Commit Graph

226 Commits

Author SHA1 Message Date
f00f6d2998
Update chain head exporter CLI to support running multiple instances (#512)
* Update chain-head-exported CLI for running multiple instances

* Update upstream RPC for metrics with active endpoint

* Export metrics for watcher package version and commitHash

* Upgrade package versions
2024-05-30 14:37:35 +05:30
4fd6bf4ad7
[WIP] Handle GQL query fragment for single entity query (#511)
* Handle GQL query fragment for single entity query

* Upgrade package versions
2024-05-17 18:47:15 +05:30
b57aa76d9f
Support fragments in GQL queries for subgraph watchers (#510)
* Avoid updating latest block metrics on RPC errors

* Handle fragments in subgraph GQL queries

* Upgrade package versions

* Move private method in util graph database

---------

Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-05-17 17:20:29 +05:30
20fa6ceaa6
Remove check for isFEVM flag when filtering event logs by topics (#508)
* Remove check for FEVM flag when filtering logs by topics

* Update package versions
2024-05-16 11:24:41 +05:30
prathamesh0
c7e6baa263
Add metrics to monitor errors and duration for ETH RPC requests (#507)
* Add metrics to monitor errors and duration for ETH RPC requests

* Check for server error

* Add a metric with configured upstream ETH RPC endpoints

* Use Gauge for RPC requests duration metric

* Filter out unknown events while loading event count on start

* Update package versions

* Rethrow errors in overridden send provider method
2024-05-15 19:11:22 +05:30
c9696c3d9f
Implement failover for RPC endpoints in watcher (#506)
* Handle RPC endpoint server errors and switch failover endpoints

* Add config maxNewBlockRetries for switching to failover endpoint

* Upgrade package versions

* Move unknown events removal after event processing for historical sync

* Rename doFailOverEndpoints to switchClients
2024-05-09 16:03:06 +05:30
6d837dc824
Fix address filter param in eth_getLogs request (#505)
* Fix eth_getLogs address filter param

* Upgrade package versions
2024-05-03 14:24:05 +05:30
edaec5a028
Fetch logs filtered by block number instead of block hash for FEVM (#504)
* Fetch logs by block number for FEVM and use node-fetch for eth_getLogs

* Upgrade package versions
2024-05-02 16:04:31 +05:30
prathamesh0
67425690e9
[WIP] Skip contract events if no match found in ABI (#503)
* Skip contract events if no match found in ABI

* Update package versions
2024-03-19 17:52:00 +05:30
prathamesh0
59edc178c9
Fix for generating nested filters and loading subgraph entities from db (#502)
* Handle ThunkComposer field types when generating nested filters

* Fix type setting for relational fields when loading a subgraph entity from db

* Fix loading nullable relational fields in GQL queries

* Update package versions
2024-03-14 11:21:52 +05:30
prathamesh0
ff44fd8717
Fix event processing and entities query for filecoin watchers (#501)
* Filter out pruned subgraph entities while fetching from db

* Sort events by tx index first incase upstream is FEVM

* Upgrade package versions

---------

Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
2024-03-06 11:10:41 +05:30
1cf1a3baa2
Upgrade package versions to 0.2.80 (#500) 2023-12-28 15:17:06 +05:30
33e4455f92
Fix pruning of canonical block if null block is encountered (#499) 2023-12-28 15:07:12 +05:30
a6deed9c27
Fix block processing during chain reorg (#498)
* Fix block processing during chain reorg

* Add new method in test dummy indexer

* Add missing semicolon
2023-12-28 15:06:47 +05:30
prathamesh0
78e43bc088
Export metrics for watcher config and upstream and external chain heads (#497)
* Add a cli to export chain head block numbers

* Use ETH RPC endpoint and allow env overrides

* Use ethers provider

* Export upstream chain head block number in watcher metrics

* Remove unnecessary exports

* Upgrade package versions

* Fix defaults usage

* Export watcher config in metrics

* Add metric for watcher sync mode

* Remove cache flag from watcher config metrics

* Update watcher config field names
2023-12-19 15:23:25 +05:30
prathamesh0
37732d1b1e
Upgrade package versions (#496) 2023-12-05 18:27:09 +05:30
220f3ddf24
Fix events processing error check in historical processing (#491)
* Fix events queue error check in historical processing

* Upgrade package version
2023-11-26 12:42:43 +05:30
3fa3cc7068
Upgrade package versions to 0.2.76 (#490) 2023-11-24 12:04:11 +05:30
6decf61d4c
Update subgraph watcher demo readme (#480)
* Update subgraph watcher demo readme

* Pass extra block data required in subgraph block handler (#486)

* Use codegen config directory name instead of file path

* Get block data from extra event data

* Add extra event data in processBlockAfterEvents

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>

* Update example subgraph mapping code for graph-cli upgrade

* Return 0 value for block size in ipld-eth-client

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2023-11-24 11:13:40 +05:30
937aed139f
Fix sync status update in historical processing job (#488)
* Fill watcher start block only in active mode

* Fix sync status update in historical processing

* Upgrade package versions

* Upgrade package versions in codegen template

* Fix graph-node test indexer
2023-11-23 20:25:39 +05:30
prathamesh0
ffd1d267d8
Implement remaining json host APIs (#487)
* Instantiate a new wasm string to convert JSON value to BigInt

* Implement remaining json host APIs

* Upgrade graph-ts and graph-cli in test example subgraph

* Handle null context for dataSource context host API

* Use JSONBig for parsing JSON strings
2023-11-22 18:03:21 +05:30
729d862aac
Fix fetching blocks near chain head in job-runner (#485)
* Fix null blocks check in job-runner when fetching near chain head

* Upgrade package versions
2023-11-21 17:44:54 +05:30
prathamesh0
e6b957e921
Mark entities as removed on store remove host API call (#484) 2023-11-21 16:00:26 +05:30
prathamesh0
2faf905d99
Implement remaining graph-node host APIs for templates (#475)
* Implement createWithContext graph-node dataSource host API

* Parse individual context entries

* Implement graph-node dataSource context host API

* Handle array type fields and refactor code

* Resolve all values when parsing an array field

* Fix wasm entity creation with array type fields

* Required codegen changes
2023-11-20 15:02:08 +05:30
c2070a80cb
Handle null block error in rpc-eth-client (#477)
* Remove block-size-cache util methods as full block with size fetched already

* Handle null block error in rpc-eth-client

* Upgrade package versions
2023-11-17 11:58:43 +05:30
e54f4c9276
Implement host API for big decimal equals (#476)
* Implement host api for big decimal equals

* Add test cases for big decimal equals

* Use boolean type conversion

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-17 10:26:03 +05:30
7c8f84b1dc
Remove prefetchBlocksInMem flag along with functionality (#474)
* Remove prefetchBlocksInMem flag

* Remove Indexer method fetchEventsAndSaveBlocks

* Throw error if pruning not safe

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-16 19:03:13 +05:30
3e898914ba
Add GQL where filter and nested order by params based on entity fields (#472)
* Add where filter equals fields based on entity fields

* Add different where filter operations for entity fields

* Add nested where filter for relation fields and filter params or & and

* Remove filters not required for derived fields

* Add nested order by params in schema

* Throw error on trying to sort by array or derived type fields

* Handle array fields while generating order by and where clauses

* Upgrade package versions

---------

Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2023-11-16 12:38:47 +05:30
5b95335fde
Filter logs by events specified in subgraph config (#473)
* Use topic hash of events from config file

* Add eventSignaturesMap field in IndexerInterface

* Add eventSignaturesMap property in Indexer class

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-16 11:54:49 +05:30
6ce8d4746d
Upgrade package versions before release (#469) 2023-11-14 15:01:52 +05:30
803f7c3b3d
Fix retry on error for events processing job with exponential backoff (#468)
* Throw error in job-runner events processing queue for retry

* Restart block processing after completion of retried events queue job

* Check events queue size in retry state from active historical processing job

* Clear watched contracts from memory map on reset

* Update sync status with zero hash during historical processing

* Update codegen for changes

* Add clearProcessedBlockData in graph-node test dummy indexer
2023-11-14 14:53:23 +05:30
7b5fbf3d13
Implement new graph-node host API for BigDecimal exponentiation (#467)
* Implement host API for BigDecimal pow

* Handle FEVM future block error in rpc-eth-client getFullBlocks

* Handle future block error in rpc-eth-client getBlockWithTransactions

* Upgrade package versions
2023-11-13 12:22:23 +05:30
66d613767e
Upgrade version to 0.2.69 (#466) 2023-11-10 19:36:21 +05:30
prathamesh0
be24166ea7
Update package versions (#463) 2023-11-10 14:48:35 +05:30
edf72c1dd6
Update indexer template in codegen for block optimization changes (#464)
* Update indexer template for block optimization

* Trim address strings in graph-node host API
2023-11-10 12:14:38 +05:30
695723955f
Prefetch block and txs in historical processing instead of fetching them in events processing (#460)
* Set gzip true in ethersjs provider

* Add timer logs and use StaticJsonRpcProvider

* Fetch block data in historical processing and cache in map

* Fetch txs required for event logs in historical processing

* Process events with prefetched block and txs data in realtime processing

* Clear old TODOs
2023-11-09 18:42:37 +05:30
f2c5f67777
Handle restarts during historical processing in watcher (#455)
* Reset to latest processed block on restarting job-runner

* Update sync status during historical processing in job-runner

* Codegen changes

* Use sync status latest processed block for subgraph _meta GQL query

* Set job per interval for subscribing events queue to 1

* Fix events processing skipped for blocks after template create
2023-11-08 14:28:53 +05:30
prathamesh0
5efab298a3
Add support for meta query in watcher GQL API (#453)
* Add a method to get meta data for watcher indexing status

* Add a flag indicating indexing error to sync status

* Codegen changes

* Clear indexing error on job-runner startup

* Fix lint errors
2023-11-07 14:37:05 +05:30
8547876764
Add isFEVM flag in config to avoid filtering event logs by topics (#454)
* Pass upstream config to indexer instance

* Add isFEVM flag and refactor watcher config fields

* Codegen changes for indexer

* Add missing getter in dummy indexer for graph-node tests
2023-11-07 12:07:49 +05:30
prathamesh0
546af92638
Support logical filter operations in plural GQL queries (#449)
* Handle logical operators in where input

* Handle logical filter operators while building the database query

* Refactor code to build where clause for database query
2023-11-06 11:17:54 +05:30
0d7e3ddc8b
Handle subgraph template create when filterLogsByAddresses set to true (#447)
* Handle subgraph template create when filterLogsByAddresses set to true

* Fix historical processing stopping after running for multiple batches

* Add new method in graph-node test dummy indexer
2023-11-03 14:08:55 +05:30
prathamesh0
92f3fb8252
Support nested filters for plural queries on subgraph entities (#446)
* Handle nested filters while building the database query

* Build nested filters for relational fields

* Avoid queries to get canonical block hashes for loading relations

* Handle nested filters in all query types

* Give precedence to block hash over number in time travel queries
2023-11-02 10:37:33 +05:30
9fb51e89f6
Implement fetch and save of filtered event logs and required blocks (#445)
* Fix async block size caching for missing blocks in historical processing

* Start historical block processing only if filter logs is set to true

* Fetch filtered logs by topics and save required blocks

* Fix realtime processing start block after historical processing

* Avoid publishing events and blocks in historical processing

* Add new method to graph-node test indexer

* Get full block data for subgraph block handler only if configured

* Add useBlockRanges flag for switching between historical and realtime processing
2023-11-01 19:07:06 +05:30
prathamesh0
1b6ca6edeb
Implement block filter for plural queries on subgraph entities (#444)
* Fix bigint values transformation

* Fix starts and ends filter operator resolution

* Support case insensitive filters for string fields

* Add support for global filter _change_block

* Handle _change_block filter in all query types
2023-11-01 10:42:56 +05:30
6c17662cad
Add new job queue for historical blocks processing (#442)
* Add TODOs for historical blocks processing

* Add new job for historical blocks processing

* Handle historical job completion

* Fetch latest block in chain and start historical block processing

* Fix starting realtime block processing from latest canonical block

* Refactor historical block processing method and add logs

* Add dummy indexer methods in graph-node to pass test

* Changes in codegen for historical processing in generated watcher
2023-11-01 10:38:59 +05:30
prathamesh0
a8e59eb6b9
Support filtering by topics when fetching logs in rpc-eth-client (#441)
* Support filtering by topics when fetching logs

* Include event signatures from all contracts in topics list

* Refactor common code

* Update package versions
2023-10-26 17:25:56 +05:30
43463af1f2
Avoid updating StateSyncStatus table when enableState flag is set to false (#440)
* Handle zero hash canonical block incase of FEVM null block

* Fix json-bigint parse in processBatchEvents

* Avoid updating SyncStatus table when enableState is false
2023-10-26 15:14:46 +05:30
prathamesh0
445d5a9293
Process template events at last when following subgraph events order (#439)
* Process template events at last when following subgraph events order

* Update mock test indexer
2023-10-26 12:05:29 +05:30
07887c160e
Use block number for eth_call in rpc-eth-client (#435)
* Update subgraph readme to run fill before job-runner

* Fix getContractEntitiesMap incase of template data sources

* Use rpcSupportsBlockHashParam flag to use blockNumber for rpc client eth_call

* Fix optional baseFeePerGas in rpc-eth-client

* Fix graph-node tests after changes

* Remove completed TODO
2023-10-25 11:04:12 +05:30
prathamesh0
e8d8476bef
Fetch event logs for a block range in a single upstream call (#433)
* Support fetching logs for a block range in rpc-eth-client

* Add a method to fetch block events for multiple blocks at once

* Add a method to save blocks with fetched events in a block range

* Fix transactions destructuring

* Fix get logs call args

* Add a separate ETH client method to get logs in a block range

* Codegen changes
2023-10-23 16:10:09 +05:30