Commit Graph

606 Commits

Author SHA1 Message Date
b9a899aec1
Implement switching endpoints after slow eth_getLogs RPC requests (#525)
* Switch upstream endpoint if getLogs requests are too slow

* Refactor methods for switching client to indexer

* Update codegen indexer template

* Add dummy methods in graph-node test Indexer

* Upgrade package versions to 0.2.101

---------

Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-06-20 17:57:01 +05:30
ff471da287
If subgraph block handlers exist force realtime processing in watcher (#524)
* Avoid historical sync for block handlers and eth_getLogs without event handlers

* Stop processing after endBlock in watcher fill CLI

* Codegen changes for avoiding eth_getLogs when no subgraph event handlers exist

* Upgrade package version to 0.2.100

* Check for blockHandlerExists first before block processing

* Fix comments for historical block processing checks
2024-06-20 10:05:53 +05:30
981f70ec9b
Replace null values when converting bytes to string in graph-node host API (#523)
* Replace null value in strings for postgres text data type

* Upgrade package versions to 0.2.99
2024-06-14 13:47:54 +05:30
prathamesh0
9c4b06652e
Export codegen config in a generated watcher (#522)
* Export codegen config in a generated watcher

* Update instructions with build step
2024-06-13 16:41:49 +05:30
467c173a0d
Use blockHash filter in eth_getLogs query for FEVM (#521)
* Fetch logs by blockHash at head for FEVM

* Upgrade watcher package versions to 0.2.98
2024-06-11 19:07:55 +05:30
7884941e75
Update codegen to add subgraph source to watcher readme (#514)
* Update codegen to add subgraph source to watcher readme

* Update exported metrics for watcher config

* Add steps to update package json and config to codegen instructions

---------

Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-06-11 14:42:54 +05:30
prathamesh0
acf69dd554
Add a config option for block processing offset (#520)
* Add config option for block processing offset

* Upgrade package versions
2024-06-10 15:09:33 +05:30
52082fc874
Process block with MAX_REORG_DEPTH delay from head (#519)
* Process block behind latest head by reorg height

* Upgrade package versions
2024-06-07 21:12:03 +05:30
4d81c14597
Fix order of events processing for FEVM (#518)
* Fix events processing order for FEVM

* Upgrade watcher package version

* Fix getResultEvent for eventsInRange GQL query

* Fix eventFields parsing
2024-06-07 17:23:55 +05:30
8d052add2d
Log GQL requests in watcher (#517)
* Update config for server GQL

* Add winston logger for GQL requests

* Fix codegen templates for resolver and package

* Update package versions

---------

Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-06-06 16:54:49 +05:30
836fe45aa5
Add metrics for GQL query duration (#516)
* Record GQL query durations by operation name

* Use try finally for timer metric

* Export watcher repo URL in metrics

* Remove unnecessary prefix from repo link

* Update repository label name

---------

Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-06-06 13:57:33 +05:30
7cc61579d8
Use ethers.js replace util for errors in bytesToString host API (#515)
* Use replacement strategy for ethers.utils.toUtf8String in graph-node host API

* Upgrade package versions
2024-06-04 11:41:54 +05:30
3b67fa1827
Use module read-pkg for reading package.json (#513)
* Use module read-pkg for reading package.json

* Upgrade package versions
2024-05-30 15:42:57 +05:30
f00f6d2998
Update chain head exporter CLI to support running multiple instances (#512)
* Update chain-head-exported CLI for running multiple instances

* Update upstream RPC for metrics with active endpoint

* Export metrics for watcher package version and commitHash

* Upgrade package versions
2024-05-30 14:37:35 +05:30
4fd6bf4ad7
[WIP] Handle GQL query fragment for single entity query (#511)
* Handle GQL query fragment for single entity query

* Upgrade package versions
2024-05-17 18:47:15 +05:30
b57aa76d9f
Support fragments in GQL queries for subgraph watchers (#510)
* Avoid updating latest block metrics on RPC errors

* Handle fragments in subgraph GQL queries

* Upgrade package versions

* Move private method in util graph database

---------

Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-05-17 17:20:29 +05:30
prathamesh0
1ca74548ff
Fetch job queue counts for metrics on scraping (#509) 2024-05-16 16:34:07 +05:30
20fa6ceaa6
Remove check for isFEVM flag when filtering event logs by topics (#508)
* Remove check for FEVM flag when filtering logs by topics

* Update package versions
2024-05-16 11:24:41 +05:30
prathamesh0
c7e6baa263
Add metrics to monitor errors and duration for ETH RPC requests (#507)
* Add metrics to monitor errors and duration for ETH RPC requests

* Check for server error

* Add a metric with configured upstream ETH RPC endpoints

* Use Gauge for RPC requests duration metric

* Filter out unknown events while loading event count on start

* Update package versions

* Rethrow errors in overridden send provider method
2024-05-15 19:11:22 +05:30
c9696c3d9f
Implement failover for RPC endpoints in watcher (#506)
* Handle RPC endpoint server errors and switch failover endpoints

* Add config maxNewBlockRetries for switching to failover endpoint

* Upgrade package versions

* Move unknown events removal after event processing for historical sync

* Rename doFailOverEndpoints to switchClients
2024-05-09 16:03:06 +05:30
6d837dc824
Fix address filter param in eth_getLogs request (#505)
* Fix eth_getLogs address filter param

* Upgrade package versions
2024-05-03 14:24:05 +05:30
edaec5a028
Fetch logs filtered by block number instead of block hash for FEVM (#504)
* Fetch logs by block number for FEVM and use node-fetch for eth_getLogs

* Upgrade package versions
2024-05-02 16:04:31 +05:30
prathamesh0
67425690e9
[WIP] Skip contract events if no match found in ABI (#503)
* Skip contract events if no match found in ABI

* Update package versions
2024-03-19 17:52:00 +05:30
prathamesh0
59edc178c9
Fix for generating nested filters and loading subgraph entities from db (#502)
* Handle ThunkComposer field types when generating nested filters

* Fix type setting for relational fields when loading a subgraph entity from db

* Fix loading nullable relational fields in GQL queries

* Update package versions
2024-03-14 11:21:52 +05:30
prathamesh0
ff44fd8717
Fix event processing and entities query for filecoin watchers (#501)
* Filter out pruned subgraph entities while fetching from db

* Sort events by tx index first incase upstream is FEVM

* Upgrade package versions

---------

Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
2024-03-06 11:10:41 +05:30
1cf1a3baa2
Upgrade package versions to 0.2.80 (#500) 2023-12-28 15:17:06 +05:30
33e4455f92
Fix pruning of canonical block if null block is encountered (#499) 2023-12-28 15:07:12 +05:30
a6deed9c27
Fix block processing during chain reorg (#498)
* Fix block processing during chain reorg

* Add new method in test dummy indexer

* Add missing semicolon
2023-12-28 15:06:47 +05:30
prathamesh0
78e43bc088
Export metrics for watcher config and upstream and external chain heads (#497)
* Add a cli to export chain head block numbers

* Use ETH RPC endpoint and allow env overrides

* Use ethers provider

* Export upstream chain head block number in watcher metrics

* Remove unnecessary exports

* Upgrade package versions

* Fix defaults usage

* Export watcher config in metrics

* Add metric for watcher sync mode

* Remove cache flag from watcher config metrics

* Update watcher config field names
2023-12-19 15:23:25 +05:30
prathamesh0
37732d1b1e
Upgrade package versions (#496) 2023-12-05 18:27:09 +05:30
prathamesh0
304b116c6d
Add a CLI for running a proxy server (#495) 2023-12-05 18:17:27 +05:30
ceb0a3de23
Add CLI tool to compare watcher GQL responses (#494)
* Integrate gql response compare tool

* Add README for compare-gql tool

* Update readme

* Update CLI output logs

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-28 09:51:26 +05:30
dab2d6d3e7
Publish events for GQL subscription in historical processing (#492)
* Add realtime block complete event

* Use realtime block complete event for realtime processing

* Refactor realtimeBlockComplete event interface
2023-11-27 14:13:02 +05:30
220f3ddf24
Fix events processing error check in historical processing (#491)
* Fix events queue error check in historical processing

* Upgrade package version
2023-11-26 12:42:43 +05:30
3fa3cc7068
Upgrade package versions to 0.2.76 (#490) 2023-11-24 12:04:11 +05:30
9ee423f38e
Fix update in sync status (#489) 2023-11-24 11:54:45 +05:30
6decf61d4c
Update subgraph watcher demo readme (#480)
* Update subgraph watcher demo readme

* Pass extra block data required in subgraph block handler (#486)

* Use codegen config directory name instead of file path

* Get block data from extra event data

* Add extra event data in processBlockAfterEvents

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>

* Update example subgraph mapping code for graph-cli upgrade

* Return 0 value for block size in ipld-eth-client

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2023-11-24 11:13:40 +05:30
937aed139f
Fix sync status update in historical processing job (#488)
* Fill watcher start block only in active mode

* Fix sync status update in historical processing

* Upgrade package versions

* Upgrade package versions in codegen template

* Fix graph-node test indexer
2023-11-23 20:25:39 +05:30
prathamesh0
ffd1d267d8
Implement remaining json host APIs (#487)
* Instantiate a new wasm string to convert JSON value to BigInt

* Implement remaining json host APIs

* Upgrade graph-ts and graph-cli in test example subgraph

* Handle null context for dataSource context host API

* Use JSONBig for parsing JSON strings
2023-11-22 18:03:21 +05:30
729d862aac
Fix fetching blocks near chain head in job-runner (#485)
* Fix null blocks check in job-runner when fetching near chain head

* Upgrade package versions
2023-11-21 17:44:54 +05:30
prathamesh0
e6b957e921
Mark entities as removed on store remove host API call (#484) 2023-11-21 16:00:26 +05:30
8c44fbbe6d
Add gqlPath in config for GQL server URL (#483) 2023-11-21 15:21:10 +05:30
4a88ff76e0
Remove check for blocks count in eventsInRange query and check sync status instead (#482)
* Add check for from and to block in eventsInRange query

* Update sync status query for more fields

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-21 14:14:36 +05:30
prathamesh0
d0aca37a74
Skip meta query generation for non-subgraph watchers (#481)
* Skip generating meta query if non subgraph watcher

* Fix linting errors in generated watcher

* Add space for meta query

* Add meta query types after adding block height type

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-21 11:44:30 +05:30
7b19d383ec
Run block fill for watcher automatically (#479)
* Find minimum start block number

* Perform fill if db is empty

* Make getStartBlock method public in GraphWatcher

* Use dataSources property

* Update server template to fill block on start

* Avoid passing graphWatcher from generated watcher

* Ensure block filling for non-subgraph watchers as well

* Remove getStartBlock method from GraphWatcher

* Remove graphWatcher property from ServerCmd

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-20 17:13:49 +05:30
07c0827a2a
Automate steps in codegen to build subgraph (#478)
* Use shelljs to run graph-cli codegen command

* Replace graph-cli and graph-ts with cerc-io forks

* Add steps to build subgraph

* Add config packageManager for different subgraph repo setup

* Copy over the subgraph build to generated watcher

* Remove TODO

* Change subgraph path in generated watcher config
2023-11-20 17:09:51 +05:30
prathamesh0
2faf905d99
Implement remaining graph-node host APIs for templates (#475)
* Implement createWithContext graph-node dataSource host API

* Parse individual context entries

* Implement graph-node dataSource context host API

* Handle array type fields and refactor code

* Resolve all values when parsing an array field

* Fix wasm entity creation with array type fields

* Required codegen changes
2023-11-20 15:02:08 +05:30
c2070a80cb
Handle null block error in rpc-eth-client (#477)
* Remove block-size-cache util methods as full block with size fetched already

* Handle null block error in rpc-eth-client

* Upgrade package versions
2023-11-17 11:58:43 +05:30
e54f4c9276
Implement host API for big decimal equals (#476)
* Implement host api for big decimal equals

* Add test cases for big decimal equals

* Use boolean type conversion

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-17 10:26:03 +05:30
7c8f84b1dc
Remove prefetchBlocksInMem flag along with functionality (#474)
* Remove prefetchBlocksInMem flag

* Remove Indexer method fetchEventsAndSaveBlocks

* Throw error if pruning not safe

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-16 19:03:13 +05:30