Commit Graph
568 Commits
Author SHA1 Message Date
nabarun edf72c1dd6 Update indexer template in codegen for block optimization changes (#464)
* Update indexer template for block optimization

* Trim address strings in graph-node host API
2023-11-10 12:14:38 +05:30
prathamesh0 32d5cd10d0 Prioritize block hash over number in singular time travel queries (#462)
* Keep return value for singular subgraph entity queries nullable

* Prioritize block hash over number in time travel queries

* Throw error when block with given hash doesn't exist in db
2023-11-10 09:39:58 +05:30
nabarun 695723955f Prefetch block and txs in historical processing instead of fetching them in events processing (#460)
* Set gzip true in ethersjs provider

* Add timer logs and use StaticJsonRpcProvider

* Fetch block data in historical processing and cache in map

* Fetch txs required for event logs in historical processing

* Process events with prefetched block and txs data in realtime processing

* Clear old TODOs
2023-11-09 18:42:37 +05:30
prathamesh0 c7bcd4c276 Generate resolvers for plural subgraph GQL queries (#461) 2023-11-09 16:33:35 +05:30
prathamesh0 6860f1a26f Handle tsvector type columns when forming the db query (#459)
* Support full text search queries

* Handle text search for all query types
2023-11-09 14:15:31 +05:30
prathamesh0andneeraj 483a73fc3e Load schema file path from subgraph config (#458)
* Get subgraph schema from config file

* Use default value if schema not given

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-09 12:17:09 +05:30
nabarun 5aa98326cb Send transaction data directly in rpc-eth-client instead of serialization (#457)
* Fix tx serialization using ethers v6

* Send tx data directly in rpc-eth-client instead of serializing it
2023-11-08 17:32:26 +05:30
nabarun f2c5f67777 Handle restarts during historical processing in watcher (#455)
* Reset to latest processed block on restarting job-runner

* Update sync status during historical processing in job-runner

* Codegen changes

* Use sync status latest processed block for subgraph _meta GQL query

* Set job per interval for subscribing events queue to 1

* Fix events processing skipped for blocks after template create
2023-11-08 14:28:53 +05:30
prathamesh0 97bd4014d6 Support nested entity sorting in plural GQL queries (#456)
* Move method to apply block filter to base database

* Handle nested entity sorting

* Handle nested entity sorting for distinct on type queries

* Handle nested entity sorting for all query types
2023-11-08 12:20:32 +05:30
prathamesh0 5efab298a3 Add support for meta query in watcher GQL API (#453)
* Add a method to get meta data for watcher indexing status

* Add a flag indicating indexing error to sync status

* Codegen changes

* Clear indexing error on job-runner startup

* Fix lint errors
2023-11-07 14:37:05 +05:30
nabarun 8547876764 Add isFEVM flag in config to avoid filtering event logs by topics (#454)
* Pass upstream config to indexer instance

* Add isFEVM flag and refactor watcher config fields

* Codegen changes for indexer

* Add missing getter in dummy indexer for graph-node tests
2023-11-07 12:07:49 +05:30
nabarun dd92b4feb2 Fix bigint and decimal transformers not added for null type entity field in codegen (#451) 2023-11-06 15:01:48 +05:30
prathamesh0 d5c92aa150 Add BigInt and BigDecimal GraphQL scalars (#452) 2023-11-06 15:00:24 +05:30
nabarun 8a720ef175 Wait for events processing to complete before continuing historical blocks processing (#450)
* Wait for events queue to be empty before continuing historical processing

* Make historicalLogsBlockRange and historicalMaxFetchAhead configurable

* Perform single RPC request for multiple addresses
2023-11-06 11:34:48 +05:30
prathamesh0 546af92638 Support logical filter operations in plural GQL queries (#449)
* Handle logical operators in where input

* Handle logical filter operators while building the database query

* Refactor code to build where clause for database query
2023-11-06 11:17:54 +05:30
nabarun 0d7e3ddc8b Handle subgraph template create when filterLogsByAddresses set to true (#447)
* Handle subgraph template create when filterLogsByAddresses set to true

* Fix historical processing stopping after running for multiple batches

* Add new method in graph-node test dummy indexer
2023-11-03 14:08:55 +05:30
prathamesh0 7f37dac888 Support filters for array and derived type fields in plural GQL queries (#448)
* Handle nested filters for array type fields

* Transform Decimal param values to strings for db queries

* Handle nested filters for derived fields

* Handle filters for array type fields
2023-11-03 12:14:32 +05:30
prathamesh0 92f3fb8252 Support nested filters for plural queries on subgraph entities (#446)
* Handle nested filters while building the database query

* Build nested filters for relational fields

* Avoid queries to get canonical block hashes for loading relations

* Handle nested filters in all query types

* Give precedence to block hash over number in time travel queries
2023-11-02 10:37:33 +05:30
nabarun 9fb51e89f6 Implement fetch and save of filtered event logs and required blocks (#445)
* Fix async block size caching for missing blocks in historical processing

* Start historical block processing only if filter logs is set to true

* Fetch filtered logs by topics and save required blocks

* Fix realtime processing start block after historical processing

* Avoid publishing events and blocks in historical processing

* Add new method to graph-node test indexer

* Get full block data for subgraph block handler only if configured

* Add useBlockRanges flag for switching between historical and realtime processing
2023-11-01 19:07:06 +05:30
prathamesh0 1b6ca6edeb Implement block filter for plural queries on subgraph entities (#444)
* Fix bigint values transformation

* Fix starts and ends filter operator resolution

* Support case insensitive filters for string fields

* Add support for global filter _change_block

* Handle _change_block filter in all query types
2023-11-01 10:42:56 +05:30
nabarun 6c17662cad Add new job queue for historical blocks processing (#442)
* Add TODOs for historical blocks processing

* Add new job for historical blocks processing

* Handle historical job completion

* Fetch latest block in chain and start historical block processing

* Fix starting realtime block processing from latest canonical block

* Refactor historical block processing method and add logs

* Add dummy indexer methods in graph-node to pass test

* Changes in codegen for historical processing in generated watcher
2023-11-01 10:38:59 +05:30
prathamesh0 15ee523e71 Generate plural GQL query schema for subgraph entities (#443)
* Generate plural GQL queries for subgraph entities

* Use pluralize for plural query names

* Handle bigint values for filters in db query
2023-10-31 09:24:49 +05:30
prathamesh0 a8e59eb6b9 Support filtering by topics when fetching logs in rpc-eth-client (#441)
* Support filtering by topics when fetching logs

* Include event signatures from all contracts in topics list

* Refactor common code

* Update package versions
v0.2.67
2023-10-26 17:25:56 +05:30
nabarun 43463af1f2 Avoid updating StateSyncStatus table when enableState flag is set to false (#440)
* Handle zero hash canonical block incase of FEVM null block

* Fix json-bigint parse in processBatchEvents

* Avoid updating SyncStatus table when enableState is false
2023-10-26 15:14:46 +05:30
prathamesh0 445d5a9293 Process template events at last when following subgraph events order (#439)
* Process template events at last when following subgraph events order

* Update mock test indexer
2023-10-26 12:05:29 +05:30
nabarun b63a93d8a0 Skip checking of events order while processing due to incorrect logIndex in FEVM (#438)
* Skip log index order in events processing

* Use json-bigint stringify in processBatchEvents

* Set eventSignature in event processing for contracts watched later

* Check for null block in async caching of block sizes
2023-10-26 11:21:48 +05:30
prathamesh0 8bba0aeb94 Handle null blocks for FEVM (#437)
* Handle null blocks in chain pruning

* Skip FEVM null blocks while prefetching

* Fix BaseFee assignment while fetching block in rpc-eth-client

* Convert blockNumber field to number type before saving block to db

* Avoid saving blocks after null block twice
2023-10-25 18:37:02 +05:30
prathamesh0 320bf02938 Fetch txs only for blocks having relevant events (#436)
* Fix address filter usage in rpc-eth-client getLogs implementation

* Avoid fetching block txs when no relevant logs exist
2023-10-25 14:36:03 +05:30
nabarun 07887c160e Use block number for eth_call in rpc-eth-client (#435)
* Update subgraph readme to run fill before job-runner

* Fix getContractEntitiesMap incase of template data sources

* Use rpcSupportsBlockHashParam flag to use blockNumber for rpc client eth_call

* Fix optional baseFeePerGas in rpc-eth-client

* Fix graph-node tests after changes

* Remove completed TODO
2023-10-25 11:04:12 +05:30
prathamesh0 e8d8476bef Fetch event logs for a block range in a single upstream call (#433)
* Support fetching logs for a block range in rpc-eth-client

* Add a method to fetch block events for multiple blocks at once

* Add a method to save blocks with fetched events in a block range

* Fix transactions destructuring

* Fix get logs call args

* Add a separate ETH client method to get logs in a block range

* Codegen changes
2023-10-23 16:10:09 +05:30
nabarun 45b7489115 Update codegen reamde for subgraph build using cerc-io dependencies (#434) 2023-10-23 12:21:38 +05:30
nabarun f8d5404c25 Fix codegen for creating subgraph watcher for Sushiswap (#432)
* Fix EthClient import in codegen and graph-node package

* Fix codegen for resolving common events between contracts
2023-10-23 09:23:20 +05:30
prathamesh0andneeraj 72ccaa0381 Upgrade nitro-node dependency (#430)
* Upgrade nitro-node to v0.1.15

* Upgrade packages version to v0.2.66

* Add default value for Nitro node store path

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
v0.2.66
2023-10-17 18:39:08 +05:30
nabarunandShreerang Kale dc06ada7c4 Add validation methods for watcher config (#425)
* Add validation for config file

* Add support for websocket endpoints to be checked for contract deployment

* Add flag to toggle validation

* Update log message for paid rpc method validation

* Update log message for validation

* Rename enableValidation to enableConfigValidation

* Check deployment only if contract address is in a valid format

* Update httpValidateEndpoint method to check if endpoint is up instead of response as ok

* Update log messages

* Update log messages

---------

Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
2023-10-16 16:39:21 +05:30
nabarunandneeraj 75b177cffd Add useDurableStore flag in Nitro setupNode (#428)
* Add useDurableStore flag in seUpNode

* Upgrade nitro-node to 0.1.14

* Change payment amount to string

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-10-13 12:47:23 +05:30
prathamesh0andShreerang Kale e6767c3aca Pass vouchers for RPC requests in header (#429)
* Send voucher in request header instead of url params

* Handle review changes

* Update package versions

---------

Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
v0.2.65
2023-10-12 17:42:19 +05:30
nabarun 2b83f38ab6 Upgrade package version to 0.2.64 (#427)
* Upgrade package version to 0.2.64

* Comment CI workflow to publish docker image

* Update CI workflow names
v0.2.64
2023-10-12 11:43:17 +05:30
nabarun 8515630f49 Add validateGQLRequest method in PaymentsManager (#426)
* Refactor GQL validation for payment

* Export payment util const PAYMENT_HEADER_KEY

* Check payment header only for paid queries
2023-10-12 10:58:13 +05:30
prathamesh0 caa8da7090 Upgrade nitro-node and add Nitro config option for chain URL (#424)
* Upgrade @cerc-io/nitro-node dependency

* Add a separate config option for chain URL used by Nitro node

* Avoid adding ETH RPC method to request URL as no longer needed

* Update package versions
v0.2.63
2023-10-05 11:28:13 +05:30
prathamesh0andShreerang Kale ae60bbe1ce Setup config for paid RPC methods and funding amounts with upstream ETH server (#423)
* Add config option for paid RPC methods

* Change rpcProviderNitroNode to nitro

* Add config option for funding amounts

* Handle review changes

* Refactor config and add a method to setup provider with payments

* Upgrade package versions

---------

Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
v0.2.62
2023-10-04 16:10:54 +05:30
prathamesh0 e203154ac7 Support payments to upstream ETH server for RPC requests (#418)
* Add config option for upstream RPC mutation endpoint

* Use mutation endpoint as chain URL for Nitro node

* Add a method to setup an upstream payment channel

* Add a method to send payments to upstream Nitro node

* Upgrade package versions
v0.2.61
2023-09-28 15:24:45 +05:30
prathamesh0andShreerang Kale 32c78a907c Add a config option to set gossipub direct peers (#414)
* Add support for setting gossipsub direct peers

* Use string array for direct peers config

* Handle empty nitro and consensus config

* Increase consensus write retry attempts

* Rename config option for watcher party peers file

* Handle directPeers array being passed undefined

* Upgrade package versions

---------

Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
v0.2.60
2023-09-22 13:46:20 +05:30
prathamesh0 48e7785367 Upgrade nitro-node package (#415)
* Upgrade nitro-node package

* Update package versions
v0.2.59
2023-09-21 18:32:03 +05:30
nabarunandneeraj 77312a2f95 Add yamux stream multiplexer in peer package (#413)
* Add yamux muxer to peer

* Upgrade version to 0.2.58

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
v0.2.58
2023-09-15 12:19:51 +05:30
prathamesh0 bd73dae1b1 Implement consensus mechanism using mokka (#412)
* Add Consensus class with placeholders

* Implement Consensus constructor

* Move PubsubType to peer package

* Implement remaining methods in Consensus class

* Use existing consensus stream if it exists

* Setup send and receive pipes on consensus streams

* Refactor P2P and consensus setup in server command

* Add Nitro node initialization in server command

* Return objects from server initializations

* Use dynamic imports for ES modules in util package

* Fix util deps

* Change target to es6 to allow creating a Mokka instance

* Set moduleResolution to node16 in util for dynamic imports

* Upgrade @cerc-io/nitro-node package

* Implement retries while sending consensus messages

* Use bunyan for consensus logs and subscribe to state changes

* Use debug for logging state change events

* Handle empty watcher party file path

* Return object from initP2P

* Upgrade @cerc-io/nitro-node package

* Update package versions
v0.2.57
2023-09-14 13:47:31 +05:30
prathamesh0 c80e4d0ab7 Add support for gossipsub (#410)
* Replace floodsub with gossipsub

* Add a pubsub flag to support both floodsub and gossipsub

* Fix deps

* Handle empty string values

* Upgrade @cerc-io/libp2p dependency

* Upgrade package versions
v0.2.56
2023-09-05 17:54:52 +05:30
nabarun f89eea0d2f Upgrade package versions (#407) v0.2.55 2023-08-30 15:55:35 +05:30
nabarunandneeraj 1d3bbf89c8 Update graph-cli version to fix concat-stream dependency issue (#405)
* Update yarn lock

* Upgrade @cerc-io/graph-cli version to 0.22.4-watcher-ts-0.1.2

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-08-25 14:26:00 +05:30
nabarunandneeraj 444901eb87 Upgrade package versions to 0.2.54 (#403)
Co-authored-by: neeraj <neeraj.rtly@gmail.com>
v0.2.54
2023-08-16 19:07:30 +05:30
nabarunandneeraj f2bc054c19 Rename nitro-client to nitro-node (#402)
* Rename nitro-client to nitro-node

* Upgrade package versions to 0.2.53

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
v0.2.53
2023-08-16 17:57:15 +05:30