Commit Graph

101 Commits

Author SHA1 Message Date
7b19d383ec
Run block fill for watcher automatically (#479)
* Find minimum start block number

* Perform fill if db is empty

* Make getStartBlock method public in GraphWatcher

* Use dataSources property

* Update server template to fill block on start

* Avoid passing graphWatcher from generated watcher

* Ensure block filling for non-subgraph watchers as well

* Remove getStartBlock method from GraphWatcher

* Remove graphWatcher property from ServerCmd

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-11-20 17:13:49 +05:30
c2070a80cb
Handle null block error in rpc-eth-client (#477)
* Remove block-size-cache util methods as full block with size fetched already

* Handle null block error in rpc-eth-client

* Upgrade package versions
2023-11-17 11:58:43 +05:30
3e898914ba
Add GQL where filter and nested order by params based on entity fields (#472)
* Add where filter equals fields based on entity fields

* Add different where filter operations for entity fields

* Add nested where filter for relation fields and filter params or & and

* Remove filters not required for derived fields

* Add nested order by params in schema

* Throw error on trying to sort by array or derived type fields

* Handle array fields while generating order by and where clauses

* Upgrade package versions

---------

Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2023-11-16 12:38:47 +05:30
6ce8d4746d
Upgrade package versions before release (#469) 2023-11-14 15:01:52 +05:30
7b5fbf3d13
Implement new graph-node host API for BigDecimal exponentiation (#467)
* Implement host API for BigDecimal pow

* Handle FEVM future block error in rpc-eth-client getFullBlocks

* Handle future block error in rpc-eth-client getBlockWithTransactions

* Upgrade package versions
2023-11-13 12:22:23 +05:30
66d613767e
Upgrade version to 0.2.69 (#466) 2023-11-10 19:36:21 +05:30
prathamesh0
be24166ea7
Update package versions (#463) 2023-11-10 14:48:35 +05:30
695723955f
Prefetch block and txs in historical processing instead of fetching them in events processing (#460)
* Set gzip true in ethersjs provider

* Add timer logs and use StaticJsonRpcProvider

* Fetch block data in historical processing and cache in map

* Fetch txs required for event logs in historical processing

* Process events with prefetched block and txs data in realtime processing

* Clear old TODOs
2023-11-09 18:42:37 +05:30
f2c5f67777
Handle restarts during historical processing in watcher (#455)
* Reset to latest processed block on restarting job-runner

* Update sync status during historical processing in job-runner

* Codegen changes

* Use sync status latest processed block for subgraph _meta GQL query

* Set job per interval for subscribing events queue to 1

* Fix events processing skipped for blocks after template create
2023-11-08 14:28:53 +05:30
prathamesh0
5efab298a3
Add support for meta query in watcher GQL API (#453)
* Add a method to get meta data for watcher indexing status

* Add a flag indicating indexing error to sync status

* Codegen changes

* Clear indexing error on job-runner startup

* Fix lint errors
2023-11-07 14:37:05 +05:30
8547876764
Add isFEVM flag in config to avoid filtering event logs by topics (#454)
* Pass upstream config to indexer instance

* Add isFEVM flag and refactor watcher config fields

* Codegen changes for indexer

* Add missing getter in dummy indexer for graph-node tests
2023-11-07 12:07:49 +05:30
dd92b4feb2
Fix bigint and decimal transformers not added for null type entity field in codegen (#451) 2023-11-06 15:01:48 +05:30
8a720ef175
Wait for events processing to complete before continuing historical blocks processing (#450)
* Wait for events queue to be empty before continuing historical processing

* Make historicalLogsBlockRange and historicalMaxFetchAhead configurable

* Perform single RPC request for multiple addresses
2023-11-06 11:34:48 +05:30
9fb51e89f6
Implement fetch and save of filtered event logs and required blocks (#445)
* Fix async block size caching for missing blocks in historical processing

* Start historical block processing only if filter logs is set to true

* Fetch filtered logs by topics and save required blocks

* Fix realtime processing start block after historical processing

* Avoid publishing events and blocks in historical processing

* Add new method to graph-node test indexer

* Get full block data for subgraph block handler only if configured

* Add useBlockRanges flag for switching between historical and realtime processing
2023-11-01 19:07:06 +05:30
prathamesh0
a8e59eb6b9
Support filtering by topics when fetching logs in rpc-eth-client (#441)
* Support filtering by topics when fetching logs

* Include event signatures from all contracts in topics list

* Refactor common code

* Update package versions
2023-10-26 17:25:56 +05:30
prathamesh0
445d5a9293
Process template events at last when following subgraph events order (#439)
* Process template events at last when following subgraph events order

* Update mock test indexer
2023-10-26 12:05:29 +05:30
prathamesh0
72ccaa0381
Upgrade nitro-node dependency (#430)
* Upgrade nitro-node to v0.1.15

* Upgrade packages version to v0.2.66

* Add default value for Nitro node store path

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-10-17 18:39:08 +05:30
75b177cffd
Add useDurableStore flag in Nitro setupNode (#428)
* Add useDurableStore flag in seUpNode

* Upgrade nitro-node to 0.1.14

* Change payment amount to string

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-10-13 12:47:23 +05:30
prathamesh0
e6767c3aca
Pass vouchers for RPC requests in header (#429)
* Send voucher in request header instead of url params

* Handle review changes

* Update package versions

---------

Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
2023-10-12 17:42:19 +05:30
2b83f38ab6
Upgrade package version to 0.2.64 (#427)
* Upgrade package version to 0.2.64

* Comment CI workflow to publish docker image

* Update CI workflow names
2023-10-12 11:43:17 +05:30
prathamesh0
caa8da7090
Upgrade nitro-node and add Nitro config option for chain URL (#424)
* Upgrade @cerc-io/nitro-node dependency

* Add a separate config option for chain URL used by Nitro node

* Avoid adding ETH RPC method to request URL as no longer needed

* Update package versions
2023-10-05 11:28:13 +05:30
prathamesh0
ae60bbe1ce
Setup config for paid RPC methods and funding amounts with upstream ETH server (#423)
* Add config option for paid RPC methods

* Change rpcProviderNitroNode to nitro

* Add config option for funding amounts

* Handle review changes

* Refactor config and add a method to setup provider with payments

* Upgrade package versions

---------

Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
2023-10-04 16:10:54 +05:30
prathamesh0
e203154ac7
Support payments to upstream ETH server for RPC requests (#418)
* Add config option for upstream RPC mutation endpoint

* Use mutation endpoint as chain URL for Nitro node

* Add a method to setup an upstream payment channel

* Add a method to send payments to upstream Nitro node

* Upgrade package versions
2023-09-28 15:24:45 +05:30
prathamesh0
32c78a907c
Add a config option to set gossipub direct peers (#414)
* Add support for setting gossipsub direct peers

* Use string array for direct peers config

* Handle empty nitro and consensus config

* Increase consensus write retry attempts

* Rename config option for watcher party peers file

* Handle directPeers array being passed undefined

* Upgrade package versions

---------

Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
2023-09-22 13:46:20 +05:30
prathamesh0
48e7785367
Upgrade nitro-node package (#415)
* Upgrade nitro-node package

* Update package versions
2023-09-21 18:32:03 +05:30
77312a2f95
Add yamux stream multiplexer in peer package (#413)
* Add yamux muxer to peer

* Upgrade version to 0.2.58

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-09-15 12:19:51 +05:30
prathamesh0
bd73dae1b1
Implement consensus mechanism using mokka (#412)
* Add Consensus class with placeholders

* Implement Consensus constructor

* Move PubsubType to peer package

* Implement remaining methods in Consensus class

* Use existing consensus stream if it exists

* Setup send and receive pipes on consensus streams

* Refactor P2P and consensus setup in server command

* Add Nitro node initialization in server command

* Return objects from server initializations

* Use dynamic imports for ES modules in util package

* Fix util deps

* Change target to es6 to allow creating a Mokka instance

* Set moduleResolution to node16 in util for dynamic imports

* Upgrade @cerc-io/nitro-node package

* Implement retries while sending consensus messages

* Use bunyan for consensus logs and subscribe to state changes

* Use debug for logging state change events

* Handle empty watcher party file path

* Return object from initP2P

* Upgrade @cerc-io/nitro-node package

* Update package versions
2023-09-14 13:47:31 +05:30
prathamesh0
c80e4d0ab7
Add support for gossipsub (#410)
* Replace floodsub with gossipsub

* Add a pubsub flag to support both floodsub and gossipsub

* Fix deps

* Handle empty string values

* Upgrade @cerc-io/libp2p dependency

* Upgrade package versions
2023-09-05 17:54:52 +05:30
f89eea0d2f
Upgrade package versions (#407) 2023-08-30 15:55:35 +05:30
444901eb87
Upgrade package versions to 0.2.54 (#403)
Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-08-16 19:07:30 +05:30
f2bc054c19
Rename nitro-client to nitro-node (#402)
* Rename nitro-client to nitro-node

* Upgrade package versions to 0.2.53

---------

Co-authored-by: neeraj <neeraj.rtly@gmail.com>
2023-08-16 17:57:15 +05:30
b600524c8c
Fix dependencies in cli and eth-client packages (#401)
* Fix dependencies in cli and eth-client packages

* Upgrade package versions
2023-08-10 09:58:50 +05:30
prathamesh0
3b090b3987
Handle null blocks in FEVM (#400)
* Avoid gasPrice while serializing tx in getRawTransaction

* Convert base fee to bigint when fetching block data

* Fix methods to get block data in rpc-eth-client

* Fix error checking in rpc-eth-client

* Continue processig after missing null block in FEVM

* Self review fix

* Upgrade package versions

---------

Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
2023-08-09 17:31:11 +05:30
c06330dd06
Implement rpc-eth-client for Ethereum compatible JSON-RPC endpoint (#398)
* Implement rpc-eth-client with getStorageAt method

* Add test for comparing RPC and GQL eth-client getStorageAt method

* Add getBlockWithTransactions and getBlocks method

* Implement getFullBlocks with RLP encoded data

* Implement getFullTransaction method with raw tx

* Implement getBlockByHash and getLogs methods

* Add flag and interface to switch between RPC and GQL eth clients

* Fix getBlocks to return empty array when block not present

* Return empty array in getBlocks for missing block and use blockNumber in getLogs

* Fix getRawTransaction method for zero signature.v value

* Remove duplicate util from rpc-eth-client
2023-08-08 18:48:55 +05:30
prathamesh0
47d4b667f4
Payment plugin fixes (#396)
* Avoid deducting from free quota for queries with unconfigured rates

* Check query selections on an introspection query

* Update comment
2023-07-31 17:09:58 +05:30
prathamesh0
198d3e65db
Add payments config and charge according to configured rates (#395)
* Track amount paid on payment channels and charge as per query cost

* Add payments manager config

* Fix received payment checks

* Separate out payment not received and amount insufficient errors

* Serve a query for free if its rate is not configured

* Load open payment channels from the stored state

* Take list of free queries from config

* Take mutation rates from config

* Rename query rates config fields

* Take cache settings and request timeout from config

* Upgrade ts-nitro packages

* Upgrade package versions
2023-07-27 15:52:36 +05:30
prathamesh0
6731a640a2
Use a single payment header in payments plugin (#393)
* Add client address field to payments manager

* Use a single payment header in payments plugin

* Upgrade ts-nitro packages

* Rename method to authenticateVoucherForSender

* Upgrade package versions

---------

Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
2023-07-25 13:34:02 +05:30
10fa9d6b4b
Wait for receiving voucher on GQL query (#392)
* Wait for receiving voucher on GQL query

* Add timeout while authenticating voucher and remove listener channel

* Clear timeout after payment received

* Rename variable to paymentListener

* Upgrade package versions

* Add rpcSupportsBlockHashParam flag in config
2023-07-24 13:36:13 +05:30
prathamesh0
eec1965980
Add a payments plugin to GQL server (#391)
* Add a custom payments plugin in server process

* Fix voucher channel read in voucher subscription

* Use voucher hash and signature for payment authentication

* Use a LRU hash map for storing received vouchers

* Avoid hoisting level types to resolve build errors

* Upgrade ts-nitro packages

* Add payments related logs

* Upgrade package versions

* Always allow latestBlock query

* Upgrade ts-nitro packages
2023-07-20 17:22:48 +05:30
11930c0107
Expose peer instance in server CLI and add nitro config options (#390)
* Expose peer instance in server CLI and add nitro config options

* Update package versions

---------

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2023-07-13 15:30:31 +05:30
c6e685fc13
Use local lerna for publishing package in CI (#389)
* Use local lerna for publishing package in CI

* Upgrade package versions

* Add new line at EOF
2023-06-28 12:14:51 +05:30
b0780da52f
Upgrade package versions before publish (#388) 2023-06-28 11:38:07 +05:30
prathamesh0
c5cbe33059
Pin typeorm version to avoid discrepancies with generated watchers (#385) 2023-05-15 17:55:36 +05:30
dfd38d3329
Upgrade package versions (#384)
* Upgrade package versions

* Use node 18.x in github CI
2023-05-15 12:30:24 +05:30
76522afa8d
Implement CLI to create checkpoint state from existing GQL endpoint (#382)
* Add util method for preparing state from GQL response entity

* Refactor method to create or update state data

* Fix typeorm bigint transformer and convert to checksum contract address

* Skip resetting to previous block in job-runner if isComplete set to true

* Fix creating subgraph event with struct params

* Use quotes for table names in custom queries

* Fix indexer prepareStateEntry method

* Fix toEthereumValue method when used with ethereum.decode

* Add CLI for creating snapshot checkpoint state from GQL endpoint

* Skip import-state if block is already indexed

* Review changes
2023-05-15 12:10:27 +05:30
prathamesh0
79898bc3df
Update package version (#383) 2023-05-12 15:11:33 +05:30
prathamesh0
2e1652d772
Block connections with blacklisted multiaddrs (#373)
* Block connections from blacklisted multiaddrs in relay node

* Block connections from blacklisted multiaddrs in peer nodes

* Block dials for blacklisted multiaddrs

* Update package version
2023-05-04 16:33:29 +05:30
ca844dbe47
Upgrade package version before release (#370) 2023-04-25 18:29:07 +05:30
aaa60430e4
Update readme content and package versions (#366)
* Upgrade package versions

* Update readme for watchers
2023-04-14 18:11:15 +05:30
7f91fa7ed8
Remove watchers after moving them out to their own repos (#365)
* Remove all watchers after moving them out

* Add missing dependencies
2023-04-14 17:22:44 +05:30