Commit Graph

43 Commits

Author SHA1 Message Date
prathamesh0
2e1652d772
Block connections with blacklisted multiaddrs (#373)
* Block connections from blacklisted multiaddrs in relay node

* Block connections from blacklisted multiaddrs in peer nodes

* Block dials for blacklisted multiaddrs

* Update package version
2023-05-04 16:33:29 +05:30
7dada71fa3
Add boolean flag to enable sending L2 txs from watcher peer (#354) 2023-04-10 09:56:55 +05:30
ac5e030b13
Integrate peer in watcher server and send txs to L2 chain (#344)
* Integrate peer in watcher and send txs to L2

* Remove passing debug log in functions

* Remove unnecessary relay advertise config option

---------

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2023-03-24 16:49:16 +04:00
prathamesh0
c46d5c3f33
Broadcast peer info over floodsub when requested (#332)
* Refactor discovery handler

* Broadcast peer info over floodsub on requests

* Broadcast peer info from relay nodes

* Make debug reponse handler optional

* Register debug info request handler on peer init

* Move debug info types to types dir

* Return method to unsubscribe from the debug topic

* Make debug info flag optional for relay nodes

* Restructure peer connection info data

* Refactor getting peer info to be used in react app

* Refactor duplicate code to utils

* Rename peer methods

---------

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2023-03-06 10:20:51 +05:30
90d60f54a6
Improve connection check with peers by retrying ping on failures (#330)
* Improve connection check with peers by retrying ping on failures

* Use DEFAULT_PING_INTERVAL in cli

* Log only error message

* Add config option to pass peer id file path

* Finish retrying pings before connection check interval duration

* Handle duplicate connections to relay nodes

* Increase default max dial retries from one relay to another

* Update connection manager config for relay nodes

* Use debug for logs in relay node

---------

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2023-02-27 12:47:49 +05:30
prathamesh0
6a8b9a2385
Pass initialization options when starting a peer (#328)
* Pass initialization options when starting a peer

* Update config used for relay and peer nodes in watcher

* Rename types

* Refactor mobymask libp2p message parsing

* Enable laconic debug logs in server command
2023-02-21 17:57:25 +05:30
prathamesh0
6fa3ee28b5
Add mobymask-v2-watcher with integrated relay and peer functionality (#327)
* Add an option to run relay node with watcher server

* Add a v2 mobymask-watcher

* Add an option to run peer node with watcher server

* Ignore ts error when importing types from peer package

* Rename mobymask v2 watcher

* Parse mobymask libp2p messages

* Refactor and add comments

* Add a note in peer package about nodejs exports

* Update copyright
2023-02-20 18:09:06 +05:30
prathamesh0
f3c65cbd64
Refactor util code to be reused (#241)
* Ignore watch contract jobs in event processing complete handler

* Update job-queue config and handle errors on job completion hook

* Update graph decimal implementation

* Return generic type from method to read watcher config

* Export fill prefetch batch size default value
2022-11-18 16:29:06 +05:30
prathamesh0
a52bdf64b1
Add config for GQL requests cache (#228)
* Add config and a method to set cache control settings for GQL requests

* Add option for max size in GQL cache config

* Fix failing tests
2022-11-15 14:56:08 +05:30
6f8ededd52
Add flag to load relations sequentially or concurrently (#225)
* Add flag to load relations sequentially or concurrently

* Fix database init in graph-node test

* Fix graph-node watcher query method and add changes to codegen
2022-11-14 14:23:46 +05:30
prathamesh0
861420a10f
Add a flag to enable state creation (#222)
* Add a flag to enable state creation

* Update flag in mock config for testing
2022-11-10 16:56:06 +05:30
Ashwin Phatak
99c6823564
GQL API request queue params (#216) 2022-11-04 12:01:07 +05:30
prathamesh0
266442a6d9
Update getFullBlocks usage (#213)
* Update getFullBlocks usage to add block number arg

* Add skipStateFieldsUpdate config for uniswap-watcher

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-11-03 16:49:41 +05:30
prathamesh0
306bbb73ca
Use prefetching of blocks with events in watchers and codegen (#206)
* Avoid refetching block while fetching events

* Prefetch a batch of blocks with events while indexing

* Update mock indexer used in graph-node testing

* Process available blocks while prefetching

* Refactor events fetching to a method in util

* Move method to get GQL event query result to util
2022-10-20 18:46:56 +05:30
prathamesh0
5af90bd388
Refactor state creation code (#204)
* Remove support for pushing state to IPFS

* Move job handlers for state creation to util

* Rename state creation related methods and objects

* Update mock indexer used in graph-node testing

* Fetch and merge diffs in batches while creating a state checkpoint

* Fix timing logs while for state checkpoint creation

* Refactor method to get state query result to util

* Accept contracts for state verification in compare CLI config

* Make method to update state status map synchronous
2022-10-19 15:24:14 +05:30
ce182bce85
Clear cache of latest entities on event processing error (#201)
* Clear cache of latest entities on event processing error

* Remove lighthouse-watcher and update ethersjs version

* Handle GraphDecimal type in state entity

* Add option for comparing all entities using paginate

* Clear pruned cached entities at intervals

* Move ipld-demo to graph-node package and remove reset-dbs script

* Implement changes in all watchers and codegen
2022-10-19 14:26:10 +05:30
prathamesh0
bad5ec754a
Prefetch blocks with events in memory (#199)
* Prefetch blocks along with events in memory

* Fix prefetched block destructuring

* Wait while blocks are being prefetched

* Retry only if block not returned by upstream eth-server

* Log time taken to prefetch blocks

* Avoid creating a separate job for prefetching blocks

* Skip syncStatus check while fetching blocks

* Convert block number string to number for comparison

* Prefetch blocks in a batch only when required

* Keep existing pattern for watchers in this repo

* Fix logging while fetching a batch of blocks
2022-10-13 17:07:46 +05:30
e1aef1a7e9
Changes to use packages in uniswap-watcher (#196)
* Changes to use util from uniswap-watcher

* Refactor ResultIPLDBlock to util

* Verify state in compare CLI for uniswap multiple queries

* Prepare watcher-ts for publishing packages

* Fix verify state in compare CLI for multiple entities query

* Fix codegen util imports
2022-10-11 13:41:26 +05:30
f3230e3822
Change package name prefixes to cerc-io (#177) 2022-09-09 17:13:01 +05:30
prathamesh0
e30af92901
Add a CLI in eden-watcher to fill state for a given range (#176)
* Add a CLI to fill state for a given range

* Refactor code

* Add a CLI to reset IPLD state

* Replace ORDER BY clause in the query to get latest IPLD block

* Optimize delete query in CLI to reset IPLD state

* Add an option to decouple subgraph state creation from mapping code

* Use a raw SQL query to delete IPLD blocks in a block range

* Accomodate changes in codegen
2022-09-09 16:23:41 +05:30
8af7417df6
Implement query for multiple entities and nested relation fields in eden-watcher (#166)
* Implement query for multiple entities in eden-watcher

* Implement nested relation queries

* Implement GQL query params first, skip, orderBy, orderDirection

* Add blockNumber index to subgraph entities

* Add logs for timing eth-calls and storage calls

* Add prometheus metrics to monitor GQL queries

* Fix default limit and order of 1-N related field in GQL entitiy query

* Add timer logs for block processing

* Run transpiled js in all watchers

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-09-01 14:17:43 +05:30
a15305450c
Add prometheus metrics support for watchers (#152)
* Add prometheus metrics endpoint in watcher

* Add event, sync status and DB size metrics

* Fix subgraph watchers DB entities directory path

* Make watcher metrics optional using config
2022-08-03 16:26:51 +05:30
1bcabd64f2
Update codegen with changes implemented in mobymask watcher (#148)
* Update codegen with index-block CLI and remove graph-node

* Add filter logs by contract flag

* Skip generating GQL API for immutable variables

* Add config for maxEventsBlockRange

* Add new flags in existing watchers
2022-07-22 13:17:56 +05:30
58bbf4d756
Handle getStorage calls for old blocks (#145)
* Handle getStorage calls for old blocks

* Refactor getStorage code in mobymask watcher

* Fix returning proof as null
2022-07-20 10:37:17 +05:30
e2668690b5
Add filterLogs flag to fetch logs by contract (#144) 2022-07-18 14:04:59 +05:30
prathamesh0
71d34331e9
Deprecate postgraphile usage (#122)
* Deprecate postgraphile usage

* Change endpoint in watcher config

* Remove subscription queries from eth-client

* Remove postgraphile config and client from watchers

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-06-08 12:13:52 +05:30
198e49e5a0 Implement WASM instance restart to handle out of memory error (#81)
* Test case for wasm out of memory error

* Restart wasm instance after N blocks

* Handle out of memory error and re instantiate WASM

* Remove old instance from map before reinstantiating WASM
2021-12-28 16:08:05 +05:30
6cca55a1ab Integrate generated watcher to invoke handlers in graph-node (#33)
* Invoke handlers based on watcher-ts events

* Read subgraph yaml and listen to events from watcher-ts

* Create GraphWatcher class to use in generated example contract watcher

* Call graph-node event handler from generated watcher
2021-12-28 16:08:05 +05:30
prathamesh0
bb1345c696 Refactor code to initialize clients (#290)
* Refactor CLI code

* Refactor code to initialize clients

* Don't return config from function to initialize clients
2021-12-28 16:08:04 +05:30
prathamesh0
a846917f1d Export-import watcher state and IPFS integration (#289)
* Add export-watcher CLI generation

* Export snapshot block and add import-watcher CLI gen

* Update codegen docs with export and import commands

* Use different function to get latest checkpoint in export CLI

* Update generated watcher docs with import state instructions

* Push IPLDBlocks to IPFS

* Change ts type for data column to Buffer

* Add checkpoint hook

* Push CLI checkpoint to IPFS

* Fix latest IPLDBlock query to filter out irrelevant blocks

* Add CLI to inspect a CID

* Decode data before exporting

* Encode data in JSON for getState GQL API

* Use go-ipfs and export cbor-encoded buffer

* Remove empty post-block methods from existing watchers

* Call hooks and checkpoint job complete handlers in fill

* Remove subscription to hooks queue from existing watchers
2021-12-28 16:08:04 +05:30
prathamesh0
51b200709b Generate IPLD blocks table and related GQL API (#260)
* Add ipld-blocks entity generation

* Populate ipld-blocks table

* Rename ipld-block entity and update after each event

* Move ipld-hook to hooks.ts

* Change IPLD block structure

* Add cid field in blocks

* Fetch prev. IPLDBlock for a contract

* GQL API to query IPLDBlock by CID

* Save cid in blocks in existing watchers

* Update codegen docs

* GQL API for getting last derived state (#3)

* GQL API for getting last derived state

* Rename query to getState

* Change query names to getState and getStateByCid

* Save BigInt as string

* Move function to prepare IPLDBlock to indexer

* Refactor IPLDBlock hook

* Add genesis hook

* Call post-block hook after a block is marked as complete

* Add IPLDBlock checkpointing

* Use queryRunner instead of a new repo for queries

* Add a query to get block in ipld-eth-client

* Get latest checkpoints for all contracts for checkpointing.

* Call post-block hook in a queue

* Pass server config to Indexer in watch-contract cli

Co-authored-by: nikugogoi <nabarun@deepstacksoft.com>
2021-12-28 16:08:04 +05:30
f56f7a823f
Process events in singe job and avoid block progress query to improve performance (#306)
* Avoid database query by passing event directly to job-queue

* Avoid block progress query by returning from update query

* Process batch of events for a block in a single job

* Fix smoke test for subscribed events and use teamSize for job queue
2021-12-10 10:44:10 +05:30
9d95e49ec9
Change block processing to be pull based (#288)
* Implement pull based watcher for uni-watcher

* Fix same block processed multiple times

* Implement wait time for fetching block from config

* Use blockProgress event to fetch and process next block

* Rename utils index to misc
2021-10-26 17:36:21 +05:30
137f7d1a21
Fix eth_calls with blockHash (#287)
* Fix eth_calls with blockHash

* Use blockHash in erc20 eth_calls
2021-10-25 19:51:16 +05:30
47b9e6bbbd
Implement reset job queue and state CLI in watchers (#276)
* Implement clean jobs CLI in watchers

* Pull common code cleanJobs to util

* Implement commands for reset job-queue and state

* Reset command for job-queues and watcher state

* Reset sync status on reset state

* Fix using cli from built js

* Implement skipCheck to reset syncStatus table and skip complete handler on fail

* Check for block isComplete on reset state

* Set default value for force param to update syncStatus to false

* Reset tables in erc20 watcher

* Push job in fill after previos block is complete
2021-10-20 16:06:03 +05:30
prathamesh0
8e3093c684
Codegen flag for watcher kind and eventsInRange query support (#254)
* Add watcher kind argument.

* Process eventsInRange query.
2021-09-28 10:22:21 +05:30
Ashwin Phatak
3ff2fdf11b
Refactoring, extract uni-info-watcher GQL client (#226)
* Refactor common code in uni-watcher and uni-info-watcher.

* Implement client.ts in uni-info-watcher for queries and use in smoke-test.

* Disable cache in graphql client.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-24 11:55:29 +05:30
Ashwin Phatak
b8b216ea5b
Extract block filler into common package, implement in uni-info-watcher (#220)
* Implement block filler for uni-info-watcher.

* Pull common code for Block filler to util.

* Watch for job events in block filler command.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-18 15:50:44 +05:30
Ashwin Phatak
a7ec3d8da8
Add APGL license and copyright notices (#212)
* Add license & copyright declarations for add-watcher.

* Add copyright declarations for cache.

* Add copyright declarations for erc20-watcher.

* Add copyright declarations for ipld-eth-client.

* Add copyright declarations for tracing-client.

* Add copyright declarations for uni-watcher.

* Add copyright declarations for solidity-mapper.

* Add copyright declarations for uni-info-watcher.

* Add copyright declarations for util.

* Add copyright declarations for lighthouse-watcher.

* Change license identifier in .sol files.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-12 15:28:13 +05:30
Ashwin Phatak
73e0475dfa
Misc changes for uni-info-watcher resolvers (#183)
* Implement transaction from to set entity origin field.

* Add all transaction fields specified in schema and add delay in uni-info block processing.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-04 18:42:59 +05:30
Ashwin Phatak
c677e5942c
ERC20 watcher based on eth_call (#165)
* Implement eth_call for ERC20 totalSupply.

* Use eth_call for erc20-watcher.

* Implement fallback for ERC20 symbol method call.

* Implement fallback for ERC20 name and totalSupply.

* implement fallback for erc20 decimals method.

* Lint fixes.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-07-28 10:04:07 +05:30
Ashwin Phatak
70e88b1004
Block filler for uni-watcher. (#140) 2021-07-15 13:10:07 +05:30
Ashwin Phatak
61f211f2d5
Handle pool initialize event (#127)
* Move getConfig to util package.

* Handle Pool initialize event.

* Update Bundle entity ethPriceUSD.

* Update Pool day and hour data.

* Update token derivedETH and complete handleInitialize.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-07-09 12:38:25 +05:30