Commit Graph

43 Commits

Author SHA1 Message Date
prathamesh0
1e639c1af3
Refactor reset CLIs to cli package (#246)
* Refactor reset CLIs to cli package

* Use reset CLIs from cli package in watchers

* Fix method to canonicalize latest entities

* Use composition with common code for refactored CLIs
2022-11-21 15:14:10 +04:00
prathamesh0
7520e9012c
Refactor watch-contract CLI to cli package (#245)
* Refactor watch-contract CLI to cli package

* Use watch-contract from cli in eden and erc721 watchers

* Use watch-contract from cli in all watchers

* Pass an object containing clients to refactored CLI

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-11-21 11:47:23 +04:00
662c79a5e7
Refactor event-watcher and move code to util (#242) 2022-11-18 17:23:33 +05:30
prathamesh0
f3c65cbd64
Refactor util code to be reused (#241)
* Ignore watch contract jobs in event processing complete handler

* Update job-queue config and handle errors on job completion hook

* Update graph decimal implementation

* Return generic type from method to read watcher config

* Export fill prefetch batch size default value
2022-11-18 16:29:06 +05:30
prathamesh0
f53371e17b
Accomodate GQL requests caching in code generator (#237)
* Accomodate GQL requests caching in code generator

* Add GQL API request queuing
2022-11-17 12:02:08 +05:30
prathamesh0
408a3927c0
Add a table for entites in frothy region for subgraph watchers (#231)
* Add a table for entites in frothy region and update it in a subscriber

* Accommodate changes to other watchers and codegen
2022-11-16 17:12:54 +05:30
prathamesh0
62c57d8005
Upgrade Apollo server to v3 (#229)
* Upgrade apollo server to v3 in eden-watcher

* Upgrade apollo server to v3 in other watchers and codegen

* Refactor common server code to util

* Pass express application object in the method to create server

* Remove unnecessary import
2022-11-16 11:14:54 +05:30
prathamesh0
861420a10f
Add a flag to enable state creation (#222)
* Add a flag to enable state creation

* Update flag in mock config for testing
2022-11-10 16:56:06 +05:30
prathamesh0
2517f110ea
Update GQL and getBlock API usage (#210)
* Update getLogs and getBlockWithTransactions usage to add block number arg

* Update getFullTransaction usage to add block number arg
2022-11-03 14:14:58 +05:30
52c42f4e84
Reset watcher to previous indexed block on start (#207)
* Reset watcher to previous indexed block before start

* Implement changes in other watchers

* Save successfully fetched blocks and events to prefetch cache

* Add unique query for transaction table

* Check db for blocks before fetching from eth-server

* Show all mismatches at a block

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-11-03 14:01:10 +05:30
prathamesh0
306bbb73ca
Use prefetching of blocks with events in watchers and codegen (#206)
* Avoid refetching block while fetching events

* Prefetch a batch of blocks with events while indexing

* Update mock indexer used in graph-node testing

* Process available blocks while prefetching

* Refactor events fetching to a method in util

* Move method to get GQL event query result to util
2022-10-20 18:46:56 +05:30
prathamesh0
5af90bd388
Refactor state creation code (#204)
* Remove support for pushing state to IPFS

* Move job handlers for state creation to util

* Rename state creation related methods and objects

* Update mock indexer used in graph-node testing

* Fetch and merge diffs in batches while creating a state checkpoint

* Fix timing logs while for state checkpoint creation

* Refactor method to get state query result to util

* Accept contracts for state verification in compare CLI config

* Make method to update state status map synchronous
2022-10-19 15:24:14 +05:30
ce182bce85
Clear cache of latest entities on event processing error (#201)
* Clear cache of latest entities on event processing error

* Remove lighthouse-watcher and update ethersjs version

* Handle GraphDecimal type in state entity

* Add option for comparing all entities using paginate

* Clear pruned cached entities at intervals

* Move ipld-demo to graph-node package and remove reset-dbs script

* Implement changes in all watchers and codegen
2022-10-19 14:26:10 +05:30
prathamesh0
74747ce49a
Add scripts for ERC721 token interaction (#203) 2022-10-17 15:38:59 +05:30
521f6ed3fa
v0.2.13 (#197) 2022-10-11 16:12:06 +05:30
e1aef1a7e9
Changes to use packages in uniswap-watcher (#196)
* Changes to use util from uniswap-watcher

* Refactor ResultIPLDBlock to util

* Verify state in compare CLI for uniswap multiple queries

* Prepare watcher-ts for publishing packages

* Fix verify state in compare CLI for multiple entities query

* Fix codegen util imports
2022-10-11 13:41:26 +05:30
prathamesh0
978f0bb456
Update getLogs API usage (#195)
* Update client query to get logs

* Update calls to get logs for contracts
2022-10-05 15:15:00 +05:30
18861eaf79
Implement cache for latest updated entities to be used in mapping code (#194)
* Load relations according to GQL query

* Implement cache for latest entities to used in mapping code

* Add metrics for cache hit and fix caching pruned entities

* Changes in codegen and graph-test-watcher

* Remove entity load counter reset to zero
2022-10-04 13:31:29 +05:30
87224a4673
Implement eden-watcher changes in other watchers and codegen (#192)
* Implement eden-watcher changes in other watchers and codegen

* Use node space size only for eden-watcher
2022-09-23 15:35:15 +05:30
a8fdcca866
Update watcher import CLI to create entities from checkpoint (#186)
* Import entities from checkpoint

* Fix IPLD state when updating subgraph entity

* Changes in codegen and other watchers

* Update IPLD state with all Block entities

* Add verify and create sub commands to checkpoint

* Add option for specifying snapshot block in export state CLI
2022-09-22 15:26:06 +05:30
prathamesh0
8960f67f1b
Update compare CLI to verify IPLD block meta data (#179)
* Update compare CLI to compare IPLD block meta data

* Include init and checkpoints in IPLD blocks verification

* Accomodate changes in codegen
2022-09-12 17:44:53 +05:30
f3230e3822
Change package name prefixes to cerc-io (#177) 2022-09-09 17:13:01 +05:30
prathamesh0
f3091dee3d
Batch diff creation for subgraph entities updated in mapping code (#172)
* Batch diff creation for subgraph entities updated in mapping code

* Propagate changes to graph-test-watcher and codegen
2022-09-07 13:05:21 +05:30
288e153287
Implement graceful shutdown in job-runner (#170)
* Implement graceful shutdown in job-runner

* Improve reset CLI delete query

* Exit job-runner immediately in development

* Implement changes in other watchers
2022-09-06 19:21:24 +05:30
prathamesh0
1e284bd07e
Update fill prefetch CLI (#171)
* Update fill prefetch CLI to accept env variables

* Clean up assets before building util package

* Accomodate change in other watchers and codegen

* Add logs for debugging failing test in CI
2022-09-06 13:19:14 +05:30
8af7417df6
Implement query for multiple entities and nested relation fields in eden-watcher (#166)
* Implement query for multiple entities in eden-watcher

* Implement nested relation queries

* Implement GQL query params first, skip, orderBy, orderDirection

* Add blockNumber index to subgraph entities

* Add logs for timing eth-calls and storage calls

* Add prometheus metrics to monitor GQL queries

* Fix default limit and order of 1-N related field in GQL entitiy query

* Add timer logs for block processing

* Run transpiled js in all watchers

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-09-01 14:17:43 +05:30
97e88ab5f0
Fixes and improvements for eden-watcher job-runner and compare CLI (#165)
* Compare IPLD state entity without derived fields

* Apply default limit to array relation fields in IPLD state entity

* Mark block as complete after processing of block handler

* Avoid re processing of block handler

* Use LIMIT 1 in the query to get latest IPLD block

* Replace eth_calls in eden-watcher with getStorageValue

* Add checkpoint verification to export state CLI

* Fix get diff blocks query when creating checkpoint

* Fix subgraph staker sort and remove entities sequentially in reset CLI

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-26 12:02:39 +05:30
ec56de057f
Implement method for storage based access in subgraph mapping code (#162)
* Implement storage call in subgraph mapping code

* Add test for mapping type variable storage call

* Use vulcanize graph-ts

* Revert to graph-ts version 0.22.1
2022-08-17 16:25:49 +05:30
bc1c267813
Add GQL query for sync status (#159)
* Add sync status GQL query

* Add sync status GQL query in codegen
2022-08-09 13:25:46 +05:30
1a903fccc6
CLI to compare eden-watcher entities and fix mapping code (#149)
* Make sumStaked variable local in eden network mapping

* Implement compare CLI to fetch and query by ids

* Set filterLogs to true for eden-watcher

* Use varchar for bigint array type in eden-watcher

* Store subgraph entities by id in IPLD state

* Store bigint vales as string in IPLD state

* Update eden watcher hook to store single Block entity in IPLD checkpoint

* Fix entity enum type property

* Fix parsing big numbers in event params

* Fix event bigint params parsing in all watchers

* Set default limit to query result and process block after events
2022-08-08 15:58:13 +05:30
6990eb892a
Handle array and struct type event params in codegen (#150)
* Handle array and struct type event params

* Refactor parse event in watcher indexer code
2022-08-04 17:09:30 +05:30
a15305450c
Add prometheus metrics support for watchers (#152)
* Add prometheus metrics endpoint in watcher

* Add event, sync status and DB size metrics

* Fix subgraph watchers DB entities directory path

* Make watcher metrics optional using config
2022-08-03 16:26:51 +05:30
1bcabd64f2
Update codegen with changes implemented in mobymask watcher (#148)
* Update codegen with index-block CLI and remove graph-node

* Add filter logs by contract flag

* Skip generating GQL API for immutable variables

* Add config for maxEventsBlockRange

* Add new flags in existing watchers
2022-07-22 13:17:56 +05:30
58bbf4d756
Handle getStorage calls for old blocks (#145)
* Handle getStorage calls for old blocks

* Refactor getStorage code in mobymask watcher

* Fix returning proof as null
2022-07-20 10:37:17 +05:30
e2668690b5
Add filterLogs flag to fetch logs by contract (#144) 2022-07-18 14:04:59 +05:30
3cee10607e
Parse batched transactions for mobymask-watcher events (#141)
* Remove using graph-node from mobymask-watcher

* Parse batched transactions for MobyMask events

* Update to use same ethers package version

* Add CLI for processing block out of order

* Fix job-runner for already processed blocks out of order
2022-07-11 11:29:33 +05:30
74f798b5c4
Rename to watcher-ts (#140) 2022-07-07 12:38:32 +05:30
4b256016da
Update docs and work around GQL client caching in ipld-eth-client (#138) 2022-07-04 15:36:47 +05:30
5a0d844d7f
Use stack-orchestrator for erc721-watcher demo (#132)
* Use stack-orchestrator for erc721-watcher demo

* Run all services in stack-orchestrator with auto mining of blocks

* Add step to check docker-compose version
2022-06-22 17:01:05 +05:30
7d6b86942f
Add GQL API for transferCount entity (#129) 2022-06-16 17:19:57 +05:30
012d2e1a47
Update erc721-watcher demo for IPLD blocks (#125)
* Update demo for IPLD blocks

* Add update for custom properties in state diff

* Implement custom update of transfer counter in state diff

* Comment code for updating custom state prop

* Separate markdown for erc721-watcher Demo

* Run workflow on main branch commit
2022-06-15 10:40:40 +05:30
baa20de443 Add demo for erc721-watcher 2022-06-08 16:43:15 +05:30
99aaa9ae0a Generate erc721-watcher using codegen 2022-06-08 16:43:15 +05:30