Commit Graph
362 Commits
Author SHA1 Message Date
prathamesh0 306bbb73ca Use prefetching of blocks with events in watchers and codegen (#206)
* Avoid refetching block while fetching events

* Prefetch a batch of blocks with events while indexing

* Update mock indexer used in graph-node testing

* Process available blocks while prefetching

* Refactor events fetching to a method in util

* Move method to get GQL event query result to util
2022-10-20 18:46:56 +05:30
nabarun 668875b3a0 Refactor code in graph-node to use in uniswap watcher (#205)
* Refactor code in graph-node to use in uniswap

* Move over database methods to fetch entities for GQL query
2022-10-20 14:39:32 +05:30
prathamesh0 5af90bd388 Refactor state creation code (#204)
* Remove support for pushing state to IPFS

* Move job handlers for state creation to util

* Rename state creation related methods and objects

* Update mock indexer used in graph-node testing

* Fetch and merge diffs in batches while creating a state checkpoint

* Fix timing logs while for state checkpoint creation

* Refactor method to get state query result to util

* Accept contracts for state verification in compare CLI config

* Make method to update state status map synchronous
2022-10-19 15:24:14 +05:30
nabarun ce182bce85 Clear cache of latest entities on event processing error (#201)
* Clear cache of latest entities on event processing error

* Remove lighthouse-watcher and update ethersjs version

* Handle GraphDecimal type in state entity

* Add option for comparing all entities using paginate

* Clear pruned cached entities at intervals

* Move ipld-demo to graph-node package and remove reset-dbs script

* Implement changes in all watchers and codegen
2022-10-19 14:26:10 +05:30
prathamesh0 74747ce49a Add scripts for ERC721 token interaction (#203) v0.2.14 2022-10-17 15:38:59 +05:30
prathamesh0 45111e68ac Add an option to run compare CLI in batched intervals (#202) 2022-10-17 12:07:21 +05:30
Michael 5ffe13b723 Npm gitea publish (#200)
* update gitignore for goland and first attempt at git.vdb.to npm publish

* whitespace yaml nonsense

* missing run entry

* lerna is not cicd friendly, yet

* try from-git arg in lerna.json

* attempt default lerna publish with full checkout

* explicit cli args... lerna.json ???

* explicit cli args... from-package is not --arg

* make publish yaml match manual workflow
2022-10-13 15:12:24 -04:00
prathamesh0 bad5ec754a Prefetch blocks with events in memory (#199)
* Prefetch blocks along with events in memory

* Fix prefetched block destructuring

* Wait while blocks are being prefetched

* Retry only if block not returned by upstream eth-server

* Log time taken to prefetch blocks

* Avoid creating a separate job for prefetching blocks

* Skip syncStatus check while fetching blocks

* Convert block number string to number for comparison

* Prefetch blocks in a batch only when required

* Keep existing pattern for watchers in this repo

* Fix logging while fetching a batch of blocks
2022-10-13 17:07:46 +05:30
Michael 0ef458734e Create manual_npm_publishl.yml 2022-10-12 15:03:50 -04:00
nabarun 521f6ed3fa v0.2.13 (#197) 2022-10-11 16:12:06 +05:30
nabarun e1aef1a7e9 Changes to use packages in uniswap-watcher (#196)
* Changes to use util from uniswap-watcher

* Refactor ResultIPLDBlock to util

* Verify state in compare CLI for uniswap multiple queries

* Prepare watcher-ts for publishing packages

* Fix verify state in compare CLI for multiple entities query

* Fix codegen util imports
2022-10-11 13:41:26 +05:30
prathamesh0 978f0bb456 Update getLogs API usage (#195)
* Update client query to get logs

* Update calls to get logs for contracts
v0.2.13
2022-10-05 15:15:00 +05:30
nabarun 18861eaf79 Implement cache for latest updated entities to be used in mapping code (#194)
* Load relations according to GQL query

* Implement cache for latest entities to used in mapping code

* Add metrics for cache hit and fix caching pruned entities

* Changes in codegen and graph-test-watcher

* Remove entity load counter reset to zero
2022-10-04 13:31:29 +05:30
nabarun 6149690126 Update ipld-status table in reset and fill state CLI (#193) 2022-10-04 11:22:04 +05:30
nabarun 00a9c247f5 Compare CLI enhancements for verifying uniswap-watcher entities (#188)
* Update compare CLI to verify only updated entities

* Show time difference between GQL requests
2022-10-04 11:18:55 +05:30
nabarun 87224a4673 Implement eden-watcher changes in other watchers and codegen (#192)
* Implement eden-watcher changes in other watchers and codegen

* Use node space size only for eden-watcher
2022-09-23 15:35:15 +05:30
nabarun a8fdcca866 Update watcher import CLI to create entities from checkpoint (#186)
* Import entities from checkpoint

* Fix IPLD state when updating subgraph entity

* Changes in codegen and other watchers

* Update IPLD state with all Block entities

* Add verify and create sub commands to checkpoint

* Add option for specifying snapshot block in export state CLI
2022-09-22 15:26:06 +05:30
prathamesh0 b2cf997900 Update CLI to fill state to create checkpoints (#187)
* Update CLI to fill state to create checkpoints

* Accomodate changes in codegen
2022-09-16 09:23:41 +05:30
prathamesh0 c2bbaa6867 Update meta data verification to allow branched checkpoints (#183) 2022-09-16 09:15:11 +05:30
nabarun 5a7dcbd20f Load relations for single entity GQL query with default limit 100 (#182)
* Check endBlock greater than chainHeadBlock in fill blocks CLI

* Load relations for single entity with limit 100

* Use single tx query runner for entity GQL query

* Remove assert for CLI input checks
2022-09-13 17:24:14 +05:30
prathamesh0 8960f67f1b Update compare CLI to verify IPLD block meta data (#179)
* Update compare CLI to compare IPLD block meta data

* Include init and checkpoints in IPLD blocks verification

* Accomodate changes in codegen
v0.2.12
2022-09-12 17:44:53 +05:30
nabarun 996f68a390 Fix hex conversion and rename to cerc-io packages (#178)
* Fix ethersjs hexlify number by removing leading zeros

* Rename to cerc-io packages in codegen templates
2022-09-09 19:39:52 +05:30
nabarun f3230e3822 Change package name prefixes to cerc-io (#177) 2022-09-09 17:13:01 +05:30
prathamesh0 e30af92901 Add a CLI in eden-watcher to fill state for a given range (#176)
* Add a CLI to fill state for a given range

* Refactor code

* Add a CLI to reset IPLD state

* Replace ORDER BY clause in the query to get latest IPLD block

* Optimize delete query in CLI to reset IPLD state

* Add an option to decouple subgraph state creation from mapping code

* Use a raw SQL query to delete IPLD blocks in a block range

* Accomodate changes in codegen
2022-09-09 16:23:41 +05:30
nabarun 4e5ec36f07 Add config for query limit in compare CLI (#175)
* Add config for query limit in compare CLI

* Fix JSON stringify bigint for GQL query params

* Take last N entity ids to compare in GQL
2022-09-08 16:54:02 +05:30
nabarun 93584bf28d Update mapping code to restore eth_calls (#174) 2022-09-08 11:21:35 +05:30
nabarun 73ca225779 Asynchronous prefetching of block size with eth_getBlockByNumber (#173)
* Prefetch block size from eth_blockByHash

* Fix updating of blockSizeMapLatestHeight

* Move block size caching to separate file

* Remove timer logs from graph-node store set and get
2022-09-07 18:29:04 +05:30
prathamesh0 f3091dee3d Batch diff creation for subgraph entities updated in mapping code (#172)
* Batch diff creation for subgraph entities updated in mapping code

* Propagate changes to graph-test-watcher and codegen
2022-09-07 13:05:21 +05:30
nabarun 288e153287 Implement graceful shutdown in job-runner (#170)
* Implement graceful shutdown in job-runner

* Improve reset CLI delete query

* Exit job-runner immediately in development

* Implement changes in other watchers
2022-09-06 19:21:24 +05:30
prathamesh0 1e284bd07e Update fill prefetch CLI (#171)
* Update fill prefetch CLI to accept env variables

* Clean up assets before building util package

* Accomodate change in other watchers and codegen

* Add logs for debugging failing test in CI
2022-09-06 13:19:14 +05:30
nabarunandprathamesh 8af7417df6 Implement query for multiple entities and nested relation fields in eden-watcher (#166)
* Implement query for multiple entities in eden-watcher

* Implement nested relation queries

* Implement GQL query params first, skip, orderBy, orderDirection

* Add blockNumber index to subgraph entities

* Add logs for timing eth-calls and storage calls

* Add prometheus metrics to monitor GQL queries

* Fix default limit and order of 1-N related field in GQL entitiy query

* Add timer logs for block processing

* Run transpiled js in all watchers

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
v0.2.11
2022-09-01 14:17:43 +05:30
nabarunandprathamesh 97e88ab5f0 Fixes and improvements for eden-watcher job-runner and compare CLI (#165)
* Compare IPLD state entity without derived fields

* Apply default limit to array relation fields in IPLD state entity

* Mark block as complete after processing of block handler

* Avoid re processing of block handler

* Use LIMIT 1 in the query to get latest IPLD block

* Replace eth_calls in eden-watcher with getStorageValue

* Add checkpoint verification to export state CLI

* Fix get diff blocks query when creating checkpoint

* Fix subgraph staker sort and remove entities sequentially in reset CLI

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-26 12:02:39 +05:30
prathamesh0 a5b3c7942d Add test package (#164)
* Add test package

* Add a separate command to perform an eth-call

* Add snapshot test suite

* Add eth-calls for UniswapV2 Pair, USDC, Compound, Dai contracts

* Add args for Uniswap and USDC contracts

* Add args for Compound and Dai contracts

* Add getStorageAt calls to the test suite

* Refactor code and add documentation

* Loop over test slots for getStorageAt calls

* Add support for individual calls

* Use debug for logging
2022-08-19 11:06:11 +05:30
nabarun 7238f614c0 Update CLI to compare only updated entities and verify IPLD state (#161)
* Change compare CLI to verify only updated entities

* Implement IPLD state verification in compare CLI

* Changes to IPLD state to match with GQL result entity
2022-08-17 19:11:40 +05:30
nabarun ec56de057f Implement method for storage based access in subgraph mapping code (#162)
* Implement storage call in subgraph mapping code

* Add test for mapping type variable storage call

* Use vulcanize graph-ts

* Revert to graph-ts version 0.22.1
2022-08-17 16:25:49 +05:30
prathamesh0 80682e2755 Remove Uniswap watchers and related artifacts (#163)
* Remove Uniswap watchers and related code

* Remove Uniswap artifacts from graph-node

* Remove Uniswap artifacts from graph-test-watcher
2022-08-17 14:04:20 +05:30
nabarun bc1c267813 Add GQL query for sync status (#159)
* Add sync status GQL query

* Add sync status GQL query in codegen
v0.2.10
2022-08-09 13:25:46 +05:30
nabarun 5077abc90f Compute gas price for EIP-1559 transaction (#157) 2022-08-09 10:50:25 +05:30
nabarun 1a903fccc6 CLI to compare eden-watcher entities and fix mapping code (#149)
* Make sumStaked variable local in eden network mapping

* Implement compare CLI to fetch and query by ids

* Set filterLogs to true for eden-watcher

* Use varchar for bigint array type in eden-watcher

* Store subgraph entities by id in IPLD state

* Store bigint vales as string in IPLD state

* Update eden watcher hook to store single Block entity in IPLD checkpoint

* Fix entity enum type property

* Fix parsing big numbers in event params

* Fix event bigint params parsing in all watchers

* Set default limit to query result and process block after events
v0.2.9
2022-08-08 15:58:13 +05:30
nabarun 6990eb892a Handle array and struct type event params in codegen (#150)
* Handle array and struct type event params

* Refactor parse event in watcher indexer code
2022-08-04 17:09:30 +05:30
nabarun a15305450c Add prometheus metrics support for watchers (#152)
* Add prometheus metrics endpoint in watcher

* Add event, sync status and DB size metrics

* Fix subgraph watchers DB entities directory path

* Make watcher metrics optional using config
v0.2.8
2022-08-03 16:26:51 +05:30
nabarun 1bcabd64f2 Update codegen with changes implemented in mobymask watcher (#148)
* Update codegen with index-block CLI and remove graph-node

* Add filter logs by contract flag

* Skip generating GQL API for immutable variables

* Add config for maxEventsBlockRange

* Add new flags in existing watchers
2022-07-22 13:17:56 +05:30
nabarun b577db287f Add doc to index blocks out of order using eth-statediff-service (#146)
* Add doc to index blocks using eth-statediff-service

* Add example commands and watcher queries
2022-07-20 12:55:24 +05:30
nabarun 58bbf4d756 Handle getStorage calls for old blocks (#145)
* Handle getStorage calls for old blocks

* Refactor getStorage code in mobymask watcher

* Fix returning proof as null
v0.2.7
2022-07-20 10:37:17 +05:30
nabarun e2668690b5 Add filterLogs flag to fetch logs by contract (#144) v0.2.6 2022-07-18 14:04:59 +05:30
nabarun 9a35955166 Fix job-runner for missing parent block (#143) v0.2.5 2022-07-15 19:08:33 +05:30
nabarun 129b9e71f0 Add option for no check in max events block range (#142) v0.2.4 2022-07-13 14:40:42 +05:30
nabarun 3cee10607e Parse batched transactions for mobymask-watcher events (#141)
* Remove using graph-node from mobymask-watcher

* Parse batched transactions for MobyMask events

* Update to use same ethers package version

* Add CLI for processing block out of order

* Fix job-runner for already processed blocks out of order
v0.2.3
2022-07-11 11:29:33 +05:30
nabarun 74f798b5c4 Rename to watcher-ts (#140) v0.2.2 2022-07-07 12:38:32 +05:30
nabarun bd3113069a Rename to mobymask-watcher (#139) 2022-07-07 11:50:32 +05:30