Commit Graph

390 Commits

Author SHA1 Message Date
prathamesh0
bad5ec754a
Prefetch blocks with events in memory (#199)
* Prefetch blocks along with events in memory

* Fix prefetched block destructuring

* Wait while blocks are being prefetched

* Retry only if block not returned by upstream eth-server

* Log time taken to prefetch blocks

* Avoid creating a separate job for prefetching blocks

* Skip syncStatus check while fetching blocks

* Convert block number string to number for comparison

* Prefetch blocks in a batch only when required

* Keep existing pattern for watchers in this repo

* Fix logging while fetching a batch of blocks
2022-10-13 17:07:46 +05:30
521f6ed3fa
v0.2.13 (#197) 2022-10-11 16:12:06 +05:30
e1aef1a7e9
Changes to use packages in uniswap-watcher (#196)
* Changes to use util from uniswap-watcher

* Refactor ResultIPLDBlock to util

* Verify state in compare CLI for uniswap multiple queries

* Prepare watcher-ts for publishing packages

* Fix verify state in compare CLI for multiple entities query

* Fix codegen util imports
2022-10-11 13:41:26 +05:30
prathamesh0
978f0bb456
Update getLogs API usage (#195)
* Update client query to get logs

* Update calls to get logs for contracts
2022-10-05 15:15:00 +05:30
18861eaf79
Implement cache for latest updated entities to be used in mapping code (#194)
* Load relations according to GQL query

* Implement cache for latest entities to used in mapping code

* Add metrics for cache hit and fix caching pruned entities

* Changes in codegen and graph-test-watcher

* Remove entity load counter reset to zero
2022-10-04 13:31:29 +05:30
6149690126
Update ipld-status table in reset and fill state CLI (#193) 2022-10-04 11:22:04 +05:30
00a9c247f5
Compare CLI enhancements for verifying uniswap-watcher entities (#188)
* Update compare CLI to verify only updated entities

* Show time difference between GQL requests
2022-10-04 11:18:55 +05:30
87224a4673
Implement eden-watcher changes in other watchers and codegen (#192)
* Implement eden-watcher changes in other watchers and codegen

* Use node space size only for eden-watcher
2022-09-23 15:35:15 +05:30
a8fdcca866
Update watcher import CLI to create entities from checkpoint (#186)
* Import entities from checkpoint

* Fix IPLD state when updating subgraph entity

* Changes in codegen and other watchers

* Update IPLD state with all Block entities

* Add verify and create sub commands to checkpoint

* Add option for specifying snapshot block in export state CLI
2022-09-22 15:26:06 +05:30
prathamesh0
b2cf997900
Update CLI to fill state to create checkpoints (#187)
* Update CLI to fill state to create checkpoints

* Accomodate changes in codegen
2022-09-16 09:23:41 +05:30
prathamesh0
c2bbaa6867
Update meta data verification to allow branched checkpoints (#183) 2022-09-16 09:15:11 +05:30
5a7dcbd20f
Load relations for single entity GQL query with default limit 100 (#182)
* Check endBlock greater than chainHeadBlock in fill blocks CLI

* Load relations for single entity with limit 100

* Use single tx query runner for entity GQL query

* Remove assert for CLI input checks
2022-09-13 17:24:14 +05:30
prathamesh0
8960f67f1b
Update compare CLI to verify IPLD block meta data (#179)
* Update compare CLI to compare IPLD block meta data

* Include init and checkpoints in IPLD blocks verification

* Accomodate changes in codegen
2022-09-12 17:44:53 +05:30
996f68a390
Fix hex conversion and rename to cerc-io packages (#178)
* Fix ethersjs hexlify number by removing leading zeros

* Rename to cerc-io packages in codegen templates
2022-09-09 19:39:52 +05:30
f3230e3822
Change package name prefixes to cerc-io (#177) 2022-09-09 17:13:01 +05:30
prathamesh0
e30af92901
Add a CLI in eden-watcher to fill state for a given range (#176)
* Add a CLI to fill state for a given range

* Refactor code

* Add a CLI to reset IPLD state

* Replace ORDER BY clause in the query to get latest IPLD block

* Optimize delete query in CLI to reset IPLD state

* Add an option to decouple subgraph state creation from mapping code

* Use a raw SQL query to delete IPLD blocks in a block range

* Accomodate changes in codegen
2022-09-09 16:23:41 +05:30
4e5ec36f07
Add config for query limit in compare CLI (#175)
* Add config for query limit in compare CLI

* Fix JSON stringify bigint for GQL query params

* Take last N entity ids to compare in GQL
2022-09-08 16:54:02 +05:30
93584bf28d
Update mapping code to restore eth_calls (#174) 2022-09-08 11:21:35 +05:30
73ca225779
Asynchronous prefetching of block size with eth_getBlockByNumber (#173)
* Prefetch block size from eth_blockByHash

* Fix updating of blockSizeMapLatestHeight

* Move block size caching to separate file

* Remove timer logs from graph-node store set and get
2022-09-07 18:29:04 +05:30
prathamesh0
f3091dee3d
Batch diff creation for subgraph entities updated in mapping code (#172)
* Batch diff creation for subgraph entities updated in mapping code

* Propagate changes to graph-test-watcher and codegen
2022-09-07 13:05:21 +05:30
288e153287
Implement graceful shutdown in job-runner (#170)
* Implement graceful shutdown in job-runner

* Improve reset CLI delete query

* Exit job-runner immediately in development

* Implement changes in other watchers
2022-09-06 19:21:24 +05:30
prathamesh0
1e284bd07e
Update fill prefetch CLI (#171)
* Update fill prefetch CLI to accept env variables

* Clean up assets before building util package

* Accomodate change in other watchers and codegen

* Add logs for debugging failing test in CI
2022-09-06 13:19:14 +05:30
8af7417df6
Implement query for multiple entities and nested relation fields in eden-watcher (#166)
* Implement query for multiple entities in eden-watcher

* Implement nested relation queries

* Implement GQL query params first, skip, orderBy, orderDirection

* Add blockNumber index to subgraph entities

* Add logs for timing eth-calls and storage calls

* Add prometheus metrics to monitor GQL queries

* Fix default limit and order of 1-N related field in GQL entitiy query

* Add timer logs for block processing

* Run transpiled js in all watchers

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-09-01 14:17:43 +05:30
97e88ab5f0
Fixes and improvements for eden-watcher job-runner and compare CLI (#165)
* Compare IPLD state entity without derived fields

* Apply default limit to array relation fields in IPLD state entity

* Mark block as complete after processing of block handler

* Avoid re processing of block handler

* Use LIMIT 1 in the query to get latest IPLD block

* Replace eth_calls in eden-watcher with getStorageValue

* Add checkpoint verification to export state CLI

* Fix get diff blocks query when creating checkpoint

* Fix subgraph staker sort and remove entities sequentially in reset CLI

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-26 12:02:39 +05:30
prathamesh0
a5b3c7942d
Add test package (#164)
* Add test package

* Add a separate command to perform an eth-call

* Add snapshot test suite

* Add eth-calls for UniswapV2 Pair, USDC, Compound, Dai contracts

* Add args for Uniswap and USDC contracts

* Add args for Compound and Dai contracts

* Add getStorageAt calls to the test suite

* Refactor code and add documentation

* Loop over test slots for getStorageAt calls

* Add support for individual calls

* Use debug for logging
2022-08-19 11:06:11 +05:30
7238f614c0
Update CLI to compare only updated entities and verify IPLD state (#161)
* Change compare CLI to verify only updated entities

* Implement IPLD state verification in compare CLI

* Changes to IPLD state to match with GQL result entity
2022-08-17 19:11:40 +05:30
ec56de057f
Implement method for storage based access in subgraph mapping code (#162)
* Implement storage call in subgraph mapping code

* Add test for mapping type variable storage call

* Use vulcanize graph-ts

* Revert to graph-ts version 0.22.1
2022-08-17 16:25:49 +05:30
prathamesh0
80682e2755
Remove Uniswap watchers and related artifacts (#163)
* Remove Uniswap watchers and related code

* Remove Uniswap artifacts from graph-node

* Remove Uniswap artifacts from graph-test-watcher
2022-08-17 14:04:20 +05:30
bc1c267813
Add GQL query for sync status (#159)
* Add sync status GQL query

* Add sync status GQL query in codegen
2022-08-09 13:25:46 +05:30
5077abc90f
Compute gas price for EIP-1559 transaction (#157) 2022-08-09 10:50:25 +05:30
1a903fccc6
CLI to compare eden-watcher entities and fix mapping code (#149)
* Make sumStaked variable local in eden network mapping

* Implement compare CLI to fetch and query by ids

* Set filterLogs to true for eden-watcher

* Use varchar for bigint array type in eden-watcher

* Store subgraph entities by id in IPLD state

* Store bigint vales as string in IPLD state

* Update eden watcher hook to store single Block entity in IPLD checkpoint

* Fix entity enum type property

* Fix parsing big numbers in event params

* Fix event bigint params parsing in all watchers

* Set default limit to query result and process block after events
2022-08-08 15:58:13 +05:30
6990eb892a
Handle array and struct type event params in codegen (#150)
* Handle array and struct type event params

* Refactor parse event in watcher indexer code
2022-08-04 17:09:30 +05:30
a15305450c
Add prometheus metrics support for watchers (#152)
* Add prometheus metrics endpoint in watcher

* Add event, sync status and DB size metrics

* Fix subgraph watchers DB entities directory path

* Make watcher metrics optional using config
2022-08-03 16:26:51 +05:30
1bcabd64f2
Update codegen with changes implemented in mobymask watcher (#148)
* Update codegen with index-block CLI and remove graph-node

* Add filter logs by contract flag

* Skip generating GQL API for immutable variables

* Add config for maxEventsBlockRange

* Add new flags in existing watchers
2022-07-22 13:17:56 +05:30
b577db287f
Add doc to index blocks out of order using eth-statediff-service (#146)
* Add doc to index blocks using eth-statediff-service

* Add example commands and watcher queries
2022-07-20 12:55:24 +05:30
58bbf4d756
Handle getStorage calls for old blocks (#145)
* Handle getStorage calls for old blocks

* Refactor getStorage code in mobymask watcher

* Fix returning proof as null
2022-07-20 10:37:17 +05:30
e2668690b5
Add filterLogs flag to fetch logs by contract (#144) 2022-07-18 14:04:59 +05:30
9a35955166
Fix job-runner for missing parent block (#143) 2022-07-15 19:08:33 +05:30
129b9e71f0
Add option for no check in max events block range (#142) 2022-07-13 14:40:42 +05:30
3cee10607e
Parse batched transactions for mobymask-watcher events (#141)
* Remove using graph-node from mobymask-watcher

* Parse batched transactions for MobyMask events

* Update to use same ethers package version

* Add CLI for processing block out of order

* Fix job-runner for already processed blocks out of order
2022-07-11 11:29:33 +05:30
74f798b5c4
Rename to watcher-ts (#140) 2022-07-07 12:38:32 +05:30
bd3113069a
Rename to mobymask-watcher (#139) 2022-07-07 11:50:32 +05:30
4b256016da
Update docs and work around GQL client caching in ipld-eth-client (#138) 2022-07-04 15:36:47 +05:30
c919d784bf
Fetch transaction arguments for indexed event (#137) 2022-06-30 15:50:49 +05:30
26d998d3a7 Update codegen to store hash for indexed reference type event args 2022-06-29 14:53:57 +05:30
14a32a7d47 Fix for indexed reference type event arg in moby mask watcher 2022-06-29 14:53:57 +05:30
6cb424b315 Remove ipfs setup from demo 2022-06-29 14:53:57 +05:30
ed75a4dc82 Demo to deploy contract and make transactions 2022-06-29 14:53:57 +05:30
ba73cd24c2 Generate MobyMask watcher using codegen 2022-06-29 14:53:57 +05:30
5a0d844d7f
Use stack-orchestrator for erc721-watcher demo (#132)
* Use stack-orchestrator for erc721-watcher demo

* Run all services in stack-orchestrator with auto mining of blocks

* Add step to check docker-compose version
2022-06-22 17:01:05 +05:30
7d6b86942f
Add GQL API for transferCount entity (#129) 2022-06-16 17:19:57 +05:30
012d2e1a47
Update erc721-watcher demo for IPLD blocks (#125)
* Update demo for IPLD blocks

* Add update for custom properties in state diff

* Implement custom update of transfer counter in state diff

* Comment code for updating custom state prop

* Separate markdown for erc721-watcher Demo

* Run workflow on main branch commit
2022-06-15 10:40:40 +05:30
baa20de443 Add demo for erc721-watcher 2022-06-08 16:43:15 +05:30
99aaa9ae0a Generate erc721-watcher using codegen 2022-06-08 16:43:15 +05:30
prathamesh0
71d34331e9
Deprecate postgraphile usage (#122)
* Deprecate postgraphile usage

* Change endpoint in watcher config

* Remove subscription queries from eth-client

* Remove postgraphile config and client from watchers

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-06-08 12:13:52 +05:30
e4b9596be7
Update ipfs daemon version in readme (#121) 2022-05-26 20:01:31 +05:30
168689a7c7
Fix codegen for non subgraph watchers (#120)
* Fix codegen for non subgraph watchers

* Remove graphWatcher.init from generated non subgraph watcher
2022-05-26 18:00:17 +05:30
b2233e16ef
Fix running erc20-watcher as active kind watcher (#116) 2022-05-23 18:54:14 +05:30
977564d17f Update example subgraph startBlock for demo 2022-05-06 15:43:45 +05:30
ce6568aad2 Add demo for graph-watcher IPLD statediff and checkpointing 2022-05-06 15:43:45 +05:30
ba6b996e65
Deploy example contract in graph-node (#113)
* Deploy example contract in graph-node

* Add auth token in docker build for github packages

* Add steps for running tests in CI
2022-04-29 15:11:40 +05:30
eca09ad66b
Change postgraphile queries to work with v3 schema (#112) 2022-04-28 17:13:32 +05:30
654edf4008
Update README and graph-node tests (#110)
* Update README with proper steps for installing github packages

* Update graph-node readme to run tests
2022-04-27 15:37:13 +05:30
9a8ae3f308
Implement compare CLI for multiple entities for given block range (#108)
* Implement compare CLI for multiple entities at blocks

* Implement caching for gql requests
2022-01-17 15:18:44 +05:30
aabf9f8e15
Implement data source templates (#106)
* Implement data source templates

* Remove optional param from wasm instantiate

* Add all types for solidity to typescript mapping

* Set contract address in context for data source templates

* Implement block handlers for data source templates
2022-01-06 18:32:08 +05:30
4867530da7
Implement subgraph JSON host APIs fromBytes and try_fromBytes (#105)
* Implement JSON fromBytes host API

* Implement json.try_fromBytes host API

* Fill host API code for json number methods
2022-01-03 18:15:20 +05:30
6a2c99a0bb
Implement data source and crypto host APIs (#104)
* Implement data source host APIs

* Implement crypto host API
2021-12-30 17:57:34 +05:30
8b913af93f
Implement ethereum ABI encode and decode in subgraph (#103)
* Implement ethereum ABI encode in subgraph

* Implement ethereum ABI decoding host API

* Implement ABI encode decode for array type

* Implement ABI encode decode for bytes type
2021-12-30 13:12:32 +05:30
9b1aa29afd
Get missing fields in transaction data for subgraph event handlers (#102)
* Get missing fields in event transaction

* Cache transaction data for event handlers in subgraph
2021-12-29 13:21:39 +05:30
561c2c9066 Generate schema from ABI when using subgraph path (#101) 2021-12-28 16:08:05 +05:30
prathamesh0
a267058d51 Add IpldStatus table and IPLD jobs in job-runner in codegen (#100) 2021-12-28 16:08:05 +05:30
prathamesh0
5f4dd14d7a Handle event name conflicts in schema and update hooks in codegen (#97)
* Change return type for subgraph resolvers and update hooks generation

* Handle event name conflicts in schema generation
2021-12-28 16:08:05 +05:30
7a4d0b6bb4 Fixes in eden watcher after comparing with generated watcher (#96) 2021-12-28 16:08:05 +05:30
prathamesh0
e883463aa6 Parse events for multiple contracts in the generated code (#95)
* Parse events for multiple contracts in the generated code

* Use contract wise artifacts in the generated indexer methods

* Update codegen docs to use config file to generate a watcher

* Add watcher generation config to eden-watcher
2021-12-28 16:08:05 +05:30
e5faba8e68 Fixes after comparing with existing eden watcher (#94) 2021-12-28 16:08:05 +05:30
feac6bad16 Remove default ERC20 code from generated hooks (#93) 2021-12-28 16:08:05 +05:30
prathamesh0
44e18fee48 Add watcher generation mode none (#92) 2021-12-28 16:08:05 +05:30
8b5d408f77 Implement Block_height input and definition for custom scalar BigDecimal (#91) 2021-12-28 16:08:05 +05:30
2a204d8a32 Handle additional subgraph types BigDecimal and Bytes in codegen (#89)
* Handle additional subgraph types BigDecimal and Bytes

* Use bigint and Decimal array transformers
2021-12-28 16:08:05 +05:30
prathamesh0
18f73e57c0 Generate relations and entity types maps in indexer (#88) 2021-12-28 16:08:05 +05:30
d96cc095bb Handle relation fields in subgraph entities (#87) 2021-12-28 16:08:05 +05:30
4b1b0e0ed6 Handle subgraph entities field name conflicts and enum types in codegen (#86)
* Handle entity field conflicts

* Handle enum type fields in subgraph entities
2021-12-28 16:08:05 +05:30
prathamesh0
3ed4ab95af Implement IPLD refactoring in codegen (#85) 2021-12-28 16:08:05 +05:30
1e7d84879b Implement changes in codegen for performance improvements (#84) 2021-12-28 16:08:05 +05:30
prathamesh0
97b529f3f2 Update CLI generation (#83) 2021-12-28 16:08:05 +05:30
prathamesh0
5b12db541b Support artifacts generation for multiple contracts (#82) 2021-12-28 16:08:05 +05:30
prathamesh0
3638d56787 Add checks for and create hooks, checkpoint and IPFS jobs in job-runner (#99)
* Add fields for checkpoint and IPFS in IpldStatus table

* Mark block processing job complete in watchers
2021-12-28 16:08:05 +05:30
dee517e444 Implement IpldStatus table for IPLD related jobs (#98) 2021-12-28 16:08:05 +05:30
prathamesh0
d385db2775 Add missing relation fields in eden-watcher relations map (#90) 2021-12-28 16:08:05 +05:30
198e49e5a0 Implement WASM instance restart to handle out of memory error (#81)
* Test case for wasm out of memory error

* Restart wasm instance after N blocks

* Handle out of memory error and re instantiate WASM

* Remove old instance from map before reinstantiating WASM
2021-12-28 16:08:05 +05:30
68bc1c00db Refactor watchers for util package changes 2021-12-28 16:08:05 +05:30
prathamesh0
560df57ac7 Performance improvements for IPLDBlocks in eden watcher (#80)
* Avoid fetching contracts while creating a checkpoint

* Use enum for state kind and cache ipld status

* Avoid fetching block twice while finalizing a staged diff IPLDBlock

* Create checkpoints at fixed block numbers and refactor checkpointing code

* Avoid calling block handler until start block is reached

* Use delete while finalizing staged diff IPLDBlocks

* Add a check to ensure hooks job is created only once

* Avoid check for initial state while creating a checkpoint
2021-12-28 16:08:05 +05:30
prathamesh0
952f68ee58 Add time logs for eden-watcher (#78) 2021-12-28 16:08:05 +05:30
f4d16afa6c Refactor watchers according to changes in util package 2021-12-28 16:08:05 +05:30
768a4d0818 Use Promise all while fetching events and watch contracts only once 2021-12-28 16:08:05 +05:30
ba890e7d9a Add method in watchers to cache contract and fixes for running tests 2021-12-28 16:08:05 +05:30
prathamesh0
cb2fe2aa45 Store initial state and refactor IPLDBlocks related code (#76)
* Process hooks for the first block in the watchers

* Store initial state from a hook in an IPLDBlock for eden-watcher

* Extract watcher method to prepare IPLDBlocks in util

* Extract common IPLDBlocks related code in util

* Move IPFSClient to util

* Use constants for state kind
2021-12-28 16:08:05 +05:30
prathamesh0
4c422e3ea2 Avoid setting null entity properties while fetching in store get API (#75) 2021-12-28 16:08:05 +05:30
prathamesh0
3b835e81f8 Pass provider while instantiating the loader and use block hash while making the eth-calls (#74)
* Pass provider while instantiating the loader

* Use custom provider for graph-node tests

* Use block hash while making an eth-call
2021-12-28 16:08:05 +05:30
prathamesh0
3d3ebb0e43 Get event handler using cleaned subgraph event signature (#73) 2021-12-28 16:08:05 +05:30