Commit Graph

80 Commits

Author SHA1 Message Date
prathamesh0
4ddb8c4af6 Auto generating state from indexer methods (#277)
* Generate default derived state for Mapping type variables

* Update IPLDBlock in storage methods instead of using a private class variable

* Default state flag for indexer storage methods

* Helper functions to update state objects

* Add checkpoint flag in Contract table and corresponding changes in existing watchers

* Update codegen docs

* Add examples to generated docs

* Turn default state off by default

* Make state parameter to indexer storage methods default to none

* Add method to get prev. state in indexer
2021-12-28 16:08:04 +05:30
prathamesh0
421e7498d3 Ensuring chronological execution of hooks and checkpointing (#264)
* Subscribe to hooks queue in existing watchers

* Change naming strategy for generated get and save functions

* Push checkpointing job after post-block hook job completed

* Using hooks status to ensure their chronological execution

* Add default indices to IPLDBlock table

* Add kind parameter to getState GQL API

* Add checkpoint CLI

* Add blockHash arg to checkpoint CLI and update codegen docs

* Print out block hash for checkpoint CLI

* Use log from debug for logging

* Filter using contract at start in hierarchical query

* Make kind argument to prepare IPLDBlock required
2021-12-28 16:08:04 +05:30
prathamesh0
51b200709b Generate IPLD blocks table and related GQL API (#260)
* Add ipld-blocks entity generation

* Populate ipld-blocks table

* Rename ipld-block entity and update after each event

* Move ipld-hook to hooks.ts

* Change IPLD block structure

* Add cid field in blocks

* Fetch prev. IPLDBlock for a contract

* GQL API to query IPLDBlock by CID

* Save cid in blocks in existing watchers

* Update codegen docs

* GQL API for getting last derived state (#3)

* GQL API for getting last derived state

* Rename query to getState

* Change query names to getState and getStateByCid

* Save BigInt as string

* Move function to prepare IPLDBlock to indexer

* Refactor IPLDBlock hook

* Add genesis hook

* Call post-block hook after a block is marked as complete

* Add IPLDBlock checkpointing

* Use queryRunner instead of a new repo for queries

* Add a query to get block in ipld-eth-client

* Get latest checkpoints for all contracts for checkpointing.

* Call post-block hook in a queue

* Pass server config to Indexer in watch-contract cli

Co-authored-by: nikugogoi <nabarun@deepstacksoft.com>
2021-12-28 16:08:04 +05:30
cfd293f11b
Fix job-runner stop and restart in uni-watcher (#319)
* Fix job-runner stop and restart in uni-watcher

* Fix blocks processed twice after processing missing parent block
2021-12-17 11:57:09 +05:30
105b26d6a3
Performance improvements for fill CLI (#314)
* Fix fill cli to work with watcher server

* Add createdAt column and insert events in batches

* Implement prefetch to fill block and events in parallel

* Fix getPrevEntity and increase fill prefetch default batch size

* Fix watcher creating mulitple jobs for a block
2021-12-16 17:16:48 +05:30
b06d931054
Add time logs for uniswap watchers (#310)
* Log time taken in watchers

* Log time for all events and for each event batch

Co-authored-by: prathamesh0 <prathamesh@deepstacksoft.com>
2021-12-14 14:48:29 +05:30
b345d25bb8
Optimize and reduce queries in block processing job (#308)
* Improvement TODOs

* Get parent and current block in single query

* Remove block progress queries and events query

* Reduce queries in fetching batch events by using query builder

* Implement changes in codegen package
2021-12-13 15:38:34 +05:30
f56f7a823f
Process events in singe job and avoid block progress query to improve performance (#306)
* Avoid database query by passing event directly to job-queue

* Avoid block progress query by returning from update query

* Process batch of events for a block in a single job

* Fix smoke test for subscribed events and use teamSize for job queue
2021-12-10 10:44:10 +05:30
prathamesh0
a2ad139769
Use node v16 and pprof to generate profile data for watchers (#299)
* Use pprof to generate profile data for watchers

* Update docs and github workflows to use node v16
2021-12-09 10:39:35 +05:30
6f98166c49
Changes to improve performance for event processing job (#304)
* Perf improvement TODOs

* Move watch contract to util package

* Get block progress from event instead of querying

* Use index field in event to check order of processing

* Use watched contract map to avoid querying database

* Use update query for blockProgress entity

Co-authored-by: Ashwin Phatak <ashwinpphatak@gmail.com>
2021-12-08 11:11:29 +05:30
prathamesh0
f89a7a07aa
Use eth client for getStorageAt query (#302) 2021-12-06 17:22:09 +05:30
63ce6fd55f
Handle missing blocks in postgraphile by making RPC call (#296) 2021-12-03 16:23:11 +05:30
32fea1f2cb
Fix uni-watcher to handle mainnet data (#293)
* Fix watchers to handle mainnet data

* Tweak jobs fetched per interval to reduce event processing time
2021-12-02 15:28:03 +05:30
9d95e49ec9
Change block processing to be pull based (#288)
* Implement pull based watcher for uni-watcher

* Fix same block processed multiple times

* Implement wait time for fetching block from config

* Use blockProgress event to fetch and process next block

* Rename utils index to misc
2021-10-26 17:36:21 +05:30
137f7d1a21
Fix eth_calls with blockHash (#287)
* Fix eth_calls with blockHash

* Use blockHash in erc20 eth_calls
2021-10-25 19:51:16 +05:30
737d9a9b6e
Handle missing blocks by making eth call to ipld-eth-server (#285) 2021-10-22 15:20:11 +05:30
6f3b8029b2
Use blockHash for eth_calls in watchers (#283) 2021-10-21 15:23:46 +05:30
7953ba2949
Replace storage calls in uni-info-watcher with eth_calls (#281)
* Replace storage calls in uni-info-watcher with eth_calls

* Handle eth_call reverts and catch them in uni-info-watcher
2021-10-21 13:24:46 +05:30
5cbcd455d2
Purge unknown events after block processing complete (#280) 2021-10-20 17:49:44 +05:30
47b9e6bbbd
Implement reset job queue and state CLI in watchers (#276)
* Implement clean jobs CLI in watchers

* Pull common code cleanJobs to util

* Implement commands for reset job-queue and state

* Reset command for job-queues and watcher state

* Reset sync status on reset state

* Fix using cli from built js

* Implement skipCheck to reset syncStatus table and skip complete handler on fail

* Check for block isComplete on reset state

* Set default value for force param to update syncStatus to false

* Reset tables in erc20 watcher

* Push job in fill after previos block is complete
2021-10-20 16:06:03 +05:30
f00ea0c1f0
Implement lazy watcher mode in erc20-watcher (#275) 2021-10-18 13:03:55 +05:30
08c0668ddb
Get endpoint URL from config in tests (#262) 2021-10-13 11:01:36 +05:30
d456a3f676
Include NFPM collect event in smoke tests. (#257) 2021-09-29 17:14:16 +05:30
Ashwin Phatak
4f8f1d8cd7
Update code for log_cids change in vdb core (#243)
* Remove fields no longer present in schema.

* Refactor code to work with log_cids change in eth-server.

* Refactor process event to save events from logs.

* Use constants for erc20 event names.

* Implement watch blocks in erc20-watcher similar to uni-watcher.

* Moved common methods to util.

* Implement eventsInRange query in erc20-watcher.

* Filter unknown event in database query.

* Change dependencies version to be same in all packages.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-21 16:43:55 +05:30
Ashwin Phatak
a9dc34f704
Fix null parent. (#244)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-09 10:51:58 +05:30
Ashwin Phatak
06bed1c13b
Enable source maps for transpiled files (#242)
* Rename .spec files to .test files.

* Enable source maps.

* Pass enable-source-maps flag for transpiled files.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-07 16:01:15 +05:30
Ashwin Phatak
2d493b3838
Run services from transpiled ES5 (#239)
* Transpile ES5 code.

* Use transpiled ES5 code in all packages.

* Refactor tracing-client to transpile ES5 code.

* Order scripts in package json.

* Update readme in packages with new scripts.

* Refactor code to use entity config programatically.

* Remove unused depencdencies.

* Use DEBUG environment for scripts.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-02 10:27:56 +05:30
Ashwin Phatak
f1f70ad7d4
Move chain pruning to block processing queue with higher priority (#235)
* Move chain pruning to block processing queue.

* Refactor common code in util.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-30 11:16:54 +05:30
Ashwin Phatak
afd7c954a2
Refactoring pruning and reorg handling code (#227)
* Refactor db methods getPrevEntityVersion and getFrothyRegion.

* Filter out entities from pruned blocks.

* Pull saveContract and getModelEntities to util.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-25 12:47:53 +05:30
Ashwin Phatak
3ff2fdf11b
Refactoring, extract uni-info-watcher GQL client (#226)
* Refactor common code in uni-watcher and uni-info-watcher.

* Implement client.ts in uni-info-watcher for queries and use in smoke-test.

* Disable cache in graphql client.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-24 11:55:29 +05:30
Ashwin Phatak
885a55e513
Implement hierarchical query for checking ancestor block during pruning (#225)
* Implement hierarchical query for checking ancestor block.

* Prune blocks together in single query.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-23 17:06:35 +05:30
Ashwin Phatak
2ddedfefe7
Refactoring common code into util (#224)
* Refactor and pull common database code to util.

* Refactor and pull common indexer code to util.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-20 19:02:57 +05:30
8489b5632a
Tests for pruning abandonded branches (#223)
* Create dummy test blocks and refactor test code.

* Intial test for pruning block in a chain without branches.

* Add test for pruning with multiple branches.

* Add test for pruning at frothy region.

* Test for pruning block at height more than max reorg depth.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-20 12:26:37 +05:30
Ashwin Phatak
35068b2c3d
Implement chain pruning in uni-info-watcher (#222)
* Refactor code for chain pruning.

* Implement chain pruning in uni-info-watcher.

* Refactor pruning code to util.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-19 13:27:32 +05:30
Ashwin Phatak
b8b216ea5b
Extract block filler into common package, implement in uni-info-watcher (#220)
* Implement block filler for uni-info-watcher.

* Pull common code for Block filler to util.

* Watch for job events in block filler command.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-18 15:50:44 +05:30
Ashwin Phatak
a7ec3d8da8
Add APGL license and copyright notices (#212)
* Add license & copyright declarations for add-watcher.

* Add copyright declarations for cache.

* Add copyright declarations for erc20-watcher.

* Add copyright declarations for ipld-eth-client.

* Add copyright declarations for tracing-client.

* Add copyright declarations for uni-watcher.

* Add copyright declarations for solidity-mapper.

* Add copyright declarations for uni-info-watcher.

* Add copyright declarations for util.

* Add copyright declarations for lighthouse-watcher.

* Change license identifier in .sol files.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-12 15:28:13 +05:30
Ashwin Phatak
053695a3e8
Add test:init in uni-watcher. (#204)
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-11 16:27:42 +05:30
Ashwin Phatak
1372fc74ef
Test for entities after CollectEvent. (#202)
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-10 17:31:50 +05:30
Ashwin Phatak
ed9577b830
Test for entities after DecreaseLiquidityEvent. (#199)
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-10 16:32:35 +05:30
Ashwin Phatak
6c05600b41
Test for entities after IncreaseLiquidityEvent. (#198)
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-10 15:53:59 +05:30
Ashwin Phatak
0c2efde028
Test for entities after TransferEvent (#196)
* Add resolver for Position entity in uni-info-watcher.

* Test for entities after TransferEvent.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-10 12:09:00 +05:30
nikugogoi
abd3175c09
Import test utils separately. (#194)
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-09 21:34:55 +05:30
Ashwin Phatak
88121f9390
Test for entities after SwapEvent. (#191)
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-06 19:10:25 +05:30
Ashwin Phatak
91463136d7
Implement transactions in uni-watcher. (#190)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-06 17:53:55 +05:30
Ashwin Phatak
b6fe8a8c47
Import TICK_MIN from util and fix lint errors. (#187)
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-05 18:33:47 +05:30
Ashwin Phatak
df85f961db
Smoke test for entities after Mint event (#185)
* Test for Token & Factory entities after MintEvent.

* Test for Pool, Mint & Tick entities after MintEvent.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-05 17:12:11 +05:30
Ashwin Phatak
a9d411c6df
Smoke test for uni-info-watcher (#184)
* Smoke test for uni-info-watcher and refactor token deployment code.

* Test for Token entity after PoolCreated event.

* Test for Pool entity after PoolCreated event.

* Tests for entities after InitializeEvent.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-04 18:57:44 +05:30
Ashwin Phatak
73e0475dfa
Misc changes for uni-info-watcher resolvers (#183)
* Implement transaction from to set entity origin field.

* Add all transaction fields specified in schema and add delay in uni-info block processing.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-04 18:42:59 +05:30
Ashwin Phatak
d9366017da
Smoke test for NFPM events (#178)
* Added NFPM tests.

* Refactor smoke test.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-08-02 16:13:56 +05:30
Ashwin Phatak
9a3ac28a5d
uni-watcher smoke test (#172)
* Created a smoke test for uni-watcher.

* Added test for createPool.

* Added more pool tests.

* Using ethers instead of hardhat-ethers in smoke test.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-07-30 19:50:56 +05:30