Commit Graph

28 Commits

Author SHA1 Message Date
prathamesh0
bd8f003322
Gracefully shutdown server (#265)
* Gracefully shutdown server

* Forbid lint warnings

* Avoid setting subgraph path in template contex in codegen
2022-11-28 12:01:28 +05:30
prathamesh0
94c8ed9575
Move event watcher to util (#262) 2022-11-25 17:19:37 +05:30
prathamesh0
6737ec756c
Refactor import-state CLI to cli package (#250)
* Remove unnecessary upstream config arg to event watcher

* Initialize event watcher in CLI package

* Refactor import-state CLI to cli package
2022-11-22 16:41:15 +05:30
a0ba657d17
Rename variables and fix comments (#243)
* Rename entity to entityType

* Rename variables and fix comments
2022-11-18 19:16:42 +05:30
662c79a5e7
Refactor event-watcher and move code to util (#242) 2022-11-18 17:23:33 +05:30
prathamesh0
f3c65cbd64
Refactor util code to be reused (#241)
* Ignore watch contract jobs in event processing complete handler

* Update job-queue config and handle errors on job completion hook

* Update graph decimal implementation

* Return generic type from method to read watcher config

* Export fill prefetch batch size default value
2022-11-18 16:29:06 +05:30
prathamesh0
62c57d8005
Upgrade Apollo server to v3 (#229)
* Upgrade apollo server to v3 in eden-watcher

* Upgrade apollo server to v3 in other watchers and codegen

* Refactor common server code to util

* Pass express application object in the method to create server

* Remove unnecessary import
2022-11-16 11:14:54 +05:30
prathamesh0
306bbb73ca
Use prefetching of blocks with events in watchers and codegen (#206)
* Avoid refetching block while fetching events

* Prefetch a batch of blocks with events while indexing

* Update mock indexer used in graph-node testing

* Process available blocks while prefetching

* Refactor events fetching to a method in util

* Move method to get GQL event query result to util
2022-10-20 18:46:56 +05:30
f3230e3822
Change package name prefixes to cerc-io (#177) 2022-09-09 17:13:01 +05:30
3cee10607e
Parse batched transactions for mobymask-watcher events (#141)
* Remove using graph-node from mobymask-watcher

* Parse batched transactions for MobyMask events

* Update to use same ethers package version

* Add CLI for processing block out of order

* Fix job-runner for already processed blocks out of order
2022-07-11 11:29:33 +05:30
prathamesh0
71d34331e9
Deprecate postgraphile usage (#122)
* Deprecate postgraphile usage

* Change endpoint in watcher config

* Remove subscription queries from eth-client

* Remove postgraphile config and client from watchers

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-06-08 12:13:52 +05:30
30f3c9e694 Fix block processing job created twice when processing missing parent block (#67)
* Fix block processing job created twice in watcher

* Fix block processing job for missing parent blocks
2021-12-28 16:08:05 +05:30
faf046d181 Implement graph-ts numbers, typeConversion and log host APIs (#64)
* Implement host api for bigint operations

* Implement type conversion APIs

* Avoid use of exports as variable name for conflict

* Implement log API with debug and levels

* Fix job-runnner hook error after stopping and starting chain
2021-12-28 16:08:05 +05:30
prathamesh0
51b200709b Generate IPLD blocks table and related GQL API (#260)
* Add ipld-blocks entity generation

* Populate ipld-blocks table

* Rename ipld-block entity and update after each event

* Move ipld-hook to hooks.ts

* Change IPLD block structure

* Add cid field in blocks

* Fetch prev. IPLDBlock for a contract

* GQL API to query IPLDBlock by CID

* Save cid in blocks in existing watchers

* Update codegen docs

* GQL API for getting last derived state (#3)

* GQL API for getting last derived state

* Rename query to getState

* Change query names to getState and getStateByCid

* Save BigInt as string

* Move function to prepare IPLDBlock to indexer

* Refactor IPLDBlock hook

* Add genesis hook

* Call post-block hook after a block is marked as complete

* Add IPLDBlock checkpointing

* Use queryRunner instead of a new repo for queries

* Add a query to get block in ipld-eth-client

* Get latest checkpoints for all contracts for checkpointing.

* Call post-block hook in a queue

* Pass server config to Indexer in watch-contract cli

Co-authored-by: nikugogoi <nabarun@deepstacksoft.com>
2021-12-28 16:08:04 +05:30
7b6f6e468f
Move update sync status outside job-runner (#324) 2021-12-21 15:46:05 +05:30
prathamesh0
2bcf579859
Remove unknown events of parent block when processing a block (#323) 2021-12-20 17:00:14 +05:30
cfd293f11b
Fix job-runner stop and restart in uni-watcher (#319)
* Fix job-runner stop and restart in uni-watcher

* Fix blocks processed twice after processing missing parent block
2021-12-17 11:57:09 +05:30
prathamesh0
d087667177
Update sync status after pruning before processing next block (#318)
* Update sync status after pruning before processing next block

* Exit gracefully on error while prefetching blocks
2021-12-17 10:23:04 +05:30
105b26d6a3
Performance improvements for fill CLI (#314)
* Fix fill cli to work with watcher server

* Add createdAt column and insert events in batches

* Implement prefetch to fill block and events in parallel

* Fix getPrevEntity and increase fill prefetch default batch size

* Fix watcher creating mulitple jobs for a block
2021-12-16 17:16:48 +05:30
b345d25bb8
Optimize and reduce queries in block processing job (#308)
* Improvement TODOs

* Get parent and current block in single query

* Remove block progress queries and events query

* Reduce queries in fetching batch events by using query builder

* Implement changes in codegen package
2021-12-13 15:38:34 +05:30
f56f7a823f
Process events in singe job and avoid block progress query to improve performance (#306)
* Avoid database query by passing event directly to job-queue

* Avoid block progress query by returning from update query

* Process batch of events for a block in a single job

* Fix smoke test for subscribed events and use teamSize for job queue
2021-12-10 10:44:10 +05:30
63ce6fd55f
Handle missing blocks in postgraphile by making RPC call (#296) 2021-12-03 16:23:11 +05:30
32fea1f2cb
Fix uni-watcher to handle mainnet data (#293)
* Fix watchers to handle mainnet data

* Tweak jobs fetched per interval to reduce event processing time
2021-12-02 15:28:03 +05:30
9d95e49ec9
Change block processing to be pull based (#288)
* Implement pull based watcher for uni-watcher

* Fix same block processed multiple times

* Implement wait time for fetching block from config

* Use blockProgress event to fetch and process next block

* Rename utils index to misc
2021-10-26 17:36:21 +05:30
5cbcd455d2
Purge unknown events after block processing complete (#280) 2021-10-20 17:49:44 +05:30
Ashwin Phatak
f1f70ad7d4
Move chain pruning to block processing queue with higher priority (#235)
* Move chain pruning to block processing queue.

* Refactor common code in util.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-30 11:16:54 +05:30
Ashwin Phatak
35068b2c3d
Implement chain pruning in uni-info-watcher (#222)
* Refactor code for chain pruning.

* Implement chain pruning in uni-info-watcher.

* Refactor pruning code to util.

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-19 13:27:32 +05:30
Ashwin Phatak
b8b216ea5b
Extract block filler into common package, implement in uni-info-watcher (#220)
* Implement block filler for uni-info-watcher.

* Pull common code for Block filler to util.

* Watch for job events in block filler command.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-18 15:50:44 +05:30