Commit Graph

483 Commits

Author SHA1 Message Date
prathamesh0
1ad223db4d
Add GQL requests caching in eden-watcher (#234) 2022-11-16 20:01:18 +05:30
74741184ee
Use lateral query for time travel queries using latest entity tables (#233)
* Avoid joins in id only queries

* Use lateral query for time travel queries using latest entity tables
2022-11-16 18:57:04 +05:30
a084b4e40c
Latest entity queries to optimize frontend app GQL requests (#232)
* Add latest entity table query pattern

* Add subscriber config to typeorm
2022-11-16 18:11:27 +05:30
prathamesh0
408a3927c0
Add a table for entites in frothy region for subgraph watchers (#231)
* Add a table for entites in frothy region and update it in a subscriber

* Accommodate changes to other watchers and codegen
2022-11-16 17:12:54 +05:30
7e5974ccf7
Denormalize is_pruned flag in eden-watcher (#230)
* Add is_pruned flag to all entities

* Mark entities as pruned once the corresponding block gets pruned
2022-11-16 15:30:40 +05:30
prathamesh0
62c57d8005
Upgrade Apollo server to v3 (#229)
* Upgrade apollo server to v3 in eden-watcher

* Upgrade apollo server to v3 in other watchers and codegen

* Refactor common server code to util

* Pass express application object in the method to create server

* Remove unnecessary import
2022-11-16 11:14:54 +05:30
prathamesh0
a52bdf64b1
Add config for GQL requests cache (#228)
* Add config and a method to set cache control settings for GQL requests

* Add option for max size in GQL cache config

* Fix failing tests
2022-11-15 14:56:08 +05:30
6f8ededd52
Add flag to load relations sequentially or concurrently (#225)
* Add flag to load relations sequentially or concurrently

* Fix database init in graph-node test

* Fix graph-node watcher query method and add changes to codegen
2022-11-14 14:23:46 +05:30
prathamesh0
13edff143b
Update cerc-io packages version in watcher generation (#224) 2022-11-11 15:00:48 +05:30
prathamesh0
861420a10f
Add a flag to enable state creation (#222)
* Add a flag to enable state creation

* Update flag in mock config for testing
2022-11-10 16:56:06 +05:30
prathamesh0
8d3c68873b
Avoid creating a separate base db instance for graph database (#221)
* Avoid creating a separate base db instance for graph database

* Fix failing test file build

* Use getter method to get base database
2022-11-10 14:51:18 +05:30
ecd8b2474a
Remove eventName filter in events count query (#220) 2022-11-10 12:09:17 +05:30
prathamesh0
3862f9ce2e
Use FindManyOptions to get entities (#219) 2022-11-09 14:01:39 +05:30
772fdbec25
Add unique entities query without checking pruned block (#218) 2022-11-09 13:31:26 +05:30
7a4b459b1c
Add DB queries without checking pruned block (#217) 2022-11-08 17:01:35 +05:30
Ashwin Phatak
99c6823564
GQL API request queue params (#216) 2022-11-04 12:01:07 +05:30
0e464385c0
Remove contracts on reset watcher (#215) 2022-11-04 10:41:48 +05:30
prathamesh0
266442a6d9
Update getFullBlocks usage (#213)
* Update getFullBlocks usage to add block number arg

* Add skipStateFieldsUpdate config for uniswap-watcher

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-11-03 16:49:41 +05:30
be65ca45ea
Load relations sequentially (#212) 2022-11-03 14:32:34 +05:30
prathamesh0
2517f110ea
Update GQL and getBlock API usage (#210)
* Update getLogs and getBlockWithTransactions usage to add block number arg

* Update getFullTransaction usage to add block number arg
2022-11-03 14:14:58 +05:30
52c42f4e84
Reset watcher to previous indexed block on start (#207)
* Reset watcher to previous indexed block before start

* Implement changes in other watchers

* Save successfully fetched blocks and events to prefetch cache

* Add unique query for transaction table

* Check db for blocks before fetching from eth-server

* Show all mismatches at a block

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-11-03 14:01:10 +05:30
prathamesh0
306bbb73ca
Use prefetching of blocks with events in watchers and codegen (#206)
* Avoid refetching block while fetching events

* Prefetch a batch of blocks with events while indexing

* Update mock indexer used in graph-node testing

* Process available blocks while prefetching

* Refactor events fetching to a method in util

* Move method to get GQL event query result to util
2022-10-20 18:46:56 +05:30
668875b3a0
Refactor code in graph-node to use in uniswap watcher (#205)
* Refactor code in graph-node to use in uniswap

* Move over database methods to fetch entities for GQL query
2022-10-20 14:39:32 +05:30
prathamesh0
5af90bd388
Refactor state creation code (#204)
* Remove support for pushing state to IPFS

* Move job handlers for state creation to util

* Rename state creation related methods and objects

* Update mock indexer used in graph-node testing

* Fetch and merge diffs in batches while creating a state checkpoint

* Fix timing logs while for state checkpoint creation

* Refactor method to get state query result to util

* Accept contracts for state verification in compare CLI config

* Make method to update state status map synchronous
2022-10-19 15:24:14 +05:30
ce182bce85
Clear cache of latest entities on event processing error (#201)
* Clear cache of latest entities on event processing error

* Remove lighthouse-watcher and update ethersjs version

* Handle GraphDecimal type in state entity

* Add option for comparing all entities using paginate

* Clear pruned cached entities at intervals

* Move ipld-demo to graph-node package and remove reset-dbs script

* Implement changes in all watchers and codegen
2022-10-19 14:26:10 +05:30
prathamesh0
74747ce49a
Add scripts for ERC721 token interaction (#203) 2022-10-17 15:38:59 +05:30
prathamesh0
45111e68ac
Add an option to run compare CLI in batched intervals (#202) 2022-10-17 12:07:21 +05:30
Michael
5ffe13b723
Npm gitea publish (#200)
* update gitignore for goland and first attempt at git.vdb.to npm publish

* whitespace yaml nonsense

* missing run entry

* lerna is not cicd friendly, yet

* try from-git arg in lerna.json

* attempt default lerna publish with full checkout

* explicit cli args... lerna.json ???

* explicit cli args... from-package is not --arg

* make publish yaml match manual workflow
2022-10-13 15:12:24 -04:00
prathamesh0
bad5ec754a
Prefetch blocks with events in memory (#199)
* Prefetch blocks along with events in memory

* Fix prefetched block destructuring

* Wait while blocks are being prefetched

* Retry only if block not returned by upstream eth-server

* Log time taken to prefetch blocks

* Avoid creating a separate job for prefetching blocks

* Skip syncStatus check while fetching blocks

* Convert block number string to number for comparison

* Prefetch blocks in a batch only when required

* Keep existing pattern for watchers in this repo

* Fix logging while fetching a batch of blocks
2022-10-13 17:07:46 +05:30
Michael
0ef458734e
Create manual_npm_publishl.yml 2022-10-12 15:03:50 -04:00
521f6ed3fa
v0.2.13 (#197) 2022-10-11 16:12:06 +05:30
e1aef1a7e9
Changes to use packages in uniswap-watcher (#196)
* Changes to use util from uniswap-watcher

* Refactor ResultIPLDBlock to util

* Verify state in compare CLI for uniswap multiple queries

* Prepare watcher-ts for publishing packages

* Fix verify state in compare CLI for multiple entities query

* Fix codegen util imports
2022-10-11 13:41:26 +05:30
prathamesh0
978f0bb456
Update getLogs API usage (#195)
* Update client query to get logs

* Update calls to get logs for contracts
2022-10-05 15:15:00 +05:30
18861eaf79
Implement cache for latest updated entities to be used in mapping code (#194)
* Load relations according to GQL query

* Implement cache for latest entities to used in mapping code

* Add metrics for cache hit and fix caching pruned entities

* Changes in codegen and graph-test-watcher

* Remove entity load counter reset to zero
2022-10-04 13:31:29 +05:30
6149690126
Update ipld-status table in reset and fill state CLI (#193) 2022-10-04 11:22:04 +05:30
00a9c247f5
Compare CLI enhancements for verifying uniswap-watcher entities (#188)
* Update compare CLI to verify only updated entities

* Show time difference between GQL requests
2022-10-04 11:18:55 +05:30
87224a4673
Implement eden-watcher changes in other watchers and codegen (#192)
* Implement eden-watcher changes in other watchers and codegen

* Use node space size only for eden-watcher
2022-09-23 15:35:15 +05:30
a8fdcca866
Update watcher import CLI to create entities from checkpoint (#186)
* Import entities from checkpoint

* Fix IPLD state when updating subgraph entity

* Changes in codegen and other watchers

* Update IPLD state with all Block entities

* Add verify and create sub commands to checkpoint

* Add option for specifying snapshot block in export state CLI
2022-09-22 15:26:06 +05:30
prathamesh0
b2cf997900
Update CLI to fill state to create checkpoints (#187)
* Update CLI to fill state to create checkpoints

* Accomodate changes in codegen
2022-09-16 09:23:41 +05:30
prathamesh0
c2bbaa6867
Update meta data verification to allow branched checkpoints (#183) 2022-09-16 09:15:11 +05:30
5a7dcbd20f
Load relations for single entity GQL query with default limit 100 (#182)
* Check endBlock greater than chainHeadBlock in fill blocks CLI

* Load relations for single entity with limit 100

* Use single tx query runner for entity GQL query

* Remove assert for CLI input checks
2022-09-13 17:24:14 +05:30
prathamesh0
8960f67f1b
Update compare CLI to verify IPLD block meta data (#179)
* Update compare CLI to compare IPLD block meta data

* Include init and checkpoints in IPLD blocks verification

* Accomodate changes in codegen
2022-09-12 17:44:53 +05:30
996f68a390
Fix hex conversion and rename to cerc-io packages (#178)
* Fix ethersjs hexlify number by removing leading zeros

* Rename to cerc-io packages in codegen templates
2022-09-09 19:39:52 +05:30
f3230e3822
Change package name prefixes to cerc-io (#177) 2022-09-09 17:13:01 +05:30
prathamesh0
e30af92901
Add a CLI in eden-watcher to fill state for a given range (#176)
* Add a CLI to fill state for a given range

* Refactor code

* Add a CLI to reset IPLD state

* Replace ORDER BY clause in the query to get latest IPLD block

* Optimize delete query in CLI to reset IPLD state

* Add an option to decouple subgraph state creation from mapping code

* Use a raw SQL query to delete IPLD blocks in a block range

* Accomodate changes in codegen
2022-09-09 16:23:41 +05:30
4e5ec36f07
Add config for query limit in compare CLI (#175)
* Add config for query limit in compare CLI

* Fix JSON stringify bigint for GQL query params

* Take last N entity ids to compare in GQL
2022-09-08 16:54:02 +05:30
93584bf28d
Update mapping code to restore eth_calls (#174) 2022-09-08 11:21:35 +05:30
73ca225779
Asynchronous prefetching of block size with eth_getBlockByNumber (#173)
* Prefetch block size from eth_blockByHash

* Fix updating of blockSizeMapLatestHeight

* Move block size caching to separate file

* Remove timer logs from graph-node store set and get
2022-09-07 18:29:04 +05:30
prathamesh0
f3091dee3d
Batch diff creation for subgraph entities updated in mapping code (#172)
* Batch diff creation for subgraph entities updated in mapping code

* Propagate changes to graph-test-watcher and codegen
2022-09-07 13:05:21 +05:30
288e153287
Implement graceful shutdown in job-runner (#170)
* Implement graceful shutdown in job-runner

* Improve reset CLI delete query

* Exit job-runner immediately in development

* Implement changes in other watchers
2022-09-06 19:21:24 +05:30