Commit Graph

163 Commits

Author SHA1 Message Date
prathamesh0
94c8ed9575
Move event watcher to util (#262) 2022-11-25 17:19:37 +05:30
e47aab2ed7
Upgrade package versions to 0.2.16 (#263) 2022-11-25 16:09:35 +05:30
b66dcb4af9
Refactor graph-node database and move to util (#259)
* Move graph-database from graph-node to util

* Refactor and remove graph-node dependency from cli package

* Modify dependencies using depcheck

* Implement CLI refactoring changes in other watchers

* Review changes to remove eden comment and fix local import in util

* Import GraphDatabase from util instead of graph-node

* Move graph-node non assemblyscript code to util package

* Implement CLI refactoring changes in codegen

* Fix graph-node tests after refactoring

* Move fromStateEntityValues to graph state utils
2022-11-25 15:54:35 +05:30
prathamesh0
590482ee53
Merge watcher job-runner into base job-runner (#260) 2022-11-24 16:26:40 +05:30
prathamesh0
aba0c665f3
Refactor fill and fill-state CLIs to cli package (#257)
* Refactor fill CLI to cli package

* Refactor method to fill-state to graph-node

* Refactor fill-state CLI to cli package

* Move subgraph state utils to a separate file

* Refactor subgraph state helper methods to graph-node

* Update mock indexer

* Move watcher job-runner to util

* Remove mock server and data from erc20-watcher

* Import watcher job-runner from util
2022-11-24 15:28:38 +05:30
b08fcebe54
Upgrade package versions (#256) 2022-11-23 18:23:26 +05:30
prathamesh0
4bfb007a7e
Refactor job-runner CLI to cli package (#255)
* Fix eden-watcher server initialization

* Add an indexer method to watch subgraph contracts

* Refactor job-runner CLI to cli package

* Move watcher reset commands to refactored code
2022-11-23 18:20:16 +05:30
cc28474537
Accommodate GQL optimizations in codegen (#254)
* Prune optional methods in indexer and database interfaces

* Implement GQL optimization changes in codegen

* Fix graph-node test indexer

* Add demos to codegen package
2022-11-23 17:42:25 +05:30
prathamesh0
0b33cc98c9
Refactor server CLI to cli package (#253)
* Refactor server CLI to cli package

* Use server CLI from cli package in all watchers
2022-11-23 12:57:59 +05:30
prathamesh0
122a64c2f9
Refactor export-state and verify checkpoint CLIs to cli package (#252)
* Refactor export-state CLI to cli package

* Refactor verify checkpoint CLI to cli package

* Update mock indexer object
2022-11-22 18:27:49 +05:30
prathamesh0
6737ec756c
Refactor import-state CLI to cli package (#250)
* Remove unnecessary upstream config arg to event watcher

* Initialize event watcher in CLI package

* Refactor import-state CLI to cli package
2022-11-22 16:41:15 +05:30
prathamesh0
570640d4bc
Refactor inspect-cid CLI to cli package (#248) 2022-11-22 11:45:45 +05:30
prathamesh0
03050e5323
Refactor create checkpoint CLI to cli package (#247)
* Refactor create checkpoint CLI to cli package

* Update mock indexer
2022-11-22 10:38:23 +05:30
06e5c62fa8
Remove saveBlockAndFetchEvents method callback pattern (#244)
* Remove saveBlockAndFetchEvents method callback pattern

* Accomodate change in code generator

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-11-21 15:54:28 +04:00
prathamesh0
1e639c1af3
Refactor reset CLIs to cli package (#246)
* Refactor reset CLIs to cli package

* Use reset CLIs from cli package in watchers

* Fix method to canonicalize latest entities

* Use composition with common code for refactored CLIs
2022-11-21 15:14:10 +04:00
prathamesh0
7520e9012c
Refactor watch-contract CLI to cli package (#245)
* Refactor watch-contract CLI to cli package

* Use watch-contract from cli in eden and erc721 watchers

* Use watch-contract from cli in all watchers

* Pass an object containing clients to refactored CLI

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-11-21 11:47:23 +04:00
a0ba657d17
Rename variables and fix comments (#243)
* Rename entity to entityType

* Rename variables and fix comments
2022-11-18 19:16:42 +05:30
662c79a5e7
Refactor event-watcher and move code to util (#242) 2022-11-18 17:23:33 +05:30
prathamesh0
f3c65cbd64
Refactor util code to be reused (#241)
* Ignore watch contract jobs in event processing complete handler

* Update job-queue config and handle errors on job completion hook

* Update graph decimal implementation

* Return generic type from method to read watcher config

* Export fill prefetch batch size default value
2022-11-18 16:29:06 +05:30
cc8fcffaa1
Fix use of prefetchBlocksInMem flag in block processing (#240)
* Fix use of prefetchBlocksInMem flag in block processing

* Rename prefetchedBlocksMap to blockAndEventsMap
2022-11-18 11:01:09 +05:30
prathamesh0
f53371e17b
Accomodate GQL requests caching in code generator (#237)
* Accomodate GQL requests caching in code generator

* Add GQL API request queuing
2022-11-17 12:02:08 +05:30
693c23d192
Update latest entity tables on chain pruning (#235) 2022-11-16 20:16:48 +05:30
prathamesh0
1ad223db4d
Add GQL requests caching in eden-watcher (#234) 2022-11-16 20:01:18 +05:30
a084b4e40c
Latest entity queries to optimize frontend app GQL requests (#232)
* Add latest entity table query pattern

* Add subscriber config to typeorm
2022-11-16 18:11:27 +05:30
prathamesh0
408a3927c0
Add a table for entites in frothy region for subgraph watchers (#231)
* Add a table for entites in frothy region and update it in a subscriber

* Accommodate changes to other watchers and codegen
2022-11-16 17:12:54 +05:30
prathamesh0
62c57d8005
Upgrade Apollo server to v3 (#229)
* Upgrade apollo server to v3 in eden-watcher

* Upgrade apollo server to v3 in other watchers and codegen

* Refactor common server code to util

* Pass express application object in the method to create server

* Remove unnecessary import
2022-11-16 11:14:54 +05:30
prathamesh0
a52bdf64b1
Add config for GQL requests cache (#228)
* Add config and a method to set cache control settings for GQL requests

* Add option for max size in GQL cache config

* Fix failing tests
2022-11-15 14:56:08 +05:30
6f8ededd52
Add flag to load relations sequentially or concurrently (#225)
* Add flag to load relations sequentially or concurrently

* Fix database init in graph-node test

* Fix graph-node watcher query method and add changes to codegen
2022-11-14 14:23:46 +05:30
prathamesh0
861420a10f
Add a flag to enable state creation (#222)
* Add a flag to enable state creation

* Update flag in mock config for testing
2022-11-10 16:56:06 +05:30
prathamesh0
8d3c68873b
Avoid creating a separate base db instance for graph database (#221)
* Avoid creating a separate base db instance for graph database

* Fix failing test file build

* Use getter method to get base database
2022-11-10 14:51:18 +05:30
ecd8b2474a
Remove eventName filter in events count query (#220) 2022-11-10 12:09:17 +05:30
prathamesh0
3862f9ce2e
Use FindManyOptions to get entities (#219) 2022-11-09 14:01:39 +05:30
7a4b459b1c
Add DB queries without checking pruned block (#217) 2022-11-08 17:01:35 +05:30
Ashwin Phatak
99c6823564
GQL API request queue params (#216) 2022-11-04 12:01:07 +05:30
0e464385c0
Remove contracts on reset watcher (#215) 2022-11-04 10:41:48 +05:30
prathamesh0
266442a6d9
Update getFullBlocks usage (#213)
* Update getFullBlocks usage to add block number arg

* Add skipStateFieldsUpdate config for uniswap-watcher

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-11-03 16:49:41 +05:30
prathamesh0
2517f110ea
Update GQL and getBlock API usage (#210)
* Update getLogs and getBlockWithTransactions usage to add block number arg

* Update getFullTransaction usage to add block number arg
2022-11-03 14:14:58 +05:30
52c42f4e84
Reset watcher to previous indexed block on start (#207)
* Reset watcher to previous indexed block before start

* Implement changes in other watchers

* Save successfully fetched blocks and events to prefetch cache

* Add unique query for transaction table

* Check db for blocks before fetching from eth-server

* Show all mismatches at a block

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-11-03 14:01:10 +05:30
prathamesh0
306bbb73ca
Use prefetching of blocks with events in watchers and codegen (#206)
* Avoid refetching block while fetching events

* Prefetch a batch of blocks with events while indexing

* Update mock indexer used in graph-node testing

* Process available blocks while prefetching

* Refactor events fetching to a method in util

* Move method to get GQL event query result to util
2022-10-20 18:46:56 +05:30
668875b3a0
Refactor code in graph-node to use in uniswap watcher (#205)
* Refactor code in graph-node to use in uniswap

* Move over database methods to fetch entities for GQL query
2022-10-20 14:39:32 +05:30
prathamesh0
5af90bd388
Refactor state creation code (#204)
* Remove support for pushing state to IPFS

* Move job handlers for state creation to util

* Rename state creation related methods and objects

* Update mock indexer used in graph-node testing

* Fetch and merge diffs in batches while creating a state checkpoint

* Fix timing logs while for state checkpoint creation

* Refactor method to get state query result to util

* Accept contracts for state verification in compare CLI config

* Make method to update state status map synchronous
2022-10-19 15:24:14 +05:30
ce182bce85
Clear cache of latest entities on event processing error (#201)
* Clear cache of latest entities on event processing error

* Remove lighthouse-watcher and update ethersjs version

* Handle GraphDecimal type in state entity

* Add option for comparing all entities using paginate

* Clear pruned cached entities at intervals

* Move ipld-demo to graph-node package and remove reset-dbs script

* Implement changes in all watchers and codegen
2022-10-19 14:26:10 +05:30
prathamesh0
bad5ec754a
Prefetch blocks with events in memory (#199)
* Prefetch blocks along with events in memory

* Fix prefetched block destructuring

* Wait while blocks are being prefetched

* Retry only if block not returned by upstream eth-server

* Log time taken to prefetch blocks

* Avoid creating a separate job for prefetching blocks

* Skip syncStatus check while fetching blocks

* Convert block number string to number for comparison

* Prefetch blocks in a batch only when required

* Keep existing pattern for watchers in this repo

* Fix logging while fetching a batch of blocks
2022-10-13 17:07:46 +05:30
521f6ed3fa
v0.2.13 (#197) 2022-10-11 16:12:06 +05:30
e1aef1a7e9
Changes to use packages in uniswap-watcher (#196)
* Changes to use util from uniswap-watcher

* Refactor ResultIPLDBlock to util

* Verify state in compare CLI for uniswap multiple queries

* Prepare watcher-ts for publishing packages

* Fix verify state in compare CLI for multiple entities query

* Fix codegen util imports
2022-10-11 13:41:26 +05:30
18861eaf79
Implement cache for latest updated entities to be used in mapping code (#194)
* Load relations according to GQL query

* Implement cache for latest entities to used in mapping code

* Add metrics for cache hit and fix caching pruned entities

* Changes in codegen and graph-test-watcher

* Remove entity load counter reset to zero
2022-10-04 13:31:29 +05:30
6149690126
Update ipld-status table in reset and fill state CLI (#193) 2022-10-04 11:22:04 +05:30
87224a4673
Implement eden-watcher changes in other watchers and codegen (#192)
* Implement eden-watcher changes in other watchers and codegen

* Use node space size only for eden-watcher
2022-09-23 15:35:15 +05:30
a8fdcca866
Update watcher import CLI to create entities from checkpoint (#186)
* Import entities from checkpoint

* Fix IPLD state when updating subgraph entity

* Changes in codegen and other watchers

* Update IPLD state with all Block entities

* Add verify and create sub commands to checkpoint

* Add option for specifying snapshot block in export state CLI
2022-09-22 15:26:06 +05:30
prathamesh0
b2cf997900
Update CLI to fill state to create checkpoints (#187)
* Update CLI to fill state to create checkpoints

* Accomodate changes in codegen
2022-09-16 09:23:41 +05:30