Commit Graph
192 Commits
Author SHA1 Message Date
nabarun cfd293f11b Fix job-runner stop and restart in uni-watcher (#319)
* Fix job-runner stop and restart in uni-watcher

* Fix blocks processed twice after processing missing parent block
2021-12-17 11:57:09 +05:30
prathamesh0 d087667177 Update sync status after pruning before processing next block (#318)
* Update sync status after pruning before processing next block

* Exit gracefully on error while prefetching blocks
2021-12-17 10:23:04 +05:30
nabarun 105b26d6a3 Performance improvements for fill CLI (#314)
* Fix fill cli to work with watcher server

* Add createdAt column and insert events in batches

* Implement prefetch to fill block and events in parallel

* Fix getPrevEntity and increase fill prefetch default batch size

* Fix watcher creating mulitple jobs for a block
2021-12-16 17:16:48 +05:30
nabarunandprathamesh b06d931054 Add time logs for uniswap watchers (#310)
* Log time taken in watchers

* Log time for all events and for each event batch

Co-authored-by: prathamesh0 <prathamesh@deepstacksoft.com>
2021-12-14 14:48:29 +05:30
nabarun b345d25bb8 Optimize and reduce queries in block processing job (#308)
* Improvement TODOs

* Get parent and current block in single query

* Remove block progress queries and events query

* Reduce queries in fetching batch events by using query builder

* Implement changes in codegen package
2021-12-13 15:38:34 +05:30
nabarun f56f7a823f Process events in singe job and avoid block progress query to improve performance (#306)
* Avoid database query by passing event directly to job-queue

* Avoid block progress query by returning from update query

* Process batch of events for a block in a single job

* Fix smoke test for subscribed events and use teamSize for job queue
2021-12-10 10:44:10 +05:30
prathamesh0 a2ad139769 Use node v16 and pprof to generate profile data for watchers (#299)
* Use pprof to generate profile data for watchers

* Update docs and github workflows to use node v16
2021-12-09 10:39:35 +05:30
nabarunandAshwin Phatak 6f98166c49 Changes to improve performance for event processing job (#304)
* Perf improvement TODOs

* Move watch contract to util package

* Get block progress from event instead of querying

* Use index field in event to check order of processing

* Use watched contract map to avoid querying database

* Use update query for blockProgress entity

Co-authored-by: Ashwin Phatak <ashwinpphatak@gmail.com>
2021-12-08 11:11:29 +05:30
prathamesh0 f89a7a07aa Use eth client for getStorageAt query (#302) 2021-12-06 17:22:09 +05:30
nabarun ec586216db Fix events out of order in job-queue (#300) 2021-12-06 17:17:55 +05:30
nabarun cda55646d2 Update uni-info-watcher with changes in uniswap subgraph mapping (#298)
* Update mapping code with latest subgraph changes

* Add mapping code for TickDayData entity
2021-12-03 16:24:13 +05:30
nabarun 63ce6fd55f Handle missing blocks in postgraphile by making RPC call (#296) 2021-12-03 16:23:11 +05:30
nabarun 32fea1f2cb Fix uni-watcher to handle mainnet data (#293)
* Fix watchers to handle mainnet data

* Tweak jobs fetched per interval to reduce event processing time
2021-12-02 15:28:03 +05:30
prathamesh0 08c712d766 Handle bigint and decimal types for entities in watchers (#295)
* Add a GraphDecimal class to replace Decimal with

* Replace Decimal in uni-info-watcher with GraphDecimal

* Add definition for custom scalar BigDecimal in uni-info-watcher

* Use pg-type numeric and transformer for bigint columns in typeorm entities
2021-12-02 13:22:29 +05:30
nabarun 9d95e49ec9 Change block processing to be pull based (#288)
* Implement pull based watcher for uni-watcher

* Fix same block processed multiple times

* Implement wait time for fetching block from config

* Use blockProgress event to fetch and process next block

* Rename utils index to misc
2021-10-26 17:36:21 +05:30
nabarun 137f7d1a21 Fix eth_calls with blockHash (#287)
* Fix eth_calls with blockHash

* Use blockHash in erc20 eth_calls
2021-10-25 19:51:16 +05:30
nabarun 737d9a9b6e Handle missing blocks by making eth call to ipld-eth-server (#285) 2021-10-22 15:20:11 +05:30
nabarun 6f3b8029b2 Use blockHash for eth_calls in watchers (#283) 2021-10-21 15:23:46 +05:30
nabarun 7953ba2949 Replace storage calls in uni-info-watcher with eth_calls (#281)
* Replace storage calls in uni-info-watcher with eth_calls

* Handle eth_call reverts and catch them in uni-info-watcher
2021-10-21 13:24:46 +05:30
nabarun 5cbcd455d2 Purge unknown events after block processing complete (#280) 2021-10-20 17:49:44 +05:30
nabarun 47b9e6bbbd Implement reset job queue and state CLI in watchers (#276)
* Implement clean jobs CLI in watchers

* Pull common code cleanJobs to util

* Implement commands for reset job-queue and state

* Reset command for job-queues and watcher state

* Reset sync status on reset state

* Fix using cli from built js

* Implement skipCheck to reset syncStatus table and skip complete handler on fail

* Check for block isComplete on reset state

* Set default value for force param to update syncStatus to false

* Reset tables in erc20 watcher

* Push job in fill after previos block is complete
2021-10-20 16:06:03 +05:30
nabarun f00ea0c1f0 Implement lazy watcher mode in erc20-watcher (#275) 2021-10-18 13:03:55 +05:30
nabarun 3b27177389 Fix query when filtering by bigint type field (#268) 2021-10-14 17:19:25 +05:30
nabarun e0933f10b2 Resolve nested relations required by frontend app (#267) 2021-10-14 12:30:23 +05:30
nabarun bc7588599d Increase sleep in smoke test to process events (#263) 2021-10-13 15:34:37 +05:30
nabarun 08c0668ddb Get endpoint URL from config in tests (#262) 2021-10-13 11:01:36 +05:30
prathamesh0 d3971b5258 Generate GQL client (#259)
* Add GQL client generation

* Add subscription, events, eventsInRange in client gen

* Add mutation in client gen and return event tx info from indexer

* Capitalize class names for storage-mode entities
2021-10-04 11:04:06 +05:30
Ashwin Phatakandprathamesh a9067c4374 Update codegen docs (#258)
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-30 10:44:30 +05:30
nabarun d456a3f676 Include NFPM collect event in smoke tests. (#257) 2021-09-29 17:14:16 +05:30
prathamesh0 278fe30a2c Add GQL mutations support (#256) 2021-09-29 13:49:01 +05:30
prathamesh0 40574cf3d9 Support GQL subscriptions, fill and custom hook for indexing on event (#255)
* Custom hook support for indexing on events.

* Add fill support.

* Process GQL subscriptions.

* Add hooks example.

* Update hooks example.
2021-09-29 10:04:09 +05:30
prathamesh0 8e3093c684 Codegen flag for watcher kind and eventsInRange query support (#254)
* Add watcher kind argument.

* Process eventsInRange query.
2021-09-28 10:22:21 +05:30
Ashwin Phatakandprathamesh 338cef9954 Generate entities from YAML templates and lint support in generated watchers (#253)
* Add lint files generation.

* Fix lint errors in generated code.

* Load default entities from yaml files.

Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-27 18:03:04 +05:30
Ashwin Phatakandnabarun 11d16b9870 Implement active watcher. (#252)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-27 15:24:17 +05:30
482be8cfaa Support storage mode for code generator (#251)
* Implement storage mode in code generator.

* Fix indexer template for eth_call.

* Add support for both eth_call and storage modes.

* Add eventsInRange, events file gen and default entites gen.

* Use static property to column maps in database.

* Process events query.

* Avoid adding duplicate events in indexer.

* Fix generated watcher query with bigint arguments.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-27 10:13:50 +05:30
Ashwin Phatakandprathamesh 92b7967895 Generating eth_call based lazy watcher (#249)
* Add entity generation.

* Add resolvers generation.

* Add queries in resolvers generation.

* Add indexer generation.

* Extract helper code in utils.

* Add server and artifacts generation.

* Fix solidity-flattener issue.

* Update readme and cleanup misc files.

* Add queries to entity generation.

* Add database generation.

* Use snakecase in database.

* Add readme generation.

* Change template file names.

* Add method descriptions.

* Change mode to eth_call in readme.

Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-23 16:55:46 +05:30
Ashwin Phatakandnabarun 4f8f1d8cd7 Update code for log_cids change in vdb core (#243)
* Remove fields no longer present in schema.

* Refactor code to work with log_cids change in eth-server.

* Refactor process event to save events from logs.

* Use constants for erc20 event names.

* Implement watch blocks in erc20-watcher similar to uni-watcher.

* Moved common methods to util.

* Implement eventsInRange query in erc20-watcher.

* Filter unknown event in database query.

* Change dependencies version to be same in all packages.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-21 16:43:55 +05:30
Ashwin Phatakandnabarun 0a96bbd94d Change dependencies version to be same in all packages. (#250)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-21 10:27:34 +05:30
Ashwin Phatakandnabarun c973f3b343 Add husky for setting lint in commit hook. (#248)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-20 16:25:54 +05:30
Ashwin Phatakandprathamesh 6560639a68 Use flatten function and modify ResultEvent in schema. (#247)
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-17 17:10:08 +05:30
Ashwin Phatakandprathamesh f9934675b2 CLI to flatten and generate GQL schema from Solidity file (#245)
* Add schema generation.

* Use yargs and add a script to generate schema.

* Add command to generate schema to README.md

* Remove use of arrow functions in visitor.

* Add function to export schema in visitor.

* Create Event union while adding to it.

* Add bool to typemappings.

* Add method descriptions and remove output type mappings.

* Add storage mode.

* Add URL support and a flattening script.

* Fix parameter for flattening in README.md.

* Add setup and references to README.md.

Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-16 17:06:10 +05:30
Ashwin Phatakandnabarun a9dc34f704 Fix null parent. (#244)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-09 10:51:58 +05:30
06bed1c13b Enable source maps for transpiled files (#242)
* Rename .spec files to .test files.

* Enable source maps.

* Pass enable-source-maps flag for transpiled files.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-07 16:01:15 +05:30
Ashwin Phatakandnabarun d3a50d2b18 Fix package versions after cleaning dependencies. (#241)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-02 16:13:55 +05:30
Ashwin Phatakandnabarun 2d493b3838 Run services from transpiled ES5 (#239)
* Transpile ES5 code.

* Use transpiled ES5 code in all packages.

* Refactor tracing-client to transpile ES5 code.

* Order scripts in package json.

* Update readme in packages with new scripts.

* Refactor code to use entity config programatically.

* Remove unused depencdencies.

* Use DEBUG environment for scripts.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-02 10:27:56 +05:30
Ashwin Phatakandnabarun fc4f49113b Bug fixes from smoke testing (#238)
* Fix update for entity relations.

* Implement demo values in uni-info-watcher indexer code.

* amend

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-31 18:05:29 +05:30
Ashwin Phatakandnabarun f1f70ad7d4 Move chain pruning to block processing queue with higher priority (#235)
* Move chain pruning to block processing queue.

* Refactor common code in util.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-30 11:16:54 +05:30
Ashwin Phatakandnabarun 458956a3a1 Fix ts-node watch for file changes. (#234)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-27 18:14:23 +05:30
Ashwin Phatakandnabarun cf0713eda7 Tasks in erc20-watcher to deploy and transfer tokens. (#233)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-27 17:47:23 +05:30
Ashwin Phatakandnabarun 8fa6229562 Fix queries, demo mode to display all tokens (#231)
* Fix queries for frontend info app.

* Fix query for nested relations.

* Add demo mode in uni-info-watcher.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-08-26 15:36:16 +05:30