Commit Graph

33 Commits

Author SHA1 Message Date
62521d7ccc Use postgraphile client to fetch blocks 2021-12-28 16:08:04 +05:30
prathamesh0
bb1345c696 Refactor code to initialize clients (#290)
* Refactor CLI code

* Refactor code to initialize clients

* Don't return config from function to initialize clients
2021-12-28 16:08:04 +05:30
prathamesh0
a846917f1d Export-import watcher state and IPFS integration (#289)
* Add export-watcher CLI generation

* Export snapshot block and add import-watcher CLI gen

* Update codegen docs with export and import commands

* Use different function to get latest checkpoint in export CLI

* Update generated watcher docs with import state instructions

* Push IPLDBlocks to IPFS

* Change ts type for data column to Buffer

* Add checkpoint hook

* Push CLI checkpoint to IPFS

* Fix latest IPLDBlock query to filter out irrelevant blocks

* Add CLI to inspect a CID

* Decode data before exporting

* Encode data in JSON for getState GQL API

* Use go-ipfs and export cbor-encoded buffer

* Remove empty post-block methods from existing watchers

* Call hooks and checkpoint job complete handlers in fill

* Remove subscription to hooks queue from existing watchers
2021-12-28 16:08:04 +05:30
prathamesh0
f2b150995b Boolean state param to indexer methods and IPLDBlocks only in the pruned region (#286)
* Remove checkpoint option for state param. to indexer methods

* Change state param. to indexer methods to boolean

* Use hierarchical query to get the last checkpoint of a contract

* Filter out blocks before prev checkpoint for a new checkpoint

* Add checkpointing job on completion of chain pruning

* Update codegen docs with reset commands

* Create initial checkpoint in the checkpoint queue

* Handle initial checkpoint along with other checkpoints

* Create initial checkpoint job in indexer

* Create IPLDBlocks only in the pruned region

* Create staged diff IPLDBlock for default state
2021-12-28 16:08:04 +05:30
prathamesh0
27104f9d74 Optional server port arg. for watcher generation (#284) 2021-12-28 16:08:04 +05:30
c349b23f29 Add reset CLI generation 2021-12-28 16:08:04 +05:30
prathamesh0
4ddb8c4af6 Auto generating state from indexer methods (#277)
* Generate default derived state for Mapping type variables

* Update IPLDBlock in storage methods instead of using a private class variable

* Default state flag for indexer storage methods

* Helper functions to update state objects

* Add checkpoint flag in Contract table and corresponding changes in existing watchers

* Update codegen docs

* Add examples to generated docs

* Turn default state off by default

* Make state parameter to indexer storage methods default to none

* Add method to get prev. state in indexer
2021-12-28 16:08:04 +05:30
prathamesh0
2aa0234da5 Change initial checkpoint hook and hook status entity naming (#269) 2021-12-28 16:08:04 +05:30
prathamesh0
421e7498d3 Ensuring chronological execution of hooks and checkpointing (#264)
* Subscribe to hooks queue in existing watchers

* Change naming strategy for generated get and save functions

* Push checkpointing job after post-block hook job completed

* Using hooks status to ensure their chronological execution

* Add default indices to IPLDBlock table

* Add kind parameter to getState GQL API

* Add checkpoint CLI

* Add blockHash arg to checkpoint CLI and update codegen docs

* Print out block hash for checkpoint CLI

* Use log from debug for logging

* Filter using contract at start in hierarchical query

* Make kind argument to prepare IPLDBlock required
2021-12-28 16:08:04 +05:30
prathamesh0
51b200709b Generate IPLD blocks table and related GQL API (#260)
* Add ipld-blocks entity generation

* Populate ipld-blocks table

* Rename ipld-block entity and update after each event

* Move ipld-hook to hooks.ts

* Change IPLD block structure

* Add cid field in blocks

* Fetch prev. IPLDBlock for a contract

* GQL API to query IPLDBlock by CID

* Save cid in blocks in existing watchers

* Update codegen docs

* GQL API for getting last derived state (#3)

* GQL API for getting last derived state

* Rename query to getState

* Change query names to getState and getStateByCid

* Save BigInt as string

* Move function to prepare IPLDBlock to indexer

* Refactor IPLDBlock hook

* Add genesis hook

* Call post-block hook after a block is marked as complete

* Add IPLDBlock checkpointing

* Use queryRunner instead of a new repo for queries

* Add a query to get block in ipld-eth-client

* Get latest checkpoints for all contracts for checkpointing.

* Call post-block hook in a queue

* Pass server config to Indexer in watch-contract cli

Co-authored-by: nikugogoi <nabarun@deepstacksoft.com>
2021-12-28 16:08:04 +05:30
b345d25bb8
Optimize and reduce queries in block processing job (#308)
* Improvement TODOs

* Get parent and current block in single query

* Remove block progress queries and events query

* Reduce queries in fetching batch events by using query builder

* Implement changes in codegen package
2021-12-13 15:38:34 +05:30
f56f7a823f
Process events in singe job and avoid block progress query to improve performance (#306)
* Avoid database query by passing event directly to job-queue

* Avoid block progress query by returning from update query

* Process batch of events for a block in a single job

* Fix smoke test for subscribed events and use teamSize for job queue
2021-12-10 10:44:10 +05:30
prathamesh0
f89a7a07aa
Use eth client for getStorageAt query (#302) 2021-12-06 17:22:09 +05:30
63ce6fd55f
Handle missing blocks in postgraphile by making RPC call (#296) 2021-12-03 16:23:11 +05:30
32fea1f2cb
Fix uni-watcher to handle mainnet data (#293)
* Fix watchers to handle mainnet data

* Tweak jobs fetched per interval to reduce event processing time
2021-12-02 15:28:03 +05:30
9d95e49ec9
Change block processing to be pull based (#288)
* Implement pull based watcher for uni-watcher

* Fix same block processed multiple times

* Implement wait time for fetching block from config

* Use blockProgress event to fetch and process next block

* Rename utils index to misc
2021-10-26 17:36:21 +05:30
137f7d1a21
Fix eth_calls with blockHash (#287)
* Fix eth_calls with blockHash

* Use blockHash in erc20 eth_calls
2021-10-25 19:51:16 +05:30
737d9a9b6e
Handle missing blocks by making eth call to ipld-eth-server (#285) 2021-10-22 15:20:11 +05:30
6f3b8029b2
Use blockHash for eth_calls in watchers (#283) 2021-10-21 15:23:46 +05:30
5cbcd455d2
Purge unknown events after block processing complete (#280) 2021-10-20 17:49:44 +05:30
47b9e6bbbd
Implement reset job queue and state CLI in watchers (#276)
* Implement clean jobs CLI in watchers

* Pull common code cleanJobs to util

* Implement commands for reset job-queue and state

* Reset command for job-queues and watcher state

* Reset sync status on reset state

* Fix using cli from built js

* Implement skipCheck to reset syncStatus table and skip complete handler on fail

* Check for block isComplete on reset state

* Set default value for force param to update syncStatus to false

* Reset tables in erc20 watcher

* Push job in fill after previos block is complete
2021-10-20 16:06:03 +05:30
prathamesh0
d3971b5258
Generate GQL client (#259)
* Add GQL client generation

* Add subscription, events, eventsInRange in client gen

* Add mutation in client gen and return event tx info from indexer

* Capitalize class names for storage-mode entities
2021-10-04 11:04:06 +05:30
Ashwin Phatak
a9067c4374
Update codegen docs (#258)
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-30 10:44:30 +05:30
prathamesh0
278fe30a2c
Add GQL mutations support (#256) 2021-09-29 13:49:01 +05:30
prathamesh0
40574cf3d9
Support GQL subscriptions, fill and custom hook for indexing on event (#255)
* Custom hook support for indexing on events.

* Add fill support.

* Process GQL subscriptions.

* Add hooks example.

* Update hooks example.
2021-09-29 10:04:09 +05:30
prathamesh0
8e3093c684
Codegen flag for watcher kind and eventsInRange query support (#254)
* Add watcher kind argument.

* Process eventsInRange query.
2021-09-28 10:22:21 +05:30
Ashwin Phatak
338cef9954
Generate entities from YAML templates and lint support in generated watchers (#253)
* Add lint files generation.

* Fix lint errors in generated code.

* Load default entities from yaml files.

Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-27 18:03:04 +05:30
Ashwin Phatak
11d16b9870
Implement active watcher. (#252)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-27 15:24:17 +05:30
Ashwin Phatak
482be8cfaa
Support storage mode for code generator (#251)
* Implement storage mode in code generator.

* Fix indexer template for eth_call.

* Add support for both eth_call and storage modes.

* Add eventsInRange, events file gen and default entites gen.

* Use static property to column maps in database.

* Process events query.

* Avoid adding duplicate events in indexer.

* Fix generated watcher query with bigint arguments.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-27 10:13:50 +05:30
Ashwin Phatak
92b7967895
Generating eth_call based lazy watcher (#249)
* Add entity generation.

* Add resolvers generation.

* Add queries in resolvers generation.

* Add indexer generation.

* Extract helper code in utils.

* Add server and artifacts generation.

* Fix solidity-flattener issue.

* Update readme and cleanup misc files.

* Add queries to entity generation.

* Add database generation.

* Use snakecase in database.

* Add readme generation.

* Change template file names.

* Add method descriptions.

* Change mode to eth_call in readme.

Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-23 16:55:46 +05:30
Ashwin Phatak
0a96bbd94d
Change dependencies version to be same in all packages. (#250)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-09-21 10:27:34 +05:30
Ashwin Phatak
6560639a68
Use flatten function and modify ResultEvent in schema. (#247)
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-17 17:10:08 +05:30
Ashwin Phatak
f9934675b2
CLI to flatten and generate GQL schema from Solidity file (#245)
* Add schema generation.

* Use yargs and add a script to generate schema.

* Add command to generate schema to README.md

* Remove use of arrow functions in visitor.

* Add function to export schema in visitor.

* Create Event union while adding to it.

* Add bool to typemappings.

* Add method descriptions and remove output type mappings.

* Add storage mode.

* Add URL support and a flattening script.

* Fix parameter for flattening in README.md.

* Add setup and references to README.md.

Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-09-16 17:06:10 +05:30