Commit Graph

416 Commits

Author SHA1 Message Date
prathamesh0
a5b3c7942d
Add test package (#164)
* Add test package

* Add a separate command to perform an eth-call

* Add snapshot test suite

* Add eth-calls for UniswapV2 Pair, USDC, Compound, Dai contracts

* Add args for Uniswap and USDC contracts

* Add args for Compound and Dai contracts

* Add getStorageAt calls to the test suite

* Refactor code and add documentation

* Loop over test slots for getStorageAt calls

* Add support for individual calls

* Use debug for logging
2022-08-19 11:06:11 +05:30
7238f614c0
Update CLI to compare only updated entities and verify IPLD state (#161)
* Change compare CLI to verify only updated entities

* Implement IPLD state verification in compare CLI

* Changes to IPLD state to match with GQL result entity
2022-08-17 19:11:40 +05:30
ec56de057f
Implement method for storage based access in subgraph mapping code (#162)
* Implement storage call in subgraph mapping code

* Add test for mapping type variable storage call

* Use vulcanize graph-ts

* Revert to graph-ts version 0.22.1
2022-08-17 16:25:49 +05:30
prathamesh0
80682e2755
Remove Uniswap watchers and related artifacts (#163)
* Remove Uniswap watchers and related code

* Remove Uniswap artifacts from graph-node

* Remove Uniswap artifacts from graph-test-watcher
2022-08-17 14:04:20 +05:30
bc1c267813
Add GQL query for sync status (#159)
* Add sync status GQL query

* Add sync status GQL query in codegen
2022-08-09 13:25:46 +05:30
5077abc90f
Compute gas price for EIP-1559 transaction (#157) 2022-08-09 10:50:25 +05:30
1a903fccc6
CLI to compare eden-watcher entities and fix mapping code (#149)
* Make sumStaked variable local in eden network mapping

* Implement compare CLI to fetch and query by ids

* Set filterLogs to true for eden-watcher

* Use varchar for bigint array type in eden-watcher

* Store subgraph entities by id in IPLD state

* Store bigint vales as string in IPLD state

* Update eden watcher hook to store single Block entity in IPLD checkpoint

* Fix entity enum type property

* Fix parsing big numbers in event params

* Fix event bigint params parsing in all watchers

* Set default limit to query result and process block after events
2022-08-08 15:58:13 +05:30
6990eb892a
Handle array and struct type event params in codegen (#150)
* Handle array and struct type event params

* Refactor parse event in watcher indexer code
2022-08-04 17:09:30 +05:30
a15305450c
Add prometheus metrics support for watchers (#152)
* Add prometheus metrics endpoint in watcher

* Add event, sync status and DB size metrics

* Fix subgraph watchers DB entities directory path

* Make watcher metrics optional using config
2022-08-03 16:26:51 +05:30
1bcabd64f2
Update codegen with changes implemented in mobymask watcher (#148)
* Update codegen with index-block CLI and remove graph-node

* Add filter logs by contract flag

* Skip generating GQL API for immutable variables

* Add config for maxEventsBlockRange

* Add new flags in existing watchers
2022-07-22 13:17:56 +05:30
b577db287f
Add doc to index blocks out of order using eth-statediff-service (#146)
* Add doc to index blocks using eth-statediff-service

* Add example commands and watcher queries
2022-07-20 12:55:24 +05:30
58bbf4d756
Handle getStorage calls for old blocks (#145)
* Handle getStorage calls for old blocks

* Refactor getStorage code in mobymask watcher

* Fix returning proof as null
2022-07-20 10:37:17 +05:30
e2668690b5
Add filterLogs flag to fetch logs by contract (#144) 2022-07-18 14:04:59 +05:30
9a35955166
Fix job-runner for missing parent block (#143) 2022-07-15 19:08:33 +05:30
129b9e71f0
Add option for no check in max events block range (#142) 2022-07-13 14:40:42 +05:30
3cee10607e
Parse batched transactions for mobymask-watcher events (#141)
* Remove using graph-node from mobymask-watcher

* Parse batched transactions for MobyMask events

* Update to use same ethers package version

* Add CLI for processing block out of order

* Fix job-runner for already processed blocks out of order
2022-07-11 11:29:33 +05:30
74f798b5c4
Rename to watcher-ts (#140) 2022-07-07 12:38:32 +05:30
bd3113069a
Rename to mobymask-watcher (#139) 2022-07-07 11:50:32 +05:30
4b256016da
Update docs and work around GQL client caching in ipld-eth-client (#138) 2022-07-04 15:36:47 +05:30
c919d784bf
Fetch transaction arguments for indexed event (#137) 2022-06-30 15:50:49 +05:30
26d998d3a7 Update codegen to store hash for indexed reference type event args 2022-06-29 14:53:57 +05:30
14a32a7d47 Fix for indexed reference type event arg in moby mask watcher 2022-06-29 14:53:57 +05:30
6cb424b315 Remove ipfs setup from demo 2022-06-29 14:53:57 +05:30
ed75a4dc82 Demo to deploy contract and make transactions 2022-06-29 14:53:57 +05:30
ba73cd24c2 Generate MobyMask watcher using codegen 2022-06-29 14:53:57 +05:30
5a0d844d7f
Use stack-orchestrator for erc721-watcher demo (#132)
* Use stack-orchestrator for erc721-watcher demo

* Run all services in stack-orchestrator with auto mining of blocks

* Add step to check docker-compose version
2022-06-22 17:01:05 +05:30
7d6b86942f
Add GQL API for transferCount entity (#129) 2022-06-16 17:19:57 +05:30
012d2e1a47
Update erc721-watcher demo for IPLD blocks (#125)
* Update demo for IPLD blocks

* Add update for custom properties in state diff

* Implement custom update of transfer counter in state diff

* Comment code for updating custom state prop

* Separate markdown for erc721-watcher Demo

* Run workflow on main branch commit
2022-06-15 10:40:40 +05:30
baa20de443 Add demo for erc721-watcher 2022-06-08 16:43:15 +05:30
99aaa9ae0a Generate erc721-watcher using codegen 2022-06-08 16:43:15 +05:30
prathamesh0
71d34331e9
Deprecate postgraphile usage (#122)
* Deprecate postgraphile usage

* Change endpoint in watcher config

* Remove subscription queries from eth-client

* Remove postgraphile config and client from watchers

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-06-08 12:13:52 +05:30
e4b9596be7
Update ipfs daemon version in readme (#121) 2022-05-26 20:01:31 +05:30
168689a7c7
Fix codegen for non subgraph watchers (#120)
* Fix codegen for non subgraph watchers

* Remove graphWatcher.init from generated non subgraph watcher
2022-05-26 18:00:17 +05:30
b2233e16ef
Fix running erc20-watcher as active kind watcher (#116) 2022-05-23 18:54:14 +05:30
977564d17f Update example subgraph startBlock for demo 2022-05-06 15:43:45 +05:30
ce6568aad2 Add demo for graph-watcher IPLD statediff and checkpointing 2022-05-06 15:43:45 +05:30
ba6b996e65
Deploy example contract in graph-node (#113)
* Deploy example contract in graph-node

* Add auth token in docker build for github packages

* Add steps for running tests in CI
2022-04-29 15:11:40 +05:30
eca09ad66b
Change postgraphile queries to work with v3 schema (#112) 2022-04-28 17:13:32 +05:30
654edf4008
Update README and graph-node tests (#110)
* Update README with proper steps for installing github packages

* Update graph-node readme to run tests
2022-04-27 15:37:13 +05:30
9a8ae3f308
Implement compare CLI for multiple entities for given block range (#108)
* Implement compare CLI for multiple entities at blocks

* Implement caching for gql requests
2022-01-17 15:18:44 +05:30
aabf9f8e15
Implement data source templates (#106)
* Implement data source templates

* Remove optional param from wasm instantiate

* Add all types for solidity to typescript mapping

* Set contract address in context for data source templates

* Implement block handlers for data source templates
2022-01-06 18:32:08 +05:30
4867530da7
Implement subgraph JSON host APIs fromBytes and try_fromBytes (#105)
* Implement JSON fromBytes host API

* Implement json.try_fromBytes host API

* Fill host API code for json number methods
2022-01-03 18:15:20 +05:30
6a2c99a0bb
Implement data source and crypto host APIs (#104)
* Implement data source host APIs

* Implement crypto host API
2021-12-30 17:57:34 +05:30
8b913af93f
Implement ethereum ABI encode and decode in subgraph (#103)
* Implement ethereum ABI encode in subgraph

* Implement ethereum ABI decoding host API

* Implement ABI encode decode for array type

* Implement ABI encode decode for bytes type
2021-12-30 13:12:32 +05:30
9b1aa29afd
Get missing fields in transaction data for subgraph event handlers (#102)
* Get missing fields in event transaction

* Cache transaction data for event handlers in subgraph
2021-12-29 13:21:39 +05:30
561c2c9066 Generate schema from ABI when using subgraph path (#101) 2021-12-28 16:08:05 +05:30
prathamesh0
a267058d51 Add IpldStatus table and IPLD jobs in job-runner in codegen (#100) 2021-12-28 16:08:05 +05:30
prathamesh0
5f4dd14d7a Handle event name conflicts in schema and update hooks in codegen (#97)
* Change return type for subgraph resolvers and update hooks generation

* Handle event name conflicts in schema generation
2021-12-28 16:08:05 +05:30
7a4d0b6bb4 Fixes in eden watcher after comparing with generated watcher (#96) 2021-12-28 16:08:05 +05:30
prathamesh0
e883463aa6 Parse events for multiple contracts in the generated code (#95)
* Parse events for multiple contracts in the generated code

* Use contract wise artifacts in the generated indexer methods

* Update codegen docs to use config file to generate a watcher

* Add watcher generation config to eden-watcher
2021-12-28 16:08:05 +05:30
e5faba8e68 Fixes after comparing with existing eden watcher (#94) 2021-12-28 16:08:05 +05:30
feac6bad16 Remove default ERC20 code from generated hooks (#93) 2021-12-28 16:08:05 +05:30
prathamesh0
44e18fee48 Add watcher generation mode none (#92) 2021-12-28 16:08:05 +05:30
8b5d408f77 Implement Block_height input and definition for custom scalar BigDecimal (#91) 2021-12-28 16:08:05 +05:30
2a204d8a32 Handle additional subgraph types BigDecimal and Bytes in codegen (#89)
* Handle additional subgraph types BigDecimal and Bytes

* Use bigint and Decimal array transformers
2021-12-28 16:08:05 +05:30
prathamesh0
18f73e57c0 Generate relations and entity types maps in indexer (#88) 2021-12-28 16:08:05 +05:30
d96cc095bb Handle relation fields in subgraph entities (#87) 2021-12-28 16:08:05 +05:30
4b1b0e0ed6 Handle subgraph entities field name conflicts and enum types in codegen (#86)
* Handle entity field conflicts

* Handle enum type fields in subgraph entities
2021-12-28 16:08:05 +05:30
prathamesh0
3ed4ab95af Implement IPLD refactoring in codegen (#85) 2021-12-28 16:08:05 +05:30
1e7d84879b Implement changes in codegen for performance improvements (#84) 2021-12-28 16:08:05 +05:30
prathamesh0
97b529f3f2 Update CLI generation (#83) 2021-12-28 16:08:05 +05:30
prathamesh0
5b12db541b Support artifacts generation for multiple contracts (#82) 2021-12-28 16:08:05 +05:30
prathamesh0
3638d56787 Add checks for and create hooks, checkpoint and IPFS jobs in job-runner (#99)
* Add fields for checkpoint and IPFS in IpldStatus table

* Mark block processing job complete in watchers
2021-12-28 16:08:05 +05:30
dee517e444 Implement IpldStatus table for IPLD related jobs (#98) 2021-12-28 16:08:05 +05:30
prathamesh0
d385db2775 Add missing relation fields in eden-watcher relations map (#90) 2021-12-28 16:08:05 +05:30
198e49e5a0 Implement WASM instance restart to handle out of memory error (#81)
* Test case for wasm out of memory error

* Restart wasm instance after N blocks

* Handle out of memory error and re instantiate WASM

* Remove old instance from map before reinstantiating WASM
2021-12-28 16:08:05 +05:30
68bc1c00db Refactor watchers for util package changes 2021-12-28 16:08:05 +05:30
prathamesh0
560df57ac7 Performance improvements for IPLDBlocks in eden watcher (#80)
* Avoid fetching contracts while creating a checkpoint

* Use enum for state kind and cache ipld status

* Avoid fetching block twice while finalizing a staged diff IPLDBlock

* Create checkpoints at fixed block numbers and refactor checkpointing code

* Avoid calling block handler until start block is reached

* Use delete while finalizing staged diff IPLDBlocks

* Add a check to ensure hooks job is created only once

* Avoid check for initial state while creating a checkpoint
2021-12-28 16:08:05 +05:30
prathamesh0
952f68ee58 Add time logs for eden-watcher (#78) 2021-12-28 16:08:05 +05:30
f4d16afa6c Refactor watchers according to changes in util package 2021-12-28 16:08:05 +05:30
768a4d0818 Use Promise all while fetching events and watch contracts only once 2021-12-28 16:08:05 +05:30
ba890e7d9a Add method in watchers to cache contract and fixes for running tests 2021-12-28 16:08:05 +05:30
prathamesh0
cb2fe2aa45 Store initial state and refactor IPLDBlocks related code (#76)
* Process hooks for the first block in the watchers

* Store initial state from a hook in an IPLDBlock for eden-watcher

* Extract watcher method to prepare IPLDBlocks in util

* Extract common IPLDBlocks related code in util

* Move IPFSClient to util

* Use constants for state kind
2021-12-28 16:08:05 +05:30
prathamesh0
4c422e3ea2 Avoid setting null entity properties while fetching in store get API (#75) 2021-12-28 16:08:05 +05:30
prathamesh0
3b835e81f8 Pass provider while instantiating the loader and use block hash while making the eth-calls (#74)
* Pass provider while instantiating the loader

* Use custom provider for graph-node tests

* Use block hash while making an eth-call
2021-12-28 16:08:05 +05:30
prathamesh0
3d3ebb0e43 Get event handler using cleaned subgraph event signature (#73) 2021-12-28 16:08:05 +05:30
d4e79ceee4 Add a method in test indexer to get blocks 2021-12-28 16:08:05 +05:30
prathamesh0
3e0c84b333 Use entity column type map to create entity in store get API (#72)
* Use typeof to distinguish between BigInt and BigDecimal in store get API

* Use entity column type map to create entity in store get API

* Add entity column type map in eden-watcher
2021-12-28 16:08:05 +05:30
prathamesh0
5f03ad5029 Use pg-type numeric for bigint in entities and use custom decimal in graph-node (#71)
* Use pg-type numeric for bigint columns in typeorm entities

* Use custom decimal implementation from util in graph-node
2021-12-28 16:08:05 +05:30
prathamesh0
475c34b3fa Add an executable for compare-entity CLI (#69) 2021-12-28 16:08:05 +05:30
prathamesh0
c79647548e Add more test cases for numeric host APIs (#66)
* Add test cases for bigInt host APIs

* Add test cases for bigDecimal fromString, plus and minus host APIs

* Add test cases for bigDecimal times and divideBy host APIs

* Throw an error on underflow or overflow in bigDecimal toString host API

* Avoid rounding of out of range bigDecimal values

* Use big number's byte size to set storage size of bigInt

* Add test cases for errors in bigDecimal host API
2021-12-28 16:08:05 +05:30
c75be57146 Get block size using JSON RPC API from ipld-eth-server (#68) 2021-12-28 16:08:05 +05:30
86dcee12f7 Get author field for block data 2021-12-28 16:08:05 +05:30
30f3c9e694 Fix block processing job created twice when processing missing parent block (#67)
* Fix block processing job created twice in watcher

* Fix block processing job for missing parent blocks
2021-12-28 16:08:05 +05:30
faf046d181 Implement graph-ts numbers, typeConversion and log host APIs (#64)
* Implement host api for bigint operations

* Implement type conversion APIs

* Avoid use of exports as variable name for conflict

* Implement log API with debug and levels

* Fix job-runnner hook error after stopping and starting chain
2021-12-28 16:08:05 +05:30
prathamesh0
970092ece2 Update graph-node tests to use dummy indexer (#65)
* Update test to call handlers to use dummy indexer

* Update graph-node tests to use dummy indexer
2021-12-28 16:08:05 +05:30
prathamesh0
94e9182dd3 Handle BigNumber event params and customize Decimal (#63)
* Handle BigNumber event params in watchers

* Customize decimal according to limits of IEEE-754 decimal128

* Add definition for custom scalar BigDecimal
2021-12-28 16:08:05 +05:30
238ad21189 Handle subgraph schema field with derivedFrom directive (#60)
* Handle subgraph schema field with derivedFrom directive

* Handle derivedFrom directive in eden-watcher

* Fix 1 to N relation error by removing limit from query

* Order by id for derivedFrom relations to match graph-node

* Refactor example subgraph schema entities

* Fix watcher queries to return correct relation field values

* Fix hierarchical query for getting two entities at same block
2021-12-28 16:08:05 +05:30
prathamesh0
b04f6f2fba Implement bigDecimal host API (#62)
* Handle negative bigInt in store host API

* Increase size of bigInt in fromString API to support UInt256

* Implement bigDecimal divideBy host API

* Implement bigDecimal plus, minus and times host API

* Implement bigInt dividedByDecimal and mod host API

* Change BN js version to that used by ethers js
2021-12-28 16:08:05 +05:30
prathamesh0
129ba694fb Create external checkpoints after hook processing and clear subgraph entities on reset (#61)
* Create checkpoint using CLI after the hooks have been processed

* Push checkpoints created while exporting state to IPFS

* Clear subgraph entities on reset state

* Remove unnecessary staged IPLD blocks created while importing state
2021-12-28 16:08:05 +05:30
prathamesh0
d979e51723 Add a CLI to compare entity query results from two GQL endpoints (#57)
* Add CLI to compare entities from two GQL endpoints

* Print out result diffs in compare-entity CLI

* Get the colorized result diff in compare-entity CLI

* Read query dir from config file or as an arg

* Make config file arg required

* Make queries in Example schema similar to that in graph-node

* Get non-colorized output on choosing raw-json diff option

* Make queries in eden-watcher similar to that in graph-node
2021-12-28 16:08:05 +05:30
31b302c9b5 Implement eth_call for methods returning struct type (#59)
* Handle tuple return type in ethereum host API

* Update graph-cli version to fix eth_call error

* Handle all types in struct based on abi

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-12-28 16:08:05 +05:30
f52467f724 Implement gql queries for relation entities similar to subgraph (#53)
* Add implementation for one to one relation

* Implement one to many relation in gql queries

* Make changes for gql relation queries in eden-watcher

* Implement subgraph gql relation queries with joins
2021-12-28 16:08:05 +05:30
prathamesh0
44b3fd59e8 Add watched contract from subgraph yaml on startup (#56) 2021-12-28 16:08:05 +05:30
prathamesh0
af259a32f0 Update eden subgraph build and watch subgraph contracts (#54)
* Update eden subgraph build

* Watch subgraph contracts in the job-runner

* Change eden subgraph startBlocks to original values
2021-12-28 16:08:05 +05:30
nikugogoi
d76268e506 Call subgraph block handlers and fix eden-watcher dependency (#52)
* Call subgraph block handler in watchers

* Fix eden-watcher dependency for build

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2021-12-28 16:08:05 +05:30
prathamesh0
ec3a8a31a7 Get missing block fields (#51)
* Decode header and get missing block fields

* Move method to get block data to misc in util

* Remove unnecessary encoding of header data
2021-12-28 16:08:05 +05:30
nikugogoi
800ad79baf Handle subgraph schema entity array type and relation fields (#49)
* Handle relation entities in subgraph

* Modify eden-watcher entities to handle subgraph schema data types

* Modify schema gql to match eden subgraph gql shape

* Handle array type fields in subgraph schema

* Fix store get api for array type fields in subgraph entities

* Handle array type in eden-watcher and format await used in params
2021-12-28 16:08:05 +05:30
prathamesh0
158c3928c9 Handle type and field name conflicts for eden-watcher subgraph entities (#50)
* Rename Block, Transaction types and add block query in eden-watcher schema

* Handle field name conflicts in eden subgraph entities

* Resolve entity field name conflicts while sending data for auto-diff
2021-12-28 16:08:05 +05:30
prathamesh0
fd86b4d5e3 Parse events of all contracts in eden-watcher (#48)
* Parse events of all contracts in eden-watcher

* Parse event of each contract in a separate function
2021-12-28 16:08:05 +05:30
prathamesh0
b7ae655c20 Add eden-watcher generated using codegen (#47)
* Add a watcher for EdenNetwork contract

* Add events from MerkleDistributor contract to eden-watcher

* Add ERC721 Transfer and Approval events to eden-watcher schema

* Add artifacts for DistributorGovernance and MerkleDistributor contracts
2021-12-28 16:08:05 +05:30
nikugogoi
47cef8d5d7 Add support for subgraph schema data types (#46)
* Fix packages build

* Add support for boolean type in subgraph schema

* Add BigDecimal fromString and toString implementation

* Add implementation for subgraph Bytes type

* Implement enum type for subgraph schema

* Add implementation for subgraph schema BigDecimal type

* JSON stringify and parse entity data for auto-diff
2021-12-28 16:08:05 +05:30
prathamesh0
73dcb882a6 Update graph-test-watcher (#45) 2021-12-28 16:08:05 +05:30
prathamesh0
16bb955213 Invoke subgraph block handlers (#43)
* Add a test case to eden test to call the block handler

* Add a block handler in example subgraph and call it in a watcher

* Use an array map to call all the block handlers for a contract

* Await on all the promises returned by block handlers map
2021-12-28 16:08:05 +05:30
prathamesh0
06ba24e38f Generate GQL API for subgraph entities and auto-diff based on store set (#38)
* Add subgraph schema types to the generated schema

* Add queries for subgraph entities

* Add entity generation for subgraph entities

* Call subgraph event handler in indexer

* Refactor subgraph schema and entity generation

* Add resolvers generation for subgraph entities

* Get event signature in the event

* Add NonNullType check for field type in entity generation

* Auto-diff based on store set

* Use contract address from data source in loader

* Change subgraph-schema arg to subgraph-path arg
2021-12-28 16:08:05 +05:30
83775608ec Implement subgraph store host API (#35)
* Implement store get api without blockHash and blockNumber

* Pass database instance to GraphWatcher

* Implement store set without block data

* Store blockHash and blockNumber in database entity table

* Implement getting entity in subgraph from store.get

* Add block data present in postgraphile

* Pass db and context to instantiate method in tests

* GQL API in graph-test-watcher to test store.set

* Remove contract address from subgraph file

* Fix block in dummy event data

* Pass just blockHash to get an entity from the database

* Review changes and add TODOs

Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
2021-12-28 16:08:05 +05:30
43d64f9e4b Invoke subgraph handler in watcher event processing (#34)
* Invoke subgraph handler in watcher event processing

* Fix error when invoking subgraph handler

* Parse events using event signature specified in subgraph yaml

* Use contract abi to parse event params

* Invoke event handler based on event signature

* Fill event with block and transaction data

* Comment missing fields in block and transaction data
2021-12-28 16:08:05 +05:30
6cca55a1ab Integrate generated watcher to invoke handlers in graph-node (#33)
* Invoke handlers based on watcher-ts events

* Read subgraph yaml and listen to events from watcher-ts

* Create GraphWatcher class to use in generated example contract watcher

* Call graph-node event handler from generated watcher
2021-12-28 16:08:05 +05:30
1c15c1eedb Test case for EdenNetwork event handler (#31)
* Test case for calling EdenNetwork event handler

* Test cases for calling EdenNetwork event handlers

* Test cases for calling event handlers in EdenNetworkDistribution

* Test cases for calling event handlers in EdenNetworkGovernance

* Perform eth-calls according to contract name in param

* Pass function params in eth-call host API

* Return value from eth-call to wasm

* Add instructions for running tests

* Install assemblyscript from github packages

* Add instrcutions for installing private github packages

* Use vulcanize assemblyscript package
2021-12-28 16:08:05 +05:30
1ce07bbb6e Implement BigInt times and dividedBy APIs (#30)
* Add script for building example subgraph

* Implement BigInt times and dividedBy API
2021-12-28 16:08:05 +05:30
ca01fa788d Test for calling ethereum host API and updated eden subgraph (#29)
* Implement graph-ts eth-call API

* Tests for eden subgraph
2021-12-28 16:08:05 +05:30
f34d83c04b Use async functions in wasm imports (#28)
* Use async functions in wasm imports

* Complete implementing asyncify with assemblyscript loader
2021-12-28 16:08:05 +05:30
Ashwin Phatak
bf54e85d05 eth_call and invoking event handler in WASM (#25)
* Perform eth_call and get result in js import

* Use dummy value for eth_call

* Called handler in wasm with event param

* Implement passing event to subgraph handler function

* Use generated event class Test for passing to handler

* Pass event params to handler

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-12-28 16:08:05 +05:30
2bc40896b0 Point graph-cli to use fork for automatic graph-ts exports (#22) 2021-12-28 16:08:05 +05:30
e10f61ba61 Implement typeConversion APIs bytesToHex, bigIntToString and stringToH160 (#19)
* Implement host api typeConversion.bytesToHex

* Complete host api bigIntToString.

* Create TypeId for assemblyscript loader.

* Implement host apis bigInt fromString, plus and minus
2021-12-28 16:08:04 +05:30
Ashwin Phatak
d247815ce2 Example subgraph, ethCall test to drive implementation (#18)
* Setup example subgraph.

* Implement eth_call in subgraph.

* eth-call test stub

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-12-28 16:08:04 +05:30
Ashwin Phatak
f078db688a Eden subgraph artifacts, host API stub (#15)
* Skip AssemblyScript during TS transpilation

* Eden subgraph build files

* Eden subgraph host API stubs
2021-12-28 16:08:04 +05:30
Ashwin Phatak
9e6f6b4519 AssemblyScript loader experiments (#13) 2021-12-28 16:08:04 +05:30
Ashwin Phatak
889e96572d Host API stubs (#12) 2021-12-28 16:08:04 +05:30
Ashwin Phatak
1421ba5a9b WASM instance setup (#11)
* Code for running wasm file.

* Add test for wasm exported function.

* Use target names for build files.

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-12-28 16:08:04 +05:30
Ashwin Phatak
987457992a Create graph-node package. (#10)
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2021-12-28 16:08:04 +05:30
62521d7ccc Use postgraphile client to fetch blocks 2021-12-28 16:08:04 +05:30
prathamesh0
bb1345c696 Refactor code to initialize clients (#290)
* Refactor CLI code

* Refactor code to initialize clients

* Don't return config from function to initialize clients
2021-12-28 16:08:04 +05:30
prathamesh0
a846917f1d Export-import watcher state and IPFS integration (#289)
* Add export-watcher CLI generation

* Export snapshot block and add import-watcher CLI gen

* Update codegen docs with export and import commands

* Use different function to get latest checkpoint in export CLI

* Update generated watcher docs with import state instructions

* Push IPLDBlocks to IPFS

* Change ts type for data column to Buffer

* Add checkpoint hook

* Push CLI checkpoint to IPFS

* Fix latest IPLDBlock query to filter out irrelevant blocks

* Add CLI to inspect a CID

* Decode data before exporting

* Encode data in JSON for getState GQL API

* Use go-ipfs and export cbor-encoded buffer

* Remove empty post-block methods from existing watchers

* Call hooks and checkpoint job complete handlers in fill

* Remove subscription to hooks queue from existing watchers
2021-12-28 16:08:04 +05:30
prathamesh0
f2b150995b Boolean state param to indexer methods and IPLDBlocks only in the pruned region (#286)
* Remove checkpoint option for state param. to indexer methods

* Change state param. to indexer methods to boolean

* Use hierarchical query to get the last checkpoint of a contract

* Filter out blocks before prev checkpoint for a new checkpoint

* Add checkpointing job on completion of chain pruning

* Update codegen docs with reset commands

* Create initial checkpoint in the checkpoint queue

* Handle initial checkpoint along with other checkpoints

* Create initial checkpoint job in indexer

* Create IPLDBlocks only in the pruned region

* Create staged diff IPLDBlock for default state
2021-12-28 16:08:04 +05:30
prathamesh0
27104f9d74 Optional server port arg. for watcher generation (#284) 2021-12-28 16:08:04 +05:30
c349b23f29 Add reset CLI generation 2021-12-28 16:08:04 +05:30
prathamesh0
4ddb8c4af6 Auto generating state from indexer methods (#277)
* Generate default derived state for Mapping type variables

* Update IPLDBlock in storage methods instead of using a private class variable

* Default state flag for indexer storage methods

* Helper functions to update state objects

* Add checkpoint flag in Contract table and corresponding changes in existing watchers

* Update codegen docs

* Add examples to generated docs

* Turn default state off by default

* Make state parameter to indexer storage methods default to none

* Add method to get prev. state in indexer
2021-12-28 16:08:04 +05:30
prathamesh0
2aa0234da5 Change initial checkpoint hook and hook status entity naming (#269) 2021-12-28 16:08:04 +05:30
prathamesh0
421e7498d3 Ensuring chronological execution of hooks and checkpointing (#264)
* Subscribe to hooks queue in existing watchers

* Change naming strategy for generated get and save functions

* Push checkpointing job after post-block hook job completed

* Using hooks status to ensure their chronological execution

* Add default indices to IPLDBlock table

* Add kind parameter to getState GQL API

* Add checkpoint CLI

* Add blockHash arg to checkpoint CLI and update codegen docs

* Print out block hash for checkpoint CLI

* Use log from debug for logging

* Filter using contract at start in hierarchical query

* Make kind argument to prepare IPLDBlock required
2021-12-28 16:08:04 +05:30
prathamesh0
51b200709b Generate IPLD blocks table and related GQL API (#260)
* Add ipld-blocks entity generation

* Populate ipld-blocks table

* Rename ipld-block entity and update after each event

* Move ipld-hook to hooks.ts

* Change IPLD block structure

* Add cid field in blocks

* Fetch prev. IPLDBlock for a contract

* GQL API to query IPLDBlock by CID

* Save cid in blocks in existing watchers

* Update codegen docs

* GQL API for getting last derived state (#3)

* GQL API for getting last derived state

* Rename query to getState

* Change query names to getState and getStateByCid

* Save BigInt as string

* Move function to prepare IPLDBlock to indexer

* Refactor IPLDBlock hook

* Add genesis hook

* Call post-block hook after a block is marked as complete

* Add IPLDBlock checkpointing

* Use queryRunner instead of a new repo for queries

* Add a query to get block in ipld-eth-client

* Get latest checkpoints for all contracts for checkpointing.

* Call post-block hook in a queue

* Pass server config to Indexer in watch-contract cli

Co-authored-by: nikugogoi <nabarun@deepstacksoft.com>
2021-12-28 16:08:04 +05:30
7b6f6e468f
Move update sync status outside job-runner (#324) 2021-12-21 15:46:05 +05:30
prathamesh0
2bcf579859
Remove unknown events of parent block when processing a block (#323) 2021-12-20 17:00:14 +05:30
prathamesh0
5b632a72aa
Store contract addresses in uni-info-watcher entities in lowercase (#322) 2021-12-17 17:55:41 +05:30
5dc6582ec8
Log time taken for processing single block inside job-runner (#321) 2021-12-17 15:20:33 +05:30
prathamesh0
01fa6e2184
Fix asserts for nullable values in uni-info-watcher mapping code (#320) 2021-12-17 11:57:32 +05:30
cfd293f11b
Fix job-runner stop and restart in uni-watcher (#319)
* Fix job-runner stop and restart in uni-watcher

* Fix blocks processed twice after processing missing parent block
2021-12-17 11:57:09 +05:30
prathamesh0
d087667177
Update sync status after pruning before processing next block (#318)
* Update sync status after pruning before processing next block

* Exit gracefully on error while prefetching blocks
2021-12-17 10:23:04 +05:30
105b26d6a3
Performance improvements for fill CLI (#314)
* Fix fill cli to work with watcher server

* Add createdAt column and insert events in batches

* Implement prefetch to fill block and events in parallel

* Fix getPrevEntity and increase fill prefetch default batch size

* Fix watcher creating mulitple jobs for a block
2021-12-16 17:16:48 +05:30
b06d931054
Add time logs for uniswap watchers (#310)
* Log time taken in watchers

* Log time for all events and for each event batch

Co-authored-by: prathamesh0 <prathamesh@deepstacksoft.com>
2021-12-14 14:48:29 +05:30
b345d25bb8
Optimize and reduce queries in block processing job (#308)
* Improvement TODOs

* Get parent and current block in single query

* Remove block progress queries and events query

* Reduce queries in fetching batch events by using query builder

* Implement changes in codegen package
2021-12-13 15:38:34 +05:30
f56f7a823f
Process events in singe job and avoid block progress query to improve performance (#306)
* Avoid database query by passing event directly to job-queue

* Avoid block progress query by returning from update query

* Process batch of events for a block in a single job

* Fix smoke test for subscribed events and use teamSize for job queue
2021-12-10 10:44:10 +05:30
prathamesh0
a2ad139769
Use node v16 and pprof to generate profile data for watchers (#299)
* Use pprof to generate profile data for watchers

* Update docs and github workflows to use node v16
2021-12-09 10:39:35 +05:30
6f98166c49
Changes to improve performance for event processing job (#304)
* Perf improvement TODOs

* Move watch contract to util package

* Get block progress from event instead of querying

* Use index field in event to check order of processing

* Use watched contract map to avoid querying database

* Use update query for blockProgress entity

Co-authored-by: Ashwin Phatak <ashwinpphatak@gmail.com>
2021-12-08 11:11:29 +05:30
prathamesh0
f89a7a07aa
Use eth client for getStorageAt query (#302) 2021-12-06 17:22:09 +05:30
ec586216db
Fix events out of order in job-queue (#300) 2021-12-06 17:17:55 +05:30
cda55646d2
Update uni-info-watcher with changes in uniswap subgraph mapping (#298)
* Update mapping code with latest subgraph changes

* Add mapping code for TickDayData entity
2021-12-03 16:24:13 +05:30
63ce6fd55f
Handle missing blocks in postgraphile by making RPC call (#296) 2021-12-03 16:23:11 +05:30
32fea1f2cb
Fix uni-watcher to handle mainnet data (#293)
* Fix watchers to handle mainnet data

* Tweak jobs fetched per interval to reduce event processing time
2021-12-02 15:28:03 +05:30
prathamesh0
08c712d766
Handle bigint and decimal types for entities in watchers (#295)
* Add a GraphDecimal class to replace Decimal with

* Replace Decimal in uni-info-watcher with GraphDecimal

* Add definition for custom scalar BigDecimal in uni-info-watcher

* Use pg-type numeric and transformer for bigint columns in typeorm entities
2021-12-02 13:22:29 +05:30