Commit Graph

479 Commits

Author SHA1 Message Date
f619c71b42 Add connection check for all remote peers 2023-02-02 19:50:27 +05:30
69e9402dc1
Fix getting bytes value length from storage slot in solidity mapper (#313)
* Add failing test case for bytes with leading zeros

* Fix getting bytes value length from storage slot
2023-02-02 17:49:31 +05:30
28023d834a
Remove peer-test-app react app (#312) 2023-02-02 12:39:55 +05:30
abb47fc7cc
Upgrade version to 0.2.25 (#311) 2023-02-01 19:07:45 +05:30
dfc3f5a63e
Move out react-peer package to its own repo (#308) 2023-02-01 09:54:52 +05:30
prathamesh0
ff64dcb43c
Upgrade js-libp2p and related dependencies (#298)
* Upgrade js-libp2p and related dependencies

* Dial all multiaddr of a discovered peer in parallel

* Dial using peer id instead of dialling individual multiaddr

* Log peer id on discovery
2023-01-20 10:34:55 +05:30
prathamesh0
7069a7f455
Upgrade version to 0.2.24 (#299) 2023-01-19 15:48:38 +05:30
b07e288756
Ensure connection with peers after going offline (#297)
* Redial relay node on failure and after going offline

* Redial only relay node if not connected

* Refactor and rename methods

* Only close existing connection to relay node

* Hearbeat check only for relay node

* Refactor startHeartbeatCheck method
2023-01-19 15:01:40 +05:30
prathamesh0
9d38306fe9
Federated relay nodes and limiting connections (#296)
* Pass relay node an optional list of relay peers to connect to

* Catch protocol selection failure on dialling to new relay nodes

* Restrict max number of connections for a peer

* Tag the relay node with a high value to avoid disconnects

* Use debug for connect/disconnect logs in relay nodes

* Ignore incomplete multiaddr on a peer discovery

* Increase max connections for a peer to 10

* Refactor and descriptive comments
2023-01-19 11:35:09 +05:30
prathamesh0
1e5485c6ef
Use StrictSign signature policy in pubsub (#295)
* Log number of connected peers on connect/disconnect

* Log generated peer id if file path not provided

* Use StrictSign signature policy in pubsub
2023-01-17 13:54:24 +05:30
zramsay
ae97b3fc65
fixup main README, link to an SO stack where applicable 2023-01-16 07:40:40 -05:00
zramsay
67dd22e761
deduplicate the Customize section 2023-01-16 07:40:40 -05:00
zramsay
dd9f85f268
moar cli & complete address-watcher README 2023-01-16 07:40:39 -05:00
zramsay
85452a870f
combine demo & readme, graph-test-watcher; start docs/cli 2023-01-16 07:40:39 -05:00
zramsay
4ee889eaa7
deduplicate into a docs/README 2023-01-16 07:40:39 -05:00
zramsay
4080e2ded7
combine demo & readme, mobymask-watcher 2023-01-16 07:40:39 -05:00
zramsay
c47a435588
combine demo & readme, erc721-watcher 2023-01-16 07:40:38 -05:00
zramsay
c03041f09d
combine demo & readme, erc20-watcher 2023-01-16 07:40:38 -05:00
prathamesh0
38daa31029
Add a method to send messages on the network using pubsub (#293)
* Add a method to send messages using pubsub on a topic

* Update version to 0.2.22
2023-01-11 17:19:21 +05:30
55884a8ecd
Fix publish failing for react-peer package (#292)
* Remove empty repository from package.json

* Update version to 0.2.21
2023-01-10 22:11:40 +05:30
4f82dfbf74
Update version to 0.2.20 (#291) 2023-01-10 20:32:05 +05:30
cd29b47ecc
Implement peer package to send messages between peers (#279)
* Initial implementation of class and discoving other peers

* Implement peer connection and sending messages between peers

* Add react app and use peer package for broadcasting

* Maintain stream for each remote peer

* Refactor code in peer package

* Add serve package for react-app

* Add readme for running react app

* Add peer package readme

* Add logs for events with details

* Add a chat CLI using peer package (#280)

* Add a flag to instantiate Peer for nodejs

* Add a basic chat CLI using peer

* Add a signal server arg to chat CLI

* Add instructions for chat CLI

* Fix typescript and ESM issues after adding peer package (#282)

* Fix build issues in util package

* Update eslint TS plugins

* Scope react app package name

* Convert cli package back to CJS and dynamically import ESM

* Upgrade ts-node version

* Fix tests

* Setup a relay node and pubsub based discovery (#284)

* Add a script to setup a relay node

* Use pubsub based peer discovery

* Add peer multiaddr to connection log

* Catch relay node dial errors

* Increase discovery interval and dial all mutiaddr

* Add UI to display self peer ID and multiaddrs

* Add UI for displaying live remote connections

* Send js objects in peer broadcast messages

* Add react-peer package for using peer in react app

* Reduce disconnect frequency between peers (#287)

* Restrict number of max concurrent dials per peer

* Increase hop relay timeout to 1 day

* Self review changes

* Review changes

* Increase pubsub discovery interval and add steps to create a peer id  (#290)

* Increase pubsub discovery interval

* Disable autodial to avoid peer dials without protocol on a reconnect

* Add steps to create a peer id and use for relay node

* Add back dependency to run signalling server

* Avoid bootstrapping and dial to relay node directly

Co-authored-by: prathamesh0 <42446521+prathamesh0@users.noreply.github.com>
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2023-01-10 20:10:27 +05:30
aa4a954330
Use graph-ts and graph-cli packages from gitea (#283)
* Use published packages from gitea

* Remove token used for github packages
2022-12-29 16:59:42 +05:30
prathamesh0
a834250259
Update demo hardhat configs to use env variable for eth endpoint (#277)
* Use an env variable for eth RPC endpoint in hardhat config

* Add a task to get main account address in erc20 watcher
2022-12-21 17:59:31 +05:30
9566dff466
Fix job-runner to process missing parent block (#274)
* Fix job-runner to process missing parent block

* Upgrade package versions to 0.2.18
2022-12-13 11:59:55 +05:30
cc66f44bab
Use assemblyscript from gitea packages (#272)
* Use assemblyscript from gitea packages

* Add back github registry for vulcanize/graph-ts

* Use cerc-io scoped assemblyscript package
2022-12-07 15:13:41 +05:30
d0ac2aa895
Add script to set package versions in codegen (#270) 2022-11-29 14:52:39 +05:30
c5908c4c0e
Upgrade package versions to 0.2.17 (#269) 2022-11-28 19:10:36 +05:30
prathamesh0
847837c2fb
Add a demo for erc20-watcher (#268)
* Fix event subscription in erc20-watcher

* Add demo for erc20-watcher

* Update README
2022-11-28 18:54:09 +05:30
c299737aab
Add demo for graph-test-watcher (#267)
* Add demo for graph-test-watcher

* Remove stack-orchestrator from demos
2022-11-28 16:35:19 +05:30
05bfd57278
Add isPruned flag to erc721-watcher custom transferCount entity (#266)
* Remove block number param from job-queue reset CLI in readme

* Add isPruned flag to erc721-watcher custom transferCount entity
2022-11-28 14:14:02 +05:30
prathamesh0
bd8f003322
Gracefully shutdown server (#265)
* Gracefully shutdown server

* Forbid lint warnings

* Avoid setting subgraph path in template contex in codegen
2022-11-28 12:01:28 +05:30
prathamesh0
2417e3feb1
Fix linter warnings and dependencies (#264)
* Fix minor linting errors

* Fix CLI getters linting errors

* Fix resolvers template in codegen

* Fix dependencies
2022-11-25 20:03:58 +05:30
prathamesh0
94c8ed9575
Move event watcher to util (#262) 2022-11-25 17:19:37 +05:30
e47aab2ed7
Upgrade package versions to 0.2.16 (#263) 2022-11-25 16:09:35 +05:30
b66dcb4af9
Refactor graph-node database and move to util (#259)
* Move graph-database from graph-node to util

* Refactor and remove graph-node dependency from cli package

* Modify dependencies using depcheck

* Implement CLI refactoring changes in other watchers

* Review changes to remove eden comment and fix local import in util

* Import GraphDatabase from util instead of graph-node

* Move graph-node non assemblyscript code to util package

* Implement CLI refactoring changes in codegen

* Fix graph-node tests after refactoring

* Move fromStateEntityValues to graph state utils
2022-11-25 15:54:35 +05:30
prathamesh0
63a2c5804e
Accommodate CLI refactoring changes to codegen (#261)
* Use watch-contract CLI from cli package in codegen

* Use reset CLIs from cli package in codegen

* Use create checkpoint CLI from cli package in codegen

* Use inspect-cid CLI from cli package in codegen

* Use import state CLI from cli package in codegen

* Use export-state and verify-checkpoint CLI from cli package in codegen

* Use server CLI from cli package in codegen

* Use job-runner CLI from cli package in codegen

* Use index-block CLI from cli package in codegen

* Use fill CLI from cli package in codegen

* Minor codegen fixes

* Update watcher instructions to run job-runner before server
2022-11-25 11:31:20 +05:30
prathamesh0
590482ee53
Merge watcher job-runner into base job-runner (#260) 2022-11-24 16:26:40 +05:30
prathamesh0
aba0c665f3
Refactor fill and fill-state CLIs to cli package (#257)
* Refactor fill CLI to cli package

* Refactor method to fill-state to graph-node

* Refactor fill-state CLI to cli package

* Move subgraph state utils to a separate file

* Refactor subgraph state helper methods to graph-node

* Update mock indexer

* Move watcher job-runner to util

* Remove mock server and data from erc20-watcher

* Import watcher job-runner from util
2022-11-24 15:28:38 +05:30
prathamesh0
7717601408
Refactor index-block CLI to cli package (#258) 2022-11-23 20:02:24 +05:30
b08fcebe54
Upgrade package versions (#256) 2022-11-23 18:23:26 +05:30
prathamesh0
4bfb007a7e
Refactor job-runner CLI to cli package (#255)
* Fix eden-watcher server initialization

* Add an indexer method to watch subgraph contracts

* Refactor job-runner CLI to cli package

* Move watcher reset commands to refactored code
2022-11-23 18:20:16 +05:30
cc28474537
Accommodate GQL optimizations in codegen (#254)
* Prune optional methods in indexer and database interfaces

* Implement GQL optimization changes in codegen

* Fix graph-node test indexer

* Add demos to codegen package
2022-11-23 17:42:25 +05:30
prathamesh0
0b33cc98c9
Refactor server CLI to cli package (#253)
* Refactor server CLI to cli package

* Use server CLI from cli package in all watchers
2022-11-23 12:57:59 +05:30
prathamesh0
122a64c2f9
Refactor export-state and verify checkpoint CLIs to cli package (#252)
* Refactor export-state CLI to cli package

* Refactor verify checkpoint CLI to cli package

* Update mock indexer object
2022-11-22 18:27:49 +05:30
ace52d9da3
Add GQL metrics to ERC20 and ERC721 watchers (#251)
* Add changes in erc20-watcher

* Add changes in erc721-watcher

* Add changes in mobymask-watcher

* Add reset state CLI to all watchers
2022-11-22 17:04:38 +05:30
prathamesh0
6737ec756c
Refactor import-state CLI to cli package (#250)
* Remove unnecessary upstream config arg to event watcher

* Initialize event watcher in CLI package

* Refactor import-state CLI to cli package
2022-11-22 16:41:15 +05:30
6622d0874e
Accommodate GQL optimizations in graph-test-watcher (#249)
* Use separate subgraph entities list

* Implement changes in graph-test-watcher

* Reset latest entity tables and use cache directives in schema GQL
2022-11-22 15:20:44 +05:30
prathamesh0
570640d4bc
Refactor inspect-cid CLI to cli package (#248) 2022-11-22 11:45:45 +05:30
prathamesh0
03050e5323
Refactor create checkpoint CLI to cli package (#247)
* Refactor create checkpoint CLI to cli package

* Update mock indexer
2022-11-22 10:38:23 +05:30
06e5c62fa8
Remove saveBlockAndFetchEvents method callback pattern (#244)
* Remove saveBlockAndFetchEvents method callback pattern

* Accomodate change in code generator

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-11-21 15:54:28 +04:00
prathamesh0
1e639c1af3
Refactor reset CLIs to cli package (#246)
* Refactor reset CLIs to cli package

* Use reset CLIs from cli package in watchers

* Fix method to canonicalize latest entities

* Use composition with common code for refactored CLIs
2022-11-21 15:14:10 +04:00
prathamesh0
7520e9012c
Refactor watch-contract CLI to cli package (#245)
* Refactor watch-contract CLI to cli package

* Use watch-contract from cli in eden and erc721 watchers

* Use watch-contract from cli in all watchers

* Pass an object containing clients to refactored CLI

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-11-21 11:47:23 +04:00
a0ba657d17
Rename variables and fix comments (#243)
* Rename entity to entityType

* Rename variables and fix comments
2022-11-18 19:16:42 +05:30
662c79a5e7
Refactor event-watcher and move code to util (#242) 2022-11-18 17:23:33 +05:30
prathamesh0
f3c65cbd64
Refactor util code to be reused (#241)
* Ignore watch contract jobs in event processing complete handler

* Update job-queue config and handle errors on job completion hook

* Update graph decimal implementation

* Return generic type from method to read watcher config

* Export fill prefetch batch size default value
2022-11-18 16:29:06 +05:30
cc8fcffaa1
Fix use of prefetchBlocksInMem flag in block processing (#240)
* Fix use of prefetchBlocksInMem flag in block processing

* Rename prefetchedBlocksMap to blockAndEventsMap
2022-11-18 11:01:09 +05:30
prathamesh0
92fd3cac03
Upgrade client libraries (#239) 2022-11-17 19:06:07 +05:30
635b26aa0c
Use entity is_pruned flag in queries (#238) 2022-11-17 13:55:53 +05:30
prathamesh0
f53371e17b
Accomodate GQL requests caching in code generator (#237)
* Accomodate GQL requests caching in code generator

* Add GQL API request queuing
2022-11-17 12:02:08 +05:30
79e903b396
Reset latest and frothy entity tables on watcher reset (#236) 2022-11-17 10:14:59 +05:30
693c23d192
Update latest entity tables on chain pruning (#235) 2022-11-16 20:16:48 +05:30
prathamesh0
1ad223db4d
Add GQL requests caching in eden-watcher (#234) 2022-11-16 20:01:18 +05:30
74741184ee
Use lateral query for time travel queries using latest entity tables (#233)
* Avoid joins in id only queries

* Use lateral query for time travel queries using latest entity tables
2022-11-16 18:57:04 +05:30
a084b4e40c
Latest entity queries to optimize frontend app GQL requests (#232)
* Add latest entity table query pattern

* Add subscriber config to typeorm
2022-11-16 18:11:27 +05:30
prathamesh0
408a3927c0
Add a table for entites in frothy region for subgraph watchers (#231)
* Add a table for entites in frothy region and update it in a subscriber

* Accommodate changes to other watchers and codegen
2022-11-16 17:12:54 +05:30
7e5974ccf7
Denormalize is_pruned flag in eden-watcher (#230)
* Add is_pruned flag to all entities

* Mark entities as pruned once the corresponding block gets pruned
2022-11-16 15:30:40 +05:30
prathamesh0
62c57d8005
Upgrade Apollo server to v3 (#229)
* Upgrade apollo server to v3 in eden-watcher

* Upgrade apollo server to v3 in other watchers and codegen

* Refactor common server code to util

* Pass express application object in the method to create server

* Remove unnecessary import
2022-11-16 11:14:54 +05:30
prathamesh0
a52bdf64b1
Add config for GQL requests cache (#228)
* Add config and a method to set cache control settings for GQL requests

* Add option for max size in GQL cache config

* Fix failing tests
2022-11-15 14:56:08 +05:30
6f8ededd52
Add flag to load relations sequentially or concurrently (#225)
* Add flag to load relations sequentially or concurrently

* Fix database init in graph-node test

* Fix graph-node watcher query method and add changes to codegen
2022-11-14 14:23:46 +05:30
prathamesh0
13edff143b
Update cerc-io packages version in watcher generation (#224) 2022-11-11 15:00:48 +05:30
prathamesh0
861420a10f
Add a flag to enable state creation (#222)
* Add a flag to enable state creation

* Update flag in mock config for testing
2022-11-10 16:56:06 +05:30
prathamesh0
8d3c68873b
Avoid creating a separate base db instance for graph database (#221)
* Avoid creating a separate base db instance for graph database

* Fix failing test file build

* Use getter method to get base database
2022-11-10 14:51:18 +05:30
ecd8b2474a
Remove eventName filter in events count query (#220) 2022-11-10 12:09:17 +05:30
prathamesh0
3862f9ce2e
Use FindManyOptions to get entities (#219) 2022-11-09 14:01:39 +05:30
772fdbec25
Add unique entities query without checking pruned block (#218) 2022-11-09 13:31:26 +05:30
7a4b459b1c
Add DB queries without checking pruned block (#217) 2022-11-08 17:01:35 +05:30
Ashwin Phatak
99c6823564
GQL API request queue params (#216) 2022-11-04 12:01:07 +05:30
0e464385c0
Remove contracts on reset watcher (#215) 2022-11-04 10:41:48 +05:30
prathamesh0
266442a6d9
Update getFullBlocks usage (#213)
* Update getFullBlocks usage to add block number arg

* Add skipStateFieldsUpdate config for uniswap-watcher

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-11-03 16:49:41 +05:30
be65ca45ea
Load relations sequentially (#212) 2022-11-03 14:32:34 +05:30
prathamesh0
2517f110ea
Update GQL and getBlock API usage (#210)
* Update getLogs and getBlockWithTransactions usage to add block number arg

* Update getFullTransaction usage to add block number arg
2022-11-03 14:14:58 +05:30
52c42f4e84
Reset watcher to previous indexed block on start (#207)
* Reset watcher to previous indexed block before start

* Implement changes in other watchers

* Save successfully fetched blocks and events to prefetch cache

* Add unique query for transaction table

* Check db for blocks before fetching from eth-server

* Show all mismatches at a block

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-11-03 14:01:10 +05:30
prathamesh0
306bbb73ca
Use prefetching of blocks with events in watchers and codegen (#206)
* Avoid refetching block while fetching events

* Prefetch a batch of blocks with events while indexing

* Update mock indexer used in graph-node testing

* Process available blocks while prefetching

* Refactor events fetching to a method in util

* Move method to get GQL event query result to util
2022-10-20 18:46:56 +05:30
668875b3a0
Refactor code in graph-node to use in uniswap watcher (#205)
* Refactor code in graph-node to use in uniswap

* Move over database methods to fetch entities for GQL query
2022-10-20 14:39:32 +05:30
prathamesh0
5af90bd388
Refactor state creation code (#204)
* Remove support for pushing state to IPFS

* Move job handlers for state creation to util

* Rename state creation related methods and objects

* Update mock indexer used in graph-node testing

* Fetch and merge diffs in batches while creating a state checkpoint

* Fix timing logs while for state checkpoint creation

* Refactor method to get state query result to util

* Accept contracts for state verification in compare CLI config

* Make method to update state status map synchronous
2022-10-19 15:24:14 +05:30
ce182bce85
Clear cache of latest entities on event processing error (#201)
* Clear cache of latest entities on event processing error

* Remove lighthouse-watcher and update ethersjs version

* Handle GraphDecimal type in state entity

* Add option for comparing all entities using paginate

* Clear pruned cached entities at intervals

* Move ipld-demo to graph-node package and remove reset-dbs script

* Implement changes in all watchers and codegen
2022-10-19 14:26:10 +05:30
prathamesh0
74747ce49a
Add scripts for ERC721 token interaction (#203) 2022-10-17 15:38:59 +05:30
prathamesh0
45111e68ac
Add an option to run compare CLI in batched intervals (#202) 2022-10-17 12:07:21 +05:30
prathamesh0
bad5ec754a
Prefetch blocks with events in memory (#199)
* Prefetch blocks along with events in memory

* Fix prefetched block destructuring

* Wait while blocks are being prefetched

* Retry only if block not returned by upstream eth-server

* Log time taken to prefetch blocks

* Avoid creating a separate job for prefetching blocks

* Skip syncStatus check while fetching blocks

* Convert block number string to number for comparison

* Prefetch blocks in a batch only when required

* Keep existing pattern for watchers in this repo

* Fix logging while fetching a batch of blocks
2022-10-13 17:07:46 +05:30
521f6ed3fa
v0.2.13 (#197) 2022-10-11 16:12:06 +05:30
e1aef1a7e9
Changes to use packages in uniswap-watcher (#196)
* Changes to use util from uniswap-watcher

* Refactor ResultIPLDBlock to util

* Verify state in compare CLI for uniswap multiple queries

* Prepare watcher-ts for publishing packages

* Fix verify state in compare CLI for multiple entities query

* Fix codegen util imports
2022-10-11 13:41:26 +05:30
prathamesh0
978f0bb456
Update getLogs API usage (#195)
* Update client query to get logs

* Update calls to get logs for contracts
2022-10-05 15:15:00 +05:30
18861eaf79
Implement cache for latest updated entities to be used in mapping code (#194)
* Load relations according to GQL query

* Implement cache for latest entities to used in mapping code

* Add metrics for cache hit and fix caching pruned entities

* Changes in codegen and graph-test-watcher

* Remove entity load counter reset to zero
2022-10-04 13:31:29 +05:30
6149690126
Update ipld-status table in reset and fill state CLI (#193) 2022-10-04 11:22:04 +05:30
00a9c247f5
Compare CLI enhancements for verifying uniswap-watcher entities (#188)
* Update compare CLI to verify only updated entities

* Show time difference between GQL requests
2022-10-04 11:18:55 +05:30
87224a4673
Implement eden-watcher changes in other watchers and codegen (#192)
* Implement eden-watcher changes in other watchers and codegen

* Use node space size only for eden-watcher
2022-09-23 15:35:15 +05:30
a8fdcca866
Update watcher import CLI to create entities from checkpoint (#186)
* Import entities from checkpoint

* Fix IPLD state when updating subgraph entity

* Changes in codegen and other watchers

* Update IPLD state with all Block entities

* Add verify and create sub commands to checkpoint

* Add option for specifying snapshot block in export state CLI
2022-09-22 15:26:06 +05:30
prathamesh0
b2cf997900
Update CLI to fill state to create checkpoints (#187)
* Update CLI to fill state to create checkpoints

* Accomodate changes in codegen
2022-09-16 09:23:41 +05:30
prathamesh0
c2bbaa6867
Update meta data verification to allow branched checkpoints (#183) 2022-09-16 09:15:11 +05:30