* Import entities from checkpoint
* Fix IPLD state when updating subgraph entity
* Changes in codegen and other watchers
* Update IPLD state with all Block entities
* Add verify and create sub commands to checkpoint
* Add option for specifying snapshot block in export state CLI
* Check endBlock greater than chainHeadBlock in fill blocks CLI
* Load relations for single entity with limit 100
* Use single tx query runner for entity GQL query
* Remove assert for CLI input checks
* Add a CLI to fill state for a given range
* Refactor code
* Add a CLI to reset IPLD state
* Replace ORDER BY clause in the query to get latest IPLD block
* Optimize delete query in CLI to reset IPLD state
* Add an option to decouple subgraph state creation from mapping code
* Use a raw SQL query to delete IPLD blocks in a block range
* Accomodate changes in codegen
* Prefetch block size from eth_blockByHash
* Fix updating of blockSizeMapLatestHeight
* Move block size caching to separate file
* Remove timer logs from graph-node store set and get
* Implement graceful shutdown in job-runner
* Improve reset CLI delete query
* Exit job-runner immediately in development
* Implement changes in other watchers
* Update fill prefetch CLI to accept env variables
* Clean up assets before building util package
* Accomodate change in other watchers and codegen
* Add logs for debugging failing test in CI
* Implement query for multiple entities in eden-watcher
* Implement nested relation queries
* Implement GQL query params first, skip, orderBy, orderDirection
* Add blockNumber index to subgraph entities
* Add logs for timing eth-calls and storage calls
* Add prometheus metrics to monitor GQL queries
* Fix default limit and order of 1-N related field in GQL entitiy query
* Add timer logs for block processing
* Run transpiled js in all watchers
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Compare IPLD state entity without derived fields
* Apply default limit to array relation fields in IPLD state entity
* Mark block as complete after processing of block handler
* Avoid re processing of block handler
* Use LIMIT 1 in the query to get latest IPLD block
* Replace eth_calls in eden-watcher with getStorageValue
* Add checkpoint verification to export state CLI
* Fix get diff blocks query when creating checkpoint
* Fix subgraph staker sort and remove entities sequentially in reset CLI
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Add test package
* Add a separate command to perform an eth-call
* Add snapshot test suite
* Add eth-calls for UniswapV2 Pair, USDC, Compound, Dai contracts
* Add args for Uniswap and USDC contracts
* Add args for Compound and Dai contracts
* Add getStorageAt calls to the test suite
* Refactor code and add documentation
* Loop over test slots for getStorageAt calls
* Add support for individual calls
* Use debug for logging
* Change compare CLI to verify only updated entities
* Implement IPLD state verification in compare CLI
* Changes to IPLD state to match with GQL result entity
* Implement storage call in subgraph mapping code
* Add test for mapping type variable storage call
* Use vulcanize graph-ts
* Revert to graph-ts version 0.22.1
* Make sumStaked variable local in eden network mapping
* Implement compare CLI to fetch and query by ids
* Set filterLogs to true for eden-watcher
* Use varchar for bigint array type in eden-watcher
* Store subgraph entities by id in IPLD state
* Store bigint vales as string in IPLD state
* Update eden watcher hook to store single Block entity in IPLD checkpoint
* Fix entity enum type property
* Fix parsing big numbers in event params
* Fix event bigint params parsing in all watchers
* Set default limit to query result and process block after events
* Add prometheus metrics endpoint in watcher
* Add event, sync status and DB size metrics
* Fix subgraph watchers DB entities directory path
* Make watcher metrics optional using config
* Update codegen with index-block CLI and remove graph-node
* Add filter logs by contract flag
* Skip generating GQL API for immutable variables
* Add config for maxEventsBlockRange
* Add new flags in existing watchers
* Remove using graph-node from mobymask-watcher
* Parse batched transactions for MobyMask events
* Update to use same ethers package version
* Add CLI for processing block out of order
* Fix job-runner for already processed blocks out of order
* Use stack-orchestrator for erc721-watcher demo
* Run all services in stack-orchestrator with auto mining of blocks
* Add step to check docker-compose version
* Update demo for IPLD blocks
* Add update for custom properties in state diff
* Implement custom update of transfer counter in state diff
* Comment code for updating custom state prop
* Separate markdown for erc721-watcher Demo
* Run workflow on main branch commit
* Implement data source templates
* Remove optional param from wasm instantiate
* Add all types for solidity to typescript mapping
* Set contract address in context for data source templates
* Implement block handlers for data source templates
* Implement ethereum ABI encode in subgraph
* Implement ethereum ABI decoding host API
* Implement ABI encode decode for array type
* Implement ABI encode decode for bytes type
* Parse events for multiple contracts in the generated code
* Use contract wise artifacts in the generated indexer methods
* Update codegen docs to use config file to generate a watcher
* Add watcher generation config to eden-watcher
* Test case for wasm out of memory error
* Restart wasm instance after N blocks
* Handle out of memory error and re instantiate WASM
* Remove old instance from map before reinstantiating WASM
* Avoid fetching contracts while creating a checkpoint
* Use enum for state kind and cache ipld status
* Avoid fetching block twice while finalizing a staged diff IPLDBlock
* Create checkpoints at fixed block numbers and refactor checkpointing code
* Avoid calling block handler until start block is reached
* Use delete while finalizing staged diff IPLDBlocks
* Add a check to ensure hooks job is created only once
* Avoid check for initial state while creating a checkpoint
* Process hooks for the first block in the watchers
* Store initial state from a hook in an IPLDBlock for eden-watcher
* Extract watcher method to prepare IPLDBlocks in util
* Extract common IPLDBlocks related code in util
* Move IPFSClient to util
* Use constants for state kind
* Use typeof to distinguish between BigInt and BigDecimal in store get API
* Use entity column type map to create entity in store get API
* Add entity column type map in eden-watcher
* Add test cases for bigInt host APIs
* Add test cases for bigDecimal fromString, plus and minus host APIs
* Add test cases for bigDecimal times and divideBy host APIs
* Throw an error on underflow or overflow in bigDecimal toString host API
* Avoid rounding of out of range bigDecimal values
* Use big number's byte size to set storage size of bigInt
* Add test cases for errors in bigDecimal host API
* Implement host api for bigint operations
* Implement type conversion APIs
* Avoid use of exports as variable name for conflict
* Implement log API with debug and levels
* Fix job-runnner hook error after stopping and starting chain
* Handle BigNumber event params in watchers
* Customize decimal according to limits of IEEE-754 decimal128
* Add definition for custom scalar BigDecimal
* Handle subgraph schema field with derivedFrom directive
* Handle derivedFrom directive in eden-watcher
* Fix 1 to N relation error by removing limit from query
* Order by id for derivedFrom relations to match graph-node
* Refactor example subgraph schema entities
* Fix watcher queries to return correct relation field values
* Fix hierarchical query for getting two entities at same block
* Handle negative bigInt in store host API
* Increase size of bigInt in fromString API to support UInt256
* Implement bigDecimal divideBy host API
* Implement bigDecimal plus, minus and times host API
* Implement bigInt dividedByDecimal and mod host API
* Change BN js version to that used by ethers js
* Create checkpoint using CLI after the hooks have been processed
* Push checkpoints created while exporting state to IPFS
* Clear subgraph entities on reset state
* Remove unnecessary staged IPLD blocks created while importing state
* Add CLI to compare entities from two GQL endpoints
* Print out result diffs in compare-entity CLI
* Get the colorized result diff in compare-entity CLI
* Read query dir from config file or as an arg
* Make config file arg required
* Make queries in Example schema similar to that in graph-node
* Get non-colorized output on choosing raw-json diff option
* Make queries in eden-watcher similar to that in graph-node
* Handle tuple return type in ethereum host API
* Update graph-cli version to fix eth_call error
* Handle all types in struct based on abi
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Add implementation for one to one relation
* Implement one to many relation in gql queries
* Make changes for gql relation queries in eden-watcher
* Implement subgraph gql relation queries with joins
* Handle relation entities in subgraph
* Modify eden-watcher entities to handle subgraph schema data types
* Modify schema gql to match eden subgraph gql shape
* Handle array type fields in subgraph schema
* Fix store get api for array type fields in subgraph entities
* Handle array type in eden-watcher and format await used in params
* Rename Block, Transaction types and add block query in eden-watcher schema
* Handle field name conflicts in eden subgraph entities
* Resolve entity field name conflicts while sending data for auto-diff
* Add a watcher for EdenNetwork contract
* Add events from MerkleDistributor contract to eden-watcher
* Add ERC721 Transfer and Approval events to eden-watcher schema
* Add artifacts for DistributorGovernance and MerkleDistributor contracts
* Fix packages build
* Add support for boolean type in subgraph schema
* Add BigDecimal fromString and toString implementation
* Add implementation for subgraph Bytes type
* Implement enum type for subgraph schema
* Add implementation for subgraph schema BigDecimal type
* JSON stringify and parse entity data for auto-diff
* Add a test case to eden test to call the block handler
* Add a block handler in example subgraph and call it in a watcher
* Use an array map to call all the block handlers for a contract
* Await on all the promises returned by block handlers map
* Add subgraph schema types to the generated schema
* Add queries for subgraph entities
* Add entity generation for subgraph entities
* Call subgraph event handler in indexer
* Refactor subgraph schema and entity generation
* Add resolvers generation for subgraph entities
* Get event signature in the event
* Add NonNullType check for field type in entity generation
* Auto-diff based on store set
* Use contract address from data source in loader
* Change subgraph-schema arg to subgraph-path arg
* Implement store get api without blockHash and blockNumber
* Pass database instance to GraphWatcher
* Implement store set without block data
* Store blockHash and blockNumber in database entity table
* Implement getting entity in subgraph from store.get
* Add block data present in postgraphile
* Pass db and context to instantiate method in tests
* GQL API in graph-test-watcher to test store.set
* Remove contract address from subgraph file
* Fix block in dummy event data
* Pass just blockHash to get an entity from the database
* Review changes and add TODOs
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
* Invoke subgraph handler in watcher event processing
* Fix error when invoking subgraph handler
* Parse events using event signature specified in subgraph yaml
* Use contract abi to parse event params
* Invoke event handler based on event signature
* Fill event with block and transaction data
* Comment missing fields in block and transaction data
* Invoke handlers based on watcher-ts events
* Read subgraph yaml and listen to events from watcher-ts
* Create GraphWatcher class to use in generated example contract watcher
* Call graph-node event handler from generated watcher
* Test case for calling EdenNetwork event handler
* Test cases for calling EdenNetwork event handlers
* Test cases for calling event handlers in EdenNetworkDistribution
* Test cases for calling event handlers in EdenNetworkGovernance
* Perform eth-calls according to contract name in param
* Pass function params in eth-call host API
* Return value from eth-call to wasm
* Add instructions for running tests
* Install assemblyscript from github packages
* Add instrcutions for installing private github packages
* Use vulcanize assemblyscript package
* Perform eth_call and get result in js import
* Use dummy value for eth_call
* Called handler in wasm with event param
* Implement passing event to subgraph handler function
* Use generated event class Test for passing to handler
* Pass event params to handler
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Implement host api typeConversion.bytesToHex
* Complete host api bigIntToString.
* Create TypeId for assemblyscript loader.
* Implement host apis bigInt fromString, plus and minus
* Code for running wasm file.
* Add test for wasm exported function.
* Use target names for build files.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Add export-watcher CLI generation
* Export snapshot block and add import-watcher CLI gen
* Update codegen docs with export and import commands
* Use different function to get latest checkpoint in export CLI
* Update generated watcher docs with import state instructions
* Push IPLDBlocks to IPFS
* Change ts type for data column to Buffer
* Add checkpoint hook
* Push CLI checkpoint to IPFS
* Fix latest IPLDBlock query to filter out irrelevant blocks
* Add CLI to inspect a CID
* Decode data before exporting
* Encode data in JSON for getState GQL API
* Use go-ipfs and export cbor-encoded buffer
* Remove empty post-block methods from existing watchers
* Call hooks and checkpoint job complete handlers in fill
* Remove subscription to hooks queue from existing watchers
* Remove checkpoint option for state param. to indexer methods
* Change state param. to indexer methods to boolean
* Use hierarchical query to get the last checkpoint of a contract
* Filter out blocks before prev checkpoint for a new checkpoint
* Add checkpointing job on completion of chain pruning
* Update codegen docs with reset commands
* Create initial checkpoint in the checkpoint queue
* Handle initial checkpoint along with other checkpoints
* Create initial checkpoint job in indexer
* Create IPLDBlocks only in the pruned region
* Create staged diff IPLDBlock for default state
* Generate default derived state for Mapping type variables
* Update IPLDBlock in storage methods instead of using a private class variable
* Default state flag for indexer storage methods
* Helper functions to update state objects
* Add checkpoint flag in Contract table and corresponding changes in existing watchers
* Update codegen docs
* Add examples to generated docs
* Turn default state off by default
* Make state parameter to indexer storage methods default to none
* Add method to get prev. state in indexer
* Subscribe to hooks queue in existing watchers
* Change naming strategy for generated get and save functions
* Push checkpointing job after post-block hook job completed
* Using hooks status to ensure their chronological execution
* Add default indices to IPLDBlock table
* Add kind parameter to getState GQL API
* Add checkpoint CLI
* Add blockHash arg to checkpoint CLI and update codegen docs
* Print out block hash for checkpoint CLI
* Use log from debug for logging
* Filter using contract at start in hierarchical query
* Make kind argument to prepare IPLDBlock required
* Add ipld-blocks entity generation
* Populate ipld-blocks table
* Rename ipld-block entity and update after each event
* Move ipld-hook to hooks.ts
* Change IPLD block structure
* Add cid field in blocks
* Fetch prev. IPLDBlock for a contract
* GQL API to query IPLDBlock by CID
* Save cid in blocks in existing watchers
* Update codegen docs
* GQL API for getting last derived state (#3)
* GQL API for getting last derived state
* Rename query to getState
* Change query names to getState and getStateByCid
* Save BigInt as string
* Move function to prepare IPLDBlock to indexer
* Refactor IPLDBlock hook
* Add genesis hook
* Call post-block hook after a block is marked as complete
* Add IPLDBlock checkpointing
* Use queryRunner instead of a new repo for queries
* Add a query to get block in ipld-eth-client
* Get latest checkpoints for all contracts for checkpointing.
* Call post-block hook in a queue
* Pass server config to Indexer in watch-contract cli
Co-authored-by: nikugogoi <nabarun@deepstacksoft.com>
* Fix fill cli to work with watcher server
* Add createdAt column and insert events in batches
* Implement prefetch to fill block and events in parallel
* Fix getPrevEntity and increase fill prefetch default batch size
* Fix watcher creating mulitple jobs for a block
* Improvement TODOs
* Get parent and current block in single query
* Remove block progress queries and events query
* Reduce queries in fetching batch events by using query builder
* Implement changes in codegen package
* Avoid database query by passing event directly to job-queue
* Avoid block progress query by returning from update query
* Process batch of events for a block in a single job
* Fix smoke test for subscribed events and use teamSize for job queue
* Perf improvement TODOs
* Move watch contract to util package
* Get block progress from event instead of querying
* Use index field in event to check order of processing
* Use watched contract map to avoid querying database
* Use update query for blockProgress entity
Co-authored-by: Ashwin Phatak <ashwinpphatak@gmail.com>
* Add a GraphDecimal class to replace Decimal with
* Replace Decimal in uni-info-watcher with GraphDecimal
* Add definition for custom scalar BigDecimal in uni-info-watcher
* Use pg-type numeric and transformer for bigint columns in typeorm entities
* Implement pull based watcher for uni-watcher
* Fix same block processed multiple times
* Implement wait time for fetching block from config
* Use blockProgress event to fetch and process next block
* Rename utils index to misc
* Implement clean jobs CLI in watchers
* Pull common code cleanJobs to util
* Implement commands for reset job-queue and state
* Reset command for job-queues and watcher state
* Reset sync status on reset state
* Fix using cli from built js
* Implement skipCheck to reset syncStatus table and skip complete handler on fail
* Check for block isComplete on reset state
* Set default value for force param to update syncStatus to false
* Reset tables in erc20 watcher
* Push job in fill after previos block is complete
* Add GQL client generation
* Add subscription, events, eventsInRange in client gen
* Add mutation in client gen and return event tx info from indexer
* Capitalize class names for storage-mode entities
* Implement storage mode in code generator.
* Fix indexer template for eth_call.
* Add support for both eth_call and storage modes.
* Add eventsInRange, events file gen and default entites gen.
* Use static property to column maps in database.
* Process events query.
* Avoid adding duplicate events in indexer.
* Fix generated watcher query with bigint arguments.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
* Remove fields no longer present in schema.
* Refactor code to work with log_cids change in eth-server.
* Refactor process event to save events from logs.
* Use constants for erc20 event names.
* Implement watch blocks in erc20-watcher similar to uni-watcher.
* Moved common methods to util.
* Implement eventsInRange query in erc20-watcher.
* Filter unknown event in database query.
* Change dependencies version to be same in all packages.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Add schema generation.
* Use yargs and add a script to generate schema.
* Add command to generate schema to README.md
* Remove use of arrow functions in visitor.
* Add function to export schema in visitor.
* Create Event union while adding to it.
* Add bool to typemappings.
* Add method descriptions and remove output type mappings.
* Add storage mode.
* Add URL support and a flattening script.
* Fix parameter for flattening in README.md.
* Add setup and references to README.md.
Co-authored-by: prathamesh <prathamesh.musale0@gmail.com>
* Transpile ES5 code.
* Use transpiled ES5 code in all packages.
* Refactor tracing-client to transpile ES5 code.
* Order scripts in package json.
* Update readme in packages with new scripts.
* Refactor code to use entity config programatically.
* Remove unused depencdencies.
* Use DEBUG environment for scripts.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Refactor db methods getPrevEntityVersion and getFrothyRegion.
* Filter out entities from pruned blocks.
* Pull saveContract and getModelEntities to util.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Refactor common code in uni-watcher and uni-info-watcher.
* Implement client.ts in uni-info-watcher for queries and use in smoke-test.
* Disable cache in graphql client.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Implement hierarchical query for checking ancestor block.
* Prune blocks together in single query.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Create dummy test blocks and refactor test code.
* Intial test for pruning block in a chain without branches.
* Add test for pruning with multiple branches.
* Add test for pruning at frothy region.
* Test for pruning block at height more than max reorg depth.
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Implement block filler for uni-info-watcher.
* Pull common code for Block filler to util.
* Watch for job events in block filler command.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Implement heirarchical query for getting previous entity.
* Check entity id in hierarchical query to limit number of recursions.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Insert and remove a dummy BlockProgress from db.
* Test to fetch a Token entity from pruned region.
* Check if db is empty initially.
* Test to fetch a Token entity from frothy region.
* Test to fetch a Token from frothy region (same block num).
* Test to fetch a Token entity from another branch.
* Tests to fetch a Token entity (multiple tokens).
* Test to fetch updated Token entity in pruned region.
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Implement watcher for Lighthouse StorageRequest event.
* Add smoke test for lighthouse watcher.
* Add fileCost as value to Lighthouse store call.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Add resolver for Position entity in uni-info-watcher.
* Test for entities after TransferEvent.
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Run db operations inside event handlers atomically using transaction.
* Implement database transaction for Pool Initialize event.
* Implement typeorm transaction without callback.
* Implement transaction for NFPM event handlers.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Test for Token & Factory entities after MintEvent.
* Test for Pool, Mint & Tick entities after MintEvent.
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Smoke test for uni-info-watcher and refactor token deployment code.
* Test for Token entity after PoolCreated event.
* Test for Pool entity after PoolCreated event.
* Tests for entities after InitializeEvent.
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Implement transaction from to set entity origin field.
* Add all transaction fields specified in schema and add delay in uni-info block processing.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Created a smoke test for uni-watcher.
* Added test for createPool.
* Added more pool tests.
* Using ethers instead of hardhat-ethers in smoke test.
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Use getEntities to get result for queries.
* Implement resolver for Mint entity.
* Implement resolvers for Pool and PoolDayData entities.
* Add query resolver for Swap entity.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Implement last processed event in uni-info-watcher.
* Check if block or event is already processed.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Implement eth_call for ERC20 totalSupply.
* Use eth_call for erc20-watcher.
* Implement fallback for ERC20 symbol method call.
* Implement fallback for ERC20 name and totalSupply.
* implement fallback for erc20 decimals method.
* Lint fixes.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Watch all blocks and fetch events for each block.
* Implement getPrevVersionEntity in database.
* Implement getPrevVersionEntity for PoolCreated event.
* Implement getPrevEntityVersion for Pool events.
* Implement getPrevEntityVersion for NFPM events.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Split get and set entity for PoolCreated and Initialize events.
* Split get/set entity for Mint, Burn and Swap events.
* Split get/set entity for IncreaseLiquidity and DecreaseLiquidity events.
* Split get/set for NFPM Transfer and Collect events.
* Add blockHash to entities.
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Handle reorgs upto known canonical block hash.
* Process block events strictly in order.
* Support filling old blocks, process in-order.
* Always publish block progress.
