mirror of
https://github.com/cerc-io/watcher-ts
synced 2026-09-08 00:44:06 +00:00
Clear cache of latest entities on event processing error (#201)
* Clear cache of latest entities on event processing error * Remove lighthouse-watcher and update ethersjs version * Handle GraphDecimal type in state entity * Add option for comparing all entities using paginate * Clear pruned cached entities at intervals * Move ipld-demo to graph-node package and remove reset-dbs script * Implement changes in all watchers and codegen
This commit is contained in:
@@ -0,0 +1,208 @@
|
||||
# Demo for IPLD statediff and checkpointing
|
||||
|
||||
* In the root of `watcher-ts`, run:
|
||||
|
||||
```bash
|
||||
yarn && yarn build
|
||||
```
|
||||
|
||||
* In console, run the IPFS daemon:
|
||||
|
||||
```bash
|
||||
# Verify ipfs version
|
||||
ipfs version
|
||||
# ipfs version 0.12.2
|
||||
|
||||
ipfs daemon
|
||||
```
|
||||
|
||||
* The following services should be running to work with watcher:
|
||||
|
||||
* [vulcanize/go-ethereum](https://github.com/vulcanize/go-ethereum) ([v1.10.17-statediff-3.2.0](https://github.com/vulcanize/go-ethereum/releases/tag/v1.10.17-statediff-3.2.0)) on port 8545.
|
||||
* [vulcanize/ipld-eth-server](https://github.com/vulcanize/ipld-eth-server) ([v3.0.0](https://github.com/vulcanize/ipld-eth-server/releases/tag/v3.0.0)) with native GQL API enabled on port 8082 and RPC API enabled on port 8081.
|
||||
|
||||
* Deploy `Example` contract:
|
||||
|
||||
```bash
|
||||
cd packages/graph-node
|
||||
|
||||
yarn example:deploy
|
||||
```
|
||||
|
||||
* Set the returned address to the variable `$EXAMPLE_ADDRESS`:
|
||||
|
||||
```bash
|
||||
EXAMPLE_ADDRESS=
|
||||
```
|
||||
|
||||
* In `packages/graph-node`, run:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
* In `.env` file, set `EXAMPLE_CONTRACT_ADDRESS` to the `EXAMPLE_ADDRESS`.
|
||||
|
||||
* In [packages/graph-node/test/subgraph/example1/subgraph.yaml](./packages/graph-node/test/subgraph/example1/subgraph.yaml), set the source address for `Example1` datasource to the `EXAMPLE_ADDRESS`.
|
||||
|
||||
```bash
|
||||
yarn build:example
|
||||
```
|
||||
|
||||
* In `packages/codegen`, create a `config.yaml` file with the following contents:
|
||||
|
||||
```yaml
|
||||
contracts:
|
||||
- name: Example
|
||||
path: ../graph-node/test/contracts/Example.sol
|
||||
kind: Example1
|
||||
|
||||
outputFolder: ../demo-example-watcher
|
||||
mode: all
|
||||
kind: active
|
||||
port: 3008
|
||||
flatten: true
|
||||
subgraphPath: ../graph-node/test/subgraph/example1/build
|
||||
```
|
||||
|
||||
Reference: [packages/codegen/README.md](./packages/codegen/README.md#run)
|
||||
|
||||
* Generate watcher:
|
||||
|
||||
```bash
|
||||
cd packages/codegen
|
||||
|
||||
yarn codegen --config-file ./config.yaml
|
||||
```
|
||||
|
||||
* In `packages/demo-example-watcher`, run:
|
||||
|
||||
```bash
|
||||
yarn
|
||||
```
|
||||
|
||||
* Create dbs:
|
||||
|
||||
```bash
|
||||
sudo su - postgres
|
||||
# Delete databases if they already exist.
|
||||
dropdb demo-example-watcher
|
||||
dropdb demo-example-watcher-job-queue
|
||||
|
||||
# Create databases
|
||||
createdb demo-example-watcher
|
||||
createdb demo-example-watcher-job-queue
|
||||
```
|
||||
|
||||
Enable the `pgcrypto` extension.
|
||||
```
|
||||
psql -U postgres -h localhost demo-example-watcher-job-queue
|
||||
|
||||
demo-example-watcher-job-queue=# CREATE EXTENSION pgcrypto;
|
||||
demo-example-watcher-job-queue=# exit
|
||||
```
|
||||
|
||||
* In a new terminal, in `packages/demo-example-watcher`, run:
|
||||
|
||||
```bash
|
||||
yarn server
|
||||
```
|
||||
|
||||
```bash
|
||||
yarn job-runner
|
||||
```
|
||||
|
||||
* Run the following GQL subscription at the [graphql endpoint](http://127.0.0.1:3008/graphql):
|
||||
|
||||
```graphql
|
||||
subscription {
|
||||
onEvent {
|
||||
event {
|
||||
__typename
|
||||
... on TestEvent {
|
||||
param1
|
||||
param2
|
||||
param3
|
||||
},
|
||||
},
|
||||
block {
|
||||
number
|
||||
hash
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
* Trigger the `Test` event by calling example contract method:
|
||||
|
||||
```bash
|
||||
cd packages/graph-node
|
||||
|
||||
yarn example:test --address $EXAMPLE_ADDRESS
|
||||
```
|
||||
|
||||
A `Test` event shall be visible in the subscription at endpoint.
|
||||
|
||||
* Run the `getState` query at the endpoint to get the latest `IPLDBlock` for `EXAMPLE_ADDRESS`:
|
||||
|
||||
```graphql
|
||||
query {
|
||||
getState (
|
||||
blockHash: "EVENT_BLOCK_HASH"
|
||||
contractAddress: "EXAMPLE_ADDRESS"
|
||||
kind: "diff_staged"
|
||||
) {
|
||||
cid
|
||||
block {
|
||||
cid
|
||||
hash
|
||||
number
|
||||
timestamp
|
||||
parentHash
|
||||
}
|
||||
contractAddress
|
||||
data
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
* Run the query for entity at the endpoint:
|
||||
|
||||
```graphql
|
||||
query {
|
||||
author (
|
||||
block: {
|
||||
hash: "EVENT_BLOCK_HASH"
|
||||
}
|
||||
id: "0xdc7d7a8920c8eecc098da5b7522a5f31509b5bfc"
|
||||
) {
|
||||
__typename
|
||||
name
|
||||
paramInt
|
||||
paramBigInt
|
||||
paramBytes
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
* `diff` IPLDBlocks get created corresponding to the `diff_staged` blocks when their respective `eth_block`s reach the pruned region.
|
||||
|
||||
* In `packages/demo-example-watcher`:
|
||||
|
||||
* After the `diff` block has been created, create a `checkpoint`:
|
||||
|
||||
```bash
|
||||
cd packages/demo-example-watcher
|
||||
|
||||
yarn checkpoint --address $EXAMPLE_ADDRESS
|
||||
```
|
||||
|
||||
* A `checkpoint` IPLDBlock should be created at the latest canonical block hash.
|
||||
|
||||
* Run the `getState` query again at the endpoint with the output `blockHash` and kind `checkpoint`.
|
||||
|
||||
* All the `IPLDBlock` entries can be seen in `pg-admin` in table `ipld_block`.
|
||||
|
||||
* All the `diff` and `checkpoint` IPLDBlocks should be pushed to `IPFS`.
|
||||
|
||||
* Open IPFS WebUI http://127.0.0.1:5001/webui and search for `IPLDBlock`s using their `CID`s.
|
||||
@@ -16,6 +16,8 @@ import { checkGQLEntityInIPLDState, compareQuery, Config, getIPLDsByBlock, check
|
||||
import { Database } from '../../database';
|
||||
import { getSubgraphConfig } from '../../utils';
|
||||
|
||||
const DEFAULT_ENTITIES_LIMIT = 100;
|
||||
|
||||
const log = debug('vulcanize:compare-blocks');
|
||||
|
||||
export const main = async (): Promise<void> => {
|
||||
@@ -70,10 +72,33 @@ export const main = async (): Promise<void> => {
|
||||
type: 'boolean',
|
||||
describe: 'Compare time taken between GQL queries',
|
||||
default: false
|
||||
},
|
||||
queryEntitiesLimit: {
|
||||
type: 'number',
|
||||
default: DEFAULT_ENTITIES_LIMIT,
|
||||
describe: 'Limit for entities returned in query'
|
||||
},
|
||||
paginate: {
|
||||
type: 'boolean',
|
||||
describe: 'Paginate in multiple entities query and compare',
|
||||
default: false
|
||||
}
|
||||
}).argv;
|
||||
|
||||
const { startBlock, endBlock, batchSize, interval, rawJson, queryDir, fetchIds, configFile, timeDiff } = argv;
|
||||
const {
|
||||
startBlock,
|
||||
endBlock,
|
||||
batchSize,
|
||||
interval,
|
||||
rawJson,
|
||||
queryDir,
|
||||
fetchIds,
|
||||
configFile,
|
||||
timeDiff,
|
||||
queryEntitiesLimit,
|
||||
paginate
|
||||
} = argv;
|
||||
|
||||
const config: Config = await getConfig(configFile);
|
||||
const snakeNamingStrategy = new SnakeNamingStrategy();
|
||||
const clients = await getClients(config, timeDiff, queryDir);
|
||||
@@ -195,23 +220,41 @@ export const main = async (): Promise<void> => {
|
||||
} else {
|
||||
if (updatedEntities.has(entityName)) {
|
||||
let result;
|
||||
let skip = 0;
|
||||
|
||||
({ diff: resultDiff, result1: result } = await compareQuery(
|
||||
clients,
|
||||
queryName,
|
||||
{ block },
|
||||
rawJson,
|
||||
timeDiff
|
||||
));
|
||||
do {
|
||||
({ diff: resultDiff, result1: result } = await compareQuery(
|
||||
clients,
|
||||
queryName,
|
||||
{
|
||||
block,
|
||||
skip,
|
||||
first: queryEntitiesLimit
|
||||
},
|
||||
rawJson,
|
||||
timeDiff
|
||||
));
|
||||
|
||||
if (config.watcher.verifyState) {
|
||||
const ipldDiff = await checkGQLEntitiesInIPLDState(ipldStateByBlock, entityName, result[queryName], rawJson, config.watcher.skipFields);
|
||||
if (config.watcher.verifyState) {
|
||||
const ipldDiff = await checkGQLEntitiesInIPLDState(ipldStateByBlock, entityName, result[queryName], rawJson, config.watcher.skipFields);
|
||||
|
||||
if (ipldDiff) {
|
||||
log('Results mismatch for IPLD state:', ipldDiff);
|
||||
diffFound = true;
|
||||
if (ipldDiff) {
|
||||
log('Results mismatch for IPLD state:', ipldDiff);
|
||||
diffFound = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
skip += queryEntitiesLimit;
|
||||
} while (
|
||||
// Check if needed to query more entities.
|
||||
result[queryName].length === queryEntitiesLimit &&
|
||||
// Check if diff found.
|
||||
!diffFound &&
|
||||
!resultDiff &&
|
||||
// Check paginate flag
|
||||
// eslint-disable-next-line no-unmodified-loop-condition
|
||||
paginate
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ import {
|
||||
QueryRunner,
|
||||
Repository
|
||||
} from 'typeorm';
|
||||
import { ColumnMetadata } from 'typeorm/metadata/ColumnMetadata';
|
||||
import { SelectionNode } from 'graphql';
|
||||
import _ from 'lodash';
|
||||
import debug from 'debug';
|
||||
@@ -647,9 +648,9 @@ export class Database {
|
||||
return this.getStateEntityValues(block, stateEntity, entityFields, relations);
|
||||
}
|
||||
|
||||
getStateEntityValues (block: BlockProgressInterface, stateEntity: any, entityFields: any, relations: { [key: string]: any } = {}): { [key: string]: any } {
|
||||
const entityValues = entityFields.map((field: any) => {
|
||||
const { propertyName } = field;
|
||||
getStateEntityValues (block: BlockProgressInterface, stateEntity: any, entityFields: ColumnMetadata[], relations: { [key: string]: any } = {}): { [key: string]: any } {
|
||||
const entityValues = entityFields.map((field) => {
|
||||
const { propertyName, transformer } = field;
|
||||
|
||||
// Get blockHash property for db entry from block instance.
|
||||
if (propertyName === 'blockHash') {
|
||||
@@ -663,10 +664,10 @@ export class Database {
|
||||
|
||||
// Get blockNumber as _blockNumber and blockHash as _blockHash from the entityInstance (wasm).
|
||||
if (['_blockNumber', '_blockHash'].includes(propertyName)) {
|
||||
return fromStateEntityValues(stateEntity, propertyName.slice(1), relations);
|
||||
return fromStateEntityValues(stateEntity, propertyName.slice(1), relations, transformer);
|
||||
}
|
||||
|
||||
return fromStateEntityValues(stateEntity, propertyName, relations);
|
||||
return fromStateEntityValues(stateEntity, propertyName, relations, transformer);
|
||||
}, {});
|
||||
|
||||
return entityFields.reduce((acc: { [key: string]: any }, field: any, index: number) => {
|
||||
|
||||
@@ -3,6 +3,7 @@ import path from 'path';
|
||||
import fs from 'fs-extra';
|
||||
import debug from 'debug';
|
||||
import yaml from 'js-yaml';
|
||||
import { ValueTransformer } from 'typeorm';
|
||||
import { ColumnMetadata } from 'typeorm/metadata/ColumnMetadata';
|
||||
import assert from 'assert';
|
||||
|
||||
@@ -840,7 +841,12 @@ export const prepareEntityState = (updatedEntity: any, entityName: string, relat
|
||||
return diffData;
|
||||
};
|
||||
|
||||
export const fromStateEntityValues = (stateEntity: any, propertyName: string, relations: { [key: string]: any } = {}): any => {
|
||||
export const fromStateEntityValues = (
|
||||
stateEntity: any,
|
||||
propertyName: string,
|
||||
relations: { [key: string]: any } = {},
|
||||
transformer?: ValueTransformer | ValueTransformer[]
|
||||
): any => {
|
||||
// Parse DB data value from state entity data.
|
||||
if (relations) {
|
||||
const relation = relations[propertyName];
|
||||
@@ -854,5 +860,16 @@ export const fromStateEntityValues = (stateEntity: any, propertyName: string, re
|
||||
}
|
||||
}
|
||||
|
||||
if (transformer) {
|
||||
if (Array.isArray(transformer)) {
|
||||
// Apply transformer in reverse order similar to when reading from DB.
|
||||
return transformer.reduceRight((acc, elTransformer) => {
|
||||
return elTransformer.from(acc);
|
||||
}, stateEntity[propertyName]);
|
||||
}
|
||||
|
||||
return transformer.from(stateEntity[propertyName]);
|
||||
}
|
||||
|
||||
return stateEntity[propertyName];
|
||||
};
|
||||
|
||||
@@ -12,7 +12,7 @@ import { SelectionNode } from 'graphql';
|
||||
|
||||
import { ResultObject } from '@vulcanize/assemblyscript/lib/loader';
|
||||
import { EthClient } from '@cerc-io/ipld-eth-client';
|
||||
import { getFullBlock, BlockHeight, ServerConfig, getFullTransaction, QueryOptions, IPLDBlockInterface, IndexerInterface, BlockProgressInterface } from '@cerc-io/util';
|
||||
import { getFullBlock, BlockHeight, ServerConfig, getFullTransaction, QueryOptions, IPLDBlockInterface, IndexerInterface, BlockProgressInterface, cachePrunedEntitiesCount } from '@cerc-io/util';
|
||||
|
||||
import { createBlock, createEvent, getSubgraphConfig, resolveEntityFieldConflicts, Transaction } from './utils';
|
||||
import { Context, GraphData, instantiate } from './loader';
|
||||
@@ -186,8 +186,12 @@ export class GraphWatcher {
|
||||
|
||||
// Create ethereum event to be passed to the wasm event handler.
|
||||
const ethereumEvent = await createEvent(instanceExports, contract, data);
|
||||
|
||||
await this._handleMemoryError(instanceExports[eventHandler.handler](ethereumEvent), dataSource.name);
|
||||
try {
|
||||
await this._handleMemoryError(instanceExports[eventHandler.handler](ethereumEvent), dataSource.name);
|
||||
} catch (error) {
|
||||
this._clearCachedEntities();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async handleBlock (blockHash: string) {
|
||||
@@ -249,7 +253,12 @@ export class GraphWatcher {
|
||||
await instanceExports[blockHandler.handler](ethereumBlock);
|
||||
});
|
||||
|
||||
await this._handleMemoryError(Promise.all(blockHandlerPromises), dataSource.name);
|
||||
try {
|
||||
await this._handleMemoryError(Promise.all(blockHandlerPromises), dataSource.name);
|
||||
} catch (error) {
|
||||
this._clearCachedEntities();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -375,8 +384,18 @@ export class GraphWatcher {
|
||||
entities: new Map()
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
log(`Size of cachedEntities.frothyBlocks map: ${this._database.cachedEntities.frothyBlocks.size}`);
|
||||
log(`Size of cachedEntities.frothyBlocks map: ${this._database.cachedEntities.frothyBlocks.size}`);
|
||||
this._measureCachedPrunedEntities();
|
||||
|
||||
assert(this._indexer);
|
||||
// Check if it is time to clear entities cache.
|
||||
if (blockProgress.blockNumber % this._indexer.serverConfig.clearEntitiesCacheInterval === 0) {
|
||||
log(`Clearing cachedEntities.latestPrunedEntities at block ${blockProgress.blockNumber}`);
|
||||
// Clearing only pruned region as frothy region cache gets updated in pruning queue.
|
||||
this._database.cachedEntities.latestPrunedEntities.clear();
|
||||
log(`Cleared cachedEntities.latestPrunedEntities. Map size: ${this._database.cachedEntities.latestPrunedEntities.size}`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -407,6 +426,19 @@ export class GraphWatcher {
|
||||
prunedBlockHashes.forEach(blockHash => this._database.cachedEntities.frothyBlocks.delete(blockHash));
|
||||
}
|
||||
|
||||
_clearCachedEntities () {
|
||||
this._database.cachedEntities.frothyBlocks.clear();
|
||||
this._database.cachedEntities.latestPrunedEntities.clear();
|
||||
}
|
||||
|
||||
_measureCachedPrunedEntities () {
|
||||
const totalEntities = Array.from(this._database.cachedEntities.latestPrunedEntities.values())
|
||||
.reduce((acc, idEntitiesMap) => acc + idEntitiesMap.size, 0);
|
||||
|
||||
log(`Total entities in cachedEntities.latestPrunedEntities map: ${totalEntities}`);
|
||||
cachePrunedEntitiesCount.set(totalEntities);
|
||||
}
|
||||
|
||||
/**
|
||||
* Method to reinstantiate WASM instance for specified dataSource.
|
||||
* @param dataSourceName
|
||||
|
||||
@@ -211,6 +211,7 @@ class ServerConfig implements ServerConfigInterface {
|
||||
wasmRestartBlocksInterval: number;
|
||||
filterLogs: boolean;
|
||||
maxEventsBlockRange: number;
|
||||
clearEntitiesCacheInterval: number;
|
||||
|
||||
constructor () {
|
||||
this.host = '';
|
||||
@@ -225,5 +226,6 @@ class ServerConfig implements ServerConfigInterface {
|
||||
this.wasmRestartBlocksInterval = 0;
|
||||
this.filterLogs = false;
|
||||
this.maxEventsBlockRange = 0;
|
||||
this.clearEntitiesCacheInterval = 0;
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user