commit
fea4ce0c7a
33
README.md
33
README.md
@ -1,8 +1,6 @@
|
|||||||
# Vulcanize DB
|
# Vulcanize DB
|
||||||
|
|
||||||
[![Join the chat at https://gitter.im/vulcanizeio/VulcanizeDB](https://badges.gitter.im/vulcanizeio/VulcanizeDB.svg)](https://gitter.im/vulcanizeio/VulcanizeDB?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
[![Build Status](https://travis-ci.org/vulcanize/vulcanizedb.svg?branch=master)](https://travis-ci.org/vulcanize/vulcanizedb)
|
||||||
|
|
||||||
[![Build Status](https://travis-ci.com/vulcanize/maker-vulcanizedb.svg?token=MKcE2K7CRvKtdxSSnbap&branch=staging)](https://travis-ci.com/vulcanize/maker-vulcanizedb)
|
|
||||||
|
|
||||||
## About
|
## About
|
||||||
|
|
||||||
@ -41,7 +39,6 @@ In order to install packages with `dep`, ensure you are in the project directory
|
|||||||
After `dep` finishes, dependencies should be installed within your `GOPATH` at the versions specified in `Gopkg.toml`.
|
After `dep` finishes, dependencies should be installed within your `GOPATH` at the versions specified in `Gopkg.toml`.
|
||||||
|
|
||||||
Because we are working with a modified version of the go-ethereum accounts/abi package, after running `dep ensure` you will need to run `git checkout vendor/github/ethereum/go-ethereum/accounts/abi` to checkout the modified dependency.
|
Because we are working with a modified version of the go-ethereum accounts/abi package, after running `dep ensure` you will need to run `git checkout vendor/github/ethereum/go-ethereum/accounts/abi` to checkout the modified dependency.
|
||||||
This is explained in greater detail [here](https://github.com/vulcanize/maker-vulcanizedb/issues/41).
|
|
||||||
|
|
||||||
Lastly, ensure that `GOPATH` is defined in your shell. If necessary, `GOPATH` can be set in `~/.bashrc` or `~/.bash_profile`, depending upon your system. It can be additionally helpful to add `$GOPATH/bin` to your shell's `$PATH`.
|
Lastly, ensure that `GOPATH` is defined in your shell. If necessary, `GOPATH` can be set in `~/.bashrc` or `~/.bash_profile`, depending upon your system. It can be additionally helpful to add `$GOPATH/bin` to your shell's `$PATH`.
|
||||||
|
|
||||||
@ -318,17 +315,17 @@ Also notice that the contract address used for the schema name has been down-cas
|
|||||||
## composeAndExecute
|
## composeAndExecute
|
||||||
The `composeAndExecute` command is used to compose and execute over an arbitrary set of custom transformers.
|
The `composeAndExecute` command is used to compose and execute over an arbitrary set of custom transformers.
|
||||||
This is accomplished by generating a Go pluggin which allows our `vulcanizedb` binary to link to external transformers, so
|
This is accomplished by generating a Go pluggin which allows our `vulcanizedb` binary to link to external transformers, so
|
||||||
long as they abide by our standard [interfaces](https://github.com/vulcanize/maker-vulcanizedb/tree/compose_and_execute/libraries/shared/transformer).
|
long as they abide by our standard [interfaces](../staging/libraries/shared/transformer).
|
||||||
|
|
||||||
This command requires Go 1.11+ and [Go plugins](https://golang.org/pkg/plugin/) only work on Unix based systems.
|
This command requires Go 1.11+ and [Go plugins](https://golang.org/pkg/plugin/) only work on Unix based systems.
|
||||||
|
|
||||||
### Writing custom transformers
|
### Writing custom transformers
|
||||||
Storage Transformers
|
Storage Transformers
|
||||||
* [Guide](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/factories/storage/README.md)
|
* [Guide](../staging/libraries/shared/factories/storage/README.md)
|
||||||
* [Example](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/factories/storage/EXAMPLE.md)
|
* [Example](../staging/libraries/shared/factories/storage/EXAMPLE.md)
|
||||||
|
|
||||||
Event Transformers
|
Event Transformers
|
||||||
* [Guide](https://github.com/vulcanize/maker-vulcanizedb/blob/event_docs/libraries/shared/factories/README.md)
|
* [Guide](../staging/libraries/shared/factories/event/README.md)
|
||||||
* [Example](https://github.com/vulcanize/ens_transformers/tree/working)
|
* [Example](https://github.com/vulcanize/ens_transformers/tree/working)
|
||||||
|
|
||||||
### composeAndExecute configuration
|
### composeAndExecute configuration
|
||||||
@ -392,12 +389,12 @@ The config provides information for composing a set of transformers:
|
|||||||
- `path` is the relative path from `repository` to the transformer's `TransformerInitializer` directory (initializer package).
|
- `path` is the relative path from `repository` to the transformer's `TransformerInitializer` directory (initializer package).
|
||||||
- Transformer repositories need to be cloned into the user's $GOPATH (`go get`)
|
- Transformer repositories need to be cloned into the user's $GOPATH (`go get`)
|
||||||
- `type` is the type of the transformer; indicating which type of watcher it works with (for now, there are only two options: `eth_event` and `eth_storage`)
|
- `type` is the type of the transformer; indicating which type of watcher it works with (for now, there are only two options: `eth_event` and `eth_storage`)
|
||||||
- `eth_storage` indicates the transformer works with the [storage watcher](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/libraries/shared/watcher/storage_watcher.go)
|
- `eth_storage` indicates the transformer works with the [storage watcher](../staging/libraries/shared/watcher/storage_watcher.go)
|
||||||
that fetches state and storage diffs from an ETH node (instead of, for example, from IPFS)
|
that fetches state and storage diffs from an ETH node (instead of, for example, from IPFS)
|
||||||
- `eth_event` indicates the transformer works with the [event watcher](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/libraries/shared/watcher/event_watcher.go)
|
- `eth_event` indicates the transformer works with the [event watcher](../staging/libraries/shared/watcher/event_watcher.go)
|
||||||
that fetches event logs from an ETH node
|
that fetches event logs from an ETH node
|
||||||
- `eth_contract` indicates the transformer works with the [contract watcher](https://github.com/vulcanize/maker-vulcanizedb/blob/omni_update/libraries/shared/watcher/generic_watcher.go)
|
- `eth_contract` indicates the transformer works with the [contract watcher](../staging/libraries/shared/watcher/contract_watcher.go)
|
||||||
that is made to work with [contract_watcher pkg](https://github.com/vulcanize/maker-vulcanizedb/tree/staging/pkg/omni)
|
that is made to work with [contract_watcher pkg](../staging/pkg/contract_watcher)
|
||||||
based transformers which work with either a light or full sync vDB to watch events and poll public methods ([example](https://github.com/vulcanize/ens_transformers/blob/working/transformers/domain_records/transformer.go))
|
based transformers which work with either a light or full sync vDB to watch events and poll public methods ([example](https://github.com/vulcanize/ens_transformers/blob/working/transformers/domain_records/transformer.go))
|
||||||
- `migrations` is the relative path from `repository` to the db migrations directory for the transformer
|
- `migrations` is the relative path from `repository` to the db migrations directory for the transformer
|
||||||
- `rank` determines the order that migrations are ran, with lower ranked migrations running first
|
- `rank` determines the order that migrations are ran, with lower ranked migrations running first
|
||||||
@ -446,12 +443,12 @@ func (e exporter) Export() []interface1.EventTransformerInitializer, []interface
|
|||||||
To plug in an external transformer we need to:
|
To plug in an external transformer we need to:
|
||||||
|
|
||||||
* Create a [package](https://github.com/vulcanize/ens_transformers/blob/working/transformers/registry/new_owner/initializer/initializer.go)
|
* Create a [package](https://github.com/vulcanize/ens_transformers/blob/working/transformers/registry/new_owner/initializer/initializer.go)
|
||||||
that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/transformer/event_transformer.go#L33)
|
that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](../staging/libraries/shared/transformer/event_transformer.go#L33)
|
||||||
or [StorageTransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/transformer/storage_transformer.go#L31),
|
or [StorageTransformerInitializer](../staging/libraries/shared/transformer/storage_transformer.go#L31),
|
||||||
or [ContractTransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/omni_update/libraries/shared/transformer/contract_transformer.go#L31), respectively
|
or [ContractTransformerInitializer](../staging/libraries/shared/transformer/contract_transformer.go#L31), respectively
|
||||||
* Design the transformers to work in the context of their [event](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/watcher/event_watcher.go#L83),
|
* Design the transformers to work in the context of their [event](../staging/libraries/shared/watcher/event_watcher.go#L83),
|
||||||
[storage](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/watcher/storage_watcher.go#L53),
|
[storage](../staging/libraries/shared/watcher/storage_watcher.go#L53),
|
||||||
or [contract](https://github.com/vulcanize/maker-vulcanizedb/blob/omni_update/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes
|
or [contract](../staging/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes
|
||||||
* Create db migrations to run against vulcanizeDB so that we can store the transformer output
|
* Create db migrations to run against vulcanizeDB so that we can store the transformer output
|
||||||
* Do not `goose fix` the transformer migrations
|
* Do not `goose fix` the transformer migrations
|
||||||
* Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields
|
* Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields
|
||||||
|
@ -17,13 +17,13 @@ Looking forward, we will be building fetchers that enable sourcing data from IPF
|
|||||||
|
|
||||||
VulcanizeDB has shared code built out for building and plugging in event transformers
|
VulcanizeDB has shared code built out for building and plugging in event transformers
|
||||||
|
|
||||||
### [Event Watcher (light sync)](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/libraries/shared/watcher/event_watcher.go)
|
### [Event Watcher (light sync)](../staging/libraries/shared/watcher/event_watcher.go)
|
||||||
|
|
||||||
The event watcher is responsible for continuously fetching and delegating chunks of logs and their associated header to the appropriate transformers.
|
The event watcher is responsible for continuously fetching and delegating chunks of logs and their associated header to the appropriate transformers.
|
||||||
|
|
||||||
Using the `compose` or `composeAndExecute` command, event watchers can be loaded with plugin event transformers and execute over them.
|
Using the `compose` or `composeAndExecute` command, event watchers can be loaded with plugin event transformers and execute over them.
|
||||||
|
|
||||||
### [Event Transformer](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/libraries/shared/transformer/event_transformer.go)
|
### [Event Transformer](../staging/libraries/shared/transformer/event_transformer.go)
|
||||||
|
|
||||||
The event transformer is responsible for converting event logs into more useful data objects and storing them in Postgres.
|
The event transformer is responsible for converting event logs into more useful data objects and storing them in Postgres.
|
||||||
The event transformer is composed of converter and repository interfaces and a config struct:
|
The event transformer is composed of converter and repository interfaces and a config struct:
|
||||||
@ -371,10 +371,10 @@ ALTER TABLE public.checked_headers
|
|||||||
|
|
||||||
|
|
||||||
-- +goose Down
|
-- +goose Down
|
||||||
DROP TABLE maker.bite;
|
DROP TABLE example_schema.example_event;
|
||||||
|
|
||||||
ALTER TABLE public.checked_headers
|
ALTER TABLE public.checked_headers
|
||||||
DROP COLUMN bite_checked;
|
DROP COLUMN example_event_checked;
|
||||||
```
|
```
|
||||||
|
|
||||||
Notice that we have also added a column to the `checked_headers` table for this event so that we can keep track
|
Notice that we have also added a column to the `checked_headers` table for this event so that we can keep track
|
||||||
@ -385,4 +385,4 @@ of which headers we have already filtered through for this event.
|
|||||||
To create a transformer for a contract event we need to create entities for unpacking the raw log, models to represent
|
To create a transformer for a contract event we need to create entities for unpacking the raw log, models to represent
|
||||||
the final data structure, a converter to mediate this unpacking and conversion between entities to models, a repository to write
|
the final data structure, a converter to mediate this unpacking and conversion between entities to models, a repository to write
|
||||||
these models to Postgres, db migrations to accommodate these models in Postgres, and a EventTransformerInitializer to export the
|
these models to Postgres, db migrations to accommodate these models in Postgres, and a EventTransformerInitializer to export the
|
||||||
configured transformer and load it as a plugin to the `compose` or `composeAndExecute` commands as described in the [main readme](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/README.md#composeandexecute-configuration).
|
configured transformer and load it as a plugin to the `compose` or `composeAndExecute` commands as described in the main readme.
|
||||||
|
@ -9,7 +9,6 @@ The current approach for caching smart contract storage diffs assumes that you a
|
|||||||
The CSV file is expected to have 5 columns: contract address, block hash, block number, storage key, storage value.
|
The CSV file is expected to have 5 columns: contract address, block hash, block number, storage key, storage value.
|
||||||
|
|
||||||
We have [a branch on vulcanize/parity-ethereum](https://github.com/vulcanize/parity-ethereum/tree/watch-storage-diffs) that enables running a node that writes storage diffs this way.
|
We have [a branch on vulcanize/parity-ethereum](https://github.com/vulcanize/parity-ethereum/tree/watch-storage-diffs) that enables running a node that writes storage diffs this way.
|
||||||
We also have [sample data](https://github.com/8thlight/maker-vulcanizedb/pull/132/files) that comes from running that node against Kovan through block 9796184.
|
|
||||||
|
|
||||||
Looking forward, we would like to isolate this assumption as much as possible.
|
Looking forward, we would like to isolate this assumption as much as possible.
|
||||||
We may end up needing to read CSV data that is formatted differently, or reading data from a non-CSV source, and we do not want resulting changes to cascade throughout the codebase.
|
We may end up needing to read CSV data that is formatted differently, or reading data from a non-CSV source, and we do not want resulting changes to cascade throughout the codebase.
|
||||||
|
@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "vulcanizedb",
|
"name": "vulcanizedb",
|
||||||
"version": "1.0.0",
|
"version": "1.0.0",
|
||||||
"description": "[![Join the chat at https://gitter.im/vulcanizeio/VulcanizeDB](https://badges.gitter.im/vulcanizeio/VulcanizeDB.svg)](https://gitter.im/vulcanizeio/VulcanizeDB?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)",
|
"description": "",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"build": "rm -rf ./build/dist && webpack --config=./webpack.config.js",
|
"build": "rm -rf ./build/dist && webpack --config=./webpack.config.js",
|
||||||
"lint": "tslint --project ./tsconfig.json --config ./tslint.json",
|
"lint": "tslint --project ./tsconfig.json --config ./tslint.json",
|
||||||
|
@ -1,4 +1,4 @@
|
|||||||
// NOTE (jchristie@8thlight.com) This file helps Jasmine
|
// NOTE: This file helps Jasmine
|
||||||
// comprehend TS sourcemaps by installing a reporter
|
// comprehend TS sourcemaps by installing a reporter
|
||||||
// specifically for TypeScript
|
// specifically for TypeScript
|
||||||
const TSConsoleReporter = require('jasmine-ts-console-reporter');
|
const TSConsoleReporter = require('jasmine-ts-console-reporter');
|
||||||
|
@ -2,7 +2,7 @@ import { RequestHandler } from 'express';
|
|||||||
import { Server } from 'http';
|
import { Server } from 'http';
|
||||||
import { PluginHookFn } from 'postgraphile/build/postgraphile/pluginHook';
|
import { PluginHookFn } from 'postgraphile/build/postgraphile/pluginHook';
|
||||||
|
|
||||||
// NOTE (jchristie@8thlight.com) Shape of the middleware is not
|
// NOTE: Shape of the middleware is not
|
||||||
// currently important to this application, but if a need arises,
|
// currently important to this application, but if a need arises,
|
||||||
// any needed shape can be assigned from a custom type here. For
|
// any needed shape can be assigned from a custom type here. For
|
||||||
// the time being, this is a named stub to provide clarity.
|
// the time being, this is a named stub to provide clarity.
|
||||||
@ -14,7 +14,7 @@ export interface PostgraphileOptions {
|
|||||||
watchPg: boolean;
|
watchPg: boolean;
|
||||||
enableCors: boolean;
|
enableCors: boolean;
|
||||||
graphiql: boolean;
|
graphiql: boolean;
|
||||||
// NOTE (jchristie@8thlight.com) Shape of the middlewares is not
|
// NOTE: Shape of the middlewares is not
|
||||||
// currently important to this application, but if a need arises,
|
// currently important to this application, but if a need arises,
|
||||||
// any needed shape can be assigned from a custom type here.
|
// any needed shape can be assigned from a custom type here.
|
||||||
webSocketMiddlewares: object[];
|
webSocketMiddlewares: object[];
|
||||||
|
Loading…
Reference in New Issue
Block a user