Merge pull request #51 from vulcanize/cleanup-links

Cleanup links
This commit is contained in:
Rob Mulholand 2019-04-02 08:45:08 -05:00 committed by GitHub
commit fea4ce0c7a
6 changed files with 24 additions and 28 deletions

View File

@ -1,8 +1,6 @@
# Vulcanize DB
[![Join the chat at https://gitter.im/vulcanizeio/VulcanizeDB](https://badges.gitter.im/vulcanizeio/VulcanizeDB.svg)](https://gitter.im/vulcanizeio/VulcanizeDB?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![Build Status](https://travis-ci.com/vulcanize/maker-vulcanizedb.svg?token=MKcE2K7CRvKtdxSSnbap&branch=staging)](https://travis-ci.com/vulcanize/maker-vulcanizedb)
[![Build Status](https://travis-ci.org/vulcanize/vulcanizedb.svg?branch=master)](https://travis-ci.org/vulcanize/vulcanizedb)
## About
@ -41,7 +39,6 @@ In order to install packages with `dep`, ensure you are in the project directory
After `dep` finishes, dependencies should be installed within your `GOPATH` at the versions specified in `Gopkg.toml`.
Because we are working with a modified version of the go-ethereum accounts/abi package, after running `dep ensure` you will need to run `git checkout vendor/github/ethereum/go-ethereum/accounts/abi` to checkout the modified dependency.
This is explained in greater detail [here](https://github.com/vulcanize/maker-vulcanizedb/issues/41).
Lastly, ensure that `GOPATH` is defined in your shell. If necessary, `GOPATH` can be set in `~/.bashrc` or `~/.bash_profile`, depending upon your system. It can be additionally helpful to add `$GOPATH/bin` to your shell's `$PATH`.
@ -318,17 +315,17 @@ Also notice that the contract address used for the schema name has been down-cas
## composeAndExecute
The `composeAndExecute` command is used to compose and execute over an arbitrary set of custom transformers.
This is accomplished by generating a Go pluggin which allows our `vulcanizedb` binary to link to external transformers, so
long as they abide by our standard [interfaces](https://github.com/vulcanize/maker-vulcanizedb/tree/compose_and_execute/libraries/shared/transformer).
long as they abide by our standard [interfaces](../staging/libraries/shared/transformer).
This command requires Go 1.11+ and [Go plugins](https://golang.org/pkg/plugin/) only work on Unix based systems.
### Writing custom transformers
Storage Transformers
* [Guide](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/factories/storage/README.md)
* [Example](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/factories/storage/EXAMPLE.md)
* [Guide](../staging/libraries/shared/factories/storage/README.md)
* [Example](../staging/libraries/shared/factories/storage/EXAMPLE.md)
Event Transformers
* [Guide](https://github.com/vulcanize/maker-vulcanizedb/blob/event_docs/libraries/shared/factories/README.md)
* [Guide](../staging/libraries/shared/factories/event/README.md)
* [Example](https://github.com/vulcanize/ens_transformers/tree/working)
### composeAndExecute configuration
@ -392,12 +389,12 @@ The config provides information for composing a set of transformers:
- `path` is the relative path from `repository` to the transformer's `TransformerInitializer` directory (initializer package).
- Transformer repositories need to be cloned into the user's $GOPATH (`go get`)
- `type` is the type of the transformer; indicating which type of watcher it works with (for now, there are only two options: `eth_event` and `eth_storage`)
- `eth_storage` indicates the transformer works with the [storage watcher](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/libraries/shared/watcher/storage_watcher.go)
- `eth_storage` indicates the transformer works with the [storage watcher](../staging/libraries/shared/watcher/storage_watcher.go)
that fetches state and storage diffs from an ETH node (instead of, for example, from IPFS)
- `eth_event` indicates the transformer works with the [event watcher](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/libraries/shared/watcher/event_watcher.go)
- `eth_event` indicates the transformer works with the [event watcher](../staging/libraries/shared/watcher/event_watcher.go)
that fetches event logs from an ETH node
- `eth_contract` indicates the transformer works with the [contract watcher](https://github.com/vulcanize/maker-vulcanizedb/blob/omni_update/libraries/shared/watcher/generic_watcher.go)
that is made to work with [contract_watcher pkg](https://github.com/vulcanize/maker-vulcanizedb/tree/staging/pkg/omni)
- `eth_contract` indicates the transformer works with the [contract watcher](../staging/libraries/shared/watcher/contract_watcher.go)
that is made to work with [contract_watcher pkg](../staging/pkg/contract_watcher)
based transformers which work with either a light or full sync vDB to watch events and poll public methods ([example](https://github.com/vulcanize/ens_transformers/blob/working/transformers/domain_records/transformer.go))
- `migrations` is the relative path from `repository` to the db migrations directory for the transformer
- `rank` determines the order that migrations are ran, with lower ranked migrations running first
@ -446,12 +443,12 @@ func (e exporter) Export() []interface1.EventTransformerInitializer, []interface
To plug in an external transformer we need to:
* Create a [package](https://github.com/vulcanize/ens_transformers/blob/working/transformers/registry/new_owner/initializer/initializer.go)
that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/transformer/event_transformer.go#L33)
or [StorageTransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/transformer/storage_transformer.go#L31),
or [ContractTransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/omni_update/libraries/shared/transformer/contract_transformer.go#L31), respectively
* Design the transformers to work in the context of their [event](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/watcher/event_watcher.go#L83),
[storage](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/watcher/storage_watcher.go#L53),
or [contract](https://github.com/vulcanize/maker-vulcanizedb/blob/omni_update/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes
that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](../staging/libraries/shared/transformer/event_transformer.go#L33)
or [StorageTransformerInitializer](../staging/libraries/shared/transformer/storage_transformer.go#L31),
or [ContractTransformerInitializer](../staging/libraries/shared/transformer/contract_transformer.go#L31), respectively
* Design the transformers to work in the context of their [event](../staging/libraries/shared/watcher/event_watcher.go#L83),
[storage](../staging/libraries/shared/watcher/storage_watcher.go#L53),
or [contract](../staging/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes
* Create db migrations to run against vulcanizeDB so that we can store the transformer output
* Do not `goose fix` the transformer migrations
* Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields

View File

@ -17,13 +17,13 @@ Looking forward, we will be building fetchers that enable sourcing data from IPF
VulcanizeDB has shared code built out for building and plugging in event transformers
### [Event Watcher (light sync)](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/libraries/shared/watcher/event_watcher.go)
### [Event Watcher (light sync)](../staging/libraries/shared/watcher/event_watcher.go)
The event watcher is responsible for continuously fetching and delegating chunks of logs and their associated header to the appropriate transformers.
Using the `compose` or `composeAndExecute` command, event watchers can be loaded with plugin event transformers and execute over them.
### [Event Transformer](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/libraries/shared/transformer/event_transformer.go)
### [Event Transformer](../staging/libraries/shared/transformer/event_transformer.go)
The event transformer is responsible for converting event logs into more useful data objects and storing them in Postgres.
The event transformer is composed of converter and repository interfaces and a config struct:
@ -371,10 +371,10 @@ ALTER TABLE public.checked_headers
-- +goose Down
DROP TABLE maker.bite;
DROP TABLE example_schema.example_event;
ALTER TABLE public.checked_headers
DROP COLUMN bite_checked;
DROP COLUMN example_event_checked;
```
Notice that we have also added a column to the `checked_headers` table for this event so that we can keep track
@ -385,4 +385,4 @@ of which headers we have already filtered through for this event.
To create a transformer for a contract event we need to create entities for unpacking the raw log, models to represent
the final data structure, a converter to mediate this unpacking and conversion between entities to models, a repository to write
these models to Postgres, db migrations to accommodate these models in Postgres, and a EventTransformerInitializer to export the
configured transformer and load it as a plugin to the `compose` or `composeAndExecute` commands as described in the [main readme](https://github.com/vulcanize/maker-vulcanizedb/blob/staging/README.md#composeandexecute-configuration).
configured transformer and load it as a plugin to the `compose` or `composeAndExecute` commands as described in the main readme.

View File

@ -9,7 +9,6 @@ The current approach for caching smart contract storage diffs assumes that you a
The CSV file is expected to have 5 columns: contract address, block hash, block number, storage key, storage value.
We have [a branch on vulcanize/parity-ethereum](https://github.com/vulcanize/parity-ethereum/tree/watch-storage-diffs) that enables running a node that writes storage diffs this way.
We also have [sample data](https://github.com/8thlight/maker-vulcanizedb/pull/132/files) that comes from running that node against Kovan through block 9796184.
Looking forward, we would like to isolate this assumption as much as possible.
We may end up needing to read CSV data that is formatted differently, or reading data from a non-CSV source, and we do not want resulting changes to cascade throughout the codebase.

View File

@ -1,7 +1,7 @@
{
"name": "vulcanizedb",
"version": "1.0.0",
"description": "[![Join the chat at https://gitter.im/vulcanizeio/VulcanizeDB](https://badges.gitter.im/vulcanizeio/VulcanizeDB.svg)](https://gitter.im/vulcanizeio/VulcanizeDB?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)",
"description": "",
"scripts": {
"build": "rm -rf ./build/dist && webpack --config=./webpack.config.js",
"lint": "tslint --project ./tsconfig.json --config ./tslint.json",

View File

@ -1,4 +1,4 @@
// NOTE (jchristie@8thlight.com) This file helps Jasmine
// NOTE: This file helps Jasmine
// comprehend TS sourcemaps by installing a reporter
// specifically for TypeScript
const TSConsoleReporter = require('jasmine-ts-console-reporter');

View File

@ -2,7 +2,7 @@ import { RequestHandler } from 'express';
import { Server } from 'http';
import { PluginHookFn } from 'postgraphile/build/postgraphile/pluginHook';
// NOTE (jchristie@8thlight.com) Shape of the middleware is not
// NOTE: Shape of the middleware is not
// currently important to this application, but if a need arises,
// any needed shape can be assigned from a custom type here. For
// the time being, this is a named stub to provide clarity.
@ -14,7 +14,7 @@ export interface PostgraphileOptions {
watchPg: boolean;
enableCors: boolean;
graphiql: boolean;
// NOTE (jchristie@8thlight.com) Shape of the middlewares is not
// NOTE: Shape of the middlewares is not
// currently important to this application, but if a need arises,
// any needed shape can be assigned from a custom type here.
webSocketMiddlewares: object[];