cerc-io/ipld-eth-server
18
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity

add some info for separate compose and execute commands; note about

Browse Source 
flaky test
This commit is contained in:
Ian Norden 2019-04-19 10:13:27 -05:00
parent 9c06e9387c
commit 01ffc80ef2
4 changed files with 81 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@ -108,11 +108,15 @@ These commands are described in detail [here](../staging/documentation/sync.md).
### Data transformation ### Data transformation
Contract watchers use the raw data that has been synced into Postgres to filter out and apply transformations to specific data of interest. Contract watchers use the raw data that has been synced into Postgres to filter out and apply transformations to specific data of interest.
There is a built-in `contractWatcher` command which provides generic transformation of most contract data. This command is described in detail [here](../staging/documentation/contractWatcher.md). There is a built-in `contractWatcher` command which provides generic transformation of most contract data.
The `contractWatcher` command is described further [here](../staging/documentation/contractWatcher.md).
In many cases a custom transformer or set of transformers will need to be written to provide complete or more comprehensive coverage or to optimize other aspects of the output for a specific end-use. In many cases a custom transformer or set of transformers will need to be written to provide complete or more comprehensive coverage or to optimize other aspects of the output for a specific end-use.
In this case we have provided the `compose`, `execute`, and `composeAndExecute` commands for running custom transformers from external repositories. This is described in detail [here](../staging/documentation/composeAndExecute.md). In this case we have provided the `compose`, `execute`, and `composeAndExecute` commands for running custom transformers from external repositories.
Usage of the `compose`, `execute`, and `composeAndExecute` commands is described further [here](../staging/documentation/composeAndExecute.md).
Documentation on how to build custom transformers to work with these commands can be found [here](../staging/documentation/transformers.md).
## Tests ## Tests
- Replace the empty `ipcPath` in the `environments/infura.toml` with a path to a full node's eth_jsonrpc endpoint (e.g. local geth node ipc path or infura url) - Replace the empty `ipcPath` in the `environments/infura.toml` with a path to a full node's eth_jsonrpc endpoint (e.g. local geth node ipc path or infura url)

62
documentation/composeAndExecute.md
View File

@ -3,48 +3,42 @@ The `composeAndExecute` command is used to compose and execute over an arbitrary
This is accomplished by generating a Go pluggin which allows the `vulcanizedb` binary to link to external transformers, so This is accomplished by generating a Go pluggin which allows the `vulcanizedb` binary to link to external transformers, so
long as they abide by one of the standard [interfaces](../staging/libraries/shared/transformer). long as they abide by one of the standard [interfaces](../staging/libraries/shared/transformer).
This command requires Go 1.11+ and [Go plugins](https://golang.org/pkg/plugin/) only work on Unix-based systems. Additionally, there are separate `compose` and `execute` commands to allow pre-building and linking to a pre-built .so file.
## Writing custom transformers **NOTE:**
Storage Transformers 1. It is necessary that the .so file was built with the same exact dependencies that are present in the execution environment,
* [Guide](../../staging/libraries/shared/factories/storage/README.md) i.e. we need to `compose` and `execute` the plugin .so file with the same exact version of vulcanizeDB.
* [Example](../../staging/libraries/shared/factories/storage/EXAMPLE.md) 1. The plugin migrations are run during the plugin's composition. As such, if `execute` is used to run a prebuilt .so in a different
environment than the one it was composed in then the migrations for that plugin will first need to be manually ran against that environment's Postgres database.
Event Transformers
* [Guide](../../staging/libraries/shared/factories/event/README.md)
* [Example 1](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registar)
* [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registry)
* [Example 3](https://github.com/vulcanize/ens_transformers/tree/master/transformers/resolver)
Contract Transformers These commands require Go 1.11+ and use [Go plugins](https://golang.org/pkg/plugin/) which only work on Unix-based systems.
* [Example 1](https://github.com/vulcanize/account_transformers) There is also an ongoing [conflict](https://github.com/golang/go/issues/20481) between Go plugins and the use vendored dependencies which
* [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/domain_records) imposes certain limitations on how the plugins are built.
## Preparing transformers to work as a plugin for composeAndExecute ## Commands
To plug in an external transformer we need to: The `compose` and `composeAndExecute` commands assume you are in the vulcanizdb directory located at your system's `$GOPATH`,
and that all of the transformer repositories for building the plugin are present at their `$GOPATH` directories.
1. Create a package that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](../staging/libraries/shared/transformer/event_transformer.go#L33) The `execute` command does not require the plugin transformer dependencies be located in their
or [StorageTransformerInitializer](../../staging/libraries/shared/transformer/storage_transformer.go#L31), `$GOPATH` directories, instead it expects a prebuilt .so file (of the name specified in the config file)
or [ContractTransformerInitializer](../../staging/libraries/shared/transformer/contract_transformer.go#L31), respectively to be in `$GOPATH/src/github.com/vulcanize/vulcanizedb/plugins/` and, as noted above, also expects the plugin
2. Design the transformers to work in the context of their [event](../staging/libraries/shared/watcher/event_watcher.go#L83), db migrations to have already been ran against the database.
[storage](../../staging/libraries/shared/watcher/storage_watcher.go#L53),
or [contract](../../staging/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes
3. Create db migrations to run against vulcanizeDB so that we can store the transformer output
* Do not `goose fix` the transformer migrations, this is to ensure they are always ran after the core vulcanizedb migrations which are kept in their fixed form
* Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields
* If the base vDB migrations occupy this path as well, they need to be in their `goose fix`ed form
as they are [here](../../staging/db/migrations)
To update a plugin repository with changes to the core vulcanizedb repository, replace the vulcanizedb vendored in the plugin repo (`plugin_repo/vendor/github.com/vulcanize/vulcanizedb`) compose:
with the newly updated version
* The entire vendor lib within the vendored vulcanizedb needs to be deleted (`plugin_repo/vendor/github.com/vulcanize/vulcanizedb/vendor`) `./vulcanizedb compose --config=./environments/config_name.toml`
* These complications arise due to this [conflict](https://github.com/golang/go/issues/20481) between `dep` and Go plugins
execute:
`./vulcanizedb execute --config=./environments/config_name.toml`
composeAndExecute:
## Configuration
A .toml config file is specified when executing the command:
`./vulcanizedb composeAndExecute --config=./environments/config_name.toml` `./vulcanizedb composeAndExecute --config=./environments/config_name.toml`
The config provides information for composing a set of transformers: ## Configuration
A .toml config file is specified when executing the commands.
The config provides information for composing a set of transformers from external repositories:
```toml ```toml
[database] [database]

40
documentation/transformers.md Normal file
View File

@ -0,0 +1,40 @@
# Custom transformers
When the capabilities of the generic `contractWatcher` are not sufficient, custom transformers tailored to a specific
purpose can be leveraged.
Individual transformers can be composed together from any number of external repositories and executed as a single process using
the `compose` and `execute` commands or the `composeAndExecute` command.
## Writing custom transformers
For help with writing different types of custom transformers for the `composeAndExecute` set of commands, please see the below:
Storage Transformers
* [Guide](../../staging/libraries/shared/factories/storage/README.md)
* [Example](../../staging/libraries/shared/factories/storage/EXAMPLE.md)
Event Transformers
* [Guide](../../staging/libraries/shared/factories/event/README.md)
* [Example 1](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registar)
* [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registry)
* [Example 3](https://github.com/vulcanize/ens_transformers/tree/master/transformers/resolver)
Contract Transformers
* [Example 1](https://github.com/vulcanize/account_transformers)
* [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/domain_records)
## Preparing custom transformers to work as part of a plugin
To plug in an external transformer we need to:
1. Create a package that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](../staging/libraries/shared/transformer/event_transformer.go#L33)
or [StorageTransformerInitializer](../../staging/libraries/shared/transformer/storage_transformer.go#L31),
or [ContractTransformerInitializer](../../staging/libraries/shared/transformer/contract_transformer.go#L31), respectively
2. Design the transformers to work in the context of their [event](../staging/libraries/shared/watcher/event_watcher.go#L83),
[storage](../../staging/libraries/shared/watcher/storage_watcher.go#L53),
or [contract](../../staging/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes
3. Create db migrations to run against vulcanizeDB so that we can store the transformer output
* Do not `goose fix` the transformer migrations, this is to ensure they are always ran after the core vulcanizedb migrations which are kept in their fixed form
* Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields
* If the base vDB migrations occupy this path as well, they need to be in their `goose fix`ed form
as they are [here](../../staging/db/migrations)
To update a plugin repository with changes to the core vulcanizedb repository, run `dep ensure` to update its dependencies.

7
integration_test/contract_watcher_light_transformer_test.go
View File

@ -49,6 +49,13 @@ var _ = Describe("contractWatcher light transformer", func() {
c, ok := t.Contracts[tusdAddr] c, ok := t.Contracts[tusdAddr]
Expect(ok).To(Equal(true)) Expect(ok).To(Equal(true))
// TODO: Fix this
// This test sometimes randomly fails because
// for some reason the starting block number is not updated from
// its original value (5197514) to the block number (6194632)
// of the earliest header (mocks.MockHeader1) in the repository
// It is not clear how this happens without one of the above insertErrs
// having been thrown and without any errors thrown during the Init() call
Expect(c.StartingBlock).To(Equal(int64(6194632))) Expect(c.StartingBlock).To(Equal(int64(6194632)))
Expect(c.Abi).To(Equal(constants.TusdAbiString)) Expect(c.Abi).To(Equal(constants.TusdAbiString))
Expect(c.Name).To(Equal("TrueUSD")) Expect(c.Name).To(Equal("TrueUSD"))