add some info for separate compose and execute commands; note about
flaky test
This commit is contained in:
parent
9c06e9387c
commit
01ffc80ef2
@ -108,11 +108,15 @@ These commands are described in detail [here](../staging/documentation/sync.md).
|
|||||||
### Data transformation
|
### Data transformation
|
||||||
Contract watchers use the raw data that has been synced into Postgres to filter out and apply transformations to specific data of interest.
|
Contract watchers use the raw data that has been synced into Postgres to filter out and apply transformations to specific data of interest.
|
||||||
|
|
||||||
There is a built-in `contractWatcher` command which provides generic transformation of most contract data. This command is described in detail [here](../staging/documentation/contractWatcher.md).
|
There is a built-in `contractWatcher` command which provides generic transformation of most contract data.
|
||||||
|
The `contractWatcher` command is described further [here](../staging/documentation/contractWatcher.md).
|
||||||
|
|
||||||
In many cases a custom transformer or set of transformers will need to be written to provide complete or more comprehensive coverage or to optimize other aspects of the output for a specific end-use.
|
In many cases a custom transformer or set of transformers will need to be written to provide complete or more comprehensive coverage or to optimize other aspects of the output for a specific end-use.
|
||||||
In this case we have provided the `compose`, `execute`, and `composeAndExecute` commands for running custom transformers from external repositories. This is described in detail [here](../staging/documentation/composeAndExecute.md).
|
In this case we have provided the `compose`, `execute`, and `composeAndExecute` commands for running custom transformers from external repositories.
|
||||||
|
|
||||||
|
Usage of the `compose`, `execute`, and `composeAndExecute` commands is described further [here](../staging/documentation/composeAndExecute.md).
|
||||||
|
|
||||||
|
Documentation on how to build custom transformers to work with these commands can be found [here](../staging/documentation/transformers.md).
|
||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
- Replace the empty `ipcPath` in the `environments/infura.toml` with a path to a full node's eth_jsonrpc endpoint (e.g. local geth node ipc path or infura url)
|
- Replace the empty `ipcPath` in the `environments/infura.toml` with a path to a full node's eth_jsonrpc endpoint (e.g. local geth node ipc path or infura url)
|
||||||
|
@ -3,48 +3,42 @@ The `composeAndExecute` command is used to compose and execute over an arbitrary
|
|||||||
This is accomplished by generating a Go pluggin which allows the `vulcanizedb` binary to link to external transformers, so
|
This is accomplished by generating a Go pluggin which allows the `vulcanizedb` binary to link to external transformers, so
|
||||||
long as they abide by one of the standard [interfaces](../staging/libraries/shared/transformer).
|
long as they abide by one of the standard [interfaces](../staging/libraries/shared/transformer).
|
||||||
|
|
||||||
This command requires Go 1.11+ and [Go plugins](https://golang.org/pkg/plugin/) only work on Unix-based systems.
|
Additionally, there are separate `compose` and `execute` commands to allow pre-building and linking to a pre-built .so file.
|
||||||
|
|
||||||
## Writing custom transformers
|
**NOTE:**
|
||||||
Storage Transformers
|
1. It is necessary that the .so file was built with the same exact dependencies that are present in the execution environment,
|
||||||
* [Guide](../../staging/libraries/shared/factories/storage/README.md)
|
i.e. we need to `compose` and `execute` the plugin .so file with the same exact version of vulcanizeDB.
|
||||||
* [Example](../../staging/libraries/shared/factories/storage/EXAMPLE.md)
|
1. The plugin migrations are run during the plugin's composition. As such, if `execute` is used to run a prebuilt .so in a different
|
||||||
|
environment than the one it was composed in then the migrations for that plugin will first need to be manually ran against that environment's Postgres database.
|
||||||
|
|
||||||
Event Transformers
|
These commands require Go 1.11+ and use [Go plugins](https://golang.org/pkg/plugin/) which only work on Unix-based systems.
|
||||||
* [Guide](../../staging/libraries/shared/factories/event/README.md)
|
There is also an ongoing [conflict](https://github.com/golang/go/issues/20481) between Go plugins and the use vendored dependencies which
|
||||||
* [Example 1](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registar)
|
imposes certain limitations on how the plugins are built.
|
||||||
* [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registry)
|
|
||||||
* [Example 3](https://github.com/vulcanize/ens_transformers/tree/master/transformers/resolver)
|
|
||||||
|
|
||||||
Contract Transformers
|
## Commands
|
||||||
* [Example 1](https://github.com/vulcanize/account_transformers)
|
The `compose` and `composeAndExecute` commands assume you are in the vulcanizdb directory located at your system's `$GOPATH`,
|
||||||
* [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/domain_records)
|
and that all of the transformer repositories for building the plugin are present at their `$GOPATH` directories.
|
||||||
|
|
||||||
## Preparing transformers to work as a plugin for composeAndExecute
|
The `execute` command does not require the plugin transformer dependencies be located in their
|
||||||
To plug in an external transformer we need to:
|
`$GOPATH` directories, instead it expects a prebuilt .so file (of the name specified in the config file)
|
||||||
|
to be in `$GOPATH/src/github.com/vulcanize/vulcanizedb/plugins/` and, as noted above, also expects the plugin
|
||||||
|
db migrations to have already been ran against the database.
|
||||||
|
|
||||||
1. Create a package that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](../staging/libraries/shared/transformer/event_transformer.go#L33)
|
compose:
|
||||||
or [StorageTransformerInitializer](../../staging/libraries/shared/transformer/storage_transformer.go#L31),
|
|
||||||
or [ContractTransformerInitializer](../../staging/libraries/shared/transformer/contract_transformer.go#L31), respectively
|
|
||||||
2. Design the transformers to work in the context of their [event](../staging/libraries/shared/watcher/event_watcher.go#L83),
|
|
||||||
[storage](../../staging/libraries/shared/watcher/storage_watcher.go#L53),
|
|
||||||
or [contract](../../staging/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes
|
|
||||||
3. Create db migrations to run against vulcanizeDB so that we can store the transformer output
|
|
||||||
* Do not `goose fix` the transformer migrations, this is to ensure they are always ran after the core vulcanizedb migrations which are kept in their fixed form
|
|
||||||
* Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields
|
|
||||||
* If the base vDB migrations occupy this path as well, they need to be in their `goose fix`ed form
|
|
||||||
as they are [here](../../staging/db/migrations)
|
|
||||||
|
|
||||||
To update a plugin repository with changes to the core vulcanizedb repository, replace the vulcanizedb vendored in the plugin repo (`plugin_repo/vendor/github.com/vulcanize/vulcanizedb`)
|
`./vulcanizedb compose --config=./environments/config_name.toml`
|
||||||
with the newly updated version
|
|
||||||
* The entire vendor lib within the vendored vulcanizedb needs to be deleted (`plugin_repo/vendor/github.com/vulcanize/vulcanizedb/vendor`)
|
execute:
|
||||||
* These complications arise due to this [conflict](https://github.com/golang/go/issues/20481) between `dep` and Go plugins
|
|
||||||
|
`./vulcanizedb execute --config=./environments/config_name.toml`
|
||||||
|
|
||||||
|
composeAndExecute:
|
||||||
|
|
||||||
## Configuration
|
|
||||||
A .toml config file is specified when executing the command:
|
|
||||||
`./vulcanizedb composeAndExecute --config=./environments/config_name.toml`
|
`./vulcanizedb composeAndExecute --config=./environments/config_name.toml`
|
||||||
|
|
||||||
The config provides information for composing a set of transformers:
|
## Configuration
|
||||||
|
A .toml config file is specified when executing the commands.
|
||||||
|
The config provides information for composing a set of transformers from external repositories:
|
||||||
|
|
||||||
```toml
|
```toml
|
||||||
[database]
|
[database]
|
||||||
|
40
documentation/transformers.md
Normal file
40
documentation/transformers.md
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
# Custom transformers
|
||||||
|
When the capabilities of the generic `contractWatcher` are not sufficient, custom transformers tailored to a specific
|
||||||
|
purpose can be leveraged.
|
||||||
|
|
||||||
|
Individual transformers can be composed together from any number of external repositories and executed as a single process using
|
||||||
|
the `compose` and `execute` commands or the `composeAndExecute` command.
|
||||||
|
|
||||||
|
## Writing custom transformers
|
||||||
|
For help with writing different types of custom transformers for the `composeAndExecute` set of commands, please see the below:
|
||||||
|
|
||||||
|
Storage Transformers
|
||||||
|
* [Guide](../../staging/libraries/shared/factories/storage/README.md)
|
||||||
|
* [Example](../../staging/libraries/shared/factories/storage/EXAMPLE.md)
|
||||||
|
|
||||||
|
Event Transformers
|
||||||
|
* [Guide](../../staging/libraries/shared/factories/event/README.md)
|
||||||
|
* [Example 1](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registar)
|
||||||
|
* [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registry)
|
||||||
|
* [Example 3](https://github.com/vulcanize/ens_transformers/tree/master/transformers/resolver)
|
||||||
|
|
||||||
|
Contract Transformers
|
||||||
|
* [Example 1](https://github.com/vulcanize/account_transformers)
|
||||||
|
* [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/domain_records)
|
||||||
|
|
||||||
|
## Preparing custom transformers to work as part of a plugin
|
||||||
|
To plug in an external transformer we need to:
|
||||||
|
|
||||||
|
1. Create a package that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](../staging/libraries/shared/transformer/event_transformer.go#L33)
|
||||||
|
or [StorageTransformerInitializer](../../staging/libraries/shared/transformer/storage_transformer.go#L31),
|
||||||
|
or [ContractTransformerInitializer](../../staging/libraries/shared/transformer/contract_transformer.go#L31), respectively
|
||||||
|
2. Design the transformers to work in the context of their [event](../staging/libraries/shared/watcher/event_watcher.go#L83),
|
||||||
|
[storage](../../staging/libraries/shared/watcher/storage_watcher.go#L53),
|
||||||
|
or [contract](../../staging/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes
|
||||||
|
3. Create db migrations to run against vulcanizeDB so that we can store the transformer output
|
||||||
|
* Do not `goose fix` the transformer migrations, this is to ensure they are always ran after the core vulcanizedb migrations which are kept in their fixed form
|
||||||
|
* Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields
|
||||||
|
* If the base vDB migrations occupy this path as well, they need to be in their `goose fix`ed form
|
||||||
|
as they are [here](../../staging/db/migrations)
|
||||||
|
|
||||||
|
To update a plugin repository with changes to the core vulcanizedb repository, run `dep ensure` to update its dependencies.
|
@ -49,6 +49,13 @@ var _ = Describe("contractWatcher light transformer", func() {
|
|||||||
c, ok := t.Contracts[tusdAddr]
|
c, ok := t.Contracts[tusdAddr]
|
||||||
Expect(ok).To(Equal(true))
|
Expect(ok).To(Equal(true))
|
||||||
|
|
||||||
|
// TODO: Fix this
|
||||||
|
// This test sometimes randomly fails because
|
||||||
|
// for some reason the starting block number is not updated from
|
||||||
|
// its original value (5197514) to the block number (6194632)
|
||||||
|
// of the earliest header (mocks.MockHeader1) in the repository
|
||||||
|
// It is not clear how this happens without one of the above insertErrs
|
||||||
|
// having been thrown and without any errors thrown during the Init() call
|
||||||
Expect(c.StartingBlock).To(Equal(int64(6194632)))
|
Expect(c.StartingBlock).To(Equal(int64(6194632)))
|
||||||
Expect(c.Abi).To(Equal(constants.TusdAbiString))
|
Expect(c.Abi).To(Equal(constants.TusdAbiString))
|
||||||
Expect(c.Name).To(Equal("TrueUSD"))
|
Expect(c.Name).To(Equal("TrueUSD"))
|
||||||
|
Loading…
Reference in New Issue
Block a user