From 01ffc80ef2d32d817f26bf7ef4bdd8df98245fad Mon Sep 17 00:00:00 2001 From: Ian Norden Date: Fri, 19 Apr 2019 10:13:27 -0500 Subject: [PATCH] add some info for separate compose and execute commands; note about flaky test --- README.md | 8 ++- documentation/composeAndExecute.md | 62 +++++++++---------- documentation/transformers.md | 40 ++++++++++++ ...contract_watcher_light_transformer_test.go | 7 +++ 4 files changed, 81 insertions(+), 36 deletions(-) create mode 100644 documentation/transformers.md diff --git a/README.md b/README.md index bbbd626c..440264c1 100644 --- a/README.md +++ b/README.md @@ -108,11 +108,15 @@ These commands are described in detail [here](../staging/documentation/sync.md). ### Data transformation Contract watchers use the raw data that has been synced into Postgres to filter out and apply transformations to specific data of interest. -There is a built-in `contractWatcher` command which provides generic transformation of most contract data. This command is described in detail [here](../staging/documentation/contractWatcher.md). +There is a built-in `contractWatcher` command which provides generic transformation of most contract data. +The `contractWatcher` command is described further [here](../staging/documentation/contractWatcher.md). In many cases a custom transformer or set of transformers will need to be written to provide complete or more comprehensive coverage or to optimize other aspects of the output for a specific end-use. -In this case we have provided the `compose`, `execute`, and `composeAndExecute` commands for running custom transformers from external repositories. This is described in detail [here](../staging/documentation/composeAndExecute.md). +In this case we have provided the `compose`, `execute`, and `composeAndExecute` commands for running custom transformers from external repositories. +Usage of the `compose`, `execute`, and `composeAndExecute` commands is described further [here](../staging/documentation/composeAndExecute.md). + +Documentation on how to build custom transformers to work with these commands can be found [here](../staging/documentation/transformers.md). ## Tests - Replace the empty `ipcPath` in the `environments/infura.toml` with a path to a full node's eth_jsonrpc endpoint (e.g. local geth node ipc path or infura url) diff --git a/documentation/composeAndExecute.md b/documentation/composeAndExecute.md index 6efbcf75..0ac226a9 100644 --- a/documentation/composeAndExecute.md +++ b/documentation/composeAndExecute.md @@ -3,48 +3,42 @@ The `composeAndExecute` command is used to compose and execute over an arbitrary This is accomplished by generating a Go pluggin which allows the `vulcanizedb` binary to link to external transformers, so long as they abide by one of the standard [interfaces](../staging/libraries/shared/transformer). -This command requires Go 1.11+ and [Go plugins](https://golang.org/pkg/plugin/) only work on Unix-based systems. +Additionally, there are separate `compose` and `execute` commands to allow pre-building and linking to a pre-built .so file. -## Writing custom transformers -Storage Transformers - * [Guide](../../staging/libraries/shared/factories/storage/README.md) - * [Example](../../staging/libraries/shared/factories/storage/EXAMPLE.md) - -Event Transformers - * [Guide](../../staging/libraries/shared/factories/event/README.md) - * [Example 1](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registar) - * [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registry) - * [Example 3](https://github.com/vulcanize/ens_transformers/tree/master/transformers/resolver) +**NOTE:** +1. It is necessary that the .so file was built with the same exact dependencies that are present in the execution environment, +i.e. we need to `compose` and `execute` the plugin .so file with the same exact version of vulcanizeDB. +1. The plugin migrations are run during the plugin's composition. As such, if `execute` is used to run a prebuilt .so in a different +environment than the one it was composed in then the migrations for that plugin will first need to be manually ran against that environment's Postgres database. -Contract Transformers - * [Example 1](https://github.com/vulcanize/account_transformers) - * [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/domain_records) +These commands require Go 1.11+ and use [Go plugins](https://golang.org/pkg/plugin/) which only work on Unix-based systems. +There is also an ongoing [conflict](https://github.com/golang/go/issues/20481) between Go plugins and the use vendored dependencies which +imposes certain limitations on how the plugins are built. -## Preparing transformers to work as a plugin for composeAndExecute -To plug in an external transformer we need to: +## Commands +The `compose` and `composeAndExecute` commands assume you are in the vulcanizdb directory located at your system's `$GOPATH`, +and that all of the transformer repositories for building the plugin are present at their `$GOPATH` directories. -1. Create a package that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](../staging/libraries/shared/transformer/event_transformer.go#L33) -or [StorageTransformerInitializer](../../staging/libraries/shared/transformer/storage_transformer.go#L31), -or [ContractTransformerInitializer](../../staging/libraries/shared/transformer/contract_transformer.go#L31), respectively -2. Design the transformers to work in the context of their [event](../staging/libraries/shared/watcher/event_watcher.go#L83), -[storage](../../staging/libraries/shared/watcher/storage_watcher.go#L53), -or [contract](../../staging/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes -3. Create db migrations to run against vulcanizeDB so that we can store the transformer output - * Do not `goose fix` the transformer migrations, this is to ensure they are always ran after the core vulcanizedb migrations which are kept in their fixed form - * Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields - * If the base vDB migrations occupy this path as well, they need to be in their `goose fix`ed form - as they are [here](../../staging/db/migrations) +The `execute` command does not require the plugin transformer dependencies be located in their +`$GOPATH` directories, instead it expects a prebuilt .so file (of the name specified in the config file) +to be in `$GOPATH/src/github.com/vulcanize/vulcanizedb/plugins/` and, as noted above, also expects the plugin +db migrations to have already been ran against the database. -To update a plugin repository with changes to the core vulcanizedb repository, replace the vulcanizedb vendored in the plugin repo (`plugin_repo/vendor/github.com/vulcanize/vulcanizedb`) -with the newly updated version -* The entire vendor lib within the vendored vulcanizedb needs to be deleted (`plugin_repo/vendor/github.com/vulcanize/vulcanizedb/vendor`) -* These complications arise due to this [conflict](https://github.com/golang/go/issues/20481) between `dep` and Go plugins +compose: + +`./vulcanizedb compose --config=./environments/config_name.toml` + +execute: + +`./vulcanizedb execute --config=./environments/config_name.toml` + +composeAndExecute: -## Configuration -A .toml config file is specified when executing the command: `./vulcanizedb composeAndExecute --config=./environments/config_name.toml` -The config provides information for composing a set of transformers: +## Configuration +A .toml config file is specified when executing the commands. +The config provides information for composing a set of transformers from external repositories: ```toml [database] diff --git a/documentation/transformers.md b/documentation/transformers.md new file mode 100644 index 00000000..b9a26ffd --- /dev/null +++ b/documentation/transformers.md @@ -0,0 +1,40 @@ +# Custom transformers +When the capabilities of the generic `contractWatcher` are not sufficient, custom transformers tailored to a specific +purpose can be leveraged. + +Individual transformers can be composed together from any number of external repositories and executed as a single process using +the `compose` and `execute` commands or the `composeAndExecute` command. + +## Writing custom transformers +For help with writing different types of custom transformers for the `composeAndExecute` set of commands, please see the below: + +Storage Transformers + * [Guide](../../staging/libraries/shared/factories/storage/README.md) + * [Example](../../staging/libraries/shared/factories/storage/EXAMPLE.md) + +Event Transformers + * [Guide](../../staging/libraries/shared/factories/event/README.md) + * [Example 1](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registar) + * [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/registry) + * [Example 3](https://github.com/vulcanize/ens_transformers/tree/master/transformers/resolver) + +Contract Transformers + * [Example 1](https://github.com/vulcanize/account_transformers) + * [Example 2](https://github.com/vulcanize/ens_transformers/tree/master/transformers/domain_records) + +## Preparing custom transformers to work as part of a plugin +To plug in an external transformer we need to: + +1. Create a package that exports a variable `TransformerInitializer`, `StorageTransformerInitializer`, or `ContractTransformerInitializer` that are of type [TransformerInitializer](../staging/libraries/shared/transformer/event_transformer.go#L33) +or [StorageTransformerInitializer](../../staging/libraries/shared/transformer/storage_transformer.go#L31), +or [ContractTransformerInitializer](../../staging/libraries/shared/transformer/contract_transformer.go#L31), respectively +2. Design the transformers to work in the context of their [event](../staging/libraries/shared/watcher/event_watcher.go#L83), +[storage](../../staging/libraries/shared/watcher/storage_watcher.go#L53), +or [contract](../../staging/libraries/shared/watcher/contract_watcher.go#L68) watcher execution modes +3. Create db migrations to run against vulcanizeDB so that we can store the transformer output + * Do not `goose fix` the transformer migrations, this is to ensure they are always ran after the core vulcanizedb migrations which are kept in their fixed form + * Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields + * If the base vDB migrations occupy this path as well, they need to be in their `goose fix`ed form + as they are [here](../../staging/db/migrations) + +To update a plugin repository with changes to the core vulcanizedb repository, run `dep ensure` to update its dependencies. diff --git a/integration_test/contract_watcher_light_transformer_test.go b/integration_test/contract_watcher_light_transformer_test.go index 3aaa3267..442d2269 100644 --- a/integration_test/contract_watcher_light_transformer_test.go +++ b/integration_test/contract_watcher_light_transformer_test.go @@ -49,6 +49,13 @@ var _ = Describe("contractWatcher light transformer", func() { c, ok := t.Contracts[tusdAddr] Expect(ok).To(Equal(true)) + // TODO: Fix this + // This test sometimes randomly fails because + // for some reason the starting block number is not updated from + // its original value (5197514) to the block number (6194632) + // of the earliest header (mocks.MockHeader1) in the repository + // It is not clear how this happens without one of the above insertErrs + // having been thrown and without any errors thrown during the Init() call Expect(c.StartingBlock).To(Equal(int64(6194632))) Expect(c.Abi).To(Equal(constants.TusdAbiString)) Expect(c.Name).To(Equal("TrueUSD"))