removing mcd/maker references; delete test_data; update READMEs; use
logrus
This commit is contained in:
Ian Norden
+1
-101
@@ -1,102 +1,2 @@
|
||||
## Plugins
|
||||
|
||||
This directory is for Exporter plugins (.go and .so files) generated by, output from, and linked to from the composeAndExecute command
|
||||
These plugins are generated using information provided in a .toml config file
|
||||
|
||||
The config file requires, at a minimum, the below fields:
|
||||
|
||||
```toml
|
||||
[database]
|
||||
name = "vulcanize_public"
|
||||
hostname = "localhost"
|
||||
user = "vulcanize"
|
||||
password = "vulcanize"
|
||||
port = 5432
|
||||
|
||||
[client]
|
||||
ipcPath = "http://kovan0.vulcanize.io:8545"
|
||||
|
||||
[exporter]
|
||||
name = "exporter"
|
||||
[exporter.transformers]
|
||||
transformer1 = "path/to/transformer1"
|
||||
transformer2 = "path/to/transformer2"
|
||||
transformer3 = "path/to/transformer3"
|
||||
transformer4 = "path/to/transformer4"
|
||||
[exporter.types]
|
||||
transformer1 = "eth_event"
|
||||
transformer2 = "eth_event"
|
||||
transformer3 = "eth_event"
|
||||
transformer4 = "eth_storage"
|
||||
[exporter.repositories]
|
||||
transformers = "github.com/account/repo"
|
||||
transformer4 = "github.com/account2/repo2"
|
||||
[exporter.migrations]
|
||||
transformers = "db/migrations"
|
||||
transformer4 = "to/db/migrations"
|
||||
```
|
||||
- `exporter.transformers` are mappings of import aliases to paths to `TransformerInitializer`s
|
||||
- Import aliases can be arbitrarily named but note that `interface1` is a reserved alias needed for the generic TransformerInitializer type
|
||||
- `exporter.repositores` are the paths to the repositories which contain the transformers
|
||||
- `exporter.migrations` are the relative paths to the db migrations found within the `exporter.repositores`
|
||||
- Migrations need to be located in the repos in `exporter.repositores`
|
||||
- Keys should match the keys for the corresponding repo
|
||||
|
||||
Note: If any of the imported transformer need additional
|
||||
config variables do not forget to include those as well
|
||||
|
||||
This information is used to write and build a go plugin with a transformer
|
||||
set composed from the transformer imports specified in the config file
|
||||
This plugin is loaded and the set of transformer initializers is exported
|
||||
from it and loaded into and executed over by the appropriate watcher
|
||||
|
||||
The type of watcher that the transformer works with is specified using the
|
||||
exporter.types config variable as shown above
|
||||
Currently there are watchers for event data from an eth node (eth_event)
|
||||
and storage data from an eth node (eth_storage)
|
||||
In the future there will be watchers for ipfs (ipfs_event and ipfs_storage)
|
||||
|
||||
Transformers of different types can be ran together in the same command using a
|
||||
single config file or in separate command instances using different config files
|
||||
|
||||
Specify config location when executing the command:
|
||||
`./vulcanizedb composeAndExecute --config=./environments/config_name.toml`
|
||||
|
||||
The general structure of a plugin .go file, and what we would see with the above config is shown below
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
interface1 "github.com/vulcanize/vulcanizedb/libraries/shared/transformer"
|
||||
transformer1 "github.com/path/to/transformer1"
|
||||
transformer2 "github.com/path/to/transformer2"
|
||||
transformer3 "github.com/path/to/transformer3"
|
||||
transformer4 "github.com/different/path/to/transformer1"
|
||||
)
|
||||
|
||||
type exporter string
|
||||
|
||||
var Exporter exporter
|
||||
|
||||
func (e exporter) Export() []interface1.TransformerInitializer, []interface1.StorageTransformerInitializer {
|
||||
return []interface1.TransformerInitializer{
|
||||
transformer1.TransformerInitializer,
|
||||
transformer2.TransformerInitializer,
|
||||
transformer3.TransformerInitializer,
|
||||
}, []interface1.StorageTransformerInitializer{
|
||||
transformer4.StorageTransformerInitializer,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
To plug in an external transformer we need to:
|
||||
* create a [package](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/pkg/autogen/test_helpers/bite/initializer.go)
|
||||
that exports a variable `TransformerInitializer` or `StorageTransformerInitializer` that are of type [TransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/transformer/event_transformer.go#L33)
|
||||
and [StorageTransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/transformer/storage_transformer.go#L31), respectively
|
||||
* design the transformers to work in the context of the [event](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/watcher/event_watcher.go#L83)
|
||||
or [storage](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/watcher/storage_watcher.go#L53) watchers
|
||||
* create db migrations to run against vulcanizeDB so that we can store the transformed data
|
||||
* store the db migrations required for a transformer in the same repository as the transformer(s) that require them
|
||||
* specify their relative paths in that repo in the config, as discussed above
|
||||
* NOTE: due to a bug with plugin migrations, currently need to leave the `exporter.migrations` blank and manually run migrations before running composeAndExecute
|
||||
This empty directory is for Exporter plugins (.go and .so files) written, built, and linked to by the composeAndExecute command
|
||||
|
||||
Reference in New Issue
Block a user