.. | ||
example_maker_exporter | ||
README.md |
Plugins
This directory is for Exporter plugins (.go and .so files) generated by, output from, and linked to from the composeAndExecute command
These plugins are generated using information provided in a .toml config file
The config file requires, at a minimum, the below fields:
[database]
name = "vulcanize_public"
hostname = "localhost"
user = "vulcanize"
password = "vulcanize"
port = 5432
[client]
ipcPath = "http://kovan0.vulcanize.io:8545"
[exporter]
name = "exporter"
[exporter.transformers]
transformer1 = "path/to/transformer1"
transformer2 = "path/to/transformer2"
transformer3 = "path/to/transformer3"
transformer4 = "path/to/transformer4"
[exporter.types]
transformer1 = "eth_event"
transformer2 = "eth_event"
transformer3 = "eth_event"
transformer4 = "eth_storage"
[exporter.repositories]
transformers = "github.com/account/repo"
transformer4 = "github.com/account2/repo2"
[exporter.migrations]
transformers = "db/migrations"
transformer4 = "to/db/migrations"
exporter.transformers
are mappings of import aliases to paths toTransformerInitializer
s- Import aliases can be arbitrarily named but note that
interface1
is a reserved alias needed for the generic TransformerInitializer type
- Import aliases can be arbitrarily named but note that
exporter.repositores
are the paths to the repositories which contain the transformersexporter.migrations
are the relative paths to the db migrations found within theexporter.repositores
- Migrations need to be located in the repos in
exporter.repositores
- Keys should match the keys for the corresponding repo
- Migrations need to be located in the repos in
Note: If any of the imported transformer need additional config variables do not forget to include those as well
This information is used to write and build a go plugin with a transformer
set composed from the transformer imports specified in the config file
This plugin is loaded and the set of transformer initializers is exported
from it and loaded into and executed over by the appropriate watcher
The type of watcher that the transformer works with is specified using the
exporter.types config variable as shown above
Currently there are watchers for event data from an eth node (eth_event)
and storage data from an eth node (eth_storage)
In the future there will be watchers for ipfs (ipfs_event and ipfs_storage)
Transformers of different types can be ran together in the same command using a single config file or in separate command instances using different config files
Specify config location when executing the command:
./vulcanizedb composeAndExecute --config=./environments/config_name.toml
The general structure of a plugin .go file, and what we would see with the above config is shown below
package main
import (
interface1 "github.com/vulcanize/vulcanizedb/libraries/shared/transformer"
transformer1 "github.com/path/to/transformer1"
transformer2 "github.com/path/to/transformer2"
transformer3 "github.com/path/to/transformer3"
transformer4 "github.com/different/path/to/transformer1"
)
type exporter string
var Exporter exporter
func (e exporter) Export() []interface1.TransformerInitializer, []interface1.StorageTransformerInitializer {
return []interface1.TransformerInitializer{
transformer1.TransformerInitializer,
transformer2.TransformerInitializer,
transformer3.TransformerInitializer,
}, []interface1.StorageTransformerInitializer{
transformer4.StorageTransformerInitializer,
}
}
To plug in an external transformer we need to:
- create a package
that exports a variable
TransformerInitializer
orStorageTransformerInitializer
that are of type TransformerInitializer
and StorageTransformerInitializer, respectively - design the transformers to work in the context of the event or storage watchers
- create db migrations to run against vulcanizeDB so that we can store the transformed data
- store the db migrations required for a transformer in the same repository as the transformer(s) that require them
- specify their relative paths in that repo in the config, as discussed above
- NOTE: due to a bug with plugin migrations, currently need to leave the
exporter.migrations
blank and manually run migrations before running composeAndExecute