ipld-eth-server/plugins

Plugins

This directory is for Exporter plugins (.go and .so files) generated by, output from, and linked to from the composeAndExecute command
These plugins are generated using information provided in a .toml config file

The config file requires, at a minimum, the below fields:

[database]
   name = "vulcanize_public"
   hostname = "localhost"
   user = "vulcanize"
   password = "vulcanize"
   port = 5432

[client]
   ipcPath = "http://kovan0.vulcanize.io:8545"

[exporter]
   name = "exporter"
   [exporter.transformers]
           transformer1 = "path/to/transformer1"
           transformer2 = "path/to/transformer2"
           transformer3 = "path/to/transformer3"
           transformer4 = "path/to/transformer4"
[exporter.types]
           transformer1 = "eth_event"
           transformer2 = "eth_event"
           transformer3 = "eth_event"
           transformer4 = "eth_storage"
   [exporter.repositories]
           transformers = "github.com/account/repo"
           transformer4 = "github.com/account2/repo2"
   [exporter.migrations]
           transformers = "db/migrations"
           transformer4 = "to/db/migrations"

• exporter.transformers are mappings of import aliases to paths to TransformerInitializers
  • Import aliases can be arbitrarily named but note that interface1 is a reserved alias needed for the generic TransformerInitializer type
• exporter.repositores are the paths to the repositories which contain the transformers
• exporter.migrations are the relative paths to the db migrations found within the exporter.repositores
  • Migrations need to be located in the repos in exporter.repositores
  • Keys should match the keys for the corresponding repo

Note: If any of the imported transformer need additional config variables do not forget to include those as well

This information is used to write and build a go plugin with a transformer set composed from the transformer imports specified in the config file
This plugin is loaded and the set of transformer initializers is exported from it and loaded into and executed over by the appropriate watcher

The type of watcher that the transformer works with is specified using the exporter.types config variable as shown above
Currently there are watchers for event data from an eth node (eth_event) and storage data from an eth node (eth_storage)
In the future there will be watchers for ipfs (ipfs_event and ipfs_storage)

Transformers of different types can be ran together in the same command using a single config file or in separate command instances using different config files

Specify config location when executing the command:
./vulcanizedb composeAndExecute --config=./environments/config_name.toml

The general structure of a plugin .go file, and what we would see with the above config is shown below

package main

import (
	interface1 "github.com/vulcanize/vulcanizedb/libraries/shared/transformer"
	transformer1 "github.com/path/to/transformer1"
	transformer2 "github.com/path/to/transformer2"
	transformer3 "github.com/path/to/transformer3"
	transformer4 "github.com/different/path/to/transformer1"
)

type exporter string

var Exporter exporter

func (e exporter) Export() []interface1.TransformerInitializer, []interface1.StorageTransformerInitializer {
	return []interface1.TransformerInitializer{
		transformer1.TransformerInitializer,
		transformer2.TransformerInitializer,
		transformer3.TransformerInitializer,
	},     []interface1.StorageTransformerInitializer{
		transformer4.StorageTransformerInitializer,
    }
}

To plug in an external transformer we need to:

• create a package that exports a variable TransformerInitializer or StorageTransformerInitializer that are of type TransformerInitializer
  and StorageTransformerInitializer, respectively
• design the transformers to work in the context of the event or storage watchers
• create db migrations to run against vulcanizeDB so that we can store the transformed data
  • store the db migrations required for a transformer in the same repository as the transformer(s) that require them
  • specify their relative paths in that repo in the config, as discussed above
  • NOTE: due to a bug with plugin migrations, currently need to leave the exporter.migrations blank and manually run migrations before running composeAndExecute