ipld-eth-server/README.md at 55fa9b8364329dd669b697bbc6a7b44b275e86b3

Ian Norden 55fa9b8364 auto management of dependencies temporarily needed to build plugin and transformer

migrations; new test showing it is working with external transformers runs
properly when migrations were manually performed but still need
to test automated migration management

2019-02-24 15:23:35 -06:00

3.6 KiB

Raw Blame History

Plugins

This directory is for Exporter plugins (.go and .so files) generated by, output from, and linked to from the composeAndExecute command
These plugins are generated using information provided in a .toml config file

The config file requires, at a minimum, the below fields:

[database]
    name = "vulcanize_public"
    hostname = "localhost"
    user = "vulcanize"
    password = "vulcanize"
    port = 5432

[client]
    ipcPath = "http://kovan0.vulcanize.io:8545"

[exporter]
    filePath = "$GOPATH/src/github.com/vulcanize/vulcanizedb/plugins/"
    fileName = "exporter"
    [exporter.transformers]
            transformer1 = "github.com/path/to/transformer1"
            transformer2 = "github.com/path/to/transformer2"
            transformer3 = "github.com/path/to/transformer3"
            transformer4 = "github.com/different/path/to/transformer1"
    [exporter.repositories]
            transformers = "github.com/path/to"
            transformer4 = "github.com/different/path
    [exporter.migrations]
            transformers = "db/migrations"
            transformer4 = "to/db/migrations"

exporter.transformers are mappings of import aliases to paths to TransformerInitializers
- Import aliases can be arbitrarily named but note that interface1 is a reserved alias needed for the generic TransformerInitializer type
exporter.repositores are the paths to the repositories which contain the transformers
exporter.migrations are the relative paths to the db migrations found within the exporter.repositores
- Migrations need to be located in the repos in exporter.repositores
- Keys should match the keys for the corresponding repo

If the individual transformers require additional configuration variables be sure to include them in the .toml file

The general structure of a plugin .go file, and what we would see with the above config is shown below

package main

import (
	interface1 "github.com/vulcanize/vulcanizedb/libraries/shared/transformer"
	transformer1 "github.com/path/to/transformer1"
	transformer2 "github.com/path/to/transformer2"
	transformer3 "github.com/path/to/transformer3"
	transformer4 "github.com/different/path/to/transformer1"
)

type exporter string

var Exporter exporter

func (e exporter) Export() []interface1.TransformerInitializer {
	return []interface1.TransformerInitializer{
		transformer1.TransformerInitializer,
		transformer2.TransformerInitializer,
		transformer3.TransformerInitializer,
		transformer4.TransformerInitializer,
	}
}

As such, to plug in an external transformer we need to create a package that exports a variable TransformerInitializer that is of type TransformerInitializer
As long as the imported transformers abide by the required interfaces, we can execute over any arbitrary set of them
Note: currently the transformers must also operate using the watcher's execution mode

For each transformer we will also need to create db migrations to run against vulcanizeDB so that we can store the transformed data
The migrations needed for a specific transformer need to be included in the same repository as the transformers that require them, and their relative paths in that repo specified in the config as discussed above

3.6 KiB Raw Blame History

Plugins

3.6 KiB

Raw Blame History