cerc-io/ipld-eth-server
18
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity

compose and execute as separate commands; update README and add git checkout vendor/github.com/ethereum/go-ethereum/accounts/abi to make build

Browse Source
This commit is contained in:
Ian Norden 2019-02-28 11:51:54 -06:00
parent 2c5ddd03ba
commit 83f7daf37d
10 changed files with 377 additions and 116 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -13,3 +13,4 @@ postgraphile/node_modules/
postgraphile/package-lock.json
vulcanizedb.log
db/migrations/20*.sql
plugins/*.so

2
Gopkg.lock generated
View File

@ -573,6 +573,7 @@
"github.com/ethereum/go-ethereum/params",
"github.com/ethereum/go-ethereum/rpc",
"github.com/hashicorp/golang-lru",
"github.com/hpcloud/tail",
"github.com/jmoiron/sqlx",
"github.com/lib/pq",
"github.com/mitchellh/go-homedir",
@ -585,6 +586,7 @@
"golang.org/x/net/context",
"golang.org/x/sync/errgroup",
"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer",
"gopkg.in/tomb.v1",
]
solver-name = "gps-cdcl"
solver-version = 1

3
Makefile
View File

@ -63,6 +63,7 @@ dep: | $(DEP)
$(DEP) ensure
build: dep
git checkout vendor/github.com/ethereum/go-ethereum/accounts/abi
go fmt ./...
go build
@ -158,4 +159,4 @@ rinkeby_env_migrate:
.PHONY: rinkeby_env_down
rinkeby_env_down:
docker-compose -f $(RINKEBY_COMPOSE_FILE) down
docker-compose -f $(RINKEBY_COMPOSE_FILE) down

24
README.md
View File

@ -10,7 +10,7 @@ Vulcanize DB is a set of tools that make it easier for developers to write appli
## Dependencies
- Go 1.11+
- Postgres 10
- Postgres 10.6
- Ethereum Node
- [Go Ethereum](https://ethereum.github.io/go-ethereum/downloads/) (1.8.21+)
- [Parity 1.8.11+](https://github.com/paritytech/parity/releases)
@ -41,6 +41,8 @@ In order to install packages with `dep`, ensure you are in the project directory
After `dep` finishes, dependencies should be installed within your `GOPATH` at the versions specified in `Gopkg.toml`.
Because we are working with a modified version of the go-ethereum accounts/abi package, after running `dep ensure` you will need to run `git checkout vendor/github/ethereum/go-ethereum/accounts/abi` to checkout the modified dependency
Lastly, ensure that `GOPATH` is defined in your shell. If necessary, `GOPATH` can be set in `~/.bashrc` or `~/.bash_profile`, depending upon your system. It can be additionally helpful to add `$GOPATH/bin` to your shell's `$PATH`.
### Setting up the Database
@ -207,7 +209,7 @@ It produces and populates a schema with three tables:
`light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.mint_event`
`light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.balanceof_method`
Column ids and types for these tables are generated based on the event and method argument names and types and method return types, resulting in tables such as
Column ids and types for these tables are generated based on the event and method argument names and types and method return types, resulting in tables such as:
Table "light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.transfer_event"
@ -223,7 +225,6 @@ Table "light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.transfer_event"
| to_ | character varying(66) | | not null | | extended | | |
| value_ | numeric | | not null | | main | | |
and
Table "light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.balanceof_method"
@ -303,7 +304,7 @@ The config provides information for composing a set of transformers:
migrations = "to/db/migrations"
```
- `home` is the name of the package you are building the plugin for, in most cases this is github.com/vulcanize/vulcanizedb
- `clone` this signifies whether or not to retrieve transformer packages by cloning them; by default we attempt to work with transformer packages located in
- `clone` this signifies whether or not to retrieve plugin transformer packages by `git clone`ing them; by default we attempt to work with transformer packages located in
our `$GOPATH` but setting this to `true` overrides that. This needs to be set to `true` for the configs used in tests in order for them to work with Travis.
- `name` is the name used for the plugin files (.so and .go)
- `save` indicates whether or not the user wants to save the .go file instead of removing it after .so compilation. Sometimes useful for debugging/trouble-shooting purposes.
@ -355,11 +356,16 @@ func (e exporter) Export() []interface1.TransformerInitializer, []interface1.Sto
#### Preparing transformer(s) to work as pluggins for composeAndExecute
To plug in an external transformer we need to:
* create a [package](https://github.com/vulcanize/mcd_transformers/blob/staging/transformers/bite/initializer/initializer.go)
* Create a [package](https://github.com/vulcanize/mcd_transformers/blob/staging/transformers/bite/initializer/initializer.go)
that exports a variable `TransformerInitializer` or `StorageTransformerInitializer` that are of type [TransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/transformer/event_transformer.go#L33)
or [StorageTransformerInitializer](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/transformer/storage_transformer.go#L31), respectively
* design the transformers to work in the context of their [event](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/watcher/event_watcher.go#L83)
* Design the transformers to work in the context of their [event](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/watcher/event_watcher.go#L83)
or [storage](https://github.com/vulcanize/maker-vulcanizedb/blob/compose_and_execute/libraries/shared/watcher/storage_watcher.go#L53) watchers
* create db migrations to run against vulcanizeDB so that we can store the transformer output
* specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields
* do not `goose fix` the transformer migrations
* Create db migrations to run against vulcanizeDB so that we can store the transformer output
* Specify migration locations for each transformer in the config with the `exporter.transformer.migrations` fields
* Do not `goose fix` the transformer migrations
To update a plugin repository with changes to the core vulcanizedb repository, replace the vulcanizedb vendored in the plugin repo (`plugin_repo/vendor/github.com/vulcanize/vulcanizedb`)
with the newly updated version
* The entire vendor lib within the vendored vulcanizedb needs to be deleted (`plugin_repo/vendor/github.com/vulcanize/vulcanizedb/vendor`)
* These complications arise due to this [conflict](https://github.com/golang/go/issues/20481) between `dep` and Go plugins

173
cmd/compose.go Normal file
View File

@ -0,0 +1,173 @@
// Copyright © 2019 Vulcanize, Inc
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"errors"
"fmt"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/vulcanize/vulcanizedb/pkg/config"
p2 "github.com/vulcanize/vulcanizedb/pkg/plugin"
)
// composeCmd represents the compose command
var composeCmd = &cobra.Command{
Use: "compose",
Short: "Composes transformer initializer plugin",
Long: `This command needs a config .toml file of form:
[database]
name = "vulcanize_public"
hostname = "localhost"
user = "vulcanize"
password = "vulcanize"
port = 5432
[client]
ipcPath = "http://kovan0.vulcanize.io:8545"
[exporter]
home = "github.com/vulcanize/vulcanizedb"
clone = false
name = "exampleTransformerExporter"
save = false
transformerNames = [
"transformer1",
"transformer2",
"transformer3",
"transformer4",
]
[exporter.transformer1]
path = "path/to/transformer1"
type = "eth_event"
repository = "github.com/account/repo"
migrations = "db/migrations"
[exporter.transformer2]
path = "path/to/transformer2"
type = "eth_event"
repository = "github.com/account/repo"
migrations = "db/migrations"
[exporter.transformer3]
path = "path/to/transformer3"
type = "eth_event"
repository = "github.com/account/repo"
migrations = "db/migrations"
[exporter.transformer4]
path = "path/to/transformer4"
type = "eth_storage"
repository = "github.com/account2/repo2"
migrations = "to/db/migrations"
Note: If any of the plugin transformer need additional
configuration variables include them in the .toml file as well
This information is used to write and build a go plugin with a transformer
set composed from the transformer imports specified in the config file
This plugin is loaded and the set of transformer initializers is exported
from it and loaded into and executed over by the appropriate watcher.
The type of watcher that the transformer works with is specified using the
type variable for each transformer in the config. Currently there are watchers
of event data from an eth node (eth_event) and storage data from an eth node
(eth_storage).
Transformers of different types can be ran together in the same command using a
single config file or in separate command instances using different config files
Specify config location when executing the command:
./vulcanizedb compose --config=./environments/config_name.toml`,
Run: func(cmd *cobra.Command, args []string) {
compose()
},
}
func compose() {
// Build plugin generator config
prepConfig()
// Generate code to build the plugin according to the config file
log.Info("generating plugin")
generator, err := p2.NewGenerator(genConfig, databaseConfig)
if err != nil {
log.Fatal(err)
}
err = generator.GenerateExporterPlugin()
if err != nil {
log.Debug("generating plugin failed")
log.Fatal(err)
}
// TODO: Embed versioning info in the .so files so we know which version of vulcanizedb to run them with
_, pluginPath, err := genConfig.GetPluginPaths()
if err != nil {
log.Fatal(err)
}
fmt.Printf("Composed plugin %s", pluginPath)
log.Info("plugin .so file output to", pluginPath)
}
func init() {
rootCmd.AddCommand(composeCmd)
}
func prepConfig() {
log.Info("configuring plugin")
names := viper.GetStringSlice("exporter.transformerNames")
transformers := make(map[string]config.Transformer)
for _, name := range names {
transformer := viper.GetStringMapString("exporter." + name)
p, ok := transformer["path"]
if !ok || p == "" {
log.Fatal(fmt.Sprintf("%s transformer config is missing `path` value", name))
}
r, ok := transformer["repository"]
if !ok || r == "" {
log.Fatal(fmt.Sprintf("%s transformer config is missing `repository` value", name))
}
m, ok := transformer["migrations"]
if !ok || m == "" {
log.Fatal(fmt.Sprintf("%s transformer config is missing `migrations` value", name))
}
t, ok := transformer["type"]
if !ok {
log.Fatal(fmt.Sprintf("%s transformer config is missing `type` value", name))
}
transformerType := config.GetTransformerType(t)
if transformerType == config.UnknownTransformerType {
log.Fatal(errors.New(`unknown transformer type in exporter config accepted types are "eth_event", "eth_storage"`))
}
transformers[name] = config.Transformer{
Path: p,
Type: transformerType,
RepositoryPath: r,
MigrationPath: m,
}
}
genConfig = config.Plugin{
Transformers: transformers,
FilePath: "$GOPATH/src/github.com/vulcanize/vulcanizedb/plugins",
FileName: viper.GetString("exporter.name"),
Save: viper.GetBool("exporter.save"),
Home: viper.GetString("exporter.home"),
Clone: viper.GetBool("exporter.clone"),
}
}

108
cmd/composeAndExecute.go
View File

@ -16,28 +16,19 @@
package cmd
import (
"errors"
"fmt"
"os"
"plugin"
syn "sync"
"time"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/vulcanize/vulcanizedb/libraries/shared/constants"
"github.com/vulcanize/vulcanizedb/libraries/shared/transformer"
"github.com/vulcanize/vulcanizedb/libraries/shared/watcher"
"github.com/vulcanize/vulcanizedb/pkg/config"
"github.com/vulcanize/vulcanizedb/pkg/fs"
p2 "github.com/vulcanize/vulcanizedb/pkg/plugin"
"github.com/vulcanize/vulcanizedb/pkg/plugin/helpers"
"github.com/vulcanize/vulcanizedb/utils"
"os"
"plugin"
syn "sync"
)
// executePluginCmd represents the execute command
// composeAndExecuteCmd represents the composeAndExecute command
var composeAndExecuteCmd = &cobra.Command{
Use: "composeAndExecute",
Short: "Composes, loads, and executes transformer initializer plugin",
@ -86,8 +77,8 @@ var composeAndExecuteCmd = &cobra.Command{
migrations = "to/db/migrations"
Note: If any of the imported transformer need additional
config variables do not forget to include those as well
Note: If any of the plugin transformer need additional
configuration variables include them in the .toml file as well
This information is used to write and build a go plugin with a transformer
set composed from the transformer imports specified in the config file
@ -133,10 +124,10 @@ func composeAndExecute() {
if !genConfig.Save {
defer helpers.ClearFiles(pluginPath)
}
log.Info("opening plugin")
log.Info("linking plugin", pluginPath)
plug, err := plugin.Open(pluginPath)
if err != nil {
log.Debug("opening pluggin failed")
log.Debug("linking plugin failed")
log.Fatal(err)
}
@ -182,90 +173,7 @@ func composeAndExecute() {
wg.Wait()
}
type Exporter interface {
Export() ([]transformer.TransformerInitializer, []transformer.StorageTransformerInitializer)
}
func init() {
rootCmd.AddCommand(composeAndExecuteCmd)
composeAndExecuteCmd.Flags().BoolVar(&recheckHeadersArg, "recheckHeaders", false, "checks headers that are already checked for each transformer.")
}
func watchEthEvents(w *watcher.EventWatcher, wg *syn.WaitGroup) {
defer wg.Done()
// Execute over the TransformerInitializer set using the watcher
log.Info("executing event transformers")
var recheck constants.TransformerExecution
if recheckHeadersArg {
recheck = constants.HeaderRecheck
} else {
recheck = constants.HeaderMissing
}
ticker := time.NewTicker(pollingInterval)
defer ticker.Stop()
for range ticker.C {
err := w.Execute(recheck)
if err != nil {
// TODO Handle watcher errors in composeAndExecute
}
}
}
func watchEthStorage(w *watcher.StorageWatcher, wg *syn.WaitGroup) {
defer wg.Done()
// Execute over the TransformerInitializer set using the watcher
log.Info("executing storage transformers")
ticker := time.NewTicker(pollingInterval)
defer ticker.Stop()
for range ticker.C {
err := w.Execute()
if err != nil {
// TODO Handle watcher errors in composeAndExecute
}
}
}
func prepConfig() {
log.Info("configuring plugin")
names := viper.GetStringSlice("exporter.transformerNames")
transformers := make(map[string]config.Transformer)
for _, name := range names {
transformer := viper.GetStringMapString("exporter." + name)
p, ok := transformer["path"]
if !ok || p == "" {
log.Fatal(fmt.Sprintf("%s transformer config is missing `path` value", name))
}
r, ok := transformer["repository"]
if !ok || r == "" {
log.Fatal(fmt.Sprintf("%s transformer config is missing `repository` value", name))
}
m, ok := transformer["migrations"]
if !ok || m == "" {
log.Fatal(fmt.Sprintf("%s transformer config is missing `migrations` value", name))
}
t, ok := transformer["type"]
if !ok {
log.Fatal(fmt.Sprintf("%s transformer config is missing `type` value", name))
}
transformerType := config.GetTransformerType(t)
if transformerType == config.UnknownTransformerType {
log.Fatal(errors.New(`unknown transformer type in exporter config accepted types are "eth_event", "eth_storage"`))
}
transformers[name] = config.Transformer{
Path: p,
Type: transformerType,
RepositoryPath: r,
MigrationPath: m,
}
}
genConfig = config.Plugin{
Transformers: transformers,
FilePath: "$GOPATH/src/github.com/vulcanize/vulcanizedb/plugins",
FileName: viper.GetString("exporter.name"),
Save: viper.GetBool("exporter.save"),
Home: viper.GetString("exporter.home"),
Clone: viper.GetBool("exporter.clone"),
}
}

170
cmd/execute.go Normal file
View File

@ -0,0 +1,170 @@
// Copyright © 2019 Vulcanize, Inc
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"fmt"
"os"
"plugin"
syn "sync"
"time"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/vulcanize/vulcanizedb/libraries/shared/constants"
"github.com/vulcanize/vulcanizedb/libraries/shared/transformer"
"github.com/vulcanize/vulcanizedb/libraries/shared/watcher"
"github.com/vulcanize/vulcanizedb/pkg/fs"
"github.com/vulcanize/vulcanizedb/utils"
)
// executeCmd represents the execute command
var executeCmd = &cobra.Command{
Use: "execute",
Short: "executes a precomposed transformer initializer plugin",
Long: `This command needs a config .toml file of form:
[database]
name = "vulcanize_public"
hostname = "localhost"
user = "vulcanize"
password = "vulcanize"
port = 5432
[client]
ipcPath = "http://kovan0.vulcanize.io:8545"
[exporter]
name = "exampleTransformerExporter"
Note: If any of the plugin transformer need additional
configuration variables include them in the .toml file as well
The exporter.name is the name (without extension) of the plugin to be loaded.
The plugin file needs to be located in the /plugins directory and this command assumes
the db migrations remain from when the plugin was composed. Additionally, the plugin
must have been composed by the same version of vulcanizedb or else it will not be compatible.
Specify config location when executing the command:
./vulcanizedb execute --config=./environments/config_name.toml`,
Run: func(cmd *cobra.Command, args []string) {
execute()
},
}
func execute() {
// Build plugin generator config
prepConfig()
// Get the plugin path and load the plugin
_, pluginPath, err := genConfig.GetPluginPaths()
if err != nil {
log.Fatal(err)
}
fmt.Printf("Executing plugin %s", pluginPath)
log.Info("linking plugin", pluginPath)
plug, err := plugin.Open(pluginPath)
if err != nil {
log.Debug("linking plugin failed")
log.Fatal(err)
}
// Load the `Exporter` symbol from the plugin
log.Info("loading transformers from plugin")
symExporter, err := plug.Lookup("Exporter")
if err != nil {
log.Debug("loading Exporter symbol failed")
log.Fatal(err)
}
// Assert that the symbol is of type Exporter
exporter, ok := symExporter.(Exporter)
if !ok {
log.Debug("plugged-in symbol not of type Exporter")
os.Exit(1)
}
// Use the Exporters export method to load the TransformerInitializer and StorageTransformerInitializer sets
ethEventInitializers, ethStorageInitializers := exporter.Export()
// Setup bc and db objects
blockChain := getBlockChain()
db := utils.LoadPostgres(databaseConfig, blockChain.Node())
// Execute over transformer sets returned by the exporter
// Use WaitGroup to wait on both goroutines
var wg syn.WaitGroup
if len(ethEventInitializers) > 0 {
ew := watcher.NewEventWatcher(&db, blockChain)
ew.AddTransformers(ethEventInitializers)
wg.Add(1)
go watchEthEvents(&ew, &wg)
}
if len(ethStorageInitializers) > 0 {
tailer := fs.FileTailer{Path: storageDiffsPath}
sw := watcher.NewStorageWatcher(tailer, &db)
sw.AddTransformers(ethStorageInitializers)
wg.Add(1)
go watchEthStorage(&sw, &wg)
}
wg.Wait()
}
func init() {
rootCmd.AddCommand(executeCmd)
executeCmd.Flags().BoolVar(&recheckHeadersArg, "recheckHeaders", false, "checks headers that are already checked for each transformer.")
}
type Exporter interface {
Export() ([]transformer.TransformerInitializer, []transformer.StorageTransformerInitializer)
}
func watchEthEvents(w *watcher.EventWatcher, wg *syn.WaitGroup) {
defer wg.Done()
// Execute over the TransformerInitializer set using the watcher
log.Info("executing event transformers")
var recheck constants.TransformerExecution
if recheckHeadersArg {
recheck = constants.HeaderRecheck
} else {
recheck = constants.HeaderMissing
}
ticker := time.NewTicker(pollingInterval)
defer ticker.Stop()
for range ticker.C {
err := w.Execute(recheck)
if err != nil {
// TODO Handle watcher errors in execute
}
}
}
func watchEthStorage(w *watcher.StorageWatcher, wg *syn.WaitGroup) {
defer wg.Done()
// Execute over the TransformerInitializer set using the watcher
log.Info("executing storage transformers")
ticker := time.NewTicker(pollingInterval)
defer ticker.Stop()
for range ticker.C {
err := w.Execute()
if err != nil {
// TODO Handle watcher errors in execute
}
}
}

6
pkg/omni/shared/parser/parser_test.go
View File

@ -112,7 +112,7 @@ var _ = Describe("Parser", func() {
Expect(abiTy).To(Equal(abi.UintTy))
pgTy = e.Fields[2].PgType
Expect(pgTy).To(Equal("DECIMAL"))
Expect(pgTy).To(Equal("NUMERIC"))
_, ok = events["Approval"]
Expect(ok).To(Equal(false))
@ -143,7 +143,7 @@ var _ = Describe("Parser", func() {
Expect(abiTy).To(Equal(abi.UintTy))
pgTy = balOf.Return[0].PgType
Expect(pgTy).To(Equal("DECIMAL"))
Expect(pgTy).To(Equal("NUMERIC"))
})
@ -200,7 +200,7 @@ var _ = Describe("Parser", func() {
Expect(abiTy).To(Equal(abi.UintTy))
pgTy = balOf.Return[0].PgType
Expect(pgTy).To(Equal("DECIMAL"))
Expect(pgTy).To(Equal("NUMERIC"))
})

2
pkg/omni/shared/types/event.go
View File

@ -64,7 +64,7 @@ func NewEvent(e abi.Event) Event {
case abi.HashTy, abi.AddressTy:
fields[i].PgType = "CHARACTER VARYING(66)"
case abi.IntTy, abi.UintTy:
fields[i].PgType = "DECIMAL"
fields[i].PgType = "NUMERIC"
case abi.BoolTy:
fields[i].PgType = "BOOLEAN"
case abi.BytesTy, abi.FixedBytesTy:

4
pkg/omni/shared/types/method.go
View File

@ -53,7 +53,7 @@ func NewMethod(m abi.Method) Method {
case abi.HashTy, abi.AddressTy:
inputs[i].PgType = "CHARACTER VARYING(66)"
case abi.IntTy, abi.UintTy:
inputs[i].PgType = "DECIMAL"
inputs[i].PgType = "NUMERIC"
case abi.BoolTy:
inputs[i].PgType = "BOOLEAN"
case abi.BytesTy, abi.FixedBytesTy:
@ -77,7 +77,7 @@ func NewMethod(m abi.Method) Method {
case abi.HashTy, abi.AddressTy:
outputs[i].PgType = "CHARACTER VARYING(66)"
case abi.IntTy, abi.UintTy:
outputs[i].PgType = "DECIMAL"
outputs[i].PgType = "NUMERIC"
case abi.BoolTy:
outputs[i].PgType = "BOOLEAN"
case abi.BytesTy, abi.FixedBytesTy: