cerc-io/ipld-eth-server
17
0
Fork 1
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 37 Wiki Activity

update deps/paths/names

Browse Source
This commit is contained in:
Ian Norden 2020-08-31 10:52:47 -05:00
parent 5830df44a4
commit f4591a6beb
10 changed files with 43 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@ -7,7 +7,7 @@ Vagrantfile
vagrant*.sh vagrant*.sh
.vagrant .vagrant
test_scripts/ test_scripts/
ipfs-blockchain-watcher ipld-eth-server
postgraphile/build/ postgraphile/build/
postgraphile/node_modules/ postgraphile/node_modules/
postgraphile/package-lock.json postgraphile/package-lock.json

32
README.md
View File

@ -1,8 +1,8 @@
# ipfs-blockchain-watcher # ipld-eth-server
[![Go Report Card](https://goreportcard.com/badge/github.com/vulcanize/ipfs-blockchain-watcher)](https://goreportcard.com/report/github.com/vulcanize/ipfs-blockchain-watcher) [![Go Report Card](https://goreportcard.com/badge/github.com/vulcanize/ipld-eth-server)](https://goreportcard.com/report/github.com/vulcanize/ipld-eth-server)
> ipfs-blockchain-watcher is used to extract, transform, and load all eth or btc data into an IPFS-backing Postgres datastore while generating useful secondary indexes around the data in other Postgres tables > ipld-eth-server is used to extract, transform, and load all eth or btc data into an IPFS-backing Postgres datastore while generating useful secondary indexes around the data in other Postgres tables
## Table of Contents ## Table of Contents
1. [Background](#background) 1. [Background](#background)
@ -13,13 +13,13 @@
1. [License](#license) 1. [License](#license)
## Background ## Background
ipfs-blockchain-watcher is a collection of interfaces that are used to extract, process, store, and index ipld-eth-server is a collection of interfaces that are used to extract, process, store, and index
all blockchain data in Postgres-IPFS. The raw data indexed by ipfs-blockchain-watcher serves as the basis for more specific watchers and applications. all blockchain data in Postgres-IPFS. The raw data indexed by ipld-eth-server serves as the basis for more specific watchers and applications.
Currently the service supports complete processing of all Bitcoin and Ethereum data. Currently the service supports complete processing of all Bitcoin and Ethereum data.
## Architecture ## Architecture
More details on the design of ipfs-blockchain-watcher can be found in [here](./documentation/architecture.md) More details on the design of ipld-eth-server can be found in [here](./documentation/architecture.md)
## Dependencies ## Dependencies
Minimal build dependencies Minimal build dependencies
@ -49,7 +49,7 @@ is required for running the automated tests and is used by the `make migrate` co
1. [Install Postgres](https://wiki.postgresql.org/wiki/Detailed_installation_guides) 1. [Install Postgres](https://wiki.postgresql.org/wiki/Detailed_installation_guides)
1. Create a superuser for yourself and make sure `psql --list` works without prompting for a password. 1. Create a superuser for yourself and make sure `psql --list` works without prompting for a password.
1. `createdb vulcanize_public` 1. `createdb vulcanize_public`
1. `cd $GOPATH/src/github.com/vulcanize/ipfs-blockchain-watcher` 1. `cd $GOPATH/src/github.com/vulcanize/ipld-eth-server`
1. Run the migrations: `make migrate HOST_NAME=localhost NAME=vulcanize_public PORT=5432` 1. Run the migrations: `make migrate HOST_NAME=localhost NAME=vulcanize_public PORT=5432`
- There are optional vars `USER=username:password` if the database user is not the default user `postgres` and/or a password is present - There are optional vars `USER=username:password` if the database user is not the default user `postgres` and/or a password is present
- To rollback a single step: `make rollback NAME=vulcanize_public` - To rollback a single step: `make rollback NAME=vulcanize_public`
@ -68,7 +68,7 @@ Data is stored in an [IPFS-backing Postgres datastore](https://github.com/ipfs/g
By default data is written directly to the ipfs blockstore in Postgres; the public.blocks table. By default data is written directly to the ipfs blockstore in Postgres; the public.blocks table.
In this case no further IPFS configuration is needed at this time. In this case no further IPFS configuration is needed at this time.
Optionally, ipfs-blockchain-watcher can be configured to function through an internal ipfs node interface using the flag: `-ipfs-mode=interface`. Optionally, ipld-eth-server can be configured to function through an internal ipfs node interface using the flag: `-ipfs-mode=interface`.
Operating through the ipfs interface provides the option to configure a block exchange that can search remotely for IPLD data found missing in the local datastore. Operating through the ipfs interface provides the option to configure a block exchange that can search remotely for IPLD data found missing in the local datastore.
This option is irrelevant in most cases and this mode has some disadvantages, namely: This option is irrelevant in most cases and this mode has some disadvantages, namely:
@ -79,7 +79,7 @@ This option is irrelevant in most cases and this mode has some disadvantages, na
More information for configuring Postgres-IPFS can be found [here](./documentation/ipfs.md) More information for configuring Postgres-IPFS can be found [here](./documentation/ipfs.md)
### Blockchain ### Blockchain
This section describes how to setup an Ethereum or Bitcoin node to serve as a data source for ipfs-blockchain-watcher This section describes how to setup an Ethereum or Bitcoin node to serve as a data source for ipld-eth-server
#### Ethereum #### Ethereum
For Ethereum, [a special fork of go-ethereum](https://github.com/vulcanize/go-ethereum/tree/statediff_at_anyblock-1.9.11) is currently *requirde*. For Ethereum, [a special fork of go-ethereum](https://github.com/vulcanize/go-ethereum/tree/statediff_at_anyblock-1.9.11) is currently *requirde*.
@ -122,7 +122,7 @@ The default ws url is "127.0.0.1:8546" and the default http url is "127.0.0.1:85
These values will be used as the `ethereum.wsPath` and `ethereum.httpPath` in the config, respectively. These values will be used as the `ethereum.wsPath` and `ethereum.httpPath` in the config, respectively.
#### Bitcoin #### Bitcoin
For Bitcoin, ipfs-blockchain-watcher is able to operate entirely through the universally exposed JSON-RPC interfaces. For Bitcoin, ipld-eth-server is able to operate entirely through the universally exposed JSON-RPC interfaces.
This means any of the standard full nodes can be used (e.g. bitcoind, btcd) as the data source. This means any of the standard full nodes can be used (e.g. bitcoind, btcd) as the data source.
Point at a remote node or set one up locally using the instructions for [bitcoind](https://github.com/bitcoin/bitcoin) and [btcd](https://github.com/btcsuite/btcd). Point at a remote node or set one up locally using the instructions for [bitcoind](https://github.com/bitcoin/bitcoin) and [btcd](https://github.com/btcsuite/btcd).
@ -133,11 +133,11 @@ The default http url is "127.0.0.1:8332". We will use the http endpoint as both
### Watcher ### Watcher
Finally, setup the watcher process itself. Finally, setup the watcher process itself.
Start by downloading ipfs-blockchain-watcher and moving into the repo: Start by downloading ipld-eth-server and moving into the repo:
`GO111MODULE=off go get -d github.com/vulcanize/ipfs-blockchain-watcher` `GO111MODULE=off go get -d github.com/vulcanize/ipld-eth-server`
`cd $GOPATH/src/github.com/vulcanize/ipfs-blockchain-watcher` `cd $GOPATH/src/github.com/vulcanize/ipld-eth-server`
Then, build the binary: Then, build the binary:
@ -146,11 +146,11 @@ Then, build the binary:
## Usage ## Usage
After building the binary, run as After building the binary, run as
`./ipfs-blockchain-watcher watch --config=<the name of your config file.toml>` `./ipld-eth-server watch --config=<the name of your config file.toml>`
### Configuration ### Configuration
Below is the set of universal config parameters for the ipfs-blockchain-watcher command, in .toml form, with the respective environmental variables commented to the side. Below is the set of universal config parameters for the ipld-eth-server command, in .toml form, with the respective environmental variables commented to the side.
This set of parameters needs to be set no matter the chain type. This set of parameters needs to be set no matter the chain type.
```toml ```toml
@ -207,7 +207,7 @@ For Ethereum:
``` ```
### Exposing the data ### Exposing the data
A number of different APIs for remote access to ipfs-blockchain-watcher data can be exposed, these are discussed in more detail [here](./documentation/apis.md) A number of different APIs for remote access to ipld-eth-server data can be exposed, these are discussed in more detail [here](./documentation/apis.md)
### Testing ### Testing
`make test` will run the unit tests `make test` will run the unit tests

10
dockerfiles/migrations/Dockerfile
View File

@ -7,15 +7,15 @@ RUN apk add busybox-extras
# this is probably a noob move, but I want apk from alpine for the above but need to avoid Go 1.13 below as this error still occurs https://github.com/ipfs/go-ipfs/issues/6603 # this is probably a noob move, but I want apk from alpine for the above but need to avoid Go 1.13 below as this error still occurs https://github.com/ipfs/go-ipfs/issues/6603
FROM golang:1.12.4 as builder FROM golang:1.12.4 as builder
# Get and build ipfs-blockchain-watcher # Get and build ipld-eth-server
ADD . /go/src/github.com/vulcanize/ipfs-blockchain-watcher ADD . /go/src/github.com/vulcanize/ipld-eth-server
# Build migration tool # Build migration tool
RUN go get -u -d github.com/pressly/goose/cmd/goose RUN go get -u -d github.com/pressly/goose/cmd/goose
WORKDIR /go/src/github.com/pressly/goose/cmd/goose WORKDIR /go/src/github.com/pressly/goose/cmd/goose
RUN GCO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -tags='no_mysql no_sqlite' -o goose . RUN GCO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -tags='no_mysql no_sqlite' -o goose .
WORKDIR /go/src/github.com/vulcanize/ipfs-blockchain-watcher WORKDIR /go/src/github.com/vulcanize/ipld-eth-server
# app container # app container
FROM alpine FROM alpine
@ -29,12 +29,12 @@ USER $USER
# chown first so dir is writable # chown first so dir is writable
# note: using $USER is merged, but not in the stable release yet # note: using $USER is merged, but not in the stable release yet
COPY --chown=5000:5000 --from=builder /go/src/github.com/vulcanize/ipfs-blockchain-watcher/dockerfiles/migrations/startup_script.sh . COPY --chown=5000:5000 --from=builder /go/src/github.com/vulcanize/ipld-eth-server/dockerfiles/migrations/startup_script.sh .
# keep binaries immutable # keep binaries immutable
COPY --from=builder /go/src/github.com/pressly/goose/cmd/goose/goose goose COPY --from=builder /go/src/github.com/pressly/goose/cmd/goose/goose goose
COPY --from=builder /go/src/github.com/vulcanize/ipfs-blockchain-watcher/db/migrations migrations/vulcanizedb COPY --from=builder /go/src/github.com/vulcanize/ipld-eth-server/db/migrations migrations/vulcanizedb
# XXX dir is already writeable RUN touch vulcanizedb.log # XXX dir is already writeable RUN touch vulcanizedb.log
CMD ["./startup_script.sh"] CMD ["./startup_script.sh"]

22
dockerfiles/super_node/Dockerfile
View File

@ -4,10 +4,10 @@ RUN apk --update --no-cache add make git g++ linux-headers
# DEBUG # DEBUG
RUN apk add busybox-extras RUN apk add busybox-extras
# Get and build ipfs-blockchain-watcher # Get and build ipld-eth-server
ADD . /go/src/github.com/vulcanize/ipfs-blockchain-watcher ADD . /go/src/github.com/vulcanize/ipld-eth-server
WORKDIR /go/src/github.com/vulcanize/ipfs-blockchain-watcher WORKDIR /go/src/github.com/vulcanize/ipld-eth-server
RUN GO111MODULE=on GCO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o ipfs-blockchain-watcher . RUN GO111MODULE=on GCO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o ipld-eth-server .
# Build migration tool # Build migration tool
WORKDIR / WORKDIR /
@ -15,7 +15,7 @@ RUN go get -u -d github.com/pressly/goose/cmd/goose
WORKDIR /go/src/github.com/pressly/goose/cmd/goose WORKDIR /go/src/github.com/pressly/goose/cmd/goose
RUN GCO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -tags='no_mysql no_sqlite' -o goose . RUN GCO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -tags='no_mysql no_sqlite' -o goose .
WORKDIR /go/src/github.com/vulcanize/ipfs-blockchain-watcher WORKDIR /go/src/github.com/vulcanize/ipld-eth-server
# app container # app container
FROM alpine FROM alpine
@ -32,16 +32,16 @@ USER $USER
# chown first so dir is writable # chown first so dir is writable
# note: using $USER is merged, but not in the stable release yet # note: using $USER is merged, but not in the stable release yet
COPY --chown=5000:5000 --from=builder /go/src/github.com/vulcanize/ipfs-blockchain-watcher/$CONFIG_FILE config.toml COPY --chown=5000:5000 --from=builder /go/src/github.com/vulcanize/ipld-eth-server/$CONFIG_FILE config.toml
COPY --chown=5000:5000 --from=builder /go/src/github.com/vulcanize/ipfs-blockchain-watcher/dockerfiles/super_node/startup_script.sh . COPY --chown=5000:5000 --from=builder /go/src/github.com/vulcanize/ipld-eth-server/dockerfiles/super_node/startup_script.sh .
COPY --chown=5000:5000 --from=builder /go/src/github.com/vulcanize/ipfs-blockchain-watcher/dockerfiles/super_node/entrypoint.sh . COPY --chown=5000:5000 --from=builder /go/src/github.com/vulcanize/ipld-eth-server/dockerfiles/super_node/entrypoint.sh .
# keep binaries immutable # keep binaries immutable
COPY --from=builder /go/src/github.com/vulcanize/ipfs-blockchain-watcher/ipfs-blockchain-watcher ipfs-blockchain-watcher COPY --from=builder /go/src/github.com/vulcanize/ipld-eth-server/ipld-eth-server ipld-eth-server
COPY --from=builder /go/src/github.com/pressly/goose/cmd/goose/goose goose COPY --from=builder /go/src/github.com/pressly/goose/cmd/goose/goose goose
COPY --from=builder /go/src/github.com/vulcanize/ipfs-blockchain-watcher/db/migrations migrations/vulcanizedb COPY --from=builder /go/src/github.com/vulcanize/ipld-eth-server/db/migrations migrations/vulcanizedb
COPY --from=builder /go/src/github.com/vulcanize/ipfs-blockchain-watcher/environments environments COPY --from=builder /go/src/github.com/vulcanize/ipld-eth-server/environments environments
EXPOSE $EXPOSE_PORT_1 EXPOSE $EXPOSE_PORT_1
EXPOSE $EXPOSE_PORT_2 EXPOSE $EXPOSE_PORT_2

6
dockerfiles/super_node/entrypoint.sh
View File

@ -35,7 +35,7 @@ echo "Beginning the vulcanizedb process"
VDB_CONFIG_FILE=${VDB_CONFIG_FILE:-config.toml} VDB_CONFIG_FILE=${VDB_CONFIG_FILE:-config.toml}
DEFAULT_OPTIONS="--config=$VDB_CONFIG_FILE" DEFAULT_OPTIONS="--config=$VDB_CONFIG_FILE"
VDB_FULL_CL=${VDB_FULL_CL:-$VDB_COMMAND $DEFAULT_OPTIONS} VDB_FULL_CL=${VDB_FULL_CL:-$VDB_COMMAND $DEFAULT_OPTIONS}
echo running: ./ipfs-blockchain-watcher $VDB_FULL_CL $@ echo running: ./ipld-eth-server $VDB_FULL_CL $@
case "$1" in case "$1" in
"/bin/sh" ) "/bin/sh" )
@ -49,8 +49,8 @@ if [[ -z "$vdb_args" ]]; then
vdb_args="--config=config.toml" vdb_args="--config=config.toml"
fi fi
echo running: ./ipfs-blockchain-watcher $vdb_args echo running: ./ipld-eth-server $vdb_args
./ipfs-blockchain-watcher $vdb_args ./ipld-eth-server $vdb_args
rv=$? rv=$?
if [ $rv != 0 ]; then if [ $rv != 0 ]; then

2
dockerfiles/super_node/startup_script.sh
View File

@ -49,7 +49,7 @@ fi
# If IPFS initialization was successful # If IPFS initialization was successful
if [[ $? -eq 0 ]]; then if [[ $? -eq 0 ]]; then
echo "Running the VulcanizeDB process" echo "Running the VulcanizeDB process"
./ipfs-blockchain-watcher ${VDB_COMMAND} --config=config.toml ./ipld-eth-server ${VDB_COMMAND} --config=config.toml
else else
echo "Could not initialize IPFS." echo "Could not initialize IPFS."
exit 1 exit 1

4
scripts/reset_db
View File

@ -11,9 +11,9 @@ fi
db=$1 db=$1
dir=$(basename "$(pwd)") dir=$(basename "$(pwd)")
if [ $dir != "ipfs-blockchain-watcher" ] if [ $dir != "ipld-eth-server" ]
then then
echo "Run me from the ipfs-blockchain-watcher root dir" echo "Run me from the ipld-eth-server root dir"
exit 1 exit 1
fi fi

2
test_config/test_config.go
View File

@ -34,7 +34,7 @@ func init() {
func setTestConfig() { func setTestConfig() {
vip := viper.New() vip := viper.New()
vip.SetConfigName("testing") vip.SetConfigName("testing")
vip.AddConfigPath("$GOPATH/src/github.com/vulcanize/ipfs-blockchain-watcher/environments/") vip.AddConfigPath("$GOPATH/src/github.com/vulcanize/ipld-eth-server/environments/")
if err := vip.ReadInConfig(); err != nil { if err := vip.ReadInConfig(); err != nil {
logrus.Fatal(err) logrus.Fatal(err)
} }

4
utils/utils.go
View File

@ -22,9 +22,9 @@ import (
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/config" "github.com/vulcanize/ipfs-blockchain-watcher/pkg/config"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/node" "github.com/vulcanize/ipld-eth-server/pkg/node"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/postgres" "github.com/vulcanize/ipfs-blockchain-watcher/pkg/postgres"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/shared" "github.com/vulcanize/ipld-eth-server/pkg/shared"
) )
func LoadPostgres(database config.Database, node node.Node) postgres.DB { func LoadPostgres(database config.Database, node node.Node) postgres.DB {

2
utils/utils_test.go
View File

@ -20,7 +20,7 @@ import (
. "github.com/onsi/ginkgo" . "github.com/onsi/ginkgo"
. "github.com/onsi/gomega" . "github.com/onsi/gomega"
"github.com/vulcanize/ipfs-blockchain-watcher/utils" "github.com/vulcanize/ipld-eth-server/utils"
) )
var _ = Describe("GetBlockHeightBins", func() { var _ = Describe("GetBlockHeightBins", func() {