cerc-io/ipld-eth-server
12
0
Fork 1
Code Issues 10 Pull Requests 2 Actions Packages Projects Releases 37 Wiki Activity

continue refactor/purge

Browse Source
This commit is contained in:
Ian Norden
2020-09-01 16:15:31 -05:00
parent f4b7b92dd8
commit 0d28234804
24 changed files with 69 additions and 1075 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/apis.md
+40 -40
View File
@@ -1,5 +1,5 @@
## ipfs-blockchain-watcher APIs
We can expose a number of different APIs for remote access to ipfs-blockchain-watcher data
## ipld-eth-server APIs
We can expose a number of different APIs for remote access to ipld-eth-server data
### Table of Contents
@@ -9,7 +9,7 @@ We can expose a number of different APIs for remote access to ipfs-blockchain-wa
### Postgraphile
ipfs-blockchain-watcher stores all processed data in Postgres using PG-IPFS, this includes all of the IPLD objects.
ipld-eth-server stores all processed data in Postgres using PG-IPFS, this includes all of the IPLD objects.
[Postgraphile](https://www.graphile.org/postgraphile/) can be used to expose GraphQL endpoints for the Postgres tables.
e.g.
@@ -22,15 +22,15 @@ All of their data can then be queried with standard [GraphQL](https://graphql.or
### RPC Subscription Interface
A direct, real-time subscription to the data being processed by ipfs-blockchain-watcher can be established over WS or IPC through the [Stream](../pkg/watch/api.go#L53) RPC method.
A direct, real-time subscription to the data being processed by ipld-eth-server can be established over WS or IPC through the [Stream](../pkg/serve/api.go#L53) RPC method.
This method is not chain-specific and each chain-type supports it, it is accessed under the "vdb" namespace rather than a chain-specific namespace. An interface for
subscribing to this endpoint is provided [here](../pkg/client/client.go).
When subscribing to this endpoint, the subscriber provides a set of RLP-encoded subscription parameters. These parameters will be chain-specific, and are used
by ipfs-blockchain-watcher to filter and return a requested subset of chain data to the subscriber. (e.g. [BTC](../pkg/btc/subscription_config.go), [ETH](../../pkg/eth/subscription_config.go)).
by ipld-eth-server to filter and return a requested subset of chain data to the subscriber. (e.g. [BTC](../pkg/btc/subscription_config.go), [ETH](../../pkg/eth/subscription_config.go)).
#### Ethereum RPC Subscription
An example of how to subscribe to a real-time Ethereum data feed from ipfs-blockchain-watcher using the `Stream` RPC method is provided below
An example of how to subscribe to a real-time Ethereum data feed from ipld-eth-server using the `Stream` RPC method is provided below
```go
package main
@@ -40,9 +40,9 @@ An example of how to subscribe to a real-time Ethereum data feed from ipfs-block
"github.com/ethereum/go-ethereum/rpc"
"github.com/spf13/viper"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/client"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/eth"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/watch"
"github.com/vulcanize/ipld-eth-server/pkg/client"
"github.com/vulcanize/ipld-eth-server/pkg/eth"
"github.com/vulcanize/ipld-eth-server/pkg/watch"
)
config, _ := eth.NewEthSubscriptionConfig()
@@ -101,10 +101,10 @@ These configuration parameters are broken down as follows:
`ethSubscription.wsPath` is used to define the watcher ws url OR ipc endpoint to subscribe to
`ethSubscription.historicalData` specifies whether or not ipfs-blockchain-watcher should look up historical data in its cache and
`ethSubscription.historicalData` specifies whether or not ipld-eth-server should look up historical data in its cache and
send that to the subscriber, if this is set to `false` then only newly synced/incoming data is streamed
`ethSubscription.historicalDataOnly` will tell ipfs-blockchain-watcher to only send historical data with the specified range and
`ethSubscription.historicalDataOnly` will tell ipld-eth-server to only send historical data with the specified range and
not stream forward syncing data
`ethSubscription.startingBlock` is the starting block number for the range to receive data in
@@ -114,43 +114,43 @@ setting to 0 means the process will continue streaming indefinitely.
`ethSubscription.headerFilter` has two sub-options: `off` and `uncles`.
- Setting `off` to true tells ipfs-blockchain-watcher to not send any headers to the subscriber
- setting `uncles` to true tells ipfs-blockchain-watcher to send uncles in addition to normal headers.
- Setting `off` to true tells ipld-eth-server to not send any headers to the subscriber
- setting `uncles` to true tells ipld-eth-server to send uncles in addition to normal headers.
`ethSubscription.txFilter` has three sub-options: `off`, `src`, and `dst`.
- Setting `off` to true tells ipfs-blockchain-watcher to not send any transactions to the subscriber
- Setting `off` to true tells ipld-eth-server to not send any transactions to the subscriber
- `src` and `dst` are string arrays which can be filled with ETH addresses to filter transactions for,
if they have any addresses then ipfs-blockchain-watcher will only send transactions that were sent or received by the addresses contained
if they have any addresses then ipld-eth-server will only send transactions that were sent or received by the addresses contained
in `src` and `dst`, respectively.
`ethSubscription.receiptFilter` has four sub-options: `off`, `topics`, `contracts` and `matchTxs`.
- Setting `off` to true tells ipfs-blockchain-watcher to not send any receipts to the subscriber
- Setting `off` to true tells ipld-eth-server to not send any receipts to the subscriber
- `topic0s` is a string array which can be filled with event topics to filter for,
if it has any topics then ipfs-blockchain-watcher will only send receipts that contain logs which have that topic0.
if it has any topics then ipld-eth-server will only send receipts that contain logs which have that topic0.
- `contracts` is a string array which can be filled with contract addresses to filter for, if it contains any contract addresses the watcher will
only send receipts that correspond to one of those contracts.
- `matchTrxs` is a bool which when set to true any receipts that correspond to filtered for transactions will be sent by the watcher, regardless of whether or not the receipt satisfies the `topics` or `contracts` filters.
`ethSubscription.stateFilter` has three sub-options: `off`, `addresses`, and `intermediateNodes`.
- Setting `off` to true tells ipfs-blockchain-watcher to not send any state data to the subscriber
- Setting `off` to true tells ipld-eth-server to not send any state data to the subscriber
- `addresses` is a string array which can be filled with ETH addresses to filter state for,
if it has any addresses then ipfs-blockchain-watcher will only send state leafs (accounts) corresponding to those account addresses.
- By default ipfs-blockchain-watcher only sends along state leafs, to receive branch and extension nodes as well `intermediateNodes` can be set to `true`.
if it has any addresses then ipld-eth-server will only send state leafs (accounts) corresponding to those account addresses.
- By default ipld-eth-server only sends along state leafs, to receive branch and extension nodes as well `intermediateNodes` can be set to `true`.
`ethSubscription.storageFilter` has four sub-options: `off`, `addresses`, `storageKeys`, and `intermediateNodes`.
- Setting `off` to true tells ipfs-blockchain-watcher to not send any storage data to the subscriber
- Setting `off` to true tells ipld-eth-server to not send any storage data to the subscriber
- `addresses` is a string array which can be filled with ETH addresses to filter storage for,
if it has any addresses then ipfs-blockchain-watcher will only send storage nodes from the storage tries at those state addresses.
if it has any addresses then ipld-eth-server will only send storage nodes from the storage tries at those state addresses.
- `storageKeys` is another string array that can be filled with storage keys to filter storage data for. It is important to note that the storage keys need to be the actual keccak256 hashes, whereas
the addresses in the `addresses` fields are pre-hashed ETH addresses.
- By default ipfs-blockchain-watcher only sends along storage leafs, to receive branch and extension nodes as well `intermediateNodes` can be set to `true`.
- By default ipld-eth-server only sends along storage leafs, to receive branch and extension nodes as well `intermediateNodes` can be set to `true`.
### Bitcoin RPC Subscription:
An example of how to subscribe to a real-time Bitcoin data feed from ipfs-blockchain-watcher using the `Stream` RPC method is provided below
An example of how to subscribe to a real-time Bitcoin data feed from ipld-eth-server using the `Stream` RPC method is provided below
```go
package main
@@ -160,9 +160,9 @@ An example of how to subscribe to a real-time Bitcoin data feed from ipfs-blockc
"github.com/ethereum/go-ethereum/rpc"
"github.com/spf13/viper"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/btc"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/client"
"github.com/vulcanize/ipfs-blockchain-watcher/pkg/watch"
"github.com/vulcanize/ipld-eth-server/pkg/btc"
"github.com/vulcanize/ipld-eth-server/pkg/client"
"github.com/vulcanize/ipld-eth-server/pkg/watch"
)
config, _ := btc.NewBtcSubscriptionConfig()
@@ -206,12 +206,12 @@ The .toml file being used to fill the Bitcoin subscription config would look som
These configuration parameters are broken down as follows:
`btcSubscription.wsPath` is used to define the ipfs-blockchain-watcher ws url OR ipc endpoint to subscribe to
`btcSubscription.wsPath` is used to define the ipld-eth-server ws url OR ipc endpoint to subscribe to
`btcSubscription.historicalData` specifies whether or not ipfs-blockchain-watcher should look up historical data in its cache and
send that to the subscriber, if this is set to `false` then ipfs-blockchain-watcher only streams newly synced/incoming data
`btcSubscription.historicalData` specifies whether or not ipld-eth-server should look up historical data in its cache and
send that to the subscriber, if this is set to `false` then ipld-eth-server only streams newly synced/incoming data
`btcSubscription.historicalDataOnly` will tell ipfs-blockchain-watcher to only send historical data with the specified range and
`btcSubscription.historicalDataOnly` will tell ipld-eth-server to only send historical data with the specified range and
not stream forward syncing data
`btcSubscription.startingBlock` is the starting block number for the range to receive data in
@@ -221,20 +221,20 @@ setting to 0 means the process will continue streaming indefinitely.
`btcSubscription.headerFilter` has one sub-option: `off`.
- Setting `off` to true tells ipfs-blockchain-watcher to
- Setting `off` to true tells ipld-eth-server to
not send any headers to the subscriber.
- Additional header-filtering options will be added in the future.
`btcSubscription.txFilter` has seven sub-options: `off`, `segwit`, `witnessHashes`, `indexes`, `pkScriptClass`, `multiSig`, and `addresses`.
- Setting `off` to true tells ipfs-blockchain-watcher to not send any transactions to the subscriber.
- Setting `segwit` to true tells ipfs-blockchain-watcher to only send segwit transactions.
- `witnessHashes` is a string array that can be filled with witness hash string; if it contains any hashes ipfs-blockchain-watcher will only send transactions that contain one of those hashes.
- `indexes` is an int64 array that can be filled with tx index numbers; if it contains any integers ipfs-blockchain-watcher will only send transactions at those indexes (e.g. `[0]` will send only coinbase transactions)
- `pkScriptClass` is an uint8 array that can be filled with pk script class numbers; if it contains any integers ipfs-blockchain-watcher will only send transactions that have at least one tx output with one of the specified pkscript classes;
- Setting `off` to true tells ipld-eth-server to not send any transactions to the subscriber.
- Setting `segwit` to true tells ipld-eth-server to only send segwit transactions.
- `witnessHashes` is a string array that can be filled with witness hash string; if it contains any hashes ipld-eth-server will only send transactions that contain one of those hashes.
- `indexes` is an int64 array that can be filled with tx index numbers; if it contains any integers ipld-eth-server will only send transactions at those indexes (e.g. `[0]` will send only coinbase transactions)
- `pkScriptClass` is an uint8 array that can be filled with pk script class numbers; if it contains any integers ipld-eth-server will only send transactions that have at least one tx output with one of the specified pkscript classes;
possible class types are 0 through 8 as defined [here](https://github.com/btcsuite/btcd/blob/master/txscript/standard.go#L52).
- Setting `multisig` to true tells ipfs-blockchain-watcher to send only multi-sig transactions- to send only transaction that have at least one tx output that requires more than one signature to spend.
- `addresses` is a string array that can be filled with btc address strings; if it contains any addresses ipfs-blockchain-watcher will only send transactions that have at least one tx output with at least one of the provided addresses.
- Setting `multisig` to true tells ipld-eth-server to send only multi-sig transactions- to send only transaction that have at least one tx output that requires more than one signature to spend.
- `addresses` is a string array that can be filled with btc address strings; if it contains any addresses ipld-eth-server will only send transactions that have at least one tx output with at least one of the provided addresses.
### Native API Recapitulation:
@@ -242,7 +242,7 @@ In addition to providing novel Postgraphile and RPC-Subscription endpoints, we a
standard chain APIs. This will allow direct compatibility with software that already makes use of the standard interfaces.
#### Ethereum JSON-RPC API
ipfs-blockchain-watcher currently faithfully recapitulates portions of the Ethereum JSON-RPC api standard.
ipld-eth-server currently faithfully recapitulates portions of the Ethereum JSON-RPC api standard.
The currently supported endpoints include:
`eth_blockNumber`
documentation/ipfs.md
-53
View File
@@ -1,53 +0,0 @@
### PG-IPFS configuration
This doc walks through the steps to install IPFS and configure it to use Postgres as its backing datastore.
1. Start by downloading and moving into the IPFS repo:
`go get github.com/ipfs/go-ipfs`
`cd $GOPATH/src/github.com/ipfs/go-ipfs`
2. Add the [Postgres-supporting fork](https://github.com/vulcanize/go-ipfs) and switch over to it:
`git remote add vulcanize https://github.com/vulcanize/go-ipfs.git`
`git fetch vulcanize`
`git checkout -b postgres_update tags/v0.4.22-alpha`
3. Now install this fork of ipfs, first be sure to remove any previous installation:
`make install`
4. Check that is installed properly by running:
`ipfs`
You should see the CLI info/help output.
5. Now we initialize with the `postgresds` profile.
If ipfs was previously initialized we will need to remove the old profile first.
We also need to provide env variables for the postgres connection:
We can either set these manually, e.g.
```bash
export IPFS_PGHOST=
export IPFS_PGUSER=
export IPFS_PGDATABASE=
export IPFS_PGPORT=
export IPFS_PGPASSWORD=
```
And then run the ipfs command:
`ipfs init --profile=postgresds`
Or we can use the pre-made script at `GOPATH/src/github.com/ipfs/go-ipfs/misc/utility/ipfs_postgres.sh`
which has usage:
`./ipfs_postgres.sh <IPFS_PGHOST> <IPFS_PGPORT> <IPFS_PGUSER> <IPFS_PGDATABASE>"`
and will ask us to enter the password, avoiding storing it to an ENV variable.
Once we have initialized ipfs, that is all we need to do with it- we do not need to run a daemon during the subsequent processes.