Statediffing geth

* Write state diff to CSV (#2)

* port statediff from 9b7fd9af80/statediff/statediff.go; minor fixes

* integrating state diff extracting, building, and persisting into geth processes

* work towards persisting created statediffs in ipfs; based off github.com/vulcanize/eth-block-extractor

* Add a state diff service

* Remove diff extractor from blockchain

* Update imports

* Move statediff on/off check to geth cmd config

* Update starting state diff service

* Add debugging logs for creating diff

* Add statediff extractor and builder tests and small refactoring

* Start to write statediff to a CSV

* Restructure statediff directory

* Pull CSV publishing methods into their own file

* Reformatting due to go fmt

* Add gomega to vendor dir

* Remove testing focuses

* Update statediff tests to use golang test pkg

instead of ginkgo

- builder_test
- extractor_test
- publisher_test

* Use hexutil.Encode instead of deprecated common.ToHex

* Remove OldValue from DiffBigInt and DiffUint64 fields

* Update builder test

* Remove old storage value from updated accounts

* Remove old values from created/deleted accounts

* Update publisher to account for only storing current account values

* Update service loop and fetching previous block

* Update testing

- remove statediff ginkgo test suite file
- move mocks to their own dir

* Updates per go fmt

* Updates to tests

* Pass statediff mode and path in through cli

* Return filename from publisher

* Remove some duplication in builder

* Remove code field from state diff output

this is the contract byte code, and it can still be obtained by querying
the db by the codeHash

* Consolidate acct diff structs for updated & updated/deleted accts

* Include block number in csv filename

* Clean up error logging

* Cleanup formatting, spelling, etc

* Address PR comments

* Add contract address and storage value to csv

* Refactor accumulating account row in csv publisher

* Add DiffStorage struct

* Add storage key to csv

* Address PR comments

* Fix publisher to include rows for accounts that don't have store updates

* Update builder test after merging in release/1.8

* Update test contract to include storage on contract intialization

- so that we're able to test that storage diffing works for created and
deleted accounts (not just updated accounts).

* Factor out a common trie iterator method in builder

* Apply goimports to statediff

* Apply gosimple changes to statediff

* Gracefully exit geth command(#4)

* Statediff for full node (#6)

* Open a trie from the in-memory database

* Use a node's LeafKey as an identifier instead of the address

It was proving difficult to find look the address up from a given path
with a full node (sometimes the value wouldn't exist in the disk db).
So, instead, for now we are using the node's LeafKey with is a Keccak256
hash of the address, so if we know the address we can figure out which
LeafKey it matches up to.

* Make sure that statediff has been processed before pruning

* Use blockchain stateCache.OpenTrie for storage diffs

* Clean up log lines and remove unnecessary fields from builder

* Apply go fmt changes

* Add a sleep to the blockchain test

* refactoring/reorganizing packages

* refactoring statediff builder and types and adjusted to relay proofs and paths (still need to make this optional)

* refactoring state diff service and adding api which allows for streaming state diff payloads over an rpc websocket subscription

* make proofs and paths optional + compress service loop into single for loop (may be missing something here)

* option to process intermediate nodes

* make state diff rlp serializable

* cli parameter to limit statediffing to select account addresses + test

* review fixes and fixes for issues ran into in integration

* review fixes; proper method signature for api; adjust service so that statediff processing is halted/paused until there is at least one subscriber listening for the results

* adjust buffering to improve stability; doc.go; fix notifier
err handling

* relay receipts with the rest of the data + review fixes/changes

* rpc method to get statediff at specific block; requires archival node or the block be within the pruning range

* fix linter issues

* include total difficulty to the payload

* fix state diff builder: emit actual leaf nodes instead of value nodes; diff on the leaf not on the value; emit correct path for intermediate nodes

* adjust statediff builder tests to changes and extend to test intermediate nodes; golint

* add genesis block to test; handle block 0 in StateDiffAt

* rlp files for mainnet blocks 0-3, for tests

* builder test on mainnet blocks

* common.BytesToHash(path) => crypto.Keaccak256(hash) in builder; BytesToHash produces same hash for e.g. []byte{} and []byte{\x00} - prefix \x00 steps are inconsequential to the hash result

* complete tests for early mainnet blocks

* diff type for representing deleted accounts

* fix builder so that we handle account deletions properly and properly diff storage when an account is moved to a new path; update params

* remove cli params; moving them to subscriber defined

* remove unneeded bc methods

* update service and api; statediffing params are now defined by user through api rather than by service provider by cli

* update top level tests

* add ability to watch specific storage slots (leaf keys) only

* comments; explain logic

* update mainnet blocks test

* update api_test.go

* storage leafkey filter test

* cleanup chain maker

* adjust chain maker for tests to add an empty account in block1 and switch to EIP-158 afterwards (now we just need to generate enough accounts until one causes the empty account to be touched and removed post-EIP-158 so we can simulate and test that process...); also added 2 new blocks where more contract storage is set and old slots are set to zero so they are removed so we can test that

* found an account whose creation causes the empty account to be moved to a new path; this should count as 'touching; the empty account and cause it to be removed according to eip-158... but it doesn't

* use new contract in unit tests that has self-destruct ability, so we can test eip-158 since simply moving an account to new path doesn't count as 'touchin' it

* handle storage deletions

* tests for eip-158 account removal and storage value deletions; there is one edge case left to test where we remove 1 account when only two exist such that the remaining account is moved up and replaces the root branch node

* finish testing known edge cases

* add endpoint to fetch all state and storage nodes at a given blockheight; useful for generating a recent atate cache/snapshot that we can diff forward from rather than needing to collect all diffs from genesis

* test for state trie builder

* if statediffing is on, lock tries in triedb until the statediffing service signals they are done using them

* fix mock blockchain; golint; bump patch

* increase maxRequestContentLength; bump patch

* log the sizes of the state objects we are sending

* CI build (#20)

* CI: run build on PR and on push to master

* CI: debug building geth

* CI: fix coping file

* CI: fix coping file v2

* CI: temporary upload file to release asset

* CI: get release upload_url by tag, upload asset to current relase

* CI: fix tag name

* fix ci build on statediff_at_anyblock-1.9.11 branch

* fix publishing assets in release

* use context deadline for timeout in eth_call

* collect and emit codehash=>code mappings for state objects

* subscription endpoint for retrieving all the codehash=>code mappings that exist at provided height

* Implement WriteStateDiffAt

* Writes state diffs directly to postgres

* Adds CLI flags to configure PG

* Refactors builder output with callbacks

* Copies refactored postgres handling code from ipld-eth-indexer

* rename PostgresCIDWriter.{index->upsert}*

* go.mod update

* rm unused

* cleanup

* output code & codehash iteratively

* had to rf some types for this

* prometheus metrics output

* duplicate recent eth-indexer changes

* migrations and metrics...

* [wip] prom.Init() here? another CLI flag?

* tidy & DRY

* statediff WriteLoop service + CLI flag

* [wip] update test mocks

* todo - do something meaningful to test write loop

* logging

* use geth log

* port tests to go testing

* drop ginkgo/gomega

* fix and cleanup tests

* fail before defer statement

* delete vendor/ dir

* fixes after rebase onto 1.9.23

* fix API registration

* use golang 1.15.5 version (#34)

* bump version meta; add 0.0.11 branch to actions

* bump version meta; update github actions workflows

* statediff: refactor metrics

* Remove redundant statediff/indexer/prom tooling and use existing
prometheus integration.

* "indexer" namespace for metrics

* add reporting loop for db metrics

* doc

* metrics for statediff stats

* metrics namespace/subsystem = statediff/{indexer,service}

* statediff: use a worker pool (for direct writes)

* fix test

* fix chain event subscription

* log tweaks

* func name

* unused import

* intermediate chain event channel for metrics

* update github actions; linting

* add poststate and status to receipt ipld indexes

* stateDiffFor endpoints for fetching or writing statediff object by blockhash; bump statediff version

* fixes after rebase on to v1.10.1

* update github actions and version meta; go fmt

* add leaf key to removed 'nodes'

* include Postgres migrations and schema

* service documentation

* touching up

update github actions after rebase

fix connection leak (misplaced defer) and perform proper rollback on errs

improve error logging; handle PushBlock internal err

* build docker image and publish it to Docker Hub on release

* add access list tx to unit tests

* MarshalBinary and UnmarshalBinary methods for receipt

* fix error caused by 2718 by using MarshalBinary instead of EncodeRLP methods

* ipld encoding/decoding tests

* update TxModel; add AccessListElementModel

* index tx type and access lists

* add access list metrics

* unit tests for tx_type and access list table

* unit tests for receipt marshal/unmarshal binary methods

* improve documentation of the encoding methods

* fix issue identified in linting

.github/workflows/on-master.yaml

@@ -0,0 +1,30 @@
+name: Docker Build and publish to Github
+
+on:
+  push:
+    branches:
+      - v1.10.2-statediff
+      - v1.10.1-statediff
+      - v1.9.25-statediff
+      - v1.9.24-statediff
+      - v1.9.23-statediff
+      - v1.9.11-statediff
+
+jobs:
+  build:
+    name: Run docker build and publish
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Run docker build
+        run: docker build -t vulcanize/go-ethereum -f Dockerfile .
+      - name: Get the version
+        id: vars
+        run: echo ::set-output name=sha::$(echo ${GITHUB_SHA:0:7})
+      - name: Tag docker image
+        run: docker tag vulcanize/go-ethereum docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}}
+      - name: Docker Login
+        run: echo ${{ secrets.GITHUB_TOKEN }} | docker login https://docker.pkg.github.com -u vulcanize --password-stdin
+      - name: Docker Push
+        run: docker push docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}}
+

.github/workflows/on-pr.yml

@@ -0,0 +1,12 @@
+name: Docker Build
+
+on: [pull_request]
+
+jobs:
+  build:
+    name: Run docker build
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Run docker build
+        run: docker build -t vulcanize/go-ethereum .

.github/workflows/publish.yaml

@@ -0,0 +1,41 @@
+name: Publish geth to release
+on:
+  release:
+    types: [published]
+jobs:
+  push_to_registries:
+    name: Publish assets to Release
+    runs-on: ubuntu-latest
+    steps:
+      - name: Get the version
+        id: vars
+        run: |
+          echo ::set-output name=sha::$(echo ${GITHUB_SHA:0:7})
+          echo ::set-output name=tag::$(echo ${GITHUB_REF#refs/tags/})
+      - name: Docker Login to Github Registry
+        run: echo ${{ secrets.GITHUB_TOKEN }} | docker login https://docker.pkg.github.com -u vulcanize --password-stdin
+      - name: Docker Pull
+        run: docker pull docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}}
+      - name: Copy ethereum binary file
+        run: docker run --rm --entrypoint cat docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}} /usr/local/bin/geth > geth-linux-amd64
+      - name: Docker Login to Docker Registry
+        run: echo ${{ secrets.VULCANIZEJENKINS_PAT }} | docker login -u vulcanizejenkins --password-stdin
+      - name: Tag docker image
+        run: docker tag docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}} vulcanize/vdb-geth:${{steps.vars.outputs.tag}}
+      - name: Docker Push to Docker Hub
+        run: docker push vulcanize/vdb-geth:${{steps.vars.outputs.tag}}
+      - name: Get release
+        id: get_release
+        uses: bruceadams/get-release@v1.2.0
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - name: Upload Release Asset
+        id: upload-release-asset
+        uses: actions/upload-release-asset@v1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          upload_url: ${{ steps.get_release.outputs.upload_url }}
+          asset_path: geth-linux-amd64
+          asset_name: geth-linux-amd64
+          asset_content_type: application/octet-stream

Dockerfile.amd64

@@ -0,0 +1,7 @@
+# Build Geth in a stock Go builder container
+FROM golang:1.15.5 as builder
+
+#RUN apk add --no-cache make gcc musl-dev linux-headers git
+
+ADD . /go-ethereum
+RUN cd /go-ethereum && make geth

cmd/geth/config.go

@@ -25,6 +25,8 @@ import (
 	"reflect"
 	"unicode"
 
+	"github.com/ethereum/go-ethereum/eth/downloader"
+	"github.com/ethereum/go-ethereum/statediff"
 	"gopkg.in/urfave/cli.v1"
 
 	"github.com/ethereum/go-ethereum/cmd/utils"
@@ -134,6 +136,9 @@ func makeConfigNode(ctx *cli.Context) (*node.Node, gethConfig) {
 		cfg.Ethstats.URL = ctx.GlobalString(utils.EthStatsURLFlag.Name)
 	}
 	applyMetricConfig(ctx, &cfg)
+	if ctx.GlobalBool(utils.StateDiffFlag.Name) {
+		cfg.Eth.Diffing = true
+	}
 
 	return stack, cfg
 }
@@ -144,6 +149,11 @@ func makeFullNode(ctx *cli.Context) (*node.Node, ethapi.Backend) {
 	if ctx.GlobalIsSet(utils.OverrideBerlinFlag.Name) {
 		cfg.Eth.OverrideBerlin = new(big.Int).SetUint64(ctx.GlobalUint64(utils.OverrideBerlinFlag.Name))
 	}
+
+	if cfg.Eth.SyncMode == downloader.LightSync {
+		return makeLightNode(ctx, stack, cfg)
+	}
+
 	backend, eth := utils.RegisterEthService(stack, &cfg.Eth)
 
 	// Configure catalyst.
@@ -156,6 +166,34 @@ func makeFullNode(ctx *cli.Context) (*node.Node, ethapi.Backend) {
 		}
 	}
 
+	if ctx.GlobalBool(utils.StateDiffFlag.Name) {
+		var dbParams *statediff.DBParams
+		if ctx.GlobalIsSet(utils.StateDiffDBFlag.Name) {
+			dbParams = new(statediff.DBParams)
+			dbParams.ConnectionURL = ctx.GlobalString(utils.StateDiffDBFlag.Name)
+			if ctx.GlobalIsSet(utils.StateDiffDBNodeIDFlag.Name) {
+				dbParams.ID = ctx.GlobalString(utils.StateDiffDBNodeIDFlag.Name)
+			} else {
+				utils.Fatalf("Must specify node ID for statediff DB output")
+			}
+			if ctx.GlobalIsSet(utils.StateDiffDBClientNameFlag.Name) {
+				dbParams.ClientName = ctx.GlobalString(utils.StateDiffDBClientNameFlag.Name)
+			} else {
+				utils.Fatalf("Must specify client name for statediff DB output")
+			}
+		} else {
+			if ctx.GlobalBool(utils.StateDiffWritingFlag.Name) {
+				utils.Fatalf("Must pass DB parameters if enabling statediff write loop")
+			}
+		}
+		p := statediff.ServiceParams{
+			DBParams:        dbParams,
+			EnableWriteLoop: ctx.GlobalBool(utils.StateDiffWritingFlag.Name),
+			NumWorkers:      ctx.GlobalUint(utils.StateDiffWorkersFlag.Name),
+		}
+		utils.RegisterStateDiffService(stack, eth, &cfg.Eth, p)
+	}
+
 	// Configure GraphQL if requested
 	if ctx.GlobalIsSet(utils.GraphQLEnabledFlag.Name) {
 		utils.RegisterGraphQLService(stack, backend, cfg.Node)
@@ -167,6 +205,20 @@ func makeFullNode(ctx *cli.Context) (*node.Node, ethapi.Backend) {
 	return stack, backend
 }
 
+func makeLightNode(ctx *cli.Context, stack *node.Node, cfg gethConfig) (*node.Node, ethapi.Backend) {
+	backend := utils.RegisterLesEthService(stack, &cfg.Eth)
+
+	// Configure GraphQL if requested
+	if ctx.GlobalIsSet(utils.GraphQLEnabledFlag.Name) {
+		utils.RegisterGraphQLService(stack, backend.ApiBackend, cfg.Node)
+	}
+	// Add the Ethereum Stats daemon if requested.
+	if cfg.Ethstats.URL != "" {
+		utils.RegisterEthStatsService(stack, backend.ApiBackend, cfg.Ethstats.URL)
+	}
+	return stack, backend.ApiBackend
+}
+
 // dumpConfig is the dumpconfig command.
 func dumpConfig(ctx *cli.Context) error {
 	_, cfg := makeConfigNode(ctx)

cmd/geth/main.go

@@ -150,6 +150,12 @@ var (
 		utils.EWASMInterpreterFlag,
 		utils.EVMInterpreterFlag,
 		utils.MinerNotifyFullFlag,
+		utils.StateDiffFlag,
+		utils.StateDiffDBFlag,
+		utils.StateDiffDBNodeIDFlag,
+		utils.StateDiffDBClientNameFlag,
+		utils.StateDiffWritingFlag,
+		utils.StateDiffWorkersFlag,
 		configFileFlag,
 		utils.CatalystFlag,
 	}

cmd/geth/usage.go

@@ -229,6 +229,17 @@ var AppHelpFlagGroups = []flags.FlagGroup{
 			utils.LegacyRPCApiFlag,
 		},
 	},
+	{
+		Name: "STATE DIFF",
+		Flags: []cli.Flag{
+			utils.StateDiffFlag,
+			utils.StateDiffDBFlag,
+			utils.StateDiffDBNodeIDFlag,
+			utils.StateDiffDBClientNameFlag,
+			utils.StateDiffWritingFlag,
+			utils.StateDiffWorkersFlag,
+		},
+	},
 	{
 		Name: "MISC",
 		Flags: []cli.Flag{

cmd/utils/flags.go

@@ -66,6 +66,8 @@ import (
 	"github.com/ethereum/go-ethereum/p2p/nat"
 	"github.com/ethereum/go-ethereum/p2p/netutil"
 	"github.com/ethereum/go-ethereum/params"
+	"github.com/ethereum/go-ethereum/statediff"
+
 	pcsclite "github.com/gballet/go-libpcsclite"
 	gopsutil "github.com/shirou/gopsutil/mem"
 	"gopkg.in/urfave/cli.v1"
@@ -760,6 +762,30 @@ var (
 		Name:  "catalyst",
 		Usage: "Catalyst mode (eth2 integration testing)",
 	}
+	StateDiffFlag = cli.BoolFlag{
+		Name:  "statediff",
+		Usage: "Enables the processing of state diffs between each block",
+	}
+	StateDiffDBFlag = cli.StringFlag{
+		Name:  "statediff.db",
+		Usage: "PostgreSQL database connection string for writing state diffs",
+	}
+	StateDiffDBNodeIDFlag = cli.StringFlag{
+		Name:  "statediff.dbnodeid",
+		Usage: "Node ID to use when writing state diffs to database",
+	}
+	StateDiffDBClientNameFlag = cli.StringFlag{
+		Name:  "statediff.dbclientname",
+		Usage: "Client name to use when writing state diffs to database",
+	}
+	StateDiffWritingFlag = cli.BoolFlag{
+		Name:  "statediff.writing",
+		Usage: "Activates progressive writing of state diffs to database as new block are synced",
+	}
+	StateDiffWorkersFlag = cli.UintFlag{
+		Name:  "statediff.workers",
+		Usage: "Number of concurrent workers to use during statediff processing (0 = 1)",
+	}
 )
 
 // MakeDataDir retrieves the currently requested data directory, terminating
@@ -1000,6 +1026,10 @@ func setWS(ctx *cli.Context, cfg *node.Config) {
 	if ctx.GlobalIsSet(WSPathPrefixFlag.Name) {
 		cfg.WSPathPrefix = ctx.GlobalString(WSPathPrefixFlag.Name)
 	}
+
+	if ctx.GlobalBool(StateDiffFlag.Name) {
+		cfg.WSModules = append(cfg.WSModules, "statediff")
+	}
 }
 
 // setIPC creates an IPC path configuration from the set command line flags,
@@ -1720,6 +1750,15 @@ func RegisterEthService(stack *node.Node, cfg *ethconfig.Config) (ethapi.Backend
 	return backend.APIBackend, backend
 }
 
+// RegisterLesEthService adds an Ethereum les client to the stack.
+func RegisterLesEthService(stack *node.Node, cfg *eth.Config) *les.LightEthereum {
+	backend, err := les.New(stack, cfg)
+	if err != nil {
+		Fatalf("Failed to register the Ethereum service: %v", err)
+	}
+	return backend
+}
+
 // RegisterEthStatsService configures the Ethereum Stats daemon and adds it to
 // the given node.
 func RegisterEthStatsService(stack *node.Node, backend ethapi.Backend, url string) {
@@ -1735,6 +1774,13 @@ func RegisterGraphQLService(stack *node.Node, backend ethapi.Backend, cfg node.C
 	}
 }
 
+// RegisterStateDiffService configures and registers a service to stream state diff data over RPC
+func RegisterStateDiffService(stack *node.Node, ethServ *eth.Ethereum, cfg *ethconfig.Config, params statediff.ServiceParams) {
+	if err := statediff.New(stack, ethServ, cfg, params); err != nil {
+		Fatalf("Failed to register the Statediff service: %v", err)
+	}
+}
+
 func SetupMetrics(ctx *cli.Context) {
 	if metrics.Enabled {
 		log.Info("Enabling metrics collection")

core/blockchain.go

@@ -131,6 +131,7 @@ type CacheConfig struct {
 	Preimages           bool          // Whether to store preimage of trie key to the disk
 
 	SnapshotWait bool // Wait for snapshot construction on startup. TODO(karalabe): This is a dirty hack for testing, nuke it
+	StateDiffing bool // Whether or not the statediffing service is running
 }
 
 // defaultCacheConfig are the default caching values if none are specified by the
@@ -209,6 +210,10 @@ type BlockChain struct {
 
 	shouldPreserve  func(*types.Block) bool        // Function used to determine whether should preserve the given block.
 	terminateInsert func(common.Hash, uint64) bool // Testing hook used to terminate ancient receipt chain insertion.
+
+	// Locked roots and their mutex
+	trieLock    sync.Mutex
+	lockedRoots map[common.Hash]bool
 }
 
 // NewBlockChain returns a fully initialised block chain using information
@@ -245,6 +250,7 @@ func NewBlockChain(db ethdb.Database, cacheConfig *CacheConfig, chainConfig *par
 		futureBlocks:   futureBlocks,
 		engine:         engine,
 		vmConfig:       vmConfig,
+		lockedRoots:    make(map[common.Hash]bool),
 	}
 	bc.validator = NewBlockValidator(chainConfig, bc, engine)
 	bc.prefetcher = newStatePrefetcher(chainConfig, bc, engine)
@@ -1031,7 +1037,10 @@ func (bc *BlockChain) Stop() {
 			}
 		}
 		for !bc.triegc.Empty() {
-			triedb.Dereference(bc.triegc.PopItem().(common.Hash))
+			pruneRoot := bc.triegc.PopItem().(common.Hash)
+			if !bc.TrieLocked(pruneRoot) {
+				triedb.Dereference(pruneRoot)
+			}
 		}
 		if size, _ := triedb.Size(); size != 0 {
 			log.Error("Dangling trie nodes after full cleanup")
@@ -1488,6 +1497,11 @@ func (bc *BlockChain) writeBlockWithState(block *types.Block, receipts []*types.
 		triedb.Reference(root, common.Hash{}) // metadata reference to keep trie alive
 		bc.triegc.Push(root, -int64(block.NumberU64()))
 
+		// If we are statediffing, lock the trie until the statediffing service is done using it
+		if bc.cacheConfig.StateDiffing {
+			bc.LockTrie(root)
+		}
+
 		if current := block.NumberU64(); current > TriesInMemory {
 			// If we exceeded our memory allowance, flush matured singleton nodes to disk
 			var (
@@ -1526,7 +1540,11 @@ func (bc *BlockChain) writeBlockWithState(block *types.Block, receipts []*types.
 					bc.triegc.Push(root, number)
 					break
 				}
-				triedb.Dereference(root.(common.Hash))
+				pruneRoot := root.(common.Hash)
+				if !bc.TrieLocked(pruneRoot) {
+					log.Debug("Dereferencing", "root", root.(common.Hash).Hex())
+					triedb.Dereference(pruneRoot)
+				}
 			}
 		}
 	}
@@ -2510,3 +2528,28 @@ func (bc *BlockChain) SubscribeLogsEvent(ch chan<- []*types.Log) event.Subscript
 func (bc *BlockChain) SubscribeBlockProcessingEvent(ch chan<- bool) event.Subscription {
 	return bc.scope.Track(bc.blockProcFeed.Subscribe(ch))
 }
+
+// TrieLocked returns whether the trie associated with the provided root is locked for use
+func (bc *BlockChain) TrieLocked(root common.Hash) bool {
+	bc.trieLock.Lock()
+	locked, ok := bc.lockedRoots[root]
+	bc.trieLock.Unlock()
+	if !ok {
+		return false
+	}
+	return locked
+}
+
+// LockTrie prevents dereferencing of the provided root
+func (bc *BlockChain) LockTrie(root common.Hash) {
+	bc.trieLock.Lock()
+	bc.lockedRoots[root] = true
+	bc.trieLock.Unlock()
+}
+
+// UnlockTrie allows dereferencing of the provided root- provided it was previously locked
+func (bc *BlockChain) UnlockTrie(root common.Hash) {
+	bc.trieLock.Lock()
+	bc.lockedRoots[root] = false
+	bc.trieLock.Unlock()
+}

core/types/receipt.go

@@ -136,6 +136,9 @@ func NewReceipt(root []byte, failed bool, cumulativeGasUsed uint64) *Receipt {
 
 // EncodeRLP implements rlp.Encoder, and flattens the consensus fields of a receipt
 // into an RLP stream. If no post state is present, byzantium fork is assumed.
+// For a legacy Receipt this returns RLP([PostStateOrStatus, CumulativeGasUsed, Bloom, Logs])
+// For a EIP-2718 Receipt this returns RLP(TxType || ReceiptPayload)
+// For a EIP-2930 Receipt, TxType == 0x01 and ReceiptPayload == RLP([PostStateOrStatus, CumulativeGasUsed, Bloom, Logs])
 func (r *Receipt) EncodeRLP(w io.Writer) error {
 	data := &receiptRLP{r.statusEncoding(), r.CumulativeGasUsed, r.Bloom, r.Logs}
 	if r.Type == LegacyTxType {
@@ -148,13 +151,34 @@ func (r *Receipt) EncodeRLP(w io.Writer) error {
 	buf := encodeBufferPool.Get().(*bytes.Buffer)
 	defer encodeBufferPool.Put(buf)
 	buf.Reset()
-	buf.WriteByte(r.Type)
+	if err := r.encodeTyped(data, buf); err != nil {
-	if err := rlp.Encode(buf, data); err != nil {
 		return err
 	}
 	return rlp.Encode(w, buf.Bytes())
 }
 
+// encodeTyped writes the canonical encoding of a typed receipt to w.
+func (r *Receipt) encodeTyped(data *receiptRLP, w *bytes.Buffer) error {
+	w.WriteByte(r.Type)
+	return rlp.Encode(w, data)
+}
+
+// MarshalBinary returns the canonical consensus encoding of the receipt.
+// For a legacy Receipt this returns RLP([PostStateOrStatus, CumulativeGasUsed, Bloom, Logs])
+// For a EIP-2718 Receipt this returns TxType || ReceiptPayload
+// For a EIP-2930, TxType == 0x01 and ReceiptPayload == RLP([PostStateOrStatus, CumulativeGasUsed, Bloom, Logs])
+func (r *Receipt) MarshalBinary() ([]byte, error) {
+	if r.Type == LegacyTxType {
+		return rlp.EncodeToBytes(r)
+	}
+	data := &receiptRLP{r.statusEncoding(), r.CumulativeGasUsed, r.Bloom, r.Logs}
+	buf := encodeBufferPool.Get().(*bytes.Buffer)
+	defer encodeBufferPool.Put(buf)
+	buf.Reset()
+	err := r.encodeTyped(data, buf)
+	return buf.Bytes(), err
+}
+
 // DecodeRLP implements rlp.Decoder, and loads the consensus fields of a receipt
 // from an RLP stream.
 func (r *Receipt) DecodeRLP(s *rlp.Stream) error {
@@ -193,6 +217,42 @@ func (r *Receipt) DecodeRLP(s *rlp.Stream) error {
 	}
 }
 
+// UnmarshalBinary decodes the canonical encoding of receipts.
+// It supports legacy RLP receipts and EIP-2718 typed receipts.
+func (r *Receipt) UnmarshalBinary(b []byte) error {
+	if len(b) > 0 && b[0] > 0x7f {
+		// It's a legacy receipt decode the RLP
+		var data receiptRLP
+		err := rlp.DecodeBytes(b, &data)
+		if err != nil {
+			return err
+		}
+		r.Type = LegacyTxType
+		return r.setFromRLP(data)
+	}
+	// It's an EIP2718 typed transaction envelope.
+	return r.decodeTyped(b)
+}
+
+// decodeTyped decodes a typed receipt from the canonical format.
+func (r *Receipt) decodeTyped(b []byte) error {
+	if len(b) == 0 {
+		return errEmptyTypedReceipt
+	}
+	switch b[0] {
+	case AccessListTxType:
+		var data receiptRLP
+		err := rlp.DecodeBytes(b[1:], &data)
+		if err != nil {
+			return err
+		}
+		r.Type = AccessListTxType
+		return r.setFromRLP(data)
+	default:
+		return ErrTxTypeNotSupported
+	}
+}
+
 func (r *Receipt) setFromRLP(data receiptRLP) error {
 	r.CumulativeGasUsed, r.Bloom, r.Logs = data.CumulativeGasUsed, data.Bloom, data.Logs
 	return r.setStatus(data.PostStateOrStatus)
@@ -355,42 +415,42 @@ func (rs Receipts) EncodeIndex(i int, w *bytes.Buffer) {
 
 // DeriveFields fills the receipts with their computed fields based on consensus
 // data and contextual infos like containing block and transactions.
-func (r Receipts) DeriveFields(config *params.ChainConfig, hash common.Hash, number uint64, txs Transactions) error {
+func (rs Receipts) DeriveFields(config *params.ChainConfig, hash common.Hash, number uint64, txs Transactions) error {
 	signer := MakeSigner(config, new(big.Int).SetUint64(number))
 
 	logIndex := uint(0)
-	if len(txs) != len(r) {
+	if len(txs) != len(rs) {
 		return errors.New("transaction and receipt count mismatch")
 	}
-	for i := 0; i < len(r); i++ {
+	for i := 0; i < len(rs); i++ {
 		// The transaction type and hash can be retrieved from the transaction itself
-		r[i].Type = txs[i].Type()
+		rs[i].Type = txs[i].Type()
-		r[i].TxHash = txs[i].Hash()
+		rs[i].TxHash = txs[i].Hash()
 
 		// block location fields
-		r[i].BlockHash = hash
+		rs[i].BlockHash = hash
-		r[i].BlockNumber = new(big.Int).SetUint64(number)
+		rs[i].BlockNumber = new(big.Int).SetUint64(number)
-		r[i].TransactionIndex = uint(i)
+		rs[i].TransactionIndex = uint(i)
 
 		// The contract address can be derived from the transaction itself
 		if txs[i].To() == nil {
 			// Deriving the signer is expensive, only do if it's actually needed
 			from, _ := Sender(signer, txs[i])
-			r[i].ContractAddress = crypto.CreateAddress(from, txs[i].Nonce())
+			rs[i].ContractAddress = crypto.CreateAddress(from, txs[i].Nonce())
 		}
 		// The used gas can be calculated based on previous r
 		if i == 0 {
-			r[i].GasUsed = r[i].CumulativeGasUsed
+			rs[i].GasUsed = rs[i].CumulativeGasUsed
 		} else {
-			r[i].GasUsed = r[i].CumulativeGasUsed - r[i-1].CumulativeGasUsed
+			rs[i].GasUsed = rs[i].CumulativeGasUsed - rs[i-1].CumulativeGasUsed
 		}
 		// The derived log fields can simply be set from the block and transaction
-		for j := 0; j < len(r[i].Logs); j++ {
+		for j := 0; j < len(rs[i].Logs); j++ {
-			r[i].Logs[j].BlockNumber = number
+			rs[i].Logs[j].BlockNumber = number
-			r[i].Logs[j].BlockHash = hash
+			rs[i].Logs[j].BlockHash = hash
-			r[i].Logs[j].TxHash = r[i].TxHash
+			rs[i].Logs[j].TxHash = rs[i].TxHash
-			r[i].Logs[j].TxIndex = uint(i)
+			rs[i].Logs[j].TxIndex = uint(i)
-			r[i].Logs[j].Index = logIndex
+			rs[i].Logs[j].Index = logIndex
 			logIndex++
 		}
 	}

core/types/receipt_test.go

@@ -29,6 +29,42 @@ import (
 	"github.com/ethereum/go-ethereum/rlp"
 )
 
+var (
+	legacyReceipt = &Receipt{
+		Status:            ReceiptStatusFailed,
+		CumulativeGasUsed: 1,
+		Logs: []*Log{
+			{
+				Address: common.BytesToAddress([]byte{0x11}),
+				Topics:  []common.Hash{common.HexToHash("dead"), common.HexToHash("beef")},
+				Data:    []byte{0x01, 0x00, 0xff},
+			},
+			{
+				Address: common.BytesToAddress([]byte{0x01, 0x11}),
+				Topics:  []common.Hash{common.HexToHash("dead"), common.HexToHash("beef")},
+				Data:    []byte{0x01, 0x00, 0xff},
+			},
+		},
+	}
+	accessListReceipt = &Receipt{
+		Status:            ReceiptStatusFailed,
+		CumulativeGasUsed: 1,
+		Logs: []*Log{
+			{
+				Address: common.BytesToAddress([]byte{0x11}),
+				Topics:  []common.Hash{common.HexToHash("dead"), common.HexToHash("beef")},
+				Data:    []byte{0x01, 0x00, 0xff},
+			},
+			{
+				Address: common.BytesToAddress([]byte{0x01, 0x11}),
+				Topics:  []common.Hash{common.HexToHash("dead"), common.HexToHash("beef")},
+				Data:    []byte{0x01, 0x00, 0xff},
+			},
+		},
+		Type: AccessListTxType,
+	}
+)
+
 func TestDecodeEmptyTypedReceipt(t *testing.T) {
 	input := []byte{0x80}
 	var r Receipt
@@ -117,6 +153,76 @@ func TestLegacyReceiptDecoding(t *testing.T) {
 	}
 }
 
+func TestReceiptMarshalBinary(t *testing.T) {
+	// Legacy Receipt
+	legacyReceipt.Bloom = CreateBloom(Receipts{legacyReceipt})
+	have, err := legacyReceipt.MarshalBinary()
+	if err != nil {
+		t.Fatalf("marshal binary error: %v", err)
+	}
+	legacyReceipts := Receipts{legacyReceipt}
+	buf := new(bytes.Buffer)
+	legacyReceipts.EncodeIndex(0, buf)
+	haveEncodeIndex := buf.Bytes()
+	if !bytes.Equal(have, haveEncodeIndex) {
+		t.Errorf("BinaryMarshal and EncodeIndex mismatch, got %x want %x", have, haveEncodeIndex)
+	}
+	buf.Reset()
+	if err := legacyReceipt.EncodeRLP(buf); err != nil {
+		t.Fatalf("encode rlp error: %v", err)
+	}
+	haveRLPEncode := buf.Bytes()
+	if !bytes.Equal(have, haveRLPEncode) {
+		t.Errorf("BinaryMarshal and EncodeRLP mismatch for legacy tx, got %x want %x", have, haveRLPEncode)
+	}
+	legacyWant := common.FromHex("f901c58001b9010000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000500000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000000000000000010000080000000000000000000004000000000000000000000000000040000000000000000000000000000800000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000f8bef85d940000000000000000000000000000000000000011f842a0000000000000000000000000000000000000000000000000000000000000deada0000000000000000000000000000000000000000000000000000000000000beef830100fff85d940000000000000000000000000000000000000111f842a0000000000000000000000000000000000000000000000000000000000000deada0000000000000000000000000000000000000000000000000000000000000beef830100ff")
+	if !bytes.Equal(have, legacyWant) {
+		t.Errorf("encoded RLP mismatch, got %x want %x", have, legacyWant)
+	}
+
+	// 2930 Receipt
+	buf.Reset()
+	accessListReceipt.Bloom = CreateBloom(Receipts{accessListReceipt})
+	have, err = accessListReceipt.MarshalBinary()
+	if err != nil {
+		t.Fatalf("marshal binary error: %v", err)
+	}
+	accessListReceipts := Receipts{accessListReceipt}
+	accessListReceipts.EncodeIndex(0, buf)
+	haveEncodeIndex = buf.Bytes()
+	if !bytes.Equal(have, haveEncodeIndex) {
+		t.Errorf("BinaryMarshal and EncodeIndex mismatch, got %x want %x", have, haveEncodeIndex)
+	}
+	accessListWant := common.FromHex("01f901c58001b9010000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000500000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000000000000000010000080000000000000000000004000000000000000000000000000040000000000000000000000000000800000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000f8bef85d940000000000000000000000000000000000000011f842a0000000000000000000000000000000000000000000000000000000000000deada0000000000000000000000000000000000000000000000000000000000000beef830100fff85d940000000000000000000000000000000000000111f842a0000000000000000000000000000000000000000000000000000000000000deada0000000000000000000000000000000000000000000000000000000000000beef830100ff")
+	if !bytes.Equal(have, accessListWant) {
+		t.Errorf("encoded RLP mismatch, got %x want %x", have, accessListWant)
+	}
+}
+
+func TestReceiptUnmarshalBinary(t *testing.T) {
+	// Legacy Receipt
+	legacyBinary := common.FromHex("f901c58001b9010000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000500000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000000000000000010000080000000000000000000004000000000000000000000000000040000000000000000000000000000800000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000f8bef85d940000000000000000000000000000000000000011f842a0000000000000000000000000000000000000000000000000000000000000deada0000000000000000000000000000000000000000000000000000000000000beef830100fff85d940000000000000000000000000000000000000111f842a0000000000000000000000000000000000000000000000000000000000000deada0000000000000000000000000000000000000000000000000000000000000beef830100ff")
+	gotLegacyReceipt := new(Receipt)
+	if err := gotLegacyReceipt.UnmarshalBinary(legacyBinary); err != nil {
+		t.Fatalf("unmarshal binary error: %v", err)
+	}
+	legacyReceipt.Bloom = CreateBloom(Receipts{legacyReceipt})
+	if !reflect.DeepEqual(gotLegacyReceipt, legacyReceipt) {
+		t.Errorf("receipt unmarshalled from binary mismatch, got %v want %v", gotLegacyReceipt, legacyReceipt)
+	}
+
+	// 2930 Receipt
+	accessListBinary := common.FromHex("01f901c58001b9010000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000500000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000000000000000010000080000000000000000000004000000000000000000000000000040000000000000000000000000000800000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000f8bef85d940000000000000000000000000000000000000011f842a0000000000000000000000000000000000000000000000000000000000000deada0000000000000000000000000000000000000000000000000000000000000beef830100fff85d940000000000000000000000000000000000000111f842a0000000000000000000000000000000000000000000000000000000000000deada0000000000000000000000000000000000000000000000000000000000000beef830100ff")
+	gotAccessListReceipt := new(Receipt)
+	if err := gotAccessListReceipt.UnmarshalBinary(accessListBinary); err != nil {
+		t.Fatalf("unmarshal binary error: %v", err)
+	}
+	accessListReceipt.Bloom = CreateBloom(Receipts{accessListReceipt})
+	if !reflect.DeepEqual(gotAccessListReceipt, accessListReceipt) {
+		t.Errorf("receipt unmarshalled from binary mismatch, got %v want %v", gotAccessListReceipt, accessListReceipt)
+	}
+}
+
 func encodeAsStoredReceiptRLP(want *Receipt) ([]byte, error) {
 	stored := &storedReceiptRLP{
 		PostStateOrStatus: want.statusEncoding(),

core/types/transaction.go

@@ -83,6 +83,9 @@ type TxData interface {
 }
 
 // EncodeRLP implements rlp.Encoder
+// For a legacy Transaction this returns RLP([AccountNonce, GasPrice, GasLimit, Recipient, Amount, Data, V, R, S])
+// For a EIP-2718 Transaction this returns RLP(TxType || TxPayload)
+// For a EIP-2930 Transaction, TxType == 0x01 and TxPayload == RLP([ChainID, AccountNonce, GasPrice, GasLimit, Recipient, Amount, Data, AccessList, V, R, S]
 func (tx *Transaction) EncodeRLP(w io.Writer) error {
 	if tx.Type() == LegacyTxType {
 		return rlp.Encode(w, tx.inner)
@@ -103,9 +106,10 @@ func (tx *Transaction) encodeTyped(w *bytes.Buffer) error {
 	return rlp.Encode(w, tx.inner)
 }
 
-// MarshalBinary returns the canonical encoding of the transaction.
+// MarshalBinary returns the canonical consensus encoding of the transaction.
-// For legacy transactions, it returns the RLP encoding. For EIP-2718 typed
+// For a legacy Transaction this returns RLP([AccountNonce, GasPrice, GasLimit, Recipient, Amount, Data, V, R, S])
-// transactions, it returns the type and payload.
+// For a EIP-2718 Transaction this returns TxType || TxPayload
+// For a EIP-2930 Transaction, TxType == 0x01 and TxPayload == RLP([ChainID, AccountNonce, GasPrice, GasLimit, Recipient, Amount, Data, AccessList, V, R, S]
 func (tx *Transaction) MarshalBinary() ([]byte, error) {
 	if tx.Type() == LegacyTxType {
 		return rlp.EncodeToBytes(tx.inner)

eth/backend.go

@@ -188,6 +188,7 @@ func New(stack *node.Node, config *ethconfig.Config) (*Ethereum, error) {
 			TrieTimeLimit:       config.TrieTimeout,
 			SnapshotLimit:       config.SnapshotCache,
 			Preimages:           config.Preimages,
+			StateDiffing:        config.Diffing,
 		}
 	)
 	eth.blockchain, err = core.NewBlockChain(chainDb, cacheConfig, chainConfig, eth.engine, vmConfig, eth.shouldPreserve, &config.TxLookupLimit)

eth/ethconfig/config.go

@@ -201,6 +201,10 @@ type Config struct {
 
 	// Berlin block override (TODO: remove after the fork)
 	OverrideBerlin *big.Int `toml:",omitempty"`
+
+	// Signify whether or not we are producing statediffs
+	// If we are, do not dereference state roots until the statediffing service is done with them
+	Diffing bool
 }
 
 // CreateConsensusEngine creates a consensus engine for the given chain configuration.

go.mod

@@ -39,13 +39,21 @@ require (
 	github.com/holiman/uint256 v1.1.1
 	github.com/huin/goupnp v1.0.1-0.20210310174557-0ca763054c88
 	github.com/influxdata/influxdb v1.8.3
+	github.com/ipfs/go-block-format v0.0.2
+	github.com/ipfs/go-cid v0.0.7
+	github.com/ipfs/go-ipfs-blockstore v1.0.1
+	github.com/ipfs/go-ipfs-ds-help v1.0.0
+	github.com/ipfs/go-ipld-format v0.2.0
 	github.com/jackpal/go-nat-pmp v1.0.2-0.20160603034137-1fa385a6f458
 	github.com/jedisct1/go-minisign v0.0.0-20190909160543-45766022959e
+	github.com/jmoiron/sqlx v1.2.0
 	github.com/julienschmidt/httprouter v1.2.0
 	github.com/karalabe/usb v0.0.0-20190919080040-51dc0efba356
 	github.com/kylelemons/godebug v1.1.0 // indirect
-	github.com/mattn/go-colorable v0.1.0
+	github.com/lib/pq v1.8.0
-	github.com/mattn/go-isatty v0.0.5-0.20180830101745-3fb116b82035
+	github.com/mattn/go-colorable v0.1.1
+	github.com/mattn/go-isatty v0.0.5
+	github.com/multiformats/go-multihash v0.0.14
 	github.com/naoina/go-stringutil v0.1.0 // indirect
 	github.com/naoina/toml v0.1.2-0.20170918210437-9fafd6967416
 	github.com/olekukonko/tablewriter v0.0.5

go.sum

@@ -149,11 +149,15 @@ github.com/go-ole/go-ole v1.2.1 h1:2lOsA72HgjxAuMlKpFiCbHTvu44PIVkZ5hqm3RSdI/E=
 github.com/go-ole/go-ole v1.2.1/go.mod h1:7FAglXiTm7HKlQRDeOQ6ZNUHidzCWXuZWq/1dTyBNF8=
 github.com/go-sourcemap/sourcemap v2.1.2+incompatible h1:0b/xya7BKGhXuqFESKM4oIiRo9WOt2ebz7KxfreD6ug=
 github.com/go-sourcemap/sourcemap v2.1.2+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg=
+github.com/go-sql-driver/mysql v1.4.0/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
+github.com/go-sql-driver/mysql v1.4.1 h1:g24URVg0OFbNUTx9qqY1IRZ9D9z3iPyi5zKhQZpNwpA=
 github.com/go-sql-driver/mysql v1.4.1/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
 github.com/go-stack/stack v1.8.0 h1:5SgMzNM5HxrEjV0ww2lTmX6E2Izsfxas4+YHWRs3Lsk=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
 github.com/gofrs/uuid v3.3.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
+github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
+github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls=
 github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o=
 github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0/go.mod h1:E/TSTwGwJL78qG/PmXZO1EjYhfJinVAhrmmHX6Z8B9k=
 github.com/golang/geo v0.0.0-20190916061304-5b978397cfec/go.mod h1:QZ0nwyI2jOfgRAoBvP+ab5aRr7c9x7lhGEJrKvBwjWI=
@@ -195,6 +199,7 @@ github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OI
 github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
 github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
+github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/uuid v1.1.5 h1:kxhtnfFVi+rYdOALN0B3k9UT86zVJKfBimRaciULW4I=
 github.com/google/uuid v1.1.5/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
@@ -204,8 +209,27 @@ github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0U
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/graph-gophers/graphql-go v0.0.0-20201113091052-beb923fada29 h1:sezaKhEfPFg8W0Enm61B9Gs911H8iesGY5R8NDPtd1M=
 github.com/graph-gophers/graphql-go v0.0.0-20201113091052-beb923fada29/go.mod h1:9CQHMSxwO4MprSdzoIEobiHpoLtHm77vfxsvsIN5Vuc=
+github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
+github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
+github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
+github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU=
+github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48=
+github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q=
+github.com/hashicorp/consul/sdk v0.1.1/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8=
+github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
+github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
+github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
+github.com/hashicorp/go-msgpack v0.5.3/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM=
+github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk=
+github.com/hashicorp/go-rootcerts v1.0.0/go.mod h1:K6zTfqpRlCUIjkwsN4Z+hiSfzSTQa6eBIzfwKfwNnHU=
+github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU=
+github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdvsLplgctolz4=
+github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
+github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
+github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90=
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
 github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d h1:dg1dEPuWpEqDnvIw251EVy4zlP8gWbsGj4BsUKCRpYs=
 github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
 github.com/holiman/bloomfilter/v2 v2.0.3 h1:73e0e/V0tCydx14a0SCYS/EWCxgwLZ18CZcZKVu0fao=
@@ -227,13 +251,43 @@ github.com/influxdata/promql/v2 v2.12.0/go.mod h1:fxOPu+DY0bqCTCECchSRtWfc+0X19y
 github.com/influxdata/roaring v0.4.13-0.20180809181101-fc520f41fab6/go.mod h1:bSgUQ7q5ZLSO+bKBGqJiCBGAl+9DxyW63zLTujjUlOE=
 github.com/influxdata/tdigest v0.0.0-20181121200506-bf2b5ad3c0a9/go.mod h1:Js0mqiSBE6Ffsg94weZZ2c+v/ciT8QRHFOap7EKDrR0=
 github.com/influxdata/usage-client v0.0.0-20160829180054-6d3895376368/go.mod h1:Wbbw6tYNvwa5dlB6304Sd+82Z3f7PmVZHVKU637d4po=
+github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
+github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
+github.com/ipfs/go-block-format v0.0.2 h1:qPDvcP19izTjU8rgo6p7gTXZlkMkF5bz5G3fqIsSCPE=
+github.com/ipfs/go-block-format v0.0.2/go.mod h1:AWR46JfpcObNfg3ok2JHDUfdiHRgWhJgCQF+KIgOPJY=
+github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
+github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
+github.com/ipfs/go-cid v0.0.5/go.mod h1:plgt+Y5MnOey4vO4UlUazGqdbEXuFYitED67FexhXog=
+github.com/ipfs/go-cid v0.0.7 h1:ysQJVJA3fNDF1qigJbsSQOdjhVLsOEoPdh0+R97k3jY=
+github.com/ipfs/go-cid v0.0.7/go.mod h1:6Ux9z5e+HpkQdckYoX1PG/6xqKspzlEIR5SDmgqgC/I=
+github.com/ipfs/go-datastore v0.4.1/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA=
+github.com/ipfs/go-datastore v0.4.2 h1:h8/n7WPzhp239kkLws+epN3Ic7YtcBPgcaXfEfdVDWM=
+github.com/ipfs/go-datastore v0.4.2/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA=
+github.com/ipfs/go-ipfs-blockstore v1.0.1 h1:fnuVj4XdZp4yExhd0CnUwAiMNJHiPnfInhiuwz4lW1w=
+github.com/ipfs/go-ipfs-blockstore v1.0.1/go.mod h1:MGNZlHNEnR4KGgPHM3/k8lBySIOK2Ve+0KjZubKlaOE=
+github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw=
+github.com/ipfs/go-ipfs-ds-help v1.0.0 h1:bEQ8hMGs80h0sR8O4tfDgV6B01aaF9qeTrujrTLYV3g=
+github.com/ipfs/go-ipfs-ds-help v1.0.0/go.mod h1:ujAbkeIgkKAWtxxNkoZHWLCyk5JpPoKnGyCcsoF6ueE=
+github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50=
+github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc=
+github.com/ipfs/go-ipld-format v0.2.0 h1:xGlJKkArkmBvowr+GMCX0FEZtkro71K1AwiKnL37mwA=
+github.com/ipfs/go-ipld-format v0.2.0/go.mod h1:3l3C1uKoadTPbeNfrDi+xMInYKlx2Cvg1BuydPSdzQs=
+github.com/ipfs/go-log v0.0.1 h1:9XTUN/rW64BCG1YhPK9Hoy3q8nr4gOmHHBpgFdfw6Lc=
+github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM=
+github.com/ipfs/go-metrics-interface v0.0.1 h1:j+cpbjYvu4R8zbleSs36gvB7jR+wsL2fGD6n0jO4kdg=
+github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j/b/tL7HTWtJ4VPgWY=
 github.com/jackpal/go-nat-pmp v1.0.2-0.20160603034137-1fa385a6f458 h1:6OvNmYgJyexcZ3pYbTI9jWx5tHo1Dee/tWbLMfPe2TA=
 github.com/jackpal/go-nat-pmp v1.0.2-0.20160603034137-1fa385a6f458/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc=
+github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 h1:bspPhN+oKYFk5fcGNuQzp6IGzYQSenLEgH3s6jkXrWw=
+github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8/go.mod h1:Ly/wlsjFq/qrU3Rar62tu1gASgGw6chQbSh/XgIIXCY=
 github.com/jedisct1/go-minisign v0.0.0-20190909160543-45766022959e h1:UvSe12bq+Uj2hWd8aOlwPmoZ+CITRFrdit+sDGfAg8U=
 github.com/jedisct1/go-minisign v0.0.0-20190909160543-45766022959e/go.mod h1:G1CVv03EnqU1wYL2dFwXxW2An0az9JTl/ZsqXQeBlkU=
 github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
 github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
 github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
+github.com/jmoiron/sqlx v1.2.0 h1:41Ip0zITnmWNR/vHV+S4m+VoUivnWY5E4OJfLZjCJMA=
+github.com/jmoiron/sqlx v1.2.0/go.mod h1:1FEQNm3xlJgrMD+FBdI9+xvCksHtbpVBBw5dYhBSsks=
+github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ=
 github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
 github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
@@ -246,6 +300,7 @@ github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5/go.mod h1:7Id9E
 github.com/jwilder/encoding v0.0.0-20170811194829-b4e1701a28ef/go.mod h1:Ct9fl0F6iIOGgxJ5npU/IUOhOhqlVrGjyIZc8/MagT0=
 github.com/karalabe/usb v0.0.0-20190919080040-51dc0efba356 h1:I/yrLt2WilKxlQKCM52clh5rGzTKpVctGT1lH4Dc8Jw=
 github.com/karalabe/usb v0.0.0-20190919080040-51dc0efba356/go.mod h1:Od972xHfMJowv7NGVDiWVxk2zxnWgjLlJzE+F4F7AGU=
+github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
 github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4=
@@ -266,24 +321,64 @@ github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+
 github.com/leanovate/gopter v0.2.9 h1:fQjYxZaynp97ozCzfOyOuAGOU4aU/z37zf/tOujFk7c=
 github.com/leanovate/gopter v0.2.9/go.mod h1:U2L/78B+KVFIx2VmW6onHJQzXtFb+p5y3y2Sh+Jxxv8=
 github.com/lib/pq v1.0.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
+github.com/lib/pq v1.8.0 h1:9xohqzkUwzR4Ga4ivdTcawVS89YSDVxXMa3xJX3cGzg=
+github.com/lib/pq v1.8.0/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
+github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM=
+github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
 github.com/mattn/go-colorable v0.1.0 h1:v2XXALHHh6zHfYTJ+cSkwtyffnaOyR1MXaA91mTrb8o=
 github.com/mattn/go-colorable v0.1.0/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
+github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg=
+github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
 github.com/mattn/go-ieproxy v0.0.0-20190610004146-91bb50d98149/go.mod h1:31jz6HNzdxOmlERGGEc4v/dMssOfmp2p5bT/okiKFFc=
 github.com/mattn/go-ieproxy v0.0.0-20190702010315-6dee0af9227d h1:oNAwILwmgWKFpuU+dXvI6dl9jG2mAWAZLX3r9s0PPiw=
 github.com/mattn/go-ieproxy v0.0.0-20190702010315-6dee0af9227d/go.mod h1:31jz6HNzdxOmlERGGEc4v/dMssOfmp2p5bT/okiKFFc=
 github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
 github.com/mattn/go-isatty v0.0.5-0.20180830101745-3fb116b82035 h1:USWjF42jDCSEeikX/G1g40ZWnsPXN5WkZ4jMHZWyBK4=
 github.com/mattn/go-isatty v0.0.5-0.20180830101745-3fb116b82035/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
+github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw=
+github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
 github.com/mattn/go-runewidth v0.0.3/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
 github.com/mattn/go-runewidth v0.0.9 h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0=
 github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
+github.com/mattn/go-sqlite3 v1.9.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
+github.com/mattn/go-sqlite3 v1.11.0 h1:LDdKkqtYlom37fkvqs8rMPFKAMe8+SgjbwZ6ex1/A/Q=
 github.com/mattn/go-sqlite3 v1.11.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
 github.com/mattn/go-tty v0.0.0-20180907095812-13ff1204f104/go.mod h1:XPvLUNfbS4fJH25nqRHfWLMa1ONC8Amw+mIA639KxkE=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
+github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
+github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g=
+github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ=
+github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
+github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771 h1:MHkK1uRtFbVqvAgvWxafZe54+5uBxLluGylDiKgdhwo=
+github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc=
+github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/mitchellh/go-testing-interface v1.0.0/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI=
+github.com/mitchellh/gox v0.4.0/go.mod h1:Sd9lOJ0+aimLBi73mGofS1ycjY8lL3uZM3JPS42BGNg=
+github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0QubkSMEySY=
+github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
+github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8=
+github.com/mr-tron/base58 v1.1.3 h1:v+sk57XuaCKGXpWtVBX8YJzO7hMGx4Aajh4TQbdEFdc=
+github.com/mr-tron/base58 v1.1.3/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
 github.com/mschoch/smat v0.0.0-20160514031455-90eadee771ae/go.mod h1:qAyveg+e4CE+eKJXWVjKXM4ck2QobLqTDytGJbLLhJg=
+github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI=
+github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA=
+github.com/multiformats/go-base36 v0.1.0 h1:JR6TyF7JjGd3m6FbLU2cOxhC0Li8z8dLNGQ89tUg4F4=
+github.com/multiformats/go-base36 v0.1.0/go.mod h1:kFGE83c6s80PklsHO9sRn2NCoffoRdUUOENyW/Vv6sM=
+github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs=
+github.com/multiformats/go-multibase v0.0.3 h1:l/B6bJDQjvQ5G52jw4QGSYeOTZoAwIO77RblWplfIqk=
+github.com/multiformats/go-multibase v0.0.3/go.mod h1:5+1R4eQrT3PkYZ24C3W2Ue2tPwIdYQD509ZjSb5y9Oc=
+github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U=
+github.com/multiformats/go-multihash v0.0.13/go.mod h1:VdAWLKTwram9oKAatUcLxBNUjdtcVwxObEQBtRfuyjc=
+github.com/multiformats/go-multihash v0.0.14 h1:QoBceQYQQtNUuf6s7wHxnE2c8bhbMqhfGzNI032se/I=
+github.com/multiformats/go-multihash v0.0.14/go.mod h1:VdAWLKTwram9oKAatUcLxBNUjdtcVwxObEQBtRfuyjc=
+github.com/multiformats/go-varint v0.0.5 h1:XVZwSo04Cs3j/jS0uAEPpT3JY6DzMcVLLoWOSnCxOjg=
+github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/naoina/go-stringutil v0.1.0 h1:rCUeRUHjBjGTSHl0VC00jUPLz8/F9dDzYI70Hzifhks=
 github.com/naoina/go-stringutil v0.1.0/go.mod h1:XJ2SJL9jCtBh+P9q5btrd/Ylo8XwT/h1USek5+NqSA0=
@@ -349,6 +444,9 @@ github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPx
 github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
 github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
 github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
+github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
+github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
+github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
 github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
 github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
@@ -372,6 +470,8 @@ github.com/tklauser/numcpus v0.2.2/go.mod h1:x3qojaO3uyYt0i56EW/VUYs7uBvdl2fkfZF
 github.com/tyler-smith/go-bip39 v1.0.1-0.20181017060643-dbb3b84ba2ef h1:wHSqTBrZW24CsNJDfeh9Ex6Pm0Rcpc7qrgKBiL44vF4=
 github.com/tyler-smith/go-bip39 v1.0.1-0.20181017060643-dbb3b84ba2ef/go.mod h1:sJ5fKU0s6JVwZjjcUEX2zFOnvq0ASQ2K9Zr6cf67kNs=
 github.com/urfave/cli/v2 v2.3.0/go.mod h1:LJmUH05zAU44vOAcrfzZQKsZbVcdbOG8rtL3/XcUArI=
+github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc h1:9lDbC6Rz4bwmou+oE6Dt4Cb2BGMur5eR/GYptkKUVHo=
+github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM=
 github.com/willf/bitset v1.1.3/go.mod h1:RjeCKbqT1RxIR/KWY6phxZiaY1IyutSBfGjNPySAYV4=
 github.com/xlab/treeprint v0.0.0-20180616005107-d6fb6747feb6/go.mod h1:ce1O1j6UtZfjr22oyGxGLbauSBp2YVXpARAosm7dHBg=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -379,13 +479,19 @@ go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
 go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
 go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
+go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
+go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk=
+go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
 go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
 go.uber.org/zap v1.9.1/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190909091759-094676da4a83/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
@@ -411,6 +517,7 @@ golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHl
 golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
 golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
 golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f h1:J5lckAjkw6qYlOZNj90mLYNTEKDvWeuc1yieZ8qUzUE=
 golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
 golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
 golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
@@ -426,6 +533,7 @@ golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73r
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
@@ -462,6 +570,8 @@ golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5h
 golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -496,6 +606,7 @@ golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxb
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20201208040808-7e3f01d25324 h1:Hir2P/De0WpUhtrKGGjvSb2YxUgyZ7EFOSLIcSSpiwE=
 golang.org/x/time v0.0.0-20201208040808-7e3f01d25324/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -515,6 +626,8 @@ golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgw
 golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191112195655-aa38f8e97acc/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
@@ -522,6 +635,8 @@ golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtn
 golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
 golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
 golang.org/x/tools v0.0.0-20200108203644-89082a384178/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
+golang.org/x/tools v0.0.0-20200117012304-6edc0a871e69 h1:yBHHx+XZqXJBm6Exke3N7V9gnlsyXxoCPEb1yVenjfk=
+golang.org/x/tools v0.0.0-20200117012304-6edc0a871e69/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
 golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -545,6 +660,7 @@ google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9Ywl
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
+google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
 google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
 google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
 google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
@@ -576,6 +692,8 @@ gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLks
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
 gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
 gopkg.in/natefinch/npipe.v2 v2.0.0-20160621034901-c1b8fa8bdcce h1:+JknDZhAj8YMt7GC73Ei8pv4MzjDUNPHgQWJdtMAaDU=

internal/ethapi/api.go

@@ -944,7 +944,12 @@ func (e *revertError) ErrorData() interface{} {
 // Note, this function doesn't make and changes in the state/blockchain and is
 // useful to execute and retrieve values.
 func (s *PublicBlockChainAPI) Call(ctx context.Context, args CallArgs, blockNrOrHash rpc.BlockNumberOrHash, overrides *StateOverride) (hexutil.Bytes, error) {
-	result, err := DoCall(ctx, s.b, args, blockNrOrHash, overrides, vm.Config{}, 5*time.Second, s.b.RPCGasCap())
+	timeout := 5 * time.Second
+	d, ok := ctx.Deadline()
+	if ok {
+		timeout = time.Until(d)
+	}
+	result, err := DoCall(ctx, s.b, args, blockNrOrHash, overrides, vm.Config{}, timeout, s.b.RPCGasCap())
 	if err != nil {
 		return nil, err
 	}

miner/stress_clique.go

@@ -36,6 +36,7 @@ import (
 	"github.com/ethereum/go-ethereum/crypto"
 	"github.com/ethereum/go-ethereum/eth"
 	"github.com/ethereum/go-ethereum/eth/downloader"
+	"github.com/ethereum/go-ethereum/eth/ethconfig"
 	"github.com/ethereum/go-ethereum/log"
 	"github.com/ethereum/go-ethereum/miner"
 	"github.com/ethereum/go-ethereum/node"

miner/stress_ethash.go

@@ -37,6 +37,7 @@ import (
 	"github.com/ethereum/go-ethereum/crypto"
 	"github.com/ethereum/go-ethereum/eth"
 	"github.com/ethereum/go-ethereum/eth/downloader"
+	"github.com/ethereum/go-ethereum/eth/ethconfig"
 	"github.com/ethereum/go-ethereum/log"
 	"github.com/ethereum/go-ethereum/miner"
 	"github.com/ethereum/go-ethereum/node"

rpc/http.go

@@ -32,7 +32,7 @@ import (
 )
 
 const (
-	maxRequestContentLength = 1024 * 1024 * 5
+	maxRequestContentLength = 1024 * 1024 * 12
 	contentType             = "application/json"
 )
 

statediff/api.go

@@ -0,0 +1,151 @@
+// Copyright 2019 The go-ethereum Authors
+// This file is part of the go-ethereum library.
+//
+// The go-ethereum library is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Lesser General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// The go-ethereum library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public License
+// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
+
+package statediff
+
+import (
+	"context"
+
+	"github.com/ethereum/go-ethereum/common"
+
+	"github.com/ethereum/go-ethereum/log"
+	"github.com/ethereum/go-ethereum/rpc"
+	. "github.com/ethereum/go-ethereum/statediff/types"
+)
+
+// APIName is the namespace used for the state diffing service API
+const APIName = "statediff"
+
+// APIVersion is the version of the state diffing service API
+const APIVersion = "0.0.1"
+
+// PublicStateDiffAPI provides an RPC subscription interface
+// that can be used to stream out state diffs as they
+// are produced by a full node
+type PublicStateDiffAPI struct {
+	sds IService
+}
+
+// NewPublicStateDiffAPI creates an rpc subscription interface for the underlying statediff service
+func NewPublicStateDiffAPI(sds IService) *PublicStateDiffAPI {
+	return &PublicStateDiffAPI{
+		sds: sds,
+	}
+}
+
+// Stream is the public method to setup a subscription that fires off statediff service payloads as they are created
+func (api *PublicStateDiffAPI) Stream(ctx context.Context, params Params) (*rpc.Subscription, error) {
+	// ensure that the RPC connection supports subscriptions
+	notifier, supported := rpc.NotifierFromContext(ctx)
+	if !supported {
+		return nil, rpc.ErrNotificationsUnsupported
+	}
+
+	// create subscription and start waiting for events
+	rpcSub := notifier.CreateSubscription()
+
+	go func() {
+		// subscribe to events from the statediff service
+		payloadChannel := make(chan Payload, chainEventChanSize)
+		quitChan := make(chan bool, 1)
+		api.sds.Subscribe(rpcSub.ID, payloadChannel, quitChan, params)
+		// loop and await payloads and relay them to the subscriber with the notifier
+		for {
+			select {
+			case payload := <-payloadChannel:
+				if err := notifier.Notify(rpcSub.ID, payload); err != nil {
+					log.Error("Failed to send state diff packet; error: " + err.Error())
+					if err := api.sds.Unsubscribe(rpcSub.ID); err != nil {
+						log.Error("Failed to unsubscribe from the state diff service; error: " + err.Error())
+					}
+					return
+				}
+			case err := <-rpcSub.Err():
+				if err != nil {
+					log.Error("State diff service rpcSub error: " + err.Error())
+					err = api.sds.Unsubscribe(rpcSub.ID)
+					if err != nil {
+						log.Error("Failed to unsubscribe from the state diff service; error: " + err.Error())
+					}
+					return
+				}
+			case <-quitChan:
+				// don't need to unsubscribe, service does so before sending the quit signal
+				return
+			}
+		}
+	}()
+
+	return rpcSub, nil
+}
+
+// StateDiffAt returns a state diff payload at the specific blockheight
+func (api *PublicStateDiffAPI) StateDiffAt(ctx context.Context, blockNumber uint64, params Params) (*Payload, error) {
+	return api.sds.StateDiffAt(blockNumber, params)
+}
+
+// StateDiffFor returns a state diff payload for the specific blockhash
+func (api *PublicStateDiffAPI) StateDiffFor(ctx context.Context, blockHash common.Hash, params Params) (*Payload, error) {
+	return api.sds.StateDiffFor(blockHash, params)
+}
+
+// StateTrieAt returns a state trie payload at the specific blockheight
+func (api *PublicStateDiffAPI) StateTrieAt(ctx context.Context, blockNumber uint64, params Params) (*Payload, error) {
+	return api.sds.StateTrieAt(blockNumber, params)
+}
+
+// StreamCodeAndCodeHash writes all of the codehash=>code pairs out to a websocket channel
+func (api *PublicStateDiffAPI) StreamCodeAndCodeHash(ctx context.Context, blockNumber uint64) (*rpc.Subscription, error) {
+	// ensure that the RPC connection supports subscriptions
+	notifier, supported := rpc.NotifierFromContext(ctx)
+	if !supported {
+		return nil, rpc.ErrNotificationsUnsupported
+	}
+
+	// create subscription and start waiting for events
+	rpcSub := notifier.CreateSubscription()
+	payloadChan := make(chan CodeAndCodeHash, chainEventChanSize)
+	quitChan := make(chan bool)
+	api.sds.StreamCodeAndCodeHash(blockNumber, payloadChan, quitChan)
+	go func() {
+		for {
+			select {
+			case payload := <-payloadChan:
+				if err := notifier.Notify(rpcSub.ID, payload); err != nil {
+					log.Error("Failed to send code and codehash packet", "err", err)
+					return
+				}
+			case err := <-rpcSub.Err():
+				log.Error("State diff service rpcSub error", "err", err)
+				return
+			case <-quitChan:
+				return
+			}
+		}
+	}()
+
+	return rpcSub, nil
+}
+
+// WriteStateDiffAt writes a state diff object directly to DB at the specific blockheight
+func (api *PublicStateDiffAPI) WriteStateDiffAt(ctx context.Context, blockNumber uint64, params Params) error {
+	return api.sds.WriteStateDiffAt(blockNumber, params)
+}
+
+// WriteStateDiffFor writes a state diff object directly to DB for the specific block hash
+func (api *PublicStateDiffAPI) WriteStateDiffFor(ctx context.Context, blockHash common.Hash, params Params) error {
+	return api.sds.WriteStateDiffFor(blockHash, params)
+}

statediff/builder.go

@@ -0,0 +1,768 @@
+// Copyright 2019 The go-ethereum Authors
+// This file is part of the go-ethereum library.
+//
+// The go-ethereum library is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Lesser General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// The go-ethereum library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public License
+// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
+
+// Contains a batch of utility type declarations used by the tests. As the node
+// operates on unique types, a lot of them are needed to check various features.
+
+package statediff
+
+import (
+	"bytes"
+	"fmt"
+
+	"github.com/ethereum/go-ethereum/common"
+	"github.com/ethereum/go-ethereum/core/state"
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/crypto"
+	"github.com/ethereum/go-ethereum/log"
+	"github.com/ethereum/go-ethereum/rlp"
+	. "github.com/ethereum/go-ethereum/statediff/types"
+	"github.com/ethereum/go-ethereum/trie"
+)
+
+var (
+	nullHashBytes     = common.Hex2Bytes("0000000000000000000000000000000000000000000000000000000000000000")
+	emptyNode, _      = rlp.EncodeToBytes([]byte{})
+	emptyContractRoot = crypto.Keccak256Hash(emptyNode)
+	nullCodeHash      = crypto.Keccak256Hash([]byte{}).Bytes()
+)
+
+// Builder interface exposes the method for building a state diff between two blocks
+type Builder interface {
+	BuildStateDiffObject(args Args, params Params) (StateObject, error)
+	BuildStateTrieObject(current *types.Block) (StateObject, error)
+	WriteStateDiffObject(args StateRoots, params Params, output StateNodeSink, codeOutput CodeSink) error
+}
+
+type builder struct {
+	stateCache state.Database
+}
+
+func resolveNode(it trie.NodeIterator, trieDB *trie.Database) (StateNode, []interface{}, error) {
+	nodePath := make([]byte, len(it.Path()))
+	copy(nodePath, it.Path())
+	node, err := trieDB.Node(it.Hash())
+	if err != nil {
+		return StateNode{}, nil, err
+	}
+	var nodeElements []interface{}
+	if err := rlp.DecodeBytes(node, &nodeElements); err != nil {
+		return StateNode{}, nil, err
+	}
+	ty, err := CheckKeyType(nodeElements)
+	if err != nil {
+		return StateNode{}, nil, err
+	}
+	return StateNode{
+		NodeType:  ty,
+		Path:      nodePath,
+		NodeValue: node,
+	}, nodeElements, nil
+}
+
+// convenience
+func stateNodeAppender(nodes *[]StateNode) StateNodeSink {
+	return func(node StateNode) error {
+		*nodes = append(*nodes, node)
+		return nil
+	}
+}
+func storageNodeAppender(nodes *[]StorageNode) StorageNodeSink {
+	return func(node StorageNode) error {
+		*nodes = append(*nodes, node)
+		return nil
+	}
+}
+func codeMappingAppender(codeAndCodeHashes *[]CodeAndCodeHash) CodeSink {
+	return func(c CodeAndCodeHash) error {
+		*codeAndCodeHashes = append(*codeAndCodeHashes, c)
+		return nil
+	}
+}
+
+// NewBuilder is used to create a statediff builder
+func NewBuilder(stateCache state.Database) Builder {
+	return &builder{
+		stateCache: stateCache, // state cache is safe for concurrent reads
+	}
+}
+
+// BuildStateTrieObject builds a state trie object from the provided block
+func (sdb *builder) BuildStateTrieObject(current *types.Block) (StateObject, error) {
+	currentTrie, err := sdb.stateCache.OpenTrie(current.Root())
+	if err != nil {
+		return StateObject{}, fmt.Errorf("error creating trie for block %d: %v", current.Number(), err)
+	}
+	it := currentTrie.NodeIterator([]byte{})
+	stateNodes, codeAndCodeHashes, err := sdb.buildStateTrie(it)
+	if err != nil {
+		return StateObject{}, fmt.Errorf("error collecting state nodes for block %d: %v", current.Number(), err)
+	}
+	return StateObject{
+		BlockNumber:       current.Number(),
+		BlockHash:         current.Hash(),
+		Nodes:             stateNodes,
+		CodeAndCodeHashes: codeAndCodeHashes,
+	}, nil
+}
+
+func (sdb *builder) buildStateTrie(it trie.NodeIterator) ([]StateNode, []CodeAndCodeHash, error) {
+	stateNodes := make([]StateNode, 0)
+	codeAndCodeHashes := make([]CodeAndCodeHash, 0)
+	for it.Next(true) {
+		// skip value nodes
+		if it.Leaf() || bytes.Equal(nullHashBytes, it.Hash().Bytes()) {
+			continue
+		}
+		node, nodeElements, err := resolveNode(it, sdb.stateCache.TrieDB())
+		if err != nil {
+			return nil, nil, err
+		}
+		switch node.NodeType {
+		case Leaf:
+			var account state.Account
+			if err := rlp.DecodeBytes(nodeElements[1].([]byte), &account); err != nil {
+				return nil, nil, fmt.Errorf("error decoding account for leaf node at path %x nerror: %v", node.Path, err)
+			}
+			partialPath := trie.CompactToHex(nodeElements[0].([]byte))
+			valueNodePath := append(node.Path, partialPath...)
+			encodedPath := trie.HexToCompact(valueNodePath)
+			leafKey := encodedPath[1:]
+			node.LeafKey = leafKey
+			if !bytes.Equal(account.CodeHash, nullCodeHash) {
+				var storageNodes []StorageNode
+				err := sdb.buildStorageNodesEventual(account.Root, nil, true, storageNodeAppender(&storageNodes))
+				if err != nil {
+					return nil, nil, fmt.Errorf("failed building eventual storage diffs for account %+v\r\nerror: %v", account, err)
+				}
+				node.StorageNodes = storageNodes
+				// emit codehash => code mappings for cod
+				codeHash := common.BytesToHash(account.CodeHash)
+				code, err := sdb.stateCache.ContractCode(common.Hash{}, codeHash)
+				if err != nil {
+					return nil, nil, fmt.Errorf("failed to retrieve code for codehash %s\r\n error: %v", codeHash.String(), err)
+				}
+				codeAndCodeHashes = append(codeAndCodeHashes, CodeAndCodeHash{
+					Hash: codeHash,
+					Code: code,
+				})
+			}
+			stateNodes = append(stateNodes, node)
+		case Extension, Branch:
+			stateNodes = append(stateNodes, node)
+		default:
+			return nil, nil, fmt.Errorf("unexpected node type %s", node.NodeType)
+		}
+	}
+	return stateNodes, codeAndCodeHashes, it.Error()
+}
+
+// BuildStateDiffObject builds a statediff object from two blocks and the provided parameters
+func (sdb *builder) BuildStateDiffObject(args Args, params Params) (StateObject, error) {
+	var stateNodes []StateNode
+	var codeAndCodeHashes []CodeAndCodeHash
+	err := sdb.WriteStateDiffObject(
+		StateRoots{OldStateRoot: args.OldStateRoot, NewStateRoot: args.NewStateRoot},
+		params, stateNodeAppender(&stateNodes), codeMappingAppender(&codeAndCodeHashes))
+	if err != nil {
+		return StateObject{}, err
+	}
+	return StateObject{
+		BlockHash:         args.BlockHash,
+		BlockNumber:       args.BlockNumber,
+		Nodes:             stateNodes,
+		CodeAndCodeHashes: codeAndCodeHashes,
+	}, nil
+}
+
+// Writes a statediff object to output callback
+func (sdb *builder) WriteStateDiffObject(args StateRoots, params Params, output StateNodeSink, codeOutput CodeSink) error {
+	if !params.IntermediateStateNodes || len(params.WatchedAddresses) > 0 {
+		// if we are watching only specific accounts then we are only diffing leaf nodes
+		return sdb.buildStateDiffWithoutIntermediateStateNodes(args, params, output, codeOutput)
+	} else {
+		return sdb.buildStateDiffWithIntermediateStateNodes(args, params, output, codeOutput)
+	}
+}
+
+func (sdb *builder) buildStateDiffWithIntermediateStateNodes(args StateRoots, params Params, output StateNodeSink, codeOutput CodeSink) error {
+	// Load tries for old and new states
+	oldTrie, err := sdb.stateCache.OpenTrie(args.OldStateRoot)
+	if err != nil {
+		return fmt.Errorf("error creating trie for oldStateRoot: %v", err)
+	}
+	newTrie, err := sdb.stateCache.OpenTrie(args.NewStateRoot)
+	if err != nil {
+		return fmt.Errorf("error creating trie for newStateRoot: %v", err)
+	}
+
+	// collect a slice of all the intermediate nodes that were touched and exist at B
+	// a map of their leafkey to all the accounts that were touched and exist at B
+	// and a slice of all the paths for the nodes in both of the above sets
+	diffAccountsAtB, diffPathsAtB, err := sdb.createdAndUpdatedStateWithIntermediateNodes(
+		oldTrie.NodeIterator([]byte{}), newTrie.NodeIterator([]byte{}),
+		output)
+	if err != nil {
+		return fmt.Errorf("error collecting createdAndUpdatedNodes: %v", err)
+	}
+
+	// collect a slice of all the nodes that existed at a path in A that doesn't exist in B
+	// a map of their leafkey to all the accounts that were touched and exist at A
+	diffAccountsAtA, err := sdb.deletedOrUpdatedState(
+		oldTrie.NodeIterator([]byte{}), newTrie.NodeIterator([]byte{}),
+		diffPathsAtB, output)
+	if err != nil {
+		return fmt.Errorf("error collecting deletedOrUpdatedNodes: %v", err)
+	}
+
+	// collect and sort the leafkey keys for both account mappings into a slice
+	createKeys := sortKeys(diffAccountsAtB)
+	deleteKeys := sortKeys(diffAccountsAtA)
+
+	// and then find the intersection of these keys
+	// these are the leafkeys for the accounts which exist at both A and B but are different
+	// this also mutates the passed in createKeys and deleteKeys, removing the intersection keys
+	// and leaving the truly created or deleted keys in place
+	updatedKeys := findIntersection(createKeys, deleteKeys)
+
+	// build the diff nodes for the updated accounts using the mappings at both A and B as directed by the keys found as the intersection of the two
+	err = sdb.buildAccountUpdates(
+		diffAccountsAtB, diffAccountsAtA, updatedKeys,
+		params.WatchedStorageSlots, params.IntermediateStorageNodes, output)
+	if err != nil {
+		return fmt.Errorf("error building diff for updated accounts: %v", err)
+	}
+	// build the diff nodes for created accounts
+	err = sdb.buildAccountCreations(diffAccountsAtB, params.WatchedStorageSlots, params.IntermediateStorageNodes, output, codeOutput)
+	if err != nil {
+		return fmt.Errorf("error building diff for created accounts: %v", err)
+	}
+	return nil
+}
+
+func (sdb *builder) buildStateDiffWithoutIntermediateStateNodes(args StateRoots, params Params, output StateNodeSink, codeOutput CodeSink) error {
+	// Load tries for old (A) and new (B) states
+	oldTrie, err := sdb.stateCache.OpenTrie(args.OldStateRoot)
+	if err != nil {
+		return fmt.Errorf("error creating trie for oldStateRoot: %v", err)
+	}
+	newTrie, err := sdb.stateCache.OpenTrie(args.NewStateRoot)
+	if err != nil {
+		return fmt.Errorf("error creating trie for newStateRoot: %v", err)
+	}
+
+	// collect a map of their leafkey to all the accounts that were touched and exist at B
+	// and a slice of all the paths for the nodes in both of the above sets
+	diffAccountsAtB, diffPathsAtB, err := sdb.createdAndUpdatedState(
+		oldTrie.NodeIterator([]byte{}), newTrie.NodeIterator([]byte{}),
+		params.WatchedAddresses)
+	if err != nil {
+		return fmt.Errorf("error collecting createdAndUpdatedNodes: %v", err)
+	}
+
+	// collect a slice of all the nodes that existed at a path in A that doesn't exist in B
+	// a map of their leafkey to all the accounts that were touched and exist at A
+	diffAccountsAtA, err := sdb.deletedOrUpdatedState(
+		oldTrie.NodeIterator([]byte{}), newTrie.NodeIterator([]byte{}),
+		diffPathsAtB, output)
+	if err != nil {
+		return fmt.Errorf("error collecting deletedOrUpdatedNodes: %v", err)
+	}
+
+	// collect and sort the leafkeys for both account mappings into a slice
+	createKeys := sortKeys(diffAccountsAtB)
+	deleteKeys := sortKeys(diffAccountsAtA)
+
+	// and then find the intersection of these keys
+	// these are the leafkeys for the accounts which exist at both A and B but are different
+	// this also mutates the passed in createKeys and deleteKeys, removing in intersection keys
+	// and leaving the truly created or deleted keys in place
+	updatedKeys := findIntersection(createKeys, deleteKeys)
+
+	// build the diff nodes for the updated accounts using the mappings at both A and B as directed by the keys found as the intersection of the two
+	err = sdb.buildAccountUpdates(
+		diffAccountsAtB, diffAccountsAtA, updatedKeys,
+		params.WatchedStorageSlots, params.IntermediateStorageNodes, output)
+	if err != nil {
+		return fmt.Errorf("error building diff for updated accounts: %v", err)
+	}
+	// build the diff nodes for created accounts
+	err = sdb.buildAccountCreations(diffAccountsAtB, params.WatchedStorageSlots, params.IntermediateStorageNodes, output, codeOutput)
+	if err != nil {
+		return fmt.Errorf("error building diff for created accounts: %v", err)
+	}
+	return nil
+}
+
+// createdAndUpdatedState returns
+// a mapping of their leafkeys to all the accounts that exist in a different state at B than A
+// and a slice of the paths for all of the nodes included in both
+func (sdb *builder) createdAndUpdatedState(a, b trie.NodeIterator, watchedAddresses []common.Address) (AccountMap, map[string]bool, error) {
+	diffPathsAtB := make(map[string]bool)
+	diffAcountsAtB := make(AccountMap)
+	it, _ := trie.NewDifferenceIterator(a, b)
+	for it.Next(true) {
+		// skip value nodes
+		if it.Leaf() || bytes.Equal(nullHashBytes, it.Hash().Bytes()) {
+			continue
+		}
+		node, nodeElements, err := resolveNode(it, sdb.stateCache.TrieDB())
+		if err != nil {
+			return nil, nil, err
+		}
+		if node.NodeType == Leaf {
+			// created vs updated is important for leaf nodes since we need to diff their storage
+			// so we need to map all changed accounts at B to their leafkey, since account can change pathes but not leafkey
+			var account state.Account
+			if err := rlp.DecodeBytes(nodeElements[1].([]byte), &account); err != nil {
+				return nil, nil, fmt.Errorf("error decoding account for leaf node at path %x nerror: %v", node.Path, err)
+			}
+			partialPath := trie.CompactToHex(nodeElements[0].([]byte))
+			valueNodePath := append(node.Path, partialPath...)
+			encodedPath := trie.HexToCompact(valueNodePath)
+			leafKey := encodedPath[1:]
+			if isWatchedAddress(watchedAddresses, leafKey) {
+				diffAcountsAtB[common.Bytes2Hex(leafKey)] = accountWrapper{
+					NodeType:  node.NodeType,
+					Path:      node.Path,
+					NodeValue: node.NodeValue,
+					LeafKey:   leafKey,
+					Account:   &account,
+				}
+			}
+		}
+		// add both intermediate and leaf node paths to the list of diffPathsAtB
+		diffPathsAtB[common.Bytes2Hex(node.Path)] = true
+	}
+	return diffAcountsAtB, diffPathsAtB, it.Error()
+}
+
+// createdAndUpdatedStateWithIntermediateNodes returns
+// a slice of all the intermediate nodes that exist in a different state at B than A
+// a mapping of their leafkeys to all the accounts that exist in a different state at B than A
+// and a slice of the paths for all of the nodes included in both
+func (sdb *builder) createdAndUpdatedStateWithIntermediateNodes(a, b trie.NodeIterator, output StateNodeSink) (AccountMap, map[string]bool, error) {
+	diffPathsAtB := make(map[string]bool)
+	diffAcountsAtB := make(AccountMap)
+	it, _ := trie.NewDifferenceIterator(a, b)
+	for it.Next(true) {
+		// skip value nodes
+		if it.Leaf() || bytes.Equal(nullHashBytes, it.Hash().Bytes()) {
+			continue
+		}
+		node, nodeElements, err := resolveNode(it, sdb.stateCache.TrieDB())
+		if err != nil {
+			return nil, nil, err
+		}
+		switch node.NodeType {
+		case Leaf:
+			// created vs updated is important for leaf nodes since we need to diff their storage
+			// so we need to map all changed accounts at B to their leafkey, since account can change paths but not leafkey
+			var account state.Account
+			if err := rlp.DecodeBytes(nodeElements[1].([]byte), &account); err != nil {
+				return nil, nil, fmt.Errorf("error decoding account for leaf node at path %x nerror: %v", node.Path, err)
+			}
+			partialPath := trie.CompactToHex(nodeElements[0].([]byte))
+			valueNodePath := append(node.Path, partialPath...)
+			encodedPath := trie.HexToCompact(valueNodePath)
+			leafKey := encodedPath[1:]
+			diffAcountsAtB[common.Bytes2Hex(leafKey)] = accountWrapper{
+				NodeType:  node.NodeType,
+				Path:      node.Path,
+				NodeValue: node.NodeValue,
+				LeafKey:   leafKey,
+				Account:   &account,
+			}
+		case Extension, Branch:
+			// create a diff for any intermediate node that has changed at b
+			// created vs updated makes no difference for intermediate nodes since we do not need to diff storage
+			if err := output(StateNode{
+				NodeType:  node.NodeType,
+				Path:      node.Path,
+				NodeValue: node.NodeValue,
+			}); err != nil {
+				return nil, nil, err
+			}
+		default:
+			return nil, nil, fmt.Errorf("unexpected node type %s", node.NodeType)
+		}
+		// add both intermediate and leaf node paths to the list of diffPathsAtB
+		diffPathsAtB[common.Bytes2Hex(node.Path)] = true
+	}
+	return diffAcountsAtB, diffPathsAtB, it.Error()
+}
+
+// deletedOrUpdatedState returns a slice of all the pathes that are emptied at B
+// and a mapping of their leafkeys to all the accounts that exist in a different state at A than B
+func (sdb *builder) deletedOrUpdatedState(a, b trie.NodeIterator, diffPathsAtB map[string]bool, output StateNodeSink) (AccountMap, error) {
+	diffAccountAtA := make(AccountMap)
+	it, _ := trie.NewDifferenceIterator(b, a)
+	for it.Next(true) {
+		// skip value nodes
+		if it.Leaf() || bytes.Equal(nullHashBytes, it.Hash().Bytes()) {
+			continue
+		}
+		node, nodeElements, err := resolveNode(it, sdb.stateCache.TrieDB())
+		if err != nil {
+			return nil, err
+		}
+		switch node.NodeType {
+		case Leaf:
+			// map all different accounts at A to their leafkey
+			var account state.Account
+			if err := rlp.DecodeBytes(nodeElements[1].([]byte), &account); err != nil {
+				return nil, fmt.Errorf("error decoding account for leaf node at path %x nerror: %v", node.Path, err)
+			}
+			partialPath := trie.CompactToHex(nodeElements[0].([]byte))
+			valueNodePath := append(node.Path, partialPath...)
+			encodedPath := trie.HexToCompact(valueNodePath)
+			leafKey := encodedPath[1:]
+			diffAccountAtA[common.Bytes2Hex(leafKey)] = accountWrapper{
+				NodeType:  node.NodeType,
+				Path:      node.Path,
+				NodeValue: node.NodeValue,
+				LeafKey:   leafKey,
+				Account:   &account,
+			}
+			// if this node's path did not show up in diffPathsAtB
+			// that means the node at this path was deleted (or moved) in B
+			// emit an empty "removed" diff to signify as such
+			if _, ok := diffPathsAtB[common.Bytes2Hex(node.Path)]; !ok {
+				if err := output(StateNode{
+					Path:      node.Path,
+					NodeValue: []byte{},
+					NodeType:  Removed,
+					LeafKey:   leafKey,
+				}); err != nil {
+					return nil, err
+				}
+			}
+		case Extension, Branch:
+			// if this node's path did not show up in diffPathsAtB
+			// that means the node at this path was deleted (or moved) in B
+			// emit an empty "removed" diff to signify as such
+			if _, ok := diffPathsAtB[common.Bytes2Hex(node.Path)]; !ok {
+				if err := output(StateNode{
+					Path:      node.Path,
+					NodeValue: []byte{},
+					NodeType:  Removed,
+				}); err != nil {
+					return nil, err
+				}
+			}
+			// fall through, we did everything we need to do with these node types
+		default:
+			return nil, fmt.Errorf("unexpected node type %s", node.NodeType)
+		}
+	}
+	return diffAccountAtA, it.Error()
+}
+
+// buildAccountUpdates uses the account diffs maps for A => B and B => A and the known intersection of their leafkeys
+// to generate the statediff node objects for all of the accounts that existed at both A and B but in different states
+// needs to be called before building account creations and deletions as this mutates
+// those account maps to remove the accounts which were updated
+func (sdb *builder) buildAccountUpdates(creations, deletions AccountMap, updatedKeys []string,
+	watchedStorageKeys []common.Hash, intermediateStorageNodes bool, output StateNodeSink) error {
+	var err error
+	for _, key := range updatedKeys {
+		createdAcc := creations[key]
+		deletedAcc := deletions[key]
+		var storageDiffs []StorageNode
+		if deletedAcc.Account != nil && createdAcc.Account != nil {
+			oldSR := deletedAcc.Account.Root
+			newSR := createdAcc.Account.Root
+			err = sdb.buildStorageNodesIncremental(
+				oldSR, newSR, watchedStorageKeys, intermediateStorageNodes,
+				storageNodeAppender(&storageDiffs))
+			if err != nil {
+				return fmt.Errorf("failed building incremental storage diffs for account with leafkey %s\r\nerror: %v", key, err)
+			}
+		}
+		if err = output(StateNode{
+			NodeType:     createdAcc.NodeType,
+			Path:         createdAcc.Path,
+			NodeValue:    createdAcc.NodeValue,
+			LeafKey:      createdAcc.LeafKey,
+			StorageNodes: storageDiffs,
+		}); err != nil {
+			return err
+		}
+		delete(creations, key)
+		delete(deletions, key)
+	}
+
+	return nil
+}
+
+// buildAccountCreations returns the statediff node objects for all the accounts that exist at B but not at A
+// it also returns the code and codehash for created contract accounts
+func (sdb *builder) buildAccountCreations(accounts AccountMap, watchedStorageKeys []common.Hash, intermediateStorageNodes bool, output StateNodeSink, codeOutput CodeSink) error {
+	for _, val := range accounts {
+		diff := StateNode{
+			NodeType:  val.NodeType,
+			Path:      val.Path,
+			LeafKey:   val.LeafKey,
+			NodeValue: val.NodeValue,
+		}
+		if !bytes.Equal(val.Account.CodeHash, nullCodeHash) {
+			// For contract creations, any storage node contained is a diff
+			var storageDiffs []StorageNode
+			err := sdb.buildStorageNodesEventual(val.Account.Root, watchedStorageKeys, intermediateStorageNodes, storageNodeAppender(&storageDiffs))
+			if err != nil {
+				return fmt.Errorf("failed building eventual storage diffs for node %x\r\nerror: %v", val.Path, err)
+			}
+			diff.StorageNodes = storageDiffs
+			// emit codehash => code mappings for cod
+			codeHash := common.BytesToHash(val.Account.CodeHash)
+			code, err := sdb.stateCache.ContractCode(common.Hash{}, codeHash)
+			if err != nil {
+				return fmt.Errorf("failed to retrieve code for codehash %s\r\n error: %v", codeHash.String(), err)
+			}
+			if err := codeOutput(CodeAndCodeHash{
+				Hash: codeHash,
+				Code: code,
+			}); err != nil {
+				return err
+			}
+		}
+		if err := output(diff); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// buildStorageNodesEventual builds the storage diff node objects for a created account
+// i.e. it returns all the storage nodes at this state, since there is no previous state
+func (sdb *builder) buildStorageNodesEventual(sr common.Hash, watchedStorageKeys []common.Hash, intermediateNodes bool, output StorageNodeSink) error {
+	if bytes.Equal(sr.Bytes(), emptyContractRoot.Bytes()) {
+		return nil
+	}
+	log.Debug("Storage Root For Eventual Diff", "root", sr.Hex())
+	sTrie, err := sdb.stateCache.OpenTrie(sr)
+	if err != nil {
+		log.Info("error in build storage diff eventual", "error", err)
+		return err
+	}
+	it := sTrie.NodeIterator(make([]byte, 0))
+	err = sdb.buildStorageNodesFromTrie(it, watchedStorageKeys, intermediateNodes, output)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+// buildStorageNodesFromTrie returns all the storage diff node objects in the provided node interator
+// if any storage keys are provided it will only return those leaf nodes
+// including intermediate nodes can be turned on or off
+func (sdb *builder) buildStorageNodesFromTrie(it trie.NodeIterator, watchedStorageKeys []common.Hash, intermediateNodes bool, output StorageNodeSink) error {
+	for it.Next(true) {
+		// skip value nodes
+		if it.Leaf() || bytes.Equal(nullHashBytes, it.Hash().Bytes()) {
+			continue
+		}
+		node, nodeElements, err := resolveNode(it, sdb.stateCache.TrieDB())
+		if err != nil {
+			return err
+		}
+		switch node.NodeType {
+		case Leaf:
+			partialPath := trie.CompactToHex(nodeElements[0].([]byte))
+			valueNodePath := append(node.Path, partialPath...)
+			encodedPath := trie.HexToCompact(valueNodePath)
+			leafKey := encodedPath[1:]
+			if isWatchedStorageKey(watchedStorageKeys, leafKey) {
+				if err := output(StorageNode{
+					NodeType:  node.NodeType,
+					Path:      node.Path,
+					NodeValue: node.NodeValue,
+					LeafKey:   leafKey,
+				}); err != nil {
+					return err
+				}
+			}
+		case Extension, Branch:
+			if intermediateNodes {
+				if err := output(StorageNode{
+					NodeType:  node.NodeType,
+					Path:      node.Path,
+					NodeValue: node.NodeValue,
+				}); err != nil {
+					return err
+				}
+			}
+		default:
+			return fmt.Errorf("unexpected node type %s", node.NodeType)
+		}
+	}
+	return it.Error()
+}
+
+// buildStorageNodesIncremental builds the storage diff node objects for all nodes that exist in a different state at B than A
+func (sdb *builder) buildStorageNodesIncremental(oldSR common.Hash, newSR common.Hash, watchedStorageKeys []common.Hash, intermediateNodes bool, output StorageNodeSink) error {
+	if bytes.Equal(newSR.Bytes(), oldSR.Bytes()) {
+		return nil
+	}
+	log.Debug("Storage Roots for Incremental Diff", "old", oldSR.Hex(), "new", newSR.Hex())
+	oldTrie, err := sdb.stateCache.OpenTrie(oldSR)
+	if err != nil {
+		return err
+	}
+	newTrie, err := sdb.stateCache.OpenTrie(newSR)
+	if err != nil {
+		return err
+	}
+
+	diffPathsAtB, err := sdb.createdAndUpdatedStorage(
+		oldTrie.NodeIterator([]byte{}), newTrie.NodeIterator([]byte{}),
+		watchedStorageKeys, intermediateNodes, output)
+	if err != nil {
+		return err
+	}
+	err = sdb.deletedOrUpdatedStorage(oldTrie.NodeIterator([]byte{}), newTrie.NodeIterator([]byte{}),
+		diffPathsAtB, watchedStorageKeys, intermediateNodes, output)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (sdb *builder) createdAndUpdatedStorage(a, b trie.NodeIterator, watchedKeys []common.Hash, intermediateNodes bool, output StorageNodeSink) (map[string]bool, error) {
+	diffPathsAtB := make(map[string]bool)
+	it, _ := trie.NewDifferenceIterator(a, b)
+	for it.Next(true) {
+		// skip value nodes
+		if it.Leaf() || bytes.Equal(nullHashBytes, it.Hash().Bytes()) {
+			continue
+		}
+		node, nodeElements, err := resolveNode(it, sdb.stateCache.TrieDB())
+		if err != nil {
+			return nil, err
+		}
+		switch node.NodeType {
+		case Leaf:
+			partialPath := trie.CompactToHex(nodeElements[0].([]byte))
+			valueNodePath := append(node.Path, partialPath...)
+			encodedPath := trie.HexToCompact(valueNodePath)
+			leafKey := encodedPath[1:]
+			if isWatchedStorageKey(watchedKeys, leafKey) {
+				if err := output(StorageNode{
+					NodeType:  node.NodeType,
+					Path:      node.Path,
+					NodeValue: node.NodeValue,
+					LeafKey:   leafKey,
+				}); err != nil {
+					return nil, err
+				}
+			}
+		case Extension, Branch:
+			if intermediateNodes {
+				if err := output(StorageNode{
+					NodeType:  node.NodeType,
+					Path:      node.Path,
+					NodeValue: node.NodeValue,
+				}); err != nil {
+					return nil, err
+				}
+			}
+		default:
+			return nil, fmt.Errorf("unexpected node type %s", node.NodeType)
+		}
+		diffPathsAtB[common.Bytes2Hex(node.Path)] = true
+	}
+	return diffPathsAtB, it.Error()
+}
+
+func (sdb *builder) deletedOrUpdatedStorage(a, b trie.NodeIterator, diffPathsAtB map[string]bool, watchedKeys []common.Hash, intermediateNodes bool, output StorageNodeSink) error {
+	it, _ := trie.NewDifferenceIterator(b, a)
+	for it.Next(true) {
+		// skip value nodes
+		if it.Leaf() || bytes.Equal(nullHashBytes, it.Hash().Bytes()) {
+			continue
+		}
+		node, nodeElements, err := resolveNode(it, sdb.stateCache.TrieDB())
+		if err != nil {
+			return err
+		}
+		// if this node path showed up in diffPathsAtB
+		// that means this node was updated at B and we already have the updated diff for it
+		// otherwise that means this node was deleted in B and we need to add a "removed" diff to represent that event
+		if _, ok := diffPathsAtB[common.Bytes2Hex(node.Path)]; ok {
+			continue
+		}
+		switch node.NodeType {
+		case Leaf:
+			partialPath := trie.CompactToHex(nodeElements[0].([]byte))
+			valueNodePath := append(node.Path, partialPath...)
+			encodedPath := trie.HexToCompact(valueNodePath)
+			leafKey := encodedPath[1:]
+			if isWatchedStorageKey(watchedKeys, leafKey) {
+				if err := output(StorageNode{
+					NodeType:  Removed,
+					Path:      node.Path,
+					NodeValue: []byte{},
+					LeafKey:   leafKey,
+				}); err != nil {
+					return err
+				}
+			}
+		case Extension, Branch:
+			if intermediateNodes {
+				if err := output(StorageNode{
+					NodeType:  Removed,
+					Path:      node.Path,
+					NodeValue: []byte{},
+				}); err != nil {
+					return err
+				}
+			}
+		default:
+			return fmt.Errorf("unexpected node type %s", node.NodeType)
+		}
+	}
+	return it.Error()
+}
+
+// isWatchedAddress is used to check if a state account corresponds to one of the addresses the builder is configured to watch
+func isWatchedAddress(watchedAddresses []common.Address, stateLeafKey []byte) bool {
+	// If we aren't watching any specific addresses, we are watching everything
+	if len(watchedAddresses) == 0 {
+		return true
+	}
+	for _, addr := range watchedAddresses {
+		addrHashKey := crypto.Keccak256(addr.Bytes())
+		if bytes.Equal(addrHashKey, stateLeafKey) {
+			return true
+		}
+	}
+	return false
+}
+
+// isWatchedStorageKey is used to check if a storage leaf corresponds to one of the storage slots the builder is configured to watch
+func isWatchedStorageKey(watchedKeys []common.Hash, storageLeafKey []byte) bool {
+	// If we aren't watching any specific addresses, we are watching everything
+	if len(watchedKeys) == 0 {
+		return true
+	}
+	for _, hashKey := range watchedKeys {
+		if bytes.Equal(hashKey.Bytes(), storageLeafKey) {
+			return true
+		}
+	}
+	return false
+}

statediff/db/migrations/00001_create_ipfs_blocks_table.sql

@@ -0,0 +1,8 @@
+-- +goose Up
+CREATE TABLE IF NOT EXISTS public.blocks (
+  key TEXT UNIQUE NOT NULL,
+  data BYTEA NOT NULL
+);
+
+-- +goose Down
+DROP TABLE public.blocks;

statediff/db/migrations/00002_create_nodes_table.sql

@@ -0,0 +1,13 @@
+-- +goose Up
+CREATE TABLE nodes (
+  id            SERIAL PRIMARY KEY,
+  client_name   VARCHAR,
+  genesis_block VARCHAR(66),
+  network_id    VARCHAR,
+  node_id       VARCHAR(128),
+  chain_id      INTEGER DEFAULT 1,
+  CONSTRAINT node_uc UNIQUE (genesis_block, network_id, node_id, chain_id)
+);
+
+-- +goose Down
+DROP TABLE nodes;

statediff/db/migrations/00003_create_eth_schema.sql

@@ -0,0 +1,5 @@
+-- +goose Up
+CREATE SCHEMA eth;
+
+-- +goose Down
+DROP SCHEMA eth;

statediff/db/migrations/00004_create_eth_header_cids_table.sql

@@ -0,0 +1,23 @@
+-- +goose Up
+CREATE TABLE eth.header_cids (
+  id                    SERIAL PRIMARY KEY,
+  block_number          BIGINT NOT NULL,
+  block_hash            VARCHAR(66) NOT NULL,
+  parent_hash           VARCHAR(66) NOT NULL,
+  cid                   TEXT NOT NULL,
+  mh_key                TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  td                    NUMERIC NOT NULL,
+  node_id               INTEGER NOT NULL REFERENCES nodes (id) ON DELETE CASCADE,
+  reward                NUMERIC NOT NULL,
+  state_root            VARCHAR(66) NOT NULL,
+  tx_root               VARCHAR(66) NOT NULL,
+  receipt_root          VARCHAR(66) NOT NULL,
+  uncle_root            VARCHAR(66) NOT NULL,
+  bloom                 BYTEA NOT NULL,
+  timestamp             NUMERIC NOT NULL,
+  times_validated       INTEGER NOT NULL DEFAULT 1,
+  UNIQUE (block_number, block_hash)
+);
+
+-- +goose Down
+DROP TABLE eth.header_cids;

statediff/db/migrations/00005_create_eth_uncle_cids_table.sql

@@ -0,0 +1,14 @@
+-- +goose Up
+CREATE TABLE eth.uncle_cids (
+  id                    SERIAL PRIMARY KEY,
+  header_id             INTEGER NOT NULL REFERENCES eth.header_cids (id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  block_hash            VARCHAR(66) NOT NULL,
+  parent_hash           VARCHAR(66) NOT NULL,
+  cid                   TEXT NOT NULL,
+  mh_key                TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  reward                NUMERIC NOT NULL,
+  UNIQUE (header_id, block_hash)
+);
+
+-- +goose Down
+DROP TABLE eth.uncle_cids;

statediff/db/migrations/00006_create_eth_transaction_cids_table.sql

@@ -0,0 +1,17 @@
+-- +goose Up
+CREATE TABLE eth.transaction_cids (
+  id                    SERIAL PRIMARY KEY,
+  header_id             INTEGER NOT NULL REFERENCES eth.header_cids (id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  tx_hash               VARCHAR(66) NOT NULL,
+  index                 INTEGER NOT NULL,
+  cid                   TEXT NOT NULL,
+  mh_key                TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  dst                   VARCHAR(66) NOT NULL,
+  src                   VARCHAR(66) NOT NULL,
+  tx_data               BYTEA,
+  tx_type               BYTEA,
+  UNIQUE (header_id, tx_hash)
+);
+
+-- +goose Down
+DROP TABLE eth.transaction_cids;

statediff/db/migrations/00007_create_eth_receipt_cids_table.sql

@@ -0,0 +1,20 @@
+-- +goose Up
+CREATE TABLE eth.receipt_cids (
+  id                    SERIAL PRIMARY KEY,
+  tx_id                 INTEGER NOT NULL REFERENCES eth.transaction_cids (id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  cid                   TEXT NOT NULL,
+  mh_key                TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  contract              VARCHAR(66),
+  contract_hash         VARCHAR(66),
+  topic0s               VARCHAR(66)[],
+  topic1s               VARCHAR(66)[],
+  topic2s               VARCHAR(66)[],
+  topic3s               VARCHAR(66)[],
+  log_contracts         VARCHAR(66)[],
+  post_state            VARCHAR(66),
+  post_status           INTEGER,
+  UNIQUE (tx_id)
+);
+
+-- +goose Down
+DROP TABLE eth.receipt_cids;

statediff/db/migrations/00008_create_eth_state_cids_table.sql

@@ -0,0 +1,15 @@
+-- +goose Up
+CREATE TABLE eth.state_cids (
+  id                    BIGSERIAL PRIMARY KEY,
+  header_id             INTEGER NOT NULL REFERENCES eth.header_cids (id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  state_leaf_key        VARCHAR(66),
+  cid                   TEXT NOT NULL,
+  mh_key                TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  state_path            BYTEA,
+  node_type             INTEGER NOT NULL,
+  diff                  BOOLEAN NOT NULL DEFAULT FALSE,
+  UNIQUE (header_id, state_path)
+);
+
+-- +goose Down
+DROP TABLE eth.state_cids;

statediff/db/migrations/00009_create_eth_storage_cids_table.sql

@@ -0,0 +1,15 @@
+-- +goose Up
+CREATE TABLE eth.storage_cids (
+  id                    BIGSERIAL PRIMARY KEY,
+  state_id              BIGINT NOT NULL REFERENCES eth.state_cids (id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  storage_leaf_key      VARCHAR(66),
+  cid                   TEXT NOT NULL,
+  mh_key                TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  storage_path          BYTEA,
+  node_type             INTEGER NOT NULL,
+  diff                  BOOLEAN NOT NULL DEFAULT FALSE,
+  UNIQUE (state_id, storage_path)
+);
+
+-- +goose Down
+DROP TABLE eth.storage_cids;

statediff/db/migrations/00010_create_eth_state_accouts_table.sql

@@ -0,0 +1,13 @@
+-- +goose Up
+CREATE TABLE eth.state_accounts (
+  id                    SERIAL PRIMARY KEY,
+  state_id              BIGINT NOT NULL REFERENCES eth.state_cids (id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+  balance               NUMERIC NOT NULL,
+  nonce                 INTEGER NOT NULL,
+  code_hash             BYTEA NOT NULL,
+  storage_root          VARCHAR(66) NOT NULL,
+  UNIQUE (state_id)
+);
+
+-- +goose Down
+DROP TABLE eth.state_accounts;

statediff/db/migrations/00011_create_postgraphile_comments.sql

@@ -0,0 +1,6 @@
+-- +goose Up
+COMMENT ON TABLE public.nodes IS E'@name NodeInfo';
+COMMENT ON TABLE eth.transaction_cids IS E'@name EthTransactionCids';
+COMMENT ON TABLE eth.header_cids IS E'@name EthHeaderCids';
+COMMENT ON COLUMN public.nodes.node_id IS E'@name ChainNodeID';
+COMMENT ON COLUMN eth.header_cids.node_id IS E'@name EthNodeID';

statediff/db/migrations/00012_potgraphile_triggers.sql

@@ -0,0 +1,69 @@
+-- +goose Up
+-- +goose StatementBegin
+CREATE FUNCTION eth.graphql_subscription() returns TRIGGER as $$
+declare
+    table_name text = TG_ARGV[0];
+    attribute text = TG_ARGV[1];
+    id text;
+begin
+    execute 'select $1.' || quote_ident(attribute)
+        using new
+        into id;
+    perform pg_notify('postgraphile:' || table_name,
+                      json_build_object(
+                              '__node__', json_build_array(
+                              table_name,
+                              id
+                          )
+                          )::text
+        );
+    return new;
+end;
+$$ language plpgsql;
+-- +goose StatementEnd
+
+CREATE TRIGGER header_cids_ai
+    after INSERT ON eth.header_cids
+    for each row
+    execute procedure eth.graphql_subscription('header_cids', 'id');
+
+CREATE TRIGGER receipt_cids_ai
+    after INSERT ON eth.receipt_cids
+    for each row
+    execute procedure eth.graphql_subscription('receipt_cids', 'id');
+
+CREATE TRIGGER state_accounts_ai
+    after INSERT ON eth.state_accounts
+    for each row
+    execute procedure eth.graphql_subscription('state_accounts', 'id');
+
+CREATE TRIGGER state_cids_ai
+    after INSERT ON eth.state_cids
+    for each row
+    execute procedure eth.graphql_subscription('state_cids', 'id');
+
+CREATE TRIGGER storage_cids_ai
+    after INSERT ON eth.storage_cids
+    for each row
+    execute procedure eth.graphql_subscription('storage_cids', 'id');
+
+CREATE TRIGGER transaction_cids_ai
+    after INSERT ON eth.transaction_cids
+    for each row
+    execute procedure eth.graphql_subscription('transaction_cids', 'id');
+
+CREATE TRIGGER uncle_cids_ai
+    after INSERT ON eth.uncle_cids
+    for each row
+    execute procedure eth.graphql_subscription('uncle_cids', 'id');
+
+-- +goose Down
+DROP TRIGGER uncle_cids_ai ON eth.uncle_cids;
+DROP TRIGGER transaction_cids_ai ON eth.transaction_cids;
+DROP TRIGGER storage_cids_ai ON eth.storage_cids;
+DROP TRIGGER state_cids_ai ON eth.state_cids;
+DROP TRIGGER state_accounts_ai ON eth.state_accounts;
+DROP TRIGGER receipt_cids_ai ON eth.receipt_cids;
+DROP TRIGGER header_cids_ai ON eth.header_cids;
+
+DROP FUNCTION eth.graphql_subscription();

statediff/db/migrations/00013_create_cid_indexes.sql

@@ -0,0 +1,121 @@
+-- +goose Up
+-- header indexes
+CREATE INDEX block_number_index ON eth.header_cids USING brin (block_number);
+
+CREATE INDEX block_hash_index ON eth.header_cids USING btree (block_hash);
+
+CREATE INDEX header_cid_index ON eth.header_cids USING btree (cid);
+
+CREATE INDEX header_mh_index ON eth.header_cids USING btree (mh_key);
+
+CREATE INDEX state_root_index ON eth.header_cids USING btree (state_root);
+
+CREATE INDEX timestamp_index ON eth.header_cids USING brin (timestamp);
+
+-- transaction indexes
+CREATE INDEX tx_header_id_index ON eth.transaction_cids USING btree (header_id);
+
+CREATE INDEX tx_hash_index ON eth.transaction_cids USING btree (tx_hash);
+
+CREATE INDEX tx_cid_index ON eth.transaction_cids USING btree (cid);
+
+CREATE INDEX tx_mh_index ON eth.transaction_cids USING btree (mh_key);
+
+CREATE INDEX tx_dst_index ON eth.transaction_cids USING btree (dst);
+
+CREATE INDEX tx_src_index ON eth.transaction_cids USING btree (src);
+
+-- receipt indexes
+CREATE INDEX rct_tx_id_index ON eth.receipt_cids USING btree (tx_id);
+
+CREATE INDEX rct_cid_index ON eth.receipt_cids USING btree (cid);
+
+CREATE INDEX rct_mh_index ON eth.receipt_cids USING btree (mh_key);
+
+CREATE INDEX rct_contract_index ON eth.receipt_cids USING btree (contract);
+
+CREATE INDEX rct_contract_hash_index ON eth.receipt_cids USING btree (contract_hash);
+
+CREATE INDEX rct_topic0_index ON eth.receipt_cids USING gin (topic0s);
+
+CREATE INDEX rct_topic1_index ON eth.receipt_cids USING gin (topic1s);
+
+CREATE INDEX rct_topic2_index ON eth.receipt_cids USING gin (topic2s);
+
+CREATE INDEX rct_topic3_index ON eth.receipt_cids USING gin (topic3s);
+
+CREATE INDEX rct_log_contract_index ON eth.receipt_cids USING gin (log_contracts);
+
+-- state node indexes
+CREATE INDEX state_header_id_index ON eth.state_cids USING btree (header_id);
+
+CREATE INDEX state_leaf_key_index ON eth.state_cids USING btree (state_leaf_key);
+
+CREATE INDEX state_cid_index ON eth.state_cids USING btree (cid);
+
+CREATE INDEX state_mh_index ON eth.state_cids USING btree (mh_key);
+
+CREATE INDEX state_path_index ON eth.state_cids USING btree (state_path);
+
+-- storage node indexes
+CREATE INDEX storage_state_id_index ON eth.storage_cids USING btree (state_id);
+
+CREATE INDEX storage_leaf_key_index ON eth.storage_cids USING btree (storage_leaf_key);
+
+CREATE INDEX storage_cid_index ON eth.storage_cids USING btree (cid);
+
+CREATE INDEX storage_mh_index ON eth.storage_cids USING btree (mh_key);
+
+CREATE INDEX storage_path_index ON eth.storage_cids USING btree (storage_path);
+
+-- state accounts indexes
+CREATE INDEX account_state_id_index ON eth.state_accounts USING btree (state_id);
+
+CREATE INDEX storage_root_index ON eth.state_accounts USING btree (storage_root);
+
+-- +goose Down
+-- state account indexes
+DROP INDEX eth.storage_root_index;
+DROP INDEX eth.account_state_id_index;
+
+-- storage node indexes
+DROP INDEX eth.storage_path_index;
+DROP INDEX eth.storage_mh_index;
+DROP INDEX eth.storage_cid_index;
+DROP INDEX eth.storage_leaf_key_index;
+DROP INDEX eth.storage_state_id_index;
+
+-- state node indexes
+DROP INDEX eth.state_path_index;
+DROP INDEX eth.state_mh_index;
+DROP INDEX eth.state_cid_index;
+DROP INDEX eth.state_leaf_key_index;
+DROP INDEX eth.state_header_id_index;
+
+-- receipt indexes
+DROP INDEX eth.rct_log_contract_index;
+DROP INDEX eth.rct_topic3_index;
+DROP INDEX eth.rct_topic2_index;
+DROP INDEX eth.rct_topic1_index;
+DROP INDEX eth.rct_topic0_index;
+DROP INDEX eth.rct_contract_hash_index;
+DROP INDEX eth.rct_contract_index;
+DROP INDEX eth.rct_mh_index;
+DROP INDEX eth.rct_cid_index;
+DROP INDEX eth.rct_tx_id_index;
+
+-- transaction indexes
+DROP INDEX eth.tx_src_index;
+DROP INDEX eth.tx_dst_index;
+DROP INDEX eth.tx_mh_index;
+DROP INDEX eth.tx_cid_index;
+DROP INDEX eth.tx_hash_index;
+DROP INDEX eth.tx_header_id_index;
+
+-- header indexes
+DROP INDEX eth.timestamp_index;
+DROP INDEX eth.state_root_index;
+DROP INDEX eth.header_mh_index;
+DROP INDEX eth.header_cid_index;
+DROP INDEX eth.block_hash_index;
+DROP INDEX eth.block_number_index;

statediff/db/migrations/00014_create_stored_functions.sql

@@ -0,0 +1,158 @@
+-- +goose Up
+-- +goose StatementBegin
+-- returns if a storage node at the provided path was removed in the range > the provided height and <= the provided block hash
+CREATE OR REPLACE FUNCTION was_storage_removed(path BYTEA, height BIGINT, hash VARCHAR(66)) RETURNS BOOLEAN
+AS $$
+SELECT exists(SELECT 1
+              FROM eth.storage_cids
+                INNER JOIN eth.state_cids ON (storage_cids.state_id = state_cids.id)
+                INNER JOIN eth.header_cids ON (state_cids.header_id = header_cids.id)
+              WHERE storage_path = path
+                AND block_number > height
+                AND block_number <= (SELECT block_number
+                                     FROM eth.header_cids
+                                     WHERE block_hash = hash)
+                AND storage_cids.node_type = 3
+              LIMIT 1);
+$$ LANGUAGE SQL;
+-- +goose StatementEnd
+
+-- +goose StatementBegin
+-- returns if a state node at the provided path was removed in the range > the provided height and <= the provided block hash
+CREATE OR REPLACE FUNCTION was_state_removed(path BYTEA, height BIGINT, hash VARCHAR(66)) RETURNS BOOLEAN
+AS $$
+SELECT exists(SELECT 1
+              FROM eth.state_cids
+                INNER JOIN eth.header_cids ON (state_cids.header_id = header_cids.id)
+              WHERE state_path = path
+                AND block_number > height
+                AND block_number <= (SELECT block_number
+                                     FROM eth.header_cids
+                                     WHERE block_hash = hash)
+                AND state_cids.node_type = 3
+              LIMIT 1);
+$$ LANGUAGE SQL;
+-- +goose StatementEnd
+
+-- +goose StatementBegin
+CREATE TYPE child_result AS (
+    has_child BOOLEAN,
+    children eth.header_cids[]
+);
+
+CREATE OR REPLACE FUNCTION has_child(hash VARCHAR(66), height BIGINT) RETURNS child_result AS
+$BODY$
+DECLARE
+  child_height INT;
+  temp_child eth.header_cids;
+  new_child_result child_result;
+BEGIN
+  child_height = height + 1;
+  -- short circuit if there are no children
+  SELECT exists(SELECT 1
+              FROM eth.header_cids
+              WHERE parent_hash = hash
+                AND block_number = child_height
+              LIMIT 1)
+  INTO new_child_result.has_child;
+  -- collect all the children for this header
+  IF new_child_result.has_child THEN
+    FOR temp_child IN
+    SELECT * FROM eth.header_cids WHERE parent_hash = hash AND block_number = child_height
+    LOOP
+      new_child_result.children = array_append(new_child_result.children, temp_child);
+    END LOOP;
+  END IF;
+RETURN new_child_result;
+END
+$BODY$
+LANGUAGE 'plpgsql';
+-- +goose StatementEnd
+
+-- +goose StatementBegin
+CREATE OR REPLACE FUNCTION canonical_header_from_array(headers eth.header_cids[]) RETURNS eth.header_cids AS
+$BODY$
+DECLARE
+  canonical_header eth.header_cids;
+  canonical_child eth.header_cids;
+  header eth.header_cids;
+  current_child_result child_result;
+  child_headers eth.header_cids[];
+  current_header_with_child eth.header_cids;
+  has_children_count INT DEFAULT 0;
+BEGIN
+  -- for each header in the provided set
+  FOREACH header IN ARRAY headers
+  LOOP
+    -- check if it has any children
+    current_child_result = has_child(header.block_hash, header.block_number);
+    IF current_child_result.has_child THEN
+      -- if it does, take note
+      has_children_count = has_children_count + 1;
+      current_header_with_child = header;
+      -- and add the children to the growing set of child headers
+      child_headers = array_cat(child_headers, current_child_result.children);
+    END IF;
+  END LOOP;
+  -- if none of the headers had children, none is more canonical than the other
+  IF has_children_count = 0 THEN
+    -- return the first one selected
+    SELECT * INTO canonical_header FROM unnest(headers) LIMIT 1;
+  -- if only one header had children, it can be considered the heaviest/canonical header of the set
+  ELSIF has_children_count = 1 THEN
+    -- return the only header with a child
+    canonical_header = current_header_with_child;
+  -- if there are multiple headers with children
+  ELSE
+    -- find the canonical header from the child set
+    canonical_child = canonical_header_from_array(child_headers);
+    -- the header that is parent to this header, is the canonical header at this level
+    SELECT * INTO canonical_header FROM unnest(headers)
+    WHERE block_hash = canonical_child.parent_hash;
+  END IF;
+  RETURN canonical_header;
+END
+$BODY$
+LANGUAGE 'plpgsql';
+-- +goose StatementEnd
+
+-- +goose StatementBegin
+CREATE OR REPLACE FUNCTION canonical_header_id(height BIGINT) RETURNS INTEGER AS
+$BODY$
+DECLARE
+  canonical_header eth.header_cids;
+  headers eth.header_cids[];
+  header_count INT;
+  temp_header eth.header_cids;
+BEGIN
+  -- collect all headers at this height
+  FOR temp_header IN
+  SELECT * FROM eth.header_cids WHERE block_number = height
+  LOOP
+    headers = array_append(headers, temp_header);
+  END LOOP;
+  -- count the number of headers collected
+  header_count = array_length(headers, 1);
+  -- if we have less than 1 header, return NULL
+  IF header_count IS NULL OR header_count < 1 THEN
+    RETURN NULL;
+  -- if we have one header, return its id
+  ELSIF header_count = 1 THEN
+    RETURN headers[1].id;
+  -- if we have multiple headers we need to determine which one is canonical
+  ELSE
+    canonical_header = canonical_header_from_array(headers);
+    RETURN canonical_header.id;
+  END IF;
+END;
+$BODY$
+LANGUAGE 'plpgsql';
+-- +goose StatementEnd
+
+-- +goose Down
+DROP FUNCTION was_storage_removed;
+DROP FUNCTION was_state_removed;
+DROP FUNCTION canonical_header_id;
+DROP FUNCTION canonical_header_from_array;
+DROP FUNCTION has_child;
+DROP TYPE child_result;

statediff/db/migrations/00015_create_access_list_table.sql

@@ -0,0 +1,15 @@
+-- +goose Up
+CREATE TABLE eth.access_list_element (
+   id                    SERIAL PRIMARY KEY,
+   tx_id                 INTEGER NOT NULL REFERENCES eth.transaction_cids (id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+   index                 INTEGER NOT NULL,
+   address               VARCHAR(66),
+   storage_keys          VARCHAR(66)[],
+   UNIQUE (tx_id, index)
+);
+
+CREATE INDEX accesss_list_element_address_index ON eth.access_list_element USING btree (address);
+
+-- +goose Down
+DROP INDEX eth.accesss_list_element_address_index;
+DROP TABLE eth.access_list_element;

statediff/doc.md

@@ -0,0 +1,216 @@
+# Statediff
+
+This package provides an auxiliary service that asynchronously processes state diff objects from chain events,
+either relaying the state objects to RPC subscribers or writing them directly to Postgres as IPLD objects.
+
+It also exposes RPC endpoints for fetching or writing to Postgres the state diff at a specific block height
+or for a specific block hash, this operates on historical block and state data and so depends on a complete state archive.
+
+Data is emitted in this differential format in order to make it feasible to IPLD-ize and index the *entire* Ethereum state
+(including intermediate state and storage trie nodes). If this state diff process is ran continuously from genesis,
+the entire state at any block can be materialized from the cumulative differentials up to that point.
+
+## Statediff object
+A state diff `StateObject` is the collection of all the state and storage trie nodes that have been updated in a given block.
+For convenience, we also associate these nodes with the block number and hash, and optionally the set of code hashes and code for any
+contracts deployed in this block.
+
+A complete state diff `StateObject` will include all state and storage intermediate nodes, which is necessary for generating proofs and for
+traversing the tries.
+
+```go
+// StateObject is a collection of state (and linked storage nodes) as well as the associated block number, block hash,
+// and a set of code hashes and their code
+type StateObject struct {
+	BlockNumber       *big.Int                `json:"blockNumber"     gencodec:"required"`
+	BlockHash         common.Hash             `json:"blockHash"       gencodec:"required"`
+	Nodes             []StateNode             `json:"nodes"           gencodec:"required"`
+	CodeAndCodeHashes []CodeAndCodeHash       `json:"codeMapping"`
+}
+
+// StateNode holds the data for a single state diff node
+type StateNode struct {
+	NodeType     NodeType      `json:"nodeType"        gencodec:"required"`
+	Path         []byte        `json:"path"            gencodec:"required"`
+	NodeValue    []byte        `json:"value"           gencodec:"required"`
+	StorageNodes []StorageNode `json:"storage"`
+	LeafKey      []byte        `json:"leafKey"`
+}
+
+// StorageNode holds the data for a single storage diff node
+type StorageNode struct {
+	NodeType  NodeType `json:"nodeType"        gencodec:"required"`
+	Path      []byte   `json:"path"            gencodec:"required"`
+	NodeValue []byte   `json:"value"           gencodec:"required"`
+	LeafKey   []byte   `json:"leafKey"`
+}
+
+// CodeAndCodeHash struct for holding codehash => code mappings
+// we can't use an actual map because they are not rlp serializable
+type CodeAndCodeHash struct {
+	Hash common.Hash `json:"codeHash"`
+	Code []byte      `json:"code"`
+}
+```
+These objects are packed into a `Payload` structure which can additionally associate the `StateObject`
+with the block (header, uncles, and transactions), receipts, and total difficulty.
+This `Payload` encapsulates all of the differential data at a given block, and allows us to index the entire Ethereum data structure
+as hash-linked IPLD objects.
+
+```go
+// Payload packages the data to send to state diff subscriptions
+type Payload struct {
+	BlockRlp        []byte   `json:"blockRlp"`
+	TotalDifficulty *big.Int `json:"totalDifficulty"`
+	ReceiptsRlp     []byte   `json:"receiptsRlp"`
+	StateObjectRlp  []byte   `json:"stateObjectRlp"    gencodec:"required"`
+
+	encoded []byte
+	err     error
+}
+```
+
+## Usage
+This state diffing service runs as an auxiliary service concurrent to the regular syncing process of the geth node.
+
+
+### CLI configuration
+This service introduces a CLI flag namespace `statediff`
+
+`--statediff` flag is used to turn on the service
+`--statediff.writing` is used to tell the service to write state diff objects it produces from synced ChainEvents directly to a configured Postgres database
+`--statediff.workers` is used to set the number of concurrent workers to process state diff objects and write them into the database
+`--statediff.db` is the connection string for the Postgres database to write to
+`--statediff.dbnodeid` is the node id to use in the Postgres database
+`--statediff.dbclientname` is the client name to use in the Postgres database
+
+The service can only operate in full sync mode (`--syncmode=full`), but only the historical RPC endpoints require an archive node (`--gcmode=archive`)
+
+e.g.
+`
+./build/bin/geth --syncmode=full --gcmode=archive --statediff --statediff.writing --statediff.db=postgres://localhost:5432/vulcanize_testing?sslmode=disable --statediff.dbnodeid={nodeId} --statediff.dbclientname={dbClientName}
+`
+
+### RPC endpoints
+The state diffing service exposes both a WS subscription endpoint, and a number of HTTP unary endpoints.
+
+Each of these endpoints requires a set of parameters provided by the caller
+
+```go
+// Params is used to carry in parameters from subscribing/requesting clients configuration
+type Params struct {
+	IntermediateStateNodes   bool
+	IntermediateStorageNodes bool
+	IncludeBlock             bool
+	IncludeReceipts          bool
+	IncludeTD                bool
+	IncludeCode              bool
+	WatchedAddresses         []common.Address
+	WatchedStorageSlots      []common.Hash
+}
+```
+
+Using these params we can tell the service whether to include state and/or storage intermediate nodes; whether
+to include the associated block (header, uncles, and transactions); whether to include the associated receipts;
+whether to include the total difficulty for this block; whether to include the set of code hashes and code for
+contracts deployed in this block; whether to limit the diffing process to a list of specific addresses; and/or
+whether to limit the diffing process to a list of specific storage slot keys.
+
+#### Subscription endpoint
+A websocket supporting RPC endpoint is exposed for subscribing to state diff `StateObjects` that come off the head of the chain while the geth node syncs.
+
+```go
+// Stream is a subscription endpoint that fires off state diff payloads as they are created
+Stream(ctx context.Context, params Params) (*rpc.Subscription, error)
+```
+
+To expose this endpoint the node needs to have the websocket server turned on (`--ws`),
+and the `statediff` namespace exposed (`--ws.api=statediff`).
+
+Go code subscriptions to this endpoint can be created using the `rpc.Client.Subscribe()` method,
+with the "statediff" namespace, a `statediff.Payload` channel, and the name of the statediff api's rpc method: "stream".
+
+e.g.
+
+```go
+
+cli, err := rpc.Dial("ipcPathOrWsURL")
+if err != nil {
+	// handle error
+}
+stateDiffPayloadChan := make(chan statediff.Payload, 20000)
+methodName := "stream"
+params := statediff.Params{
+    IncludeBlock:             true,
+    IncludeTD:                true,
+    IncludeReceipts:          true,
+    IntermediateStorageNodes: true,
+    IntermediateStateNodes:   true,
+}
+rpcSub, err := cli.Subscribe(context.Background(), statediff.APIName, stateDiffPayloadChan, methodName, params)
+if err != nil {
+	// handle error
+}
+for {
+	select {
+	case stateDiffPayload := <- stateDiffPayloadChan:
+            // process the payload
+        case err := <- rpcSub.Err():
+    	    // handle rpc subscription error
+        }
+}
+```
+
+#### Unary endpoints
+The service also exposes unary RPC endpoints for retrieving the state diff `StateObject` for a specific block height/hash.
+```go
+// StateDiffAt returns a state diff payload at the specific blockheight
+StateDiffAt(ctx context.Context, blockNumber uint64, params Params) (*Payload, error)
+
+// StateDiffFor returns a state diff payload for the specific blockhash
+StateDiffFor(ctx context.Context, blockHash common.Hash, params Params) (*Payload, error)
+```
+
+To expose this endpoint the node needs to have the HTTP server turned on (`--http`),
+and the `statediff` namespace exposed (`--http.api=statediff`).
+
+### Direct indexing into Postgres
+If `--statediff.writing` is set, the service will convert the state diff `StateObject` data into IPLD objects, persist them directly to Postgres,
+and generate secondary indexes around the IPLD data.
+
+The schema and migrations for this Postgres database are provided in `statediff/db/`.
+
+#### Postgres setup
+We use [pressly/goose](https://github.com/pressly/goose) as our Postgres migration manager.
+You can also load the Postgres schema directly into a database using
+
+`psql database_name < schema.sql`
+
+This will only work on a version 12.4 Postgres database.
+
+#### Schema overview
+Our Postgres schemas are built around a single IPFS backing Postgres IPLD blockstore table (`public.blocks`) that conforms with [go-ds-sql](https://github.com/ipfs/go-ds-sql/blob/master/postgres/postgres.go).
+All IPLD objects are stored in this table, where `key` is the blockstore-prefixed multihash key for the IPLD object and `data` contains
+the bytes for the IPLD block (in the case of all Ethereum IPLDs, this is the RLP byte encoding of the Ethereum object).
+
+The IPLD objects in this table can be traversed using an IPLD DAG interface, but since this table only maps multihash to raw IPLD object
+it is not particularly useful for searching through the data by looking up Ethereum objects by their constituent fields
+(e.g. by block number, tx source/recipient, state/storage trie node path). To improve the accessibility of these objects
+we create an Ethereum [advanced data layout](https://github.com/ipld/specs#schemas-and-advanced-data-layouts) (ADL) by generating secondary
+indexes on top of the raw IPLDs in other Postgres tables.
+
+These secondary index tables fall under the `eth` schema and follow an `{objectType}_cids` naming convention.
+These tables provide a view into individual fields of the underlying Ethereum IPLD objects, allowing lookups on these fields, and reference the raw IPLD objects stored in `public.blocks`
+by foreign keys to their multihash keys.
+Additionally, these tables maintain the hash-linked nature of Ethereum objects to one another. E.g. a storage trie node entry in the `storage_cids`
+table contains a `state_id` foreign key which references the `id` for the `state_cids` entry that contains the state leaf node for the contract that storage node belongs to,
+and in turn that `state_cids` entry contains a `header_id` foreign key which references the `id` of the `header_cids` entry that contains the header for the block these state and storage nodes were updated (diffed).
+
+### Optimization
+On mainnet this process is extremely IO intensive and requires significant resources to allow it to keep up with the head of the chain.
+The state diff processing time for a specific block is dependent on the number and complexity of the state changes that occur in a block and
+the number of updated state nodes that are available in the in-memory cache vs must be retrieved from disc.
+
+If memory permits, one means of improving the efficiency of this process is to increase the in-memory trie cache allocation.
+This can be done by increasing the overall `--cache` allocation and/or by increasing the % of the cache allocated to trie
+usage with `--cache.trie`.

statediff/helpers.go

@@ -0,0 +1,98 @@
+// Copyright 2019 The go-ethereum Authors
+// This file is part of the go-ethereum library.
+//
+// The go-ethereum library is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Lesser General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// The go-ethereum library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public License
+// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
+
+// Contains a batch of utility type declarations used by the tests. As the node
+// operates on unique types, a lot of them are needed to check various features.
+
+package statediff
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+
+	sdtypes "github.com/ethereum/go-ethereum/statediff/types"
+)
+
+func sortKeys(data AccountMap) []string {
+	keys := make([]string, 0, len(data))
+	for key := range data {
+		keys = append(keys, key)
+	}
+	sort.Strings(keys)
+
+	return keys
+}
+
+// findIntersection finds the set of strings from both arrays that are equivalent
+// a and b must first be sorted
+// this is used to find which keys have been both "deleted" and "created" i.e. they were updated
+func findIntersection(a, b []string) []string {
+	lenA := len(a)
+	lenB := len(b)
+	iOfA, iOfB := 0, 0
+	updates := make([]string, 0)
+	if iOfA >= lenA || iOfB >= lenB {
+		return updates
+	}
+	for {
+		switch strings.Compare(a[iOfA], b[iOfB]) {
+		// -1 when a[iOfA] < b[iOfB]
+		case -1:
+			iOfA++
+			if iOfA >= lenA {
+				return updates
+			}
+			// 0 when a[iOfA] == b[iOfB]
+		case 0:
+			updates = append(updates, a[iOfA])
+			iOfA++
+			iOfB++
+			if iOfA >= lenA || iOfB >= lenB {
+				return updates
+			}
+			// 1 when a[iOfA] > b[iOfB]
+		case 1:
+			iOfB++
+			if iOfB >= lenB {
+				return updates
+			}
+		}
+	}
+
+}
+
+// CheckKeyType checks what type of key we have
+func CheckKeyType(elements []interface{}) (sdtypes.NodeType, error) {
+	if len(elements) > 2 {
+		return sdtypes.Branch, nil
+	}
+	if len(elements) < 2 {
+		return sdtypes.Unknown, fmt.Errorf("node cannot be less than two elements in length")
+	}
+	switch elements[0].([]byte)[0] / 16 {
+	case '\x00':
+		return sdtypes.Extension, nil
+	case '\x01':
+		return sdtypes.Extension, nil
+	case '\x02':
+		return sdtypes.Leaf, nil
+	case '\x03':
+		return sdtypes.Leaf, nil
+	default:
+		return sdtypes.Unknown, fmt.Errorf("unknown hex prefix")
+	}
+}

statediff/indexer/helpers.go

@@ -0,0 +1,55 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package indexer
+
+import (
+	"fmt"
+
+	"github.com/ethereum/go-ethereum/params"
+	"github.com/ethereum/go-ethereum/statediff/types"
+)
+
+func ResolveFromNodeType(nodeType types.NodeType) int {
+	switch nodeType {
+	case types.Branch:
+		return 0
+	case types.Extension:
+		return 1
+	case types.Leaf:
+		return 2
+	case types.Removed:
+		return 3
+	default:
+		return -1
+	}
+}
+
+// ChainConfig returns the appropriate ethereum chain config for the provided chain id
+func ChainConfig(chainID uint64) (*params.ChainConfig, error) {
+	switch chainID {
+	case 1:
+		return params.MainnetChainConfig, nil
+	case 3:
+		return params.RopstenChainConfig, nil
+	case 4:
+		return params.RinkebyChainConfig, nil
+	case 5:
+		return params.GoerliChainConfig, nil
+	default:
+		return nil, fmt.Errorf("chain config for chainid %d not available", chainID)
+	}
+}

statediff/indexer/indexer.go

@@ -0,0 +1,459 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+// This package provides an interface for pushing and indexing IPLD objects into a Postgres database
+// Metrics for reporting processing and connection stats are defined in ./metrics.go
+package indexer
+
+import (
+	"fmt"
+	"math/big"
+	"time"
+
+	"github.com/lib/pq"
+
+	"github.com/ethereum/go-ethereum/common"
+	"github.com/ethereum/go-ethereum/core/state"
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/crypto"
+	"github.com/ethereum/go-ethereum/log"
+	"github.com/ethereum/go-ethereum/metrics"
+	"github.com/ethereum/go-ethereum/params"
+	"github.com/ethereum/go-ethereum/rlp"
+	"github.com/ethereum/go-ethereum/statediff/indexer/ipfs/ipld"
+	"github.com/ethereum/go-ethereum/statediff/indexer/models"
+	"github.com/ethereum/go-ethereum/statediff/indexer/postgres"
+	"github.com/ethereum/go-ethereum/statediff/indexer/shared"
+	sdtypes "github.com/ethereum/go-ethereum/statediff/types"
+
+	node "github.com/ipfs/go-ipld-format"
+	"github.com/jmoiron/sqlx"
+	"github.com/multiformats/go-multihash"
+)
+
+var (
+	indexerMetrics = RegisterIndexerMetrics(metrics.DefaultRegistry)
+	dbMetrics      = RegisterDBMetrics(metrics.DefaultRegistry)
+)
+
+// Indexer interface to allow substitution of mocks for testing
+type Indexer interface {
+	PushBlock(block *types.Block, receipts types.Receipts, totalDifficulty *big.Int) (*BlockTx, error)
+	PushStateNode(tx *BlockTx, stateNode sdtypes.StateNode) error
+	PushCodeAndCodeHash(tx *BlockTx, codeAndCodeHash sdtypes.CodeAndCodeHash) error
+	ReportDBMetrics(delay time.Duration, quit <-chan bool)
+}
+
+// StateDiffIndexer satisfies the Indexer interface for ethereum statediff objects
+type StateDiffIndexer struct {
+	chainConfig *params.ChainConfig
+	dbWriter    *PostgresCIDWriter
+}
+
+// NewStateDiffIndexer creates a pointer to a new PayloadConverter which satisfies the PayloadConverter interface
+func NewStateDiffIndexer(chainConfig *params.ChainConfig, db *postgres.DB) *StateDiffIndexer {
+	return &StateDiffIndexer{
+		chainConfig: chainConfig,
+		dbWriter:    NewPostgresCIDWriter(db),
+	}
+}
+
+type BlockTx struct {
+	dbtx        *sqlx.Tx
+	BlockNumber uint64
+	headerID    int64
+	Close       func(err error) error
+}
+
+// Reporting function to run as goroutine
+func (sdi *StateDiffIndexer) ReportDBMetrics(delay time.Duration, quit <-chan bool) {
+	if !metrics.Enabled {
+		return
+	}
+	ticker := time.NewTicker(delay)
+	go func() {
+		for {
+			select {
+			case <-ticker.C:
+				dbMetrics.Update(sdi.dbWriter.db.Stats())
+			case <-quit:
+				ticker.Stop()
+				return
+			}
+		}
+	}()
+}
+
+// Pushes and indexes block data in database, except state & storage nodes (includes header, uncles, transactions & receipts)
+// Returns an initiated DB transaction which must be Closed via defer to commit or rollback
+func (sdi *StateDiffIndexer) PushBlock(block *types.Block, receipts types.Receipts, totalDifficulty *big.Int) (*BlockTx, error) {
+	start, t := time.Now(), time.Now()
+	blockHash := block.Hash()
+	blockHashStr := blockHash.String()
+	height := block.NumberU64()
+	traceMsg := fmt.Sprintf("indexer stats for statediff at %d with hash %s:\r\n", height, blockHashStr)
+	transactions := block.Transactions()
+	// Derive any missing fields
+	if err := receipts.DeriveFields(sdi.chainConfig, blockHash, height, transactions); err != nil {
+		return nil, err
+	}
+	// Generate the block iplds
+	headerNode, uncleNodes, txNodes, txTrieNodes, rctNodes, rctTrieNodes, err := ipld.FromBlockAndReceipts(block, receipts)
+	if err != nil {
+		return nil, fmt.Errorf("error creating IPLD nodes from block and receipts: %v", err)
+	}
+	if len(txNodes) != len(txTrieNodes) && len(rctNodes) != len(rctTrieNodes) && len(txNodes) != len(rctNodes) {
+		return nil, fmt.Errorf("expected number of transactions (%d), transaction trie nodes (%d), receipts (%d), and receipt trie nodes (%d)to be equal", len(txNodes), len(txTrieNodes), len(rctNodes), len(rctTrieNodes))
+	}
+	// Calculate reward
+	reward := CalcEthBlockReward(block.Header(), block.Uncles(), block.Transactions(), receipts)
+	t = time.Now()
+	// Begin new db tx for everything
+	tx, err := sdi.dbWriter.db.Beginx()
+	if err != nil {
+		return nil, err
+	}
+	defer func() {
+		if p := recover(); p != nil {
+			shared.Rollback(tx)
+			panic(p)
+		} else if err != nil {
+			shared.Rollback(tx)
+		}
+	}()
+	blockTx := &BlockTx{
+		dbtx: tx,
+		// handle transaction commit or rollback for any return case
+		Close: func(err error) error {
+			if p := recover(); p != nil {
+				shared.Rollback(tx)
+				panic(p)
+			} else if err != nil {
+				shared.Rollback(tx)
+			} else {
+				tDiff := time.Since(t)
+				indexerMetrics.tStateStoreCodeProcessing.Update(tDiff)
+				traceMsg += fmt.Sprintf("state, storage, and code storage processing time: %s\r\n", tDiff.String())
+				t = time.Now()
+				err = tx.Commit()
+				tDiff = time.Since(t)
+				indexerMetrics.tPostgresCommit.Update(tDiff)
+				traceMsg += fmt.Sprintf("postgres transaction commit duration: %s\r\n", tDiff.String())
+			}
+			traceMsg += fmt.Sprintf(" TOTAL PROCESSING DURATION: %s\r\n", time.Since(start).String())
+			log.Debug(traceMsg)
+			return err
+		},
+	}
+	tDiff := time.Since(t)
+	indexerMetrics.tFreePostgres.Update(tDiff)
+
+	traceMsg += fmt.Sprintf("time spent waiting for free postgres tx: %s:\r\n", tDiff.String())
+	t = time.Now()
+
+	// Publish and index header, collect headerID
+	var headerID int64
+	headerID, err = sdi.processHeader(tx, block.Header(), headerNode, reward, totalDifficulty)
+	if err != nil {
+		return nil, err
+	}
+	tDiff = time.Since(t)
+	indexerMetrics.tHeaderProcessing.Update(tDiff)
+	traceMsg += fmt.Sprintf("header processing time: %s\r\n", tDiff.String())
+	t = time.Now()
+	// Publish and index uncles
+	err = sdi.processUncles(tx, headerID, height, uncleNodes)
+	if err != nil {
+		return nil, err
+	}
+	tDiff = time.Since(t)
+	indexerMetrics.tUncleProcessing.Update(tDiff)
+	traceMsg += fmt.Sprintf("uncle processing time: %s\r\n", tDiff.String())
+	t = time.Now()
+	// Publish and index receipts and txs
+	err = sdi.processReceiptsAndTxs(tx, processArgs{
+		headerID:     headerID,
+		blockNumber:  block.Number(),
+		receipts:     receipts,
+		txs:          transactions,
+		rctNodes:     rctNodes,
+		rctTrieNodes: rctTrieNodes,
+		txNodes:      txNodes,
+		txTrieNodes:  txTrieNodes,
+	})
+	if err != nil {
+		return nil, err
+	}
+	tDiff = time.Since(t)
+	indexerMetrics.tTxAndRecProcessing.Update(tDiff)
+	traceMsg += fmt.Sprintf("tx and receipt processing time: %s\r\n", tDiff.String())
+	t = time.Now()
+
+	blockTx.BlockNumber = height
+	blockTx.headerID = headerID
+	return blockTx, err
+}
+
+// processHeader publishes and indexes a header IPLD in Postgres
+// it returns the headerID
+func (sdi *StateDiffIndexer) processHeader(tx *sqlx.Tx, header *types.Header, headerNode node.Node, reward, td *big.Int) (int64, error) {
+	// publish header
+	if err := shared.PublishIPLD(tx, headerNode); err != nil {
+		return 0, fmt.Errorf("error publishing header IPLD: %v", err)
+	}
+	// index header
+	return sdi.dbWriter.upsertHeaderCID(tx, models.HeaderModel{
+		CID:             headerNode.Cid().String(),
+		MhKey:           shared.MultihashKeyFromCID(headerNode.Cid()),
+		ParentHash:      header.ParentHash.String(),
+		BlockNumber:     header.Number.String(),
+		BlockHash:       header.Hash().String(),
+		TotalDifficulty: td.String(),
+		Reward:          reward.String(),
+		Bloom:           header.Bloom.Bytes(),
+		StateRoot:       header.Root.String(),
+		RctRoot:         header.ReceiptHash.String(),
+		TxRoot:          header.TxHash.String(),
+		UncleRoot:       header.UncleHash.String(),
+		Timestamp:       header.Time,
+	})
+}
+
+func (sdi *StateDiffIndexer) processUncles(tx *sqlx.Tx, headerID int64, blockNumber uint64, uncleNodes []*ipld.EthHeader) error {
+	// publish and index uncles
+	for _, uncleNode := range uncleNodes {
+		if err := shared.PublishIPLD(tx, uncleNode); err != nil {
+			return fmt.Errorf("error publishing uncle IPLD: %v", err)
+		}
+		uncleReward := CalcUncleMinerReward(blockNumber, uncleNode.Number.Uint64())
+		uncle := models.UncleModel{
+			CID:        uncleNode.Cid().String(),
+			MhKey:      shared.MultihashKeyFromCID(uncleNode.Cid()),
+			ParentHash: uncleNode.ParentHash.String(),
+			BlockHash:  uncleNode.Hash().String(),
+			Reward:     uncleReward.String(),
+		}
+		if err := sdi.dbWriter.upsertUncleCID(tx, uncle, headerID); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// processArgs bundles arguments to processReceiptsAndTxs
+type processArgs struct {
+	headerID     int64
+	blockNumber  *big.Int
+	receipts     types.Receipts
+	txs          types.Transactions
+	rctNodes     []*ipld.EthReceipt
+	rctTrieNodes []*ipld.EthRctTrie
+	txNodes      []*ipld.EthTx
+	txTrieNodes  []*ipld.EthTxTrie
+}
+
+// processReceiptsAndTxs publishes and indexes receipt and transaction IPLDs in Postgres
+func (sdi *StateDiffIndexer) processReceiptsAndTxs(tx *sqlx.Tx, args processArgs) error {
+	// Process receipts and txs
+	signer := types.MakeSigner(sdi.chainConfig, args.blockNumber)
+	for i, receipt := range args.receipts {
+		// tx that corresponds with this receipt
+		trx := args.txs[i]
+		from, err := types.Sender(signer, trx)
+		if err != nil {
+			return fmt.Errorf("error deriving tx sender: %v", err)
+		}
+
+		// Publishing
+		// publish trie nodes, these aren't indexed directly
+		if err := shared.PublishIPLD(tx, args.txTrieNodes[i]); err != nil {
+			return fmt.Errorf("error publishing tx trie node IPLD: %v", err)
+		}
+		if err := shared.PublishIPLD(tx, args.rctTrieNodes[i]); err != nil {
+			return fmt.Errorf("error publishing rct trie node IPLD: %v", err)
+		}
+		// publish the txs and receipts
+		txNode, rctNode := args.txNodes[i], args.rctNodes[i]
+		if err := shared.PublishIPLD(tx, txNode); err != nil {
+			return fmt.Errorf("error publishing tx IPLD: %v", err)
+		}
+		if err := shared.PublishIPLD(tx, rctNode); err != nil {
+			return fmt.Errorf("error publishing rct IPLD: %v", err)
+		}
+
+		// Indexing
+		// extract topic and contract data from the receipt for indexing
+		topicSets := make([][]string, 4)
+		mappedContracts := make(map[string]bool) // use map to avoid duplicate addresses
+		for _, log := range receipt.Logs {
+			for i, topic := range log.Topics {
+				topicSets[i] = append(topicSets[i], topic.Hex())
+			}
+			mappedContracts[log.Address.String()] = true
+		}
+		// these are the contracts seen in the logs
+		logContracts := make([]string, 0, len(mappedContracts))
+		for addr := range mappedContracts {
+			logContracts = append(logContracts, addr)
+		}
+		// this is the contract address if this receipt is for a contract creation tx
+		contract := shared.HandleZeroAddr(receipt.ContractAddress)
+		var contractHash string
+		if contract != "" {
+			contractHash = crypto.Keccak256Hash(common.HexToAddress(contract).Bytes()).String()
+		}
+		// index tx first so that the receipt can reference it by FK
+		txModel := models.TxModel{
+			Dst:    shared.HandleZeroAddrPointer(trx.To()),
+			Src:    shared.HandleZeroAddr(from),
+			TxHash: trx.Hash().String(),
+			Index:  int64(i),
+			Data:   trx.Data(),
+			CID:    txNode.Cid().String(),
+			MhKey:  shared.MultihashKeyFromCID(txNode.Cid()),
+		}
+		txType := trx.Type()
+		if txType != types.LegacyTxType {
+			txModel.Type = &txType
+		}
+		txID, err := sdi.dbWriter.upsertTransactionCID(tx, txModel, args.headerID)
+		if err != nil {
+			return err
+		}
+
+		// AccessListEntryModel is the db model for eth.access_list_entry
+		type AccessListElementModel struct {
+			ID          int64          `db:"id"`
+			Index       int64          `db:"index"`
+			TxID        int64          `db:"tx_id"`
+			Address     string         `db:"address"`
+			StorageKeys pq.StringArray `db:"storage_keys"`
+		}
+		// index access list if this is one
+		for j, accessListElement := range trx.AccessList() {
+			storageKeys := make([]string, len(accessListElement.StorageKeys))
+			for k, storageKey := range accessListElement.StorageKeys {
+				storageKeys[k] = storageKey.Hex()
+			}
+			accessListElementModel := models.AccessListElementModel{
+				Index:       int64(j),
+				Address:     accessListElement.Address.Hex(),
+				StorageKeys: storageKeys,
+			}
+			if err := sdi.dbWriter.upsertAccessListElement(tx, accessListElementModel, txID); err != nil {
+				return err
+			}
+		}
+		// index the receipt
+		rctModel := models.ReceiptModel{
+			Topic0s:      topicSets[0],
+			Topic1s:      topicSets[1],
+			Topic2s:      topicSets[2],
+			Topic3s:      topicSets[3],
+			Contract:     contract,
+			ContractHash: contractHash,
+			LogContracts: logContracts,
+			CID:          rctNode.Cid().String(),
+			MhKey:        shared.MultihashKeyFromCID(rctNode.Cid()),
+		}
+		if len(receipt.PostState) == 0 {
+			rctModel.PostStatus = receipt.Status
+		} else {
+			rctModel.PostState = common.Bytes2Hex(receipt.PostState)
+		}
+		if err := sdi.dbWriter.upsertReceiptCID(tx, rctModel, txID); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (sdi *StateDiffIndexer) PushStateNode(tx *BlockTx, stateNode sdtypes.StateNode) error {
+	// publish the state node
+	stateCIDStr, err := shared.PublishRaw(tx.dbtx, ipld.MEthStateTrie, multihash.KECCAK_256, stateNode.NodeValue)
+	if err != nil {
+		return fmt.Errorf("error publishing state node IPLD: %v", err)
+	}
+	mhKey, _ := shared.MultihashKeyFromCIDString(stateCIDStr)
+	stateModel := models.StateNodeModel{
+		Path:     stateNode.Path,
+		StateKey: common.BytesToHash(stateNode.LeafKey).String(),
+		CID:      stateCIDStr,
+		MhKey:    mhKey,
+		NodeType: ResolveFromNodeType(stateNode.NodeType),
+	}
+	// index the state node, collect the stateID to reference by FK
+	stateID, err := sdi.dbWriter.upsertStateCID(tx.dbtx, stateModel, tx.headerID)
+	if err != nil {
+		return err
+	}
+	// if we have a leaf, decode and index the account data
+	if stateNode.NodeType == sdtypes.Leaf {
+		var i []interface{}
+		if err := rlp.DecodeBytes(stateNode.NodeValue, &i); err != nil {
+			return fmt.Errorf("error decoding state leaf node rlp: %s", err.Error())
+		}
+		if len(i) != 2 {
+			return fmt.Errorf("eth IPLDPublisher expected state leaf node rlp to decode into two elements")
+		}
+		var account state.Account
+		if err := rlp.DecodeBytes(i[1].([]byte), &account); err != nil {
+			return fmt.Errorf("error decoding state account rlp: %s", err.Error())
+		}
+		accountModel := models.StateAccountModel{
+			Balance:     account.Balance.String(),
+			Nonce:       account.Nonce,
+			CodeHash:    account.CodeHash,
+			StorageRoot: account.Root.String(),
+		}
+		if err := sdi.dbWriter.upsertStateAccount(tx.dbtx, accountModel, stateID); err != nil {
+			return err
+		}
+	}
+	// if there are any storage nodes associated with this node, publish and index them
+	for _, storageNode := range stateNode.StorageNodes {
+		storageCIDStr, err := shared.PublishRaw(tx.dbtx, ipld.MEthStorageTrie, multihash.KECCAK_256, storageNode.NodeValue)
+		if err != nil {
+			return fmt.Errorf("error publishing storage node IPLD: %v", err)
+		}
+		mhKey, _ := shared.MultihashKeyFromCIDString(storageCIDStr)
+		storageModel := models.StorageNodeModel{
+			Path:       storageNode.Path,
+			StorageKey: common.BytesToHash(storageNode.LeafKey).String(),
+			CID:        storageCIDStr,
+			MhKey:      mhKey,
+			NodeType:   ResolveFromNodeType(storageNode.NodeType),
+		}
+		if err := sdi.dbWriter.upsertStorageCID(tx.dbtx, storageModel, stateID); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// Publishes code and codehash pairs to the ipld database
+func (sdi *StateDiffIndexer) PushCodeAndCodeHash(tx *BlockTx, codeAndCodeHash sdtypes.CodeAndCodeHash) error {
+	// codec doesn't matter since db key is multihash-based
+	mhKey, err := shared.MultihashKeyFromKeccak256(codeAndCodeHash.Hash)
+	if err != nil {
+		return fmt.Errorf("error deriving multihash key from codehash: %v", err)
+	}
+	if err := shared.PublishDirect(tx.dbtx, mhKey, codeAndCodeHash.Code); err != nil {
+		return fmt.Errorf("error publishing code IPLD: %v", err)
+	}
+	return nil
+}

statediff/indexer/indexer_test.go

@@ -0,0 +1,450 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package indexer_test
+
+import (
+	"bytes"
+	"testing"
+
+	"github.com/ethereum/go-ethereum/core/types"
+
+	"github.com/ethereum/go-ethereum/common"
+	"github.com/ethereum/go-ethereum/params"
+	"github.com/ethereum/go-ethereum/statediff/indexer"
+	"github.com/ethereum/go-ethereum/statediff/indexer/ipfs/ipld"
+	"github.com/ethereum/go-ethereum/statediff/indexer/mocks"
+	"github.com/ethereum/go-ethereum/statediff/indexer/models"
+	"github.com/ethereum/go-ethereum/statediff/indexer/postgres"
+	"github.com/ethereum/go-ethereum/statediff/indexer/shared"
+
+	"github.com/ipfs/go-cid"
+	blockstore "github.com/ipfs/go-ipfs-blockstore"
+	dshelp "github.com/ipfs/go-ipfs-ds-help"
+	"github.com/multiformats/go-multihash"
+)
+
+var (
+	db        *postgres.DB
+	err       error
+	ind       *indexer.StateDiffIndexer
+	ipfsPgGet = `SELECT data FROM public.blocks
+					WHERE key = $1`
+	tx1, tx2, tx3, tx4, rct1, rct2, rct3, rct4    []byte
+	mockBlock                                     *types.Block
+	headerCID, trx1CID, trx2CID, trx3CID, trx4CID cid.Cid
+	rct1CID, rct2CID, rct3CID, rct4CID            cid.Cid
+	state1CID, state2CID, storageCID              cid.Cid
+)
+
+func expectTrue(t *testing.T, value bool) {
+	if !value {
+		t.Fatalf("Assertion failed")
+	}
+}
+
+func init() {
+	mockBlock = mocks.MockBlock
+	txs, rcts := mocks.MockBlock.Transactions(), mocks.MockReceipts
+
+	buf := new(bytes.Buffer)
+	txs.EncodeIndex(0, buf)
+	tx1 = make([]byte, buf.Len())
+	copy(tx1, buf.Bytes())
+	buf.Reset()
+
+	txs.EncodeIndex(1, buf)
+	tx2 = make([]byte, buf.Len())
+	copy(tx2, buf.Bytes())
+	buf.Reset()
+
+	txs.EncodeIndex(2, buf)
+	tx3 = make([]byte, buf.Len())
+	copy(tx3, buf.Bytes())
+	buf.Reset()
+
+	txs.EncodeIndex(3, buf)
+	tx4 = make([]byte, buf.Len())
+	copy(tx4, buf.Bytes())
+	buf.Reset()
+
+	rcts.EncodeIndex(0, buf)
+	rct1 = make([]byte, buf.Len())
+	copy(rct1, buf.Bytes())
+	buf.Reset()
+
+	rcts.EncodeIndex(1, buf)
+	rct2 = make([]byte, buf.Len())
+	copy(rct2, buf.Bytes())
+	buf.Reset()
+
+	rcts.EncodeIndex(2, buf)
+	rct3 = make([]byte, buf.Len())
+	copy(rct3, buf.Bytes())
+	buf.Reset()
+
+	rcts.EncodeIndex(3, buf)
+	rct4 = make([]byte, buf.Len())
+	copy(rct4, buf.Bytes())
+	buf.Reset()
+
+	headerCID, _ = ipld.RawdataToCid(ipld.MEthHeader, mocks.MockHeaderRlp, multihash.KECCAK_256)
+	trx1CID, _ = ipld.RawdataToCid(ipld.MEthTx, tx1, multihash.KECCAK_256)
+	trx2CID, _ = ipld.RawdataToCid(ipld.MEthTx, tx2, multihash.KECCAK_256)
+	trx3CID, _ = ipld.RawdataToCid(ipld.MEthTx, tx3, multihash.KECCAK_256)
+	trx4CID, _ = ipld.RawdataToCid(ipld.MEthTx, tx4, multihash.KECCAK_256)
+	rct1CID, _ = ipld.RawdataToCid(ipld.MEthTxReceipt, rct1, multihash.KECCAK_256)
+	rct2CID, _ = ipld.RawdataToCid(ipld.MEthTxReceipt, rct2, multihash.KECCAK_256)
+	rct3CID, _ = ipld.RawdataToCid(ipld.MEthTxReceipt, rct3, multihash.KECCAK_256)
+	rct4CID, _ = ipld.RawdataToCid(ipld.MEthTxReceipt, rct4, multihash.KECCAK_256)
+	state1CID, _ = ipld.RawdataToCid(ipld.MEthStateTrie, mocks.ContractLeafNode, multihash.KECCAK_256)
+	state2CID, _ = ipld.RawdataToCid(ipld.MEthStateTrie, mocks.AccountLeafNode, multihash.KECCAK_256)
+	storageCID, _ = ipld.RawdataToCid(ipld.MEthStorageTrie, mocks.StorageLeafNode, multihash.KECCAK_256)
+}
+
+func setup(t *testing.T) {
+	db, err = shared.SetupDB()
+	if err != nil {
+		t.Fatal(err)
+	}
+	ind = indexer.NewStateDiffIndexer(params.MainnetChainConfig, db)
+	var tx *indexer.BlockTx
+	tx, err = ind.PushBlock(
+		mockBlock,
+		mocks.MockReceipts,
+		mocks.MockBlock.Difficulty())
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer tx.Close(err)
+	for _, node := range mocks.StateDiffs {
+		err = ind.PushStateNode(tx, node)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	shared.ExpectEqual(t, tx.BlockNumber, mocks.BlockNumber.Uint64())
+}
+
+func tearDown(t *testing.T) {
+	indexer.TearDownDB(t, db)
+}
+
+func TestPublishAndIndexer(t *testing.T) {
+	t.Run("Publish and index header IPLDs in a single tx", func(t *testing.T) {
+		setup(t)
+		defer tearDown(t)
+		pgStr := `SELECT cid, td, reward, id
+				FROM eth.header_cids
+				WHERE block_number = $1`
+		// check header was properly indexed
+		type res struct {
+			CID    string
+			TD     string
+			Reward string
+			ID     int
+		}
+		header := new(res)
+		err = db.QueryRowx(pgStr, mocks.BlockNumber.Uint64()).StructScan(header)
+		if err != nil {
+			t.Fatal(err)
+		}
+		shared.ExpectEqual(t, header.CID, headerCID.String())
+		shared.ExpectEqual(t, header.TD, mocks.MockBlock.Difficulty().String())
+		shared.ExpectEqual(t, header.Reward, "2000000000000021250")
+		dc, err := cid.Decode(header.CID)
+		if err != nil {
+			t.Fatal(err)
+		}
+		mhKey := dshelp.MultihashToDsKey(dc.Hash())
+		prefixedKey := blockstore.BlockPrefix.String() + mhKey.String()
+		var data []byte
+		err = db.Get(&data, ipfsPgGet, prefixedKey)
+		if err != nil {
+			t.Fatal(err)
+		}
+		shared.ExpectEqual(t, data, mocks.MockHeaderRlp)
+	})
+
+	t.Run("Publish and index transaction IPLDs in a single tx", func(t *testing.T) {
+		setup(t)
+		defer tearDown(t)
+		// check that txs were properly indexed
+		trxs := make([]string, 0)
+		pgStr := `SELECT transaction_cids.cid FROM eth.transaction_cids INNER JOIN eth.header_cids ON (transaction_cids.header_id = header_cids.id)
+				WHERE header_cids.block_number = $1`
+		err = db.Select(&trxs, pgStr, mocks.BlockNumber.Uint64())
+		if err != nil {
+			t.Fatal(err)
+		}
+		shared.ExpectEqual(t, len(trxs), 4)
+		expectTrue(t, shared.ListContainsString(trxs, trx1CID.String()))
+		expectTrue(t, shared.ListContainsString(trxs, trx2CID.String()))
+		expectTrue(t, shared.ListContainsString(trxs, trx3CID.String()))
+		expectTrue(t, shared.ListContainsString(trxs, trx4CID.String()))
+		// and published
+		for _, c := range trxs {
+			dc, err := cid.Decode(c)
+			if err != nil {
+				t.Fatal(err)
+			}
+			mhKey := dshelp.MultihashToDsKey(dc.Hash())
+			prefixedKey := blockstore.BlockPrefix.String() + mhKey.String()
+			var data []byte
+			err = db.Get(&data, ipfsPgGet, prefixedKey)
+			if err != nil {
+				t.Fatal(err)
+			}
+			switch c {
+			case trx1CID.String():
+				shared.ExpectEqual(t, data, tx1)
+				var txType *uint8
+				pgStr = `SELECT tx_type FROM eth.transaction_cids WHERE cid = $1`
+				err = db.Get(&txType, pgStr, c)
+				if err != nil {
+					t.Fatal(err)
+				}
+				if txType != nil {
+					t.Fatalf("expected nil tx_type, got %d", *txType)
+				}
+			case trx2CID.String():
+				shared.ExpectEqual(t, data, tx2)
+				var txType *uint8
+				pgStr = `SELECT tx_type FROM eth.transaction_cids WHERE cid = $1`
+				err = db.Get(&txType, pgStr, c)
+				if err != nil {
+					t.Fatal(err)
+				}
+				if txType != nil {
+					t.Fatalf("expected nil tx_type, got %d", *txType)
+				}
+			case trx3CID.String():
+				shared.ExpectEqual(t, data, tx3)
+				var txType *uint8
+				pgStr = `SELECT tx_type FROM eth.transaction_cids WHERE cid = $1`
+				err = db.Get(&txType, pgStr, c)
+				if err != nil {
+					t.Fatal(err)
+				}
+				if txType != nil {
+					t.Fatalf("expected nil tx_type, got %d", *txType)
+				}
+			case trx4CID.String():
+				shared.ExpectEqual(t, data, tx4)
+				var txType *uint8
+				pgStr = `SELECT tx_type FROM eth.transaction_cids WHERE cid = $1`
+				err = db.Get(&txType, pgStr, c)
+				if err != nil {
+					t.Fatal(err)
+				}
+				if *txType != types.AccessListTxType {
+					t.Fatalf("expected AccessListTxType (1), got %d", *txType)
+				}
+				accessListElementModels := make([]models.AccessListElementModel, 0)
+				pgStr = `SELECT access_list_element.* FROM eth.access_list_element INNER JOIN eth.transaction_cids ON (tx_id = transaction_cids.id) WHERE cid = $1 ORDER BY access_list_element.index ASC`
+				err = db.Select(&accessListElementModels, pgStr, c)
+				if err != nil {
+					t.Fatal(err)
+				}
+				if len(accessListElementModels) != 2 {
+					t.Fatalf("expected two access list entries, got %d", len(accessListElementModels))
+				}
+				model1 := models.AccessListElementModel{
+					Index:   accessListElementModels[0].Index,
+					Address: accessListElementModels[0].Address,
+				}
+				model2 := models.AccessListElementModel{
+					Index:       accessListElementModels[1].Index,
+					Address:     accessListElementModels[1].Address,
+					StorageKeys: accessListElementModels[1].StorageKeys,
+				}
+				shared.ExpectEqual(t, model1, mocks.AccessListEntry1Model)
+				shared.ExpectEqual(t, model2, mocks.AccessListEntry2Model)
+			}
+		}
+	})
+
+	t.Run("Publish and index receipt IPLDs in a single tx", func(t *testing.T) {
+		setup(t)
+		defer tearDown(t)
+		// check receipts were properly indexed
+		rcts := make([]string, 0)
+		pgStr := `SELECT receipt_cids.cid FROM eth.receipt_cids, eth.transaction_cids, eth.header_cids
+				WHERE receipt_cids.tx_id = transaction_cids.id
+				AND transaction_cids.header_id = header_cids.id
+				AND header_cids.block_number = $1`
+		err = db.Select(&rcts, pgStr, mocks.BlockNumber.Uint64())
+		if err != nil {
+			t.Fatal(err)
+		}
+		shared.ExpectEqual(t, len(rcts), 4)
+		expectTrue(t, shared.ListContainsString(rcts, rct1CID.String()))
+		expectTrue(t, shared.ListContainsString(rcts, rct2CID.String()))
+		expectTrue(t, shared.ListContainsString(rcts, rct3CID.String()))
+		expectTrue(t, shared.ListContainsString(rcts, rct4CID.String()))
+		// and published
+		for _, c := range rcts {
+			dc, err := cid.Decode(c)
+			if err != nil {
+				t.Fatal(err)
+			}
+			mhKey := dshelp.MultihashToDsKey(dc.Hash())
+			prefixedKey := blockstore.BlockPrefix.String() + mhKey.String()
+			var data []byte
+			err = db.Get(&data, ipfsPgGet, prefixedKey)
+			if err != nil {
+				t.Fatal(err)
+			}
+			switch c {
+			case rct1CID.String():
+				shared.ExpectEqual(t, data, rct1)
+				var postStatus uint64
+				pgStr = `SELECT post_status FROM eth.receipt_cids WHERE cid = $1`
+				err = db.Get(&postStatus, pgStr, c)
+				if err != nil {
+					t.Fatal(err)
+				}
+				shared.ExpectEqual(t, postStatus, mocks.ExpectedPostStatus)
+			case rct2CID.String():
+				shared.ExpectEqual(t, data, rct2)
+				var postState string
+				pgStr = `SELECT post_state FROM eth.receipt_cids WHERE cid = $1`
+				err = db.Get(&postState, pgStr, c)
+				if err != nil {
+					t.Fatal(err)
+				}
+				shared.ExpectEqual(t, postState, mocks.ExpectedPostState1)
+			case rct3CID.String():
+				shared.ExpectEqual(t, data, rct3)
+				var postState string
+				pgStr = `SELECT post_state FROM eth.receipt_cids WHERE cid = $1`
+				err = db.Get(&postState, pgStr, c)
+				if err != nil {
+					t.Fatal(err)
+				}
+				shared.ExpectEqual(t, postState, mocks.ExpectedPostState2)
+			case rct4CID.String():
+				shared.ExpectEqual(t, data, rct4)
+				var postState string
+				pgStr = `SELECT post_state FROM eth.receipt_cids WHERE cid = $1`
+				err = db.Get(&postState, pgStr, c)
+				if err != nil {
+					t.Fatal(err)
+				}
+				shared.ExpectEqual(t, postState, mocks.ExpectedPostState3)
+			}
+		}
+	})
+
+	t.Run("Publish and index state IPLDs in a single tx", func(t *testing.T) {
+		setup(t)
+		defer tearDown(t)
+		// check that state nodes were properly indexed and published
+		stateNodes := make([]models.StateNodeModel, 0)
+		pgStr := `SELECT state_cids.id, state_cids.cid, state_cids.state_leaf_key, state_cids.node_type, state_cids.state_path, state_cids.header_id
+				FROM eth.state_cids INNER JOIN eth.header_cids ON (state_cids.header_id = header_cids.id)
+				WHERE header_cids.block_number = $1`
+		err = db.Select(&stateNodes, pgStr, mocks.BlockNumber.Uint64())
+		if err != nil {
+			t.Fatal(err)
+		}
+		shared.ExpectEqual(t, len(stateNodes), 2)
+		for _, stateNode := range stateNodes {
+			var data []byte
+			dc, err := cid.Decode(stateNode.CID)
+			if err != nil {
+				t.Fatal(err)
+			}
+			mhKey := dshelp.MultihashToDsKey(dc.Hash())
+			prefixedKey := blockstore.BlockPrefix.String() + mhKey.String()
+			err = db.Get(&data, ipfsPgGet, prefixedKey)
+			if err != nil {
+				t.Fatal(err)
+			}
+			pgStr = `SELECT * from eth.state_accounts WHERE state_id = $1`
+			var account models.StateAccountModel
+			err = db.Get(&account, pgStr, stateNode.ID)
+			if err != nil {
+				t.Fatal(err)
+			}
+			if stateNode.CID == state1CID.String() {
+				shared.ExpectEqual(t, stateNode.NodeType, 2)
+				shared.ExpectEqual(t, stateNode.StateKey, common.BytesToHash(mocks.ContractLeafKey).Hex())
+				shared.ExpectEqual(t, stateNode.Path, []byte{'\x06'})
+				shared.ExpectEqual(t, data, mocks.ContractLeafNode)
+				shared.ExpectEqual(t, account, models.StateAccountModel{
+					ID:          account.ID,
+					StateID:     stateNode.ID,
+					Balance:     "0",
+					CodeHash:    mocks.ContractCodeHash.Bytes(),
+					StorageRoot: mocks.ContractRoot,
+					Nonce:       1,
+				})
+			}
+			if stateNode.CID == state2CID.String() {
+				shared.ExpectEqual(t, stateNode.NodeType, 2)
+				shared.ExpectEqual(t, stateNode.StateKey, common.BytesToHash(mocks.AccountLeafKey).Hex())
+				shared.ExpectEqual(t, stateNode.Path, []byte{'\x0c'})
+				shared.ExpectEqual(t, data, mocks.AccountLeafNode)
+				shared.ExpectEqual(t, account, models.StateAccountModel{
+					ID:          account.ID,
+					StateID:     stateNode.ID,
+					Balance:     "1000",
+					CodeHash:    mocks.AccountCodeHash.Bytes(),
+					StorageRoot: mocks.AccountRoot,
+					Nonce:       0,
+				})
+			}
+		}
+	})
+
+	t.Run("Publish and index storage IPLDs in a single tx", func(t *testing.T) {
+		setup(t)
+		defer tearDown(t)
+		// check that storage nodes were properly indexed
+		storageNodes := make([]models.StorageNodeWithStateKeyModel, 0)
+		pgStr := `SELECT storage_cids.cid, state_cids.state_leaf_key, storage_cids.storage_leaf_key, storage_cids.node_type, storage_cids.storage_path
+				FROM eth.storage_cids, eth.state_cids, eth.header_cids
+				WHERE storage_cids.state_id = state_cids.id
+				AND state_cids.header_id = header_cids.id
+				AND header_cids.block_number = $1`
+		err = db.Select(&storageNodes, pgStr, mocks.BlockNumber.Uint64())
+		if err != nil {
+			t.Fatal(err)
+		}
+		shared.ExpectEqual(t, len(storageNodes), 1)
+		shared.ExpectEqual(t, storageNodes[0], models.StorageNodeWithStateKeyModel{
+			CID:        storageCID.String(),
+			NodeType:   2,
+			StorageKey: common.BytesToHash(mocks.StorageLeafKey).Hex(),
+			StateKey:   common.BytesToHash(mocks.ContractLeafKey).Hex(),
+			Path:       []byte{},
+		})
+		var data []byte
+		dc, err := cid.Decode(storageNodes[0].CID)
+		if err != nil {
+			t.Fatal(err)
+		}
+		mhKey := dshelp.MultihashToDsKey(dc.Hash())
+		prefixedKey := blockstore.BlockPrefix.String() + mhKey.String()
+		err = db.Get(&data, ipfsPgGet, prefixedKey)
+		if err != nil {
+			t.Fatal(err)
+		}
+		shared.ExpectEqual(t, data, mocks.StorageLeafNode)
+	})
+}

statediff/indexer/ipfs/ipld/eth_account.go

@@ -0,0 +1,175 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"encoding/json"
+	"fmt"
+	"math/big"
+
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+)
+
+// EthAccountSnapshot (eth-account-snapshot codec 0x97)
+// represents an ethereum account, i.e. a wallet address or
+// a smart contract
+type EthAccountSnapshot struct {
+	*EthAccount
+
+	cid     cid.Cid
+	rawdata []byte
+}
+
+// EthAccount is the building block of EthAccountSnapshot.
+// Or, is the former stripped of its cid and rawdata components.
+type EthAccount struct {
+	Nonce    uint64
+	Balance  *big.Int
+	Root     []byte // This is the storage root trie
+	CodeHash []byte // This is the hash of the EVM code
+}
+
+// Static (compile time) check that EthAccountSnapshot satisfies the
+// node.Node interface.
+var _ node.Node = (*EthAccountSnapshot)(nil)
+
+/*
+  INPUT
+*/
+
+// Input should be managed by EthStateTrie
+
+/*
+   OUTPUT
+*/
+
+// Output should be managed by EthStateTrie
+
+/*
+   Block INTERFACE
+*/
+
+// RawData returns the binary of the RLP encode of the account snapshot.
+func (as *EthAccountSnapshot) RawData() []byte {
+	return as.rawdata
+}
+
+// Cid returns the cid of the transaction.
+func (as *EthAccountSnapshot) Cid() cid.Cid {
+	return as.cid
+}
+
+// String is a helper for output
+func (as *EthAccountSnapshot) String() string {
+	return fmt.Sprintf("<EthereumAccountSnapshot %s>", as.cid)
+}
+
+// Loggable returns in a map the type of IPLD Link.
+func (as *EthAccountSnapshot) Loggable() map[string]interface{} {
+	return map[string]interface{}{
+		"type": "eth-account-snapshot",
+	}
+}
+
+/*
+   Node INTERFACE
+*/
+
+// Resolve resolves a path through this node, stopping at any link boundary
+// and returning the object found as well as the remaining path to traverse
+func (as *EthAccountSnapshot) Resolve(p []string) (interface{}, []string, error) {
+	if len(p) == 0 {
+		return as, nil, nil
+	}
+
+	if len(p) > 1 {
+		return nil, nil, fmt.Errorf("unexpected path elements past %s", p[0])
+	}
+
+	switch p[0] {
+	case "balance":
+		return as.Balance, nil, nil
+	case "codeHash":
+		return &node.Link{Cid: keccak256ToCid(RawBinary, as.CodeHash)}, nil, nil
+	case "nonce":
+		return as.Nonce, nil, nil
+	case "root":
+		return &node.Link{Cid: keccak256ToCid(MEthStorageTrie, as.Root)}, nil, nil
+	default:
+		return nil, nil, fmt.Errorf("no such link")
+	}
+}
+
+// Tree lists all paths within the object under 'path', and up to the given depth.
+// To list the entire object (similar to `find .`) pass "" and -1
+func (as *EthAccountSnapshot) Tree(p string, depth int) []string {
+	if p != "" || depth == 0 {
+		return nil
+	}
+	return []string{"balance", "codeHash", "nonce", "root"}
+}
+
+// ResolveLink is a helper function that calls resolve and asserts the
+// output is a link
+func (as *EthAccountSnapshot) ResolveLink(p []string) (*node.Link, []string, error) {
+	obj, rest, err := as.Resolve(p)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if lnk, ok := obj.(*node.Link); ok {
+		return lnk, rest, nil
+	}
+
+	return nil, nil, fmt.Errorf("resolved item was not a link")
+}
+
+// Copy will go away. It is here to comply with the interface.
+func (as *EthAccountSnapshot) Copy() node.Node {
+	panic("implement me")
+}
+
+// Links is a helper function that returns all links within this object
+func (as *EthAccountSnapshot) Links() []*node.Link {
+	return nil
+}
+
+// Stat will go away. It is here to comply with the interface.
+func (as *EthAccountSnapshot) Stat() (*node.NodeStat, error) {
+	return &node.NodeStat{}, nil
+}
+
+// Size will go away. It is here to comply with the interface.
+func (as *EthAccountSnapshot) Size() (uint64, error) {
+	return 0, nil
+}
+
+/*
+  EthAccountSnapshot functions
+*/
+
+// MarshalJSON processes the transaction into readable JSON format.
+func (as *EthAccountSnapshot) MarshalJSON() ([]byte, error) {
+	out := map[string]interface{}{
+		"balance":  as.Balance,
+		"codeHash": keccak256ToCid(RawBinary, as.CodeHash),
+		"nonce":    as.Nonce,
+		"root":     keccak256ToCid(MEthStorageTrie, as.Root),
+	}
+	return json.Marshal(out)
+}

statediff/indexer/ipfs/ipld/eth_account_test.go

@@ -0,0 +1,292 @@
+package ipld
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"regexp"
+	"testing"
+)
+
+/*
+  Block INTERFACE
+*/
+
+func TestAccountSnapshotBlockElements(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	if fmt.Sprintf("%x", eas.RawData())[:10] != "f84e808a03" {
+		t.Fatal("Wrong Data")
+	}
+
+	if eas.Cid().String() !=
+		"baglqcgzasckx2alxk43cksshnztjvhfyvbbh6bkp376gtcndm5cg4fkrkhsa" {
+		t.Fatal("Wrong Cid")
+	}
+}
+
+func TestAccountSnapshotString(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	if eas.String() !=
+		"<EthereumAccountSnapshot baglqcgzasckx2alxk43cksshnztjvhfyvbbh6bkp376gtcndm5cg4fkrkhsa>" {
+		t.Fatalf("Wrong String()")
+	}
+}
+
+func TestAccountSnapshotLoggable(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	l := eas.Loggable()
+	if _, ok := l["type"]; !ok {
+		t.Fatal("Loggable map expected the field 'type'")
+	}
+
+	if l["type"] != "eth-account-snapshot" {
+		t.Fatalf("Wrong Loggable 'type' value\r\nexpected %s\r\ngot %s", "eth-account-snapshot", l["type"])
+	}
+}
+
+/*
+  Node INTERFACE
+*/
+func TestAccountSnapshotResolve(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	// Empty path
+	obj, rest, err := eas.Resolve([]string{})
+	reas, ok := obj.(*EthAccountSnapshot)
+	if !ok {
+		t.Fatalf("Wrong type of returned object\r\nexpected %T\r\ngot %T", &EthAccountSnapshot{}, reas)
+	}
+	if reas.Cid() != eas.Cid() {
+		t.Fatalf("wrong returned CID\r\nexpected %s\r\ngot %s", eas.Cid().String(), reas.Cid().String())
+	}
+	if rest != nil {
+		t.Fatal("rest should be nil")
+	}
+	if err != nil {
+		t.Fatal("err should be nil")
+	}
+
+	// len(p) > 1
+	badCases := [][]string{
+		{"two", "elements"},
+		{"here", "three", "elements"},
+		{"and", "here", "four", "elements"},
+	}
+
+	for _, bc := range badCases {
+		obj, rest, err = eas.Resolve(bc)
+		if obj != nil {
+			t.Fatal("obj should be nil")
+		}
+		if rest != nil {
+			t.Fatal("rest should be nil")
+		}
+		if err.Error() != fmt.Sprintf("unexpected path elements past %s", bc[0]) {
+			t.Fatal("wrong error")
+		}
+	}
+
+	moreBadCases := []string{
+		"i",
+		"am",
+		"not",
+		"an",
+		"account",
+		"field",
+	}
+	for _, mbc := range moreBadCases {
+		obj, rest, err = eas.Resolve([]string{mbc})
+		if obj != nil {
+			t.Fatal("obj should be nil")
+		}
+		if rest != nil {
+			t.Fatal("rest should be nil")
+		}
+		if err.Error() != fmt.Sprintf("no such link") {
+			t.Fatal("wrong error")
+		}
+	}
+
+	goodCases := []string{
+		"balance",
+		"codeHash",
+		"nonce",
+		"root",
+	}
+	for _, gc := range goodCases {
+		_, _, err = eas.Resolve([]string{gc})
+		if err != nil {
+			t.Fatalf("error should be nil %v", gc)
+		}
+	}
+
+}
+
+func TestAccountSnapshotTree(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	// Bad cases
+	tree := eas.Tree("non-empty-string", 0)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	tree = eas.Tree("non-empty-string", 1)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	tree = eas.Tree("", 0)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	// Good cases
+	tree = eas.Tree("", 1)
+	lookupElements := map[string]interface{}{
+		"balance":  nil,
+		"codeHash": nil,
+		"nonce":    nil,
+		"root":     nil,
+	}
+
+	if len(tree) != len(lookupElements) {
+		t.Fatalf("Wrong number of elements\r\nexpected %d\r\ngot %d", len(lookupElements), len(tree))
+	}
+
+	for _, te := range tree {
+		if _, ok := lookupElements[te]; !ok {
+			t.Fatalf("Unexpected Element: %v", te)
+		}
+	}
+}
+
+func TestAccountSnapshotResolveLink(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	// bad case
+	obj, rest, err := eas.ResolveLink([]string{"supercalifragilist"})
+	if obj != nil {
+		t.Fatalf("Expected obj to be nil")
+	}
+	if rest != nil {
+		t.Fatal("Expected rest to be nil")
+	}
+	if err.Error() != "no such link" {
+		t.Fatal("Wrong error")
+	}
+
+	// good case
+	obj, rest, err = eas.ResolveLink([]string{"nonce"})
+	if obj != nil {
+		t.Fatalf("Expected obj to be nil")
+	}
+	if rest != nil {
+		t.Fatal("Expected rest to be nil")
+	}
+	if err.Error() != "resolved item was not a link" {
+		t.Fatal("Wrong error")
+	}
+}
+
+func TestAccountSnapshotCopy(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	defer func() {
+		r := recover()
+		if r == nil {
+			t.Fatal("Expected panic")
+		}
+		if r != "implement me" {
+			t.Fatalf("Wrong panic message\r\n expected %s\r\ngot %s", "'implement me'", r)
+		}
+	}()
+
+	_ = eas.Copy()
+}
+
+func TestAccountSnapshotLinks(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	if eas.Links() != nil {
+		t.Fatal("Links() expected to return nil")
+	}
+}
+
+func TestAccountSnapshotStat(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	obj, err := eas.Stat()
+	if obj == nil {
+		t.Fatal("Expected a not null object node.NodeStat")
+	}
+
+	if err != nil {
+		t.Fatal("Expected a nil error")
+	}
+}
+
+func TestAccountSnapshotSize(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	size, err := eas.Size()
+	if size != uint64(0) {
+		t.Fatalf("Wrong size\r\nexpected %d\r\ngot %d", 0, size)
+	}
+
+	if err != nil {
+		t.Fatal("Expected a nil error")
+	}
+}
+
+/*
+  EthAccountSnapshot functions
+*/
+
+func TestAccountSnapshotMarshalJSON(t *testing.T) {
+	eas := prepareEthAccountSnapshot(t)
+
+	jsonOutput, err := eas.MarshalJSON()
+	checkError(err, t)
+
+	var data map[string]interface{}
+	err = json.Unmarshal(jsonOutput, &data)
+	checkError(err, t)
+
+	balanceExpression := regexp.MustCompile(`{"balance":16011846000000000000000,`)
+	if !balanceExpression.MatchString(string(jsonOutput)) {
+		t.Fatal("Balance expression not found")
+	}
+
+	code, _ := data["codeHash"].(map[string]interface{})
+	if fmt.Sprintf("%s", code["/"]) !=
+		"bafkrwigf2jdadbxxem6je7t5wlomoa6a4ualmu6kqittw6723acf3bneoa" {
+		t.Fatalf("Wrong Marshaled Value\r\nexpected %s\r\ngot %s", "bafkrwigf2jdadbxxem6je7t5wlomoa6a4ualmu6kqittw6723acf3bneoa", fmt.Sprintf("%s", code["/"]))
+	}
+
+	if fmt.Sprintf("%v", data["nonce"]) != "0" {
+		t.Fatalf("Wrong Marshaled Value\r\nexpected %s\r\ngot %s", "0", fmt.Sprintf("%v", data["nonce"]))
+	}
+
+	root, _ := data["root"].(map[string]interface{})
+	if fmt.Sprintf("%s", root["/"]) !=
+		"bagmacgzak3ub6fy3zrk2n74dixtjfqhynznurya3tfwk3qabmix3ly3dwqqq" {
+		t.Fatalf("Wrong Marshaled Value\r\nexpected %s\r\ngot %s", "bagmacgzak3ub6fy3zrk2n74dixtjfqhynznurya3tfwk3qabmix3ly3dwqqq", fmt.Sprintf("%s", root["/"]))
+	}
+}
+
+/*
+  AUXILIARS
+*/
+func prepareEthAccountSnapshot(t *testing.T) *EthAccountSnapshot {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-c9070d")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	return output.elements[1].(*EthAccountSnapshot)
+}

statediff/indexer/ipfs/ipld/eth_header.go

@@ -0,0 +1,293 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/ethereum/go-ethereum/common"
+
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/rlp"
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+	mh "github.com/multiformats/go-multihash"
+)
+
+// EthHeader (eth-block, codec 0x90), represents an ethereum block header
+type EthHeader struct {
+	*types.Header
+
+	cid     cid.Cid
+	rawdata []byte
+}
+
+// Static (compile time) check that EthHeader satisfies the node.Node interface.
+var _ node.Node = (*EthHeader)(nil)
+
+/*
+  INPUT
+*/
+
+// NewEthHeader converts a *types.Header into an EthHeader IPLD node
+func NewEthHeader(header *types.Header) (*EthHeader, error) {
+	headerRLP, err := rlp.EncodeToBytes(header)
+	if err != nil {
+		return nil, err
+	}
+	c, err := RawdataToCid(MEthHeader, headerRLP, mh.KECCAK_256)
+	if err != nil {
+		return nil, err
+	}
+	return &EthHeader{
+		Header:  header,
+		cid:     c,
+		rawdata: headerRLP,
+	}, nil
+}
+
+/*
+ OUTPUT
+*/
+
+// DecodeEthHeader takes a cid and its raw binary data
+// from IPFS and returns an EthTx object for further processing.
+func DecodeEthHeader(c cid.Cid, b []byte) (*EthHeader, error) {
+	h := new(types.Header)
+	if err := rlp.DecodeBytes(b, h); err != nil {
+		return nil, err
+	}
+	return &EthHeader{
+		Header:  h,
+		cid:     c,
+		rawdata: b,
+	}, nil
+}
+
+/*
+  Block INTERFACE
+*/
+
+// RawData returns the binary of the RLP encode of the block header.
+func (b *EthHeader) RawData() []byte {
+	return b.rawdata
+}
+
+// Cid returns the cid of the block header.
+func (b *EthHeader) Cid() cid.Cid {
+	return b.cid
+}
+
+// String is a helper for output
+func (b *EthHeader) String() string {
+	return fmt.Sprintf("<EthHeader %s>", b.cid)
+}
+
+// Loggable returns a map the type of IPLD Link.
+func (b *EthHeader) Loggable() map[string]interface{} {
+	return map[string]interface{}{
+		"type": "eth-header",
+	}
+}
+
+/*
+  Node INTERFACE
+*/
+
+// Resolve resolves a path through this node, stopping at any link boundary
+// and returning the object found as well as the remaining path to traverse
+func (b *EthHeader) Resolve(p []string) (interface{}, []string, error) {
+	if len(p) == 0 {
+		return b, nil, nil
+	}
+
+	first, rest := p[0], p[1:]
+
+	switch first {
+	case "parent":
+		return &node.Link{Cid: commonHashToCid(MEthHeader, b.ParentHash)}, rest, nil
+	case "receipts":
+		return &node.Link{Cid: commonHashToCid(MEthTxReceiptTrie, b.ReceiptHash)}, rest, nil
+	case "root":
+		return &node.Link{Cid: commonHashToCid(MEthStateTrie, b.Root)}, rest, nil
+	case "tx":
+		return &node.Link{Cid: commonHashToCid(MEthTxTrie, b.TxHash)}, rest, nil
+	case "uncles":
+		return &node.Link{Cid: commonHashToCid(MEthHeaderList, b.UncleHash)}, rest, nil
+	}
+
+	if len(p) != 1 {
+		return nil, nil, fmt.Errorf("unexpected path elements past %s", first)
+	}
+
+	switch first {
+	case "bloom":
+		return b.Bloom, nil, nil
+	case "coinbase":
+		return b.Coinbase, nil, nil
+	case "difficulty":
+		return b.Difficulty, nil, nil
+	case "extra":
+		// This is a []byte. By default they are marshalled into Base64.
+		return fmt.Sprintf("0x%x", b.Extra), nil, nil
+	case "gaslimit":
+		return b.GasLimit, nil, nil
+	case "gasused":
+		return b.GasUsed, nil, nil
+	case "mixdigest":
+		return b.MixDigest, nil, nil
+	case "nonce":
+		return b.Nonce, nil, nil
+	case "number":
+		return b.Number, nil, nil
+	case "time":
+		return b.Time, nil, nil
+	default:
+		return nil, nil, fmt.Errorf("no such link")
+	}
+}
+
+// Tree lists all paths within the object under 'path', and up to the given depth.
+// To list the entire object (similar to `find .`) pass "" and -1
+func (b *EthHeader) Tree(p string, depth int) []string {
+	if p != "" || depth == 0 {
+		return nil
+	}
+
+	return []string{
+		"time",
+		"bloom",
+		"coinbase",
+		"difficulty",
+		"extra",
+		"gaslimit",
+		"gasused",
+		"mixdigest",
+		"nonce",
+		"number",
+		"parent",
+		"receipts",
+		"root",
+		"tx",
+		"uncles",
+	}
+}
+
+// ResolveLink is a helper function that allows easier traversal of links through blocks
+func (b *EthHeader) ResolveLink(p []string) (*node.Link, []string, error) {
+	obj, rest, err := b.Resolve(p)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if lnk, ok := obj.(*node.Link); ok {
+		return lnk, rest, nil
+	}
+
+	return nil, nil, fmt.Errorf("resolved item was not a link")
+}
+
+// Copy will go away. It is here to comply with the Node interface.
+func (b *EthHeader) Copy() node.Node {
+	panic("implement me")
+}
+
+// Links is a helper function that returns all links within this object
+// HINT: Use `ipfs refs <cid>`
+func (b *EthHeader) Links() []*node.Link {
+	return []*node.Link{
+		{Cid: commonHashToCid(MEthHeader, b.ParentHash)},
+		{Cid: commonHashToCid(MEthTxReceiptTrie, b.ReceiptHash)},
+		{Cid: commonHashToCid(MEthStateTrie, b.Root)},
+		{Cid: commonHashToCid(MEthTxTrie, b.TxHash)},
+		{Cid: commonHashToCid(MEthHeaderList, b.UncleHash)},
+	}
+}
+
+// Stat will go away. It is here to comply with the Node interface.
+func (b *EthHeader) Stat() (*node.NodeStat, error) {
+	return &node.NodeStat{}, nil
+}
+
+// Size will go away. It is here to comply with the Node interface.
+func (b *EthHeader) Size() (uint64, error) {
+	return 0, nil
+}
+
+/*
+  EthHeader functions
+*/
+
+// MarshalJSON processes the block header into readable JSON format,
+// converting the right links into their cids, and keeping the original
+// hex hash, allowing the user to simplify external queries.
+func (b *EthHeader) MarshalJSON() ([]byte, error) {
+	out := map[string]interface{}{
+		"time":       b.Time,
+		"bloom":      b.Bloom,
+		"coinbase":   b.Coinbase,
+		"difficulty": b.Difficulty,
+		"extra":      fmt.Sprintf("0x%x", b.Extra),
+		"gaslimit":   b.GasLimit,
+		"gasused":    b.GasUsed,
+		"mixdigest":  b.MixDigest,
+		"nonce":      b.Nonce,
+		"number":     b.Number,
+		"parent":     commonHashToCid(MEthHeader, b.ParentHash),
+		"receipts":   commonHashToCid(MEthTxReceiptTrie, b.ReceiptHash),
+		"root":       commonHashToCid(MEthStateTrie, b.Root),
+		"tx":         commonHashToCid(MEthTxTrie, b.TxHash),
+		"uncles":     commonHashToCid(MEthHeaderList, b.UncleHash),
+	}
+	return json.Marshal(out)
+}
+
+// objJSONHeader defines the output of the JSON RPC API for either
+// "eth_BlockByHash" or "eth_BlockByHeader".
+type objJSONHeader struct {
+	Result objJSONHeaderResult `json:"result"`
+}
+
+// objJSONBLockResult is the  nested struct that takes
+// the contents of the JSON field "result".
+type objJSONHeaderResult struct {
+	types.Header            // Use its fields and unmarshaler
+	*objJSONHeaderResultExt // Add these fields to the parsing
+}
+
+// objJSONBLockResultExt facilitates the composition
+// of the field "result", adding to the
+// `types.Header` fields, both ommers (their hashes) and transactions.
+type objJSONHeaderResultExt struct {
+	OmmerHashes  []common.Hash        `json:"uncles"`
+	Transactions []*types.Transaction `json:"transactions"`
+}
+
+// UnmarshalJSON overrides the function types.Header.UnmarshalJSON, allowing us
+// to parse the fields of Header, plus ommer hashes and transactions.
+// (yes, ommer hashes. You will need to "eth_getUncleCountByBlockHash" per each ommer)
+func (o *objJSONHeaderResult) UnmarshalJSON(input []byte) error {
+	err := o.Header.UnmarshalJSON(input)
+	if err != nil {
+		return err
+	}
+
+	o.objJSONHeaderResultExt = &objJSONHeaderResultExt{}
+	err = json.Unmarshal(input, o.objJSONHeaderResultExt)
+	return err
+}

statediff/indexer/ipfs/ipld/eth_header_test.go

@@ -0,0 +1,585 @@
+package ipld
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"runtime"
+	"strconv"
+	"testing"
+
+	block "github.com/ipfs/go-block-format"
+	node "github.com/ipfs/go-ipld-format"
+	"github.com/multiformats/go-multihash"
+
+	"github.com/ethereum/go-ethereum/core/types"
+)
+
+func TestBlockBodyRlpParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-block-body-rlp-999999")
+	checkError(err, t)
+
+	output, _, _, err := FromBlockRLP(fi)
+	checkError(err, t)
+
+	testEthBlockFields(output, t)
+}
+
+func TestBlockHeaderRlpParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-block-header-rlp-999999")
+	checkError(err, t)
+
+	output, _, _, err := FromBlockRLP(fi)
+	checkError(err, t)
+
+	testEthBlockFields(output, t)
+}
+
+func TestBlockBodyJsonParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-block-body-json-999999")
+	checkError(err, t)
+
+	output, _, _, err := FromBlockJSON(fi)
+	checkError(err, t)
+
+	testEthBlockFields(output, t)
+}
+
+func TestEthBlockProcessTransactionsError(t *testing.T) {
+	// Let's just change one byte in a field of one of these transactions.
+	fi, err := os.Open("test_data/error-tx-eth-block-body-json-999999")
+	checkError(err, t)
+
+	_, _, _, err = FromBlockJSON(fi)
+	if err == nil {
+		t.Fatal("Expected an error")
+	}
+}
+
+// TestDecodeBlockHeader should work for both inputs (block header and block body)
+// as what we are storing is just the block header
+func TestDecodeBlockHeader(t *testing.T) {
+	storedEthBlock := prepareStoredEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	ethBlock, err := DecodeEthHeader(storedEthBlock.Cid(), storedEthBlock.RawData())
+	checkError(err, t)
+
+	testEthBlockFields(ethBlock, t)
+}
+
+func TestEthBlockString(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+	if ethBlock.String() != "<EthHeader bagiacgzawt5236hkiuvrhfyy4jya3qitlt6icfcqgheew6vsptlraokppm4a>" {
+		t.Fatalf("Wrong String()\r\nexpected %s\r\ngot %s", "<EthHeader bagiacgzawt5236hkiuvrhfyy4jya3qitlt6icfcqgheew6vsptlraokppm4a>", ethBlock.String())
+	}
+}
+
+func TestEthBlockLoggable(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	l := ethBlock.Loggable()
+	if _, ok := l["type"]; !ok {
+		t.Fatal("Loggable map expected the field 'type'")
+	}
+
+	if l["type"] != "eth-header" {
+		t.Fatalf("Wrong Loggable 'type' value\r\nexpected %s\r\ngot %s", "eth-header", l["type"])
+	}
+}
+
+func TestEthBlockJSONMarshal(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	jsonOutput, err := ethBlock.MarshalJSON()
+	checkError(err, t)
+
+	var data map[string]interface{}
+	err = json.Unmarshal(jsonOutput, &data)
+	checkError(err, t)
+
+	// Testing all fields is boring, but can help us to avoid
+	// that dreaded regression
+	if data["bloom"].(string)[:10] != "0x00000000" {
+		t.Fatalf("Wrong Bloom\r\nexpected %s\r\ngot %s", "0x00000000", data["bloom"].(string)[:10])
+		t.Fatal("Wrong Bloom")
+	}
+	if data["coinbase"] != "0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5" {
+		t.Fatalf("Wrong coinbase\r\nexpected %s\r\ngot %s", "0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5", data["coinbase"])
+	}
+	if parseFloat(data["difficulty"]) != "12555463106190" {
+		t.Fatalf("Wrong Difficulty\r\nexpected %s\r\ngot %s", "12555463106190", parseFloat(data["difficulty"]))
+	}
+	if data["extra"] != "0xd783010303844765746887676f312e342e32856c696e7578" {
+		t.Fatalf("Wrong Extra\r\nexpected %s\r\ngot %s", "0xd783010303844765746887676f312e342e32856c696e7578", data["extra"])
+	}
+	if parseFloat(data["gaslimit"]) != "3141592" {
+		t.Fatalf("Wrong Gas limit\r\nexpected %s\r\ngot %s", "3141592", parseFloat(data["gaslimit"]))
+	}
+	if parseFloat(data["gasused"]) != "231000" {
+		t.Fatalf("Wrong Gas used\r\nexpected %s\r\ngot %s", "231000", parseFloat(data["gasused"]))
+	}
+	if data["mixdigest"] != "0x5b10f4a08a6c209d426f6158bd24b574f4f7b7aa0099c67c14a1f693b4dd04d0" {
+		t.Fatalf("Wrong Mix digest\r\nexpected %s\r\ngot %s", "0x5b10f4a08a6c209d426f6158bd24b574f4f7b7aa0099c67c14a1f693b4dd04d0", data["mixdigest"])
+	}
+	if data["nonce"] != "0xf491f46b60fe04b3" {
+		t.Fatalf("Wrong nonce\r\nexpected %s\r\ngot %s", "0xf491f46b60fe04b3", data["nonce"])
+	}
+	if parseFloat(data["number"]) != "999999" {
+		t.Fatalf("Wrong block number\r\nexpected %s\r\ngot %s", "999999", parseFloat(data["number"]))
+	}
+	if parseMapElement(data["parent"]) != "bagiacgza2m6j3xu774hlvjxhd2fsnuv5ufom6ei4ply3mm3jrleeozt7b62a" {
+		t.Fatalf("Wrong Parent cid\r\nexpected %s\r\ngot %s", "bagiacgza2m6j3xu774hlvjxhd2fsnuv5ufom6ei4ply3mm3jrleeozt7b62a", parseMapElement(data["parent"]))
+	}
+	if parseMapElement(data["receipts"]) != "bagkacgzap6qpnsrkagbdecgybaa63ljx4pr2aa5vlsetdg2f5mpzpbrk2iuq" {
+		t.Fatalf("Wrong Receipt root cid\r\nexpected %s\r\ngot %s", "bagkacgzap6qpnsrkagbdecgybaa63ljx4pr2aa5vlsetdg2f5mpzpbrk2iuq", parseMapElement(data["receipts"]))
+	}
+	if parseMapElement(data["root"]) != "baglacgza5wmkus23dhec7m2tmtyikcfobjw6yzs7uv3ghxfjjroxavkm3yia" {
+		t.Fatalf("Wrong root hash cid\r\nexpected %s\r\ngot %s", "baglacgza5wmkus23dhec7m2tmtyikcfobjw6yzs7uv3ghxfjjroxavkm3yia", parseMapElement(data["root"]))
+	}
+	if parseFloat(data["time"]) != "1455404037" {
+		t.Fatalf("Wrong Time\r\nexpected %s\r\ngot %s", "1455404037", parseFloat(data["time"]))
+	}
+	if parseMapElement(data["tx"]) != "bagjacgzair6l3dci6smknejlccbrzx7vtr737s56onoksked2t5anxgxvzka" {
+		t.Fatalf("Wrong Tx root cid\r\nexpected %s\r\ngot %s", "bagjacgzair6l3dci6smknejlccbrzx7vtr737s56onoksked2t5anxgxvzka", parseMapElement(data["tx"]))
+	}
+	if parseMapElement(data["uncles"]) != "bagiqcgzadxge32g6y5oxvk4fwvt3ntgudljreri3ssfhie7qufbp2qgusndq" {
+		t.Fatalf("Wrong Uncle hash cid\r\nexpected %s\r\ngot %s", "bagiqcgzadxge32g6y5oxvk4fwvt3ntgudljreri3ssfhie7qufbp2qgusndq", parseMapElement(data["uncles"]))
+	}
+}
+
+func TestEthBlockLinks(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	links := ethBlock.Links()
+	if links[0].Cid.String() != "bagiacgza2m6j3xu774hlvjxhd2fsnuv5ufom6ei4ply3mm3jrleeozt7b62a" {
+		t.Fatalf("Wrong cid for parent link\r\nexpected: %s\r\ngot %s", "bagiacgza2m6j3xu774hlvjxhd2fsnuv5ufom6ei4ply3mm3jrleeozt7b62a", links[0].Cid.String())
+	}
+	if links[1].Cid.String() != "bagkacgzap6qpnsrkagbdecgybaa63ljx4pr2aa5vlsetdg2f5mpzpbrk2iuq" {
+		t.Fatalf("Wrong cid for receipt root link\r\nexpected: %s\r\ngot %s", "bagkacgzap6qpnsrkagbdecgybaa63ljx4pr2aa5vlsetdg2f5mpzpbrk2iuq", links[1].Cid.String())
+	}
+	if links[2].Cid.String() != "baglacgza5wmkus23dhec7m2tmtyikcfobjw6yzs7uv3ghxfjjroxavkm3yia" {
+		t.Fatalf("Wrong cid for state root link\r\nexpected: %s\r\ngot %s", "baglacgza5wmkus23dhec7m2tmtyikcfobjw6yzs7uv3ghxfjjroxavkm3yia", links[2].Cid.String())
+	}
+	if links[3].Cid.String() != "bagjacgzair6l3dci6smknejlccbrzx7vtr737s56onoksked2t5anxgxvzka" {
+		t.Fatalf("Wrong cid for tx root link\r\nexpected: %s\r\ngot %s", "bagjacgzair6l3dci6smknejlccbrzx7vtr737s56onoksked2t5anxgxvzka", links[3].Cid.String())
+	}
+	if links[4].Cid.String() != "bagiqcgzadxge32g6y5oxvk4fwvt3ntgudljreri3ssfhie7qufbp2qgusndq" {
+		t.Fatalf("Wrong cid for uncles root link\r\nexpected: %s\r\ngot %s", "bagiqcgzadxge32g6y5oxvk4fwvt3ntgudljreri3ssfhie7qufbp2qgusndq", links[4].Cid.String())
+	}
+}
+
+func TestEthBlockResolveEmptyPath(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	obj, rest, err := ethBlock.Resolve([]string{})
+	checkError(err, t)
+
+	if ethBlock != obj.(*EthHeader) {
+		t.Fatal("Should have returned the same eth-block object")
+	}
+
+	if len(rest) != 0 {
+		t.Fatalf("Wrong len of rest of the path returned\r\nexpected %d\r\ngot %d", 0, len(rest))
+	}
+}
+
+func TestEthBlockResolveNoSuchLink(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	_, _, err := ethBlock.Resolve([]string{"wewonthavethisfieldever"})
+	if err == nil {
+		t.Fatal("Should have failed with unknown field")
+	}
+
+	if err.Error() != "no such link" {
+		t.Fatalf("Wrong error message\r\nexpected %s\r\ngot %s", "no such link", err.Error())
+	}
+}
+
+func TestEthBlockResolveBloom(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	obj, rest, err := ethBlock.Resolve([]string{"bloom"})
+	checkError(err, t)
+
+	// The marshaler of types.Bloom should output it as 0x
+	bloomInText := fmt.Sprintf("%x", obj.(types.Bloom))
+	if bloomInText[:10] != "0000000000" {
+		t.Fatalf("Wrong Bloom\r\nexpected %s\r\ngot %s", "0000000000", bloomInText[:10])
+	}
+
+	if len(rest) != 0 {
+		t.Fatalf("Wrong len of rest of the path returned\r\nexpected %d\r\ngot %d", 0, len(rest))
+	}
+}
+
+func TestEthBlockResolveBloomExtraPathElements(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	obj, rest, err := ethBlock.Resolve([]string{"bloom", "unexpected", "extra", "elements"})
+	if obj != nil {
+		t.Fatal("Returned obj should be nil")
+	}
+
+	if rest != nil {
+		t.Fatal("Returned rest should be nil")
+	}
+
+	if err.Error() != "unexpected path elements past bloom" {
+		t.Fatalf("Wrong error\r\nexpected %s\r\ngot %s", "unexpected path elements past bloom", err.Error())
+	}
+}
+
+func TestEthBlockResolveNonLinkFields(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	testCases := map[string][]string{
+		"coinbase":   {"%x", "52bc44d5378309ee2abf1539bf71de1b7d7be3b5"},
+		"difficulty": {"%s", "12555463106190"},
+		"extra":      {"%s", "0xd783010303844765746887676f312e342e32856c696e7578"},
+		"gaslimit":   {"%d", "3141592"},
+		"gasused":    {"%d", "231000"},
+		"mixdigest":  {"%x", "5b10f4a08a6c209d426f6158bd24b574f4f7b7aa0099c67c14a1f693b4dd04d0"},
+		"nonce":      {"%x", "f491f46b60fe04b3"},
+		"number":     {"%s", "999999"},
+		"time":       {"%d", "1455404037"},
+	}
+
+	for field, value := range testCases {
+		obj, rest, err := ethBlock.Resolve([]string{field})
+		checkError(err, t)
+
+		format := value[0]
+		result := value[1]
+		if fmt.Sprintf(format, obj) != result {
+			t.Fatalf("Wrong %v\r\nexpected %v\r\ngot %s", field, result, fmt.Sprintf(format, obj))
+		}
+
+		if len(rest) != 0 {
+			t.Fatalf("Wrong len of rest of the path returned\r\nexpected %d\r\ngot %d", 0, len(rest))
+		}
+	}
+}
+
+func TestEthBlockResolveNonLinkFieldsExtraPathElements(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	testCases := []string{
+		"coinbase",
+		"difficulty",
+		"extra",
+		"gaslimit",
+		"gasused",
+		"mixdigest",
+		"nonce",
+		"number",
+		"time",
+	}
+
+	for _, field := range testCases {
+		obj, rest, err := ethBlock.Resolve([]string{field, "unexpected", "extra", "elements"})
+		if obj != nil {
+			t.Fatal("Returned obj should be nil")
+		}
+
+		if rest != nil {
+			t.Fatal("Returned rest should be nil")
+		}
+
+		if err.Error() != "unexpected path elements past "+field {
+			t.Fatalf("Wrong error\r\nexpected %s\r\ngot %s", "unexpected path elements past "+field, err.Error())
+		}
+
+	}
+}
+
+func TestEthBlockResolveLinkFields(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	testCases := map[string]string{
+		"parent":   "bagiacgza2m6j3xu774hlvjxhd2fsnuv5ufom6ei4ply3mm3jrleeozt7b62a",
+		"receipts": "bagkacgzap6qpnsrkagbdecgybaa63ljx4pr2aa5vlsetdg2f5mpzpbrk2iuq",
+		"root":     "baglacgza5wmkus23dhec7m2tmtyikcfobjw6yzs7uv3ghxfjjroxavkm3yia",
+		"tx":       "bagjacgzair6l3dci6smknejlccbrzx7vtr737s56onoksked2t5anxgxvzka",
+		"uncles":   "bagiqcgzadxge32g6y5oxvk4fwvt3ntgudljreri3ssfhie7qufbp2qgusndq",
+	}
+
+	for field, result := range testCases {
+		obj, rest, err := ethBlock.Resolve([]string{field, "anything", "goes", "here"})
+		checkError(err, t)
+
+		lnk, ok := obj.(*node.Link)
+		if !ok {
+			t.Fatal("Returned object is not a link")
+		}
+
+		if lnk.Cid.String() != result {
+			t.Fatalf("Wrong %s cid\r\nexpected %v\r\ngot %v", field, result, lnk.Cid.String())
+		}
+
+		for i, p := range []string{"anything", "goes", "here"} {
+			if rest[i] != p {
+				t.Fatalf("Wrong rest of the path returned\r\nexpected %s\r\ngot %s", p, rest[i])
+			}
+		}
+	}
+}
+
+func TestEthBlockTreeBadParams(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	tree := ethBlock.Tree("non-empty-string", 0)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	tree = ethBlock.Tree("non-empty-string", 1)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	tree = ethBlock.Tree("", 0)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+}
+
+func TestEThBlockTree(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	tree := ethBlock.Tree("", 1)
+	lookupElements := map[string]interface{}{
+		"bloom":      nil,
+		"coinbase":   nil,
+		"difficulty": nil,
+		"extra":      nil,
+		"gaslimit":   nil,
+		"gasused":    nil,
+		"mixdigest":  nil,
+		"nonce":      nil,
+		"number":     nil,
+		"parent":     nil,
+		"receipts":   nil,
+		"root":       nil,
+		"time":       nil,
+		"tx":         nil,
+		"uncles":     nil,
+	}
+
+	if len(tree) != len(lookupElements) {
+		t.Fatalf("Wrong number of elements\r\nexpected %d\r\ngot %d", len(lookupElements), len(tree))
+	}
+
+	for _, te := range tree {
+		if _, ok := lookupElements[te]; !ok {
+			t.Fatalf("Unexpected Element: %v", te)
+		}
+	}
+}
+
+/*
+  The two functions above: TestEthBlockResolveNonLinkFields and
+  TestEthBlockResolveLinkFields did all the heavy lifting. Then, we will
+  just test two use cases.
+*/
+func TestEthBlockResolveLinksBadLink(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	obj, rest, err := ethBlock.ResolveLink([]string{"supercalifragilist"})
+	if obj != nil {
+		t.Fatalf("Expected obj to be nil")
+	}
+	if rest != nil {
+		t.Fatal("Expected rest to be nil")
+	}
+	if err.Error() != "no such link" {
+		t.Fatalf("Expected error\r\nexpected %s\r\ngot %s", "no such link", err.Error())
+	}
+}
+
+func TestEthBlockResolveLinksGoodLink(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	obj, rest, err := ethBlock.ResolveLink([]string{"tx", "0", "0", "0"})
+	if obj == nil {
+		t.Fatalf("Expected valid *node.Link obj to be returned")
+	}
+
+	if rest == nil {
+		t.Fatal("Expected rest to be returned")
+	}
+	for i, p := range []string{"0", "0", "0"} {
+		if rest[i] != p {
+			t.Fatalf("Wrong rest of the path returned\r\nexpected %s\r\ngot %s", p, rest[i])
+		}
+	}
+
+	if err != nil {
+		t.Fatal("Non error expected")
+	}
+}
+
+/*
+  These functions below should go away
+  We are working on test coverage anyways...
+*/
+func TestEthBlockCopy(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	defer func() {
+		r := recover()
+		if r == nil {
+			t.Fatal("Expected panic")
+		}
+		if r != "implement me" {
+			t.Fatalf("Wrong panic message\r\nexpected %s\r\ngot %s", "'implement me'", r)
+		}
+	}()
+
+	_ = ethBlock.Copy()
+}
+
+func TestEthBlockStat(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	obj, err := ethBlock.Stat()
+	if obj == nil {
+		t.Fatal("Expected a not null object node.NodeStat")
+	}
+
+	if err != nil {
+		t.Fatal("Expected a nil error")
+	}
+}
+
+func TestEthBlockSize(t *testing.T) {
+	ethBlock := prepareDecodedEthBlock("test_data/eth-block-header-rlp-999999", t)
+
+	size, err := ethBlock.Size()
+	if size != 0 {
+		t.Fatalf("Wrong size\r\nexpected %d\r\ngot %d", 0, size)
+	}
+
+	if err != nil {
+		t.Fatal("Expected a nil error")
+	}
+}
+
+/*
+  AUXILIARS
+*/
+
+// checkError makes 3 lines into 1.
+func checkError(err error, t *testing.T) {
+	if err != nil {
+		_, fn, line, _ := runtime.Caller(1)
+		t.Fatalf("[%v:%v] %v", fn, line, err)
+	}
+}
+
+// parseFloat is a convenience function to test json output
+func parseFloat(v interface{}) string {
+	return strconv.FormatFloat(v.(float64), 'f', 0, 64)
+}
+
+// parseMapElement is a convenience function to tets json output
+func parseMapElement(v interface{}) string {
+	return v.(map[string]interface{})["/"].(string)
+}
+
+// prepareStoredEthBlock reads the block from a file source to get its rawdata
+// and computes its cid, for then, feeding it into a new IPLD block function.
+// So we can pretend that we got this block from the datastore
+func prepareStoredEthBlock(filepath string, t *testing.T) *block.BasicBlock {
+	// Prepare the "fetched block". This one is supposed to be in the datastore
+	// and given away by github.com/ipfs/go-ipfs/merkledag
+	fi, err := os.Open(filepath)
+	checkError(err, t)
+
+	b, err := ioutil.ReadAll(fi)
+	checkError(err, t)
+
+	c, err := RawdataToCid(MEthHeader, b, multihash.KECCAK_256)
+	checkError(err, t)
+
+	// It's good to clarify that this one below is an IPLD block
+	storedEthBlock, err := block.NewBlockWithCid(b, c)
+	checkError(err, t)
+
+	return storedEthBlock
+}
+
+// prepareDecodedEthBlock is more complex than function above, as it stores a
+// basic block and RLP-decodes it
+func prepareDecodedEthBlock(filepath string, t *testing.T) *EthHeader {
+	// Get the block from the datastore and decode it.
+	storedEthBlock := prepareStoredEthBlock("test_data/eth-block-header-rlp-999999", t)
+	ethBlock, err := DecodeEthHeader(storedEthBlock.Cid(), storedEthBlock.RawData())
+	checkError(err, t)
+
+	return ethBlock
+}
+
+// testEthBlockFields checks the fields of EthBlock one by one.
+func testEthBlockFields(ethBlock *EthHeader, t *testing.T) {
+	// Was the cid calculated?
+	if ethBlock.Cid().String() != "bagiacgzawt5236hkiuvrhfyy4jya3qitlt6icfcqgheew6vsptlraokppm4a" {
+		t.Fatalf("Wrong cid\r\nexpected %s\r\ngot %s", "bagiacgzawt5236hkiuvrhfyy4jya3qitlt6icfcqgheew6vsptlraokppm4a", ethBlock.Cid().String())
+	}
+
+	// Do we have the rawdata available?
+	if fmt.Sprintf("%x", ethBlock.RawData()[:10]) != "f90218a0d33c9dde9fff" {
+		t.Fatalf("Wrong Rawdata\r\nexpected %s\r\ngot %s", "f90218a0d33c9dde9fff", fmt.Sprintf("%x", ethBlock.RawData()[:10]))
+	}
+
+	// Proper Fields of types.Header
+	if fmt.Sprintf("%x", ethBlock.ParentHash) != "d33c9dde9fff0ebaa6e71e8b26d2bda15ccf111c7af1b633698ac847667f0fb4" {
+		t.Fatalf("Wrong ParentHash\r\nexpected %s\r\ngot %s", "d33c9dde9fff0ebaa6e71e8b26d2bda15ccf111c7af1b633698ac847667f0fb4", fmt.Sprintf("%x", ethBlock.ParentHash))
+	}
+	if fmt.Sprintf("%x", ethBlock.UncleHash) != "1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347" {
+		t.Fatalf("Wrong UncleHash field\r\nexpected %s\r\ngot %s", "1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347", fmt.Sprintf("%x", ethBlock.UncleHash))
+	}
+	if fmt.Sprintf("%x", ethBlock.Coinbase) != "52bc44d5378309ee2abf1539bf71de1b7d7be3b5" {
+		t.Fatalf("Wrong Coinbase\r\nexpected %s\r\ngot %s", "52bc44d5378309ee2abf1539bf71de1b7d7be3b5", fmt.Sprintf("%x", ethBlock.Coinbase))
+	}
+	if fmt.Sprintf("%x", ethBlock.Root) != "ed98aa4b5b19c82fb35364f08508ae0a6dec665fa57663dca94c5d70554cde10" {
+		t.Fatalf("Wrong Root\r\nexpected %s\r\ngot %s", "ed98aa4b5b19c82fb35364f08508ae0a6dec665fa57663dca94c5d70554cde10", fmt.Sprintf("%x", ethBlock.Root))
+	}
+	if fmt.Sprintf("%x", ethBlock.TxHash) != "447cbd8c48f498a6912b10831cdff59c7fbfcbbe735ca92883d4fa06dcd7ae54" {
+		t.Fatalf("Wrong TxHash\r\nexpected %s\r\ngot %s", "447cbd8c48f498a6912b10831cdff59c7fbfcbbe735ca92883d4fa06dcd7ae54", fmt.Sprintf("%x", ethBlock.TxHash))
+	}
+	if fmt.Sprintf("%x", ethBlock.ReceiptHash) != "7fa0f6ca2a01823208d80801edad37e3e3a003b55c89319b45eb1f97862ad229" {
+		t.Fatalf("Wrong ReceiptHash\r\nexpected %s\r\ngot %s", "7fa0f6ca2a01823208d80801edad37e3e3a003b55c89319b45eb1f97862ad229", fmt.Sprintf("%x", ethBlock.ReceiptHash))
+	}
+	if len(ethBlock.Bloom) != 256 {
+		t.Fatalf("Wrong Bloom Length\r\nexpected %d\r\ngot %d", 256, len(ethBlock.Bloom))
+	}
+	if fmt.Sprintf("%x", ethBlock.Bloom[71:76]) != "0000000000" { // You wouldn't want me to print out the whole bloom field?
+		t.Fatalf("Wrong Bloom\r\nexpected %s\r\ngot %s", "0000000000", fmt.Sprintf("%x", ethBlock.Bloom[71:76]))
+	}
+	if ethBlock.Difficulty.String() != "12555463106190" {
+		t.Fatalf("Wrong Difficulty\r\nexpected %s\r\ngot %s", "12555463106190", ethBlock.Difficulty.String())
+	}
+	if ethBlock.Number.String() != "999999" {
+		t.Fatalf("Wrong Block Number\r\nexpected %s\r\ngot %s", "999999", ethBlock.Number.String())
+	}
+	if ethBlock.GasLimit != uint64(3141592) {
+		t.Fatalf("Wrong Gas Limit\r\nexpected %d\r\ngot %d", 3141592, ethBlock.GasLimit)
+	}
+	if ethBlock.GasUsed != uint64(231000) {
+		t.Fatalf("Wrong Gas Used\r\nexpected %d\r\ngot %d", 231000, ethBlock.GasUsed)
+	}
+	if ethBlock.Time != uint64(1455404037) {
+		t.Fatalf("Wrong Time\r\nexpected %d\r\ngot %d", 1455404037, ethBlock.Time)
+	}
+	if fmt.Sprintf("%x", ethBlock.Extra) != "d783010303844765746887676f312e342e32856c696e7578" {
+		t.Fatalf("Wrong Extra\r\nexpected %s\r\ngot %s", "d783010303844765746887676f312e342e32856c696e7578", fmt.Sprintf("%x", ethBlock.Extra))
+	}
+	if fmt.Sprintf("%x", ethBlock.Nonce) != "f491f46b60fe04b3" {
+		t.Fatalf("Wrong Nonce\r\nexpected %s\r\ngot %s", "f491f46b60fe04b3", fmt.Sprintf("%x", ethBlock.Nonce))
+	}
+	if fmt.Sprintf("%x", ethBlock.MixDigest) != "5b10f4a08a6c209d426f6158bd24b574f4f7b7aa0099c67c14a1f693b4dd04d0" {
+		t.Fatalf("Wrong MixDigest\r\nexpected %s\r\ngot %s", "5b10f4a08a6c209d426f6158bd24b574f4f7b7aa0099c67c14a1f693b4dd04d0", fmt.Sprintf("%x", ethBlock.MixDigest))
+	}
+}

statediff/indexer/ipfs/ipld/eth_parser.go

@@ -0,0 +1,198 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+
+	"github.com/multiformats/go-multihash"
+
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/rlp"
+)
+
+// FromBlockRLP takes an RLP message representing
+// an ethereum block header or body (header, ommers and txs)
+// to return it as a set of IPLD nodes for further processing.
+func FromBlockRLP(r io.Reader) (*EthHeader, []*EthTx, []*EthTxTrie, error) {
+	// We may want to use this stream several times
+	rawdata, err := ioutil.ReadAll(r)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+
+	// Let's try to decode the received element as a block body
+	var decodedBlock types.Block
+	err = rlp.Decode(bytes.NewBuffer(rawdata), &decodedBlock)
+	if err != nil {
+		if err.Error()[:41] != "rlp: expected input list for types.Header" {
+			return nil, nil, nil, err
+		}
+
+		// Maybe it is just a header... (body sans ommers and txs)
+		var decodedHeader types.Header
+		err := rlp.Decode(bytes.NewBuffer(rawdata), &decodedHeader)
+		if err != nil {
+			return nil, nil, nil, err
+		}
+
+		c, err := RawdataToCid(MEthHeader, rawdata, multihash.KECCAK_256)
+		if err != nil {
+			return nil, nil, nil, err
+		}
+		// It was a header
+		return &EthHeader{
+			Header:  &decodedHeader,
+			cid:     c,
+			rawdata: rawdata,
+		}, nil, nil, nil
+	}
+
+	// This is a block body (header + ommers + txs)
+	// We'll extract the header bits here
+	headerRawData := getRLP(decodedBlock.Header())
+	c, err := RawdataToCid(MEthHeader, headerRawData, multihash.KECCAK_256)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+	ethBlock := &EthHeader{
+		Header:  decodedBlock.Header(),
+		cid:     c,
+		rawdata: headerRawData,
+	}
+
+	// Process the found eth-tx objects
+	ethTxNodes, ethTxTrieNodes, err := processTransactions(decodedBlock.Transactions(),
+		decodedBlock.Header().TxHash[:])
+	if err != nil {
+		return nil, nil, nil, err
+	}
+
+	return ethBlock, ethTxNodes, ethTxTrieNodes, nil
+}
+
+// FromBlockJSON takes the output of an ethereum client JSON API
+// (i.e. parity or geth) and returns a set of IPLD nodes.
+func FromBlockJSON(r io.Reader) (*EthHeader, []*EthTx, []*EthTxTrie, error) {
+	var obj objJSONHeader
+	dec := json.NewDecoder(r)
+	err := dec.Decode(&obj)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+
+	headerRawData := getRLP(obj.Result.Header)
+	c, err := RawdataToCid(MEthHeader, headerRawData, multihash.KECCAK_256)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+	ethBlock := &EthHeader{
+		Header:  &obj.Result.Header,
+		cid:     c,
+		rawdata: headerRawData,
+	}
+
+	// Process the found eth-tx objects
+	ethTxNodes, ethTxTrieNodes, err := processTransactions(obj.Result.Transactions,
+		obj.Result.Header.TxHash[:])
+	if err != nil {
+		return nil, nil, nil, err
+	}
+
+	return ethBlock, ethTxNodes, ethTxTrieNodes, nil
+}
+
+// FromBlockAndReceipts takes a block and processes it
+// to return it a set of IPLD nodes for further processing.
+func FromBlockAndReceipts(block *types.Block, receipts []*types.Receipt) (*EthHeader, []*EthHeader, []*EthTx, []*EthTxTrie, []*EthReceipt, []*EthRctTrie, error) {
+	// Process the header
+	headerNode, err := NewEthHeader(block.Header())
+	if err != nil {
+		return nil, nil, nil, nil, nil, nil, err
+	}
+	// Process the uncles
+	uncleNodes := make([]*EthHeader, len(block.Uncles()))
+	for i, uncle := range block.Uncles() {
+		uncleNode, err := NewEthHeader(uncle)
+		if err != nil {
+			return nil, nil, nil, nil, nil, nil, err
+		}
+		uncleNodes[i] = uncleNode
+	}
+	// Process the txs
+	ethTxNodes, ethTxTrieNodes, err := processTransactions(block.Transactions(),
+		block.Header().TxHash[:])
+	if err != nil {
+		return nil, nil, nil, nil, nil, nil, err
+	}
+	// Process the receipts
+	ethRctNodes, ethRctTrieNodes, err := processReceipts(receipts,
+		block.Header().ReceiptHash[:])
+	return headerNode, uncleNodes, ethTxNodes, ethTxTrieNodes, ethRctNodes, ethRctTrieNodes, err
+}
+
+// processTransactions will take the found transactions in a parsed block body
+// to return IPLD node slices for eth-tx and eth-tx-trie
+func processTransactions(txs []*types.Transaction, expectedTxRoot []byte) ([]*EthTx, []*EthTxTrie, error) {
+	var ethTxNodes []*EthTx
+	transactionTrie := newTxTrie()
+
+	for idx, tx := range txs {
+		ethTx, err := NewEthTx(tx)
+		if err != nil {
+			return nil, nil, err
+		}
+		ethTxNodes = append(ethTxNodes, ethTx)
+		if err := transactionTrie.add(idx, ethTx.RawData()); err != nil {
+			return nil, nil, err
+		}
+	}
+
+	if !bytes.Equal(transactionTrie.rootHash(), expectedTxRoot) {
+		return nil, nil, fmt.Errorf("wrong transaction hash computed")
+	}
+	txTrieNodes, err := transactionTrie.getNodes()
+	return ethTxNodes, txTrieNodes, err
+}
+
+// processReceipts will take in receipts
+// to return IPLD node slices for eth-rct and eth-rct-trie
+func processReceipts(rcts []*types.Receipt, expectedRctRoot []byte) ([]*EthReceipt, []*EthRctTrie, error) {
+	var ethRctNodes []*EthReceipt
+	receiptTrie := newRctTrie()
+
+	for idx, rct := range rcts {
+		ethRct, err := NewReceipt(rct)
+		if err != nil {
+			return nil, nil, err
+		}
+		ethRctNodes = append(ethRctNodes, ethRct)
+		if err := receiptTrie.add(idx, ethRct.RawData()); err != nil {
+			return nil, nil, err
+		}
+	}
+
+	if !bytes.Equal(receiptTrie.rootHash(), expectedRctRoot) {
+		return nil, nil, fmt.Errorf("wrong receipt hash computed")
+	}
+	rctTrieNodes, err := receiptTrie.getNodes()
+	return ethRctNodes, rctTrieNodes, err
+}

statediff/indexer/ipfs/ipld/eth_receipt.go

@@ -0,0 +1,198 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"encoding/json"
+	"fmt"
+	"strconv"
+
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+	mh "github.com/multiformats/go-multihash"
+)
+
+type EthReceipt struct {
+	*types.Receipt
+
+	rawdata []byte
+	cid     cid.Cid
+}
+
+// Static (compile time) check that EthReceipt satisfies the node.Node interface.
+var _ node.Node = (*EthReceipt)(nil)
+
+/*
+  INPUT
+*/
+
+// NewReceipt converts a types.ReceiptForStorage to an EthReceipt IPLD node
+func NewReceipt(receipt *types.Receipt) (*EthReceipt, error) {
+	rctRaw, err := receipt.MarshalBinary()
+	if err != nil {
+		return nil, err
+	}
+	c, err := RawdataToCid(MEthTxReceipt, rctRaw, mh.KECCAK_256)
+	if err != nil {
+		return nil, err
+	}
+	return &EthReceipt{
+		Receipt: receipt,
+		cid:     c,
+		rawdata: rctRaw,
+	}, nil
+}
+
+/*
+ OUTPUT
+*/
+
+// DecodeEthReceipt takes a cid and its raw binary data
+// from IPFS and returns an EthTx object for further processing.
+func DecodeEthReceipt(c cid.Cid, b []byte) (*EthReceipt, error) {
+	r := new(types.Receipt)
+	if err := r.UnmarshalBinary(b); err != nil {
+		return nil, err
+	}
+	return &EthReceipt{
+		Receipt: r,
+		cid:     c,
+		rawdata: b,
+	}, nil
+}
+
+/*
+  Block INTERFACE
+*/
+
+func (node *EthReceipt) RawData() []byte {
+	return node.rawdata
+}
+
+func (node *EthReceipt) Cid() cid.Cid {
+	return node.cid
+}
+
+// String is a helper for output
+func (r *EthReceipt) String() string {
+	return fmt.Sprintf("<EthereumReceipt %s>", r.cid)
+}
+
+// Loggable returns in a map the type of IPLD Link.
+func (r *EthReceipt) Loggable() map[string]interface{} {
+	return map[string]interface{}{
+		"type": "eth-receipt",
+	}
+}
+
+// Resolve resolves a path through this node, stopping at any link boundary
+// and returning the object found as well as the remaining path to traverse
+func (r *EthReceipt) Resolve(p []string) (interface{}, []string, error) {
+	if len(p) == 0 {
+		return r, nil, nil
+	}
+
+	if len(p) > 1 {
+		return nil, nil, fmt.Errorf("unexpected path elements past %s", p[0])
+	}
+
+	switch p[0] {
+
+	case "root":
+		return r.PostState, nil, nil
+	case "status":
+		return r.Status, nil, nil
+	case "cumulativeGasUsed":
+		return r.CumulativeGasUsed, nil, nil
+	case "logsBloom":
+		return r.Bloom, nil, nil
+	case "logs":
+		return r.Logs, nil, nil
+	case "transactionHash":
+		return r.TxHash, nil, nil
+	case "contractAddress":
+		return r.ContractAddress, nil, nil
+	case "gasUsed":
+		return r.GasUsed, nil, nil
+	default:
+		return nil, nil, fmt.Errorf("no such link")
+	}
+}
+
+// Tree lists all paths within the object under 'path', and up to the given depth.
+// To list the entire object (similar to `find .`) pass "" and -1
+func (r *EthReceipt) Tree(p string, depth int) []string {
+	if p != "" || depth == 0 {
+		return nil
+	}
+	return []string{"root", "status", "cumulativeGasUsed", "logsBloom", "logs", "transactionHash", "contractAddress", "gasUsed"}
+}
+
+// ResolveLink is a helper function that calls resolve and asserts the
+// output is a link
+func (r *EthReceipt) ResolveLink(p []string) (*node.Link, []string, error) {
+	obj, rest, err := r.Resolve(p)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if lnk, ok := obj.(*node.Link); ok {
+		return lnk, rest, nil
+	}
+
+	return nil, nil, fmt.Errorf("resolved item was not a link")
+}
+
+// Copy will go away. It is here to comply with the Node interface.
+func (*EthReceipt) Copy() node.Node {
+	panic("implement me")
+}
+
+// Links is a helper function that returns all links within this object
+func (*EthReceipt) Links() []*node.Link {
+	return nil
+}
+
+// Stat will go away. It is here to comply with the interface.
+func (r *EthReceipt) Stat() (*node.NodeStat, error) {
+	return &node.NodeStat{}, nil
+}
+
+// Size will go away. It is here to comply with the interface.
+func (r *EthReceipt) Size() (uint64, error) {
+	return strconv.ParseUint(r.Receipt.Size().String(), 10, 64)
+}
+
+/*
+  EthReceipt functions
+*/
+
+// MarshalJSON processes the receipt into readable JSON format.
+func (r *EthReceipt) MarshalJSON() ([]byte, error) {
+	out := map[string]interface{}{
+		"root":              r.PostState,
+		"status":            r.Status,
+		"cumulativeGasUsed": r.CumulativeGasUsed,
+		"logsBloom":         r.Bloom,
+		"logs":              r.Logs,
+		"transactionHash":   r.TxHash,
+		"contractAddress":   r.ContractAddress,
+		"gasUsed":           r.GasUsed,
+	}
+	return json.Marshal(out)
+}

statediff/indexer/ipfs/ipld/eth_receipt_trie.go

@@ -0,0 +1,152 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"fmt"
+
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+	"github.com/multiformats/go-multihash"
+
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/rlp"
+)
+
+// EthRctTrie (eth-tx-trie codec 0x92) represents
+// a node from the transaction trie in ethereum.
+type EthRctTrie struct {
+	*TrieNode
+}
+
+// Static (compile time) check that EthRctTrie satisfies the node.Node interface.
+var _ node.Node = (*EthRctTrie)(nil)
+
+/*
+ INPUT
+*/
+
+// To create a proper trie of the eth-tx-trie objects, it is required
+// to input all transactions belonging to a forest in a single step.
+// We are adding the transactions, and creating its trie on
+// block body parsing time.
+
+/*
+  OUTPUT
+*/
+
+// DecodeEthRctTrie returns an EthRctTrie object from its cid and rawdata.
+func DecodeEthRctTrie(c cid.Cid, b []byte) (*EthRctTrie, error) {
+	tn, err := decodeTrieNode(c, b, decodeEthRctTrieLeaf)
+	if err != nil {
+		return nil, err
+	}
+	return &EthRctTrie{TrieNode: tn}, nil
+}
+
+// decodeEthRctTrieLeaf parses a eth-rct-trie leaf
+//from decoded RLP elements
+func decodeEthRctTrieLeaf(i []interface{}) ([]interface{}, error) {
+	var r types.Receipt
+	err := rlp.DecodeBytes(i[1].([]byte), &r)
+	if err != nil {
+		return nil, err
+	}
+	c, err := RawdataToCid(MEthTxReceipt, i[1].([]byte), multihash.KECCAK_256)
+	if err != nil {
+		return nil, err
+	}
+	return []interface{}{
+		i[0].([]byte),
+		&EthReceipt{
+			Receipt: &r,
+			cid:     c,
+			rawdata: i[1].([]byte),
+		},
+	}, nil
+}
+
+/*
+  Block INTERFACE
+*/
+
+// RawData returns the binary of the RLP encode of the transaction.
+func (t *EthRctTrie) RawData() []byte {
+	return t.rawdata
+}
+
+// Cid returns the cid of the transaction.
+func (t *EthRctTrie) Cid() cid.Cid {
+	return t.cid
+}
+
+// String is a helper for output
+func (t *EthRctTrie) String() string {
+	return fmt.Sprintf("<EthereumRctTrie %s>", t.cid)
+}
+
+// Loggable returns in a map the type of IPLD Link.
+func (t *EthRctTrie) Loggable() map[string]interface{} {
+	return map[string]interface{}{
+		"type": "eth-rct-trie",
+	}
+}
+
+/*
+  EthRctTrie functions
+*/
+
+// rctTrie wraps a localTrie for use on the receipt trie.
+type rctTrie struct {
+	*localTrie
+}
+
+// newRctTrie initializes and returns a rctTrie.
+func newRctTrie() *rctTrie {
+	return &rctTrie{
+		localTrie: newLocalTrie(),
+	}
+}
+
+// getNodes invokes the localTrie, which computes the root hash of the
+// transaction trie and returns its database keys, to return a slice
+// of EthRctTrie nodes.
+func (rt *rctTrie) getNodes() ([]*EthRctTrie, error) {
+	keys, err := rt.getKeys()
+	if err != nil {
+		return nil, err
+	}
+	var out []*EthRctTrie
+
+	for _, k := range keys {
+		rawdata, err := rt.db.Get(k)
+		if err != nil {
+			panic(err)
+		}
+		c, err := RawdataToCid(MEthTxReceiptTrie, rawdata, multihash.KECCAK_256)
+		if err != nil {
+			return nil, err
+		}
+		tn := &TrieNode{
+			cid:     c,
+			rawdata: rawdata,
+		}
+		out = append(out, &EthRctTrie{TrieNode: tn})
+	}
+
+	return out, nil
+}

statediff/indexer/ipfs/ipld/eth_state.go

@@ -0,0 +1,126 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+	"github.com/multiformats/go-multihash"
+
+	"github.com/ethereum/go-ethereum/rlp"
+)
+
+// EthStateTrie (eth-state-trie, codec 0x96), represents
+// a node from the satte trie in ethereum.
+type EthStateTrie struct {
+	*TrieNode
+}
+
+// Static (compile time) check that EthStateTrie satisfies the node.Node interface.
+var _ node.Node = (*EthStateTrie)(nil)
+
+/*
+  INPUT
+*/
+
+// FromStateTrieRLPFile takes the RLP representation of an ethereum
+// state trie node to return it as an IPLD node for further processing.
+func FromStateTrieRLPFile(r io.Reader) (*EthStateTrie, error) {
+	raw, err := ioutil.ReadAll(r)
+	if err != nil {
+		return nil, err
+	}
+	return FromStateTrieRLP(raw)
+}
+
+// FromStateTrieRLP takes the RLP representation of an ethereum
+// state trie node to return it as an IPLD node for further processing.
+func FromStateTrieRLP(raw []byte) (*EthStateTrie, error) {
+	c, err := RawdataToCid(MEthStateTrie, raw, multihash.KECCAK_256)
+	if err != nil {
+		return nil, err
+	}
+	// Let's run the whole mile and process the nodeKind and
+	// its elements, in case somebody would need this function
+	// to parse an RLP element from the filesystem
+	return DecodeEthStateTrie(c, raw)
+}
+
+/*
+  OUTPUT
+*/
+
+// DecodeEthStateTrie returns an EthStateTrie object from its cid and rawdata.
+func DecodeEthStateTrie(c cid.Cid, b []byte) (*EthStateTrie, error) {
+	tn, err := decodeTrieNode(c, b, decodeEthStateTrieLeaf)
+	if err != nil {
+		return nil, err
+	}
+	return &EthStateTrie{TrieNode: tn}, nil
+}
+
+// decodeEthStateTrieLeaf parses a eth-tx-trie leaf
+// from decoded RLP elements
+func decodeEthStateTrieLeaf(i []interface{}) ([]interface{}, error) {
+	var account EthAccount
+	err := rlp.DecodeBytes(i[1].([]byte), &account)
+	if err != nil {
+		return nil, err
+	}
+	c, err := RawdataToCid(MEthAccountSnapshot, i[1].([]byte), multihash.KECCAK_256)
+	if err != nil {
+		return nil, err
+	}
+	return []interface{}{
+		i[0].([]byte),
+		&EthAccountSnapshot{
+			EthAccount: &account,
+			cid:        c,
+			rawdata:    i[1].([]byte),
+		},
+	}, nil
+}
+
+/*
+  Block INTERFACE
+*/
+
+// RawData returns the binary of the RLP encode of the state trie node.
+func (st *EthStateTrie) RawData() []byte {
+	return st.rawdata
+}
+
+// Cid returns the cid of the state trie node.
+func (st *EthStateTrie) Cid() cid.Cid {
+	return st.cid
+}
+
+// String is a helper for output
+func (st *EthStateTrie) String() string {
+	return fmt.Sprintf("<EthereumStateTrie %s>", st.cid)
+}
+
+// Loggable returns in a map the type of IPLD Link.
+func (st *EthStateTrie) Loggable() map[string]interface{} {
+	return map[string]interface{}{
+		"type": "eth-state-trie",
+	}
+}

statediff/indexer/ipfs/ipld/eth_state_test.go

@@ -0,0 +1,326 @@
+package ipld
+
+import (
+	"fmt"
+	"os"
+	"testing"
+
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+)
+
+/*
+  INPUT
+  OUTPUT
+*/
+
+func TestStateTrieNodeEvenExtensionParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-eb2f5f")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	if output.nodeKind != "extension" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "extension", output.nodeKind)
+	}
+
+	if len(output.elements) != 2 {
+		t.Fatalf("Wrong number of elements for an extension node\r\nexpected %d\r\ngot %d", 2, len(output.elements))
+	}
+
+	if fmt.Sprintf("%x", output.elements[0]) != "0d08" {
+		t.Fatalf("Wrong key\r\nexpected %s\r\ngot %s", "0d08", fmt.Sprintf("%x", output.elements[0]))
+	}
+
+	if output.elements[1].(cid.Cid).String() !=
+		"baglacgzalnzmhhnxudxtga6t3do2rctb6ycgyj6mjnycoamlnc733nnbkd6q" {
+		t.Fatalf("Wrong CID\r\nexpected %s\r\ngot %s", "baglacgzalnzmhhnxudxtga6t3do2rctb6ycgyj6mjnycoamlnc733nnbkd6q", output.elements[1].(cid.Cid).String())
+	}
+}
+
+func TestStateTrieNodeOddExtensionParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-56864f")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	if output.nodeKind != "extension" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "extension", output.nodeKind)
+	}
+
+	if len(output.elements) != 2 {
+		t.Fatalf("Wrong number of elements for an extension node\r\nexpected %d\r\ngot %d", 2, len(output.elements))
+	}
+
+	if fmt.Sprintf("%x", output.elements[0]) != "02" {
+		t.Fatalf("Wrong key\r\nexpected %s\r\ngot %s", "02", fmt.Sprintf("%x", output.elements[0]))
+	}
+
+	if output.elements[1].(cid.Cid).String() !=
+		"baglacgzaizf2czb7wztoox4lu23qkwkbfamqsdzcmejzr3rsszrvkaktpfeq" {
+		t.Fatalf("Wrong CID\r\nexpected %s\r\ngot %s", "baglacgzaizf2czb7wztoox4lu23qkwkbfamqsdzcmejzr3rsszrvkaktpfeq", output.elements[1].(cid.Cid).String())
+	}
+}
+
+func TestStateTrieNodeEvenLeafParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-0e8b34")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	if output.nodeKind != "leaf" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "leaf", output.nodeKind)
+	}
+
+	if len(output.elements) != 2 {
+		t.Fatalf("Wrong number of elements for an extension node\r\nexpected %d\r\ngot %d", 2, len(output.elements))
+	}
+
+	// bd66f60e5b954e1af93ded1b02cb575ff0ed6d9241797eff7576b0bf0637
+	if fmt.Sprintf("%x", output.elements[0].([]byte)[0:10]) != "0b0d06060f06000e050b" {
+		t.Fatalf("Wrong key\r\nexpected %s\r\ngot %s", "0b0d06060f06000e050b", fmt.Sprintf("%x", output.elements[0].([]byte)[0:10]))
+	}
+
+	if output.elements[1].(*EthAccountSnapshot).String() !=
+		"<EthereumAccountSnapshot baglqcgzaf5tapdf2fwb6mo4ijtovqpoi4n3f4jv2yx6avvz6sjypp6vytfva>" {
+		t.Fatalf("Wrong String()\r\nexpected %s\r\ngot %s", "<EthereumAccountSnapshot baglqcgzaf5tapdf2fwb6mo4ijtovqpoi4n3f4jv2yx6avvz6sjypp6vytfva>", output.elements[1].(*EthAccountSnapshot).String())
+	}
+}
+
+func TestStateTrieNodeOddLeafParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-c9070d")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	if output.nodeKind != "leaf" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "leaf", output.nodeKind)
+	}
+
+	if len(output.elements) != 2 {
+		t.Fatalf("Wrong number of elements for an extension node\r\nexpected %d\r\ngot %d", 2, len(output.elements))
+	}
+
+	// 6c9db9bb545a03425e300f3ee72bae098110336dd3eaf48c20a2e5b6865fc
+	if fmt.Sprintf("%x", output.elements[0].([]byte)[0:10]) != "060c090d0b090b0b0504" {
+		t.Fatalf("Wrong key\r\nexpected %s\r\ngot %s", "060c090d0b090b0b0504", fmt.Sprintf("%x", output.elements[0].([]byte)[0:10]))
+	}
+
+	if output.elements[1].(*EthAccountSnapshot).String() !=
+		"<EthereumAccountSnapshot baglqcgzasckx2alxk43cksshnztjvhfyvbbh6bkp376gtcndm5cg4fkrkhsa>" {
+		t.Fatalf("Wrong String()\r\nexpected %s\r\ngot %s", "<EthereumAccountSnapshot baglqcgzasckx2alxk43cksshnztjvhfyvbbh6bkp376gtcndm5cg4fkrkhsa>", output.elements[1].(*EthAccountSnapshot).String())
+	}
+}
+
+/*
+  Block INTERFACE
+*/
+func TestStateTrieBlockElements(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-d7f897")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	if fmt.Sprintf("%x", output.RawData())[:10] != "f90211a090" {
+		t.Fatalf("Wrong Data\r\nexpected %s\r\ngot %s", "f90211a090", fmt.Sprintf("%x", output.RawData())[:10])
+	}
+
+	if output.Cid().String() !=
+		"baglacgza274jot5vvr4ntlajtonnkaml5xbm4cts3liye6qxbhndawapavca" {
+		t.Fatalf("Wrong Cid\r\nexpected %s\r\ngot %s", "baglacgza274jot5vvr4ntlajtonnkaml5xbm4cts3liye6qxbhndawapavca", output.Cid().String())
+	}
+}
+
+func TestStateTrieString(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-d7f897")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	if output.String() !=
+		"<EthereumStateTrie baglacgza274jot5vvr4ntlajtonnkaml5xbm4cts3liye6qxbhndawapavca>" {
+		t.Fatalf("Wrong String()\r\nexpected %s\r\ngot %s", "<EthereumStateTrie baglacgza274jot5vvr4ntlajtonnkaml5xbm4cts3liye6qxbhndawapavca>", output.String())
+	}
+}
+
+func TestStateTrieLoggable(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-d7f897")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	l := output.Loggable()
+	if _, ok := l["type"]; !ok {
+		t.Fatal("Loggable map expected the field 'type'")
+	}
+
+	if l["type"] != "eth-state-trie" {
+		t.Fatalf("Wrong Loggable 'type' value\r\nexpected %s\r\ngot %s", "eth-state-trie", l["type"])
+	}
+}
+
+/*
+  TRIE NODE (Through EthStateTrie)
+  Node INTERFACE
+*/
+
+func TestTraverseStateTrieWithResolve(t *testing.T) {
+	var err error
+
+	stMap := prepareStateTrieMap(t)
+
+	// This is the cid of the root of the block 0
+	// baglacgza274jot5vvr4ntlajtonnkaml5xbm4cts3liye6qxbhndawapavca
+	currentNode := stMap["baglacgza274jot5vvr4ntlajtonnkaml5xbm4cts3liye6qxbhndawapavca"]
+
+	// This is the path we want to traverse
+	// The eth address is 0x5abfec25f74cd88437631a7731906932776356f9
+	// Its keccak-256 is cdd3e25edec0a536a05f5e5ab90a5603624c0ed77453b2e8f955cf8b43d4d0fb
+	// We use the keccak-256(addr) to traverse the state trie in ethereum.
+	var traversePath []string
+	for _, s := range "cdd3e25edec0a536a05f5e5ab90a5603624c0ed77453b2e8f955cf8b43d4d0fb" {
+		traversePath = append(traversePath, string(s))
+	}
+	traversePath = append(traversePath, "balance")
+
+	var obj interface{}
+	for {
+		obj, traversePath, err = currentNode.Resolve(traversePath)
+		link, ok := obj.(*node.Link)
+		if !ok {
+			break
+		}
+		if err != nil {
+			t.Fatal("Error should be nil")
+		}
+
+		currentNode = stMap[link.Cid.String()]
+		if currentNode == nil {
+			t.Fatal("state trie node not found in memory map")
+		}
+	}
+
+	if fmt.Sprintf("%v", obj) != "11901484239480000000000000" {
+		t.Fatalf("Wrong balance value\r\nexpected %s\r\ngot %s", "11901484239480000000000000", fmt.Sprintf("%v", obj))
+	}
+}
+
+func TestStateTrieResolveLinks(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-eb2f5f")
+	checkError(err, t)
+
+	stNode, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	// bad case
+	obj, rest, err := stNode.ResolveLink([]string{"supercalifragilist"})
+	if obj != nil {
+		t.Fatalf("Expected obj to be nil")
+	}
+	if rest != nil {
+		t.Fatal("Expected rest to be nil")
+	}
+	if err.Error() != "invalid path element" {
+		t.Fatalf("Wrong error\r\nexpected %s\r\ngot %s", "invalid path element", err.Error())
+	}
+
+	// good case
+	obj, rest, err = stNode.ResolveLink([]string{"d8"})
+	if obj == nil {
+		t.Fatalf("Expected a not nil obj to be returned")
+	}
+	if rest != nil {
+		t.Fatal("Expected rest to be nil")
+	}
+	if err != nil {
+		t.Fatal("Expected error to be nil")
+	}
+}
+
+func TestStateTrieCopy(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-eb2f5f")
+	checkError(err, t)
+
+	stNode, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	defer func() {
+		r := recover()
+		if r == nil {
+			t.Fatal("Expected panic")
+		}
+		if r != "implement me" {
+			t.Fatalf("Wrong panic message\r\nexpected %s\r\ngot %s", "'implement me'", r)
+		}
+	}()
+
+	_ = stNode.Copy()
+}
+
+func TestStateTrieStat(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-eb2f5f")
+	checkError(err, t)
+
+	stNode, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	obj, err := stNode.Stat()
+	if obj == nil {
+		t.Fatal("Expected a not null object node.NodeStat")
+	}
+
+	if err != nil {
+		t.Fatal("Expected a nil error")
+	}
+}
+
+func TestStateTrieSize(t *testing.T) {
+	fi, err := os.Open("test_data/eth-state-trie-rlp-eb2f5f")
+	checkError(err, t)
+
+	stNode, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	size, err := stNode.Size()
+	if size != uint64(0) {
+		t.Fatalf("Wrong size\r\nexpected %d\r\ngot %d", 0, size)
+	}
+
+	if err != nil {
+		t.Fatal("Expected a nil error")
+	}
+}
+
+func prepareStateTrieMap(t *testing.T) map[string]*EthStateTrie {
+	filepaths := []string{
+		"test_data/eth-state-trie-rlp-0e8b34",
+		"test_data/eth-state-trie-rlp-56864f",
+		"test_data/eth-state-trie-rlp-6fc2d7",
+		"test_data/eth-state-trie-rlp-727994",
+		"test_data/eth-state-trie-rlp-c9070d",
+		"test_data/eth-state-trie-rlp-d5be90",
+		"test_data/eth-state-trie-rlp-d7f897",
+		"test_data/eth-state-trie-rlp-eb2f5f",
+	}
+
+	out := make(map[string]*EthStateTrie)
+
+	for _, fp := range filepaths {
+		fi, err := os.Open(fp)
+		checkError(err, t)
+
+		stateTrieNode, err := FromStateTrieRLPFile(fi)
+		checkError(err, t)
+
+		out[stateTrieNode.Cid().String()] = stateTrieNode
+	}
+
+	return out
+}

statediff/indexer/ipfs/ipld/eth_storage.go

@@ -0,0 +1,112 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+	"github.com/multiformats/go-multihash"
+)
+
+// EthStorageTrie (eth-storage-trie, codec 0x98), represents
+// a node from the storage trie in ethereum.
+type EthStorageTrie struct {
+	*TrieNode
+}
+
+// Static (compile time) check that EthStorageTrie satisfies the node.Node interface.
+var _ node.Node = (*EthStorageTrie)(nil)
+
+/*
+  INPUT
+*/
+
+// FromStorageTrieRLPFile takes the RLP representation of an ethereum
+// storage trie node to return it as an IPLD node for further processing.
+func FromStorageTrieRLPFile(r io.Reader) (*EthStorageTrie, error) {
+	raw, err := ioutil.ReadAll(r)
+	if err != nil {
+		return nil, err
+	}
+	return FromStorageTrieRLP(raw)
+}
+
+// FromStorageTrieRLP takes the RLP representation of an ethereum
+// storage trie node to return it as an IPLD node for further processing.
+func FromStorageTrieRLP(raw []byte) (*EthStorageTrie, error) {
+	c, err := RawdataToCid(MEthStorageTrie, raw, multihash.KECCAK_256)
+	if err != nil {
+		return nil, err
+	}
+
+	// Let's run the whole mile and process the nodeKind and
+	// its elements, in case somebody would need this function
+	// to parse an RLP element from the filesystem
+	return DecodeEthStorageTrie(c, raw)
+}
+
+/*
+  OUTPUT
+*/
+
+// DecodeEthStorageTrie returns an EthStorageTrie object from its cid and rawdata.
+func DecodeEthStorageTrie(c cid.Cid, b []byte) (*EthStorageTrie, error) {
+	tn, err := decodeTrieNode(c, b, decodeEthStorageTrieLeaf)
+	if err != nil {
+		return nil, err
+	}
+	return &EthStorageTrie{TrieNode: tn}, nil
+}
+
+// decodeEthStorageTrieLeaf parses a eth-tx-trie leaf
+// from decoded RLP elements
+func decodeEthStorageTrieLeaf(i []interface{}) ([]interface{}, error) {
+	return []interface{}{
+		i[0].([]byte),
+		i[1].([]byte),
+	}, nil
+}
+
+/*
+  Block INTERFACE
+*/
+
+// RawData returns the binary of the RLP encode of the storage trie node.
+func (st *EthStorageTrie) RawData() []byte {
+	return st.rawdata
+}
+
+// Cid returns the cid of the storage trie node.
+func (st *EthStorageTrie) Cid() cid.Cid {
+	return st.cid
+}
+
+// String is a helper for output
+func (st *EthStorageTrie) String() string {
+	return fmt.Sprintf("<EthereumStorageTrie %s>", st.cid)
+}
+
+// Loggable returns in a map the type of IPLD Link.
+func (st *EthStorageTrie) Loggable() map[string]interface{} {
+	return map[string]interface{}{
+		"type": "eth-storage-trie",
+	}
+}

statediff/indexer/ipfs/ipld/eth_storage_test.go

@@ -0,0 +1,140 @@
+package ipld
+
+import (
+	"fmt"
+	"os"
+	"testing"
+
+	"github.com/ipfs/go-cid"
+)
+
+/*
+  INPUT
+  OUTPUT
+*/
+
+func TestStorageTrieNodeExtensionParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-storage-trie-rlp-113049")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	if output.nodeKind != "extension" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "extension", output.nodeKind)
+	}
+
+	if len(output.elements) != 2 {
+		t.Fatalf("Wrong number of elements for an extension node\r\nexpected %d\r\ngot %d", 2, len(output.elements))
+	}
+
+	if fmt.Sprintf("%x", output.elements[0]) != "0a" {
+		t.Fatalf("Wrong key\r\nexpected %s\r\ngot %s", "0a", fmt.Sprintf("%x", output.elements[0]))
+	}
+
+	if output.elements[1].(cid.Cid).String() !=
+		"baglacgzautxeutufae7owyrezfvwpan2vusocmxgzwqhzrhjbwprp2texgsq" {
+		t.Fatalf("Wrong CID\r\nexpected %s\r\ngot %s", "baglacgzautxeutufae7owyrezfvwpan2vusocmxgzwqhzrhjbwprp2texgsq", output.elements[1].(cid.Cid).String())
+	}
+}
+
+func TestStateTrieNodeLeafParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-storage-trie-rlp-ffbcad")
+	checkError(err, t)
+
+	output, err := FromStorageTrieRLPFile(fi)
+	checkError(err, t)
+
+	if output.nodeKind != "leaf" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "leaf", output.nodeKind)
+	}
+
+	if len(output.elements) != 2 {
+		t.Fatalf("Wrong number of elements for an leaf node\r\nexpected %d\r\ngot %d", 2, len(output.elements))
+	}
+
+	// 2ee1ae9c502e48e0ed528b7b39ac569cef69d7844b5606841a7f3fe898a2
+	if fmt.Sprintf("%x", output.elements[0].([]byte)[:10]) != "020e0e010a0e090c0500" {
+		t.Fatalf("Wrong key\r\nexpected %s\r\ngot %s", "020e0e010a0e090c0500", fmt.Sprintf("%x", output.elements[0].([]byte)[:10]))
+	}
+
+	if fmt.Sprintf("%x", output.elements[1]) != "89056c31f304b2530000" {
+		t.Fatalf("Wrong Value\r\nexpected %s\r\ngot %s", "89056c31f304b2530000", fmt.Sprintf("%x", output.elements[1]))
+	}
+}
+
+func TestStateTrieNodeBranchParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-storage-trie-rlp-ffc25c")
+	checkError(err, t)
+
+	output, err := FromStateTrieRLPFile(fi)
+	checkError(err, t)
+
+	if output.nodeKind != "branch" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "branch", output.nodeKind)
+	}
+
+	if len(output.elements) != 17 {
+		t.Fatalf("Wrong number of elements for an branch node\r\nexpected %d\r\ngot %d", 17, len(output.elements))
+	}
+
+	if fmt.Sprintf("%s", output.elements[4]) !=
+		"baglacgzadqhbmlxrxtw5hplcq5jn74p4dceryzw664w3237ra52dnghbjpva" {
+		t.Fatalf("Wrong Cid\r\nexpected %s\r\ngot %s", "baglacgzadqhbmlxrxtw5hplcq5jn74p4dceryzw664w3237ra52dnghbjpva", fmt.Sprintf("%s", output.elements[4]))
+	}
+
+	if fmt.Sprintf("%s", output.elements[10]) !=
+		"baglacgza77d37i2v6uhtzeeq4vngragjbgbwq3lylpoc3lihenvzimybzxmq" {
+		t.Fatalf("Wrong Cid\r\nexpected %s\r\ngot %s", "baglacgza77d37i2v6uhtzeeq4vngragjbgbwq3lylpoc3lihenvzimybzxmq", fmt.Sprintf("%s", output.elements[10]))
+	}
+}
+
+/*
+  Block INTERFACE
+*/
+func TestStorageTrieBlockElements(t *testing.T) {
+	fi, err := os.Open("test_data/eth-storage-trie-rlp-ffbcad")
+	checkError(err, t)
+
+	output, err := FromStorageTrieRLPFile(fi)
+	checkError(err, t)
+
+	if fmt.Sprintf("%x", output.RawData())[:10] != "eb9f202ee1" {
+		t.Fatalf("Wrong Data\r\nexpected %s\r\ngot %s", "eb9f202ee1", fmt.Sprintf("%x", output.RawData())[:10])
+	}
+
+	if output.Cid().String() !=
+		"bagmacgza766k3oprj2qxn36eycw55pogmu3dwtfay6zdh6ajrhvw3b2nqg5a" {
+		t.Fatalf("Wrong Cid\r\nexpected %s\r\ngot %s", "bagmacgza766k3oprj2qxn36eycw55pogmu3dwtfay6zdh6ajrhvw3b2nqg5a", output.Cid().String())
+	}
+}
+
+func TestStorageTrieString(t *testing.T) {
+	fi, err := os.Open("test_data/eth-storage-trie-rlp-ffbcad")
+	checkError(err, t)
+
+	output, err := FromStorageTrieRLPFile(fi)
+	checkError(err, t)
+
+	if output.String() !=
+		"<EthereumStorageTrie bagmacgza766k3oprj2qxn36eycw55pogmu3dwtfay6zdh6ajrhvw3b2nqg5a>" {
+		t.Fatalf("Wrong String()\r\nexpected %s\r\ngot %s", "<EthereumStorageTrie bagmacgza766k3oprj2qxn36eycw55pogmu3dwtfay6zdh6ajrhvw3b2nqg5a>", output.String())
+	}
+}
+
+func TestStorageTrieLoggable(t *testing.T) {
+	fi, err := os.Open("test_data/eth-storage-trie-rlp-ffbcad")
+	checkError(err, t)
+
+	output, err := FromStorageTrieRLPFile(fi)
+	checkError(err, t)
+
+	l := output.Loggable()
+	if _, ok := l["type"]; !ok {
+		t.Fatal("Loggable map expected the field 'type'")
+	}
+
+	if l["type"] != "eth-storage-trie" {
+		t.Fatalf("Wrong Loggable 'type' value\r\nexpected %s\r\ngot %s", "eth-storage-trie", l["type"])
+	}
+}

statediff/indexer/ipfs/ipld/eth_tx.go

@@ -0,0 +1,236 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"encoding/json"
+	"fmt"
+	"strconv"
+	"strings"
+
+	"github.com/ethereum/go-ethereum/common/hexutil"
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+	mh "github.com/multiformats/go-multihash"
+)
+
+// EthTx (eth-tx codec 0x93) represents an ethereum transaction
+type EthTx struct {
+	*types.Transaction
+
+	cid     cid.Cid
+	rawdata []byte
+}
+
+// Static (compile time) check that EthTx satisfies the node.Node interface.
+var _ node.Node = (*EthTx)(nil)
+
+/*
+  INPUT
+*/
+
+// NewEthTx converts a *types.Transaction to an EthTx IPLD node
+func NewEthTx(tx *types.Transaction) (*EthTx, error) {
+	txRaw, err := tx.MarshalBinary()
+	if err != nil {
+		return nil, err
+	}
+	c, err := RawdataToCid(MEthTx, txRaw, mh.KECCAK_256)
+	if err != nil {
+		return nil, err
+	}
+	return &EthTx{
+		Transaction: tx,
+		cid:         c,
+		rawdata:     txRaw,
+	}, nil
+}
+
+/*
+ OUTPUT
+*/
+
+// DecodeEthTx takes a cid and its raw binary data
+// from IPFS and returns an EthTx object for further processing.
+func DecodeEthTx(c cid.Cid, b []byte) (*EthTx, error) {
+	t := new(types.Transaction)
+	if err := t.UnmarshalBinary(b); err != nil {
+		return nil, err
+	}
+	return &EthTx{
+		Transaction: t,
+		cid:         c,
+		rawdata:     b,
+	}, nil
+}
+
+/*
+  Block INTERFACE
+*/
+
+// RawData returns the binary of the RLP encode of the transaction.
+func (t *EthTx) RawData() []byte {
+	return t.rawdata
+}
+
+// Cid returns the cid of the transaction.
+func (t *EthTx) Cid() cid.Cid {
+	return t.cid
+}
+
+// String is a helper for output
+func (t *EthTx) String() string {
+	return fmt.Sprintf("<EthereumTx %s>", t.cid)
+}
+
+// Loggable returns in a map the type of IPLD Link.
+func (t *EthTx) Loggable() map[string]interface{} {
+	return map[string]interface{}{
+		"type": "eth-tx",
+	}
+}
+
+/*
+  Node INTERFACE
+*/
+
+// Resolve resolves a path through this node, stopping at any link boundary
+// and returning the object found as well as the remaining path to traverse
+func (t *EthTx) Resolve(p []string) (interface{}, []string, error) {
+	if len(p) == 0 {
+		return t, nil, nil
+	}
+
+	if len(p) > 1 {
+		return nil, nil, fmt.Errorf("unexpected path elements past %s", p[0])
+	}
+
+	switch p[0] {
+
+	case "gas":
+		return t.Gas(), nil, nil
+	case "gasPrice":
+		return t.GasPrice(), nil, nil
+	case "input":
+		return fmt.Sprintf("%x", t.Data()), nil, nil
+	case "nonce":
+		return t.Nonce(), nil, nil
+	case "r":
+		_, r, _ := t.RawSignatureValues()
+		return hexutil.EncodeBig(r), nil, nil
+	case "s":
+		_, _, s := t.RawSignatureValues()
+		return hexutil.EncodeBig(s), nil, nil
+	case "toAddress":
+		return t.To(), nil, nil
+	case "v":
+		v, _, _ := t.RawSignatureValues()
+		return hexutil.EncodeBig(v), nil, nil
+	case "value":
+		return hexutil.EncodeBig(t.Value()), nil, nil
+	default:
+		return nil, nil, fmt.Errorf("no such link")
+	}
+}
+
+// Tree lists all paths within the object under 'path', and up to the given depth.
+// To list the entire object (similar to `find .`) pass "" and -1
+func (t *EthTx) Tree(p string, depth int) []string {
+	if p != "" || depth == 0 {
+		return nil
+	}
+	return []string{"gas", "gasPrice", "input", "nonce", "r", "s", "toAddress", "v", "value"}
+}
+
+// ResolveLink is a helper function that calls resolve and asserts the
+// output is a link
+func (t *EthTx) ResolveLink(p []string) (*node.Link, []string, error) {
+	obj, rest, err := t.Resolve(p)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if lnk, ok := obj.(*node.Link); ok {
+		return lnk, rest, nil
+	}
+
+	return nil, nil, fmt.Errorf("resolved item was not a link")
+}
+
+// Copy will go away. It is here to comply with the interface.
+func (t *EthTx) Copy() node.Node {
+	panic("implement me")
+}
+
+// Links is a helper function that returns all links within this object
+func (t *EthTx) Links() []*node.Link {
+	return nil
+}
+
+// Stat will go away. It is here to comply with the interface.
+func (t *EthTx) Stat() (*node.NodeStat, error) {
+	return &node.NodeStat{}, nil
+}
+
+// Size will go away. It is here to comply with the interface. It returns the byte size for the transaction
+func (t *EthTx) Size() (uint64, error) {
+	spl := strings.Split(t.Transaction.Size().String(), " ")
+	size, units := spl[0], spl[1]
+	floatSize, err := strconv.ParseFloat(size, 64)
+	if err != nil {
+		return 0, err
+	}
+	var byteSize uint64
+	switch units {
+	case "B":
+		byteSize = uint64(floatSize)
+	case "KB":
+		byteSize = uint64(floatSize * 1000)
+	case "MB":
+		byteSize = uint64(floatSize * 1000000)
+	case "GB":
+		byteSize = uint64(floatSize * 1000000000)
+	case "TB":
+		byteSize = uint64(floatSize * 1000000000000)
+	default:
+		return 0, fmt.Errorf("unreconginized units %s", units)
+	}
+	return byteSize, nil
+}
+
+/*
+  EthTx functions
+*/
+
+// MarshalJSON processes the transaction into readable JSON format.
+func (t *EthTx) MarshalJSON() ([]byte, error) {
+	v, r, s := t.RawSignatureValues()
+
+	out := map[string]interface{}{
+		"gas":       t.Gas(),
+		"gasPrice":  hexutil.EncodeBig(t.GasPrice()),
+		"input":     fmt.Sprintf("%x", t.Data()),
+		"nonce":     t.Nonce(),
+		"r":         hexutil.EncodeBig(r),
+		"s":         hexutil.EncodeBig(s),
+		"toAddress": t.To(),
+		"v":         hexutil.EncodeBig(v),
+		"value":     hexutil.EncodeBig(t.Value()),
+	}
+	return json.Marshal(out)
+}

statediff/indexer/ipfs/ipld/eth_tx_test.go

@@ -0,0 +1,410 @@
+package ipld
+
+import (
+	"encoding/hex"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+	"testing"
+
+	block "github.com/ipfs/go-block-format"
+	"github.com/multiformats/go-multihash"
+)
+
+/*
+  EthBlock
+  INPUT
+*/
+
+func TestTxInBlockBodyRlpParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-block-body-rlp-999999")
+	checkError(err, t)
+
+	_, output, _, err := FromBlockRLP(fi)
+	checkError(err, t)
+
+	if len(output) != 11 {
+		t.Fatalf("Wrong number of parsed txs\r\nexpected %d\r\ngot %d", 11, len(output))
+	}
+
+	// Oh, let's just grab the last element and one from the middle
+	testTx05Fields(output[5], t)
+	testTx10Fields(output[10], t)
+}
+
+func TestTxInBlockHeaderRlpParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-block-header-rlp-999999")
+	checkError(err, t)
+
+	_, output, _, err := FromBlockRLP(fi)
+	checkError(err, t)
+
+	if len(output) != 0 {
+		t.Fatalf("Wrong number of txs\r\nexpected %d\r\ngot %d", 0, len(output))
+	}
+}
+
+func TestTxInBlockBodyJsonParsing(t *testing.T) {
+	fi, err := os.Open("test_data/eth-block-body-json-999999")
+	checkError(err, t)
+
+	_, output, _, err := FromBlockJSON(fi)
+	checkError(err, t)
+
+	if len(output) != 11 {
+		t.Fatalf("Wrong number of parsed txs\r\nexpected %d\r\ngot %d", 11, len(output))
+	}
+
+	testTx05Fields(output[5], t)
+	testTx10Fields(output[10], t)
+}
+
+/*
+  OUTPUT
+*/
+
+func TestDecodeTransaction(t *testing.T) {
+	// Prepare the "fetched transaction".
+	// This one is supposed to be in the datastore already,
+	// and given away by github.com/ipfs/go-ipfs/merkledag
+	rawTransactionString :=
+		"f86c34850df84758008252089432be343b94f860124dc4fee278fdcbd38c102d88880f25" +
+			"8512af0d4000801ba0e9a25c929c26d1a95232ba75aef419a91b470651eb77614695e16c" +
+			"5ba023e383a0679fb2fc0d0b0f3549967c0894ee7d947f07d238a83ef745bc3ced5143a4af36"
+	rawTransaction, err := hex.DecodeString(rawTransactionString)
+	checkError(err, t)
+	c, err := RawdataToCid(MEthTx, rawTransaction, multihash.KECCAK_256)
+	checkError(err, t)
+
+	// Just to clarify: This `block` is an IPFS block
+	storedTransaction, err := block.NewBlockWithCid(rawTransaction, c)
+	checkError(err, t)
+
+	// Now the proper test
+	ethTransaction, err := DecodeEthTx(storedTransaction.Cid(), storedTransaction.RawData())
+	checkError(err, t)
+
+	testTx05Fields(ethTransaction, t)
+}
+
+/*
+  Block INTERFACE
+*/
+
+func TestEthTxLoggable(t *testing.T) {
+	txs := prepareParsedTxs(t)
+
+	l := txs[0].Loggable()
+	if _, ok := l["type"]; !ok {
+		t.Fatal("Loggable map expected the field 'type'")
+	}
+
+	if l["type"] != "eth-tx" {
+		t.Fatalf("Wrong Loggable 'type' value\r\nexpected %s\r\ngot %s", "eth-tx", l["type"])
+	}
+}
+
+/*
+  Node INTERFACE
+*/
+
+func TestEthTxResolve(t *testing.T) {
+	tx := prepareParsedTxs(t)[0]
+
+	// Empty path
+	obj, rest, err := tx.Resolve([]string{})
+	rtx, ok := obj.(*EthTx)
+	if !ok {
+		t.Fatal("Wrong type of returned object")
+	}
+	if rtx.Cid() != tx.Cid() {
+		t.Fatalf("Wrong CID\r\nexpected %s\r\ngot %s", tx.Cid().String(), rtx.Cid().String())
+	}
+	if rest != nil {
+		t.Fatal("est should be nil")
+	}
+	if err != nil {
+		t.Fatal("err should be nil")
+	}
+
+	// len(p) > 1
+	badCases := [][]string{
+		{"two", "elements"},
+		{"here", "three", "elements"},
+		{"and", "here", "four", "elements"},
+	}
+
+	for _, bc := range badCases {
+		obj, rest, err = tx.Resolve(bc)
+		if obj != nil {
+			t.Fatal("obj should be nil")
+		}
+		if rest != nil {
+			t.Fatal("rest should be nil")
+		}
+		if err.Error() != fmt.Sprintf("unexpected path elements past %s", bc[0]) {
+			t.Fatalf("wrong error\r\nexpected %s\r\ngot %s", fmt.Sprintf("unexpected path elements past %s", bc[0]), err.Error())
+		}
+	}
+
+	moreBadCases := []string{
+		"i",
+		"am",
+		"not",
+		"a",
+		"tx",
+		"field",
+	}
+	for _, mbc := range moreBadCases {
+		obj, rest, err = tx.Resolve([]string{mbc})
+		if obj != nil {
+			t.Fatal("obj should be nil")
+		}
+		if rest != nil {
+			t.Fatal("rest should be nil")
+		}
+		if err.Error() != "no such link" {
+			t.Fatalf("wrong error\r\nexpected %s\r\ngot %s", "no such link", err.Error())
+		}
+	}
+
+	goodCases := []string{
+		"gas",
+		"gasPrice",
+		"input",
+		"nonce",
+		"r",
+		"s",
+		"toAddress",
+		"v",
+		"value",
+	}
+	for _, gc := range goodCases {
+		_, _, err = tx.Resolve([]string{gc})
+		if err != nil {
+			t.Fatalf("error should be nil %v", gc)
+		}
+	}
+
+}
+
+func TestEthTxTree(t *testing.T) {
+	tx := prepareParsedTxs(t)[0]
+	_ = tx
+
+	// Bad cases
+	tree := tx.Tree("non-empty-string", 0)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	tree = tx.Tree("non-empty-string", 1)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	tree = tx.Tree("", 0)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	// Good cases
+	tree = tx.Tree("", 1)
+	lookupElements := map[string]interface{}{
+		"gas":       nil,
+		"gasPrice":  nil,
+		"input":     nil,
+		"nonce":     nil,
+		"r":         nil,
+		"s":         nil,
+		"toAddress": nil,
+		"v":         nil,
+		"value":     nil,
+	}
+
+	if len(tree) != len(lookupElements) {
+		t.Fatalf("Wrong number of elements\r\nexpected %d\r\ngot %d", len(lookupElements), len(tree))
+	}
+
+	for _, te := range tree {
+		if _, ok := lookupElements[te]; !ok {
+			t.Fatalf("Unexpected Element: %v", te)
+		}
+	}
+}
+
+func TestEthTxResolveLink(t *testing.T) {
+	tx := prepareParsedTxs(t)[0]
+
+	// bad case
+	obj, rest, err := tx.ResolveLink([]string{"supercalifragilist"})
+	if obj != nil {
+		t.Fatalf("Expected obj to be nil")
+	}
+	if rest != nil {
+		t.Fatal("Expected rest to be nil")
+	}
+	if err.Error() != "no such link" {
+		t.Fatalf("Wrong error\r\nexpected %s\r\ngot %s", "no such link", err.Error())
+	}
+
+	// good case
+	obj, rest, err = tx.ResolveLink([]string{"nonce"})
+	if obj != nil {
+		t.Fatalf("Expected obj to be nil")
+	}
+	if rest != nil {
+		t.Fatal("Expected rest to be nil")
+	}
+	if err.Error() != "resolved item was not a link" {
+		t.Fatalf("Wrong error\r\nexpected %s\r\ngot %s", "resolved item was not a link", err.Error())
+	}
+}
+
+func TestEthTxCopy(t *testing.T) {
+	tx := prepareParsedTxs(t)[0]
+
+	defer func() {
+		r := recover()
+		if r == nil {
+			t.Fatal("Expected panic")
+		}
+		if r != "implement me" {
+			t.Fatalf("Wrong panic message\r\nexpected %s\r\ngot %s", "'implement me'", r)
+		}
+	}()
+
+	_ = tx.Copy()
+}
+
+func TestEthTxLinks(t *testing.T) {
+	tx := prepareParsedTxs(t)[0]
+
+	if tx.Links() != nil {
+		t.Fatal("Links() expected to return nil")
+	}
+}
+
+func TestEthTxStat(t *testing.T) {
+	tx := prepareParsedTxs(t)[0]
+
+	obj, err := tx.Stat()
+	if obj == nil {
+		t.Fatal("Expected a not null object node.NodeStat")
+	}
+
+	if err != nil {
+		t.Fatal("Expected a nil error")
+	}
+}
+
+func TestEthTxSize(t *testing.T) {
+	tx := prepareParsedTxs(t)[0]
+
+	size, err := tx.Size()
+	checkError(err, t)
+
+	spl := strings.Split(tx.Transaction.Size().String(), " ")
+	expectedSize, units := spl[0], spl[1]
+	floatSize, err := strconv.ParseFloat(expectedSize, 64)
+	checkError(err, t)
+
+	var byteSize uint64
+	switch units {
+	case "B":
+		byteSize = uint64(floatSize)
+	case "KB":
+		byteSize = uint64(floatSize * 1000)
+	case "MB":
+		byteSize = uint64(floatSize * 1000000)
+	case "GB":
+		byteSize = uint64(floatSize * 1000000000)
+	case "TB":
+		byteSize = uint64(floatSize * 1000000000000)
+	default:
+		t.Fatal("Unexpected size units")
+	}
+	if size != byteSize {
+		t.Fatalf("Wrong size\r\nexpected %d\r\ngot %d", byteSize, size)
+	}
+}
+
+/*
+  AUXILIARS
+*/
+
+// prepareParsedTxs is a convenienve method
+func prepareParsedTxs(t *testing.T) []*EthTx {
+	fi, err := os.Open("test_data/eth-block-body-rlp-999999")
+	checkError(err, t)
+
+	_, output, _, err := FromBlockRLP(fi)
+	checkError(err, t)
+
+	return output
+}
+
+func testTx05Fields(ethTx *EthTx, t *testing.T) {
+	// Was the cid calculated?
+	if ethTx.Cid().String() != "bagjqcgzawhfnvdnpmpcfoug7d3tz53k2ht3cidr45pnw3y7snpd46azbpp2a" {
+		t.Fatalf("Wrong cid\r\nexpected %s\r\ngot %s\r\n", "bagjqcgzawhfnvdnpmpcfoug7d3tz53k2ht3cidr45pnw3y7snpd46azbpp2a", ethTx.Cid().String())
+	}
+
+	// Do we have the rawdata available?
+	if fmt.Sprintf("%x", ethTx.RawData()[:10]) != "f86c34850df847580082" {
+		t.Fatalf("Wrong Rawdata\r\nexpected %s\r\ngot %s", "f86c34850df847580082", fmt.Sprintf("%x", ethTx.RawData()[:10]))
+	}
+
+	// Proper Fields of types.Transaction
+	if fmt.Sprintf("%x", ethTx.To()) != "32be343b94f860124dc4fee278fdcbd38c102d88" {
+		t.Fatalf("Wrong Recipient\r\nexpected %s\r\ngot %s", "32be343b94f860124dc4fee278fdcbd38c102d88", fmt.Sprintf("%x", ethTx.To()))
+	}
+	if len(ethTx.Data()) != 0 {
+		t.Fatalf("Wrong len of Data\r\nexpected %d\r\ngot %d", 0, len(ethTx.Data()))
+	}
+	if fmt.Sprintf("%v", ethTx.Gas()) != "21000" {
+		t.Fatalf("Wrong Gas\r\nexpected %s\r\ngot %s", "21000", fmt.Sprintf("%v", ethTx.Gas()))
+	}
+	if fmt.Sprintf("%v", ethTx.Value()) != "1091424800000000000" {
+		t.Fatalf("Wrong Value\r\nexpected %s\r\ngot %s", "1091424800000000000", fmt.Sprintf("%v", ethTx.Value()))
+	}
+	if fmt.Sprintf("%v", ethTx.Nonce()) != "52" {
+		t.Fatalf("Wrong Nonce\r\nexpected %s\r\ngot %s", "52", fmt.Sprintf("%v", ethTx.Nonce()))
+	}
+	if fmt.Sprintf("%v", ethTx.GasPrice()) != "60000000000" {
+		t.Fatalf("Wrong Gas Price\r\nexpected %s\r\ngot %s", "60000000000", fmt.Sprintf("%v", ethTx.GasPrice()))
+	}
+}
+
+func testTx10Fields(ethTx *EthTx, t *testing.T) {
+	// Was the cid calculated?
+	if ethTx.Cid().String() != "bagjqcgzaykakwayoec6j55zmq62cbvmplgf5u5j67affge3ksi4ermgitjoa" {
+		t.Fatalf("Wrong Cid\r\nexpected %s\r\ngot %s", "bagjqcgzaykakwayoec6j55zmq62cbvmplgf5u5j67affge3ksi4ermgitjoa", ethTx.Cid().String())
+	}
+
+	// Do we have the rawdata available?
+	if fmt.Sprintf("%x", ethTx.RawData()[:10]) != "f8708302a120850ba43b" {
+		t.Fatalf("Wrong Rawdata\r\nexpected %s\r\ngot %s", "f8708302a120850ba43b", fmt.Sprintf("%x", ethTx.RawData()[:10]))
+	}
+
+	// Proper Fields of types.Transaction
+	if fmt.Sprintf("%x", ethTx.To()) != "1c51bf013add0857c5d9cf2f71a7f15ca93d4816" {
+		t.Fatalf("Wrong Recipient\r\nexpected %s\r\ngot %s", "1c51bf013add0857c5d9cf2f71a7f15ca93d4816", fmt.Sprintf("%x", ethTx.To()))
+	}
+	if len(ethTx.Data()) != 0 {
+		t.Fatalf("Wrong len of Data\r\nexpected %d\r\ngot %d", 0, len(ethTx.Data()))
+	}
+	if fmt.Sprintf("%v", ethTx.Gas()) != "90000" {
+		t.Fatalf("Wrong Gas\r\nexpected %s\r\ngot %s", "90000", fmt.Sprintf("%v", ethTx.Gas()))
+	}
+	if fmt.Sprintf("%v", ethTx.Value()) != "1049756850000000000" {
+		t.Fatalf("Wrong Value\r\nexpected %s\r\ngot %s", "1049756850000000000", fmt.Sprintf("%v", ethTx.Value()))
+	}
+	if fmt.Sprintf("%v", ethTx.Nonce()) != "172320" {
+		t.Fatalf("Wrong Nonce\r\nexpected %s\r\ngot %s", "172320", fmt.Sprintf("%v", ethTx.Nonce()))
+	}
+	if fmt.Sprintf("%v", ethTx.GasPrice()) != "50000000000" {
+		t.Fatalf("Wrong Gas Price\r\nexpected %s\r\ngot %s", "50000000000", fmt.Sprintf("%v", ethTx.GasPrice()))
+	}
+}

statediff/indexer/ipfs/ipld/eth_tx_trie.go

@@ -0,0 +1,152 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"fmt"
+
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+	"github.com/multiformats/go-multihash"
+
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/rlp"
+)
+
+// EthTxTrie (eth-tx-trie codec 0x92) represents
+// a node from the transaction trie in ethereum.
+type EthTxTrie struct {
+	*TrieNode
+}
+
+// Static (compile time) check that EthTxTrie satisfies the node.Node interface.
+var _ node.Node = (*EthTxTrie)(nil)
+
+/*
+ INPUT
+*/
+
+// To create a proper trie of the eth-tx-trie objects, it is required
+// to input all transactions belonging to a forest in a single step.
+// We are adding the transactions, and creating its trie on
+// block body parsing time.
+
+/*
+  OUTPUT
+*/
+
+// DecodeEthTxTrie returns an EthTxTrie object from its cid and rawdata.
+func DecodeEthTxTrie(c cid.Cid, b []byte) (*EthTxTrie, error) {
+	tn, err := decodeTrieNode(c, b, decodeEthTxTrieLeaf)
+	if err != nil {
+		return nil, err
+	}
+	return &EthTxTrie{TrieNode: tn}, nil
+}
+
+// decodeEthTxTrieLeaf parses a eth-tx-trie leaf
+//from decoded RLP elements
+func decodeEthTxTrieLeaf(i []interface{}) ([]interface{}, error) {
+	var t types.Transaction
+	err := rlp.DecodeBytes(i[1].([]byte), &t)
+	if err != nil {
+		return nil, err
+	}
+	c, err := RawdataToCid(MEthTx, i[1].([]byte), multihash.KECCAK_256)
+	if err != nil {
+		return nil, err
+	}
+	return []interface{}{
+		i[0].([]byte),
+		&EthTx{
+			Transaction: &t,
+			cid:         c,
+			rawdata:     i[1].([]byte),
+		},
+	}, nil
+}
+
+/*
+  Block INTERFACE
+*/
+
+// RawData returns the binary of the RLP encode of the transaction.
+func (t *EthTxTrie) RawData() []byte {
+	return t.rawdata
+}
+
+// Cid returns the cid of the transaction.
+func (t *EthTxTrie) Cid() cid.Cid {
+	return t.cid
+}
+
+// String is a helper for output
+func (t *EthTxTrie) String() string {
+	return fmt.Sprintf("<EthereumTxTrie %s>", t.cid)
+}
+
+// Loggable returns in a map the type of IPLD Link.
+func (t *EthTxTrie) Loggable() map[string]interface{} {
+	return map[string]interface{}{
+		"type": "eth-tx-trie",
+	}
+}
+
+/*
+  EthTxTrie functions
+*/
+
+// txTrie wraps a localTrie for use on the transaction trie.
+type txTrie struct {
+	*localTrie
+}
+
+// newTxTrie initializes and returns a txTrie.
+func newTxTrie() *txTrie {
+	return &txTrie{
+		localTrie: newLocalTrie(),
+	}
+}
+
+// getNodes invokes the localTrie, which computes the root hash of the
+// transaction trie and returns its database keys, to return a slice
+// of EthTxTrie nodes.
+func (tt *txTrie) getNodes() ([]*EthTxTrie, error) {
+	keys, err := tt.getKeys()
+	if err != nil {
+		return nil, err
+	}
+	var out []*EthTxTrie
+
+	for _, k := range keys {
+		rawdata, err := tt.db.Get(k)
+		if err != nil {
+			panic(err)
+		}
+		c, err := RawdataToCid(MEthTxTrie, rawdata, multihash.KECCAK_256)
+		if err != nil {
+			return nil, err
+		}
+		tn := &TrieNode{
+			cid:     c,
+			rawdata: rawdata,
+		}
+		out = append(out, &EthTxTrie{TrieNode: tn})
+	}
+
+	return out, nil
+}

statediff/indexer/ipfs/ipld/eth_tx_trie_test.go

@@ -0,0 +1,505 @@
+package ipld
+
+import (
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"os"
+	"testing"
+
+	block "github.com/ipfs/go-block-format"
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+	"github.com/multiformats/go-multihash"
+)
+
+/*
+  EthBlock
+*/
+
+func TestTxTriesInBlockBodyJSONParsing(t *testing.T) {
+	// HINT: 306 txs
+	// cat test_data/eth-block-body-json-4139497 | jsontool | grep transactionIndex | wc -l
+	// or, https://etherscan.io/block/4139497
+	fi, err := os.Open("test_data/eth-block-body-json-4139497")
+	checkError(err, t)
+
+	_, _, output, err := FromBlockJSON(fi)
+	checkError(err, t)
+	if len(output) != 331 {
+		t.Fatalf("Wrong number of obtained tx trie nodes\r\nexpected %d\r\n got %d", 331, len(output))
+	}
+}
+
+/*
+  OUTPUT
+*/
+
+func TestTxTrieDecodeExtension(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieExtension(t)
+
+	if ethTxTrie.nodeKind != "extension" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "extension", ethTxTrie.nodeKind)
+	}
+
+	if len(ethTxTrie.elements) != 2 {
+		t.Fatalf("Wrong number of elements for an extension node\r\nexpected %d\r\ngot %d", 2, len(ethTxTrie.elements))
+	}
+
+	if fmt.Sprintf("%x", ethTxTrie.elements[0].([]byte)) != "0001" {
+		t.Fatalf("Wrong key\r\nexpected %s\r\ngot %s", "0001", fmt.Sprintf("%x", ethTxTrie.elements[0].([]byte)))
+	}
+
+	if ethTxTrie.elements[1].(cid.Cid).String() !=
+		"bagjacgzak6wdjvshdtb7lrvlteweyd7f5qjr3dmzmh7g2xpi4xrwoujsio2a" {
+		t.Fatalf("Wrong CID\r\nexpected %s\r\ngot %s", "bagjacgzak6wdjvshdtb7lrvlteweyd7f5qjr3dmzmh7g2xpi4xrwoujsio2a", ethTxTrie.elements[1].(cid.Cid).String())
+	}
+}
+
+func TestTxTrieDecodeLeaf(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieLeaf(t)
+
+	if ethTxTrie.nodeKind != "leaf" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "leaf", ethTxTrie.nodeKind)
+	}
+
+	if len(ethTxTrie.elements) != 2 {
+		t.Fatalf("Wrong number of elements for a leaf node\r\nexpected %d\r\ngot %d", 2, len(ethTxTrie.elements))
+	}
+
+	if fmt.Sprintf("%x", ethTxTrie.elements[0].([]byte)) != "" {
+		t.Fatalf("Wrong key\r\nexpected %s\r\ngot %s", "", fmt.Sprintf("%x", ethTxTrie.elements[0].([]byte)))
+	}
+
+	if _, ok := ethTxTrie.elements[1].(*EthTx); !ok {
+		t.Fatal("Expected element to be an EthTx")
+	}
+
+	if ethTxTrie.elements[1].(*EthTx).String() !=
+		"<EthereumTx bagjqcgzaqsbvff5xrqh5lobxmhuharvkqdc4jmsqfalsu2xs4pbyix7dvfzq>" {
+		t.Fatalf("Wrong String()\r\nexpected %s\r\ngot %s", "<EthereumTx bagjqcgzaqsbvff5xrqh5lobxmhuharvkqdc4jmsqfalsu2xs4pbyix7dvfzq>", ethTxTrie.elements[1].(*EthTx).String())
+	}
+}
+
+func TestTxTrieDecodeBranch(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieBranch(t)
+
+	if ethTxTrie.nodeKind != "branch" {
+		t.Fatalf("Wrong nodeKind\r\nexpected %s\r\ngot %s", "branch", ethTxTrie.nodeKind)
+	}
+
+	if len(ethTxTrie.elements) != 17 {
+		t.Fatalf("Wrong number of elements for a branch node\r\nexpected %d\r\ngot %d", 17, len(ethTxTrie.elements))
+	}
+
+	for i, element := range ethTxTrie.elements {
+		switch {
+		case i < 9:
+			if _, ok := element.(cid.Cid); !ok {
+				t.Fatal("Expected element to be a cid")
+			}
+			continue
+		default:
+			if element != nil {
+				t.Fatal("Expected element to be a nil")
+			}
+		}
+	}
+}
+
+/*
+  Block INTERFACE
+*/
+
+func TestEthTxTrieBlockElements(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieExtension(t)
+
+	if fmt.Sprintf("%x", ethTxTrie.RawData())[:10] != "e4820001a0" {
+		t.Fatalf("Wrong Data\r\nexpected %s\r\ngot %s", "e4820001a0", fmt.Sprintf("%x", ethTxTrie.RawData())[:10])
+	}
+
+	if ethTxTrie.Cid().String() !=
+		"bagjacgzaw6ccgrfc3qnrl6joodbjjiet4haufnt2xww725luwgfhijnmg36q" {
+		t.Fatalf("Wrong Cid\r\nexpected %s\r\ngot %s", "bagjacgzaw6ccgrfc3qnrl6joodbjjiet4haufnt2xww725luwgfhijnmg36q", ethTxTrie.Cid().String())
+	}
+}
+
+func TestEthTxTrieString(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieExtension(t)
+
+	if ethTxTrie.String() != "<EthereumTxTrie bagjacgzaw6ccgrfc3qnrl6joodbjjiet4haufnt2xww725luwgfhijnmg36q>" {
+		t.Fatalf("Wrong String()\r\nexpected %s\r\ngot %s", "<EthereumTxTrie bagjacgzaw6ccgrfc3qnrl6joodbjjiet4haufnt2xww725luwgfhijnmg36q>", ethTxTrie.String())
+	}
+}
+
+func TestEthTxTrieLoggable(t *testing.T) {
+
+	ethTxTrie := prepareDecodedEthTxTrieExtension(t)
+	l := ethTxTrie.Loggable()
+	if _, ok := l["type"]; !ok {
+		t.Fatal("Loggable map expected the field 'type'")
+	}
+
+	if l["type"] != "eth-tx-trie" {
+		t.Fatalf("Wrong Loggable 'type' value\r\nexpected %s\r\ngot %s", "eth-tx-trie", l["type"])
+	}
+}
+
+/*
+  Node INTERFACE
+*/
+
+func TestTxTrieResolveExtension(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieExtension(t)
+
+	_ = ethTxTrie
+}
+
+func TestTxTrieResolveLeaf(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieLeaf(t)
+
+	_ = ethTxTrie
+}
+
+func TestTxTrieResolveBranch(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieBranch(t)
+
+	indexes := []string{"0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "d", "e", "f"}
+
+	for j, index := range indexes {
+		obj, rest, err := ethTxTrie.Resolve([]string{index, "nonce"})
+
+		switch {
+		case j < 9:
+			_, ok := obj.(*node.Link)
+			if !ok {
+				t.Fatalf("Returned object is not a link (index: %d)", j)
+			}
+
+			if rest[0] != "nonce" {
+				t.Fatalf("Wrong rest of the path returned\r\nexpected %s\r\ngot %s", "nonce", rest[0])
+			}
+
+			if err != nil {
+				t.Fatal("Error should be nil")
+			}
+
+		default:
+			if obj != nil {
+				t.Fatalf("Returned object should have been nil")
+			}
+
+			if rest != nil {
+				t.Fatalf("Rest of the path returned should be nil")
+			}
+
+			if err.Error() != "no such link in this branch" {
+				t.Fatalf("Wrong error")
+			}
+		}
+	}
+
+	otherSuccessCases := [][]string{
+		{"0", "1", "banana"},
+		{"1", "banana"},
+		{"7bc", "def"},
+		{"bc", "def"},
+	}
+
+	for i := 0; i < len(otherSuccessCases); i = i + 2 {
+		osc := otherSuccessCases[i]
+		expectedRest := otherSuccessCases[i+1]
+
+		obj, rest, err := ethTxTrie.Resolve(osc)
+		_, ok := obj.(*node.Link)
+		if !ok {
+			t.Fatalf("Returned object is not a link")
+		}
+
+		for j, _ := range expectedRest {
+			if rest[j] != expectedRest[j] {
+				t.Fatalf("Wrong rest of the path returned\r\nexpected %s\r\ngot %s", expectedRest[j], rest[j])
+			}
+		}
+
+		if err != nil {
+			t.Fatal("Error should be nil")
+		}
+
+	}
+}
+
+func TestTraverseTxTrieWithResolve(t *testing.T) {
+	var err error
+
+	txMap := prepareTxTrieMap(t)
+
+	// This is the cid of the tx root at the block 4,139,497
+	currentNode := txMap["bagjacgzaqolvvlyflkdiylijcu4ts6myxczkb2y3ewxmln5oyrsrkfc4v7ua"]
+
+	// This is the path we want to traverse
+	// the transaction id 256, which is RLP encoded to 820100
+	var traversePath []string
+	for _, s := range "820100" {
+		traversePath = append(traversePath, string(s))
+	}
+	traversePath = append(traversePath, "value")
+
+	var obj interface{}
+	for {
+		obj, traversePath, err = currentNode.Resolve(traversePath)
+		link, ok := obj.(*node.Link)
+		if !ok {
+			break
+		}
+		if err != nil {
+			t.Fatal("Error should be nil")
+		}
+
+		currentNode = txMap[link.Cid.String()]
+		if currentNode == nil {
+			t.Fatal("transaction trie node not found in memory map")
+		}
+	}
+
+	if fmt.Sprintf("%v", obj) != "0xc495a958603400" {
+		t.Fatalf("Wrong value\r\nexpected %s\r\ngot %s", "0xc495a958603400", fmt.Sprintf("%v", obj))
+	}
+}
+
+func TestTxTrieTreeBadParams(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieBranch(t)
+
+	tree := ethTxTrie.Tree("non-empty-string", 0)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	tree = ethTxTrie.Tree("non-empty-string", 1)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+
+	tree = ethTxTrie.Tree("", 0)
+	if tree != nil {
+		t.Fatal("Expected nil to be returned")
+	}
+}
+
+func TestTxTrieTreeExtension(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieExtension(t)
+
+	tree := ethTxTrie.Tree("", -1)
+
+	if len(tree) != 1 {
+		t.Fatalf("An extension should have one element")
+	}
+
+	if tree[0] != "01" {
+		t.Fatalf("Wrong trie element\r\nexpected %s\r\ngot %s", "01", tree[0])
+	}
+}
+
+func TestTxTrieTreeBranch(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieBranch(t)
+
+	tree := ethTxTrie.Tree("", -1)
+
+	lookupElements := map[string]interface{}{
+		"0": nil,
+		"1": nil,
+		"2": nil,
+		"3": nil,
+		"4": nil,
+		"5": nil,
+		"6": nil,
+		"7": nil,
+		"8": nil,
+	}
+
+	if len(tree) != len(lookupElements) {
+		t.Fatalf("Wrong number of elements\r\nexpected %d\r\ngot %d", len(lookupElements), len(tree))
+	}
+
+	for _, te := range tree {
+		if _, ok := lookupElements[te]; !ok {
+			t.Fatalf("Unexpected Element: %v", te)
+		}
+	}
+}
+
+func TestTxTrieLinksBranch(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieBranch(t)
+
+	desiredValues := []string{
+		"bagjacgzakhtcfpja453ydiaqxgidqmxhh7jwmxujib663deebwfs3m2n3hoa",
+		"bagjacgza2p2fuqh4vumknq6x5w7i47usvtu5ixqins6qjjtcks4zge3vx3qq",
+		"bagjacgza4fkhn7et3ra66yjkzbtvbxjefuketda6jctlut6it7gfahxhywga",
+		"bagjacgzacnryeybs52xryrka5uxi4eg4hi2mh66esaghu7cetzu6fsukrynq",
+		"bagjacgzastu5tc7lwz4ap3gznjwkyyepswquub7gvhags5mgdyfynnwbi43a",
+		"bagjacgza5qgp76ovvorkydni2lchew6ieu5wb55w6hdliiu6vft7zlxtdhjq",
+		"bagjacgzafnssc4yvln6zxmks5roskw4ckngta5n4yfy2skhlu435ve4b575a",
+		"bagjacgzagkuei7qxfxefufme2d3xizxokkq4ad3rzl2x4dq2uao6dcr4va2a",
+		"bagjacgzaxpaehtananrdxjghwukh2wwkkzcqwveppf6xclkrtd26rm27kqwq",
+	}
+
+	links := ethTxTrie.Links()
+
+	for i, v := range desiredValues {
+		if links[i].Cid.String() != v {
+			t.Fatalf("Wrong cid for link %d\r\nexpected %s\r\ngot %s", i, v, links[i].Cid.String())
+		}
+	}
+}
+
+/*
+  EthTxTrie Functions
+*/
+
+func TestTxTrieJSONMarshalExtension(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieExtension(t)
+
+	jsonOutput, err := ethTxTrie.MarshalJSON()
+	checkError(err, t)
+
+	var data map[string]interface{}
+	err = json.Unmarshal(jsonOutput, &data)
+	checkError(err, t)
+
+	if parseMapElement(data["01"]) !=
+		"bagjacgzak6wdjvshdtb7lrvlteweyd7f5qjr3dmzmh7g2xpi4xrwoujsio2a" {
+		t.Fatalf("Wrong Marshaled Value\r\nexpected %s\r\ngot %s", "bagjacgzak6wdjvshdtb7lrvlteweyd7f5qjr3dmzmh7g2xpi4xrwoujsio2a", parseMapElement(data["01"]))
+	}
+
+	if data["type"] != "extension" {
+		t.Fatalf("Wrong node type\r\nexpected %s\r\ngot %s", "extension", data["type"])
+	}
+}
+
+func TestTxTrieJSONMarshalLeaf(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieLeaf(t)
+
+	jsonOutput, err := ethTxTrie.MarshalJSON()
+	checkError(err, t)
+
+	var data map[string]interface{}
+	err = json.Unmarshal(jsonOutput, &data)
+	checkError(err, t)
+
+	if data["type"] != "leaf" {
+		t.Fatalf("Wrong node type\r\nexpected %s\r\ngot %s", "leaf", data["type"])
+	}
+
+	if fmt.Sprintf("%v", data[""].(map[string]interface{})["nonce"]) !=
+		"40243" {
+		t.Fatalf("Wrong nonce value\r\nexepcted %s\r\ngot %s", "40243", fmt.Sprintf("%v", data[""].(map[string]interface{})["nonce"]))
+	}
+}
+
+func TestTxTrieJSONMarshalBranch(t *testing.T) {
+	ethTxTrie := prepareDecodedEthTxTrieBranch(t)
+
+	jsonOutput, err := ethTxTrie.MarshalJSON()
+	checkError(err, t)
+
+	var data map[string]interface{}
+	err = json.Unmarshal(jsonOutput, &data)
+	checkError(err, t)
+
+	desiredValues := map[string]string{
+		"0": "bagjacgzakhtcfpja453ydiaqxgidqmxhh7jwmxujib663deebwfs3m2n3hoa",
+		"1": "bagjacgza2p2fuqh4vumknq6x5w7i47usvtu5ixqins6qjjtcks4zge3vx3qq",
+		"2": "bagjacgza4fkhn7et3ra66yjkzbtvbxjefuketda6jctlut6it7gfahxhywga",
+		"3": "bagjacgzacnryeybs52xryrka5uxi4eg4hi2mh66esaghu7cetzu6fsukrynq",
+		"4": "bagjacgzastu5tc7lwz4ap3gznjwkyyepswquub7gvhags5mgdyfynnwbi43a",
+		"5": "bagjacgza5qgp76ovvorkydni2lchew6ieu5wb55w6hdliiu6vft7zlxtdhjq",
+		"6": "bagjacgzafnssc4yvln6zxmks5roskw4ckngta5n4yfy2skhlu435ve4b575a",
+		"7": "bagjacgzagkuei7qxfxefufme2d3xizxokkq4ad3rzl2x4dq2uao6dcr4va2a",
+		"8": "bagjacgzaxpaehtananrdxjghwukh2wwkkzcqwveppf6xclkrtd26rm27kqwq",
+	}
+
+	for k, v := range desiredValues {
+		if parseMapElement(data[k]) != v {
+			t.Fatalf("Wrong Marshaled Value %s\r\nexpected %s\r\ngot %s", k, v, parseMapElement(data[k]))
+		}
+	}
+
+	for _, v := range []string{"a", "b", "c", "d", "e", "f"} {
+		if data[v] != nil {
+			t.Fatal("Expected value to be nil")
+		}
+	}
+
+	if data["type"] != "branch" {
+		t.Fatalf("Wrong node type\r\nexpected %s\r\ngot %s", "branch", data["type"])
+	}
+}
+
+/*
+  AUXILIARS
+*/
+
+// prepareDecodedEthTxTrie simulates an IPLD block available in the datastore,
+// checks the source RLP and tests for the absence of errors during the decoding fase.
+func prepareDecodedEthTxTrie(branchDataRLP string, t *testing.T) *EthTxTrie {
+	b, err := hex.DecodeString(branchDataRLP)
+	checkError(err, t)
+
+	c, err := RawdataToCid(MEthTxTrie, b, multihash.KECCAK_256)
+	checkError(err, t)
+
+	storedEthTxTrie, err := block.NewBlockWithCid(b, c)
+	checkError(err, t)
+
+	ethTxTrie, err := DecodeEthTxTrie(storedEthTxTrie.Cid(), storedEthTxTrie.RawData())
+	checkError(err, t)
+
+	return ethTxTrie
+}
+
+func prepareDecodedEthTxTrieExtension(t *testing.T) *EthTxTrie {
+	extensionDataRLP :=
+		"e4820001a057ac34d6471cc3f5c6ab992c4c0fe5ec131d8d9961fe6d5de8e5e367513243b4"
+	return prepareDecodedEthTxTrie(extensionDataRLP, t)
+}
+
+func prepareDecodedEthTxTrieLeaf(t *testing.T) *EthTxTrie {
+	leafDataRLP :=
+		"f87220b86ff86d829d3384ee6b280083015f9094e0e6c781b8cba08bc840" +
+			"7eac0101b668d1fa6f4987c495a9586034008026a0981b6223c9d3c31971" +
+			"6da3cf057da84acf0fef897f4003d8a362d7bda42247dba066be134c4bc4" +
+			"32125209b5056ef274b7423bcac7cc398cf60b83aaff7b95469f"
+	return prepareDecodedEthTxTrie(leafDataRLP, t)
+}
+
+func prepareDecodedEthTxTrieBranch(t *testing.T) *EthTxTrie {
+	branchDataRLP :=
+		"f90131a051e622bd20e77781a010b9903832e73fd3665e89407ded8c840d8b2db34dd9" +
+			"dca0d3f45a40fcad18a6c3d7edbe8e7e92ace9d45e086cbd04a66254b9931375bee1a0" +
+			"e15476fc93dc41ef612ac86750dd242d14498c1e48a6ba4fc89fcc501ee7c58ca01363" +
+			"826032eeaf1c4540ed2e8e10dc3a34c3fbc4900c7a7c449e69e2ca8a8e1ba094e9d98b" +
+			"ebb67807ecd96a6cac608f95a14a07e6a9c06975861e0b86b6c14736a0ec0cfff9d5ab" +
+			"a2ac0da8d2c4725bc8253b60f7b6f1c6b4229ea967fcaef319d3a02b652173155b7d9b" +
+			"b152ec5d255b82534d3075bcc171a928eba737da9381effaa032a8447e172dc85a1584" +
+			"d0f77466ee52a1c00f71caf57e0e1aa01de18a3ca834a0bbc043cc0d03623ba4c7b514" +
+			"7d5aca56450b548f797d712d5198f5e8b35f542d8080808080808080"
+	return prepareDecodedEthTxTrie(branchDataRLP, t)
+}
+
+func prepareTxTrieMap(t *testing.T) map[string]*EthTxTrie {
+	fi, err := os.Open("test_data/eth-block-body-json-4139497")
+	checkError(err, t)
+
+	_, _, txTrieNodes, err := FromBlockJSON(fi)
+	checkError(err, t)
+
+	out := make(map[string]*EthTxTrie)
+
+	for _, txTrieNode := range txTrieNodes {
+		decodedNode, err := DecodeEthTxTrie(txTrieNode.Cid(), txTrieNode.RawData())
+		checkError(err, t)
+		out[txTrieNode.Cid().String()] = decodedNode
+	}
+
+	return out
+}

statediff/indexer/ipfs/ipld/shared.go

@@ -0,0 +1,159 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"bytes"
+
+	"github.com/ipfs/go-cid"
+	mh "github.com/multiformats/go-multihash"
+
+	"github.com/ethereum/go-ethereum/common"
+	"github.com/ethereum/go-ethereum/core/rawdb"
+	"github.com/ethereum/go-ethereum/ethdb"
+	"github.com/ethereum/go-ethereum/rlp"
+	"github.com/ethereum/go-ethereum/trie"
+)
+
+// IPLD Codecs for Ethereum
+// See the authoritative document:
+// https://github.com/multiformats/multicodec/blob/master/table.csv
+const (
+	RawBinary           = 0x55
+	MEthHeader          = 0x90
+	MEthHeaderList      = 0x91
+	MEthTxTrie          = 0x92
+	MEthTx              = 0x93
+	MEthTxReceiptTrie   = 0x94
+	MEthTxReceipt       = 0x95
+	MEthStateTrie       = 0x96
+	MEthAccountSnapshot = 0x97
+	MEthStorageTrie     = 0x98
+)
+
+var (
+	nullHashBytes = common.Hex2Bytes("0000000000000000000000000000000000000000000000000000000000000000")
+)
+
+// RawdataToCid takes the desired codec and a slice of bytes
+// and returns the proper cid of the object.
+func RawdataToCid(codec uint64, rawdata []byte, multiHash uint64) (cid.Cid, error) {
+	c, err := cid.Prefix{
+		Codec:    codec,
+		Version:  1,
+		MhType:   multiHash,
+		MhLength: -1,
+	}.Sum(rawdata)
+	if err != nil {
+		return cid.Cid{}, err
+	}
+	return c, nil
+}
+
+// keccak256ToCid takes a keccak256 hash and returns its cid based on
+// the codec given.
+func keccak256ToCid(codec uint64, h []byte) cid.Cid {
+	buf, err := mh.Encode(h, mh.KECCAK_256)
+	if err != nil {
+		panic(err)
+	}
+
+	return cid.NewCidV1(codec, mh.Multihash(buf))
+}
+
+// commonHashToCid takes a go-ethereum common.Hash and returns its
+// cid based on the codec given,
+func commonHashToCid(codec uint64, h common.Hash) cid.Cid {
+	mhash, err := mh.Encode(h[:], mh.KECCAK_256)
+	if err != nil {
+		panic(err)
+	}
+
+	return cid.NewCidV1(codec, mhash)
+}
+
+// localTrie wraps a go-ethereum trie and its underlying memory db.
+// It contributes to the creation of the trie node objects.
+type localTrie struct {
+	db     ethdb.Database
+	trieDB *trie.Database
+	trie   *trie.Trie
+}
+
+// newLocalTrie initializes and returns a localTrie object
+func newLocalTrie() *localTrie {
+	var err error
+	lt := &localTrie{}
+	lt.db = rawdb.NewMemoryDatabase()
+	lt.trieDB = trie.NewDatabase(lt.db)
+	lt.trie, err = trie.New(common.Hash{}, lt.trieDB)
+	if err != nil {
+		panic(err)
+	}
+	return lt
+}
+
+// add receives the index of an object and its rawdata value
+// and includes it into the localTrie
+func (lt *localTrie) add(idx int, rawdata []byte) error {
+	key, err := rlp.EncodeToBytes(uint(idx))
+	if err != nil {
+		panic(err)
+	}
+	return lt.trie.TryUpdate(key, rawdata)
+}
+
+// rootHash returns the computed trie root.
+// Useful for sanity checks on parsed data.
+func (lt *localTrie) rootHash() []byte {
+	return lt.trie.Hash().Bytes()
+}
+
+// getKeys returns the stored keys of the memory database
+// of the localTrie for further processing.
+func (lt *localTrie) getKeys() ([][]byte, error) {
+	// commit trie nodes to trieDB
+	var err error
+	_, err = lt.trie.Commit(nil)
+	if err != nil {
+		return nil, err
+	}
+	// commit trieDB to the underlying ethdb.Database
+	if err := lt.trieDB.Commit(lt.trie.Hash(), false, nil); err != nil {
+		return nil, err
+	}
+	// collect all of the node keys
+	it := lt.trie.NodeIterator([]byte{})
+	keyBytes := make([][]byte, 0)
+	for it.Next(true) {
+		if it.Leaf() || bytes.Equal(nullHashBytes, it.Hash().Bytes()) {
+			continue
+		}
+		keyBytes = append(keyBytes, it.Hash().Bytes())
+	}
+	return keyBytes, nil
+}
+
+// getRLP encodes the given object to RLP returning its bytes.
+func getRLP(object interface{}) []byte {
+	buf := new(bytes.Buffer)
+	if err := rlp.Encode(buf, object); err != nil {
+		panic(err)
+	}
+
+	return buf.Bytes()
+}

statediff/indexer/ipfs/ipld/test_data/eth-block-body-json-0

@@ -0,0 +1 @@
+{"jsonrpc":"2.0","id":1,"result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x400000000","extraData":"0x11bbe8db4e347b4e8c937c1c8370e4b5ed33adb3db69cbdb7a38e1e50b1b82fa","gasLimit":"0x1388","gasUsed":"0x0","hash":"0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","mixHash":"0x0000000000000000000000000000000000000000000000000000000000000000","nonce":"0x0000000000000042","number":"0x0","parentHash":"0x0000000000000000000000000000000000000000000000000000000000000000","receiptsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000042"],"sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":"0x21c","stateRoot":"0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544","timestamp":"0x0","totalDifficulty":"0x400000000","transactions":[],"transactionsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","uncles":[]}}

statediff/indexer/ipfs/ipld/test_data/eth-block-body-json-997522

@@ -0,0 +1 @@
+{"jsonrpc":"2.0","result":{"author":"0x4bb96091ee9d802ed039c4d1a5f6216f90f81b01","difficulty":"0xae22b2113ed","extraData":"0xd783010400844765746887676f312e352e31856c696e7578","gasLimit":"0x2fefd8","gasUsed":"0x5208","hash":"0x79851e1adb52a8c5490da2df5d8c060b1cc44a3b6eeaada2e20edba5a8e84523","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x4bb96091ee9d802ed039c4d1a5f6216f90f81b01","mixHash":"0x2565992ba4dbd7ab3bb08d1da34051ae1d90c79bc637a21aa2f51f6380bf5f6a","nonce":"0xf7a14147c2320b2d","number":"0xf3892","parentHash":"0x8ad6d5cbe7ec75ed71d5153dd58f2fd413b17c398ad2a7d9309459ce884e6c9b","receiptsRoot":"0xa73a95d90de29c66220c8b8da825cf34ae969efc7f9a878d8ed893565e4b4676","sealFields":["0xa02565992ba4dbd7ab3bb08d1da34051ae1d90c79bc637a21aa2f51f6380bf5f6a","0x88f7a14147c2320b2d"],"sha3Uncles":"0x08793b633d0b21b980107f3e3277c6693f2f3739e0c676a238cbe24d9ae6e252","size":"0x6c0","stateRoot":"0x11e5ea49ecbee25a9b8f267492a5d296ac09cf6179b43bc334242d052bac5963","timestamp":"0x56bf10c5","totalDifficulty":"0x629a0a89232bcd5b","transactions":[{"blockHash":"0x79851e1adb52a8c5490da2df5d8c060b1cc44a3b6eeaada2e20edba5a8e84523","blockNumber":"0xf3892","condition":null,"creates":null,"from":"0x4bb96091ee9d802ed039c4d1a5f6216f90f81b01","gas":"0x15f90","gasPrice":"0xa","hash":"0xd0fc6b051f16468862c462c672532427efef537ea3737b25b10716949d0e2228","input":"0x","networkId":null,"nonce":"0x7c37","publicKey":"0xa9177f27b99a4ad938359d77e0dca4b64e7ce3722c835d8087d4eecb27c8a54d59e2917e6b31ec12e44b1064d102d35815f9707af9571f15e92d1b6fbcd207e9","r":"0x76933e91718154f18db2e993bc96e82abd9a0fac2bae284875341cbecafa837b","raw":"0xf86a827c370a83015f909404a6c6a293340fc3f2244d097b0cfd84d5317ba58844b1eec616322c1c801ba076933e91718154f18db2e993bc96e82abd9a0fac2bae284875341cbecafa837ba02f165c2c4b5f4b786a95e106c48bccc7e065647af5a1942025b6fbfafeabbbf6","s":"0x2f165c2c4b5f4b786a95e106c48bccc7e065647af5a1942025b6fbfafeabbbf6","standardV":"0x0","to":"0x04a6c6a293340fc3f2244d097b0cfd84d5317ba5","transactionIndex":"0x0","v":"0x1b","value":"0x44b1eec616322c1c"}],"transactionsRoot":"0x7ab22cfcf6db5d1628ac888c25e6bc49aba2faaa200fc880f800f1db1e8bd3cc","uncles":["0x319e0dc9a53711579c4ba88062c927a0045443cca57625903ef471d760506a94","0x0324272e484e509c3c9e9e75ad8b48c7d34556e6b269dd72331033fd5cdc1b2a"]},"id":1}

statediff/indexer/ipfs/ipld/test_data/eth-state-trie-rlp-56864f

@@ -0,0 +1 @@
+â FKˇd?¶fç_‹¦·YA(	"a<13>î2–cU
SyI

statediff/indexer/ipfs/ipld/test_data/eth-state-trie-rlp-6fc2d7

@@ -0,0 +1,5 @@
+ù Úä<C39A>[G“(»o½Uå,ÔrBÇõSsµ^²€^âä©) 7ó7ì€.Ytâç5_ñ¤ƒ+9¸FÙÃYzFv Ú<C2A0>b{¸ûî­³à¨äõ(Û1Y¶«-í¤©÷Ê<C3B7>*µ —f&HÕ‚•ÐЪK€UX<55> v• R€%IÙJ/ ÌÇ
ïä³A?Ö¦lŸ@éU¯wFI¨Ùý!-jZ9Ý»g Öͳ.+Ö5î/ž¼”ݽ	±À<>fbŽfzìW [‰ =É@æúpì­N
ÐIÓ¥º ¨ùÀRRVíIŸ ¸B'Ô<>öŠìÇr“šY¯©á¤«W<C2AB>{i‹Û‰â›`DfŽ ý™ p¹JÎW䌿e¡j§pÆEùõﺇ»å<C2BB>
+) áj|ΦtŠé
é/Šï;=ÂH¥W¹¬N)i41?$÷üí_ B7<ô 0ÙMé
+#¸óŒík|¸¸’_î<5F>*(¢Z _‰ÒAÿBˆd÷ˆ˜fHLïb-å:Fç•ßÞÃ61Ÿ u— fE&ÈǕ΢{‹rE\IeqàEURÛÀhź1 Õ¾<C395>‰
/Ú,XZ–˜Ž¥ïÍ:˜Ž
+†‚	iK7Å ÷°5.8ò١MQºêMÞáw	tÈâ	5R3ÃÈ<C383>În I¿n<C2BF>ð¬¯Ðïømïî³VŽDÕ-"5Ï4
+á\`4â²A€

statediff/indexer/ipfs/ipld/test_data/eth-storage-trie-rlp-113049

@@ -0,0 +1 @@
+β ¤ξJN…
>λb$Ιkg<6B>Ί­$α2ζΝ |Δι
<0A>κdΉ¥

statediff/indexer/ipfs/ipld/test_data/eth-storage-trie-rlp-9d1860

@@ -0,0 +1 @@
+<EFBFBD>

statediff/indexer/ipfs/ipld/test_data/eth-storage-trie-rlp-ffc25c

@@ -0,0 +1 @@
+<EFBFBD>Q<EFBFBD><EFBFBD><EFBFBD><EFBFBD> .ّ<>ٍس<D98D>b<EFBFBD>R<EFBFBD>ّ<EFBFBD><18>f<><66>-<2D>oّt6<74>فKي<4B><D98A><EFBFBD><EFBFBD><EFBFBD> <EFBFBD>ا؟<D8A7>U<EFBFBD><<3C><>مZh<5A>ة	<09>hmx[<5B>-­#k<>3
حع<D8AD><D8B9><EFBFBD><EFBFBD><EFBFBD><EFBFBD>

statediff/indexer/ipfs/ipld/test_data/eth-uncle-json-997522-0

@@ -0,0 +1 @@
+{"jsonrpc":"2.0","result":{"author":"0x68795c4aa09d6f4ed3e5deddf8c2ad3049a601da","difficulty":"0xae387bd92cc","extraData":"0xd783010400844765746887676f312e352e31856c696e7578","gasLimit":"0x2fefd8","gasUsed":"0x0","hash":"0x319e0dc9a53711579c4ba88062c927a0045443cca57625903ef471d760506a94","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x68795c4aa09d6f4ed3e5deddf8c2ad3049a601da","mixHash":"0x2d4fd3be43fd50f5c0ba7f1c86c8f468b5c14f75b6143da927a2994383f26640","nonce":"0x0aaaa7fe9d7cf7f4","number":"0xf388f","parentHash":"0xac74216bbdb0ebec6612ad5f26301ab50e588aabe75a804bc2068f83980eefc6","receiptsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","sealFields":["0xa02d4fd3be43fd50f5c0ba7f1c86c8f468b5c14f75b6143da927a2994383f26640","0x880aaaa7fe9d7cf7f4"],"sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":null,"stateRoot":"0xf9309492322aab44243f8c38240874b37dd0c563bac85f1a816941acc945b21d","timestamp":"0x56bf1097","totalDifficulty":"0x6299e9e3fdb6eb4d","transactions":[],"transactionsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","uncles":[]},"id":1}

statediff/indexer/ipfs/ipld/test_data/eth-uncle-json-997522-1

@@ -0,0 +1 @@
+{"jsonrpc":"2.0","result":{"author":"0x68795c4aa09d6f4ed3e5deddf8c2ad3049a601da","difficulty":"0xae22b4c9b9a","extraData":"0xd783010400844765746887676f312e352e31856c696e7578","gasLimit":"0x2fefd8","gasUsed":"0xf618","hash":"0x0324272e484e509c3c9e9e75ad8b48c7d34556e6b269dd72331033fd5cdc1b2a","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x68795c4aa09d6f4ed3e5deddf8c2ad3049a601da","mixHash":"0x0f3bdea5170d6af74b70fcf0df81969f6bb1b740f4a6c78df1d354f172865594","nonce":"0x4c691de262b2b3d9","number":"0xf3890","parentHash":"0xcb9efe9bc3c59be7fb673576d661aff9ca75b1522f58fd38d03d3d49b32bddb3","receiptsRoot":"0x5cf73738487f67f1c0a1c2d1083ae014f38e1aab5eb26a8929a511c48b07ea03","sealFields":["0xa00f3bdea5170d6af74b70fcf0df81969f6bb1b740f4a6c78df1d354f172865594","0x884c691de262b2b3d9"],"sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":null,"stateRoot":"0x968e8d8d099572ac783f4511724ec646f59bb33f7395edf858f98b37c8c3b265","timestamp":"0x56bf10b1","totalDifficulty":"0x6299f4c6290386e7","transactions":[],"transactionsRoot":"0x9cea6a59a5df69111ead7406a431c764b2357120e5b61425388df62f87cbcbc3","uncles":[]},"id":1}

statediff/indexer/ipfs/ipld/trie_node.go

@@ -0,0 +1,456 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipld
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/ethereum/go-ethereum/rlp"
+	"github.com/ipfs/go-cid"
+	node "github.com/ipfs/go-ipld-format"
+)
+
+const (
+	extension = "extension"
+	leaf      = "leaf"
+	branch    = "branch"
+)
+
+// TrieNode is the general abstraction for
+//ethereum IPLD trie nodes.
+type TrieNode struct {
+	// leaf, extension or branch
+	nodeKind string
+
+	// If leaf or extension: [0] is key, [1] is val.
+	// If branch: [0] - [16] are children.
+	elements []interface{}
+
+	// IPLD block information
+	cid     cid.Cid
+	rawdata []byte
+}
+
+/*
+  OUTPUT
+*/
+
+type trieNodeLeafDecoder func([]interface{}) ([]interface{}, error)
+
+// decodeTrieNode returns a TrieNode object from an IPLD block's
+// cid and rawdata.
+func decodeTrieNode(c cid.Cid, b []byte,
+	leafDecoder trieNodeLeafDecoder) (*TrieNode, error) {
+	var (
+		i, decoded, elements []interface{}
+		nodeKind             string
+		err                  error
+	)
+
+	if err = rlp.DecodeBytes(b, &i); err != nil {
+		return nil, err
+	}
+
+	codec := c.Type()
+	switch len(i) {
+	case 2:
+		nodeKind, decoded, err = decodeCompactKey(i)
+		if err != nil {
+			return nil, err
+		}
+
+		if nodeKind == extension {
+			elements, err = parseTrieNodeExtension(decoded, codec)
+			if err != nil {
+				return nil, err
+			}
+		}
+		if nodeKind == leaf {
+			elements, err = leafDecoder(decoded)
+			if err != nil {
+				return nil, err
+			}
+		}
+		if nodeKind != extension && nodeKind != leaf {
+			return nil, fmt.Errorf("unexpected nodeKind returned from decoder")
+		}
+	case 17:
+		nodeKind = branch
+		elements, err = parseTrieNodeBranch(i, codec)
+		if err != nil {
+			return nil, err
+		}
+	default:
+		return nil, fmt.Errorf("unknown trie node type")
+	}
+
+	return &TrieNode{
+		nodeKind: nodeKind,
+		elements: elements,
+		rawdata:  b,
+		cid:      c,
+	}, nil
+}
+
+// decodeCompactKey takes a compact key, and returns its nodeKind and value.
+func decodeCompactKey(i []interface{}) (string, []interface{}, error) {
+	first := i[0].([]byte)
+	last := i[1].([]byte)
+
+	switch first[0] / 16 {
+	case '\x00':
+		return extension, []interface{}{
+			nibbleToByte(first)[2:],
+			last,
+		}, nil
+	case '\x01':
+		return extension, []interface{}{
+			nibbleToByte(first)[1:],
+			last,
+		}, nil
+	case '\x02':
+		return leaf, []interface{}{
+			nibbleToByte(first)[2:],
+			last,
+		}, nil
+	case '\x03':
+		return leaf, []interface{}{
+			nibbleToByte(first)[1:],
+			last,
+		}, nil
+	default:
+		return "", nil, fmt.Errorf("unknown hex prefix")
+	}
+}
+
+// parseTrieNodeExtension helper improves readability
+func parseTrieNodeExtension(i []interface{}, codec uint64) ([]interface{}, error) {
+	return []interface{}{
+		i[0].([]byte),
+		keccak256ToCid(codec, i[1].([]byte)),
+	}, nil
+}
+
+// parseTrieNodeBranch helper improves readability
+func parseTrieNodeBranch(i []interface{}, codec uint64) ([]interface{}, error) {
+	var out []interface{}
+
+	for i, vi := range i {
+		v, ok := vi.([]byte)
+		// Sometimes this throws "panic: interface conversion: interface {} is []interface {}, not []uint8"
+		// Figure out why, and if it is okay to continue
+		if !ok {
+			return nil, fmt.Errorf("unable to decode branch node entry into []byte at position: %d value: %+v", i, vi)
+		}
+
+		switch len(v) {
+		case 0:
+			out = append(out, nil)
+		case 32:
+			out = append(out, keccak256ToCid(codec, v))
+		default:
+			return nil, fmt.Errorf("unrecognized object: %v", v)
+		}
+	}
+
+	return out, nil
+}
+
+/*
+  Node INTERFACE
+*/
+
+// Resolve resolves a path through this node, stopping at any link boundary
+// and returning the object found as well as the remaining path to traverse
+func (t *TrieNode) Resolve(p []string) (interface{}, []string, error) {
+	switch t.nodeKind {
+	case extension:
+		return t.resolveTrieNodeExtension(p)
+	case leaf:
+		return t.resolveTrieNodeLeaf(p)
+	case branch:
+		return t.resolveTrieNodeBranch(p)
+	default:
+		return nil, nil, fmt.Errorf("nodeKind case not implemented")
+	}
+}
+
+// Tree lists all paths within the object under 'path', and up to the given depth.
+// To list the entire object (similar to `find .`) pass "" and -1
+func (t *TrieNode) Tree(p string, depth int) []string {
+	if p != "" || depth == 0 {
+		return nil
+	}
+
+	var out []string
+
+	switch t.nodeKind {
+	case extension:
+		var val string
+		for _, e := range t.elements[0].([]byte) {
+			val += fmt.Sprintf("%x", e)
+		}
+		return []string{val}
+	case branch:
+		for i, elem := range t.elements {
+			if _, ok := elem.(cid.Cid); ok {
+				out = append(out, fmt.Sprintf("%x", i))
+			}
+		}
+		return out
+
+	default:
+		return nil
+	}
+}
+
+// ResolveLink is a helper function that calls resolve and asserts the
+// output is a link
+func (t *TrieNode) ResolveLink(p []string) (*node.Link, []string, error) {
+	obj, rest, err := t.Resolve(p)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	lnk, ok := obj.(*node.Link)
+	if !ok {
+		return nil, nil, fmt.Errorf("was not a link")
+	}
+
+	return lnk, rest, nil
+}
+
+// Copy will go away. It is here to comply with the interface.
+func (t *TrieNode) Copy() node.Node {
+	panic("implement me")
+}
+
+// Links is a helper function that returns all links within this object
+func (t *TrieNode) Links() []*node.Link {
+	var out []*node.Link
+
+	for _, i := range t.elements {
+		c, ok := i.(cid.Cid)
+		if ok {
+			out = append(out, &node.Link{Cid: c})
+		}
+	}
+
+	return out
+}
+
+// Stat will go away. It is here to comply with the interface.
+func (t *TrieNode) Stat() (*node.NodeStat, error) {
+	return &node.NodeStat{}, nil
+}
+
+// Size will go away. It is here to comply with the interface.
+func (t *TrieNode) Size() (uint64, error) {
+	return 0, nil
+}
+
+/*
+  TrieNode functions
+*/
+
+// MarshalJSON processes the transaction trie into readable JSON format.
+func (t *TrieNode) MarshalJSON() ([]byte, error) {
+	var out map[string]interface{}
+
+	switch t.nodeKind {
+	case extension:
+		fallthrough
+	case leaf:
+		var hexPrefix string
+		for _, e := range t.elements[0].([]byte) {
+			hexPrefix += fmt.Sprintf("%x", e)
+		}
+
+		// if we got a byte we need to do this casting otherwise
+		// it will be marshaled to a base64 encoded value
+		if _, ok := t.elements[1].([]byte); ok {
+			var hexVal string
+			for _, e := range t.elements[1].([]byte) {
+				hexVal += fmt.Sprintf("%x", e)
+			}
+
+			t.elements[1] = hexVal
+		}
+
+		out = map[string]interface{}{
+			"type":    t.nodeKind,
+			hexPrefix: t.elements[1],
+		}
+
+	case branch:
+		out = map[string]interface{}{
+			"type": branch,
+			"0":    t.elements[0],
+			"1":    t.elements[1],
+			"2":    t.elements[2],
+			"3":    t.elements[3],
+			"4":    t.elements[4],
+			"5":    t.elements[5],
+			"6":    t.elements[6],
+			"7":    t.elements[7],
+			"8":    t.elements[8],
+			"9":    t.elements[9],
+			"a":    t.elements[10],
+			"b":    t.elements[11],
+			"c":    t.elements[12],
+			"d":    t.elements[13],
+			"e":    t.elements[14],
+			"f":    t.elements[15],
+		}
+	default:
+		return nil, fmt.Errorf("nodeKind %s not supported", t.nodeKind)
+	}
+
+	return json.Marshal(out)
+}
+
+// nibbleToByte expands the nibbles of a byte slice into their own bytes.
+func nibbleToByte(k []byte) []byte {
+	var out []byte
+
+	for _, b := range k {
+		out = append(out, b/16)
+		out = append(out, b%16)
+	}
+
+	return out
+}
+
+// Resolve reading conveniences
+func (t *TrieNode) resolveTrieNodeExtension(p []string) (interface{}, []string, error) {
+	nibbles := t.elements[0].([]byte)
+	idx, rest := shiftFromPath(p, len(nibbles))
+	if len(idx) < len(nibbles) {
+		return nil, nil, fmt.Errorf("not enough nibbles to traverse this extension")
+	}
+
+	for _, i := range idx {
+		if getHexIndex(string(i)) == -1 {
+			return nil, nil, fmt.Errorf("invalid path element")
+		}
+	}
+
+	for i, n := range nibbles {
+		if string(idx[i]) != fmt.Sprintf("%x", n) {
+			return nil, nil, fmt.Errorf("no such link in this extension")
+		}
+	}
+
+	return &node.Link{Cid: t.elements[1].(cid.Cid)}, rest, nil
+}
+
+func (t *TrieNode) resolveTrieNodeLeaf(p []string) (interface{}, []string, error) {
+	nibbles := t.elements[0].([]byte)
+
+	if len(nibbles) != 0 {
+		idx, rest := shiftFromPath(p, len(nibbles))
+		if len(idx) < len(nibbles) {
+			return nil, nil, fmt.Errorf("not enough nibbles to traverse this leaf")
+		}
+
+		for _, i := range idx {
+			if getHexIndex(string(i)) == -1 {
+				return nil, nil, fmt.Errorf("invalid path element")
+			}
+		}
+
+		for i, n := range nibbles {
+			if string(idx[i]) != fmt.Sprintf("%x", n) {
+				return nil, nil, fmt.Errorf("no such link in this extension")
+			}
+		}
+
+		p = rest
+	}
+
+	link, ok := t.elements[1].(node.Node)
+	if !ok {
+		return nil, nil, fmt.Errorf("leaf children is not an IPLD node")
+	}
+
+	return link.Resolve(p)
+}
+
+func (t *TrieNode) resolveTrieNodeBranch(p []string) (interface{}, []string, error) {
+	idx, rest := shiftFromPath(p, 1)
+	hidx := getHexIndex(idx)
+	if hidx == -1 {
+		return nil, nil, fmt.Errorf("incorrect path")
+	}
+
+	child := t.elements[hidx]
+	if child != nil {
+		return &node.Link{Cid: child.(cid.Cid)}, rest, nil
+	}
+	return nil, nil, fmt.Errorf("no such link in this branch")
+}
+
+// shiftFromPath extracts from a given path (as a slice of strings)
+// the given number of elements as a single string, returning whatever
+// it has not taken.
+//
+// Examples:
+// ["0", "a", "something"] and 1 -> "0" and ["a", "something"]
+// ["ab", "c", "d", "1"] and 2 -> "ab" and ["c", "d", "1"]
+// ["abc", "d", "1"] and 2 -> "ab" and ["c", "d", "1"]
+func shiftFromPath(p []string, i int) (string, []string) {
+	var (
+		out  string
+		rest []string
+	)
+
+	for _, pe := range p {
+		re := ""
+		for _, c := range pe {
+			if len(out) < i {
+				out += string(c)
+			} else {
+				re += string(c)
+			}
+		}
+
+		if len(out) == i && re != "" {
+			rest = append(rest, re)
+		}
+	}
+
+	return out, rest
+}
+
+// getHexIndex returns to you the integer 0 - 15 equivalent to your
+// string character if applicable, or -1 otherwise.
+func getHexIndex(s string) int {
+	if len(s) != 1 {
+		return -1
+	}
+
+	c := s[0]
+	switch {
+	case '0' <= c && c <= '9':
+		return int(c - '0')
+	case 'a' <= c && c <= 'f':
+		return int(c - 'a' + 10)
+	}
+
+	return -1
+}

statediff/indexer/ipfs/models.go

@@ -0,0 +1,22 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package ipfs
+
+type BlockModel struct {
+	CID  string `db:"key"`
+	Data []byte `db:"data"`
+}

statediff/indexer/metrics.go

@@ -0,0 +1,128 @@
+package indexer
+
+import (
+	"database/sql"
+	"strings"
+
+	"github.com/ethereum/go-ethereum/metrics"
+)
+
+const (
+	namespace = "statediff"
+)
+
+// Build a fully qualified metric name
+func metricName(subsystem, name string) string {
+	if name == "" {
+		return ""
+	}
+	parts := []string{namespace, name}
+	if subsystem != "" {
+		parts = []string{namespace, subsystem, name}
+	}
+	// Prometheus uses _ but geth metrics uses / and replaces
+	return strings.Join(parts, "/")
+}
+
+type indexerMetricsHandles struct {
+	// The total number of processed blocks
+	blocks metrics.Counter
+	// The total number of processed transactions
+	transactions metrics.Counter
+	// The total number of processed receipts
+	receipts metrics.Counter
+	// The total number of access list entries processed
+	accessListEntries metrics.Counter
+	// Time spent waiting for free postgres tx
+	tFreePostgres metrics.Timer
+	// Postgres transaction commit duration
+	tPostgresCommit metrics.Timer
+	// Header processing time
+	tHeaderProcessing metrics.Timer
+	// Uncle processing time
+	tUncleProcessing metrics.Timer
+	// Tx and receipt processing time
+	tTxAndRecProcessing metrics.Timer
+	// State, storage, and code combined processing time
+	tStateStoreCodeProcessing metrics.Timer
+}
+
+func RegisterIndexerMetrics(reg metrics.Registry) indexerMetricsHandles {
+	ctx := indexerMetricsHandles{
+		blocks:                    metrics.NewCounter(),
+		transactions:              metrics.NewCounter(),
+		receipts:                  metrics.NewCounter(),
+		accessListEntries:         metrics.NewCounter(),
+		tFreePostgres:             metrics.NewTimer(),
+		tPostgresCommit:           metrics.NewTimer(),
+		tHeaderProcessing:         metrics.NewTimer(),
+		tUncleProcessing:          metrics.NewTimer(),
+		tTxAndRecProcessing:       metrics.NewTimer(),
+		tStateStoreCodeProcessing: metrics.NewTimer(),
+	}
+	subsys := "indexer"
+	reg.Register(metricName(subsys, "blocks"), ctx.blocks)
+	reg.Register(metricName(subsys, "transactions"), ctx.transactions)
+	reg.Register(metricName(subsys, "receipts"), ctx.receipts)
+	reg.Register(metricName(subsys, "access_list_entries"), ctx.accessListEntries)
+	reg.Register(metricName(subsys, "t_free_postgres"), ctx.tFreePostgres)
+	reg.Register(metricName(subsys, "t_postgres_commit"), ctx.tPostgresCommit)
+	reg.Register(metricName(subsys, "t_header_processing"), ctx.tHeaderProcessing)
+	reg.Register(metricName(subsys, "t_uncle_processing"), ctx.tUncleProcessing)
+	reg.Register(metricName(subsys, "t_tx_receipt_processing"), ctx.tTxAndRecProcessing)
+	reg.Register(metricName(subsys, "t_state_store_code_processing"), ctx.tStateStoreCodeProcessing)
+	return ctx
+}
+
+type dbMetricsHandles struct {
+	// Maximum number of open connections to the database
+	maxOpen metrics.Gauge
+	// The number of established connections both in use and idle
+	open metrics.Gauge
+	// The number of connections currently in use
+	inUse metrics.Gauge
+	// The number of idle connections
+	idle metrics.Gauge
+	// The total number of connections waited for
+	waitedFor metrics.Counter
+	// The total time blocked waiting for a new connection
+	blockedMilliseconds metrics.Counter
+	// The total number of connections closed due to SetMaxIdleConns
+	closedMaxIdle metrics.Counter
+	// The total number of connections closed due to SetConnMaxLifetime
+	closedMaxLifetime metrics.Counter
+}
+
+func RegisterDBMetrics(reg metrics.Registry) dbMetricsHandles {
+	ctx := dbMetricsHandles{
+		maxOpen:             metrics.NewGauge(),
+		open:                metrics.NewGauge(),
+		inUse:               metrics.NewGauge(),
+		idle:                metrics.NewGauge(),
+		waitedFor:           metrics.NewCounter(),
+		blockedMilliseconds: metrics.NewCounter(),
+		closedMaxIdle:       metrics.NewCounter(),
+		closedMaxLifetime:   metrics.NewCounter(),
+	}
+	subsys := "connections"
+	reg.Register(metricName(subsys, "max_open"), ctx.maxOpen)
+	reg.Register(metricName(subsys, "open"), ctx.open)
+	reg.Register(metricName(subsys, "in_use"), ctx.inUse)
+	reg.Register(metricName(subsys, "idle"), ctx.idle)
+	reg.Register(metricName(subsys, "waited_for"), ctx.waitedFor)
+	reg.Register(metricName(subsys, "blocked_milliseconds"), ctx.blockedMilliseconds)
+	reg.Register(metricName(subsys, "closed_max_idle"), ctx.closedMaxIdle)
+	reg.Register(metricName(subsys, "closed_max_lifetime"), ctx.closedMaxLifetime)
+	return ctx
+}
+
+func (met *dbMetricsHandles) Update(stats sql.DBStats) {
+	met.maxOpen.Update(int64(stats.MaxOpenConnections))
+	met.open.Update(int64(stats.OpenConnections))
+	met.inUse.Update(int64(stats.InUse))
+	met.idle.Update(int64(stats.Idle))
+	met.waitedFor.Inc(stats.WaitCount)
+	met.blockedMilliseconds.Inc(stats.WaitDuration.Milliseconds())
+	met.closedMaxIdle.Inc(stats.MaxIdleClosed)
+	met.closedMaxLifetime.Inc(stats.MaxLifetimeClosed)
+}

statediff/indexer/mocks/test_data.go

@@ -0,0 +1,249 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package mocks
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"math/big"
+
+	"github.com/ethereum/go-ethereum/statediff/indexer/models"
+
+	"github.com/ethereum/go-ethereum/trie"
+
+	"github.com/ethereum/go-ethereum/common"
+	"github.com/ethereum/go-ethereum/core/state"
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/crypto"
+	"github.com/ethereum/go-ethereum/log"
+	"github.com/ethereum/go-ethereum/params"
+	"github.com/ethereum/go-ethereum/rlp"
+	"github.com/ethereum/go-ethereum/statediff/testhelpers"
+	sdtypes "github.com/ethereum/go-ethereum/statediff/types"
+)
+
+// Test variables
+var (
+	// block data
+	BlockNumber = big.NewInt(12244001)
+	MockHeader  = types.Header{
+		Time:        0,
+		Number:      new(big.Int).Set(BlockNumber),
+		Root:        common.HexToHash("0x0"),
+		TxHash:      common.HexToHash("0x0"),
+		ReceiptHash: common.HexToHash("0x0"),
+		Difficulty:  big.NewInt(5000000),
+		Extra:       []byte{},
+	}
+	MockTransactions, MockReceipts, SenderAddr        = createTransactionsAndReceipts()
+	MockBlock                                         = types.NewBlock(&MockHeader, MockTransactions, nil, MockReceipts, new(trie.Trie))
+	MockHeaderRlp, _                                  = rlp.EncodeToBytes(MockBlock.Header())
+	Address                                           = common.HexToAddress("0xaE9BEa628c4Ce503DcFD7E305CaB4e29E7476592")
+	AnotherAddress                                    = common.HexToAddress("0xaE9BEa628c4Ce503DcFD7E305CaB4e29E7476593")
+	ContractAddress                                   = crypto.CreateAddress(SenderAddr, MockTransactions[2].Nonce())
+	MockContractByteCode                              = []byte{0, 1, 2, 3, 4, 5}
+	mockTopic11                                       = common.HexToHash("0x04")
+	mockTopic12                                       = common.HexToHash("0x06")
+	mockTopic21                                       = common.HexToHash("0x05")
+	mockTopic22                                       = common.HexToHash("0x07")
+	ExpectedPostStatus                         uint64 = 1
+	ExpectedPostState1                                = common.Bytes2Hex(common.HexToHash("0x1").Bytes())
+	ExpectedPostState2                                = common.Bytes2Hex(common.HexToHash("0x2").Bytes())
+	ExpectedPostState3                                = common.Bytes2Hex(common.HexToHash("0x3").Bytes())
+	MockLog1                                          = &types.Log{
+		Address: Address,
+		Topics:  []common.Hash{mockTopic11, mockTopic12},
+		Data:    []byte{},
+	}
+	MockLog2 = &types.Log{
+		Address: AnotherAddress,
+		Topics:  []common.Hash{mockTopic21, mockTopic22},
+		Data:    []byte{},
+	}
+
+	// access list entries
+	AccessListEntry1 = types.AccessTuple{
+		Address: Address,
+	}
+	AccessListEntry2 = types.AccessTuple{
+		Address:     AnotherAddress,
+		StorageKeys: []common.Hash{common.BytesToHash(StorageLeafKey), common.BytesToHash(MockStorageLeafKey)},
+	}
+	AccessListEntry1Model = models.AccessListElementModel{
+		Index:   0,
+		Address: Address.Hex(),
+	}
+	AccessListEntry2Model = models.AccessListElementModel{
+		Index:       1,
+		Address:     AnotherAddress.Hex(),
+		StorageKeys: []string{common.BytesToHash(StorageLeafKey).Hex(), common.BytesToHash(MockStorageLeafKey).Hex()},
+	}
+
+	// statediff data
+	storageLocation     = common.HexToHash("0")
+	StorageLeafKey      = crypto.Keccak256Hash(storageLocation[:]).Bytes()
+	mockStorageLocation = common.HexToHash("1")
+	MockStorageLeafKey  = crypto.Keccak256Hash(mockStorageLocation[:]).Bytes()
+	StorageValue        = common.Hex2Bytes("01")
+	StoragePartialPath  = common.Hex2Bytes("20290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563")
+	StorageLeafNode, _  = rlp.EncodeToBytes([]interface{}{
+		StoragePartialPath,
+		StorageValue,
+	})
+
+	nonce1             = uint64(1)
+	ContractRoot       = "0x821e2556a290c86405f8160a2d662042a431ba456b9db265c79bb837c04be5f0"
+	ContractCodeHash   = common.HexToHash("0x753f98a8d4328b15636e46f66f2cb4bc860100aa17967cc145fcd17d1d4710ea")
+	ContractLeafKey    = testhelpers.AddressToLeafKey(ContractAddress)
+	ContractAccount, _ = rlp.EncodeToBytes(state.Account{
+		Nonce:    nonce1,
+		Balance:  big.NewInt(0),
+		CodeHash: ContractCodeHash.Bytes(),
+		Root:     common.HexToHash(ContractRoot),
+	})
+	ContractPartialPath = common.Hex2Bytes("3114658a74d9cc9f7acf2c5cd696c3494d7c344d78bfec3add0d91ec4e8d1c45")
+	ContractLeafNode, _ = rlp.EncodeToBytes([]interface{}{
+		ContractPartialPath,
+		ContractAccount,
+	})
+
+	nonce0          = uint64(0)
+	AccountRoot     = "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421"
+	AccountCodeHash = common.HexToHash("0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470")
+	AccountLeafKey  = testhelpers.Account2LeafKey
+	Account, _      = rlp.EncodeToBytes(state.Account{
+		Nonce:    nonce0,
+		Balance:  big.NewInt(1000),
+		CodeHash: AccountCodeHash.Bytes(),
+		Root:     common.HexToHash(AccountRoot),
+	})
+	AccountPartialPath = common.Hex2Bytes("3957f3e2f04a0764c3a0491b175f69926da61efbcc8f61fa1455fd2d2b4cdd45")
+	AccountLeafNode, _ = rlp.EncodeToBytes([]interface{}{
+		AccountPartialPath,
+		Account,
+	})
+
+	StateDiffs = []sdtypes.StateNode{
+		{
+			Path:      []byte{'\x06'},
+			NodeType:  sdtypes.Leaf,
+			LeafKey:   ContractLeafKey,
+			NodeValue: ContractLeafNode,
+			StorageNodes: []sdtypes.StorageNode{
+				{
+					Path:      []byte{},
+					NodeType:  sdtypes.Leaf,
+					LeafKey:   StorageLeafKey,
+					NodeValue: StorageLeafNode,
+				},
+			},
+		},
+		{
+			Path:         []byte{'\x0c'},
+			NodeType:     sdtypes.Leaf,
+			LeafKey:      AccountLeafKey,
+			NodeValue:    AccountLeafNode,
+			StorageNodes: []sdtypes.StorageNode{},
+		},
+	}
+)
+
+/*
+// AccessListTx is the data of EIP-2930 access list transactions.
+type AccessListTx struct {
+	ChainID    *big.Int        // destination chain ID
+	Nonce      uint64          // nonce of sender account
+	GasPrice   *big.Int        // wei per gas
+	Gas        uint64          // gas limit
+	To         *common.Address `rlp:"nil"` // nil means contract creation
+	Value      *big.Int        // wei amount
+	Data       []byte          // contract invocation input data
+	AccessList AccessList      // EIP-2930 access list
+	V, R, S    *big.Int        // signature values
+}
+
+*/
+
+// createTransactionsAndReceipts is a helper function to generate signed mock transactions and mock receipts with mock logs
+func createTransactionsAndReceipts() (types.Transactions, types.Receipts, common.Address) {
+	// make transactions
+	trx1 := types.NewTransaction(0, Address, big.NewInt(1000), 50, big.NewInt(100), []byte{})
+	trx2 := types.NewTransaction(1, AnotherAddress, big.NewInt(2000), 100, big.NewInt(200), []byte{})
+	trx3 := types.NewContractCreation(2, big.NewInt(1500), 75, big.NewInt(150), MockContractByteCode)
+	trx4 := types.NewTx(&types.AccessListTx{
+		ChainID:  big.NewInt(1),
+		Nonce:    0,
+		GasPrice: big.NewInt(100),
+		Gas:      50,
+		To:       &AnotherAddress,
+		Value:    big.NewInt(1000),
+		Data:     []byte{},
+		AccessList: types.AccessList{
+			AccessListEntry1,
+			AccessListEntry2,
+		},
+	})
+
+	transactionSigner := types.NewEIP2930Signer(params.MainnetChainConfig.ChainID)
+	mockCurve := elliptic.P256()
+	mockPrvKey, err := ecdsa.GenerateKey(mockCurve, rand.Reader)
+	if err != nil {
+		log.Crit(err.Error())
+	}
+	signedTrx1, err := types.SignTx(trx1, transactionSigner, mockPrvKey)
+	if err != nil {
+		log.Crit(err.Error())
+	}
+	signedTrx2, err := types.SignTx(trx2, transactionSigner, mockPrvKey)
+	if err != nil {
+		log.Crit(err.Error())
+	}
+	signedTrx3, err := types.SignTx(trx3, transactionSigner, mockPrvKey)
+	if err != nil {
+		log.Crit(err.Error())
+	}
+	signedTrx4, err := types.SignTx(trx4, transactionSigner, mockPrvKey)
+	if err != nil {
+		println(err.Error())
+		log.Crit(err.Error())
+	}
+	senderAddr, err := types.Sender(transactionSigner, signedTrx1) // same for both trx
+	if err != nil {
+		log.Crit(err.Error())
+	}
+	// make receipts
+	mockReceipt1 := types.NewReceipt(nil, false, 50)
+	mockReceipt1.Logs = []*types.Log{MockLog1}
+	mockReceipt1.TxHash = signedTrx1.Hash()
+	mockReceipt2 := types.NewReceipt(common.HexToHash("0x1").Bytes(), false, 100)
+	mockReceipt2.Logs = []*types.Log{MockLog2}
+	mockReceipt2.TxHash = signedTrx2.Hash()
+	mockReceipt3 := types.NewReceipt(common.HexToHash("0x2").Bytes(), false, 75)
+	mockReceipt3.Logs = []*types.Log{}
+	mockReceipt3.TxHash = signedTrx3.Hash()
+	mockReceipt4 := &types.Receipt{
+		Type:              types.AccessListTxType,
+		PostState:         common.HexToHash("0x3").Bytes(),
+		Status:            types.ReceiptStatusSuccessful,
+		CumulativeGasUsed: 175,
+		Logs:              []*types.Log{},
+		TxHash:            signedTrx4.Hash(),
+	}
+
+	return types.Transactions{signedTrx1, signedTrx2, signedTrx3, signedTrx4}, types.Receipts{mockReceipt1, mockReceipt2, mockReceipt3, mockReceipt4}, senderAddr
+}

statediff/indexer/models/models.go

@@ -0,0 +1,137 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package models
+
+import "github.com/lib/pq"
+
+// HeaderModel is the db model for eth.header_cids
+type HeaderModel struct {
+	ID              int64  `db:"id"`
+	BlockNumber     string `db:"block_number"`
+	BlockHash       string `db:"block_hash"`
+	ParentHash      string `db:"parent_hash"`
+	CID             string `db:"cid"`
+	MhKey           string `db:"mh_key"`
+	TotalDifficulty string `db:"td"`
+	NodeID          int64  `db:"node_id"`
+	Reward          string `db:"reward"`
+	StateRoot       string `db:"state_root"`
+	UncleRoot       string `db:"uncle_root"`
+	TxRoot          string `db:"tx_root"`
+	RctRoot         string `db:"receipt_root"`
+	Bloom           []byte `db:"bloom"`
+	Timestamp       uint64 `db:"timestamp"`
+	TimesValidated  int64  `db:"times_validated"`
+}
+
+// UncleModel is the db model for eth.uncle_cids
+type UncleModel struct {
+	ID         int64  `db:"id"`
+	HeaderID   int64  `db:"header_id"`
+	BlockHash  string `db:"block_hash"`
+	ParentHash string `db:"parent_hash"`
+	CID        string `db:"cid"`
+	MhKey      string `db:"mh_key"`
+	Reward     string `db:"reward"`
+}
+
+// TxModel is the db model for eth.transaction_cids
+type TxModel struct {
+	ID       int64  `db:"id"`
+	HeaderID int64  `db:"header_id"`
+	Index    int64  `db:"index"`
+	TxHash   string `db:"tx_hash"`
+	CID      string `db:"cid"`
+	MhKey    string `db:"mh_key"`
+	Dst      string `db:"dst"`
+	Src      string `db:"src"`
+	Data     []byte `db:"tx_data"`
+	Type     *uint8 `db:"tx_type"`
+}
+
+// AccessListEntryModel is the db model for eth.access_list_entry
+type AccessListElementModel struct {
+	ID          int64          `db:"id"`
+	Index       int64          `db:"index"`
+	TxID        int64          `db:"tx_id"`
+	Address     string         `db:"address"`
+	StorageKeys pq.StringArray `db:"storage_keys"`
+}
+
+// ReceiptModel is the db model for eth.receipt_cids
+type ReceiptModel struct {
+	ID           int64          `db:"id"`
+	TxID         int64          `db:"tx_id"`
+	CID          string         `db:"cid"`
+	MhKey        string         `db:"mh_key"`
+	PostStatus   uint64         `db:"post_status"`
+	PostState    string         `db:"post_state"`
+	Contract     string         `db:"contract"`
+	ContractHash string         `db:"contract_hash"`
+	LogContracts pq.StringArray `db:"log_contracts"`
+	Topic0s      pq.StringArray `db:"topic0s"`
+	Topic1s      pq.StringArray `db:"topic1s"`
+	Topic2s      pq.StringArray `db:"topic2s"`
+	Topic3s      pq.StringArray `db:"topic3s"`
+}
+
+// StateNodeModel is the db model for eth.state_cids
+type StateNodeModel struct {
+	ID       int64  `db:"id"`
+	HeaderID int64  `db:"header_id"`
+	Path     []byte `db:"state_path"`
+	StateKey string `db:"state_leaf_key"`
+	NodeType int    `db:"node_type"`
+	CID      string `db:"cid"`
+	MhKey    string `db:"mh_key"`
+	Diff     bool   `db:"diff"`
+}
+
+// StorageNodeModel is the db model for eth.storage_cids
+type StorageNodeModel struct {
+	ID         int64  `db:"id"`
+	StateID    int64  `db:"state_id"`
+	Path       []byte `db:"storage_path"`
+	StorageKey string `db:"storage_leaf_key"`
+	NodeType   int    `db:"node_type"`
+	CID        string `db:"cid"`
+	MhKey      string `db:"mh_key"`
+	Diff       bool   `db:"diff"`
+}
+
+// StorageNodeWithStateKeyModel is a db model for eth.storage_cids + eth.state_cids.state_key
+type StorageNodeWithStateKeyModel struct {
+	ID         int64  `db:"id"`
+	StateID    int64  `db:"state_id"`
+	Path       []byte `db:"storage_path"`
+	StateKey   string `db:"state_leaf_key"`
+	StorageKey string `db:"storage_leaf_key"`
+	NodeType   int    `db:"node_type"`
+	CID        string `db:"cid"`
+	MhKey      string `db:"mh_key"`
+	Diff       bool   `db:"diff"`
+}
+
+// StateAccountModel is a db model for an eth state account (decoded value of state leaf node)
+type StateAccountModel struct {
+	ID          int64  `db:"id"`
+	StateID     int64  `db:"state_id"`
+	Balance     string `db:"balance"`
+	Nonce       uint64 `db:"nonce"`
+	CodeHash    []byte `db:"code_hash"`
+	StorageRoot string `db:"storage_root"`
+}

statediff/indexer/node/node.go

@@ -0,0 +1,25 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package node
+
+type Info struct {
+	GenesisBlock string
+	NetworkID    string
+	ChainID      uint64
+	ID           string
+	ClientName   string
+}

statediff/indexer/postgres/config.go

@@ -0,0 +1,59 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package postgres
+
+import (
+	"fmt"
+)
+
+// Env variables
+const (
+	DATABASE_NAME                 = "DATABASE_NAME"
+	DATABASE_HOSTNAME             = "DATABASE_HOSTNAME"
+	DATABASE_PORT                 = "DATABASE_PORT"
+	DATABASE_USER                 = "DATABASE_USER"
+	DATABASE_PASSWORD             = "DATABASE_PASSWORD"
+	DATABASE_MAX_IDLE_CONNECTIONS = "DATABASE_MAX_IDLE_CONNECTIONS"
+	DATABASE_MAX_OPEN_CONNECTIONS = "DATABASE_MAX_OPEN_CONNECTIONS"
+	DATABASE_MAX_CONN_LIFETIME    = "DATABASE_MAX_CONN_LIFETIME"
+)
+
+type ConnectionParams struct {
+	Hostname string
+	Name     string
+	User     string
+	Password string
+	Port     int
+}
+
+type ConnectionConfig struct {
+	MaxIdle     int
+	MaxOpen     int
+	MaxLifetime int
+}
+
+func DbConnectionString(params ConnectionParams) string {
+	if len(params.User) > 0 && len(params.Password) > 0 {
+		return fmt.Sprintf("postgresql://%s:%s@%s:%d/%s?sslmode=disable",
+			params.User, params.Password, params.Hostname, params.Port, params.Name)
+	}
+	if len(params.User) > 0 && len(params.Password) == 0 {
+		return fmt.Sprintf("postgresql://%s@%s:%d/%s?sslmode=disable",
+			params.User, params.Hostname, params.Port, params.Name)
+	}
+	return fmt.Sprintf("postgresql://%s:%d/%s?sslmode=disable", params.Hostname, params.Port, params.Name)
+}

statediff/indexer/postgres/errors.go

@@ -0,0 +1,38 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package postgres
+
+import (
+	"fmt"
+)
+
+const (
+	DbConnectionFailedMsg = "db connection failed"
+	SettingNodeFailedMsg  = "unable to set db node"
+)
+
+func ErrDBConnectionFailed(connectErr error) error {
+	return formatError(DbConnectionFailedMsg, connectErr.Error())
+}
+
+func ErrUnableToSetNode(setErr error) error {
+	return formatError(SettingNodeFailedMsg, setErr.Error())
+}
+
+func formatError(msg, err string) error {
+	return fmt.Errorf("%s: %s", msg, err)
+}

statediff/indexer/postgres/postgres.go

@@ -0,0 +1,76 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package postgres
+
+import (
+	"time"
+
+	"github.com/jmoiron/sqlx"
+	_ "github.com/lib/pq" //postgres driver
+
+	"github.com/ethereum/go-ethereum/statediff/indexer/node"
+)
+
+type DB struct {
+	*sqlx.DB
+	Node   node.Info
+	NodeID int64
+}
+
+func NewDB(connectString string, config ConnectionConfig, node node.Info) (*DB, error) {
+	db, connectErr := sqlx.Connect("postgres", connectString)
+	if connectErr != nil {
+		return &DB{}, ErrDBConnectionFailed(connectErr)
+	}
+	if config.MaxOpen > 0 {
+		db.SetMaxOpenConns(config.MaxOpen)
+	}
+	if config.MaxIdle > 0 {
+		db.SetMaxIdleConns(config.MaxIdle)
+	}
+	if config.MaxLifetime > 0 {
+		lifetime := time.Duration(config.MaxLifetime) * time.Second
+		db.SetConnMaxLifetime(lifetime)
+	}
+	pg := DB{DB: db, Node: node}
+	nodeErr := pg.CreateNode(&node)
+	if nodeErr != nil {
+		return &DB{}, ErrUnableToSetNode(nodeErr)
+	}
+	return &pg, nil
+}
+
+func (db *DB) CreateNode(node *node.Info) error {
+	var nodeID int64
+	err := db.QueryRow(
+		`INSERT INTO nodes (genesis_block, network_id, node_id, client_name, chain_id)
+                VALUES ($1, $2, $3, $4, $5)
+                ON CONFLICT (genesis_block, network_id, node_id, chain_id)
+                  DO UPDATE
+                    SET genesis_block = $1,
+                        network_id = $2,
+                        node_id = $3,
+                        client_name = $4,
+						chain_id = $5
+                RETURNING id`,
+		node.GenesisBlock, node.NetworkID, node.ID, node.ClientName, node.ChainID).Scan(&nodeID)
+	if err != nil {
+		return ErrUnableToSetNode(err)
+	}
+	db.NodeID = nodeID
+	return nil
+}

statediff/indexer/postgres/postgres_suite_test.go

@@ -0,0 +1,25 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package postgres_test
+
+import (
+	"github.com/ethereum/go-ethereum/log"
+)
+
+func init() {
+	log.Root().SetHandler(log.DiscardHandler())
+}

statediff/indexer/postgres/postgres_test.go

@@ -0,0 +1,137 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package postgres_test
+
+import (
+	"fmt"
+	"strings"
+	"testing"
+
+	"math/big"
+
+	"github.com/jmoiron/sqlx"
+	_ "github.com/lib/pq"
+
+	"github.com/ethereum/go-ethereum/statediff/indexer/node"
+	"github.com/ethereum/go-ethereum/statediff/indexer/postgres"
+	"github.com/ethereum/go-ethereum/statediff/indexer/shared"
+)
+
+var DBParams = postgres.ConnectionParams{
+	Name:     "vulcanize_testing",
+	Password: "",
+	Port:     5432,
+	Hostname: "localhost",
+	User:     "postgres",
+}
+
+func expectContainsSubstring(t *testing.T, full string, sub string) {
+	if !strings.Contains(full, sub) {
+		t.Fatalf("Expected \"%v\" to contain substring \"%v\"\n", full, sub)
+	}
+}
+
+func TestPostgresDB(t *testing.T) {
+	var sqlxdb *sqlx.DB
+
+	t.Run("connects to the database", func(t *testing.T) {
+		var err error
+		pgConfig := postgres.DbConnectionString(DBParams)
+
+		sqlxdb, err = sqlx.Connect("postgres", pgConfig)
+
+		if err != nil {
+			t.Fatalf("failed to connect to db with connection string: %s err: %v", pgConfig, err)
+		}
+		if sqlxdb == nil {
+			t.Fatal("DB is nil")
+		}
+	})
+
+	t.Run("serializes big.Int to db", func(t *testing.T) {
+		// postgres driver doesn't support go big.Int type
+		// various casts in golang uint64, int64, overflow for
+		// transaction value (in wei) even though
+		// postgres numeric can handle an arbitrary
+		// sized int, so use string representation of big.Int
+		// and cast on insert
+
+		pgConnectString := postgres.DbConnectionString(DBParams)
+		db, err := sqlx.Connect("postgres", pgConnectString)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		bi := new(big.Int)
+		bi.SetString("34940183920000000000", 10)
+		shared.ExpectEqual(t, bi.String(), "34940183920000000000")
+
+		defer db.Exec(`DROP TABLE IF EXISTS example`)
+		_, err = db.Exec("CREATE TABLE example ( id INTEGER, data NUMERIC )")
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		sqlStatement := `  
+			INSERT INTO example (id, data)
+			VALUES (1, cast($1 AS NUMERIC))`
+		_, err = db.Exec(sqlStatement, bi.String())
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		var data string
+		err = db.QueryRow(`SELECT data FROM example WHERE id = 1`).Scan(&data)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		shared.ExpectEqual(t, bi.String(), data)
+		actual := new(big.Int)
+		actual.SetString(data, 10)
+		shared.ExpectEqual(t, actual, bi)
+	})
+
+	t.Run("throws error when can't connect to the database", func(t *testing.T) {
+		invalidDatabase := postgres.ConnectionParams{}
+		node := node.Info{GenesisBlock: "GENESIS", NetworkID: "1", ID: "x123", ClientName: "geth"}
+
+		_, err := postgres.NewDB(postgres.DbConnectionString(invalidDatabase),
+			postgres.ConnectionConfig{}, node)
+
+		if err == nil {
+			t.Fatal("Expected an error")
+		}
+
+		expectContainsSubstring(t, err.Error(), postgres.DbConnectionFailedMsg)
+	})
+
+	t.Run("throws error when can't create node", func(t *testing.T) {
+		badHash := fmt.Sprintf("x %s", strings.Repeat("1", 100))
+		node := node.Info{GenesisBlock: badHash, NetworkID: "1", ID: "x123", ClientName: "geth"}
+
+		_, err := postgres.NewDB(postgres.DbConnectionString(DBParams), postgres.ConnectionConfig{}, node)
+
+		if err == nil {
+			t.Fatal("Expected an error")
+		}
+		expectContainsSubstring(t, err.Error(), postgres.SettingNodeFailedMsg)
+	})
+}

statediff/indexer/reward.go

@@ -0,0 +1,76 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package indexer
+
+import (
+	"math/big"
+
+	"github.com/ethereum/go-ethereum/core/types"
+)
+
+func CalcEthBlockReward(header *types.Header, uncles []*types.Header, txs types.Transactions, receipts types.Receipts) *big.Int {
+	staticBlockReward := staticRewardByBlockNumber(header.Number.Uint64())
+	transactionFees := calcEthTransactionFees(txs, receipts)
+	uncleInclusionRewards := calcEthUncleInclusionRewards(header, uncles)
+	tmp := transactionFees.Add(transactionFees, uncleInclusionRewards)
+	return tmp.Add(tmp, staticBlockReward)
+}
+
+func CalcUncleMinerReward(blockNumber, uncleBlockNumber uint64) *big.Int {
+	staticBlockReward := staticRewardByBlockNumber(blockNumber)
+	rewardDiv8 := staticBlockReward.Div(staticBlockReward, big.NewInt(8))
+	mainBlock := new(big.Int).SetUint64(blockNumber)
+	uncleBlock := new(big.Int).SetUint64(uncleBlockNumber)
+	uncleBlockPlus8 := uncleBlock.Add(uncleBlock, big.NewInt(8))
+	uncleBlockPlus8MinusMainBlock := uncleBlockPlus8.Sub(uncleBlockPlus8, mainBlock)
+	return rewardDiv8.Mul(rewardDiv8, uncleBlockPlus8MinusMainBlock)
+}
+
+func staticRewardByBlockNumber(blockNumber uint64) *big.Int {
+	staticBlockReward := new(big.Int)
+	//https://blog.ethereum.org/2017/10/12/byzantium-hf-announcement/
+	if blockNumber >= 7280000 {
+		staticBlockReward.SetString("2000000000000000000", 10)
+	} else if blockNumber >= 4370000 {
+		staticBlockReward.SetString("3000000000000000000", 10)
+	} else {
+		staticBlockReward.SetString("5000000000000000000", 10)
+	}
+	return staticBlockReward
+}
+
+func calcEthTransactionFees(txs types.Transactions, receipts types.Receipts) *big.Int {
+	transactionFees := new(big.Int)
+	for i, transaction := range txs {
+		receipt := receipts[i]
+		gasPrice := big.NewInt(transaction.GasPrice().Int64())
+		gasUsed := big.NewInt(int64(receipt.GasUsed))
+		transactionFee := gasPrice.Mul(gasPrice, gasUsed)
+		transactionFees = transactionFees.Add(transactionFees, transactionFee)
+	}
+	return transactionFees
+}
+
+func calcEthUncleInclusionRewards(header *types.Header, uncles []*types.Header) *big.Int {
+	uncleInclusionRewards := new(big.Int)
+	for range uncles {
+		staticBlockReward := staticRewardByBlockNumber(header.Number.Uint64())
+		staticBlockReward.Div(staticBlockReward, big.NewInt(32))
+		uncleInclusionRewards.Add(uncleInclusionRewards, staticBlockReward)
+	}
+	return uncleInclusionRewards
+}

statediff/indexer/shared/chain_type.go

@@ -0,0 +1,78 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package shared
+
+import (
+	"errors"
+	"strings"
+)
+
+// ChainType enum for specifying blockchain
+type ChainType int
+
+const (
+	UnknownChain ChainType = iota
+	Ethereum
+	Bitcoin
+	Omni
+	EthereumClassic
+)
+
+func (c ChainType) String() string {
+	switch c {
+	case Ethereum:
+		return "Ethereum"
+	case Bitcoin:
+		return "Bitcoin"
+	case Omni:
+		return "Omni"
+	case EthereumClassic:
+		return "EthereumClassic"
+	default:
+		return ""
+	}
+}
+
+func (c ChainType) API() string {
+	switch c {
+	case Ethereum:
+		return "eth"
+	case Bitcoin:
+		return "btc"
+	case Omni:
+		return "omni"
+	case EthereumClassic:
+		return "etc"
+	default:
+		return ""
+	}
+}
+
+func NewChainType(name string) (ChainType, error) {
+	switch strings.ToLower(name) {
+	case "ethereum", "eth":
+		return Ethereum, nil
+	case "bitcoin", "btc", "xbt":
+		return Bitcoin, nil
+	case "omni":
+		return Omni, nil
+	case "classic", "etc":
+		return EthereumClassic, nil
+	default:
+		return UnknownChain, errors.New("invalid name for chain")
+	}
+}

statediff/indexer/shared/constants.go

@@ -0,0 +1,22 @@
+// VulcanizeDB
+// Copyright © 2019 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package shared
+
+const (
+	DefaultMaxBatchSize   uint64 = 100
+	DefaultMaxBatchNumber int64  = 50
+)