Make writeStateDiffAt call on missing block and add metrics (#20)

* Add chainID flag and make chainConfig optional * Add prometheus metrics to monitor validation progress * Make writeStateDiffAt calls on missing blocks * Update docs and config improvements * Use standard logger * Add copyright to files * Upgrade dependencies
2022-08-23 13:05:50 +05:30 · 2022-08-23 13:05:50 +05:30 · acf8e6f96a
commit acf8e6f96a
parent 716978e23c
21 changed files with 787 additions and 144 deletions
--- a/README.md
+++ b/README.md
@ -1,75 +1,124 @@
- [Validator-README](#validator-readme)
- [Overview](#overview)
- [Intention for the Validator](#intention-for-the-validator)
-  - [Edge Cases](#edge-cases)
- [Instructions for Testing](#instructions-for-testing)
- [Code Overview](#code-overview)
- [Known Bugs](#known-bugs)
- [Tests on 03/03/22](#tests-on-03-03-22)
-  - [Set Up](#set-up)
-  - [Testing Failures](#testing-failures)
+# ipld-eth-db-validator

-<small><i><a href='http://ecotrust-canada.github.io/markdown-toc/'>Table of contents generated with markdown-toc</a></i></small>
+> `ipld-eth-db-validator` performs validation checks on indexed Ethereum IPLD objects in a Postgres database:
+> * Attempt to apply transactions in each block and validate resultant block hash
+> * Check referential integrity between IPLD blocks and index tables

-# Overview
+## Setup

-This repository contains the validator. The purpose of the validator is to ensure that the data in the Core Postgres database match the data on the blockchain.
+Build the binary:

-# Intention for the Validator
+```bash
+make build
+```

-The perfect scenario for the validator is as follows:
+## Configuration

-1. The validator will have the capacity to perform historical checks for the Core Postgres database. Users can contain these historical checks to specified configurations (block range).
-2. The validator will validate a certain number of trailing blocks, `t`, trailing the head, `n`. Therefore the validator will constantly perform real-time validation starting at `n` and ending at `n - t`.
-3. The validator validates the IPLD blocks in the Core Database; it will update the core database to indicate that the validator validated it.
+An example config file:

-## Edge Cases
+```toml
+[database]
+  # db credentials
+  name     = "vulcanize_public" # DATABASE_NAME
+  hostname = "localhost"        # DATABASE_HOSTNAME
+  port     = 5432               # DATABASE_PORT
+  user     = "vdbm"             # DATABASE_USER
+  password = "..."              # DATABASE_PASSWORD

-We must consider the following edge cases for the validator.
+[validate]
+  # block height to initiate database validation at
+  blockHeight = 1      # VALIDATE_BLOCK_HEIGHT  (default: 1)
+  # number of blocks to trail behind the head
+  trail  = 16         # VALIDATE_TRAIL  (default: 16)
+  # sleep interval after validator has caught up to (head-trail) height (in sec)
+  sleepInterval = 10  # VALIDATE_SLEEP_INTERVAL (default: 10)

- There are three different data types that the validator must account for.
+  # whether to perform a statediffing call on a missing block
+  stateDiffMissingBlock = true # (default: false)
+  # statediffing call timeout period (in sec)
+  stateDiffTimeout = 240 # (default: 240)

-# Instructions for Testing
+[ethereum]
+  # node info
+  # path to json chain config (optional)
+  chainConfig = ""            # ETH_CHAIN_CONFIG
+  # eth chain id for config (overridden by chainConfig)
+  chainID = "1"               # ETH_CHAIN_ID (default: 1)
+  # http RPC endpoint URL for a statediffing node
+  httpPath = "localhost:8545" # ETH_HTTP_PATH

-Follow steps in [test/README.md](./test/README.md)
+[prom]
+  # prometheus metrics
+  metrics = true        # PROM_METRICS    (default: false)
+  http = true           # PROM_HTTP       (default: false)
+  httpAddr = "0.0.0.0"  # PROM_HTTP_ADDR  (default: 127.0.0.1)
+  httpPort = "9001"     # PROM_HTTP_PORT  (default: 9001)
+  dbStats = true        # PROM_DB_STATS   (default: false)

-# Code Overview
+[log]
+  # log level (trace, debug, info, warn, error, fatal, panic)
+  level = "info"  # LOG_LEVEL (default: info)
+  # file path for logging, leave unset to log to stdout
+  file  = ""      # LOG_FILE_PATH
+```

-This section will provide some insight into specific files and their purpose.

- `validator_test/chain_maker.go` - This file contains the code for creating a “test” blockchain.
- `validator_test/validator_test.go` - This file contains testing to validate the validator. It leverages `chain_maker.go` to create a blockchain to validate.
- `pkg/validator/validator.go` - This file contains most of the core logic for the validator.
+* The validation process trails behind the latest block number in the database by config parameter `validate.trail`.

-# Known Bugs
+* If the validator has caught up to (head-trail) height, it waits for a configured time interval (`validate.sleepInterval`) before again querying the database.

-1. The validator is improperly handling missing headers from the database.
-   1. Scenario
-      1. The IPLD blocks from the mock blockchain are inserted into the Postgres Data.
-      2. The validator runs, and all tests pass.
-      3. Users manually remove the last few rows from the database.
-      4. The validator runs, and all tests pass - This behavior is neither expected nor wanted.
+* If the validator encounters a missing block (gap) in the database, it makes a `writeStateDiffAt` call to the configured statediffing endpoint (`ethereum.httpPath`) if `validate.stateDiffMissingBlock` is set to `true`. Here it is assumed that the statediffing node pointed to is writing out to the database.

-# Tests on 03/03/22
+### Local Setup

-The tests highlighted below were conducted to validate the initial behavior of the validator.
+* Create a chain config file `chain.json` according to chain config in genesis json file used by local geth.

-## Set Up
+  Example:

-Below are the steps utilized to set up the test environment.
+  ```json
+  {
+    "chainId": 41337,
+    "homesteadBlock": 0,
+    "eip150Block": 0,
+    "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
+    "eip155Block": 0,
+    "eip158Block": 0,
+    "byzantiumBlock": 0,
+    "constantinopleBlock": 0,
+    "petersburgBlock": 0,
+    "istanbulBlock": 0,
+    "clique": {
+      "period": 5,
+      "epoch": 30000
+    }
+  }
+  ```

-1. Run the `scripts/run_integration_test.sh` script.
-   1. First comment outline 130 to 133 from `validator_test/validator_test.go`
-2. Once the code has completed running, comment out lines 55 to 126, 38 to 40, and 42 to 44.
-   1. Make the following change `db, err = setupDB() --> db, _ = setupDB()`
-3. Run the following command: `ginkgo -r validator_test/ -v`
-   1. All tests should pass
+  Provide the path to the above file in the config.

-## Testing Failures
+## Usage

-Once we had populated the database, we tested for failures.
+* Create / update the config file (refer to example config above).

-1. Removing a Transaction from `transaction_cids` - If we removed a transaction from the database and ran the test, the test would fail. **This is the expected behavior.**
-2. Removing Headers from `eth.header_cids`
-   1. If we removed a header block sandwiched between two header blocks, the test would fail (For example, we removed the entry for block 4, and the block range is 1-10). **This is the expected behavior.**
-   2. If we removed the tail block(s) from the table, the test would pass (For example, we remove the entry for blocks 8, 9, 10, and the block range is 1-10). **This is _not_ the expected behavior.**
+* Run validator:
+
+  ```bash
+  ./ipld-eth-db-validator stateValidator --config=<config path>
+  ```
+
+  Example:
+
+  ```bash
+  ./ipld-eth-db-validator stateValidator --config=environments/example.toml
+  ```
+
+## Monitoring
+
+* Enable metrics using config parameters `prom.metrics` and `prom.http`.
+* `ipld-eth-db-validator` exposes following prometheus metrics at `/metrics` endpoint:
+  * `last_validated_block`: Last validated block number.
+  * DB stats if `prom.dbStats` set to `true`.
+
+## Tests
+
+* Follow [Test Instructions](./test/README.md) to run unit and integration tests locally.
--- a/cmd/env.go
+++ b/cmd/env.go
@ -0,0 +1,84 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"github.com/spf13/viper"
+)
+
+const (
+	LOG_LEVEL     = "LOG_LEVEL"
+	LOG_FILE_PATH = "LOG_FILE_PATH"
+
+	PROM_METRICS   = "PROM_METRICS"
+	PROM_HTTP      = "PROM_HTTP"
+	PROM_HTTP_ADDR = "PROM_HTTP_ADDR"
+	PROM_HTTP_PORT = "PROM_HTTP_PORT"
+	PROM_DB_STATS  = "PROM_DB_STATS"
+
+	DATABASE_NAME     = "DATABASE_NAME"
+	DATABASE_HOSTNAME = "DATABASE_HOSTNAME"
+	DATABASE_PORT     = "DATABASE_PORT"
+	DATABASE_USER     = "DATABASE_USER"
+	DATABASE_PASSWORD = "DATABASE_PASSWORD"
+
+	DATABASE_MAX_IDLE_CONNECTIONS = "DATABASE_MAX_IDLE_CONNECTIONS"
+	DATABASE_MAX_OPEN_CONNECTIONS = "DATABASE_MAX_OPEN_CONNECTIONS"
+	DATABASE_MAX_CONN_LIFETIME    = "DATABASE_MAX_CONN_LIFETIME"
+
+	ETH_CHAIN_CONFIG = "ETH_CHAIN_CONFIG"
+	ETH_CHAIN_ID     = "ETH_CHAIN_ID"
+	ETH_HTTP_PATH    = "ETH_HTTP_PATH"
+
+	VALIDATE_BLOCK_HEIGHT            = "VALIDATE_BLOCK_HEIGHT"
+	VALIDATE_TRAIL                   = "VALIDATE_TRAIL"
+	VALIDATE_SLEEP_INTERVAL          = "VALIDATE_SLEEP_INTERVAL"
+	VALIDATE_STATEDIFF_MISSING_BLOCK = "VALIDATE_STATEDIFF_MISSING_BLOCK"
+	VALIDATE_STATEDIFF_TIMEOUT       = "VALIDATE_STATEDIFF_TIMEOUT"
+)
+
+// Bind env vars
+func init() {
+	viper.BindEnv("log.level", LOG_LEVEL)
+	viper.BindEnv("log.file", LOG_FILE_PATH)
+
+	viper.BindEnv("prom.metrics", PROM_METRICS)
+	viper.BindEnv("prom.http", PROM_HTTP)
+	viper.BindEnv("prom.httpAddr", PROM_HTTP_ADDR)
+	viper.BindEnv("prom.httpPort", PROM_HTTP_PORT)
+	viper.BindEnv("prom.dbStats", PROM_DB_STATS)
+
+	viper.BindEnv("database.name", DATABASE_NAME)
+	viper.BindEnv("database.hostname", DATABASE_HOSTNAME)
+	viper.BindEnv("database.port", DATABASE_PORT)
+	viper.BindEnv("database.user", DATABASE_USER)
+	viper.BindEnv("database.password", DATABASE_PASSWORD)
+
+	viper.BindEnv("database.maxIdle", DATABASE_MAX_IDLE_CONNECTIONS)
+	viper.BindEnv("database.maxOpen", DATABASE_MAX_OPEN_CONNECTIONS)
+	viper.BindEnv("database.maxLifetime", DATABASE_MAX_CONN_LIFETIME)
+
+	viper.BindEnv("ethereum.chainConfig", ETH_CHAIN_CONFIG)
+	viper.BindEnv("ethereum.chainID", ETH_CHAIN_ID)
+	viper.BindEnv("ethereum.httpPath", ETH_HTTP_PATH)
+
+	viper.BindEnv("validate.blockHeight", VALIDATE_BLOCK_HEIGHT)
+	viper.BindEnv("validate.trail", VALIDATE_TRAIL)
+	viper.BindEnv("validate.sleepInterval", VALIDATE_SLEEP_INTERVAL)
+	viper.BindEnv("validate.stateDiffMissingBlock", VALIDATE_STATEDIFF_MISSING_BLOCK)
+	viper.BindEnv("validate.stateDiffTimeout", VALIDATE_STATEDIFF_TIMEOUT)
+}
--- a/cmd/root.go
+++ b/cmd/root.go
@ -1,12 +1,31 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 package cmd

 import (
+	"fmt"
 	"os"
 	"strings"

 	log "github.com/sirupsen/logrus"
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
+
+	"github.com/vulcanize/ipld-eth-db-validator/pkg/prom"
 )

 var (
@ -31,7 +50,7 @@ func Execute() {
 }

 func initFunc(cmd *cobra.Command, args []string) {
-	logfile := viper.GetString("logfile")
+	logfile := viper.GetString("log.file")
 	if logfile != "" {
 		file, err := os.OpenFile(logfile,
 			os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
@ -49,6 +68,21 @@ func initFunc(cmd *cobra.Command, args []string) {
 	if err := logLevel(); err != nil {
 		log.Fatal("Could not set log level: ", err)
 	}
+
+	if viper.GetBool("prom.metrics") {
+		log.Info("initializing prometheus metrics")
+		prom.Init()
+	}
+
+	if viper.GetBool("prom.http") {
+		addr := fmt.Sprintf(
+			"%s:%s",
+			viper.GetString("prom.httpAddr"),
+			viper.GetString("prom.httpPort"),
+		)
+		log.Info("starting prometheus server")
+		prom.Serve(addr)
+	}
 }

 func init() {
@ -57,21 +91,33 @@ func init() {
 	viper.AutomaticEnv()

 	rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file location")
-	rootCmd.PersistentFlags().String("logfile", "", "file path for logging")
 	rootCmd.PersistentFlags().String("database-name", "vulcanize_public", "database name")
 	rootCmd.PersistentFlags().Int("database-port", 5432, "database port")
 	rootCmd.PersistentFlags().String("database-hostname", "localhost", "database hostname")
 	rootCmd.PersistentFlags().String("database-user", "", "database user")
 	rootCmd.PersistentFlags().String("database-password", "", "database password")
+	rootCmd.PersistentFlags().String("log-file", "", "file path for logging")
 	rootCmd.PersistentFlags().String("log-level", log.InfoLevel.String(), "Log level (trace, debug, info, warn, error, fatal, panic")

-	_ = viper.BindPFlag("logfile", rootCmd.PersistentFlags().Lookup("logfile"))
+	rootCmd.PersistentFlags().Bool("prom-metrics", false, "enable prometheus metrics")
+	rootCmd.PersistentFlags().Bool("prom-http", false, "enable prometheus http service")
+	rootCmd.PersistentFlags().String("prom-httpAddr", "127.0.0.1", "prometheus http host")
+	rootCmd.PersistentFlags().String("prom-httpPort", "9001", "prometheus http port")
+	rootCmd.PersistentFlags().Bool("prom-dbStats", false, "enables prometheus db stats")
+
 	_ = viper.BindPFlag("database.name", rootCmd.PersistentFlags().Lookup("database-name"))
 	_ = viper.BindPFlag("database.port", rootCmd.PersistentFlags().Lookup("database-port"))
 	_ = viper.BindPFlag("database.hostname", rootCmd.PersistentFlags().Lookup("database-hostname"))
 	_ = viper.BindPFlag("database.user", rootCmd.PersistentFlags().Lookup("database-user"))
 	_ = viper.BindPFlag("database.password", rootCmd.PersistentFlags().Lookup("database-password"))
+	_ = viper.BindPFlag("log.file", rootCmd.PersistentFlags().Lookup("log-file"))
 	_ = viper.BindPFlag("log.level", rootCmd.PersistentFlags().Lookup("log-level"))
+
+	_ = viper.BindPFlag("prom.metrics", rootCmd.PersistentFlags().Lookup("prom-metrics"))
+	_ = viper.BindPFlag("prom.http", rootCmd.PersistentFlags().Lookup("prom-http"))
+	_ = viper.BindPFlag("prom.httpAddr", rootCmd.PersistentFlags().Lookup("prom-httpAddr"))
+	_ = viper.BindPFlag("prom.httpPort", rootCmd.PersistentFlags().Lookup("prom-httpPort"))
+	_ = viper.BindPFlag("prom.dbStats", rootCmd.PersistentFlags().Lookup("prom-dbStats"))
 }

 func logLevel() error {
--- a/cmd/state_validator.go
+++ b/cmd/state_validator.go
@ -1,3 +1,19 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 package cmd

 import (
@ -52,14 +68,22 @@ func init() {
 	stateValidatorCmd.PersistentFlags().String("block-height", "1", "block height to initiate state validation")
 	stateValidatorCmd.PersistentFlags().String("trail", "16", "trail of block height to validate")
 	stateValidatorCmd.PersistentFlags().String("sleep-interval", "10", "sleep interval in seconds after validator has caught up to (head-trail) height")
+	stateValidatorCmd.PersistentFlags().Bool("statediff-missing-block", false, "whether to perform a statediffing call on a missing block")
+	stateValidatorCmd.PersistentFlags().Uint("statediff-timeout", 240, "statediffing call timeout period (in sec)")

-	stateValidatorCmd.PersistentFlags().String("chain-config", "", "path to chain config")
+	stateValidatorCmd.PersistentFlags().String("eth-chain-config", "", "path to json chain config")
+	stateValidatorCmd.PersistentFlags().String("eth-chain-id", "1", "eth chain id")
+	stateValidatorCmd.PersistentFlags().String("eth-http-path", "", "http url for a statediffing node")

-	_ = viper.BindPFlag("validate.block-height", stateValidatorCmd.PersistentFlags().Lookup("block-height"))
+	_ = viper.BindPFlag("validate.blockHeight", stateValidatorCmd.PersistentFlags().Lookup("block-height"))
 	_ = viper.BindPFlag("validate.trail", stateValidatorCmd.PersistentFlags().Lookup("trail"))
 	_ = viper.BindPFlag("validate.sleepInterval", stateValidatorCmd.PersistentFlags().Lookup("sleep-interval"))
+	_ = viper.BindPFlag("validate.stateDiffMissingBlock", stateValidatorCmd.PersistentFlags().Lookup("statediff-missing-block"))
+	_ = viper.BindPFlag("validate.stateDiffTimeout", stateValidatorCmd.PersistentFlags().Lookup("statediff-timeout"))

-	_ = viper.BindPFlag("ethereum.chainConfig", stateValidatorCmd.PersistentFlags().Lookup("chain-config"))
+	_ = viper.BindPFlag("ethereum.chainConfig", stateValidatorCmd.PersistentFlags().Lookup("eth-chain-config"))
+	_ = viper.BindPFlag("ethereum.chainID", stateValidatorCmd.PersistentFlags().Lookup("eth-chain-id"))
+	_ = viper.BindPFlag("ethereum.httpPath", stateValidatorCmd.PersistentFlags().Lookup("eth-http-path"))
 }

 func initConfig() {
--- a/environments/example.toml
+++ b/environments/example.toml
@ -6,9 +6,24 @@
    user     = "vdbm"

 [validate]
-    block-height = 1
+    blockHeight = 1
    trail  = 16
    sleepInterval = 10
+    stateDiffMissingBlock = true
+    stateDiffTimeout = 240

 [ethereum]
-    chainConfig = "./chain.json"
+    chainConfig = ""
+    chainID = "1"
+    httpPath = "localhost:8545"
+
+[prom]
+    metrics = true
+    http = true
+    httpAddr = "localhost"
+    httpPort = "9001"
+    dbStats = true
+
+[log]
+    file = ""
+    level = "info"
--- a/go.mod
+++ b/go.mod
@ -7,11 +7,12 @@ require (
 	github.com/jmoiron/sqlx v1.3.5
 	github.com/onsi/ginkgo v1.16.5
 	github.com/onsi/gomega v1.19.0
+	github.com/prometheus/client_golang v1.12.1
 	github.com/sirupsen/logrus v1.8.1
 	github.com/spf13/cobra v1.4.0
 	github.com/spf13/viper v1.11.0
-	github.com/vulcanize/ipfs-ethdb/v4 v4.0.5-alpha
-	github.com/vulcanize/ipld-eth-server/v4 v4.1.4-alpha
+	github.com/vulcanize/ipfs-ethdb/v4 v4.0.6-alpha
+	github.com/vulcanize/ipld-eth-server/v4 v4.1.5-alpha
 )

 require (
@ -206,7 +207,6 @@ require (
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/polydawn/refmt v0.0.0-20201211092308-30ac6d18308e // indirect
-	github.com/prometheus/client_golang v1.12.1 // indirect
 	github.com/prometheus/client_model v0.2.0 // indirect
 	github.com/prometheus/common v0.33.0 // indirect
 	github.com/prometheus/procfs v0.7.3 // indirect
@ -256,7 +256,7 @@ require (
 	go.uber.org/multierr v1.8.0 // indirect
 	go.uber.org/zap v1.21.0 // indirect
 	go4.org v0.0.0-20200411211856-f5505b9728dd // indirect
-	golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e // indirect
+	golang.org/x/crypto v0.0.0-20220817201139-bc19a97f63c8 // indirect
 	golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 // indirect
 	golang.org/x/net v0.0.0-20220607020251-c690dde0001d // indirect
 	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect
--- a/go.sum
+++ b/go.sum
@ -1678,10 +1678,10 @@ github.com/vulcanize/eth-ipfs-state-validator/v4 v4.0.6-alpha h1:cr039FSz9KUKt6+
 github.com/vulcanize/eth-ipfs-state-validator/v4 v4.0.6-alpha/go.mod h1:yd90/EemgaKlmy+rhoTVDtQqwiStNnBi4mDo27oPcoI=
 github.com/vulcanize/go-ethereum v1.10.21-statediff-4.1.2-alpha h1:ct+8FGuQnHA6SOGOQoAMINWdeexuSF40+IjF48J094A=
 github.com/vulcanize/go-ethereum v1.10.21-statediff-4.1.2-alpha/go.mod h1:dNJkmCSbaasX0zfQM6pm1g3rWlW3EGhLOEZMScyrRAs=
-github.com/vulcanize/ipfs-ethdb/v4 v4.0.5-alpha h1:NFRwWeMB3Q+QqLM9qdcHvfvWBxOk0lPwhOqXJpkIg30=
-github.com/vulcanize/ipfs-ethdb/v4 v4.0.5-alpha/go.mod h1:WvYj0m0cLPAtoytTbcbE2nZ3Hg9iuuF+lY14dBVRWZQ=
-github.com/vulcanize/ipld-eth-server/v4 v4.1.4-alpha h1:r/unaDcJKHzQC9gCOXNs5YhYBiIgG1w2eA7pQVl/IdE=
-github.com/vulcanize/ipld-eth-server/v4 v4.1.4-alpha/go.mod h1:bL5EeJrHQQoXCFc7rN611dXho3ahuVQNqUJoO6e8NO4=
+github.com/vulcanize/ipfs-ethdb/v4 v4.0.6-alpha h1:iKpv+Bvc0HScak+NiGK4NeYGLWMZ1pyLmrZecHoUGYA=
+github.com/vulcanize/ipfs-ethdb/v4 v4.0.6-alpha/go.mod h1:WvYj0m0cLPAtoytTbcbE2nZ3Hg9iuuF+lY14dBVRWZQ=
+github.com/vulcanize/ipld-eth-server/v4 v4.1.5-alpha h1:qeD4RCz9nOhdyRNw+L/5u6jNE+xgTd6rlgpYtTa349g=
+github.com/vulcanize/ipld-eth-server/v4 v4.1.5-alpha/go.mod h1:GssD69QmLIFH419aiz5ywTUH9XRGK9gUbL4n4UTvycA=
 github.com/wI2L/jsondiff v0.2.0 h1:dE00WemBa1uCjrzQUUTE/17I6m5qAaN0EMFOg2Ynr/k=
 github.com/wI2L/jsondiff v0.2.0/go.mod h1:axTcwtBkY4TsKuV+RgoMhHyHKKFRI6nnjRLi8LLYQnA=
 github.com/wangjia184/sortedset v0.0.0-20160527075905-f5d03557ba30/go.mod h1:YkocrP2K2tcw938x9gCOmT5G5eCD6jsTz0SZuyAqwIE=
@ -1849,8 +1849,8 @@ golang.org/x/crypto v0.0.0-20210813211128-0a44fdfbc16e/go.mod h1:GvvjBRRGRdwPK5y
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20220518034528-6f7dac969898/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
-golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e h1:T8NU3HyQ8ClP4SEE+KbFlg6n0NhuTsN4MyznaarGsZM=
-golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/crypto v0.0.0-20220817201139-bc19a97f63c8 h1:GIAS/yBem/gq2MUqgNIzUHW7cJMmx3TGZOrnyYaNQ6c=
+golang.org/x/crypto v0.0.0-20220817201139-bc19a97f63c8/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
--- a/main.go
+++ b/main.go
@ -1,3 +1,19 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 package main

 import "github.com/vulcanize/ipld-eth-db-validator/cmd"
--- a/pkg/prom/db_stats_collector.go
+++ b/pkg/prom/db_stats_collector.go
@ -0,0 +1,157 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package prom
+
+import (
+	"database/sql"
+
+	"github.com/prometheus/client_golang/prometheus"
+)
+
+// DBStatsGetter is an interface that gets sql.DBStats.
+type DBStatsGetter interface {
+	Stats() sql.DBStats
+}
+
+// DBStatsCollector implements the prometheus.Collector interface.
+type DBStatsCollector struct {
+	sg DBStatsGetter
+
+	// descriptions of exported metrics
+	maxOpenDesc           *prometheus.Desc
+	openDesc              *prometheus.Desc
+	inUseDesc             *prometheus.Desc
+	idleDesc              *prometheus.Desc
+	waitedForDesc         *prometheus.Desc
+	blockedSecondsDesc    *prometheus.Desc
+	closedMaxIdleDesc     *prometheus.Desc
+	closedMaxLifetimeDesc *prometheus.Desc
+}
+
+// NewDBStatsCollector creates a new DBStatsCollector.
+func NewDBStatsCollector(dbName string, sg DBStatsGetter) *DBStatsCollector {
+	labels := prometheus.Labels{"db_name": dbName}
+	return &DBStatsCollector{
+		sg: sg,
+		maxOpenDesc: prometheus.NewDesc(
+			prometheus.BuildFQName(namespace, connSubsystem, "max_open"),
+			"Maximum number of open connections to the database.",
+			nil,
+			labels,
+		),
+		openDesc: prometheus.NewDesc(
+			prometheus.BuildFQName(namespace, connSubsystem, "open"),
+			"The number of established connections both in use and idle.",
+			nil,
+			labels,
+		),
+		inUseDesc: prometheus.NewDesc(
+			prometheus.BuildFQName(namespace, connSubsystem, "in_use"),
+			"The number of connections currently in use.",
+			nil,
+			labels,
+		),
+		idleDesc: prometheus.NewDesc(
+			prometheus.BuildFQName(namespace, connSubsystem, "idle"),
+			"The number of idle connections.",
+			nil,
+			labels,
+		),
+		waitedForDesc: prometheus.NewDesc(
+			prometheus.BuildFQName(namespace, connSubsystem, "waited_for"),
+			"The total number of connections waited for.",
+			nil,
+			labels,
+		),
+		blockedSecondsDesc: prometheus.NewDesc(
+			prometheus.BuildFQName(namespace, connSubsystem, "blocked_seconds"),
+			"The total time blocked waiting for a new connection.",
+			nil,
+			labels,
+		),
+		closedMaxIdleDesc: prometheus.NewDesc(
+			prometheus.BuildFQName(namespace, connSubsystem, "closed_max_idle"),
+			"The total number of connections closed due to SetMaxIdleConns.",
+			nil,
+			labels,
+		),
+		closedMaxLifetimeDesc: prometheus.NewDesc(
+			prometheus.BuildFQName(namespace, connSubsystem, "closed_max_lifetime"),
+			"The total number of connections closed due to SetConnMaxLifetime.",
+			nil,
+			labels,
+		),
+	}
+}
+
+// Describe implements the prometheus.Collector interface.
+func (c DBStatsCollector) Describe(ch chan<- *prometheus.Desc) {
+	ch <- c.maxOpenDesc
+	ch <- c.openDesc
+	ch <- c.inUseDesc
+	ch <- c.idleDesc
+	ch <- c.waitedForDesc
+	ch <- c.blockedSecondsDesc
+	ch <- c.closedMaxIdleDesc
+	ch <- c.closedMaxLifetimeDesc
+}
+
+// Collect implements the prometheus.Collector interface.
+func (c DBStatsCollector) Collect(ch chan<- prometheus.Metric) {
+	stats := c.sg.Stats()
+
+	ch <- prometheus.MustNewConstMetric(
+		c.maxOpenDesc,
+		prometheus.GaugeValue,
+		float64(stats.MaxOpenConnections),
+	)
+	ch <- prometheus.MustNewConstMetric(
+		c.openDesc,
+		prometheus.GaugeValue,
+		float64(stats.OpenConnections),
+	)
+	ch <- prometheus.MustNewConstMetric(
+		c.inUseDesc,
+		prometheus.GaugeValue,
+		float64(stats.InUse),
+	)
+	ch <- prometheus.MustNewConstMetric(
+		c.idleDesc,
+		prometheus.GaugeValue,
+		float64(stats.Idle),
+	)
+	ch <- prometheus.MustNewConstMetric(
+		c.waitedForDesc,
+		prometheus.CounterValue,
+		float64(stats.WaitCount),
+	)
+	ch <- prometheus.MustNewConstMetric(
+		c.blockedSecondsDesc,
+		prometheus.CounterValue,
+		stats.WaitDuration.Seconds(),
+	)
+	ch <- prometheus.MustNewConstMetric(
+		c.closedMaxIdleDesc,
+		prometheus.CounterValue,
+		float64(stats.MaxIdleClosed),
+	)
+	ch <- prometheus.MustNewConstMetric(
+		c.closedMaxLifetimeDesc,
+		prometheus.CounterValue,
+		float64(stats.MaxLifetimeClosed),
+	)
+}
--- a/pkg/prom/prom.go
+++ b/pkg/prom/prom.go
@ -0,0 +1,59 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package prom
+
+import (
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promauto"
+)
+
+const (
+	namespace = "ipld_eth_state_snapshot"
+
+	connSubsystem  = "connections"
+	statsSubsystem = "stats"
+)
+
+var (
+	metrics            bool
+	lastValidatedBlock prometheus.Gauge
+)
+
+func Init() {
+	metrics = true
+
+	lastValidatedBlock = promauto.NewGauge(prometheus.GaugeOpts{
+		Namespace: namespace,
+		Subsystem: statsSubsystem,
+		Name:      "last_validated_block",
+		Help:      "Last validated block number",
+	})
+}
+
+// RegisterDBCollector create metric collector for given connection
+func RegisterDBCollector(name string, db DBStatsGetter) {
+	if metrics {
+		prometheus.Register(NewDBStatsCollector(name, db))
+	}
+}
+
+// SetLastValidatedBlock sets the last validated block number
+func SetLastValidatedBlock(blockNumber float64) {
+	if metrics {
+		lastValidatedBlock.Set(blockNumber)
+	}
+}
--- a/pkg/prom/serve.go
+++ b/pkg/prom/serve.go
@ -0,0 +1,47 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package prom
+
+import (
+	"errors"
+	"net/http"
+
+	"github.com/prometheus/client_golang/prometheus/promhttp"
+	"github.com/sirupsen/logrus"
+)
+
+var errPromHTTP = errors.New("can't start http server for prometheus")
+
+// Serve start listening http
+func Serve(addr string) *http.Server {
+	mux := http.NewServeMux()
+	mux.Handle("/metrics", promhttp.Handler())
+	srv := http.Server{
+		Addr:    addr,
+		Handler: mux,
+	}
+	go func() {
+		if err := srv.ListenAndServe(); err != nil {
+			logrus.
+				WithError(err).
+				WithField("module", "prom").
+				WithField("addr", addr).
+				Fatal(errPromHTTP)
+		}
+	}()
+	return &srv
+}
--- a/pkg/validator/config.go
+++ b/pkg/validator/config.go
@ -1,3 +1,19 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 package validator

 import (
@ -6,22 +22,14 @@ import (
 	"time"

 	"github.com/ethereum/go-ethereum/params"
+	"github.com/ethereum/go-ethereum/rpc"
 	"github.com/ethereum/go-ethereum/statediff"
 	"github.com/ethereum/go-ethereum/statediff/indexer/database/sql/postgres"
 	"github.com/jmoiron/sqlx"
 	"github.com/spf13/viper"
 	"github.com/vulcanize/ipld-eth-server/v4/pkg/shared"
-)

-var (
-	DATABASE_NAME                 = "DATABASE_NAME"
-	DATABASE_HOSTNAME             = "DATABASE_HOSTNAME"
-	DATABASE_PORT                 = "DATABASE_PORT"
-	DATABASE_USER                 = "DATABASE_USER"
-	DATABASE_PASSWORD             = "DATABASE_PASSWORD"
-	DATABASE_MAX_IDLE_CONNECTIONS = "DATABASE_MAX_IDLE_CONNECTIONS"
-	DATABASE_MAX_OPEN_CONNECTIONS = "DATABASE_MAX_OPEN_CONNECTIONS"
-	DATABASE_MAX_CONN_LIFETIME    = "DATABASE_MAX_CONN_LIFETIME"
+	"github.com/vulcanize/ipld-eth-db-validator/pkg/prom"
 )

 var IntegrationTestChainConfig = &params.ChainConfig{
@ -62,6 +70,9 @@ type Config struct {
 	DB       *sqlx.DB

 	ChainCfg              *params.ChainConfig
+	Client                *rpc.Client
+	StateDiffMissingBlock bool
+	StateDiffTimeout      uint

 	BlockNum, Trail uint64
 	SleepInterval   uint
@ -74,16 +85,12 @@ func NewConfig() (*Config, error) {
 		return nil, err
 	}

-	cfg.BlockNum = viper.GetUint64("validate.block-height")
-	if cfg.BlockNum < 1 {
-		return nil, fmt.Errorf("block height cannot be less the 1")
+	err = cfg.setupEth()
+	if err != nil {
+		return nil, err
 	}

-	cfg.Trail = viper.GetUint64("validate.trail")
-	cfg.SleepInterval = viper.GetUint("validate.sleepInterval")
-
-	chainConfigPath := viper.GetString("ethereum.chainConfig")
-	cfg.ChainCfg, err = statediff.LoadConfig(chainConfigPath)
+	err = cfg.setupValidator()
 	if err != nil {
 		return nil, err
 	}
@ -92,15 +99,6 @@ func NewConfig() (*Config, error) {
 }

 func (c *Config) setupDB() error {
-	_ = viper.BindEnv("database.name", DATABASE_NAME)
-	_ = viper.BindEnv("database.hostname", DATABASE_HOSTNAME)
-	_ = viper.BindEnv("database.port", DATABASE_PORT)
-	_ = viper.BindEnv("database.user", DATABASE_USER)
-	_ = viper.BindEnv("database.password", DATABASE_PASSWORD)
-	_ = viper.BindEnv("database.maxIdle", DATABASE_MAX_IDLE_CONNECTIONS)
-	_ = viper.BindEnv("database.maxOpen", DATABASE_MAX_OPEN_CONNECTIONS)
-	_ = viper.BindEnv("database.maxLifetime", DATABASE_MAX_CONN_LIFETIME)
-
 	// DB Config
 	c.dbConfig.DatabaseName = viper.GetString("database.name")
 	c.dbConfig.Hostname = viper.GetString("database.hostname")
@ -117,7 +115,53 @@ func (c *Config) setupDB() error {
 	if err != nil {
 		return fmt.Errorf("failed to create config: %w", err)
 	}
-
 	c.DB = db
+
+	// Enable DB stats
+	if viper.GetBool("prom.dbStats") {
+		prom.RegisterDBCollector(c.dbConfig.DatabaseName, c.DB)
+	}
+
 	return nil
 }
+
+func (c *Config) setupEth() error {
+	var err error
+	chainConfigPath := viper.GetString("ethereum.chainConfig")
+	if chainConfigPath != "" {
+		c.ChainCfg, err = statediff.LoadConfig(chainConfigPath)
+	} else {
+		// read chainID if chain config path not provided
+		chainID := viper.GetUint64("ethereum.chainID")
+		c.ChainCfg, err = statediff.ChainConfig(chainID)
+	}
+	if err != nil {
+		return err
+	}
+
+	// setup a statediffing client
+	ethHTTP := viper.GetString("ethereum.httpPath")
+	if ethHTTP != "" {
+		ethHTTPEndpoint := fmt.Sprintf("http://%s", ethHTTP)
+		c.Client, err = rpc.Dial(ethHTTPEndpoint)
+	}
+
+	return err
+}
+
+func (c *Config) setupValidator() error {
+	var err error
+	c.BlockNum = viper.GetUint64("validate.blockHeight")
+	if c.BlockNum < 1 {
+		return fmt.Errorf("block height cannot be less the 1")
+	}
+
+	c.Trail = viper.GetUint64("validate.trail")
+	c.SleepInterval = viper.GetUint("validate.sleepInterval")
+	c.StateDiffMissingBlock = viper.GetBool("validate.stateDiffMissingBlock")
+	if c.StateDiffMissingBlock {
+		c.StateDiffTimeout = viper.GetUint("validate.stateDiffTimeout")
+	}
+
+	return err
+}
--- a/pkg/validator/database.go
+++ b/pkg/validator/database.go
@ -1,3 +1,19 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 package validator

 import (
--- a/pkg/validator/ref_integrity.go
+++ b/pkg/validator/ref_integrity.go
@ -1,3 +1,19 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 package validator

 import (
@ -8,7 +24,6 @@ import (

 // ValidateReferentialIntegrity validates referential integrity at the given height
 func ValidateReferentialIntegrity(db *sqlx.DB, blockNumber uint64) error {
-
 	err := ValidateHeaderCIDsRef(db, blockNumber)
 	if err != nil {
 		return err
--- a/pkg/validator/ref_integrity_queries.go
+++ b/pkg/validator/ref_integrity_queries.go
@ -1,3 +1,19 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 package validator

 // Queries to validate referential integrity in the indexed data:
--- a/pkg/validator/validator.go
+++ b/pkg/validator/validator.go
@ -1,7 +1,24 @@
+// VulcanizeDB
+// Copyright © 2022 Vulcanize
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 package validator

 import (
 	"context"
+	"encoding/json"
 	"errors"
 	"fmt"
 	"math/big"
@ -18,12 +35,15 @@ import (
 	"github.com/ethereum/go-ethereum/core/vm"
 	"github.com/ethereum/go-ethereum/params"
 	"github.com/ethereum/go-ethereum/rpc"
+	"github.com/ethereum/go-ethereum/statediff"
 	"github.com/jmoiron/sqlx"
 	log "github.com/sirupsen/logrus"

 	ipfsethdb "github.com/vulcanize/ipfs-ethdb/v4/postgres"
 	ipldEth "github.com/vulcanize/ipld-eth-server/v4/pkg/eth"
 	ethServerShared "github.com/vulcanize/ipld-eth-server/v4/pkg/shared"
+
+	"github.com/vulcanize/ipld-eth-db-validator/pkg/prom"
 )

 var (
@ -38,8 +58,12 @@ type service struct {
 	db              *sqlx.DB
 	blockNum, trail uint64
 	sleepInterval   uint
-	logger          *log.Logger
 	chainCfg        *params.ChainConfig
+
+	stateDiffMissingBlock bool
+	stateDiffTimeout      uint
+	ethClient             *rpc.Client
+
 	quitChan     chan bool
 	progressChan chan uint64
 }
@ -50,8 +74,10 @@ func NewService(cfg *Config, progressChan chan uint64) *service {
 		blockNum:              cfg.BlockNum,
 		trail:                 cfg.Trail,
 		sleepInterval:         cfg.SleepInterval,
-		logger:        log.New(),
 		chainCfg:              cfg.ChainCfg,
+		stateDiffMissingBlock: cfg.StateDiffMissingBlock,
+		stateDiffTimeout:      cfg.StateDiffTimeout,
+		ethClient:             cfg.Client,
 		quitChan:              make(chan bool),
 		progressChan:          progressChan,
 	}
@ -92,7 +118,7 @@ func (s *service) Start(ctx context.Context, wg *sync.WaitGroup) {

 	api, err := EthAPI(ctx, s.db, s.chainCfg)
 	if err != nil {
-		s.logger.Fatal(err)
+		log.Fatal(err)
 		return
 	}

@ -101,7 +127,7 @@ func (s *service) Start(ctx context.Context, wg *sync.WaitGroup) {
 	for {
 		select {
 		case <-s.quitChan:
-			s.logger.Infof("last validated block %v", idxBlockNum-1)
+			log.Infof("last validated block %v", idxBlockNum-1)
 			if s.progressChan != nil {
 				close(s.progressChan)
 			}
@ -109,17 +135,19 @@ func (s *service) Start(ctx context.Context, wg *sync.WaitGroup) {
 		default:
 			idxBlockNum, err = s.Validate(ctx, api, idxBlockNum)
 			if err != nil {
-				s.logger.Infof("last validated block %v", idxBlockNum-1)
-				s.logger.Fatal(err)
+				log.Infof("last validated block %v", idxBlockNum-1)
+				log.Fatal(err)
 				return
 			}
+
+			prom.SetLastValidatedBlock(float64(idxBlockNum))
 		}
 	}
 }

 // Stop is used to gracefully stop the service
 func (s *service) Stop() {
-	s.logger.Info("stopping ipld-eth-db-validator process")
+	log.Info("stopping ipld-eth-db-validator process")
 	close(s.quitChan)
 }

@ -131,20 +159,32 @@ func (s *service) Validate(ctx context.Context, api *ipldEth.PublicEthAPI, idxBl

 	// Check if it block at height idxBlockNum can be validated
 	if idxBlockNum <= headBlockNum-s.trail {
-		err = ValidateBlock(ctx, api, idxBlockNum)
+		blockToBeValidated, err := api.B.BlockByNumber(ctx, rpc.BlockNumber(idxBlockNum))
 		if err != nil {
-			s.logger.Errorf("failed to verify state root at block %d", idxBlockNum)
+			log.Errorf("failed to fetch block at height %d", idxBlockNum)
 			return idxBlockNum, err
 		}

-		s.logger.Infof("state root verified for block %d", idxBlockNum)
+		// Make a writeStateDiffAt call if block not found in the db
+		if blockToBeValidated == nil {
+			err = s.writeStateDiffAt(idxBlockNum)
+			return idxBlockNum, err
+		}
+
+		err = ValidateBlock(blockToBeValidated, api.B, idxBlockNum)
+		if err != nil {
+			log.Errorf("failed to verify state root at block %d", idxBlockNum)
+			return idxBlockNum, err
+		}
+
+		log.Infof("state root verified for block %d", idxBlockNum)

 		err = ValidateReferentialIntegrity(s.db, idxBlockNum)
 		if err != nil {
-			s.logger.Errorf("failed to verify referential integrity at block %d", idxBlockNum)
+			log.Errorf("failed to verify referential integrity at block %d", idxBlockNum)
 			return idxBlockNum, err
 		}
-		s.logger.Infof("referential integrity verified for block %d", idxBlockNum)
+		log.Infof("referential integrity verified for block %d", idxBlockNum)

 		if s.progressChan != nil {
 			s.progressChan <- idxBlockNum
@ -159,14 +199,37 @@ func (s *service) Validate(ctx context.Context, api *ipldEth.PublicEthAPI, idxBl
 	return idxBlockNum, nil
 }

-// ValidateBlock validates block at the given height
-func ValidateBlock(ctx context.Context, api *ipldEth.PublicEthAPI, blockNumber uint64) error {
-	blockToBeValidated, err := api.B.BlockByNumber(ctx, rpc.BlockNumber(blockNumber))
-	if err != nil {
+// writeStateDiffAt calls out to a statediffing geth client to fill in a gap in the index
+func (s *service) writeStateDiffAt(height uint64) error {
+	if !s.stateDiffMissingBlock {
+		return nil
+	}
+
+	var data json.RawMessage
+	params := statediff.Params{
+		IntermediateStateNodes:   true,
+		IntermediateStorageNodes: true,
+		IncludeBlock:             true,
+		IncludeReceipts:          true,
+		IncludeTD:                true,
+		IncludeCode:              true,
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(s.stateDiffTimeout*uint(time.Second)))
+	defer cancel()
+
+	log.Warnf("making writeStateDiffAt call at height %d", height)
+	if err := s.ethClient.CallContext(ctx, &data, "statediff_writeStateDiffAt", height, params); err != nil {
+		log.Errorf("writeStateDiffAt %d faild with err %s", height, err.Error())
 		return err
 	}

-	stateDB, err := applyTransaction(blockToBeValidated, api.B)
+	return nil
+}
+
+// ValidateBlock validates block at the given height
+func ValidateBlock(blockToBeValidated *types.Block, b *ipldEth.Backend, blockNumber uint64) error {
+	stateDB, err := applyTransaction(blockToBeValidated, b)
 	if err != nil {
 		return err
 	}
--- a/pkg/validator/validator_suite_test.go
+++ b/pkg/validator/validator_suite_test.go
@ -1,13 +0,0 @@
-package validator_test
-
-import (
-	"testing"
-
-	. "github.com/onsi/ginkgo"
-	. "github.com/onsi/gomega"
-)
-
-func TestValidator(t *testing.T) {
-	RegisterFailHandler(Fail)
-	RunSpecs(t, "Validator Suite")
-}
--- a/test/README.md
+++ b/test/README.md
@ -6,18 +6,18 @@

  - Clone [stack-orchestrator](https://github.com/vulcanize/stack-orchestrator), [go-ethereum](https://github.com/vulcanize/go-ethereum) and [ipld-eth-db](https://github.com/vulcanize/ipld-eth-db) repositories.

-  - Checkout [v4 release](https://github.com/vulcanize/ipld-eth-db/releases/tag/v4.2.0-alpha) in ipld-eth-db repo.
+  - Checkout [v4 release](https://github.com/vulcanize/ipld-eth-db/releases/tag/v4.2.1-alpha) in ipld-eth-db repo.

    ```bash
    # In ipld-eth-db repo.
-    git checkout v4.2.0-alpha
+    git checkout v4.2.1-alpha
    ```

-  - Checkout [v4 release](https://github.com/vulcanize/go-ethereum/releases/tag/v1.10.19-statediff-4.1.0-alpha) in go-ethereum repo.
+  - Checkout [v4 release](https://github.com/vulcanize/go-ethereum/releases/tag/v1.10.21-statediff-4.1.2-alpha) in go-ethereum repo.

    ```bash
    # In go-ethereum repo.
-    git checkout v1.10.19-statediff-4.1.0-alpha
+    git checkout v1.10.21-statediff-4.1.2-alpha
    ```

  - Checkout working commit in stack-orchestrator repo.
--- a/validator_test/ref_integrity_test.go
+++ b/validator_test/ref_integrity_test.go
--- a/validator_test/validator_suite_test.go
+++ b/validator_test/validator_suite_test.go
@ -1,4 +1,4 @@
-package validator_test_test
+package validator_test

 import (
 	"io/ioutil"
--- a/validator_test/validator_test.go
+++ b/validator_test/validator_test.go
@ -7,6 +7,7 @@ import (
 	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/core"
 	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/rpc"
 	"github.com/ethereum/go-ethereum/statediff"
 	sdtypes "github.com/ethereum/go-ethereum/statediff/types"
 	"github.com/jmoiron/sqlx"
@ -118,7 +119,11 @@ var _ = Describe("eth state reading tests", func() {
 			Expect(err).ToNot(HaveOccurred())

 			for i := uint64(blockHeight); i <= chainLength-trail; i++ {
-				err = validator.ValidateBlock(context.Background(), api, i)
+				blockToBeValidated, err := api.B.BlockByNumber(context.Background(), rpc.BlockNumber(i))
+				Expect(err).ToNot(HaveOccurred())
+				Expect(blockToBeValidated).ToNot(BeNil())
+
+				err = validator.ValidateBlock(blockToBeValidated, api.B, i)
 				Expect(err).ToNot(HaveOccurred())

 				err = validator.ValidateReferentialIntegrity(db, i)