# Index missing blocks with eth-statediff-service

This readme can be followed to index required blocks out of order for a contract. This indexed data can then be used by the watcher further.

* For indexing the required blocks the following core services will be used:

  * [ipld-eth-db](https://github.com/vulcanize/ipld-eth-db)

    * Run ipld-eth-db database using docker:

      ```bash
      docker-compose -f docker-compose.yml up
      ```
    
  * [leveldb-ethdb-rpc](https://github.com/vulcanize/leveldb-ethdb-rpc)

    It is an RPC wrapper around LevelDB. The endpoint can be used by eth-statediff-service to access LevelDB.

  * [eth-statediff-service](https://github.com/vulcanize/eth-statediff-service)

    * The [config file](https://github.com/vulcanize/eth-statediff-service/blob/sharding/environments/config.toml) can be updated with the following for running eth-statediff-service:

      ```toml
      [leveldb]
        mode = "remote"
        # leveldb-ethdb-rpc endpoint
        url = "http://127.0.0.1:8082/"

      [server]
        httpPath = "0.0.0.0:8545"

      [statediff]
        prerun = false
        serviceWorkers = 2
        workerQueueSize = 1024
        trieWorkers = 16

      [log]
        level = "info"

      [database]
        # Credentials for ipld-eth-db database
        name     = "vulcanize_testing"
        hostname = "localhost"
        port     = 8077
        user     = "vdbm"
        password = "password"
        type = "postgres"
        driver = "sqlx"

      [cache]
        database = 1024
        trie = 4096

      [ethereum]
        # Config for mainnet
        nodeID = "1"
        clientName = "eth-statediff-service"
        networkID = 1
        chainID = 1
      ```

    * Run eth-statediff-service:

      ```bash
      make build && ./eth-statediff-service serve --config environments/config.toml
      ```
  
* Indexing required blocks can be done in the following way:

  * Call `writeStateDiffAt` API with watched addresses for required blocks:

    ```bash
    # Replace $BLOCK_NUMBER with required block number to index and $CONTRACT_ADDRESS with the contract of interest.
    curl -X POST -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"statediff_writeStateDiffAt","params":[$BLOCK_NUMBER, {"intermediateStateNodes":true,"intermediateStorageNodes":true,"includeBlock":true,"includeReceipts":true,"includeTD":true,"includeCode":true,"watchedAddresses":["$CONTRACT_ADDRESS"]}],"id":1}' "127.0.0.1":"8545"
    ```

    Example for indexing [mainnet MobyMask blocks](https://etherscan.io/address/0xb06e6db9288324738f04fcaac910f5a60102c1f8):
    - 14869713
    - 14875233
    - 14876405
    - 14884873
    - 14885755

    ```bash
    curl -X POST -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"statediff_writeStateDiffAt","params":[14869713, {"intermediateStateNodes":true,"intermediateStorageNodes":true,"includeBlock":true,"includeReceipts":true,"includeTD":true,"includeCode":true,"watchedAddresses":["0xB06E6DB9288324738f04fCAAc910f5A60102C1F8"]}],"id":1}' "127.0.0.1":"8545"
    ```

    After successfully completing writeStateDiffAt for a block the returned response is:

    ```bash
    curl: (52) Empty reply from server
    ```

    **NOTE**: Using remote leveldb-ethdb-rpc takes long time (6-20 minutes).

  * Stop the eth-statediff-service after all required blocks are indexed.

* Start the [ipld-eth-server](https://github.com/vulcanize/eth-statediff-service) to query the indexed data from watcher.

  * Create the following config.toml file for ipld-eth-server in [environments directory](https://github.com/vulcanize/ipld-eth-server/tree/sharding/environments):

    ```toml
    [database]
      # Credentials for ipld-eth-db database
      name     = "vulcanize_testing" # $DATABASE_NAME
      hostname = "localhost" # $DATABASE_HOSTNAME
      port     = 8077 # $DATABASE_PORT
      user     = "vdbm" # $DATABASE_USER
      password = "password" # $DATABASE_PASSWORD

    [log]
      level = "info" # $LOGRUS_LEVEL

    [ethereum]
      # Config for mainnet
      chainID = "1" # $ETH_CHAIN_ID
      nodeID = "arch1" # $ETH_NODE_ID
      clientName = "Geth" # $ETH_CLIENT_NAME
      networkID = "1" # $ETH_NETWORK_ID
    ```
  
  * Run the server with the config above:

    ```bash
    make build && ./ipld-eth-server serve --config=./environments/config.toml --eth-server-graphql --log-level info
    ```

* The following steps are for indexing blocks out of order in the watcher:

  * Follow [steps in the readme](./README.md#setup) to setup the watcher.

  * Watch the contract:

    ```bash
    # Replace $CONTRACT_ADDRESS and $CONTRACT_NAME witch actual values
    yarn watch:contract --address $CONTRACT_ADDRESS --kind $CONTRACT_NAME --checkpoint true
    
    # Example for mobymask-watcher
    yarn watch:contract --address 0xB06E6DB9288324738f04fCAAc910f5A60102C1F8 --kind PhisherRegistry --checkpoint true
    ```

  * Index the required blocks. They should be the same blocks indexed by eth-statediff-service above.

    ```bash
    # Replace $BLOCK_NUMBER with required block number to index
    yarn index-block --block $BLOCK_NUMBER
    ```

    Example for [mainnet MobyMask blocks](https://etherscan.io/address/0xb06e6db9288324738f04fcaac910f5a60102c1f8) indexed above:
    ```bash
    yarn index-block --block 14869713
    ```
  
  * Check the `event` and `block_progress` table to confirm that the required blocks have been indexed properly.

* The watcher can be started to perform queries on the indexed data:
  
  * The watcher can be started in lazy mode:

    * Update `server.kind` in [config](./environments/local.toml):

      ```toml
      [server]
        kind = "lazy"
      ```

    * Run server:

      ```bash
      yarn server
      ```

  * Run query in [GraphQL endpoint](http://127.0.0.1:3010/graphql) to get events in a range. Following query is for getting events in the range of mainnet blocks indexed for mobymask-watcher:

    ```graphql
    query {
      eventsInRange(
        # Range for mainnet data blocks
        fromBlockNumber: 14869713
        toBlockNumber: 14885755
      ) {
        block {
          hash
          number
        }
        tx {
          hash
        }
        contract
        eventIndex
        event {
          __typename
        }
        proof {
          data
        }
      }
    }
    ```

  * Run query to get contract storage variable values. The following query is for getting value of `isMember` variable in MobyMask contract:

    ```graphql
    query {
      isMember(
        # BlockHash of an indexed mainnet block that can be taken from the events returned above
        blockHash: "0x28cb16e740cd5d7de869bee2957e7442790e9d774e6e71804a67933c7e628038"
        contractAddress: "0xB06E6DB9288324738f04fCAAc910f5A60102C1F8"
        key0: "TWT:danfinlay"
      ) {
        value
        proof {
          data
        }
      }
    }
    ```