update docs

This commit is contained in:
Roy Crihfield 2023-06-23 21:43:35 +08:00
parent e24178a8a9
commit 951946d164
5 changed files with 13 additions and 41 deletions

View File

@ -1,17 +0,0 @@
# Overview
This document will provide some insight into the `known_gaps` table, their use cases, and implementation. Please refer to the [following PR](https://github.com/vulcanize/go-ethereum/pull/217) and the [following epic](https://github.com/vulcanize/ops/issues/143) to grasp their inception.
![known gaps](diagrams/KnownGapsProcess.png)
# Use Cases
The known gaps table is updated when the following events occur:
1. At start up we check the latest block from the `eth.headers_cid` table. We compare the first block that we are processing with the latest block from the DB. If they are not one unit of expectedDifference away from each other, add the gap between the two blocks.
2. If there is any error in processing a block (db connection, deadlock, etc), add that block to the knownErrorBlocks slice, when the next block is successfully written, write this slice into the DB.
# Glossary
1. `expectedDifference (number)` - This number indicates what the difference between two blocks should be. If we are capturing all events on a geth node then this number would be `1`. But once we scale nodes, the `expectedDifference` might be `2` or greater.
2. `processingKey (number)` - This number can be used to keep track of different geth nodes and their specific `expectedDifference`.

View File

@ -1,3 +0,0 @@
# Overview
This folder keeps tracks of random documents as they relate to the `statediff` service.

View File

@ -1,21 +0,0 @@
# Overview
This document will go through some notes on the database component of the statediff service.
# Components
- Indexer: The indexer creates IPLD and DB models to insert to the Postgres DB. It performs the insert utilizing and atomic function.
- Builder: The builder constructs the statediff object that needs to be inserted.
- Known Gaps: Captures any gaps that might have occured and either writes them to the DB, local sql file, to prometeus, or a local error.
# Making Code Changes
## Adding a New Function to the Indexer
If you want to implement a new feature for adding data to the database. Keep the following in mind:
1. You need to handle `sql`, `file`, and `dump`.
1. `sql` - Contains the code needed to write directly to the `sql` db.
2. `file` - Contains all the code required to write the SQL statements to a file.
3. `dump` - Contains all the code for outputting events to the console.
2. You will have to add it to the `interfaces.StateDiffIndexer` interface.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 33 KiB

13
docs/indexer.md Normal file
View File

@ -0,0 +1,13 @@
# Statediff database indexing
To process data in real time as Geth syncs updates to the Ethereum execution layer, the statediff
service is able to directly transform and load data into a Postgres database. The `indexer` package
contains abstractions for handling this ingestion.
## Interface
A `StateDiffIndexer` object is responsible for inserting statediff data into a database, as well as managing watched address lists for a given database.
Three implementations are currently maintained:
* `sql` for direct insertion to Postgres
* `file` which writes to CSV for SQL files for insertion in a separate step
* `dump` which simply dumps to stdout