[Feature] - Create a known_gaps table. #29

New Issue

2022-05-09T18:55:29Z

abdulrabbani00 commented

2022-05-09 18:55:29 +00:00

(Migrated from github.com)

Overview
- Request
Potential Solution
- Alternative Solutions
Additional Context

Table of contents generated with markdown-toc

Overview

Requester: @abdulrabbani00
Epic Link: https://github.com/vulcanize/ipld-ethcl-indexer/issues/19

Request

We need to create a table for the known_gaps. Here are some thoughts:

We should write all the blocks we want to process to the ethcl.known_gaps table.
When the application is running in historic mode, it will periodically read the ethcl.known_gaps table.
As new entries as found in the table, new goroutines will be spun up to handle the entries.
We can also add a flag to head tracking to handle known_gaps.

Potential Solution

Create a new table called ethcl.known_gaps.
Keep each block range in the table capped off to a certain number, x. For example, each row should contain no more than "1000" slots. This way if an error occurs when processing in the middle of the processing x block range, we can contain it within that range. On the other hand if instead, we had "1,000,000" entries per row, if an error occurred, we might have to start from the first slot when reprocessing after the error.
If its a low lift, capture errors and add them to the table. Slots that result in errors should be marked as such. In this way, we can keep track of errors in the DB, and if they persist, we can manually see them and study them.
If we are unable to write to the ethcl.known_gaps table, then we need to shut down the application. The risk of skipping blocks without knowing is too great, and if it occurs, then the application should be shut down.

Alternative Solutions

Additional Context

- [Overview](#overview) - [Request](#request) - [Potential Solution](#potential-solution) - [Alternative Solutions](#alternative-solutions) - [Additional Context](#additional-context) <small><i><a href='http://ecotrust-canada.github.io/markdown-toc/'>Table of contents generated with markdown-toc</a></i></small> # Overview - **Requester**: @abdulrabbani00 - **Epic Link**: https://github.com/vulcanize/ipld-ethcl-indexer/issues/19 ## Request  We need to create a table for the `known_gaps`. Here are some thoughts: * We should write all the blocks we want to process to the `ethcl.known_gaps` table. * When the application is running in `historic` mode, it will periodically read the `ethcl.known_gaps` table. * As new entries as found in the table, new goroutines will be spun up to handle the entries. * We can also add a flag to `head` tracking to handle `known_gaps`. # Potential Solution  * Create a new table called `ethcl.known_gaps`. * Keep each block range in the table capped off to a certain number, `x`. For example, each row should contain no more than "1000" slots. This way if an error occurs when processing in the middle of the processing `x` block range, we can contain it within that range. On the other hand if instead, we had "1,000,000" entries per row, if an error occurred, we might have to start from the first slot when reprocessing after the error. * If its a low lift, capture errors and add them to the table. Slots that result in errors should be marked as such. In this way, we can keep track of errors in the DB, and if they persist, we can manually see them and study them. * If we are unable to write to the `ethcl.known_gaps` table, then we need to shut down the application. The risk of skipping blocks without knowing is too great, and if it occurs, then the application should be shut down. ## Alternative Solutions  # Additional Context

Sign in to join this conversation.

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/ipld-eth-beacon-indexer#29

[Feature] - Create a known_gaps table. #29

Overview

Request

Potential Solution

Alternative Solutions

Additional Context

[Feature] - Create a `known_gaps` table. #29