This application will capture all the BeaconState's and SignedBeaconBlock's from the consensus chain on Ethereum.
Go to file
Ian Norden 23cbe8f919
multihash key gen func (#36)
* multihash key gen func

* go mod updates

* Added test to ensure the application shuts down gracefully or within a timeframe.

* Disregard race condition since its with the test not the application itself

* Capture the head block in the DB entirely. (#27)

* -- Intermediary Commit --

Just want to commit my code over the weekend, in case I spill coffee on my workstation.

* Create DB models ready for write.

* Handle SSE events

* Update ref for stack-orchestrator

* Use env in one place only.

* Boot Application on PR

* Update syntax

* Update syntax

* Correct command

* Use bash instead of sh

* Use until instead of while

* Make linter happy and check sse subscription err

* Handle Reorgs - Untested

* Feature/22 test handling incoming events - Intermediary Commit (#28)

* Checkpoint before the weekend

* Update location for SetupPostgresDB

* Feature/22 test handling incoming events (#30)

* Checkpoint before the weekend

* Update location for SetupPostgresDB

* Include first functioning tests for processing head

* Fix gitignore

* Test CaptureHead | Add Metrics | Handle Test Race Conditions

This Commit allows us to:

* Test the `CaptureHead` function.
* Test parsing a single Head message.
* Test a Reorg condition.
* Add Metrics. This is primarily used for testing but can have future use cases.
* Rearrange the test due to race conditions introduced by reusing a variable. `BeforeEach` can't be used to update `BC`.

* Update and finalize testing at this stage

* Update code and CI/CD

* Fix lint errors

* Update CICD and fail when file not found.

* Update test to have failed as expected.

* Remove Test file

* Add KnownGaps Errors (#33)

* Handle Skipped Slots (#34)

* Ensure that the node is synced at boot time

* Update test + add logic for checking skipped slots

* Update boot check

* Add skip_sync to config.

* Update a test so it fails

* go mod updates

* Integrate MHKey into existing code base.

* Update go.mod and go.sum

* Utilize the MHkey

* Stop tests from running forever on failure.

* Use sszRoot instead of sszObj for MhKey

* Update entrypoint script

* Update config parameter

Co-authored-by: Abdul Rabbani <abdulrabbani00@gmail.com>
Co-authored-by: Abdul Rabbani <58230246+abdulrabbani00@users.noreply.github.com>
2022-05-13 10:46:13 -04:00
.github multihash key gen func (#36) 2022-05-13 10:46:13 -04:00
cmd Handle Skipped Slots (#34) 2022-05-13 08:48:31 -04:00
internal Handle Skipped Slots (#34) 2022-05-13 08:48:31 -04:00
pkg multihash key gen func (#36) 2022-05-13 10:46:13 -04:00
.gitignore Remove Test file 2022-05-12 09:55:42 -04:00
application_component.md Add KnownGaps Errors (#33) 2022-05-12 15:44:05 -04:00
Dockerfile Add KnownGaps Errors (#33) 2022-05-12 15:44:05 -04:00
entrypoint.sh multihash key gen func (#36) 2022-05-13 10:46:13 -04:00
go.mod multihash key gen func (#36) 2022-05-13 10:46:13 -04:00
go.sum multihash key gen func (#36) 2022-05-13 10:46:13 -04:00
LICENSE Initial commit 2022-04-19 14:42:32 -04:00
main.go Utilize Cobra 2022-04-19 17:09:59 -04:00
Makefile Feature/22 test handling incoming events (#30) 2022-05-12 09:52:13 -04:00
README.md Handle Skipped Slots (#34) 2022-05-13 08:48:31 -04:00

Table of contents generated with markdown-toc

ipld-ethcl-indexer

This application will capture all the BeaconState's and SignedBeaconBlock's from the consensus chain on Ethereum. This application is going to connect to the lighthouse client, but hypothetically speaking, it should be interchangeable with any eth2 beacon node.

To learn more about the applications individual components, please read the application components.

Quick Start

Running the Application

To run the application, do as follows:

  1. Setup the prerequisite applications. a. Run a beacon client (such as lighthouse). b. Run a postgres DB. c. You can utilize the stack-orchestrator repository.

    ./wrapper.sh -e skip \
    -d ../docker/local/docker-compose-db.yml \
    -d ../docker/latest/docker-compose-lighthouse.yml \
    -v remove \
    -p ../local-config.sh
    
    
  2. Run the start up command.

go run -race main.go capture head --db.address localhost \
  --db.password password \
  --db.port 8077 \
  --db.username vdbm \
  --db.name vulcanize_testing \
  --db.driver PGX \
  --bc.address localhost \
  --bc.port 5052 \
  --bc.connectionProtocol http \
  --t.skipSync=true \
  --log.level info \
  --log.output=true \
  --kg.increment 100

Running Tests

To run tests, you will need to clone another repository which contains all the ssz files.

  1. git clone git@github.com:vulcanize/ssz-data.git pkg/beaconclient/ssz-data
  2. To run unit tests, make sure you have a DB running: make unit-test-local
  3. To run integration tests, make sure you have a lighthouse client and a DB running: make integration-test-local-no-race .

Development Patterns

This section will cover some generic development patterns utilizes.

Logging

For logging, please keep the following in mind:

  • Utilize logrus.
  • Use log.Debug to highlight that you are about to do something.
  • Use log.Info-Fatal when the thing you were about to do has been completed, along with the result.
log.Debug("Adding 1 + 2")
a := 1 + 2
log.Info("1 + 2 successfully Added, outcome is: ", a)
  • loghelper.LogError(err) is a pretty wrapper to output errors.

Testing

This project utilizes ginkgo for testing. A few notes on testing:

  • All tests within this code base will test public methods only.
  • All test packages are named {base_package}_test. This ensures we only test the public methods.
  • If there is a need to test a private method, please include why in the testing file.
  • Unit tests must contain the Label("unit").
  • Unit tests should not rely on any running service (except for a postgres DB). If a running service is needed. Utilize an integration test.
  • Integration tests must contain the Label("integration").

Understanding Testing Components

A few notes about the testing components.

  • The TestEvents map contains several events for testers to leverage when testing.
  • Any object ending in -dummy is not a real object. You will also notice it has a present field called MimicConfig. This object will use an existing SSZ object, and update the parameters from the Head and MimicConfig.
    • This is done because creating an empty or minimal SignedBeaconBlock and BeaconState is fairly challenging.
    • By slightly modifying an existing object, we can test re-org, malformed objects, and other negative conditions.

Contribution

If you want to contribute please make sure you do the following:

  • Create a Github issue before starting your work.
  • Follow the branching structure.
  • Delete your branch once it has been merged.
    • Do not delete the develop branch. We can add branch protection once we make the branch public.

Branching Structure

The branching structure is as follows: main <-- develop <-- your-branch.

It is adviced that your-branch follows the following structure: {type}/{issue-number}-{description}.

  • type - This can be anything identifying the reason for this PR, for example: bug, feature, release.
  • issue-number - This is the issue number of the GitHub issue. It will help users easily find a full description of the issue you are trying to solve.
  • description - A few words to identify your issue.