Commit Graph

28 Commits

Author SHA1 Message Date
Abdul Rabbani
3da8094e60 Update flags and minor shutdown updates 2022-06-03 11:12:54 -04:00
Abdul Rabbani
0da384517a Handle graceful shutdown for historical processing 2022-06-02 17:04:27 -04:00
Abdul Rabbani
4a85428b53 Capture the unique identifier everywhere its needed. 2022-05-25 10:59:25 -04:00
Abdul Rabbani
9160dded11
Add Prometheus Metrics (#49)
* Add Prometheus Metrics

* Fix Prometheus duplicate error

* Handle duplicate registrations
2022-05-25 10:19:29 -04:00
Abdul Rabbani
347984a547
Feature/44 read write historic slots (#46)
* Update boot to set processing type.

* Handle entries from the historic_process table.

* Update on-pr.yml

* Fix head processing Error

* Update names and debug

* Seperate checking for new entries and locking them

* Application can process historic and known gaps

Untested

* Handle genesis

* Update remove entry for knownGaps, viper

* Disregard unused code from linter
2022-05-24 16:18:55 -04:00
Abdul Rabbani
041276da81
Update the boot process (#45)
* Update the boot process

* Update the CI/CD reference for stack-orchestrator
2022-05-19 09:46:38 -04:00
Abdul Rabbani
891f4c7ef3 Update Vulcanize License in a .go files 2022-05-18 12:12:54 -04:00
Abdul Rabbani
de38c531d5 Test the application for v1 release. (#37)
* Allow the application to process events in parallel

there is a main thread that tracks incoming messages, but then it spawns goroutines to actually process each slot so that they can happen concurrently.

* Control knownGaps in existing test

* Use Interfaces for different fork version

Use interfaces for `SignedBeaconBlock` and `BeaconState`, this allows the application to determine the correct forked struct.

In the test we also use a switch condition to properly serve the correct mimics.

* Utilize new ipld-ethcl-db repository

* Add final tests

* Update timeout and secret

* Update token

* Update docker compose

* Update expected inserts
2022-05-17 16:45:40 -04:00
Ian Norden
7a242a4c71 multihash key gen func (#36)
* multihash key gen func

* go mod updates

* Added test to ensure the application shuts down gracefully or within a timeframe.

* Disregard race condition since its with the test not the application itself

* Capture the head block in the DB entirely. (#27)

* -- Intermediary Commit --

Just want to commit my code over the weekend, in case I spill coffee on my workstation.

* Create DB models ready for write.

* Handle SSE events

* Update ref for stack-orchestrator

* Use env in one place only.

* Boot Application on PR

* Update syntax

* Update syntax

* Correct command

* Use bash instead of sh

* Use until instead of while

* Make linter happy and check sse subscription err

* Handle Reorgs - Untested

* Feature/22 test handling incoming events - Intermediary Commit (#28)

* Checkpoint before the weekend

* Update location for SetupPostgresDB

* Feature/22 test handling incoming events (#30)

* Checkpoint before the weekend

* Update location for SetupPostgresDB

* Include first functioning tests for processing head

* Fix gitignore

* Test CaptureHead | Add Metrics | Handle Test Race Conditions

This Commit allows us to:

* Test the `CaptureHead` function.
* Test parsing a single Head message.
* Test a Reorg condition.
* Add Metrics. This is primarily used for testing but can have future use cases.
* Rearrange the test due to race conditions introduced by reusing a variable. `BeforeEach` can't be used to update `BC`.

* Update and finalize testing at this stage

* Update code and CI/CD

* Fix lint errors

* Update CICD and fail when file not found.

* Update test to have failed as expected.

* Remove Test file

* Add KnownGaps Errors (#33)

* Handle Skipped Slots (#34)

* Ensure that the node is synced at boot time

* Update test + add logic for checking skipped slots

* Update boot check

* Add skip_sync to config.

* Update a test so it fails

* go mod updates

* Integrate MHKey into existing code base.

* Update go.mod and go.sum

* Utilize the MHkey

* Stop tests from running forever on failure.

* Use sszRoot instead of sszObj for MhKey

* Update entrypoint script

* Update config parameter

Co-authored-by: Abdul Rabbani <abdulrabbani00@gmail.com>
Co-authored-by: Abdul Rabbani <58230246+abdulrabbani00@users.noreply.github.com>
2022-05-17 16:45:40 -04:00
Abdul Rabbani
a31c9226ec Handle Skipped Slots (#34)
* Ensure that the node is synced at boot time

* Update test + add logic for checking skipped slots

* Update boot check

* Add skip_sync to config.

* Update a test so it fails
2022-05-17 16:45:40 -04:00
Abdul Rabbani
12675f85c2 Add KnownGaps Errors (#33) 2022-05-17 16:45:40 -04:00
Abdul Rabbani
78089d3fd8 Remove Test file 2022-05-17 16:45:40 -04:00
Abdul Rabbani
6123fa40e8 Feature/22 test handling incoming events (#30)
* Checkpoint before the weekend

* Update location for SetupPostgresDB

* Include first functioning tests for processing head

* Fix gitignore

* Test CaptureHead | Add Metrics | Handle Test Race Conditions

This Commit allows us to:

* Test the `CaptureHead` function.
* Test parsing a single Head message.
* Test a Reorg condition.
* Add Metrics. This is primarily used for testing but can have future use cases.
* Rearrange the test due to race conditions introduced by reusing a variable. `BeforeEach` can't be used to update `BC`.

* Update and finalize testing at this stage

* Update code and CI/CD

* Fix lint errors

* Update CICD and fail when file not found.

* Update test to have failed as expected.
2022-05-17 16:45:40 -04:00
Abdul Rabbani
361dbd73a0 Feature/22 test handling incoming events - Intermediary Commit (#28)
* Checkpoint before the weekend

* Update location for SetupPostgresDB
2022-05-17 16:45:40 -04:00
Abdul Rabbani
d13144ca90 Capture the head block in the DB entirely. (#27)
* -- Intermediary Commit --

Just want to commit my code over the weekend, in case I spill coffee on my workstation.

* Create DB models ready for write.

* Handle SSE events

* Update ref for stack-orchestrator

* Use env in one place only.

* Boot Application on PR

* Update syntax

* Update syntax

* Correct command

* Use bash instead of sh

* Use until instead of while

* Make linter happy and check sse subscription err

* Handle Reorgs - Untested
2022-05-17 16:45:40 -04:00
Abdul Rabbani
6215f7e66f Added test to ensure the application shuts down gracefully or within a timeframe. 2022-05-17 16:45:40 -04:00
Abdul Rabbani
f46c4410a2 Clean up + Get State and Block 2022-04-27 14:01:59 -04:00
Abdul Rabbani
d496886f95 Process SSE messages, and include a graceful shutdown 2022-04-27 10:28:42 -04:00
Abdul Rabbani
87313887a4 Graceful Shutdown | Handle SSE | Requires Testing
This Commit contains the following:
* Graceful shutdowns.
* Handling all incoming SSE events for reorgs, finalizations, and head.

The structure of the `BeaconClient` has drastically changed and generics are used.
2022-04-26 13:57:01 -04:00
Abdul Rabbani
5b75f5a257 Create a single beaconclient pkg instead of having subpackages
The main reason to do this is so they can all access the `BeaconClient` Struct.
2022-04-25 11:32:46 -04:00
Abdul Rabbani
594d1e6ee8 Move healthcheck to beaconclient pkg. 2022-04-25 10:41:50 -04:00
Abdul Rabbani
3d46a029f1
Feature/11 testing add tests for boot (#12)
* Utilize Versioning

* Update Testing and CI/CD

* Update Testing

* Add Linting and Remove timeout

* Update Lint

* handle errors to make the linter happy
2022-04-22 12:27:54 -04:00
Abdul Rabbani
e2f7fa381b
Utilize Ginkgo toreplace testing library, and add CI/CD. (#9)
* Utilize Ginkgo and replace `testing` library.

* Add TOC to docs

* Add Docker specific files

* Remove -e

* Update on-pr-manual.yml

* Add depth

* Add repositories

* Remove $ from path

* Update path for make

* Setup Go and Ginkgo

* Use go mod download

* Use go install

* Update on-pr-manual.yml

* Use latest

* Remove install of GINKGO

* Add explicit gopath

* Explicitly specify the gopath

* Update on-pr-manual.yml

* Update on-pr-manual.yml

* Update on-pr-manual.yml

* Update on-pr-manual.yml

* Update on-pr-manual.yml

* Update on-pr-manual.yml

* Use which ginkgo

* Try with make now

* Final working Make
2022-04-22 08:28:01 -04:00
Abdul Rabbani
7a2c4b3cef Last second clean ups 2022-04-20 18:23:21 -04:00
Abdul Rabbani
d7ad4108a7 Code clean up + Beacon Chain Connection
This concludes all the code needed to connect to the DB and beacon node. We will no longer reference the lighthouse client because this application should work interchangeably with any beacon node.

I have also standardized logging.
2022-04-20 18:12:44 -04:00
Abdul Rabbani
827475f029 Add DB Connection and Logging
* Utilize LogRus
* Create a DB connection using PGX.
* Create an internal boot package for starting the application.
2022-04-20 15:44:15 -04:00
Abdul Rabbani
d0d4f2498e Put PGX and other future Postgres Drivers behind SQL package
This PR makes allows users to provide a config, along with a driver string. It will then provide the respective driver.
2022-04-20 13:06:00 -04:00
Abdul Rabbani
ed3d0be2b7 A very generic package for implementing PGX driver
I copied most of the code from the `statediff` service from within geth. The idea is that I can create formal DB packages, that can be utilized in other projects down the road.
2022-04-20 12:12:55 -04:00