1. Updates or replaces outdated dependencies (eg, replacing a version of the Prysm client with the latest zrnt).
2. Add support for parsing Bellatrix-era BeaconState and BeaconBlocks
3. Adds flags for toggling the processing of BeaconBlocks and BeaconState. This is particularly important because processing and storing the BeaconState at this time would be too expensive to really do (see: Temporarily disable BeaconState indexing #75 and [Feature] Reduce the Amount of DB Space the Beacon Chain Needs #71)
4. Fixes flaky event handling. The previous code would not reconnect in the case of errors with the SSE connection. This enables automatic reconnection in the case of error (default in the updated v2 SSE library dependency), and also adds a timeout so that if no event is received in 2.5x the block time, the SSE connection is closed and re-established.
5. Other refactoring and cleanup (eg, changing the type of slot from int to Slot (uint64)).
* Slight Performance improvements with Logging
* Use exec in docker container
* Update packages for dockerfile
* Add flakey test and boot command specific logic
* Remove excess logging and add config val
* Update README and condition for cleanup
* Utilize config file for ipld-eth-beacon
* Update CICD to use config file on container build.
* Add env file for build cp
* Add a sleep before running the test
* Set starting slot and improve error gap capturing
* Set starting slot and improve error gap capturing
* Tests + Significant Refactor
The code for historical processing has been significantly refactored to use a context to signify a shutdown.
There have also been many tests added for historical and knownGaps processing.
* Update MhKeys in test
* Update correct values
* Update Max Retry
Genesis is not working as expected.
* Ensure we release locks properly
* Add ordered testing
* Include system tests
* Update workflow calls
* Add secrets
* Add required secrets
* update path
* Try using the absolute path
* Remove volumes at the end.
* Update system-tests.yml
* Update system-tests.yml
* Update test err
* Update and test the shutdown
* rename ethcl --> eth-beacon
* Try forcing /bin/bash for docker-compose
* Update system-tests.yml
* Update system-tests.yml
* Update system-tests.yml
* Update system-tests.yml
* Update system-tests.yml
* Update system-tests.yml
* Use single quote cron
* Dont run generic on schedule
* Capture the unique identifier everywhere its needed.
* Handle graceful shutdown for historical processing
* Update flags and minor shutdown updates
* Update checkout statement for historic
* update cicd pipeline
* Add secret
* Use deploy-key instead of personal PAT
* update variables
* Update boot to set processing type.
* Handle entries from the historic_process table.
* Update on-pr.yml
* Fix head processing Error
* Update names and debug
* Seperate checking for new entries and locking them
* Application can process historic and known gaps
Untested
* Handle genesis
* Update remove entry for knownGaps, viper
* Disregard unused code from linter
* Allow the application to process events in parallel
there is a main thread that tracks incoming messages, but then it spawns goroutines to actually process each slot so that they can happen concurrently.
* Control knownGaps in existing test
* Use Interfaces for different fork version
Use interfaces for `SignedBeaconBlock` and `BeaconState`, this allows the application to determine the correct forked struct.
In the test we also use a switch condition to properly serve the correct mimics.
* Utilize new ipld-ethcl-db repository
* Add final tests
* Update timeout and secret
* Update token
* Update docker compose
* Update expected inserts
* multihash key gen func
* go mod updates
* Added test to ensure the application shuts down gracefully or within a timeframe.
* Disregard race condition since its with the test not the application itself
* Capture the head block in the DB entirely. (#27)
* -- Intermediary Commit --
Just want to commit my code over the weekend, in case I spill coffee on my workstation.
* Create DB models ready for write.
* Handle SSE events
* Update ref for stack-orchestrator
* Use env in one place only.
* Boot Application on PR
* Update syntax
* Update syntax
* Correct command
* Use bash instead of sh
* Use until instead of while
* Make linter happy and check sse subscription err
* Handle Reorgs - Untested
* Feature/22 test handling incoming events - Intermediary Commit (#28)
* Checkpoint before the weekend
* Update location for SetupPostgresDB
* Feature/22 test handling incoming events (#30)
* Checkpoint before the weekend
* Update location for SetupPostgresDB
* Include first functioning tests for processing head
* Fix gitignore
* Test CaptureHead | Add Metrics | Handle Test Race Conditions
This Commit allows us to:
* Test the `CaptureHead` function.
* Test parsing a single Head message.
* Test a Reorg condition.
* Add Metrics. This is primarily used for testing but can have future use cases.
* Rearrange the test due to race conditions introduced by reusing a variable. `BeforeEach` can't be used to update `BC`.
* Update and finalize testing at this stage
* Update code and CI/CD
* Fix lint errors
* Update CICD and fail when file not found.
* Update test to have failed as expected.
* Remove Test file
* Add KnownGaps Errors (#33)
* Handle Skipped Slots (#34)
* Ensure that the node is synced at boot time
* Update test + add logic for checking skipped slots
* Update boot check
* Add skip_sync to config.
* Update a test so it fails
* go mod updates
* Integrate MHKey into existing code base.
* Update go.mod and go.sum
* Utilize the MHkey
* Stop tests from running forever on failure.
* Use sszRoot instead of sszObj for MhKey
* Update entrypoint script
* Update config parameter
Co-authored-by: Abdul Rabbani <abdulrabbani00@gmail.com>
Co-authored-by: Abdul Rabbani <58230246+abdulrabbani00@users.noreply.github.com>
* Ensure that the node is synced at boot time
* Update test + add logic for checking skipped slots
* Update boot check
* Add skip_sync to config.
* Update a test so it fails
* Checkpoint before the weekend
* Update location for SetupPostgresDB
* Include first functioning tests for processing head
* Fix gitignore
* Test CaptureHead | Add Metrics | Handle Test Race Conditions
This Commit allows us to:
* Test the `CaptureHead` function.
* Test parsing a single Head message.
* Test a Reorg condition.
* Add Metrics. This is primarily used for testing but can have future use cases.
* Rearrange the test due to race conditions introduced by reusing a variable. `BeforeEach` can't be used to update `BC`.
* Update and finalize testing at this stage
* Update code and CI/CD
* Fix lint errors
* Update CICD and fail when file not found.
* Update test to have failed as expected.
* -- Intermediary Commit --
Just want to commit my code over the weekend, in case I spill coffee on my workstation.
* Create DB models ready for write.
* Handle SSE events
* Update ref for stack-orchestrator
* Use env in one place only.
* Boot Application on PR
* Update syntax
* Update syntax
* Correct command
* Use bash instead of sh
* Use until instead of while
* Make linter happy and check sse subscription err
* Handle Reorgs - Untested
* Update cobra to require `head` or `historic` when using `capture`.
* A very generic package for implementing PGX driver
I copied most of the code from the `statediff` service from within geth. The idea is that I can create formal DB packages, that can be utilized in other projects down the road.
* Put PGX and other future Postgres Drivers behind SQL package
This PR makes allows users to provide a config, along with a driver string. It will then provide the respective driver.
* Add DB Connection and Logging
* Utilize LogRus
* Create a DB connection using PGX.
* Create an internal boot package for starting the application.
* Code clean up + Beacon Chain Connection
This concludes all the code needed to connect to the DB and beacon node. We will no longer reference the lighthouse client because this application should work interchangeably with any beacon node.
I have also standardized logging.
* Last second clean ups
* Utilize Ginkgo toreplace `testing` library, and add CI/CD. (#9)
* Utilize Ginkgo and replace `testing` library.
* Add TOC to docs
* Add Docker specific files
* Remove -e
* Update on-pr-manual.yml
* Add depth
* Add repositories
* Remove $ from path
* Update path for make
* Setup Go and Ginkgo
* Use go mod download
* Use go install
* Update on-pr-manual.yml
* Use latest
* Remove install of GINKGO
* Add explicit gopath
* Explicitly specify the gopath
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Use which ginkgo
* Try with make now
* Final working Make
* Feature/11 testing add tests for boot (#12)
* Utilize Versioning
* Update Testing and CI/CD
* Update Testing
* Add Linting and Remove timeout
* Update Lint
* handle errors to make the linter happy
* Utilized interfaces for DB
* Update Manual Testing CI/CD
* Try combining everything into one pipeline (#13)
* Try combining everything into one pipeline
* Update on-pr-manual.yml
* Remove PAth for testing
* Combine single YML workflow file for PR
* Add a comment
* Update cobra to require `head` or `historic` when using `capture`.
* A very generic package for implementing PGX driver
I copied most of the code from the `statediff` service from within geth. The idea is that I can create formal DB packages, that can be utilized in other projects down the road.
* Put PGX and other future Postgres Drivers behind SQL package
This PR makes allows users to provide a config, along with a driver string. It will then provide the respective driver.
* Add DB Connection and Logging
* Utilize LogRus
* Create a DB connection using PGX.
* Create an internal boot package for starting the application.
* Code clean up + Beacon Chain Connection
This concludes all the code needed to connect to the DB and beacon node. We will no longer reference the lighthouse client because this application should work interchangeably with any beacon node.
I have also standardized logging.
* Last second clean ups
* Utilize Ginkgo toreplace `testing` library, and add CI/CD. (#9)
* Utilize Ginkgo and replace `testing` library.
* Add TOC to docs
* Add Docker specific files
* Remove -e
* Update on-pr-manual.yml
* Add depth
* Add repositories
* Remove $ from path
* Update path for make
* Setup Go and Ginkgo
* Use go mod download
* Use go install
* Update on-pr-manual.yml
* Use latest
* Remove install of GINKGO
* Add explicit gopath
* Explicitly specify the gopath
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Use which ginkgo
* Try with make now
* Final working Make
* Utilize Ginkgo and replace `testing` library.
* Add TOC to docs
* Add Docker specific files
* Remove -e
* Update on-pr-manual.yml
* Add depth
* Add repositories
* Remove $ from path
* Update path for make
* Setup Go and Ginkgo
* Use go mod download
* Use go install
* Update on-pr-manual.yml
* Use latest
* Remove install of GINKGO
* Add explicit gopath
* Explicitly specify the gopath
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Update on-pr-manual.yml
* Use which ginkgo
* Try with make now
* Final working Make
