Commit Graph

21 Commits

Author SHA1 Message Date
Abdul Rabbani
53c4e4243c Use pprof + pointers 2022-06-20 13:56:55 -04:00
Abdul Rabbani
24fc6358d6
Testing for Batch Processing (#56)
* Set starting slot and improve error gap capturing

* Set starting slot and improve error gap capturing

* Tests + Significant Refactor

The code for historical processing has been significantly refactored to use a context to signify a shutdown.

There have also been many tests added for historical and knownGaps processing.

* Update MhKeys in test

* Update correct values

* Update Max Retry

Genesis is not working as expected.

* Ensure we release locks properly

* Add ordered testing

* Include system tests

* Update workflow calls

* Add secrets

* Add required secrets

* update path

* Try using the absolute path

* Remove volumes at the end.

* Update system-tests.yml

* Update system-tests.yml

* Update test err

* Update and test the shutdown

* rename ethcl --> eth-beacon

* Try forcing /bin/bash for docker-compose

* Update system-tests.yml

* Update system-tests.yml

* Update system-tests.yml

* Update system-tests.yml

* Update system-tests.yml

* Update system-tests.yml

* Use single quote cron

* Dont run generic on schedule
2022-06-09 17:32:46 -04:00
Abdul Rabbani
4d7781e2b0
Use a transaction for writing, knowngaps, and reorgs (#53)
* Use a transaction for writing, knowngaps, and reorgs

* Checkpoint - Check the DB before writing

* Ensure an error exists when writing a transaction to knowngaps

* Update unneccesary parameter
2022-06-06 09:02:43 -04:00
Abdul Rabbani
e3b4fad3c7
Capture the unique identifier everywhere its needed. (#51)
* Capture the unique identifier everywhere its needed.

* Handle graceful shutdown for historical processing

* Update flags and minor shutdown updates

* Update checkout statement for historic

* update cicd pipeline

* Add secret

* Use deploy-key instead of personal PAT

* update variables
2022-06-03 12:47:13 -04:00
Abdul Rabbani
9160dded11
Add Prometheus Metrics (#49)
* Add Prometheus Metrics

* Fix Prometheus duplicate error

* Handle duplicate registrations
2022-05-25 10:19:29 -04:00
Abdul Rabbani
347984a547
Feature/44 read write historic slots (#46)
* Update boot to set processing type.

* Handle entries from the historic_process table.

* Update on-pr.yml

* Fix head processing Error

* Update names and debug

* Seperate checking for new entries and locking them

* Application can process historic and known gaps

Untested

* Handle genesis

* Update remove entry for knownGaps, viper

* Disregard unused code from linter
2022-05-24 16:18:55 -04:00
Abdul Rabbani
041276da81
Update the boot process (#45)
* Update the boot process

* Update the CI/CD reference for stack-orchestrator
2022-05-19 09:46:38 -04:00
Abdul Rabbani
891f4c7ef3 Update Vulcanize License in a .go files 2022-05-18 12:12:54 -04:00
Abdul Rabbani
a31c9226ec Handle Skipped Slots (#34)
* Ensure that the node is synced at boot time

* Update test + add logic for checking skipped slots

* Update boot check

* Add skip_sync to config.

* Update a test so it fails
2022-05-17 16:45:40 -04:00
Abdul Rabbani
12675f85c2 Add KnownGaps Errors (#33) 2022-05-17 16:45:40 -04:00
Abdul Rabbani
d13144ca90 Capture the head block in the DB entirely. (#27)
* -- Intermediary Commit --

Just want to commit my code over the weekend, in case I spill coffee on my workstation.

* Create DB models ready for write.

* Handle SSE events

* Update ref for stack-orchestrator

* Use env in one place only.

* Boot Application on PR

* Update syntax

* Update syntax

* Correct command

* Use bash instead of sh

* Use until instead of while

* Make linter happy and check sse subscription err

* Handle Reorgs - Untested
2022-05-17 16:45:40 -04:00
Abdul Rabbani
6215f7e66f Added test to ensure the application shuts down gracefully or within a timeframe. 2022-05-17 16:45:40 -04:00
Abdul Rabbani
f46c4410a2 Clean up + Get State and Block 2022-04-27 14:01:59 -04:00
Abdul Rabbani
d496886f95 Process SSE messages, and include a graceful shutdown 2022-04-27 10:28:42 -04:00
Abdul Rabbani
87313887a4 Graceful Shutdown | Handle SSE | Requires Testing
This Commit contains the following:
* Graceful shutdowns.
* Handling all incoming SSE events for reorgs, finalizations, and head.

The structure of the `BeaconClient` has drastically changed and generics are used.
2022-04-26 13:57:01 -04:00
Abdul Rabbani
3d46a029f1
Feature/11 testing add tests for boot (#12)
* Utilize Versioning

* Update Testing and CI/CD

* Update Testing

* Add Linting and Remove timeout

* Update Lint

* handle errors to make the linter happy
2022-04-22 12:27:54 -04:00
Abdul Rabbani
7a2c4b3cef Last second clean ups 2022-04-20 18:23:21 -04:00
Abdul Rabbani
d7ad4108a7 Code clean up + Beacon Chain Connection
This concludes all the code needed to connect to the DB and beacon node. We will no longer reference the lighthouse client because this application should work interchangeably with any beacon node.

I have also standardized logging.
2022-04-20 18:12:44 -04:00
Abdul Rabbani
827475f029 Add DB Connection and Logging
* Utilize LogRus
* Create a DB connection using PGX.
* Create an internal boot package for starting the application.
2022-04-20 15:44:15 -04:00
Abdul Rabbani
42923d15c1 Update cobra to require head or historic when using capture. 2022-04-20 09:25:47 -04:00
Abdul Rabbani
daf99c7279 Utilize Cobra 2022-04-19 17:09:59 -04:00