Commit Graph

86 Commits

Author SHA1 Message Date
Abdul Rabbani
4825a54e4e Fix test conditions 2022-06-23 13:47:26 -04:00
Abdul Rabbani
0954dc6159 Update head testing 2022-06-23 11:35:18 -04:00
Abdul Rabbani
16d034c844 Close idle connections and chan's properly. 2022-06-23 09:17:46 -04:00
Abdul Rabbani
32ec784e1e Close channels in the correct location 2022-06-23 08:50:09 -04:00
Abdul Rabbani
9664783eef Remove pprof 2022-06-23 08:26:23 -04:00
Abdul Rabbani
ac1e3075e2 Update system testing 2022-06-23 08:07:19 -04:00
Abdul Rabbani
041f862c11 Make channels buffered 2022-06-23 07:50:36 -04:00
Abdul Rabbani
2299e54197 Use context to end goroutine 2022-06-22 14:44:26 -04:00
Abdul Rabbani
6cecc69520 Don't close DB 2022-06-22 14:09:39 -04:00
Abdul Rabbani
43e0c71639 Improve test stop 2022-06-22 12:36:49 -04:00
Abdul Rabbani
8253201f95 Limit head workers + use single context + skipSse 2022-06-22 12:06:52 -04:00
Abdul Rabbani
67305e07c3 Add condition for SSE subscription 2022-06-21 16:42:51 -04:00
Abdul Rabbani
a2f2603d38 Use context to stop head tracking 2022-06-21 14:34:10 -04:00
Abdul Rabbani
30e8e955ee fix cicd 2022-06-20 14:06:06 -04:00
Abdul Rabbani
79250eed1a Update CICD pipeline for duplicate jobs 2022-06-20 14:04:17 -04:00
Abdul Rabbani
53c4e4243c Use pprof + pointers 2022-06-20 13:56:55 -04:00
Abdul Rabbani
117a2deea9
Slight Performance improvements with Logging (#67)
* Slight Performance improvements with Logging

* Use exec in docker container

* Update packages for dockerfile

* Add flakey test and boot command specific logic

* Remove excess logging and add config val
2022-06-15 11:49:30 -04:00
Abdul Rabbani
d5a9f999b3
Merge pull request #64 from vulcanize/feature/bug-remove-processed-slots
Send slots to processedCh and remove
2022-06-14 09:35:28 -04:00
Abdul Rabbani
cdeb43177c Send slots to processedCh and remove 2022-06-14 09:27:59 -04:00
Abdul Rabbani
67c937418e Improve entrypoint to be more helpful 2022-06-14 09:06:51 -04:00
Abdul Rabbani
64a89d8a46 Fix datarace condition
By creating local variables within each goroutine.
2022-06-13 12:15:18 -04:00
Abdul Rabbani
72eea46bab
Update README and condition for cleanup (#58)
* Update README and condition for cleanup

* Utilize config file for ipld-eth-beacon

* Update CICD to use config file on container build.

* Add env file for build cp

* Add a sleep before running the test
2022-06-10 09:30:52 -04:00
Abdul Rabbani
3c6a9787fd Merge branch 'main' of github.com:vulcanize/ipld-ethcl-indexer into develop 2022-06-09 17:37:30 -04:00
Abdul Rabbani
24fc6358d6
Testing for Batch Processing (#56)
* Set starting slot and improve error gap capturing

* Set starting slot and improve error gap capturing

* Tests + Significant Refactor

The code for historical processing has been significantly refactored to use a context to signify a shutdown.

There have also been many tests added for historical and knownGaps processing.

* Update MhKeys in test

* Update correct values

* Update Max Retry

Genesis is not working as expected.

* Ensure we release locks properly

* Add ordered testing

* Include system tests

* Update workflow calls

* Add secrets

* Add required secrets

* update path

* Try using the absolute path

* Remove volumes at the end.

* Update system-tests.yml

* Update system-tests.yml

* Update test err

* Update and test the shutdown

* rename ethcl --> eth-beacon

* Try forcing /bin/bash for docker-compose

* Update system-tests.yml

* Update system-tests.yml

* Update system-tests.yml

* Update system-tests.yml

* Update system-tests.yml

* Update system-tests.yml

* Use single quote cron

* Dont run generic on schedule
2022-06-09 17:32:46 -04:00
Abdul Rabbani
d674df1ef0
Merge pull request #55 from vulcanize/ian/update_multihash_prefix
update multihash prefix
2022-06-06 15:52:12 -04:00
i-norden
87aa87cb51 update multihash prefix 2022-06-06 13:10:09 -05:00
Abdul Rabbani
4d7781e2b0
Use a transaction for writing, knowngaps, and reorgs (#53)
* Use a transaction for writing, knowngaps, and reorgs

* Checkpoint - Check the DB before writing

* Ensure an error exists when writing a transaction to knowngaps

* Update unneccesary parameter
2022-06-06 09:02:43 -04:00
Abdul Rabbani
e3b4fad3c7
Capture the unique identifier everywhere its needed. (#51)
* Capture the unique identifier everywhere its needed.

* Handle graceful shutdown for historical processing

* Update flags and minor shutdown updates

* Update checkout statement for historic

* update cicd pipeline

* Add secret

* Use deploy-key instead of personal PAT

* update variables
2022-06-03 12:47:13 -04:00
Abdul Rabbani
9160dded11
Add Prometheus Metrics (#49)
* Add Prometheus Metrics

* Fix Prometheus duplicate error

* Handle duplicate registrations
2022-05-25 10:19:29 -04:00
Abdul Rabbani
347984a547
Feature/44 read write historic slots (#46)
* Update boot to set processing type.

* Handle entries from the historic_process table.

* Update on-pr.yml

* Fix head processing Error

* Update names and debug

* Seperate checking for new entries and locking them

* Application can process historic and known gaps

Untested

* Handle genesis

* Update remove entry for knownGaps, viper

* Disregard unused code from linter
2022-05-24 16:18:55 -04:00
Abdul Rabbani
041276da81
Update the boot process (#45)
* Update the boot process

* Update the CI/CD reference for stack-orchestrator
2022-05-19 09:46:38 -04:00
Abdul Rabbani
95e7090087 Update Vulcanize License in a .go files 2022-05-18 12:24:22 -04:00
Abdul Rabbani
891f4c7ef3 Update Vulcanize License in a .go files 2022-05-18 12:12:54 -04:00
Abdul Rabbani
dd6be9ccba Debug docker issues. (#39) 2022-05-17 16:45:40 -04:00
Abdul Rabbani
de38c531d5 Test the application for v1 release. (#37)
* Allow the application to process events in parallel

there is a main thread that tracks incoming messages, but then it spawns goroutines to actually process each slot so that they can happen concurrently.

* Control knownGaps in existing test

* Use Interfaces for different fork version

Use interfaces for `SignedBeaconBlock` and `BeaconState`, this allows the application to determine the correct forked struct.

In the test we also use a switch condition to properly serve the correct mimics.

* Utilize new ipld-ethcl-db repository

* Add final tests

* Update timeout and secret

* Update token

* Update docker compose

* Update expected inserts
2022-05-17 16:45:40 -04:00
Abdul Rabbani
b79c57d5ac Build and Push Image to Docker Registry on Publish or Edit 2022-05-17 16:45:40 -04:00
Ian Norden
7a242a4c71 multihash key gen func (#36)
* multihash key gen func

* go mod updates

* Added test to ensure the application shuts down gracefully or within a timeframe.

* Disregard race condition since its with the test not the application itself

* Capture the head block in the DB entirely. (#27)

* -- Intermediary Commit --

Just want to commit my code over the weekend, in case I spill coffee on my workstation.

* Create DB models ready for write.

* Handle SSE events

* Update ref for stack-orchestrator

* Use env in one place only.

* Boot Application on PR

* Update syntax

* Update syntax

* Correct command

* Use bash instead of sh

* Use until instead of while

* Make linter happy and check sse subscription err

* Handle Reorgs - Untested

* Feature/22 test handling incoming events - Intermediary Commit (#28)

* Checkpoint before the weekend

* Update location for SetupPostgresDB

* Feature/22 test handling incoming events (#30)

* Checkpoint before the weekend

* Update location for SetupPostgresDB

* Include first functioning tests for processing head

* Fix gitignore

* Test CaptureHead | Add Metrics | Handle Test Race Conditions

This Commit allows us to:

* Test the `CaptureHead` function.
* Test parsing a single Head message.
* Test a Reorg condition.
* Add Metrics. This is primarily used for testing but can have future use cases.
* Rearrange the test due to race conditions introduced by reusing a variable. `BeforeEach` can't be used to update `BC`.

* Update and finalize testing at this stage

* Update code and CI/CD

* Fix lint errors

* Update CICD and fail when file not found.

* Update test to have failed as expected.

* Remove Test file

* Add KnownGaps Errors (#33)

* Handle Skipped Slots (#34)

* Ensure that the node is synced at boot time

* Update test + add logic for checking skipped slots

* Update boot check

* Add skip_sync to config.

* Update a test so it fails

* go mod updates

* Integrate MHKey into existing code base.

* Update go.mod and go.sum

* Utilize the MHkey

* Stop tests from running forever on failure.

* Use sszRoot instead of sszObj for MhKey

* Update entrypoint script

* Update config parameter

Co-authored-by: Abdul Rabbani <abdulrabbani00@gmail.com>
Co-authored-by: Abdul Rabbani <58230246+abdulrabbani00@users.noreply.github.com>
2022-05-17 16:45:40 -04:00
Abdul Rabbani
a31c9226ec Handle Skipped Slots (#34)
* Ensure that the node is synced at boot time

* Update test + add logic for checking skipped slots

* Update boot check

* Add skip_sync to config.

* Update a test so it fails
2022-05-17 16:45:40 -04:00
Abdul Rabbani
12675f85c2 Add KnownGaps Errors (#33) 2022-05-17 16:45:40 -04:00
Abdul Rabbani
78089d3fd8 Remove Test file 2022-05-17 16:45:40 -04:00
Abdul Rabbani
6123fa40e8 Feature/22 test handling incoming events (#30)
* Checkpoint before the weekend

* Update location for SetupPostgresDB

* Include first functioning tests for processing head

* Fix gitignore

* Test CaptureHead | Add Metrics | Handle Test Race Conditions

This Commit allows us to:

* Test the `CaptureHead` function.
* Test parsing a single Head message.
* Test a Reorg condition.
* Add Metrics. This is primarily used for testing but can have future use cases.
* Rearrange the test due to race conditions introduced by reusing a variable. `BeforeEach` can't be used to update `BC`.

* Update and finalize testing at this stage

* Update code and CI/CD

* Fix lint errors

* Update CICD and fail when file not found.

* Update test to have failed as expected.
2022-05-17 16:45:40 -04:00
Abdul Rabbani
361dbd73a0 Feature/22 test handling incoming events - Intermediary Commit (#28)
* Checkpoint before the weekend

* Update location for SetupPostgresDB
2022-05-17 16:45:40 -04:00
Abdul Rabbani
d13144ca90 Capture the head block in the DB entirely. (#27)
* -- Intermediary Commit --

Just want to commit my code over the weekend, in case I spill coffee on my workstation.

* Create DB models ready for write.

* Handle SSE events

* Update ref for stack-orchestrator

* Use env in one place only.

* Boot Application on PR

* Update syntax

* Update syntax

* Correct command

* Use bash instead of sh

* Use until instead of while

* Make linter happy and check sse subscription err

* Handle Reorgs - Untested
2022-05-17 16:45:40 -04:00
Abdul Rabbani
71348321ae Disregard race condition since its with the test not the application itself 2022-05-17 16:45:40 -04:00
Abdul Rabbani
6215f7e66f Added test to ensure the application shuts down gracefully or within a timeframe. 2022-05-17 16:45:40 -04:00
Abdul Rabbani
91a1287974
Merge pull request #24 from vulcanize/develop
Interact with Beacon Node endpoints
2022-04-28 11:50:03 -04:00
Abdul Rabbani
164ef4eab4 Remove Binaries 2022-04-28 11:46:14 -04:00
Abdul Rabbani
0b712935a3
Merge pull request #18 from vulcanize/feature/17-interact-with-the-beacon-client
Add functions to interact with the beacon client
2022-04-27 14:07:46 -04:00
Abdul Rabbani
f46c4410a2 Clean up + Get State and Block 2022-04-27 14:01:59 -04:00
Abdul Rabbani
d496886f95 Process SSE messages, and include a graceful shutdown 2022-04-27 10:28:42 -04:00