Commit Graph

13484 Commits

Author SHA1 Message Date
Abdul Rabbani
1778c0c3f1 minor syntax change 2022-06-27 15:28:44 -04:00
Abdul Rabbani
54a623004d Add uniqueness 2022-06-27 15:28:44 -04:00
Abdul Rabbani
5b598d71bc Use stack orchestrator properly 2022-06-27 15:28:44 -04:00
Abdul Rabbani
6805070b1e Update "file" and CICD 2022-06-27 15:28:44 -04:00
Abdul Rabbani
601b8e778c Update upsert for eth.header_cids 2022-06-27 15:28:44 -04:00
Abdul Rabbani
569bfaad0b Combine build and push 2022-06-27 15:28:44 -04:00
ed370ccb84 Changes for using single-node v4 db schema 2022-06-27 15:28:35 -04:00
05b0ddcffc Add new target branch for github actions 2022-06-27 15:28:35 -04:00
a78d590b65 Implemented method to get private watchedAddressesLeafKeys 2022-06-27 15:28:34 -04:00
2e79df1884 Export builder config param WatchedAddressesLeafKeys 2022-06-27 15:28:34 -04:00
8711a970d3 Update more test queries to use COALESCE 2022-06-27 15:28:34 -04:00
8ba6f5de1d Use COALESCE to avoid possible NULL values from database 2022-06-27 15:28:34 -04:00
cc14556448 Change ON CONFLICT clauses to do nothing 2022-06-27 15:28:34 -04:00
Abdul Rabbani
cd70c6e741 Remove related-repository 2022-06-27 15:28:34 -04:00
Abdul Rabbani
04f08a0ae0 Remove redundant deferences for pointers 2022-06-27 15:28:34 -04:00
Abdul Rabbani
c55f7d3098 Handle all breaking changes for the patch
The major change to integrate the patch has been:

* Updating the arguments for `rlp.EncodeToBytes`. This function accepts interfaces, but since the last update, it is better to pass in a pointer to the function.
* From the Ethereum Release Notes: "
Compatibility note about `core/types`: For optimization purposes, `types.Header` and other types in this package now implement the `rlp.Encoder` interface. This change can cause incompatibilities because the new method is implemented with pointer receiver. Attempting to RLP-encode unadressable (i.e. non-pointer) values of type `Header` does not work anymore and will result in an error."
* Instead of just updating all the headers. I have updated all parameters for the `rlp.EncodeToBytes` to be pointers instead of values.

__Please take a close look at the updates__. The functions won't fail if a non-pointer is passed (in most cases), but we could be unexpected behaviors.
2022-06-27 15:28:34 -04:00
Abdul Rabbani
3fab9cf7a8 Update a few go dependencies + GHA workflow 2022-06-27 15:28:34 -04:00
94664b1b4a Updated unit tests for changes in builder 2022-06-27 15:28:34 -04:00
536cd32b46 Create file for watched addresses only when required 2022-06-27 15:28:34 -04:00
a47a90e9d3 Add indexer unit tests for loading watched addresses 2022-06-27 15:28:34 -04:00
319b13c44e Add file mode indexer unit tests for watched address methods 2022-06-27 15:28:34 -04:00
7481861d58 Implement watch address indexer methods for file mode 2022-06-27 15:28:34 -04:00
3cdb78d194 Add builder test for removal of watched addresses 2022-06-27 15:28:34 -04:00
02b1d6bf81 Add a fix in builder for removal of a non-watched address 2022-06-27 15:28:34 -04:00
cf8ba30102 Add tests for indexer methods used to change addresses being watched 2022-06-27 15:28:34 -04:00
a40f02a3cb Statediff API to change addresses being watched in direct indexing 2022-06-27 15:28:34 -04:00
Abdul Rabbani
695e38718c Add Workflow dispatch for docker builds 2022-06-27 15:28:34 -04:00
Abdul Rabbani
08053b039c Update known_gaps table name 2022-06-27 15:28:34 -04:00
Abdul Rabbani
f2b4c0b1ec PR Updates as per Ian's request 2022-06-27 15:28:34 -04:00
Abdul Rabbani
68a0c09657 Code Clean Up 2022-06-27 15:28:34 -04:00
Abdul Rabbani
bcfc34dd81 Add Prometheus metrics and write SQL file if it has data. 2022-06-27 15:28:34 -04:00
Abdul Rabbani
9f92fe6e7c Refactor: Decouple knownGaps and Indexer
This commit decouples knownGaps and Indexer. All knownGaps logic is in its own file. This makes testing and maintainability easier.

We have also removed all efforts to check the `lastProcessedblock` - This is because we won't ever run into this issue (hyptothetically), because geth won't let it happen.
2022-06-27 15:28:34 -04:00
Abdul Rabbani
2afccedd22 ** Write to file if you can't write to DB
This commit contains many new features:

1. Write the insert statement to file if there is an error when trying to write to DB.
2. Test the write to DB and write to file features in unit tests. They work as expected.
3. Insert logic for calling the knownGaps function (untested).

This PR makes a major change to the way statediff is started. Unless it is started in `file` mode, it will create a `file` indexer. This indexer will be used for writing errors to disk.
2022-06-27 15:28:34 -04:00
Abdul Rabbani
4774b01573 % Checkpoint - Added the feature to write to File if writing to DB errors out. NOT TESTED 2022-06-27 15:28:33 -04:00
Abdul Rabbani
a2bbeaa1d1 Handle All Gaps within Geth
Including an updated doc which keeps track of events in this PR.
2022-06-27 15:28:33 -04:00
Abdul Rabbani
22b1b245bb Find Gaps and at them to known_gaps
This contains all the logic needed to add a gap to the `known_gaps` table.

We can now add this code to various parts of the application to calculate if a gap has occured.
2022-06-27 15:28:33 -04:00
Abdul Rabbani
07cc0b3b34 Improve Testing for Insert, add a query function 2022-06-27 15:28:33 -04:00
Abdul Rabbani
e3df60c506 Remove the need to pass a transaction when upserting to known_gaps
Thanks Ian :D
2022-06-27 15:28:33 -04:00
Abdul Rabbani
2807a8953c Delete geth-linux-amd64 2022-06-27 15:28:33 -04:00
Abdul Rabbani
7e9dfeec85 Function To Add entry to known_gaps
This commit contains all the code needed to update the `known_gaps` table. It uses a generic function `PushKnownGaps` to insert gaps to the database.

The current test is very "basic." All it does it insert a dummy gap into the database. As long as their is no error it will return succesfully.

I have also removed the foundry subfolder. I am using the `related-repository` setup for testing.
2022-06-27 15:28:32 -04:00
Abdul Rabbani
1d34c1c26f Add a flag to start processing statediff if we are caught up to the head of the chain 2022-06-27 15:28:32 -04:00
Abdul Rabbani
5b7af4b41d Basic updates, add prints and variables 2022-06-27 15:28:32 -04:00
i-norden
493d71cc13 update unit tests 2022-06-27 15:28:32 -04:00
i-norden
16ac439546 update file writing mode 2022-06-27 15:28:32 -04:00
i-norden
5e77593c0e update sql indexer to use new v4 schema that denormalizes by block_number for the purposes of partitioning & sharding 2022-06-27 15:28:31 -04:00
Abdul Rabbani
22f77861db Fix Linter 2022-06-27 15:28:31 -04:00
Abdul Rabbani
ac73289ed4 Working Foundry 2022-06-27 15:28:31 -04:00
Abdul Rabbani
91fe51fd54 forge install: ds-test 2022-06-27 15:28:31 -04:00
Abdul Rabbani
9ec5f6930d Starting point for Foundry
I still need to finish it up, but geth runs properly.
2022-06-27 15:28:31 -04:00
i-norden
93a6978b05 add new target branch for github actions 2022-06-27 15:28:31 -04:00