Add a service (v3) to fill indexing gap for watched addresses #149

Merged
ashwinphatak merged 16 commits from ng-watched-addresses-v3 into master 2022-05-18 08:54:04 +00:00
ashwinphatak commented 2022-05-02 07:52:29 +00:00 (Migrated from github.com)

Part of https://github.com/vulcanize/go-ethereum/issues/127

  • An API to change addresses being watched by geth for direct indexing
  • Integration tests for the API
  • A service to fill indexing gap for watched addresses
  • Unit and Integration tests for the service
  • Integration tests for redeployment of a destroyed contract
  • Upgrade ipfs-ethdb to v3.0.1 and eth-ipfs-state-validator to v3.0.0
  • Use foundry setup to run integration tests
  • Upgrade vulcanize/geth to v1.10.17-statediff-3.2.1
  • Change module path from github.com/vulcanize/ipld-eth-server to github.com/vulcanize/ipld-eth-server/v3
Part of https://github.com/vulcanize/go-ethereum/issues/127 - An API to change addresses being watched by geth for direct indexing - Integration tests for the API - A service to fill indexing gap for watched addresses - Unit and Integration tests for the service - Integration tests for redeployment of a destroyed contract - Upgrade ipfs-ethdb to `v3.0.1` and eth-ipfs-state-validator to `v3.0.0` - Use foundry setup to run integration tests - Upgrade vulcanize/geth to `v1.10.17-statediff-3.2.1` - Change module path from `github.com/vulcanize/ipld-eth-server` to `github.com/vulcanize/ipld-eth-server/v3`
i-norden reviewed 2022-05-10 04:59:10 +00:00
i-norden left a comment
Member

Code looks great but I don't yet grasp why the pkg/fill/service.go belongs in this repo. We've already moved a db validation command here for convenience but a difference with this new fill process is that it mutates the database directly (granted only an "eth_meta" table and not any of the "eth" schema).

Code looks great but I don't yet grasp why the pkg/fill/service.go belongs in this repo. We've already moved a db validation command here for convenience but a difference with this new fill process is that it mutates the database directly (granted only an "eth_meta" table and not any of the "eth" schema).
Member

Should the service have some type of shutdown procedure in response to the interrupt notify below?

Should the service have some type of shutdown procedure in response to the interrupt notify below?
Member

I don't see/remember why this and eth-ipfs-state-validator need to be tagged with v3 since we aren't importing and using multiple versions of them

I don't see/remember why this and eth-ipfs-state-validator need to be tagged with v3 since we aren't importing and using multiple versions of them
Member

If this is a standalone background process that has no server component and is disconnected from the rest of ipld-eth-server I'm not sure it belongs in this repo.

If this is a standalone background process that has no server component and is disconnected from the rest of ipld-eth-server I'm not sure it belongs in this repo.
@ -0,0 +171,4 @@
// UpdateLastFilledAt updates the fill status for the provided addresses in the db
func (s *Service) UpdateLastFilledAt(blockNumber uint64, fillAddresses []interface{}) {
// Prepare the query
query := "UPDATE eth_meta.watched_addresses SET last_filled_at=? WHERE address IN (?" + strings.Repeat(",?", len(fillAddresses)-1) + ")"
Member

ipld-eth-server writing to the DB itself is outside of its usual scope

ipld-eth-server writing to the DB itself is outside of its usual scope
prathamesh0 (Migrated from github.com) reviewed 2022-05-10 08:27:08 +00:00
@ -0,0 +171,4 @@
// UpdateLastFilledAt updates the fill status for the provided addresses in the db
func (s *Service) UpdateLastFilledAt(blockNumber uint64, fillAddresses []interface{}) {
// Prepare the query
query := "UPDATE eth_meta.watched_addresses SET last_filled_at=? WHERE address IN (?" + strings.Repeat(",?", len(fillAddresses)-1) + ")"
prathamesh0 (Migrated from github.com) commented 2022-05-10 08:27:08 +00:00

Implemented as per the discussion here: https://github.com/vulcanize/go-ethereum/issues/127#issuecomment-1011819114
We can move the service to a new repository if desired.

Implemented as per the discussion here: https://github.com/vulcanize/go-ethereum/issues/127#issuecomment-1011819114 We can move the service to a new repository if desired.
prathamesh0 (Migrated from github.com) reviewed 2022-05-10 08:49:31 +00:00
prathamesh0 (Migrated from github.com) commented 2022-05-10 08:49:31 +00:00

Required if we want to include them as a dependency with major version > 1.
See: https://github.com/golang/go/wiki/Modules#semantic-import-versioning

Required if we want to include them as a dependency with major version > 1. See: https://github.com/golang/go/wiki/Modules#semantic-import-versioning
i-norden reviewed 2022-05-10 19:00:58 +00:00
Member

I see, I must have been manifesting false memories of having imported > 1 semantically versioned stuff before. Thanks!

I see, I must have been manifesting false memories of having imported > 1 semantically versioned stuff before. Thanks!
i-norden reviewed 2022-05-10 19:02:07 +00:00
@ -0,0 +171,4 @@
// UpdateLastFilledAt updates the fill status for the provided addresses in the db
func (s *Service) UpdateLastFilledAt(blockNumber uint64, fillAddresses []interface{}) {
// Prepare the query
query := "UPDATE eth_meta.watched_addresses SET last_filled_at=? WHERE address IN (?" + strings.Repeat(",?", len(fillAddresses)-1) + ")"
Member

Not sure I was part of that discussion

Not sure I was part of that discussion
i-norden reviewed 2022-05-10 19:05:46 +00:00
@ -0,0 +171,4 @@
// UpdateLastFilledAt updates the fill status for the provided addresses in the db
func (s *Service) UpdateLastFilledAt(blockNumber uint64, fillAddresses []interface{}) {
// Prepare the query
query := "UPDATE eth_meta.watched_addresses SET last_filled_at=? WHERE address IN (?" + strings.Repeat(",?", len(fillAddresses)-1) + ")"
Member

Curious what @ashwinphatak thinks, I just want to be cautious about adding more and more misc bells and whistles to this repo as the original intent was for it to have a focused and well-defined function that was encapsulated by its name. We already broke the convention with the addition of the validation command but imo that was supposed to be an exception to the rule and not meant to set a new precedent, and that validation command is at least read only.

Curious what @ashwinphatak thinks, I just want to be cautious about adding more and more misc bells and whistles to this repo as the original intent was for it to have a focused and well-defined function that was encapsulated by its name. We already broke the convention with the addition of the validation command but imo that was supposed to be an exception to the rule and not meant to set a new precedent, and that validation command is at least read only.
i-norden reviewed 2022-05-10 19:07:57 +00:00
@ -0,0 +171,4 @@
// UpdateLastFilledAt updates the fill status for the provided addresses in the db
func (s *Service) UpdateLastFilledAt(blockNumber uint64, fillAddresses []interface{}) {
// Prepare the query
query := "UPDATE eth_meta.watched_addresses SET last_filled_at=? WHERE address IN (?" + strings.Repeat(",?", len(fillAddresses)-1) + ")"
Member

And the designation that this binary is read-only on the DB is kind of nice from an ops perspective.

And the designation that this binary is read-only on the DB is kind of nice from an ops perspective.
i-norden reviewed 2022-05-10 19:10:56 +00:00
@ -0,0 +171,4 @@
// UpdateLastFilledAt updates the fill status for the provided addresses in the db
func (s *Service) UpdateLastFilledAt(blockNumber uint64, fillAddresses []interface{}) {
// Prepare the query
query := "UPDATE eth_meta.watched_addresses SET last_filled_at=? WHERE address IN (?" + strings.Repeat(",?", len(fillAddresses)-1) + ")"
Member

But on the otherhand, Shane not having to keep track of another binary has its benefits from the ops perspective.

But on the otherhand, Shane not having to keep track of another binary has its benefits from the ops perspective.
i-norden approved these changes 2022-05-18 04:31:34 +00:00
i-norden left a comment
Member

Approved per our on-call discussions

Approved per our on-call discussions
prathamesh0 (Migrated from github.com) reviewed 2022-05-18 08:35:56 +00:00
@ -0,0 +171,4 @@
// UpdateLastFilledAt updates the fill status for the provided addresses in the db
func (s *Service) UpdateLastFilledAt(blockNumber uint64, fillAddresses []interface{}) {
// Prepare the query
query := "UPDATE eth_meta.watched_addresses SET last_filled_at=? WHERE address IN (?" + strings.Repeat(",?", len(fillAddresses)-1) + ")"
prathamesh0 (Migrated from github.com) commented 2022-05-18 08:35:55 +00:00

The watched addresses fill service will be extracted to a separate repository.
See: https://github.com/vulcanize/ops/issues/156

The watched addresses fill service will be extracted to a separate repository. See: https://github.com/vulcanize/ops/issues/156
prathamesh0 (Migrated from github.com) reviewed 2022-05-18 08:36:14 +00:00
prathamesh0 (Migrated from github.com) commented 2022-05-18 08:36:14 +00:00

Graceful shutdown implemented.

Graceful shutdown implemented.
Sign in to join this conversation.
No reviewers
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/ipld-eth-server#149
No description provided.