cerc-io/go-ethereum
12
0
Fork 0
Code Issues 9 Pull Requests Packages Projects Releases 66 Wiki Activity

Minor README fixes, and getting started with doc updates #185

New Issue
Closed
opened 2022-01-07 19:56:27 +00:00 by justindotpub · 3 comments
justindotpub commented 2022-01-07 19:56:27 +00:00 (Migrated from github.com)
Copy Link
Copy Source

System information

./build/bin/geth versionGeth
Version: 1.10.13-statediff-0.0.28
Git Commit: 9d0ec30de1eaf87cd9985bf07a38f9d36f7232fd
Git Commit Date: 20211217
Architecture: arm64
Go Version: go1.17.5
Operating System: darwin
GOPATH=/Users/justin/.go
GOROOT=go

Used 3fb695c1e60d958d229be2c495a90d309dc800e9 of ipld-eth-db to initialize db.

Background

I started working at Fission this week and have been looking at the vulcanize repos, comparing with the Laconic whitepaper, and attempting to understand the moving parts and update docs. I'm not sure if this is considered too early to be submitting issues or PRs for doc clean up, but I wanted to reach out and see how I can help and if someone closer to the code might have random notes or doc they can feed me to help kickstart both my learning and any doc contributions I can make.

Thanks in advance for any guidance and info sharing. 🙏🏻

Details

Here are some issues I've noticed related to this repo's statediff README.

# Run as documented, using dbnodeid of 1 and dbclientname of test1.  Everything else as doc'd.
$ ./build/bin/geth --syncmode=full --gcmode=archive --statediff --statediff.writing --statediff.db=postgres://localhost:5432/vulcanize_testing?sslmode=disable --statediff.db.init=true --statediff.dbnodeid=1 --statediff.dbclientname=test1         
zsh: no matches found: --statediff.db=postgres://localhost:5432/vulcanize_testing?sslmode=disable

# Wrap statediff.db value in quotes
$ ./build/bin/geth --syncmode=full --gcmode=archive --statediff --statediff.writing --statediff.db="postgres://localhost:5432/vulcanize_testing?sslmode=disable" --statediff.db.init=true --statediff.dbnodeid=1 --statediff.dbclientname=test1
Incorrect Usage. flag provided but not defined: -statediff.db.init

...usage info elided...

flag provided but not defined: -statediff.db.init

# Remove statediff.db.init option (after initializing the db manually and using Makefile from the ipfs-eth-db repo to set up the schema)
$ ./build/bin/geth --syncmode=full --gcmode=archive --statediff --statediff.writing --statediff.db="postgres://localhost:5432/vulcanize_testing?sslmode=disable" --statediff.dbnodeid=1 --statediff.dbclientname=test1
...misc output and then this fatal error...
Fatal: Failed to register the Statediff service: unable to set db node: unable to set db node: pq: column "id" does not exist

Does this last error relate to me setting up the database using the wrong version of the schema perhaps?

To set up that db I did the following. Note that I'm not using Docker for the current explorations, though that's on my list of things to look into further as well.

The README mentioned importing from schema.sql, but that file doesn't exist in this repo. It looked to me like running the goose migrations using the Makefile was the way to go.

createdb vulcanize_testing
cd ipld-eth-db
# on main branch
make installtools
make -e NAME=vulcanize_testing migrate
# with the following output in my case
test -n "localhost" # $HOST_NAME
test -n "5432" # $PORT
test -n "vulcanize_testing" # $NAME
postgresql://justin:password@localhost:5432/vulcanize_testing?sslmode=disable
/Users/justin/.go/bin/goose -dir db/migrations postgres "postgresql://justin:password@localhost:5432/vulcanize_testing?sslmode=disable" up
2022/01/07 13:53:36 OK    00001_create_ipfs_blocks_table.sql
2022/01/07 13:53:36 OK    00002_create_nodes_table.sql
2022/01/07 13:53:36 OK    00003_create_eth_schema.sql
2022/01/07 13:53:36 OK    00004_create_eth_header_cids_table.sql
2022/01/07 13:53:36 OK    00005_create_eth_uncle_cids_table.sql
2022/01/07 13:53:36 OK    00006_create_eth_transaction_cids_table.sql
2022/01/07 13:53:36 OK    00007_create_eth_receipt_cids_table.sql
2022/01/07 13:53:36 OK    00008_create_eth_state_cids_table.sql
2022/01/07 13:53:36 OK    00009_create_eth_storage_cids_table.sql
2022/01/07 13:53:36 OK    00010_create_eth_state_accounts_table.sql
2022/01/07 13:53:36 OK    00011_create_eth_access_list_elements_table.sql
2022/01/07 13:53:36 OK    00012_create_eth_log_cids_table.sql
2022/01/07 13:53:36 OK    00013_create_postgraphile_comments.sql
2022/01/07 13:53:36 OK    00014_create_cid_indexes.sql
2022/01/07 13:53:36 OK    00015_create_stored_functions.sql
2022/01/07 13:53:36 OK    00016_create_postgraphile_triggers.sql
2022/01/07 13:53:36 OK    00017_create_db_version_table.sql
2022/01/07 13:53:36 OK    00018_update_db_version.sql
2022/01/07 13:53:36 goose: no migrations to run. current version: 18
pg_dump -O -s postgresql://justin:password@localhost:5432/vulcanize_testing?sslmode=disable > schema.sql

Thank you. 🙇🏻‍♂️

#### System information ``` ./build/bin/geth versionGeth Version: 1.10.13-statediff-0.0.28 Git Commit: 9d0ec30de1eaf87cd9985bf07a38f9d36f7232fd Git Commit Date: 20211217 Architecture: arm64 Go Version: go1.17.5 Operating System: darwin GOPATH=/Users/justin/.go GOROOT=go ``` Used `3fb695c1e60d958d229be2c495a90d309dc800e9` of `ipld-eth-db` to initialize db. #### Background I started working at Fission this week and have been looking at the vulcanize repos, comparing with the Laconic whitepaper, and attempting to understand the moving parts and update docs. I'm not sure if this is considered too early to be submitting issues or PRs for doc clean up, but I wanted to reach out and see how I can help and if someone closer to the code might have random notes or doc they can feed me to help kickstart both my learning and any doc contributions I can make. Thanks in advance for any guidance and info sharing. 🙏🏻 #### Details Here are some issues I've noticed related to this repo's statediff README. ```shell # Run as documented, using dbnodeid of 1 and dbclientname of test1. Everything else as doc'd. $ ./build/bin/geth --syncmode=full --gcmode=archive --statediff --statediff.writing --statediff.db=postgres://localhost:5432/vulcanize_testing?sslmode=disable --statediff.db.init=true --statediff.dbnodeid=1 --statediff.dbclientname=test1 zsh: no matches found: --statediff.db=postgres://localhost:5432/vulcanize_testing?sslmode=disable # Wrap statediff.db value in quotes $ ./build/bin/geth --syncmode=full --gcmode=archive --statediff --statediff.writing --statediff.db="postgres://localhost:5432/vulcanize_testing?sslmode=disable" --statediff.db.init=true --statediff.dbnodeid=1 --statediff.dbclientname=test1 Incorrect Usage. flag provided but not defined: -statediff.db.init ...usage info elided... flag provided but not defined: -statediff.db.init # Remove statediff.db.init option (after initializing the db manually and using Makefile from the ipfs-eth-db repo to set up the schema) $ ./build/bin/geth --syncmode=full --gcmode=archive --statediff --statediff.writing --statediff.db="postgres://localhost:5432/vulcanize_testing?sslmode=disable" --statediff.dbnodeid=1 --statediff.dbclientname=test1 ...misc output and then this fatal error... Fatal: Failed to register the Statediff service: unable to set db node: unable to set db node: pq: column "id" does not exist ``` Does this last error relate to me setting up the database using the wrong version of the schema perhaps? To set up that db I did the following. Note that I'm not using Docker for the current explorations, though that's on my list of things to look into further as well. The README mentioned importing from schema.sql, but that file doesn't exist in this repo. It looked to me like running the goose migrations using the Makefile was the way to go. ``` createdb vulcanize_testing cd ipld-eth-db # on main branch make installtools make -e NAME=vulcanize_testing migrate # with the following output in my case test -n "localhost" # $HOST_NAME test -n "5432" # $PORT test -n "vulcanize_testing" # $NAME postgresql://justin:password@localhost:5432/vulcanize_testing?sslmode=disable /Users/justin/.go/bin/goose -dir db/migrations postgres "postgresql://justin:password@localhost:5432/vulcanize_testing?sslmode=disable" up 2022/01/07 13:53:36 OK 00001_create_ipfs_blocks_table.sql 2022/01/07 13:53:36 OK 00002_create_nodes_table.sql 2022/01/07 13:53:36 OK 00003_create_eth_schema.sql 2022/01/07 13:53:36 OK 00004_create_eth_header_cids_table.sql 2022/01/07 13:53:36 OK 00005_create_eth_uncle_cids_table.sql 2022/01/07 13:53:36 OK 00006_create_eth_transaction_cids_table.sql 2022/01/07 13:53:36 OK 00007_create_eth_receipt_cids_table.sql 2022/01/07 13:53:36 OK 00008_create_eth_state_cids_table.sql 2022/01/07 13:53:36 OK 00009_create_eth_storage_cids_table.sql 2022/01/07 13:53:36 OK 00010_create_eth_state_accounts_table.sql 2022/01/07 13:53:36 OK 00011_create_eth_access_list_elements_table.sql 2022/01/07 13:53:36 OK 00012_create_eth_log_cids_table.sql 2022/01/07 13:53:36 OK 00013_create_postgraphile_comments.sql 2022/01/07 13:53:36 OK 00014_create_cid_indexes.sql 2022/01/07 13:53:36 OK 00015_create_stored_functions.sql 2022/01/07 13:53:36 OK 00016_create_postgraphile_triggers.sql 2022/01/07 13:53:36 OK 00017_create_db_version_table.sql 2022/01/07 13:53:36 OK 00018_update_db_version.sql 2022/01/07 13:53:36 goose: no migrations to run. current version: 18 pg_dump -O -s postgresql://justin:password@localhost:5432/vulcanize_testing?sslmode=disable > schema.sql ``` Thank you. 🙇🏻‍♂️
justindotpub commented 2022-01-07 22:16:41 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Thinking aloud about things I'm sure you don't need me to tell you...

It looks like the id error is due to https://github.com/vulcanize/go-ethereum/blob/statediff/statediff/indexer/postgres/postgres.go#L69 returning id but the nodes table not having an id column. Git history isn't giving me any clues that something changed recently in this code or the migrations in ipld-eth-db, so I'll speculate that the db changed structure (or ways the schema is managed) maybe a month back and statediff geth hasn't caught up? Or perhaps somehow related to the migration from v2 to v3 of the database mentioned in migration-tools? The statediff branch seems to be tracking with the v3 tags so I think I ran them from the right branch.

Any whom, enough speculating for tonight. Thanks again. 🤝

Thinking aloud about things I'm sure you don't need me to tell you... It looks like the `id` error is due to https://github.com/vulcanize/go-ethereum/blob/statediff/statediff/indexer/postgres/postgres.go#L69 returning `id` but the `nodes` table not having an `id` column. Git history isn't giving me any clues that something changed recently in this code or the [migrations](https://github.com/vulcanize/ipld-eth-db/tree/main/db/migrations) in `ipld-eth-db`, so I'll speculate that the db changed structure (or ways the schema is managed) maybe a month back and statediff geth hasn't caught up? Or perhaps somehow related to the migration from v2 to v3 of the database mentioned in `migration-tools`? The `statediff` branch seems to be tracking with the v3 tags so I think I ran them from the right branch. Any whom, enough speculating for tonight. Thanks again. 🤝
justindotpub commented 2022-01-08 11:17:46 +00:00 (Migrated from github.com)
Copy Link
Copy Source

See https://github.com/vulcanize/go-ethereum/pull/186.

See https://github.com/vulcanize/go-ethereum/pull/186.
i-norden commented 2022-01-08 19:06:34 +00:00
Member
Copy Link
Copy Source

Hi @justincjohnson thank you and sorry I didn't get to you sooner, the statediff branch was outdated. No great excuse for letting that happen, but it is in part due to the fact that we are in a transitory phase where we need to maintain, and continually rebase, two versions of the statediffing code: one for our v0.2.x version of the DB (v1.10.14-statediff-0.0.29) schema and one for our v0.3.x version of the DB (v1.10.11-statediff-0.1.0).

The cause of the id error is due to this statediff branch tracking a version of the code that works with the v0.2.x DB schema, whereas the main branch of ipld-eth-db is at the v0.3.x version.

The safest way to checkout the code- until we finish the v2 => v3 migration and clean up our organization and documentation of this statediff branch- is to checkout one of the tagged release. In the release notes there will be a note indicating which version of the DB schema it works with.

Again, sorry for the confusion, this issue is very helpful in highlighting the issues we have with tracking versions in and across repos.

Hi @justincjohnson thank you and sorry I didn't get to you sooner, the `statediff` branch was outdated. No great excuse for letting that happen, but it is in part due to the fact that we are in a transitory phase where we need to maintain, and continually rebase, two versions of the statediffing code: one for our [v0.2.x](https://github.com/vulcanize/ipld-eth-db/releases/tag/v0.2.1) version of the DB (v1.10.14-statediff-0.0.29) schema and one for our [v0.3.x](https://github.com/vulcanize/ipld-eth-db/releases/tag/v0.3.3) version of the DB (v1.10.11-statediff-0.1.0). The cause of the `id` error is due to this `statediff` branch tracking a version of the code that works with the v0.2.x DB schema, whereas the main branch of ipld-eth-db is at the v0.3.x version. The safest way to checkout the code- until we finish the v2 => v3 migration and clean up our organization and documentation of this statediff branch- is to checkout one of the [tagged release](https://github.com/vulcanize/go-ethereum/releases). In the release notes there will be a note indicating which version of the DB schema it works with. Again, sorry for the confusion, this issue is very helpful in highlighting the issues we have with tracking versions in and across repos.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
critical
duplicate
enhancement
epic
help wanted
in progress
invalid
low priority
question
rebase
v1
v5
wontfix
Copied from Github
An issue or PR manually copied from GitHub.
 Kind/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
AFDudley (A. F. Dudley) Alex ashwin cerccicd ericlewis i-norden (Ian Norden) prathamesh (Prathamesh Musale) rachmaninovquar rjk-laconic roysc (Roy Crihfield) srw (srw) zramsay
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cerc-io/go-ethereum#185