cerc-io/stack-orchestrator
17
1
Fork 2
Code Issues 112 Pull Requests 15 Actions Packages Projects Releases 473 Wiki Activity

Fatal: Failed to start mining: etherbase missing: etherbase must be explicitly specified #215

New Issue
Open
opened 2023-03-10 18:30:26 +00:00 by zramsay · 14 comments
zramsay commented 2023-03-10 18:30:26 +00:00
Member
Copy Link

While running the demo, this command led to the following error:

# docker exec 6bc2aa641a1c yarn token:deploy:docker
yarn run v1.22.19
$ hardhat --network docker token-deploy
Downloading compiler 0.8.0
Compiled 5 Solidity files successfully

Error HH108: Cannot connect to the network docker.
Please make sure your node is running, and check your internet connection and networks config
For more info go to https://hardhat.org/HH108 or run Hardhat with --show-stack-traces

It's never been a blocker that the erc20 container is unhealthy (it's in the demo :/), however, I notice smth new with foundry (it hangs at health: starting:

6bc2aa641a1c   cerc/watcher-erc20:local           "docker-entrypoint.s…"   6 minutes ago   Up 5 minutes (unhealthy)           0.0.0.0:3002->3001/tcp, 0.0.0.0:9002->9001/tcp                                                                                                                                         laconic-f50bfdf8077c439670ec741760811ee7-erc20-watcher-1
70f6db331d79   cerc/go-ethereum-foundry:local     "./start-private-net…"   6 minutes ago   Up 45 seconds (health: starting)   127.0.0.1:8545-8546->8545-8546/tcp

then buried in docker logs 70f6db331d79:

Curling: http://0.0.0.0:8545 complete
Geth Start up completed!
testnet:  Account: 0x515B23E9adB9735d69A774f082113C711fA4456B (default)
Fatal: Failed to start mining: etherbase missing: etherbase must be explicitly specified

using: Version: v1.0.26-ee77835
discovered by @Escape613

While running [the demo](http://docs-staging.laconic.com/docs/tutorials/quick-start/), this command led to the following error: ``` # docker exec 6bc2aa641a1c yarn token:deploy:docker yarn run v1.22.19 $ hardhat --network docker token-deploy Downloading compiler 0.8.0 Compiled 5 Solidity files successfully Error HH108: Cannot connect to the network docker. Please make sure your node is running, and check your internet connection and networks config For more info go to https://hardhat.org/HH108 or run Hardhat with --show-stack-traces ``` It's never been a blocker that the erc20 container is unhealthy (it's in the demo :/), however, I notice smth new with foundry (it hangs at `health: starting`: ``` 6bc2aa641a1c cerc/watcher-erc20:local "docker-entrypoint.s…" 6 minutes ago Up 5 minutes (unhealthy) 0.0.0.0:3002->3001/tcp, 0.0.0.0:9002->9001/tcp laconic-f50bfdf8077c439670ec741760811ee7-erc20-watcher-1 70f6db331d79 cerc/go-ethereum-foundry:local "./start-private-net…" 6 minutes ago Up 45 seconds (health: starting) 127.0.0.1:8545-8546->8545-8546/tcp ``` then buried in `docker logs 70f6db331d79`: ``` Curling: http://0.0.0.0:8545 complete Geth Start up completed! testnet: Account: 0x515B23E9adB9735d69A774f082113C711fA4456B (default) Fatal: Failed to start mining: etherbase missing: etherbase must be explicitly specified ``` using: Version: v1.0.26-ee77835 discovered by @Escape613
dboreham commented 2023-03-28 16:50:24 +00:00
Owner
Copy Link

I attempted to reproduce this with the new Debian-based go-ethereum-foundry container.
I wasn't able to see the problem above, but I did notice errors in a couple of the container healthchecks which I fixed here:
https://github.com/cerc-io/stack-orchestrator/pull/252

After those fixes, I get this:

$ docker ps
CONTAINER ID   IMAGE                              COMMAND                  CREATED              STATUS                                                                                                             NAMES
2147d0f8c94f   cerc/watcher-erc20:local           "docker-entrypoint.s…"   About a minute ago   Up 44 seconds (healthy)       
791fefee6bec   cerc/ipld-eth-server:local         "/app/entrypoint.sh"     About a minute ago   Up About a minute (healthy)   
aa9ba2ae1f34   cerc/go-ethereum-foundry:local     "./start-private-net…"   About a minute ago   Up About a minute (healthy)   
52838b1cf396   cerc/ipld-eth-db:local             "/app/startup_script…"   About a minute ago   Up About a minute                                                                                                                   
4bd72d5adb25   postgres:14-alpine                 "docker-entrypoint.s…"   About a minute ago   Up About a minute (healthy)   
61ceca4b2b2d   timescale/timescaledb:2.8.1-pg14   "docker-entrypoint.s…"   About a minute ago   Up About a minute (healthy)
I attempted to reproduce this with the new Debian-based go-ethereum-foundry container. I wasn't able to see the problem above, but I did notice errors in a couple of the container healthchecks which I fixed here: https://github.com/cerc-io/stack-orchestrator/pull/252 After those fixes, I get this: ``` $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES 2147d0f8c94f cerc/watcher-erc20:local "docker-entrypoint.s…" About a minute ago Up 44 seconds (healthy) 791fefee6bec cerc/ipld-eth-server:local "/app/entrypoint.sh" About a minute ago Up About a minute (healthy) aa9ba2ae1f34 cerc/go-ethereum-foundry:local "./start-private-net…" About a minute ago Up About a minute (healthy) 52838b1cf396 cerc/ipld-eth-db:local "/app/startup_script…" About a minute ago Up About a minute 4bd72d5adb25 postgres:14-alpine "docker-entrypoint.s…" About a minute ago Up About a minute (healthy) 61ceca4b2b2d timescale/timescaledb:2.8.1-pg14 "docker-entrypoint.s…" About a minute ago Up About a minute (healthy) ```
dboreham commented 2023-03-28 16:50:38 +00:00
Owner
Copy Link

Closing as WFM. Please re-open if problem persists.

Closing as WFM. Please re-open if problem persists.
zramsay commented 2023-03-29 01:30:08 +00:00
Author
Member
Copy Link

can confirm this is solved, once getting around #245

can confirm this is solved, once getting around #245
dboreham commented 2023-03-30 13:00:55 +00:00
Owner
Copy Link

Error re-occurred with @nikugogoi

Followed the steps in erc20 watcher stack after clearing out all docker images and build cache. Also cleared out local repositories in cerc directory.

# Clear all images
docker rmi $(docker images -aq)

# Clear volumes
docker volume prune

# Clear build cache
docker system prune

# Clear local repositories
rm -rf ~/cerc/

In the setup steps we face issue after running deploy-system up command. On checking health using docker ps, the go-ethereum-foundry container is not healthy.

NOTE: In build-containers step there should be an error for cerc/foundry image not found.
The fix is to include cerc/foundry container and dboreham/foundry repo in erc20 watcher stack.yml
But we get around this by running the following:

laconic-so setup-repositories --include dboreham/foundry

laconic-so build-containers --include cerc/foundry

In container logs we see the following:

Geth started
Geth PID: 64
Curling: http://0.0.0.0:8545/
--datadir /root/ethereum --networkid 99 --port=38545 --mine --miner.threads=1 --allow-insecure-unlock --http --http.api admin,debug,eth,miner,net,personal,txpool,web3,statediff --http.corsdomain * --http.vhosts * --nodiscover --http.addr=0.0.0.0 --http.port=8545 --syncmode=full --gcmode=archive --statediff --statediff.db.host=ipld-eth-db --statediff.db.port=5432 --statediff.db.user=vdbm --statediff.db.password=password --statediff.db.name=cerc_testing --statediff.db.nodeid 1 --statediff.db.clientname test1 --statediff.writing=true --statediff.db.type=postgres --statediff.db.driver=sqlx --ws --ws.addr=0.0.0.0 --ws.origins * --ws.api=admin,debug,eth,miner,net,personal,txpool,web3 --nat=none --miner.gasprice 16000000000 --nat=none --unlock=0xB2fcBbb669f3dEeF689a6A94eA6D843149C266F7 --password=/root/ethereum/config/password --metrics --metrics.expensive --metrics.addr 0.0.0.0 --metrics.port 6060
testnet:  Account: 0xB2fcBbb669f3dEeF689a6A94eA6D843149C266F7 (default)
Curling: http://0.0.0.0:8545/ complete
Geth Start up completed!
Fatal: Failed to start mining: etherbase missing: etherbase must be explicitly specified
++ nc -vz localhost 8545
nc: connect to localhost (127.0.0.1) port 8545 (tcp) failed: Connection refused
nc: connect to localhost (::1) port 8545 (tcp) failed: Cannot assign requested address

This issue should have been resolved as mentioned in https://github.com/cerc-io/stack-orchestrator/issues/215.
But facing it still after clearing out images and repositories.

Error re-occurred with @nikugogoi Followed the steps in [erc20 watcher stack](https://github.com/cerc-io/stack-orchestrator/tree/v1.0.35/app/data/stacks/erc20#readme) after clearing out all docker images and build cache. Also cleared out local repositories in cerc directory. ```bash # Clear all images docker rmi $(docker images -aq) # Clear volumes docker volume prune # Clear build cache docker system prune # Clear local repositories rm -rf ~/cerc/ ``` In the [setup steps](https://github.com/cerc-io/stack-orchestrator/tree/v1.0.35/app/data/stacks/erc20#setup) we face issue after running `deploy-system up` command. On checking health using `docker ps`, the `go-ethereum-foundry` container is not healthy. NOTE: In `build-containers` step there should be an error for `cerc/foundry` image not found. The fix is to include `cerc/foundry` container and `dboreham/foundry` repo in erc20 watcher [stack.yml](https://github.com/cerc-io/stack-orchestrator/blob/v1.0.35/app/data/stacks/erc20/stack.yml) But we get around this by running the following: ``` laconic-so setup-repositories --include dboreham/foundry laconic-so build-containers --include cerc/foundry ``` In container logs we see the following: ``` Geth started Geth PID: 64 Curling: http://0.0.0.0:8545/ --datadir /root/ethereum --networkid 99 --port=38545 --mine --miner.threads=1 --allow-insecure-unlock --http --http.api admin,debug,eth,miner,net,personal,txpool,web3,statediff --http.corsdomain * --http.vhosts * --nodiscover --http.addr=0.0.0.0 --http.port=8545 --syncmode=full --gcmode=archive --statediff --statediff.db.host=ipld-eth-db --statediff.db.port=5432 --statediff.db.user=vdbm --statediff.db.password=password --statediff.db.name=cerc_testing --statediff.db.nodeid 1 --statediff.db.clientname test1 --statediff.writing=true --statediff.db.type=postgres --statediff.db.driver=sqlx --ws --ws.addr=0.0.0.0 --ws.origins * --ws.api=admin,debug,eth,miner,net,personal,txpool,web3 --nat=none --miner.gasprice 16000000000 --nat=none --unlock=0xB2fcBbb669f3dEeF689a6A94eA6D843149C266F7 --password=/root/ethereum/config/password --metrics --metrics.expensive --metrics.addr 0.0.0.0 --metrics.port 6060 testnet: Account: 0xB2fcBbb669f3dEeF689a6A94eA6D843149C266F7 (default) Curling: http://0.0.0.0:8545/ complete Geth Start up completed! Fatal: Failed to start mining: etherbase missing: etherbase must be explicitly specified ++ nc -vz localhost 8545 nc: connect to localhost (127.0.0.1) port 8545 (tcp) failed: Connection refused nc: connect to localhost (::1) port 8545 (tcp) failed: Cannot assign requested address ``` This issue should have been resolved as mentioned in https://github.com/cerc-io/stack-orchestrator/issues/215. But facing it still after clearing out images and repositories.
dboreham commented 2023-04-05 14:21:10 +00:00
Owner
Copy Link

Is this still a problem? I've spun up the geth fixturenet multiple times on several machines lately and not seen this issue, fwiw.

Is this still a problem? I've spun up the geth fixturenet multiple times on several machines lately and not seen this issue, fwiw.
zramsay commented 2023-04-05 19:50:29 +00:00
Author
Member
Copy Link

I've just encountered it running the erc20 stack on the latest S-O version, using the pulled (rather than built) foundry docker image

I've just encountered it running the erc20 stack on the latest S-O version, using the pulled (rather than built) foundry docker image
zramsay commented 2023-04-05 19:55:04 +00:00
Author
Member
Copy Link

mmm, this appears to be a hangover from the PoW -> PoS transition:
https://github.com/ethereum/go-ethereum/issues/26860

https://github.com/ethereum/go-ethereum/issues/19547#issuecomment-492124555

mmm, this appears to be a hangover from the PoW -> PoS transition: https://github.com/ethereum/go-ethereum/issues/26860 https://github.com/ethereum/go-ethereum/issues/19547#issuecomment-492124555
dboreham commented 2023-04-05 20:08:58 +00:00
Owner
Copy Link

Hmm. None of that should be relevant (e.g. foundry isn't able to affect geth).
I wonder if it is some sort of timing thing.
When fixturenet-eth is deployed I think it goes through several phases before reaching the operating state (post-merge).
My hunch is that on some machines (perhaps slower ones?) something tries to do something that doesn't work at the then-pertaining chain state. e.g. it normally passes through the pre-merge state quick enough to not try mining, but if the host is slower it does get a chance to try mining and then fails because we didn't configure it for mining.

@telackey any thoughts on this?

Hmm. None of that should be relevant (e.g. foundry isn't able to affect geth). I wonder if it is some sort of timing thing. When fixturenet-eth is deployed I think it goes through several phases before reaching the operating state (post-merge). My hunch is that on some machines (perhaps slower ones?) something tries to do something that doesn't work at the then-pertaining chain state. e.g. it normally passes through the pre-merge state quick enough to not try mining, but if the host is slower it does get a chance to try mining and then fails because we didn't configure it for mining. @telackey any thoughts on this?
zramsay commented 2023-04-05 20:18:01 +00:00
Author
Member
Copy Link

your hunch might be correct, I think when I had it working last week it was on a 16gb machine and today's failure was on a 8gb. Foundry not building on an 8gb was the canary...

(trying on a 16gb right now, will report back, then we have a clear answer to #229 )

your hunch might be correct, I think when I had it working last week it was on a 16gb machine and today's failure was on a 8gb. Foundry not building on an 8gb was the canary... (trying on a 16gb right now, will report back, then we have a clear answer to #229 )
zramsay commented 2023-04-05 22:07:23 +00:00
Author
Member
Copy Link

:( still same error on a 16g machine

:( still same error on a 16g machine
zramsay commented 2023-04-06 13:47:43 +00:00
Author
Member
Copy Link

same on a 64g machine

same on a 64g machine
dboreham commented 2023-06-23 12:55:32 +00:00
Owner
Copy Link

Closing since this hasn't been reported again.

Closing since this hasn't been reported again.
zramsay commented 2023-08-23 20:39:32 +00:00
Author
Member
Copy Link

this error persists, and can be reproduced with:

laconic-so --stack erc20 setup-repositories
laconic-so --stack erc20 build-containers
laconic-so --stack erc20 deploy up

wait some time

docker ps | grep "watcher-erc20" # get $CONTAINER_ID
docker exec $CONTAINER_ID yarn token:deploy:docker
this error persists, and can be reproduced with: ``` laconic-so --stack erc20 setup-repositories laconic-so --stack erc20 build-containers laconic-so --stack erc20 deploy up ``` wait some time ``` docker ps | grep "watcher-erc20" # get $CONTAINER_ID docker exec $CONTAINER_ID yarn token:deploy:docker ```
telackey commented 2023-08-23 20:49:06 +00:00
Member
Copy Link

In the fixturenet-eth-geth case, we do set the etherbase (https://git.vdb.to/cerc-io/stack-orchestrator/src/branch/main/app/data/container-build/cerc-fixturenet-eth-geth/run-el.sh#L7)

Could something similar be done here?

In the fixturenet-eth-geth case, we do set the etherbase (https://git.vdb.to/cerc-io/stack-orchestrator/src/branch/main/app/data/container-build/cerc-fixturenet-eth-geth/run-el.sh#L7) Could something similar be done here?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
feature

   
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
in progress

   
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

  
Copied from Github

An issue or PR manually copied from GitHub.

  
Kind/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No Label
bug
documentation
duplicate
enhancement
feature
good first issue
help wanted
in progress
invalid
question
wontfix
Copied from Github
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/stack-orchestrator#215
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?