cerc-io/stack-orchestrator
17
1
Fork 2
Code Issues 112 Pull Requests 15 Actions Packages Projects Releases 473 Wiki Activity

Error on trying to deploy the Lotus fixturenet #481

New Issue
Closed
opened 2023-08-03 06:10:33 +00:00 by prathamesh0 · 4 comments
prathamesh0 commented 2023-08-03 06:10:33 +00:00 (Migrated from github.com)
Copy Link

On trying to run the fixturenet-lotus stack following steps from fixturenet-lotus (stack-orchestrator v1.1.0-2da03d7-202308010420), getting an error in the deploy up step:

(venv) prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/stack-orchestrator$ laconic-so --stack fixturenet-lotus deploy up
[+] Running 5/5
 ✔ Network laconic-01dfd0e5e8a0e82ba6858771df1e3da8_default           Created                                                                                                     0.1s 
 ✔ Volume "laconic-01dfd0e5e8a0e82ba6858771df1e3da8_lotus-shared"     Created                                                                                                     0.0s 
 ✘ Container laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1   Error                                                                                                     152.1s 
 ✔ Container laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-node-1-1  Created                                                                                                     0.1s 
 ✔ Container laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-node-2-1  Created                                                                                                     0.1s 
dependency failed to start: container laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1 is unhealthy
Traceback (most recent call last):
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/bin/laconic-so", line 33, in <module>
    sys.exit(load_entry_point('laconic-stack-orchestrator', 'console_scripts', 'laconic-so')())
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/prathamesh/deepstack/stack-orchestrator/app/deploy.py", line 164, in up
    up_operation(ctx, extra_args_list)
  File "/home/prathamesh/deepstack/stack-orchestrator/app/deploy.py", line 71, in up_operation
    deploy_context.docker.compose.up(detach=not stay_attached, services=services_list)
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/python_on_whales/components/compose/cli_wrapper.py", line 749, in up
    run(full_cmd, capture_stdout=quiet, capture_stderr=quiet)
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/python_on_whales/utils.py", line 191, in run
    raise DockerException(
python_on_whales.exceptions.DockerException: The docker command executed was `/usr/bin/docker compose --file /home/prathamesh/deepstack/stack-orchestrator/app/data/compose/docker-compose-fixturenet-lotus.yml --project-name laconic-01dfd0e5e8a0e82ba6858771df1e3da8 up --detach`.
It returned with code 1
The content of stdout can be found above the stacktrace (it wasn't captured).
The content of stderr can be found above the stacktrace (it wasn't captured).

lotus-miner container logs:

(venv) prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/stack-orchestrator$ laconic-so --stack fixturenet-lotus deploy logs lotus-miner
laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1  | lotus version 1.23.2+2k+git.8347235b5.dirty
laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1  | Waiting for daemon to start...
On trying to run the `fixturenet-lotus` stack following steps from [fixturenet-lotus](https://github.com/cerc-io/stack-orchestrator/tree/main/app/data/stacks/fixturenet-lotus) (stack-orchestrator [`v1.1.0-2da03d7-202308010420`](https://github.com/cerc-io/stack-orchestrator/releases/tag/v1.1.0-2da03d7-202308010420)), getting an error in the `deploy up` step: ```bash (venv) prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/stack-orchestrator$ laconic-so --stack fixturenet-lotus deploy up [+] Running 5/5 ✔ Network laconic-01dfd0e5e8a0e82ba6858771df1e3da8_default Created 0.1s ✔ Volume "laconic-01dfd0e5e8a0e82ba6858771df1e3da8_lotus-shared" Created 0.0s ✘ Container laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1 Error 152.1s ✔ Container laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-node-1-1 Created 0.1s ✔ Container laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-node-2-1 Created 0.1s dependency failed to start: container laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1 is unhealthy Traceback (most recent call last): File "/home/prathamesh/deepstack/stack-orchestrator/venv/bin/laconic-so", line 33, in <module> sys.exit(load_entry_point('laconic-stack-orchestrator', 'console_scripts', 'laconic-so')()) File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/home/prathamesh/deepstack/stack-orchestrator/app/deploy.py", line 164, in up up_operation(ctx, extra_args_list) File "/home/prathamesh/deepstack/stack-orchestrator/app/deploy.py", line 71, in up_operation deploy_context.docker.compose.up(detach=not stay_attached, services=services_list) File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/python_on_whales/components/compose/cli_wrapper.py", line 749, in up run(full_cmd, capture_stdout=quiet, capture_stderr=quiet) File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/python_on_whales/utils.py", line 191, in run raise DockerException( python_on_whales.exceptions.DockerException: The docker command executed was `/usr/bin/docker compose --file /home/prathamesh/deepstack/stack-orchestrator/app/data/compose/docker-compose-fixturenet-lotus.yml --project-name laconic-01dfd0e5e8a0e82ba6858771df1e3da8 up --detach`. It returned with code 1 The content of stdout can be found above the stacktrace (it wasn't captured). The content of stderr can be found above the stacktrace (it wasn't captured). ``` `lotus-miner` container logs: ```bash (venv) prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/stack-orchestrator$ laconic-so --stack fixturenet-lotus deploy logs lotus-miner laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1 | lotus version 1.23.2+2k+git.8347235b5.dirty laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1 | Waiting for daemon to start... ```
prathamesh0 commented 2023-08-03 06:55:16 +00:00 (Migrated from github.com)
Copy Link

Getting same error on trying with stack-orchestrator release where fixturenet-lotus was added (v1.1.0-5ecfcae-202305172112)

Getting same error on trying with `stack-orchestrator` release where `fixturenet-lotus` was added ([`v1.1.0-5ecfcae-202305172112`](https://github.com/cerc-io/stack-orchestrator/releases/tag/v1.1.0-5ecfcae-202305172112))
dboreham commented 2023-08-03 13:07:01 +00:00
Owner
Copy Link

Could it be a healthcheck failure? Try commenting out the healthcheck here: https://github.com/cerc-io/stack-orchestrator/blob/main/app/data/compose/docker-compose-fixturenet-lotus.yml#L17 and see if compose completes the start up. Then it might be easier to diagnose why the healthcheck failed.

Could it be a healthcheck failure? Try commenting out the healthcheck here: https://github.com/cerc-io/stack-orchestrator/blob/main/app/data/compose/docker-compose-fixturenet-lotus.yml#L17 and see if compose completes the start up. Then it might be easier to diagnose why the healthcheck failed.
prathamesh0 commented 2023-08-03 13:54:10 +00:00 (Migrated from github.com)
Copy Link

After removing the healthcheck, the miner node seems to have fetched the params but has failed to initialize:

prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/uniswap-v3-periphery$ docker logs -f laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1
lotus version 1.23.2+2k+git.8347235b5.dirty
Waiting for daemon to start...
Daemon started.
ERROR: open /root/.genesis-sectors/pre-seal-t01000.key: no such file or directory

2023-08-03T13:22:03.559Z	INFO	main	lotus-miner/init.go:129	Initializing lotus miner
2023-08-03T13:22:03.562Z	INFO	main	lotus-miner/init.go:156	Checking proof parameters
2023-08-03T13:22:03.566Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-8-2-102e1444a7e9a97ebf1e3d6855dcc77e66c011ea66f936d9b2c508f87f2f83a7.vk is ok
2023-08-03T13:22:03.566Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0cfb4f178bbb71cf2ecfcd42accce558b27199ab4fb59cb78f2483fe21ef36d9.vk is ok
2023-08-03T13:22:03.565Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-5294475db5237a2e83c3e52fd6c2b03859a1831d45ed08c4f35dbf9a803165a9.vk is ok
2023-08-03T13:22:03.567Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-ecd683648512ab1765faa2a5f14bab48f676e633467f0aa8aad4b55dcb0652bb.vk is ok
2023-08-03T13:22:03.567Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-92180959e1918d26350b8e6cfe217bbdd0a2d8de51ebec269078b364b715ad63.vk is ok
2023-08-03T13:22:03.567Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.vk is ok
2023-08-03T13:22:03.566Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.vk is ok
2023-08-03T13:22:03.565Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:244	Fetching /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.params from https://proofs.filecoin.io/ipfs/
2023-08-03T13:22:03.569Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:262	GET https://proofs.filecoin.io/ipfs/QmVxjFRyhmyQaZEtCh7nk2abc7LhFkzhnRX4rcHqCCpikR
2023-08-03T13:22:03.569Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.vk is ok
2023-08-03T13:22:03.567Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.vk is ok
2023-08-03T13:22:03.572Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.vk is ok
2023-08-03T13:22:03.567Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-8-0-3b7f44a9362e3985369454947bc94022e118211e49fd672d52bec1cbfd599d18.vk is ok
2023-08-03T13:22:03.572Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-559e581f022bb4e4ec6e719e563bf0e026ad6de42e56c18714a2c692b1b88d7e.vk is ok
2023-08-03T13:22:03.572Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-2-sha256_hasher-96f1b4a04c5c51e4759bbf224bbc2ef5a42c7100f16ec0637123f16a845ddfb2.vk is ok
2023-08-03T13:22:03.572Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-6babf46ce344ae495d558e7770a585b2382d54f225af8ed0397b8be7c3fcd472.vk is ok
2023-08-03T13:22:03.572Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-61fa69f38b9cc771ba27b670124714b4ea77fbeae05e377fb859c4a43b73a30c.vk is ok
2023-08-03T13:22:03.572Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-b62098629d07946e9028127e70295ed996fe3ed25b0f9f88eb610a0ab4385a3c.vk is ok
2023-08-03T13:22:03.573Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-7d739b8cf60f1b0709eeebee7730e297683552e4b69cab6984ec0285663c5781.vk is ok
2023-08-03T13:22:03.573Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-50c7368dea9593ed0989e70974d28024efa9d156d585b7eea1be22b2e753f331.vk is ok
2023-08-03T13:22:03.580Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk is ok
2023-08-03T13:22:03.581Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-2627e4006b67f99cef990c0a47d5426cb7ab0a0ad58fc1061547bf2d28b09def.vk is ok
2023-08-03T13:22:03.931Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-fil-inner-product-v1.srs is ok
 10.97 MiB / 10.97 MiB  100.00% 1.16 MiB/s 9s01sm16s
2023-08-03T13:22:13.705Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.params is ok
2023-08-03T13:22:13.706Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:244	Fetching /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.params from https://proofs.filecoin.io/ipfs/
2023-08-03T13:22:13.706Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:262	GET https://proofs.filecoin.io/ipfs/QmWceMgnWYLopMuM4AoGMvGEau7tNe5UK83XFjH5V9B17h
 1.04 GiB / 1.04 GiB  100.00% 1.78 MiB/s 9m56ss22s1ss
2023-08-03T13:32:12.319Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.params is ok
2023-08-03T13:32:12.326Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:244	Fetching /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.params from https://proofs.filecoin.io/ipfs/
2023-08-03T13:32:12.329Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:262	GET https://proofs.filecoin.io/ipfs/QmcSTqDcFVLGGVYz1njhUZ7B6fkKtBumsLUwx4nkh22TzS
 45.11 MiB / 45.11 MiB  100.00% 1.59 MiB/s 28s01s17s
2023-08-03T13:32:42.637Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.params is ok
2023-08-03T13:32:42.637Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:244	Fetching /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.params from https://proofs.filecoin.io/ipfs/
2023-08-03T13:32:42.637Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:262	GET https://proofs.filecoin.io/ipfs/QmYM6Hg7mjmvA3ZHTsqkss1fkdyDju5dDmLiBZGJ5pz9y9
 625.41 MiB / 625.41 MiB  100.00% 2.38 MiB/s 4m22ss9s
2023-08-03T13:37:06.972Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.params is ok
2023-08-03T13:37:06.972Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:233	parameter and key-fetching complete
2023-08-03T13:37:06.973Z	INFO	main	lotus-miner/init.go:162	Trying to connect to full node RPC
2023-08-03T13:37:07.013Z	INFO	main	lotus-miner/init.go:174	Checking full node sync status
2023-08-03T13:37:07.013Z	INFO	main	lotus-miner/init.go:182	Checking if repo exists
2023-08-03T13:37:07.013Z	INFO	main	lotus-miner/init.go:198	Checking full node version
2023-08-03T13:37:07.014Z	INFO	main	lotus-miner/init.go:209	Initializing repo
2023-08-03T13:37:07.014Z	INFO	repo	repo/fsrepo.go:266	Initializing repo at '/root/.lotus-miner-local-net'
2023-08-03T13:37:07.027Z	INFO	main	lotus-miner/init.go:224	Setting up storage config with presealed sectors: [~/.genesis-sectors]
2023-08-03T13:37:07.028Z	INFO	main	lotus-miner/init.go:424	Initializing libp2p identity
2023-08-03T13:37:07.093Z	INFO	badger	v2@v2.2007.4/levels.go:183	All 0 tables opened in 0s

2023-08-03T13:37:07.118Z	INFO	badger	v2@v2.2007.4/levels.go:183	All 0 tables opened in 0s

2023-08-03T13:37:07.139Z	ERROR	main	lotus-miner/init.go:269	Failed to initialize lotus-miner: opening path /root/.genesis-sectors:
    github.com/filecoin-project/lotus/storage/paths.(*Local).open
        /opt/filecoin/storage/paths/local.go:235
  - reading storage metadata for /root/.genesis-sectors:
    github.com/filecoin-project/lotus/storage/paths.(*Local).OpenPath
        /opt/filecoin/storage/paths/local.go:155
  - open /root/.genesis-sectors/sectorstore.json: no such file or directory
2023-08-03T13:37:07.139Z	INFO	main	lotus-miner/init.go:274	Cleaning up /root/.lotus-miner-local-net after attempt...
ERROR: Storage-miner init failed

2023-08-03T13:37:07.279Z	INFO	main	lotus-miner/run.go:98	Checking full node sync status
ERROR: repo at '~/.lotus-miner-local-net' is not initialized, run 'lotus-miner init' to set it up

The nodes fetched some params but seems like they too didn't start properly:

prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/uniswap-v3-periphery$ docker logs -f laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-node-1-1
lotus version 1.23.2+2k+git.8347235b5.dirty
Bootstrap peer info not found, unable to configure. Manual peering will be required.
2023-08-03T13:16:55.205Z	INFO	main	lotus/daemon.go:222	lotus repo: /root/.lotus-local-net
2023-08-03T13:16:55.206Z	INFO	repo	repo/fsrepo.go:266	Initializing repo at '/root/.lotus-local-net'
2023-08-03T13:16:55.207Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:244	Fetching /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.vk from https://proofs.filecoin.io/ipfs/
2023-08-03T13:16:55.208Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:262	GET https://proofs.filecoin.io/ipfs/QmaQsTLL3nc5dw6wAvaioJSBfd1jhQrA2o6ucFf7XeV74P
 1.32 KiB / 1.32 KiB  100.00% 3.60 MiB/s 0s
2023-08-03T13:16:55.838Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.vk is ok
2023-08-03T13:16:55.838Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:244	Fetching /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-50c7368dea9593ed0989e70974d28024efa9d156d585b7eea1be22b2e753f331.vk from https://proofs.filecoin.io/ipfs/
2023-08-03T13:16:55.838Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:262	GET https://proofs.filecoin.io/ipfs/QmbmUMa3TbbW3X5kFhExs6WgC4KeWT18YivaVmXDkB6ANG
.
.
2023-08-03T13:22:58.193Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:209	Parameter file /var/tmp/filecoin-proof-parameters/v28-fil-inner-product-v1.srs is ok
2023-08-03T13:22:58.193Z	INFO	paramfetch	go-paramfetch@v0.0.4/paramfetch.go:233	parameter and key-fetching complete
2023-08-03T13:22:58.207Z	INFO	modules	modules/core.go:63	memory limits initialized	{"max_mem_heap": 0, "total_system_mem": 16607203328, "effective_mem_limit": 16607203328}
2023-08-03T13:22:58.208Z	WARN	modules	modules/core.go:123	failed to initialize cgroup-driven watchdog; err: failed to load cgroup for process: cgroups: cgroup mountpoint does not exist
2023-08-03T13:22:58.208Z	WARN	modules	modules/core.go:124	trying a system-driven watchdog
2023-08-03T13:22:58.208Z	INFO	watchdog	go-watchdog@v1.3.0/watermarks.go:16	initialized watermark watchdog policy; watermarks: []; thresholds: [8303601664 9964321996 11625042329 14116122828 14946482995 15361663078 15776843161]
2023-08-03T13:22:58.208Z	INFO	watchdog	go-watchdog@v1.3.0/watchdog.go:472	initialized heap profile capture; threshold: 0.900000; max captures: 10; dir: /root/.lotus-local-net/heapprof
2023-08-03T13:22:58.208Z	INFO	modules	modules/core.go:129	initialized system-driven watchdog
2023-08-03T13:22:58.237Z	INFO	badger	v2@v2.2007.4/levels.go:183	All 0 tables opened in 0s

2023-08-03T13:22:58.258Z	INFO	badger	v2@v2.2007.4/levels.go:183	All 0 tables opened in 0s

2023-08-03T13:22:58.271Z	INFO	p2pnode	lp2p/rcmgr.go:40	libp2p resource manager is enabled
2023-08-03T13:22:58.272Z	INFO	p2pnode	lp2p/rcmgr.go:88	adjusted default resource manager limits
2023-08-03T13:22:58.272Z	DEBUG	rcmgr	resource-manager/limit.go:84	initializing new limiter with config	{"limits": {}}
2023-08-03T13:22:58.290Z	INFO	badgerbs	v2@v2.2007.4/levels.go:183	All 0 tables opened in 0s

2023-08-03T13:22:58.307Z	INFO	badgerbs	v2@v2.2007.4/levels.go:183	All 0 tables opened in 0s

2023-08-03T13:22:58.315Z	WARN	chainstore	store/store.go:222	no previous chain state found
2023-08-03T13:22:58.387Z	INFO	chainstore	store/store.go:644	New heaviest tipset! [bafy2bzacedcky4dj7v7jnqzapa6tp2h2fd7xh2yzh7mb7uy3jsdiea5byf4bk] (height=0)
2023-08-03T13:22:58.389Z	WARN	chainstore	store/store.go:662	no previous heaviest tipset found, using [bafy2bzacedcky4dj7v7jnqzapa6tp2h2fd7xh2yzh7mb7uy3jsdiea5byf4bk]
2023-08-03T13:22:58.393Z	INFO	p2pnode	lp2p/addrs.go:101	Swarm listening at: [/ip4/127.0.0.1/tcp/33209 /ip4/172.22.0.4/tcp/33209]
2023-08-03T13:22:58.598Z	INFO	modules	modules/services.go:163	subscribing to pubsub topic /fil/blocks/localnet-6d52dae5-ff29-4bac-a45d-f84e6c07564c
2023-08-03T13:22:58.598Z	INFO	messagepool	messagepool/messagepool.go:443	mpool ready
2023-08-03T13:22:58.599Z	INFO	modules	modules/services.go:228	relaying messages for pubsub topic /indexer/ingest/localnet-6d52dae5-ff29-4bac-a45d-f84e6c07564c
2023-08-03T13:22:58.599Z	WARN	modules	modules/core.go:147	Generating new API secret
2023-08-03T13:22:58.600Z	INFO	importmgr	imports/manager.go:61	sanity checking imports
2023/08/03 13:22:58 storess: Query
2023/08/03 13:22:58 storess: q.Prefix: 
2023/08/03 13:22:58 storess: q.KeysOnly: true
2023/08/03 13:22:58 storess: q.Filters: 0
2023/08/03 13:22:58 storess: q.Orders: 0
2023/08/03 13:22:58 storess: q.Offset: 0
2023-08-03T13:22:58.600Z	INFO	importmgr	imports/manager.go:98	sanity check completed	{"broken": 0, "total": 0}
2023-08-03T13:22:58.606Z	INFO	splitstore	splitstore/splitstore.go:794	starting splitstore	{"baseEpoch": "0", "warmupEpoch": "0"}
2023-08-03T13:22:58.606Z	INFO	splitstore	splitstore/splitstore_warmup.go:35	warming up hotstore
2023-08-03T13:22:58.606Z	WARN	statemgr	stmgr/forks.go:245	STARTING pre-migration
2023-08-03T13:22:58.606Z	INFO	dt-impl	impl/impl.go:138	start data-transfer module
manifest cid: bafy2bzaceay35go4xbjb45km6o46e5bib3bi46panhovcbedrynzwmm3drr4i
2023-08-03T13:22:58.609Z	INFO	markets	loggers/loggers.go:65	module ready	{"module": "discovery"}
2023-08-03T13:22:58.612Z	INFO	healthcheck	node/health.go:57	waiting 1s before starting ChainNotify channel
2023-08-03T13:22:58.613Z	INFO	markets	loggers/loggers.go:65	module ready	{"module": "client data transfer"}
2023-08-03T13:22:58.613Z	INFO	markets	loggers/loggers.go:65	module ready	{"module": "storage client"}
2023-08-03T13:22:58.613Z	INFO	markets	loggers/loggers.go:65	module ready	{"module": "retrieval client"}
2023-08-03T13:22:58.634Z	INFO	splitstore	splitstore/splitstore_compact.go:1127	chain walk done	{"walked": 1, "scanned": 1, "walk size": 8912}
2023-08-03T13:22:58.634Z	INFO	splitstore	splitstore/splitstore_warmup.go:126	warmup stats	{"visited": 79, "warm": 0, "missing": 0}
2023-08-03T13:22:58.645Z	INFO	splitstore	splitstore/splitstore_warmup.go:44	warm up done	{"took": 0.038772103}
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Started 6 workers
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Creating migration jobs
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Result writer started
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Done creating 17 migration jobs after 134.001µs
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Worker 5 done
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Worker 3 done
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Worker 2 done
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Worker 0 done
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Worker 4 done
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Worker 1 done
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	All workers done after 232.418µs
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	Result writer wrote 17 results to state tree after 239.858µs
2023-08-03T13:22:58.687Z	INFO	fil-consensus	filcns/upgrades.go:1971	All 17 done after 250.85µs (67770/s)
2023-08-03T13:22:58.734Z	WARN	statemgr	stmgr/forks.go:258	COMPLETED pre-migration	{"duration": 0.127130646}
2023-08-03T13:22:59.613Z	INFO	healthcheck	node/health.go:70	started ChainNotify channel
2023-08-03T13:23:03.274Z	INFO	basichost	basic/natmgr.go:93	DiscoverNAT error:no NAT found
2023-08-03T13:23:03.598Z	WARN	peermgr	peermgr/peermgr.go:205	no peers connected, and no bootstrappers configured
2023-08-03T13:23:08.598Z	WARN	peermgr	peermgr/peermgr.go:205	no peers connected, and no bootstrappers configured
2023-08-03T13:23:13.598Z	WARN	peermgr	peermgr/peermgr.go:205	no peers connected, and no bootstrappers configured
2023-08-03T13:23:18.598Z	WARN	peermgr	peermgr/peermgr.go:205	no peers connected, and no bootstrappers configured
2023-08-03T13:23:23.598Z	WARN	peermgr	peermgr/peermgr.go:205	no peers connected, and no bootstrappers configured
.
.

Tried querying the RPC endpoints as well, but don't seem to be working:

prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/uniswap-v3-periphery$ curl -X POST http://localhost:32790/rpc/v1 -H "Content-Type: application/json" -d '{
  "jsonrpc": "2.0",
  "method": "Filecoin.WalletList",
  "params": [],
  "id": 1
}'
curl: (56) Recv failure: Connection reset by peer
After removing the healthcheck, the miner node seems to have fetched the params but has failed to initialize: ```bash prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/uniswap-v3-periphery$ docker logs -f laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-miner-1 lotus version 1.23.2+2k+git.8347235b5.dirty Waiting for daemon to start... Daemon started. ERROR: open /root/.genesis-sectors/pre-seal-t01000.key: no such file or directory 2023-08-03T13:22:03.559Z INFO main lotus-miner/init.go:129 Initializing lotus miner 2023-08-03T13:22:03.562Z INFO main lotus-miner/init.go:156 Checking proof parameters 2023-08-03T13:22:03.566Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-8-2-102e1444a7e9a97ebf1e3d6855dcc77e66c011ea66f936d9b2c508f87f2f83a7.vk is ok 2023-08-03T13:22:03.566Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0cfb4f178bbb71cf2ecfcd42accce558b27199ab4fb59cb78f2483fe21ef36d9.vk is ok 2023-08-03T13:22:03.565Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-5294475db5237a2e83c3e52fd6c2b03859a1831d45ed08c4f35dbf9a803165a9.vk is ok 2023-08-03T13:22:03.567Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-ecd683648512ab1765faa2a5f14bab48f676e633467f0aa8aad4b55dcb0652bb.vk is ok 2023-08-03T13:22:03.567Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-92180959e1918d26350b8e6cfe217bbdd0a2d8de51ebec269078b364b715ad63.vk is ok 2023-08-03T13:22:03.567Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.vk is ok 2023-08-03T13:22:03.566Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.vk is ok 2023-08-03T13:22:03.565Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:244 Fetching /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.params from https://proofs.filecoin.io/ipfs/ 2023-08-03T13:22:03.569Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:262 GET https://proofs.filecoin.io/ipfs/QmVxjFRyhmyQaZEtCh7nk2abc7LhFkzhnRX4rcHqCCpikR 2023-08-03T13:22:03.569Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.vk is ok 2023-08-03T13:22:03.567Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.vk is ok 2023-08-03T13:22:03.572Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.vk is ok 2023-08-03T13:22:03.567Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-8-0-3b7f44a9362e3985369454947bc94022e118211e49fd672d52bec1cbfd599d18.vk is ok 2023-08-03T13:22:03.572Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-559e581f022bb4e4ec6e719e563bf0e026ad6de42e56c18714a2c692b1b88d7e.vk is ok 2023-08-03T13:22:03.572Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-2-sha256_hasher-96f1b4a04c5c51e4759bbf224bbc2ef5a42c7100f16ec0637123f16a845ddfb2.vk is ok 2023-08-03T13:22:03.572Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-6babf46ce344ae495d558e7770a585b2382d54f225af8ed0397b8be7c3fcd472.vk is ok 2023-08-03T13:22:03.572Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-61fa69f38b9cc771ba27b670124714b4ea77fbeae05e377fb859c4a43b73a30c.vk is ok 2023-08-03T13:22:03.572Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-b62098629d07946e9028127e70295ed996fe3ed25b0f9f88eb610a0ab4385a3c.vk is ok 2023-08-03T13:22:03.573Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-7d739b8cf60f1b0709eeebee7730e297683552e4b69cab6984ec0285663c5781.vk is ok 2023-08-03T13:22:03.573Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-50c7368dea9593ed0989e70974d28024efa9d156d585b7eea1be22b2e753f331.vk is ok 2023-08-03T13:22:03.580Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk is ok 2023-08-03T13:22:03.581Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-2627e4006b67f99cef990c0a47d5426cb7ab0a0ad58fc1061547bf2d28b09def.vk is ok 2023-08-03T13:22:03.931Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-fil-inner-product-v1.srs is ok 10.97 MiB / 10.97 MiB 100.00% 1.16 MiB/s 9s01sm16s 2023-08-03T13:22:13.705Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.params is ok 2023-08-03T13:22:13.706Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:244 Fetching /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.params from https://proofs.filecoin.io/ipfs/ 2023-08-03T13:22:13.706Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:262 GET https://proofs.filecoin.io/ipfs/QmWceMgnWYLopMuM4AoGMvGEau7tNe5UK83XFjH5V9B17h 1.04 GiB / 1.04 GiB 100.00% 1.78 MiB/s 9m56ss22s1ss 2023-08-03T13:32:12.319Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.params is ok 2023-08-03T13:32:12.326Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:244 Fetching /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.params from https://proofs.filecoin.io/ipfs/ 2023-08-03T13:32:12.329Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:262 GET https://proofs.filecoin.io/ipfs/QmcSTqDcFVLGGVYz1njhUZ7B6fkKtBumsLUwx4nkh22TzS 45.11 MiB / 45.11 MiB 100.00% 1.59 MiB/s 28s01s17s 2023-08-03T13:32:42.637Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.params is ok 2023-08-03T13:32:42.637Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:244 Fetching /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.params from https://proofs.filecoin.io/ipfs/ 2023-08-03T13:32:42.637Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:262 GET https://proofs.filecoin.io/ipfs/QmYM6Hg7mjmvA3ZHTsqkss1fkdyDju5dDmLiBZGJ5pz9y9 625.41 MiB / 625.41 MiB 100.00% 2.38 MiB/s 4m22ss9s 2023-08-03T13:37:06.972Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.params is ok 2023-08-03T13:37:06.972Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:233 parameter and key-fetching complete 2023-08-03T13:37:06.973Z INFO main lotus-miner/init.go:162 Trying to connect to full node RPC 2023-08-03T13:37:07.013Z INFO main lotus-miner/init.go:174 Checking full node sync status 2023-08-03T13:37:07.013Z INFO main lotus-miner/init.go:182 Checking if repo exists 2023-08-03T13:37:07.013Z INFO main lotus-miner/init.go:198 Checking full node version 2023-08-03T13:37:07.014Z INFO main lotus-miner/init.go:209 Initializing repo 2023-08-03T13:37:07.014Z INFO repo repo/fsrepo.go:266 Initializing repo at '/root/.lotus-miner-local-net' 2023-08-03T13:37:07.027Z INFO main lotus-miner/init.go:224 Setting up storage config with presealed sectors: [~/.genesis-sectors] 2023-08-03T13:37:07.028Z INFO main lotus-miner/init.go:424 Initializing libp2p identity 2023-08-03T13:37:07.093Z INFO badger v2@v2.2007.4/levels.go:183 All 0 tables opened in 0s 2023-08-03T13:37:07.118Z INFO badger v2@v2.2007.4/levels.go:183 All 0 tables opened in 0s 2023-08-03T13:37:07.139Z ERROR main lotus-miner/init.go:269 Failed to initialize lotus-miner: opening path /root/.genesis-sectors: github.com/filecoin-project/lotus/storage/paths.(*Local).open /opt/filecoin/storage/paths/local.go:235 - reading storage metadata for /root/.genesis-sectors: github.com/filecoin-project/lotus/storage/paths.(*Local).OpenPath /opt/filecoin/storage/paths/local.go:155 - open /root/.genesis-sectors/sectorstore.json: no such file or directory 2023-08-03T13:37:07.139Z INFO main lotus-miner/init.go:274 Cleaning up /root/.lotus-miner-local-net after attempt... ERROR: Storage-miner init failed 2023-08-03T13:37:07.279Z INFO main lotus-miner/run.go:98 Checking full node sync status ERROR: repo at '~/.lotus-miner-local-net' is not initialized, run 'lotus-miner init' to set it up ``` The nodes fetched some params but seems like they too didn't start properly: ```bash prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/uniswap-v3-periphery$ docker logs -f laconic-01dfd0e5e8a0e82ba6858771df1e3da8-lotus-node-1-1 lotus version 1.23.2+2k+git.8347235b5.dirty Bootstrap peer info not found, unable to configure. Manual peering will be required. 2023-08-03T13:16:55.205Z INFO main lotus/daemon.go:222 lotus repo: /root/.lotus-local-net 2023-08-03T13:16:55.206Z INFO repo repo/fsrepo.go:266 Initializing repo at '/root/.lotus-local-net' 2023-08-03T13:16:55.207Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:244 Fetching /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.vk from https://proofs.filecoin.io/ipfs/ 2023-08-03T13:16:55.208Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:262 GET https://proofs.filecoin.io/ipfs/QmaQsTLL3nc5dw6wAvaioJSBfd1jhQrA2o6ucFf7XeV74P 1.32 KiB / 1.32 KiB 100.00% 3.60 MiB/s 0s 2023-08-03T13:16:55.838Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.vk is ok 2023-08-03T13:16:55.838Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:244 Fetching /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-50c7368dea9593ed0989e70974d28024efa9d156d585b7eea1be22b2e753f331.vk from https://proofs.filecoin.io/ipfs/ 2023-08-03T13:16:55.838Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:262 GET https://proofs.filecoin.io/ipfs/QmbmUMa3TbbW3X5kFhExs6WgC4KeWT18YivaVmXDkB6ANG . . 2023-08-03T13:22:58.193Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:209 Parameter file /var/tmp/filecoin-proof-parameters/v28-fil-inner-product-v1.srs is ok 2023-08-03T13:22:58.193Z INFO paramfetch go-paramfetch@v0.0.4/paramfetch.go:233 parameter and key-fetching complete 2023-08-03T13:22:58.207Z INFO modules modules/core.go:63 memory limits initialized {"max_mem_heap": 0, "total_system_mem": 16607203328, "effective_mem_limit": 16607203328} 2023-08-03T13:22:58.208Z WARN modules modules/core.go:123 failed to initialize cgroup-driven watchdog; err: failed to load cgroup for process: cgroups: cgroup mountpoint does not exist 2023-08-03T13:22:58.208Z WARN modules modules/core.go:124 trying a system-driven watchdog 2023-08-03T13:22:58.208Z INFO watchdog go-watchdog@v1.3.0/watermarks.go:16 initialized watermark watchdog policy; watermarks: []; thresholds: [8303601664 9964321996 11625042329 14116122828 14946482995 15361663078 15776843161] 2023-08-03T13:22:58.208Z INFO watchdog go-watchdog@v1.3.0/watchdog.go:472 initialized heap profile capture; threshold: 0.900000; max captures: 10; dir: /root/.lotus-local-net/heapprof 2023-08-03T13:22:58.208Z INFO modules modules/core.go:129 initialized system-driven watchdog 2023-08-03T13:22:58.237Z INFO badger v2@v2.2007.4/levels.go:183 All 0 tables opened in 0s 2023-08-03T13:22:58.258Z INFO badger v2@v2.2007.4/levels.go:183 All 0 tables opened in 0s 2023-08-03T13:22:58.271Z INFO p2pnode lp2p/rcmgr.go:40 libp2p resource manager is enabled 2023-08-03T13:22:58.272Z INFO p2pnode lp2p/rcmgr.go:88 adjusted default resource manager limits 2023-08-03T13:22:58.272Z DEBUG rcmgr resource-manager/limit.go:84 initializing new limiter with config {"limits": {}} 2023-08-03T13:22:58.290Z INFO badgerbs v2@v2.2007.4/levels.go:183 All 0 tables opened in 0s 2023-08-03T13:22:58.307Z INFO badgerbs v2@v2.2007.4/levels.go:183 All 0 tables opened in 0s 2023-08-03T13:22:58.315Z WARN chainstore store/store.go:222 no previous chain state found 2023-08-03T13:22:58.387Z INFO chainstore store/store.go:644 New heaviest tipset! [bafy2bzacedcky4dj7v7jnqzapa6tp2h2fd7xh2yzh7mb7uy3jsdiea5byf4bk] (height=0) 2023-08-03T13:22:58.389Z WARN chainstore store/store.go:662 no previous heaviest tipset found, using [bafy2bzacedcky4dj7v7jnqzapa6tp2h2fd7xh2yzh7mb7uy3jsdiea5byf4bk] 2023-08-03T13:22:58.393Z INFO p2pnode lp2p/addrs.go:101 Swarm listening at: [/ip4/127.0.0.1/tcp/33209 /ip4/172.22.0.4/tcp/33209] 2023-08-03T13:22:58.598Z INFO modules modules/services.go:163 subscribing to pubsub topic /fil/blocks/localnet-6d52dae5-ff29-4bac-a45d-f84e6c07564c 2023-08-03T13:22:58.598Z INFO messagepool messagepool/messagepool.go:443 mpool ready 2023-08-03T13:22:58.599Z INFO modules modules/services.go:228 relaying messages for pubsub topic /indexer/ingest/localnet-6d52dae5-ff29-4bac-a45d-f84e6c07564c 2023-08-03T13:22:58.599Z WARN modules modules/core.go:147 Generating new API secret 2023-08-03T13:22:58.600Z INFO importmgr imports/manager.go:61 sanity checking imports 2023/08/03 13:22:58 storess: Query 2023/08/03 13:22:58 storess: q.Prefix: 2023/08/03 13:22:58 storess: q.KeysOnly: true 2023/08/03 13:22:58 storess: q.Filters: 0 2023/08/03 13:22:58 storess: q.Orders: 0 2023/08/03 13:22:58 storess: q.Offset: 0 2023-08-03T13:22:58.600Z INFO importmgr imports/manager.go:98 sanity check completed {"broken": 0, "total": 0} 2023-08-03T13:22:58.606Z INFO splitstore splitstore/splitstore.go:794 starting splitstore {"baseEpoch": "0", "warmupEpoch": "0"} 2023-08-03T13:22:58.606Z INFO splitstore splitstore/splitstore_warmup.go:35 warming up hotstore 2023-08-03T13:22:58.606Z WARN statemgr stmgr/forks.go:245 STARTING pre-migration 2023-08-03T13:22:58.606Z INFO dt-impl impl/impl.go:138 start data-transfer module manifest cid: bafy2bzaceay35go4xbjb45km6o46e5bib3bi46panhovcbedrynzwmm3drr4i 2023-08-03T13:22:58.609Z INFO markets loggers/loggers.go:65 module ready {"module": "discovery"} 2023-08-03T13:22:58.612Z INFO healthcheck node/health.go:57 waiting 1s before starting ChainNotify channel 2023-08-03T13:22:58.613Z INFO markets loggers/loggers.go:65 module ready {"module": "client data transfer"} 2023-08-03T13:22:58.613Z INFO markets loggers/loggers.go:65 module ready {"module": "storage client"} 2023-08-03T13:22:58.613Z INFO markets loggers/loggers.go:65 module ready {"module": "retrieval client"} 2023-08-03T13:22:58.634Z INFO splitstore splitstore/splitstore_compact.go:1127 chain walk done {"walked": 1, "scanned": 1, "walk size": 8912} 2023-08-03T13:22:58.634Z INFO splitstore splitstore/splitstore_warmup.go:126 warmup stats {"visited": 79, "warm": 0, "missing": 0} 2023-08-03T13:22:58.645Z INFO splitstore splitstore/splitstore_warmup.go:44 warm up done {"took": 0.038772103} 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Started 6 workers 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Creating migration jobs 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Result writer started 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Done creating 17 migration jobs after 134.001µs 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Worker 5 done 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Worker 3 done 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Worker 2 done 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Worker 0 done 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Worker 4 done 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Worker 1 done 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 All workers done after 232.418µs 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 Result writer wrote 17 results to state tree after 239.858µs 2023-08-03T13:22:58.687Z INFO fil-consensus filcns/upgrades.go:1971 All 17 done after 250.85µs (67770/s) 2023-08-03T13:22:58.734Z WARN statemgr stmgr/forks.go:258 COMPLETED pre-migration {"duration": 0.127130646} 2023-08-03T13:22:59.613Z INFO healthcheck node/health.go:70 started ChainNotify channel 2023-08-03T13:23:03.274Z INFO basichost basic/natmgr.go:93 DiscoverNAT error:no NAT found 2023-08-03T13:23:03.598Z WARN peermgr peermgr/peermgr.go:205 no peers connected, and no bootstrappers configured 2023-08-03T13:23:08.598Z WARN peermgr peermgr/peermgr.go:205 no peers connected, and no bootstrappers configured 2023-08-03T13:23:13.598Z WARN peermgr peermgr/peermgr.go:205 no peers connected, and no bootstrappers configured 2023-08-03T13:23:18.598Z WARN peermgr peermgr/peermgr.go:205 no peers connected, and no bootstrappers configured 2023-08-03T13:23:23.598Z WARN peermgr peermgr/peermgr.go:205 no peers connected, and no bootstrappers configured . . ``` Tried querying the RPC endpoints as well, but don't seem to be working: ```bash prathamesh@prathamesh-Strix-GL504GM-GL504GM:~/deepstack/uniswap-v3-periphery$ curl -X POST http://localhost:32790/rpc/v1 -H "Content-Type: application/json" -d '{ "jsonrpc": "2.0", "method": "Filecoin.WalletList", "params": [], "id": 1 }' curl: (56) Recv failure: Connection reset by peer ````
iskay commented 2023-08-03 15:08:19 +00:00 (Migrated from github.com)
Copy Link

The health check was being used to wait to bring the other nodes online until after network prep was completed, so commenting it out likely caused other problems... but it does point to the health check causing a premature timeout if fetching parameters and other prep takes longer than expected. Removing the health check and moving the logic to wait for network prep inside the startup scripts is more reliable.

Tested this solution and it's working on my (very slow) laptop
https://github.com/cerc-io/stack-orchestrator/pull/483

The health check was being used to wait to bring the other nodes online until after network prep was completed, so commenting it out likely caused other problems... but it does point to the health check causing a premature timeout if fetching parameters and other prep takes longer than expected. Removing the health check and moving the logic to wait for network prep inside the startup scripts is more reliable. Tested this solution and it's working on my (very slow) laptop https://github.com/cerc-io/stack-orchestrator/pull/483
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
feature

   
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
in progress

   
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

  
Copied from Github

An issue or PR manually copied from GitHub.

  
Kind/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No Label
bug
documentation
duplicate
enhancement
feature
good first issue
help wanted
in progress
invalid
question
wontfix
Copied from Github
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/stack-orchestrator#481
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?