cerc-io/stack-orchestrator
17
1
Fork 2
Code Issues 112 Pull Requests 15 Actions Packages Projects Releases 473 Wiki Activity

fixturenet payments demo deployment error on ARM #565

New Issue
Open
opened 2023-10-09 12:31:58 +00:00 by zramsay · 3 comments
zramsay commented 2023-10-09 12:31:58 +00:00
Member
Copy Link

With #564, all the images build on both: Mac M2 and this Ubuntu server running on UTM on the same Mac. However, upon deploying, in both cases:

The watcher server container never comes up; this command has no output

docker logs -f $(docker ps -aq --filter name="mobymask-watcher-server")

After a few minutes, the fixturenet-eth-geth-1-1 status switches from Created/Starting to Error. Its logs look like:

docker logs fixturenet-eth-geth-1-1

0x5929AD4A1d6B899065AcF2a66d5EB086a2863bEe
INFO [10-06|23:11:47.852] Maximum peer count                       ETH=50 LES=0 total=50
INFO [10-06|23:11:47.852] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-06|23:11:47.852] Set global gas cap                       cap=50,000,000
Address: {5929ad4a1d6b899065acf2a66d5eb086a2863bee}
Statediff DB at ipld-eth-db
Waiting for statediff DB...

and every other service is stuck in retrying finding what it depends on

With #564, all the images build on both: Mac M2 and [this Ubuntu server](https://ubuntu.com/download/server/arm) running on UTM on the same Mac. However, upon deploying, in both cases: The watcher server container never comes up; this command has no output ``` docker logs -f $(docker ps -aq --filter name="mobymask-watcher-server") ``` After a few minutes, the `fixturenet-eth-geth-1-1` status switches from Created/Starting to Error. Its logs look like: ``` docker logs fixturenet-eth-geth-1-1 ``` ``` 0x5929AD4A1d6B899065AcF2a66d5EB086a2863bEe INFO [10-06|23:11:47.852] Maximum peer count ETH=50 LES=0 total=50 INFO [10-06|23:11:47.852] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" INFO [10-06|23:11:47.852] Set global gas cap cap=50,000,000 Address: {5929ad4a1d6b899065acf2a66d5eb086a2863bee} Statediff DB at ipld-eth-db Waiting for statediff DB... ``` and every other service is stuck in retrying finding what it depends on
dboreham commented 2023-10-16 12:50:03 +00:00
Owner
Copy Link

Looks like the issue is that it is not seeing the statediff DB.

Looks like the issue is that it is not seeing the statediff DB.
iskay commented 2023-10-18 01:07:43 +00:00 (Migrated from github.com)
Copy Link

payments-migrations-1 container is the root cause -- The ipld-eth-db Dockerfile fetches the x86 version of Goose from the releases page, so no error is thrown when building on ARM but at runtime the container can't start the Goose binary, and the statediff DB never comes online.

We can change the dockerfile from

ADD https://github.com/pressly/goose/releases/download/${GOOSE_VER}/goose_linux_x86_64 ./goose

to something like

RUN arch=$(arch | sed s/aarch64/arm64/) && \
  wget -O ./goose https://github.com/pressly/goose/releases/download/${GOOSE_VER}/goose_linux_${arch}

Tested this change and the stack started normally on ARM

`payments-migrations-1` container is the root cause -- The ipld-eth-db Dockerfile fetches the x86 version of Goose from the releases page, so no error is thrown when building on ARM but at runtime the container can't start the Goose binary, and the statediff DB never comes online. We can change the dockerfile from ``` ADD https://github.com/pressly/goose/releases/download/${GOOSE_VER}/goose_linux_x86_64 ./goose ``` to something like ``` RUN arch=$(arch | sed s/aarch64/arm64/) && \ wget -O ./goose https://github.com/pressly/goose/releases/download/${GOOSE_VER}/goose_linux_${arch} ``` Tested this change and the stack started normally on ARM
🎉 1
zramsay commented 2023-10-19 13:22:03 +00:00
Author
Member
Copy Link

using the branch with that fix (and the latest fixturenet-payment stack), it hangs seemingly indefinitely on deploy, and 10 minutes into it, no docker containers have been created.

[+] Building 0.0s (0/0)                                                                                                                    
[+] Running 5/20
[+] Running 5/20nts_erc20_deployment"                       Created                                                                   0.0s 
[+] Running 5/20nts_erc20_deployment"                       Created                                                                   0.0s 
 ✔ Volume "payments_erc20_deployment"                       Created                                                                   0.0s 
 ✔ Volume "payments_ponder_watcher_nitro_data"              Created                                                                   0.0s 
 ✔ Volume "payments_eth_server_nitro_data"                  Created                                                                   0.0s 
 ✔ Volume "payments_ponder_indexer_2_nitro_data"            Created                                                                   0.0s 
 ✔ Volume "payments_ponder_indexer_1_nitro_data"            Created                                                                   0.0s 
 ⠼ Container payments-nitro-contracts-1                     Creating                                                                557.4s 
 ⠼ Container payments-fixturenet-eth-bootnode-lighthouse-1  Creating                                                                557.4s 
 ⠼ Container payments-mobymask-1                            Creating                                                                557.4s 
 ⠼ Container payments-fixturenet-eth-bootnode-geth-1        Creating                                                                557.4s 
 ⠼ Container payments-ponder-app-indexer-1-1                Creating                                                                557.4s 
 ⠼ Container payments-peer-ids-gen-1                        Creating                                                                557.4s 
 ⠼ Container payments-ponder-er20-contracts-1               Creating                                                                557.4s 
 ⠼ Container payments-ponder-app-indexer-2-1                Creating                                                                557.4s 
 ⠼ Container payments-mobymask-v3-app-1                     Creating                                                                557.4s 
 ⠼ Container payments-mobymask-snap-1                       Creating                                                                557.4s 
 ⠼ Container payments-nitro-rpc-client-1                    Creating                                                                557.4s 
 ⠼ Container payments-migrations-1                          Creating                                                                557.4s 
 ⠼ Container payments-ipld-eth-server-1-1                   Creating                                                                557.4s 
 ⠼ Container payments-ipld-eth-server-2-1                   Creating                                                                557.4s 
 ⠼ Container payments-mobymask-watcher-db-1                 Recreate                                                                557.4s

I'll try next on UTM

using the branch with that fix (and the latest fixturenet-payment stack), it hangs seemingly indefinitely on deploy, and 10 minutes into it, no docker containers have been created. ``` [+] Building 0.0s (0/0) [+] Running 5/20 [+] Running 5/20nts_erc20_deployment" Created 0.0s [+] Running 5/20nts_erc20_deployment" Created 0.0s ✔ Volume "payments_erc20_deployment" Created 0.0s ✔ Volume "payments_ponder_watcher_nitro_data" Created 0.0s ✔ Volume "payments_eth_server_nitro_data" Created 0.0s ✔ Volume "payments_ponder_indexer_2_nitro_data" Created 0.0s ✔ Volume "payments_ponder_indexer_1_nitro_data" Created 0.0s ⠼ Container payments-nitro-contracts-1 Creating 557.4s ⠼ Container payments-fixturenet-eth-bootnode-lighthouse-1 Creating 557.4s ⠼ Container payments-mobymask-1 Creating 557.4s ⠼ Container payments-fixturenet-eth-bootnode-geth-1 Creating 557.4s ⠼ Container payments-ponder-app-indexer-1-1 Creating 557.4s ⠼ Container payments-peer-ids-gen-1 Creating 557.4s ⠼ Container payments-ponder-er20-contracts-1 Creating 557.4s ⠼ Container payments-ponder-app-indexer-2-1 Creating 557.4s ⠼ Container payments-mobymask-v3-app-1 Creating 557.4s ⠼ Container payments-mobymask-snap-1 Creating 557.4s ⠼ Container payments-nitro-rpc-client-1 Creating 557.4s ⠼ Container payments-migrations-1 Creating 557.4s ⠼ Container payments-ipld-eth-server-1-1 Creating 557.4s ⠼ Container payments-ipld-eth-server-2-1 Creating 557.4s ⠼ Container payments-mobymask-watcher-db-1 Recreate 557.4s ``` I'll try next on UTM
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
feature

   
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
in progress

   
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

  
Copied from Github

An issue or PR manually copied from GitHub.

  
Kind/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No Label
bug
documentation
duplicate
enhancement
feature
good first issue
help wanted
in progress
invalid
question
wontfix
Copied from Github
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/stack-orchestrator#565
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?