cerc-io/stack-orchestrator
17
1
Fork 3
Code Issues 113 Pull Requests 18 Actions Packages Projects Releases 480 Wiki Activity

Deployer displays only a white screen #782

New Issue
Closed
opened 2024-03-21 17:03:53 +00:00 by zramsay · 4 comments
zramsay commented 2024-03-21 17:03:53 +00:00
Member
Copy Link

https://webapp-deployer.apps.snowballtools.com/

https://webapp-deployer.apps.snowballtools.com/
Screenshot 2024-03-21 at 12.58.30.png
702 KiB
Screenshot 2024-03-21 at 12.58.30.png
zramsay changed title from Deployer displays on a white screen to Deployer displays only a white screen 2024-03-21 17:04:02 +00:00
telackey was assigned by dboreham 2024-03-21 21:01:03 +00:00
telackey was unassigned by dboreham 2024-03-21 21:01:11 +00:00
telackey was assigned by dboreham 2024-03-21 21:01:29 +00:00
telackey was unassigned by dboreham 2024-03-21 21:01:38 +00:00
telackey was assigned by dboreham 2024-03-21 21:01:50 +00:00
telackey was unassigned by dboreham 2024-03-21 21:02:19 +00:00
srw commented 2024-03-25 13:53:25 +00:00
Owner
Copy Link

The deployer ingress url is generating a bad gateway webapp-deployer-api.apps.snowballtools.com

looking at the log for the api pod it looks like it is caught in a loop, cycling through the same list of deployments.

########### UNDEPLOY ############
Found removal record for indicated deployment bafyreihsil2dfxwuj43ck23hesd72rc4ws32xblfndq76eudr2qovxdrzm at bafyreifktqzez7g65evzmcekkn4ynyyu2ycheszsxb3j6eufhnqifhtrfq
Skipping unsatisfied request bafyreibem6zcrk5hgpl5m73tcihowbhf4bczupaj46tpuikxs5nu25pul4 because we have seen it before.
Found 0 unsatisfied request(s) to process.
############ DEPLOY #############
Found request bafyreibq6fq4q3k6hjbyza4y7tsxeenypmwmqsmmreh4o5bnbr55llj3aa to run application crn://snowballtools/applications/dsfsdfdsfdslfds@0.1.24 on VPhung24-dsfsdfdsfdslfds.
Found request bafyreieq7vwnl4uh2utq53hddyhay5vzybo2lkrvvlma7zeoshl4m7futm to run application crn://snowballtools/applications/dsfsdfdsfdslfds@0.1.24 on VPhung24-dsfsdfdsfdslfds-f0zo9w5t.
Found request bafyreigt3ndoqqfskhkuipe7tgexpw7p6q55ynrym3u6euavj4gu7wvrtm to run application crn://snowballtools/applications/plume-copy@0.0.0 on danielz-mnx-plume-copy.
Found request bafyreiawulgciezdvlr52ckasjhct2h2vyhznlsk36gir4ycvthljgd65q to run application crn://snowballtools/applications/plume-copy@0.0.0 on danielz-mnx-plume-copy-5j6owy7h.
Found request bafyreicwmpnmzkd7ya32fzb2vyzpatfgvgnwnglyqkgwt7qpc2xfwiz7t4 to run application crn://snowballtools/applications/cosmjs@undefined on danielz-mnx-cosmjs.
Found request bafyreictui5kn6o7z2p32l77ygnllsdtteji5b5du7vhm2tu6mafqsrlqu to run application crn://snowballtools/applications/cosmjs@undefined on danielz-mnx-cosmjs-fmj381fo.
Found request bafyreihy6tbtoa27ch53eo4tqrdaxlmi3ai5ywdbwaxjqhv6gwj77fvsky to run application crn://snowballtools/applications/plume@0.0.0 on DeanVanNiekerk-plume.
Found request bafyreiaxeantx6uxy5anxmdrq2fl5hppj64iaf26c7wxupjn4ghyivnlua to run application crn://snowballtools/applications/plume@0.0.0 on DeanVanNiekerk-plume-9aozo9pp.
...
The deployer ingress url is generating a bad gateway `webapp-deployer-api.apps.snowballtools.com` looking at the log for the api pod it looks like it is caught in a loop, cycling through the same list of deployments. ``` ########### UNDEPLOY ############ Found removal record for indicated deployment bafyreihsil2dfxwuj43ck23hesd72rc4ws32xblfndq76eudr2qovxdrzm at bafyreifktqzez7g65evzmcekkn4ynyyu2ycheszsxb3j6eufhnqifhtrfq Skipping unsatisfied request bafyreibem6zcrk5hgpl5m73tcihowbhf4bczupaj46tpuikxs5nu25pul4 because we have seen it before. Found 0 unsatisfied request(s) to process. ############ DEPLOY ############# Found request bafyreibq6fq4q3k6hjbyza4y7tsxeenypmwmqsmmreh4o5bnbr55llj3aa to run application crn://snowballtools/applications/dsfsdfdsfdslfds@0.1.24 on VPhung24-dsfsdfdsfdslfds. Found request bafyreieq7vwnl4uh2utq53hddyhay5vzybo2lkrvvlma7zeoshl4m7futm to run application crn://snowballtools/applications/dsfsdfdsfdslfds@0.1.24 on VPhung24-dsfsdfdsfdslfds-f0zo9w5t. Found request bafyreigt3ndoqqfskhkuipe7tgexpw7p6q55ynrym3u6euavj4gu7wvrtm to run application crn://snowballtools/applications/plume-copy@0.0.0 on danielz-mnx-plume-copy. Found request bafyreiawulgciezdvlr52ckasjhct2h2vyhznlsk36gir4ycvthljgd65q to run application crn://snowballtools/applications/plume-copy@0.0.0 on danielz-mnx-plume-copy-5j6owy7h. Found request bafyreicwmpnmzkd7ya32fzb2vyzpatfgvgnwnglyqkgwt7qpc2xfwiz7t4 to run application crn://snowballtools/applications/cosmjs@undefined on danielz-mnx-cosmjs. Found request bafyreictui5kn6o7z2p32l77ygnllsdtteji5b5du7vhm2tu6mafqsrlqu to run application crn://snowballtools/applications/cosmjs@undefined on danielz-mnx-cosmjs-fmj381fo. Found request bafyreihy6tbtoa27ch53eo4tqrdaxlmi3ai5ywdbwaxjqhv6gwj77fvsky to run application crn://snowballtools/applications/plume@0.0.0 on DeanVanNiekerk-plume. Found request bafyreiaxeantx6uxy5anxmdrq2fl5hppj64iaf26c7wxupjn4ghyivnlua to run application crn://snowballtools/applications/plume@0.0.0 on DeanVanNiekerk-plume-9aozo9pp. ... ```
telackey commented 2024-03-25 15:56:11 +00:00
Member
Copy Link

No, the looping is normal (though it would probably be a good idea to tone down the logging a bit).

The issue in this case was that the node process for the status API had stopped. I restarted it and it was fine.

There are two things at least that should follow... some additional resilience for the process (eg, auto-restart), and also we need to add node selection to SO. One concern I had was that the deployer got moved between nodes and lost its state. That didnt' actually happen, but it could.

No, the looping is normal (though it would probably be a good idea to tone down the logging a bit). The issue in this case was that the node process for the status API had stopped. I restarted it and it was fine. There are two things at least that should follow... some additional resilience for the process (eg, auto-restart), and also we need to add node selection to SO. One concern I had was that the deployer got moved between nodes and lost its state. That didnt' actually happen, but it could.
telackey commented 2024-03-25 17:43:57 +00:00
Member
Copy Link

I updated the script to auto-restart the node process (just in case).

I updated the script to auto-restart the node process (just in case).
srw commented 2024-03-25 18:56:23 +00:00
Owner
Copy Link

yeah in regards to moving between nodes, we may want to look at longhorn.io which will sync PVs

yeah in regards to moving between nodes, we may want to look at longhorn.io which will sync PVs
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
feature

   
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
in progress

   
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

  
Copied from Github

An issue or PR manually copied from GitHub.

  
Kind/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No Label
bug
documentation
duplicate
enhancement
feature
good first issue
help wanted
in progress
invalid
question
wontfix
Copied from Github
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/stack-orchestrator#782
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?