cerc-io/stack-orchestrator
Archived
12
1
Fork 3
Code Issues 117 Pull Requests 25 Actions Packages Projects Releases 493 Wiki Activity

Investigate odd lighthouse error during catch up sync #466

New Issue
Open
opened 2023-07-27 18:50:09 +00:00 by dboreham · 3 comments
dboreham commented 2023-07-27 18:50:09 +00:00
Contributor
Copy Link
Copy Source

We keep seeing this error in the lighthouse logs after a warm restart:

WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("mainnet-eth-geth-1")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec

Evidence suggests it is not as simple as a timeout on a request to the sibling geth (geth seems perfectly happy).

We keep seeing this error in the lighthouse logs after a warm restart: ``` WARN Error whilst processing payload status error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("mainnet-eth-geth-1")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec ``` Evidence suggests it is not as simple as a timeout on a request to the sibling geth (geth seems perfectly happy).
dboreham commented 2023-07-27 18:52:48 +00:00
Author
Contributor
Copy Link
Copy Source

Perhaps relevant that this message seems to only be seen when lighthouse is not synced up to head:

INFO Syncing                                 est_time: --, distance: 58 slots (11 mins), peers: 17, service: slot_notifier
Perhaps relevant that this message seems to only be seen when lighthouse is not synced up to head: ``` INFO Syncing est_time: --, distance: 58 slots (11 mins), peers: 17, service: slot_notifier ```
dboreham commented 2023-07-27 18:56:05 +00:00
Author
Contributor
Copy Link
Copy Source

Actually since the slot backlog count keeps increasing, it may be that sync is totally stalled. Perhaps the stalled sync and the message about whilst etc are related.

Actually since the slot backlog count keeps increasing, it may be that sync is totally stalled. Perhaps the stalled sync and the message about whilst etc are related.
dboreham commented 2023-07-27 19:02:45 +00:00
Author
Contributor
Copy Link
Copy Source

Suspicions supported by the fact that after a stack restart the errors do not reappear.
Theory is that lighthouse put itself in some bad perpetual stuck state.

Suspicions supported by the fact that after a stack restart the errors do not reappear. Theory is that lighthouse put itself in some bad perpetual stuck state.
This repo is archived. You cannot comment on issues.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
K8s
bug
Something isn't working
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 feature
good first issue
Good for newcomers
 help wanted
Extra attention is needed
 in progress
invalid
This doesn't seem right
 question
Further information is requested
 wontfix
This will not be worked on
Copied from Github
An issue or PR manually copied from GitHub.
 Kind/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
AFDudley Alex ashwin cerccicd ericlewis i-norden (Ian Norden) prathamesh (Prathamesh Musale) rachmaninovquar rjk-laconic roysc (Roy Crihfield) srw zramsay
No Assignees
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cerc-io/stack-orchestrator#466