cerc-io/ipld-eth-server
14
0
Fork 1
Code Issues 10 Pull Requests 2 Actions Packages Projects Releases 37 Wiki Activity

faild with err Post "http://geth:8545": context deadline exceeded #210

New Issue
Closed
opened 2022-12-22 20:05:40 +00:00 by telackey · 5 comments
telackey commented 2022-12-22 20:05:40 +00:00
Contributor
Copy Link

While running a tool that executes a single eth_call, eth_blockNumber, eth_getStorageAt at a time after a few minutes of it looping, I observed repeated errors in the logs of the form:

time="2022-12-22T20:01:02Z" level=error msg="writeStateDiffFor 0xd9ee39c319ebb4a2afde56d292397f0dc524aca89d85e100d534b0565121fdb6 faild with err Post \"http://geth:8545\": context deadline exceeded"

or

time="2022-12-22T18:43:52Z" level=error msg="writeStateDiffFor 0xd9ee39c319ebb4a2afde56d292397f0dc524aca89d85e100d534b0565121fdb6 faild with err Post \"http://geth:8545/\": EOF"

I am assuming these are related to forwarded eth_calls, but that is not confirmed.

While running a tool that executes a single eth_call, eth_blockNumber, eth_getStorageAt at a time after a few minutes of it looping, I observed repeated errors in the logs of the form: ``` time="2022-12-22T20:01:02Z" level=error msg="writeStateDiffFor 0xd9ee39c319ebb4a2afde56d292397f0dc524aca89d85e100d534b0565121fdb6 faild with err Post \"http://geth:8545\": context deadline exceeded" ``` or ``` time="2022-12-22T18:43:52Z" level=error msg="writeStateDiffFor 0xd9ee39c319ebb4a2afde56d292397f0dc524aca89d85e100d534b0565121fdb6 faild with err Post \"http://geth:8545/\": EOF" ``` I am assuming these are related to forwarded eth_calls, but that is not confirmed.
dboreham commented 2022-12-22 20:18:29 +00:00
Copy Link

Does the error message have anything to do with your requests? It says "writeStateDiffFor" which implies some code to do with state diffing, which is not code to do with your requests. Perhaps this is some general load interference issue? Your requests imposed load, and under load some problem in state diffing arises?

Does the error message have anything to do with your requests? It says "writeStateDiffFor" which implies some code to do with state diffing, which is not code to do with your requests. Perhaps this is some general load interference issue? Your requests imposed load, and under load some problem in state diffing arises?
dboreham commented 2022-12-22 20:19:28 +00:00
Copy Link

Also, do we know why ipld-eth-server is doing state diffing? Or is it? Could the error actually be coming from geth?

Also, do we know why ipld-eth-server is doing state diffing? Or is it? Could the error actually be coming from geth?
telackey commented 2022-12-22 20:33:23 +00:00
Author
Contributor
Copy Link

Also, do we know why ipld-eth-server is doing state diffing? Or is it? Could the error actually be coming from geth?

It seems to be triggering/asking for the statediffing to be done by geth, which geth either fails to respond to in a timely manner (causing this error) or responds with an error, such as #209. #209 has additional info. (I strongly suspect they are two manifestations of the same underlying issue).

> Also, do we know why ipld-eth-server is doing state diffing? Or is it? Could the error actually be coming from geth? It seems to be triggering/asking for the statediffing to be done by geth, which geth either fails to respond to in a timely manner (causing this error) or responds with an error, such as #209. #209 has additional info. (I strongly suspect they are two manifestations of the same underlying issue).
i-norden commented 2022-12-30 19:30:59 +00:00
Member
Copy Link

Yeah I think they are related. If it is due to https://github.com/cerc-io/ipld-eth-server/issues/209#issuecomment-1368058574 and the geth node being overloaded with more statediffing requests than it can handle, then as it tries to process more requests concurrently it takes longer for any single one to be finished and they begin to either timeout or hit errors due to the connection pool being overloaded.

Default timeout of 10s is being used https://github.com/cerc-io/go-ethereum/blob/v1.10.25-statediff-v4/rpc/client.go#L43 which is already cutting it close to how long a high-complexity statediff could take to generate in isolation on "decent" hardware.

Yeah I think they are related. If it is due to https://github.com/cerc-io/ipld-eth-server/issues/209#issuecomment-1368058574 and the geth node being overloaded with more statediffing requests than it can handle, then as it tries to process more requests concurrently it takes longer for any single one to be finished and they begin to either timeout or hit errors due to the connection pool being overloaded. Default timeout of 10s is being used https://github.com/cerc-io/go-ethereum/blob/v1.10.25-statediff-v4/rpc/client.go#L43 which is already cutting it close to how long a high-complexity statediff could take to generate in isolation on "decent" hardware.
telackey commented 2023-01-11 23:19:43 +00:00
Author
Contributor
Copy Link

#223 and #221 should prevent this from the ipld-eth-server side.

#223 and #221 should prevent this from the ipld-eth-server side.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something isn't working

  
critical

   
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
Epic

   
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
Integration tests

   
invalid

This doesn't seem right

  
question

Further information is requested

  
v5

   
wontfix

This will not be worked on

  
Copied from Github

An issue or PR manually copied from GitHub.

  
Kind/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No Label
bug
critical
documentation
duplicate
enhancement
Epic
good first issue
help wanted
Integration tests
invalid
question
v5
wontfix
Copied from Github
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/ipld-eth-server#210
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?