Commit Graph

903 Commits

Author SHA1 Message Date
Felix Lange
69f5d5ba1f
node, rpc: add configurable HTTP request limit (#28948)
Adds a configurable HTTP request limit, and bumps the engine default
2024-02-07 21:06:38 +01:00
Martin HS
5a45e7a631
rpc: fix ns/µs mismatch in metrics (#28649)
The rpc/duration/all meter was in nanoseconds, the individual meter in microseconds.
This PR changes it so both of them use nanoseconds.
2023-12-08 09:40:50 +01:00
Delweng
d6cea4832a
rpc: improve performance of subscription notification encoding (#28328)
It turns out that encoding json.RawMessage is slow because
package json basically parses the message again to ensure it is valid.
We can avoid the slowdown by encoding the entire RPC notification once,
which yields a 30% speedup.
2023-11-22 11:24:54 +01:00
Marius van der Wijden
146e8d999c
core, trie, rpc: speed up tests (#28461)
* rpc: make subscription test faster

reduces time for TestClientSubscriptionChannelClose
from 25 sec to < 1 sec.

* trie: cache trie nodes for faster sanity check

This reduces the time spent on TestIncompleteSyncHash
from ~25s to ~16s.

* core/forkid: speed up validation test

This takes the validation test from > 5s to sub 1 sec

* core/state: improve snapshot test run
brings the time for TestSnapshotRandom from 13s down to 6s

* accounts/keystore: improve keyfile test

This removes some unnecessary waits and reduces the
runtime of TestUpdatedKeyfileContents from 5 to 3 seconds

* trie: remove resolver
* trie: only check ~5% of all trie nodes
2023-11-21 12:19:28 +01:00
Håvard Anda Estensen
a75a2d6db6
all: replace some cases of strings.SplitN with strings.Cut (#28446) 2023-11-15 14:42:33 +01:00
Adrian Sutton
d8c6ae054c
rpc: use correct stringer-method for serializing BlockNumberOrHash (#28358)
The String() version of BlockNumberOrHash uses decimal for all block numbers, including negative ones used to indicate labels. Switch to using BlockNumber.String() which encodes it correctly for use in the JSON-RPC API.
2023-10-25 15:52:42 +02:00
Brandon Liu
ec1a0502bf
rpc: increase timeout in TestClientWebsocketPing (#28371) 2023-10-19 10:08:36 +02:00
hyunchel
7963c4e808
rpc: fix erroneous error-message in test (#28227) 2023-10-03 07:48:36 -04:00
tylerni7
705a51e566
eth, rpc: add configurable option for wsMessageSizeLimit (#27801)
This change adds a configurable limit to websocket message. 
---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-10-03 03:23:19 -04:00
Martin Holst Swende
16946d218a
rpc: use go-winio for named pipes (#27972)
We're trying a new named pipe library, which should hopefully fix some occasional failures in CI.

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
2023-08-23 04:39:01 +02:00
Felix Lange
8f8ef2bc0c
rpc: attempt to fix ping/pong logic race (#27733)
This should fix #27726. With enough load, it might happen that the SetPongHandler 
callback gets invoked before the call to SetReadDeadline is made in pingLoop. When 
this occurs, the socket will end up with a 30s read deadline even though it got the pong,
which will lead to a timeout.

The fix here is processing the pong on pingLoop, synchronizing with the code that 
sends the ping.
2023-08-11 15:16:45 +02:00
Koichi Shiraishi
f0b5af74a3
rpc: avoid use of cgo by hard-coding maxPathSize (#27447)
Package rpc uses cgo to find the maximum UNIX domain socket path 
length. If exceeded, a warning is printed. This is the only use of cgo in this
package. It seems excessive to depend on cgo just for this warning, so
we now hard-code the usual limit for Linux instead.

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
2023-06-19 08:06:58 +02:00
zhiqiangxu
6f08c2f3f1
rpc: add method to test for subscription support (#25942)
This adds two ways to check for subscription support. First, one can now check
whether the transport method (HTTP/WS/etc.) is capable of subscriptions using
the new Client.SupportsSubscriptions method.

Second, the error returned by Subscribe can now reliably be tested using this
pattern:
    
    sub, err := client.Subscribe(...)
    if errors.Is(err, rpc.ErrNotificationsUnsupported) {
        // no subscription support
    }

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
2023-06-14 14:04:41 +02:00
mmsqe
f3314bb6df
rpc: add limit for batch request items and response size (#26681)
This PR adds server-side limits for JSON-RPC batch requests. Before this change, batches
were limited only by processing time. The server would pick calls from the batch and
answer them until the response timeout occurred, then stop processing the remaining batch
items.

Here, we are adding two additional limits which can be configured:

- the 'item limit': batches can have at most N items
- the 'response size limit': batches can contain at most X response bytes

These limits are optional in package rpc. In Geth, we set a default limit of 1000 items
and 25MB response size.

When a batch goes over the limit, an error response is returned to the client. However,
doing this correctly isn't always possible. In JSON-RPC, only method calls with a valid
`id` can be responded to. Since batches may also contain non-call messages or
notifications, the best effort thing we can do to report an error with the batch itself is
reporting the limit violation as an error for the first method call in the batch. If a batch is
too large, but contains only notifications and responses, the error will be reported with
a null `id`.

The RPC client was also changed so it can deal with errors resulting from too large
batches. An older client connected to the server code in this PR could get stuck
until the request timeout occurred when the batch is too large. **Upgrading to a version
of the RPC client containing this change is strongly recommended to avoid timeout issues.**

For some weird reason, when writing the original client implementation, @fjl worked off of
the assumption that responses could be distributed across batches arbitrarily. So for a
batch request containing requests `[A B C]`, the server could respond with `[A B C]` but
also with `[A B] [C]` or even `[A] [B] [C]` and it wouldn't make a difference to the
client.

So in the implementation of BatchCallContext, the client waited for all requests in the
batch individually. If the server didn't respond to some of the requests in the batch, the
client would eventually just time out (if a context was used).

With the addition of batch limits into the server, we anticipate that people will hit this
kind of error way more often. To handle this properly, the client now waits for a single
response batch and expects it to contain all responses to the requests.

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-06-13 13:38:58 +02:00
Martin Holst Swende
9231770811
rpc: change BlockNumber constant values to match ethclient (#27219)
ethclient accepts certain negative block number values as specifiers for the "pending",
"safe" and "finalized" block. In case of "pending", the value accepted by ethclient (-1)
did not match rpc.PendingBlockNumber (-2).

This wasn't really a problem, but other values accepted by ethclient did match the
definitions in package rpc, and it's weird to have this one special case where they don't.

To fix it, we decided to change the values of the constants rather than changing ethclient.
The constant values are not otherwise significant. This is a breaking API change, but we
believe not a dangerous one.

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
2023-05-23 13:18:38 +02:00
Delweng
6fe0252571
rpc, internal/cmdtest: increase timeout in tests (#27083)
This change gives the cmd-tests have a bit more time to finish before getting forcibly torn down.
2023-05-22 08:13:03 -04:00
Stephen Guo
84c3799e21
rpc: more accurate checking of handler method signatures (#27287)
This changes the RPC server to ignore methods using *context.Context as parameter
and *error as return value type. Methods with such types would crash the server when
called.
2023-05-17 12:27:51 +02:00
oseau
7369752999
rpc: websocket should respect the "HTTP_PROXY" by default (#27264)
rpc: the default dialer for websocket should respect the proxy environment variables like "HTTP_PROXY"
2023-05-15 08:48:06 +03:00
s7v7nislands
ffda2c64c4
rpc: use atomic types (#27214)
rpc: use atomic type
2023-05-04 04:54:45 -04:00
Delweng
117530b0e6
metrics/librato: ensure resp.body closed (#26969)
This change ensures that we call Close on a http response body, in various places in the source code (mostly tests)
2023-03-27 07:44:41 -04:00
Jonathan Otto
6bc68f8d94
Increase websocket frame size (from erigon rpc client) (#26883)
This increases the maximum allowed message size to 32MB.

Originally submitted at https://github.com/ledgerwatch/erigon/pull/2739

example block failure: https://etherscan.io/tx/0x1317d973a55cedf9b0f2df6ea48e8077dd176f5444a3423368a46d6e4db89982#internal
2023-03-14 18:41:28 +01:00
Felix Lange
1db978ca6b
rpc: fix unmarshaling of null result in CallContext (#26723)
The change fixes unmarshaling of JSON null results into json.RawMessage.

---------

Co-authored-by: Jason Yuan <jason.yuan@curvegrid.com>
Co-authored-by: Jason Yuan <jason.yuan869@gmail.com>
2023-02-19 14:23:18 -05:00
Felix Lange
31d401ea68
rpc: remove DecimalOrHex type (#26629)
It's the same as math.HexOrDecimal64, which has more uses across the codebase.
2023-02-07 15:44:27 +01:00
Martin Holst Swende
8e92881a3d
rpc: fix off-by-one in ipc endpoint length check (#26614)
This change fixes a minor flaw in the check for ipc endpoint length. The max_path_size is the max path that an ipc endpoint can have, which is 208. However, that size concerns the null-terminated pathname, so we need to account for an extra null-character too.
2023-02-06 07:25:42 -05:00
Shude Li
163e996d0e
all: use http package to replace http method names (#26535) 2023-01-24 11:12:25 +02:00
lightclient
6f858fa806
rpc: fix setting client in DialHTTPWithClient (#26470) 2023-01-12 11:04:34 +01:00
Sina Mahmoodi
f20eba426a
graphql, node, rpc: improve HTTP write timeout handling (#25457)
Here we add special handling for sending an error response when the write timeout of the
HTTP server is just about to expire. This is surprisingly difficult to get right, since is
must be ensured that all output is fully flushed in time, which needs support from
multiple levels of the RPC handler stack:

The timeout response can't use chunked transfer-encoding because there is no way to write
the final terminating chunk. net/http writes it when the topmost handler returns, but the
timeout will already be over by the time that happens. We decided to disable chunked
encoding by setting content-length explicitly.

Gzip compression must also be disabled for timeout responses because we don't know the
true content-length before compressing all output, i.e. compression would reintroduce
chunked transfer-encoding.
2022-12-07 14:02:14 +01:00
Wihan de Beer
8c6e74eed4
rpc: decrease websocket ping interval (#26253)
This is to cater for more node providers.
2022-11-25 16:29:38 +01:00
Martin Holst Swende
97c563e055
rpc, internal/guide: speed up tests a bit (#26193)
This removes an RPC test which takes > 90s to execute, and updates the
internal/guide tests to use lighter scrypt parameters.

Co-authored-by: Felix Lange <fjl@twurst.com>
2022-11-17 15:30:53 +01:00
storyicon
add337e0f7
rpc: support injecting HTTP headers through context (#26023)
This adds a way to specify HTTP headers per request.

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
2022-11-16 15:22:12 +01:00
Felix Lange
ae42148093
rpc: fix connection tracking set in Server (#26180)
rpc: fix connection tracking in Server

When upgrading to mapset/v2 with generics, the set element type used in
rpc.Server had to be changed to *ServerCodec because ServerCodec is not
'comparable'. While the distinction is technically correct, we know all
possible ServerCodec types, and all of them are comparable. So just use
a map instead.
2022-11-15 14:05:16 +01:00
Jolly Zhao
f58ebd9696
all: use github.com/deckarep/golang-set/v2 (generic set) (#26159)
Co-authored-by: Felix Lange <fjl@twurst.com>
2022-11-14 15:16:52 +01:00
Jordan Krage
05037eaffc
rpc: handle wrong HTTP batch response length (#26064) 2022-11-02 15:29:33 +01:00
s7v7nislands
fb4a97f33f
rpc: update docstring (#26072)
Co-authored-by: seven <seven@nodereal.io>
2022-10-31 09:42:25 +01:00
Felix Lange
b628d72766
build: upgrade to go 1.19 (#25726)
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.

In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.

With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2022-09-10 13:25:40 +02:00
Nicholas
610cf02c4a
rpc: improve error codes for internal server errors (#25678)
This changes the error code returned by the RPC server in certain situations:

- handler panic: code -32603
- result marshaling error: code -32603
- attempt to subscribe via HTTP: code -32001

In all of the above cases, the server previously returned the default error
code -32000.

Co-authored-by: Nicholas Zhao <nicholas.zhao@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
2022-09-09 14:03:23 +02:00
Seungbae Yu
38e002f464
rpc: check that "version" is "2.0" in request objects (#25570)
The JSON-RPC spec requires the "version" field to be exactly "2.0",
so we should verify that. This change is not backwards-compatible with
sloppy client implementations, but I decided to go ahead with it anyway
because the failure will be caught via the returned error.
2022-09-02 17:47:29 +02:00
protolambda
90711efb0a
node, rpc: add JWT auth support in client (#24911)
This adds a generic mechanism for 'dial options' in the RPC client,
and also implements a specific dial option for the JWT authentication
mechanism used by the engine API. Some real tests for the server-side
authentication handling are also added.

Co-authored-by: Joshua Gutow <jgutow@optimism.io>
Co-authored-by: Felix Lange <fjl@twurst.com>
2022-09-02 17:40:41 +02:00
Justin Traglia
2c5648d891
all: fix some typos (#25551)
* Fix some typos

* Fix some mistakes

* Revert 4byte.json

* Fix an incorrect fix

* Change files to fails
2022-08-19 09:00:21 +03:00
Tristan-Wilson
9244f87dc1
node, rpc: add ReadHeaderTimeout config option (#25338)
This change makes http.Server.ReadHeaderTimeout configurable separately
from ReadTimeout for RPC servers. The default is set to the same as
ReadTimeout, which in order to cause no change in existing deployments.
2022-08-03 16:50:12 +02:00
Marius van der Wijden
c6dcd018d2
core: eth: rpc: implement safe rpc block (#25165)
* core: eth: rpc: implement safe rpc block

* core: fix setHead, panics
2022-07-25 18:42:05 +03:00
lightclient@protonmail.com
0c6f81f888
all: remove version field from rpc.API 2022-06-27 12:39:46 +02:00
lightclient
119f955686
all: remove public field from rpc.API (#25059)
all: remove public field from rpc.API
2022-06-27 13:33:13 +03:00
Zachinquarantine
f49e298330
README,rpc: remove mention of "shh" RPC API (#25137) 2022-06-21 11:59:02 +02:00
Martin Holst Swende
a907d7e81a
all: more linters (#24783)
This enables the following linters

- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec

WIth a few exceptions.

- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
2022-06-13 16:24:45 +02:00
Péter Szilágyi
106a162b7c
rpc: swap out timer metrics to histograms 2022-06-08 16:24:33 +03:00
Péter Szilágyi
34bda5eae3
core, eth, les, rpc: polish catalyst errors, add context 2022-05-20 12:15:54 +03:00
Marius van der Wijden
e6fa102eb0
core, eth, internal, rpc: implement final block (#24282)
* eth: core: implement finalized block

* eth/catalyst: fix final block

* eth/catalyst: update finalized head gauge

* internal/jsre/deps: updated web3.js to allow for finalized block

* eth/catalyst: make sure only one thread can call fcu

* eth/catalyst: nitpicks

* eth/catalyst: use plain mutex

* eth: nitpicks
2022-05-18 17:30:42 +03:00
Håvard Anda Estensen
4b309c7006
all: replace non-trivial uses of package ioutil with os (#24886)
Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-05-17 14:27:04 +02:00
Håvard Anda Estensen
07508ac0e9
all: replace uses of ioutil with io and os (#24869) 2022-05-16 11:59:35 +02:00