This does some dark magic with reflect to enable plugins to offer
subscriptions without requiring them to use the rpc.Subscriptions
import.
Basically, plugin services can have a channel as a return value,
and the shim will pull items off of that chanel and send them to
the notifier. It makes sure that context.Done() will fire when
the user disconnects, and will shut everything down if the channel
closes.
Currently rpc.BlockNumber is marshalled to JSON as a numeric value, which is
wrong because BlockNumber.UnmarshalJSON() wants it to either be hex-encoded
or string "earliest"/"latest"/"pending". As a result, the call chain
rpc.BlockNumberOrHashWithNumber(123) -> json.Marshal() -> json.Unmarshal()
fails with error "cannot unmarshal object into Go value of type string".
The new error type is returned by client operations contains details of
the response error code and response body.
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes a rare issue where the client subscription forwarding loop
would attempt send on the subscription's channel after Unsubscribe has
returned, leading to a panic if the subscription channel was already
closed by the user. Example:
sub, _ := client.Subscribe(..., channel, ...)
sub.Unsubscribe()
close(channel)
The race occurred because Unsubscribe called quitWithServer to tell the
forwarding loop to stop sending on sub.channel, but did not wait for the
loop to actually come down. This is fixed by adding an additional channel
to track the shutdown, on which Unsubscribe now waits.
Fixes#22322
* trie: fix tests to work on 32-bit systems
* les: make test work on 32-bit platform
* cmd/geth: fix windows-issues on tests
* trie: improve balance
* cmd/geth: make account tests less verbose + less mem intense
* rpc: make debug-level log output less verbose
* cmd/geth: lint
* Only compare hostnames in ws.origins
Also using a helper function for ToLower consolidates all preparation steps in one function for more maintainable consistency.
Spaces => tabs
Remove a semicolon
Add space at start of comment
Remove parens around conditional
Handle case wehre parsed hostname is empty
When passing a single word like "localhost" the parsed hostname is an empty string. Handle this and the error-parsing case together as default, and the nonempty hostname case in the conditional.
Refactor with new originIsAllowed functions
Adds originIsAllowed() & ruleAllowsOrigin(); removes prepOriginForComparison
Remove blank line
Added tests for simple allowed-orign rule
which does not specify a protocol or port, just a hostname
Fix copy-paste: `:=` => `=`
Remove parens around conditional
Remove autoadded whitespace on blank lines
Compare scheme, hostname, and port with rule
if the rule specifies those portions.
Remove one autoadded trailing whitespace
Better handle case where only origin host is given
e.g. "localhost"
Remove parens around conditional
Refactor: attemptWebsocketConnectionFromOrigin DRY
Include return type on helper function
Provide srv obj in helper fn
Provide srv to helper fn
Remove stray underscore
Remove blank line
parent 93e666b4c1e7e49b8406dc83ed93f4a02ea49ac1
author wbt <wbt@users.noreply.github.com> 1598559718 -0400
committer Martin Holst Swende <martin@swende.se> 1605602257 +0100
gpgsig -----BEGIN PGP SIGNATURE-----
iQFFBAABCAAvFiEEypmrtbNuJK1doP1AaDtDjAWl3fAFAl+zi9ARHG1hcnRpbkBz
d2VuZGUuc2UACgkQaDtDjAWl3fDRiwgAoMtzU8dwRV7Q9xkCwWEx9Wz2f3n6jUr2
VWBycDKGKwRkPPOER3oc9kzjGU/P1tFlK07PjfnAKZ9KWzxpDcJZwYM3xCBurG7A
16y4YsQnzgPNONv3xIkdi3RZtDBIiPFFEmdZFFvZ/jKexfI6JIYPngCAoqdTIFb9
On/aPvvVWQn1ExfmarsvvJ7kUDUG77tZipuacEH5FfFsfelBWOEYPe+I9ToUHskv
+qO6rOkV1Ojk8eBc6o0R1PnApwCAlEhJs7aM/SEOg4B4ZJJneiFuEXBIG9+0yS2I
NOicuDPLGucOB5nBsfIKI3USPeE+3jxdT8go2lN5Nrhm6MimoILDsQ==
=sgUp
-----END PGP SIGNATURE-----
Refactor: drop err var for more concise test lines
Add several tests for new WebSocket origin checks
Remove autoadded whitespace on blank lines
Restore TestWebsocketOrigins originally-named test
and rename the others to be helpers rather than full tests
Remove autoadded whitespace on blank line
Temporarily comment out new test sets
Uncomment test around origin rule with scheme
Remove tests without scheme on browser origin
per https://github.com/ethereum/go-ethereum/pull/21481/files#r479371498
Uncomment tests with port; remove some blank lines
Handle when browser does not specify scheme/port
Uncomment test for including scheme & port in rule
Add IP tests
* node: more tests + table-driven, ws origin changes
Co-authored-by: Martin Holst Swende <martin@swende.se>
* internal/ethapi: return revert reason for eth_call
* internal/ethapi: moved revert reason logic to doCall
* accounts/abi/bind/backends: added revert reason logic to simulated backend
* internal/ethapi: fixed linting error
* internal/ethapi: check if require reason can be unpacked
* internal/ethapi: better error logic
* internal/ethapi: simplify logic
* internal/ethapi: return vmError()
* internal/ethapi: move handling of revert out of docall
* graphql: removed revert logic until spec change
* rpc: internal/ethapi: added custom error types
* graphql: use returndata instead of return
Return() checks if there is an error. If an error is found, we return nil.
For most use cases it can be beneficial to return the output even if there
was an error. This code should be changed anyway once the spec supports
error reasons in graphql responses
* accounts/abi/bind/backends: added tests for revert reason
* internal/ethapi: add errorCode to revert error
* internal/ethapi: add errorCode of 3 to revertError
* internal/ethapi: unified estimateGasErrors, simplified logic
* internal/ethapi: unified handling of errors in DoEstimateGas
* rpc: print error data field
* accounts/abi/bind/backends: unify simulatedBackend and RPC
* internal/ethapi: added binary data to revertError data
* internal/ethapi: refactored unpacking logic into newRevertError
* accounts/abi/bind/backends: fix EstimateGas
* accounts, console, internal, rpc: minor error interface cleanups
* Revert "accounts, console, internal, rpc: minor error interface cleanups"
This reverts commit 2d3ef53c5304e429a04983210a417c1f4e0dafb7.
* re-apply the good parts of 2d3ef53c53
* rpc: add test for returning server error data from client
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This corrects the call to eth_getBlockByNumber, which previously
returned this error:
can't get latest block: missing value for required argument 1
Co-authored-by: Felix Lange <fjl@twurst.com>
This change makes it possible to run geth with JSON-RPC over HTTP and
WebSocket on the same TCP port. The default port for WebSocket
is still 8546.
geth --rpc --rpcport 8545 --ws --wsport 8545
This also removes a lot of deprecated API surface from package rpc.
The rpc package is now purely about serving JSON-RPC and no longer
provides a way to start an HTTP server.
The leaks were mostly in unit tests, and could all be resolved by
adding suitably-sized channel buffers or by restructuring the test
to not send on a channel after an error has occurred.
There is an unavoidable goroutine leak in Console.Interactive: when
we receive a signal, the line reader cannot be unblocked and will get
stuck. This leak is now documented and I've tried to make it slightly
less bad by adding a one-element buffer to the output channels of
the line-reading loop. Should the reader eventually awake from its
blocked state (i.e. when stdin is closed), at least it won't get stuck
trying to send to the interpreter loop which has quit long ago.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds a couple of metrics for tracking the timing
and frequency of method calls:
- rpc/requests gauge counts all requests
- rpc/success gauge counts requests which return err == nil
- rpc/failure gauge counts requests which return err != nil
- rpc/duration/all timer tracks timing of all requests
- rpc/duration/<method>/<success/failure> tracks per-method timing
This just prevents a false negative ERROR warning when, for some unknown
reason, a user attempts to turn on the module rpc even though it's already going
to be on.
This removes the error added in #20597 in favor of a log message at
error level. Failing to start broke a bunch of people's setups and is
probably not the right thing to do for this check.
This change makes the client attempt to reconnect when a write fails.
We already had reconnect support, but the reconnect would previously
happen on the next call after an error. Being more eager leads to a
smoother experience overall.
This commit intents to replicate the DialHTTPWithClient function which allows
creating a RPC Client using a custom dialer but for websockets.
We introduce a new DialWebsocketWithDialer function which allows the caller
to instantiate a new websocket client using a custom dialer.
* rpc: remove 'exported or builtin' restriction for parameters
There is no technial reason for this restriction because package reflect
can create values of any type. Requiring parameters and return values to
be exported causes a lot of noise in package exports.
* rpc: fix staticcheck warnings
* rpc: improve codec abstraction
rpc.ServerCodec is an opaque interface. There was only one way to get a
codec using existing APIs: rpc.NewJSONCodec. This change exports
newCodec (as NewFuncCodec) and NewJSONCodec (as NewCodec). It also makes
all codec methods non-public to avoid showing internals in godoc.
While here, remove codec options in tests because they are not
supported anymore.
* p2p/simulations: use github.com/gorilla/websocket
This package was the last remaining user of golang.org/x/net/websocket.
Migrating to the new library wasn't straightforward because it is no
longer possible to treat WebSocket connections as a net.Conn.
* vendor: delete golang.org/x/net/websocket
* rpc: fix godoc comments and run gofmt
This change adds support for gzip encoding on HTTP responses.
Gzip encoding is used when the client sets the 'accept-encoding: gzip' header.
Original change by @brianosaurus, with fixes from @SjonHortensius.
* rpc: implement websockets with github.com/gorilla/websocket
This change makes package rpc use the github.com/gorilla/websocket
package for WebSockets instead of golang.org/x/net/websocket. The new
library is more robust and supports all WebSocket features including
continuation frames.
There are new tests for two issues with the previously-used library:
- TestWebsocketClientPing checks handling of Ping frames.
- TestWebsocketLargeCall checks whether the request size limit is
applied correctly.
* rpc: raise HTTP/WebSocket request size limit to 5MB
* rpc: remove default origin for client connections
The client used to put the local hostname into the Origin header because
the server wanted an origin to accept the connection, but that's silly:
Origin is for browsers/websites. The nobody would whitelist a particular
hostname.
Now that the server doesn't need Origin anymore, don't bother setting
one for clients. Users who need an origin can use DialWebsocket to
create a client with arbitrary origin if needed.
* vendor: put golang.org/x/net/websocket back
* rpc: don't set Origin header for empty (default) origin
* rpc: add HTTP status code to handshake error
This makes it easier to debug failing connections.
* ethstats: use github.com/gorilla/websocket
* rpc: fix lint
This PR updates a comment about the maximum client subscription buffer
to reflect changes made previously, and fixes a test that wouldn't fail
when wantError == true but execution did not return an error.
When cancelling the context for a call on a HTTP-based client while the
call is running, the select in requestOp.wait may hit the <-context.Done()
case instead of the <-op.resp case. This doesn't happen often -- our
cancel test hasn't caught this even though it ran thousands of times
on CI since the RPC client was added.
Fixes#19714
New APIs added:
client.RegisterName(namespace, service) // makes service available to server
client.Notify(ctx, method, args...) // sends a notification
ClientFromContext(ctx) // to get a client in handler method
This is essentially a rewrite of the server-side code. JSON-RPC
processing code is now the same on both server and client side. Many
minor issues were fixed in the process and there is a new test suite for
JSON-RPC spec compliance (and non-compliance in some cases).
List of behavior changes:
- Method handlers are now called with a per-request context instead of a
per-connection context. The context is canceled right after the method
returns.
- Subscription error channels are always closed when the connection
ends. There is no need to also wait on the Notifier's Closed channel
to detect whether the subscription has ended.
- Client now omits "params" instead of sending "params": null when there
are no arguments to a call. The previous behavior was not compliant
with the spec. The server still accepts "params": null.
- Floating point numbers are allowed as "id". The spec doesn't allow
them, but we handle request "id" as json.RawMessage and guarantee that
the same number will be sent back.
- Logging is improved significantly. There is now a message at DEBUG
level for each RPC call served.
* Initial work on a graphql API
* Added receipts, and more transaction fields.
* Finish receipts, add logs
* Add transactionCount to block
* Add types and .
* Update Block type to be compatible with ethql
* Rename nonce to transactionCount in Account, to be compatible with ethql
* Update transaction, receipt and log to match ethql
* Add query operator, for a range of blocks
* Added ommerCount to Block
* Add transactionAt and ommerAt to Block
* Added sendRawTransaction mutation
* Add Call and EstimateGas to graphQL API
* Refactored to use hexutil.Bytes instead of HexBytes
* Replace BigNum with hexutil.Big
* Refactor call and estimateGas to use ethapi struct type
* Replace ethgraphql.Address with common.Address
* Replace ethgraphql.Hash with common.Hash
* Converted most quantities to Long instead of Int
* Add support for logs
* Fix bug in runFilter
* Restructured Transaction to work primarily with headers, so uncle data is reported properly
* Add gasPrice API
* Add protocolVersion API
* Add syncing API
* Moved schema into its own source file
* Move some single use args types into anonymous structs
* Add doc-comments
* Fixed backend fetching to use context
* Added (very) basic tests
* Add documentation to the graphql schema
* Fix reversion for formatting of big numbers
* Correct spelling error
* s/BigInt/Long/
* Update common/types.go
* Fixes in response to review
* Fix lint error
* Updated calls on private functions
* Fix typo in graphql.go
* Rollback ethapi breaking changes for graphql support
Co-Authored-By: Arachnid <arachnid@notdot.net>
Notifier tracks whether subscription are 'active'. A subscription
becomes active when the subscription ID has been sent to the client. If
the client sends notifications in the request handler before the
subscription becomes active they are dropped. The tests tried to work
around this problem by always waiting 5s before sending the first
notification.
Fix it by buffering notifications until the subscription becomes active.
This speeds up all subscription tests.
Also fix TestSubscriptionMultipleNamespaces to wait for three messages
per subscription instead of six. The test now finishes just after all
notifications have been received and doesn't hit the 30s timeout anymore.
* signer: remove local path disclosure from extapi
* signer: show more data in cli ui
* rpc: make http server forward UA and Origin via Context
* signer, clef/core: ui changes + display UA and Origin
* signer: cliui - indicate less trust in remote headers, see https://github.com/ethereum/go-ethereum/issues/17637
* signer: prevent possibility swap KV-entries in aes_gcm storage, fixes#17635
* signer: remove ecrecover from external API
* signer,clef: default reject instead of warn + valideate new passwords. fixes#17632 and #17631
* signer: check calldata length even if no ABI signature is present
* signer: fix failing testcase
* clef: remove account import from external api
* signer: allow space in passwords, improve error messsage
* signer/storage: fix typos
* rpc: Make HTTP server timeout values configurable
* rpc: Remove flags for setting HTTP Timeouts, configuring via .toml is sufficient.
* rpc: Replace separate constants with a single default struct.
* rpc: Update HTTP Server Read and Write Timeouts to 30s.
* rpc: Remove redundant NewDefaultHTTPTimeouts function.
* rpc: document HTTPTimeouts.
* rpc: sanitize timeout values for library use
This commit adds all changes needed for the merge of swarm-network-rewrite.
The changes:
- build: increase linter timeout
- contracts/ens: export ensNode
- log: add Output method and enable fractional seconds in format
- metrics: relax test timeout
- p2p: reduced some log levels, updates to simulation packages
- rpc: increased maxClientSubscriptionBuffer to 20000
* cmd,node,rpc: add allowedHosts to prevent dns rebinding attacks
* p2p,node: Fix bug with dumpconfig introduced in r54aeb8e4c0bb9f0e7a6c67258af67df3b266af3d
* rpc: add wildcard support for rpcallowedhosts + go fmt
* cmd/geth, cmd/utils, node, rpc: ignore direct ip(v4/6) addresses in rpc virtual hostnames check
* http, rpc, utils: make vhosts into map, address review concerns
* node: change log messages to use geth standard (not sprintf)
* rpc: fix spelling
* rpc: Support specifying HTTP client in RPC dialing
Adds a minimal interface that captures http.Client and adds a new method
rpc.DialHTTPClient that takes a client using that interface. The existing
rpc.DialHTTP method is then alternatively implemented by using the new
rpc.DialHTTPClient method provided with a standard *http.Client.
* rpc: fix minor doc typos
This commit introduces a network simulation framework which
can be used to run simulated networks of devp2p nodes. The
intention is to use this for testing protocols, performing
benchmarks and visualising emergent network behaviour.
Currently http cors and websocket origins are a comma separated string in the
config object. These are replaced with string arrays that are more expressive in
case of a config file.
* p2p/discover, p2p/discv5: add marshaling methods to Node
* p2p/netutil: make Netlist decodable from TOML
* common/math: encode nil HexOrDecimal256 as 0x0
* cmd/geth: add --config file flag
* cmd/geth: add missing license header
* eth: prettify Config again, fix tests
* eth: use gasprice.Config instead of duplicating its fields
* eth/gasprice: hide nil default from dumpconfig output
* cmd/geth: hide genesis block in dumpconfig output
* node: make tests compile
* console: fix tests
* cmd/geth: make TOML keys look exactly like Go struct fields
* p2p: use discovery by default
This makes the zero Config slightly more useful. It also fixes package
node tests because Node detects reuse of the datadir through the
NodeDatabase.
* cmd/geth: make ethstats URL settable through config file
* cmd/faucet: fix configuration
* cmd/geth: dedup attach tests
* eth: add comment for DefaultConfig
* eth: pass downloader.SyncMode in Config
This removes the FastSync, LightSync flags in favour of a more
general SyncMode flag.
* cmd/utils: remove jitvm flags
* cmd/utils: make mutually exclusive flag error prettier
It now reads:
Fatal: flags --dev, --testnet can't be used at the same time
* p2p: fix typo
* node: add DefaultConfig, use it for geth
* mobile: add missing NoDiscovery option
* cmd/utils: drop MakeNode
This exposed a couple of places that needed to be updated to use
node.DefaultConfig.
* node: fix typo
* eth: make fast sync the default mode
* cmd/utils: remove IPCApiFlag (unused)
* node: remove default IPC path
Set it in the frontends instead.
* cmd/geth: add --syncmode
* cmd/utils: make --ipcdisable and --ipcpath mutually exclusive
* cmd/utils: don't enable WS, HTTP when setting addr
* cmd/utils: fix --identity
There is no need to depend on the old context package now that the
minimum Go version is 1.7. The move to "context" eliminates our weird
vendoring setup. Some vendored code still uses golang.org/x/net/context
and it is now vendored in the normal way.
This change triggered new vet checks around context.WithTimeout which
didn't fire with golang.org/x/net/context.
- returned headers didn't include mixHash
- returned transactions didn't include signature fields
- empty transaction input was returned as "", but should be "0x"
- returned receipts didn't include the bloom filter
- "root" in receipts was missing 0x prefix
I initially made the client block if the 100-element buffer was
exceeded. It turns out that this is inconvenient for simple uses of the
client which subscribe and perform calls on the same goroutine, e.g.
client, _ := rpc.Dial(...)
ch := make(chan int) // note: no buffer
sub, _ := client.EthSubscribe(ch, "something")
for event := range ch {
client.Call(...)
}
This innocent looking code will lock up if the server suddenly decides
to send 2000 notifications. In this case, the client's main loop won't
accept the call because it is trying to deliver a notification to ch.
The issue is kind of hard to explain in the docs and few people will
actually read them. Buffering is the simple option and works with close
to no overhead for subscribers that always listen.
If a batch request contained an invalid method, the server would reply
with a non-batch error response. Fix this by tracking an error for each
batch element.
The server delayed closing of connections for 3s when stopping. This was
supposed to allow for slow handlers, but it didn't really work. When
geth quits, it will just exit immediately after quitting the server.
Removing the timer makes testing easier because all connections will be
closed after Stop returns.
Context keys must have a unique type in order to prevent
any unintented clashes. The code used int(1) as key.
Fix it by implementing the pattern recommended by package context.
rpc: be less restrictive on the request id
rpc: improved documentation
console: upgrade web3.js to version 0.16.0
rpc: cache http connections
rpc: rename wsDomains parameter to wsOrigins
