Commit Graph

75 Commits

Author SHA1 Message Date
Felix Lange
7194c847b6
p2p/rlpx: reduce allocation and syscalls (#22899)
This change significantly improves the performance of RLPx message reads
and writes. In the previous implementation, reading and writing of
message frames performed multiple reads and writes on the underlying
network connection, and allocated a new []byte buffer for every read.

In the new implementation, reads and writes re-use buffers, and perform
much fewer system calls on the underlying connection. This doubles the
theoretically achievable throughput on a single connection, as shown by
the benchmark result:

    name             old speed      new speed       delta
    Throughput-8     70.3MB/s ± 0%  155.4MB/s ± 0%  +121.11%  (p=0.000 n=9+8)

The change also removes support for the legacy, pre-EIP-8 handshake encoding.
As of May 2021, no actively maintained client sends this format.
2021-05-27 10:19:13 +02:00
Felix Lange
4d33de9b49
rlp: optimize big.Int decoding for size <= 32 bytes (#22927)
This change grows the static integer buffer in Stream to 32 bytes,
making it possible to decode 256bit integers without allocating a
temporary buffer.

In the recent commit 088da24, Stream struct size decreased from 120
bytes down to 88 bytes. This commit grows the struct to 112 bytes again,
but the size change will not degrade performance because Stream
instances are internally cached in sync.Pool.

    name             old time/op    new time/op    delta
    DecodeBigInts-8    12.2µs ± 0%     8.6µs ± 4%  -29.58%  (p=0.000 n=9+10)

    name             old speed      new speed      delta
    DecodeBigInts-8   230MB/s ± 0%   326MB/s ± 4%  +42.04%  (p=0.000 n=9+10)
2021-05-25 21:56:25 +02:00
ucwong
017cf71fbd
rlp, tests/fuzzers/bls12381: gofmt (#22937) 2021-05-25 10:14:39 +02:00
Felix Lange
154ca32a8a
rlp: optimize byte array handling (#22924)
This change improves the performance of encoding/decoding [N]byte.

    name                     old time/op    new time/op    delta
    DecodeByteArrayStruct-8     336ns ± 0%     246ns ± 0%  -26.98%  (p=0.000 n=9+10)
    EncodeByteArrayStruct-8     225ns ± 1%     148ns ± 1%  -34.12%  (p=0.000 n=10+10)

    name                     old alloc/op   new alloc/op   delta
    DecodeByteArrayStruct-8      120B ± 0%       48B ± 0%  -60.00%  (p=0.000 n=10+10)
    EncodeByteArrayStruct-8     0.00B          0.00B          ~     (all equal)
2021-05-22 15:10:16 +02:00
Felix Lange
0d076d92db
rlp: use atomic.Value for type cache (#22902)
All encoding/decoding operations read the type cache to find the
writer/decoder function responsible for a type. When analyzing CPU
profiles of geth during sync, I found that the use of sync.RWMutex in
cache lookups appears in the profiles. It seems we are running into
CPU cache contention problems when package rlp is heavily used
on all CPU cores during sync.

This change makes it use atomic.Value + a writer lock instead of
sync.RWMutex. In the common case where the typeinfo entry is present in
the cache, we simply fetch the map and lookup the type.
2021-05-22 13:34:29 +02:00
Felix Lange
088da24ebf
rlp: improve decoder stream implementation (#22858)
This commit makes various cleanup changes to rlp.Stream.

* rlp: shrink Stream struct

This removes a lot of unused padding space in Stream by reordering the
fields. The size of Stream changes from 120 bytes to 88 bytes. Stream
instances are internally cached and reused using sync.Pool, so this does
not improve performance.

* rlp: simplify list stack

The list stack kept track of the size of the current list context as
well as the current offset into it. The size had to be stored in the
stack in order to subtract it from the remaining bytes of any enclosing
list in ListEnd. It seems that this can be implemented in a simpler
way: just subtract the size from the enclosing list context in List instead.
2021-05-18 12:10:27 +02:00
Felix Lange
700df1442d
rlp: add support for optional struct fields (#22832)
This adds support for a new struct tag "optional". Using this tag, structs used
for RLP encoding/decoding can be extended in a backwards-compatible way,
by adding new fields at the end.
2021-05-07 14:37:13 +02:00
Martin Holst Swende
94ab4ea341
core/rawdb: fix transaction indexing/unindexing hashing error (#22457)
* core/rawdb: more verbose error logs + better hashing

* core/rawdb: add failing testcase

* core/rawdb: properly hash transactions while indexing/unindexing

* core/rawdb: exit on error + better log msg
2021-03-16 12:15:14 +02:00
Or Neeman
9ec32a9e7b
rlp: handle case of normal EOF in Stream.readFull (#22336)
io.Reader may return n > 0 and io.EOF at the end of the input stream.
readFull did not handle this correctly, looking only at the error. This fixes
it to check for n == len(buf) as well.
2021-02-18 10:19:49 +01:00
Alex Prut
c92faee66e
all: simplify nested complexity and if blocks ending with a return statement (#21854)
Changes:

    Simplify nested complexity
    If an if blocks ends with a return statement then remove the else nesting.

Most of the changes has also been reported in golint https://goreportcard.com/report/github.com/ethereum/go-ethereum#golint
2020-11-25 09:24:50 +01:00
Martin Holst Swende
175506e7fd
core/types, rlp: optimize derivesha (#21728)
This PR contains a minor optimization in derivesha, by exposing the RLP
int-encoding and making use of it to write integers directly to a
buffer (an RLP integer is known to never require more than 9 bytes
total). rlp.AppendUint64 might be useful in other places too.

The code assumes, just as before, that the hasher (a trie) will copy the
key internally, which it does when doing keybytesToHex(key).

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-11-04 19:29:24 +01:00
Felix Lange
f7112cc182
rlp: add SplitUint64 (#21563)
This can be useful when working with raw RLP data.
2020-09-14 19:23:01 +02:00
Felix Lange
6315b6fcc0
rlp: reduce allocations for big.Int and byte array encoding (#21291)
This change further improves the performance of RLP encoding by removing
allocations for big.Int and [...]byte types. I have added a new benchmark
that measures RLP encoding of types.Block to verify that performance is
improved.
2020-07-06 11:17:09 +02:00
Marius van der Wijden
8dfd66f701
rlp: avoid list header allocation in encoder (#21274)
List headers made up 11% of all allocations during sync. This change
removes most of those allocations by keeping the list header values
cached in the encoder buffer instead. Since encoder buffers are pooled,
list headers are no longer allocated in the common case where an
encoder buffer is available for reuse.

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-07-01 13:49:19 +02:00
Martin Holst Swende
4535230059
cmd, core, eth: background transaction indexing (#20302)
* cmd, core, eth: init tx lookup in background

* core/rawdb: tiny log fixes to make it clearer what's happening

* core, eth: fix rebase errors

* core/rawdb: make reindexing less generic, but more optimal

* rlp: implement rlp list iterator

* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data

* core/rawdb, cmd/utils: fix review concerns

* cmd/utils: fix merge issue

* core/rawdb: add some log formatting polishes

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-11 18:58:43 +03:00
Felix Lange
44b41641f8 rlp: fix staticcheck warnings (#20368)
* rlp: fix staticcheck warnings

* rlp: fix ExampleDecode test
2019-11-25 14:41:53 +02:00
Felix Lange
689486449d build: use golangci-lint (#20295)
* build: use golangci-lint

This changes build/ci.go to download and run golangci-lint instead
of gometalinter.

* core/state: fix unnecessary conversion

* p2p/simulations: fix lock copying (found by go vet)

* signer/core: fix unnecessary conversions

* crypto/ecies: remove unused function cmpPublic

* core/rawdb: remove unused function print

* core/state: remove unused function xTestFuzzCutter

* core/vm: disable TestWriteExpectedValues in a different way

* core/forkid: remove unused function checksum

* les: remove unused type proofsData

* cmd/utils: remove unused functions prefixedNames, prefixFor

* crypto/bn256: run goimports

* p2p/nat: fix goimports lint issue

* cmd/clef: avoid using unkeyed struct fields

* les: cancel context in testRequest

* rlp: delete unreachable code

* core: gofmt

* internal/build: simplify DownloadFile for Go 1.11 compatibility

* build: remove go test --short flag

* .travis.yml: disable build cache

* whisper/whisperv6: fix ineffectual assignment in TestWhisperIdentityManagement

* .golangci.yml: enable goconst and ineffassign linters

* build: print message when there are no lint issues

* internal/build: refactor download a bit
2019-11-18 10:49:17 +02:00
Felix Lange
96fb839133
rlp: improve nil pointer handling (#20064)
* rlp: improve nil pointer handling

In both encoder and decoder, the rules for encoding nil pointers were a
bit hard to understand, and didn't leave much choice. Since RLP allows
two empty values (empty list, empty string), any protocol built on RLP
must choose either of these values to represent the null value in a
certain context.

This change adds choice in the form of two new struct tags, "nilString"
and "nilList". These can be used to specify how a nil pointer value is
encoded. The "nil" tag still exists, but its implementation is now
explicit and defines exactly how nil pointers are handled in a single
place.

Another important change in this commit is how nil pointers and the
Encoder interface interact. The EncodeRLP method was previously called
even on nil values, which was supposed to give users a choice of how
their value would be handled when nil. It turns out this is a stupid
idea. If you create a network protocol containing an object defined in
another package, it's better to be able to say that the object should be
a list or string when nil in the definition of the protocol message
rather than defining the encoding of nil on the object itself.

As of this commit, the encoding rules for pointers now take precedence
over the Encoder interface rule. I think the "nil" tag will work fine
for most cases. For special kinds of objects which are a struct in Go
but strings in RLP, code using the object can specify the desired
encoding of nil using the "nilString" and "nilList" tags.

* rlp: propagate struct field type errors

If a struct contained fields of undecodable type, the encoder and
decoder would panic instead of returning an error. Fix this by
propagating type errors in makeStruct{Writer,Decoder} and add a test.
2019-09-13 11:10:57 +02:00
Felix Lange
8deec2e45a
rlp: fixes for two corner cases and documentation (#19527)
These changes fix two corner cases related to internal handling of types
in package rlp: The "tail" struct tag can only be applied to the last field.
The check for this was wrong and didn't allow for private fields after the
field with the tag. Unsupported types (e.g. structs containing int) which
implement either the Encoder or Decoder interface but not both 
couldn't be encoded/decoded.

Also fixes #19367
2019-05-14 15:09:56 +02:00
Marius van der Wijden
7d881e45bd rlp: added pooling of streams using sync (#19044)
Prevents reallocation, improves performance
2019-02-25 12:34:08 +02:00
chenyufeng
2d98099c25 rlp: fix comment typo (#17640) 2018-09-11 18:05:28 +03:00
kiel barry
a42be3b78d rlp: fix some golint warnings (#16659) 2018-05-08 11:48:07 +02:00
Felix Lange
2499b1b139
rlp: fix string size check in readKind (#15625)
Issue found by @guidovranken
2017-12-11 22:47:10 +01:00
Péter Szilágyi
63246e2542
rlp: fix decoding long strings into RawValue types 2017-08-24 13:10:50 +03:00
S. Matthew English
061889d4ea rlp, trie, contracts, compression, consensus: improve comments (#14580) 2017-06-12 14:45:17 +02:00
Felix Lange
eee96a5bb7 rlp: add support for "-" struct tag 2017-03-07 12:45:12 +01:00
Felix Lange
6fdd0893c3 all: fix go vet warnings 2016-04-15 11:17:27 +02:00
Felix Lange
d04a2e7557 all: update license information 2016-04-15 09:48:05 +02:00
Felix Lange
bb07ce3eed rlp: add "tail" struct tag 2016-02-19 11:14:29 +01:00
Felix Lange
fc8b246109 rlp: move ListSize to raw.go 2015-09-10 19:41:51 +02:00
Felix Lange
24bb68e7cf rlp: add RawValue 2015-09-10 19:41:51 +02:00
Felix Lange
bc17dba8fb rlp: add Split functions
These functions allow destructuring of raw rlp-encoded bytes
without the overhead of reflection or copying.
2015-09-10 19:41:51 +02:00
Felix Lange
ac32f52ca6 rlp: fix encReader returning nil buffers to the pool
The bug can cause crashes if Read is called after EOF has been returned.
No code performs such calls right now, but hitting the bug gets more
likely as rlp.EncodeToReader gets used in more places.
2015-09-10 19:12:32 +02:00
Péter Szilágyi
1d2420323c rlp: add support for boolean encoding/decoding 2015-08-13 12:05:39 +03:00
Felix Lange
bfbcfbe4a9 all: fix license headers one more time
I forgot to update one instance of "go-ethereum" in commit 3f047be5a.
2015-07-23 18:35:11 +02:00
Felix Lange
3f047be5aa all: update license headers to distiguish GPL/LGPL
All code outside of cmd/ is licensed as LGPL. The headers
now reflect this by calling the whole work "the go-ethereum library".
2015-07-22 18:51:45 +02:00
Felix Lange
345590f27f rlp: fix check for canonical byte array size
Decoding did not reject byte arrays of length one with a single element
b where 55 < b < 128. Such byte arrays must be rejected because
they must be encoded as the single byte b instead.
2015-07-18 16:23:35 +02:00
Felix Lange
cefd948267 rlp: reject trailing data when using DecodeBytes 2015-07-18 01:47:17 +02:00
Felix Lange
ea54283b30 all: update license information 2015-07-07 14:12:44 +02:00
Felix Lange
8743cc1c1c rlp: add ListSize 2015-06-29 18:51:48 +02:00
Felix Lange
a0566c1058 rlp: remove Flat 2015-06-29 18:51:47 +02:00
Felix Lange
3d0c6a8345 rlp: pool encoder allocations 2015-06-29 18:51:47 +02:00
Felix Lange
dd49c8e43d rlp: fix list bounds check overflow (found by go-fuzz)
The list size checking overflowed if the size information
for a value was bigger than the list. This is resolved by
always performing the check before reading.
2015-04-28 10:28:15 +02:00
Felix Lange
7180699d40 rlp: require declared number of input elements for array types 2015-04-17 14:45:10 +02:00
Felix Lange
4d5a518a0b rlp: stop accepting lists for byte slices and byte arrays 2015-04-17 14:45:10 +02:00
Felix Lange
cad64fb911 rlp: stricter rules for structs and pointers
The rules have changed as follows:

* When decoding into pointers, empty values no longer produce
  a nil pointer. This can be overriden for struct fields using the
  struct tag "nil".
* When decoding into structs, the input list must contain an element
  for each field.
2015-04-17 14:45:09 +02:00
Felix Lange
1e2c93aa2d rlp: reject non-minimal input strings
Input strings of length 1 containing a byte < 56 are non-minimal and
should be encoded as a single byte instead. Reject such strings.
2015-04-17 14:45:09 +02:00
Felix Lange
6e9f8035a1 rlp: stricter validation of canonical integer format
All integers (including size information in type tags) need to be
encoded using the smallest possible encoding. This commit expands the
stricter validation introduced for *big.Int in commit 59597d23a5
to all integer types and size tags.
2015-04-17 14:45:09 +02:00
Felix Lange
6788f955c2 rlp: fix handling of single byte zero when decoding into a pointer
A single zero byte carries information and should not set the pointer
to nil. This is arguably a corner case. While here, fix the comment
to explain pointer reuse.
2015-04-17 14:45:09 +02:00
Felix Lange
2750ec47b7 rlp: fix integer overflow in list element size validation
It is not safe to add anything to s.size.
2015-04-17 14:45:09 +02:00