Commit Graph

52 Commits

Author SHA1 Message Date
Felix Lange
b628d72766
build: upgrade to go 1.19 (#25726)
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.

In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.

With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2022-09-10 13:25:40 +02:00
Felix Lange
9b93564e21
rlp/rlpgen: RLP encoder code generator (#24251)
This change adds a code generator tool for creating EncodeRLP method
implementations. The generated methods will behave identically to the
reflect-based encoder, but run faster because there is no reflection overhead.

Package rlp now provides the EncoderBuffer type for incremental encoding. This
is used by generated code, but the new methods can also be useful for
hand-written encoders.

There is also experimental support for generating DecodeRLP, and some new
methods have been added to the existing Stream type to support this. Creating
decoders with rlpgen is not recommended at this time because the generated
methods create very poor error reporting.

More detail about package rlp changes:

* rlp: externalize struct field processing / validation

This adds a new package, rlp/internal/rlpstruct, in preparation for the
RLP encoder generator.

I think the struct field rules are subtle enough to warrant extracting
this into their own package, even though it means that a bunch of
adapter code is needed for converting to/from rlpstruct.Type.

* rlp: add more decoder methods (for rlpgen)

This adds new methods on rlp.Stream:

- Uint64, Uint32, Uint16, Uint8, BigInt
- ReadBytes for decoding into []byte
- MoreDataInList - useful for optional list elements

* rlp: expose encoder buffer (for rlpgen)

This exposes the internal encoder buffer type for use in EncodeRLP
implementations.

The new EncoderBuffer type is a sort-of 'opaque handle' for a pointer to
encBuffer. It is implemented this way to ensure the global encBuffer pool
is handled correctly.
2022-02-16 18:14:12 +01:00
Felix Lange
32c576bd3c
rlp: minor optimizations for slice/array encoding (#23467)
As per benchmark results below, these changes speed up encoding/decoding of
consensus objects a bit.

    name                             old time/op    new time/op    delta
    EncodeRLP/legacy-header-8           384ns ± 1%     331ns ± 3%  -13.83%  (p=0.000 n=7+8)
    EncodeRLP/london-header-8           411ns ± 1%     359ns ± 2%  -12.53%  (p=0.000 n=8+8)
    EncodeRLP/receipt-for-storage-8     251ns ± 0%     239ns ± 0%   -4.97%  (p=0.000 n=8+8)
    EncodeRLP/receipt-full-8            319ns ± 0%     300ns ± 0%   -5.89%  (p=0.000 n=8+7)
    EncodeRLP/legacy-transaction-8      389ns ± 1%     387ns ± 1%     ~     (p=0.099 n=8+8)
    EncodeRLP/access-transaction-8      607ns ± 0%     581ns ± 0%   -4.26%  (p=0.000 n=8+8)
    EncodeRLP/1559-transaction-8        627ns ± 0%     606ns ± 1%   -3.44%  (p=0.000 n=8+8)
    DecodeRLP/legacy-header-8           831ns ± 1%     813ns ± 1%   -2.20%  (p=0.000 n=8+8)
    DecodeRLP/london-header-8           824ns ± 0%     804ns ± 1%   -2.44%  (p=0.000 n=8+7)

* rlp: pass length to byteArrayBytes

This makes it possible to inline byteArrayBytes. For arrays, the length is known
at encoder construction time, so the call to v.Len() can be avoided.

* rlp: avoid IsNil for pointer encoding

It's actually cheaper to use Elem first, because it performs less checks
on the value. If the pointer was nil, the result of Elem is 'invalid'.

* rlp: minor optimizations for slice/array encoding

For empty slices/arrays, we can avoid storing a list header entry in the
encoder buffer. Also avoid doing the tail check at encoding time because
it is already known at encoder construction time.
2021-08-25 19:01:10 +02:00
Felix Lange
4d33de9b49
rlp: optimize big.Int decoding for size <= 32 bytes (#22927)
This change grows the static integer buffer in Stream to 32 bytes,
making it possible to decode 256bit integers without allocating a
temporary buffer.

In the recent commit 088da24, Stream struct size decreased from 120
bytes down to 88 bytes. This commit grows the struct to 112 bytes again,
but the size change will not degrade performance because Stream
instances are internally cached in sync.Pool.

    name             old time/op    new time/op    delta
    DecodeBigInts-8    12.2µs ± 0%     8.6µs ± 4%  -29.58%  (p=0.000 n=9+10)

    name             old speed      new speed      delta
    DecodeBigInts-8   230MB/s ± 0%   326MB/s ± 4%  +42.04%  (p=0.000 n=9+10)
2021-05-25 21:56:25 +02:00
Felix Lange
154ca32a8a
rlp: optimize byte array handling (#22924)
This change improves the performance of encoding/decoding [N]byte.

    name                     old time/op    new time/op    delta
    DecodeByteArrayStruct-8     336ns ± 0%     246ns ± 0%  -26.98%  (p=0.000 n=9+10)
    EncodeByteArrayStruct-8     225ns ± 1%     148ns ± 1%  -34.12%  (p=0.000 n=10+10)

    name                     old alloc/op   new alloc/op   delta
    DecodeByteArrayStruct-8      120B ± 0%       48B ± 0%  -60.00%  (p=0.000 n=10+10)
    EncodeByteArrayStruct-8     0.00B          0.00B          ~     (all equal)
2021-05-22 15:10:16 +02:00
Felix Lange
0d076d92db
rlp: use atomic.Value for type cache (#22902)
All encoding/decoding operations read the type cache to find the
writer/decoder function responsible for a type. When analyzing CPU
profiles of geth during sync, I found that the use of sync.RWMutex in
cache lookups appears in the profiles. It seems we are running into
CPU cache contention problems when package rlp is heavily used
on all CPU cores during sync.

This change makes it use atomic.Value + a writer lock instead of
sync.RWMutex. In the common case where the typeinfo entry is present in
the cache, we simply fetch the map and lookup the type.
2021-05-22 13:34:29 +02:00
Felix Lange
088da24ebf
rlp: improve decoder stream implementation (#22858)
This commit makes various cleanup changes to rlp.Stream.

* rlp: shrink Stream struct

This removes a lot of unused padding space in Stream by reordering the
fields. The size of Stream changes from 120 bytes to 88 bytes. Stream
instances are internally cached and reused using sync.Pool, so this does
not improve performance.

* rlp: simplify list stack

The list stack kept track of the size of the current list context as
well as the current offset into it. The size had to be stored in the
stack in order to subtract it from the remaining bytes of any enclosing
list in ListEnd. It seems that this can be implemented in a simpler
way: just subtract the size from the enclosing list context in List instead.
2021-05-18 12:10:27 +02:00
Felix Lange
700df1442d
rlp: add support for optional struct fields (#22832)
This adds support for a new struct tag "optional". Using this tag, structs used
for RLP encoding/decoding can be extended in a backwards-compatible way,
by adding new fields at the end.
2021-05-07 14:37:13 +02:00
Or Neeman
9ec32a9e7b
rlp: handle case of normal EOF in Stream.readFull (#22336)
io.Reader may return n > 0 and io.EOF at the end of the input stream.
readFull did not handle this correctly, looking only at the error. This fixes
it to check for n == len(buf) as well.
2021-02-18 10:19:49 +01:00
Felix Lange
44b41641f8 rlp: fix staticcheck warnings (#20368)
* rlp: fix staticcheck warnings

* rlp: fix ExampleDecode test
2019-11-25 14:41:53 +02:00
Felix Lange
689486449d build: use golangci-lint (#20295)
* build: use golangci-lint

This changes build/ci.go to download and run golangci-lint instead
of gometalinter.

* core/state: fix unnecessary conversion

* p2p/simulations: fix lock copying (found by go vet)

* signer/core: fix unnecessary conversions

* crypto/ecies: remove unused function cmpPublic

* core/rawdb: remove unused function print

* core/state: remove unused function xTestFuzzCutter

* core/vm: disable TestWriteExpectedValues in a different way

* core/forkid: remove unused function checksum

* les: remove unused type proofsData

* cmd/utils: remove unused functions prefixedNames, prefixFor

* crypto/bn256: run goimports

* p2p/nat: fix goimports lint issue

* cmd/clef: avoid using unkeyed struct fields

* les: cancel context in testRequest

* rlp: delete unreachable code

* core: gofmt

* internal/build: simplify DownloadFile for Go 1.11 compatibility

* build: remove go test --short flag

* .travis.yml: disable build cache

* whisper/whisperv6: fix ineffectual assignment in TestWhisperIdentityManagement

* .golangci.yml: enable goconst and ineffassign linters

* build: print message when there are no lint issues

* internal/build: refactor download a bit
2019-11-18 10:49:17 +02:00
Felix Lange
96fb839133
rlp: improve nil pointer handling (#20064)
* rlp: improve nil pointer handling

In both encoder and decoder, the rules for encoding nil pointers were a
bit hard to understand, and didn't leave much choice. Since RLP allows
two empty values (empty list, empty string), any protocol built on RLP
must choose either of these values to represent the null value in a
certain context.

This change adds choice in the form of two new struct tags, "nilString"
and "nilList". These can be used to specify how a nil pointer value is
encoded. The "nil" tag still exists, but its implementation is now
explicit and defines exactly how nil pointers are handled in a single
place.

Another important change in this commit is how nil pointers and the
Encoder interface interact. The EncodeRLP method was previously called
even on nil values, which was supposed to give users a choice of how
their value would be handled when nil. It turns out this is a stupid
idea. If you create a network protocol containing an object defined in
another package, it's better to be able to say that the object should be
a list or string when nil in the definition of the protocol message
rather than defining the encoding of nil on the object itself.

As of this commit, the encoding rules for pointers now take precedence
over the Encoder interface rule. I think the "nil" tag will work fine
for most cases. For special kinds of objects which are a struct in Go
but strings in RLP, code using the object can specify the desired
encoding of nil using the "nilString" and "nilList" tags.

* rlp: propagate struct field type errors

If a struct contained fields of undecodable type, the encoder and
decoder would panic instead of returning an error. Fix this by
propagating type errors in makeStruct{Writer,Decoder} and add a test.
2019-09-13 11:10:57 +02:00
Felix Lange
8deec2e45a
rlp: fixes for two corner cases and documentation (#19527)
These changes fix two corner cases related to internal handling of types
in package rlp: The "tail" struct tag can only be applied to the last field.
The check for this was wrong and didn't allow for private fields after the
field with the tag. Unsupported types (e.g. structs containing int) which
implement either the Encoder or Decoder interface but not both 
couldn't be encoded/decoded.

Also fixes #19367
2019-05-14 15:09:56 +02:00
Marius van der Wijden
7d881e45bd rlp: added pooling of streams using sync (#19044)
Prevents reallocation, improves performance
2019-02-25 12:34:08 +02:00
kiel barry
a42be3b78d rlp: fix some golint warnings (#16659) 2018-05-08 11:48:07 +02:00
Péter Szilágyi
63246e2542
rlp: fix decoding long strings into RawValue types 2017-08-24 13:10:50 +03:00
S. Matthew English
061889d4ea rlp, trie, contracts, compression, consensus: improve comments (#14580) 2017-06-12 14:45:17 +02:00
Felix Lange
eee96a5bb7 rlp: add support for "-" struct tag 2017-03-07 12:45:12 +01:00
Felix Lange
bb07ce3eed rlp: add "tail" struct tag 2016-02-19 11:14:29 +01:00
Felix Lange
24bb68e7cf rlp: add RawValue 2015-09-10 19:41:51 +02:00
Péter Szilágyi
1d2420323c rlp: add support for boolean encoding/decoding 2015-08-13 12:05:39 +03:00
Felix Lange
bfbcfbe4a9 all: fix license headers one more time
I forgot to update one instance of "go-ethereum" in commit 3f047be5a.
2015-07-23 18:35:11 +02:00
Felix Lange
3f047be5aa all: update license headers to distiguish GPL/LGPL
All code outside of cmd/ is licensed as LGPL. The headers
now reflect this by calling the whole work "the go-ethereum library".
2015-07-22 18:51:45 +02:00
Felix Lange
345590f27f rlp: fix check for canonical byte array size
Decoding did not reject byte arrays of length one with a single element
b where 55 < b < 128. Such byte arrays must be rejected because
they must be encoded as the single byte b instead.
2015-07-18 16:23:35 +02:00
Felix Lange
cefd948267 rlp: reject trailing data when using DecodeBytes 2015-07-18 01:47:17 +02:00
Felix Lange
ea54283b30 all: update license information 2015-07-07 14:12:44 +02:00
Felix Lange
dd49c8e43d rlp: fix list bounds check overflow (found by go-fuzz)
The list size checking overflowed if the size information
for a value was bigger than the list. This is resolved by
always performing the check before reading.
2015-04-28 10:28:15 +02:00
Felix Lange
7180699d40 rlp: require declared number of input elements for array types 2015-04-17 14:45:10 +02:00
Felix Lange
4d5a518a0b rlp: stop accepting lists for byte slices and byte arrays 2015-04-17 14:45:10 +02:00
Felix Lange
cad64fb911 rlp: stricter rules for structs and pointers
The rules have changed as follows:

* When decoding into pointers, empty values no longer produce
  a nil pointer. This can be overriden for struct fields using the
  struct tag "nil".
* When decoding into structs, the input list must contain an element
  for each field.
2015-04-17 14:45:09 +02:00
Felix Lange
1e2c93aa2d rlp: reject non-minimal input strings
Input strings of length 1 containing a byte < 56 are non-minimal and
should be encoded as a single byte instead. Reject such strings.
2015-04-17 14:45:09 +02:00
Felix Lange
6e9f8035a1 rlp: stricter validation of canonical integer format
All integers (including size information in type tags) need to be
encoded using the smallest possible encoding. This commit expands the
stricter validation introduced for *big.Int in commit 59597d23a5
to all integer types and size tags.
2015-04-17 14:45:09 +02:00
Felix Lange
6788f955c2 rlp: fix handling of single byte zero when decoding into a pointer
A single zero byte carries information and should not set the pointer
to nil. This is arguably a corner case. While here, fix the comment
to explain pointer reuse.
2015-04-17 14:45:09 +02:00
Felix Lange
2750ec47b7 rlp: fix integer overflow in list element size validation
It is not safe to add anything to s.size.
2015-04-17 14:45:09 +02:00
Felix Lange
c35f4fd0bd rlp: check top-level value sizes against input limit
This is a preliminary fix for #420 (SEC-18 RLP decoder unsafe
allocation). If a sane input limit is set on the rlp.Stream,
it should no longer be possible to cause huge []byte allocations.
2015-04-17 14:42:41 +02:00
obscuren
59597d23a5 Reject integers w/ appended zero's 2015-04-04 21:29:23 +02:00
Felix Lange
a829a56587 rlp: add Stream.Raw 2015-03-21 00:49:31 +01:00
Felix Lange
b41185a68f rlp: fix nil pointer decoding
The generic pointer decoder did not advance the input position
for empty values. This can lead to strange issues and even
infinite loops.
2015-03-20 22:33:40 +01:00
Felix Lange
b94a6a0193 rlp: add DecodeBytes
Über-convenience.
2015-03-18 13:17:39 +01:00
Felix Lange
fc92abec2c rlp: allow encoding non-empty interface values
This needs to be supported because []someInterface does occur sometimes.

Funny enough, the fix involves changes to the decoder. makeDecoder
cannot return an error for non-empty interfaces anymore because the type
cache builds both decoder and writer. Do the check at 'runtime' instead.
2015-01-15 23:35:26 +01:00
Felix Lange
552f5b2693 rlp: add functions for encoding
I'm reasonably confident that the encoding matches the output of
ethutil.Encode for values that it supports. Some of the tests have been
adpated from the Ethereum testing repository.

There are still TODOs in the code.
2015-01-15 11:00:19 +01:00
obscuren
6abf8ef78f Merge 2015-01-05 17:10:42 +01:00
Felix Lange
6cf4e0329c rlp: display decoder target type in more error messages
Decode error messages now say "expected input list for foo.MyStruct"
instead of just "expected List".
2014-12-09 22:28:49 +01:00
Felix Lange
829730dec4 rlp: remove dead code 2014-12-09 12:49:20 +01:00
Felix Lange
c084a7daa5 rlp: fix panic in decodeList on go 1.4+
The documentation for reflect.Value.Index states that it will
panic for out-of-bounds indices. Since go 1.4, it actually panics.
2014-12-09 10:58:46 +01:00
Felix Lange
93e858f88e rlp: remove support for signed integer types
There is no agreement on how to encode negative integers
across implementations. cpp-ethereum doesn't support them either.
2014-12-09 10:58:46 +01:00
Felix Lange
4f12f0697e rlp: move decoder type switch to decode.go 2014-12-09 10:52:05 +01:00
Felix Lange
f816fdcb69 rlp: include target type in decoder error messages 2014-11-25 16:00:48 +01:00
Felix Lange
205af02a1f rlp: add NewListStream (for p2p) 2014-11-24 19:03:17 +01:00
Felix Lange
5a5560f105 rlp: add Stream.Reset and accept any reader (for p2p) 2014-11-24 19:03:11 +01:00