Optimizes and tests FormatInt by removing inefficient string
concatenation but also making it so much clearer with how one would
express adding thousand separators in natural language. It uses
a combination of strings.Builder whose values can be grown
The performance improvement is stark in every dimension:
```shell
$ benchstat before.txt after3.txt
name old time/op new time/op delta
DecimalValueRendererFormat-8 4.48µs ± 1% 2.11µs ± 2% -52.90% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
DecimalValueRendererFormat-8 3.62kB ± 0% 0.78kB ± 0% -78.59% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
DecimalValueRendererFormat-8 83.0 ± 0% 28.0 ± 0% -66.27% (p=0.000 n=10+10)
```
While here, also simplified zero padding for LegacyNewDecFromStr
simply by using strings.Repeat instead of a convoluted
fmt.Sprintf+strconv.Itoa.
Fixes#14008Fixes#14003
Co-authored-by: Marko <marbar3778@yahoo.com>
Noticed while examining a bunch of profiles, that the for loop inside
(*CompactBitArry).NumTrueBitsBefore unnecessarily consumed a bunch of
time:
```shell
7.55s 9.88s (flat, cum) 93.38% of Total
240ms 250ms 88:func (bA *CompactBitArray) NumTrueBitsBefore(index int) int {
. . 89: onesCount := 0
70ms 340ms 90: max := bA.Count()
70ms 70ms 91: if index > max {
. . 92: index = max
. . 93: }
. . 94: // below we iterate over the bytes then over bits (in low endian) and count bits set to 1
2.54s 2.76s 95: for elem := 0; elem < len(bA.Elems); elem++ {
```
but we can use the native for loop that produces indices while iterating
over slices. Just by simply changing the form results in an improvement
```shell
7.50s 9.95s (flat, cum) 94.94% of Total
240ms 320ms 88:func (bA *CompactBitArray) NumTrueBitsBefore(index int) int {
. . 89: onesCount := 0
170ms 420ms 90: max := bA.Count()
90ms 100ms 91: if index > max {
. . 92: index = max
. . 93: }
. . 94: // below we iterate over the bytes then over bits (in low endian) and count bits set to 1
1.49s 1.62s 95: for elem := range bA.Elems {
```
and an improvement in CPU time
```shell
$ benchstat before.txt after.txt
name old time/op new time/op delta
NumTrueBitsBefore/new-8 13.3ns ± 1% 12.5ns ± 1% -6.07% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
NumTrueBitsBefore/new-8 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
NumTrueBitsBefore/new-8 0.00 0.00 ~ (all equal)
```
Fixes#13999
Simplifies and makes clearer the code in removeZeroCoins by
removing unnecessary checks that boiled down to still running
in the same final for loop.
Fixes#13958
Enumerates and improves on the error checking for non-existance errors
to clearly return unhandled errors such as access restrictions,
unaccessible volumes etc, when trying to download binaries.
The prior code assumed that either the file existed or the only error
to be returned would be about the file not existing, yet this isn't
always true.
Fixes#13469
Brings in a static analyzer that catches leaking resources like
unclosed file handles, networking handles etc that can't easily
be caught except in very critical code reviews as well as a bunch
of other static analyzers that Orijtech Inc produces.
Adds more rigorous checks to ensure that the output
is as expected with range failures. While here also
added control tests to ensure that we can parse integers
within range of int.
Updates #13346
Fixes an oversight that hadn't been considered when the build
guards were added to the other files in the same directory, that
this test requires linking with code that hooks into cgo. This change
adds the build guard
+build cgo
Fixes#13266
This change removes a code pattern that I noticed while on a late night
audit of cosmovisor in which
strings.Builder.WriteString(fmt.Sprintf(...))
calls were being made, yet that's counterproductive to using fmt.Fprintf
which will check whether the writer implements .WriteString and then
avoids the need to firstly build a string using fmt.Sprintf.
The performance wins from this change transcend all dimensions as
exhibited below:
```shell
$ benchstat before.txt after.txt
name old time/op new time/op delta
DetailString-8 5.48µs ±23% 4.40µs ±11% -19.79% (p=0.000 n=20+17)
name old alloc/op new alloc/op delta
DetailString-8 2.63kB ± 0% 2.11kB ± 0% -19.76% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
DetailString-8 63.0 ± 0% 50.0 ± 0% -20.63% (p=0.000 n=20+20)
```
Fixes#13229
Noticed in an audit that the differeent value renderers perform an
expensive and unnecessary string->byteslice in cases where the output
write implements io.StringWriter. This change instead invokes
io.WriteString(w, formatted)
instead of:
w.Write([]byte(formatted))
and added benchmarks that show an improvement from just the 1 line change:
```shell
$ benchstat before.txt after.txt
name old time/op new time/op delta
IntValueRendererFormat-8 4.13µs ± 3% 3.95µs ± 6% -4.55% (p=0.000 n=15+14)
BytesValueRendererFormat-8 5.22ms ± 3% 4.77ms ± 5% -8.60% (p=0.000 n=15+14)
name old alloc/op new alloc/op delta
IntValueRendererFormat-8 3.64kB ± 0% 3.31kB ± 0% -9.01% (p=0.000 n=15+15)
BytesValueRendererFormat-8 12.6MB ± 0% 8.4MB ± 0% -33.22% (p=0.000 n=15+15)
name old allocs/op new allocs/op delta
IntValueRendererFormat-8 76.0 ± 0% 67.0 ± 0% -11.84% (p=0.000 n=15+15)
BytesValueRendererFormat-8 27.0 ± 0% 18.0 ± 0% -33.33% (p=0.000 n=15+15)
```
While here, implemented negative sign preservation because previously
the code wasn't tested for negative values so passing in negative values
such as:
"-10000000.11"
would produce:
"10'000'000.11"
instead of the proper value with the negative sign preserved:
"-10'000'000.11"
Fixes#12810Fixes#12812
Reported by informalsystems/gosec's map iteration pass, this
change fixes non-determinism from iterating with a map, whose
definition in the Go spec guarantees that the order of keys
and values presented during iteration will be randomized.
Fixes#12428Fixes#12430Fixes#12431
This change fixes resource leaks that were present due a rigid
error return path and didn't invoke .Close until the last line
of each method. This change fixes the problem by using a named
return error variable and if the return error value is nil, we
derive its final value from the unconditionally defer os.File.Close
value, which matches the prior logic but now fixes the flaw.
Fixes#12336Fixes#12337Fixes#12338
The specification of "copy", the builtin function per
https://pkg.go.dev/builtin#copy, says that it returns the minimum of
len(src) and len(dst) when invoked as:
copy(dst, src)
of which the prior code blindly assumed that everytime that
copy is invoked that the buffer provided had enough size
to accomodate the contents of *.MarshalTo but this isn't true
at all if len(data) is less than the values of .Marshal()
This change adds a negative value check to Uint.Unmarshal,
which coincidentally is fixed by refactoring for code reuse.
While here, added tests to ensure we don't regress.
Fixes#11995
We can shave off some milliseconds, but also cut down some Megabytes of
RAM consumed by only requesting from the cache if needed, but also using
the map clearing idiom which is recognized by the compiler to make fast
code.
Noticed in profiles from Tharsis' Ethermint per https://github.com/tharsis/ethermint/issues/710
- Before
* Memory profiles
```shell
19.50MB 19.50MB 134: store.cache = make(map[string]*cValue)
18.50MB 18.50MB 135: store.deleted = make(map[string]struct{})
15.50MB 15.50MB 136: store.unsortedCache = make(map[string]struct{})
```
* CPU profiles
```go
. . 118: // TODO: Consider allowing usage of Batch, which would allow the write to
. . 119: // at least happen atomically.
150ms 150ms 120: for _, key := range keys {
220ms 3.64s 121: cacheValue := store.cache[key]
. . 122:
. . 123: switch {
. 250ms 124: case store.isDeleted(key):
. . 125: store.parent.Delete([]byte(key))
210ms 210ms 126: case cacheValue.value == nil:
. . 127: // Skip, it already doesn't exist in parent.
. . 128: default:
240ms 27.94s 129: store.parent.Set([]byte(key), cacheValue.value)
. . 130: }
. . 131: }
...
10ms 60ms 134: store.cache = make(map[string]*cValue)
. 40ms 135: store.deleted = make(map[string]struct{})
. 50ms 136: store.unsortedCache = make(map[string]struct{})
. 110ms 137: store.sortedCache = dbm.NewMemDB()
```
- After
* Memory profiles
```shell
. . 130: // Clear the cache using the map clearing idiom
. . 131: // and not allocating fresh objects.
. . 132: // Please see https://bencher.orijtech.com/perfclinic/mapclearing/
. . 133: for key := range store.cache {
. . 134: delete(store.cache, key)
. . 135: }
. . 136: for key := range store.deleted {
. . 137: delete(store.deleted, key)
. . 138: }
. . 139: for key := range store.unsortedCache {
. . 140: delete(store.unsortedCache, key)
. . 141: }
```
* CPU profiles
```shell
. . 111: // TODO: Consider allowing usage of Batch, which would allow the write to
. . 112: // at least happen atomically.
110ms 110ms 113: for _, key := range keys {
. 210ms 114: if store.isDeleted(key) {
. . 115: // We use []byte(key) instead of conv.UnsafeStrToBytes because we cannot
. . 116: // be sure if the underlying store might do a save with the byteslice or
. . 117: // not. Once we get confirmation that .Delete is guaranteed not to
. . 118: // save the byteslice, then we can assume only a read-only copy is sufficient.
. . 119: store.parent.Delete([]byte(key))
. . 120: continue
. . 121: }
. . 122:
50ms 2.45s 123: cacheValue := store.cache[key]
910ms 920ms 124: if cacheValue.value != nil {
. . 125: // It already exists in the parent, hence delete it.
120ms 29.56s 126: store.parent.Set([]byte(key), cacheValue.value)
. . 127: }
. . 128: }
. . 129:
. . 130: // Clear the cache using the map clearing idiom
. . 131: // and not allocating fresh objects.
. . 132: // Please see https://bencher.orijtech.com/perfclinic/mapclearing/
. 210ms 133: for key := range store.cache {
. . 134: delete(store.cache, key)
. . 135: }
. 10ms 136: for key := range store.deleted {
. . 137: delete(store.deleted, key)
. . 138: }
. 170ms 139: for key := range store.unsortedCache {
. . 140: delete(store.unsortedCache, key)
. . 141: }
. 260ms 142: store.sortedCache = dbm.NewMemDB()
. 10ms 143:}
```
Fixes#10487
Updates https://github.com/tharsis/ethermint/issues/710
Per https://golang.org/pkg/fmt#hdr-Printing, Go's fmt "%T" specifier
prints out the underlying type hence we don't need to invoke
reflect.Typeof(key).Name() just to get it. This change was discovered
while examining Informal Systems' static analyzers that want to flag
certain packages.
Fixes#10391
This change takes the observation that previous dbm.IsKeyInDomain
which searches for [start, end) was performing too many byteslice
comparisons. Instead we start off by sorting all the values in the
store.unsortedCache, and then apply a modified binary search to
look for values that fall within the domain [start, end)
The procedure involves:
* iterating over all items to build a list of all keys -- O(n)
* invoking sort.Strings immediately, of which
we anyways eventually invoke sort.Slice(unsorted, ...) which uses
Quicksort -- O(nlog(n)) or O(n^2) worst case
* invoking modified binary search which is O(log(n)) * 2 ~ O(log(n))
to search for the [start, end) range indices
for a total approximate complexity of:
Best case: O(n) + O(n(log(n))) + O(log(n)) ~= O(nlog(n))
Worst case: O(n) + O(n^2) + O(log(n)) ~= O(n^2)
instead of previously:
* iterating over all the unsorted items and invoking dbm.IsKeyInDomain:
bytes.Compare ~ O(n) + O(n*s*e) where s -- len(start), e -- len(end)
for overall complexity of O(n*s*e)
* invoking sort.Slice(unsorted, ...) which uses
Quicksort -- O(nlog(n)) or O(n^2) worst case
for a total approximate complexity of:
Best case: O(n) + O(n*s*e) + O(nlog(n)) ~= O(n*s*e) ~ O(n^2)
Worst case: O(n) + O(n*s*e) + O(n^2) ~= O(n*s*e) ~ O(n^2)
Ordinarily we'd combine the n*s*e to be n*m, but really the comparisons
between (start & key, end & key) are profound that it makes sense to
keep them as factors. The overall benchmark results vindicate our choice
of isolating the factors (n*s*e)
The benchmarks show that as the number of keys to iterate grows, the
new code grows gracefully in a somewhat linear growth, notice for
CAcheKVStoreIterator*, when we go from:
* 1,000 to 10,000 keys: 120us->1,600us (13X) old vs 95us->900us (9.47X) new
* 50,000 to 100,000 keys: 19ms->100ms (5.3X) old vs 5.5ms->17ms (3X) new
```shell
time/op
GetValidator-8 5.8ms ± 2% 4.7ms ± 1% -17.69% (p=0.000 n=10+10)
OneBankSendTxPerBlock-8 3.2ms ± 2% 2.8ms ± 1% -10.80% (p=0.000 n=7+10)
OneBankMultiSendTxPerBlock-8 3.1ms ± 3% 2.9ms ± 2% -8.36% (p=0.000 n=10+10)
AccountMapperSetAccount-8 8.6µs ± 1% 7.8µs ± 1% -9.74% (p=0.000 n=10+10)
CacheKVStoreIterator500-8 64µs ± 6% 51µs ± 6% -19.22% (p=0.000 n=10+9)
CacheKVStoreIterator1000-8 0.12ms ± 4% 95µs ± 4% -19.55% (p=0.000 n=10+10)
CacheKVStoreIterator10000-8 1.6ms ± 4% 0.90ms ± 1% -42.11% (p=0.000 n=10+10)
CacheKVStoreIterator50000-8 19ms ± 5% 5.5ms ± 1% -71.35% (p=0.000 n=10+10)
CacheKVStoreIterator100000-8 0.10s ± 23% 17ms ± 7% -83.44% (p=0.000 n=10+10)
CacheKVStoreGetNoKeyFound-8 1.3µs ± 6% 0.90µs ± 3% -31.19% (p=0.000 n=9+9)
CacheKVStoreGetKeyFound-8 0.66µs ± 6% 0.56µs ± 2% -14.81% (p=0.000 n=10+9)
alloc/op
B/op
BlockProvision-8 0.11kB ± 0% 0.10kB ± 0% -7.14% (p=0.000 n=10+10)
CacheKVStoreIterator50000-8 0.89MB ± 6% 0.53MB ± 1% -40.85% (p=0.000 n=10+10)
CacheKVStoreIterator100000-8 6.3MB ± 23% 1.6MB ± 6% -74.17% (p=0.000 n=10+10)
CacheKVStoreGetNoKeyFound-8 0.26kB ± 0% 0.23kB ± 1% -11.53% (p=0.000 n=10+8)
allocs/op (count)
AccountMapperSetAccount-8 42 ± 0% 38 ± 0% -9.52% (p=0.000 n=10+10)
BlockProvision-8 6.0 ± 0% 5.0 ± 0% -16.67% (p=0.000 n=10+10)
CacheKVStoreIterator1000-8 14 ± 0% 13 ± 0% -7.14% (p=0.002 n=8+10)
CacheKVStoreIterator10000-8 0.15k ± 2% 76 ± 1% -49.00% (p=0.000 n=7+10)
CacheKVStoreIterator50000-8 8.9k ± 11% 2.0k ± 2% -77.60% (p=0.000 n=10+10)
CacheKVStoreIterator100000-8 0.10M ± 26% 13k ± 12% -86.89% (p=0.000 n=10+10)
CacheKVStoreGetNoKeyFound-8 5.0 ± 0% 4.0 ± 0% -20.00% (p=0.000 n=10+10)
```
Note: Purposefully using a commit off master that doesn't
include the buggy code that caused x/bank.BenchmarkOneBank* to fail
per issue https://github.com/cosmos/cosmos-sdk/issues/10023
Updates #9876
/cc @cuonglm @kirbyquerby
<!--
The default pull request template is for types feat, fix, or refactor.
For other templates, add one of the following parameters to the url:
- template=docs.md
- template=other.md
-->
## Description
Closes: #XXXX
<!-- Add a description of the changes that this PR introduces and the files that
are the most critical to review. -->
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [ ] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
fmt.Printf or fmt.Sprintf already know how to convert a
byteslice into a string when building the output; we shouldn't
incur the unnecessary string(byteslice) conversion.
Using Bencher, we can see improvements such as
https://dashboard.github.orijtech.com/benchmark/3245b8e4bbbd44a597480319aaa4b9fe
which in independent experiments show:
* time/op (ns/op)
FormatIt-8 1.2µs ± 2% 1.1µs ± 10% -11.77% (p=0.000 n=10+9)
* speed (MB/s)
FormatIt-8 0.71GB/s ± 2% 0.80GB/s ± 9% +13.59% (p=0.000 n=10+9)
* allocs/op (B/op)
FormatIt-8 2.0kB ± 0% 1.1kB ± 0% -45.62% (p=0.000 n=10+10)
* allocs/op (count/op)
FormatIt-8 11 ± 0% 9.0 ± 0% -18.18% (p=0.000 n=10+10)
Fixes#10363
Fixes unchecked negative index access that'd cause panics, in CompactBitArray's:
* CompactUnmarshal, which blindly used the result of binary.Uvarint
* GetIndex
* SetIndex
Fixes#9164Fixes#9165
Ensure that we don't pass overflowed values into make, because
a clever attacker could see that to cause:
(bits+7)/8
to become negative, they just have to make (bits+7) become negative
simply by >=maxint-6
but also reject unreasonably large element count like >2**32, which
while arbitrary is super duper large for a bit array.
Fixes#9162
* Skips very noisy benchmarks that end up running only for b.N=1 because
their entire time is spent in setup, and varying parameters doesn't change
much given that the number of stores is what dominates the expense. To
ensure we can provide reliable benchmarks, progressively for the project,
skip these until there is a proper re-work of what the benchmarks need to do
* Previously sub-benchmarks: b.Run(...) did not b.ReportAllocs() due to a faulty
assumption that invoking b.ReportAllocs() at the top would be inherited by
all sub-benchmarks. This change fixes that
Fixes#8779Fixes#8855
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Reduces CPU burn by using a typed List to avoid the expensive type
assertions from using an interface. This is the only option for now
until Go makes generics generally available.
The change brings time spent on the time assertion cummulatively to:
580ms down from 6.88s
Explanation:
The type assertions were showing up heavily in profiles:
* Before this commit
```shell
Total: 42.18s
ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator
in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go
14.01s 18.87s (flat, cum) 44.74% of Total
. . 17: items []*kv.Pair
. . 18: ascending bool
. . 19:}
. . 20:
. . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator {
. 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len())
. . 23:
. . 24: var entered bool
. . 25:
510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() {
6.85s 6.88s 27: item := e.Value.(*kv.Pair)
5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) {
120ms 120ms 29: if entered {
. . 30: break
. . 31: }
. . 32:
. . 33: continue
. . 34: }
. . 35:
820ms 980ms 36: itemsInDomain = append(itemsInDomain, item)
. . 37: entered = true
. . 38: }
. . 39:
. 1.21s 40: return &memIterator{
. . 41: start: start,
. . 42: end: end,
. . 43: items: itemsInDomain,
. . 44: ascending: ascending,
. . 45: }
```
and given that the list only uses that type, it is only right to lift the
code from container/list.List, and only modify Element.Value's type.
For emphasis, the code is basically just a retrofit of
container/list/list.go but with a typing, and we'll keep it as is
until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after
generics have landed.
* After this commit
```shell
Total: 45.25s
ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator
in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go
4.84s 6.77s (flat, cum) 14.96% of Total
. . 16: items []*kv.Pair
. . 17: ascending bool
. . 18:}
. . 19:
. . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator {
. 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len())
. . 22:
. . 23: var entered bool
. . 24:
60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() {
580ms 580ms 26: item := e.Value
3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) {
80ms 80ms 28: if entered {
. . 29: break
. . 30: }
. . 31:
. . 32: continue
. . 33: }
. . 34:
440ms 580ms 35: itemsInDomain = append(itemsInDomain, item)
. . 36: entered = true
. . 37: }
. . 38:
. 260ms 39: return &memIterator{
. . 40: start: start,
. . 41: end: end,
. . 42: items: itemsInDomain,
. . 43: ascending: ascending,
. . 44: }
```
Fixes#8810
This benchmark examines how ValidateGenesis behaves.
In a follow-up issue, I'll show how it reveals an inefficiency
that'll affect trying to load repeatedly retrieve Validators'
ConsAddress values.
Updates #8744
After continuously profiling InitGensis with 100K accounts, it showed
pathologically slow code, that was the result of a couple of patterns:
* Unconditional and not always necessary map lookups
* O(n^2) sdk.AccAddressFromBech32 retrievals when the code is expensive,
during a quicksort
The remedy involved 4 parts:
* O(n) sdk.AccAddressFromBech32 invocations, down from O(n^2) in the quicksort
* Only doing map lookups when the domain key check has passed
* Using a black magic compiler technique of the map clearing idiom
* Zero allocation []byte<->string conversion
With 100K accounts, this brings InitGenesis down to ~6min, instead of
20+min, it reduces the sort code from ~7sec down to 50ms.
Also some simple benchmark reflect the change:
```shell
name old time/op new time/op delta
SanitizeBalances500-8 19.3ms ±10% 1.5ms ± 5% -92.46% (p=0.000 n=20+20)
SanitizeBalances1000-8 41.9ms ± 8% 3.0ms ±12% -92.92% (p=0.000 n=20+20)
name old alloc/op new alloc/op delta
SanitizeBalances500-8 9.05MB ± 6% 0.56MB ± 0% -93.76% (p=0.000 n=20+18)
SanitizeBalances1000-8 20.2MB ± 3% 1.1MB ± 0% -94.37% (p=0.000 n=20+19)
name old allocs/op new allocs/op delta
SanitizeBalances500-8 72.4k ± 6% 4.5k ± 0% -93.76% (p=0.000 n=20+20)
SanitizeBalances1000-8 162k ± 3% 9k ± 0% -94.40% (p=0.000 n=20+20)
```
The CPU profiles show the radical change as per
https://github.com/cosmos/cosmos-sdk/issues/7766#issuecomment-786671734
Later on, we shall do more profiling and fixes but for now this brings
down the run-time for InitGenesis.
Fixes#7766
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
Given that AccAddress is a []byte type, and its .String() method is
quite expensive, continuously invoking the method doesn't easily provide
a way for the result to be memoized. In memory profiles from
benchmarking OneBankSendTxPerBlock and run for a while, it showed >2GiB burnt
and SendCoins consuming a bunch of memory: >2GiB.
This change introduces a simple global cache with a map to intern
AccAddress values, so that we quickly look up the expensively computed
values. With it, the prior memory profiles are erased, and benchmarks
show improvements:
```shell
$ benchstat before.txt after.txt
name old time/op new time/op delta
OneBankSendTxPerBlock-8 1.94ms ± 9% 1.92ms ±11% -1.34% (p=0.011 n=58+57)
name old alloc/op new alloc/op delta
OneBankSendTxPerBlock-8 428kB ± 0% 365kB ± 0% -14.67% (p=0.000 n=58+54)
name old allocs/op new allocs/op delta
OneBankSendTxPerBlock-8 6.34k ± 0% 5.90k ± 0% -7.06% (p=0.000 n=58+57)
```
Fixes#8693
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
The code in NewDecodeStore decoded the wrong proposal due
to a typographical error, but the tests used the exact same
value for the key value pairs hence the typo could never be caught.
I noticed it during an audit of the code, and I've fixed the
tests to pass in varying values for the various key value pairs.
Fixes#8570
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
Detected during my audit, right before fuzzing, the code that
checked for presence of hyphens per path segment assumed that
the part would always be non-empty. However, with paths such as:
* m/4/
* /44/
* m/4///
it'd panic with a runtime slice out of bounds.
With this new change, we now:
* firstly strip the right trailing slash
* on finding any empty segments of a path return an error
Fixes#8557
Avoids a bleed out attack in which a node can be made to allocate
memory slowly or very fast in small strides, by sending bad data
to code that invokes NewAnyWithCustomTypeURL, in which we
unconditionally returned a new Any object. On a 64-bit machine,
this would waste 96 bytes per invocation even on error.
Added a test to ensure zero allocations with a fixed error returned.
Also added a benchmark which shows reduction in wasted allocations and
wasted CPU time:
```shell
$ benchstat before.txt after.txt
name old time/op new time/op delta
NewAnyWithCustomTypeURLWithErrorReturned-8 142ns ± 6% 55ns ±12% -61.65% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
NewAnyWithCustomTypeURLWithErrorReturned-8 96.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
NewAnyWithCustomTypeURLWithErrorReturned-8 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
```
Fixes#8537
With this change, we'll get details on the number of
allocations performed by code. Later on when we have
continuous benchmarking infrastructure, this change
will prove useful to flag regressions.
Fixes#8459
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
Simplify the code in Dec.Power to unconditionally divide by 2, since:
x /= 2
is the same result for both even and odd integers, and produces the
floor of the result of x/2 -> ⌊x/2⌋:
99/2 ~> 49.5 ⌊49.5⌋ = 49
98/2 ~> 49.0 ⌊49.0⌋ = 49
aka "integer truncation".
Fixes#7924
Follows up PR #4876 which removed the return on non-nil errors
to always return a non-nil REST response regardless of error.
The checks performed seem benign and the return result is the
same.
This change uses the reverse conditional refactoring method
to gut out nesting and many conditions.
Noticed while hunting through types.ParseABCILogs.
Fixes#8181
Updates cosmos/go-bip39 to version v1.0.0 which
features performance upgrades that bring 12+ searches
down from 24.3us to 1.4us (so 108.33ns per fetch)
after PR https://github.com/cosmos/go-bip39/pull/2.
The prior code that tried to get the newest/larger time
between the too performed:
* 2 time.Time.UnixNano() retrievals: 2 multiplications, 2 additions
* 1 math.Max comparison
* 1 time.Unix(0, minTime) creation
but really this code could just use native operations on:
time.Time.After() simply.
This now becomes a single operation, single comparison for the same effect.
* x/ibc/core/23-commitment/types: fix MerkleProof.Empty comparisons
Fixes invalid pointer creation, reduces on unnecessary allocations
inside MerkleProof.Empty, and changes the method to a pointer; the
invalid pointer creation was symptomatic of broad use of values.
With this change we have improvements whose benchmarks produce:
```shell
name old time/op new time/op delta
Empty-8 311ns ± 5% 232ns ± 5% -25.49% (p=0.000 n=20+19)
name old alloc/op new alloc/op delta
Empty-8 56.0B ± 0% 8.0B ± 0% -85.71% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
Empty-8 3.00 ± 0% 1.00 ± 0% -66.67% (p=0.000 n=20+20)
```
Fixes#8091
* Move Empty godoc to right place + add comments for blank*
* x/bank/keeper: fix a missing gRPC error in AllBalances
Fixes an incoherent return which had:
(non-nil value, non-nil-non-grpc error)
to
(nil value, non-nil-gRPC error)
and it is a follow-up of PR #7814.
* types/query: fix pagination test (#8096)
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
* types/Coin: compile and reuse Regexps to reduce massive RAM+CPU burn
Fixes a resource (CPU+RAM) burn from recompiling Regexps which would
consume 1.2+MiB of RAM and 1.1ms per ParseCoin invocation, which means
that 1,000 invocations would consume 1.2GiB of RAM and >1second.
With the change, ParseCoin now takes 3.2us down from 1.1ms, with
dramatic results:
name old time/op new time/op delta
ParseCoin-8 1.04ms ± 4% 0.00ms ± 1% -99.69% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
ParseCoin-8 1.20MB ± 0% 0.00MB ± 0% -99.94% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ParseCoin-8 17.8k ± 0% 0.0k ± 0% -99.89% (p=0.000 n=10+10)
Fixes#7986.
* replace the CoinDenomRegex variable with SetCoinDenomRegex() (#7998)
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
Invoking:
c := make(chan os.Signal)
signal.Notify(c, signals...)
is explicitly called out as a bug in the os/signal docs
and should instead make that channel buffered lest
a signal could be lost.
Fixes#7950
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
* unknownproto: check result from protowire.ConsumeFieldValue and return error
Given that protowire.ConsumeFieldValue returns -1 when it encounters an
error, perform a check for n < 0 and return the respectively obtained
error with context about the details.
Fixes an issue identified from a go-fuzz session, thanks to Ethan
Buchman and the IBC auditors from Informal Systems et al.
Fixes#7739.
* Address AlexanderBez's suggestions
* Use require in tests
* Add issue #7846 to TODO
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* crypto/hd: properly catch index overflows to ensure conformance with BIP 32
Uses 31 bits as the bitsize argument to strconv.ParseUint to ensure
that we correctly parse values in the range [0, max(int32)]
Adds tests too to prevent future regressions of this subtlety.
Fixes#7627.
* Address Fedekunze's testing review
* baseapp, client: reject gRPC connections with out-of-range/nefarious x-cosmos-block-height values
Rejects gRPC connections that send out-of-range x-cosmos-block-height
values that previously weren't checked for. We now reject any negative
values and any value greater than max(int64) aka >9223372036854775807.
Also added an enforcement for returning an error if any negative heights
are passed into (*BaseApp).createQueryContext.
Fixes#7662
* baseapp, client: reject gRPC connections with out-of-range/nefarious x-cosmos-block-height values
Rejects gRPC connections that send out-of-range x-cosmos-block-height
values that previously weren't checked for. We now reject any negative
values and any value greater than max(int64) aka >9223372036854775807.
Also added an enforcement for returning an error if any negative heights
are passed into (*BaseApp).createQueryContext.
Fixes#7662
* Address Robert's feedback to extract negative height checker
* Fix tests
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Catches and returns unhandled errors after stat-ing for the configFile,
instead of assuming that we'd only encounter a non-existent error.
Added a regression test to lock-in checks for unhandled errors like
Permission errors, which is representative of all the errors that
haven't been accounted for.
Fixes#7578
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* x/genutil: fix CollectTxs traversal logic
Fixes the file traversal of CollectTxs to correctly skip over
directories instead of trying to read them, failing and erroring
out.
Also while here, changed the order to perform the os read
only after the AppState UnmarshalJSON has succeeded, otherwise
an attack vector can be to purposefully request many file stats
which touches kernel resources, while just causing failures
indefinitely.
Fixes#6788
* address comments
* Fix test by passing in blank AppState
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: SaReN <sahithnarahari@gmail.com>
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
Co-authored-by: Federico Kunze <federico.kunze94@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Use a for loop to correctly and fully dereference and
extract the type for the type of ParamSetPair.Value.
Fixes#6785
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* codec: remove unnecessary allocations in ProtoCodec.MarshalBinaryLengthPrefixed
Improve ProtoCodec.MarshalBinaryLengthPrefixed by removing the need
to use a *bytes.Buffer and instead use an array and invoke
binary.PutUvarint, and directly create the respective length prefixed
concatentation.
With this change we get the following improvement in all dimenions
of throughput, bytes allocated, number of allocations per operation.
```shell
$ benchstat before.txt after.txt
name old time/op new time/op delta
ProtoCodecMarshalBinaryLengthPrefixed-8 295ns ± 2% 177ns ± 3% -39.92% (p=0.000 n=20+20)
name old speed new speed delta
ProtoCodecMarshalBinaryLengthPrefixed-8 505MB/s ± 2% 841MB/s ± 3% +66.44% (p=0.000 n=20+20)
name old alloc/op new alloc/op delta
ProtoCodecMarshalBinaryLengthPrefixed-8 576B ± 0% 336B ± 0% -41.67% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
ProtoCodecMarshalBinaryLengthPrefixed-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=20+20)
```
Fixes#6875
* Address @marbar3778's feedback
Use binary.MaxVarintLen64 instead of 10
Co-authored-by: Marko <marbar3778@yahoo.com>
Co-authored-by: Marko <marbar3778@yahoo.com>
Co-authored-by: Aaron Craelius <aaron@regen.network>
Noticed during my audit, the code for cache.CommitKVStoreCacheManager.Reset()
discarded the prior on every invocation i.e.
m = make(map[T(key)]T(value))
However, this can be made fast and conserve memory by using the map clearing idiom
that the Go compiler recognizes i.e.
for key := range m {
delete(m, key)
}
and turns into very fast code, instead of the extraneous map discarding.
The speed up generated is:
```shell
$ benchstat before after
name old time/op new time/op delta
Reset-8 204ns ± 2% 66ns ± 9% -67.86% (p=0.000 n=20+20)
name old alloc/op new alloc/op delta
Reset-8 384B ± 0% 0B -100.00% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
Reset-8 3.00 ± 0% 0.00 -100.00% (p=0.000 n=20+20)
```
Fixes#6681
Noticed during an audit, we've got duplicated code that's unused.
There were 2 type definitions:
* type kvPair types.Pair
* type KVPair types.Pair
and each had a .bytes() and .Bytes() method respectively that
then used some extra code.
This change deletes the duplicated/unnecessary code but also
while here improves the performance by removing the use of a
bytes.Buffer which is unnecessary given that we are only
length prefixed the key, length prefixing the value hence
the various helpers are unnecessary.
The added benchmarks shows the performance boost
```shell
$ benchstat before after
name old time/op new time/op delta
KVPairBytes-8 146µs ± 1% 142µs ± 2% -3.05% (p=0.000 n=18+17)
name old speed new speed delta
KVPairBytes-8 6.84GB/s ± 1% 7.06GB/s ± 2% +3.15% (p=0.000 n=18+17)
name old alloc/op new alloc/op delta
KVPairBytes-8 1.01MB ± 0% 1.01MB ± 0% -0.04% (p=0.000 n=17+20)
name old allocs/op new allocs/op delta
KVPairBytes-8 6.00 ± 0% 1.00 ± 0% -83.33% (p=0.000 n=20+20)
```
Closes: #6688
As part of an audit, adds as much visible coverage from
73.4% to about 84.2%, but after #6668 is merged, we'll
have it about 88.X% and the coverage report seems to show
only missing cases of not common scenarios, e.g. a case that'll
make jsonpb.Marshaling to fail, and which will return an error
anyways.
Fixes an invalid int->string conversion that will become
a vet error for Go1.15. The correct conversion was to use
fmt.Sprintf("COINZ_%d", i)
instead of
"COINZ_" + string(i)
Noticed during a coverage audit.
Emmanuel T OdekeandMarko
