## Description
- Return an exported type rather than an unexported type, from an exported function
- Assign error values to _
- Add reason to nolint directive
- Remove ineffectual assignment
- Remove checked cast to same type as value
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
- [ ] ~~provided a link to the relevant issue or specification~~
- [ ] ~~reviewed "Files changed" and left comments if necessary~~
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed all author checklist items have been addressed
- [ ] confirmed that this PR does not change production code
Co-authored-by: Marko <marbar3778@yahoo.com>
Co-authored-by: Marko <marko@baricevic.me>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
## Description
This pr removes base app from the store package. This is un order to decouple store from the sdk. There are a few things we can clean up but the telemetry package may be difficult with how it's done.
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
- [ ] provided a link to the relevant issue or specification
- [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules)
- [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [ ] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
* fix: state listener observe writes at wrong time
Closes: #13457
Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.
It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.
It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync
* synchronous abci call, and format doc
* fix comment
* update file streamer readme and fix typos
* typo
* fix: state listener observe writes at wrong time
Closes: #13457
Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.
It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.
It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync
synchronous abci call, and format doc
fix comment
update file streamer readme and fix typos
typo
* improve UX of file streamer, make it immediately usable after enabled
- set default value to write_dir.
- make write_dir based on home directory by default.
- auto-create the directory if not exists.
* get homePage from opts
Co-authored-by: Marko <marbar3778@yahoo.com>
This change fixes a bounty by the Juno team. Juno's invariant checks took 10 hours during their most recent chain halt. This PR cuts that down to 30 seconds. See https://github.com/CosmosContracts/bounties#improve-speed-of-invariant-checks.
The root problem is deep in the `can-withdraw` invariant check, which calls this repeatedly: https://github.com/cosmos/cosmos-sdk/blob/main/x/distribution/keeper/store.go#L337. Iterators have a chain of parents and in this case creates an iterator from the `cachekv` store. For the genesis file, it has a cache of 500,000+ unsorted entries, which are sorted as strings here: https://github.com/cosmos/cosmos-sdk/blob/main/store/cachekv/store.go#L314. Each delegation from `can-withdraw` uses this cache and many of the cache checks miss or are a very small range. This means very few entries get removed from the unsorted cache and they have to be re-sorted on the next call. With a full cache it takes about 180ms on my machine to sort them.
This change introduce a minimum number of entries that will get processed and removed from the unsorted list. It's set at the same value that directs the code to sort them in the first place. This ensures the unsorted values get removed in a relative short amount of time, and amortizes the cost to ensure an individual check does not have to process the entire cache.
## Benchmarks
On running the benchmarks included in this change produces:
```shell
name old time/op new time/op delta
LargeUnsortedMisses-32 21.2s ± 9% 0.0s ± 1% -99.91% (p=0.000 n=20+17)
name old alloc/op new alloc/op delta
LargeUnsortedMisses-32 1.64GB ± 0% 0.00GB ± 0% -99.83% (p=0.000 n=19+19)
name old allocs/op new allocs/op delta
LargeUnsortedMisses-32 20.0k ± 0% 41.1k ± 0% +105.23% (p=0.000 n=19+20)
```
## Invariant checks results
This is what the invariant checks for Juno look like with this change (on a Hetzner AX101):
```shell
INF starting node with ABCI Tendermint in-process
4:11PM INF Starting multiAppConn service impl=multiAppConn module=proxy
4:11PM INF Starting localClient service connection=query impl=localClient module=abci-client
4:11PM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
4:11PM INF Starting localClient service connection=mempool impl=localClient module=abci-client
4:11PM INF Starting localClient service connection=consensus impl=localClient module=abci-client
4:11PM INF Starting EventBus service impl=EventBus module=events
4:11PM INF Starting PubSub service impl=PubSub module=pubsub
4:11PM INF Starting IndexerService service impl=IndexerService module=txindex
4:11PM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 software-version=v9.0.0-36-g8fd6f16
4:11PM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
4:12PM INF asserting crisis invariants inv=1/11 module=x/crisis name=gov/module-account
4:12PM INF asserting crisis invariants inv=2/11 module=x/crisis name=distribution/nonnegative-outstanding
4:12PM INF asserting crisis invariants inv=3/11 module=x/crisis name=distribution/can-withdraw
4:12PM INF asserting crisis invariants inv=4/11 module=x/crisis name=distribution/reference-count
4:12PM INF asserting crisis invariants inv=5/11 module=x/crisis name=distribution/module-account
4:12PM INF asserting crisis invariants inv=6/11 module=x/crisis name=bank/nonnegative-outstanding
4:12PM INF asserting crisis invariants inv=7/11 module=x/crisis name=bank/total-supply
4:12PM INF asserting crisis invariants inv=8/11 module=x/crisis name=staking/module-accounts
4:12PM INF asserting crisis invariants inv=9/11 module=x/crisis name=staking/nonnegative-power
4:12PM INF asserting crisis invariants inv=10/11 module=x/crisis name=staking/positive-delegation
4:12PM INF asserting crisis invariants inv=11/11 module=x/crisis name=staking/delegator-shares
4:12PM INF asserted all invariants duration=28383.559601 height=4136532 module=x/crisis
```
## Alternatives
There is another PR which fixes this problem for the Juno genesis file https://github.com/cosmos/cosmos-sdk/pull/12886. However, because of its concurrent nature, it happens to hit a large range relatively early, clearing the unsorted entries and allowing the rest of the checks to not sort it.
## Description
Closes: #12661
Adds support for nil end semantics to iterators in cachekv store, addressing [this workaround](4176b287d4/osmoutils/store_helper.go (L37-L41)).
Note that this has the effect of sorting the dirty cache into the BTree cache store in the bounds `[startIndex, end-of-cache-space]`
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
## Description
Ref: https://github.com/cosmos/cosmos-sdk/issues/11362
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/building-modules)
- [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [ ] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
* [10948]: Add changelog entry.
* [10948]: Deprecate the types.DBBackend variable and the NewLevelDB function. Create a NewDB function to replace them.
* [10948]: Add a DBBackend string to the simulation config and a flag for setting it. Update the simulation setup to use that instead of the compile-time DBBackend variable.
* [10948]: Update the mock app creator to use the NewDB function. Not sure what to do about the db backend in that case though.
* [10948]: Update changelog to reflect new db-backend field name.
* [10948]: Use the tendermint db-backend type for the snapshot db.
* [10948]: Update the last use of NewLevelDB by adding a parameter to openDB and uppdating calls to that to provide the db type to use.
* [10948]: Upddate the NewDB function to also have a default db backend type if an empty string is provided there.
* [10948]: Remove the new TODO in mock.NewApp. After looking through it's uses, there doesn't seem to be any desire to change it, and there's no easy way to communicate it.
* [10948]: Enhance the NewDB defer function to also add info to any err that is being returned.
* [10948]: Add some unit tests for NewDB.
* [10948]: Lint fixes.
* [10948]: Add a changelog entry to the deprecated section.
* [10948]: Update the makefile to no longer set the types.DBBackend value.
* [10948]: Use memdb for the mock app instead of goleveldb. I know it was a goleveldb before, but for a mock app, a memdb feels like a better choice (assuming 'mock' and 'mem' mean what I assume they mean).
* [10948]: Fix the store benchmark tests (had some index-out-of-range issues).
* [10948]: Fix cachekv store bench test calling iter.Key() before checking iter.Valid().
* [10948]: Remove the panic recovery from types.NewDB since dbm.NewDB returns an error now (it didn't originally, when NewLevelDB was first written).
* [10948]: Add changlog entry indicationg an API breaking change due to the DBBackend change.
* [10948]: Get rid of the types.NewDB function in favor of just using the tm-db version of it.
* [10948]: Fix Update the codeql-analysis github action to use go v1.17.
* [10948]: Add config file option for the app db backend type.
* [10948]: Adjust the comment on the app-db-backend config entry to clarify fallback behavior.
* [10948]: Add a default of GoLevelDBBackend to GetAppDBBackend. The old DBBackend variable defaulted to that, and some unit tests assume that behavior still exists.
* [10948]: Add the missing quotes around the app-db-backend value.
* [10948]: Small tweak to the changelog's deprecated entry.
* Add the go version declaration back into the codeql-analysis github action.
* [10948]: Update new use of openDB.
* [10948]: Put a brief delay after closing the test network. Hopefully that helps with address-in-use and non-empty directory errors.
Co-authored-by: Marko <marbar3778@yahoo.com>
## Description
Cref discussion in #10026, updates the CacheKV Benchmark naming and implementation to correspond to whats actually going on, and remove many irrelevant/incorrect components from being in the benchmarks timing.
Basically the old Benchmark's iterator creation was very flawed, and never hit the complex case, only repeatedly performing the best case performance of the iterator. Instead it really benchmarked iterator set and next operations.
This PR splits out the benchmarks for those two accordingly.
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [x] added a changelog entry to `CHANGELOG.md` - N/A imo
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [x] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [x] confirmed all CI checks have passed - lint failure unrelated
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
We can shave off some milliseconds, but also cut down some Megabytes of
RAM consumed by only requesting from the cache if needed, but also using
the map clearing idiom which is recognized by the compiler to make fast
code.
Noticed in profiles from Tharsis' Ethermint per https://github.com/tharsis/ethermint/issues/710
- Before
* Memory profiles
```shell
19.50MB 19.50MB 134: store.cache = make(map[string]*cValue)
18.50MB 18.50MB 135: store.deleted = make(map[string]struct{})
15.50MB 15.50MB 136: store.unsortedCache = make(map[string]struct{})
```
* CPU profiles
```go
. . 118: // TODO: Consider allowing usage of Batch, which would allow the write to
. . 119: // at least happen atomically.
150ms 150ms 120: for _, key := range keys {
220ms 3.64s 121: cacheValue := store.cache[key]
. . 122:
. . 123: switch {
. 250ms 124: case store.isDeleted(key):
. . 125: store.parent.Delete([]byte(key))
210ms 210ms 126: case cacheValue.value == nil:
. . 127: // Skip, it already doesn't exist in parent.
. . 128: default:
240ms 27.94s 129: store.parent.Set([]byte(key), cacheValue.value)
. . 130: }
. . 131: }
...
10ms 60ms 134: store.cache = make(map[string]*cValue)
. 40ms 135: store.deleted = make(map[string]struct{})
. 50ms 136: store.unsortedCache = make(map[string]struct{})
. 110ms 137: store.sortedCache = dbm.NewMemDB()
```
- After
* Memory profiles
```shell
. . 130: // Clear the cache using the map clearing idiom
. . 131: // and not allocating fresh objects.
. . 132: // Please see https://bencher.orijtech.com/perfclinic/mapclearing/
. . 133: for key := range store.cache {
. . 134: delete(store.cache, key)
. . 135: }
. . 136: for key := range store.deleted {
. . 137: delete(store.deleted, key)
. . 138: }
. . 139: for key := range store.unsortedCache {
. . 140: delete(store.unsortedCache, key)
. . 141: }
```
* CPU profiles
```shell
. . 111: // TODO: Consider allowing usage of Batch, which would allow the write to
. . 112: // at least happen atomically.
110ms 110ms 113: for _, key := range keys {
. 210ms 114: if store.isDeleted(key) {
. . 115: // We use []byte(key) instead of conv.UnsafeStrToBytes because we cannot
. . 116: // be sure if the underlying store might do a save with the byteslice or
. . 117: // not. Once we get confirmation that .Delete is guaranteed not to
. . 118: // save the byteslice, then we can assume only a read-only copy is sufficient.
. . 119: store.parent.Delete([]byte(key))
. . 120: continue
. . 121: }
. . 122:
50ms 2.45s 123: cacheValue := store.cache[key]
910ms 920ms 124: if cacheValue.value != nil {
. . 125: // It already exists in the parent, hence delete it.
120ms 29.56s 126: store.parent.Set([]byte(key), cacheValue.value)
. . 127: }
. . 128: }
. . 129:
. . 130: // Clear the cache using the map clearing idiom
. . 131: // and not allocating fresh objects.
. . 132: // Please see https://bencher.orijtech.com/perfclinic/mapclearing/
. 210ms 133: for key := range store.cache {
. . 134: delete(store.cache, key)
. . 135: }
. 10ms 136: for key := range store.deleted {
. . 137: delete(store.deleted, key)
. . 138: }
. 170ms 139: for key := range store.unsortedCache {
. . 140: delete(store.unsortedCache, key)
. . 141: }
. 260ms 142: store.sortedCache = dbm.NewMemDB()
. 10ms 143:}
```
Fixes#10487
Updates https://github.com/tharsis/ethermint/issues/710
This change takes the observation that previous dbm.IsKeyInDomain
which searches for [start, end) was performing too many byteslice
comparisons. Instead we start off by sorting all the values in the
store.unsortedCache, and then apply a modified binary search to
look for values that fall within the domain [start, end)
The procedure involves:
* iterating over all items to build a list of all keys -- O(n)
* invoking sort.Strings immediately, of which
we anyways eventually invoke sort.Slice(unsorted, ...) which uses
Quicksort -- O(nlog(n)) or O(n^2) worst case
* invoking modified binary search which is O(log(n)) * 2 ~ O(log(n))
to search for the [start, end) range indices
for a total approximate complexity of:
Best case: O(n) + O(n(log(n))) + O(log(n)) ~= O(nlog(n))
Worst case: O(n) + O(n^2) + O(log(n)) ~= O(n^2)
instead of previously:
* iterating over all the unsorted items and invoking dbm.IsKeyInDomain:
bytes.Compare ~ O(n) + O(n*s*e) where s -- len(start), e -- len(end)
for overall complexity of O(n*s*e)
* invoking sort.Slice(unsorted, ...) which uses
Quicksort -- O(nlog(n)) or O(n^2) worst case
for a total approximate complexity of:
Best case: O(n) + O(n*s*e) + O(nlog(n)) ~= O(n*s*e) ~ O(n^2)
Worst case: O(n) + O(n*s*e) + O(n^2) ~= O(n*s*e) ~ O(n^2)
Ordinarily we'd combine the n*s*e to be n*m, but really the comparisons
between (start & key, end & key) are profound that it makes sense to
keep them as factors. The overall benchmark results vindicate our choice
of isolating the factors (n*s*e)
The benchmarks show that as the number of keys to iterate grows, the
new code grows gracefully in a somewhat linear growth, notice for
CAcheKVStoreIterator*, when we go from:
* 1,000 to 10,000 keys: 120us->1,600us (13X) old vs 95us->900us (9.47X) new
* 50,000 to 100,000 keys: 19ms->100ms (5.3X) old vs 5.5ms->17ms (3X) new
```shell
time/op
GetValidator-8 5.8ms ± 2% 4.7ms ± 1% -17.69% (p=0.000 n=10+10)
OneBankSendTxPerBlock-8 3.2ms ± 2% 2.8ms ± 1% -10.80% (p=0.000 n=7+10)
OneBankMultiSendTxPerBlock-8 3.1ms ± 3% 2.9ms ± 2% -8.36% (p=0.000 n=10+10)
AccountMapperSetAccount-8 8.6µs ± 1% 7.8µs ± 1% -9.74% (p=0.000 n=10+10)
CacheKVStoreIterator500-8 64µs ± 6% 51µs ± 6% -19.22% (p=0.000 n=10+9)
CacheKVStoreIterator1000-8 0.12ms ± 4% 95µs ± 4% -19.55% (p=0.000 n=10+10)
CacheKVStoreIterator10000-8 1.6ms ± 4% 0.90ms ± 1% -42.11% (p=0.000 n=10+10)
CacheKVStoreIterator50000-8 19ms ± 5% 5.5ms ± 1% -71.35% (p=0.000 n=10+10)
CacheKVStoreIterator100000-8 0.10s ± 23% 17ms ± 7% -83.44% (p=0.000 n=10+10)
CacheKVStoreGetNoKeyFound-8 1.3µs ± 6% 0.90µs ± 3% -31.19% (p=0.000 n=9+9)
CacheKVStoreGetKeyFound-8 0.66µs ± 6% 0.56µs ± 2% -14.81% (p=0.000 n=10+9)
alloc/op
B/op
BlockProvision-8 0.11kB ± 0% 0.10kB ± 0% -7.14% (p=0.000 n=10+10)
CacheKVStoreIterator50000-8 0.89MB ± 6% 0.53MB ± 1% -40.85% (p=0.000 n=10+10)
CacheKVStoreIterator100000-8 6.3MB ± 23% 1.6MB ± 6% -74.17% (p=0.000 n=10+10)
CacheKVStoreGetNoKeyFound-8 0.26kB ± 0% 0.23kB ± 1% -11.53% (p=0.000 n=10+8)
allocs/op (count)
AccountMapperSetAccount-8 42 ± 0% 38 ± 0% -9.52% (p=0.000 n=10+10)
BlockProvision-8 6.0 ± 0% 5.0 ± 0% -16.67% (p=0.000 n=10+10)
CacheKVStoreIterator1000-8 14 ± 0% 13 ± 0% -7.14% (p=0.002 n=8+10)
CacheKVStoreIterator10000-8 0.15k ± 2% 76 ± 1% -49.00% (p=0.000 n=7+10)
CacheKVStoreIterator50000-8 8.9k ± 11% 2.0k ± 2% -77.60% (p=0.000 n=10+10)
CacheKVStoreIterator100000-8 0.10M ± 26% 13k ± 12% -86.89% (p=0.000 n=10+10)
CacheKVStoreGetNoKeyFound-8 5.0 ± 0% 4.0 ± 0% -20.00% (p=0.000 n=10+10)
```
Note: Purposefully using a commit off master that doesn't
include the buggy code that caused x/bank.BenchmarkOneBank* to fail
per issue https://github.com/cosmos/cosmos-sdk/issues/10023
Updates #9876
/cc @cuonglm @kirbyquerby
<!--
The default pull request template is for types feat, fix, or refactor.
For other templates, add one of the following parameters to the url:
- template=docs.md
- template=other.md
-->
## Description
Closes: #XXXX
<!-- Add a description of the changes that this PR introduces and the files that
are the most critical to review. -->
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [ ] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
<!--
The default pull request template is for types feat, fix, or refactor.
For other templates, add one of the following parameters to the url:
- template=docs.md
- template=other.md
-->
## Description
Followup from #10077
Closes: #XXXX
<!-- Add a description of the changes that this PR introduces and the files that
are the most critical to review. -->
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [x] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [x] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
## Description
Closes: #10072
Significantly speeds up all the GasKvStore and CacheKVStore operations. Now Get/Set for these stores no longer even appears on my Osmosis benchmarks, saving ~8% of those benchmark's times. (Only one of the internal methods for setCacheValue appear -- will try to separately get a PR for reducing those memory allocations if #10026 is merged)
Talked to @alexanderbez about this on Discord, and he seemed in agreement with the approach of removing telemetry from these stores.
This does technically change telemetry, but I don't know if this is / should be considered a breaking change?
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification - is there something I should be updating?
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
## Description
So we don't burn un-necessary CPU in case of dirty store/delete.
```
name old time/op new time/op delta
CacheKVStoreIterator500-8 23.3µs ± 2% 23.4µs ± 1% ~ (p=0.151 n=5+5)
CacheKVStoreIterator1000-8 47.0µs ± 2% 46.3µs ± 0% -1.50% (p=0.008 n=5+5)
CacheKVStoreIterator10000-8 462µs ± 3% 458µs ± 1% ~ (p=0.690 n=5+5)
CacheKVStoreIterator50000-8 2.63ms ± 5% 2.51ms ± 2% -4.63% (p=0.032 n=5+5)
CacheKVStoreIterator100000-8 8.09ms ±21% 6.98ms ± 4% ~ (p=0.151 n=5+5)
CacheKVStoreGetNoKeyFound-8 393ns ± 2% 397ns ± 2% ~ (p=0.421 n=5+5)
CacheKVStoreGetKeyFound-8 269ns ± 5% 268ns ± 3% ~ (p=1.000 n=5+5)
```
Fixes#9991
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [ ] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
* StoreKVPair protobuf message definition and generated go types
* store WriteListener
* update MultiStore, CacheWrap, CacheWrapper interfaces
* adjust KVStores to fit new CacheWrapper interface
* new ListenKVStore
* adjust multistores to fit new MultiStore interface and enable wrapping returned KVStores with the new ListenKVStore
* typo fixes in adr
* ListenKV Store test
* update server mock KVStore and MultiStore
* multistore unit test; fix multistore constructor
* update changelog
* fix bug identified in CI
* improve codecov, minor fixes/adjustments
* review fixes
* review updates; flip set to delete in KVStorePair, updated proto-docs from running 'make proto-gen'
Reduces CPU burn by using a typed List to avoid the expensive type
assertions from using an interface. This is the only option for now
until Go makes generics generally available.
The change brings time spent on the time assertion cummulatively to:
580ms down from 6.88s
Explanation:
The type assertions were showing up heavily in profiles:
* Before this commit
```shell
Total: 42.18s
ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator
in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go
14.01s 18.87s (flat, cum) 44.74% of Total
. . 17: items []*kv.Pair
. . 18: ascending bool
. . 19:}
. . 20:
. . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator {
. 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len())
. . 23:
. . 24: var entered bool
. . 25:
510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() {
6.85s 6.88s 27: item := e.Value.(*kv.Pair)
5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) {
120ms 120ms 29: if entered {
. . 30: break
. . 31: }
. . 32:
. . 33: continue
. . 34: }
. . 35:
820ms 980ms 36: itemsInDomain = append(itemsInDomain, item)
. . 37: entered = true
. . 38: }
. . 39:
. 1.21s 40: return &memIterator{
. . 41: start: start,
. . 42: end: end,
. . 43: items: itemsInDomain,
. . 44: ascending: ascending,
. . 45: }
```
and given that the list only uses that type, it is only right to lift the
code from container/list.List, and only modify Element.Value's type.
For emphasis, the code is basically just a retrofit of
container/list/list.go but with a typing, and we'll keep it as is
until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after
generics have landed.
* After this commit
```shell
Total: 45.25s
ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator
in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go
4.84s 6.77s (flat, cum) 14.96% of Total
. . 16: items []*kv.Pair
. . 17: ascending bool
. . 18:}
. . 19:
. . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator {
. 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len())
. . 22:
. . 23: var entered bool
. . 24:
60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() {
580ms 580ms 26: item := e.Value
3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) {
80ms 80ms 28: if entered {
. . 29: break
. . 30: }
. . 31:
. . 32: continue
. . 33: }
. . 34:
440ms 580ms 35: itemsInDomain = append(itemsInDomain, item)
. . 36: entered = true
. . 37: }
. . 38:
. 260ms 39: return &memIterator{
. . 40: start: start,
. . 41: end: end,
. . 42: items: itemsInDomain,
. . 43: ascending: ascending,
. . 44: }
```
Fixes#8810
