* Move snapshotstore under store
* add changelog entry
* errors store
* remove dep on sdk/types/errors
* clean up
* fix var
* reduce further reliance on sdk
* import fixes
* rename based off comments
* Move snapshotstore under store
* add changelog entry
* errors store
* remove dep on sdk/types/errors
* clean up
* fix var
* undo some changes, clean up fmt.Errorf usage
* remove pkg/errors
* fix: state listener observe writes at wrong time
Closes: #13457
Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.
It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.
It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync
* synchronous abci call, and format doc
* fix comment
* update file streamer readme and fix typos
* typo
* fix: state listener observe writes at wrong time
Closes: #13457
Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.
It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.
It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync
synchronous abci call, and format doc
fix comment
update file streamer readme and fix typos
typo
* improve UX of file streamer, make it immediately usable after enabled
- set default value to write_dir.
- make write_dir based on home directory by default.
- auto-create the directory if not exists.
* get homePage from opts
Co-authored-by: Marko <marbar3778@yahoo.com>
* fix: state listener could observe uncommitted writes
Closes: #13457
don't pass listeners to nested cached store,
only the most inner layer's cache writes should be observed.
* Update CHANGELOG.md
* add unit test
* rename
Co-authored-by: Marko <marbar3778@yahoo.com>
* Revert "chore: change id to use uint64 in `AccountAddressByID` (#13411)"
This reverts commit 3d0e214446.
* Add since
* Update CLI command
* make proto-gen
* id->accNum
* No API breaking changes
* improve err msg
* Update CHANGELOG.md
Co-authored-by: Julien Robert <julien@rbrt.fr>
* Update CHANGELOG.md
Co-authored-by: Julien Robert <julien@rbrt.fr>
Co-authored-by: Julien Robert <julien@rbrt.fr>
* Add go.mod for simapp
* creating dep check script
* new version of cosmos-sdk
* tests/ must be a module also if it is to test simapp
* maybe add a github action which should fail
* mv tests/mocks -> testutil/mock
* Refactor usages of tests/mocks
* update build command
* fix rosetta tests
* go mod tidy
* use cosmossdk.io/simapp
* Update sim entrypoints
* use simapp as a module
* go mod tidy
* Add replaced for vuln package
* fix vuln dep
* this CI run should fail
* this CI run should succeed
* use absolute path in makefile
## Description
Closes: #12763
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
- [ ] provided a link to the relevant issue or specification
- [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/building-modules)
- [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [ ] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
This change fixes a bounty by the Juno team. Juno's invariant checks took 10 hours during their most recent chain halt. This PR cuts that down to 30 seconds. See https://github.com/CosmosContracts/bounties#improve-speed-of-invariant-checks.
The root problem is deep in the `can-withdraw` invariant check, which calls this repeatedly: https://github.com/cosmos/cosmos-sdk/blob/main/x/distribution/keeper/store.go#L337. Iterators have a chain of parents and in this case creates an iterator from the `cachekv` store. For the genesis file, it has a cache of 500,000+ unsorted entries, which are sorted as strings here: https://github.com/cosmos/cosmos-sdk/blob/main/store/cachekv/store.go#L314. Each delegation from `can-withdraw` uses this cache and many of the cache checks miss or are a very small range. This means very few entries get removed from the unsorted cache and they have to be re-sorted on the next call. With a full cache it takes about 180ms on my machine to sort them.
This change introduce a minimum number of entries that will get processed and removed from the unsorted list. It's set at the same value that directs the code to sort them in the first place. This ensures the unsorted values get removed in a relative short amount of time, and amortizes the cost to ensure an individual check does not have to process the entire cache.
## Benchmarks
On running the benchmarks included in this change produces:
```shell
name old time/op new time/op delta
LargeUnsortedMisses-32 21.2s ± 9% 0.0s ± 1% -99.91% (p=0.000 n=20+17)
name old alloc/op new alloc/op delta
LargeUnsortedMisses-32 1.64GB ± 0% 0.00GB ± 0% -99.83% (p=0.000 n=19+19)
name old allocs/op new allocs/op delta
LargeUnsortedMisses-32 20.0k ± 0% 41.1k ± 0% +105.23% (p=0.000 n=19+20)
```
## Invariant checks results
This is what the invariant checks for Juno look like with this change (on a Hetzner AX101):
```shell
INF starting node with ABCI Tendermint in-process
4:11PM INF Starting multiAppConn service impl=multiAppConn module=proxy
4:11PM INF Starting localClient service connection=query impl=localClient module=abci-client
4:11PM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
4:11PM INF Starting localClient service connection=mempool impl=localClient module=abci-client
4:11PM INF Starting localClient service connection=consensus impl=localClient module=abci-client
4:11PM INF Starting EventBus service impl=EventBus module=events
4:11PM INF Starting PubSub service impl=PubSub module=pubsub
4:11PM INF Starting IndexerService service impl=IndexerService module=txindex
4:11PM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 software-version=v9.0.0-36-g8fd6f16
4:11PM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
4:12PM INF asserting crisis invariants inv=1/11 module=x/crisis name=gov/module-account
4:12PM INF asserting crisis invariants inv=2/11 module=x/crisis name=distribution/nonnegative-outstanding
4:12PM INF asserting crisis invariants inv=3/11 module=x/crisis name=distribution/can-withdraw
4:12PM INF asserting crisis invariants inv=4/11 module=x/crisis name=distribution/reference-count
4:12PM INF asserting crisis invariants inv=5/11 module=x/crisis name=distribution/module-account
4:12PM INF asserting crisis invariants inv=6/11 module=x/crisis name=bank/nonnegative-outstanding
4:12PM INF asserting crisis invariants inv=7/11 module=x/crisis name=bank/total-supply
4:12PM INF asserting crisis invariants inv=8/11 module=x/crisis name=staking/module-accounts
4:12PM INF asserting crisis invariants inv=9/11 module=x/crisis name=staking/nonnegative-power
4:12PM INF asserting crisis invariants inv=10/11 module=x/crisis name=staking/positive-delegation
4:12PM INF asserting crisis invariants inv=11/11 module=x/crisis name=staking/delegator-shares
4:12PM INF asserted all invariants duration=28383.559601 height=4136532 module=x/crisis
```
## Alternatives
There is another PR which fixes this problem for the Juno genesis file https://github.com/cosmos/cosmos-sdk/pull/12886. However, because of its concurrent nature, it happens to hit a large range relatively early, clearing the unsorted entries and allowing the rest of the checks to not sort it.
