chore: Update ADR-065 (#18862)
Co-authored-by: Marko <marbar3778@yahoo.com> Co-authored-by: Cool Developer <cool199966@outlook.com> Co-authored-by: cool-developer <51834436+cool-develope@users.noreply.github.com>
This commit is contained in:
Aleksandr Bezobchuk
co-authored by
Marko
Cool Developer
cool-developer
Marko
Cool Developer
cool-developer
parent
77cbb3cd4f
commit
a298594263
@@ -61,6 +61,7 @@ When writing ADRs, follow the same best practices for writing RFCs. When writing
|
||||
* [ADR 060: ABCI 1.0 (Phase I)](adr-060-abci-1.0.md)
|
||||
* [ADR 061: Liquid Staking](./adr-061-liquid-staking.md)
|
||||
* [ADR 070: Un-Ordered Transaction Inclusion](./adr-070-unordered-account.md)
|
||||
* [ADR 065: Store v2](./adr-065-store-v2.md)
|
||||
|
||||
### Proposed
|
||||
|
||||
@@ -86,7 +87,6 @@ When writing ADRs, follow the same best practices for writing RFCs. When writing
|
||||
* [ADR 059: Test Scopes](./adr-059-test-scopes.md)
|
||||
* [ADR 062: Collections State Layer](./adr-062-collections-state-layer.md)
|
||||
* [ADR 063: Core Module API](./adr-063-core-module-api.md)
|
||||
* [ADR 065: Store v2](./adr-065-store-v2.md)
|
||||
* [ADR 067: Simulator v2](./adr-067-simulator-v2.md)
|
||||
* [ADR 069: `x/gov` modularity, multiple choice and optimisic proposals](./adr-069-gov-improvements.md)
|
||||
|
||||
|
||||
@@ -3,10 +3,11 @@
|
||||
## Changelog
|
||||
|
||||
* Feb 14, 2023: Initial Draft (@alexanderbez)
|
||||
* Dec 21, 2023: Updates after implementation (@alexanderbez)
|
||||
|
||||
## Status
|
||||
|
||||
DRAFT
|
||||
ACCEPTED
|
||||
|
||||
## Abstract
|
||||
|
||||
@@ -51,7 +52,7 @@ design of storage in the Cosmos SDK:
|
||||
* Note, we can still allow modules to have their own IAVL `KVStore`, but the
|
||||
IAVL library will need to support the ability to pass a DB instance as an
|
||||
argument to various IAVL APIs.
|
||||
* Since IAVL is responsible for both state storage and commitment, running an
|
||||
* Since IAVL is responsible for both state storage and commitment, running an
|
||||
archive node becomes increasingly expensive as disk space grows exponentially.
|
||||
* As the size of a network increases, various performance bottlenecks start to
|
||||
emerge in many areas such as query performance, network upgrades, state
|
||||
@@ -81,73 +82,73 @@ less intrusive. Specifically, we propose to:
|
||||
* Separate the concerns of state commitment (**SC**), needed for consensus, and
|
||||
state storage (**SS**), needed for state machine and clients.
|
||||
* Reduce layers of abstractions necessary between the RMS and underlying stores.
|
||||
* Provide atomic module store commitments by providing a batch database object
|
||||
to core IAVL APIs.
|
||||
* Reduce complexities in the `CacheKVStore` implementation while also improving
|
||||
performance<sup>[3]</sup>.
|
||||
* Remove unnecessary store types and implementations such as `CacheKVStore`.
|
||||
* Simplify the branching logic.
|
||||
* Ensure the `RootStore` interface remains as lightweight as possible.
|
||||
* Allow application developers to easily swap out SS and SC backends.
|
||||
|
||||
Furthermore, we will keep the IAVL is the backing [commitment](https://cryptography.fandom.com/wiki/Commitment_scheme)
|
||||
store for the time being. While we might not fully settle on the use of IAVL in
|
||||
Furthermore, we will keep IAVL as the default [SC](https://cryptography.fandom.com/wiki/Commitment_scheme)
|
||||
backend for the time being. While we might not fully settle on the use of IAVL in
|
||||
the long term, we do not have strong empirical evidence to suggest a better
|
||||
alternative. Given that the SDK provides interfaces for stores, it should be sufficient
|
||||
to change the backing commitment store in the future should evidence arise to
|
||||
warrant a better alternative. However there is promising work being done to IAVL
|
||||
that should result in significant performance improvement <sup>[1,2]</sup>.
|
||||
|
||||
Note, we will provide applications with the ability to use IAVL v1 and IAVL v2 as
|
||||
either SC backend, with the latter showing extremely promising performance improvements
|
||||
over IAVL v0 and v1, at the cost of a state migration.
|
||||
|
||||
### Separating SS and SC
|
||||
|
||||
By separating SS and SC, it will allow for us to optimize against primary use cases
|
||||
and access patterns to state. Specifically, The SS layer will be responsible for
|
||||
direct access to data in the form of (key, value) pairs, whereas the SC layer (IAVL)
|
||||
direct access to data in the form of (key, value) pairs, whereas the SC layer (e.g. IAVL)
|
||||
will be responsible for committing to data and providing Merkle proofs.
|
||||
|
||||
Note, the underlying physical storage database will be the same between both the
|
||||
SS and SC layers. So to avoid collisions between (key, value) pairs, both layers
|
||||
will be namespaced.
|
||||
|
||||
#### State Commitment (SC)
|
||||
|
||||
Given that the existing solution today acts as both SS and SC, we can simply
|
||||
repurpose it to act solely as the SC layer without any significant changes to
|
||||
access patterns or behavior. In other words, the entire collection of existing
|
||||
IAVL-backed module `KVStore`s will act as the SC layer.
|
||||
A foremost design goal is that SC backends should be easily swappable, i.e. not
|
||||
necessarily IAVL. To this end, the scope of SC has been reduced, it must only:
|
||||
|
||||
However, in order for the SC layer to remain lightweight and not duplicate a
|
||||
majority of the data held in the SS layer, we encourage node operators to keep
|
||||
tight pruning strategies.
|
||||
* Provide a stateful root app hash for height h resulting from applying a batch
|
||||
of key-value set/deletes to height h-1.
|
||||
* Fulfill (though not necessarily provide) historical proofs for all heights < h.
|
||||
* Provide an API for snapshot create/restore to fulfill state sync requests.
|
||||
|
||||
An SC implementation may choose not to provide historical proofs past height h - n (n can be 0)
|
||||
due to the time and space constraints, but since store v2 defines an API for historical
|
||||
proofs there should be at least one configuration of a given SC backend which
|
||||
supports this.
|
||||
|
||||
#### State Storage (SS)
|
||||
|
||||
In the RMS, we will expose a *single* `KVStore` backed by the same physical
|
||||
database that backs the SC layer. This `KVStore` will be explicitly namespaced
|
||||
to avoid collisions and will act as the primary storage for (key, value) pairs.
|
||||
The goal of SS is to provide a modular storage backend, i.e. multiple implementations,
|
||||
to facilitate storing versioned raw key/value pairs in a fast embedded database.
|
||||
The responsibility and functions of SS include the following:
|
||||
|
||||
While we most likely will continue the use of `cosmos-db`, or some local interface,
|
||||
to allow for flexibility and iteration over preferred physical storage backends
|
||||
as research and benchmarking continues. However, we propose to hardcode the use
|
||||
of RocksDB as the primary physical storage backend.
|
||||
* Provided fast and efficient queries for versioned raw key/value pairs
|
||||
* Provide versioned CRUD operations
|
||||
* Provide versioned batching functionality
|
||||
* Provide versioned iteration (forward and reverse) functionality
|
||||
* Provide pruning functionality
|
||||
|
||||
Since the SS layer will be implemented as a `KVStore`, it will support the
|
||||
following functionality:
|
||||
All of the functionality provided by an SS backend should work under a versioned
|
||||
scheme, i.e. a user should be able to get, store, and iterate over keys for the latest
|
||||
and historical versions efficiently and a store key, which is used for name-spacing
|
||||
purposes.
|
||||
|
||||
* Range queries
|
||||
* CRUD operations
|
||||
* Historical queries and versioning
|
||||
* Pruning
|
||||
We propose to have three defaulting SS backends for applications to choose from:
|
||||
|
||||
The RMS will keep track of all buffered writes using a dedicated and internal
|
||||
`MemoryListener` for each `StoreKey`. For each block height, upon `Commit`, the
|
||||
SS layer will write all buffered (key, value) pairs under a [RocksDB user-defined timestamp](https://github.com/facebook/rocksdb/wiki/User-defined-Timestamp-%28Experimental%29) column
|
||||
family using the block height as the timestamp, which is an unsigned integer.
|
||||
This will allow a client to fetch (key, value) pairs at historical and current
|
||||
heights along with making iteration and range queries relatively performant as
|
||||
the timestamp is the key suffix.
|
||||
|
||||
Note, we choose not to use a more general approach of allowing any embedded key/value
|
||||
database, such as LevelDB or PebbleDB, using height key-prefixed keys to
|
||||
effectively version state because most of these databases use variable length
|
||||
keys which would effectively make actions likes iteration and range queries less
|
||||
performant.
|
||||
* RocksDB
|
||||
* CGO based
|
||||
* Usage of User-Defined Timestamps as a built-in versioning mechanism
|
||||
* PebbleDB
|
||||
* Native
|
||||
* Manual implementation of MVCC keys for versioning
|
||||
* SQLite
|
||||
* CGO based
|
||||
* Single table for all state
|
||||
|
||||
Since operators might want pruning strategies to differ in SS compared to SC,
|
||||
e.g. having a very tight pruning strategy in SC while having a looser pruning
|
||||
@@ -165,58 +166,61 @@ otherwise, a snapshot could be triggered on a height that is not available in SC
|
||||
The state sync process should be largely unaffected by the separation of the SC
|
||||
and SS layers. However, if a node syncs via state sync, the SS layer of the node
|
||||
will not have the state synced height available, since the IAVL import process is
|
||||
not setup in way to easily allow direct key/value insertion. A modification of
|
||||
the IAVL import process would be necessary to facilitate having the state sync
|
||||
height available.
|
||||
not setup in way to easily allow direct key/value insertion.
|
||||
|
||||
Note, this is not problematic for the state machine itself because when a query
|
||||
is made, the RMS will automatically direct the query correctly (see [Queries](#queries)).
|
||||
We propose a simple `SnapshotManager` that consumes and produces snapshots. SC
|
||||
backends will be responsible for providing a snapshot of the state at a given
|
||||
height and both SS and SC consume snapshots to restore state.
|
||||
|
||||
#### Queries
|
||||
#### RootStore
|
||||
|
||||
To consolidate the query routing between both the SC and SS layers, we propose to
|
||||
have a notion of a "query router" that is constructed in the RMS. This query router
|
||||
will be supplied to each `KVStore` implementation. The query router will route
|
||||
queries to either the SC layer or the SS layer based on a few parameters. If
|
||||
`prove: true`, then the query must be routed to the SC layer. Otherwise, if the
|
||||
query height is available in the SS layer, the query will be served from the SS
|
||||
layer. Otherwise, we fall back on the SC layer.
|
||||
We will define a `RootStore` interface and default implementation that will be
|
||||
the primary interface for the application to interact with. The `RootStore` will
|
||||
be responsible for housing SS and SC backends. Specifically, a `RootStore` will
|
||||
provide the following functionality:
|
||||
|
||||
If no height is provided, the SS layer will assume the latest height. The SS
|
||||
layer will store a reverse index to lookup `LatestVersion -> timestamp(version)`
|
||||
which is set on `Commit`.
|
||||
* Manage commitment of state (both SS and SC)
|
||||
* Provide modules access to state
|
||||
* Query delegation (i.e. get a value for a <key, height> tuple)
|
||||
* Providing commitment proofs
|
||||
|
||||
#### Store Keys
|
||||
|
||||
Naturally, if a single SC tree is used in all RootStore implementations, then the
|
||||
notion of a store key becomes entirely useless. However, we cannot dictate or
|
||||
predicate how all applications will implement their RooStore (if they choose to).
|
||||
|
||||
Since an app can choose to have multiple SC trees, we need to keep the notion of
|
||||
store keys. Unlike store v1, we represent store keys as simple strings as opposed
|
||||
to concrete types to provide OCAP functionality. The store key strings act to
|
||||
solely provide key prefixing/namespacing functionality for modules.
|
||||
|
||||
#### Proofs
|
||||
|
||||
Since the SS layer is naturally a storage layer only, without any commitments
|
||||
to (key, value) pairs, it cannot provide Merkle proofs to clients during queries.
|
||||
|
||||
Since the pruning strategy against the SC layer is configured by the operator,
|
||||
we can therefore have the RMS route the query SC layer if the version exists and
|
||||
`prove: true`. Otherwise, the query will fall back to the SS layer without a proof.
|
||||
So providing inclusion and exclusion proofs, via a `CommitmentOp` type, will be
|
||||
the responsibility of the SC backend. Retrieving proofs will be done through the
|
||||
a `RootStore`, which will internally route the request to the SC backend.
|
||||
|
||||
We could explore the idea of using state snapshots to rebuild an in-memory IAVL
|
||||
tree in real time against a version closest to the one provided in the query.
|
||||
However, it is not clear what the performance implications will be of this approach.
|
||||
#### Commitment
|
||||
|
||||
### Atomic Commitment
|
||||
Before ABCI 2.0, specifically before `FinalizeBlock` was introduced, the flow of state
|
||||
commitment in BaseApp was defined by writes being written to the `RootMultiStore`
|
||||
and then a single Commit call on the `RootMultiStore` during the ABCI Commit method.
|
||||
|
||||
We propose to modify the existing IAVL APIs to accept a batch DB object instead
|
||||
of relying on an internal batch object in `nodeDB`. Since each underlying IAVL
|
||||
`KVStore` shares the same DB in the SC layer, this will allow commits to be
|
||||
atomic.
|
||||
With the advent of ABCI 2.0, the commitment flow has now changed to `WorkingHash` being
|
||||
called during `FinalizeBlock` and then Commit being called on ABCI Commit. Note,
|
||||
`WorkingHash` does not actually commit state to disk, but rather computes an
|
||||
uncommitted work-in-progress hash, which is returned in `FinalizeBlock`. Then,
|
||||
during the ABCI Commit phase, the state is finally flushed to disk.
|
||||
|
||||
Specifically, we propose to:
|
||||
|
||||
* Remove the `dbm.Batch` field from `nodeDB`
|
||||
* Update the `SaveVersion` method of the `MutableTree` IAVL type to accept a batch object
|
||||
* Update the `Commit` method of the `CommitKVStore` interface to accept a batch object
|
||||
* Create a batch object in the RMS during `Commit` and pass this object to each
|
||||
`KVStore`
|
||||
* Write the database batch after all stores have committed successfully
|
||||
|
||||
Note, this will require IAVL to be updated to not rely or assume on any batch
|
||||
being present during `SaveVersion`.
|
||||
In store v2, we must respect this flow. Thus, a caller is expected to call `WorkingHash`
|
||||
during `FinalizeBlock`, which takes the latest changeset in the `RootStore`,
|
||||
writes that to the SC tree in a single batch and returns a hash. Finally, during
|
||||
the ABCI Commit phase, we call `Commit` on the `RootStore` which commits the SC
|
||||
tree and flushes the changeset to the SS backend.
|
||||
|
||||
## Consequences
|
||||
|
||||
@@ -252,6 +256,11 @@ be broken or modified.
|
||||
|
||||
### Neutral
|
||||
|
||||
* Removal of OCAP-based store keys in favor of simple strings for state retrieval
|
||||
and name-spacing. We consider this neutral as removal of OCAP functionality can
|
||||
be seen as a negative, however, we're simply moving the OCAP functionality upstream
|
||||
to the KVStore service. The SS and SC layers shouldn't have to concern themselves
|
||||
with OCAP responsibilities.
|
||||
* Keeping IAVL as the primary commitment data structure, although drastic
|
||||
performance improvements are being made
|
||||
|
||||
@@ -275,16 +284,9 @@ commitment proofs for historical state. While solutions can be devised such as
|
||||
rebuilding trees on the fly based on state snapshots, it is not clear what the
|
||||
performance implications are for such solutions.
|
||||
|
||||
### Physical DB Backends
|
||||
|
||||
This ADR proposes usage of RocksDB to utilize user-defined timestamps as a
|
||||
versioning mechanism. However, other physical DB backends are available that may
|
||||
offer alternative ways to implement versioning while also providing performance
|
||||
improvements over RocksDB. E.g. PebbleDB supports MVCC timestamps as well, but
|
||||
we'll need to explore how PebbleDB handles compaction and state growth over time.
|
||||
|
||||
## References
|
||||
|
||||
* [1] https://github.com/cosmos/iavl/pull/676
|
||||
* [2] https://github.com/cosmos/iavl/pull/664
|
||||
* [3] https://github.com/cosmos/cosmos-sdk/issues/14990
|
||||
* [4] https://docs.google.com/document/d/e/2PACX-1vSCFfXZm2vsRsACOPoxGqysMaUg7jY833LwR3YyjA1S3FNHfXRiJor-qLjzx833TavLXLPSIcFZJhyh/pub
|
||||
|
||||
Reference in New Issue
Block a user