From b853d3fe8fc203267adb20aba26ea89d7a4490dc Mon Sep 17 00:00:00 2001 From: Aleksandr Bezobchuk Date: Fri, 23 Sep 2022 12:46:28 -0400 Subject: [PATCH] chore: store package specification (#13209) * updates * updates * updates * updates * updates * updates * updates * updates * updates * updates * updates * updates * updates * updates * updates * updates * updates Co-authored-by: Marko --- docs/spec/store/README.md | 219 ++++++++++++++++++++++++++++++++++++++ store/README.md | 129 ---------------------- 2 files changed, 219 insertions(+), 129 deletions(-) delete mode 100644 store/README.md diff --git a/docs/spec/store/README.md b/docs/spec/store/README.md index e69de29bb2..cd9d23fc45 100644 --- a/docs/spec/store/README.md +++ b/docs/spec/store/README.md @@ -0,0 +1,219 @@ +# Store + +The store package defines the interfaces, types and abstractions for Cosmos SDK +modules to read and write to Merkleized state within a Cosmos SDK application. +The store package provides many primitives for developers to use in order to +work with both state storage and state commitment. Below we describe the various +abstractions. + +## Types + +### `Store` + +The bulk of the store interfaces are defined [here](https://github.com/cosmos/cosmos-sdk/blob/main/store/types/store.go), +where the base primitive interface, for which other interfaces build off of, is +the `Store` type. The `Store` interface defines the ability to tell the type of +the implementing store and the ability to cache wrap via the `CacheWrapper` interface. + +### `CacheWrapper` & `CacheWrap` + +One of the most important features a store has the ability to perform is the +ability to cache wrap. Cache wrapping is essentially the underlying store wrapping +itself within another store type that performs caching for both reads and writes +with the ability to flush writes via `Write()`. + +### `KVStore` & `CacheKVStore` + +One of the most important interfaces that both developers and modules interface +with, which also provides the basis of most state storage and commitment operations, +is the `KVStore`. The `KVStore` interface provides basic CRUD abilities and +prefix-based iteration, including reverse iteration. + +Typically, each module has it's own dedicated `KVStore` instance, which it can +get access to via the `sdk.Context` and the use of a pointer-based named key -- +`KVStoreKey`. The `KVStoreKey` provides pseudo-OCAP. How a exactly a `KVStoreKey` +maps to a `KVStore` will be illustrated below through the `CommitMultiStore`. + +Note, a `KVStore` cannot directly commit state. Instead, a `KVStore` can be wrapped +by a `CacheKVStore` which extends a `KVStore` and provides the ability for the +caller to execute `Write()` which commits state to the underlying state storage. +Note, this doesn't actually flush writes to disk as writes are held in memory +until `Commit()` is called on the `CommitMultiStore`. + +### `CommitMultiStore` + +The `CommitMultiStore` interface exposes the the top-level interface that is used +to manage state commitment and storage by an SDK application and abstracts the +concept of multiple `KVStore`s which are used by multiple modules. Specifically, +it supports the following high-level primitives: + +* Allows for a caller to retrieve a `KVStore` by providing a `KVStoreKey`. +* Exposes pruning mechanisms to remove state pinned against a specific height/version + in the past. +* Allows for loading state storage at a particular height/version in the past to + provide current head and historical queries. +* Provides the ability to rollback state to a previous height/version. +* Provides the ability to to load state storage at a particular height/version + while also performing store upgrades, which are used during live hard-fork + application state migrations. +* Provides the ability to commit all current accumulated state to disk and performs + Merkle commitment. + +## Implementation Details + +While there are many interfaces that the `store` package provides, there is +typically a core implementation for each main interface that modules and +developers interact with that are defined in the Cosmos SDK. + +### `iavl.Store` + +The `iavl.Store` provides the core implementation for state storage and commitment +by implementing the following interfaces: + +* `KVStore` +* `CommitStore` +* `CommitKVStore` +* `Queryable` +* `StoreWithInitialVersion` + +It allows for all CRUD operations to be performed along with allowing current +and historical state queries, prefix iteration, and state commitment along with +Merkle proof operations. The `iavl.Store` also provides the ability to remove +historical state from the state commitment layer. + +An overview of the IAVL implementation can be found [here](https://github.com/cosmos/iavl/blob/master/docs/overview.md). +It is important to note that the IAVL store provides both state commitment and +logical storage operations, which comes with drawbacks as there are various +performance impacts, some of which are very drastic, when it comes to the +operations mentioned above. + +When dealing with state management in modules and clients, the Cosmos SDK provides +various layers of abstractions or "store wrapping", where the `iavl.Store` is the +bottom most layer. When requesting a store to perform reads or writes in a module, +the typical abstraction layer in order is defined as follows: + +```text +iavl.Store <- cachekv.Store <- gaskv.Store <- cachemulti.Store <- rootmulti.Store +``` + +### `cachekv.Store` + +The `cachekv.Store` store wraps an underlying `KVStore`, typically a `iavl.Store` +and contains an in-memory cache for storing pending writes to underlying `KVStore`. +`Set` and `Delete` calls are executed on the in-memory cache, whereas `Has` calls +are proxied to the underlying `KVStore`. + +One of the most important calls to a `cachekv.Store` is `Write()`, which ensures +that key-value pairs are written to the underlying `KVStore` in a deterministic +and ordered manner by sorting the keys first. The store keeps track of "dirty" +keys and uses these to determine what keys to sort. In addition, it also keeps +track of deleted keys and ensures these are also removed from the underlying +`KVStore`. + +The `cachekv.Store` also provides the ability to perform iteration and reverse +iteration. Iteration is performed through the `cacheMergeIterator` type and uses +both the dirty cache and underlying `KVStore` to iterate over key-value pairs. + +Note, all calls to CRUD and iteration operations on a `cachekv.Store` are thread-safe. + +### `gaskv.Store` + +The `gaskv.Store` store provides a simple implementation of a `KVStore`. +Specifically, it just wraps an existing `KVStore`, such as a cache-wrapped +`iavl.Store`, and incurs configurable gas costs for CRUD operations via +`ConsumeGas()` calls defined on the `GasMeter` which exists in a `sdk.Context` +and then proxies the underlying CRUD call to the underlying store. Note, the +`GasMeter` is reset on each block. + +### `cachemulti.Store` & `rootmulti.Store` + +The `rootmulti.Store` acts as an abstraction around a series of stores. Namely, +it implements the `CommitMultiStore` an `Queryable` interfaces. Through the +`rootmulti.Store`, an SDK module can request access to a `KVStore` to perform +state CRUD operations and queries by holding access to a unique `KVStoreKey`. + +The `rootmulti.Store` ensures these queries and state operations are performed +through cached-wrapped instances of `cachekv.Store` which is described above. The +`rootmulti.Store` implementation is also responsible for committing all accumulated +state from each `KVStore` to disk and returning an application state Merkle root. + +Queries can be performed to return state data along with associated state +commitment proofs for both previous heights/versions and the current state root. +Queries are routed based on store name, i.e. a module, along with other parameters +which are defined in `abci.RequestQuery`. + +The `rootmulti.Store` also provides primitives for pruning data at a given +height/version from state storage. When a height is committed, the `rootmulti.Store` +will determine if other previous heights should be considered for removal based +on the operator's pruning settings defined by `PruningOptions`, which defines +how many recent versions to keep on disk and the interval at which to remove +"staged" pruned heights from disk. During each interval, the staged heights are +removed from each `KVStore`. Note, it is up to the underlying `KVStore` +implementation to determine how pruning is actually performed. The `PruningOptions` +are defined as follows: + +```golang +type PruningOptions struct { + // KeepRecent defines how many recent heights to keep on disk. + KeepRecent uint64 + + // Interval defines when the pruned heights are removed from disk. + Interval uint64 + + // Strategy defines the kind of pruning strategy. See below for more information on each. + Strategy PruningStrategy +} +``` + +The Cosmos SDK defines a preset number of pruning "strategies": `default`, `everything` +`nothing`, and `custom`. + +It is important to note that the `rootmulti.Store` considers each `KVStore` as a +separate logical store. In other words, they do not share a Merkle tree or +comparable data structure. This means that when state is committed via +`rootmulti.Store`, each store is committed in sequence and thus is not atomic. + +In terms of store construction and wiring, each Cosmos SDK application contains +a `BaseApp` instance which internally has a reference to a `CommitMultiStore` +that is implemented by a `rootmulti.Store`. The application then registers one or +more `KVStoreKey` that pertain to a unique module and thus a `KVStore`. Through +the use of an `sdk.Context` and a `KVStoreKey`, each module can get direct access +to it's respective `KVStore` instance. + +Example: + +```golang +func NewApp(...) Application { + // ... + + bApp := baseapp.NewBaseApp(appName, logger, db, txConfig.TxDecoder(), baseAppOptions...) + bApp.SetCommitMultiStoreTracer(traceStore) + bApp.SetVersion(version.Version) + bApp.SetInterfaceRegistry(interfaceRegistry) + + // ... + + keys := sdk.NewKVStoreKeys(...) + transientKeys := sdk.NewTransientStoreKeys(...) + memKeys := sdk.NewMemoryStoreKeys(...) + + // ... + + // initialize stores + app.MountKVStores(keys) + app.MountTransientStores(transientKeys) + app.MountMemoryStores(memKeys) + + // ... +} +``` + +The `rootmulti.Store` itself can be cache-wrapped which returns an instance of a +`cachemulti.Store`. For each block, `BaseApp` ensures that the proper abstractions +are created on the `CommitMultiStore`, i.e. ensuring that the `rootmulti.Store` +is cached-wrapped and uses the resulting `cachemulti.Store` to be set on the +`sdk.Context` which is then used for block and transaction execution. As a result, +all state mutations due to block and transaction execution are actually held +ephemerally until `Commit()` is called by the ABCI client. This concept is further +expanded upon when the AnteHandler is executed per transaction to ensure state +is not committed for transactions that failed CheckTx. diff --git a/store/README.md b/store/README.md deleted file mode 100644 index 115f9816be..0000000000 --- a/store/README.md +++ /dev/null @@ -1,129 +0,0 @@ -# Store - -## CacheKV - -`cachekv.Store` is a wrapper `KVStore` which provides buffered writing / cached reading functionalities over the underlying `KVStore`. - -```go -type Store struct { - cache map[string]cValue - parent types.KVStore -} -``` - -### Get - -`Store.Get()` checks `Store.cache` first in order to find if there is any cached value associated with the key. If the value exists, the function returns it. If not, the function calls `Store.parent.Get()`, sets the key-value pair in the `Store.cache`, and returns it. - -### Set - -`Store.Set()` sets the key-value pair to the `Store.cache`. `cValue` has the field `dirty bool` which indicates whether the cached value is different from the underlying value. When `Store.Set()` cache new pair, the `cValue.dirty` is set true so when `Store.Write()` is called it can be written to the underlying store. - -### Iterator - -`Store.Iterator()` have to traverse on both caches items and the original items. In `Store.iterator()`, two iterators are generated for each of them, and merged. `memIterator` is essentially a slice of the `KVPair`s, used for cached items. `mergeIterator` is a combination of two iterators, where traverse happens ordered on both iterators. - -## CacheMulti - -`cachemulti.Store` is a wrapper `MultiStore` which provides buffered writing / cached reading functionalities over the underlying `MutliStore` - -```go -type Store struct { - db types.CacheKVStore - stores map[types.StoreKey] types.CacheWrap -} -``` - -`cachemulti.Store` branches all substores in its constructor and hold them in `Store.stores`. `Store.GetKVStore()` returns the store from `Store.stores`, and `Store.Write()` recursively calls `CacheWrap.Write()` on the substores. - -## DBAdapter - -`dbadapter.Store` is a adapter for `dbm.DB` making it fulfilling the `KVStore` interface. - -```go -type Store struct { - dbm.DB -} -``` - -`dbadapter.Store` embeds `dbm.DB`, so most of the `KVStore` interface functions are implemented. The other functions(mostly miscellaneous) are manually implemented. - -## IAVL - -`iavl.Store` is a base-layer self-balancing merkle tree. It is guaranteed that - -1. Get & set operations are `O(log n)`, where `n` is the number of elements in the tree -2. Iteration efficiently returns the sorted elements within the range -3. Each tree version is immutable and can be retrieved even after a commit(depending on the pruning settings) - -Specification and implementation of IAVL tree can be found in https://github.com/cosmos/iavl. - -## GasKV - -`gaskv.Store` is a wrapper `KVStore` which provides gas consuming functionalities over the underlying `KVStore`. - -```go -type Store struct { - gasMeter types.GasMeter - gasConfig types.GasConfig - parent types.KVStore -} -``` - -When each `KVStore` methods are called, `gaskv.Store` automatically consumes appropriate amount of gas depending on the `Store.gasConfig`. - -## Prefix - -`prefix.Store` is a wrapper `KVStore` which provides automatic key-prefixing functionalities over the underlying `KVStore`. - -```go -type Store struct { - parent types.KVStore - prefix []byte -} -``` - -When `Store.{Get, Set}()` is called, the store forwards the call to its parent, with the key prefixed with the `Store.prefix`. - -When `Store.Iterator()` is called, it does not simply prefix the `Store.prefix`, since it does not work as intended. In that case, some of the elements are traversed even they are not starting with the prefix. - -## RootMulti - -`rootmulti.Store` is a base-layer `MultiStore` where multiple `KVStore` can be mounted on it and retrieved via object-capability keys. The keys are memory addresses, so it is impossible to forge the key unless an object is a valid owner(or a receiver) of the key, according to the object capability principles. - -## TraceKV - -`tracekv.Store` is a wrapper `KVStore` which provides operation tracing functionalities over the underlying `KVStore`. - -```go -type Store struct { - parent types.KVStore - writer io.Writer - context types.TraceContext -} -``` - -When each `KVStore` methods are called, `tracekv.Store` automatically logs `traceOperation` to the `Store.writer`. - -```go -type traceOperation struct { - Operation operation - Key string - Value string - Metadata map[string]interface{} -} -``` - -`traceOperation.Metadata` is filled with `Store.context` when it is not nil. `TraceContext` is a `map[string]interface{}`. - -## Transient - -`transient.Store` is a base-layer `KVStore` which is automatically discarded at the end of the block. - -```go -type Store struct { - dbadapter.Store -} -``` - -`Store.Store` is a `dbadapter.Store` with a `dbm.NewMemDB()`. All `KVStore` methods are reused. When `Store.Commit()` is called, new `dbadapter.Store` is assigned, discarding previous reference and making it garbage collected.