From 8b83a2e29d3590324b9023f0b903f27bcbbdf031 Mon Sep 17 00:00:00 2001 From: Aleksandr Bezobchuk Date: Fri, 23 Feb 2024 08:57:57 -0800 Subject: [PATCH] docs(store/v2): update store v2 docs (#19502) Co-authored-by: cool-developer <51834436+cool-develope@users.noreply.github.com> --- store/README.md | 31 ++++++++ store/commitment/README.md | 46 +++++++++++- store/storage/README.md | 109 +++++++++++++++++++++++++++- store/storage/storage_bench_test.go | 4 + 4 files changed, 188 insertions(+), 2 deletions(-) create mode 100644 store/README.md diff --git a/store/README.md b/store/README.md new file mode 100644 index 0000000000..70f8dfb9a6 --- /dev/null +++ b/store/README.md @@ -0,0 +1,31 @@ +# Store + +The `store` package contains the implementation of store/v2, which is the SDK's +abstraction around managing historical and committed state. See [ADR-065](../docs/architecture/adr-065-store-v2.md) +and [Store v2 Design](https://docs.google.com/document/d/1l6uXIjTPHOOWM5N4sUUmUfCZvePoa5SNfIEtmgvgQSU/edit#heading=h.nz8dqy6wa4g1) for a high-level overview of the design and rationale. + +## Migration + + + +## Pruning + +The `root.Store` is NOT responsible for pruning. Rather, pruning is the responsibility +of the underlying SS and SC layers. This means pruning can be implementation specific, +such as being synchronous or asynchronous. + +## Usage + +The `store` package contains a `root.Store` type which is intended to act as an +abstraction layer around it's two primary constituent components - state storage (SS) +and state commitment (SC). It acts as the main entry point into storage for an +application to use in server/v2. Through `root.Store`, an application can query +and iterate over both current and historical data, commit new state, perform state +sync, and fetch commitment proofs. + +A `root.Store` is intended to be initialized with already constructed SS and SC +backends (see relevant package documentation for instantiation details). Note, +from the perspective of `root.Store`, there is no notion of multi or single tree/store, +rather these are implementation details of SS and SC. For SS, we utilize store keys +to namespace raw key/value pairs. For SC, we utilize an abstraction, `commitment.CommitStore`, +to map store keys to a commitment trees. diff --git a/store/commitment/README.md b/store/commitment/README.md index 4843ef0db5..1e48cc057a 100644 --- a/store/commitment/README.md +++ b/store/commitment/README.md @@ -1,3 +1,47 @@ # State Commitment (SC) -TODO +The `commitment` package contains the state commitment (SC) implementation. +Specifically, it contains an IAVL v1 implementation of SC and the necessary types +and abstractions to support other SC backends, as well as supporting general integration +into store/v2, specifically the `RootStore` type. + +A foremost design goal is that SC backends should be easily swappable, i.e. not +necessarily IAVL. To this end, the scope of SC has been reduced, it must only: + +* Provide a stateful root app hash for height h resulting from applying a batch + of key-value set/deletes to height h-1. +* Fulfill (though not necessarily provide) historical proofs for all heights < `h`. +* Provide an API for snapshot create/restore to fulfill state sync requests. + +Notably, SC is not required to provide key iteration or value retrieval for either +queries or state machine execution, this now being the responsibility of state +storage. + +An SC implementation may choose not to provide historical proofs past height `h - n` +(`n` can be 0) due to the time and space constraints, but since store/v2 defines +an API for historical proofs there should be at least one configuration of a +given SC backend which supports this. + +## Benchmarks + +See this [section](https://docs.google.com/document/d/1l6uXIjTPHOOWM5N4sUUmUfCZvePoa5SNfIEtmgvgQSU/edit#heading=h.7l0i621y5vgm) for specifics on SC benchmarks on various implementations. + +## Pruning + + + +## State Sync + +State commitment (SC) does not have a direct notion of state sync. Rather, +`snapshots.Manager` is responsible for creating and restoring snapshots of the +entire state. The `snapshots.Manager` has a `CommitSnapshotter` field which is +fulfilled by the `CommitStore` type, specifically it implements the `Snapshot` +and `Restore` methods. + +## Usage + +Similar to the `storage` package, the `commitment` package is designed to be used +in a broader store implementation, i.e. it fulfills the role of the SC backend. +Specifically, it provides a `CommitStore` type which accepts a `store.RawDB` and +a mapping from store key, a string meant to represent a single module, to a `Tree`, +which reflects the commitment structure. diff --git a/store/storage/README.md b/store/storage/README.md index c82c3f3a2f..3a05d77864 100644 --- a/store/storage/README.md +++ b/store/storage/README.md @@ -1,3 +1,110 @@ # State Storage (SS) -TODO +The `storage` package contains the state storage (SS) implementation. Specifically, +it contains RocksDB, PebbleDB, and SQLite (Btree) backend implementations of the +`VersionedDatabase` interface. + +The goal of SS is to provide a modular storage backend, i.e. multiple implementations, +to facilitate storing versioned raw key/value pairs in a fast embedded database, +although an embedded database is not required, i.e. you could use a replicated +RDBMS system. + +The responsibility and functions of SS include the following: + +* Provide fast and efficient queries for versioned raw key/value pairs +* Provide versioned CRUD operations +* Provide versioned batching functionality +* Provide versioned iteration (forward and reverse) functionality +* Provide pruning functionality + +All of the functionality provided by an SS backend should work under a versioned +scheme, i.e. a user should be able to get, store, and iterate over keys for the +latest and historical versions efficiently. + +## Backends + +### RocksDB + +The RocksDB implementation is a CGO-based SS implementation. It fully supports +the `VersionedDatabase` API and is arguably the most efficient implementation. It +also supports versioning out-of-the-box using User-defined Timestamps in +ColumnFamilies (CF). However, it requires the CGO dependency which can complicate +an app’s build process. + +### PebbleDB + +The PebbleDB implementation is a native Go SS implementation that is primarily an +alternative to RocksDB. Since it does not support CF, results in the fact that we +need to implement versioning (MVCC) ourselves. This comes with added implementation +complexity and potential performance overhead. However, it is a pure Go implementation +and does not require CGO. + +### SQLite (Btree) + +The SQLite implementation is another CGO-based SS implementation. It fully supports +the `VersionedDatabase` API. The implementation is relatively straightforward and +easy to understand as it’s entirely SQL-based. However, benchmarks show that this +options is least performant, even for reads. This SS backend has a lot of promise, +but needs more benchmarking and potential SQL optimizations, like dedicated tables +for certain aspects of state, e.g. latest state, to be extremely performant. + +## Benchmarks + +Benchmarks for basic operations on all supported native SS implementations can +be found in `store/storage/storage_bench_test.go`. + +At the time of writing, the following benchmarks were performed: + +```shell +name time/op +Get/backend_rocksdb_versiondb_opts-10 7.41µs ± 0% +Get/backend_pebbledb_default_opts-10 6.17µs ± 0% +Get/backend_btree_sqlite-10 29.1µs ± 0% +ApplyChangeset/backend_pebbledb_default_opts-10 5.73ms ± 0% +ApplyChangeset/backend_btree_sqlite-10 56.9ms ± 0% +ApplyChangeset/backend_rocksdb_versiondb_opts-10 4.07ms ± 0% +Iterate/backend_pebbledb_default_opts-10 1.04s ± 0% +Iterate/backend_btree_sqlite-10 1.59s ± 0% +Iterate/backend_rocksdb_versiondb_opts-10 778ms ± 0% +``` + +## Pruning + +Pruning is an implementation and responsibility of the underlying SS backend. +Specifically, the `StorageStore` accepts `store.PruneOptions` which defines the +pruning configuration. During `ApplyChangeset`, the `StorageStore` will check if +pruning should occur based on the current height being committed. If so, it will +delegate a `Prune` call on the underlying SS backend, which can be defined specific +to the implementation, e.g. asynchronous or synchronous. + + +## State Sync + +State storage (SS) does not have a direct notion of state sync. Rather, `snapshots.Manager` +is responsible for creating and restoring snapshots of the entire state. The +`snapshots.Manager` has a `StorageSnapshotter` field which is fulfilled by the +`StorageStore` type, specifically it implements the `Restore` method. The `Restore` +method reads off of a provided channel and writes key/value pairs directly to a +batch object which is committed to the underlying SS engine. + +## Non-Consensus Data + + + +## Usage + +An SS backend is meant to be used within a broader store implementation, as it +only stores data for direct and historical query purposes. We define a `Database` +interface in the `storage` package which is mean to be represent a `VersionedDatabase` +with only the necessary methods. The `StorageStore` interface is meant to wrap or +accept this `Database` type, e.g. RocksDB. + +The `StorageStore` interface is an abstraction or wrapper around the backing SS +engine can be seen as the the main entry point to using SS. + +Higher up the stack, there should exist a `root.Store` implementation. The `root.Store` +is meant to encapsulate both an SS backend and an SC backend. The SS backend is +defined by this `StorageStore` implementation. + +In short, initialize your SS engine of choice and then provide that to `NewStorageStore` +which will further be provided to `root.Store` as the SS backend. diff --git a/store/storage/storage_bench_test.go b/store/storage/storage_bench_test.go index b345398d1b..b66343d215 100644 --- a/store/storage/storage_bench_test.go +++ b/store/storage/storage_bench_test.go @@ -32,6 +32,10 @@ var ( }, "pebbledb_default_opts": func(dataDir string) (store.VersionedDatabase, error) { db, err := pebbledb.New(dataDir) + if err == nil && db != nil { + db.SetSync(false) + } + return storage.NewStorageStore(db, nil, log.NewNopLogger()), err }, "btree_sqlite": func(dataDir string) (store.VersionedDatabase, error) {