From c3edbb22cab8678c35e21fe0253919996b780c01 Mon Sep 17 00:00:00 2001 From: Aaron Craelius Date: Wed, 4 May 2022 10:59:08 -0400 Subject: [PATCH] docs: add ADR 055 ORM (#11807) ## Description Adds an ADR for the `orm` module which is already implemented. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) --- docs/architecture/README.md | 1 + docs/architecture/adr-055-orm.md | 111 +++++++++++++++++++++++++++++++ 2 files changed, 112 insertions(+) create mode 100644 docs/architecture/adr-055-orm.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md index fecaa118be..4257d1e87b 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -56,6 +56,7 @@ When writing ADRs, follow the same best practices for writing RFCs. When writing * [ADR 029: Fee Grant Module](./adr-029-fee-grant-module.md) * [ADR 030: Message Authorization Module](./adr-030-authz-module.md) * [ADR 031: Protobuf Msg Services](./adr-031-msg-service.md) +* [ADR 055: ORM](./adr-055-orm.md) ### Proposed diff --git a/docs/architecture/adr-055-orm.md b/docs/architecture/adr-055-orm.md new file mode 100644 index 0000000000..71a7595260 --- /dev/null +++ b/docs/architecture/adr-055-orm.md @@ -0,0 +1,111 @@ +# ADR 055: ORM + +## Changelog + +* 2022-04-27: First draft + +## Status + +ACCEPTED Implemented + +## Abstract + +In order to make it easier for developers to build Cosmos SDK modules and for clients to query, index and verify proofs +against state data, we have implemented an ORM (object-relational mapping) layer for the Cosmos SDK. + +## Context + +Historically modules in the Cosmos SDK have always used the key-value store directly and created various handwritten +functions for managing key format as well as constructing secondary indexes. This consumes a significant amount of +time when building a module and is error-prone. Because key formats are non-standard, sometimes poorly documented, +and subject to change, it is hard for clients to generically index, query and verify merkle proofs against state data. + +The known first instance of an "ORM" in the Cosmos ecosystem was in [weave](https://github.com/iov-one/weave/tree/master/orm). +A later version was built for [regen-ledger](https://github.com/regen-network/regen-ledger/tree/157181f955823149e1825263a317ad8e16096da4/orm) for +use in the group module and later [ported to the SDK](https://github.com/cosmos/cosmos-sdk/tree/35d3312c3be306591fcba39892223f1244c8d108/x/group/internal/orm) +just for that purpose. + +While these earlier designs made it significantly easier to write state machines, they still required a lot of manual +configuration, didn't expose state format directly to clients, and were limited in their support of different types +of index keys, composite keys, and range queries. + +Discussions about the design continued in https://github.com/cosmos/cosmos-sdk/discussions/9156 and more +sophisticated proofs of concept were created in https://github.com/allinbits/cosmos-sdk-poc/tree/master/runtime/orm +and https://github.com/cosmos/cosmos-sdk/pull/10454. + +## Decision + +These prior efforts culminated in the creation of the Cosmos SDK `orm` go module which uses protobuf annotations +for specifying ORM table definitions. This ORM is based on the new `google.golang.org/protobuf/reflect/protoreflect` +API and supports: +* sorted indexes for all simple protobuf types (except `bytes`, `enum`, `float`, `double`) as well as `Timestamp` and `Duration` +* unsorted `bytes` and `enum` indexes +* composite primary and secondary keys +* unique indexes +* auto-incrementing `uint64` primary keys +* complex prefix and range queries +* paginated queries +* complete logical decoding of KV-store data + +Almost all the information needed to decode state directly is specified in .proto files. Each table definition specifies +an ID which is unique per .proto file and each index within a table is unique within that table. Clients then only need +to know the name of a module and the prefix ORM data for a specific .proto file within that module in order to decode +state data directly. This additional information will be exposed directly through app configs which will be explained +in a future ADR related to app wiring. + +The ORM makes optimizations around storage space by not repeating values in the primary key in the key value +when storing primary key records. For example, if the object `{"a":0,"b":1}` has the primary key `a`, it will +be stored in the key value store as `Key: '0', Value: {"b":1}` (with more efficient protobuf binary encoding). +Also, the generated code from https://github.com/cosmos/cosmos-proto does optimizations around the +`google.golang.org/protobuf/reflect/protoreflect` API to improve performance. + +A code generator is included with the ORM which creates type safe wrappers around the ORM's dynamic `Table` +implementation and is the recommended way for modules to use the ORM. + +The ORM tests provide a simplified bank module demonstration which illustrates: +- [ORM proto options](https://github.com/cosmos/cosmos-sdk/blob/0d846ae2f0424b2eb640f6679a703b52d407813d/orm/internal/testpb/bank.proto) +- [Generated Code](https://github.com/cosmos/cosmos-sdk/blob/0d846ae2f0424b2eb640f6679a703b52d407813d/orm/internal/testpb/bank.cosmos_orm.go) +- [Example Usage in a Module Keeper](https://github.com/cosmos/cosmos-sdk/blob/0d846ae2f0424b2eb640f6679a703b52d407813d/orm/model/ormdb/module_test.go) + +## Consequences + +### Backwards Compatibility + +State machine code that adopts the ORM will need migrations as the state layout is generally backwards incompatible. +These state machines will also need to migrate to https://github.com/cosmos/cosmos-proto at least for state data. + +### Positive + +* easier to build modules +* easier to add secondary indexes to state +* possible to write a generic indexer for ORM state +* easier to write clients that do state proofs +* possible to automatically write query layers rather than needing to manually implement gRPC queries + +### Negative + +* worse performance than handwritten keys (for now). See [Further Discussions](#further-discussions) +for potential improvements + +### Neutral + +## Further Discussions + +Further discussions will happen within the Cosmos SDK Framework Working Group. Current planned and ongoing work includes: +* automatically generate client-facing query layer +* client-side query libraries that transparently verify light client proofs +* index ORM data to SQL databases +* improve performance by: + * optimizing existing reflection based code to avoid unnecessary gets when doing deletes & updates of simple tables + * more sophisticated code generation such as making fast path reflection even faster (avoiding `switch` statements), + or even fully generating code that equals handwritten performance + + +## References + +* https://github.com/iov-one/weave/tree/master/orm). +* https://github.com/regen-network/regen-ledger/tree/157181f955823149e1825263a317ad8e16096da4/orm +* https://github.com/cosmos/cosmos-sdk/tree/35d3312c3be306591fcba39892223f1244c8d108/x/group/internal/orm +* https://github.com/cosmos/cosmos-sdk/discussions/9156 +* https://github.com/allinbits/cosmos-sdk-poc/tree/master/runtime/orm +* https://github.com/cosmos/cosmos-sdk/pull/10454 \ No newline at end of file