diff --git a/docs/architecture/README.md b/docs/architecture/README.md index a3446ce1a2..65e15c4920 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -81,9 +81,10 @@ When writing ADRs, follow the same best practices for writing RFCs. When writing * [ADR 046: Module Params](./adr-046-module-params.md) * [ADR 057: App Wiring Part I](./adr-057-app-wiring-1.md) * [ADR 059: Test Scopes](./adr-059-test-scopes.md) +* [ADR 060: ABCI++ (Phase I)](./adr-060-abci%2B%2B-phase-i.md) ### Draft -- [ADR 044: Guidelines for Updating Protobuf Definitions](./adr-044-protobuf-updates-guidelines.md) -- [ADR 047: Extend Upgrade Plan](./adr-047-extend-upgrade-plan.md) -- [ADR 053: Go Module Refactoring](./adr-053-go-module-refactoring.md) +* [ADR 044: Guidelines for Updating Protobuf Definitions](./adr-044-protobuf-updates-guidelines.md) +* [ADR 047: Extend Upgrade Plan](./adr-047-extend-upgrade-plan.md) +* [ADR 053: Go Module Refactoring](./adr-053-go-module-refactoring.md) diff --git a/docs/architecture/adr-060-abci++-phase-i.md b/docs/architecture/adr-060-abci++-phase-i.md new file mode 100644 index 0000000000..f463222a33 --- /dev/null +++ b/docs/architecture/adr-060-abci++-phase-i.md @@ -0,0 +1,251 @@ +# ADR 60: ABCI++ (Phase I) + +## Changelog + +* 2022-08-10: Initial Draft (@alexanderbez, @marbar3778) + +## Status + +PROPOSED + +## Abstract + +This ADR describes the initial adoption of [ABCI++](https://github.com/tendermint/tendermint/blob/master/spec/abci%2B%2B/README.md), +the next evolution of ABCI, within the Cosmos SDK. ABCI++ aims to provide +application developers with more flexibility and control over application and +consensus semantics, e.g. in-application mempools, in-process oracles, and +order-book style matching engines. + +## Context + +Tendermint will release ABCI++ in phases. Notably, at the time of this writing, +Tendermint is releasing v0.37.0 which will include `PrepareProposal` and `ProcessProposal`. + +The `PrepareProposal` ABCI method is concerned with a block proposer requesting +the application to evaluate a series of transactions to be included in the next +block, defined as a slice of `TxRecord` objects. The application can either +accept, reject, or completely ignore some or all of these transactions. This is +an important consideration to make as the application can essentially define and +control its own mempool allowing it to define sophisticated transaction priority +and filtering mechanisms, by completely ignoring the `TxRecords` Tendermint +sends it, favoring its own transactions. This essentially means that the Tendermint +mempool acts more like a gossip data structure. + +The second ABCI method, `ProcessProposal`, is used to process the block proposer's +proposal as defined by `PrepareProposal`. This ABCI method requests that the +application evaluate the entire proposed block for validity. + +It is important to note that in this phase of ABCI++ integration, the application +is NOT responsible for locking semantics -- Tendermint will still be responsible +for that. In the future, however, the application will be responsible for locking, +which allows for parallel execution possibilities. + +## Decision + +We will integrate the first phase of ABCI++, which will be introduced in Tendermint +v0.37.0, in the next major release of the Cosmos SDK. We will integrate the two +aforementioned ABCI++ methods on the `BaseApp` type. We describe the implementations +of the two methods individually below. + +Prior to describing the implementation of the two new methods, it is important to +note that the existing ABCI methods, `CheckTx`, `DeliverTx`, etc, still exist and +serve the same functions as they do now. + +### `PrepareProposal` + +Prior to evaluating the decision for how to implement `PrepareProposal`, it is +important to note that `CheckTx` will still be executed and will be responsible +for evaluating transaction validity as it does now, with one very important +_additive_ distinction. + +When executing transactions in `CheckTx`, the application will now add valid +transactions, i.e. passing the AnteHandler, to its own mempool data structure. +In order to provide a flexible approach to meet the varying needs of application +developers, we will define both a mempool interface and a data structure utilizing +Golang generics, allowing developers to focus only on transaction +ordering. Developers requiring absolute full control can implement their own +custom mempool implementation. + +We define the general mempool interface as follows (subject to change): + +```go +// MempoolTx we define an app-side mempool transaction interface that is as +// minimal as possible, only requiring applications to define the size of the +// transaction to be used when reaping and getting the transaction itself. +// Interface type casting can be used in the actual app-side mempool implementation. +type MempoolTx interface { + // Size returns the size of the transaction in bytes. + Size(codec.Codec) int + Tx() sdk.Tx +} + +// PrepareTxRecord defines a wrapper around a MempoolTx that is returned from +// PrepareProposal which includes an Action to inform Tendermint what to do with +// the transaction. +type PrepareTxRecord[T MempoolTx] struct { + Tx T + Action abci.TxAction +} + +type Mempool[T MempoolTx] interface { + // Insert attempts to insert a MempoolTx into the app-side mempool returning + // an error upon failure. + Insert(sdk.Context, T) error + // ReapMaxBytes returns the next set of available transactions from the app-side + // mempool, up to maxBytes or until the mempool is empty. The application can + // decide to return transactions from its own mempool or from the incoming + // TxRecords or some combination of both. The notion of 'available' or 'next' + // is defined by the application's mempool implementation. + ReapMaxBytes(ctx sdk.Context, txRecords abci.TxRecords, maxBytes int) ([]PrepareTxRecord[T], error) + // NumTxs returns the number of transactions currently in the mempool. + NumTxs() int + // Remove attempts to remove a transaction from the mempool, returning an error + // upon failure. + Remove(sdk.Context, T) error +} +``` + +We will define an implementation of `Mempool[T MempoolTx]` that will cover a +majority of application use cases. Namely, it will prioritize transactions by +priority and transaction sender, allowing for multiple prioritized transactions +from the same sender. The app-side mempool will be defined as a wrapper around a +simple priority queue using a max binary heap, along with additional indexes/metadata +to store senders and their nonces, allowing for simple multi-dimensional +prioritization (2-ary). + +Transaction reaping will essentially happen via a two-phase approach: + +1. Reap one or more transactions from the priority queue and collect them into + one of two buffers -- _valid_ and _invalid_. +2. For transactions that DO NOT violate the nonce validation, they are included + in the _valid_ buffer. +3. For transactions that DO violate the nonce validation, they are included in + the _invalid_ buffer. +4. Continue this process until the desired number of valid transactions are + reaped or until the mempool is empty. +5. Provide Tendermint the list of all transactions from the _valid_ buffer. +6. Re-insert all transactions, from both buffers, back into app-side mempool. + This is to ensure we do not discard transactions from the app-side mempool in + case `ProcessProposal` fails or in case that the proposal, while passing + `ProcessProposal` is not the one decided for that height, i.e. the height took + more than one round. + +```go +type PriorityMempool[T MempoolTx] struct { + queue *PriorityQueue[MempoolTx] + + // senders will contain a mapping from tx sender account addresses to all + // sequence numbers (nonces) or txs that they have in the app-side mempool. + senders map[string][]int64 + + // ... +} +``` + +> The `PriorityMempool[T MempoolTx]` implementation will support Options such as +> limiting the mempool size by a fixed number of bytes. + +Previous discussions1 have come to the agreement that Tendermint will +perform a request to the application, via `RequestPrepareProposal`, with a certain +amount of transactions reaped from Tendermint's local mempool. The exact amount +of transactions reaped will be determined by a local operator configuration. +This is referred to as the "one-shot approach" seen in discussions. + +When Tendermint reaps transactions from the local mempool and sends them to the +application via `RequestPrepareProposal`, the application will have to evaluate +the transactions. Specifically, it will need to inform Tendermint if it should +reject and or include each transaction. Note, the application can even _replace_ +transactions entirely with other transactions. + +When evaluating transactions from `RequestPrepareProposal`, the application will +ignore _all_ transactions sent to it in the request and instead reap up to +`RequestPrepareProposal.max_tx_bytes` from it's own mempool. There is no need to +execute the transactions for validity as they have already passed CheckTx. + +### `ProcessProposal` + +The `ProcessProposal` ABCI method is relatively straightforward. It is responsible +for ensuring validity of the proposed block containing transactions that were +selected from the `PrepareProposal` step. However, how an application determines +validity of a proposed block depends on the application and its varying use cases. +For most applications, simply calling the `AnteHandler` chain would suffice, but +there could easily be other applications that need more control over the validation +process of the proposed block, such as ensuring txs are in a certain order or +that certain transactions are included. While this theoretically could be achieved +with a custom `AnteHandler` implementation, it's not the cleanest UX or the most +efficient solution. + +Instead, we will define an additional ABCI interface method on the existing +`Application` interface, similar to the existing ABCI methods such as `BeginBlock` +or `EndBlock`. This new interface method will be defined as follows: + +```go +ProcessProposal(sdk.Context, abci.RequestProcessProposal) error {} +``` + +Note, we must call `ProcessProposal` with a new internal branched state on the +`Context` argument as we cannot simply just use the existing `checkState` because +`BaseApp` already has a modified `checkState` at this point. So when executing +`ProcessProposal`, we create a similar branched state, `processProposalState`, +off of `deliverState`. Note, the `processProposalState` is never committed and +is completely discarded after `ProcessProposal` finishes execution. + +We will only populate the `Status` field of the `ResponseProcessProposal` with +`ACCEPT` if ALL the transactions were accepted as valid, otherwise we will +populate with `REJECT`. + +### `DeliverTx` + +Since transactions are not truly removed from the app-side mempool during +`PrepareProposal`, since `ProcessProposal` can fail or take multiple rounds and +we do not want to lose transactions, we need to finally remove the transaction +from the app-side mempool during `DeliverTx` since during this phase, the +transactions are being included in the proposed block. + +Alternatively, we can keep the transactions as truly being removed during the +reaping phase in `PrepareProposal` and add them back to the app-side mempool in +case `ProcessProposal` fails. + +## Consequences + +### Backwards Compatibility + +ABCI++ is naturally not backwards compatible with prior versions of the Cosmos SDK +and Tendermint. For example, an application that requests `RequestPrepareProposal` +to the same application that does not speak ABCI++ will naturally fail. + +However, in the first phase of the integration, the existing ABCI methods as we +know them today will still exist and function as they currently do. + +### Positive + +* Applications now have full control over transaction ordering and priority. +* Lays the groundwork for the full integration of ABCI++, which will unlock more + app-side use cases around block construction and integration with the Tendermint + consensus engine. + +### Negative + +* Requires that the "mempool", as a general data structure that collects and stores + uncommitted transactions will be duplicated between both Tendermint and the + Cosmos SDK. +* Additional requests between Tendermint and the Cosmos SDK in the context of + block execution. Albeit, the overhead should be negligible. +* Not backwards compatible with previous versions of Tendermint and the Cosmos SDK. + +### Neutral + +## Further Discussions + +It is possible to design the app-side implementation of the `Mempool[T MempoolTx]` +in many different ways using different data structures and implementations. All +of which have different tradeoffs. The proposed solution keeps things simple +and covers cases that would be required for most basic applications. There are +tradeoffs that can be made to improve performance of reaping and inserting into +the provided mempool implementation. + +## References + +* https://github.com/tendermint/tendermint/blob/master/spec/abci%2B%2B/README.md +* [1] https://github.com/tendermint/tendermint/issues/7750#issuecomment-1076806155 +* [2] https://github.com/tendermint/tendermint/issues/7750#issuecomment-1075717151