LaconicNetwork/cosmos-sdk
7
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 2 Wiki Activity

docs: adr-070 unordered transactions refactor (#23974)

Browse Source 
Co-authored-by: Alex | Interchain Labs <alex@interchainlabs.io>
This commit is contained in:
Tyler 2025-03-19 12:46:44 -07:00 committed by GitHub
parent 7f5f3fccf8
commit 6577e1eeef
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 216 additions and 229 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

445
docs/architecture/adr-070-unordered-account.md
View File

@ -2,221 +2,132 @@
## Changelog
* Dec 4, 2023: Initial Draft (@yihuang, @tac0turtle, @alexanderbez)
* Jan 30, 2024: Include section on deterministic transaction encoding
- Dec 4, 2023: Initial Draft (@yihuang, @tac0turtle, @alexanderbez)
- Jan 30, 2024: Include section on deterministic transaction encoding
- Mar 18, 2025: Revise implementation to use Cosmos SDK KV Store and require unique timeouts per-address (@technicallyty)
## Status
ACCEPTED
ACCEPTED Not Implemented
## Abstract
We propose a way to do replay-attack protection without enforcing the order of
transactions, without requiring the use of nonces. In this way, we can support
un-ordered transaction inclusion.
transactions and without requiring the use of monotonically increasing sequences. Instead, we propose
the use of a time-based, ephemeral sequence.
## Context
As of today, the nonce value (account sequence number) prevents replay-attack and
ensures the transactions from the same sender are included into blocks and executed
in sequential order. However it makes it tricky to send many transactions from the
same sender concurrently in a reliable way. IBC relayer and crypto exchanges are
typical examples of such use cases.
Account sequence values serve to prevent replay attacks and ensure transactions from the same sender are included into blocks and executed
in sequential order. Unfortunately, this makes it difficult to reliably send many concurrent transactions from the
same sender. Victims of such limitations include IBC relayers and crypto exchanges.
## Decision
We propose to add a boolean field `unordered` to transaction body to mark "un-ordered"
transactions.
We propose adding a boolean field `unordered` and a google.protobuf.Timestamp field `timeout_timestamp` to the transaction body.
Un-ordered transactions will bypass the nonce rules and follow the rules described
below instead, in contrary, the default ordered transactions are not impacted by
this proposal, they'll follow the nonce rules the same as before.
Unordered transactions will bypass the traditional account sequence rules and follow the rules described
below, without impacting traditional ordered transactions which will follow the same sequence rules as before.
When an un-ordered transaction is included into a block, the transaction hash is
recorded in a dictionary. New transactions are checked against this dictionary for
duplicates, and to prevent the dictionary grow indefinitely, the transaction must
specify `timeout_timestamp` for expiration, so it's safe to removed it from the
dictionary after it's expired.
We will introduce new storage of time-based, ephemeral unordered sequences using the SDK's existing KV Store library.
Specifically, we will leverage the existing x/auth KV store to store the unordered sequences.
The dictionary can be simply implemented as an in-memory golang map, a preliminary
analysis shows that the memory consumption won't be too big, for example `32M = 32 * 1024 * 1024`
can support 1024 blocks where each block contains 1024 unordered transactions. For
safety, we should limit the range of `timeout_timestamp` to prevent very long expiration,
and limit the size of the dictionary.
When an unordered transaction is included in a block, a concatenation of the `timeout_timestamp` and sender’s address bytes
will be recorded to state (i.e. `542939323/<address_bytes>`). In cases of multi-party signing, one entry per signer
will be recorded to state.
New transactions will be checked against the state to prevent duplicate submissions. To prevent the state from growing indefinitely, we propose the following:
- Define an upper bound for the value of `timeout_timestamp` (i.e. 10 minutes).
- Add PreBlocker method x/auth that removes state entries with a `timeout_timestamp` earlier than the current block time.
### Transaction Format
```protobuf
message TxBody {
...
bool unordered = 4;
google.protobuf.Timestamp timeout_timestamp = 5
}
```
### Replay Protection
In order to provide replay protection, a user should ensure that the transaction's
TTL value is relatively short-lived but long enough to provide enough time to be
included in a block, e.g. ~10 minutes.
We facilitate replay protection by storing the unordered sequence in the Cosmos SDK KV store. Upon transaction ingress, we check if the transaction's unordered
sequence exists in state, or if the TTL value is stale, i.e. before the current block time. If so, we reject it. Otherwise,
we add the unordered sequence to the state. This section of the state will belong to the `x/auth` module.
We facilitate this by storing the transaction's hash in a durable map, `UnorderedTxManager`,
to prevent duplicates, i.e. replay attacks. Upon transaction ingress during `CheckTx`,
we check if the transaction's hash exists in this map or if the TTL value is stale,
i.e. before the current block time. If so, we reject it. Upon inclusion in a block
during `DeliverTx`, the transaction's hash is set in the map along with it's TTL
value.
The state is evaluated during x/auth's `PreBlocker`. All transactions with an unordered sequence earlier than the current block time
will be deleted.
This map is evaluated at the end of each block, e.g. ABCI `Commit`, and all stale
transactions, i.e. transactions's TTL value who's now beyond the committed block,
are purged from the map.
An important point to note is that in theory, it may be possible to submit an unordered
transaction twice, or multiple times, before the transaction is included in a block.
However, we'll note a few important layers of protection and mitigation:
* Assuming CometBFT is used as the underlying consensus engine and a non-noop mempool
is used, CometBFT will reject the duplicate for you.
* For applications that leverage ABCI++, `ProcessProposal` should evaluate and reject
malicious proposals with duplicate transactions.
* For applications that leverage their own application mempool, their mempool should
reject the duplicate for you.
* Finally, worst case if the duplicate transaction is somehow selected for a block
proposal, 2nd and all further attempts to evaluate it, will fail during `DeliverTx`,
so worst case you just end up filling up block space with a duplicate transaction.
```go
func (am AppModule) PreBlock(ctx context.Context) (appmodule.ResponsePreBlock, error) {
err := am.accountKeeper.RemoveExpired(sdk.UnwrapSDKContext(ctx))
if err != nil {
return nil, err
}
return &sdk.ResponsePreBlock{ConsensusParamsChanged: false}, nil
}
```
```golang
type TxHash [32]byte
package keeper
const PurgeLoopSleepMS = 500
import (
sdk "github.com/cosmos/cosmos-sdk/types"
// UnorderedTxManager contains the tx hash dictionary for duplicates checking,
// and expire them when block production progresses.
type UnorderedTxManager struct {
// blockCh defines a channel to receive newly committed block time
blockCh chan time.Time
"cosmossdk.io/collections"
"cosmossdk.io/core/store"
)
mu sync.RWMutex
// txHashes defines a map from tx hash -> TTL value, which is used for duplicate
// checking and replay protection, as well as purging the map when the TTL is
// expired.
txHashes map[TxHash]time.Time
var (
// just arbitrarily picking some upper bound number.
unorderedSequencePrefix = collections.NewPrefix(90)
)
type AccountKeeper struct {
// ...
unorderedSequences collections.KeySet[collections.Pair[uint64, []byte]]
}
func NewUnorderedTxManager() *UnorderedTxManager {
m := &UnorderedTxManager{
blockCh: make(chan time.Time, 16),
txHashes: make(map[TxHash]time.Time),
}
return m
func (m *AccountKeeper) Contains(ctx sdk.Context, sender []byte, timestamp uint64) (bool, error) {
return m.unorderedSequences.Has(ctx, collections.Join(timestamp, sender))
}
func (m *UnorderedTxManager) Start() {
go m.purgeLoop()
func (m *AccountKeeper) Add(ctx sdk.Context, sender []byte, timestamp uint64) error {
return m.unorderedSequences.Set(ctx, collections.Join(timestamp, sender))
}
func (m *UnorderedTxManager) Close() error {
close(m.blockCh)
m.blockCh = nil
return nil
}
func (m *AccountKeeper) RemoveExpired(ctx sdk.Context) error {
blkTime := ctx.BlockTime().UnixNano()
it, err := m.unorderedSequences.Iterate(ctx, collections.NewPrefixUntilPairRange[uint64, []byte](uint64(blkTime)))
if err != nil {
return err
}
defer it.Close()
func (m *UnorderedTxManager) Contains(hash TxHash) bool{
m.mu.RLock()
defer m.mu.RUnlock()
keys, err := it.Keys()
if err != nil {
return err
}
_, ok := m.txHashes[hash]
return ok
}
func (m *UnorderedTxManager) Size() int {
m.mu.RLock()
defer m.mu.RUnlock()
return len(m.txHashes)
}
func (m *UnorderedTxManager) Add(hash TxHash, expire time.Time) {
m.mu.Lock()
defer m.mu.Unlock()
m.txHashes[hash] = expire
}
// OnNewBlock send the latest block time to the background purge loop, which
// should be called in ABCI Commit event.
func (m *UnorderedTxManager) OnNewBlock(blockTime time.Time) {
m.blockCh <- blockTime
}
// expiredTxs returns expired tx hashes based on the provided block time.
func (m *UnorderedTxManager) expiredTxs(blockTime time.Time) []TxHash {
m.mu.RLock()
defer m.mu.RUnlock()
var result []TxHash
for txHash, expire := range m.txHashes {
if blockTime.After(expire) {
result = append(result, txHash)
}
}
return result
}
func (m *UnorderedTxManager) purge(txHashes []TxHash) {
m.mu.Lock()
defer m.mu.Unlock()
for _, txHash := range txHashes {
delete(m.txHashes, txHash)
}
}
// purgeLoop removes expired tx hashes in the background
func (m *UnorderedTxManager) purgeLoop() error {
for {
latestTime, ok := m.batchReceive()
if !ok {
// channel closed
return
}
hashes := m.expiredTxs(latestTime)
if len(hashes) > 0 {
m.purge(hashes)
for _, key := range keys {
if err := m.unorderedSequences.Remove(ctx, key); err != nil {
return err
}
}
return nil
}
// channelBatchRecv try to exhaust the channel buffer when it's not empty,
// and block when it's empty.
func channelBatchRecv[T any](ch <-chan *T) []*T {
item := <-ch // block if channel is empty
if item == nil {
// channel is closed
return nil
}
remaining := len(ch)
result := make([]*T, 0, remaining+1)
result = append(result, item)
for i := 0; i < remaining; i++ {
result = append(result, <-ch)
}
return result
}
```
### AnteHandler Decorator
In order to facilitate bypassing nonce verification, we have to modify the existing
To facilitate bypassing nonce verification, we must modify the existing
`IncrementSequenceDecorator` AnteHandler decorator to skip the nonce verification
when the transaction is marked as un-ordered.
when the transaction is marked as unordered.
```golang
func (isd IncrementSequenceDecorator) AnteHandle(ctx sdk.Context, tx sdk.Tx, simulate bool, next sdk.AnteHandler) (sdk.Context, error) {
@ -228,110 +139,186 @@ func (isd IncrementSequenceDecorator) AnteHandle(ctx sdk.Context, tx sdk.Tx, sim
}
```
In addition, we need to introduce a new decorator to perform the un-ordered transaction
verification and map lookup.
We also introduce a new decorator to perform the unordered transaction verification.
```golang
const (
// DefaultMaxTimeoutDuration defines the default maximum duration an un-ordered transaction
// can set.
DefaultMaxTimeoutDuration = time.Minute * 40
package ante
import (
"slices"
"strings"
"time"
sdk "github.com/cosmos/cosmos-sdk/types"
sdkerrors "github.com/cosmos/cosmos-sdk/types/errors"
authkeeper "github.com/cosmos/cosmos-sdk/x/auth/keeper"
authsigning "github.com/cosmos/cosmos-sdk/x/auth/signing"
errorsmod "cosmossdk.io/errors"
)
type DedupTxDecorator struct {
m *UnorderedTxManager
maxTimeoutDuration time.Time
var _ sdk.AnteDecorator = (*UnorderedTxDecorator)(nil)
// UnorderedTxDecorator defines an AnteHandler decorator that is responsible for
// checking if a transaction is intended to be unordered and, if so, evaluates
// the transaction accordingly. An unordered transaction will bypass having its
// nonce incremented, which allows fire-and-forget transaction broadcasting,
// removing the necessity of ordering on the sender-side.
//
// The transaction sender must ensure that unordered=true and a timeout_height
// is appropriately set. The AnteHandler will check that the transaction is not
// a duplicate and will evict it from state when the timeout is reached.
//
// The UnorderedTxDecorator should be placed as early as possible in the AnteHandler
// chain to ensure that during DeliverTx, the transaction is added to the unordered sequence state.
type UnorderedTxDecorator struct {
// maxUnOrderedTTL defines the maximum TTL a transaction can define.
maxTimeoutDuration time.Duration
txManager authkeeper.UnorderedTxManager
}
func (d *DedupTxDecorator) AnteHandle(ctx sdk.Context, tx sdk.Tx, simulate bool, next sdk.AnteHandler) (sdk.Context, error) {
// only apply to un-ordered transactions
if !tx.UnOrdered() {
return next(ctx, tx, simulate)
}
func NewUnorderedTxDecorator(
utxm authkeeper.UnorderedTxManager,
) *UnorderedTxDecorator {
return &UnorderedTxDecorator{
maxTimeoutDuration: 10 * time.Minute,
txManager: utxm,
}
}
headerInfo := d.env.HeaderService.HeaderInfo(ctx)
func (d *UnorderedTxDecorator) AnteHandle(
ctx sdk.Context,
tx sdk.Tx,
_ bool,
next sdk.AnteHandler,
) (sdk.Context, error) {
if err := d.ValidateTx(ctx, tx); err != nil {
return ctx, err
}
return next(ctx, tx, false)
}
func (d *UnorderedTxDecorator) ValidateTx(ctx sdk.Context, tx sdk.Tx) error {
unorderedTx, ok := tx.(sdk.TxWithUnordered)
if !ok || !unorderedTx.GetUnordered() {
// If the transaction does not implement unordered capabilities or has the
// unordered value as false, we bypass.
return nil
}
blockTime := ctx.BlockTime()
timeoutTimestamp := unorderedTx.GetTimeoutTimeStamp()
if timeoutTimestamp.IsZero() || timeoutTimestamp.Unix() == 0 {
return ctx, errorsmod.Wrap(sdkerrors.ErrInvalidRequest, "unordered transaction must have timeout_timestamp set")
return errorsmod.Wrap(
sdkerrors.ErrInvalidRequest,
"unordered transaction must have timeout_timestamp set",
)
}
if timeoutTimestamp.Before(headerInfo.Time) {
return ctx, errorsmod.Wrap(sdkerrors.ErrInvalidRequest, "unordered transaction has a timeout_timestamp that has already passed")
if timeoutTimestamp.Before(blockTime) {
return errorsmod.Wrap(
sdkerrors.ErrInvalidRequest,
"unordered transaction has a timeout_timestamp that has already passed",
)
}
if timeoutTimestamp.After(headerInfo.Time.Add(d.maxTimeoutDuration)) {
return ctx, errorsmod.Wrapf(sdkerrors.ErrInvalidRequest, "unordered tx ttl exceeds %s", d.maxTimeoutDuration.String())
if timeoutTimestamp.After(blockTime.Add(d.maxTimeoutDuration)) {
return errorsmod.Wrapf(
sdkerrors.ErrInvalidRequest,
"unordered tx ttl exceeds %s",
d.maxTimeoutDuration.String(),
)
}
// in order to create a deterministic hash based on the tx, we need to hash the contents of the tx with signature
// Get a Buffer from the pool
buf := bufPool.Get().(*bytes.Buffer)
// Make sure to reset the buffer
buf.Reset()
execMode := ctx.ExecMode()
if execMode == sdk.ExecModeSimulate {
return nil
}
// Use the buffer
for _, msg := range tx.GetMsgs() {
// loop through the messages and write them to the buffer
// encoding the msg to bytes makes it deterministic within the state machine.
// Malleability is not a concern here because the state machine will encode the transaction deterministically.
bz, err := proto.Marshal(msg)
signerAddrs, err := getSigners(tx)
if err != nil {
return err
}
for _, signer := range signerAddrs {
contains, err := d.txManager.Contains(ctx, signer, uint64(unorderedTx.GetTimeoutTimeStamp().Unix()))
if err != nil {
return ctx, errorsmod.Wrap(sdkerrors.ErrInvalidRequest, "failed to marshal message")
return errorsmod.Wrap(
sdkerrors.ErrIO,
"failed to check contains",
)
}
if contains {
return errorsmod.Wrapf(
sdkerrors.ErrInvalidRequest,
"tx is duplicated for signer %x", signer,
)
}
buf.Write(bz)
}
if err := d.txManager.Add(ctx, signer, uint64(unorderedTx.GetTimeoutTimeStamp().Unix())); err != nil {
return errorsmod.Wrap(
sdkerrors.ErrIO,
"failed to add unordered sequence to state",
)
}
}
return nil
}
// check for duplicates
// check for duplicates
if d.txManager.Contains(txHash) {
return ctx, errorsmod.Wrap(sdkerrors.ErrInvalidRequest, "tx %X is duplicated")
func getSigners(tx sdk.Tx) ([][]byte, error) {
sigTx, ok := tx.(authsigning.SigVerifiableTx)
if !ok {
return nil, errorsmod.Wrap(sdkerrors.ErrTxDecode, "invalid tx type")
}
return sigTx.GetSigners()
}
if d.env.TransactionService.ExecMode(ctx) == transaction.ExecModeFinalize {
// a new tx included in the block, add the hash to the unordered tx manager
d.txManager.Add(txHash, ttl)
}
```
return next(ctx, tx, simulate)
### Unordered Sequences
Unordered sequences provide a simple, straightforward mechanism to protect against both transaction malleability and
transaction duplication. It is important to note that the unordered sequence must still be unique. However,
the value is not required to be strictly increasing as with regular sequences, and the order in which the node receives
the transactions no longer matters. Clients can handle building unordered transactions similarly to the code below:
```go
for _, tx := range txs {
tx.SetUnordered(true)
tx.SetTimeoutTimestamp(time.Now() + 1 * time.Nanosecond)
}
```
### Transaction Hashes
It is absolutely vital that transaction hashes are deterministic, i.e. transaction
encoding is not malleable. If a given transaction, which is otherwise valid, can
be encoded to produce different hashes, which reflect the same valid transaction,
then a duplicate unordered transaction can be submitted and included in a block.
In order to prevent this, the decoded transaction contents is taken. Starting with the content of the transaction we marshal the transaction in order to prevent a client reordering the transaction. Next we include the gas and timeout timestamp as part of the identifier. All these fields are signed over in the transaction payload. If one of them changes the signature will not match the transaction.
### State Management
On start up, the node needs to ensure the TxManager's state contains all un-expired
transactions that have been committed to the chain. This is critical since if the
state is not properly initialized, the node will not reject duplicate transactions
and thus will not provide replay protection, and will likely get an app hash mismatch error.
The storage of unordered sequences will be facilitated using the Cosmos SDK's KV Store service.
We propose to write all un-expired unordered transactions from the TxManager's to
file on disk. On start up, the node will read this file and re-populate the TxManager's
map. The write to file will happen when the node gracefully shuts down on `Close()`.
## Note On Previous Design Iteration
Note, this is not a perfect solution, in the context of store v1. With store v2,
we can omit explicit file handling altogether and simply write the all the transactions
to non-consensus state, i.e State Storage (SS).
The previous iteration of unordered transactions worked by using an ad-hoc state-management system that posed severe
risks and a vector for duplicated tx processing. It relied on graceful app closure which would flush the current state
of the unordered sequence mapping. If the 2/3's of the network crashed, and the graceful closure did not trigger,
the system would lose track of all sequences in the mapping, allowing those transactions to be replayed. The
implementation proposed in the updated version of this ADR solves this by writing directly to the Cosmos KV Store.
While this is less performant, for the initial implementation, we opted to choose a safer path and postpone performance optimizations until we have more data on real-world impacts and a more battle-tested approach to optimization.
Alternatively, we can write all the transactions to consensus state.
Additionally, the previous iteration relied on using hashes to create what we call an "unordered sequence." There are known
issues with transaction malleability in Cosmos SDK signing modes. This ADR gets away from this problem by enforcing
single-use unordered nonces, instead of deriving nonces from bytes in the transaction.
## Consequences
### Positive
* Support un-ordered and concurrent transaction inclusion.
* Support unordered transaction inclusion, enabling the ability to "fire and forget" many transactions at once.
### Negative
* Requires additional storage overhead and management of processed unordered
transactions that exist outside of consensus state.
* Requires additional storage overhead.
* Requirement of unique timestamps per transaction causes a small amount of additional overhead for clients. Clients must ensure each transaction's timeout timestamp is different. However, nanosecond differentials suffice.
* Usage of Cosmos SDK KV store is slower in comparison to using a non-merklized store or ad-hoc methods, and block times may slow down as a result.
## References
* https://github.com/cosmos/cosmos-sdk/issues/13009