more docs

This commit is contained in:
David Terpay 2023-08-16 10:51:55 -04:00
parent 6529c39605
commit a550384569
No known key found for this signature in database
GPG Key ID: 627EFB00DADF0CD1
2 changed files with 45 additions and 520 deletions

113
README.md
View File

@ -9,80 +9,57 @@
[![License: Apache-2.0](https://img.shields.io/github/license/skip-mev/pob.svg?style=flat-square)](https://github.com/skip-mev/pob/blob/main/LICENSE)
[![Lines Of Code](https://img.shields.io/tokei/lines/github/skip-mev/pob?style=flat-square)](https://github.com/skip-mev/pob)
## 📖 Overview
### 🤔 What is the Block SDK?
The Block SDK is a set of Cosmos SDK and ABCI++ primitives that allow chains to fully customize blocks to specific use cases. It turns your chain's blocks into a **transaction highway** consisting of individual lanes with their own special functionality.
## 🤔 How does it work
### 🔁 Transaction Lifecycle
The best way to understand how lanes work is to first understand the lifecycle
of a transaction. A transaction begins its lifecycle when it is first signed and
broadcasted to a chain. After it is broadcasted to a validator, it will be checked
in `CheckTx` by the base application. If the transaction is valid, it will be
inserted into the applications mempool.
The transaction then waits in the mempool until a new block needs to be proposed.
When a new block needs to be proposed, the application will call `PrepareProposal`
(which is a new ABCI++ addition) to request a new block from the current
proposer. The proposer will look at what transactions currently waiting to
be included in a block by looking at their mempool. The proposer will then
iteratively select transactions until the block is full. The proposer will then
send the block to other validators in the network.
When a validator receives a proposed block, the validator will first want to
verify the contents of the block before signing off on it. The validator will
call `ProcessProposal` to verify the contents of the block. If the block is
valid, the validator will sign off on the block and broadcast their vote to the
network. If the block is invalid, the validator will reject the block. Once a
block is accepted by the network, it is committed and the transactions that
were included in the block are removed from the validator's mempool (as they no
longer need to be considered).
### 🛣️ Lane Lifecycle
After a transaction is verified in CheckTx, it will attempt to be inserted
into the `LanedMempool`. A LanedMempool is composed of several distinct `Lanes`
that have the ability to store their own transactions. The LanedMempool will
insert the transaction into all lanes that will accept it. The criteria for
whether a lane will accept a transaction is defined by the lane's
`MatchHandler`. The default implementation of a MatchHandler will accept all transactions.
> **🌐 The Block SDK is a toolkit for building customized blocks**
> The Block SDK is a set of Cosmos SDK and ABCI++ primitives that allow chains to fully customize blocks to specific use cases. It turns your chain's blocks into a **`highway`** consisting of individual **`lanes`** with their own special functionality.
When a new block is proposed, the `PrepareProposalHandler` will iteratively call
`PrepareLane` on each lane (in the order in which they are defined in the
LanedMempool). The PrepareLane method is anaolgous to PrepareProposal. Calling
PrepareLane on a lane will trigger the lane to reap transactions from its mempool
and add them to the proposal (given they are valid respecting the verification rules
of the lane).
Skip has built out a number of plug-and-play `lanes` on the SDK that your protocol can use, including in-protocol MEV recapture and Oracles! Additionally, the Block SDK can be extended to add **your own custom `lanes`** to configure your blocks to exactly fit your application needs.
When proposals need to be verified in `ProcessProposal`, the `ProcessProposalHandler`
defined in `abci/abci.go` will call `ProcessLane` on each lane in the same order
as they were called in the PrepareProposalHandler. Each subsequent call to
ProcessLane will filter out transactions that belong to previous lanes. A given
lane's ProcessLane will only verify transactions that belong to that lane.
### ❌ Problems: Blocks are not Customizable
> **Scenario**
>
> Let's say we have a LanedMempool composed of two lanes: LaneA and LaneB.
> LaneA is defined first in the LanedMempool and LaneB is defined second.
> LaneA contains transactions Tx1 and Tx2 and LaneB contains transactions
> Tx3 and Tx4.
Most Cosmos chains today utilize standard `CometBFT` block construction - which is too limited.
- The standard `CometBFT` block building is susceptible to MEV-related issues, such as front-running and sandwich attacks, since proposers have monopolistic rights on ordering and no verification of good behavior. MEV that is created cannot be redistributed to the protocol.
- The standard `CometBFT` block building uses a one-size-fits-all approach, which can result in inefficient transaction processing for specific applications or use cases and sub-optimal fee markets.
- Transactions tailored for specific applications may need custom prioritization, ordering or validation rules that the mempool is otherwise unaware of because transactions within a block are currently in-differentiable when a blockchain might want them to be.
### ✅ Solution: The Block SDK
You can think of the Block SDK as a **transaction `highway` system**, where each
`lane` on the highway serves a specific purpose and has its own set of rules and
traffic flow.
In the Block SDK, each `lane` has its own set of rules and transaction flow management systems.
* A `lane` is what we might traditionally consider to be a standard mempool
where transaction **_validation_**, **_ordering_** and **_prioritization_** for
contained transactions are shared.
* `lanes` implement a **standard interface** that allows each individual `lane` to
propose and validate a portion of a block.
* `lanes` are ordered with each other, configurable by developers. All `lanes`
together define the desired block structure of a chain.
### ✨ Block SDK Use Cases
A block with separate `lanes` can be used for:
1. **MEV mitigation**: a top of block lane could be designed to create an in-protocol top-of-block auction (as we are doing with POB) to recapture MEV in a transparent and governable way.
2. **Free/reduced fee txs**: transactions with certain properties (e.g. from trusted accounts or performing encouraged actions) could leverage a free lane to reward behavior.
3. **Dedicated oracle space** Oracles could be included before other kinds of transactions to ensure that price updates occur first, and are not able to be sandwiched or manipulated.
4. **Orderflow auctions**: an OFA lane could be constructed such that order flow providers can have their submitted transactions bundled with specific backrunners, to guarantee MEV rewards are attributed back to users. Imagine MEV-share but in protocol.
5. **Enhanced and customizable privacy**: privacy-enhancing features could be introduced, such as threshold encrypted lanes, to protect user data and maintain privacy for specific use cases.
6. **Fee market improvements**: one or many fee markets - such as EIP-1559 - could be easily adopted for different lanes (potentially custom for certain dApps). Each smart contract/exchange could have its own fee market or auction for transaction ordering.
7. **Congestion management**: segmentation of transactions to lanes can help mitigate network congestion by capping usage of certain applications and tailoring fee markets.
When a new block needs to be proposed, the PrepareProposalHandler will call
PrepareLane on LaneA first and LaneB second. When PrepareLane is called
on LaneA, LaneA will reap transactions from its mempool and add them to the
proposal. Same applies for LaneB. Say LaneA reaps transactions Tx1 and Tx2
and LaneB reaps transactions Tx3 and Tx4. This gives us a proposal composed
of the following:
### 📚 Block SDK Documentation
* Tx1, Tx2, Tx3, Tx4
#### Lane App Store
When the ProcessProposalHandler is called, it will call ProcessLane on LaneA
with the proposal composed of Tx1, Tx2, Tx3, and Tx4. LaneA will then
verify Tx1 and Tx2 and return the remaining transactions - Tx3 and Tx4.
The ProcessProposalHandler will then call ProcessLane on LaneB with the
remaining transactions - Tx3 and Tx4. LaneB will then verify Tx3 and Tx4
and return no remaining transactions.
To read more about Skip's pre-built `lanes` and how to use them, check out the [Lane App Store]().
#### Lane Development
To read more about how to build your own custom `lanes`, check out the [Build Your Own Lane]().

View File

@ -1,452 +0,0 @@
# 🎨 Base Lane
> 🏗️ Build your own lane in less than 10 minutes using the Base Lane
## 💡 Overview
The Base Lane is a generic implementation of a lane. It comes out of the
box with default implementations for all the required interfaces. It is meant to
be used as a starting point for building your own lane.
## 🤔 How to use it
> **Default Implementations**
>
> There are default implementations for all of the below which can be found in
> the `block/base` package. It is highly recommended that developers overview
> the default implementations before building their own lane.
There are **three** critical components to building a custom lane using the lane
constructor:
1. `LaneConfig` - The lane configuration which determines the basic properties
of the lane including the maximum block space that the lane can fill up.
2. `LaneMempool` - The lane mempool which is responsible for storing
transactions that have been verified and are waiting to be included in proposals.
3. `MatchHandler` - This is responsible for determining whether a transaction should
belong to this lane.
4. [**OPTIONAL**] `PrepareLaneHandler` - Allows developers to define their own
handler to customize the how transactions are verified and ordered before they
are included into a proposal.
5. [**OPTIONAL**] `CheckOrderHandler` - Allows developers to define their own
handler that will run any custom checks on whether transactions included in
block proposals are in the correct order (respecting the ordering rules of the
lane and the ordering rules of the other lanes).
6. [**OPTIONAL**] `ProcessLaneHandler` - Allows developers to define their own
handler for processing transactions that are included in block proposals.
### 1. 📝 Lane Config
The lane config (`LaneConfig`) is a simple configuration
object that defines the desired amount of block space the lane should
utilize when building a proposal, an antehandler that is used to verify
transactions as they are added/verified to/in a proposal, and more. By default,
we recommend that user's pass in all of the base apps configurations (txDecoder,
logger, etc.). A sample `LaneConfig` might look like the following:
```golang
config := block.LaneConfig{
Logger: app.Logger(),
TxDecoder: app.TxDecoder(),
TxEncoder: app.TxEncoder(),
AnteHandler: app.AnteHandler(),
MaxTxs: 0,
MaxBlockSpace: math.LegacyZeroDec(),
IgnoreList: []block.Lane{},
}
```
The three most important parameters to set are the `AnteHandler`, `MaxTxs`, and
`MaxBlockSpace`.
#### **AnteHandler**
With the default implementation, the `AnteHandler` is responsible for verifying
transactions as they are being considered for a new proposal or are being processed
in a proposed block. We recommend user's utilize the same antehandler chain that
is used in the base app. If developers want a certain `AnteDecorator` to be
ignored if it qualifies for a given lane, they can do so by using the `NewIgnoreDecorator`
defined in `block/utils/ante.go`.
For example, a free lane might want to ignore the `DeductFeeDecorator` so that it's
transactions are not charged any fees. Where ever the `AnteHandler` is defined,
we could add the following to ignore the `DeductFeeDecorator`:
```golang
anteDecorators := []sdk.AnteDecorator{
ante.NewSetUpContextDecorator(),
...,
utils.NewIgnoreDecorator(
ante.NewDeductFeeDecorator(
options.BaseOptions.AccountKeeper,
options.BaseOptions.BankKeeper,
options.BaseOptions.FeegrantKeeper,
options.BaseOptions.TxFeeChecker,
),
options.FreeLane,
),
...,
}
```
Anytime a transaction that qualifies for the free lane is being processed, the
`DeductFeeDecorator` will be ignored and no fees will be deducted!
#### **MaxTxs**
This sets the maximum number of transactions allowed in the mempool with
the semantics:
* if `MaxTxs` == 0, there is no cap on the number of transactions in the mempool
* if `MaxTxs` > 0, the mempool will cap the number of transactions it stores,
and will prioritize transactions by their priority and sender-nonce
(sequence number) when evicting transactions.
* if `MaxTxs` < 0, `Insert` is a no-op.
#### **MaxBlockSpace**
MaxBlockSpace is the maximum amount of block space that the lane will attempt to
fill when building a proposal. This parameter may be useful lanes that should be
limited (such as a free or onboarding lane) in space usage. Setting this to 0
will allow the lane to fill the block with as many transactions as possible.
If a block proposal request has a `MaxTxBytes` of 1000 and the lane has a
`MaxBlockSpace` of 0.5, the lane will attempt to fill the block with 500 bytes.
#### **[OPTIONAL] IgnoreList**
`IgnoreList` defines the list of lanes to ignore when processing transactions.
For example, say there are two lanes: default and free. The free lane is
processed after the default lane. In this case, the free lane should be added
to the ignore list of the default lane. Otherwise, the transactions that belong
to the free lane will be processed by the default lane (which accepts all
transactions by default).
### 2. 🗄️ LaneMempool
This is the data structure that is responsible for storing transactions
as they are being verified and are waiting to be included in proposals. `block/base/mempool.go`
provides an out-of-the-box implementation that should be used as a starting
point for building out the mempool and should cover most use cases. To
utilize the mempool, you must implement a `TxPriority[C]` struct that does the
following:
* Implements a `GetTxPriority` method that returns the priority (as defined
by the type `[C]`) of a given transaction.
* Implements a `Compare` method that returns the relative priority of two
transactions. If the first transaction has a higher priority, the method
should return -1, if the second transaction has a higher priority the method
should return 1, otherwise the method should return 0.
* Implements a `MinValue` method that returns the minimum priority value
that a transaction can have.
The default implementation can be found in `block/base/mempool.go`. What
if we wanted to prioritize transactions by the amount they have staked on a chain?
Well we could do something like the following:
```golang
// CustomTxPriority returns a TxPriority that prioritizes transactions by the
// amount they have staked on chain. This means that transactions with a higher
// amount staked will be prioritized over transactions with a lower amount staked.
func (p *CustomTxPriority) CustomTxPriority() TxPriority[string] {
return TxPriority[string]{
GetTxPriority: func(ctx context.Context, tx sdk.Tx) string {
// Get the signer of the transaction.
signer := p.getTransactionSigner(tx)
// Get the total amount staked by the signer on chain.
// This is abstracted away in the example, but you can
// implement this using the staking keeper.
totalStake, err := p.getTotalStake(ctx, signer)
if err != nil {
return ""
}
return totalStake.String()
},
Compare: func(a, b string) int {
aCoins, _ := sdk.ParseCoinsNormalized(a)
bCoins, _ := sdk.ParseCoinsNormalized(b)
switch {
case aCoins == nil && bCoins == nil:
return 0
case aCoins == nil:
return -1
case bCoins == nil:
return 1
default:
switch {
case aCoins.IsAllGT(bCoins):
return 1
case aCoins.IsAllLT(bCoins):
return -1
default:
return 0
}
}
},
MinValue: "",
}
}
```
#### Using a Custom TxPriority
To utilize this new priority configuration in a lane, all you have to then do
is pass in the `TxPriority[C]` to the `NewLaneMempool` function.
```golang
// Create the lane config
laneCfg := NewLaneConfig(
...
MaxTxs: 100,
...
)
// Pseudocode for creating the custom tx priority
priorityCfg := NewPriorityConfig(
stakingKeeper,
accountKeeper,
...
)
// define your mempool that orders transactions by on-chain stake
mempool := constructor.NewMempool[string](
priorityCfg.CustomTxPriority(),
laneCfg.TxEncoder,
laneCfg.MaxTxs,
)
// Initialize your lane with the mempool
lane := constructor.NewBaseLane(
laneCfg,
LaneName,
mempool,
constructor.DefaultMatchHandler(),
)
```
### 3. 🤝 MatchHandler
`MatchHandler` is utilized to determine if a transaction should be included in
the lane. This function can be a stateless or stateful check on the transaction.
The default implementation can be found in `block/base/handlers.go`.
The match handler can be as custom as desired. Following the example above, if
we wanted to make a lane that only accepts transactions if they have a large
amount staked, we could do the following:
```golang
// CustomMatchHandler returns a custom implementation of the MatchHandler. It
// matches transactions that have a large amount staked. These transactions
// will then be charged no fees at execution time.
//
// NOTE: This is a stateful check on the transaction. The details of how to
// implement this are abstracted away in the example, but you can implement
// this using the staking keeper.
func (h *Handler) CustomMatchHandler() block.MatchHandler {
return func(ctx sdk.Context, tx sdk.Tx) bool {
if !h.IsStakingTx(tx) {
return false
}
signer, err := getTxSigner(tx)
if err != nil {
return false
}
stakedAmount, err := h.GetStakedAmount(signer)
if err != nil {
return false
}
// The transaction can only be considered for inclusion if the amount
// staked is greater than some predetermined threshold.
return stakeAmount.GT(h.Threshold)
}
}
```
#### Using a Custom MatchHandler
If we wanted to create the lane using the custom match handler along with the
custom mempool, we could do the following:
```golang
// Pseudocode for creating the custom match handler
handler := NewHandler(
stakingKeeper,
accountKeeper,
...
)
// define your mempool that orders transactions by on chain stake
mempool := constructor.NewMempool[string](
priorityCfg.CustomTxPriority(),
cfg.TxEncoder,
cfg.MaxTxs,
)
// Initialize your lane with the mempool
lane := constructor.NewBaseLane(
cfg,
LaneName,
mempool,
handler.CustomMatchHandler(),
)
```
### Summary on Steps 1-3
The following is a summary of the steps above:
1. Create a custom `LaneConfig` struct that defines the configuration of the lane.
2. Create a custom `TxPriority[C]` struct to have a custom mempool that orders
transactions via a custom priority mechanism.
3. Create a custom `MatchHandler` that implements the `block.MatchHandler` to
have a custom lane that only accepts transactions that match a custom criteria.
### [OPTIONAL] Steps 4-6
The remaining steps walk through the process of creating custom block
building/verification logic. The default implementation found in `block/base/handlers.go`
should fit most use cases. Please reference that file for more details on
the default implementation and whether it fits your use case.
Implementing custom block building/verification logic is a bit more involved
than the previous steps and is a all or nothing approach. This means that if
you implement any of the handlers, you must implement all of them in most cases.
If you do not implement all of them, the lane may have unintended behavior.
### 4. 🛠️ PrepareLaneHandler
The `PrepareLaneHandler` is an optional field you can set on the lane constructor.
This handler is responsible for the transaction selection logic when a new proposal
is requested.
The handler should return the following for a given lane:
1. The transactions to be included in the block proposal.
2. The transactions to be removed from the lane's mempool.
3. An error if the lane is unable to prepare a block proposal.
```golang
// PrepareLaneHandler is responsible for preparing transactions to be included
// in the block from a given lane. Given a lane, this function should return
// the transactions to include in the block, the transactions that must be
// removed from the lane, and an error if one occurred.
PrepareLaneHandler func(ctx sdk.Context,proposal BlockProposal,maxTxBytes int64)
(txsToInclude [][]byte, txsToRemove []sdk.Tx, err error)
```
The default implementation is simple. It will continue to select transactions
from its mempool under the following criteria:
1. The transactions is not already included in the block proposal.
2. The transaction is valid and passes the AnteHandler check.
3. The transaction is not too large to be included in the block.
If a more involved selection process is required, you can implement your own
`PrepareLaneHandler` and and set it after creating the lane constructor.
```golang
// Pseudocode for creating the custom prepare lane handler
// This assumes that the CustomLane inherits from the constructor
// lane.
customLane := constructor.NewCustomLane(
cfg,
LaneName,
mempool,
handler.CustomMatchHandler(),
)
// Set the custom PrepareLaneHandler on the lane
customLane.SetPrepareLaneHandler(customlane.PrepareLaneHandler())
```
### 5. ✅ CheckOrderHandler
The `CheckOrderHandler` is an optional field you can set on the lane constructor.
This handler is responsible for verifying the ordering of the transactions in the
block proposal that belong to the lane.
```golang
// CheckOrderHandler is responsible for checking the order of transactions that
// belong to a given lane. This handler should be used to verify that the
// ordering of transactions passed into the function respect the ordering logic
// of the lane (if any transactions from the lane are included). This function
// should also ensure that transactions that belong to this lane are contiguous
// and do not have any transactions from other lanes in between them.
CheckOrderHandler func(ctx sdk.Context, txs []sdk.Tx) error
```
The default implementation is simple and utilizes the same `TxPriority` struct
that the mempool uses to determine if transactions are in order. The criteria
for determining if transactions are in order is as follows:
1. The transactions are in order according to the `TxPriority` struct. i.e. any
two transactions (that match to the lane) `tx1` and `tx2` where `tx1` has a
higher priority than `tx2` should be ordered before `tx2`.
2. The transactions are contiguous. i.e. there are no transactions from other
lanes in between the transactions that belong to this lane. i.e. if `tx1` and
`tx2` belong to the lane, there should be no transactions from other lanes in
between `tx1` and `tx2`.
If a more involved ordering process is required, you can implement your own
`CheckOrderHandler` and and set it after creating the lane constructor.
```golang
// Pseudocode for creating the custom check order handler
// This assumes that the CustomLane inherits from the constructor
// lane.
customLane := constructor.NewCustomLane(
cfg,
LaneName,
mempool,
handler.CustomMatchHandler(),
)
// Set the custom CheckOrderHandler on the lane
customLane.SetCheckOrderHandler(customlane.CheckOrderHandler())
```
### 6. 🆗 ProcessLaneHandler
The `ProcessLaneHandler` is an optional field you can set on the lane constructor.
This handler is responsible for verifying the transactions in the block proposal
that belong to the lane. This handler is executed after the `CheckOrderHandler`
so the transactions passed into this function SHOULD already be in order
respecting the ordering rules of the lane and respecting the ordering rules of
mempool relative to the lanes it has. This means that if the first transaction
does not belong to the lane, the remaining transactions should not belong to the
lane either.
```golang
// ProcessLaneHandler is responsible for processing transactions that are
// included in a block and belong to a given lane. ProcessLaneHandler is
// executed after CheckOrderHandler so the transactions passed into this
// function SHOULD already be in order respecting the ordering rules of the
// lane and respecting the ordering rules of mempool relative to the lanes it has.
ProcessLaneHandler func(ctx sdk.Context, txs []sdk.Tx) ([]sdk.Tx, error)
```
Given the invarients above, the default implementation is simple. It will
continue to verify transactions in the block proposal under the following
criteria:
1. If a transaction matches to this lane, verify it and continue. If it is not
valid, return an error.
2. If a transaction does not match to this lane, return the remaining transactions
to the next lane to process.