From 1ff85aae430cef067db7e2a9933942fa158a4c25 Mon Sep 17 00:00:00 2001 From: antstalepresh Date: Fri, 6 Nov 2020 00:15:31 +1000 Subject: [PATCH 1/8] cosmos sdk master sync on interval --- .github/workflows/repo-sync.yml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 .github/workflows/repo-sync.yml diff --git a/.github/workflows/repo-sync.yml b/.github/workflows/repo-sync.yml new file mode 100644 index 0000000000..8e9cbed5e4 --- /dev/null +++ b/.github/workflows/repo-sync.yml @@ -0,0 +1,19 @@ +on: + schedule: + - cron: "*/15 * * * *" + workflow_dispatch: + +jobs: + repo-sync: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + with: + persist-credentials: false + - name: repo-sync + uses: repo-sync/github-sync@v2 + with: + source_repo: "https://github.com/cosmos/cosmos-sdk" + source_branch: "master" + destination_branch: "master" + github_token: ${{ secrets.PAT }} \ No newline at end of file From e63f9f28ea3f7c23c3f12cdf83d1d3087b4c5d1c Mon Sep 17 00:00:00 2001 From: antstalepresh Date: Fri, 6 Nov 2020 00:38:14 +1000 Subject: [PATCH 2/8] add line break at the end --- .github/workflows/repo-sync.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/repo-sync.yml b/.github/workflows/repo-sync.yml index 8e9cbed5e4..7844b46fa8 100644 --- a/.github/workflows/repo-sync.yml +++ b/.github/workflows/repo-sync.yml @@ -16,4 +16,4 @@ jobs: source_repo: "https://github.com/cosmos/cosmos-sdk" source_branch: "master" destination_branch: "master" - github_token: ${{ secrets.PAT }} \ No newline at end of file + github_token: ${{ secrets.PAT }} From 352a19e6eb75eb17e329acd1ba53e0f25a77721a Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Thu, 11 Feb 2021 11:30:42 -0600 Subject: [PATCH 3/8] Add epoched staking ADR --- docs/architecture/adr-033-epoched-staking.md | 111 +++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 docs/architecture/adr-033-epoched-staking.md diff --git a/docs/architecture/adr-033-epoched-staking.md b/docs/architecture/adr-033-epoched-staking.md new file mode 100644 index 0000000000..01929758be --- /dev/null +++ b/docs/architecture/adr-033-epoched-staking.md @@ -0,0 +1,111 @@ +# ADR 032: Typed Events + +## Changelog + +- 10-Feb-2021: Initial Draft + +## Authors + +- Dev Ojha (@valardragon) +- Sunny Aggarwall (@sunnya97) + +## Status + +Proposed + +## Abstract + +This ADR updates the proof of stake module to buffer the staking weight updates for a number of blocks before updating the consensus' staking weights. The length of the buffer is dubbed an epoch. The prior functionality of the staking module is then a special case of the abstracted module, with the epoch being set to 1 block. + +## Context + +The current proof of stake module takes the design decision to apply staking weight changes to the consensus engine immediately. This means that delegations and unbonds get applied immediately to the validator set. This decision was primarily done as it was implementationally simplest, and because we at the time believed that this would lead to better UX for clients. + +An alternative design choice is to allow buffering staking updates (delegations, unbonds, validators joining) for a number of blocks. This 'epoch'd proof of stake consensus provides the guarantee that the consensus weights for validators will not change mid-epoch, except in the event of a slash condition. + +The decision to have immediate execution of staking changes was primarily done as it was implementationally simplest, and because we at the time believed that this would lead to better UX for clients. The UX hurdle may not be as significant as was previously thought, since it is possible to provide users acknowledgement that their bond was recorded and will be executed. + +Furthermore, it has become clearer over time that immediate execution of staking events comes with limitations, such as: + +* Threshold based cryptography. One of the main limitations is that because the validator set can change so regularly, it makes the running of multiparty computation by a fixed validator set difficult. Many threshold-based cryptographic features for blockchains such as randomness beacons and threshold decryption require a computationally-expensive DKG process (will take much longer than 1 block to create). To productively use these, we need to guarantee that the result of the DKG will be used for a reasonably long time. It wouldn't be feasible to rerun the DKG every block. By epoching staking, it guarantees we'll only need to run a new DKG once every epoch. + +* Light client efficiency. This would lessen the overhead for IBC. Because of the lite client bisection algorithm, the number of headers you need to verify is related to bounding the validator set diffs between two successively verified headers. By limiting the frequency of validator set changes, we can reduce the size of IBC lite client proofs. + +* Fairness of deterministic leader election. Currently we have no ways of reasoning of fairness of deterministic leader election in the presence of staking changes without epochs (tendermint/spec#217). Adding epochs at least makes it easier for our deterministic leader election to match something we can prove secure. (Albeit, we still haven’t proven if our current algorithm is fair with > 2 validators) + +* Staking derivative design. Currently, reward distribution is done lazily using the F1 fee distribution. While saving computational complexity, lazy accounting increases “statefulness of staking”. Right now, each delegation entry has to track the time of last withdrawal. Handling this can be a challenge for some staking derivatives designs (see example). Force-withdrawing rewards to users can help solve this, however it is infeasible to force-withdraw rewards to users on a per block basis. With epoching, a chain could more easily alter the design to have rewards be forcefully withdrawn (iterating over delegator accounts only once per-epoch), and thus remove the time of delegation from state. This preliminarily seems like it may be of utility in certain staking derivative designs. + +## Design considerations + +### Slashing + +There is a design consideration for whether to apply a slash immediately or at the end of an epoch. A slash event should apply to only members who are actually staked during the time of the infraction, namely during the epoch the slash event occured. + +Applying it immediately can be viewed as offering greater consensus layer security, at potential costs to the aforementioned usecases. The benefits of immediate slashing for consensus layer security can be all be obtained by executing the validator jailing immediately (thus removing it from the validator set), and delaying the actual slash change to the validator's weight until the epoch boundary. For the use cases mentioned above, workarounds can be integrated to avoid problems, as follows: + +- For threshold based cryptography, it can keep using the original keep epoch weights for the cryptography thresholds, but allow the underlying finality to benefit from extra security more quickly. +- For light client efficiency, there can be a bit included in the header indicating an intra-epoch slash (ala https://github.com/tendermint/spec/issues/199). +- For fairness of deterministic leader election, this will cause problems with the formalization of it / proximity of implementation to formally provable spec. However, a less formal claim can be made that the amount lost due to the slash should hopefully outweigh slight biases into the leader election process. This claim is dubious with the presence of MEV, but potentially formal upperbounds on MEV for fairness here could be derived. +- For staking derivative design, this will not cause problems with the suggested design there, nor does it increase the stateful of staking. (As whether a slash has occured is fully queryable given the validator address) + +However, for achieving consensus layer security, it suffices to apply the validator jailing immediately, but still delay the actual slash changes to waiting until the end of the epoch. This largely mitigates the concern for the fairness of deterministic leader election as well, since that validator is removed the set being rotated from immediately. + +### Token lockup + +When someone makes a transaction to delegate, even though they are not immediately staked, their tokens should be moved into a pool managed by the staking module which will then be used at the end of an epoch. This prevents concerns where they stake, and then spend those tokens not realizing they were already allocated for staking, and thus having their staking tx fail. + +### Pipelining the epochs + +For threshold based cryptography in particular, we need a pipeline for epoch changes. This is because when we are in epoch N, we want the epoch N+1 weights to be fixed so that the validator set can do the DKG accordingly. So if we are currently in epoch N, the stake weights for epoch N+1 should already be fixed, and new stake changes should be getting applied to epoch N + 2. + +This can be handled by making a parameter for the epoch pipeline. This parameter should not be alterable except during hard forks, to mitigate implementation complexity of switching the pipeline length. + +### Rewards + +Even though all staking updates are applied at epoch boundaries, rewards can still be distributed immediately when they are claimed. This is because they do not affect the current stake weights, as we do not implement auto-bonding of rewards. If such a feature were to be implemented, it would have to be setup so that rewards are auto-bonded at the epoch boundary. + +## Decision + +__Step-1__: Implement buffering of all staking and slashing messages. + +First we create a pool for storing tokens that are being bonded, but should be applied at the epoch boundary called the `EpochDelegationPool`. Then, we have two separate queues, one for staking, one for slashing. We describe what happens on each message being delivered below: + +### Staking messages +- **MsgCreateValidator**: Move user's self-bond to `EpochDelegationPool` immediately. Queue a message for the epoch boundary to handle the self-bond, taking the funds from the `EpochDelegationPool`. If Epoch execution fail, return back funds from `EpochDelegationPool` to user's account. +- **MsgEditValidator**: Validate message and if valid queue the message for execution at the end of the Epoch. +- **MsgDelegate**: Move user's funds to `EpochDelegationPool` immediately. Queue a message for the epoch boundary to handle the delegation, taking the funds from the `EpochDelegationPool`. If Epoch execution fail, return back funds from `EpochDelegationPool` to user's account. +- **MsgBeginRedelegate**: Validate message and if valid queue the message for execution at the end of the Epoch. +- **MsgUndelegate**: Validate message and if valid queue the message for execution at the end of the Epoch. + +### Slashing messages +- **MsgUnjail**: Validate message and if valid queue the message for execution at the end of the Epoch. +- **Slash Event**: Whenever a slash event is created, it gets queued in the slashing module to apply at the end of the epoch. The queues should be setup such that this slash applies immediately. + +### Evidence Messages + - **MsgSubmitEvidence**: This gets executed immediately, and the validator gets jailed immediately. However in slashing, the actual slash event gets queued. + +Then we add methods to the end blockers, to ensure that at the epoch boundary the queues are cleared and delegation updates are applied. + + +__Step-2__: Implement querying of queued staking txs. + +When querying the staking activity of a given address, the status should return not only the amount of tokens staked, but also if there are any queued stake events for that address. This will require nodes supporting querying to either do some more indexing to have this be efficiently queryable, or to have transactions + +__Step-3__: Adjust gas + +Currently gas represents the cost of executing a transaction when its done immediately. (Merging together costs of p2p overhead, state access overhead, and computational overhead) However, now a transaction can cause computation in a future block, namely at the epoch boundary. + +To handle this, we should initially include parameters for estimating the amount of future computation (denominated in gas), and add that as a flat charge needed for the message. +We leave it as out of scope for how to weight future computation versus current computation in gas pricing, and have it set such that the are weighted equally for now. + +## Consequences + +### Positive + +* Abstracts the proof of stake module that allows retaining the existing functionality +* Enables new features such as validator-set based threshold cryptography + +### Negative + +* Increases complexity of integrating more complex gas pricing mechanisms, as they now have to consider future execution costs as well. + From 3a7cc3bd3a620f45949dce2524c6652e0346eb69 Mon Sep 17 00:00:00 2001 From: Sunny Aggarwal Date: Thu, 11 Feb 2021 16:23:53 -0500 Subject: [PATCH 4/8] epoched staking --- docs/architecture/README.md | 3 +- docs/architecture/adr-039-epoched-staking.md | 110 +++++++++++++++++++ 2 files changed, 112 insertions(+), 1 deletion(-) create mode 100644 docs/architecture/adr-039-epoched-staking.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 19e6d6e2db..da4f77898d 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -74,4 +74,5 @@ Read about the [PROCESS](./PROCESS.md). - [ADR 032: Typed Events](./adr-032-typed-events.md) - [ADR 035: Rosetta API Support](./adr-035-rosetta-api-support.md) - [ADR 037: Governance Split Votes](./adr-037-gov-split-vote.md) -- [ADR 038: State Listening](./adr-038-state-listening.md) \ No newline at end of file +- [ADR 038: State Listening](./adr-038-state-listening.md) +- [ADR 039: Epoched Staking](./adr-039-epoched-staking.md) \ No newline at end of file diff --git a/docs/architecture/adr-039-epoched-staking.md b/docs/architecture/adr-039-epoched-staking.md new file mode 100644 index 0000000000..9e7720627c --- /dev/null +++ b/docs/architecture/adr-039-epoched-staking.md @@ -0,0 +1,110 @@ +# ADR 039: Epoched Staking + +## Changelog + +- 10-Feb-2021: Initial Draft + +## Authors + +- Dev Ojha (@valardragon) +- Sunny Aggarwall (@sunnya97) + +## Status + +Proposed + +## Abstract + +This ADR updates the proof of stake module to buffer the staking weight updates for a number of blocks before updating the consensus' staking weights. The length of the buffer is dubbed an epoch. The prior functionality of the staking module is then a special case of the abstracted module, with the epoch being set to 1 block. + +## Context + +The current proof of stake module takes the design decision to apply staking weight changes to the consensus engine immediately. This means that delegations and unbonds get applied immediately to the validator set. This decision was primarily done as it was implementationally simplest, and because we at the time believed that this would lead to better UX for clients. + +An alternative design choice is to allow buffering staking updates (delegations, unbonds, validators joining) for a number of blocks. This 'epoch'd proof of stake consensus provides the guarantee that the consensus weights for validators will not change mid-epoch, except in the event of a slash condition. + +The decision to have immediate execution of staking changes was primarily done as it was implementationally simplest, and because we at the time believed that this would lead to better UX for clients. The UX hurdle may not be as significant as was previously thought, since it is possible to provide users acknowledgement that their bond was recorded and will be executed. + +Furthermore, it has become clearer over time that immediate execution of staking events comes with limitations, such as: + +* Threshold based cryptography. One of the main limitations is that because the validator set can change so regularly, it makes the running of multiparty computation by a fixed validator set difficult. Many threshold-based cryptographic features for blockchains such as randomness beacons and threshold decryption require a computationally-expensive DKG process (will take much longer than 1 block to create). To productively use these, we need to guarantee that the result of the DKG will be used for a reasonably long time. It wouldn't be feasible to rerun the DKG every block. By epoching staking, it guarantees we'll only need to run a new DKG once every epoch. + +* Light client efficiency. This would lessen the overhead for IBC. Because of the lite client bisection algorithm, the number of headers you need to verify is related to bounding the validator set diffs between two successively verified headers. By limiting the frequency of validator set changes, we can reduce the size of IBC lite client proofs. + +* Fairness of deterministic leader election. Currently we have no ways of reasoning of fairness of deterministic leader election in the presence of staking changes without epochs (tendermint/spec#217). Adding epochs at least makes it easier for our deterministic leader election to match something we can prove secure. (Albeit, we still haven’t proven if our current algorithm is fair with > 2 validators) + +* Staking derivative design. Currently, reward distribution is done lazily using the F1 fee distribution. While saving computational complexity, lazy accounting increases “statefulness of staking”. Right now, each delegation entry has to track the time of last withdrawal. Handling this can be a challenge for some staking derivatives designs (see example). Force-withdrawing rewards to users can help solve this, however it is infeasible to force-withdraw rewards to users on a per block basis. With epoching, a chain could more easily alter the design to have rewards be forcefully withdrawn (iterating over delegator accounts only once per-epoch), and thus remove the time of delegation from state. This preliminarily seems like it may be of utility in certain staking derivative designs. + +## Design considerations + +### Slashing + +There is a design consideration for whether to apply a slash immediately or at the end of an epoch. A slash event should apply to only members who are actually staked during the time of the infraction, namely during the epoch the slash event occured. + +Applying it immediately can be viewed as offering greater consensus layer security, at potential costs to the aforementioned usecases. The benefits of immediate slashing for consensus layer security can be all be obtained by executing the validator jailing immediately (thus removing it from the validator set), and delaying the actual slash change to the validator's weight until the epoch boundary. For the use cases mentioned above, workarounds can be integrated to avoid problems, as follows: + +- For threshold based cryptography, it can keep using the original keep epoch weights for the cryptography thresholds, but allow the underlying finality to benefit from extra security more quickly. +- For light client efficiency, there can be a bit included in the header indicating an intra-epoch slash (ala https://github.com/tendermint/spec/issues/199). +- For fairness of deterministic leader election, this will cause problems with the formalization of it / proximity of implementation to formally provable spec. However, a less formal claim can be made that the amount lost due to the slash should hopefully outweigh slight biases into the leader election process. This claim is dubious with the presence of MEV, but potentially formal upperbounds on MEV for fairness here could be derived. +- For staking derivative design, this will not cause problems with the suggested design there, nor does it increase the stateful of staking. (As whether a slash has occured is fully queryable given the validator address) + +However, for achieving consensus layer security, it suffices to apply the validator jailing immediately, but still delay the actual slash changes to waiting until the end of the epoch. This largely mitigates the concern for the fairness of deterministic leader election as well, since that validator is removed the set being rotated from immediately. + +### Token lockup + +When someone makes a transaction to delegate, even though they are not immediately staked, their tokens should be moved into a pool managed by the staking module which will then be used at the end of an epoch. This prevents concerns where they stake, and then spend those tokens not realizing they were already allocated for staking, and thus having their staking tx fail. + +### Pipelining the epochs + +For threshold based cryptography in particular, we need a pipeline for epoch changes. This is because when we are in epoch N, we want the epoch N+1 weights to be fixed so that the validator set can do the DKG accordingly. So if we are currently in epoch N, the stake weights for epoch N+1 should already be fixed, and new stake changes should be getting applied to epoch N + 2. + +This can be handled by making a parameter for the epoch pipeline. This parameter should not be alterable except during hard forks, to mitigate implementation complexity of switching the pipeline length. + +### Rewards + +Even though all staking updates are applied at epoch boundaries, rewards can still be distributed immediately when they are claimed. This is because they do not affect the current stake weights, as we do not implement auto-bonding of rewards. If such a feature were to be implemented, it would have to be setup so that rewards are auto-bonded at the epoch boundary. + +## Decision + +__Step-1__: Implement buffering of all staking and slashing messages. + +First we create a pool for storing tokens that are being bonded, but should be applied at the epoch boundary called the `EpochDelegationPool`. Then, we have two separate queues, one for staking, one for slashing. We describe what happens on each message being delivered below: + +### Staking messages +- **MsgCreateValidator**: Move user's self-bond to `EpochDelegationPool` immediately. Queue a message for the epoch boundary to handle the self-bond, taking the funds from the `EpochDelegationPool`. If Epoch execution fail, return back funds from `EpochDelegationPool` to user's account. +- **MsgEditValidator**: Validate message and if valid queue the message for execution at the end of the Epoch. +- **MsgDelegate**: Move user's funds to `EpochDelegationPool` immediately. Queue a message for the epoch boundary to handle the delegation, taking the funds from the `EpochDelegationPool`. If Epoch execution fail, return back funds from `EpochDelegationPool` to user's account. +- **MsgBeginRedelegate**: Validate message and if valid queue the message for execution at the end of the Epoch. +- **MsgUndelegate**: Validate message and if valid queue the message for execution at the end of the Epoch. + +### Slashing messages +- **MsgUnjail**: Validate message and if valid queue the message for execution at the end of the Epoch. +- **Slash Event**: Whenever a slash event is created, it gets queued in the slashing module to apply at the end of the epoch. The queues should be setup such that this slash applies immediately. + +### Evidence Messages + - **MsgSubmitEvidence**: This gets executed immediately, and the validator gets jailed immediately. However in slashing, the actual slash event gets queued. + +Then we add methods to the end blockers, to ensure that at the epoch boundary the queues are cleared and delegation updates are applied. + + +__Step-2__: Implement querying of queued staking txs. + +When querying the staking activity of a given address, the status should return not only the amount of tokens staked, but also if there are any queued stake events for that address. This will require nodes supporting querying to either do some more indexing to have this be efficiently queryable, or to have transactions + +__Step-3__: Adjust gas + +Currently gas represents the cost of executing a transaction when its done immediately. (Merging together costs of p2p overhead, state access overhead, and computational overhead) However, now a transaction can cause computation in a future block, namely at the epoch boundary. + +To handle this, we should initially include parameters for estimating the amount of future computation (denominated in gas), and add that as a flat charge needed for the message. +We leave it as out of scope for how to weight future computation versus current computation in gas pricing, and have it set such that the are weighted equally for now. + +## Consequences + +### Positive + +* Abstracts the proof of stake module that allows retaining the existing functionality +* Enables new features such as validator-set based threshold cryptography + +### Negative + +* Increases complexity of integrating more complex gas pricing mechanisms, as they now have to consider future execution costs as well. From 846cede5bfc330826c72b776f73c17d718489238 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Mon, 22 Feb 2021 14:33:47 -0600 Subject: [PATCH 5/8] Repsond to comments, improve descriptions --- docs/architecture/adr-039-epoched-staking.md | 27 ++++++++++++-------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/docs/architecture/adr-039-epoched-staking.md b/docs/architecture/adr-039-epoched-staking.md index 9e7720627c..82d453b7a5 100644 --- a/docs/architecture/adr-039-epoched-staking.md +++ b/docs/architecture/adr-039-epoched-staking.md @@ -7,7 +7,7 @@ ## Authors - Dev Ojha (@valardragon) -- Sunny Aggarwall (@sunnya97) +- Sunny Aggarwal (@sunnya97) ## Status @@ -23,17 +23,17 @@ The current proof of stake module takes the design decision to apply staking wei An alternative design choice is to allow buffering staking updates (delegations, unbonds, validators joining) for a number of blocks. This 'epoch'd proof of stake consensus provides the guarantee that the consensus weights for validators will not change mid-epoch, except in the event of a slash condition. -The decision to have immediate execution of staking changes was primarily done as it was implementationally simplest, and because we at the time believed that this would lead to better UX for clients. The UX hurdle may not be as significant as was previously thought, since it is possible to provide users acknowledgement that their bond was recorded and will be executed. +Additionally, the UX hurdle may not be as significant as was previously thought. This is because it is possible to provide users immediate acknowledgement that their bond was recorded and will be executed. Furthermore, it has become clearer over time that immediate execution of staking events comes with limitations, such as: * Threshold based cryptography. One of the main limitations is that because the validator set can change so regularly, it makes the running of multiparty computation by a fixed validator set difficult. Many threshold-based cryptographic features for blockchains such as randomness beacons and threshold decryption require a computationally-expensive DKG process (will take much longer than 1 block to create). To productively use these, we need to guarantee that the result of the DKG will be used for a reasonably long time. It wouldn't be feasible to rerun the DKG every block. By epoching staking, it guarantees we'll only need to run a new DKG once every epoch. -* Light client efficiency. This would lessen the overhead for IBC. Because of the lite client bisection algorithm, the number of headers you need to verify is related to bounding the validator set diffs between two successively verified headers. By limiting the frequency of validator set changes, we can reduce the size of IBC lite client proofs. +* Light client efficiency. This would lessen the overhead for IBC. Because of the light client bisection algorithm, the number of headers you need to verify is related to bounding the difference in validator sets between two successively verified headers. By limiting the frequency of validator set changes, we can reduce the size of IBC lite client proofs. -* Fairness of deterministic leader election. Currently we have no ways of reasoning of fairness of deterministic leader election in the presence of staking changes without epochs (tendermint/spec#217). Adding epochs at least makes it easier for our deterministic leader election to match something we can prove secure. (Albeit, we still haven’t proven if our current algorithm is fair with > 2 validators) +* Fairness of deterministic leader election. Currently we have no ways of reasoning of fairness of deterministic leader election in the presence of staking changes without epochs (tendermint/spec#217). Breaking fairness of leader election is profitable for validators, as they earn additional rewards from being the proposer. Adding epochs at least makes it easier for our deterministic leader election to match something we can prove secure. (Albeit, we still haven’t proven if our current algorithm is fair with > 2 validators in the presence of stake changes) -* Staking derivative design. Currently, reward distribution is done lazily using the F1 fee distribution. While saving computational complexity, lazy accounting increases “statefulness of staking”. Right now, each delegation entry has to track the time of last withdrawal. Handling this can be a challenge for some staking derivatives designs (see example). Force-withdrawing rewards to users can help solve this, however it is infeasible to force-withdraw rewards to users on a per block basis. With epoching, a chain could more easily alter the design to have rewards be forcefully withdrawn (iterating over delegator accounts only once per-epoch), and thus remove the time of delegation from state. This preliminarily seems like it may be of utility in certain staking derivative designs. +* Staking derivative design. Currently, reward distribution is done lazily using the F1 fee distribution. While saving computational complexity, lazy accounting requires a more stateful staking implementation. Right now, each delegation entry has to track the time of last withdrawal. Handling this can be a challenge for some staking derivatives designs that seek to provide fungibility for all tokens staked to a single validator. Force-withdrawing rewards to users can help solve this, however it is infeasible to force-withdraw rewards to users on a per block basis. With epochs, a chain could more easily alter the design to have rewards be forcefully withdrawn (iterating over delegator accounts only once per-epoch), and can thus remove delegation timing from state. This may be useful for certain staking derivative designs. ## Design considerations @@ -43,12 +43,12 @@ There is a design consideration for whether to apply a slash immediately or at t Applying it immediately can be viewed as offering greater consensus layer security, at potential costs to the aforementioned usecases. The benefits of immediate slashing for consensus layer security can be all be obtained by executing the validator jailing immediately (thus removing it from the validator set), and delaying the actual slash change to the validator's weight until the epoch boundary. For the use cases mentioned above, workarounds can be integrated to avoid problems, as follows: -- For threshold based cryptography, it can keep using the original keep epoch weights for the cryptography thresholds, but allow the underlying finality to benefit from extra security more quickly. +- For threshold based cryptography, this setting will have the threshold cryptography use the original epoch weights, while consensus has an update that lets it more rapidly benefit from additional security. If the threshold based cryptography blocks liveness of the chain, then we have effectively raised the liveness threshold of the remaining validators for the rest of the epoch. (Alternatively, jailed nodes could still contribute shares) This plan will fail in the extreme case that more than 1/3rd of the validators have been jailed within a single epoch. For such an extreme scenario, the chain already have its own custom incident response plan, and defining how to handle the threshold cryptography should be a part of that. - For light client efficiency, there can be a bit included in the header indicating an intra-epoch slash (ala https://github.com/tendermint/spec/issues/199). -- For fairness of deterministic leader election, this will cause problems with the formalization of it / proximity of implementation to formally provable spec. However, a less formal claim can be made that the amount lost due to the slash should hopefully outweigh slight biases into the leader election process. This claim is dubious with the presence of MEV, but potentially formal upperbounds on MEV for fairness here could be derived. +- For fairness of deterministic leader election, applying a slash or jailing within an epoch would break the guarantee we were seeking to provide. This then re-introduces a new (but significantly simpler) problem for trying to provide fairness guarantees. Namely, that validators can adversarially elect to remove themself from the set of proposers. From a security perspective, this could potentially be handled by two different mechanisms (or prove to still be too difficult to achieve). One is making a security statement acknowledging the ability for an adversary to force an ahead-of-time fixed threshold of users to drop out of the proposer set within an epoch. The second method would be to parameterize such that the cost of a slash within the epoch far outweights benefits due to being a proposer. However, this latter criterion is quite dubious, since being a proposer can have many advantageous side-effects in chains with complex state machines. (Namely, DeFi games such as Fomo3D) - For staking derivative design, this will not cause problems with the suggested design there, nor does it increase the stateful of staking. (As whether a slash has occured is fully queryable given the validator address) -However, for achieving consensus layer security, it suffices to apply the validator jailing immediately, but still delay the actual slash changes to waiting until the end of the epoch. This largely mitigates the concern for the fairness of deterministic leader election as well, since that validator is removed the set being rotated from immediately. +However, for achieving consensus layer security, it suffices to apply the validator jailing immediately, but still delay the actual slash changes to waiting until the end of the epoch. This largely mitigates the concern for fairness of deterministic leader election as well, since the validator is removed the set of potential proposers immediately. ### Token lockup @@ -58,7 +58,11 @@ When someone makes a transaction to delegate, even though they are not immediate For threshold based cryptography in particular, we need a pipeline for epoch changes. This is because when we are in epoch N, we want the epoch N+1 weights to be fixed so that the validator set can do the DKG accordingly. So if we are currently in epoch N, the stake weights for epoch N+1 should already be fixed, and new stake changes should be getting applied to epoch N + 2. -This can be handled by making a parameter for the epoch pipeline. This parameter should not be alterable except during hard forks, to mitigate implementation complexity of switching the pipeline length. +This can be handled by making a parameter for the epoch pipeline length. This parameter should not be alterable except during hard forks, to mitigate implementation complexity of switching the pipeline length. + + +With pipeline length 1, if I redelegate during epoch N, then my redelegation is applied prior to the beginning of epoch N+1. +With pipeline length 2, if I redelegate during epoch N, then my redelegation is applied prior to the beginning of epoch N+2. ### Rewards @@ -89,7 +93,9 @@ Then we add methods to the end blockers, to ensure that at the epoch boundary th __Step-2__: Implement querying of queued staking txs. -When querying the staking activity of a given address, the status should return not only the amount of tokens staked, but also if there are any queued stake events for that address. This will require nodes supporting querying to either do some more indexing to have this be efficiently queryable, or to have transactions +When querying the staking activity of a given address, the status should return not only the amount of tokens staked, but also if there are any queued stake events for that address. This will require more work to be done in the querying logic, to trace the queued upcoming staking events. + +As an initial implementation, this can be implemented as a linear search over all queued staking events, but eventually nodes that support querying should maintain an auxilliary hashmap to make these queries constant time. __Step-3__: Adjust gas @@ -108,3 +114,4 @@ We leave it as out of scope for how to weight future computation versus current ### Negative * Increases complexity of integrating more complex gas pricing mechanisms, as they now have to consider future execution costs as well. +* When epoch > 1, validators can no longer leave the network immediately, and must wait until an epoch boundary. From e858c5179991453116146ae81a3b8c8096550be2 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Mon, 22 Feb 2021 14:38:08 -0600 Subject: [PATCH 6/8] Missed a comment --- docs/architecture/adr-039-epoched-staking.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/architecture/adr-039-epoched-staking.md b/docs/architecture/adr-039-epoched-staking.md index 82d453b7a5..6443c7b61a 100644 --- a/docs/architecture/adr-039-epoched-staking.md +++ b/docs/architecture/adr-039-epoched-staking.md @@ -48,8 +48,6 @@ Applying it immediately can be viewed as offering greater consensus layer securi - For fairness of deterministic leader election, applying a slash or jailing within an epoch would break the guarantee we were seeking to provide. This then re-introduces a new (but significantly simpler) problem for trying to provide fairness guarantees. Namely, that validators can adversarially elect to remove themself from the set of proposers. From a security perspective, this could potentially be handled by two different mechanisms (or prove to still be too difficult to achieve). One is making a security statement acknowledging the ability for an adversary to force an ahead-of-time fixed threshold of users to drop out of the proposer set within an epoch. The second method would be to parameterize such that the cost of a slash within the epoch far outweights benefits due to being a proposer. However, this latter criterion is quite dubious, since being a proposer can have many advantageous side-effects in chains with complex state machines. (Namely, DeFi games such as Fomo3D) - For staking derivative design, this will not cause problems with the suggested design there, nor does it increase the stateful of staking. (As whether a slash has occured is fully queryable given the validator address) -However, for achieving consensus layer security, it suffices to apply the validator jailing immediately, but still delay the actual slash changes to waiting until the end of the epoch. This largely mitigates the concern for fairness of deterministic leader election as well, since the validator is removed the set of potential proposers immediately. - ### Token lockup When someone makes a transaction to delegate, even though they are not immediately staked, their tokens should be moved into a pool managed by the staking module which will then be used at the end of an epoch. This prevents concerns where they stake, and then spend those tokens not realizing they were already allocated for staking, and thus having their staking tx fail. From 649a4e603a0214f033a8626559ac9de9028f7dc2 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Mon, 22 Feb 2021 15:37:53 -0600 Subject: [PATCH 7/8] Add section for parameterizing the epoch length --- docs/architecture/adr-039-epoched-staking.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/architecture/adr-039-epoched-staking.md b/docs/architecture/adr-039-epoched-staking.md index 6443c7b61a..6ad61c8840 100644 --- a/docs/architecture/adr-039-epoched-staking.md +++ b/docs/architecture/adr-039-epoched-staking.md @@ -29,7 +29,7 @@ Furthermore, it has become clearer over time that immediate execution of staking * Threshold based cryptography. One of the main limitations is that because the validator set can change so regularly, it makes the running of multiparty computation by a fixed validator set difficult. Many threshold-based cryptographic features for blockchains such as randomness beacons and threshold decryption require a computationally-expensive DKG process (will take much longer than 1 block to create). To productively use these, we need to guarantee that the result of the DKG will be used for a reasonably long time. It wouldn't be feasible to rerun the DKG every block. By epoching staking, it guarantees we'll only need to run a new DKG once every epoch. -* Light client efficiency. This would lessen the overhead for IBC. Because of the light client bisection algorithm, the number of headers you need to verify is related to bounding the difference in validator sets between two successively verified headers. By limiting the frequency of validator set changes, we can reduce the size of IBC lite client proofs. +* Light client efficiency. This would lessen the overhead for IBC when there is high churn in the validator set. In the Tendermint light client bisection algorithm, the number of headers you need to verify is related to bounding the difference in validator sets between a trusted header and the latest header. If the difference is too great, you verify more header in between the two. By limiting the frequency of validator set changes, we can reduce the worst case size of IBC lite client proofs, which occurs when a validator set has high churn. * Fairness of deterministic leader election. Currently we have no ways of reasoning of fairness of deterministic leader election in the presence of staking changes without epochs (tendermint/spec#217). Breaking fairness of leader election is profitable for validators, as they earn additional rewards from being the proposer. Adding epochs at least makes it easier for our deterministic leader election to match something we can prove secure. (Albeit, we still haven’t proven if our current algorithm is fair with > 2 validators in the presence of stake changes) @@ -46,7 +46,7 @@ Applying it immediately can be viewed as offering greater consensus layer securi - For threshold based cryptography, this setting will have the threshold cryptography use the original epoch weights, while consensus has an update that lets it more rapidly benefit from additional security. If the threshold based cryptography blocks liveness of the chain, then we have effectively raised the liveness threshold of the remaining validators for the rest of the epoch. (Alternatively, jailed nodes could still contribute shares) This plan will fail in the extreme case that more than 1/3rd of the validators have been jailed within a single epoch. For such an extreme scenario, the chain already have its own custom incident response plan, and defining how to handle the threshold cryptography should be a part of that. - For light client efficiency, there can be a bit included in the header indicating an intra-epoch slash (ala https://github.com/tendermint/spec/issues/199). - For fairness of deterministic leader election, applying a slash or jailing within an epoch would break the guarantee we were seeking to provide. This then re-introduces a new (but significantly simpler) problem for trying to provide fairness guarantees. Namely, that validators can adversarially elect to remove themself from the set of proposers. From a security perspective, this could potentially be handled by two different mechanisms (or prove to still be too difficult to achieve). One is making a security statement acknowledging the ability for an adversary to force an ahead-of-time fixed threshold of users to drop out of the proposer set within an epoch. The second method would be to parameterize such that the cost of a slash within the epoch far outweights benefits due to being a proposer. However, this latter criterion is quite dubious, since being a proposer can have many advantageous side-effects in chains with complex state machines. (Namely, DeFi games such as Fomo3D) -- For staking derivative design, this will not cause problems with the suggested design there, nor does it increase the stateful of staking. (As whether a slash has occured is fully queryable given the validator address) +- For staking derivative design, there is no issue introduced. This does not increase the state size of staking records, since whether a slash has occured is fully queryable given the validator address. ### Token lockup @@ -66,6 +66,12 @@ With pipeline length 2, if I redelegate during epoch N, then my redelegation is Even though all staking updates are applied at epoch boundaries, rewards can still be distributed immediately when they are claimed. This is because they do not affect the current stake weights, as we do not implement auto-bonding of rewards. If such a feature were to be implemented, it would have to be setup so that rewards are auto-bonded at the epoch boundary. +### Parameterizing the epoch length + +When choosing the epoch length, there is a trade-off queued state/computation buildup, and countering the previously discussed limitations of immediate execution if they apply to a given chain. + +Until an ABCI mechanism for variable block times is introduced, it is ill-advised to be using high epoch lengths due to the computation buildup. This is because when a block's execution time is greater than the expected block time from Tendermint, rounds may increment. + ## Decision __Step-1__: Implement buffering of all staking and slashing messages. @@ -93,7 +99,7 @@ __Step-2__: Implement querying of queued staking txs. When querying the staking activity of a given address, the status should return not only the amount of tokens staked, but also if there are any queued stake events for that address. This will require more work to be done in the querying logic, to trace the queued upcoming staking events. -As an initial implementation, this can be implemented as a linear search over all queued staking events, but eventually nodes that support querying should maintain an auxilliary hashmap to make these queries constant time. +As an initial implementation, this can be implemented as a linear search over all queued staking events. However, for chains that need long epochs, they should eventually build additional support for nodes that support querying to be able to produce results in constant time. (This is do-able by maintaining an auxilliary hashmap for indexing upcoming staking events by address) __Step-3__: Adjust gas From d880d744984e519b199e107a078dd09bdb56c5f3 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Mon, 22 Feb 2021 17:39:50 -0600 Subject: [PATCH 8/8] Fix accidentally committed files --- .github/workflows/repo-sync.yml | 19 ---- docs/architecture/adr-033-epoched-staking.md | 111 ------------------- 2 files changed, 130 deletions(-) delete mode 100644 .github/workflows/repo-sync.yml delete mode 100644 docs/architecture/adr-033-epoched-staking.md diff --git a/.github/workflows/repo-sync.yml b/.github/workflows/repo-sync.yml deleted file mode 100644 index 7844b46fa8..0000000000 --- a/.github/workflows/repo-sync.yml +++ /dev/null @@ -1,19 +0,0 @@ -on: - schedule: - - cron: "*/15 * * * *" - workflow_dispatch: - -jobs: - repo-sync: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - with: - persist-credentials: false - - name: repo-sync - uses: repo-sync/github-sync@v2 - with: - source_repo: "https://github.com/cosmos/cosmos-sdk" - source_branch: "master" - destination_branch: "master" - github_token: ${{ secrets.PAT }} diff --git a/docs/architecture/adr-033-epoched-staking.md b/docs/architecture/adr-033-epoched-staking.md deleted file mode 100644 index 01929758be..0000000000 --- a/docs/architecture/adr-033-epoched-staking.md +++ /dev/null @@ -1,111 +0,0 @@ -# ADR 032: Typed Events - -## Changelog - -- 10-Feb-2021: Initial Draft - -## Authors - -- Dev Ojha (@valardragon) -- Sunny Aggarwall (@sunnya97) - -## Status - -Proposed - -## Abstract - -This ADR updates the proof of stake module to buffer the staking weight updates for a number of blocks before updating the consensus' staking weights. The length of the buffer is dubbed an epoch. The prior functionality of the staking module is then a special case of the abstracted module, with the epoch being set to 1 block. - -## Context - -The current proof of stake module takes the design decision to apply staking weight changes to the consensus engine immediately. This means that delegations and unbonds get applied immediately to the validator set. This decision was primarily done as it was implementationally simplest, and because we at the time believed that this would lead to better UX for clients. - -An alternative design choice is to allow buffering staking updates (delegations, unbonds, validators joining) for a number of blocks. This 'epoch'd proof of stake consensus provides the guarantee that the consensus weights for validators will not change mid-epoch, except in the event of a slash condition. - -The decision to have immediate execution of staking changes was primarily done as it was implementationally simplest, and because we at the time believed that this would lead to better UX for clients. The UX hurdle may not be as significant as was previously thought, since it is possible to provide users acknowledgement that their bond was recorded and will be executed. - -Furthermore, it has become clearer over time that immediate execution of staking events comes with limitations, such as: - -* Threshold based cryptography. One of the main limitations is that because the validator set can change so regularly, it makes the running of multiparty computation by a fixed validator set difficult. Many threshold-based cryptographic features for blockchains such as randomness beacons and threshold decryption require a computationally-expensive DKG process (will take much longer than 1 block to create). To productively use these, we need to guarantee that the result of the DKG will be used for a reasonably long time. It wouldn't be feasible to rerun the DKG every block. By epoching staking, it guarantees we'll only need to run a new DKG once every epoch. - -* Light client efficiency. This would lessen the overhead for IBC. Because of the lite client bisection algorithm, the number of headers you need to verify is related to bounding the validator set diffs between two successively verified headers. By limiting the frequency of validator set changes, we can reduce the size of IBC lite client proofs. - -* Fairness of deterministic leader election. Currently we have no ways of reasoning of fairness of deterministic leader election in the presence of staking changes without epochs (tendermint/spec#217). Adding epochs at least makes it easier for our deterministic leader election to match something we can prove secure. (Albeit, we still haven’t proven if our current algorithm is fair with > 2 validators) - -* Staking derivative design. Currently, reward distribution is done lazily using the F1 fee distribution. While saving computational complexity, lazy accounting increases “statefulness of staking”. Right now, each delegation entry has to track the time of last withdrawal. Handling this can be a challenge for some staking derivatives designs (see example). Force-withdrawing rewards to users can help solve this, however it is infeasible to force-withdraw rewards to users on a per block basis. With epoching, a chain could more easily alter the design to have rewards be forcefully withdrawn (iterating over delegator accounts only once per-epoch), and thus remove the time of delegation from state. This preliminarily seems like it may be of utility in certain staking derivative designs. - -## Design considerations - -### Slashing - -There is a design consideration for whether to apply a slash immediately or at the end of an epoch. A slash event should apply to only members who are actually staked during the time of the infraction, namely during the epoch the slash event occured. - -Applying it immediately can be viewed as offering greater consensus layer security, at potential costs to the aforementioned usecases. The benefits of immediate slashing for consensus layer security can be all be obtained by executing the validator jailing immediately (thus removing it from the validator set), and delaying the actual slash change to the validator's weight until the epoch boundary. For the use cases mentioned above, workarounds can be integrated to avoid problems, as follows: - -- For threshold based cryptography, it can keep using the original keep epoch weights for the cryptography thresholds, but allow the underlying finality to benefit from extra security more quickly. -- For light client efficiency, there can be a bit included in the header indicating an intra-epoch slash (ala https://github.com/tendermint/spec/issues/199). -- For fairness of deterministic leader election, this will cause problems with the formalization of it / proximity of implementation to formally provable spec. However, a less formal claim can be made that the amount lost due to the slash should hopefully outweigh slight biases into the leader election process. This claim is dubious with the presence of MEV, but potentially formal upperbounds on MEV for fairness here could be derived. -- For staking derivative design, this will not cause problems with the suggested design there, nor does it increase the stateful of staking. (As whether a slash has occured is fully queryable given the validator address) - -However, for achieving consensus layer security, it suffices to apply the validator jailing immediately, but still delay the actual slash changes to waiting until the end of the epoch. This largely mitigates the concern for the fairness of deterministic leader election as well, since that validator is removed the set being rotated from immediately. - -### Token lockup - -When someone makes a transaction to delegate, even though they are not immediately staked, their tokens should be moved into a pool managed by the staking module which will then be used at the end of an epoch. This prevents concerns where they stake, and then spend those tokens not realizing they were already allocated for staking, and thus having their staking tx fail. - -### Pipelining the epochs - -For threshold based cryptography in particular, we need a pipeline for epoch changes. This is because when we are in epoch N, we want the epoch N+1 weights to be fixed so that the validator set can do the DKG accordingly. So if we are currently in epoch N, the stake weights for epoch N+1 should already be fixed, and new stake changes should be getting applied to epoch N + 2. - -This can be handled by making a parameter for the epoch pipeline. This parameter should not be alterable except during hard forks, to mitigate implementation complexity of switching the pipeline length. - -### Rewards - -Even though all staking updates are applied at epoch boundaries, rewards can still be distributed immediately when they are claimed. This is because they do not affect the current stake weights, as we do not implement auto-bonding of rewards. If such a feature were to be implemented, it would have to be setup so that rewards are auto-bonded at the epoch boundary. - -## Decision - -__Step-1__: Implement buffering of all staking and slashing messages. - -First we create a pool for storing tokens that are being bonded, but should be applied at the epoch boundary called the `EpochDelegationPool`. Then, we have two separate queues, one for staking, one for slashing. We describe what happens on each message being delivered below: - -### Staking messages -- **MsgCreateValidator**: Move user's self-bond to `EpochDelegationPool` immediately. Queue a message for the epoch boundary to handle the self-bond, taking the funds from the `EpochDelegationPool`. If Epoch execution fail, return back funds from `EpochDelegationPool` to user's account. -- **MsgEditValidator**: Validate message and if valid queue the message for execution at the end of the Epoch. -- **MsgDelegate**: Move user's funds to `EpochDelegationPool` immediately. Queue a message for the epoch boundary to handle the delegation, taking the funds from the `EpochDelegationPool`. If Epoch execution fail, return back funds from `EpochDelegationPool` to user's account. -- **MsgBeginRedelegate**: Validate message and if valid queue the message for execution at the end of the Epoch. -- **MsgUndelegate**: Validate message and if valid queue the message for execution at the end of the Epoch. - -### Slashing messages -- **MsgUnjail**: Validate message and if valid queue the message for execution at the end of the Epoch. -- **Slash Event**: Whenever a slash event is created, it gets queued in the slashing module to apply at the end of the epoch. The queues should be setup such that this slash applies immediately. - -### Evidence Messages - - **MsgSubmitEvidence**: This gets executed immediately, and the validator gets jailed immediately. However in slashing, the actual slash event gets queued. - -Then we add methods to the end blockers, to ensure that at the epoch boundary the queues are cleared and delegation updates are applied. - - -__Step-2__: Implement querying of queued staking txs. - -When querying the staking activity of a given address, the status should return not only the amount of tokens staked, but also if there are any queued stake events for that address. This will require nodes supporting querying to either do some more indexing to have this be efficiently queryable, or to have transactions - -__Step-3__: Adjust gas - -Currently gas represents the cost of executing a transaction when its done immediately. (Merging together costs of p2p overhead, state access overhead, and computational overhead) However, now a transaction can cause computation in a future block, namely at the epoch boundary. - -To handle this, we should initially include parameters for estimating the amount of future computation (denominated in gas), and add that as a flat charge needed for the message. -We leave it as out of scope for how to weight future computation versus current computation in gas pricing, and have it set such that the are weighted equally for now. - -## Consequences - -### Positive - -* Abstracts the proof of stake module that allows retaining the existing functionality -* Enables new features such as validator-set based threshold cryptography - -### Negative - -* Increases complexity of integrating more complex gas pricing mechanisms, as they now have to consider future execution costs as well. -