cmd/evm: implement input txs via rlp in t8n tool (#23138)

In many cases, it's desireable to use already-signed transactions as input to the state transition, instead of having the evm sign them internally (for example to use malformed or not-yet-valid transactions). This PR adds support + docs for that feature.

cmd/evm/README.md

@@ -4,15 +4,15 @@ The `evm t8n` tool is a stateless state transition utility. It is a utility
 which can
 
 1. Take a prestate, including
-  - Accounts,
+- Accounts,
-  - Block context information,
+- Block context information,
-  - Previous blockshashes (*optional)
+- Previous blockshashes (*optional)
 2. Apply a set of transactions,
 3. Apply a mining-reward (*optional),
 4. And generate a post-state, including
-  - State root, transaction root, receipt root,
+- State root, transaction root, receipt root,
-  - Information about rejected transactions,
+- Information about rejected transactions,
-  - Optionally: a full or partial post-state dump
+- Optionally: a full or partial post-state dump
 
 ## Specification
 
@@ -37,6 +37,8 @@ Command line params that has to be supported are
    --output.result result             Determines where to put the result (stateroot, txroot etc) of the post-state.
                                       `stdout` - into the stdout output
                                       `stderr` - into the stderr output
+   --output.body value                If set, the RLP of the transactions (block body) will be written to this file.
+   --input.txs stdin                  stdin or file name of where to find the transactions to apply. If the file prefix is '.rlp', then the data is interpreted as an RLP list of signed transactions.The '.rlp' format is identical to the output.body format. (default: "txs.json")
    --state.fork value                 Name of ruleset to use.
    --state.chainid value              ChainID to use (default: 1)
    --state.reward value               Mining reward. Set to -1 to disable (default: 0)
@@ -110,7 +112,10 @@ Two resulting files:
   }
  ],
  "rejected": [
-  1
+  {
+   "index": 1,
+   "error": "nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+  }
  ]
 }
 ```
@@ -156,7 +161,10 @@ Output:
    }
   ],
   "rejected": [
-   1
+   {
+    "index": 1,
+    "error": "nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+   }
   ]
  }
 }
@@ -168,9 +176,9 @@ Mining rewards and ommer rewards might need to be added. This is how those are a
 
 - `block_reward` is the block mining reward for the miner (`0xaa`), of a block at height `N`.
 - For each ommer (mined by `0xbb`), with blocknumber `N-delta`
-   - (where `delta` is the difference between the current block and the ommer)
+  - (where `delta` is the difference between the current block and the ommer)
-   - The account `0xbb` (ommer miner) is awarded `(8-delta)/ 8 * block_reward`
+  - The account `0xbb` (ommer miner) is awarded `(8-delta)/ 8 * block_reward`
-   - The account `0xaa` (block miner) is awarded `block_reward / 32`
+  - The account `0xaa` (block miner) is awarded `block_reward / 32`
 
 To make `state_t8n` apply these, the following inputs are required:
 
@@ -220,7 +228,7 @@ Output:
 ### Future EIPS
 
 It is also possible to experiment with future eips that are not yet defined in a hard fork.
-Example, putting EIP-1344 into Frontier: 
+Example, putting EIP-1344 into Frontier:
 ```
 ./evm t8n --state.fork=Frontier+1344 --input.pre=./testdata/1/pre.json --input.txs=./testdata/1/txs.json --input.env=/testdata/1/env.json
 ```
@@ -229,41 +237,102 @@ Example, putting EIP-1344 into Frontier:
 
 The `BLOCKHASH` opcode requires blockhashes to be provided by the caller, inside the `env`.
 If a required blockhash is not provided, the exit code should be `4`:
-Example where blockhashes are provided: 
+Example where blockhashes are provided:
 ```
-./evm t8n --input.alloc=./testdata/3/alloc.json --input.txs=./testdata/3/txs.json --input.env=./testdata/3/env.json --trace
+./evm --verbosity=1 t8n --input.alloc=./testdata/3/alloc.json --input.txs=./testdata/3/txs.json --input.env=./testdata/3/env.json  --trace
+INFO [07-27|11:53:40.960] Trie dumping started                     root=b7341d..857ea1
+INFO [07-27|11:53:40.960] Trie dumping complete                    accounts=3 elapsed="103.298µs"
+INFO [07-27|11:53:40.960] Wrote file                               file=alloc.json
+INFO [07-27|11:53:40.960] Wrote file                               file=result.json
+
 ```
+
 ```
 cat trace-0-0x72fadbef39cd251a437eea619cfeda752271a5faaaa2147df012e112159ffb81.jsonl | grep BLOCKHASH -C2
 ```
 ```
-{"pc":0,"op":96,"gas":"0x5f58ef8","gasCost":"0x3","memory":"0x","memSize":0,"stack":[],"returnStack":[],"returnData":"0x","depth":1,"refund":0,"opName":"PUSH1","error":""}
+{"pc":0,"op":96,"gas":"0x5f58ef8","gasCost":"0x3","memory":"0x","memSize":0,"stack":[],"returnData":"0x","depth":1,"refund":0,"opName":"PUSH1","error":""}
-{"pc":2,"op":64,"gas":"0x5f58ef5","gasCost":"0x14","memory":"0x","memSize":0,"stack":["0x1"],"returnStack":[],"returnData":"0x","depth":1,"refund":0,"opName":"BLOCKHASH","error":""}
+{"pc":2,"op":64,"gas":"0x5f58ef5","gasCost":"0x14","memory":"0x","memSize":0,"stack":["0x1"],"returnData":"0x","depth":1,"refund":0,"opName":"BLOCKHASH","error":""}
-{"pc":3,"op":0,"gas":"0x5f58ee1","gasCost":"0x0","memory":"0x","memSize":0,"stack":["0xdac58aa524e50956d0c0bae7f3f8bb9d35381365d07804dd5b48a5a297c06af4"],"returnStack":[],"returnData":"0x","depth":1,"refund":0,"opName":"STOP","error":""}
+{"pc":3,"op":0,"gas":"0x5f58ee1","gasCost":"0x0","memory":"0x","memSize":0,"stack":["0xdac58aa524e50956d0c0bae7f3f8bb9d35381365d07804dd5b48a5a297c06af4"],"returnData":"0x","depth":1,"refund":0,"opName":"STOP","error":""}
-{"output":"","gasUsed":"0x17","time":142709}
+{"output":"","gasUsed":"0x17","time":156276}
 ```
 
 In this example, the caller has not provided the required blockhash:
 ```
 ./evm t8n --input.alloc=./testdata/4/alloc.json --input.txs=./testdata/4/txs.json --input.env=./testdata/4/env.json --trace
-```
-```
 ERROR(4): getHash(3) invoked, blockhash for that block not provided
 ```
 Error code: 4
+
 ### Chaining
 
 Another thing that can be done, is to chain invocations:
 ```
 ./evm t8n --input.alloc=./testdata/1/alloc.json --input.txs=./testdata/1/txs.json --input.env=./testdata/1/env.json --output.alloc=stdout | ./evm t8n --input.alloc=stdin --input.env=./testdata/1/env.json --input.txs=./testdata/1/txs.json
-INFO [01-21|22:41:22.963] rejected tx                              index=1 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.049] rejected tx                              index=1 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
-INFO [01-21|22:41:22.966] rejected tx                              index=0 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.050] Trie dumping started                     root=84208a..ae4e13
-INFO [01-21|22:41:22.967] rejected tx                              index=1 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.050] Trie dumping complete                    accounts=3 elapsed="59.412µs"
+INFO [07-27|11:53:41.050] Wrote file                               file=result.json
+INFO [07-27|11:53:41.051] rejected tx                              index=0 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.051] rejected tx                              index=1 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.052] Trie dumping started                     root=84208a..ae4e13
+INFO [07-27|11:53:41.052] Trie dumping complete                    accounts=3 elapsed="45.734µs"
+INFO [07-27|11:53:41.052] Wrote file                               file=alloc.json
+INFO [07-27|11:53:41.052] Wrote file                               file=result.json
 
 ```
-What happened here, is that we first applied two identical transactions, so the second one was rejected. 
+What happened here, is that we first applied two identical transactions, so the second one was rejected.
 Then, taking the poststate alloc as the input for the next state, we tried again to include
 the same two transactions: this time, both failed due to too low nonce.
 
 In order to meaningfully chain invocations, one would need to provide meaningful new `env`, otherwise the
 actual blocknumber (exposed to the EVM) would not increase.
+
+### Transactions in RLP form
+
+It is possible to provide already-signed transactions as input to, using an `input.txs` which ends with the `rlp` suffix.
+The input format for RLP-form transactions is _identical_ to the _output_ format for block bodies. Therefore, it's fully possible
+to use the evm to go from `json` input to `rlp` input.
+
+The following command takes **json** the transactions in `./testdata/13/txs.json` and signs them. After execution, they are output to `signed_txs.rlp`.:
+```
+./evm t8n --state.fork=London --input.alloc=./testdata/13/alloc.json --input.txs=./testdata/13/txs.json --input.env=./testdata/13/env.json --output.result=alloc_jsontx.json --output.body=signed_txs.rlp
+INFO [07-27|11:53:41.124] Trie dumping started                     root=e4b924..6aef61
+INFO [07-27|11:53:41.124] Trie dumping complete                    accounts=3 elapsed="94.284µs"
+INFO [07-27|11:53:41.125] Wrote file                               file=alloc.json
+INFO [07-27|11:53:41.125] Wrote file                               file=alloc_jsontx.json
+INFO [07-27|11:53:41.125] Wrote file                               file=signed_txs.rlp
+
+```
+
+The `output.body` is the rlp-list of transactions, encoded in hex and placed in a string a'la `json` encoding rules:
+```
+cat signed_txs.rlp
+"0xf8d2b86702f864010180820fa08284d09411111111111111111111111111111111111111118080c001a0b7dfab36232379bb3d1497a4f91c1966b1f932eae3ade107bf5d723b9cb474e0a06261c359a10f2132f126d250485b90cf20f30340801244a08ef6142ab33d1904b86702f864010280820fa08284d09411111111111111111111111111111111111111118080c080a0d4ec563b6568cd42d998fc4134b36933c6568d01533b5adf08769270243c6c7fa072bf7c21eac6bbeae5143371eef26d5e279637f3bd73482b55979d76d935b1e9"
+```
+
+We can use `rlpdump` to check what the contents are:
+```
+rlpdump -hex $(cat signed_txs.rlp | jq -r )
+[
+  02f864010180820fa08284d09411111111111111111111111111111111111111118080c001a0b7dfab36232379bb3d1497a4f91c1966b1f932eae3ade107bf5d723b9cb474e0a06261c359a10f2132f126d250485b90cf20f30340801244a08ef6142ab33d1904,
+  02f864010280820fa08284d09411111111111111111111111111111111111111118080c080a0d4ec563b6568cd42d998fc4134b36933c6568d01533b5adf08769270243c6c7fa072bf7c21eac6bbeae5143371eef26d5e279637f3bd73482b55979d76d935b1e9,
+]
+```
+Now, we can now use those (or any other already signed transactions), as input, like so:
+```
+./evm t8n --state.fork=London --input.alloc=./testdata/13/alloc.json --input.txs=./signed_txs.rlp --input.env=./testdata/13/env.json --output.result=alloc_rlptx.json
+INFO [07-27|11:53:41.253] Trie dumping started                     root=e4b924..6aef61
+INFO [07-27|11:53:41.253] Trie dumping complete                    accounts=3 elapsed="128.445µs"
+INFO [07-27|11:53:41.253] Wrote file                               file=alloc.json
+INFO [07-27|11:53:41.255] Wrote file                               file=alloc_rlptx.json
+
+```
+
+You might have noticed that the results from these two invocations were stored in two separate files.
+And we can now finally check that they match.
+```
+cat alloc_jsontx.json | jq .stateRoot && cat alloc_rlptx.json | jq .stateRoot
+"0xe4b924a6adb5959fccf769d5b7bb2f6359e26d1e76a2443c5a91a36d826aef61"
+"0xe4b924a6adb5959fccf769d5b7bb2f6359e26d1e76a2443c5a91a36d826aef61"
+```

cmd/evm/internal/t8ntool/flags.go

@@ -79,8 +79,10 @@ var (
 		Value: "env.json",
 	}
 	InputTxsFlag = cli.StringFlag{
-		Name:  "input.txs",
+		Name: "input.txs",
-		Usage: "`stdin` or file name of where to find the transactions to apply.",
+		Usage: "`stdin` or file name of where to find the transactions to apply. " +
+			"If the file prefix is '.rlp', then the data is interpreted as an RLP list of signed transactions." +
+			"The '.rlp' format is identical to the output.body format.",
 		Value: "txs.json",
 	}
 	RewardFlag = cli.Int64Flag{

cmd/evm/internal/t8ntool/transition.go

@@ -25,6 +25,7 @@ import (
 	"math/big"
 	"os"
 	"path"
+	"strings"
 
 	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/common/hexutil"
@@ -72,6 +73,7 @@ type input struct {
 	Alloc core.GenesisAlloc `json:"alloc,omitempty"`
 	Env   *stEnv            `json:"env,omitempty"`
 	Txs   []*txWithKey      `json:"txs,omitempty"`
+	TxRlp string            `json:"txsRlp,omitempty"`
 }
 
 func Main(ctx *cli.Context) error {
@@ -199,11 +201,44 @@ func Main(ctx *cli.Context) error {
 		}
 		defer inFile.Close()
 		decoder := json.NewDecoder(inFile)
-		if err := decoder.Decode(&txsWithKeys); err != nil {
+		if strings.HasSuffix(txStr, ".rlp") {
-			return NewError(ErrorJson, fmt.Errorf("failed unmarshaling txs-file: %v", err))
+			var body hexutil.Bytes
+			if err := decoder.Decode(&body); err != nil {
+				return err
+			}
+			var txs types.Transactions
+			if err := rlp.DecodeBytes(body, &txs); err != nil {
+				return err
+			}
+			for _, tx := range txs {
+				txsWithKeys = append(txsWithKeys, &txWithKey{
+					key: nil,
+					tx:  tx,
+				})
+			}
+		} else {
+			if err := decoder.Decode(&txsWithKeys); err != nil {
+				return NewError(ErrorJson, fmt.Errorf("failed unmarshaling txs-file: %v", err))
+			}
 		}
 	} else {
-		txsWithKeys = inputData.Txs
+		if len(inputData.TxRlp) > 0 {
+			// Decode the body of already signed transactions
+			body := common.FromHex(inputData.TxRlp)
+			var txs types.Transactions
+			if err := rlp.DecodeBytes(body, &txs); err != nil {
+				return err
+			}
+			for _, tx := range txs {
+				txsWithKeys = append(txsWithKeys, &txWithKey{
+					key: nil,
+					tx:  tx,
+				})
+			}
+		} else {
+			// JSON encoded transactions
+			txsWithKeys = inputData.Txs
+		}
 	}
 	// We may have to sign the transactions.
 	signer := types.MakeSigner(chainConfig, big.NewInt(int64(prestate.Env.Number)))
@@ -365,6 +400,7 @@ func dispatchOutput(ctx *cli.Context, baseDir string, result *ExecutionResult, a
 			return NewError(ErrorJson, fmt.Errorf("failed marshalling output: %v", err))
 		}
 		os.Stdout.Write(b)
+		os.Stdout.Write([]byte("\n"))
 	}
 	if len(stdErrObject) > 0 {
 		b, err := json.MarshalIndent(stdErrObject, "", " ")
@@ -372,6 +408,7 @@ func dispatchOutput(ctx *cli.Context, baseDir string, result *ExecutionResult, a
 			return NewError(ErrorJson, fmt.Errorf("failed marshalling output: %v", err))
 		}
 		os.Stderr.Write(b)
+		os.Stderr.Write([]byte("\n"))
 	}
 	return nil
 }

cmd/evm/testdata/13/alloc.json

@@ -0,0 +1,23 @@
+{
+  "0x1111111111111111111111111111111111111111" : {
+    "balance" : "0x010000000000",
+    "code" : "0xfe",
+    "nonce" : "0x01",
+    "storage" : {
+    }
+  },
+  "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
+    "balance" : "0x010000000000",
+    "code" : "0x",
+    "nonce" : "0x01",
+    "storage" : {
+    }
+  },
+  "0xd02d72e067e77158444ef2020ff2d325f929b363" : {
+    "balance" : "0x01000000000000",
+    "code" : "0x",
+    "nonce" : "0x01",
+    "storage" : {
+    }
+  }
+}

cmd/evm/testdata/13/env.json

@@ -0,0 +1,12 @@
+{
+  "currentCoinbase" : "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
+  "currentDifficulty" : "0x020000",
+  "currentNumber" : "0x01",
+  "currentTimestamp" : "0x079e",
+  "previousHash" : "0xcb23ee65a163121f640673b41788ee94633941405f95009999b502eedfbbfd4f",
+  "currentGasLimit" : "0x40000000",
+  "currentBaseFee" : "0x036b",
+  "blockHashes" : {
+    "0" : "0xcb23ee65a163121f640673b41788ee94633941405f95009999b502eedfbbfd4f"
+  }
+}

cmd/evm/testdata/13/readme.md

@@ -0,0 +1,4 @@
+## Input transactions in RLP form
+
+This testdata folder is used to examplify how transaction input can be provided in rlp form. 
+Please see the README in `evm` folder for how this is performed. 

cmd/evm/testdata/13/txs.json

@@ -0,0 +1,34 @@
+[
+  {
+    "input" : "0x",
+    "gas" : "0x84d0",
+    "nonce" : "0x1",
+    "to" : "0x1111111111111111111111111111111111111111",
+    "value" : "0x0",
+    "v" : "0x0",
+    "r" : "0x0",
+    "s" : "0x0",
+    "secretKey" : "0x41f6e321b31e72173f8ff2e292359e1862f24fba42fe6f97efaf641980eff298",
+    "chainId" : "0x1",
+    "type" : "0x2",
+    "maxFeePerGas" : "0xfa0",
+    "maxPriorityFeePerGas" : "0x0",
+    "accessList" : []
+  },
+  {
+    "input" : "0x",
+    "gas" : "0x84d0",
+    "nonce" : "0x2",
+    "to" : "0x1111111111111111111111111111111111111111",
+    "value" : "0x0",
+    "v" : "0x0",
+    "r" : "0x0",
+    "s" : "0x0",
+    "secretKey" : "0x41f6e321b31e72173f8ff2e292359e1862f24fba42fe6f97efaf641980eff298",
+    "chainId" : "0x1",
+    "type" : "0x2",
+    "maxFeePerGas" : "0xfa0",
+    "maxPriorityFeePerGas" : "0x0",
+    "accessList" : []
+  }
+]

cmd/evm/transition-test.sh

@@ -11,6 +11,8 @@ function showjson(){
 function demo(){
   echo "$ticks"
   echo "$1"
+  $1
+  echo ""
   echo "$ticks"
   echo ""
 }
@@ -152,9 +154,7 @@ echo ""
 echo "The \`BLOCKHASH\` opcode requires blockhashes to be provided by the caller, inside the \`env\`."
 echo "If a required blockhash is not provided, the exit code should be \`4\`:"
 echo "Example where blockhashes are provided: "
-cmd="./evm t8n --input.alloc=./testdata/3/alloc.json --input.txs=./testdata/3/txs.json --input.env=./testdata/3/env.json  --trace"
+demo "./evm --verbosity=1 t8n --input.alloc=./testdata/3/alloc.json --input.txs=./testdata/3/txs.json --input.env=./testdata/3/env.json  --trace"
-tick && echo $cmd && tick
-$cmd 2>&1 >/dev/null
 cmd="cat trace-0-0x72fadbef39cd251a437eea619cfeda752271a5faaaa2147df012e112159ffb81.jsonl | grep BLOCKHASH -C2"
 tick && echo $cmd && tick
 echo "$ticks"
@@ -164,13 +164,11 @@ echo ""
 
 echo "In this example, the caller has not provided the required blockhash:"
 cmd="./evm t8n --input.alloc=./testdata/4/alloc.json --input.txs=./testdata/4/txs.json --input.env=./testdata/4/env.json  --trace"
-tick && echo $cmd && tick
+tick && echo $cmd && $cmd
-tick
-$cmd
 errc=$?
 tick
 echo "Error code: $errc"
-
+echo ""
 
 echo "### Chaining"
 echo ""
@@ -189,3 +187,28 @@ echo ""
 echo "In order to meaningfully chain invocations, one would need to provide meaningful new \`env\`, otherwise the"
 echo "actual blocknumber (exposed to the EVM) would not increase."
 echo ""
+
+echo "### Transactions in RLP form"
+echo ""
+echo "It is possible to provide already-signed transactions as input to, using an \`input.txs\` which ends with the \`rlp\` suffix."
+echo "The input format for RLP-form transactions is _identical_ to the _output_ format for block bodies. Therefore, it's fully possible"
+echo "to use the evm to go from \`json\` input to \`rlp\` input."
+echo ""
+echo "The following command takes **json** the transactions in \`./testdata/13/txs.json\` and signs them. After execution, they are output to \`signed_txs.rlp\`.:"
+demo "./evm t8n --state.fork=London --input.alloc=./testdata/13/alloc.json --input.txs=./testdata/13/txs.json --input.env=./testdata/13/env.json --output.result=alloc_jsontx.json --output.body=signed_txs.rlp"
+echo "The \`output.body\` is the rlp-list of transactions, encoded in hex and placed in a string a'la \`json\` encoding rules:"
+demo "cat signed_txs.rlp"
+echo "We can use \`rlpdump\` to check what the contents are: "
+echo "$ticks"
+echo "rlpdump -hex \$(cat signed_txs.rlp | jq -r )"
+rlpdump -hex $(cat signed_txs.rlp | jq -r )
+echo "$ticks"
+echo "Now, we can now use those (or any other already signed transactions), as input, like so: "
+demo "./evm t8n --state.fork=London --input.alloc=./testdata/13/alloc.json --input.txs=./signed_txs.rlp --input.env=./testdata/13/env.json --output.result=alloc_rlptx.json"
+
+echo "You might have noticed that the results from these two invocations were stored in two separate files. "
+echo "And we can now finally check that they match."
+echo "$ticks"
+echo "cat alloc_jsontx.json | jq .stateRoot && cat alloc_rlptx.json | jq .stateRoot"
+cat alloc_jsontx.json | jq .stateRoot && cat alloc_rlptx.json | jq .stateRoot
+echo "$ticks"