Merge pull request #5331 from filecoin-project/raulk-patch-1

README: refine Contribute section.

CHANGELOG.md

@@ -1,5 +1,18 @@
 # Lotus changelog
 
+# 1.4.0 / 2020-12-19
+
+This is a MANDATORY hotfix release of Lotus that resolves a chain halt at height 336,459 caused by nondeterminism in specs-actors. The fix is to update actors to 2.3.3 in order to incorporate this fix https://github.com/filecoin-project/specs-actors/pull/1334.
+
+# 1.3.0 / 2020-12-16
+
+This is a mandatory release of Lotus that introduces the third post-liftoff upgrade to the Filecoin network. The network upgrade occurs at height 343200, before which time all nodes must have updated to this release (or later). The change that breaks consensus is an implementation of FIP-0009(https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0009.md).
+
+## Changes
+
+- Disable gas burning for window post messages (https://github.com/filecoin-project/lotus/pull/5200)
+- fix lock propose (https://github.com/filecoin-project/lotus/pull/5197)
+
 # 1.2.3 / 2020-12-15
 
 This is an optional Lotus release that introduces many performance improvements, bugfixes, and UX improvements.
@@ -120,7 +133,7 @@ This is a very small release of Lotus that fixes an issue users are experiencing
 
 # 1.2.0 / 2020-11-18
 
-This is a mandatory release of Lotus that introduces the second post-liftoff upgrade to the Filecoin network. The network upgrade occurs at height 265200, before which time all nodes must have update to this release (or later). This release also bumps the required version of Go to 1.15.
+This is a mandatory release of Lotus that introduces the second post-liftoff upgrade to the Filecoin network. The network upgrade occurs at height 265200, before which time all nodes must have updated to this release (or later). This release also bumps the required version of Go to 1.15.
 
 The changes that break consensus are:
 

Makefile

@@ -63,6 +63,9 @@ debug: lotus lotus-miner lotus-worker lotus-seed
 2k: GOFLAGS+=-tags=2k
 2k: lotus lotus-miner lotus-worker lotus-seed
 
+calibnet: GOFLAGS+=-tags=calibnet
+calibnet: lotus lotus-miner lotus-worker lotus-seed
+
 lotus: $(BUILD_DEPS)
 	rm -f lotus
 	go build $(GOFLAGS) -o lotus ./cmd/lotus

README.md

@@ -24,24 +24,31 @@ For instructions on how to build, install and setup lotus, please visit [https:/
 
 Please send an email to security@filecoin.org. See our [security policy](SECURITY.md) for more details.
 
-## Development
+## Related packages
 
-The main branches under development at the moment are:
-* [`master`](https://github.com/filecoin-project/lotus): current testnet.
-* [`next`](https://github.com/filecoin-project/lotus/tree/next): working branch with chain-breaking changes.
-* [`ntwk-calibration`](https://github.com/filecoin-project/lotus/tree/ntwk-calibration): devnet running one of `next` commits.
+These repos are independent and reusable modules, but are tightly integrated into Lotus to make up a fully featured Filecoin implementation:
 
-### Tracker
-
-All work is tracked via issues. An attempt at keeping an up-to-date view on remaining work towards Mainnet launch can be seen at the [lotus github project board](https://github.com/orgs/filecoin-project/projects/8). The issues labeled with `incentives` are there to identify the issues needed for Space Race launch.
-
-### Packages
-
-The lotus Filecoin implementation unfolds into the following packages:
-
-- [This repo](https://github.com/filecoin-project/lotus)
 - [go-fil-markets](https://github.com/filecoin-project/go-fil-markets) which has its own [kanban work tracker available here](https://app.zenhub.com/workspaces/markets-shared-components-5daa144a7046a60001c6e253/board)
-- [spec-actors](https://github.com/filecoin-project/specs-actors) which has its own [kanban work tracker available here](https://app.zenhub.com/workspaces/actors-5ee6f3aa87591f0016c05685/board)
+- [specs-actors](https://github.com/filecoin-project/specs-actors) which has its own [kanban work tracker available here](https://app.zenhub.com/workspaces/actors-5ee6f3aa87591f0016c05685/board)
+
+## Contribute
+
+Lotus is a universally open project and welcomes contributions of all kinds: code, docs, and more. However, before making a contribution, we ask you to heed these recommendations:
+
+1. If the proposal entails a protocol change, please first submit a [Filecoin Improvement Proposal](https://github.com/filecoin-project/FIPs).
+2. If the change is complex and requires prior discussion, [open an issue](github.com/filecoin-project/lotus/issues) or a [discussion](https://github.com/filecoin-project/lotus/discussions) to request feedback before you start working on a pull request. This is to avoid disappointment and sunk costs, in case the change is not actually needed or accepted.
+3. Please refrain from submitting PRs to adapt existing code to subjective preferences. The changeset should contain functional or technical improvements/enhancements, bug fixes, new features, or some other clear material contribution. Simple stylistic changes are likely to be rejected in order to reduce code churn.
+
+When implementing a change:
+
+1. Adhere to the standard Go formatting guidelines, e.g. [Effective Go](https://golang.org/doc/effective_go.html). Run `go fmt`.
+2. Stick to the idioms and patterns used in the codebase. Familiar-looking code has a higher chance of being accepted than eerie code. Pay attention to commonly used variable and parameter names, avoidance of naked returns, error handling patterns, etc.
+3. Comments: follow the advice on the [Commentary](https://golang.org/doc/effective_go.html#commentary) section of Effective Go.
+4. Minimize code churn. Modify only what is strictly necessary. Well-encapsulated changesets will get a quicker response from maintainers.
+5. Lint your code with [`golangci-lint`](https://golangci-lint.run) (CI will reject your PR if unlinted).
+6. Add tests.
+7. Title the PR in a meaningful way and describe the rationale and the thought process in the PR description.
+8. Write clean, thoughtful, and detailed [commit messages](https://chris.beams.io/posts/git-commit/). This is even more important than the PR description, because commit messages are stored _inside_ the Git history. One good rule is: if you are happy posting the commit message as the PR description, then it's a good commit message.
 
 ## License
 

SECURITY.md

@@ -2,11 +2,11 @@
 
 ## Reporting a Vulnerability
 
-For *critical* bugs, please consult our Security Policy and Responsible Disclosure Program information at https://github.com/filecoin-project/community/blob/master/SECURITY.md
+For reporting security vulnerabilities/bugs, please consult our Security Policy and Responsible Disclosure Program information at https://github.com/filecoin-project/community/blob/master/SECURITY.md. Security vulnerabilities should be reported via our [Vulnerability Reporting channels](https://github.com/filecoin-project/community/blob/master/SECURITY.md#vulnerability-reporting) and will be eligible for a [Bug Bounty](https://security.filecoin.io/bug-bounty/).
 
 Please try to provide a clear description of any bugs reported, along with how to reproduce the bug if possible. More detailed bug reports (especially those with a PoC included) will help us move forward much faster. Additionally, please avoid reporting bugs that already have open issues. Take a moment to search the issue list of the related GitHub repositories before writing up a new report.
 
-Here are some examples of bugs we would consider 'critical':
+Here are some examples of bugs we would consider to be security vulnerabilities:
 
 * If you can spend from a `multisig` wallet you do not control the keys for.
 * If you can cause a miner to be slashed without them actually misbehaving.
@@ -16,8 +16,8 @@ Here are some examples of bugs we would consider 'critical':
 * If you can craft a message that causes a persistent fork in the network.
 * If you can cause the total amount of Filecoin in the network to no longer be 2 billion.
 
-This is not an exhaustive list, but should provide some idea of what we consider 'critical'.
+This is not an exhaustive list, but should provide some idea of what we consider as a security vulnerability, . 
 
 ## Reporting a non security bug
 
-For non-critical bugs, please simply file a GitHub [issue](https://github.com/filecoin-project/lotus/issues/new?template=bug_report.md). 
+For non-security bugs, please simply file a GitHub [issue](https://github.com/filecoin-project/lotus/issues/new?template=bug_report.md). 

api/api_full.go

@@ -515,6 +515,10 @@ type FullNode interface {
 	// along with the address removal.
 	MsigRemoveSigner(ctx context.Context, msig address.Address, proposer address.Address, toRemove address.Address, decrease bool) (cid.Cid, error)
 
+	// MarketAddBalance adds funds to the market actor
+	MarketAddBalance(ctx context.Context, wallet, addr address.Address, amt types.BigInt) (cid.Cid, error)
+	// MarketGetReserved gets the amount of funds that are currently reserved for the address
+	MarketGetReserved(ctx context.Context, addr address.Address) (types.BigInt, error)
 	// MarketReserveFunds reserves funds for a deal
 	MarketReserveFunds(ctx context.Context, wallet address.Address, addr address.Address, amt types.BigInt) (cid.Cid, error)
 	// MarketReleaseFunds releases funds reserved by MarketReserveFunds

api/apistruct/struct.go

@@ -244,6 +244,8 @@ type FullNodeStruct struct {
 		MsigSwapCancel          func(context.Context, address.Address, address.Address, uint64, address.Address, address.Address) (cid.Cid, error)                               `perm:"sign"`
 		MsigRemoveSigner        func(ctx context.Context, msig address.Address, proposer address.Address, toRemove address.Address, decrease bool) (cid.Cid, error)              `perm:"sign"`
 
+		MarketAddBalance   func(ctx context.Context, wallet, addr address.Address, amt types.BigInt) (cid.Cid, error)                 `perm:"sign"`
+		MarketGetReserved  func(ctx context.Context, addr address.Address) (types.BigInt, error)                                      `perm:"sign"`
 		MarketReserveFunds func(ctx context.Context, wallet address.Address, addr address.Address, amt types.BigInt) (cid.Cid, error) `perm:"sign"`
 		MarketReleaseFunds func(ctx context.Context, addr address.Address, amt types.BigInt) error                                    `perm:"sign"`
 		MarketWithdraw     func(ctx context.Context, wallet, addr address.Address, amt types.BigInt) (cid.Cid, error)                 `perm:"sign"`
@@ -1148,6 +1150,14 @@ func (c *FullNodeStruct) MsigRemoveSigner(ctx context.Context, msig address.Addr
 	return c.Internal.MsigRemoveSigner(ctx, msig, proposer, toRemove, decrease)
 }
 
+func (c *FullNodeStruct) MarketAddBalance(ctx context.Context, wallet address.Address, addr address.Address, amt types.BigInt) (cid.Cid, error) {
+	return c.Internal.MarketAddBalance(ctx, wallet, addr, amt)
+}
+
+func (c *FullNodeStruct) MarketGetReserved(ctx context.Context, addr address.Address) (types.BigInt, error) {
+	return c.Internal.MarketGetReserved(ctx, addr)
+}
+
 func (c *FullNodeStruct) MarketReserveFunds(ctx context.Context, wallet address.Address, addr address.Address, amt types.BigInt) (cid.Cid, error) {
 	return c.Internal.MarketReserveFunds(ctx, wallet, addr, amt)
 }

build/bootstrap.go

@@ -2,11 +2,9 @@ package build
 
 import (
 	"context"
-	"os"
 	"strings"
 
 	"github.com/filecoin-project/lotus/lib/addrutil"
-	"golang.org/x/xerrors"
 
 	rice "github.com/GeertJohan/go.rice"
 	"github.com/libp2p/go-libp2p-core/peer"
@@ -17,24 +15,16 @@ func BuiltinBootstrap() ([]peer.AddrInfo, error) {
 		return nil, nil
 	}
 
-	var out []peer.AddrInfo
-
 	b := rice.MustFindBox("bootstrap")
-	err := b.Walk("", func(path string, info os.FileInfo, err error) error {
-		if err != nil {
-			return xerrors.Errorf("failed to walk box: %w", err)
+
+	if BootstrappersFile != "" {
+		spi := b.MustString(BootstrappersFile)
+		if spi == "" {
+			return nil, nil
 		}
 
-		if !strings.HasSuffix(path, ".pi") {
-			return nil
-		}
-		spi := b.MustString(path)
-		if spi == "" {
-			return nil
-		}
-		pi, err := addrutil.ParseAddresses(context.TODO(), strings.Split(strings.TrimSpace(spi), "\n"))
-		out = append(out, pi...)
-		return err
-	})
-	return out, err
+		return addrutil.ParseAddresses(context.TODO(), strings.Split(strings.TrimSpace(spi), "\n"))
+	}
+
+	return nil, nil
 }

build/bootstrap/mainnet.pi

@@ -0,0 +1,15 @@
+/dns4/bootstrap-0.mainnet.filops.net/tcp/1347/p2p/12D3KooWCVe8MmsEMes2FzgTpt9fXtmCY7wrq91GRiaC8PHSCCBj
+/dns4/bootstrap-1.mainnet.filops.net/tcp/1347/p2p/12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc
+/dns4/bootstrap-2.mainnet.filops.net/tcp/1347/p2p/12D3KooWEWVwHGn2yR36gKLozmb4YjDJGerotAPGxmdWZx2nxMC4
+/dns4/bootstrap-3.mainnet.filops.net/tcp/1347/p2p/12D3KooWKhgq8c7NQ9iGjbyK7v7phXvG6492HQfiDaGHLHLQjk7R
+/dns4/bootstrap-4.mainnet.filops.net/tcp/1347/p2p/12D3KooWL6PsFNPhYftrJzGgF5U18hFoaVhfGk7xwzD8yVrHJ3Uc
+/dns4/bootstrap-5.mainnet.filops.net/tcp/1347/p2p/12D3KooWLFynvDQiUpXoHroV1YxKHhPJgysQGH2k3ZGwtWzR4dFH
+/dns4/bootstrap-6.mainnet.filops.net/tcp/1347/p2p/12D3KooWP5MwCiqdMETF9ub1P3MbCvQCcfconnYHbWg6sUJcDRQQ
+/dns4/bootstrap-7.mainnet.filops.net/tcp/1347/p2p/12D3KooWRs3aY1p3juFjPy8gPN95PEQChm2QKGUCAdcDCC4EBMKf
+/dns4/bootstrap-8.mainnet.filops.net/tcp/1347/p2p/12D3KooWScFR7385LTyR4zU1bYdzSiiAb5rnNABfVahPvVSzyTkR
+/dns4/lotus-bootstrap.forceup.cn/tcp/41778/p2p/12D3KooWFQsv3nRMUevZNWWsY1Wu6NUzUbawnWU5NcRhgKuJA37C
+/dns4/bootstrap-0.starpool.in/tcp/12757/p2p/12D3KooWGHpBMeZbestVEWkfdnC9u7p6uFHXL1n7m1ZBqsEmiUzz
+/dns4/bootstrap-1.starpool.in/tcp/12757/p2p/12D3KooWQZrGH1PxSNZPum99M1zNvjNFM33d1AAu5DcvdHptuU7u
+/dns4/node.glif.io/tcp/1235/p2p/12D3KooWBF8cpp65hp2u9LK5mh19x67ftAam84z9LsfaquTDSBpt
+/dns4/bootstrap-0.ipfsmain.cn/tcp/34721/p2p/12D3KooWQnwEGNqcM2nAcPtRR9rAX8Hrg4k9kJLCHoTR5chJfz6d
+/dns4/bootstrap-1.ipfsmain.cn/tcp/34723/p2p/12D3KooWMKxMkD5DMpSWsW7dBddKxKT7L2GgbNuckz9otxvkvByP

build/genesis.go

@@ -14,7 +14,7 @@ func MaybeGenesis() []byte {
 		log.Warnf("loading built-in genesis: %s", err)
 		return nil
 	}
-	genBytes, err := builtinGen.Bytes("devnet.car")
+	genBytes, err := builtinGen.Bytes(GenesisFile)
 	if err != nil {
 		log.Warnf("loading built-in genesis: %s", err)
 	}

build/isnearupgrade.go

@@ -0,0 +1,9 @@
+package build
+
+import (
+	"github.com/filecoin-project/go-state-types/abi"
+)
+
+func IsNearUpgrade(epoch, upgradeEpoch abi.ChainEpoch) bool {
+	return epoch > upgradeEpoch-Finality && epoch < upgradeEpoch+Finality
+}

build/params_2k.go

@@ -3,13 +3,13 @@
 package build
 
 import (
-	"math"
-	"os"
-
 	"github.com/filecoin-project/go-state-types/abi"
 	"github.com/filecoin-project/lotus/chain/actors/policy"
 )
 
+const BootstrappersFile = ""
+const GenesisFile = ""
+
 const UpgradeBreezeHeight = -1
 const BreezeGasTampingDuration = 0
 
@@ -18,12 +18,13 @@ const UpgradeIgnitionHeight = -2
 const UpgradeRefuelHeight = -3
 const UpgradeTapeHeight = -4
 
-var UpgradeActorsV2Height = abi.ChainEpoch(10)
-var UpgradeLiftoffHeight = abi.ChainEpoch(-5)
+const UpgradeActorsV2Height = 10
+const UpgradeLiftoffHeight = -5
 
 const UpgradeKumquatHeight = 15
 const UpgradeCalicoHeight = 20
 const UpgradePersianHeight = 25
+const UpgradeOrangeHeight = 27
 const UpgradeClausHeight = 30
 
 var DrandSchedule = map[abi.ChainEpoch]DrandEnum{
@@ -35,11 +36,6 @@ func init() {
 	policy.SetConsensusMinerMinPower(abi.NewStoragePower(2048))
 	policy.SetMinVerifiedDealSize(abi.NewStoragePower(256))
 
-	if os.Getenv("LOTUS_DISABLE_V2_ACTOR_MIGRATION") == "1" {
-		UpgradeActorsV2Height = math.MaxInt64
-		UpgradeLiftoffHeight = 11
-	}
-
 	BuildType |= Build2k
 }
 

build/params_calibnet.go

@@ -0,0 +1,65 @@
+// +build calibnet
+
+package build
+
+import (
+	"github.com/filecoin-project/go-address"
+	"github.com/filecoin-project/go-state-types/abi"
+	"github.com/filecoin-project/lotus/chain/actors/policy"
+	builtin2 "github.com/filecoin-project/specs-actors/v2/actors/builtin"
+)
+
+var DrandSchedule = map[abi.ChainEpoch]DrandEnum{
+	0: DrandMainnet,
+}
+
+const BootstrappersFile = "calibnet.pi"
+const GenesisFile = "calibnet.car"
+
+const UpgradeBreezeHeight = -1
+const BreezeGasTampingDuration = 120
+
+const UpgradeSmokeHeight = -2
+
+const UpgradeIgnitionHeight = -3
+const UpgradeRefuelHeight = -4
+
+var UpgradeActorsV2Height = abi.ChainEpoch(30)
+
+const UpgradeTapeHeight = 60
+
+// This signals our tentative epoch for mainnet launch. Can make it later, but not earlier.
+// Miners, clients, developers, custodians all need time to prepare.
+// We still have upgrades and state changes to do, but can happen after signaling timing here.
+const UpgradeLiftoffHeight = -5
+
+const UpgradeKumquatHeight = 90
+
+const UpgradeCalicoHeight = 92000
+const UpgradePersianHeight = UpgradeCalicoHeight + (builtin2.EpochsInHour * 60)
+
+// 2020-12-17T19:00:00Z
+const UpgradeClausHeight = 161386
+
+// 2021-01-17T19:00:00Z
+const UpgradeOrangeHeight = 250666
+
+func init() {
+	policy.SetConsensusMinerMinPower(abi.NewStoragePower(10 << 30))
+	policy.SetSupportedProofTypes(
+		abi.RegisteredSealProof_StackedDrg512MiBV1,
+		abi.RegisteredSealProof_StackedDrg32GiBV1,
+		abi.RegisteredSealProof_StackedDrg64GiBV1,
+	)
+
+	SetAddressNetwork(address.Testnet)
+
+	Devnet = true
+}
+
+const BlockDelaySecs = uint64(builtin2.EpochDurationSeconds)
+
+const PropagationDelaySecs = uint64(6)
+
+// BootstrapPeerThreshold is the minimum number peers we need to track for a sync worker to start
+const BootstrapPeerThreshold = 4

build/params_mainnet.go

@@ -1,10 +1,13 @@
 // +build !debug
 // +build !2k
 // +build !testground
+// +build !calibnet
 
 package build
 
 import (
+	"os"
+
 	"github.com/filecoin-project/go-address"
 	"github.com/filecoin-project/go-state-types/abi"
 	"github.com/filecoin-project/lotus/chain/actors/policy"
@@ -12,45 +15,49 @@ import (
 )
 
 var DrandSchedule = map[abi.ChainEpoch]DrandEnum{
-	0: DrandMainnet,
+	0:                  DrandIncentinet,
+	UpgradeSmokeHeight: DrandMainnet,
 }
 
-const UpgradeBreezeHeight = -1
+const BootstrappersFile = "mainnet.pi"
+const GenesisFile = "mainnet.car"
+
+const UpgradeBreezeHeight = 41280
+
 const BreezeGasTampingDuration = 120
 
-const UpgradeSmokeHeight = -2
+const UpgradeSmokeHeight = 51000
 
-const UpgradeIgnitionHeight = -3
-const UpgradeRefuelHeight = -4
+const UpgradeIgnitionHeight = 94000
+const UpgradeRefuelHeight = 130800
 
-var UpgradeActorsV2Height = abi.ChainEpoch(30)
+const UpgradeActorsV2Height = 138720
 
-const UpgradeTapeHeight = 60
+const UpgradeTapeHeight = 140760
 
 // This signals our tentative epoch for mainnet launch. Can make it later, but not earlier.
 // Miners, clients, developers, custodians all need time to prepare.
 // We still have upgrades and state changes to do, but can happen after signaling timing here.
-const UpgradeLiftoffHeight = -5
+const UpgradeLiftoffHeight = 148888
 
-const UpgradeKumquatHeight = 90
+const UpgradeKumquatHeight = 170000
 
-const UpgradeCalicoHeight = 92000
+const UpgradeCalicoHeight = 265200
 const UpgradePersianHeight = UpgradeCalicoHeight + (builtin2.EpochsInHour * 60)
 
-// 2020-12-17T19:00:00Z
-const UpgradeClausHeight = 161386
+const UpgradeOrangeHeight = 336458
+
+// 2020-12-22T02:00:00Z
+const UpgradeClausHeight = 343200
 
 func init() {
-	policy.SetConsensusMinerMinPower(abi.NewStoragePower(10 << 30))
-	policy.SetSupportedProofTypes(
-		abi.RegisteredSealProof_StackedDrg512MiBV1,
-		abi.RegisteredSealProof_StackedDrg32GiBV1,
-		abi.RegisteredSealProof_StackedDrg64GiBV1,
-	)
+	policy.SetConsensusMinerMinPower(abi.NewStoragePower(10 << 40))
 
-	SetAddressNetwork(address.Testnet)
+	if os.Getenv("LOTUS_USE_TEST_ADDRESSES") != "1" {
+		SetAddressNetwork(address.Mainnet)
+	}
 
-	Devnet = true
+	Devnet = false
 }
 
 const BlockDelaySecs = uint64(builtin2.EpochDurationSeconds)

build/params_shared_vals.go

@@ -25,7 +25,7 @@ const UnixfsLinksPerLevel = 1024
 // Consensus / Network
 
 const AllowableClockDriftSecs = uint64(1)
-const NewestNetworkVersion = network.Version8
+const NewestNetworkVersion = network.Version9
 const ActorUpgradeNetworkVersion = network.Version4
 
 // Epochs

build/params_testground.go

@@ -91,16 +91,20 @@ var (
 	UpgradeCalicoHeight   abi.ChainEpoch = -7
 	UpgradePersianHeight  abi.ChainEpoch = -8
 	UpgradeClausHeight    abi.ChainEpoch = -9
+	UpgradeOrangeHeight   abi.ChainEpoch = -10
 
 	DrandSchedule = map[abi.ChainEpoch]DrandEnum{
 		0: DrandMainnet,
 	}
 
-	NewestNetworkVersion       = network.Version8
+	NewestNetworkVersion       = network.Version9
 	ActorUpgradeNetworkVersion = network.Version4
 
 	Devnet      = true
 	ZeroAddress = MustParseAddress("f3yaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaby2smx7a")
+
+	BootstrappersFile = ""
+	GenesisFile       = ""
 )
 
 const BootstrapPeerThreshold = 1

build/version.go

@@ -29,7 +29,7 @@ func buildType() string {
 }
 
 // BuildVersion is the local build version, set by build system
-const BuildVersion = "1.2.3"
+const BuildVersion = "1.4.0"
 
 func UserVersion() string {
 	return BuildVersion + buildType() + CurrentCommit

chain/actors/version.go

@@ -18,7 +18,7 @@ func VersionForNetwork(version network.Version) Version {
 	switch version {
 	case network.Version0, network.Version1, network.Version2, network.Version3:
 		return Version0
-	case network.Version4, network.Version5, network.Version6, network.Version7, network.Version8:
+	case network.Version4, network.Version5, network.Version6, network.Version7, network.Version8, network.Version9:
 		return Version2
 	default:
 		panic(fmt.Sprintf("unsupported network version %d", version))

chain/exchange/server.go

@@ -56,7 +56,11 @@ func (s *server) HandleStream(stream inet.Stream) {
 	}
 
 	_ = stream.SetDeadline(time.Now().Add(WriteResDeadline))
-	if err := cborutil.WriteCborRPC(stream, resp); err != nil {
+	buffered := bufio.NewWriter(stream)
+	if err = cborutil.WriteCborRPC(buffered, resp); err == nil {
+		err = buffered.Flush()
+	}
+	if err != nil {
 		_ = stream.SetDeadline(time.Time{})
 		log.Warnw("failed to write back response for handle stream",
 			"err", err, "peer", stream.Conn().RemotePeer())

chain/gen/slashfilter/slashfilter.go

@@ -3,6 +3,8 @@ package slashfilter
 import (
 	"fmt"
 
+	"github.com/filecoin-project/lotus/build"
+
 	"golang.org/x/xerrors"
 
 	"github.com/ipfs/go-cid"
@@ -26,6 +28,10 @@ func New(dstore ds.Batching) *SlashFilter {
 }
 
 func (f *SlashFilter) MinedBlock(bh *types.BlockHeader, parentEpoch abi.ChainEpoch) error {
+	if build.IsNearUpgrade(bh.Height, build.UpgradeOrangeHeight) {
+		return nil
+	}
+
 	epochKey := ds.NewKey(fmt.Sprintf("/%s/%d", bh.Miner, bh.Height))
 	{
 		// double-fork mining (2 blocks at one epoch)

chain/market/fundmanager.go

@@ -2,6 +2,7 @@ package market
 
 import (
 	"context"
+	"fmt"
 	"sync"
 
 	"github.com/filecoin-project/go-address"
@@ -129,6 +130,11 @@ func (fm *FundManager) Withdraw(ctx context.Context, wallet, addr address.Addres
 	return fm.getFundedAddress(addr).withdraw(ctx, wallet, amt)
 }
 
+// GetReserved returns the amount that is currently reserved for the address
+func (fm *FundManager) GetReserved(addr address.Address) abi.TokenAmount {
+	return fm.getFundedAddress(addr).getReserved()
+}
+
 // FundedAddressState keeps track of the state of an address with funds in the
 // datastore
 type FundedAddressState struct {
@@ -147,7 +153,7 @@ type fundedAddress struct {
 	env *fundManagerEnvironment
 	str *Store
 
-	lk    sync.Mutex
+	lk    sync.RWMutex
 	state *FundedAddressState
 
 	// Note: These request queues are ephemeral, they are not saved to store
@@ -183,6 +189,13 @@ func (a *fundedAddress) start() {
 	}
 }
 
+func (a *fundedAddress) getReserved() abi.TokenAmount {
+	a.lk.RLock()
+	defer a.lk.RUnlock()
+
+	return a.state.AmtReserved
+}
+
 func (a *fundedAddress) reserve(ctx context.Context, wallet address.Address, amt abi.TokenAmount) (cid.Cid, error) {
 	return a.requestAndWait(ctx, wallet, amt, &a.reservations)
 }
@@ -501,7 +514,13 @@ func (a *fundedAddress) processWithdrawals(withdrawals []*fundRequest) (msgCid c
 		// request with an error
 		newWithdrawalAmt := types.BigAdd(withdrawalAmt, amt)
 		if newWithdrawalAmt.GreaterThan(netAvail) {
-			err := xerrors.Errorf("insufficient funds for withdrawal of %d", amt)
+			msg := fmt.Sprintf("insufficient funds for withdrawal of %s: ", types.FIL(amt))
+			msg += fmt.Sprintf("net available (%s) = available (%s) - reserved (%s)",
+				types.FIL(types.BigSub(netAvail, withdrawalAmt)), types.FIL(avail), types.FIL(a.state.AmtReserved))
+			if !withdrawalAmt.IsZero() {
+				msg += fmt.Sprintf(" - queued withdrawals (%s)", types.FIL(withdrawalAmt))
+			}
+			err := xerrors.Errorf(msg)
 			a.debugf("%s", err)
 			req.Complete(cid.Undef, err)
 			continue

chain/stmgr/forks.go

@@ -96,6 +96,10 @@ func DefaultUpgradeSchedule() UpgradeSchedule {
 		Height:    build.UpgradePersianHeight,
 		Network:   network.Version8,
 		Migration: nil,
+	}, {
+		Height:    build.UpgradeOrangeHeight,
+		Network:   network.Version9,
+		Migration: nil,
 	}}
 
 	if build.UpgradeActorsV2Height == math.MaxInt64 { // disable actors upgrade

chain/sync.go

@@ -250,18 +250,6 @@ func (syncer *Syncer) InformNewHead(from peer.ID, fts *store.FullTipSet) bool {
 
 	syncer.incoming.Pub(fts.TipSet().Blocks(), LocalIncoming)
 
-	if from == syncer.self {
-		// TODO: this is kindof a hack...
-		log.Debug("got block from ourselves")
-
-		if err := syncer.Sync(ctx, fts.TipSet()); err != nil {
-			log.Errorf("failed to sync our own block %s: %+v", fts.TipSet().Cids(), err)
-			return false
-		}
-
-		return true
-	}
-
 	// TODO: IMPORTANT(GARBAGE) this needs to be put in the 'temporary' side of
 	// the blockstore
 	if err := syncer.store.PersistBlockHeaders(fts.TipSet().Blocks()...); err != nil {
@@ -1449,7 +1437,7 @@ func (syncer *Syncer) syncFork(ctx context.Context, incoming *types.TipSet, know
 		return nil, ErrForkCheckpoint
 	}
 
-	// TODO: Does this mean we always ask for ForkLengthThreshold blocks from the network, even if we just need, like, 2?
+	// TODO: Does this mean we always ask for ForkLengthThreshold blocks from the network, even if we just need, like, 2? Yes.
 	// Would it not be better to ask in smaller chunks, given that an ~ForkLengthThreshold is very rare?
 	tips, err := syncer.Exchange.GetBlocks(ctx, incoming.Parents(), int(build.ForkLengthThreshold))
 	if err != nil {
@@ -1460,6 +1448,10 @@ func (syncer *Syncer) syncFork(ctx context.Context, incoming *types.TipSet, know
 	if err != nil {
 		return nil, xerrors.Errorf("failed to load next local tipset: %w", err)
 	}
+	// Track the fork length on our side of the synced chain to enforce
+	// `ForkLengthThreshold`. Initialized to 1 because we already walked back
+	// one tipset from `known` (our synced head).
+	forkLengthInHead := 1
 
 	for cur := 0; cur < len(tips); {
 		if nts.Height() == 0 {
@@ -1476,6 +1468,13 @@ func (syncer *Syncer) syncFork(ctx context.Context, incoming *types.TipSet, know
 		if nts.Height() < tips[cur].Height() {
 			cur++
 		} else {
+			// Walk back one block in our synced chain to try to meet the fork's
+			// height.
+			forkLengthInHead++
+			if forkLengthInHead > int(build.ForkLengthThreshold) {
+				return nil, ErrForkTooLong
+			}
+
 			// We will be forking away from nts, check that it isn't checkpointed
 			if nts.Key() == chkpt {
 				return nil, ErrForkCheckpoint

chain/types/fil.go

@@ -26,7 +26,7 @@ func (f FIL) Unitless() string {
 var unitPrefixes = []string{"a", "f", "p", "n", "μ", "m"}
 
 func (f FIL) Short() string {
-	n := BigInt(f)
+	n := BigInt(f).Abs()
 
 	dn := uint64(1)
 	var prefix string
@@ -70,7 +70,7 @@ func (f FIL) UnmarshalText(text []byte) error {
 }
 
 func ParseFIL(s string) (FIL, error) {
-	suffix := strings.TrimLeft(s, ".1234567890")
+	suffix := strings.TrimLeft(s, "-.1234567890")
 	s = s[:len(s)-len(suffix)]
 	var attofil bool
 	if suffix != "" {

chain/types/fil_test.go

@@ -57,6 +57,52 @@ func TestFilShort(t *testing.T) {
 		{fil: "0.000221234", expect: "221.234 μFIL"},
 		{fil: "0.0002212344", expect: "221.234 μFIL"},
 		{fil: "0.00022123444", expect: "221.234 μFIL"},
+
+		{fil: "-1", expect: "-1 FIL"},
+		{fil: "-1.1", expect: "-1.1 FIL"},
+		{fil: "-12", expect: "-12 FIL"},
+		{fil: "-123", expect: "-123 FIL"},
+		{fil: "-123456", expect: "-123456 FIL"},
+		{fil: "-123.23", expect: "-123.23 FIL"},
+		{fil: "-123456.234", expect: "-123456.234 FIL"},
+		{fil: "-123456.2341234", expect: "-123456.234 FIL"},
+		{fil: "-123456.234123445", expect: "-123456.234 FIL"},
+
+		{fil: "-0.1", expect: "-100 mFIL"},
+		{fil: "-0.01", expect: "-10 mFIL"},
+		{fil: "-0.001", expect: "-1 mFIL"},
+
+		{fil: "-0.0001", expect: "-100 μFIL"},
+		{fil: "-0.00001", expect: "-10 μFIL"},
+		{fil: "-0.000001", expect: "-1 μFIL"},
+
+		{fil: "-0.0000001", expect: "-100 nFIL"},
+		{fil: "-0.00000001", expect: "-10 nFIL"},
+		{fil: "-0.000000001", expect: "-1 nFIL"},
+
+		{fil: "-0.0000000001", expect: "-100 pFIL"},
+		{fil: "-0.00000000001", expect: "-10 pFIL"},
+		{fil: "-0.000000000001", expect: "-1 pFIL"},
+
+		{fil: "-0.0000000000001", expect: "-100 fFIL"},
+		{fil: "-0.00000000000001", expect: "-10 fFIL"},
+		{fil: "-0.000000000000001", expect: "-1 fFIL"},
+
+		{fil: "-0.0000000000000001", expect: "-100 aFIL"},
+		{fil: "-0.00000000000000001", expect: "-10 aFIL"},
+		{fil: "-0.000000000000000001", expect: "-1 aFIL"},
+
+		{fil: "-0.0000012", expect: "-1.2 μFIL"},
+		{fil: "-0.00000123", expect: "-1.23 μFIL"},
+		{fil: "-0.000001234", expect: "-1.234 μFIL"},
+		{fil: "-0.0000012344", expect: "-1.234 μFIL"},
+		{fil: "-0.00000123444", expect: "-1.234 μFIL"},
+
+		{fil: "-0.0002212", expect: "-221.2 μFIL"},
+		{fil: "-0.00022123", expect: "-221.23 μFIL"},
+		{fil: "-0.000221234", expect: "-221.234 μFIL"},
+		{fil: "-0.0002212344", expect: "-221.234 μFIL"},
+		{fil: "-0.00022123444", expect: "-221.234 μFIL"},
 	} {
 		s := s
 		t.Run(s.fil, func(t *testing.T) {

chain/vm/syscalls.go

@@ -17,6 +17,7 @@ import (
 	"github.com/filecoin-project/go-state-types/abi"
 	"github.com/filecoin-project/go-state-types/crypto"
 	"github.com/filecoin-project/go-state-types/network"
+	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/actors/adt"
 	"github.com/filecoin-project/lotus/chain/actors/builtin/miner"
 	"github.com/filecoin-project/lotus/chain/actors/policy"
@@ -107,11 +108,18 @@ func (ss *syscallShim) VerifyConsensusFault(a, b, extra []byte) (*runtime2.Conse
 		return nil, xerrors.Errorf("cannot decode second block header: %f", decodeErr)
 	}
 
+	// workaround chain halt
+	if build.IsNearUpgrade(blockA.Height, build.UpgradeOrangeHeight) {
+		return nil, xerrors.Errorf("consensus reporting disabled around Upgrade Orange")
+	}
+	if build.IsNearUpgrade(blockB.Height, build.UpgradeOrangeHeight) {
+		return nil, xerrors.Errorf("consensus reporting disabled around Upgrade Orange")
+	}
+
 	// are blocks the same?
 	if blockA.Cid().Equals(blockB.Cid()) {
 		return nil, fmt.Errorf("no consensus fault: submitted blocks are the same")
 	}
-
 	// (1) check conditions necessary to any consensus fault
 
 	// were blocks mined by same miner?

cli/chain.go

@@ -1105,10 +1105,6 @@ var slashConsensusFault = &cli.Command{
 	Usage:     "Report consensus fault",
 	ArgsUsage: "[blockCid1 blockCid2]",
 	Flags: []cli.Flag{
-		&cli.StringFlag{
-			Name:  "miner",
-			Usage: "Miner address",
-		},
 		&cli.StringFlag{
 			Name:  "extra",
 			Usage: "Extra block cid",
@@ -1142,6 +1138,10 @@ var slashConsensusFault = &cli.Command{
 			return xerrors.Errorf("getting block 2: %w", err)
 		}
 
+		if b1.Miner != b2.Miner {
+			return xerrors.Errorf("block1.miner:%s block2.miner:%s", b1.Miner, b2.Miner)
+		}
+
 		def, err := api.WalletDefaultAddress(ctx)
 		if err != nil {
 			return err
@@ -1186,17 +1186,8 @@ var slashConsensusFault = &cli.Command{
 			return err
 		}
 
-		if cctx.String("miner") == "" {
-			return xerrors.Errorf("--miner flag is required")
-		}
-
-		maddr, err := address.NewFromString(cctx.String("miner"))
-		if err != nil {
-			return err
-		}
-
 		msg := &types.Message{
-			To:     maddr,
+			To:     b2.Miner,
 			From:   def,
 			Value:  types.NewInt(0),
 			Method: builtin.MethodsMiner.ReportConsensusFault,

cli/client.go

@@ -91,7 +91,7 @@ var clientCmd = &cli.Command{
 		WithCategory("retrieval", clientRetrieveCmd),
 		WithCategory("util", clientCommPCmd),
 		WithCategory("util", clientCarGenCmd),
-		WithCategory("util", clientInfoCmd),
+		WithCategory("util", clientBalancesCmd),
 		WithCategory("util", clientListTransfers),
 		WithCategory("util", clientRestartTransfer),
 		WithCategory("util", clientCancelTransfer),
@@ -1732,9 +1732,9 @@ var clientGetDealCmd = &cli.Command{
 	},
 }
 
-var clientInfoCmd = &cli.Command{
-	Name:  "info",
-	Usage: "Print storage market client information",
+var clientBalancesCmd = &cli.Command{
+	Name:  "balances",
+	Usage: "Print storage market client balances",
 	Flags: []cli.Flag{
 		&cli.StringFlag{
 			Name:  "client",
@@ -1751,7 +1751,7 @@ var clientInfoCmd = &cli.Command{
 
 		var addr address.Address
 		if clientFlag := cctx.String("client"); clientFlag != "" {
-			ca, err := address.NewFromString("client")
+			ca, err := address.NewFromString(clientFlag)
 			if err != nil {
 				return err
 			}
@@ -1770,10 +1770,22 @@ var clientInfoCmd = &cli.Command{
 			return err
 		}
 
-		fmt.Printf("Client Market Info:\n")
+		reserved, err := api.MarketGetReserved(ctx, addr)
+		if err != nil {
+			return err
+		}
 
-		fmt.Printf("Locked Funds:\t%s\n", types.FIL(balance.Locked))
-		fmt.Printf("Escrowed Funds:\t%s\n", types.FIL(balance.Escrow))
+		avail := big.Sub(big.Sub(balance.Escrow, balance.Locked), reserved)
+		if avail.LessThan(big.Zero()) {
+			avail = big.Zero()
+		}
+
+		fmt.Printf("Client Market Balance for address %s:\n", addr)
+
+		fmt.Printf("  Escrowed Funds:        %s\n", types.FIL(balance.Escrow))
+		fmt.Printf("  Locked Funds:          %s\n", types.FIL(balance.Locked))
+		fmt.Printf("  Reserved Funds:        %s\n", types.FIL(reserved))
+		fmt.Printf("  Available to Withdraw: %s\n", types.FIL(avail))
 
 		return nil
 	},
@@ -1943,6 +1955,11 @@ var clientListTransfers = &cli.Command{
 	Name:  "list-transfers",
 	Usage: "List ongoing data transfers for deals",
 	Flags: []cli.Flag{
+		&cli.BoolFlag{
+			Name:    "verbose",
+			Aliases: []string{"v"},
+			Usage:   "print verbose transfer details",
+		},
 		&cli.BoolFlag{
 			Name:  "color",
 			Usage: "use color in display output",
@@ -1974,6 +1991,7 @@ var clientListTransfers = &cli.Command{
 			return err
 		}
 
+		verbose := cctx.Bool("verbose")
 		completed := cctx.Bool("completed")
 		color := cctx.Bool("color")
 		watch := cctx.Bool("watch")
@@ -1989,7 +2007,7 @@ var clientListTransfers = &cli.Command{
 
 				tm.MoveCursor(1, 1)
 
-				OutputDataTransferChannels(tm.Screen, channels, completed, color, showFailed)
+				OutputDataTransferChannels(tm.Screen, channels, verbose, completed, color, showFailed)
 
 				tm.Flush()
 
@@ -2014,13 +2032,13 @@ var clientListTransfers = &cli.Command{
 				}
 			}
 		}
-		OutputDataTransferChannels(os.Stdout, channels, completed, color, showFailed)
+		OutputDataTransferChannels(os.Stdout, channels, verbose, completed, color, showFailed)
 		return nil
 	},
 }
 
 // OutputDataTransferChannels generates table output for a list of channels
-func OutputDataTransferChannels(out io.Writer, channels []lapi.DataTransferChannel, completed bool, color bool, showFailed bool) {
+func OutputDataTransferChannels(out io.Writer, channels []lapi.DataTransferChannel, verbose, completed, color, showFailed bool) {
 	sort.Slice(channels, func(i, j int) bool {
 		return channels[i].TransferID < channels[j].TransferID
 	})
@@ -2050,7 +2068,7 @@ func OutputDataTransferChannels(out io.Writer, channels []lapi.DataTransferChann
 		tablewriter.Col("Voucher"),
 		tablewriter.NewLineCol("Message"))
 	for _, channel := range sendingChannels {
-		w.Write(toChannelOutput(color, "Sending To", channel))
+		w.Write(toChannelOutput(color, "Sending To", channel, verbose))
 	}
 	w.Flush(out) //nolint:errcheck
 
@@ -2064,7 +2082,7 @@ func OutputDataTransferChannels(out io.Writer, channels []lapi.DataTransferChann
 		tablewriter.Col("Voucher"),
 		tablewriter.NewLineCol("Message"))
 	for _, channel := range receivingChannels {
-		w.Write(toChannelOutput(color, "Receiving From", channel))
+		w.Write(toChannelOutput(color, "Receiving From", channel, verbose))
 	}
 	w.Flush(out) //nolint:errcheck
 }
@@ -2085,9 +2103,13 @@ func channelStatusString(useColor bool, status datatransfer.Status) string {
 	}
 }
 
-func toChannelOutput(useColor bool, otherPartyColumn string, channel lapi.DataTransferChannel) map[string]interface{} {
-	rootCid := ellipsis(channel.BaseCID.String(), 8)
-	otherParty := ellipsis(channel.OtherPeer.String(), 8)
+func toChannelOutput(useColor bool, otherPartyColumn string, channel lapi.DataTransferChannel, verbose bool) map[string]interface{} {
+	rootCid := channel.BaseCID.String()
+	otherParty := channel.OtherPeer.String()
+	if !verbose {
+		rootCid = ellipsis(rootCid, 8)
+		otherParty = ellipsis(otherParty, 8)
+	}
 
 	initiated := "N"
 	if channel.IsInitiator {
@@ -2095,7 +2117,7 @@ func toChannelOutput(useColor bool, otherPartyColumn string, channel lapi.DataTr
 	}
 
 	voucher := channel.Voucher
-	if len(voucher) > 40 {
+	if len(voucher) > 40 && !verbose {
 		voucher = ellipsis(voucher, 37)
 	}
 

cli/mpool.go

@@ -362,15 +362,15 @@ var mpoolReplaceCmd = &cli.Command{
 	Flags: []cli.Flag{
 		&cli.StringFlag{
 			Name:  "gas-feecap",
-			Usage: "gas feecap for new message",
+			Usage: "gas feecap for new message (burn and pay to miner, attoFIL/GasUnit)",
 		},
 		&cli.StringFlag{
 			Name:  "gas-premium",
-			Usage: "gas price for new message",
+			Usage: "gas price for new message (pay to miner, attoFIL/GasUnit)",
 		},
 		&cli.Int64Flag{
 			Name:  "gas-limit",
-			Usage: "gas price for new message",
+			Usage: "gas limit for new message (GasUnit)",
 		},
 		&cli.BoolFlag{
 			Name:  "auto",
@@ -378,7 +378,7 @@ var mpoolReplaceCmd = &cli.Command{
 		},
 		&cli.StringFlag{
 			Name:  "max-fee",
-			Usage: "Spend up to X FIL for this message (applicable for auto mode)",
+			Usage: "Spend up to X attoFIL for this message (applicable for auto mode)",
 		},
 	},
 	ArgsUsage: "<from nonce> | <message-cid>",

cli/send.go

@@ -15,6 +15,7 @@ import (
 	"github.com/filecoin-project/go-state-types/abi"
 
 	"github.com/filecoin-project/lotus/api"
+	"github.com/filecoin-project/lotus/chain/actors/builtin"
 	"github.com/filecoin-project/lotus/chain/stmgr"
 	"github.com/filecoin-project/lotus/chain/types"
 )
@@ -51,7 +52,7 @@ var sendCmd = &cli.Command{
 		&cli.Uint64Flag{
 			Name:  "method",
 			Usage: "specify method to invoke",
-			Value: 0,
+			Value: uint64(builtin.MethodSend),
 		},
 		&cli.StringFlag{
 			Name:  "params-json",
@@ -61,6 +62,10 @@ var sendCmd = &cli.Command{
 			Name:  "params-hex",
 			Usage: "specify invocation parameters in hex",
 		},
+		&cli.BoolFlag{
+			Name:  "force",
+			Usage: "must be specified for the action to take effect if maybe SysErrInsufficientFunds etc",
+		},
 	},
 	Action: func(cctx *cli.Context) error {
 		if cctx.Args().Len() != 2 {
@@ -143,6 +148,20 @@ var sendCmd = &cli.Command{
 			Params:     params,
 		}
 
+		if !cctx.Bool("force") {
+			// Funds insufficient check
+			fromBalance, err := api.WalletBalance(ctx, msg.From)
+			if err != nil {
+				return err
+			}
+			totalCost := types.BigAdd(types.BigMul(msg.GasFeeCap, types.NewInt(uint64(msg.GasLimit))), msg.Value)
+
+			if fromBalance.LessThan(totalCost) {
+				fmt.Printf("WARNING: From balance %s less than total cost %s\n", types.FIL(fromBalance), types.FIL(totalCost))
+				return fmt.Errorf("--force must be specified for this action to have an effect; you have been warned")
+			}
+		}
+
 		if cctx.IsSet("nonce") {
 			msg.Nonce = cctx.Uint64("nonce")
 			sm, err := api.WalletSignMessage(ctx, fromAddr, msg)

cli/state.go

@@ -1617,7 +1617,7 @@ func parseParamsForMethod(act cid.Cid, method uint64, args []string) ([]byte, er
 		return nil, fmt.Errorf("unknown method %d for actor %s", method, act)
 	}
 
-	paramObj := methodMeta.Params
+	paramObj := methodMeta.Params.Elem()
 	if paramObj.NumField() != len(args) {
 		return nil, fmt.Errorf("not enough arguments given to call that method (expecting %d)", paramObj.NumField())
 	}

cli/wallet.go

@@ -509,6 +509,7 @@ var walletMarket = &cli.Command{
 	Usage: "Interact with market balances",
 	Subcommands: []*cli.Command{
 		walletMarketWithdraw,
+		walletMarketAdd,
 	},
 }
 
@@ -518,13 +519,13 @@ var walletMarketWithdraw = &cli.Command{
 	ArgsUsage: "[amount (FIL) optional, otherwise will withdraw max available]",
 	Flags: []cli.Flag{
 		&cli.StringFlag{
-			Name:    "from",
-			Usage:   "Specify address to withdraw funds from, otherwise it will use the default wallet address",
-			Aliases: []string{"f"},
+			Name:    "wallet",
+			Usage:   "Specify address to withdraw funds to, otherwise it will use the default wallet address",
+			Aliases: []string{"w"},
 		},
 		&cli.StringFlag{
 			Name:    "address",
-			Usage:   "Market address to withdraw from (account or miner actor address, defaults to --from address)",
+			Usage:   "Market address to withdraw from (account or miner actor address, defaults to --wallet address)",
 			Aliases: []string{"a"},
 		},
 	},
@@ -536,6 +537,123 @@ var walletMarketWithdraw = &cli.Command{
 		defer closer()
 		ctx := ReqContext(cctx)
 
+		var wallet address.Address
+		if cctx.String("wallet") != "" {
+			wallet, err = address.NewFromString(cctx.String("wallet"))
+			if err != nil {
+				return xerrors.Errorf("parsing from address: %w", err)
+			}
+		} else {
+			wallet, err = api.WalletDefaultAddress(ctx)
+			if err != nil {
+				return xerrors.Errorf("getting default wallet address: %w", err)
+			}
+		}
+
+		addr := wallet
+		if cctx.String("address") != "" {
+			addr, err = address.NewFromString(cctx.String("address"))
+			if err != nil {
+				return xerrors.Errorf("parsing market address: %w", err)
+			}
+		}
+
+		// Work out if there are enough unreserved, unlocked funds to withdraw
+		bal, err := api.StateMarketBalance(ctx, addr, types.EmptyTSK)
+		if err != nil {
+			return xerrors.Errorf("getting market balance for address %s: %w", addr.String(), err)
+		}
+
+		reserved, err := api.MarketGetReserved(ctx, addr)
+		if err != nil {
+			return xerrors.Errorf("getting market reserved amount for address %s: %w", addr.String(), err)
+		}
+
+		avail := big.Subtract(big.Subtract(bal.Escrow, bal.Locked), reserved)
+
+		notEnoughErr := func(msg string) error {
+			return xerrors.Errorf("%s; "+
+				"available (%s) = escrow (%s) - locked (%s) - reserved (%s)",
+				msg, types.FIL(avail), types.FIL(bal.Escrow), types.FIL(bal.Locked), types.FIL(reserved))
+		}
+
+		if avail.IsZero() || avail.LessThan(big.Zero()) {
+			avail = big.Zero()
+			return notEnoughErr("no funds available to withdraw")
+		}
+
+		// Default to withdrawing all available funds
+		amt := avail
+
+		// If there was an amount argument, only withdraw that amount
+		if cctx.Args().Present() {
+			f, err := types.ParseFIL(cctx.Args().First())
+			if err != nil {
+				return xerrors.Errorf("parsing 'amount' argument: %w", err)
+			}
+
+			amt = abi.TokenAmount(f)
+		}
+
+		// Check the amount is positive
+		if amt.IsZero() || amt.LessThan(big.Zero()) {
+			return xerrors.Errorf("amount must be > 0")
+		}
+
+		// Check there are enough available funds
+		if amt.GreaterThan(avail) {
+			msg := fmt.Sprintf("can't withdraw more funds than available; requested: %s", types.FIL(amt))
+			return notEnoughErr(msg)
+		}
+
+		fmt.Printf("Submitting WithdrawBalance message for amount %s for address %s\n", types.FIL(amt), wallet.String())
+		smsg, err := api.MarketWithdraw(ctx, wallet, addr, amt)
+		if err != nil {
+			return xerrors.Errorf("fund manager withdraw error: %w", err)
+		}
+
+		fmt.Printf("WithdrawBalance message cid: %s\n", smsg)
+
+		return nil
+	},
+}
+
+var walletMarketAdd = &cli.Command{
+	Name:      "add",
+	Usage:     "Add funds to the Storage Market Actor",
+	ArgsUsage: "<amount>",
+	Flags: []cli.Flag{
+		&cli.StringFlag{
+			Name:    "from",
+			Usage:   "Specify address to move funds from, otherwise it will use the default wallet address",
+			Aliases: []string{"f"},
+		},
+		&cli.StringFlag{
+			Name:    "address",
+			Usage:   "Market address to move funds to (account or miner actor address, defaults to --from address)",
+			Aliases: []string{"a"},
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		api, closer, err := GetFullNodeAPI(cctx)
+		if err != nil {
+			return xerrors.Errorf("getting node API: %w", err)
+		}
+		defer closer()
+		ctx := ReqContext(cctx)
+
+		// Get amount param
+		if !cctx.Args().Present() {
+			return fmt.Errorf("must pass amount to add")
+		}
+		f, err := types.ParseFIL(cctx.Args().First())
+		if err != nil {
+			return xerrors.Errorf("parsing 'amount' argument: %w", err)
+		}
+
+		amt := abi.TokenAmount(f)
+
+		// Get from param
 		var from address.Address
 		if cctx.String("from") != "" {
 			from, err = address.NewFromString(cctx.String("from"))
@@ -549,6 +667,7 @@ var walletMarketWithdraw = &cli.Command{
 			}
 		}
 
+		// Get address param
 		addr := from
 		if cctx.String("address") != "" {
 			addr, err = address.NewFromString(cctx.String("address"))
@@ -557,38 +676,14 @@ var walletMarketWithdraw = &cli.Command{
 			}
 		}
 
-		bal, err := api.StateMarketBalance(ctx, addr, types.EmptyTSK)
+		// Add balance to market actor
+		fmt.Printf("Submitting Add Balance message for amount %s for address %s\n", types.FIL(amt), addr)
+		smsg, err := api.MarketAddBalance(ctx, from, addr, amt)
 		if err != nil {
-			return xerrors.Errorf("getting market balance for address %s: %w", addr.String(), err)
+			return xerrors.Errorf("add balance error: %w", err)
 		}
 
-		avail := big.Subtract(bal.Escrow, bal.Locked)
-		amt := avail
-
-		if cctx.Args().Present() {
-			f, err := types.ParseFIL(cctx.Args().First())
-			if err != nil {
-				return xerrors.Errorf("parsing 'amount' argument: %w", err)
-			}
-
-			amt = abi.TokenAmount(f)
-		}
-
-		if amt.GreaterThan(avail) {
-			return xerrors.Errorf("can't withdraw more funds than available; requested: %s; available: %s", types.FIL(amt), types.FIL(avail))
-		}
-
-		if avail.IsZero() {
-			return xerrors.Errorf("zero unlocked funds available to withdraw")
-		}
-
-		fmt.Printf("Submitting WithdrawBalance message for amount %s for address %s\n", types.FIL(amt), from.String())
-		smsg, err := api.MarketWithdraw(ctx, from, addr, amt)
-		if err != nil {
-			return xerrors.Errorf("fund manager withdraw error: %w", err)
-		}
-
-		fmt.Printf("WithdrawBalance message cid: %s\n", smsg)
+		fmt.Printf("AddBalance message cid: %s\n", smsg)
 
 		return nil
 	},

cmd/chain-noise/main.go

@@ -27,6 +27,16 @@ func main() {
 				Hidden:  true,
 				Value:   "~/.lotus", // TODO: Consider XDG_DATA_HOME
 			},
+			&cli.IntFlag{
+				Name:  "limit",
+				Usage: "spam transaction count limit, <= 0 is no limit",
+				Value: 0,
+			},
+			&cli.IntFlag{
+				Name:  "rate",
+				Usage: "spam transaction rate, count per second",
+				Value: 5,
+			},
 		},
 		Commands: []*cli.Command{runCmd},
 	}
@@ -52,11 +62,17 @@ var runCmd = &cli.Command{
 		defer closer()
 		ctx := lcli.ReqContext(cctx)
 
-		return sendSmallFundsTxs(ctx, api, addr, 5)
+		rate := cctx.Int("rate")
+		if rate <= 0 {
+			rate = 5
+		}
+		limit := cctx.Int("limit")
+
+		return sendSmallFundsTxs(ctx, api, addr, rate, limit)
 	},
 }
 
-func sendSmallFundsTxs(ctx context.Context, api api.FullNode, from address.Address, rate int) error {
+func sendSmallFundsTxs(ctx context.Context, api api.FullNode, from address.Address, rate, limit int) error {
 	var sendSet []address.Address
 	for i := 0; i < 20; i++ {
 		naddr, err := api.WalletNew(ctx, types.KTSecp256k1)
@@ -66,9 +82,14 @@ func sendSmallFundsTxs(ctx context.Context, api api.FullNode, from address.Addre
 
 		sendSet = append(sendSet, naddr)
 	}
+	count := limit
 
 	tick := build.Clock.Ticker(time.Second / time.Duration(rate))
 	for {
+		if count <= 0 && limit > 0 {
+			fmt.Printf("%d messages sent.\n", limit)
+			return nil
+		}
 		select {
 		case <-tick.C:
 			msg := &types.Message{
@@ -81,6 +102,7 @@ func sendSmallFundsTxs(ctx context.Context, api api.FullNode, from address.Addre
 			if err != nil {
 				return err
 			}
+			count--
 			fmt.Println("Message sent: ", smsg.Cid())
 		case <-ctx.Done():
 			return nil

cmd/lotus-bench/main.go

@@ -40,6 +40,8 @@ import (
 var log = logging.Logger("lotus-bench")
 
 type BenchResults struct {
+	EnvVar map[string]string
+
 	SectorSize   abi.SectorSize
 	SectorNumber int
 
@@ -446,6 +448,15 @@ var sealBenchCmd = &cli.Command{
 			bo.VerifyWindowPostHot = verifyWindowpost2.Sub(verifyWindowpost1)
 		}
 
+		bo.EnvVar = make(map[string]string)
+		for _, envKey := range []string{"BELLMAN_NO_GPU", "FIL_PROOFS_MAXIMIZE_CACHING", "FIL_PROOFS_USE_GPU_COLUMN_BUILDER",
+			"FIL_PROOFS_USE_GPU_TREE_BUILDER", "FIL_PROOFS_USE_MULTICORE_SDR", "BELLMAN_CUSTOM_GPU"} {
+			envValue, found := os.LookupEnv(envKey)
+			if found {
+				bo.EnvVar[envKey] = envValue
+			}
+		}
+
 		if c.Bool("json-out") {
 			data, err := json.MarshalIndent(bo, "", "  ")
 			if err != nil {
@@ -454,6 +465,10 @@ var sealBenchCmd = &cli.Command{
 
 			fmt.Println(string(data))
 		} else {
+			fmt.Println("environment variable list:")
+			for envKey, envValue := range bo.EnvVar {
+				fmt.Printf("%s=%s\n", envKey, envValue)
+			}
 			fmt.Printf("----\nresults (v28) SectorSize:(%d), SectorNumber:(%d)\n", sectorSize, sectorNumber)
 			if robench == "" {
 				fmt.Printf("seal: addPiece: %s (%s)\n", bo.SealingSum.AddPiece, bps(bo.SectorSize, bo.SectorNumber, bo.SealingSum.AddPiece))

cmd/lotus-storage-miner/market.go

@@ -744,6 +744,11 @@ var transfersListCmd = &cli.Command{
 	Name:  "list",
 	Usage: "List ongoing data transfers for this miner",
 	Flags: []cli.Flag{
+		&cli.BoolFlag{
+			Name:    "verbose",
+			Aliases: []string{"v"},
+			Usage:   "print verbose transfer details",
+		},
 		&cli.BoolFlag{
 			Name:  "color",
 			Usage: "use color in display output",
@@ -775,6 +780,7 @@ var transfersListCmd = &cli.Command{
 			return err
 		}
 
+		verbose := cctx.Bool("verbose")
 		completed := cctx.Bool("completed")
 		color := cctx.Bool("color")
 		watch := cctx.Bool("watch")
@@ -790,7 +796,7 @@ var transfersListCmd = &cli.Command{
 
 				tm.MoveCursor(1, 1)
 
-				lcli.OutputDataTransferChannels(tm.Screen, channels, completed, color, showFailed)
+				lcli.OutputDataTransferChannels(tm.Screen, channels, verbose, completed, color, showFailed)
 
 				tm.Flush()
 
@@ -815,7 +821,7 @@ var transfersListCmd = &cli.Command{
 				}
 			}
 		}
-		lcli.OutputDataTransferChannels(os.Stdout, channels, completed, color, showFailed)
+		lcli.OutputDataTransferChannels(os.Stdout, channels, verbose, completed, color, showFailed)
 		return nil
 	},
 }

cmd/lotus-storage-miner/proving.go

@@ -446,7 +446,7 @@ var provingCheckProvableCmd = &cli.Command{
 		for parIdx, par := range partitions {
 			sectors := make(map[abi.SectorNumber]struct{})
 
-			sectorInfos, err := api.StateMinerSectors(ctx, addr, &par.AllSectors, types.EmptyTSK)
+			sectorInfos, err := api.StateMinerSectors(ctx, addr, &par.LiveSectors, types.EmptyTSK)
 			if err != nil {
 				return err
 			}

cmd/tvx/codenames_test.go

@@ -18,7 +18,7 @@ func TestProtocolCodenames(t *testing.T) {
 		t.Fatal("expected breeze codename")
 	}
 
-	if height := build.UpgradeActorsV2Height + 1; GetProtocolCodename(height) != "actorsv2" {
+	if height := build.UpgradeActorsV2Height + 1; GetProtocolCodename(abi.ChainEpoch(height)) != "actorsv2" {
 		t.Fatal("expected actorsv2 codename")
 	}
 

cmd/tvx/exec.go

@@ -1,33 +1,48 @@
 package main
 
 import (
+	"bufio"
 	"encoding/json"
 	"fmt"
 	"io"
 	"log"
 	"os"
+	"path/filepath"
+	"strings"
 
 	"github.com/fatih/color"
+	"github.com/filecoin-project/go-address"
+	cbornode "github.com/ipfs/go-ipld-cbor"
 	"github.com/urfave/cli/v2"
 
-	"github.com/filecoin-project/lotus/conformance"
-
 	"github.com/filecoin-project/test-vectors/schema"
+
+	"github.com/filecoin-project/lotus/chain/state"
+	"github.com/filecoin-project/lotus/chain/types"
+	"github.com/filecoin-project/lotus/conformance"
+	"github.com/filecoin-project/lotus/lib/blockstore"
 )
 
 var execFlags struct {
 	file               string
+	out                string
+	driverOpts         cli.StringSlice
 	fallbackBlockstore bool
 }
 
+const (
+	optSaveBalances = "save-balances"
+)
+
 var execCmd = &cli.Command{
 	Name:        "exec",
-	Description: "execute one or many test vectors against Lotus; supplied as a single JSON file, or a ndjson stdin stream",
-	Action:      runExecLotus,
+	Description: "execute one or many test vectors against Lotus; supplied as a single JSON file, a directory, or a ndjson stdin stream",
+	Action:      runExec,
 	Flags: []cli.Flag{
+		&repoFlag,
 		&cli.StringFlag{
 			Name:        "file",
-			Usage:       "input file; if not supplied, the vector will be read from stdin",
+			Usage:       "input file or directory; if not supplied, the vector will be read from stdin",
 			TakesFile:   true,
 			Destination: &execFlags.file,
 		},
@@ -36,10 +51,20 @@ var execCmd = &cli.Command{
 			Usage:       "sets the full node API as a fallback blockstore; use this if you're transplanting vectors and get block not found errors",
 			Destination: &execFlags.fallbackBlockstore,
 		},
+		&cli.StringFlag{
+			Name:        "out",
+			Usage:       "output directory where to save the results, only used when the input is a directory",
+			Destination: &execFlags.out,
+		},
+		&cli.StringSliceFlag{
+			Name:        "driver-opt",
+			Usage:       "comma-separated list of driver options (EXPERIMENTAL; will change), supported: 'save-balances=<dst>', 'pipeline-basefee' (unimplemented); only available in single-file mode",
+			Destination: &execFlags.driverOpts,
+		},
 	},
 }
 
-func runExecLotus(c *cli.Context) error {
+func runExec(c *cli.Context) error {
 	if execFlags.fallbackBlockstore {
 		if err := initialize(c); err != nil {
 			return fmt.Errorf("fallback blockstore was enabled, but could not resolve lotus API endpoint: %w", err)
@@ -48,30 +73,97 @@ func runExecLotus(c *cli.Context) error {
 		conformance.FallbackBlockstoreGetter = FullAPI
 	}
 
-	if file := execFlags.file; file != "" {
-		// we have a single test vector supplied as a file.
-		file, err := os.Open(file)
-		if err != nil {
-			return fmt.Errorf("failed to open test vector: %w", err)
-		}
-
-		var (
-			dec = json.NewDecoder(file)
-			tv  schema.TestVector
-		)
-
-		if err = dec.Decode(&tv); err != nil {
-			return fmt.Errorf("failed to decode test vector: %w", err)
-		}
-
-		return executeTestVector(tv)
+	path := execFlags.file
+	if path == "" {
+		return execVectorsStdin()
 	}
 
+	fi, err := os.Stat(path)
+	if err != nil {
+		return err
+	}
+
+	if fi.IsDir() {
+		// we're in directory mode; ensure the out directory exists.
+		outdir := execFlags.out
+		if outdir == "" {
+			return fmt.Errorf("no output directory provided")
+		}
+		if err := ensureDir(outdir); err != nil {
+			return err
+		}
+		return execVectorDir(path, outdir)
+	}
+
+	// process tipset vector options.
+	if err := processTipsetOpts(); err != nil {
+		return err
+	}
+
+	_, err = execVectorFile(new(conformance.LogReporter), path)
+	return err
+}
+
+func processTipsetOpts() error {
+	for _, opt := range execFlags.driverOpts.Value() {
+		switch ss := strings.Split(opt, "="); {
+		case ss[0] == optSaveBalances:
+			filename := ss[1]
+			log.Printf("saving balances after each tipset in: %s", filename)
+			balancesFile, err := os.Create(filename)
+			if err != nil {
+				return err
+			}
+			w := bufio.NewWriter(balancesFile)
+			cb := func(bs blockstore.Blockstore, params *conformance.ExecuteTipsetParams, res *conformance.ExecuteTipsetResult) {
+				cst := cbornode.NewCborStore(bs)
+				st, err := state.LoadStateTree(cst, res.PostStateRoot)
+				if err != nil {
+					return
+				}
+				_ = st.ForEach(func(addr address.Address, actor *types.Actor) error {
+					_, err := fmt.Fprintln(w, params.ExecEpoch, addr, actor.Balance)
+					return err
+				})
+				_ = w.Flush()
+			}
+			conformance.TipsetVectorOpts.OnTipsetApplied = append(conformance.TipsetVectorOpts.OnTipsetApplied, cb)
+
+		}
+
+	}
+	return nil
+}
+
+func execVectorDir(path string, outdir string) error {
+	files, err := filepath.Glob(filepath.Join(path, "*"))
+	if err != nil {
+		return fmt.Errorf("failed to glob input directory %s: %w", path, err)
+	}
+	for _, f := range files {
+		outfile := strings.TrimSuffix(filepath.Base(f), filepath.Ext(f)) + ".out"
+		outpath := filepath.Join(outdir, outfile)
+		outw, err := os.Create(outpath)
+		if err != nil {
+			return fmt.Errorf("failed to create file %s: %w", outpath, err)
+		}
+
+		log.Printf("processing vector %s; sending output to %s", f, outpath)
+		log.SetOutput(io.MultiWriter(os.Stderr, outw)) // tee the output.
+		_, _ = execVectorFile(new(conformance.LogReporter), f)
+		log.SetOutput(os.Stderr)
+		_ = outw.Close()
+	}
+	return nil
+}
+
+func execVectorsStdin() error {
+	r := new(conformance.LogReporter)
 	for dec := json.NewDecoder(os.Stdin); ; {
 		var tv schema.TestVector
 		switch err := dec.Decode(&tv); err {
 		case nil:
-			if err = executeTestVector(tv); err != nil {
+			if _, err = executeTestVector(r, tv); err != nil {
 				return err
 			}
 		case io.EOF:
@@ -84,19 +176,30 @@ func runExecLotus(c *cli.Context) error {
 	}
 }
 
-func executeTestVector(tv schema.TestVector) error {
+func execVectorFile(r conformance.Reporter, path string) (diffs []string, error error) {
+	file, err := os.Open(path)
+	if err != nil {
+		return nil, fmt.Errorf("failed to open test vector: %w", err)
+	}
+
+	var tv schema.TestVector
+	if err = json.NewDecoder(file).Decode(&tv); err != nil {
+		return nil, fmt.Errorf("failed to decode test vector: %w", err)
+	}
+	return executeTestVector(r, tv)
+}
+
+func executeTestVector(r conformance.Reporter, tv schema.TestVector) (diffs []string, err error) {
 	log.Println("executing test vector:", tv.Meta.ID)
 
 	for _, v := range tv.Pre.Variants {
-		r := new(conformance.LogReporter)
-
 		switch class, v := tv.Class, v; class {
 		case "message":
-			conformance.ExecuteMessageVector(r, &tv, &v)
+			diffs, err = conformance.ExecuteMessageVector(r, &tv, &v)
 		case "tipset":
-			conformance.ExecuteTipsetVector(r, &tv, &v)
+			diffs, err = conformance.ExecuteTipsetVector(r, &tv, &v)
 		default:
-			return fmt.Errorf("test vector class %s not supported", class)
+			return nil, fmt.Errorf("test vector class %s not supported", class)
 		}
 
 		if r.Failed() {
@@ -106,5 +209,5 @@ func executeTestVector(tv schema.TestVector) error {
 		}
 	}
 
-	return nil
+	return diffs, err
 }

cmd/tvx/extract.go

@@ -1,8 +1,14 @@
 package main
 
 import (
+	"encoding/json"
 	"fmt"
+	"io"
+	"log"
+	"os"
+	"path/filepath"
 
+	"github.com/filecoin-project/test-vectors/schema"
 	"github.com/urfave/cli/v2"
 )
 
@@ -21,6 +27,7 @@ type extractOpts struct {
 	retain             string
 	precursor          string
 	ignoreSanityChecks bool
+	squash             bool
 }
 
 var extractFlags extractOpts
@@ -62,13 +69,13 @@ var extractCmd = &cli.Command{
 		},
 		&cli.StringFlag{
 			Name:        "tsk",
-			Usage:       "tipset key to extract into a vector",
+			Usage:       "tipset key to extract into a vector, or range of tipsets in tsk1..tsk2 form",
 			Destination: &extractFlags.tsk,
 		},
 		&cli.StringFlag{
 			Name:        "out",
 			Aliases:     []string{"o"},
-			Usage:       "file to write test vector to",
+			Usage:       "file to write test vector to, or directory to write the batch to",
 			Destination: &extractFlags.file,
 		},
 		&cli.StringFlag{
@@ -93,6 +100,12 @@ var extractCmd = &cli.Command{
 			Value:       false,
 			Destination: &extractFlags.ignoreSanityChecks,
 		},
+		&cli.BoolFlag{
+			Name:        "squash",
+			Usage:       "when extracting a tipset range, squash all tipsets into a single vector",
+			Value:       false,
+			Destination: &extractFlags.squash,
+		},
 	},
 }
 
@@ -106,3 +119,43 @@ func runExtract(_ *cli.Context) error {
 		return fmt.Errorf("unsupported vector class")
 	}
 }
+
+// writeVector writes the vector into the specified file, or to stdout if
+// file is empty.
+func writeVector(vector *schema.TestVector, file string) (err error) {
+	output := io.WriteCloser(os.Stdout)
+	if file := file; file != "" {
+		dir := filepath.Dir(file)
+		if err := os.MkdirAll(dir, 0755); err != nil {
+			return fmt.Errorf("unable to create directory %s: %w", dir, err)
+		}
+		output, err = os.Create(file)
+		if err != nil {
+			return err
+		}
+		defer output.Close() //nolint:errcheck
+		defer log.Printf("wrote test vector to file: %s", file)
+	}
+
+	enc := json.NewEncoder(output)
+	enc.SetIndent("", "  ")
+	return enc.Encode(&vector)
+}
+
+// writeVectors writes each vector to a different file under the specified
+// directory.
+func writeVectors(dir string, vectors ...*schema.TestVector) error {
+	// verify the output directory exists.
+	if err := ensureDir(dir); err != nil {
+		return err
+	}
+	// write each vector to its file.
+	for _, v := range vectors {
+		id := v.Meta.ID
+		path := filepath.Join(dir, fmt.Sprintf("%s.json", id))
+		if err := writeVector(v, path); err != nil {
+			return err
+		}
+	}
+	return nil
+}

cmd/tvx/extract_message.go

@@ -4,12 +4,9 @@ import (
 	"bytes"
 	"compress/gzip"
 	"context"
-	"encoding/json"
 	"fmt"
 	"io"
 	"log"
-	"os"
-	"path/filepath"
 
 	"github.com/fatih/color"
 	"github.com/filecoin-project/go-address"
@@ -316,28 +313,7 @@ func doExtractMessage(opts extractOpts) error {
 			},
 		},
 	}
-
-	return writeVector(vector, opts.file)
-}
-
-func writeVector(vector schema.TestVector, file string) (err error) {
-	output := io.WriteCloser(os.Stdout)
-	if file := file; file != "" {
-		dir := filepath.Dir(file)
-		if err := os.MkdirAll(dir, 0755); err != nil {
-			return fmt.Errorf("unable to create directory %s: %w", dir, err)
-		}
-		output, err = os.Create(file)
-		if err != nil {
-			return err
-		}
-		defer output.Close() //nolint:errcheck
-		defer log.Printf("wrote test vector to file: %s", file)
-	}
-
-	enc := json.NewEncoder(output)
-	enc.SetIndent("", "  ")
-	return enc.Encode(&vector)
+	return writeVector(&vector, opts.file)
 }
 
 // resolveFromChain queries the chain for the provided message, using the block CID to

cmd/tvx/extract_tipset.go

@@ -6,10 +6,12 @@ import (
 	"context"
 	"fmt"
 	"log"
+	"strings"
 
 	"github.com/filecoin-project/test-vectors/schema"
 	"github.com/ipfs/go-cid"
 
+	"github.com/filecoin-project/lotus/chain/types"
 	lcli "github.com/filecoin-project/lotus/cli"
 	"github.com/filecoin-project/lotus/conformance"
 )
@@ -17,170 +19,259 @@ import (
 func doExtractTipset(opts extractOpts) error {
 	ctx := context.Background()
 
-	if opts.tsk == "" {
-		return fmt.Errorf("tipset key cannot be empty")
-	}
-
 	if opts.retain != "accessed-cids" {
 		return fmt.Errorf("tipset extraction only supports 'accessed-cids' state retention")
 	}
 
-	ts, err := lcli.ParseTipSetRef(ctx, FullAPI, opts.tsk)
-	if err != nil {
-		return fmt.Errorf("failed to fetch tipset: %w", err)
+	if opts.tsk == "" {
+		return fmt.Errorf("tipset key cannot be empty")
 	}
 
-	log.Printf("tipset block count: %d", len(ts.Blocks()))
-
-	var blocks []schema.Block
-	for _, b := range ts.Blocks() {
-		msgs, err := FullAPI.ChainGetBlockMessages(ctx, b.Cid())
+	ss := strings.Split(opts.tsk, "..")
+	switch len(ss) {
+	case 1: // extracting a single tipset.
+		ts, err := lcli.ParseTipSetRef(ctx, FullAPI, opts.tsk)
 		if err != nil {
-			return fmt.Errorf("failed to get block messages (cid: %s): %w", b.Cid(), err)
+			return fmt.Errorf("failed to fetch tipset: %w", err)
+		}
+		v, err := extractTipsets(ctx, ts)
+		if err != nil {
+			return err
+		}
+		return writeVector(v, opts.file)
+
+	case 2: // extracting a range of tipsets.
+		left, err := lcli.ParseTipSetRef(ctx, FullAPI, ss[0])
+		if err != nil {
+			return fmt.Errorf("failed to fetch tipset %s: %w", ss[0], err)
+		}
+		right, err := lcli.ParseTipSetRef(ctx, FullAPI, ss[1])
+		if err != nil {
+			return fmt.Errorf("failed to fetch tipset %s: %w", ss[1], err)
 		}
 
-		log.Printf("block %s has %d messages", b.Cid(), len(msgs.Cids))
+		// resolve the tipset range.
+		tss, err := resolveTipsetRange(ctx, left, right)
+		if err != nil {
+			return err
+		}
 
-		packed := make([]schema.Base64EncodedBytes, 0, len(msgs.Cids))
-		for _, m := range msgs.BlsMessages {
-			b, err := m.Serialize()
+		// are are squashing all tipsets into a single multi-tipset vector?
+		if opts.squash {
+			vector, err := extractTipsets(ctx, tss...)
 			if err != nil {
-				return fmt.Errorf("failed to serialize message: %w", err)
+				return err
 			}
-			packed = append(packed, b)
+			return writeVector(vector, opts.file)
 		}
-		for _, m := range msgs.SecpkMessages {
-			b, err := m.Message.Serialize()
-			if err != nil {
-				return fmt.Errorf("failed to serialize message: %w", err)
-			}
-			packed = append(packed, b)
+
+		// we are generating a single-tipset vector per tipset.
+		vectors, err := extractIndividualTipsets(ctx, tss...)
+		if err != nil {
+			return err
 		}
-		blocks = append(blocks, schema.Block{
-			MinerAddr: b.Miner,
-			WinCount:  b.ElectionProof.WinCount,
-			Messages:  packed,
-		})
+		return writeVectors(opts.file, vectors...)
+
+	default:
+		return fmt.Errorf("unrecognized tipset format")
 	}
+}
 
+func resolveTipsetRange(ctx context.Context, left *types.TipSet, right *types.TipSet) (tss []*types.TipSet, err error) {
+	// start from the right tipset and walk back the chain until the left tipset, inclusive.
+	for curr := right; curr.Key() != left.Parents(); {
+		tss = append(tss, curr)
+		curr, err = FullAPI.ChainGetTipSet(ctx, curr.Parents())
+		if err != nil {
+			return nil, fmt.Errorf("failed to get tipset %s (height: %d): %w", curr.Parents(), curr.Height()-1, err)
+		}
+	}
+	// reverse the slice.
+	for i, j := 0, len(tss)-1; i < j; i, j = i+1, j-1 {
+		tss[i], tss[j] = tss[j], tss[i]
+	}
+	return tss, nil
+}
+
+func extractIndividualTipsets(ctx context.Context, tss ...*types.TipSet) (vectors []*schema.TestVector, err error) {
+	for _, ts := range tss {
+		v, err := extractTipsets(ctx, ts)
+		if err != nil {
+			return nil, err
+		}
+		vectors = append(vectors, v)
+	}
+	return vectors, nil
+}
+
+func extractTipsets(ctx context.Context, tss ...*types.TipSet) (*schema.TestVector, error) {
 	var (
 		// create a read-through store that uses ChainGetObject to fetch unknown CIDs.
 		pst = NewProxyingStores(ctx, FullAPI)
 		g   = NewSurgeon(ctx, FullAPI, pst)
+
+		// recordingRand will record randomness so we can embed it in the test vector.
+		recordingRand = conformance.NewRecordingRand(new(conformance.LogReporter), FullAPI)
 	)
 
+	tbs, ok := pst.Blockstore.(TracingBlockstore)
+	if !ok {
+		return nil, fmt.Errorf("requested 'accessed-cids' state retention, but no tracing blockstore was present")
+	}
+
 	driver := conformance.NewDriver(ctx, schema.Selector{}, conformance.DriverOpts{
 		DisableVMFlush: true,
 	})
 
+	base := tss[0]
+	last := tss[len(tss)-1]
+
 	// this is the root of the state tree we start with.
-	root := ts.ParentState()
+	root := base.ParentState()
 	log.Printf("base state tree root CID: %s", root)
 
-	basefee := ts.Blocks()[0].ParentBaseFee
-	log.Printf("basefee: %s", basefee)
-
-	tipset := schema.Tipset{
-		BaseFee: *basefee.Int,
-		Blocks:  blocks,
+	codename := GetProtocolCodename(base.Height())
+	nv, err := FullAPI.StateNetworkVersion(ctx, base.Key())
+	if err != nil {
+		return nil, err
 	}
 
-	// recordingRand will record randomness so we can embed it in the test vector.
-	recordingRand := conformance.NewRecordingRand(new(conformance.LogReporter), FullAPI)
+	version, err := FullAPI.Version(ctx)
+	if err != nil {
+		return nil, err
+	}
 
-	log.Printf("using state retention strategy: %s", extractFlags.retain)
+	ntwkName, err := FullAPI.StateNetworkName(ctx)
+	if err != nil {
+		return nil, err
+	}
 
-	tbs, ok := pst.Blockstore.(TracingBlockstore)
-	if !ok {
-		return fmt.Errorf("requested 'accessed-cids' state retention, but no tracing blockstore was present")
+	vector := schema.TestVector{
+		Class: schema.ClassTipset,
+		Meta: &schema.Metadata{
+			ID: fmt.Sprintf("@%d..@%d", base.Height(), last.Height()),
+			Gen: []schema.GenerationData{
+				{Source: fmt.Sprintf("network:%s", ntwkName)},
+				{Source: "github.com/filecoin-project/lotus", Version: version.String()}},
+			// will be completed by extra tipset stamps.
+		},
+		Selector: schema.Selector{
+			schema.SelectorMinProtocolVersion: codename,
+		},
+		Pre: &schema.Preconditions{
+			Variants: []schema.Variant{
+				{ID: codename, Epoch: int64(base.Height()), NetworkVersion: uint(nv)},
+			},
+			StateTree: &schema.StateTree{
+				RootCID: base.ParentState(),
+			},
+		},
+		Post: &schema.Postconditions{
+			StateTree: new(schema.StateTree),
+		},
 	}
 
 	tbs.StartTracing()
 
-	params := conformance.ExecuteTipsetParams{
-		Preroot:     ts.ParentState(),
-		ParentEpoch: ts.Height() - 1,
-		Tipset:      &tipset,
-		ExecEpoch:   ts.Height(),
-		Rand:        recordingRand,
-	}
-	result, err := driver.ExecuteTipset(pst.Blockstore, pst.Datastore, params)
-	if err != nil {
-		return fmt.Errorf("failed to execute tipset: %w", err)
+	roots := []cid.Cid{base.ParentState()}
+	for i, ts := range tss {
+		log.Printf("tipset %s block count: %d", ts.Key(), len(ts.Blocks()))
+
+		var blocks []schema.Block
+		for _, b := range ts.Blocks() {
+			msgs, err := FullAPI.ChainGetBlockMessages(ctx, b.Cid())
+			if err != nil {
+				return nil, fmt.Errorf("failed to get block messages (cid: %s): %w", b.Cid(), err)
+			}
+
+			log.Printf("block %s has %d messages", b.Cid(), len(msgs.Cids))
+
+			packed := make([]schema.Base64EncodedBytes, 0, len(msgs.Cids))
+			for _, m := range msgs.BlsMessages {
+				b, err := m.Serialize()
+				if err != nil {
+					return nil, fmt.Errorf("failed to serialize message: %w", err)
+				}
+				packed = append(packed, b)
+			}
+			for _, m := range msgs.SecpkMessages {
+				b, err := m.Message.Serialize()
+				if err != nil {
+					return nil, fmt.Errorf("failed to serialize message: %w", err)
+				}
+				packed = append(packed, b)
+			}
+			blocks = append(blocks, schema.Block{
+				MinerAddr: b.Miner,
+				WinCount:  b.ElectionProof.WinCount,
+				Messages:  packed,
+			})
+		}
+
+		basefee := base.Blocks()[0].ParentBaseFee
+		log.Printf("tipset basefee: %s", basefee)
+
+		tipset := schema.Tipset{
+			BaseFee:     *basefee.Int,
+			Blocks:      blocks,
+			EpochOffset: int64(i),
+		}
+
+		params := conformance.ExecuteTipsetParams{
+			Preroot:     roots[len(roots)-1],
+			ParentEpoch: ts.Height() - 1,
+			Tipset:      &tipset,
+			ExecEpoch:   ts.Height(),
+			Rand:        recordingRand,
+		}
+
+		result, err := driver.ExecuteTipset(pst.Blockstore, pst.Datastore, params)
+		if err != nil {
+			return nil, fmt.Errorf("failed to execute tipset: %w", err)
+		}
+
+		roots = append(roots, result.PostStateRoot)
+
+		// update the vector.
+		vector.ApplyTipsets = append(vector.ApplyTipsets, tipset)
+		vector.Post.ReceiptsRoots = append(vector.Post.ReceiptsRoots, result.ReceiptsRoot)
+
+		for _, res := range result.AppliedResults {
+			vector.Post.Receipts = append(vector.Post.Receipts, &schema.Receipt{
+				ExitCode:    int64(res.ExitCode),
+				ReturnValue: res.Return,
+				GasUsed:     res.GasUsed,
+			})
+		}
+
+		vector.Meta.Gen = append(vector.Meta.Gen, schema.GenerationData{
+			Source: "tipset:" + ts.Key().String(),
+		})
 	}
 
 	accessed := tbs.FinishTracing()
 
+	//
+	// ComputeBaseFee(ctx, baseTs)
+
 	// write a CAR with the accessed state into a buffer.
 	var (
 		out = new(bytes.Buffer)
 		gw  = gzip.NewWriter(out)
 	)
-	if err := g.WriteCARIncluding(gw, accessed, ts.ParentState(), result.PostStateRoot); err != nil {
-		return err
+	if err := g.WriteCARIncluding(gw, accessed, roots...); err != nil {
+		return nil, err
 	}
 	if err = gw.Flush(); err != nil {
-		return err
+		return nil, err
 	}
 	if err = gw.Close(); err != nil {
-		return err
+		return nil, err
 	}
 
-	codename := GetProtocolCodename(ts.Height())
-	nv, err := FullAPI.StateNetworkVersion(ctx, ts.Key())
-	if err != nil {
-		return err
-	}
+	vector.Randomness = recordingRand.Recorded()
+	vector.Post.StateTree.RootCID = roots[len(roots)-1]
+	vector.CAR = out.Bytes()
 
-	version, err := FullAPI.Version(ctx)
-	if err != nil {
-		return err
-	}
-
-	ntwkName, err := FullAPI.StateNetworkName(ctx)
-	if err != nil {
-		return err
-	}
-
-	vector := schema.TestVector{
-		Class: schema.ClassTipset,
-		Meta: &schema.Metadata{
-			ID: opts.id,
-			Gen: []schema.GenerationData{
-				{Source: fmt.Sprintf("network:%s", ntwkName)},
-				{Source: fmt.Sprintf("tipset:%s", ts.Key())},
-				{Source: "github.com/filecoin-project/lotus", Version: version.String()}},
-		},
-		Selector: schema.Selector{
-			schema.SelectorMinProtocolVersion: codename,
-		},
-		Randomness: recordingRand.Recorded(),
-		CAR:        out.Bytes(),
-		Pre: &schema.Preconditions{
-			Variants: []schema.Variant{
-				{ID: codename, Epoch: int64(ts.Height()), NetworkVersion: uint(nv)},
-			},
-			BaseFee: basefee.Int,
-			StateTree: &schema.StateTree{
-				RootCID: ts.ParentState(),
-			},
-		},
-		ApplyTipsets: []schema.Tipset{tipset},
-		Post: &schema.Postconditions{
-			StateTree: &schema.StateTree{
-				RootCID: result.PostStateRoot,
-			},
-			ReceiptsRoots: []cid.Cid{result.ReceiptsRoot},
-		},
-	}
-
-	for _, res := range result.AppliedResults {
-		vector.Post.Receipts = append(vector.Post.Receipts, &schema.Receipt{
-			ExitCode:    int64(res.ExitCode),
-			ReturnValue: res.Return,
-			GasUsed:     res.GasUsed,
-		})
-	}
-
-	return writeVector(vector, opts.file)
+	return &vector, nil
 }

cmd/tvx/main.go

@@ -113,3 +113,19 @@ func destroy(_ *cli.Context) error {
 	}
 	return nil
 }
+
+func ensureDir(path string) error {
+	switch fi, err := os.Stat(path); {
+	case os.IsNotExist(err):
+		if err := os.MkdirAll(path, 0755); err != nil {
+			return fmt.Errorf("failed to create directory %s: %w", path, err)
+		}
+	case err == nil:
+		if !fi.IsDir() {
+			return fmt.Errorf("path %s is not a directory: %w", path, err)
+		}
+	default:
+		return fmt.Errorf("failed to stat directory %s: %w", path, err)
+	}
+	return nil
+}

cmd/tvx/simulate.go

@@ -202,7 +202,7 @@ func runSimulateCmd(_ *cli.Context) error {
 		},
 	}
 
-	if err := writeVector(vector, simulateFlags.out); err != nil {
+	if err := writeVector(&vector, simulateFlags.out); err != nil {
 		return fmt.Errorf("failed to write vector: %w", err)
 	}
 

cmd/tvx/stores.go

@@ -149,3 +149,14 @@ func (pb *proxyingBlockstore) Put(block blocks.Block) error {
 	pb.lk.Unlock()
 	return pb.Blockstore.Put(block)
 }
+
+func (pb *proxyingBlockstore) PutMany(blocks []blocks.Block) error {
+	pb.lk.Lock()
+	if pb.tracing {
+		for _, b := range blocks {
+			pb.traced[b.Cid()] = struct{}{}
+		}
+	}
+	pb.lk.Unlock()
+	return pb.Blockstore.PutMany(blocks)
+}

conformance/corpus_test.go

@@ -11,7 +11,7 @@ import (
 	"github.com/filecoin-project/test-vectors/schema"
 )
 
-var invokees = map[schema.Class]func(Reporter, *schema.TestVector, *schema.Variant){
+var invokees = map[schema.Class]func(Reporter, *schema.TestVector, *schema.Variant) ([]string, error){
 	schema.ClassMessage: ExecuteMessageVector,
 	schema.ClassTipset:  ExecuteTipsetVector,
 }
@@ -133,7 +133,7 @@ func TestConformance(t *testing.T) {
 			for _, variant := range vector.Pre.Variants {
 				variant := variant
 				t.Run(variant.ID, func(t *testing.T) {
-					invokee(t, &vector, &variant)
+					_, _ = invokee(t, &vector, &variant) //nolint:errcheck
 				})
 			}
 		})

conformance/driver.go

@@ -71,6 +71,9 @@ type ExecuteTipsetResult struct {
 	AppliedMessages []*types.Message
 	// AppliedResults stores the results of AppliedMessages, in the same order.
 	AppliedResults []*vm.ApplyRet
+
+	// PostBaseFee returns the basefee after applying this tipset.
+	PostBaseFee abi.TokenAmount
 }
 
 type ExecuteTipsetParams struct {

conformance/runner.go

@@ -14,6 +14,7 @@ import (
 	"github.com/fatih/color"
 	"github.com/filecoin-project/go-state-types/abi"
 	"github.com/filecoin-project/go-state-types/exitcode"
+	"github.com/hashicorp/go-multierror"
 	blocks "github.com/ipfs/go-block-format"
 	"github.com/ipfs/go-blockservice"
 	"github.com/ipfs/go-cid"
@@ -38,8 +39,19 @@ var FallbackBlockstoreGetter interface {
 	ChainReadObj(context.Context, cid.Cid) ([]byte, error)
 }
 
+var TipsetVectorOpts struct {
+	// PipelineBaseFee pipelines the basefee in multi-tipset vectors from one
+	// tipset to another. Basefees in the vector are ignored, except for that of
+	// the first tipset. UNUSED.
+	PipelineBaseFee bool
+
+	// OnTipsetApplied contains callback functions called after a tipset has been
+	// applied.
+	OnTipsetApplied []func(bs blockstore.Blockstore, params *ExecuteTipsetParams, res *ExecuteTipsetResult)
+}
+
 // ExecuteMessageVector executes a message-class test vector.
-func ExecuteMessageVector(r Reporter, vector *schema.TestVector, variant *schema.Variant) {
+func ExecuteMessageVector(r Reporter, vector *schema.TestVector, variant *schema.Variant) (diffs []string, err error) {
 	var (
 		ctx       = context.Background()
 		baseEpoch = variant.Epoch
@@ -88,14 +100,16 @@ func ExecuteMessageVector(r Reporter, vector *schema.TestVector, variant *schema
 	// Once all messages are applied, assert that the final state root matches
 	// the expected postcondition root.
 	if expected, actual := vector.Post.StateTree.RootCID, root; expected != actual {
-		r.Errorf("wrong post root cid; expected %v, but got %v", expected, actual)
-		dumpThreeWayStateDiff(r, vector, bs, root)
-		r.FailNow()
+		ierr := fmt.Errorf("wrong post root cid; expected %v, but got %v", expected, actual)
+		r.Errorf(ierr.Error())
+		err = multierror.Append(err, ierr)
+		diffs = dumpThreeWayStateDiff(r, vector, bs, root)
 	}
+	return diffs, err
 }
 
 // ExecuteTipsetVector executes a tipset-class test vector.
-func ExecuteTipsetVector(r Reporter, vector *schema.TestVector, variant *schema.Variant) {
+func ExecuteTipsetVector(r Reporter, vector *schema.TestVector, variant *schema.Variant) (diffs []string, err error) {
 	var (
 		ctx       = context.Background()
 		baseEpoch = abi.ChainEpoch(variant.Epoch)
@@ -107,6 +121,7 @@ func ExecuteTipsetVector(r Reporter, vector *schema.TestVector, variant *schema.
 	bs, err := LoadBlockstore(vector.CAR)
 	if err != nil {
 		r.Fatalf("failed to load the vector CAR: %w", err)
+		return nil, err
 	}
 
 	// Create a new Driver.
@@ -118,15 +133,22 @@ func ExecuteTipsetVector(r Reporter, vector *schema.TestVector, variant *schema.
 	for i, ts := range vector.ApplyTipsets {
 		ts := ts // capture
 		execEpoch := baseEpoch + abi.ChainEpoch(ts.EpochOffset)
-		ret, err := driver.ExecuteTipset(bs, tmpds, ExecuteTipsetParams{
+		params := ExecuteTipsetParams{
 			Preroot:     root,
 			ParentEpoch: prevEpoch,
 			Tipset:      &ts,
 			ExecEpoch:   execEpoch,
 			Rand:        NewReplayingRand(r, vector.Randomness),
-		})
+		}
+		ret, err := driver.ExecuteTipset(bs, tmpds, params)
 		if err != nil {
 			r.Fatalf("failed to apply tipset %d: %s", i, err)
+			return nil, err
+		}
+
+		// invoke callbacks.
+		for _, cb := range TipsetVectorOpts.OnTipsetApplied {
+			cb(bs, &params, ret)
 		}
 
 		for j, v := range ret.AppliedResults {
@@ -136,7 +158,9 @@ func ExecuteTipsetVector(r Reporter, vector *schema.TestVector, variant *schema.
 
 		// Compare the receipts root.
 		if expected, actual := vector.Post.ReceiptsRoots[i], ret.ReceiptsRoot; expected != actual {
-			r.Errorf("post receipts root doesn't match; expected: %s, was: %s", expected, actual)
+			ierr := fmt.Errorf("post receipts root doesn't match; expected: %s, was: %s", expected, actual)
+			r.Errorf(ierr.Error())
+			err = multierror.Append(err, ierr)
 		}
 
 		prevEpoch = execEpoch
@@ -146,10 +170,12 @@ func ExecuteTipsetVector(r Reporter, vector *schema.TestVector, variant *schema.
 	// Once all messages are applied, assert that the final state root matches
 	// the expected postcondition root.
 	if expected, actual := vector.Post.StateTree.RootCID, root; expected != actual {
-		r.Errorf("wrong post root cid; expected %v, but got %v", expected, actual)
-		dumpThreeWayStateDiff(r, vector, bs, root)
-		r.FailNow()
+		ierr := fmt.Errorf("wrong post root cid; expected %v, but got %v", expected, actual)
+		r.Errorf(ierr.Error())
+		err = multierror.Append(err, ierr)
+		diffs = dumpThreeWayStateDiff(r, vector, bs, root)
 	}
+	return diffs, err
 }
 
 // AssertMsgResult compares a message result. It takes the expected receipt
@@ -169,7 +195,7 @@ func AssertMsgResult(r Reporter, expected *schema.Receipt, actual *vm.ApplyRet,
 	}
 }
 
-func dumpThreeWayStateDiff(r Reporter, vector *schema.TestVector, bs blockstore.Blockstore, actual cid.Cid) {
+func dumpThreeWayStateDiff(r Reporter, vector *schema.TestVector, bs blockstore.Blockstore, actual cid.Cid) []string {
 	// check if statediff exists; if not, skip.
 	if err := exec.Command("statediff", "--help").Run(); err != nil {
 		r.Log("could not dump 3-way state tree diff upon test failure: statediff command not found")
@@ -178,7 +204,7 @@ func dumpThreeWayStateDiff(r Reporter, vector *schema.TestVector, bs blockstore.
 		r.Log("$ cd statediff")
 		r.Log("$ go generate ./...")
 		r.Log("$ go install ./cmd/statediff")
-		return
+		return nil
 	}
 
 	tmpCar, err := writeStateToTempCAR(bs,
@@ -188,6 +214,7 @@ func dumpThreeWayStateDiff(r Reporter, vector *schema.TestVector, bs blockstore.
 	)
 	if err != nil {
 		r.Fatalf("failed to write temporary state CAR: %s", err)
+		return nil
 	}
 	defer os.RemoveAll(tmpCar) //nolint:errcheck
 
@@ -202,28 +229,43 @@ func dumpThreeWayStateDiff(r Reporter, vector *schema.TestVector, bs blockstore.
 		d3 = color.New(color.FgGreen, color.Bold).Sprint("[Δ3]")
 	)
 
-	printDiff := func(left, right cid.Cid) {
+	diff := func(left, right cid.Cid) string {
 		cmd := exec.Command("statediff", "car", "--file", tmpCar, left.String(), right.String())
 		b, err := cmd.CombinedOutput()
 		if err != nil {
 			r.Fatalf("statediff failed: %s", err)
 		}
-		r.Log(string(b))
+		return string(b)
 	}
 
 	bold := color.New(color.Bold).SprintfFunc()
 
+	r.Log(bold("-----BEGIN STATEDIFF-----"))
+
 	// run state diffs.
 	r.Log(bold("=== dumping 3-way diffs between %s, %s, %s ===", a, b, c))
 
 	r.Log(bold("--- %s left: %s; right: %s ---", d1, a, b))
-	printDiff(vector.Post.StateTree.RootCID, actual)
+	diffA := diff(vector.Post.StateTree.RootCID, actual)
+	r.Log(bold("----------BEGIN STATEDIFF A----------"))
+	r.Log(diffA)
+	r.Log(bold("----------END STATEDIFF A----------"))
 
 	r.Log(bold("--- %s left: %s; right: %s ---", d2, c, b))
-	printDiff(vector.Pre.StateTree.RootCID, actual)
+	diffB := diff(vector.Pre.StateTree.RootCID, actual)
+	r.Log(bold("----------BEGIN STATEDIFF B----------"))
+	r.Log(diffB)
+	r.Log(bold("----------END STATEDIFF B----------"))
 
 	r.Log(bold("--- %s left: %s; right: %s ---", d3, c, a))
-	printDiff(vector.Pre.StateTree.RootCID, vector.Post.StateTree.RootCID)
+	diffC := diff(vector.Pre.StateTree.RootCID, vector.Post.StateTree.RootCID)
+	r.Log(bold("----------BEGIN STATEDIFF C----------"))
+	r.Log(diffC)
+	r.Log(bold("----------END STATEDIFF C----------"))
+
+	r.Log(bold("-----END STATEDIFF-----"))
+
+	return []string{diffA, diffB, diffC}
 }
 
 // writeStateToTempCAR writes the provided roots to a temporary CAR that'll be

documentation/en/api-methods.md

@@ -68,6 +68,8 @@
   * [LogList](#LogList)
   * [LogSetLevel](#LogSetLevel)
 * [Market](#Market)
+  * [MarketAddBalance](#MarketAddBalance)
+  * [MarketGetReserved](#MarketGetReserved)
   * [MarketReleaseFunds](#MarketReleaseFunds)
   * [MarketReserveFunds](#MarketReserveFunds)
   * [MarketWithdraw](#MarketWithdraw)
@@ -1653,6 +1655,43 @@ Response: `{}`
 ## Market
 
 
+### MarketAddBalance
+MarketAddBalance adds funds to the market actor
+
+
+Perms: sign
+
+Inputs:
+```json
+[
+  "f01234",
+  "f01234",
+  "0"
+]
+```
+
+Response:
+```json
+{
+  "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"
+}
+```
+
+### MarketGetReserved
+MarketGetReserved gets the amount of funds that are currently reserved for the address
+
+
+Perms: sign
+
+Inputs:
+```json
+[
+  "f01234"
+]
+```
+
+Response: `"0"`
+
 ### MarketReleaseFunds
 MarketReleaseFunds releases funds reserved by MarketReserveFunds
 
@@ -4308,7 +4347,7 @@ Inputs:
 ]
 ```
 
-Response: `8`
+Response: `9`
 
 ### StateReadState
 StateReadState returns the indicated actor's state.

extern/sector-storage/ffiwrapper/sealer_cgo.go

@@ -45,6 +45,10 @@ func (sb *Sealer) NewSector(ctx context.Context, sector storage.SectorRef) error
 }
 
 func (sb *Sealer) AddPiece(ctx context.Context, sector storage.SectorRef, existingPieceSizes []abi.UnpaddedPieceSize, pieceSize abi.UnpaddedPieceSize, file storage.Data) (abi.PieceInfo, error) {
+	// TODO: allow tuning those:
+	chunk := abi.PaddedPieceSize(4 << 20)
+	parallel := runtime.NumCPU()
+
 	var offset abi.UnpaddedPieceSize
 	for _, size := range existingPieceSizes {
 		offset += size
@@ -108,10 +112,16 @@ func (sb *Sealer) AddPiece(ctx context.Context, sector storage.SectorRef, existi
 
 	pr := io.TeeReader(io.LimitReader(file, int64(pieceSize)), pw)
 
-	chunk := abi.PaddedPieceSize(4 << 20)
+	throttle := make(chan []byte, parallel)
+	piecePromises := make([]func() (abi.PieceInfo, error), 0)
 
 	buf := make([]byte, chunk.Unpadded())
-	var pieceCids []abi.PieceInfo
+	for i := 0; i < parallel; i++ {
+		if abi.UnpaddedPieceSize(i)*chunk.Unpadded() >= pieceSize {
+			break // won't use this many buffers
+		}
+		throttle <- make([]byte, chunk.Unpadded())
+	}
 
 	for {
 		var read int
@@ -132,13 +142,39 @@ func (sb *Sealer) AddPiece(ctx context.Context, sector storage.SectorRef, existi
 			break
 		}
 
-		c, err := sb.pieceCid(sector.ProofType, buf[:read])
-		if err != nil {
-			return abi.PieceInfo{}, xerrors.Errorf("pieceCid error: %w", err)
-		}
-		pieceCids = append(pieceCids, abi.PieceInfo{
-			Size:     abi.UnpaddedPieceSize(len(buf[:read])).Padded(),
-			PieceCID: c,
+		done := make(chan struct {
+			cid.Cid
+			error
+		}, 1)
+		pbuf := <-throttle
+		copy(pbuf, buf[:read])
+
+		go func(read int) {
+			defer func() {
+				throttle <- pbuf
+			}()
+
+			c, err := sb.pieceCid(sector.ProofType, pbuf[:read])
+			done <- struct {
+				cid.Cid
+				error
+			}{c, err}
+		}(read)
+
+		piecePromises = append(piecePromises, func() (abi.PieceInfo, error) {
+			select {
+			case e := <-done:
+				if e.error != nil {
+					return abi.PieceInfo{}, e.error
+				}
+
+				return abi.PieceInfo{
+					Size:     abi.UnpaddedPieceSize(len(buf[:read])).Padded(),
+					PieceCID: e.Cid,
+				}, nil
+			case <-ctx.Done():
+				return abi.PieceInfo{}, ctx.Err()
+			}
 		})
 	}
 
@@ -155,8 +191,16 @@ func (sb *Sealer) AddPiece(ctx context.Context, sector storage.SectorRef, existi
 	}
 	stagedFile = nil
 
-	if len(pieceCids) == 1 {
-		return pieceCids[0], nil
+	if len(piecePromises) == 1 {
+		return piecePromises[0]()
+	}
+
+	pieceCids := make([]abi.PieceInfo, len(piecePromises))
+	for i, promise := range piecePromises {
+		pieceCids[i], err = promise()
+		if err != nil {
+			return abi.PieceInfo{}, err
+		}
 	}
 
 	pieceCID, err := ffi.GenerateUnsealedCID(sector.ProofType, pieceCids)

extern/sector-storage/ffiwrapper/sealer_test.go

@@ -33,6 +33,7 @@ import (
 
 	"github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper/basicfs"
 	"github.com/filecoin-project/lotus/extern/sector-storage/storiface"
+	"github.com/filecoin-project/lotus/extern/storage-sealing/lib/nullreader"
 )
 
 func init() {
@@ -622,3 +623,89 @@ func TestGenerateUnsealedCID(t *testing.T) {
 		[][]byte{barr(1, 16), barr(0, 16), barr(2, 8), barr(3, 16), barr(0, 16), barr(0, 8), barr(4, 4), barr(5, 16), barr(0, 16), barr(0, 8)},
 	)
 }
+
+func TestAddPiece512M(t *testing.T) {
+	sz := abi.PaddedPieceSize(512 << 20).Unpadded()
+
+	cdir, err := ioutil.TempDir("", "sbtest-c-")
+	if err != nil {
+		t.Fatal(err)
+	}
+	miner := abi.ActorID(123)
+
+	sp := &basicfs.Provider{
+		Root: cdir,
+	}
+	sb, err := New(sp)
+	if err != nil {
+		t.Fatalf("%+v", err)
+	}
+	cleanup := func() {
+		if t.Failed() {
+			fmt.Printf("not removing %s\n", cdir)
+			return
+		}
+		if err := os.RemoveAll(cdir); err != nil {
+			t.Error(err)
+		}
+	}
+	t.Cleanup(cleanup)
+
+	r := rand.New(rand.NewSource(0x7e5))
+
+	c, err := sb.AddPiece(context.TODO(), storage.SectorRef{
+		ID: abi.SectorID{
+			Miner:  miner,
+			Number: 0,
+		},
+		ProofType: abi.RegisteredSealProof_StackedDrg512MiBV1_1,
+	}, nil, sz, io.LimitReader(r, int64(sz)))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	require.Equal(t, "baga6ea4seaqhyticusemlcrjhvulpfng4nint6bu3wpe5s3x4bnuj2rs47hfacy", c.PieceCID.String())
+}
+
+func BenchmarkAddPiece512M(b *testing.B) {
+	sz := abi.PaddedPieceSize(512 << 20).Unpadded()
+	b.SetBytes(int64(sz))
+
+	cdir, err := ioutil.TempDir("", "sbtest-c-")
+	if err != nil {
+		b.Fatal(err)
+	}
+	miner := abi.ActorID(123)
+
+	sp := &basicfs.Provider{
+		Root: cdir,
+	}
+	sb, err := New(sp)
+	if err != nil {
+		b.Fatalf("%+v", err)
+	}
+	cleanup := func() {
+		if b.Failed() {
+			fmt.Printf("not removing %s\n", cdir)
+			return
+		}
+		if err := os.RemoveAll(cdir); err != nil {
+			b.Error(err)
+		}
+	}
+	b.Cleanup(cleanup)
+
+	for i := 0; i < b.N; i++ {
+		c, err := sb.AddPiece(context.TODO(), storage.SectorRef{
+			ID: abi.SectorID{
+				Miner:  miner,
+				Number: abi.SectorNumber(i),
+			},
+			ProofType: abi.RegisteredSealProof_StackedDrg512MiBV1_1,
+		}, nil, sz, io.LimitReader(&nullreader.Reader{}, int64(sz)))
+		if err != nil {
+			b.Fatal(err)
+		}
+		fmt.Println(c)
+	}
+}

extern/sector-storage/mock/mock.go

@@ -347,7 +347,7 @@ func generateFakePoSt(sectorInfo []proof2.SectorInfo, rpt func(abi.RegisteredSea
 }
 
 func (mgr *SectorMgr) ReadPiece(ctx context.Context, w io.Writer, sectorID storage.SectorRef, offset storiface.UnpaddedByteIndex, size abi.UnpaddedPieceSize, randomness abi.SealRandomness, c cid.Cid) error {
-	if len(mgr.sectors[sectorID.ID].pieces) > 1 || offset != 0 {
+	if offset != 0 {
 		panic("implme")
 	}
 

extern/sector-storage/sched_worker.go

@@ -57,6 +57,7 @@ func (sh *scheduler) runWorker(ctx context.Context, w Worker) error {
 		log.Warnw("duplicated worker added", "id", wid)
 
 		// this is ok, we're already handling this worker in a different goroutine
+		sh.workersLk.Unlock()
 		return nil
 	}
 

extern/sector-storage/stores/index.go

@@ -155,6 +155,10 @@ func (i *Index) StorageAttach(ctx context.Context, si StorageInfo, st fsutil.FsS
 			i.stores[si.ID].info.URLs = append(i.stores[si.ID].info.URLs, u)
 		}
 
+		i.stores[si.ID].info.Weight = si.Weight
+		i.stores[si.ID].info.CanSeal = si.CanSeal
+		i.stores[si.ID].info.CanStore = si.CanStore
+
 		return nil
 	}
 	i.stores[si.ID] = &storageEntry{

extern/storage-sealing/fsm.go

@@ -196,7 +196,7 @@ func (m *Sealing) plan(events []statemachine.Event, state *SectorInfo) (func(sta
 				Kind:      fmt.Sprintf("truncate"),
 			}
 
-			state.Log = append(state.Log[:2000], state.Log[:6000]...)
+			state.Log = append(state.Log[:2000], state.Log[6000:]...)
 		}
 
 		state.Log = append(state.Log, l)
@@ -409,8 +409,9 @@ func (m *Sealing) restartSectors(ctx context.Context) error {
 		return err
 	}
 
-	m.unsealedInfoMap.lk.Lock()
+	// m.unsealedInfoMap.lk.Lock() taken early in .New to prevent races
 	defer m.unsealedInfoMap.lk.Unlock()
+
 	for _, sector := range trackedSectors {
 		if err := m.sectors.Send(uint64(sector.SectorNumber), SectorRestart{}); err != nil {
 			log.Errorf("restarting sector %d: %+v", sector.SectorNumber, err)

extern/storage-sealing/sealing.go

@@ -145,6 +145,8 @@ func New(api SealingAPI, fc FeeConfig, events Events, maddr address.Address, ds
 
 	s.sectors = statemachine.New(namespace.Wrap(ds, datastore.NewKey(SectorStorePrefix)), s, SectorInfo{})
 
+	s.unsealedInfoMap.lk.Lock() // released after initialized in .Run()
+
 	return s
 }
 

go.mod

@@ -43,7 +43,7 @@ require (
 	github.com/filecoin-project/go-statestore v0.1.0
 	github.com/filecoin-project/go-storedcounter v0.0.0-20200421200003-1c99c62e8a5b
 	github.com/filecoin-project/specs-actors v0.9.13
-	github.com/filecoin-project/specs-actors/v2 v2.3.2
+	github.com/filecoin-project/specs-actors/v2 v2.3.3
 	github.com/filecoin-project/specs-storage v0.1.1-0.20201105051918-5188d9774506
 	github.com/filecoin-project/test-vectors/schema v0.0.5
 	github.com/gbrlsnchs/jwt/v3 v3.0.0-beta.1
@@ -104,7 +104,7 @@ require (
 	github.com/libp2p/go-libp2p-mplex v0.3.0
 	github.com/libp2p/go-libp2p-noise v0.1.2
 	github.com/libp2p/go-libp2p-peerstore v0.2.6
-	github.com/libp2p/go-libp2p-pubsub v0.4.0
+	github.com/libp2p/go-libp2p-pubsub v0.4.1
 	github.com/libp2p/go-libp2p-quic-transport v0.9.0
 	github.com/libp2p/go-libp2p-record v0.1.3
 	github.com/libp2p/go-libp2p-routing-helpers v0.2.3

go.sum

@@ -296,6 +296,8 @@ github.com/filecoin-project/specs-actors v0.9.13/go.mod h1:TS1AW/7LbG+615j4NsjMK
 github.com/filecoin-project/specs-actors/v2 v2.0.1/go.mod h1:v2NZVYinNIKA9acEMBm5wWXxqv5+frFEbekBFemYghY=
 github.com/filecoin-project/specs-actors/v2 v2.3.2 h1:2Vcf4CGa29kRh4JJ02m+FbvD/p3YNnLGsaHfw7Uj49g=
 github.com/filecoin-project/specs-actors/v2 v2.3.2/go.mod h1:UuJQLoTx/HPvvWeqlIFmC/ywlOLHNe8SNQ3OunFbu2Y=
+github.com/filecoin-project/specs-actors/v2 v2.3.3 h1:5Pd6pjU7VjUye+Hz4gYBCPAFdBxtEbHsgGYvWmfc83w=
+github.com/filecoin-project/specs-actors/v2 v2.3.3/go.mod h1:UuJQLoTx/HPvvWeqlIFmC/ywlOLHNe8SNQ3OunFbu2Y=
 github.com/filecoin-project/specs-storage v0.1.1-0.20201105051918-5188d9774506 h1:Ur/l2+6qN+lQiqjozWWc5p9UDaAMDZKTlDS98oRnlIw=
 github.com/filecoin-project/specs-storage v0.1.1-0.20201105051918-5188d9774506/go.mod h1:nJRRM7Aa9XVvygr3W9k6xGF46RWzr2zxF/iGoAIfA/g=
 github.com/filecoin-project/test-vectors/schema v0.0.5 h1:w3zHQhzM4pYxJDl21avXjOKBLF8egrvwUwjpT8TquDg=
@@ -923,8 +925,8 @@ github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1
 github.com/libp2p/go-libp2p-protocol v0.1.0/go.mod h1:KQPHpAabB57XQxGrXCNvbL6UEXfQqUgC/1adR2Xtflk=
 github.com/libp2p/go-libp2p-pubsub v0.1.1/go.mod h1:ZwlKzRSe1eGvSIdU5bD7+8RZN/Uzw0t1Bp9R1znpR/Q=
 github.com/libp2p/go-libp2p-pubsub v0.3.2-0.20200527132641-c0712c6e92cf/go.mod h1:TxPOBuo1FPdsTjFnv+FGZbNbWYsp74Culx+4ViQpato=
-github.com/libp2p/go-libp2p-pubsub v0.4.0 h1:YNVRyXqBgv9i4RG88jzoTtkSOaSB45CqHkL29NNBZb4=
-github.com/libp2p/go-libp2p-pubsub v0.4.0/go.mod h1:izkeMLvz6Ht8yAISXjx60XUQZMq9ZMe5h2ih4dLIBIQ=
+github.com/libp2p/go-libp2p-pubsub v0.4.1 h1:j4umIg5nyus+sqNfU+FWvb9aeYFQH/A+nDFhWj+8yy8=
+github.com/libp2p/go-libp2p-pubsub v0.4.1/go.mod h1:izkeMLvz6Ht8yAISXjx60XUQZMq9ZMe5h2ih4dLIBIQ=
 github.com/libp2p/go-libp2p-quic-transport v0.1.1/go.mod h1:wqG/jzhF3Pu2NrhJEvE+IE0NTHNXslOPn9JQzyCAxzU=
 github.com/libp2p/go-libp2p-quic-transport v0.5.0/go.mod h1:IEcuC5MLxvZ5KuHKjRu+dr3LjCT1Be3rcD/4d8JrX8M=
 github.com/libp2p/go-libp2p-quic-transport v0.9.0 h1:WPuq5nV/chmIZIzvrkC2ulSdAQ0P0BDvgvAhZFOZ59E=

lib/blockstore/badger/blockstore_test_suite.go

@@ -183,6 +183,8 @@ func (s *Suite) TestAllKeysRespectsContext(t *testing.T) {
 	require.True(t, ok)
 
 	cancel()
+	// pull one value out to avoid race
+	_, _ = <-ch
 
 	v, ok = <-ch
 	require.Equal(t, cid.Undef, v)

markets/storageadapter/client.go

@@ -264,7 +264,7 @@ func (c *ClientNodeAdapter) OnDealExpiredOrSlashed(ctx context.Context, dealID a
 	// and the chain has advanced to the confidence height
 	stateChanged := func(ts *types.TipSet, ts2 *types.TipSet, states events.StateChange, h abi.ChainEpoch) (more bool, err error) {
 		// Check if the deal has already expired
-		if sd.Proposal.EndEpoch <= ts2.Height() {
+		if ts2 == nil || sd.Proposal.EndEpoch <= ts2.Height() {
 			onDealExpired(nil)
 			return false, nil
 		}

markets/storageadapter/provider.go

@@ -344,7 +344,7 @@ func (n *ProviderNodeAdapter) OnDealExpiredOrSlashed(ctx context.Context, dealID
 	// and the chain has advanced to the confidence height
 	stateChanged := func(ts *types.TipSet, ts2 *types.TipSet, states events.StateChange, h abi.ChainEpoch) (more bool, err error) {
 		// Check if the deal has already expired
-		if sd.Proposal.EndEpoch <= ts2.Height() {
+		if ts2 == nil || sd.Proposal.EndEpoch <= ts2.Height() {
 			onDealExpired(nil)
 			return false, nil
 		}

node/impl/market/market.go

@@ -3,21 +3,49 @@ package market
 import (
 	"context"
 
+	"github.com/ipfs/go-cid"
 	"go.uber.org/fx"
 
-	"github.com/ipfs/go-cid"
-
 	"github.com/filecoin-project/go-address"
+	"github.com/filecoin-project/lotus/chain/actors"
+	marketactor "github.com/filecoin-project/lotus/chain/actors/builtin/market"
 	"github.com/filecoin-project/lotus/chain/market"
 	"github.com/filecoin-project/lotus/chain/types"
+	"github.com/filecoin-project/lotus/node/impl/full"
 )
 
 type MarketAPI struct {
 	fx.In
 
+	full.MpoolAPI
 	FMgr *market.FundManager
 }
 
+func (a *MarketAPI) MarketAddBalance(ctx context.Context, wallet, addr address.Address, amt types.BigInt) (cid.Cid, error) {
+	params, err := actors.SerializeParams(&addr)
+	if err != nil {
+		return cid.Undef, err
+	}
+
+	smsg, aerr := a.MpoolPushMessage(ctx, &types.Message{
+		To:     marketactor.Address,
+		From:   wallet,
+		Value:  amt,
+		Method: marketactor.Methods.AddBalance,
+		Params: params,
+	}, nil)
+
+	if aerr != nil {
+		return cid.Undef, aerr
+	}
+
+	return smsg.Cid(), nil
+}
+
+func (a *MarketAPI) MarketGetReserved(ctx context.Context, addr address.Address) (types.BigInt, error) {
+	return a.FMgr.GetReserved(addr), nil
+}
+
 func (a *MarketAPI) MarketReserveFunds(ctx context.Context, wallet address.Address, addr address.Address, amt types.BigInt) (cid.Cid, error) {
 	return a.FMgr.Reserve(ctx, wallet, addr, amt)
 }