fix setup of burnt funds actor state

Address potential mpool state inconsistencies

CHANGELOG.md

@@ -1,7 +1,148 @@
-# lotus changelog
+# Lotus changelog
+
+## 0.5.0 / 2020-08-2020
+
+This version of Lotus will be used for the incentivized testnet Space Race competition,
+and can be considered mainnet-ready code. It includes some protocol
+changes, upgrades of core dependencies, and various bugfixes and UX/performance improvements.
+
+# Highlights
+
+Among the highlights included in this release are:
+
+- Gas changes: We implemented EIP-1559 and introduced real gas values.
+- Deal-making: We now support "Committed Capacity" sectors, "fast-retrieval" deals,
+and the packing of multiple deals into a single sector.
+- Renamed features: We renamed some of the binaries, environment variables, and default
+paths associated with a Lotus node.
+
+## Gas changes
+
+We made some significant changes to the mechanics of gas in this release.
+
+#### Network fee
+
+We implemented something similar to 
+[Ethereum's EIP-1559](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1559.md).
+The `Message` structure had three changes:
+- The `GasPrice` field has been removed
+- A new `GasFeeCap` field has been added, which controls the maximum cost
+the sender incurs for the message
+- A new `GasPremium` field has been added, which controls the reward a miner
+earns for including the message
+
+A sender will never be charged more than `GasFeeCap * GasLimit`. 
+A miner will typically earn `GasPremium * GasLimit` as a reward.
+
+The `Blockheader` structure has one new field, called `ParentBaseFee`. 
+Informally speaking,the `ParentBaseFee`
+is increased when blocks are densely packed with messages, and decreased otherwise.
+
+The `ParentBaseFee` is used when calculating how much a sender burns when executing a message. _Burning_ simply refers to sending attoFIL to a dedicated, unreachable account.
+A message causes `ParentBaseFee * GasUsed` attoFIL to be burnt.
+
+#### Real gas values
+
+This release also includes our first "real" gas costs for primitive operations.
+The costs were designed to account for both the _time_ that message execution takes,
+as well as the _space_ a message adds to the state tree.
+
+## Deal-making changes
+
+There are three key changes to the deal-making process.
+
+#### Committed Capacity sectors
+
+Miners can now pledge "Committed Capacity" (CC) sectors, which are explicitly
+stated as containing junk data, and must not include any deals. Miners can do this
+to increase their storage power, and win block rewards from this pledged storage.
+
+They can mark these sectors as "upgradable" with `lotus-miner sectors mark-for-upgrade`.
+If the miner receives and accepts one or more storage deals, the sector that includes
+those deals will _replace_ the CC sector. This is intended to maximize the amount of useful
+storage on the Filecoin network.
+
+#### Fast-retrieval deals
+
+Clients can now include a `fast-retrieval` flag when proposing deals with storage miners.
+If set to true, the miner will include an extra copy of the deal data. This
+data can be quickly served in a retrieval deal, since it will not need to be unsealed.
+
+#### Multiple deals per sector
+
+Miners can now pack multiple deals into a single sector, so long as all the deals
+fit into the sector capacity. This should increase the packing efficiency of miners.
+
+## Renamed features
+
+To improve the user experience, we updated several names to mainatin
+standard prefixing, and to better reflect the meaning of the features being referenced.
+
+In particular, the Lotus miner binary is now called `lotus-miner`, the default
+path for miner data is now `~/.lotusminer`, and the environment variable
+that sets the path for miner data is now `$LOTUS_MINER_PATH`. A full list of renamed
+features can be found [here](https://github.com/filecoin-project/lotus/issues/2304).
+
+# Changelog
+
+#### Downstream upgrades
+- Upgrades markets to v0.5.6 (https://github.com/filecoin-project/lotus/pull/3058)
+- Upgrades specs-actors to v0.9.3 (https://github.com/filecoin-project/lotus/pull/3151)
+
+#### Core protocol
+- Introduces gas values, replacing placeholders (https://github.com/filecoin-project/lotus/pull/2343)
+- Implements EIP-1559, introducing a network base fee, message gas fee cap, and message gas fee premium (https://github.com/filecoin-project/lotus/pull/2874)
+- Implements Poisson Sortition for elections (https://github.com/filecoin-project/lotus/pull/2084)
+
+#### Deal-making lifecycle
+- Introduces "Committed Capacity" sectors (https://github.com/filecoin-project/lotus/pull/2220)
+- Introduces "fast-retrieval" flag for deals (https://github.com/filecoin-project/lotus/pull/2323
+- Supports packing multiple deals into one sector (https://github.com/filecoin-project/storage-fsm/pull/38)
+
+#### Enhancements
+
+- Optimized message pool selection logic (https://github.com/filecoin-project/lotus/pull/2838)
+- Window-based scheduling of sealing tasks (https://github.com/filecoin-project/sector-storage/pull/67)
+- Faster window PoSt (https://github.com/filecoin-project/lotus/pull/2209/files)
+- Refactors the payment channel manager (https://github.com/filecoin-project/lotus/pull/2640)
+- Refactors blocksync (https://github.com/filecoin-project/lotus/pull/2715/files)
+
+#### UX
+
+- Provide status updates for data-transfer (https://github.com/filecoin-project/lotus/pull/3162, https://github.com/filecoin-project/lotus/pull/3191)
+- Miners can customise asks (https://github.com/filecoin-project/lotus/pull/2046)
+- Miners can toggle auto-acceptance of deals (https://github.com/filecoin-project/lotus/pull/1994)
+- Miners can maintain a blocklist of piece CIDs (https://github.com/filecoin-project/lotus/pull/2069)
+
+# Contributors
+
+The following contributors had 10 or more commits go into this release.
+We are grateful for every contribution!
+
+| Contributor        | Commits | Lines ±       |
+|--------------------|---------|---------------|
+| magik6k            | 361     | +13197/-6136  |
+| Kubuxu             | 227     | +5670/-2587   |
+| arajasek           | 120     | +2916/-1264   |
+| whyrusleeping      | 112     | +3979/-1089   |
+| vyzo               | 99      | +3343/-1305   |
+| dirkmc             | 68      | +8732/-3621   |
+| laser              | 45      | +1489/-501    |
+| hannahhoward       | 43      | +2654/-990    |
+| frrist             | 37      | +6630/-4338   |
+| schomatis          | 28      | +3016/-1368   |
+| placer14           | 27      | +824/-350     |
+| raulk              | 25      | +28718/-29849 |
+| mrsmkl             | 22      | +560/-368     |
+| travisperson       | 18      | +1354/-314    |
+| nonsense           | 16      | +2956/-2842   |
+| ingar              | 13      | +331/-123     |
+| daviddias          | 11      | +311/-11      |
+| Stebalien          | 11      | +1204/-980    |
+| RobQuistNL         | 10      | +69/-74       |
 
 ## 0.1.0 / 2019-12-11
 
-We are very excited to release **lotus** 0.1.0. This is our testnet release. To install lotus and join the testnet, please visit [docs.lotu.sh](docs.lotu.sh). Please file bug reports as [issues](https://github.com/filecoin-project/lotus/issues).
+We are very excited to release **lotus** 0.1.0. This is our testnet release. To install lotus and join the testnet, please visit [lotu.sh](lotu.sh). Please file bug reports as [issues](https://github.com/filecoin-project/lotus/issues).
 
 A huge thank you to all contributors for this testnet release!

README.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <a href="https://docs.lotu.sh/" title="Lotus Docs">
+  <a href="https://lotu.sh/" title="Lotus Docs">
     <img src="documentation/images/lotus_logo_h.png" alt="Project Lotus Logo" width="244" />
   </a>
 </p>
@@ -18,7 +18,7 @@ Lotus is an implementation of the Filecoin Distributed Storage Network. For more
 
 ## Building & Documentation
 
-For instructions on how to build lotus from source, please visit [https://docs.lotu.sh](https://docs.lotu.sh) or read the source [here](https://github.com/filecoin-project/lotus/tree/master/documentation).
+For instructions on how to build lotus from source, please visit [https://lotu.sh](https://lotu.sh) or read the source [here](https://github.com/filecoin-project/lotus/tree/master/documentation).
 
 ## Reporting a Vulnerability
 

api/api_full.go

@@ -184,6 +184,9 @@ type FullNode interface {
 	MpoolGetNonce(context.Context, address.Address) (uint64, error)
 	MpoolSub(context.Context) (<-chan MpoolUpdate, error)
 
+	// MpoolClear clears pending messages from the mpool
+	MpoolClear(context.Context, bool) error
+
 	// MpoolGetConfig returns (a copy of) the current mpool config
 	MpoolGetConfig(context.Context) (*types.MpoolConfig, error)
 	// MpoolSetConfig sets the mpool config to (a copy of) the supplied config
@@ -478,6 +481,8 @@ type DealInfo struct {
 	Duration      uint64
 
 	DealID abi.DealID
+
+	CreationTime time.Time
 }
 
 type MsgLookup struct {

api/api_storage.go

@@ -79,6 +79,8 @@ type StorageMiner interface {
 	MarketGetAsk(ctx context.Context) (*storagemarket.SignedStorageAsk, error)
 	MarketSetRetrievalAsk(ctx context.Context, rask *retrievalmarket.Ask) error
 	MarketGetRetrievalAsk(ctx context.Context) (*retrievalmarket.Ask, error)
+	MarketListDataTransfers(ctx context.Context) ([]DataTransferChannel, error)
+	MarketDataTransferUpdates(ctx context.Context) (<-chan DataTransferChannel, error)
 
 	DealsImportData(ctx context.Context, dealPropCid cid.Cid, file string) error
 	DealsList(ctx context.Context) ([]storagemarket.StorageDeal, error)

api/apistruct/struct.go

@@ -106,7 +106,9 @@ type FullNodeStruct struct {
 
 		MpoolSelect func(context.Context, types.TipSetKey, float64) ([]*types.SignedMessage, error) `perm:"read"`
 
-		MpoolPending     func(context.Context, types.TipSetKey) ([]*types.SignedMessage, error)                    `perm:"read"`
+		MpoolPending func(context.Context, types.TipSetKey) ([]*types.SignedMessage, error) `perm:"read"`
+		MpoolClear   func(context.Context, bool) error                                      `perm:"write"`
+
 		MpoolPush        func(context.Context, *types.SignedMessage) (cid.Cid, error)                              `perm:"write"`
 		MpoolPushMessage func(context.Context, *types.Message, *api.MessageSendSpec) (*types.SignedMessage, error) `perm:"sign"`
 		MpoolGetNonce    func(context.Context, address.Address) (uint64, error)                                    `perm:"read"`
@@ -238,6 +240,8 @@ type StorageMinerStruct struct {
 		MarketGetAsk              func(ctx context.Context) (*storagemarket.SignedStorageAsk, error)                                                                                                           `perm:"read"`
 		MarketSetRetrievalAsk     func(ctx context.Context, rask *retrievalmarket.Ask) error                                                                                                                   `perm:"admin"`
 		MarketGetRetrievalAsk     func(ctx context.Context) (*retrievalmarket.Ask, error)                                                                                                                      `perm:"read"`
+		MarketListDataTransfers   func(ctx context.Context) ([]api.DataTransferChannel, error)                                                                                                                 `perm:"write"`
+		MarketDataTransferUpdates func(ctx context.Context) (<-chan api.DataTransferChannel, error)                                                                                                            `perm:"write"`
 
 		PledgeSector func(context.Context) error `perm:"write"`
 
@@ -492,6 +496,10 @@ func (c *FullNodeStruct) MpoolPending(ctx context.Context, tsk types.TipSetKey)
 	return c.Internal.MpoolPending(ctx, tsk)
 }
 
+func (c *FullNodeStruct) MpoolClear(ctx context.Context, local bool) error {
+	return c.Internal.MpoolClear(ctx, local)
+}
+
 func (c *FullNodeStruct) MpoolPush(ctx context.Context, smsg *types.SignedMessage) (cid.Cid, error) {
 	return c.Internal.MpoolPush(ctx, smsg)
 }
@@ -1080,6 +1088,14 @@ func (c *StorageMinerStruct) MarketGetRetrievalAsk(ctx context.Context) (*retrie
 	return c.Internal.MarketGetRetrievalAsk(ctx)
 }
 
+func (c *StorageMinerStruct) MarketListDataTransfers(ctx context.Context) ([]api.DataTransferChannel, error) {
+	return c.Internal.MarketListDataTransfers(ctx)
+}
+
+func (c *StorageMinerStruct) MarketDataTransferUpdates(ctx context.Context) (<-chan api.DataTransferChannel, error) {
+	return c.Internal.MarketDataTransferUpdates(ctx)
+}
+
 func (c *StorageMinerStruct) DealsImportData(ctx context.Context, dealPropCid cid.Cid, file string) error {
 	return c.Internal.DealsImportData(ctx, dealPropCid, file)
 }

api/types.go

@@ -2,6 +2,7 @@ package api
 
 import (
 	"encoding/json"
+	"fmt"
 
 	"github.com/filecoin-project/go-address"
 	datatransfer "github.com/filecoin-project/go-data-transfer"
@@ -117,3 +118,35 @@ type DataTransferChannel struct {
 	OtherPeer   peer.ID
 	Transferred uint64
 }
+
+// NewDataTransferChannel constructs an API DataTransferChannel type from full channel state snapshot and a host id
+func NewDataTransferChannel(hostID peer.ID, channelState datatransfer.ChannelState) DataTransferChannel {
+	channel := DataTransferChannel{
+		TransferID: channelState.TransferID(),
+		Status:     channelState.Status(),
+		BaseCID:    channelState.BaseCID(),
+		IsSender:   channelState.Sender() == hostID,
+		Message:    channelState.Message(),
+	}
+	stringer, ok := channelState.Voucher().(fmt.Stringer)
+	if ok {
+		channel.Voucher = stringer.String()
+	} else {
+		voucherJSON, err := json.Marshal(channelState.Voucher())
+		if err != nil {
+			channel.Voucher = fmt.Errorf("Voucher Serialization: %w", err).Error()
+		} else {
+			channel.Voucher = string(voucherJSON)
+		}
+	}
+	if channel.IsSender {
+		channel.IsInitiator = !channelState.IsPull()
+		channel.Transferred = channelState.Sent()
+		channel.OtherPeer = channelState.Recipient()
+	} else {
+		channel.IsInitiator = channelState.IsPull()
+		channel.Transferred = channelState.Received()
+		channel.OtherPeer = channelState.Sender()
+	}
+	return channel
+}

build/bootstrap/bootstrappers.pi

@@ -1,12 +1,6 @@
-/dns4/bootstrap-0-sin.fil-test.net/tcp/1347/p2p/12D3KooWPdUquftaQvoQEtEdsRBAhwD6jopbF2oweVTzR59VbHEd
-/ip4/86.109.15.57/tcp/1347/p2p/12D3KooWPdUquftaQvoQEtEdsRBAhwD6jopbF2oweVTzR59VbHEd
-/dns4/bootstrap-0-dfw.fil-test.net/tcp/1347/p2p/12D3KooWQSCkHCzosEyrh8FgYfLejKgEPM5VB6qWzZE3yDAuXn8d
-/ip4/139.178.84.45/tcp/1347/p2p/12D3KooWQSCkHCzosEyrh8FgYfLejKgEPM5VB6qWzZE3yDAuXn8d
-/dns4/bootstrap-0-fra.fil-test.net/tcp/1347/p2p/12D3KooWEXN2eQmoyqnNjde9PBAQfQLHN67jcEdWU6JougWrgXJK
-/ip4/136.144.49.17/tcp/1347/p2p/12D3KooWEXN2eQmoyqnNjde9PBAQfQLHN67jcEdWU6JougWrgXJK
-/dns4/bootstrap-1-sin.fil-test.net/tcp/1347/p2p/12D3KooWLmJkZd33mJhjg5RrpJ6NFep9SNLXWc4uVngV4TXKwzYw
-/ip4/86.109.15.123/tcp/1347/p2p/12D3KooWLmJkZd33mJhjg5RrpJ6NFep9SNLXWc4uVngV4TXKwzYw
-/dns4/bootstrap-1-dfw.fil-test.net/tcp/1347/p2p/12D3KooWGXLHjiz6pTRu7x2pkgTVCoxcCiVxcNLpMnWcJ3JiNEy5
-/ip4/139.178.86.3/tcp/1347/p2p/12D3KooWGXLHjiz6pTRu7x2pkgTVCoxcCiVxcNLpMnWcJ3JiNEy5
-/dns4/bootstrap-1-fra.fil-test.net/tcp/1347/p2p/12D3KooW9szZmKttS9A1FafH3Zc2pxKwwmvCWCGKkRP4KmbhhC4R
-/ip4/136.144.49.131/tcp/1347/p2p/12D3KooW9szZmKttS9A1FafH3Zc2pxKwwmvCWCGKkRP4KmbhhC4R
+/dns4/bootstrap-0.testnet.fildev.network/tcp/1347/p2p/12D3KooWJTUBUjtzWJGWU1XSiY21CwmHaCNLNYn2E7jqHEHyZaP7
+/dns4/bootstrap-1.testnet.fildev.network/tcp/1347/p2p/12D3KooW9yeKXha4hdrJKq74zEo99T8DhriQdWNoojWnnQbsgB3v
+/dns4/bootstrap-2.testnet.fildev.network/tcp/1347/p2p/12D3KooWCrx8yVG9U9Kf7w8KLN3Edkj5ZKDhgCaeMqQbcQUoB6CT
+/dns4/bootstrap-4.testnet.fildev.network/tcp/1347/p2p/12D3KooWPkL9LrKRQgHtq7kn9ecNhGU9QaziG8R5tX8v9v7t3h34
+/dns4/bootstrap-3.testnet.fildev.network/tcp/1347/p2p/12D3KooWKYSsbpgZ3HAjax5M1BXCwXLa6gVkUARciz7uN3FNtr7T
+/dns4/bootstrap-5.testnet.fildev.network/tcp/1347/p2p/12D3KooWQYzqnLASJAabyMpPb1GcWZvNSe7JDcRuhdRqonFoiK9W

build/drand.go

@@ -2,7 +2,7 @@ package build
 
 import "github.com/filecoin-project/lotus/node/modules/dtypes"
 
-var DrandNetwork = DrandMainnet
+var DrandNetwork = DrandIncentinet
 
 func DrandConfig() dtypes.DrandConfig {
 	return DrandConfigs[DrandNetwork]
@@ -15,6 +15,7 @@ const (
 	DrandTestnet
 	DrandDevnet
 	DrandLocalnet
+	DrandIncentinet
 )
 
 var DrandConfigs = map[DrandEnum]dtypes.DrandConfig{
@@ -55,4 +56,17 @@ var DrandConfigs = map[DrandEnum]dtypes.DrandConfig{
 		},
 		ChainInfoJSON: `{"public_key":"8cda589f88914aa728fd183f383980b35789ce81b274e5daee1f338b77d02566ef4d3fb0098af1f844f10f9c803c1827","period":25,"genesis_time":1595348225,"hash":"e73b7dc3c4f6a236378220c0dd6aa110eb16eed26c11259606e07ee122838d4f","groupHash":"567d4785122a5a3e75a9bc9911d7ea807dd85ff76b78dc4ff06b075712898607"}`,
 	},
+	DrandIncentinet: {
+		Servers: []string{
+			"https://pl-eu.incentinet.drand.sh",
+			"https://pl-us.incentinet.drand.sh",
+			"https://pl-sin.incentinet.drand.sh",
+		},
+		Relays: []string{
+			"/dnsaddr/pl-eu.incentinet.drand.sh/",
+			"/dnsaddr/pl-us.incentinet.drand.sh/",
+			"/dnsaddr/pl-sin.incentinet.drand.sh/",
+		},
+		ChainInfoJSON: `{"public_key":"8cad0c72c606ab27d36ee06de1d5b2db1faf92e447025ca37575ab3a8aac2eaae83192f846fc9e158bc738423753d000","period":30,"genesis_time":1595873820,"hash":"80c8b872c714f4c00fdd3daa465d5514049f457f01f85a4caf68cdcd394ba039","groupHash":"d9406aaed487f7af71851b4399448e311f2328923d454e971536c05398ce2d9b"}`,
+	},
 }

build/params_testnet.go

@@ -13,7 +13,7 @@ import (
 )
 
 func init() {
-	power.ConsensusMinerMinPower = big.NewInt(1024 << 30)
+	power.ConsensusMinerMinPower = big.NewInt(10 << 40)
 	miner.SupportedProofTypes = map[abi.RegisteredSealProof]struct{}{
 		abi.RegisteredSealProof_StackedDrg32GiBV1: {},
 		abi.RegisteredSealProof_StackedDrg64GiBV1: {},

chain/gen/genesis/genesis.go

@@ -2,6 +2,7 @@ package genesis
 
 import (
 	"context"
+	"crypto/rand"
 	"encoding/json"
 	"fmt"
 
@@ -118,11 +119,6 @@ func MakeInitialStateTree(ctx context.Context, bs bstore.Blockstore, template ge
 		return nil, nil, xerrors.Errorf("making new state tree: %w", err)
 	}
 
-	emptyobject, err := cst.Put(context.TODO(), []struct{}{})
-	if err != nil {
-		return nil, nil, xerrors.Errorf("failed putting empty object: %w", err)
-	}
-
 	// Create system actor
 
 	sysact, err := SetupSystemActor(bs)
@@ -191,11 +187,18 @@ func MakeInitialStateTree(ctx context.Context, bs bstore.Blockstore, template ge
 		return nil, nil, xerrors.Errorf("set market actor: %w", err)
 	}
 
+	burntRoot, err := cst.Put(ctx, &account.State{
+		Address: builtin.BurntFundsActorAddr,
+	})
+	if err != nil {
+		return nil, nil, xerrors.Errorf("failed to setup burnt funds actor state: %w", err)
+	}
+
 	// Setup burnt-funds
 	err = state.SetActor(builtin.BurntFundsActorAddr, &types.Actor{
 		Code:    builtin.AccountActorCodeID,
 		Balance: types.NewInt(0),
-		Head:    emptyobject,
+		Head:    burntRoot,
 	})
 	if err != nil {
 		return nil, nil, xerrors.Errorf("set burnt funds account actor: %w", err)
@@ -297,6 +300,8 @@ func MakeInitialStateTree(ctx context.Context, bs bstore.Blockstore, template ge
 		return nil, nil, err
 	}
 
+	template.RemainderAccount.Balance = remainingFil
+
 	if err := createMultisigAccount(ctx, bs, cst, state, remAccKey, template.RemainderAccount, keyIDs); err != nil {
 		return nil, nil, xerrors.Errorf("failed to set up remainder account: %w", err)
 	}
@@ -508,8 +513,10 @@ func MakeGenesisBlock(ctx context.Context, bs bstore.Blockstore, sys vm.SyscallB
 
 	log.Infof("Empty Genesis root: %s", emptyroot)
 
+	tickBuf := make([]byte, 32)
+	_, _ = rand.Read(tickBuf)
 	genesisticket := &types.Ticket{
-		VRFProof: []byte("vrf proof0000000vrf proof0000000"),
+		VRFProof: tickBuf,
 	}
 
 	filecoinGenesisCid, err := cid.Decode("bafyreiaqpwbbyjo4a42saasj36kkrpv4tsherf2e7bvezkert2a7dhonoi")

chain/messagepool/messagepool.go

@@ -12,6 +12,7 @@ import (
 
 	"github.com/filecoin-project/specs-actors/actors/abi"
 	"github.com/filecoin-project/specs-actors/actors/crypto"
+	"github.com/hashicorp/go-multierror"
 	lru "github.com/hashicorp/golang-lru"
 	"github.com/ipfs/go-cid"
 	"github.com/ipfs/go-datastore"
@@ -744,30 +745,42 @@ func (mp *MessagePool) HeadChange(revert []*types.TipSet, apply []*types.TipSet)
 		}
 	}
 
+	var merr error
+
 	for _, ts := range revert {
 		pts, err := mp.api.LoadTipSet(ts.Parents())
 		if err != nil {
-			return err
-		}
-
-		msgs, err := mp.MessagesForBlocks(ts.Blocks())
-		if err != nil {
-			return err
+			log.Errorf("error loading reverted tipset parent: %s", err)
+			merr = multierror.Append(merr, err)
+			continue
 		}
 
 		mp.curTs = pts
 
+		msgs, err := mp.MessagesForBlocks(ts.Blocks())
+		if err != nil {
+			log.Errorf("error retrieving messages for reverted block: %s", err)
+			merr = multierror.Append(merr, err)
+			continue
+		}
+
 		for _, msg := range msgs {
 			add(msg)
 		}
 	}
 
 	for _, ts := range apply {
+		mp.curTs = ts
+
 		for _, b := range ts.Blocks() {
 			bmsgs, smsgs, err := mp.api.MessagesForBlock(b)
 			if err != nil {
-				return xerrors.Errorf("failed to get messages for apply block %s(height %d) (msgroot = %s): %w", b.Cid(), b.Height, b.Messages, err)
+				xerr := xerrors.Errorf("failed to get messages for apply block %s(height %d) (msgroot = %s): %w", b.Cid(), b.Height, b.Messages, err)
+				log.Errorf("error retrieving messages for block: %s", xerr)
+				merr = multierror.Append(merr, xerr)
+				continue
 			}
+
 			for _, msg := range smsgs {
 				rm(msg.Message.From, msg.Message.Nonce)
 				maybeRepub(msg.Cid())
@@ -778,8 +791,6 @@ func (mp *MessagePool) HeadChange(revert []*types.TipSet, apply []*types.TipSet)
 				maybeRepub(msg.Cid())
 			}
 		}
-
-		mp.curTs = ts
 	}
 
 	if repubTrigger {
@@ -862,7 +873,7 @@ func (mp *MessagePool) HeadChange(revert []*types.TipSet, apply []*types.TipSet)
 		}
 	}
 
-	return nil
+	return merr
 }
 
 type statBucket struct {
@@ -962,3 +973,40 @@ func (mp *MessagePool) loadLocal() error {
 
 	return nil
 }
+
+func (mp *MessagePool) Clear(local bool) {
+	mp.lk.Lock()
+	defer mp.lk.Unlock()
+
+	// remove everything if local is true, including removing local messages from
+	// the datastore
+	if local {
+		for a := range mp.localAddrs {
+			mset, ok := mp.pending[a]
+			if !ok {
+				continue
+			}
+
+			for _, m := range mset.msgs {
+				err := mp.localMsgs.Delete(datastore.NewKey(string(m.Cid().Bytes())))
+				if err != nil {
+					log.Warnf("error deleting local message: %s", err)
+				}
+			}
+		}
+
+		mp.pending = make(map[address.Address]*msgSet)
+		mp.republished = nil
+
+		return
+	}
+
+	// remove everything except the local messages
+	for a := range mp.pending {
+		_, isLocal := mp.localAddrs[a]
+		if isLocal {
+			continue
+		}
+		delete(mp.pending, a)
+	}
+}

cli/client.go

@@ -997,6 +997,10 @@ var clientListDeals = &cli.Command{
 			return err
 		}
 
+		sort.Slice(localDeals, func(i, j int) bool {
+			return localDeals[i].CreationTime.Before(localDeals[j].CreationTime)
+		})
+
 		var deals []deal
 		for _, v := range localDeals {
 			if v.DealID == 0 {
@@ -1025,7 +1029,7 @@ var clientListDeals = &cli.Command{
 
 		if cctx.Bool("verbose") {
 			w := tabwriter.NewWriter(os.Stdout, 2, 4, 2, ' ', 0)
-			fmt.Fprintf(w, "DealCid\tDealId\tProvider\tState\tOn Chain?\tSlashed?\tPieceCID\tSize\tPrice\tDuration\tMessage\n")
+			fmt.Fprintf(w, "Created\tDealCid\tDealId\tProvider\tState\tOn Chain?\tSlashed?\tPieceCID\tSize\tPrice\tDuration\tMessage\n")
 			for _, d := range deals {
 				onChain := "N"
 				if d.OnChainDealState.SectorStartEpoch != -1 {
@@ -1038,7 +1042,7 @@ var clientListDeals = &cli.Command{
 				}
 
 				price := types.FIL(types.BigMul(d.LocalDeal.PricePerEpoch, types.NewInt(d.LocalDeal.Duration)))
-				fmt.Fprintf(w, "%s\t%d\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%d\t%s\n", d.LocalDeal.ProposalCid, d.LocalDeal.DealID, d.LocalDeal.Provider, dealStateString(color, d.LocalDeal.State), onChain, slashed, d.LocalDeal.PieceCID, types.SizeStr(types.NewInt(d.LocalDeal.Size)), price, d.LocalDeal.Duration, d.LocalDeal.Message)
+				fmt.Fprintf(w, "%s\t%s\t%d\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%d\t%s\n", d.LocalDeal.CreationTime.Format(time.Stamp), d.LocalDeal.ProposalCid, d.LocalDeal.DealID, d.LocalDeal.Provider, dealStateString(color, d.LocalDeal.State), onChain, slashed, d.LocalDeal.PieceCID, types.SizeStr(types.NewInt(d.LocalDeal.Size)), price, d.LocalDeal.Duration, d.LocalDeal.Message)
 			}
 			return w.Flush()
 		} else {
@@ -1254,7 +1258,7 @@ var clientListTransfers = &cli.Command{
 
 				tm.MoveCursor(1, 1)
 
-				outputChannels(tm.Screen, channels, completed, color)
+				OutputDataTransferChannels(tm.Screen, channels, completed, color)
 
 				tm.Flush()
 
@@ -1279,12 +1283,13 @@ var clientListTransfers = &cli.Command{
 				}
 			}
 		}
-		outputChannels(os.Stdout, channels, completed, color)
+		OutputDataTransferChannels(os.Stdout, channels, completed, color)
 		return nil
 	},
 }
 
-func outputChannels(out io.Writer, channels []api.DataTransferChannel, completed bool, color bool) {
+// OutputDataTransferChannels generates table output for a list of channels
+func OutputDataTransferChannels(out io.Writer, channels []lapi.DataTransferChannel, completed bool, color bool) {
 	sort.Slice(channels, func(i, j int) bool {
 		return channels[i].TransferID < channels[j].TransferID
 	})
@@ -1346,7 +1351,7 @@ func channelStatusString(useColor bool, status datatransfer.Status) string {
 	}
 }
 
-func toChannelOutput(useColor bool, otherPartyColumn string, channel api.DataTransferChannel) map[string]interface{} {
+func toChannelOutput(useColor bool, otherPartyColumn string, channel lapi.DataTransferChannel) map[string]interface{} {
 	rootCid := channel.BaseCID.String()
 	rootCid = "..." + rootCid[len(rootCid)-8:]
 

cli/mpool.go

@@ -20,6 +20,7 @@ var mpoolCmd = &cli.Command{
 	Usage: "Manage message pool",
 	Subcommands: []*cli.Command{
 		mpoolPending,
+		mpoolClear,
 		mpoolSub,
 		mpoolStat,
 		mpoolReplaceCmd,
@@ -83,6 +84,38 @@ var mpoolPending = &cli.Command{
 	},
 }
 
+var mpoolClear = &cli.Command{
+	Name:  "clear",
+	Usage: "Clear all pending messages from the mpool (USE WITH CARE)",
+	Flags: []cli.Flag{
+		&cli.BoolFlag{
+			Name:  "local",
+			Usage: "also clear local messages",
+		},
+		&cli.BoolFlag{
+			Name:  "really-do-it",
+			Usage: "must be specified for the action to take effect",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		api, closer, err := GetFullNodeAPI(cctx)
+		if err != nil {
+			return err
+		}
+		defer closer()
+
+		really := cctx.Bool("really-do-it")
+		if !really {
+			return fmt.Errorf("--really-do-it must be specified for this action to have an effect; you have been warned.")
+		}
+
+		local := cctx.Bool("local")
+
+		ctx := ReqContext(cctx)
+		return api.MpoolClear(ctx, local)
+	},
+}
+
 var mpoolSub = &cli.Command{
 	Name:  "sub",
 	Usage: "Subscribe to mpool changes",

cmd/lotus-chainwatch/processor/reward.go

@@ -280,6 +280,9 @@ func (p *Processor) storeRewardSmoothingEstimates(rewards []rewardActorInfo) err
 	}
 
 	for _, rewardState := range rewards {
+		if rewardState.epochSmoothingEstimate == nil {
+			continue
+		}
 		if _, err := stmt.Exec(
 			rewardState.common.stateroot.String(),
 			rewardState.epochSmoothingEstimate.PositionEstimate.String(),

cmd/lotus-fountain/site/_miner.html

@@ -29,7 +29,7 @@
             <span>When creating miner, DO NOT REFRESH THE PAGE, wait for it to load. This can take more than 5min.</span>
         </div>
         <div class="Index-node">
-            <span>If you don't have an owner/worker address, you can create it by following <a target="_blank" href="https://docs.lotu.sh/en+mining#get-started-22083">these instructions</a>.</span>
+            <span>If you don't have an owner/worker address, you can create it by following <a target="_blank" href="https://lotu.sh/en+mining#get-started-22083">these instructions</a>.</span>
         </div>
     </div>
     <div class="Index-footer">

cmd/lotus-pcr/main.go

@@ -394,7 +394,7 @@ func (r *refunder) ProcessTipset(ctx context.Context, tipset *types.TipSet) (*Mi
 		}
 
 		if r.percentExtra > 0 {
-			refundValue = types.BigAdd(refundValue, types.BigDiv(types.BigMul(refundValue, types.NewInt(100)), types.NewInt(uint64(r.percentExtra))))
+			refundValue = types.BigAdd(refundValue, types.BigMul(types.BigDiv(refundValue, types.NewInt(100)), types.NewInt(uint64(r.percentExtra))))
 		}
 
 		log.Debugw("processing message", "method", messageMethod, "cid", msg.Cid, "from", m.From, "to", m.To, "value", m.Value, "gas_fee_cap", m.GasFeeCap, "gas_premium", m.GasPremium, "gas_used", recps[i].GasUsed, "refund", refundValue)

cmd/lotus-storage-miner/init.go

@@ -7,12 +7,13 @@ import (
 	"encoding/binary"
 	"encoding/json"
 	"fmt"
-	"github.com/filecoin-project/specs-actors/actors/abi/big"
 	"io/ioutil"
 	"os"
 	"path/filepath"
 	"strconv"
 
+	"github.com/filecoin-project/specs-actors/actors/abi/big"
+
 	"github.com/docker/go-units"
 	"github.com/google/uuid"
 	"github.com/ipfs/go-datastore"
@@ -107,6 +108,10 @@ var initCmd = &cli.Command{
 			Usage: "set gas premium for initialization messages in AttoFIL",
 			Value: "0",
 		},
+		&cli.StringFlag{
+			Name:  "from",
+			Usage: "select which address to send actor creation message from",
+		},
 	},
 	Action: func(cctx *cli.Context) error {
 		log.Info("Initializing lotus miner")
@@ -631,9 +636,18 @@ func createStorageMiner(ctx context.Context, api lapi.FullNode, peerid peer.ID,
 		return address.Undef, err
 	}
 
+	sender := owner
+	if fromstr := cctx.String("from"); fromstr != "" {
+		faddr, err := address.NewFromString(fromstr)
+		if err != nil {
+			return address.Undef, fmt.Errorf("could not parse from address: %w", err)
+		}
+		sender = faddr
+	}
+
 	createStorageMinerMsg := &types.Message{
 		To:    builtin.StoragePowerActorAddr,
-		From:  owner,
+		From:  sender,
 		Value: big.Zero(),
 
 		Method: builtin.MethodsPower.CreateMiner,

cmd/lotus-storage-miner/main.go

@@ -37,6 +37,7 @@ func main() {
 		lcli.WithCategory("chain", infoCmd),
 		lcli.WithCategory("market", storageDealsCmd),
 		lcli.WithCategory("market", retrievalDealsCmd),
+		lcli.WithCategory("market", dataTransfersCmd),
 		lcli.WithCategory("storage", sectorsCmd),
 		lcli.WithCategory("storage", provingCmd),
 		lcli.WithCategory("storage", storageCmd),

cmd/lotus-storage-miner/market.go

@@ -5,10 +5,12 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"sort"
 	"strconv"
 	"text/tabwriter"
 	"time"
 
+	tm "github.com/buger/goterm"
 	"github.com/docker/go-units"
 	"github.com/ipfs/go-cid"
 	"github.com/ipfs/go-cidutil/cidenc"
@@ -358,12 +360,16 @@ var dealsListCmd = &cli.Command{
 			return err
 		}
 
+		sort.Slice(deals, func(i, j int) bool {
+			return deals[i].CreationTime.Time().Before(deals[j].CreationTime.Time())
+		})
+
 		w := tabwriter.NewWriter(os.Stdout, 2, 4, 2, ' ', 0)
 
 		verbose := cctx.Bool("verbose")
 
 		if verbose {
-			_, _ = fmt.Fprintf(w, "ProposalCid\tDealId\tState\tClient\tSize\tPrice\tDuration\tMessage\n")
+			_, _ = fmt.Fprintf(w, "Creation\tProposalCid\tDealId\tState\tClient\tSize\tPrice\tDuration\tMessage\n")
 		} else {
 			_, _ = fmt.Fprintf(w, "ProposalCid\tDealId\tState\tClient\tSize\tPrice\tDuration\n")
 		}
@@ -376,6 +382,10 @@ var dealsListCmd = &cli.Command{
 
 			fil := types.FIL(types.BigMul(deal.Proposal.StoragePricePerEpoch, types.NewInt(uint64(deal.Proposal.Duration()))))
 
+			if verbose {
+				_, _ = fmt.Fprintf(w, "%s\t", deal.CreationTime.Time().Format(time.Stamp))
+			}
+
 			_, _ = fmt.Fprintf(w, "%s\t%d\t%s\t%s\t%s\t%s\t%s", propcid, deal.DealID, storagemarket.DealStates[deal.State], deal.Proposal.Client, units.BytesSize(float64(deal.Proposal.PieceSize)), fil, deal.Proposal.Duration())
 			if verbose {
 				_, _ = fmt.Fprintf(w, "\t%s", deal.Message)
@@ -506,3 +516,87 @@ var setSealDurationCmd = &cli.Command{
 		return nodeApi.SectorSetExpectedSealDuration(ctx, time.Duration(delay))
 	},
 }
+
+var dataTransfersCmd = &cli.Command{
+	Name:  "data-transfers",
+	Usage: "Manage data transfers",
+	Subcommands: []*cli.Command{
+		transfersListCmd,
+	},
+}
+
+var transfersListCmd = &cli.Command{
+	Name:  "list",
+	Usage: "List ongoing data transfers for this miner",
+	Flags: []cli.Flag{
+		&cli.BoolFlag{
+			Name:  "color",
+			Usage: "use color in display output",
+			Value: true,
+		},
+		&cli.BoolFlag{
+			Name:  "completed",
+			Usage: "show completed data transfers",
+		},
+		&cli.BoolFlag{
+			Name:  "watch",
+			Usage: "watch deal updates in real-time, rather than a one time list",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		if err != nil {
+			return err
+		}
+		defer closer()
+		ctx := lcli.ReqContext(cctx)
+
+		channels, err := api.MarketListDataTransfers(ctx)
+		if err != nil {
+			return err
+		}
+
+		completed := cctx.Bool("completed")
+		color := cctx.Bool("color")
+		watch := cctx.Bool("watch")
+
+		if watch {
+			channelUpdates, err := api.MarketDataTransferUpdates(ctx)
+			if err != nil {
+				return err
+			}
+
+			for {
+				tm.Clear() // Clear current screen
+
+				tm.MoveCursor(1, 1)
+
+				lcli.OutputDataTransferChannels(tm.Screen, channels, completed, color)
+
+				tm.Flush()
+
+				select {
+				case <-ctx.Done():
+					return nil
+				case channelUpdate := <-channelUpdates:
+					var found bool
+					for i, existing := range channels {
+						if existing.TransferID == channelUpdate.TransferID &&
+							existing.OtherPeer == channelUpdate.OtherPeer &&
+							existing.IsSender == channelUpdate.IsSender &&
+							existing.IsInitiator == channelUpdate.IsInitiator {
+							channels[i] = channelUpdate
+							found = true
+							break
+						}
+					}
+					if !found {
+						channels = append(channels, channelUpdate)
+					}
+				}
+			}
+		}
+		lcli.OutputDataTransferChannels(os.Stdout, channels, completed, color)
+		return nil
+	},
+}

documentation/en/.library.json

@@ -67,6 +67,12 @@
           "slug": "en+setup-troubleshooting",
           "github": "en/setup-troubleshooting.md",
           "value": null
+        },
+        {
+          "title": "Environment Variables",
+          "slug": "en+env-vars",
+          "github": "en/environment-vars.md",
+          "value": null
         }
       ]
     },

documentation/en/dev-tools-pond-ui.md

@@ -1,6 +1,6 @@
 # Pond UI
 
-Pond is a graphical testbed for [Lotus](https://docs.lotu.sh). Using it will setup a separate local network which is helpful for debugging. Pond will spin up nodes, connect them in a given topology, start them mining, and observe how they function over time.
+Pond is a graphical testbed for [Lotus](https://lotu.sh). Using it will setup a separate local network which is helpful for debugging. Pond will spin up nodes, connect them in a given topology, start them mining, and observe how they function over time.
 
 ## Build
 

documentation/en/dev/create-miner.md

@@ -37,7 +37,7 @@ It is important to note that the key store also holds the *private* key associat
 
 There is no explicit Filecoin message or CLI command that creates an account actor but rather it is implicit in the public-key address reflected by the wallet. The first time the VM sees it as the receiving address of a Filecoin message it will automatically create the associated account actor.
 
-We will trigger its creation then by sending funds to it, needed for the creation of the storage miner (note this document reflects the Filecoin Testnet, where we actually don't need funds to create a miner but will do the full procedure regardless to show the full cycle). In the Testnet we can send funds to an account trough the [Faucet](https://faucet.testnet.filecoin.io/funds.html), inserting the address returned by the `wallet new` command.
+We will trigger its creation then by sending funds to it, needed for the creation of the storage miner (note this document reflects the Filecoin Testnet, where we actually don't need funds to create a miner but will do the full procedure regardless to show the full cycle). In the Testnet we can send funds to an account through the [Faucet](http://spacerace.faucet.glif.io/), inserting the address returned by the `wallet new` command.
 
 FIXME: Is there an easy way to visualize the message generated by the Faucet?
 

documentation/en/environment-vars.md

@@ -0,0 +1,60 @@
+# Lotus Environment Variables
+
+## Building
+
+## Common
+
+The environment variables are common across most lotus binaries.
+
+### `LOTUS_FD_MAX`
+
+Sets the file descriptor limit for the process. This should be set high (8192
+or higher) if you ever notice 'too many open file descriptor' errors.
+
+### `LOTUS_JAEGER`
+
+This can be set to enable jaeger trace reporting. The value should be the url
+of the jaeger trace collector, the default for most jaeger setups should be
+`localhost:6831`.
+
+### `LOTUS_DEV`
+
+If set to a non-empty value, certain parts of the application will print more
+verbose information to aid in development of the software. Not recommended for
+end users.
+
+## Lotus Daemon
+
+### `LOTUS_PATH`
+
+Sets the location for the lotus daemon on-disk repo. If left empty, this defaults to `~/.lotus`.
+
+### `LOTUS_SKIP_GENESIS_CHECK`
+
+Can be set to `_yes_` if you wish to run a lotus network with a different
+genesis than the default one built into your lotus binary.
+
+### `LOTUS_CHAIN_TIPSET_CACHE`
+
+Sets the cache size for the chainstore tipset cache. The default value is 8192,
+but if your usage of the lotus API involves frequent arbitrary tipset lookups,
+you may want to increase this.
+
+### `LOTUS_CHAIN_INDEX_CACHE`
+
+Sets the cache size for the chainstore epoch index cache. The default value is 32768,
+but if your usage of the lotus API involves frequent deep chain lookups for
+block heights that are very far from the current chain height, you may want to
+increase this.
+
+
+
+## Lotus Miner
+
+A number of environment variables are respected for configuring the behavior of the filecoin proving subsystem. For more details on those [see here](https://github.com/filecoin-project/rust-fil-proofs/#settings).
+
+### `LOTUS_MINER_PATH`
+
+Sets the location for the lotus miners on-disk repo. If left empty, this defaults to `~/.lotusminer`.
+
+

documentation/en/faqs.md

@@ -30,7 +30,7 @@ to a Lotus Node over the JSON-RPC API.
 
 ### How do I set up a Lotus Node?
 
-Follow the instructions found [here](https://docs.lotu.sh/en+getting-started).
+Follow the instructions found [here](https://lotu.sh/en+getting-started).
 
 ### Where can I get the latest version of Lotus?
 
@@ -52,7 +52,7 @@ Stop the Lotus daemon, and delete all related files, including sealed and chain
 running `rm ~/.lotus ~/.lotusminer`.
 
 Then, install Lotus afresh by following the instructions 
-found [here](https://docs.lotu.sh/en+getting-started).
+found [here](https://lotu.sh/en+getting-started).
 
 ### Can I configure where the node's config and data goes?
 

documentation/en/getting-started.md

@@ -6,10 +6,10 @@ For more details about Filecoin, check out the [Filecoin Docs](https://docs.file
 
 ## What can I learn here?
 
-- How to install Lotus on [Arch Linux](https://docs.lotu.sh/en+install-lotus-arch), [Ubuntu](https://docs.lotu.sh/en+install-lotus-ubuntu), or [MacOS](https://docs.lotu.sh/en+install-lotus-macos).
-- Joining the [Lotus Testnet](https://docs.lotu.sh/en+join-testnet).
-- [Storing](https://docs.lotu.sh/en+storing-data) or [retrieving](https://docs.lotu.sh/en+retrieving-data) data.
-- Mining Filecoin using the **Lotus Miner** in your [CLI](https://docs.lotu.sh/en+mining).
+- How to install Lotus on [Arch Linux](https://lotu.sh/en+install-lotus-arch), [Ubuntu](https://lotu.sh/en+install-lotus-ubuntu), or [MacOS](https://lotu.sh/en+install-lotus-macos).
+- Joining the [Lotus Testnet](https://lotu.sh/en+join-testnet).
+- [Storing](https://lotu.sh/en+storing-data) or [retrieving](https://lotu.sh/en+retrieving-data) data.
+- Mining Filecoin using the **Lotus Miner** in your [CLI](https://lotu.sh/en+mining).
 
 ## How is Lotus designed?
 

documentation/en/install-lotus-arch.md

@@ -48,4 +48,4 @@ env env RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make clean d
 sudo make install
 ```
 
-After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus Testnet](https://docs.lotu.sh/en+join-testnet).
+After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus Testnet](https://lotu.sh/en+join-testnet).

documentation/en/install-lotus-fedora.md

@@ -51,4 +51,4 @@ env env RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make clean d
 sudo make install
 ```
 
-After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus TestNet](https://docs.lotu.sh/en+join-testnet).
+After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus TestNet](https://lotu.sh/en+join-testnet).

documentation/en/install-lotus-macos.md

@@ -59,4 +59,4 @@ make clean && make all
 sudo make install
 ```
 
-After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus Testnet](https://docs.lotu.sh/en+join-testnet).
+After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus Testnet](https://lotu.sh/en+join-testnet).

documentation/en/install-lotus-ubuntu.md

@@ -51,4 +51,4 @@ sudo make install
 ```
 
 
-After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus Testnet](https://docs.lotu.sh/en+join-testnet).
+After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus Testnet](https://lotu.sh/en+join-testnet).

documentation/en/join-testnet.md

@@ -26,9 +26,9 @@ In another terminal window, check your connection with peers:
 lotus net peers | wc -l
 ```
 
-In order to connect to the network, you need to be connected to at least 1 peer. If you’re seeing 0 peers, read our [troubleshooting notes](https://docs.lotu.sh/en+setup-troubleshooting).
+In order to connect to the network, you need to be connected to at least 1 peer. If you’re seeing 0 peers, read our [troubleshooting notes](https://lotu.sh/en+setup-troubleshooting).
 
-Make sure that you have a reasonable "open files limit" set on your machine, such as 10000. If you're seeing a lower value, such as 256 (default on macOS), read our [troubleshooting notes](https://docs.lotu.sh/en+setup-troubleshooting) on how to update it prior to starting the Lotus daemon.
+Make sure that you have a reasonable "open files limit" set on your machine, such as 10000. If you're seeing a lower value, such as 256 (default on macOS), read our [troubleshooting notes](https://lotu.sh/en+setup-troubleshooting) on how to update it prior to starting the Lotus daemon.
 
 ## Chain sync
 
@@ -57,7 +57,7 @@ Here is an example of the response:
 t1aswwvjsae63tcrniz6x5ykvsuotlgkvlulnqpsi
 ```
 
-- Visit the [faucet](https://faucet.testnet.filecoin.io) to add funds.
+- Visit the [faucet](http://spacerace.faucet.glif.io/) to add funds.
 - Paste the address you created.
 - Press the send button.
 

documentation/en/mining-lotus-worker.md

@@ -34,11 +34,11 @@ RemoteListenAddress = "127.0.0.1:2345"
 
 To make your node accessible over the local area network, you will need to determine your machines IP on the LAN, and change the `127.0.0.1` in the file to that address.
 
-A more permissive and less secure option is to change it to `0.0.0.0`. This will allow anyone who can connect to your computer on that port to access the [API](https://docs.lotu.sh/en+api). They will still need an auth token.
+A more permissive and less secure option is to change it to `0.0.0.0`. This will allow anyone who can connect to your computer on that port to access the [API](https://lotu.sh/en+api). They will still need an auth token.
 
 `RemoteListenAddress` must be set to an address which other nodes on your network will be able to reach.
 
-Next, you will need to [create an authentication token](https://docs.lotu.sh/en+api-scripting-support#generate-a-jwt-46). All Lotus APIs require authentication tokens to ensure your processes are as secure against attackers attempting to make unauthenticated requests to them.
+Next, you will need to [create an authentication token](https://lotu.sh/en+api-scripting-support#generate-a-jwt-46). All Lotus APIs require authentication tokens to ensure your processes are as secure against attackers attempting to make unauthenticated requests to them.
 
 ### Connect the Lotus Worker
 

documentation/en/mining-troubleshooting.md

@@ -25,7 +25,7 @@ lotus-miner info
 # WARN  main  lotus-storage-miner/main.go:73  failed to get api endpoint: (/Users/myrmidon/.lotusminer) %!w(*errors.errorString=&{API not running (no endpoint)}):
 ```
 
-If you see this, that means your **Lotus Miner** isn't ready yet. You need to finish [syncing the chain](https://docs.lotu.sh/en+join-testnet).
+If you see this, that means your **Lotus Miner** isn't ready yet. You need to finish [syncing the chain](https://lotu.sh/en+join-testnet).
 
 ## Error: Your computer may not be fast enough
 

documentation/en/mining.md

@@ -1,8 +1,8 @@
 # Storage Mining
 
-Here are instructions to learn how to perform storage mining. For hardware specifications please read [this](https://docs.lotu.sh/en+hardware-mining).
+Here are instructions to learn how to perform storage mining. For hardware specifications please read [this](https://lotu.sh/en+hardware-mining).
 
-It is useful to [join the Testnet](https://docs.lotu.sh/en+join-testnet) prior to attempting storage mining for the first time.
+It is useful to [join the Testnet](https://lotu.sh/en+join-testnet) prior to attempting storage mining for the first time.
 
 ## Note: Using the Lotus Miner from China
 
@@ -28,7 +28,7 @@ lotus wallet new bls
 
 With your wallet address:
 
-- Visit the [faucet](https://faucet.testnet.filecoin.io)
+- Visit the [faucet](http://spacerace.faucet.glif.io/)
 - Click "Create Miner"
 - DO NOT REFRESH THE PAGE. THIS OPERATION CAN TAKE SOME TIME.
 

documentation/en/mpool.md

@@ -20,6 +20,7 @@ The full node API defines the following methods for interacting with the mpool:
     MpoolSub(context.Context) (<-chan MpoolUpdate, error)
     MpoolGetConfig(context.Context) (*types.MpoolConfig, error)
     MpoolSetConfig(context.Context, *types.MpoolConfig) error
+    MpoolClear(context.Context, local bool) error
 ```
 
 ### MpoolPending
@@ -61,6 +62,13 @@ Returns (a copy of) the current mpool configuration.
 
 Sets the mpool configuration to (a copy of) the supplied configuration object.
 
+### MpoolClear
+
+Clears pending messages from the mpool; if `local` is `true` then local messages are also cleared and removed from the datastore.
+
+This should be used with extreme care and only in the case of errors during head changes that
+would leave the mpool in an inconsistent state.
+
 
 ## Command Line Interfae
 
@@ -75,6 +83,7 @@ lotus mpool stat [--local]
 lotus mpool replace [--gas-feecap <feecap>] [--gas-premium <premium>] [--gas-limit <limit>] [from] [nonce]
 lotus mpool find [--from <address>] [--to <address>] [--method <int>]
 lotus mpool config [<configuration>]
+lotus mpool clear [--local]
 ```
 
 ### lotus mpool pending
@@ -98,6 +107,12 @@ Searches for messages in the mpool.
 ### lotus mpool config
 Gets or sets the current mpool configuration.
 
+### lotus mpool clear
+Unconditionally clears pending messages from the mpool.
+If the `--local` flag is passed, then local messages are also cleared; otherwise local messages are retained.
+
+*Warning*: this command should only be used in the case of head change errors leaving the mpool in an  state.
+
 ## Configuration
 
 The mpool a few parameters that can be configured by the user, either through the API

documentation/en/retrieving-data.md

@@ -4,7 +4,7 @@
 
 Here are the operations you can perform after you have stored and sealed a **Data CID** with the **Lotus Miner** in the network.
 
-If you would like to learn how to store a **Data CID** on a miner, read the instructions [here](https://docs.lotu.sh/en+storing-data).
+If you would like to learn how to store a **Data CID** on a miner, read the instructions [here](https://lotu.sh/en+storing-data).
 
 ## Find by Data CID
 

documentation/en/storing-data-troubleshooting.md

@@ -24,4 +24,4 @@ In order to retrieve a file, it must be sealed. Miners can check sealing progres
 lotus-miner sectors list
 ```
 
-When sealing is complete, `pSet: NO` will become `pSet: YES`. From now on the **Data CID** is [retrievable](https://docs.lotu.sh/en+retrieving-data) from the **Lotus Miner**.
+When sealing is complete, `pSet: NO` will become `pSet: YES`. From now on the **Data CID** is [retrievable](https://lotu.sh/en+retrieving-data) from the **Lotus Miner**.

extern/storage-sealing/checks.go

@@ -79,7 +79,7 @@ func checkPrecommit(ctx context.Context, maddr address.Address, si SectorInfo, t
 		return &ErrApi{xerrors.Errorf("calling StateComputeDataCommitment: %w", err)}
 	}
 
-	if !commD.Equals(*si.CommD) {
+	if si.CommD == nil || !commD.Equals(*si.CommD) {
 		return &ErrBadCommD{xerrors.Errorf("on chain CommD differs from sector: %s != %s", commD, si.CommD)}
 	}
 

extern/storage-sealing/states_sealing.go

@@ -308,6 +308,10 @@ func (m *Sealing) handleCommitting(ctx statemachine.Context, sector SectorInfo)
 
 	log.Infof("KOMIT %d %x(%d); %x(%d); %v; r:%x; d:%x", sector.SectorNumber, sector.TicketValue, sector.TicketEpoch, sector.SeedValue, sector.SeedEpoch, sector.pieceInfos(), sector.CommR, sector.CommD)
 
+	if sector.CommD == nil || sector.CommR == nil {
+		return ctx.Send(SectorCommitFailed{xerrors.Errorf("sector had nil commR or commD")})
+	}
+
 	cids := storage.SectorCids{
 		Unsealed: *sector.CommD,
 		Sealed:   *sector.CommR,

extern/storage-sealing/upgrade_queue.go

@@ -44,6 +44,9 @@ func (m *Sealing) MarkForUpgrade(id abi.SectorNumber) error {
 }
 
 func (m *Sealing) tryUpgradeSector(ctx context.Context, params *miner.SectorPreCommitInfo) big.Int {
+	if len(params.DealIDs) == 0 {
+		return big.Zero()
+	}
 	replace := m.maybeUpgradableSector()
 	if replace != nil {
 		loc, err := m.api.StateSectorPartition(ctx, m.maddr, *replace, nil)

go.mod

@@ -28,9 +28,9 @@ require (
 	github.com/filecoin-project/go-bitfield v0.2.0
 	github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2
 	github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03
-	github.com/filecoin-project/go-data-transfer v0.6.1
+	github.com/filecoin-project/go-data-transfer v0.6.2
 	github.com/filecoin-project/go-fil-commcid v0.0.0-20200716160307-8f644712406f
-	github.com/filecoin-project/go-fil-markets v0.5.6
+	github.com/filecoin-project/go-fil-markets v0.5.7
 	github.com/filecoin-project/go-jsonrpc v0.1.2-0.20200817153016-2ea5cbaf5ec0
 	github.com/filecoin-project/go-multistore v0.0.3
 	github.com/filecoin-project/go-padreader v0.0.0-20200210211231-548257017ca6
@@ -94,7 +94,7 @@ require (
 	github.com/libp2p/go-libp2p-noise v0.1.1
 	github.com/libp2p/go-libp2p-peer v0.2.0
 	github.com/libp2p/go-libp2p-peerstore v0.2.6
-	github.com/libp2p/go-libp2p-pubsub v0.3.4
+	github.com/libp2p/go-libp2p-pubsub v0.3.5-0.20200821075113-efd56962bced
 	github.com/libp2p/go-libp2p-quic-transport v0.7.1
 	github.com/libp2p/go-libp2p-record v0.1.3
 	github.com/libp2p/go-libp2p-routing-helpers v0.2.3
@@ -119,7 +119,7 @@ require (
 	github.com/syndtr/goleveldb v1.0.0
 	github.com/urfave/cli/v2 v2.2.0
 	github.com/whyrusleeping/bencher v0.0.0-20190829221104-bb6607aa8bba
-	github.com/whyrusleeping/cbor-gen v0.0.0-20200812213548-958ddffe352c
+	github.com/whyrusleeping/cbor-gen v0.0.0-20200814224545-656e08ce49ee
 	github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7
 	github.com/whyrusleeping/pubsub v0.0.0-20131020042734-02de8aa2db3d
 	github.com/xorcare/golden v0.6.1-0.20191112154924-b87f686d7542

go.sum

@@ -240,13 +240,13 @@ github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2 h1:a
 github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2/go.mod h1:pqTiPHobNkOVM5thSRsHYjyQfq7O5QSCMhvuu9JoDlg=
 github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03 h1:2pMXdBnCiXjfCYx/hLqFxccPoqsSveQFxVLvNxy9bus=
 github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
-github.com/filecoin-project/go-data-transfer v0.6.1 h1:EA6X8fSiBRNVVwKm5pA7+njZnBbdqpRtedZagrrwHHI=
-github.com/filecoin-project/go-data-transfer v0.6.1/go.mod h1:uRYBRKVBVM12CSusBtVrzDHkVw/3DKZpkxKJVP1Ydas=
+github.com/filecoin-project/go-data-transfer v0.6.2 h1:IgbkwcHoyWGglzfsY7P9L1GapzoiLNKuzfZY2bxER8E=
+github.com/filecoin-project/go-data-transfer v0.6.2/go.mod h1:uRYBRKVBVM12CSusBtVrzDHkVw/3DKZpkxKJVP1Ydas=
 github.com/filecoin-project/go-fil-commcid v0.0.0-20200208005934-2b8bd03caca5/go.mod h1:JbkIgFF/Z9BDlvrJO1FuKkaWsH673/UdFaiVS6uIHlA=
 github.com/filecoin-project/go-fil-commcid v0.0.0-20200716160307-8f644712406f h1:GxJzR3oRIMTPtpZ0b7QF8FKPK6/iPAc7trhlL5k/g+s=
 github.com/filecoin-project/go-fil-commcid v0.0.0-20200716160307-8f644712406f/go.mod h1:Eaox7Hvus1JgPrL5+M3+h7aSPHc0cVqpSxA+TxIEpZQ=
-github.com/filecoin-project/go-fil-markets v0.5.6 h1:WmBbV0qBU4NvLJ64xROpzrKUbkZxZqszZiEiCGmCEIY=
-github.com/filecoin-project/go-fil-markets v0.5.6/go.mod h1:SJApXAKr5jyGpbzDEOhvemui0pih7hhT8r2MXJxCP1E=
+github.com/filecoin-project/go-fil-markets v0.5.7 h1:kzyMHqez8ssxchj5s9M1hkC3CTwRGh2MeglJGfUksQU=
+github.com/filecoin-project/go-fil-markets v0.5.7/go.mod h1:KnvFG3kSQ77vKYSY/QdrXET81wVCBByHXjG7AyxnbUw=
 github.com/filecoin-project/go-jsonrpc v0.1.2-0.20200817153016-2ea5cbaf5ec0 h1:/GT3V+3f+H5w5odb7LcCWJ1zPw8H8m9TsGQcU0cGSHo=
 github.com/filecoin-project/go-jsonrpc v0.1.2-0.20200817153016-2ea5cbaf5ec0/go.mod h1:XBBpuKIMaXIIzeqzO1iucq4GvbF8CxmXRFoezRh+Cx4=
 github.com/filecoin-project/go-multistore v0.0.3 h1:vaRBY4YiA2UZFPK57RNuewypB8u0DzzQwqsL0XarpnI=
@@ -885,8 +885,8 @@ github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1
 github.com/libp2p/go-libp2p-protocol v0.1.0/go.mod h1:KQPHpAabB57XQxGrXCNvbL6UEXfQqUgC/1adR2Xtflk=
 github.com/libp2p/go-libp2p-pubsub v0.1.1/go.mod h1:ZwlKzRSe1eGvSIdU5bD7+8RZN/Uzw0t1Bp9R1znpR/Q=
 github.com/libp2p/go-libp2p-pubsub v0.3.2-0.20200527132641-c0712c6e92cf/go.mod h1:TxPOBuo1FPdsTjFnv+FGZbNbWYsp74Culx+4ViQpato=
-github.com/libp2p/go-libp2p-pubsub v0.3.4 h1:8PollxXtUvzy0DMn5XFMg/JihjaKboWyk3ML6yRW1Lk=
-github.com/libp2p/go-libp2p-pubsub v0.3.4/go.mod h1:DTMSVmZZfXodB/pvdTGrY2eHPZ9W2ev7hzTH83OKHrI=
+github.com/libp2p/go-libp2p-pubsub v0.3.5-0.20200821075113-efd56962bced h1:mgcPX7FRZNpIhg2i8B91T4fO0UEpeuhRS0YX5uMAic4=
+github.com/libp2p/go-libp2p-pubsub v0.3.5-0.20200821075113-efd56962bced/go.mod h1:DTMSVmZZfXodB/pvdTGrY2eHPZ9W2ev7hzTH83OKHrI=
 github.com/libp2p/go-libp2p-quic-transport v0.1.1/go.mod h1:wqG/jzhF3Pu2NrhJEvE+IE0NTHNXslOPn9JQzyCAxzU=
 github.com/libp2p/go-libp2p-quic-transport v0.5.0/go.mod h1:IEcuC5MLxvZ5KuHKjRu+dr3LjCT1Be3rcD/4d8JrX8M=
 github.com/libp2p/go-libp2p-quic-transport v0.7.1 h1:X6Ond9GANspXpgwJlSR9yxcMMD6SLBnGKRtwjBG5awc=
@@ -1403,6 +1403,8 @@ github.com/whyrusleeping/cbor-gen v0.0.0-20200810223238-211df3b9e24c h1:BMg3YUwL
 github.com/whyrusleeping/cbor-gen v0.0.0-20200810223238-211df3b9e24c/go.mod h1:fgkXqYy7bV2cFeIEOkVTZS/WjXARfBqSH6Q2qHL33hQ=
 github.com/whyrusleeping/cbor-gen v0.0.0-20200812213548-958ddffe352c h1:otRnI08JoahNBxUFqX3372Ab9GnTj8L5J9iP5ImyxGU=
 github.com/whyrusleeping/cbor-gen v0.0.0-20200812213548-958ddffe352c/go.mod h1:fgkXqYy7bV2cFeIEOkVTZS/WjXARfBqSH6Q2qHL33hQ=
+github.com/whyrusleeping/cbor-gen v0.0.0-20200814224545-656e08ce49ee h1:U7zWWvvAjT76EiuWPSOiZlQDnaQYPxPoxugTtTAcJK0=
+github.com/whyrusleeping/cbor-gen v0.0.0-20200814224545-656e08ce49ee/go.mod h1:fgkXqYy7bV2cFeIEOkVTZS/WjXARfBqSH6Q2qHL33hQ=
 github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f h1:jQa4QT2UP9WYv2nzyawpKMOCl+Z/jW7djv2/J50lj9E=
 github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f/go.mod h1:p9UJB6dDgdPgMJZs7UjUOdulKyRr9fqkS+6JKAInPy8=
 github.com/whyrusleeping/go-ctrlnet v0.0.0-20180313164037-f564fbbdaa95/go.mod h1:SJqKCCPXRfBFCwXjfNT/skfsceF7+MBFLI2OrvuRA7g=

lib/ulimit/ulimit.go

@@ -28,14 +28,19 @@ const minFds = 2048
 // default max file descriptor limit.
 const maxFds = 16 << 10
 
-// userMaxFDs returns the value of IPFS_FD_MAX
+// userMaxFDs returns the value of LOTUS_FD_MAX
 func userMaxFDs() uint64 {
-	// check if the IPFS_FD_MAX is set up and if it does
+	// check if the LOTUS_FD_MAX is set up and if it does
 	// not have a valid fds number notify the user
-	if val := os.Getenv("IPFS_FD_MAX"); val != "" {
+	val := os.Getenv("LOTUS_FD_MAX")
+	if val == "" {
+		val = os.Getenv("IPFS_FD_MAX")
+	}
+
+	if val != "" {
 		fds, err := strconv.ParseUint(val, 10, 64)
 		if err != nil {
-			log.Errorf("bad value for IPFS_FD_MAX: %s", err)
+			log.Errorf("bad value for LOTUS_FD_MAX: %s", err)
 			return 0
 		}
 		return fds
@@ -44,7 +49,7 @@ func userMaxFDs() uint64 {
 }
 
 // ManageFdLimit raise the current max file descriptor count
-// of the process based on the IPFS_FD_MAX value
+// of the process based on the LOTUS_FD_MAX value
 func ManageFdLimit() (changed bool, newLimit uint64, err error) {
 	if !supportsFDManagement {
 		return false, 0, nil
@@ -93,7 +98,7 @@ func ManageFdLimit() (changed bool, newLimit uint64, err error) {
 
 		if newLimit < userLimit {
 			err = fmt.Errorf(
-				"failed to raise ulimit to IPFS_FD_MAX (%d): set to %d",
+				"failed to raise ulimit to LOTUS_FD_MAX (%d): set to %d",
 				userLimit,
 				newLimit,
 			)

lib/ulimit/ulimit_test.go

@@ -44,7 +44,7 @@ func TestManageInvalidNFds(t *testing.T) {
 		t.Errorf("ManageFdLimit should return an error: changed %t, new: %d", changed, new)
 	} else if err != nil {
 		flag := strings.Contains(err.Error(),
-			"failed to raise ulimit to IPFS_FD_MAX")
+			"failed to raise ulimit to LOTUS_FD_MAX")
 		if !flag {
 			t.Error("ManageFdLimit returned unexpected error", err)
 		}

metrics/metrics.go

@@ -44,6 +44,9 @@ var (
 	PubsubDeliverMessage                = stats.Int64("pubsub/delivered", "Counter for total delivered messages", stats.UnitDimensionless)
 	PubsubRejectMessage                 = stats.Int64("pubsub/rejected", "Counter for total rejected messages", stats.UnitDimensionless)
 	PubsubDuplicateMessage              = stats.Int64("pubsub/duplicate", "Counter for total duplicate messages", stats.UnitDimensionless)
+	PubsubRecvRPC                       = stats.Int64("pubsub/recv_rpc", "Counter for total received RPCs", stats.UnitDimensionless)
+	PubsubSendRPC                       = stats.Int64("pubsub/send_rpc", "Counter for total sent RPCs", stats.UnitDimensionless)
+	PubsubDropRPC                       = stats.Int64("pubsub/drop_rpc", "Counter for total dropped RPCs", stats.UnitDimensionless)
 )
 
 var (

node/builder.go

@@ -174,7 +174,7 @@ func libp2p() Option {
 		Override(AddrsFactoryKey, lp2p.AddrsFactory(nil, nil)),
 		Override(SmuxTransportKey, lp2p.SmuxTransport(true)),
 		Override(RelayKey, lp2p.NoRelay()),
-		Override(SecurityKey, lp2p.Security(true, true)),
+		Override(SecurityKey, lp2p.Security(true, false)),
 
 		Override(BaseRoutingKey, lp2p.BaseRouting),
 		Override(new(routing.Routing), lp2p.Routing),

node/impl/client/client.go

@@ -2,7 +2,6 @@ package client
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
 	"io"
 	"os"
@@ -197,6 +196,7 @@ func (a *API) ClientListDeals(ctx context.Context) ([]api.DealInfo, error) {
 			PricePerEpoch: v.Proposal.StoragePricePerEpoch,
 			Duration:      uint64(v.Proposal.Duration()),
 			DealID:        v.DealID,
+			CreationTime:  v.CreationTime.Time(),
 		}
 	}
 
@@ -219,6 +219,7 @@ func (a *API) ClientGetDealInfo(ctx context.Context, d cid.Cid) (*api.DealInfo,
 		PricePerEpoch: v.Proposal.StoragePricePerEpoch,
 		Duration:      uint64(v.Proposal.Duration()),
 		DealID:        v.DealID,
+		CreationTime:  v.CreationTime.Time(),
 	}, nil
 }
 
@@ -773,7 +774,7 @@ func (a *API) ClientListDataTransfers(ctx context.Context) ([]api.DataTransferCh
 
 	apiChannels := make([]api.DataTransferChannel, 0, len(inProgressChannels))
 	for _, channelState := range inProgressChannels {
-		apiChannels = append(apiChannels, toAPIChannel(a.Host.ID(), channelState))
+		apiChannels = append(apiChannels, api.NewDataTransferChannel(a.Host.ID(), channelState))
 	}
 
 	return apiChannels, nil
@@ -783,7 +784,7 @@ func (a *API) ClientDataTransferUpdates(ctx context.Context) (<-chan api.DataTra
 	channels := make(chan api.DataTransferChannel)
 
 	unsub := a.DataTransfer.SubscribeToEvents(func(evt datatransfer.Event, channelState datatransfer.ChannelState) {
-		channel := toAPIChannel(a.Host.ID(), channelState)
+		channel := api.NewDataTransferChannel(a.Host.ID(), channelState)
 		select {
 		case <-ctx.Done():
 		case channels <- channel:
@@ -797,34 +798,3 @@ func (a *API) ClientDataTransferUpdates(ctx context.Context) (<-chan api.DataTra
 
 	return channels, nil
 }
-
-func toAPIChannel(hostID peer.ID, channelState datatransfer.ChannelState) api.DataTransferChannel {
-	channel := api.DataTransferChannel{
-		TransferID: channelState.TransferID(),
-		Status:     channelState.Status(),
-		BaseCID:    channelState.BaseCID(),
-		IsSender:   channelState.Sender() == hostID,
-		Message:    channelState.Message(),
-	}
-	stringer, ok := channelState.Voucher().(fmt.Stringer)
-	if ok {
-		channel.Voucher = stringer.String()
-	} else {
-		voucherJSON, err := json.Marshal(channelState.Voucher())
-		if err != nil {
-			channel.Voucher = fmt.Errorf("Voucher Serialization: %w", err).Error()
-		} else {
-			channel.Voucher = string(voucherJSON)
-		}
-	}
-	if channel.IsSender {
-		channel.IsInitiator = !channelState.IsPull()
-		channel.Transferred = channelState.Sent()
-		channel.OtherPeer = channelState.Recipient()
-	} else {
-		channel.IsInitiator = channelState.IsPull()
-		channel.Transferred = channelState.Received()
-		channel.OtherPeer = channelState.Sender()
-	}
-	return channel
-}

node/impl/full/mpool.go

@@ -105,6 +105,11 @@ func (a *MpoolAPI) MpoolPending(ctx context.Context, tsk types.TipSetKey) ([]*ty
 	}
 }
 
+func (a *MpoolAPI) MpoolClear(ctx context.Context, local bool) error {
+	a.Mpool.Clear(local)
+	return nil
+}
+
 func (a *MpoolAPI) MpoolPush(ctx context.Context, smsg *types.SignedMessage) (cid.Cid, error) {
 	return a.Mpool.Push(smsg)
 }

node/impl/storminer.go

@@ -8,8 +8,10 @@ import (
 	"strconv"
 	"time"
 
+	datatransfer "github.com/filecoin-project/go-data-transfer"
 	"github.com/filecoin-project/specs-actors/actors/abi/big"
 	"github.com/ipfs/go-cid"
+	"github.com/libp2p/go-libp2p-core/host"
 	"golang.org/x/xerrors"
 
 	"github.com/filecoin-project/go-address"
@@ -51,6 +53,8 @@ type StorageMinerAPI struct {
 	StorageMgr        *sectorstorage.Manager `optional:"true"`
 	IStorageMgr       sectorstorage.SectorManager
 	*stores.Index
+	DataTransfer dtypes.ProviderDataTransfer
+	Host         host.Host
 
 	ConsiderOnlineStorageDealsConfigFunc       dtypes.ConsiderOnlineStorageDealsConfigFunc
 	SetConsiderOnlineStorageDealsConfigFunc    dtypes.SetConsiderOnlineStorageDealsConfigFunc
@@ -354,6 +358,39 @@ func (sm *StorageMinerAPI) MarketGetRetrievalAsk(ctx context.Context) (*retrieva
 	return sm.RetrievalProvider.GetAsk(), nil
 }
 
+func (sm *StorageMinerAPI) MarketListDataTransfers(ctx context.Context) ([]api.DataTransferChannel, error) {
+	inProgressChannels, err := sm.DataTransfer.InProgressChannels(ctx)
+	if err != nil {
+		return nil, err
+	}
+
+	apiChannels := make([]api.DataTransferChannel, 0, len(inProgressChannels))
+	for _, channelState := range inProgressChannels {
+		apiChannels = append(apiChannels, api.NewDataTransferChannel(sm.Host.ID(), channelState))
+	}
+
+	return apiChannels, nil
+}
+
+func (sm *StorageMinerAPI) MarketDataTransferUpdates(ctx context.Context) (<-chan api.DataTransferChannel, error) {
+	channels := make(chan api.DataTransferChannel)
+
+	unsub := sm.DataTransfer.SubscribeToEvents(func(evt datatransfer.Event, channelState datatransfer.ChannelState) {
+		channel := api.NewDataTransferChannel(sm.Host.ID(), channelState)
+		select {
+		case <-ctx.Done():
+		case channels <- channel:
+		}
+	})
+
+	go func() {
+		defer unsub()
+		<-ctx.Done()
+	}()
+
+	return channels, nil
+}
+
 func (sm *StorageMinerAPI) DealsList(ctx context.Context) ([]storagemarket.StorageDeal, error) {
 	return sm.StorageProvider.ListDeals(ctx)
 }

node/modules/lp2p/pubsub.go

@@ -31,6 +31,8 @@ func init() {
 	pubsub.GossipSubDhi = 12
 	pubsub.GossipSubDlazy = 12
 	pubsub.GossipSubDirectConnectInitialDelay = 30 * time.Second
+	pubsub.GossipSubIWantFollowupTime = 5 * time.Second
+	pubsub.GossipSubHistoryLength = 10
 }
 func ScoreKeeper() *dtypes.ScoreKeeper {
 	return new(dtypes.ScoreKeeper)
@@ -110,8 +112,9 @@ func GossipSub(in GossipIn) (service *pubsub.PubSub, err error) {
 				// IPColocationFactorWhitelist: map[string]struct{}{},
 
 				// P7: behavioural penalties, decay after 1hr
-				BehaviourPenaltyWeight: -10,
-				BehaviourPenaltyDecay:  pubsub.ScoreParameterDecay(time.Hour),
+				BehaviourPenaltyThreshold: 6,
+				BehaviourPenaltyWeight:    -10,
+				BehaviourPenaltyDecay:     pubsub.ScoreParameterDecay(time.Hour),
 
 				DecayInterval: pubsub.DefaultDecayInterval,
 				DecayToZero:   pubsub.DefaultDecayToZero,
@@ -371,5 +374,11 @@ func (trw *tracerWrapper) Trace(evt *pubsub_pb.TraceEvent) {
 		if trw.tr != nil {
 			trw.tr.Trace(evt)
 		}
+	case pubsub_pb.TraceEvent_RECV_RPC:
+		stats.Record(context.TODO(), metrics.PubsubRecvRPC.M(1))
+	case pubsub_pb.TraceEvent_SEND_RPC:
+		stats.Record(context.TODO(), metrics.PubsubSendRPC.M(1))
+	case pubsub_pb.TraceEvent_DROP_RPC:
+		stats.Record(context.TODO(), metrics.PubsubDropRPC.M(1))
 	}
 }