Merge tag 'tags/v1.26.2' into 1.26.2

v1.26.2
Git-EVTag-v0-SHA512: f94cc711cb948b08eea54dedd0ddedf9b3443f4b1ea4e606154264b39f8a47f4949283c9a76c1564ce59e3aeab3e3232bc4a9491138035597159cb322f85e2f4

CHANGELOG.md

@@ -6,8 +6,43 @@
 
 ## Improvements
 
+# v1.26.2 / 2024-04-08
+
+**This is a mandatory patch release for the Filecoin network version 22 mainnet upgrade, for all node operators.**
+
+There is an update in the upgrade epoch for nv22, you can read the [full discussion in Slack here.](https://filecoinproject.slack.com/archives/C05P37R9KQD/p1712548103521969)
+
+The new upgrade epoch is scheduled to be on **epoch `3855360 - 2024-04-24 - 14:00:00Z`**. That means:
+
+- **All mainnet node operators that have upgraded to v1.26.x, must upgrade to this patch release before 2024-04-11T14:00:00Z.**
+- **All mainnet node operators that are on a version lower the v1.26.x, must upgrade to this patch release before 2024-04-24T14:00:00Z.**
+
+This patch also includes fixes for node operators who want to index builtin-actor events after the nv22 upgrade. Specifically, it ensures the builtin actor event entries are ordered by insertion order when selected ([#11834](https://github.com/filecoin-project/lotus/pull/11834)). It also includes a couple Lotus-Miner patch fixes, ensuring that SnapDeals works properly and are using the new ProveReplicaUpdate3 message after the network version 22 upgrade, ensuring that DDO-sectors has the correct sector expirations, as well as DDO-sector visibility in the `lotus-miner sectors list` cmd.
+
+## Upgrade Warnings
+
+For users currently on a version of Lotus lower than v1.26.0, please note that **this release requires a minimum Go version of v1.21.7 or higher to successfully build Lotus.**
+
+## v1.26.x Inclusions
+
+See the [v1.26.0](#v1260--2024-03-21) release notes below for inclusions and notes on the v1.26.x series.
+
+* [v13 Builtin Actor Bundle](#v13-builtin-actor-bundle)
+* [Migration](#migration)
+* [New features](#new-features-1)
+  * [Tracing API](#tracing-api)
+  * [Ethereum Tracing API (`trace_block` and `trace_replayBlockTransactions`)](#ethereum-tracing-api-trace_block-and-trace_replayblocktransactions)
+  * [GetActorEventsRaw and SubscribeActorEventsRaw](#getactoreventsraw-and-subscribeactoreventsraw)
+  * [Events Configuration Changes](#events-configuration-changes)
+  * [GetAllClaims and GetAllAlocations](#getallclaims-and-getallalocations)
+  * [Lotus CLI](#lotus-cli)
+
+#v1260--2024-03-21
+
 # v1.26.1 / 2024-03-27
 
+***RETRACTED: Due to a change in network version 22 upgrade epoch, Lotus v1.26.1 should not be used prior to the new upgrade epoch. See v1.26.2 release notes above.***
+
 **This is a patch release for the Calibration network user.** The Calibration network is scheduled for an upgrade to include the two additional built-in actor events to ease the transition and observability of DDO for the ecosystem ([#964](https://github.com/filecoin-project/FIPs/pull/964) and [#968](https://github.com/filecoin-project/FIPs/pull/968)).
 
 The agreed-upon epoch between the Filecoin implementer team for the update is `1493854`, corresponding to `2024-04-03T11:00:00Z`. All Calibration network users need to upgrade to this patch release before that.
@@ -16,6 +51,8 @@ The agreed-upon epoch between the Filecoin implementer team for the update is `1
 
 # v1.26.0 / 2024-03-21
 
+***RETRACTED: Due to a change in network version 22 upgrade epoch, Lotus v1.26.0 should not be used prior to the new upgrade epoch. See v1.26.2 release notes above.***
+
 This is the stable release for the upcoming MANDATORY Filecoin network upgrade v22, codenamed Dragon 🐉, at `epoch 3817920 - 2024-04-11 - 14:00:00Z`
 
 The Filecoin network version 22 delivers the following FIPs:
@@ -76,7 +113,7 @@ For certain node operators, such as full archival nodes or systems that need to
 - feat: implement FIP-0063 ([filecoin-project/lotus#11572](https://github.com/filecoin-project/lotus/pull/11572))
 - feat: events: Add Lotus APIs to consume smart contract and built-in actor events ([filecoin-project/lotus#11618](https://github.com/filecoin-project/lotus/pull/11618))
 
-## Tracing API
+### Tracing API
 
 Replace the `CodeCid` field in the message trace (added in 1.23.4) with an `InvokedActor` field.
 
@@ -411,8 +448,6 @@ This patch release allows for up to 10k messages per block. Additionally, it int
 ## Improvements
 - fix: exchange: allow up to 10k messages per block ([filecoin-project/lotus#11506](https://github.com/filecoin-project/lotus/pull/11506))
 
->>>>>>> releases
-
 # v 1.25.0 / 2023-11-22
 
 This is a highly recommended feature release of Lotus. This optional release supports the Filecoin network version 21 upgrade, codenamed Watermelon 🍉, in addition to the numerous improvements and enhancements for node operators, ETH RPC-providers and storage providers.
@@ -500,6 +535,7 @@ Lotus-workers can now be built to leverage the SupraSeal C2 sealing optimization
 - fix: lotus-provider: lotus-provider msg sending ([filecoin-project/lotus#11480](https://github.com/filecoin-project/lotus/pull/11480))
 - fix: lotus-provider: Fix winning PoSt ([filecoin-project/lotus#11483](https://github.com/filecoin-project/lotus/pull/11483))
 - chore: fix: sql Scan cannot write to an object ([filecoin-project/lotus#11487](https://github.com/filecoin-project/lotus/pull/11487))
+- fix: Exclude reverted events in `eth_getLogs` results [filecoin-project/lotus#11318](https://github.com/filecoin-project/lotus/pull/11318)
 
 ## Dependencies
 - deps: update go-libp2p to v0.28.1 ([filecoin-project/lotus#10998](https://github.com/filecoin-project/lotus/pull/10998))

build/params_mainnet.go

@@ -99,8 +99,8 @@ const UpgradeThunderHeight = UpgradeLightningHeight + 2880*21
 // 2023-12-12T13:30:00Z
 const UpgradeWatermelonHeight = 3469380
 
-// 2024-04-11T14:00:00Z
-var UpgradeDragonHeight = abi.ChainEpoch(3817920)
+// 2024-04-24T14:00:00Z
+var UpgradeDragonHeight = abi.ChainEpoch(3855360)
 
 // This epoch, 120 epochs after the "rest" of the nv22 upgrade, is when we switch to Drand quicknet
 // 2024-04-11T15:00:00Z
@@ -132,6 +132,7 @@ func init() {
 
 	if os.Getenv("LOTUS_DISABLE_DRAGON") == "1" {
 		UpgradeDragonHeight = math.MaxInt64 - 1
+		delete(DrandSchedule, UpgradePhoenixHeight)
 		UpgradePhoenixHeight = math.MaxInt64
 	}
 

build/version.go

@@ -37,7 +37,7 @@ func BuildTypeString() string {
 }
 
 // BuildVersion is the local build version
-const BuildVersion = "1.26.1"
+const BuildVersion = "1.26.2"
 
 func UserVersion() string {
 	if os.Getenv("LOTUS_VERSION_IGNORE_COMMIT") == "1" {

chain/events/filter/index.go

@@ -46,6 +46,7 @@ var ddls = []string{
 	)`,
 
 	`CREATE INDEX IF NOT EXISTS height_tipset_key ON event (height,tipset_key)`,
+	`CREATE INDEX IF NOT EXISTS event_emitter_addr ON event (emitter_addr)`,
 
 	`CREATE TABLE IF NOT EXISTS event_entry (
 		event_id INTEGER,
@@ -56,6 +57,8 @@ var ddls = []string{
 		value BLOB NOT NULL
 	)`,
 
+	`CREATE INDEX IF NOT EXISTS event_entry_key_index ON event_entry (key)`,
+
 	// metadata containing version of schema
 	`CREATE TABLE IF NOT EXISTS _meta (
     	version UINT64 NOT NULL UNIQUE
@@ -63,6 +66,7 @@ var ddls = []string{
 
 	`INSERT OR IGNORE INTO _meta (version) VALUES (1)`,
 	`INSERT OR IGNORE INTO _meta (version) VALUES (2)`,
+	`INSERT OR IGNORE INTO _meta (version) VALUES (3)`,
 }
 
 var (
@@ -70,7 +74,7 @@ var (
 )
 
 const (
-	schemaVersion = 2
+	schemaVersion = 3
 
 	eventExists          = `SELECT MAX(id) FROM event WHERE height=? AND tipset_key=? AND tipset_key_cid=? AND emitter_addr=? AND event_index=? AND message_cid=? AND message_index=?`
 	insertEvent          = `INSERT OR IGNORE INTO event(height, tipset_key, tipset_key_cid, emitter_addr, event_index, message_cid, message_index, reverted) VALUES(?, ?, ?, ?, ?, ?, ?, ?)`
@@ -321,6 +325,22 @@ func NewEventIndex(ctx context.Context, path string, chainStore *store.ChainStor
 			version = 2
 		}
 
+		if version == 2 {
+			log.Infof("upgrading event index from version 2 to version 3")
+
+			// to upgrade to version 3 we only need to create an index on the event_entry.key column
+			// and on the event.emitter_addr column
+			// which means we can just reapply the schema (it will not have any effect on existing data)
+			for _, ddl := range ddls {
+				if _, err := db.Exec(ddl); err != nil {
+					_ = db.Close()
+					return nil, xerrors.Errorf("could not upgrade index to version 3, exec ddl %q: %w", ddl, err)
+				}
+			}
+
+			version = 3
+		}
+
 		if version != schemaVersion {
 			_ = db.Close()
 			return nil, xerrors.Errorf("invalid database version: got %d, expected %d", version, schemaVersion)
@@ -557,7 +577,8 @@ func (ei *EventIndex) prefillFilter(ctx context.Context, f *eventFilter, exclude
 		s = s + " WHERE " + strings.Join(clauses, " AND ")
 	}
 
-	s += " ORDER BY event.height DESC"
+	// retain insertion order of event_entry rows with the implicit _rowid_ column
+	s += " ORDER BY event.height DESC, event_entry._rowid_ ASC"
 
 	stmt, err := ei.db.Prepare(s)
 	if err != nil {

chain/types/ethtypes/eth_types.go

@@ -229,13 +229,25 @@ type EthCall struct {
 }
 
 func (c *EthCall) UnmarshalJSON(b []byte) error {
-	type TempEthCall EthCall
-	var params TempEthCall
+	type EthCallRaw EthCall // Avoid a recursive call.
+	type EthCallDecode struct {
+		// The field should be "input" by spec, but many clients use "data" so we support
+		// both, but prefer "input".
+		Input *EthBytes `json:"input"`
+		EthCallRaw
+	}
 
+	var params EthCallDecode
 	if err := json.Unmarshal(b, &params); err != nil {
 		return err
 	}
-	*c = EthCall(params)
+
+	// If input is specified, prefer it.
+	if params.Input != nil {
+		params.Data = *params.Input
+	}
+
+	*c = EthCall(params.EthCallRaw)
 	return nil
 }
 

chain/types/ethtypes/eth_types_test.go

@@ -194,11 +194,40 @@ func TestMaskedIDInF4(t *testing.T) {
 }
 
 func TestUnmarshalEthCall(t *testing.T) {
-	data := `{"from":"0x4D6D86b31a112a05A473c4aE84afaF873f632325","to":"0xFe01CC39f5Ae8553D6914DBb9dC27D219fa22D7f","gas":"0x5","gasPrice":"0x6","value":"0x123","data":""}`
+	data := `{"from":"0x4D6D86b31a112a05A473c4aE84afaF873f632325","to":"0xFe01CC39f5Ae8553D6914DBb9dC27D219fa22D7f","gas":"0x5","gasPrice":"0x6","value":"0x123","data":"0xFF"}`
 
 	var c EthCall
 	err := c.UnmarshalJSON([]byte(data))
 	require.Nil(t, err)
+	require.EqualValues(t, []byte{0xff}, c.Data)
+}
+
+func TestUnmarshalEthCallInput(t *testing.T) {
+	data := `{"from":"0x4D6D86b31a112a05A473c4aE84afaF873f632325","to":"0xFe01CC39f5Ae8553D6914DBb9dC27D219fa22D7f","gas":"0x5","gasPrice":"0x6","value":"0x123","input":"0xFF"}`
+
+	var c EthCall
+	err := c.UnmarshalJSON([]byte(data))
+	require.Nil(t, err)
+	require.EqualValues(t, []byte{0xff}, c.Data)
+}
+
+func TestUnmarshalEthCallInputAndData(t *testing.T) {
+	data := `{"from":"0x4D6D86b31a112a05A473c4aE84afaF873f632325","to":"0xFe01CC39f5Ae8553D6914DBb9dC27D219fa22D7f","gas":"0x5","gasPrice":"0x6","value":"0x123","data":"0xFE","input":"0xFF"}`
+
+	var c EthCall
+	err := c.UnmarshalJSON([]byte(data))
+	require.Nil(t, err)
+	require.EqualValues(t, []byte{0xff}, c.Data)
+}
+
+func TestUnmarshalEthCallInputAndDataEmpty(t *testing.T) {
+	// Even if the input is empty, it should be used when specified.
+	data := `{"from":"0x4D6D86b31a112a05A473c4aE84afaF873f632325","to":"0xFe01CC39f5Ae8553D6914DBb9dC27D219fa22D7f","gas":"0x5","gasPrice":"0x6","value":"0x123","data":"0xFE","input":""}`
+
+	var c EthCall
+	err := c.UnmarshalJSON([]byte(data))
+	require.Nil(t, err)
+	require.EqualValues(t, []byte{}, c.Data)
 }
 
 func TestUnmarshalEthBytes(t *testing.T) {

cmd/lotus-miner/sectors.go

@@ -487,6 +487,13 @@ var sectorsListCmd = &cli.Command{
 				}
 			}
 
+			var pams int
+			for _, p := range st.Pieces {
+				if p.DealInfo != nil && p.DealInfo.PieceActivationManifest != nil {
+					pams++
+				}
+			}
+
 			exp := st.Expiration
 			if st.OnTime > 0 && st.OnTime < exp {
 				exp = st.OnTime // Can be different when the sector was CC upgraded
@@ -501,6 +508,8 @@ var sectorsListCmd = &cli.Command{
 
 			if deals > 0 {
 				m["Deals"] = color.GreenString("%d", deals)
+			} else if pams > 0 {
+				m["Deals"] = color.MagentaString("DDO:%d", pams)
 			} else {
 				m["Deals"] = color.BlueString("CC")
 				if st.ToUpgrade {

documentation/en/cli-lotus-miner.md

@@ -7,7 +7,7 @@ USAGE:
    lotus-miner [global options] command [command options] [arguments...]
 
 VERSION:
-   1.26.1
+   1.26.2
 
 COMMANDS:
    init     Initialize a lotus miner repo

documentation/en/cli-lotus-provider.md

@@ -7,7 +7,7 @@ USAGE:
    lotus-provider [global options] command [command options] [arguments...]
 
 VERSION:
-   1.26.1
+   1.26.2
 
 COMMANDS:
    run      Start a lotus provider process

documentation/en/cli-lotus-worker.md

@@ -7,7 +7,7 @@ USAGE:
    lotus-worker [global options] command [command options] [arguments...]
 
 VERSION:
-   1.26.1
+   1.26.2
 
 COMMANDS:
    run        Start lotus worker

documentation/en/cli-lotus.md

@@ -7,7 +7,7 @@ USAGE:
    lotus [global options] command [command options] [arguments...]
 
 VERSION:
-   1.26.1
+   1.26.2
 
 COMMANDS:
    daemon   Start a lotus daemon process

go.mod

@@ -111,7 +111,7 @@ require (
 	github.com/kelseyhightower/envconfig v1.4.0
 	github.com/koalacxr/quantile v0.0.1
 	github.com/libp2p/go-buffer-pool v0.1.0
-	github.com/libp2p/go-libp2p v0.33.1
+	github.com/libp2p/go-libp2p v0.33.2
 	github.com/libp2p/go-libp2p-consensus v0.0.1
 	github.com/libp2p/go-libp2p-gorpc v0.6.0
 	github.com/libp2p/go-libp2p-kad-dht v0.25.2
@@ -126,7 +126,7 @@ require (
 	github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1
 	github.com/mitchellh/go-homedir v1.1.0
 	github.com/multiformats/go-base32 v0.1.0
-	github.com/multiformats/go-multiaddr v0.12.2
+	github.com/multiformats/go-multiaddr v0.12.3
 	github.com/multiformats/go-multiaddr-dns v0.3.1
 	github.com/multiformats/go-multibase v0.2.0
 	github.com/multiformats/go-multicodec v0.9.0

go.sum

@@ -1012,8 +1012,8 @@ github.com/libp2p/go-libp2p v0.7.0/go.mod h1:hZJf8txWeCduQRDC/WSqBGMxaTHCOYHt2xS
 github.com/libp2p/go-libp2p v0.7.4/go.mod h1:oXsBlTLF1q7pxr+9w6lqzS1ILpyHsaBPniVO7zIHGMw=
 github.com/libp2p/go-libp2p v0.8.1/go.mod h1:QRNH9pwdbEBpx5DTJYg+qxcVaDMAz3Ee/qDKwXujH5o=
 github.com/libp2p/go-libp2p v0.14.3/go.mod h1:d12V4PdKbpL0T1/gsUNN8DfgMuRPDX8bS2QxCZlwRH0=
-github.com/libp2p/go-libp2p v0.33.1 h1:tvJl9b9M6nSLBtZSXSguq+/lRhRj2oLRkyhBmQNMFLA=
-github.com/libp2p/go-libp2p v0.33.1/go.mod h1:zOUTMjG4I7TXwMndNyOBn/CNtVBLlvBlnxfi+8xzx+E=
+github.com/libp2p/go-libp2p v0.33.2 h1:vCdwnFxoGOXMKmaGHlDSnL4bM3fQeW8pgIa9DECnb40=
+github.com/libp2p/go-libp2p v0.33.2/go.mod h1:zTeppLuCvUIkT118pFVzA8xzP/p2dJYOMApCkFh0Yww=
 github.com/libp2p/go-libp2p-asn-util v0.4.1 h1:xqL7++IKD9TBFMgnLPZR6/6iYhawHKHl950SO9L6n94=
 github.com/libp2p/go-libp2p-asn-util v0.4.1/go.mod h1:d/NI6XZ9qxw67b4e+NgpQexCIiFYJjErASrYW4PFDN8=
 github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3PtvYlJmg7NeVtPRqH8=
@@ -1309,8 +1309,8 @@ github.com/multiformats/go-multiaddr v0.2.2/go.mod h1:NtfXiOtHvghW9KojvtySjH5y0u
 github.com/multiformats/go-multiaddr v0.3.0/go.mod h1:dF9kph9wfJ+3VLAaeBqo9Of8x4fJxp6ggJGteB8HQTI=
 github.com/multiformats/go-multiaddr v0.3.1/go.mod h1:uPbspcUPd5AfaP6ql3ujFY+QWzmBD8uLLL4bXW0XfGc=
 github.com/multiformats/go-multiaddr v0.3.3/go.mod h1:lCKNGP1EQ1eZ35Za2wlqnabm9xQkib3fyB+nZXHLag0=
-github.com/multiformats/go-multiaddr v0.12.2 h1:9G9sTY/wCYajKa9lyfWPmpZAwe6oV+Wb1zcmMS1HG24=
-github.com/multiformats/go-multiaddr v0.12.2/go.mod h1:GKyaTYjZRdcUhyOetrxTk9z0cW+jA/YrnqTOvKgi44M=
+github.com/multiformats/go-multiaddr v0.12.3 h1:hVBXvPRcKG0w80VinQ23P5t7czWgg65BmIvQKjDydU8=
+github.com/multiformats/go-multiaddr v0.12.3/go.mod h1:sBXrNzucqkFJhvKOiwwLyqamGa/P5EIXNPLovyhQCII=
 github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q=
 github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q=
 github.com/multiformats/go-multiaddr-dns v0.2.0/go.mod h1:TJ5pr5bBO7Y1B18djPuRsVkduhQH2YqYSbxWJzYGdK0=

itests/direct_data_onboard_test.go

@@ -11,6 +11,7 @@ import (
 	"github.com/ipfs/go-cid"
 	"github.com/ipld/go-ipld-prime"
 	"github.com/ipld/go-ipld-prime/codec/dagcbor"
+	cidlink "github.com/ipld/go-ipld-prime/linking/cid"
 	"github.com/ipld/go-ipld-prime/node/basicnode"
 	"github.com/multiformats/go-multicodec"
 	"github.com/stretchr/testify/require"
@@ -24,6 +25,7 @@ import (
 	"github.com/filecoin-project/go-state-types/exitcode"
 	"github.com/filecoin-project/go-state-types/network"
 
+	"github.com/filecoin-project/lotus/api"
 	"github.com/filecoin-project/lotus/chain/actors/builtin/market"
 	minertypes "github.com/filecoin-project/lotus/chain/actors/builtin/miner"
 	"github.com/filecoin-project/lotus/chain/consensus/filcns"
@@ -131,15 +133,17 @@ func TestOnboardMixedMarketDDO(t *testing.T) {
 	var pieces []abi.PieceInfo
 	var dealID abi.DealID
 
+	// market ddo piece
+	var marketSector api.SectorOffset
+	var marketPiece abi.PieceInfo
+	marketPieceSize := abi.PaddedPieceSize(2048 / 2).Unpadded()
 	{
-		// market piece
-		pieceSize := abi.PaddedPieceSize(2048 / 2).Unpadded()
-		pieceData := make([]byte, pieceSize)
+		pieceData := make([]byte, marketPieceSize)
 		_, _ = rand.Read(pieceData)
 
-		dc, err := miner.ComputeDataCid(ctx, pieceSize, bytes.NewReader(pieceData))
+		marketPiece, err = miner.ComputeDataCid(ctx, marketPieceSize, bytes.NewReader(pieceData))
 		require.NoError(t, err)
-		pieces = append(pieces, dc)
+		pieces = append(pieces, marketPiece)
 
 		head, err := client.ChainHead(ctx)
 		require.NoError(t, err)
@@ -148,7 +152,7 @@ func TestOnboardMixedMarketDDO(t *testing.T) {
 
 		psdParams := market2.PublishStorageDealsParams{
 			Deals: []market2.ClientDealProposal{
-				makeMarketDealProposal(t, client, miner, dc.PieceCID, pieceSize.Padded(), head.Height()+2880*2, head.Height()+2880*400),
+				makeMarketDealProposal(t, client, miner, marketPiece.PieceCID, marketPieceSize.Padded(), head.Height()+2880*2, head.Height()+2880*400),
 			},
 		}
 
@@ -177,7 +181,7 @@ func TestOnboardMixedMarketDDO(t *testing.T) {
 
 		mcid := smsg.Cid()
 
-		so, err := miner.SectorAddPieceToAny(ctx, pieceSize, bytes.NewReader(pieceData), piece.PieceDealInfo{
+		marketSector, err = miner.SectorAddPieceToAny(ctx, marketPieceSize, bytes.NewReader(pieceData), piece.PieceDealInfo{
 			PublishCid:   &mcid,
 			DealID:       dealID,
 			DealProposal: &psdParams.Deals[0].Proposal,
@@ -190,25 +194,26 @@ func TestOnboardMixedMarketDDO(t *testing.T) {
 		})
 		require.NoError(t, err)
 
-		require.Equal(t, abi.PaddedPieceSize(0), so.Offset)
-		require.Equal(t, abi.SectorNumber(2), so.Sector)
+		require.Equal(t, abi.PaddedPieceSize(0), marketSector.Offset)
+		require.Equal(t, abi.SectorNumber(2), marketSector.Sector)
 	}
 
+	// raw ddo piece
+	var rawSector api.SectorOffset
+	var rawPiece abi.PieceInfo
+	rawPieceSize := abi.PaddedPieceSize(2048 / 2).Unpadded()
 	{
-		// raw ddo piece
-
-		pieceSize := abi.PaddedPieceSize(2048 / 2).Unpadded()
-		pieceData := make([]byte, pieceSize)
+		pieceData := make([]byte, rawPieceSize)
 		_, _ = rand.Read(pieceData)
 
-		dc, err := miner.ComputeDataCid(ctx, pieceSize, bytes.NewReader(pieceData))
+		rawPiece, err = miner.ComputeDataCid(ctx, rawPieceSize, bytes.NewReader(pieceData))
 		require.NoError(t, err)
-		pieces = append(pieces, dc)
+		pieces = append(pieces, rawPiece)
 
 		head, err := client.ChainHead(ctx)
 		require.NoError(t, err)
 
-		so, err := miner.SectorAddPieceToAny(ctx, pieceSize, bytes.NewReader(pieceData), piece.PieceDealInfo{
+		rawSector, err = miner.SectorAddPieceToAny(ctx, rawPieceSize, bytes.NewReader(pieceData), piece.PieceDealInfo{
 			PublishCid:   nil,
 			DealID:       0,
 			DealProposal: nil,
@@ -218,18 +223,20 @@ func TestOnboardMixedMarketDDO(t *testing.T) {
 			},
 			KeepUnsealed: false,
 			PieceActivationManifest: &minertypes.PieceActivationManifest{
-				CID:                   dc.PieceCID,
-				Size:                  dc.Size,
+				CID:                   rawPiece.PieceCID,
+				Size:                  rawPiece.Size,
 				VerifiedAllocationKey: nil,
 				Notify:                nil,
 			},
 		})
 		require.NoError(t, err)
 
-		require.Equal(t, abi.PaddedPieceSize(1024), so.Offset)
-		require.Equal(t, abi.SectorNumber(2), so.Sector)
+		require.Equal(t, abi.PaddedPieceSize(1024), rawSector.Offset)
+		require.Equal(t, abi.SectorNumber(2), rawSector.Sector)
 	}
 
+	require.Equal(t, marketSector.Sector, rawSector.Sector) // sanity check same sector
+
 	toCheck := map[abi.SectorNumber]struct{}{
 		2: {},
 	}
@@ -272,7 +279,7 @@ func TestOnboardMixedMarketDDO(t *testing.T) {
 			fmt.Println("piece", piece.PieceCID, piece.Size)
 		}
 
-		// check "deal-published" actor event
+		// check some actor events
 		var epochZero abi.ChainEpoch
 		allEvents, err := miner.FullNode.GetActorEventsRaw(ctx, &types.ActorEventFilter{
 			FromHeight: &epochZero,
@@ -282,21 +289,32 @@ func TestOnboardMixedMarketDDO(t *testing.T) {
 			var found bool
 			keyBytes := must.One(ipld.Encode(basicnode.NewString(key), dagcbor.Encode))
 			for _, event := range allEvents {
-				for _, e := range event.Entries {
-					if e.Key == "$type" && bytes.Equal(e.Value, keyBytes) {
-						found = true
-						switch key {
-						case "deal-published", "deal-activated":
-							expectedEntries := []types.EventEntry{
-								{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "$type", Value: keyBytes},
-								{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "id", Value: must.One(ipld.Encode(basicnode.NewInt(2), dagcbor.Encode))},
-								{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "client", Value: must.One(ipld.Encode(basicnode.NewInt(int64(clientId)), dagcbor.Encode))},
-								{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "provider", Value: must.One(ipld.Encode(basicnode.NewInt(int64(minerId)), dagcbor.Encode))},
-							}
-							require.ElementsMatch(t, expectedEntries, event.Entries)
+				require.True(t, len(event.Entries) > 0)
+				if event.Entries[0].Key == "$type" && bytes.Equal(event.Entries[0].Value, keyBytes) {
+					found = true
+					switch key {
+					case "deal-published", "deal-activated":
+						expectedEntries := []types.EventEntry{
+							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "$type", Value: keyBytes},
+							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "id", Value: must.One(ipld.Encode(basicnode.NewInt(2), dagcbor.Encode))},
+							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "client", Value: must.One(ipld.Encode(basicnode.NewInt(int64(clientId)), dagcbor.Encode))},
+							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "provider", Value: must.One(ipld.Encode(basicnode.NewInt(int64(minerId)), dagcbor.Encode))},
 						}
-						break
+						require.Equal(t, expectedEntries, event.Entries)
+					case "sector-activated":
+						// only one sector, that has both our pieces in it
+						expectedEntries := []types.EventEntry{
+							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "$type", Value: must.One(ipld.Encode(basicnode.NewString("sector-activated"), dagcbor.Encode))},
+							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "sector", Value: must.One(ipld.Encode(basicnode.NewInt(int64(rawSector.Sector)), dagcbor.Encode))},
+							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "unsealed-cid", Value: must.One(ipld.Encode(basicnode.NewLink(cidlink.Link{Cid: expectCommD}), dagcbor.Encode))},
+							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "piece-cid", Value: must.One(ipld.Encode(basicnode.NewLink(cidlink.Link{Cid: marketPiece.PieceCID}), dagcbor.Encode))},
+							{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "piece-size", Value: must.One(ipld.Encode(basicnode.NewInt(int64(marketPieceSize.Padded())), dagcbor.Encode))},
+							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "piece-cid", Value: must.One(ipld.Encode(basicnode.NewLink(cidlink.Link{Cid: rawPiece.PieceCID}), dagcbor.Encode))},
+							{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "piece-size", Value: must.One(ipld.Encode(basicnode.NewInt(int64(rawPieceSize.Padded())), dagcbor.Encode))},
+						}
+						require.Equal(t, expectedEntries, event.Entries)
 					}
+					break
 				}
 			}
 			require.True(t, found, "expected to find event %s", key)

itests/direct_data_onboard_verified_test.go

@@ -175,11 +175,11 @@ func TestOnboardRawPieceVerified_WithActorEvents(t *testing.T) {
 			// first sector to start mining is CC
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "sector", Value: must.One(ipld.Encode(basicnode.NewInt(int64(so.Sector)-1), dagcbor.Encode))},
 		}
-		require.ElementsMatch(t, expectedEntries, precommitedEvents[0].Entries)
+		require.Equal(t, expectedEntries, precommitedEvents[0].Entries)
 
 		// second sector has our piece
 		expectedEntries[1].Value = must.One(ipld.Encode(basicnode.NewInt(int64(so.Sector)), dagcbor.Encode))
-		require.ElementsMatch(t, expectedEntries, precommitedEvents[1].Entries)
+		require.Equal(t, expectedEntries, precommitedEvents[1].Entries)
 	}
 
 	{
@@ -192,7 +192,7 @@ func TestOnboardRawPieceVerified_WithActorEvents(t *testing.T) {
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "sector", Value: must.One(ipld.Encode(basicnode.NewInt(int64(so.Sector)-1), dagcbor.Encode))},
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "unsealed-cid", Value: must.One(ipld.Encode(datamodel.Null, dagcbor.Encode))},
 		}
-		require.ElementsMatch(t, expectedEntries, activatedEvents[0].Entries)
+		require.Equal(t, expectedEntries, activatedEvents[0].Entries)
 
 		// second sector has our piece, and only our piece, so usealed-cid matches piece-cid,
 		// unfortunately we don't have a case with multiple pieces
@@ -202,7 +202,7 @@ func TestOnboardRawPieceVerified_WithActorEvents(t *testing.T) {
 			types.EventEntry{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "piece-cid", Value: must.One(ipld.Encode(basicnode.NewLink(cidlink.Link{Cid: dc.PieceCID}), dagcbor.Encode))},
 			types.EventEntry{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "piece-size", Value: must.One(ipld.Encode(basicnode.NewInt(int64(pieceSize.Padded())), dagcbor.Encode))},
 		)
-		require.ElementsMatch(t, expectedEntries, activatedEvents[1].Entries)
+		require.Equal(t, expectedEntries, activatedEvents[1].Entries)
 	}
 
 	{
@@ -232,21 +232,21 @@ func TestOnboardRawPieceVerified_WithActorEvents(t *testing.T) {
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "$type", Value: must.One(ipld.Encode(basicnode.NewString("allocation"), dagcbor.Encode))},
 			// first, bogus, allocation
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "id", Value: must.One(ipld.Encode(basicnode.NewInt(int64(allocationId)-1), dagcbor.Encode))},
-			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "provider", Value: must.One(ipld.Encode(basicnode.NewInt(int64(minerId)), dagcbor.Encode))},
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "client", Value: must.One(ipld.Encode(basicnode.NewInt(int64(clientId)), dagcbor.Encode))},
+			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "provider", Value: must.One(ipld.Encode(basicnode.NewInt(int64(minerId)), dagcbor.Encode))},
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "piece-cid", Value: must.One(ipld.Encode(basicnode.NewLink(cidlink.Link{Cid: bogusPieceCid}), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "piece-size", Value: must.One(ipld.Encode(basicnode.NewInt(int64(pieceSize.Padded())), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "term-min", Value: must.One(ipld.Encode(basicnode.NewInt(verifregtypes13.MinimumVerifiedAllocationTerm), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "term-max", Value: must.One(ipld.Encode(basicnode.NewInt(verifregtypes13.MaximumVerifiedAllocationTerm), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "expiration", Value: must.One(ipld.Encode(basicnode.NewInt(int64(bogusAllocationExpiry)), dagcbor.Encode))},
 		}
-		require.ElementsMatch(t, expectedEntries, allocationEvents[0].Entries)
+		require.Equal(t, expectedEntries, allocationEvents[0].Entries)
 
 		// the second, real allocation
 		expectedEntries[1].Value = must.One(ipld.Encode(basicnode.NewInt(int64(allocationId)), dagcbor.Encode))                                 // "id"
 		expectedEntries[4].Value = must.One(ipld.Encode(basicnode.NewLink(cidlink.Link{Cid: dc.PieceCID}), dagcbor.Encode))                     // "piece-cid"
 		expectedEntries[8].Value = must.One(ipld.Encode(basicnode.NewInt(verifregtypes13.MaximumVerifiedAllocationExpiration), dagcbor.Encode)) // "expiration"
-		require.ElementsMatch(t, expectedEntries, allocationEvents[1].Entries)
+		require.Equal(t, expectedEntries, allocationEvents[1].Entries)
 	}
 
 	{
@@ -257,15 +257,15 @@ func TestOnboardRawPieceVerified_WithActorEvents(t *testing.T) {
 		expectedEntries := []types.EventEntry{
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "$type", Value: must.One(ipld.Encode(basicnode.NewString("allocation-removed"), dagcbor.Encode))},
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "id", Value: must.One(ipld.Encode(basicnode.NewInt(int64(allocationId)-1), dagcbor.Encode))},
-			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "provider", Value: must.One(ipld.Encode(basicnode.NewInt(int64(minerId)), dagcbor.Encode))},
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "client", Value: must.One(ipld.Encode(basicnode.NewInt(int64(clientId)), dagcbor.Encode))},
+			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "provider", Value: must.One(ipld.Encode(basicnode.NewInt(int64(minerId)), dagcbor.Encode))},
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "piece-cid", Value: must.One(ipld.Encode(basicnode.NewLink(cidlink.Link{Cid: bogusPieceCid}), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "piece-size", Value: must.One(ipld.Encode(basicnode.NewInt(int64(pieceSize.Padded())), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "term-min", Value: must.One(ipld.Encode(basicnode.NewInt(verifregtypes13.MinimumVerifiedAllocationTerm), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "term-max", Value: must.One(ipld.Encode(basicnode.NewInt(verifregtypes13.MaximumVerifiedAllocationTerm), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "expiration", Value: must.One(ipld.Encode(basicnode.NewInt(int64(bogusAllocationExpiry)), dagcbor.Encode))},
 		}
-		require.ElementsMatch(t, expectedEntries, allocationEvents[0].Entries)
+		require.Equal(t, expectedEntries, allocationEvents[0].Entries)
 	}
 
 	{
@@ -276,8 +276,8 @@ func TestOnboardRawPieceVerified_WithActorEvents(t *testing.T) {
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "$type", Value: must.One(ipld.Encode(basicnode.NewString("claim"), dagcbor.Encode))},
 			// claimId inherits from its original allocationId
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "id", Value: must.One(ipld.Encode(basicnode.NewInt(int64(allocationId)), dagcbor.Encode))},
-			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "provider", Value: must.One(ipld.Encode(basicnode.NewInt(int64(minerId)), dagcbor.Encode))},
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "client", Value: must.One(ipld.Encode(basicnode.NewInt(int64(clientId)), dagcbor.Encode))},
+			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "provider", Value: must.One(ipld.Encode(basicnode.NewInt(int64(minerId)), dagcbor.Encode))},
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "piece-cid", Value: must.One(ipld.Encode(basicnode.NewLink(cidlink.Link{Cid: dc.PieceCID}), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "piece-size", Value: must.One(ipld.Encode(basicnode.NewInt(int64(pieceSize.Padded())), dagcbor.Encode))},
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "term-min", Value: must.One(ipld.Encode(basicnode.NewInt(verifregtypes13.MinimumVerifiedAllocationTerm), dagcbor.Encode))},
@@ -285,7 +285,7 @@ func TestOnboardRawPieceVerified_WithActorEvents(t *testing.T) {
 			{Flags: 0x01, Codec: uint64(multicodec.Cbor), Key: "term-start", Value: must.One(ipld.Encode(basicnode.NewInt(int64(claimEvents[0].Height)), dagcbor.Encode))},
 			{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "sector", Value: must.One(ipld.Encode(basicnode.NewInt(int64(si.SectorID)), dagcbor.Encode))},
 		}
-		require.ElementsMatch(t, expectedEntries, claimEvents[0].Entries)
+		require.Equal(t, expectedEntries, claimEvents[0].Entries)
 	}
 
 	// verify that we can trace a datacap allocation through to a claim with the events, since this

itests/sector_terminate_test.go

@@ -189,7 +189,7 @@ loop:
 							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "$type", Value: keyBytes},
 							{Flags: 0x03, Codec: uint64(multicodec.Cbor), Key: "sector", Value: must.One(ipld.Encode(basicnode.NewInt(int64(toTerminate)), dagcbor.Encode))},
 						}
-						require.ElementsMatch(t, expectedEntries, event.Entries)
+						require.Equal(t, expectedEntries, event.Entries)
 					}
 					break
 				}

storage/pipeline/piece/piece_info.go

@@ -147,7 +147,7 @@ func (ds *PieceDealInfo) EndEpoch() (abi.ChainEpoch, error) {
 	default:
 		// note - when implementing make sure to cache any dynamically computed values
 		// todo do we want a smarter mechanism here
-		return ds.DealSchedule.StartEpoch, nil
+		return ds.DealSchedule.EndEpoch, nil
 	}
 }
 

storage/pipeline/precommit_policy_test.go

@@ -18,6 +18,7 @@ import (
 	"github.com/filecoin-project/lotus/api"
 	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/actors/builtin"
+	"github.com/filecoin-project/lotus/chain/actors/builtin/miner"
 	"github.com/filecoin-project/lotus/chain/actors/policy"
 	"github.com/filecoin-project/lotus/chain/types"
 	"github.com/filecoin-project/lotus/node/modules/dtypes"
@@ -242,3 +243,37 @@ func TestMissingDealIsIgnored(t *testing.T) {
 
 	assert.Equal(t, 547300, int(exp))
 }
+
+func TestBasicPolicyDDO(t *testing.T) {
+	cfg := fakeConfigGetter(nil)
+	pcp := pipeline.NewBasicPreCommitPolicy(&fakeChain{
+		h: abi.ChainEpoch(55),
+	}, cfg, 0)
+
+	pieces := []pipeline.SafeSectorPiece{
+		pipeline.SafePiece(api.SectorPiece{
+			Piece: abi.PieceInfo{
+				Size:     abi.PaddedPieceSize(1024),
+				PieceCID: fakePieceCid(t),
+			},
+			DealInfo: &piece.PieceDealInfo{
+				PublishCid: nil,
+				DealID:     abi.DealID(44),
+				DealSchedule: piece.DealSchedule{
+					StartEpoch: abi.ChainEpoch(100_000),
+					EndEpoch:   abi.ChainEpoch(1500_000),
+				},
+				PieceActivationManifest: &miner.PieceActivationManifest{
+					Size:                  0,
+					VerifiedAllocationKey: nil,
+					Notify:                nil,
+				},
+			},
+		}),
+	}
+
+	exp, err := pcp.Expiration(context.Background(), pieces...)
+	require.NoError(t, err)
+
+	assert.Equal(t, abi.ChainEpoch(1500_000), exp)
+}

storage/pipeline/states_replica_update.go

@@ -11,6 +11,7 @@ import (
 	"github.com/filecoin-project/go-state-types/big"
 	"github.com/filecoin-project/go-state-types/builtin"
 	"github.com/filecoin-project/go-state-types/exitcode"
+	"github.com/filecoin-project/go-state-types/network"
 	"github.com/filecoin-project/go-statemachine"
 
 	"github.com/filecoin-project/lotus/api"
@@ -170,7 +171,12 @@ func (m *Sealing) handleSubmitReplicaUpdate(ctx statemachine.Context, sector Sec
 
 	// figure out message type
 
-	pams, deals, err := m.processPieces(ctx.Context(), sector)
+	nv, err := m.Api.StateNetworkVersion(ctx.Context(), ts.Key())
+	if err != nil {
+		log.Errorf("failed to get network version: %+v", err)
+	}
+
+	pams, deals, err := m.processPieces(ctx.Context(), sector, nv >= network.Version22)
 	if err != nil {
 		log.Errorf("failed to process pieces: %+v", err)
 		return ctx.Send(SectorSubmitReplicaUpdateFailed{})

storage/pipeline/states_sealing.go

@@ -748,30 +748,33 @@ func (m *Sealing) handleSubmitCommit(ctx statemachine.Context, sector SectorInfo
 // processPieces returns either:
 // - a list of piece activation manifests
 // - a list of deal IDs, if all non-filler pieces are deal-id pieces
-func (m *Sealing) processPieces(ctx context.Context, sector SectorInfo) ([]miner.PieceActivationManifest, []abi.DealID, error) {
+func (m *Sealing) processPieces(ctx context.Context, sector SectorInfo, forceDDO bool) ([]miner.PieceActivationManifest, []abi.DealID, error) {
 	pams := make([]miner.PieceActivationManifest, 0, len(sector.Pieces))
 	dealIDs := make([]abi.DealID, 0, len(sector.Pieces))
-	var hasDDO bool
+	hasDDO := forceDDO
 
-	for _, piece := range sector.Pieces {
-		piece := piece
+	if !forceDDO {
+		// if not forcing DDO, check if we have any DDO pieces
+		for _, piece := range sector.Pieces {
+			piece := piece
 
-		// first figure out if this is a ddo sector
-		err := piece.handleDealInfo(handleDealInfoParams{
-			FillerHandler: func(info UniversalPieceInfo) error {
-				// Fillers are implicit (todo review: Are they??)
-				return nil
-			},
-			BuiltinMarketHandler: func(info UniversalPieceInfo) error {
-				return nil
-			},
-			DDOHandler: func(info UniversalPieceInfo) error {
-				hasDDO = true
-				return nil
-			},
-		})
-		if err != nil {
-			return nil, nil, xerrors.Errorf("handleDealInfo: %w", err)
+			// first figure out if this is a ddo sector
+			err := piece.handleDealInfo(handleDealInfoParams{
+				FillerHandler: func(info UniversalPieceInfo) error {
+					// Fillers are implicit (todo review: Are they??)
+					return nil
+				},
+				BuiltinMarketHandler: func(info UniversalPieceInfo) error {
+					return nil
+				},
+				DDOHandler: func(info UniversalPieceInfo) error {
+					hasDDO = true
+					return nil
+				},
+			})
+			if err != nil {
+				return nil, nil, xerrors.Errorf("handleDealInfo: %w", err)
+			}
 		}
 	}
 	for _, piece := range sector.Pieces {
@@ -847,7 +850,7 @@ func (m *Sealing) handleSubmitCommitAggregate(ctx statemachine.Context, sector S
 		return ctx.Send(SectorCommitFailed{xerrors.Errorf("sector had nil commR or commD")})
 	}
 
-	pams, dealIDs, err := m.processPieces(ctx.Context(), sector)
+	pams, dealIDs, err := m.processPieces(ctx.Context(), sector, false)
 	if err != nil {
 		return err
 	}