Merge branch 'release/v1.11.1' into feat/replace-multistore-carv2

Fixes in master -> release

CHANGELOG.md

@@ -1,5 +1,184 @@
 # Lotus changelog
 
+# 1.11.1-rc1 / 2021-07-27
+
+This is the first release candidate for the **highly-recommended** but optional Lotus v1.11.1 release that introduces many deal making and datastore improvements and new features along with other bug fixes. A more organized and detailed release note will be shared in the next few days with future RCs, highlights are:
+- [lotus-miner market subsystem](https://docs.filecoin.io/mine/lotus/split-markets-miners/#frontmatter-title)
+- [splistore](https://github.com/filecoin-project/lotus/blob/master/blockstore/splitstore/README.md)
+
+- github.com/filecoin-project/lotus:
+  - Merge branch 'releases' into release/v1.11.1
+  - Update to proof v8.0.3 ([filecoin-project/lotus#6890](https://github.com/filecoin-project/lotus/pull/6890))
+  - lotus-shed: initial export cmd for markets related metadata ([filecoin-project/lotus#6840](https://github.com/filecoin-project/lotus/pull/6840))
+  - add a very verbose -vv flag to lotus and lotus-miner. ([filecoin-project/lotus#6888](https://github.com/filecoin-project/lotus/pull/6888))
+  - Update RELEASE_ISSUE_TEMPLATE.md ([filecoin-project/lotus#6880](https://github.com/filecoin-project/lotus/pull/6880))
+  - Moving GC for badger ([filecoin-project/lotus#6854](https://github.com/filecoin-project/lotus/pull/6854))
+  - Add github actions for staled pr ([filecoin-project/lotus#6879](https://github.com/filecoin-project/lotus/pull/6879))
+  - Add allocated sectorid vis ([filecoin-project/lotus#4638](https://github.com/filecoin-project/lotus/pull/4638))
+  - rename `cmd/lotus{-storage=>}-miner` to match binary. ([filecoin-project/lotus#6886](https://github.com/filecoin-project/lotus/pull/6886))
+  - update to go-fil-market v1.6.0 ([filecoin-project/lotus#6885](https://github.com/filecoin-project/lotus/pull/6885))
+  - Bump go-multihash, adjust test for supported version ([filecoin-project/lotus#6674](https://github.com/filecoin-project/lotus/pull/6674))
+  - Fix padding of deals, which only partially shipped in #5988 ([filecoin-project/lotus#6683](https://github.com/filecoin-project/lotus/pull/6683))
+  - fix racy TestSimultanenousTransferLimit. ([filecoin-project/lotus#6862](https://github.com/filecoin-project/lotus/pull/6862))
+  - Improve splitstore warmup ([filecoin-project/lotus#6867](https://github.com/filecoin-project/lotus/pull/6867))
+  - ValidateBlock: Assert that block header height's are greater than parents ([filecoin-project/lotus#6872](https://github.com/filecoin-project/lotus/pull/6872))
+  - feat: Don't panic when api impl is nil ([filecoin-project/lotus#6857](https://github.com/filecoin-project/lotus/pull/6857))
+  - splitstore shed utils ([filecoin-project/lotus#6811](https://github.com/filecoin-project/lotus/pull/6811))
+  - Fix links in issue templates
+  - Update issue templates and add templates for M1 ([filecoin-project/lotus#6856](https://github.com/filecoin-project/lotus/pull/6856))
+  - Splitstore: support on-disk marksets using badger ([filecoin-project/lotus#6833](https://github.com/filecoin-project/lotus/pull/6833))
+  - Config UX improvements ([filecoin-project/lotus#6848](https://github.com/filecoin-project/lotus/pull/6848))
+  - fix deal concurrency test failures by upgrading graphsync and others ([filecoin-project/lotus#6724](https://github.com/filecoin-project/lotus/pull/6724))
+  - Update issue templates to forms ([filecoin-project/lotus#6798](https://github.com/filecoin-project/lotus/pull/6798))
+  - Nerpa v13 upgrade ([filecoin-project/lotus#6837](https://github.com/filecoin-project/lotus/pull/6837))
+  - add docker-compose file ([filecoin-project/lotus#6544](https://github.com/filecoin-project/lotus/pull/6544))
+  - fix warmup by decoupling state from message receipt walk ([filecoin-project/lotus#6841](https://github.com/filecoin-project/lotus/pull/6841))
+  - add a command for compacting sector numbers bitfield ([filecoin-project/lotus#4640](https://github.com/filecoin-project/lotus/pull/4640))
+  - PriceListByVersion ([filecoin-project/lotus#6766](https://github.com/filecoin-project/lotus/pull/6766))
+  - easy way to make install app ([filecoin-project/lotus#5183](https://github.com/filecoin-project/lotus/pull/5183))
+  - api: Separate the Net interface from Common ([filecoin-project/lotus#6627](https://github.com/filecoin-project/lotus/pull/6627))
+  - cache loaded block messages ([filecoin-project/lotus#6760](https://github.com/filecoin-project/lotus/pull/6760))
+  - fix: on randomness change, use new rand ([filecoin-project/lotus#6805](https://github.com/filecoin-project/lotus/pull/6805))
+  - Splitstore: add retention policy option for keeping messages in the hotstore ([filecoin-project/lotus#6775](https://github.com/filecoin-project/lotus/pull/6775))
+  - Introduce the LOTUS_CHAIN_BADGERSTORE_DISABLE_FSYNC envvar ([filecoin-project/lotus#6817](https://github.com/filecoin-project/lotus/pull/6817))
+  - add StateReadState to gateway api ([filecoin-project/lotus#6818](https://github.com/filecoin-project/lotus/pull/6818))
+  - add SealProof in SectorBuilder ([filecoin-project/lotus#6815](https://github.com/filecoin-project/lotus/pull/6815))
+  - release -> master ([filecoin-project/lotus#6828](https://github.com/filecoin-project/lotus/pull/6828))
+  - sealing: Handle preCommitParams errors more correctly ([filecoin-project/lotus#6763](https://github.com/filecoin-project/lotus/pull/6763))
+  - fix: always check if StateSearchMessage returns nil ([filecoin-project/lotus#6802](https://github.com/filecoin-project/lotus/pull/6802))
+  - ClientFindData: always fetch peer id from chain ([filecoin-project/lotus#6807](https://github.com/filecoin-project/lotus/pull/6807))
+  - test: fix flaky window post tests ([filecoin-project/lotus#6804](https://github.com/filecoin-project/lotus/pull/6804))
+  -  ([filecoin-project/lotus#6800](https://github.com/filecoin-project/lotus/pull/6800))
+  - fixes #6786 segfault ([filecoin-project/lotus#6787](https://github.com/filecoin-project/lotus/pull/6787))
+  - Splitstore: add support for protecting out of chain references in the blockstore ([filecoin-project/lotus#6777](https://github.com/filecoin-project/lotus/pull/6777))
+  - Resurrect CODEOWNERS, but for maintainers group ([filecoin-project/lotus#6773](https://github.com/filecoin-project/lotus/pull/6773))
+  - update go-libp2p-pubsub to v0.5.0 ([filecoin-project/lotus#6764](https://github.com/filecoin-project/lotus/pull/6764))
+  - Implement exposed splitstore ([filecoin-project/lotus#6762](https://github.com/filecoin-project/lotus/pull/6762))
+  - Add ChainGetMessagesInTipset API ([filecoin-project/lotus#6642](https://github.com/filecoin-project/lotus/pull/6642))
+  - test: handle null blocks in TestForkRefuseCall ([filecoin-project/lotus#6758](https://github.com/filecoin-project/lotus/pull/6758))
+  - Master disclaimer  ([filecoin-project/lotus#6757](https://github.com/filecoin-project/lotus/pull/6757))
+  - Splitstore code reorg ([filecoin-project/lotus#6756](https://github.com/filecoin-project/lotus/pull/6756))
+  - Create stale.yml ([filecoin-project/lotus#6747](https://github.com/filecoin-project/lotus/pull/6747))
+  - Splitstore: Some small fixes ([filecoin-project/lotus#6754](https://github.com/filecoin-project/lotus/pull/6754))
+  -  ([filecoin-project/lotus#6746](https://github.com/filecoin-project/lotus/pull/6746))
+  - Handle the --color flag via proper global state ([filecoin-project/lotus#6743](https://github.com/filecoin-project/lotus/pull/6743))
+  - Config for collateral from miner available balance ([filecoin-project/lotus#6629](https://github.com/filecoin-project/lotus/pull/6629))
+  - Support standalone miner-market process ([filecoin-project/lotus#6356](https://github.com/filecoin-project/lotus/pull/6356))
+  - Splitstore Enhanchements ([filecoin-project/lotus#6474](https://github.com/filecoin-project/lotus/pull/6474))
+  -  ([filecoin-project/lotus#6739](https://github.com/filecoin-project/lotus/pull/6739))
+  - Add more deal details to lotus-miner info ([filecoin-project/lotus#6708](https://github.com/filecoin-project/lotus/pull/6708))
+  - Release template: Update all testnet infra at once ([filecoin-project/lotus#6710](https://github.com/filecoin-project/lotus/pull/6710))
+  - Fix Lotus shed
+  - Fix bugs in sectors extend --v1-sectors ([filecoin-project/lotus#6066](https://github.com/filecoin-project/lotus/pull/6066))
+  - add election backtest ([filecoin-project/lotus#5950](https://github.com/filecoin-project/lotus/pull/5950))
+  - Envvar to disable slash filter ([filecoin-project/lotus#6620](https://github.com/filecoin-project/lotus/pull/6620))
+  - Release Template: remove binary validation step ([filecoin-project/lotus#6709](https://github.com/filecoin-project/lotus/pull/6709))
+  - Config for deal publishing control addresses ([filecoin-project/lotus#6697](https://github.com/filecoin-project/lotus/pull/6697))
+  - Reset of the interop network ([filecoin-project/lotus#6689](https://github.com/filecoin-project/lotus/pull/6689))
+  - Enable color by default only if os.Stdout is a TTY ([filecoin-project/lotus#6696](https://github.com/filecoin-project/lotus/pull/6696))
+  - Stop outputing ANSI color on non-TTY ([filecoin-project/lotus#6694](https://github.com/filecoin-project/lotus/pull/6694))
+  - add dollar sign ([filecoin-project/lotus#6690](https://github.com/filecoin-project/lotus/pull/6690))
+  - get-actor cli spelling fix ([filecoin-project/lotus#6681](https://github.com/filecoin-project/lotus/pull/6681))
+  - fix "lotus-seed genesis car" error "merkledag: not found" ([filecoin-project/lotus#6688](https://github.com/filecoin-project/lotus/pull/6688))
+  - polish(statetree): accept a context in statetree diff for timeouts ([filecoin-project/lotus#6639](https://github.com/filecoin-project/lotus/pull/6639))
+  - Add helptext to lotus chain export ([filecoin-project/lotus#6672](https://github.com/filecoin-project/lotus/pull/6672))
+  - Get retrieval pricing input should not error out on a deal state fetch ([filecoin-project/lotus#6679](https://github.com/filecoin-project/lotus/pull/6679))
+  - Fix more CID double-encoding as hex ([filecoin-project/lotus#6680](https://github.com/filecoin-project/lotus/pull/6680))
+  - add an incremental nonce itest. ([filecoin-project/lotus#6663](https://github.com/filecoin-project/lotus/pull/6663))
+  - storage: Fix FinalizeSector with sectors in stoage paths ([filecoin-project/lotus#6653](https://github.com/filecoin-project/lotus/pull/6653))
+  - Fix tiny error in check-client-datacap ([filecoin-project/lotus#6664](https://github.com/filecoin-project/lotus/pull/6664))
+  - Fix: precommit_batch method used the wrong cfg.CommitBatchWait ([filecoin-project/lotus#6658](https://github.com/filecoin-project/lotus/pull/6658))
+  - fix ticket expiration check ([filecoin-project/lotus#6635](https://github.com/filecoin-project/lotus/pull/6635))
+  - commit batch: AggregateAboveBaseFee config ([filecoin-project/lotus#6650](https://github.com/filecoin-project/lotus/pull/6650))
+  - commit batch: Initialize the FailedSectors map ([filecoin-project/lotus#6647](https://github.com/filecoin-project/lotus/pull/6647))
+  - Fast-path retry submitting commit aggregate if commit is still valid  ([filecoin-project/lotus#6638](https://github.com/filecoin-project/lotus/pull/6638))
+  - remove precommit check in handleCommitFailed ([filecoin-project/lotus#6634](https://github.com/filecoin-project/lotus/pull/6634))
+  - Reuse timers in sealing batch logic ([filecoin-project/lotus#6636](https://github.com/filecoin-project/lotus/pull/6636))
+  - shed tool to estimate aggregate network fees ([filecoin-project/lotus#6631](https://github.com/filecoin-project/lotus/pull/6631))
+  - fix prove commit aggregate send token amount ([filecoin-project/lotus#6625](https://github.com/filecoin-project/lotus/pull/6625))
+  - Update version.go to 1.11.1 ([filecoin-project/lotus#6621](https://github.com/filecoin-project/lotus/pull/6621))
+- github.com/filecoin-project/go-data-transfer (v1.6.0 -> v1.7.0):
+  - release: v1.7.0
+  - Fire a transfer queued event when a transfer is queued in Graphsync (#221) ([filecoin-project/go-data-transfer#221](https://github.com/filecoin-project/go-data-transfer/pull/221))
+  - feat: pass ChannelID to ValidatePush & ValidatePull (#220) ([filecoin-project/go-data-transfer#220](https://github.com/filecoin-project/go-data-transfer/pull/220))
+  - release: v1.6.1 ([filecoin-project/go-data-transfer#218](https://github.com/filecoin-project/go-data-transfer/pull/218))
+  - Remove CID lists (#217) ([filecoin-project/go-data-transfer#217](https://github.com/filecoin-project/go-data-transfer/pull/217))
+  - Merge v1.6.0 ([filecoin-project/go-data-transfer#214](https://github.com/filecoin-project/go-data-transfer/pull/214))
+  - Remove restart ack timeout (#211) ([filecoin-project/go-data-transfer#211](https://github.com/filecoin-project/go-data-transfer/pull/211))
+  - feat: use different extension names to fit multiple hooks data in same graphsync message (#204) ([filecoin-project/go-data-transfer#204](https://github.com/filecoin-project/go-data-transfer/pull/204))
+  - fix: map race in GS transport (#208) ([filecoin-project/go-data-transfer#208](https://github.com/filecoin-project/go-data-transfer/pull/208))
+  - refactor: simplify graphsync transport (#203) ([filecoin-project/go-data-transfer#203](https://github.com/filecoin-project/go-data-transfer/pull/203))
+  - release: v1.5.0 (#200) ([filecoin-project/go-data-transfer#200](https://github.com/filecoin-project/go-data-transfer/pull/200))
+- github.com/filecoin-project/go-fil-markets (v1.5.0 -> v1.6.0):
+  - release: v1.6.0
+  - support padding out smaller files (#536) ([filecoin-project/go-fil-markets#536](https://github.com/filecoin-project/go-fil-markets/pull/536))
+  - On overloaded CI 10 seconds just isn't enough (#587) ([filecoin-project/go-fil-markets#587](https://github.com/filecoin-project/go-fil-markets/pull/587))
+  - Do not hex-encode CIDs in logs (#561) ([filecoin-project/go-fil-markets#561](https://github.com/filecoin-project/go-fil-markets/pull/561))
+  - remove wrong peer check in push deal validation (#585) ([filecoin-project/go-fil-markets#585](https://github.com/filecoin-project/go-fil-markets/pull/585))
+  - fix: circleci docs-gen task (#574) ([filecoin-project/go-fil-markets#574](https://github.com/filecoin-project/go-fil-markets/pull/574))
+  - Storage market request queued event and validation interface changes (#555) ([filecoin-project/go-fil-markets#555](https://github.com/filecoin-project/go-fil-markets/pull/555))
+  - build(deps): bump ws from 6.2.1 to 6.2.2 (#554) ([filecoin-project/go-fil-markets#554](https://github.com/filecoin-project/go-fil-markets/pull/554))
+  - release: v1.5.0 ([filecoin-project/go-fil-markets#553](https://github.com/filecoin-project/go-fil-markets/pull/553))
+- github.com/filecoin-project/go-padreader (v0.0.0-20200903213702-ed5fae088b20 -> v0.0.0-20210723183308-812a16dc01b1):
+  - New method to pad harder (#6) ([filecoin-project/go-padreader#6](https://github.com/filecoin-project/go-padreader/pull/6))
+  - Create SECURITY.md (#5) ([filecoin-project/go-padreader#5](https://github.com/filecoin-project/go-padreader/pull/5))
+- github.com/filecoin-project/go-state-types (v0.1.1-0.20210506134452-99b279731c48 -> v0.1.1-0.20210722133031-ad9bfe54c124):
+  - Add version 6.5 (#30) ([filecoin-project/go-state-types#30](https://github.com/filecoin-project/go-state-types/pull/30))
+  - rename file
+- github.com/filecoin-project/specs-actors/v5 (v5.0.1 -> v5.0.3):
+  - Adjust code for subtle change in go-multihash 0.0.15 (#1463) ([filecoin-project/specs-actors#1463](https://github.com/filecoin-project/specs-actors/pull/1463))
+  - Bump go state types (#1464) ([filecoin-project/specs-actors#1464](https://github.com/filecoin-project/specs-actors/pull/1464))
+  - Create CODEOWNERS (#1465) ([filecoin-project/specs-actors#1465](https://github.com/filecoin-project/specs-actors/pull/1465))
+  - Test deterministic offset (#1462) ([filecoin-project/specs-actors#1462](https://github.com/filecoin-project/specs-actors/pull/1462))
+
+Contributors
+
+| Contributor | Commits | Lines ± | Files Changed |
+|-------------|---------|---------|---------------|
+| vyzo | 295 | +8700/-5936 | 397 |
+| Anton Evangelatov | 94 | +4680/-2965 | 277 |
+| Łukasz Magiera | 37 | +3851/-1611 | 146 |
+| Mike Greenberg | 1 | +2310/-578 | 8 |
+| dirkmc | 7 | +1154/-726 | 29 |
+| Jennifer Wang | 9 | +485/-341 | 26 |
+| Peter Rabbitson | 18 | +469/-273 | 64 |
+| Cory Schwartz | 5 | +576/-135 | 14 |
+| hunjixin | 7 | +404/-82 | 19 |
+| ZenGround0 | 17 | +284/-135 | 44 |
+| Dirk McCormick | 17 | +348/-47 | 17 |
+| Raúl Kripalani | 18 | +254/-97 | 62 |
+| tchardin | 1 | +261/-33 | 4 |
+| Jakub Sztandera | 4 | +254/-16 | 4 |
+| Aarsh Shah | 2 | +196/-40 | 28 |
+| whyrusleeping | 3 | +150/-9 | 8 |
+| Whyrusleeping | 2 | +87/-66 | 10 |
+| Aayush Rajasekaran | 10 | +81/-53 | 13 |
+| zgfzgf | 2 | +104/-4 | 2 |
+| aarshkshah1992 | 4 | +73/-7 | 6 |
+| llifezou | 4 | +59/-20 | 4 |
+| Steven Allen | 7 | +47/-17 | 9 |
+| johnli-helloworld | 3 | +46/-15 | 5 |
+| frrist | 1 | +28/-23 | 2 |
+| Jennifer | 4 | +31/-2 | 4 |
+| wangchao | 1 | +1/-27 | 1 |
+| Jiaying Wang | 2 | +7/-21 | 2 |
+| hannahhoward | 3 | +21/-2 | 3 |
+| chadwick2143 | 1 | +15/-1 | 1 |
+| Jerry | 2 | +9/-4 | 2 |
+| Steve Loeppky | 2 | +12/-0 | 2 |
+| David Dias | 1 | +9/-0 | 1 |
+| dependabot[bot] | 1 | +3/-3 | 1 |
+| zhoutian527 | 1 | +2/-2 | 1 |
+| xloem | 1 | +4/-0 | 1 |
+| Travis Person | 2 | +2/-2 | 3 |
+| Liviu Damian | 2 | +2/-2 | 2 |
+| Jim Pick | 2 | +2/-2 | 2 |
+| Frank | 1 | +3/-0 | 1 |
+| turuslan | 1 | +1/-1 | 1 |
+| Kirk Baird | 1 | +0/-0 | 1 |
+
+
+
 # 1.11.0 / 2021-07-22
 
 This is a **highly recommended** release of Lotus that have many bug fixes, improvements and new features. 
@@ -253,6 +432,13 @@ Contributors
 | @zhoutian527 | 1 | +2/-2 | 1 |
 | @ribasushi| 1 | +1/-1 | 1 |
 
+
+<<<<<<< HEAD
+||||||| merged common ancestors
+>>>>>>>>> Temporary merge branch 2
+=======
+>>>>>>> releases
+>>>>>>> releases
 # 1.10.0 / 2021-06-23
 
 This is a mandatory release of Lotus that introduces Filecoin network v13, codenamed the HyperDrive upgrade. The

api/api_storage.go

@@ -166,6 +166,10 @@ type StorageMiner interface {
 	MarketPendingDeals(ctx context.Context) (PendingDealInfo, error)                                                             //perm:write
 	MarketPublishPendingDeals(ctx context.Context) error                                                                         //perm:admin
 
+	// RuntimeSubsystems returns the subsystems that are enabled
+	// in this instance.
+	RuntimeSubsystems(ctx context.Context) (MinerSubsystems, error) //perm:read
+
 	DealsImportData(ctx context.Context, dealPropCid cid.Cid, file string) error //perm:admin
 	DealsList(ctx context.Context) ([]MarketDeal, error)                         //perm:admin
 	DealsConsiderOnlineStorageDeals(context.Context) (bool, error)               //perm:admin

api/docgen/docgen.go

@@ -18,10 +18,10 @@ import (
 	"github.com/google/uuid"
 	"github.com/ipfs/go-cid"
 	"github.com/ipfs/go-filestore"
-	metrics "github.com/libp2p/go-libp2p-core/metrics"
+	"github.com/libp2p/go-libp2p-core/metrics"
 	"github.com/libp2p/go-libp2p-core/network"
 	"github.com/libp2p/go-libp2p-core/peer"
-	protocol "github.com/libp2p/go-libp2p-core/protocol"
+	"github.com/libp2p/go-libp2p-core/protocol"
 	pubsub "github.com/libp2p/go-libp2p-pubsub"
 	"github.com/multiformats/go-multiaddr"
 
@@ -47,11 +47,12 @@ import (
 )
 
 var ExampleValues = map[reflect.Type]interface{}{
-	reflect.TypeOf(auth.Permission("")): auth.Permission("write"),
+	reflect.TypeOf(api.MinerSubsystem(0)): api.MinerSubsystem(1),
-	reflect.TypeOf(""):                  "string value",
+	reflect.TypeOf(auth.Permission("")):   auth.Permission("write"),
-	reflect.TypeOf(uint64(42)):          uint64(42),
+	reflect.TypeOf(""):                    "string value",
-	reflect.TypeOf(byte(7)):             byte(7),
+	reflect.TypeOf(uint64(42)):            uint64(42),
-	reflect.TypeOf([]byte{}):            []byte("byte array"),
+	reflect.TypeOf(byte(7)):               byte(7),
+	reflect.TypeOf([]byte{}):              []byte("byte array"),
 }
 
 func addExample(v interface{}) {
@@ -268,6 +269,12 @@ func init() {
 
 	addExample(api.CheckStatusCode(0))
 	addExample(map[string]interface{}{"abc": 123})
+	addExample(api.MinerSubsystems{
+		api.SubsystemMining,
+		api.SubsystemSealing,
+		api.SubsystemSectorStorage,
+		api.SubsystemMarkets,
+	})
 }
 
 func GetAPIType(name, pkg string) (i interface{}, t reflect.Type, permStruct []reflect.Type) {

api/miner_subsystems.go

@@ -0,0 +1,79 @@
+package api
+
+import (
+	"encoding/json"
+)
+
+// MinerSubsystem represents a miner subsystem. Int and string values are not
+// guaranteed to be stable over time  is not
+// guaranteed to be stable over time.
+type MinerSubsystem int
+
+const (
+	// SubsystemUnknown is a placeholder for the zero value. It should never
+	// be used.
+	SubsystemUnknown MinerSubsystem = iota
+	// SubsystemMarkets signifies the storage and retrieval
+	// deal-making subsystem.
+	SubsystemMarkets
+	// SubsystemMining signifies the mining subsystem.
+	SubsystemMining
+	// SubsystemSealing signifies the sealing subsystem.
+	SubsystemSealing
+	// SubsystemSectorStorage signifies the sector storage subsystem.
+	SubsystemSectorStorage
+)
+
+var MinerSubsystemToString = map[MinerSubsystem]string{
+	SubsystemUnknown:       "Unknown",
+	SubsystemMarkets:       "Markets",
+	SubsystemMining:        "Mining",
+	SubsystemSealing:       "Sealing",
+	SubsystemSectorStorage: "SectorStorage",
+}
+
+var MinerSubsystemToID = map[string]MinerSubsystem{
+	"Unknown":       SubsystemUnknown,
+	"Markets":       SubsystemMarkets,
+	"Mining":        SubsystemMining,
+	"Sealing":       SubsystemSealing,
+	"SectorStorage": SubsystemSectorStorage,
+}
+
+func (ms MinerSubsystem) MarshalJSON() ([]byte, error) {
+	return json.Marshal(MinerSubsystemToString[ms])
+}
+
+func (ms *MinerSubsystem) UnmarshalJSON(b []byte) error {
+	var j string
+	err := json.Unmarshal(b, &j)
+	if err != nil {
+		return err
+	}
+	s, ok := MinerSubsystemToID[j]
+	if !ok {
+		*ms = SubsystemUnknown
+	} else {
+		*ms = s
+	}
+	return nil
+}
+
+type MinerSubsystems []MinerSubsystem
+
+func (ms MinerSubsystems) Has(entry MinerSubsystem) bool {
+	for _, v := range ms {
+		if v == entry {
+			return true
+		}
+	}
+	return false
+}
+
+func (ms MinerSubsystem) String() string {
+	s, ok := MinerSubsystemToString[ms]
+	if !ok {
+		return MinerSubsystemToString[SubsystemUnknown]
+	}
+	return s
+}

api/proxy_gen.go

@@ -699,6 +699,8 @@ type StorageMinerStruct struct {
 
 		ReturnUnsealPiece func(p0 context.Context, p1 storiface.CallID, p2 *storiface.CallError) error `perm:"admin"`
 
+		RuntimeSubsystems func(p0 context.Context) (MinerSubsystems, error) `perm:"read"`
+
 		SealingAbort func(p0 context.Context, p1 storiface.CallID) error `perm:"admin"`
 
 		SealingSchedDiag func(p0 context.Context, p1 bool) (interface{}, error) `perm:"admin"`
@@ -4095,6 +4097,17 @@ func (s *StorageMinerStub) ReturnUnsealPiece(p0 context.Context, p1 storiface.Ca
 	return ErrNotSupported
 }
 
+func (s *StorageMinerStruct) RuntimeSubsystems(p0 context.Context) (MinerSubsystems, error) {
+	if s.Internal.RuntimeSubsystems == nil {
+		return *new(MinerSubsystems), ErrNotSupported
+	}
+	return s.Internal.RuntimeSubsystems(p0)
+}
+
+func (s *StorageMinerStub) RuntimeSubsystems(p0 context.Context) (MinerSubsystems, error) {
+	return *new(MinerSubsystems), ErrNotSupported
+}
+
 func (s *StorageMinerStruct) SealingAbort(p0 context.Context, p1 storiface.CallID) error {
 	if s.Internal.SealingAbort == nil {
 		return ErrNotSupported

blockstore/badger/blockstore.go

@@ -258,16 +258,16 @@ func (b *Blockstore) movingGC() error {
 	b.moveCond.Broadcast()
 	b.moveMx.Unlock()
 
-	var path string
+	var newPath string
 
 	defer func() {
 		b.lockMove()
 
-		db2 := b.dbNext
+		dbNext := b.dbNext
 		b.dbNext = nil
 
 		var state bsMoveState
-		if db2 != nil {
+		if dbNext != nil {
 			state = moveStateCleanup
 		} else {
 			state = moveStateNone
@@ -275,12 +275,13 @@ func (b *Blockstore) movingGC() error {
 
 		b.unlockMove(state)
 
-		if db2 != nil {
+		if dbNext != nil {
-			err := db2.Close()
+			// the move failed and we have a left-over db; delete it.
+			err := dbNext.Close()
 			if err != nil {
 				log.Warnf("error closing badger db: %s", err)
 			}
-			b.deleteDB(path)
+			b.deleteDB(newPath)
 
 			b.lockMove()
 			b.unlockMove(moveStateNone)
@@ -296,68 +297,96 @@ func (b *Blockstore) movingGC() error {
 	}
 
 	if basePath == linkPath {
-		path = basePath
+		newPath = basePath
 	} else {
+		// we do this dance to create a name adjacent to the current one, while avoiding clown
+		// shoes with multiple moves (i.e. we can't just take the basename of the linkPath, as it
+		// could have been created in a previous move and have the timestamp suffix, which would then
+		// perpetuate itself.
 		name := filepath.Base(basePath)
 		dir := filepath.Dir(linkPath)
-		path = filepath.Join(dir, name)
+		newPath = filepath.Join(dir, name)
 	}
-	path = fmt.Sprintf("%s.%d", path, time.Now().UnixNano())
+	newPath = fmt.Sprintf("%s.%d", newPath, time.Now().UnixNano())
 
-	log.Infof("moving blockstore from %s to %s", b.opts.Dir, path)
+	log.Infof("moving blockstore from %s to %s", b.opts.Dir, newPath)
 
 	opts := b.opts
-	opts.Dir = path
+	opts.Dir = newPath
-	opts.ValueDir = path
+	opts.ValueDir = newPath
 
-	db2, err := badger.Open(opts.Options)
+	dbNew, err := badger.Open(opts.Options)
 	if err != nil {
-		return fmt.Errorf("failed to open badger blockstore in %s: %w", path, err)
+		return fmt.Errorf("failed to open badger blockstore in %s: %w", newPath, err)
 	}
 
 	b.lockMove()
-	b.dbNext = db2
+	b.dbNext = dbNew
 	b.unlockMove(moveStateMoving)
 
 	log.Info("copying blockstore")
 	err = b.doCopy(b.db, b.dbNext)
 	if err != nil {
-		return fmt.Errorf("error moving badger blockstore to %s: %w", path, err)
+		return fmt.Errorf("error moving badger blockstore to %s: %w", newPath, err)
 	}
 
 	b.lockMove()
-	db1 := b.db
+	dbOld := b.db
 	b.db = b.dbNext
 	b.dbNext = nil
 	b.unlockMove(moveStateCleanup)
 
-	err = db1.Close()
+	err = dbOld.Close()
 	if err != nil {
 		log.Warnf("error closing old badger db: %s", err)
 	}
 
-	dbpath := b.opts.Dir
+	// this is the canonical db path; this is where our db lives.
-	oldpath := fmt.Sprintf("%s.old.%d", dbpath, time.Now().Unix())
+	dbPath := b.opts.Dir
 
-	if err = os.Rename(dbpath, oldpath); err != nil {
+	// we first move the existing db out of the way, and only delete it after we have symlinked the
+	// new db to the canonical path
+	backupPath := fmt.Sprintf("%s.old.%d", dbPath, time.Now().Unix())
+	if err = os.Rename(dbPath, backupPath); err != nil {
 		// this is not catastrophic in the sense that we have not lost any data.
 		// but it is pretty bad, as the db path points to the old db, while we are now using to the new
 		// db; we can't continue and leave a ticking bomb for the next restart.
 		// so a panic is appropriate and user can fix.
-		panic(fmt.Errorf("error renaming old badger db dir from %s to %s: %w; USER ACTION REQUIRED", dbpath, oldpath, err)) //nolint
+		panic(fmt.Errorf("error renaming old badger db dir from %s to %s: %w; USER ACTION REQUIRED", dbPath, backupPath, err)) //nolint
 	}
 
-	if err = os.Symlink(path, dbpath); err != nil {
+	if err = symlink(newPath, dbPath); err != nil {
 		// same here; the db path is pointing to the void. panic and let the user fix.
-		panic(fmt.Errorf("error symlinking new badger db dir from %s to %s: %w; USER ACTION REQUIRED", path, dbpath, err)) //nolint
+		panic(fmt.Errorf("error symlinking new badger db dir from %s to %s: %w; USER ACTION REQUIRED", newPath, dbPath, err)) //nolint
 	}
 
-	b.deleteDB(oldpath)
+	// the delete follows symlinks
+	b.deleteDB(backupPath)
 
 	log.Info("moving blockstore done")
 	return nil
 }
 
+// symlink creates a symlink from path to linkTo; the link is relative if the two are
+// in the same directory
+func symlink(path, linkTo string) error {
+	resolvedPathDir, err := filepath.EvalSymlinks(filepath.Dir(path))
+	if err != nil {
+		return fmt.Errorf("error resolving links in %s: %w", path, err)
+	}
+
+	resolvedLinkDir, err := filepath.EvalSymlinks(filepath.Dir(linkTo))
+	if err != nil {
+		return fmt.Errorf("error resolving links in %s: %w", linkTo, err)
+	}
+
+	if resolvedPathDir == resolvedLinkDir {
+		path = filepath.Base(path)
+	}
+
+	return os.Symlink(path, linkTo)
+}
+
 // doCopy copies a badger blockstore to another, with an optional filter; if the filter
 // is not nil, then only cids that satisfy the filter will be copied.
 func (b *Blockstore) doCopy(from, to *badger.DB) error {
@@ -390,19 +419,19 @@ func (b *Blockstore) doCopy(from, to *badger.DB) error {
 
 func (b *Blockstore) deleteDB(path string) {
 	// follow symbolic links, otherwise the data wil be left behind
-	lpath, err := filepath.EvalSymlinks(path)
+	linkPath, err := filepath.EvalSymlinks(path)
 	if err != nil {
 		log.Warnf("error resolving symlinks in %s", path)
 		return
 	}
 
-	log.Infof("removing data directory %s", lpath)
+	log.Infof("removing data directory %s", linkPath)
-	if err := os.RemoveAll(lpath); err != nil {
+	if err := os.RemoveAll(linkPath); err != nil {
-		log.Warnf("error deleting db at %s: %s", lpath, err)
+		log.Warnf("error deleting db at %s: %s", linkPath, err)
 		return
 	}
 
-	if path != lpath {
+	if path != linkPath {
 		log.Infof("removing link %s", path)
 		if err := os.Remove(path); err != nil {
 			log.Warnf("error removing symbolic link %s", err)

blockstore/badger/blockstore_test.go

@@ -245,6 +245,21 @@ func testMove(t *testing.T, optsF func(string) Options) {
 
 	checkBlocks()
 	checkPath()
+
+	// reopen the db to make sure our relative link works:
+	err = db.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	db, err = Open(optsF(dbPath))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// db.Close() is already deferred
+
+	checkBlocks()
 }
 
 func TestMoveNoPrefix(t *testing.T) {

build/version.go

@@ -34,7 +34,7 @@ func buildType() string {
 }
 
 // BuildVersion is the local build version, set by build system
-const BuildVersion = "1.11.1-dev"
+const BuildVersion = "1.11.1-rc1"
 
 func UserVersion() string {
 	if os.Getenv("LOTUS_VERSION_IGNORE_COMMIT") == "1" {

cli/auth.go

@@ -113,7 +113,7 @@ var AuthApiInfoToken = &cli.Command{
 
 		ti, ok := cctx.App.Metadata["repoType"]
 		if !ok {
-			log.Errorf("unknown repo type, are you sure you want to use GetAPI?")
+			log.Errorf("unknown repo type, are you sure you want to use GetCommonAPI?")
 			ti = repo.FullNode
 		}
 		t, ok := ti.(repo.RepoType)
@@ -128,7 +128,8 @@ var AuthApiInfoToken = &cli.Command{
 
 		// TODO: Log in audit log when it is implemented
 
-		fmt.Printf("%s=%s:%s\n", cliutil.EnvForRepo(t), string(token), ainfo.Addr)
+		currentEnv, _, _ := cliutil.EnvsForAPIInfos(t)
+		fmt.Printf("%s=%s:%s\n", currentEnv, string(token), ainfo.Addr)
 		return nil
 	},
 }

cli/cmd.go

@@ -44,7 +44,7 @@ func GetFullNodeServices(ctx *cli.Context) (ServicesAPI, error) {
 
 var GetAPIInfo = cliutil.GetAPIInfo
 var GetRawAPI = cliutil.GetRawAPI
-var GetAPI = cliutil.GetAPI
+var GetAPI = cliutil.GetCommonAPI
 
 var DaemonContext = cliutil.DaemonContext
 var ReqContext = cliutil.ReqContext
@@ -54,6 +54,7 @@ var GetFullNodeAPIV1 = cliutil.GetFullNodeAPIV1
 var GetGatewayAPI = cliutil.GetGatewayAPI
 
 var GetStorageMinerAPI = cliutil.GetStorageMinerAPI
+var GetMarketsAPI = cliutil.GetMarketsAPI
 var GetWorkerAPI = cliutil.GetWorkerAPI
 
 var CommonCommands = []*cli.Command{

cli/util/api.go

@@ -27,112 +27,145 @@ const (
 	metadataTraceContext = "traceContext"
 )
 
-// The flag passed on the command line with the listen address of the API
+// flagsForAPI returns flags passed on the command line with the listen address
-// server (only used by the tests)
+// of the API server (only used by the tests), in the order of precedence they
-func flagForAPI(t repo.RepoType) string {
+// should be applied for the requested kind of node.
+func flagsForAPI(t repo.RepoType) []string {
 	switch t {
 	case repo.FullNode:
-		return "api-url"
+		return []string{"api-url"}
 	case repo.StorageMiner:
-		return "miner-api-url"
+		return []string{"miner-api-url"}
 	case repo.Worker:
-		return "worker-api-url"
+		return []string{"worker-api-url"}
+	case repo.Markets:
+		// support split markets-miner and monolith deployments.
+		return []string{"markets-api-url", "miner-api-url"}
 	default:
 		panic(fmt.Sprintf("Unknown repo type: %v", t))
 	}
 }
 
-func flagForRepo(t repo.RepoType) string {
+func flagsForRepo(t repo.RepoType) []string {
 	switch t {
 	case repo.FullNode:
-		return "repo"
+		return []string{"repo"}
 	case repo.StorageMiner:
-		return "miner-repo"
+		return []string{"miner-repo"}
 	case repo.Worker:
-		return "worker-repo"
+		return []string{"worker-repo"}
+	case repo.Markets:
+		// support split markets-miner and monolith deployments.
+		return []string{"markets-repo", "miner-repo"}
 	default:
 		panic(fmt.Sprintf("Unknown repo type: %v", t))
 	}
 }
 
-func EnvForRepo(t repo.RepoType) string {
+// EnvsForAPIInfos returns the environment variables to use in order of precedence
+// to determine the API endpoint of the specified node type.
+//
+// It returns the current variables and deprecated ones separately, so that
+// the user can log a warning when deprecated ones are found to be in use.
+func EnvsForAPIInfos(t repo.RepoType) (primary string, fallbacks []string, deprecated []string) {
 	switch t {
 	case repo.FullNode:
-		return "FULLNODE_API_INFO"
+		return "FULLNODE_API_INFO", nil, nil
 	case repo.StorageMiner:
-		return "MINER_API_INFO"
+		// TODO remove deprecated deprecation period
+		return "MINER_API_INFO", nil, []string{"STORAGE_API_INFO"}
 	case repo.Worker:
-		return "WORKER_API_INFO"
+		return "WORKER_API_INFO", nil, nil
-	default:
+	case repo.Markets:
-		panic(fmt.Sprintf("Unknown repo type: %v", t))
+		// support split markets-miner and monolith deployments.
-	}
+		return "MARKETS_API_INFO", []string{"MINER_API_INFO"}, nil
-}
-
-// TODO remove after deprecation period
-func envForRepoDeprecation(t repo.RepoType) string {
-	switch t {
-	case repo.FullNode:
-		return "FULLNODE_API_INFO"
-	case repo.StorageMiner:
-		return "STORAGE_API_INFO"
-	case repo.Worker:
-		return "WORKER_API_INFO"
 	default:
 		panic(fmt.Sprintf("Unknown repo type: %v", t))
 	}
 }
 
+// GetAPIInfo returns the API endpoint to use for the specified kind of repo.
+//
+// The order of precedence is as follows:
+//
+//  1. *-api-url command line flags.
+//  2. *_API_INFO environment variables
+//  3. deprecated *_API_INFO environment variables
+//  4. *-repo command line flags.
 func GetAPIInfo(ctx *cli.Context, t repo.RepoType) (APIInfo, error) {
 	// Check if there was a flag passed with the listen address of the API
 	// server (only used by the tests)
-	apiFlag := flagForAPI(t)
+	apiFlags := flagsForAPI(t)
-	if ctx.IsSet(apiFlag) {
+	for _, f := range apiFlags {
-		strma := ctx.String(apiFlag)
+		if !ctx.IsSet(f) {
+			continue
+		}
+		strma := ctx.String(f)
 		strma = strings.TrimSpace(strma)
 
 		return APIInfo{Addr: strma}, nil
 	}
 
-	envKey := EnvForRepo(t)
+	//
-	env, ok := os.LookupEnv(envKey)
+	// Note: it is not correct/intuitive to prefer environment variables over
-	if !ok {
+	// CLI flags (repo flags below).
-		// TODO remove after deprecation period
+	//
-		envKey = envForRepoDeprecation(t)
+	primaryEnv, fallbacksEnvs, deprecatedEnvs := EnvsForAPIInfos(t)
-		env, ok = os.LookupEnv(envKey)
+	env, ok := os.LookupEnv(primaryEnv)
-		if ok {
-			log.Warnf("Use deprecation env(%s) value, please use env(%s) instead.", envKey, EnvForRepo(t))
-		}
-	}
 	if ok {
 		return ParseApiInfo(env), nil
 	}
 
-	repoFlag := flagForRepo(t)
+	for _, env := range deprecatedEnvs {
-
+		env, ok := os.LookupEnv(env)
-	p, err := homedir.Expand(ctx.String(repoFlag))
+		if ok {
-	if err != nil {
+			log.Warnf("Using deprecated env(%s) value, please use env(%s) instead.", env, primaryEnv)
-		return APIInfo{}, xerrors.Errorf("could not expand home dir (%s): %w", repoFlag, err)
+			return ParseApiInfo(env), nil
+		}
 	}
 
-	r, err := repo.NewFS(p)
+	repoFlags := flagsForRepo(t)
-	if err != nil {
+	for _, f := range repoFlags {
-		return APIInfo{}, xerrors.Errorf("could not open repo at path: %s; %w", p, err)
+		// cannot use ctx.IsSet because it ignores default values
+		path := ctx.String(f)
+		if path == "" {
+			continue
+		}
+
+		p, err := homedir.Expand(path)
+		if err != nil {
+			return APIInfo{}, xerrors.Errorf("could not expand home dir (%s): %w", f, err)
+		}
+
+		r, err := repo.NewFS(p)
+		if err != nil {
+			return APIInfo{}, xerrors.Errorf("could not open repo at path: %s; %w", p, err)
+		}
+
+		ma, err := r.APIEndpoint()
+		if err != nil {
+			return APIInfo{}, xerrors.Errorf("could not get api endpoint: %w", err)
+		}
+
+		token, err := r.APIToken()
+		if err != nil {
+			log.Warnf("Couldn't load CLI token, capabilities may be limited: %v", err)
+		}
+
+		return APIInfo{
+			Addr:  ma.String(),
+			Token: token,
+		}, nil
 	}
 
-	ma, err := r.APIEndpoint()
+	for _, env := range fallbacksEnvs {
-	if err != nil {
+		env, ok := os.LookupEnv(env)
-		return APIInfo{}, xerrors.Errorf("could not get api endpoint: %w", err)
+		if ok {
+			return ParseApiInfo(env), nil
+		}
 	}
 
-	token, err := r.APIToken()
+	return APIInfo{}, fmt.Errorf("could not determine API endpoint for node type: %v", t)
-	if err != nil {
-		log.Warnf("Couldn't load CLI token, capabilities may be limited: %v", err)
-	}
-
-	return APIInfo{
-		Addr:  ma.String(),
-		Token: token,
-	}, nil
 }
 
 func GetRawAPI(ctx *cli.Context, t repo.RepoType, version string) (string, http.Header, error) {
@@ -153,10 +186,10 @@ func GetRawAPI(ctx *cli.Context, t repo.RepoType, version string) (string, http.
 	return addr, ainfo.AuthHeader(), nil
 }
 
-func GetAPI(ctx *cli.Context) (api.CommonNet, jsonrpc.ClientCloser, error) {
+func GetCommonAPI(ctx *cli.Context) (api.CommonNet, jsonrpc.ClientCloser, error) {
 	ti, ok := ctx.App.Metadata["repoType"]
 	if !ok {
-		log.Errorf("unknown repo type, are you sure you want to use GetAPI?")
+		log.Errorf("unknown repo type, are you sure you want to use GetCommonAPI?")
 		ti = repo.FullNode
 	}
 	t, ok := ti.(repo.RepoType)
@@ -274,6 +307,27 @@ func GetWorkerAPI(ctx *cli.Context) (api.Worker, jsonrpc.ClientCloser, error) {
 	return client.NewWorkerRPCV0(ctx.Context, addr, headers)
 }
 
+func GetMarketsAPI(ctx *cli.Context) (api.StorageMiner, jsonrpc.ClientCloser, error) {
+	// to support lotus-miner cli tests.
+	if tn, ok := ctx.App.Metadata["testnode-storage"]; ok {
+		return tn.(api.StorageMiner), func() {}, nil
+	}
+
+	addr, headers, err := GetRawAPI(ctx, repo.Markets, "v0")
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if IsVeryVerbose {
+		_, _ = fmt.Fprintln(ctx.App.Writer, "using markets API v0 endpoint:", addr)
+	}
+
+	// the markets node is a specialised miner's node, supporting only the
+	// markets API, which is a subset of the miner API. All non-markets
+	// operations will error out with "unsupported".
+	return client.NewStorageMinerRPCV0(ctx.Context, addr, headers)
+}
+
 func GetGatewayAPI(ctx *cli.Context) (api.Gateway, jsonrpc.ClientCloser, error) {
 	addr, headers, err := GetRawAPI(ctx, repo.FullNode, "v1")
 	if err != nil {

cmd/lotus-miner/info.go

@@ -21,6 +21,7 @@ import (
 	"github.com/filecoin-project/go-fil-markets/storagemarket"
 	"github.com/filecoin-project/go-state-types/abi"
 	"github.com/filecoin-project/go-state-types/big"
+	"github.com/filecoin-project/lotus/api/v0api"
 	sealing "github.com/filecoin-project/lotus/extern/storage-sealing"
 
 	"github.com/filecoin-project/lotus/api"
@@ -49,13 +50,19 @@ var infoCmd = &cli.Command{
 }
 
 func infoCmdAct(cctx *cli.Context) error {
-	nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
+	minerApi, closer, err := lcli.GetStorageMinerAPI(cctx)
 	if err != nil {
 		return err
 	}
 	defer closer()
 
-	api, acloser, err := lcli.GetFullNodeAPI(cctx)
+	marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
+	if err != nil {
+		return err
+	}
+	defer closer()
+
+	fullapi, acloser, err := lcli.GetFullNodeAPI(cctx)
 	if err != nil {
 		return err
 	}
@@ -63,9 +70,23 @@ func infoCmdAct(cctx *cli.Context) error {
 
 	ctx := lcli.ReqContext(cctx)
 
+	subsystems, err := minerApi.RuntimeSubsystems(ctx)
+	if err != nil {
+		return err
+	}
+
+	fmt.Println("Enabled subsystems (from miner API):", subsystems)
+
+	subsystems, err = marketsApi.RuntimeSubsystems(ctx)
+	if err != nil {
+		return err
+	}
+
+	fmt.Println("Enabled subsystems (from markets API):", subsystems)
+
 	fmt.Print("Chain: ")
 
-	head, err := api.ChainHead(ctx)
+	head, err := fullapi.ChainHead(ctx)
 	if err != nil {
 		return err
 	}
@@ -95,24 +116,38 @@ func infoCmdAct(cctx *cli.Context) error {
 
 	fmt.Println()
 
+	err = handleMiningInfo(ctx, cctx, fullapi, minerApi)
+	if err != nil {
+		return err
+	}
+
+	err = handleMarketsInfo(ctx, marketsApi)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func handleMiningInfo(ctx context.Context, cctx *cli.Context, fullapi v0api.FullNode, nodeApi api.StorageMiner) error {
 	maddr, err := getActorAddress(ctx, cctx)
 	if err != nil {
 		return err
 	}
 
-	mact, err := api.StateGetActor(ctx, maddr, types.EmptyTSK)
+	mact, err := fullapi.StateGetActor(ctx, maddr, types.EmptyTSK)
 	if err != nil {
 		return err
 	}
 
-	tbs := blockstore.NewTieredBstore(blockstore.NewAPIBlockstore(api), blockstore.NewMemory())
+	tbs := blockstore.NewTieredBstore(blockstore.NewAPIBlockstore(fullapi), blockstore.NewMemory())
 	mas, err := miner.Load(adt.WrapStore(ctx, cbor.NewCborStore(tbs)), mact)
 	if err != nil {
 		return err
 	}
 
 	// Sector size
-	mi, err := api.StateMinerInfo(ctx, maddr, types.EmptyTSK)
+	mi, err := fullapi.StateMinerInfo(ctx, maddr, types.EmptyTSK)
 	if err != nil {
 		return err
 	}
@@ -120,7 +155,7 @@ func infoCmdAct(cctx *cli.Context) error {
 	ssize := types.SizeStr(types.NewInt(uint64(mi.SectorSize)))
 	fmt.Printf("Miner: %s (%s sectors)\n", color.BlueString("%s", maddr), ssize)
 
-	pow, err := api.StateMinerPower(ctx, maddr, types.EmptyTSK)
+	pow, err := fullapi.StateMinerPower(ctx, maddr, types.EmptyTSK)
 	if err != nil {
 		return err
 	}
@@ -142,7 +177,7 @@ func infoCmdAct(cctx *cli.Context) error {
 			pow.TotalPower.RawBytePower,
 		),
 	)
-	secCounts, err := api.StateMinerSectorCount(ctx, maddr, types.EmptyTSK)
+	secCounts, err := fullapi.StateMinerSectorCount(ctx, maddr, types.EmptyTSK)
 	if err != nil {
 		return err
 	}
@@ -219,6 +254,75 @@ func infoCmdAct(cctx *cli.Context) error {
 
 	fmt.Println()
 
+	spendable := big.Zero()
+
+	// NOTE: there's no need to unlock anything here. Funds only
+	// vest on deadline boundaries, and they're unlocked by cron.
+	lockedFunds, err := mas.LockedFunds()
+	if err != nil {
+		return xerrors.Errorf("getting locked funds: %w", err)
+	}
+	availBalance, err := mas.AvailableBalance(mact.Balance)
+	if err != nil {
+		return xerrors.Errorf("getting available balance: %w", err)
+	}
+	spendable = big.Add(spendable, availBalance)
+
+	fmt.Printf("Miner Balance:    %s\n", color.YellowString("%s", types.FIL(mact.Balance).Short()))
+	fmt.Printf("      PreCommit:  %s\n", types.FIL(lockedFunds.PreCommitDeposits).Short())
+	fmt.Printf("      Pledge:     %s\n", types.FIL(lockedFunds.InitialPledgeRequirement).Short())
+	fmt.Printf("      Vesting:    %s\n", types.FIL(lockedFunds.VestingFunds).Short())
+	colorTokenAmount("      Available:  %s\n", availBalance)
+
+	mb, err := fullapi.StateMarketBalance(ctx, maddr, types.EmptyTSK)
+	if err != nil {
+		return xerrors.Errorf("getting market balance: %w", err)
+	}
+	spendable = big.Add(spendable, big.Sub(mb.Escrow, mb.Locked))
+
+	fmt.Printf("Market Balance:   %s\n", types.FIL(mb.Escrow).Short())
+	fmt.Printf("       Locked:    %s\n", types.FIL(mb.Locked).Short())
+	colorTokenAmount("       Available: %s\n", big.Sub(mb.Escrow, mb.Locked))
+
+	wb, err := fullapi.WalletBalance(ctx, mi.Worker)
+	if err != nil {
+		return xerrors.Errorf("getting worker balance: %w", err)
+	}
+	spendable = big.Add(spendable, wb)
+	color.Cyan("Worker Balance:   %s", types.FIL(wb).Short())
+	if len(mi.ControlAddresses) > 0 {
+		cbsum := big.Zero()
+		for _, ca := range mi.ControlAddresses {
+			b, err := fullapi.WalletBalance(ctx, ca)
+			if err != nil {
+				return xerrors.Errorf("getting control address balance: %w", err)
+			}
+			cbsum = big.Add(cbsum, b)
+		}
+		spendable = big.Add(spendable, cbsum)
+
+		fmt.Printf("       Control:   %s\n", types.FIL(cbsum).Short())
+	}
+	colorTokenAmount("Total Spendable:  %s\n", spendable)
+
+	fmt.Println()
+
+	if !cctx.Bool("hide-sectors-info") {
+		fmt.Println("Sectors:")
+		err = sectorsInfo(ctx, nodeApi)
+		if err != nil {
+			return err
+		}
+	}
+
+	// TODO: grab actr state / info
+	//  * Sealed sectors (count / bytes)
+	//  * Power
+
+	return nil
+}
+
+func handleMarketsInfo(ctx context.Context, nodeApi api.StorageMiner) error {
 	deals, err := nodeApi.MarketListIncompleteDeals(ctx)
 	if err != nil {
 		return err
@@ -282,6 +386,7 @@ func infoCmdAct(cctx *cli.Context) error {
 		return sorted[i].status > sorted[j].status
 	})
 
+	fmt.Println()
 	fmt.Printf("Storage Deals: %d, %s\n", total.count, types.SizeStr(types.NewInt(total.bytes)))
 
 	tw := tabwriter.NewWriter(os.Stdout, 1, 1, 1, ' ', 0)
@@ -309,70 +414,6 @@ func infoCmdAct(cctx *cli.Context) error {
 
 	fmt.Println()
 
-	spendable := big.Zero()
-
-	// NOTE: there's no need to unlock anything here. Funds only
-	// vest on deadline boundaries, and they're unlocked by cron.
-	lockedFunds, err := mas.LockedFunds()
-	if err != nil {
-		return xerrors.Errorf("getting locked funds: %w", err)
-	}
-	availBalance, err := mas.AvailableBalance(mact.Balance)
-	if err != nil {
-		return xerrors.Errorf("getting available balance: %w", err)
-	}
-	spendable = big.Add(spendable, availBalance)
-
-	fmt.Printf("Miner Balance:    %s\n", color.YellowString("%s", types.FIL(mact.Balance).Short()))
-	fmt.Printf("      PreCommit:  %s\n", types.FIL(lockedFunds.PreCommitDeposits).Short())
-	fmt.Printf("      Pledge:     %s\n", types.FIL(lockedFunds.InitialPledgeRequirement).Short())
-	fmt.Printf("      Vesting:    %s\n", types.FIL(lockedFunds.VestingFunds).Short())
-	colorTokenAmount("      Available:  %s\n", availBalance)
-
-	mb, err := api.StateMarketBalance(ctx, maddr, types.EmptyTSK)
-	if err != nil {
-		return xerrors.Errorf("getting market balance: %w", err)
-	}
-	spendable = big.Add(spendable, big.Sub(mb.Escrow, mb.Locked))
-
-	fmt.Printf("Market Balance:   %s\n", types.FIL(mb.Escrow).Short())
-	fmt.Printf("       Locked:    %s\n", types.FIL(mb.Locked).Short())
-	colorTokenAmount("       Available: %s\n", big.Sub(mb.Escrow, mb.Locked))
-
-	wb, err := api.WalletBalance(ctx, mi.Worker)
-	if err != nil {
-		return xerrors.Errorf("getting worker balance: %w", err)
-	}
-	spendable = big.Add(spendable, wb)
-	color.Cyan("Worker Balance:   %s", types.FIL(wb).Short())
-	if len(mi.ControlAddresses) > 0 {
-		cbsum := big.Zero()
-		for _, ca := range mi.ControlAddresses {
-			b, err := api.WalletBalance(ctx, ca)
-			if err != nil {
-				return xerrors.Errorf("getting control address balance: %w", err)
-			}
-			cbsum = big.Add(cbsum, b)
-		}
-		spendable = big.Add(spendable, cbsum)
-
-		fmt.Printf("       Control:   %s\n", types.FIL(cbsum).Short())
-	}
-	colorTokenAmount("Total Spendable:  %s\n", spendable)
-
-	fmt.Println()
-
-	if !cctx.Bool("hide-sectors-info") {
-		fmt.Println("Sectors:")
-		err = sectorsInfo(ctx, nodeApi)
-		if err != nil {
-			return err
-		}
-	}
-
-	// TODO: grab actr state / info
-	//  * Sealed sectors (count / bytes)
-	//  * Power
 	return nil
 }
 

cmd/lotus-miner/init_restore.go

@@ -17,7 +17,7 @@ import (
 	"gopkg.in/cheggaaa/pb.v1"
 
 	"github.com/filecoin-project/go-address"
-	paramfetch "github.com/filecoin-project/go-paramfetch"
+	"github.com/filecoin-project/go-paramfetch"
 	"github.com/filecoin-project/go-state-types/big"
 
 	lapi "github.com/filecoin-project/lotus/api"
@@ -72,7 +72,9 @@ var restoreCmd = &cli.Command{
 			}
 		}
 
-		if err := restore(ctx, cctx, storageCfg, nil, func(api lapi.FullNode, maddr address.Address, peerid peer.ID, mi miner.MinerInfo) error {
+		repoPath := cctx.String(FlagMinerRepo)
+
+		if err := restore(ctx, cctx, repoPath, storageCfg, nil, func(api lapi.FullNode, maddr address.Address, peerid peer.ID, mi miner.MinerInfo) error {
 			log.Info("Checking proof parameters")
 
 			if err := paramfetch.GetParams(ctx, build.ParametersJSON(), build.SrsJSON(), uint64(mi.SectorSize)); err != nil {
@@ -94,7 +96,7 @@ var restoreCmd = &cli.Command{
 	},
 }
 
-func restore(ctx context.Context, cctx *cli.Context, strConfig *stores.StorageConfig, manageConfig func(*config.StorageMiner) error, after func(api lapi.FullNode, addr address.Address, peerid peer.ID, mi miner.MinerInfo) error) error {
+func restore(ctx context.Context, cctx *cli.Context, targetPath string, strConfig *stores.StorageConfig, manageConfig func(*config.StorageMiner) error, after func(api lapi.FullNode, addr address.Address, peerid peer.ID, mi miner.MinerInfo) error) error {
 	if cctx.Args().Len() != 1 {
 		return xerrors.Errorf("expected 1 argument")
 	}
@@ -142,8 +144,7 @@ func restore(ctx context.Context, cctx *cli.Context, strConfig *stores.StorageCo
 
 	log.Info("Checking if repo exists")
 
-	repoPath := cctx.String(FlagMinerRepo)
+	r, err := repo.NewFS(targetPath)
-	r, err := repo.NewFS(repoPath)
 	if err != nil {
 		return err
 	}

cmd/lotus-miner/init_service.go

@@ -71,7 +71,12 @@ var serviceCmd = &cli.Command{
 			return xerrors.Errorf("--api-sector-index is required without the sector storage module enabled")
 		}
 
-		if err := restore(ctx, cctx, &stores.StorageConfig{}, func(cfg *config.StorageMiner) error {
+		repoPath := cctx.String(FlagMarketsRepo)
+		if repoPath == "" {
+			return xerrors.Errorf("please provide Lotus markets repo path via flag %s", FlagMarketsRepo)
+		}
+
+		if err := restore(ctx, cctx, repoPath, &stores.StorageConfig{}, func(cfg *config.StorageMiner) error {
 			cfg.Subsystems.EnableMarkets = es.Contains(MarketsService)
 			cfg.Subsystems.EnableMining = false
 			cfg.Subsystems.EnableSealing = false

cmd/lotus-miner/main.go

@@ -22,7 +22,10 @@ import (
 
 var log = logging.Logger("main")
 
-const FlagMinerRepo = "miner-repo"
+const (
+	FlagMinerRepo   = "miner-repo"
+	FlagMarketsRepo = "markets-repo"
+)
 
 // TODO remove after deprecation period
 const FlagMinerRepoDeprecation = "storagerepo"
@@ -106,14 +109,29 @@ func main() {
 				Value:   "~/.lotusminer", // TODO: Consider XDG_DATA_HOME
 				Usage:   fmt.Sprintf("Specify miner repo path. flag(%s) and env(LOTUS_STORAGE_PATH) are DEPRECATION, will REMOVE SOON", FlagMinerRepoDeprecation),
 			},
+			&cli.StringFlag{
+				Name:    FlagMarketsRepo,
+				EnvVars: []string{"LOTUS_MARKETS_PATH"},
+				Usage:   fmt.Sprintf("Markets repo path"),
+			},
+			&cli.BoolFlag{
+				Name:  "call-on-markets",
+				Usage: "(experimental; may be removed) call this command against a markets node; use only with common commands like net, auth, pprof, etc. whose target may be ambiguous",
+			},
 			cliutil.FlagVeryVerbose,
 		},
-
 		Commands: append(local, lcli.CommonCommands...),
+		Before: func(c *cli.Context) error {
+			// this command is explicitly called on markets, inform
+			// common commands by overriding the repoType.
+			if c.Bool("call-on-markets") {
+				c.App.Metadata["repoType"] = repo.Markets
+			}
+			return nil
+		},
 	}
 	app.Setup()
 	app.Metadata["repoType"] = repo.StorageMiner
-
 	lcli.RunApp(app)
 }
 

cmd/lotus-miner/market.go

@@ -73,7 +73,7 @@ var storageDealSelectionShowCmd = &cli.Command{
 	Name:  "list",
 	Usage: "List storage deal proposal selection criteria",
 	Action: func(cctx *cli.Context) error {
-		smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		smapi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -100,7 +100,7 @@ var storageDealSelectionResetCmd = &cli.Command{
 	Name:  "reset",
 	Usage: "Reset storage deal proposal selection criteria to default values",
 	Action: func(cctx *cli.Context) error {
-		smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		smapi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -148,7 +148,7 @@ var storageDealSelectionRejectCmd = &cli.Command{
 		},
 	},
 	Action: func(cctx *cli.Context) error {
-		smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		smapi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -215,7 +215,13 @@ var setAskCmd = &cli.Command{
 	Action: func(cctx *cli.Context) error {
 		ctx := lcli.DaemonContext(cctx)
 
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		minerApi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		if err != nil {
+			return err
+		}
+		defer closer()
+
+		marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -252,12 +258,12 @@ var setAskCmd = &cli.Command{
 			return xerrors.Errorf("cannot parse max-piece-size to quantity of bytes: %w", err)
 		}
 
-		maddr, err := api.ActorAddress(ctx)
+		maddr, err := minerApi.ActorAddress(ctx)
 		if err != nil {
 			return err
 		}
 
-		ssize, err := api.ActorSectorSize(ctx, maddr)
+		ssize, err := minerApi.ActorSectorSize(ctx, maddr)
 		if err != nil {
 			return err
 		}
@@ -272,7 +278,7 @@ var setAskCmd = &cli.Command{
 			return xerrors.Errorf("max piece size (w/bit-padding) %s cannot exceed miner sector size %s", types.SizeStr(types.NewInt(uint64(max))), types.SizeStr(types.NewInt(uint64(smax))))
 		}
 
-		return api.MarketSetAsk(ctx, types.BigInt(pri), types.BigInt(vpri), abi.ChainEpoch(qty), abi.PaddedPieceSize(min), abi.PaddedPieceSize(max))
+		return marketsApi.MarketSetAsk(ctx, types.BigInt(pri), types.BigInt(vpri), abi.ChainEpoch(qty), abi.PaddedPieceSize(min), abi.PaddedPieceSize(max))
 	},
 }
 
@@ -289,7 +295,7 @@ var getAskCmd = &cli.Command{
 		}
 		defer closer()
 
-		smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		smapi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -352,7 +358,7 @@ var dealsImportDataCmd = &cli.Command{
 	Usage:     "Manually import data for a deal",
 	ArgsUsage: "<proposal CID> <file>",
 	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -390,7 +396,7 @@ var dealsListCmd = &cli.Command{
 		},
 	},
 	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -494,7 +500,7 @@ var getBlocklistCmd = &cli.Command{
 		&CidBaseFlag,
 	},
 	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -524,7 +530,7 @@ var setBlocklistCmd = &cli.Command{
 	ArgsUsage: "[<path-of-file-containing-newline-delimited-piece-CIDs> (optional, will read from stdin if omitted)]",
 	Flags:     []cli.Flag{},
 	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -570,7 +576,7 @@ var resetBlocklistCmd = &cli.Command{
 	Usage: "Remove all entries from the miner's piece CID blocklist",
 	Flags: []cli.Flag{},
 	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -634,7 +640,7 @@ var marketRestartTransfer = &cli.Command{
 		if !cctx.Args().Present() {
 			return cli.ShowCommandHelp(cctx, cctx.Command.Name)
 		}
-		nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -699,7 +705,7 @@ var marketCancelTransfer = &cli.Command{
 		if !cctx.Args().Present() {
 			return cli.ShowCommandHelp(cctx, cctx.Command.Name)
 		}
-		nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -775,7 +781,7 @@ var transfersListCmd = &cli.Command{
 			color.NoColor = !cctx.Bool("color")
 		}
 
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -842,7 +848,7 @@ var dealsPendingPublish = &cli.Command{
 		},
 	},
 	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}

cmd/lotus-miner/retrieval-deals.go

@@ -39,7 +39,7 @@ var retrievalDealSelectionShowCmd = &cli.Command{
 	Name:  "list",
 	Usage: "List retrieval deal proposal selection criteria",
 	Action: func(cctx *cli.Context) error {
-		smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		smapi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -66,7 +66,7 @@ var retrievalDealSelectionResetCmd = &cli.Command{
 	Name:  "reset",
 	Usage: "Reset retrieval deal proposal selection criteria to default values",
 	Action: func(cctx *cli.Context) error {
-		smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		smapi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -98,7 +98,7 @@ var retrievalDealSelectionRejectCmd = &cli.Command{
 		},
 	},
 	Action: func(cctx *cli.Context) error {
-		smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
+		smapi, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -126,7 +126,7 @@ var retrievalDealsListCmd = &cli.Command{
 	Name:  "list",
 	Usage: "List all active retrieval deals for this miner",
 	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -186,7 +186,7 @@ var retrievalSetAskCmd = &cli.Command{
 	Action: func(cctx *cli.Context) error {
 		ctx := lcli.DaemonContext(cctx)
 
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -240,7 +240,7 @@ var retrievalGetAskCmd = &cli.Command{
 	Action: func(cctx *cli.Context) error {
 		ctx := lcli.DaemonContext(cctx)
 
-		api, closer, err := lcli.GetStorageMinerAPI(cctx)
+		api, closer, err := lcli.GetMarketsAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -252,13 +252,13 @@ var retrievalGetAskCmd = &cli.Command{
 		}
 
 		w := tabwriter.NewWriter(os.Stdout, 2, 4, 2, ' ', 0)
-		fmt.Fprintf(w, "Price per Byte\tUnseal Price\tPayment Interval\tPayment Interval Increase\n")
+		_, _ = fmt.Fprintf(w, "Price per Byte\tUnseal Price\tPayment Interval\tPayment Interval Increase\n")
 		if ask == nil {
-			fmt.Fprintf(w, "<miner does not have an retrieval ask set>\n")
+			_, _ = fmt.Fprintf(w, "<miner does not have an retrieval ask set>\n")
 			return w.Flush()
 		}
 
-		fmt.Fprintf(w, "%s\t%s\t%s\t%s\n",
+		_, _ = fmt.Fprintf(w, "%s\t%s\t%s\t%s\n",
 			types.FIL(ask.PricePerByte),
 			types.FIL(ask.UnsealPrice),
 			units.BytesSize(float64(ask.PaymentInterval)),

documentation/en/api-v0-methods-miner.md

@@ -94,6 +94,8 @@
   * [ReturnSealPreCommit1](#ReturnSealPreCommit1)
   * [ReturnSealPreCommit2](#ReturnSealPreCommit2)
   * [ReturnUnsealPiece](#ReturnUnsealPiece)
+* [Runtime](#Runtime)
+  * [RuntimeSubsystems](#RuntimeSubsystems)
 * [Sealing](#Sealing)
   * [SealingAbort](#SealingAbort)
   * [SealingSchedDiag](#SealingSchedDiag)
@@ -1522,6 +1524,28 @@ Inputs:
 
 Response: `{}`
 
+## Runtime
+
+
+### RuntimeSubsystems
+RuntimeSubsystems returns the subsystems that are enabled
+in this instance.
+
+
+Perms: read
+
+Inputs: `null`
+
+Response:
+```json
+[
+  "Mining",
+  "Sealing",
+  "SectorStorage",
+  "Markets"
+]
+```
+
 ## Sealing
 
 

documentation/en/cli-lotus-miner.md

@@ -7,7 +7,7 @@ USAGE:
    lotus-miner [global options] command [command options] [arguments...]
 
 VERSION:
-   1.11.1-dev
+   1.11.1-rc1
 
 COMMANDS:
    init     Initialize a lotus miner repo
@@ -43,6 +43,8 @@ GLOBAL OPTIONS:
    --actor value, -a value                  specify other actor to check state for (read only)
    --color                                  use color in display output (default: depends on output being a TTY)
    --miner-repo value, --storagerepo value  Specify miner repo path. flag(storagerepo) and env(LOTUS_STORAGE_PATH) are DEPRECATION, will REMOVE SOON (default: "~/.lotusminer") [$LOTUS_MINER_PATH, $LOTUS_STORAGE_PATH]
+   --markets-repo value                     Markets repo path [$LOTUS_MARKETS_PATH]
+   --call-on-markets                        (experimental; may be removed) call this command against a markets node; use only with common commands like net, auth, pprof, etc. whose target may be ambiguous (default: false)
    --vv                                     enables very verbose mode, useful for debugging the CLI (default: false)
    --help, -h                               show help (default: false)
    --version, -v                            print the version (default: false)

documentation/en/cli-lotus-worker.md

@@ -7,7 +7,7 @@ USAGE:
    lotus-worker [global options] command [command options] [arguments...]
 
 VERSION:
-   1.11.1-dev
+   1.11.1-rc1
 
 COMMANDS:
    run         Start lotus worker

documentation/en/cli-lotus.md

@@ -7,7 +7,7 @@ USAGE:
    lotus [global options] command [command options] [arguments...]
 
 VERSION:
-   1.11.1-dev
+   1.11.1-rc1
 
 COMMANDS:
    daemon   Start a lotus daemon process

go.mod

@@ -26,23 +26,23 @@ require (
 	github.com/elastic/gosigar v0.12.0
 	github.com/etclabscore/go-openrpc-reflect v0.0.36
 	github.com/fatih/color v1.9.0
-	github.com/filecoin-project/dagstore v0.3.1
+	github.com/filecoin-project/dagstore v0.3.2-0.20210728152352-8e0b04b9b384
 	github.com/filecoin-project/filecoin-ffi v0.30.4-0.20200910194244-f640612a1a1f
 	github.com/filecoin-project/go-address v0.0.5
 	github.com/filecoin-project/go-bitfield v0.2.4
 	github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2
 	github.com/filecoin-project/go-commp-utils v0.1.1-0.20210427191551-70bf140d31c7
 	github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03
-	github.com/filecoin-project/go-data-transfer v1.7.0
+	github.com/filecoin-project/go-data-transfer v1.7.2
 	github.com/filecoin-project/go-fil-commcid v0.1.0
 	github.com/filecoin-project/go-fil-commp-hashhash v0.1.0
-	github.com/filecoin-project/go-fil-markets v1.6.1-0.20210727201249-2213f80f0403
+	github.com/filecoin-project/go-fil-markets v1.6.3-0.20210729173742-a44f98a4f8c1
 	github.com/filecoin-project/go-jsonrpc v0.1.4-0.20210217175800-45ea43ac2bec
 	github.com/filecoin-project/go-multistore v0.0.3
 	github.com/filecoin-project/go-padreader v0.0.0-20210723183308-812a16dc01b1
 	github.com/filecoin-project/go-paramfetch v0.0.2-0.20210614165157-25a6c7769498
 	github.com/filecoin-project/go-state-types v0.1.1-0.20210722133031-ad9bfe54c124
-	github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe
+	github.com/filecoin-project/go-statemachine v1.0.1
 	github.com/filecoin-project/go-statestore v0.1.1
 	github.com/filecoin-project/go-storedcounter v0.0.0-20200421200003-1c99c62e8a5b
 	github.com/filecoin-project/specs-actors v0.9.14
@@ -78,7 +78,7 @@ require (
 	github.com/ipfs/go-ds-pebble v0.0.2-0.20200921225637-ce220f8ac459
 	github.com/ipfs/go-filestore v1.0.0
 	github.com/ipfs/go-fs-lock v0.0.6
-	github.com/ipfs/go-graphsync v0.6.5
+	github.com/ipfs/go-graphsync v0.6.6
 	github.com/ipfs/go-ipfs-blockstore v1.0.4
 	github.com/ipfs/go-ipfs-blocksutil v0.0.1
 	github.com/ipfs/go-ipfs-chunker v0.0.5

go.sum

@@ -256,8 +256,9 @@ github.com/fatih/color v1.8.0/go.mod h1:3l45GVGkyrnYNl9HoIjnp2NnNWvh6hLAqD8yTfGj
 github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
 github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
 github.com/fd/go-nat v1.0.0/go.mod h1:BTBu/CKvMmOMUPkKVef1pngt2WFH/lg7E6yQnulfp6E=
-github.com/filecoin-project/dagstore v0.3.1 h1:IShahbR3zEPE1pfHa5DiYMt/S40b9JuWZx6+kZPuyTQ=
 github.com/filecoin-project/dagstore v0.3.1/go.mod h1:WY5OoLfnwISCk6eASSF927KKPqLPIlTwmG1qHpA08KY=
+github.com/filecoin-project/dagstore v0.3.2-0.20210728152352-8e0b04b9b384 h1:0LrfJHR6YAa8t7VEZmbNECW7Deov8Di9CZHsZaamiM4=
+github.com/filecoin-project/dagstore v0.3.2-0.20210728152352-8e0b04b9b384/go.mod h1:WY5OoLfnwISCk6eASSF927KKPqLPIlTwmG1qHpA08KY=
 github.com/filecoin-project/go-address v0.0.3/go.mod h1:jr8JxKsYx+lQlQZmF5i2U0Z+cGQ59wMIps/8YW/lDj8=
 github.com/filecoin-project/go-address v0.0.5 h1:SSaFT/5aLfPXycUlFyemoHYhRgdyXClXCyDdNJKPlDM=
 github.com/filecoin-project/go-address v0.0.5/go.mod h1:jr8JxKsYx+lQlQZmF5i2U0Z+cGQ59wMIps/8YW/lDj8=
@@ -277,8 +278,9 @@ github.com/filecoin-project/go-commp-utils v0.1.1-0.20210427191551-70bf140d31c7/
 github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03 h1:2pMXdBnCiXjfCYx/hLqFxccPoqsSveQFxVLvNxy9bus=
 github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
 github.com/filecoin-project/go-data-transfer v1.0.1/go.mod h1:UxvfUAY9v3ub0a21BSK9u3pB2aq30Y0KMsG+w9/ysyo=
-github.com/filecoin-project/go-data-transfer v1.7.0 h1:mFRn+UuTdPROmhplLSekzd4rAs9ug8ubtSY4nw9wYkU=
 github.com/filecoin-project/go-data-transfer v1.7.0/go.mod h1:GLRr5BmLEqsLwXfiRDG7uJvph22KGL2M4iOuF8EINaU=
+github.com/filecoin-project/go-data-transfer v1.7.2 h1:iL3q5pxSloA7V2QucFofoVN3lquULz+Ml0KrNqMT5ZU=
+github.com/filecoin-project/go-data-transfer v1.7.2/go.mod h1:GLRr5BmLEqsLwXfiRDG7uJvph22KGL2M4iOuF8EINaU=
 github.com/filecoin-project/go-ds-versioning v0.1.0 h1:y/X6UksYTsK8TLCI7rttCKEvl8btmWxyFMEeeWGUxIQ=
 github.com/filecoin-project/go-ds-versioning v0.1.0/go.mod h1:mp16rb4i2QPmxBnmanUx8i/XANp+PFCCJWiAb+VW4/s=
 github.com/filecoin-project/go-fil-commcid v0.0.0-20200716160307-8f644712406f/go.mod h1:Eaox7Hvus1JgPrL5+M3+h7aSPHc0cVqpSxA+TxIEpZQ=
@@ -288,8 +290,8 @@ github.com/filecoin-project/go-fil-commcid v0.1.0/go.mod h1:Eaox7Hvus1JgPrL5+M3+
 github.com/filecoin-project/go-fil-commp-hashhash v0.1.0 h1:imrrpZWEHRnNqqv0tN7LXep5bFEVOVmQWHJvl2mgsGo=
 github.com/filecoin-project/go-fil-commp-hashhash v0.1.0/go.mod h1:73S8WSEWh9vr0fDJVnKADhfIv/d6dCbAGaAGWbdJEI8=
 github.com/filecoin-project/go-fil-markets v1.0.5-0.20201113164554-c5eba40d5335/go.mod h1:AJySOJC00JRWEZzRG2KsfUnqEf5ITXxeX09BE9N4f9c=
-github.com/filecoin-project/go-fil-markets v1.6.1-0.20210727201249-2213f80f0403 h1:Zrr2YRH186x0w4U5kaCYX+ccDD+g09yjYIO6QsICLpA=
+github.com/filecoin-project/go-fil-markets v1.6.3-0.20210729173742-a44f98a4f8c1 h1:xXgCsfEaA3wdof/N5mkD7PphDolhc9MAsIrO/QBH5Pg=
-github.com/filecoin-project/go-fil-markets v1.6.1-0.20210727201249-2213f80f0403/go.mod h1:13+DUe7AaHekzgpQPbacdppRoqz0SyPlx48g0f/pRmA=
+github.com/filecoin-project/go-fil-markets v1.6.3-0.20210729173742-a44f98a4f8c1/go.mod h1:13+DUe7AaHekzgpQPbacdppRoqz0SyPlx48g0f/pRmA=
 github.com/filecoin-project/go-hamt-ipld v0.1.5 h1:uoXrKbCQZ49OHpsTCkrThPNelC4W3LPEk0OrS/ytIBM=
 github.com/filecoin-project/go-hamt-ipld v0.1.5/go.mod h1:6Is+ONR5Cd5R6XZoCse1CWaXZc0Hdb/JeX+EQCQzX24=
 github.com/filecoin-project/go-hamt-ipld/v2 v2.0.0 h1:b3UDemBYN2HNfk3KOXNuxgTTxlWi3xVvbQP0IT38fvM=
@@ -314,8 +316,9 @@ github.com/filecoin-project/go-state-types v0.1.0/go.mod h1:ezYnPf0bNkTsDibL/psS
 github.com/filecoin-project/go-state-types v0.1.1-0.20210506134452-99b279731c48/go.mod h1:ezYnPf0bNkTsDibL/psSz5dy4B5awOJ/E7P2Saeep8g=
 github.com/filecoin-project/go-state-types v0.1.1-0.20210722133031-ad9bfe54c124 h1:veGrNABg/9I7prngrowkhwbvW5d5JN55MNKmbsr5FqA=
 github.com/filecoin-project/go-state-types v0.1.1-0.20210722133031-ad9bfe54c124/go.mod h1:ezYnPf0bNkTsDibL/psSz5dy4B5awOJ/E7P2Saeep8g=
-github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe h1:dF8u+LEWeIcTcfUcCf3WFVlc81Fr2JKg8zPzIbBDKDw=
 github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe/go.mod h1:FGwQgZAt2Gh5mjlwJUlVB62JeYdo+if0xWxSEfBD9ig=
+github.com/filecoin-project/go-statemachine v1.0.1 h1:LQ60+JDVjMdLxXmVFM2jjontzOYnfVE7u02CXV3WKSw=
+github.com/filecoin-project/go-statemachine v1.0.1/go.mod h1:jZdXXiHa61n4NmgWFG4w8tnqgvZVHYbJ3yW7+y8bF54=
 github.com/filecoin-project/go-statestore v0.1.0/go.mod h1:LFc9hD+fRxPqiHiaqUEZOinUJB4WARkRfNl10O7kTnI=
 github.com/filecoin-project/go-statestore v0.1.1 h1:ufMFq00VqnT2CAuDpcGnwLnCX1I/c3OROw/kXVNSTZk=
 github.com/filecoin-project/go-statestore v0.1.1/go.mod h1:LFc9hD+fRxPqiHiaqUEZOinUJB4WARkRfNl10O7kTnI=
@@ -637,8 +640,8 @@ github.com/ipfs/go-graphsync v0.1.0/go.mod h1:jMXfqIEDFukLPZHqDPp8tJMbHO9Rmeb9CE
 github.com/ipfs/go-graphsync v0.4.2/go.mod h1:/VmbZTUdUMTbNkgzAiCEucIIAU3BkLE2cZrDCVUhyi0=
 github.com/ipfs/go-graphsync v0.4.3/go.mod h1:mPOwDYv128gf8gxPFgXnz4fNrSYPsWyqisJ7ych+XDY=
 github.com/ipfs/go-graphsync v0.6.4/go.mod h1:5WyaeigpNdpiYQuW2vwpuecOoEfB4h747ZGEOKmAGTg=
-github.com/ipfs/go-graphsync v0.6.5 h1:YAJl6Yit23PQcaawzb1rPK9PSnbbq2jjMRPpRpJ0Y5U=
+github.com/ipfs/go-graphsync v0.6.6 h1:In7jjzvSXlrAUz4OjN41lxYf/dzkf1bVeVxLpwKMRo8=
-github.com/ipfs/go-graphsync v0.6.5/go.mod h1:GdHT8JeuIZ0R4lSjFR16Oe4zPi5dXwKi9zR9ADVlcdk=
+github.com/ipfs/go-graphsync v0.6.6/go.mod h1:GdHT8JeuIZ0R4lSjFR16Oe4zPi5dXwKi9zR9ADVlcdk=
 github.com/ipfs/go-hamt-ipld v0.1.1/go.mod h1:1EZCr2v0jlCnhpa+aZ0JZYp8Tt2w16+JJOAVz17YcDk=
 github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08=
 github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw=

itests/deals_test.go

@@ -42,4 +42,4 @@ func TestDealsWithSealingAndRPC(t *testing.T) {
 		dh.RunConcurrentDeals(kit.RunConcurrentDealsOpts{N: 1, UseCARFileForStorageDeal: true})
 	})
 
-}
+}

markets/dagstore/lotusaccessor.go

@@ -2,7 +2,10 @@ package dagstore
 
 import (
 	"context"
+	"fmt"
 	"io"
+	"os"
+	"strconv"
 
 	"github.com/filecoin-project/dagstore/throttle"
 	"github.com/ipfs/go-cid"
@@ -13,10 +16,19 @@ import (
 	"github.com/filecoin-project/go-fil-markets/shared"
 )
 
-// MaxConcurrentUnsealedFetches caps the amount of concurrent fetches for
+// MaxConcurrentStorageCalls caps the amount of concurrent calls to the
-// unsealed pieces, so that we don't saturate IO or the network too much,
+// storage, so that we don't spam it during heavy processes like bulk migration.
-// especially when bulk processing (e.g. at migration).
+var MaxConcurrentStorageCalls = func() int {
-var MaxConcurrentUnsealedFetches = 3
+	// TODO replace env with config.toml attribute.
+	v, ok := os.LookupEnv("LOTUS_DAGSTORE_MOUNT_CONCURRENCY")
+	if ok {
+		concurrency, err := strconv.Atoi(v)
+		if err == nil {
+			return concurrency
+		}
+	}
+	return 100
+}()
 
 type LotusAccessor interface {
 	FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error)
@@ -37,35 +49,13 @@ func NewLotusAccessor(store piecestore.PieceStore, rm retrievalmarket.RetrievalP
 	return &lotusAccessor{
 		pieceStore: store,
 		rm:         rm,
-		throttle:   throttle.Fixed(MaxConcurrentUnsealedFetches),
+		throttle:   throttle.Fixed(MaxConcurrentStorageCalls),
 		readyMgr:   shared.NewReadyManager(),
 	}
 }
 
-func (m *lotusAccessor) Start(ctx context.Context) error {
+func (m *lotusAccessor) Start(_ context.Context) error {
-	// Wait for the piece store to startup
+	return m.readyMgr.FireReady(nil)
-	ready := make(chan error)
-	m.pieceStore.OnReady(func(err error) {
-		select {
-		case <-ctx.Done():
-		case ready <- err:
-		}
-	})
-
-	select {
-	case <-ctx.Done():
-		err := xerrors.Errorf("context cancelled waiting for piece store startup: %w", ctx.Err())
-		if ferr := m.readyMgr.FireReady(err); ferr != nil {
-			log.Warnw("failed to publish ready event", "err", ferr)
-		}
-		return err
-
-	case err := <-ready:
-		if ferr := m.readyMgr.FireReady(err); ferr != nil {
-			log.Warnw("failed to publish ready event", "err", ferr)
-		}
-		return err
-	}
 }
 
 func (m *lotusAccessor) FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error) {
@@ -74,34 +64,48 @@ func (m *lotusAccessor) FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid
 		return nil, err
 	}
 
-	pieceInfo, err := m.pieceStore.GetPieceInfo(pieceCid)
+	// Throttle this path to avoid flooding the storage subsystem.
+	var pieceInfo piecestore.PieceInfo
+	err = m.throttle.Do(ctx, func(ctx context.Context) (err error) {
+		pieceInfo, err = m.pieceStore.GetPieceInfo(pieceCid)
+		return err
+	})
+
 	if err != nil {
 		return nil, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
 	}
 
 	if len(pieceInfo.Deals) == 0 {
-		return nil, xerrors.Errorf("no storage deals found for Piece %s", pieceCid)
+		return nil, xerrors.Errorf("no storage deals found for piece %s", pieceCid)
 	}
 
 	// prefer an unsealed sector containing the piece if one exists
 	for _, deal := range pieceInfo.Deals {
-		isUnsealed, err := m.rm.IsUnsealed(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
+		deal := deal
+
+		// Throttle this path to avoid flooding the storage subsystem.
+		var reader io.ReadCloser
+		err := m.throttle.Do(ctx, func(ctx context.Context) (err error) {
+			isUnsealed, err := m.rm.IsUnsealed(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
+			if err != nil {
+				return fmt.Errorf("failed to check if sector %d for deal %d was unsealed: %w", deal.SectorID, deal.DealID, err)
+			}
+			if !isUnsealed {
+				return nil
+			}
+			// Because we know we have an unsealed copy, this UnsealSector call will actually not perform any unsealing.
+			reader, err = m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
+			return err
+		})
+
 		if err != nil {
-			log.Warnf("failed to check if deal %d unsealed: %s", deal.DealID, err)
+			log.Warnf("failed to check/retrieve unsealed sector: %s", err)
-			continue
+			continue // move on to the next match.
 		}
-		if isUnsealed {
+
-			var reader io.ReadCloser
+		if reader != nil {
-			// We want to throttle this path, as these copies will be downloaded
+			// we were able to obtain a reader for an already unsealed piece
-			// immediately from the storage cluster without any unsealing
+			return reader, nil
-			// necessary.
-			deal := deal // calm the linter
-			err := m.throttle.Do(ctx, func(ctx context.Context) (err error) {
-				// UnsealSector will NOT unseal a sector if we already have an unsealed copy lying around.
-				reader, err = m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
-				return err
-			})
-			return reader, err
 		}
 	}
 
@@ -110,6 +114,8 @@ func (m *lotusAccessor) FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid
 	for _, deal := range pieceInfo.Deals {
 		// Note that if the deal data is not already unsealed, unsealing may
 		// block for a long time with the current PoRep
+		//
+		// This path is unthrottled.
 		reader, err := m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
 		if err != nil {
 			lastErr = xerrors.Errorf("failed to unseal deal %d: %w", deal.DealID, err)

markets/dagstore/lotusaccessor_test.go

@@ -123,6 +123,8 @@ func TestLotusAccessorGetUnpaddedCARSize(t *testing.T) {
 }
 
 func TestThrottle(t *testing.T) {
+	MaxConcurrentStorageCalls = 3
+
 	ctx := context.Background()
 	cid1, err := cid.Parse("bafkqaaa")
 	require.NoError(t, err)
@@ -160,7 +162,7 @@ func TestThrottle(t *testing.T) {
 	}
 
 	time.Sleep(500 * time.Millisecond)
-	require.EqualValues(t, MaxConcurrentUnsealedFetches, atomic.LoadInt32(&rpn.calls)) // throttled
+	require.EqualValues(t, MaxConcurrentStorageCalls, atomic.LoadInt32(&rpn.calls)) // throttled
 
 	// allow to proceed.
 	rpn.lk.Unlock()

markets/dagstore/wrapper.go

@@ -27,11 +27,12 @@ var log = logging.Logger("dagstore-wrapper")
 
 // MarketDAGStoreConfig is the config the market needs to then construct a DAG Store.
 type MarketDAGStoreConfig struct {
-	TransientsDir      string
+	TransientsDir       string
-	IndexDir           string
+	IndexDir            string
-	Datastore          ds.Datastore
+	Datastore           ds.Datastore
-	MaxConcurrentIndex int
+	MaxConcurrentIndex  int
-	GCInterval         time.Duration
+	MaxConcurrentCopies int
+	GCInterval          time.Duration
 }
 
 // DAGStore provides an interface for the DAG store that can be mocked out
@@ -89,8 +90,9 @@ func NewDagStoreWrapper(cfg MarketDAGStoreConfig, mountApi LotusAccessor) (*Wrap
 		TraceCh:       traceCh,
 		// not limiting fetches globally, as the Lotus mount does
 		// conditional throttling.
-		MaxConcurrentIndex: cfg.MaxConcurrentIndex,
+		MaxConcurrentIndex:  cfg.MaxConcurrentIndex,
-		RecoverOnStart:     dagstore.RecoverOnAcquire,
+		MaxConcurrentCopies: cfg.MaxConcurrentCopies,
+		RecoverOnStart:      dagstore.RecoverOnAcquire,
 	}
 	dagStore, err := dagstore.NewDAGStore(dcfg)
 	if err != nil {

node/builder_miner.go

@@ -73,6 +73,7 @@ func ConfigStorageMiner(c interface{}) Option {
 	return Options(
 		ConfigCommon(&cfg.Common, enableLibp2pNode),
 
+		Override(new(api.MinerSubsystems), modules.ExtractEnabledMinerSubsystems(cfg.Subsystems)),
 		Override(new(stores.LocalStorage), From(new(repo.LockedRepo))),
 		Override(new(*stores.Local), modules.LocalStorage),
 		Override(new(*stores.Remote), modules.RemoteStorage),

node/impl/storminer.go

@@ -23,8 +23,8 @@ import (
 	"github.com/filecoin-project/go-address"
 	datatransfer "github.com/filecoin-project/go-data-transfer"
 	"github.com/filecoin-project/go-fil-markets/piecestore"
-	retrievalmarket "github.com/filecoin-project/go-fil-markets/retrievalmarket"
+	"github.com/filecoin-project/go-fil-markets/retrievalmarket"
-	storagemarket "github.com/filecoin-project/go-fil-markets/storagemarket"
+	"github.com/filecoin-project/go-fil-markets/storagemarket"
 	"github.com/filecoin-project/go-state-types/abi"
 
 	sectorstorage "github.com/filecoin-project/lotus/extern/sector-storage"
@@ -51,6 +51,8 @@ type StorageMinerAPI struct {
 	api.Common
 	api.Net
 
+	EnabledSubsystems api.MinerSubsystems
+
 	Full        api.FullNode
 	LocalStore  *stores.Local
 	RemoteStore *stores.Remote
@@ -703,4 +705,8 @@ func (sm *StorageMinerAPI) ComputeProof(ctx context.Context, ssi []builtin.Secto
 	return sm.Epp.ComputeProof(ctx, ssi, rand)
 }
 
+func (sm *StorageMinerAPI) RuntimeSubsystems(context.Context) (res api.MinerSubsystems, err error) {
+	return sm.EnabledSubsystems, nil
+}
+
 var _ api.StorageMiner = &StorageMinerAPI{}

node/modules/storageminer.go

@@ -585,8 +585,15 @@ func NewLotusAccessor(lc fx.Lifecycle,
 	rpn retrievalmarket.RetrievalProviderNode,
 ) (dagstore.LotusAccessor, error) {
 	mountApi := dagstore.NewLotusAccessor(pieceStore, rpn)
+	ready := make(chan error, 1)
+	pieceStore.OnReady(func(err error) {
+		ready <- err
+	})
 	lc.Append(fx.Hook{
 		OnStart: func(ctx context.Context) error {
+			if err := <-ready; err != nil {
+				return fmt.Errorf("aborting dagstore start; piecestore failed to start: %s", err)
+			}
 			return mountApi.Start(ctx)
 		},
 		OnStop: func(context.Context) error {
@@ -609,11 +616,12 @@ func DagStoreWrapper(
 	}
 
 	cfg := dagstore.MarketDAGStoreConfig{
-		TransientsDir:      filepath.Join(dagStoreDir, "transients"),
+		TransientsDir:       filepath.Join(dagStoreDir, "transients"),
-		IndexDir:           filepath.Join(dagStoreDir, "index"),
+		IndexDir:            filepath.Join(dagStoreDir, "index"),
-		Datastore:          dagStoreDS,
+		Datastore:           dagStoreDS,
-		GCInterval:         1 * time.Minute,
+		GCInterval:          1 * time.Minute,
-		MaxConcurrentIndex: 5,
+		MaxConcurrentIndex:  5,
+		MaxConcurrentCopies: 2,
 	}
 
 	dsw, err := dagstore.NewDagStoreWrapper(cfg, lotusAccessor)
@@ -1082,3 +1090,19 @@ func mutateCfg(r repo.LockedRepo, mutator func(*config.StorageMiner)) error {
 
 	return multierr.Combine(typeErr, setConfigErr)
 }
+
+func ExtractEnabledMinerSubsystems(cfg config.MinerSubsystemConfig) (res api.MinerSubsystems) {
+	if cfg.EnableMining {
+		res = append(res, api.SubsystemMining)
+	}
+	if cfg.EnableSealing {
+		res = append(res, api.SubsystemSealing)
+	}
+	if cfg.EnableSectorStorage {
+		res = append(res, api.SubsystemSectorStorage)
+	}
+	if cfg.EnableMarkets {
+		res = append(res, api.SubsystemMarkets)
+	}
+	return res
+}

node/repo/fsrepo.go

@@ -49,13 +49,31 @@ const (
 	StorageMiner
 	Worker
 	Wallet
+	Markets
 )
 
+func (t RepoType) String() string {
+	s := [...]string{
+		"__invalid__",
+		"FullNode",
+		"StorageMiner",
+		"Worker",
+		"Wallet",
+		"Markets",
+	}
+	if t < 0 || int(t) > len(s) {
+		return "__invalid__"
+	}
+	return s[t]
+}
+
 func defConfForType(t RepoType) interface{} {
 	switch t {
 	case FullNode:
 		return config.DefaultFullNode()
-	case StorageMiner:
+	case StorageMiner, Markets:
+		// markets is a specialised miner service
+		// this taxonomy needs to be cleaned up
 		return config.DefaultStorageMiner()
 	case Worker:
 		return &struct{}{}

v1.11.1

@@ -0,0 +1,170 @@
+- github.com/filecoin-project/lotus:
+  - Merge branch 'releases' into release/v1.11.1
+  - Update to proof v8.0.3 ([filecoin-project/lotus#6890](https://github.com/filecoin-project/lotus/pull/6890))
+  - lotus-shed: initial export cmd for markets related metadata ([filecoin-project/lotus#6840](https://github.com/filecoin-project/lotus/pull/6840))
+  - add a very verbose -vv flag to lotus and lotus-miner. ([filecoin-project/lotus#6888](https://github.com/filecoin-project/lotus/pull/6888))
+  - Update RELEASE_ISSUE_TEMPLATE.md ([filecoin-project/lotus#6880](https://github.com/filecoin-project/lotus/pull/6880))
+  - Moving GC for badger ([filecoin-project/lotus#6854](https://github.com/filecoin-project/lotus/pull/6854))
+  - Add github actions for staled pr ([filecoin-project/lotus#6879](https://github.com/filecoin-project/lotus/pull/6879))
+  - Add allocated sectorid vis ([filecoin-project/lotus#4638](https://github.com/filecoin-project/lotus/pull/4638))
+  - rename `cmd/lotus{-storage=>}-miner` to match binary. ([filecoin-project/lotus#6886](https://github.com/filecoin-project/lotus/pull/6886))
+  - update to go-fil-market v1.6.0 ([filecoin-project/lotus#6885](https://github.com/filecoin-project/lotus/pull/6885))
+  - Bump go-multihash, adjust test for supported version ([filecoin-project/lotus#6674](https://github.com/filecoin-project/lotus/pull/6674))
+  - Fix padding of deals, which only partially shipped in #5988 ([filecoin-project/lotus#6683](https://github.com/filecoin-project/lotus/pull/6683))
+  - fix racy TestSimultanenousTransferLimit. ([filecoin-project/lotus#6862](https://github.com/filecoin-project/lotus/pull/6862))
+  - Improve splitstore warmup ([filecoin-project/lotus#6867](https://github.com/filecoin-project/lotus/pull/6867))
+  - ValidateBlock: Assert that block header height's are greater than parents ([filecoin-project/lotus#6872](https://github.com/filecoin-project/lotus/pull/6872))
+  - feat: Don't panic when api impl is nil ([filecoin-project/lotus#6857](https://github.com/filecoin-project/lotus/pull/6857))
+  - splitstore shed utils ([filecoin-project/lotus#6811](https://github.com/filecoin-project/lotus/pull/6811))
+  - Fix links in issue templates
+  - Update issue templates and add templates for M1 ([filecoin-project/lotus#6856](https://github.com/filecoin-project/lotus/pull/6856))
+  - Splitstore: support on-disk marksets using badger ([filecoin-project/lotus#6833](https://github.com/filecoin-project/lotus/pull/6833))
+  - Config UX improvements ([filecoin-project/lotus#6848](https://github.com/filecoin-project/lotus/pull/6848))
+  - fix deal concurrency test failures by upgrading graphsync and others ([filecoin-project/lotus#6724](https://github.com/filecoin-project/lotus/pull/6724))
+  - Update issue templates to forms ([filecoin-project/lotus#6798](https://github.com/filecoin-project/lotus/pull/6798))
+  - Nerpa v13 upgrade ([filecoin-project/lotus#6837](https://github.com/filecoin-project/lotus/pull/6837))
+  - add docker-compose file ([filecoin-project/lotus#6544](https://github.com/filecoin-project/lotus/pull/6544))
+  - fix warmup by decoupling state from message receipt walk ([filecoin-project/lotus#6841](https://github.com/filecoin-project/lotus/pull/6841))
+  - add a command for compacting sector numbers bitfield ([filecoin-project/lotus#4640](https://github.com/filecoin-project/lotus/pull/4640))
+  - PriceListByVersion ([filecoin-project/lotus#6766](https://github.com/filecoin-project/lotus/pull/6766))
+  - easy way to make install app ([filecoin-project/lotus#5183](https://github.com/filecoin-project/lotus/pull/5183))
+  - api: Separate the Net interface from Common ([filecoin-project/lotus#6627](https://github.com/filecoin-project/lotus/pull/6627))
+  - cache loaded block messages ([filecoin-project/lotus#6760](https://github.com/filecoin-project/lotus/pull/6760))
+  - fix: on randomness change, use new rand ([filecoin-project/lotus#6805](https://github.com/filecoin-project/lotus/pull/6805))
+  - Splitstore: add retention policy option for keeping messages in the hotstore ([filecoin-project/lotus#6775](https://github.com/filecoin-project/lotus/pull/6775))
+  - Introduce the LOTUS_CHAIN_BADGERSTORE_DISABLE_FSYNC envvar ([filecoin-project/lotus#6817](https://github.com/filecoin-project/lotus/pull/6817))
+  - add StateReadState to gateway api ([filecoin-project/lotus#6818](https://github.com/filecoin-project/lotus/pull/6818))
+  - add SealProof in SectorBuilder ([filecoin-project/lotus#6815](https://github.com/filecoin-project/lotus/pull/6815))
+  - release -> master ([filecoin-project/lotus#6828](https://github.com/filecoin-project/lotus/pull/6828))
+  - sealing: Handle preCommitParams errors more correctly ([filecoin-project/lotus#6763](https://github.com/filecoin-project/lotus/pull/6763))
+  - fix: always check if StateSearchMessage returns nil ([filecoin-project/lotus#6802](https://github.com/filecoin-project/lotus/pull/6802))
+  - ClientFindData: always fetch peer id from chain ([filecoin-project/lotus#6807](https://github.com/filecoin-project/lotus/pull/6807))
+  - test: fix flaky window post tests ([filecoin-project/lotus#6804](https://github.com/filecoin-project/lotus/pull/6804))
+  -  ([filecoin-project/lotus#6800](https://github.com/filecoin-project/lotus/pull/6800))
+  - fixes #6786 segfault ([filecoin-project/lotus#6787](https://github.com/filecoin-project/lotus/pull/6787))
+  - Splitstore: add support for protecting out of chain references in the blockstore ([filecoin-project/lotus#6777](https://github.com/filecoin-project/lotus/pull/6777))
+  - Resurrect CODEOWNERS, but for maintainers group ([filecoin-project/lotus#6773](https://github.com/filecoin-project/lotus/pull/6773))
+  - update go-libp2p-pubsub to v0.5.0 ([filecoin-project/lotus#6764](https://github.com/filecoin-project/lotus/pull/6764))
+  - Implement exposed splitstore ([filecoin-project/lotus#6762](https://github.com/filecoin-project/lotus/pull/6762))
+  - Add ChainGetMessagesInTipset API ([filecoin-project/lotus#6642](https://github.com/filecoin-project/lotus/pull/6642))
+  - test: handle null blocks in TestForkRefuseCall ([filecoin-project/lotus#6758](https://github.com/filecoin-project/lotus/pull/6758))
+  - Master disclaimer  ([filecoin-project/lotus#6757](https://github.com/filecoin-project/lotus/pull/6757))
+  - Splitstore code reorg ([filecoin-project/lotus#6756](https://github.com/filecoin-project/lotus/pull/6756))
+  - Create stale.yml ([filecoin-project/lotus#6747](https://github.com/filecoin-project/lotus/pull/6747))
+  - Splitstore: Some small fixes ([filecoin-project/lotus#6754](https://github.com/filecoin-project/lotus/pull/6754))
+  -  ([filecoin-project/lotus#6746](https://github.com/filecoin-project/lotus/pull/6746))
+  - Handle the --color flag via proper global state ([filecoin-project/lotus#6743](https://github.com/filecoin-project/lotus/pull/6743))
+  - Config for collateral from miner available balance ([filecoin-project/lotus#6629](https://github.com/filecoin-project/lotus/pull/6629))
+  - Support standalone miner-market process ([filecoin-project/lotus#6356](https://github.com/filecoin-project/lotus/pull/6356))
+  - Splitstore Enhanchements ([filecoin-project/lotus#6474](https://github.com/filecoin-project/lotus/pull/6474))
+  -  ([filecoin-project/lotus#6739](https://github.com/filecoin-project/lotus/pull/6739))
+  - Add more deal details to lotus-miner info ([filecoin-project/lotus#6708](https://github.com/filecoin-project/lotus/pull/6708))
+  - Release template: Update all testnet infra at once ([filecoin-project/lotus#6710](https://github.com/filecoin-project/lotus/pull/6710))
+  - Fix Lotus shed
+  - Fix bugs in sectors extend --v1-sectors ([filecoin-project/lotus#6066](https://github.com/filecoin-project/lotus/pull/6066))
+  - add election backtest ([filecoin-project/lotus#5950](https://github.com/filecoin-project/lotus/pull/5950))
+  - Envvar to disable slash filter ([filecoin-project/lotus#6620](https://github.com/filecoin-project/lotus/pull/6620))
+  - Release Template: remove binary validation step ([filecoin-project/lotus#6709](https://github.com/filecoin-project/lotus/pull/6709))
+  - Config for deal publishing control addresses ([filecoin-project/lotus#6697](https://github.com/filecoin-project/lotus/pull/6697))
+  - Reset of the interop network ([filecoin-project/lotus#6689](https://github.com/filecoin-project/lotus/pull/6689))
+  - Enable color by default only if os.Stdout is a TTY ([filecoin-project/lotus#6696](https://github.com/filecoin-project/lotus/pull/6696))
+  - Stop outputing ANSI color on non-TTY ([filecoin-project/lotus#6694](https://github.com/filecoin-project/lotus/pull/6694))
+  - add dollar sign ([filecoin-project/lotus#6690](https://github.com/filecoin-project/lotus/pull/6690))
+  - get-actor cli spelling fix ([filecoin-project/lotus#6681](https://github.com/filecoin-project/lotus/pull/6681))
+  - fix "lotus-seed genesis car" error "merkledag: not found" ([filecoin-project/lotus#6688](https://github.com/filecoin-project/lotus/pull/6688))
+  - polish(statetree): accept a context in statetree diff for timeouts ([filecoin-project/lotus#6639](https://github.com/filecoin-project/lotus/pull/6639))
+  - Add helptext to lotus chain export ([filecoin-project/lotus#6672](https://github.com/filecoin-project/lotus/pull/6672))
+  - Get retrieval pricing input should not error out on a deal state fetch ([filecoin-project/lotus#6679](https://github.com/filecoin-project/lotus/pull/6679))
+  - Fix more CID double-encoding as hex ([filecoin-project/lotus#6680](https://github.com/filecoin-project/lotus/pull/6680))
+  - add an incremental nonce itest. ([filecoin-project/lotus#6663](https://github.com/filecoin-project/lotus/pull/6663))
+  - storage: Fix FinalizeSector with sectors in stoage paths ([filecoin-project/lotus#6653](https://github.com/filecoin-project/lotus/pull/6653))
+  - Fix tiny error in check-client-datacap ([filecoin-project/lotus#6664](https://github.com/filecoin-project/lotus/pull/6664))
+  - Fix: precommit_batch method used the wrong cfg.CommitBatchWait ([filecoin-project/lotus#6658](https://github.com/filecoin-project/lotus/pull/6658))
+  - fix ticket expiration check ([filecoin-project/lotus#6635](https://github.com/filecoin-project/lotus/pull/6635))
+  - commit batch: AggregateAboveBaseFee config ([filecoin-project/lotus#6650](https://github.com/filecoin-project/lotus/pull/6650))
+  - commit batch: Initialize the FailedSectors map ([filecoin-project/lotus#6647](https://github.com/filecoin-project/lotus/pull/6647))
+  - Fast-path retry submitting commit aggregate if commit is still valid  ([filecoin-project/lotus#6638](https://github.com/filecoin-project/lotus/pull/6638))
+  - remove precommit check in handleCommitFailed ([filecoin-project/lotus#6634](https://github.com/filecoin-project/lotus/pull/6634))
+  - Reuse timers in sealing batch logic ([filecoin-project/lotus#6636](https://github.com/filecoin-project/lotus/pull/6636))
+  - shed tool to estimate aggregate network fees ([filecoin-project/lotus#6631](https://github.com/filecoin-project/lotus/pull/6631))
+  - fix prove commit aggregate send token amount ([filecoin-project/lotus#6625](https://github.com/filecoin-project/lotus/pull/6625))
+  - Update version.go to 1.11.1 ([filecoin-project/lotus#6621](https://github.com/filecoin-project/lotus/pull/6621))
+- github.com/filecoin-project/go-data-transfer (v1.6.0 -> v1.7.0):
+  - release: v1.7.0
+  - Fire a transfer queued event when a transfer is queued in Graphsync (#221) ([filecoin-project/go-data-transfer#221](https://github.com/filecoin-project/go-data-transfer/pull/221))
+  - feat: pass ChannelID to ValidatePush & ValidatePull (#220) ([filecoin-project/go-data-transfer#220](https://github.com/filecoin-project/go-data-transfer/pull/220))
+  - release: v1.6.1 ([filecoin-project/go-data-transfer#218](https://github.com/filecoin-project/go-data-transfer/pull/218))
+  - Remove CID lists (#217) ([filecoin-project/go-data-transfer#217](https://github.com/filecoin-project/go-data-transfer/pull/217))
+  - Merge v1.6.0 ([filecoin-project/go-data-transfer#214](https://github.com/filecoin-project/go-data-transfer/pull/214))
+  - Remove restart ack timeout (#211) ([filecoin-project/go-data-transfer#211](https://github.com/filecoin-project/go-data-transfer/pull/211))
+  - feat: use different extension names to fit multiple hooks data in same graphsync message (#204) ([filecoin-project/go-data-transfer#204](https://github.com/filecoin-project/go-data-transfer/pull/204))
+  - fix: map race in GS transport (#208) ([filecoin-project/go-data-transfer#208](https://github.com/filecoin-project/go-data-transfer/pull/208))
+  - refactor: simplify graphsync transport (#203) ([filecoin-project/go-data-transfer#203](https://github.com/filecoin-project/go-data-transfer/pull/203))
+  - release: v1.5.0 (#200) ([filecoin-project/go-data-transfer#200](https://github.com/filecoin-project/go-data-transfer/pull/200))
+- github.com/filecoin-project/go-fil-markets (v1.5.0 -> v1.6.0):
+  - release: v1.6.0
+  - support padding out smaller files (#536) ([filecoin-project/go-fil-markets#536](https://github.com/filecoin-project/go-fil-markets/pull/536))
+  - On overloaded CI 10 seconds just isn't enough (#587) ([filecoin-project/go-fil-markets#587](https://github.com/filecoin-project/go-fil-markets/pull/587))
+  - Do not hex-encode CIDs in logs (#561) ([filecoin-project/go-fil-markets#561](https://github.com/filecoin-project/go-fil-markets/pull/561))
+  - remove wrong peer check in push deal validation (#585) ([filecoin-project/go-fil-markets#585](https://github.com/filecoin-project/go-fil-markets/pull/585))
+  - fix: circleci docs-gen task (#574) ([filecoin-project/go-fil-markets#574](https://github.com/filecoin-project/go-fil-markets/pull/574))
+  - Storage market request queued event and validation interface changes (#555) ([filecoin-project/go-fil-markets#555](https://github.com/filecoin-project/go-fil-markets/pull/555))
+  - build(deps): bump ws from 6.2.1 to 6.2.2 (#554) ([filecoin-project/go-fil-markets#554](https://github.com/filecoin-project/go-fil-markets/pull/554))
+  - release: v1.5.0 ([filecoin-project/go-fil-markets#553](https://github.com/filecoin-project/go-fil-markets/pull/553))
+- github.com/filecoin-project/go-padreader (v0.0.0-20200903213702-ed5fae088b20 -> v0.0.0-20210723183308-812a16dc01b1):
+  - New method to pad harder (#6) ([filecoin-project/go-padreader#6](https://github.com/filecoin-project/go-padreader/pull/6))
+  - Create SECURITY.md (#5) ([filecoin-project/go-padreader#5](https://github.com/filecoin-project/go-padreader/pull/5))
+- github.com/filecoin-project/go-state-types (v0.1.1-0.20210506134452-99b279731c48 -> v0.1.1-0.20210722133031-ad9bfe54c124):
+  - Add version 6.5 (#30) ([filecoin-project/go-state-types#30](https://github.com/filecoin-project/go-state-types/pull/30))
+  - rename file
+- github.com/filecoin-project/specs-actors/v5 (v5.0.1 -> v5.0.3):
+  - Adjust code for subtle change in go-multihash 0.0.15 (#1463) ([filecoin-project/specs-actors#1463](https://github.com/filecoin-project/specs-actors/pull/1463))
+  - Bump go state types (#1464) ([filecoin-project/specs-actors#1464](https://github.com/filecoin-project/specs-actors/pull/1464))
+  - Create CODEOWNERS (#1465) ([filecoin-project/specs-actors#1465](https://github.com/filecoin-project/specs-actors/pull/1465))
+  - Test deterministic offset (#1462) ([filecoin-project/specs-actors#1462](https://github.com/filecoin-project/specs-actors/pull/1462))
+
+Contributors
+
+| Contributor | Commits | Lines ± | Files Changed |
+|-------------|---------|---------|---------------|
+| vyzo | 295 | +8700/-5936 | 397 |
+| Anton Evangelatov | 94 | +4680/-2965 | 277 |
+| Łukasz Magiera | 37 | +3851/-1611 | 146 |
+| Mike Greenberg | 1 | +2310/-578 | 8 |
+| dirkmc | 7 | +1154/-726 | 29 |
+| Jennifer Wang | 9 | +485/-341 | 26 |
+| Peter Rabbitson | 18 | +469/-273 | 64 |
+| Cory Schwartz | 5 | +576/-135 | 14 |
+| hunjixin | 7 | +404/-82 | 19 |
+| ZenGround0 | 17 | +284/-135 | 44 |
+| Dirk McCormick | 17 | +348/-47 | 17 |
+| Raúl Kripalani | 18 | +254/-97 | 62 |
+| tchardin | 1 | +261/-33 | 4 |
+| Jakub Sztandera | 4 | +254/-16 | 4 |
+| Aarsh Shah | 2 | +196/-40 | 28 |
+| whyrusleeping | 3 | +150/-9 | 8 |
+| Whyrusleeping | 2 | +87/-66 | 10 |
+| Aayush Rajasekaran | 10 | +81/-53 | 13 |
+| zgfzgf | 2 | +104/-4 | 2 |
+| aarshkshah1992 | 4 | +73/-7 | 6 |
+| llifezou | 4 | +59/-20 | 4 |
+| Steven Allen | 7 | +47/-17 | 9 |
+| johnli-helloworld | 3 | +46/-15 | 5 |
+| frrist | 1 | +28/-23 | 2 |
+| Jennifer | 4 | +31/-2 | 4 |
+| wangchao | 1 | +1/-27 | 1 |
+| Jiaying Wang | 2 | +7/-21 | 2 |
+| hannahhoward | 3 | +21/-2 | 3 |
+| chadwick2143 | 1 | +15/-1 | 1 |
+| Jerry | 2 | +9/-4 | 2 |
+| Steve Loeppky | 2 | +12/-0 | 2 |
+| David Dias | 1 | +9/-0 | 1 |
+| dependabot[bot] | 1 | +3/-3 | 1 |
+| zhoutian527 | 1 | +2/-2 | 1 |
+| xloem | 1 | +4/-0 | 1 |
+| Travis Person | 2 | +2/-2 | 3 |
+| Liviu Damian | 2 | +2/-2 | 2 |
+| Jim Pick | 2 | +2/-2 | 2 |
+| Frank | 1 | +3/-0 | 1 |
+| turuslan | 1 | +1/-1 | 1 |
+| Kirk Baird | 1 | +0/-0 | 1 |