Compare commits

..
1 Commits
Author SHA1 Message Date
Jennifer Wang 1757ffcb2c Merge branch 'releases' into jen/m1.1 2021-07-27 22:17:34 -04:00
57 changed files with 769 additions and 2824 deletions
-188
View File
@@ -1,186 +1,5 @@
# Lotus changelog
# 1.11.1-rc2 / 2021-08-01
> Note: for discussion about this release, please comment [here](https://github.com/filecoin-project/lotus/discussions/6904)
This is the second release candidate for the **highly-recommended** but optional Lotus v1.11.1 release that introduces many deal making and datastore improvements and new features along with other bug fixes. A more organized and detailed release note will be shared in the next few days with future RCs, highlights are:
- **Experimental** [lotus-miner market subsystem](https://docs.filecoin.io/mine/lotus/split-markets-miners/#frontmatter-title)
- **Under Testing** [splistore](https://github.com/filecoin-project/lotus/blob/master/blockstore/splitstore/README.md)
- github.com/filecoin-project/lotus:
- Merge branch 'releases' into release/v1.11.1
- Update to proof v8.0.3 ([filecoin-project/lotus#6890](https://github.com/filecoin-project/lotus/pull/6890))
- lotus-shed: initial export cmd for markets related metadata ([filecoin-project/lotus#6840](https://github.com/filecoin-project/lotus/pull/6840))
- add a very verbose -vv flag to lotus and lotus-miner. ([filecoin-project/lotus#6888](https://github.com/filecoin-project/lotus/pull/6888))
- Update RELEASE_ISSUE_TEMPLATE.md ([filecoin-project/lotus#6880](https://github.com/filecoin-project/lotus/pull/6880))
- Moving GC for badger ([filecoin-project/lotus#6854](https://github.com/filecoin-project/lotus/pull/6854))
- Add github actions for staled pr ([filecoin-project/lotus#6879](https://github.com/filecoin-project/lotus/pull/6879))
- Add allocated sectorid vis ([filecoin-project/lotus#4638](https://github.com/filecoin-project/lotus/pull/4638))
- rename `cmd/lotus{-storage=>}-miner` to match binary. ([filecoin-project/lotus#6886](https://github.com/filecoin-project/lotus/pull/6886))
- update to go-fil-market v1.6.0 ([filecoin-project/lotus#6885](https://github.com/filecoin-project/lotus/pull/6885))
- Bump go-multihash, adjust test for supported version ([filecoin-project/lotus#6674](https://github.com/filecoin-project/lotus/pull/6674))
- Fix padding of deals, which only partially shipped in #5988 ([filecoin-project/lotus#6683](https://github.com/filecoin-project/lotus/pull/6683))
- fix racy TestSimultanenousTransferLimit. ([filecoin-project/lotus#6862](https://github.com/filecoin-project/lotus/pull/6862))
- Improve splitstore warmup ([filecoin-project/lotus#6867](https://github.com/filecoin-project/lotus/pull/6867))
- ValidateBlock: Assert that block header height's are greater than parents ([filecoin-project/lotus#6872](https://github.com/filecoin-project/lotus/pull/6872))
- feat: Don't panic when api impl is nil ([filecoin-project/lotus#6857](https://github.com/filecoin-project/lotus/pull/6857))
- splitstore shed utils ([filecoin-project/lotus#6811](https://github.com/filecoin-project/lotus/pull/6811))
- Fix links in issue templates
- Update issue templates and add templates for M1 ([filecoin-project/lotus#6856](https://github.com/filecoin-project/lotus/pull/6856))
- Splitstore: support on-disk marksets using badger ([filecoin-project/lotus#6833](https://github.com/filecoin-project/lotus/pull/6833))
- Config UX improvements ([filecoin-project/lotus#6848](https://github.com/filecoin-project/lotus/pull/6848))
- fix deal concurrency test failures by upgrading graphsync and others ([filecoin-project/lotus#6724](https://github.com/filecoin-project/lotus/pull/6724))
- Update issue templates to forms ([filecoin-project/lotus#6798](https://github.com/filecoin-project/lotus/pull/6798))
- Nerpa v13 upgrade ([filecoin-project/lotus#6837](https://github.com/filecoin-project/lotus/pull/6837))
- add docker-compose file ([filecoin-project/lotus#6544](https://github.com/filecoin-project/lotus/pull/6544))
- fix warmup by decoupling state from message receipt walk ([filecoin-project/lotus#6841](https://github.com/filecoin-project/lotus/pull/6841))
- add a command for compacting sector numbers bitfield ([filecoin-project/lotus#4640](https://github.com/filecoin-project/lotus/pull/4640))
- PriceListByVersion ([filecoin-project/lotus#6766](https://github.com/filecoin-project/lotus/pull/6766))
- easy way to make install app ([filecoin-project/lotus#5183](https://github.com/filecoin-project/lotus/pull/5183))
- api: Separate the Net interface from Common ([filecoin-project/lotus#6627](https://github.com/filecoin-project/lotus/pull/6627))
- cache loaded block messages ([filecoin-project/lotus#6760](https://github.com/filecoin-project/lotus/pull/6760))
- fix: on randomness change, use new rand ([filecoin-project/lotus#6805](https://github.com/filecoin-project/lotus/pull/6805))
- Splitstore: add retention policy option for keeping messages in the hotstore ([filecoin-project/lotus#6775](https://github.com/filecoin-project/lotus/pull/6775))
- Introduce the LOTUS_CHAIN_BADGERSTORE_DISABLE_FSYNC envvar ([filecoin-project/lotus#6817](https://github.com/filecoin-project/lotus/pull/6817))
- add StateReadState to gateway api ([filecoin-project/lotus#6818](https://github.com/filecoin-project/lotus/pull/6818))
- add SealProof in SectorBuilder ([filecoin-project/lotus#6815](https://github.com/filecoin-project/lotus/pull/6815))
- release -> master ([filecoin-project/lotus#6828](https://github.com/filecoin-project/lotus/pull/6828))
- sealing: Handle preCommitParams errors more correctly ([filecoin-project/lotus#6763](https://github.com/filecoin-project/lotus/pull/6763))
- fix: always check if StateSearchMessage returns nil ([filecoin-project/lotus#6802](https://github.com/filecoin-project/lotus/pull/6802))
- ClientFindData: always fetch peer id from chain ([filecoin-project/lotus#6807](https://github.com/filecoin-project/lotus/pull/6807))
- test: fix flaky window post tests ([filecoin-project/lotus#6804](https://github.com/filecoin-project/lotus/pull/6804))
- ([filecoin-project/lotus#6800](https://github.com/filecoin-project/lotus/pull/6800))
- fixes #6786 segfault ([filecoin-project/lotus#6787](https://github.com/filecoin-project/lotus/pull/6787))
- Splitstore: add support for protecting out of chain references in the blockstore ([filecoin-project/lotus#6777](https://github.com/filecoin-project/lotus/pull/6777))
- Resurrect CODEOWNERS, but for maintainers group ([filecoin-project/lotus#6773](https://github.com/filecoin-project/lotus/pull/6773))
- update go-libp2p-pubsub to v0.5.0 ([filecoin-project/lotus#6764](https://github.com/filecoin-project/lotus/pull/6764))
- Implement exposed splitstore ([filecoin-project/lotus#6762](https://github.com/filecoin-project/lotus/pull/6762))
- Add ChainGetMessagesInTipset API ([filecoin-project/lotus#6642](https://github.com/filecoin-project/lotus/pull/6642))
- test: handle null blocks in TestForkRefuseCall ([filecoin-project/lotus#6758](https://github.com/filecoin-project/lotus/pull/6758))
- Master disclaimer ([filecoin-project/lotus#6757](https://github.com/filecoin-project/lotus/pull/6757))
- Splitstore code reorg ([filecoin-project/lotus#6756](https://github.com/filecoin-project/lotus/pull/6756))
- Create stale.yml ([filecoin-project/lotus#6747](https://github.com/filecoin-project/lotus/pull/6747))
- Splitstore: Some small fixes ([filecoin-project/lotus#6754](https://github.com/filecoin-project/lotus/pull/6754))
- ([filecoin-project/lotus#6746](https://github.com/filecoin-project/lotus/pull/6746))
- Handle the --color flag via proper global state ([filecoin-project/lotus#6743](https://github.com/filecoin-project/lotus/pull/6743))
- Config for collateral from miner available balance ([filecoin-project/lotus#6629](https://github.com/filecoin-project/lotus/pull/6629))
- Support standalone miner-market process ([filecoin-project/lotus#6356](https://github.com/filecoin-project/lotus/pull/6356))
- Splitstore Enhanchements ([filecoin-project/lotus#6474](https://github.com/filecoin-project/lotus/pull/6474))
- ([filecoin-project/lotus#6739](https://github.com/filecoin-project/lotus/pull/6739))
- Add more deal details to lotus-miner info ([filecoin-project/lotus#6708](https://github.com/filecoin-project/lotus/pull/6708))
- Release template: Update all testnet infra at once ([filecoin-project/lotus#6710](https://github.com/filecoin-project/lotus/pull/6710))
- Fix Lotus shed
- Fix bugs in sectors extend --v1-sectors ([filecoin-project/lotus#6066](https://github.com/filecoin-project/lotus/pull/6066))
- add election backtest ([filecoin-project/lotus#5950](https://github.com/filecoin-project/lotus/pull/5950))
- Envvar to disable slash filter ([filecoin-project/lotus#6620](https://github.com/filecoin-project/lotus/pull/6620))
- Release Template: remove binary validation step ([filecoin-project/lotus#6709](https://github.com/filecoin-project/lotus/pull/6709))
- Config for deal publishing control addresses ([filecoin-project/lotus#6697](https://github.com/filecoin-project/lotus/pull/6697))
- Reset of the interop network ([filecoin-project/lotus#6689](https://github.com/filecoin-project/lotus/pull/6689))
- Enable color by default only if os.Stdout is a TTY ([filecoin-project/lotus#6696](https://github.com/filecoin-project/lotus/pull/6696))
- Stop outputing ANSI color on non-TTY ([filecoin-project/lotus#6694](https://github.com/filecoin-project/lotus/pull/6694))
- add dollar sign ([filecoin-project/lotus#6690](https://github.com/filecoin-project/lotus/pull/6690))
- get-actor cli spelling fix ([filecoin-project/lotus#6681](https://github.com/filecoin-project/lotus/pull/6681))
- fix "lotus-seed genesis car" error "merkledag: not found" ([filecoin-project/lotus#6688](https://github.com/filecoin-project/lotus/pull/6688))
- polish(statetree): accept a context in statetree diff for timeouts ([filecoin-project/lotus#6639](https://github.com/filecoin-project/lotus/pull/6639))
- Add helptext to lotus chain export ([filecoin-project/lotus#6672](https://github.com/filecoin-project/lotus/pull/6672))
- Get retrieval pricing input should not error out on a deal state fetch ([filecoin-project/lotus#6679](https://github.com/filecoin-project/lotus/pull/6679))
- Fix more CID double-encoding as hex ([filecoin-project/lotus#6680](https://github.com/filecoin-project/lotus/pull/6680))
- add an incremental nonce itest. ([filecoin-project/lotus#6663](https://github.com/filecoin-project/lotus/pull/6663))
- storage: Fix FinalizeSector with sectors in stoage paths ([filecoin-project/lotus#6653](https://github.com/filecoin-project/lotus/pull/6653))
- Fix tiny error in check-client-datacap ([filecoin-project/lotus#6664](https://github.com/filecoin-project/lotus/pull/6664))
- Fix: precommit_batch method used the wrong cfg.CommitBatchWait ([filecoin-project/lotus#6658](https://github.com/filecoin-project/lotus/pull/6658))
- fix ticket expiration check ([filecoin-project/lotus#6635](https://github.com/filecoin-project/lotus/pull/6635))
- commit batch: AggregateAboveBaseFee config ([filecoin-project/lotus#6650](https://github.com/filecoin-project/lotus/pull/6650))
- commit batch: Initialize the FailedSectors map ([filecoin-project/lotus#6647](https://github.com/filecoin-project/lotus/pull/6647))
- Fast-path retry submitting commit aggregate if commit is still valid ([filecoin-project/lotus#6638](https://github.com/filecoin-project/lotus/pull/6638))
- remove precommit check in handleCommitFailed ([filecoin-project/lotus#6634](https://github.com/filecoin-project/lotus/pull/6634))
- Reuse timers in sealing batch logic ([filecoin-project/lotus#6636](https://github.com/filecoin-project/lotus/pull/6636))
- shed tool to estimate aggregate network fees ([filecoin-project/lotus#6631](https://github.com/filecoin-project/lotus/pull/6631))
- fix prove commit aggregate send token amount ([filecoin-project/lotus#6625](https://github.com/filecoin-project/lotus/pull/6625))
- Update version.go to 1.11.1 ([filecoin-project/lotus#6621](https://github.com/filecoin-project/lotus/pull/6621))
- github.com/filecoin-project/go-data-transfer (v1.6.0 -> v1.7.0):
- release: v1.7.0
- Fire a transfer queued event when a transfer is queued in Graphsync (#221) ([filecoin-project/go-data-transfer#221](https://github.com/filecoin-project/go-data-transfer/pull/221))
- feat: pass ChannelID to ValidatePush & ValidatePull (#220) ([filecoin-project/go-data-transfer#220](https://github.com/filecoin-project/go-data-transfer/pull/220))
- release: v1.6.1 ([filecoin-project/go-data-transfer#218](https://github.com/filecoin-project/go-data-transfer/pull/218))
- Remove CID lists (#217) ([filecoin-project/go-data-transfer#217](https://github.com/filecoin-project/go-data-transfer/pull/217))
- Merge v1.6.0 ([filecoin-project/go-data-transfer#214](https://github.com/filecoin-project/go-data-transfer/pull/214))
- Remove restart ack timeout (#211) ([filecoin-project/go-data-transfer#211](https://github.com/filecoin-project/go-data-transfer/pull/211))
- feat: use different extension names to fit multiple hooks data in same graphsync message (#204) ([filecoin-project/go-data-transfer#204](https://github.com/filecoin-project/go-data-transfer/pull/204))
- fix: map race in GS transport (#208) ([filecoin-project/go-data-transfer#208](https://github.com/filecoin-project/go-data-transfer/pull/208))
- refactor: simplify graphsync transport (#203) ([filecoin-project/go-data-transfer#203](https://github.com/filecoin-project/go-data-transfer/pull/203))
- release: v1.5.0 (#200) ([filecoin-project/go-data-transfer#200](https://github.com/filecoin-project/go-data-transfer/pull/200))
- github.com/filecoin-project/go-fil-markets (v1.5.0 -> v1.6.0):
- release: v1.6.0
- support padding out smaller files (#536) ([filecoin-project/go-fil-markets#536](https://github.com/filecoin-project/go-fil-markets/pull/536))
- On overloaded CI 10 seconds just isn't enough (#587) ([filecoin-project/go-fil-markets#587](https://github.com/filecoin-project/go-fil-markets/pull/587))
- Do not hex-encode CIDs in logs (#561) ([filecoin-project/go-fil-markets#561](https://github.com/filecoin-project/go-fil-markets/pull/561))
- remove wrong peer check in push deal validation (#585) ([filecoin-project/go-fil-markets#585](https://github.com/filecoin-project/go-fil-markets/pull/585))
- fix: circleci docs-gen task (#574) ([filecoin-project/go-fil-markets#574](https://github.com/filecoin-project/go-fil-markets/pull/574))
- Storage market request queued event and validation interface changes (#555) ([filecoin-project/go-fil-markets#555](https://github.com/filecoin-project/go-fil-markets/pull/555))
- build(deps): bump ws from 6.2.1 to 6.2.2 (#554) ([filecoin-project/go-fil-markets#554](https://github.com/filecoin-project/go-fil-markets/pull/554))
- release: v1.5.0 ([filecoin-project/go-fil-markets#553](https://github.com/filecoin-project/go-fil-markets/pull/553))
- github.com/filecoin-project/go-padreader (v0.0.0-20200903213702-ed5fae088b20 -> v0.0.0-20210723183308-812a16dc01b1):
- New method to pad harder (#6) ([filecoin-project/go-padreader#6](https://github.com/filecoin-project/go-padreader/pull/6))
- Create SECURITY.md (#5) ([filecoin-project/go-padreader#5](https://github.com/filecoin-project/go-padreader/pull/5))
- github.com/filecoin-project/go-state-types (v0.1.1-0.20210506134452-99b279731c48 -> v0.1.1-0.20210722133031-ad9bfe54c124):
- Add version 6.5 (#30) ([filecoin-project/go-state-types#30](https://github.com/filecoin-project/go-state-types/pull/30))
- rename file
- github.com/filecoin-project/specs-actors/v5 (v5.0.1 -> v5.0.3):
- Adjust code for subtle change in go-multihash 0.0.15 (#1463) ([filecoin-project/specs-actors#1463](https://github.com/filecoin-project/specs-actors/pull/1463))
- Bump go state types (#1464) ([filecoin-project/specs-actors#1464](https://github.com/filecoin-project/specs-actors/pull/1464))
- Create CODEOWNERS (#1465) ([filecoin-project/specs-actors#1465](https://github.com/filecoin-project/specs-actors/pull/1465))
- Test deterministic offset (#1462) ([filecoin-project/specs-actors#1462](https://github.com/filecoin-project/specs-actors/pull/1462))
Contributors
| Contributor | Commits | Lines ± | Files Changed |
|-------------|---------|---------|---------------|
| vyzo | 295 | +8700/-5936 | 397 |
| Anton Evangelatov | 94 | +4680/-2965 | 277 |
| Łukasz Magiera | 37 | +3851/-1611 | 146 |
| Mike Greenberg | 1 | +2310/-578 | 8 |
| dirkmc | 7 | +1154/-726 | 29 |
| Jennifer Wang | 9 | +485/-341 | 26 |
| Peter Rabbitson | 18 | +469/-273 | 64 |
| Cory Schwartz | 5 | +576/-135 | 14 |
| hunjixin | 7 | +404/-82 | 19 |
| ZenGround0 | 17 | +284/-135 | 44 |
| Dirk McCormick | 17 | +348/-47 | 17 |
| Raúl Kripalani | 18 | +254/-97 | 62 |
| tchardin | 1 | +261/-33 | 4 |
| Jakub Sztandera | 4 | +254/-16 | 4 |
| Aarsh Shah | 2 | +196/-40 | 28 |
| whyrusleeping | 3 | +150/-9 | 8 |
| Whyrusleeping | 2 | +87/-66 | 10 |
| Aayush Rajasekaran | 10 | +81/-53 | 13 |
| zgfzgf | 2 | +104/-4 | 2 |
| aarshkshah1992 | 4 | +73/-7 | 6 |
| llifezou | 4 | +59/-20 | 4 |
| Steven Allen | 7 | +47/-17 | 9 |
| johnli-helloworld | 3 | +46/-15 | 5 |
| frrist | 1 | +28/-23 | 2 |
| Jennifer | 4 | +31/-2 | 4 |
| wangchao | 1 | +1/-27 | 1 |
| Jiaying Wang | 2 | +7/-21 | 2 |
| hannahhoward | 3 | +21/-2 | 3 |
| chadwick2143 | 1 | +15/-1 | 1 |
| Jerry | 2 | +9/-4 | 2 |
| Steve Loeppky | 2 | +12/-0 | 2 |
| David Dias | 1 | +9/-0 | 1 |
| dependabot[bot] | 1 | +3/-3 | 1 |
| zhoutian527 | 1 | +2/-2 | 1 |
| xloem | 1 | +4/-0 | 1 |
| Travis Person | 2 | +2/-2 | 3 |
| Liviu Damian | 2 | +2/-2 | 2 |
| Jim Pick | 2 | +2/-2 | 2 |
| Frank | 1 | +3/-0 | 1 |
| turuslan | 1 | +1/-1 | 1 |
| Kirk Baird | 1 | +0/-0 | 1 |
# 1.11.0 / 2021-07-22
This is a **highly recommended** release of Lotus that have many bug fixes, improvements and new features.
@@ -434,13 +253,6 @@ Contributors
| @zhoutian527 | 1 | +2/-2 | 1 |
| @ribasushi| 1 | +1/-1 | 1 |
<<<<<<< HEAD
||||||| merged common ancestors
>>>>>>>>> Temporary merge branch 2
=======
>>>>>>> releases
>>>>>>> releases
# 1.10.0 / 2021-06-23
This is a mandatory release of Lotus that introduces Filecoin network v13, codenamed the HyperDrive upgrade. The
+3 -80
View File
@@ -13,13 +13,12 @@ import (
"github.com/filecoin-project/go-address"
datatransfer "github.com/filecoin-project/go-data-transfer"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/specs-actors/v2/actors/builtin/market"
"github.com/filecoin-project/specs-storage/storage"
"github.com/filecoin-project/go-fil-markets/piecestore"
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
"github.com/filecoin-project/go-fil-markets/storagemarket"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/specs-actors/v2/actors/builtin/market"
"github.com/filecoin-project/specs-storage/storage"
"github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/extern/sector-storage/fsutil"
@@ -167,52 +166,6 @@ type StorageMiner interface {
MarketPendingDeals(ctx context.Context) (PendingDealInfo, error) //perm:write
MarketPublishPendingDeals(ctx context.Context) error //perm:admin
// DagstoreListShards returns information about all shards known to the
// DAG store. Only available on nodes running the markets subsystem.
DagstoreListShards(ctx context.Context) ([]DagstoreShardInfo, error) //perm:read
// DagstoreInitializeShard initializes an uninitialized shard.
//
// Initialization consists of fetching the shard's data (deal payload) from
// the storage subsystem, generating an index, and persisting the index
// to facilitate later retrievals, and/or to publish to external sources.
//
// This operation is intended to complement the initial migration. The
// migration registers a shard for every unique piece CID, with lazy
// initialization. Thus, shards are not initialized immediately to avoid
// IO activity competing with proving. Instead, shard are initialized
// when first accessed. This method forces the initialization of a shard by
// accessing it and immediately releasing it. This is useful to warm up the
// cache to facilitate subsequent retrievals, and to generate the indexes
// to publish them externally.
//
// This operation fails if the shard is not in ShardStateNew state.
// It blocks until initialization finishes.
DagstoreInitializeShard(ctx context.Context, key string) error //perm:write
// DagstoreRecoverShard attempts to recover a failed shard.
//
// This operation fails if the shard is not in ShardStateErrored state.
// It blocks until recovery finishes. If recovery failed, it returns the
// error.
DagstoreRecoverShard(ctx context.Context, key string) error //perm:write
// DagstoreInitializeAll initializes all uninitialized shards in bulk,
// according to the policy passed in the parameters.
//
// It is recommended to set a maximum concurrency to avoid extreme
// IO pressure if the storage subsystem has a large amount of deals.
//
// It returns a stream of events to report progress.
DagstoreInitializeAll(ctx context.Context, params DagstoreInitializeAllParams) (<-chan DagstoreInitializeAllEvent, error) //perm:write
// DagstoreGC runs garbage collection on the DAG store.
DagstoreGC(ctx context.Context) ([]DagstoreShardResult, error) //perm:admin
// RuntimeSubsystems returns the subsystems that are enabled
// in this instance.
RuntimeSubsystems(ctx context.Context) (MinerSubsystems, error) //perm:read
DealsImportData(ctx context.Context, dealPropCid cid.Cid, file string) error //perm:admin
DealsList(ctx context.Context) ([]MarketDeal, error) //perm:admin
DealsConsiderOnlineStorageDeals(context.Context) (bool, error) //perm:admin
@@ -379,33 +332,3 @@ type DealSchedule struct {
StartEpoch abi.ChainEpoch
EndEpoch abi.ChainEpoch
}
// DagstoreShardInfo is the serialized form of dagstore.DagstoreShardInfo that
// we expose through JSON-RPC to avoid clients having to depend on the
// dagstore lib.
type DagstoreShardInfo struct {
Key string
State string
Error string
}
// DagstoreShardResult enumerates results per shard.
type DagstoreShardResult struct {
Key string
Success bool
Error string
}
type DagstoreInitializeAllParams struct {
MaxConcurrency int
}
// DagstoreInitializeAllEvent represents an initialization event.
type DagstoreInitializeAllEvent struct {
Key string
Event string // "start", "end"
Success bool
Error string
Total int
Current int
}
+1 -1
View File
@@ -76,7 +76,7 @@ func TestReturnTypes(t *testing.T) {
seen[typ] = struct{}{}
if typ.Kind() == reflect.Interface && typ != bareIface && !typ.Implements(jmarsh) {
t.Error("methods can't return interfaces or struct types not implementing json.Marshaller", m.Name)
t.Error("methods can't return interfaces", m.Name)
}
switch typ.Kind() {
+8 -25
View File
@@ -14,18 +14,17 @@ import (
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-bitfield"
"github.com/filecoin-project/go-multistore"
"github.com/filecoin-project/lotus/node/repo/importmgr"
"github.com/google/uuid"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-filestore"
"github.com/libp2p/go-libp2p-core/metrics"
metrics "github.com/libp2p/go-libp2p-core/metrics"
"github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/protocol"
protocol "github.com/libp2p/go-libp2p-core/protocol"
pubsub "github.com/libp2p/go-libp2p-pubsub"
"github.com/multiformats/go-multiaddr"
"github.com/filecoin-project/lotus/node/repo/importmgr"
datatransfer "github.com/filecoin-project/go-data-transfer"
filestore2 "github.com/filecoin-project/go-fil-markets/filestore"
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
@@ -48,12 +47,11 @@ import (
)
var ExampleValues = map[reflect.Type]interface{}{
reflect.TypeOf(api.MinerSubsystem(0)): api.MinerSubsystem(1),
reflect.TypeOf(auth.Permission("")): auth.Permission("write"),
reflect.TypeOf(""): "string value",
reflect.TypeOf(uint64(42)): uint64(42),
reflect.TypeOf(byte(7)): byte(7),
reflect.TypeOf([]byte{}): []byte("byte array"),
reflect.TypeOf(auth.Permission("")): auth.Permission("write"),
reflect.TypeOf(""): "string value",
reflect.TypeOf(uint64(42)): uint64(42),
reflect.TypeOf(byte(7)): byte(7),
reflect.TypeOf([]byte{}): []byte("byte array"),
}
func addExample(v interface{}) {
@@ -270,21 +268,6 @@ func init() {
addExample(api.CheckStatusCode(0))
addExample(map[string]interface{}{"abc": 123})
addExample(api.MinerSubsystems{
api.SubsystemMining,
api.SubsystemSealing,
api.SubsystemSectorStorage,
api.SubsystemMarkets,
})
addExample(api.DagstoreShardResult{
Key: "baga6ea4seaqecmtz7iak33dsfshi627abz4i4665dfuzr3qfs4bmad6dx3iigdq",
Error: "<error>",
})
addExample(api.DagstoreShardInfo{
Key: "baga6ea4seaqecmtz7iak33dsfshi627abz4i4665dfuzr3qfs4bmad6dx3iigdq",
State: "ShardStateAvailable",
Error: "<error>",
})
}
func GetAPIType(name, pkg string) (i interface{}, t reflect.Type, permStruct []reflect.Type) {
-79
View File
@@ -1,79 +0,0 @@
package api
import (
"encoding/json"
)
// MinerSubsystem represents a miner subsystem. Int and string values are not
// guaranteed to be stable over time is not
// guaranteed to be stable over time.
type MinerSubsystem int
const (
// SubsystemUnknown is a placeholder for the zero value. It should never
// be used.
SubsystemUnknown MinerSubsystem = iota
// SubsystemMarkets signifies the storage and retrieval
// deal-making subsystem.
SubsystemMarkets
// SubsystemMining signifies the mining subsystem.
SubsystemMining
// SubsystemSealing signifies the sealing subsystem.
SubsystemSealing
// SubsystemSectorStorage signifies the sector storage subsystem.
SubsystemSectorStorage
)
var MinerSubsystemToString = map[MinerSubsystem]string{
SubsystemUnknown: "Unknown",
SubsystemMarkets: "Markets",
SubsystemMining: "Mining",
SubsystemSealing: "Sealing",
SubsystemSectorStorage: "SectorStorage",
}
var MinerSubsystemToID = map[string]MinerSubsystem{
"Unknown": SubsystemUnknown,
"Markets": SubsystemMarkets,
"Mining": SubsystemMining,
"Sealing": SubsystemSealing,
"SectorStorage": SubsystemSectorStorage,
}
func (ms MinerSubsystem) MarshalJSON() ([]byte, error) {
return json.Marshal(MinerSubsystemToString[ms])
}
func (ms *MinerSubsystem) UnmarshalJSON(b []byte) error {
var j string
err := json.Unmarshal(b, &j)
if err != nil {
return err
}
s, ok := MinerSubsystemToID[j]
if !ok {
*ms = SubsystemUnknown
} else {
*ms = s
}
return nil
}
type MinerSubsystems []MinerSubsystem
func (ms MinerSubsystems) Has(entry MinerSubsystem) bool {
for _, v := range ms {
if v == entry {
return true
}
}
return false
}
func (ms MinerSubsystem) String() string {
s, ok := MinerSubsystemToString[ms]
if !ok {
return MinerSubsystemToString[SubsystemUnknown]
}
return s
}
-78
View File
@@ -603,16 +603,6 @@ type StorageMinerStruct struct {
CreateBackup func(p0 context.Context, p1 string) error `perm:"admin"`
DagstoreGC func(p0 context.Context) ([]DagstoreShardResult, error) `perm:"admin"`
DagstoreInitializeAll func(p0 context.Context, p1 DagstoreInitializeAllParams) (<-chan DagstoreInitializeAllEvent, error) `perm:"write"`
DagstoreInitializeShard func(p0 context.Context, p1 string) error `perm:"write"`
DagstoreListShards func(p0 context.Context) ([]DagstoreShardInfo, error) `perm:"read"`
DagstoreRecoverShard func(p0 context.Context, p1 string) error `perm:"write"`
DealsConsiderOfflineRetrievalDeals func(p0 context.Context) (bool, error) `perm:"admin"`
DealsConsiderOfflineStorageDeals func(p0 context.Context) (bool, error) `perm:"admin"`
@@ -709,8 +699,6 @@ type StorageMinerStruct struct {
ReturnUnsealPiece func(p0 context.Context, p1 storiface.CallID, p2 *storiface.CallError) error `perm:"admin"`
RuntimeSubsystems func(p0 context.Context) (MinerSubsystems, error) `perm:"read"`
SealingAbort func(p0 context.Context, p1 storiface.CallID) error `perm:"admin"`
SealingSchedDiag func(p0 context.Context, p1 bool) (interface{}, error) `perm:"admin"`
@@ -3579,61 +3567,6 @@ func (s *StorageMinerStub) CreateBackup(p0 context.Context, p1 string) error {
return ErrNotSupported
}
func (s *StorageMinerStruct) DagstoreGC(p0 context.Context) ([]DagstoreShardResult, error) {
if s.Internal.DagstoreGC == nil {
return *new([]DagstoreShardResult), ErrNotSupported
}
return s.Internal.DagstoreGC(p0)
}
func (s *StorageMinerStub) DagstoreGC(p0 context.Context) ([]DagstoreShardResult, error) {
return *new([]DagstoreShardResult), ErrNotSupported
}
func (s *StorageMinerStruct) DagstoreInitializeAll(p0 context.Context, p1 DagstoreInitializeAllParams) (<-chan DagstoreInitializeAllEvent, error) {
if s.Internal.DagstoreInitializeAll == nil {
return nil, ErrNotSupported
}
return s.Internal.DagstoreInitializeAll(p0, p1)
}
func (s *StorageMinerStub) DagstoreInitializeAll(p0 context.Context, p1 DagstoreInitializeAllParams) (<-chan DagstoreInitializeAllEvent, error) {
return nil, ErrNotSupported
}
func (s *StorageMinerStruct) DagstoreInitializeShard(p0 context.Context, p1 string) error {
if s.Internal.DagstoreInitializeShard == nil {
return ErrNotSupported
}
return s.Internal.DagstoreInitializeShard(p0, p1)
}
func (s *StorageMinerStub) DagstoreInitializeShard(p0 context.Context, p1 string) error {
return ErrNotSupported
}
func (s *StorageMinerStruct) DagstoreListShards(p0 context.Context) ([]DagstoreShardInfo, error) {
if s.Internal.DagstoreListShards == nil {
return *new([]DagstoreShardInfo), ErrNotSupported
}
return s.Internal.DagstoreListShards(p0)
}
func (s *StorageMinerStub) DagstoreListShards(p0 context.Context) ([]DagstoreShardInfo, error) {
return *new([]DagstoreShardInfo), ErrNotSupported
}
func (s *StorageMinerStruct) DagstoreRecoverShard(p0 context.Context, p1 string) error {
if s.Internal.DagstoreRecoverShard == nil {
return ErrNotSupported
}
return s.Internal.DagstoreRecoverShard(p0, p1)
}
func (s *StorageMinerStub) DagstoreRecoverShard(p0 context.Context, p1 string) error {
return ErrNotSupported
}
func (s *StorageMinerStruct) DealsConsiderOfflineRetrievalDeals(p0 context.Context) (bool, error) {
if s.Internal.DealsConsiderOfflineRetrievalDeals == nil {
return false, ErrNotSupported
@@ -4162,17 +4095,6 @@ func (s *StorageMinerStub) ReturnUnsealPiece(p0 context.Context, p1 storiface.Ca
return ErrNotSupported
}
func (s *StorageMinerStruct) RuntimeSubsystems(p0 context.Context) (MinerSubsystems, error) {
if s.Internal.RuntimeSubsystems == nil {
return *new(MinerSubsystems), ErrNotSupported
}
return s.Internal.RuntimeSubsystems(p0)
}
func (s *StorageMinerStub) RuntimeSubsystems(p0 context.Context) (MinerSubsystems, error) {
return *new(MinerSubsystems), ErrNotSupported
}
func (s *StorageMinerStruct) SealingAbort(p0 context.Context, p1 storiface.CallID) error {
if s.Internal.SealingAbort == nil {
return ErrNotSupported
+30 -59
View File
@@ -258,16 +258,16 @@ func (b *Blockstore) movingGC() error {
b.moveCond.Broadcast()
b.moveMx.Unlock()
var newPath string
var path string
defer func() {
b.lockMove()
dbNext := b.dbNext
db2 := b.dbNext
b.dbNext = nil
var state bsMoveState
if dbNext != nil {
if db2 != nil {
state = moveStateCleanup
} else {
state = moveStateNone
@@ -275,13 +275,12 @@ func (b *Blockstore) movingGC() error {
b.unlockMove(state)
if dbNext != nil {
// the move failed and we have a left-over db; delete it.
err := dbNext.Close()
if db2 != nil {
err := db2.Close()
if err != nil {
log.Warnf("error closing badger db: %s", err)
}
b.deleteDB(newPath)
b.deleteDB(path)
b.lockMove()
b.unlockMove(moveStateNone)
@@ -297,96 +296,68 @@ func (b *Blockstore) movingGC() error {
}
if basePath == linkPath {
newPath = basePath
path = basePath
} else {
// we do this dance to create a name adjacent to the current one, while avoiding clown
// shoes with multiple moves (i.e. we can't just take the basename of the linkPath, as it
// could have been created in a previous move and have the timestamp suffix, which would then
// perpetuate itself.
name := filepath.Base(basePath)
dir := filepath.Dir(linkPath)
newPath = filepath.Join(dir, name)
path = filepath.Join(dir, name)
}
newPath = fmt.Sprintf("%s.%d", newPath, time.Now().UnixNano())
path = fmt.Sprintf("%s.%d", path, time.Now().UnixNano())
log.Infof("moving blockstore from %s to %s", b.opts.Dir, newPath)
log.Infof("moving blockstore from %s to %s", b.opts.Dir, path)
opts := b.opts
opts.Dir = newPath
opts.ValueDir = newPath
opts.Dir = path
opts.ValueDir = path
dbNew, err := badger.Open(opts.Options)
db2, err := badger.Open(opts.Options)
if err != nil {
return fmt.Errorf("failed to open badger blockstore in %s: %w", newPath, err)
return fmt.Errorf("failed to open badger blockstore in %s: %w", path, err)
}
b.lockMove()
b.dbNext = dbNew
b.dbNext = db2
b.unlockMove(moveStateMoving)
log.Info("copying blockstore")
err = b.doCopy(b.db, b.dbNext)
if err != nil {
return fmt.Errorf("error moving badger blockstore to %s: %w", newPath, err)
return fmt.Errorf("error moving badger blockstore to %s: %w", path, err)
}
b.lockMove()
dbOld := b.db
db1 := b.db
b.db = b.dbNext
b.dbNext = nil
b.unlockMove(moveStateCleanup)
err = dbOld.Close()
err = db1.Close()
if err != nil {
log.Warnf("error closing old badger db: %s", err)
}
// this is the canonical db path; this is where our db lives.
dbPath := b.opts.Dir
dbpath := b.opts.Dir
oldpath := fmt.Sprintf("%s.old.%d", dbpath, time.Now().Unix())
// we first move the existing db out of the way, and only delete it after we have symlinked the
// new db to the canonical path
backupPath := fmt.Sprintf("%s.old.%d", dbPath, time.Now().Unix())
if err = os.Rename(dbPath, backupPath); err != nil {
if err = os.Rename(dbpath, oldpath); err != nil {
// this is not catastrophic in the sense that we have not lost any data.
// but it is pretty bad, as the db path points to the old db, while we are now using to the new
// db; we can't continue and leave a ticking bomb for the next restart.
// so a panic is appropriate and user can fix.
panic(fmt.Errorf("error renaming old badger db dir from %s to %s: %w; USER ACTION REQUIRED", dbPath, backupPath, err)) //nolint
panic(fmt.Errorf("error renaming old badger db dir from %s to %s: %w; USER ACTION REQUIRED", dbpath, oldpath, err)) //nolint
}
if err = symlink(newPath, dbPath); err != nil {
if err = os.Symlink(path, dbpath); err != nil {
// same here; the db path is pointing to the void. panic and let the user fix.
panic(fmt.Errorf("error symlinking new badger db dir from %s to %s: %w; USER ACTION REQUIRED", newPath, dbPath, err)) //nolint
panic(fmt.Errorf("error symlinking new badger db dir from %s to %s: %w; USER ACTION REQUIRED", path, dbpath, err)) //nolint
}
// the delete follows symlinks
b.deleteDB(backupPath)
b.deleteDB(oldpath)
log.Info("moving blockstore done")
return nil
}
// symlink creates a symlink from path to linkTo; the link is relative if the two are
// in the same directory
func symlink(path, linkTo string) error {
resolvedPathDir, err := filepath.EvalSymlinks(filepath.Dir(path))
if err != nil {
return fmt.Errorf("error resolving links in %s: %w", path, err)
}
resolvedLinkDir, err := filepath.EvalSymlinks(filepath.Dir(linkTo))
if err != nil {
return fmt.Errorf("error resolving links in %s: %w", linkTo, err)
}
if resolvedPathDir == resolvedLinkDir {
path = filepath.Base(path)
}
return os.Symlink(path, linkTo)
}
// doCopy copies a badger blockstore to another, with an optional filter; if the filter
// is not nil, then only cids that satisfy the filter will be copied.
func (b *Blockstore) doCopy(from, to *badger.DB) error {
@@ -419,19 +390,19 @@ func (b *Blockstore) doCopy(from, to *badger.DB) error {
func (b *Blockstore) deleteDB(path string) {
// follow symbolic links, otherwise the data wil be left behind
linkPath, err := filepath.EvalSymlinks(path)
lpath, err := filepath.EvalSymlinks(path)
if err != nil {
log.Warnf("error resolving symlinks in %s", path)
return
}
log.Infof("removing data directory %s", linkPath)
if err := os.RemoveAll(linkPath); err != nil {
log.Warnf("error deleting db at %s: %s", linkPath, err)
log.Infof("removing data directory %s", lpath)
if err := os.RemoveAll(lpath); err != nil {
log.Warnf("error deleting db at %s: %s", lpath, err)
return
}
if path != linkPath {
if path != lpath {
log.Infof("removing link %s", path)
if err := os.Remove(path); err != nil {
log.Warnf("error removing symbolic link %s", err)
-15
View File
@@ -245,21 +245,6 @@ func testMove(t *testing.T, optsF func(string) Options) {
checkBlocks()
checkPath()
// reopen the db to make sure our relative link works:
err = db.Close()
if err != nil {
t.Fatal(err)
}
db, err = Open(optsF(dbPath))
if err != nil {
t.Fatal(err)
}
// db.Close() is already deferred
checkBlocks()
}
func TestMoveNoPrefix(t *testing.T) {
Binary file not shown.
Binary file not shown.
Binary file not shown.
+1 -1
View File
@@ -34,7 +34,7 @@ func buildType() string {
}
// BuildVersion is the local build version, set by build system
const BuildVersion = "1.11.1-m1.3.4"
const BuildVersion = "1.11.1-dev"
func UserVersion() string {
if os.Getenv("LOTUS_VERSION_IGNORE_COMMIT") == "1" {
+2 -3
View File
@@ -113,7 +113,7 @@ var AuthApiInfoToken = &cli.Command{
ti, ok := cctx.App.Metadata["repoType"]
if !ok {
log.Errorf("unknown repo type, are you sure you want to use GetCommonAPI?")
log.Errorf("unknown repo type, are you sure you want to use GetAPI?")
ti = repo.FullNode
}
t, ok := ti.(repo.RepoType)
@@ -128,8 +128,7 @@ var AuthApiInfoToken = &cli.Command{
// TODO: Log in audit log when it is implemented
currentEnv, _, _ := cliutil.EnvsForAPIInfos(t)
fmt.Printf("%s=%s:%s\n", currentEnv, string(token), ainfo.Addr)
fmt.Printf("%s=%s:%s\n", cliutil.EnvForRepo(t), string(token), ainfo.Addr)
return nil
},
}
+1 -2
View File
@@ -44,7 +44,7 @@ func GetFullNodeServices(ctx *cli.Context) (ServicesAPI, error) {
var GetAPIInfo = cliutil.GetAPIInfo
var GetRawAPI = cliutil.GetRawAPI
var GetAPI = cliutil.GetCommonAPI
var GetAPI = cliutil.GetAPI
var DaemonContext = cliutil.DaemonContext
var ReqContext = cliutil.ReqContext
@@ -54,7 +54,6 @@ var GetFullNodeAPIV1 = cliutil.GetFullNodeAPIV1
var GetGatewayAPI = cliutil.GetGatewayAPI
var GetStorageMinerAPI = cliutil.GetStorageMinerAPI
var GetMarketsAPI = cliutil.GetMarketsAPI
var GetWorkerAPI = cliutil.GetWorkerAPI
var CommonCommands = []*cli.Command{
+63 -127
View File
@@ -2,7 +2,6 @@ package cliutil
import (
"context"
"errors"
"fmt"
"net/http"
"net/url"
@@ -28,154 +27,112 @@ const (
metadataTraceContext = "traceContext"
)
// flagsForAPI returns flags passed on the command line with the listen address
// of the API server (only used by the tests), in the order of precedence they
// should be applied for the requested kind of node.
func flagsForAPI(t repo.RepoType) []string {
// The flag passed on the command line with the listen address of the API
// server (only used by the tests)
func flagForAPI(t repo.RepoType) string {
switch t {
case repo.FullNode:
return []string{"api-url"}
return "api-url"
case repo.StorageMiner:
return []string{"miner-api-url"}
return "miner-api-url"
case repo.Worker:
return []string{"worker-api-url"}
case repo.Markets:
// support split markets-miner and monolith deployments.
return []string{"markets-api-url", "miner-api-url"}
return "worker-api-url"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
func flagsForRepo(t repo.RepoType) []string {
func flagForRepo(t repo.RepoType) string {
switch t {
case repo.FullNode:
return []string{"repo"}
return "repo"
case repo.StorageMiner:
return []string{"miner-repo"}
return "miner-repo"
case repo.Worker:
return []string{"worker-repo"}
case repo.Markets:
// support split markets-miner and monolith deployments.
return []string{"markets-repo", "miner-repo"}
return "worker-repo"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
// EnvsForAPIInfos returns the environment variables to use in order of precedence
// to determine the API endpoint of the specified node type.
//
// It returns the current variables and deprecated ones separately, so that
// the user can log a warning when deprecated ones are found to be in use.
func EnvsForAPIInfos(t repo.RepoType) (primary string, fallbacks []string, deprecated []string) {
func EnvForRepo(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "FULLNODE_API_INFO", nil, nil
return "FULLNODE_API_INFO"
case repo.StorageMiner:
// TODO remove deprecated deprecation period
return "MINER_API_INFO", nil, []string{"STORAGE_API_INFO"}
return "MINER_API_INFO"
case repo.Worker:
return "WORKER_API_INFO", nil, nil
case repo.Markets:
// support split markets-miner and monolith deployments.
return "MARKETS_API_INFO", []string{"MINER_API_INFO"}, nil
return "WORKER_API_INFO"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
// TODO remove after deprecation period
func envForRepoDeprecation(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "FULLNODE_API_INFO"
case repo.StorageMiner:
return "STORAGE_API_INFO"
case repo.Worker:
return "WORKER_API_INFO"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
// GetAPIInfo returns the API endpoint to use for the specified kind of repo.
//
// The order of precedence is as follows:
//
// 1. *-api-url command line flags.
// 2. *_API_INFO environment variables
// 3. deprecated *_API_INFO environment variables
// 4. *-repo command line flags.
func GetAPIInfo(ctx *cli.Context, t repo.RepoType) (APIInfo, error) {
// Check if there was a flag passed with the listen address of the API
// server (only used by the tests)
apiFlags := flagsForAPI(t)
for _, f := range apiFlags {
if !ctx.IsSet(f) {
continue
}
strma := ctx.String(f)
apiFlag := flagForAPI(t)
if ctx.IsSet(apiFlag) {
strma := ctx.String(apiFlag)
strma = strings.TrimSpace(strma)
return APIInfo{Addr: strma}, nil
}
//
// Note: it is not correct/intuitive to prefer environment variables over
// CLI flags (repo flags below).
//
primaryEnv, fallbacksEnvs, deprecatedEnvs := EnvsForAPIInfos(t)
env, ok := os.LookupEnv(primaryEnv)
envKey := EnvForRepo(t)
env, ok := os.LookupEnv(envKey)
if !ok {
// TODO remove after deprecation period
envKey = envForRepoDeprecation(t)
env, ok = os.LookupEnv(envKey)
if ok {
log.Warnf("Use deprecation env(%s) value, please use env(%s) instead.", envKey, EnvForRepo(t))
}
}
if ok {
return ParseApiInfo(env), nil
}
for _, env := range deprecatedEnvs {
env, ok := os.LookupEnv(env)
if ok {
log.Warnf("Using deprecated env(%s) value, please use env(%s) instead.", env, primaryEnv)
return ParseApiInfo(env), nil
}
repoFlag := flagForRepo(t)
p, err := homedir.Expand(ctx.String(repoFlag))
if err != nil {
return APIInfo{}, xerrors.Errorf("could not expand home dir (%s): %w", repoFlag, err)
}
repoFlags := flagsForRepo(t)
for _, f := range repoFlags {
// cannot use ctx.IsSet because it ignores default values
path := ctx.String(f)
if path == "" {
continue
}
p, err := homedir.Expand(path)
if err != nil {
return APIInfo{}, xerrors.Errorf("could not expand home dir (%s): %w", f, err)
}
r, err := repo.NewFS(p)
if err != nil {
return APIInfo{}, xerrors.Errorf("could not open repo at path: %s; %w", p, err)
}
exists, err := r.Exists()
if err != nil {
return APIInfo{}, xerrors.Errorf("repo.Exists returned an error: %w", err)
}
if !exists {
return APIInfo{}, errors.New("repo directory does not exist. Make sure your configuration is correct")
}
ma, err := r.APIEndpoint()
if err != nil {
return APIInfo{}, xerrors.Errorf("could not get api endpoint: %w", err)
}
token, err := r.APIToken()
if err != nil {
log.Warnf("Couldn't load CLI token, capabilities may be limited: %v", err)
}
return APIInfo{
Addr: ma.String(),
Token: token,
}, nil
r, err := repo.NewFS(p)
if err != nil {
return APIInfo{}, xerrors.Errorf("could not open repo at path: %s; %w", p, err)
}
for _, env := range fallbacksEnvs {
env, ok := os.LookupEnv(env)
if ok {
return ParseApiInfo(env), nil
}
ma, err := r.APIEndpoint()
if err != nil {
return APIInfo{}, xerrors.Errorf("could not get api endpoint: %w", err)
}
return APIInfo{}, fmt.Errorf("could not determine API endpoint for node type: %v", t)
token, err := r.APIToken()
if err != nil {
log.Warnf("Couldn't load CLI token, capabilities may be limited: %v", err)
}
return APIInfo{
Addr: ma.String(),
Token: token,
}, nil
}
func GetRawAPI(ctx *cli.Context, t repo.RepoType, version string) (string, http.Header, error) {
@@ -196,10 +153,10 @@ func GetRawAPI(ctx *cli.Context, t repo.RepoType, version string) (string, http.
return addr, ainfo.AuthHeader(), nil
}
func GetCommonAPI(ctx *cli.Context) (api.CommonNet, jsonrpc.ClientCloser, error) {
func GetAPI(ctx *cli.Context) (api.CommonNet, jsonrpc.ClientCloser, error) {
ti, ok := ctx.App.Metadata["repoType"]
if !ok {
log.Errorf("unknown repo type, are you sure you want to use GetCommonAPI?")
log.Errorf("unknown repo type, are you sure you want to use GetAPI?")
ti = repo.FullNode
}
t, ok := ti.(repo.RepoType)
@@ -317,27 +274,6 @@ func GetWorkerAPI(ctx *cli.Context) (api.Worker, jsonrpc.ClientCloser, error) {
return client.NewWorkerRPCV0(ctx.Context, addr, headers)
}
func GetMarketsAPI(ctx *cli.Context) (api.StorageMiner, jsonrpc.ClientCloser, error) {
// to support lotus-miner cli tests.
if tn, ok := ctx.App.Metadata["testnode-storage"]; ok {
return tn.(api.StorageMiner), func() {}, nil
}
addr, headers, err := GetRawAPI(ctx, repo.Markets, "v0")
if err != nil {
return nil, nil, err
}
if IsVeryVerbose {
_, _ = fmt.Fprintln(ctx.App.Writer, "using markets API v0 endpoint:", addr)
}
// the markets node is a specialised miner's node, supporting only the
// markets API, which is a subset of the miner API. All non-markets
// operations will error out with "unsupported".
return client.NewStorageMinerRPCV0(ctx.Context, addr, headers)
}
func GetGatewayAPI(ctx *cli.Context) (api.Gateway, jsonrpc.ClientCloser, error) {
addr, headers, err := GetRawAPI(ctx, repo.FullNode, "v1")
if err != nil {
+1 -1
View File
@@ -22,7 +22,7 @@ var WaitApiCmd = &cli.Command{
ctx := ReqContext(cctx)
_, err = api.Version(ctx)
_, err = api.ID(ctx)
if err != nil {
return err
}
-209
View File
@@ -1,209 +0,0 @@
package main
import (
"fmt"
"os"
"github.com/fatih/color"
"github.com/urfave/cli/v2"
"github.com/filecoin-project/lotus/api"
lcli "github.com/filecoin-project/lotus/cli"
"github.com/filecoin-project/lotus/lib/tablewriter"
)
var dagstoreCmd = &cli.Command{
Name: "dagstore",
Usage: "Manage the dagstore on the markets subsystem",
Subcommands: []*cli.Command{
dagstoreListShardsCmd,
dagstoreInitializeShardCmd,
dagstoreRecoverShardCmd,
dagstoreInitializeAllCmd,
dagstoreGcCmd,
},
}
var dagstoreListShardsCmd = &cli.Command{
Name: "list-shards",
Usage: "List all shards known to the dagstore, with their current status",
Action: func(cctx *cli.Context) error {
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
if err != nil {
return err
}
defer closer()
ctx := lcli.ReqContext(cctx)
shards, err := marketsApi.DagstoreListShards(ctx)
if err != nil {
return err
}
if len(shards) == 0 {
return nil
}
tw := tablewriter.New(
tablewriter.Col("Key"),
tablewriter.Col("State"),
tablewriter.Col("Error"),
)
colors := map[string]color.Attribute{
"ShardStateAvailable": color.FgGreen,
"ShardStateServing": color.FgBlue,
"ShardStateErrored": color.FgRed,
"ShardStateNew": color.FgYellow,
}
for _, s := range shards {
m := map[string]interface{}{
"Key": s.Key,
"State": func() string {
if c, ok := colors[s.State]; ok {
return color.New(c).Sprint(s.State)
}
return s.State
}(),
"Error": s.Error,
}
tw.Write(m)
}
return tw.Flush(os.Stdout)
},
}
var dagstoreInitializeShardCmd = &cli.Command{
Name: "initialize-shard",
ArgsUsage: "[key]",
Usage: "Initialize the specified shard",
Action: func(cctx *cli.Context) error {
if cctx.NArg() != 1 {
return fmt.Errorf("must provide a single shard key")
}
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
if err != nil {
return err
}
defer closer()
ctx := lcli.ReqContext(cctx)
return marketsApi.DagstoreInitializeShard(ctx, cctx.Args().First())
},
}
var dagstoreRecoverShardCmd = &cli.Command{
Name: "recover-shard",
ArgsUsage: "[key]",
Usage: "Attempt to recover a shard in errored state",
Action: func(cctx *cli.Context) error {
if cctx.NArg() != 1 {
return fmt.Errorf("must provide a single shard key")
}
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
if err != nil {
return err
}
defer closer()
ctx := lcli.ReqContext(cctx)
return marketsApi.DagstoreRecoverShard(ctx, cctx.Args().First())
},
}
var dagstoreInitializeAllCmd = &cli.Command{
Name: "initialize-all",
Usage: "Initialize all uninitialized shards, streaming results as they're produced",
Flags: []cli.Flag{
&cli.UintFlag{
Name: "concurrency",
Usage: "maximum shards to initialize concurrently at a time; use 0 for unlimited",
Required: true,
},
},
Action: func(cctx *cli.Context) error {
concurrency := cctx.Uint("concurrency")
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
if err != nil {
return err
}
defer closer()
ctx := lcli.ReqContext(cctx)
params := api.DagstoreInitializeAllParams{
MaxConcurrency: int(concurrency),
}
ch, err := marketsApi.DagstoreInitializeAll(ctx, params)
if err != nil {
return err
}
for {
select {
case evt, ok := <-ch:
if !ok {
return nil
}
_, _ = fmt.Fprint(os.Stdout, color.New(color.BgHiBlack).Sprintf("(%d/%d)", evt.Current, evt.Total))
_, _ = fmt.Fprint(os.Stdout, " ")
if evt.Event == "start" {
_, _ = fmt.Fprintln(os.Stdout, evt.Key, color.New(color.Reset).Sprint("STARTING"))
} else {
if evt.Success {
_, _ = fmt.Fprintln(os.Stdout, evt.Key, color.New(color.FgGreen).Sprint("SUCCESS"))
} else {
_, _ = fmt.Fprintln(os.Stdout, evt.Key, color.New(color.FgRed).Sprint("ERROR"), evt.Error)
}
}
case <-ctx.Done():
return fmt.Errorf("aborted")
}
}
},
}
var dagstoreGcCmd = &cli.Command{
Name: "gc",
Usage: "Garbage collect the dagstore",
Action: func(cctx *cli.Context) error {
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
if err != nil {
return err
}
defer closer()
ctx := lcli.ReqContext(cctx)
collected, err := marketsApi.DagstoreGC(ctx)
if err != nil {
return err
}
if len(collected) == 0 {
_, _ = fmt.Fprintln(os.Stdout, "no shards collected")
return nil
}
for _, e := range collected {
if e.Error == "" {
_, _ = fmt.Fprintln(os.Stdout, e.Key, color.New(color.FgGreen).Sprint("SUCCESS"))
} else {
_, _ = fmt.Fprintln(os.Stdout, e.Key, color.New(color.FgRed).Sprint("ERROR"), e.Error)
}
}
return nil
},
}
+72 -113
View File
@@ -21,7 +21,6 @@ import (
"github.com/filecoin-project/go-fil-markets/storagemarket"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-state-types/big"
"github.com/filecoin-project/lotus/api/v0api"
sealing "github.com/filecoin-project/lotus/extern/storage-sealing"
"github.com/filecoin-project/lotus/api"
@@ -50,19 +49,13 @@ var infoCmd = &cli.Command{
}
func infoCmdAct(cctx *cli.Context) error {
minerApi, closer, err := lcli.GetStorageMinerAPI(cctx)
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
defer closer()
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
if err != nil {
return err
}
defer closer()
fullapi, acloser, err := lcli.GetFullNodeAPI(cctx)
api, acloser, err := lcli.GetFullNodeAPI(cctx)
if err != nil {
return err
}
@@ -70,23 +63,9 @@ func infoCmdAct(cctx *cli.Context) error {
ctx := lcli.ReqContext(cctx)
subsystems, err := minerApi.RuntimeSubsystems(ctx)
if err != nil {
return err
}
fmt.Println("Enabled subsystems (from miner API):", subsystems)
subsystems, err = marketsApi.RuntimeSubsystems(ctx)
if err != nil {
return err
}
fmt.Println("Enabled subsystems (from markets API):", subsystems)
fmt.Print("Chain: ")
head, err := fullapi.ChainHead(ctx)
head, err := api.ChainHead(ctx)
if err != nil {
return err
}
@@ -116,38 +95,24 @@ func infoCmdAct(cctx *cli.Context) error {
fmt.Println()
err = handleMiningInfo(ctx, cctx, fullapi, minerApi)
if err != nil {
return err
}
err = handleMarketsInfo(ctx, marketsApi)
if err != nil {
return err
}
return nil
}
func handleMiningInfo(ctx context.Context, cctx *cli.Context, fullapi v0api.FullNode, nodeApi api.StorageMiner) error {
maddr, err := getActorAddress(ctx, cctx)
if err != nil {
return err
}
mact, err := fullapi.StateGetActor(ctx, maddr, types.EmptyTSK)
mact, err := api.StateGetActor(ctx, maddr, types.EmptyTSK)
if err != nil {
return err
}
tbs := blockstore.NewTieredBstore(blockstore.NewAPIBlockstore(fullapi), blockstore.NewMemory())
tbs := blockstore.NewTieredBstore(blockstore.NewAPIBlockstore(api), blockstore.NewMemory())
mas, err := miner.Load(adt.WrapStore(ctx, cbor.NewCborStore(tbs)), mact)
if err != nil {
return err
}
// Sector size
mi, err := fullapi.StateMinerInfo(ctx, maddr, types.EmptyTSK)
mi, err := api.StateMinerInfo(ctx, maddr, types.EmptyTSK)
if err != nil {
return err
}
@@ -155,7 +120,7 @@ func handleMiningInfo(ctx context.Context, cctx *cli.Context, fullapi v0api.Full
ssize := types.SizeStr(types.NewInt(uint64(mi.SectorSize)))
fmt.Printf("Miner: %s (%s sectors)\n", color.BlueString("%s", maddr), ssize)
pow, err := fullapi.StateMinerPower(ctx, maddr, types.EmptyTSK)
pow, err := api.StateMinerPower(ctx, maddr, types.EmptyTSK)
if err != nil {
return err
}
@@ -177,7 +142,7 @@ func handleMiningInfo(ctx context.Context, cctx *cli.Context, fullapi v0api.Full
pow.TotalPower.RawBytePower,
),
)
secCounts, err := fullapi.StateMinerSectorCount(ctx, maddr, types.EmptyTSK)
secCounts, err := api.StateMinerSectorCount(ctx, maddr, types.EmptyTSK)
if err != nil {
return err
}
@@ -254,75 +219,6 @@ func handleMiningInfo(ctx context.Context, cctx *cli.Context, fullapi v0api.Full
fmt.Println()
spendable := big.Zero()
// NOTE: there's no need to unlock anything here. Funds only
// vest on deadline boundaries, and they're unlocked by cron.
lockedFunds, err := mas.LockedFunds()
if err != nil {
return xerrors.Errorf("getting locked funds: %w", err)
}
availBalance, err := mas.AvailableBalance(mact.Balance)
if err != nil {
return xerrors.Errorf("getting available balance: %w", err)
}
spendable = big.Add(spendable, availBalance)
fmt.Printf("Miner Balance: %s\n", color.YellowString("%s", types.FIL(mact.Balance).Short()))
fmt.Printf(" PreCommit: %s\n", types.FIL(lockedFunds.PreCommitDeposits).Short())
fmt.Printf(" Pledge: %s\n", types.FIL(lockedFunds.InitialPledgeRequirement).Short())
fmt.Printf(" Vesting: %s\n", types.FIL(lockedFunds.VestingFunds).Short())
colorTokenAmount(" Available: %s\n", availBalance)
mb, err := fullapi.StateMarketBalance(ctx, maddr, types.EmptyTSK)
if err != nil {
return xerrors.Errorf("getting market balance: %w", err)
}
spendable = big.Add(spendable, big.Sub(mb.Escrow, mb.Locked))
fmt.Printf("Market Balance: %s\n", types.FIL(mb.Escrow).Short())
fmt.Printf(" Locked: %s\n", types.FIL(mb.Locked).Short())
colorTokenAmount(" Available: %s\n", big.Sub(mb.Escrow, mb.Locked))
wb, err := fullapi.WalletBalance(ctx, mi.Worker)
if err != nil {
return xerrors.Errorf("getting worker balance: %w", err)
}
spendable = big.Add(spendable, wb)
color.Cyan("Worker Balance: %s", types.FIL(wb).Short())
if len(mi.ControlAddresses) > 0 {
cbsum := big.Zero()
for _, ca := range mi.ControlAddresses {
b, err := fullapi.WalletBalance(ctx, ca)
if err != nil {
return xerrors.Errorf("getting control address balance: %w", err)
}
cbsum = big.Add(cbsum, b)
}
spendable = big.Add(spendable, cbsum)
fmt.Printf(" Control: %s\n", types.FIL(cbsum).Short())
}
colorTokenAmount("Total Spendable: %s\n", spendable)
fmt.Println()
if !cctx.Bool("hide-sectors-info") {
fmt.Println("Sectors:")
err = sectorsInfo(ctx, nodeApi)
if err != nil {
return err
}
}
// TODO: grab actr state / info
// * Sealed sectors (count / bytes)
// * Power
return nil
}
func handleMarketsInfo(ctx context.Context, nodeApi api.StorageMiner) error {
deals, err := nodeApi.MarketListIncompleteDeals(ctx)
if err != nil {
return err
@@ -386,7 +282,6 @@ func handleMarketsInfo(ctx context.Context, nodeApi api.StorageMiner) error {
return sorted[i].status > sorted[j].status
})
fmt.Println()
fmt.Printf("Storage Deals: %d, %s\n", total.count, types.SizeStr(types.NewInt(total.bytes)))
tw := tabwriter.NewWriter(os.Stdout, 1, 1, 1, ' ', 0)
@@ -414,6 +309,70 @@ func handleMarketsInfo(ctx context.Context, nodeApi api.StorageMiner) error {
fmt.Println()
spendable := big.Zero()
// NOTE: there's no need to unlock anything here. Funds only
// vest on deadline boundaries, and they're unlocked by cron.
lockedFunds, err := mas.LockedFunds()
if err != nil {
return xerrors.Errorf("getting locked funds: %w", err)
}
availBalance, err := mas.AvailableBalance(mact.Balance)
if err != nil {
return xerrors.Errorf("getting available balance: %w", err)
}
spendable = big.Add(spendable, availBalance)
fmt.Printf("Miner Balance: %s\n", color.YellowString("%s", types.FIL(mact.Balance).Short()))
fmt.Printf(" PreCommit: %s\n", types.FIL(lockedFunds.PreCommitDeposits).Short())
fmt.Printf(" Pledge: %s\n", types.FIL(lockedFunds.InitialPledgeRequirement).Short())
fmt.Printf(" Vesting: %s\n", types.FIL(lockedFunds.VestingFunds).Short())
colorTokenAmount(" Available: %s\n", availBalance)
mb, err := api.StateMarketBalance(ctx, maddr, types.EmptyTSK)
if err != nil {
return xerrors.Errorf("getting market balance: %w", err)
}
spendable = big.Add(spendable, big.Sub(mb.Escrow, mb.Locked))
fmt.Printf("Market Balance: %s\n", types.FIL(mb.Escrow).Short())
fmt.Printf(" Locked: %s\n", types.FIL(mb.Locked).Short())
colorTokenAmount(" Available: %s\n", big.Sub(mb.Escrow, mb.Locked))
wb, err := api.WalletBalance(ctx, mi.Worker)
if err != nil {
return xerrors.Errorf("getting worker balance: %w", err)
}
spendable = big.Add(spendable, wb)
color.Cyan("Worker Balance: %s", types.FIL(wb).Short())
if len(mi.ControlAddresses) > 0 {
cbsum := big.Zero()
for _, ca := range mi.ControlAddresses {
b, err := api.WalletBalance(ctx, ca)
if err != nil {
return xerrors.Errorf("getting control address balance: %w", err)
}
cbsum = big.Add(cbsum, b)
}
spendable = big.Add(spendable, cbsum)
fmt.Printf(" Control: %s\n", types.FIL(cbsum).Short())
}
colorTokenAmount("Total Spendable: %s\n", spendable)
fmt.Println()
if !cctx.Bool("hide-sectors-info") {
fmt.Println("Sectors:")
err = sectorsInfo(ctx, nodeApi)
if err != nil {
return err
}
}
// TODO: grab actr state / info
// * Sealed sectors (count / bytes)
// * Power
return nil
}
+5 -6
View File
@@ -17,7 +17,7 @@ import (
"gopkg.in/cheggaaa/pb.v1"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-paramfetch"
paramfetch "github.com/filecoin-project/go-paramfetch"
"github.com/filecoin-project/go-state-types/big"
lapi "github.com/filecoin-project/lotus/api"
@@ -72,9 +72,7 @@ var restoreCmd = &cli.Command{
}
}
repoPath := cctx.String(FlagMinerRepo)
if err := restore(ctx, cctx, repoPath, storageCfg, nil, func(api lapi.FullNode, maddr address.Address, peerid peer.ID, mi miner.MinerInfo) error {
if err := restore(ctx, cctx, storageCfg, nil, func(api lapi.FullNode, maddr address.Address, peerid peer.ID, mi miner.MinerInfo) error {
log.Info("Checking proof parameters")
if err := paramfetch.GetParams(ctx, build.ParametersJSON(), build.SrsJSON(), uint64(mi.SectorSize)); err != nil {
@@ -96,7 +94,7 @@ var restoreCmd = &cli.Command{
},
}
func restore(ctx context.Context, cctx *cli.Context, targetPath string, strConfig *stores.StorageConfig, manageConfig func(*config.StorageMiner) error, after func(api lapi.FullNode, addr address.Address, peerid peer.ID, mi miner.MinerInfo) error) error {
func restore(ctx context.Context, cctx *cli.Context, strConfig *stores.StorageConfig, manageConfig func(*config.StorageMiner) error, after func(api lapi.FullNode, addr address.Address, peerid peer.ID, mi miner.MinerInfo) error) error {
if cctx.Args().Len() != 1 {
return xerrors.Errorf("expected 1 argument")
}
@@ -144,7 +142,8 @@ func restore(ctx context.Context, cctx *cli.Context, targetPath string, strConfi
log.Info("Checking if repo exists")
r, err := repo.NewFS(targetPath)
repoPath := cctx.String(FlagMinerRepo)
r, err := repo.NewFS(repoPath)
if err != nil {
return err
}
+1 -6
View File
@@ -71,12 +71,7 @@ var serviceCmd = &cli.Command{
return xerrors.Errorf("--api-sector-index is required without the sector storage module enabled")
}
repoPath := cctx.String(FlagMarketsRepo)
if repoPath == "" {
return xerrors.Errorf("please provide Lotus markets repo path via flag %s", FlagMarketsRepo)
}
if err := restore(ctx, cctx, repoPath, &stores.StorageConfig{}, func(cfg *config.StorageMiner) error {
if err := restore(ctx, cctx, &stores.StorageConfig{}, func(cfg *config.StorageMiner) error {
cfg.Subsystems.EnableMarkets = es.Contains(MarketsService)
cfg.Subsystems.EnableMining = false
cfg.Subsystems.EnableSealing = false
+4 -25
View File
@@ -5,15 +5,13 @@ import (
"fmt"
"github.com/fatih/color"
cliutil "github.com/filecoin-project/lotus/cli/util"
logging "github.com/ipfs/go-log/v2"
"github.com/urfave/cli/v2"
"go.opencensus.io/trace"
"golang.org/x/xerrors"
cliutil "github.com/filecoin-project/lotus/cli/util"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/build"
lcli "github.com/filecoin-project/lotus/cli"
@@ -24,10 +22,7 @@ import (
var log = logging.Logger("main")
const (
FlagMinerRepo = "miner-repo"
FlagMarketsRepo = "markets-repo"
)
const FlagMinerRepo = "miner-repo"
// TODO remove after deprecation period
const FlagMinerRepoDeprecation = "storagerepo"
@@ -48,7 +43,6 @@ func main() {
lcli.WithCategory("market", storageDealsCmd),
lcli.WithCategory("market", retrievalDealsCmd),
lcli.WithCategory("market", dataTransfersCmd),
lcli.WithCategory("market", dagstoreCmd),
lcli.WithCategory("storage", sectorsCmd),
lcli.WithCategory("storage", provingCmd),
lcli.WithCategory("storage", storageCmd),
@@ -112,29 +106,14 @@ func main() {
Value: "~/.lotusminer", // TODO: Consider XDG_DATA_HOME
Usage: fmt.Sprintf("Specify miner repo path. flag(%s) and env(LOTUS_STORAGE_PATH) are DEPRECATION, will REMOVE SOON", FlagMinerRepoDeprecation),
},
&cli.StringFlag{
Name: FlagMarketsRepo,
EnvVars: []string{"LOTUS_MARKETS_PATH"},
Usage: fmt.Sprintf("Markets repo path"),
},
&cli.BoolFlag{
Name: "call-on-markets",
Usage: "(experimental; may be removed) call this command against a markets node; use only with common commands like net, auth, pprof, etc. whose target may be ambiguous",
},
cliutil.FlagVeryVerbose,
},
Commands: append(local, lcli.CommonCommands...),
Before: func(c *cli.Context) error {
// this command is explicitly called on markets, inform
// common commands by overriding the repoType.
if c.Bool("call-on-markets") {
c.App.Metadata["repoType"] = repo.Markets
}
return nil
},
}
app.Setup()
app.Metadata["repoType"] = repo.StorageMiner
lcli.RunApp(app)
}
+18 -24
View File
@@ -73,7 +73,7 @@ var storageDealSelectionShowCmd = &cli.Command{
Name: "list",
Usage: "List storage deal proposal selection criteria",
Action: func(cctx *cli.Context) error {
smapi, closer, err := lcli.GetMarketsAPI(cctx)
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -100,7 +100,7 @@ var storageDealSelectionResetCmd = &cli.Command{
Name: "reset",
Usage: "Reset storage deal proposal selection criteria to default values",
Action: func(cctx *cli.Context) error {
smapi, closer, err := lcli.GetMarketsAPI(cctx)
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -148,7 +148,7 @@ var storageDealSelectionRejectCmd = &cli.Command{
},
},
Action: func(cctx *cli.Context) error {
smapi, closer, err := lcli.GetMarketsAPI(cctx)
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -215,13 +215,7 @@ var setAskCmd = &cli.Command{
Action: func(cctx *cli.Context) error {
ctx := lcli.DaemonContext(cctx)
minerApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
defer closer()
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -258,12 +252,12 @@ var setAskCmd = &cli.Command{
return xerrors.Errorf("cannot parse max-piece-size to quantity of bytes: %w", err)
}
maddr, err := minerApi.ActorAddress(ctx)
maddr, err := api.ActorAddress(ctx)
if err != nil {
return err
}
ssize, err := minerApi.ActorSectorSize(ctx, maddr)
ssize, err := api.ActorSectorSize(ctx, maddr)
if err != nil {
return err
}
@@ -278,7 +272,7 @@ var setAskCmd = &cli.Command{
return xerrors.Errorf("max piece size (w/bit-padding) %s cannot exceed miner sector size %s", types.SizeStr(types.NewInt(uint64(max))), types.SizeStr(types.NewInt(uint64(smax))))
}
return marketsApi.MarketSetAsk(ctx, types.BigInt(pri), types.BigInt(vpri), abi.ChainEpoch(qty), abi.PaddedPieceSize(min), abi.PaddedPieceSize(max))
return api.MarketSetAsk(ctx, types.BigInt(pri), types.BigInt(vpri), abi.ChainEpoch(qty), abi.PaddedPieceSize(min), abi.PaddedPieceSize(max))
},
}
@@ -295,7 +289,7 @@ var getAskCmd = &cli.Command{
}
defer closer()
smapi, closer, err := lcli.GetMarketsAPI(cctx)
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -358,7 +352,7 @@ var dealsImportDataCmd = &cli.Command{
Usage: "Manually import data for a deal",
ArgsUsage: "<proposal CID> <file>",
Action: func(cctx *cli.Context) error {
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -396,7 +390,7 @@ var dealsListCmd = &cli.Command{
},
},
Action: func(cctx *cli.Context) error {
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -500,7 +494,7 @@ var getBlocklistCmd = &cli.Command{
&CidBaseFlag,
},
Action: func(cctx *cli.Context) error {
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -530,7 +524,7 @@ var setBlocklistCmd = &cli.Command{
ArgsUsage: "[<path-of-file-containing-newline-delimited-piece-CIDs> (optional, will read from stdin if omitted)]",
Flags: []cli.Flag{},
Action: func(cctx *cli.Context) error {
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -576,7 +570,7 @@ var resetBlocklistCmd = &cli.Command{
Usage: "Remove all entries from the miner's piece CID blocklist",
Flags: []cli.Flag{},
Action: func(cctx *cli.Context) error {
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -591,7 +585,7 @@ var setSealDurationCmd = &cli.Command{
Usage: "Set the expected time, in minutes, that you expect sealing sectors to take. Deals that start before this duration will be rejected.",
ArgsUsage: "<minutes>",
Action: func(cctx *cli.Context) error {
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -640,7 +634,7 @@ var marketRestartTransfer = &cli.Command{
if !cctx.Args().Present() {
return cli.ShowCommandHelp(cctx, cctx.Command.Name)
}
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -705,7 +699,7 @@ var marketCancelTransfer = &cli.Command{
if !cctx.Args().Present() {
return cli.ShowCommandHelp(cctx, cctx.Command.Name)
}
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -781,7 +775,7 @@ var transfersListCmd = &cli.Command{
color.NoColor = !cctx.Bool("color")
}
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -848,7 +842,7 @@ var dealsPendingPublish = &cli.Command{
},
},
Action: func(cctx *cli.Context) error {
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
+4 -4
View File
@@ -26,7 +26,7 @@ var piecesListPiecesCmd = &cli.Command{
Name: "list-pieces",
Usage: "list registered pieces",
Action: func(cctx *cli.Context) error {
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -49,7 +49,7 @@ var piecesListCidInfosCmd = &cli.Command{
Name: "list-cids",
Usage: "list registered payload CIDs",
Action: func(cctx *cli.Context) error {
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -76,7 +76,7 @@ var piecesInfoCmd = &cli.Command{
return lcli.ShowHelp(cctx, fmt.Errorf("must specify piece cid"))
}
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -111,7 +111,7 @@ var piecesCidInfoCmd = &cli.Command{
return lcli.ShowHelp(cctx, fmt.Errorf("must specify payload cid"))
}
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
+9 -9
View File
@@ -39,7 +39,7 @@ var retrievalDealSelectionShowCmd = &cli.Command{
Name: "list",
Usage: "List retrieval deal proposal selection criteria",
Action: func(cctx *cli.Context) error {
smapi, closer, err := lcli.GetMarketsAPI(cctx)
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -66,7 +66,7 @@ var retrievalDealSelectionResetCmd = &cli.Command{
Name: "reset",
Usage: "Reset retrieval deal proposal selection criteria to default values",
Action: func(cctx *cli.Context) error {
smapi, closer, err := lcli.GetMarketsAPI(cctx)
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -98,7 +98,7 @@ var retrievalDealSelectionRejectCmd = &cli.Command{
},
},
Action: func(cctx *cli.Context) error {
smapi, closer, err := lcli.GetMarketsAPI(cctx)
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -126,7 +126,7 @@ var retrievalDealsListCmd = &cli.Command{
Name: "list",
Usage: "List all active retrieval deals for this miner",
Action: func(cctx *cli.Context) error {
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -186,7 +186,7 @@ var retrievalSetAskCmd = &cli.Command{
Action: func(cctx *cli.Context) error {
ctx := lcli.DaemonContext(cctx)
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -240,7 +240,7 @@ var retrievalGetAskCmd = &cli.Command{
Action: func(cctx *cli.Context) error {
ctx := lcli.DaemonContext(cctx)
api, closer, err := lcli.GetMarketsAPI(cctx)
api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
@@ -252,13 +252,13 @@ var retrievalGetAskCmd = &cli.Command{
}
w := tabwriter.NewWriter(os.Stdout, 2, 4, 2, ' ', 0)
_, _ = fmt.Fprintf(w, "Price per Byte\tUnseal Price\tPayment Interval\tPayment Interval Increase\n")
fmt.Fprintf(w, "Price per Byte\tUnseal Price\tPayment Interval\tPayment Interval Increase\n")
if ask == nil {
_, _ = fmt.Fprintf(w, "<miner does not have an retrieval ask set>\n")
fmt.Fprintf(w, "<miner does not have an retrieval ask set>\n")
return w.Flush()
}
_, _ = fmt.Fprintf(w, "%s\t%s\t%s\t%s\n",
fmt.Fprintf(w, "%s\t%s\t%s\t%s\n",
types.FIL(ask.PricePerByte),
types.FIL(ask.UnsealPrice),
units.BytesSize(float64(ask.PaymentInterval)),
+1 -1
View File
@@ -396,7 +396,7 @@ var sectorsRefsCmd = &cli.Command{
Name: "refs",
Usage: "List References to sectors",
Action: func(cctx *cli.Context) error {
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
+1 -138
View File
@@ -18,12 +18,6 @@
* [ComputeProof](#ComputeProof)
* [Create](#Create)
* [CreateBackup](#CreateBackup)
* [Dagstore](#Dagstore)
* [DagstoreGC](#DagstoreGC)
* [DagstoreInitializeAll](#DagstoreInitializeAll)
* [DagstoreInitializeShard](#DagstoreInitializeShard)
* [DagstoreListShards](#DagstoreListShards)
* [DagstoreRecoverShard](#DagstoreRecoverShard)
* [Deals](#Deals)
* [DealsConsiderOfflineRetrievalDeals](#DealsConsiderOfflineRetrievalDeals)
* [DealsConsiderOfflineStorageDeals](#DealsConsiderOfflineStorageDeals)
@@ -100,8 +94,6 @@
* [ReturnSealPreCommit1](#ReturnSealPreCommit1)
* [ReturnSealPreCommit2](#ReturnSealPreCommit2)
* [ReturnUnsealPiece](#ReturnUnsealPiece)
* [Runtime](#Runtime)
* [RuntimeSubsystems](#RuntimeSubsystems)
* [Sealing](#Sealing)
* [SealingAbort](#SealingAbort)
* [SealingSchedDiag](#SealingSchedDiag)
@@ -351,113 +343,6 @@ Inputs:
Response: `{}`
## Dagstore
### DagstoreGC
DagstoreGC runs garbage collection on the DAG store.
Perms: admin
Inputs: `null`
Response: `null`
### DagstoreInitializeAll
DagstoreInitializeAll initializes all uninitialized shards in bulk,
according to the policy passed in the parameters.
It is recommended to set a maximum concurrency to avoid extreme
IO pressure if the storage subsystem has a large amount of deals.
It returns a stream of events to report progress.
Perms: write
Inputs:
```json
[
{
"MaxConcurrency": 123
}
]
```
Response:
```json
{
"Key": "string value",
"Event": "string value",
"Success": true,
"Error": "string value",
"Total": 123,
"Current": 123
}
```
### DagstoreInitializeShard
DagstoreInitializeShard initializes an uninitialized shard.
Initialization consists of fetching the shard's data (deal payload) from
the storage subsystem, generating an index, and persisting the index
to facilitate later retrievals, and/or to publish to external sources.
This operation is intended to complement the initial migration. The
migration registers a shard for every unique piece CID, with lazy
initialization. Thus, shards are not initialized immediately to avoid
IO activity competing with proving. Instead, shard are initialized
when first accessed. This method forces the initialization of a shard by
accessing it and immediately releasing it. This is useful to warm up the
cache to facilitate subsequent retrievals, and to generate the indexes
to publish them externally.
This operation fails if the shard is not in ShardStateNew state.
It blocks until initialization finishes.
Perms: write
Inputs:
```json
[
"string value"
]
```
Response: `{}`
### DagstoreListShards
DagstoreListShards returns information about all shards known to the
DAG store. Only available on nodes running the markets subsystem.
Perms: read
Inputs: `null`
Response: `null`
### DagstoreRecoverShard
DagstoreRecoverShard attempts to recover a failed shard.
This operation fails if the shard is not in ShardStateErrored state.
It blocks until recovery finishes. If recovery failed, it returns the
error.
Perms: write
Inputs:
```json
[
"string value"
]
```
Response: `{}`
## Deals
@@ -822,7 +707,7 @@ Response:
"ID": 3
},
"SectorNumber": 9,
"InboundCAR": "string value"
"CARv2FilePath": "string value"
}
```
@@ -1637,28 +1522,6 @@ Inputs:
Response: `{}`
## Runtime
### RuntimeSubsystems
RuntimeSubsystems returns the subsystems that are enabled
in this instance.
Perms: read
Inputs: `null`
Response:
```json
[
"Mining",
"Sealing",
"SectorStorage",
"Markets"
]
```
## Sealing
+1 -92
View File
@@ -7,7 +7,7 @@ USAGE:
lotus-miner [global options] command [command options] [arguments...]
VERSION:
1.11.1-m1.3.4
1.11.1-dev
COMMANDS:
init Initialize a lotus miner repo
@@ -29,7 +29,6 @@ COMMANDS:
storage-deals Manage storage deals and related configuration
retrieval-deals Manage retrieval deals and related configuration
data-transfers Manage data transfers
dagstore Manage the dagstore on the markets subsystem
NETWORK:
net Manage P2P Network
RETRIEVAL:
@@ -44,8 +43,6 @@ GLOBAL OPTIONS:
--actor value, -a value specify other actor to check state for (read only)
--color use color in display output (default: depends on output being a TTY)
--miner-repo value, --storagerepo value Specify miner repo path. flag(storagerepo) and env(LOTUS_STORAGE_PATH) are DEPRECATION, will REMOVE SOON (default: "~/.lotusminer") [$LOTUS_MINER_PATH, $LOTUS_STORAGE_PATH]
--markets-repo value Markets repo path [$LOTUS_MARKETS_PATH]
--call-on-markets (experimental; may be removed) call this command against a markets node; use only with common commands like net, auth, pprof, etc. whose target may be ambiguous (default: false)
--vv enables very verbose mode, useful for debugging the CLI (default: false)
--help, -h show help (default: false)
--version, -v print the version (default: false)
@@ -1001,94 +998,6 @@ OPTIONS:
```
## lotus-miner dagstore
```
NAME:
lotus-miner dagstore - Manage the dagstore on the markets subsystem
USAGE:
lotus-miner dagstore command [command options] [arguments...]
COMMANDS:
list-shards List all shards known to the dagstore, with their current status
initialize-shard Initialize the specified shard
recover-shard Attempt to recover a shard in errored state
initialize-all Initialize all uninitialized shards, streaming results as they're produced
gc Garbage collect the dagstore
help, h Shows a list of commands or help for one command
OPTIONS:
--help, -h show help (default: false)
--version, -v print the version (default: false)
```
### lotus-miner dagstore list-shards
```
NAME:
lotus-miner dagstore list-shards - List all shards known to the dagstore, with their current status
USAGE:
lotus-miner dagstore list-shards [command options] [arguments...]
OPTIONS:
--help, -h show help (default: false)
```
### lotus-miner dagstore initialize-shard
```
NAME:
lotus-miner dagstore initialize-shard - Initialize the specified shard
USAGE:
lotus-miner dagstore initialize-shard [command options] [key]
OPTIONS:
--help, -h show help (default: false)
```
### lotus-miner dagstore recover-shard
```
NAME:
lotus-miner dagstore recover-shard - Attempt to recover a shard in errored state
USAGE:
lotus-miner dagstore recover-shard [command options] [key]
OPTIONS:
--help, -h show help (default: false)
```
### lotus-miner dagstore initialize-all
```
NAME:
lotus-miner dagstore initialize-all - Initialize all uninitialized shards, streaming results as they're produced
USAGE:
lotus-miner dagstore initialize-all [command options] [arguments...]
OPTIONS:
--concurrency value maximum shards to initialize concurrently at a time; use 0 for unlimited (default: 0)
--help, -h show help (default: false)
```
### lotus-miner dagstore gc
```
NAME:
lotus-miner dagstore gc - Garbage collect the dagstore
USAGE:
lotus-miner dagstore gc [command options] [arguments...]
OPTIONS:
--help, -h show help (default: false)
```
## lotus-miner net
```
NAME:
+1 -1
View File
@@ -7,7 +7,7 @@ USAGE:
lotus-worker [global options] command [command options] [arguments...]
VERSION:
1.11.1-m1.3.4
1.11.1-dev
COMMANDS:
run Start lotus worker
+1 -1
View File
@@ -7,7 +7,7 @@ USAGE:
lotus [global options] command [command options] [arguments...]
VERSION:
1.11.1-m1.3.4
1.11.1-dev
COMMANDS:
daemon Start a lotus daemon process
+7 -7
View File
@@ -26,23 +26,23 @@ require (
github.com/elastic/gosigar v0.12.0
github.com/etclabscore/go-openrpc-reflect v0.0.36
github.com/fatih/color v1.9.0
github.com/filecoin-project/dagstore v0.4.2
github.com/filecoin-project/dagstore v0.3.1
github.com/filecoin-project/filecoin-ffi v0.30.4-0.20200910194244-f640612a1a1f
github.com/filecoin-project/go-address v0.0.5
github.com/filecoin-project/go-bitfield v0.2.4
github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2
github.com/filecoin-project/go-commp-utils v0.1.1-0.20210427191551-70bf140d31c7
github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03
github.com/filecoin-project/go-data-transfer v1.7.3
github.com/filecoin-project/go-data-transfer v1.7.0
github.com/filecoin-project/go-fil-commcid v0.1.0
github.com/filecoin-project/go-fil-commp-hashhash v0.1.0
github.com/filecoin-project/go-fil-markets v1.6.3-0.20210806151421-967d8717c917
github.com/filecoin-project/go-fil-markets v1.6.1-0.20210727201249-2213f80f0403
github.com/filecoin-project/go-jsonrpc v0.1.4-0.20210217175800-45ea43ac2bec
github.com/filecoin-project/go-multistore v0.0.3
github.com/filecoin-project/go-padreader v0.0.0-20210723183308-812a16dc01b1
github.com/filecoin-project/go-paramfetch v0.0.2-0.20210614165157-25a6c7769498
github.com/filecoin-project/go-state-types v0.1.1-0.20210722133031-ad9bfe54c124
github.com/filecoin-project/go-statemachine v1.0.1
github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe
github.com/filecoin-project/go-statestore v0.1.1
github.com/filecoin-project/go-storedcounter v0.0.0-20200421200003-1c99c62e8a5b
github.com/filecoin-project/specs-actors v0.9.14
@@ -56,7 +56,7 @@ require (
github.com/gdamore/tcell/v2 v2.2.0
github.com/go-kit/kit v0.10.0
github.com/go-ole/go-ole v1.2.4 // indirect
github.com/golang/mock v1.6.0
github.com/golang/mock v1.5.0
github.com/google/uuid v1.2.0
github.com/gorilla/mux v1.7.4
github.com/gorilla/websocket v1.4.2
@@ -78,7 +78,7 @@ require (
github.com/ipfs/go-ds-pebble v0.0.2-0.20200921225637-ce220f8ac459
github.com/ipfs/go-filestore v1.0.0
github.com/ipfs/go-fs-lock v0.0.6
github.com/ipfs/go-graphsync v0.6.8
github.com/ipfs/go-graphsync v0.6.5
github.com/ipfs/go-ipfs-blockstore v1.0.4
github.com/ipfs/go-ipfs-blocksutil v0.0.1
github.com/ipfs/go-ipfs-chunker v0.0.5
@@ -99,7 +99,7 @@ require (
github.com/ipfs/go-unixfs v0.2.6
github.com/ipfs/interface-go-ipfs-core v0.2.3
github.com/ipld/go-car v0.1.1-0.20201119040415-11b6074b6d4d
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210803143113-d0e44a62f0a3
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210721090610-5a9d1b217d25
github.com/ipld/go-ipld-prime v0.5.1-0.20201021195245-109253e8a018
github.com/kelseyhightower/envconfig v1.4.0
github.com/lib/pq v1.7.0
+11 -15
View File
@@ -256,8 +256,8 @@ github.com/fatih/color v1.8.0/go.mod h1:3l45GVGkyrnYNl9HoIjnp2NnNWvh6hLAqD8yTfGj
github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
github.com/fd/go-nat v1.0.0/go.mod h1:BTBu/CKvMmOMUPkKVef1pngt2WFH/lg7E6yQnulfp6E=
github.com/filecoin-project/dagstore v0.4.2 h1:Ae2+O1DhKCI1JbOZCBkqUksKYofdbRbjkS7OF0A6Jw0=
github.com/filecoin-project/dagstore v0.4.2/go.mod h1:WY5OoLfnwISCk6eASSF927KKPqLPIlTwmG1qHpA08KY=
github.com/filecoin-project/dagstore v0.3.1 h1:IShahbR3zEPE1pfHa5DiYMt/S40b9JuWZx6+kZPuyTQ=
github.com/filecoin-project/dagstore v0.3.1/go.mod h1:WY5OoLfnwISCk6eASSF927KKPqLPIlTwmG1qHpA08KY=
github.com/filecoin-project/go-address v0.0.3/go.mod h1:jr8JxKsYx+lQlQZmF5i2U0Z+cGQ59wMIps/8YW/lDj8=
github.com/filecoin-project/go-address v0.0.5 h1:SSaFT/5aLfPXycUlFyemoHYhRgdyXClXCyDdNJKPlDM=
github.com/filecoin-project/go-address v0.0.5/go.mod h1:jr8JxKsYx+lQlQZmF5i2U0Z+cGQ59wMIps/8YW/lDj8=
@@ -277,9 +277,8 @@ github.com/filecoin-project/go-commp-utils v0.1.1-0.20210427191551-70bf140d31c7/
github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03 h1:2pMXdBnCiXjfCYx/hLqFxccPoqsSveQFxVLvNxy9bus=
github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
github.com/filecoin-project/go-data-transfer v1.0.1/go.mod h1:UxvfUAY9v3ub0a21BSK9u3pB2aq30Y0KMsG+w9/ysyo=
github.com/filecoin-project/go-data-transfer v1.7.0 h1:mFRn+UuTdPROmhplLSekzd4rAs9ug8ubtSY4nw9wYkU=
github.com/filecoin-project/go-data-transfer v1.7.0/go.mod h1:GLRr5BmLEqsLwXfiRDG7uJvph22KGL2M4iOuF8EINaU=
github.com/filecoin-project/go-data-transfer v1.7.3 h1:2F3K1LmT6Q97+Bq9gBv/VVqFZ/0FHsRPqYDbCzrGaYg=
github.com/filecoin-project/go-data-transfer v1.7.3/go.mod h1:Cbl9lzKOuAyyIxp1tE+VbV5Aix4bxzA7uJGA9wGM4fM=
github.com/filecoin-project/go-ds-versioning v0.1.0 h1:y/X6UksYTsK8TLCI7rttCKEvl8btmWxyFMEeeWGUxIQ=
github.com/filecoin-project/go-ds-versioning v0.1.0/go.mod h1:mp16rb4i2QPmxBnmanUx8i/XANp+PFCCJWiAb+VW4/s=
github.com/filecoin-project/go-fil-commcid v0.0.0-20200716160307-8f644712406f/go.mod h1:Eaox7Hvus1JgPrL5+M3+h7aSPHc0cVqpSxA+TxIEpZQ=
@@ -289,8 +288,8 @@ github.com/filecoin-project/go-fil-commcid v0.1.0/go.mod h1:Eaox7Hvus1JgPrL5+M3+
github.com/filecoin-project/go-fil-commp-hashhash v0.1.0 h1:imrrpZWEHRnNqqv0tN7LXep5bFEVOVmQWHJvl2mgsGo=
github.com/filecoin-project/go-fil-commp-hashhash v0.1.0/go.mod h1:73S8WSEWh9vr0fDJVnKADhfIv/d6dCbAGaAGWbdJEI8=
github.com/filecoin-project/go-fil-markets v1.0.5-0.20201113164554-c5eba40d5335/go.mod h1:AJySOJC00JRWEZzRG2KsfUnqEf5ITXxeX09BE9N4f9c=
github.com/filecoin-project/go-fil-markets v1.6.3-0.20210806151421-967d8717c917 h1:A7B4EShovtH4B8tMp6+Smst7B+H+CLSJ4tcjcS15ta4=
github.com/filecoin-project/go-fil-markets v1.6.3-0.20210806151421-967d8717c917/go.mod h1:9DxT29OFLUJPswugvIbqD5Ab768pJQxkxbnTEFTI8lk=
github.com/filecoin-project/go-fil-markets v1.6.1-0.20210727201249-2213f80f0403 h1:Zrr2YRH186x0w4U5kaCYX+ccDD+g09yjYIO6QsICLpA=
github.com/filecoin-project/go-fil-markets v1.6.1-0.20210727201249-2213f80f0403/go.mod h1:13+DUe7AaHekzgpQPbacdppRoqz0SyPlx48g0f/pRmA=
github.com/filecoin-project/go-hamt-ipld v0.1.5 h1:uoXrKbCQZ49OHpsTCkrThPNelC4W3LPEk0OrS/ytIBM=
github.com/filecoin-project/go-hamt-ipld v0.1.5/go.mod h1:6Is+ONR5Cd5R6XZoCse1CWaXZc0Hdb/JeX+EQCQzX24=
github.com/filecoin-project/go-hamt-ipld/v2 v2.0.0 h1:b3UDemBYN2HNfk3KOXNuxgTTxlWi3xVvbQP0IT38fvM=
@@ -315,9 +314,8 @@ github.com/filecoin-project/go-state-types v0.1.0/go.mod h1:ezYnPf0bNkTsDibL/psS
github.com/filecoin-project/go-state-types v0.1.1-0.20210506134452-99b279731c48/go.mod h1:ezYnPf0bNkTsDibL/psSz5dy4B5awOJ/E7P2Saeep8g=
github.com/filecoin-project/go-state-types v0.1.1-0.20210722133031-ad9bfe54c124 h1:veGrNABg/9I7prngrowkhwbvW5d5JN55MNKmbsr5FqA=
github.com/filecoin-project/go-state-types v0.1.1-0.20210722133031-ad9bfe54c124/go.mod h1:ezYnPf0bNkTsDibL/psSz5dy4B5awOJ/E7P2Saeep8g=
github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe h1:dF8u+LEWeIcTcfUcCf3WFVlc81Fr2JKg8zPzIbBDKDw=
github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe/go.mod h1:FGwQgZAt2Gh5mjlwJUlVB62JeYdo+if0xWxSEfBD9ig=
github.com/filecoin-project/go-statemachine v1.0.1 h1:LQ60+JDVjMdLxXmVFM2jjontzOYnfVE7u02CXV3WKSw=
github.com/filecoin-project/go-statemachine v1.0.1/go.mod h1:jZdXXiHa61n4NmgWFG4w8tnqgvZVHYbJ3yW7+y8bF54=
github.com/filecoin-project/go-statestore v0.1.0/go.mod h1:LFc9hD+fRxPqiHiaqUEZOinUJB4WARkRfNl10O7kTnI=
github.com/filecoin-project/go-statestore v0.1.1 h1:ufMFq00VqnT2CAuDpcGnwLnCX1I/c3OROw/kXVNSTZk=
github.com/filecoin-project/go-statestore v0.1.1/go.mod h1:LFc9hD+fRxPqiHiaqUEZOinUJB4WARkRfNl10O7kTnI=
@@ -434,8 +432,8 @@ github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFU
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/mock v1.6.0 h1:ErTB+efbowRARo13NNdxyJji2egdxLGQhRaY+DUumQc=
github.com/golang/mock v1.6.0/go.mod h1:p6yTPP+5HYm5mzsMV8JkE6ZKdX+/wYM6Hr+LicevLPs=
github.com/golang/mock v1.5.0 h1:jlYHihg//f7RRwuPfptm04yp4s7O6Kw8EZiVYIGcH0g=
github.com/golang/mock v1.5.0/go.mod h1:CWnOUgYIOo4TcNZ0wHX3YZCqsaM1I1Jvs6v3mP3KVu8=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.0/go.mod h1:Qd/q+1AKNOZr9uGQzbzCmRO6sUih6GTPZv6a1/R87v0=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -639,8 +637,8 @@ github.com/ipfs/go-graphsync v0.1.0/go.mod h1:jMXfqIEDFukLPZHqDPp8tJMbHO9Rmeb9CE
github.com/ipfs/go-graphsync v0.4.2/go.mod h1:/VmbZTUdUMTbNkgzAiCEucIIAU3BkLE2cZrDCVUhyi0=
github.com/ipfs/go-graphsync v0.4.3/go.mod h1:mPOwDYv128gf8gxPFgXnz4fNrSYPsWyqisJ7ych+XDY=
github.com/ipfs/go-graphsync v0.6.4/go.mod h1:5WyaeigpNdpiYQuW2vwpuecOoEfB4h747ZGEOKmAGTg=
github.com/ipfs/go-graphsync v0.6.8 h1:mgyPdBDPcgL8ujO132grQjP3rfQv+KN/riQzbmTVgg4=
github.com/ipfs/go-graphsync v0.6.8/go.mod h1:GdHT8JeuIZ0R4lSjFR16Oe4zPi5dXwKi9zR9ADVlcdk=
github.com/ipfs/go-graphsync v0.6.5 h1:YAJl6Yit23PQcaawzb1rPK9PSnbbq2jjMRPpRpJ0Y5U=
github.com/ipfs/go-graphsync v0.6.5/go.mod h1:GdHT8JeuIZ0R4lSjFR16Oe4zPi5dXwKi9zR9ADVlcdk=
github.com/ipfs/go-hamt-ipld v0.1.1/go.mod h1:1EZCr2v0jlCnhpa+aZ0JZYp8Tt2w16+JJOAVz17YcDk=
github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08=
github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw=
@@ -759,9 +757,8 @@ github.com/ipld/go-car v0.1.0/go.mod h1:RCWzaUh2i4mOEkB3W45Vc+9jnS/M6Qay5ooytiBH
github.com/ipld/go-car v0.1.1-0.20200923150018-8cdef32e2da4/go.mod h1:xrMEcuSq+D1vEwl+YAXsg/JfA98XGpXDwnkIL4Aimqw=
github.com/ipld/go-car v0.1.1-0.20201119040415-11b6074b6d4d h1:iphSzTuPqyDgH7WUVZsdqUnQNzYgIblsVr1zhVNA33U=
github.com/ipld/go-car v0.1.1-0.20201119040415-11b6074b6d4d/go.mod h1:2Gys8L8MJ6zkh1gktTSXreY63t4UbyvNp5JaudTyxHQ=
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210721090610-5a9d1b217d25 h1:SeNfgKCuIhGJKpaSVvn8WX7uHxWeHvwVUq1MIh2KM2o=
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210721090610-5a9d1b217d25/go.mod h1:I2ACeeg6XNBe5pdh5TaR7Ambhfa7If9KXxmXgZsYENU=
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210803143113-d0e44a62f0a3 h1:TPOuaAmTWpzWFbGPsJ+QJsR6Qc+fnHKgvtA6Jy2A22w=
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210803143113-d0e44a62f0a3/go.mod h1:I2ACeeg6XNBe5pdh5TaR7Ambhfa7If9KXxmXgZsYENU=
github.com/ipld/go-ipld-prime v0.0.2-0.20191108012745-28a82f04c785/go.mod h1:bDDSvVz7vaK12FNvMeRYnpRFkSUPNQOiCYQezMD/P3w=
github.com/ipld/go-ipld-prime v0.0.2-0.20200428162820-8b59dc292b8e/go.mod h1:uVIwe/u0H4VdKv3kaN1ck7uCb6yD9cFLS9/ELyXbsw8=
github.com/ipld/go-ipld-prime v0.5.1-0.20200828233916-988837377a7f/go.mod h1:0xEgdD6MKbZ1vF0GC+YcR/C4SQCAlRuOjIJ2i0HxqzM=
@@ -1989,7 +1986,6 @@ golang.org/x/tools v0.0.0-20200827010519-17fd2f27a9e3/go.mod h1:njjCfa9FT2d7l9Bc
golang.org/x/tools v0.0.0-20201112185108-eeaa07dd7696/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.5 h1:ouewzE6p+/VEB31YYnTbEJdi8pFqKp4P4n85vwo3DHA=
golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+1 -1
View File
@@ -42,4 +42,4 @@ func TestDealsWithSealingAndRPC(t *testing.T) {
dh.RunConcurrentDeals(kit.RunConcurrentDealsOpts{N: 1, UseCARFileForStorageDeal: true})
})
}
}
-33
View File
@@ -1,33 +0,0 @@
package dagstore
import (
"io"
blocks "github.com/ipfs/go-block-format"
"github.com/ipfs/go-cid"
bstore "github.com/ipfs/go-ipfs-blockstore"
"golang.org/x/xerrors"
"github.com/filecoin-project/dagstore"
)
// Blockstore promotes a dagstore.ReadBlockstore to a full closeable Blockstore,
// stubbing out the write methods with erroring implementations.
type Blockstore struct {
dagstore.ReadBlockstore
io.Closer
}
var _ bstore.Blockstore = (*Blockstore)(nil)
func (b *Blockstore) DeleteBlock(c cid.Cid) error {
return xerrors.Errorf("DeleteBlock called but not implemented")
}
func (b *Blockstore) Put(block blocks.Block) error {
return xerrors.Errorf("Put called but not implemented")
}
func (b *Blockstore) PutMany(blocks []blocks.Block) error {
return xerrors.Errorf("PutMany called but not implemented")
}
+145
View File
@@ -0,0 +1,145 @@
package dagstore
import (
"context"
"io"
"github.com/filecoin-project/dagstore/throttle"
"github.com/ipfs/go-cid"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-fil-markets/piecestore"
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
"github.com/filecoin-project/go-fil-markets/shared"
)
// MaxConcurrentUnsealedFetches caps the amount of concurrent fetches for
// unsealed pieces, so that we don't saturate IO or the network too much,
// especially when bulk processing (e.g. at migration).
var MaxConcurrentUnsealedFetches = 3
type LotusAccessor interface {
FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error)
GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid) (uint64, error)
Start(ctx context.Context) error
}
type lotusAccessor struct {
pieceStore piecestore.PieceStore
rm retrievalmarket.RetrievalProviderNode
throttle throttle.Throttler
readyMgr *shared.ReadyManager
}
var _ LotusAccessor = (*lotusAccessor)(nil)
func NewLotusAccessor(store piecestore.PieceStore, rm retrievalmarket.RetrievalProviderNode) LotusAccessor {
return &lotusAccessor{
pieceStore: store,
rm: rm,
throttle: throttle.Fixed(MaxConcurrentUnsealedFetches),
readyMgr: shared.NewReadyManager(),
}
}
func (m *lotusAccessor) Start(ctx context.Context) error {
// Wait for the piece store to startup
ready := make(chan error)
m.pieceStore.OnReady(func(err error) {
select {
case <-ctx.Done():
case ready <- err:
}
})
select {
case <-ctx.Done():
err := xerrors.Errorf("context cancelled waiting for piece store startup: %w", ctx.Err())
if ferr := m.readyMgr.FireReady(err); ferr != nil {
log.Warnw("failed to publish ready event", "err", ferr)
}
return err
case err := <-ready:
if ferr := m.readyMgr.FireReady(err); ferr != nil {
log.Warnw("failed to publish ready event", "err", ferr)
}
return err
}
}
func (m *lotusAccessor) FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error) {
err := m.readyMgr.AwaitReady()
if err != nil {
return nil, err
}
pieceInfo, err := m.pieceStore.GetPieceInfo(pieceCid)
if err != nil {
return nil, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
}
if len(pieceInfo.Deals) == 0 {
return nil, xerrors.Errorf("no storage deals found for Piece %s", pieceCid)
}
// prefer an unsealed sector containing the piece if one exists
for _, deal := range pieceInfo.Deals {
isUnsealed, err := m.rm.IsUnsealed(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
if err != nil {
log.Warnf("failed to check if deal %d unsealed: %s", deal.DealID, err)
continue
}
if isUnsealed {
var reader io.ReadCloser
// We want to throttle this path, as these copies will be downloaded
// immediately from the storage cluster without any unsealing
// necessary.
deal := deal // calm the linter
err := m.throttle.Do(ctx, func(ctx context.Context) (err error) {
// UnsealSector will NOT unseal a sector if we already have an unsealed copy lying around.
reader, err = m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
return err
})
return reader, err
}
}
lastErr := xerrors.New("no sectors found to unseal from")
// if there is no unsealed sector containing the piece, just read the piece from the first sector we are able to unseal.
for _, deal := range pieceInfo.Deals {
// Note that if the deal data is not already unsealed, unsealing may
// block for a long time with the current PoRep
reader, err := m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
if err != nil {
lastErr = xerrors.Errorf("failed to unseal deal %d: %w", deal.DealID, err)
log.Warn(lastErr.Error())
continue
}
// Successfully fetched the deal data so return a reader over the data
return reader, nil
}
return nil, lastErr
}
func (m *lotusAccessor) GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid) (uint64, error) {
err := m.readyMgr.AwaitReady()
if err != nil {
return 0, err
}
pieceInfo, err := m.pieceStore.GetPieceInfo(pieceCid)
if err != nil {
return 0, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
}
if len(pieceInfo.Deals) == 0 {
return 0, xerrors.Errorf("no storage deals found for piece %s", pieceCid)
}
len := pieceInfo.Deals[0].Length
return uint64(len), nil
}
@@ -45,9 +45,7 @@ func TestLotusAccessorFetchUnsealedPiece(t *testing.T) {
name string
deals []abi.SectorNumber
fetchedData string
isUnsealed bool
expectErr bool
expectErr bool
}{{
// Expect error if there is no deal info for piece CID
name: "no deals",
@@ -58,13 +56,11 @@ func TestLotusAccessorFetchUnsealedPiece(t *testing.T) {
name: "prefer unsealed deal",
deals: []abi.SectorNumber{unsealedSectorID, sealedSectorID},
fetchedData: unsealedSectorData,
isUnsealed: true,
}, {
// Expect the API to unseal the data if there are no unsealed deals
name: "unseal if necessary",
deals: []abi.SectorNumber{sealedSectorID},
fetchedData: sealedSectorData,
isUnsealed: false,
}}
for _, tc := range testCases {
@@ -74,7 +70,7 @@ func TestLotusAccessorFetchUnsealedPiece(t *testing.T) {
rpn := &mockRPN{
sectors: mockData,
}
api := NewMinerAPI(ps, rpn, 100)
api := NewLotusAccessor(ps, rpn)
require.NoError(t, api.Start(ctx))
// Add deals to piece store
@@ -99,10 +95,6 @@ func TestLotusAccessorFetchUnsealedPiece(t *testing.T) {
require.NoError(t, err)
require.Equal(t, tc.fetchedData, string(bz))
uns, err := api.IsUnsealed(ctx, cid1)
require.NoError(t, err)
require.Equal(t, tc.isUnsealed, uns)
})
}
}
@@ -114,7 +106,7 @@ func TestLotusAccessorGetUnpaddedCARSize(t *testing.T) {
ps := getPieceStore(t)
rpn := &mockRPN{}
api := NewMinerAPI(ps, rpn, 100)
api := NewLotusAccessor(ps, rpn)
require.NoError(t, api.Start(ctx))
// Add a deal with data Length 10
@@ -141,7 +133,7 @@ func TestThrottle(t *testing.T) {
unsealedSectorID: "foo",
},
}
api := NewMinerAPI(ps, rpn, 3)
api := NewLotusAccessor(ps, rpn)
require.NoError(t, api.Start(ctx))
// Add a deal with data Length 10
@@ -168,7 +160,7 @@ func TestThrottle(t *testing.T) {
}
time.Sleep(500 * time.Millisecond)
require.EqualValues(t, 3, atomic.LoadInt32(&rpn.calls)) // throttled
require.EqualValues(t, MaxConcurrentUnsealedFetches, atomic.LoadInt32(&rpn.calls)) // throttled
// allow to proceed.
rpn.lk.Unlock()
@@ -185,14 +177,8 @@ func getPieceStore(t *testing.T) piecestore.PieceStore {
ps, err := piecestoreimpl.NewPieceStore(ds_sync.MutexWrap(ds.NewMapDatastore()))
require.NoError(t, err)
ch := make(chan struct{}, 1)
ps.OnReady(func(_ error) {
ch <- struct{}{}
})
err = ps.Start(context.Background())
require.NoError(t, err)
<-ch
return ps
}
-184
View File
@@ -1,184 +0,0 @@
package dagstore
import (
"context"
"fmt"
"io"
"github.com/filecoin-project/dagstore/throttle"
"github.com/ipfs/go-cid"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-fil-markets/piecestore"
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
"github.com/filecoin-project/go-fil-markets/shared"
)
type MinerAPI interface {
FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error)
GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid) (uint64, error)
IsUnsealed(ctx context.Context, pieceCid cid.Cid) (bool, error)
Start(ctx context.Context) error
}
type minerAPI struct {
pieceStore piecestore.PieceStore
rm retrievalmarket.RetrievalProviderNode
throttle throttle.Throttler
readyMgr *shared.ReadyManager
}
var _ MinerAPI = (*minerAPI)(nil)
func NewMinerAPI(store piecestore.PieceStore, rm retrievalmarket.RetrievalProviderNode, concurrency int) MinerAPI {
return &minerAPI{
pieceStore: store,
rm: rm,
throttle: throttle.Fixed(concurrency),
readyMgr: shared.NewReadyManager(),
}
}
func (m *minerAPI) Start(_ context.Context) error {
return m.readyMgr.FireReady(nil)
}
func (m *minerAPI) IsUnsealed(ctx context.Context, pieceCid cid.Cid) (bool, error) {
err := m.readyMgr.AwaitReady()
if err != nil {
return false, xerrors.Errorf("failed while waiting for accessor to start: %w", err)
}
var pieceInfo piecestore.PieceInfo
err = m.throttle.Do(ctx, func(ctx context.Context) (err error) {
pieceInfo, err = m.pieceStore.GetPieceInfo(pieceCid)
return err
})
if err != nil {
return false, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
}
if len(pieceInfo.Deals) == 0 {
return false, xerrors.Errorf("no storage deals found for piece %s", pieceCid)
}
// check if we have an unsealed deal for the given piece in any of the unsealed sectors.
for _, deal := range pieceInfo.Deals {
deal := deal
var isUnsealed bool
// Throttle this path to avoid flooding the storage subsystem.
err := m.throttle.Do(ctx, func(ctx context.Context) (err error) {
isUnsealed, err = m.rm.IsUnsealed(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
if err != nil {
return fmt.Errorf("failed to check if sector %d for deal %d was unsealed: %w", deal.SectorID, deal.DealID, err)
}
return nil
})
if err != nil {
log.Warnf("failed to check/retrieve unsealed sector: %s", err)
continue // move on to the next match.
}
if isUnsealed {
return true, nil
}
}
// we don't have an unsealed sector containing the piece
return false, nil
}
func (m *minerAPI) FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error) {
err := m.readyMgr.AwaitReady()
if err != nil {
return nil, err
}
// Throttle this path to avoid flooding the storage subsystem.
var pieceInfo piecestore.PieceInfo
err = m.throttle.Do(ctx, func(ctx context.Context) (err error) {
pieceInfo, err = m.pieceStore.GetPieceInfo(pieceCid)
return err
})
if err != nil {
return nil, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
}
if len(pieceInfo.Deals) == 0 {
return nil, xerrors.Errorf("no storage deals found for piece %s", pieceCid)
}
// prefer an unsealed sector containing the piece if one exists
for _, deal := range pieceInfo.Deals {
deal := deal
// Throttle this path to avoid flooding the storage subsystem.
var reader io.ReadCloser
err := m.throttle.Do(ctx, func(ctx context.Context) (err error) {
isUnsealed, err := m.rm.IsUnsealed(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
if err != nil {
return fmt.Errorf("failed to check if sector %d for deal %d was unsealed: %w", deal.SectorID, deal.DealID, err)
}
if !isUnsealed {
return nil
}
// Because we know we have an unsealed copy, this UnsealSector call will actually not perform any unsealing.
reader, err = m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
return err
})
if err != nil {
log.Warnf("failed to check/retrieve unsealed sector: %s", err)
continue // move on to the next match.
}
if reader != nil {
// we were able to obtain a reader for an already unsealed piece
return reader, nil
}
}
lastErr := xerrors.New("no sectors found to unseal from")
// if there is no unsealed sector containing the piece, just read the piece from the first sector we are able to unseal.
for _, deal := range pieceInfo.Deals {
// Note that if the deal data is not already unsealed, unsealing may
// block for a long time with the current PoRep
//
// This path is unthrottled.
reader, err := m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
if err != nil {
lastErr = xerrors.Errorf("failed to unseal deal %d: %w", deal.DealID, err)
log.Warn(lastErr.Error())
continue
}
// Successfully fetched the deal data so return a reader over the data
return reader, nil
}
return nil, lastErr
}
func (m *minerAPI) GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid) (uint64, error) {
err := m.readyMgr.AwaitReady()
if err != nil {
return 0, err
}
pieceInfo, err := m.pieceStore.GetPieceInfo(pieceCid)
if err != nil {
return 0, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
}
if len(pieceInfo.Deals) == 0 {
return 0, xerrors.Errorf("no storage deals found for piece %s", pieceCid)
}
len := pieceInfo.Deals[0].Length
return uint64(len), nil
}
+5 -30
View File
@@ -1,5 +1,5 @@
// Code generated by MockGen. DO NOT EDIT.
// Source: lotusaccessor.go
// Source: markets/dagstore/lotusaccessor.go
// Package mock_dagstore is a generated GoMock package.
package mock_dagstore
@@ -31,6 +31,10 @@ func NewMockLotusAccessor(ctrl *gomock.Controller) *MockLotusAccessor {
return mock
}
func (mr *MockLotusAccessor) Start(_ context.Context) error {
return nil
}
// EXPECT returns an object that allows the caller to indicate expected use.
func (m *MockLotusAccessor) EXPECT() *MockLotusAccessorMockRecorder {
return m.recorder
@@ -65,32 +69,3 @@ func (mr *MockLotusAccessorMockRecorder) GetUnpaddedCARSize(ctx, pieceCid interf
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetUnpaddedCARSize", reflect.TypeOf((*MockLotusAccessor)(nil).GetUnpaddedCARSize), ctx, pieceCid)
}
// IsUnsealed mocks base method.
func (m *MockLotusAccessor) IsUnsealed(ctx context.Context, pieceCid cid.Cid) (bool, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "IsUnsealed", ctx, pieceCid)
ret0, _ := ret[0].(bool)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// IsUnsealed indicates an expected call of IsUnsealed.
func (mr *MockLotusAccessorMockRecorder) IsUnsealed(ctx, pieceCid interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "IsUnsealed", reflect.TypeOf((*MockLotusAccessor)(nil).IsUnsealed), ctx, pieceCid)
}
// Start mocks base method.
func (m *MockLotusAccessor) Start(ctx context.Context) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Start", ctx)
ret0, _ := ret[0].(error)
return ret0
}
// Start indicates an expected call of Start.
func (mr *MockLotusAccessorMockRecorder) Start(ctx interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Start", reflect.TypeOf((*MockLotusAccessor)(nil).Start), ctx)
}
+19 -24
View File
@@ -15,29 +15,28 @@ const lotusScheme = "lotus"
var _ mount.Mount = (*LotusMount)(nil)
// mountTemplate returns a templated LotusMount containing the supplied API.
//
// It is called when registering a mount type with the mount registry
// of the DAG store. It is used to reinstantiate mounts after a restart.
//
// When the registry needs to deserialize a mount it clones the template then
// calls Deserialize on the cloned instance, which will have a reference to the
// lotus mount API supplied here.
func mountTemplate(api MinerAPI) *LotusMount {
return &LotusMount{API: api}
}
// LotusMount is a DAGStore mount implementation that fetches deal data
// from a PieceCID.
// LotusMount is the Lotus implementation of a Sharded DAG Store Mount.
// A Filecoin Piece is treated as a Shard by this implementation.
type LotusMount struct {
API MinerAPI
Api LotusAccessor
PieceCid cid.Cid
}
func NewLotusMount(pieceCid cid.Cid, api MinerAPI) (*LotusMount, error) {
// NewLotusMountTemplate is called when registering a mount with
// the DAG store registry.
//
// The DAG store registry receives an instance of the mount (a "template").
// When the registry needs to deserialize a mount it clones the template then
// calls Deserialize on the cloned instance, which will have a reference to the
// lotus mount API supplied here.
func NewLotusMountTemplate(api LotusAccessor) *LotusMount {
return &LotusMount{Api: api}
}
func NewLotusMount(pieceCid cid.Cid, api LotusAccessor) (*LotusMount, error) {
return &LotusMount{
PieceCid: pieceCid,
API: api,
Api: api,
}, nil
}
@@ -52,12 +51,13 @@ func (l *LotusMount) Deserialize(u *url.URL) error {
if err != nil {
return xerrors.Errorf("failed to parse PieceCid from host '%s': %w", u.Host, err)
}
l.PieceCid = pieceCid
return nil
}
func (l *LotusMount) Fetch(ctx context.Context) (mount.Reader, error) {
r, err := l.API.FetchUnsealedPiece(ctx, l.PieceCid)
r, err := l.Api.FetchUnsealedPiece(ctx, l.PieceCid)
if err != nil {
return nil, xerrors.Errorf("failed to fetch unsealed piece %s: %w", l.PieceCid, err)
}
@@ -78,20 +78,15 @@ func (l *LotusMount) Close() error {
}
func (l *LotusMount) Stat(ctx context.Context) (mount.Stat, error) {
size, err := l.API.GetUnpaddedCARSize(ctx, l.PieceCid)
size, err := l.Api.GetUnpaddedCARSize(ctx, l.PieceCid)
if err != nil {
return mount.Stat{}, xerrors.Errorf("failed to fetch piece size for piece %s: %w", l.PieceCid, err)
}
isUnsealed, err := l.API.IsUnsealed(ctx, l.PieceCid)
if err != nil {
return mount.Stat{}, xerrors.Errorf("failed to verify if we have the unsealed piece %s: %w", l.PieceCid, err)
}
// TODO Mark false when storage deal expires.
return mount.Stat{
Exists: true,
Size: int64(size),
Ready: isUnsealed,
}, nil
}
+5 -10
View File
@@ -27,9 +27,6 @@ func TestLotusMount(t *testing.T) {
// create a mock lotus api that returns the reader we want
mockLotusMountAPI := mock_dagstore.NewMockLotusAccessor(mockCtrl)
mockLotusMountAPI.EXPECT().IsUnsealed(gomock.Any(), cid).Return(true, nil).Times(1)
mockLotusMountAPI.EXPECT().FetchUnsealedPiece(gomock.Any(), cid).Return(&readCloser{ioutil.NopCloser(strings.NewReader("testing"))}, nil).Times(1)
mockLotusMountAPI.EXPECT().FetchUnsealedPiece(gomock.Any(), cid).Return(&readCloser{ioutil.NopCloser(strings.NewReader("testing"))}, nil).Times(1)
mockLotusMountAPI.EXPECT().GetUnpaddedCARSize(ctx, cid).Return(uint64(100), nil).Times(1)
@@ -55,7 +52,7 @@ func TestLotusMount(t *testing.T) {
// serialize url then deserialize from mount template -> should get back
// the same mount
url := mnt.Serialize()
mnt2 := mountTemplate(mockLotusMountAPI)
mnt2 := NewLotusMountTemplate(mockLotusMountAPI)
err = mnt2.Deserialize(url)
require.NoError(t, err)
@@ -69,7 +66,7 @@ func TestLotusMount(t *testing.T) {
}
func TestLotusMountDeserialize(t *testing.T) {
api := &minerAPI{}
api := &lotusAccessor{}
bgen := blocksutil.NewBlockGenerator()
cid := bgen.Next().Cid()
@@ -79,12 +76,12 @@ func TestLotusMountDeserialize(t *testing.T) {
u, err := url.Parse(us)
require.NoError(t, err)
mnt := mountTemplate(api)
mnt := NewLotusMountTemplate(api)
err = mnt.Deserialize(u)
require.NoError(t, err)
require.Equal(t, cid, mnt.PieceCid)
require.Equal(t, api, mnt.API)
require.Equal(t, api, mnt.Api)
// fails if cid is not valid
us = lotusScheme + "://" + "rand"
@@ -111,16 +108,14 @@ func TestLotusMountRegistration(t *testing.T) {
mockLotusMountAPI := mock_dagstore.NewMockLotusAccessor(mockCtrl)
registry := mount.NewRegistry()
err = registry.Register(lotusScheme, mountTemplate(mockLotusMountAPI))
err = registry.Register(lotusScheme, NewLotusMountTemplate(mockLotusMountAPI))
require.NoError(t, err)
mnt, err := registry.Instantiate(u)
require.NoError(t, err)
mockLotusMountAPI.EXPECT().IsUnsealed(ctx, cid).Return(true, nil)
mockLotusMountAPI.EXPECT().GetUnpaddedCARSize(ctx, cid).Return(uint64(100), nil).Times(1)
stat, err := mnt.Stat(context.Background())
require.NoError(t, err)
require.EqualValues(t, 100, stat.Size)
require.True(t, stat.Ready)
}
+33
View File
@@ -0,0 +1,33 @@
package dagstore
import (
blocks "github.com/ipfs/go-block-format"
"github.com/ipfs/go-cid"
bstore "github.com/ipfs/go-ipfs-blockstore"
"golang.org/x/xerrors"
"github.com/filecoin-project/dagstore"
)
// ReadOnlyBlockstore stubs out Blockstore mutators with methods that error out
type ReadOnlyBlockstore struct {
dagstore.ReadBlockstore
}
func NewReadOnlyBlockstore(rbs dagstore.ReadBlockstore) bstore.Blockstore {
return ReadOnlyBlockstore{ReadBlockstore: rbs}
}
func (r ReadOnlyBlockstore) DeleteBlock(c cid.Cid) error {
return xerrors.Errorf("DeleteBlock called but not implemented")
}
func (r ReadOnlyBlockstore) Put(block blocks.Block) error {
return xerrors.Errorf("Put called but not implemented")
}
func (r ReadOnlyBlockstore) PutMany(blocks []blocks.Block) error {
return xerrors.Errorf("PutMany called but not implemented")
}
var _ bstore.Blockstore = (*ReadOnlyBlockstore)(nil)
+87 -252
View File
@@ -3,202 +3,169 @@ package dagstore
import (
"context"
"errors"
"math"
"os"
"path/filepath"
"io"
"sync"
"time"
"github.com/filecoin-project/dagstore/index"
"github.com/ipfs/go-cid"
ds "github.com/ipfs/go-datastore"
levelds "github.com/ipfs/go-ds-leveldb"
measure "github.com/ipfs/go-ds-measure"
bstore "github.com/ipfs/go-ipfs-blockstore"
logging "github.com/ipfs/go-log/v2"
ldbopts "github.com/syndtr/goleveldb/leveldb/opt"
"golang.org/x/xerrors"
"github.com/filecoin-project/lotus/node/config"
"github.com/filecoin-project/go-statemachine/fsm"
"github.com/filecoin-project/dagstore"
"github.com/filecoin-project/dagstore/index"
"github.com/filecoin-project/dagstore/mount"
"github.com/filecoin-project/dagstore/shard"
"github.com/filecoin-project/go-fil-markets/storagemarket"
"github.com/filecoin-project/go-fil-markets/storagemarket/impl/providerstates"
"github.com/filecoin-project/go-fil-markets/stores"
"github.com/filecoin-project/go-fil-markets/carstore"
"github.com/filecoin-project/go-fil-markets/shared"
)
const (
maxRecoverAttempts = 1
shardRegMarker = ".shard-registration-complete"
)
const maxRecoverAttempts = 1
var log = logging.Logger("dagstore")
var log = logging.Logger("dagstore-wrapper")
// MarketDAGStoreConfig is the config the market needs to then construct a DAG Store.
type MarketDAGStoreConfig struct {
TransientsDir string
IndexDir string
Datastore ds.Datastore
MaxConcurrentIndex int
GCInterval time.Duration
}
// DAGStore provides an interface for the DAG store that can be mocked out
// by tests
type DAGStore interface {
RegisterShard(ctx context.Context, key shard.Key, mnt mount.Mount, out chan dagstore.ShardResult, opts dagstore.RegisterOpts) error
AcquireShard(ctx context.Context, key shard.Key, out chan dagstore.ShardResult, _ dagstore.AcquireOpts) error
RecoverShard(ctx context.Context, key shard.Key, out chan dagstore.ShardResult, _ dagstore.RecoverOpts) error
GC(ctx context.Context) (*dagstore.GCResult, error)
Close() error
}
type closableBlockstore struct {
bstore.Blockstore
io.Closer
}
type Wrapper struct {
ctx context.Context
cancel context.CancelFunc
backgroundWg sync.WaitGroup
cfg config.DAGStoreConfig
dagst dagstore.Interface
minerAPI MinerAPI
dagStore DAGStore
mountApi LotusAccessor
failureCh chan dagstore.ShardResult
traceCh chan dagstore.Trace
gcInterval time.Duration
}
var _ stores.DAGStoreWrapper = (*Wrapper)(nil)
var _ shared.DagStoreWrapper = (*Wrapper)(nil)
func NewDAGStore(cfg config.DAGStoreConfig, mountApi MinerAPI) (*dagstore.DAGStore, *Wrapper, error) {
func NewDagStoreWrapper(cfg MarketDAGStoreConfig, mountApi LotusAccessor) (*Wrapper, error) {
// construct the DAG Store.
registry := mount.NewRegistry()
if err := registry.Register(lotusScheme, mountTemplate(mountApi)); err != nil {
return nil, nil, xerrors.Errorf("failed to create registry: %w", err)
if err := registry.Register(lotusScheme, NewLotusMountTemplate(mountApi)); err != nil {
return nil, xerrors.Errorf("failed to create registry: %w", err)
}
// The dagstore will write Shard failures to the `failureCh` here.
failureCh := make(chan dagstore.ShardResult, 1)
// The dagstore will write Trace events to the `traceCh` here.
traceCh := make(chan dagstore.Trace, 32)
var (
transientsDir = filepath.Join(cfg.RootDir, "transients")
datastoreDir = filepath.Join(cfg.RootDir, "datastore")
indexDir = filepath.Join(cfg.RootDir, "index")
)
dstore, err := newDatastore(datastoreDir)
irepo, err := index.NewFSRepo(cfg.IndexDir)
if err != nil {
return nil, nil, xerrors.Errorf("failed to create dagstore datastore in %s: %w", datastoreDir, err)
}
irepo, err := index.NewFSRepo(indexDir)
if err != nil {
return nil, nil, xerrors.Errorf("failed to initialise dagstore index repo")
return nil, xerrors.Errorf("failed to initialise dagstore index repo")
}
dcfg := dagstore.Config{
TransientsDir: transientsDir,
TransientsDir: cfg.TransientsDir,
IndexRepo: irepo,
Datastore: dstore,
Datastore: cfg.Datastore,
MountRegistry: registry,
FailureCh: failureCh,
TraceCh: traceCh,
// not limiting fetches globally, as the Lotus mount does
// conditional throttling.
MaxConcurrentIndex: cfg.MaxConcurrentIndex,
MaxConcurrentReadyFetches: cfg.MaxConcurrentReadyFetches,
RecoverOnStart: dagstore.RecoverOnAcquire,
MaxConcurrentIndex: cfg.MaxConcurrentIndex,
RecoverOnStart: dagstore.RecoverOnAcquire,
}
dagst, err := dagstore.NewDAGStore(dcfg)
dagStore, err := dagstore.NewDAGStore(dcfg)
if err != nil {
return nil, nil, xerrors.Errorf("failed to create DAG store: %w", err)
return nil, xerrors.Errorf("failed to create DAG store: %w", err)
}
w := &Wrapper{
cfg: cfg,
dagst: dagst,
minerAPI: mountApi,
return &Wrapper{
dagStore: dagStore,
mountApi: mountApi,
failureCh: failureCh,
traceCh: traceCh,
gcInterval: time.Duration(cfg.GCInterval),
}
return dagst, w, nil
gcInterval: cfg.GCInterval,
}, nil
}
// newDatastore creates a datastore under the given base directory
// for dagstore metadata.
func newDatastore(dir string) (ds.Batching, error) {
// Create the datastore directory if it doesn't exist yet.
if err := os.MkdirAll(dir, 0755); err != nil {
return nil, xerrors.Errorf("failed to create directory %s for DAG store datastore: %w", dir, err)
}
// Create a new LevelDB datastore
dstore, err := levelds.NewDatastore(dir, &levelds.Options{
Compression: ldbopts.NoCompression,
NoSync: false,
Strict: ldbopts.StrictAll,
ReadOnly: false,
})
if err != nil {
return nil, xerrors.Errorf("failed to open datastore for DAG store: %w", err)
}
// Keep statistics about the datastore
mds := measure.New("measure.", dstore)
return mds, nil
}
func (w *Wrapper) Start(ctx context.Context) error {
w.ctx, w.cancel = context.WithCancel(ctx)
func (ds *Wrapper) Start(ctx context.Context) {
ds.ctx, ds.cancel = context.WithCancel(ctx)
// Run a go-routine to do DagStore GC.
w.backgroundWg.Add(1)
go w.gcLoop()
ds.backgroundWg.Add(1)
go ds.dagStoreGCLoop()
// run a go-routine to read the trace for debugging.
w.backgroundWg.Add(1)
go w.traceLoop()
ds.backgroundWg.Add(1)
go ds.traceLoop()
// Run a go-routine for shard recovery
if dss, ok := w.dagst.(*dagstore.DAGStore); ok {
w.backgroundWg.Add(1)
go dagstore.RecoverImmediately(w.ctx, dss, w.failureCh, maxRecoverAttempts, w.backgroundWg.Done)
if dss, ok := ds.dagStore.(*dagstore.DAGStore); ok {
ds.backgroundWg.Add(1)
go dagstore.RecoverImmediately(ds.ctx, dss, ds.failureCh, maxRecoverAttempts, ds.backgroundWg.Done)
}
return w.dagst.Start(ctx)
}
func (w *Wrapper) traceLoop() {
defer w.backgroundWg.Done()
func (ds *Wrapper) traceLoop() {
defer ds.backgroundWg.Done()
for w.ctx.Err() == nil {
for ds.ctx.Err() == nil {
select {
// Log trace events from the DAG store
case tr := <-w.traceCh:
case tr := <-ds.traceCh:
log.Debugw("trace",
"shard-key", tr.Key.String(),
"op-type", tr.Op.String(),
"after", tr.After.String())
case <-w.ctx.Done():
case <-ds.ctx.Done():
return
}
}
}
func (w *Wrapper) gcLoop() {
defer w.backgroundWg.Done()
func (ds *Wrapper) dagStoreGCLoop() {
defer ds.backgroundWg.Done()
ticker := time.NewTicker(w.gcInterval)
defer ticker.Stop()
gcTicker := time.NewTicker(ds.gcInterval)
defer gcTicker.Stop()
for w.ctx.Err() == nil {
for ds.ctx.Err() == nil {
select {
// GC the DAG store on every tick
case <-ticker.C:
_, _ = w.dagst.GC(w.ctx)
case <-gcTicker.C:
_, _ = ds.dagStore.GC(ds.ctx)
// Exit when the DAG store wrapper is shutdown
case <-w.ctx.Done():
case <-ds.ctx.Done():
return
}
}
}
func (w *Wrapper) LoadShard(ctx context.Context, pieceCid cid.Cid) (stores.ClosableBlockstore, error) {
func (ds *Wrapper) LoadShard(ctx context.Context, pieceCid cid.Cid) (carstore.ClosableBlockstore, error) {
log.Debugf("acquiring shard for piece CID %s", pieceCid)
key := shard.KeyFromCID(pieceCid)
resch := make(chan dagstore.ShardResult, 1)
err := w.dagst.AcquireShard(ctx, key, resch, dagstore.AcquireOpts{})
err := ds.dagStore.AcquireShard(ctx, key, resch, dagstore.AcquireOpts{})
log.Debugf("sent message to acquire shard for piece CID %s", pieceCid)
if err != nil {
@@ -213,13 +180,13 @@ func (w *Wrapper) LoadShard(ctx context.Context, pieceCid cid.Cid) (stores.Closa
// we already have a transient file. However, we don't have it here
// and therefore we pass an empty file path.
carPath := ""
if err := stores.RegisterShardSync(ctx, w, pieceCid, carPath, false); err != nil {
if err := shared.RegisterShardSync(ctx, ds, pieceCid, carPath, false); err != nil {
return nil, xerrors.Errorf("failed to re-register shard during loading piece CID %s: %w", pieceCid, err)
}
log.Warnw("successfully re-registered shard", "pieceCID", pieceCid)
resch = make(chan dagstore.ShardResult, 1)
if err := w.dagst.AcquireShard(ctx, key, resch, dagstore.AcquireOpts{}); err != nil {
if err := ds.dagStore.AcquireShard(ctx, key, resch, dagstore.AcquireOpts{}); err != nil {
return nil, xerrors.Errorf("failed to acquire Shard for piece CID %s after re-registering: %w", pieceCid, err)
}
}
@@ -244,13 +211,13 @@ func (w *Wrapper) LoadShard(ctx context.Context, pieceCid cid.Cid) (stores.Closa
}
log.Debugf("successfully loaded blockstore for piece CID %s", pieceCid)
return &Blockstore{ReadBlockstore: bs, Closer: res.Accessor}, nil
return &closableBlockstore{Blockstore: NewReadOnlyBlockstore(bs), Closer: res.Accessor}, nil
}
func (w *Wrapper) RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath string, eagerInit bool, resch chan dagstore.ShardResult) error {
func (ds *Wrapper) RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath string, eagerInit bool, resch chan dagstore.ShardResult) error {
// Create a lotus mount with the piece CID
key := shard.KeyFromCID(pieceCid)
mt, err := NewLotusMount(pieceCid, w.minerAPI)
mt, err := NewLotusMount(pieceCid, ds.mountApi)
if err != nil {
return xerrors.Errorf("failed to create lotus mount for piece CID %s: %w", pieceCid, err)
}
@@ -260,7 +227,7 @@ func (w *Wrapper) RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath s
ExistingTransient: carPath,
LazyInitialization: !eagerInit,
}
err = w.dagst.RegisterShard(ctx, key, mt, resch, opts)
err = ds.dagStore.RegisterShard(ctx, key, mt, resch, opts)
if err != nil {
return xerrors.Errorf("failed to schedule register shard for piece CID %s: %w", pieceCid, err)
}
@@ -269,153 +236,21 @@ func (w *Wrapper) RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath s
return nil
}
func (w *Wrapper) MigrateDeals(ctx context.Context, deals []storagemarket.MinerDeal) (bool, error) {
log := log.Named("migrator")
// Check if all deals have already been registered as shards
isComplete, err := w.registrationComplete()
if err != nil {
return false, xerrors.Errorf("failed to get dagstore migration status: %w", err)
}
if isComplete {
// All deals have been registered as shards, bail out
log.Info("no shard migration necessary; already marked complete")
return false, nil
}
log.Infow("registering shards for all active deals in sealing subsystem", "count", len(deals))
inSealingSubsystem := make(map[fsm.StateKey]struct{}, len(providerstates.StatesKnownBySealingSubsystem))
for _, s := range providerstates.StatesKnownBySealingSubsystem {
inSealingSubsystem[s] = struct{}{}
}
// channel where results will be received, and channel where the total
// number of registered shards will be sent.
resch := make(chan dagstore.ShardResult, 32)
totalCh := make(chan int)
doneCh := make(chan struct{})
// Start making progress consuming results. We won't know how many to
// actually consume until we register all shards.
//
// If there are any problems registering shards, just log an error
go func() {
defer close(doneCh)
var total = math.MaxInt64
var res dagstore.ShardResult
for rcvd := 0; rcvd < total; {
select {
case total = <-totalCh:
// we now know the total number of registered shards
// nullify so that we no longer consume from it after closed.
close(totalCh)
totalCh = nil
case res = <-resch:
rcvd++
if res.Error == nil {
log.Infow("async shard registration completed successfully", "shard_key", res.Key)
} else {
log.Warnw("async shard registration failed", "shard_key", res.Key, "error", res.Error)
}
}
}
}()
// Filter for deals that are handed off.
//
// If the deal has not yet been handed off to the sealing subsystem, we
// don't need to call RegisterShard in this migration; RegisterShard will
// be called in the new code once the deal reaches the state where it's
// handed off to the sealing subsystem.
var registered int
for _, deal := range deals {
if deal.Ref.PieceCid == nil {
log.Warnw("deal has nil piece CID; skipping", "deal_id", deal.DealID)
continue
}
// enrich log statements in this iteration with deal ID and piece CID.
log := log.With("deal_id", deal.DealID, "piece_cid", deal.Ref.PieceCid)
// Filter for deals that have been handed off to the sealing subsystem
if _, ok := inSealingSubsystem[deal.State]; !ok {
log.Infow("deal not ready; skipping")
continue
}
log.Infow("registering deal in dagstore with lazy init")
// Register the deal as a shard with the DAG store with lazy initialization.
// The index will be populated the first time the deal is retrieved, or
// through the bulk initialization script.
err = w.RegisterShard(ctx, *deal.Ref.PieceCid, "", false, resch)
if err != nil {
log.Warnw("failed to register shard", "error", err)
continue
}
registered++
}
log.Infow("finished registering all shards", "total", registered)
totalCh <- registered
<-doneCh
log.Infow("confirmed registration of all shards")
// Completed registering all shards, so mark the migration as complete
err = w.markRegistrationComplete()
if err != nil {
log.Errorf("failed to mark shards as registered: %s", err)
} else {
log.Info("successfully marked migration as complete")
}
log.Infow("dagstore migration complete")
return true, nil
}
// Check for the existence of a "marker" file indicating that the migration
// has completed
func (w *Wrapper) registrationComplete() (bool, error) {
path := filepath.Join(w.cfg.RootDir, shardRegMarker)
_, err := os.Stat(path)
if os.IsNotExist(err) {
return false, nil
}
if err != nil {
return false, err
}
return true, nil
}
// Create a "marker" file indicating that the migration has completed
func (w *Wrapper) markRegistrationComplete() error {
path := filepath.Join(w.cfg.RootDir, shardRegMarker)
file, err := os.Create(path)
if err != nil {
return err
}
return file.Close()
}
func (w *Wrapper) Close() error {
func (ds *Wrapper) Close() error {
// Cancel the context
w.cancel()
ds.cancel()
// Close the DAG store
log.Info("will close the dagstore")
if err := w.dagst.Close(); err != nil {
return xerrors.Errorf("failed to close dagstore: %w", err)
if err := ds.dagStore.Close(); err != nil {
return xerrors.Errorf("failed to close DAG store: %w", err)
}
log.Info("dagstore closed")
// Wait for the background go routine to exit
log.Info("waiting for dagstore background wrapper goroutines to exit")
w.backgroundWg.Wait()
log.Info("exited dagstore background wrapper goroutines")
log.Info("waiting for dagstore background wrapper routines to exist")
ds.backgroundWg.Wait()
log.Info("exited dagstore background warpper routines")
return nil
}
-110
View File
@@ -1,110 +0,0 @@
package dagstore
import (
"context"
"testing"
"github.com/filecoin-project/dagstore"
"github.com/stretchr/testify/require"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-fil-markets/retrievalmarket/impl/testnodes"
tut "github.com/filecoin-project/go-fil-markets/shared_testutil"
"github.com/filecoin-project/go-fil-markets/storagemarket"
"github.com/filecoin-project/lotus/node/config"
)
func TestShardRegistration(t *testing.T) {
ps := tut.NewTestPieceStore()
providerNode := testnodes.NewTestRetrievalProviderNode()
ctx := context.Background()
cids := tut.GenerateCids(4)
pieceCidUnsealed := cids[0]
pieceCidSealed := cids[1]
pieceCidUnsealed2 := cids[2]
pieceCidUnsealed3 := cids[3]
sealedSector := abi.SectorNumber(1)
unsealedSector1 := abi.SectorNumber(2)
unsealedSector2 := abi.SectorNumber(3)
unsealedSector3 := abi.SectorNumber(4)
// ps.ExpectPiece(pieceCidUnsealed, piecestore.PieceInfo{
// PieceCID: pieceCidUnsealed,
// Deals: []piecestore.DealInfo{
// {
// SectorID: unsealedSector1,
// },
// },
// })
//
// ps.ExpectPiece(pieceCidSealed, piecestore.PieceInfo{
// PieceCID: pieceCidSealed,
// Deals: []piecestore.DealInfo{
// {
// SectorID: sealedSector,
// },
// },
// })
deals := []storagemarket.MinerDeal{{
// Should be registered
State: storagemarket.StorageDealSealing,
SectorNumber: unsealedSector1,
Ref: &storagemarket.DataRef{
PieceCid: &pieceCidUnsealed,
},
}, {
// Should be registered with lazy registration (because sector is sealed)
State: storagemarket.StorageDealSealing,
SectorNumber: sealedSector,
Ref: &storagemarket.DataRef{
PieceCid: &pieceCidSealed,
},
}, {
// Should be ignored because deal is no longer active
State: storagemarket.StorageDealError,
SectorNumber: unsealedSector2,
Ref: &storagemarket.DataRef{
PieceCid: &pieceCidUnsealed2,
},
}, {
// Should be ignored because deal is not yet sealing
State: storagemarket.StorageDealFundsReserved,
SectorNumber: unsealedSector3,
Ref: &storagemarket.DataRef{
PieceCid: &pieceCidUnsealed3,
},
}}
cfg := config.DefaultStorageMiner().DAGStore
cfg.RootDir = t.TempDir()
mapi := NewMinerAPI(ps, providerNode, 10)
dagst, w, err := NewDAGStore(cfg, mapi)
require.NoError(t, err)
require.NotNil(t, dagst)
require.NotNil(t, w)
err = dagst.Start(context.Background())
require.NoError(t, err)
migrated, err := w.MigrateDeals(ctx, deals)
require.True(t, migrated)
require.NoError(t, err)
info := dagst.AllShardsInfo()
require.Len(t, info, 2)
for _, i := range info {
require.Equal(t, dagstore.ShardStateNew, i.ShardState)
}
// Run register shard migration again
migrated, err = w.MigrateDeals(ctx, deals)
require.False(t, migrated)
require.NoError(t, err)
// ps.VerifyExpectations(t)
}
+11 -36
View File
@@ -10,8 +10,6 @@ import (
"golang.org/x/xerrors"
"github.com/filecoin-project/lotus/node/config"
"github.com/filecoin-project/dagstore"
"github.com/filecoin-project/dagstore/mount"
"github.com/filecoin-project/dagstore/shard"
@@ -28,14 +26,13 @@ func TestWrapperAcquireRecovery(t *testing.T) {
require.NoError(t, err)
// Create a DAG store wrapper
dagst, w, err := NewDAGStore(config.DAGStoreConfig{
RootDir: t.TempDir(),
GCInterval: config.Duration(1 * time.Millisecond),
w, err := NewDagStoreWrapper(MarketDAGStoreConfig{
TransientsDir: t.TempDir(),
IndexDir: t.TempDir(),
GCInterval: time.Millisecond,
}, mockLotusMount{})
require.NoError(t, err)
defer dagst.Close() //nolint:errcheck
// Return an error from acquire shard the first time
acquireShardErr := make(chan error, 1)
acquireShardErr <- xerrors.Errorf("unknown shard: %w", dagstore.ErrShardUnknown)
@@ -48,7 +45,7 @@ func TestWrapperAcquireRecovery(t *testing.T) {
},
register: make(chan shard.Key, 1),
}
w.dagst = mock
w.dagStore = mock
mybs, err := w.LoadShard(ctx, pieceCid)
require.NoError(t, err)
@@ -79,25 +76,23 @@ func TestWrapperBackground(t *testing.T) {
ctx := context.Background()
// Create a DAG store wrapper
dagst, w, err := NewDAGStore(config.DAGStoreConfig{
RootDir: t.TempDir(),
GCInterval: config.Duration(1 * time.Millisecond),
w, err := NewDagStoreWrapper(MarketDAGStoreConfig{
TransientsDir: t.TempDir(),
IndexDir: t.TempDir(),
GCInterval: time.Millisecond,
}, mockLotusMount{})
require.NoError(t, err)
defer dagst.Close() //nolint:errcheck
// Create a mock DAG store in place of the real DAG store
mock := &mockDagStore{
gc: make(chan struct{}, 1),
recover: make(chan shard.Key, 1),
close: make(chan struct{}, 1),
}
w.dagst = mock
w.dagStore = mock
// Start up the wrapper
err = w.Start(ctx)
require.NoError(t, err)
w.Start(ctx)
// Expect GC to be called automatically
tctx, cancel := context.WithTimeout(ctx, time.Second)
@@ -132,22 +127,6 @@ type mockDagStore struct {
close chan struct{}
}
func (m *mockDagStore) DestroyShard(ctx context.Context, key shard.Key, out chan dagstore.ShardResult, _ dagstore.DestroyOpts) error {
panic("implement me")
}
func (m *mockDagStore) GetShardInfo(k shard.Key) (dagstore.ShardInfo, error) {
panic("implement me")
}
func (m *mockDagStore) AllShardsInfo() dagstore.AllShardsInfo {
panic("implement me")
}
func (m *mockDagStore) Start(_ context.Context) error {
return nil
}
func (m *mockDagStore) RegisterShard(ctx context.Context, key shard.Key, mnt mount.Mount, out chan dagstore.ShardResult, opts dagstore.RegisterOpts) error {
m.register <- key
out <- dagstore.ShardResult{Key: key}
@@ -199,10 +178,6 @@ func (m mockLotusMount) GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid
panic("implement me")
}
func (m mockLotusMount) IsUnsealed(ctx context.Context, pieceCid cid.Cid) (bool, error) {
panic("implement me")
}
func getShardAccessor(t *testing.T) *dagstore.ShardAccessor {
data, err := os.ReadFile("./fixtures/sample-rw-bs-v2.car")
require.NoError(t, err)
-1
View File
@@ -66,7 +66,6 @@ var (
AutoNATSvcKey = special{10} // Libp2p option
BandwidthReporterKey = special{11} // Libp2p option
ConnGaterKey = special{12} // libp2p option
DAGStoreKey = special{13} // constructor returns multiple values
)
type invoke int
+2 -3
View File
@@ -73,7 +73,6 @@ func ConfigStorageMiner(c interface{}) Option {
return Options(
ConfigCommon(&cfg.Common, enableLibp2pNode),
Override(new(api.MinerSubsystems), modules.ExtractEnabledMinerSubsystems(cfg.Subsystems)),
Override(new(stores.LocalStorage), From(new(repo.LockedRepo))),
Override(new(*stores.Local), modules.LocalStorage),
Override(new(*stores.Remote), modules.RemoteStorage),
@@ -148,8 +147,8 @@ func ConfigStorageMiner(c interface{}) Option {
Override(new(dtypes.RetrievalPricingFunc), modules.RetrievalPricingFunc(cfg.Dealmaking)),
// DAG Store
Override(new(dagstore.MinerAPI), modules.NewMinerAPI),
Override(DAGStoreKey, modules.DAGStore),
Override(new(dagstore.LotusAccessor), modules.NewLotusAccessor),
Override(new(*dagstore.Wrapper), modules.DagStoreWrapper),
// Markets (retrieval)
Override(new(retrievalmarket.RetrievalProviderNode), retrievaladapter.NewRetrievalProviderNode),
-6
View File
@@ -188,12 +188,6 @@ func DefaultStorageMiner() *StorageMiner {
TerminateControl: []string{},
DealPublishControl: []string{},
},
DAGStore: DAGStoreConfig{
MaxConcurrentIndex: 5,
MaxConcurrencyStorageCalls: 100,
GCInterval: Duration(1 * time.Minute),
},
}
cfg.Common.API.ListenAddress = "/ip4/127.0.0.1/tcp/2345/http"
cfg.Common.API.RemoteListenAddress = "127.0.0.1:2345"
-46
View File
@@ -125,46 +125,6 @@ and storage providers`,
Comment: ``,
},
},
"DAGStoreConfig": []DocField{
{
Name: "RootDir",
Type: "string",
Comment: `Default value: $LOTUS_MARKETS_PATH/dagStore (split deployment) or
$LOTUS_MINER_PATH/dagStore (monolith deployment)`,
},
{
Name: "MaxConcurrentIndex",
Type: "int",
Comment: `The maximum amount of indexing jobs that can run simultaneously.
Default value: 5.`,
},
{
Name: "MaxConcurrentReadyFetches",
Type: "int",
Comment: `The maximum amount of unsealed deals that can be fetched simultaneously
from the storage subsystem.
Default value: 2.`,
},
{
Name: "MaxConcurrencyStorageCalls",
Type: "int",
Comment: `The maximum number of simultaneous inflight API calls to the storage
subsystem.
Default value: 100.`,
},
{
Name: "GCInterval",
Type: "Duration",
Comment: `The time between calls to periodic dagstore GC, in time.Duration string
representation, e.g. 1m, 5m, 1h.
Default value: 1 minute.`,
},
},
"DealmakingConfig": []DocField{
{
Name: "ConsiderOnlineStorageDeals",
@@ -793,12 +753,6 @@ Default is 20 (about once a week).`,
Name: "Addresses",
Type: "MinerAddressConfig",
Comment: ``,
},
{
Name: "DAGStore",
Type: "DAGStoreConfig",
Comment: ``,
},
},
-36
View File
@@ -50,42 +50,6 @@ type StorageMiner struct {
Storage sectorstorage.SealerConfig
Fees MinerFeeConfig
Addresses MinerAddressConfig
DAGStore DAGStoreConfig
}
type DAGStoreConfig struct {
// Path to the dagstore root directory. This directory contains three
// subdirectories, which can be symlinked to alternative locations if
// need be:
//
// - ./transients: caches unsealed deals that have been fetched from the
// storage subsystem for serving retrievals.
// - ./indices: stores shard indices.
// - ./datastore: holds the KV store tracking the state of every shard
// known to the DAG store.
//
// Default value: $LOTUS_MARKETS_PATH/dagStore (split deployment) or
// $LOTUS_MINER_PATH/dagStore (monolith deployment)
RootDir string
// The maximum amount of indexing jobs that can run simultaneously.
// Default value: 5.
MaxConcurrentIndex int
// The maximum amount of unsealed deals that can be fetched simultaneously
// from the storage subsystem.
// Default value: 2.
MaxConcurrentReadyFetches int
// The maximum number of simultaneous inflight API calls to the storage
// subsystem.
// Default value: 100.
MaxConcurrencyStorageCalls int
// The time between calls to periodic dagstore GC, in time.Duration string
// representation, e.g. 1m, 5m, 1h.
// Default value: 1 minute.
GCInterval Duration
}
type MinerSubsystemConfig struct {
+34 -33
View File
@@ -10,7 +10,8 @@ import (
"sort"
"time"
"github.com/ipld/go-car"
"github.com/filecoin-project/go-fil-markets/filestorecaradapter"
"github.com/filecoin-project/lotus/chain/actors/builtin/miner"
carv2 "github.com/ipld/go-car/v2"
"github.com/ipld/go-car/v2/blockstore"
@@ -25,7 +26,7 @@ import (
files "github.com/ipfs/go-ipfs-files"
"github.com/ipfs/go-merkledag"
unixfile "github.com/ipfs/go-unixfs/file"
"github.com/ipld/go-car"
basicnode "github.com/ipld/go-ipld-prime/node/basic"
"github.com/ipld/go-ipld-prime/traversal/selector"
"github.com/ipld/go-ipld-prime/traversal/selector/builder"
@@ -46,7 +47,6 @@ import (
"github.com/filecoin-project/go-fil-markets/shared"
"github.com/filecoin-project/go-fil-markets/storagemarket"
"github.com/filecoin-project/go-fil-markets/storagemarket/network"
"github.com/filecoin-project/go-fil-markets/stores"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/specs-actors/v3/actors/builtin/market"
@@ -54,7 +54,6 @@ import (
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/chain/actors/builtin/miner"
"github.com/filecoin-project/lotus/chain/store"
"github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/markets/utils"
@@ -189,17 +188,17 @@ func (a *API) dealStarter(ctx context.Context, params *api.StartDealParams, isSt
providerInfo := utils.NewStorageProviderInfo(params.Miner, mi.Worker, mi.SectorSize, *mi.PeerId, mi.Multiaddrs)
result, err := a.SMDealClient.ProposeStorageDeal(ctx, storagemarket.ProposeStorageDealParams{
Addr: params.Wallet,
Info: &providerInfo,
Data: params.Data,
StartEpoch: dealStart,
EndEpoch: calcDealExpiration(params.MinBlocksDuration, md, dealStart),
Price: params.EpochPrice,
Collateral: params.ProviderCollateral,
Rt: st,
FastRetrieval: params.FastRetrieval,
VerifiedDeal: params.VerifiedDeal,
IndexedCAR: CARV2FilePath,
Addr: params.Wallet,
Info: &providerInfo,
Data: params.Data,
StartEpoch: dealStart,
EndEpoch: calcDealExpiration(params.MinBlocksDuration, md, dealStart),
Price: params.EpochPrice,
Collateral: params.ProviderCollateral,
Rt: st,
FastRetrieval: params.FastRetrieval,
VerifiedDeal: params.VerifiedDeal,
FilestoreCARv2FilePath: CARV2FilePath,
})
if err != nil {
@@ -520,7 +519,7 @@ func (a *API) ClientImport(ctx context.Context, ref api.FileRef) (res *api.Impor
}
}()
root, err = a.doImport(ctx, ref.Path, carFile)
root, err = a.importNormalFileToFilestoreCARv2(ctx, id, ref.Path, carFile)
if err != nil {
return nil, xerrors.Errorf("failed to import normal file to CARv2: %w", err)
}
@@ -1032,21 +1031,21 @@ func (w *lenWriter) Write(p []byte) (n int, err error) {
}
func (a *API) ClientDealSize(ctx context.Context, root cid.Cid) (api.DataSize, error) {
path, err := a.imgr().FilestoreCARV2FilePathFor(root)
fc, err := a.imgr().FilestoreCARV2FilePathFor(root)
if err != nil {
return api.DataSize{}, xerrors.Errorf("failed to find CARv2 file for root: %w", err)
}
if len(path) == 0 {
if len(fc) == 0 {
return api.DataSize{}, xerrors.New("no CARv2 file for root")
}
fs, err := stores.ReadOnlyFilestore(path)
rdOnly, err := filestorecaradapter.NewReadOnlyFileStore(fc)
if err != nil {
return api.DataSize{}, xerrors.Errorf("failed to open filestore from carv2 in path %s: %w", path, err)
return api.DataSize{}, xerrors.Errorf("failed to open read only filestore: %w", err)
}
defer fs.Close() //nolint:errcheck
defer rdOnly.Close() //nolint:errcheck
dag := merkledag.NewDAGService(blockservice.New(fs, offline.Exchange(fs)))
dag := merkledag.NewDAGService(blockservice.New(rdOnly, offline.Exchange(rdOnly)))
w := lenWriter(0)
err = car.WriteCar(ctx, dag, []cid.Cid{root}, &w)
@@ -1063,21 +1062,21 @@ func (a *API) ClientDealSize(ctx context.Context, root cid.Cid) (api.DataSize, e
}
func (a *API) ClientDealPieceCID(ctx context.Context, root cid.Cid) (api.DataCIDSize, error) {
path, err := a.imgr().FilestoreCARV2FilePathFor(root)
fc, err := a.imgr().FilestoreCARV2FilePathFor(root)
if err != nil {
return api.DataCIDSize{}, xerrors.Errorf("failed to find CARv2 file for root: %w", err)
}
if len(path) == 0 {
if len(fc) == 0 {
return api.DataCIDSize{}, xerrors.New("no CARv2 file for root")
}
fs, err := stores.ReadOnlyFilestore(path)
rdOnly, err := filestorecaradapter.NewReadOnlyFileStore(fc)
if err != nil {
return api.DataCIDSize{}, xerrors.Errorf("failed to open filestore from carv2 in path %s: %w", path, err)
return api.DataCIDSize{}, xerrors.Errorf("failed to open read only blockstore: %w", err)
}
defer fs.Close() //nolint:errcheck
defer rdOnly.Close() //nolint:errcheck
dag := merkledag.NewDAGService(blockservice.New(fs, offline.Exchange(fs)))
dag := merkledag.NewDAGService(blockservice.New(rdOnly, offline.Exchange(rdOnly)))
w := &writer.Writer{}
bw := bufio.NewWriterSize(w, int(writer.CommPBuf))
@@ -1096,20 +1095,22 @@ func (a *API) ClientDealPieceCID(ctx context.Context, root cid.Cid) (api.DataCID
func (a *API) ClientGenCar(ctx context.Context, ref api.FileRef, outputPath string) error {
id := importmgr.ImportID(rand.Uint64())
tmp, err := a.imgr().NewTempFile(id)
tmpCARv2File, err := a.imgr().NewTempFile(id)
if err != nil {
return xerrors.Errorf("failed to create temp file: %w", err)
}
defer os.Remove(tmp) //nolint:errcheck
defer os.Remove(tmpCARv2File) //nolint:errcheck
root, err := a.doImport(ctx, ref.Path, tmp)
root, err := a.importNormalFileToFilestoreCARv2(ctx, id, ref.Path, tmpCARv2File)
if err != nil {
return xerrors.Errorf("failed to import normal file to CARv2")
}
fs, err := stores.ReadOnlyFilestore(tmp)
// generate a deterministic CARv1 payload from the UnixFS DAG by doing an IPLD
// traversal over the Unixfs DAG in the CARv2 file using the "all selector" i.e the entire DAG selector.
fs, err := filestorecaradapter.NewReadOnlyFileStore(tmpCARv2File)
if err != nil {
return xerrors.Errorf("failed to open filestore from carv2 in path %s: %w", tmp, err)
return xerrors.Errorf("failed to open read only CARv2 blockstore: %w", err)
}
defer fs.Close() //nolint:errcheck
+37 -59
View File
@@ -2,13 +2,18 @@ package client
import (
"context"
"io/ioutil"
"os"
"github.com/filecoin-project/go-fil-markets/stores"
"github.com/filecoin-project/go-fil-markets/filestorecaradapter"
bstore "github.com/filecoin-project/lotus/blockstore"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/node/repo/importmgr"
"github.com/ipfs/go-blockservice"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-cidutil"
"github.com/ipfs/go-datastore"
ds_sync "github.com/ipfs/go-datastore/sync"
"github.com/ipfs/go-filestore"
chunker "github.com/ipfs/go-ipfs-chunker"
offline "github.com/ipfs/go-ipfs-exchange-offline"
files2 "github.com/ipfs/go-ipfs-files"
@@ -16,83 +21,56 @@ import (
"github.com/ipfs/go-merkledag"
"github.com/ipfs/go-unixfs/importer/balanced"
ihelper "github.com/ipfs/go-unixfs/importer/helpers"
"github.com/ipld/go-car/v2"
"github.com/ipld/go-car/v2/blockstore"
"golang.org/x/xerrors"
"github.com/filecoin-project/lotus/build"
)
// doImport takes a standard file (src), forms a UnixFS DAG, and writes a
// CARv2 file with positional mapping (backed by the go-filestore library).
func (a *API) doImport(ctx context.Context, src string, dst string) (cid.Cid, error) {
// This method uses a two-phase approach with a staging CAR blockstore and
// a final CAR blockstore.
//
// This is necessary because of https://github.com/ipld/go-car/issues/196
//
// TODO: do we need to chunk twice? Isn't the first output already in the
// right order? Can't we just copy the CAR file and replace the header?
f, err := ioutil.TempFile("", "")
if err != nil {
return cid.Undef, xerrors.Errorf("failed to create temp file: %w", err)
}
_ = f.Close() // close; we only want the path.
tmp := f.Name()
defer os.Remove(tmp) //nolint:errcheck
// Step 1. Compute the UnixFS DAG and write it to a CARv2 file to get
// the root CID of the DAG.
fstore, err := stores.ReadWriteFilestore(tmp)
if err != nil {
return cid.Undef, xerrors.Errorf("failed to create temporary filestore: %w", err)
}
// importNormalFileToFilestoreCARv2 transforms the client's "normal file" to a Unixfs IPLD DAG and writes out the DAG to a CARv2 file
// that can be used to back a filestore.
func (a *API) importNormalFileToFilestoreCARv2(ctx context.Context, importID importmgr.ImportID, inputFilePath string, outputCARv2Path string) (c cid.Cid, finalErr error) {
// TODO: We've currently put in a hack to create the Unixfs DAG as a CARv2 without using Badger.
// We first create the Unixfs DAG using a filestore to get the root of the Unixfs DAG.
// We can't create the UnixfsDAG right away using a CARv2 read-write blockstore as the blockstore
// needs the root of the DAG during instantiation to write out a valid CARv2 file.
//
// In the second pass, we create a CARv2 file with the root present using the root node we get in the above step.
// This hack should be fixed when CARv2 allows specifying the root AFTER finishing the CARv2 streaming write.
fm := filestore.NewFileManager(ds_sync.MutexWrap(datastore.NewMapDatastore()), "/")
fm.AllowFiles = true
fstore := filestore.NewFilestore(bstore.NewMemorySync(), fm)
bsvc := blockservice.New(fstore, offline.Exchange(fstore))
dags := merkledag.NewDAGService(bsvc)
defer bsvc.Close() //nolint:errcheck
root, err := buildUnixFS(ctx, src, dags)
// ---- First Pass --- Write out the UnixFS DAG to a rootless CARv2 file by instantiating a read-write CARv2 blockstore without the root.
root, err := importNormalFileToUnixfsDAG(ctx, inputFilePath, merkledag.NewDAGService(bsvc))
if err != nil {
_ = fstore.Close()
return cid.Undef, xerrors.Errorf("failed to import file to store to compute root: %w", err)
return cid.Undef, xerrors.Errorf("failed to import file to store: %w", err)
}
if err := fstore.Close(); err != nil {
return cid.Undef, xerrors.Errorf("failed to finalize car filestore: %w", err)
}
// Step 2. We now have the root of the UnixFS DAG, and we can write the
// final CAR for real under `dst`.
bs, err := blockstore.OpenReadWrite(dst, []cid.Cid{root},
car.ZeroLengthSectionAsEOF(true),
blockstore.UseWholeCIDs(true))
//------ Second Pass --- Now that we have the root of the Unixfs DAG -> write out the Unixfs DAG to a CARv2 file with the root present by using a
// filestore backed by a read-write CARv2 blockstore.
fsb, err := filestorecaradapter.NewReadWriteFileStore(outputCARv2Path, []cid.Cid{root})
if err != nil {
return cid.Undef, xerrors.Errorf("failed to create a carv2 read/write blockstore: %w", err)
return cid.Undef, xerrors.Errorf("failed to create a CARv2 read-write blockstore: %w", err)
}
defer fsb.Close() //nolint:errcheck
bsvc = blockservice.New(bs, offline.Exchange(bs))
dags = merkledag.NewDAGService(bsvc)
finalRoot, err := buildUnixFS(ctx, src, dags)
bsvc = blockservice.New(fsb, offline.Exchange(fsb))
root2, err := importNormalFileToUnixfsDAG(ctx, inputFilePath, merkledag.NewDAGService(bsvc))
if err != nil {
_ = bs.Close()
return cid.Undef, xerrors.Errorf("failed to create UnixFS DAG with carv2 blockstore: %w", err)
return cid.Undef, xerrors.Errorf("failed to create Unixfs DAG with CARv2 blockstore: %w", err)
}
if err := bs.Finalize(); err != nil {
return cid.Undef, xerrors.Errorf("failed to finalize car blockstore: %w", err)
}
if root != finalRoot {
if root != root2 {
return cid.Undef, xerrors.New("roots do not match")
}
return root, nil
}
// buildUnixFS builds a UnixFS DAG out of the supplied ordinary file,
// and imports the DAG into the supplied service.
func buildUnixFS(ctx context.Context, src string, dag ipld.DAGService) (cid.Cid, error) {
f, err := os.Open(src)
// importNormalFileToUnixfsDAG transforms a client's normal file to a UnixfsDAG and imports the DAG to the given DAG service.
func importNormalFileToUnixfsDAG(ctx context.Context, inputFilePath string, dag ipld.DAGService) (cid.Cid, error) {
f, err := os.Open(inputFilePath)
if err != nil {
return cid.Undef, xerrors.Errorf("failed to open input file: %w", err)
}
@@ -103,7 +81,7 @@ func buildUnixFS(ctx context.Context, src string, dag ipld.DAGService) (cid.Cid,
return cid.Undef, xerrors.Errorf("failed to stat file :%w", err)
}
file, err := files2.NewReaderPathFile(src, f, stat)
file, err := files2.NewReaderPathFile(inputFilePath, f, stat)
if err != nil {
return cid.Undef, xerrors.Errorf("failed to create reader path file: %w", err)
}
+33 -44
View File
@@ -4,11 +4,12 @@ import (
"context"
"io"
"io/ioutil"
"math/rand"
"os"
"strings"
"testing"
"github.com/filecoin-project/go-fil-markets/stores"
"github.com/filecoin-project/go-fil-markets/filestorecaradapter"
"github.com/filecoin-project/lotus/node/repo/importmgr"
"github.com/ipfs/go-blockservice"
"github.com/ipfs/go-cid"
@@ -21,50 +22,39 @@ import (
"github.com/stretchr/testify/require"
)
// This test uses a full "dense" CARv2, and not a filestore (positional mapping).
func TestRoundtripUnixFS_Dense(t *testing.T) {
func TestImportNormalFileToUnixfsDAG(t *testing.T) {
ctx := context.Background()
inputPath, inputContents := genInputFile(t)
inputPath, inputContents := genNormalInputFile(t)
defer os.Remove(inputPath) //nolint:errcheck
carv2File := newTmpFile(t)
carv2File := genTmpFile(t)
defer os.Remove(carv2File) //nolint:errcheck
// import a file to a Unixfs DAG using a CARv2 read/write blockstore.
path, err := blockstore.OpenReadWrite(carv2File, nil,
carv2.ZeroLengthSectionAsEOF(true),
blockstore.UseWholeCIDs(true))
// import a normal file to a Unixfs DAG using a CARv2 read-write blockstore and flush it out to a CARv2 file.
tempCARv2Store, err := blockstore.OpenReadWrite(carv2File, []cid.Cid{}, carv2.ZeroLengthSectionAsEOF(true), blockstore.UseWholeCIDs(true))
require.NoError(t, err)
bsvc := blockservice.New(path, offline.Exchange(path))
dags := merkledag.NewDAGService(bsvc)
root, err := buildUnixFS(ctx, inputPath, dags)
bsvc := blockservice.New(tempCARv2Store, offline.Exchange(tempCARv2Store))
root, err := importNormalFileToUnixfsDAG(ctx, inputPath, merkledag.NewDAGService(bsvc))
require.NoError(t, err)
require.NotEqual(t, cid.Undef, root)
require.NoError(t, path.Finalize())
require.NoError(t, tempCARv2Store.Finalize())
// reconstruct the file.
readOnly, err := blockstore.OpenReadOnly(carv2File,
carv2.ZeroLengthSectionAsEOF(true),
blockstore.UseWholeCIDs(true))
// convert the CARv2 file to a normal file again and ensure the contents match.
readOnly, err := blockstore.OpenReadOnly(carv2File, carv2.ZeroLengthSectionAsEOF(true), blockstore.UseWholeCIDs(true))
require.NoError(t, err)
defer readOnly.Close() //nolint:errcheck
dag := merkledag.NewDAGService(blockservice.New(readOnly, offline.Exchange(readOnly)))
dags = merkledag.NewDAGService(blockservice.New(readOnly, offline.Exchange(readOnly)))
nd, err := dags.Get(ctx, root)
nd, err := dag.Get(ctx, root)
require.NoError(t, err)
file, err := unixfile.NewUnixfsFile(ctx, dag, nd)
require.NoError(t, err)
file, err := unixfile.NewUnixfsFile(ctx, dags, nd)
require.NoError(t, err)
tmpOutput := newTmpFile(t)
tmpOutput := genTmpFile(t)
defer os.Remove(tmpOutput) //nolint:errcheck
require.NoError(t, files.WriteTo(file, tmpOutput))
// ensure contents of the initial input file and the output file are identical.
fo, err := os.Open(tmpOutput)
require.NoError(t, err)
bz2, err := ioutil.ReadAll(fo)
@@ -73,36 +63,35 @@ func TestRoundtripUnixFS_Dense(t *testing.T) {
require.Equal(t, inputContents, bz2)
}
func TestRoundtripUnixFS_Filestore(t *testing.T) {
func TestImportNormalFileToCARv2(t *testing.T) {
ctx := context.Background()
a := &API{
Imports: &importmgr.Mgr{},
}
importID := importmgr.ImportID(rand.Uint64())
inputFilePath, inputContents := genInputFile(t)
inputFilePath, inputContents := genNormalInputFile(t)
defer os.Remove(inputFilePath) //nolint:errcheck
path := newTmpFile(t)
defer os.Remove(path) //nolint:errcheck
outputCARv2 := genTmpFile(t)
defer os.Remove(outputCARv2) //nolint:errcheck
root, err := a.doImport(ctx, inputFilePath, path)
root, err := a.importNormalFileToFilestoreCARv2(ctx, importID, inputFilePath, outputCARv2)
require.NoError(t, err)
require.NotEqual(t, cid.Undef, root)
// convert the CARv2 to a normal file again and ensure the contents match
fs, err := stores.ReadOnlyFilestore(path)
readOnly, err := filestorecaradapter.NewReadOnlyFileStore(outputCARv2)
require.NoError(t, err)
defer fs.Close() //nolint:errcheck
defer readOnly.Close() //nolint:errcheck
dag := merkledag.NewDAGService(blockservice.New(readOnly, offline.Exchange(readOnly)))
dags := merkledag.NewDAGService(blockservice.New(fs, offline.Exchange(fs)))
nd, err := dags.Get(ctx, root)
nd, err := dag.Get(ctx, root)
require.NoError(t, err)
file, err := unixfile.NewUnixfsFile(ctx, dag, nd)
require.NoError(t, err)
file, err := unixfile.NewUnixfsFile(ctx, dags, nd)
require.NoError(t, err)
tmpOutput := newTmpFile(t)
tmpOutput := genTmpFile(t)
defer os.Remove(tmpOutput) //nolint:errcheck
require.NoError(t, files.WriteTo(file, tmpOutput))
@@ -115,14 +104,14 @@ func TestRoundtripUnixFS_Filestore(t *testing.T) {
require.Equal(t, inputContents, bz2)
}
func newTmpFile(t *testing.T) string {
func genTmpFile(t *testing.T) string {
f, err := os.CreateTemp("", "")
require.NoError(t, err)
require.NoError(t, f.Close())
return f.Name()
}
func genInputFile(t *testing.T) (filepath string, contents []byte) {
func genNormalInputFile(t *testing.T) (filepath string, contents []byte) {
s := strings.Repeat("abcde", 100)
tmp, err := os.CreateTemp("", "")
require.NoError(t, err)
+5 -238
View File
@@ -3,20 +3,16 @@ package impl
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"sort"
"strconv"
"time"
"github.com/filecoin-project/dagstore"
"github.com/filecoin-project/dagstore/shard"
"github.com/filecoin-project/go-jsonrpc/auth"
"github.com/filecoin-project/lotus/chain/actors/builtin"
"github.com/filecoin-project/lotus/chain/gen"
"github.com/filecoin-project/lotus/build"
"github.com/google/uuid"
"github.com/ipfs/go-cid"
"github.com/libp2p/go-libp2p-core/host"
@@ -24,13 +20,11 @@ import (
"go.uber.org/fx"
"golang.org/x/xerrors"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/go-address"
datatransfer "github.com/filecoin-project/go-data-transfer"
"github.com/filecoin-project/go-fil-markets/piecestore"
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
"github.com/filecoin-project/go-fil-markets/storagemarket"
retrievalmarket "github.com/filecoin-project/go-fil-markets/retrievalmarket"
storagemarket "github.com/filecoin-project/go-fil-markets/storagemarket"
"github.com/filecoin-project/go-state-types/abi"
sectorstorage "github.com/filecoin-project/lotus/extern/sector-storage"
@@ -40,8 +34,6 @@ import (
sealing "github.com/filecoin-project/lotus/extern/storage-sealing"
"github.com/filecoin-project/lotus/extern/storage-sealing/sealiface"
sto "github.com/filecoin-project/specs-storage/storage"
"github.com/filecoin-project/lotus/api"
apitypes "github.com/filecoin-project/lotus/api/types"
"github.com/filecoin-project/lotus/chain/types"
@@ -50,6 +42,7 @@ import (
"github.com/filecoin-project/lotus/node/modules/dtypes"
"github.com/filecoin-project/lotus/storage"
"github.com/filecoin-project/lotus/storage/sectorblocks"
sto "github.com/filecoin-project/specs-storage/storage"
)
type StorageMinerAPI struct {
@@ -58,8 +51,6 @@ type StorageMinerAPI struct {
api.Common
api.Net
EnabledSubsystems api.MinerSubsystems
Full api.FullNode
LocalStore *stores.Local
RemoteStore *stores.Remote
@@ -72,7 +63,6 @@ type StorageMinerAPI struct {
DealPublisher *storageadapter.DealPublisher `optional:"true"`
SectorBlocks *sectorblocks.SectorBlocks `optional:"true"`
Host host.Host `optional:"true"`
DAGStore *dagstore.DAGStore `optional:"true"`
// Miner / storage
Miner *storage.Miner `optional:"true"`
@@ -106,8 +96,6 @@ type StorageMinerAPI struct {
SetExpectedSealDurationFunc dtypes.SetExpectedSealDurationFunc `optional:"true"`
}
var _ api.StorageMiner = &StorageMinerAPI{}
func (sm *StorageMinerAPI) ServeRemote(perm bool) func(w http.ResponseWriter, r *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
if perm == true {
@@ -555,225 +543,6 @@ func (sm *StorageMinerAPI) MarketPublishPendingDeals(ctx context.Context) error
return nil
}
func (sm *StorageMinerAPI) DagstoreListShards(ctx context.Context) ([]api.DagstoreShardInfo, error) {
if sm.DAGStore == nil {
return nil, fmt.Errorf("dagstore not available on this node")
}
info := sm.DAGStore.AllShardsInfo()
ret := make([]api.DagstoreShardInfo, 0, len(info))
for k, i := range info {
ret = append(ret, api.DagstoreShardInfo{
Key: k.String(),
State: i.ShardState.String(),
Error: func() string {
if i.Error == nil {
return ""
}
return i.Error.Error()
}(),
})
}
// order by key.
sort.SliceStable(ret, func(i, j int) bool {
return ret[i].Key < ret[j].Key
})
return ret, nil
}
func (sm *StorageMinerAPI) DagstoreInitializeShard(ctx context.Context, key string) error {
if sm.DAGStore == nil {
return fmt.Errorf("dagstore not available on this node")
}
k := shard.KeyFromString(key)
info, err := sm.DAGStore.GetShardInfo(k)
if err != nil {
return fmt.Errorf("failed to get shard info: %w", err)
}
if st := info.ShardState; st != dagstore.ShardStateNew {
return fmt.Errorf("cannot initialize shard; expected state ShardStateNew, was: %s", st.String())
}
ch := make(chan dagstore.ShardResult, 1)
if err = sm.DAGStore.AcquireShard(ctx, k, ch, dagstore.AcquireOpts{}); err != nil {
return fmt.Errorf("failed to acquire shard: %w", err)
}
var res dagstore.ShardResult
select {
case res = <-ch:
case <-ctx.Done():
return ctx.Err()
}
if err := res.Error; err != nil {
return fmt.Errorf("failed to acquire shard: %w", err)
}
if res.Accessor != nil {
err = res.Accessor.Close()
if err != nil {
log.Warnw("failed to close shard accessor; continuing", "shard_key", k, "error", err)
}
}
return nil
}
func (sm *StorageMinerAPI) DagstoreInitializeAll(ctx context.Context, params api.DagstoreInitializeAllParams) (<-chan api.DagstoreInitializeAllEvent, error) {
if sm.DAGStore == nil {
return nil, fmt.Errorf("dagstore not available on this node")
}
// prepare the thottler tokens.
var throttle chan struct{}
if c := params.MaxConcurrency; c > 0 {
throttle = make(chan struct{}, c)
for i := 0; i < c; i++ {
throttle <- struct{}{}
}
}
info := sm.DAGStore.AllShardsInfo()
var uninit []string
for k, i := range info {
if i.ShardState != dagstore.ShardStateNew {
continue
}
uninit = append(uninit, k.String())
}
total := len(uninit)
if total == 0 {
out := make(chan api.DagstoreInitializeAllEvent)
close(out)
return out, nil
}
// response channel must be closed when we're done, or the context is cancelled.
// this buffering is necessary to prevent inflight children goroutines from
// publishing to a closed channel (res) when the context is cancelled.
out := make(chan api.DagstoreInitializeAllEvent, 32) // internal buffer.
res := make(chan api.DagstoreInitializeAllEvent, 32) // returned to caller.
// pump events back to caller.
// two events per shard.
go func() {
defer close(res)
for i := 0; i < total*2; i++ {
select {
case res <- <-out:
case <-ctx.Done():
return
}
}
}()
go func() {
for i, k := range uninit {
select {
case <-throttle:
// acquired a throttle token, proceed.
case <-ctx.Done():
return
}
go func(k string, i int) {
r := api.DagstoreInitializeAllEvent{
Key: k,
Event: "start",
Total: total,
Current: i + 1, // start with 1
}
select {
case out <- r:
case <-ctx.Done():
return
}
err := sm.DagstoreInitializeShard(ctx, k)
throttle <- struct{}{}
r.Event = "end"
if err == nil {
r.Success = true
} else {
r.Success = false
r.Error = err.Error()
}
select {
case out <- r:
case <-ctx.Done():
}
}(k, i)
}
}()
return res, nil
}
func (sm *StorageMinerAPI) DagstoreRecoverShard(ctx context.Context, key string) error {
if sm.DAGStore == nil {
return fmt.Errorf("dagstore not available on this node")
}
k := shard.KeyFromString(key)
info, err := sm.DAGStore.GetShardInfo(k)
if err != nil {
return fmt.Errorf("failed to get shard info: %w", err)
}
if st := info.ShardState; st != dagstore.ShardStateErrored {
return fmt.Errorf("cannot recover shard; expected state ShardStateErrored, was: %s", st.String())
}
ch := make(chan dagstore.ShardResult, 1)
if err = sm.DAGStore.RecoverShard(ctx, k, ch, dagstore.RecoverOpts{}); err != nil {
return fmt.Errorf("failed to recover shard: %w", err)
}
var res dagstore.ShardResult
select {
case res = <-ch:
case <-ctx.Done():
return ctx.Err()
}
return res.Error
}
func (sm *StorageMinerAPI) DagstoreGC(ctx context.Context) ([]api.DagstoreShardResult, error) {
if sm.DAGStore == nil {
return nil, fmt.Errorf("dagstore not available on this node")
}
res, err := sm.DAGStore.GC(ctx)
if err != nil {
return nil, fmt.Errorf("failed to gc: %w", err)
}
ret := make([]api.DagstoreShardResult, 0, len(res.Shards))
for k, err := range res.Shards {
r := api.DagstoreShardResult{Key: k.String()}
if err == nil {
r.Success = true
} else {
r.Success = false
r.Error = err.Error()
}
ret = append(ret, r)
}
return ret, nil
}
func (sm *StorageMinerAPI) DealsList(ctx context.Context) ([]api.MarketDeal, error) {
return sm.listDeals(ctx)
}
@@ -934,6 +703,4 @@ func (sm *StorageMinerAPI) ComputeProof(ctx context.Context, ssi []builtin.Secto
return sm.Epp.ComputeProof(ctx, ssi, rand)
}
func (sm *StorageMinerAPI) RuntimeSubsystems(context.Context) (res api.MinerSubsystems, err error) {
return sm.EnabledSubsystems, nil
}
var _ api.StorageMiner = &StorageMinerAPI{}
+1 -1
View File
@@ -182,7 +182,7 @@ func StorageClient(lc fx.Lifecycle, h host.Host, dataTransfer dtypes.ClientDataT
func RetrievalClient(lc fx.Lifecycle, h host.Host, r repo.LockedRepo, dt dtypes.ClientDataTransfer, payAPI payapi.PaychAPI, resolver discovery.PeerResolver,
ds dtypes.MetadataDS, chainAPI full.ChainAPI, stateAPI full.StateAPI, j journal.Journal) (retrievalmarket.RetrievalClient, error) {
carsPath := filepath.Join(r.Path(), DefaultDAGStoreDir, "retrieval-cars")
carsPath := filepath.Join(r.Path(), dagStore, "retrieval-cars")
if err := os.MkdirAll(carsPath, 0755); err != nil {
return nil, xerrors.Errorf("failed to create dir")
+99 -47
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"net/http"
"os"
"path"
"path/filepath"
"strings"
"time"
@@ -15,28 +16,26 @@ import (
"go.uber.org/multierr"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-address"
dtimpl "github.com/filecoin-project/go-data-transfer/impl"
dtnet "github.com/filecoin-project/go-data-transfer/network"
dtgstransport "github.com/filecoin-project/go-data-transfer/transport/graphsync"
"github.com/filecoin-project/go-jsonrpc/auth"
"github.com/filecoin-project/go-paramfetch"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-statestore"
"github.com/filecoin-project/go-storedcounter"
"github.com/ipfs/go-bitswap"
"github.com/ipfs/go-bitswap/network"
"github.com/ipfs/go-blockservice"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-datastore"
"github.com/ipfs/go-datastore/namespace"
levelds "github.com/ipfs/go-ds-leveldb"
measure "github.com/ipfs/go-ds-measure"
graphsync "github.com/ipfs/go-graphsync/impl"
gsnet "github.com/ipfs/go-graphsync/network"
"github.com/ipfs/go-graphsync/storeutil"
"github.com/ipfs/go-merkledag"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/routing"
ldbopts "github.com/syndtr/goleveldb/leveldb/opt"
"github.com/filecoin-project/go-address"
dtimpl "github.com/filecoin-project/go-data-transfer/impl"
dtnet "github.com/filecoin-project/go-data-transfer/network"
dtgstransport "github.com/filecoin-project/go-data-transfer/transport/graphsync"
piecefilestore "github.com/filecoin-project/go-fil-markets/filestore"
piecestoreimpl "github.com/filecoin-project/go-fil-markets/piecestore/impl"
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
@@ -47,6 +46,11 @@ import (
storageimpl "github.com/filecoin-project/go-fil-markets/storagemarket/impl"
"github.com/filecoin-project/go-fil-markets/storagemarket/impl/storedask"
smnet "github.com/filecoin-project/go-fil-markets/storagemarket/network"
"github.com/filecoin-project/go-jsonrpc/auth"
"github.com/filecoin-project/go-paramfetch"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-statestore"
"github.com/filecoin-project/go-storedcounter"
sectorstorage "github.com/filecoin-project/lotus/extern/sector-storage"
"github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper"
@@ -77,6 +81,7 @@ import (
)
var StorageCounterDSPrefix = "/storage/nextid"
var dagStore = "dagStore"
func minerAddrFromDS(ds dtypes.MetadataDS) (address.Address, error) {
maddrb, err := ds.Get(datastore.NewKey("miner-address"))
@@ -575,6 +580,84 @@ func BasicDealFilter(user dtypes.StorageDealFilter) func(onlineOk dtypes.Conside
}
}
func NewLotusAccessor(lc fx.Lifecycle,
pieceStore dtypes.ProviderPieceStore,
rpn retrievalmarket.RetrievalProviderNode,
) (dagstore.LotusAccessor, error) {
mountApi := dagstore.NewLotusAccessor(pieceStore, rpn)
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
return mountApi.Start(ctx)
},
OnStop: func(context.Context) error {
return nil
},
})
return mountApi, nil
}
func DagStoreWrapper(
lc fx.Lifecycle,
r repo.LockedRepo,
lotusAccessor dagstore.LotusAccessor,
) (*dagstore.Wrapper, error) {
dagStoreDir := filepath.Join(r.Path(), dagStore)
dagStoreDS, err := createDAGStoreDatastore(dagStoreDir)
if err != nil {
return nil, err
}
cfg := dagstore.MarketDAGStoreConfig{
TransientsDir: filepath.Join(dagStoreDir, "transients"),
IndexDir: filepath.Join(dagStoreDir, "index"),
Datastore: dagStoreDS,
GCInterval: 1 * time.Minute,
MaxConcurrentIndex: 5,
}
dsw, err := dagstore.NewDagStoreWrapper(cfg, lotusAccessor)
if err != nil {
return nil, xerrors.Errorf("failed to create DAG store wrapper: %w", err)
}
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
dsw.Start(ctx)
return nil
},
OnStop: func(context.Context) error {
return dsw.Close()
},
})
return dsw, nil
}
// createDAGStoreDatastore creates a datastore under the given base directory
// for DAG store metadata
func createDAGStoreDatastore(baseDir string) (datastore.Batching, error) {
// Create a datastore directory under the base dir if it doesn't already exist
dsDir := path.Join(baseDir, "datastore")
if err := os.MkdirAll(dsDir, 0755); err != nil {
return nil, xerrors.Errorf("failed to create directory %s for DAG store datastore: %w", dsDir, err)
}
// Create a new LevelDB datastore
ds, err := levelds.NewDatastore(dsDir, &levelds.Options{
Compression: ldbopts.NoCompression,
NoSync: false,
Strict: ldbopts.StrictAll,
ReadOnly: false,
})
if err != nil {
return nil, xerrors.Errorf("failed to open datastore for DAG store: %w", err)
}
// Keep statistics about the datastore
mds := measure.New("measure.", ds)
return mds, nil
}
func StorageProvider(minerAddress dtypes.MinerAddress,
storedAsk *storedask.StoredAsk,
h host.Host, ds dtypes.MetadataDS,
@@ -586,33 +669,18 @@ func StorageProvider(minerAddress dtypes.MinerAddress,
dsw *dagstore.Wrapper,
) (storagemarket.StorageProvider, error) {
net := smnet.NewFromLibp2pHost(h)
// Create a directory under the repo path for temporary files
repoTmpPath := filepath.Join(r.Path(), "tmp")
err := os.MkdirAll(repoTmpPath, 0755) //nolint: gosec
if err != nil {
return nil, xerrors.Errorf("creating tmp dir for storage provider repo: %w", err)
}
store, err := piecefilestore.NewLocalFileStore(piecefilestore.OsPath(repoTmpPath))
store, err := piecefilestore.NewLocalFileStore(piecefilestore.OsPath(r.Path()))
if err != nil {
return nil, err
}
opt := storageimpl.CustomDealDecisionLogic(storageimpl.DealDeciderFunc(df))
dagStorePath := filepath.Join(r.Path(), dagStore)
return storageimpl.NewProvider(
net,
namespace.Wrap(ds, datastore.NewKey("/deals/provider")),
store,
dsw,
pieceStore,
dataTransfer,
spn,
address.Address(minerAddress),
storedAsk,
opt,
)
opt := storageimpl.CustomDealDecisionLogic(storageimpl.DealDeciderFunc(df))
shardMigrator := storageimpl.NewShardMigrator(address.Address(minerAddress), dagStorePath, dsw, pieceStore, spn)
return storageimpl.NewProvider(net, namespace.Wrap(ds, datastore.NewKey("/deals/provider")), store, dsw, pieceStore,
dataTransfer, spn, address.Address(minerAddress), storedAsk, shardMigrator, opt)
}
func RetrievalDealFilter(userFilter dtypes.RetrievalDealFilter) func(onlineOk dtypes.ConsiderOnlineRetrievalDealsConfigFunc,
@@ -1014,19 +1082,3 @@ func mutateCfg(r repo.LockedRepo, mutator func(*config.StorageMiner)) error {
return multierr.Combine(typeErr, setConfigErr)
}
func ExtractEnabledMinerSubsystems(cfg config.MinerSubsystemConfig) (res api.MinerSubsystems) {
if cfg.EnableMining {
res = append(res, api.SubsystemMining)
}
if cfg.EnableSealing {
res = append(res, api.SubsystemSealing)
}
if cfg.EnableSectorStorage {
res = append(res, api.SubsystemSectorStorage)
}
if cfg.EnableMarkets {
res = append(res, api.SubsystemMarkets)
}
return res
}
-112
View File
@@ -1,112 +0,0 @@
package modules
import (
"context"
"fmt"
"os"
"path/filepath"
"strconv"
"github.com/filecoin-project/dagstore"
"go.uber.org/fx"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
mdagstore "github.com/filecoin-project/lotus/markets/dagstore"
"github.com/filecoin-project/lotus/node/config"
"github.com/filecoin-project/lotus/node/modules/dtypes"
"github.com/filecoin-project/lotus/node/repo"
)
const (
EnvDAGStoreCopyConcurrency = "LOTUS_DAGSTORE_COPY_CONCURRENCY"
DefaultDAGStoreDir = "dagstore"
)
// NewMinerAPI creates a new MinerAPI adaptor for the dagstore mounts.
func NewMinerAPI(lc fx.Lifecycle, r repo.LockedRepo, pieceStore dtypes.ProviderPieceStore, rpn retrievalmarket.RetrievalProviderNode) (mdagstore.MinerAPI, error) {
cfg, err := extractDAGStoreConfig(r)
if err != nil {
return nil, err
}
// caps the amount of concurrent calls to the storage, so that we don't
// spam it during heavy processes like bulk migration.
if v, ok := os.LookupEnv("LOTUS_DAGSTORE_MOUNT_CONCURRENCY"); ok {
concurrency, err := strconv.Atoi(v)
if err == nil {
cfg.MaxConcurrencyStorageCalls = concurrency
}
}
mountApi := mdagstore.NewMinerAPI(pieceStore, rpn, cfg.MaxConcurrencyStorageCalls)
ready := make(chan error, 1)
pieceStore.OnReady(func(err error) {
ready <- err
})
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
if err := <-ready; err != nil {
return fmt.Errorf("aborting dagstore start; piecestore failed to start: %s", err)
}
return mountApi.Start(ctx)
},
OnStop: func(context.Context) error {
return nil
},
})
return mountApi, nil
}
// DAGStore constructs a DAG store using the supplied minerAPI, and the
// user configuration. It returns both the DAGStore and the Wrapper suitable for
// passing to markets.
func DAGStore(lc fx.Lifecycle, r repo.LockedRepo, minerAPI mdagstore.MinerAPI) (*dagstore.DAGStore, *mdagstore.Wrapper, error) {
cfg, err := extractDAGStoreConfig(r)
if err != nil {
return nil, nil, err
}
// fall back to default root directory if not explicitly set in the config.
if cfg.RootDir == "" {
cfg.RootDir = filepath.Join(r.Path(), DefaultDAGStoreDir)
}
v, ok := os.LookupEnv(EnvDAGStoreCopyConcurrency)
if ok {
concurrency, err := strconv.Atoi(v)
if err == nil {
cfg.MaxConcurrentReadyFetches = concurrency
}
}
dagst, w, err := mdagstore.NewDAGStore(cfg, minerAPI)
if err != nil {
return nil, nil, xerrors.Errorf("failed to create DAG store: %w", err)
}
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
return w.Start(ctx)
},
OnStop: func(context.Context) error {
return w.Close()
},
})
return dagst, w, nil
}
func extractDAGStoreConfig(r repo.LockedRepo) (config.DAGStoreConfig, error) {
cfg, err := r.Config()
if err != nil {
return config.DAGStoreConfig{}, xerrors.Errorf("could not load config: %w", err)
}
mcfg, ok := cfg.(*config.StorageMiner)
if !ok {
return config.DAGStoreConfig{}, xerrors.Errorf("config not expected type; expected config.StorageMiner, got: %T", cfg)
}
return mcfg.DAGStore, nil
}
+1 -19
View File
@@ -49,31 +49,13 @@ const (
StorageMiner
Worker
Wallet
Markets
)
func (t RepoType) String() string {
s := [...]string{
"__invalid__",
"FullNode",
"StorageMiner",
"Worker",
"Wallet",
"Markets",
}
if t < 0 || int(t) > len(s) {
return "__invalid__"
}
return s[t]
}
func defConfForType(t RepoType) interface{} {
switch t {
case FullNode:
return config.DefaultFullNode()
case StorageMiner, Markets:
// markets is a specialised miner service
// this taxonomy needs to be cleaned up
case StorageMiner:
return config.DefaultStorageMiner()
case Worker:
return &struct{}{}
-170
View File
@@ -1,170 +0,0 @@
- github.com/filecoin-project/lotus:
- Merge branch 'releases' into release/v1.11.1
- Update to proof v8.0.3 ([filecoin-project/lotus#6890](https://github.com/filecoin-project/lotus/pull/6890))
- lotus-shed: initial export cmd for markets related metadata ([filecoin-project/lotus#6840](https://github.com/filecoin-project/lotus/pull/6840))
- add a very verbose -vv flag to lotus and lotus-miner. ([filecoin-project/lotus#6888](https://github.com/filecoin-project/lotus/pull/6888))
- Update RELEASE_ISSUE_TEMPLATE.md ([filecoin-project/lotus#6880](https://github.com/filecoin-project/lotus/pull/6880))
- Moving GC for badger ([filecoin-project/lotus#6854](https://github.com/filecoin-project/lotus/pull/6854))
- Add github actions for staled pr ([filecoin-project/lotus#6879](https://github.com/filecoin-project/lotus/pull/6879))
- Add allocated sectorid vis ([filecoin-project/lotus#4638](https://github.com/filecoin-project/lotus/pull/4638))
- rename `cmd/lotus{-storage=>}-miner` to match binary. ([filecoin-project/lotus#6886](https://github.com/filecoin-project/lotus/pull/6886))
- update to go-fil-market v1.6.0 ([filecoin-project/lotus#6885](https://github.com/filecoin-project/lotus/pull/6885))
- Bump go-multihash, adjust test for supported version ([filecoin-project/lotus#6674](https://github.com/filecoin-project/lotus/pull/6674))
- Fix padding of deals, which only partially shipped in #5988 ([filecoin-project/lotus#6683](https://github.com/filecoin-project/lotus/pull/6683))
- fix racy TestSimultanenousTransferLimit. ([filecoin-project/lotus#6862](https://github.com/filecoin-project/lotus/pull/6862))
- Improve splitstore warmup ([filecoin-project/lotus#6867](https://github.com/filecoin-project/lotus/pull/6867))
- ValidateBlock: Assert that block header height's are greater than parents ([filecoin-project/lotus#6872](https://github.com/filecoin-project/lotus/pull/6872))
- feat: Don't panic when api impl is nil ([filecoin-project/lotus#6857](https://github.com/filecoin-project/lotus/pull/6857))
- splitstore shed utils ([filecoin-project/lotus#6811](https://github.com/filecoin-project/lotus/pull/6811))
- Fix links in issue templates
- Update issue templates and add templates for M1 ([filecoin-project/lotus#6856](https://github.com/filecoin-project/lotus/pull/6856))
- Splitstore: support on-disk marksets using badger ([filecoin-project/lotus#6833](https://github.com/filecoin-project/lotus/pull/6833))
- Config UX improvements ([filecoin-project/lotus#6848](https://github.com/filecoin-project/lotus/pull/6848))
- fix deal concurrency test failures by upgrading graphsync and others ([filecoin-project/lotus#6724](https://github.com/filecoin-project/lotus/pull/6724))
- Update issue templates to forms ([filecoin-project/lotus#6798](https://github.com/filecoin-project/lotus/pull/6798))
- Nerpa v13 upgrade ([filecoin-project/lotus#6837](https://github.com/filecoin-project/lotus/pull/6837))
- add docker-compose file ([filecoin-project/lotus#6544](https://github.com/filecoin-project/lotus/pull/6544))
- fix warmup by decoupling state from message receipt walk ([filecoin-project/lotus#6841](https://github.com/filecoin-project/lotus/pull/6841))
- add a command for compacting sector numbers bitfield ([filecoin-project/lotus#4640](https://github.com/filecoin-project/lotus/pull/4640))
- PriceListByVersion ([filecoin-project/lotus#6766](https://github.com/filecoin-project/lotus/pull/6766))
- easy way to make install app ([filecoin-project/lotus#5183](https://github.com/filecoin-project/lotus/pull/5183))
- api: Separate the Net interface from Common ([filecoin-project/lotus#6627](https://github.com/filecoin-project/lotus/pull/6627))
- cache loaded block messages ([filecoin-project/lotus#6760](https://github.com/filecoin-project/lotus/pull/6760))
- fix: on randomness change, use new rand ([filecoin-project/lotus#6805](https://github.com/filecoin-project/lotus/pull/6805))
- Splitstore: add retention policy option for keeping messages in the hotstore ([filecoin-project/lotus#6775](https://github.com/filecoin-project/lotus/pull/6775))
- Introduce the LOTUS_CHAIN_BADGERSTORE_DISABLE_FSYNC envvar ([filecoin-project/lotus#6817](https://github.com/filecoin-project/lotus/pull/6817))
- add StateReadState to gateway api ([filecoin-project/lotus#6818](https://github.com/filecoin-project/lotus/pull/6818))
- add SealProof in SectorBuilder ([filecoin-project/lotus#6815](https://github.com/filecoin-project/lotus/pull/6815))
- release -> master ([filecoin-project/lotus#6828](https://github.com/filecoin-project/lotus/pull/6828))
- sealing: Handle preCommitParams errors more correctly ([filecoin-project/lotus#6763](https://github.com/filecoin-project/lotus/pull/6763))
- fix: always check if StateSearchMessage returns nil ([filecoin-project/lotus#6802](https://github.com/filecoin-project/lotus/pull/6802))
- ClientFindData: always fetch peer id from chain ([filecoin-project/lotus#6807](https://github.com/filecoin-project/lotus/pull/6807))
- test: fix flaky window post tests ([filecoin-project/lotus#6804](https://github.com/filecoin-project/lotus/pull/6804))
- ([filecoin-project/lotus#6800](https://github.com/filecoin-project/lotus/pull/6800))
- fixes #6786 segfault ([filecoin-project/lotus#6787](https://github.com/filecoin-project/lotus/pull/6787))
- Splitstore: add support for protecting out of chain references in the blockstore ([filecoin-project/lotus#6777](https://github.com/filecoin-project/lotus/pull/6777))
- Resurrect CODEOWNERS, but for maintainers group ([filecoin-project/lotus#6773](https://github.com/filecoin-project/lotus/pull/6773))
- update go-libp2p-pubsub to v0.5.0 ([filecoin-project/lotus#6764](https://github.com/filecoin-project/lotus/pull/6764))
- Implement exposed splitstore ([filecoin-project/lotus#6762](https://github.com/filecoin-project/lotus/pull/6762))
- Add ChainGetMessagesInTipset API ([filecoin-project/lotus#6642](https://github.com/filecoin-project/lotus/pull/6642))
- test: handle null blocks in TestForkRefuseCall ([filecoin-project/lotus#6758](https://github.com/filecoin-project/lotus/pull/6758))
- Master disclaimer ([filecoin-project/lotus#6757](https://github.com/filecoin-project/lotus/pull/6757))
- Splitstore code reorg ([filecoin-project/lotus#6756](https://github.com/filecoin-project/lotus/pull/6756))
- Create stale.yml ([filecoin-project/lotus#6747](https://github.com/filecoin-project/lotus/pull/6747))
- Splitstore: Some small fixes ([filecoin-project/lotus#6754](https://github.com/filecoin-project/lotus/pull/6754))
- ([filecoin-project/lotus#6746](https://github.com/filecoin-project/lotus/pull/6746))
- Handle the --color flag via proper global state ([filecoin-project/lotus#6743](https://github.com/filecoin-project/lotus/pull/6743))
- Config for collateral from miner available balance ([filecoin-project/lotus#6629](https://github.com/filecoin-project/lotus/pull/6629))
- Support standalone miner-market process ([filecoin-project/lotus#6356](https://github.com/filecoin-project/lotus/pull/6356))
- Splitstore Enhanchements ([filecoin-project/lotus#6474](https://github.com/filecoin-project/lotus/pull/6474))
- ([filecoin-project/lotus#6739](https://github.com/filecoin-project/lotus/pull/6739))
- Add more deal details to lotus-miner info ([filecoin-project/lotus#6708](https://github.com/filecoin-project/lotus/pull/6708))
- Release template: Update all testnet infra at once ([filecoin-project/lotus#6710](https://github.com/filecoin-project/lotus/pull/6710))
- Fix Lotus shed
- Fix bugs in sectors extend --v1-sectors ([filecoin-project/lotus#6066](https://github.com/filecoin-project/lotus/pull/6066))
- add election backtest ([filecoin-project/lotus#5950](https://github.com/filecoin-project/lotus/pull/5950))
- Envvar to disable slash filter ([filecoin-project/lotus#6620](https://github.com/filecoin-project/lotus/pull/6620))
- Release Template: remove binary validation step ([filecoin-project/lotus#6709](https://github.com/filecoin-project/lotus/pull/6709))
- Config for deal publishing control addresses ([filecoin-project/lotus#6697](https://github.com/filecoin-project/lotus/pull/6697))
- Reset of the interop network ([filecoin-project/lotus#6689](https://github.com/filecoin-project/lotus/pull/6689))
- Enable color by default only if os.Stdout is a TTY ([filecoin-project/lotus#6696](https://github.com/filecoin-project/lotus/pull/6696))
- Stop outputing ANSI color on non-TTY ([filecoin-project/lotus#6694](https://github.com/filecoin-project/lotus/pull/6694))
- add dollar sign ([filecoin-project/lotus#6690](https://github.com/filecoin-project/lotus/pull/6690))
- get-actor cli spelling fix ([filecoin-project/lotus#6681](https://github.com/filecoin-project/lotus/pull/6681))
- fix "lotus-seed genesis car" error "merkledag: not found" ([filecoin-project/lotus#6688](https://github.com/filecoin-project/lotus/pull/6688))
- polish(statetree): accept a context in statetree diff for timeouts ([filecoin-project/lotus#6639](https://github.com/filecoin-project/lotus/pull/6639))
- Add helptext to lotus chain export ([filecoin-project/lotus#6672](https://github.com/filecoin-project/lotus/pull/6672))
- Get retrieval pricing input should not error out on a deal state fetch ([filecoin-project/lotus#6679](https://github.com/filecoin-project/lotus/pull/6679))
- Fix more CID double-encoding as hex ([filecoin-project/lotus#6680](https://github.com/filecoin-project/lotus/pull/6680))
- add an incremental nonce itest. ([filecoin-project/lotus#6663](https://github.com/filecoin-project/lotus/pull/6663))
- storage: Fix FinalizeSector with sectors in stoage paths ([filecoin-project/lotus#6653](https://github.com/filecoin-project/lotus/pull/6653))
- Fix tiny error in check-client-datacap ([filecoin-project/lotus#6664](https://github.com/filecoin-project/lotus/pull/6664))
- Fix: precommit_batch method used the wrong cfg.CommitBatchWait ([filecoin-project/lotus#6658](https://github.com/filecoin-project/lotus/pull/6658))
- fix ticket expiration check ([filecoin-project/lotus#6635](https://github.com/filecoin-project/lotus/pull/6635))
- commit batch: AggregateAboveBaseFee config ([filecoin-project/lotus#6650](https://github.com/filecoin-project/lotus/pull/6650))
- commit batch: Initialize the FailedSectors map ([filecoin-project/lotus#6647](https://github.com/filecoin-project/lotus/pull/6647))
- Fast-path retry submitting commit aggregate if commit is still valid ([filecoin-project/lotus#6638](https://github.com/filecoin-project/lotus/pull/6638))
- remove precommit check in handleCommitFailed ([filecoin-project/lotus#6634](https://github.com/filecoin-project/lotus/pull/6634))
- Reuse timers in sealing batch logic ([filecoin-project/lotus#6636](https://github.com/filecoin-project/lotus/pull/6636))
- shed tool to estimate aggregate network fees ([filecoin-project/lotus#6631](https://github.com/filecoin-project/lotus/pull/6631))
- fix prove commit aggregate send token amount ([filecoin-project/lotus#6625](https://github.com/filecoin-project/lotus/pull/6625))
- Update version.go to 1.11.1 ([filecoin-project/lotus#6621](https://github.com/filecoin-project/lotus/pull/6621))
- github.com/filecoin-project/go-data-transfer (v1.6.0 -> v1.7.0):
- release: v1.7.0
- Fire a transfer queued event when a transfer is queued in Graphsync (#221) ([filecoin-project/go-data-transfer#221](https://github.com/filecoin-project/go-data-transfer/pull/221))
- feat: pass ChannelID to ValidatePush & ValidatePull (#220) ([filecoin-project/go-data-transfer#220](https://github.com/filecoin-project/go-data-transfer/pull/220))
- release: v1.6.1 ([filecoin-project/go-data-transfer#218](https://github.com/filecoin-project/go-data-transfer/pull/218))
- Remove CID lists (#217) ([filecoin-project/go-data-transfer#217](https://github.com/filecoin-project/go-data-transfer/pull/217))
- Merge v1.6.0 ([filecoin-project/go-data-transfer#214](https://github.com/filecoin-project/go-data-transfer/pull/214))
- Remove restart ack timeout (#211) ([filecoin-project/go-data-transfer#211](https://github.com/filecoin-project/go-data-transfer/pull/211))
- feat: use different extension names to fit multiple hooks data in same graphsync message (#204) ([filecoin-project/go-data-transfer#204](https://github.com/filecoin-project/go-data-transfer/pull/204))
- fix: map race in GS transport (#208) ([filecoin-project/go-data-transfer#208](https://github.com/filecoin-project/go-data-transfer/pull/208))
- refactor: simplify graphsync transport (#203) ([filecoin-project/go-data-transfer#203](https://github.com/filecoin-project/go-data-transfer/pull/203))
- release: v1.5.0 (#200) ([filecoin-project/go-data-transfer#200](https://github.com/filecoin-project/go-data-transfer/pull/200))
- github.com/filecoin-project/go-fil-markets (v1.5.0 -> v1.6.0):
- release: v1.6.0
- support padding out smaller files (#536) ([filecoin-project/go-fil-markets#536](https://github.com/filecoin-project/go-fil-markets/pull/536))
- On overloaded CI 10 seconds just isn't enough (#587) ([filecoin-project/go-fil-markets#587](https://github.com/filecoin-project/go-fil-markets/pull/587))
- Do not hex-encode CIDs in logs (#561) ([filecoin-project/go-fil-markets#561](https://github.com/filecoin-project/go-fil-markets/pull/561))
- remove wrong peer check in push deal validation (#585) ([filecoin-project/go-fil-markets#585](https://github.com/filecoin-project/go-fil-markets/pull/585))
- fix: circleci docs-gen task (#574) ([filecoin-project/go-fil-markets#574](https://github.com/filecoin-project/go-fil-markets/pull/574))
- Storage market request queued event and validation interface changes (#555) ([filecoin-project/go-fil-markets#555](https://github.com/filecoin-project/go-fil-markets/pull/555))
- build(deps): bump ws from 6.2.1 to 6.2.2 (#554) ([filecoin-project/go-fil-markets#554](https://github.com/filecoin-project/go-fil-markets/pull/554))
- release: v1.5.0 ([filecoin-project/go-fil-markets#553](https://github.com/filecoin-project/go-fil-markets/pull/553))
- github.com/filecoin-project/go-padreader (v0.0.0-20200903213702-ed5fae088b20 -> v0.0.0-20210723183308-812a16dc01b1):
- New method to pad harder (#6) ([filecoin-project/go-padreader#6](https://github.com/filecoin-project/go-padreader/pull/6))
- Create SECURITY.md (#5) ([filecoin-project/go-padreader#5](https://github.com/filecoin-project/go-padreader/pull/5))
- github.com/filecoin-project/go-state-types (v0.1.1-0.20210506134452-99b279731c48 -> v0.1.1-0.20210722133031-ad9bfe54c124):
- Add version 6.5 (#30) ([filecoin-project/go-state-types#30](https://github.com/filecoin-project/go-state-types/pull/30))
- rename file
- github.com/filecoin-project/specs-actors/v5 (v5.0.1 -> v5.0.3):
- Adjust code for subtle change in go-multihash 0.0.15 (#1463) ([filecoin-project/specs-actors#1463](https://github.com/filecoin-project/specs-actors/pull/1463))
- Bump go state types (#1464) ([filecoin-project/specs-actors#1464](https://github.com/filecoin-project/specs-actors/pull/1464))
- Create CODEOWNERS (#1465) ([filecoin-project/specs-actors#1465](https://github.com/filecoin-project/specs-actors/pull/1465))
- Test deterministic offset (#1462) ([filecoin-project/specs-actors#1462](https://github.com/filecoin-project/specs-actors/pull/1462))
Contributors
| Contributor | Commits | Lines ± | Files Changed |
|-------------|---------|---------|---------------|
| vyzo | 295 | +8700/-5936 | 397 |
| Anton Evangelatov | 94 | +4680/-2965 | 277 |
| Łukasz Magiera | 37 | +3851/-1611 | 146 |
| Mike Greenberg | 1 | +2310/-578 | 8 |
| dirkmc | 7 | +1154/-726 | 29 |
| Jennifer Wang | 9 | +485/-341 | 26 |
| Peter Rabbitson | 18 | +469/-273 | 64 |
| Cory Schwartz | 5 | +576/-135 | 14 |
| hunjixin | 7 | +404/-82 | 19 |
| ZenGround0 | 17 | +284/-135 | 44 |
| Dirk McCormick | 17 | +348/-47 | 17 |
| Raúl Kripalani | 18 | +254/-97 | 62 |
| tchardin | 1 | +261/-33 | 4 |
| Jakub Sztandera | 4 | +254/-16 | 4 |
| Aarsh Shah | 2 | +196/-40 | 28 |
| whyrusleeping | 3 | +150/-9 | 8 |
| Whyrusleeping | 2 | +87/-66 | 10 |
| Aayush Rajasekaran | 10 | +81/-53 | 13 |
| zgfzgf | 2 | +104/-4 | 2 |
| aarshkshah1992 | 4 | +73/-7 | 6 |
| llifezou | 4 | +59/-20 | 4 |
| Steven Allen | 7 | +47/-17 | 9 |
| johnli-helloworld | 3 | +46/-15 | 5 |
| frrist | 1 | +28/-23 | 2 |
| Jennifer | 4 | +31/-2 | 4 |
| wangchao | 1 | +1/-27 | 1 |
| Jiaying Wang | 2 | +7/-21 | 2 |
| hannahhoward | 3 | +21/-2 | 3 |
| chadwick2143 | 1 | +15/-1 | 1 |
| Jerry | 2 | +9/-4 | 2 |
| Steve Loeppky | 2 | +12/-0 | 2 |
| David Dias | 1 | +9/-0 | 1 |
| dependabot[bot] | 1 | +3/-3 | 1 |
| zhoutian527 | 1 | +2/-2 | 1 |
| xloem | 1 | +4/-0 | 1 |
| Travis Person | 2 | +2/-2 | 3 |
| Liviu Damian | 2 | +2/-2 | 2 |
| Jim Pick | 2 | +2/-2 | 2 |
| Frank | 1 | +3/-0 | 1 |
| turuslan | 1 | +1/-1 | 1 |
| Kirk Baird | 1 | +0/-0 | 1 |