Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1757ffcb2c |
-188
@@ -1,186 +1,5 @@
|
||||
# Lotus changelog
|
||||
|
||||
# 1.11.1-rc2 / 2021-08-01
|
||||
|
||||
> Note: for discussion about this release, please comment [here](https://github.com/filecoin-project/lotus/discussions/6904)
|
||||
|
||||
This is the second release candidate for the **highly-recommended** but optional Lotus v1.11.1 release that introduces many deal making and datastore improvements and new features along with other bug fixes. A more organized and detailed release note will be shared in the next few days with future RCs, highlights are:
|
||||
- **Experimental** [lotus-miner market subsystem](https://docs.filecoin.io/mine/lotus/split-markets-miners/#frontmatter-title)
|
||||
- **Under Testing** [splistore](https://github.com/filecoin-project/lotus/blob/master/blockstore/splitstore/README.md)
|
||||
|
||||
- github.com/filecoin-project/lotus:
|
||||
- Merge branch 'releases' into release/v1.11.1
|
||||
- Update to proof v8.0.3 ([filecoin-project/lotus#6890](https://github.com/filecoin-project/lotus/pull/6890))
|
||||
- lotus-shed: initial export cmd for markets related metadata ([filecoin-project/lotus#6840](https://github.com/filecoin-project/lotus/pull/6840))
|
||||
- add a very verbose -vv flag to lotus and lotus-miner. ([filecoin-project/lotus#6888](https://github.com/filecoin-project/lotus/pull/6888))
|
||||
- Update RELEASE_ISSUE_TEMPLATE.md ([filecoin-project/lotus#6880](https://github.com/filecoin-project/lotus/pull/6880))
|
||||
- Moving GC for badger ([filecoin-project/lotus#6854](https://github.com/filecoin-project/lotus/pull/6854))
|
||||
- Add github actions for staled pr ([filecoin-project/lotus#6879](https://github.com/filecoin-project/lotus/pull/6879))
|
||||
- Add allocated sectorid vis ([filecoin-project/lotus#4638](https://github.com/filecoin-project/lotus/pull/4638))
|
||||
- rename `cmd/lotus{-storage=>}-miner` to match binary. ([filecoin-project/lotus#6886](https://github.com/filecoin-project/lotus/pull/6886))
|
||||
- update to go-fil-market v1.6.0 ([filecoin-project/lotus#6885](https://github.com/filecoin-project/lotus/pull/6885))
|
||||
- Bump go-multihash, adjust test for supported version ([filecoin-project/lotus#6674](https://github.com/filecoin-project/lotus/pull/6674))
|
||||
- Fix padding of deals, which only partially shipped in #5988 ([filecoin-project/lotus#6683](https://github.com/filecoin-project/lotus/pull/6683))
|
||||
- fix racy TestSimultanenousTransferLimit. ([filecoin-project/lotus#6862](https://github.com/filecoin-project/lotus/pull/6862))
|
||||
- Improve splitstore warmup ([filecoin-project/lotus#6867](https://github.com/filecoin-project/lotus/pull/6867))
|
||||
- ValidateBlock: Assert that block header height's are greater than parents ([filecoin-project/lotus#6872](https://github.com/filecoin-project/lotus/pull/6872))
|
||||
- feat: Don't panic when api impl is nil ([filecoin-project/lotus#6857](https://github.com/filecoin-project/lotus/pull/6857))
|
||||
- splitstore shed utils ([filecoin-project/lotus#6811](https://github.com/filecoin-project/lotus/pull/6811))
|
||||
- Fix links in issue templates
|
||||
- Update issue templates and add templates for M1 ([filecoin-project/lotus#6856](https://github.com/filecoin-project/lotus/pull/6856))
|
||||
- Splitstore: support on-disk marksets using badger ([filecoin-project/lotus#6833](https://github.com/filecoin-project/lotus/pull/6833))
|
||||
- Config UX improvements ([filecoin-project/lotus#6848](https://github.com/filecoin-project/lotus/pull/6848))
|
||||
- fix deal concurrency test failures by upgrading graphsync and others ([filecoin-project/lotus#6724](https://github.com/filecoin-project/lotus/pull/6724))
|
||||
- Update issue templates to forms ([filecoin-project/lotus#6798](https://github.com/filecoin-project/lotus/pull/6798))
|
||||
- Nerpa v13 upgrade ([filecoin-project/lotus#6837](https://github.com/filecoin-project/lotus/pull/6837))
|
||||
- add docker-compose file ([filecoin-project/lotus#6544](https://github.com/filecoin-project/lotus/pull/6544))
|
||||
- fix warmup by decoupling state from message receipt walk ([filecoin-project/lotus#6841](https://github.com/filecoin-project/lotus/pull/6841))
|
||||
- add a command for compacting sector numbers bitfield ([filecoin-project/lotus#4640](https://github.com/filecoin-project/lotus/pull/4640))
|
||||
- PriceListByVersion ([filecoin-project/lotus#6766](https://github.com/filecoin-project/lotus/pull/6766))
|
||||
- easy way to make install app ([filecoin-project/lotus#5183](https://github.com/filecoin-project/lotus/pull/5183))
|
||||
- api: Separate the Net interface from Common ([filecoin-project/lotus#6627](https://github.com/filecoin-project/lotus/pull/6627))
|
||||
- cache loaded block messages ([filecoin-project/lotus#6760](https://github.com/filecoin-project/lotus/pull/6760))
|
||||
- fix: on randomness change, use new rand ([filecoin-project/lotus#6805](https://github.com/filecoin-project/lotus/pull/6805))
|
||||
- Splitstore: add retention policy option for keeping messages in the hotstore ([filecoin-project/lotus#6775](https://github.com/filecoin-project/lotus/pull/6775))
|
||||
- Introduce the LOTUS_CHAIN_BADGERSTORE_DISABLE_FSYNC envvar ([filecoin-project/lotus#6817](https://github.com/filecoin-project/lotus/pull/6817))
|
||||
- add StateReadState to gateway api ([filecoin-project/lotus#6818](https://github.com/filecoin-project/lotus/pull/6818))
|
||||
- add SealProof in SectorBuilder ([filecoin-project/lotus#6815](https://github.com/filecoin-project/lotus/pull/6815))
|
||||
- release -> master ([filecoin-project/lotus#6828](https://github.com/filecoin-project/lotus/pull/6828))
|
||||
- sealing: Handle preCommitParams errors more correctly ([filecoin-project/lotus#6763](https://github.com/filecoin-project/lotus/pull/6763))
|
||||
- fix: always check if StateSearchMessage returns nil ([filecoin-project/lotus#6802](https://github.com/filecoin-project/lotus/pull/6802))
|
||||
- ClientFindData: always fetch peer id from chain ([filecoin-project/lotus#6807](https://github.com/filecoin-project/lotus/pull/6807))
|
||||
- test: fix flaky window post tests ([filecoin-project/lotus#6804](https://github.com/filecoin-project/lotus/pull/6804))
|
||||
- ([filecoin-project/lotus#6800](https://github.com/filecoin-project/lotus/pull/6800))
|
||||
- fixes #6786 segfault ([filecoin-project/lotus#6787](https://github.com/filecoin-project/lotus/pull/6787))
|
||||
- Splitstore: add support for protecting out of chain references in the blockstore ([filecoin-project/lotus#6777](https://github.com/filecoin-project/lotus/pull/6777))
|
||||
- Resurrect CODEOWNERS, but for maintainers group ([filecoin-project/lotus#6773](https://github.com/filecoin-project/lotus/pull/6773))
|
||||
- update go-libp2p-pubsub to v0.5.0 ([filecoin-project/lotus#6764](https://github.com/filecoin-project/lotus/pull/6764))
|
||||
- Implement exposed splitstore ([filecoin-project/lotus#6762](https://github.com/filecoin-project/lotus/pull/6762))
|
||||
- Add ChainGetMessagesInTipset API ([filecoin-project/lotus#6642](https://github.com/filecoin-project/lotus/pull/6642))
|
||||
- test: handle null blocks in TestForkRefuseCall ([filecoin-project/lotus#6758](https://github.com/filecoin-project/lotus/pull/6758))
|
||||
- Master disclaimer ([filecoin-project/lotus#6757](https://github.com/filecoin-project/lotus/pull/6757))
|
||||
- Splitstore code reorg ([filecoin-project/lotus#6756](https://github.com/filecoin-project/lotus/pull/6756))
|
||||
- Create stale.yml ([filecoin-project/lotus#6747](https://github.com/filecoin-project/lotus/pull/6747))
|
||||
- Splitstore: Some small fixes ([filecoin-project/lotus#6754](https://github.com/filecoin-project/lotus/pull/6754))
|
||||
- ([filecoin-project/lotus#6746](https://github.com/filecoin-project/lotus/pull/6746))
|
||||
- Handle the --color flag via proper global state ([filecoin-project/lotus#6743](https://github.com/filecoin-project/lotus/pull/6743))
|
||||
- Config for collateral from miner available balance ([filecoin-project/lotus#6629](https://github.com/filecoin-project/lotus/pull/6629))
|
||||
- Support standalone miner-market process ([filecoin-project/lotus#6356](https://github.com/filecoin-project/lotus/pull/6356))
|
||||
- Splitstore Enhanchements ([filecoin-project/lotus#6474](https://github.com/filecoin-project/lotus/pull/6474))
|
||||
- ([filecoin-project/lotus#6739](https://github.com/filecoin-project/lotus/pull/6739))
|
||||
- Add more deal details to lotus-miner info ([filecoin-project/lotus#6708](https://github.com/filecoin-project/lotus/pull/6708))
|
||||
- Release template: Update all testnet infra at once ([filecoin-project/lotus#6710](https://github.com/filecoin-project/lotus/pull/6710))
|
||||
- Fix Lotus shed
|
||||
- Fix bugs in sectors extend --v1-sectors ([filecoin-project/lotus#6066](https://github.com/filecoin-project/lotus/pull/6066))
|
||||
- add election backtest ([filecoin-project/lotus#5950](https://github.com/filecoin-project/lotus/pull/5950))
|
||||
- Envvar to disable slash filter ([filecoin-project/lotus#6620](https://github.com/filecoin-project/lotus/pull/6620))
|
||||
- Release Template: remove binary validation step ([filecoin-project/lotus#6709](https://github.com/filecoin-project/lotus/pull/6709))
|
||||
- Config for deal publishing control addresses ([filecoin-project/lotus#6697](https://github.com/filecoin-project/lotus/pull/6697))
|
||||
- Reset of the interop network ([filecoin-project/lotus#6689](https://github.com/filecoin-project/lotus/pull/6689))
|
||||
- Enable color by default only if os.Stdout is a TTY ([filecoin-project/lotus#6696](https://github.com/filecoin-project/lotus/pull/6696))
|
||||
- Stop outputing ANSI color on non-TTY ([filecoin-project/lotus#6694](https://github.com/filecoin-project/lotus/pull/6694))
|
||||
- add dollar sign ([filecoin-project/lotus#6690](https://github.com/filecoin-project/lotus/pull/6690))
|
||||
- get-actor cli spelling fix ([filecoin-project/lotus#6681](https://github.com/filecoin-project/lotus/pull/6681))
|
||||
- fix "lotus-seed genesis car" error "merkledag: not found" ([filecoin-project/lotus#6688](https://github.com/filecoin-project/lotus/pull/6688))
|
||||
- polish(statetree): accept a context in statetree diff for timeouts ([filecoin-project/lotus#6639](https://github.com/filecoin-project/lotus/pull/6639))
|
||||
- Add helptext to lotus chain export ([filecoin-project/lotus#6672](https://github.com/filecoin-project/lotus/pull/6672))
|
||||
- Get retrieval pricing input should not error out on a deal state fetch ([filecoin-project/lotus#6679](https://github.com/filecoin-project/lotus/pull/6679))
|
||||
- Fix more CID double-encoding as hex ([filecoin-project/lotus#6680](https://github.com/filecoin-project/lotus/pull/6680))
|
||||
- add an incremental nonce itest. ([filecoin-project/lotus#6663](https://github.com/filecoin-project/lotus/pull/6663))
|
||||
- storage: Fix FinalizeSector with sectors in stoage paths ([filecoin-project/lotus#6653](https://github.com/filecoin-project/lotus/pull/6653))
|
||||
- Fix tiny error in check-client-datacap ([filecoin-project/lotus#6664](https://github.com/filecoin-project/lotus/pull/6664))
|
||||
- Fix: precommit_batch method used the wrong cfg.CommitBatchWait ([filecoin-project/lotus#6658](https://github.com/filecoin-project/lotus/pull/6658))
|
||||
- fix ticket expiration check ([filecoin-project/lotus#6635](https://github.com/filecoin-project/lotus/pull/6635))
|
||||
- commit batch: AggregateAboveBaseFee config ([filecoin-project/lotus#6650](https://github.com/filecoin-project/lotus/pull/6650))
|
||||
- commit batch: Initialize the FailedSectors map ([filecoin-project/lotus#6647](https://github.com/filecoin-project/lotus/pull/6647))
|
||||
- Fast-path retry submitting commit aggregate if commit is still valid ([filecoin-project/lotus#6638](https://github.com/filecoin-project/lotus/pull/6638))
|
||||
- remove precommit check in handleCommitFailed ([filecoin-project/lotus#6634](https://github.com/filecoin-project/lotus/pull/6634))
|
||||
- Reuse timers in sealing batch logic ([filecoin-project/lotus#6636](https://github.com/filecoin-project/lotus/pull/6636))
|
||||
- shed tool to estimate aggregate network fees ([filecoin-project/lotus#6631](https://github.com/filecoin-project/lotus/pull/6631))
|
||||
- fix prove commit aggregate send token amount ([filecoin-project/lotus#6625](https://github.com/filecoin-project/lotus/pull/6625))
|
||||
- Update version.go to 1.11.1 ([filecoin-project/lotus#6621](https://github.com/filecoin-project/lotus/pull/6621))
|
||||
- github.com/filecoin-project/go-data-transfer (v1.6.0 -> v1.7.0):
|
||||
- release: v1.7.0
|
||||
- Fire a transfer queued event when a transfer is queued in Graphsync (#221) ([filecoin-project/go-data-transfer#221](https://github.com/filecoin-project/go-data-transfer/pull/221))
|
||||
- feat: pass ChannelID to ValidatePush & ValidatePull (#220) ([filecoin-project/go-data-transfer#220](https://github.com/filecoin-project/go-data-transfer/pull/220))
|
||||
- release: v1.6.1 ([filecoin-project/go-data-transfer#218](https://github.com/filecoin-project/go-data-transfer/pull/218))
|
||||
- Remove CID lists (#217) ([filecoin-project/go-data-transfer#217](https://github.com/filecoin-project/go-data-transfer/pull/217))
|
||||
- Merge v1.6.0 ([filecoin-project/go-data-transfer#214](https://github.com/filecoin-project/go-data-transfer/pull/214))
|
||||
- Remove restart ack timeout (#211) ([filecoin-project/go-data-transfer#211](https://github.com/filecoin-project/go-data-transfer/pull/211))
|
||||
- feat: use different extension names to fit multiple hooks data in same graphsync message (#204) ([filecoin-project/go-data-transfer#204](https://github.com/filecoin-project/go-data-transfer/pull/204))
|
||||
- fix: map race in GS transport (#208) ([filecoin-project/go-data-transfer#208](https://github.com/filecoin-project/go-data-transfer/pull/208))
|
||||
- refactor: simplify graphsync transport (#203) ([filecoin-project/go-data-transfer#203](https://github.com/filecoin-project/go-data-transfer/pull/203))
|
||||
- release: v1.5.0 (#200) ([filecoin-project/go-data-transfer#200](https://github.com/filecoin-project/go-data-transfer/pull/200))
|
||||
- github.com/filecoin-project/go-fil-markets (v1.5.0 -> v1.6.0):
|
||||
- release: v1.6.0
|
||||
- support padding out smaller files (#536) ([filecoin-project/go-fil-markets#536](https://github.com/filecoin-project/go-fil-markets/pull/536))
|
||||
- On overloaded CI 10 seconds just isn't enough (#587) ([filecoin-project/go-fil-markets#587](https://github.com/filecoin-project/go-fil-markets/pull/587))
|
||||
- Do not hex-encode CIDs in logs (#561) ([filecoin-project/go-fil-markets#561](https://github.com/filecoin-project/go-fil-markets/pull/561))
|
||||
- remove wrong peer check in push deal validation (#585) ([filecoin-project/go-fil-markets#585](https://github.com/filecoin-project/go-fil-markets/pull/585))
|
||||
- fix: circleci docs-gen task (#574) ([filecoin-project/go-fil-markets#574](https://github.com/filecoin-project/go-fil-markets/pull/574))
|
||||
- Storage market request queued event and validation interface changes (#555) ([filecoin-project/go-fil-markets#555](https://github.com/filecoin-project/go-fil-markets/pull/555))
|
||||
- build(deps): bump ws from 6.2.1 to 6.2.2 (#554) ([filecoin-project/go-fil-markets#554](https://github.com/filecoin-project/go-fil-markets/pull/554))
|
||||
- release: v1.5.0 ([filecoin-project/go-fil-markets#553](https://github.com/filecoin-project/go-fil-markets/pull/553))
|
||||
- github.com/filecoin-project/go-padreader (v0.0.0-20200903213702-ed5fae088b20 -> v0.0.0-20210723183308-812a16dc01b1):
|
||||
- New method to pad harder (#6) ([filecoin-project/go-padreader#6](https://github.com/filecoin-project/go-padreader/pull/6))
|
||||
- Create SECURITY.md (#5) ([filecoin-project/go-padreader#5](https://github.com/filecoin-project/go-padreader/pull/5))
|
||||
- github.com/filecoin-project/go-state-types (v0.1.1-0.20210506134452-99b279731c48 -> v0.1.1-0.20210722133031-ad9bfe54c124):
|
||||
- Add version 6.5 (#30) ([filecoin-project/go-state-types#30](https://github.com/filecoin-project/go-state-types/pull/30))
|
||||
- rename file
|
||||
- github.com/filecoin-project/specs-actors/v5 (v5.0.1 -> v5.0.3):
|
||||
- Adjust code for subtle change in go-multihash 0.0.15 (#1463) ([filecoin-project/specs-actors#1463](https://github.com/filecoin-project/specs-actors/pull/1463))
|
||||
- Bump go state types (#1464) ([filecoin-project/specs-actors#1464](https://github.com/filecoin-project/specs-actors/pull/1464))
|
||||
- Create CODEOWNERS (#1465) ([filecoin-project/specs-actors#1465](https://github.com/filecoin-project/specs-actors/pull/1465))
|
||||
- Test deterministic offset (#1462) ([filecoin-project/specs-actors#1462](https://github.com/filecoin-project/specs-actors/pull/1462))
|
||||
|
||||
Contributors
|
||||
|
||||
| Contributor | Commits | Lines ± | Files Changed |
|
||||
|-------------|---------|---------|---------------|
|
||||
| vyzo | 295 | +8700/-5936 | 397 |
|
||||
| Anton Evangelatov | 94 | +4680/-2965 | 277 |
|
||||
| Łukasz Magiera | 37 | +3851/-1611 | 146 |
|
||||
| Mike Greenberg | 1 | +2310/-578 | 8 |
|
||||
| dirkmc | 7 | +1154/-726 | 29 |
|
||||
| Jennifer Wang | 9 | +485/-341 | 26 |
|
||||
| Peter Rabbitson | 18 | +469/-273 | 64 |
|
||||
| Cory Schwartz | 5 | +576/-135 | 14 |
|
||||
| hunjixin | 7 | +404/-82 | 19 |
|
||||
| ZenGround0 | 17 | +284/-135 | 44 |
|
||||
| Dirk McCormick | 17 | +348/-47 | 17 |
|
||||
| Raúl Kripalani | 18 | +254/-97 | 62 |
|
||||
| tchardin | 1 | +261/-33 | 4 |
|
||||
| Jakub Sztandera | 4 | +254/-16 | 4 |
|
||||
| Aarsh Shah | 2 | +196/-40 | 28 |
|
||||
| whyrusleeping | 3 | +150/-9 | 8 |
|
||||
| Whyrusleeping | 2 | +87/-66 | 10 |
|
||||
| Aayush Rajasekaran | 10 | +81/-53 | 13 |
|
||||
| zgfzgf | 2 | +104/-4 | 2 |
|
||||
| aarshkshah1992 | 4 | +73/-7 | 6 |
|
||||
| llifezou | 4 | +59/-20 | 4 |
|
||||
| Steven Allen | 7 | +47/-17 | 9 |
|
||||
| johnli-helloworld | 3 | +46/-15 | 5 |
|
||||
| frrist | 1 | +28/-23 | 2 |
|
||||
| Jennifer | 4 | +31/-2 | 4 |
|
||||
| wangchao | 1 | +1/-27 | 1 |
|
||||
| Jiaying Wang | 2 | +7/-21 | 2 |
|
||||
| hannahhoward | 3 | +21/-2 | 3 |
|
||||
| chadwick2143 | 1 | +15/-1 | 1 |
|
||||
| Jerry | 2 | +9/-4 | 2 |
|
||||
| Steve Loeppky | 2 | +12/-0 | 2 |
|
||||
| David Dias | 1 | +9/-0 | 1 |
|
||||
| dependabot[bot] | 1 | +3/-3 | 1 |
|
||||
| zhoutian527 | 1 | +2/-2 | 1 |
|
||||
| xloem | 1 | +4/-0 | 1 |
|
||||
| Travis Person | 2 | +2/-2 | 3 |
|
||||
| Liviu Damian | 2 | +2/-2 | 2 |
|
||||
| Jim Pick | 2 | +2/-2 | 2 |
|
||||
| Frank | 1 | +3/-0 | 1 |
|
||||
| turuslan | 1 | +1/-1 | 1 |
|
||||
| Kirk Baird | 1 | +0/-0 | 1 |
|
||||
|
||||
|
||||
|
||||
# 1.11.0 / 2021-07-22
|
||||
|
||||
This is a **highly recommended** release of Lotus that have many bug fixes, improvements and new features.
|
||||
@@ -434,13 +253,6 @@ Contributors
|
||||
| @zhoutian527 | 1 | +2/-2 | 1 |
|
||||
| @ribasushi| 1 | +1/-1 | 1 |
|
||||
|
||||
|
||||
<<<<<<< HEAD
|
||||
||||||| merged common ancestors
|
||||
>>>>>>>>> Temporary merge branch 2
|
||||
=======
|
||||
>>>>>>> releases
|
||||
>>>>>>> releases
|
||||
# 1.10.0 / 2021-06-23
|
||||
|
||||
This is a mandatory release of Lotus that introduces Filecoin network v13, codenamed the HyperDrive upgrade. The
|
||||
|
||||
+3
-80
@@ -13,13 +13,12 @@ import (
|
||||
|
||||
"github.com/filecoin-project/go-address"
|
||||
datatransfer "github.com/filecoin-project/go-data-transfer"
|
||||
"github.com/filecoin-project/go-state-types/abi"
|
||||
"github.com/filecoin-project/specs-actors/v2/actors/builtin/market"
|
||||
"github.com/filecoin-project/specs-storage/storage"
|
||||
|
||||
"github.com/filecoin-project/go-fil-markets/piecestore"
|
||||
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
|
||||
"github.com/filecoin-project/go-fil-markets/storagemarket"
|
||||
"github.com/filecoin-project/go-state-types/abi"
|
||||
"github.com/filecoin-project/specs-actors/v2/actors/builtin/market"
|
||||
"github.com/filecoin-project/specs-storage/storage"
|
||||
|
||||
"github.com/filecoin-project/lotus/chain/types"
|
||||
"github.com/filecoin-project/lotus/extern/sector-storage/fsutil"
|
||||
@@ -167,52 +166,6 @@ type StorageMiner interface {
|
||||
MarketPendingDeals(ctx context.Context) (PendingDealInfo, error) //perm:write
|
||||
MarketPublishPendingDeals(ctx context.Context) error //perm:admin
|
||||
|
||||
// DagstoreListShards returns information about all shards known to the
|
||||
// DAG store. Only available on nodes running the markets subsystem.
|
||||
DagstoreListShards(ctx context.Context) ([]DagstoreShardInfo, error) //perm:read
|
||||
|
||||
// DagstoreInitializeShard initializes an uninitialized shard.
|
||||
//
|
||||
// Initialization consists of fetching the shard's data (deal payload) from
|
||||
// the storage subsystem, generating an index, and persisting the index
|
||||
// to facilitate later retrievals, and/or to publish to external sources.
|
||||
//
|
||||
// This operation is intended to complement the initial migration. The
|
||||
// migration registers a shard for every unique piece CID, with lazy
|
||||
// initialization. Thus, shards are not initialized immediately to avoid
|
||||
// IO activity competing with proving. Instead, shard are initialized
|
||||
// when first accessed. This method forces the initialization of a shard by
|
||||
// accessing it and immediately releasing it. This is useful to warm up the
|
||||
// cache to facilitate subsequent retrievals, and to generate the indexes
|
||||
// to publish them externally.
|
||||
//
|
||||
// This operation fails if the shard is not in ShardStateNew state.
|
||||
// It blocks until initialization finishes.
|
||||
DagstoreInitializeShard(ctx context.Context, key string) error //perm:write
|
||||
|
||||
// DagstoreRecoverShard attempts to recover a failed shard.
|
||||
//
|
||||
// This operation fails if the shard is not in ShardStateErrored state.
|
||||
// It blocks until recovery finishes. If recovery failed, it returns the
|
||||
// error.
|
||||
DagstoreRecoverShard(ctx context.Context, key string) error //perm:write
|
||||
|
||||
// DagstoreInitializeAll initializes all uninitialized shards in bulk,
|
||||
// according to the policy passed in the parameters.
|
||||
//
|
||||
// It is recommended to set a maximum concurrency to avoid extreme
|
||||
// IO pressure if the storage subsystem has a large amount of deals.
|
||||
//
|
||||
// It returns a stream of events to report progress.
|
||||
DagstoreInitializeAll(ctx context.Context, params DagstoreInitializeAllParams) (<-chan DagstoreInitializeAllEvent, error) //perm:write
|
||||
|
||||
// DagstoreGC runs garbage collection on the DAG store.
|
||||
DagstoreGC(ctx context.Context) ([]DagstoreShardResult, error) //perm:admin
|
||||
|
||||
// RuntimeSubsystems returns the subsystems that are enabled
|
||||
// in this instance.
|
||||
RuntimeSubsystems(ctx context.Context) (MinerSubsystems, error) //perm:read
|
||||
|
||||
DealsImportData(ctx context.Context, dealPropCid cid.Cid, file string) error //perm:admin
|
||||
DealsList(ctx context.Context) ([]MarketDeal, error) //perm:admin
|
||||
DealsConsiderOnlineStorageDeals(context.Context) (bool, error) //perm:admin
|
||||
@@ -379,33 +332,3 @@ type DealSchedule struct {
|
||||
StartEpoch abi.ChainEpoch
|
||||
EndEpoch abi.ChainEpoch
|
||||
}
|
||||
|
||||
// DagstoreShardInfo is the serialized form of dagstore.DagstoreShardInfo that
|
||||
// we expose through JSON-RPC to avoid clients having to depend on the
|
||||
// dagstore lib.
|
||||
type DagstoreShardInfo struct {
|
||||
Key string
|
||||
State string
|
||||
Error string
|
||||
}
|
||||
|
||||
// DagstoreShardResult enumerates results per shard.
|
||||
type DagstoreShardResult struct {
|
||||
Key string
|
||||
Success bool
|
||||
Error string
|
||||
}
|
||||
|
||||
type DagstoreInitializeAllParams struct {
|
||||
MaxConcurrency int
|
||||
}
|
||||
|
||||
// DagstoreInitializeAllEvent represents an initialization event.
|
||||
type DagstoreInitializeAllEvent struct {
|
||||
Key string
|
||||
Event string // "start", "end"
|
||||
Success bool
|
||||
Error string
|
||||
Total int
|
||||
Current int
|
||||
}
|
||||
|
||||
+1
-1
@@ -76,7 +76,7 @@ func TestReturnTypes(t *testing.T) {
|
||||
seen[typ] = struct{}{}
|
||||
|
||||
if typ.Kind() == reflect.Interface && typ != bareIface && !typ.Implements(jmarsh) {
|
||||
t.Error("methods can't return interfaces or struct types not implementing json.Marshaller", m.Name)
|
||||
t.Error("methods can't return interfaces", m.Name)
|
||||
}
|
||||
|
||||
switch typ.Kind() {
|
||||
|
||||
+8
-25
@@ -14,18 +14,17 @@ import (
|
||||
"github.com/filecoin-project/go-address"
|
||||
"github.com/filecoin-project/go-bitfield"
|
||||
"github.com/filecoin-project/go-multistore"
|
||||
"github.com/filecoin-project/lotus/node/repo/importmgr"
|
||||
"github.com/google/uuid"
|
||||
"github.com/ipfs/go-cid"
|
||||
"github.com/ipfs/go-filestore"
|
||||
"github.com/libp2p/go-libp2p-core/metrics"
|
||||
metrics "github.com/libp2p/go-libp2p-core/metrics"
|
||||
"github.com/libp2p/go-libp2p-core/network"
|
||||
"github.com/libp2p/go-libp2p-core/peer"
|
||||
"github.com/libp2p/go-libp2p-core/protocol"
|
||||
protocol "github.com/libp2p/go-libp2p-core/protocol"
|
||||
pubsub "github.com/libp2p/go-libp2p-pubsub"
|
||||
"github.com/multiformats/go-multiaddr"
|
||||
|
||||
"github.com/filecoin-project/lotus/node/repo/importmgr"
|
||||
|
||||
datatransfer "github.com/filecoin-project/go-data-transfer"
|
||||
filestore2 "github.com/filecoin-project/go-fil-markets/filestore"
|
||||
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
|
||||
@@ -48,12 +47,11 @@ import (
|
||||
)
|
||||
|
||||
var ExampleValues = map[reflect.Type]interface{}{
|
||||
reflect.TypeOf(api.MinerSubsystem(0)): api.MinerSubsystem(1),
|
||||
reflect.TypeOf(auth.Permission("")): auth.Permission("write"),
|
||||
reflect.TypeOf(""): "string value",
|
||||
reflect.TypeOf(uint64(42)): uint64(42),
|
||||
reflect.TypeOf(byte(7)): byte(7),
|
||||
reflect.TypeOf([]byte{}): []byte("byte array"),
|
||||
reflect.TypeOf(auth.Permission("")): auth.Permission("write"),
|
||||
reflect.TypeOf(""): "string value",
|
||||
reflect.TypeOf(uint64(42)): uint64(42),
|
||||
reflect.TypeOf(byte(7)): byte(7),
|
||||
reflect.TypeOf([]byte{}): []byte("byte array"),
|
||||
}
|
||||
|
||||
func addExample(v interface{}) {
|
||||
@@ -270,21 +268,6 @@ func init() {
|
||||
|
||||
addExample(api.CheckStatusCode(0))
|
||||
addExample(map[string]interface{}{"abc": 123})
|
||||
addExample(api.MinerSubsystems{
|
||||
api.SubsystemMining,
|
||||
api.SubsystemSealing,
|
||||
api.SubsystemSectorStorage,
|
||||
api.SubsystemMarkets,
|
||||
})
|
||||
addExample(api.DagstoreShardResult{
|
||||
Key: "baga6ea4seaqecmtz7iak33dsfshi627abz4i4665dfuzr3qfs4bmad6dx3iigdq",
|
||||
Error: "<error>",
|
||||
})
|
||||
addExample(api.DagstoreShardInfo{
|
||||
Key: "baga6ea4seaqecmtz7iak33dsfshi627abz4i4665dfuzr3qfs4bmad6dx3iigdq",
|
||||
State: "ShardStateAvailable",
|
||||
Error: "<error>",
|
||||
})
|
||||
}
|
||||
|
||||
func GetAPIType(name, pkg string) (i interface{}, t reflect.Type, permStruct []reflect.Type) {
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
)
|
||||
|
||||
// MinerSubsystem represents a miner subsystem. Int and string values are not
|
||||
// guaranteed to be stable over time is not
|
||||
// guaranteed to be stable over time.
|
||||
type MinerSubsystem int
|
||||
|
||||
const (
|
||||
// SubsystemUnknown is a placeholder for the zero value. It should never
|
||||
// be used.
|
||||
SubsystemUnknown MinerSubsystem = iota
|
||||
// SubsystemMarkets signifies the storage and retrieval
|
||||
// deal-making subsystem.
|
||||
SubsystemMarkets
|
||||
// SubsystemMining signifies the mining subsystem.
|
||||
SubsystemMining
|
||||
// SubsystemSealing signifies the sealing subsystem.
|
||||
SubsystemSealing
|
||||
// SubsystemSectorStorage signifies the sector storage subsystem.
|
||||
SubsystemSectorStorage
|
||||
)
|
||||
|
||||
var MinerSubsystemToString = map[MinerSubsystem]string{
|
||||
SubsystemUnknown: "Unknown",
|
||||
SubsystemMarkets: "Markets",
|
||||
SubsystemMining: "Mining",
|
||||
SubsystemSealing: "Sealing",
|
||||
SubsystemSectorStorage: "SectorStorage",
|
||||
}
|
||||
|
||||
var MinerSubsystemToID = map[string]MinerSubsystem{
|
||||
"Unknown": SubsystemUnknown,
|
||||
"Markets": SubsystemMarkets,
|
||||
"Mining": SubsystemMining,
|
||||
"Sealing": SubsystemSealing,
|
||||
"SectorStorage": SubsystemSectorStorage,
|
||||
}
|
||||
|
||||
func (ms MinerSubsystem) MarshalJSON() ([]byte, error) {
|
||||
return json.Marshal(MinerSubsystemToString[ms])
|
||||
}
|
||||
|
||||
func (ms *MinerSubsystem) UnmarshalJSON(b []byte) error {
|
||||
var j string
|
||||
err := json.Unmarshal(b, &j)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s, ok := MinerSubsystemToID[j]
|
||||
if !ok {
|
||||
*ms = SubsystemUnknown
|
||||
} else {
|
||||
*ms = s
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type MinerSubsystems []MinerSubsystem
|
||||
|
||||
func (ms MinerSubsystems) Has(entry MinerSubsystem) bool {
|
||||
for _, v := range ms {
|
||||
if v == entry {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (ms MinerSubsystem) String() string {
|
||||
s, ok := MinerSubsystemToString[ms]
|
||||
if !ok {
|
||||
return MinerSubsystemToString[SubsystemUnknown]
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -603,16 +603,6 @@ type StorageMinerStruct struct {
|
||||
|
||||
CreateBackup func(p0 context.Context, p1 string) error `perm:"admin"`
|
||||
|
||||
DagstoreGC func(p0 context.Context) ([]DagstoreShardResult, error) `perm:"admin"`
|
||||
|
||||
DagstoreInitializeAll func(p0 context.Context, p1 DagstoreInitializeAllParams) (<-chan DagstoreInitializeAllEvent, error) `perm:"write"`
|
||||
|
||||
DagstoreInitializeShard func(p0 context.Context, p1 string) error `perm:"write"`
|
||||
|
||||
DagstoreListShards func(p0 context.Context) ([]DagstoreShardInfo, error) `perm:"read"`
|
||||
|
||||
DagstoreRecoverShard func(p0 context.Context, p1 string) error `perm:"write"`
|
||||
|
||||
DealsConsiderOfflineRetrievalDeals func(p0 context.Context) (bool, error) `perm:"admin"`
|
||||
|
||||
DealsConsiderOfflineStorageDeals func(p0 context.Context) (bool, error) `perm:"admin"`
|
||||
@@ -709,8 +699,6 @@ type StorageMinerStruct struct {
|
||||
|
||||
ReturnUnsealPiece func(p0 context.Context, p1 storiface.CallID, p2 *storiface.CallError) error `perm:"admin"`
|
||||
|
||||
RuntimeSubsystems func(p0 context.Context) (MinerSubsystems, error) `perm:"read"`
|
||||
|
||||
SealingAbort func(p0 context.Context, p1 storiface.CallID) error `perm:"admin"`
|
||||
|
||||
SealingSchedDiag func(p0 context.Context, p1 bool) (interface{}, error) `perm:"admin"`
|
||||
@@ -3579,61 +3567,6 @@ func (s *StorageMinerStub) CreateBackup(p0 context.Context, p1 string) error {
|
||||
return ErrNotSupported
|
||||
}
|
||||
|
||||
func (s *StorageMinerStruct) DagstoreGC(p0 context.Context) ([]DagstoreShardResult, error) {
|
||||
if s.Internal.DagstoreGC == nil {
|
||||
return *new([]DagstoreShardResult), ErrNotSupported
|
||||
}
|
||||
return s.Internal.DagstoreGC(p0)
|
||||
}
|
||||
|
||||
func (s *StorageMinerStub) DagstoreGC(p0 context.Context) ([]DagstoreShardResult, error) {
|
||||
return *new([]DagstoreShardResult), ErrNotSupported
|
||||
}
|
||||
|
||||
func (s *StorageMinerStruct) DagstoreInitializeAll(p0 context.Context, p1 DagstoreInitializeAllParams) (<-chan DagstoreInitializeAllEvent, error) {
|
||||
if s.Internal.DagstoreInitializeAll == nil {
|
||||
return nil, ErrNotSupported
|
||||
}
|
||||
return s.Internal.DagstoreInitializeAll(p0, p1)
|
||||
}
|
||||
|
||||
func (s *StorageMinerStub) DagstoreInitializeAll(p0 context.Context, p1 DagstoreInitializeAllParams) (<-chan DagstoreInitializeAllEvent, error) {
|
||||
return nil, ErrNotSupported
|
||||
}
|
||||
|
||||
func (s *StorageMinerStruct) DagstoreInitializeShard(p0 context.Context, p1 string) error {
|
||||
if s.Internal.DagstoreInitializeShard == nil {
|
||||
return ErrNotSupported
|
||||
}
|
||||
return s.Internal.DagstoreInitializeShard(p0, p1)
|
||||
}
|
||||
|
||||
func (s *StorageMinerStub) DagstoreInitializeShard(p0 context.Context, p1 string) error {
|
||||
return ErrNotSupported
|
||||
}
|
||||
|
||||
func (s *StorageMinerStruct) DagstoreListShards(p0 context.Context) ([]DagstoreShardInfo, error) {
|
||||
if s.Internal.DagstoreListShards == nil {
|
||||
return *new([]DagstoreShardInfo), ErrNotSupported
|
||||
}
|
||||
return s.Internal.DagstoreListShards(p0)
|
||||
}
|
||||
|
||||
func (s *StorageMinerStub) DagstoreListShards(p0 context.Context) ([]DagstoreShardInfo, error) {
|
||||
return *new([]DagstoreShardInfo), ErrNotSupported
|
||||
}
|
||||
|
||||
func (s *StorageMinerStruct) DagstoreRecoverShard(p0 context.Context, p1 string) error {
|
||||
if s.Internal.DagstoreRecoverShard == nil {
|
||||
return ErrNotSupported
|
||||
}
|
||||
return s.Internal.DagstoreRecoverShard(p0, p1)
|
||||
}
|
||||
|
||||
func (s *StorageMinerStub) DagstoreRecoverShard(p0 context.Context, p1 string) error {
|
||||
return ErrNotSupported
|
||||
}
|
||||
|
||||
func (s *StorageMinerStruct) DealsConsiderOfflineRetrievalDeals(p0 context.Context) (bool, error) {
|
||||
if s.Internal.DealsConsiderOfflineRetrievalDeals == nil {
|
||||
return false, ErrNotSupported
|
||||
@@ -4162,17 +4095,6 @@ func (s *StorageMinerStub) ReturnUnsealPiece(p0 context.Context, p1 storiface.Ca
|
||||
return ErrNotSupported
|
||||
}
|
||||
|
||||
func (s *StorageMinerStruct) RuntimeSubsystems(p0 context.Context) (MinerSubsystems, error) {
|
||||
if s.Internal.RuntimeSubsystems == nil {
|
||||
return *new(MinerSubsystems), ErrNotSupported
|
||||
}
|
||||
return s.Internal.RuntimeSubsystems(p0)
|
||||
}
|
||||
|
||||
func (s *StorageMinerStub) RuntimeSubsystems(p0 context.Context) (MinerSubsystems, error) {
|
||||
return *new(MinerSubsystems), ErrNotSupported
|
||||
}
|
||||
|
||||
func (s *StorageMinerStruct) SealingAbort(p0 context.Context, p1 storiface.CallID) error {
|
||||
if s.Internal.SealingAbort == nil {
|
||||
return ErrNotSupported
|
||||
|
||||
@@ -258,16 +258,16 @@ func (b *Blockstore) movingGC() error {
|
||||
b.moveCond.Broadcast()
|
||||
b.moveMx.Unlock()
|
||||
|
||||
var newPath string
|
||||
var path string
|
||||
|
||||
defer func() {
|
||||
b.lockMove()
|
||||
|
||||
dbNext := b.dbNext
|
||||
db2 := b.dbNext
|
||||
b.dbNext = nil
|
||||
|
||||
var state bsMoveState
|
||||
if dbNext != nil {
|
||||
if db2 != nil {
|
||||
state = moveStateCleanup
|
||||
} else {
|
||||
state = moveStateNone
|
||||
@@ -275,13 +275,12 @@ func (b *Blockstore) movingGC() error {
|
||||
|
||||
b.unlockMove(state)
|
||||
|
||||
if dbNext != nil {
|
||||
// the move failed and we have a left-over db; delete it.
|
||||
err := dbNext.Close()
|
||||
if db2 != nil {
|
||||
err := db2.Close()
|
||||
if err != nil {
|
||||
log.Warnf("error closing badger db: %s", err)
|
||||
}
|
||||
b.deleteDB(newPath)
|
||||
b.deleteDB(path)
|
||||
|
||||
b.lockMove()
|
||||
b.unlockMove(moveStateNone)
|
||||
@@ -297,96 +296,68 @@ func (b *Blockstore) movingGC() error {
|
||||
}
|
||||
|
||||
if basePath == linkPath {
|
||||
newPath = basePath
|
||||
path = basePath
|
||||
} else {
|
||||
// we do this dance to create a name adjacent to the current one, while avoiding clown
|
||||
// shoes with multiple moves (i.e. we can't just take the basename of the linkPath, as it
|
||||
// could have been created in a previous move and have the timestamp suffix, which would then
|
||||
// perpetuate itself.
|
||||
name := filepath.Base(basePath)
|
||||
dir := filepath.Dir(linkPath)
|
||||
newPath = filepath.Join(dir, name)
|
||||
path = filepath.Join(dir, name)
|
||||
}
|
||||
newPath = fmt.Sprintf("%s.%d", newPath, time.Now().UnixNano())
|
||||
path = fmt.Sprintf("%s.%d", path, time.Now().UnixNano())
|
||||
|
||||
log.Infof("moving blockstore from %s to %s", b.opts.Dir, newPath)
|
||||
log.Infof("moving blockstore from %s to %s", b.opts.Dir, path)
|
||||
|
||||
opts := b.opts
|
||||
opts.Dir = newPath
|
||||
opts.ValueDir = newPath
|
||||
opts.Dir = path
|
||||
opts.ValueDir = path
|
||||
|
||||
dbNew, err := badger.Open(opts.Options)
|
||||
db2, err := badger.Open(opts.Options)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to open badger blockstore in %s: %w", newPath, err)
|
||||
return fmt.Errorf("failed to open badger blockstore in %s: %w", path, err)
|
||||
}
|
||||
|
||||
b.lockMove()
|
||||
b.dbNext = dbNew
|
||||
b.dbNext = db2
|
||||
b.unlockMove(moveStateMoving)
|
||||
|
||||
log.Info("copying blockstore")
|
||||
err = b.doCopy(b.db, b.dbNext)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error moving badger blockstore to %s: %w", newPath, err)
|
||||
return fmt.Errorf("error moving badger blockstore to %s: %w", path, err)
|
||||
}
|
||||
|
||||
b.lockMove()
|
||||
dbOld := b.db
|
||||
db1 := b.db
|
||||
b.db = b.dbNext
|
||||
b.dbNext = nil
|
||||
b.unlockMove(moveStateCleanup)
|
||||
|
||||
err = dbOld.Close()
|
||||
err = db1.Close()
|
||||
if err != nil {
|
||||
log.Warnf("error closing old badger db: %s", err)
|
||||
}
|
||||
|
||||
// this is the canonical db path; this is where our db lives.
|
||||
dbPath := b.opts.Dir
|
||||
dbpath := b.opts.Dir
|
||||
oldpath := fmt.Sprintf("%s.old.%d", dbpath, time.Now().Unix())
|
||||
|
||||
// we first move the existing db out of the way, and only delete it after we have symlinked the
|
||||
// new db to the canonical path
|
||||
backupPath := fmt.Sprintf("%s.old.%d", dbPath, time.Now().Unix())
|
||||
if err = os.Rename(dbPath, backupPath); err != nil {
|
||||
if err = os.Rename(dbpath, oldpath); err != nil {
|
||||
// this is not catastrophic in the sense that we have not lost any data.
|
||||
// but it is pretty bad, as the db path points to the old db, while we are now using to the new
|
||||
// db; we can't continue and leave a ticking bomb for the next restart.
|
||||
// so a panic is appropriate and user can fix.
|
||||
panic(fmt.Errorf("error renaming old badger db dir from %s to %s: %w; USER ACTION REQUIRED", dbPath, backupPath, err)) //nolint
|
||||
panic(fmt.Errorf("error renaming old badger db dir from %s to %s: %w; USER ACTION REQUIRED", dbpath, oldpath, err)) //nolint
|
||||
}
|
||||
|
||||
if err = symlink(newPath, dbPath); err != nil {
|
||||
if err = os.Symlink(path, dbpath); err != nil {
|
||||
// same here; the db path is pointing to the void. panic and let the user fix.
|
||||
panic(fmt.Errorf("error symlinking new badger db dir from %s to %s: %w; USER ACTION REQUIRED", newPath, dbPath, err)) //nolint
|
||||
panic(fmt.Errorf("error symlinking new badger db dir from %s to %s: %w; USER ACTION REQUIRED", path, dbpath, err)) //nolint
|
||||
}
|
||||
|
||||
// the delete follows symlinks
|
||||
b.deleteDB(backupPath)
|
||||
b.deleteDB(oldpath)
|
||||
|
||||
log.Info("moving blockstore done")
|
||||
return nil
|
||||
}
|
||||
|
||||
// symlink creates a symlink from path to linkTo; the link is relative if the two are
|
||||
// in the same directory
|
||||
func symlink(path, linkTo string) error {
|
||||
resolvedPathDir, err := filepath.EvalSymlinks(filepath.Dir(path))
|
||||
if err != nil {
|
||||
return fmt.Errorf("error resolving links in %s: %w", path, err)
|
||||
}
|
||||
|
||||
resolvedLinkDir, err := filepath.EvalSymlinks(filepath.Dir(linkTo))
|
||||
if err != nil {
|
||||
return fmt.Errorf("error resolving links in %s: %w", linkTo, err)
|
||||
}
|
||||
|
||||
if resolvedPathDir == resolvedLinkDir {
|
||||
path = filepath.Base(path)
|
||||
}
|
||||
|
||||
return os.Symlink(path, linkTo)
|
||||
}
|
||||
|
||||
// doCopy copies a badger blockstore to another, with an optional filter; if the filter
|
||||
// is not nil, then only cids that satisfy the filter will be copied.
|
||||
func (b *Blockstore) doCopy(from, to *badger.DB) error {
|
||||
@@ -419,19 +390,19 @@ func (b *Blockstore) doCopy(from, to *badger.DB) error {
|
||||
|
||||
func (b *Blockstore) deleteDB(path string) {
|
||||
// follow symbolic links, otherwise the data wil be left behind
|
||||
linkPath, err := filepath.EvalSymlinks(path)
|
||||
lpath, err := filepath.EvalSymlinks(path)
|
||||
if err != nil {
|
||||
log.Warnf("error resolving symlinks in %s", path)
|
||||
return
|
||||
}
|
||||
|
||||
log.Infof("removing data directory %s", linkPath)
|
||||
if err := os.RemoveAll(linkPath); err != nil {
|
||||
log.Warnf("error deleting db at %s: %s", linkPath, err)
|
||||
log.Infof("removing data directory %s", lpath)
|
||||
if err := os.RemoveAll(lpath); err != nil {
|
||||
log.Warnf("error deleting db at %s: %s", lpath, err)
|
||||
return
|
||||
}
|
||||
|
||||
if path != linkPath {
|
||||
if path != lpath {
|
||||
log.Infof("removing link %s", path)
|
||||
if err := os.Remove(path); err != nil {
|
||||
log.Warnf("error removing symbolic link %s", err)
|
||||
|
||||
@@ -245,21 +245,6 @@ func testMove(t *testing.T, optsF func(string) Options) {
|
||||
|
||||
checkBlocks()
|
||||
checkPath()
|
||||
|
||||
// reopen the db to make sure our relative link works:
|
||||
err = db.Close()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
db, err = Open(optsF(dbPath))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// db.Close() is already deferred
|
||||
|
||||
checkBlocks()
|
||||
}
|
||||
|
||||
func TestMoveNoPrefix(t *testing.T) {
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
+1
-1
@@ -34,7 +34,7 @@ func buildType() string {
|
||||
}
|
||||
|
||||
// BuildVersion is the local build version, set by build system
|
||||
const BuildVersion = "1.11.1-m1.3.4"
|
||||
const BuildVersion = "1.11.1-dev"
|
||||
|
||||
func UserVersion() string {
|
||||
if os.Getenv("LOTUS_VERSION_IGNORE_COMMIT") == "1" {
|
||||
|
||||
+2
-3
@@ -113,7 +113,7 @@ var AuthApiInfoToken = &cli.Command{
|
||||
|
||||
ti, ok := cctx.App.Metadata["repoType"]
|
||||
if !ok {
|
||||
log.Errorf("unknown repo type, are you sure you want to use GetCommonAPI?")
|
||||
log.Errorf("unknown repo type, are you sure you want to use GetAPI?")
|
||||
ti = repo.FullNode
|
||||
}
|
||||
t, ok := ti.(repo.RepoType)
|
||||
@@ -128,8 +128,7 @@ var AuthApiInfoToken = &cli.Command{
|
||||
|
||||
// TODO: Log in audit log when it is implemented
|
||||
|
||||
currentEnv, _, _ := cliutil.EnvsForAPIInfos(t)
|
||||
fmt.Printf("%s=%s:%s\n", currentEnv, string(token), ainfo.Addr)
|
||||
fmt.Printf("%s=%s:%s\n", cliutil.EnvForRepo(t), string(token), ainfo.Addr)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
+1
-2
@@ -44,7 +44,7 @@ func GetFullNodeServices(ctx *cli.Context) (ServicesAPI, error) {
|
||||
|
||||
var GetAPIInfo = cliutil.GetAPIInfo
|
||||
var GetRawAPI = cliutil.GetRawAPI
|
||||
var GetAPI = cliutil.GetCommonAPI
|
||||
var GetAPI = cliutil.GetAPI
|
||||
|
||||
var DaemonContext = cliutil.DaemonContext
|
||||
var ReqContext = cliutil.ReqContext
|
||||
@@ -54,7 +54,6 @@ var GetFullNodeAPIV1 = cliutil.GetFullNodeAPIV1
|
||||
var GetGatewayAPI = cliutil.GetGatewayAPI
|
||||
|
||||
var GetStorageMinerAPI = cliutil.GetStorageMinerAPI
|
||||
var GetMarketsAPI = cliutil.GetMarketsAPI
|
||||
var GetWorkerAPI = cliutil.GetWorkerAPI
|
||||
|
||||
var CommonCommands = []*cli.Command{
|
||||
|
||||
+63
-127
@@ -2,7 +2,6 @@ package cliutil
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
@@ -28,154 +27,112 @@ const (
|
||||
metadataTraceContext = "traceContext"
|
||||
)
|
||||
|
||||
// flagsForAPI returns flags passed on the command line with the listen address
|
||||
// of the API server (only used by the tests), in the order of precedence they
|
||||
// should be applied for the requested kind of node.
|
||||
func flagsForAPI(t repo.RepoType) []string {
|
||||
// The flag passed on the command line with the listen address of the API
|
||||
// server (only used by the tests)
|
||||
func flagForAPI(t repo.RepoType) string {
|
||||
switch t {
|
||||
case repo.FullNode:
|
||||
return []string{"api-url"}
|
||||
return "api-url"
|
||||
case repo.StorageMiner:
|
||||
return []string{"miner-api-url"}
|
||||
return "miner-api-url"
|
||||
case repo.Worker:
|
||||
return []string{"worker-api-url"}
|
||||
case repo.Markets:
|
||||
// support split markets-miner and monolith deployments.
|
||||
return []string{"markets-api-url", "miner-api-url"}
|
||||
return "worker-api-url"
|
||||
default:
|
||||
panic(fmt.Sprintf("Unknown repo type: %v", t))
|
||||
}
|
||||
}
|
||||
|
||||
func flagsForRepo(t repo.RepoType) []string {
|
||||
func flagForRepo(t repo.RepoType) string {
|
||||
switch t {
|
||||
case repo.FullNode:
|
||||
return []string{"repo"}
|
||||
return "repo"
|
||||
case repo.StorageMiner:
|
||||
return []string{"miner-repo"}
|
||||
return "miner-repo"
|
||||
case repo.Worker:
|
||||
return []string{"worker-repo"}
|
||||
case repo.Markets:
|
||||
// support split markets-miner and monolith deployments.
|
||||
return []string{"markets-repo", "miner-repo"}
|
||||
return "worker-repo"
|
||||
default:
|
||||
panic(fmt.Sprintf("Unknown repo type: %v", t))
|
||||
}
|
||||
}
|
||||
|
||||
// EnvsForAPIInfos returns the environment variables to use in order of precedence
|
||||
// to determine the API endpoint of the specified node type.
|
||||
//
|
||||
// It returns the current variables and deprecated ones separately, so that
|
||||
// the user can log a warning when deprecated ones are found to be in use.
|
||||
func EnvsForAPIInfos(t repo.RepoType) (primary string, fallbacks []string, deprecated []string) {
|
||||
func EnvForRepo(t repo.RepoType) string {
|
||||
switch t {
|
||||
case repo.FullNode:
|
||||
return "FULLNODE_API_INFO", nil, nil
|
||||
return "FULLNODE_API_INFO"
|
||||
case repo.StorageMiner:
|
||||
// TODO remove deprecated deprecation period
|
||||
return "MINER_API_INFO", nil, []string{"STORAGE_API_INFO"}
|
||||
return "MINER_API_INFO"
|
||||
case repo.Worker:
|
||||
return "WORKER_API_INFO", nil, nil
|
||||
case repo.Markets:
|
||||
// support split markets-miner and monolith deployments.
|
||||
return "MARKETS_API_INFO", []string{"MINER_API_INFO"}, nil
|
||||
return "WORKER_API_INFO"
|
||||
default:
|
||||
panic(fmt.Sprintf("Unknown repo type: %v", t))
|
||||
}
|
||||
}
|
||||
|
||||
// TODO remove after deprecation period
|
||||
func envForRepoDeprecation(t repo.RepoType) string {
|
||||
switch t {
|
||||
case repo.FullNode:
|
||||
return "FULLNODE_API_INFO"
|
||||
case repo.StorageMiner:
|
||||
return "STORAGE_API_INFO"
|
||||
case repo.Worker:
|
||||
return "WORKER_API_INFO"
|
||||
default:
|
||||
panic(fmt.Sprintf("Unknown repo type: %v", t))
|
||||
}
|
||||
}
|
||||
|
||||
// GetAPIInfo returns the API endpoint to use for the specified kind of repo.
|
||||
//
|
||||
// The order of precedence is as follows:
|
||||
//
|
||||
// 1. *-api-url command line flags.
|
||||
// 2. *_API_INFO environment variables
|
||||
// 3. deprecated *_API_INFO environment variables
|
||||
// 4. *-repo command line flags.
|
||||
func GetAPIInfo(ctx *cli.Context, t repo.RepoType) (APIInfo, error) {
|
||||
// Check if there was a flag passed with the listen address of the API
|
||||
// server (only used by the tests)
|
||||
apiFlags := flagsForAPI(t)
|
||||
for _, f := range apiFlags {
|
||||
if !ctx.IsSet(f) {
|
||||
continue
|
||||
}
|
||||
strma := ctx.String(f)
|
||||
apiFlag := flagForAPI(t)
|
||||
if ctx.IsSet(apiFlag) {
|
||||
strma := ctx.String(apiFlag)
|
||||
strma = strings.TrimSpace(strma)
|
||||
|
||||
return APIInfo{Addr: strma}, nil
|
||||
}
|
||||
|
||||
//
|
||||
// Note: it is not correct/intuitive to prefer environment variables over
|
||||
// CLI flags (repo flags below).
|
||||
//
|
||||
primaryEnv, fallbacksEnvs, deprecatedEnvs := EnvsForAPIInfos(t)
|
||||
env, ok := os.LookupEnv(primaryEnv)
|
||||
envKey := EnvForRepo(t)
|
||||
env, ok := os.LookupEnv(envKey)
|
||||
if !ok {
|
||||
// TODO remove after deprecation period
|
||||
envKey = envForRepoDeprecation(t)
|
||||
env, ok = os.LookupEnv(envKey)
|
||||
if ok {
|
||||
log.Warnf("Use deprecation env(%s) value, please use env(%s) instead.", envKey, EnvForRepo(t))
|
||||
}
|
||||
}
|
||||
if ok {
|
||||
return ParseApiInfo(env), nil
|
||||
}
|
||||
|
||||
for _, env := range deprecatedEnvs {
|
||||
env, ok := os.LookupEnv(env)
|
||||
if ok {
|
||||
log.Warnf("Using deprecated env(%s) value, please use env(%s) instead.", env, primaryEnv)
|
||||
return ParseApiInfo(env), nil
|
||||
}
|
||||
repoFlag := flagForRepo(t)
|
||||
|
||||
p, err := homedir.Expand(ctx.String(repoFlag))
|
||||
if err != nil {
|
||||
return APIInfo{}, xerrors.Errorf("could not expand home dir (%s): %w", repoFlag, err)
|
||||
}
|
||||
|
||||
repoFlags := flagsForRepo(t)
|
||||
for _, f := range repoFlags {
|
||||
// cannot use ctx.IsSet because it ignores default values
|
||||
path := ctx.String(f)
|
||||
if path == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
p, err := homedir.Expand(path)
|
||||
if err != nil {
|
||||
return APIInfo{}, xerrors.Errorf("could not expand home dir (%s): %w", f, err)
|
||||
}
|
||||
|
||||
r, err := repo.NewFS(p)
|
||||
if err != nil {
|
||||
return APIInfo{}, xerrors.Errorf("could not open repo at path: %s; %w", p, err)
|
||||
}
|
||||
|
||||
exists, err := r.Exists()
|
||||
if err != nil {
|
||||
return APIInfo{}, xerrors.Errorf("repo.Exists returned an error: %w", err)
|
||||
}
|
||||
|
||||
if !exists {
|
||||
return APIInfo{}, errors.New("repo directory does not exist. Make sure your configuration is correct")
|
||||
}
|
||||
|
||||
ma, err := r.APIEndpoint()
|
||||
if err != nil {
|
||||
return APIInfo{}, xerrors.Errorf("could not get api endpoint: %w", err)
|
||||
}
|
||||
|
||||
token, err := r.APIToken()
|
||||
if err != nil {
|
||||
log.Warnf("Couldn't load CLI token, capabilities may be limited: %v", err)
|
||||
}
|
||||
|
||||
return APIInfo{
|
||||
Addr: ma.String(),
|
||||
Token: token,
|
||||
}, nil
|
||||
r, err := repo.NewFS(p)
|
||||
if err != nil {
|
||||
return APIInfo{}, xerrors.Errorf("could not open repo at path: %s; %w", p, err)
|
||||
}
|
||||
|
||||
for _, env := range fallbacksEnvs {
|
||||
env, ok := os.LookupEnv(env)
|
||||
if ok {
|
||||
return ParseApiInfo(env), nil
|
||||
}
|
||||
ma, err := r.APIEndpoint()
|
||||
if err != nil {
|
||||
return APIInfo{}, xerrors.Errorf("could not get api endpoint: %w", err)
|
||||
}
|
||||
|
||||
return APIInfo{}, fmt.Errorf("could not determine API endpoint for node type: %v", t)
|
||||
token, err := r.APIToken()
|
||||
if err != nil {
|
||||
log.Warnf("Couldn't load CLI token, capabilities may be limited: %v", err)
|
||||
}
|
||||
|
||||
return APIInfo{
|
||||
Addr: ma.String(),
|
||||
Token: token,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func GetRawAPI(ctx *cli.Context, t repo.RepoType, version string) (string, http.Header, error) {
|
||||
@@ -196,10 +153,10 @@ func GetRawAPI(ctx *cli.Context, t repo.RepoType, version string) (string, http.
|
||||
return addr, ainfo.AuthHeader(), nil
|
||||
}
|
||||
|
||||
func GetCommonAPI(ctx *cli.Context) (api.CommonNet, jsonrpc.ClientCloser, error) {
|
||||
func GetAPI(ctx *cli.Context) (api.CommonNet, jsonrpc.ClientCloser, error) {
|
||||
ti, ok := ctx.App.Metadata["repoType"]
|
||||
if !ok {
|
||||
log.Errorf("unknown repo type, are you sure you want to use GetCommonAPI?")
|
||||
log.Errorf("unknown repo type, are you sure you want to use GetAPI?")
|
||||
ti = repo.FullNode
|
||||
}
|
||||
t, ok := ti.(repo.RepoType)
|
||||
@@ -317,27 +274,6 @@ func GetWorkerAPI(ctx *cli.Context) (api.Worker, jsonrpc.ClientCloser, error) {
|
||||
return client.NewWorkerRPCV0(ctx.Context, addr, headers)
|
||||
}
|
||||
|
||||
func GetMarketsAPI(ctx *cli.Context) (api.StorageMiner, jsonrpc.ClientCloser, error) {
|
||||
// to support lotus-miner cli tests.
|
||||
if tn, ok := ctx.App.Metadata["testnode-storage"]; ok {
|
||||
return tn.(api.StorageMiner), func() {}, nil
|
||||
}
|
||||
|
||||
addr, headers, err := GetRawAPI(ctx, repo.Markets, "v0")
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
if IsVeryVerbose {
|
||||
_, _ = fmt.Fprintln(ctx.App.Writer, "using markets API v0 endpoint:", addr)
|
||||
}
|
||||
|
||||
// the markets node is a specialised miner's node, supporting only the
|
||||
// markets API, which is a subset of the miner API. All non-markets
|
||||
// operations will error out with "unsupported".
|
||||
return client.NewStorageMinerRPCV0(ctx.Context, addr, headers)
|
||||
}
|
||||
|
||||
func GetGatewayAPI(ctx *cli.Context) (api.Gateway, jsonrpc.ClientCloser, error) {
|
||||
addr, headers, err := GetRawAPI(ctx, repo.FullNode, "v1")
|
||||
if err != nil {
|
||||
|
||||
+1
-1
@@ -22,7 +22,7 @@ var WaitApiCmd = &cli.Command{
|
||||
|
||||
ctx := ReqContext(cctx)
|
||||
|
||||
_, err = api.Version(ctx)
|
||||
_, err = api.ID(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -1,209 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/fatih/color"
|
||||
"github.com/urfave/cli/v2"
|
||||
|
||||
"github.com/filecoin-project/lotus/api"
|
||||
lcli "github.com/filecoin-project/lotus/cli"
|
||||
"github.com/filecoin-project/lotus/lib/tablewriter"
|
||||
)
|
||||
|
||||
var dagstoreCmd = &cli.Command{
|
||||
Name: "dagstore",
|
||||
Usage: "Manage the dagstore on the markets subsystem",
|
||||
Subcommands: []*cli.Command{
|
||||
dagstoreListShardsCmd,
|
||||
dagstoreInitializeShardCmd,
|
||||
dagstoreRecoverShardCmd,
|
||||
dagstoreInitializeAllCmd,
|
||||
dagstoreGcCmd,
|
||||
},
|
||||
}
|
||||
|
||||
var dagstoreListShardsCmd = &cli.Command{
|
||||
Name: "list-shards",
|
||||
Usage: "List all shards known to the dagstore, with their current status",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
ctx := lcli.ReqContext(cctx)
|
||||
|
||||
shards, err := marketsApi.DagstoreListShards(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(shards) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
tw := tablewriter.New(
|
||||
tablewriter.Col("Key"),
|
||||
tablewriter.Col("State"),
|
||||
tablewriter.Col("Error"),
|
||||
)
|
||||
|
||||
colors := map[string]color.Attribute{
|
||||
"ShardStateAvailable": color.FgGreen,
|
||||
"ShardStateServing": color.FgBlue,
|
||||
"ShardStateErrored": color.FgRed,
|
||||
"ShardStateNew": color.FgYellow,
|
||||
}
|
||||
|
||||
for _, s := range shards {
|
||||
m := map[string]interface{}{
|
||||
"Key": s.Key,
|
||||
"State": func() string {
|
||||
if c, ok := colors[s.State]; ok {
|
||||
return color.New(c).Sprint(s.State)
|
||||
}
|
||||
return s.State
|
||||
}(),
|
||||
"Error": s.Error,
|
||||
}
|
||||
tw.Write(m)
|
||||
}
|
||||
|
||||
return tw.Flush(os.Stdout)
|
||||
},
|
||||
}
|
||||
|
||||
var dagstoreInitializeShardCmd = &cli.Command{
|
||||
Name: "initialize-shard",
|
||||
ArgsUsage: "[key]",
|
||||
Usage: "Initialize the specified shard",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
if cctx.NArg() != 1 {
|
||||
return fmt.Errorf("must provide a single shard key")
|
||||
}
|
||||
|
||||
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
ctx := lcli.ReqContext(cctx)
|
||||
|
||||
return marketsApi.DagstoreInitializeShard(ctx, cctx.Args().First())
|
||||
},
|
||||
}
|
||||
|
||||
var dagstoreRecoverShardCmd = &cli.Command{
|
||||
Name: "recover-shard",
|
||||
ArgsUsage: "[key]",
|
||||
Usage: "Attempt to recover a shard in errored state",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
if cctx.NArg() != 1 {
|
||||
return fmt.Errorf("must provide a single shard key")
|
||||
}
|
||||
|
||||
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
ctx := lcli.ReqContext(cctx)
|
||||
|
||||
return marketsApi.DagstoreRecoverShard(ctx, cctx.Args().First())
|
||||
},
|
||||
}
|
||||
|
||||
var dagstoreInitializeAllCmd = &cli.Command{
|
||||
Name: "initialize-all",
|
||||
Usage: "Initialize all uninitialized shards, streaming results as they're produced",
|
||||
Flags: []cli.Flag{
|
||||
&cli.UintFlag{
|
||||
Name: "concurrency",
|
||||
Usage: "maximum shards to initialize concurrently at a time; use 0 for unlimited",
|
||||
Required: true,
|
||||
},
|
||||
},
|
||||
Action: func(cctx *cli.Context) error {
|
||||
concurrency := cctx.Uint("concurrency")
|
||||
|
||||
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
ctx := lcli.ReqContext(cctx)
|
||||
|
||||
params := api.DagstoreInitializeAllParams{
|
||||
MaxConcurrency: int(concurrency),
|
||||
}
|
||||
|
||||
ch, err := marketsApi.DagstoreInitializeAll(ctx, params)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for {
|
||||
select {
|
||||
case evt, ok := <-ch:
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
_, _ = fmt.Fprint(os.Stdout, color.New(color.BgHiBlack).Sprintf("(%d/%d)", evt.Current, evt.Total))
|
||||
_, _ = fmt.Fprint(os.Stdout, " ")
|
||||
if evt.Event == "start" {
|
||||
_, _ = fmt.Fprintln(os.Stdout, evt.Key, color.New(color.Reset).Sprint("STARTING"))
|
||||
} else {
|
||||
if evt.Success {
|
||||
_, _ = fmt.Fprintln(os.Stdout, evt.Key, color.New(color.FgGreen).Sprint("SUCCESS"))
|
||||
} else {
|
||||
_, _ = fmt.Fprintln(os.Stdout, evt.Key, color.New(color.FgRed).Sprint("ERROR"), evt.Error)
|
||||
}
|
||||
}
|
||||
|
||||
case <-ctx.Done():
|
||||
return fmt.Errorf("aborted")
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
var dagstoreGcCmd = &cli.Command{
|
||||
Name: "gc",
|
||||
Usage: "Garbage collect the dagstore",
|
||||
|
||||
Action: func(cctx *cli.Context) error {
|
||||
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
ctx := lcli.ReqContext(cctx)
|
||||
|
||||
collected, err := marketsApi.DagstoreGC(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(collected) == 0 {
|
||||
_, _ = fmt.Fprintln(os.Stdout, "no shards collected")
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, e := range collected {
|
||||
if e.Error == "" {
|
||||
_, _ = fmt.Fprintln(os.Stdout, e.Key, color.New(color.FgGreen).Sprint("SUCCESS"))
|
||||
} else {
|
||||
_, _ = fmt.Fprintln(os.Stdout, e.Key, color.New(color.FgRed).Sprint("ERROR"), e.Error)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
},
|
||||
}
|
||||
+72
-113
@@ -21,7 +21,6 @@ import (
|
||||
"github.com/filecoin-project/go-fil-markets/storagemarket"
|
||||
"github.com/filecoin-project/go-state-types/abi"
|
||||
"github.com/filecoin-project/go-state-types/big"
|
||||
"github.com/filecoin-project/lotus/api/v0api"
|
||||
sealing "github.com/filecoin-project/lotus/extern/storage-sealing"
|
||||
|
||||
"github.com/filecoin-project/lotus/api"
|
||||
@@ -50,19 +49,13 @@ var infoCmd = &cli.Command{
|
||||
}
|
||||
|
||||
func infoCmdAct(cctx *cli.Context) error {
|
||||
minerApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
fullapi, acloser, err := lcli.GetFullNodeAPI(cctx)
|
||||
api, acloser, err := lcli.GetFullNodeAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -70,23 +63,9 @@ func infoCmdAct(cctx *cli.Context) error {
|
||||
|
||||
ctx := lcli.ReqContext(cctx)
|
||||
|
||||
subsystems, err := minerApi.RuntimeSubsystems(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Println("Enabled subsystems (from miner API):", subsystems)
|
||||
|
||||
subsystems, err = marketsApi.RuntimeSubsystems(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Println("Enabled subsystems (from markets API):", subsystems)
|
||||
|
||||
fmt.Print("Chain: ")
|
||||
|
||||
head, err := fullapi.ChainHead(ctx)
|
||||
head, err := api.ChainHead(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -116,38 +95,24 @@ func infoCmdAct(cctx *cli.Context) error {
|
||||
|
||||
fmt.Println()
|
||||
|
||||
err = handleMiningInfo(ctx, cctx, fullapi, minerApi)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = handleMarketsInfo(ctx, marketsApi)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func handleMiningInfo(ctx context.Context, cctx *cli.Context, fullapi v0api.FullNode, nodeApi api.StorageMiner) error {
|
||||
maddr, err := getActorAddress(ctx, cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
mact, err := fullapi.StateGetActor(ctx, maddr, types.EmptyTSK)
|
||||
mact, err := api.StateGetActor(ctx, maddr, types.EmptyTSK)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tbs := blockstore.NewTieredBstore(blockstore.NewAPIBlockstore(fullapi), blockstore.NewMemory())
|
||||
tbs := blockstore.NewTieredBstore(blockstore.NewAPIBlockstore(api), blockstore.NewMemory())
|
||||
mas, err := miner.Load(adt.WrapStore(ctx, cbor.NewCborStore(tbs)), mact)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Sector size
|
||||
mi, err := fullapi.StateMinerInfo(ctx, maddr, types.EmptyTSK)
|
||||
mi, err := api.StateMinerInfo(ctx, maddr, types.EmptyTSK)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -155,7 +120,7 @@ func handleMiningInfo(ctx context.Context, cctx *cli.Context, fullapi v0api.Full
|
||||
ssize := types.SizeStr(types.NewInt(uint64(mi.SectorSize)))
|
||||
fmt.Printf("Miner: %s (%s sectors)\n", color.BlueString("%s", maddr), ssize)
|
||||
|
||||
pow, err := fullapi.StateMinerPower(ctx, maddr, types.EmptyTSK)
|
||||
pow, err := api.StateMinerPower(ctx, maddr, types.EmptyTSK)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -177,7 +142,7 @@ func handleMiningInfo(ctx context.Context, cctx *cli.Context, fullapi v0api.Full
|
||||
pow.TotalPower.RawBytePower,
|
||||
),
|
||||
)
|
||||
secCounts, err := fullapi.StateMinerSectorCount(ctx, maddr, types.EmptyTSK)
|
||||
secCounts, err := api.StateMinerSectorCount(ctx, maddr, types.EmptyTSK)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -254,75 +219,6 @@ func handleMiningInfo(ctx context.Context, cctx *cli.Context, fullapi v0api.Full
|
||||
|
||||
fmt.Println()
|
||||
|
||||
spendable := big.Zero()
|
||||
|
||||
// NOTE: there's no need to unlock anything here. Funds only
|
||||
// vest on deadline boundaries, and they're unlocked by cron.
|
||||
lockedFunds, err := mas.LockedFunds()
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting locked funds: %w", err)
|
||||
}
|
||||
availBalance, err := mas.AvailableBalance(mact.Balance)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting available balance: %w", err)
|
||||
}
|
||||
spendable = big.Add(spendable, availBalance)
|
||||
|
||||
fmt.Printf("Miner Balance: %s\n", color.YellowString("%s", types.FIL(mact.Balance).Short()))
|
||||
fmt.Printf(" PreCommit: %s\n", types.FIL(lockedFunds.PreCommitDeposits).Short())
|
||||
fmt.Printf(" Pledge: %s\n", types.FIL(lockedFunds.InitialPledgeRequirement).Short())
|
||||
fmt.Printf(" Vesting: %s\n", types.FIL(lockedFunds.VestingFunds).Short())
|
||||
colorTokenAmount(" Available: %s\n", availBalance)
|
||||
|
||||
mb, err := fullapi.StateMarketBalance(ctx, maddr, types.EmptyTSK)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting market balance: %w", err)
|
||||
}
|
||||
spendable = big.Add(spendable, big.Sub(mb.Escrow, mb.Locked))
|
||||
|
||||
fmt.Printf("Market Balance: %s\n", types.FIL(mb.Escrow).Short())
|
||||
fmt.Printf(" Locked: %s\n", types.FIL(mb.Locked).Short())
|
||||
colorTokenAmount(" Available: %s\n", big.Sub(mb.Escrow, mb.Locked))
|
||||
|
||||
wb, err := fullapi.WalletBalance(ctx, mi.Worker)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting worker balance: %w", err)
|
||||
}
|
||||
spendable = big.Add(spendable, wb)
|
||||
color.Cyan("Worker Balance: %s", types.FIL(wb).Short())
|
||||
if len(mi.ControlAddresses) > 0 {
|
||||
cbsum := big.Zero()
|
||||
for _, ca := range mi.ControlAddresses {
|
||||
b, err := fullapi.WalletBalance(ctx, ca)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting control address balance: %w", err)
|
||||
}
|
||||
cbsum = big.Add(cbsum, b)
|
||||
}
|
||||
spendable = big.Add(spendable, cbsum)
|
||||
|
||||
fmt.Printf(" Control: %s\n", types.FIL(cbsum).Short())
|
||||
}
|
||||
colorTokenAmount("Total Spendable: %s\n", spendable)
|
||||
|
||||
fmt.Println()
|
||||
|
||||
if !cctx.Bool("hide-sectors-info") {
|
||||
fmt.Println("Sectors:")
|
||||
err = sectorsInfo(ctx, nodeApi)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// TODO: grab actr state / info
|
||||
// * Sealed sectors (count / bytes)
|
||||
// * Power
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func handleMarketsInfo(ctx context.Context, nodeApi api.StorageMiner) error {
|
||||
deals, err := nodeApi.MarketListIncompleteDeals(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -386,7 +282,6 @@ func handleMarketsInfo(ctx context.Context, nodeApi api.StorageMiner) error {
|
||||
return sorted[i].status > sorted[j].status
|
||||
})
|
||||
|
||||
fmt.Println()
|
||||
fmt.Printf("Storage Deals: %d, %s\n", total.count, types.SizeStr(types.NewInt(total.bytes)))
|
||||
|
||||
tw := tabwriter.NewWriter(os.Stdout, 1, 1, 1, ' ', 0)
|
||||
@@ -414,6 +309,70 @@ func handleMarketsInfo(ctx context.Context, nodeApi api.StorageMiner) error {
|
||||
|
||||
fmt.Println()
|
||||
|
||||
spendable := big.Zero()
|
||||
|
||||
// NOTE: there's no need to unlock anything here. Funds only
|
||||
// vest on deadline boundaries, and they're unlocked by cron.
|
||||
lockedFunds, err := mas.LockedFunds()
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting locked funds: %w", err)
|
||||
}
|
||||
availBalance, err := mas.AvailableBalance(mact.Balance)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting available balance: %w", err)
|
||||
}
|
||||
spendable = big.Add(spendable, availBalance)
|
||||
|
||||
fmt.Printf("Miner Balance: %s\n", color.YellowString("%s", types.FIL(mact.Balance).Short()))
|
||||
fmt.Printf(" PreCommit: %s\n", types.FIL(lockedFunds.PreCommitDeposits).Short())
|
||||
fmt.Printf(" Pledge: %s\n", types.FIL(lockedFunds.InitialPledgeRequirement).Short())
|
||||
fmt.Printf(" Vesting: %s\n", types.FIL(lockedFunds.VestingFunds).Short())
|
||||
colorTokenAmount(" Available: %s\n", availBalance)
|
||||
|
||||
mb, err := api.StateMarketBalance(ctx, maddr, types.EmptyTSK)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting market balance: %w", err)
|
||||
}
|
||||
spendable = big.Add(spendable, big.Sub(mb.Escrow, mb.Locked))
|
||||
|
||||
fmt.Printf("Market Balance: %s\n", types.FIL(mb.Escrow).Short())
|
||||
fmt.Printf(" Locked: %s\n", types.FIL(mb.Locked).Short())
|
||||
colorTokenAmount(" Available: %s\n", big.Sub(mb.Escrow, mb.Locked))
|
||||
|
||||
wb, err := api.WalletBalance(ctx, mi.Worker)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting worker balance: %w", err)
|
||||
}
|
||||
spendable = big.Add(spendable, wb)
|
||||
color.Cyan("Worker Balance: %s", types.FIL(wb).Short())
|
||||
if len(mi.ControlAddresses) > 0 {
|
||||
cbsum := big.Zero()
|
||||
for _, ca := range mi.ControlAddresses {
|
||||
b, err := api.WalletBalance(ctx, ca)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("getting control address balance: %w", err)
|
||||
}
|
||||
cbsum = big.Add(cbsum, b)
|
||||
}
|
||||
spendable = big.Add(spendable, cbsum)
|
||||
|
||||
fmt.Printf(" Control: %s\n", types.FIL(cbsum).Short())
|
||||
}
|
||||
colorTokenAmount("Total Spendable: %s\n", spendable)
|
||||
|
||||
fmt.Println()
|
||||
|
||||
if !cctx.Bool("hide-sectors-info") {
|
||||
fmt.Println("Sectors:")
|
||||
err = sectorsInfo(ctx, nodeApi)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// TODO: grab actr state / info
|
||||
// * Sealed sectors (count / bytes)
|
||||
// * Power
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
"gopkg.in/cheggaaa/pb.v1"
|
||||
|
||||
"github.com/filecoin-project/go-address"
|
||||
"github.com/filecoin-project/go-paramfetch"
|
||||
paramfetch "github.com/filecoin-project/go-paramfetch"
|
||||
"github.com/filecoin-project/go-state-types/big"
|
||||
|
||||
lapi "github.com/filecoin-project/lotus/api"
|
||||
@@ -72,9 +72,7 @@ var restoreCmd = &cli.Command{
|
||||
}
|
||||
}
|
||||
|
||||
repoPath := cctx.String(FlagMinerRepo)
|
||||
|
||||
if err := restore(ctx, cctx, repoPath, storageCfg, nil, func(api lapi.FullNode, maddr address.Address, peerid peer.ID, mi miner.MinerInfo) error {
|
||||
if err := restore(ctx, cctx, storageCfg, nil, func(api lapi.FullNode, maddr address.Address, peerid peer.ID, mi miner.MinerInfo) error {
|
||||
log.Info("Checking proof parameters")
|
||||
|
||||
if err := paramfetch.GetParams(ctx, build.ParametersJSON(), build.SrsJSON(), uint64(mi.SectorSize)); err != nil {
|
||||
@@ -96,7 +94,7 @@ var restoreCmd = &cli.Command{
|
||||
},
|
||||
}
|
||||
|
||||
func restore(ctx context.Context, cctx *cli.Context, targetPath string, strConfig *stores.StorageConfig, manageConfig func(*config.StorageMiner) error, after func(api lapi.FullNode, addr address.Address, peerid peer.ID, mi miner.MinerInfo) error) error {
|
||||
func restore(ctx context.Context, cctx *cli.Context, strConfig *stores.StorageConfig, manageConfig func(*config.StorageMiner) error, after func(api lapi.FullNode, addr address.Address, peerid peer.ID, mi miner.MinerInfo) error) error {
|
||||
if cctx.Args().Len() != 1 {
|
||||
return xerrors.Errorf("expected 1 argument")
|
||||
}
|
||||
@@ -144,7 +142,8 @@ func restore(ctx context.Context, cctx *cli.Context, targetPath string, strConfi
|
||||
|
||||
log.Info("Checking if repo exists")
|
||||
|
||||
r, err := repo.NewFS(targetPath)
|
||||
repoPath := cctx.String(FlagMinerRepo)
|
||||
r, err := repo.NewFS(repoPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -71,12 +71,7 @@ var serviceCmd = &cli.Command{
|
||||
return xerrors.Errorf("--api-sector-index is required without the sector storage module enabled")
|
||||
}
|
||||
|
||||
repoPath := cctx.String(FlagMarketsRepo)
|
||||
if repoPath == "" {
|
||||
return xerrors.Errorf("please provide Lotus markets repo path via flag %s", FlagMarketsRepo)
|
||||
}
|
||||
|
||||
if err := restore(ctx, cctx, repoPath, &stores.StorageConfig{}, func(cfg *config.StorageMiner) error {
|
||||
if err := restore(ctx, cctx, &stores.StorageConfig{}, func(cfg *config.StorageMiner) error {
|
||||
cfg.Subsystems.EnableMarkets = es.Contains(MarketsService)
|
||||
cfg.Subsystems.EnableMining = false
|
||||
cfg.Subsystems.EnableSealing = false
|
||||
|
||||
+4
-25
@@ -5,15 +5,13 @@ import (
|
||||
"fmt"
|
||||
|
||||
"github.com/fatih/color"
|
||||
cliutil "github.com/filecoin-project/lotus/cli/util"
|
||||
logging "github.com/ipfs/go-log/v2"
|
||||
"github.com/urfave/cli/v2"
|
||||
"go.opencensus.io/trace"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
cliutil "github.com/filecoin-project/lotus/cli/util"
|
||||
|
||||
"github.com/filecoin-project/go-address"
|
||||
|
||||
"github.com/filecoin-project/lotus/api"
|
||||
"github.com/filecoin-project/lotus/build"
|
||||
lcli "github.com/filecoin-project/lotus/cli"
|
||||
@@ -24,10 +22,7 @@ import (
|
||||
|
||||
var log = logging.Logger("main")
|
||||
|
||||
const (
|
||||
FlagMinerRepo = "miner-repo"
|
||||
FlagMarketsRepo = "markets-repo"
|
||||
)
|
||||
const FlagMinerRepo = "miner-repo"
|
||||
|
||||
// TODO remove after deprecation period
|
||||
const FlagMinerRepoDeprecation = "storagerepo"
|
||||
@@ -48,7 +43,6 @@ func main() {
|
||||
lcli.WithCategory("market", storageDealsCmd),
|
||||
lcli.WithCategory("market", retrievalDealsCmd),
|
||||
lcli.WithCategory("market", dataTransfersCmd),
|
||||
lcli.WithCategory("market", dagstoreCmd),
|
||||
lcli.WithCategory("storage", sectorsCmd),
|
||||
lcli.WithCategory("storage", provingCmd),
|
||||
lcli.WithCategory("storage", storageCmd),
|
||||
@@ -112,29 +106,14 @@ func main() {
|
||||
Value: "~/.lotusminer", // TODO: Consider XDG_DATA_HOME
|
||||
Usage: fmt.Sprintf("Specify miner repo path. flag(%s) and env(LOTUS_STORAGE_PATH) are DEPRECATION, will REMOVE SOON", FlagMinerRepoDeprecation),
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: FlagMarketsRepo,
|
||||
EnvVars: []string{"LOTUS_MARKETS_PATH"},
|
||||
Usage: fmt.Sprintf("Markets repo path"),
|
||||
},
|
||||
&cli.BoolFlag{
|
||||
Name: "call-on-markets",
|
||||
Usage: "(experimental; may be removed) call this command against a markets node; use only with common commands like net, auth, pprof, etc. whose target may be ambiguous",
|
||||
},
|
||||
cliutil.FlagVeryVerbose,
|
||||
},
|
||||
|
||||
Commands: append(local, lcli.CommonCommands...),
|
||||
Before: func(c *cli.Context) error {
|
||||
// this command is explicitly called on markets, inform
|
||||
// common commands by overriding the repoType.
|
||||
if c.Bool("call-on-markets") {
|
||||
c.App.Metadata["repoType"] = repo.Markets
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
app.Setup()
|
||||
app.Metadata["repoType"] = repo.StorageMiner
|
||||
|
||||
lcli.RunApp(app)
|
||||
}
|
||||
|
||||
|
||||
+18
-24
@@ -73,7 +73,7 @@ var storageDealSelectionShowCmd = &cli.Command{
|
||||
Name: "list",
|
||||
Usage: "List storage deal proposal selection criteria",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
smapi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -100,7 +100,7 @@ var storageDealSelectionResetCmd = &cli.Command{
|
||||
Name: "reset",
|
||||
Usage: "Reset storage deal proposal selection criteria to default values",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
smapi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -148,7 +148,7 @@ var storageDealSelectionRejectCmd = &cli.Command{
|
||||
},
|
||||
},
|
||||
Action: func(cctx *cli.Context) error {
|
||||
smapi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -215,13 +215,7 @@ var setAskCmd = &cli.Command{
|
||||
Action: func(cctx *cli.Context) error {
|
||||
ctx := lcli.DaemonContext(cctx)
|
||||
|
||||
minerApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
marketsApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -258,12 +252,12 @@ var setAskCmd = &cli.Command{
|
||||
return xerrors.Errorf("cannot parse max-piece-size to quantity of bytes: %w", err)
|
||||
}
|
||||
|
||||
maddr, err := minerApi.ActorAddress(ctx)
|
||||
maddr, err := api.ActorAddress(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
ssize, err := minerApi.ActorSectorSize(ctx, maddr)
|
||||
ssize, err := api.ActorSectorSize(ctx, maddr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -278,7 +272,7 @@ var setAskCmd = &cli.Command{
|
||||
return xerrors.Errorf("max piece size (w/bit-padding) %s cannot exceed miner sector size %s", types.SizeStr(types.NewInt(uint64(max))), types.SizeStr(types.NewInt(uint64(smax))))
|
||||
}
|
||||
|
||||
return marketsApi.MarketSetAsk(ctx, types.BigInt(pri), types.BigInt(vpri), abi.ChainEpoch(qty), abi.PaddedPieceSize(min), abi.PaddedPieceSize(max))
|
||||
return api.MarketSetAsk(ctx, types.BigInt(pri), types.BigInt(vpri), abi.ChainEpoch(qty), abi.PaddedPieceSize(min), abi.PaddedPieceSize(max))
|
||||
},
|
||||
}
|
||||
|
||||
@@ -295,7 +289,7 @@ var getAskCmd = &cli.Command{
|
||||
}
|
||||
defer closer()
|
||||
|
||||
smapi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -358,7 +352,7 @@ var dealsImportDataCmd = &cli.Command{
|
||||
Usage: "Manually import data for a deal",
|
||||
ArgsUsage: "<proposal CID> <file>",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -396,7 +390,7 @@ var dealsListCmd = &cli.Command{
|
||||
},
|
||||
},
|
||||
Action: func(cctx *cli.Context) error {
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -500,7 +494,7 @@ var getBlocklistCmd = &cli.Command{
|
||||
&CidBaseFlag,
|
||||
},
|
||||
Action: func(cctx *cli.Context) error {
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -530,7 +524,7 @@ var setBlocklistCmd = &cli.Command{
|
||||
ArgsUsage: "[<path-of-file-containing-newline-delimited-piece-CIDs> (optional, will read from stdin if omitted)]",
|
||||
Flags: []cli.Flag{},
|
||||
Action: func(cctx *cli.Context) error {
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -576,7 +570,7 @@ var resetBlocklistCmd = &cli.Command{
|
||||
Usage: "Remove all entries from the miner's piece CID blocklist",
|
||||
Flags: []cli.Flag{},
|
||||
Action: func(cctx *cli.Context) error {
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -591,7 +585,7 @@ var setSealDurationCmd = &cli.Command{
|
||||
Usage: "Set the expected time, in minutes, that you expect sealing sectors to take. Deals that start before this duration will be rejected.",
|
||||
ArgsUsage: "<minutes>",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -640,7 +634,7 @@ var marketRestartTransfer = &cli.Command{
|
||||
if !cctx.Args().Present() {
|
||||
return cli.ShowCommandHelp(cctx, cctx.Command.Name)
|
||||
}
|
||||
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -705,7 +699,7 @@ var marketCancelTransfer = &cli.Command{
|
||||
if !cctx.Args().Present() {
|
||||
return cli.ShowCommandHelp(cctx, cctx.Command.Name)
|
||||
}
|
||||
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -781,7 +775,7 @@ var transfersListCmd = &cli.Command{
|
||||
color.NoColor = !cctx.Bool("color")
|
||||
}
|
||||
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -848,7 +842,7 @@ var dealsPendingPublish = &cli.Command{
|
||||
},
|
||||
},
|
||||
Action: func(cctx *cli.Context) error {
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -26,7 +26,7 @@ var piecesListPiecesCmd = &cli.Command{
|
||||
Name: "list-pieces",
|
||||
Usage: "list registered pieces",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -49,7 +49,7 @@ var piecesListCidInfosCmd = &cli.Command{
|
||||
Name: "list-cids",
|
||||
Usage: "list registered payload CIDs",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -76,7 +76,7 @@ var piecesInfoCmd = &cli.Command{
|
||||
return lcli.ShowHelp(cctx, fmt.Errorf("must specify piece cid"))
|
||||
}
|
||||
|
||||
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -111,7 +111,7 @@ var piecesCidInfoCmd = &cli.Command{
|
||||
return lcli.ShowHelp(cctx, fmt.Errorf("must specify payload cid"))
|
||||
}
|
||||
|
||||
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -39,7 +39,7 @@ var retrievalDealSelectionShowCmd = &cli.Command{
|
||||
Name: "list",
|
||||
Usage: "List retrieval deal proposal selection criteria",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
smapi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -66,7 +66,7 @@ var retrievalDealSelectionResetCmd = &cli.Command{
|
||||
Name: "reset",
|
||||
Usage: "Reset retrieval deal proposal selection criteria to default values",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
smapi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -98,7 +98,7 @@ var retrievalDealSelectionRejectCmd = &cli.Command{
|
||||
},
|
||||
},
|
||||
Action: func(cctx *cli.Context) error {
|
||||
smapi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
smapi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -126,7 +126,7 @@ var retrievalDealsListCmd = &cli.Command{
|
||||
Name: "list",
|
||||
Usage: "List all active retrieval deals for this miner",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -186,7 +186,7 @@ var retrievalSetAskCmd = &cli.Command{
|
||||
Action: func(cctx *cli.Context) error {
|
||||
ctx := lcli.DaemonContext(cctx)
|
||||
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -240,7 +240,7 @@ var retrievalGetAskCmd = &cli.Command{
|
||||
Action: func(cctx *cli.Context) error {
|
||||
ctx := lcli.DaemonContext(cctx)
|
||||
|
||||
api, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
api, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -252,13 +252,13 @@ var retrievalGetAskCmd = &cli.Command{
|
||||
}
|
||||
|
||||
w := tabwriter.NewWriter(os.Stdout, 2, 4, 2, ' ', 0)
|
||||
_, _ = fmt.Fprintf(w, "Price per Byte\tUnseal Price\tPayment Interval\tPayment Interval Increase\n")
|
||||
fmt.Fprintf(w, "Price per Byte\tUnseal Price\tPayment Interval\tPayment Interval Increase\n")
|
||||
if ask == nil {
|
||||
_, _ = fmt.Fprintf(w, "<miner does not have an retrieval ask set>\n")
|
||||
fmt.Fprintf(w, "<miner does not have an retrieval ask set>\n")
|
||||
return w.Flush()
|
||||
}
|
||||
|
||||
_, _ = fmt.Fprintf(w, "%s\t%s\t%s\t%s\n",
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\n",
|
||||
types.FIL(ask.PricePerByte),
|
||||
types.FIL(ask.UnsealPrice),
|
||||
units.BytesSize(float64(ask.PaymentInterval)),
|
||||
|
||||
@@ -396,7 +396,7 @@ var sectorsRefsCmd = &cli.Command{
|
||||
Name: "refs",
|
||||
Usage: "List References to sectors",
|
||||
Action: func(cctx *cli.Context) error {
|
||||
nodeApi, closer, err := lcli.GetMarketsAPI(cctx)
|
||||
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -18,12 +18,6 @@
|
||||
* [ComputeProof](#ComputeProof)
|
||||
* [Create](#Create)
|
||||
* [CreateBackup](#CreateBackup)
|
||||
* [Dagstore](#Dagstore)
|
||||
* [DagstoreGC](#DagstoreGC)
|
||||
* [DagstoreInitializeAll](#DagstoreInitializeAll)
|
||||
* [DagstoreInitializeShard](#DagstoreInitializeShard)
|
||||
* [DagstoreListShards](#DagstoreListShards)
|
||||
* [DagstoreRecoverShard](#DagstoreRecoverShard)
|
||||
* [Deals](#Deals)
|
||||
* [DealsConsiderOfflineRetrievalDeals](#DealsConsiderOfflineRetrievalDeals)
|
||||
* [DealsConsiderOfflineStorageDeals](#DealsConsiderOfflineStorageDeals)
|
||||
@@ -100,8 +94,6 @@
|
||||
* [ReturnSealPreCommit1](#ReturnSealPreCommit1)
|
||||
* [ReturnSealPreCommit2](#ReturnSealPreCommit2)
|
||||
* [ReturnUnsealPiece](#ReturnUnsealPiece)
|
||||
* [Runtime](#Runtime)
|
||||
* [RuntimeSubsystems](#RuntimeSubsystems)
|
||||
* [Sealing](#Sealing)
|
||||
* [SealingAbort](#SealingAbort)
|
||||
* [SealingSchedDiag](#SealingSchedDiag)
|
||||
@@ -351,113 +343,6 @@ Inputs:
|
||||
|
||||
Response: `{}`
|
||||
|
||||
## Dagstore
|
||||
|
||||
|
||||
### DagstoreGC
|
||||
DagstoreGC runs garbage collection on the DAG store.
|
||||
|
||||
|
||||
Perms: admin
|
||||
|
||||
Inputs: `null`
|
||||
|
||||
Response: `null`
|
||||
|
||||
### DagstoreInitializeAll
|
||||
DagstoreInitializeAll initializes all uninitialized shards in bulk,
|
||||
according to the policy passed in the parameters.
|
||||
|
||||
It is recommended to set a maximum concurrency to avoid extreme
|
||||
IO pressure if the storage subsystem has a large amount of deals.
|
||||
|
||||
It returns a stream of events to report progress.
|
||||
|
||||
|
||||
Perms: write
|
||||
|
||||
Inputs:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"MaxConcurrency": 123
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"Key": "string value",
|
||||
"Event": "string value",
|
||||
"Success": true,
|
||||
"Error": "string value",
|
||||
"Total": 123,
|
||||
"Current": 123
|
||||
}
|
||||
```
|
||||
|
||||
### DagstoreInitializeShard
|
||||
DagstoreInitializeShard initializes an uninitialized shard.
|
||||
|
||||
Initialization consists of fetching the shard's data (deal payload) from
|
||||
the storage subsystem, generating an index, and persisting the index
|
||||
to facilitate later retrievals, and/or to publish to external sources.
|
||||
|
||||
This operation is intended to complement the initial migration. The
|
||||
migration registers a shard for every unique piece CID, with lazy
|
||||
initialization. Thus, shards are not initialized immediately to avoid
|
||||
IO activity competing with proving. Instead, shard are initialized
|
||||
when first accessed. This method forces the initialization of a shard by
|
||||
accessing it and immediately releasing it. This is useful to warm up the
|
||||
cache to facilitate subsequent retrievals, and to generate the indexes
|
||||
to publish them externally.
|
||||
|
||||
This operation fails if the shard is not in ShardStateNew state.
|
||||
It blocks until initialization finishes.
|
||||
|
||||
|
||||
Perms: write
|
||||
|
||||
Inputs:
|
||||
```json
|
||||
[
|
||||
"string value"
|
||||
]
|
||||
```
|
||||
|
||||
Response: `{}`
|
||||
|
||||
### DagstoreListShards
|
||||
DagstoreListShards returns information about all shards known to the
|
||||
DAG store. Only available on nodes running the markets subsystem.
|
||||
|
||||
|
||||
Perms: read
|
||||
|
||||
Inputs: `null`
|
||||
|
||||
Response: `null`
|
||||
|
||||
### DagstoreRecoverShard
|
||||
DagstoreRecoverShard attempts to recover a failed shard.
|
||||
|
||||
This operation fails if the shard is not in ShardStateErrored state.
|
||||
It blocks until recovery finishes. If recovery failed, it returns the
|
||||
error.
|
||||
|
||||
|
||||
Perms: write
|
||||
|
||||
Inputs:
|
||||
```json
|
||||
[
|
||||
"string value"
|
||||
]
|
||||
```
|
||||
|
||||
Response: `{}`
|
||||
|
||||
## Deals
|
||||
|
||||
|
||||
@@ -822,7 +707,7 @@ Response:
|
||||
"ID": 3
|
||||
},
|
||||
"SectorNumber": 9,
|
||||
"InboundCAR": "string value"
|
||||
"CARv2FilePath": "string value"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -1637,28 +1522,6 @@ Inputs:
|
||||
|
||||
Response: `{}`
|
||||
|
||||
## Runtime
|
||||
|
||||
|
||||
### RuntimeSubsystems
|
||||
RuntimeSubsystems returns the subsystems that are enabled
|
||||
in this instance.
|
||||
|
||||
|
||||
Perms: read
|
||||
|
||||
Inputs: `null`
|
||||
|
||||
Response:
|
||||
```json
|
||||
[
|
||||
"Mining",
|
||||
"Sealing",
|
||||
"SectorStorage",
|
||||
"Markets"
|
||||
]
|
||||
```
|
||||
|
||||
## Sealing
|
||||
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ USAGE:
|
||||
lotus-miner [global options] command [command options] [arguments...]
|
||||
|
||||
VERSION:
|
||||
1.11.1-m1.3.4
|
||||
1.11.1-dev
|
||||
|
||||
COMMANDS:
|
||||
init Initialize a lotus miner repo
|
||||
@@ -29,7 +29,6 @@ COMMANDS:
|
||||
storage-deals Manage storage deals and related configuration
|
||||
retrieval-deals Manage retrieval deals and related configuration
|
||||
data-transfers Manage data transfers
|
||||
dagstore Manage the dagstore on the markets subsystem
|
||||
NETWORK:
|
||||
net Manage P2P Network
|
||||
RETRIEVAL:
|
||||
@@ -44,8 +43,6 @@ GLOBAL OPTIONS:
|
||||
--actor value, -a value specify other actor to check state for (read only)
|
||||
--color use color in display output (default: depends on output being a TTY)
|
||||
--miner-repo value, --storagerepo value Specify miner repo path. flag(storagerepo) and env(LOTUS_STORAGE_PATH) are DEPRECATION, will REMOVE SOON (default: "~/.lotusminer") [$LOTUS_MINER_PATH, $LOTUS_STORAGE_PATH]
|
||||
--markets-repo value Markets repo path [$LOTUS_MARKETS_PATH]
|
||||
--call-on-markets (experimental; may be removed) call this command against a markets node; use only with common commands like net, auth, pprof, etc. whose target may be ambiguous (default: false)
|
||||
--vv enables very verbose mode, useful for debugging the CLI (default: false)
|
||||
--help, -h show help (default: false)
|
||||
--version, -v print the version (default: false)
|
||||
@@ -1001,94 +998,6 @@ OPTIONS:
|
||||
|
||||
```
|
||||
|
||||
## lotus-miner dagstore
|
||||
```
|
||||
NAME:
|
||||
lotus-miner dagstore - Manage the dagstore on the markets subsystem
|
||||
|
||||
USAGE:
|
||||
lotus-miner dagstore command [command options] [arguments...]
|
||||
|
||||
COMMANDS:
|
||||
list-shards List all shards known to the dagstore, with their current status
|
||||
initialize-shard Initialize the specified shard
|
||||
recover-shard Attempt to recover a shard in errored state
|
||||
initialize-all Initialize all uninitialized shards, streaming results as they're produced
|
||||
gc Garbage collect the dagstore
|
||||
help, h Shows a list of commands or help for one command
|
||||
|
||||
OPTIONS:
|
||||
--help, -h show help (default: false)
|
||||
--version, -v print the version (default: false)
|
||||
|
||||
```
|
||||
|
||||
### lotus-miner dagstore list-shards
|
||||
```
|
||||
NAME:
|
||||
lotus-miner dagstore list-shards - List all shards known to the dagstore, with their current status
|
||||
|
||||
USAGE:
|
||||
lotus-miner dagstore list-shards [command options] [arguments...]
|
||||
|
||||
OPTIONS:
|
||||
--help, -h show help (default: false)
|
||||
|
||||
```
|
||||
|
||||
### lotus-miner dagstore initialize-shard
|
||||
```
|
||||
NAME:
|
||||
lotus-miner dagstore initialize-shard - Initialize the specified shard
|
||||
|
||||
USAGE:
|
||||
lotus-miner dagstore initialize-shard [command options] [key]
|
||||
|
||||
OPTIONS:
|
||||
--help, -h show help (default: false)
|
||||
|
||||
```
|
||||
|
||||
### lotus-miner dagstore recover-shard
|
||||
```
|
||||
NAME:
|
||||
lotus-miner dagstore recover-shard - Attempt to recover a shard in errored state
|
||||
|
||||
USAGE:
|
||||
lotus-miner dagstore recover-shard [command options] [key]
|
||||
|
||||
OPTIONS:
|
||||
--help, -h show help (default: false)
|
||||
|
||||
```
|
||||
|
||||
### lotus-miner dagstore initialize-all
|
||||
```
|
||||
NAME:
|
||||
lotus-miner dagstore initialize-all - Initialize all uninitialized shards, streaming results as they're produced
|
||||
|
||||
USAGE:
|
||||
lotus-miner dagstore initialize-all [command options] [arguments...]
|
||||
|
||||
OPTIONS:
|
||||
--concurrency value maximum shards to initialize concurrently at a time; use 0 for unlimited (default: 0)
|
||||
--help, -h show help (default: false)
|
||||
|
||||
```
|
||||
|
||||
### lotus-miner dagstore gc
|
||||
```
|
||||
NAME:
|
||||
lotus-miner dagstore gc - Garbage collect the dagstore
|
||||
|
||||
USAGE:
|
||||
lotus-miner dagstore gc [command options] [arguments...]
|
||||
|
||||
OPTIONS:
|
||||
--help, -h show help (default: false)
|
||||
|
||||
```
|
||||
|
||||
## lotus-miner net
|
||||
```
|
||||
NAME:
|
||||
|
||||
@@ -7,7 +7,7 @@ USAGE:
|
||||
lotus-worker [global options] command [command options] [arguments...]
|
||||
|
||||
VERSION:
|
||||
1.11.1-m1.3.4
|
||||
1.11.1-dev
|
||||
|
||||
COMMANDS:
|
||||
run Start lotus worker
|
||||
|
||||
@@ -7,7 +7,7 @@ USAGE:
|
||||
lotus [global options] command [command options] [arguments...]
|
||||
|
||||
VERSION:
|
||||
1.11.1-m1.3.4
|
||||
1.11.1-dev
|
||||
|
||||
COMMANDS:
|
||||
daemon Start a lotus daemon process
|
||||
|
||||
@@ -26,23 +26,23 @@ require (
|
||||
github.com/elastic/gosigar v0.12.0
|
||||
github.com/etclabscore/go-openrpc-reflect v0.0.36
|
||||
github.com/fatih/color v1.9.0
|
||||
github.com/filecoin-project/dagstore v0.4.2
|
||||
github.com/filecoin-project/dagstore v0.3.1
|
||||
github.com/filecoin-project/filecoin-ffi v0.30.4-0.20200910194244-f640612a1a1f
|
||||
github.com/filecoin-project/go-address v0.0.5
|
||||
github.com/filecoin-project/go-bitfield v0.2.4
|
||||
github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2
|
||||
github.com/filecoin-project/go-commp-utils v0.1.1-0.20210427191551-70bf140d31c7
|
||||
github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03
|
||||
github.com/filecoin-project/go-data-transfer v1.7.3
|
||||
github.com/filecoin-project/go-data-transfer v1.7.0
|
||||
github.com/filecoin-project/go-fil-commcid v0.1.0
|
||||
github.com/filecoin-project/go-fil-commp-hashhash v0.1.0
|
||||
github.com/filecoin-project/go-fil-markets v1.6.3-0.20210806151421-967d8717c917
|
||||
github.com/filecoin-project/go-fil-markets v1.6.1-0.20210727201249-2213f80f0403
|
||||
github.com/filecoin-project/go-jsonrpc v0.1.4-0.20210217175800-45ea43ac2bec
|
||||
github.com/filecoin-project/go-multistore v0.0.3
|
||||
github.com/filecoin-project/go-padreader v0.0.0-20210723183308-812a16dc01b1
|
||||
github.com/filecoin-project/go-paramfetch v0.0.2-0.20210614165157-25a6c7769498
|
||||
github.com/filecoin-project/go-state-types v0.1.1-0.20210722133031-ad9bfe54c124
|
||||
github.com/filecoin-project/go-statemachine v1.0.1
|
||||
github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe
|
||||
github.com/filecoin-project/go-statestore v0.1.1
|
||||
github.com/filecoin-project/go-storedcounter v0.0.0-20200421200003-1c99c62e8a5b
|
||||
github.com/filecoin-project/specs-actors v0.9.14
|
||||
@@ -56,7 +56,7 @@ require (
|
||||
github.com/gdamore/tcell/v2 v2.2.0
|
||||
github.com/go-kit/kit v0.10.0
|
||||
github.com/go-ole/go-ole v1.2.4 // indirect
|
||||
github.com/golang/mock v1.6.0
|
||||
github.com/golang/mock v1.5.0
|
||||
github.com/google/uuid v1.2.0
|
||||
github.com/gorilla/mux v1.7.4
|
||||
github.com/gorilla/websocket v1.4.2
|
||||
@@ -78,7 +78,7 @@ require (
|
||||
github.com/ipfs/go-ds-pebble v0.0.2-0.20200921225637-ce220f8ac459
|
||||
github.com/ipfs/go-filestore v1.0.0
|
||||
github.com/ipfs/go-fs-lock v0.0.6
|
||||
github.com/ipfs/go-graphsync v0.6.8
|
||||
github.com/ipfs/go-graphsync v0.6.5
|
||||
github.com/ipfs/go-ipfs-blockstore v1.0.4
|
||||
github.com/ipfs/go-ipfs-blocksutil v0.0.1
|
||||
github.com/ipfs/go-ipfs-chunker v0.0.5
|
||||
@@ -99,7 +99,7 @@ require (
|
||||
github.com/ipfs/go-unixfs v0.2.6
|
||||
github.com/ipfs/interface-go-ipfs-core v0.2.3
|
||||
github.com/ipld/go-car v0.1.1-0.20201119040415-11b6074b6d4d
|
||||
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210803143113-d0e44a62f0a3
|
||||
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210721090610-5a9d1b217d25
|
||||
github.com/ipld/go-ipld-prime v0.5.1-0.20201021195245-109253e8a018
|
||||
github.com/kelseyhightower/envconfig v1.4.0
|
||||
github.com/lib/pq v1.7.0
|
||||
|
||||
@@ -256,8 +256,8 @@ github.com/fatih/color v1.8.0/go.mod h1:3l45GVGkyrnYNl9HoIjnp2NnNWvh6hLAqD8yTfGj
|
||||
github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
|
||||
github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
|
||||
github.com/fd/go-nat v1.0.0/go.mod h1:BTBu/CKvMmOMUPkKVef1pngt2WFH/lg7E6yQnulfp6E=
|
||||
github.com/filecoin-project/dagstore v0.4.2 h1:Ae2+O1DhKCI1JbOZCBkqUksKYofdbRbjkS7OF0A6Jw0=
|
||||
github.com/filecoin-project/dagstore v0.4.2/go.mod h1:WY5OoLfnwISCk6eASSF927KKPqLPIlTwmG1qHpA08KY=
|
||||
github.com/filecoin-project/dagstore v0.3.1 h1:IShahbR3zEPE1pfHa5DiYMt/S40b9JuWZx6+kZPuyTQ=
|
||||
github.com/filecoin-project/dagstore v0.3.1/go.mod h1:WY5OoLfnwISCk6eASSF927KKPqLPIlTwmG1qHpA08KY=
|
||||
github.com/filecoin-project/go-address v0.0.3/go.mod h1:jr8JxKsYx+lQlQZmF5i2U0Z+cGQ59wMIps/8YW/lDj8=
|
||||
github.com/filecoin-project/go-address v0.0.5 h1:SSaFT/5aLfPXycUlFyemoHYhRgdyXClXCyDdNJKPlDM=
|
||||
github.com/filecoin-project/go-address v0.0.5/go.mod h1:jr8JxKsYx+lQlQZmF5i2U0Z+cGQ59wMIps/8YW/lDj8=
|
||||
@@ -277,9 +277,8 @@ github.com/filecoin-project/go-commp-utils v0.1.1-0.20210427191551-70bf140d31c7/
|
||||
github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03 h1:2pMXdBnCiXjfCYx/hLqFxccPoqsSveQFxVLvNxy9bus=
|
||||
github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
|
||||
github.com/filecoin-project/go-data-transfer v1.0.1/go.mod h1:UxvfUAY9v3ub0a21BSK9u3pB2aq30Y0KMsG+w9/ysyo=
|
||||
github.com/filecoin-project/go-data-transfer v1.7.0 h1:mFRn+UuTdPROmhplLSekzd4rAs9ug8ubtSY4nw9wYkU=
|
||||
github.com/filecoin-project/go-data-transfer v1.7.0/go.mod h1:GLRr5BmLEqsLwXfiRDG7uJvph22KGL2M4iOuF8EINaU=
|
||||
github.com/filecoin-project/go-data-transfer v1.7.3 h1:2F3K1LmT6Q97+Bq9gBv/VVqFZ/0FHsRPqYDbCzrGaYg=
|
||||
github.com/filecoin-project/go-data-transfer v1.7.3/go.mod h1:Cbl9lzKOuAyyIxp1tE+VbV5Aix4bxzA7uJGA9wGM4fM=
|
||||
github.com/filecoin-project/go-ds-versioning v0.1.0 h1:y/X6UksYTsK8TLCI7rttCKEvl8btmWxyFMEeeWGUxIQ=
|
||||
github.com/filecoin-project/go-ds-versioning v0.1.0/go.mod h1:mp16rb4i2QPmxBnmanUx8i/XANp+PFCCJWiAb+VW4/s=
|
||||
github.com/filecoin-project/go-fil-commcid v0.0.0-20200716160307-8f644712406f/go.mod h1:Eaox7Hvus1JgPrL5+M3+h7aSPHc0cVqpSxA+TxIEpZQ=
|
||||
@@ -289,8 +288,8 @@ github.com/filecoin-project/go-fil-commcid v0.1.0/go.mod h1:Eaox7Hvus1JgPrL5+M3+
|
||||
github.com/filecoin-project/go-fil-commp-hashhash v0.1.0 h1:imrrpZWEHRnNqqv0tN7LXep5bFEVOVmQWHJvl2mgsGo=
|
||||
github.com/filecoin-project/go-fil-commp-hashhash v0.1.0/go.mod h1:73S8WSEWh9vr0fDJVnKADhfIv/d6dCbAGaAGWbdJEI8=
|
||||
github.com/filecoin-project/go-fil-markets v1.0.5-0.20201113164554-c5eba40d5335/go.mod h1:AJySOJC00JRWEZzRG2KsfUnqEf5ITXxeX09BE9N4f9c=
|
||||
github.com/filecoin-project/go-fil-markets v1.6.3-0.20210806151421-967d8717c917 h1:A7B4EShovtH4B8tMp6+Smst7B+H+CLSJ4tcjcS15ta4=
|
||||
github.com/filecoin-project/go-fil-markets v1.6.3-0.20210806151421-967d8717c917/go.mod h1:9DxT29OFLUJPswugvIbqD5Ab768pJQxkxbnTEFTI8lk=
|
||||
github.com/filecoin-project/go-fil-markets v1.6.1-0.20210727201249-2213f80f0403 h1:Zrr2YRH186x0w4U5kaCYX+ccDD+g09yjYIO6QsICLpA=
|
||||
github.com/filecoin-project/go-fil-markets v1.6.1-0.20210727201249-2213f80f0403/go.mod h1:13+DUe7AaHekzgpQPbacdppRoqz0SyPlx48g0f/pRmA=
|
||||
github.com/filecoin-project/go-hamt-ipld v0.1.5 h1:uoXrKbCQZ49OHpsTCkrThPNelC4W3LPEk0OrS/ytIBM=
|
||||
github.com/filecoin-project/go-hamt-ipld v0.1.5/go.mod h1:6Is+ONR5Cd5R6XZoCse1CWaXZc0Hdb/JeX+EQCQzX24=
|
||||
github.com/filecoin-project/go-hamt-ipld/v2 v2.0.0 h1:b3UDemBYN2HNfk3KOXNuxgTTxlWi3xVvbQP0IT38fvM=
|
||||
@@ -315,9 +314,8 @@ github.com/filecoin-project/go-state-types v0.1.0/go.mod h1:ezYnPf0bNkTsDibL/psS
|
||||
github.com/filecoin-project/go-state-types v0.1.1-0.20210506134452-99b279731c48/go.mod h1:ezYnPf0bNkTsDibL/psSz5dy4B5awOJ/E7P2Saeep8g=
|
||||
github.com/filecoin-project/go-state-types v0.1.1-0.20210722133031-ad9bfe54c124 h1:veGrNABg/9I7prngrowkhwbvW5d5JN55MNKmbsr5FqA=
|
||||
github.com/filecoin-project/go-state-types v0.1.1-0.20210722133031-ad9bfe54c124/go.mod h1:ezYnPf0bNkTsDibL/psSz5dy4B5awOJ/E7P2Saeep8g=
|
||||
github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe h1:dF8u+LEWeIcTcfUcCf3WFVlc81Fr2JKg8zPzIbBDKDw=
|
||||
github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe/go.mod h1:FGwQgZAt2Gh5mjlwJUlVB62JeYdo+if0xWxSEfBD9ig=
|
||||
github.com/filecoin-project/go-statemachine v1.0.1 h1:LQ60+JDVjMdLxXmVFM2jjontzOYnfVE7u02CXV3WKSw=
|
||||
github.com/filecoin-project/go-statemachine v1.0.1/go.mod h1:jZdXXiHa61n4NmgWFG4w8tnqgvZVHYbJ3yW7+y8bF54=
|
||||
github.com/filecoin-project/go-statestore v0.1.0/go.mod h1:LFc9hD+fRxPqiHiaqUEZOinUJB4WARkRfNl10O7kTnI=
|
||||
github.com/filecoin-project/go-statestore v0.1.1 h1:ufMFq00VqnT2CAuDpcGnwLnCX1I/c3OROw/kXVNSTZk=
|
||||
github.com/filecoin-project/go-statestore v0.1.1/go.mod h1:LFc9hD+fRxPqiHiaqUEZOinUJB4WARkRfNl10O7kTnI=
|
||||
@@ -434,8 +432,8 @@ github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFU
|
||||
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
|
||||
github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
|
||||
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
|
||||
github.com/golang/mock v1.6.0 h1:ErTB+efbowRARo13NNdxyJji2egdxLGQhRaY+DUumQc=
|
||||
github.com/golang/mock v1.6.0/go.mod h1:p6yTPP+5HYm5mzsMV8JkE6ZKdX+/wYM6Hr+LicevLPs=
|
||||
github.com/golang/mock v1.5.0 h1:jlYHihg//f7RRwuPfptm04yp4s7O6Kw8EZiVYIGcH0g=
|
||||
github.com/golang/mock v1.5.0/go.mod h1:CWnOUgYIOo4TcNZ0wHX3YZCqsaM1I1Jvs6v3mP3KVu8=
|
||||
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/golang/protobuf v1.3.0/go.mod h1:Qd/q+1AKNOZr9uGQzbzCmRO6sUih6GTPZv6a1/R87v0=
|
||||
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
@@ -639,8 +637,8 @@ github.com/ipfs/go-graphsync v0.1.0/go.mod h1:jMXfqIEDFukLPZHqDPp8tJMbHO9Rmeb9CE
|
||||
github.com/ipfs/go-graphsync v0.4.2/go.mod h1:/VmbZTUdUMTbNkgzAiCEucIIAU3BkLE2cZrDCVUhyi0=
|
||||
github.com/ipfs/go-graphsync v0.4.3/go.mod h1:mPOwDYv128gf8gxPFgXnz4fNrSYPsWyqisJ7ych+XDY=
|
||||
github.com/ipfs/go-graphsync v0.6.4/go.mod h1:5WyaeigpNdpiYQuW2vwpuecOoEfB4h747ZGEOKmAGTg=
|
||||
github.com/ipfs/go-graphsync v0.6.8 h1:mgyPdBDPcgL8ujO132grQjP3rfQv+KN/riQzbmTVgg4=
|
||||
github.com/ipfs/go-graphsync v0.6.8/go.mod h1:GdHT8JeuIZ0R4lSjFR16Oe4zPi5dXwKi9zR9ADVlcdk=
|
||||
github.com/ipfs/go-graphsync v0.6.5 h1:YAJl6Yit23PQcaawzb1rPK9PSnbbq2jjMRPpRpJ0Y5U=
|
||||
github.com/ipfs/go-graphsync v0.6.5/go.mod h1:GdHT8JeuIZ0R4lSjFR16Oe4zPi5dXwKi9zR9ADVlcdk=
|
||||
github.com/ipfs/go-hamt-ipld v0.1.1/go.mod h1:1EZCr2v0jlCnhpa+aZ0JZYp8Tt2w16+JJOAVz17YcDk=
|
||||
github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08=
|
||||
github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw=
|
||||
@@ -759,9 +757,8 @@ github.com/ipld/go-car v0.1.0/go.mod h1:RCWzaUh2i4mOEkB3W45Vc+9jnS/M6Qay5ooytiBH
|
||||
github.com/ipld/go-car v0.1.1-0.20200923150018-8cdef32e2da4/go.mod h1:xrMEcuSq+D1vEwl+YAXsg/JfA98XGpXDwnkIL4Aimqw=
|
||||
github.com/ipld/go-car v0.1.1-0.20201119040415-11b6074b6d4d h1:iphSzTuPqyDgH7WUVZsdqUnQNzYgIblsVr1zhVNA33U=
|
||||
github.com/ipld/go-car v0.1.1-0.20201119040415-11b6074b6d4d/go.mod h1:2Gys8L8MJ6zkh1gktTSXreY63t4UbyvNp5JaudTyxHQ=
|
||||
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210721090610-5a9d1b217d25 h1:SeNfgKCuIhGJKpaSVvn8WX7uHxWeHvwVUq1MIh2KM2o=
|
||||
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210721090610-5a9d1b217d25/go.mod h1:I2ACeeg6XNBe5pdh5TaR7Ambhfa7If9KXxmXgZsYENU=
|
||||
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210803143113-d0e44a62f0a3 h1:TPOuaAmTWpzWFbGPsJ+QJsR6Qc+fnHKgvtA6Jy2A22w=
|
||||
github.com/ipld/go-car/v2 v2.0.0-beta1.0.20210803143113-d0e44a62f0a3/go.mod h1:I2ACeeg6XNBe5pdh5TaR7Ambhfa7If9KXxmXgZsYENU=
|
||||
github.com/ipld/go-ipld-prime v0.0.2-0.20191108012745-28a82f04c785/go.mod h1:bDDSvVz7vaK12FNvMeRYnpRFkSUPNQOiCYQezMD/P3w=
|
||||
github.com/ipld/go-ipld-prime v0.0.2-0.20200428162820-8b59dc292b8e/go.mod h1:uVIwe/u0H4VdKv3kaN1ck7uCb6yD9cFLS9/ELyXbsw8=
|
||||
github.com/ipld/go-ipld-prime v0.5.1-0.20200828233916-988837377a7f/go.mod h1:0xEgdD6MKbZ1vF0GC+YcR/C4SQCAlRuOjIJ2i0HxqzM=
|
||||
@@ -1989,7 +1986,6 @@ golang.org/x/tools v0.0.0-20200827010519-17fd2f27a9e3/go.mod h1:njjCfa9FT2d7l9Bc
|
||||
golang.org/x/tools v0.0.0-20201112185108-eeaa07dd7696/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
|
||||
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
|
||||
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
|
||||
golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
|
||||
golang.org/x/tools v0.1.5 h1:ouewzE6p+/VEB31YYnTbEJdi8pFqKp4P4n85vwo3DHA=
|
||||
golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
|
||||
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
|
||||
@@ -42,4 +42,4 @@ func TestDealsWithSealingAndRPC(t *testing.T) {
|
||||
dh.RunConcurrentDeals(kit.RunConcurrentDealsOpts{N: 1, UseCARFileForStorageDeal: true})
|
||||
})
|
||||
|
||||
}
|
||||
}
|
||||
@@ -1,33 +0,0 @@
|
||||
package dagstore
|
||||
|
||||
import (
|
||||
"io"
|
||||
|
||||
blocks "github.com/ipfs/go-block-format"
|
||||
"github.com/ipfs/go-cid"
|
||||
bstore "github.com/ipfs/go-ipfs-blockstore"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/dagstore"
|
||||
)
|
||||
|
||||
// Blockstore promotes a dagstore.ReadBlockstore to a full closeable Blockstore,
|
||||
// stubbing out the write methods with erroring implementations.
|
||||
type Blockstore struct {
|
||||
dagstore.ReadBlockstore
|
||||
io.Closer
|
||||
}
|
||||
|
||||
var _ bstore.Blockstore = (*Blockstore)(nil)
|
||||
|
||||
func (b *Blockstore) DeleteBlock(c cid.Cid) error {
|
||||
return xerrors.Errorf("DeleteBlock called but not implemented")
|
||||
}
|
||||
|
||||
func (b *Blockstore) Put(block blocks.Block) error {
|
||||
return xerrors.Errorf("Put called but not implemented")
|
||||
}
|
||||
|
||||
func (b *Blockstore) PutMany(blocks []blocks.Block) error {
|
||||
return xerrors.Errorf("PutMany called but not implemented")
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package dagstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
|
||||
"github.com/filecoin-project/dagstore/throttle"
|
||||
"github.com/ipfs/go-cid"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/go-fil-markets/piecestore"
|
||||
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
|
||||
"github.com/filecoin-project/go-fil-markets/shared"
|
||||
)
|
||||
|
||||
// MaxConcurrentUnsealedFetches caps the amount of concurrent fetches for
|
||||
// unsealed pieces, so that we don't saturate IO or the network too much,
|
||||
// especially when bulk processing (e.g. at migration).
|
||||
var MaxConcurrentUnsealedFetches = 3
|
||||
|
||||
type LotusAccessor interface {
|
||||
FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error)
|
||||
GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid) (uint64, error)
|
||||
Start(ctx context.Context) error
|
||||
}
|
||||
|
||||
type lotusAccessor struct {
|
||||
pieceStore piecestore.PieceStore
|
||||
rm retrievalmarket.RetrievalProviderNode
|
||||
throttle throttle.Throttler
|
||||
readyMgr *shared.ReadyManager
|
||||
}
|
||||
|
||||
var _ LotusAccessor = (*lotusAccessor)(nil)
|
||||
|
||||
func NewLotusAccessor(store piecestore.PieceStore, rm retrievalmarket.RetrievalProviderNode) LotusAccessor {
|
||||
return &lotusAccessor{
|
||||
pieceStore: store,
|
||||
rm: rm,
|
||||
throttle: throttle.Fixed(MaxConcurrentUnsealedFetches),
|
||||
readyMgr: shared.NewReadyManager(),
|
||||
}
|
||||
}
|
||||
|
||||
func (m *lotusAccessor) Start(ctx context.Context) error {
|
||||
// Wait for the piece store to startup
|
||||
ready := make(chan error)
|
||||
m.pieceStore.OnReady(func(err error) {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
case ready <- err:
|
||||
}
|
||||
})
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
err := xerrors.Errorf("context cancelled waiting for piece store startup: %w", ctx.Err())
|
||||
if ferr := m.readyMgr.FireReady(err); ferr != nil {
|
||||
log.Warnw("failed to publish ready event", "err", ferr)
|
||||
}
|
||||
return err
|
||||
|
||||
case err := <-ready:
|
||||
if ferr := m.readyMgr.FireReady(err); ferr != nil {
|
||||
log.Warnw("failed to publish ready event", "err", ferr)
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
func (m *lotusAccessor) FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error) {
|
||||
err := m.readyMgr.AwaitReady()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
pieceInfo, err := m.pieceStore.GetPieceInfo(pieceCid)
|
||||
if err != nil {
|
||||
return nil, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
|
||||
}
|
||||
|
||||
if len(pieceInfo.Deals) == 0 {
|
||||
return nil, xerrors.Errorf("no storage deals found for Piece %s", pieceCid)
|
||||
}
|
||||
|
||||
// prefer an unsealed sector containing the piece if one exists
|
||||
for _, deal := range pieceInfo.Deals {
|
||||
isUnsealed, err := m.rm.IsUnsealed(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
|
||||
if err != nil {
|
||||
log.Warnf("failed to check if deal %d unsealed: %s", deal.DealID, err)
|
||||
continue
|
||||
}
|
||||
if isUnsealed {
|
||||
var reader io.ReadCloser
|
||||
// We want to throttle this path, as these copies will be downloaded
|
||||
// immediately from the storage cluster without any unsealing
|
||||
// necessary.
|
||||
deal := deal // calm the linter
|
||||
err := m.throttle.Do(ctx, func(ctx context.Context) (err error) {
|
||||
// UnsealSector will NOT unseal a sector if we already have an unsealed copy lying around.
|
||||
reader, err = m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
|
||||
return err
|
||||
})
|
||||
return reader, err
|
||||
}
|
||||
}
|
||||
|
||||
lastErr := xerrors.New("no sectors found to unseal from")
|
||||
// if there is no unsealed sector containing the piece, just read the piece from the first sector we are able to unseal.
|
||||
for _, deal := range pieceInfo.Deals {
|
||||
// Note that if the deal data is not already unsealed, unsealing may
|
||||
// block for a long time with the current PoRep
|
||||
reader, err := m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
|
||||
if err != nil {
|
||||
lastErr = xerrors.Errorf("failed to unseal deal %d: %w", deal.DealID, err)
|
||||
log.Warn(lastErr.Error())
|
||||
continue
|
||||
}
|
||||
|
||||
// Successfully fetched the deal data so return a reader over the data
|
||||
return reader, nil
|
||||
}
|
||||
|
||||
return nil, lastErr
|
||||
}
|
||||
|
||||
func (m *lotusAccessor) GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid) (uint64, error) {
|
||||
err := m.readyMgr.AwaitReady()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
pieceInfo, err := m.pieceStore.GetPieceInfo(pieceCid)
|
||||
if err != nil {
|
||||
return 0, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
|
||||
}
|
||||
|
||||
if len(pieceInfo.Deals) == 0 {
|
||||
return 0, xerrors.Errorf("no storage deals found for piece %s", pieceCid)
|
||||
}
|
||||
|
||||
len := pieceInfo.Deals[0].Length
|
||||
|
||||
return uint64(len), nil
|
||||
}
|
||||
@@ -45,9 +45,7 @@ func TestLotusAccessorFetchUnsealedPiece(t *testing.T) {
|
||||
name string
|
||||
deals []abi.SectorNumber
|
||||
fetchedData string
|
||||
isUnsealed bool
|
||||
|
||||
expectErr bool
|
||||
expectErr bool
|
||||
}{{
|
||||
// Expect error if there is no deal info for piece CID
|
||||
name: "no deals",
|
||||
@@ -58,13 +56,11 @@ func TestLotusAccessorFetchUnsealedPiece(t *testing.T) {
|
||||
name: "prefer unsealed deal",
|
||||
deals: []abi.SectorNumber{unsealedSectorID, sealedSectorID},
|
||||
fetchedData: unsealedSectorData,
|
||||
isUnsealed: true,
|
||||
}, {
|
||||
// Expect the API to unseal the data if there are no unsealed deals
|
||||
name: "unseal if necessary",
|
||||
deals: []abi.SectorNumber{sealedSectorID},
|
||||
fetchedData: sealedSectorData,
|
||||
isUnsealed: false,
|
||||
}}
|
||||
|
||||
for _, tc := range testCases {
|
||||
@@ -74,7 +70,7 @@ func TestLotusAccessorFetchUnsealedPiece(t *testing.T) {
|
||||
rpn := &mockRPN{
|
||||
sectors: mockData,
|
||||
}
|
||||
api := NewMinerAPI(ps, rpn, 100)
|
||||
api := NewLotusAccessor(ps, rpn)
|
||||
require.NoError(t, api.Start(ctx))
|
||||
|
||||
// Add deals to piece store
|
||||
@@ -99,10 +95,6 @@ func TestLotusAccessorFetchUnsealedPiece(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Equal(t, tc.fetchedData, string(bz))
|
||||
|
||||
uns, err := api.IsUnsealed(ctx, cid1)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, tc.isUnsealed, uns)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -114,7 +106,7 @@ func TestLotusAccessorGetUnpaddedCARSize(t *testing.T) {
|
||||
|
||||
ps := getPieceStore(t)
|
||||
rpn := &mockRPN{}
|
||||
api := NewMinerAPI(ps, rpn, 100)
|
||||
api := NewLotusAccessor(ps, rpn)
|
||||
require.NoError(t, api.Start(ctx))
|
||||
|
||||
// Add a deal with data Length 10
|
||||
@@ -141,7 +133,7 @@ func TestThrottle(t *testing.T) {
|
||||
unsealedSectorID: "foo",
|
||||
},
|
||||
}
|
||||
api := NewMinerAPI(ps, rpn, 3)
|
||||
api := NewLotusAccessor(ps, rpn)
|
||||
require.NoError(t, api.Start(ctx))
|
||||
|
||||
// Add a deal with data Length 10
|
||||
@@ -168,7 +160,7 @@ func TestThrottle(t *testing.T) {
|
||||
}
|
||||
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
require.EqualValues(t, 3, atomic.LoadInt32(&rpn.calls)) // throttled
|
||||
require.EqualValues(t, MaxConcurrentUnsealedFetches, atomic.LoadInt32(&rpn.calls)) // throttled
|
||||
|
||||
// allow to proceed.
|
||||
rpn.lk.Unlock()
|
||||
@@ -185,14 +177,8 @@ func getPieceStore(t *testing.T) piecestore.PieceStore {
|
||||
ps, err := piecestoreimpl.NewPieceStore(ds_sync.MutexWrap(ds.NewMapDatastore()))
|
||||
require.NoError(t, err)
|
||||
|
||||
ch := make(chan struct{}, 1)
|
||||
ps.OnReady(func(_ error) {
|
||||
ch <- struct{}{}
|
||||
})
|
||||
|
||||
err = ps.Start(context.Background())
|
||||
require.NoError(t, err)
|
||||
<-ch
|
||||
return ps
|
||||
}
|
||||
|
||||
@@ -1,184 +0,0 @@
|
||||
package dagstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"github.com/filecoin-project/dagstore/throttle"
|
||||
"github.com/ipfs/go-cid"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/go-fil-markets/piecestore"
|
||||
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
|
||||
"github.com/filecoin-project/go-fil-markets/shared"
|
||||
)
|
||||
|
||||
type MinerAPI interface {
|
||||
FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error)
|
||||
GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid) (uint64, error)
|
||||
IsUnsealed(ctx context.Context, pieceCid cid.Cid) (bool, error)
|
||||
Start(ctx context.Context) error
|
||||
}
|
||||
|
||||
type minerAPI struct {
|
||||
pieceStore piecestore.PieceStore
|
||||
rm retrievalmarket.RetrievalProviderNode
|
||||
throttle throttle.Throttler
|
||||
readyMgr *shared.ReadyManager
|
||||
}
|
||||
|
||||
var _ MinerAPI = (*minerAPI)(nil)
|
||||
|
||||
func NewMinerAPI(store piecestore.PieceStore, rm retrievalmarket.RetrievalProviderNode, concurrency int) MinerAPI {
|
||||
return &minerAPI{
|
||||
pieceStore: store,
|
||||
rm: rm,
|
||||
throttle: throttle.Fixed(concurrency),
|
||||
readyMgr: shared.NewReadyManager(),
|
||||
}
|
||||
}
|
||||
|
||||
func (m *minerAPI) Start(_ context.Context) error {
|
||||
return m.readyMgr.FireReady(nil)
|
||||
}
|
||||
|
||||
func (m *minerAPI) IsUnsealed(ctx context.Context, pieceCid cid.Cid) (bool, error) {
|
||||
err := m.readyMgr.AwaitReady()
|
||||
if err != nil {
|
||||
return false, xerrors.Errorf("failed while waiting for accessor to start: %w", err)
|
||||
}
|
||||
|
||||
var pieceInfo piecestore.PieceInfo
|
||||
err = m.throttle.Do(ctx, func(ctx context.Context) (err error) {
|
||||
pieceInfo, err = m.pieceStore.GetPieceInfo(pieceCid)
|
||||
return err
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
return false, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
|
||||
}
|
||||
|
||||
if len(pieceInfo.Deals) == 0 {
|
||||
return false, xerrors.Errorf("no storage deals found for piece %s", pieceCid)
|
||||
}
|
||||
|
||||
// check if we have an unsealed deal for the given piece in any of the unsealed sectors.
|
||||
for _, deal := range pieceInfo.Deals {
|
||||
deal := deal
|
||||
|
||||
var isUnsealed bool
|
||||
// Throttle this path to avoid flooding the storage subsystem.
|
||||
err := m.throttle.Do(ctx, func(ctx context.Context) (err error) {
|
||||
isUnsealed, err = m.rm.IsUnsealed(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to check if sector %d for deal %d was unsealed: %w", deal.SectorID, deal.DealID, err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
log.Warnf("failed to check/retrieve unsealed sector: %s", err)
|
||||
continue // move on to the next match.
|
||||
}
|
||||
|
||||
if isUnsealed {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
|
||||
// we don't have an unsealed sector containing the piece
|
||||
return false, nil
|
||||
}
|
||||
|
||||
func (m *minerAPI) FetchUnsealedPiece(ctx context.Context, pieceCid cid.Cid) (io.ReadCloser, error) {
|
||||
err := m.readyMgr.AwaitReady()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Throttle this path to avoid flooding the storage subsystem.
|
||||
var pieceInfo piecestore.PieceInfo
|
||||
err = m.throttle.Do(ctx, func(ctx context.Context) (err error) {
|
||||
pieceInfo, err = m.pieceStore.GetPieceInfo(pieceCid)
|
||||
return err
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
return nil, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
|
||||
}
|
||||
|
||||
if len(pieceInfo.Deals) == 0 {
|
||||
return nil, xerrors.Errorf("no storage deals found for piece %s", pieceCid)
|
||||
}
|
||||
|
||||
// prefer an unsealed sector containing the piece if one exists
|
||||
for _, deal := range pieceInfo.Deals {
|
||||
deal := deal
|
||||
|
||||
// Throttle this path to avoid flooding the storage subsystem.
|
||||
var reader io.ReadCloser
|
||||
err := m.throttle.Do(ctx, func(ctx context.Context) (err error) {
|
||||
isUnsealed, err := m.rm.IsUnsealed(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to check if sector %d for deal %d was unsealed: %w", deal.SectorID, deal.DealID, err)
|
||||
}
|
||||
if !isUnsealed {
|
||||
return nil
|
||||
}
|
||||
// Because we know we have an unsealed copy, this UnsealSector call will actually not perform any unsealing.
|
||||
reader, err = m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
|
||||
return err
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
log.Warnf("failed to check/retrieve unsealed sector: %s", err)
|
||||
continue // move on to the next match.
|
||||
}
|
||||
|
||||
if reader != nil {
|
||||
// we were able to obtain a reader for an already unsealed piece
|
||||
return reader, nil
|
||||
}
|
||||
}
|
||||
|
||||
lastErr := xerrors.New("no sectors found to unseal from")
|
||||
// if there is no unsealed sector containing the piece, just read the piece from the first sector we are able to unseal.
|
||||
for _, deal := range pieceInfo.Deals {
|
||||
// Note that if the deal data is not already unsealed, unsealing may
|
||||
// block for a long time with the current PoRep
|
||||
//
|
||||
// This path is unthrottled.
|
||||
reader, err := m.rm.UnsealSector(ctx, deal.SectorID, deal.Offset.Unpadded(), deal.Length.Unpadded())
|
||||
if err != nil {
|
||||
lastErr = xerrors.Errorf("failed to unseal deal %d: %w", deal.DealID, err)
|
||||
log.Warn(lastErr.Error())
|
||||
continue
|
||||
}
|
||||
|
||||
// Successfully fetched the deal data so return a reader over the data
|
||||
return reader, nil
|
||||
}
|
||||
|
||||
return nil, lastErr
|
||||
}
|
||||
|
||||
func (m *minerAPI) GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid) (uint64, error) {
|
||||
err := m.readyMgr.AwaitReady()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
pieceInfo, err := m.pieceStore.GetPieceInfo(pieceCid)
|
||||
if err != nil {
|
||||
return 0, xerrors.Errorf("failed to fetch pieceInfo for piece %s: %w", pieceCid, err)
|
||||
}
|
||||
|
||||
if len(pieceInfo.Deals) == 0 {
|
||||
return 0, xerrors.Errorf("no storage deals found for piece %s", pieceCid)
|
||||
}
|
||||
|
||||
len := pieceInfo.Deals[0].Length
|
||||
|
||||
return uint64(len), nil
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
// Code generated by MockGen. DO NOT EDIT.
|
||||
// Source: lotusaccessor.go
|
||||
// Source: markets/dagstore/lotusaccessor.go
|
||||
|
||||
// Package mock_dagstore is a generated GoMock package.
|
||||
package mock_dagstore
|
||||
@@ -31,6 +31,10 @@ func NewMockLotusAccessor(ctrl *gomock.Controller) *MockLotusAccessor {
|
||||
return mock
|
||||
}
|
||||
|
||||
func (mr *MockLotusAccessor) Start(_ context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// EXPECT returns an object that allows the caller to indicate expected use.
|
||||
func (m *MockLotusAccessor) EXPECT() *MockLotusAccessorMockRecorder {
|
||||
return m.recorder
|
||||
@@ -65,32 +69,3 @@ func (mr *MockLotusAccessorMockRecorder) GetUnpaddedCARSize(ctx, pieceCid interf
|
||||
mr.mock.ctrl.T.Helper()
|
||||
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetUnpaddedCARSize", reflect.TypeOf((*MockLotusAccessor)(nil).GetUnpaddedCARSize), ctx, pieceCid)
|
||||
}
|
||||
|
||||
// IsUnsealed mocks base method.
|
||||
func (m *MockLotusAccessor) IsUnsealed(ctx context.Context, pieceCid cid.Cid) (bool, error) {
|
||||
m.ctrl.T.Helper()
|
||||
ret := m.ctrl.Call(m, "IsUnsealed", ctx, pieceCid)
|
||||
ret0, _ := ret[0].(bool)
|
||||
ret1, _ := ret[1].(error)
|
||||
return ret0, ret1
|
||||
}
|
||||
|
||||
// IsUnsealed indicates an expected call of IsUnsealed.
|
||||
func (mr *MockLotusAccessorMockRecorder) IsUnsealed(ctx, pieceCid interface{}) *gomock.Call {
|
||||
mr.mock.ctrl.T.Helper()
|
||||
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "IsUnsealed", reflect.TypeOf((*MockLotusAccessor)(nil).IsUnsealed), ctx, pieceCid)
|
||||
}
|
||||
|
||||
// Start mocks base method.
|
||||
func (m *MockLotusAccessor) Start(ctx context.Context) error {
|
||||
m.ctrl.T.Helper()
|
||||
ret := m.ctrl.Call(m, "Start", ctx)
|
||||
ret0, _ := ret[0].(error)
|
||||
return ret0
|
||||
}
|
||||
|
||||
// Start indicates an expected call of Start.
|
||||
func (mr *MockLotusAccessorMockRecorder) Start(ctx interface{}) *gomock.Call {
|
||||
mr.mock.ctrl.T.Helper()
|
||||
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Start", reflect.TypeOf((*MockLotusAccessor)(nil).Start), ctx)
|
||||
}
|
||||
|
||||
+19
-24
@@ -15,29 +15,28 @@ const lotusScheme = "lotus"
|
||||
|
||||
var _ mount.Mount = (*LotusMount)(nil)
|
||||
|
||||
// mountTemplate returns a templated LotusMount containing the supplied API.
|
||||
//
|
||||
// It is called when registering a mount type with the mount registry
|
||||
// of the DAG store. It is used to reinstantiate mounts after a restart.
|
||||
//
|
||||
// When the registry needs to deserialize a mount it clones the template then
|
||||
// calls Deserialize on the cloned instance, which will have a reference to the
|
||||
// lotus mount API supplied here.
|
||||
func mountTemplate(api MinerAPI) *LotusMount {
|
||||
return &LotusMount{API: api}
|
||||
}
|
||||
|
||||
// LotusMount is a DAGStore mount implementation that fetches deal data
|
||||
// from a PieceCID.
|
||||
// LotusMount is the Lotus implementation of a Sharded DAG Store Mount.
|
||||
// A Filecoin Piece is treated as a Shard by this implementation.
|
||||
type LotusMount struct {
|
||||
API MinerAPI
|
||||
Api LotusAccessor
|
||||
PieceCid cid.Cid
|
||||
}
|
||||
|
||||
func NewLotusMount(pieceCid cid.Cid, api MinerAPI) (*LotusMount, error) {
|
||||
// NewLotusMountTemplate is called when registering a mount with
|
||||
// the DAG store registry.
|
||||
//
|
||||
// The DAG store registry receives an instance of the mount (a "template").
|
||||
// When the registry needs to deserialize a mount it clones the template then
|
||||
// calls Deserialize on the cloned instance, which will have a reference to the
|
||||
// lotus mount API supplied here.
|
||||
func NewLotusMountTemplate(api LotusAccessor) *LotusMount {
|
||||
return &LotusMount{Api: api}
|
||||
}
|
||||
|
||||
func NewLotusMount(pieceCid cid.Cid, api LotusAccessor) (*LotusMount, error) {
|
||||
return &LotusMount{
|
||||
PieceCid: pieceCid,
|
||||
API: api,
|
||||
Api: api,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -52,12 +51,13 @@ func (l *LotusMount) Deserialize(u *url.URL) error {
|
||||
if err != nil {
|
||||
return xerrors.Errorf("failed to parse PieceCid from host '%s': %w", u.Host, err)
|
||||
}
|
||||
|
||||
l.PieceCid = pieceCid
|
||||
return nil
|
||||
}
|
||||
|
||||
func (l *LotusMount) Fetch(ctx context.Context) (mount.Reader, error) {
|
||||
r, err := l.API.FetchUnsealedPiece(ctx, l.PieceCid)
|
||||
r, err := l.Api.FetchUnsealedPiece(ctx, l.PieceCid)
|
||||
if err != nil {
|
||||
return nil, xerrors.Errorf("failed to fetch unsealed piece %s: %w", l.PieceCid, err)
|
||||
}
|
||||
@@ -78,20 +78,15 @@ func (l *LotusMount) Close() error {
|
||||
}
|
||||
|
||||
func (l *LotusMount) Stat(ctx context.Context) (mount.Stat, error) {
|
||||
size, err := l.API.GetUnpaddedCARSize(ctx, l.PieceCid)
|
||||
size, err := l.Api.GetUnpaddedCARSize(ctx, l.PieceCid)
|
||||
if err != nil {
|
||||
return mount.Stat{}, xerrors.Errorf("failed to fetch piece size for piece %s: %w", l.PieceCid, err)
|
||||
}
|
||||
isUnsealed, err := l.API.IsUnsealed(ctx, l.PieceCid)
|
||||
if err != nil {
|
||||
return mount.Stat{}, xerrors.Errorf("failed to verify if we have the unsealed piece %s: %w", l.PieceCid, err)
|
||||
}
|
||||
|
||||
// TODO Mark false when storage deal expires.
|
||||
return mount.Stat{
|
||||
Exists: true,
|
||||
Size: int64(size),
|
||||
Ready: isUnsealed,
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -27,9 +27,6 @@ func TestLotusMount(t *testing.T) {
|
||||
|
||||
// create a mock lotus api that returns the reader we want
|
||||
mockLotusMountAPI := mock_dagstore.NewMockLotusAccessor(mockCtrl)
|
||||
|
||||
mockLotusMountAPI.EXPECT().IsUnsealed(gomock.Any(), cid).Return(true, nil).Times(1)
|
||||
|
||||
mockLotusMountAPI.EXPECT().FetchUnsealedPiece(gomock.Any(), cid).Return(&readCloser{ioutil.NopCloser(strings.NewReader("testing"))}, nil).Times(1)
|
||||
mockLotusMountAPI.EXPECT().FetchUnsealedPiece(gomock.Any(), cid).Return(&readCloser{ioutil.NopCloser(strings.NewReader("testing"))}, nil).Times(1)
|
||||
mockLotusMountAPI.EXPECT().GetUnpaddedCARSize(ctx, cid).Return(uint64(100), nil).Times(1)
|
||||
@@ -55,7 +52,7 @@ func TestLotusMount(t *testing.T) {
|
||||
// serialize url then deserialize from mount template -> should get back
|
||||
// the same mount
|
||||
url := mnt.Serialize()
|
||||
mnt2 := mountTemplate(mockLotusMountAPI)
|
||||
mnt2 := NewLotusMountTemplate(mockLotusMountAPI)
|
||||
err = mnt2.Deserialize(url)
|
||||
require.NoError(t, err)
|
||||
|
||||
@@ -69,7 +66,7 @@ func TestLotusMount(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestLotusMountDeserialize(t *testing.T) {
|
||||
api := &minerAPI{}
|
||||
api := &lotusAccessor{}
|
||||
|
||||
bgen := blocksutil.NewBlockGenerator()
|
||||
cid := bgen.Next().Cid()
|
||||
@@ -79,12 +76,12 @@ func TestLotusMountDeserialize(t *testing.T) {
|
||||
u, err := url.Parse(us)
|
||||
require.NoError(t, err)
|
||||
|
||||
mnt := mountTemplate(api)
|
||||
mnt := NewLotusMountTemplate(api)
|
||||
err = mnt.Deserialize(u)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Equal(t, cid, mnt.PieceCid)
|
||||
require.Equal(t, api, mnt.API)
|
||||
require.Equal(t, api, mnt.Api)
|
||||
|
||||
// fails if cid is not valid
|
||||
us = lotusScheme + "://" + "rand"
|
||||
@@ -111,16 +108,14 @@ func TestLotusMountRegistration(t *testing.T) {
|
||||
|
||||
mockLotusMountAPI := mock_dagstore.NewMockLotusAccessor(mockCtrl)
|
||||
registry := mount.NewRegistry()
|
||||
err = registry.Register(lotusScheme, mountTemplate(mockLotusMountAPI))
|
||||
err = registry.Register(lotusScheme, NewLotusMountTemplate(mockLotusMountAPI))
|
||||
require.NoError(t, err)
|
||||
|
||||
mnt, err := registry.Instantiate(u)
|
||||
require.NoError(t, err)
|
||||
|
||||
mockLotusMountAPI.EXPECT().IsUnsealed(ctx, cid).Return(true, nil)
|
||||
mockLotusMountAPI.EXPECT().GetUnpaddedCARSize(ctx, cid).Return(uint64(100), nil).Times(1)
|
||||
stat, err := mnt.Stat(context.Background())
|
||||
require.NoError(t, err)
|
||||
require.EqualValues(t, 100, stat.Size)
|
||||
require.True(t, stat.Ready)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
package dagstore
|
||||
|
||||
import (
|
||||
blocks "github.com/ipfs/go-block-format"
|
||||
"github.com/ipfs/go-cid"
|
||||
bstore "github.com/ipfs/go-ipfs-blockstore"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/dagstore"
|
||||
)
|
||||
|
||||
// ReadOnlyBlockstore stubs out Blockstore mutators with methods that error out
|
||||
type ReadOnlyBlockstore struct {
|
||||
dagstore.ReadBlockstore
|
||||
}
|
||||
|
||||
func NewReadOnlyBlockstore(rbs dagstore.ReadBlockstore) bstore.Blockstore {
|
||||
return ReadOnlyBlockstore{ReadBlockstore: rbs}
|
||||
}
|
||||
|
||||
func (r ReadOnlyBlockstore) DeleteBlock(c cid.Cid) error {
|
||||
return xerrors.Errorf("DeleteBlock called but not implemented")
|
||||
}
|
||||
|
||||
func (r ReadOnlyBlockstore) Put(block blocks.Block) error {
|
||||
return xerrors.Errorf("Put called but not implemented")
|
||||
}
|
||||
|
||||
func (r ReadOnlyBlockstore) PutMany(blocks []blocks.Block) error {
|
||||
return xerrors.Errorf("PutMany called but not implemented")
|
||||
}
|
||||
|
||||
var _ bstore.Blockstore = (*ReadOnlyBlockstore)(nil)
|
||||
+87
-252
@@ -3,202 +3,169 @@ package dagstore
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"io"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/filecoin-project/dagstore/index"
|
||||
"github.com/ipfs/go-cid"
|
||||
ds "github.com/ipfs/go-datastore"
|
||||
levelds "github.com/ipfs/go-ds-leveldb"
|
||||
measure "github.com/ipfs/go-ds-measure"
|
||||
bstore "github.com/ipfs/go-ipfs-blockstore"
|
||||
logging "github.com/ipfs/go-log/v2"
|
||||
ldbopts "github.com/syndtr/goleveldb/leveldb/opt"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/lotus/node/config"
|
||||
|
||||
"github.com/filecoin-project/go-statemachine/fsm"
|
||||
|
||||
"github.com/filecoin-project/dagstore"
|
||||
"github.com/filecoin-project/dagstore/index"
|
||||
"github.com/filecoin-project/dagstore/mount"
|
||||
"github.com/filecoin-project/dagstore/shard"
|
||||
|
||||
"github.com/filecoin-project/go-fil-markets/storagemarket"
|
||||
"github.com/filecoin-project/go-fil-markets/storagemarket/impl/providerstates"
|
||||
"github.com/filecoin-project/go-fil-markets/stores"
|
||||
"github.com/filecoin-project/go-fil-markets/carstore"
|
||||
"github.com/filecoin-project/go-fil-markets/shared"
|
||||
)
|
||||
|
||||
const (
|
||||
maxRecoverAttempts = 1
|
||||
shardRegMarker = ".shard-registration-complete"
|
||||
)
|
||||
const maxRecoverAttempts = 1
|
||||
|
||||
var log = logging.Logger("dagstore")
|
||||
var log = logging.Logger("dagstore-wrapper")
|
||||
|
||||
// MarketDAGStoreConfig is the config the market needs to then construct a DAG Store.
|
||||
type MarketDAGStoreConfig struct {
|
||||
TransientsDir string
|
||||
IndexDir string
|
||||
Datastore ds.Datastore
|
||||
MaxConcurrentIndex int
|
||||
GCInterval time.Duration
|
||||
}
|
||||
|
||||
// DAGStore provides an interface for the DAG store that can be mocked out
|
||||
// by tests
|
||||
type DAGStore interface {
|
||||
RegisterShard(ctx context.Context, key shard.Key, mnt mount.Mount, out chan dagstore.ShardResult, opts dagstore.RegisterOpts) error
|
||||
AcquireShard(ctx context.Context, key shard.Key, out chan dagstore.ShardResult, _ dagstore.AcquireOpts) error
|
||||
RecoverShard(ctx context.Context, key shard.Key, out chan dagstore.ShardResult, _ dagstore.RecoverOpts) error
|
||||
GC(ctx context.Context) (*dagstore.GCResult, error)
|
||||
Close() error
|
||||
}
|
||||
|
||||
type closableBlockstore struct {
|
||||
bstore.Blockstore
|
||||
io.Closer
|
||||
}
|
||||
|
||||
type Wrapper struct {
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
backgroundWg sync.WaitGroup
|
||||
|
||||
cfg config.DAGStoreConfig
|
||||
dagst dagstore.Interface
|
||||
minerAPI MinerAPI
|
||||
dagStore DAGStore
|
||||
mountApi LotusAccessor
|
||||
failureCh chan dagstore.ShardResult
|
||||
traceCh chan dagstore.Trace
|
||||
gcInterval time.Duration
|
||||
}
|
||||
|
||||
var _ stores.DAGStoreWrapper = (*Wrapper)(nil)
|
||||
var _ shared.DagStoreWrapper = (*Wrapper)(nil)
|
||||
|
||||
func NewDAGStore(cfg config.DAGStoreConfig, mountApi MinerAPI) (*dagstore.DAGStore, *Wrapper, error) {
|
||||
func NewDagStoreWrapper(cfg MarketDAGStoreConfig, mountApi LotusAccessor) (*Wrapper, error) {
|
||||
// construct the DAG Store.
|
||||
registry := mount.NewRegistry()
|
||||
if err := registry.Register(lotusScheme, mountTemplate(mountApi)); err != nil {
|
||||
return nil, nil, xerrors.Errorf("failed to create registry: %w", err)
|
||||
if err := registry.Register(lotusScheme, NewLotusMountTemplate(mountApi)); err != nil {
|
||||
return nil, xerrors.Errorf("failed to create registry: %w", err)
|
||||
}
|
||||
|
||||
// The dagstore will write Shard failures to the `failureCh` here.
|
||||
failureCh := make(chan dagstore.ShardResult, 1)
|
||||
|
||||
// The dagstore will write Trace events to the `traceCh` here.
|
||||
traceCh := make(chan dagstore.Trace, 32)
|
||||
|
||||
var (
|
||||
transientsDir = filepath.Join(cfg.RootDir, "transients")
|
||||
datastoreDir = filepath.Join(cfg.RootDir, "datastore")
|
||||
indexDir = filepath.Join(cfg.RootDir, "index")
|
||||
)
|
||||
|
||||
dstore, err := newDatastore(datastoreDir)
|
||||
irepo, err := index.NewFSRepo(cfg.IndexDir)
|
||||
if err != nil {
|
||||
return nil, nil, xerrors.Errorf("failed to create dagstore datastore in %s: %w", datastoreDir, err)
|
||||
}
|
||||
|
||||
irepo, err := index.NewFSRepo(indexDir)
|
||||
if err != nil {
|
||||
return nil, nil, xerrors.Errorf("failed to initialise dagstore index repo")
|
||||
return nil, xerrors.Errorf("failed to initialise dagstore index repo")
|
||||
}
|
||||
|
||||
dcfg := dagstore.Config{
|
||||
TransientsDir: transientsDir,
|
||||
TransientsDir: cfg.TransientsDir,
|
||||
IndexRepo: irepo,
|
||||
Datastore: dstore,
|
||||
Datastore: cfg.Datastore,
|
||||
MountRegistry: registry,
|
||||
FailureCh: failureCh,
|
||||
TraceCh: traceCh,
|
||||
// not limiting fetches globally, as the Lotus mount does
|
||||
// conditional throttling.
|
||||
MaxConcurrentIndex: cfg.MaxConcurrentIndex,
|
||||
MaxConcurrentReadyFetches: cfg.MaxConcurrentReadyFetches,
|
||||
RecoverOnStart: dagstore.RecoverOnAcquire,
|
||||
MaxConcurrentIndex: cfg.MaxConcurrentIndex,
|
||||
RecoverOnStart: dagstore.RecoverOnAcquire,
|
||||
}
|
||||
|
||||
dagst, err := dagstore.NewDAGStore(dcfg)
|
||||
dagStore, err := dagstore.NewDAGStore(dcfg)
|
||||
if err != nil {
|
||||
return nil, nil, xerrors.Errorf("failed to create DAG store: %w", err)
|
||||
return nil, xerrors.Errorf("failed to create DAG store: %w", err)
|
||||
}
|
||||
|
||||
w := &Wrapper{
|
||||
cfg: cfg,
|
||||
dagst: dagst,
|
||||
minerAPI: mountApi,
|
||||
return &Wrapper{
|
||||
dagStore: dagStore,
|
||||
mountApi: mountApi,
|
||||
failureCh: failureCh,
|
||||
traceCh: traceCh,
|
||||
gcInterval: time.Duration(cfg.GCInterval),
|
||||
}
|
||||
|
||||
return dagst, w, nil
|
||||
gcInterval: cfg.GCInterval,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// newDatastore creates a datastore under the given base directory
|
||||
// for dagstore metadata.
|
||||
func newDatastore(dir string) (ds.Batching, error) {
|
||||
// Create the datastore directory if it doesn't exist yet.
|
||||
if err := os.MkdirAll(dir, 0755); err != nil {
|
||||
return nil, xerrors.Errorf("failed to create directory %s for DAG store datastore: %w", dir, err)
|
||||
}
|
||||
|
||||
// Create a new LevelDB datastore
|
||||
dstore, err := levelds.NewDatastore(dir, &levelds.Options{
|
||||
Compression: ldbopts.NoCompression,
|
||||
NoSync: false,
|
||||
Strict: ldbopts.StrictAll,
|
||||
ReadOnly: false,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, xerrors.Errorf("failed to open datastore for DAG store: %w", err)
|
||||
}
|
||||
// Keep statistics about the datastore
|
||||
mds := measure.New("measure.", dstore)
|
||||
return mds, nil
|
||||
}
|
||||
|
||||
func (w *Wrapper) Start(ctx context.Context) error {
|
||||
w.ctx, w.cancel = context.WithCancel(ctx)
|
||||
func (ds *Wrapper) Start(ctx context.Context) {
|
||||
ds.ctx, ds.cancel = context.WithCancel(ctx)
|
||||
|
||||
// Run a go-routine to do DagStore GC.
|
||||
w.backgroundWg.Add(1)
|
||||
go w.gcLoop()
|
||||
ds.backgroundWg.Add(1)
|
||||
go ds.dagStoreGCLoop()
|
||||
|
||||
// run a go-routine to read the trace for debugging.
|
||||
w.backgroundWg.Add(1)
|
||||
go w.traceLoop()
|
||||
ds.backgroundWg.Add(1)
|
||||
go ds.traceLoop()
|
||||
|
||||
// Run a go-routine for shard recovery
|
||||
if dss, ok := w.dagst.(*dagstore.DAGStore); ok {
|
||||
w.backgroundWg.Add(1)
|
||||
go dagstore.RecoverImmediately(w.ctx, dss, w.failureCh, maxRecoverAttempts, w.backgroundWg.Done)
|
||||
if dss, ok := ds.dagStore.(*dagstore.DAGStore); ok {
|
||||
ds.backgroundWg.Add(1)
|
||||
go dagstore.RecoverImmediately(ds.ctx, dss, ds.failureCh, maxRecoverAttempts, ds.backgroundWg.Done)
|
||||
}
|
||||
|
||||
return w.dagst.Start(ctx)
|
||||
}
|
||||
|
||||
func (w *Wrapper) traceLoop() {
|
||||
defer w.backgroundWg.Done()
|
||||
func (ds *Wrapper) traceLoop() {
|
||||
defer ds.backgroundWg.Done()
|
||||
|
||||
for w.ctx.Err() == nil {
|
||||
for ds.ctx.Err() == nil {
|
||||
select {
|
||||
// Log trace events from the DAG store
|
||||
case tr := <-w.traceCh:
|
||||
case tr := <-ds.traceCh:
|
||||
log.Debugw("trace",
|
||||
"shard-key", tr.Key.String(),
|
||||
"op-type", tr.Op.String(),
|
||||
"after", tr.After.String())
|
||||
|
||||
case <-w.ctx.Done():
|
||||
case <-ds.ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Wrapper) gcLoop() {
|
||||
defer w.backgroundWg.Done()
|
||||
func (ds *Wrapper) dagStoreGCLoop() {
|
||||
defer ds.backgroundWg.Done()
|
||||
|
||||
ticker := time.NewTicker(w.gcInterval)
|
||||
defer ticker.Stop()
|
||||
gcTicker := time.NewTicker(ds.gcInterval)
|
||||
defer gcTicker.Stop()
|
||||
|
||||
for w.ctx.Err() == nil {
|
||||
for ds.ctx.Err() == nil {
|
||||
select {
|
||||
// GC the DAG store on every tick
|
||||
case <-ticker.C:
|
||||
_, _ = w.dagst.GC(w.ctx)
|
||||
case <-gcTicker.C:
|
||||
_, _ = ds.dagStore.GC(ds.ctx)
|
||||
|
||||
// Exit when the DAG store wrapper is shutdown
|
||||
case <-w.ctx.Done():
|
||||
case <-ds.ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Wrapper) LoadShard(ctx context.Context, pieceCid cid.Cid) (stores.ClosableBlockstore, error) {
|
||||
func (ds *Wrapper) LoadShard(ctx context.Context, pieceCid cid.Cid) (carstore.ClosableBlockstore, error) {
|
||||
log.Debugf("acquiring shard for piece CID %s", pieceCid)
|
||||
|
||||
key := shard.KeyFromCID(pieceCid)
|
||||
resch := make(chan dagstore.ShardResult, 1)
|
||||
err := w.dagst.AcquireShard(ctx, key, resch, dagstore.AcquireOpts{})
|
||||
err := ds.dagStore.AcquireShard(ctx, key, resch, dagstore.AcquireOpts{})
|
||||
log.Debugf("sent message to acquire shard for piece CID %s", pieceCid)
|
||||
|
||||
if err != nil {
|
||||
@@ -213,13 +180,13 @@ func (w *Wrapper) LoadShard(ctx context.Context, pieceCid cid.Cid) (stores.Closa
|
||||
// we already have a transient file. However, we don't have it here
|
||||
// and therefore we pass an empty file path.
|
||||
carPath := ""
|
||||
if err := stores.RegisterShardSync(ctx, w, pieceCid, carPath, false); err != nil {
|
||||
if err := shared.RegisterShardSync(ctx, ds, pieceCid, carPath, false); err != nil {
|
||||
return nil, xerrors.Errorf("failed to re-register shard during loading piece CID %s: %w", pieceCid, err)
|
||||
}
|
||||
log.Warnw("successfully re-registered shard", "pieceCID", pieceCid)
|
||||
|
||||
resch = make(chan dagstore.ShardResult, 1)
|
||||
if err := w.dagst.AcquireShard(ctx, key, resch, dagstore.AcquireOpts{}); err != nil {
|
||||
if err := ds.dagStore.AcquireShard(ctx, key, resch, dagstore.AcquireOpts{}); err != nil {
|
||||
return nil, xerrors.Errorf("failed to acquire Shard for piece CID %s after re-registering: %w", pieceCid, err)
|
||||
}
|
||||
}
|
||||
@@ -244,13 +211,13 @@ func (w *Wrapper) LoadShard(ctx context.Context, pieceCid cid.Cid) (stores.Closa
|
||||
}
|
||||
|
||||
log.Debugf("successfully loaded blockstore for piece CID %s", pieceCid)
|
||||
return &Blockstore{ReadBlockstore: bs, Closer: res.Accessor}, nil
|
||||
return &closableBlockstore{Blockstore: NewReadOnlyBlockstore(bs), Closer: res.Accessor}, nil
|
||||
}
|
||||
|
||||
func (w *Wrapper) RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath string, eagerInit bool, resch chan dagstore.ShardResult) error {
|
||||
func (ds *Wrapper) RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath string, eagerInit bool, resch chan dagstore.ShardResult) error {
|
||||
// Create a lotus mount with the piece CID
|
||||
key := shard.KeyFromCID(pieceCid)
|
||||
mt, err := NewLotusMount(pieceCid, w.minerAPI)
|
||||
mt, err := NewLotusMount(pieceCid, ds.mountApi)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("failed to create lotus mount for piece CID %s: %w", pieceCid, err)
|
||||
}
|
||||
@@ -260,7 +227,7 @@ func (w *Wrapper) RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath s
|
||||
ExistingTransient: carPath,
|
||||
LazyInitialization: !eagerInit,
|
||||
}
|
||||
err = w.dagst.RegisterShard(ctx, key, mt, resch, opts)
|
||||
err = ds.dagStore.RegisterShard(ctx, key, mt, resch, opts)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("failed to schedule register shard for piece CID %s: %w", pieceCid, err)
|
||||
}
|
||||
@@ -269,153 +236,21 @@ func (w *Wrapper) RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath s
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *Wrapper) MigrateDeals(ctx context.Context, deals []storagemarket.MinerDeal) (bool, error) {
|
||||
log := log.Named("migrator")
|
||||
|
||||
// Check if all deals have already been registered as shards
|
||||
isComplete, err := w.registrationComplete()
|
||||
if err != nil {
|
||||
return false, xerrors.Errorf("failed to get dagstore migration status: %w", err)
|
||||
}
|
||||
if isComplete {
|
||||
// All deals have been registered as shards, bail out
|
||||
log.Info("no shard migration necessary; already marked complete")
|
||||
return false, nil
|
||||
}
|
||||
|
||||
log.Infow("registering shards for all active deals in sealing subsystem", "count", len(deals))
|
||||
|
||||
inSealingSubsystem := make(map[fsm.StateKey]struct{}, len(providerstates.StatesKnownBySealingSubsystem))
|
||||
for _, s := range providerstates.StatesKnownBySealingSubsystem {
|
||||
inSealingSubsystem[s] = struct{}{}
|
||||
}
|
||||
|
||||
// channel where results will be received, and channel where the total
|
||||
// number of registered shards will be sent.
|
||||
resch := make(chan dagstore.ShardResult, 32)
|
||||
totalCh := make(chan int)
|
||||
doneCh := make(chan struct{})
|
||||
|
||||
// Start making progress consuming results. We won't know how many to
|
||||
// actually consume until we register all shards.
|
||||
//
|
||||
// If there are any problems registering shards, just log an error
|
||||
go func() {
|
||||
defer close(doneCh)
|
||||
|
||||
var total = math.MaxInt64
|
||||
var res dagstore.ShardResult
|
||||
for rcvd := 0; rcvd < total; {
|
||||
select {
|
||||
case total = <-totalCh:
|
||||
// we now know the total number of registered shards
|
||||
// nullify so that we no longer consume from it after closed.
|
||||
close(totalCh)
|
||||
totalCh = nil
|
||||
case res = <-resch:
|
||||
rcvd++
|
||||
if res.Error == nil {
|
||||
log.Infow("async shard registration completed successfully", "shard_key", res.Key)
|
||||
} else {
|
||||
log.Warnw("async shard registration failed", "shard_key", res.Key, "error", res.Error)
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
// Filter for deals that are handed off.
|
||||
//
|
||||
// If the deal has not yet been handed off to the sealing subsystem, we
|
||||
// don't need to call RegisterShard in this migration; RegisterShard will
|
||||
// be called in the new code once the deal reaches the state where it's
|
||||
// handed off to the sealing subsystem.
|
||||
var registered int
|
||||
for _, deal := range deals {
|
||||
if deal.Ref.PieceCid == nil {
|
||||
log.Warnw("deal has nil piece CID; skipping", "deal_id", deal.DealID)
|
||||
continue
|
||||
}
|
||||
|
||||
// enrich log statements in this iteration with deal ID and piece CID.
|
||||
log := log.With("deal_id", deal.DealID, "piece_cid", deal.Ref.PieceCid)
|
||||
|
||||
// Filter for deals that have been handed off to the sealing subsystem
|
||||
if _, ok := inSealingSubsystem[deal.State]; !ok {
|
||||
log.Infow("deal not ready; skipping")
|
||||
continue
|
||||
}
|
||||
|
||||
log.Infow("registering deal in dagstore with lazy init")
|
||||
|
||||
// Register the deal as a shard with the DAG store with lazy initialization.
|
||||
// The index will be populated the first time the deal is retrieved, or
|
||||
// through the bulk initialization script.
|
||||
err = w.RegisterShard(ctx, *deal.Ref.PieceCid, "", false, resch)
|
||||
if err != nil {
|
||||
log.Warnw("failed to register shard", "error", err)
|
||||
continue
|
||||
}
|
||||
registered++
|
||||
}
|
||||
|
||||
log.Infow("finished registering all shards", "total", registered)
|
||||
totalCh <- registered
|
||||
<-doneCh
|
||||
|
||||
log.Infow("confirmed registration of all shards")
|
||||
|
||||
// Completed registering all shards, so mark the migration as complete
|
||||
err = w.markRegistrationComplete()
|
||||
if err != nil {
|
||||
log.Errorf("failed to mark shards as registered: %s", err)
|
||||
} else {
|
||||
log.Info("successfully marked migration as complete")
|
||||
}
|
||||
|
||||
log.Infow("dagstore migration complete")
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// Check for the existence of a "marker" file indicating that the migration
|
||||
// has completed
|
||||
func (w *Wrapper) registrationComplete() (bool, error) {
|
||||
path := filepath.Join(w.cfg.RootDir, shardRegMarker)
|
||||
_, err := os.Stat(path)
|
||||
if os.IsNotExist(err) {
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// Create a "marker" file indicating that the migration has completed
|
||||
func (w *Wrapper) markRegistrationComplete() error {
|
||||
path := filepath.Join(w.cfg.RootDir, shardRegMarker)
|
||||
file, err := os.Create(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return file.Close()
|
||||
}
|
||||
|
||||
func (w *Wrapper) Close() error {
|
||||
func (ds *Wrapper) Close() error {
|
||||
// Cancel the context
|
||||
w.cancel()
|
||||
ds.cancel()
|
||||
|
||||
// Close the DAG store
|
||||
log.Info("will close the dagstore")
|
||||
if err := w.dagst.Close(); err != nil {
|
||||
return xerrors.Errorf("failed to close dagstore: %w", err)
|
||||
if err := ds.dagStore.Close(); err != nil {
|
||||
return xerrors.Errorf("failed to close DAG store: %w", err)
|
||||
}
|
||||
log.Info("dagstore closed")
|
||||
|
||||
// Wait for the background go routine to exit
|
||||
log.Info("waiting for dagstore background wrapper goroutines to exit")
|
||||
w.backgroundWg.Wait()
|
||||
log.Info("exited dagstore background wrapper goroutines")
|
||||
log.Info("waiting for dagstore background wrapper routines to exist")
|
||||
ds.backgroundWg.Wait()
|
||||
log.Info("exited dagstore background warpper routines")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
package dagstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/filecoin-project/dagstore"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/filecoin-project/go-state-types/abi"
|
||||
|
||||
"github.com/filecoin-project/go-fil-markets/retrievalmarket/impl/testnodes"
|
||||
tut "github.com/filecoin-project/go-fil-markets/shared_testutil"
|
||||
"github.com/filecoin-project/go-fil-markets/storagemarket"
|
||||
"github.com/filecoin-project/lotus/node/config"
|
||||
)
|
||||
|
||||
func TestShardRegistration(t *testing.T) {
|
||||
ps := tut.NewTestPieceStore()
|
||||
providerNode := testnodes.NewTestRetrievalProviderNode()
|
||||
|
||||
ctx := context.Background()
|
||||
cids := tut.GenerateCids(4)
|
||||
pieceCidUnsealed := cids[0]
|
||||
pieceCidSealed := cids[1]
|
||||
pieceCidUnsealed2 := cids[2]
|
||||
pieceCidUnsealed3 := cids[3]
|
||||
|
||||
sealedSector := abi.SectorNumber(1)
|
||||
unsealedSector1 := abi.SectorNumber(2)
|
||||
unsealedSector2 := abi.SectorNumber(3)
|
||||
unsealedSector3 := abi.SectorNumber(4)
|
||||
|
||||
// ps.ExpectPiece(pieceCidUnsealed, piecestore.PieceInfo{
|
||||
// PieceCID: pieceCidUnsealed,
|
||||
// Deals: []piecestore.DealInfo{
|
||||
// {
|
||||
// SectorID: unsealedSector1,
|
||||
// },
|
||||
// },
|
||||
// })
|
||||
//
|
||||
// ps.ExpectPiece(pieceCidSealed, piecestore.PieceInfo{
|
||||
// PieceCID: pieceCidSealed,
|
||||
// Deals: []piecestore.DealInfo{
|
||||
// {
|
||||
// SectorID: sealedSector,
|
||||
// },
|
||||
// },
|
||||
// })
|
||||
|
||||
deals := []storagemarket.MinerDeal{{
|
||||
// Should be registered
|
||||
State: storagemarket.StorageDealSealing,
|
||||
SectorNumber: unsealedSector1,
|
||||
Ref: &storagemarket.DataRef{
|
||||
PieceCid: &pieceCidUnsealed,
|
||||
},
|
||||
}, {
|
||||
// Should be registered with lazy registration (because sector is sealed)
|
||||
State: storagemarket.StorageDealSealing,
|
||||
SectorNumber: sealedSector,
|
||||
Ref: &storagemarket.DataRef{
|
||||
PieceCid: &pieceCidSealed,
|
||||
},
|
||||
}, {
|
||||
// Should be ignored because deal is no longer active
|
||||
State: storagemarket.StorageDealError,
|
||||
SectorNumber: unsealedSector2,
|
||||
Ref: &storagemarket.DataRef{
|
||||
PieceCid: &pieceCidUnsealed2,
|
||||
},
|
||||
}, {
|
||||
// Should be ignored because deal is not yet sealing
|
||||
State: storagemarket.StorageDealFundsReserved,
|
||||
SectorNumber: unsealedSector3,
|
||||
Ref: &storagemarket.DataRef{
|
||||
PieceCid: &pieceCidUnsealed3,
|
||||
},
|
||||
}}
|
||||
|
||||
cfg := config.DefaultStorageMiner().DAGStore
|
||||
cfg.RootDir = t.TempDir()
|
||||
|
||||
mapi := NewMinerAPI(ps, providerNode, 10)
|
||||
dagst, w, err := NewDAGStore(cfg, mapi)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, dagst)
|
||||
require.NotNil(t, w)
|
||||
|
||||
err = dagst.Start(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
migrated, err := w.MigrateDeals(ctx, deals)
|
||||
require.True(t, migrated)
|
||||
require.NoError(t, err)
|
||||
|
||||
info := dagst.AllShardsInfo()
|
||||
require.Len(t, info, 2)
|
||||
for _, i := range info {
|
||||
require.Equal(t, dagstore.ShardStateNew, i.ShardState)
|
||||
}
|
||||
|
||||
// Run register shard migration again
|
||||
migrated, err = w.MigrateDeals(ctx, deals)
|
||||
require.False(t, migrated)
|
||||
require.NoError(t, err)
|
||||
|
||||
// ps.VerifyExpectations(t)
|
||||
}
|
||||
@@ -10,8 +10,6 @@ import (
|
||||
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/lotus/node/config"
|
||||
|
||||
"github.com/filecoin-project/dagstore"
|
||||
"github.com/filecoin-project/dagstore/mount"
|
||||
"github.com/filecoin-project/dagstore/shard"
|
||||
@@ -28,14 +26,13 @@ func TestWrapperAcquireRecovery(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create a DAG store wrapper
|
||||
dagst, w, err := NewDAGStore(config.DAGStoreConfig{
|
||||
RootDir: t.TempDir(),
|
||||
GCInterval: config.Duration(1 * time.Millisecond),
|
||||
w, err := NewDagStoreWrapper(MarketDAGStoreConfig{
|
||||
TransientsDir: t.TempDir(),
|
||||
IndexDir: t.TempDir(),
|
||||
GCInterval: time.Millisecond,
|
||||
}, mockLotusMount{})
|
||||
require.NoError(t, err)
|
||||
|
||||
defer dagst.Close() //nolint:errcheck
|
||||
|
||||
// Return an error from acquire shard the first time
|
||||
acquireShardErr := make(chan error, 1)
|
||||
acquireShardErr <- xerrors.Errorf("unknown shard: %w", dagstore.ErrShardUnknown)
|
||||
@@ -48,7 +45,7 @@ func TestWrapperAcquireRecovery(t *testing.T) {
|
||||
},
|
||||
register: make(chan shard.Key, 1),
|
||||
}
|
||||
w.dagst = mock
|
||||
w.dagStore = mock
|
||||
|
||||
mybs, err := w.LoadShard(ctx, pieceCid)
|
||||
require.NoError(t, err)
|
||||
@@ -79,25 +76,23 @@ func TestWrapperBackground(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a DAG store wrapper
|
||||
dagst, w, err := NewDAGStore(config.DAGStoreConfig{
|
||||
RootDir: t.TempDir(),
|
||||
GCInterval: config.Duration(1 * time.Millisecond),
|
||||
w, err := NewDagStoreWrapper(MarketDAGStoreConfig{
|
||||
TransientsDir: t.TempDir(),
|
||||
IndexDir: t.TempDir(),
|
||||
GCInterval: time.Millisecond,
|
||||
}, mockLotusMount{})
|
||||
require.NoError(t, err)
|
||||
|
||||
defer dagst.Close() //nolint:errcheck
|
||||
|
||||
// Create a mock DAG store in place of the real DAG store
|
||||
mock := &mockDagStore{
|
||||
gc: make(chan struct{}, 1),
|
||||
recover: make(chan shard.Key, 1),
|
||||
close: make(chan struct{}, 1),
|
||||
}
|
||||
w.dagst = mock
|
||||
w.dagStore = mock
|
||||
|
||||
// Start up the wrapper
|
||||
err = w.Start(ctx)
|
||||
require.NoError(t, err)
|
||||
w.Start(ctx)
|
||||
|
||||
// Expect GC to be called automatically
|
||||
tctx, cancel := context.WithTimeout(ctx, time.Second)
|
||||
@@ -132,22 +127,6 @@ type mockDagStore struct {
|
||||
close chan struct{}
|
||||
}
|
||||
|
||||
func (m *mockDagStore) DestroyShard(ctx context.Context, key shard.Key, out chan dagstore.ShardResult, _ dagstore.DestroyOpts) error {
|
||||
panic("implement me")
|
||||
}
|
||||
|
||||
func (m *mockDagStore) GetShardInfo(k shard.Key) (dagstore.ShardInfo, error) {
|
||||
panic("implement me")
|
||||
}
|
||||
|
||||
func (m *mockDagStore) AllShardsInfo() dagstore.AllShardsInfo {
|
||||
panic("implement me")
|
||||
}
|
||||
|
||||
func (m *mockDagStore) Start(_ context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *mockDagStore) RegisterShard(ctx context.Context, key shard.Key, mnt mount.Mount, out chan dagstore.ShardResult, opts dagstore.RegisterOpts) error {
|
||||
m.register <- key
|
||||
out <- dagstore.ShardResult{Key: key}
|
||||
@@ -199,10 +178,6 @@ func (m mockLotusMount) GetUnpaddedCARSize(ctx context.Context, pieceCid cid.Cid
|
||||
panic("implement me")
|
||||
}
|
||||
|
||||
func (m mockLotusMount) IsUnsealed(ctx context.Context, pieceCid cid.Cid) (bool, error) {
|
||||
panic("implement me")
|
||||
}
|
||||
|
||||
func getShardAccessor(t *testing.T) *dagstore.ShardAccessor {
|
||||
data, err := os.ReadFile("./fixtures/sample-rw-bs-v2.car")
|
||||
require.NoError(t, err)
|
||||
|
||||
@@ -66,7 +66,6 @@ var (
|
||||
AutoNATSvcKey = special{10} // Libp2p option
|
||||
BandwidthReporterKey = special{11} // Libp2p option
|
||||
ConnGaterKey = special{12} // libp2p option
|
||||
DAGStoreKey = special{13} // constructor returns multiple values
|
||||
)
|
||||
|
||||
type invoke int
|
||||
|
||||
@@ -73,7 +73,6 @@ func ConfigStorageMiner(c interface{}) Option {
|
||||
return Options(
|
||||
ConfigCommon(&cfg.Common, enableLibp2pNode),
|
||||
|
||||
Override(new(api.MinerSubsystems), modules.ExtractEnabledMinerSubsystems(cfg.Subsystems)),
|
||||
Override(new(stores.LocalStorage), From(new(repo.LockedRepo))),
|
||||
Override(new(*stores.Local), modules.LocalStorage),
|
||||
Override(new(*stores.Remote), modules.RemoteStorage),
|
||||
@@ -148,8 +147,8 @@ func ConfigStorageMiner(c interface{}) Option {
|
||||
Override(new(dtypes.RetrievalPricingFunc), modules.RetrievalPricingFunc(cfg.Dealmaking)),
|
||||
|
||||
// DAG Store
|
||||
Override(new(dagstore.MinerAPI), modules.NewMinerAPI),
|
||||
Override(DAGStoreKey, modules.DAGStore),
|
||||
Override(new(dagstore.LotusAccessor), modules.NewLotusAccessor),
|
||||
Override(new(*dagstore.Wrapper), modules.DagStoreWrapper),
|
||||
|
||||
// Markets (retrieval)
|
||||
Override(new(retrievalmarket.RetrievalProviderNode), retrievaladapter.NewRetrievalProviderNode),
|
||||
|
||||
@@ -188,12 +188,6 @@ func DefaultStorageMiner() *StorageMiner {
|
||||
TerminateControl: []string{},
|
||||
DealPublishControl: []string{},
|
||||
},
|
||||
|
||||
DAGStore: DAGStoreConfig{
|
||||
MaxConcurrentIndex: 5,
|
||||
MaxConcurrencyStorageCalls: 100,
|
||||
GCInterval: Duration(1 * time.Minute),
|
||||
},
|
||||
}
|
||||
cfg.Common.API.ListenAddress = "/ip4/127.0.0.1/tcp/2345/http"
|
||||
cfg.Common.API.RemoteListenAddress = "127.0.0.1:2345"
|
||||
|
||||
@@ -125,46 +125,6 @@ and storage providers`,
|
||||
Comment: ``,
|
||||
},
|
||||
},
|
||||
"DAGStoreConfig": []DocField{
|
||||
{
|
||||
Name: "RootDir",
|
||||
Type: "string",
|
||||
|
||||
Comment: `Default value: $LOTUS_MARKETS_PATH/dagStore (split deployment) or
|
||||
$LOTUS_MINER_PATH/dagStore (monolith deployment)`,
|
||||
},
|
||||
{
|
||||
Name: "MaxConcurrentIndex",
|
||||
Type: "int",
|
||||
|
||||
Comment: `The maximum amount of indexing jobs that can run simultaneously.
|
||||
Default value: 5.`,
|
||||
},
|
||||
{
|
||||
Name: "MaxConcurrentReadyFetches",
|
||||
Type: "int",
|
||||
|
||||
Comment: `The maximum amount of unsealed deals that can be fetched simultaneously
|
||||
from the storage subsystem.
|
||||
Default value: 2.`,
|
||||
},
|
||||
{
|
||||
Name: "MaxConcurrencyStorageCalls",
|
||||
Type: "int",
|
||||
|
||||
Comment: `The maximum number of simultaneous inflight API calls to the storage
|
||||
subsystem.
|
||||
Default value: 100.`,
|
||||
},
|
||||
{
|
||||
Name: "GCInterval",
|
||||
Type: "Duration",
|
||||
|
||||
Comment: `The time between calls to periodic dagstore GC, in time.Duration string
|
||||
representation, e.g. 1m, 5m, 1h.
|
||||
Default value: 1 minute.`,
|
||||
},
|
||||
},
|
||||
"DealmakingConfig": []DocField{
|
||||
{
|
||||
Name: "ConsiderOnlineStorageDeals",
|
||||
@@ -793,12 +753,6 @@ Default is 20 (about once a week).`,
|
||||
Name: "Addresses",
|
||||
Type: "MinerAddressConfig",
|
||||
|
||||
Comment: ``,
|
||||
},
|
||||
{
|
||||
Name: "DAGStore",
|
||||
Type: "DAGStoreConfig",
|
||||
|
||||
Comment: ``,
|
||||
},
|
||||
},
|
||||
|
||||
@@ -50,42 +50,6 @@ type StorageMiner struct {
|
||||
Storage sectorstorage.SealerConfig
|
||||
Fees MinerFeeConfig
|
||||
Addresses MinerAddressConfig
|
||||
DAGStore DAGStoreConfig
|
||||
}
|
||||
|
||||
type DAGStoreConfig struct {
|
||||
// Path to the dagstore root directory. This directory contains three
|
||||
// subdirectories, which can be symlinked to alternative locations if
|
||||
// need be:
|
||||
//
|
||||
// - ./transients: caches unsealed deals that have been fetched from the
|
||||
// storage subsystem for serving retrievals.
|
||||
// - ./indices: stores shard indices.
|
||||
// - ./datastore: holds the KV store tracking the state of every shard
|
||||
// known to the DAG store.
|
||||
//
|
||||
// Default value: $LOTUS_MARKETS_PATH/dagStore (split deployment) or
|
||||
// $LOTUS_MINER_PATH/dagStore (monolith deployment)
|
||||
RootDir string
|
||||
|
||||
// The maximum amount of indexing jobs that can run simultaneously.
|
||||
// Default value: 5.
|
||||
MaxConcurrentIndex int
|
||||
|
||||
// The maximum amount of unsealed deals that can be fetched simultaneously
|
||||
// from the storage subsystem.
|
||||
// Default value: 2.
|
||||
MaxConcurrentReadyFetches int
|
||||
|
||||
// The maximum number of simultaneous inflight API calls to the storage
|
||||
// subsystem.
|
||||
// Default value: 100.
|
||||
MaxConcurrencyStorageCalls int
|
||||
|
||||
// The time between calls to periodic dagstore GC, in time.Duration string
|
||||
// representation, e.g. 1m, 5m, 1h.
|
||||
// Default value: 1 minute.
|
||||
GCInterval Duration
|
||||
}
|
||||
|
||||
type MinerSubsystemConfig struct {
|
||||
|
||||
+34
-33
@@ -10,7 +10,8 @@ import (
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/ipld/go-car"
|
||||
"github.com/filecoin-project/go-fil-markets/filestorecaradapter"
|
||||
"github.com/filecoin-project/lotus/chain/actors/builtin/miner"
|
||||
carv2 "github.com/ipld/go-car/v2"
|
||||
"github.com/ipld/go-car/v2/blockstore"
|
||||
|
||||
@@ -25,7 +26,7 @@ import (
|
||||
files "github.com/ipfs/go-ipfs-files"
|
||||
"github.com/ipfs/go-merkledag"
|
||||
unixfile "github.com/ipfs/go-unixfs/file"
|
||||
|
||||
"github.com/ipld/go-car"
|
||||
basicnode "github.com/ipld/go-ipld-prime/node/basic"
|
||||
"github.com/ipld/go-ipld-prime/traversal/selector"
|
||||
"github.com/ipld/go-ipld-prime/traversal/selector/builder"
|
||||
@@ -46,7 +47,6 @@ import (
|
||||
"github.com/filecoin-project/go-fil-markets/shared"
|
||||
"github.com/filecoin-project/go-fil-markets/storagemarket"
|
||||
"github.com/filecoin-project/go-fil-markets/storagemarket/network"
|
||||
"github.com/filecoin-project/go-fil-markets/stores"
|
||||
"github.com/filecoin-project/go-state-types/abi"
|
||||
"github.com/filecoin-project/specs-actors/v3/actors/builtin/market"
|
||||
|
||||
@@ -54,7 +54,6 @@ import (
|
||||
|
||||
"github.com/filecoin-project/lotus/api"
|
||||
"github.com/filecoin-project/lotus/build"
|
||||
"github.com/filecoin-project/lotus/chain/actors/builtin/miner"
|
||||
"github.com/filecoin-project/lotus/chain/store"
|
||||
"github.com/filecoin-project/lotus/chain/types"
|
||||
"github.com/filecoin-project/lotus/markets/utils"
|
||||
@@ -189,17 +188,17 @@ func (a *API) dealStarter(ctx context.Context, params *api.StartDealParams, isSt
|
||||
providerInfo := utils.NewStorageProviderInfo(params.Miner, mi.Worker, mi.SectorSize, *mi.PeerId, mi.Multiaddrs)
|
||||
|
||||
result, err := a.SMDealClient.ProposeStorageDeal(ctx, storagemarket.ProposeStorageDealParams{
|
||||
Addr: params.Wallet,
|
||||
Info: &providerInfo,
|
||||
Data: params.Data,
|
||||
StartEpoch: dealStart,
|
||||
EndEpoch: calcDealExpiration(params.MinBlocksDuration, md, dealStart),
|
||||
Price: params.EpochPrice,
|
||||
Collateral: params.ProviderCollateral,
|
||||
Rt: st,
|
||||
FastRetrieval: params.FastRetrieval,
|
||||
VerifiedDeal: params.VerifiedDeal,
|
||||
IndexedCAR: CARV2FilePath,
|
||||
Addr: params.Wallet,
|
||||
Info: &providerInfo,
|
||||
Data: params.Data,
|
||||
StartEpoch: dealStart,
|
||||
EndEpoch: calcDealExpiration(params.MinBlocksDuration, md, dealStart),
|
||||
Price: params.EpochPrice,
|
||||
Collateral: params.ProviderCollateral,
|
||||
Rt: st,
|
||||
FastRetrieval: params.FastRetrieval,
|
||||
VerifiedDeal: params.VerifiedDeal,
|
||||
FilestoreCARv2FilePath: CARV2FilePath,
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
@@ -520,7 +519,7 @@ func (a *API) ClientImport(ctx context.Context, ref api.FileRef) (res *api.Impor
|
||||
}
|
||||
}()
|
||||
|
||||
root, err = a.doImport(ctx, ref.Path, carFile)
|
||||
root, err = a.importNormalFileToFilestoreCARv2(ctx, id, ref.Path, carFile)
|
||||
if err != nil {
|
||||
return nil, xerrors.Errorf("failed to import normal file to CARv2: %w", err)
|
||||
}
|
||||
@@ -1032,21 +1031,21 @@ func (w *lenWriter) Write(p []byte) (n int, err error) {
|
||||
}
|
||||
|
||||
func (a *API) ClientDealSize(ctx context.Context, root cid.Cid) (api.DataSize, error) {
|
||||
path, err := a.imgr().FilestoreCARV2FilePathFor(root)
|
||||
fc, err := a.imgr().FilestoreCARV2FilePathFor(root)
|
||||
if err != nil {
|
||||
return api.DataSize{}, xerrors.Errorf("failed to find CARv2 file for root: %w", err)
|
||||
}
|
||||
if len(path) == 0 {
|
||||
if len(fc) == 0 {
|
||||
return api.DataSize{}, xerrors.New("no CARv2 file for root")
|
||||
}
|
||||
|
||||
fs, err := stores.ReadOnlyFilestore(path)
|
||||
rdOnly, err := filestorecaradapter.NewReadOnlyFileStore(fc)
|
||||
if err != nil {
|
||||
return api.DataSize{}, xerrors.Errorf("failed to open filestore from carv2 in path %s: %w", path, err)
|
||||
return api.DataSize{}, xerrors.Errorf("failed to open read only filestore: %w", err)
|
||||
}
|
||||
defer fs.Close() //nolint:errcheck
|
||||
defer rdOnly.Close() //nolint:errcheck
|
||||
|
||||
dag := merkledag.NewDAGService(blockservice.New(fs, offline.Exchange(fs)))
|
||||
dag := merkledag.NewDAGService(blockservice.New(rdOnly, offline.Exchange(rdOnly)))
|
||||
w := lenWriter(0)
|
||||
|
||||
err = car.WriteCar(ctx, dag, []cid.Cid{root}, &w)
|
||||
@@ -1063,21 +1062,21 @@ func (a *API) ClientDealSize(ctx context.Context, root cid.Cid) (api.DataSize, e
|
||||
}
|
||||
|
||||
func (a *API) ClientDealPieceCID(ctx context.Context, root cid.Cid) (api.DataCIDSize, error) {
|
||||
path, err := a.imgr().FilestoreCARV2FilePathFor(root)
|
||||
fc, err := a.imgr().FilestoreCARV2FilePathFor(root)
|
||||
if err != nil {
|
||||
return api.DataCIDSize{}, xerrors.Errorf("failed to find CARv2 file for root: %w", err)
|
||||
}
|
||||
if len(path) == 0 {
|
||||
if len(fc) == 0 {
|
||||
return api.DataCIDSize{}, xerrors.New("no CARv2 file for root")
|
||||
}
|
||||
|
||||
fs, err := stores.ReadOnlyFilestore(path)
|
||||
rdOnly, err := filestorecaradapter.NewReadOnlyFileStore(fc)
|
||||
if err != nil {
|
||||
return api.DataCIDSize{}, xerrors.Errorf("failed to open filestore from carv2 in path %s: %w", path, err)
|
||||
return api.DataCIDSize{}, xerrors.Errorf("failed to open read only blockstore: %w", err)
|
||||
}
|
||||
defer fs.Close() //nolint:errcheck
|
||||
defer rdOnly.Close() //nolint:errcheck
|
||||
|
||||
dag := merkledag.NewDAGService(blockservice.New(fs, offline.Exchange(fs)))
|
||||
dag := merkledag.NewDAGService(blockservice.New(rdOnly, offline.Exchange(rdOnly)))
|
||||
w := &writer.Writer{}
|
||||
bw := bufio.NewWriterSize(w, int(writer.CommPBuf))
|
||||
|
||||
@@ -1096,20 +1095,22 @@ func (a *API) ClientDealPieceCID(ctx context.Context, root cid.Cid) (api.DataCID
|
||||
|
||||
func (a *API) ClientGenCar(ctx context.Context, ref api.FileRef, outputPath string) error {
|
||||
id := importmgr.ImportID(rand.Uint64())
|
||||
tmp, err := a.imgr().NewTempFile(id)
|
||||
tmpCARv2File, err := a.imgr().NewTempFile(id)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("failed to create temp file: %w", err)
|
||||
}
|
||||
defer os.Remove(tmp) //nolint:errcheck
|
||||
defer os.Remove(tmpCARv2File) //nolint:errcheck
|
||||
|
||||
root, err := a.doImport(ctx, ref.Path, tmp)
|
||||
root, err := a.importNormalFileToFilestoreCARv2(ctx, id, ref.Path, tmpCARv2File)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("failed to import normal file to CARv2")
|
||||
}
|
||||
|
||||
fs, err := stores.ReadOnlyFilestore(tmp)
|
||||
// generate a deterministic CARv1 payload from the UnixFS DAG by doing an IPLD
|
||||
// traversal over the Unixfs DAG in the CARv2 file using the "all selector" i.e the entire DAG selector.
|
||||
fs, err := filestorecaradapter.NewReadOnlyFileStore(tmpCARv2File)
|
||||
if err != nil {
|
||||
return xerrors.Errorf("failed to open filestore from carv2 in path %s: %w", tmp, err)
|
||||
return xerrors.Errorf("failed to open read only CARv2 blockstore: %w", err)
|
||||
}
|
||||
defer fs.Close() //nolint:errcheck
|
||||
|
||||
|
||||
+37
-59
@@ -2,13 +2,18 @@ package client
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io/ioutil"
|
||||
"os"
|
||||
|
||||
"github.com/filecoin-project/go-fil-markets/stores"
|
||||
"github.com/filecoin-project/go-fil-markets/filestorecaradapter"
|
||||
bstore "github.com/filecoin-project/lotus/blockstore"
|
||||
"github.com/filecoin-project/lotus/build"
|
||||
"github.com/filecoin-project/lotus/node/repo/importmgr"
|
||||
"github.com/ipfs/go-blockservice"
|
||||
"github.com/ipfs/go-cid"
|
||||
"github.com/ipfs/go-cidutil"
|
||||
"github.com/ipfs/go-datastore"
|
||||
ds_sync "github.com/ipfs/go-datastore/sync"
|
||||
"github.com/ipfs/go-filestore"
|
||||
chunker "github.com/ipfs/go-ipfs-chunker"
|
||||
offline "github.com/ipfs/go-ipfs-exchange-offline"
|
||||
files2 "github.com/ipfs/go-ipfs-files"
|
||||
@@ -16,83 +21,56 @@ import (
|
||||
"github.com/ipfs/go-merkledag"
|
||||
"github.com/ipfs/go-unixfs/importer/balanced"
|
||||
ihelper "github.com/ipfs/go-unixfs/importer/helpers"
|
||||
"github.com/ipld/go-car/v2"
|
||||
"github.com/ipld/go-car/v2/blockstore"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/lotus/build"
|
||||
)
|
||||
|
||||
// doImport takes a standard file (src), forms a UnixFS DAG, and writes a
|
||||
// CARv2 file with positional mapping (backed by the go-filestore library).
|
||||
func (a *API) doImport(ctx context.Context, src string, dst string) (cid.Cid, error) {
|
||||
// This method uses a two-phase approach with a staging CAR blockstore and
|
||||
// a final CAR blockstore.
|
||||
//
|
||||
// This is necessary because of https://github.com/ipld/go-car/issues/196
|
||||
//
|
||||
// TODO: do we need to chunk twice? Isn't the first output already in the
|
||||
// right order? Can't we just copy the CAR file and replace the header?
|
||||
f, err := ioutil.TempFile("", "")
|
||||
if err != nil {
|
||||
return cid.Undef, xerrors.Errorf("failed to create temp file: %w", err)
|
||||
}
|
||||
_ = f.Close() // close; we only want the path.
|
||||
tmp := f.Name()
|
||||
defer os.Remove(tmp) //nolint:errcheck
|
||||
|
||||
// Step 1. Compute the UnixFS DAG and write it to a CARv2 file to get
|
||||
// the root CID of the DAG.
|
||||
fstore, err := stores.ReadWriteFilestore(tmp)
|
||||
if err != nil {
|
||||
return cid.Undef, xerrors.Errorf("failed to create temporary filestore: %w", err)
|
||||
}
|
||||
// importNormalFileToFilestoreCARv2 transforms the client's "normal file" to a Unixfs IPLD DAG and writes out the DAG to a CARv2 file
|
||||
// that can be used to back a filestore.
|
||||
func (a *API) importNormalFileToFilestoreCARv2(ctx context.Context, importID importmgr.ImportID, inputFilePath string, outputCARv2Path string) (c cid.Cid, finalErr error) {
|
||||
|
||||
// TODO: We've currently put in a hack to create the Unixfs DAG as a CARv2 without using Badger.
|
||||
// We first create the Unixfs DAG using a filestore to get the root of the Unixfs DAG.
|
||||
// We can't create the UnixfsDAG right away using a CARv2 read-write blockstore as the blockstore
|
||||
// needs the root of the DAG during instantiation to write out a valid CARv2 file.
|
||||
//
|
||||
// In the second pass, we create a CARv2 file with the root present using the root node we get in the above step.
|
||||
// This hack should be fixed when CARv2 allows specifying the root AFTER finishing the CARv2 streaming write.
|
||||
fm := filestore.NewFileManager(ds_sync.MutexWrap(datastore.NewMapDatastore()), "/")
|
||||
fm.AllowFiles = true
|
||||
fstore := filestore.NewFilestore(bstore.NewMemorySync(), fm)
|
||||
bsvc := blockservice.New(fstore, offline.Exchange(fstore))
|
||||
dags := merkledag.NewDAGService(bsvc)
|
||||
defer bsvc.Close() //nolint:errcheck
|
||||
|
||||
root, err := buildUnixFS(ctx, src, dags)
|
||||
// ---- First Pass --- Write out the UnixFS DAG to a rootless CARv2 file by instantiating a read-write CARv2 blockstore without the root.
|
||||
root, err := importNormalFileToUnixfsDAG(ctx, inputFilePath, merkledag.NewDAGService(bsvc))
|
||||
if err != nil {
|
||||
_ = fstore.Close()
|
||||
return cid.Undef, xerrors.Errorf("failed to import file to store to compute root: %w", err)
|
||||
return cid.Undef, xerrors.Errorf("failed to import file to store: %w", err)
|
||||
}
|
||||
|
||||
if err := fstore.Close(); err != nil {
|
||||
return cid.Undef, xerrors.Errorf("failed to finalize car filestore: %w", err)
|
||||
}
|
||||
|
||||
// Step 2. We now have the root of the UnixFS DAG, and we can write the
|
||||
// final CAR for real under `dst`.
|
||||
bs, err := blockstore.OpenReadWrite(dst, []cid.Cid{root},
|
||||
car.ZeroLengthSectionAsEOF(true),
|
||||
blockstore.UseWholeCIDs(true))
|
||||
//------ Second Pass --- Now that we have the root of the Unixfs DAG -> write out the Unixfs DAG to a CARv2 file with the root present by using a
|
||||
// filestore backed by a read-write CARv2 blockstore.
|
||||
fsb, err := filestorecaradapter.NewReadWriteFileStore(outputCARv2Path, []cid.Cid{root})
|
||||
if err != nil {
|
||||
return cid.Undef, xerrors.Errorf("failed to create a carv2 read/write blockstore: %w", err)
|
||||
return cid.Undef, xerrors.Errorf("failed to create a CARv2 read-write blockstore: %w", err)
|
||||
}
|
||||
defer fsb.Close() //nolint:errcheck
|
||||
|
||||
bsvc = blockservice.New(bs, offline.Exchange(bs))
|
||||
dags = merkledag.NewDAGService(bsvc)
|
||||
finalRoot, err := buildUnixFS(ctx, src, dags)
|
||||
bsvc = blockservice.New(fsb, offline.Exchange(fsb))
|
||||
root2, err := importNormalFileToUnixfsDAG(ctx, inputFilePath, merkledag.NewDAGService(bsvc))
|
||||
if err != nil {
|
||||
_ = bs.Close()
|
||||
return cid.Undef, xerrors.Errorf("failed to create UnixFS DAG with carv2 blockstore: %w", err)
|
||||
return cid.Undef, xerrors.Errorf("failed to create Unixfs DAG with CARv2 blockstore: %w", err)
|
||||
}
|
||||
|
||||
if err := bs.Finalize(); err != nil {
|
||||
return cid.Undef, xerrors.Errorf("failed to finalize car blockstore: %w", err)
|
||||
}
|
||||
|
||||
if root != finalRoot {
|
||||
if root != root2 {
|
||||
return cid.Undef, xerrors.New("roots do not match")
|
||||
}
|
||||
|
||||
return root, nil
|
||||
}
|
||||
|
||||
// buildUnixFS builds a UnixFS DAG out of the supplied ordinary file,
|
||||
// and imports the DAG into the supplied service.
|
||||
func buildUnixFS(ctx context.Context, src string, dag ipld.DAGService) (cid.Cid, error) {
|
||||
f, err := os.Open(src)
|
||||
// importNormalFileToUnixfsDAG transforms a client's normal file to a UnixfsDAG and imports the DAG to the given DAG service.
|
||||
func importNormalFileToUnixfsDAG(ctx context.Context, inputFilePath string, dag ipld.DAGService) (cid.Cid, error) {
|
||||
f, err := os.Open(inputFilePath)
|
||||
if err != nil {
|
||||
return cid.Undef, xerrors.Errorf("failed to open input file: %w", err)
|
||||
}
|
||||
@@ -103,7 +81,7 @@ func buildUnixFS(ctx context.Context, src string, dag ipld.DAGService) (cid.Cid,
|
||||
return cid.Undef, xerrors.Errorf("failed to stat file :%w", err)
|
||||
}
|
||||
|
||||
file, err := files2.NewReaderPathFile(src, f, stat)
|
||||
file, err := files2.NewReaderPathFile(inputFilePath, f, stat)
|
||||
if err != nil {
|
||||
return cid.Undef, xerrors.Errorf("failed to create reader path file: %w", err)
|
||||
}
|
||||
|
||||
@@ -4,11 +4,12 @@ import (
|
||||
"context"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/filecoin-project/go-fil-markets/stores"
|
||||
"github.com/filecoin-project/go-fil-markets/filestorecaradapter"
|
||||
"github.com/filecoin-project/lotus/node/repo/importmgr"
|
||||
"github.com/ipfs/go-blockservice"
|
||||
"github.com/ipfs/go-cid"
|
||||
@@ -21,50 +22,39 @@ import (
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// This test uses a full "dense" CARv2, and not a filestore (positional mapping).
|
||||
func TestRoundtripUnixFS_Dense(t *testing.T) {
|
||||
func TestImportNormalFileToUnixfsDAG(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
inputPath, inputContents := genInputFile(t)
|
||||
inputPath, inputContents := genNormalInputFile(t)
|
||||
defer os.Remove(inputPath) //nolint:errcheck
|
||||
|
||||
carv2File := newTmpFile(t)
|
||||
carv2File := genTmpFile(t)
|
||||
defer os.Remove(carv2File) //nolint:errcheck
|
||||
|
||||
// import a file to a Unixfs DAG using a CARv2 read/write blockstore.
|
||||
path, err := blockstore.OpenReadWrite(carv2File, nil,
|
||||
carv2.ZeroLengthSectionAsEOF(true),
|
||||
blockstore.UseWholeCIDs(true))
|
||||
// import a normal file to a Unixfs DAG using a CARv2 read-write blockstore and flush it out to a CARv2 file.
|
||||
tempCARv2Store, err := blockstore.OpenReadWrite(carv2File, []cid.Cid{}, carv2.ZeroLengthSectionAsEOF(true), blockstore.UseWholeCIDs(true))
|
||||
require.NoError(t, err)
|
||||
|
||||
bsvc := blockservice.New(path, offline.Exchange(path))
|
||||
dags := merkledag.NewDAGService(bsvc)
|
||||
|
||||
root, err := buildUnixFS(ctx, inputPath, dags)
|
||||
bsvc := blockservice.New(tempCARv2Store, offline.Exchange(tempCARv2Store))
|
||||
root, err := importNormalFileToUnixfsDAG(ctx, inputPath, merkledag.NewDAGService(bsvc))
|
||||
require.NoError(t, err)
|
||||
require.NotEqual(t, cid.Undef, root)
|
||||
require.NoError(t, path.Finalize())
|
||||
require.NoError(t, tempCARv2Store.Finalize())
|
||||
|
||||
// reconstruct the file.
|
||||
readOnly, err := blockstore.OpenReadOnly(carv2File,
|
||||
carv2.ZeroLengthSectionAsEOF(true),
|
||||
blockstore.UseWholeCIDs(true))
|
||||
// convert the CARv2 file to a normal file again and ensure the contents match.
|
||||
readOnly, err := blockstore.OpenReadOnly(carv2File, carv2.ZeroLengthSectionAsEOF(true), blockstore.UseWholeCIDs(true))
|
||||
require.NoError(t, err)
|
||||
defer readOnly.Close() //nolint:errcheck
|
||||
dag := merkledag.NewDAGService(blockservice.New(readOnly, offline.Exchange(readOnly)))
|
||||
|
||||
dags = merkledag.NewDAGService(blockservice.New(readOnly, offline.Exchange(readOnly)))
|
||||
|
||||
nd, err := dags.Get(ctx, root)
|
||||
nd, err := dag.Get(ctx, root)
|
||||
require.NoError(t, err)
|
||||
file, err := unixfile.NewUnixfsFile(ctx, dag, nd)
|
||||
require.NoError(t, err)
|
||||
|
||||
file, err := unixfile.NewUnixfsFile(ctx, dags, nd)
|
||||
require.NoError(t, err)
|
||||
|
||||
tmpOutput := newTmpFile(t)
|
||||
tmpOutput := genTmpFile(t)
|
||||
defer os.Remove(tmpOutput) //nolint:errcheck
|
||||
require.NoError(t, files.WriteTo(file, tmpOutput))
|
||||
|
||||
// ensure contents of the initial input file and the output file are identical.
|
||||
|
||||
fo, err := os.Open(tmpOutput)
|
||||
require.NoError(t, err)
|
||||
bz2, err := ioutil.ReadAll(fo)
|
||||
@@ -73,36 +63,35 @@ func TestRoundtripUnixFS_Dense(t *testing.T) {
|
||||
require.Equal(t, inputContents, bz2)
|
||||
}
|
||||
|
||||
func TestRoundtripUnixFS_Filestore(t *testing.T) {
|
||||
func TestImportNormalFileToCARv2(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a := &API{
|
||||
Imports: &importmgr.Mgr{},
|
||||
}
|
||||
importID := importmgr.ImportID(rand.Uint64())
|
||||
|
||||
inputFilePath, inputContents := genInputFile(t)
|
||||
inputFilePath, inputContents := genNormalInputFile(t)
|
||||
defer os.Remove(inputFilePath) //nolint:errcheck
|
||||
|
||||
path := newTmpFile(t)
|
||||
defer os.Remove(path) //nolint:errcheck
|
||||
outputCARv2 := genTmpFile(t)
|
||||
defer os.Remove(outputCARv2) //nolint:errcheck
|
||||
|
||||
root, err := a.doImport(ctx, inputFilePath, path)
|
||||
root, err := a.importNormalFileToFilestoreCARv2(ctx, importID, inputFilePath, outputCARv2)
|
||||
require.NoError(t, err)
|
||||
require.NotEqual(t, cid.Undef, root)
|
||||
|
||||
// convert the CARv2 to a normal file again and ensure the contents match
|
||||
fs, err := stores.ReadOnlyFilestore(path)
|
||||
readOnly, err := filestorecaradapter.NewReadOnlyFileStore(outputCARv2)
|
||||
require.NoError(t, err)
|
||||
defer fs.Close() //nolint:errcheck
|
||||
defer readOnly.Close() //nolint:errcheck
|
||||
dag := merkledag.NewDAGService(blockservice.New(readOnly, offline.Exchange(readOnly)))
|
||||
|
||||
dags := merkledag.NewDAGService(blockservice.New(fs, offline.Exchange(fs)))
|
||||
|
||||
nd, err := dags.Get(ctx, root)
|
||||
nd, err := dag.Get(ctx, root)
|
||||
require.NoError(t, err)
|
||||
file, err := unixfile.NewUnixfsFile(ctx, dag, nd)
|
||||
require.NoError(t, err)
|
||||
|
||||
file, err := unixfile.NewUnixfsFile(ctx, dags, nd)
|
||||
require.NoError(t, err)
|
||||
|
||||
tmpOutput := newTmpFile(t)
|
||||
tmpOutput := genTmpFile(t)
|
||||
defer os.Remove(tmpOutput) //nolint:errcheck
|
||||
require.NoError(t, files.WriteTo(file, tmpOutput))
|
||||
|
||||
@@ -115,14 +104,14 @@ func TestRoundtripUnixFS_Filestore(t *testing.T) {
|
||||
require.Equal(t, inputContents, bz2)
|
||||
}
|
||||
|
||||
func newTmpFile(t *testing.T) string {
|
||||
func genTmpFile(t *testing.T) string {
|
||||
f, err := os.CreateTemp("", "")
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, f.Close())
|
||||
return f.Name()
|
||||
}
|
||||
|
||||
func genInputFile(t *testing.T) (filepath string, contents []byte) {
|
||||
func genNormalInputFile(t *testing.T) (filepath string, contents []byte) {
|
||||
s := strings.Repeat("abcde", 100)
|
||||
tmp, err := os.CreateTemp("", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
+5
-238
@@ -3,20 +3,16 @@ package impl
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/filecoin-project/dagstore"
|
||||
"github.com/filecoin-project/dagstore/shard"
|
||||
"github.com/filecoin-project/go-jsonrpc/auth"
|
||||
|
||||
"github.com/filecoin-project/lotus/chain/actors/builtin"
|
||||
"github.com/filecoin-project/lotus/chain/gen"
|
||||
|
||||
"github.com/filecoin-project/lotus/build"
|
||||
"github.com/google/uuid"
|
||||
"github.com/ipfs/go-cid"
|
||||
"github.com/libp2p/go-libp2p-core/host"
|
||||
@@ -24,13 +20,11 @@ import (
|
||||
"go.uber.org/fx"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/lotus/build"
|
||||
|
||||
"github.com/filecoin-project/go-address"
|
||||
datatransfer "github.com/filecoin-project/go-data-transfer"
|
||||
"github.com/filecoin-project/go-fil-markets/piecestore"
|
||||
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
|
||||
"github.com/filecoin-project/go-fil-markets/storagemarket"
|
||||
retrievalmarket "github.com/filecoin-project/go-fil-markets/retrievalmarket"
|
||||
storagemarket "github.com/filecoin-project/go-fil-markets/storagemarket"
|
||||
"github.com/filecoin-project/go-state-types/abi"
|
||||
|
||||
sectorstorage "github.com/filecoin-project/lotus/extern/sector-storage"
|
||||
@@ -40,8 +34,6 @@ import (
|
||||
sealing "github.com/filecoin-project/lotus/extern/storage-sealing"
|
||||
"github.com/filecoin-project/lotus/extern/storage-sealing/sealiface"
|
||||
|
||||
sto "github.com/filecoin-project/specs-storage/storage"
|
||||
|
||||
"github.com/filecoin-project/lotus/api"
|
||||
apitypes "github.com/filecoin-project/lotus/api/types"
|
||||
"github.com/filecoin-project/lotus/chain/types"
|
||||
@@ -50,6 +42,7 @@ import (
|
||||
"github.com/filecoin-project/lotus/node/modules/dtypes"
|
||||
"github.com/filecoin-project/lotus/storage"
|
||||
"github.com/filecoin-project/lotus/storage/sectorblocks"
|
||||
sto "github.com/filecoin-project/specs-storage/storage"
|
||||
)
|
||||
|
||||
type StorageMinerAPI struct {
|
||||
@@ -58,8 +51,6 @@ type StorageMinerAPI struct {
|
||||
api.Common
|
||||
api.Net
|
||||
|
||||
EnabledSubsystems api.MinerSubsystems
|
||||
|
||||
Full api.FullNode
|
||||
LocalStore *stores.Local
|
||||
RemoteStore *stores.Remote
|
||||
@@ -72,7 +63,6 @@ type StorageMinerAPI struct {
|
||||
DealPublisher *storageadapter.DealPublisher `optional:"true"`
|
||||
SectorBlocks *sectorblocks.SectorBlocks `optional:"true"`
|
||||
Host host.Host `optional:"true"`
|
||||
DAGStore *dagstore.DAGStore `optional:"true"`
|
||||
|
||||
// Miner / storage
|
||||
Miner *storage.Miner `optional:"true"`
|
||||
@@ -106,8 +96,6 @@ type StorageMinerAPI struct {
|
||||
SetExpectedSealDurationFunc dtypes.SetExpectedSealDurationFunc `optional:"true"`
|
||||
}
|
||||
|
||||
var _ api.StorageMiner = &StorageMinerAPI{}
|
||||
|
||||
func (sm *StorageMinerAPI) ServeRemote(perm bool) func(w http.ResponseWriter, r *http.Request) {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if perm == true {
|
||||
@@ -555,225 +543,6 @@ func (sm *StorageMinerAPI) MarketPublishPendingDeals(ctx context.Context) error
|
||||
return nil
|
||||
}
|
||||
|
||||
func (sm *StorageMinerAPI) DagstoreListShards(ctx context.Context) ([]api.DagstoreShardInfo, error) {
|
||||
if sm.DAGStore == nil {
|
||||
return nil, fmt.Errorf("dagstore not available on this node")
|
||||
}
|
||||
|
||||
info := sm.DAGStore.AllShardsInfo()
|
||||
ret := make([]api.DagstoreShardInfo, 0, len(info))
|
||||
for k, i := range info {
|
||||
ret = append(ret, api.DagstoreShardInfo{
|
||||
Key: k.String(),
|
||||
State: i.ShardState.String(),
|
||||
Error: func() string {
|
||||
if i.Error == nil {
|
||||
return ""
|
||||
}
|
||||
return i.Error.Error()
|
||||
}(),
|
||||
})
|
||||
}
|
||||
|
||||
// order by key.
|
||||
sort.SliceStable(ret, func(i, j int) bool {
|
||||
return ret[i].Key < ret[j].Key
|
||||
})
|
||||
|
||||
return ret, nil
|
||||
}
|
||||
|
||||
func (sm *StorageMinerAPI) DagstoreInitializeShard(ctx context.Context, key string) error {
|
||||
if sm.DAGStore == nil {
|
||||
return fmt.Errorf("dagstore not available on this node")
|
||||
}
|
||||
|
||||
k := shard.KeyFromString(key)
|
||||
|
||||
info, err := sm.DAGStore.GetShardInfo(k)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get shard info: %w", err)
|
||||
}
|
||||
if st := info.ShardState; st != dagstore.ShardStateNew {
|
||||
return fmt.Errorf("cannot initialize shard; expected state ShardStateNew, was: %s", st.String())
|
||||
}
|
||||
|
||||
ch := make(chan dagstore.ShardResult, 1)
|
||||
if err = sm.DAGStore.AcquireShard(ctx, k, ch, dagstore.AcquireOpts{}); err != nil {
|
||||
return fmt.Errorf("failed to acquire shard: %w", err)
|
||||
}
|
||||
|
||||
var res dagstore.ShardResult
|
||||
select {
|
||||
case res = <-ch:
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
if err := res.Error; err != nil {
|
||||
return fmt.Errorf("failed to acquire shard: %w", err)
|
||||
}
|
||||
|
||||
if res.Accessor != nil {
|
||||
err = res.Accessor.Close()
|
||||
if err != nil {
|
||||
log.Warnw("failed to close shard accessor; continuing", "shard_key", k, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (sm *StorageMinerAPI) DagstoreInitializeAll(ctx context.Context, params api.DagstoreInitializeAllParams) (<-chan api.DagstoreInitializeAllEvent, error) {
|
||||
if sm.DAGStore == nil {
|
||||
return nil, fmt.Errorf("dagstore not available on this node")
|
||||
}
|
||||
|
||||
// prepare the thottler tokens.
|
||||
var throttle chan struct{}
|
||||
if c := params.MaxConcurrency; c > 0 {
|
||||
throttle = make(chan struct{}, c)
|
||||
for i := 0; i < c; i++ {
|
||||
throttle <- struct{}{}
|
||||
}
|
||||
}
|
||||
|
||||
info := sm.DAGStore.AllShardsInfo()
|
||||
var uninit []string
|
||||
for k, i := range info {
|
||||
if i.ShardState != dagstore.ShardStateNew {
|
||||
continue
|
||||
}
|
||||
uninit = append(uninit, k.String())
|
||||
}
|
||||
|
||||
total := len(uninit)
|
||||
if total == 0 {
|
||||
out := make(chan api.DagstoreInitializeAllEvent)
|
||||
close(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// response channel must be closed when we're done, or the context is cancelled.
|
||||
// this buffering is necessary to prevent inflight children goroutines from
|
||||
// publishing to a closed channel (res) when the context is cancelled.
|
||||
out := make(chan api.DagstoreInitializeAllEvent, 32) // internal buffer.
|
||||
res := make(chan api.DagstoreInitializeAllEvent, 32) // returned to caller.
|
||||
|
||||
// pump events back to caller.
|
||||
// two events per shard.
|
||||
go func() {
|
||||
defer close(res)
|
||||
|
||||
for i := 0; i < total*2; i++ {
|
||||
select {
|
||||
case res <- <-out:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
go func() {
|
||||
for i, k := range uninit {
|
||||
select {
|
||||
case <-throttle:
|
||||
// acquired a throttle token, proceed.
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
|
||||
go func(k string, i int) {
|
||||
r := api.DagstoreInitializeAllEvent{
|
||||
Key: k,
|
||||
Event: "start",
|
||||
Total: total,
|
||||
Current: i + 1, // start with 1
|
||||
}
|
||||
select {
|
||||
case out <- r:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
|
||||
err := sm.DagstoreInitializeShard(ctx, k)
|
||||
throttle <- struct{}{}
|
||||
|
||||
r.Event = "end"
|
||||
if err == nil {
|
||||
r.Success = true
|
||||
} else {
|
||||
r.Success = false
|
||||
r.Error = err.Error()
|
||||
}
|
||||
|
||||
select {
|
||||
case out <- r:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
}(k, i)
|
||||
}
|
||||
}()
|
||||
|
||||
return res, nil
|
||||
|
||||
}
|
||||
|
||||
func (sm *StorageMinerAPI) DagstoreRecoverShard(ctx context.Context, key string) error {
|
||||
if sm.DAGStore == nil {
|
||||
return fmt.Errorf("dagstore not available on this node")
|
||||
}
|
||||
|
||||
k := shard.KeyFromString(key)
|
||||
|
||||
info, err := sm.DAGStore.GetShardInfo(k)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get shard info: %w", err)
|
||||
}
|
||||
if st := info.ShardState; st != dagstore.ShardStateErrored {
|
||||
return fmt.Errorf("cannot recover shard; expected state ShardStateErrored, was: %s", st.String())
|
||||
}
|
||||
|
||||
ch := make(chan dagstore.ShardResult, 1)
|
||||
if err = sm.DAGStore.RecoverShard(ctx, k, ch, dagstore.RecoverOpts{}); err != nil {
|
||||
return fmt.Errorf("failed to recover shard: %w", err)
|
||||
}
|
||||
|
||||
var res dagstore.ShardResult
|
||||
select {
|
||||
case res = <-ch:
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
return res.Error
|
||||
}
|
||||
|
||||
func (sm *StorageMinerAPI) DagstoreGC(ctx context.Context) ([]api.DagstoreShardResult, error) {
|
||||
if sm.DAGStore == nil {
|
||||
return nil, fmt.Errorf("dagstore not available on this node")
|
||||
}
|
||||
|
||||
res, err := sm.DAGStore.GC(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to gc: %w", err)
|
||||
}
|
||||
|
||||
ret := make([]api.DagstoreShardResult, 0, len(res.Shards))
|
||||
for k, err := range res.Shards {
|
||||
r := api.DagstoreShardResult{Key: k.String()}
|
||||
if err == nil {
|
||||
r.Success = true
|
||||
} else {
|
||||
r.Success = false
|
||||
r.Error = err.Error()
|
||||
}
|
||||
ret = append(ret, r)
|
||||
}
|
||||
|
||||
return ret, nil
|
||||
}
|
||||
|
||||
func (sm *StorageMinerAPI) DealsList(ctx context.Context) ([]api.MarketDeal, error) {
|
||||
return sm.listDeals(ctx)
|
||||
}
|
||||
@@ -934,6 +703,4 @@ func (sm *StorageMinerAPI) ComputeProof(ctx context.Context, ssi []builtin.Secto
|
||||
return sm.Epp.ComputeProof(ctx, ssi, rand)
|
||||
}
|
||||
|
||||
func (sm *StorageMinerAPI) RuntimeSubsystems(context.Context) (res api.MinerSubsystems, err error) {
|
||||
return sm.EnabledSubsystems, nil
|
||||
}
|
||||
var _ api.StorageMiner = &StorageMinerAPI{}
|
||||
|
||||
@@ -182,7 +182,7 @@ func StorageClient(lc fx.Lifecycle, h host.Host, dataTransfer dtypes.ClientDataT
|
||||
func RetrievalClient(lc fx.Lifecycle, h host.Host, r repo.LockedRepo, dt dtypes.ClientDataTransfer, payAPI payapi.PaychAPI, resolver discovery.PeerResolver,
|
||||
ds dtypes.MetadataDS, chainAPI full.ChainAPI, stateAPI full.StateAPI, j journal.Journal) (retrievalmarket.RetrievalClient, error) {
|
||||
|
||||
carsPath := filepath.Join(r.Path(), DefaultDAGStoreDir, "retrieval-cars")
|
||||
carsPath := filepath.Join(r.Path(), dagStore, "retrieval-cars")
|
||||
|
||||
if err := os.MkdirAll(carsPath, 0755); err != nil {
|
||||
return nil, xerrors.Errorf("failed to create dir")
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -15,28 +16,26 @@ import (
|
||||
"go.uber.org/multierr"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/go-address"
|
||||
dtimpl "github.com/filecoin-project/go-data-transfer/impl"
|
||||
dtnet "github.com/filecoin-project/go-data-transfer/network"
|
||||
dtgstransport "github.com/filecoin-project/go-data-transfer/transport/graphsync"
|
||||
"github.com/filecoin-project/go-jsonrpc/auth"
|
||||
"github.com/filecoin-project/go-paramfetch"
|
||||
"github.com/filecoin-project/go-state-types/abi"
|
||||
"github.com/filecoin-project/go-statestore"
|
||||
"github.com/filecoin-project/go-storedcounter"
|
||||
"github.com/ipfs/go-bitswap"
|
||||
"github.com/ipfs/go-bitswap/network"
|
||||
"github.com/ipfs/go-blockservice"
|
||||
"github.com/ipfs/go-cid"
|
||||
"github.com/ipfs/go-datastore"
|
||||
"github.com/ipfs/go-datastore/namespace"
|
||||
levelds "github.com/ipfs/go-ds-leveldb"
|
||||
measure "github.com/ipfs/go-ds-measure"
|
||||
graphsync "github.com/ipfs/go-graphsync/impl"
|
||||
gsnet "github.com/ipfs/go-graphsync/network"
|
||||
"github.com/ipfs/go-graphsync/storeutil"
|
||||
"github.com/ipfs/go-merkledag"
|
||||
"github.com/libp2p/go-libp2p-core/host"
|
||||
"github.com/libp2p/go-libp2p-core/routing"
|
||||
ldbopts "github.com/syndtr/goleveldb/leveldb/opt"
|
||||
|
||||
"github.com/filecoin-project/go-address"
|
||||
dtimpl "github.com/filecoin-project/go-data-transfer/impl"
|
||||
dtnet "github.com/filecoin-project/go-data-transfer/network"
|
||||
dtgstransport "github.com/filecoin-project/go-data-transfer/transport/graphsync"
|
||||
piecefilestore "github.com/filecoin-project/go-fil-markets/filestore"
|
||||
piecestoreimpl "github.com/filecoin-project/go-fil-markets/piecestore/impl"
|
||||
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
|
||||
@@ -47,6 +46,11 @@ import (
|
||||
storageimpl "github.com/filecoin-project/go-fil-markets/storagemarket/impl"
|
||||
"github.com/filecoin-project/go-fil-markets/storagemarket/impl/storedask"
|
||||
smnet "github.com/filecoin-project/go-fil-markets/storagemarket/network"
|
||||
"github.com/filecoin-project/go-jsonrpc/auth"
|
||||
"github.com/filecoin-project/go-paramfetch"
|
||||
"github.com/filecoin-project/go-state-types/abi"
|
||||
"github.com/filecoin-project/go-statestore"
|
||||
"github.com/filecoin-project/go-storedcounter"
|
||||
|
||||
sectorstorage "github.com/filecoin-project/lotus/extern/sector-storage"
|
||||
"github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper"
|
||||
@@ -77,6 +81,7 @@ import (
|
||||
)
|
||||
|
||||
var StorageCounterDSPrefix = "/storage/nextid"
|
||||
var dagStore = "dagStore"
|
||||
|
||||
func minerAddrFromDS(ds dtypes.MetadataDS) (address.Address, error) {
|
||||
maddrb, err := ds.Get(datastore.NewKey("miner-address"))
|
||||
@@ -575,6 +580,84 @@ func BasicDealFilter(user dtypes.StorageDealFilter) func(onlineOk dtypes.Conside
|
||||
}
|
||||
}
|
||||
|
||||
func NewLotusAccessor(lc fx.Lifecycle,
|
||||
pieceStore dtypes.ProviderPieceStore,
|
||||
rpn retrievalmarket.RetrievalProviderNode,
|
||||
) (dagstore.LotusAccessor, error) {
|
||||
mountApi := dagstore.NewLotusAccessor(pieceStore, rpn)
|
||||
lc.Append(fx.Hook{
|
||||
OnStart: func(ctx context.Context) error {
|
||||
return mountApi.Start(ctx)
|
||||
},
|
||||
OnStop: func(context.Context) error {
|
||||
return nil
|
||||
},
|
||||
})
|
||||
|
||||
return mountApi, nil
|
||||
}
|
||||
|
||||
func DagStoreWrapper(
|
||||
lc fx.Lifecycle,
|
||||
r repo.LockedRepo,
|
||||
lotusAccessor dagstore.LotusAccessor,
|
||||
) (*dagstore.Wrapper, error) {
|
||||
dagStoreDir := filepath.Join(r.Path(), dagStore)
|
||||
dagStoreDS, err := createDAGStoreDatastore(dagStoreDir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
cfg := dagstore.MarketDAGStoreConfig{
|
||||
TransientsDir: filepath.Join(dagStoreDir, "transients"),
|
||||
IndexDir: filepath.Join(dagStoreDir, "index"),
|
||||
Datastore: dagStoreDS,
|
||||
GCInterval: 1 * time.Minute,
|
||||
MaxConcurrentIndex: 5,
|
||||
}
|
||||
|
||||
dsw, err := dagstore.NewDagStoreWrapper(cfg, lotusAccessor)
|
||||
if err != nil {
|
||||
return nil, xerrors.Errorf("failed to create DAG store wrapper: %w", err)
|
||||
}
|
||||
|
||||
lc.Append(fx.Hook{
|
||||
OnStart: func(ctx context.Context) error {
|
||||
dsw.Start(ctx)
|
||||
return nil
|
||||
},
|
||||
OnStop: func(context.Context) error {
|
||||
return dsw.Close()
|
||||
},
|
||||
})
|
||||
return dsw, nil
|
||||
}
|
||||
|
||||
// createDAGStoreDatastore creates a datastore under the given base directory
|
||||
// for DAG store metadata
|
||||
func createDAGStoreDatastore(baseDir string) (datastore.Batching, error) {
|
||||
// Create a datastore directory under the base dir if it doesn't already exist
|
||||
dsDir := path.Join(baseDir, "datastore")
|
||||
if err := os.MkdirAll(dsDir, 0755); err != nil {
|
||||
return nil, xerrors.Errorf("failed to create directory %s for DAG store datastore: %w", dsDir, err)
|
||||
}
|
||||
|
||||
// Create a new LevelDB datastore
|
||||
ds, err := levelds.NewDatastore(dsDir, &levelds.Options{
|
||||
Compression: ldbopts.NoCompression,
|
||||
NoSync: false,
|
||||
Strict: ldbopts.StrictAll,
|
||||
ReadOnly: false,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, xerrors.Errorf("failed to open datastore for DAG store: %w", err)
|
||||
}
|
||||
|
||||
// Keep statistics about the datastore
|
||||
mds := measure.New("measure.", ds)
|
||||
return mds, nil
|
||||
}
|
||||
|
||||
func StorageProvider(minerAddress dtypes.MinerAddress,
|
||||
storedAsk *storedask.StoredAsk,
|
||||
h host.Host, ds dtypes.MetadataDS,
|
||||
@@ -586,33 +669,18 @@ func StorageProvider(minerAddress dtypes.MinerAddress,
|
||||
dsw *dagstore.Wrapper,
|
||||
) (storagemarket.StorageProvider, error) {
|
||||
net := smnet.NewFromLibp2pHost(h)
|
||||
|
||||
// Create a directory under the repo path for temporary files
|
||||
repoTmpPath := filepath.Join(r.Path(), "tmp")
|
||||
err := os.MkdirAll(repoTmpPath, 0755) //nolint: gosec
|
||||
if err != nil {
|
||||
return nil, xerrors.Errorf("creating tmp dir for storage provider repo: %w", err)
|
||||
}
|
||||
|
||||
store, err := piecefilestore.NewLocalFileStore(piecefilestore.OsPath(repoTmpPath))
|
||||
store, err := piecefilestore.NewLocalFileStore(piecefilestore.OsPath(r.Path()))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
opt := storageimpl.CustomDealDecisionLogic(storageimpl.DealDeciderFunc(df))
|
||||
dagStorePath := filepath.Join(r.Path(), dagStore)
|
||||
|
||||
return storageimpl.NewProvider(
|
||||
net,
|
||||
namespace.Wrap(ds, datastore.NewKey("/deals/provider")),
|
||||
store,
|
||||
dsw,
|
||||
pieceStore,
|
||||
dataTransfer,
|
||||
spn,
|
||||
address.Address(minerAddress),
|
||||
storedAsk,
|
||||
opt,
|
||||
)
|
||||
opt := storageimpl.CustomDealDecisionLogic(storageimpl.DealDeciderFunc(df))
|
||||
shardMigrator := storageimpl.NewShardMigrator(address.Address(minerAddress), dagStorePath, dsw, pieceStore, spn)
|
||||
|
||||
return storageimpl.NewProvider(net, namespace.Wrap(ds, datastore.NewKey("/deals/provider")), store, dsw, pieceStore,
|
||||
dataTransfer, spn, address.Address(minerAddress), storedAsk, shardMigrator, opt)
|
||||
}
|
||||
|
||||
func RetrievalDealFilter(userFilter dtypes.RetrievalDealFilter) func(onlineOk dtypes.ConsiderOnlineRetrievalDealsConfigFunc,
|
||||
@@ -1014,19 +1082,3 @@ func mutateCfg(r repo.LockedRepo, mutator func(*config.StorageMiner)) error {
|
||||
|
||||
return multierr.Combine(typeErr, setConfigErr)
|
||||
}
|
||||
|
||||
func ExtractEnabledMinerSubsystems(cfg config.MinerSubsystemConfig) (res api.MinerSubsystems) {
|
||||
if cfg.EnableMining {
|
||||
res = append(res, api.SubsystemMining)
|
||||
}
|
||||
if cfg.EnableSealing {
|
||||
res = append(res, api.SubsystemSealing)
|
||||
}
|
||||
if cfg.EnableSectorStorage {
|
||||
res = append(res, api.SubsystemSectorStorage)
|
||||
}
|
||||
if cfg.EnableMarkets {
|
||||
res = append(res, api.SubsystemMarkets)
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
@@ -1,112 +0,0 @@
|
||||
package modules
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
|
||||
"github.com/filecoin-project/dagstore"
|
||||
"go.uber.org/fx"
|
||||
"golang.org/x/xerrors"
|
||||
|
||||
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
|
||||
|
||||
mdagstore "github.com/filecoin-project/lotus/markets/dagstore"
|
||||
"github.com/filecoin-project/lotus/node/config"
|
||||
"github.com/filecoin-project/lotus/node/modules/dtypes"
|
||||
"github.com/filecoin-project/lotus/node/repo"
|
||||
)
|
||||
|
||||
const (
|
||||
EnvDAGStoreCopyConcurrency = "LOTUS_DAGSTORE_COPY_CONCURRENCY"
|
||||
DefaultDAGStoreDir = "dagstore"
|
||||
)
|
||||
|
||||
// NewMinerAPI creates a new MinerAPI adaptor for the dagstore mounts.
|
||||
func NewMinerAPI(lc fx.Lifecycle, r repo.LockedRepo, pieceStore dtypes.ProviderPieceStore, rpn retrievalmarket.RetrievalProviderNode) (mdagstore.MinerAPI, error) {
|
||||
cfg, err := extractDAGStoreConfig(r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// caps the amount of concurrent calls to the storage, so that we don't
|
||||
// spam it during heavy processes like bulk migration.
|
||||
if v, ok := os.LookupEnv("LOTUS_DAGSTORE_MOUNT_CONCURRENCY"); ok {
|
||||
concurrency, err := strconv.Atoi(v)
|
||||
if err == nil {
|
||||
cfg.MaxConcurrencyStorageCalls = concurrency
|
||||
}
|
||||
}
|
||||
|
||||
mountApi := mdagstore.NewMinerAPI(pieceStore, rpn, cfg.MaxConcurrencyStorageCalls)
|
||||
ready := make(chan error, 1)
|
||||
pieceStore.OnReady(func(err error) {
|
||||
ready <- err
|
||||
})
|
||||
lc.Append(fx.Hook{
|
||||
OnStart: func(ctx context.Context) error {
|
||||
if err := <-ready; err != nil {
|
||||
return fmt.Errorf("aborting dagstore start; piecestore failed to start: %s", err)
|
||||
}
|
||||
return mountApi.Start(ctx)
|
||||
},
|
||||
OnStop: func(context.Context) error {
|
||||
return nil
|
||||
},
|
||||
})
|
||||
|
||||
return mountApi, nil
|
||||
}
|
||||
|
||||
// DAGStore constructs a DAG store using the supplied minerAPI, and the
|
||||
// user configuration. It returns both the DAGStore and the Wrapper suitable for
|
||||
// passing to markets.
|
||||
func DAGStore(lc fx.Lifecycle, r repo.LockedRepo, minerAPI mdagstore.MinerAPI) (*dagstore.DAGStore, *mdagstore.Wrapper, error) {
|
||||
cfg, err := extractDAGStoreConfig(r)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
// fall back to default root directory if not explicitly set in the config.
|
||||
if cfg.RootDir == "" {
|
||||
cfg.RootDir = filepath.Join(r.Path(), DefaultDAGStoreDir)
|
||||
}
|
||||
|
||||
v, ok := os.LookupEnv(EnvDAGStoreCopyConcurrency)
|
||||
if ok {
|
||||
concurrency, err := strconv.Atoi(v)
|
||||
if err == nil {
|
||||
cfg.MaxConcurrentReadyFetches = concurrency
|
||||
}
|
||||
}
|
||||
|
||||
dagst, w, err := mdagstore.NewDAGStore(cfg, minerAPI)
|
||||
if err != nil {
|
||||
return nil, nil, xerrors.Errorf("failed to create DAG store: %w", err)
|
||||
}
|
||||
|
||||
lc.Append(fx.Hook{
|
||||
OnStart: func(ctx context.Context) error {
|
||||
return w.Start(ctx)
|
||||
},
|
||||
OnStop: func(context.Context) error {
|
||||
return w.Close()
|
||||
},
|
||||
})
|
||||
|
||||
return dagst, w, nil
|
||||
}
|
||||
|
||||
func extractDAGStoreConfig(r repo.LockedRepo) (config.DAGStoreConfig, error) {
|
||||
cfg, err := r.Config()
|
||||
if err != nil {
|
||||
return config.DAGStoreConfig{}, xerrors.Errorf("could not load config: %w", err)
|
||||
}
|
||||
mcfg, ok := cfg.(*config.StorageMiner)
|
||||
if !ok {
|
||||
return config.DAGStoreConfig{}, xerrors.Errorf("config not expected type; expected config.StorageMiner, got: %T", cfg)
|
||||
}
|
||||
return mcfg.DAGStore, nil
|
||||
}
|
||||
+1
-19
@@ -49,31 +49,13 @@ const (
|
||||
StorageMiner
|
||||
Worker
|
||||
Wallet
|
||||
Markets
|
||||
)
|
||||
|
||||
func (t RepoType) String() string {
|
||||
s := [...]string{
|
||||
"__invalid__",
|
||||
"FullNode",
|
||||
"StorageMiner",
|
||||
"Worker",
|
||||
"Wallet",
|
||||
"Markets",
|
||||
}
|
||||
if t < 0 || int(t) > len(s) {
|
||||
return "__invalid__"
|
||||
}
|
||||
return s[t]
|
||||
}
|
||||
|
||||
func defConfForType(t RepoType) interface{} {
|
||||
switch t {
|
||||
case FullNode:
|
||||
return config.DefaultFullNode()
|
||||
case StorageMiner, Markets:
|
||||
// markets is a specialised miner service
|
||||
// this taxonomy needs to be cleaned up
|
||||
case StorageMiner:
|
||||
return config.DefaultStorageMiner()
|
||||
case Worker:
|
||||
return &struct{}{}
|
||||
|
||||
@@ -1,170 +0,0 @@
|
||||
- github.com/filecoin-project/lotus:
|
||||
- Merge branch 'releases' into release/v1.11.1
|
||||
- Update to proof v8.0.3 ([filecoin-project/lotus#6890](https://github.com/filecoin-project/lotus/pull/6890))
|
||||
- lotus-shed: initial export cmd for markets related metadata ([filecoin-project/lotus#6840](https://github.com/filecoin-project/lotus/pull/6840))
|
||||
- add a very verbose -vv flag to lotus and lotus-miner. ([filecoin-project/lotus#6888](https://github.com/filecoin-project/lotus/pull/6888))
|
||||
- Update RELEASE_ISSUE_TEMPLATE.md ([filecoin-project/lotus#6880](https://github.com/filecoin-project/lotus/pull/6880))
|
||||
- Moving GC for badger ([filecoin-project/lotus#6854](https://github.com/filecoin-project/lotus/pull/6854))
|
||||
- Add github actions for staled pr ([filecoin-project/lotus#6879](https://github.com/filecoin-project/lotus/pull/6879))
|
||||
- Add allocated sectorid vis ([filecoin-project/lotus#4638](https://github.com/filecoin-project/lotus/pull/4638))
|
||||
- rename `cmd/lotus{-storage=>}-miner` to match binary. ([filecoin-project/lotus#6886](https://github.com/filecoin-project/lotus/pull/6886))
|
||||
- update to go-fil-market v1.6.0 ([filecoin-project/lotus#6885](https://github.com/filecoin-project/lotus/pull/6885))
|
||||
- Bump go-multihash, adjust test for supported version ([filecoin-project/lotus#6674](https://github.com/filecoin-project/lotus/pull/6674))
|
||||
- Fix padding of deals, which only partially shipped in #5988 ([filecoin-project/lotus#6683](https://github.com/filecoin-project/lotus/pull/6683))
|
||||
- fix racy TestSimultanenousTransferLimit. ([filecoin-project/lotus#6862](https://github.com/filecoin-project/lotus/pull/6862))
|
||||
- Improve splitstore warmup ([filecoin-project/lotus#6867](https://github.com/filecoin-project/lotus/pull/6867))
|
||||
- ValidateBlock: Assert that block header height's are greater than parents ([filecoin-project/lotus#6872](https://github.com/filecoin-project/lotus/pull/6872))
|
||||
- feat: Don't panic when api impl is nil ([filecoin-project/lotus#6857](https://github.com/filecoin-project/lotus/pull/6857))
|
||||
- splitstore shed utils ([filecoin-project/lotus#6811](https://github.com/filecoin-project/lotus/pull/6811))
|
||||
- Fix links in issue templates
|
||||
- Update issue templates and add templates for M1 ([filecoin-project/lotus#6856](https://github.com/filecoin-project/lotus/pull/6856))
|
||||
- Splitstore: support on-disk marksets using badger ([filecoin-project/lotus#6833](https://github.com/filecoin-project/lotus/pull/6833))
|
||||
- Config UX improvements ([filecoin-project/lotus#6848](https://github.com/filecoin-project/lotus/pull/6848))
|
||||
- fix deal concurrency test failures by upgrading graphsync and others ([filecoin-project/lotus#6724](https://github.com/filecoin-project/lotus/pull/6724))
|
||||
- Update issue templates to forms ([filecoin-project/lotus#6798](https://github.com/filecoin-project/lotus/pull/6798))
|
||||
- Nerpa v13 upgrade ([filecoin-project/lotus#6837](https://github.com/filecoin-project/lotus/pull/6837))
|
||||
- add docker-compose file ([filecoin-project/lotus#6544](https://github.com/filecoin-project/lotus/pull/6544))
|
||||
- fix warmup by decoupling state from message receipt walk ([filecoin-project/lotus#6841](https://github.com/filecoin-project/lotus/pull/6841))
|
||||
- add a command for compacting sector numbers bitfield ([filecoin-project/lotus#4640](https://github.com/filecoin-project/lotus/pull/4640))
|
||||
- PriceListByVersion ([filecoin-project/lotus#6766](https://github.com/filecoin-project/lotus/pull/6766))
|
||||
- easy way to make install app ([filecoin-project/lotus#5183](https://github.com/filecoin-project/lotus/pull/5183))
|
||||
- api: Separate the Net interface from Common ([filecoin-project/lotus#6627](https://github.com/filecoin-project/lotus/pull/6627))
|
||||
- cache loaded block messages ([filecoin-project/lotus#6760](https://github.com/filecoin-project/lotus/pull/6760))
|
||||
- fix: on randomness change, use new rand ([filecoin-project/lotus#6805](https://github.com/filecoin-project/lotus/pull/6805))
|
||||
- Splitstore: add retention policy option for keeping messages in the hotstore ([filecoin-project/lotus#6775](https://github.com/filecoin-project/lotus/pull/6775))
|
||||
- Introduce the LOTUS_CHAIN_BADGERSTORE_DISABLE_FSYNC envvar ([filecoin-project/lotus#6817](https://github.com/filecoin-project/lotus/pull/6817))
|
||||
- add StateReadState to gateway api ([filecoin-project/lotus#6818](https://github.com/filecoin-project/lotus/pull/6818))
|
||||
- add SealProof in SectorBuilder ([filecoin-project/lotus#6815](https://github.com/filecoin-project/lotus/pull/6815))
|
||||
- release -> master ([filecoin-project/lotus#6828](https://github.com/filecoin-project/lotus/pull/6828))
|
||||
- sealing: Handle preCommitParams errors more correctly ([filecoin-project/lotus#6763](https://github.com/filecoin-project/lotus/pull/6763))
|
||||
- fix: always check if StateSearchMessage returns nil ([filecoin-project/lotus#6802](https://github.com/filecoin-project/lotus/pull/6802))
|
||||
- ClientFindData: always fetch peer id from chain ([filecoin-project/lotus#6807](https://github.com/filecoin-project/lotus/pull/6807))
|
||||
- test: fix flaky window post tests ([filecoin-project/lotus#6804](https://github.com/filecoin-project/lotus/pull/6804))
|
||||
- ([filecoin-project/lotus#6800](https://github.com/filecoin-project/lotus/pull/6800))
|
||||
- fixes #6786 segfault ([filecoin-project/lotus#6787](https://github.com/filecoin-project/lotus/pull/6787))
|
||||
- Splitstore: add support for protecting out of chain references in the blockstore ([filecoin-project/lotus#6777](https://github.com/filecoin-project/lotus/pull/6777))
|
||||
- Resurrect CODEOWNERS, but for maintainers group ([filecoin-project/lotus#6773](https://github.com/filecoin-project/lotus/pull/6773))
|
||||
- update go-libp2p-pubsub to v0.5.0 ([filecoin-project/lotus#6764](https://github.com/filecoin-project/lotus/pull/6764))
|
||||
- Implement exposed splitstore ([filecoin-project/lotus#6762](https://github.com/filecoin-project/lotus/pull/6762))
|
||||
- Add ChainGetMessagesInTipset API ([filecoin-project/lotus#6642](https://github.com/filecoin-project/lotus/pull/6642))
|
||||
- test: handle null blocks in TestForkRefuseCall ([filecoin-project/lotus#6758](https://github.com/filecoin-project/lotus/pull/6758))
|
||||
- Master disclaimer ([filecoin-project/lotus#6757](https://github.com/filecoin-project/lotus/pull/6757))
|
||||
- Splitstore code reorg ([filecoin-project/lotus#6756](https://github.com/filecoin-project/lotus/pull/6756))
|
||||
- Create stale.yml ([filecoin-project/lotus#6747](https://github.com/filecoin-project/lotus/pull/6747))
|
||||
- Splitstore: Some small fixes ([filecoin-project/lotus#6754](https://github.com/filecoin-project/lotus/pull/6754))
|
||||
- ([filecoin-project/lotus#6746](https://github.com/filecoin-project/lotus/pull/6746))
|
||||
- Handle the --color flag via proper global state ([filecoin-project/lotus#6743](https://github.com/filecoin-project/lotus/pull/6743))
|
||||
- Config for collateral from miner available balance ([filecoin-project/lotus#6629](https://github.com/filecoin-project/lotus/pull/6629))
|
||||
- Support standalone miner-market process ([filecoin-project/lotus#6356](https://github.com/filecoin-project/lotus/pull/6356))
|
||||
- Splitstore Enhanchements ([filecoin-project/lotus#6474](https://github.com/filecoin-project/lotus/pull/6474))
|
||||
- ([filecoin-project/lotus#6739](https://github.com/filecoin-project/lotus/pull/6739))
|
||||
- Add more deal details to lotus-miner info ([filecoin-project/lotus#6708](https://github.com/filecoin-project/lotus/pull/6708))
|
||||
- Release template: Update all testnet infra at once ([filecoin-project/lotus#6710](https://github.com/filecoin-project/lotus/pull/6710))
|
||||
- Fix Lotus shed
|
||||
- Fix bugs in sectors extend --v1-sectors ([filecoin-project/lotus#6066](https://github.com/filecoin-project/lotus/pull/6066))
|
||||
- add election backtest ([filecoin-project/lotus#5950](https://github.com/filecoin-project/lotus/pull/5950))
|
||||
- Envvar to disable slash filter ([filecoin-project/lotus#6620](https://github.com/filecoin-project/lotus/pull/6620))
|
||||
- Release Template: remove binary validation step ([filecoin-project/lotus#6709](https://github.com/filecoin-project/lotus/pull/6709))
|
||||
- Config for deal publishing control addresses ([filecoin-project/lotus#6697](https://github.com/filecoin-project/lotus/pull/6697))
|
||||
- Reset of the interop network ([filecoin-project/lotus#6689](https://github.com/filecoin-project/lotus/pull/6689))
|
||||
- Enable color by default only if os.Stdout is a TTY ([filecoin-project/lotus#6696](https://github.com/filecoin-project/lotus/pull/6696))
|
||||
- Stop outputing ANSI color on non-TTY ([filecoin-project/lotus#6694](https://github.com/filecoin-project/lotus/pull/6694))
|
||||
- add dollar sign ([filecoin-project/lotus#6690](https://github.com/filecoin-project/lotus/pull/6690))
|
||||
- get-actor cli spelling fix ([filecoin-project/lotus#6681](https://github.com/filecoin-project/lotus/pull/6681))
|
||||
- fix "lotus-seed genesis car" error "merkledag: not found" ([filecoin-project/lotus#6688](https://github.com/filecoin-project/lotus/pull/6688))
|
||||
- polish(statetree): accept a context in statetree diff for timeouts ([filecoin-project/lotus#6639](https://github.com/filecoin-project/lotus/pull/6639))
|
||||
- Add helptext to lotus chain export ([filecoin-project/lotus#6672](https://github.com/filecoin-project/lotus/pull/6672))
|
||||
- Get retrieval pricing input should not error out on a deal state fetch ([filecoin-project/lotus#6679](https://github.com/filecoin-project/lotus/pull/6679))
|
||||
- Fix more CID double-encoding as hex ([filecoin-project/lotus#6680](https://github.com/filecoin-project/lotus/pull/6680))
|
||||
- add an incremental nonce itest. ([filecoin-project/lotus#6663](https://github.com/filecoin-project/lotus/pull/6663))
|
||||
- storage: Fix FinalizeSector with sectors in stoage paths ([filecoin-project/lotus#6653](https://github.com/filecoin-project/lotus/pull/6653))
|
||||
- Fix tiny error in check-client-datacap ([filecoin-project/lotus#6664](https://github.com/filecoin-project/lotus/pull/6664))
|
||||
- Fix: precommit_batch method used the wrong cfg.CommitBatchWait ([filecoin-project/lotus#6658](https://github.com/filecoin-project/lotus/pull/6658))
|
||||
- fix ticket expiration check ([filecoin-project/lotus#6635](https://github.com/filecoin-project/lotus/pull/6635))
|
||||
- commit batch: AggregateAboveBaseFee config ([filecoin-project/lotus#6650](https://github.com/filecoin-project/lotus/pull/6650))
|
||||
- commit batch: Initialize the FailedSectors map ([filecoin-project/lotus#6647](https://github.com/filecoin-project/lotus/pull/6647))
|
||||
- Fast-path retry submitting commit aggregate if commit is still valid ([filecoin-project/lotus#6638](https://github.com/filecoin-project/lotus/pull/6638))
|
||||
- remove precommit check in handleCommitFailed ([filecoin-project/lotus#6634](https://github.com/filecoin-project/lotus/pull/6634))
|
||||
- Reuse timers in sealing batch logic ([filecoin-project/lotus#6636](https://github.com/filecoin-project/lotus/pull/6636))
|
||||
- shed tool to estimate aggregate network fees ([filecoin-project/lotus#6631](https://github.com/filecoin-project/lotus/pull/6631))
|
||||
- fix prove commit aggregate send token amount ([filecoin-project/lotus#6625](https://github.com/filecoin-project/lotus/pull/6625))
|
||||
- Update version.go to 1.11.1 ([filecoin-project/lotus#6621](https://github.com/filecoin-project/lotus/pull/6621))
|
||||
- github.com/filecoin-project/go-data-transfer (v1.6.0 -> v1.7.0):
|
||||
- release: v1.7.0
|
||||
- Fire a transfer queued event when a transfer is queued in Graphsync (#221) ([filecoin-project/go-data-transfer#221](https://github.com/filecoin-project/go-data-transfer/pull/221))
|
||||
- feat: pass ChannelID to ValidatePush & ValidatePull (#220) ([filecoin-project/go-data-transfer#220](https://github.com/filecoin-project/go-data-transfer/pull/220))
|
||||
- release: v1.6.1 ([filecoin-project/go-data-transfer#218](https://github.com/filecoin-project/go-data-transfer/pull/218))
|
||||
- Remove CID lists (#217) ([filecoin-project/go-data-transfer#217](https://github.com/filecoin-project/go-data-transfer/pull/217))
|
||||
- Merge v1.6.0 ([filecoin-project/go-data-transfer#214](https://github.com/filecoin-project/go-data-transfer/pull/214))
|
||||
- Remove restart ack timeout (#211) ([filecoin-project/go-data-transfer#211](https://github.com/filecoin-project/go-data-transfer/pull/211))
|
||||
- feat: use different extension names to fit multiple hooks data in same graphsync message (#204) ([filecoin-project/go-data-transfer#204](https://github.com/filecoin-project/go-data-transfer/pull/204))
|
||||
- fix: map race in GS transport (#208) ([filecoin-project/go-data-transfer#208](https://github.com/filecoin-project/go-data-transfer/pull/208))
|
||||
- refactor: simplify graphsync transport (#203) ([filecoin-project/go-data-transfer#203](https://github.com/filecoin-project/go-data-transfer/pull/203))
|
||||
- release: v1.5.0 (#200) ([filecoin-project/go-data-transfer#200](https://github.com/filecoin-project/go-data-transfer/pull/200))
|
||||
- github.com/filecoin-project/go-fil-markets (v1.5.0 -> v1.6.0):
|
||||
- release: v1.6.0
|
||||
- support padding out smaller files (#536) ([filecoin-project/go-fil-markets#536](https://github.com/filecoin-project/go-fil-markets/pull/536))
|
||||
- On overloaded CI 10 seconds just isn't enough (#587) ([filecoin-project/go-fil-markets#587](https://github.com/filecoin-project/go-fil-markets/pull/587))
|
||||
- Do not hex-encode CIDs in logs (#561) ([filecoin-project/go-fil-markets#561](https://github.com/filecoin-project/go-fil-markets/pull/561))
|
||||
- remove wrong peer check in push deal validation (#585) ([filecoin-project/go-fil-markets#585](https://github.com/filecoin-project/go-fil-markets/pull/585))
|
||||
- fix: circleci docs-gen task (#574) ([filecoin-project/go-fil-markets#574](https://github.com/filecoin-project/go-fil-markets/pull/574))
|
||||
- Storage market request queued event and validation interface changes (#555) ([filecoin-project/go-fil-markets#555](https://github.com/filecoin-project/go-fil-markets/pull/555))
|
||||
- build(deps): bump ws from 6.2.1 to 6.2.2 (#554) ([filecoin-project/go-fil-markets#554](https://github.com/filecoin-project/go-fil-markets/pull/554))
|
||||
- release: v1.5.0 ([filecoin-project/go-fil-markets#553](https://github.com/filecoin-project/go-fil-markets/pull/553))
|
||||
- github.com/filecoin-project/go-padreader (v0.0.0-20200903213702-ed5fae088b20 -> v0.0.0-20210723183308-812a16dc01b1):
|
||||
- New method to pad harder (#6) ([filecoin-project/go-padreader#6](https://github.com/filecoin-project/go-padreader/pull/6))
|
||||
- Create SECURITY.md (#5) ([filecoin-project/go-padreader#5](https://github.com/filecoin-project/go-padreader/pull/5))
|
||||
- github.com/filecoin-project/go-state-types (v0.1.1-0.20210506134452-99b279731c48 -> v0.1.1-0.20210722133031-ad9bfe54c124):
|
||||
- Add version 6.5 (#30) ([filecoin-project/go-state-types#30](https://github.com/filecoin-project/go-state-types/pull/30))
|
||||
- rename file
|
||||
- github.com/filecoin-project/specs-actors/v5 (v5.0.1 -> v5.0.3):
|
||||
- Adjust code for subtle change in go-multihash 0.0.15 (#1463) ([filecoin-project/specs-actors#1463](https://github.com/filecoin-project/specs-actors/pull/1463))
|
||||
- Bump go state types (#1464) ([filecoin-project/specs-actors#1464](https://github.com/filecoin-project/specs-actors/pull/1464))
|
||||
- Create CODEOWNERS (#1465) ([filecoin-project/specs-actors#1465](https://github.com/filecoin-project/specs-actors/pull/1465))
|
||||
- Test deterministic offset (#1462) ([filecoin-project/specs-actors#1462](https://github.com/filecoin-project/specs-actors/pull/1462))
|
||||
|
||||
Contributors
|
||||
|
||||
| Contributor | Commits | Lines ± | Files Changed |
|
||||
|-------------|---------|---------|---------------|
|
||||
| vyzo | 295 | +8700/-5936 | 397 |
|
||||
| Anton Evangelatov | 94 | +4680/-2965 | 277 |
|
||||
| Łukasz Magiera | 37 | +3851/-1611 | 146 |
|
||||
| Mike Greenberg | 1 | +2310/-578 | 8 |
|
||||
| dirkmc | 7 | +1154/-726 | 29 |
|
||||
| Jennifer Wang | 9 | +485/-341 | 26 |
|
||||
| Peter Rabbitson | 18 | +469/-273 | 64 |
|
||||
| Cory Schwartz | 5 | +576/-135 | 14 |
|
||||
| hunjixin | 7 | +404/-82 | 19 |
|
||||
| ZenGround0 | 17 | +284/-135 | 44 |
|
||||
| Dirk McCormick | 17 | +348/-47 | 17 |
|
||||
| Raúl Kripalani | 18 | +254/-97 | 62 |
|
||||
| tchardin | 1 | +261/-33 | 4 |
|
||||
| Jakub Sztandera | 4 | +254/-16 | 4 |
|
||||
| Aarsh Shah | 2 | +196/-40 | 28 |
|
||||
| whyrusleeping | 3 | +150/-9 | 8 |
|
||||
| Whyrusleeping | 2 | +87/-66 | 10 |
|
||||
| Aayush Rajasekaran | 10 | +81/-53 | 13 |
|
||||
| zgfzgf | 2 | +104/-4 | 2 |
|
||||
| aarshkshah1992 | 4 | +73/-7 | 6 |
|
||||
| llifezou | 4 | +59/-20 | 4 |
|
||||
| Steven Allen | 7 | +47/-17 | 9 |
|
||||
| johnli-helloworld | 3 | +46/-15 | 5 |
|
||||
| frrist | 1 | +28/-23 | 2 |
|
||||
| Jennifer | 4 | +31/-2 | 4 |
|
||||
| wangchao | 1 | +1/-27 | 1 |
|
||||
| Jiaying Wang | 2 | +7/-21 | 2 |
|
||||
| hannahhoward | 3 | +21/-2 | 3 |
|
||||
| chadwick2143 | 1 | +15/-1 | 1 |
|
||||
| Jerry | 2 | +9/-4 | 2 |
|
||||
| Steve Loeppky | 2 | +12/-0 | 2 |
|
||||
| David Dias | 1 | +9/-0 | 1 |
|
||||
| dependabot[bot] | 1 | +3/-3 | 1 |
|
||||
| zhoutian527 | 1 | +2/-2 | 1 |
|
||||
| xloem | 1 | +4/-0 | 1 |
|
||||
| Travis Person | 2 | +2/-2 | 3 |
|
||||
| Liviu Damian | 2 | +2/-2 | 2 |
|
||||
| Jim Pick | 2 | +2/-2 | 2 |
|
||||
| Frank | 1 | +3/-0 | 1 |
|
||||
| turuslan | 1 | +1/-1 | 1 |
|
||||
| Kirk Baird | 1 | +0/-0 | 1 |
|
||||
Reference in New Issue
Block a user