zenground0
7d2b3f05db
WIP sector storage and integration test
2021-11-29 10:24:00 -05:00
Łukasz Magiera
05aa860459
Request correct read size with startOffset in pieceProvider
2021-11-27 00:05:45 +01:00
Łukasz Magiera
743ce5a40f
Add startOffset support to mock SectorMgr.ReadPiece
2021-11-26 18:48:52 +01:00
Łukasz Magiera
f6de16e95a
Fix sector-storage tests
2021-11-26 18:16:53 +01:00
Łukasz Magiera
8d955d5f30
dagstore mount: Add random access support
2021-11-26 17:40:53 +01:00
Łukasz Magiera
8454abcf45
storage: Use 1M buffers for Tar transfers
2021-11-24 20:08:37 +01:00
Łukasz Magiera
2a1505b364
storage: Test StorageFindSector with groups
2021-11-23 16:11:04 +01:00
Łukasz Magiera
8b548ac02f
storage: Check allowlists in StorageFindSector
2021-11-23 16:11:04 +01:00
Łukasz Magiera
5c77c25747
storage: Add Group tags to StorageInfo
2021-11-23 16:11:04 +01:00
Łukasz Magiera
d1a63e4173
remote store: Remove debug printf
2021-11-22 17:50:12 +01:00
Łukasz Magiera
2620eab45e
Merge pull request #7555 from diancun/dc/check-commit
...
checkCommit should return SectorCommitFailed
2021-11-22 12:28:16 +01:00
Aayush Rajasekaran
91fb111462
Update FFI
2021-11-17 17:50:36 -05:00
Aayush Rajasekaran
8665e32221
Update deps
2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
07f1be0ff6
Plug in the FFI call
2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
1449644c34
Integrate v7 actors
2021-11-17 15:55:14 -05:00
Jennifer Wang
1f339f623e
update to proof v10.1.0
2021-10-26 14:36:19 -04:00
yaohcn
3123400759
checkCommit should return SectorCommitFailed
2021-10-22 10:09:44 +08:00
Łukasz Magiera
f7884c4f45
Merge pull request #7527 from filecoin-project/feat/show-prepared-waitres
...
Show prepared tasks in sealing jobs
2021-10-20 15:22:35 +01:00
Łukasz Magiera
726d9b86ea
fix lint
2021-10-19 18:56:30 +02:00
Łukasz Magiera
852ac4c178
Expose per-state sector counts on the prometheus endpoint
2021-10-19 18:53:07 +02:00
Łukasz Magiera
e508055dc1
make gen
2021-10-19 11:13:23 +02:00
Łukasz Magiera
080aa3356a
Fix locks in worker-tracked
2021-10-18 20:19:21 +02:00
Łukasz Magiera
70589e4406
Block work in tracked worker before it is started
2021-10-18 18:44:56 +02:00
Łukasz Magiera
261238e157
Show prepared tasks in sealing jobs
2021-10-18 18:44:56 +02:00
Łukasz Magiera
11d738eee0
Track prepared work
2021-10-18 18:44:56 +02:00
Łukasz Magiera
98ff1c4d20
Merge pull request #7531 from marten-seemann/statfs-build-constraint
...
add missing build constraint to statfs_unix.go
2021-10-18 16:02:48 +01:00
Marten Seemann
03806f7063
add missing build constraint to statfs_unix.go
2021-10-18 16:29:01 +02:00
Łukasz Magiera
63c8b8edd1
Fix used sectors space accounting after AddPieceFailed
2021-10-18 09:55:28 +02:00
Łukasz Magiera
f352c18290
Don't remove sector data when moving data into a shared path
2021-10-11 21:11:38 +02:00
Łukasz Magiera
aed6e8d0be
Merge pull request #7335 from filecoin-project/feat/sched-ready-work
...
sealing: Improve scheduling of ready work
2021-10-07 18:49:02 +01:00
Łukasz Magiera
1993efe201
Merge pull request #7444 from filecoin-project/feat/ap-fail-retry
...
sealing: Recover sectors after failed AddPiece
2021-10-07 03:18:15 +01:00
Łukasz Magiera
cadbd00ac4
sealing: Recover sectors after failed AddPiece
2021-10-04 20:00:07 +02:00
Jennifer Wang
38e01cba83
Merge branch 'release/v1.12.0' into jen/12tomaster
2021-10-04 02:38:00 -04:00
Łukasz Magiera
9af82f2d68
sched: Fix taskDone chan deadlock
2021-10-03 17:09:43 +02:00
Łukasz Magiera
b87142ec8e
wip improve scheduling of ready work
2021-10-03 10:38:08 +02:00
Jennifer Wang
2e1d463933
make lint happy
2021-10-01 20:36:13 -04:00
Jennifer Wang
5f8c44125b
Merge branch 'releases' into jen/12rc1prep
2021-10-01 20:31:29 -04:00
Łukasz Magiera
fc10281d96
Merge pull request #7420 from filecoin-project/deps/proofs-v10.0.0
...
Update proofs to v10.0.0
2021-10-01 22:20:09 +01:00
Aayush Rajasekaran
a335cb3767
Update to latest actors
2021-10-01 17:11:05 -04:00
Aayush Rajasekaran
f004d036dc
Set BatchPreCommitAboveBaseFee correctly
2021-10-01 14:23:41 -04:00
Aayush Rajasekaran
f8a89cafd8
Fix logs
2021-10-01 14:23:41 -04:00
Aayush Rajasekaran
d6dd86b419
Fix lint
2021-10-01 14:23:41 -04:00
Łukasz Magiera
6fd9d5f28b
Precommit batch balancer support/config
2021-10-01 14:23:41 -04:00
Łukasz Magiera
d7fbd8b67d
Update proofs to v10.0.0
2021-10-01 18:38:27 +02:00
Anton Evangelatov
809289f5ef
add Dealmaking.StartEpochSealingBuffer config
2021-10-01 17:44:15 +02:00
Anton Evangelatov
c2fa54ca9c
check for deal start epoch on SectorAddPieceToAny
2021-10-01 17:42:09 +02:00
Łukasz Magiera
a8a9818043
Expose storage states on the metrics endpoint
2021-10-01 14:45:01 +02:00
Aayush Rajasekaran
8756df2447
Fix tests after specs update
2021-09-29 21:27:49 -04:00
Aayush Rajasekaran
84b6734063
Incorporate the new PublishStorageDealsReturn
2021-09-29 19:13:13 -04:00
Jennifer Wang
d7690aa719
Merge branch 'releases' into jen/backport
2021-09-29 03:21:19 -04:00
Adrian Lanzafame
92190c7e0b
Fix formatting
2021-09-23 12:39:39 +10:00
Anton Evangelatov
0c1e29390a
unit test where StateMarketStorageDeal return nil, err
2021-09-21 22:13:51 -04:00
Anton Evangelatov
bf9d0bca40
GetCurrentDealInfo err: handle correctly err case
2021-09-21 22:13:46 -04:00
Jennifer Wang
e996c27619
update to ffi to update-bellperson-proofs-v9-0-2
2021-09-21 17:43:27 -04:00
Aayush Rajasekaran
b9bfcc4b69
Randomness: Move getters from ChainAPI to StateAPI
2021-09-21 14:20:15 -04:00
Łukasz Magiera
2437a4182d
Merge pull request #7322 from filecoin-project/asr/drand-state
...
Randomness: Move getters from ChainAPI to StateAPI
2021-09-21 09:20:12 +01:00
Anton Evangelatov
605953565b
unit test where StateMarketStorageDeal return nil, err
2021-09-17 15:14:53 +02:00
Anton Evangelatov
34e3dc2b37
GetCurrentDealInfo err: handle correctly err case
2021-09-17 14:55:56 +02:00
Łukasz Magiera
438d5ce78e
fix a panic in HandleRecoverDealIDs
2021-09-15 18:13:01 +02:00
Łukasz Magiera
ef03314c6d
storagemgr: Cleanup workerLk around worker resources
2021-09-15 16:35:19 +02:00
Aayush Rajasekaran
1d6db306ec
Randomness: Move getters from ChainAPI to StateAPI
2021-09-15 00:03:13 +02:00
swift-mx
b0b016b58c
fix: check padSector Cid
2021-09-10 18:13:07 +08:00
Jennifer Wang
f751291f1a
update to proof v0.9.2
2021-09-08 01:03:11 -04:00
frank
179458efe9
index out of range
2021-09-06 16:20:23 +08:00
Jennifer Wang
6379d5129f
integrate the proof patch: tag proofs-v9-revert-deps-hotfix
2021-09-02 15:40:15 -04:00
Łukasz Magiera
7ef1b62b41
sealing: Fix sector state accounting with FinalizeEarly
2021-09-02 19:27:10 +02:00
Łukasz Magiera
41db98d49f
sealing: Fix tests
2021-09-01 12:09:44 +02:00
Łukasz Magiera
b280e29409
sealing: Fix retry loop in SubmitCommitAggregate
2021-09-01 11:33:23 +02:00
Łukasz Magiera
3118bd1039
stores: Fix reserved disk usage log spam
2021-08-31 13:36:09 +02:00
Steven Allen
1cf556c3a2
feat: expose ChainGetPath on the gateway
2021-08-30 16:43:21 -07:00
Łukasz Magiera
2293ecd8e8
Reduce lotus-miner startup spam
2021-08-27 19:41:54 +02:00
Łukasz Magiera
b42171dc04
sealing: Check piece CIDs after AddPiece
2021-08-26 13:02:15 -07:00
Łukasz Magiera
62769e3b11
sealing: Fix RecoverDealIDs loop with changed PieceCID
2021-08-20 16:00:40 +02:00
Aayush Rajasekaran
113ea6dc4b
Update to latest FFI
2021-08-17 18:15:12 -04:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ( #6671 )
...
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.
Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.
When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.
Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.
Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.
Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.
Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.
A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.
Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.
NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:
- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport
At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Steven Allen
498644acab
Merge pull request #7011 from filecoin-project/fix/no-t-aux
...
fix: don't check for t_aux when proving
2021-08-11 11:28:29 -07:00
Aayush Rajasekaran
a52352b13a
PreCommitPolicy: Don't try to align expirations on proving period boundaries
2021-08-11 12:44:12 -04:00
ZenGround0
6da26dcefc
Remove network version panic
2021-08-10 13:07:30 -04:00
Steven Allen
18f39be3ba
fix: don't check for t_aux when proving
...
We don't need it.
2021-08-09 11:07:35 -07:00
mx
cfc10e9b52
fix: more logging in maybeStartBatch error
2021-08-06 16:50:37 +08:00
He Weidong
4273fb067c
reduce diff
2021-07-29 15:01:52 +08:00
He Weidong
ef6620ad8e
reasonable max value for initial sector expiration
2021-07-29 14:57:06 +08:00
He Weidong
ee20d30480
reasonable min and max value for initial sector expiration
2021-07-29 14:55:11 +08:00
Steven Allen
dbe4891a39
Merge branch 'master' into fix/check_ticket_msg
2021-07-28 15:25:40 -07:00
Aayush Rajasekaran
21244704fc
Merge branch 'master' into mg/feat/configurable-sector-expirations
2021-07-28 16:45:27 -04:00
Jennifer Wang
2790c9439f
Update to proof v8.0.3
2021-07-27 13:29:07 -04:00
hunjixin
ac8937245c
fix ticket check
2021-07-27 16:21:47 +08:00
Mike Greenberg
41bce7925a
fix: Remove unnecessary Truncate
2021-07-23 20:12:30 -04:00
Mike Greenberg
3829d6bd9a
fix: Remove actor method from autogen source; Move inline to pkg
2021-07-22 15:10:49 -04:00
Mike Greenberg
123a976f62
chore: Add unit test for custom CC lifetime value expiration
2021-07-22 15:07:03 -04:00
Mike Greenberg
7ee46ad4e0
fix: PreCommitPolicy unit tests
2021-07-22 15:07:03 -04:00
Mike Greenberg
adb62a3fff
chore: Move cfg getter into PCPolicy; Clamp values on get
2021-07-22 15:07:03 -04:00
Mike Greenberg
9e34cce1f2
feat(config): support configurable CC sector expiration
2021-07-22 15:07:03 -04:00
Łukasz Magiera
8029a9ffe3
sealing: Address riview on preCommitParams send fix
2021-07-20 17:46:19 -07:00
johnli-helloworld
aea338ecf0
fix handleSubmitPreCommitBatch() preCommitParams error
2021-07-20 17:46:19 -07:00
Steven Allen
865e7ca80a
test(sealing): test message not found
2021-07-20 12:09:50 -07:00
Steven Allen
ad1dd36193
fix: better error message on missing message
2021-07-20 11:20:45 -07:00
xloem
e0208e3595
fixes #6786 segfault
2021-07-20 11:20:45 -07:00
Łukasz Magiera
583a8a13d9
Merge pull request #6629 from filecoin-project/feat/pledge-from-miner-balance
...
Config for collateral from miner available balance
2021-07-13 17:22:00 +02:00
Łukasz Magiera
0678b3fa05
sealing: Cleanup DisableCollateralFallback handling
2021-07-13 17:06:41 +02:00