cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
14,306 Commits 24 Branches 342 Tags 314 MiB
229cfafa8d
Commit Graph

336 Commits

Author SHA1 Message Date
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
Aayush Rajasekaran
a52352b13a PreCommitPolicy: Don't try to align expirations on proving period boundaries 2021-08-11 12:44:12 -04:00
ZenGround0
6da26dcefc Remove network version panic 2021-08-10 13:07:30 -04:00
mx
cfc10e9b52 fix: more logging in maybeStartBatch error 2021-08-06 16:50:37 +08:00
He Weidong
4273fb067c reduce diff 2021-07-29 15:01:52 +08:00
He Weidong
ef6620ad8e reasonable max value for initial sector expiration 2021-07-29 14:57:06 +08:00
He Weidong
ee20d30480 reasonable min and max value for initial sector expiration 2021-07-29 14:55:11 +08:00
Steven Allen
dbe4891a39
Merge branch 'master' into fix/check_ticket_msg 2021-07-28 15:25:40 -07:00
hunjixin
ac8937245c fix ticket check 2021-07-27 16:21:47 +08:00
Mike Greenberg
41bce7925a fix: Remove unnecessary Truncate 2021-07-23 20:12:30 -04:00
Mike Greenberg
3829d6bd9a fix: Remove actor method from autogen source; Move inline to pkg 2021-07-22 15:10:49 -04:00
Mike Greenberg
123a976f62 chore: Add unit test for custom CC lifetime value expiration 2021-07-22 15:07:03 -04:00
Mike Greenberg
7ee46ad4e0 fix: PreCommitPolicy unit tests 2021-07-22 15:07:03 -04:00
Mike Greenberg
adb62a3fff chore: Move cfg getter into PCPolicy; Clamp values on get 2021-07-22 15:07:03 -04:00
Mike Greenberg
9e34cce1f2 feat(config): support configurable CC sector expiration 2021-07-22 15:07:03 -04:00
Łukasz Magiera
8029a9ffe3 sealing: Address riview on preCommitParams send fix 2021-07-20 17:46:19 -07:00
johnli-helloworld
aea338ecf0 fix handleSubmitPreCommitBatch() preCommitParams error 2021-07-20 17:46:19 -07:00
Steven Allen
865e7ca80a test(sealing): test message not found 2021-07-20 12:09:50 -07:00
Steven Allen
ad1dd36193 fix: better error message on missing message 2021-07-20 11:20:45 -07:00
xloem
e0208e3595 fixes #6786 segfault 2021-07-20 11:20:45 -07:00
Łukasz Magiera
583a8a13d9
Merge pull request #6629 from filecoin-project/feat/pledge-from-miner-balance 
Config for collateral from miner available balance
 2021-07-13 17:22:00 +02:00
Łukasz Magiera
0678b3fa05 sealing: Cleanup DisableCollateralFallback handling 2021-07-13 17:06:41 +02:00
Łukasz Magiera
7526a074d9 sealing: collateral buffer / falback config 2021-07-13 17:04:58 +02:00
Łukasz Magiera
83f2368507 Add CollateralFromMinerBalance config 2021-07-12 14:11:58 +02:00
Anton Evangelatov
ff2772a58c resolved conflicts 2021-07-06 16:00:41 +02:00
Anton Evangelatov
4be0a7a215 resolve conflicts with master 2021-07-05 13:13:32 +02:00
Peter Rabbitson
a544fa0f97 Fix more CID doucle-encoding as hex 2021-07-05 12:51:15 +02:00
Łukasz Magiera
35c0e3cf8c
Merge pull request #6658 from sunday527/master 
Fix: precommit_batch method used the wrong cfg.CommitBatchWait
 2021-07-02 15:18:48 +02:00
Łukasz Magiera
60ecbdee8f
Merge pull request #6635 from llifezou/fix_checkTicketExpired 
fix ticket expiration check
 2021-07-02 15:18:04 +02:00
zhoutian527
71e1577a62 Fix: precommit_batch method used the wrong cfg.PreCommitBatchWait 2021-07-02 15:53:21 +08:00
llifezou
df53b97fe4
Update extern/storage-sealing/states_sealing.go 
fix sector precommitted but expired judgment

Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-02 11:44:46 +08:00
llifezou
fe3ad4fd60
Update extern/storage-sealing/states_sealing.go 
fix log

Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2021-07-02 11:38:04 +08:00
Łukasz Magiera
7c2c8b2a95 commit batch: Regression test nil FailedSectors map 2021-07-01 13:51:11 +02:00
Łukasz Magiera
c094aa82ec commit batch: AggregateAboveBaseFee config 2021-07-01 13:33:54 +02:00
Łukasz Magiera
88bb9f422e commit batch: Initialize the FailedSectors map 2021-07-01 12:15:58 +02:00
llifezou
73e58f7af1 fix getTicket: sector precommitted but expired case 2021-07-01 10:53:42 +08:00
johnli-helloworld
229d5e5c80 handleSubmitCommitAggregate() exception handling 2021-06-30 15:29:21 +02:00
Łukasz Magiera
2dad06d59f
Merge pull request #6634 from GFZRZK/GFZRZK/correct_check_commit 
remove precommit check in handleCommitFailed
 2021-06-30 15:23:33 +02:00
Łukasz Magiera
f45340461d
gofmt 2021-06-30 15:19:29 +02:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
johnli-helloworld
b0b070ffe7 to optimize the batchwait 2021-06-30 16:56:40 +08:00
llifezou
3dd3476bfd fix ticket expiration check, otherwise it may cause a large number of loops to retry GetTicket when retrying PreCommit1 2021-06-30 16:32:44 +08:00
wangchao
a4342f3997 remove precommit check in handleCommitFailed 2021-06-30 11:09:07 +08:00
Łukasz Magiera
7b1b082381 Merge branch 'master' into fix/prove-commit-aggregate 2021-06-29 11:48:52 +02:00
Jerry
73704c92ba ensure agg fee is adequate 2021-06-29 14:36:26 +08:00
Jerry
5c3d67a811 fix: miner balance is not enough, so that ProveCommitAggregate msg exec failed 2021-06-29 14:06:41 +08:00
Łukasz Magiera
84dbcb63d2
Merge pull request #6566 from GFZRZK/GFZRZK/correct_sector_expire_time 
scale up sector expiration to avoid sector expire in batch-pre-commit waitting
 2021-06-24 10:59:49 +02:00
Łukasz Magiera
c3480dc0e8 Merge remote-tracking branch 'origin/releases' into chore/merge-release 2021-06-24 09:39:11 +02:00
Aayush Rajasekaran
616e5688fc Remove MinPreCommitBatch 2021-06-23 12:30:32 -04:00
wangchao
03261771e2 scale up sector expiration to avoid sector expire in batch-pre-commit waitting 2021-06-23 14:10:28 +08:00
Aayush Rajasekaran
93f7cbe587 Add a helpful comment 2021-06-22 19:31:36 -04:00
Aayush Rajasekaran
6ca2a14819 Always flush when timer goes off 2021-06-22 19:16:36 -04:00
Anton Evangelatov
6720463799 resolve merge conflicts 2021-06-22 11:28:23 +02:00
Łukasz Magiera
b37a66c7c9 Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-21 17:03:03 +02:00
yaohcn
7c6736b3ef fix commit finalize failed 2021-06-18 12:09:02 +08:00
Łukasz Magiera
0a5a7cf45d storagefsm: Fix too-long log handling 2021-06-17 20:52:46 -04:00
Łukasz Magiera
b0128bd99e storagefsm: Fix race spawning more than one new sector at once 2021-06-17 20:52:42 -04:00
Łukasz Magiera
097baeb9b0 Make batch deal input test less flaky 2021-06-17 20:52:09 -04:00
Łukasz Magiera
5c8498b603 storagefsm: Fix batch deal packing behavior 2021-06-17 20:50:14 -04:00
Anton Evangelatov
d45bb14015 Merge branch 'master' into nonsense/split-market-miner-processes 2021-06-16 18:49:15 +02:00
Łukasz Magiera
1f914053b4 sealing: Wire up context to batchers 2021-06-16 11:43:06 +02:00
Łukasz Magiera
81b412399e sealing: Fix restartSectors race 2021-06-15 17:55:58 -04:00
Łukasz Magiera
74db586fdf sealing: Fix restartSectors race 2021-06-15 17:43:32 -04:00
Aayush Rajasekaran
520a0091f5 Merge branch 'releases' into release/v1.10.0 2021-06-15 17:40:36 -04:00
Łukasz Magiera
f9596dd730
Merge pull request #6432 from filecoin-project/tests/batching-unit 
Unit tests for sector batchers
 2021-06-14 21:05:34 +02:00
wangchao
4da30931c9 failed sectors should be added into res correctly 2021-06-14 11:33:38 -04:00
Łukasz Magiera
0514f38dc4 Merge remote-tracking branch 'origin/master' into chore/merge-1.10 2021-06-14 13:27:53 +02:00
Anton Evangelatov
31e6fb154e resolve conflicts 2021-06-14 11:27:54 +02:00
wangchao
f392c1295c failed sectors should be added into res correctly 2021-06-12 23:22:32 +08:00
Łukasz Magiera
733240a2bb sealing: Test early finalization fsm planners 2021-06-11 11:52:00 +02:00
Łukasz Magiera
05d9b5ce0f sealing: Add missing planner for CommitFinalizeFailed 2021-06-11 11:44:37 +02:00
Łukasz Magiera
ec06f086ef sealing: Early finalization option 2021-06-11 11:41:28 +02:00
Łukasz Magiera
06195bc8e1 Unit tests for sector batchers 2021-06-10 13:46:36 +02:00
Łukasz Magiera
f0a2e97cb5 fee config for sector batching 2021-06-09 14:23:06 -04:00
Aayush Rajasekaran
dede1b4758 Merge branch 'release/v1.10.0' into asr/merge-release 2021-06-09 14:05:41 -04:00
Aayush Rajasekaran
b0c9dd49f0 Fund miners with the aggregate fee when ProveCommitting 2021-06-09 13:08:54 -04:00
Aayush Rajasekaran
dbb4e9fcc5 Drop soms logs 2021-06-09 12:26:20 -04:00
Łukasz Magiera
92bb874327
Use correct batch fee config in commit batcher 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2021-06-09 12:17:18 +02:00
Aayush Rajasekaran
ba27d45214 Split the getSectorCutoff methods between precommit and commit batchers 2021-06-08 16:46:35 -04:00
Łukasz Magiera
78171055e7 fee config for sector batching 2021-06-08 15:45:32 +02:00
Aayush Rajasekaran
b13169f071 Rename deadlines to cutoffs in the batchers 2021-06-07 20:20:06 -04:00
Anton Evangelatov
8bd09e39ca resolve merge conflicts 2021-06-04 16:17:00 +02:00
Łukasz Magiera
dc642d0b7b Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11 2021-06-01 21:06:58 +02:00
Łukasz Magiera
8b84e499c5 storagefsm: Handle preCommitParams errors better 2021-06-01 20:36:20 +02:00
Łukasz Magiera
482e1110c2 precommit batcher: Improve error propagation 2021-06-01 14:35:30 +02:00
Łukasz Magiera
cb4eb487f4 commit batcher: Fix min aggregate size check 2021-06-01 12:27:22 +02:00
Łukasz Magiera
084b0e7f60 Handle collateral when submitting aggregated commits 2021-06-01 12:09:15 +02:00
Łukasz Magiera
9fcb564bef Make commit batcher more robust 2021-06-01 11:56:19 +02:00
Anton Evangelatov
0a79f49dc5 fix tests 2021-06-01 11:45:34 +02:00
Łukasz Magiera
ffa47659a1 Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11 2021-05-31 21:38:34 +02:00
Łukasz Magiera
c3e8eddb9b Merge remote-tracking branch 'origin/master' into feat/nv13-1.11 2021-05-31 21:24:56 +02:00
Aayush Rajasekaran
61554cf3e0 Update to latest actors 2021-05-31 14:47:41 -04:00
Anton Evangelatov
d195a12565 Merge branch 'master' into nonsense/split-market-miner-processes 2021-05-31 11:26:11 +02:00
Łukasz Magiera
f3bf773152 storagefsm: Fix too-long log handling 2021-05-30 19:24:42 +02:00
Łukasz Magiera
6e1919c67f storagefsm: Fix race spawning more than one new sector at once 2021-05-30 18:30:38 +02:00
Łukasz Magiera
9475079b97 Make batch deal input test less flaky 2021-05-30 17:46:22 +02:00
Łukasz Magiera
c8d603557b storagefsm: Fix batch deal packing behavior 2021-05-30 17:46:22 +02:00
Łukasz Magiera
1e4456138e Merge master into feat/nv13 2021-05-27 12:28:20 +02:00
yaohcn
d04e7d98ce Get current seal proof when necessary 2021-05-27 11:53:33 +08:00
Łukasz Magiera
fc76a09b19 mock: Use real aggregate lengths 2021-05-26 10:06:41 +02:00