Darko Brdareski
e51ce5c508
Merge remote-tracking branch 'upstream/master' into bloxico/system-test-matrix
2022-01-27 10:57:56 +01:00
Aayush
e7123d1a8e
fix: sealer: correctly pipe through errors for SectorAbortUpgrade
2022-01-25 13:02:00 -05:00
Aayush
50aba9a8e6
fix: sealer: don't replica update sectors unless they have deals in them
2022-01-25 13:01:56 -05:00
Aayush
1b18236f91
feat: sealer: allow users to abort in-flight snap upgrades
2022-01-25 13:01:51 -05:00
Aayush Rajasekaran
0c9c94bad1
fix: checkReplica incorrectly returns ErrBadPR
2022-01-25 13:01:12 -05:00
Aayush Rajasekaran
3ff23ecbfa
:fix: checkReplica incorrectly returns ErrBadPR
2022-01-19 12:00:27 -05:00
Aayush Rajasekaran
aad8aa0893
Appease the linter
2022-01-14 17:15:44 -05:00
Jennifer Wang
30013c1f06
fix lint
2022-01-14 17:15:44 -05:00
Jennifer Wang
6901e998e6
Check piece before PRU2 instead of PRU1 as PRU2 is the heavy computation part
2022-01-14 17:15:44 -05:00
Jennifer Wang
8939d5982f
just use checkPiece
2022-01-14 17:15:44 -05:00
Jennifer Wang
a20916f9af
Add more deal expiration handling for snap deals
2022-01-14 17:15:44 -05:00
zenground0
5b0a0baa9a
Fix hande deal recover return value bug
2022-01-14 17:14:32 -05:00
zenground0
a9a523d8c0
Fix TooManyMarkedForUpgrade
2022-01-14 17:14:32 -05:00
zenground0
d6aa17e21f
Snap Deals Integration
...
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
bda4e5be95
Appease the linter
2022-01-12 18:10:07 -05:00
Jennifer Wang
fd50cd128a
fix lint
2022-01-11 18:34:26 -05:00
Jennifer Wang
e99b98873c
Check piece before PRU2 instead of PRU1 as PRU2 is the heavy computation part
2022-01-11 18:34:20 -05:00
Jennifer Wang
6b953a03d0
just use checkPiece
2022-01-11 18:34:15 -05:00
Jennifer Wang
37a3e610b7
Add more deal expiration handling for snap deals
2022-01-11 18:34:10 -05:00
zenground0
d16c5d0e93
Fix hande deal recover return value bug
2022-01-10 15:47:20 +05:30
zenground0
c309686679
Fix TooManyMarkedForUpgrade
2022-01-10 15:39:38 +05:30
zenground0
33f2d24f54
Snap Deals Integration
...
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-10 15:39:38 +05:30
Darko Brdareski
dda1a42a2a
Merge branch 'bloxico/system-test-matrix' of https://github.com/filecoin-project/lotus into merge_lotus
2021-12-20 15:48:16 +01:00
Darko Brdareski
0169d0dafd
Annotate state feature tests
2021-12-10 16:08:25 +01:00
yaohcn
3123400759
checkCommit should return SectorCommitFailed
2021-10-22 10:09:44 +08:00
Łukasz Magiera
726d9b86ea
fix lint
2021-10-19 18:56:30 +02:00
Łukasz Magiera
852ac4c178
Expose per-state sector counts on the prometheus endpoint
2021-10-19 18:53:07 +02:00
Łukasz Magiera
63c8b8edd1
Fix used sectors space accounting after AddPieceFailed
2021-10-18 09:55:28 +02:00
Łukasz Magiera
1993efe201
Merge pull request #7444 from filecoin-project/feat/ap-fail-retry
...
sealing: Recover sectors after failed AddPiece
2021-10-07 03:18:15 +01:00
Łukasz Magiera
cadbd00ac4
sealing: Recover sectors after failed AddPiece
2021-10-04 20:00:07 +02:00
Jennifer Wang
38e01cba83
Merge branch 'release/v1.12.0' into jen/12tomaster
2021-10-04 02:38:00 -04:00
Jennifer Wang
2e1d463933
make lint happy
2021-10-01 20:36:13 -04:00
Jennifer Wang
5f8c44125b
Merge branch 'releases' into jen/12rc1prep
2021-10-01 20:31:29 -04:00
Aayush Rajasekaran
a335cb3767
Update to latest actors
2021-10-01 17:11:05 -04:00
Aayush Rajasekaran
f004d036dc
Set BatchPreCommitAboveBaseFee correctly
2021-10-01 14:23:41 -04:00
Aayush Rajasekaran
f8a89cafd8
Fix logs
2021-10-01 14:23:41 -04:00
Aayush Rajasekaran
d6dd86b419
Fix lint
2021-10-01 14:23:41 -04:00
Łukasz Magiera
6fd9d5f28b
Precommit batch balancer support/config
2021-10-01 14:23:41 -04:00
Anton Evangelatov
809289f5ef
add Dealmaking.StartEpochSealingBuffer config
2021-10-01 17:44:15 +02:00
Anton Evangelatov
c2fa54ca9c
check for deal start epoch on SectorAddPieceToAny
2021-10-01 17:42:09 +02:00
Aayush Rajasekaran
8756df2447
Fix tests after specs update
2021-09-29 21:27:49 -04:00
Aayush Rajasekaran
84b6734063
Incorporate the new PublishStorageDealsReturn
2021-09-29 19:13:13 -04:00
Jennifer Wang
d7690aa719
Merge branch 'releases' into jen/backport
2021-09-29 03:21:19 -04:00
Adrian Lanzafame
92190c7e0b
Fix formatting
2021-09-23 12:39:39 +10:00
Anton Evangelatov
0c1e29390a
unit test where StateMarketStorageDeal return nil, err
2021-09-21 22:13:51 -04:00
Anton Evangelatov
bf9d0bca40
GetCurrentDealInfo err: handle correctly err case
2021-09-21 22:13:46 -04:00
Aayush Rajasekaran
b9bfcc4b69
Randomness: Move getters from ChainAPI to StateAPI
2021-09-21 14:20:15 -04:00
Łukasz Magiera
2437a4182d
Merge pull request #7322 from filecoin-project/asr/drand-state
...
Randomness: Move getters from ChainAPI to StateAPI
2021-09-21 09:20:12 +01:00
Anton Evangelatov
605953565b
unit test where StateMarketStorageDeal return nil, err
2021-09-17 15:14:53 +02:00
Anton Evangelatov
34e3dc2b37
GetCurrentDealInfo err: handle correctly err case
2021-09-17 14:55:56 +02:00
Łukasz Magiera
438d5ce78e
fix a panic in HandleRecoverDealIDs
2021-09-15 18:13:01 +02:00
Aayush Rajasekaran
1d6db306ec
Randomness: Move getters from ChainAPI to StateAPI
2021-09-15 00:03:13 +02:00
swift-mx
b0b016b58c
fix: check padSector Cid
2021-09-10 18:13:07 +08:00
frank
179458efe9
index out of range
2021-09-06 16:20:23 +08:00
Łukasz Magiera
7ef1b62b41
sealing: Fix sector state accounting with FinalizeEarly
2021-09-02 19:27:10 +02:00
Łukasz Magiera
41db98d49f
sealing: Fix tests
2021-09-01 12:09:44 +02:00
Łukasz Magiera
b280e29409
sealing: Fix retry loop in SubmitCommitAggregate
2021-09-01 11:33:23 +02:00
Łukasz Magiera
2293ecd8e8
Reduce lotus-miner startup spam
2021-08-27 19:41:54 +02:00
Łukasz Magiera
b42171dc04
sealing: Check piece CIDs after AddPiece
2021-08-26 13:02:15 -07:00
Łukasz Magiera
62769e3b11
sealing: Fix RecoverDealIDs loop with changed PieceCID
2021-08-20 16:00:40 +02:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ( #6671 )
...
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.
Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.
When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.
Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.
Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.
Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.
Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.
A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.
Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.
NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:
- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport
At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Aayush Rajasekaran
a52352b13a
PreCommitPolicy: Don't try to align expirations on proving period boundaries
2021-08-11 12:44:12 -04:00
ZenGround0
6da26dcefc
Remove network version panic
2021-08-10 13:07:30 -04:00
mx
cfc10e9b52
fix: more logging in maybeStartBatch error
2021-08-06 16:50:37 +08:00
He Weidong
4273fb067c
reduce diff
2021-07-29 15:01:52 +08:00
He Weidong
ef6620ad8e
reasonable max value for initial sector expiration
2021-07-29 14:57:06 +08:00
He Weidong
ee20d30480
reasonable min and max value for initial sector expiration
2021-07-29 14:55:11 +08:00
Steven Allen
dbe4891a39
Merge branch 'master' into fix/check_ticket_msg
2021-07-28 15:25:40 -07:00
hunjixin
ac8937245c
fix ticket check
2021-07-27 16:21:47 +08:00
Mike Greenberg
41bce7925a
fix: Remove unnecessary Truncate
2021-07-23 20:12:30 -04:00
Mike Greenberg
3829d6bd9a
fix: Remove actor method from autogen source; Move inline to pkg
2021-07-22 15:10:49 -04:00
Mike Greenberg
123a976f62
chore: Add unit test for custom CC lifetime value expiration
2021-07-22 15:07:03 -04:00
Mike Greenberg
7ee46ad4e0
fix: PreCommitPolicy unit tests
2021-07-22 15:07:03 -04:00
Mike Greenberg
adb62a3fff
chore: Move cfg getter into PCPolicy; Clamp values on get
2021-07-22 15:07:03 -04:00
Mike Greenberg
9e34cce1f2
feat(config): support configurable CC sector expiration
2021-07-22 15:07:03 -04:00
Łukasz Magiera
8029a9ffe3
sealing: Address riview on preCommitParams send fix
2021-07-20 17:46:19 -07:00
johnli-helloworld
aea338ecf0
fix handleSubmitPreCommitBatch() preCommitParams error
2021-07-20 17:46:19 -07:00
Steven Allen
865e7ca80a
test(sealing): test message not found
2021-07-20 12:09:50 -07:00
Steven Allen
ad1dd36193
fix: better error message on missing message
2021-07-20 11:20:45 -07:00
xloem
e0208e3595
fixes #6786 segfault
2021-07-20 11:20:45 -07:00
Łukasz Magiera
583a8a13d9
Merge pull request #6629 from filecoin-project/feat/pledge-from-miner-balance
...
Config for collateral from miner available balance
2021-07-13 17:22:00 +02:00
Łukasz Magiera
0678b3fa05
sealing: Cleanup DisableCollateralFallback handling
2021-07-13 17:06:41 +02:00
Łukasz Magiera
7526a074d9
sealing: collateral buffer / falback config
2021-07-13 17:04:58 +02:00
Łukasz Magiera
83f2368507
Add CollateralFromMinerBalance config
2021-07-12 14:11:58 +02:00
Anton Evangelatov
ff2772a58c
resolved conflicts
2021-07-06 16:00:41 +02:00
Anton Evangelatov
4be0a7a215
resolve conflicts with master
2021-07-05 13:13:32 +02:00
Peter Rabbitson
a544fa0f97
Fix more CID doucle-encoding as hex
2021-07-05 12:51:15 +02:00
Łukasz Magiera
35c0e3cf8c
Merge pull request #6658 from sunday527/master
...
Fix: precommit_batch method used the wrong cfg.CommitBatchWait
2021-07-02 15:18:48 +02:00
Łukasz Magiera
60ecbdee8f
Merge pull request #6635 from llifezou/fix_checkTicketExpired
...
fix ticket expiration check
2021-07-02 15:18:04 +02:00
zhoutian527
71e1577a62
Fix: precommit_batch method used the wrong cfg.PreCommitBatchWait
2021-07-02 15:53:21 +08:00
llifezou
df53b97fe4
Update extern/storage-sealing/states_sealing.go
...
fix sector precommitted but expired judgment
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-07-02 11:44:46 +08:00
llifezou
fe3ad4fd60
Update extern/storage-sealing/states_sealing.go
...
fix log
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
2021-07-02 11:38:04 +08:00
Łukasz Magiera
7c2c8b2a95
commit batch: Regression test nil FailedSectors map
2021-07-01 13:51:11 +02:00
Łukasz Magiera
c094aa82ec
commit batch: AggregateAboveBaseFee config
2021-07-01 13:33:54 +02:00
Łukasz Magiera
88bb9f422e
commit batch: Initialize the FailedSectors map
2021-07-01 12:15:58 +02:00
llifezou
73e58f7af1
fix getTicket: sector precommitted but expired case
2021-07-01 10:53:42 +08:00
johnli-helloworld
229d5e5c80
handleSubmitCommitAggregate() exception handling
2021-06-30 15:29:21 +02:00
Łukasz Magiera
2dad06d59f
Merge pull request #6634 from GFZRZK/GFZRZK/correct_check_commit
...
remove precommit check in handleCommitFailed
2021-06-30 15:23:33 +02:00
Łukasz Magiera
f45340461d
gofmt
2021-06-30 15:19:29 +02:00
Anton Evangelatov
4f0a96c9c7
resolved conflicts
2021-06-30 13:16:52 +02:00
johnli-helloworld
b0b070ffe7
to optimize the batchwait
2021-06-30 16:56:40 +08:00
llifezou
3dd3476bfd
fix ticket expiration check, otherwise it may cause a large number of loops to retry GetTicket when retrying PreCommit1
2021-06-30 16:32:44 +08:00
wangchao
a4342f3997
remove precommit check in handleCommitFailed
2021-06-30 11:09:07 +08:00
Łukasz Magiera
7b1b082381
Merge branch 'master' into fix/prove-commit-aggregate
2021-06-29 11:48:52 +02:00
Jerry
73704c92ba
ensure agg fee is adequate
2021-06-29 14:36:26 +08:00
Jerry
5c3d67a811
fix: miner balance is not enough, so that ProveCommitAggregate msg exec failed
2021-06-29 14:06:41 +08:00
Łukasz Magiera
84dbcb63d2
Merge pull request #6566 from GFZRZK/GFZRZK/correct_sector_expire_time
...
scale up sector expiration to avoid sector expire in batch-pre-commit waitting
2021-06-24 10:59:49 +02:00
Łukasz Magiera
c3480dc0e8
Merge remote-tracking branch 'origin/releases' into chore/merge-release
2021-06-24 09:39:11 +02:00
Aayush Rajasekaran
616e5688fc
Remove MinPreCommitBatch
2021-06-23 12:30:32 -04:00
wangchao
03261771e2
scale up sector expiration to avoid sector expire in batch-pre-commit waitting
2021-06-23 14:10:28 +08:00
Aayush Rajasekaran
93f7cbe587
Add a helpful comment
2021-06-22 19:31:36 -04:00
Aayush Rajasekaran
6ca2a14819
Always flush when timer goes off
2021-06-22 19:16:36 -04:00
Anton Evangelatov
6720463799
resolve merge conflicts
2021-06-22 11:28:23 +02:00
Łukasz Magiera
b37a66c7c9
Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10
2021-06-21 17:03:03 +02:00
yaohcn
7c6736b3ef
fix commit finalize failed
2021-06-18 12:09:02 +08:00
Łukasz Magiera
0a5a7cf45d
storagefsm: Fix too-long log handling
2021-06-17 20:52:46 -04:00
Łukasz Magiera
b0128bd99e
storagefsm: Fix race spawning more than one new sector at once
2021-06-17 20:52:42 -04:00
Łukasz Magiera
097baeb9b0
Make batch deal input test less flaky
2021-06-17 20:52:09 -04:00
Łukasz Magiera
5c8498b603
storagefsm: Fix batch deal packing behavior
2021-06-17 20:50:14 -04:00
Anton Evangelatov
d45bb14015
Merge branch 'master' into nonsense/split-market-miner-processes
2021-06-16 18:49:15 +02:00
Łukasz Magiera
1f914053b4
sealing: Wire up context to batchers
2021-06-16 11:43:06 +02:00
Łukasz Magiera
81b412399e
sealing: Fix restartSectors race
2021-06-15 17:55:58 -04:00
Łukasz Magiera
74db586fdf
sealing: Fix restartSectors race
2021-06-15 17:43:32 -04:00
Aayush Rajasekaran
520a0091f5
Merge branch 'releases' into release/v1.10.0
2021-06-15 17:40:36 -04:00
Łukasz Magiera
f9596dd730
Merge pull request #6432 from filecoin-project/tests/batching-unit
...
Unit tests for sector batchers
2021-06-14 21:05:34 +02:00
wangchao
4da30931c9
failed sectors should be added into res correctly
2021-06-14 11:33:38 -04:00
Łukasz Magiera
0514f38dc4
Merge remote-tracking branch 'origin/master' into chore/merge-1.10
2021-06-14 13:27:53 +02:00
Anton Evangelatov
31e6fb154e
resolve conflicts
2021-06-14 11:27:54 +02:00
wangchao
f392c1295c
failed sectors should be added into res correctly
2021-06-12 23:22:32 +08:00
Łukasz Magiera
733240a2bb
sealing: Test early finalization fsm planners
2021-06-11 11:52:00 +02:00
Łukasz Magiera
05d9b5ce0f
sealing: Add missing planner for CommitFinalizeFailed
2021-06-11 11:44:37 +02:00
Łukasz Magiera
ec06f086ef
sealing: Early finalization option
2021-06-11 11:41:28 +02:00
Łukasz Magiera
06195bc8e1
Unit tests for sector batchers
2021-06-10 13:46:36 +02:00
Łukasz Magiera
f0a2e97cb5
fee config for sector batching
2021-06-09 14:23:06 -04:00
Aayush Rajasekaran
dede1b4758
Merge branch 'release/v1.10.0' into asr/merge-release
2021-06-09 14:05:41 -04:00
Aayush Rajasekaran
b0c9dd49f0
Fund miners with the aggregate fee when ProveCommitting
2021-06-09 13:08:54 -04:00
Aayush Rajasekaran
dbb4e9fcc5
Drop soms logs
2021-06-09 12:26:20 -04:00
Łukasz Magiera
92bb874327
Use correct batch fee config in commit batcher
...
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
2021-06-09 12:17:18 +02:00
Aayush Rajasekaran
ba27d45214
Split the getSectorCutoff methods between precommit and commit batchers
2021-06-08 16:46:35 -04:00
Łukasz Magiera
78171055e7
fee config for sector batching
2021-06-08 15:45:32 +02:00
Aayush Rajasekaran
b13169f071
Rename deadlines to cutoffs in the batchers
2021-06-07 20:20:06 -04:00
Anton Evangelatov
8bd09e39ca
resolve merge conflicts
2021-06-04 16:17:00 +02:00
Łukasz Magiera
dc642d0b7b
Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11
2021-06-01 21:06:58 +02:00
Łukasz Magiera
8b84e499c5
storagefsm: Handle preCommitParams errors better
2021-06-01 20:36:20 +02:00
Łukasz Magiera
482e1110c2
precommit batcher: Improve error propagation
2021-06-01 14:35:30 +02:00
Łukasz Magiera
cb4eb487f4
commit batcher: Fix min aggregate size check
2021-06-01 12:27:22 +02:00
Łukasz Magiera
084b0e7f60
Handle collateral when submitting aggregated commits
2021-06-01 12:09:15 +02:00
Łukasz Magiera
9fcb564bef
Make commit batcher more robust
2021-06-01 11:56:19 +02:00
Anton Evangelatov
0a79f49dc5
fix tests
2021-06-01 11:45:34 +02:00
Łukasz Magiera
ffa47659a1
Merge remote-tracking branch 'origin/feat/nv13' into feat/nv13-1.11
2021-05-31 21:38:34 +02:00