Łukasz Magiera
aac1f5abef
sealing: Fix build
2022-03-16 20:09:18 +01:00
Łukasz Magiera
724d80cf02
sealing: Address snap queue review
2022-03-16 20:04:48 +01:00
Łukasz Magiera
e7ee5b5932
sealing: Run more checks when considering Available sectors
2022-03-16 19:57:37 +01:00
Łukasz Magiera
4d4739497d
sealing: Use bitfields in sectorActive
2022-03-16 19:53:00 +01:00
Łukasz Magiera
4ded059a1c
sealing: Add MakeCCSectorsAvailable config
2022-03-16 19:29:47 +01:00
Łukasz Magiera
99c6ef564e
worker: RemoveCopies expects one type at a time
2022-03-16 19:07:42 +01:00
Łukasz Magiera
39e3ddb0ed
storagemgr: MoveStorage in separate steps
2022-03-16 19:07:42 +01:00
Łukasz Magiera
defdea7b49
worker: Call RemoveCopies in MoveStorage
2022-03-16 19:07:42 +01:00
Łukasz Magiera
d249eb3f93
stores: Support RemoveCopies on remote store
2022-03-16 19:07:42 +01:00
Łukasz Magiera
668c33fdcd
stores: http: Support multiple storage IDs in ?keep
2022-03-16 19:07:42 +01:00
Łukasz Magiera
628102ab6c
sealing: Support abort uprade for Available sectors
2022-03-16 18:53:54 +01:00
Łukasz Magiera
f2c0ba1df5
Merge pull request #8329 from filecoin-project/fix/snap-worker-cleanup
...
fix: sealing: Remove sector copies from workers after snapdeals
2022-03-16 18:48:21 +01:00
Łukasz Magiera
193c84811a
feat: sealing: Linter did have a good point
2022-03-16 17:45:22 +01:00
Łukasz Magiera
aef2ec513f
feat: sealing: Sector upgrade queue
2022-03-16 17:33:05 +01:00
Aayush
16128a0212
Merge branch 'master' into feat/fvm
2022-03-16 11:57:46 -04:00
Łukasz Magiera
9f6f94bd02
Merge pull request #8215 from filecoin-project/feat/remove-mark-for-upgrade
...
chore:sealing:remove endpoint from cli
2022-03-16 15:33:21 +01:00
Łukasz Magiera
c4259cb594
worker: RemoveCopies expects one type at a time
2022-03-16 12:28:56 +01:00
Łukasz Magiera
5ed5744cb6
storagemgr: MoveStorage in separate steps
2022-03-16 12:28:56 +01:00
Łukasz Magiera
a88edeb79d
worker: Call RemoveCopies in MoveStorage
2022-03-16 12:28:56 +01:00
Łukasz Magiera
4d65e0101d
stores: Support RemoveCopies on remote store
2022-03-16 12:28:56 +01:00
Łukasz Magiera
45b07674e5
stores: http: Support multiple storage IDs in ?keep
2022-03-16 12:28:56 +01:00
Aayush
6c51adc334
Update FFI: fix cargo.lock
2022-03-15 11:07:25 -04:00
Eng Zer Jun
c229842f57
test: use T.TempDir to create temporary test directory
...
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-15 00:21:31 +08:00
Aayush
67889b473c
Merge branch 'master' into feat/fvm
2022-03-12 11:12:23 -05:00
Aayush
d2054e8653
FVM: support nv15
2022-03-11 17:03:34 -05:00
Łukasz Magiera
ce99aad438
Fix provingCheckProvableCmd
2022-03-11 18:13:51 +01:00
Łukasz Magiera
135aef78d7
Merge remote-tracking branch 'origin/master' into feat/post-worker
2022-03-11 17:04:58 +01:00
Łukasz Magiera
ad66ad4f7d
fix: sealing fsm: Handle inputLk correctly
2022-03-11 13:34:14 +01:00
Łukasz Magiera
91ccb7fbda
fix: piece provider: Don't log CIDs as binary
2022-03-10 14:28:18 +01:00
Łukasz Magiera
046a9f8af0
Merge remote-tracking branch 'origin/master' into feat/post-worker
2022-03-09 16:27:03 +01:00
zenground0
b81d189ec1
Log instead of error normal shutdown of state machine
2022-03-02 13:03:03 -07:00
zenground0
874124c4e7
Handle finalize failures in fsm
2022-03-02 09:33:33 -07:00
Aayush
0a67b6e222
Merge branch 'master' into feat/fvm
2022-03-02 11:24:48 -05:00
Łukasz Magiera
6d464c478e
Merge pull request #8160 from filecoin-project/fix/addpiece-idempotent
...
feat: Make add piece idempotent
2022-03-02 15:10:16 +00:00
Łukasz Magiera
a6892f956e
Merge pull request #7844 from llifezou/add_workerName_in_sealing_err
...
feat: #6147 : Include worker name in sealing errors
2022-03-02 13:13:34 +00:00
Jiaying Wang
e0922a7dd2
pay for the collateral difference needed if the miner available balance is insufficient
2022-03-01 21:56:12 -05:00
zenground0
59ef2fe1b8
Remove unused fields from sealing
2022-03-01 11:46:36 -07:00
zenground0
a463c90057
remove endpoint from cli
2022-03-01 11:27:03 -07:00
zenground0
f138ae8882
Less verbose sector manager logging
2022-03-01 10:52:09 -07:00
Aayush
dd91857bf6
Merge branch 'master' into feat/fvm
2022-02-28 22:17:20 -05:00
Jerry
8f95cd4149
release worker tracker lock when call cb func
2022-02-28 16:55:04 +08:00
jennijuju
f21ddf0c50
Merge branch 'releases' into jen/releasesbacktomaster
2022-02-25 20:54:11 -05:00
llifezou
dac5518005
Update extern/sector-storage/worker_local.go
...
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2022-02-25 11:12:18 +08:00
Jiaying Wang
ec6b786fbf
Merge pull request #8182 from filecoin-project/jen/backports
...
chore: backports: snappy related improvements/bug fixes
2022-02-24 18:07:41 -05:00
zenground0
e5de237990
Fix fault tracker to handle snap deals
2022-02-24 17:53:21 -05:00
Jiaying Wang
f9641d70af
fix: sealing: missing file type
2022-02-24 16:27:24 -05:00
zenground0
abe04c33c1
Fix fault tracker to handle snap deals
2022-02-23 09:56:47 -07:00
Aarsh Shah
f0f2b3e1f4
changes as per review
2022-02-22 14:22:51 +04:00
Aarsh Shah
8e62fec1d3
address review
2022-02-22 13:48:39 +04:00
Aarsh Shah
56926b349a
changes as per review
2022-02-21 18:27:51 +04:00
Aarsh Shah
c2522e8ca9
idempotent add piece
2022-02-21 13:51:43 +04:00
Aarsh Shah
1245871004
guard with ctx
2022-02-21 11:56:49 +04:00
Aarsh Shah
26707a5e03
make add piece idempotent
2022-02-21 11:51:25 +04:00
Aayush
d35216434f
Merge branch 'releases' into asr/releases-master
2022-02-18 13:45:08 -05:00
Aayush
6e1d5c5733
Merge branch 'master' into feat/fvm
2022-02-18 13:33:06 -05:00
Aayush
393479ea69
Fvm: impl VerifyConsensusFault
2022-02-18 13:21:01 -05:00
Aayush
b708fbcd20
sealer: fix error message
2022-02-17 14:31:59 -05:00
Steven Allen
d835cade15
chore: update FFI
...
Updates the FVM
2022-02-16 17:47:25 -08:00
Aayush
36aa243c56
sealer: fix error message
2022-02-16 19:26:00 -05:00
Aayush
56df886b55
fix: sealer: allow enable/disabling ReplicaUpdate tasks
2022-02-14 15:24:13 -05:00
Aayush
eebe784191
fix: sealer: allow enable/disabling ReplicaUpdate tasks
2022-02-14 15:08:28 -05:00
Aayush
ee6989970f
Merge branch 'master' into feat/fvm
2022-02-13 14:25:39 -05:00
zenground0
407c6cf120
Stop recovery attempts after fault
2022-02-10 17:33:57 -05:00
Łukasz Magiera
9e76a788e2
sealing: Use ChainAt in handleUpdateActivating
2022-02-10 17:32:31 -05:00
Łukasz Magiera
02c0fb7b65
ffiwrapper: Use ClearCache for update cache cleanup
2022-02-10 17:32:13 -05:00
Łukasz Magiera
a5bddbcdc6
proofs master
2022-02-10 17:32:04 -05:00
Łukasz Magiera
9f54c4d002
Fix CheckProvable with updated sectors
2022-02-10 17:30:23 -05:00
Łukasz Magiera
4289ce10e4
fsm: Call ReleaseSectorKey after WinningPoSt lookback period
2022-02-10 17:30:18 -05:00
Łukasz Magiera
ecc1f4b8f1
Make FinalizeReplicaUpdate actually do cleanup
2022-02-10 17:29:52 -05:00
Łukasz Magiera
ea32b415f0
Fix missing FinalizeReplicaUpdate in tests
2022-02-10 17:29:47 -05:00
Łukasz Magiera
53e1f75c2c
Add FinalizeReplicaUpdate into some more places
2022-02-10 17:27:49 -05:00
Łukasz Magiera
9a117fa39a
try ClearCache for update cache
2022-02-10 17:27:24 -05:00
Łukasz Magiera
212f5ddb4f
wip FinalizeReplicaUpdate
2022-02-10 17:24:26 -05:00
Steven Allen
7ef1513f9b
ffi: update fvm
2022-02-10 07:56:27 -08:00
Jennifer Wang
1d67b7ffb6
undo ffi
2022-02-09 19:31:50 -05:00
Jennifer Wang
de1f4792cd
Merge remote-tracking branch 'origin/master' into jen/v15
2022-02-09 19:27:34 -05:00
Steven Allen
0d6eb7fefc
ffi: update for fixed FVM lifetime management
2022-02-09 14:13:58 -08:00
Łukasz Magiera
e7038d286d
Merge pull request #8018 from filecoin-project/feat/fru
...
FinalizeReplicaUpdate
2022-02-09 17:40:47 +00:00
Łukasz Magiera
84d70caa87
sealing: Use ChainAt in handleUpdateActivating
2022-02-09 17:50:56 +01:00
Łukasz Magiera
cce77196cd
proofs v11.0.2
2022-02-09 16:22:35 +01:00
Łukasz Magiera
f33cf74d0f
ffiwrapper: Use ClearCache for update cache cleanup
2022-02-09 14:54:36 +01:00
Łukasz Magiera
6cb411e5bb
proofs master
2022-02-09 14:46:29 +01:00
Łukasz Magiera
b1a74d71f1
Fix CheckProvable with updated sectors
2022-02-09 14:41:36 +01:00
Łukasz Magiera
947000c94f
fsm: Call ReleaseSectorKey after WinningPoSt lookback period
2022-02-09 12:41:39 +01:00
Jennifer Wang
6ed97f0d04
Merge branch 'release/v1.14.0' into jen/v15
2022-02-09 03:17:38 -05:00
Steven Allen
743862857e
ffi: update FFI for fvm changes
2022-02-08 20:57:20 -08:00
zenground0
1e524b7210
Review Response
2022-02-08 12:39:35 -05:00
zenground0
a19c0a8255
Remove extra file
2022-02-08 12:39:35 -05:00
zenground0
4d8b912acc
New config for turning off auto deal creation
2022-02-08 12:39:35 -05:00
zenground0
1af8987c79
StartEpochSealingBuffer triggers packing on timer
2022-02-08 12:39:35 -05:00
zenground0
6fdf76b5d2
Stop recovery attempts after fault
...
Check sector is active before PRU
Fix log
nit: edit a comment
2022-02-08 12:25:24 -05:00
Aayush Rajasekaran
899ae8aa8e
Merge pull request #8014 from filecoin-project/fix/8011
...
fix: sealing: Stop recovery attempts after fault
2022-02-08 12:06:55 -05:00
Aayush Rajasekaran
b6c141c059
nit: edit a comment
2022-02-08 12:06:33 -05:00
Łukasz Magiera
6123339baf
Make FinalizeReplicaUpdate actually do cleanup
2022-02-08 17:45:26 +01:00
Łukasz Magiera
c8e0341248
Fix missing FinalizeReplicaUpdate in tests
2022-02-08 17:22:41 +01:00
Łukasz Magiera
09cfad9d71
Add FinalizeReplicaUpdate into some more places
2022-02-08 17:22:41 +01:00
Łukasz Magiera
e271bae5ec
try ClearCache for update cache
2022-02-08 17:22:41 +01:00
Łukasz Magiera
142ba6660a
wip FinalizeReplicaUpdate
2022-02-08 17:22:41 +01:00
Aayush Rajasekaran
f476aa937e
update to latest FFI
2022-02-08 10:45:58 -05:00
Steven Allen
279cdd0760
fvm: fix implicit messages and message inclusion gas charging
2022-02-07 10:33:57 -08:00
zenground0
1ab2744c84
Fix log
2022-02-07 09:15:23 -05:00
zenground0
47ffceef0d
Check sector is active before PRU
2022-02-07 07:41:48 -05:00
Steven Allen
04092f3e6b
update ffi
2022-02-04 12:46:15 -08:00
Steven Allen
fc74a6c8ca
update fvm/ffi bindings
2022-02-04 12:46:15 -08:00
Aayush Rajasekaran
2a669b95fb
:Hook up the FVM
2022-02-04 12:46:15 -08:00
Jennifer Wang
33b53c4a0d
update to ffi v11.0.1
2022-02-03 11:15:47 -05:00
Łukasz Magiera
b74e085272
don't sort
2022-02-02 19:08:21 +00:00
zenground0
13ccb8cbfe
Stop recovery attempts after fault
2022-02-01 11:39:42 +05:30
Łukasz Magiera
efdb854a7c
fix some races
2022-01-31 20:53:25 +00:00
Łukasz Magiera
98f4d37f1a
Merge remote-tracking branch 'origin/master' into feat/post-worker
2022-01-31 19:50:52 +00:00
Aayush
37a345b39d
Update FFI
2022-01-27 15:30:01 -05:00
Darko Brdareski
e51ce5c508
Merge remote-tracking branch 'upstream/master' into bloxico/system-test-matrix
2022-01-27 10:57:56 +01:00
Aayush
817c155287
chore: deps: update to latest proofs
2022-01-25 13:23:00 -05:00
Aayush Rajasekaran
a6460be37b
remove a log
2022-01-25 13:02:45 -05:00
Aayush
e7123d1a8e
fix: sealer: correctly pipe through errors for SectorAbortUpgrade
2022-01-25 13:02:00 -05:00
Aayush
50aba9a8e6
fix: sealer: don't replica update sectors unless they have deals in them
2022-01-25 13:01:56 -05:00
Aayush
1b18236f91
feat: sealer: allow users to abort in-flight snap upgrades
2022-01-25 13:01:51 -05:00
Aayush
e17ae2eaf4
fix: sealer: manager should lock Unsealed for ReplicaUpdate
2022-01-25 13:01:37 -05:00
Aayush
6d567b36e3
Fix: sealer: ReplicaUpdate should fetch the correct files
2022-01-25 13:01:27 -05:00
Aayush Rajasekaran
7b7ab016db
create replica update paths in acquireSectors
2022-01-25 13:01:18 -05:00
Aayush Rajasekaran
0c9c94bad1
fix: checkReplica incorrectly returns ErrBadPR
2022-01-25 13:01:12 -05:00
Aayush Rajasekaran
a3c5fadcc0
feat: sealing: Add ReplicaUpdate work to Resource table
2022-01-25 13:01:05 -05:00
Aayush Rajasekaran
2d0929e305
remove a log
2022-01-25 12:55:56 -05:00
Aayush Rajasekaran
92e6f29cc8
chore: sealer: quieten a log
2022-01-24 18:28:52 -05:00
Aayush
5cafdc2f29
fix: sealer: manager should lock Unsealed for ReplicaUpdate
2022-01-21 11:12:12 -05:00
Aayush Rajasekaran
ff845aa793
Merge pull request #7977 from filecoin-project/chore/sealtasks-comment
...
chore: remove inaccurate comment in sealtasks
2022-01-21 10:53:04 -05:00
Łukasz Magiera
f148397e1b
post workers: Fix race in setting vproofs
2022-01-21 12:31:24 +01:00
Łukasz Magiera
82c9e72aab
post workers: Fix skipped handling
2022-01-21 10:39:14 +01:00
Łukasz Magiera
4682e72cf4
Address review
2022-01-21 10:11:04 +01:00
Łukasz Magiera
1e5e69ec9d
Typos
...
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
2022-01-21 10:02:00 +01:00
Aayush
752f4a3d67
Fix: sealer: ReplicaUpdate should fetch the correct files
2022-01-20 15:06:53 -05:00
Łukasz Magiera
33dfcf2663
stores: Fix single post proof reading with updated sectors
2022-01-20 15:27:42 +01:00
Łukasz Magiera
e476cf7968
Merge remote-tracking branch 'origin/master' into feat/post-worker
2022-01-20 13:15:48 +01:00
Aayush Rajasekaran
3ff23ecbfa
:fix: checkReplica incorrectly returns ErrBadPR
2022-01-19 12:00:27 -05:00
Aayush Rajasekaran
ab8bf393c2
create replica update paths in acquireSectors
2022-01-19 11:41:38 -05:00
Aayush Rajasekaran
d0390181ec
feat: sealing: Add ReplicaUpdate work to Resource table
2022-01-19 11:41:32 -05:00
Łukasz Magiera
c41ccb6c37
chore: remove inaccurate comment in sealtasks
2022-01-19 10:46:37 +01:00
Łukasz Magiera
fa09b9afb1
post workers: Share resource accounting code
2022-01-18 16:02:37 +01:00
Łukasz Magiera
8ac20305df
Fix double unlock in WorkerStats
2022-01-18 12:40:04 +01:00
Łukasz Magiera
c79f2b9d38
post workers: Fix lint
2022-01-18 12:11:59 +01:00
Łukasz Magiera
b38141601c
Untangle ffi from api
2022-01-18 11:57:04 +01:00
Łukasz Magiera
03fa955e9e
Cleanup imports
2022-01-18 11:37:24 +01:00
Łukasz Magiera
f2496d87c9
post workers: Snapdeals updates
2022-01-18 11:25:04 +01:00
Łukasz Magiera
0c062f9c08
Merge remote-tracking branch 'origin/master' into feat/post-worker
2022-01-18 10:49:42 +01:00
Aayush Rajasekaran
aad8aa0893
Appease the linter
2022-01-14 17:15:44 -05:00
Jennifer Wang
30013c1f06
fix lint
2022-01-14 17:15:44 -05:00
Jennifer Wang
6901e998e6
Check piece before PRU2 instead of PRU1 as PRU2 is the heavy computation part
2022-01-14 17:15:44 -05:00
Jennifer Wang
8939d5982f
just use checkPiece
2022-01-14 17:15:44 -05:00
Jennifer Wang
a20916f9af
Add more deal expiration handling for snap deals
2022-01-14 17:15:44 -05:00
Jennifer Wang
ac3bea489b
Integrate proof v11.0.0
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
ca57546ef5
Remove unnecessary params from VerifyWinningPost
2022-01-14 17:14:32 -05:00
zenground0
5b0a0baa9a
Fix hande deal recover return value bug
2022-01-14 17:14:32 -05:00
zenground0
a9a523d8c0
Fix TooManyMarkedForUpgrade
2022-01-14 17:14:32 -05:00
zenground0
d6aa17e21f
Snap Deals Integration
...
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
3a8ac6dffa
Update FFI
2022-01-14 17:14:32 -05:00
zenground0
d1480c36c0
RemoveData and Decode
...
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
2022-01-14 17:14:32 -05:00
zenground0
4936b4ea44
Review Response
2022-01-14 17:14:32 -05:00
zenground0
c4069824f7
WIP
2022-01-14 17:14:32 -05:00
zenground0
93656e65f8
WIP sector storage and integration test
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
073b7b4ff5
Update FFI
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
393d8541e2
Update deps
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
1ef780d96f
Plug in the FFI call
2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
a8cb027c08
Integrate v7 actors
2022-01-14 17:14:32 -05:00
Łukasz Magiera
4a874eff70
post workers: Cleanup, tests
2022-01-14 14:17:52 +01:00
Aayush Rajasekaran
bda4e5be95
Appease the linter
2022-01-12 18:10:07 -05:00
Jennifer Wang
fd50cd128a
fix lint
2022-01-11 18:34:26 -05:00
Jennifer Wang
e99b98873c
Check piece before PRU2 instead of PRU1 as PRU2 is the heavy computation part
2022-01-11 18:34:20 -05:00
Jennifer Wang
6b953a03d0
just use checkPiece
2022-01-11 18:34:15 -05:00
Jennifer Wang
37a3e610b7
Add more deal expiration handling for snap deals
2022-01-11 18:34:10 -05:00
Jiaying Wang
0130b28879
Merge pull request #7923 from filecoin-project/jen/proofv11
...
chore: deps: Integrate proof v11.0.0
2022-01-11 17:49:43 -05:00
Aayush Rajasekaran
d645c5fbab
Remove unnecessary params from VerifyWinningPost
2022-01-11 12:06:39 -05:00
Jennifer Wang
5b7da270c9
Integrate proof v11.0.0
2022-01-10 23:45:04 -05:00
Łukasz Magiera
e216aefd23
fix make gen
2022-01-10 18:24:00 +01:00
zenground0
d16c5d0e93
Fix hande deal recover return value bug
2022-01-10 15:47:20 +05:30
zenground0
c309686679
Fix TooManyMarkedForUpgrade
2022-01-10 15:39:38 +05:30
zenground0
33f2d24f54
Snap Deals Integration
...
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-10 15:39:38 +05:30
mz-sirius
793b5c7cc3
fix ci err
2022-01-05 21:41:21 +08:00
mz-sirius
6867f017ec
update ffi
2022-01-05 01:50:53 +08:00
mz-sirius
3fd55fa56b
decoupling winningpost and windowpost from lotus-miner
2022-01-05 01:50:49 +08:00
llifezou
4b685c5e26
Include worker name in sealing errors
2021-12-23 17:44:43 +08:00
Darko Brdareski
dda1a42a2a
Merge branch 'bloxico/system-test-matrix' of https://github.com/filecoin-project/lotus into merge_lotus
2021-12-20 15:48:16 +01:00
shotcollin
d10d0a20b1
fix typo in log warning
...
very minor but this warning comes up a lot so it'd be nicer if it wasn't a grammatical error too
2021-12-19 17:07:11 -07:00
Darko Brdareski
2f1f35cc71
Annotate storage miner features
2021-12-15 15:30:42 +01:00
Aayush Rajasekaran
3e288f1066
Update FFI
2021-12-13 15:47:17 -05:00
Aayush Rajasekaran
80d5e52923
Merge branch 'master' into next
2021-12-13 13:24:28 -05:00
Łukasz Magiera
ba3c96f8c6
stores: Reduce log spam during retrievals
2021-12-10 16:47:32 -05:00
Łukasz Magiera
dafdb7689c
Fix mock ReadPiece
2021-12-10 16:47:12 -05:00
Łukasz Magiera
46ba2b6b4f
fr32: Reduce MTTresh from 32M to 512k per core
...
This results in 64x less bytes allocated when spawning new readers
for larger pieces.
Results in about 30% speedup in 1G unpad benchmark on AMD TR 2950x
2021-12-10 16:47:07 -05:00
Łukasz Magiera
b4c1e340ea
piecereader: Avoid allocating 1024MB slices per read
2021-12-10 16:47:01 -05:00
Łukasz Magiera
a438e6fa73
piecereader: Avoid redundant roundtrips when seeking
2021-12-10 16:46:57 -05:00
Łukasz Magiera
b21d3ded2f
piecereader: Move closer to storage
2021-12-10 16:46:52 -05:00
Darko Brdareski
0169d0dafd
Annotate state feature tests
2021-12-10 16:08:25 +01:00
Łukasz Magiera
e8ef39e734
stores: Reduce log spam during retrievals
2021-12-10 11:28:04 +01:00
Łukasz Magiera
c31f4de7d5
Fix mock ReadPiece
2021-12-09 16:26:59 +01:00
Łukasz Magiera
6fd1609410
fr32: Reduce MTTresh from 32M to 512k per core
...
This results in 64x less bytes allocated when spawning new readers
for larger pieces.
Results in about 30% speedup in 1G unpad benchmark on AMD TR 2950x
2021-12-09 16:14:47 +01:00
Łukasz Magiera
9c75a3aaa8
piecereader: Avoid allocating 1024MB slices per read
2021-12-09 15:49:43 +01:00
Łukasz Magiera
a3d8494a04
piecereader: Avoid redundant roundtrips when seeking
2021-12-09 14:52:33 +01:00
Łukasz Magiera
13b260e7f7
piecereader: Move closer to storage
2021-12-08 23:20:20 +01:00
zenground0
a5be80828a
RemoveData and Decode
...
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
2021-12-03 15:21:06 -05:00
Łukasz Magiera
727765b248
Command to list active sector locks
2021-12-03 12:33:23 +01:00
Łukasz Magiera
71329f6c41
Address Scheduler enhancements ( #7703 ) review
2021-11-30 20:50:40 +01:00
zenground0
40d16a8f88
Review Response
2021-11-30 13:53:37 -05:00
zenground0
f88fcdbcfc
WIP
2021-11-30 12:40:14 -05:00
Łukasz Magiera
001ecbb561
fix lint
2021-11-30 02:06:58 +01:00
Łukasz Magiera
a597b072b8
fix sched tests
2021-11-30 02:06:58 +01:00
Łukasz Magiera
f25efecb74
worker: Test resource table overrides
2021-11-30 02:06:58 +01:00
Łukasz Magiera
6d52d8552b
Fix docsgen
2021-11-30 02:06:58 +01:00
Łukasz Magiera
c9a2ff4007
cleanup worker resource overrides
2021-11-30 02:06:58 +01:00
Łukasz Magiera
b961e1aab5
sched resources: Separate Parallelism defaults depending on GPU presence
2021-11-30 02:06:58 +01:00
Łukasz Magiera
36868a8749
sched: C2 is not all-core load
2021-11-30 02:06:58 +01:00
Clint Armstrong
4ef8543128
Permit workers to override resource table
...
In an environment with heterogenious worker nodes, a universal resource
table for all workers does not allow effective scheduling of tasks. Some
workers may have different proof cache settings, changing the required
memory for different tasks. Some workers may have a different count of
CPUs per core-complex, changing the max parallelism of PC1.
This change allows workers to customize these parameters with
environment variables. A worker could set the environment variable
PC1_MIN_MEMORY for example to customize the minimum memory requirement
for PC1 tasks. If no environment variables are specified, the resource
table on the miner is used, except for PC1 parallelism.
If PC1_MAX_PARALLELISM is not specified, and
FIL_PROOFS_USE_MULTICORE_SDR is set, PC1_MAX_PARALLELSIM will
automatically be set to FIL_PROOFS_MULTICORE_SDR_PRODUCERS + 1.
2021-11-30 02:06:58 +01:00
Clint Armstrong
93e4656a27
Use a float to represent GPU utilization
...
Before this change workers can only be allocated one GPU task,
regardless of how much of the GPU resources that task uses, or how many
GPUs are in the system.
This makes GPUUtilization a float which can represent that a task needs
a portion, or multiple GPUs. GPUs are accounted for like RAM and CPUs so
that workers with more GPUs can be allocated more tasks.
A known issue is that PC2 cannot use multiple GPUs. And even if the
worker has multiple GPUs and is allocated multiple PC2 tasks, those
tasks will only run on the first GPU.
This could result in unexpected behavior when a worker with multiple
GPUs is assigned multiple PC2 tasks. But this should not suprise any
existing users who upgrade, as any existing users who run workers with
multiple GPUs should already know this and be running a worker per GPU
for PC2. But now those users have the freedom to customize the GPU
utilization of PC2 to be less than one and effectively run multiple PC2
processes in a single worker.
C2 is capable of utilizing multiple GPUs, and now workers can be
customized for C2 accordingly.
2021-11-30 02:06:58 +01:00
Clint Armstrong
c4f46171ae
Report memory used and swap used in worker res
...
Attempting to report "memory used by other processes" in the MemReserved
field fails to take into account the fact that the system's memory used
includes memory used by ongoing tasks.
To properly account for this, worker should report the memory and swap
used, then the scheduler that is aware of the memory requirements for a
task can determine if there is sufficient memory available for a task.
2021-11-30 02:06:58 +01:00
Clint Armstrong
e2a1ca7caa
Use cgroup limits in worker memory calculations
...
Worker processes may have memory limitations imposed by Systemd. But
/proc/meminfo shows the entire system memory regardless of these limits.
This results in the scheduler believing the worker has the entire system
memory avaliable and the worker being allocated too many tasks.
This change attempts to read cgroup memory limits for the worker
process. It supports cgroups v1 and v2, and compares cgroup limits
against the system memory and returns the most conservative values to
prevent the worker from being allocated too many tasks and potentially
triggering an OOM event.
2021-11-30 02:06:58 +01:00
Łukasz Magiera
d21c44e266
ffiwrapper: Validate PC2 by calling C1 with random seeds
2021-11-30 01:33:05 +01:00
zenground0
7d2b3f05db
WIP sector storage and integration test
2021-11-29 10:24:00 -05:00
Łukasz Magiera
05aa860459
Request correct read size with startOffset in pieceProvider
2021-11-27 00:05:45 +01:00
Łukasz Magiera
743ce5a40f
Add startOffset support to mock SectorMgr.ReadPiece
2021-11-26 18:48:52 +01:00
Łukasz Magiera
f6de16e95a
Fix sector-storage tests
2021-11-26 18:16:53 +01:00
Łukasz Magiera
8d955d5f30
dagstore mount: Add random access support
2021-11-26 17:40:53 +01:00
Łukasz Magiera
8454abcf45
storage: Use 1M buffers for Tar transfers
2021-11-24 20:08:37 +01:00
Łukasz Magiera
2a1505b364
storage: Test StorageFindSector with groups
2021-11-23 16:11:04 +01:00
Łukasz Magiera
8b548ac02f
storage: Check allowlists in StorageFindSector
2021-11-23 16:11:04 +01:00
Łukasz Magiera
5c77c25747
storage: Add Group tags to StorageInfo
2021-11-23 16:11:04 +01:00
Łukasz Magiera
d1a63e4173
remote store: Remove debug printf
2021-11-22 17:50:12 +01:00
Łukasz Magiera
2620eab45e
Merge pull request #7555 from diancun/dc/check-commit
...
checkCommit should return SectorCommitFailed
2021-11-22 12:28:16 +01:00
Aayush Rajasekaran
91fb111462
Update FFI
2021-11-17 17:50:36 -05:00
Aayush Rajasekaran
8665e32221
Update deps
2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
07f1be0ff6
Plug in the FFI call
2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
1449644c34
Integrate v7 actors
2021-11-17 15:55:14 -05:00
Jennifer Wang
1f339f623e
update to proof v10.1.0
2021-10-26 14:36:19 -04:00
yaohcn
3123400759
checkCommit should return SectorCommitFailed
2021-10-22 10:09:44 +08:00
Łukasz Magiera
f7884c4f45
Merge pull request #7527 from filecoin-project/feat/show-prepared-waitres
...
Show prepared tasks in sealing jobs
2021-10-20 15:22:35 +01:00
Łukasz Magiera
726d9b86ea
fix lint
2021-10-19 18:56:30 +02:00
Łukasz Magiera
852ac4c178
Expose per-state sector counts on the prometheus endpoint
2021-10-19 18:53:07 +02:00
Łukasz Magiera
e508055dc1
make gen
2021-10-19 11:13:23 +02:00
Łukasz Magiera
080aa3356a
Fix locks in worker-tracked
2021-10-18 20:19:21 +02:00
Łukasz Magiera
70589e4406
Block work in tracked worker before it is started
2021-10-18 18:44:56 +02:00
Łukasz Magiera
261238e157
Show prepared tasks in sealing jobs
2021-10-18 18:44:56 +02:00
Łukasz Magiera
11d738eee0
Track prepared work
2021-10-18 18:44:56 +02:00
Łukasz Magiera
98ff1c4d20
Merge pull request #7531 from marten-seemann/statfs-build-constraint
...
add missing build constraint to statfs_unix.go
2021-10-18 16:02:48 +01:00
Marten Seemann
03806f7063
add missing build constraint to statfs_unix.go
2021-10-18 16:29:01 +02:00
Łukasz Magiera
63c8b8edd1
Fix used sectors space accounting after AddPieceFailed
2021-10-18 09:55:28 +02:00
Łukasz Magiera
f352c18290
Don't remove sector data when moving data into a shared path
2021-10-11 21:11:38 +02:00
Łukasz Magiera
aed6e8d0be
Merge pull request #7335 from filecoin-project/feat/sched-ready-work
...
sealing: Improve scheduling of ready work
2021-10-07 18:49:02 +01:00
Łukasz Magiera
1993efe201
Merge pull request #7444 from filecoin-project/feat/ap-fail-retry
...
sealing: Recover sectors after failed AddPiece
2021-10-07 03:18:15 +01:00
Łukasz Magiera
cadbd00ac4
sealing: Recover sectors after failed AddPiece
2021-10-04 20:00:07 +02:00
Jennifer Wang
38e01cba83
Merge branch 'release/v1.12.0' into jen/12tomaster
2021-10-04 02:38:00 -04:00
Łukasz Magiera
9af82f2d68
sched: Fix taskDone chan deadlock
2021-10-03 17:09:43 +02:00
Łukasz Magiera
b87142ec8e
wip improve scheduling of ready work
2021-10-03 10:38:08 +02:00
Jennifer Wang
2e1d463933
make lint happy
2021-10-01 20:36:13 -04:00
Jennifer Wang
5f8c44125b
Merge branch 'releases' into jen/12rc1prep
2021-10-01 20:31:29 -04:00
Łukasz Magiera
fc10281d96
Merge pull request #7420 from filecoin-project/deps/proofs-v10.0.0
...
Update proofs to v10.0.0
2021-10-01 22:20:09 +01:00
Aayush Rajasekaran
a335cb3767
Update to latest actors
2021-10-01 17:11:05 -04:00
Aayush Rajasekaran
f004d036dc
Set BatchPreCommitAboveBaseFee correctly
2021-10-01 14:23:41 -04:00
Aayush Rajasekaran
f8a89cafd8
Fix logs
2021-10-01 14:23:41 -04:00
Aayush Rajasekaran
d6dd86b419
Fix lint
2021-10-01 14:23:41 -04:00
Łukasz Magiera
6fd9d5f28b
Precommit batch balancer support/config
2021-10-01 14:23:41 -04:00
Łukasz Magiera
d7fbd8b67d
Update proofs to v10.0.0
2021-10-01 18:38:27 +02:00
Anton Evangelatov
809289f5ef
add Dealmaking.StartEpochSealingBuffer config
2021-10-01 17:44:15 +02:00
Anton Evangelatov
c2fa54ca9c
check for deal start epoch on SectorAddPieceToAny
2021-10-01 17:42:09 +02:00
Łukasz Magiera
a8a9818043
Expose storage states on the metrics endpoint
2021-10-01 14:45:01 +02:00
Aayush Rajasekaran
8756df2447
Fix tests after specs update
2021-09-29 21:27:49 -04:00
Aayush Rajasekaran
84b6734063
Incorporate the new PublishStorageDealsReturn
2021-09-29 19:13:13 -04:00
Jennifer Wang
d7690aa719
Merge branch 'releases' into jen/backport
2021-09-29 03:21:19 -04:00
Adrian Lanzafame
92190c7e0b
Fix formatting
2021-09-23 12:39:39 +10:00
Anton Evangelatov
0c1e29390a
unit test where StateMarketStorageDeal return nil, err
2021-09-21 22:13:51 -04:00
Anton Evangelatov
bf9d0bca40
GetCurrentDealInfo err: handle correctly err case
2021-09-21 22:13:46 -04:00
Jennifer Wang
e996c27619
update to ffi to update-bellperson-proofs-v9-0-2
2021-09-21 17:43:27 -04:00
Aayush Rajasekaran
b9bfcc4b69
Randomness: Move getters from ChainAPI to StateAPI
2021-09-21 14:20:15 -04:00
Łukasz Magiera
2437a4182d
Merge pull request #7322 from filecoin-project/asr/drand-state
...
Randomness: Move getters from ChainAPI to StateAPI
2021-09-21 09:20:12 +01:00
Anton Evangelatov
605953565b
unit test where StateMarketStorageDeal return nil, err
2021-09-17 15:14:53 +02:00
Anton Evangelatov
34e3dc2b37
GetCurrentDealInfo err: handle correctly err case
2021-09-17 14:55:56 +02:00
Łukasz Magiera
438d5ce78e
fix a panic in HandleRecoverDealIDs
2021-09-15 18:13:01 +02:00
Łukasz Magiera
ef03314c6d
storagemgr: Cleanup workerLk around worker resources
2021-09-15 16:35:19 +02:00
Aayush Rajasekaran
1d6db306ec
Randomness: Move getters from ChainAPI to StateAPI
2021-09-15 00:03:13 +02:00
swift-mx
b0b016b58c
fix: check padSector Cid
2021-09-10 18:13:07 +08:00
Jennifer Wang
f751291f1a
update to proof v0.9.2
2021-09-08 01:03:11 -04:00
frank
179458efe9
index out of range
2021-09-06 16:20:23 +08:00
Jennifer Wang
6379d5129f
integrate the proof patch: tag proofs-v9-revert-deps-hotfix
2021-09-02 15:40:15 -04:00
Łukasz Magiera
7ef1b62b41
sealing: Fix sector state accounting with FinalizeEarly
2021-09-02 19:27:10 +02:00
Łukasz Magiera
41db98d49f
sealing: Fix tests
2021-09-01 12:09:44 +02:00
Łukasz Magiera
b280e29409
sealing: Fix retry loop in SubmitCommitAggregate
2021-09-01 11:33:23 +02:00
Łukasz Magiera
3118bd1039
stores: Fix reserved disk usage log spam
2021-08-31 13:36:09 +02:00
Steven Allen
1cf556c3a2
feat: expose ChainGetPath on the gateway
2021-08-30 16:43:21 -07:00
Łukasz Magiera
2293ecd8e8
Reduce lotus-miner startup spam
2021-08-27 19:41:54 +02:00
Łukasz Magiera
b42171dc04
sealing: Check piece CIDs after AddPiece
2021-08-26 13:02:15 -07:00
Łukasz Magiera
62769e3b11
sealing: Fix RecoverDealIDs loop with changed PieceCID
2021-08-20 16:00:40 +02:00
Aayush Rajasekaran
113ea6dc4b
Update to latest FFI
2021-08-17 18:15:12 -04:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ( #6671 )
...
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.
Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.
When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.
Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.
Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.
Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.
Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.
A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.
Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.
NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:
- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport
At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Steven Allen
498644acab
Merge pull request #7011 from filecoin-project/fix/no-t-aux
...
fix: don't check for t_aux when proving
2021-08-11 11:28:29 -07:00
Aayush Rajasekaran
a52352b13a
PreCommitPolicy: Don't try to align expirations on proving period boundaries
2021-08-11 12:44:12 -04:00
ZenGround0
6da26dcefc
Remove network version panic
2021-08-10 13:07:30 -04:00
Steven Allen
18f39be3ba
fix: don't check for t_aux when proving
...
We don't need it.
2021-08-09 11:07:35 -07:00
mx
cfc10e9b52
fix: more logging in maybeStartBatch error
2021-08-06 16:50:37 +08:00
He Weidong
4273fb067c
reduce diff
2021-07-29 15:01:52 +08:00
He Weidong
ef6620ad8e
reasonable max value for initial sector expiration
2021-07-29 14:57:06 +08:00
He Weidong
ee20d30480
reasonable min and max value for initial sector expiration
2021-07-29 14:55:11 +08:00
Steven Allen
dbe4891a39
Merge branch 'master' into fix/check_ticket_msg
2021-07-28 15:25:40 -07:00
Aayush Rajasekaran
21244704fc
Merge branch 'master' into mg/feat/configurable-sector-expirations
2021-07-28 16:45:27 -04:00
