cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
16,038 Commits 24 Branches 342 Tags 314 MiB
89531e33a1
Commit Graph

1209 Commits

Author SHA1 Message Date
Aayush
e17ae2eaf4 fix: sealer: manager should lock Unsealed for ReplicaUpdate 2022-01-25 13:01:37 -05:00
Aayush
6d567b36e3 Fix: sealer: ReplicaUpdate should fetch the correct files 2022-01-25 13:01:27 -05:00
Aayush Rajasekaran
7b7ab016db create replica update paths in acquireSectors 2022-01-25 13:01:18 -05:00
Aayush Rajasekaran
0c9c94bad1 fix: checkReplica incorrectly returns ErrBadPR 2022-01-25 13:01:12 -05:00
Aayush Rajasekaran
a3c5fadcc0 feat: sealing: Add ReplicaUpdate work to Resource table 2022-01-25 13:01:05 -05:00
Aayush Rajasekaran
2d0929e305
remove a log 2022-01-25 12:55:56 -05:00
Aayush Rajasekaran
92e6f29cc8
chore: sealer: quieten a log 2022-01-24 18:28:52 -05:00
Aayush
5cafdc2f29 fix: sealer: manager should lock Unsealed for ReplicaUpdate 2022-01-21 11:12:12 -05:00
Aayush Rajasekaran
ff845aa793
Merge pull request #7977 from filecoin-project/chore/sealtasks-comment 
chore: remove inaccurate comment in sealtasks
 2022-01-21 10:53:04 -05:00
Łukasz Magiera
f148397e1b post workers: Fix race in setting vproofs 2022-01-21 12:31:24 +01:00
Łukasz Magiera
82c9e72aab post workers: Fix skipped handling 2022-01-21 10:39:14 +01:00
Łukasz Magiera
4682e72cf4 Address review 2022-01-21 10:11:04 +01:00
Łukasz Magiera
1e5e69ec9d
Typos 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2022-01-21 10:02:00 +01:00
Aayush
752f4a3d67 Fix: sealer: ReplicaUpdate should fetch the correct files 2022-01-20 15:06:53 -05:00
Łukasz Magiera
33dfcf2663 stores: Fix single post proof reading with updated sectors 2022-01-20 15:27:42 +01:00
Łukasz Magiera
e476cf7968 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-01-20 13:15:48 +01:00
Aayush Rajasekaran
3ff23ecbfa :fix: checkReplica incorrectly returns ErrBadPR 2022-01-19 12:00:27 -05:00
Aayush Rajasekaran
ab8bf393c2 create replica update paths in acquireSectors 2022-01-19 11:41:38 -05:00
Aayush Rajasekaran
d0390181ec feat: sealing: Add ReplicaUpdate work to Resource table 2022-01-19 11:41:32 -05:00
Łukasz Magiera
c41ccb6c37
chore: remove inaccurate comment in sealtasks 2022-01-19 10:46:37 +01:00
Łukasz Magiera
fa09b9afb1 post workers: Share resource accounting code 2022-01-18 16:02:37 +01:00
Łukasz Magiera
8ac20305df Fix double unlock in WorkerStats 2022-01-18 12:40:04 +01:00
Łukasz Magiera
c79f2b9d38 post workers: Fix lint 2022-01-18 12:11:59 +01:00
Łukasz Magiera
b38141601c Untangle ffi from api 2022-01-18 11:57:04 +01:00
Łukasz Magiera
03fa955e9e Cleanup imports 2022-01-18 11:37:24 +01:00
Łukasz Magiera
f2496d87c9 post workers: Snapdeals updates 2022-01-18 11:25:04 +01:00
Łukasz Magiera
0c062f9c08 Merge remote-tracking branch 'origin/master' into feat/post-worker 2022-01-18 10:49:42 +01:00
Aayush Rajasekaran
aad8aa0893 Appease the linter 2022-01-14 17:15:44 -05:00
Jennifer Wang
30013c1f06 fix lint 2022-01-14 17:15:44 -05:00
Jennifer Wang
6901e998e6 Check piece before PRU2 instead of PRU1 as PRU2 is the heavy computation part 2022-01-14 17:15:44 -05:00
Jennifer Wang
8939d5982f just use checkPiece 2022-01-14 17:15:44 -05:00
Jennifer Wang
a20916f9af Add more deal expiration handling for snap deals 2022-01-14 17:15:44 -05:00
Jennifer Wang
ac3bea489b Integrate proof v11.0.0 2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
ca57546ef5 Remove unnecessary params from VerifyWinningPost 2022-01-14 17:14:32 -05:00
zenground0
5b0a0baa9a Fix hande deal recover return value bug 2022-01-14 17:14:32 -05:00
zenground0
a9a523d8c0 Fix TooManyMarkedForUpgrade 2022-01-14 17:14:32 -05:00
zenground0
d6aa17e21f Snap Deals Integration 
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
 2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
3a8ac6dffa Update FFI 2022-01-14 17:14:32 -05:00
zenground0
d1480c36c0 RemoveData and Decode 
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
 2022-01-14 17:14:32 -05:00
zenground0
4936b4ea44 Review Response 2022-01-14 17:14:32 -05:00
zenground0
c4069824f7 WIP 2022-01-14 17:14:32 -05:00
zenground0
93656e65f8 WIP sector storage and integration test 2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
073b7b4ff5 Update FFI 2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
393d8541e2 Update deps 2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
1ef780d96f Plug in the FFI call 2022-01-14 17:14:32 -05:00
Aayush Rajasekaran
a8cb027c08 Integrate v7 actors 2022-01-14 17:14:32 -05:00
Łukasz Magiera
4a874eff70 post workers: Cleanup, tests 2022-01-14 14:17:52 +01:00
Aayush Rajasekaran
bda4e5be95 Appease the linter 2022-01-12 18:10:07 -05:00
Jennifer Wang
fd50cd128a fix lint 2022-01-11 18:34:26 -05:00
Jennifer Wang
e99b98873c Check piece before PRU2 instead of PRU1 as PRU2 is the heavy computation part 2022-01-11 18:34:20 -05:00
Jennifer Wang
6b953a03d0 just use checkPiece 2022-01-11 18:34:15 -05:00
Jennifer Wang
37a3e610b7 Add more deal expiration handling for snap deals 2022-01-11 18:34:10 -05:00
Jiaying Wang
0130b28879
Merge pull request #7923 from filecoin-project/jen/proofv11 
chore: deps: Integrate proof v11.0.0
 2022-01-11 17:49:43 -05:00
Aayush Rajasekaran
d645c5fbab Remove unnecessary params from VerifyWinningPost 2022-01-11 12:06:39 -05:00
Jennifer Wang
5b7da270c9 Integrate proof v11.0.0 2022-01-10 23:45:04 -05:00
Łukasz Magiera
e216aefd23 fix make gen 2022-01-10 18:24:00 +01:00
zenground0
d16c5d0e93 Fix hande deal recover return value bug 2022-01-10 15:47:20 +05:30
zenground0
c309686679 Fix TooManyMarkedForUpgrade 2022-01-10 15:39:38 +05:30
zenground0
33f2d24f54 Snap Deals Integration 
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
 2022-01-10 15:39:38 +05:30
mz-sirius
793b5c7cc3 fix ci err 2022-01-05 21:41:21 +08:00
mz-sirius
6867f017ec update ffi 2022-01-05 01:50:53 +08:00
mz-sirius
3fd55fa56b decoupling winningpost and windowpost from lotus-miner 2022-01-05 01:50:49 +08:00
llifezou
4b685c5e26 Include worker name in sealing errors 2021-12-23 17:44:43 +08:00
Darko Brdareski
dda1a42a2a Merge branch 'bloxico/system-test-matrix' of https://github.com/filecoin-project/lotus into merge_lotus 2021-12-20 15:48:16 +01:00
shotcollin
d10d0a20b1
fix typo in log warning 
very minor but this warning comes up a lot so it'd be nicer if it wasn't a grammatical error too
 2021-12-19 17:07:11 -07:00
Darko Brdareski
2f1f35cc71 Annotate storage miner features 2021-12-15 15:30:42 +01:00
Aayush Rajasekaran
3e288f1066 Update FFI 2021-12-13 15:47:17 -05:00
Aayush Rajasekaran
80d5e52923 Merge branch 'master' into next 2021-12-13 13:24:28 -05:00
Łukasz Magiera
ba3c96f8c6 stores: Reduce log spam during retrievals 2021-12-10 16:47:32 -05:00
Łukasz Magiera
dafdb7689c Fix mock ReadPiece 2021-12-10 16:47:12 -05:00
Łukasz Magiera
46ba2b6b4f fr32: Reduce MTTresh from 32M to 512k per core 
This results in 64x less bytes allocated when spawning new readers
for larger pieces.

Results in about 30% speedup in 1G unpad benchmark on AMD TR 2950x
 2021-12-10 16:47:07 -05:00
Łukasz Magiera
b4c1e340ea piecereader: Avoid allocating 1024MB slices per read 2021-12-10 16:47:01 -05:00
Łukasz Magiera
a438e6fa73 piecereader: Avoid redundant roundtrips when seeking 2021-12-10 16:46:57 -05:00
Łukasz Magiera
b21d3ded2f piecereader: Move closer to storage 2021-12-10 16:46:52 -05:00
Darko Brdareski
0169d0dafd Annotate state feature tests 2021-12-10 16:08:25 +01:00
Łukasz Magiera
e8ef39e734 stores: Reduce log spam during retrievals 2021-12-10 11:28:04 +01:00
Łukasz Magiera
c31f4de7d5 Fix mock ReadPiece 2021-12-09 16:26:59 +01:00
Łukasz Magiera
6fd1609410 fr32: Reduce MTTresh from 32M to 512k per core 
This results in 64x less bytes allocated when spawning new readers
for larger pieces.

Results in about 30% speedup in 1G unpad benchmark on AMD TR 2950x
 2021-12-09 16:14:47 +01:00
Łukasz Magiera
9c75a3aaa8 piecereader: Avoid allocating 1024MB slices per read 2021-12-09 15:49:43 +01:00
Łukasz Magiera
a3d8494a04 piecereader: Avoid redundant roundtrips when seeking 2021-12-09 14:52:33 +01:00
Łukasz Magiera
13b260e7f7 piecereader: Move closer to storage 2021-12-08 23:20:20 +01:00
zenground0
a5be80828a RemoveData and Decode 
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
 2021-12-03 15:21:06 -05:00
Łukasz Magiera
727765b248 Command to list active sector locks 2021-12-03 12:33:23 +01:00
Łukasz Magiera
71329f6c41 Address Scheduler enhancements (#7703) review 2021-11-30 20:50:40 +01:00
zenground0
40d16a8f88 Review Response 2021-11-30 13:53:37 -05:00
zenground0
f88fcdbcfc WIP 2021-11-30 12:40:14 -05:00
Łukasz Magiera
001ecbb561 fix lint 2021-11-30 02:06:58 +01:00
Łukasz Magiera
a597b072b8 fix sched tests 2021-11-30 02:06:58 +01:00
Łukasz Magiera
f25efecb74 worker: Test resource table overrides 2021-11-30 02:06:58 +01:00
Łukasz Magiera
6d52d8552b Fix docsgen 2021-11-30 02:06:58 +01:00
Łukasz Magiera
c9a2ff4007 cleanup worker resource overrides 2021-11-30 02:06:58 +01:00
Łukasz Magiera
b961e1aab5 sched resources: Separate Parallelism defaults depending on GPU presence 2021-11-30 02:06:58 +01:00
Łukasz Magiera
36868a8749 sched: C2 is not all-core load 2021-11-30 02:06:58 +01:00
Clint Armstrong
4ef8543128 Permit workers to override resource table 
In an environment with heterogenious worker nodes, a universal resource
table for all workers does not allow effective scheduling of tasks. Some
workers may have different proof cache settings, changing the required
memory for different tasks. Some workers may have a different count of
CPUs per core-complex, changing the max parallelism of PC1.

This change allows workers to customize these parameters with
environment variables. A worker could set the environment variable
PC1_MIN_MEMORY for example to customize the minimum memory requirement
for PC1 tasks. If no environment variables are specified, the resource
table on the miner is used, except for PC1 parallelism.

If PC1_MAX_PARALLELISM is not specified, and
FIL_PROOFS_USE_MULTICORE_SDR is set, PC1_MAX_PARALLELSIM will
automatically be set to FIL_PROOFS_MULTICORE_SDR_PRODUCERS + 1.
 2021-11-30 02:06:58 +01:00
Clint Armstrong
93e4656a27 Use a float to represent GPU utilization 
Before this change workers can only be allocated one GPU task,
regardless of how much of the GPU resources that task uses, or how many
GPUs are in the system.

This makes GPUUtilization a float which can represent that a task needs
a portion, or multiple GPUs. GPUs are accounted for like RAM and CPUs so
that workers with more GPUs can be allocated more tasks.

A known issue is that PC2 cannot use multiple GPUs. And even if the
worker has multiple GPUs and is allocated multiple PC2 tasks, those
tasks will only run on the first GPU.

This could result in unexpected behavior when a worker with multiple
GPUs is assigned multiple PC2 tasks. But this should not suprise any
existing users who upgrade, as any existing users who run workers with
multiple GPUs should already know this and be running a worker per GPU
for PC2. But now those users have the freedom to customize the GPU
utilization of PC2 to be less than one and effectively run multiple PC2
processes in a single worker.

C2 is capable of utilizing multiple GPUs, and now workers can be
customized for C2 accordingly.
 2021-11-30 02:06:58 +01:00
Clint Armstrong
c4f46171ae Report memory used and swap used in worker res 
Attempting to report "memory used by other processes" in the MemReserved
field fails to take into account the fact that the system's memory used
includes memory used by ongoing tasks.

To properly account for this, worker should report the memory and swap
used, then the scheduler that is aware of the memory requirements for a
task can determine if there is sufficient memory available for a task.
 2021-11-30 02:06:58 +01:00
Clint Armstrong
e2a1ca7caa Use cgroup limits in worker memory calculations 
Worker processes may have memory limitations imposed by Systemd. But
/proc/meminfo shows the entire system memory regardless of these limits.
This results in the scheduler believing the worker has the entire system
memory avaliable and the worker being allocated too many tasks.

This change attempts to read cgroup memory limits for the worker
process. It supports cgroups v1 and v2, and compares cgroup limits
against the system memory and returns the most conservative values to
prevent the worker from being allocated too many tasks and potentially
triggering an OOM event.
 2021-11-30 02:06:58 +01:00
Łukasz Magiera
d21c44e266 ffiwrapper: Validate PC2 by calling C1 with random seeds 2021-11-30 01:33:05 +01:00
zenground0
7d2b3f05db WIP sector storage and integration test 2021-11-29 10:24:00 -05:00
Łukasz Magiera
05aa860459 Request correct read size with startOffset in pieceProvider 2021-11-27 00:05:45 +01:00
Łukasz Magiera
743ce5a40f Add startOffset support to mock SectorMgr.ReadPiece 2021-11-26 18:48:52 +01:00
Łukasz Magiera
f6de16e95a Fix sector-storage tests 2021-11-26 18:16:53 +01:00
Łukasz Magiera
8d955d5f30 dagstore mount: Add random access support 2021-11-26 17:40:53 +01:00
Łukasz Magiera
8454abcf45 storage: Use 1M buffers for Tar transfers 2021-11-24 20:08:37 +01:00
Łukasz Magiera
2a1505b364 storage: Test StorageFindSector with groups 2021-11-23 16:11:04 +01:00
Łukasz Magiera
8b548ac02f storage: Check allowlists in StorageFindSector 2021-11-23 16:11:04 +01:00
Łukasz Magiera
5c77c25747 storage: Add Group tags to StorageInfo 2021-11-23 16:11:04 +01:00
Łukasz Magiera
d1a63e4173 remote store: Remove debug printf 2021-11-22 17:50:12 +01:00
Łukasz Magiera
2620eab45e
Merge pull request #7555 from diancun/dc/check-commit 
checkCommit should return SectorCommitFailed
 2021-11-22 12:28:16 +01:00
Aayush Rajasekaran
91fb111462 Update FFI 2021-11-17 17:50:36 -05:00
Aayush Rajasekaran
8665e32221 Update deps 2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
07f1be0ff6 Plug in the FFI call 2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
1449644c34 Integrate v7 actors 2021-11-17 15:55:14 -05:00
Jennifer Wang
1f339f623e update to proof v10.1.0 2021-10-26 14:36:19 -04:00
yaohcn
3123400759 checkCommit should return SectorCommitFailed 2021-10-22 10:09:44 +08:00
Łukasz Magiera
f7884c4f45
Merge pull request #7527 from filecoin-project/feat/show-prepared-waitres 
Show prepared tasks in sealing jobs
 2021-10-20 15:22:35 +01:00
Łukasz Magiera
726d9b86ea fix lint 2021-10-19 18:56:30 +02:00
Łukasz Magiera
852ac4c178 Expose per-state sector counts on the prometheus endpoint 2021-10-19 18:53:07 +02:00
Łukasz Magiera
e508055dc1 make gen 2021-10-19 11:13:23 +02:00
Łukasz Magiera
080aa3356a Fix locks in worker-tracked 2021-10-18 20:19:21 +02:00
Łukasz Magiera
70589e4406 Block work in tracked worker before it is started 2021-10-18 18:44:56 +02:00
Łukasz Magiera
261238e157 Show prepared tasks in sealing jobs 2021-10-18 18:44:56 +02:00
Łukasz Magiera
11d738eee0 Track prepared work 2021-10-18 18:44:56 +02:00
Łukasz Magiera
98ff1c4d20
Merge pull request #7531 from marten-seemann/statfs-build-constraint 
add missing build constraint to statfs_unix.go
 2021-10-18 16:02:48 +01:00
Marten Seemann
03806f7063 add missing build constraint to statfs_unix.go 2021-10-18 16:29:01 +02:00
Łukasz Magiera
63c8b8edd1 Fix used sectors space accounting after AddPieceFailed 2021-10-18 09:55:28 +02:00
Łukasz Magiera
f352c18290 Don't remove sector data when moving data into a shared path 2021-10-11 21:11:38 +02:00
Łukasz Magiera
aed6e8d0be
Merge pull request #7335 from filecoin-project/feat/sched-ready-work 
sealing: Improve scheduling of ready work
 2021-10-07 18:49:02 +01:00
Łukasz Magiera
1993efe201
Merge pull request #7444 from filecoin-project/feat/ap-fail-retry 
sealing: Recover sectors after failed AddPiece
 2021-10-07 03:18:15 +01:00
Łukasz Magiera
cadbd00ac4 sealing: Recover sectors after failed AddPiece 2021-10-04 20:00:07 +02:00
Jennifer Wang
38e01cba83 Merge branch 'release/v1.12.0' into jen/12tomaster 2021-10-04 02:38:00 -04:00
Łukasz Magiera
9af82f2d68 sched: Fix taskDone chan deadlock 2021-10-03 17:09:43 +02:00
Łukasz Magiera
b87142ec8e wip improve scheduling of ready work 2021-10-03 10:38:08 +02:00
Jennifer Wang
2e1d463933 make lint happy 2021-10-01 20:36:13 -04:00
Jennifer Wang
5f8c44125b Merge branch 'releases' into jen/12rc1prep 2021-10-01 20:31:29 -04:00
Łukasz Magiera
fc10281d96
Merge pull request #7420 from filecoin-project/deps/proofs-v10.0.0 
Update proofs to v10.0.0
 2021-10-01 22:20:09 +01:00
Aayush Rajasekaran
a335cb3767 Update to latest actors 2021-10-01 17:11:05 -04:00
Aayush Rajasekaran
f004d036dc Set BatchPreCommitAboveBaseFee correctly 2021-10-01 14:23:41 -04:00
Aayush Rajasekaran
f8a89cafd8 Fix logs 2021-10-01 14:23:41 -04:00
Aayush Rajasekaran
d6dd86b419 Fix lint 2021-10-01 14:23:41 -04:00
Łukasz Magiera
6fd9d5f28b Precommit batch balancer support/config 2021-10-01 14:23:41 -04:00
Łukasz Magiera
d7fbd8b67d Update proofs to v10.0.0 2021-10-01 18:38:27 +02:00
Anton Evangelatov
809289f5ef add Dealmaking.StartEpochSealingBuffer config 2021-10-01 17:44:15 +02:00
Anton Evangelatov
c2fa54ca9c check for deal start epoch on SectorAddPieceToAny 2021-10-01 17:42:09 +02:00
Łukasz Magiera
a8a9818043 Expose storage states on the metrics endpoint 2021-10-01 14:45:01 +02:00
Aayush Rajasekaran
8756df2447 Fix tests after specs update 2021-09-29 21:27:49 -04:00
Aayush Rajasekaran
84b6734063 Incorporate the new PublishStorageDealsReturn 2021-09-29 19:13:13 -04:00
Jennifer Wang
d7690aa719 Merge branch 'releases' into jen/backport 2021-09-29 03:21:19 -04:00
Adrian Lanzafame
92190c7e0b
Fix formatting 2021-09-23 12:39:39 +10:00
Anton Evangelatov
0c1e29390a unit test where StateMarketStorageDeal return nil, err 2021-09-21 22:13:51 -04:00
Anton Evangelatov
bf9d0bca40 GetCurrentDealInfo err: handle correctly err case 2021-09-21 22:13:46 -04:00
Jennifer Wang
e996c27619 update to ffi to update-bellperson-proofs-v9-0-2 2021-09-21 17:43:27 -04:00
Aayush Rajasekaran
b9bfcc4b69 Randomness: Move getters from ChainAPI to StateAPI 2021-09-21 14:20:15 -04:00
Łukasz Magiera
2437a4182d
Merge pull request #7322 from filecoin-project/asr/drand-state 
Randomness: Move getters from ChainAPI to StateAPI
 2021-09-21 09:20:12 +01:00
Anton Evangelatov
605953565b unit test where StateMarketStorageDeal return nil, err 2021-09-17 15:14:53 +02:00
Anton Evangelatov
34e3dc2b37 GetCurrentDealInfo err: handle correctly err case 2021-09-17 14:55:56 +02:00
Łukasz Magiera
438d5ce78e fix a panic in HandleRecoverDealIDs 2021-09-15 18:13:01 +02:00
Łukasz Magiera
ef03314c6d storagemgr: Cleanup workerLk around worker resources 2021-09-15 16:35:19 +02:00
Aayush Rajasekaran
1d6db306ec Randomness: Move getters from ChainAPI to StateAPI 2021-09-15 00:03:13 +02:00
swift-mx
b0b016b58c fix: check padSector Cid 2021-09-10 18:13:07 +08:00
Jennifer Wang
f751291f1a update to proof v0.9.2 2021-09-08 01:03:11 -04:00
frank
179458efe9 index out of range 2021-09-06 16:20:23 +08:00
Jennifer Wang
6379d5129f integrate the proof patch: tag proofs-v9-revert-deps-hotfix 2021-09-02 15:40:15 -04:00
Łukasz Magiera
7ef1b62b41 sealing: Fix sector state accounting with FinalizeEarly 2021-09-02 19:27:10 +02:00
Łukasz Magiera
41db98d49f sealing: Fix tests 2021-09-01 12:09:44 +02:00
Łukasz Magiera
b280e29409 sealing: Fix retry loop in SubmitCommitAggregate 2021-09-01 11:33:23 +02:00
Łukasz Magiera
3118bd1039 stores: Fix reserved disk usage log spam 2021-08-31 13:36:09 +02:00
Steven Allen
1cf556c3a2 feat: expose ChainGetPath on the gateway 2021-08-30 16:43:21 -07:00
Łukasz Magiera
2293ecd8e8 Reduce lotus-miner startup spam 2021-08-27 19:41:54 +02:00
Łukasz Magiera
b42171dc04 sealing: Check piece CIDs after AddPiece 2021-08-26 13:02:15 -07:00
Łukasz Magiera
62769e3b11 sealing: Fix RecoverDealIDs loop with changed PieceCID 2021-08-20 16:00:40 +02:00
Aayush Rajasekaran
113ea6dc4b Update to latest FFI 2021-08-17 18:15:12 -04:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
Steven Allen
498644acab
Merge pull request #7011 from filecoin-project/fix/no-t-aux 
fix: don't check for t_aux when proving
 2021-08-11 11:28:29 -07:00
Aayush Rajasekaran
a52352b13a PreCommitPolicy: Don't try to align expirations on proving period boundaries 2021-08-11 12:44:12 -04:00
ZenGround0
6da26dcefc Remove network version panic 2021-08-10 13:07:30 -04:00
Steven Allen
18f39be3ba fix: don't check for t_aux when proving 
We don't need it.
 2021-08-09 11:07:35 -07:00
mx
cfc10e9b52 fix: more logging in maybeStartBatch error 2021-08-06 16:50:37 +08:00
He Weidong
4273fb067c reduce diff 2021-07-29 15:01:52 +08:00
He Weidong
ef6620ad8e reasonable max value for initial sector expiration 2021-07-29 14:57:06 +08:00
He Weidong
ee20d30480 reasonable min and max value for initial sector expiration 2021-07-29 14:55:11 +08:00
Steven Allen
dbe4891a39
Merge branch 'master' into fix/check_ticket_msg 2021-07-28 15:25:40 -07:00
Aayush Rajasekaran
21244704fc
Merge branch 'master' into mg/feat/configurable-sector-expirations 2021-07-28 16:45:27 -04:00
Jennifer Wang
2790c9439f Update to proof v8.0.3 2021-07-27 13:29:07 -04:00
hunjixin
ac8937245c fix ticket check 2021-07-27 16:21:47 +08:00
Mike Greenberg
41bce7925a fix: Remove unnecessary Truncate 2021-07-23 20:12:30 -04:00
Mike Greenberg
3829d6bd9a fix: Remove actor method from autogen source; Move inline to pkg 2021-07-22 15:10:49 -04:00
Mike Greenberg
123a976f62 chore: Add unit test for custom CC lifetime value expiration 2021-07-22 15:07:03 -04:00
Mike Greenberg
7ee46ad4e0 fix: PreCommitPolicy unit tests 2021-07-22 15:07:03 -04:00
Mike Greenberg
adb62a3fff chore: Move cfg getter into PCPolicy; Clamp values on get 2021-07-22 15:07:03 -04:00
Mike Greenberg
9e34cce1f2 feat(config): support configurable CC sector expiration 2021-07-22 15:07:03 -04:00
Łukasz Magiera
8029a9ffe3 sealing: Address riview on preCommitParams send fix 2021-07-20 17:46:19 -07:00
johnli-helloworld
aea338ecf0 fix handleSubmitPreCommitBatch() preCommitParams error 2021-07-20 17:46:19 -07:00
Steven Allen
865e7ca80a test(sealing): test message not found 2021-07-20 12:09:50 -07:00
Steven Allen
ad1dd36193 fix: better error message on missing message 2021-07-20 11:20:45 -07:00
xloem
e0208e3595 fixes #6786 segfault 2021-07-20 11:20:45 -07:00
Łukasz Magiera
583a8a13d9
Merge pull request #6629 from filecoin-project/feat/pledge-from-miner-balance 
Config for collateral from miner available balance
 2021-07-13 17:22:00 +02:00
Łukasz Magiera
0678b3fa05 sealing: Cleanup DisableCollateralFallback handling 2021-07-13 17:06:41 +02:00
Łukasz Magiera
7526a074d9 sealing: collateral buffer / falback config 2021-07-13 17:04:58 +02:00
Łukasz Magiera
83f2368507 Add CollateralFromMinerBalance config 2021-07-12 14:11:58 +02:00
Anton Evangelatov
16784aa2cc remove pieceProvider from DI; small refactors 2021-07-12 11:30:26 +02:00
Anton Evangelatov
566a9d5541 fix signature 2021-07-06 18:12:30 +02:00
Anton Evangelatov
7a78527fde remove not used sa; cleanup 2021-07-06 17:46:21 +02:00
Anton Evangelatov
604be5fc82
Update extern/sector-storage/stores/remote.go 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-06 17:24:56 +02:00
Anton Evangelatov
ff2772a58c resolved conflicts 2021-07-06 16:00:41 +02:00
Anton Evangelatov
4be0a7a215 resolve conflicts with master 2021-07-05 13:13:32 +02:00
Peter Rabbitson
a544fa0f97 Fix more CID doucle-encoding as hex 2021-07-05 12:51:15 +02:00
Łukasz Magiera
e2f48b21ab
Merge pull request #6653 from filecoin-project/fix/finalize-in-storage 
storage: Fix FinalizeSector with sectors in stoage paths
 2021-07-02 20:38:25 +02:00
Łukasz Magiera
8a94ab676e storage: Fix FinalizeSector with sectors in stoage paths 2021-07-02 19:54:45 +02:00
Łukasz Magiera
35c0e3cf8c
Merge pull request #6658 from sunday527/master 
Fix: precommit_batch method used the wrong cfg.CommitBatchWait
 2021-07-02 15:18:48 +02:00
Łukasz Magiera
60ecbdee8f
Merge pull request #6635 from llifezou/fix_checkTicketExpired 
fix ticket expiration check
 2021-07-02 15:18:04 +02:00
zhoutian527
71e1577a62 Fix: precommit_batch method used the wrong cfg.PreCommitBatchWait 2021-07-02 15:53:21 +08:00
llifezou
df53b97fe4
Update extern/storage-sealing/states_sealing.go 
fix sector precommitted but expired judgment

Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-02 11:44:46 +08:00
llifezou
fe3ad4fd60
Update extern/storage-sealing/states_sealing.go 
fix log

Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2021-07-02 11:38:04 +08:00
Łukasz Magiera
7c2c8b2a95 commit batch: Regression test nil FailedSectors map 2021-07-01 13:51:11 +02:00
Łukasz Magiera
c094aa82ec commit batch: AggregateAboveBaseFee config 2021-07-01 13:33:54 +02:00
Łukasz Magiera
88bb9f422e commit batch: Initialize the FailedSectors map 2021-07-01 12:15:58 +02:00
llifezou
73e58f7af1 fix getTicket: sector precommitted but expired case 2021-07-01 10:53:42 +08:00
johnli-helloworld
229d5e5c80 handleSubmitCommitAggregate() exception handling 2021-06-30 15:29:21 +02:00
Łukasz Magiera
2dad06d59f
Merge pull request #6634 from GFZRZK/GFZRZK/correct_check_commit 
remove precommit check in handleCommitFailed
 2021-06-30 15:23:33 +02:00
Łukasz Magiera
f45340461d
gofmt 2021-06-30 15:19:29 +02:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
johnli-helloworld
b0b070ffe7 to optimize the batchwait 2021-06-30 16:56:40 +08:00
llifezou
3dd3476bfd fix ticket expiration check, otherwise it may cause a large number of loops to retry GetTicket when retrying PreCommit1 2021-06-30 16:32:44 +08:00
wangchao
a4342f3997 remove precommit check in handleCommitFailed 2021-06-30 11:09:07 +08:00
Łukasz Magiera
7b1b082381 Merge branch 'master' into fix/prove-commit-aggregate 2021-06-29 11:48:52 +02:00
Jerry
73704c92ba ensure agg fee is adequate 2021-06-29 14:36:26 +08:00
Jerry
5c3d67a811 fix: miner balance is not enough, so that ProveCommitAggregate msg exec failed 2021-06-29 14:06:41 +08:00
Łukasz Magiera
84dbcb63d2
Merge pull request #6566 from GFZRZK/GFZRZK/correct_sector_expire_time 
scale up sector expiration to avoid sector expire in batch-pre-commit waitting
 2021-06-24 10:59:49 +02:00
Łukasz Magiera
c3480dc0e8 Merge remote-tracking branch 'origin/releases' into chore/merge-release 2021-06-24 09:39:11 +02:00
Aayush Rajasekaran
616e5688fc Remove MinPreCommitBatch 2021-06-23 12:30:32 -04:00
Anton Evangelatov
21b51328f9 adding TestDealWithMarketAndMinerNode 2021-06-23 12:44:52 +02:00
wangchao
03261771e2 scale up sector expiration to avoid sector expire in batch-pre-commit waitting 2021-06-23 14:10:28 +08:00
Aayush Rajasekaran
93f7cbe587 Add a helpful comment 2021-06-22 19:31:36 -04:00
Aayush Rajasekaran
6ca2a14819 Always flush when timer goes off 2021-06-22 19:16:36 -04:00
Anton Evangelatov
6720463799 resolve merge conflicts 2021-06-22 11:28:23 +02:00
Raúl Kripalani
684cce198f add a unit test. 2021-06-21 20:49:24 +01:00
Raúl Kripalani
59eab2df25 move scheduling filtering logic down. 2021-06-21 20:49:16 +01:00
Raúl Kripalani
83839362c5 fix boolean condition. 2021-06-21 20:35:51 +01:00
Raúl Kripalani
b6147fb27f extern/storage: retype resource filtering strategy to enum. 2021-06-21 20:28:15 +01:00
Raúl Kripalani
f3b6f8de1a add ability to ignore worker resources when scheduling. 2021-06-21 20:08:18 +01:00
Łukasz Magiera
b37a66c7c9 Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-21 17:03:03 +02:00
Łukasz Magiera
738ac926d1 Update ffi 2021-06-18 16:27:37 +02:00
Łukasz Magiera
5516f3492e ffiwrapper: Assert aggregate length 2021-06-18 12:02:42 +02:00
Łukasz Magiera
7f00a6e59c Proofs v8.0.2 with fixed aggregate lengths 2021-06-18 11:17:18 +02:00
yaohcn
7c6736b3ef fix commit finalize failed 2021-06-18 12:09:02 +08:00
Łukasz Magiera
0a5a7cf45d storagefsm: Fix too-long log handling 2021-06-17 20:52:46 -04:00
Łukasz Magiera
b0128bd99e storagefsm: Fix race spawning more than one new sector at once 2021-06-17 20:52:42 -04:00
Łukasz Magiera
097baeb9b0 Make batch deal input test less flaky 2021-06-17 20:52:09 -04:00
Łukasz Magiera
5c8498b603 storagefsm: Fix batch deal packing behavior 2021-06-17 20:50:14 -04:00
Łukasz Magiera
44de67cf7d
Merge pull request #6175 from filecoin-project/feat/dynamic-retreival-pricing 
Dynamic Retrieval pricing
 2021-06-17 10:25:48 +02:00
aarshkshah1992
9002fcbe47 local unsealed file wont have the unsealed piece 2021-06-17 09:46:29 +05:30
Anton Evangelatov
d45bb14015 Merge branch 'master' into nonsense/split-market-miner-processes 2021-06-16 18:49:15 +02:00
Łukasz Magiera
1f914053b4 sealing: Wire up context to batchers 2021-06-16 11:43:06 +02:00
Łukasz Magiera
4bd2587e47
Merge pull request #6479 from filecoin-project/test/multicore-sdr 
Test multicore SDR support
 2021-06-16 11:02:09 +02:00
Łukasz Magiera
81b412399e sealing: Fix restartSectors race 2021-06-15 17:55:58 -04:00
Łukasz Magiera
74db586fdf sealing: Fix restartSectors race 2021-06-15 17:43:32 -04:00
Aayush Rajasekaran
520a0091f5 Merge branch 'releases' into release/v1.10.0 2021-06-15 17:40:36 -04:00
Łukasz Magiera
c7c593c74e TestMulticoreSDR: Setup rust logger after envvar check 2021-06-15 17:59:46 +02:00
Łukasz Magiera
6b0aed9317 Setup rust logger in the test 2021-06-15 17:53:26 +02:00
Łukasz Magiera
265afd696e Run TestMulticoreSDR on Circle 2021-06-15 17:46:06 +02:00
Łukasz Magiera
9ae780902a Test multicore SDR support 2021-06-15 17:36:10 +02:00
Łukasz Magiera
20c4fe9bba Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-15 16:50:23 +02:00
Aayush Rajasekaran
18b13f6f58 Update to fixed Bellperson 2021-06-14 20:19:26 -04:00
Łukasz Magiera
f9596dd730
Merge pull request #6432 from filecoin-project/tests/batching-unit 
Unit tests for sector batchers
 2021-06-14 21:05:34 +02:00
wangchao
4da30931c9 failed sectors should be added into res correctly 2021-06-14 11:33:38 -04:00
Łukasz Magiera
40efafb7cd Update ffi with fixed multicore sdr support 2021-06-14 17:29:21 +02:00
Łukasz Magiera
ff43d29bbd Update ffi with fixed multicore sdr support 2021-06-14 16:59:55 +02:00
Łukasz Magiera
0514f38dc4 Merge remote-tracking branch 'origin/master' into chore/merge-1.10 2021-06-14 13:27:53 +02:00
Anton Evangelatov
31e6fb154e resolve conflicts 2021-06-14 11:27:54 +02:00
wangchao
f392c1295c failed sectors should be added into res correctly 2021-06-12 23:22:32 +08:00
aarshkshah1992
ea9bed2746 undo ffi change 2021-06-11 16:22:09 +05:30
Łukasz Magiera
733240a2bb sealing: Test early finalization fsm planners 2021-06-11 11:52:00 +02:00
Łukasz Magiera
05d9b5ce0f sealing: Add missing planner for CommitFinalizeFailed 2021-06-11 11:44:37 +02:00
Łukasz Magiera
ec06f086ef sealing: Early finalization option 2021-06-11 11:41:28 +02:00
aarshkshah1992
fed5afa704 merge master 2021-06-11 09:35:20 +05:30
Łukasz Magiera
06195bc8e1 Unit tests for sector batchers 2021-06-10 13:46:36 +02:00
Aayush Rajasekaran
3766980331 Merge branch 'release/v1.10.0' into asr/merge-release 2021-06-09 18:41:10 -04:00
Łukasz Magiera
52199c9af3 Proofs v8.0.1 2021-06-10 00:17:13 +02:00
Łukasz Magiera
f0a2e97cb5 fee config for sector batching 2021-06-09 14:23:06 -04:00
Aayush Rajasekaran
dede1b4758 Merge branch 'release/v1.10.0' into asr/merge-release 2021-06-09 14:05:41 -04:00
Aayush Rajasekaran
b0c9dd49f0 Fund miners with the aggregate fee when ProveCommitting 2021-06-09 13:08:54 -04:00
Aayush Rajasekaran
dbb4e9fcc5 Drop soms logs 2021-06-09 12:26:20 -04:00
Anton Evangelatov
beb5bffaf6 fix log.Error; rename RemoteStore 2021-06-09 13:05:54 +02:00
Łukasz Magiera
92bb874327
Use correct batch fee config in commit batcher 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2021-06-09 12:17:18 +02:00
Aayush Rajasekaran
ba27d45214 Split the getSectorCutoff methods between precommit and commit batchers 2021-06-08 16:46:35 -04:00
Łukasz Magiera
78171055e7 fee config for sector batching 2021-06-08 15:45:32 +02:00
Anton Evangelatov
30ba9a2751 remove double http handler decl 2021-06-08 15:36:54 +02:00
Anton Evangelatov
4dd093b160 fix piece provider test 2021-06-08 14:57:45 +02:00
Anton Evangelatov
9f3ec82cc1 initial resolution of conflicts 2021-06-08 13:01:22 +02:00
aarshkshah1992
4e9bb16532 changes as per review 2021-06-08 10:09:27 +05:30
aarshkshah1992
212d0fc264 fix remote store diff 2021-06-08 09:54:45 +05:30
aarshkshah1992
e4e60d7af4 fix mock 2021-06-08 09:37:12 +05:30
aarshkshah1992
16166504f4 merged master 2021-06-08 09:17:40 +05:30
Aayush Rajasekaran
b13169f071 Rename deadlines to cutoffs in the batchers 2021-06-07 20:20:06 -04:00
aarshkshah1992
ddd9bf610e fix CI 2021-06-07 16:45:52 +05:30
aarshkshah1992
1a90d3bbb5 update ffi 2021-06-07 16:12:39 +05:30
Anton Evangelatov
b0cb0c1a4a do not depend on filecoin-ffi in api package 2021-06-07 11:51:25 +02:00
aarshkshah1992
8d9cef17af changes as per review 2021-06-07 15:03:09 +05:30
aarshkshah1992
2a134887c3 logs to debug read & unseal 2021-06-07 15:03:09 +05:30