Commit Graph

352 Commits

Author SHA1 Message Date
Łukasz Magiera
1e4456138e Merge master into feat/nv13 2021-05-27 12:28:20 +02:00
Łukasz Magiera
fc76a09b19 mock: Use real aggregate lengths 2021-05-26 10:06:41 +02:00
Łukasz Magiera
04658e1cae fix lint 2021-05-25 16:26:59 +02:00
Łukasz Magiera
f5409845b5 Some review addressing 2021-05-25 16:07:45 +02:00
aarshkshah1992
0f3ff9e06a fix typo 2021-05-24 15:22:11 +05:30
aarshkshah1992
dc6dbc9a11 dpr changes and test based on new unsealing PR 2021-05-22 22:40:21 +05:30
aarshkshah1992
50e023edd3 changes as per review 2021-05-21 19:15:05 +05:30
aarshkshah1992
2c9f5922b5 logs to debug read & unseal 2021-05-21 19:02:37 +05:30
aarshkshah1992
8ff5bce7a3 logs to debug read and unseal 2021-05-21 19:01:17 +05:30
aarshkshah1992
fb29f782df integration test should remove unsealed files 2021-05-21 16:15:08 +05:30
aarshkshah1992
536d7c4275 clean up logging 2021-05-21 15:50:25 +05:30
aarshkshah1992
40642b2cad better logging 2021-05-21 15:16:20 +05:30
aarshkshah1992
85f2ac8789 more logging 2021-05-21 14:56:37 +05:30
aarshkshah1992
78a0458ada finish integration tests 2021-05-21 11:00:17 +05:30
aarshkshah1992
c17300dc1f remove read task type and run gen and docsgen 2021-05-20 15:25:46 -06:00
aarshkshah1992
50d7acfa0c fix go mod 2021-05-20 15:25:46 -06:00
aarshkshah1992
bd9959070e unit tests for the remote store Reader 2021-05-20 15:25:46 -06:00
aarshkshah1992
9b34494501 use mockgen 2021-05-20 15:25:46 -06:00
aarshkshah1992
31a5f68dfa use an actual worker in the integration tests 2021-05-20 15:25:46 -06:00
Dirk McCormick
ec85a973b1 feat: TestPieceProviderReadPiece 2021-05-20 15:25:46 -06:00
aarshkshah1992
8c4c26ca4c fix linting problems 2021-05-20 15:25:46 -06:00
aarshkshah1992
dd9c9fd4fd test http handler 2021-05-20 15:25:46 -06:00
aarshkshah1992
c58048d16a address review comments 2021-05-20 15:25:46 -06:00
Aarsh Shah
d33d426692 Apply suggestions from code review
Co-authored-by: dirkmc <dirkmdev@gmail.com>
2021-05-20 15:25:46 -06:00
aarshkshah1992
73613ee883 docs, logs and green ci 2021-05-20 15:25:46 -06:00
aarshkshah1992
2a40c802ea bypass task scheduler for reading unsealed pieces 2021-05-20 15:25:46 -06:00
Anton Evangelatov
9a7b0b657e comment 2021-05-20 13:18:56 +02:00
Anton Evangelatov
19bd5beb96 update remoteGetSector 2021-05-20 13:15:57 +02:00
Anton Evangelatov
1a9b5760a7 add remoteGetAllocated http handler 2021-05-20 13:14:25 +02:00
Anton Evangelatov
76bb424de0 update StorageMinerAPI 2021-05-20 13:01:14 +02:00
Anton Evangelatov
cb603c62d9 update retrievaladapter ; add piece_provider 2021-05-20 12:49:53 +02:00
Anton Evangelatov
c12d802811 update reader to use ReadAtLeast 2021-05-20 12:38:38 +02:00
Anton Evangelatov
2aad7b6979 update sectorstorage.New (Manager) interface 2021-05-20 12:32:29 +02:00
Łukasz Magiera
e088c71b9a marketadapter: Handle batch sealing messages 2021-05-19 20:07:20 +02:00
Łukasz Magiera
a5677d1b7a ffiwrapper: Separate Prover interface 2021-05-19 15:20:23 +02:00
Anton Evangelatov
55401116bb add UnpaddedByteIndex.Valid() 2021-05-19 13:16:29 +02:00
Anton Evangelatov
e275b54f55 export PartialFile and OpenPartialFile 2021-05-19 13:14:16 +02:00
Anton Evangelatov
a989f60e27 add SectorAddPieceToAny and SectorUnsealPiece to StorageMiner iface; model moved to api package - PieceDealInfo, DealSchedule 2021-05-19 13:05:07 +02:00
Łukasz Magiera
dd393b470f Fix aggregation inputs 2021-05-18 20:36:47 +02:00
Łukasz Magiera
e400bdf87a Order proofs before aggregation 2021-05-18 18:58:41 +02:00
Łukasz Magiera
9690bc882c Test to trigger batching logic 2021-05-18 18:41:42 +02:00
Łukasz Magiera
fe9311e435 update ffi 2021-05-18 13:24:10 +02:00
Łukasz Magiera
6278bdc69a Make things build 2021-05-17 20:47:41 +02:00
Łukasz Magiera
506f39b294 WIP: Integrate FIP0013 2021-05-11 22:10:29 -04:00
Anton Evangelatov
eb13c74dce panic on unknown pathType 2021-05-11 18:14:01 +02:00
Anton Evangelatov
e07438417c consider storiface.PathStorage when calculating storage requirements 2021-05-11 13:19:26 +02:00
whyrusleeping
ac77c51d5e address nit 2021-04-29 10:24:16 -07:00
whyrusleeping
48feb52cbf add a test for adding padded pieces 2021-04-27 16:28:25 -07:00
whyrusleeping
6daaf6ac2e attempt to do better padding on pieces being written into sectors 2021-04-27 11:39:06 -07:00
Łukasz Magiera
a24a73bb52 Merge tag 'v1.8.0' into release/v1.9.0 2021-04-27 10:30:33 +02:00
Łukasz Magiera
c4137a438d Merge commit 'f1ded63d608e1b191d79c780b8065f8cff367c98' into feat/nv12-1.11 2021-04-27 08:22:55 +02:00
Aayush Rajasekaran
f1ded63d60 v4 specs-actors integration, nv12 migration 2021-04-27 02:02:44 -04:00
Dirk McCormick
61344644a4 feat: add more debug logging for unsealing 2021-04-22 11:33:28 +02:00
yaohcn
844d1d364d fix health report 2021-04-11 14:47:04 +08:00
Łukasz Magiera
9d4935a736
Merge pull request #5976 from filecoin-project/fix/storage-demote-stat-err
localstorage: Demote reservation stat error to debug
2021-04-09 21:08:08 +02:00
Łukasz Magiera
bed5ec81bd localstorage: Demote reservation stat error to debug 2021-04-06 16:06:47 +02:00
Peter Rabbitson
25a77d905d Centralize everything on ipfs/go-log/v2
I am not entirely sure this is right, but everything seems to build...
2021-04-06 15:04:32 +02:00
Łukasz Magiera
f372964a60 fix lint 2021-04-01 16:40:11 +02:00
Łukasz Magiera
c7de7882a1 Merge branch 'master' of github.com:ppiaas/lotus-ppiaas into fix/stg-fetch-close 2021-04-01 16:38:17 +02:00
yaohcn
3895b2ccca fix reservations 2021-03-31 14:30:24 +08:00
Steven Allen
b74b1658b2 test: mock proof side effects on verify as well 2021-03-12 09:06:41 -08:00
Łukasz Magiera
d9923125ff mock sealer: Simulate randomness sideeffects 2021-03-12 17:33:43 +01:00
Łukasz Magiera
cc490b983a
Merge pull request #5761 from filecoin-project/fix/testunpadreader-go116
Fix TestUnpadReader on Go 1.16
2021-03-10 13:13:27 +01:00
Łukasz Magiera
7fbb4bdd3b Fix TestUnpadReader on Go 1.16 2021-03-10 13:05:25 +01:00
Łukasz Magiera
40fdf6c180
Merge pull request #5624 from filecoin-project/feat/miner-storage-limit
Configurable storage path storage limit
2021-03-10 11:15:28 +01:00
Łukasz Magiera
fe230f901e Collect worker task metrics 2021-03-05 12:48:17 +01:00
Łukasz Magiera
35759fa07e
Merge pull request #5635 from filecoin-project/deps/cbg-soft-map
Update markets, cbor-gen with soft map decoding
2021-02-22 18:27:38 +01:00
Aayush Rajasekaran
6732758d0f Tweak mock ProveCommits to respect proof sizes 2021-02-19 17:22:59 -05:00
Łukasz Magiera
26399dba70 Update markets, cbor-gen with soft map decoding 2021-02-19 20:11:43 +01:00
Łukasz Magiera
37b1fb97fb Configurable storage path storage limit 2021-02-18 16:44:34 +01:00
yaohcn
3f66e731fa fix drop sector 2021-02-18 12:00:28 +08:00
Łukasz Magiera
67780346e2 storagemgr: Always unseal full sectors 2021-01-25 21:39:15 +01:00
Łukasz Magiera
48c6c55f68
Merge pull request #5384 from filecoin-project/fix-storage-size-estimation
fix FileSize to return correct disk usage recursively
2021-01-19 19:22:47 +01:00
Łukasz Magiera
560a127ce6
fsutil: Defensive check for Stat cast 2021-01-19 18:53:37 +01:00
Anton Evangelatov
834a6148b0 use stat.Blocks and not info.Size 2021-01-19 18:07:37 +01:00
Anton Evangelatov
16d07d3f18 fix FileSize to return correct disk usage recursively 2021-01-19 15:21:31 +01:00
Łukasz Magiera
5462ecf6da gofmt 2021-01-11 16:44:11 +01:00
Łukasz Magiera
9e74c3c4e8 Multicore AddPiece CommP 2021-01-11 16:44:11 +01:00
Łukasz Magiera
2c9f4ad5ad
Merge pull request #5251 from liismn/master
Fix deadlock in runWorker in sched_worker.go
2021-01-05 19:35:49 +01:00
Łukasz Magiera
44a56c7e33
Merge pull request #5208 from jsign/jsign/imprreadpiece
sector-storage/mock: improve mocked readpiece
2021-01-05 18:37:48 +01:00
15ho
5b235db366 fix: stores.FetchHandler: defer close file and increase buffer size 2020-12-29 10:07:28 +08:00
liismn
e9cc697ee8
Update sched_worker.go
Avoid possible deadlock by sh.workersLk
2020-12-25 15:05:01 +08:00
quanzhan lu
d806fafee8 update weight canSeal and canStore when attach
update  weight canSeal and canStore when attach storage
2020-12-22 19:07:17 +08:00
Ignacio Hagopian
f6cf27c781
improve mocked readpiece
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
2020-12-16 10:15:38 -03:00
Łukasz Magiera
46a5013dc1 stores: Simplify remote copy buf size logic 2020-12-10 20:54:42 +01:00
caoao
d4582fb32e write size according to RegisteredSealProof 2020-12-10 11:59:09 +08:00
beck
d3fa728894
Remove extra spaces 2020-12-09 19:22:21 +08:00
Łukasz Magiera
289ef910a0 fix imports, docsgen 2020-12-02 00:39:55 +01:00
Łukasz Magiera
95eaf13b5a sectorstorage: Fix tests 2020-12-02 00:36:32 +01:00
Łukasz Magiera
c6f85886bf miner: Add slow mode to proving check 2020-12-02 00:32:14 +01:00
Łukasz Magiera
614f45dcfb
Merge pull request #5041 from filecoin-project/feat/5013
Sector check command
2020-12-01 14:59:01 +01:00
Łukasz Magiera
a4f3758f4c worker api: better grouping 2020-11-30 23:16:30 +01:00
Łukasz Magiera
e8d2560581 Fix tests 2020-11-27 16:47:48 +01:00
Łukasz Magiera
3672053ae9 worker: Support setting task types at runtime 2020-11-26 17:33:34 +01:00
s1m0n21
663b3f4f21 add a sector check command 2020-11-26 15:02:43 +08:00
Łukasz Magiera
a63ef1dcd5
Merge pull request #4984 from yaohcn/fix-log-warn
fix log format
2020-11-24 18:01:56 +01:00
Łukasz Magiera
fe92a6e2d4
Merge pull request #4024 from zgfzgf/sector-sched
modify for unsafe
2020-11-24 16:19:17 +01:00
yaohcn
7c0b6f41d8 fix log format 2020-11-24 19:09:48 +08:00
Łukasz Magiera
0e39e760e5
Merge pull request #4929 from filecoin-project/feat/update-markets-commp-extraction
Remove all temp file generation for deals
2020-11-23 18:43:00 +01:00
Dirk McCormick
3634ca50ca fix: sector-storage tests 2020-11-23 16:09:57 +01:00
Dirk McCormick
c965696586 fix: ffiwrapper sealer test 2020-11-23 14:21:46 +01:00
Dan Shao
f3838a47c7 Format workerID as string 2020-11-23 15:07:50 +08:00
zgfzgf
b6893b0a3f solve merage problem 2020-11-22 16:15:30 +08:00
hannahhoward
218f7c2c5d feat(markets): update markets, extract commp 2020-11-20 13:29:14 -08:00
S1M0N21
2b611b3a99 Remove unused key 2020-11-19 23:48:28 +08:00
Łukasz Magiera
b242d69805 Make storiface.CallError json-friendly 2020-11-17 16:28:41 +01:00
Łukasz Magiera
b8853aa4d5 Add error codes to worker return 2020-11-17 16:17:55 +01:00
Łukasz Magiera
e0ade92b29 Fix lint warnings 2020-11-16 19:20:18 +01:00
Łukasz Magiera
915c5e0fdd Get most tests to pass 2020-11-16 19:03:30 +01:00
Łukasz Magiera
2a7e4af5ee Fix lotus-miner/lotus-worker build 2020-11-16 19:03:30 +01:00
Steven Allen
584907269a make sector-storage compile 2020-11-16 19:03:30 +01:00
Łukasz Magiera
6bea9dd178 Making sealing logic work with multiple seal proof types 2020-11-16 19:03:30 +01:00
Łukasz Magiera
408183c7a6 docsgen: Fix CI 2020-11-12 20:15:57 +01:00
Łukasz Magiera
8409e155d4 docsgen: Support miner/worker structs 2020-11-12 10:29:42 +01:00
Łukasz Magiera
8ac495723e gofmt 2020-11-11 17:42:23 +01:00
Łukasz Magiera
09f9f871a3 Create a command to abort sealing calls 2020-11-11 17:39:12 +01:00
Łukasz Magiera
bd71bf2379 Expand sched-diag 2020-11-11 16:47:44 +01:00
Łukasz Magiera
18aa97f962
Merge pull request #4779 from filecoin-project/feat/work-track-hostname
storage: Track worker hostnames with work
2020-11-10 20:38:44 +01:00
Łukasz Magiera
5caa277261 storage: Track abandoned work more correctly 2020-11-09 23:38:20 +01:00
Łukasz Magiera
f819e71d12 storage: Separate returned jobs in jobs cli 2020-11-09 23:13:29 +01:00
Łukasz Magiera
27a9dd3bbb storage: Track worker hostnames with work 2020-11-09 23:09:04 +01:00
Łukasz Magiera
5b8f555221 sched: Update after tasks finish preparing 2020-11-09 21:44:28 +01:00
Łukasz Magiera
fbc05fa5a8 stores: Report storage changes quickly 2020-11-09 21:43:52 +01:00
zgfzgf
5bcc6339b4 optimize code replace strings with constants 2020-11-09 16:21:16 +08:00
stirlingx
b5d2f01993
Update selector_task.go 2020-11-03 09:20:48 +08:00
Łukasz Magiera
094ea3fe97
Merge pull request #4666 from filecoin-project/fix/sched-issues
Fix worker reenabling, handle multiple restarts in worker
2020-10-30 21:34:07 +01:00
Łukasz Magiera
69e44ebf07 sched: re-register worker windows after re-enabling correctly 2020-10-30 18:32:59 +01:00
Łukasz Magiera
f90a387f96 sched: Print worker UUIDs in shed-diag correctly 2020-10-30 18:32:16 +01:00
Łukasz Magiera
774e2ecebf sched: Fix worker reenabling 2020-10-30 18:01:37 +01:00
Łukasz Magiera
7fbb868513 Debug flag to force running sealing scheduler 2020-10-30 11:07:35 +01:00
Łukasz Magiera
c6b03ce62b sectorstorage: Missing unlock in waitWork 2020-10-29 15:18:51 +01:00
Łukasz Magiera
ea5bb5cdab sectorstorage: Fix manager restart edge-case 2020-10-29 12:14:21 +01:00
Łukasz Magiera
f0f75e2d2c
Merge pull request #4627 from karalabe/fix-gpu-usage-tracking
extern/sector-storage: fix GPU usage overwrite bug
2020-10-29 10:13:54 +01:00
Péter Szilágyi
5f657b4333
extern/sector-storage: fix GPU usage overwrite bug 2020-10-28 20:52:33 +02:00
Łukasz Magiera
da7ecc1527 Fix flaky sealing manager tests 2020-10-28 16:15:17 +01:00
Łukasz Magiera
4100f6eead fix TestWDPostDoPost 2020-10-28 15:23:21 +01:00
Łukasz Magiera
ed2f81da2f sched: Fix tests 2020-10-28 14:34:28 +01:00
Łukasz Magiera
4cf00b8b42 worker_local: address review 2020-10-28 14:29:17 +01:00
Łukasz Magiera
96c5ff7e7f sched: use more letters for variables 2020-10-28 14:23:38 +01:00
Łukasz Magiera
8731fe9112 sched: split worker handling into more funcs 2020-10-28 14:14:50 +01:00
Łukasz Magiera
84b567c790 sched: move worker funcs to a separate file 2020-10-28 13:39:28 +01:00
Łukasz Magiera
660236b224 Merge remote-tracking branch 'origin/master' into feat/async-restartable-workers 2020-10-23 23:25:35 +02:00
Łukasz Magiera
29e334de54
Merge pull request #4511 from filecoin-project/steb/generalize-window-post
Manage sectors by size instead of proof type.
2020-10-22 21:27:48 +02:00
Steven Allen
4e730b5ec8 port to v2 imports 2020-10-21 12:16:23 -07:00
Steven Allen
00dcb1bce9 Manage sectors by size instead of proof type.
* We may have multiple sectors with the same size and different proof types, but all these management functions stay the same.
* This simplifies PoSt logic.
2020-10-20 18:30:56 -07:00
Łukasz Magiera
8c86ea6b75 localworker: Try very hard to get ruselts to manager 2020-10-18 19:45:11 +02:00
Łukasz Magiera
dbb421c4f7 localworker: Use better context for calling returnFunc 2020-10-18 19:32:43 +02:00
Łukasz Magiera
8d06cca073 sched: Handle workers using sessions instead of connections 2020-10-18 12:36:06 +02:00
Łukasz Magiera
71b3b9075d Merge remote-tracking branch 'origin/master' into feat/async-restartable-workers 2020-10-13 21:33:21 +02:00
Steven Allen
dc4e73c737 Test the tape upgrade 2020-10-12 00:01:25 -07:00