cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
15,816 Commits 24 Branches 342 Tags 314 MiB
8c6cba7a03
Commit Graph

518 Commits

Author SHA1 Message Date
Łukasz Magiera
ef03314c6d storagemgr: Cleanup workerLk around worker resources 2021-09-15 16:35:19 +02:00
Łukasz Magiera
3118bd1039 stores: Fix reserved disk usage log spam 2021-08-31 13:36:09 +02:00
Steven Allen
1cf556c3a2 feat: expose ChainGetPath on the gateway 2021-08-30 16:43:21 -07:00
Łukasz Magiera
2293ecd8e8 Reduce lotus-miner startup spam 2021-08-27 19:41:54 +02:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
Steven Allen
18f39be3ba fix: don't check for t_aux when proving 
We don't need it.
 2021-08-09 11:07:35 -07:00
Anton Evangelatov
16784aa2cc remove pieceProvider from DI; small refactors 2021-07-12 11:30:26 +02:00
Anton Evangelatov
566a9d5541 fix signature 2021-07-06 18:12:30 +02:00
Anton Evangelatov
7a78527fde remove not used sa; cleanup 2021-07-06 17:46:21 +02:00
Anton Evangelatov
604be5fc82
Update extern/sector-storage/stores/remote.go 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-06 17:24:56 +02:00
Anton Evangelatov
4be0a7a215 resolve conflicts with master 2021-07-05 13:13:32 +02:00
Łukasz Magiera
8a94ab676e storage: Fix FinalizeSector with sectors in stoage paths 2021-07-02 19:54:45 +02:00
Anton Evangelatov
21b51328f9 adding TestDealWithMarketAndMinerNode 2021-06-23 12:44:52 +02:00
Anton Evangelatov
6720463799 resolve merge conflicts 2021-06-22 11:28:23 +02:00
Raúl Kripalani
684cce198f add a unit test. 2021-06-21 20:49:24 +01:00
Raúl Kripalani
59eab2df25 move scheduling filtering logic down. 2021-06-21 20:49:16 +01:00
Raúl Kripalani
83839362c5 fix boolean condition. 2021-06-21 20:35:51 +01:00
Raúl Kripalani
b6147fb27f extern/storage: retype resource filtering strategy to enum. 2021-06-21 20:28:15 +01:00
Raúl Kripalani
f3b6f8de1a add ability to ignore worker resources when scheduling. 2021-06-21 20:08:18 +01:00
Łukasz Magiera
b37a66c7c9 Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-21 17:03:03 +02:00
Łukasz Magiera
5516f3492e ffiwrapper: Assert aggregate length 2021-06-18 12:02:42 +02:00
Łukasz Magiera
44de67cf7d
Merge pull request #6175 from filecoin-project/feat/dynamic-retreival-pricing 
Dynamic Retrieval pricing
 2021-06-17 10:25:48 +02:00
aarshkshah1992
9002fcbe47 local unsealed file wont have the unsealed piece 2021-06-17 09:46:29 +05:30
Anton Evangelatov
d45bb14015 Merge branch 'master' into nonsense/split-market-miner-processes 2021-06-16 18:49:15 +02:00
Aayush Rajasekaran
520a0091f5 Merge branch 'releases' into release/v1.10.0 2021-06-15 17:40:36 -04:00
Łukasz Magiera
c7c593c74e TestMulticoreSDR: Setup rust logger after envvar check 2021-06-15 17:59:46 +02:00
Łukasz Magiera
6b0aed9317 Setup rust logger in the test 2021-06-15 17:53:26 +02:00
Łukasz Magiera
265afd696e Run TestMulticoreSDR on Circle 2021-06-15 17:46:06 +02:00
Łukasz Magiera
9ae780902a Test multicore SDR support 2021-06-15 17:36:10 +02:00
Anton Evangelatov
beb5bffaf6 fix log.Error; rename RemoteStore 2021-06-09 13:05:54 +02:00
Anton Evangelatov
30ba9a2751 remove double http handler decl 2021-06-08 15:36:54 +02:00
Anton Evangelatov
4dd093b160 fix piece provider test 2021-06-08 14:57:45 +02:00
Anton Evangelatov
9f3ec82cc1 initial resolution of conflicts 2021-06-08 13:01:22 +02:00
aarshkshah1992
4e9bb16532 changes as per review 2021-06-08 10:09:27 +05:30
aarshkshah1992
212d0fc264 fix remote store diff 2021-06-08 09:54:45 +05:30
aarshkshah1992
e4e60d7af4 fix mock 2021-06-08 09:37:12 +05:30
aarshkshah1992
16166504f4 merged master 2021-06-08 09:17:40 +05:30
aarshkshah1992
ddd9bf610e fix CI 2021-06-07 16:45:52 +05:30
Anton Evangelatov
b0cb0c1a4a do not depend on filecoin-ffi in api package 2021-06-07 11:51:25 +02:00
aarshkshah1992
8d9cef17af changes as per review 2021-06-07 15:03:09 +05:30
aarshkshah1992
2a134887c3 logs to debug read & unseal 2021-06-07 15:03:09 +05:30
aarshkshah1992
ec6a49693f logs to debug read and unseal 2021-06-07 15:03:09 +05:30
aarshkshah1992
207f0d901a integration test should remove unsealed files 2021-06-07 15:03:09 +05:30
aarshkshah1992
73f7825fbb clean up logging 2021-06-07 15:03:09 +05:30
aarshkshah1992
3b792a32c3 better logging 2021-06-07 15:03:09 +05:30
aarshkshah1992
22f36483cb more logging 2021-06-07 15:03:09 +05:30
aarshkshah1992
acfa3d7370 finish integration tests 2021-06-07 15:03:09 +05:30
aarshkshah1992
ad4b182bfe remove read task type and run gen and docsgen 2021-06-07 15:03:06 +05:30
aarshkshah1992
35a0dbfa8c fix go mod 2021-06-07 15:02:52 +05:30
aarshkshah1992
4efaa8d688 unit tests for the remote store Reader 2021-06-07 15:02:52 +05:30
aarshkshah1992
0d88800eb9 use mockgen 2021-06-07 15:02:52 +05:30
aarshkshah1992
77b5e8d045 use an actual worker in the integration tests 2021-06-07 15:02:52 +05:30
Dirk McCormick
6879ae9e6a feat: TestPieceProviderReadPiece 2021-06-07 15:02:52 +05:30
aarshkshah1992
74372d3e81 fix linting problems 2021-06-07 15:02:52 +05:30
aarshkshah1992
759d8f090b test http handler 2021-06-07 15:02:52 +05:30
aarshkshah1992
db5c88196d address review comments 2021-06-07 15:02:52 +05:30
Aarsh Shah
c853350bdf Apply suggestions from code review 
Co-authored-by: dirkmc <dirkmdev@gmail.com>
 2021-06-07 15:02:52 +05:30
aarshkshah1992
65eb610ec3 docs, logs and green ci 2021-06-07 15:02:49 +05:30
aarshkshah1992
670835fca0 bypass task scheduler for reading unsealed pieces 2021-06-07 15:02:04 +05:30
Anton Evangelatov
8bd09e39ca resolve merge conflicts 2021-06-04 16:17:00 +02:00
Anton Evangelatov
88756f3ebf fix TestAPIDeal tests 2021-06-04 15:41:38 +02:00
aarshkshah1992
05ba3de5cc changes as per review 2021-06-04 10:27:43 +05:30
Łukasz Magiera
565bb4f589 mock: Log debug info on bad aggregates 2021-06-03 17:35:10 +02:00
aarshkshah1992
43453bb633 fix broken test 2021-06-03 10:15:23 +05:30
Łukasz Magiera
40cc29d723 Skip FD check in TestDownloadParams 2021-06-02 19:50:17 +02:00
Anton Evangelatov
d195a12565 Merge branch 'master' into nonsense/split-market-miner-processes 2021-05-31 11:26:11 +02:00
Łukasz Magiera
1e4456138e Merge master into feat/nv13 2021-05-27 12:28:20 +02:00
Łukasz Magiera
fc76a09b19 mock: Use real aggregate lengths 2021-05-26 10:06:41 +02:00
Łukasz Magiera
04658e1cae fix lint 2021-05-25 16:26:59 +02:00
Łukasz Magiera
f5409845b5 Some review addressing 2021-05-25 16:07:45 +02:00
aarshkshah1992
0f3ff9e06a fix typo 2021-05-24 15:22:11 +05:30
aarshkshah1992
dc6dbc9a11 dpr changes and test based on new unsealing PR 2021-05-22 22:40:21 +05:30
aarshkshah1992
50e023edd3 changes as per review 2021-05-21 19:15:05 +05:30
aarshkshah1992
2c9f5922b5 logs to debug read & unseal 2021-05-21 19:02:37 +05:30
aarshkshah1992
8ff5bce7a3 logs to debug read and unseal 2021-05-21 19:01:17 +05:30
aarshkshah1992
fb29f782df integration test should remove unsealed files 2021-05-21 16:15:08 +05:30
aarshkshah1992
536d7c4275 clean up logging 2021-05-21 15:50:25 +05:30
aarshkshah1992
40642b2cad better logging 2021-05-21 15:16:20 +05:30
aarshkshah1992
85f2ac8789 more logging 2021-05-21 14:56:37 +05:30
aarshkshah1992
78a0458ada finish integration tests 2021-05-21 11:00:17 +05:30
aarshkshah1992
c17300dc1f remove read task type and run gen and docsgen 2021-05-20 15:25:46 -06:00
aarshkshah1992
50d7acfa0c fix go mod 2021-05-20 15:25:46 -06:00
aarshkshah1992
bd9959070e unit tests for the remote store Reader 2021-05-20 15:25:46 -06:00
aarshkshah1992
9b34494501 use mockgen 2021-05-20 15:25:46 -06:00
aarshkshah1992
31a5f68dfa use an actual worker in the integration tests 2021-05-20 15:25:46 -06:00
Dirk McCormick
ec85a973b1 feat: TestPieceProviderReadPiece 2021-05-20 15:25:46 -06:00
aarshkshah1992
8c4c26ca4c fix linting problems 2021-05-20 15:25:46 -06:00
aarshkshah1992
dd9c9fd4fd test http handler 2021-05-20 15:25:46 -06:00
aarshkshah1992
c58048d16a address review comments 2021-05-20 15:25:46 -06:00
Aarsh Shah
d33d426692 Apply suggestions from code review 
Co-authored-by: dirkmc <dirkmdev@gmail.com>
 2021-05-20 15:25:46 -06:00
aarshkshah1992
73613ee883 docs, logs and green ci 2021-05-20 15:25:46 -06:00
aarshkshah1992
2a40c802ea bypass task scheduler for reading unsealed pieces 2021-05-20 15:25:46 -06:00
Anton Evangelatov
9a7b0b657e comment 2021-05-20 13:18:56 +02:00
Anton Evangelatov
19bd5beb96 update remoteGetSector 2021-05-20 13:15:57 +02:00
Anton Evangelatov
1a9b5760a7 add remoteGetAllocated http handler 2021-05-20 13:14:25 +02:00
Anton Evangelatov
76bb424de0 update StorageMinerAPI 2021-05-20 13:01:14 +02:00
Anton Evangelatov
cb603c62d9 update retrievaladapter ; add piece_provider 2021-05-20 12:49:53 +02:00
Anton Evangelatov
c12d802811 update reader to use ReadAtLeast 2021-05-20 12:38:38 +02:00
Anton Evangelatov
2aad7b6979 update sectorstorage.New (Manager) interface 2021-05-20 12:32:29 +02:00
Łukasz Magiera
e088c71b9a marketadapter: Handle batch sealing messages 2021-05-19 20:07:20 +02:00
Łukasz Magiera
a5677d1b7a ffiwrapper: Separate Prover interface 2021-05-19 15:20:23 +02:00
Anton Evangelatov
55401116bb add UnpaddedByteIndex.Valid() 2021-05-19 13:16:29 +02:00
Anton Evangelatov
e275b54f55 export PartialFile and OpenPartialFile 2021-05-19 13:14:16 +02:00
Anton Evangelatov
a989f60e27 add SectorAddPieceToAny and SectorUnsealPiece to StorageMiner iface; model moved to api package - PieceDealInfo, DealSchedule 2021-05-19 13:05:07 +02:00
Łukasz Magiera
dd393b470f Fix aggregation inputs 2021-05-18 20:36:47 +02:00
Łukasz Magiera
e400bdf87a Order proofs before aggregation 2021-05-18 18:58:41 +02:00
Łukasz Magiera
9690bc882c Test to trigger batching logic 2021-05-18 18:41:42 +02:00
Łukasz Magiera
fe9311e435 update ffi 2021-05-18 13:24:10 +02:00
Łukasz Magiera
6278bdc69a Make things build 2021-05-17 20:47:41 +02:00
Łukasz Magiera
506f39b294 WIP: Integrate FIP0013 2021-05-11 22:10:29 -04:00
Anton Evangelatov
eb13c74dce panic on unknown pathType 2021-05-11 18:14:01 +02:00
Anton Evangelatov
e07438417c consider storiface.PathStorage when calculating storage requirements 2021-05-11 13:19:26 +02:00
whyrusleeping
ac77c51d5e address nit 2021-04-29 10:24:16 -07:00
whyrusleeping
48feb52cbf add a test for adding padded pieces 2021-04-27 16:28:25 -07:00
whyrusleeping
6daaf6ac2e attempt to do better padding on pieces being written into sectors 2021-04-27 11:39:06 -07:00
Łukasz Magiera
a24a73bb52 Merge tag 'v1.8.0' into release/v1.9.0 2021-04-27 10:30:33 +02:00
Łukasz Magiera
c4137a438d Merge commit 'f1ded63d608e1b191d79c780b8065f8cff367c98' into feat/nv12-1.11 2021-04-27 08:22:55 +02:00
Aayush Rajasekaran
f1ded63d60 v4 specs-actors integration, nv12 migration 2021-04-27 02:02:44 -04:00
Dirk McCormick
61344644a4 feat: add more debug logging for unsealing 2021-04-22 11:33:28 +02:00
yaohcn
844d1d364d fix health report 2021-04-11 14:47:04 +08:00
Łukasz Magiera
9d4935a736
Merge pull request #5976 from filecoin-project/fix/storage-demote-stat-err 
localstorage: Demote reservation stat error to debug
 2021-04-09 21:08:08 +02:00
Łukasz Magiera
bed5ec81bd localstorage: Demote reservation stat error to debug 2021-04-06 16:06:47 +02:00
Peter Rabbitson
25a77d905d Centralize everything on ipfs/go-log/v2 
I am not entirely sure this is right, but everything seems to build...
 2021-04-06 15:04:32 +02:00
Łukasz Magiera
f372964a60 fix lint 2021-04-01 16:40:11 +02:00
Łukasz Magiera
c7de7882a1 Merge branch 'master' of github.com:ppiaas/lotus-ppiaas into fix/stg-fetch-close 2021-04-01 16:38:17 +02:00
yaohcn
3895b2ccca fix reservations 2021-03-31 14:30:24 +08:00
Steven Allen
b74b1658b2 test: mock proof side effects on verify as well 2021-03-12 09:06:41 -08:00
Łukasz Magiera
d9923125ff mock sealer: Simulate randomness sideeffects 2021-03-12 17:33:43 +01:00
Łukasz Magiera
cc490b983a
Merge pull request #5761 from filecoin-project/fix/testunpadreader-go116 
Fix TestUnpadReader on Go 1.16
 2021-03-10 13:13:27 +01:00
Łukasz Magiera
7fbb4bdd3b Fix TestUnpadReader on Go 1.16 2021-03-10 13:05:25 +01:00
Łukasz Magiera
40fdf6c180
Merge pull request #5624 from filecoin-project/feat/miner-storage-limit 
Configurable storage path storage limit
 2021-03-10 11:15:28 +01:00
Łukasz Magiera
fe230f901e Collect worker task metrics 2021-03-05 12:48:17 +01:00
Łukasz Magiera
35759fa07e
Merge pull request #5635 from filecoin-project/deps/cbg-soft-map 
Update markets, cbor-gen with soft map decoding
 2021-02-22 18:27:38 +01:00
Aayush Rajasekaran
6732758d0f Tweak mock ProveCommits to respect proof sizes 2021-02-19 17:22:59 -05:00
Łukasz Magiera
26399dba70 Update markets, cbor-gen with soft map decoding 2021-02-19 20:11:43 +01:00
Łukasz Magiera
37b1fb97fb Configurable storage path storage limit 2021-02-18 16:44:34 +01:00
yaohcn
3f66e731fa fix drop sector 2021-02-18 12:00:28 +08:00
Łukasz Magiera
67780346e2 storagemgr: Always unseal full sectors 2021-01-25 21:39:15 +01:00
Łukasz Magiera
48c6c55f68
Merge pull request #5384 from filecoin-project/fix-storage-size-estimation 
fix FileSize to return correct disk usage recursively
 2021-01-19 19:22:47 +01:00
Łukasz Magiera
560a127ce6
fsutil: Defensive check for Stat cast 2021-01-19 18:53:37 +01:00
Anton Evangelatov
834a6148b0 use stat.Blocks and not info.Size 2021-01-19 18:07:37 +01:00
Anton Evangelatov
16d07d3f18 fix FileSize to return correct disk usage recursively 2021-01-19 15:21:31 +01:00
Łukasz Magiera
5462ecf6da gofmt 2021-01-11 16:44:11 +01:00
Łukasz Magiera
9e74c3c4e8 Multicore AddPiece CommP 2021-01-11 16:44:11 +01:00
Łukasz Magiera
2c9f4ad5ad
Merge pull request #5251 from liismn/master 
Fix deadlock in runWorker in sched_worker.go
 2021-01-05 19:35:49 +01:00
Łukasz Magiera
44a56c7e33
Merge pull request #5208 from jsign/jsign/imprreadpiece 
sector-storage/mock: improve mocked readpiece
 2021-01-05 18:37:48 +01:00
15ho
5b235db366 fix: stores.FetchHandler: defer close file and increase buffer size 2020-12-29 10:07:28 +08:00
liismn
e9cc697ee8
Update sched_worker.go 
Avoid possible deadlock by sh.workersLk
 2020-12-25 15:05:01 +08:00
quanzhan lu
d806fafee8 update weight canSeal and canStore when attach 
update  weight canSeal and canStore when attach storage
 2020-12-22 19:07:17 +08:00
Ignacio Hagopian
f6cf27c781
improve mocked readpiece 
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
 2020-12-16 10:15:38 -03:00
Łukasz Magiera
46a5013dc1 stores: Simplify remote copy buf size logic 2020-12-10 20:54:42 +01:00
caoao
d4582fb32e write size according to RegisteredSealProof 2020-12-10 11:59:09 +08:00
beck
d3fa728894
Remove extra spaces 2020-12-09 19:22:21 +08:00
Łukasz Magiera
289ef910a0 fix imports, docsgen 2020-12-02 00:39:55 +01:00
Łukasz Magiera
95eaf13b5a sectorstorage: Fix tests 2020-12-02 00:36:32 +01:00
Łukasz Magiera
c6f85886bf miner: Add slow mode to proving check 2020-12-02 00:32:14 +01:00
Łukasz Magiera
614f45dcfb
Merge pull request #5041 from filecoin-project/feat/5013 
Sector check command
 2020-12-01 14:59:01 +01:00
Łukasz Magiera
a4f3758f4c worker api: better grouping 2020-11-30 23:16:30 +01:00
Łukasz Magiera
e8d2560581 Fix tests 2020-11-27 16:47:48 +01:00
Łukasz Magiera
3672053ae9 worker: Support setting task types at runtime 2020-11-26 17:33:34 +01:00
s1m0n21
663b3f4f21 add a sector check command 2020-11-26 15:02:43 +08:00
Łukasz Magiera
a63ef1dcd5
Merge pull request #4984 from yaohcn/fix-log-warn 
fix log format
 2020-11-24 18:01:56 +01:00
Łukasz Magiera
fe92a6e2d4
Merge pull request #4024 from zgfzgf/sector-sched 
modify for unsafe
 2020-11-24 16:19:17 +01:00
yaohcn
7c0b6f41d8 fix log format 2020-11-24 19:09:48 +08:00
Łukasz Magiera
0e39e760e5
Merge pull request #4929 from filecoin-project/feat/update-markets-commp-extraction 
Remove all temp file generation for deals
 2020-11-23 18:43:00 +01:00
Dirk McCormick
3634ca50ca fix: sector-storage tests 2020-11-23 16:09:57 +01:00
Dirk McCormick
c965696586 fix: ffiwrapper sealer test 2020-11-23 14:21:46 +01:00
Dan Shao
f3838a47c7 Format workerID as string 2020-11-23 15:07:50 +08:00
zgfzgf
b6893b0a3f solve merage problem 2020-11-22 16:15:30 +08:00
hannahhoward
218f7c2c5d feat(markets): update markets, extract commp 2020-11-20 13:29:14 -08:00
S1M0N21
2b611b3a99 Remove unused key 2020-11-19 23:48:28 +08:00
Łukasz Magiera
b242d69805 Make storiface.CallError json-friendly 2020-11-17 16:28:41 +01:00
Łukasz Magiera
b8853aa4d5 Add error codes to worker return 2020-11-17 16:17:55 +01:00
Łukasz Magiera
e0ade92b29 Fix lint warnings 2020-11-16 19:20:18 +01:00
Łukasz Magiera
915c5e0fdd Get most tests to pass 2020-11-16 19:03:30 +01:00
Łukasz Magiera
2a7e4af5ee Fix lotus-miner/lotus-worker build 2020-11-16 19:03:30 +01:00
Steven Allen
584907269a make sector-storage compile 2020-11-16 19:03:30 +01:00
Łukasz Magiera
6bea9dd178 Making sealing logic work with multiple seal proof types 2020-11-16 19:03:30 +01:00
Łukasz Magiera
408183c7a6 docsgen: Fix CI 2020-11-12 20:15:57 +01:00
Łukasz Magiera
8409e155d4 docsgen: Support miner/worker structs 2020-11-12 10:29:42 +01:00
Łukasz Magiera
8ac495723e gofmt 2020-11-11 17:42:23 +01:00
Łukasz Magiera
09f9f871a3 Create a command to abort sealing calls 2020-11-11 17:39:12 +01:00
Łukasz Magiera
bd71bf2379 Expand sched-diag 2020-11-11 16:47:44 +01:00
Łukasz Magiera
18aa97f962
Merge pull request #4779 from filecoin-project/feat/work-track-hostname 
storage: Track worker hostnames with work
 2020-11-10 20:38:44 +01:00
Łukasz Magiera
5caa277261 storage: Track abandoned work more correctly 2020-11-09 23:38:20 +01:00
Łukasz Magiera
f819e71d12 storage: Separate returned jobs in jobs cli 2020-11-09 23:13:29 +01:00
Łukasz Magiera
27a9dd3bbb storage: Track worker hostnames with work 2020-11-09 23:09:04 +01:00
Łukasz Magiera
5b8f555221 sched: Update after tasks finish preparing 2020-11-09 21:44:28 +01:00
Łukasz Magiera
fbc05fa5a8 stores: Report storage changes quickly 2020-11-09 21:43:52 +01:00
zgfzgf
5bcc6339b4 optimize code replace strings with constants 2020-11-09 16:21:16 +08:00
stirlingx
b5d2f01993
Update selector_task.go 2020-11-03 09:20:48 +08:00
Łukasz Magiera
094ea3fe97
Merge pull request #4666 from filecoin-project/fix/sched-issues 
Fix worker reenabling, handle multiple restarts in worker
 2020-10-30 21:34:07 +01:00
Łukasz Magiera
69e44ebf07 sched: re-register worker windows after re-enabling correctly 2020-10-30 18:32:59 +01:00
Łukasz Magiera
f90a387f96 sched: Print worker UUIDs in shed-diag correctly 2020-10-30 18:32:16 +01:00
Łukasz Magiera
774e2ecebf sched: Fix worker reenabling 2020-10-30 18:01:37 +01:00
Łukasz Magiera
7fbb868513 Debug flag to force running sealing scheduler 2020-10-30 11:07:35 +01:00
Łukasz Magiera
c6b03ce62b sectorstorage: Missing unlock in waitWork 2020-10-29 15:18:51 +01:00
Łukasz Magiera
ea5bb5cdab sectorstorage: Fix manager restart edge-case 2020-10-29 12:14:21 +01:00
Łukasz Magiera
f0f75e2d2c
Merge pull request #4627 from karalabe/fix-gpu-usage-tracking 
extern/sector-storage: fix GPU usage overwrite bug
 2020-10-29 10:13:54 +01:00
Péter Szilágyi
5f657b4333
extern/sector-storage: fix GPU usage overwrite bug 2020-10-28 20:52:33 +02:00
Łukasz Magiera
da7ecc1527 Fix flaky sealing manager tests 2020-10-28 16:15:17 +01:00
Łukasz Magiera
4100f6eead fix TestWDPostDoPost 2020-10-28 15:23:21 +01:00
Łukasz Magiera
ed2f81da2f sched: Fix tests 2020-10-28 14:34:28 +01:00
Łukasz Magiera
4cf00b8b42 worker_local: address review 2020-10-28 14:29:17 +01:00
Łukasz Magiera
96c5ff7e7f sched: use more letters for variables 2020-10-28 14:23:38 +01:00
Łukasz Magiera
8731fe9112 sched: split worker handling into more funcs 2020-10-28 14:14:50 +01:00
Łukasz Magiera
84b567c790 sched: move worker funcs to a separate file 2020-10-28 13:39:28 +01:00
Łukasz Magiera
660236b224 Merge remote-tracking branch 'origin/master' into feat/async-restartable-workers 2020-10-23 23:25:35 +02:00
Łukasz Magiera
29e334de54
Merge pull request #4511 from filecoin-project/steb/generalize-window-post 
Manage sectors by size instead of proof type.
 2020-10-22 21:27:48 +02:00
Steven Allen
4e730b5ec8 port to v2 imports 2020-10-21 12:16:23 -07:00
Steven Allen
00dcb1bce9 Manage sectors by size instead of proof type. 
* We may have multiple sectors with the same size and different proof types, but all these management functions stay the same.
* This simplifies PoSt logic.
 2020-10-20 18:30:56 -07:00
Łukasz Magiera
8c86ea6b75 localworker: Try very hard to get ruselts to manager 2020-10-18 19:45:11 +02:00
Łukasz Magiera
dbb421c4f7 localworker: Use better context for calling returnFunc 2020-10-18 19:32:43 +02:00
Łukasz Magiera
8d06cca073 sched: Handle workers using sessions instead of connections 2020-10-18 12:36:06 +02:00
Łukasz Magiera
71b3b9075d Merge remote-tracking branch 'origin/master' into feat/async-restartable-workers 2020-10-13 21:33:21 +02:00
Steven Allen
dc4e73c737 Test the tape upgrade 2020-10-12 00:01:25 -07:00
Steven Allen
83dfc460d4 fix race in unseal 
1. Remove an invalid error check.
2. Make sure to shadow the outer error type from within the goroutine instead or
reading the outer type.

This may have been causing test issues (caught in TestMinerAllInfo with the race
detector).
 2020-10-09 15:39:41 -07:00
Łukasz Magiera
0de3051821 Merge remote-tracking branch 'origin/master' into feat/async-restartable-workers 2020-10-08 13:10:41 +02:00
Łukasz Magiera
cfd126ee9f
Merge pull request #3961 from filecoin-project/docs/miner-storage 
lotus-miner: add more help text to storage / attach
 2020-10-07 14:55:42 +02:00
Łukasz Magiera
1fc23fb466 lotus-miner: Cleanup storage attach helptext a bit 2020-10-03 11:30:22 +02:00
Łukasz Magiera
5932f28519 Merge remote-tracking branch 'origin/master' into feat/async-restartable-workers 2020-10-01 02:39:48 +02:00
Łukasz Magiera
6981f776f4 Lower PC2 memory requirements 2020-10-01 00:54:53 +02:00
Łukasz Magiera
1b7cdb9341 Fix storage manager tests 2020-10-01 00:54:34 +02:00
Łukasz Magiera
5e08d56630 sched: Allow some single-thread tasks to run in parallel with PC2/C2 2020-10-01 00:28:44 +02:00
Łukasz Magiera
79d2ddf24f Review 2020-09-30 21:18:12 +02:00
Łukasz Magiera
2cfe22d4e5 Merge remote-tracking branch 'origin/master' into feat/async-restartable-workers 2020-09-30 20:48:16 +02:00
Łukasz Magiera
e3ee4e4718 Fix lint errors 2020-09-30 20:24:03 +02:00
Łukasz Magiera
2d16af6ee6 sectorstorage: Fix TestRedoPC1 2020-09-30 19:18:38 +02:00
Łukasz Magiera
c228598098 sectorstorage: Variable scopes are really hard 2020-09-30 18:16:07 +02:00
Łukasz Magiera
54fdd6ba5a sectorstorage: Variable scopes are hard 2020-09-30 17:48:48 +02:00
Łukasz Magiera
6855284d88 sectorstorage: Cancel non-running work in case of abort in sched 2020-09-30 17:26:09 +02:00
Łukasz Magiera
bc85e3ce50
Merge pull request #4107 from shaodan/worker-no-swap 
Add --no-swap flag for worker
 2020-09-30 09:25:57 +02:00
Dan Shao
1affd498c1 Add --no-swap flag for worker 2020-09-30 14:23:35 +08:00
Łukasz Magiera
baef3c8dd2 sectorstorage: Fix potential panic in FinalizeSector 2020-09-29 15:22:46 +02:00
Łukasz Magiera
0f2dcf28b1 fsm: Reuse tickets in PC1 on retry 2020-09-29 10:07:49 +02:00
Łukasz Magiera
1e6a69f8aa localworker: Don't mark calls as returned when returning fails 2020-09-28 22:10:02 +02:00
Łukasz Magiera
9bd2537971 stores: Fix error printing in http handler 2020-09-28 22:06:03 +02:00
Łukasz Magiera
810c767200 worker: Redeclare storage on reconnect 2020-09-28 21:06:49 +02:00
Łukasz Magiera
4ba7af6061 worker: Mark return methods as retry-safe 2020-09-28 20:46:44 +02:00
Łukasz Magiera
9e7d6823b1 sectorstorage: Cleanup callToWork mapping after work is done 2020-09-28 13:34:45 +02:00
zgfzgf
1a7aea1906 modify error 2020-09-25 22:59:21 +08:00
zgfzgf
3207bc4704 optimize trySched 2020-09-25 22:41:29 +08:00
zgfzgf
60e950015c modify for unsafe 2020-09-25 22:13:27 +08:00
Łukasz Magiera
04ee53e061 sectorstorage: Show task type of ret-wait jobs 2020-09-24 11:55:11 +02:00
Łukasz Magiera
d817dceb05 Show lost calls in sealing jobs cli 2020-09-23 19:26:35 +02:00
Łukasz Magiera
c17f0d7e61 sectorstorage: Fix panic in returnResult 2020-09-23 17:37:05 +02:00
Łukasz Magiera
86c222ab58 sectorstorage: fix work tracking 2020-09-23 14:56:50 +02:00
Łukasz Magiera
ce6b92484f Merge remote-tracking branch 'origin/master' into feat/async-restartable-workers 2020-09-23 11:31:21 +02:00
Łukasz Magiera
6185e157e9 sectorstorage: calltracker: work around cbor-gen bytearray len limit 2020-09-23 00:29:10 +02:00
Łukasz Magiera
04ad1791b0 localworker: Fix contexts 2020-09-23 00:10:36 +02:00