Commit Graph

1230 Commits

Author SHA1 Message Date
Aarsh Shah
9ee940098d Merge remote-tracking branch 'origin/master' into feat/cid-to-piece-idx 2022-01-20 16:21:48 +04:00
vyzo
b360c9403f make gen and friends 2022-01-20 11:44:01 +02:00
vyzo
39bf59d372 add examples to docgen 2022-01-20 11:36:11 +02:00
vyzo
554848fe6b add resource manager Net(Set)Limit api 2022-01-20 11:36:11 +02:00
vyzo
d05d5bcb70 add resource manager NetStat api 2022-01-20 11:36:11 +02:00
Aarsh Shah
8c8652e7bb add a new cli 2022-01-12 18:06:48 +04:00
Aarsh Shah
edfc2c215c Merge remote-tracking branch 'origin/main' into feat/cid-to-piece-idx 2022-01-12 12:48:54 +04:00
Łukasz Magiera
960759d22b address review 2022-01-11 17:31:27 +01:00
zenground0
33f2d24f54 Snap Deals Integration
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-10 15:39:38 +05:30
zl
4172a3c8b7 ExampleValue for a silce is nil 2022-01-04 14:27:10 +08:00
hannahhoward
cddf63efe9 feat(storageminer): add api for transfer diagnostics
Add API + CLI for inspecting in depth diagnostics on graphsync transfers with a given peer
2021-12-22 13:41:29 -08:00
Dirk McCormick
b1734f84b3 feat: retrieval ask CLI command 2021-12-17 15:55:12 +01:00
Aayush Rajasekaran
80d5e52923 Merge branch 'master' into next 2021-12-13 13:24:28 -05:00
zenground0
a5be80828a RemoveData and Decode
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
2021-12-03 15:21:06 -05:00
Łukasz Magiera
727765b248 Command to list active sector locks 2021-12-03 12:33:23 +01:00
Łukasz Magiera
6d52d8552b Fix docsgen 2021-11-30 02:06:58 +01:00
Łukasz Magiera
c9a2ff4007 cleanup worker resource overrides 2021-11-30 02:06:58 +01:00
Clint Armstrong
4ef8543128 Permit workers to override resource table
In an environment with heterogenious worker nodes, a universal resource
table for all workers does not allow effective scheduling of tasks. Some
workers may have different proof cache settings, changing the required
memory for different tasks. Some workers may have a different count of
CPUs per core-complex, changing the max parallelism of PC1.

This change allows workers to customize these parameters with
environment variables. A worker could set the environment variable
PC1_MIN_MEMORY for example to customize the minimum memory requirement
for PC1 tasks. If no environment variables are specified, the resource
table on the miner is used, except for PC1 parallelism.

If PC1_MAX_PARALLELISM is not specified, and
FIL_PROOFS_USE_MULTICORE_SDR is set, PC1_MAX_PARALLELSIM will
automatically be set to FIL_PROOFS_MULTICORE_SDR_PRODUCERS + 1.
2021-11-30 02:06:58 +01:00
Clint Armstrong
93e4656a27 Use a float to represent GPU utilization
Before this change workers can only be allocated one GPU task,
regardless of how much of the GPU resources that task uses, or how many
GPUs are in the system.

This makes GPUUtilization a float which can represent that a task needs
a portion, or multiple GPUs. GPUs are accounted for like RAM and CPUs so
that workers with more GPUs can be allocated more tasks.

A known issue is that PC2 cannot use multiple GPUs. And even if the
worker has multiple GPUs and is allocated multiple PC2 tasks, those
tasks will only run on the first GPU.

This could result in unexpected behavior when a worker with multiple
GPUs is assigned multiple PC2 tasks. But this should not suprise any
existing users who upgrade, as any existing users who run workers with
multiple GPUs should already know this and be running a worker per GPU
for PC2. But now those users have the freedom to customize the GPU
utilization of PC2 to be less than one and effectively run multiple PC2
processes in a single worker.

C2 is capable of utilizing multiple GPUs, and now workers can be
customized for C2 accordingly.
2021-11-30 02:06:58 +01:00
Clint Armstrong
c4f46171ae Report memory used and swap used in worker res
Attempting to report "memory used by other processes" in the MemReserved
field fails to take into account the fact that the system's memory used
includes memory used by ongoing tasks.

To properly account for this, worker should report the memory and swap
used, then the scheduler that is aware of the memory requirements for a
task can determine if there is sufficient memory available for a task.
2021-11-30 02:06:58 +01:00
Łukasz Magiera
320b36495d gofmt 2021-11-29 21:39:27 +01:00
Łukasz Magiera
d019853687
review: Cleanup some comments
Co-authored-by: Peter Rabbitson <ribasushi@protocol.ai>
2021-11-29 21:29:00 +01:00
Łukasz Magiera
5b5e6b9e44 retrieval: DagSpec.MatchPath -> ExportMerkleProof 2021-11-29 21:14:00 +01:00
Łukasz Magiera
61791b90ea retrieval: Only output matching nodes, MatchPath dagspec 2021-11-29 20:40:55 +01:00
zenground0
7d2b3f05db WIP sector storage and integration test 2021-11-29 10:24:00 -05:00
Peter Rabbitson
d5b4d93575 Cleanup partial retrieval codepaths ( zero functional changes )
- Adjust the ExportRef comment to reflect change from #7673
- Simplify control flow in parseDagSpec() ( read diff under -w )
2021-11-25 10:35:05 +01:00
Łukasz Magiera
407c2ed114 retrieval: Drop the RootSelector hack 2021-11-23 17:42:43 +01:00
Łukasz Magiera
bdac11ade7 retrieval: Update some API comments 2021-11-22 14:19:03 +01:00
Łukasz Magiera
763659b8a3 retrieval: Update lotus-soup 2021-11-22 12:46:29 +01:00
Łukasz Magiera
ec2bfb99bb make gen 2021-11-22 12:46:25 +01:00
Łukasz Magiera
b83a9b902a gofmt 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b26906963b retrieval: Support multi-root export 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b0c043cc2f make gen 2021-11-22 12:46:15 +01:00
Łukasz Magiera
b868769ec8 more retrieval api work 2021-11-22 12:46:02 +01:00
Łukasz Magiera
89138bab4d Simplify retrieval APIs 2021-11-22 12:45:42 +01:00
Travis Person
17cb5117f4 add additional methods to lotus gateway 2021-11-22 12:19:27 +01:00
Aayush Rajasekaran
32fc03886d CLI: Add a lotus multisig cancel command 2021-11-18 19:21:17 -05:00
Anton Evangelatov
5454aebf13
CLI tools for the DAGStore inverted index (#7361)
* cli commands for dagstore

* address comments from Dirk

* rename inverted index to piece index
2021-11-17 15:39:45 +02:00
Dirk McCormick
501308239f feat: CLI command to announce deal to indexers 2021-11-17 12:16:22 +01:00
Jennifer Wang
724dfc6a4d tidy 2021-10-26 20:10:18 -04:00
Anton Evangelatov
6e5ccc87cf cli: add retry for deals stuck in Publish with no funds 2021-10-11 14:03:55 +02:00
Peter Rabbitson
c4a7de9d37 Remove dead example code + dep 2021-10-07 09:44:50 +02:00
Peter Rabbitson
2a1094fc17 Merge banch feat/go-fil-markets-v1.13.0 into feat/datamodel-selector-retrieval 2021-10-04 21:56:11 +02:00
Aayush Rajasekaran
1d6db306ec Randomness: Move getters from ChainAPI to StateAPI 2021-09-15 00:03:13 +02:00
Peter Rabbitson
0444435589 Expose basic text-based datamodel selector on retrieval
Syntaxt of selection is located at
https://pkg.go.dev/github.com/ipld/go-ipld-selector-text-lite#SelectorSpecFromPath

Example use, assuming that:
  - The root of the deal is a plain dag-pb unixfs directory
  - The directory is not sharded
  - The user wants to retrieve the first entry in that directory

  lotus client retrieve --miner f0XXXXX --datamodel-path-selector 'Links/0/Hash' bafyROOTCID ~/output

For a much more elaborate example see the top of ./itests/deals_partial_retrieval_test.go
2021-09-10 09:44:11 +02:00
Łukasz Magiera
e4044151f0 Show deal sizes is sealing sectors 2021-09-07 18:49:53 +02:00
hannahhoward
91804d5746 feat(deps): update go-graphsync v0.9.0 2021-09-03 16:22:51 +02:00
Steven Allen
1cf556c3a2 feat: expose ChainGetPath on the gateway 2021-08-30 16:43:21 -07:00
Łukasz Magiera
e2ba650a8c cli for checking alerts 2021-08-26 15:45:17 +02:00
Łukasz Magiera
5366821144 api for getting alert states 2021-08-26 15:44:45 +02:00