Commit Graph

14939 Commits

Author SHA1 Message Date
Łukasz Magiera
b4bdb89f65 CARv2 v2.1.0 2021-12-08 01:52:00 -05:00
Łukasz Magiera
ea9130256c dagstore pieceReader: Fix wrong ErrUnexpectedEOF return in ReadAt 2021-12-08 01:51:56 -05:00
Łukasz Magiera
cde687bbe1 dagstore pieceReader: Always read full in ReadAt 2021-12-08 01:51:47 -05:00
Łukasz Magiera
9725d72a80 Add metrics to dagstore piecereader 2021-12-08 01:51:43 -05:00
Travis Person
fbd3c90b80 disable building of appimage on release
appimage testing is currently broken due to a requirement to have access
to /dev/snd, which is not present on the image we are using in circleci.
2021-12-08 01:02:17 -05:00
Łukasz Magiera
49c619d65d
Merge pull request #7735 from filecoin-project/feat/storlocks-cmd
Command to list active sector locks
2021-12-08 00:11:17 +01:00
Aayush Rajasekaran
b0a9a27288 Shed: Allow send-csv to specify params and method 2021-12-07 17:14:29 -05:00
Aayush Rajasekaran
c9a557e0d9 Address review 2021-12-07 17:09:07 -05:00
Aayush Rajasekaran
fcb7cd0eb0 Shed: Add a util to send a batch of messages 2021-12-07 17:09:07 -05:00
Łukasz Magiera
9ccd4ee240
Merge pull request #7737 from filecoin-project/feat/opt-ds-pr
dagstore pieceReader: Always read full in ReadAt
2021-12-06 23:14:21 +01:00
Łukasz Magiera
84c48de5d9 CARv2 v2.1.0 2021-12-06 15:39:03 +01:00
Łukasz Magiera
9f6265e0b4 dagstore pieceReader: Fix wrong ErrUnexpectedEOF return in ReadAt 2021-12-03 23:36:36 +01:00
zenground0
a5be80828a RemoveData and Decode
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
2021-12-03 15:21:06 -05:00
Łukasz Magiera
3969d6b767 dagstore pieceReader: Always read full in ReadAt 2021-12-03 20:14:35 +01:00
Łukasz Magiera
dad9190142 Add metrics to dagstore piecereader 2021-12-03 17:07:14 +01:00
Łukasz Magiera
9d143426ee
Merge pull request #7734 from filecoin-project/feat/splitstore-upgrade-protection
SplitStore: supress compaction near upgrades
2021-12-03 15:18:07 +01:00
Łukasz Magiera
727765b248 Command to list active sector locks 2021-12-03 12:33:23 +01:00
vyzo
489782e21b satisfy the spellchecker that masquarades as a linter 2021-12-03 12:15:28 +02:00
vyzo
5d6398f20e hook the upgrade schedule to splitstore start 2021-12-03 12:11:54 +02:00
vyzo
6ce5879071 add unit test for compaction supression 2021-12-03 12:05:15 +02:00
vyzo
05c1325885 add logic for supressing compaction near upgrade boundaries 2021-12-03 11:50:35 +02:00
Łukasz Magiera
9331714089
Merge pull request #7707 from filecoin-project/fix/app-image-build
Disable appimage
2021-12-02 14:39:48 +01:00
Łukasz Magiera
d5e1cc095c
Merge pull request #7725 from filecoin-project/feat/open-telemetry
Initial switch to OpenTelemetry
2021-12-02 14:39:20 +01:00
Jiaying Wang
c2437b7bf2
Create pull_request_template.md
This is the very first iteration of the lotus PR template. The goal of adding PR template is to standardize PR requests and encourage contributors to:
- come up with good PR descriptions to give code reviewers a clear overview of what's in the PR 
- have a clear PR title as lotus generates a change log based on it
- check that tests and documentation for the codes that changed are icnluded 

The PR type follows the https://www.conventionalcommits.org/en/v1.0.0-beta.2/.  The [contribution guideline](https://github.com/filecoin-project/lotus#contribute) should be updated with how to create a pr after the template is accepted.
2021-12-01 22:37:22 -05:00
hannahhoward
bfd1099ef3 fix(main): fix bug in startup when tracing is not present 2021-12-01 18:53:00 -08:00
hannahhoward
f3b691d618 feat(tracing): switch to OpenTelemetry
Switch the underlying tracing library to OpenTelemetry, with a bridge to OpenCensus for
compatibility
2021-12-01 16:19:47 -08:00
Travis Person
66c8c66f40 disable building of appimage on release
appimage testing is currently broken due to a requirement to have access
to /dev/snd, which is not present on the image we are using in circleci.
2021-12-01 19:15:35 +00:00
Jiaying Wang
52c7fa2427
Merge pull request #7718 from filecoin-project/jen/new-prep
v1.13.2-rc1
2021-11-30 19:23:09 -05:00
Jiaying Wang
0cfc344f82
Merge pull request #7717 from filecoin-project/jen/bumpmaster
Bump the master version to v1.13.3-dev
2021-11-30 19:06:46 -05:00
Jennifer Wang
02fa0df81c update change log 2021-11-30 19:06:11 -05:00
Jennifer Wang
027017d0d2 update the version to v1.13.2-rc1 2021-11-30 19:03:24 -05:00
Jennifer Wang
c3ecf8355d docs gen 2021-11-30 18:29:13 -05:00
Jennifer Wang
a09aa0aa72 Bump the master version to v1.13.3-dev 2021-11-30 18:24:34 -05:00
Łukasz Magiera
26c9120d22
Merge pull request #7714 from filecoin-project/feat/sched-review
Address Scheduler enhancements (#7703) review
2021-11-30 23:03:03 +01:00
Łukasz Magiera
71329f6c41 Address Scheduler enhancements (#7703) review 2021-11-30 20:50:40 +01:00
ZenGround0
91f6d4b013
Merge pull request #7615 from filecoin-project/feat/snap-deals-storage
Feat/snap deals storage
2021-11-30 14:18:24 -05:00
zenground0
40d16a8f88 Review Response 2021-11-30 13:53:37 -05:00
Łukasz Magiera
73f16f08e3
Merge pull request #7703 from filecoin-project/feat/scheduler-enhancements
Scheduler enhancements
2021-11-30 19:14:25 +01:00
zenground0
f88fcdbcfc WIP 2021-11-30 12:40:14 -05:00
Łukasz Magiera
330cfc33ee worker: Typo in resources cmd usage
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
2021-11-30 02:06:58 +01:00
Łukasz Magiera
cf20b0b2b8 worker: Command to print resource-table env vars 2021-11-30 02:06:58 +01:00
Łukasz Magiera
001ecbb561 fix lint 2021-11-30 02:06:58 +01:00
Łukasz Magiera
a597b072b8 fix sched tests 2021-11-30 02:06:58 +01:00
Łukasz Magiera
f25efecb74 worker: Test resource table overrides 2021-11-30 02:06:58 +01:00
Łukasz Magiera
6d52d8552b Fix docsgen 2021-11-30 02:06:58 +01:00
Łukasz Magiera
c9a2ff4007 cleanup worker resource overrides 2021-11-30 02:06:58 +01:00
Łukasz Magiera
b961e1aab5 sched resources: Separate Parallelism defaults depending on GPU presence 2021-11-30 02:06:58 +01:00
Łukasz Magiera
36868a8749 sched: C2 is not all-core load 2021-11-30 02:06:58 +01:00
Clint Armstrong
4ef8543128 Permit workers to override resource table
In an environment with heterogenious worker nodes, a universal resource
table for all workers does not allow effective scheduling of tasks. Some
workers may have different proof cache settings, changing the required
memory for different tasks. Some workers may have a different count of
CPUs per core-complex, changing the max parallelism of PC1.

This change allows workers to customize these parameters with
environment variables. A worker could set the environment variable
PC1_MIN_MEMORY for example to customize the minimum memory requirement
for PC1 tasks. If no environment variables are specified, the resource
table on the miner is used, except for PC1 parallelism.

If PC1_MAX_PARALLELISM is not specified, and
FIL_PROOFS_USE_MULTICORE_SDR is set, PC1_MAX_PARALLELSIM will
automatically be set to FIL_PROOFS_MULTICORE_SDR_PRODUCERS + 1.
2021-11-30 02:06:58 +01:00
Clint Armstrong
93e4656a27 Use a float to represent GPU utilization
Before this change workers can only be allocated one GPU task,
regardless of how much of the GPU resources that task uses, or how many
GPUs are in the system.

This makes GPUUtilization a float which can represent that a task needs
a portion, or multiple GPUs. GPUs are accounted for like RAM and CPUs so
that workers with more GPUs can be allocated more tasks.

A known issue is that PC2 cannot use multiple GPUs. And even if the
worker has multiple GPUs and is allocated multiple PC2 tasks, those
tasks will only run on the first GPU.

This could result in unexpected behavior when a worker with multiple
GPUs is assigned multiple PC2 tasks. But this should not suprise any
existing users who upgrade, as any existing users who run workers with
multiple GPUs should already know this and be running a worker per GPU
for PC2. But now those users have the freedom to customize the GPU
utilization of PC2 to be less than one and effectively run multiple PC2
processes in a single worker.

C2 is capable of utilizing multiple GPUs, and now workers can be
customized for C2 accordingly.
2021-11-30 02:06:58 +01:00