Łukasz Magiera
587c3fde58
piecereader: Fix parallel buffer access, close readers in readInto
2023-05-23 12:26:13 +02:00
Łukasz Magiera
cd75ea0fe4
piecereader: Retune to allow parallel ReadAt calls
2023-05-23 12:26:13 +02:00
Łukasz Magiera
8b2ef40f4e
storage: Weave end offset into unsealed reader code
2023-05-23 12:26:13 +02:00
Łukasz Magiera
6fd93ed170
fix: sched: Address GET_32G_MAX_CONCURRENT regression ( #10850 )
...
* Fix 1.21 regression: GET_32G_MAX_CONCURRENT + mixed prepared/executing leads to stuck scheduler
If you have 12 GET tasks and GET_32G_MAX_CONCURRENT=1, sealing jobs will only show assigned tasks for GET of the miner
and is stuck.
I believe this to be a regression of 1.21 unifying the counters, in the case of GETs where PrepType and TaskType
both being seal/v0/fetch leading to a state where tasks are blocked since already counted towards the limit.
* itests: Repro issue from PR #10633
* make counters int (non-working)
* fix: worker sched: Send taskDone notifs after tasks are done
* itests: Make TestPledgeMaxConcurrentGet actually reproduce the issue
* make the linter happy
---------
Co-authored-by: Steffen Butzer <steffen.butzer@outlook.com>
2023-05-10 15:43:42 -04:00
Łukasz Magiera
2f469c2f58
fix: tests: Use mutex-wrapped datastore in storage tests
2023-05-09 18:06:41 +02:00
Shrenuj Bansal
78800a4e77
fix: sealing: Make lotus-worker report GPU usage to miner during ReplicaUpdate task ( #10806 )
...
* Make lotus-worker report GPU usage to miner during ReplicaUpdate task
* make gen
---------
Co-authored-by: Jacob Crowther <jcrowthe@users.noreply.github.com>
2023-05-03 12:28:03 -04:00
Łukasz Magiera
9d2d53b58e
fix: prover: Propagate skipped sectors in local PoSt
2023-04-27 09:26:12 +02:00
Łukasz Magiera
aeee542fb9
Merge pull request #10696 from filecoin-project/fix/nil-terminate-wait
...
fix: sealing pipeline: Allow nil message in TerminateWait
2023-04-25 19:28:52 +02:00
Łukasz Magiera
7e34b49b5d
Merge pull request #10661 from filecoin-project/fix/cleanup-temp-failed-fetch
...
fix: storage: Remove temp fetching files after failed fetch
2023-04-25 19:28:33 +02:00
Aayush
5a061e4677
Merge branch 'releases' into asr/merge-release-into-master
2023-04-23 14:58:50 -04:00
Łukasz Magiera
b44ae9a7d8
Merge pull request #10647 from filecoin-project/sbansal/split-pcb
...
feat: sealing: Split PCA/PCB batches if gas used exceeds block limit
2023-04-21 18:30:27 +02:00
sirius
07416e8c0c
update: builtin to builtin11 in wdpost worker
2023-04-21 12:19:06 +08:00
Shrenuj Bansal
d1f3380850
change comment
2023-04-20 12:15:51 -04:00
Shrenuj Bansal
0c83781a7f
Add tests for PCB/PCA batch splitting
2023-04-19 18:44:32 -04:00
ZenGround0
327760acff
chore: build: Merge/v22 into 21 for 23 ( #10702 )
...
* chore: update ffi to increase execution parallelism
* Don't enforce walking receipt tree during compaction
* fix: build: drop drand incentinet servers
* chore: release lotus v1.20.4
* Apply suggestions from code review
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
* feat: Introduce nv19 skeleton
Update to go-state-types v0.11.0-alpha-1
Introduce dummy v11 actor bundles
Make new actors adapters
Add upgrade to Upgrade Schedules
make jen
Update to go-state-types v0.11.0-alpha-2
* feat: vm: switch to the new exec trace format (#10372 )
This is now "FVM" native. Changes include:
1. Don't treat "trace" messages like off-chain messages. E.g., don't
include CIDs, versions, etc.
2. Include IPLD codecs where applicable.
3. Remove fields that aren't filled by the FVM (timing, some errors,
code locations, etc.).
* feat: implement FIP-0061
* Address review
* Add and test the FIP-0061 migration
* Update actors bundles to fip/20230406
* Update to go-state-types master
* Update to actors v11.0.0-rc1
* - Update go state types
- Keep current expiration defaults on creation, extension some tests
- Update ffi
* ffi experiment
* Integration nv19 migration
- Open splitstore in migration shed tool
- Update state root version
* Post rebase fixup
* Fix
* gen
* nv19 invariant checking
* Try fixig blockstore so bundle is loaded
* Debug
* Fix
* Make butterfly upgrades happen
* Another ffi experiment
* Fix copy paste error
* Actually schedule migration (#10656 )
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
* Butterfly artifacts
* Set calibration net upgrade height
* Review Response
* Fix state tree version assert
* Quick butterfly upgrade to sanity check (#10660 )
* Quick butterfly upgrade to sanity check
* Update butterfly artifacts
* Revert fake fix
* Give butterfly net correct genesis
* Butterfly artifacts
* Give time before upgrade
---------
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
* chore:releasepolish v1.22 release (#10666 )
* Update butterfly artifacts
* register actors v11
* Update calibration upgrade time
* State inspection shed cmds
* Fix
* make gen
* Fix swallowed errors
* Lint fixup
---------
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
* v1.22.0-rc3
* bundle fix
* Feat/expedite nv19 (#10681 )
* Update go-state-types
* Modify upgrade schedule and params
* Revert fip 0052
* Update gst
* docsgen
* fast butterfly migration to validate migration
* Correct epoch to match specified date
* Update actors v11
* Update changelog build version
* Update butterfly artifacts
* Fix lotus-miner init to work after upgrade
---------
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
* fix:deps:stable ffi for stable release (#10698 )
* Point to stable ffi for stable lotus release
* go mod tidy
---------
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
* Update CHANGELOG.md
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
---------
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: jennijuju <jiayingw703@gmail.com>
2023-04-19 18:40:18 -04:00
Łukasz Magiera
4bdb6b34b3
fix: sealing pipeline: Allow nil message in TerminateWait
2023-04-19 15:43:50 +02:00
Shrenuj Bansal
79826447f5
fix unit and integration test breaks
2023-04-12 21:45:43 -04:00
Łukasz Magiera
7a4f69721c
fix: storage: Remove temp fetching files after failed fetch
2023-04-12 17:00:10 -07:00
Shrenuj Bansal
4eb4af639a
Split PCA msg into smaller batches
2023-04-12 00:30:19 -04:00
Shrenuj Bansal
6f91dc7c5b
populate result error on exit conditions
2023-04-10 16:24:13 -04:00
Shrenuj Bansal
8893c62a42
make gen
2023-04-10 15:39:42 -04:00
Shrenuj Bansal
1e25d7b453
Split precommit batches if gas used exceeds block limit
2023-04-10 14:52:39 -04:00
Phi
9fd69377df
fix: unseal: check if sealed sector exists
...
Check if sealed or update sector exists when `SectorsUnsealPiece` is called.
2023-04-10 09:53:19 +02:00
Shrenuj Bansal
2278a209e2
Add feature to stagger sector prove commit submission ( #10543 )
...
* Add feature to stagger sector prove commit submission
* make gen and docsgen as usual
* address comments and lint
* Update comment
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
* make gen for stupid comment
* make docsgen
* address comments
---------
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2023-04-01 19:30:32 -04:00
Łukasz Magiera
3af5ef1167
Merge pull request #10588 from filecoin-project/fix/file-does-not-exsist-logs
...
fix: log: Stop logging `file does not exists`
2023-03-30 20:24:59 +02:00
Aayush
66fc6dc3e5
refactor: stop using deprecated io/ioutil
2023-03-29 15:40:49 -04:00
Aayush
a724a713da
refactor: stop using deprecated io/ioutil
2023-03-29 15:38:46 -04:00
Phi
a99946937c
fix: log: Stop logging file does not exists
...
Stop logging `file does not exists` errors when retrieving disk usage information.
2023-03-29 15:32:32 +02:00
Łukasz Magiera
7e414ac390
Merge remote-tracking branch 'origin/master' into feat/go-data-transfer-v2
2023-03-15 11:37:53 +01:00
Steven Allen
dcb49dc8ee
refactor: update cache to the new generic version ( #10463 )
...
- Adds type safety.
- Reduces allocations.
- Fixes the drand cache (was storing by value, but retrieving by pointer)
2023-03-13 15:29:09 -07:00
Łukasz Magiera
80ccd14447
Merge pull request #10356 from filecoin-project/feat/assigner-experiments
...
feat: sched: Assigner experiments
2023-03-09 01:28:31 +01:00
Łukasz Magiera
52ae215104
Merge pull request #10363 from filecoin-project/fix/fsm-shutdown-removed-sectors
...
fix: fsm: shutdown removed sectors FSMs
2023-03-08 19:32:47 +01:00
Łukasz Magiera
d114d8fe7e
deprecate MarketListRetrievalDeals, make gen
2023-03-08 17:15:16 +01:00
Łukasz Magiera
b0ebdb6882
make gen
2023-03-06 15:07:49 +01:00
Łukasz Magiera
9a295e58b0
post worker sched: Retry on alternative worker on RPC errors
2023-03-06 14:30:58 +01:00
Łukasz Magiera
e170487faf
post worker sched: Filter out disabled workers correctly
2023-03-06 13:56:23 +01:00
Łukasz Magiera
c484c38735
worker sched: Separate resource def for preparing window
2023-02-28 10:52:50 +01:00
Łukasz Magiera
04fe9062a6
fix: fsm: shutdown removed sectors FSMs
2023-02-28 09:48:17 +01:00
Łukasz Magiera
2316363f7a
sched: Share active/preparing task counters
2023-02-28 09:08:49 +01:00
Łukasz Magiera
745476c9ab
feat: sched: Assigner experiments
2023-02-27 17:54:25 +01:00
Łukasz Magiera
a0771b12eb
Merge pull request #9878 from filecoin-project/gstuart/check-allocation-expiry
...
feat: Check for allocation expiry when waiting to seal sectors
2023-01-27 13:05:09 +01:00
Łukasz Magiera
fc6721cff7
feat: wdpost: Emit more detailed errors
2023-01-26 11:57:16 +01:00
Adin Schmahmann
d35818293a
fix: stub out the FileSize command so lotus libraries can build on Windows
2023-01-17 01:03:10 -05:00
Geoff Stuart
787470c2b4
Remove workaround that is no longer needed
2023-01-12 12:39:57 -05:00
Geoff Stuart
12a06defd1
Check sectors for allocation expiry before committing
2023-01-11 16:24:34 -05:00
Phi
bd28a4bca4
Add snap-sector types
...
Add snap-sector types
2022-12-15 14:29:58 +01:00
Phi
87e384e7fd
Update svg-image
...
We do not depend on specs-storage anymore, this has been moved into lotus (storiface)
2022-12-15 10:49:58 +01:00
Phi
a8b406e7b7
Update svg-file
...
Update svg-file and headings
2022-12-15 10:15:07 +01:00
Geoff Stuart
ae4010ffb6
Drop actor keys from chain/actors/manifest.go
2022-12-13 18:02:34 -05:00
Geoff Stuart
f95e6602a0
Check for nil returns from StateSectorGetInfo
2022-12-12 10:44:08 -05:00
Łukasz Magiera
f0bf3408bc
sealing: Add a guard check to getExpirationCached when sector not found
2022-12-09 09:56:11 +01:00
Łukasz Magiera
2fd0cb4aff
fix: sealing: Avoid nil dereference in debug log
2022-12-09 09:54:28 +01:00
Łukasz Magiera
7b5e8d1e96
Merge pull request #9748 from filecoin-project/feat/metrics-tag-storage-use
...
feat: metrics: Tag storage path types
2022-11-29 18:06:23 +01:00
Ian Davis
0a4efd4820
Merge branch 'main' into id/update-go-version
2022-11-29 14:29:11 +00:00
Łukasz Magiera
57c31a79e7
feat: metrics: Tag storage path types
2022-11-29 13:06:56 +01:00
Łukasz Magiera
1597e85906
sched: Address review
2022-11-29 11:46:51 +01:00
Łukasz Magiera
5a458a60c2
fix sched_test
2022-11-29 11:44:57 +01:00
Łukasz Magiera
5881edb75e
feat: sched: Cache worker calls
2022-11-29 11:44:57 +01:00
Łukasz Magiera
d82b2a5804
Merge pull request #9738 from filecoin-project/feat/sched-cycle-metrics
...
feat: sched: Add metrics around sched cycle
2022-11-28 23:57:58 +01:00
Łukasz Magiera
285cc66773
Merge pull request #9648 from filecoin-project/fix/snap-abort-cleanup
...
fix: sealing: More complete snapdeals abort cleanup
2022-11-28 23:26:32 +01:00
Łukasz Magiera
8ba4355cab
Merge pull request #9694 from filecoin-project/9159-allow-lotus-worker-to-unseal-sector-even-if-does-not-store-unsealed-data-long-term
...
fix: sealing: Set all path types for Unseal pipeline to sealing storage
2022-11-28 20:15:17 +01:00
Łukasz Magiera
c0fce07889
sched: Metrics around scheduldng workload
2022-11-28 19:51:18 +01:00
Łukasz Magiera
bc879c5bd8
feat: sched: Add metrics around sched cycle
2022-11-28 19:07:35 +01:00
Ian Davis
9f85d3dca7
Address simple linter issues
2022-11-24 16:32:27 +00:00
Shrenuj Bansal
fa596eee5b
Move unsealed sector from sealing to long term storage after Unseal
2022-11-23 15:22:14 -05:00
Aayush
355a08cd04
fix: sealer: Tweak confidence for handleUpdateActivating
2022-11-23 14:38:57 -05:00
Łukasz Magiera
ea9a8303c8
sealing: tasks: TTReleaseUnsealed -> TTFinalizeUnsealed
2022-11-23 18:57:17 +01:00
Łukasz Magiera
d6f73c1a81
sealing: Address snap cleanup review
2022-11-23 18:57:17 +01:00
Łukasz Magiera
dc6c2b68fe
sealing: Fix tests after finalize changes
2022-11-23 18:57:16 +01:00
Łukasz Magiera
87e5549af5
sealing: Split unsealed cleanup from Finalize
2022-11-23 18:57:16 +01:00
Łukasz Magiera
0bba2bd1ba
sealing: Use FinalizeSector in snap abort for cleanup
2022-11-23 18:57:16 +01:00
Łukasz Magiera
211712bf6d
itests: Test snapdeals abort cleanup
2022-11-23 18:57:16 +01:00
Shrenuj Bansal
e4ce056ac6
Set all path types for Unseal pipeline to sealing storage
2022-11-21 14:40:50 -05:00
Łukasz Magiera
71a84bb092
feat: wdpost: Configurable pre-check timeouts
2022-11-17 18:25:30 +01:00
Łukasz Magiera
0e07fa94d0
Merge pull request #9613 from filecoin-project/feat/wdpost-robustness
...
feat: storage: Force exit GenerateSingleVanillaProof on cancelled context
2022-11-15 10:30:25 +01:00
Łukasz Magiera
e476c1ab17
storage: paths: Fix typo in proof error message
...
Co-authored-by: Piotr Bochenek <pbochenek@worldremit.com>
2022-11-14 21:04:18 +01:00
Łukasz Magiera
51e92c97c3
Merge pull request #9642 from filecoin-project/fix/snapsector-clear-ctime
...
fix: sealing pipeline: Clear CreationTime when starting sector upgrade
2022-11-14 20:52:58 +01:00
Łukasz Magiera
c79085e5b7
Merge pull request #9598 from filecoin-project/fix/ancient-snap-deals-terrible
...
fix:sealing-fsm:wait mutable fsm state for immutable sector upgrade error
2022-11-14 19:53:37 +01:00
Łukasz Magiera
efe2d8da8a
sealing pipeline: Test clearing CreationTime
2022-11-14 18:46:50 +01:00
Łukasz Magiera
d8c1b67a22
fix: sealing pipeline: Clear CreationTime when starting sector upgrade
2022-11-14 18:36:26 +01:00
zenground0
087da50ecc
Lint
2022-11-14 10:16:37 -05:00
zenground0
d86a68f713
Review Response
2022-11-14 10:07:22 -05:00
Łukasz Magiera
73aabdfeff
Merge pull request #9601 from filecoin-project/fix/fsm-release-dealq-on-snap-abort
...
fix: sealing pipeine: Release assigned deals on snapdeals abort
2022-11-14 11:07:33 +01:00
Aayush
69c55d807a
Sealing manager: minor refactor
2022-11-09 09:09:59 -05:00
Łukasz Magiera
cbcd0d49ae
storage: Log a detailed error on PoSt challenge read timeout
2022-11-09 12:49:52 +01:00
Łukasz Magiera
42d02dd448
feat: storage: Force exit GenerateSingleVanillaProof on cancelled context
2022-11-09 12:38:23 +01:00
zenground0
3a6aae3e10
Remove another panic
2022-11-08 14:39:22 +00:00
zenground0
10b7961a99
Correctly pass abort upgrade an error
2022-11-08 14:11:28 +00:00
zenground0
7f650ad7dc
Go is hard
2022-11-08 12:30:53 +00:00
Łukasz Magiera
fcad93dc10
netbs: Fix lint
2022-11-08 09:37:43 +00:00
Łukasz Magiera
ec89424c42
make repo not depend on ffi
2022-11-08 09:37:43 +00:00
Łukasz Magiera
5f5cc794f0
make config not depend on ffi
2022-11-08 09:37:43 +00:00
Łukasz Magiera
33f7746296
fix: sealing pipeine: Release assigned deals on snapdeals abort
2022-11-07 16:50:56 +00:00
zenground0
c902218baf
Respect context
2022-11-07 16:03:22 +00:00
zenground0
6dee0d8ca8
Rough draft no tests
2022-11-07 14:56:53 +00:00
Aayush
c0b7343e60
Merge branch 'release/v1.18.0' into asr/merge-release-into-master
2022-11-06 14:40:13 -05:00
Łukasz Magiera
32de871243
fix: post worker sched: Don't check worker session in a busy loop
2022-10-14 15:58:25 +02:00
Aayush
65e37c29e4
address review
2022-10-12 13:54:04 -04:00
Łukasz Magiera
dc102f076a
sealing pipeline: Get allocationId with StateApi
2022-10-12 13:54:03 -04:00
Łukasz Magiera
0af9888b12
sealing pipeline: Prepare deal assigning logic for FIP-45
2022-10-12 13:42:53 -04:00
