Commit Graph

103 Commits

Author SHA1 Message Date
Rod Vagg
26d3fd2ecc chore: lint: fix lint errors with new linting config
Ref: https://github.com/filecoin-project/lotus/issues/11967
2024-05-13 14:52:09 +10:00
Andrew Jackson (Ajax)
441f5995d3
fix: curio: node UI & darwin gpu count (#11950)
* fix: curio: Update pgx imports, fix db_storage alloc

* feat: curioweb: Improve task_history indexes (#11911)

* 1

* relatable

* add and delete layer

* chore: curio: merge master (#11956)

* Fixing dead links (#11907)

* ci: ci: create gh workflow that updates sorted pr checks (#11861)

* ci: create gh workflow that updates sorted pr checks

* ci: use grouped_by_result template for pr checks sticky comment

* chore: apply pr review suggestion

* Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916)

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* Fix mismatched method names in comments (#11913)

Signed-off-by: forcedebug <forcedebug@outlook.com>

* release: v1.26.3 (#11908) (#11915) (#11922)

* deps: update dependencies to address migration memory bloat

to address memory concerns during a heavy migration

Ref: https://github.com/filecoin-project/go-state-types/pull/260
Ref: https://github.com/whyrusleeping/cbor-gen/pull/96
Ref: https://github.com/filecoin-project/go-amt-ipld/pull/90

* release: prep v1.26.3 patch

Prep v1.26.3 patch release:
- Update changelog, version and make gen + make docsgen-cli

* deps: update cbor-gen to tagged version

deps: update cbor-gen to tagged version

* deps: update go-state-types to tagged version

deps: update go-state-types to tagged version v0.13.2

* chore: deps: update go-state-types to v0.13.3

Fixes a panic when we have fewer than 1k proposals.

---------

Co-authored-by: Phi-rjan <orjan.roren@gmail.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Steven Allen <steven@stebalien.com>

* Refactor `LookupID*` APIs in `StateManager` and `StateTree`

The naming of `LookupID` can cause confusion when resolving actor IDs vs
 ID addresses. To avoid this:

* Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it
returns ID address.
* Refactor `StateManager` `LookupID` to
`LookupIDAddress` because it also returns ID address via a chain call to
`StateTree`.
* Introduce a new API `StateManager` dedicated to resolving address to
actor ID, called `LookupID` which returns `abi.ActorID`.

For context, see:
 * https://github.com/filecoin-project/lotus/pull/11723#discussion_r1534728607

* Add v13 support to invariants-checker (#11931)

Add v13 support to invariants-checker

* chore: docs: nv-skeleton documentation (#11065)

* nv-skeleton documentation

Add a tutorial for how one can create a nv-skeleton in Lotus

* Add footnote for `Add migration` step

Add footnote for `Add migration` step

* Indent migration-code

Indent migration-code to make it show properly as a footnote.

* Add ref-fvm and filecoin-ffi checklist

Add ref-fvm and filecoin-ffi checklist

* Add Filecoin-FFI steps

Add Filecoin-FFI steps

* Add step to params_butterfly.go

Add step to params_butterfly.go

* Fix typo

Fix typo

* Add links to reference PRs

Add links to reference PRs

* Update ref-fvm list

Update ref-fvm list

* feat: curio: add StorageInit api (#11918)

* feat: add StorageInit api

* remove unused variables

* fix gen check

* feat: curio: simpler reservation release logic (#11900)

* simpler release logic

* oops, plus simpler

* simpler

* fix NewLine (#11893)

* fix(events): check for sync-in-progress (#11932)

* feat(events): adjust indexes in event index db to match query patterns

Introduces a v4 migration that just adjusts indexes.

Copies some improvements from https://github.com/filecoin-project/lotus/pull/11723

Closes: https://github.com/filecoin-project/lotus/issues/11909

* fix(pipeline): should return if error occurred when get network version (#11902)

* fix(events): correct log msg for v4 events index db migration

* chore: remove duplicate words in strings and comments

* fix(events): register events index db migration v4

* fix: curio seal: Failed commit retry strategy (#11870)

* ffi: improved-error-handling

* curio seal: Failed commit retry strategy

* use master ffi

* mod tidy

* fix: curio: Update pgx imports, fix db_storage alloc

* feat: curioweb: Improve task_history indexes (#11911)

* mod tidy

* Event index should be unique for tipsets (#11952)

* event index should be unique for tipsets

* fix formatting

* migrate to version 5

* chore: bump build version in master (#11946)

* Bump version

Bump version in master branch in preperation for cutting v1.27.0-rc1

* chore: bump build-version

chore: bump build-version

* feat: curioweb: Show piece info on the sector page (#11955)

* curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895)

* break trees task

* fix TreeD reservation

* fix nil pointer err

* apply suggestions

* fix allocate file types

* fix dbIndex inserts

* set resource, move release func

* refactor func(), update memory

* remove extra release

---------

Signed-off-by: forcedebug <forcedebug@outlook.com>
Co-authored-by: parthshah1 <parthhshah171@gmail.com>
Co-authored-by: Piotr Galar <piotr.galar@gmail.com>
Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Phi-rjan <orjan.roren@gmail.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Masih H. Derkani <m@derkani.org>
Co-authored-by: Lee <octalgah@gmail.com>
Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com>
Co-authored-by: beck <34204218+beck-8@users.noreply.github.com>
Co-authored-by: 0x5459 <0x5459@protonmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>

* linter oops

* gen cleanup

* fix

* named returns are confusing

---------

Signed-off-by: forcedebug <forcedebug@outlook.com>
Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: parthshah1 <parthhshah171@gmail.com>
Co-authored-by: Piotr Galar <piotr.galar@gmail.com>
Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Phi-rjan <orjan.roren@gmail.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Masih H. Derkani <m@derkani.org>
Co-authored-by: Lee <octalgah@gmail.com>
Co-authored-by: beck <34204218+beck-8@users.noreply.github.com>
Co-authored-by: 0x5459 <0x5459@protonmail.com>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
2024-05-04 10:35:30 +02:00
LexLuthr
00edad4e4d
curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895)
* break trees task

* fix TreeD reservation

* fix nil pointer err

* apply suggestions

* fix allocate file types

* fix dbIndex inserts

* set resource, move release func

* refactor func(), update memory

* remove extra release
2024-05-02 11:17:33 +02:00
Łukasz Magiera
1b6bffd339 fix: curio: Update pgx imports, fix db_storage alloc 2024-04-30 20:43:56 +02:00
Andrew Jackson (Ajax)
5e1d8f661e
feat: curio: simpler reservation release logic (#11900)
* simpler release logic

* oops, plus simpler

* simpler
2024-04-26 11:11:42 +02:00
Andrew Jackson (Ajax)
6b3e9b109f
feat: curio: sectors UI (#11869)
* cfg edit 1

* jsonschema deps

* feat: lp mig - first few steps

* lp mig: default tasks

* code comments

* docs

* lp-mig-progress

* shared

* comments and todos

* fix: curio: rename lotus-provider to curio (#11645)

* rename provider to curio

* install gotext

* fix lint errors, mod tidy

* fix typo

* fix API_INFO and add gotext to circleCI

* add back gotext

* add gotext after remerge

* lp: channels doc

* finish easy-migration TODOs

* out generate

* merging and more renames

* avoid make-all

* minor doc stuff

* cu: make gen

* make gen fix

* make gen

* tryfix

* go mod tidy

* minor ez migration fixes

* ez setup - ui cleanups

* better error message

* guided setup colors

* better path to saveconfigtolayer

* loadconfigwithupgrades fix

* readMiner oops

* guided - homedir

* err if miner is running

* prompt error should exit

* process already running, miner_id sectors in migration

* dont prompt for language a second time

* check miner stopped

* unlock repo

* render and sql oops

* curio easyMig - some fixes

* easyMigration runs successfully

* lint

* part 2 of last

* message

* merge addtl

* fixing guided setup for myself

* warn-on-no-post

* EditorLoads

* cleanups and styles

* create info

* fix tests

* make gen

* sector early bird

* sectors v2

* sector termination v1

* terminate2

* mjs

* minor things

* flag bad sectors

* fix errors

* add dealweight and deals

* change column width

* refactor sql, handle sealing sectors

* fix estimates

---------

Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: LexLuthr <lexluthr@protocol.ai>
Co-authored-by: LexLuthr <lexluthr@curiostorage.org>
2024-04-18 14:57:29 -05:00
Łukasz Magiera
c785e59371
feat: curio: storage index gc task (#11884)
* curio storage path gc: lay out the structure

* curio gc: Implement storage metadata gc

* move bored singleton task impl to harmonytask

* curio: run storage gc task on storage node

* make gen
2024-04-16 16:34:48 -05:00
LexLuthr
3ebb92a976
fix: curio: fix incorrect null check for varchar column (#11881)
* err check empty string

* log skipped tasks
2024-04-15 19:27:31 -05:00
Łukasz Magiera
bc43bd6d69
fix: local storage reservations fixes (#11866)
* paths: Debugging local storage reservations

* paths: Log when individual reservation is less than on-disk space

* paths: fix debug reservations print

* paths: More reserve logs

* paths: More more reserve logs

* paths: add stacks to duplicate done call log

* curio: task storage: Release storage at most once

* curio: cleanup before restarting sdr

* address review

* paths: Simplify reservation release logic
2024-04-15 17:33:06 -05:00
Łukasz Magiera
3cc62e04e9
feat: curio: Cleanup data copies after seal ops (#11847)
* feat: curio: Cleanup data copies after seal ops

* curio: make ensureOneCopy safer

* curio: Don't randomly remove cache files

* Address review
2024-04-11 11:35:09 -05:00
Łukasz Magiera
d7d849cf20
feat: curioweb: Sector info page (#11846)
* feat: curioweb: Sector info page

* Address review
2024-04-11 11:30:53 -05:00
Łukasz Magiera
fd64e38b39
fix: paths/local: Fix on-disk storage accounting in new reservations (#11825) 2024-04-04 11:17:04 -05:00
LexLuthr
a24028c3c0 apply suggestions 2024-04-04 18:06:19 +02:00
LexLuthr
243bf1a0b3 refactor MinerFilter func, update comments 2024-04-04 18:06:19 +02:00
LexLuthr
d1a52cf04d refactor filter decision code 2024-04-04 18:06:19 +02:00
LexLuthr
2951d038a8 add miner fliter 2024-04-04 18:06:19 +02:00
Łukasz Magiera
2ccc0db1cc feat: curio: Storage reservations in MoveStorage 2024-04-03 21:40:06 +02:00
Łukasz Magiera
021385116a paths: Fix local existing allocate requests in Finalize 2024-03-26 20:32:46 +01:00
Łukasz Magiera
c7ca271507 rebuild storage mocks 2024-03-26 20:32:46 +01:00
Łukasz Magiera
7b00cc5827 curio: Add task storage to SDRTrees 2024-03-26 20:32:46 +01:00
Łukasz Magiera
c7b64bd6a9 curio: Storage reservations when fetching 2024-03-26 20:32:46 +01:00
Łukasz Magiera
e86d81b26a fix: curio seal: Fix partial Finalize retry 2024-03-22 16:05:48 +01:00
Łukasz Magiera
6ca55d18a7 address review 2024-03-15 14:10:48 +01:00
Łukasz Magiera
b90cf19604 lpseal: PiecePark in SDRTrees 2024-03-14 09:36:51 +01:00
Łukasz Magiera
0800e6e5a7 make gen 2024-03-14 09:36:47 +01:00
Łukasz Magiera
e060cd2f37 lppiece: Implement Piece Park 2024-03-14 09:36:03 +01:00
Łukasz Magiera
5ab75d35f9 fix mac build 2024-02-22 14:00:47 +01:00
Łukasz Magiera
0634dfce07 Fix lint 2024-02-22 11:39:17 +01:00
Łukasz Magiera
4ae73f059c provider: Fix sql query capitalization 2024-02-21 14:19:48 +01:00
Łukasz Magiera
91d3edecfd lp: Address review 2024-02-21 13:28:49 +01:00
Łukasz Magiera
9f9e2f7fe4 make gen 2024-02-17 18:13:30 +01:00
Łukasz Magiera
c293bcebb6 storage: Make remote Generate Vanilla more robust 2024-02-17 11:20:49 +01:00
Łukasz Magiera
e22ab4c8fe lp: makePhase1Out for large sectors 2024-02-17 01:40:01 +01:00
Łukasz Magiera
06a2f8c670 paths: Fix dbindex heartbeat 2024-02-17 00:57:27 +01:00
Łukasz Magiera
bc8517baa0 paths: Support multiple endpoints in FsStat 2024-02-16 21:23:21 +01:00
Łukasz Magiera
9ddfe23702 lp: Small migration improvements 2024-02-16 21:23:19 +01:00
Łukasz Magiera
4c2235a686 make gen, mod tidy 2024-02-16 21:21:25 +01:00
Łukasz Magiera
fe7558368a storage: Support C1 directly from sector storage 2024-02-16 21:21:25 +01:00
Łukasz Magiera
725460bc45 dbindex: Fix heartbeat checks 2024-02-16 21:21:25 +01:00
Łukasz Magiera
47672e0933 storage: Fix dbindex heartbeat 2024-02-16 21:20:13 +01:00
Andrew Jackson (Ajax)
96353e63ea exp backoff, short stack err 2023-12-11 10:50:49 -06:00
Andrew Jackson (Ajax)
0e49673c49 fix: db serialize txn - retry 2023-12-07 15:32:35 -06:00
Andrew Jackson (Ajax)
81e59b7fce fix:sector_filetype undef 2023-11-28 09:05:59 -06:00
Łukasz Magiera
f0a931edc9 fix: miner: Don't require db config when it's not used 2023-11-23 16:28:39 +01:00
Andrew Jackson (Ajax)
848c20d15f Merge branch 'master' into feat/sturdypost 2023-11-06 16:10:57 -06:00
Andrew Jackson (Ajax)
43680400c3 lp runs. todo ds-sectorInfo move & cleanups 2023-10-24 19:26:13 -05:00
Łukasz Magiera
d981baa362 storage: Return soft err when sector alloc fails in acquire 2023-10-17 11:42:41 +02:00
Andrew Jackson (Ajax)
a520ee85d6 master-to-sturdy 2023-09-21 10:37:02 -05:00
Łukasz Magiera
bcc2d7fd10 ffiwrapper: Fix RegenerateSectorKey 2023-08-31 09:46:10 -04:00
Shrenuj Bansal
15247486ae anotha one 2023-08-22 12:19:18 -04:00