535 Commits
Author SHA1 Message Date
Steven Allen e0b3ea7294 chore: deps: update boxo/libp2p/graphsync (#11739)
- Pulls in the latest version of `kubo-api-client` (extracted from kubo
  0.27.0).
- Update go-libp2p to v0.33.0
- Update boxo to v0.18.0
- Update go-graphsync to v0.16.0
2024-03-22 07:00:28 +01:00
Rod Vagg d8fb28e36d AggregateProofType nil when doing batch updates
Use latest nv22 go-state-types version with matching update
2024-03-22 07:00:28 +01:00
6a0f16b084 feat: sealing: Support nv22 DDO features in the sealing pipeline (#11226)
* Initial work supporting DDO pieces in lotus-miner

* sealing: Update pipeline input to operate on UniversalPiece

* sealing: Update pipeline checks/sealing states to operate on UniversalPiece

* sealing: Make pipeline build with UniversalPiece

* move PieceDealInfo out of api

* make gen

* make sealing pipeline unit tests pass

* fix itest ensemble build

* don't panic in SectorsStatus with deals

* stop linter from complaining about checkPieces

* fix sector import tests

* mod tidy

* sealing: Add logic for (pre)committing DDO sectors

* sealing: state-types with method defs

* DDO non-snap pipeline works(?), DDO Itests

* DDO support in snapdeals pipeline

* make gen

* update actor bundles

* update the gst market fix

* fix: chain: use PreCommitSectorsBatch2 when setting up genesis

* some bug fixes

* integration working changes

* update actor bundles

* Make TestOnboardRawPieceSnap pass

* Appease the linter

* Make deadlines test pass with v12 actors

* Update go-state-types, abstract market DealState

* make gen

* mod tidy, lint fixes

* Fix some more tests

* Bump version in master

Bump version in master

* Make gen

Make gen

* fix sender

* fix: lotus-provider: Fix winning PoSt

* fix: sql Scan cannot write to an object

* Actually show miner-addrs in info-log

Actually show miner-addrs in lotus-provider info-log

* [WIP] feat: Add nv22 skeleton

Addition of Network Version 22 skeleton

* update FFI

* ddo is now nv22

* make gen

* temp actor bundle with ddo

* use working go-state-types

* gst with v13 market migration

* update bundle, builtin.MethodsMiner.ProveCommitSectors2 -> 3

* actually working v13 migration, v13 migration itest

* Address review

* sealing: Correct DDO snap pledge math

* itests: Mixed ddo itest

* pipeline: Fix sectorWeight

* sealing: convert market deals into PAMs in mixed sectors

* sealing: make market to ddo conversion work

* fix lint

* update gst

* Update actors and GST to lastest integ branch

* commit batcher: Update ProveCommitSectors3Params builder logic

* make gen

* use builtin-actors master

* ddo: address review

* itests: Add commd assertions to ddo tests

* make gen

* gst with fixed types

* config knobs for RequireActivationSuccess

* storage: Drop obsolete flaky tasts

---------

Co-authored-by: Jennifer Wang <jiayingw703@gmail.com>
Co-authored-by: Aayush <arajasek94@gmail.com>
Co-authored-by: Shrenuj Bansal <shrenuj.bansal@protocol.ai>
Co-authored-by: Phi <orjan.roren@gmail.com>
Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com>
Co-authored-by: TippyFlits <james.bluett@protocol.ai>
2024-03-22 07:00:28 +01:00
Andrew Jackson (Ajax) 848c20d15f Merge branch 'master' into feat/sturdypost 2023-11-06 16:10:57 -06:00
Steven Allen 87801b5855 feat: shed: fix blockstore prune (#11197)
1. Implement a pass-through for garbage collection on the idstore.
2. Fix the `lotus-shed state-prune` command.

NOTE: the new performance of running a full prune will be significantly
less as this version doesn't batch. However, it should now actually
_work_ and most users will be using the splitstore anyways.
2023-10-27 11:33:18 -07:00
Aayush 5e76b05b17 Merge branch 'feat/nv21' into asr/merge-feat-nv21 2023-10-16 11:13:33 -04:00
Steven Allen c6debaadf0 make gen 2023-09-23 17:54:44 -04:00
Andrew Jackson (Ajax) a888b6320c harmonytask bump http protocol spec & lints 2023-09-22 18:06:26 -05:00
Andrew Jackson (Ajax) a520ee85d6 master-to-sturdy 2023-09-21 10:37:02 -05:00
Fridrik Asmundsson 2d7a9b0551 Moved work duration to a constant 2023-09-18 15:43:01 +00:00
Fridrik Asmundsson 087d799547 Increase time spent pausing to 4sec 2023-09-15 10:57:43 +00:00
Fridrik Asmundsson 131f87afb1 Update changelog 2023-09-14 13:58:01 +00:00
Fridrik Asmundsson d9755dfb1c Add time slicing ot splitstore purging to reduce lock congestion 2023-09-14 13:39:15 +00:00
Mikers 5e5a81bf23 add go linter - "unused" (#11235)
* add go linter - "unused"

* use _ to name unused but needed padding variable

* remove unused code

* add queue test to appease unused linter

* remove unused code in test

* remove unused func

* remove unused struct identified by linter

* remove unused variable

* remove unused code

* remove unused file

* remove unused struct

* remove unused function

* remove unused observe peers function in raft

* remove unused declareFaults function

* annotate nolint:unused on needed methods
2023-09-05 12:19:43 -10:00
Łukasz Magiera b8b9158074 make gen 2023-06-22 10:43:56 +02:00
hannahhoward cc207ede88 refactor(deps): use forked kubo-api-client 2023-06-19 14:45:06 -07:00
Adin Schmahmann 73f09878e1 chore(blockstore): switch ipfs blockstore from go-ipfs-http-api to kubo/client/rpc 2023-06-19 14:45:06 -07:00
Jorropo 6c01310728 chore: migrate to boxo
This migrates everything except the `go-car` librairy: https://github.com/ipfs/boxo/issues/218#issuecomment-1529922103

I didn't migrated everything in the previous release because all the boxo code wasn't compatible with the go-ipld-prime one due to a an in flight (/ aftermath) revert of github.com/ipfs/go-block-format. go-block-format has been unmigrated since slight bellow absolutely everything depends on it that would have required everything to be moved on boxo or everything to optin into using boxo which were all deal breakers for different groups.

This worked fine because lotus's codebase could live hapely on the first multirepo setup however boost is now trying to use boxo's code with lotus's (still on multirepo) setup: https://filecoinproject.slack.com/archives/C03AQ3QAUG1/p1685022344779649

The alternative would be for boost to write shim types which just forward calls and return with the different interface definitions.

Btw why is that an issue in the first place is because unlike what go's duck typing model suggest interfaces are not transparent https://github.com/golang/go/issues/58112, interfaces are strongly typed but they have implicit narrowing. The issue is if you return an interface from an interface Go does not have a function definition to insert the implicit conversion thus instead the type checker complains you are not returning the right type.

Stubbing types were reverted https://github.com/ipfs/boxo/issues/218#issuecomment-1478650351

Last time I only migrated `go-bitswap` to `boxo/bitswap` because of the security issues and because we never had the interface return an interface problem (we had concrete wrappers where the implicit conversion took place).
2023-06-19 14:45:05 -07:00
Aayush f4a3207ede fix: splitstore: remove deadlock around waiting for sync 2023-05-11 09:32:40 -04:00
Aayush 52e7546e98 address review: make warmupEpoch an atomic.Int64 2023-05-10 14:46:26 -04:00
Aayush 92dcfe530b feat: splitstore: access warmupepoch atomically instead of locking 2023-05-10 14:19:33 -04:00
zenground0 bca0023756 Don't block when potentially holding txnLk as writer 2023-05-03 09:55:22 -06:00
Aayush Rajasekaran 7903224ae5 Merge pull request #10680 from filecoin-project/steb/blockstore-has-check
feat: badger: add a has check before writing to reduce duplicates
2023-04-26 16:17:16 -04:00
Aayush 5a061e4677 Merge branch 'releases' into asr/merge-release-into-master 2023-04-23 14:58:50 -04:00
Jorropo fe42d974a2 chore: all: migrate from go-libipfs to boxo
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Steven Allen 0cff56a16d feat: badger: add a has check before writing to reduce duplicates 2023-04-17 10:57:41 -07:00
Steven Allen 34f46d736f fix: revert chain sync mutex to a regular lock
Conditions always call "unlock" so we can't safely use the condition
with both the read and write side of lock. So we might as well revert
back to a regular lock.

fixes #10616
2023-04-10 17:16:20 +02:00
zenground0 d77ac5ddcb Stop swallowing errors 2023-04-10 17:16:13 +02:00
zenground0 0ce9ae4809 Implement yield friendly online GC 2023-04-10 17:16:01 +02:00
zenground0 fd66db680f check yield before GC 2023-04-10 17:15:52 +02:00
zenground0 b5da33c013 signal chain in and out of sync to compaction workers 2023-04-10 17:15:42 +02:00
Phi-rjanandzenground0 b240031605 Limit moving gc threads (#10623)
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2023-04-05 11:21:32 -04:00
zenground0 c281d053d2 Limit moving gc threads 2023-04-04 14:07:36 -06:00
Aayush 66fc6dc3e5 refactor: stop using deprecated io/ioutil 2023-03-29 15:40:49 -04:00
Aayush a724a713da refactor: stop using deprecated io/ioutil 2023-03-29 15:38:46 -04:00
ZenGround0andzenground0 2b3a86eefb Demote now common logs (#10516)
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2023-03-20 10:54:30 -04:00
zenground0 252387768d Don't enforce walking receipt tree during compaction 2023-03-17 09:29:18 -06:00
Peter Rabbitson 6afa99d0a1 Plumb through a proper Flush() method on all blockstores 2023-03-14 02:30:15 +01:00
zenground0 87d5a3723f lint 2023-03-09 08:57:14 -07:00
zenground0 bd0c010be6 Configur buffer and threshold 2023-03-09 08:40:14 -07:00
zenground0 aac30cd840 Review Response 2023-03-09 07:57:35 -07:00
zenground0 fb7eb08624 lint 2023-03-09 06:36:35 -07:00
zenground0 140f2099bd lint 2023-03-08 22:49:30 -07:00
zenground0 0b0913f2f3 lint 2023-03-08 18:40:23 -07:00
zenground0 7931414735 Lint 2023-03-08 18:20:27 -07:00
zenground0 efbb63582e Review Response 2023-03-08 18:11:39 -07:00
zenground0 ee9ff563d2 computing szPurge leads to deadlock and unneeded, remove 2023-03-08 17:51:59 -07:00
zenground0 e5553554d1 cid key size 2023-03-08 17:30:42 -07:00
zenground0 efbc0ff12e Fix up approximation and logging 2023-03-08 09:43:37 -07:00
zenground0 e17ec37d10 fix logging 2023-03-08 09:13:00 -07:00