Commit Graph
529 Commits
Author SHA1 Message Date
Steven Allen 87801b5855 feat: shed: fix blockstore prune (#11197)
1. Implement a pass-through for garbage collection on the idstore.
2. Fix the `lotus-shed state-prune` command.

NOTE: the new performance of running a full prune will be significantly
less as this version doesn't batch. However, it should now actually
_work_ and most users will be using the splitstore anyways.
2023-10-27 11:33:18 -07:00
Aayush 5e76b05b17 Merge branch 'feat/nv21' into asr/merge-feat-nv21 2023-10-16 11:13:33 -04:00
Steven Allen c6debaadf0 make gen 2023-09-23 17:54:44 -04:00
Fridrik Asmundsson 2d7a9b0551 Moved work duration to a constant 2023-09-18 15:43:01 +00:00
Fridrik Asmundsson 087d799547 Increase time spent pausing to 4sec 2023-09-15 10:57:43 +00:00
Fridrik Asmundsson 131f87afb1 Update changelog 2023-09-14 13:58:01 +00:00
Fridrik Asmundsson d9755dfb1c Add time slicing ot splitstore purging to reduce lock congestion 2023-09-14 13:39:15 +00:00
Mikers 5e5a81bf23 add go linter - "unused" (#11235)
* add go linter - "unused"

* use _ to name unused but needed padding variable

* remove unused code

* add queue test to appease unused linter

* remove unused code in test

* remove unused func

* remove unused struct identified by linter

* remove unused variable

* remove unused code

* remove unused file

* remove unused struct

* remove unused function

* remove unused observe peers function in raft

* remove unused declareFaults function

* annotate nolint:unused on needed methods
2023-09-05 12:19:43 -10:00
Łukasz Magiera b8b9158074 make gen 2023-06-22 10:43:56 +02:00
hannahhoward cc207ede88 refactor(deps): use forked kubo-api-client 2023-06-19 14:45:06 -07:00
Adin Schmahmann 73f09878e1 chore(blockstore): switch ipfs blockstore from go-ipfs-http-api to kubo/client/rpc 2023-06-19 14:45:06 -07:00
Jorropo 6c01310728 chore: migrate to boxo
This migrates everything except the `go-car` librairy: https://github.com/ipfs/boxo/issues/218#issuecomment-1529922103

I didn't migrated everything in the previous release because all the boxo code wasn't compatible with the go-ipld-prime one due to a an in flight (/ aftermath) revert of github.com/ipfs/go-block-format. go-block-format has been unmigrated since slight bellow absolutely everything depends on it that would have required everything to be moved on boxo or everything to optin into using boxo which were all deal breakers for different groups.

This worked fine because lotus's codebase could live hapely on the first multirepo setup however boost is now trying to use boxo's code with lotus's (still on multirepo) setup: https://filecoinproject.slack.com/archives/C03AQ3QAUG1/p1685022344779649

The alternative would be for boost to write shim types which just forward calls and return with the different interface definitions.

Btw why is that an issue in the first place is because unlike what go's duck typing model suggest interfaces are not transparent https://github.com/golang/go/issues/58112, interfaces are strongly typed but they have implicit narrowing. The issue is if you return an interface from an interface Go does not have a function definition to insert the implicit conversion thus instead the type checker complains you are not returning the right type.

Stubbing types were reverted https://github.com/ipfs/boxo/issues/218#issuecomment-1478650351

Last time I only migrated `go-bitswap` to `boxo/bitswap` because of the security issues and because we never had the interface return an interface problem (we had concrete wrappers where the implicit conversion took place).
2023-06-19 14:45:05 -07:00
Aayush f4a3207ede fix: splitstore: remove deadlock around waiting for sync 2023-05-11 09:32:40 -04:00
Aayush 52e7546e98 address review: make warmupEpoch an atomic.Int64 2023-05-10 14:46:26 -04:00
Aayush 92dcfe530b feat: splitstore: access warmupepoch atomically instead of locking 2023-05-10 14:19:33 -04:00
zenground0 bca0023756 Don't block when potentially holding txnLk as writer 2023-05-03 09:55:22 -06:00
Aayush Rajasekaran 7903224ae5 Merge pull request #10680 from filecoin-project/steb/blockstore-has-check
feat: badger: add a has check before writing to reduce duplicates
2023-04-26 16:17:16 -04:00
Aayush 5a061e4677 Merge branch 'releases' into asr/merge-release-into-master 2023-04-23 14:58:50 -04:00
Jorropo fe42d974a2 chore: all: migrate from go-libipfs to boxo
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Steven Allen 0cff56a16d feat: badger: add a has check before writing to reduce duplicates 2023-04-17 10:57:41 -07:00
Steven Allen 34f46d736f fix: revert chain sync mutex to a regular lock
Conditions always call "unlock" so we can't safely use the condition
with both the read and write side of lock. So we might as well revert
back to a regular lock.

fixes #10616
2023-04-10 17:16:20 +02:00
zenground0 d77ac5ddcb Stop swallowing errors 2023-04-10 17:16:13 +02:00
zenground0 0ce9ae4809 Implement yield friendly online GC 2023-04-10 17:16:01 +02:00
zenground0 fd66db680f check yield before GC 2023-04-10 17:15:52 +02:00
zenground0 b5da33c013 signal chain in and out of sync to compaction workers 2023-04-10 17:15:42 +02:00
Phi-rjanandzenground0 b240031605 Limit moving gc threads (#10623)
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2023-04-05 11:21:32 -04:00
zenground0 c281d053d2 Limit moving gc threads 2023-04-04 14:07:36 -06:00
Aayush 66fc6dc3e5 refactor: stop using deprecated io/ioutil 2023-03-29 15:40:49 -04:00
Aayush a724a713da refactor: stop using deprecated io/ioutil 2023-03-29 15:38:46 -04:00
ZenGround0andzenground0 2b3a86eefb Demote now common logs (#10516)
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2023-03-20 10:54:30 -04:00
zenground0 252387768d Don't enforce walking receipt tree during compaction 2023-03-17 09:29:18 -06:00
Peter Rabbitson 6afa99d0a1 Plumb through a proper Flush() method on all blockstores 2023-03-14 02:30:15 +01:00
zenground0 87d5a3723f lint 2023-03-09 08:57:14 -07:00
zenground0 bd0c010be6 Configur buffer and threshold 2023-03-09 08:40:14 -07:00
zenground0 aac30cd840 Review Response 2023-03-09 07:57:35 -07:00
zenground0 fb7eb08624 lint 2023-03-09 06:36:35 -07:00
zenground0 140f2099bd lint 2023-03-08 22:49:30 -07:00
zenground0 0b0913f2f3 lint 2023-03-08 18:40:23 -07:00
zenground0 7931414735 Lint 2023-03-08 18:20:27 -07:00
zenground0 efbb63582e Review Response 2023-03-08 18:11:39 -07:00
zenground0 ee9ff563d2 computing szPurge leads to deadlock and unneeded, remove 2023-03-08 17:51:59 -07:00
zenground0 e5553554d1 cid key size 2023-03-08 17:30:42 -07:00
zenground0 efbc0ff12e Fix up approximation and logging 2023-03-08 09:43:37 -07:00
zenground0 e17ec37d10 fix logging 2023-03-08 09:13:00 -07:00
zenground0 bf29d41993 lint 2023-03-07 08:21:15 -07:00
zenground0 a994153e27 GC respects target for max hotstore space 2023-03-07 07:39:17 -07:00
zenground0 0fe91846cd Plan out moving GC limiting 2023-03-07 07:39:17 -07:00
zenground0 5d9739a863 Track size of dags relevant to compaction 2023-03-07 07:39:17 -07:00
zenground0 dd998d6b24 Begin account for size during walks 2023-03-07 07:39:17 -07:00
zenground0 5534755f2d Add command to trigger moving GC of hotstore manually 2023-03-06 11:59:08 -07:00