Commit Graph

532 Commits

Author SHA1 Message Date
Andrew Jackson (Ajax)
848c20d15f Merge branch 'master' into feat/sturdypost 2023-11-06 16:10:57 -06:00
Steven Allen
87801b5855
feat: shed: fix blockstore prune (#11197)
1. Implement a pass-through for garbage collection on the idstore.
2. Fix the `lotus-shed state-prune` command.

NOTE: the new performance of running a full prune will be significantly
less as this version doesn't batch. However, it should now actually
_work_ and most users will be using the splitstore anyways.
2023-10-27 11:33:18 -07:00
Aayush
5e76b05b17 Merge branch 'feat/nv21' into asr/merge-feat-nv21 2023-10-16 11:13:33 -04:00
Steven Allen
c6debaadf0 make gen 2023-09-23 17:54:44 -04:00
Andrew Jackson (Ajax)
a888b6320c harmonytask bump http protocol spec & lints 2023-09-22 18:06:26 -05:00
Andrew Jackson (Ajax)
a520ee85d6 master-to-sturdy 2023-09-21 10:37:02 -05:00
Fridrik Asmundsson
2d7a9b0551 Moved work duration to a constant 2023-09-18 15:43:01 +00:00
Fridrik Asmundsson
087d799547 Increase time spent pausing to 4sec 2023-09-15 10:57:43 +00:00
Fridrik Asmundsson
131f87afb1 Update changelog 2023-09-14 13:58:01 +00:00
Fridrik Asmundsson
d9755dfb1c Add time slicing ot splitstore purging to reduce lock congestion 2023-09-14 13:39:15 +00:00
Mikers
5e5a81bf23
add go linter - "unused" (#11235)
* add go linter - "unused"

* use _ to name unused but needed padding variable

* remove unused code

* add queue test to appease unused linter

* remove unused code in test

* remove unused func

* remove unused struct identified by linter

* remove unused variable

* remove unused code

* remove unused file

* remove unused struct

* remove unused function

* remove unused observe peers function in raft

* remove unused declareFaults function

* annotate nolint:unused on needed methods
2023-09-05 12:19:43 -10:00
Łukasz Magiera
b8b9158074 make gen 2023-06-22 10:43:56 +02:00
hannahhoward
cc207ede88
refactor(deps): use forked kubo-api-client 2023-06-19 14:45:06 -07:00
Adin Schmahmann
73f09878e1
chore(blockstore): switch ipfs blockstore from go-ipfs-http-api to kubo/client/rpc 2023-06-19 14:45:06 -07:00
Jorropo
6c01310728
chore: migrate to boxo
This migrates everything except the `go-car` librairy: https://github.com/ipfs/boxo/issues/218#issuecomment-1529922103

I didn't migrated everything in the previous release because all the boxo code wasn't compatible with the go-ipld-prime one due to a an in flight (/ aftermath) revert of github.com/ipfs/go-block-format. go-block-format has been unmigrated since slight bellow absolutely everything depends on it that would have required everything to be moved on boxo or everything to optin into using boxo which were all deal breakers for different groups.

This worked fine because lotus's codebase could live hapely on the first multirepo setup however boost is now trying to use boxo's code with lotus's (still on multirepo) setup: https://filecoinproject.slack.com/archives/C03AQ3QAUG1/p1685022344779649

The alternative would be for boost to write shim types which just forward calls and return with the different interface definitions.

Btw why is that an issue in the first place is because unlike what go's duck typing model suggest interfaces are not transparent https://github.com/golang/go/issues/58112, interfaces are strongly typed but they have implicit narrowing. The issue is if you return an interface from an interface Go does not have a function definition to insert the implicit conversion thus instead the type checker complains you are not returning the right type.

Stubbing types were reverted https://github.com/ipfs/boxo/issues/218#issuecomment-1478650351

Last time I only migrated `go-bitswap` to `boxo/bitswap` because of the security issues and because we never had the interface return an interface problem (we had concrete wrappers where the implicit conversion took place).
2023-06-19 14:45:05 -07:00
Aayush
f4a3207ede fix: splitstore: remove deadlock around waiting for sync 2023-05-11 09:32:40 -04:00
Aayush
52e7546e98 address review: make warmupEpoch an atomic.Int64 2023-05-10 14:46:26 -04:00
Aayush
92dcfe530b feat: splitstore: access warmupepoch atomically instead of locking 2023-05-10 14:19:33 -04:00
zenground0
bca0023756 Don't block when potentially holding txnLk as writer 2023-05-03 09:55:22 -06:00
Aayush Rajasekaran
7903224ae5
Merge pull request #10680 from filecoin-project/steb/blockstore-has-check
feat: badger: add a has check before writing to reduce duplicates
2023-04-26 16:17:16 -04:00
Aayush
5a061e4677 Merge branch 'releases' into asr/merge-release-into-master 2023-04-23 14:58:50 -04:00
Jorropo
fe42d974a2 chore: all: migrate from go-libipfs to boxo
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Steven Allen
0cff56a16d feat: badger: add a has check before writing to reduce duplicates 2023-04-17 10:57:41 -07:00
Steven Allen
34f46d736f fix: revert chain sync mutex to a regular lock
Conditions always call "unlock" so we can't safely use the condition
with both the read and write side of lock. So we might as well revert
back to a regular lock.

fixes #10616
2023-04-10 17:16:20 +02:00
zenground0
d77ac5ddcb Stop swallowing errors 2023-04-10 17:16:13 +02:00
zenground0
0ce9ae4809 Implement yield friendly online GC 2023-04-10 17:16:01 +02:00
zenground0
fd66db680f check yield before GC 2023-04-10 17:15:52 +02:00
zenground0
b5da33c013 signal chain in and out of sync to compaction workers 2023-04-10 17:15:42 +02:00
Phi-rjan
b240031605
Limit moving gc threads (#10623)
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2023-04-05 11:21:32 -04:00
zenground0
c281d053d2 Limit moving gc threads 2023-04-04 14:07:36 -06:00
Aayush
66fc6dc3e5 refactor: stop using deprecated io/ioutil 2023-03-29 15:40:49 -04:00
Aayush
a724a713da refactor: stop using deprecated io/ioutil 2023-03-29 15:38:46 -04:00
ZenGround0
2b3a86eefb
Demote now common logs (#10516)
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2023-03-20 10:54:30 -04:00
zenground0
252387768d Don't enforce walking receipt tree during compaction 2023-03-17 09:29:18 -06:00
Peter Rabbitson
6afa99d0a1 Plumb through a proper Flush() method on all blockstores 2023-03-14 02:30:15 +01:00
zenground0
87d5a3723f lint 2023-03-09 08:57:14 -07:00
zenground0
bd0c010be6 Configur buffer and threshold 2023-03-09 08:40:14 -07:00
zenground0
aac30cd840 Review Response 2023-03-09 07:57:35 -07:00
zenground0
fb7eb08624 lint 2023-03-09 06:36:35 -07:00
zenground0
140f2099bd lint 2023-03-08 22:49:30 -07:00
zenground0
0b0913f2f3 lint 2023-03-08 18:40:23 -07:00
zenground0
7931414735 Lint 2023-03-08 18:20:27 -07:00
zenground0
efbb63582e Review Response 2023-03-08 18:11:39 -07:00
zenground0
ee9ff563d2 computing szPurge leads to deadlock and unneeded, remove 2023-03-08 17:51:59 -07:00
zenground0
e5553554d1 cid key size 2023-03-08 17:30:42 -07:00
zenground0
efbc0ff12e Fix up approximation and logging 2023-03-08 09:43:37 -07:00
zenground0
e17ec37d10 fix logging 2023-03-08 09:13:00 -07:00
zenground0
bf29d41993 lint 2023-03-07 08:21:15 -07:00
zenground0
a994153e27 GC respects target for max hotstore space 2023-03-07 07:39:17 -07:00
zenground0
0fe91846cd Plan out moving GC limiting 2023-03-07 07:39:17 -07:00