Commit Graph

254 Commits

Author SHA1 Message Date
Andrew Jackson (Ajax)
a520ee85d6 master-to-sturdy 2023-09-21 10:37:02 -05:00
Aayush
52e7546e98 address review: make warmupEpoch an atomic.Int64 2023-05-10 14:46:26 -04:00
Aayush
92dcfe530b feat: splitstore: access warmupepoch atomically instead of locking 2023-05-10 14:19:33 -04:00
Aayush
5a061e4677 Merge branch 'releases' into asr/merge-release-into-master 2023-04-23 14:58:50 -04:00
Jorropo
fe42d974a2 chore: all: migrate from go-libipfs to boxo
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Steven Allen
34f46d736f fix: revert chain sync mutex to a regular lock
Conditions always call "unlock" so we can't safely use the condition
with both the read and write side of lock. So we might as well revert
back to a regular lock.

fixes 
2023-04-10 17:16:20 +02:00
zenground0
b5da33c013 signal chain in and out of sync to compaction workers 2023-04-10 17:15:42 +02:00
Peter Rabbitson
6afa99d0a1 Plumb through a proper Flush() method on all blockstores 2023-03-14 02:30:15 +01:00
zenground0
bd0c010be6 Configur buffer and threshold 2023-03-09 08:40:14 -07:00
zenground0
ee9ff563d2 computing szPurge leads to deadlock and unneeded, remove 2023-03-08 17:51:59 -07:00
zenground0
a994153e27 GC respects target for max hotstore space 2023-03-07 07:39:17 -07:00
zenground0
5d9739a863 Track size of dags relevant to compaction 2023-03-07 07:39:17 -07:00
Jorropo
f572852d06 chore: all: bump go-libipfs to replace go-block-format
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58
2023-01-26 17:03:18 +01:00
ZenGround0
4ffded6fef
feat:splitstore:single compaction that can handle prune aka two marksets one compaction ()
* begin

* rough draft -- this should probably actually work?

* WIP

* Start testing

* message mode

* Fix tests, make gen

* Better default

* docsgen-cli

* Review Response

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-11-07 16:31:12 -05:00
ZenGround0
8b7be6d47e
feat:chain:splitstore auto prune ()
Auto Prune

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-08-08 16:06:32 -04:00
ZenGround0
0c91b0dc10
feat:chain:splitstore chain prune ()
* Splitstore chain prune
* Protect on reification for simpler logic and sound cold compact protect
* Recovery from checkpoint during chain prune
* Splitstore (discard and universal mode) running in itests
* Add pause and restart functions to itest block miner
* Add config options to itest full nodes
* Add FsRepo support for itest full ndoes

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-08-05 16:34:16 -04:00
Masih H. Derkani
6f686cae11 Upgrade to go-blockstore 1.2.0 and go-cid 0.2.0
Upgrade required dependencies and fix breaking changes.
2022-06-29 14:55:22 -04:00
Łukasz Magiera
05cdeb80c3 chore: remove redundant import prefixes 2022-06-15 12:06:22 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
vyzo
4524fbe936 wait for reify workers to finish when closing 2022-02-14 16:10:54 +02:00
vyzo
b576785aac rename GetHotView to IsHotView 2022-02-14 16:03:17 +02:00
vyzo
268366e446 cold object reification context option 2022-02-04 16:07:58 +02:00
vyzo
4cddfd1074 background cold object reification 2022-02-04 14:29:50 +02:00
vyzo
b13aa8f172 unblock waitForSync on close 2022-02-01 11:30:21 +02:00
vyzo
7b4ab2077b wait for sync in a non racey way 2022-02-01 11:10:51 +02:00
vyzo
3aabb0314a synchronously mark live refs on put/putmany 2022-02-01 09:07:14 +02:00
vyzo
710fda49f0 fix putmany marking 2022-01-31 17:35:30 +02:00
vyzo
2b14bda6b8 recursively mark puts during the critical section 2022-01-31 17:26:08 +02:00
vyzo
ee63be26a1 fix race in protectView 2022-01-31 12:40:26 +02:00
vyzo
1900c907fd account for missing refs in the markset in Has 2022-01-31 09:44:51 +02:00
vyzo
a9d4495d83 use both hot and cold when doing fetches for markset positive objects 2022-01-30 22:47:20 +02:00
vyzo
7931f1f8f9 fix lint 2022-01-30 15:43:52 +02:00
vyzo
dbc8903bac sortless compaction 2022-01-30 15:33:15 +02:00
vyzo
25284b5325 refactor: eliminate distinction between markset and markset visitors 2022-01-25 16:31:45 +02:00
Aayush Rajasekaran
dfb65ed89f Plumb contexts through 2021-12-11 17:04:00 -05:00
vyzo
489782e21b satisfy the spellchecker that masquarades as a linter 2021-12-03 12:15:28 +02:00
vyzo
05c1325885 add logic for supressing compaction near upgrade boundaries 2021-12-03 11:50:35 +02:00
vyzo
742c85bf81 add SupportsVisitor in the markset env interface 2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6 widen MarkSetVisitor interface and get rid of the casts 2021-08-10 10:47:01 +03:00
vyzo
9d25464703 use functional options for hotstore gc, rename MovingGC to FullGC 2021-07-27 09:53:22 +03:00
vyzo
5acae50e07 add config option for splitstore moving gc frequency 2021-07-26 15:05:57 +03:00
vyzo
fb3986226f do hotstore moving GC in splitstore with a user-specified frequency 2021-07-26 15:05:57 +03:00
vyzo
1918ffda4e implement splitstore check 2021-07-25 10:35:37 +03:00
vyzo
12c3432b8d document the "badger" markset type option 2021-07-23 12:47:18 +03:00
Łukasz Magiera
da5aeda197
Merge branch 'master' into feat/splitstore-hot-messages 2021-07-22 12:58:06 +02:00
vyzo
ebbaf23af8 support out-of-chain reference protection 2021-07-20 09:02:40 +03:00
vyzo
1b77361301 add option for hotstore message retention 2021-07-17 08:35:35 +03:00
vyzo
5a23f64b3b code reorg: break splitstore.go into smaller logical units 2021-07-14 13:11:15 -07:00
vyzo
023146803d use Broadcast for view barrier 2021-07-14 22:59:53 +03:00
vyzo
3d77ae1f4d make trackTxnRefMany consistent with trackTxnRef 2021-07-14 22:59:53 +03:00