Aayush
5a061e4677
Merge branch 'releases' into asr/merge-release-into-master
2023-04-23 14:58:50 -04:00
Jorropo
fe42d974a2
chore: all: migrate from go-libipfs to boxo
...
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Steven Allen
34f46d736f
fix: revert chain sync mutex to a regular lock
...
Conditions always call "unlock" so we can't safely use the condition
with both the read and write side of lock. So we might as well revert
back to a regular lock.
fixes #10616
2023-04-10 17:16:20 +02:00
zenground0
b5da33c013
signal chain in and out of sync to compaction workers
2023-04-10 17:15:42 +02:00
Peter Rabbitson
6afa99d0a1
Plumb through a proper Flush() method on all blockstores
2023-03-14 02:30:15 +01:00
zenground0
bd0c010be6
Configur buffer and threshold
2023-03-09 08:40:14 -07:00
zenground0
ee9ff563d2
computing szPurge leads to deadlock and unneeded, remove
2023-03-08 17:51:59 -07:00
zenground0
a994153e27
GC respects target for max hotstore space
2023-03-07 07:39:17 -07:00
zenground0
5d9739a863
Track size of dags relevant to compaction
2023-03-07 07:39:17 -07:00
Jorropo
f572852d06
chore: all: bump go-libipfs to replace go-block-format
...
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58
2023-01-26 17:03:18 +01:00
ZenGround0
4ffded6fef
feat:splitstore:single compaction that can handle prune aka two marksets one compaction ( #9571 )
...
* begin
* rough draft -- this should probably actually work?
* WIP
* Start testing
* message mode
* Fix tests, make gen
* Better default
* docsgen-cli
* Review Response
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-11-07 16:31:12 -05:00
ZenGround0
8b7be6d47e
feat:chain:splitstore auto prune ( #9123 )
...
Auto Prune
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-08-08 16:06:32 -04:00
ZenGround0
0c91b0dc10
feat:chain:splitstore chain prune ( #9056 )
...
* Splitstore chain prune
* Protect on reification for simpler logic and sound cold compact protect
* Recovery from checkpoint during chain prune
* Splitstore (discard and universal mode) running in itests
* Add pause and restart functions to itest block miner
* Add config options to itest full nodes
* Add FsRepo support for itest full ndoes
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-08-05 16:34:16 -04:00
Masih H. Derkani
6f686cae11
Upgrade to go-blockstore 1.2.0 and go-cid 0.2.0
...
Upgrade required dependencies and fix breaking changes.
2022-06-29 14:55:22 -04:00
Łukasz Magiera
05cdeb80c3
chore: remove redundant import prefixes
2022-06-15 12:06:22 +02:00
Łukasz Magiera
e65fae28de
chore: fix imports
2022-06-14 17:00:51 +02:00
vyzo
4524fbe936
wait for reify workers to finish when closing
2022-02-14 16:10:54 +02:00
vyzo
b576785aac
rename GetHotView to IsHotView
2022-02-14 16:03:17 +02:00
vyzo
268366e446
cold object reification context option
2022-02-04 16:07:58 +02:00
vyzo
4cddfd1074
background cold object reification
2022-02-04 14:29:50 +02:00
vyzo
b13aa8f172
unblock waitForSync on close
2022-02-01 11:30:21 +02:00
vyzo
7b4ab2077b
wait for sync in a non racey way
2022-02-01 11:10:51 +02:00
vyzo
3aabb0314a
synchronously mark live refs on put/putmany
2022-02-01 09:07:14 +02:00
vyzo
710fda49f0
fix putmany marking
2022-01-31 17:35:30 +02:00
vyzo
2b14bda6b8
recursively mark puts during the critical section
2022-01-31 17:26:08 +02:00
vyzo
ee63be26a1
fix race in protectView
2022-01-31 12:40:26 +02:00
vyzo
1900c907fd
account for missing refs in the markset in Has
2022-01-31 09:44:51 +02:00
vyzo
a9d4495d83
use both hot and cold when doing fetches for markset positive objects
2022-01-30 22:47:20 +02:00
vyzo
7931f1f8f9
fix lint
2022-01-30 15:43:52 +02:00
vyzo
dbc8903bac
sortless compaction
2022-01-30 15:33:15 +02:00
vyzo
25284b5325
refactor: eliminate distinction between markset and markset visitors
2022-01-25 16:31:45 +02:00
Aayush Rajasekaran
dfb65ed89f
Plumb contexts through
2021-12-11 17:04:00 -05:00
vyzo
489782e21b
satisfy the spellchecker that masquarades as a linter
2021-12-03 12:15:28 +02:00
vyzo
05c1325885
add logic for supressing compaction near upgrade boundaries
2021-12-03 11:50:35 +02:00
vyzo
742c85bf81
add SupportsVisitor in the markset env interface
2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6
widen MarkSetVisitor interface and get rid of the casts
2021-08-10 10:47:01 +03:00
vyzo
9d25464703
use functional options for hotstore gc, rename MovingGC to FullGC
2021-07-27 09:53:22 +03:00
vyzo
5acae50e07
add config option for splitstore moving gc frequency
2021-07-26 15:05:57 +03:00
vyzo
fb3986226f
do hotstore moving GC in splitstore with a user-specified frequency
2021-07-26 15:05:57 +03:00
vyzo
1918ffda4e
implement splitstore check
2021-07-25 10:35:37 +03:00
vyzo
12c3432b8d
document the "badger" markset type option
2021-07-23 12:47:18 +03:00
Łukasz Magiera
da5aeda197
Merge branch 'master' into feat/splitstore-hot-messages
2021-07-22 12:58:06 +02:00
vyzo
ebbaf23af8
support out-of-chain reference protection
2021-07-20 09:02:40 +03:00
vyzo
1b77361301
add option for hotstore message retention
2021-07-17 08:35:35 +03:00
vyzo
5a23f64b3b
code reorg: break splitstore.go into smaller logical units
2021-07-14 13:11:15 -07:00
vyzo
023146803d
use Broadcast for view barrier
2021-07-14 22:59:53 +03:00
vyzo
3d77ae1f4d
make trackTxnRefMany consistent with trackTxnRef
2021-07-14 22:59:53 +03:00
vyzo
6f126c80bf
remove redundant log, more descriptive error message for closing condition
2021-07-14 22:59:53 +03:00
vyzo
ff093fae00
use a missing compactionIndex as an indicator for warmup
...
so that splitstore v0 nodes upgrading will get a fresh warmup.
2021-07-14 22:59:53 +03:00
vyzo
818b8de182
keep track of the compaction serial (index)
...
it is useful so that:
- we only do slow (but very effective) moving gc every 10 compactions
- we can detect a splitstore v0 upgrade and re-warm up
2021-07-14 22:59:53 +03:00
