zenground0
bd0c010be6
Configur buffer and threshold
2023-03-09 08:40:14 -07:00
zenground0
ee9ff563d2
computing szPurge leads to deadlock and unneeded, remove
2023-03-08 17:51:59 -07:00
zenground0
a994153e27
GC respects target for max hotstore space
2023-03-07 07:39:17 -07:00
zenground0
5d9739a863
Track size of dags relevant to compaction
2023-03-07 07:39:17 -07:00
Jorropo
f572852d06
chore: all: bump go-libipfs to replace go-block-format
...
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58
2023-01-26 17:03:18 +01:00
ZenGround0
4ffded6fef
feat:splitstore:single compaction that can handle prune aka two marksets one compaction ( #9571 )
...
* begin
* rough draft -- this should probably actually work?
* WIP
* Start testing
* message mode
* Fix tests, make gen
* Better default
* docsgen-cli
* Review Response
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-11-07 16:31:12 -05:00
ZenGround0
8b7be6d47e
feat:chain:splitstore auto prune ( #9123 )
...
Auto Prune
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-08-08 16:06:32 -04:00
ZenGround0
0c91b0dc10
feat:chain:splitstore chain prune ( #9056 )
...
* Splitstore chain prune
* Protect on reification for simpler logic and sound cold compact protect
* Recovery from checkpoint during chain prune
* Splitstore (discard and universal mode) running in itests
* Add pause and restart functions to itest block miner
* Add config options to itest full nodes
* Add FsRepo support for itest full ndoes
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-08-05 16:34:16 -04:00
Masih H. Derkani
6f686cae11
Upgrade to go-blockstore 1.2.0 and go-cid 0.2.0
...
Upgrade required dependencies and fix breaking changes.
2022-06-29 14:55:22 -04:00
Łukasz Magiera
05cdeb80c3
chore: remove redundant import prefixes
2022-06-15 12:06:22 +02:00
Łukasz Magiera
e65fae28de
chore: fix imports
2022-06-14 17:00:51 +02:00
vyzo
4524fbe936
wait for reify workers to finish when closing
2022-02-14 16:10:54 +02:00
vyzo
b576785aac
rename GetHotView to IsHotView
2022-02-14 16:03:17 +02:00
vyzo
268366e446
cold object reification context option
2022-02-04 16:07:58 +02:00
vyzo
4cddfd1074
background cold object reification
2022-02-04 14:29:50 +02:00
vyzo
b13aa8f172
unblock waitForSync on close
2022-02-01 11:30:21 +02:00
vyzo
7b4ab2077b
wait for sync in a non racey way
2022-02-01 11:10:51 +02:00
vyzo
3aabb0314a
synchronously mark live refs on put/putmany
2022-02-01 09:07:14 +02:00
vyzo
710fda49f0
fix putmany marking
2022-01-31 17:35:30 +02:00
vyzo
2b14bda6b8
recursively mark puts during the critical section
2022-01-31 17:26:08 +02:00
vyzo
ee63be26a1
fix race in protectView
2022-01-31 12:40:26 +02:00
vyzo
1900c907fd
account for missing refs in the markset in Has
2022-01-31 09:44:51 +02:00
vyzo
a9d4495d83
use both hot and cold when doing fetches for markset positive objects
2022-01-30 22:47:20 +02:00
vyzo
7931f1f8f9
fix lint
2022-01-30 15:43:52 +02:00
vyzo
dbc8903bac
sortless compaction
2022-01-30 15:33:15 +02:00
vyzo
25284b5325
refactor: eliminate distinction between markset and markset visitors
2022-01-25 16:31:45 +02:00
Aayush Rajasekaran
dfb65ed89f
Plumb contexts through
2021-12-11 17:04:00 -05:00
vyzo
489782e21b
satisfy the spellchecker that masquarades as a linter
2021-12-03 12:15:28 +02:00
vyzo
05c1325885
add logic for supressing compaction near upgrade boundaries
2021-12-03 11:50:35 +02:00
vyzo
742c85bf81
add SupportsVisitor in the markset env interface
2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6
widen MarkSetVisitor interface and get rid of the casts
2021-08-10 10:47:01 +03:00
vyzo
9d25464703
use functional options for hotstore gc, rename MovingGC to FullGC
2021-07-27 09:53:22 +03:00
vyzo
5acae50e07
add config option for splitstore moving gc frequency
2021-07-26 15:05:57 +03:00
vyzo
fb3986226f
do hotstore moving GC in splitstore with a user-specified frequency
2021-07-26 15:05:57 +03:00
vyzo
1918ffda4e
implement splitstore check
2021-07-25 10:35:37 +03:00
vyzo
12c3432b8d
document the "badger" markset type option
2021-07-23 12:47:18 +03:00
Łukasz Magiera
da5aeda197
Merge branch 'master' into feat/splitstore-hot-messages
2021-07-22 12:58:06 +02:00
vyzo
ebbaf23af8
support out-of-chain reference protection
2021-07-20 09:02:40 +03:00
vyzo
1b77361301
add option for hotstore message retention
2021-07-17 08:35:35 +03:00
vyzo
5a23f64b3b
code reorg: break splitstore.go into smaller logical units
2021-07-14 13:11:15 -07:00
vyzo
023146803d
use Broadcast for view barrier
2021-07-14 22:59:53 +03:00
vyzo
3d77ae1f4d
make trackTxnRefMany consistent with trackTxnRef
2021-07-14 22:59:53 +03:00
vyzo
6f126c80bf
remove redundant log, more descriptive error message for closing condition
2021-07-14 22:59:53 +03:00
vyzo
ff093fae00
use a missing compactionIndex as an indicator for warmup
...
so that splitstore v0 nodes upgrading will get a fresh warmup.
2021-07-14 22:59:53 +03:00
vyzo
818b8de182
keep track of the compaction serial (index)
...
it is useful so that:
- we only do slow (but very effective) moving gc every 10 compactions
- we can detect a splitstore v0 upgrade and re-warm up
2021-07-14 22:59:53 +03:00
vyzo
c93328b036
use the new traits for hotstore gc
2021-07-14 22:59:52 +03:00
vyzo
af399529ec
finetune view waiting
2021-07-13 09:06:40 +03:00
vyzo
257423e917
fix view waiting issues with the WaitGroup
...
We can add after Wait is called, which is problematic with WaitGroups.
This instead uses a mx/cond combo and waits while the count is > 0.
The only downside is that we might needlessly wait for (a bunch) of views
that started while the txn is active, but we can live with that.
2021-07-13 09:01:50 +03:00
Steven Allen
04abd190ab
nit: remove useless goto
...
Because stebalien has allergies.
2021-07-12 21:46:50 -07:00
vyzo
60212c86cb
put a mutex around HeadChange
2021-07-13 03:14:13 +03:00
