vyzo
ee63be26a1
fix race in protectView
2022-01-31 12:40:26 +02:00
vyzo
1900c907fd
account for missing refs in the markset in Has
2022-01-31 09:44:51 +02:00
vyzo
a9d4495d83
use both hot and cold when doing fetches for markset positive objects
2022-01-30 22:47:20 +02:00
vyzo
7931f1f8f9
fix lint
2022-01-30 15:43:52 +02:00
vyzo
dbc8903bac
sortless compaction
2022-01-30 15:33:15 +02:00
vyzo
25284b5325
refactor: eliminate distinction between markset and markset visitors
2022-01-25 16:31:45 +02:00
Aayush Rajasekaran
dfb65ed89f
Plumb contexts through
2021-12-11 17:04:00 -05:00
vyzo
489782e21b
satisfy the spellchecker that masquarades as a linter
2021-12-03 12:15:28 +02:00
vyzo
05c1325885
add logic for supressing compaction near upgrade boundaries
2021-12-03 11:50:35 +02:00
vyzo
742c85bf81
add SupportsVisitor in the markset env interface
2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6
widen MarkSetVisitor interface and get rid of the casts
2021-08-10 10:47:01 +03:00
vyzo
9d25464703
use functional options for hotstore gc, rename MovingGC to FullGC
2021-07-27 09:53:22 +03:00
vyzo
5acae50e07
add config option for splitstore moving gc frequency
2021-07-26 15:05:57 +03:00
vyzo
fb3986226f
do hotstore moving GC in splitstore with a user-specified frequency
2021-07-26 15:05:57 +03:00
vyzo
1918ffda4e
implement splitstore check
2021-07-25 10:35:37 +03:00
vyzo
12c3432b8d
document the "badger" markset type option
2021-07-23 12:47:18 +03:00
Łukasz Magiera
da5aeda197
Merge branch 'master' into feat/splitstore-hot-messages
2021-07-22 12:58:06 +02:00
vyzo
ebbaf23af8
support out-of-chain reference protection
2021-07-20 09:02:40 +03:00
vyzo
1b77361301
add option for hotstore message retention
2021-07-17 08:35:35 +03:00
vyzo
5a23f64b3b
code reorg: break splitstore.go into smaller logical units
2021-07-14 13:11:15 -07:00
vyzo
023146803d
use Broadcast for view barrier
2021-07-14 22:59:53 +03:00
vyzo
3d77ae1f4d
make trackTxnRefMany consistent with trackTxnRef
2021-07-14 22:59:53 +03:00
vyzo
6f126c80bf
remove redundant log, more descriptive error message for closing condition
2021-07-14 22:59:53 +03:00
vyzo
ff093fae00
use a missing compactionIndex as an indicator for warmup
...
so that splitstore v0 nodes upgrading will get a fresh warmup.
2021-07-14 22:59:53 +03:00
vyzo
818b8de182
keep track of the compaction serial (index)
...
it is useful so that:
- we only do slow (but very effective) moving gc every 10 compactions
- we can detect a splitstore v0 upgrade and re-warm up
2021-07-14 22:59:53 +03:00
vyzo
c93328b036
use the new traits for hotstore gc
2021-07-14 22:59:52 +03:00
vyzo
af399529ec
finetune view waiting
2021-07-13 09:06:40 +03:00
vyzo
257423e917
fix view waiting issues with the WaitGroup
...
We can add after Wait is called, which is problematic with WaitGroups.
This instead uses a mx/cond combo and waits while the count is > 0.
The only downside is that we might needlessly wait for (a bunch) of views
that started while the txn is active, but we can live with that.
2021-07-13 09:01:50 +03:00
Steven Allen
04abd190ab
nit: remove useless goto
...
Because stebalien has allergies.
2021-07-12 21:46:50 -07:00
vyzo
60212c86cb
put a mutex around HeadChange
2021-07-13 03:14:13 +03:00
vyzo
759594d01c
always return the waitgroup in protectView
...
so that we preclude the following scenario:
Start compaction.
Start view.
Finish compaction.
Start compaction.
which would not wait for the view to complete.
2021-07-13 03:11:40 +03:00
vyzo
df9670c58d
fix lint
2021-07-10 16:38:40 +03:00
vyzo
0c5e336ff1
address review comments
2021-07-10 16:30:27 +03:00
vyzo
870a47f55d
handle id cids in internal versions of view/get
2021-07-09 20:07:17 +03:00
vyzo
18161fee38
remove unused lookback constructs
2021-07-09 19:12:58 +03:00
vyzo
095d7427ba
make view protection optimistic again, as there is a race window
2021-07-09 15:41:10 +03:00
vyzo
da0feb3fa4
dont mark references inline; instad rely on the main compaction thread to do concurrent marking
...
The problem is that it is possible that an inline marking might take minutes for some objects
(infrequent, but still possible for state roots and prohibitive if that's a block validation).
So we simply track references continuously and rely on the main compaction thread to trigger
concurrent marking for all references at opportune moments.
Assumption: we can mark references faster than they are created during purge or else we'll
never purge anything.
2021-07-09 15:10:02 +03:00
vyzo
acc4c374ef
properly handle protecting long-running views
2021-07-09 13:20:18 +03:00
vyzo
4f89d260b0
kill isOldBlockHeader; it's dangerous.
2021-07-09 11:35:10 +03:00
vyzo
de5e21bf1a
correctly handle identity cids
2021-07-09 11:31:04 +03:00
vyzo
abdf4a161a
explicitly switch marksets for concurrent marking
...
this has very noticeable impact in initial marking time; it also allows us
to get rid of the confusing ts monikers.
2021-07-09 04:26:36 +03:00
vyzo
b6611125b6
add environment variables to turn on the debug log without recompiling
2021-07-08 21:30:39 +03:00
vyzo
60dd97c7fc
fix potential deadlock in View
...
As pointed out by magik, it is possible to deadlock if the view callback performs
a blockstore operation while a Lock is pending.
This fixes the issue by optimistically tracking the reference before actually calling
the underlying View and limiting the scope of the lock.
2021-07-08 21:18:59 +03:00
vyzo
c0537848b3
fix typo
...
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-07-08 17:54:16 +03:00
vyzo
fa30ac8c5d
fix typo
...
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-07-08 17:53:59 +03:00
vyzo
00d7772f57
move check for closure in walkChain
...
so that we don't do it too often and also cover warmup.
2021-07-08 13:12:19 +03:00
vyzo
f5c45bd517
check the closing state variable often
...
so that we have a reasonably quick graceful shutdown
2021-07-08 10:13:44 +03:00
vyzo
fee50b13a2
check the closing state on each batch during the purge.
2021-07-07 21:32:05 +03:00
vyzo
451ddf50ab
RIP bbolt-backed markset
2021-07-07 16:39:37 +03:00
vyzo
9dbb2e0abd
don't leak tracking errors through the API
2021-07-07 16:34:02 +03:00
