vyzo
d6ebead756
add comment about the necessity of FileIO
2021-07-23 16:30:55 +03:00
vyzo
1480c22447
improve README
2021-07-23 16:27:09 +03:00
vyzo
9faa3d5fd7
adjust compaction test with badger to hit the db
2021-07-23 12:47:18 +03:00
vyzo
9d5cf4177b
add splitstore compaction test with badger markset
2021-07-23 12:47:18 +03:00
vyzo
dec5e13c85
update README with configuration options
2021-07-23 12:47:18 +03:00
vyzo
a8c0fd1654
tweak badger options
2021-07-23 12:47:18 +03:00
vyzo
4df6073382
some tweaks in badger markset
2021-07-23 12:47:18 +03:00
vyzo
f492f8d450
track in flight writes in badger markste, as they happen outside the lock
2021-07-23 12:47:18 +03:00
vyzo
f75d982c76
remove early occurs check from trackTxnRef
...
this happens inline, and it might block when using the badger markset
2021-07-23 12:47:18 +03:00
vyzo
2891a31c99
make badger markset concurrent close safe
2021-07-23 12:47:18 +03:00
vyzo
f2b7c3e6f2
reduce scope of exclusive lock in badger markset
2021-07-23 12:47:18 +03:00
vyzo
12c3432b8d
document the "badger" markset type option
2021-07-23 12:47:18 +03:00
vyzo
be9530b66e
finetune badger options
2021-07-23 12:47:18 +03:00
vyzo
2c26abc551
add badger markset test
2021-07-23 12:47:18 +03:00
vyzo
54a6968327
add badger-backed markset implementation
2021-07-23 12:47:18 +03:00
vyzo
5266b240b8
coalesce message and message receipt retention
2021-07-22 22:02:29 +03:00
vyzo
2a68ae8dad
fix warmup by decoupling state from message receipt walk
2021-07-22 20:49:36 +03:00
Łukasz Magiera
da5aeda197
Merge branch 'master' into feat/splitstore-hot-messages
2021-07-22 12:58:06 +02:00
vyzo
c47fce8d2d
test protector support
2021-07-20 09:02:45 +03:00
vyzo
ebbaf23af8
support out-of-chain reference protection
2021-07-20 09:02:40 +03:00
vyzo
006050ed27
implement hotstore message retention policy
2021-07-17 08:59:43 +03:00
vyzo
1b77361301
add option for hotstore message retention
2021-07-17 08:35:35 +03:00
vyzo
e003203bea
implement exposed splitstore
2021-07-15 13:12:10 +03:00
vyzo
5a23f64b3b
code reorg: break splitstore.go into smaller logical units
2021-07-14 13:11:15 -07:00
vyzo
3f3a12b75c
remove BlockstoreMover interface
...
we decided it's premature
2021-07-14 22:59:53 +03:00
vyzo
023146803d
use Broadcast for view barrier
2021-07-14 22:59:53 +03:00
vyzo
3d77ae1f4d
make trackTxnRefMany consistent with trackTxnRef
2021-07-14 22:59:53 +03:00
vyzo
6f126c80bf
remove redundant log, more descriptive error message for closing condition
2021-07-14 22:59:53 +03:00
vyzo
ff093fae00
use a missing compactionIndex as an indicator for warmup
...
so that splitstore v0 nodes upgrading will get a fresh warmup.
2021-07-14 22:59:53 +03:00
vyzo
669b47cfc9
do moving gc for hotstore every 20 compactions
...
that's about once a week
2021-07-14 22:59:53 +03:00
vyzo
818b8de182
keep track of the compaction serial (index)
...
it is useful so that:
- we only do slow (but very effective) moving gc every 10 compactions
- we can detect a splitstore v0 upgrade and re-warm up
2021-07-14 22:59:53 +03:00
vyzo
c93328b036
use the new traits for hotstore gc
2021-07-14 22:59:52 +03:00
vyzo
af399529ec
finetune view waiting
2021-07-13 09:06:40 +03:00
vyzo
257423e917
fix view waiting issues with the WaitGroup
...
We can add after Wait is called, which is problematic with WaitGroups.
This instead uses a mx/cond combo and waits while the count is > 0.
The only downside is that we might needlessly wait for (a bunch) of views
that started while the txn is active, but we can live with that.
2021-07-13 09:01:50 +03:00
Steven Allen
04abd190ab
nit: remove useless goto
...
Because stebalien has allergies.
2021-07-12 21:46:50 -07:00
vyzo
60212c86cb
put a mutex around HeadChange
2021-07-13 03:14:13 +03:00
vyzo
759594d01c
always return the waitgroup in protectView
...
so that we preclude the following scenario:
Start compaction.
Start view.
Finish compaction.
Start compaction.
which would not wait for the view to complete.
2021-07-13 03:11:40 +03:00
vyzo
df9670c58d
fix lint
2021-07-10 16:38:40 +03:00
vyzo
0c5e336ff1
address review comments
2021-07-10 16:30:27 +03:00
vyzo
870a47f55d
handle id cids in internal versions of view/get
2021-07-09 20:07:17 +03:00
vyzo
f5ae10e3d1
refactor debug log code to eliminate duplication
2021-07-09 19:53:51 +03:00
vyzo
41290383e2
fix test
2021-07-09 19:24:44 +03:00
vyzo
b9a5ea8f7b
update wording around discard store
2021-07-09 19:23:55 +03:00
vyzo
18161fee38
remove unused lookback constructs
2021-07-09 19:12:58 +03:00
vyzo
095d7427ba
make view protection optimistic again, as there is a race window
2021-07-09 15:41:10 +03:00
vyzo
da0feb3fa4
dont mark references inline; instad rely on the main compaction thread to do concurrent marking
...
The problem is that it is possible that an inline marking might take minutes for some objects
(infrequent, but still possible for state roots and prohibitive if that's a block validation).
So we simply track references continuously and rely on the main compaction thread to trigger
concurrent marking for all references at opportune moments.
Assumption: we can mark references faster than they are created during purge or else we'll
never purge anything.
2021-07-09 15:10:02 +03:00
vyzo
acc4c374ef
properly handle protecting long-running views
2021-07-09 13:20:18 +03:00
vyzo
565faff754
fix test
2021-07-09 11:38:09 +03:00
vyzo
4f89d260b0
kill isOldBlockHeader; it's dangerous.
2021-07-09 11:35:10 +03:00
vyzo
de5e21bf1a
correctly handle identity cids
2021-07-09 11:31:04 +03:00
