vyzo
49346f5679
use visitors instead of cidsets in walks
2021-08-10 10:47:01 +03:00
vyzo
1323d8fb20
markset-backed visitors
2021-08-10 10:47:01 +03:00
vyzo
6f22cffb6b
object visitor interface
2021-08-10 10:47:01 +03:00
vyzo
a768dd84fa
document splitstore clear shed command
2021-08-01 19:18:34 +03:00
vyzo
5e8b2c7860
make symlink helper freestanding
2021-07-29 08:35:53 +03:00
vyzo
b75ff37448
fix typo
...
Co-authored-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-07-28 17:11:04 +03:00
vyzo
c65d72fbca
fix format specifier
2021-07-28 16:20:25 +03:00
vyzo
fff1c0ae57
improve detection of relative links
2021-07-28 16:15:39 +03:00
vyzo
297c9e2f7a
extend test to check the validity of relative links
2021-07-28 11:56:35 +03:00
vyzo
0740274b7c
make relative links when the canonical and new db paths are in the same directory
2021-07-28 11:56:23 +03:00
vyzo
950ded454d
code cosmetics: rename variables for better readability and some comments
2021-07-28 11:49:42 +03:00
vyzo
da3e36acdc
Merge branch 'master' into feat/moving-gc
2021-07-27 12:12:38 +03:00
vyzo
8809707158
update README
2021-07-27 12:08:39 +03:00
vyzo
c21c413695
remove db2 to dbNext
2021-07-27 12:06:40 +03:00
vyzo
bb2d99908c
make state constants typed
2021-07-27 12:03:26 +03:00
vyzo
b82f953fd5
use the badger streaming interface in doCopy
2021-07-27 11:46:35 +03:00
vyzo
59aebba0d9
use a slab allocator for the copy
2021-07-27 11:35:21 +03:00
vyzo
0baeec0686
remove filter from doCopy; it's not used
...
it was there to support a potential CopyTo interface; but we'll cross that bridge when we get there.
2021-07-27 11:18:24 +03:00
vyzo
649fc62863
panic if we fail to correctly setup the db paths.
...
we can't really continue and leave a ticking bomb for the next restart; the user might not see it.
2021-07-27 10:27:16 +03:00
vyzo
d6ace68540
extend test to do a double move and check symlink following
2021-07-27 10:16:50 +03:00
vyzo
cbaffab9dd
use EvalSymlinks in deleteDB
2021-07-27 10:08:07 +03:00
vyzo
c03859c1b5
resolve symlinks when constructing the new db path
...
so that the new path is adjacent to the old path, allowing the user to symlink the db in
a different file system.
2021-07-27 10:05:35 +03:00
vyzo
9d25464703
use functional options for hotstore gc, rename MovingGC to FullGC
2021-07-27 09:53:22 +03:00
vyzo
96c1123c33
use functional options in the BlockstoreGC interface
2021-07-27 09:52:06 +03:00
vyzo
c52c2738ab
fix potential underflow
2021-07-26 18:38:42 +03:00
vyzo
18caefc4a1
fix missing object condition; short-circuit walk
2021-07-26 18:34:05 +03:00
vyzo
0fd8fc24e1
load a full finality worth of state during warmup
2021-07-26 18:34:05 +03:00
vyzo
938330e6c3
moveMx is not an RWMutex, just a regular mutex
2021-07-26 15:09:31 +03:00
vyzo
21e7c188da
use CollectGarbage in blockstore move test, as it is the real interface
2021-07-26 15:05:57 +03:00
vyzo
b1f60e85e9
document moving GC frequency option in README
2021-07-26 15:05:57 +03:00
vyzo
5acae50e07
add config option for splitstore moving gc frequency
2021-07-26 15:05:57 +03:00
vyzo
fb3986226f
do hotstore moving GC in splitstore with a user-specified frequency
2021-07-26 15:05:57 +03:00
vyzo
c747f2f1e2
do moving GC if the user asks for it
2021-07-26 15:05:57 +03:00
vyzo
a843665132
add options to BlockstoreGC trait
2021-07-26 15:05:57 +03:00
vyzo
94509968a0
make moveTo a private method
2021-07-26 15:05:20 +03:00
vyzo
5cf6fdf81d
don't heap allocate the cond, just set L
2021-07-26 15:04:46 +03:00
vyzo
524564e2cf
add some more logging around move
2021-07-26 15:04:46 +03:00
vyzo
f2c7b08be5
follow symbolic links when deleting old dbs
2021-07-26 15:04:46 +03:00
vyzo
608a9f84d2
fix copy: flush discards the transaction
2021-07-26 15:04:46 +03:00
vyzo
4b0b37a4ef
fix lint
...
the great spellchecker strikes again
2021-07-26 15:04:46 +03:00
vyzo
4715b1f436
fix lotus-shed
2021-07-26 15:04:46 +03:00
vyzo
aec126879e
add MoveTo test
2021-07-26 15:04:46 +03:00
vyzo
001c04f2dd
use pooled slices for the copy
2021-07-26 15:04:46 +03:00
vyzo
b741d61b20
implement BlockstoreMover in badger
2021-07-26 15:04:46 +03:00
vyzo
2cfd73c879
manually compute size when badger is being stupid
2021-07-26 09:46:21 +03:00
vyzo
74009bd67f
document lotus-shed splitstore utiilities in the README
2021-07-26 08:52:32 +03:00
vyzo
ce528a1293
implement Info in splitstore
2021-07-26 08:45:46 +03:00
vyzo
30e4b405b7
implement BlockstoreSize for badger
2021-07-26 08:43:09 +03:00
vyzo
baaa9a7738
add BlockstoreSize trait for reporting size
2021-07-26 08:42:54 +03:00
vyzo
c00b86e8a8
stop the walk on missing references
2021-07-25 13:42:20 +03:00
vyzo
5285a14d27
write check summary at the end
2021-07-25 11:42:13 +03:00
vyzo
5718da237f
handle newlines consistently in check output
2021-07-25 11:30:58 +03:00
vyzo
1918ffda4e
implement splitstore check
2021-07-25 10:35:37 +03:00
vyzo
d6ebead756
add comment about the necessity of FileIO
2021-07-23 16:30:55 +03:00
vyzo
1480c22447
improve README
2021-07-23 16:27:09 +03:00
vyzo
9faa3d5fd7
adjust compaction test with badger to hit the db
2021-07-23 12:47:18 +03:00
vyzo
9d5cf4177b
add splitstore compaction test with badger markset
2021-07-23 12:47:18 +03:00
vyzo
dec5e13c85
update README with configuration options
2021-07-23 12:47:18 +03:00
vyzo
a8c0fd1654
tweak badger options
2021-07-23 12:47:18 +03:00
vyzo
4df6073382
some tweaks in badger markset
2021-07-23 12:47:18 +03:00
vyzo
f492f8d450
track in flight writes in badger markste, as they happen outside the lock
2021-07-23 12:47:18 +03:00
vyzo
f75d982c76
remove early occurs check from trackTxnRef
...
this happens inline, and it might block when using the badger markset
2021-07-23 12:47:18 +03:00
vyzo
2891a31c99
make badger markset concurrent close safe
2021-07-23 12:47:18 +03:00
vyzo
f2b7c3e6f2
reduce scope of exclusive lock in badger markset
2021-07-23 12:47:18 +03:00
vyzo
12c3432b8d
document the "badger" markset type option
2021-07-23 12:47:18 +03:00
vyzo
be9530b66e
finetune badger options
2021-07-23 12:47:18 +03:00
vyzo
2c26abc551
add badger markset test
2021-07-23 12:47:18 +03:00
vyzo
54a6968327
add badger-backed markset implementation
2021-07-23 12:47:18 +03:00
vyzo
5266b240b8
coalesce message and message receipt retention
2021-07-22 22:02:29 +03:00
vyzo
2a68ae8dad
fix warmup by decoupling state from message receipt walk
2021-07-22 20:49:36 +03:00
Łukasz Magiera
da5aeda197
Merge branch 'master' into feat/splitstore-hot-messages
2021-07-22 12:58:06 +02:00
vyzo
c47fce8d2d
test protector support
2021-07-20 09:02:45 +03:00
vyzo
ebbaf23af8
support out-of-chain reference protection
2021-07-20 09:02:40 +03:00
vyzo
006050ed27
implement hotstore message retention policy
2021-07-17 08:59:43 +03:00
vyzo
1b77361301
add option for hotstore message retention
2021-07-17 08:35:35 +03:00
vyzo
e003203bea
implement exposed splitstore
2021-07-15 13:12:10 +03:00
vyzo
5a23f64b3b
code reorg: break splitstore.go into smaller logical units
2021-07-14 13:11:15 -07:00
vyzo
3f3a12b75c
remove BlockstoreMover interface
...
we decided it's premature
2021-07-14 22:59:53 +03:00
vyzo
023146803d
use Broadcast for view barrier
2021-07-14 22:59:53 +03:00
vyzo
3d77ae1f4d
make trackTxnRefMany consistent with trackTxnRef
2021-07-14 22:59:53 +03:00
vyzo
6f126c80bf
remove redundant log, more descriptive error message for closing condition
2021-07-14 22:59:53 +03:00
vyzo
ff093fae00
use a missing compactionIndex as an indicator for warmup
...
so that splitstore v0 nodes upgrading will get a fresh warmup.
2021-07-14 22:59:53 +03:00
vyzo
669b47cfc9
do moving gc for hotstore every 20 compactions
...
that's about once a week
2021-07-14 22:59:53 +03:00
vyzo
818b8de182
keep track of the compaction serial (index)
...
it is useful so that:
- we only do slow (but very effective) moving gc every 10 compactions
- we can detect a splitstore v0 upgrade and re-warm up
2021-07-14 22:59:53 +03:00
vyzo
c93328b036
use the new traits for hotstore gc
2021-07-14 22:59:52 +03:00
vyzo
35180b4761
merge Compact and CollectGarbage in badger
2021-07-14 22:59:52 +03:00
vyzo
dc81c0e6a2
add blockstore traits related to gc
2021-07-14 22:59:52 +03:00
vyzo
af399529ec
finetune view waiting
2021-07-13 09:06:40 +03:00
vyzo
257423e917
fix view waiting issues with the WaitGroup
...
We can add after Wait is called, which is problematic with WaitGroups.
This instead uses a mx/cond combo and waits while the count is > 0.
The only downside is that we might needlessly wait for (a bunch) of views
that started while the txn is active, but we can live with that.
2021-07-13 09:01:50 +03:00
Steven Allen
04abd190ab
nit: remove useless goto
...
Because stebalien has allergies.
2021-07-12 21:46:50 -07:00
vyzo
60212c86cb
put a mutex around HeadChange
2021-07-13 03:14:13 +03:00
vyzo
759594d01c
always return the waitgroup in protectView
...
so that we preclude the following scenario:
Start compaction.
Start view.
Finish compaction.
Start compaction.
which would not wait for the view to complete.
2021-07-13 03:11:40 +03:00
vyzo
df9670c58d
fix lint
2021-07-10 16:38:40 +03:00
vyzo
0c5e336ff1
address review comments
2021-07-10 16:30:27 +03:00
vyzo
870a47f55d
handle id cids in internal versions of view/get
2021-07-09 20:07:17 +03:00
vyzo
f5ae10e3d1
refactor debug log code to eliminate duplication
2021-07-09 19:53:51 +03:00
vyzo
41290383e2
fix test
2021-07-09 19:24:44 +03:00
vyzo
b9a5ea8f7b
update wording around discard store
2021-07-09 19:23:55 +03:00
vyzo
c0a1cfffa1
rename noopstore to discardstore
2021-07-09 19:19:37 +03:00
vyzo
18161fee38
remove unused lookback constructs
2021-07-09 19:12:58 +03:00
