vyzo
742c85bf81
add SupportsVisitor in the markset env interface
2021-08-10 10:47:01 +03:00
vyzo
79f348a01c
make the write lock scope limited within a function
...
Also avoid removing the writing set if there was an error while writing.
2021-08-10 10:47:01 +03:00
vyzo
a9403b42f0
deduplicate put code
2021-08-10 10:47:01 +03:00
vyzo
1a59b73adc
add .tmp suffix to transient db names
2021-08-10 10:47:01 +03:00
vyzo
26a5832f92
simplify and deduplicate Has/Visit using helper methods tryPending and tryDB.
2021-08-10 10:47:01 +03:00
vyzo
380e16d465
remove redundant writers state variable
2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6
widen MarkSetVisitor interface and get rid of the casts
2021-08-10 10:47:01 +03:00
vyzo
57c984cea1
improve concurrency properties of Visit with optimistic concurrency
2021-08-10 10:47:01 +03:00
vyzo
563fa1e31a
take advantage of MarkSet/Visitor unification to atomically check+mark
2021-08-10 10:47:01 +03:00
vyzo
cb3c53664d
unify marksets and visitors
2021-08-10 10:47:01 +03:00
vyzo
3c994d94aa
deduplicate some code in markset_badger
2021-08-10 10:47:01 +03:00
vyzo
32d94d4e85
reset walked set as epoch boundaries are crossed
...
the walk is BFS, so we can do this!
2021-08-10 10:47:01 +03:00
vyzo
49346f5679
use visitors instead of cidsets in walks
2021-08-10 10:47:01 +03:00
vyzo
1323d8fb20
markset-backed visitors
2021-08-10 10:47:01 +03:00
vyzo
6f22cffb6b
object visitor interface
2021-08-10 10:47:01 +03:00
vyzo
a768dd84fa
document splitstore clear shed command
2021-08-01 19:18:34 +03:00
vyzo
5e8b2c7860
make symlink helper freestanding
2021-07-29 08:35:53 +03:00
vyzo
b75ff37448
fix typo
...
Co-authored-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-07-28 17:11:04 +03:00
vyzo
c65d72fbca
fix format specifier
2021-07-28 16:20:25 +03:00
vyzo
fff1c0ae57
improve detection of relative links
2021-07-28 16:15:39 +03:00
vyzo
297c9e2f7a
extend test to check the validity of relative links
2021-07-28 11:56:35 +03:00
vyzo
0740274b7c
make relative links when the canonical and new db paths are in the same directory
2021-07-28 11:56:23 +03:00
vyzo
950ded454d
code cosmetics: rename variables for better readability and some comments
2021-07-28 11:49:42 +03:00
vyzo
da3e36acdc
Merge branch 'master' into feat/moving-gc
2021-07-27 12:12:38 +03:00
vyzo
8809707158
update README
2021-07-27 12:08:39 +03:00
vyzo
c21c413695
remove db2 to dbNext
2021-07-27 12:06:40 +03:00
vyzo
bb2d99908c
make state constants typed
2021-07-27 12:03:26 +03:00
vyzo
b82f953fd5
use the badger streaming interface in doCopy
2021-07-27 11:46:35 +03:00
vyzo
59aebba0d9
use a slab allocator for the copy
2021-07-27 11:35:21 +03:00
vyzo
0baeec0686
remove filter from doCopy; it's not used
...
it was there to support a potential CopyTo interface; but we'll cross that bridge when we get there.
2021-07-27 11:18:24 +03:00
vyzo
649fc62863
panic if we fail to correctly setup the db paths.
...
we can't really continue and leave a ticking bomb for the next restart; the user might not see it.
2021-07-27 10:27:16 +03:00
vyzo
d6ace68540
extend test to do a double move and check symlink following
2021-07-27 10:16:50 +03:00
vyzo
cbaffab9dd
use EvalSymlinks in deleteDB
2021-07-27 10:08:07 +03:00
vyzo
c03859c1b5
resolve symlinks when constructing the new db path
...
so that the new path is adjacent to the old path, allowing the user to symlink the db in
a different file system.
2021-07-27 10:05:35 +03:00
vyzo
9d25464703
use functional options for hotstore gc, rename MovingGC to FullGC
2021-07-27 09:53:22 +03:00
vyzo
96c1123c33
use functional options in the BlockstoreGC interface
2021-07-27 09:52:06 +03:00
vyzo
c52c2738ab
fix potential underflow
2021-07-26 18:38:42 +03:00
vyzo
18caefc4a1
fix missing object condition; short-circuit walk
2021-07-26 18:34:05 +03:00
vyzo
0fd8fc24e1
load a full finality worth of state during warmup
2021-07-26 18:34:05 +03:00
vyzo
938330e6c3
moveMx is not an RWMutex, just a regular mutex
2021-07-26 15:09:31 +03:00
vyzo
21e7c188da
use CollectGarbage in blockstore move test, as it is the real interface
2021-07-26 15:05:57 +03:00
vyzo
b1f60e85e9
document moving GC frequency option in README
2021-07-26 15:05:57 +03:00
vyzo
5acae50e07
add config option for splitstore moving gc frequency
2021-07-26 15:05:57 +03:00
vyzo
fb3986226f
do hotstore moving GC in splitstore with a user-specified frequency
2021-07-26 15:05:57 +03:00
vyzo
c747f2f1e2
do moving GC if the user asks for it
2021-07-26 15:05:57 +03:00
vyzo
a843665132
add options to BlockstoreGC trait
2021-07-26 15:05:57 +03:00
vyzo
94509968a0
make moveTo a private method
2021-07-26 15:05:20 +03:00
vyzo
5cf6fdf81d
don't heap allocate the cond, just set L
2021-07-26 15:04:46 +03:00
vyzo
524564e2cf
add some more logging around move
2021-07-26 15:04:46 +03:00
vyzo
f2c7b08be5
follow symbolic links when deleting old dbs
2021-07-26 15:04:46 +03:00
vyzo
608a9f84d2
fix copy: flush discards the transaction
2021-07-26 15:04:46 +03:00
vyzo
4b0b37a4ef
fix lint
...
the great spellchecker strikes again
2021-07-26 15:04:46 +03:00
vyzo
4715b1f436
fix lotus-shed
2021-07-26 15:04:46 +03:00
vyzo
aec126879e
add MoveTo test
2021-07-26 15:04:46 +03:00
vyzo
001c04f2dd
use pooled slices for the copy
2021-07-26 15:04:46 +03:00
vyzo
b741d61b20
implement BlockstoreMover in badger
2021-07-26 15:04:46 +03:00
vyzo
2cfd73c879
manually compute size when badger is being stupid
2021-07-26 09:46:21 +03:00
vyzo
74009bd67f
document lotus-shed splitstore utiilities in the README
2021-07-26 08:52:32 +03:00
vyzo
ce528a1293
implement Info in splitstore
2021-07-26 08:45:46 +03:00
vyzo
30e4b405b7
implement BlockstoreSize for badger
2021-07-26 08:43:09 +03:00
vyzo
baaa9a7738
add BlockstoreSize trait for reporting size
2021-07-26 08:42:54 +03:00
vyzo
c00b86e8a8
stop the walk on missing references
2021-07-25 13:42:20 +03:00
vyzo
5285a14d27
write check summary at the end
2021-07-25 11:42:13 +03:00
vyzo
5718da237f
handle newlines consistently in check output
2021-07-25 11:30:58 +03:00
vyzo
1918ffda4e
implement splitstore check
2021-07-25 10:35:37 +03:00
vyzo
d6ebead756
add comment about the necessity of FileIO
2021-07-23 16:30:55 +03:00
vyzo
1480c22447
improve README
2021-07-23 16:27:09 +03:00
vyzo
9faa3d5fd7
adjust compaction test with badger to hit the db
2021-07-23 12:47:18 +03:00
vyzo
9d5cf4177b
add splitstore compaction test with badger markset
2021-07-23 12:47:18 +03:00
vyzo
dec5e13c85
update README with configuration options
2021-07-23 12:47:18 +03:00
vyzo
a8c0fd1654
tweak badger options
2021-07-23 12:47:18 +03:00
vyzo
4df6073382
some tweaks in badger markset
2021-07-23 12:47:18 +03:00
vyzo
f492f8d450
track in flight writes in badger markste, as they happen outside the lock
2021-07-23 12:47:18 +03:00
vyzo
f75d982c76
remove early occurs check from trackTxnRef
...
this happens inline, and it might block when using the badger markset
2021-07-23 12:47:18 +03:00
vyzo
2891a31c99
make badger markset concurrent close safe
2021-07-23 12:47:18 +03:00
vyzo
f2b7c3e6f2
reduce scope of exclusive lock in badger markset
2021-07-23 12:47:18 +03:00
vyzo
12c3432b8d
document the "badger" markset type option
2021-07-23 12:47:18 +03:00
vyzo
be9530b66e
finetune badger options
2021-07-23 12:47:18 +03:00
vyzo
2c26abc551
add badger markset test
2021-07-23 12:47:18 +03:00
vyzo
54a6968327
add badger-backed markset implementation
2021-07-23 12:47:18 +03:00
vyzo
5266b240b8
coalesce message and message receipt retention
2021-07-22 22:02:29 +03:00
vyzo
2a68ae8dad
fix warmup by decoupling state from message receipt walk
2021-07-22 20:49:36 +03:00
Łukasz Magiera
da5aeda197
Merge branch 'master' into feat/splitstore-hot-messages
2021-07-22 12:58:06 +02:00
vyzo
c47fce8d2d
test protector support
2021-07-20 09:02:45 +03:00
vyzo
ebbaf23af8
support out-of-chain reference protection
2021-07-20 09:02:40 +03:00
vyzo
006050ed27
implement hotstore message retention policy
2021-07-17 08:59:43 +03:00
vyzo
1b77361301
add option for hotstore message retention
2021-07-17 08:35:35 +03:00
vyzo
e003203bea
implement exposed splitstore
2021-07-15 13:12:10 +03:00
vyzo
5a23f64b3b
code reorg: break splitstore.go into smaller logical units
2021-07-14 13:11:15 -07:00
vyzo
3f3a12b75c
remove BlockstoreMover interface
...
we decided it's premature
2021-07-14 22:59:53 +03:00
vyzo
023146803d
use Broadcast for view barrier
2021-07-14 22:59:53 +03:00
vyzo
3d77ae1f4d
make trackTxnRefMany consistent with trackTxnRef
2021-07-14 22:59:53 +03:00
vyzo
6f126c80bf
remove redundant log, more descriptive error message for closing condition
2021-07-14 22:59:53 +03:00
vyzo
ff093fae00
use a missing compactionIndex as an indicator for warmup
...
so that splitstore v0 nodes upgrading will get a fresh warmup.
2021-07-14 22:59:53 +03:00
vyzo
669b47cfc9
do moving gc for hotstore every 20 compactions
...
that's about once a week
2021-07-14 22:59:53 +03:00
vyzo
818b8de182
keep track of the compaction serial (index)
...
it is useful so that:
- we only do slow (but very effective) moving gc every 10 compactions
- we can detect a splitstore v0 upgrade and re-warm up
2021-07-14 22:59:53 +03:00
vyzo
c93328b036
use the new traits for hotstore gc
2021-07-14 22:59:52 +03:00
vyzo
35180b4761
merge Compact and CollectGarbage in badger
2021-07-14 22:59:52 +03:00
vyzo
dc81c0e6a2
add blockstore traits related to gc
2021-07-14 22:59:52 +03:00
vyzo
af399529ec
finetune view waiting
2021-07-13 09:06:40 +03:00
vyzo
257423e917
fix view waiting issues with the WaitGroup
...
We can add after Wait is called, which is problematic with WaitGroups.
This instead uses a mx/cond combo and waits while the count is > 0.
The only downside is that we might needlessly wait for (a bunch) of views
that started while the txn is active, but we can live with that.
2021-07-13 09:01:50 +03:00
Steven Allen
04abd190ab
nit: remove useless goto
...
Because stebalien has allergies.
2021-07-12 21:46:50 -07:00
vyzo
60212c86cb
put a mutex around HeadChange
2021-07-13 03:14:13 +03:00
vyzo
759594d01c
always return the waitgroup in protectView
...
so that we preclude the following scenario:
Start compaction.
Start view.
Finish compaction.
Start compaction.
which would not wait for the view to complete.
2021-07-13 03:11:40 +03:00
vyzo
df9670c58d
fix lint
2021-07-10 16:38:40 +03:00
vyzo
0c5e336ff1
address review comments
2021-07-10 16:30:27 +03:00
vyzo
870a47f55d
handle id cids in internal versions of view/get
2021-07-09 20:07:17 +03:00
vyzo
f5ae10e3d1
refactor debug log code to eliminate duplication
2021-07-09 19:53:51 +03:00
vyzo
41290383e2
fix test
2021-07-09 19:24:44 +03:00
vyzo
b9a5ea8f7b
update wording around discard store
2021-07-09 19:23:55 +03:00
vyzo
c0a1cfffa1
rename noopstore to discardstore
2021-07-09 19:19:37 +03:00
vyzo
18161fee38
remove unused lookback constructs
2021-07-09 19:12:58 +03:00
vyzo
095d7427ba
make view protection optimistic again, as there is a race window
2021-07-09 15:41:10 +03:00
vyzo
da0feb3fa4
dont mark references inline; instad rely on the main compaction thread to do concurrent marking
...
The problem is that it is possible that an inline marking might take minutes for some objects
(infrequent, but still possible for state roots and prohibitive if that's a block validation).
So we simply track references continuously and rely on the main compaction thread to trigger
concurrent marking for all references at opportune moments.
Assumption: we can mark references faster than they are created during purge or else we'll
never purge anything.
2021-07-09 15:10:02 +03:00
vyzo
acc4c374ef
properly handle protecting long-running views
2021-07-09 13:20:18 +03:00
vyzo
565faff754
fix test
2021-07-09 11:38:09 +03:00
vyzo
4f89d260b0
kill isOldBlockHeader; it's dangerous.
2021-07-09 11:35:10 +03:00
vyzo
de5e21bf1a
correctly handle identity cids
2021-07-09 11:31:04 +03:00
vyzo
909f7039d4
make badger Close-safe
2021-07-09 09:54:12 +03:00
vyzo
abdf4a161a
explicitly switch marksets for concurrent marking
...
this has very noticeable impact in initial marking time; it also allows us
to get rid of the confusing ts monikers.
2021-07-09 04:26:36 +03:00
vyzo
b6611125b6
add environment variables to turn on the debug log without recompiling
2021-07-08 21:30:39 +03:00
vyzo
60dd97c7fc
fix potential deadlock in View
...
As pointed out by magik, it is possible to deadlock if the view callback performs
a blockstore operation while a Lock is pending.
This fixes the issue by optimistically tracking the reference before actually calling
the underlying View and limiting the scope of the lock.
2021-07-08 21:18:59 +03:00
vyzo
c0537848b3
fix typo
...
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-07-08 17:54:16 +03:00
vyzo
fa30ac8c5d
fix typo
...
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-07-08 17:53:59 +03:00
vyzo
00d7772f57
move check for closure in walkChain
...
so that we don't do it too often and also cover warmup.
2021-07-08 13:12:19 +03:00
vyzo
5cf1e09e81
README: add instructions for how to enable
2021-07-08 13:00:31 +03:00
vyzo
9aa4f3b3b2
add README for documentation
2021-07-08 12:32:41 +03:00
vyzo
e6eacbdd56
use RW mutexes in marksets
2021-07-08 10:20:29 +03:00
vyzo
48f13a43b7
intelligently close marksets and signal errors in concurrent operations
2021-07-08 10:18:43 +03:00
vyzo
f5c45bd517
check the closing state variable often
...
so that we have a reasonably quick graceful shutdown
2021-07-08 10:13:44 +03:00
vyzo
4f808367f8
fix lint
2021-07-07 21:32:58 +03:00
vyzo
fee50b13a2
check the closing state on each batch during the purge.
2021-07-07 21:32:05 +03:00
vyzo
c6421f8a75
don't nil the mark sets on close, it's dangerous.
...
a concurrent marking can panic.
2021-07-07 21:27:36 +03:00
vyzo
aec2ba2c82
nil map/bf on markset close
2021-07-07 16:46:14 +03:00
vyzo
451ddf50ab
RIP bbolt-backed markset
2021-07-07 16:39:37 +03:00
vyzo
9dbb2e0abd
don't leak tracking errors through the API
2021-07-07 16:34:02 +03:00
vyzo
83c30dc4c0
protect assignment of warmup epoch with the mutex
2021-07-07 11:31:27 +03:00
vyzo
6cc2112749
remove the curTs state variable; we don't need it
2021-07-07 09:55:25 +03:00
vyzo
05dbbe9681
rename som Txn methods for better readability
2021-07-07 09:52:31 +03:00
vyzo
90da6227b3
transactional protect incoming tipsets
2021-07-07 02:11:37 +03:00
vyzo
0e2af11f6a
prepare the transaction before launching the compaction goroutine
2021-07-07 01:39:58 +03:00
vyzo
f2f4af669d
clean up: simplify debug log, get rid of ugly debug log
2021-07-06 17:13:38 +03:00
vyzo
c1c25868cc
improve comments
2021-07-06 15:09:04 +03:00
vyzo
fdff1bebc9
move map markset implementation to its own file
2021-07-06 14:44:40 +03:00
vyzo
5c514504f7
remove unused GetGenesis method from ChainAccessor interface
2021-07-06 14:41:41 +03:00
vyzo
dc8139a1d2
add some comments for debug only code
2021-07-06 13:23:12 +03:00
vyzo
c4ae3e0c3d
minor tweak
2021-07-06 09:17:35 +03:00
vyzo
169ab262f5
really optimize computing object weights
...
sort is still taking a long time, this should be as fast as it gets.
2021-07-06 09:02:44 +03:00
vyzo
55a9e0ccd1
short-circuit block headers on sort weight computation
2021-07-06 08:22:43 +03:00
vyzo
bf7aeb3167
optimize sort a tad
...
it's taking a long time to compute weights...
2021-07-06 08:10:57 +03:00