vyzo
|
710fda49f0
|
fix putmany marking
|
2022-01-31 17:35:30 +02:00 |
|
vyzo
|
2b14bda6b8
|
recursively mark puts during the critical section
|
2022-01-31 17:26:08 +02:00 |
|
vyzo
|
1abfc5b1cf
|
fix comment
|
2022-01-31 15:00:03 +02:00 |
|
vyzo
|
c9bd5ec452
|
mark tipset references to protect them during critical section
|
2022-01-31 14:54:23 +02:00 |
|
vyzo
|
ee63be26a1
|
fix race in protectView
|
2022-01-31 12:40:26 +02:00 |
|
vyzo
|
1900c907fd
|
account for missing refs in the markset in Has
|
2022-01-31 09:44:51 +02:00 |
|
vyzo
|
a9d4495d83
|
use both hot and cold when doing fetches for markset positive objects
|
2022-01-30 22:47:20 +02:00 |
|
vyzo
|
7b8447a95a
|
reinstante waitForMissingRefs
|
2022-01-30 19:00:52 +02:00 |
|
vyzo
|
7931f1f8f9
|
fix lint
|
2022-01-30 15:43:52 +02:00 |
|
vyzo
|
20b75022aa
|
fix mockStore for splitstore tests
|
2022-01-30 15:33:30 +02:00 |
|
vyzo
|
dbc8903bac
|
sortless compaction
|
2022-01-30 15:33:15 +02:00 |
|
vyzo
|
a4f720d866
|
coldset test
|
2022-01-30 15:11:25 +02:00 |
|
vyzo
|
4b8369c071
|
fix buffered reads
|
2022-01-30 15:11:18 +02:00 |
|
vyzo
|
72333147d3
|
prettier checkpoint close
|
2022-01-30 15:00:03 +02:00 |
|
vyzo
|
64cda4aeda
|
on disk coldsets
|
2022-01-30 14:19:05 +02:00 |
|
vyzo
|
6ede77b254
|
checkpoint test
|
2022-01-30 14:10:11 +02:00 |
|
vyzo
|
c94eee5fc3
|
on disk checkpoints
|
2022-01-30 12:47:30 +02:00 |
|
vyzo
|
322b85898f
|
make markSets synchronous in critical section
|
2022-01-30 12:10:08 +02:00 |
|
vyzo
|
cf09dd044a
|
moar markset tests
|
2022-01-30 11:45:12 +02:00 |
|
vyzo
|
a4c1a34163
|
check for existence of badger db in recover
|
2022-01-30 11:44:27 +02:00 |
|
vyzo
|
d140909562
|
add MarkMany to MarkSet interface
|
2022-01-30 11:43:45 +02:00 |
|
vyzo
|
f9fd47e7d0
|
use temporary dir for splitstore test path
|
2022-01-29 13:11:58 +02:00 |
|
vyzo
|
67fbf9eb00
|
improve peristence test
|
2022-01-28 15:55:34 +02:00 |
|
vyzo
|
730aceac6d
|
immediately flush pending writes when entering critical section
|
2022-01-28 15:55:10 +02:00 |
|
vyzo
|
1bf396f9af
|
add test for markset persistence
|
2022-01-28 15:49:41 +02:00 |
|
vyzo
|
45c2f34295
|
refactor marksets for critical section on-disk persistence
|
2022-01-28 15:41:33 +02:00 |
|
vyzo
|
f07ce297f6
|
optimize slice allocations in walk
|
2022-01-26 21:55:24 +02:00 |
|
vyzo
|
a87239e802
|
avoid extraneous assignment
|
2022-01-26 21:48:03 +02:00 |
|
vyzo
|
fe47d6a1a4
|
fix check and warmup for parallel walk
|
2022-01-26 09:01:51 +02:00 |
|
vyzo
|
10f2445a99
|
use minimum of 2 workers
|
2022-01-25 21:37:48 +02:00 |
|
vyzo
|
8e01e73de4
|
dynamically compute number of workers for parallel chain walk
|
2022-01-25 19:47:58 +02:00 |
|
vyzo
|
7c8edf5632
|
parallelize walkChain
|
2022-01-25 17:29:02 +02:00 |
|
vyzo
|
25284b5325
|
refactor: eliminate distinction between markset and markset visitors
|
2022-01-25 16:31:45 +02:00 |
|
vyzo
|
40c61a310f
|
fix splitstore test
|
2021-12-14 17:17:30 +02:00 |
|
Aayush Rajasekaran
|
dfb65ed89f
|
Plumb contexts through
|
2021-12-11 17:04:00 -05:00 |
|
vyzo
|
489782e21b
|
satisfy the spellchecker that masquarades as a linter
|
2021-12-03 12:15:28 +02:00 |
|
vyzo
|
6ce5879071
|
add unit test for compaction supression
|
2021-12-03 12:05:15 +02:00 |
|
vyzo
|
05c1325885
|
add logic for supressing compaction near upgrade boundaries
|
2021-12-03 11:50:35 +02:00 |
|
vyzo
|
eb0a62e9e9
|
RIP bloom filter markset; you weren't used anyway.
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
742c85bf81
|
add SupportsVisitor in the markset env interface
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
79f348a01c
|
make the write lock scope limited within a function
Also avoid removing the writing set if there was an error while writing.
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
a9403b42f0
|
deduplicate put code
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
1a59b73adc
|
add .tmp suffix to transient db names
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
26a5832f92
|
simplify and deduplicate Has/Visit using helper methods tryPending and tryDB.
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
380e16d465
|
remove redundant writers state variable
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
b83b5405c6
|
widen MarkSetVisitor interface and get rid of the casts
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
57c984cea1
|
improve concurrency properties of Visit with optimistic concurrency
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
563fa1e31a
|
take advantage of MarkSet/Visitor unification to atomically check+mark
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
cb3c53664d
|
unify marksets and visitors
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
3c994d94aa
|
deduplicate some code in markset_badger
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
32d94d4e85
|
reset walked set as epoch boundaries are crossed
the walk is BFS, so we can do this!
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
49346f5679
|
use visitors instead of cidsets in walks
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
1323d8fb20
|
markset-backed visitors
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
6f22cffb6b
|
object visitor interface
|
2021-08-10 10:47:01 +03:00 |
|
vyzo
|
a768dd84fa
|
document splitstore clear shed command
|
2021-08-01 19:18:34 +03:00 |
|
vyzo
|
da3e36acdc
|
Merge branch 'master' into feat/moving-gc
|
2021-07-27 12:12:38 +03:00 |
|
vyzo
|
8809707158
|
update README
|
2021-07-27 12:08:39 +03:00 |
|
vyzo
|
9d25464703
|
use functional options for hotstore gc, rename MovingGC to FullGC
|
2021-07-27 09:53:22 +03:00 |
|
vyzo
|
c52c2738ab
|
fix potential underflow
|
2021-07-26 18:38:42 +03:00 |
|
vyzo
|
18caefc4a1
|
fix missing object condition; short-circuit walk
|
2021-07-26 18:34:05 +03:00 |
|
vyzo
|
0fd8fc24e1
|
load a full finality worth of state during warmup
|
2021-07-26 18:34:05 +03:00 |
|
vyzo
|
b1f60e85e9
|
document moving GC frequency option in README
|
2021-07-26 15:05:57 +03:00 |
|
vyzo
|
5acae50e07
|
add config option for splitstore moving gc frequency
|
2021-07-26 15:05:57 +03:00 |
|
vyzo
|
fb3986226f
|
do hotstore moving GC in splitstore with a user-specified frequency
|
2021-07-26 15:05:57 +03:00 |
|
vyzo
|
74009bd67f
|
document lotus-shed splitstore utiilities in the README
|
2021-07-26 08:52:32 +03:00 |
|
vyzo
|
ce528a1293
|
implement Info in splitstore
|
2021-07-26 08:45:46 +03:00 |
|
vyzo
|
c00b86e8a8
|
stop the walk on missing references
|
2021-07-25 13:42:20 +03:00 |
|
vyzo
|
5285a14d27
|
write check summary at the end
|
2021-07-25 11:42:13 +03:00 |
|
vyzo
|
5718da237f
|
handle newlines consistently in check output
|
2021-07-25 11:30:58 +03:00 |
|
vyzo
|
1918ffda4e
|
implement splitstore check
|
2021-07-25 10:35:37 +03:00 |
|
vyzo
|
d6ebead756
|
add comment about the necessity of FileIO
|
2021-07-23 16:30:55 +03:00 |
|
vyzo
|
1480c22447
|
improve README
|
2021-07-23 16:27:09 +03:00 |
|
vyzo
|
9faa3d5fd7
|
adjust compaction test with badger to hit the db
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
9d5cf4177b
|
add splitstore compaction test with badger markset
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
dec5e13c85
|
update README with configuration options
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
a8c0fd1654
|
tweak badger options
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
4df6073382
|
some tweaks in badger markset
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
f492f8d450
|
track in flight writes in badger markste, as they happen outside the lock
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
f75d982c76
|
remove early occurs check from trackTxnRef
this happens inline, and it might block when using the badger markset
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
2891a31c99
|
make badger markset concurrent close safe
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
f2b7c3e6f2
|
reduce scope of exclusive lock in badger markset
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
12c3432b8d
|
document the "badger" markset type option
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
be9530b66e
|
finetune badger options
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
2c26abc551
|
add badger markset test
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
54a6968327
|
add badger-backed markset implementation
|
2021-07-23 12:47:18 +03:00 |
|
vyzo
|
5266b240b8
|
coalesce message and message receipt retention
|
2021-07-22 22:02:29 +03:00 |
|
vyzo
|
2a68ae8dad
|
fix warmup by decoupling state from message receipt walk
|
2021-07-22 20:49:36 +03:00 |
|
Łukasz Magiera
|
da5aeda197
|
Merge branch 'master' into feat/splitstore-hot-messages
|
2021-07-22 12:58:06 +02:00 |
|
vyzo
|
c47fce8d2d
|
test protector support
|
2021-07-20 09:02:45 +03:00 |
|
vyzo
|
ebbaf23af8
|
support out-of-chain reference protection
|
2021-07-20 09:02:40 +03:00 |
|
vyzo
|
006050ed27
|
implement hotstore message retention policy
|
2021-07-17 08:59:43 +03:00 |
|
vyzo
|
1b77361301
|
add option for hotstore message retention
|
2021-07-17 08:35:35 +03:00 |
|
vyzo
|
e003203bea
|
implement exposed splitstore
|
2021-07-15 13:12:10 +03:00 |
|
vyzo
|
5a23f64b3b
|
code reorg: break splitstore.go into smaller logical units
|
2021-07-14 13:11:15 -07:00 |
|
vyzo
|
3f3a12b75c
|
remove BlockstoreMover interface
we decided it's premature
|
2021-07-14 22:59:53 +03:00 |
|
vyzo
|
023146803d
|
use Broadcast for view barrier
|
2021-07-14 22:59:53 +03:00 |
|
vyzo
|
3d77ae1f4d
|
make trackTxnRefMany consistent with trackTxnRef
|
2021-07-14 22:59:53 +03:00 |
|
vyzo
|
6f126c80bf
|
remove redundant log, more descriptive error message for closing condition
|
2021-07-14 22:59:53 +03:00 |
|
vyzo
|
ff093fae00
|
use a missing compactionIndex as an indicator for warmup
so that splitstore v0 nodes upgrading will get a fresh warmup.
|
2021-07-14 22:59:53 +03:00 |
|
vyzo
|
669b47cfc9
|
do moving gc for hotstore every 20 compactions
that's about once a week
|
2021-07-14 22:59:53 +03:00 |
|