Steven Allen
a888ea0d1f
fix: avoid holding a lock while calling the View callback
...
Interleaved puts/views could get really slow and there's no real reason
to use view under the covers here because the underlying blockstore is
always a "memstore".
2021-03-11 20:03:38 -08:00
vyzo
ae6410d02f
use compacting atomic to make the test deterministic
2021-03-09 09:05:36 +02:00
Steven Allen
6d2e8d721d
test: attempt to make the splitstore test deterministic
...
At a minimum, make it thread-safe.
2021-03-08 16:36:25 -08:00
vyzo
90741da019
tune badger gc to repeated gc the value log until there is no rewrite
2021-03-08 21:46:44 +02:00
vyzo
3bd77701d8
deduplicate code
2021-03-08 19:46:21 +02:00
vyzo
3d1b855f20
rename GC to CollectGarbage, ignore badger.ErrNoRewrite
2021-03-08 19:42:38 +02:00
vyzo
52de95d344
also gc in compactFull, not just compactSimple
2021-03-08 18:30:09 +02:00
vyzo
8562a9bb82
garbage collect hotstore after compaction
2021-03-08 18:12:09 +02:00
vyzo
e85391b46c
quiet stupid linter
2021-03-05 20:05:32 +02:00
vyzo
09f5ba177a
add splitstore unit test
2021-03-05 19:55:32 +02:00
vyzo
0a2f2cf00d
use the right condition for triggering the miss metric
2021-03-05 14:48:59 +02:00
vyzo
2b32c2e597
add some metrics
2021-03-05 14:48:57 +02:00
vyzo
99d21573da
remove DEBUG log spam
2021-03-05 14:46:18 +02:00
vyzo
c58df3f079
don't panic on compaction errors
2021-03-05 14:46:18 +02:00
vyzo
9bd009d795
use atomics to demarkate critical section and limit close delay
2021-03-05 14:46:18 +02:00
vyzo
17be7d3919
save markSetSize
2021-03-05 14:46:18 +02:00
vyzo
aff0f1ed4c
deduplicate code for batch deletion
2021-03-05 14:46:18 +02:00
vyzo
5fb6a907cb
fix loop condition in batch deletion
2021-03-05 14:46:18 +02:00
vyzo
98a7b884fe
implement DeleteMany in union blockstore
2021-03-05 14:46:18 +02:00
vyzo
fdd877534f
walk at boundary epoch, 2 finalities from current epoch, to find live objects
...
objects written after that are retained anyway.
2021-03-05 14:46:18 +02:00
vyzo
508fcb9d26
properly close snoop at shutdown
2021-03-05 14:46:18 +02:00
vyzo
47d8c87486
fix log
2021-03-05 14:46:18 +02:00
vyzo
11b2f41804
overestimate markSetSize a bit
2021-03-05 14:46:18 +02:00
vyzo
6b680d112b
do tracker purge in smaller batches
2021-03-05 14:46:18 +02:00
vyzo
d2d0980532
don't delete in one giant batch, use smaller chunks of batchSize
2021-03-05 14:46:18 +02:00
vyzo
70ebb2ad8d
improve startup log
2021-03-05 14:46:18 +02:00
vyzo
006c55a7c9
add startup log
2021-03-05 14:46:18 +02:00
vyzo
06d8ea10b1
batch delete during the cold purge
2021-03-05 14:46:18 +02:00
vyzo
4c05ec28ba
fix FromDatastore to not do double adapting
2021-03-05 14:46:18 +02:00
vyzo
ab52e34e6a
add comment
...
Co-authored-by: raulk <raul@protocol.ai>
2021-03-05 14:46:18 +02:00
vyzo
86fdad2e31
fix typo
...
Co-authored-by: raulk <raul@protocol.ai>
2021-03-05 14:46:18 +02:00
vyzo
2ff5aec80e
satisfy linter, use Prefix for common path of non inline CIDs
2021-03-05 14:46:18 +02:00
vyzo
8a55b73146
fix the situation with WrapIDStore
2021-03-05 14:46:18 +02:00
vyzo
86b73d651e
add DeleteMany to Blockstore interface
2021-03-05 14:46:18 +02:00
vyzo
c762536dcb
deduplicate code
2021-03-05 14:46:18 +02:00
vyzo
5184bc5c40
log consistency for full compaction
2021-03-05 14:46:18 +02:00
vyzo
68213a92cb
use ioutil.TempDir for test directories
2021-03-05 14:46:18 +02:00
vyzo
35d466d847
use sha256 for bloom key rehashing
2021-03-05 14:46:18 +02:00
vyzo
f651f43c5e
improve comment accuracy
2021-03-05 14:46:18 +02:00
Raúl Kripalani
4b1e1f4b52
rename liveset => markset; rename snoop => tracking store; docs.
2021-03-05 14:46:18 +02:00
vyzo
48f253328d
increase batch size to 16K
2021-03-05 14:46:18 +02:00
vyzo
ce68b9b229
batch writes during warm up
2021-03-05 14:46:18 +02:00
Raúl Kripalani
8cfba5b092
renames and polish.
2021-03-05 14:46:18 +02:00
Raúl Kripalani
b1b452bc0f
remove dependency from blockstore/splitstore => chain/store.
2021-03-05 14:46:18 +02:00
vyzo
b9400c590f
use crypto/rand for bloom salt
2021-03-05 14:46:18 +02:00
vyzo
e612fff1fe
also estimate liveset size during warm up
2021-03-05 14:46:18 +02:00
vyzo
748dd962d8
snake current tipset from head change notification
2021-03-05 14:46:18 +02:00
vyzo
cb36d5b6a4
warm up splitstore at first head change notification
2021-03-05 14:46:18 +02:00
Raúl Kripalani
1a804fbdec
move splitstore into blockstore package.
2021-03-05 14:46:18 +02:00
Raúl Kripalani
68b8e8e9cb
implement unionBlockstore#HashOnRead.
2021-03-02 21:22:24 +00:00
