Steven Allen
25725110e7
Merge pull request #5792 from filecoin-project/fix/timed-cache-locking
...
fix: avoid holding a lock while calling the View callback
2021-03-12 09:25:00 -08:00
Łukasz Magiera
c69b26cfc6
Merge pull request #5778 from filecoin-project/feat/splitstore-compact-hotstore
...
splitstore: compact hotstore prior to garbage collection
2021-03-12 16:30:05 +01:00
Steven Allen
bba71da401
fix: return buffers after canceling badger operation
...
In theory, Delete/Put could fail. If it does, we'll return the buffers
to the pool before we're really done with them.
In practice, this is almost certainly not an issue as badger shouldn't
_use_ the buffer unless we flush. But I feel slightly safer this way.
2021-03-11 20:30:43 -08:00
Steven Allen
a888ea0d1f
fix: avoid holding a lock while calling the View callback
...
Interleaved puts/views could get really slow and there's no real reason
to use view under the covers here because the underlying blockstore is
always a "memstore".
2021-03-11 20:03:38 -08:00
vyzo
1b1d3606cd
make linter happy
2021-03-11 13:10:44 +02:00
vyzo
353bb1881f
compact hotstore if it provides the method
2021-03-11 11:45:19 +02:00
vyzo
01ce9b5c44
add Compact to badger blockstore
2021-03-11 11:45:05 +02:00
vyzo
ae6410d02f
use compacting atomic to make the test deterministic
2021-03-09 09:05:36 +02:00
Steven Allen
6d2e8d721d
test: attempt to make the splitstore test deterministic
...
At a minimum, make it thread-safe.
2021-03-08 16:36:25 -08:00
vyzo
90741da019
tune badger gc to repeated gc the value log until there is no rewrite
2021-03-08 21:46:44 +02:00
vyzo
3bd77701d8
deduplicate code
2021-03-08 19:46:21 +02:00
vyzo
3d1b855f20
rename GC to CollectGarbage, ignore badger.ErrNoRewrite
2021-03-08 19:42:38 +02:00
vyzo
52de95d344
also gc in compactFull, not just compactSimple
2021-03-08 18:30:09 +02:00
vyzo
8562a9bb82
garbage collect hotstore after compaction
2021-03-08 18:12:09 +02:00
vyzo
e85391b46c
quiet stupid linter
2021-03-05 20:05:32 +02:00
vyzo
09f5ba177a
add splitstore unit test
2021-03-05 19:55:32 +02:00
vyzo
0a2f2cf00d
use the right condition for triggering the miss metric
2021-03-05 14:48:59 +02:00
vyzo
2b32c2e597
add some metrics
2021-03-05 14:48:57 +02:00
vyzo
99d21573da
remove DEBUG log spam
2021-03-05 14:46:18 +02:00
vyzo
c58df3f079
don't panic on compaction errors
2021-03-05 14:46:18 +02:00
vyzo
9bd009d795
use atomics to demarkate critical section and limit close delay
2021-03-05 14:46:18 +02:00
vyzo
17be7d3919
save markSetSize
2021-03-05 14:46:18 +02:00
vyzo
aff0f1ed4c
deduplicate code for batch deletion
2021-03-05 14:46:18 +02:00
vyzo
5fb6a907cb
fix loop condition in batch deletion
2021-03-05 14:46:18 +02:00
vyzo
98a7b884fe
implement DeleteMany in union blockstore
2021-03-05 14:46:18 +02:00
vyzo
fdd877534f
walk at boundary epoch, 2 finalities from current epoch, to find live objects
...
objects written after that are retained anyway.
2021-03-05 14:46:18 +02:00
vyzo
508fcb9d26
properly close snoop at shutdown
2021-03-05 14:46:18 +02:00
vyzo
47d8c87486
fix log
2021-03-05 14:46:18 +02:00
vyzo
11b2f41804
overestimate markSetSize a bit
2021-03-05 14:46:18 +02:00
vyzo
6b680d112b
do tracker purge in smaller batches
2021-03-05 14:46:18 +02:00
vyzo
d2d0980532
don't delete in one giant batch, use smaller chunks of batchSize
2021-03-05 14:46:18 +02:00
vyzo
70ebb2ad8d
improve startup log
2021-03-05 14:46:18 +02:00
vyzo
006c55a7c9
add startup log
2021-03-05 14:46:18 +02:00
vyzo
06d8ea10b1
batch delete during the cold purge
2021-03-05 14:46:18 +02:00
vyzo
4c05ec28ba
fix FromDatastore to not do double adapting
2021-03-05 14:46:18 +02:00
vyzo
ab52e34e6a
add comment
...
Co-authored-by: raulk <raul@protocol.ai>
2021-03-05 14:46:18 +02:00
vyzo
86fdad2e31
fix typo
...
Co-authored-by: raulk <raul@protocol.ai>
2021-03-05 14:46:18 +02:00
vyzo
2ff5aec80e
satisfy linter, use Prefix for common path of non inline CIDs
2021-03-05 14:46:18 +02:00
vyzo
8a55b73146
fix the situation with WrapIDStore
2021-03-05 14:46:18 +02:00
vyzo
86b73d651e
add DeleteMany to Blockstore interface
2021-03-05 14:46:18 +02:00
vyzo
c762536dcb
deduplicate code
2021-03-05 14:46:18 +02:00
vyzo
5184bc5c40
log consistency for full compaction
2021-03-05 14:46:18 +02:00
vyzo
68213a92cb
use ioutil.TempDir for test directories
2021-03-05 14:46:18 +02:00
vyzo
35d466d847
use sha256 for bloom key rehashing
2021-03-05 14:46:18 +02:00
vyzo
f651f43c5e
improve comment accuracy
2021-03-05 14:46:18 +02:00
Raúl Kripalani
4b1e1f4b52
rename liveset => markset; rename snoop => tracking store; docs.
2021-03-05 14:46:18 +02:00
vyzo
48f253328d
increase batch size to 16K
2021-03-05 14:46:18 +02:00
vyzo
ce68b9b229
batch writes during warm up
2021-03-05 14:46:18 +02:00
Raúl Kripalani
8cfba5b092
renames and polish.
2021-03-05 14:46:18 +02:00
Raúl Kripalani
b1b452bc0f
remove dependency from blockstore/splitstore => chain/store.
2021-03-05 14:46:18 +02:00
