vyzo
|
d2d0980532
|
don't delete in one giant batch, use smaller chunks of batchSize
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
70ebb2ad8d
|
improve startup log
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
006c55a7c9
|
add startup log
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
06d8ea10b1
|
batch delete during the cold purge
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
4c05ec28ba
|
fix FromDatastore to not do double adapting
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
ab52e34e6a
|
add comment
Co-authored-by: raulk <raul@protocol.ai>
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
86fdad2e31
|
fix typo
Co-authored-by: raulk <raul@protocol.ai>
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
2ff5aec80e
|
satisfy linter, use Prefix for common path of non inline CIDs
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
8a55b73146
|
fix the situation with WrapIDStore
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
86b73d651e
|
add DeleteMany to Blockstore interface
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
dd0c308427
|
move Blockstore config to FullNode, rename to Chainstore and add default for HotStoreType
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
6014273e69
|
storage miner doesn't need a splitstore
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
c762536dcb
|
deduplicate code
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
5184bc5c40
|
log consistency for full compaction
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
68213a92cb
|
use ioutil.TempDir for test directories
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
35d466d847
|
use sha256 for bloom key rehashing
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
f651f43c5e
|
improve comment accuracy
|
2021-03-05 14:46:18 +02:00 |
|
Raúl Kripalani
|
4b1e1f4b52
|
rename liveset => markset; rename snoop => tracking store; docs.
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
48f253328d
|
increase batch size to 16K
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
ce68b9b229
|
batch writes during warm up
|
2021-03-05 14:46:18 +02:00 |
|
Raúl Kripalani
|
8cfba5b092
|
renames and polish.
|
2021-03-05 14:46:18 +02:00 |
|
Raúl Kripalani
|
b1b452bc0f
|
remove dependency from blockstore/splitstore => chain/store.
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
b9400c590f
|
use crypto/rand for bloom salt
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
e612fff1fe
|
also estimate liveset size during warm up
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
748dd962d8
|
snake current tipset from head change notification
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
cb36d5b6a4
|
warm up splitstore at first head change notification
|
2021-03-05 14:46:18 +02:00 |
|
Raúl Kripalani
|
1a804fbdec
|
move splitstore into blockstore package.
|
2021-03-05 14:46:18 +02:00 |
|
Raúl Kripalani
|
1b51c10d78
|
split off lmdb support to a different branch.
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
3733456bca
|
go mod tidy
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
0fc2f3a26f
|
fix post-rebase compilation errors
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
3282f856ec
|
fix tests
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
99c6e4f48f
|
adjust min bloom filter size
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
cae5ddce88
|
dynamically size bloom filters
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
5639261e44
|
make compaction parameters variable
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
7587ab6234
|
quiet the stupid linter
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
e582f0b713
|
remove references to splitstore from lotus-shed
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
05fee27840
|
remove stale references to lmdb from splitstore implementation
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
f62999d2b8
|
use named constants for bloom filter parameters
|
2021-03-05 14:46:18 +02:00 |
|
vyzo
|
44aadb931a
|
rehash salted keys in bloom filter
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
88849201ff
|
fix tests
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
f5ce7957f3
|
size bloom filter for 50M objects
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
f4c6bc6a86
|
comment nomenclature
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
4cc672d0c5
|
batch move objects from coldstore to hotstore
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
97abbe1eca
|
add (salted) bloom filter liveset
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
aba6530411
|
batch deletion for purging the tracking store
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
09cd1175a1
|
structured log for beginning of compaction
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
e52c709d8a
|
more accurate setting of skip params
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
2426ffb277
|
better logging plus moving some code around
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
2f26026991
|
compactSimple should walk the cold epoch at depth 1
So that it finds reachable objects that should stay in the hotstore
|
2021-03-05 14:46:17 +02:00 |
|
vyzo
|
783dcda19c
|
add Sync to the tracking store
|
2021-03-05 14:46:17 +02:00 |
|