Commit Graph

353 Commits

Author SHA1 Message Date
vyzo
bdb97d6186 more robust handling of sync gap walks 2021-07-04 18:38:28 +03:00
vyzo
7cf75e667d keep genesis-linked state hot 2021-07-04 18:38:28 +03:00
vyzo
e9f531b4aa don't open bolt tracking store with NoSync, it might get corrupted 2021-07-04 18:38:28 +03:00
Raúl Kripalani
b2b7eb2ded metrics: increment misses in View(). 2021-07-04 18:38:28 +03:00
vyzo
3a9b7c592d mark from current epoch to boundary epoch when necessary
this is necessary to avoid wearing clown shoes when the node stays
offline for an extended period of time (more than 1 finality).

Basically it gets quite slow if we do the full 2 finality walk, so we
try to avoid it unless necessary.
The conditions under which a full walk is necessary is if there is a
sync gap (most likely because the node was offline) during which the
tracking of writes is inaccurate because we have not yet delivered the
HeadChange notification.  In this case, it is possible to have
actually hot blocks to be tracked before the boundary and fail to mark
them accordingly.  So when we detect a sync gap, we do the full walk;
if there is no sync gap, we can just use the much faster boundary
epoch walk.
2021-07-04 18:38:28 +03:00
vyzo
d7ceef104e decrease CompactionThreshold to 3 finalities 2021-07-04 18:38:28 +03:00
vyzo
e3cbeec6ee implement chain walking 2021-07-04 18:38:28 +03:00
vyzo
04f2e102a1 kill full splitstore compaction, simplify splitstore configuration 2021-07-04 18:38:28 +03:00
Steven Allen
f353a794cb
fix: spelling
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
2021-04-28 22:08:37 -07:00
Steven Allen
63db9e1633 fix(splitstore): fix a panic on revert-only head changes
Calling, e.g., `lotus chain sethead` on an ancestor tipset won't apply
any new blocks, it'll just revert a bunch. This will lead to HeadChange
calls with no new blocks to apply.

fixes #6125
2021-04-28 20:35:30 -07:00
vyzo
1b1d3606cd make linter happy 2021-03-11 13:10:44 +02:00
vyzo
353bb1881f compact hotstore if it provides the method 2021-03-11 11:45:19 +02:00
vyzo
ae6410d02f use compacting atomic to make the test deterministic 2021-03-09 09:05:36 +02:00
Steven Allen
6d2e8d721d test: attempt to make the splitstore test deterministic
At a minimum, make it thread-safe.
2021-03-08 16:36:25 -08:00
vyzo
3bd77701d8 deduplicate code 2021-03-08 19:46:21 +02:00
vyzo
3d1b855f20 rename GC to CollectGarbage, ignore badger.ErrNoRewrite 2021-03-08 19:42:38 +02:00
vyzo
52de95d344 also gc in compactFull, not just compactSimple 2021-03-08 18:30:09 +02:00
vyzo
8562a9bb82 garbage collect hotstore after compaction 2021-03-08 18:12:09 +02:00
vyzo
e85391b46c quiet stupid linter 2021-03-05 20:05:32 +02:00
vyzo
09f5ba177a add splitstore unit test 2021-03-05 19:55:32 +02:00
vyzo
0a2f2cf00d use the right condition for triggering the miss metric 2021-03-05 14:48:59 +02:00
vyzo
2b32c2e597 add some metrics 2021-03-05 14:48:57 +02:00
vyzo
99d21573da remove DEBUG log spam 2021-03-05 14:46:18 +02:00
vyzo
c58df3f079 don't panic on compaction errors 2021-03-05 14:46:18 +02:00
vyzo
9bd009d795 use atomics to demarkate critical section and limit close delay 2021-03-05 14:46:18 +02:00
vyzo
17be7d3919 save markSetSize 2021-03-05 14:46:18 +02:00
vyzo
aff0f1ed4c deduplicate code for batch deletion 2021-03-05 14:46:18 +02:00
vyzo
5fb6a907cb fix loop condition in batch deletion 2021-03-05 14:46:18 +02:00
vyzo
fdd877534f walk at boundary epoch, 2 finalities from current epoch, to find live objects
objects written after that are retained anyway.
2021-03-05 14:46:18 +02:00
vyzo
508fcb9d26 properly close snoop at shutdown 2021-03-05 14:46:18 +02:00
vyzo
47d8c87486 fix log 2021-03-05 14:46:18 +02:00
vyzo
11b2f41804 overestimate markSetSize a bit 2021-03-05 14:46:18 +02:00
vyzo
6b680d112b do tracker purge in smaller batches 2021-03-05 14:46:18 +02:00
vyzo
d2d0980532 don't delete in one giant batch, use smaller chunks of batchSize 2021-03-05 14:46:18 +02:00
vyzo
70ebb2ad8d improve startup log 2021-03-05 14:46:18 +02:00
vyzo
006c55a7c9 add startup log 2021-03-05 14:46:18 +02:00
vyzo
06d8ea10b1 batch delete during the cold purge 2021-03-05 14:46:18 +02:00
vyzo
86b73d651e add DeleteMany to Blockstore interface 2021-03-05 14:46:18 +02:00
vyzo
c762536dcb deduplicate code 2021-03-05 14:46:18 +02:00
vyzo
5184bc5c40 log consistency for full compaction 2021-03-05 14:46:18 +02:00
vyzo
68213a92cb use ioutil.TempDir for test directories 2021-03-05 14:46:18 +02:00
vyzo
35d466d847 use sha256 for bloom key rehashing 2021-03-05 14:46:18 +02:00
vyzo
f651f43c5e improve comment accuracy 2021-03-05 14:46:18 +02:00
Raúl Kripalani
4b1e1f4b52 rename liveset => markset; rename snoop => tracking store; docs. 2021-03-05 14:46:18 +02:00
vyzo
48f253328d increase batch size to 16K 2021-03-05 14:46:18 +02:00
vyzo
ce68b9b229 batch writes during warm up 2021-03-05 14:46:18 +02:00
Raúl Kripalani
8cfba5b092 renames and polish. 2021-03-05 14:46:18 +02:00
Raúl Kripalani
b1b452bc0f remove dependency from blockstore/splitstore => chain/store. 2021-03-05 14:46:18 +02:00
vyzo
b9400c590f use crypto/rand for bloom salt 2021-03-05 14:46:18 +02:00
vyzo
e612fff1fe also estimate liveset size during warm up 2021-03-05 14:46:18 +02:00
vyzo
748dd962d8 snake current tipset from head change notification 2021-03-05 14:46:18 +02:00
vyzo
cb36d5b6a4 warm up splitstore at first head change notification 2021-03-05 14:46:18 +02:00
Raúl Kripalani
1a804fbdec move splitstore into blockstore package. 2021-03-05 14:46:18 +02:00