vyzo
c4d95de987
coalesce back-to-back compactions
...
get rid of the CompactionCold construct, run a single compaction on catch up
2021-07-04 18:38:28 +03:00
vyzo
b7897595eb
augment current epoch by +1
...
to account for off by one conditions
2021-07-04 18:38:28 +03:00
vyzo
933c786421
update write epoch in the background every second
2021-07-04 18:38:28 +03:00
vyzo
66f1630f14
fix lint issue
2021-07-04 18:38:28 +03:00
vyzo
bb17608ae0
track writeEpoch relative to current wall clock time
...
The issue: head change notifications are not emitted until after catching up,
which results in all writes during a catch up period being tracked at the base epoch.
2021-07-04 18:38:28 +03:00
vyzo
421f05eab9
save the warm up epoch only if successful in warming up
2021-07-04 18:38:28 +03:00
vyzo
9b6448518c
refactor warmup to trigger at startup and not wait for sync
2021-07-04 18:38:28 +03:00
vyzo
3fe4261f12
don't attempt compaction while still syncing
2021-07-04 18:38:28 +03:00
vyzo
7b02673620
don't try to visit genesis parent blocks
2021-07-04 18:38:28 +03:00
vyzo
997f2c098b
keep headers hot when running with a noop splitstore
2021-07-04 18:38:28 +03:00
vyzo
7c814cd2e3
refactor genesis state loading code into its own method
2021-07-04 18:38:28 +03:00
vyzo
41573f1fb2
also walk parent message receipts when including messages in the walk
2021-07-04 18:38:28 +03:00
vyzo
fa6481401d
reduce SyncGapTime to 1 minute
...
for maximal safety.
2021-07-04 18:38:28 +03:00
vyzo
fda291b876
fix test
2021-07-04 18:38:28 +03:00
vyzo
d33a44e67f
first visit the cid, then short-circuit non dagcbor objects
2021-07-04 18:38:28 +03:00
vyzo
bdb97d6186
more robust handling of sync gap walks
2021-07-04 18:38:28 +03:00
vyzo
7cf75e667d
keep genesis-linked state hot
2021-07-04 18:38:28 +03:00
vyzo
e9f531b4aa
don't open bolt tracking store with NoSync, it might get corrupted
2021-07-04 18:38:28 +03:00
Raúl Kripalani
b2b7eb2ded
metrics: increment misses in View().
2021-07-04 18:38:28 +03:00
vyzo
3a9b7c592d
mark from current epoch to boundary epoch when necessary
...
this is necessary to avoid wearing clown shoes when the node stays
offline for an extended period of time (more than 1 finality).
Basically it gets quite slow if we do the full 2 finality walk, so we
try to avoid it unless necessary.
The conditions under which a full walk is necessary is if there is a
sync gap (most likely because the node was offline) during which the
tracking of writes is inaccurate because we have not yet delivered the
HeadChange notification. In this case, it is possible to have
actually hot blocks to be tracked before the boundary and fail to mark
them accordingly. So when we detect a sync gap, we do the full walk;
if there is no sync gap, we can just use the much faster boundary
epoch walk.
2021-07-04 18:38:28 +03:00
vyzo
d7ceef104e
decrease CompactionThreshold to 3 finalities
2021-07-04 18:38:28 +03:00
vyzo
e3cbeec6ee
implement chain walking
2021-07-04 18:38:28 +03:00
vyzo
04f2e102a1
kill full splitstore compaction, simplify splitstore configuration
2021-07-04 18:38:28 +03:00
vyzo
4d3c73f4ca
noop blockstore
2021-07-04 18:38:28 +03:00
Steven Allen
f353a794cb
fix: spelling
...
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
2021-04-28 22:08:37 -07:00
Steven Allen
63db9e1633
fix(splitstore): fix a panic on revert-only head changes
...
Calling, e.g., `lotus chain sethead` on an ancestor tipset won't apply
any new blocks, it'll just revert a bunch. This will lead to HeadChange
calls with no new blocks to apply.
fixes #6125
2021-04-28 20:35:30 -07:00
Peter Rabbitson
a5fd552a0c
CachedBlockstore is not referenced as of 3795cc2bd2
2021-04-06 13:24:49 +02:00
Łukasz Magiera
5f80869fe0
Merge pull request #5794 from filecoin-project/fix/atomic-first
...
fix: make sure atomic 64bit fields are 64bit aligned
2021-03-25 13:29:59 +01:00
Steven Allen
25725110e7
Merge pull request #5792 from filecoin-project/fix/timed-cache-locking
...
fix: avoid holding a lock while calling the View callback
2021-03-12 09:25:00 -08:00
Łukasz Magiera
c69b26cfc6
Merge pull request #5778 from filecoin-project/feat/splitstore-compact-hotstore
...
splitstore: compact hotstore prior to garbage collection
2021-03-12 16:30:05 +01:00
Steven Allen
bba71da401
fix: return buffers after canceling badger operation
...
In theory, Delete/Put could fail. If it does, we'll return the buffers
to the pool before we're really done with them.
In practice, this is almost certainly not an issue as badger shouldn't
_use_ the buffer unless we flush. But I feel slightly safer this way.
2021-03-11 20:30:43 -08:00
Steven Allen
1c490f3fda
fix: make sure atomic 64bit fields are 64bit aligned
...
Otherwise, this won't work on 32bit ARM.
2021-03-11 20:10:39 -08:00
Steven Allen
a888ea0d1f
fix: avoid holding a lock while calling the View callback
...
Interleaved puts/views could get really slow and there's no real reason
to use view under the covers here because the underlying blockstore is
always a "memstore".
2021-03-11 20:03:38 -08:00
vyzo
1b1d3606cd
make linter happy
2021-03-11 13:10:44 +02:00
vyzo
353bb1881f
compact hotstore if it provides the method
2021-03-11 11:45:19 +02:00
vyzo
01ce9b5c44
add Compact to badger blockstore
2021-03-11 11:45:05 +02:00
vyzo
ae6410d02f
use compacting atomic to make the test deterministic
2021-03-09 09:05:36 +02:00
Steven Allen
6d2e8d721d
test: attempt to make the splitstore test deterministic
...
At a minimum, make it thread-safe.
2021-03-08 16:36:25 -08:00
vyzo
90741da019
tune badger gc to repeated gc the value log until there is no rewrite
2021-03-08 21:46:44 +02:00
vyzo
3bd77701d8
deduplicate code
2021-03-08 19:46:21 +02:00
vyzo
3d1b855f20
rename GC to CollectGarbage, ignore badger.ErrNoRewrite
2021-03-08 19:42:38 +02:00
vyzo
52de95d344
also gc in compactFull, not just compactSimple
2021-03-08 18:30:09 +02:00
vyzo
8562a9bb82
garbage collect hotstore after compaction
2021-03-08 18:12:09 +02:00
vyzo
e85391b46c
quiet stupid linter
2021-03-05 20:05:32 +02:00
vyzo
09f5ba177a
add splitstore unit test
2021-03-05 19:55:32 +02:00
vyzo
0a2f2cf00d
use the right condition for triggering the miss metric
2021-03-05 14:48:59 +02:00
vyzo
2b32c2e597
add some metrics
2021-03-05 14:48:57 +02:00
vyzo
99d21573da
remove DEBUG log spam
2021-03-05 14:46:18 +02:00
vyzo
c58df3f079
don't panic on compaction errors
2021-03-05 14:46:18 +02:00
vyzo
9bd009d795
use atomics to demarkate critical section and limit close delay
2021-03-05 14:46:18 +02:00
