cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
12,931 Commits 24 Branches 342 Tags 314 MiB
964df1a2ec
Commit Graph

283 Commits

Author SHA1 Message Date
vyzo
5266b240b8 coalesce message and message receipt retention 2021-07-22 22:02:29 +03:00
vyzo
2a68ae8dad fix warmup by decoupling state from message receipt walk 2021-07-22 20:49:36 +03:00
Łukasz Magiera
da5aeda197
Merge branch 'master' into feat/splitstore-hot-messages 2021-07-22 12:58:06 +02:00
vyzo
c47fce8d2d test protector support 2021-07-20 09:02:45 +03:00
vyzo
ebbaf23af8 support out-of-chain reference protection 2021-07-20 09:02:40 +03:00
vyzo
006050ed27 implement hotstore message retention policy 2021-07-17 08:59:43 +03:00
vyzo
1b77361301 add option for hotstore message retention 2021-07-17 08:35:35 +03:00
vyzo
e003203bea implement exposed splitstore 2021-07-15 13:12:10 +03:00
vyzo
5a23f64b3b code reorg: break splitstore.go into smaller logical units 2021-07-14 13:11:15 -07:00
vyzo
3f3a12b75c remove BlockstoreMover interface 
we decided it's premature
 2021-07-14 22:59:53 +03:00
vyzo
023146803d use Broadcast for view barrier 2021-07-14 22:59:53 +03:00
vyzo
3d77ae1f4d make trackTxnRefMany consistent with trackTxnRef 2021-07-14 22:59:53 +03:00
vyzo
6f126c80bf remove redundant log, more descriptive error message for closing condition 2021-07-14 22:59:53 +03:00
vyzo
ff093fae00 use a missing compactionIndex as an indicator for warmup 
so that splitstore v0 nodes upgrading will get a fresh warmup.
 2021-07-14 22:59:53 +03:00
vyzo
669b47cfc9 do moving gc for hotstore every 20 compactions 
that's about once a week
 2021-07-14 22:59:53 +03:00
vyzo
818b8de182 keep track of the compaction serial (index) 
it is useful so that:
- we only do slow (but very effective) moving gc every 10 compactions
- we can detect a splitstore v0 upgrade and re-warm up
 2021-07-14 22:59:53 +03:00
vyzo
c93328b036 use the new traits for hotstore gc 2021-07-14 22:59:52 +03:00
vyzo
35180b4761 merge Compact and CollectGarbage in badger 2021-07-14 22:59:52 +03:00
vyzo
dc81c0e6a2 add blockstore traits related to gc 2021-07-14 22:59:52 +03:00
vyzo
af399529ec finetune view waiting 2021-07-13 09:06:40 +03:00
vyzo
257423e917 fix view waiting issues with the WaitGroup 
We can add after Wait is called, which is problematic with WaitGroups.
This instead uses a mx/cond combo and waits while the count is > 0.
The only downside is that we might needlessly wait for (a bunch) of views
that started while the txn is active, but we can live with that.
 2021-07-13 09:01:50 +03:00
Steven Allen
04abd190ab nit: remove useless goto 
Because stebalien has allergies.
 2021-07-12 21:46:50 -07:00
vyzo
60212c86cb put a mutex around HeadChange 2021-07-13 03:14:13 +03:00
vyzo
759594d01c always return the waitgroup in protectView 
so that we preclude the following scenario:
    Start compaction.
    Start view.
    Finish compaction.
    Start compaction.

which would not wait for the view to complete.
 2021-07-13 03:11:40 +03:00
vyzo
df9670c58d fix lint 2021-07-10 16:38:40 +03:00
vyzo
0c5e336ff1 address review comments 2021-07-10 16:30:27 +03:00
vyzo
870a47f55d handle id cids in internal versions of view/get 2021-07-09 20:07:17 +03:00
vyzo
f5ae10e3d1 refactor debug log code to eliminate duplication 2021-07-09 19:53:51 +03:00
vyzo
41290383e2 fix test 2021-07-09 19:24:44 +03:00
vyzo
b9a5ea8f7b update wording around discard store 2021-07-09 19:23:55 +03:00
vyzo
c0a1cfffa1 rename noopstore to discardstore 2021-07-09 19:19:37 +03:00
vyzo
18161fee38 remove unused lookback constructs 2021-07-09 19:12:58 +03:00
vyzo
095d7427ba make view protection optimistic again, as there is a race window 2021-07-09 15:41:10 +03:00
vyzo
da0feb3fa4 dont mark references inline; instad rely on the main compaction thread to do concurrent marking 
The problem is that it is possible that an inline marking might take minutes for some objects
(infrequent, but still possible for state roots and prohibitive if that's a block validation).
So we simply track references continuously and rely on the main compaction thread to trigger
concurrent marking for all references at opportune moments.

Assumption: we can mark references faster than they are created during purge or else we'll
never purge anything.
 2021-07-09 15:10:02 +03:00
vyzo
acc4c374ef properly handle protecting long-running views 2021-07-09 13:20:18 +03:00
vyzo
565faff754 fix test 2021-07-09 11:38:09 +03:00
vyzo
4f89d260b0 kill isOldBlockHeader; it's dangerous. 2021-07-09 11:35:10 +03:00
vyzo
de5e21bf1a correctly handle identity cids 2021-07-09 11:31:04 +03:00
vyzo
909f7039d4 make badger Close-safe 2021-07-09 09:54:12 +03:00
vyzo
abdf4a161a explicitly switch marksets for concurrent marking 
this has very noticeable impact in initial marking time; it also allows us
to get rid of the confusing ts monikers.
 2021-07-09 04:26:36 +03:00
vyzo
b6611125b6 add environment variables to turn on the debug log without recompiling 2021-07-08 21:30:39 +03:00
vyzo
60dd97c7fc fix potential deadlock in View 
As pointed out by magik, it is possible to deadlock if the view callback performs
a blockstore operation while a Lock is pending.
This fixes the issue by optimistically tracking the reference before actually calling
the underlying View and limiting the scope of the lock.
 2021-07-08 21:18:59 +03:00
vyzo
c0537848b3
fix typo 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-08 17:54:16 +03:00
vyzo
fa30ac8c5d
fix typo 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-08 17:53:59 +03:00
vyzo
00d7772f57 move check for closure in walkChain 
so that we don't do it too often and also cover warmup.
 2021-07-08 13:12:19 +03:00
vyzo
5cf1e09e81 README: add instructions for how to enable 2021-07-08 13:00:31 +03:00
vyzo
9aa4f3b3b2 add README for documentation 2021-07-08 12:32:41 +03:00
vyzo
e6eacbdd56 use RW mutexes in marksets 2021-07-08 10:20:29 +03:00
vyzo
48f13a43b7 intelligently close marksets and signal errors in concurrent operations 2021-07-08 10:18:43 +03:00
vyzo
f5c45bd517 check the closing state variable often 
so that we have a reasonably quick graceful shutdown
 2021-07-08 10:13:44 +03:00