cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
19,253 Commits 24 Branches 342 Tags 314 MiB
07e5abbd3e
Commit Graph

416 Commits

Author SHA1 Message Date
vyzo
a4c1a34163 check for existence of badger db in recover 2022-01-30 11:44:27 +02:00
vyzo
d140909562 add MarkMany to MarkSet interface 2022-01-30 11:43:45 +02:00
vyzo
f9fd47e7d0 use temporary dir for splitstore test path 2022-01-29 13:11:58 +02:00
vyzo
67fbf9eb00 improve peristence test 2022-01-28 15:55:34 +02:00
vyzo
730aceac6d immediately flush pending writes when entering critical section 2022-01-28 15:55:10 +02:00
vyzo
1bf396f9af add test for markset persistence 2022-01-28 15:49:41 +02:00
vyzo
45c2f34295 refactor marksets for critical section on-disk persistence 2022-01-28 15:41:33 +02:00
vyzo
f07ce297f6 optimize slice allocations in walk 2022-01-26 21:55:24 +02:00
vyzo
a87239e802 avoid extraneous assignment 2022-01-26 21:48:03 +02:00
vyzo
fe47d6a1a4 fix check and warmup for parallel walk 2022-01-26 09:01:51 +02:00
vyzo
10f2445a99 use minimum of 2 workers 2022-01-25 21:37:48 +02:00
vyzo
8e01e73de4 dynamically compute number of workers for parallel chain walk 2022-01-25 19:47:58 +02:00
vyzo
7c8edf5632 parallelize walkChain 2022-01-25 17:29:02 +02:00
vyzo
25284b5325 refactor: eliminate distinction between markset and markset visitors 2022-01-25 16:31:45 +02:00
vyzo
40c61a310f fix splitstore test 2021-12-14 17:17:30 +02:00
Aayush Rajasekaran
dfb65ed89f Plumb contexts through 2021-12-11 17:04:00 -05:00
vyzo
489782e21b satisfy the spellchecker that masquarades as a linter 2021-12-03 12:15:28 +02:00
vyzo
6ce5879071 add unit test for compaction supression 2021-12-03 12:05:15 +02:00
vyzo
05c1325885 add logic for supressing compaction near upgrade boundaries 2021-12-03 11:50:35 +02:00
vyzo
eb0a62e9e9 RIP bloom filter markset; you weren't used anyway. 2021-08-10 10:47:01 +03:00
vyzo
742c85bf81 add SupportsVisitor in the markset env interface 2021-08-10 10:47:01 +03:00
vyzo
79f348a01c make the write lock scope limited within a function 
Also avoid removing the writing set if there was an error while writing.
 2021-08-10 10:47:01 +03:00
vyzo
a9403b42f0 deduplicate put code 2021-08-10 10:47:01 +03:00
vyzo
1a59b73adc add .tmp suffix to transient db names 2021-08-10 10:47:01 +03:00
vyzo
26a5832f92 simplify and deduplicate Has/Visit using helper methods tryPending and tryDB. 2021-08-10 10:47:01 +03:00
vyzo
380e16d465 remove redundant writers state variable 2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6 widen MarkSetVisitor interface and get rid of the casts 2021-08-10 10:47:01 +03:00
vyzo
57c984cea1 improve concurrency properties of Visit with optimistic concurrency 2021-08-10 10:47:01 +03:00
vyzo
563fa1e31a take advantage of MarkSet/Visitor unification to atomically check+mark 2021-08-10 10:47:01 +03:00
vyzo
cb3c53664d unify marksets and visitors 2021-08-10 10:47:01 +03:00
vyzo
3c994d94aa deduplicate some code in markset_badger 2021-08-10 10:47:01 +03:00
vyzo
32d94d4e85 reset walked set as epoch boundaries are crossed 
the walk is BFS, so we can do this!
 2021-08-10 10:47:01 +03:00
vyzo
49346f5679 use visitors instead of cidsets in walks 2021-08-10 10:47:01 +03:00
vyzo
1323d8fb20 markset-backed visitors 2021-08-10 10:47:01 +03:00
vyzo
6f22cffb6b object visitor interface 2021-08-10 10:47:01 +03:00
vyzo
a768dd84fa document splitstore clear shed command 2021-08-01 19:18:34 +03:00
vyzo
da3e36acdc
Merge branch 'master' into feat/moving-gc 2021-07-27 12:12:38 +03:00
vyzo
8809707158 update README 2021-07-27 12:08:39 +03:00
vyzo
9d25464703 use functional options for hotstore gc, rename MovingGC to FullGC 2021-07-27 09:53:22 +03:00
vyzo
c52c2738ab fix potential underflow 2021-07-26 18:38:42 +03:00
vyzo
18caefc4a1 fix missing object condition; short-circuit walk 2021-07-26 18:34:05 +03:00
vyzo
0fd8fc24e1 load a full finality worth of state during warmup 2021-07-26 18:34:05 +03:00
vyzo
b1f60e85e9 document moving GC frequency option in README 2021-07-26 15:05:57 +03:00
vyzo
5acae50e07 add config option for splitstore moving gc frequency 2021-07-26 15:05:57 +03:00
vyzo
fb3986226f do hotstore moving GC in splitstore with a user-specified frequency 2021-07-26 15:05:57 +03:00
vyzo
74009bd67f document lotus-shed splitstore utiilities in the README 2021-07-26 08:52:32 +03:00
vyzo
ce528a1293 implement Info in splitstore 2021-07-26 08:45:46 +03:00
vyzo
c00b86e8a8 stop the walk on missing references 2021-07-25 13:42:20 +03:00
vyzo
5285a14d27 write check summary at the end 2021-07-25 11:42:13 +03:00
vyzo
5718da237f handle newlines consistently in check output 2021-07-25 11:30:58 +03:00
vyzo
1918ffda4e implement splitstore check 2021-07-25 10:35:37 +03:00
vyzo
d6ebead756 add comment about the necessity of FileIO 2021-07-23 16:30:55 +03:00
vyzo
1480c22447 improve README 2021-07-23 16:27:09 +03:00
vyzo
9faa3d5fd7 adjust compaction test with badger to hit the db 2021-07-23 12:47:18 +03:00
vyzo
9d5cf4177b add splitstore compaction test with badger markset 2021-07-23 12:47:18 +03:00
vyzo
dec5e13c85 update README with configuration options 2021-07-23 12:47:18 +03:00
vyzo
a8c0fd1654 tweak badger options 2021-07-23 12:47:18 +03:00
vyzo
4df6073382 some tweaks in badger markset 2021-07-23 12:47:18 +03:00
vyzo
f492f8d450 track in flight writes in badger markste, as they happen outside the lock 2021-07-23 12:47:18 +03:00
vyzo
f75d982c76 remove early occurs check from trackTxnRef 
this happens inline, and it might block when using the badger markset
 2021-07-23 12:47:18 +03:00
vyzo
2891a31c99 make badger markset concurrent close safe 2021-07-23 12:47:18 +03:00
vyzo
f2b7c3e6f2 reduce scope of exclusive lock in badger markset 2021-07-23 12:47:18 +03:00
vyzo
12c3432b8d document the "badger" markset type option 2021-07-23 12:47:18 +03:00
vyzo
be9530b66e finetune badger options 2021-07-23 12:47:18 +03:00
vyzo
2c26abc551 add badger markset test 2021-07-23 12:47:18 +03:00
vyzo
54a6968327 add badger-backed markset implementation 2021-07-23 12:47:18 +03:00
vyzo
5266b240b8 coalesce message and message receipt retention 2021-07-22 22:02:29 +03:00
vyzo
2a68ae8dad fix warmup by decoupling state from message receipt walk 2021-07-22 20:49:36 +03:00
Łukasz Magiera
da5aeda197
Merge branch 'master' into feat/splitstore-hot-messages 2021-07-22 12:58:06 +02:00
vyzo
c47fce8d2d test protector support 2021-07-20 09:02:45 +03:00
vyzo
ebbaf23af8 support out-of-chain reference protection 2021-07-20 09:02:40 +03:00
vyzo
006050ed27 implement hotstore message retention policy 2021-07-17 08:59:43 +03:00
vyzo
1b77361301 add option for hotstore message retention 2021-07-17 08:35:35 +03:00
vyzo
e003203bea implement exposed splitstore 2021-07-15 13:12:10 +03:00
vyzo
5a23f64b3b code reorg: break splitstore.go into smaller logical units 2021-07-14 13:11:15 -07:00
vyzo
3f3a12b75c remove BlockstoreMover interface 
we decided it's premature
 2021-07-14 22:59:53 +03:00
vyzo
023146803d use Broadcast for view barrier 2021-07-14 22:59:53 +03:00
vyzo
3d77ae1f4d make trackTxnRefMany consistent with trackTxnRef 2021-07-14 22:59:53 +03:00
vyzo
6f126c80bf remove redundant log, more descriptive error message for closing condition 2021-07-14 22:59:53 +03:00
vyzo
ff093fae00 use a missing compactionIndex as an indicator for warmup 
so that splitstore v0 nodes upgrading will get a fresh warmup.
 2021-07-14 22:59:53 +03:00
vyzo
669b47cfc9 do moving gc for hotstore every 20 compactions 
that's about once a week
 2021-07-14 22:59:53 +03:00
vyzo
818b8de182 keep track of the compaction serial (index) 
it is useful so that:
- we only do slow (but very effective) moving gc every 10 compactions
- we can detect a splitstore v0 upgrade and re-warm up
 2021-07-14 22:59:53 +03:00
vyzo
c93328b036 use the new traits for hotstore gc 2021-07-14 22:59:52 +03:00
vyzo
af399529ec finetune view waiting 2021-07-13 09:06:40 +03:00
vyzo
257423e917 fix view waiting issues with the WaitGroup 
We can add after Wait is called, which is problematic with WaitGroups.
This instead uses a mx/cond combo and waits while the count is > 0.
The only downside is that we might needlessly wait for (a bunch) of views
that started while the txn is active, but we can live with that.
 2021-07-13 09:01:50 +03:00
Steven Allen
04abd190ab nit: remove useless goto 
Because stebalien has allergies.
 2021-07-12 21:46:50 -07:00
vyzo
60212c86cb put a mutex around HeadChange 2021-07-13 03:14:13 +03:00
vyzo
759594d01c always return the waitgroup in protectView 
so that we preclude the following scenario:
    Start compaction.
    Start view.
    Finish compaction.
    Start compaction.

which would not wait for the view to complete.
 2021-07-13 03:11:40 +03:00
vyzo
df9670c58d fix lint 2021-07-10 16:38:40 +03:00
vyzo
0c5e336ff1 address review comments 2021-07-10 16:30:27 +03:00
vyzo
870a47f55d handle id cids in internal versions of view/get 2021-07-09 20:07:17 +03:00
vyzo
f5ae10e3d1 refactor debug log code to eliminate duplication 2021-07-09 19:53:51 +03:00
vyzo
41290383e2 fix test 2021-07-09 19:24:44 +03:00
vyzo
b9a5ea8f7b update wording around discard store 2021-07-09 19:23:55 +03:00
vyzo
18161fee38 remove unused lookback constructs 2021-07-09 19:12:58 +03:00
vyzo
095d7427ba make view protection optimistic again, as there is a race window 2021-07-09 15:41:10 +03:00
vyzo
da0feb3fa4 dont mark references inline; instad rely on the main compaction thread to do concurrent marking 
The problem is that it is possible that an inline marking might take minutes for some objects
(infrequent, but still possible for state roots and prohibitive if that's a block validation).
So we simply track references continuously and rely on the main compaction thread to trigger
concurrent marking for all references at opportune moments.

Assumption: we can mark references faster than they are created during purge or else we'll
never purge anything.
 2021-07-09 15:10:02 +03:00
vyzo
acc4c374ef properly handle protecting long-running views 2021-07-09 13:20:18 +03:00
vyzo
565faff754 fix test 2021-07-09 11:38:09 +03:00
vyzo
4f89d260b0 kill isOldBlockHeader; it's dangerous. 2021-07-09 11:35:10 +03:00
vyzo
de5e21bf1a correctly handle identity cids 2021-07-09 11:31:04 +03:00
vyzo
abdf4a161a explicitly switch marksets for concurrent marking 
this has very noticeable impact in initial marking time; it also allows us
to get rid of the confusing ts monikers.
 2021-07-09 04:26:36 +03:00
vyzo
b6611125b6 add environment variables to turn on the debug log without recompiling 2021-07-08 21:30:39 +03:00
vyzo
60dd97c7fc fix potential deadlock in View 
As pointed out by magik, it is possible to deadlock if the view callback performs
a blockstore operation while a Lock is pending.
This fixes the issue by optimistically tracking the reference before actually calling
the underlying View and limiting the scope of the lock.
 2021-07-08 21:18:59 +03:00
vyzo
c0537848b3
fix typo 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-08 17:54:16 +03:00
vyzo
fa30ac8c5d
fix typo 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-08 17:53:59 +03:00
vyzo
00d7772f57 move check for closure in walkChain 
so that we don't do it too often and also cover warmup.
 2021-07-08 13:12:19 +03:00
vyzo
5cf1e09e81 README: add instructions for how to enable 2021-07-08 13:00:31 +03:00
vyzo
9aa4f3b3b2 add README for documentation 2021-07-08 12:32:41 +03:00
vyzo
e6eacbdd56 use RW mutexes in marksets 2021-07-08 10:20:29 +03:00
vyzo
48f13a43b7 intelligently close marksets and signal errors in concurrent operations 2021-07-08 10:18:43 +03:00
vyzo
f5c45bd517 check the closing state variable often 
so that we have a reasonably quick graceful shutdown
 2021-07-08 10:13:44 +03:00
vyzo
4f808367f8 fix lint 2021-07-07 21:32:58 +03:00
vyzo
fee50b13a2 check the closing state on each batch during the purge. 2021-07-07 21:32:05 +03:00
vyzo
c6421f8a75 don't nil the mark sets on close, it's dangerous. 
a concurrent marking can panic.
 2021-07-07 21:27:36 +03:00
vyzo
aec2ba2c82 nil map/bf on markset close 2021-07-07 16:46:14 +03:00
vyzo
451ddf50ab RIP bbolt-backed markset 2021-07-07 16:39:37 +03:00
vyzo
9dbb2e0abd don't leak tracking errors through the API 2021-07-07 16:34:02 +03:00
vyzo
83c30dc4c0 protect assignment of warmup epoch with the mutex 2021-07-07 11:31:27 +03:00
vyzo
6cc2112749 remove the curTs state variable; we don't need it 2021-07-07 09:55:25 +03:00
vyzo
05dbbe9681 rename som Txn methods for better readability 2021-07-07 09:52:31 +03:00
vyzo
90da6227b3 transactional protect incoming tipsets 2021-07-07 02:11:37 +03:00
vyzo
0e2af11f6a prepare the transaction before launching the compaction goroutine 2021-07-07 01:39:58 +03:00
vyzo
f2f4af669d clean up: simplify debug log, get rid of ugly debug log 2021-07-06 17:13:38 +03:00
vyzo
c1c25868cc improve comments 2021-07-06 15:09:04 +03:00
vyzo
fdff1bebc9 move map markset implementation to its own file 2021-07-06 14:44:40 +03:00
vyzo
5c514504f7 remove unused GetGenesis method from ChainAccessor interface 2021-07-06 14:41:41 +03:00
vyzo
dc8139a1d2 add some comments for debug only code 2021-07-06 13:23:12 +03:00
vyzo
c4ae3e0c3d minor tweak 2021-07-06 09:17:35 +03:00
vyzo
169ab262f5 really optimize computing object weights 
sort is still taking a long time, this should be as fast as it gets.
 2021-07-06 09:02:44 +03:00
vyzo
55a9e0ccd1 short-circuit block headers on sort weight computation 2021-07-06 08:22:43 +03:00
vyzo
bf7aeb3167 optimize sort a tad 
it's taking a long time to compute weights...
 2021-07-06 08:10:57 +03:00
vyzo
0659235e21 cache cid strings in sort 
so as to avoid making a gazillion of strings
 2021-07-06 07:26:13 +03:00
vyzo
525a2c71dd use hashes as keys in weight map to avoid duplicate work 
otherwise the root object will be raw, but internal references will be dag; duplicate work.
 2021-07-06 01:27:56 +03:00
vyzo
c6ad8fdaed use walkObjectRaw for computing object weights 
cids that come out of the hotstore with ForEach are raw.
 2021-07-06 01:08:44 +03:00
vyzo
2cbd3faf5a make sure to nil everything in txnEndProtect 2021-07-05 23:56:31 +03:00
vyzo
51ab891d5c quiet linter 
it's a false positive, function doesn't escape.
 2021-07-05 23:53:45 +03:00
vyzo
bd436ab9de make endTxnProtect idempotent 2021-07-05 23:51:10 +03:00
vyzo
e859942fa4 code cleanup: refactor txn state code into their own functions 2021-07-05 23:31:37 +03:00
vyzo
3477d265c6 unify the two marksets 
really, it's concurrent marking and there is no reason to have two different marksets
 2021-07-05 20:10:47 +03:00
vyzo
73d07999bf dont needlessly wait 1 min in first retry for missing refs 2021-07-05 18:24:48 +03:00
vyzo
af8cf712be handle all missing refs together 
so that we wait 6min at most, not 12.
 2021-07-05 18:16:54 +03:00
vyzo
5a099b7d05 more commentary on the missing refs situation 2021-07-05 16:12:17 +03:00
vyzo
59639a0788 reinstate some better code for handling missing references. 2021-07-05 16:08:08 +03:00
vyzo
fa195bede2 get rid of ugly missing reference handling code 
those missing objects don't seem to ever get there, are they from an abandoned fork?
 2021-07-05 14:29:55 +03:00
vyzo
59936ef468 fix log 2021-07-05 13:30:31 +03:00
vyzo
0b7153be86 use internal version of has for occurs checks 2021-07-05 12:41:11 +03:00
vyzo
d8b8d75e0f readd minute delay before trying for missing objects 2021-07-05 12:38:09 +03:00
vyzo
d7709deb2b reduce memory pressure from marksets when the size is decreased 2021-07-05 11:51:22 +03:00
vyzo
3ec834b2e3 improve logs and error messages 2021-07-05 11:41:09 +03:00