Commit Graph

12560 Commits

Author SHA1 Message Date
vyzo
abdf4a161a explicitly switch marksets for concurrent marking
this has very noticeable impact in initial marking time; it also allows us
to get rid of the confusing ts monikers.
2021-07-09 04:26:36 +03:00
vyzo
b6611125b6 add environment variables to turn on the debug log without recompiling 2021-07-08 21:30:39 +03:00
vyzo
60dd97c7fc fix potential deadlock in View
As pointed out by magik, it is possible to deadlock if the view callback performs
a blockstore operation while a Lock is pending.
This fixes the issue by optimistically tracking the reference before actually calling
the underlying View and limiting the scope of the lock.
2021-07-08 21:18:59 +03:00
vyzo
c0537848b3
fix typo
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-07-08 17:54:16 +03:00
vyzo
fa30ac8c5d
fix typo
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-07-08 17:53:59 +03:00
vyzo
00d7772f57 move check for closure in walkChain
so that we don't do it too often and also cover warmup.
2021-07-08 13:12:19 +03:00
vyzo
5cf1e09e81 README: add instructions for how to enable 2021-07-08 13:00:31 +03:00
vyzo
9aa4f3b3b2 add README for documentation 2021-07-08 12:32:41 +03:00
vyzo
e6eacbdd56 use RW mutexes in marksets 2021-07-08 10:20:29 +03:00
vyzo
48f13a43b7 intelligently close marksets and signal errors in concurrent operations 2021-07-08 10:18:43 +03:00
vyzo
f5c45bd517 check the closing state variable often
so that we have a reasonably quick graceful shutdown
2021-07-08 10:13:44 +03:00
vyzo
4f808367f8 fix lint 2021-07-07 21:32:58 +03:00
vyzo
fee50b13a2 check the closing state on each batch during the purge. 2021-07-07 21:32:05 +03:00
vyzo
c6421f8a75 don't nil the mark sets on close, it's dangerous.
a concurrent marking can panic.
2021-07-07 21:27:36 +03:00
vyzo
aec2ba2c82 nil map/bf on markset close 2021-07-07 16:46:14 +03:00
vyzo
ec586a852a remove bbolt dependency from go.mod 2021-07-07 16:41:11 +03:00
vyzo
451ddf50ab RIP bbolt-backed markset 2021-07-07 16:39:37 +03:00
vyzo
9dbb2e0abd don't leak tracking errors through the API 2021-07-07 16:34:02 +03:00
vyzo
83c30dc4c0 protect assignment of warmup epoch with the mutex 2021-07-07 11:31:27 +03:00
vyzo
6cc2112749 remove the curTs state variable; we don't need it 2021-07-07 09:55:25 +03:00
vyzo
05dbbe9681 rename som Txn methods for better readability 2021-07-07 09:52:31 +03:00
vyzo
90da6227b3 transactional protect incoming tipsets 2021-07-07 02:11:37 +03:00
vyzo
0e2af11f6a prepare the transaction before launching the compaction goroutine 2021-07-07 01:39:58 +03:00
vyzo
f2f4af669d clean up: simplify debug log, get rid of ugly debug log 2021-07-06 17:13:38 +03:00
vyzo
c1c25868cc improve comments 2021-07-06 15:09:04 +03:00
vyzo
fdff1bebc9 move map markset implementation to its own file 2021-07-06 14:44:40 +03:00
vyzo
5c514504f7 remove unused GetGenesis method from ChainAccessor interface 2021-07-06 14:41:41 +03:00
vyzo
dc8139a1d2 add some comments for debug only code 2021-07-06 13:23:12 +03:00
vyzo
c4ae3e0c3d minor tweak 2021-07-06 09:17:35 +03:00
vyzo
169ab262f5 really optimize computing object weights
sort is still taking a long time, this should be as fast as it gets.
2021-07-06 09:02:44 +03:00
vyzo
55a9e0ccd1 short-circuit block headers on sort weight computation 2021-07-06 08:22:43 +03:00
vyzo
bf7aeb3167 optimize sort a tad
it's taking a long time to compute weights...
2021-07-06 08:10:57 +03:00
vyzo
0659235e21 cache cid strings in sort
so as to avoid making a gazillion of strings
2021-07-06 07:26:13 +03:00
vyzo
525a2c71dd use hashes as keys in weight map to avoid duplicate work
otherwise the root object will be raw, but internal references will be dag; duplicate work.
2021-07-06 01:27:56 +03:00
vyzo
c6ad8fdaed use walkObjectRaw for computing object weights
cids that come out of the hotstore with ForEach are raw.
2021-07-06 01:08:44 +03:00
vyzo
2cbd3faf5a make sure to nil everything in txnEndProtect 2021-07-05 23:56:31 +03:00
vyzo
51ab891d5c quiet linter
it's a false positive, function doesn't escape.
2021-07-05 23:53:45 +03:00
vyzo
bd436ab9de make endTxnProtect idempotent 2021-07-05 23:51:10 +03:00
vyzo
e859942fa4 code cleanup: refactor txn state code into their own functions 2021-07-05 23:31:37 +03:00
vyzo
3477d265c6 unify the two marksets
really, it's concurrent marking and there is no reason to have two different marksets
2021-07-05 20:10:47 +03:00
vyzo
73d07999bf dont needlessly wait 1 min in first retry for missing refs 2021-07-05 18:24:48 +03:00
vyzo
af8cf712be handle all missing refs together
so that we wait 6min at most, not 12.
2021-07-05 18:16:54 +03:00
vyzo
5a099b7d05 more commentary on the missing refs situation 2021-07-05 16:12:17 +03:00
vyzo
59639a0788 reinstate some better code for handling missing references. 2021-07-05 16:08:08 +03:00
vyzo
fa195bede2 get rid of ugly missing reference handling code
those missing objects don't seem to ever get there, are they from an abandoned fork?
2021-07-05 14:29:55 +03:00
vyzo
59936ef468 fix log 2021-07-05 13:30:31 +03:00
vyzo
0b7153be86 use internal version of has for occurs checks 2021-07-05 12:41:11 +03:00
vyzo
d8b8d75e0f readd minute delay before trying for missing objects 2021-07-05 12:38:09 +03:00
vyzo
d7709deb2b reduce memory pressure from marksets when the size is decreased 2021-07-05 11:51:22 +03:00
vyzo
3ec834b2e3 improve logs and error messages 2021-07-05 11:41:09 +03:00