cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
12,958 Commits 24 Branches 342 Tags 314 MiB
64d2b7a122
Commit Graph

211 Commits

Author SHA1 Message Date
vyzo
c6ad8fdaed use walkObjectRaw for computing object weights 
cids that come out of the hotstore with ForEach are raw.
 2021-07-06 01:08:44 +03:00
vyzo
2cbd3faf5a make sure to nil everything in txnEndProtect 2021-07-05 23:56:31 +03:00
vyzo
51ab891d5c quiet linter 
it's a false positive, function doesn't escape.
 2021-07-05 23:53:45 +03:00
vyzo
bd436ab9de make endTxnProtect idempotent 2021-07-05 23:51:10 +03:00
vyzo
e859942fa4 code cleanup: refactor txn state code into their own functions 2021-07-05 23:31:37 +03:00
vyzo
3477d265c6 unify the two marksets 
really, it's concurrent marking and there is no reason to have two different marksets
 2021-07-05 20:10:47 +03:00
vyzo
73d07999bf dont needlessly wait 1 min in first retry for missing refs 2021-07-05 18:24:48 +03:00
vyzo
af8cf712be handle all missing refs together 
so that we wait 6min at most, not 12.
 2021-07-05 18:16:54 +03:00
vyzo
5a099b7d05 more commentary on the missing refs situation 2021-07-05 16:12:17 +03:00
vyzo
59639a0788 reinstate some better code for handling missing references. 2021-07-05 16:08:08 +03:00
vyzo
fa195bede2 get rid of ugly missing reference handling code 
those missing objects don't seem to ever get there, are they from an abandoned fork?
 2021-07-05 14:29:55 +03:00
vyzo
59936ef468 fix log 2021-07-05 13:30:31 +03:00
vyzo
0b7153be86 use internal version of has for occurs checks 2021-07-05 12:41:11 +03:00
vyzo
d8b8d75e0f readd minute delay before trying for missing objects 2021-07-05 12:38:09 +03:00
vyzo
d7709deb2b reduce memory pressure from marksets when the size is decreased 2021-07-05 11:51:22 +03:00
vyzo
3ec834b2e3 improve logs and error messages 2021-07-05 11:41:09 +03:00
vyzo
918a7ec749 a bit more fil commitment short-circuiting 2021-07-05 11:38:53 +03:00
vyzo
2ea2abc07d short-circuit fil commitments 
they don't make it to the blockstore anyway
 2021-07-05 11:32:52 +03:00
vyzo
839f7bd2b5 only occur check for DAGs 2021-07-05 11:11:08 +03:00
vyzo
c81ae5fc20 add some comments about the missing business and anothre log 2021-07-05 10:42:14 +03:00
vyzo
4c41f52828 add warning for missing objects for marking for debug purposes 2021-07-05 10:35:04 +03:00
vyzo
3597192d58 remove the sleeps and busy loop more times when waiting for missing objects 2021-07-05 10:31:47 +03:00
vyzo
1726eb993c deal with incomplete objects that need to be marked and protected 
seems that something is writing DAGs before its consituents, which causes problems.
 2021-07-05 10:22:52 +03:00
vyzo
db53859e7a reduce CompactionThreshold to 5 finalities 
so that we run compaction every finality, once we've first compacted
 2021-07-04 22:12:51 +03:00
vyzo
b08e0b7102 fix lint 2021-07-04 21:24:15 +03:00
vyzo
94efae419e reduce length of critical section 
Just the purge; the rest is not critical -- e.g. it's ok if we do some duplicate copies
to the coldstore, we'll have gc soon.
 2021-07-04 21:21:53 +03:00
vyzo
f33d4e79aa simplify transactional protection logic 
Now that we delete objects heaviest first, we don't have to do deep walk and rescan gymnastics.
 2021-07-04 20:49:39 +03:00
vyzo
40c271cda1 sort cold objects before deleting 
so that we can't shoot ourselves in the foot by deleting the constituents of a DAG while it is
still in the hotstore.
 2021-07-04 20:17:07 +03:00
vyzo
13d612f72f smarter trackTxnRefMany 2021-07-04 19:33:49 +03:00
vyzo
f124389b66 recursively protect all references 2021-07-04 19:21:00 +03:00
vyzo
4d286da593 fix error message 2021-07-04 18:58:39 +03:00
vyzo
680af8eb09 use deep object walking for more robust handling of transactional references 2021-07-04 18:38:28 +03:00
vyzo
1f02428225 fix lint 2021-07-04 18:38:28 +03:00
vyzo
2c7a89a1db short-circuit rescanning on block headers 2021-07-04 18:38:28 +03:00
vyzo
8e56fffb33 walkChain should visit the genesis state root 2021-07-04 18:38:28 +03:00
vyzo
190cb18ab0 housekeeping 
- remove defunct tracking store implementations
- update splitstore node config
- use mark set type config option (defaulting to mapts); a memory constrained node
  may want to use an on-disk one
 2021-07-04 18:38:28 +03:00
vyzo
19d1b1f532 deal with partially written objects 2021-07-04 18:38:28 +03:00
vyzo
0a1d7b3732 fix log 2021-07-04 18:38:28 +03:00
vyzo
eafffc1634 more efficient trackTxnRefMany 2021-07-04 18:38:28 +03:00
vyzo
36f93649ef fix panic from concurrent map writes in txnRefs 2021-07-04 18:38:28 +03:00
vyzo
6fa2cd232d simplify compaction model 2021-07-04 18:38:28 +03:00
vyzo
1f2b604c07 RIP tracking store 2021-07-04 18:38:28 +03:00
vyzo
68a83500bc fix bug that turned candidate filtering to dead code 2021-07-04 18:38:28 +03:00
vyzo
642f0e4740 deal with memory pressure, don't walk under the boundary 2021-07-04 18:38:28 +03:00
vyzo
c5cf8e226b remove unnecessary code 2021-07-04 18:38:28 +03:00
vyzo
d79e4da7aa more accurate stats about mark set updates 2021-07-04 18:38:28 +03:00
vyzo
6f58fdcb22 remove vm copy context detection hack 
stack tracing is slow.
 2021-07-04 18:38:28 +03:00
vyzo
2b03316cd9 fix log message 2021-07-04 18:38:28 +03:00
vyzo
184d3802b6 remove dead code 2021-07-04 18:38:28 +03:00
vyzo
228a435ba7 rework tracking logic; do it lazily and far more efficiently 2021-07-04 18:38:28 +03:00
vyzo
9d6cabd18a if it's not a dag, it's not a block 2021-07-04 18:38:28 +03:00
vyzo
8157f889ce short-circuit marking walks when encountering a block and more efficient walking 2021-07-04 18:38:28 +03:00
vyzo
736d6a3c19 only treat Has as an implicit write within vm.Copy context 2021-07-04 18:38:28 +03:00
vyzo
39723bbe60 use a single map for tracking pending writes, properly track implicits 2021-07-04 18:38:28 +03:00
vyzo
5834231e58 create the transactional protect filter before walking 2021-07-04 18:38:28 +03:00
vyzo
e4bb4be855 fix some residual purge races 2021-07-04 18:38:28 +03:00
vyzo
68bc5d2291 skip moving cold blocks when running with a noop coldstore 
it is a noop but it still takes (a lot of) time because it has to read all the cold blocks.
 2021-07-04 18:38:28 +03:00
vyzo
b87295db93 bubble up dependent txn ref errors 
This cause Has to return false if it fails to traverse/protect all links, which would cause
the vm to recompute.
 2021-07-04 18:38:28 +03:00
vyzo
637fbf6c5b fix faulty if/else logic for implicit txn protection 2021-07-04 18:38:28 +03:00
vyzo
9d6bcd7705 avoid clown shoes: only walk links for tracking in implicit writes/refs 2021-07-04 18:38:28 +03:00
vyzo
484dfaebce reused cidset across all walks when flushing pending writes 2021-07-04 18:38:28 +03:00
vyzo
1d41e1544a optimize transitive write tracking a bit 2021-07-04 18:38:28 +03:00
vyzo
da00fc66ee downgrade a couple of logs to warnings 2021-07-04 18:38:28 +03:00
vyzo
4071488ef2 first write, then track 2021-07-04 18:38:28 +03:00
vyzo
bd92c230da refactor txn reference tracking, do deep marking of DAGs 2021-07-04 18:38:28 +03:00
vyzo
a98a062347 do the dag walk for deep write tracking during flush 
avoid crawling everything to a halt
 2021-07-04 18:38:28 +03:00
vyzo
13a674330f add pending write check before tracking the object in Has 2021-07-04 18:38:28 +03:00
vyzo
982867317e transitively track dags from implicit writes in Has 2021-07-04 18:38:28 +03:00
vyzo
4de0cd9fcb move write log back to flush so that we don't crawl to a halt 2021-07-04 18:38:28 +03:00
vyzo
2faa4aa993 debug log writes at track so that we get correct stack traces 2021-07-04 18:38:28 +03:00
vyzo
aeaa59d4b5 move comments about tracking perf issues into a more pertinent place 2021-07-04 18:38:28 +03:00
vyzo
3e8e9273ca track all writes using async batching, not just implicit ones 2021-07-04 18:38:28 +03:00
vyzo
d0bfe421b5 flush implicit writes at the right time before starting compaction to avoid races 2021-07-04 18:38:28 +03:00
vyzo
7f473f56eb flush implicit writes before starting compaction 2021-07-04 18:38:28 +03:00
vyzo
a29947d47c flush implicit writes in all paths in updateWriteEpoch 2021-07-04 18:38:28 +03:00
vyzo
be6cc2c3e6 batch implicit write tracking 
bolt performance leaves something to be desired; doing a single Put takes 10ms, about the same time
as batching thousands of them.
 2021-07-04 18:38:28 +03:00
vyzo
e472cacb3e add missing return 2021-07-04 18:38:28 +03:00
vyzo
6a3cbea790 treat Has as an implicit Write 
Rationale: the VM uses the Has check to avoid issuing a duplicate Write in the blockstore.
This means that live objects that would be otherwise written are not actually written, resulting
in the first write epoch being considered the write epoch.
 2021-07-04 18:38:28 +03:00
vyzo
90dc274113 better logging for chain walk 2021-07-04 18:38:28 +03:00
vyzo
40f42db7fa walk tweaks 2021-07-04 18:38:28 +03:00
vyzo
09efed50fd check for lookback references to block headers in walk 2021-07-04 18:38:28 +03:00
vyzo
7de0771883 count txn live objects explicitly for logging 2021-07-04 18:38:28 +03:00
vyzo
e29b64c5de check both markset and txn liveset before declaring an object cold 2021-07-04 18:38:28 +03:00
vyzo
4bed3161f0 fix broken purge count log 2021-07-04 18:38:28 +03:00
vyzo
dec61fa333 deduplicate stack logs and optionally trace write stacks 2021-07-04 18:38:28 +03:00
vyzo
7ebef6d838 better log message 2021-07-04 18:38:28 +03:00
vyzo
40ff5bf164 log put errors in splitstore log 2021-07-04 18:38:28 +03:00
vyzo
9fda61abec fix error check for unreachable cids 2021-07-04 18:38:28 +03:00
vyzo
4a71c68e06 move code around for better readability 2021-07-04 18:38:28 +03:00
vyzo
31497f4bd3 use internal get during walk to avoid blowing the compaction txn 
otherwise the walk itself precludes purge... duh!
 2021-07-04 18:38:28 +03:00
vyzo
6af3a23dd4 use a map for txn protection mark set 2021-07-04 18:38:28 +03:00
vyzo
65ccc99e79 minor tweaks in purge 
- allocate once
- log purge count
 2021-07-04 18:38:28 +03:00
vyzo
cb665d07e0 fix transactional race during compaction 
It is possible for an object to be written or recreated (and checked with Has)
after the mark completes and during the purge; if this happens we will purge
a live block.
 2021-07-04 18:38:28 +03:00
vyzo
50ebaf25aa don't log read misses before warmup 2021-07-04 18:38:28 +03:00
vyzo
b187b5c301 fix lint 2021-07-04 18:38:28 +03:00
vyzo
a53c4e1597 implement debug log 2021-07-04 18:38:28 +03:00
vyzo
fce7b8dc9b flush move log when cold collection is done 2021-07-04 18:38:28 +03:00
vyzo
fc247e4223 add debug log skeleton 2021-07-04 18:38:28 +03:00
vyzo
0390285c4e always do full walks, not only when there is a sync gap 2021-07-04 18:38:28 +03:00
vyzo
30dbe4978b adjust compaction range 2021-07-04 18:38:28 +03:00