Commit Graph

12783 Commits

Author SHA1 Message Date
vyzo
680af8eb09 use deep object walking for more robust handling of transactional references 2021-07-04 18:38:28 +03:00
vyzo
1f02428225 fix lint 2021-07-04 18:38:28 +03:00
vyzo
2c7a89a1db short-circuit rescanning on block headers 2021-07-04 18:38:28 +03:00
vyzo
028a5c4942 make test do something useful again 2021-07-04 18:38:28 +03:00
vyzo
8e56fffb33 walkChain should visit the genesis state root 2021-07-04 18:38:28 +03:00
vyzo
95c3aaec9a fix test 2021-07-04 18:38:28 +03:00
vyzo
190cb18ab0 housekeeping
- remove defunct tracking store implementations
- update splitstore node config
- use mark set type config option (defaulting to mapts); a memory constrained node
  may want to use an on-disk one
2021-07-04 18:38:28 +03:00
vyzo
19d1b1f532 deal with partially written objects 2021-07-04 18:38:28 +03:00
vyzo
0a1d7b3732 fix log 2021-07-04 18:38:28 +03:00
vyzo
08cad30be2 reuse key buffer in badger ForEachKey
cid copies the bytes so it's safe
2021-07-04 18:38:28 +03:00
vyzo
eafffc1634 more efficient trackTxnRefMany 2021-07-04 18:38:28 +03:00
vyzo
36f93649ef fix panic from concurrent map writes in txnRefs 2021-07-04 18:38:28 +03:00
vyzo
6fa2cd232d simplify compaction model 2021-07-04 18:38:28 +03:00
vyzo
5f7ae1f489 update splistore DI constructor 2021-07-04 18:38:28 +03:00
vyzo
1f2b604c07 RIP tracking store 2021-07-04 18:38:28 +03:00
vyzo
d476a3db2c BlockstoreIterator trait with implementation for badger 2021-07-04 18:38:28 +03:00
vyzo
68a83500bc fix bug that turned candidate filtering to dead code 2021-07-04 18:38:28 +03:00
vyzo
00fcf6dd72 add staging cache to bolt tracking store 2021-07-04 18:38:28 +03:00
vyzo
642f0e4740 deal with memory pressure, don't walk under the boundary 2021-07-04 18:38:28 +03:00
vyzo
c5cf8e226b remove unnecessary code 2021-07-04 18:38:28 +03:00
vyzo
d79e4da7aa more accurate stats about mark set updates 2021-07-04 18:38:28 +03:00
vyzo
6f58fdcb22 remove vm copy context detection hack
stack tracing is slow.
2021-07-04 18:38:28 +03:00
vyzo
2b03316cd9 fix log message 2021-07-04 18:38:28 +03:00
vyzo
184d3802b6 remove dead code 2021-07-04 18:38:28 +03:00
vyzo
228a435ba7 rework tracking logic; do it lazily and far more efficiently 2021-07-04 18:38:28 +03:00
vyzo
9d6cabd18a if it's not a dag, it's not a block 2021-07-04 18:38:28 +03:00
vyzo
8157f889ce short-circuit marking walks when encountering a block and more efficient walking 2021-07-04 18:38:28 +03:00
vyzo
736d6a3c19 only treat Has as an implicit write within vm.Copy context 2021-07-04 18:38:28 +03:00
vyzo
39723bbe60 use a single map for tracking pending writes, properly track implicits 2021-07-04 18:38:28 +03:00
vyzo
5834231e58 create the transactional protect filter before walking 2021-07-04 18:38:28 +03:00
vyzo
e4bb4be855 fix some residual purge races 2021-07-04 18:38:28 +03:00
vyzo
68bc5d2291 skip moving cold blocks when running with a noop coldstore
it is a noop but it still takes (a lot of) time because it has to read all the cold blocks.
2021-07-04 18:38:28 +03:00
vyzo
b87295db93 bubble up dependent txn ref errors
This cause Has to return false if it fails to traverse/protect all links, which would cause
the vm to recompute.
2021-07-04 18:38:28 +03:00
vyzo
637fbf6c5b fix faulty if/else logic for implicit txn protection 2021-07-04 18:38:28 +03:00
vyzo
9d6bcd7705 avoid clown shoes: only walk links for tracking in implicit writes/refs 2021-07-04 18:38:28 +03:00
vyzo
484dfaebce reused cidset across all walks when flushing pending writes 2021-07-04 18:38:28 +03:00
vyzo
1d41e1544a optimize transitive write tracking a bit 2021-07-04 18:38:28 +03:00
vyzo
da00fc66ee downgrade a couple of logs to warnings 2021-07-04 18:38:28 +03:00
vyzo
4071488ef2 first write, then track 2021-07-04 18:38:28 +03:00
vyzo
bd92c230da refactor txn reference tracking, do deep marking of DAGs 2021-07-04 18:38:28 +03:00
vyzo
a98a062347 do the dag walk for deep write tracking during flush
avoid crawling everything to a halt
2021-07-04 18:38:28 +03:00
vyzo
13a674330f add pending write check before tracking the object in Has 2021-07-04 18:38:28 +03:00
vyzo
982867317e transitively track dags from implicit writes in Has 2021-07-04 18:38:28 +03:00
vyzo
4de0cd9fcb move write log back to flush so that we don't crawl to a halt 2021-07-04 18:38:28 +03:00
vyzo
b3ddaa5f02 fix panic at startup
genesis is written (!) before starting the splitstore, so curTs is nil
2021-07-04 18:38:28 +03:00
vyzo
2faa4aa993 debug log writes at track so that we get correct stack traces 2021-07-04 18:38:28 +03:00
vyzo
aeaa59d4b5 move comments about tracking perf issues into a more pertinent place 2021-07-04 18:38:28 +03:00
vyzo
3e8e9273ca track all writes using async batching, not just implicit ones 2021-07-04 18:38:28 +03:00
vyzo
d0bfe421b5 flush implicit writes at the right time before starting compaction to avoid races 2021-07-04 18:38:28 +03:00
vyzo
7f473f56eb flush implicit writes before starting compaction 2021-07-04 18:38:28 +03:00