vyzo
190cb18ab0
housekeeping
...
- remove defunct tracking store implementations
- update splitstore node config
- use mark set type config option (defaulting to mapts); a memory constrained node
may want to use an on-disk one
2021-07-04 18:38:28 +03:00
vyzo
19d1b1f532
deal with partially written objects
2021-07-04 18:38:28 +03:00
vyzo
0a1d7b3732
fix log
2021-07-04 18:38:28 +03:00
vyzo
08cad30be2
reuse key buffer in badger ForEachKey
...
cid copies the bytes so it's safe
2021-07-04 18:38:28 +03:00
vyzo
eafffc1634
more efficient trackTxnRefMany
2021-07-04 18:38:28 +03:00
vyzo
36f93649ef
fix panic from concurrent map writes in txnRefs
2021-07-04 18:38:28 +03:00
vyzo
6fa2cd232d
simplify compaction model
2021-07-04 18:38:28 +03:00
vyzo
5f7ae1f489
update splistore DI constructor
2021-07-04 18:38:28 +03:00
vyzo
1f2b604c07
RIP tracking store
2021-07-04 18:38:28 +03:00
vyzo
d476a3db2c
BlockstoreIterator trait with implementation for badger
2021-07-04 18:38:28 +03:00
vyzo
68a83500bc
fix bug that turned candidate filtering to dead code
2021-07-04 18:38:28 +03:00
vyzo
00fcf6dd72
add staging cache to bolt tracking store
2021-07-04 18:38:28 +03:00
vyzo
642f0e4740
deal with memory pressure, don't walk under the boundary
2021-07-04 18:38:28 +03:00
vyzo
c5cf8e226b
remove unnecessary code
2021-07-04 18:38:28 +03:00
vyzo
d79e4da7aa
more accurate stats about mark set updates
2021-07-04 18:38:28 +03:00
vyzo
6f58fdcb22
remove vm copy context detection hack
...
stack tracing is slow.
2021-07-04 18:38:28 +03:00
vyzo
2b03316cd9
fix log message
2021-07-04 18:38:28 +03:00
vyzo
184d3802b6
remove dead code
2021-07-04 18:38:28 +03:00
vyzo
228a435ba7
rework tracking logic; do it lazily and far more efficiently
2021-07-04 18:38:28 +03:00
vyzo
9d6cabd18a
if it's not a dag, it's not a block
2021-07-04 18:38:28 +03:00
vyzo
8157f889ce
short-circuit marking walks when encountering a block and more efficient walking
2021-07-04 18:38:28 +03:00
vyzo
736d6a3c19
only treat Has as an implicit write within vm.Copy context
2021-07-04 18:38:28 +03:00
vyzo
39723bbe60
use a single map for tracking pending writes, properly track implicits
2021-07-04 18:38:28 +03:00
vyzo
5834231e58
create the transactional protect filter before walking
2021-07-04 18:38:28 +03:00
vyzo
e4bb4be855
fix some residual purge races
2021-07-04 18:38:28 +03:00
vyzo
68bc5d2291
skip moving cold blocks when running with a noop coldstore
...
it is a noop but it still takes (a lot of) time because it has to read all the cold blocks.
2021-07-04 18:38:28 +03:00
vyzo
b87295db93
bubble up dependent txn ref errors
...
This cause Has to return false if it fails to traverse/protect all links, which would cause
the vm to recompute.
2021-07-04 18:38:28 +03:00
vyzo
637fbf6c5b
fix faulty if/else logic for implicit txn protection
2021-07-04 18:38:28 +03:00
vyzo
9d6bcd7705
avoid clown shoes: only walk links for tracking in implicit writes/refs
2021-07-04 18:38:28 +03:00
vyzo
484dfaebce
reused cidset across all walks when flushing pending writes
2021-07-04 18:38:28 +03:00
vyzo
1d41e1544a
optimize transitive write tracking a bit
2021-07-04 18:38:28 +03:00
vyzo
da00fc66ee
downgrade a couple of logs to warnings
2021-07-04 18:38:28 +03:00
vyzo
4071488ef2
first write, then track
2021-07-04 18:38:28 +03:00
vyzo
bd92c230da
refactor txn reference tracking, do deep marking of DAGs
2021-07-04 18:38:28 +03:00
vyzo
a98a062347
do the dag walk for deep write tracking during flush
...
avoid crawling everything to a halt
2021-07-04 18:38:28 +03:00
vyzo
13a674330f
add pending write check before tracking the object in Has
2021-07-04 18:38:28 +03:00
vyzo
982867317e
transitively track dags from implicit writes in Has
2021-07-04 18:38:28 +03:00
vyzo
4de0cd9fcb
move write log back to flush so that we don't crawl to a halt
2021-07-04 18:38:28 +03:00
vyzo
b3ddaa5f02
fix panic at startup
...
genesis is written (!) before starting the splitstore, so curTs is nil
2021-07-04 18:38:28 +03:00
vyzo
2faa4aa993
debug log writes at track so that we get correct stack traces
2021-07-04 18:38:28 +03:00
vyzo
aeaa59d4b5
move comments about tracking perf issues into a more pertinent place
2021-07-04 18:38:28 +03:00
vyzo
3e8e9273ca
track all writes using async batching, not just implicit ones
2021-07-04 18:38:28 +03:00
vyzo
d0bfe421b5
flush implicit writes at the right time before starting compaction to avoid races
2021-07-04 18:38:28 +03:00
vyzo
7f473f56eb
flush implicit writes before starting compaction
2021-07-04 18:38:28 +03:00
vyzo
a29947d47c
flush implicit writes in all paths in updateWriteEpoch
2021-07-04 18:38:28 +03:00
vyzo
be6cc2c3e6
batch implicit write tracking
...
bolt performance leaves something to be desired; doing a single Put takes 10ms, about the same time
as batching thousands of them.
2021-07-04 18:38:28 +03:00
vyzo
e472cacb3e
add missing return
2021-07-04 18:38:28 +03:00
vyzo
6a3cbea790
treat Has as an implicit Write
...
Rationale: the VM uses the Has check to avoid issuing a duplicate Write in the blockstore.
This means that live objects that would be otherwise written are not actually written, resulting
in the first write epoch being considered the write epoch.
2021-07-04 18:38:28 +03:00
vyzo
f97535d87e
store the hash in map markset
2021-07-04 18:38:28 +03:00
vyzo
90dc274113
better logging for chain walk
2021-07-04 18:38:28 +03:00
vyzo