Commit Graph

419 Commits

Author SHA1 Message Date
vyzo
899a65ae87 fix test 2022-02-17 20:13:46 +02:00
vyzo
a20c6cb04b temporarily disable reification
big reifications can use a lot of memory during sync apparently.
2022-02-17 19:56:50 +02:00
vyzo
74556edcff don't fail reification on missing references 2022-02-17 12:52:52 +02:00
Łukasz Magiera
7efed6617c
Merge pull request from filecoin-project/feat/splistore-cold-object-reification-redux
splistore cold object reification redux
2022-02-14 19:04:02 +01:00
vyzo
6bcade5e6d add comment about bigness of reification batch 2022-02-14 16:13:54 +02:00
vyzo
4524fbe936 wait for reify workers to finish when closing 2022-02-14 16:10:54 +02:00
vyzo
6c7ababd3f add comment about trackTxnRefs being noops if txnActive is false 2022-02-14 16:06:12 +02:00
vyzo
a428f44793 don't reify objects while still warming up 2022-02-14 16:04:39 +02:00
vyzo
b576785aac rename GetHotView to IsHotView 2022-02-14 16:03:17 +02:00
vyzo
e129ae3706 refactor nextBatch in badger markset 2022-02-09 18:34:50 +02:00
vyzo
8ddf4766ea update README
Adds note about 3k IOPs requirement with badger markset, updates the memory requirement for
map to 48G based on observed behaviour of test nodes.
2022-02-08 10:10:51 +02:00
vyzo
d45e2074a6 update README for map as the default 2022-02-06 11:25:27 +02:00
vyzo
713edd565c fix lint 2022-02-05 21:30:53 +02:00
vyzo
a32b7a32f3 directly mark objects in cold object reification 2022-02-05 20:00:15 +02:00
vyzo
929a05e898 add reification test 2022-02-04 16:16:34 +02:00
vyzo
268366e446 cold object reification context option 2022-02-04 16:07:58 +02:00
vyzo
4cddfd1074 background cold object reification 2022-02-04 14:29:50 +02:00
vyzo
049b48928f add note about compaction algorithm changes in README 2022-02-02 14:39:21 +02:00
vyzo
75ad0c3c2d badger markset option tweaks 2022-02-02 14:34:30 +02:00
vyzo
c1d8368fdc share a concurrent visitor between workers in markLiveRefs 2022-02-01 21:40:09 +02:00
vyzo
4b4104e400 fix comment 2022-02-01 12:10:30 +02:00
vyzo
b13aa8f172 unblock waitForSync on close 2022-02-01 11:30:21 +02:00
vyzo
9c92d77b64 improve robustness of waitForSync 2022-02-01 11:25:07 +02:00
vyzo
cd95892bf0 fix test 2022-02-01 11:13:01 +02:00
vyzo
7b4ab2077b wait for sync in a non racey way 2022-02-01 11:10:51 +02:00
vyzo
578b5691bc check for closing after the sync wait 2022-02-01 10:59:08 +02:00
vyzo
6353fa72d8 decouple SyncGapTime from wait time 2022-02-01 10:55:15 +02:00
vyzo
fd07ca87ce wait for the sync gap time befor starting the purge 2022-02-01 10:46:42 +02:00
vyzo
11ae85645b optimize single object marking in markLiveRefs 2022-02-01 09:13:21 +02:00
vyzo
3aabb0314a synchronously mark live refs on put/putmany 2022-02-01 09:07:14 +02:00
vyzo
37673c6de6 downgrade marking log to debug 2022-01-31 21:44:10 +02:00
vyzo
7896af731f use walkObjectIncomplete for marking live refs 2022-01-31 21:41:42 +02:00
vyzo
877dfbe992 hold the lock in the second protect call 2022-01-31 19:21:03 +02:00
vyzo
5b9ea1b4e0 avoid races in beginCriticalSection 2022-01-31 18:48:53 +02:00
vyzo
710fda49f0 fix putmany marking 2022-01-31 17:35:30 +02:00
vyzo
2b14bda6b8 recursively mark puts during the critical section 2022-01-31 17:26:08 +02:00
vyzo
1abfc5b1cf fix comment 2022-01-31 15:00:03 +02:00
vyzo
c9bd5ec452 mark tipset references to protect them during critical section 2022-01-31 14:54:23 +02:00
vyzo
ee63be26a1 fix race in protectView 2022-01-31 12:40:26 +02:00
vyzo
1900c907fd account for missing refs in the markset in Has 2022-01-31 09:44:51 +02:00
vyzo
a9d4495d83 use both hot and cold when doing fetches for markset positive objects 2022-01-30 22:47:20 +02:00
vyzo
7b8447a95a reinstante waitForMissingRefs 2022-01-30 19:00:52 +02:00
vyzo
7931f1f8f9 fix lint 2022-01-30 15:43:52 +02:00
vyzo
20b75022aa fix mockStore for splitstore tests 2022-01-30 15:33:30 +02:00
vyzo
dbc8903bac sortless compaction 2022-01-30 15:33:15 +02:00
vyzo
a4f720d866 coldset test 2022-01-30 15:11:25 +02:00
vyzo
4b8369c071 fix buffered reads 2022-01-30 15:11:18 +02:00
vyzo
72333147d3 prettier checkpoint close 2022-01-30 15:00:03 +02:00
vyzo
64cda4aeda on disk coldsets 2022-01-30 14:19:05 +02:00
vyzo
6ede77b254 checkpoint test 2022-01-30 14:10:11 +02:00
vyzo
c94eee5fc3 on disk checkpoints 2022-01-30 12:47:30 +02:00
vyzo
322b85898f make markSets synchronous in critical section 2022-01-30 12:10:08 +02:00
vyzo
cf09dd044a moar markset tests 2022-01-30 11:45:12 +02:00
vyzo
a4c1a34163 check for existence of badger db in recover 2022-01-30 11:44:27 +02:00
vyzo
d140909562 add MarkMany to MarkSet interface 2022-01-30 11:43:45 +02:00
vyzo
f9fd47e7d0 use temporary dir for splitstore test path 2022-01-29 13:11:58 +02:00
vyzo
67fbf9eb00 improve peristence test 2022-01-28 15:55:34 +02:00
vyzo
730aceac6d immediately flush pending writes when entering critical section 2022-01-28 15:55:10 +02:00
vyzo
1bf396f9af add test for markset persistence 2022-01-28 15:49:41 +02:00
vyzo
45c2f34295 refactor marksets for critical section on-disk persistence 2022-01-28 15:41:33 +02:00
vyzo
f07ce297f6 optimize slice allocations in walk 2022-01-26 21:55:24 +02:00
vyzo
a87239e802 avoid extraneous assignment 2022-01-26 21:48:03 +02:00
vyzo
fe47d6a1a4 fix check and warmup for parallel walk 2022-01-26 09:01:51 +02:00
vyzo
10f2445a99 use minimum of 2 workers 2022-01-25 21:37:48 +02:00
vyzo
8e01e73de4 dynamically compute number of workers for parallel chain walk 2022-01-25 19:47:58 +02:00
vyzo
7c8edf5632 parallelize walkChain 2022-01-25 17:29:02 +02:00
vyzo
25284b5325 refactor: eliminate distinction between markset and markset visitors 2022-01-25 16:31:45 +02:00
vyzo
40c61a310f fix splitstore test 2021-12-14 17:17:30 +02:00
Aayush Rajasekaran
dfb65ed89f Plumb contexts through 2021-12-11 17:04:00 -05:00
vyzo
489782e21b satisfy the spellchecker that masquarades as a linter 2021-12-03 12:15:28 +02:00
vyzo
6ce5879071 add unit test for compaction supression 2021-12-03 12:05:15 +02:00
vyzo
05c1325885 add logic for supressing compaction near upgrade boundaries 2021-12-03 11:50:35 +02:00
vyzo
eb0a62e9e9 RIP bloom filter markset; you weren't used anyway. 2021-08-10 10:47:01 +03:00
vyzo
742c85bf81 add SupportsVisitor in the markset env interface 2021-08-10 10:47:01 +03:00
vyzo
79f348a01c make the write lock scope limited within a function
Also avoid removing the writing set if there was an error while writing.
2021-08-10 10:47:01 +03:00
vyzo
a9403b42f0 deduplicate put code 2021-08-10 10:47:01 +03:00
vyzo
1a59b73adc add .tmp suffix to transient db names 2021-08-10 10:47:01 +03:00
vyzo
26a5832f92 simplify and deduplicate Has/Visit using helper methods tryPending and tryDB. 2021-08-10 10:47:01 +03:00
vyzo
380e16d465 remove redundant writers state variable 2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6 widen MarkSetVisitor interface and get rid of the casts 2021-08-10 10:47:01 +03:00
vyzo
57c984cea1 improve concurrency properties of Visit with optimistic concurrency 2021-08-10 10:47:01 +03:00
vyzo
563fa1e31a take advantage of MarkSet/Visitor unification to atomically check+mark 2021-08-10 10:47:01 +03:00
vyzo
cb3c53664d unify marksets and visitors 2021-08-10 10:47:01 +03:00
vyzo
3c994d94aa deduplicate some code in markset_badger 2021-08-10 10:47:01 +03:00
vyzo
32d94d4e85 reset walked set as epoch boundaries are crossed
the walk is BFS, so we can do this!
2021-08-10 10:47:01 +03:00
vyzo
49346f5679 use visitors instead of cidsets in walks 2021-08-10 10:47:01 +03:00
vyzo
1323d8fb20 markset-backed visitors 2021-08-10 10:47:01 +03:00
vyzo
6f22cffb6b object visitor interface 2021-08-10 10:47:01 +03:00
vyzo
a768dd84fa document splitstore clear shed command 2021-08-01 19:18:34 +03:00
vyzo
da3e36acdc
Merge branch 'master' into feat/moving-gc 2021-07-27 12:12:38 +03:00
vyzo
8809707158 update README 2021-07-27 12:08:39 +03:00
vyzo
9d25464703 use functional options for hotstore gc, rename MovingGC to FullGC 2021-07-27 09:53:22 +03:00
vyzo
c52c2738ab fix potential underflow 2021-07-26 18:38:42 +03:00
vyzo
18caefc4a1 fix missing object condition; short-circuit walk 2021-07-26 18:34:05 +03:00
vyzo
0fd8fc24e1 load a full finality worth of state during warmup 2021-07-26 18:34:05 +03:00
vyzo
b1f60e85e9 document moving GC frequency option in README 2021-07-26 15:05:57 +03:00
vyzo
5acae50e07 add config option for splitstore moving gc frequency 2021-07-26 15:05:57 +03:00
vyzo
fb3986226f do hotstore moving GC in splitstore with a user-specified frequency 2021-07-26 15:05:57 +03:00
vyzo
74009bd67f document lotus-shed splitstore utiilities in the README 2021-07-26 08:52:32 +03:00
vyzo
ce528a1293 implement Info in splitstore 2021-07-26 08:45:46 +03:00