Commit Graph

463 Commits

Author SHA1 Message Date
vyzo
a9d4495d83 use both hot and cold when doing fetches for markset positive objects 2022-01-30 22:47:20 +02:00
vyzo
7b8447a95a reinstante waitForMissingRefs 2022-01-30 19:00:52 +02:00
vyzo
7931f1f8f9 fix lint 2022-01-30 15:43:52 +02:00
vyzo
20b75022aa fix mockStore for splitstore tests 2022-01-30 15:33:30 +02:00
vyzo
dbc8903bac sortless compaction 2022-01-30 15:33:15 +02:00
vyzo
a4f720d866 coldset test 2022-01-30 15:11:25 +02:00
vyzo
4b8369c071 fix buffered reads 2022-01-30 15:11:18 +02:00
vyzo
72333147d3 prettier checkpoint close 2022-01-30 15:00:03 +02:00
vyzo
64cda4aeda on disk coldsets 2022-01-30 14:19:05 +02:00
vyzo
6ede77b254 checkpoint test 2022-01-30 14:10:11 +02:00
vyzo
c94eee5fc3 on disk checkpoints 2022-01-30 12:47:30 +02:00
vyzo
322b85898f make markSets synchronous in critical section 2022-01-30 12:10:08 +02:00
vyzo
cf09dd044a moar markset tests 2022-01-30 11:45:12 +02:00
vyzo
a4c1a34163 check for existence of badger db in recover 2022-01-30 11:44:27 +02:00
vyzo
d140909562 add MarkMany to MarkSet interface 2022-01-30 11:43:45 +02:00
vyzo
f9fd47e7d0 use temporary dir for splitstore test path 2022-01-29 13:11:58 +02:00
vyzo
67fbf9eb00 improve peristence test 2022-01-28 15:55:34 +02:00
vyzo
730aceac6d immediately flush pending writes when entering critical section 2022-01-28 15:55:10 +02:00
vyzo
1bf396f9af add test for markset persistence 2022-01-28 15:49:41 +02:00
vyzo
45c2f34295 refactor marksets for critical section on-disk persistence 2022-01-28 15:41:33 +02:00
vyzo
f07ce297f6 optimize slice allocations in walk 2022-01-26 21:55:24 +02:00
vyzo
a87239e802 avoid extraneous assignment 2022-01-26 21:48:03 +02:00
vyzo
fe47d6a1a4 fix check and warmup for parallel walk 2022-01-26 09:01:51 +02:00
vyzo
10f2445a99 use minimum of 2 workers 2022-01-25 21:37:48 +02:00
vyzo
8e01e73de4 dynamically compute number of workers for parallel chain walk 2022-01-25 19:47:58 +02:00
vyzo
7c8edf5632 parallelize walkChain 2022-01-25 17:29:02 +02:00
vyzo
25284b5325 refactor: eliminate distinction between markset and markset visitors 2022-01-25 16:31:45 +02:00
Aayush Rajasekaran
3c57b980e5
Merge pull request from filecoin-project/steb/buffered-blockstore-correctness
correctness fixes for the autobatch blockstore
2022-01-12 19:17:03 -05:00
Steven Allen
2a862d497f correctness fixes for the autobatch blockstore
1. Simplify shutdown and make it idempotent by using a context.
2. Make sure `Flush` actually _fully_ flushes if the previous flush failed.
3. Don't clear the flush batch if flushing fails.
2022-01-12 15:19:44 -08:00
Aayush Rajasekaran
efe9ec4906 Add missing locks to Get() 2022-01-12 17:22:54 -05:00
Aayush Rajasekaran
3464dc2fdf Don't lock in Get 2022-01-12 16:04:47 -05:00
Aayush Rajasekaran
893998cb70 Address review part 2 2022-01-12 15:03:34 -05:00
Aayush Rajasekaran
083c5b003c Address review 2022-01-12 12:57:34 -05:00
Aayush Rajasekaran
7559e4311e Support faster Get, retry flushes on error 2022-01-11 20:30:34 -05:00
Aayush Rajasekaran
a41b4acec3 Use channels to trigger flushes in a dedicated goroutine 2022-01-11 19:44:56 -05:00
Aayush Rajasekaran
5ff6148444 implement stubs 2022-01-11 18:03:37 -05:00
Aayush Rajasekaran
544cfa63ab cache added cids 2022-01-11 17:22:36 -05:00
Aayush Rajasekaran
25768a291e Implement an autobatcher 2022-01-11 17:20:10 -05:00
vyzo
976a3a5e7b fix blockstore test contexts 2021-12-17 13:44:05 +02:00
vyzo
dd327f0b22 plumb more contexts 2021-12-17 11:42:09 +02:00
vyzo
40c61a310f fix splitstore test 2021-12-14 17:17:30 +02:00
vyzo
84710cf27e plumb more contexts in lotus 2021-12-13 14:26:59 +02:00
Aayush Rajasekaran
dfb65ed89f Plumb contexts through 2021-12-11 17:04:00 -05:00
whyrusleeping
072297e661 WIP: updating to new datastore/blockstore code with contexts 2021-12-09 14:12:25 +02:00
vyzo
489782e21b satisfy the spellchecker that masquarades as a linter 2021-12-03 12:15:28 +02:00
vyzo
6ce5879071 add unit test for compaction supression 2021-12-03 12:05:15 +02:00
vyzo
05c1325885 add logic for supressing compaction near upgrade boundaries 2021-12-03 11:50:35 +02:00
Aayush Rajasekaran
b3966ab4eb
Fix typo in comment 2021-08-24 20:39:28 -04:00
Steven Allen
cbf69f6ded fix: make TestTimedCacheBlockstoreSimple pass reliably 2021-08-24 11:35:56 -07:00
vyzo
eb0a62e9e9 RIP bloom filter markset; you weren't used anyway. 2021-08-10 10:47:01 +03:00
vyzo
742c85bf81 add SupportsVisitor in the markset env interface 2021-08-10 10:47:01 +03:00
vyzo
79f348a01c make the write lock scope limited within a function
Also avoid removing the writing set if there was an error while writing.
2021-08-10 10:47:01 +03:00
vyzo
a9403b42f0 deduplicate put code 2021-08-10 10:47:01 +03:00
vyzo
1a59b73adc add .tmp suffix to transient db names 2021-08-10 10:47:01 +03:00
vyzo
26a5832f92 simplify and deduplicate Has/Visit using helper methods tryPending and tryDB. 2021-08-10 10:47:01 +03:00
vyzo
380e16d465 remove redundant writers state variable 2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6 widen MarkSetVisitor interface and get rid of the casts 2021-08-10 10:47:01 +03:00
vyzo
57c984cea1 improve concurrency properties of Visit with optimistic concurrency 2021-08-10 10:47:01 +03:00
vyzo
563fa1e31a take advantage of MarkSet/Visitor unification to atomically check+mark 2021-08-10 10:47:01 +03:00
vyzo
cb3c53664d unify marksets and visitors 2021-08-10 10:47:01 +03:00
vyzo
3c994d94aa deduplicate some code in markset_badger 2021-08-10 10:47:01 +03:00
vyzo
32d94d4e85 reset walked set as epoch boundaries are crossed
the walk is BFS, so we can do this!
2021-08-10 10:47:01 +03:00
vyzo
49346f5679 use visitors instead of cidsets in walks 2021-08-10 10:47:01 +03:00
vyzo
1323d8fb20 markset-backed visitors 2021-08-10 10:47:01 +03:00
vyzo
6f22cffb6b object visitor interface 2021-08-10 10:47:01 +03:00
vyzo
a768dd84fa document splitstore clear shed command 2021-08-01 19:18:34 +03:00
vyzo
5e8b2c7860 make symlink helper freestanding 2021-07-29 08:35:53 +03:00
vyzo
b75ff37448
fix typo
Co-authored-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-07-28 17:11:04 +03:00
vyzo
c65d72fbca fix format specifier 2021-07-28 16:20:25 +03:00
vyzo
fff1c0ae57 improve detection of relative links 2021-07-28 16:15:39 +03:00
vyzo
297c9e2f7a extend test to check the validity of relative links 2021-07-28 11:56:35 +03:00
vyzo
0740274b7c make relative links when the canonical and new db paths are in the same directory 2021-07-28 11:56:23 +03:00
vyzo
950ded454d code cosmetics: rename variables for better readability and some comments 2021-07-28 11:49:42 +03:00
vyzo
da3e36acdc
Merge branch 'master' into feat/moving-gc 2021-07-27 12:12:38 +03:00
vyzo
8809707158 update README 2021-07-27 12:08:39 +03:00
vyzo
c21c413695 remove db2 to dbNext 2021-07-27 12:06:40 +03:00
vyzo
bb2d99908c make state constants typed 2021-07-27 12:03:26 +03:00
vyzo
b82f953fd5 use the badger streaming interface in doCopy 2021-07-27 11:46:35 +03:00
vyzo
59aebba0d9 use a slab allocator for the copy 2021-07-27 11:35:21 +03:00
vyzo
0baeec0686 remove filter from doCopy; it's not used
it was there to support a potential CopyTo interface; but we'll cross that bridge when we get there.
2021-07-27 11:18:24 +03:00
vyzo
649fc62863 panic if we fail to correctly setup the db paths.
we can't really continue and leave a ticking bomb for the next restart; the user might not see it.
2021-07-27 10:27:16 +03:00
vyzo
d6ace68540 extend test to do a double move and check symlink following 2021-07-27 10:16:50 +03:00
vyzo
cbaffab9dd use EvalSymlinks in deleteDB 2021-07-27 10:08:07 +03:00
vyzo
c03859c1b5 resolve symlinks when constructing the new db path
so that the new path is adjacent to the old path, allowing the user to symlink the db in
a different file system.
2021-07-27 10:05:35 +03:00
vyzo
9d25464703 use functional options for hotstore gc, rename MovingGC to FullGC 2021-07-27 09:53:22 +03:00
vyzo
96c1123c33 use functional options in the BlockstoreGC interface 2021-07-27 09:52:06 +03:00
vyzo
c52c2738ab fix potential underflow 2021-07-26 18:38:42 +03:00
vyzo
18caefc4a1 fix missing object condition; short-circuit walk 2021-07-26 18:34:05 +03:00
vyzo
0fd8fc24e1 load a full finality worth of state during warmup 2021-07-26 18:34:05 +03:00
vyzo
938330e6c3 moveMx is not an RWMutex, just a regular mutex 2021-07-26 15:09:31 +03:00
vyzo
21e7c188da use CollectGarbage in blockstore move test, as it is the real interface 2021-07-26 15:05:57 +03:00
vyzo
b1f60e85e9 document moving GC frequency option in README 2021-07-26 15:05:57 +03:00
vyzo
5acae50e07 add config option for splitstore moving gc frequency 2021-07-26 15:05:57 +03:00
vyzo
fb3986226f do hotstore moving GC in splitstore with a user-specified frequency 2021-07-26 15:05:57 +03:00
vyzo
c747f2f1e2 do moving GC if the user asks for it 2021-07-26 15:05:57 +03:00
vyzo
a843665132 add options to BlockstoreGC trait 2021-07-26 15:05:57 +03:00
vyzo
94509968a0 make moveTo a private method 2021-07-26 15:05:20 +03:00
vyzo
5cf6fdf81d don't heap allocate the cond, just set L 2021-07-26 15:04:46 +03:00
vyzo
524564e2cf add some more logging around move 2021-07-26 15:04:46 +03:00
vyzo
f2c7b08be5 follow symbolic links when deleting old dbs 2021-07-26 15:04:46 +03:00