Łukasz Magiera
7efed6617c
Merge pull request #8029 from filecoin-project/feat/splistore-cold-object-reification-redux
...
splistore cold object reification redux
2022-02-14 19:04:02 +01:00
vyzo
6bcade5e6d
add comment about bigness of reification batch
2022-02-14 16:13:54 +02:00
vyzo
4524fbe936
wait for reify workers to finish when closing
2022-02-14 16:10:54 +02:00
vyzo
6c7ababd3f
add comment about trackTxnRefs being noops if txnActive is false
2022-02-14 16:06:12 +02:00
vyzo
a428f44793
don't reify objects while still warming up
2022-02-14 16:04:39 +02:00
vyzo
b576785aac
rename GetHotView to IsHotView
2022-02-14 16:03:17 +02:00
vyzo
e129ae3706
refactor nextBatch in badger markset
2022-02-09 18:34:50 +02:00
vyzo
8ddf4766ea
update README
...
Adds note about 3k IOPs requirement with badger markset, updates the memory requirement for
map to 48G based on observed behaviour of test nodes.
2022-02-08 10:10:51 +02:00
vyzo
d45e2074a6
update README for map as the default
2022-02-06 11:25:27 +02:00
vyzo
713edd565c
fix lint
2022-02-05 21:30:53 +02:00
vyzo
a32b7a32f3
directly mark objects in cold object reification
2022-02-05 20:00:15 +02:00
vyzo
929a05e898
add reification test
2022-02-04 16:16:34 +02:00
vyzo
268366e446
cold object reification context option
2022-02-04 16:07:58 +02:00
vyzo
4cddfd1074
background cold object reification
2022-02-04 14:29:50 +02:00
vyzo
049b48928f
add note about compaction algorithm changes in README
2022-02-02 14:39:21 +02:00
vyzo
75ad0c3c2d
badger markset option tweaks
2022-02-02 14:34:30 +02:00
vyzo
c1d8368fdc
share a concurrent visitor between workers in markLiveRefs
2022-02-01 21:40:09 +02:00
vyzo
4b4104e400
fix comment
2022-02-01 12:10:30 +02:00
vyzo
b13aa8f172
unblock waitForSync on close
2022-02-01 11:30:21 +02:00
vyzo
9c92d77b64
improve robustness of waitForSync
2022-02-01 11:25:07 +02:00
vyzo
cd95892bf0
fix test
2022-02-01 11:13:01 +02:00
vyzo
7b4ab2077b
wait for sync in a non racey way
2022-02-01 11:10:51 +02:00
vyzo
578b5691bc
check for closing after the sync wait
2022-02-01 10:59:08 +02:00
vyzo
6353fa72d8
decouple SyncGapTime from wait time
2022-02-01 10:55:15 +02:00
vyzo
fd07ca87ce
wait for the sync gap time befor starting the purge
2022-02-01 10:46:42 +02:00
vyzo
11ae85645b
optimize single object marking in markLiveRefs
2022-02-01 09:13:21 +02:00
vyzo
3aabb0314a
synchronously mark live refs on put/putmany
2022-02-01 09:07:14 +02:00
vyzo
37673c6de6
downgrade marking log to debug
2022-01-31 21:44:10 +02:00
vyzo
7896af731f
use walkObjectIncomplete for marking live refs
2022-01-31 21:41:42 +02:00
vyzo
877dfbe992
hold the lock in the second protect call
2022-01-31 19:21:03 +02:00
vyzo
5b9ea1b4e0
avoid races in beginCriticalSection
2022-01-31 18:48:53 +02:00
vyzo
710fda49f0
fix putmany marking
2022-01-31 17:35:30 +02:00
vyzo
2b14bda6b8
recursively mark puts during the critical section
2022-01-31 17:26:08 +02:00
vyzo
1abfc5b1cf
fix comment
2022-01-31 15:00:03 +02:00
vyzo
c9bd5ec452
mark tipset references to protect them during critical section
2022-01-31 14:54:23 +02:00
vyzo
ee63be26a1
fix race in protectView
2022-01-31 12:40:26 +02:00
vyzo
1900c907fd
account for missing refs in the markset in Has
2022-01-31 09:44:51 +02:00
vyzo
a9d4495d83
use both hot and cold when doing fetches for markset positive objects
2022-01-30 22:47:20 +02:00
vyzo
7b8447a95a
reinstante waitForMissingRefs
2022-01-30 19:00:52 +02:00
vyzo
7931f1f8f9
fix lint
2022-01-30 15:43:52 +02:00
vyzo
20b75022aa
fix mockStore for splitstore tests
2022-01-30 15:33:30 +02:00
vyzo
dbc8903bac
sortless compaction
2022-01-30 15:33:15 +02:00
vyzo
a4f720d866
coldset test
2022-01-30 15:11:25 +02:00
vyzo
4b8369c071
fix buffered reads
2022-01-30 15:11:18 +02:00
vyzo
72333147d3
prettier checkpoint close
2022-01-30 15:00:03 +02:00
vyzo
64cda4aeda
on disk coldsets
2022-01-30 14:19:05 +02:00
vyzo
6ede77b254
checkpoint test
2022-01-30 14:10:11 +02:00
vyzo
c94eee5fc3
on disk checkpoints
2022-01-30 12:47:30 +02:00
vyzo
322b85898f
make markSets synchronous in critical section
2022-01-30 12:10:08 +02:00
vyzo
cf09dd044a
moar markset tests
2022-01-30 11:45:12 +02:00
vyzo
a4c1a34163
check for existence of badger db in recover
2022-01-30 11:44:27 +02:00
vyzo
d140909562
add MarkMany to MarkSet interface
2022-01-30 11:43:45 +02:00
vyzo
f9fd47e7d0
use temporary dir for splitstore test path
2022-01-29 13:11:58 +02:00
vyzo
67fbf9eb00
improve peristence test
2022-01-28 15:55:34 +02:00
vyzo
730aceac6d
immediately flush pending writes when entering critical section
2022-01-28 15:55:10 +02:00
vyzo
1bf396f9af
add test for markset persistence
2022-01-28 15:49:41 +02:00
vyzo
45c2f34295
refactor marksets for critical section on-disk persistence
2022-01-28 15:41:33 +02:00
vyzo
f07ce297f6
optimize slice allocations in walk
2022-01-26 21:55:24 +02:00
vyzo
a87239e802
avoid extraneous assignment
2022-01-26 21:48:03 +02:00
vyzo
fe47d6a1a4
fix check and warmup for parallel walk
2022-01-26 09:01:51 +02:00
vyzo
10f2445a99
use minimum of 2 workers
2022-01-25 21:37:48 +02:00
vyzo
8e01e73de4
dynamically compute number of workers for parallel chain walk
2022-01-25 19:47:58 +02:00
vyzo
7c8edf5632
parallelize walkChain
2022-01-25 17:29:02 +02:00
vyzo
25284b5325
refactor: eliminate distinction between markset and markset visitors
2022-01-25 16:31:45 +02:00
Aayush Rajasekaran
3c57b980e5
Merge pull request #7940 from filecoin-project/steb/buffered-blockstore-correctness
...
correctness fixes for the autobatch blockstore
2022-01-12 19:17:03 -05:00
Steven Allen
2a862d497f
correctness fixes for the autobatch blockstore
...
1. Simplify shutdown and make it idempotent by using a context.
2. Make sure `Flush` actually _fully_ flushes if the previous flush failed.
3. Don't clear the flush batch if flushing fails.
2022-01-12 15:19:44 -08:00
Aayush Rajasekaran
efe9ec4906
Add missing locks to Get()
2022-01-12 17:22:54 -05:00
Aayush Rajasekaran
3464dc2fdf
Don't lock in Get
2022-01-12 16:04:47 -05:00
Aayush Rajasekaran
893998cb70
Address review part 2
2022-01-12 15:03:34 -05:00
Aayush Rajasekaran
083c5b003c
Address review
2022-01-12 12:57:34 -05:00
Aayush Rajasekaran
7559e4311e
Support faster Get, retry flushes on error
2022-01-11 20:30:34 -05:00
Aayush Rajasekaran
a41b4acec3
Use channels to trigger flushes in a dedicated goroutine
2022-01-11 19:44:56 -05:00
Aayush Rajasekaran
5ff6148444
implement stubs
2022-01-11 18:03:37 -05:00
Aayush Rajasekaran
544cfa63ab
cache added cids
2022-01-11 17:22:36 -05:00
Aayush Rajasekaran
25768a291e
Implement an autobatcher
2022-01-11 17:20:10 -05:00
vyzo
976a3a5e7b
fix blockstore test contexts
2021-12-17 13:44:05 +02:00
vyzo
dd327f0b22
plumb more contexts
2021-12-17 11:42:09 +02:00
vyzo
40c61a310f
fix splitstore test
2021-12-14 17:17:30 +02:00
vyzo
84710cf27e
plumb more contexts in lotus
2021-12-13 14:26:59 +02:00
Aayush Rajasekaran
dfb65ed89f
Plumb contexts through
2021-12-11 17:04:00 -05:00
whyrusleeping
072297e661
WIP: updating to new datastore/blockstore code with contexts
2021-12-09 14:12:25 +02:00
vyzo
489782e21b
satisfy the spellchecker that masquarades as a linter
2021-12-03 12:15:28 +02:00
vyzo
6ce5879071
add unit test for compaction supression
2021-12-03 12:05:15 +02:00
vyzo
05c1325885
add logic for supressing compaction near upgrade boundaries
2021-12-03 11:50:35 +02:00
Aayush Rajasekaran
b3966ab4eb
Fix typo in comment
2021-08-24 20:39:28 -04:00
Steven Allen
cbf69f6ded
fix: make TestTimedCacheBlockstoreSimple pass reliably
2021-08-24 11:35:56 -07:00
vyzo
eb0a62e9e9
RIP bloom filter markset; you weren't used anyway.
2021-08-10 10:47:01 +03:00
vyzo
742c85bf81
add SupportsVisitor in the markset env interface
2021-08-10 10:47:01 +03:00
vyzo
79f348a01c
make the write lock scope limited within a function
...
Also avoid removing the writing set if there was an error while writing.
2021-08-10 10:47:01 +03:00
vyzo
a9403b42f0
deduplicate put code
2021-08-10 10:47:01 +03:00
vyzo
1a59b73adc
add .tmp suffix to transient db names
2021-08-10 10:47:01 +03:00
vyzo
26a5832f92
simplify and deduplicate Has/Visit using helper methods tryPending and tryDB.
2021-08-10 10:47:01 +03:00
vyzo
380e16d465
remove redundant writers state variable
2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6
widen MarkSetVisitor interface and get rid of the casts
2021-08-10 10:47:01 +03:00
vyzo
57c984cea1
improve concurrency properties of Visit with optimistic concurrency
2021-08-10 10:47:01 +03:00
vyzo
563fa1e31a
take advantage of MarkSet/Visitor unification to atomically check+mark
2021-08-10 10:47:01 +03:00
vyzo
cb3c53664d
unify marksets and visitors
2021-08-10 10:47:01 +03:00
vyzo
3c994d94aa
deduplicate some code in markset_badger
2021-08-10 10:47:01 +03:00
vyzo
32d94d4e85
reset walked set as epoch boundaries are crossed
...
the walk is BFS, so we can do this!
2021-08-10 10:47:01 +03:00
vyzo
49346f5679
use visitors instead of cidsets in walks
2021-08-10 10:47:01 +03:00
