ZenGround0
4ffded6fef
feat:splitstore:single compaction that can handle prune aka two marksets one compaction ( #9571 )
...
* begin
* rough draft -- this should probably actually work?
* WIP
* Start testing
* message mode
* Fix tests, make gen
* Better default
* docsgen-cli
* Review Response
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-11-07 16:31:12 -05:00
Łukasz Magiera
08b22edd89
fix make gen
2022-08-29 16:25:30 +02:00
ZenGround0
8b7be6d47e
feat:chain:splitstore auto prune ( #9123 )
...
Auto Prune
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-08-08 16:06:32 -04:00
ZenGround0
0c91b0dc10
feat:chain:splitstore chain prune ( #9056 )
...
* Splitstore chain prune
* Protect on reification for simpler logic and sound cold compact protect
* Recovery from checkpoint during chain prune
* Splitstore (discard and universal mode) running in itests
* Add pause and restart functions to itest block miner
* Add config options to itest full nodes
* Add FsRepo support for itest full ndoes
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2022-08-05 16:34:16 -04:00
Masih H. Derkani
6f686cae11
Upgrade to go-blockstore 1.2.0 and go-cid 0.2.0
...
Upgrade required dependencies and fix breaking changes.
2022-06-29 14:55:22 -04:00
Łukasz Magiera
05cdeb80c3
chore: remove redundant import prefixes
2022-06-15 12:06:22 +02:00
Łukasz Magiera
e65fae28de
chore: fix imports
2022-06-14 17:00:51 +02:00
Łukasz Magiera
1c055fe83b
Merge branch 'master' into test/t.TempDir
2022-03-17 12:06:52 +01:00
Darko Brdareski
79453663b3
feat: Add additional test annotations ( #8272 )
...
* Annotate api,proxy_util,blockstore_badger, policy tests
* Annotate splitstore: bsbadger / markset
* Annotate splitstore feature
* Annotate union/timed blockstore tests
* Annotate openrpc, diff_adt tests
* Annotate error,drand,events tests
* Annotate predicates_test
* Fix annotations
* Annotate tscache, gen tests
* Annotate fundmanager test
* Annotate repub and selection tests
* Annotate statetree_test
* Annotate forks_test
* Annotate searchwait_test.go
* Fix duplicated @@ symbols
* Annotate chain stmgr/store tests
* Annotate more (types) tests
* More tests annotated
* Annotate conformance chaos actor tests
* Annotate more integration tests
* Annotate journal system tests
* Annotate more tests.
* Annotate gas,head buffer behaviors
* Fix markset annotations
* doc: test annotations for the markets dagstore wrapper
* Annotate miner_api test in dagstore
* Annotate more test files
* Remove bad annotations from fsrepo
* Annotate wdpost system
* Remove bad annotations
* Renamce "conformance" to "chaos_actor" tests
* doc: stm annotations for blockheader & election proof tests
* Annotate remaining "A" tests
* annotate: stm for error_test
* memrepo_test.go
* Annotate "b" file tests
* message_test.go
* doc: stm annotate for fsrepo_test
* Annotate "c" file tests
* Annotate "D" test files
* message_test.go
* doc: stm annotate for chain, node/config & client
* docs: stm annotate node_test
* Annotate u,v,wl tests
* doc: stm annotations for various test files
* Annotate "T" test files
* doc: stm annotate for proxy_util_test & policy_test
* doc: stm annotate for various tests
* doc: final few stm annotations
* Add mempool unit tests
* Add two more memPool Add tests
* Update submodules
* Add check function tests
* Add stm annotations, refactor test helper
* Annotate api,proxy_util,blockstore_badger, policy tests
* Annotate splitstore: bsbadger / markset
solving merge conflicts
* Annotate splitstore feature
* Annotate union/timed blockstore tests
* Annotate openrpc, diff_adt tests
* Annotate error,drand,events tests
* Annotate predicates_test
* Fix annotations
* Annotate tscache, gen tests
* Annotate fundmanager test
* Annotate statetree_test
* Annotate forks_test
* Annotate searchwait_test.go
* Fix duplicated @@ symbols
* Annotate chain stmgr/store tests
* Annotate more (types) tests
* More tests annotated
* Annotate conformance chaos actor tests
* Annotate more integration tests
* Annotate journal system tests
* Annotate more tests.
* Annotate gas,head buffer behaviors
solve merge conflict
* Fix markset annotations
* Annotate miner_api test in dagstore
* Annotate more test files
* doc: test annotations for the markets dagstore wrapper
* Annotate wdpost system
* Renamce "conformance" to "chaos_actor" tests
* Annotate remaining "A" tests
* doc: stm annotations for blockheader & election proof tests
* annotate: stm for error_test
* Annotate "b" file tests
* memrepo_test.go
* Annotate "c" file tests
* message_test.go
* Annotate "D" test files
* doc: stm annotate for fsrepo_test
* Annotate u,v,wl tests
* message_test.go
* doc: stm annotate for chain, node/config & client
* docs: stm annotate node_test
* Annotate "T" test files
* doc: stm annotations for various test files
* Add mempool unit tests
solve merge conflict
* doc: stm annotate for proxy_util_test & policy_test
* doc: stm annotate for various tests
* doc: final few stm annotations
* Add two more memPool Add tests
* Update submodules
* Add check function tests
solve conflict
* Add stm annotations, refactor test helper
solve merge conflict
* Change CLI test kinds to "unit"
* Fix double merged test
* Fix ccupgrade_test merge
* Fix lint issues
* Add stm annotation to types_Test
* Test vectors submodule
* Add file annotation to burn_test
Co-authored-by: Nikola Divic <divicnikola@gmail.com>
Co-authored-by: TheMenko <themenkoprojects@gmail.com>
2022-03-16 18:37:34 +01:00
Eng Zer Jun
c229842f57
test: use T.TempDir to create temporary test directory
...
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-15 00:21:31 +08:00
vyzo
a7b1d86533
make cidset (in memory) visitors smarter; no need to ever visit unitary objects
2022-02-18 12:35:42 +02:00
vyzo
2795995989
add reification limit test
2022-02-18 12:19:19 +02:00
vyzo
cbd23c2b1b
add reification limit
2022-02-18 12:19:09 +02:00
vyzo
899a65ae87
fix test
2022-02-17 20:13:46 +02:00
vyzo
a20c6cb04b
temporarily disable reification
...
big reifications can use a lot of memory during sync apparently.
2022-02-17 19:56:50 +02:00
vyzo
74556edcff
don't fail reification on missing references
2022-02-17 12:52:52 +02:00
Łukasz Magiera
7efed6617c
Merge pull request #8029 from filecoin-project/feat/splistore-cold-object-reification-redux
...
splistore cold object reification redux
2022-02-14 19:04:02 +01:00
vyzo
6bcade5e6d
add comment about bigness of reification batch
2022-02-14 16:13:54 +02:00
vyzo
4524fbe936
wait for reify workers to finish when closing
2022-02-14 16:10:54 +02:00
vyzo
6c7ababd3f
add comment about trackTxnRefs being noops if txnActive is false
2022-02-14 16:06:12 +02:00
vyzo
a428f44793
don't reify objects while still warming up
2022-02-14 16:04:39 +02:00
vyzo
b576785aac
rename GetHotView to IsHotView
2022-02-14 16:03:17 +02:00
vyzo
e129ae3706
refactor nextBatch in badger markset
2022-02-09 18:34:50 +02:00
vyzo
8ddf4766ea
update README
...
Adds note about 3k IOPs requirement with badger markset, updates the memory requirement for
map to 48G based on observed behaviour of test nodes.
2022-02-08 10:10:51 +02:00
vyzo
d45e2074a6
update README for map as the default
2022-02-06 11:25:27 +02:00
vyzo
713edd565c
fix lint
2022-02-05 21:30:53 +02:00
vyzo
a32b7a32f3
directly mark objects in cold object reification
2022-02-05 20:00:15 +02:00
vyzo
929a05e898
add reification test
2022-02-04 16:16:34 +02:00
vyzo
268366e446
cold object reification context option
2022-02-04 16:07:58 +02:00
vyzo
4cddfd1074
background cold object reification
2022-02-04 14:29:50 +02:00
vyzo
049b48928f
add note about compaction algorithm changes in README
2022-02-02 14:39:21 +02:00
vyzo
75ad0c3c2d
badger markset option tweaks
2022-02-02 14:34:30 +02:00
vyzo
c1d8368fdc
share a concurrent visitor between workers in markLiveRefs
2022-02-01 21:40:09 +02:00
vyzo
4b4104e400
fix comment
2022-02-01 12:10:30 +02:00
vyzo
b13aa8f172
unblock waitForSync on close
2022-02-01 11:30:21 +02:00
vyzo
9c92d77b64
improve robustness of waitForSync
2022-02-01 11:25:07 +02:00
vyzo
cd95892bf0
fix test
2022-02-01 11:13:01 +02:00
vyzo
7b4ab2077b
wait for sync in a non racey way
2022-02-01 11:10:51 +02:00
vyzo
578b5691bc
check for closing after the sync wait
2022-02-01 10:59:08 +02:00
vyzo
6353fa72d8
decouple SyncGapTime from wait time
2022-02-01 10:55:15 +02:00
vyzo
fd07ca87ce
wait for the sync gap time befor starting the purge
2022-02-01 10:46:42 +02:00
vyzo
11ae85645b
optimize single object marking in markLiveRefs
2022-02-01 09:13:21 +02:00
vyzo
3aabb0314a
synchronously mark live refs on put/putmany
2022-02-01 09:07:14 +02:00
vyzo
37673c6de6
downgrade marking log to debug
2022-01-31 21:44:10 +02:00
vyzo
7896af731f
use walkObjectIncomplete for marking live refs
2022-01-31 21:41:42 +02:00
vyzo
877dfbe992
hold the lock in the second protect call
2022-01-31 19:21:03 +02:00
vyzo
5b9ea1b4e0
avoid races in beginCriticalSection
2022-01-31 18:48:53 +02:00
vyzo
710fda49f0
fix putmany marking
2022-01-31 17:35:30 +02:00
vyzo
2b14bda6b8
recursively mark puts during the critical section
2022-01-31 17:26:08 +02:00
vyzo
1abfc5b1cf
fix comment
2022-01-31 15:00:03 +02:00
