cerc-io/lotus
15
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
21,262 Commits 24 Branches 342 Tags 314 MiB
62961c3eee
Commit Graph

427 Commits

Author SHA1 Message Date
Łukasz Magiera
05cdeb80c3 chore: remove redundant import prefixes 2022-06-15 12:06:22 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Łukasz Magiera
1c055fe83b
Merge branch 'master' into test/t.TempDir 2022-03-17 12:06:52 +01:00
Darko Brdareski
79453663b3
feat: Add additional test annotations (#8272) 
* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate repub and selection tests

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

* Fix markset annotations

* doc: test annotations for the markets dagstore wrapper

* Annotate miner_api test in dagstore

* Annotate more test files

* Remove bad annotations from fsrepo

* Annotate wdpost system

* Remove bad annotations

* Renamce "conformance" to "chaos_actor" tests

* doc: stm annotations for blockheader & election proof tests

* Annotate remaining "A" tests

* annotate: stm for error_test

* memrepo_test.go

* Annotate "b" file tests

* message_test.go

* doc: stm annotate for fsrepo_test

* Annotate "c" file tests

* Annotate "D" test files

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate u,v,wl tests

* doc: stm annotations for various test files

* Annotate "T" test files

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add mempool unit tests

* Add two more memPool Add tests

* Update submodules

* Add check function tests

* Add stm annotations, refactor test helper

* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

solving merge conflicts

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

solve merge conflict

* Fix markset annotations

* Annotate miner_api test in dagstore

* Annotate more test files

* doc: test annotations for the markets dagstore wrapper

* Annotate wdpost system

* Renamce "conformance" to "chaos_actor" tests

* Annotate remaining "A" tests

* doc: stm annotations for blockheader & election proof tests

* annotate: stm for error_test

* Annotate "b" file tests

* memrepo_test.go

* Annotate "c" file tests

* message_test.go

* Annotate "D" test files

* doc: stm annotate for fsrepo_test

* Annotate u,v,wl tests

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate "T" test files

* doc: stm annotations for various test files

* Add mempool unit tests

solve merge conflict

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add two more memPool Add tests

* Update submodules

* Add check function tests

solve conflict

* Add stm annotations, refactor test helper

solve merge conflict

* Change CLI test kinds to "unit"

* Fix double merged test

* Fix ccupgrade_test merge

* Fix lint issues

* Add stm annotation to types_Test

* Test vectors submodule

* Add file annotation to burn_test

Co-authored-by: Nikola Divic <divicnikola@gmail.com>
Co-authored-by: TheMenko <themenkoprojects@gmail.com>
 2022-03-16 18:37:34 +01:00
Eng Zer Jun
c229842f57
test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-03-15 00:21:31 +08:00
vyzo
a7b1d86533 make cidset (in memory) visitors smarter; no need to ever visit unitary objects 2022-02-18 12:35:42 +02:00
vyzo
2795995989 add reification limit test 2022-02-18 12:19:19 +02:00
vyzo
cbd23c2b1b add reification limit 2022-02-18 12:19:09 +02:00
vyzo
899a65ae87 fix test 2022-02-17 20:13:46 +02:00
vyzo
a20c6cb04b temporarily disable reification 
big reifications can use a lot of memory during sync apparently.
 2022-02-17 19:56:50 +02:00
vyzo
74556edcff don't fail reification on missing references 2022-02-17 12:52:52 +02:00
Łukasz Magiera
7efed6617c
Merge pull request #8029 from filecoin-project/feat/splistore-cold-object-reification-redux 
splistore cold object reification redux
 2022-02-14 19:04:02 +01:00
vyzo
6bcade5e6d add comment about bigness of reification batch 2022-02-14 16:13:54 +02:00
vyzo
4524fbe936 wait for reify workers to finish when closing 2022-02-14 16:10:54 +02:00
vyzo
6c7ababd3f add comment about trackTxnRefs being noops if txnActive is false 2022-02-14 16:06:12 +02:00
vyzo
a428f44793 don't reify objects while still warming up 2022-02-14 16:04:39 +02:00
vyzo
b576785aac rename GetHotView to IsHotView 2022-02-14 16:03:17 +02:00
vyzo
e129ae3706 refactor nextBatch in badger markset 2022-02-09 18:34:50 +02:00
vyzo
8ddf4766ea update README 
Adds note about 3k IOPs requirement with badger markset, updates the memory requirement for
map to 48G based on observed behaviour of test nodes.
 2022-02-08 10:10:51 +02:00
vyzo
d45e2074a6 update README for map as the default 2022-02-06 11:25:27 +02:00
vyzo
713edd565c fix lint 2022-02-05 21:30:53 +02:00
vyzo
a32b7a32f3 directly mark objects in cold object reification 2022-02-05 20:00:15 +02:00
vyzo
929a05e898 add reification test 2022-02-04 16:16:34 +02:00
vyzo
268366e446 cold object reification context option 2022-02-04 16:07:58 +02:00
vyzo
4cddfd1074 background cold object reification 2022-02-04 14:29:50 +02:00
vyzo
049b48928f add note about compaction algorithm changes in README 2022-02-02 14:39:21 +02:00
vyzo
75ad0c3c2d badger markset option tweaks 2022-02-02 14:34:30 +02:00
vyzo
c1d8368fdc share a concurrent visitor between workers in markLiveRefs 2022-02-01 21:40:09 +02:00
vyzo
4b4104e400 fix comment 2022-02-01 12:10:30 +02:00
vyzo
b13aa8f172 unblock waitForSync on close 2022-02-01 11:30:21 +02:00
vyzo
9c92d77b64 improve robustness of waitForSync 2022-02-01 11:25:07 +02:00
vyzo
cd95892bf0 fix test 2022-02-01 11:13:01 +02:00
vyzo
7b4ab2077b wait for sync in a non racey way 2022-02-01 11:10:51 +02:00
vyzo
578b5691bc check for closing after the sync wait 2022-02-01 10:59:08 +02:00
vyzo
6353fa72d8 decouple SyncGapTime from wait time 2022-02-01 10:55:15 +02:00
vyzo
fd07ca87ce wait for the sync gap time befor starting the purge 2022-02-01 10:46:42 +02:00
vyzo
11ae85645b optimize single object marking in markLiveRefs 2022-02-01 09:13:21 +02:00
vyzo
3aabb0314a synchronously mark live refs on put/putmany 2022-02-01 09:07:14 +02:00
vyzo
37673c6de6 downgrade marking log to debug 2022-01-31 21:44:10 +02:00
vyzo
7896af731f use walkObjectIncomplete for marking live refs 2022-01-31 21:41:42 +02:00
vyzo
877dfbe992 hold the lock in the second protect call 2022-01-31 19:21:03 +02:00
vyzo
5b9ea1b4e0 avoid races in beginCriticalSection 2022-01-31 18:48:53 +02:00
vyzo
710fda49f0 fix putmany marking 2022-01-31 17:35:30 +02:00
vyzo
2b14bda6b8 recursively mark puts during the critical section 2022-01-31 17:26:08 +02:00
vyzo
1abfc5b1cf fix comment 2022-01-31 15:00:03 +02:00
vyzo
c9bd5ec452 mark tipset references to protect them during critical section 2022-01-31 14:54:23 +02:00
vyzo
ee63be26a1 fix race in protectView 2022-01-31 12:40:26 +02:00
vyzo
1900c907fd account for missing refs in the markset in Has 2022-01-31 09:44:51 +02:00
vyzo
a9d4495d83 use both hot and cold when doing fetches for markset positive objects 2022-01-30 22:47:20 +02:00
vyzo
7b8447a95a reinstante waitForMissingRefs 2022-01-30 19:00:52 +02:00
vyzo
7931f1f8f9 fix lint 2022-01-30 15:43:52 +02:00
vyzo
20b75022aa fix mockStore for splitstore tests 2022-01-30 15:33:30 +02:00
vyzo
dbc8903bac sortless compaction 2022-01-30 15:33:15 +02:00
vyzo
a4f720d866 coldset test 2022-01-30 15:11:25 +02:00
vyzo
4b8369c071 fix buffered reads 2022-01-30 15:11:18 +02:00
vyzo
72333147d3 prettier checkpoint close 2022-01-30 15:00:03 +02:00
vyzo
64cda4aeda on disk coldsets 2022-01-30 14:19:05 +02:00
vyzo
6ede77b254 checkpoint test 2022-01-30 14:10:11 +02:00
vyzo
c94eee5fc3 on disk checkpoints 2022-01-30 12:47:30 +02:00
vyzo
322b85898f make markSets synchronous in critical section 2022-01-30 12:10:08 +02:00
vyzo
cf09dd044a moar markset tests 2022-01-30 11:45:12 +02:00
vyzo
a4c1a34163 check for existence of badger db in recover 2022-01-30 11:44:27 +02:00
vyzo
d140909562 add MarkMany to MarkSet interface 2022-01-30 11:43:45 +02:00
vyzo
f9fd47e7d0 use temporary dir for splitstore test path 2022-01-29 13:11:58 +02:00
vyzo
67fbf9eb00 improve peristence test 2022-01-28 15:55:34 +02:00
vyzo
730aceac6d immediately flush pending writes when entering critical section 2022-01-28 15:55:10 +02:00
vyzo
1bf396f9af add test for markset persistence 2022-01-28 15:49:41 +02:00
vyzo
45c2f34295 refactor marksets for critical section on-disk persistence 2022-01-28 15:41:33 +02:00
vyzo
f07ce297f6 optimize slice allocations in walk 2022-01-26 21:55:24 +02:00
vyzo
a87239e802 avoid extraneous assignment 2022-01-26 21:48:03 +02:00
vyzo
fe47d6a1a4 fix check and warmup for parallel walk 2022-01-26 09:01:51 +02:00
vyzo
10f2445a99 use minimum of 2 workers 2022-01-25 21:37:48 +02:00
vyzo
8e01e73de4 dynamically compute number of workers for parallel chain walk 2022-01-25 19:47:58 +02:00
vyzo
7c8edf5632 parallelize walkChain 2022-01-25 17:29:02 +02:00
vyzo
25284b5325 refactor: eliminate distinction between markset and markset visitors 2022-01-25 16:31:45 +02:00
vyzo
40c61a310f fix splitstore test 2021-12-14 17:17:30 +02:00
Aayush Rajasekaran
dfb65ed89f Plumb contexts through 2021-12-11 17:04:00 -05:00
vyzo
489782e21b satisfy the spellchecker that masquarades as a linter 2021-12-03 12:15:28 +02:00
vyzo
6ce5879071 add unit test for compaction supression 2021-12-03 12:05:15 +02:00
vyzo
05c1325885 add logic for supressing compaction near upgrade boundaries 2021-12-03 11:50:35 +02:00
vyzo
eb0a62e9e9 RIP bloom filter markset; you weren't used anyway. 2021-08-10 10:47:01 +03:00
vyzo
742c85bf81 add SupportsVisitor in the markset env interface 2021-08-10 10:47:01 +03:00
vyzo
79f348a01c make the write lock scope limited within a function 
Also avoid removing the writing set if there was an error while writing.
 2021-08-10 10:47:01 +03:00
vyzo
a9403b42f0 deduplicate put code 2021-08-10 10:47:01 +03:00
vyzo
1a59b73adc add .tmp suffix to transient db names 2021-08-10 10:47:01 +03:00
vyzo
26a5832f92 simplify and deduplicate Has/Visit using helper methods tryPending and tryDB. 2021-08-10 10:47:01 +03:00
vyzo
380e16d465 remove redundant writers state variable 2021-08-10 10:47:01 +03:00
vyzo
b83b5405c6 widen MarkSetVisitor interface and get rid of the casts 2021-08-10 10:47:01 +03:00
vyzo
57c984cea1 improve concurrency properties of Visit with optimistic concurrency 2021-08-10 10:47:01 +03:00
vyzo
563fa1e31a take advantage of MarkSet/Visitor unification to atomically check+mark 2021-08-10 10:47:01 +03:00
vyzo
cb3c53664d unify marksets and visitors 2021-08-10 10:47:01 +03:00
vyzo
3c994d94aa deduplicate some code in markset_badger 2021-08-10 10:47:01 +03:00
vyzo
32d94d4e85 reset walked set as epoch boundaries are crossed 
the walk is BFS, so we can do this!
 2021-08-10 10:47:01 +03:00
vyzo
49346f5679 use visitors instead of cidsets in walks 2021-08-10 10:47:01 +03:00
vyzo
1323d8fb20 markset-backed visitors 2021-08-10 10:47:01 +03:00
vyzo
6f22cffb6b object visitor interface 2021-08-10 10:47:01 +03:00
vyzo
a768dd84fa document splitstore clear shed command 2021-08-01 19:18:34 +03:00
vyzo
da3e36acdc
Merge branch 'master' into feat/moving-gc 2021-07-27 12:12:38 +03:00
vyzo
8809707158 update README 2021-07-27 12:08:39 +03:00
vyzo
9d25464703 use functional options for hotstore gc, rename MovingGC to FullGC 2021-07-27 09:53:22 +03:00