cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
20,853 Commits 24 Branches 342 Tags 314 MiB
8f2931072e
Commit Graph

533 Commits

Author SHA1 Message Date
zenground0
0fe91846cd Plan out moving GC limiting 2023-03-07 07:39:17 -07:00
zenground0
5d9739a863 Track size of dags relevant to compaction 2023-03-07 07:39:17 -07:00
zenground0
dd998d6b24 Begin account for size during walks 2023-03-07 07:39:17 -07:00
zenground0
5534755f2d Add command to trigger moving GC of hotstore manually 2023-03-06 11:59:08 -07:00
zenground0
71b21db0d9 chain prune hot -- hotstore online gc 2023-03-03 11:14:52 -05:00
ZenGround0
544af751cb
Fix the overzealous fix (#10366) 
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2023-02-28 09:22:18 -05:00
ZenGround0
0c9f697bf6
Two fixes, better logging and comments (#10332) 
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2023-02-23 10:05:18 -05:00
Peter Rabbitson
818b3b8daf fix: extend LOTUS_CHAIN_BADGERSTORE_DISABLE_FSYNC to the markset 
Without doing this walking a badger markset on a non-nvme knocks the
node hopelessly out of sync during a compaction.
 2023-02-01 21:00:39 +01:00
Jorropo
f572852d06 chore: all: bump go-libipfs to replace go-block-format 
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58
 2023-01-26 17:03:18 +01:00
raulk
cdf3812e40
NV18: Filecoin EVM runtime + Actor Events + EthAccount + EAM + f4 addressing (#9998) 
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Raul Kripalani <raulk@users.noreply.github.com>
Co-authored-by: Kevin Li <ychiaoli18@users.noreply.github.com>
Co-authored-by: vyzo <vyzo@hackzen.org>
Co-authored-by: Ian Davis <nospam@iandavis.com>
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Jennifer Wang <jiayingw703@gmail.com>
Co-authored-by: Geoff Stuart <geoff.vball@gmail.com>
Co-authored-by: Shrenuj Bansal <shrenuj.bansal@protocol.ai>
Co-authored-by: Shrenuj Bansal <108157875+shrenujbansal@users.noreply.github.com>
Co-authored-by: Geoff Stuart <geoffrey.stuart@protocol.ai>
Co-authored-by: Aayush Rajasekaran <aayushrajasekaran@Aayushs-MacBook-Pro.local>
Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2023-01-13 19:11:13 +00:00
Łukasz Magiera
53e43a402a netbs: Drop client code for allkeys for now 2022-11-08 09:37:43 +00:00
Łukasz Magiera
401359646a netbs: Address review 2022-11-08 09:37:43 +00:00
Łukasz Magiera
1577740bc4 bstore: Network blockstore 2022-11-08 09:37:11 +00:00
ZenGround0
4ffded6fef
feat:splitstore:single compaction that can handle prune aka two marksets one compaction (#9571) 
* begin

* rough draft -- this should probably actually work?

* WIP

* Start testing

* message mode

* Fix tests, make gen

* Better default

* docsgen-cli

* Review Response

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2022-11-07 16:31:12 -05:00
Aayush
2e78158099 fix imports 2022-11-07 15:10:27 -05:00
Steven Allen
385f787ffc fix: autobatch: remove potential deadlock when a block is missing 
Check the _underlying_ blockstore instead of recursing. Also, drop the
lock before we do that.
 2022-11-07 10:46:51 -08:00
Łukasz Magiera
516ad171d2 fix: bstore: Handle codecs correctly in membstore Get 2022-10-12 10:03:22 +02:00
Łukasz Magiera
08b22edd89 fix make gen 2022-08-29 16:25:30 +02:00
ZenGround0
8b7be6d47e
feat:chain:splitstore auto prune (#9123) 
Auto Prune

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2022-08-08 16:06:32 -04:00
ZenGround0
0c91b0dc10
feat:chain:splitstore chain prune (#9056) 
* Splitstore chain prune
* Protect on reification for simpler logic and sound cold compact protect
* Recovery from checkpoint during chain prune
* Splitstore (discard and universal mode) running in itests
* Add pause and restart functions to itest block miner
* Add config options to itest full nodes
* Add FsRepo support for itest full ndoes

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2022-08-05 16:34:16 -04:00
Masih H. Derkani
6f686cae11 Upgrade to go-blockstore 1.2.0 and go-cid 0.2.0 
Upgrade required dependencies and fix breaking changes.
 2022-06-29 14:55:22 -04:00
Łukasz Magiera
05cdeb80c3 chore: remove redundant import prefixes 2022-06-15 12:06:22 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Geoff Stuart
b7010c9e60 Implement function to migrate actors with only code changes 2022-06-10 15:52:32 -04:00
Łukasz Magiera
1c055fe83b
Merge branch 'master' into test/t.TempDir 2022-03-17 12:06:52 +01:00
Darko Brdareski
79453663b3
feat: Add additional test annotations (#8272) 
* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate repub and selection tests

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

* Fix markset annotations

* doc: test annotations for the markets dagstore wrapper

* Annotate miner_api test in dagstore

* Annotate more test files

* Remove bad annotations from fsrepo

* Annotate wdpost system

* Remove bad annotations

* Renamce "conformance" to "chaos_actor" tests

* doc: stm annotations for blockheader & election proof tests

* Annotate remaining "A" tests

* annotate: stm for error_test

* memrepo_test.go

* Annotate "b" file tests

* message_test.go

* doc: stm annotate for fsrepo_test

* Annotate "c" file tests

* Annotate "D" test files

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate u,v,wl tests

* doc: stm annotations for various test files

* Annotate "T" test files

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add mempool unit tests

* Add two more memPool Add tests

* Update submodules

* Add check function tests

* Add stm annotations, refactor test helper

* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

solving merge conflicts

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

solve merge conflict

* Fix markset annotations

* Annotate miner_api test in dagstore

* Annotate more test files

* doc: test annotations for the markets dagstore wrapper

* Annotate wdpost system

* Renamce "conformance" to "chaos_actor" tests

* Annotate remaining "A" tests

* doc: stm annotations for blockheader & election proof tests

* annotate: stm for error_test

* Annotate "b" file tests

* memrepo_test.go

* Annotate "c" file tests

* message_test.go

* Annotate "D" test files

* doc: stm annotate for fsrepo_test

* Annotate u,v,wl tests

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate "T" test files

* doc: stm annotations for various test files

* Add mempool unit tests

solve merge conflict

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add two more memPool Add tests

* Update submodules

* Add check function tests

solve conflict

* Add stm annotations, refactor test helper

solve merge conflict

* Change CLI test kinds to "unit"

* Fix double merged test

* Fix ccupgrade_test merge

* Fix lint issues

* Add stm annotation to types_Test

* Test vectors submodule

* Add file annotation to burn_test

Co-authored-by: Nikola Divic <divicnikola@gmail.com>
Co-authored-by: TheMenko <themenkoprojects@gmail.com>
 2022-03-16 18:37:34 +01:00
Eng Zer Jun
c229842f57
test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-03-15 00:21:31 +08:00
vyzo
a7b1d86533 make cidset (in memory) visitors smarter; no need to ever visit unitary objects 2022-02-18 12:35:42 +02:00
vyzo
2795995989 add reification limit test 2022-02-18 12:19:19 +02:00
vyzo
cbd23c2b1b add reification limit 2022-02-18 12:19:09 +02:00
vyzo
899a65ae87 fix test 2022-02-17 20:13:46 +02:00
vyzo
a20c6cb04b temporarily disable reification 
big reifications can use a lot of memory during sync apparently.
 2022-02-17 19:56:50 +02:00
vyzo
74556edcff don't fail reification on missing references 2022-02-17 12:52:52 +02:00
Łukasz Magiera
7efed6617c
Merge pull request #8029 from filecoin-project/feat/splistore-cold-object-reification-redux 
splistore cold object reification redux
 2022-02-14 19:04:02 +01:00
vyzo
6bcade5e6d add comment about bigness of reification batch 2022-02-14 16:13:54 +02:00
vyzo
4524fbe936 wait for reify workers to finish when closing 2022-02-14 16:10:54 +02:00
vyzo
6c7ababd3f add comment about trackTxnRefs being noops if txnActive is false 2022-02-14 16:06:12 +02:00
vyzo
a428f44793 don't reify objects while still warming up 2022-02-14 16:04:39 +02:00
vyzo
b576785aac rename GetHotView to IsHotView 2022-02-14 16:03:17 +02:00
vyzo
e129ae3706 refactor nextBatch in badger markset 2022-02-09 18:34:50 +02:00
vyzo
8ddf4766ea update README 
Adds note about 3k IOPs requirement with badger markset, updates the memory requirement for
map to 48G based on observed behaviour of test nodes.
 2022-02-08 10:10:51 +02:00
vyzo
d45e2074a6 update README for map as the default 2022-02-06 11:25:27 +02:00
vyzo
713edd565c fix lint 2022-02-05 21:30:53 +02:00
vyzo
a32b7a32f3 directly mark objects in cold object reification 2022-02-05 20:00:15 +02:00
vyzo
929a05e898 add reification test 2022-02-04 16:16:34 +02:00
vyzo
268366e446 cold object reification context option 2022-02-04 16:07:58 +02:00
vyzo
4cddfd1074 background cold object reification 2022-02-04 14:29:50 +02:00
vyzo
049b48928f add note about compaction algorithm changes in README 2022-02-02 14:39:21 +02:00
vyzo
75ad0c3c2d badger markset option tweaks 2022-02-02 14:34:30 +02:00
vyzo
c1d8368fdc share a concurrent visitor between workers in markLiveRefs 2022-02-01 21:40:09 +02:00
vyzo
4b4104e400 fix comment 2022-02-01 12:10:30 +02:00
vyzo
b13aa8f172 unblock waitForSync on close 2022-02-01 11:30:21 +02:00
vyzo
9c92d77b64 improve robustness of waitForSync 2022-02-01 11:25:07 +02:00
vyzo
cd95892bf0 fix test 2022-02-01 11:13:01 +02:00
vyzo
7b4ab2077b wait for sync in a non racey way 2022-02-01 11:10:51 +02:00
vyzo
578b5691bc check for closing after the sync wait 2022-02-01 10:59:08 +02:00
vyzo
6353fa72d8 decouple SyncGapTime from wait time 2022-02-01 10:55:15 +02:00
vyzo
fd07ca87ce wait for the sync gap time befor starting the purge 2022-02-01 10:46:42 +02:00
vyzo
11ae85645b optimize single object marking in markLiveRefs 2022-02-01 09:13:21 +02:00
vyzo
3aabb0314a synchronously mark live refs on put/putmany 2022-02-01 09:07:14 +02:00
vyzo
37673c6de6 downgrade marking log to debug 2022-01-31 21:44:10 +02:00
vyzo
7896af731f use walkObjectIncomplete for marking live refs 2022-01-31 21:41:42 +02:00
vyzo
877dfbe992 hold the lock in the second protect call 2022-01-31 19:21:03 +02:00
vyzo
5b9ea1b4e0 avoid races in beginCriticalSection 2022-01-31 18:48:53 +02:00
vyzo
710fda49f0 fix putmany marking 2022-01-31 17:35:30 +02:00
vyzo
2b14bda6b8 recursively mark puts during the critical section 2022-01-31 17:26:08 +02:00
vyzo
1abfc5b1cf fix comment 2022-01-31 15:00:03 +02:00
vyzo
c9bd5ec452 mark tipset references to protect them during critical section 2022-01-31 14:54:23 +02:00
vyzo
ee63be26a1 fix race in protectView 2022-01-31 12:40:26 +02:00
vyzo
1900c907fd account for missing refs in the markset in Has 2022-01-31 09:44:51 +02:00
vyzo
a9d4495d83 use both hot and cold when doing fetches for markset positive objects 2022-01-30 22:47:20 +02:00
vyzo
7b8447a95a reinstante waitForMissingRefs 2022-01-30 19:00:52 +02:00
vyzo
7931f1f8f9 fix lint 2022-01-30 15:43:52 +02:00
vyzo
20b75022aa fix mockStore for splitstore tests 2022-01-30 15:33:30 +02:00
vyzo
dbc8903bac sortless compaction 2022-01-30 15:33:15 +02:00
vyzo
a4f720d866 coldset test 2022-01-30 15:11:25 +02:00
vyzo
4b8369c071 fix buffered reads 2022-01-30 15:11:18 +02:00
vyzo
72333147d3 prettier checkpoint close 2022-01-30 15:00:03 +02:00
vyzo
64cda4aeda on disk coldsets 2022-01-30 14:19:05 +02:00
vyzo
6ede77b254 checkpoint test 2022-01-30 14:10:11 +02:00
vyzo
c94eee5fc3 on disk checkpoints 2022-01-30 12:47:30 +02:00
vyzo
322b85898f make markSets synchronous in critical section 2022-01-30 12:10:08 +02:00
vyzo
cf09dd044a moar markset tests 2022-01-30 11:45:12 +02:00
vyzo
a4c1a34163 check for existence of badger db in recover 2022-01-30 11:44:27 +02:00
vyzo
d140909562 add MarkMany to MarkSet interface 2022-01-30 11:43:45 +02:00
vyzo
f9fd47e7d0 use temporary dir for splitstore test path 2022-01-29 13:11:58 +02:00
vyzo
67fbf9eb00 improve peristence test 2022-01-28 15:55:34 +02:00
vyzo
730aceac6d immediately flush pending writes when entering critical section 2022-01-28 15:55:10 +02:00
vyzo
1bf396f9af add test for markset persistence 2022-01-28 15:49:41 +02:00
vyzo
45c2f34295 refactor marksets for critical section on-disk persistence 2022-01-28 15:41:33 +02:00
vyzo
f07ce297f6 optimize slice allocations in walk 2022-01-26 21:55:24 +02:00
vyzo
a87239e802 avoid extraneous assignment 2022-01-26 21:48:03 +02:00
vyzo
fe47d6a1a4 fix check and warmup for parallel walk 2022-01-26 09:01:51 +02:00
vyzo
10f2445a99 use minimum of 2 workers 2022-01-25 21:37:48 +02:00
vyzo
8e01e73de4 dynamically compute number of workers for parallel chain walk 2022-01-25 19:47:58 +02:00
vyzo
7c8edf5632 parallelize walkChain 2022-01-25 17:29:02 +02:00
vyzo
25284b5325 refactor: eliminate distinction between markset and markset visitors 2022-01-25 16:31:45 +02:00
Aayush Rajasekaran
3c57b980e5
Merge pull request #7940 from filecoin-project/steb/buffered-blockstore-correctness 
correctness fixes for the autobatch blockstore
 2022-01-12 19:17:03 -05:00
Steven Allen
2a862d497f correctness fixes for the autobatch blockstore 
1. Simplify shutdown and make it idempotent by using a context.
2. Make sure `Flush` actually _fully_ flushes if the previous flush failed.
3. Don't clear the flush batch if flushing fails.
 2022-01-12 15:19:44 -08:00
Aayush Rajasekaran
efe9ec4906 Add missing locks to Get() 2022-01-12 17:22:54 -05:00