Commit Graph

464 Commits

Author SHA1 Message Date
Steven Allen
1fc47eb7d8 feat: vm: allow raw "cbor" in state and use the new go-multicodec
1. Switch to go-multicodec as the source of multicodec code information.
This gives us a central, generated source of multicodec codes.
2. Use this library inside the VM and shapshot logic to consistently
allow CBOR, in addition to DagCBOR.
3. Remove the hard-coded CBOR constant.
2023-07-17 11:24:01 -07:00
Jakub Sztandera
cad743ec54
Appease the linter
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2023-05-19 20:02:47 +02:00
Jakub Sztandera
dfa7fc7723
feat: chainstore: sharded mutex for filling chain height index
This PR introduces as sharded mutex within the ChainIndex#GetTipsetByHeight.
It also replaces a go map with xsync.Map which doesn't require locking.

The lock is taken when it appears that ChainIndex filling work should be
started. After claiming the lock, the status of the cache is rechecked,
if the entry is still missing, the fillCache is started.

Thanks to @snissn and @arajasek for debugging and taking initial stabs at this.

Supersedes  and 10885

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2023-05-19 19:49:22 +02:00
Shrenuj Bansal
b0d1385273 Check if epoch is negative in GetTipsetByHeight 2023-05-16 21:19:03 -04:00
Aayush
596b01fba2 feat: chainstore: exit early in MaybeTakeHeavierTipset 2023-05-05 13:56:48 -04:00
Aayush
49f5825296 feat: chainstore: batch writes of tipsets 2023-05-02 17:26:07 -04:00
Steven Allen
7162c656cc
fix: chain: record heaviest tipset before notifying ()
Clearly this hasn't caused any issues, but I'm pretty sure we should be
updating the current head _before_ notifying about it.
2023-04-25 11:56:26 -07:00
Hector Sanjuan
875c09840b
chainstore: Fix raw blocks getting scanned for links during snapshots ()
We have to save raw blocks to the snapshot, but we should not be scanning them
for additional links as if they were CBOR blocks.

This cleans the logic a bit (we were checking that the parent was a CBOR block
before queueing up the children, but then scanning the children... it was weird).

Additionally, more verbose logging is added for the next time ScanForLinks
fails (currently very little info was given).

Our ScanForLinks callback should only enqueue CBOR for further processing.
2023-04-21 15:16:26 -07:00
Jorropo
fe42d974a2 chore: all: migrate from go-libipfs to boxo
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Jiaying Wang
522388820a
Merge pull request from filecoin-project/asr/optimize-blockmsgsfortipset
feat: chainstore: optimize BlockMsgsForTipset
2023-04-04 13:53:23 -04:00
Aayush Rajasekaran
4b2e74a6df
Merge pull request from hsanjuan/filcryo-v1.20.0-branch
Fix: export-range: Ignore ipld Blocks not found in Receipts.
2023-03-28 16:01:11 -04:00
Hector Sanjuan
39b27e709e export-range: use debug log instead of warn for not found receipt events 2023-03-28 15:05:43 +02:00
Aayush
b1669235f7 feat: chainstore: optimize BlockMsgsForTipset 2023-03-23 11:57:43 -04:00
vyzo
8abe0ea608
Merge pull request from filecoin-project/vyzo/feat/chain/msgindex
feat:chain: Message Index
2023-03-22 17:02:34 +02:00
Hector Sanjuan
6942020253 Ignore blockstore.Get errors when the top-level-task is Receipts 2023-03-21 21:25:50 +01:00
Hector Sanjuan
4b8442d8b9 Chain export: record epoch for every task. 2023-03-16 15:31:58 +01:00
Hector Sanjuan
a3900caeb0 Improve logging by carrying some more information in tasks 2023-03-16 13:38:40 +01:00
Steven Allen
dcb49dc8ee
refactor: update cache to the new generic version ()
- Adds type safety.
- Reduces allocations.
- Fixes the drand cache (was storing by value, but retrieving by pointer)
2023-03-13 15:29:09 -07:00
vyzo
1415391988 lint
lint

lint
2023-03-12 15:50:10 +02:00
Steven Allen
1cf57ffe2d feat: eth: optimize receipt reading
This optimizes the eth APIs (except the fee history one) to lookup the
tipset state/receipts instead of computing the state.
2023-03-10 15:33:45 -08:00
Aayush Rajasekaran
b75ebcac81
Merge pull request from filecoin-project/asr/fast-index
feat: chain: make chain tipset fetching 1000x faster
2023-03-09 15:38:55 -05:00
Aayush
90c89288b6 feat: chain: make fetching tipset by height 1000x faster 2023-03-09 15:12:13 -05:00
Travis Person
814c146626 feat: stmgr: cache migrated stateroots 2023-03-01 16:17:34 +00:00
Hector Sanjuan
1bb698619c Ranged-export: Remove CachingBlockstore
The improvements in the range-export code lead to avoid reading most blocks
twice, as well as to allowing some blocks to be written to disk multiple times.

The cache hit-rate went down from being close to 50% to a maximum of 12% at
the very end of the export. The reason is that most CIDs are never read twice
since they are correctly tracked in the CID set.

These numbers do not support the maintenance of the CachingBlockstore
code. Additional testing shows that removing it has similar memory-usage
behaviour and about 5 minute-faster execution (around 10%).

Less code to maintain and less options to mess up with.
2023-02-14 21:08:10 +01:00
Hector Sanjuan
fa93c23813 Chain ranged export: rework and address current shortcomings
This commit moderately refactors the ranged export code. It addresses several
problems:
  * Code does not finish cleanly and things hang on ctrl-c
  * Same block is read multiple times in a row (artificially increasing cached
    blockstore metrics to 50%)
  * It is unclear whether there are additional races (a single worker quits
    when reaching height 0)
  * CARs produced have duplicated blocks (~400k for an 80M-blocks CAR or
    so). Some blocks appear up to 5 times.
  * Using pointers for tasks where it is not necessary.

The changes:

  * Use a FIFO instead of stack: simpler implementation as its own type. This
has not proven to be much more memory-friendly, but it has not made things
worse either.
  * We avoid a probably not small amount of allocations by not using
    unnecessary pointers.
  * Fix duplicated blocks by atomically checking+adding to CID set.
  * Context-termination now works correctly. Worker lifetime is correctly tracked and all channels
  are closed, avoiding any memory leaks and deadlocks.
  * We ensure all work is finished before finishing, something that might have
  been broken in some edge cases previously. In practice, we would not have
  seen this except perhaps in very early snapshots close to genesis.

Initial testing shows the code is currently about 5% faster. Resulting
snapshots do not have duplicates so they are a bit smaller. We have manually
verified that no CID is lost versus previous results, with both old and recent
snapshots.
2023-02-14 21:08:10 +01:00
frrist
21efd481d8 First efficient ranged-export implementation by @frisst
This first commit contains the first and second implementation stabs (after
primary review by @hsanjuan), using a stack for task buffering.

Known issues: ctrl-c (context cancellation) results in the export code getting
deadlocked. Duplicate blocks in exports. Duplicate block reads from store.

Original commit messages:

works

works against mainnet and calibnet

feat: add internal export api method

- will hopfully make things faster by not streaming the export over the json rpc api

polish: better file nameing

fix: potential race in marking cids as seen

chore: improve logging

feat: front export with cache

fix: give hector a good channel buffer on this shit

docsgen
2023-02-14 15:41:10 +01:00
Aayush
63b03a48d0 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-10 18:39:51 -05:00
Łukasz Magiera
75f9b7d040
Merge pull request from filecoin-project/gstuart/tx-hash-on-simport-napshot
feat: eth: populate tx hash database on startup
2023-02-09 19:06:20 +01:00
Aayush
f291572b4a Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-03 09:27:04 -05:00
raulk
b46258d0bf
feat: evm: align events implementation with FIP-0049 and FIP-0054. ()
- Event keys are now t1, t2, t3, t4 for topics; and d for data.
- ref-fvm no longer stores events in the blockstore for us. It just
  returns events to the client, who is now responsible for handling
  them as it wishes / according to its configuration.
- Add a flag to VMOpts to have the events AMT be written in the blockstore.
- Add a flag to the ChainStore to advertise to the rest of the system
  if the ChainStore is storing events.
- Enable that flag if the EthRPC is enabled (can also add an explicit
  configuration flag if wanted).
2023-02-02 19:10:30 -08:00
Geoff Stuart
e0326d9d39 Populate transaction hash database if the database has not been set up before 2023-01-30 16:33:04 -05:00
Jorropo
f572852d06 chore: all: bump go-libipfs to replace go-block-format
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58
2023-01-26 17:03:18 +01:00
Aayush
4f199ada40 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-20 17:32:29 -05:00
Geoff Stuart
50f26e9721 Fix testground build 2023-01-20 13:27:04 -05:00
Aayush
5518e64574 fix: chain: put tipsetkey upon expansion of tipset 2023-01-19 11:33:49 -05:00
Shrenuj Bansal
ee54a7f3f5
feat: snapshot: Store tipset key cids in chain store during snapshot import ()
* Store tipset key cids in chain store during snapshot import

* make gen

* fix circle ci config

* fix lint

* address comments
2023-01-18 11:22:05 -05:00
Aayush
e715384173 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-16 10:51:22 -05:00
raulk
cdf3812e40
NV18: Filecoin EVM runtime + Actor Events + EthAccount + EAM + f4 addressing ()
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Raul Kripalani <raulk@users.noreply.github.com>
Co-authored-by: Kevin Li <ychiaoli18@users.noreply.github.com>
Co-authored-by: vyzo <vyzo@hackzen.org>
Co-authored-by: Ian Davis <nospam@iandavis.com>
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Jennifer Wang <jiayingw703@gmail.com>
Co-authored-by: Geoff Stuart <geoff.vball@gmail.com>
Co-authored-by: Shrenuj Bansal <shrenuj.bansal@protocol.ai>
Co-authored-by: Shrenuj Bansal <108157875+shrenujbansal@users.noreply.github.com>
Co-authored-by: Geoff Stuart <geoffrey.stuart@protocol.ai>
Co-authored-by: Aayush Rajasekaran <aayushrajasekaran@Aayushs-MacBook-Pro.local>
Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2023-01-13 19:11:13 +00:00
Alfonso de la Rocha
075216d9da Merge remote-tracking branch 'upstream/master' into adlrocha/cns-iface-master 2022-12-05 18:16:14 +01:00
Łukasz Magiera
cdf169ed83 fix: chain: Fix flaky TestChainExportImport 2022-12-02 22:13:12 +01:00
Łukasz Magiera
ac8ab3ef9e feat: chain: Faster snapshot imports, zstd imports 2022-11-29 14:10:15 +01:00
Alfonso de la Rocha
627d6e70a6 parametrized reward function for consensus 2022-10-13 16:30:14 +02:00
Aayush
27880ece2b feat: support typed errors over RPC 2022-09-27 15:34:01 +00:00
Łukasz Magiera
08b22edd89 fix make gen 2022-08-29 16:25:30 +02:00
Masih H. Derkani
6f686cae11 Upgrade to go-blockstore 1.2.0 and go-cid 0.2.0
Upgrade required dependencies and fix breaking changes.
2022-06-29 14:55:22 -04:00
Łukasz Magiera
05cdeb80c3 chore: remove redundant import prefixes 2022-06-15 12:06:22 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Steven Allen
f491e39f22 fix: vm: support raw blocks in chain export
We need this for NV16 to include code in chain snapshots.

NOTE: I've also checked the splitstore, and we appear to be doing the
right thing there already.
2022-05-20 18:56:30 -04:00
Aayush Rajasekaran
ddc46fc059 Update to filecoin pubsub 2022-05-10 16:51:36 -04:00
Darko Brdareski
79453663b3
feat: Add additional test annotations ()
* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate repub and selection tests

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

* Fix markset annotations

* doc: test annotations for the markets dagstore wrapper

* Annotate miner_api test in dagstore

* Annotate more test files

* Remove bad annotations from fsrepo

* Annotate wdpost system

* Remove bad annotations

* Renamce "conformance" to "chaos_actor" tests

* doc: stm annotations for blockheader & election proof tests

* Annotate remaining "A" tests

* annotate: stm for error_test

* memrepo_test.go

* Annotate "b" file tests

* message_test.go

* doc: stm annotate for fsrepo_test

* Annotate "c" file tests

* Annotate "D" test files

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate u,v,wl tests

* doc: stm annotations for various test files

* Annotate "T" test files

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add mempool unit tests

* Add two more memPool Add tests

* Update submodules

* Add check function tests

* Add stm annotations, refactor test helper

* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

solving merge conflicts

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

solve merge conflict

* Fix markset annotations

* Annotate miner_api test in dagstore

* Annotate more test files

* doc: test annotations for the markets dagstore wrapper

* Annotate wdpost system

* Renamce "conformance" to "chaos_actor" tests

* Annotate remaining "A" tests

* doc: stm annotations for blockheader & election proof tests

* annotate: stm for error_test

* Annotate "b" file tests

* memrepo_test.go

* Annotate "c" file tests

* message_test.go

* Annotate "D" test files

* doc: stm annotate for fsrepo_test

* Annotate u,v,wl tests

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate "T" test files

* doc: stm annotations for various test files

* Add mempool unit tests

solve merge conflict

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add two more memPool Add tests

* Update submodules

* Add check function tests

solve conflict

* Add stm annotations, refactor test helper

solve merge conflict

* Change CLI test kinds to "unit"

* Fix double merged test

* Fix ccupgrade_test merge

* Fix lint issues

* Add stm annotation to types_Test

* Test vectors submodule

* Add file annotation to burn_test

Co-authored-by: Nikola Divic <divicnikola@gmail.com>
Co-authored-by: TheMenko <themenkoprojects@gmail.com>
2022-03-16 18:37:34 +01:00