Commit Graph

472 Commits

Author SHA1 Message Date
Aayush Rajasekaran
c4214e23bf
Merge pull request from AnomalRoil/fix/ctxonlongops
fix: chain: cancel long operations upon ctx cancelation
2023-08-28 13:33:08 -04:00
Yolan Romailler
13ade62e27 fix: cancel long operations upon ctx cancelation 2023-08-24 15:30:33 +02:00
Marten Seemann
bca0ec4f9c update go-libp2p v0.30.0 2023-08-21 10:44:37 +07:00
Aayush
12e076e9ea fix: chainstore: do not get stuck in unhappy equivocation cases 2023-08-11 17:32:39 -04:00
Aayush
69f3580ca2 add more test cases covering chainstore equivocation 2023-08-09 21:00:50 -04:00
Aayush
2cbdedc209 add more test cases covering chainstore equivocation 2023-08-09 19:40:53 -04:00
Aayush
0ff334912b Address review 2023-08-09 19:28:24 -04:00
Aayush
52657c5cb0 feat: chainstore: FRC-0051: Remove all equivocated blocks from tipsets 2023-08-09 19:28:24 -04:00
Steven Allen
1fc47eb7d8 feat: vm: allow raw "cbor" in state and use the new go-multicodec
1. Switch to go-multicodec as the source of multicodec code information.
This gives us a central, generated source of multicodec codes.
2. Use this library inside the VM and shapshot logic to consistently
allow CBOR, in addition to DagCBOR.
3. Remove the hard-coded CBOR constant.
2023-07-17 11:24:01 -07:00
Jakub Sztandera
cad743ec54
Appease the linter
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2023-05-19 20:02:47 +02:00
Jakub Sztandera
dfa7fc7723
feat: chainstore: sharded mutex for filling chain height index
This PR introduces as sharded mutex within the ChainIndex#GetTipsetByHeight.
It also replaces a go map with xsync.Map which doesn't require locking.

The lock is taken when it appears that ChainIndex filling work should be
started. After claiming the lock, the status of the cache is rechecked,
if the entry is still missing, the fillCache is started.

Thanks to @snissn and @arajasek for debugging and taking initial stabs at this.

Supersedes  and 10885

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2023-05-19 19:49:22 +02:00
Shrenuj Bansal
b0d1385273 Check if epoch is negative in GetTipsetByHeight 2023-05-16 21:19:03 -04:00
Aayush
596b01fba2 feat: chainstore: exit early in MaybeTakeHeavierTipset 2023-05-05 13:56:48 -04:00
Aayush
49f5825296 feat: chainstore: batch writes of tipsets 2023-05-02 17:26:07 -04:00
Steven Allen
7162c656cc
fix: chain: record heaviest tipset before notifying ()
Clearly this hasn't caused any issues, but I'm pretty sure we should be
updating the current head _before_ notifying about it.
2023-04-25 11:56:26 -07:00
Hector Sanjuan
875c09840b
chainstore: Fix raw blocks getting scanned for links during snapshots ()
We have to save raw blocks to the snapshot, but we should not be scanning them
for additional links as if they were CBOR blocks.

This cleans the logic a bit (we were checking that the parent was a CBOR block
before queueing up the children, but then scanning the children... it was weird).

Additionally, more verbose logging is added for the next time ScanForLinks
fails (currently very little info was given).

Our ScanForLinks callback should only enqueue CBOR for further processing.
2023-04-21 15:16:26 -07:00
Jorropo
fe42d974a2 chore: all: migrate from go-libipfs to boxo
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Jiaying Wang
522388820a
Merge pull request from filecoin-project/asr/optimize-blockmsgsfortipset
feat: chainstore: optimize BlockMsgsForTipset
2023-04-04 13:53:23 -04:00
Aayush Rajasekaran
4b2e74a6df
Merge pull request from hsanjuan/filcryo-v1.20.0-branch
Fix: export-range: Ignore ipld Blocks not found in Receipts.
2023-03-28 16:01:11 -04:00
Hector Sanjuan
39b27e709e export-range: use debug log instead of warn for not found receipt events 2023-03-28 15:05:43 +02:00
Aayush
b1669235f7 feat: chainstore: optimize BlockMsgsForTipset 2023-03-23 11:57:43 -04:00
vyzo
8abe0ea608
Merge pull request from filecoin-project/vyzo/feat/chain/msgindex
feat:chain: Message Index
2023-03-22 17:02:34 +02:00
Hector Sanjuan
6942020253 Ignore blockstore.Get errors when the top-level-task is Receipts 2023-03-21 21:25:50 +01:00
Hector Sanjuan
4b8442d8b9 Chain export: record epoch for every task. 2023-03-16 15:31:58 +01:00
Hector Sanjuan
a3900caeb0 Improve logging by carrying some more information in tasks 2023-03-16 13:38:40 +01:00
Steven Allen
dcb49dc8ee
refactor: update cache to the new generic version ()
- Adds type safety.
- Reduces allocations.
- Fixes the drand cache (was storing by value, but retrieving by pointer)
2023-03-13 15:29:09 -07:00
vyzo
1415391988 lint
lint

lint
2023-03-12 15:50:10 +02:00
Steven Allen
1cf57ffe2d feat: eth: optimize receipt reading
This optimizes the eth APIs (except the fee history one) to lookup the
tipset state/receipts instead of computing the state.
2023-03-10 15:33:45 -08:00
Aayush Rajasekaran
b75ebcac81
Merge pull request from filecoin-project/asr/fast-index
feat: chain: make chain tipset fetching 1000x faster
2023-03-09 15:38:55 -05:00
Aayush
90c89288b6 feat: chain: make fetching tipset by height 1000x faster 2023-03-09 15:12:13 -05:00
Travis Person
814c146626 feat: stmgr: cache migrated stateroots 2023-03-01 16:17:34 +00:00
Hector Sanjuan
1bb698619c Ranged-export: Remove CachingBlockstore
The improvements in the range-export code lead to avoid reading most blocks
twice, as well as to allowing some blocks to be written to disk multiple times.

The cache hit-rate went down from being close to 50% to a maximum of 12% at
the very end of the export. The reason is that most CIDs are never read twice
since they are correctly tracked in the CID set.

These numbers do not support the maintenance of the CachingBlockstore
code. Additional testing shows that removing it has similar memory-usage
behaviour and about 5 minute-faster execution (around 10%).

Less code to maintain and less options to mess up with.
2023-02-14 21:08:10 +01:00
Hector Sanjuan
fa93c23813 Chain ranged export: rework and address current shortcomings
This commit moderately refactors the ranged export code. It addresses several
problems:
  * Code does not finish cleanly and things hang on ctrl-c
  * Same block is read multiple times in a row (artificially increasing cached
    blockstore metrics to 50%)
  * It is unclear whether there are additional races (a single worker quits
    when reaching height 0)
  * CARs produced have duplicated blocks (~400k for an 80M-blocks CAR or
    so). Some blocks appear up to 5 times.
  * Using pointers for tasks where it is not necessary.

The changes:

  * Use a FIFO instead of stack: simpler implementation as its own type. This
has not proven to be much more memory-friendly, but it has not made things
worse either.
  * We avoid a probably not small amount of allocations by not using
    unnecessary pointers.
  * Fix duplicated blocks by atomically checking+adding to CID set.
  * Context-termination now works correctly. Worker lifetime is correctly tracked and all channels
  are closed, avoiding any memory leaks and deadlocks.
  * We ensure all work is finished before finishing, something that might have
  been broken in some edge cases previously. In practice, we would not have
  seen this except perhaps in very early snapshots close to genesis.

Initial testing shows the code is currently about 5% faster. Resulting
snapshots do not have duplicates so they are a bit smaller. We have manually
verified that no CID is lost versus previous results, with both old and recent
snapshots.
2023-02-14 21:08:10 +01:00
frrist
21efd481d8 First efficient ranged-export implementation by @frisst
This first commit contains the first and second implementation stabs (after
primary review by @hsanjuan), using a stack for task buffering.

Known issues: ctrl-c (context cancellation) results in the export code getting
deadlocked. Duplicate blocks in exports. Duplicate block reads from store.

Original commit messages:

works

works against mainnet and calibnet

feat: add internal export api method

- will hopfully make things faster by not streaming the export over the json rpc api

polish: better file nameing

fix: potential race in marking cids as seen

chore: improve logging

feat: front export with cache

fix: give hector a good channel buffer on this shit

docsgen
2023-02-14 15:41:10 +01:00
Aayush
63b03a48d0 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-10 18:39:51 -05:00
Łukasz Magiera
75f9b7d040
Merge pull request from filecoin-project/gstuart/tx-hash-on-simport-napshot
feat: eth: populate tx hash database on startup
2023-02-09 19:06:20 +01:00
Aayush
f291572b4a Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-03 09:27:04 -05:00
raulk
b46258d0bf
feat: evm: align events implementation with FIP-0049 and FIP-0054. ()
- Event keys are now t1, t2, t3, t4 for topics; and d for data.
- ref-fvm no longer stores events in the blockstore for us. It just
  returns events to the client, who is now responsible for handling
  them as it wishes / according to its configuration.
- Add a flag to VMOpts to have the events AMT be written in the blockstore.
- Add a flag to the ChainStore to advertise to the rest of the system
  if the ChainStore is storing events.
- Enable that flag if the EthRPC is enabled (can also add an explicit
  configuration flag if wanted).
2023-02-02 19:10:30 -08:00
Geoff Stuart
e0326d9d39 Populate transaction hash database if the database has not been set up before 2023-01-30 16:33:04 -05:00
Jorropo
f572852d06 chore: all: bump go-libipfs to replace go-block-format
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58
2023-01-26 17:03:18 +01:00
Aayush
4f199ada40 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-20 17:32:29 -05:00
Geoff Stuart
50f26e9721 Fix testground build 2023-01-20 13:27:04 -05:00
Aayush
5518e64574 fix: chain: put tipsetkey upon expansion of tipset 2023-01-19 11:33:49 -05:00
Shrenuj Bansal
ee54a7f3f5
feat: snapshot: Store tipset key cids in chain store during snapshot import ()
* Store tipset key cids in chain store during snapshot import

* make gen

* fix circle ci config

* fix lint

* address comments
2023-01-18 11:22:05 -05:00
Aayush
e715384173 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-16 10:51:22 -05:00
raulk
cdf3812e40
NV18: Filecoin EVM runtime + Actor Events + EthAccount + EAM + f4 addressing ()
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Raul Kripalani <raulk@users.noreply.github.com>
Co-authored-by: Kevin Li <ychiaoli18@users.noreply.github.com>
Co-authored-by: vyzo <vyzo@hackzen.org>
Co-authored-by: Ian Davis <nospam@iandavis.com>
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Jennifer Wang <jiayingw703@gmail.com>
Co-authored-by: Geoff Stuart <geoff.vball@gmail.com>
Co-authored-by: Shrenuj Bansal <shrenuj.bansal@protocol.ai>
Co-authored-by: Shrenuj Bansal <108157875+shrenujbansal@users.noreply.github.com>
Co-authored-by: Geoff Stuart <geoffrey.stuart@protocol.ai>
Co-authored-by: Aayush Rajasekaran <aayushrajasekaran@Aayushs-MacBook-Pro.local>
Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
2023-01-13 19:11:13 +00:00
Alfonso de la Rocha
075216d9da Merge remote-tracking branch 'upstream/master' into adlrocha/cns-iface-master 2022-12-05 18:16:14 +01:00
Łukasz Magiera
cdf169ed83 fix: chain: Fix flaky TestChainExportImport 2022-12-02 22:13:12 +01:00
Łukasz Magiera
ac8ab3ef9e feat: chain: Faster snapshot imports, zstd imports 2022-11-29 14:10:15 +01:00
Alfonso de la Rocha
627d6e70a6 parametrized reward function for consensus 2022-10-13 16:30:14 +02:00