Commit Graph

18409 Commits

Author SHA1 Message Date
Hector Sanjuan
a2475bca94 range-export: write files to lotus repo path 2023-02-14 21:08:10 +01:00
Hector Sanjuan
be1614a042 range-export: check serverside that head > tail 2023-02-14 21:08:10 +01:00
Hector Sanjuan
ec01036871 Ranged export: remove non-internal ranged export method
Additionally, per feedback:

* Set "admin" permissions
* Remove from v0api
2023-02-14 21:08:10 +01:00
Hector Sanjuan
1bb698619c Ranged-export: Remove CachingBlockstore
The improvements in the range-export code lead to avoid reading most blocks
twice, as well as to allowing some blocks to be written to disk multiple times.

The cache hit-rate went down from being close to 50% to a maximum of 12% at
the very end of the export. The reason is that most CIDs are never read twice
since they are correctly tracked in the CID set.

These numbers do not support the maintenance of the CachingBlockstore
code. Additional testing shows that removing it has similar memory-usage
behaviour and about 5 minute-faster execution (around 10%).

Less code to maintain and less options to mess up with.
2023-02-14 21:08:10 +01:00
Hector Sanjuan
fa93c23813 Chain ranged export: rework and address current shortcomings
This commit moderately refactors the ranged export code. It addresses several
problems:
  * Code does not finish cleanly and things hang on ctrl-c
  * Same block is read multiple times in a row (artificially increasing cached
    blockstore metrics to 50%)
  * It is unclear whether there are additional races (a single worker quits
    when reaching height 0)
  * CARs produced have duplicated blocks (~400k for an 80M-blocks CAR or
    so). Some blocks appear up to 5 times.
  * Using pointers for tasks where it is not necessary.

The changes:

  * Use a FIFO instead of stack: simpler implementation as its own type. This
has not proven to be much more memory-friendly, but it has not made things
worse either.
  * We avoid a probably not small amount of allocations by not using
    unnecessary pointers.
  * Fix duplicated blocks by atomically checking+adding to CID set.
  * Context-termination now works correctly. Worker lifetime is correctly tracked and all channels
  are closed, avoiding any memory leaks and deadlocks.
  * We ensure all work is finished before finishing, something that might have
  been broken in some edge cases previously. In practice, we would not have
  seen this except perhaps in very early snapshots close to genesis.

Initial testing shows the code is currently about 5% faster. Resulting
snapshots do not have duplicates so they are a bit smaller. We have manually
verified that no CID is lost versus previous results, with both old and recent
snapshots.
2023-02-14 21:08:10 +01:00
frrist
21efd481d8 First efficient ranged-export implementation by @frisst
This first commit contains the first and second implementation stabs (after
primary review by @hsanjuan), using a stack for task buffering.

Known issues: ctrl-c (context cancellation) results in the export code getting
deadlocked. Duplicate blocks in exports. Duplicate block reads from store.

Original commit messages:

works

works against mainnet and calibnet

feat: add internal export api method

- will hopfully make things faster by not streaming the export over the json rpc api

polish: better file nameing

fix: potential race in marking cids as seen

chore: improve logging

feat: front export with cache

fix: give hector a good channel buffer on this shit

docsgen
2023-02-14 15:41:10 +01:00
Łukasz Magiera
61f29a84b5
Merge pull request #10238 from filecoin-project/asr/merge-release-into-master
chore: merge release/v1.20.0 into master
2023-02-13 15:25:23 +01:00
Łukasz Magiera
d108565ea2 Merge remote-tracking branch 'origin/master' into asr/merge-release-into-master 2023-02-13 13:40:32 +01:00
Łukasz Magiera
7486227c27 Merge remote-tracking branch 'origin/release/v1.20.0' into asr/merge-release-into-master 2023-02-13 13:14:09 +01:00
Łukasz Magiera
7b959ab39c
Merge pull request #10246 from filecoin-project/steb/fix-eth-get-code-none
fix: eth: correctly handle ethGetCode ege-cases
2023-02-13 13:13:00 +01:00
Łukasz Magiera
a652ffcb62 make gen 2023-02-13 13:09:09 +01:00
Łukasz Magiera
4d20011b78
Merge pull request #10249 from filecoin-project/jen/bumpmaster
build: bump version to v1.21.0-dev
2023-02-13 13:05:03 +01:00
Łukasz Magiera
98482a2795
Merge pull request #10250 from filecoin-project/jen/ipldcp
backport: deps: update ipld prime
2023-02-13 13:03:56 +01:00
Łukasz Magiera
e6b0ee7e4a
Merge pull request #10247 from filecoin-project/jen/ipld
dep: ipld: update ipld prime to v0.20.0
2023-02-13 13:03:44 +01:00
Raúl Kripalani
8ff55c027e Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-13 11:45:09 +00:00
Jennifer Wang
948a963f15 update ipld prime 2023-02-13 00:06:30 -05:00
Jennifer Wang
e73a038fe8 bump version to v1.21.0-dev 2023-02-13 00:00:07 -05:00
Steven Allen
b765112896 fix: correctly handle ethGetCode edge-cases
1. If the actor is a non-evm actor, return "nothing".
2. Correctly handle missing actors.
3. Handle self-destructed actors (they'll return "null").

fixes https://github.com/filecoin-project/ref-fvm/issues/1641
2023-02-12 20:38:47 -08:00
Jennifer Wang
94063ea4d1 update ipld prime 2023-02-12 23:33:19 -05:00
Steven Allen
a085eb1593 fix: eth: correctly return empty bytecode after selfdestruct 2023-02-12 19:31:56 -08:00
raulk
37e1ac5d93
eth: FIP-0055: implement final version of transitory delegated signature. (#10239) 2023-02-12 21:08:42 +00:00
Łukasz Magiera
1ee9516887
Merge pull request #9961 from filecoin-project/fix/EthUint64-args
Eth JSON-RPC: support passing uint64 in JSON-RPC arguments for EthUint64
2023-02-11 01:12:13 +01:00
Aayush
63b03a48d0 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-10 18:39:51 -05:00
Aayush Rajasekaran
5e12916aec
Merge pull request #10237 from filecoin-project/asr/update-everything
chore: deps: update FFI, go-state-types, actors to RCs
2023-02-10 18:37:32 -05:00
Aayush
753a0e5408 Update actors to v10.0.0-rc.1 2023-02-10 17:59:52 -05:00
Aayush
83bce7c94f disable adding git tags to bundle metadata 2023-02-10 17:58:57 -05:00
Aayush
2ec34e2c46 Update to go-state-types v0.10.0-rc2 2023-02-10 17:56:48 -05:00
Raúl Kripalani
6a76bbb6e3 improve TODO. 2023-02-10 22:46:57 +00:00
Raúl Kripalani
c61267cadc fix bad test. 2023-02-10 22:42:17 +00:00
Aayush
e167891207 Update to FFI v1.20.0-rc1 2023-02-10 17:38:29 -05:00
ychiao
8cded9f8d2 add todo 2023-02-10 17:34:53 -05:00
Raúl Kripalani
778768dbf7 eth_feeHistory: parse block param correctly. 2023-02-10 22:29:52 +00:00
ychiao
db749b8fc7 Merge branch 'release/v1.20.0' into fix/EthUint64-args 2023-02-10 17:26:38 -05:00
Steven Allen
01e616f29e
fix: stmgr: make the tipset and height agree when estimating gas (#10216)
* fix: stmgr: make the tipset and height agree when estimating gas

Specifically re-execute all messages in the current tipset, tacking the new
message onto the end. That way, the epoch is the epoch of the current tipset.

We could try to "make" a fake block and use that, but that's unlikely to
work well.

* fix: stmgr: only apply tipset messages for CallWithGas

* fix: itest: window post dispute
2023-02-10 20:49:26 +00:00
Aayush Rajasekaran
1996694a2a
Merge pull request #10215 from filecoin-project/steb/txhash-topic
feat: eth: harden event processing
2023-02-10 15:16:34 -05:00
ychiao
5e56af33df fix typo 2023-02-10 13:52:25 -05:00
ychiao
3c580403c1 implement itest and handle optional params 2023-02-10 13:33:59 -05:00
Steven Allen
b0d917d542 fix: make gen 2023-02-10 09:32:59 -08:00
Steven Allen
cc302dd68b fix: eth: log on unexpected events
We can remove these later as we add more event types, but this will aid
in debugging.
2023-02-10 07:59:47 -08:00
ychiao
37044ed3c3
Apply suggestions from code review
Co-authored-by: raulk <raul@protocol.ai>
2023-02-10 10:42:19 -05:00
Steven Allen
ece8f25511 fix: eth: strict event parsing
We now enforce the following rules:

1. No duplicate topics or data.
2. Topics must have 32 byte keys.
3. Topics may not be skipped. (e.g., no t1 & t3 without a t2).
4. Raw codecs.

We _don't_ require that topics/data be emitted in any specific order.

We _skip_ events with unknown keys. We _drop_ events that violate the
above rules.
2023-02-10 07:41:29 -08:00
Steven Allen
f1381ad535 refactor: use EthHash for event topics
This ensures they're always 32 bytes and padded, as required.
2023-02-10 07:41:29 -08:00
Łukasz Magiera
1588dd3dbe
Merge pull request #10227 from filecoin-project/fix/signing-noexport
fix: Don't call WalletExport in msg signing flows
2023-02-10 16:21:24 +01:00
Raúl Kripalani
e06e3a0654 Merge branch 'release/v1.20.0' into fix/EthUint64-args 2023-02-10 12:49:02 +00:00
Steven Allen
80fccba93f
fix: gas: update ffi & correct the message inclusion cost in nv18 (#10228)
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
2023-02-10 12:48:20 +00:00
Łukasz Magiera
7aa6663127
Merge pull request #10232 from filecoin-project/worker-all-task-flag2
fix: worker: add all tasks flag
2023-02-10 12:59:47 +01:00
Łukasz Magiera
54ed1f12ff
Merge pull request #10230 from filecoin-project/chore/nv18-butterflynet-reset-artifacts
updates butterflynet reset artifacts
2023-02-10 12:57:15 +01:00
Łukasz Magiera
37e9fe90ae
Merge pull request #10189 from filecoin-project/raulk/timestamp-null-rounds
fix: null rounds: pass correct timestamp to the FVM.
2023-02-10 12:15:44 +01:00
Łukasz Magiera
ec1b2b959a
Merge pull request #9973 from placer14/mg/chore/expose-compaction-poolsize-envvar
feat: blockstore: Envvar can adjust badger compaction worker poolsize
2023-02-10 11:38:08 +01:00
ognots
bc4ae8d383 updates butterflynet reset artifacts
* bootstrap node multiaddr
* new genesis file

these files are necessary for others to build lotus and join the new butterflynet, reset on February 08, 2023
2023-02-09 21:45:44 -05:00