cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
20,722 Commits 24 Branches 342 Tags 314 MiB
entrypoint_update
Commit Graph

4412 Commits

Author SHA1 Message Date
Steven Allen
3f74840b67
test: events: fix race when recording tipsets (#10665) 
fixes #10664
 2023-04-13 17:20:34 -07:00
Aayush
ae84f335cc feat: pubsub: treat ErrGasFeeCapTooLow as ignore, not reject 2023-04-11 10:26:43 -04:00
vyzo
dd6720c61b message data 2023-04-11 16:19:59 +03:00
vyzo
e358b3b358 add real world timing test for selection 2023-04-11 16:19:50 +03:00
vyzo
b4e2e871dc prune excess messages before selection 2023-04-11 13:12:05 +03:00
Aayush
7710a57fa3 fix: include extra messages in ComputeState InvocResult output 2023-04-05 15:25:57 -04:00
Jiaying Wang
522388820a
Merge pull request #10552 from filecoin-project/asr/optimize-blockmsgsfortipset 
feat: chainstore: optimize BlockMsgsForTipset
 2023-04-04 13:53:23 -04:00
vyzo
bf666a3f7e
Merge pull request #10551 from filecoin-project/vyzo/feat/exec-lanes 
feat: VM Execution Lanes
 2023-03-30 22:19:11 +03:00
Aayush Rajasekaran
139bde3771
Merge pull request #10561 from filecoin-project/mikers/messagepoolRWMutexes 
perf: message pool: change locks to RWMutexes for performance
 2023-03-30 12:47:44 -04:00
vyzo
d71b528253 reorg initialization code for better readability, remove unused import 2023-03-30 18:15:13 +03:00
vyzo
7b4e68249a add comment about Signal unsoundness 2023-03-30 18:13:08 +03:00
vyzo
54a80a8a97 revert dead code 2023-03-30 18:11:44 +03:00
Aayush Rajasekaran
8b2208fd9a
Merge pull request #9858 from adlrocha/adlrocha/consistent-bcast 
feat:networking:  (Synchronous) Consistent Broadcast for Filecoin EC
 2023-03-30 09:32:09 -04:00
adlrocha
682ddf6ffa
Update chain/sub/bcast/consistent.go 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2023-03-30 12:38:00 +02:00
Jiaying Wang
caa6dd9c41
Merge pull request #10594 from filecoin-project/phi/v1210-rc2 
build: release: v1.21.0-rc2
 2023-03-29 15:59:56 -04:00
Aayush
66fc6dc3e5 refactor: stop using deprecated io/ioutil 2023-03-29 15:40:49 -04:00
Aayush
a724a713da refactor: stop using deprecated io/ioutil 2023-03-29 15:38:46 -04:00
Aayush
3120852eb9 feat: supply: only grab genesis msig locks for writes 2023-03-29 14:26:15 -04:00
Aayush
24381411f2 feat: supply: drop genesis market locked funds 2023-03-29 14:26:08 -04:00
Mikers
06294351fa perf: eth: gas estimate set applyTsMessages false (#10546) 
* have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization

* set applyTsMessages = true in CallWithGas call in shed

* update test with new callwithgas api optimization for eth call

* Update chain/stmgr/call.go

Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation

---------

Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal>
 2023-03-29 14:25:40 -04:00
Friðrik Ásmundsson
36913db0cd
Merge pull request #10587 from filecoin-project/10583-enable-index-for-statewaitformessage 
feat: Use MessageIndex in WaitForMessage
 2023-03-29 18:38:39 +02:00
Mikers
41ea5a6f63
Update chain/messagepool/messagepool.go 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2023-03-29 05:44:25 -10:00
Fridrik Asmundsson
2c98a187dd Use defaultExecTraceCacheSize in case env cannot be parsed 2023-03-29 17:30:00 +02:00
vyzo
52d70d563a fix tests 2023-03-29 16:46:37 +03:00
vyzo
4184ce9c75 refactor execution lanes: hide the lock 2023-03-29 16:45:45 +03:00
Fridrik Asmundsson
f44fa5d96f Use MessageIndex in WaitForMessage 2023-03-29 13:35:27 +02:00
Fridrik Asmundsson
8aa4917741 Fix bug in searchForIndexedMsg which always returns an error 2023-03-29 13:18:26 +02:00
Fridrik Asmundsson
17505a0022 Make execution trace configurable via env variable 
We want to make the execution trace cache size configurable as SPs
may want to disable it while exchanges may want to crank it up.

We were also are going with intuition for this value, so having
ability to change it without a new build would help.

Fixes: https://github.com/filecoin-project/lotus/issues/10584
 2023-03-29 11:53:43 +02:00
Mike Seiler
3477f7ce57 use LRU cache for keyCache to make threadsafe, also have (high) upper bounds on size of cache 2023-03-28 13:14:33 -10:00
Mike Seiler
89b217ee21 move write lock to before verifyMsg 2023-03-28 12:35:09 -10:00
Mike Seiler
2fa95a09be clean up cache logic in addTs / fix a bug where the loop was incorrectly releasing the lock 2x 2023-03-28 11:14:13 -10:00
Aayush Rajasekaran
4b2e74a6df
Merge pull request #10535 from hsanjuan/filcryo-v1.20.0-branch 
Fix: export-range: Ignore ipld Blocks not found in Receipts.
 2023-03-28 16:01:11 -04:00
Aayush Rajasekaran
687f0c070f
Merge pull request #10556 from filecoin-project/10537-populate-index-after-snapshot 
feat: populate the index on snapshot import
 2023-03-28 11:41:43 -04:00
Fridrik Asmundsson
ecd13079e7 Address review comments 2023-03-28 17:08:53 +02:00
vyzo
71650cd8a4 rename newVM to makeVM for a happy yushie 2023-03-28 18:05:00 +03:00
vyzo
b27121612e rename confusing variable 2023-03-28 18:03:55 +03:00
vyzo
b2b78e9dfa
Update chain/vm/execution.go 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2023-03-28 17:56:35 +03:00
Alfonso de la Rocha
b7c297c537 Merge branch 'master' into adlrocha/consistent-bcast 2023-03-28 16:55:06 +02:00
Alfonso de la Rocha
1a771e4310
include a deeper gc round 2023-03-28 16:52:32 +02:00
vyzo
dcd9869842 make gen 2023-03-28 16:58:09 +03:00
Hector Sanjuan
39b27e709e export-range: use debug log instead of warn for not found receipt events 2023-03-28 15:05:43 +02:00
Fridrik Asmundsson
2e45f6f778 Dont do locking using defer 2023-03-28 12:36:57 +02:00
Fridrik Asmundsson
f84f8a831a Use TipSetKey as key in cache and return copies 2023-03-28 12:36:57 +02:00
Fridrik Asmundsson
1ea7e05cde feat: Add small cache to execution traces 
This PR adds a small cache to calls to ExecutionTrace which helps
 improve performance for node operators like exchanges and block
explorers.

If items is in cache calls to this function will be 2-3x faster.

Fixes: https://github.com/filecoin-project/lotus/issues/10504
 2023-03-28 12:36:52 +02:00
Mikers
831f8a499d repub needs Lock not RLock 2023-03-28 08:40:41 +00:00
Alfonso de la Rocha
df82a8240e
add comments 2023-03-28 10:33:26 +02:00
Mikers
e9d2223025 disallow infinite loop, since in testing this only runs once 2023-03-28 08:11:23 +00:00
Mikers
ad81cd18c2 cache the tipset nonce calculation before holding the write lock, and also verify that the the calculation is cached after grabbing the write lock. if it is not cached, give up the lock, calculate, and then grab the write lock again 2023-03-28 08:11:00 +00:00
Alfonso de la Rocha
92f6d3e468
global locking strategy for blockInfo map 2023-03-28 10:01:43 +02:00
Fridrik Asmundsson
90171c8bab Addressing lint errors 2023-03-25 11:17:54 +00:00
Mike Seiler
6f440a6420 change highly contented message pool locks to RWMutexes for performance 2023-03-24 15:49:02 -10:00
Fridrik Asmundsson
b8137f6b4d Delete existing message index when loading from snapshot 2023-03-24 16:45:33 +00:00
Fridrik Asmundsson
48c57d394b Improve performance when populating message indax 2023-03-24 15:36:31 +00:00
vyzo
08134552a4 address review comments 2023-03-24 15:48:58 +02:00
vyzo
4b590e2102 add vm execution metrics 2023-03-24 15:16:22 +02:00
Fridrik Asmundsson
59640a8b22 Populate the index on snapshot import 
Fixes: https://github.com/filecoin-project/lotus/issues/10537
 2023-03-24 11:41:27 +00:00
Mikers
41fce94db4
perf: eth: gas estimate set applyTsMessages false (#10546) 
* have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization

* set applyTsMessages = true in CallWithGas call in shed

* update test with new callwithgas api optimization for eth call

* Update chain/stmgr/call.go

Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation

---------

Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal>
 2023-03-23 18:27:01 -04:00
Aayush
bc87017ea5 feat: supply: only grab genesis msig locks for writes 2023-03-23 17:04:59 -04:00
Aayush
dcdd1bc214 feat: supply: drop genesis market locked funds 2023-03-23 15:56:59 -04:00
vyzo
f11a7f8940 fix incorrect deferred vm release 2023-03-23 19:51:23 +02:00
vyzo
317a87d669 add some sanity checks for execution concurrency parameters 2023-03-23 18:52:37 +02:00
Aayush
b1669235f7 feat: chainstore: optimize BlockMsgsForTipset 2023-03-23 11:57:43 -04:00
vyzo
2a0660447a make token.Done idempotent 2023-03-23 17:38:36 +02:00
vyzo
2bb89d9c30 call Executor.Done where appropriate in stmgr uses 2023-03-23 17:34:59 +02:00
vyzo
ee6c0f8570 only call Atoi on non empty strings 2023-03-23 17:28:08 +02:00
vyzo
7362556c02 update VM interface references to use the executor, and call Done where appropriate 2023-03-23 17:17:46 +02:00
vyzo
6550abdfcc introduce execution lanes 2023-03-23 16:53:50 +02:00
vyzo
8abe0ea608
Merge pull request #10452 from filecoin-project/vyzo/feat/chain/msgindex 
feat:chain: Message Index
 2023-03-22 17:02:34 +02:00
Hector Sanjuan
6942020253 Ignore blockstore.Get errors when the top-level-task is Receipts 2023-03-21 21:25:50 +01:00
Alfonso de la Rocha
90c2f9dbe2
minor fix 2023-03-16 17:17:59 +01:00
Alfonso de la Rocha
8d260d7478
address review 2023-03-16 17:03:25 +01:00
vyzo
883bbf8701 add sanity check in SearchForIndexedMsg call site 
Check that receipt is non nil, and the message was indeed found.
 2023-03-16 17:51:28 +02:00
vyzo
3710768910 add TODO for WaitForMessage to use the index 2023-03-16 17:45:16 +02:00
vyzo
ef2f2b0f89 reword funny comment 2023-03-16 17:43:56 +02:00
vyzo
9c087cc52c second error variable name to avoid confusing they yushie. 2023-03-16 17:42:09 +02:00
vyzo
1e7f5c6a1e shut down the index if there is an error during head processing 2023-03-16 17:41:06 +02:00
vyzo
ff22a46253 complain if head change processing is building backlog 2023-03-16 17:38:46 +02:00
vyzo
90c4763e00 fix typos 2023-03-16 17:33:12 +02:00
vyzo
8bceaadda8 chain errors in searchForIndexedMsg 2023-03-16 17:32:06 +02:00
vyzo
91fccc421a add ON CONFLICT REPLACE clause in messages 2023-03-16 17:25:20 +02:00
adlrocha
f59c246c7a
Update chain/sub/bcast/consistent.go 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2023-03-16 16:09:45 +01:00
Hector Sanjuan
4b8442d8b9 Chain export: record epoch for every task. 2023-03-16 15:31:58 +01:00
Hector Sanjuan
a3900caeb0 Improve logging by carrying some more information in tasks 2023-03-16 13:38:40 +01:00
Łukasz Magiera
7e414ac390 Merge remote-tracking branch 'origin/master' into feat/go-data-transfer-v2 2023-03-15 11:37:53 +01:00
Łukasz Magiera
162e07fc95
Merge pull request #10470 from filecoin-project/feat/applyblock-timing-log 
feat: consensus: log ApplyBlock timing/gas stats
 2023-03-14 15:25:31 +01:00
Łukasz Magiera
27e3c0b89f
fix: stmgr: don't attempt to lookup genesis state (#10472) 
* fix: stmgr: don't attempt to lookup genesis state

* stmgr: More correct fix for genesis state compute
 2023-03-14 06:45:23 -07:00
Łukasz Magiera
51c6b64c91 feat: consensus: log ApplyBlock timing/gas stats 2023-03-14 13:58:36 +01:00
Steven Allen
dcb49dc8ee
refactor: update cache to the new generic version (#10463) 
- Adds type safety.
- Reduces allocations.
- Fixes the drand cache (was storing by value, but retrieving by pointer)
 2023-03-13 15:29:09 -07:00
vyzo
5ade40cbac increase coalesce delays for test to deflake on CI 2023-03-13 17:27:12 +02:00
vyzo
47646cbd69 add optimization TODO comment 2023-03-13 15:19:07 +02:00
vyzo
1970792445 fix bug in searchForIndexedMsg 
Need to use and return the execution tipset
 2023-03-13 14:56:00 +02:00
vyzo
4b1a405002 basic msgindex itest 2023-03-13 14:55:26 +02:00
vyzo
05cb2428c1 add confidence check for indexed message epoch 2023-03-13 12:33:19 +02:00
vyzo
5e011d536b enhance comment about lookup cid semantics 2023-03-13 12:14:30 +02:00
vyzo
a11032b10a adjust coalescer delays 2023-03-13 11:44:10 +02:00
vyzo
bda7ef52da log rollback errors 2023-03-13 11:41:46 +02:00
vyzo
b90cfff0aa wire in lifecycle context 2023-03-13 10:42:48 +02:00
vyzo
5461548b7e fix comment typo 2023-03-13 05:57:26 +02:00
vyzo
4a20c9b60f cosmetics 2023-03-13 05:54:32 +02:00
vyzo
e7448b1bb5 fix tests 2023-03-13 05:51:33 +02:00
vyzo
df6dfdf8a9 refactor database 
- drop the execution index; we don't need it
- it is inclusion tipset
- use MessagesForTipset
- hoist db sql stuffs on top for clarity
- add index for tipset on messages
 2023-03-13 05:51:24 +02:00
vyzo
88d7a4e610 more lint 2023-03-12 16:13:05 +02:00
vyzo
1415391988 lint 
lint

lint
 2023-03-12 15:50:10 +02:00
vyzo
0077fa2a98 make gen 2023-03-12 15:30:05 +02:00
vyzo
171734ea31 hook the index into the rest of lotus 2023-03-12 15:25:07 +02:00
vyzo
0d274df977 use the transaction Luke! 2023-03-12 13:35:50 +02:00
vyzo
d9ca214309 test test test 2023-03-12 13:21:03 +02:00
vyzo
bf9ae23c98 implement Close 2023-03-11 22:09:31 +02:00
vyzo
b5dd4e31ac implement reconciliation 2023-03-11 21:26:11 +02:00
Raúl Kripalani
f7a979d825 eth_feeHistory: migrate to using TipSetState. 2023-03-11 17:25:00 +00:00
vyzo
d97c6b2f69 more implementation 2023-03-11 19:11:08 +02:00
vyzo
7fcf228bc4 better logger name 2023-03-11 18:23:05 +02:00
vyzo
3649ae373b implementation details 2023-03-11 18:21:16 +02:00
vyzo
9144f9cea7 abstract ChainStore interface 2023-03-11 17:56:23 +02:00
vyzo
eab728c4e7 we don't need the block 2023-03-11 17:25:57 +02:00
vyzo
cc83a7cd35 use tipset cid instead of key for posterity. 2023-03-11 17:12:28 +02:00
vyzo
a0e95ebe97 scaffolding 2023-03-11 16:50:29 +02:00
Steven Allen
1cf57ffe2d feat: eth: optimize receipt reading 
This optimizes the eth APIs (except the fee history one) to lookup the
tipset state/receipts instead of computing the state.
 2023-03-10 15:33:45 -08:00
Steven Allen
92bca485b6 feat: stmgr: skip tipset execution when possible 2023-03-10 13:21:17 -08:00
Steven Allen
f7603f6c13 feat: eth API: reject masked ID addresses embedded in f410f payloads 
We'll never get an actor/account deployed to one of these
addresses (although we might get a placeholder). However, converting
such an address to an f4 address is definitely wrong.
 2023-03-10 10:57:39 -08:00
Alfonso de la Rocha
c11ffa58a8
address review 2023-03-10 09:27:30 +01:00
Steven Allen
58900a7033
feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (#10416) 
However, we're leaving the default at 1.25x for backwards compatibility, for now.

Also:

1. Actually use the configured replace fee ratio.
2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2),
can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot.

fixes #10415
 2023-03-09 13:17:17 -08:00
Aayush Rajasekaran
b75ebcac81
Merge pull request #10423 from filecoin-project/asr/fast-index 
feat: chain: make chain tipset fetching 1000x faster
 2023-03-09 15:38:55 -05:00
Aayush
90c89288b6 feat: chain: make fetching tipset by height 1000x faster 2023-03-09 15:12:13 -05:00
Travis Person
f641139bf2
Merge pull request #10282 from filecoin-project/feat/cache-migrated-stateroot 
feat: stmgr: cache migrated stateroots
 2023-03-09 11:47:43 -08:00
Travis Person
b4e589a0f7 review comments 2023-03-09 18:54:29 +00:00
Łukasz Magiera
dcfbf7d0a6 fix TestPrintGroupInfo 2023-03-08 18:13:58 +01:00
Łukasz Magiera
d114d8fe7e deprecate MarketListRetrievalDeals, make gen 2023-03-08 17:15:16 +01:00
Marten Seemann
5b5ea083ce fix drand test 2023-03-08 17:50:58 +13:00
hannahhoward
7bc6185077
chore(deps): update to get to compile 2023-03-07 11:45:30 -08:00
Aayush
74d94af034 fix: state: short-circuit genesis state computation 2023-03-06 15:39:28 -05:00
Aayush
4e57626e42 feat: update renew-sectors with FIP-0045 logic 2023-03-02 16:38:05 -05:00
Steven Allen
dbbcf4b2ee
feat: vm: switch to the new exec trace format (#10372) 
This is now "FVM" native. Changes include:

1. Don't treat "trace" messages like off-chain messages. E.g., don't
include CIDs, versions, etc.
2. Include IPLD codecs where applicable.
3. Remove fields that aren't filled by the FVM (timing, some errors,
code locations, etc.).
 2023-03-01 16:02:18 -08:00
Aayush
19f1ad2762 Merge branch 'releases' into asr/merge-release-into-master 2023-03-01 14:18:35 -05:00
Travis Person
814c146626 feat: stmgr: cache migrated stateroots 2023-03-01 16:17:34 +00:00
Steven Allen
68b401a895 fix: cli: better handle sending from EthAccount actors 
This will make `lotus send` mostly just "do what the user wants" in this
case:

1. The user may not explicitly specify a method number.
2. Parameters are automatically cbor-encoded where applicable.
3. The method number is automatically selected based on the
   recipient (CreateExternal if sent to the EAM, InvokeEVM otherwise).
 2023-02-24 15:15:41 -08:00
Łukasz Magiera
ea57a1aec3 Merge remote-tracking branch 'origin/master' into hsanjuan/chain-export-range-rebased 2023-02-23 13:08:14 +01:00
Travis Person
bf2ac13609
Merge pull request #10283 from filecoin-project/feat/disable-pre-migrations 
feat: stmgr: add env to disable premigrations
 2023-02-21 09:23:13 -08:00
Travis Person
2dcaddf61d feat: stmgr: add env to disable premigrations 
Setting the environment variable `LOTUS_DISABLE_PRE_MIGRATIONS=1` will
discard premigrations for all upgrade.
 2023-02-17 23:12:26 +00:00
Aayush Rajasekaran
1ec02c5c95
Merge pull request #10296 from filecoin-project/asr/migration-test 
feat: shed: test the nv18 migration
 2023-02-17 16:20:51 -05:00
Aayush
984b4c78cb Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-17 16:08:22 -05:00
Aayush Rajasekaran
efcdec6517
Merge pull request #10300 from filecoin-project/steb/fix-eth-nonce 
fix: eth: return the correct nonce from EthGetTransactionCount
 2023-02-17 14:45:45 -05:00
Steven Allen
a3bc65ba05 fix: eth: return the correct nonce from EthGetTransactionCount 
EVM contracts track this number internally.

fixes https://github.com/filecoin-project/lotus/issues/10255
 2023-02-17 10:51:42 -08:00
Jorropo
2d3065ae8e chore: all: bump go-libipfs to replace go-block-format 
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58

(cherry picked from commit f572852d06)
 2023-02-17 18:46:11 +01:00
Aayush
fab7ea6ca1 feat: shed: test the nv18 migration 2023-02-17 11:09:52 -05:00
Steven Allen
00b6d06041
feat: eth: parse revert data (#10295) 
We don't really want to do this in the FVM because it's Ethereum
specific, but this makes sense to do in the Ethereum API.

See:

See https://docs.soliditylang.org/en/latest/control-structures.html#panic-via-assert-and-error-via-require
 2023-02-17 01:18:03 +00:00
Aayush
75c744a438 Merge branch 'release/v1.20.0' 2023-02-14 20:29:14 -05:00
Hector Sanjuan
1bb698619c Ranged-export: Remove CachingBlockstore 
The improvements in the range-export code lead to avoid reading most blocks
twice, as well as to allowing some blocks to be written to disk multiple times.

The cache hit-rate went down from being close to 50% to a maximum of 12% at
the very end of the export. The reason is that most CIDs are never read twice
since they are correctly tracked in the CID set.

These numbers do not support the maintenance of the CachingBlockstore
code. Additional testing shows that removing it has similar memory-usage
behaviour and about 5 minute-faster execution (around 10%).

Less code to maintain and less options to mess up with.
 2023-02-14 21:08:10 +01:00
Hector Sanjuan
fa93c23813 Chain ranged export: rework and address current shortcomings 
This commit moderately refactors the ranged export code. It addresses several
problems:
  * Code does not finish cleanly and things hang on ctrl-c
  * Same block is read multiple times in a row (artificially increasing cached
    blockstore metrics to 50%)
  * It is unclear whether there are additional races (a single worker quits
    when reaching height 0)
  * CARs produced have duplicated blocks (~400k for an 80M-blocks CAR or
    so). Some blocks appear up to 5 times.
  * Using pointers for tasks where it is not necessary.

The changes:

  * Use a FIFO instead of stack: simpler implementation as its own type. This
has not proven to be much more memory-friendly, but it has not made things
worse either.
  * We avoid a probably not small amount of allocations by not using
    unnecessary pointers.
  * Fix duplicated blocks by atomically checking+adding to CID set.
  * Context-termination now works correctly. Worker lifetime is correctly tracked and all channels
  are closed, avoiding any memory leaks and deadlocks.
  * We ensure all work is finished before finishing, something that might have
  been broken in some edge cases previously. In practice, we would not have
  seen this except perhaps in very early snapshots close to genesis.

Initial testing shows the code is currently about 5% faster. Resulting
snapshots do not have duplicates so they are a bit smaller. We have manually
verified that no CID is lost versus previous results, with both old and recent
snapshots.
 2023-02-14 21:08:10 +01:00
Jiaying Wang
86b4aeef1a
Merge pull request #10258 from filecoin-project/steb/fix-msg-modify 
fix: stmgr: copy the message before modifying it
 2023-02-14 10:07:24 -05:00
frrist
21efd481d8 First efficient ranged-export implementation by @frisst 
This first commit contains the first and second implementation stabs (after
primary review by @hsanjuan), using a stack for task buffering.

Known issues: ctrl-c (context cancellation) results in the export code getting
deadlocked. Duplicate blocks in exports. Duplicate block reads from store.

Original commit messages:

works

works against mainnet and calibnet

feat: add internal export api method

- will hopfully make things faster by not streaming the export over the json rpc api

polish: better file nameing

fix: potential race in marking cids as seen

chore: improve logging

feat: front export with cache

fix: give hector a good channel buffer on this shit

docsgen
 2023-02-14 15:41:10 +01:00
raulk
11ac325d1a
Revert "fix: stmgr: check message validity before invoking vm" (#10270) 
This reverts commit 2bbd1fbbb6.
 2023-02-14 14:06:33 +00:00
Aayush Rajasekaran
3c4a83916b
Merge pull request #10259 from filecoin-project/steb/call-check-message-validity 
fix: stmgr: check message validity before invoking vm
 2023-02-13 21:29:19 -05:00
ychiao
0d92c746bd
Eth JSON-RPC: populate reward in eth_feeHistory (#10245) 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
 2023-02-14 01:02:37 +00:00
Steven Allen
a16c54051c
fix: eth: rename yParity to v in transaction return values (#10264) 
fixes https://github.com/filecoin-project/ref-fvm/issues/1662
 2023-02-13 23:29:13 +00:00
Steven Allen
2bbd1fbbb6 fix: stmgr: check message validity before invoking vm 
Otherwise we may, e.g., try to estimate gas on a message to an f4
address before the nv18 migration.

I'm _not_ checking the "prior messages" here as this is just a sanity
check.
 2023-02-13 11:01:42 -08:00
Steven Allen
f427c2b566
fix: eth: correctly convert filecoin message <-> eth txn (#10257) 
1. We do allow deploying with empty initcode.
2. Make sure that the encoded "code" is non-empty, if specified.

Basically, this makes everything consistent (and it's how I specified it
in the FIP).
 2023-02-13 10:33:59 -08:00
Steven Allen
77fbc60cb8 fix: stmgr: copy the message before modifying it 2023-02-13 10:06:59 -08:00
Raúl Kripalani
8ff55c027e Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-13 11:45:09 +00:00
raulk
37e1ac5d93
eth: FIP-0055: implement final version of transitory delegated signature. (#10239) 2023-02-12 21:08:42 +00:00
Aayush
63b03a48d0 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-10 18:39:51 -05:00
ychiao
db749b8fc7 Merge branch 'release/v1.20.0' into fix/EthUint64-args 2023-02-10 17:26:38 -05:00
Steven Allen
01e616f29e
fix: stmgr: make the tipset and height agree when estimating gas (#10216) 
* fix: stmgr: make the tipset and height agree when estimating gas

Specifically re-execute all messages in the current tipset, tacking the new
message onto the end. That way, the epoch is the epoch of the current tipset.

We could try to "make" a fake block and use that, but that's unlikely to
work well.

* fix: stmgr: only apply tipset messages for CallWithGas

* fix: itest: window post dispute
 2023-02-10 20:49:26 +00:00
ychiao
5e56af33df fix typo 2023-02-10 13:52:25 -05:00
ychiao
3c580403c1 implement itest and handle optional params 2023-02-10 13:33:59 -05:00
ychiao
37044ed3c3
Apply suggestions from code review 
Co-authored-by: raulk <raul@protocol.ai>
 2023-02-10 10:42:19 -05:00
Steven Allen
ece8f25511 fix: eth: strict event parsing 
We now enforce the following rules:

1. No duplicate topics or data.
2. Topics must have 32 byte keys.
3. Topics may not be skipped. (e.g., no t1 & t3 without a t2).
4. Raw codecs.

We _don't_ require that topics/data be emitted in any specific order.

We _skip_ events with unknown keys. We _drop_ events that violate the
above rules.
 2023-02-10 07:41:29 -08:00
Steven Allen
f1381ad535 refactor: use EthHash for event topics 
This ensures they're always 32 bytes and padded, as required.
 2023-02-10 07:41:29 -08:00
Łukasz Magiera
1588dd3dbe
Merge pull request #10227 from filecoin-project/fix/signing-noexport 
fix: Don't call WalletExport in msg signing flows
 2023-02-10 16:21:24 +01:00
Raúl Kripalani
e06e3a0654 Merge branch 'release/v1.20.0' into fix/EthUint64-args 2023-02-10 12:49:02 +00:00
Steven Allen
80fccba93f
fix: gas: update ffi & correct the message inclusion cost in nv18 (#10228) 
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
 2023-02-10 12:48:20 +00:00
Łukasz Magiera
088d231cda fix: Don't call WalletExport in msg signing flows 2023-02-10 02:37:59 +01:00
Raúl Kripalani
d30ace0518 Merge branch 'release/v1.20.0' into raulk/timestamp-null-rounds 2023-02-09 22:58:08 +00:00
Steven Allen
a83f120f32 fix: chain: make sure the head is empty, not the code 2023-02-09 11:36:02 -08:00
Łukasz Magiera
0f05232199
Merge pull request #10224 from filecoin-project/feat/ethcli-strip-spaces 
feat: eth cli: Strip out empty spaces around contract bytes
 2023-02-09 19:24:49 +01:00
Jiaying Wang
1c13a591e9
Merge pull request #10223 from filecoin-project/feat/ethtx-fixes 
fix: ethrpc: Add missing fields to EthTx
 2023-02-09 13:13:26 -05:00
Łukasz Magiera
75f9b7d040
Merge pull request #10119 from filecoin-project/gstuart/tx-hash-on-simport-napshot 
feat: eth: populate tx hash database on startup
 2023-02-09 19:06:20 +01:00
Łukasz Magiera
b0fa75f13e feat: eth cli: Strip out empty spaces around contract bytes 2023-02-09 18:49:08 +01:00
Łukasz Magiera
cb8e2836bb
Merge pull request #10184 from filecoin-project/asr/merge-release-into-master 
chore: merge release/v1.20.0 into master
 2023-02-09 16:41:58 +01:00
Łukasz Magiera
e321761a72 ethrpc: Add missing fields to EthTx 2023-02-09 14:26:13 +01:00
Łukasz Magiera
554dd2f1ee itests: Test EVM bytecode getters, check bytecode hash 2023-02-09 13:23:17 +01:00
Łukasz Magiera
8d6aa2477c feat: actors: Add bytecode getter 2023-02-09 13:12:16 +01:00
Łukasz Magiera
9339267e6e
fix: ethrpc: emit correct bloom filters (#10194) 2023-02-08 23:14:53 +00:00
Steven Allen
89499bfd40 fvm: chore: update FVM 
This:

1. Updates the builtin actors bundle (for actors v10).
2. Updates the event entry type to include the codec.
3. Removes the cbor encoding and zero trimming from event data.

I've chose to:

1. _Not_ add codec handling to the event filtering system for now.
2. _Skip_ events with unexpected codecs.

We don't actually _allow_ these events in the FVM right now, and it
simplifies the implementation.

However, I _am_ recording the codecs in the database so we don't have to
migrate it later.
 2023-02-07 18:15:40 -08:00
Aayush Rajasekaran
693f230fbb Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-07 11:03:01 -05:00
Aayush Rajasekaran
6a268cfb63 chain: explicitly check that gasLimit is above zero 2023-02-06 17:53:00 -05:00
Raúl Kripalani
0cf3a5e704 fix: null rounds: pass correct timestamp to the FVM. 2023-02-06 16:58:58 +00:00
Łukasz Magiera
5dc56841ea fix: ethtypes: Correct 'no transactions' hash in NewEthBlock 2023-02-06 17:17:19 +01:00
Łukasz Magiera
424824019b fix: ethtypes: Correct 'no uncles' hash in NewEthBlock 2023-02-06 14:38:48 +01:00
Łukasz Magiera
6851b6a523 Merge remote-tracking branch 'origin/release/v1.20.0' into fix/eth-orpc-validate 2023-02-06 10:41:52 +01:00
Aayush
f291572b4a Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-03 09:27:04 -05:00
raulk
b46258d0bf
feat: evm: align events implementation with FIP-0049 and FIP-0054. (#10152) 
- Event keys are now t1, t2, t3, t4 for topics; and d for data.
- ref-fvm no longer stores events in the blockstore for us. It just
  returns events to the client, who is now responsible for handling
  them as it wishes / according to its configuration.
- Add a flag to VMOpts to have the events AMT be written in the blockstore.
- Add a flag to the ChainStore to advertise to the rest of the system
  if the ChainStore is storing events.
- Enable that flag if the EthRPC is enabled (can also add an explicit
  configuration flag if wanted).
 2023-02-02 19:10:30 -08:00
Łukasz Magiera
fe1e0974cb feat: ethrpc: Support filtering by address in subscribe 2023-02-02 14:46:36 +01:00
Ian Davis
d29a244064 Merge remote-tracking branch 'origin/release/v1.20.0' into iand/eth-openrpc-validate 2023-02-01 16:19:17 +00:00
Łukasz Magiera
ad14d71978 eth rpc: Params are optional in eth_subscribe 2023-01-31 11:08:51 +01:00
Łukasz Magiera
9701b11641 ethtypes: Serialize EthFilterID/EthSubscriptionID correctly 2023-01-31 10:28:12 +01:00
Geoff Stuart
e0326d9d39 Populate transaction hash database if the database has not been set up before 2023-01-30 16:33:04 -05:00
Geoff Stuart
af72e6f6ac
Refactor: Unify EthTx to FilecoinMessage methods v2 (#10095) 
* Refactor: Unify EthTx to FilecoinMessage methods

* Filecoin messages can again be converted to Eth Txs

* All BLS messages should calculated tx hash with unsigned message

* Refactor newEthTxReceipt

* fill in from and to for non-eth transactions

* Hoist nil check out of newEthTxFromMessageLookup

---------

Co-authored-by: Aayush <arajasek94@gmail.com>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
 2023-01-30 18:40:14 +00:00
Geoff Stuart
395cdd721e simplify transaction hash db queries, prevent fd leaks 2023-01-30 11:03:12 -05:00
Jorropo
f572852d06 chore: all: bump go-libipfs to replace go-block-format 
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58
 2023-01-26 17:03:18 +01:00
Phi
4fb9e2fd6a make jen 
- Add builtin.EthereumAddressManagerActorAddr to builtin.go.template and make jen
- Rename to EthereumAddressManagerActorAddr to match pattern of other actors (CronActorAddr/etc)
 2023-01-26 16:26:39 +01:00
Phi
8f6c569391 Add actors to circ-supply 
- Consider funds to EAM as locked
- Add evm, placeholder and ethaccount to circ supply
 2023-01-26 15:39:51 +01:00
Ian Davis
4500c02e49 Merge branch 'release/v1.20.0' into iand/eth-openrpc-validate 2023-01-26 14:23:42 +00:00
Ian Davis
c82dccc76c Merge branch 'release/v1.20.0' into iand/issue-9849-realtime 2023-01-26 12:11:14 +00:00
Ian Davis
a9cd76907e Merge branch 'release/v1.20.0' into iand/eth-openrpc-validate 2023-01-26 11:35:33 +00:00
Aayush
8842b21ec9 feat: vm: Assert empty object CID when dumping state 2023-01-25 12:35:18 -05:00
Ian Davis
58cd226d6b Change EthBlock.Extradata type to EthBytes 2023-01-24 14:46:16 +00:00
Ian Davis
8daf22c3ff Test and fix eth_FeeHistory 2023-01-24 11:27:18 +00:00
Aayush
e9f665214a Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-23 12:23:47 -05:00
Aayush Rajasekaran
a21bba8ac2
Merge pull request #10084 from filecoin-project/asr/delegated-siggy 
feat: minor improvements to Ethereum delegated siggys
 2023-01-23 12:11:51 -05:00
Ian Davis
7d1bd1d5ab Use gojsonschema 2023-01-23 16:50:40 +00:00
Ian Davis
ddd5ff9c42
fix: event filters use cbor encoding internally (#10085) 2023-01-23 14:14:25 +00:00
Aayush
4f199ada40 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-20 17:32:29 -05:00
Geoff Stuart
439b4c6e22 Cleanups 2023-01-20 16:51:01 -05:00
Geoff Stuart
50f26e9721 Fix testground build 2023-01-20 13:27:04 -05:00
ychiao
57bf1c7a89 Eth JSON-RPC: support passing uint64 in JSON-RPC arguments for EthUint64 2023-01-20 12:37:37 -05:00
Aayush
972e68a8c1 feat: minor improvements to Ethereum delegated siggys 2023-01-20 10:21:00 -05:00
Ian Davis
c5ed5dd254 itests: add event matrix tests for realtime eth filters and subscriptions 2023-01-20 14:01:25 +00:00
Raúl Kripalani
21c93f1dc1 Merge branch 'release/v1.20.0' into iand/issue-9849-filters 2023-01-20 01:11:31 +00:00
Raúl Kripalani
2a9069200d use len to conduct empty byte array checks. 2023-01-20 00:57:53 +00:00
Raúl Kripalani
2c56687527 fix: events: put the _signed_ message through the pipeline. 
We were putting the unsigned/VM message through the pipeline.
The events index was storing the _unsigned_ message CID.

However, the Eth tx hash index maps signed Delegated-signature message
CIDs to transaction hashes, i.e. it uses the _signed_ message CID.

As a result, eth_getLogs and other log-related methods were
unable to resolve the transaction hash from the index properly, and
would end up returning 0x00..00 in the transactionHash field.
 2023-01-20 00:36:06 +00:00
Shrenuj Bansal
522e96f016
fix: msg signing with delegated keys and send cli changes (#10056) 
* fix msg signing with delegated keys and send cli changes

* make gen and docsgen

* address comments
 2023-01-19 17:57:48 -05:00
Aayush
5518e64574 fix: chain: put tipsetkey upon expansion of tipset 2023-01-19 11:33:49 -05:00
Ian Davis
a08b557938
Merge branch 'release/v1.20.0' into iand/issue-9849-filters 2023-01-19 13:01:12 +00:00
Ian Davis
ca5cab4c43 Cleanup and fix eth filter tests 2023-01-19 12:58:23 +00:00
Ian Davis
061e8cca1f Add itest for EthGetLogs filter combinations 2023-01-18 17:33:54 +00:00
Shrenuj Bansal
ee54a7f3f5
feat: snapshot: Store tipset key cids in chain store during snapshot import (#10042) 
* Store tipset key cids in chain store during snapshot import

* make gen

* fix circle ci config

* fix lint

* address comments
 2023-01-18 11:22:05 -05:00
Aayush
e715384173 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-16 10:51:22 -05:00
Geoff Stuart
f8121c8f1c Add gc for eth tx database 2023-01-16 07:04:20 -05:00
Geoff Stuart
f8dee0983a review fixes 2023-01-16 01:56:55 -05:00
Geoff Stuart
a8436074a6 Store mapping from hashes for Ethereum transactions to Filecoin Message Cids 2023-01-16 01:17:56 -05:00
Steven Allen
2a3989e4bd
fix: check for unread params bytes in all cases (#10018) 2023-01-14 15:45:19 -08:00
Aayush
e0c09c0396 remove invalid test cases 2023-01-14 12:30:06 -05:00
Jennifer Wang
51d77d408b add bundle dev/20230114-pre-rc.2 2023-01-14 01:41:46 -05:00
vyzo
859cdca4f7 update go-state-types 
use CreateExternal

use CreateExternal in itest evm deployment

add missing import

update gst
 2023-01-14 01:11:11 -05:00
Aayush Rajasekaran
cc86117289
Merge pull request #10007 from filecoin-project/asr/delegated-siggy 
fix: delegated signatures: check every field of txs and roundtrip eth <-> FIL
 2023-01-14 00:14:38 -05:00
Steven Allen
1045dce1ea guarantee round-trip 2023-01-13 20:14:22 -08:00
Aayush
3421e6a127 fix: delegated signatures: check every field of txs and roundtrip eth <-> FIL 2023-01-13 20:28:51 -05:00
Raúl Kripalani
e25728cb68 remove leftover hack. 2023-01-13 19:59:09 +00:00
raulk
cdf3812e40
NV18: Filecoin EVM runtime + Actor Events + EthAccount + EAM + f4 addressing (#9998) 
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Raul Kripalani <raulk@users.noreply.github.com>
Co-authored-by: Kevin Li <ychiaoli18@users.noreply.github.com>
Co-authored-by: vyzo <vyzo@hackzen.org>
Co-authored-by: Ian Davis <nospam@iandavis.com>
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Jennifer Wang <jiayingw703@gmail.com>
Co-authored-by: Geoff Stuart <geoff.vball@gmail.com>
Co-authored-by: Shrenuj Bansal <shrenuj.bansal@protocol.ai>
Co-authored-by: Shrenuj Bansal <108157875+shrenujbansal@users.noreply.github.com>
Co-authored-by: Geoff Stuart <geoffrey.stuart@protocol.ai>
Co-authored-by: Aayush Rajasekaran <aayushrajasekaran@Aayushs-MacBook-Pro.local>
Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2023-01-13 19:11:13 +00:00
Łukasz Magiera
9a46682d9d
Merge pull request #9481 from consensus-shipyard/adlrocha/cns-iface-master 
IPC: Abstract common consensus functions and consensus interface
 2023-01-11 14:49:27 +01:00
hanabi1224
f4b4a63217 Merge remote-tracking branch 'origin/master' into close-write 2023-01-04 20:10:29 +08:00
hanabi1224
6b14b0536e fix: error checks 2023-01-04 20:10:21 +08:00
Łukasz Magiera
d8570576af
Merge pull request #9902 from filecoin-project/asr/restrict-verifblock 
chore: VM: don't export Extern::VerifyBlockSig
 2023-01-04 12:18:02 +01:00
hanabi1224
83554afebc Merge remote-tracking branch 'origin/master' into close-write 2023-01-02 16:42:21 +08:00
Łukasz Magiera
156ba420c3 fix: splitstore: Don't deadlock in mpool protector 2022-12-17 13:14:50 +01:00
Aayush
28b4ebd52b chore: VM: don't export Extern::VerifyBlockSig 2022-12-16 16:54:09 -05:00
hanabi1224
6ae7cdab64 call Stream.CloseWrite right after writing 2022-12-16 17:22:53 +08:00
Geoff Stuart
d615b1c806 Update go-state-types 2022-12-14 19:33:47 -05:00
Aayush Rajasekaran
a264893543
Merge pull request #9873 from filecoin-project/gstuart/print-addr-key-not-found 
feat: wallet: Print addr when key not found
 2022-12-14 13:36:50 -05:00