cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
20,835 Commits 24 Branches 342 Tags 314 MiB
a003b6ccda
Commit Graph

247 Commits

Author SHA1 Message Date
Hector Sanjuan
fa93c23813 Chain ranged export: rework and address current shortcomings 
This commit moderately refactors the ranged export code. It addresses several
problems:
  * Code does not finish cleanly and things hang on ctrl-c
  * Same block is read multiple times in a row (artificially increasing cached
    blockstore metrics to 50%)
  * It is unclear whether there are additional races (a single worker quits
    when reaching height 0)
  * CARs produced have duplicated blocks (~400k for an 80M-blocks CAR or
    so). Some blocks appear up to 5 times.
  * Using pointers for tasks where it is not necessary.

The changes:

  * Use a FIFO instead of stack: simpler implementation as its own type. This
has not proven to be much more memory-friendly, but it has not made things
worse either.
  * We avoid a probably not small amount of allocations by not using
    unnecessary pointers.
  * Fix duplicated blocks by atomically checking+adding to CID set.
  * Context-termination now works correctly. Worker lifetime is correctly tracked and all channels
  are closed, avoiding any memory leaks and deadlocks.
  * We ensure all work is finished before finishing, something that might have
  been broken in some edge cases previously. In practice, we would not have
  seen this except perhaps in very early snapshots close to genesis.

Initial testing shows the code is currently about 5% faster. Resulting
snapshots do not have duplicates so they are a bit smaller. We have manually
verified that no CID is lost versus previous results, with both old and recent
snapshots.
 2023-02-14 21:08:10 +01:00
frrist
21efd481d8 First efficient ranged-export implementation by @frisst 
This first commit contains the first and second implementation stabs (after
primary review by @hsanjuan), using a stack for task buffering.

Known issues: ctrl-c (context cancellation) results in the export code getting
deadlocked. Duplicate blocks in exports. Duplicate block reads from store.

Original commit messages:

works

works against mainnet and calibnet

feat: add internal export api method

- will hopfully make things faster by not streaming the export over the json rpc api

polish: better file nameing

fix: potential race in marking cids as seen

chore: improve logging

feat: front export with cache

fix: give hector a good channel buffer on this shit

docsgen
 2023-02-14 15:41:10 +01:00
Raúl Kripalani
8ff55c027e Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-13 11:45:09 +00:00
ychiao
3c580403c1 implement itest and handle optional params 2023-02-10 13:33:59 -05:00
Aayush
f291572b4a Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-03 09:27:04 -05:00
Łukasz Magiera
a6a2f0dbae gateway: Support all EthModule methods 2023-02-02 13:37:31 +01:00
Łukasz Magiera
ad14d71978 eth rpc: Params are optional in eth_subscribe 2023-01-31 11:08:51 +01:00
Łukasz Magiera
1286d76988 gateway: eth_subscribe support 2023-01-31 10:28:12 +01:00
Łukasz Magiera
6491becbe1 rpc: Switch eth_subscribe to reverse calls 2023-01-31 10:28:12 +01:00
Jorropo
f572852d06 chore: all: bump go-libipfs to replace go-block-format 
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58
 2023-01-26 17:03:18 +01:00
ychiao
ee6af9de61 Eth JSON-RPC: implement web3_clientVersion 2023-01-24 22:56:04 -05:00
Geoff Stuart
5133b9da94 Add EthGetMessageCidByTransactionHash, fix EthGetTransactionHashByCid 2023-01-18 17:48:10 -05:00
Geoff Stuart
a8436074a6 Store mapping from hashes for Ethereum transactions to Filecoin Message Cids 2023-01-16 01:17:56 -05:00
Łukasz Magiera
708618d11c feat: gateway: eth_ api support 2023-01-13 20:04:41 +01:00
Geoff Stuart
4ae498a72e Fixes 2022-12-16 17:24:46 -05:00
Steven Allen
266492fa1c Merge branch 'feat/nv18-fevm' into feat/nv18-events 2022-12-07 22:16:56 -08:00
Aayush Rajasekaran
6443e72ad7 Merge branch 'master' into feat/nv18-fevm 2022-11-29 20:16:12 -06:00
ychiao
f1493fbee7
eth: rpc: fix sendRawTransaction, transactionIndex, feeHistory (#9736) 
* Eth JSON-RPC API: add transactionIndex

* Eth JSON-RPC API: fix EthFeeHistory and EthSendRawTransaction

* Eth JSON-RPC API: fix parsing block parameter for getBlockByNumber

* fix itest
 2022-11-27 18:44:43 -08:00
Łukasz Magiera
87e5549af5 sealing: Split unsealed cleanup from Finalize 2022-11-23 18:57:16 +01:00
Steven Allen
176275267b Merge branch 'feat/nv18' into feat/nv18-fevm 2022-11-22 10:32:26 -08:00
Raúl Kripalani
32385a97cd Eth JSON-RPC API: return logs in eth_getTransactionReceipt. 2022-11-16 13:53:27 +00:00
Ian Davis
314fb31886 Fix signature of EthSubscribe 2022-11-16 12:18:25 +00:00
Shrenuj Bansal
b95d1a6323 Merge branch 'master' into sbansal/nonce-coordination-and-consensus-for-chain-nodes 2022-11-14 17:50:52 -05:00
Geoff Stuart
95af40543a Fix lotus-miner proving check 2022-11-14 15:29:50 -05:00
Shrenuj Bansal
2fa21ff091 Merge branch 'master' into sbansal/nonce-coordination-and-consensus-for-chain-nodes 2022-11-11 14:41:38 -05:00
Ian Davis
7383ecb0ba make gen 2022-11-10 15:04:58 +00:00
vyzo
deea6797c5 make gen 2022-11-10 02:52:06 +02:00
Raúl Kripalani
dc5fc9ded4 fix eth_feeHistory. 2022-11-09 19:47:07 +02:00
Raúl Kripalani
32617a61df fix eth_feeHistory: add missing param. 2022-11-09 19:47:06 +02:00
Kevin Li
64afdfc642 api: ethrpc: implement eth_feeHistory (#9539) 
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
 2022-11-09 19:46:22 +02:00
Kevin Li
e17e92775c api: ethrpc: fix eth_getCode, eth_getTransactionReceipt (#9530) 2022-11-09 19:40:00 +02:00
raulk
ee83dceef0 Eth JSON-RPC API: implement eth_getCode and eth_getStorageAt (#9397) 2022-11-09 19:30:06 +02:00
Kevin Li
6c7fd94aa7 api: ethrpc: implement eth_sendRawTransaction (#9334) 
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
 2022-11-09 19:28:17 +02:00
Kevin Li
b53d5924a1 feat: ethrpc: implement call, estimateGas, getTransactionCount (#9306) 2022-11-09 19:25:40 +02:00
Kevin Li
e5bb5b7430 feat: ethrpc: implement EthBlock and EthTx structs (#9287) 
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
 2022-11-09 19:23:49 +02:00
Kevin Li
4652d8559f api: ethrpc: implement a few ethereum JSON-RPC API methods (#9208) 2022-11-09 19:23:11 +02:00
Aayush
c0b7343e60 Merge branch 'release/v1.18.0' into asr/merge-release-into-master 2022-11-06 14:40:13 -05:00
Shrenuj Bansal
15ed1ee33c Merge branch 'master' into sbansal/nonce-coordination-and-consensus-for-chain-nodes 2022-10-17 22:55:48 -04:00
Łukasz Magiera
2c11f9d265 feat: Add node uptime rpc / output in info command 2022-10-11 10:11:09 +02:00
Geoff Stuart
e2d5d12e7f Add accessors for allocations and claims maps 2022-10-07 16:41:59 -04:00
Geoff Stuart
b5c5e6627e review fixes 2022-10-06 11:06:21 -04:00
Geoff Stuart
f55dc46a32 Add api for getting allocation 2022-10-06 11:06:21 -04:00
Shrenuj Bansal
f89a682d98 Add Mpool ref to raft state and rearrange some APIs 2022-09-29 10:56:57 +00:00
Shrenuj Bansal
559c2c6d34 Merge branch 'master' into sbansal/nonce-coordination-and-consensus-for-chain-nodes 2022-09-27 16:29:03 +00:00
Shrenuj Bansal
99e7c322eb More wip 2022-09-27 16:08:04 +00:00
Łukasz Magiera
859c2606f0 sealing: Address review 2022-09-19 12:13:06 +02:00
Łukasz Magiera
fbb487ae2b sector import: Plumbing for DownloadSectorData in the sealing system 2022-09-19 12:13:05 +02:00
Łukasz Magiera
2318419291 sector import: Fix missing perm tag on SectorReceive 2022-09-19 12:13:05 +02:00
Łukasz Magiera
29135aa77c sector import: Initial api scaffolding 2022-09-19 12:13:03 +02:00
Geoff Stuart
21906b5a63 add api and cli calls for beneficiary withdrawl 2022-09-14 12:26:35 -04:00
Shrenuj Bansal
4be8861f41 Merge branch 'master' into sbansal/nonce-coordination-and-consensus-for-chain-nodes 2022-09-13 17:41:06 -04:00
Shrenuj Bansal
a1f2fdb706 Merge branch 'master' into sbansal/nonce-coordination-and-consensus-for-chain-nodes 2022-09-12 16:23:24 -04:00
Shrenuj Bansal
8f1b1bb1ff WIP: Raft consensus for lotus nodes in a cluster 2022-09-12 16:10:15 -04:00
Geoff Stuart
642555d0fb Fix discrepancies, run fiximports 2022-09-09 21:40:15 -04:00
Aayush
0efca4d266 introduce v9 actors and nv17 2022-09-09 21:40:15 -04:00
LexLuthr
67d4f905e6
feat: proving: Introduce manual sector fault recovery (#9144) 
* rebase

* fix lint error

* fix errors

* add itest

* make gen after rebase

* apply suggestion from review

* make gen after latest rebase

* gen after rebase

* fix waitgroup

* change as per review

* refactor Recovery Batches

* fix CLI

* gen after rebase

* close the channel
 2022-09-06 21:17:30 +05:30
Łukasz Magiera
45d1bd61ce
Merge pull request #9183 from filecoin-project/feat/sectornum-mgmt 
feat: sealing: Use bitfields to manage sector numbers
 2022-08-26 10:59:24 -04:00
Łukasz Magiera
2086b219d2 Don't use go-libp2p-core 2022-08-25 14:20:41 -04:00
Łukasz Magiera
335f5df3ec numassigner: SectorNumReserveCount 2022-08-24 11:25:37 -04:00
Łukasz Magiera
b290dac3cb Merge remote-tracking branch 'origin/master' into feat/sectornum-mgmt 2022-08-23 11:44:43 -04:00
Łukasz Magiera
64290b69c9 numassigner: Fix missing api perms 2022-08-22 17:41:41 -04:00
Geoff Stuart
7d07ddd837 Reintroduce StateActorManifestCID 2022-08-22 17:17:01 -04:00
Łukasz Magiera
00bef607ca sealing: numassign: take on-chain allocated numbers into account 2022-08-22 16:55:44 -04:00
Łukasz Magiera
ef2080a800 cli for managing sector reservations 2022-08-22 16:55:41 -04:00
Shrenuj Bansal
f390c186e6 Close all storage paths on worker shutdown 2022-08-10 17:06:01 -04:00
ZenGround0
0c91b0dc10
feat:chain:splitstore chain prune (#9056) 
* Splitstore chain prune
* Protect on reification for simpler logic and sound cold compact protect
* Recovery from checkpoint during chain prune
* Splitstore (discard and universal mode) running in itests
* Add pause and restart functions to itest block miner
* Add config options to itest full nodes
* Add FsRepo support for itest full ndoes

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2022-08-05 16:34:16 -04:00
Łukasz Magiera
cb2b4fa4aa Merge remote-tracking branch 'origin/master' into feat/storage-redeclare 2022-08-05 21:26:15 +02:00
Łukasz Magiera
819314de46 Merge remote-tracking branch 'origin/master' into feat/storage-redeclare 2022-08-05 09:01:30 +02:00
LexLuthr
c736dedfa6 move requestremove inside runSched 2022-08-04 02:00:24 +05:30
LexLuthr
cf78fa99ee Rebase on master 2022-08-04 01:59:48 +05:30
Łukasz Magiera
4b47881950 Merge branch 'master' into sbansal/lotus-worker-stop-cmd 2022-08-03 10:42:19 +02:00
LexLuthr
f5a7a650e7 rename to ActorWithdrawBalance 2022-08-02 13:04:54 +05:30
LexLuthr
e0e0b0b62f don't wait for msg to land on chain 2022-08-01 23:41:24 +05:30
Łukasz Magiera
3cd8d9eb06 itests: Worker tests for storage detach/redeclare 2022-08-01 15:58:06 +02:00
Łukasz Magiera
8cff52aef6 Storage detach/attach in lotus-miner, cli commands 2022-08-01 15:58:06 +02:00
Łukasz Magiera
943d2a72c6 storage: StorageRedeclareLocal on lotus-worker 2022-08-01 15:58:06 +02:00
Łukasz Magiera
e252e666bd storage: Storage detach on lotus-worker 2022-08-01 15:58:06 +02:00
Łukasz Magiera
f947e65f15 wip storage redeclare/detach 2022-08-01 15:58:06 +02:00
Shrenuj Bansal
dfea74cca8 Add stop cmd for lotus worker 2022-07-29 18:33:30 -04:00
LexLuthr
4f03486011 miner withdrawbalance API 2022-07-29 20:09:49 +05:30
Łukasz Magiera
413183e4fa fix: Delegate storage auth on market nodes 2022-07-06 22:22:41 +02:00
Jennifer Wang
c3f3eb0812 Merge branch 'releases' into jen/masterbp 2022-06-27 15:13:12 -04:00
Geoff Stuart
e684248f48 Added api call to get actors cids 2022-06-23 14:07:23 -04:00
Łukasz Magiera
58416d0881 storage: Don't depend on specs-storage, move to storiface 2022-06-17 13:36:33 +02:00
Łukasz Magiera
9c4d10ec73 api: handle no-precommit in StateSectorPreCommitInfo gracefully 2022-06-16 15:20:58 +02:00
Łukasz Magiera
2dc1df878d api: Move StateComputeDataCID to full-node 2022-06-16 13:29:35 +02:00
Łukasz Magiera
6d770f6e10 make gen 2022-06-15 16:51:30 +02:00
Łukasz Magiera
a9600b8a6f storage: Move extern/sector-storage to storage/sealer 2022-06-14 20:03:38 +02:00
Łukasz Magiera
98a48a47f8 storage: Move extern/storage-sealing to storage/pipeline 2022-06-14 19:41:59 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Geoff Stuart
b7010c9e60 Implement function to migrate actors with only code changes 2022-06-10 15:52:32 -04:00
Geoff Stuart
801c670edd Remove ChainPutMany 2022-06-10 14:09:05 -04:00
Geoff Stuart
5c0f2c8ae6 Add putObj and putMany to apiBlockstore 2022-06-09 15:13:42 -04:00
Aayush
b28c11a57d Merge branch 'feat/nv16' 2022-06-03 14:01:49 -04:00
Jennifer Wang
2e59d0129d Merge branch 'release/v1.15.3' into jen/mergev1153to16 2022-05-31 16:33:18 -04:00
Łukasz Magiera
032e598962 feat: gateway: OpenRPC support 2022-05-27 17:03:56 +02:00
Aayush Rajasekaran
6924a3d5f2 Fix calculation of Drand round from Filecoin epochs 2022-05-25 12:43:52 -04:00
Aayush
8cca9b1970 Use new go-state-types accessors 2022-05-17 15:21:27 -04:00
LexLuthr
7e492f23b1 add dagstore register-shard command 2022-05-12 19:46:53 +05:30
simlecode
962e37e893 add StateGetNetworkParams api 2022-05-05 14:16:30 +08:00