cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
20,231 Commits 24 Branches 342 Tags 314 MiB
848c20d15f
Commit Graph

1450 Commits

Author SHA1 Message Date
Hector Sanjuan
be1614a042 range-export: check serverside that head > tail 2023-02-14 21:08:10 +01:00
Hector Sanjuan
ec01036871 Ranged export: remove non-internal ranged export method 
Additionally, per feedback:

* Set "admin" permissions
* Remove from v0api
 2023-02-14 21:08:10 +01:00
Hector Sanjuan
1bb698619c Ranged-export: Remove CachingBlockstore 
The improvements in the range-export code lead to avoid reading most blocks
twice, as well as to allowing some blocks to be written to disk multiple times.

The cache hit-rate went down from being close to 50% to a maximum of 12% at
the very end of the export. The reason is that most CIDs are never read twice
since they are correctly tracked in the CID set.

These numbers do not support the maintenance of the CachingBlockstore
code. Additional testing shows that removing it has similar memory-usage
behaviour and about 5 minute-faster execution (around 10%).

Less code to maintain and less options to mess up with.
 2023-02-14 21:08:10 +01:00
Hector Sanjuan
fa93c23813 Chain ranged export: rework and address current shortcomings 
This commit moderately refactors the ranged export code. It addresses several
problems:
  * Code does not finish cleanly and things hang on ctrl-c
  * Same block is read multiple times in a row (artificially increasing cached
    blockstore metrics to 50%)
  * It is unclear whether there are additional races (a single worker quits
    when reaching height 0)
  * CARs produced have duplicated blocks (~400k for an 80M-blocks CAR or
    so). Some blocks appear up to 5 times.
  * Using pointers for tasks where it is not necessary.

The changes:

  * Use a FIFO instead of stack: simpler implementation as its own type. This
has not proven to be much more memory-friendly, but it has not made things
worse either.
  * We avoid a probably not small amount of allocations by not using
    unnecessary pointers.
  * Fix duplicated blocks by atomically checking+adding to CID set.
  * Context-termination now works correctly. Worker lifetime is correctly tracked and all channels
  are closed, avoiding any memory leaks and deadlocks.
  * We ensure all work is finished before finishing, something that might have
  been broken in some edge cases previously. In practice, we would not have
  seen this except perhaps in very early snapshots close to genesis.

Initial testing shows the code is currently about 5% faster. Resulting
snapshots do not have duplicates so they are a bit smaller. We have manually
verified that no CID is lost versus previous results, with both old and recent
snapshots.
 2023-02-14 21:08:10 +01:00
frrist
21efd481d8 First efficient ranged-export implementation by @frisst 
This first commit contains the first and second implementation stabs (after
primary review by @hsanjuan), using a stack for task buffering.

Known issues: ctrl-c (context cancellation) results in the export code getting
deadlocked. Duplicate blocks in exports. Duplicate block reads from store.

Original commit messages:

works

works against mainnet and calibnet

feat: add internal export api method

- will hopfully make things faster by not streaming the export over the json rpc api

polish: better file nameing

fix: potential race in marking cids as seen

chore: improve logging

feat: front export with cache

fix: give hector a good channel buffer on this shit

docsgen
 2023-02-14 15:41:10 +01:00
Aayush
63b03a48d0 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-10 18:39:51 -05:00
Łukasz Magiera
0f05232199
Merge pull request #10224 from filecoin-project/feat/ethcli-strip-spaces 
feat: eth cli: Strip out empty spaces around contract bytes
 2023-02-09 19:24:49 +01:00
Łukasz Magiera
ada32395ca
Merge pull request #10160 from filecoin-project/gstuart/send-from-f4 
feat: cli: allow value send from f4 to any address type
 2023-02-09 18:53:40 +01:00
Łukasz Magiera
b0fa75f13e feat: eth cli: Strip out empty spaces around contract bytes 2023-02-09 18:49:08 +01:00
Phi-rjan
9848e0467e
Merge pull request #10209 from filecoin-project/fix-check-backupfile 
fix: cli: error if backup file already exists
 2023-02-09 18:39:17 +01:00
Phi-rjan
0e7cf61888
Update cli/backup.go 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2023-02-09 16:50:15 +01:00
Phi
b78cbcbb99 Error if backup file exists 
Error out if backup file already exists
 2023-02-08 11:33:48 +01:00
Aayush Rajasekaran
693f230fbb Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-07 11:03:01 -05:00
Aayush
f291572b4a Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-02-03 09:27:04 -05:00
Geoff Stuart
f05437d253 Check decoding params for new methods 2023-02-02 14:41:30 -05:00
Geoff Stuart
87397eaffa Allow f4 address to send to all address types if ID address exists on chain 2023-02-01 11:46:45 -05:00
Łukasz Magiera
965b1cf03c itests: Fix TestEthSubscribeLogs 2023-01-31 10:28:12 +01:00
Łukasz Magiera
1286d76988 gateway: eth_subscribe support 2023-01-31 10:28:12 +01:00
Łukasz Magiera
13fc5b688f
Merge pull request #10028 from filecoin-project/gstuart/cli-color-cleanup 
chore: cli: Remove unneeded individual color flags
 2023-01-31 09:23:16 +01:00
Phi
a0b25b9136 less strict ArgsCheck 
Less strict ArgsCheck
 2023-01-26 10:13:39 +01:00
Phi
34f376eb23 chore: cli 
Standarize cli/code functions similar to: https://github.com/filecoin-project/lotus/pull/9317

- cctx.NArg() instead of cctx.Args().xxx
- Add check for args and print help on functions that did not have it
 2023-01-25 13:13:56 +01:00
Aayush
65ac5669a9 Merge branch 'master' into asr/merge-release-into-master 2023-01-23 12:37:24 -05:00
Aayush
e9f665214a Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-23 12:23:47 -05:00
Geoff Stuart
807a60b4e4 Remove unneeded individual color flags 2023-01-23 11:11:12 -05:00
Łukasz Magiera
804bef72c9 fix: cli: Wait for all results in GetAsks 2023-01-23 12:48:21 +01:00
Aayush
4f199ada40 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-20 17:32:29 -05:00
Geoff Stuart
fff58e1368 Refactor EvmGetInfoCmd 2023-01-20 16:51:01 -05:00
Geoff Stuart
439b4c6e22 Cleanups 2023-01-20 16:51:01 -05:00
Shrenuj Bansal
522e96f016
fix: msg signing with delegated keys and send cli changes (#10056) 
* fix msg signing with delegated keys and send cli changes

* make gen and docsgen

* address comments
 2023-01-19 17:57:48 -05:00
ZenGround0
178aaf6ac4
fix broke invoke (#10031) 
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2023-01-19 11:41:11 -05:00
Travis Person
a0c58b5582 feat: add toolshed commands to inspect statetree size 2023-01-19 00:48:22 +00:00
Aayush
e715384173 Merge branch 'release/v1.20.0' into asr/merge-release-into-master 2023-01-16 10:51:22 -05:00
vyzo
859cdca4f7 update go-state-types 
use CreateExternal

use CreateExternal in itest evm deployment

add missing import

update gst
 2023-01-14 01:11:11 -05:00
raulk
cdf3812e40
NV18: Filecoin EVM runtime + Actor Events + EthAccount + EAM + f4 addressing (#9998) 
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Raul Kripalani <raulk@users.noreply.github.com>
Co-authored-by: Kevin Li <ychiaoli18@users.noreply.github.com>
Co-authored-by: vyzo <vyzo@hackzen.org>
Co-authored-by: Ian Davis <nospam@iandavis.com>
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Jennifer Wang <jiayingw703@gmail.com>
Co-authored-by: Geoff Stuart <geoff.vball@gmail.com>
Co-authored-by: Shrenuj Bansal <shrenuj.bansal@protocol.ai>
Co-authored-by: Shrenuj Bansal <108157875+shrenujbansal@users.noreply.github.com>
Co-authored-by: Geoff Stuart <geoffrey.stuart@protocol.ai>
Co-authored-by: Aayush Rajasekaran <aayushrajasekaran@Aayushs-MacBook-Pro.local>
Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
 2023-01-13 19:11:13 +00:00
Geoff Stuart
923d4128bb
Merge pull request #9875 from filecoin-project/gstuart/remove-expired-claim 
feat: cli: list claims and remove expired claims
 2023-01-12 12:44:21 -05:00
Łukasz Magiera
9a46682d9d
Merge pull request #9481 from consensus-shipyard/adlrocha/cns-iface-master 
IPC: Abstract common consensus functions and consensus interface
 2023-01-11 14:49:27 +01:00
Geoff Stuart
88d59f6a5f Set notifications for SIGTERM and SIGINT when we start the cli 2022-12-15 12:54:41 -05:00
Geoff Stuart
6ca975e83a Add cli command to get a list of claims 2022-12-15 01:07:47 -05:00
Geoff Stuart
754f294992 Add cli command for removing expired claims 2022-12-15 01:07:47 -05:00
Geoff Stuart
33a176eae4
Merge pull request #9753 from filecoin-project/gstuart/fix-devnet-deals 
fix: cli: Check upper bound for deal duration
 2022-12-12 09:33:23 -05:00
zenground0
e1a4cf02f4 Remove unused flag for clarity 2022-12-05 16:06:03 -07:00
Alfonso de la Rocha
075216d9da Merge remote-tracking branch 'upstream/master' into adlrocha/cns-iface-master 2022-12-05 18:16:14 +01:00
Łukasz Magiera
11633ff1f7 fix: cliutil: Fix URL-based API endpoint parsing 2022-12-02 20:16:40 +01:00
Geoff Stuart
c2e469095c Check upper bound for deal duration 2022-11-29 15:18:29 -05:00
Geoff Stuart
e6e50ee9f7 more linter 2022-11-25 17:05:41 -05:00
Ian Davis
9f85d3dca7 Address simple linter issues 2022-11-24 16:32:27 +00:00
Alfonso de la Rocha
0f92bced9d
Merge branch 'master' into adlrocha/cns-iface-master 2022-11-22 10:28:18 +01:00
Geoff Stuart
21afb3f5ea Review fixes 2022-11-16 15:07:23 -05:00
Geoff Stuart
62fedfbce3 Make pre-migration optional 2022-11-15 23:12:04 -05:00
Shrenuj Bansal
800d9de4d5 Address comments 2022-11-14 15:46:58 -05:00
Shrenuj Bansal
2fa21ff091 Merge branch 'master' into sbansal/nonce-coordination-and-consensus-for-chain-nodes 2022-11-11 14:41:38 -05:00
Łukasz Magiera
b540d10b48 cli: Move ClientExport to cliutil 2022-11-08 09:37:43 +00:00
Łukasz Magiera
e66d5a0537 cli: Move EpochTime to cliutil 2022-11-08 09:37:43 +00:00
Łukasz Magiera
2c89b3240f retrieval: Support retrievals into remote stores 2022-11-08 09:37:34 +00:00
Aayush
c0b7343e60 Merge branch 'release/v1.18.0' into asr/merge-release-into-master 2022-11-06 14:40:13 -05:00
Geoff Stuart
54d3e412f1
Merge pull request #9373 from sectrgt/master 
fix: chain: Update chain.go
 2022-11-04 10:52:52 -04:00
Aayush
13b59c9c23 Invoker: Use MethodMeta from go-state-types 2022-10-19 15:53:26 -04:00
Shrenuj Bansal
ad8b959880 Address more comments and add test for gorpc auth 2022-10-18 15:53:42 -04:00
Shrenuj Bansal
15ed1ee33c Merge branch 'master' into sbansal/nonce-coordination-and-consensus-for-chain-nodes 2022-10-17 22:55:48 -04:00
Shrenuj Bansal
674427a8b2 fix lint and make gen 2022-10-17 22:44:00 -04:00
Shrenuj Bansal
900525f8c2 some cleanup 2022-10-17 16:13:30 -04:00
Shrenuj Bansal
b77ca54719 Change cli cmd to API with proxy 2022-10-17 11:23:10 -04:00
Shrenuj Bansal
139f8773de fix some bugs and address some comments 2022-10-16 22:52:22 -04:00
Geoff Stuart
796ddc07b0
Update cli/filplus.go 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2022-10-13 13:07:58 -04:00
Alfonso de la Rocha
3105596c9e Clean pending files from rebase 2022-10-13 17:03:18 +02:00
Alfonso de la Rocha
88d3de7afc run make gen to make ci happy 2022-10-13 16:43:02 +02:00
Alfonso de la Rocha
627d6e70a6 parametrized reward function for consensus 2022-10-13 16:30:14 +02:00
Geoff Stuart
1ff4d1ae6c Add cli for listing allocations and removing expired allocations 2022-10-12 15:54:58 -04:00
Łukasz Magiera
e40fc14f85
Merge pull request #9436 from filecoin-project/feat/uptime 
feat: Add node uptime rpc / output in info command
 2022-10-11 18:20:41 +02:00
Łukasz Magiera
2c11f9d265 feat: Add node uptime rpc / output in info command 2022-10-11 10:11:09 +02:00
Phi
0645225801 fix: ux: specify arg in net ping cmd 
Specify that you need to ping the `peerMultiaddrs` with the `lotus net ping` and `lotus-miner net ping`.
 2022-10-10 08:31:55 +02:00
Aayush Rajasekaran
e57a27e12a
Merge pull request #9416 from filecoin-project/fix/wallet-cmd-outputs 
fix: ux: Add outputs to wallet commands
 2022-10-07 00:10:10 -04:00
Geoff Stuart
652c3707e9 update go-fil-markets 2022-10-06 11:06:21 -04:00
Geoff Stuart
1973e44f49 Update datacap actor to query datacap instead of verifreg 2022-10-06 11:06:21 -04:00
Peter Rabbitson
ce65cdd1e5 Teach shed/sim to understand --tipset=@nnn notation 2022-10-05 21:10:19 +02:00
Shrenuj Bansal
98481821d8 solution for mining loop hitting the same node 2022-10-05 16:52:49 +00:00
Phi
d8b6904492 fix: ux: Add cmd outputs 
Adding command outputs to `lotus wallet set-default` and `lotus wallet delete`
 2022-10-05 09:47:31 +02:00
Shrenuj Bansal
dde204fb6a Change Mpool push API to have an option to publish 2022-10-04 16:49:09 +00:00
Rob Quist
66b77be2d0
Add retrieval deal ID and bytes transferred to retrieval output 2022-09-30 02:43:01 +02:00
Shrenuj Bansal
f89a682d98 Add Mpool ref to raft state and rearrange some APIs 2022-09-29 10:56:57 +00:00
Shrenuj Bansal
570f61438a Retries within proxy working 2022-09-28 15:07:05 +00:00
Shrenuj Bansal
559c2c6d34 Merge branch 'master' into sbansal/nonce-coordination-and-consensus-for-chain-nodes 2022-09-27 16:29:03 +00:00
Shrenuj Bansal
99e7c322eb More wip 2022-09-27 16:08:04 +00:00
sectrgt
857c091576
Update chain.go 2022-09-25 21:01:08 -04:00
Łukasz Magiera
2d82527ee9 cli: Fix compute-state --html 2022-09-20 17:40:24 +02:00
Geoff Stuart
609dad02a2 actor-cids cli command now defaults to current network version instead of newest 2022-09-15 10:42:44 -04:00
Geoff Stuart
9d63e1412e print beneficiary info in state miner-info and lotus-miner info 2022-09-14 17:54:20 -04:00
Geoff Stuart
4f75e2041c Add helper function to print usage when wrong number of arguments are supplied 2022-09-14 15:38:15 -04:00
Geoff Stuart
3ca9b1e331 wait.Receipt.ExitCode != 0 to wait.Receipt.ExitCode.IsError() 2022-09-14 14:53:11 -04:00
Geoff Stuart
308cef950b cctx.Args().Len() to cctx.NArg() 2022-09-14 14:33:29 -04:00
Aayush
19c3232d2f more fixes 2022-09-10 12:37:03 -04:00
Geoff Stuart
642555d0fb Fix discrepancies, run fiximports 2022-09-09 21:40:15 -04:00
Aayush
0efca4d266 introduce v9 actors and nv17 2022-09-09 21:40:15 -04:00
Łukasz Magiera
783fc6a20d
Merge pull request #9233 from filecoin-project/feat/lotus-info-cmd 
feat: cli: lotus info cmd
 2022-09-05 09:47:42 +02:00
Phi
ab0699875a make gen happy 
make gen happy not sad
 2022-09-02 15:20:36 +02:00
Phi-rjan
a31b8a1b58
Update cli/info.go 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2022-09-02 15:12:32 +02:00
Phi
a5ea5ba9c1 Fix lint 
Fix lint
 2022-09-01 17:01:03 +02:00
Phi
405bfeddac Better ledger rejection error 
Print a more understandable error message when a transaction is being rejected on ledger.
 2022-09-01 16:30:36 +02:00
Phi
818153a1e7 Put SyncBasefeeCheck in function 
Put SyncBasefeeCheck in function
 2022-08-31 13:15:28 +02:00
Phi
870c2f9554 Move newline 
Move newline
 2022-08-31 10:44:59 +02:00