cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
14,238 Commits 24 Branches 342 Tags 314 MiB
a20916f9af
Commit Graph

2441 Commits

Author SHA1 Message Date
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
raulk
473a192b85
fix build after merging #6097. (#7096) 
fix build after merging #6097.
 2021-08-16 23:18:32 +01:00
ZenGround0
10615b53ad
Merge pull request #6853 from filecoin-project/frrist/miner-temp-file-path 
feat(miner deals): create subdir to miner repo for staged deals
 2021-08-16 16:19:05 -04:00
ZenGround0
2e5b492edd
Merge pull request #6097 from filcloud/sectors-renew 
Extending sectors: more practical and flexible tools
 2021-08-16 13:50:38 -04:00
Aayush Rajasekaran
90e60f7a98 Shed: Create a verifreg command for when VRK isn't a multisig 2021-08-16 12:56:27 -04:00
He Weidong
b50b1e9ac4 Optimize for ci 2021-08-14 10:41:01 +08:00
He Weidong
c5f81d35f5 minor modification 2021-08-13 13:47:48 +08:00
Raúl Kripalani
5f6166435c make lotus-miner net commands hit markets subsystem. 2021-08-12 11:15:00 +01:00
Steven Allen
dcff06bd91 test: genesis actors version 
Properly handle genesis in actors tests. Fast-forward upgrading to
actors v13 doesn't work because there needs to be at least a day between
v0 genesis and v13 (due to differences in miner cron).
 2021-08-11 13:30:15 -07:00
swift-mx
d6562bccfc
Merge branch 'filecoin-project:master' into master 2021-08-11 19:25:27 +08:00
mx
46a93bb78a fix: get other actor controlList 2021-08-11 19:23:37 +08:00
lanzafame
0b8a211e70 fix: init restore adds empty storage.json 2021-08-11 13:36:01 +10:00
Steven Allen
130fc6fce1
Merge pull request #7019 from filecoin-project/asr/genesis-shed 
Shed: Include network name in genesis-verify
 2021-08-10 16:25:02 -07:00
Aayush Rajasekaran
feb9d3c557 Shed: Include network name in genesis-verify 2021-08-10 14:46:24 -04:00
ZenGround0
6da26dcefc Remove network version panic 2021-08-10 13:07:30 -04:00
Frank
2744bb4e5a update 2021-08-04 15:49:35 +08:00
Frank
2fec0c2440 add produced blocks info 2021-08-04 15:41:06 +08:00
vyzo
c9124bd1a9 clear splitstore dir before deleting, only warn if the latter fails 
to address situations with symlinks
 2021-08-03 15:39:48 +03:00
vyzo
db6b7b91d8 add option to only clear splistore keys 2021-08-03 15:32:53 +03:00
He Weidong
94c5076710 Make flags more intuitive 2021-08-02 16:44:22 +08:00
He Weidong
66e8418125 Enable exclude 2021-08-02 16:39:16 +08:00
He Weidong
677afbcb91 update to miner5 2021-08-02 16:26:29 +08:00
He Weidong
7f6c942b81 Use max-fee instead of gas-feecap 2021-08-02 16:09:00 +08:00
He Weidong
e09cdc45a7 Add gas-feecap flag for cmd sectors renew 2021-08-02 16:08:57 +08:00
He Weidong
9ee80fd1db Prettify json output of extending params 2021-08-02 16:08:55 +08:00
He Weidong
77166f88ac Optimize search of sectors location 2021-08-02 16:08:52 +08:00
He Weidong
e0777fed15 Add a more flexible cli tool: sectors renew 2021-08-02 16:08:50 +08:00
He Weidong
075bb3b314 Add a cli tool: sectors check-expire 2021-08-02 16:08:47 +08:00
vyzo
4a1655b225 add lotus-shed splitstore clear command 2021-08-01 19:18:29 +03:00
frrist
ca697c9aa2 feat(miner deals): create subdir to miner repo for staged deals 2021-07-29 21:08:38 -07:00
Aayush Rajasekaran
e5fb4a6c04
Merge pull request #6942 from filecoin-project/asr/nit 
Improve formatting of workers CLI
 2021-07-29 15:09:06 -04:00
Aayush Rajasekaran
1f7d584517
Update cmd/lotus-miner/sealing.go 2021-07-29 14:46:19 -04:00
Aayush Rajasekaran
28e3c8bd3f Improve formatting of workers CLI 2021-07-29 11:46:42 -04:00
Raúl Kripalani
8433aae1af use fallback api infos last; init service with markets-path. 2021-07-29 16:10:04 +01:00
Raúl Kripalani
b17fcfc4d1 bugfix. 2021-07-29 14:36:04 +01:00
Raúl Kripalani
90427bc3af fix docs, add flag. 2021-07-29 13:39:03 +01:00
Raúl Kripalani
3b2c3a3f6b polish. 2021-07-29 12:56:08 +01:00
Raúl Kripalani
d9c6a32b8e refactor miner info command. 2021-07-29 12:54:08 +01:00
Raúl Kripalani
22c0884f5f target markets API for markets commands. 2021-07-29 12:37:29 +01:00
raulk
80aedcd276
Merge branch 'master' into nonsense/get-enabled-subsystems-api 2021-07-28 20:09:43 +01:00
Raúl Kripalani
07f40b9488 fix docs and nits. 2021-07-28 20:03:25 +01:00
Łukasz Magiera
25052fe48f Remove chainwatch (now filecoin-project/sentinel) 2021-07-28 17:59:01 +02:00
Łukasz Magiera
389f71251c Remove townhall 2021-07-28 17:58:46 +02:00
Anton Evangelatov
6969202a03 make linter happy 2021-07-28 17:47:01 +03:00
Anton Evangelatov
54c6a3a454 extract outputs for mining node and markets node in separate functions 2021-07-28 17:31:48 +03:00
Anton Evangelatov
de4a847078 add RuntimeSubsystems API method; use it in lotus-miner info 2021-07-28 16:02:05 +03:00
Jiaying Wang
29b2d27f00
Merge branch 'master' into chore/cleanup-chain 2021-07-27 18:04:17 -04:00
raulk
35c90349ce
Merge pull request #6840 from filecoin-project/nonsense/lotus-shed-support-rollback-for-mra 
lotus-shed: initial export cmd for markets related metadata
 2021-07-27 22:13:29 +01:00
Jiaying Wang
d45b4a7a1f
Merge branch 'master' into chore/cleanup-chain 2021-07-27 17:11:36 -04:00
Jiaying Wang
4918a7c16f
Merge branch 'master' into feat/cli-super-verbose 2021-07-27 17:07:16 -04:00
raulk
e4b225f4ba
Merge branch 'master' into feat/cli-super-verbose 2021-07-27 20:49:44 +01:00
Raúl Kripalani
b04fb75a92 rename flag to very verbose. 2021-07-27 20:49:30 +01:00
vyzo
7c28ff3826
Merge branch 'master' into feat/moving-gc 2021-07-27 21:48:51 +03:00
Łukasz Magiera
e163d556fb
Merge branch 'master' into chore/cleanup-chain 2021-07-27 17:49:42 +02:00
Jiaying Wang
e46f02f59d
Merge branch 'master' into feat/vis-sector-alloc 2021-07-27 09:49:33 -04:00
Łukasz Magiera
ae63a4b33b fix lotus-sim build 2021-07-27 15:49:01 +02:00
Łukasz Magiera
9bd312881d VMSys doesn't belong in chainstore 2021-07-27 15:30:23 +02:00
Raúl Kripalani
48cddd3644 add a super verbose -vv flag to lotus and lotus-miner. 2021-07-27 14:25:28 +01:00
Raúl Kripalani
e0522a6b1f rename cmd/lotus{-storage=>}-miner to match binary. 2021-07-27 11:54:46 +01:00
Jiaying Wang
790f27d6a2
Merge branch 'master' into feat/vis-sector-alloc 2021-07-26 16:15:15 -04:00
Anton Evangelatov
9d61f91cdf
Update cmd/lotus-shed/market.go 
Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
 2021-07-26 16:41:22 +02:00
Jakub Sztandera
7de866d881
lint = happy 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2021-07-26 16:39:12 +02:00
vyzo
4715b1f436 fix lotus-shed 2021-07-26 15:04:46 +03:00
vyzo
221dc7024f add splitstore info command 2021-07-26 08:47:02 +03:00
vyzo
c99dc3e811 add splitstore check command 2021-07-25 11:24:06 +03:00
vyzo
e317c831a8 quiet excessive badger logs 2021-07-25 09:07:27 +03:00
vyzo
ce6f410f2a add options to control compaction/gc of the coldstore and config rewrite during rollback 2021-07-25 09:01:34 +03:00
vyzo
254c489164 fix typo 
Co-authored-by: raulk <raul@protocol.ai>
 2021-07-25 09:01:34 +03:00
vyzo
33cdc90f13 fix typo 
Co-authored-by: raulk <raul@protocol.ai>
 2021-07-25 09:01:34 +03:00
vyzo
da66e7a878 fix typo 2021-07-25 09:01:34 +03:00
vyzo
e696a2c0cc fix newline in progres message 2021-07-25 09:01:34 +03:00
vyzo
36b209ca5c compact and gc coldstore after copying 2021-07-25 09:01:34 +03:00
vyzo
92b9d8c895 quiet linter 2021-07-25 09:01:34 +03:00
vyzo
0c68bccbda add splitstore rollback lotus-shed command 2021-07-25 09:01:34 +03:00
Jakub Sztandera
e615eecbea Skip also set sectors 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2021-07-23 19:01:50 -04:00
Jakub Sztandera
39cc536115 Add sectors vis 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2021-07-23 19:01:48 -04:00
Łukasz Magiera
ac2a1bfeab fix lint 2021-07-23 15:53:31 +02:00
Łukasz Magiera
4757b014bc Nicer lotus-miner config commands 2021-07-23 14:57:42 +02:00
Łukasz Magiera
8b1f19e94c Move doc-comment logic to the config pkg 2021-07-23 14:55:19 +02:00
Łukasz Magiera
97f5bb66c7 config: doc struct codegen 2021-07-23 13:55:50 +02:00
Łukasz Magiera
cf65804883 lotus daemon config utils 2021-07-23 13:09:03 +02:00
Dirk McCormick
52498f78a1 feat: add import market datastore cmd 2021-07-23 10:30:58 +02:00
Dirk McCormick
7ef167b04f refactor: simplify market datastore backup 2021-07-23 09:33:26 +02:00
Anton Evangelatov
adba595350 commit from @dirkmc - initial export cmd for martkers related metadata 2021-07-22 18:41:57 +02:00
Łukasz Magiera
15a6995cda fix lint 2021-07-22 16:31:04 +02:00
Łukasz Magiera
0656b62176 Merge remote-tracking branch 'origin/master' into feat/compact-sectors-numbers-cmd 2021-07-22 16:04:25 +02:00
Steven Allen
f9c759fcb2 fix: always check if StateSearchMessage returns nil 
It returns nil on "not found".

Fixes the cases not covered in:
https://github.com/filecoin-project/lotus/pull/6787
 2021-07-20 11:28:27 -07:00
Peter Rabbitson
a37ababc20 Minor rearrange 2021-07-14 08:32:07 +02:00
Peter Rabbitson
677c193e6c Add current power to the stats 2021-07-14 08:11:23 +02:00
Łukasz Magiera
dd09666399
Merge pull request #6743 from filecoin-project/chore/centralize_all_cli_color_handling 
Handle the --color flag via proper global state
 2021-07-13 17:34:25 +02:00
Łukasz Magiera
837322ea59
Merge pull request #6356 from filecoin-project/nonsense/split-market-miner-processes 
Support standalone miner-market process
 2021-07-13 17:16:07 +02:00
Anton Evangelatov
2af02af1c1 rename --name to --type; provide empty stores.StorageConfig when init new service 2021-07-13 12:57:43 +02:00
Peter Rabbitson
6efd2e7335 Handle the --color flag via proper global state 
Use `color.NoColor` as a global progressively modified as one descends
down command flags. Allows non-surprising operation when --color is
accepted at multiple layers, and even makes this behave predictably:

    lotus-miner --color actor control list --color=false
 2021-07-13 12:22:29 +02:00
Łukasz Magiera
4433c3cd7c
Merge pull request #6708 from filecoin-project/feat/miner-info-deals 
Add more deal details to lotus-miner info
 2021-07-12 22:18:56 +02:00
Anton Evangelatov
7728d6bfc3 remove --enable-markets flag; add --name string slice flag 2021-07-12 14:38:29 +02:00
Anton Evangelatov
4bc9fa04ba rename cfg.Subsystems.EnableStorageMarket to EnableMarkets 2021-07-12 12:12:29 +02:00
Anton Evangelatov
d89ddb9315 resolve conflicts 2021-07-12 11:34:37 +02:00
Anton Evangelatov
16784aa2cc remove pieceProvider from DI; small refactors 2021-07-12 11:30:26 +02:00
Aayush Rajasekaran
20582037c5 Fix Lotus shed 2021-07-11 14:03:22 -04:00
Aayush Rajasekaran
e983d90a28
Merge pull request #6066 from chadwick2143/extend-v1-sectors 
Fix bugs in sectors extend --v1-sectors
 2021-07-11 13:19:35 -04:00
Aayush Rajasekaran
ea9ad8c5a8
Update cmd/lotus-storage-miner/sectors.go 2021-07-11 13:01:19 -04:00
Aayush Rajasekaran
93d929a10a
Merge pull request #5950 from zgfzgf/feat-shed-election-backtest 
add election backtest
 2021-07-11 12:08:21 -04:00
Łukasz Magiera
44e81424f2 Print completed deal stats in miner info 2021-07-08 19:40:05 +02:00
Łukasz Magiera
ed0cc0b56d Handle bad peerIDs in MarketListRetrievalDeals 2021-07-08 19:27:03 +02:00
Łukasz Magiera
10825e3a95 Add more info about deals to miner info 2021-07-08 19:02:18 +02:00
Łukasz Magiera
8f7bf1df86
Merge pull request #6697 from filecoin-project/feat/dealpublish-ctl-config 
Config for deal publishing control addresses
 2021-07-08 19:00:29 +02:00
Peter Rabbitson
140a2e0200 Stabilize default text and make gen docsgen-cli 2021-07-08 10:44:13 +02:00
Peter Rabbitson
c7bb326c78 Enable color by default only if os.Stdout is a TTY 2021-07-07 18:12:24 +02:00
Łukasz Magiera
8f5c0c60f5 Config for deal publishing control addresses 2021-07-07 18:00:54 +02:00
Anton Evangelatov
6b014f57e5 pass Subsystems to StorageMiner option; add enableLibp2p bool in Settings 2021-07-07 13:56:37 +02:00
Anton Evangelatov
7a78527fde remove not used sa; cleanup 2021-07-06 17:46:21 +02:00
turuslan
8697edc8b7 fix "lotus-seed car" error "merkledag: not found" 2021-07-06 12:54:09 +03:00
Anton Evangelatov
4be0a7a215 resolve conflicts with master 2021-07-05 13:13:32 +02:00
Łukasz Magiera
8a94ab676e storage: Fix FinalizeSector with sectors in stoage paths 2021-07-02 19:54:45 +02:00
Anton Evangelatov
adba1a792a add missing err check 2021-07-02 15:15:16 +02:00
Anton Evangelatov
ab6e01e379 remove --enable-markets flag, read from repo 2021-07-02 15:08:25 +02:00
Anton Evangelatov
3aa6d03cc1 connect miner to full node , only when running markets 2021-06-30 16:45:11 +02:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
Anton Evangelatov
9ec7cd7239 enable libp2p options 2021-06-30 12:55:43 +02:00
Łukasz Magiera
022d4b548a shed tool to estimate aggregate network fees 2021-06-29 20:42:26 +02:00
Łukasz Magiera
cefd140e45
Merge pull request #6329 from filecoin-project/raulk/itests-refactor-kit 
revamped integration test kit (aka. Operation Sparks Joy)
 2021-06-28 13:20:49 +02:00
Anton Evangelatov
7fb93d4585 refactor kit.NewDealHarness to include main miner and market node 2021-06-23 11:24:55 +02:00
Steven Allen
63e2caae81 lint(lotus-sim): handle error 2021-06-22 15:06:44 -07:00
Steven Allen
87c306fd47 feat(lotus-sim): use current power instead of lookback 
I'd _really_ like to use lookback, but don't have that when starting
from a snapshot.
 2021-06-22 15:06:30 -07:00
Steven Allen
0b06de2bd3 fix(lotus-sim): unembed Node from Simulation 
I wanted to expose the node's _fields_, but this also exposed the
methods. That got rather confusing.

(probably could have used a new type, but eh)

foo
 2021-06-22 14:53:06 -07:00
Anton Evangelatov
6720463799 resolve merge conflicts 2021-06-22 11:28:23 +02:00
Steven Allen
b7c36bc02c fix(lotus-sim): make NewNode take a repo 
This is primarily for testing, so we can just pass an in-memory repo.
 2021-06-21 11:32:19 -07:00
Steven Allen
c532b1b819 fix(lotus-sim): remove unused field and function 2021-06-21 11:25:40 -07:00
Steven Allen
80eba1069a feat(lotus-sim): NewNode to construct a node without a repo 2021-06-21 11:25:40 -07:00
Steven Allen
ffb63a93ff fix(lotus-sim): make 'fund' easier to understand 2021-06-21 11:25:40 -07:00
Raúl Kripalani
83ee345b1c Merge branch 'master' into raulk/itests-refactor-kit 2021-06-21 18:19:26 +01:00
Steven Allen
d6abcff63c
fix(lotus-sim): apply code review from magik6k 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-06-21 09:05:48 -07:00
Raúl Kripalani
7ee184e965 fix lotus-storage-miner tests. 2021-06-21 11:56:10 +01:00
Steven Allen
69a8a6bc0e fix(lotus-sim): lint 2021-06-18 15:44:37 -07:00
Jakub Sztandera
2fdf49e7da Add histogram and quantiles for message sizes 
Resolves https://github.com/filecoin-project/lotus/issues/6513

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2021-06-18 15:44:37 -07:00
Steven Allen
eb2b706156 chore: fix lint errors in simulation 2021-06-18 15:44:37 -07:00
Steven Allen
2aedd82c72 fix(lotus-sim): correct window post batch sizes 2021-06-18 15:44:37 -07:00
Steven Allen
bc2698a988 fix(lotus-sim): simulate using realistic gas numbers 
Previously, we assumed the network was "optimal". Now, we're using real
numbers.
 2021-06-18 15:44:37 -07:00
Steven Allen
a26cd5a809 fix(lotus-sim): mark provecommit stage initialized 2021-06-18 15:44:37 -07:00
Steven Allen
f0d0b40bd3 fix(lotus-sim): debug log mock 2021-06-18 15:44:37 -07:00
Steven Allen
ce29a0ac17 fix(lotus-sim): initialize commit queue 2021-06-18 15:44:37 -07:00
Steven Allen
28e6fa5923 fix(lotus-sim): remove duplicate error handling 2021-06-18 15:44:37 -07:00
Steven Allen
ec5fab09a1 feat(lotus-sim): log failing proofs 2021-06-18 15:44:37 -07:00
Steven Allen
e41f0842b0 fix(lotus-sim): load prove-commits 
(regression from refactor)
 2021-06-18 15:44:37 -07:00
Steven Allen
af33d69357 fix(lotus-sim): don't close node on list 2021-06-18 15:44:37 -07:00
Steven Allen
885062f712 fix(lotus-sim): fix info state imports 2021-06-18 15:44:37 -07:00
Steven Allen
73ae1924bc feat(lotus-sim): state size command 2021-06-18 15:44:37 -07:00
Steven Allen
22267eb45d feat(lotus-sim): split info command file 2021-06-18 15:44:36 -07:00
Steven Allen
5d7b7ce5c1 feat(lotus-sim): allow profile info 2021-06-18 15:44:36 -07:00
Steven Allen
95cf577447 fix(lotus-sim): really cancel walk immediately 2021-06-18 15:44:36 -07:00