Commit Graph

220 Commits

Author SHA1 Message Date
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ()
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Peter Rabbitson
6efd2e7335 Handle the --color flag via proper global state
Use `color.NoColor` as a global progressively modified as one descends
down command flags. Allows non-surprising operation when --color is
accepted at multiple layers, and even makes this behave predictably:

    lotus-miner --color actor control list --color=false
2021-07-13 12:22:29 +02:00
Peter Rabbitson
140a2e0200 Stabilize default text and make gen docsgen-cli 2021-07-08 10:44:13 +02:00
Peter Rabbitson
c7bb326c78 Enable color by default only if os.Stdout is a TTY 2021-07-07 18:12:24 +02:00
Aayush Rajasekaran
cb4d7cb9e7 Make query-ask CLI more graceful 2021-06-24 12:57:54 -04:00
Aayush Rajasekaran
4af59e0188
Apply suggestions from code review
Co-authored-by: Jennifer <42981373+jennijuju@users.noreply.github.com>
2021-06-22 19:23:24 -04:00
Aayush Rajasekaran
07fad6a201 Fix helptext 2021-06-22 16:02:06 -04:00
aarshkshah1992
35a466f4c4 add new line 2021-06-04 21:41:09 +02:00
aarshkshah1992
ed4748e8ac fix bug 2021-06-04 21:41:09 +02:00
aarshkshah1992
182da9d4ef fix error handling 2021-06-04 21:41:09 +02:00
aarshkshah1992
8733cea902 fix success handling in retreival 2021-06-04 21:41:09 +02:00
hannahhoward
93a2530803 fix(cli): make failed retrievals show by default 2021-06-01 16:02:35 -07:00
hannahhoward
3fbe2b320d feat(v0api): add list-retrievals to v0 2021-05-27 15:00:31 -07:00
hannahhoward
9e73e43272 fix(cli): add one more error state 2021-05-27 11:48:28 -07:00
hannahhoward
9c2467b17c fix(cli): patch for output given fil-markets IsTerminalError ahving a bug 2021-05-27 11:48:28 -07:00
hannahhoward
19b6dc8d1e feat(cli): add a list retrievals command
Currently, there is no way to inspect retrievals on a client. This adds said command, allow with
corresponding APIs
2021-05-27 11:48:25 -07:00
Peter Rabbitson
0019187a4f Forgotten pre-API zero-price check 2021-05-11 04:44:07 +02:00
Peter Rabbitson
e648b72d8d Merge remote-tracking branch 'origin/master' into feat/stateless-offline-dealflow 2021-05-06 15:57:10 +02:00
Łukasz Magiera
462a44902e
Merge pull request from filcloud/ping-ask
remove duplicate ask and calculate ping before lock
2021-04-29 21:12:37 +02:00
Yusef Napora
66e8517769 add "expected duration" label to inspect-deals output 2021-04-26 13:02:29 -04:00
Frank
b9cd364535 update ping lock 2021-04-20 10:26:02 +08:00
Frank
c915170b58 remove duplicate ask and calculate ping before lock 2021-04-20 10:20:30 +08:00
Łukasz Magiera
e76935147f Merge remote-tracking branch 'origin/master' into feat/v1-api 2021-04-13 14:24:22 +02:00
Aayush Rajasekaran
18d40e62f9 Add description to the client deal CLI command 2021-04-09 00:58:29 -04:00
Peter Rabbitson
b79be2a2c2 Merge remote-tracking branch 'origin/master' into feat/stateless-offline-dealflow 2021-04-05 16:06:44 +02:00
Peter Rabbitson
7fddbb528d Introduce stateless offline dealflow, bypassing the FSM/deallists
This is aproposal for an additional flag --manual-stateless-deal and a
corresponding API endpoint ClientStatelessDeal. This allows firing off
an offline-style deal against a miner without keeping further track of
it locally.

Not keeping any local state introduces the limitation of requiring free
storage deals, as there is nothing to tie the payment channel setup to.

Rationale/need for this type of flow is the case of incredibly large
sets of data nd deals, where the client and providers have prearranged
payment ahead of time, and the client has a separate-from-lotus database
of deal inventory. This way the client can use their lotus node merely
as a network gateway, without running into any limitations currently
present in both lotus as a whole and go-fil-markets in particular.

Specific context for this work is filecoin-discover, where the requirement
is to onboard ~ 12,000,000 individual deals against a pool of miners
with whom the client has prearranged a relationship.
2021-04-05 13:11:10 +02:00
Łukasz Magiera
1b32d7f52f cli: fix build with v1 api changes 2021-04-03 12:55:29 +02:00
Łukasz Magiera
4538550999 Merge remote-tracking branch 'origin/master' into feat/local-retrieval 2021-04-02 12:36:28 +02:00
Anton Evangelatov
df003d44d2 include Stages and DealStages only on ClientDealInfo, not on ClientListDeals 2021-04-01 17:08:01 +03:00
Anton Evangelatov
686c719bb1 move inspectDeal to cli package 2021-04-01 15:07:25 +03:00
Anton Evangelatov
967fbf52c6 resolve conflicts 2021-03-31 23:14:29 +03:00
Łukasz Magiera
ee04f9b71c Merge remote-tracking branch 'origin/master' into nonsense/add-cancel-retrieval-deal-cmd 2021-03-31 19:15:21 +02:00
Łukasz Magiera
f2ab316fb1 Local retrieval support 2021-03-30 21:32:06 +02:00
Anton Evangelatov
b661f51976 better description for inspect-deal 2021-03-30 17:08:47 +03:00
Anton Evangelatov
139ff4ae64 resolved conflicts 2021-03-30 16:15:42 +03:00
Łukasz Magiera
bf6ee88539
Merge pull request from filcloud/deal-duration
use deal duration from actors
2021-03-29 20:17:34 +02:00
Anton Evangelatov
8ba0bcb375 Merge branch 'master' into nonsense/cli-show-deals-backup 2021-03-29 18:19:28 +03:00
Raúl Kripalani
fd91c095c4 rename command to cancel-retrieval; rename args to follow Lotus style. 2021-03-26 16:37:46 +00:00
Dirk McCormick
47f7208f38 fix: rename command to retrieve-cancel 2021-03-26 10:19:19 +01:00
Dirk McCormick
b13f6a3209 fix: cancel retrieval deal - disallow negative deal ID 2021-03-26 09:51:07 +01:00
aarshkshah1992
36c3086376 changes as per review 2021-03-26 09:48:50 +05:30
Łukasz Magiera
fc087f3ac0 Take latency into account when making interactive deals 2021-03-24 22:42:17 +01:00
Anton Evangelatov
a202f9d19e add cancel-retrieval-deal cmd 2021-03-24 18:18:49 +02:00
Anton Evangelatov
9d6c77d93f remove interactive UX; add inspect-deal cmd 2021-03-18 14:53:03 +02:00
Dirk McCormick
84803f82ab feat: show deals CLI command 2021-03-04 16:56:49 +01:00
Rod Vagg
b9d7de595a list-asks: add --output-format & omit progress if !stdout 2021-03-02 13:58:41 +01:00
Dirk McCormick
a6e3856776 Revert "feat: deals - show data transfer %"
This reverts commit b6c9ddccff.
2021-02-16 12:32:45 +01:00
Dirk McCormick
b6c9ddccff feat: deals - show data transfer % 2021-02-10 09:56:23 +01:00
Łukasz Magiera
c5a2cac06a Fix client flag parsing in client balances cli 2021-01-08 23:03:52 +01:00
Łukasz Magiera
32dcfb4842
Merge pull request from filecoin-project/feat/wallet-market-info
Better CLI for wallet market withdraw and client info
2021-01-08 22:54:12 +01:00