Commit Graph

242 Commits

Author SHA1 Message Date
Geoff Stuart
dc4ba81b9c Make cli deal command get Block Delay specific to build 2022-06-22 19:35:03 -04:00
Geoff Stuart
5c884b5fd1
Merge pull request #8830 from filecoin-project/fix/error-client-deal
fix: ux: Check for client deal addrs on-chain
2022-06-17 11:42:04 -04:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Phi
faa3b185bf fix: ux: client deal addrs is on-chain check
Checks if the address used in `lotus client deal` is "initialized" on-chain before running the `uiLoop`. Resolves #8817
2022-06-09 16:20:51 +02:00
Aayush
8cca9b1970 Use new go-state-types accessors 2022-05-17 15:21:27 -04:00
Łukasz Magiera
6bef1aeb82 feat: cli: lotus client list-asks --protocols 2022-04-11 19:49:52 +02:00
Florian Ruen
196030bee2
fix: cli: Reset miner/ask lists in interactive deal 'miner' step (#8155)
* Update client.go

Bug fixed : if hitting return instead of filling with a valid value, the CLI crashed (line 796)
Change return err by continue to ask a valid value again

* Fix #8100

Bug fixed : if hitting return instead of filling with a valid value, the CLI crashed (line 796)
Change return err by continue to ask a valid value again

* lint reported errors on CircleCI

Commit to fix lint reported errors during tests on CircleCI
(remove blank lines x2)

* lint reported errors on CircleCI

Commit to fix lint reported errors during tests on CircleCI
(remove trailling whitespace)

* Fix #8095
Clear the list of miner addresses and successfull get-asked before asking for new ones
Previous behavior : if get-ask failed for miner(s), the faulty miner will be retried each time,
so you have to stop the command and start again to change this faulty miner (instead of removing from the list on the new attempt)

* Clear the list of miner addresses and successfull get-asked before asking for new ones
Previous behavior : if get-ask failed for miner(s), the faulty miner will be retried each time,
so you have to stop the command and start again to change this faulty miner (instead of removing from the list on the new attempt)
2022-03-25 17:05:19 -04:00
Rjan
c5dc898ed0 fix: cli: add ArgsUsage field to clientGetDealCmd
Adds [proposalCID] as ArgsUsage field in the 'lotus client get-deal' command. Fixes #8133
2022-03-03 15:40:16 +01:00
Łukasz Magiera
74e6260fbd
Merge pull request #7560 from EmadMF/emad22552-patch-1
add check manual-stateless-deal with interactive deal making
2022-03-02 13:16:14 +00:00
florian@eyesr.fr
1c39730531 Fix #8095
Clear the list of miner addresses and successfull get-asked before asking for new ones
Previous behavior : if get-ask failed for miner(s), the faulty miner will be retried each time,
so you have to stop the command and start again to change this faulty miner (instead of removing from the list on the new attempt)
2022-02-18 15:46:40 +00:00
emad22552
135186e33d
go fmt ./client.go 2022-01-02 19:08:22 +03:30
Dirk McCormick
0e53275d40 feat: retrieval-ask - if size param is zero use QueryOffer.Size 2021-12-20 16:03:03 +01:00
Dirk McCormick
bb56e97177 fix: rename query-retrieval-ask to retrieval-ask 2021-12-17 18:01:12 +01:00
dirkmc
0f22d51309
Update cli/client.go
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-12-17 17:54:32 +01:00
Dirk McCormick
b1734f84b3 feat: retrieval ask CLI command 2021-12-17 15:55:12 +01:00
Łukasz Magiera
53a48df77e retrieval: Don't use subcommands 2021-11-22 12:46:29 +01:00
Łukasz Magiera
a1d5b2a293 retrieval: wip improved retrieval commands 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b26906963b retrieval: Support multi-root export 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b868769ec8 more retrieval api work 2021-11-22 12:46:02 +01:00
Łukasz Magiera
89138bab4d Simplify retrieval APIs 2021-11-22 12:45:42 +01:00
emad22552
4eaf89666f
fix manual-stateless-deal with interactive
fix checking manual-stateless-deal with interactive deal making as --manual-stateless-deal can not be combined with interactive deal mode.
2021-10-26 14:00:41 +03:30
Peter Rabbitson
0444435589 Expose basic text-based datamodel selector on retrieval
Syntaxt of selection is located at
https://pkg.go.dev/github.com/ipld/go-ipld-selector-text-lite#SelectorSpecFromPath

Example use, assuming that:
  - The root of the deal is a plain dag-pb unixfs directory
  - The directory is not sharded
  - The user wants to retrieve the first entry in that directory

  lotus client retrieve --miner f0XXXXX --datamodel-path-selector 'Links/0/Hash' bafyROOTCID ~/output

For a much more elaborate example see the top of ./itests/deals_partial_retrieval_test.go
2021-09-10 09:44:11 +02:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671)
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Peter Rabbitson
6efd2e7335 Handle the --color flag via proper global state
Use `color.NoColor` as a global progressively modified as one descends
down command flags. Allows non-surprising operation when --color is
accepted at multiple layers, and even makes this behave predictably:

    lotus-miner --color actor control list --color=false
2021-07-13 12:22:29 +02:00
Peter Rabbitson
140a2e0200 Stabilize default text and make gen docsgen-cli 2021-07-08 10:44:13 +02:00
Peter Rabbitson
c7bb326c78 Enable color by default only if os.Stdout is a TTY 2021-07-07 18:12:24 +02:00
Aayush Rajasekaran
cb4d7cb9e7 Make query-ask CLI more graceful 2021-06-24 12:57:54 -04:00
Aayush Rajasekaran
4af59e0188
Apply suggestions from code review
Co-authored-by: Jennifer <42981373+jennijuju@users.noreply.github.com>
2021-06-22 19:23:24 -04:00
Aayush Rajasekaran
07fad6a201 Fix helptext 2021-06-22 16:02:06 -04:00
aarshkshah1992
35a466f4c4 add new line 2021-06-04 21:41:09 +02:00
aarshkshah1992
ed4748e8ac fix bug 2021-06-04 21:41:09 +02:00
aarshkshah1992
182da9d4ef fix error handling 2021-06-04 21:41:09 +02:00
aarshkshah1992
8733cea902 fix success handling in retreival 2021-06-04 21:41:09 +02:00
hannahhoward
93a2530803 fix(cli): make failed retrievals show by default 2021-06-01 16:02:35 -07:00
hannahhoward
3fbe2b320d feat(v0api): add list-retrievals to v0 2021-05-27 15:00:31 -07:00
hannahhoward
9e73e43272 fix(cli): add one more error state 2021-05-27 11:48:28 -07:00
hannahhoward
9c2467b17c fix(cli): patch for output given fil-markets IsTerminalError ahving a bug 2021-05-27 11:48:28 -07:00
hannahhoward
19b6dc8d1e feat(cli): add a list retrievals command
Currently, there is no way to inspect retrievals on a client. This adds said command, allow with
corresponding APIs
2021-05-27 11:48:25 -07:00
Peter Rabbitson
0019187a4f Forgotten pre-API zero-price check 2021-05-11 04:44:07 +02:00
Peter Rabbitson
e648b72d8d Merge remote-tracking branch 'origin/master' into feat/stateless-offline-dealflow 2021-05-06 15:57:10 +02:00
Łukasz Magiera
462a44902e
Merge pull request #5968 from filcloud/ping-ask
remove duplicate ask and calculate ping before lock
2021-04-29 21:12:37 +02:00
Yusef Napora
66e8517769 add "expected duration" label to inspect-deals output 2021-04-26 13:02:29 -04:00
Frank
b9cd364535 update ping lock 2021-04-20 10:26:02 +08:00
Frank
c915170b58 remove duplicate ask and calculate ping before lock 2021-04-20 10:20:30 +08:00
Łukasz Magiera
e76935147f Merge remote-tracking branch 'origin/master' into feat/v1-api 2021-04-13 14:24:22 +02:00
Aayush Rajasekaran
18d40e62f9 Add description to the client deal CLI command 2021-04-09 00:58:29 -04:00
Peter Rabbitson
b79be2a2c2 Merge remote-tracking branch 'origin/master' into feat/stateless-offline-dealflow 2021-04-05 16:06:44 +02:00
Peter Rabbitson
7fddbb528d Introduce stateless offline dealflow, bypassing the FSM/deallists
This is aproposal for an additional flag --manual-stateless-deal and a
corresponding API endpoint ClientStatelessDeal. This allows firing off
an offline-style deal against a miner without keeping further track of
it locally.

Not keeping any local state introduces the limitation of requiring free
storage deals, as there is nothing to tie the payment channel setup to.

Rationale/need for this type of flow is the case of incredibly large
sets of data nd deals, where the client and providers have prearranged
payment ahead of time, and the client has a separate-from-lotus database
of deal inventory. This way the client can use their lotus node merely
as a network gateway, without running into any limitations currently
present in both lotus as a whole and go-fil-markets in particular.

Specific context for this work is filecoin-discover, where the requirement
is to onboard ~ 12,000,000 individual deals against a pool of miners
with whom the client has prearranged a relationship.
2021-04-05 13:11:10 +02:00
Łukasz Magiera
1b32d7f52f cli: fix build with v1 api changes 2021-04-03 12:55:29 +02:00
Łukasz Magiera
4538550999 Merge remote-tracking branch 'origin/master' into feat/local-retrieval 2021-04-02 12:36:28 +02:00