Commit Graph

1094 Commits

Author SHA1 Message Date
Aarsh Shah
b096887b16 update deps 2021-08-23 14:45:17 +05:30
Aarsh Shah
b3e73e4dad update markets, dt and graphsync 2021-08-23 11:23:11 +05:30
Raúl Kripalani
d494778435 upgrade upstream dependencies. 2021-08-18 11:13:11 +01:00
Aarsh Shah
88f5bcdb63
Remove replace directive for multihash dep (#7113) 2021-08-18 11:00:29 +01:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671)
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
ZenGround0
bf41240ee1 Update deps to revert version representation 2021-08-11 09:49:23 -04:00
vyzo
8d51c5a971 update go-libp2p-pubsub to v0.5.4 2021-07-30 23:15:24 +03:00
Aarsh Shah
9549087ba9 update deps for logging 2021-07-29 20:04:50 +05:30
aarshkshah1992
973476930e more logging in data transfer and markets 2021-07-29 15:28:08 +05:30
hannahhoward
5a3b4ab34c feat(deps): update to branches with improved logging
update sub repos with improved logging around data transfer processing
2021-07-28 14:46:13 -07:00
Aayush Rajasekaran
b397955f24
Merge branch 'master' into deps/update-pubsub 2021-07-28 17:15:38 -04:00
Łukasz Magiera
3b617f4528 mod tidy 2021-07-28 17:59:37 +02:00
vyzo
66e793531d update go-libp2p-pubsub to v0.5.3 2021-07-28 15:59:16 +03:00
Jennifer Wang
d24b2acc63 update to go-fil-market v1.6.0 2021-07-27 04:34:35 -04:00
Peter Rabbitson
20c5dd5bdc Merge remote-tracking branch 'origin/master' into chore/upgrade_go-multihash 2021-07-26 21:26:10 +02:00
Jiaying Wang
98c8a2abe5
Merge branch 'master' into feat/allow_padding_redux 2021-07-26 13:59:33 -04:00
Peter Rabbitson
ed387b43cc Bump go-multihash, adjust test for supported version 2021-07-26 17:50:19 +02:00
Mike Greenberg
76a9f4241b feat: Graceful error when api impl is nil 2021-07-26 13:51:37 +02:00
whyrusleeping
78f94c4c6b This pulls in forgotten parts properly implementing PR#5988
( previous testing focused exclusively on offline dealflow .cars )

Allows a workflow of:

~$ dd if=/dev/urandom bs=1M count=1 | ~/go-ipfs/cmd/ipfs/ipfs add --pin=false
added QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2 QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ ~/go-ipfs/cmd/ipfs/ipfs dag export QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2 > test_mib.car

~$ lotus client import --car ~/test_mib.car
Import 2, Root QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ ~/go/bin/stream-commp -p $(( 256 * 1024 * 1024 )) < test_mib.car
CommP:    54e4e75ddc3fffa8fd33d3ededc06e564603ac0fe62543ec6463d51b553be40b
CommPCid: baga6ea4seaqfjzhhlxod775i7uz5h3pnybxfmrqdvqh6mjkd5rsghvi3ku56icy
Raw bytes:           1049073 bytes
Unpadded piece:    266338304 bytes
Padded piece:      268435456 bytes
CARv1 detected in stream:
Blocks:         5
Roots:          1
    1: QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ curl http://127.0.0.1:1234/rpc/v0 -X POST -H "Authorization: Bearer $(cat ~/.lotus/token)" -H "Content-Type: application/json" --data '
  { "jsonrpc": "2.0", "id":1, "method": "Filecoin.ClientStartDeal", "params": [
    {
      "Wallet":"t01004",
      "Miner":"t01005",
      "EpochPrice":"0",
      "MinBlocksDuration":518400,
      "Data": {
        "Root":{ "/":"QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2" },
        "PieceCid":{ "/":"baga6ea4seaqfjzhhlxod775i7uz5h3pnybxfmrqdvqh6mjkd5rsghvi3ku56icy" },
        "PieceSize": 266338304
      }
    }
  ] }
'

~$ ~/go/bin/stream-commp -p $(( 128 * 1024 * 1024 )) < test_mib.car
CommP:    ed904105399ed346f6b03844abc14710a1748854c2781824d6bd1100e63b1807
CommPCid: baga6ea4seaqo3ecbau4z5u2g62ydqrflyfdrbilurbkme6ayetll2eia4y5rqby
Raw bytes:           1049073 bytes
Unpadded piece:    133169152 bytes
Padded piece:      134217728 bytes
CARv1 detected in stream:
Blocks:         5
Roots:          1
    1: QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ curl http://127.0.0.1:1234/rpc/v0 -X POST -H "Authorization: Bearer $(cat ~/.lotus/token)" -H "Content-Type: application/json" --data '
  { "jsonrpc": "2.0", "id":1, "method": "Filecoin.ClientStatelessDeal", "params": [
    {
      "Wallet":"t01004",
      "Miner":"t01005",
      "EpochPrice":"0",
      "ProviderCollateral":"0",
      "MinBlocksDuration":518400,
      "Data": {
        "TransferType": "manual",
        "Root":{ "/":"QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2" },
        "PieceCid":{ "/":"baga6ea4seaqo3ecbau4z5u2g62ydqrflyfdrbilurbkme6ayetll2eia4y5rqby" },
        "PieceSize": 133169152
      }
    }
  ] }
'
{"jsonrpc":"2.0","result":{"/":"bafyreianhjvev3w6q5lteap3h7tkxbe2jaobwlsi7vzbcoobjpicg3foqi"},"id":1}

~$ lotus-miner storage-deals import-data bafyreianhjvev3w6q5lteap3h7tkxbe2jaobwlsi7vzbcoobjpicg3foqi ~/test_mib.car
2021-07-24 18:17:13 +02:00
Peter Rabbitson
8d873e3edc Tests for online and offline dealpadding
Also bump the times on several flaky tests that can not complete in time
on a typical laptop ( and fail half the time on CircleCI )
2021-07-24 17:46:04 +02:00
Raúl Kripalani
21f9dc8096 update graphsync to an actual release. 2021-07-22 23:47:47 +01:00
raulk
4ffbb03a82
Merge branch 'master' into raulk/merge-gs-fix 2021-07-22 23:37:17 +01:00
ZenGround0
3155609304 Finalize deps + fix after rebase 2021-07-22 09:49:56 -04:00
ZenGround0
56d71a190f Update versions to have more digits 2021-07-22 09:49:47 -04:00
ZenGround0
f49a8248f0 PriceListByVersion 2021-07-22 09:49:47 -04:00
Raúl Kripalani
c1b304b633 update graphsync. 2021-07-20 15:43:44 +01:00
Raúl Kripalani
9042429abd Merge branch 'master' into raulk/merge-gs-fix 2021-07-16 18:19:50 +01:00
Raúl Kripalani
aeb8e57eef use https://github.com/ipfs/go-graphsync/pull/176. 2021-07-16 18:17:34 +01:00
Raúl Kripalani
b2feb7e960 upgrade go-fil-markets, go-data-transfer; add logging. 2021-07-16 17:30:45 +01:00
vyzo
1d9dcef878 update go-libp2p-pubsub to v0.5.0 2021-07-15 15:07:14 +03:00
Łukasz Magiera
c37401a1a3
Merge pull request #6474 from filecoin-project/feat/splitstore-redux
Splitstore Enhanchements
2021-07-13 12:43:57 +02:00
Raúl Kripalani
71ddf73f44 check if merging ipfs/go-graphsync#177 fixes deal concurrency. 2021-07-10 10:48:18 +01:00
vyzo
ec586a852a remove bbolt dependency from go.mod 2021-07-07 16:41:11 +03:00
Peter Rabbitson
dcfbb1998f Stop outputing ANSI color on non-TTY
No more stuff like: https://circleci.com/gh/filecoin-project/lotus/189136
2021-07-07 11:06:20 +02:00
Peter Rabbitson
5c643d6141 Remove leftover from the great de-submoduling
This is no longer used anywhere in lotus
2021-07-04 09:40:47 +02:00
Raúl Kripalani
ec69ac0b2c downgrade libp2p/go-libp2p-yamux to v0.5.1. 2021-06-25 22:39:26 +01:00
Łukasz Magiera
ebd746cdec
Merge pull request #6406 from filecoin-project/feat/lotus-sim 2021-06-24 18:20:43 +02:00
Łukasz Magiera
c3480dc0e8 Merge remote-tracking branch 'origin/releases' into chore/merge-release 2021-06-24 09:39:11 +02:00
Jennifer Wang
85f2fcf2c6 pull actor v5.0.1 2021-06-23 18:51:20 -04:00
Jakub Sztandera
7dd58efb84 Add quantiles and histogram
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-06-18 15:44:36 -07:00
Łukasz Magiera
44de67cf7d
Merge pull request #6175 from filecoin-project/feat/dynamic-retreival-pricing
Dynamic Retrieval pricing
2021-06-17 10:25:48 +02:00
Łukasz Magiera
20c4fe9bba Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-15 16:50:23 +02:00
Łukasz Magiera
b00cbcaf5e Update to go-praamfetch with fslocks 2021-06-14 18:54:59 +02:00
aarshkshah1992
fed5afa704 merge master 2021-06-11 09:35:20 +05:30
Aayush Rajasekaran
3766980331 Merge branch 'release/v1.10.0' into asr/merge-release 2021-06-09 18:41:10 -04:00
Aayush Rajasekaran
29ebdc07c9 Update to specs-actors v5-rc-3 2021-06-09 18:07:46 -04:00
aarshkshah1992
16166504f4 merged master 2021-06-08 09:17:40 +05:30
Steven Allen
5ef5680b98 chore: update to go-libp2p v0.14.2
This should fix test flakes due to a race in the go-libp2p mock network.
2021-06-07 12:14:01 -07:00
Jakub Sztandera
f9acd07987
Update libp2p to 0.14.1
This does not resolve the CI issue but prepares us for it after https://github.com/libp2p/go-libp2p/pull/1116

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-06-07 18:51:48 +02:00
aarshkshah1992
0d88800eb9 use mockgen 2021-06-07 15:02:52 +05:30