Commit Graph

81 Commits

Author SHA1 Message Date
Aayush Rajasekaran
80d5e52923 Merge branch 'master' into next 2021-12-13 13:24:28 -05:00
Łukasz Magiera
763659b8a3 retrieval: Update lotus-soup 2021-11-22 12:46:29 +01:00
Aayush Rajasekaran
1449644c34 Integrate v7 actors 2021-11-17 15:55:14 -05:00
Travis Person
2d4f5958e2 Add caches to lotus-stats and splitcode 2021-11-01 09:05:14 +00:00
Jennifer Wang
38e01cba83 Merge branch 'release/v1.12.0' into jen/12tomaster 2021-10-04 02:38:00 -04:00
Aayush Rajasekaran
ec14259b5a Prep the butterfly network for v14 2021-10-02 17:27:39 -04:00
Aayush Rajasekaran
50ce475701 Upgrade to actors v6-rc1 2021-10-01 17:57:32 -04:00
Aayush Rajasekaran
a335cb3767 Update to latest actors 2021-10-01 17:11:05 -04:00
Aayush Rajasekaran
2bafdf7271 Modify tooling to support new WithdrawBalance return 2021-09-29 13:01:04 -04:00
Aarsh Shah
7694f895c2 update lotus soup dep 2021-09-23 09:15:47 +04:00
Aarsh Shah
9a07c2fd38 revert go car update 2021-09-22 12:55:22 +04:00
Aayush Rajasekaran
7b4c657e90 Add v6 actors 2021-09-21 14:23:14 -04:00
Aarsh Shah
7863a269ad fix lotus-soup 2021-09-21 15:31:16 +04:00
Aarsh Shah
89ac8b09b5 update libp2p 2021-09-21 15:10:04 +04:00
Steven Allen
0789a2927a fix: make lotus soup use the correct dependencies
1. Add a replace directive for lotus.
2. Make sure we're not depending on commits that don't exist.
2021-08-30 08:57:24 -07:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ()
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Anton Evangelatov
bb466ede66 fix cast on role_miner in testground lotus-soup plan 2021-07-07 16:04:01 +02:00
Anton Evangelatov
1fe08f5973 refactor testkit in lotus-soup to include Subsystems when instantiating miner 2021-07-07 14:21:28 +02:00
Anton Evangelatov
6b014f57e5 pass Subsystems to StorageMiner option; add enableLibp2p bool in Settings 2021-07-07 13:56:37 +02:00
Anton Evangelatov
8dbae00847 fix lotus-soup 2021-07-02 12:45:56 +02:00
Anton Evangelatov
4217ec5308 make auth on remote rpc server optional 2021-06-23 13:27:03 +02:00
Łukasz Magiera
b37a66c7c9 Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-21 17:03:03 +02:00
Aayush Rajasekaran
514107a75b Fix soup 2021-06-16 18:04:45 -04:00
Aayush Rajasekaran
35f76f58df
Merge pull request from filecoin-project/feat/better_mining_projection_and_accuracy
Adjust various CLI display ratios to arbitrary precision
2021-06-16 17:48:23 -04:00
Aayush Rajasekaran
520a0091f5 Merge branch 'releases' into release/v1.10.0 2021-06-15 17:40:36 -04:00
Peter Rabbitson
4efc3f8db0 Copy latest version to lotus-soup 2021-06-15 22:00:17 +02:00
Anton Evangelatov
c7c029ea91 testplans: lotus-soup: new images with filecoin-ffi ; use default WPoStChallengeWindow 2021-06-07 11:28:24 +02:00
Aayush Rajasekaran
4a321c6da2 Fix nerpa build 2021-06-02 19:19:32 -04:00
Aayush Rajasekaran
cd4505dd63 Update to specs-actors v5-rc-2 2021-06-02 19:11:47 -04:00
Łukasz Magiera
08b7ab90c1 mod tidy, fix testground build 2021-06-02 15:29:54 +02:00
Łukasz Magiera
6d9779ecae Merge remote-tracking branch 'origin/master' into feat/nv13-1.11 2021-06-02 15:07:13 +02:00
Łukasz Magiera
6ead83b605 Go 1.16 in Dockerfile 2021-06-02 14:42:48 +02:00
Łukasz Magiera
cba5c34aef Fix lotus-soup build 2021-06-02 12:38:07 +02:00
Dirk McCormick
133459756b fix: update lotus-soup 2021-06-01 13:44:42 -06:00
Dirk McCormick
6e92c43dd5 feat: add testground versions composition 2021-06-01 10:40:55 -06:00
Łukasz Magiera
1e4456138e Merge master into feat/nv13 2021-05-27 12:28:20 +02:00
Peter Rabbitson
c2e5a837e6 Adjust various CLI display ratios to arbitrary precision
Originally the deviations from using float64 were insignificant, but at
exabyte scale they start to show up. Cleanup all displays, and clarify
the expectation text, adding an extra 99.9% probability calculator to
`lotus-miner info`
2021-05-25 14:09:01 +02:00
Aayush Rajasekaran
b5da2655dc Introduce v5 actors 2021-05-10 19:44:28 -04:00
Dirk McCormick
c17a340f2e feat: deals end to end test with restarts 2021-05-06 09:49:44 +02:00
Łukasz Magiera
492b026031
Merge pull request from filecoin-project/testplans-go-mod
go mod tidy for lotus-soup testplans
2021-04-29 18:31:32 +02:00
Anton Evangelatov
5769a45380 use 1 miner for payment channel tests 2021-04-28 16:58:03 +02:00
Anton Evangelatov
bc8d6a1d87 go mod tidy for lotus-soup testplans 2021-04-28 14:36:20 +02:00
Anton Evangelatov
0304bebf20 remove deals concurrency 2021-04-28 13:38:18 +02:00
Anton Evangelatov
a9e4a0d22b upgrade docker images 2021-04-27 16:28:57 +02:00
Łukasz Magiera
a5921a9bf9 fix lotus-soup build 2021-04-06 13:30:36 +02:00
Łukasz Magiera
c9866da688 Merge remote-tracking branch 'origin/master' into feat/v1-api 2021-04-02 13:15:33 +02:00
Anton Evangelatov
1626dffd4c remove overwrite for go-data-transfer in testplans 2021-03-31 23:17:16 +03:00
Łukasz Magiera
9223a8e5ff Fix lotus-soup build 2021-03-25 15:29:20 +01:00
Anton Evangelatov
b4d4766615 use go-data-transfer from personal fork 2021-03-23 14:30:26 +02:00
Raúl Kripalani
7f0f7d0b36 Merge branch 'master' into refactor/lib/blockstore 2021-02-28 19:55:23 +00:00