Commit Graph

99 Commits

Author SHA1 Message Date
Aayush Rajasekaran
316836f43b deps: update go-paramfetch to v0.0.4 2022-02-08 12:00:55 -05:00
Aayush Rajasekaran
ad39d8707d build: add empty sector update params 2022-02-08 10:41:18 -05:00
Aayush Rajasekaran
544cfa63ab cache added cids 2022-01-11 17:22:36 -05:00
Aayush Rajasekaran
19bd9cf945 Update to actors v7.0.0-rc1 2022-01-11 17:20:08 -05:00
Aayush Rajasekaran
8aabe1b488 Fast migration for v15 2022-01-11 17:19:38 -05:00
Aayush Rajasekaran
207d33eaba
Merge pull request #7810 from filecoin-project/feat/snap-deals
Snap Deals Lotus Integration: FSM Posting and integration test
2022-01-11 12:46:55 -05:00
Aayush Rajasekaran
d645c5fbab Remove unnecessary params from VerifyWinningPost 2022-01-11 12:06:39 -05:00
Aayush Rajasekaran
9614f6db8d Update go-paramfetch 2022-01-10 19:03:26 -05:00
Aayush Rajasekaran
5ec5ebac31 Fix circsuypply calc around null blocks 2022-01-05 14:17:40 -05:00
Aayush Rajasekaran
aa1b770974 Rand: Refactor so that versioning lives in StateRand 2021-12-17 13:58:53 -05:00
Łukasz Magiera
bc384c01e3 Merge remote-tracking branch 'origin/master' into deps/update-ctx-dsbs 2021-12-17 13:01:22 +01:00
Aayush Rajasekaran
0e2278cc76 Merge branch 'master' into deps/update-ctx-dsbs 2021-12-14 18:08:29 -05:00
Aayush Rajasekaran
7fb590208c Deps: Update lotus-soup to point to tagged releases 2021-12-14 18:05:52 -05:00
vyzo
0eec00055f fix lotus-soup build 2021-12-14 18:23:43 +02:00
vyzo
e8df325796 update lotus-soup deps 2021-12-14 18:23:33 +02:00
Aayush Rajasekaran
80d5e52923 Merge branch 'master' into next 2021-12-13 13:24:28 -05:00
Łukasz Magiera
763659b8a3 retrieval: Update lotus-soup 2021-11-22 12:46:29 +01:00
Aayush Rajasekaran
1449644c34 Integrate v7 actors 2021-11-17 15:55:14 -05:00
Travis Person
2d4f5958e2 Add caches to lotus-stats and splitcode 2021-11-01 09:05:14 +00:00
Peter Rabbitson
778483166f Remove obsolete GS testplan - it now lives in go-graphsync 2021-10-07 12:23:43 +02:00
Jennifer Wang
38e01cba83 Merge branch 'release/v1.12.0' into jen/12tomaster 2021-10-04 02:38:00 -04:00
Aayush Rajasekaran
ec14259b5a Prep the butterfly network for v14 2021-10-02 17:27:39 -04:00
Aayush Rajasekaran
50ce475701 Upgrade to actors v6-rc1 2021-10-01 17:57:32 -04:00
Aayush Rajasekaran
a335cb3767 Update to latest actors 2021-10-01 17:11:05 -04:00
Aayush Rajasekaran
2bafdf7271 Modify tooling to support new WithdrawBalance return 2021-09-29 13:01:04 -04:00
Aarsh Shah
7694f895c2 update lotus soup dep 2021-09-23 09:15:47 +04:00
Aarsh Shah
9a07c2fd38 revert go car update 2021-09-22 12:55:22 +04:00
Aayush Rajasekaran
7b4c657e90 Add v6 actors 2021-09-21 14:23:14 -04:00
Aarsh Shah
7863a269ad fix lotus-soup 2021-09-21 15:31:16 +04:00
Aarsh Shah
89ac8b09b5 update libp2p 2021-09-21 15:10:04 +04:00
Steven Allen
0789a2927a fix: make lotus soup use the correct dependencies
1. Add a replace directive for lotus.
2. Make sure we're not depending on commits that don't exist.
2021-08-30 08:57:24 -07:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671)
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Anton Evangelatov
bb466ede66 fix cast on role_miner in testground lotus-soup plan 2021-07-07 16:04:01 +02:00
Anton Evangelatov
1fe08f5973 refactor testkit in lotus-soup to include Subsystems when instantiating miner 2021-07-07 14:21:28 +02:00
Anton Evangelatov
6b014f57e5 pass Subsystems to StorageMiner option; add enableLibp2p bool in Settings 2021-07-07 13:56:37 +02:00
Anton Evangelatov
8dbae00847 fix lotus-soup 2021-07-02 12:45:56 +02:00
Anton Evangelatov
4217ec5308 make auth on remote rpc server optional 2021-06-23 13:27:03 +02:00
Łukasz Magiera
b37a66c7c9 Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-21 17:03:03 +02:00
Aayush Rajasekaran
514107a75b Fix soup 2021-06-16 18:04:45 -04:00
Aayush Rajasekaran
35f76f58df
Merge pull request #6309 from filecoin-project/feat/better_mining_projection_and_accuracy
Adjust various CLI display ratios to arbitrary precision
2021-06-16 17:48:23 -04:00
Aayush Rajasekaran
520a0091f5 Merge branch 'releases' into release/v1.10.0 2021-06-15 17:40:36 -04:00
Peter Rabbitson
4efc3f8db0 Copy latest version to lotus-soup 2021-06-15 22:00:17 +02:00
Anton Evangelatov
c7c029ea91 testplans: lotus-soup: new images with filecoin-ffi ; use default WPoStChallengeWindow 2021-06-07 11:28:24 +02:00
Aayush Rajasekaran
4a321c6da2 Fix nerpa build 2021-06-02 19:19:32 -04:00
Aayush Rajasekaran
cd4505dd63 Update to specs-actors v5-rc-2 2021-06-02 19:11:47 -04:00
Łukasz Magiera
08b7ab90c1 mod tidy, fix testground build 2021-06-02 15:29:54 +02:00
Łukasz Magiera
6d9779ecae Merge remote-tracking branch 'origin/master' into feat/nv13-1.11 2021-06-02 15:07:13 +02:00
Łukasz Magiera
6ead83b605 Go 1.16 in Dockerfile 2021-06-02 14:42:48 +02:00
Łukasz Magiera
cba5c34aef Fix lotus-soup build 2021-06-02 12:38:07 +02:00
Dirk McCormick
133459756b fix: update lotus-soup 2021-06-01 13:44:42 -06:00