Commit Graph

172 Commits

Author SHA1 Message Date
Aarsh Shah
7694f895c2 update lotus soup dep 2021-09-23 09:15:47 +04:00
Aarsh Shah
9a07c2fd38 revert go car update 2021-09-22 12:55:22 +04:00
Aayush Rajasekaran
7b4c657e90 Add v6 actors 2021-09-21 14:23:14 -04:00
Aarsh Shah
7863a269ad fix lotus-soup 2021-09-21 15:31:16 +04:00
Aarsh Shah
89ac8b09b5 update libp2p 2021-09-21 15:10:04 +04:00
Steven Allen
0789a2927a fix: make lotus soup use the correct dependencies
1. Add a replace directive for lotus.
2. Make sure we're not depending on commits that don't exist.
2021-08-30 08:57:24 -07:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671)
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Anton Evangelatov
bb466ede66 fix cast on role_miner in testground lotus-soup plan 2021-07-07 16:04:01 +02:00
Anton Evangelatov
1fe08f5973 refactor testkit in lotus-soup to include Subsystems when instantiating miner 2021-07-07 14:21:28 +02:00
Anton Evangelatov
6b014f57e5 pass Subsystems to StorageMiner option; add enableLibp2p bool in Settings 2021-07-07 13:56:37 +02:00
Anton Evangelatov
8dbae00847 fix lotus-soup 2021-07-02 12:45:56 +02:00
Anton Evangelatov
4217ec5308 make auth on remote rpc server optional 2021-06-23 13:27:03 +02:00
Łukasz Magiera
b37a66c7c9 Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-21 17:03:03 +02:00
Aayush Rajasekaran
514107a75b Fix soup 2021-06-16 18:04:45 -04:00
Aayush Rajasekaran
35f76f58df
Merge pull request #6309 from filecoin-project/feat/better_mining_projection_and_accuracy
Adjust various CLI display ratios to arbitrary precision
2021-06-16 17:48:23 -04:00
Aayush Rajasekaran
520a0091f5 Merge branch 'releases' into release/v1.10.0 2021-06-15 17:40:36 -04:00
Peter Rabbitson
4efc3f8db0 Copy latest version to lotus-soup 2021-06-15 22:00:17 +02:00
Anton Evangelatov
c7c029ea91 testplans: lotus-soup: new images with filecoin-ffi ; use default WPoStChallengeWindow 2021-06-07 11:28:24 +02:00
Aayush Rajasekaran
4a321c6da2 Fix nerpa build 2021-06-02 19:19:32 -04:00
Aayush Rajasekaran
cd4505dd63 Update to specs-actors v5-rc-2 2021-06-02 19:11:47 -04:00
Łukasz Magiera
08b7ab90c1 mod tidy, fix testground build 2021-06-02 15:29:54 +02:00
Łukasz Magiera
6d9779ecae Merge remote-tracking branch 'origin/master' into feat/nv13-1.11 2021-06-02 15:07:13 +02:00
Łukasz Magiera
6ead83b605 Go 1.16 in Dockerfile 2021-06-02 14:42:48 +02:00
Łukasz Magiera
cba5c34aef Fix lotus-soup build 2021-06-02 12:38:07 +02:00
Dirk McCormick
133459756b fix: update lotus-soup 2021-06-01 13:44:42 -06:00
Dirk McCormick
6e92c43dd5 feat: add testground versions composition 2021-06-01 10:40:55 -06:00
Łukasz Magiera
1e4456138e Merge master into feat/nv13 2021-05-27 12:28:20 +02:00
Peter Rabbitson
c2e5a837e6 Adjust various CLI display ratios to arbitrary precision
Originally the deviations from using float64 were insignificant, but at
exabyte scale they start to show up. Cleanup all displays, and clarify
the expectation text, adding an extra 99.9% probability calculator to
`lotus-miner info`
2021-05-25 14:09:01 +02:00
Aayush Rajasekaran
b5da2655dc Introduce v5 actors 2021-05-10 19:44:28 -04:00
Dirk McCormick
c17a340f2e feat: deals end to end test with restarts 2021-05-06 09:49:44 +02:00
Łukasz Magiera
492b026031
Merge pull request #6124 from filecoin-project/testplans-go-mod
go mod tidy for lotus-soup testplans
2021-04-29 18:31:32 +02:00
Anton Evangelatov
5769a45380 use 1 miner for payment channel tests 2021-04-28 16:58:03 +02:00
Anton Evangelatov
bc8d6a1d87 go mod tidy for lotus-soup testplans 2021-04-28 14:36:20 +02:00
Anton Evangelatov
0304bebf20 remove deals concurrency 2021-04-28 13:38:18 +02:00
Anton Evangelatov
a9e4a0d22b upgrade docker images 2021-04-27 16:28:57 +02:00
Łukasz Magiera
a5921a9bf9 fix lotus-soup build 2021-04-06 13:30:36 +02:00
Łukasz Magiera
c9866da688 Merge remote-tracking branch 'origin/master' into feat/v1-api 2021-04-02 13:15:33 +02:00
Anton Evangelatov
1626dffd4c remove overwrite for go-data-transfer in testplans 2021-03-31 23:17:16 +03:00
Łukasz Magiera
9223a8e5ff Fix lotus-soup build 2021-03-25 15:29:20 +01:00
Anton Evangelatov
b4d4766615 use go-data-transfer from personal fork 2021-03-23 14:30:26 +02:00
Raúl Kripalani
7f0f7d0b36 Merge branch 'master' into refactor/lib/blockstore 2021-02-28 19:55:23 +00:00
Anton Evangelatov
59f814028f add storageadapter.PublishMsgConfig to miner in testkit 2021-02-09 19:12:59 +01:00
Anton Evangelatov
891971edec circleci: add task to get latest master for lotus 2021-02-08 17:48:37 +01:00
Anton Evangelatov
cc91eab0fd fix lr.Datastore interface to include context 2021-02-08 17:38:51 +01:00
Anton Evangelatov
764a08b980 fix testplans build 2021-02-08 16:32:30 +01:00
Łukasz Magiera
0afe7325f6
Merge pull request #4995 from filecoin-project/lock-lotus-version-in-testplan
rename oni/lotus-soup imports to testplans/lotus-soup and go get lotus@master on trigger
2021-02-08 15:39:15 +01:00
Łukasz Magiera
a541a2500a
Merge pull request #5282 from filecoin-project/chore/snake_context_through_blockstore_init
Snake a context through the Chain-blockstore creation
2021-01-30 11:55:17 +01:00
Łukasz Magiera
bb4e3e49ea Merge remote-tracking branch 'origin/master' into feat/no-timing-compute-state 2021-01-30 10:57:32 +01:00
Raúl Kripalani
af429dba0f Merge branch 'master' into chore/snake_context_through_blockstore_init 2021-01-29 21:21:46 +00:00
Raúl Kripalani
e02fdf5064 Merge branch 'master' into refactor/lib/blockstore 2021-01-29 21:21:31 +00:00
Anton Evangelatov
52a610650a merge master; resolve conflicts 2021-01-29 22:20:10 +01:00
Anton Evangelatov
1cca0ff9e2 upgrade docker; upgrade filecoin-ffi; bump up image versions 2021-01-29 22:02:45 +01:00
Anton Evangelatov
8ed726f9e3 bump up go to 1.15 2021-01-29 21:19:27 +01:00
Raúl Kripalani
b0cbc932bd consolidate all blockstores in blockstore package. 2021-01-29 20:01:00 +00:00
Anton Evangelatov
c09571984a remove blst replace; update go.mod to use lotus@master 2021-01-29 19:53:34 +01:00
Łukasz Magiera
389e6eb70b state compute-state ---html --no-timing flag 2021-01-28 19:37:32 +01:00
Raúl Kripalani
a1da1dab85 add context to LockedRepo#Datastore(). 2021-01-26 11:01:43 +00:00
Raúl Kripalani
dbbd1d7a45 fix stale imports in testplans/lotus-soup. 2021-01-26 10:53:27 +00:00
Anton Evangelatov
655277171b baseline 1 miner 1 cluster composition for local:docker 2020-12-17 16:19:41 +01:00
Anton Evangelatov
4d69947eed resolve conflicts with master 2020-12-11 10:01:14 +01:00
Anton Evangelatov
1cfd9555d5 reduce sectors ; less verbose logging for some subsystems 2020-12-07 15:11:41 +01:00
Anton Evangelatov
064b036000 update debug image to use 8MiB sectors 2020-12-07 14:14:21 +01:00
Anton Evangelatov
2829245a46 baseline k8s configs 2020-12-07 11:48:22 +01:00
Anton Evangelatov
b1dcf5fead 8MB sectors ; 5MB deal 2020-12-07 11:48:22 +01:00
Anton Evangelatov
d0599ff1e3 dummy delay for everyone but first miner 2020-12-07 11:48:22 +01:00
Anton Evangelatov
a53fd82379 add more time to miner to bootstrap 2020-12-07 11:48:22 +01:00
Anton Evangelatov
86af05ba6f update base image with latest filecoin-ffi 2020-12-07 11:48:22 +01:00
Anton Evangelatov
c543a10ce7 rename oni/lotus-soup to testplans/lotus-soup and lock lotus version 2020-12-07 11:48:22 +01:00
Aayush Rajasekaran
370817eb60 Allow miners to filter (un)verified deals 2020-12-02 01:32:34 -05:00
Anton Evangelatov
52da5acc96 go mod replace lotus with ../../ 2020-11-23 20:43:18 +01:00
Anton Evangelatov
bf500711b3 update Dockerfiles; integrate lotus-soup and graphsync in Lotus CI; update go.mod and references 2020-11-06 21:04:11 +01:00
Anton Evangelatov
1f678c17d3 merge filecoin-project/oni subtree in testplans/ 2020-11-06 21:02:06 +01:00