Commit Graph

45 Commits

Author SHA1 Message Date
Darko Brdareski
a64f2421d2 Annotate 'incoming' subsystem 2021-12-14 11:33:33 +01:00
Darko Brdareski
0addca1070 Fix bad annotations 2021-12-13 13:41:04 +01:00
Darko Brdareski
95f86f9de0 Annotate feature syncer 2021-12-10 11:33:29 +01:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ()
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Peter Rabbitson
8d873e3edc Tests for online and offline dealpadding
Also bump the times on several flaky tests that can not complete in time
on a typical laptop ( and fail half the time on CircleCI )
2021-07-24 17:46:04 +02:00
Anton Evangelatov
4be0a7a215 resolve conflicts with master 2021-07-05 13:13:32 +02:00
Łukasz Magiera
8a94ab676e storage: Fix FinalizeSector with sectors in stoage paths 2021-07-02 19:54:45 +02:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
Anton Evangelatov
9ec7cd7239 enable libp2p options 2021-06-30 12:55:43 +02:00
Raúl Kripalani
9b3188d110 split deals tests. 2021-06-23 18:13:29 +01:00
Anton Evangelatov
a8d5ae5213 enable more tests 2021-06-23 13:39:03 +02:00
Anton Evangelatov
21b51328f9 adding TestDealWithMarketAndMinerNode 2021-06-23 12:44:52 +02:00
Anton Evangelatov
99646d209e fix TestFirstDealEnablesMining 2021-06-23 11:56:09 +02:00
Anton Evangelatov
7fb93d4585 refactor kit.NewDealHarness to include main miner and market node 2021-06-23 11:24:55 +02:00
Anton Evangelatov
9567807144 Merge branch 'raulk/itests-refactor-kit' into nonsense/split-market-miner-processes 2021-06-22 14:39:28 +02:00
Dirk McCormick
da789939b1 fix: bump blocktime of TestQuotePriceForUnsealedRetrieval to 1 second 2021-06-22 14:33:44 +02:00
Anton Evangelatov
00618672f5 policy.SetPreCommitChallengeDelay to reduce WaitSeed stage time 2021-06-22 13:08:36 +02:00
Anton Evangelatov
5e10d53ca8 wip 2021-06-22 12:16:21 +02:00
Anton Evangelatov
6720463799 resolve merge conflicts 2021-06-22 11:28:23 +02:00
Raúl Kripalani
502e104e6a typo. 2021-06-21 21:35:39 +01:00
Raúl Kripalani
e438ef99f8 fix merge error in window post dispute tests. 2021-06-21 21:25:55 +01:00
Raúl Kripalani
4fcd0b7aca disable resource filtering on scheduler. 2021-06-21 21:20:45 +01:00
Raúl Kripalani
c22e10c4a4 use mock proofs in TestQuotePriceForUnsealedRetrieval. 2021-06-21 18:29:25 +01:00
Raúl Kripalani
70929a99e6 speed up test. 2021-06-21 18:24:39 +01:00
Raúl Kripalani
0879ac496f uncomment lines in TestDealCyclesConcurrent. 2021-06-21 18:21:10 +01:00
Raúl Kripalani
83ee345b1c Merge branch 'master' into raulk/itests-refactor-kit 2021-06-21 18:19:26 +01:00
Raúl Kripalani
8a418bf982 rename {kit2=>kit}. 2021-06-18 19:45:29 +01:00
Anton Evangelatov
577b730415 wip 2021-06-18 18:42:34 +02:00
Raúl Kripalani
9436be5ff0 introduce TestFullNode#WaitTillChain(predicate). 2021-06-17 22:02:20 +01:00
aarshkshah1992
3d086dfb43 changes as per review 2021-06-14 09:40:34 +05:30
Raúl Kripalani
f1ec0d6094 fix lint. 2021-06-14 00:10:37 +01:00
Raúl Kripalani
183814a826 finish migrating deals test. 2021-06-13 23:43:22 +01:00
Raúl Kripalani
e84b8ab3a0 move new kit into kit2, re-enable unmigrated tests against kit1. 2021-06-11 18:26:25 +01:00
aarshkshah1992
fed5afa704 merge master 2021-06-11 09:35:20 +05:30
Raúl Kripalani
8b037e2da3 deals tests: migrate deals cycles tests and add coverage. 2021-06-10 18:25:02 +01:00
Raúl Kripalani
dcd6fc239b deals tests: migrate TestOfflineDealFlow. 2021-06-10 15:54:16 +01:00
Raúl Kripalani
4f2aaa54d2 deals tests: refactor/simplify TestDealMining; now TestFirstDealEnablesMining. 2021-06-10 14:04:39 +01:00
Raúl Kripalani
329970934a deals tests: begin migration. 2021-06-10 13:25:36 +01:00
Raúl Kripalani
a274fcc533 Merge branch 'master' into raulk/itests 2021-06-07 23:44:22 +01:00
Raúl Kripalani
db1a61852d Merge branch 'master' into raulk/itests 2021-06-07 23:17:44 +01:00
Raúl Kripalani
2bcedcf55f initial version of the new itest kit.
Still need to migrate all integration tests, add godocs,
and probably zap bugs.
2021-05-26 00:04:13 +01:00
Raúl Kripalani
25daa0c8e4 itests: create deal harness. 2021-05-20 16:12:42 +01:00
Raúl Kripalani
1902c4c687 itests: rename Builder methods. 2021-05-20 12:17:41 +01:00
Raúl Kripalani
0cfef0fdbb wip extract test kit. 2021-05-19 17:30:43 +01:00
Raúl Kripalani
41d0818347 wip pull all integration tests under itests/ 2021-05-19 17:30:43 +01:00