Commit Graph

233 Commits

Author SHA1 Message Date
Dirk McCormick
7b7a5b0b21 revert: changes to OnDealExpiredOrChanged in 2021-08-30 10:42:41 +02:00
dirkmc
77a19774cf fix events API timeout handling for nil blocks () 2021-08-27 09:05:00 +02:00
Raúl Kripalani
342134f853 EnableLargeSectors: restore previous supported proof types. 2021-08-20 18:07:25 +01:00
Raúl Kripalani
0030e208a0 avoid reordering imports. 2021-08-20 15:59:19 +01:00
Raúl Kripalani
2dde022a00 itests: support larger sector sizes; add large deal test. 2021-08-20 15:53:24 +01:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ()
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
ZenGround0
ce58b119bf
Merge pull request from filecoin-project/spike/restrict-api-versions
Strict major minor version checking on v0 and v1 apis
2021-08-12 14:34:31 -04:00
ZenGround0
c8720fef02 Fix tests 2021-08-12 14:01:24 -04:00
Steven Allen
dcff06bd91 test: genesis actors version
Properly handle genesis in actors tests. Fast-forward upgrading to
actors v13 doesn't work because there needs to be at least a day between
v0 genesis and v13 (due to differences in miner cron).
2021-08-11 13:30:15 -07:00
Peter Rabbitson
2edf7fd25b Rewire itest's StartDeal to take the full API struct
This allows one to use the harness for much more versatile deal conditions
2021-07-24 04:17:42 +02:00
raulk
4ffbb03a82
Merge branch 'master' into raulk/merge-gs-fix 2021-07-22 23:37:17 +01:00
Raúl Kripalani
9042429abd Merge branch 'master' into raulk/merge-gs-fix 2021-07-16 18:19:50 +01:00
Raúl Kripalani
b2feb7e960 upgrade go-fil-markets, go-data-transfer; add logging. 2021-07-16 17:30:45 +01:00
Łukasz Magiera
49e26cce7d api: Separate the Net interface from Common 2021-07-15 11:41:30 +02:00
Anton Evangelatov
4bc9fa04ba rename cfg.Subsystems.EnableStorageMarket to EnableMarkets 2021-07-12 12:12:29 +02:00
Anton Evangelatov
d89ddb9315 resolve conflicts 2021-07-12 11:34:37 +02:00
Łukasz Magiera
9bdea91a70 itests: Some kit cleanup 2021-07-07 21:40:44 +02:00
Łukasz Magiera
2dc27d6ab4 itests: Fix deal provider collateral flakiness 2021-07-07 19:41:46 +02:00
Łukasz Magiera
ee97aa0a8a dealpublisher: Use address selector 2021-07-07 19:12:03 +02:00
Anton Evangelatov
6b014f57e5 pass Subsystems to StorageMiner option; add enableLibp2p bool in Settings 2021-07-07 13:56:37 +02:00
Anton Evangelatov
caa366bc6a variadic WithSubsystems 2021-07-07 13:35:40 +02:00
Anton Evangelatov
2a7af38538 fix testServ to Start with correct listener 2021-07-06 17:57:08 +02:00
Anton Evangelatov
7a78527fde remove not used sa; cleanup 2021-07-06 17:46:21 +02:00
Anton Evangelatov
8cf3accc53
Update itests/kit/ensemble.go
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
2021-07-06 17:25:33 +02:00
Anton Evangelatov
4be0a7a215 resolve conflicts with master 2021-07-05 13:13:32 +02:00
Łukasz Magiera
8a94ab676e storage: Fix FinalizeSector with sectors in stoage paths 2021-07-02 19:54:45 +02:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
Anton Evangelatov
9ec7cd7239 enable libp2p options 2021-06-30 12:55:43 +02:00
Raúl Kripalani
6a48fbbc11 increase suspension threshold to 300. 2021-06-23 19:21:42 +01:00
Raúl Kripalani
58f348cb7f add ability to suspend deal-making until CE is stable. 2021-06-23 19:14:27 +01:00
Raúl Kripalani
9b3188d110 split deals tests. 2021-06-23 18:13:29 +01:00
Raúl Kripalani
f4d0db83d8 Merge branch 'raulk/circleci-dynamic' into raulk/itests-refactor-kit 2021-06-23 17:33:17 +01:00
Raúl Kripalani
211751f8b8 declare some tests as expensive. 2021-06-23 16:37:43 +01:00
Anton Evangelatov
4217ec5308 make auth on remote rpc server optional 2021-06-23 13:27:03 +02:00
Anton Evangelatov
21b51328f9 adding TestDealWithMarketAndMinerNode 2021-06-23 12:44:52 +02:00
Anton Evangelatov
7fb93d4585 refactor kit.NewDealHarness to include main miner and market node 2021-06-23 11:24:55 +02:00
Anton Evangelatov
d98eca7260 Merge branch 'raulk/itests-refactor-kit' into nonsense/split-market-miner-processes 2021-06-23 10:58:37 +02:00
Raúl Kripalani
2e9e1c2330 avoid double BlockMiner instantiation. 2021-06-22 18:15:38 +01:00
Anton Evangelatov
fc7feffaad Merge branch 'raulk/itests-refactor-kit' into nonsense/split-market-miner-processes 2021-06-22 17:37:50 +02:00
Raúl Kripalani
098eb6bfff try using bg context on constructor. 2021-06-22 16:36:06 +01:00
Anton Evangelatov
a828b15fbe revert MockSectorMgr 2021-06-22 15:22:08 +02:00
Anton Evangelatov
b0d1668ab9 comment out preseal sectors 2021-06-22 13:34:45 +02:00
Anton Evangelatov
5e10d53ca8 wip 2021-06-22 12:16:21 +02:00
Anton Evangelatov
6720463799 resolve merge conflicts 2021-06-22 11:28:23 +02:00
Raúl Kripalani
bb032b526c switch back to ws API. 2021-06-21 23:24:59 +01:00
Raúl Kripalani
0e2d06fc39 switch to http API. 2021-06-21 23:10:17 +01:00
Raúl Kripalani
4fcd0b7aca disable resource filtering on scheduler. 2021-06-21 21:20:45 +01:00
Raúl Kripalani
b7a5e3cd0f Merge branch 'raulk/worker-disable-resource-filtering' into raulk/itests-refactor-kit 2021-06-21 21:09:04 +01:00
Raúl Kripalani
b6147fb27f extern/storage: retype resource filtering strategy to enum. 2021-06-21 20:28:15 +01:00
Raúl Kripalani
f3b6f8de1a add ability to ignore worker resources when scheduling. 2021-06-21 20:08:18 +01:00
Raúl Kripalani
132b49728c Merge branch 'master' into raulk/itests-refactor-kit 2021-06-21 18:26:04 +01:00
Raúl Kripalani
00fa3878d4 remove debug statement. 2021-06-21 18:22:08 +01:00
Raúl Kripalani
83ee345b1c Merge branch 'master' into raulk/itests-refactor-kit 2021-06-21 18:19:26 +01:00
Łukasz Magiera
b37a66c7c9 Merge remote-tracking branch 'origin/release/v1.10.0' into chore/merge-1.10 2021-06-21 17:03:03 +02:00
Raúl Kripalani
d1b291de5e fix proof types. 2021-06-21 13:24:25 +01:00
Raúl Kripalani
8a418bf982 rename {kit2=>kit}. 2021-06-18 19:45:29 +01:00
Raúl Kripalani
a7d8d15c13 kill old kit 🎉 2021-06-18 19:42:06 +01:00
Anton Evangelatov
31e6fb154e resolve conflicts 2021-06-14 11:27:54 +02:00
aarshkshah1992
3d086dfb43 changes as per review 2021-06-14 09:40:34 +05:30
aarshkshah1992
fed5afa704 merge master 2021-06-11 09:35:20 +05:30
Raúl Kripalani
d3fc6833a5 itests/kit: add guard to ensure imports from tests only. 2021-06-10 12:05:35 +01:00
Raúl Kripalani
6d46be53bd make tests no longer create auth tokens. 2021-06-10 00:15:29 +01:00
Raúl Kripalani
6e4eae69ac fix merge error that led to test failures. 2021-06-09 23:11:34 +01:00
Raúl Kripalani
7df3f3755d itests: move init. 2021-06-09 17:32:08 +01:00
Raúl Kripalani
d1f749a808 fix log. 2021-06-09 00:46:57 +01:00
Raúl Kripalani
9eecbb1b20 fix rpc tests. 2021-06-09 00:17:39 +01:00
Raúl Kripalani
cd6f91ba6f fix lint. 2021-06-07 23:47:38 +01:00
Raúl Kripalani
a274fcc533 Merge branch 'master' into raulk/itests 2021-06-07 23:44:22 +01:00
Raúl Kripalani
db1a61852d Merge branch 'master' into raulk/itests 2021-06-07 23:17:44 +01:00
Raúl Kripalani
75c88d0385 make gateway tests use gateway rpc setup methods. 2021-05-23 18:57:04 +01:00
Raúl Kripalani
f08c792686 minor renames. 2021-05-21 20:39:41 +01:00
Raúl Kripalani
416340d227 fix paych test; re-add pubsub bootstrapping. 2021-05-21 19:46:58 +01:00
Raúl Kripalani
bf36891972 fix lint errors. 2021-05-21 15:02:03 +01:00
Raúl Kripalani
2e9b0f0895 adapt cmd/lotus-storage-miner tests. 2021-05-21 14:52:30 +01:00
Raúl Kripalani
b56568d720 fix client tests. 2021-05-21 13:56:04 +01:00
Raúl Kripalani
20dfe220f3 fix lifecycle of BlockMiner. 2021-05-21 13:39:09 +01:00
Raúl Kripalani
25daa0c8e4 itests: create deal harness. 2021-05-20 16:12:42 +01:00
Raúl Kripalani
1902c4c687 itests: rename Builder methods. 2021-05-20 12:17:41 +01:00
Raúl Kripalani
2a70ff96bf move the gateway test to itests package. 2021-05-19 17:34:48 +01:00
Raúl Kripalani
6f4349064a fix bad rename. 2021-05-19 17:30:43 +01:00
Raúl Kripalani
5d34c8b7da wip move MineUntilBlock under BlockMiner; other simplifications. 2021-05-19 17:30:43 +01:00
Raúl Kripalani
2a71c47397 simplify BlockMiner. 2021-05-19 17:30:43 +01:00
Raúl Kripalani
0cfef0fdbb wip extract test kit. 2021-05-19 17:30:43 +01:00