Commit Graph

119 Commits

Author SHA1 Message Date
Geoff Stuart
b4c04ad927 update markets 2022-10-06 11:06:21 -04:00
Łukasz Magiera
9aa5659d24 sealing pipeline: Drop TipSetToken, use TipSetKey directly 2022-06-16 12:05:58 +02:00
Łukasz Magiera
39cede946c chore: Fix more imports with '2' suffix 2022-06-14 21:37:54 +02:00
Łukasz Magiera
98a48a47f8 storage: Move extern/storage-sealing to storage/pipeline 2022-06-14 19:41:59 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Aayush
8cca9b1970 Use new go-state-types accessors 2022-05-17 15:21:27 -04:00
Aayush
339b7db2b7 Integrate FIP 0027 2022-04-14 19:32:56 +03:00
Dirk McCormick
839fc8df7d fix: when retrying Add Piece, first seek to start of reader 2021-12-17 14:44:51 +01:00
Aayush Rajasekaran
dfb65ed89f Plumb contexts through 2021-12-11 17:04:00 -05:00
Steven Allen
3846170302 refactor events system 2021-08-30 16:43:21 -07:00
Dirk McCormick
7b7a5b0b21 revert: changes to OnDealExpiredOrChanged in 2021-08-30 10:42:41 +02:00
Dirk McCormick
c7e7f89026 refactor: OnDealExpiredOrSlashed get deal by proposal instead of deal ID 2021-08-27 09:02:56 +02:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ()
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Steven Allen
be2ecf6236 fix flaky TestDealPublisher and re-enable
fixes 
2021-08-05 19:46:08 -07:00
Anton Evangelatov
a989f60e27 add SectorAddPieceToAny and SectorUnsealPiece to StorageMiner iface; model moved to api package - PieceDealInfo, DealSchedule 2021-05-19 13:05:07 +02:00
Łukasz Magiera
81bd27911f Propagate StateMsg api changes 2021-04-05 19:56:53 +02:00
Łukasz Magiera
deb2b90b6a Fix lotus/miner build 2021-04-05 13:23:46 +02:00
Dirk McCormick
ced455bba9 feat: add configurable max provider collateral 2021-03-02 10:24:57 +01:00
Łukasz Magiera
0c6aef221d Fix error logging format strings 2021-02-11 12:00:26 +01:00
Łukasz Magiera
b938ac4cc8
Merge pull request from filecoin-project/refactor/mkts-wait-publish
Move waiting for publish deals from markets into lotus
2021-02-05 10:59:18 +01:00
Dirk McCormick
fc8ee481a8 refactor: move waiting for publish deals from markets into lotus 2021-02-03 10:36:38 +01:00
Łukasz Magiera
c59e2fea5b Use correct contexts in storageadapter 2021-02-02 18:43:49 +01:00
Łukasz Magiera
332ea8a126 Merge remote-tracking branch 'origin/master' into feat/deal-batch-publish 2021-02-02 18:21:14 +01:00
Łukasz Magiera
94009f247f
Merge pull request from filecoin-project/feat/sealing-handle-batch-publish
Handle batch publish storage deals message in sealing recovery
2021-01-28 19:33:49 +01:00
Dirk McCormick
adac340f3f feat: batch publish deal messages 2021-01-25 14:25:57 +01:00
Dirk McCormick
1b494acc9e feat: handle batch publish storage deals message in sealing recovery 2021-01-25 14:10:05 +01:00
Steven Allen
03cd3760bb correctly pick the seal type based on the network version
Of course, we should really just run all of our tests post actors v2.
2021-01-21 15:21:20 -08:00
Aayush Rajasekaran
ab90a3b2bc Actors update: MinerInfo.SealProofType has been removed 2021-01-21 15:21:19 -08:00
Łukasz Magiera
ea85a2457e storageadapter: Address review 2021-01-21 15:44:27 +01:00
Łukasz Magiera
de0a452282 storageadapter: Fix tests, more testable diff api 2021-01-21 12:21:26 +01:00
Łukasz Magiera
b8bba15ed2 markets: Fix deal expired panic 2020-12-17 12:01:51 +01:00
Dirk McCormick
239e180683 feat: markets - separate watching for pre-commit from prove-commit 2020-11-24 13:24:41 -08:00
Steven Allen
638900f97f Remove static proof type configuration from market subsystem
Instead, use proof type from miner actor. This will, in turn, use the upgraded
proof type if/when it's switched at runtime.

TODO: Consider making this some form of config option instead?
2020-11-16 19:04:17 +01:00
Łukasz Magiera
961a25298b predicates: Fast StateGetActor wrapper 2020-11-12 23:45:06 +01:00
Dirk McCormick
0d243bb824 refactor: integrate new FundManager 2020-11-10 21:11:05 -08:00
hannahhoward
7519bdde42 feat(markets): check deal equality
in OnDealSectorCommitted, verify that deals looked up match the deal proposals which were made
2020-11-09 18:48:28 -08:00
hannahhoward
09a0d6e607 style(lint): fix lint issues 2020-11-09 18:48:28 -08:00
hannahhoward
a3e93554e5 refactor(markets): refactor OnDealSectorCommitted
extract OnDealSectorCommitted from adapters and test
2020-11-09 18:48:28 -08:00
hannahhoward
d2acc78787 feat(markets): handle deal ID changes
make OnDealSectorCommitted handle changes to deal ids
2020-11-09 18:48:28 -08:00
Łukasz Magiera
c6a8efff7d
Merge pull request from filecoin-project/feat/cache-deal-state-matcher
Cache deal states for most recent old/new tipset
2020-10-31 01:00:11 +01:00
Dirk McCormick
79a8ff04fd refactor: simplify chain event Called API 2020-10-30 14:00:32 +01:00
Dirk McCormick
f79652c28c feat: cache deal states for most recent old/new tipset 2020-10-28 16:55:48 +01:00
Steven Allen
bcabe7b3b5 migrate methods to abstracted methods
Method numbers never change anyways. At worst, we'll deprecate old methods and
have to explicitly import them from the correct actors version to use them.
2020-10-21 12:18:37 -07:00
Steven Allen
4e730b5ec8 port to v2 imports 2020-10-21 12:16:23 -07:00
Łukasz Magiera
8fe8a5df45 Cap market provider messages 2020-10-02 18:35:52 +02:00
Steven Allen
d33dd4f7bc more renames 2020-09-21 16:01:29 -07:00
Łukasz Magiera
6eda53565f Most tests passing 2020-09-17 17:30:24 +02:00
Aayush Rajasekaran
31ff5230bb Get State API almost working 2020-09-17 05:05:32 -04:00
Aayush Rajasekaran
053cfc1cc7 Migrate verifreg actor 2020-09-17 03:55:23 -04:00
hannahhoward
691bd9f442 feat(markets): complete markets conversion
complete markets conversion to using chain/actors types, also replacing DealProposal/DealState
interfaces with structs
2020-09-17 00:43:14 -07:00