Commit Graph

130 Commits

Author SHA1 Message Date
Jakub Sztandera
98c99b4fea
Merge pull request from filecoin-project/fix/vouchers
Verify Voucher locks in VoucherValidUnlocked
2021-09-28 14:29:04 +02:00
Steven Allen
3846170302 refactor events system 2021-08-30 16:43:21 -07:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ()
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
ZenGround0
6da26dcefc Remove network version panic 2021-08-10 13:07:30 -04:00
Jakub Sztandera
a5bcf8b5b2 Add nolint
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-07-27 09:10:41 -07:00
Jakub Sztandera
ab71bc0e78 Don't test extra as we don't support it
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-07-27 09:10:41 -07:00
Jakub Sztandera
26da526379 Verify Voucher locks in VoucherValidUnlocked
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2021-07-27 09:10:41 -07:00
Raúl Kripalani
63f929541f itests: fix gateway tests parameters. 2021-05-21 17:20:56 +01:00
Łukasz Magiera
e8f28d7b9f Fix tests 2021-04-06 12:24:58 +02:00
Łukasz Magiera
81bd27911f Propagate StateMsg api changes 2021-04-05 19:56:53 +02:00
Łukasz Magiera
deb2b90b6a Fix lotus/miner build 2021-04-05 13:23:46 +02:00
Łukasz Magiera
f4e46c9003 implement v1 api with api modules 2021-04-03 13:20:50 +02:00
Łukasz Magiera
53f915d155 Import secp sigs in paych tests 2021-03-25 12:28:53 +01:00
whyrusleeping
cbc7f1c244 fix paychmgr constructor to take an easier to implement interface 2021-03-12 00:39:20 -08:00
whyrusleeping
bf42a3be5e move DI stuff for paychmgr into modules 2021-03-11 17:35:31 -08:00
Łukasz Magiera
26399dba70 Update markets, cbor-gen with soft map decoding 2021-02-19 20:11:43 +01:00
Łukasz Magiera
0c6aef221d Fix error logging format strings 2021-02-11 12:00:26 +01:00
Łukasz Magiera
c59e2fea5b Use correct contexts in storageadapter 2021-02-02 18:43:49 +01:00
Dirk McCormick
dc8ff27b36 fix: when waiting for deal commit messages, ignore unsuccessful messages 2020-12-14 12:30:52 +01:00
Dirk McCormick
9745f676ed fix: race in paych manager when req context is cancelled 2020-11-11 14:40:48 +01:00
Dirk McCormick
79a8ff04fd refactor: simplify chain event Called API 2020-10-30 14:00:32 +01:00
Dirk McCormick
5ed57d34f0 fix: race in paych manager add funds 2020-10-26 11:09:56 +01:00
Dirk McCormick
92942d44d1 feat: lite-mode - market storage and retrieval clients 2020-10-23 15:02:26 +02:00
Steven Allen
bcabe7b3b5 migrate methods to abstracted methods
Method numbers never change anyways. At worst, we'll deprecate old methods and
have to explicitly import them from the correct actors version to use them.
2020-10-21 12:18:37 -07:00
Steven Allen
4e730b5ec8 port to v2 imports 2020-10-21 12:16:23 -07:00
Łukasz Magiera
2f70a91665 Merge remote-tracking branch 'origin/master' into next 2020-10-06 23:54:59 +02:00
Aayush Rajasekaran
b5907c665b Fix test 2020-10-06 03:43:20 -04:00
Steven Allen
9621dbb4f1 fix tests to use v2 types 2020-10-05 13:27:34 -07:00
Steven Allen
8292d60196 Merge branch 'master' into asr/spec-v1 2020-10-02 18:03:03 -07:00
Dirk McCormick
594ec8855c fix: read lane state from chain as well as datastore 2020-10-01 12:08:11 +02:00
Dirk McCormick
e2ecc35dff test: add test for voucher after add-funds 2020-10-01 11:35:41 +02:00
Dirk McCormick
9dc75a7bc4 fix: verify voucher amount check 2020-10-01 11:35:41 +02:00
Steven Allen
2e9915ac28 move common from message param into message builder constructor 2020-09-30 10:58:34 -07:00
Steven Allen
4c60d7b5ce abstract over paych messages 2020-09-30 10:04:10 -07:00
Steven Allen
23b729a056 remove proof parameter from payment channels
It never worked properly, and will be removed in actors v2.
2020-09-30 09:35:42 -07:00
Steven Allen
233d8a9b72 update even more imports 2020-09-28 14:25:58 -07:00
Aayush Rajasekaran
a876a0ba44 Use actor state addresses 2020-09-23 02:32:40 -04:00
Aayush Rajasekaran
476e7992e8 Add an error return to all actor state interface methods 2020-09-23 01:51:38 -04:00
Steven Allen
d33dd4f7bc more renames 2020-09-21 16:01:29 -07:00
Steven Allen
63f026f7c3 migrate some more imports 2020-09-21 15:24:45 -07:00
Steven Allen
1bf3b4989d rename imports to match actors code
`sed -i 's/\bv0\(\w\)\(\w*\)/\L\1\E\20/g' **/*.go`
2020-09-18 14:59:27 -07:00
hannahhoward
05c11531b1 feat(paych): convert paych actor
build abstraction for paych actor and switch to using it in payment channel manager and state
predicates
2020-09-15 21:06:04 -07:00
Aayush Rajasekaran
39755a294a Update to specs v0.9.6 2020-09-07 15:48:41 -04:00
Dirk McCormick
51ef39592f feat: paych - get available funds by address or by from/to 2020-09-04 13:44:09 +02:00
Dirk McCormick
8423325a6e refactor: paych - use channel accessor from/to instead of passing them around as params 2020-09-03 15:31:49 +02:00
Dirk McCormick
1d3a21f6c8 docs: paych - correct comments on store fields 2020-09-03 08:22:40 +02:00
Dirk McCormick
3daa0b5e63 fix: paych - clarify behaviour of current available funds for non-existent channel 2020-09-03 08:20:08 +02:00
Dirk McCormick
2c98bf0cc7 feat: PaychAvailableFunds API method 2020-09-02 14:31:32 +02:00
Dirk McCormick
340d11be38 fix: check voucher spendable should take into account submitted vouchers 2020-08-25 12:51:29 +02:00
Łukasz Magiera
fda038cd89
Merge pull request from filecoin-project/fix/paych-chk-vchr
Payment channel check voucher should get inbound channel from state
2020-08-24 23:04:34 +02:00