cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
20,632 Commits 24 Branches 342 Tags 314 MiB
226542f97c
Commit Graph

179 Commits

Author SHA1 Message Date
Steven Allen
3846170302 refactor events system 2021-08-30 16:43:21 -07:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
ZenGround0
6da26dcefc Remove network version panic 2021-08-10 13:07:30 -04:00
Jakub Sztandera
a5bcf8b5b2 Add nolint 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2021-07-27 09:10:41 -07:00
Jakub Sztandera
ab71bc0e78 Don't test extra as we don't support it 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2021-07-27 09:10:41 -07:00
Jakub Sztandera
26da526379 Verify Voucher locks in VoucherValidUnlocked 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2021-07-27 09:10:41 -07:00
Raúl Kripalani
63f929541f itests: fix gateway tests parameters. 2021-05-21 17:20:56 +01:00
Łukasz Magiera
e8f28d7b9f Fix tests 2021-04-06 12:24:58 +02:00
Łukasz Magiera
81bd27911f Propagate StateMsg api changes 2021-04-05 19:56:53 +02:00
Łukasz Magiera
deb2b90b6a Fix lotus/miner build 2021-04-05 13:23:46 +02:00
Łukasz Magiera
f4e46c9003 implement v1 api with api modules 2021-04-03 13:20:50 +02:00
Łukasz Magiera
53f915d155 Import secp sigs in paych tests 2021-03-25 12:28:53 +01:00
whyrusleeping
cbc7f1c244 fix paychmgr constructor to take an easier to implement interface 2021-03-12 00:39:20 -08:00
whyrusleeping
bf42a3be5e move DI stuff for paychmgr into modules 2021-03-11 17:35:31 -08:00
Łukasz Magiera
26399dba70 Update markets, cbor-gen with soft map decoding 2021-02-19 20:11:43 +01:00
Łukasz Magiera
0c6aef221d Fix error logging format strings 2021-02-11 12:00:26 +01:00
Łukasz Magiera
c59e2fea5b Use correct contexts in storageadapter 2021-02-02 18:43:49 +01:00
Dirk McCormick
dc8ff27b36 fix: when waiting for deal commit messages, ignore unsuccessful messages 2020-12-14 12:30:52 +01:00
Dirk McCormick
9745f676ed fix: race in paych manager when req context is cancelled 2020-11-11 14:40:48 +01:00
Dirk McCormick
79a8ff04fd refactor: simplify chain event Called API 2020-10-30 14:00:32 +01:00
Dirk McCormick
5ed57d34f0 fix: race in paych manager add funds 2020-10-26 11:09:56 +01:00
Dirk McCormick
92942d44d1 feat: lite-mode - market storage and retrieval clients 2020-10-23 15:02:26 +02:00
Steven Allen
bcabe7b3b5 migrate methods to abstracted methods 
Method numbers never change anyways. At worst, we'll deprecate old methods and
have to explicitly import them from the correct actors version to use them.
 2020-10-21 12:18:37 -07:00
Steven Allen
4e730b5ec8 port to v2 imports 2020-10-21 12:16:23 -07:00
Łukasz Magiera
2f70a91665 Merge remote-tracking branch 'origin/master' into next 2020-10-06 23:54:59 +02:00
Aayush Rajasekaran
b5907c665b Fix test 2020-10-06 03:43:20 -04:00
Steven Allen
9621dbb4f1 fix tests to use v2 types 2020-10-05 13:27:34 -07:00
Steven Allen
8292d60196 Merge branch 'master' into asr/spec-v1 2020-10-02 18:03:03 -07:00
Dirk McCormick
594ec8855c fix: read lane state from chain as well as datastore 2020-10-01 12:08:11 +02:00
Dirk McCormick
e2ecc35dff test: add test for voucher after add-funds 2020-10-01 11:35:41 +02:00
Dirk McCormick
9dc75a7bc4 fix: verify voucher amount check 2020-10-01 11:35:41 +02:00
Steven Allen
2e9915ac28 move common from message param into message builder constructor 2020-09-30 10:58:34 -07:00
Steven Allen
4c60d7b5ce abstract over paych messages 2020-09-30 10:04:10 -07:00
Steven Allen
23b729a056 remove proof parameter from payment channels 
It never worked properly, and will be removed in actors v2.
 2020-09-30 09:35:42 -07:00
Steven Allen
233d8a9b72 update even more imports 2020-09-28 14:25:58 -07:00
Aayush Rajasekaran
a876a0ba44 Use actor state addresses 2020-09-23 02:32:40 -04:00
Aayush Rajasekaran
476e7992e8 Add an error return to all actor state interface methods 2020-09-23 01:51:38 -04:00
Steven Allen
d33dd4f7bc more renames 2020-09-21 16:01:29 -07:00
Steven Allen
63f026f7c3 migrate some more imports 2020-09-21 15:24:45 -07:00
Steven Allen
1bf3b4989d rename imports to match actors code 
`sed -i 's/\bv0\(\w\)\(\w*\)/\L\1\E\20/g' **/*.go`
 2020-09-18 14:59:27 -07:00
hannahhoward
05c11531b1 feat(paych): convert paych actor 
build abstraction for paych actor and switch to using it in payment channel manager and state
predicates
 2020-09-15 21:06:04 -07:00
Aayush Rajasekaran
39755a294a Update to specs v0.9.6 2020-09-07 15:48:41 -04:00
Dirk McCormick
51ef39592f feat: paych - get available funds by address or by from/to 2020-09-04 13:44:09 +02:00
Dirk McCormick
8423325a6e refactor: paych - use channel accessor from/to instead of passing them around as params 2020-09-03 15:31:49 +02:00
Dirk McCormick
1d3a21f6c8 docs: paych - correct comments on store fields 2020-09-03 08:22:40 +02:00
Dirk McCormick
3daa0b5e63 fix: paych - clarify behaviour of current available funds for non-existent channel 2020-09-03 08:20:08 +02:00
Dirk McCormick
2c98bf0cc7 feat: PaychAvailableFunds API method 2020-09-02 14:31:32 +02:00
Dirk McCormick
340d11be38 fix: check voucher spendable should take into account submitted vouchers 2020-08-25 12:51:29 +02:00
Łukasz Magiera
fda038cd89
Merge pull request #3168 from filecoin-project/fix/paych-chk-vchr 
Payment channel check voucher should get inbound channel from state
 2020-08-24 23:04:34 +02:00
Łukasz Magiera
2f6f978cd5
Merge pull request #3111 from filecoin-project/fix/paych-addvchr-chk 
paych: check To address is owned by wallet for inbound channels
 2020-08-24 23:02:10 +02:00
Łukasz Magiera
ae19b8ebfd
Merge pull request #3146 from filecoin-project/feat/paych-cli-voucher-tests 
paych: modify `lotus paych voucher best-spendable` to output _all_ best vouchers
 2020-08-24 22:59:20 +02:00
anorth
4ce71bce2d Change Message.Version to be a uint64 2020-08-20 14:37:21 +10:00
Dirk McCormick
4a3c273072 fix: paych - check wallet for key for channel To address 2020-08-19 18:56:51 -04:00
Dirk McCormick
d317835535 fix: paych check voucher should get inbound channel from state 2020-08-19 09:17:08 -04:00
Dirk McCormick
d1592b01c3 feat: paych cli voucher tests 2020-08-18 11:33:04 -04:00
Dirk McCormick
b25215e31b fix: paych add voucher - for inbound voucher check channel To address is owned by wallet 2020-08-17 11:13:13 -04:00
Dirk McCormick
536b2ad9d7 fix: paych - bug with allocating lane 2020-08-13 10:14:11 -04:00
Steven Allen
cac3d07abb Fix payment channel for actor changes 
Also, allocate the next available lane, instead of max+1. That way, we fill
holes.

Finally, check some error cases.
 2020-08-12 17:29:00 -07:00
Dirk McCormick
335d165db6 refactor: use spec-actors paych with AMT for lane states 2020-08-12 17:06:18 -07:00
Łukasz Magiera
12f7e2d9bf Use a struct for send metadata in MpoolPushMessage 2020-08-12 22:17:29 +02:00
Łukasz Magiera
e1a1b325bc Merge remote-tracking branch 'origin/next' into feat/max-msg-fee-config 2020-08-12 21:55:03 +02:00
Łukasz Magiera
6fe743f3a9 Fix tests 2020-08-12 20:03:07 +02:00
Steven Allen
9248e5a572 Update specs-actors, cbor-gen, and go-address 2020-08-12 10:32:39 -07:00
Łukasz Magiera
d0147aa50f Add maxFee param to MpoolPushMessage 2020-08-12 19:06:16 +02:00
Dirk McCormick
f857ee159e refactor: unify manager interfaces for easier mocks 2020-08-11 13:15:39 -04:00
Łukasz Magiera
51cf4a5749
Merge pull request #2962 from filecoin-project/refactor/rip-paych-pubsub 
Paych refactor: replace custom pubsub impl with lib
 2020-08-11 18:32:05 +02:00
Łukasz Magiera
957d83f069
Merge pull request #2955 from filecoin-project/refactor/paych-collect 
Refactor: move paych collect inside manager
 2020-08-11 01:42:10 +02:00
Dirk McCormick
51444c7f18 refactor: replace custom paych pubsub with existing impl 2020-08-10 17:53:10 -04:00
Dirk McCormick
a6100b7e46 refactor: move collect inside paych manager 2020-08-10 13:21:25 -04:00
Dirk McCormick
0edf65855a feat: paychmgr - merge queued add funds 2020-08-10 11:26:13 -04:00
Dirk McCormick
d3aa5e638d refactor: remove unused method 2020-08-07 12:53:40 -04:00
Dirk McCormick
e488996fd0 fix: remove unused onComplete callback 2020-08-07 12:53:40 -04:00
Łukasz Magiera
07bf155295 Merge remote-tracking branch 'origin/next' into feat/dyn-base-fee 2020-08-06 23:41:54 +02:00
Jakub Sztandera
ab08858b45
Delete GasPrice from this world 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-08-06 23:08:42 +02:00
Dirk McCormick
42e1906d9c fix: paychmgr - bail out if there's an error writing to store 2020-08-06 14:01:38 -04:00
Dirk McCormick
0a3e51a74c refactor: remove sleeps from paych tests 2020-08-06 13:39:13 -04:00
Dirk McCormick
1ef9113ff0 fix: track payment channel by ID instead of from/to 2020-08-06 12:07:13 -04:00
Dirk McCormick
63bb09553a fix: race in paych manager test 2020-08-06 12:07:13 -04:00
Dirk McCormick
28985c8bcf fix: tighten up testing and add a couple of missing locks 2020-08-06 12:07:13 -04:00
Dirk McCormick
45d9ddc79f fix: paych manager dependency injection 2020-08-06 12:04:10 -04:00
Dirk McCormick
c897cc0cc9 docs: fix comments 2020-08-06 12:04:10 -04:00
Dirk McCormick
fdfccf0466 WIP: fix payment channel locking 2020-08-06 12:04:10 -04:00
Lucas Molas
f1f9ce5351 make gen 2020-07-30 17:25:18 -03:00
Dirk McCormick
46b7546d51 fix: zero out gas limit 2020-07-22 13:55:31 -04:00
hannahhoward
f27bd1dacb feat(settler): address pr comments 
clean up imports, improve api import, fix text, support waiting on messages
 2020-07-22 13:34:35 -04:00
hannahhoward
98535d5e06 feat(settler): monitor settled channels 
Add a routine to monitor the chain for payment channels settling, and if it's a channel that we are
the recipient of payments on, submit any outstanding vouchers to the
chain
 2020-07-22 13:34:35 -04:00
Jakub Sztandera
7da629d03b
Update gas prices 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-07-20 21:31:05 +02:00
Łukasz Magiera
cb6767a02b Merge remote-tracking branch 'origin/next' into feat/actors-miner-refactor 2020-07-17 15:18:11 +02:00
Łukasz Magiera
9f4fb428c9 gofmt 2020-07-16 17:24:55 +02:00
Jakub Sztandera
7b14d445b4
Update message gas limits 
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
 2020-07-15 20:01:57 +02:00
Łukasz Magiera
d41a07161d paychmgr: handle voucher.ChannelAddr 2020-07-15 11:12:03 +02:00
Łukasz Magiera
faebc4c948 WIP Integrating specs-actors with refactored miner state 2020-07-14 13:45:45 +02:00
Dirk McCormick
a12e5884ab fix: add TODO about lane merging 2020-07-10 15:24:13 -04:00
Dirk McCormick
f07c7377b6 feat: account for other vouchers when calculating voucher validity 2020-07-10 14:06:52 -04:00
Dirk McCormick
b888385ba1 refactor: lint fixes 2020-07-09 18:49:43 -04:00
Dirk McCormick
6c70ef7c7d refactor: simplify state management 2020-07-09 18:27:39 -04:00
Dirk McCormick
1cdb008bd5 fix: lane allocation 2020-07-09 17:20:17 -04:00
Dirk McCormick
770ee8b54d test: payment channel manager voucher and channel info tests 2020-07-09 16:35:43 -04:00
Dirk McCormick
9a45ffd459 test: add basic paychmgr store test 2020-07-08 17:34:44 -04:00
Alexey
eafb04004e
Fix to addresses in paychmgr channel creation 2020-06-26 09:06:01 +03:00