Commit Graph

76 Commits

Author SHA1 Message Date
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671)
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Raúl Kripalani
db1a61852d Merge branch 'master' into raulk/itests 2021-06-07 23:17:44 +01:00
Peter Rabbitson
1e72d8f14c Reduce noise from 'peer has different genesis' messages
After the unification of all networks behind a build-tag, the amount of these
warnings has risen sharply. Reduce it to a debug-level, since it isn't actionable
to an operator.
2021-05-31 13:43:24 +02:00
Raúl Kripalani
c46d4ae529 wip 2021-05-19 17:30:43 +01:00
Łukasz Magiera
26399dba70 Update markets, cbor-gen with soft map decoding 2021-02-19 20:11:43 +01:00
Łukasz Magiera
9270ac6358 Make some logs quieter 2020-11-03 13:28:41 +01:00
Jakub Sztandera
46d3769a44
Add peer to peer manager before fetching the tipset
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-09-08 20:13:16 +02:00
Aayush Rajasekaran
39755a294a Update to specs v0.9.6 2020-09-07 15:48:41 -04:00
anorth
4ce71bce2d Change Message.Version to be a uint64 2020-08-20 14:37:21 +10:00
Steven Allen
9248e5a572 Update specs-actors, cbor-gen, and go-address 2020-08-12 10:32:39 -07:00
Łukasz Magiera
823d8a64e1 Merge remote-tracking branch 'origin/master' into update/next-master 2020-08-07 05:09:26 +02:00
Łukasz Magiera
5ef7328fe3 Rerun cbor-gen 2020-08-06 23:27:23 +02:00
Brian Hoffman
0bb4c20e02 Fix misspelling in HelloMessage 2020-08-04 11:33:29 -04:00
Lucas Molas
f1f9ce5351 make gen 2020-07-30 17:25:18 -03:00
whyrusleeping
0594177191 add some better logging for hello handshake failures 2020-07-23 16:58:21 -07:00
Łukasz Magiera
cb6767a02b Merge remote-tracking branch 'origin/next' into feat/actors-miner-refactor 2020-07-17 15:18:11 +02:00
Łukasz Magiera
faebc4c948 WIP Integrating specs-actors with refactored miner state 2020-07-14 13:45:45 +02:00
Raúl Kripalani
13de81b3b2 introduce the ability to mock time. 2020-07-10 15:51:45 +01:00
Jakub Sztandera
d6615b6286
Cleanup many lint warnings
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-05-27 22:53:20 +02:00
Aayush Rajasekaran
f7a45f6c12 Fix 3 typos 2020-05-14 18:27:47 -04:00
Jeromy
d7af5c67e9 update to latest cbor-gen 2020-05-04 15:19:48 -07:00
Jeromy
eb4f4675a5 add version field to message type 2020-04-28 19:12:52 -04:00
Łukasz Magiera
8e13920e7b cbor-gen 2020-04-17 19:53:09 +02:00
laser
2f4732760a update CBOR encoders/decoders 2020-04-06 13:35:29 -07:00
Łukasz Magiera
a1b35aa9d5 Change api.SectorState to a string 2020-04-03 18:29:31 +02:00
Jeromy
417d434973 several fixes and improvements while debugging interop 2020-03-31 18:35:23 -07:00
Jakub Sztandera
300c89f541 Update to new sepcs actors, more debug
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-03-26 01:02:13 +01:00
Whyrusleeping
d74ede5a79 Merge pull request #1428 from filecoin-project/feat/graphsync-fetching
WIP: integrate graphsync for chainsync requests
2020-03-23 18:47:17 -07:00
Jeromy
f1dbd35407 change genesis fields to match interop 2020-03-21 15:25:00 -07:00
Jeromy
f3781e8329 update and rerun cbor gen 2020-03-21 14:17:01 -07:00
whyrusleeping
1076a1a89d WIP: integrate graphsync for chainsync requests 2020-03-20 21:30:24 -07:00
Łukasz Magiera
f83bbc2cbe Regen cbor marshalers 2020-02-28 00:34:48 +01:00
Łukasz Magiera
1920981713 hello: regen type-gen 2020-02-25 21:21:58 +01:00
Łukasz Magiera
794429dd55 post-upstream-merge fixes 2020-02-24 18:45:25 +01:00
Łukasz Magiera
d787aa5007 Merge branch 'testnet/3' into feat/specs-actors 2020-02-24 18:32:02 +01:00
Łukasz Magiera
b12482df89 move peermgr, tracing into lib 2020-02-22 12:36:22 +01:00
Łukasz Magiera
5b96b648b4 hello fixes 2020-02-21 20:25:57 +01:00
Jakub Sztandera
8610bcbec3 Use tuple encoding
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-02-12 19:23:15 +01:00
Jakub Sztandera
50702f309f Use new hello protocol
Using map encoder.
Types in LatencyMessage are int64 in spec, butl 1. tupled generator
does not support int64 right now; 2. there is no harm in using uint64 here.

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-02-12 19:08:49 +01:00
Jakub Sztandera
1ed62628a7
Update go-log to v2
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2020-01-08 20:22:31 +01:00
Łukasz Magiera
1e7f10e387 gofmt 2020-01-07 15:00:10 +01:00
hannahhoward
718acb0a86 refactor(cborutil): extract cborutil library
Use extracted cborutil library
2020-01-07 14:59:35 +01:00
whyrusleeping
6cb6167393 bump up filecoin related weights 2019-12-17 10:16:10 -08:00
whyrusleeping
eb4077b1af allow specifying some protected peers in the config file 2019-12-16 21:37:31 -08:00
Łukasz Magiera
3892e94bfe
Merge pull request #938 from filecoin-project/feat/load-tipset-cache
cache load tipset
2019-12-16 21:13:03 +01:00
whyrusleeping
6e94377469 cache load tipset 2019-12-16 11:22:56 -08:00
whyrusleeping
99eae286c2 clean up some log spam 2019-12-16 09:14:21 -08:00
Jakub Sztandera
c9eed9635b
Fix hello calling close before compleating timestamp exchange
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-12-11 22:18:22 +01:00
Jakub Sztandera
86feebc899
Fix timeoffset hello calculation
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-12-11 16:36:08 +01:00
Jakub Sztandera
32b429a69a
Disconnect if hello fails
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>
2019-12-10 22:09:48 +01:00