Commit Graph

220 Commits

Author SHA1 Message Date
Łukasz Magiera
95b128b7bc chain: Cleanup consensus logic 2021-09-02 18:09:37 +02:00
Steven Allen
003eae81ce fix: address review 2021-08-30 16:43:21 -07:00
Steven Allen
3846170302 refactor events system 2021-08-30 16:43:21 -07:00
Steven Allen
43bbde1e6b fix: close chain head subscription when the reader is slow
The reader can just re-subscribe when they're ready to catch up. This
prevents a slow reader from bogging down the entire system.
2021-08-30 16:43:21 -07:00
Dirk McCormick
7b7a5b0b21 revert: changes to OnDealExpiredOrChanged in #5431 #7201 2021-08-30 10:42:41 +02:00
Dirk McCormick
07dcb074f8 fix: catch deal slashed because sector was terminated 2021-08-27 12:56:23 +02:00
dirkmc
77a19774cf fix events API timeout handling for nil blocks (#7184) 2021-08-27 09:05:00 +02:00
Steven Allen
1105c24e8c test: disable flaky TestBatchDealInput
See #4611.
2021-08-24 15:37:52 -07:00
Steven Allen
b509f1f098
Merge pull request #7148 from filecoin-project/raulk/test-large-deals
itests: support larger sector sizes; add large deal test.
2021-08-24 12:21:11 -07:00
Steven Allen
2a4ad207a5 test: disable flaky TestSimultaneousTransferLimit
See https://github.com/filecoin-project/lotus/issues/7152 for details.
2021-08-20 14:59:01 -07:00
Raúl Kripalani
342134f853 EnableLargeSectors: restore previous supported proof types. 2021-08-20 18:07:25 +01:00
Raúl Kripalani
0030e208a0 avoid reordering imports. 2021-08-20 15:59:19 +01:00
Raúl Kripalani
2dde022a00 itests: support larger sector sizes; add large deal test. 2021-08-20 15:53:24 +01:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671)
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
ZenGround0
ce58b119bf
Merge pull request #7038 from filecoin-project/spike/restrict-api-versions
Strict major minor version checking on v0 and v1 apis
2021-08-12 14:34:31 -04:00
ZenGround0
2d8c9bba99 more fix 2021-08-12 14:03:19 -04:00
ZenGround0
c8720fef02 Fix tests 2021-08-12 14:01:24 -04:00
Steven Allen
dcff06bd91 test: genesis actors version
Properly handle genesis in actors tests. Fast-forward upgrading to
actors v13 doesn't work because there needs to be at least a day between
v0 genesis and v13 (due to differences in miner cron).
2021-08-11 13:30:15 -07:00
Steven Allen
b885d69aaa chore: disable flaky test 2021-07-30 09:01:31 -07:00
ZenGround0
823bd54484 Review 2021-07-29 20:48:27 -07:00
ZenGround0
756e5ac0b8 Add a separate node to ensemble and check propagation reached 2021-07-29 20:48:27 -07:00
ZenGround0
7c99169fb4 Fix 2021-07-29 20:48:27 -07:00
ZenGround0
2380d252e1 Post rebase fix 2021-07-29 20:48:27 -07:00
ZenGround0
f2f6c209ea Integration test forcing max batch size 2021-07-29 20:48:27 -07:00
Peter Rabbitson
39325e5463
Merge branch 'master' into feat/allow_padding_redux 2021-07-26 20:53:27 +02:00
Peter Rabbitson
8d873e3edc Tests for online and offline dealpadding
Also bump the times on several flaky tests that can not complete in time
on a typical laptop ( and fail half the time on CircleCI )
2021-07-24 17:46:04 +02:00
Raúl Kripalani
8693df4656 fix racy TestSimultanenousTransferLimit. 2021-07-24 12:53:56 +01:00
Peter Rabbitson
2edf7fd25b Rewire itest's StartDeal to take the full API struct
This allows one to use the harness for much more versatile deal conditions
2021-07-24 04:17:42 +02:00
raulk
4ffbb03a82
Merge branch 'master' into raulk/merge-gs-fix 2021-07-22 23:37:17 +01:00
Łukasz Magiera
660829703a Merge remote-tracking branch 'origin/master' into feat/split-net-api 2021-07-22 15:38:06 +02:00
Łukasz Magiera
0236f2ec5d
Merge pull request #6802 from filecoin-project/fix/6786-rest
fix: always check if StateSearchMessage returns nil
2021-07-21 15:05:57 +02:00
Steven Allen
38919f59ad test: fix flaky window post tests
Wait until the network upgrade is finished. If we try to seal while it's
happening, we have a few annoying edge cases that can fail if we try to
submit some messages right on the upgrade epoch (which is why everyone
turns that kind of stuff off for the upgrade epoch).
2021-07-20 16:31:14 -07:00
Steven Allen
f9c759fcb2 fix: always check if StateSearchMessage returns nil
It returns nil on "not found".

Fixes the cases not covered in:
https://github.com/filecoin-project/lotus/pull/6787
2021-07-20 11:28:27 -07:00
Raúl Kripalani
5e66960ab6 fix merge error. 2021-07-20 17:51:09 +01:00
raulk
353ccf0c55
Merge branch 'raulk/merge-gs-fix' into raulk/merge-gs-fix-simultaneous-fix 2021-07-20 17:37:51 +01:00
Raúl Kripalani
c1b304b633 update graphsync. 2021-07-20 15:43:44 +01:00
hannahhoward
3b4df5e10d fix(itests): fix simultaneous transfer logic
limit parallelism on client side and also fix the issue with how ongoing transfers are detected
2021-07-16 11:46:03 -07:00
Raúl Kripalani
089bcceb33 remove test concurrency guardrails. 2021-07-16 19:09:56 +01:00
Raúl Kripalani
9042429abd Merge branch 'master' into raulk/merge-gs-fix 2021-07-16 18:19:50 +01:00
Raúl Kripalani
aeb8e57eef use https://github.com/ipfs/go-graphsync/pull/176. 2021-07-16 18:17:34 +01:00
Raúl Kripalani
b2feb7e960 upgrade go-fil-markets, go-data-transfer; add logging. 2021-07-16 17:30:45 +01:00
Łukasz Magiera
49e26cce7d api: Separate the Net interface from Common 2021-07-15 11:41:30 +02:00
hunjixin
b2f8b4011a fix imports 2021-07-15 14:52:12 +08:00
hunjixin
3e9ec01d04 rm fmt 2021-07-15 14:52:12 +08:00
hunjixin
a48a6a168a add test for new api 2021-07-15 14:52:12 +08:00
hunjixin
ac03c01e4a test for messages in tipset 2021-07-15 14:52:12 +08:00
Łukasz Magiera
583a8a13d9
Merge pull request #6629 from filecoin-project/feat/pledge-from-miner-balance
Config for collateral from miner available balance
2021-07-13 17:22:00 +02:00
Łukasz Magiera
7526a074d9 sealing: collateral buffer / falback config 2021-07-13 17:04:58 +02:00
Łukasz Magiera
83f2368507 Add CollateralFromMinerBalance config 2021-07-12 14:11:58 +02:00
Anton Evangelatov
4bc9fa04ba rename cfg.Subsystems.EnableStorageMarket to EnableMarkets 2021-07-12 12:12:29 +02:00