cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity
15,819 Commits 24 Branches 342 Tags 314 MiB
70f3b98574
Commit Graph

390 Commits

Author SHA1 Message Date
Łukasz Magiera
4a874eff70 post workers: Cleanup, tests 2022-01-14 14:17:52 +01:00
Aarsh Shah
b9d3585f4c get tests working 2022-01-12 16:55:45 +04:00
Aarsh Shah
3a52f05692 green ci 2022-01-12 15:53:15 +04:00
Aarsh Shah
edfc2c215c Merge remote-tracking branch 'origin/main' into feat/cid-to-piece-idx 2022-01-12 12:48:54 +04:00
zenground0
c309686679 Fix TooManyMarkedForUpgrade 2022-01-10 15:39:38 +05:30
zenground0
3dc3f55ecb Deflake more practically 2022-01-10 15:39:38 +05:30
zenground0
f59bfd65da Deflake snap deals integration test 2022-01-10 15:39:38 +05:30
zenground0
33f2d24f54 Snap Deals Integration 
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
 2022-01-10 15:39:38 +05:30
Darko Brdareski
dda1a42a2a Merge branch 'bloxico/system-test-matrix' of https://github.com/filecoin-project/lotus into merge_lotus 2021-12-20 15:48:16 +01:00
Dirk McCormick
bb56e97177 fix: rename query-retrieval-ask to retrieval-ask 2021-12-17 18:01:12 +01:00
Dirk McCormick
b1734f84b3 feat: retrieval ask CLI command 2021-12-17 15:55:12 +01:00
Łukasz Magiera
a4728d3c72
Merge pull request #7578 from filecoin-project/rvagg/SimultaneousTransfersForStoragePerClient 
feat(graphsync): allow setting of per-peer incoming requests for miners
 2021-12-17 14:27:15 +01:00
Łukasz Magiera
bc384c01e3 Merge remote-tracking branch 'origin/master' into deps/update-ctx-dsbs 2021-12-17 13:01:22 +01:00
Rod Vagg
9e7d9affbe
feat(graphsync): allow setting of per-peer incoming requests for miners 2021-12-17 15:04:16 +11:00
Darko Brdareski
3e32aa896c Annotate client storage deals feature 2021-12-16 14:36:02 +01:00
Darko Brdareski
2f1f35cc71 Annotate storage miner features 2021-12-15 15:30:42 +01:00
vyzo
fc7d4a13b0 fix itests 2021-12-14 16:06:59 +02:00
Darko Brdareski
a64f2421d2 Annotate 'incoming' subsystem 2021-12-14 11:33:33 +01:00
Aayush Rajasekaran
80d5e52923 Merge branch 'master' into next 2021-12-13 13:24:28 -05:00
Darko Brdareski
0addca1070 Fix bad annotations 2021-12-13 13:41:04 +01:00
Darko Brdareski
0169d0dafd Annotate state feature tests 2021-12-10 16:08:25 +01:00
Darko Brdareski
95f86f9de0 Annotate feature syncer 2021-12-10 11:33:29 +01:00
zenground0
a5be80828a RemoveData and Decode 
- Unsealing replica update with sector key works and tested
- Sector key generation added and tested
 2021-12-03 15:21:06 -05:00
c r
e16e9ad343
reorder transfer checks so as to ensure sending more money than you have to yourself fails with an error (fixing issue 7596) 
PR #7637, also adds tests to make sure behavior is correct across versions.
 2021-11-29 17:26:47 -05:00
Łukasz Magiera
5b5e6b9e44 retrieval: DagSpec.MatchPath -> ExportMerkleProof 2021-11-29 21:14:00 +01:00
Łukasz Magiera
227188e908 retrieval: Test non-matching path traversal 2021-11-29 20:52:55 +01:00
Łukasz Magiera
61791b90ea retrieval: Only output matching nodes, MatchPath dagspec 2021-11-29 20:40:55 +01:00
Peter Rabbitson
407bf4961c Modify preexisting test to reflect the simple-path car export behavior 
I am open to having a flag or something, as long as the behavior reflected in
deals_partial_retrieval_dm-level_test.go can be preserved for v0, and opt-in for v1
 2021-11-24 15:31:19 +01:00
Peter Rabbitson
46937a1b9f Add tests cementing the DM-level behavior on simple paths 2021-11-24 15:31:15 +01:00
Łukasz Magiera
597b72e286 retrieval: Fix lint, cli docsgen 2021-11-22 12:46:29 +01:00
Łukasz Magiera
b83a9b902a gofmt 2021-11-22 12:46:17 +01:00
Łukasz Magiera
b26906963b retrieval: Support multi-root export 2021-11-22 12:46:17 +01:00
Łukasz Magiera
d0503d409f fix TestPartialRetrieval 2021-11-22 12:46:04 +01:00
Łukasz Magiera
b868769ec8 more retrieval api work 2021-11-22 12:46:02 +01:00
Łukasz Magiera
89138bab4d Simplify retrieval APIs 2021-11-22 12:45:42 +01:00
Aayush Rajasekaran
5f1783c9a5 Address review 2021-11-19 15:11:13 -05:00
Aarsh Shah
7655e660f3 integrate store-the-hash 2021-11-18 12:07:12 +04:00
Aayush Rajasekaran
8665e32221 Update deps 2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
07f1be0ff6 Plug in the FFI call 2021-11-17 15:55:14 -05:00
Aayush Rajasekaran
1449644c34 Integrate v7 actors 2021-11-17 15:55:14 -05:00
Aarsh Shah
60fe55cdc0 fix lint and gen 2021-11-17 15:25:25 +04:00
Aarsh Shah
caf3cfa7d8 Merge branch 'feat/cid-to-piece-idx' into feat/indexer-provider 2021-11-17 11:43:49 +04:00
Aarsh Shah
bfee6699d2 update go mod and inject store the hash index in dagstore 2021-11-17 11:42:14 +04:00
Dirk McCormick
627d4fbf74 wip: storage miner index provider 2021-11-16 17:45:42 +01:00
Aarsh Shah
189f5f8aae fix deals test 2021-11-12 09:43:42 +01:00
Dirk McCormick
e309293a47 fix: lint fixes 2021-11-12 09:43:42 +01:00
Aarsh Shah
4d87f16e95 better comparision in test 2021-11-12 09:43:42 +01:00
Aarsh Shah
053317e334 test for retrieval by any cid 2021-11-12 09:43:42 +01:00
Dirk McCormick
662c51605f feat: retrieve from any cid (not just root cid) 2021-11-12 09:43:42 +01:00
Łukasz Magiera
19f52b4541 inline-gen: Slightly nicer comment format 2021-10-18 17:21:53 +02:00
Łukasz Magiera
9606dce425 Add inline-gen to itests 2021-10-18 17:17:54 +02:00
Anton Evangelatov
a19dbb8c91 remove log 2021-10-11 14:03:56 +02:00
Anton Evangelatov
6a3bb184b7 more tests for deals when provider has no funds in control addr 2021-10-11 14:03:56 +02:00
Anton Evangelatov
1acb6b4328 update CLI command to take arg; reduce time for itest 2021-10-11 14:03:56 +02:00
Anton Evangelatov
6e5ccc87cf cli: add retry for deals stuck in Publish with no funds 2021-10-11 14:03:55 +02:00
Peter Rabbitson
f8ac98bf46 Adjust error wording 2021-10-05 18:16:14 +02:00
Peter Rabbitson
5e6aceeb0e Merge remote-tracking branch 'origin/master' into feat/datamodel-selector-retrieval 2021-10-05 18:15:51 +02:00
Jiaying Wang
cbb147de23
Merge pull request #7405 from filecoin-project/feat/update-graphsync-0.10.0 
Update to graphsync to v0.10.0, enable seperate storage and retrieval transfer limits
 2021-10-05 11:50:59 -04:00
hannahhoward
368d72ebfe
feat(graphsync): update to v0.10.0-rc1 
also add config changes
 2021-10-05 14:13:58 +11:00
Peter Rabbitson
af0d9b650d Error on corner cases with corresponding negative tests 2021-10-04 23:21:24 +02:00
Peter Rabbitson
2a1094fc17 Merge banch feat/go-fil-markets-v1.13.0 into feat/datamodel-selector-retrieval 2021-10-04 21:56:11 +02:00
Jennifer Wang
38e01cba83 Merge branch 'release/v1.12.0' into jen/12tomaster 2021-10-04 02:38:00 -04:00
Aayush Rajasekaran
f004d036dc Set BatchPreCommitAboveBaseFee correctly 2021-10-01 14:23:41 -04:00
Łukasz Magiera
8e0314b3a3
Merge pull request #7362 from filecoin-project/chore/update-libp2p 
Update go-libp2p to v0.15.0
 2021-09-23 14:24:12 +01:00
Aayush Rajasekaran
b9bfcc4b69 Randomness: Move getters from ChainAPI to StateAPI 2021-09-21 14:20:15 -04:00
Aarsh Shah
89ac8b09b5 update libp2p 2021-09-21 15:10:04 +04:00
Aayush Rajasekaran
1d6db306ec Randomness: Move getters from ChainAPI to StateAPI 2021-09-15 00:03:13 +02:00
Peter Rabbitson
0444435589 Expose basic text-based datamodel selector on retrieval 
Syntaxt of selection is located at
https://pkg.go.dev/github.com/ipld/go-ipld-selector-text-lite#SelectorSpecFromPath

Example use, assuming that:
  - The root of the deal is a plain dag-pb unixfs directory
  - The directory is not sharded
  - The user wants to retrieve the first entry in that directory

  lotus client retrieve --miner f0XXXXX --datamodel-path-selector 'Links/0/Hash' bafyROOTCID ~/output

For a much more elaborate example see the top of ./itests/deals_partial_retrieval_test.go
 2021-09-10 09:44:11 +02:00
Anton Evangelatov
567d0546ec itests: remove equality comparison for cids 2021-09-07 10:57:10 +02:00
Anton Evangelatov
992cc3ffbf itests: MaxStagingDealsBytes integration test 2021-09-06 17:39:35 +02:00
Łukasz Magiera
95b128b7bc chain: Cleanup consensus logic 2021-09-02 18:09:37 +02:00
Steven Allen
003eae81ce fix: address review 2021-08-30 16:43:21 -07:00
Steven Allen
3846170302 refactor events system 2021-08-30 16:43:21 -07:00
Steven Allen
43bbde1e6b fix: close chain head subscription when the reader is slow 
The reader can just re-subscribe when they're ready to catch up. This
prevents a slow reader from bogging down the entire system.
 2021-08-30 16:43:21 -07:00
Dirk McCormick
7b7a5b0b21 revert: changes to OnDealExpiredOrChanged in #5431 #7201 2021-08-30 10:42:41 +02:00
Dirk McCormick
07dcb074f8 fix: catch deal slashed because sector was terminated 2021-08-27 12:56:23 +02:00
dirkmc
77a19774cf fix events API timeout handling for nil blocks (#7184) 2021-08-27 09:05:00 +02:00
Steven Allen
1105c24e8c test: disable flaky TestBatchDealInput 
See #4611.
 2021-08-24 15:37:52 -07:00
Steven Allen
b509f1f098
Merge pull request #7148 from filecoin-project/raulk/test-large-deals 
itests: support larger sector sizes; add large deal test.
 2021-08-24 12:21:11 -07:00
Steven Allen
2a4ad207a5 test: disable flaky TestSimultaneousTransferLimit 
See https://github.com/filecoin-project/lotus/issues/7152 for details.
 2021-08-20 14:59:01 -07:00
Raúl Kripalani
342134f853 EnableLargeSectors: restore previous supported proof types. 2021-08-20 18:07:25 +01:00
Raúl Kripalani
0030e208a0 avoid reordering imports. 2021-08-20 15:59:19 +01:00
Raúl Kripalani
2dde022a00 itests: support larger sector sizes; add large deal test. 2021-08-20 15:53:24 +01:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making (#6671) 
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
 2021-08-16 23:34:32 +01:00
ZenGround0
ce58b119bf
Merge pull request #7038 from filecoin-project/spike/restrict-api-versions 
Strict major minor version checking on v0 and v1 apis
 2021-08-12 14:34:31 -04:00
ZenGround0
2d8c9bba99 more fix 2021-08-12 14:03:19 -04:00
ZenGround0
c8720fef02 Fix tests 2021-08-12 14:01:24 -04:00
Steven Allen
dcff06bd91 test: genesis actors version 
Properly handle genesis in actors tests. Fast-forward upgrading to
actors v13 doesn't work because there needs to be at least a day between
v0 genesis and v13 (due to differences in miner cron).
 2021-08-11 13:30:15 -07:00
Steven Allen
b885d69aaa chore: disable flaky test 2021-07-30 09:01:31 -07:00
ZenGround0
823bd54484 Review 2021-07-29 20:48:27 -07:00
ZenGround0
756e5ac0b8 Add a separate node to ensemble and check propagation reached 2021-07-29 20:48:27 -07:00
ZenGround0
7c99169fb4 Fix 2021-07-29 20:48:27 -07:00
ZenGround0
2380d252e1 Post rebase fix 2021-07-29 20:48:27 -07:00
ZenGround0
f2f6c209ea Integration test forcing max batch size 2021-07-29 20:48:27 -07:00
Peter Rabbitson
39325e5463
Merge branch 'master' into feat/allow_padding_redux 2021-07-26 20:53:27 +02:00
Peter Rabbitson
8d873e3edc Tests for online and offline dealpadding 
Also bump the times on several flaky tests that can not complete in time
on a typical laptop ( and fail half the time on CircleCI )
 2021-07-24 17:46:04 +02:00
Raúl Kripalani
8693df4656 fix racy TestSimultanenousTransferLimit. 2021-07-24 12:53:56 +01:00
Peter Rabbitson
2edf7fd25b Rewire itest's StartDeal to take the full API struct 
This allows one to use the harness for much more versatile deal conditions
 2021-07-24 04:17:42 +02:00
raulk
4ffbb03a82
Merge branch 'master' into raulk/merge-gs-fix 2021-07-22 23:37:17 +01:00
Łukasz Magiera
660829703a Merge remote-tracking branch 'origin/master' into feat/split-net-api 2021-07-22 15:38:06 +02:00
Łukasz Magiera
0236f2ec5d
Merge pull request #6802 from filecoin-project/fix/6786-rest 
fix: always check if StateSearchMessage returns nil
 2021-07-21 15:05:57 +02:00
Steven Allen
38919f59ad test: fix flaky window post tests 
Wait until the network upgrade is finished. If we try to seal while it's
happening, we have a few annoying edge cases that can fail if we try to
submit some messages right on the upgrade epoch (which is why everyone
turns that kind of stuff off for the upgrade epoch).
 2021-07-20 16:31:14 -07:00
Steven Allen
f9c759fcb2 fix: always check if StateSearchMessage returns nil 
It returns nil on "not found".

Fixes the cases not covered in:
https://github.com/filecoin-project/lotus/pull/6787
 2021-07-20 11:28:27 -07:00
Raúl Kripalani
5e66960ab6 fix merge error. 2021-07-20 17:51:09 +01:00
raulk
353ccf0c55
Merge branch 'raulk/merge-gs-fix' into raulk/merge-gs-fix-simultaneous-fix 2021-07-20 17:37:51 +01:00
Raúl Kripalani
c1b304b633 update graphsync. 2021-07-20 15:43:44 +01:00
hannahhoward
3b4df5e10d fix(itests): fix simultaneous transfer logic 
limit parallelism on client side and also fix the issue with how ongoing transfers are detected
 2021-07-16 11:46:03 -07:00
Raúl Kripalani
089bcceb33 remove test concurrency guardrails. 2021-07-16 19:09:56 +01:00
Raúl Kripalani
9042429abd Merge branch 'master' into raulk/merge-gs-fix 2021-07-16 18:19:50 +01:00
Raúl Kripalani
aeb8e57eef use https://github.com/ipfs/go-graphsync/pull/176. 2021-07-16 18:17:34 +01:00
Raúl Kripalani
b2feb7e960 upgrade go-fil-markets, go-data-transfer; add logging. 2021-07-16 17:30:45 +01:00
Łukasz Magiera
49e26cce7d api: Separate the Net interface from Common 2021-07-15 11:41:30 +02:00
hunjixin
b2f8b4011a fix imports 2021-07-15 14:52:12 +08:00
hunjixin
3e9ec01d04 rm fmt 2021-07-15 14:52:12 +08:00
hunjixin
a48a6a168a add test for new api 2021-07-15 14:52:12 +08:00
hunjixin
ac03c01e4a test for messages in tipset 2021-07-15 14:52:12 +08:00
Łukasz Magiera
583a8a13d9
Merge pull request #6629 from filecoin-project/feat/pledge-from-miner-balance 
Config for collateral from miner available balance
 2021-07-13 17:22:00 +02:00
Łukasz Magiera
7526a074d9 sealing: collateral buffer / falback config 2021-07-13 17:04:58 +02:00
Łukasz Magiera
83f2368507 Add CollateralFromMinerBalance config 2021-07-12 14:11:58 +02:00
Anton Evangelatov
4bc9fa04ba rename cfg.Subsystems.EnableStorageMarket to EnableMarkets 2021-07-12 12:12:29 +02:00
Anton Evangelatov
d89ddb9315 resolve conflicts 2021-07-12 11:34:37 +02:00
Raúl Kripalani
71ddf73f44 check if merging ipfs/go-graphsync#177 fixes deal concurrency. 2021-07-10 10:48:18 +01:00
Łukasz Magiera
9bdea91a70 itests: Some kit cleanup 2021-07-07 21:40:44 +02:00
Łukasz Magiera
d0e87a7487 dealpublisher: Fix provider address checks 2021-07-07 20:06:49 +02:00
Łukasz Magiera
2dc27d6ab4 itests: Fix deal provider collateral flakiness 2021-07-07 19:41:46 +02:00
Łukasz Magiera
ee97aa0a8a dealpublisher: Use address selector 2021-07-07 19:12:03 +02:00
Anton Evangelatov
6b014f57e5 pass Subsystems to StorageMiner option; add enableLibp2p bool in Settings 2021-07-07 13:56:37 +02:00
Anton Evangelatov
caa366bc6a variadic WithSubsystems 2021-07-07 13:35:40 +02:00
Anton Evangelatov
566a9d5541 fix signature 2021-07-06 18:12:30 +02:00
Anton Evangelatov
2a7af38538 fix testServ to Start with correct listener 2021-07-06 17:57:08 +02:00
Anton Evangelatov
7a78527fde remove not used sa; cleanup 2021-07-06 17:46:21 +02:00
Anton Evangelatov
8cf3accc53
Update itests/kit/ensemble.go 
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
 2021-07-06 17:25:33 +02:00
Anton Evangelatov
9839d5701b fix SectorsStatus to query full node for on-chain data 2021-07-06 15:41:14 +02:00
Anton Evangelatov
5a87f04e71 fix finalize sectors early test 2021-07-05 13:36:23 +02:00
Anton Evangelatov
4be0a7a215 resolve conflicts with master 2021-07-05 13:13:32 +02:00
Łukasz Magiera
13d4563c25
Merge pull request #6663 from filecoin-project/raulk/test-incremental-nonce 
add an incremental nonce itest.
 2021-07-02 23:33:52 +02:00
Łukasz Magiera
8a94ab676e storage: Fix FinalizeSector with sectors in stoage paths 2021-07-02 19:54:45 +02:00
Raúl Kripalani
592b8c4f4a add an incremental nonce itest. 2021-07-02 17:04:13 +01:00
Anton Evangelatov
724f728f60 enable more tests 2021-07-02 14:54:09 +02:00
Anton Evangelatov
9e85492b85 fix test 2021-07-02 12:24:07 +02:00
Anton Evangelatov
3aa6d03cc1 connect miner to full node , only when running markets 2021-06-30 16:45:11 +02:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
Anton Evangelatov
9ec7cd7239 enable libp2p options 2021-06-30 12:55:43 +02:00
Łukasz Magiera
e9dd3e8650 Test Miner SimultaneousTransfers 2021-06-28 18:17:29 +02:00
Raúl Kripalani
6a48fbbc11 increase suspension threshold to 300. 2021-06-23 19:21:42 +01:00
Raúl Kripalani
58f348cb7f add ability to suspend deal-making until CE is stable. 2021-06-23 19:14:27 +01:00
Raúl Kripalani
9b3188d110 split deals tests. 2021-06-23 18:13:29 +01:00
Raúl Kripalani
f4d0db83d8 Merge branch 'raulk/circleci-dynamic' into raulk/itests-refactor-kit 2021-06-23 17:33:17 +01:00
Raúl Kripalani
211751f8b8 declare some tests as expensive. 2021-06-23 16:37:43 +01:00
Anton Evangelatov
a8d5ae5213 enable more tests 2021-06-23 13:39:03 +02:00