Commit Graph

363 Commits

Author SHA1 Message Date
Łukasz Magiera
1f2621b574 Make retrieval work with reused channels 2022-01-20 18:16:00 +01:00
Łukasz Magiera
8e46b9ea5d paych: API to pre-fund channels 2022-01-20 18:15:46 +01:00
Aarsh Shah
7508d96327 fix lint 2022-01-20 16:47:15 +04:00
Aarsh Shah
9ee940098d Merge remote-tracking branch 'origin/master' into feat/cid-to-piece-idx 2022-01-20 16:21:48 +04:00
vyzo
ca4ee1e8cf fix issues with fil-markets/data-transfer updates 2022-01-20 11:38:37 +02:00
hannahhoward
92d56d82f4 feat(deps): update markets stack
update go-fil-markets, go-data-transfer 1.13.0, go-graphsync 0.12.0
2022-01-14 17:21:04 -08:00
c r
da6752eccb
feat: sealing: Adding conf variable for capping number of concurrent unsealing jobs ()
* adding the new variables- now time for logic

* putting parameters into right placeS

* adding unsealing throttle

* fixing linter issues

* removing one last thing...
2022-01-13 12:26:13 -06:00
Aarsh Shah
3a52f05692 green ci 2022-01-12 15:53:15 +04:00
Aarsh Shah
edfc2c215c Merge remote-tracking branch 'origin/main' into feat/cid-to-piece-idx 2022-01-12 12:48:54 +04:00
zenground0
33f2d24f54 Snap Deals Integration
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-10 15:39:38 +05:30
Darko Brdareski
dda1a42a2a Merge branch 'bloxico/system-test-matrix' of https://github.com/filecoin-project/lotus into merge_lotus 2021-12-20 15:48:16 +01:00
Dirk McCormick
839fc8df7d fix: when retrying Add Piece, first seek to start of reader 2021-12-17 14:44:51 +01:00
Aayush Rajasekaran
0e2278cc76 Merge branch 'master' into deps/update-ctx-dsbs 2021-12-14 18:08:29 -05:00
Aayush Rajasekaran
dfb65ed89f Plumb contexts through 2021-12-11 17:04:00 -05:00
Darko Brdareski
0169d0dafd Annotate state feature tests 2021-12-10 16:08:25 +01:00
Łukasz Magiera
74d3ba6b5e fix lint 2021-12-09 18:10:34 +01:00
Łukasz Magiera
13b260e7f7 piecereader: Move closer to storage 2021-12-08 23:20:20 +01:00
Łukasz Magiera
9f6265e0b4 dagstore pieceReader: Fix wrong ErrUnexpectedEOF return in ReadAt 2021-12-03 23:36:36 +01:00
Łukasz Magiera
3969d6b767 dagstore pieceReader: Always read full in ReadAt 2021-12-03 20:14:35 +01:00
Łukasz Magiera
dad9190142 Add metrics to dagstore piecereader 2021-12-03 17:07:14 +01:00
Łukasz Magiera
4bcde2f0ff dagstore pieceReader: Cleanup reader nil check 2021-11-29 15:32:27 +01:00
Łukasz Magiera
05aa860459 Request correct read size with startOffset in pieceProvider 2021-11-27 00:05:45 +01:00
Łukasz Magiera
9110e6f632 dagstore pieceReader: add debug log on stream restart 2021-11-26 20:24:51 +01:00
Łukasz Magiera
331702cd95 Tweak MaxPieceReaderBurnBytes 2021-11-26 18:49:41 +01:00
Łukasz Magiera
abec7dd3bc Fix imports 2021-11-26 18:40:27 +01:00
Łukasz Magiera
a9ee263682 Fix dagstore pieceReader burn logic 2021-11-26 18:01:09 +01:00
Łukasz Magiera
8d955d5f30 dagstore mount: Add random access support 2021-11-26 17:40:53 +01:00
Aarsh Shah
7655e660f3 integrate store-the-hash 2021-11-18 12:07:12 +04:00
Aarsh Shah
d196c236b8 fix lint 2021-11-17 14:55:09 +04:00
Aarsh Shah
f8c675bc5e green tests 2021-11-17 14:48:28 +04:00
Aarsh Shah
caf3cfa7d8 Merge branch 'feat/cid-to-piece-idx' into feat/indexer-provider 2021-11-17 11:43:49 +04:00
Aarsh Shah
bfee6699d2 update go mod and inject store the hash index in dagstore 2021-11-17 11:42:14 +04:00
Dirk McCormick
627d4fbf74 wip: storage miner index provider 2021-11-16 17:45:42 +01:00
Dirk McCormick
3c37078a85 fix: dagstore wrapper test 2021-11-12 09:43:42 +01:00
Dirk McCormick
662c51605f feat: retrieve from any cid (not just root cid) 2021-11-12 09:43:42 +01:00
Peter Rabbitson
5e6aceeb0e Merge remote-tracking branch 'origin/master' into feat/datamodel-selector-retrieval 2021-10-05 18:15:51 +02:00
Peter Rabbitson
2a1094fc17 Merge banch feat/go-fil-markets-v1.13.0 into feat/datamodel-selector-retrieval 2021-10-04 21:56:11 +02:00
Jennifer Wang
38e01cba83 Merge branch 'release/v1.12.0' into jen/12tomaster 2021-10-04 02:38:00 -04:00
Anton Evangelatov
809289f5ef add Dealmaking.StartEpochSealingBuffer config 2021-10-01 17:44:15 +02:00
Aayush Rajasekaran
84b6734063 Incorporate the new PublishStorageDealsReturn 2021-09-29 19:13:13 -04:00
Peter Rabbitson
0444435589 Expose basic text-based datamodel selector on retrieval
Syntaxt of selection is located at
https://pkg.go.dev/github.com/ipld/go-ipld-selector-text-lite#SelectorSpecFromPath

Example use, assuming that:
  - The root of the deal is a plain dag-pb unixfs directory
  - The directory is not sharded
  - The user wants to retrieve the first entry in that directory

  lotus client retrieve --miner f0XXXXX --datamodel-path-selector 'Links/0/Hash' bafyROOTCID ~/output

For a much more elaborate example see the top of ./itests/deals_partial_retrieval_test.go
2021-09-10 09:44:11 +02:00
Łukasz Magiera
42135ec7cc dealpublisher: Test deal verification 2021-09-03 19:40:02 +02:00
Łukasz Magiera
386910589d dealpublisher: Fully validate deals before publishing 2021-09-02 23:11:09 +02:00
Steven Allen
3846170302 refactor events system 2021-08-30 16:43:21 -07:00
Dirk McCormick
7b7a5b0b21 revert: changes to OnDealExpiredOrChanged in 2021-08-30 10:42:41 +02:00
Dirk McCormick
07dcb074f8 fix: catch deal slashed because sector was terminated 2021-08-27 12:56:23 +02:00
dirkmc
77a19774cf fix events API timeout handling for nil blocks () 2021-08-27 09:05:00 +02:00
Dirk McCormick
aa8ac05d6b fix: replace panic with err in OnDealExpiredOrSlashed 2021-08-27 09:02:56 +02:00
Dirk McCormick
c7e7f89026 refactor: OnDealExpiredOrSlashed get deal by proposal instead of deal ID 2021-08-27 09:02:56 +02:00
Steven Allen
add699d238 fix: make sure we start the publish timer before recording the time
Otherwise, our nice and deterministic test may keep repeatedly setting
the time to _right_ before the timer fires.
2021-08-24 11:05:51 -07:00
Steven Allen
16f8a35e76 fix: idiomatic go
1. No need to slice a slice.
2. Idiomatic filter (https://github.com/golang/go/wiki/SliceTricks).
2021-08-24 11:05:05 -07:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ()
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Steven Allen
be2ecf6236 fix flaky TestDealPublisher and re-enable
fixes 
2021-08-05 19:46:08 -07:00
hannahhoward
8d6bff36f2 fix(marketevents): use ReceivedCidsLen
improve data transfer logger by not reading in entire cid lists
2021-07-29 01:12:13 -04:00
Anton Evangelatov
9ad408a0ba add SealProof in sector builder 2021-07-21 16:45:57 +02:00
vyzo
c9930cb7db disable flaky test 2021-07-20 09:49:22 +03:00
Anton Evangelatov
d89ddb9315 resolve conflicts 2021-07-12 11:34:37 +02:00
Łukasz Magiera
d0e87a7487 dealpublisher: Fix provider address checks 2021-07-07 20:06:49 +02:00
Łukasz Magiera
ee97aa0a8a dealpublisher: Use address selector 2021-07-07 19:12:03 +02:00
Anton Evangelatov
b572c2b7bc fix IsUnsealed 2021-07-06 17:40:03 +02:00
Anton Evangelatov
10dc90f7fe addressing comments 2021-07-06 17:33:47 +02:00
Anton Evangelatov
ff2772a58c resolved conflicts 2021-07-06 16:00:41 +02:00
Anton Evangelatov
9839d5701b fix SectorsStatus to query full node for on-chain data 2021-07-06 15:41:14 +02:00
aarshkshah1992
add113f585 use multierror 2021-07-05 16:08:51 +05:30
aarshkshah1992
44985722d2 changes as per review 2021-07-05 15:53:41 +05:30
aarshkshah1992
8ecdc929b4 get retrieval pricing input should not error out on a deal state fetch 2021-07-05 09:34:56 +05:30
Anton Evangelatov
5e10d53ca8 wip 2021-06-22 12:16:21 +02:00
Anton Evangelatov
6720463799 resolve merge conflicts 2021-06-22 11:28:23 +02:00
Anton Evangelatov
c0feb36e70 add missing debug 2021-06-14 11:31:55 +02:00
aarshkshah1992
3d086dfb43 changes as per review 2021-06-14 09:40:34 +05:30
Anton Evangelatov
9f3ec82cc1 initial resolution of conflicts 2021-06-08 13:01:22 +02:00
aarshkshah1992
16166504f4 merged master 2021-06-08 09:17:40 +05:30
aarshkshah1992
670835fca0 bypass task scheduler for reading unsealed pieces 2021-06-07 15:02:04 +05:30
Anton Evangelatov
8bd09e39ca resolve merge conflicts 2021-06-04 16:17:00 +02:00
Anton Evangelatov
7f234809fc make linter happy 2021-06-04 15:44:13 +02:00
Anton Evangelatov
88756f3ebf fix TestAPIDeal tests 2021-06-04 15:41:38 +02:00
Łukasz Magiera
1e4456138e Merge master into feat/nv13 2021-05-27 12:28:20 +02:00
Łukasz Magiera
f5409845b5 Some review addressing 2021-05-25 16:07:45 +02:00
aarshkshah1992
dc6dbc9a11 dpr changes and test based on new unsealing PR 2021-05-22 22:40:21 +05:30
aarshkshah1992
2a40c802ea bypass task scheduler for reading unsealed pieces 2021-05-20 15:25:46 -06:00
Anton Evangelatov
cb603c62d9 update retrievaladapter ; add piece_provider 2021-05-20 12:49:53 +02:00
Łukasz Magiera
e088c71b9a marketadapter: Handle batch sealing messages 2021-05-19 20:07:20 +02:00
Anton Evangelatov
a989f60e27 add SectorAddPieceToAny and SectorUnsealPiece to StorageMiner iface; model moved to api package - PieceDealInfo, DealSchedule 2021-05-19 13:05:07 +02:00
Dirk McCormick
61344644a4 feat: add more debug logging for unsealing 2021-04-22 11:33:28 +02:00
Dirk McCormick
47145b6b82 fix: adjust client deal collateral overestimation to 1.2 2021-04-15 15:51:31 +02:00
Łukasz Magiera
81bd27911f Propagate StateMsg api changes 2021-04-05 19:56:53 +02:00
Łukasz Magiera
deb2b90b6a Fix lotus/miner build 2021-04-05 13:23:46 +02:00
dirkmc
14ede9b0e6
Fix unseal comment 2021-03-22 14:39:03 +01:00
Dirk McCormick
73d5225f3c fix: better logging when unsealing fails 2021-03-22 10:23:58 +01:00
Łukasz Magiera
d11f2def6e Merge branch 'feat/max-deal-collateral-multiplier' into next 2021-03-10 11:13:15 +01:00
Dirk McCormick
ced455bba9 feat: add configurable max provider collateral 2021-03-02 10:24:57 +01:00
Raúl Kripalani
7f0f7d0b36 Merge branch 'master' into refactor/lib/blockstore 2021-02-28 19:55:23 +00:00
Łukasz Magiera
0c6aef221d Fix error logging format strings 2021-02-11 12:00:26 +01:00
Dirk McCormick
af45b299e7 fix: flaky TestForcePublish 2021-02-08 12:05:44 +01:00
Łukasz Magiera
3ff6a6f59f address review; flush tablewriter 2021-02-05 22:33:53 +01:00
Łukasz Magiera
1b65f48808 Fix build 2021-02-05 21:58:36 +01:00
Łukasz Magiera
b3f4e50c58 market: APIs to manage pending deals 2021-02-05 18:58:55 +01:00
Dirk McCormick
01e30e0665 test: add test for pending publish deals / force publish 2021-02-05 17:09:57 +01:00
Dirk McCormick
4676fd6753 feat: add methods to list pending deals and force publish 2021-02-05 15:56:50 +01:00
Łukasz Magiera
b938ac4cc8
Merge pull request from filecoin-project/refactor/mkts-wait-publish
Move waiting for publish deals from markets into lotus
2021-02-05 10:59:18 +01:00