Commit Graph

55 Commits

Author SHA1 Message Date
Jorropo
6c01310728
chore: migrate to boxo
This migrates everything except the `go-car` librairy: https://github.com/ipfs/boxo/issues/218#issuecomment-1529922103

I didn't migrated everything in the previous release because all the boxo code wasn't compatible with the go-ipld-prime one due to a an in flight (/ aftermath) revert of github.com/ipfs/go-block-format. go-block-format has been unmigrated since slight bellow absolutely everything depends on it that would have required everything to be moved on boxo or everything to optin into using boxo which were all deal breakers for different groups.

This worked fine because lotus's codebase could live hapely on the first multirepo setup however boost is now trying to use boxo's code with lotus's (still on multirepo) setup: https://filecoinproject.slack.com/archives/C03AQ3QAUG1/p1685022344779649

The alternative would be for boost to write shim types which just forward calls and return with the different interface definitions.

Btw why is that an issue in the first place is because unlike what go's duck typing model suggest interfaces are not transparent https://github.com/golang/go/issues/58112, interfaces are strongly typed but they have implicit narrowing. The issue is if you return an interface from an interface Go does not have a function definition to insert the implicit conversion thus instead the type checker complains you are not returning the right type.

Stubbing types were reverted https://github.com/ipfs/boxo/issues/218#issuecomment-1478650351

Last time I only migrated `go-bitswap` to `boxo/bitswap` because of the security issues and because we never had the interface return an interface problem (we had concrete wrappers where the implicit conversion took place).
2023-06-19 14:45:05 -07:00
Jorropo
fe42d974a2 chore: all: migrate from go-libipfs to boxo
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Marten Seemann
db6efe6512 use go-libipfs/files instead of ipfs/go-ipfs-files 2023-03-08 17:50:56 +13:00
Łukasz Magiera
888f97a35f netbs: Add an integration test 2022-11-08 13:38:28 +01:00
Łukasz Magiera
d1173265b3 itests: Helper returning non-genesis sector list 2022-08-02 15:29:38 +02:00
Aayush
0c34699f23 unbreak itests 2022-06-29 14:55:35 -04:00
Łukasz Magiera
98a48a47f8 storage: Move extern/storage-sealing to storage/pipeline 2022-06-14 19:41:59 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Masih H. Derkani
575e6add43
Rename IndexerProvider to IndexProvider and add more docs
Rename the config section corresponding to indexing to `IndexProvider`
for better readability.

Update existing docs for better clarity and add docs for config
parameters embedded from `index-provider` `Ingest` config library.
2022-02-03 13:10:47 +00:00
Aarsh Shah
b9d3585f4c get tests working 2022-01-12 16:55:45 +04:00
Aarsh Shah
3a52f05692 green ci 2022-01-12 15:53:15 +04:00
Aarsh Shah
edfc2c215c Merge remote-tracking branch 'origin/main' into feat/cid-to-piece-idx 2022-01-12 12:48:54 +04:00
zenground0
33f2d24f54 Snap Deals Integration
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-10 15:39:38 +05:30
vyzo
fc7d4a13b0 fix itests 2021-12-14 16:06:59 +02:00
Łukasz Magiera
597b72e286 retrieval: Fix lint, cli docsgen 2021-11-22 12:46:29 +01:00
Łukasz Magiera
89138bab4d Simplify retrieval APIs 2021-11-22 12:45:42 +01:00
Dirk McCormick
627d4fbf74 wip: storage miner index provider 2021-11-16 17:45:42 +01:00
Aarsh Shah
189f5f8aae fix deals test 2021-11-12 09:43:42 +01:00
Aarsh Shah
053317e334 test for retrieval by any cid 2021-11-12 09:43:42 +01:00
Dirk McCormick
662c51605f feat: retrieve from any cid (not just root cid) 2021-11-12 09:43:42 +01:00
Anton Evangelatov
a19dbb8c91 remove log 2021-10-11 14:03:56 +02:00
Anton Evangelatov
992cc3ffbf itests: MaxStagingDealsBytes integration test 2021-09-06 17:39:35 +02:00
Dirk McCormick
7b7a5b0b21 revert: changes to OnDealExpiredOrChanged in 2021-08-30 10:42:41 +02:00
dirkmc
77a19774cf fix events API timeout handling for nil blocks () 2021-08-27 09:05:00 +02:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ()
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Peter Rabbitson
2edf7fd25b Rewire itest's StartDeal to take the full API struct
This allows one to use the harness for much more versatile deal conditions
2021-07-24 04:17:42 +02:00
Raúl Kripalani
9042429abd Merge branch 'master' into raulk/merge-gs-fix 2021-07-16 18:19:50 +01:00
Raúl Kripalani
b2feb7e960 upgrade go-fil-markets, go-data-transfer; add logging. 2021-07-16 17:30:45 +01:00
Anton Evangelatov
4f0a96c9c7 resolved conflicts 2021-06-30 13:16:52 +02:00
Raúl Kripalani
6a48fbbc11 increase suspension threshold to 300. 2021-06-23 19:21:42 +01:00
Raúl Kripalani
58f348cb7f add ability to suspend deal-making until CE is stable. 2021-06-23 19:14:27 +01:00
Raúl Kripalani
9b3188d110 split deals tests. 2021-06-23 18:13:29 +01:00
Anton Evangelatov
7fb93d4585 refactor kit.NewDealHarness to include main miner and market node 2021-06-23 11:24:55 +02:00
Raúl Kripalani
83ee345b1c Merge branch 'master' into raulk/itests-refactor-kit 2021-06-21 18:19:26 +01:00
Raúl Kripalani
8a418bf982 rename {kit2=>kit}. 2021-06-18 19:45:29 +01:00
Raúl Kripalani
a7d8d15c13 kill old kit 🎉 2021-06-18 19:42:06 +01:00
aarshkshah1992
3d086dfb43 changes as per review 2021-06-14 09:40:34 +05:30
Raúl Kripalani
e84b8ab3a0 move new kit into kit2, re-enable unmigrated tests against kit1. 2021-06-11 18:26:25 +01:00
aarshkshah1992
fed5afa704 merge master 2021-06-11 09:35:20 +05:30
Raúl Kripalani
8b037e2da3 deals tests: migrate deals cycles tests and add coverage. 2021-06-10 18:25:02 +01:00
Raúl Kripalani
dcd6fc239b deals tests: migrate TestOfflineDealFlow. 2021-06-10 15:54:16 +01:00
Raúl Kripalani
cf0150e057 deals harness: use require. 2021-06-10 12:23:27 +01:00
Raúl Kripalani
c27fdc263c deals harness: more improvements. 2021-06-10 12:22:55 +01:00
Raúl Kripalani
0d69c03a8d deals harness: use require. 2021-06-10 12:17:39 +01:00
Raúl Kripalani
0303a0297d rename DealHarness.{TestRetrieval=>PerformRetrieval}. 2021-06-10 12:15:03 +01:00
Raúl Kripalani
bcbbd73d65 Merge branch 'raulk/itests' into raulk/itests-refactor-kit 2021-06-10 12:02:08 +01:00
Raúl Kripalani
6e4eae69ac fix merge error that led to test failures. 2021-06-09 23:11:34 +01:00
Raúl Kripalani
a274fcc533 Merge branch 'master' into raulk/itests 2021-06-07 23:44:22 +01:00
Raúl Kripalani
db1a61852d Merge branch 'master' into raulk/itests 2021-06-07 23:17:44 +01:00
Raúl Kripalani
2bcedcf55f initial version of the new itest kit.
Still need to migrate all integration tests, add godocs,
and probably zap bugs.
2021-05-26 00:04:13 +01:00