Commit Graph

216 Commits

Author SHA1 Message Date
Jorropo
6c01310728
chore: migrate to boxo
This migrates everything except the `go-car` librairy: https://github.com/ipfs/boxo/issues/218#issuecomment-1529922103

I didn't migrated everything in the previous release because all the boxo code wasn't compatible with the go-ipld-prime one due to a an in flight (/ aftermath) revert of github.com/ipfs/go-block-format. go-block-format has been unmigrated since slight bellow absolutely everything depends on it that would have required everything to be moved on boxo or everything to optin into using boxo which were all deal breakers for different groups.

This worked fine because lotus's codebase could live hapely on the first multirepo setup however boost is now trying to use boxo's code with lotus's (still on multirepo) setup: https://filecoinproject.slack.com/archives/C03AQ3QAUG1/p1685022344779649

The alternative would be for boost to write shim types which just forward calls and return with the different interface definitions.

Btw why is that an issue in the first place is because unlike what go's duck typing model suggest interfaces are not transparent https://github.com/golang/go/issues/58112, interfaces are strongly typed but they have implicit narrowing. The issue is if you return an interface from an interface Go does not have a function definition to insert the implicit conversion thus instead the type checker complains you are not returning the right type.

Stubbing types were reverted https://github.com/ipfs/boxo/issues/218#issuecomment-1478650351

Last time I only migrated `go-bitswap` to `boxo/bitswap` because of the security issues and because we never had the interface return an interface problem (we had concrete wrappers where the implicit conversion took place).
2023-06-19 14:45:05 -07:00
Jorropo
fe42d974a2 chore: all: migrate from go-libipfs to boxo
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Jorropo
f572852d06 chore: all: bump go-libipfs to replace go-block-format
Includes changes from:
- https://github.com/ipfs/go-block-format/pull/37
- https://github.com/ipfs/go-libipfs/pull/58
2023-01-26 17:03:18 +01:00
Aayush
c0b7343e60 Merge branch 'release/v1.18.0' into asr/merge-release-into-master 2022-11-06 14:40:13 -05:00
Geoff Stuart
b4c04ad927 update markets 2022-10-06 11:06:21 -04:00
Aayush
54a4f14d09 feat: dealpublisher: check for duplicate deals before adding 2022-09-23 12:23:42 -04:00
Aayush
4382bbcc96 feat: add logs to markets event handler 2022-09-16 15:36:08 -04:00
Aayush
19c3232d2f more fixes 2022-09-10 12:37:03 -04:00
Aayush
0efca4d266 introduce v9 actors and nv17 2022-09-09 21:40:15 -04:00
Łukasz Magiera
08b22edd89 fix make gen 2022-08-29 16:25:30 +02:00
Łukasz Magiera
ae17b765c0 storage: Address review, no tok 2022-06-17 18:25:59 +02:00
Łukasz Magiera
a084445e6c sealing pipeline: Use non-special ChainHead 2022-06-16 13:15:49 +02:00
Łukasz Magiera
9aa5659d24 sealing pipeline: Drop TipSetToken, use TipSetKey directly 2022-06-16 12:05:58 +02:00
Łukasz Magiera
39cede946c chore: Fix more imports with '2' suffix 2022-06-14 21:37:54 +02:00
Łukasz Magiera
98a48a47f8 storage: Move extern/storage-sealing to storage/pipeline 2022-06-14 19:41:59 +02:00
Łukasz Magiera
717bda63b1 storage: Move control address logic to storage/ctladdr 2022-06-14 19:32:29 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Geoff Stuart
b7010c9e60 Implement function to migrate actors with only code changes 2022-06-10 15:52:32 -04:00
Jennifer Wang
2e59d0129d Merge branch 'release/v1.15.3' into jen/mergev1153to16 2022-05-31 16:33:18 -04:00
Aayush
8cca9b1970 Use new go-state-types accessors 2022-05-17 15:21:27 -04:00
zenground0
8562798940 Use go-fil-markets tag 2022-04-14 19:32:56 +03:00
zenground0
60d09579be More fixes 2022-04-14 19:32:56 +03:00
Aayush
339b7db2b7 Integrate FIP 0027 2022-04-14 19:32:56 +03:00
zenground0
239274cb1d Properly handle output index 2022-04-12 15:34:45 -04:00
Darko Brdareski
79453663b3
feat: Add additional test annotations ()
* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate repub and selection tests

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

* Fix markset annotations

* doc: test annotations for the markets dagstore wrapper

* Annotate miner_api test in dagstore

* Annotate more test files

* Remove bad annotations from fsrepo

* Annotate wdpost system

* Remove bad annotations

* Renamce "conformance" to "chaos_actor" tests

* doc: stm annotations for blockheader & election proof tests

* Annotate remaining "A" tests

* annotate: stm for error_test

* memrepo_test.go

* Annotate "b" file tests

* message_test.go

* doc: stm annotate for fsrepo_test

* Annotate "c" file tests

* Annotate "D" test files

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate u,v,wl tests

* doc: stm annotations for various test files

* Annotate "T" test files

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add mempool unit tests

* Add two more memPool Add tests

* Update submodules

* Add check function tests

* Add stm annotations, refactor test helper

* Annotate api,proxy_util,blockstore_badger, policy tests

* Annotate splitstore: bsbadger / markset

solving merge conflicts

* Annotate splitstore feature

* Annotate union/timed blockstore tests

* Annotate openrpc, diff_adt tests

* Annotate error,drand,events tests

* Annotate predicates_test

* Fix annotations

* Annotate tscache, gen tests

* Annotate fundmanager test

* Annotate statetree_test

* Annotate forks_test

* Annotate searchwait_test.go

* Fix duplicated @@ symbols

* Annotate chain stmgr/store tests

* Annotate more (types) tests

* More tests annotated

* Annotate conformance chaos actor tests

* Annotate more integration tests

* Annotate journal system tests

* Annotate more tests.

* Annotate gas,head buffer behaviors

solve merge conflict

* Fix markset annotations

* Annotate miner_api test in dagstore

* Annotate more test files

* doc: test annotations for the markets dagstore wrapper

* Annotate wdpost system

* Renamce "conformance" to "chaos_actor" tests

* Annotate remaining "A" tests

* doc: stm annotations for blockheader & election proof tests

* annotate: stm for error_test

* Annotate "b" file tests

* memrepo_test.go

* Annotate "c" file tests

* message_test.go

* Annotate "D" test files

* doc: stm annotate for fsrepo_test

* Annotate u,v,wl tests

* message_test.go

* doc: stm annotate for chain, node/config & client

* docs: stm annotate node_test

* Annotate "T" test files

* doc: stm annotations for various test files

* Add mempool unit tests

solve merge conflict

* doc: stm annotate for proxy_util_test & policy_test

* doc: stm annotate for various tests

* doc: final few stm annotations

* Add two more memPool Add tests

* Update submodules

* Add check function tests

solve conflict

* Add stm annotations, refactor test helper

solve merge conflict

* Change CLI test kinds to "unit"

* Fix double merged test

* Fix ccupgrade_test merge

* Fix lint issues

* Add stm annotation to types_Test

* Test vectors submodule

* Add file annotation to burn_test

Co-authored-by: Nikola Divic <divicnikola@gmail.com>
Co-authored-by: TheMenko <themenkoprojects@gmail.com>
2022-03-16 18:37:34 +01:00
Darko Brdareski
e51ce5c508 Merge remote-tracking branch 'upstream/master' into bloxico/system-test-matrix 2022-01-27 10:57:56 +01:00
zenground0
33f2d24f54 Snap Deals Integration
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-10 15:39:38 +05:30
Darko Brdareski
dda1a42a2a Merge branch 'bloxico/system-test-matrix' of https://github.com/filecoin-project/lotus into merge_lotus 2021-12-20 15:48:16 +01:00
Dirk McCormick
839fc8df7d fix: when retrying Add Piece, first seek to start of reader 2021-12-17 14:44:51 +01:00
Aayush Rajasekaran
dfb65ed89f Plumb contexts through 2021-12-11 17:04:00 -05:00
Darko Brdareski
0169d0dafd Annotate state feature tests 2021-12-10 16:08:25 +01:00
Jennifer Wang
38e01cba83 Merge branch 'release/v1.12.0' into jen/12tomaster 2021-10-04 02:38:00 -04:00
Anton Evangelatov
809289f5ef add Dealmaking.StartEpochSealingBuffer config 2021-10-01 17:44:15 +02:00
Aayush Rajasekaran
84b6734063 Incorporate the new PublishStorageDealsReturn 2021-09-29 19:13:13 -04:00
Łukasz Magiera
42135ec7cc dealpublisher: Test deal verification 2021-09-03 19:40:02 +02:00
Łukasz Magiera
386910589d dealpublisher: Fully validate deals before publishing 2021-09-02 23:11:09 +02:00
Steven Allen
3846170302 refactor events system 2021-08-30 16:43:21 -07:00
Dirk McCormick
7b7a5b0b21 revert: changes to OnDealExpiredOrChanged in 2021-08-30 10:42:41 +02:00
Dirk McCormick
07dcb074f8 fix: catch deal slashed because sector was terminated 2021-08-27 12:56:23 +02:00
dirkmc
77a19774cf fix events API timeout handling for nil blocks () 2021-08-27 09:05:00 +02:00
Dirk McCormick
aa8ac05d6b fix: replace panic with err in OnDealExpiredOrSlashed 2021-08-27 09:02:56 +02:00
Dirk McCormick
c7e7f89026 refactor: OnDealExpiredOrSlashed get deal by proposal instead of deal ID 2021-08-27 09:02:56 +02:00
Steven Allen
add699d238 fix: make sure we start the publish timer before recording the time
Otherwise, our nice and deterministic test may keep repeatedly setting
the time to _right_ before the timer fires.
2021-08-24 11:05:51 -07:00
Steven Allen
16f8a35e76 fix: idiomatic go
1. No need to slice a slice.
2. Idiomatic filter (https://github.com/golang/go/wiki/SliceTricks).
2021-08-24 11:05:05 -07:00
Aarsh Shah
d7076778e2
integrate DAG store and CARv2 in deal-making ()
This commit removes badger from the deal-making processes, and
moves to a new architecture with the dagstore as the cental
component on the miner-side, and CARv2s on the client-side.

Every deal that has been handed off to the sealing subsystem becomes
a shard in the dagstore. Shards are mounted via the LotusMount, which
teaches the dagstore how to load the related piece when serving
retrievals.

When the miner starts the Lotus for the first time with this patch,
we will perform a one-time migration of all active deals into the
dagstore. This is a lightweight process, and it consists simply
of registering the shards in the dagstore.

Shards are backed by the unsealed copy of the piece. This is currently
a CARv1. However, the dagstore keeps CARv2 indices for all pieces, so
when it's time to acquire a shard to serve a retrieval, the unsealed
CARv1 is joined with its index (safeguarded by the dagstore), to form
a read-only blockstore, thus taking the place of the monolithic
badger.

Data transfers have been adjusted to interface directly with CARv2 files.
On inbound transfers (client retrievals, miner storage deals), we stream
the received data into a CARv2 ReadWrite blockstore. On outbound transfers
(client storage deals, miner retrievals), we serve the data off a CARv2
ReadOnly blockstore.

Client-side imports are managed by the refactored *imports.Manager
component (when not using IPFS integration). Just like it before, we use
the go-filestore library to avoid duplicating the data from the original
file in the resulting UnixFS DAG (concretely the leaves). However, the
target of those imports are what we call "ref-CARv2s": CARv2 files placed
under the `$LOTUS_PATH/imports` directory, containing the intermediate
nodes in full, and the leaves as positional references to the original file
on disk.

Client-side retrievals are placed into CARv2 files in the location:
`$LOTUS_PATH/retrievals`.

A new set of `Dagstore*` JSON-RPC operations and `lotus-miner dagstore`
subcommands have been introduced on the miner-side to inspect and manage
the dagstore.

Despite moving to a CARv2-backed system, the IPFS integration has been
respected, and it continues to be possible to make storage deals with data
held in an IPFS node, and to perform retrievals directly into an IPFS node.

NOTE: because the "staging" and "client" Badger blockstores are no longer
used, existing imports on the client will be rendered useless. On startup,
Lotus will enumerate all imports and print WARN statements on the log for
each import that needs to be reimported. These log lines contain these
messages:

- import lacks carv2 path; import will not work; please reimport
- import has missing/broken carv2; please reimport

At the end, we will print a "sanity check completed" message indicating
the count of imports found, and how many were deemed broken.

Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
2021-08-16 23:34:32 +01:00
Steven Allen
be2ecf6236 fix flaky TestDealPublisher and re-enable
fixes 
2021-08-05 19:46:08 -07:00
vyzo
c9930cb7db disable flaky test 2021-07-20 09:49:22 +03:00
Anton Evangelatov
d89ddb9315 resolve conflicts 2021-07-12 11:34:37 +02:00
Łukasz Magiera
d0e87a7487 dealpublisher: Fix provider address checks 2021-07-07 20:06:49 +02:00
Łukasz Magiera
ee97aa0a8a dealpublisher: Use address selector 2021-07-07 19:12:03 +02:00