Now that host is shared having separate config for listen and announce
addresses along with graphsync limit makes no sense since all of that is
shared with the markets host and datatransfer manager.
Remove the bespoke instantiation of libp2p host and datatransfer manager
for index provider and reuse the existing instances used by markets.
The rationale for reuse is the following:
1. Separation of host introduces a discovery problem, where without
gossipsub the index provider endpoint will not be discoverable.
Using the same host as markets would mean the chain can be used to
discover addresses, putting less empassis on criticality of
gossipsub considering its set-up cost and lack of message delivery
guarantees.
2. Only a single instance of graphsync/datatransfer can be instantiated
per libp2p host; therefore, if the host is shared, so should
datatransfer manager.
3. it is not clear if the assumptions under which separation was
decided still hold.
Add the peer ID of index provider host to the list of protected peers
before connecting to full node. Otherwise, it is possible for the
connection to be reset by full node before we reach the line that adds
the ID to list of protected peers via JsonRPC API.
Relates to:
- https://github.com/filecoin-project/index-provider/issues/177
The type of extra data in go-legs gossip is bytes. But when it is parsed
as miner ID, it is cast to string then parsed. Instead, it should be
decoded from bytes.
Upgrade to the latest `index-provider` which upgrades the go-legs
protocol to allow the inclusion of extra gossip data that may be used
for gossip validation purposes. In the case of lotus gossip message
validators the miner ID is used to verify the sender's peer ID on chain.
Relates to:
- https://github.com/filecoin-project/lotus/pull/8045
Content providers announce the availability of indexer data using gossip pubsub. The content providers are not connected directly to indexers, so the pubsub messages are relayed to indexers via chain nodes. This PR makes chain nodes relay gossip pubsub messages, on the /indexer/ingest/<netname> topic.
Rename the config section corresponding to indexing to `IndexProvider`
for better readability.
Update existing docs for better clarity and add docs for config
parameters embedded from `index-provider` `Ingest` config library.
Masih H. Derkani
Aarsh Shah
Will
gammazero
Jiaying Wang
Łukasz Magiera
vyzo
Anton Evangelatov