The markets process instantiates its own `PubSub` instance with all
validators, peer scoring, etc. set up. Use that instane to join the
indexing topic, otherwise the default topic instantiated by
index-provider internally (via go-legs) has no validators.
- add opt-in env var to control instantation, until we are comfortable with testing to enble by default.
- adjust default limits if the connection manager high mark is higher than the default inbound conn limit.
Integrate the latest `index-provider` and reflect the changes to engine
configuration. Note that this commit disables announcements of indices
on the network by default as requested for initial merge to master.
Introduce dedicated index provider configuration parameters with
documentation and defaults that match the defaults in index-provider.
Re-generate code as needed.
- Add comment to clarify the reason for loop in testkit
- Trim common prefix in state printed in CLI commands for better
readability
- Upgrade to a tagged release of `go-fil-markets` that includes indexing
work; see: https://github.com/filecoin-project/go-fil-markets/pull/673
- Fix typo in CLI usage.
- Add comments to note that it is safe to use fx `OnStart` context when
starting the provider engine.
- Fix string concatenation in error message formatting.
Remove the bespoke instantiation of libp2p host and datatransfer manager
for index provider and reuse the existing instances used by markets.
The rationale for reuse is the following:
1. Separation of host introduces a discovery problem, where without
gossipsub the index provider endpoint will not be discoverable.
Using the same host as markets would mean the chain can be used to
discover addresses, putting less empassis on criticality of
gossipsub considering its set-up cost and lack of message delivery
guarantees.
2. Only a single instance of graphsync/datatransfer can be instantiated
per libp2p host; therefore, if the host is shared, so should
datatransfer manager.
3. it is not clear if the assumptions under which separation was
decided still hold.
Upgrade to the latest `index-provider` which upgrades the go-legs
protocol to allow the inclusion of extra gossip data that may be used
for gossip validation purposes. In the case of lotus gossip message
validators the miner ID is used to verify the sender's peer ID on chain.
Relates to:
- https://github.com/filecoin-project/lotus/pull/8045
Content providers announce the availability of indexer data using gossip pubsub. The content providers are not connected directly to indexers, so the pubsub messages are relayed to indexers via chain nodes. This PR makes chain nodes relay gossip pubsub messages, on the /indexer/ingest/<netname> topic.
Rename the config section corresponding to indexing to `IndexProvider`
for better readability.
Update existing docs for better clarity and add docs for config
parameters embedded from `index-provider` `Ingest` config library.
