Index provider integration uses a gossipsub topic to announce changes to
the advertised content. The topic name was fixed to the default topic
which is `/indexer/ingest/mainnet`.
In the case of lotus, the gossipsub validators enforce a list of topics
the instance is permitted to join by setting subscription filter option
when `PubSub` instance is constructed via DI.
Having the fixed topic name meant that any SP starting up a node on a
network other than `mainnet` would have to override the default config
to avoid the node crashing when index provider is enabled.
Instead of a fixed default, the changes here infer the allowed indexer
topic name from network name automatically if the topic configuration is
left empty.
Fixes#8510
The markets process instantiates its own `PubSub` instance with all
validators, peer scoring, etc. set up. Use that instane to join the
indexing topic, otherwise the default topic instantiated by
index-provider internally (via go-legs) has no validators.
The markets process instantiates its own `PubSub` instance with all
validators, peer scoring, etc. set up. Use that instane to join the
indexing topic, otherwise the default topic instantiated by
index-provider internally (via go-legs) has no validators.
- add opt-in env var to control instantation, until we are comfortable with testing to enble by default.
- adjust default limits if the connection manager high mark is higher than the default inbound conn limit.
Integrate the latest `index-provider` and reflect the changes to engine
configuration. Note that this commit disables announcements of indices
on the network by default as requested for initial merge to master.
Introduce dedicated index provider configuration parameters with
documentation and defaults that match the defaults in index-provider.
Re-generate code as needed.
- Add comment to clarify the reason for loop in testkit
- Trim common prefix in state printed in CLI commands for better
readability
- Upgrade to a tagged release of `go-fil-markets` that includes indexing
work; see: https://github.com/filecoin-project/go-fil-markets/pull/673
- Fix typo in CLI usage.
- Add comments to note that it is safe to use fx `OnStart` context when
starting the provider engine.
- Fix string concatenation in error message formatting.
Remove the bespoke instantiation of libp2p host and datatransfer manager
for index provider and reuse the existing instances used by markets.
The rationale for reuse is the following:
1. Separation of host introduces a discovery problem, where without
gossipsub the index provider endpoint will not be discoverable.
Using the same host as markets would mean the chain can be used to
discover addresses, putting less empassis on criticality of
gossipsub considering its set-up cost and lack of message delivery
guarantees.
2. Only a single instance of graphsync/datatransfer can be instantiated
per libp2p host; therefore, if the host is shared, so should
datatransfer manager.
3. it is not clear if the assumptions under which separation was
decided still hold.
Upgrade to the latest `index-provider` which upgrades the go-legs
protocol to allow the inclusion of extra gossip data that may be used
for gossip validation purposes. In the case of lotus gossip message
validators the miner ID is used to verify the sender's peer ID on chain.
Relates to:
- https://github.com/filecoin-project/lotus/pull/8045
Content providers announce the availability of indexer data using gossip pubsub. The content providers are not connected directly to indexers, so the pubsub messages are relayed to indexers via chain nodes. This PR makes chain nodes relay gossip pubsub messages, on the /indexer/ingest/<netname> topic.
Rename the config section corresponding to indexing to `IndexProvider`
for better readability.
Update existing docs for better clarity and add docs for config
parameters embedded from `index-provider` `Ingest` config library.
* adding the new variables- now time for logic
* putting parameters into right placeS
* adding unsealing throttle
* fixing linter issues
* removing one last thing...
