Question: Need an EPP for each InferencePool?

[ed-pai]

I've been testing out GAIE + Kgateway (agentgateway) + VLLM the last couple days and trying to understand a couple things. Really appreciate the help:

1. When hosting multiple VLLM engines with different models using BBR, should each unique model/engine instance get its own EPP?

For example:

# Should this have 3 separate EPP's deployed??
InferencePool 1 = VLLM (2 replicas) w/ Qwen3-32B
InferencePool 2 = VLLM (1 replica) w/ GPT-OSS-120B
InferencePool 3 = VLLM (4 replica) w/ Llama-3.1 + (3) LoRAs

2. Is there a mechanism that aggregates all /v1/models and serves that as a single endpoint?

I wrote a PoC of an aggregator that monitors the inferencepools, gets all instances of downstream VLLM, and regularly queries the downstream /v1/models and serves an aggregated upstream endpoint (includes all LoRAs this way). However, this feels like something that might be native to GAIE already. How are people solving this problem now?

[nirrozenbaum]

Hi @ed-pai,
welcome to Inference Gateway 😄 .

1. When hosting multiple VLLM engines with different models using BBR, should each unique model/engine instance get its own EPP?

the mental model of EPP today is that a pool is associated with a base model in a 1:1 relation. so in a single pool we can serve a base model and its LoRA adapters.

2. Is there a mechanism that aggregates all /v1/models and serves that as a single endpoint?

there is no such mechanism today. but, there is a WIP to leverage BBR for management of multiple inference pools without requiring any change from the user perspective. so from the user side it remains a single endpoint, internally BBR receives the request, injects a header that matches the matching InferencePool, and the request gets routed to the correct EPP using this header (defined in HttpRoute). more details on this in #1812.
This capability is targeted for completion around January, so hopefully we will have something working pretty soon.

as a side note about point (2) - this will be developed in a phased approach. in phase (1), an admin should update the available LoRAs in a configmap manually. in phase (2) that could be automated by calling /v1/models. we aim to have phase (1) in January.