FREE WEBINAR X NetApp
Traditional model serving involves static request routing and uniform load balancing where the trained models are deployed behind stateless endpoints. LLM inference is inherently different from serving traditional models, as every incoming request carries prompt context. Variable prompt lengths means that no two inference requests incur the same computational costs.
Along with it, the Key-Value (KV) cache generated at the backend during the prefill stage can be reused to reduce latency, minimize redundant computation, and improve infrastructure efficiency. The benefits can only be realized if subsequent inference requests are routed to the same inference backend (e.g., vLLM) that already holds the KV cache.
Traditional request routing with a standard Kubernetes Service is designed to distribute requests using generic load balancing, without considering characteristics like KV cache locality. llm-d, an open-source, Kubernetes-native stack for distributed LLM inference, bridges this challenge by bringing inference-aware routing capabilities to LLM serving.
Acting as an intelligent routing (or orchestration) layer, llm-d complements popular inference engines like vLLM by efficiently routing and scheduling inference requests based on characteristics such as variable prompt lengths, KV cache locality, prefill/decode asymmetry, and GPU utilization.
What Is llm-d
llm-d is a Kubernetes-native distributed LLM inference framework that sits in front of one or more LLM inference backends. llm-d is created and maintained by open-source contributors from notable organisations including Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. This distributed LLM inference framework is built with the goal of supporting any model, any accelerator, any cloud.
Built on vLLM, Kubernetes, and Inference Gateway, llm-d offers modular solutions for distributed inference with features like KV-cache aware routing and disaggregated serving.
Figure 1: Why llm-d? [Image Source]
llm-d is the distributed inference serving stack around model servers, including Router, Endpoint Picker, InferencePool, KV-cache-aware routing, disaggregated serving, and related orchestration components.As stated in llm-d official documentation, that role is pluggable across backends including vLLM, SGLang, and TensorRT-LLM. A model server acts as a computer layer and loads a model onto one or more accelerators (GPUs, TPUs, XPUs, CPUs, and emerging NPUs).
Lowest layer in the llm-d stack, model servers execute the actual prefill and decode steps that generate tokens. Apart from the Model Server, Router, and InferencePool are other essential components of the llm-d framework.
As said in the official documentation, llm-d is said to provide the fastest time-to-value and competitive performance per dollar.
Why Normal Kubernetes Routing Fails For LLM inference
Before we get into Kubernetes for LLM inference, let’s take a step back and look at how traditional model serving works. At a high-level, it is about loading the trained ML model into the memory using a serving framework such as TensorFlow Serving, TorchServe, or ONNX Runtime. Once the model is loaded successfully, it is exposed through a REST or gRPC endpoint.
All the incoming requests are handled stateless, with the replica pods holding no memory of any prior invocation. Hence, the replica behind the endpoint can be interchangeable with some other replica, since no context or state is carried between requests.
This is the very reason why K8s default service load balancing mechanism (i.e., random-probability based selection or round-robin depending on kube-proxy in iptables mode) works well for traditional model serving but not for LLM inference.
K8s services distribute traffic across interchangeable backend Pods without considering request complexity, execution cost, or application state. This approach is best-suited for stateless model serving where the pod replicas are fungible. Frontends do not care which backend they use.
In contrast, no two requests are equal in LLM inference. Consider these requests hitting the same model: one with a 200-token prompt, another with a 100,000-token prompt. However, these requests are seen with an identical lens by Kubernetes Service routing where the requests are routed without considering prompt length, expected compute cost, or memory consumption.
Prefill Cost
Once the prompt is entered by the user, the prefill phase processes the entire prompt before generating the first output token and corresponding KV cache. Longer prompts (e.g., 100,000-token prompt) require significantly more GPU time in the prefill phase in comparison to a shorter prompt (e.g., 200-token prompt).
Since a generic Kubernetes load balancer treats both requests identically, the 200-token request may end up queued behind the 100,000-token request, increasing its time-to-first-token (TTFT).
Figure 2: The Prefill Phase [Image generated using AI]
Additionally, processing a large number of input tokens consumes more GPU compute time, memory bandwidth, and KV cache memory during the prefill stage.
KV Cache Memory Footprint
During request servicing, GPU memory is consumed within the model’s KV cache by each prompt token as well as every token that is generated. Along with the increase in the prefill phase, longer prompts also create larger KV caches that consume GPU memory. Hence, the KV cache also increases linearly with the sequence length (i.e., input tokens + output tokens generated so far) during the course of a single inference request.
Figure 3: Top-level architecture of KV Caching [Image Source]
Understandably, a 100,000-token request can consume tens of GBs of GPU memory whereas a 200-token request barely registers. Post the decode phase of inference, the KV cache might be released unless the cache is preserved by the framework for prefix cache reuse. Releasing the KV cache after generation comes with a trade-off, it frees memory for subsequent requests right away but it also sacrifices the massive reuse opportunity!
Consider a case where an AI agent is a technical support assistant for AceCloud.ai:
Format: 1) Acknowledge issue 2) Provide solution, 3) Offer further help.
- Request 1: [shared prefix] + “Q: Which GPU is best for fine-tuning a 70B model?”
- Request 2: [shared prefix] + “Q: Is H200 available in the APAC region?”
Request 1 computes and caches K/V tensors for the shared prefix; Request 2 detects the match, reuses those tensors, and runs prefill only on the new question.
As seen above, AceCloud system prompt describing available GPU options, is computed once on Request 1 and reused on Request 2, since both requests share the same leading tokens. Popular serving frameworks such as vLLM, llm-d, TensorRT-LLM, etc. intentionally retain portions of the KV cache for prefix cache reuse, allowing subsequent requests with a shared prompt prefix to skip recomputing those tokens.
Decode Footprint
Once the first output token is generated, the model enters the decode phase, where it generates one output token at a time. Each new token is sequentially appended to the sequence and fed back into the model to generate the next one.
Figure 4: Decode Phase [Image Source]
While prefill is compute bound, the decode phase is memory-bandwidth-bound. As seen earlier, large-context requests could occupy a larger KV cache thereby leaving lesser GPU memory for batching other requests. As a result, the replica can serve fewer requests concurrently, reducing throughput and increasing latency for everyone else.
This is exactly why continuous batching is so critical for ensuring higher decode throughput.
Where Normal K8s Routing Falls Short
None of Kubernetes’ built-in load-balancing strategies have visibility into any of the pointers described so far in this section. Standard Kubernetes load balancers function without insights into key metrics like queue depth, the token counts of active requests, or a replica’s current KV cache occupancy.
Owing to this, a long-context request can be routed to the replica pod that is already the busiest whereas a lightly-loaded replica sits idle. All of this results in the following downsides:
- Uneven GPU utilization
- Request queuing behind long-context requests
- Memory pressure that force the serving framework to preempt in-flight requests through cache eviction
All the above gaps are closed by llm-d, as it relies on prefix-cache-aware routing rather than K8s’s connection-count-based routing. Sitting in front one or more LLM inference backends, this inference-aware orchestrator smartly routes LLM inference requests using critical signals such as KV cache occupancy, queue depth, and prefix-cache locality.
In the end, the replica pod that is best positioned to serve the request is the one that gets it!
Core Capabilities Of llm-d
llm-d, the open-source Kubernetes-native stack helps in speeding up speeds up distributed LLM inference at scale. The core distributed inference features namely, KV-cache aware routing and disaggregated serving are built on top of vLLM, Kubernetes, and Inference Gateway.
Here are main core capabilities of llm-d:
Inference-Aware Traffic Management
As seen so far, llm-d replaces the traditional connection-count-based routing in Kubernetes with well-informed routing decisions informed by what is happening inside each replica. KV cache occupancy, queue depth, and prefix-cache locality are some of the key parameters on which the routing decisions are made.
Acting as an orchestration (or routing) layer, llm-d scores candidate replicas per request and forwards to the one that can serve fastest and cheapest.
Figure 5: llm-d orchestration/routing layer [Image generated using AI]
Application teams can make use of well-lit-paths, llm-d capabilities packaged as modular-building-blocks that span the following areas:
- Intelligent routing (latency-aware scheduling + multi-model serving)
- KV cache management
- Large-model serving (prefill/decode disaggregation + expert parallelism)
- Autoscaling (SLO-aware scaling, fast model startup)
These composable pieces can be tailor-made to combine into a production setup that fits a given LLM model, hardware, and workload. Explaining these concepts are beyond the scope of this blog, would recommend checking-out the following guides:
Endpoint Picker (EPP)
Considered as the brains of llm-d deployment, Endpoint Picker (EPP) is primarily responsible for intelligent routing and fairness & prioritization.
As a part of intelligent routing, EPP selects the appropriate model server pod (e.g., vLLM) within the InferencePool to serve the model or process each inference request. It uses internal state (i.e., KV-cache utilization, prefix cache locality, request queue depth, and active request counts) of model server pods to intelligently route the inference request.
Figure 6: The EPP pipeline [Image Source]
As far as fairness and prioritization is concerned, EPP selects which inference requests should run at any given time. This enables consolidation of multiple workloads with varying priorities onto a single set of Model Servers.
EPP is an extensible component that can be seamlessly integrated with the proxy layer via Envoy’s External Processing (ext-proc) protocol. We recommend checking out EPP detailed documentation to learn about its features and implementation in more detail.
Prefix-Cache-Aware Routing
Prefix-cache-aware routing is a core routing technique that is managed by the llm-d router. It is done via the EPP component. Reduction in tail latency and increased throughput are the major benefits offered by prefix-cache-aware routing.
Figure 7: Prefix-Aware Routing vs. Load-Aware Routing [Image Source]
The system avoids redundant prefill computation by routing requests to model server replicas that already contain the relevant KV cache for a prompt’s prefix, thereby saving both time and accelerator (GPU/TPU) resources.
We recommend checking out KV Management detailed documentation to learn about its implementation in more detail.
Kubernetes Gateway API Inference Extension
The standard Kubernetes Gateway API only understands HTTP-level routing (i.e., host, path, and header matching). The Gateway API Inference Extension (GAIE), a Kubernetes SIGs project, introduces the InferencePool API and an EPP, implemented via Envoy’s external processing (ext-proc) filter.
Inference Gateway (IGW), a proxy/load-balancer which has been coupled with an EPP, provides optimized routing and load balancing for serving Kubernetes self-hosted GenAI workloads. Endpoint Picker (EPP) is the critical data-plane component (acting as a Router) that intercepts incoming inference requests and routes each request to the optimal model server replica.
Prefill/Decode Disaggregation
Since prefill (compute-bound) and decode (memory-bandwidth-bound) serve different purposes, running both the phases on the same replica pod could lead to increased inference latency and resource contention.
To avoid such situations that can also lead to inefficient GPU utilization, llm-d uses disaggregated serving that separates the prefill and decode stages of LLM inference onto different model server instances.
Figure 8: Prefill/Decode Disaggregation [Image Source]
The prefill and decode phases run as separate pods, each in its own independently scalable pool. With disaggregation in place, a larger Tensor Parallelism (TP) is used for the memory-bound decode phase while a smaller TP for the computation-bound prefill phase.
By allocating dedicated prefill instances for these lengthy requests, the system isolates the prefill phase from ongoing decoding tasks. This prevention of blocking ensures that decoding requests continue to be processed efficiently, ultimately enhancing the overall Quality of Service (QoS).
KV cache offloading across GPU, TPU, CPU, and storage tiers
The KV cache grows with every prompt and generated token. For example, a 200-token prompt creates a relatively small KV cache, whereas a 100,000-token prompt can consume tens of gigabytes of GPU memory. KV cache offloading is not a llm-d feature but llm-d is aware of KV cache state and routes requests accordingly.
With KV cache offloading in llm-d, KV cache data is moved from high-speed accelerator memory (GPU or TPU HBM) to lower-cost memory tiers such as CPU RAM or storage when it is not immediately needed.
![KV Cache Offloading [Image generated using AI]](data:image/svg+xml;base64,PHN2ZyB2aWV3Qm94PSIwIDAgMTAyNCAxMDE1IiB3aWR0aD0iMTAyNCIgaGVpZ2h0PSIxMDE1IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==)
Figure 9: KV Cache Offloading [Image generated using AI]
Accelerators have limited memory, and every request holding onto its KV cache eats into it. Offloading that cache elsewhere frees up room on the GPU so that longer contexts can be efficiently handled and more requests be served at once, without buying more accelerators.
Accelerator-Neutral Design
Officially accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project, lm-d was founded with a clear vision: any model, any accelerator, any cloud. Although llm-d was initially developed around the vLLM ecosystem, it now supports popular inference backends such as vLLM, SGLang, and TensorRT-LLM.
llm-d can be used with different model-server backends and accelerator environments, but heterogeneous accelerator routing should be treated as advanced and validated per model/backend. Feature parity, metrics, cache behavior, quantization and performance may differ across NVIDIA, AMD, TPU, XPU and CPU backends.
It can also be used with GPU cloud providers like AceCloud, giving them the flexibility to serve models across a mix of accelerators without locking their stack to a single vendor.
llm-d Architecture
llm-d is built around three primary concepts : Router, InferencePool, and Model Server. It has a purpose-built layer of components that sit on top of vanilla Kubernetes and vLLM.
Figure 10: llm-d architecture [Image Source]
Here are the major components that are a part of the llm-d architecture:
llm-d Router
The llm-d router is the entry point of the LLM inference requests. It comprises a Proxy (typically Envoy) and Endpoint Picker (EPP). The llm-d router provides LLM-aware load balancing, request queuing, and policy enforcement.
- Inference Gateway (Envoy)– A high-performance L7 proxy (typically Envoy) that accepts incoming requests and consults the EPP via the ext-proc protocol to determine where to send them
- Endpoint Picker (EPP) – The routing engine that scores and selects the model server pods on metrics such as KV cache affinity, configured policies, and real-time metrics (i.e., queue-scorer, kv-cache-utilization-scorer, and prefix-cache-scorer)
The weighted score combines the real-time metrics of the EPP, utilizing default values of 3 for the prefix-cache-hit and 2 for the queue depth.
schedulingProfiles:
-name: default
plugins:
-pluginRef: label-selector-filter# Optional: not in default profile
-pluginRef: prefix-cache-scorer# Recommended: not in default profile
weight: 3.0
-pluginRef: kv-cache-utilization-scorer# Recommended: not in default profile
weight: 2.0
-pluginRef: queue-scorer# Recommended: not in default profile
weight: 2.0
-pluginRef: max-score-picker# Default picker (auto-injected if omitted)InferencePool
It is the central resource that acts as a bridge between the Gateway, EPP, and collection of the Model Server instances. The two primary responsibilities of InferencePool are:
- Endpoint Discovery for EPP– A Kubernetes Label Selector is used for identifying the eligible model server pods to take up the LLM inference requests. The EPP watches the selected pods and continuously monitors their health and serving metrics.
The InferencePool is the Router’s discovery point. The Router queries the InferencePool which in-turn identifies the model-serving replica pods and exposes them as a logical endpoint pool for request routing.
- Define the routing pool – A Custom Resource Definition (CRD) from the Gateway API Inference Extension that defines the pool of inference endpoints (or model servers) and configures the EPP and proxy for LLM-aware routing.
For instance if the pods are labelled as,
metadata:
labels:
app: vllm
model: llama-3-70bthe InferencePool might specify:
spec:
selector:
matchLabels:
app: vllm
model: llama-3-70bConsequently, the EPP locates all Pods matching these labels, designating them as the eligible inference endpoints for that specific pool. You can find more information about the responsibilities in the InferencePool official documentation.
Model Server
It is the component that runs LLM inference on a model. vLLM, SGLang, and TensorRT-LLM (trtllm-serve) are currently the model server backends supported by llm-d.
Considered as the compute layer, model server is the lowest layer of the llm-d stack that loads a model onto one or more accelerators (GPUs, TPUs, etc.). It then and exposes a supported API, such as OpenAI-compatible API, for inference requests.
llm-d’s core design can be further extended with optional advanced patterns such as KV Cache Management, Disaggregated Serving, Batch Inference, etc., details about which can be found in the llm-d advanced patterns documentation.
Link – https://github.com/acardace/llm-d-demo
llm-d vs vLLM vs KServe vs Ray Serve
The LLM inference ecosystem includes several serving frameworks, each of which address different needs model serving. Each of these frameworks addresses specific requirements within the LLM inference ecosystem: vLLM targets high-performance execution of models, KServe delivers models serving native to Kubernetes, and Ray Serve streamlines application orchestration alongside distributed deployments.
At the first glance, llm-d, vLLM, KServe, and Ray Serve seem to appear solving a similar problem. However, they operate at different layers of the LLM serving stack. The comparison shown below will be helpful in leveraging each of them to realize the needs of production-grade LLM inference.
Shown below is a simplistic flow diagram of the different frameworks that are part of the LLM ecosystem:
Figure 10: LLM Serving Ecosystem [Image generated using AI]
As seen above, the llm-d framework complements inference engines and serving frameworks by bringing the intelligence of intelligent orchestration and scheduling to the LLM serving stack.
Benefits And Trade-offs Of llm-d
llm-d is an evolving framework that has recently joined the CNCF as a Sandbox project. Here are some of the major benefits of the llm-d framework:
- Lower Time To First Token (TTFT) – Inference-aware orchestration and request routing not only reduces request queuing but also enables faster generation of initial tokens.
- Improved Cache Locality – Prefix-cache-aware routing ensures that inference requests with similar prompt prefixes are routed to pod replicas with matching KV cache or prefix cache. This eventually minimizes redundant prefill computation.
- Higher Throughput – Distribution of inference requests based on real-time backend load rather than than Kubernetes’ generic load-balancing algorithms, increases throughput by improving overall cluster utilization.
- Efficient GPU Utilization – LLM-aware load balancing and scheduling balances workloads across GPUs. This results in improved hardware efficiency.
Here are some of the trade-offs of the llm-d framework:
- Ecosystem Maturity – The overall llm-d ecosystem is evolving, it only became a CNCF Sandbox project in March 2026.
- Operational Complexity – Introduction of additional components like the Gateway, EPP, and InferencePool can increase the operational overhead in comparison to a traditional Kubernetes service.
- Dependency on Gateway API and emerging CRDs -The architecture relies on the GAIE (Gateway API for Inference Extensions) and custom resources such as InferencePool, which are less battle-tested than core Kubernetes primitives.
- Debugging Complexity -Prefix-cache-aware routing adds several new moving parts (EPP’s scoring logic, its prefix-cache map, etc.). Hence when a request lands on an unexpected replica, you might need to additionally look into EPP-specific logs and runtime metrics.
At the time of writing this blog, the latest version of llm-d is v0.8.1. The framework is undergoing rapid development, with recent versions graduating features like Multimodal Workload Serving, batch gateway functionality, amongst others!
Enterprise Evaluation Checklist
Even before adopting the llm-d framework, it is essential to evaluate it and verify whether it aligns with operational requirements and inference workloads. The evaluation checklist consists of questions that can help in determining whether llm-d is suited for your needs.
Conclusion
Features like KV-cache aware routing, disaggregated P/D (Prefill/Decode), and Intelligent request routing & load balancing helps llm-d close the gap that generic Kubernetes routing leaves open for LLM inference.
Since llm-d is accelerator neutral by design, teams can build this on NVIDIA, AMD, or TPU hardware without vendor lock-in. Teams can also run llm-d on GPU cloud infrastructure from providers like AceCloud. This can be achieved by pairing llm-d’s accelerator-neutral orchestration with AceCloud’s GPU capacity to serve models efficiently across available hardware.
