Why does sending a model with 100K-word prompt cost more than sending a shorter one, even when the LLM model and underlying GPU hardware stay the same? Well, the answer lies in Key-Value (KV) cache, a block of working memory that grows with every token generated and can outgrow the fastest memory available for long-context or high-concurrency workloads.
AI infrastructure has always been about compute, until memory became a bottleneck too. Rapid increase in LLM model sizes and long-context LLM inference is when memory became a constraint in GPU-accelerated LLM serving. Apart from memory capacity, memory bandwidth and data movement are the other factors that have a significant inference cost and performance.
Fast but limited capacity of GPU High-Bandwidth Memory (HBM) results in large KV caches to spill into larger but slower host memory (CPU DRAM). CPU DRAM is constrained by server architecture and limited number of memory channels per CPU socket. NVMe, a cheaper option, sits further down the hierarchy but is considered too slow for active KV cache.
This performance bottleneck, termed the AI memory wall, limits how fast large AI models can be served. CXL-attached memory pools help close that gap, offering far more capacity than local DRAM.
What Is the AI Memory Wall
Along with the model weights held in the GPU memory, the GPU also needs to keep information about the request that has already been processed. This information stored in the KV cache grows in size as LLMs (or models) grow larger, handle longer contexts, and cater to more concurrent users.
The most complex models now number a trillion or more parameters, which means that most of their GPU memory is consumed just for weights. The left-over HBM is taken up by the KV cache that loosely acts as the modelโs short term memory.
LLM inference can be compute-bound during prefill and more memory-bandwidth/KV-cache-bound during decode. The bottleneck depends on prompt length, output length, batch size, concurrency, model architecture, precision and serving engine.
The KV cache for long-running conversations could even consume hundreds of GBs of memory. Operating as a critical pressure point for real-time inference, the KV cache is rapidly hitting its memory ceiling.
The AI memory wall is what happens when the memory demands of inference exceed what the GPU can supply. AI memory wall could be due to:
- Model weights and KV cache outgrowing the available GPU HBM capacity, or
- HBM bandwidth being insufficient to move data to and from the GPU’s processing units fast enough
CXL (Compute Express Link) an open standard industry-supported cache-coherent interconnect for processors, memory expansion, and accelerators can be leveraged to reduce the GPU memory pressure.
Offloading the KV cache via CXL-based memory pooling allows GPUs to expand and pool external memory, delivering a scalable approach to handle larger models and growing context windows.
Why KV cache is the pressure point
In contrast to AI model training, which is episodic and heavy on compute, AI inference relies heavily on memory while operating in real time for user-facing tasks. In order to make this clearer, letโs look closely at what happens when a prompt is provided to the LLM.
The Prefill Phase
When a user-prompt arrives at the serving engine, the model has to first process everything before a single output token is generated (i.e., TTFT – time-to-first-token). This first phase called prefill is where the entire input prompt is processed in a single pass.
The cost of prefill scales with the prompt length, as every token in the prompt can attend to every other token in parallel. Longer contexts also drive higher GPU compute demands which impacts the TTFT and lowers the overall system throughput.
The Key-Value (KV) cache is created in the prefill phase and every token processed generates the keys and values that get stored for later reuse.
The Decode Phase
The response is generated in the decodephase, one token at a time. During token generation, the model calculates a new query, key, and value for each token.
Rather than checking for pre-existing computations, it continually reads the cached KV data to process the current token while appending the newly generated key and value to the KV cache so it can attend to all prior tokens. However, each decode step depends on the entire KV cache, and that cache grows with every token.
Figure 1: LLM Inference flow with KV Cache [Image Source]
Consider a 70B-parameter LLM (e.g., Llama 2 70B) serving 100 users simultaneously, where each request has a 32K-token context.
- Prefill phase: All the 32K tokens are processed as a batch/sequence during the prefill phase, thereby making it a more computationally-intensive operation.
- Decode phase: For generating any new token, the model avoids recomputing the entire context and uses the previously computed KV cache to access information from the 32K-token history.
In the decode phase, the GPU is largely underutilized because it is waiting for memory, shifting the bottleneck from computation to data movement.
The Growing KV Cache
The KV cache is accessed frequently during LLM serving, particularly during the decode phase. Hence, the KV cache has to live somewhere (in the memory) so that it can keep up pace with decode. Ideally, that memory is the HBM, faster memory that is closest to the GPU compute. Keeping the KV cache in HBM provides the bandwidth needed for fast decode, but HBM capacity is limited and expensive.
The KV cache grows with context length, batch size, model depth, number of KV heads, and head dimension. An influx of concurrent users, extended conversations, agentic workflows, and Retrieval-Augmented Generation (RAG) pipelines all extend cache retention requirements.
Consequently, in long-context and highly concurrent serving environments, the KV cache footprint can easily surpass the size of the model weights by several folds.
All these factors including the intermediate activations can lengthen the inference latency. For large billion-parameter LLMs (e.g., Llama 3.1 70B, Llama 3.1 405B, Mistral 7B, Gemma 2 27B, etc.) serving many concurrent users with long context windows, GPU HBM can become insufficient to hold the model weights and growing KV cache.
Decode is already starved for memory bandwidth and the KV cache is the pressure point in that phase, as it grows with every token and every concurrent session.
In demanding long-context or high-concurrency cases, KV cache can exceed practical HBM headroom, forcing the serving engine to reduce concurrency, evict cache blocks, recompute, or use offload/tiering, forcing a trade-off in the LLM serving engine.
It could either be shrunk batches, context evictions, or spilling the KV cache to a slower memory tier like DRAM, NVMe, to name a few.
The LLM Memory Hierarchy
As LLM serving scales, KV cache becomes the major consumer of memory. Its size grows with the following sequence length, number of layers, number of KV heads, and precision. Additionally, total KV-cache usage scales directly with the volume of concurrent sequences being served in a production environment.
For example, Llama3 405B has 126 layers, 8 key/value heads with a head dimension of 128, 2 tensors per layer (key and value), and uses FP16 precision (2 bytes per element).
The KV cache size per token is thus 126 ร 8 ร 128 ร 2 ร 2 = 516,096 bytes โ 504 KB per token. For a sequence of 8,192 tokens, a single request therefore requires approximately 4 GB (i.e., 504 KB * 8192) of KV cache memory, quickly exhausting GPU HBM at scale.
Figure 2: KV Cache Size Calculation [Image generated using AI from information available in this whitepaper]
So what happens when the fastest tier of memory or GPU HBM is exhausted? After GPU HBM, the next usable tiers may include local CPU DRAM, CXL-attached memory, RDMA-accessed remote memory, and NVMe/storage. Their order depends on platform topology, latency, bandwidth, access path and software support.
| Tier | Strength | Where it sits | Latency | Weakness |
|---|---|---|---|---|
| GPU HBM | – Highest bandwidth – Closest to GPU compute – Ideal for model weights, active KV cache, intermediate activations | On/close to the GPU | Very low | – Capacity-constrained – Expensive – Restricts both the volume and scale of requests that can be kept in active memory |
| CPU DRAM | – Provides substantially more capacity at lower cost – Can accommodate larger working sets | Attached to the GPU | Higher than HBM | – Higher latency – Lower bandwidth than HBM, with capacity constrained by CPU memory channels and server architecture |
| NVMe | – Extensively high capacity at low cost – Best suited for cold or infrequently accessed data | Attached to the PCIe slot | Higher than HBM, DRAM, and CXL memories | – Bandwidth limitations and elevated storage latency – Impractical for active KV-cache access during frequent inference workloads |
| RDMA memory pool | – Allows memory pooling and sharing across multiple servers – Enables workloads to scale beyond the capacity constraints of an individual physical machine | Remote server memory over network | Network dependent | – High access latency – Complex communication protocols – Synchronization overhead – Bandwidth contention, and infrastructure complexity can inversely affect inference performance |
| CXL memory | – Provides larger, memory-semantic memory pools – Capacity can be extended beyond local DRAM – Supports memory disaggregation | Attached through CXL | Lower than NVMe, higher than HBM/DRAM | – CXL performance depends on the underlying hardware and topology – Evolving ecosystem |
The KV cache has emerged as the component that most tightly couples compute throughput, memory capacity, and communication efficiency. Modern architectures aim to optimize KV-cache management by implementing multi-tiered storage strategies across various nodes, which effectively minimizes redundant computations through enhanced data reuse and intelligent distribution of storage resources.
Figure 3: AI Inferencing with GPUs and CXL [Image Source]
A research article states that Beluga is a pioneering Compute Express Link (CXL)-based memory architecture for LLM KV-cache management. Designed to optimize LLM inference, its implementation (Beluga-KVCache) reports huge TTFT and throughput improvements over RDMA-based approaches in vLLM-based experiments.
CXLโs Role in LLM Memory Scaling
For starters, CXL (Compute Express Link) is an industry-supported cache-coherent interconnect for processors, memory expansion, and accelerators. By keeping the memory space of the CPU coherent with that of connected devices, CXL technology enables seamless resource sharing. This unity boosts performance, streamlines the software stack, and lowers total system expenses.
With the recent release of the CXL 4.0 specification, transfer rates have doubled from 64 GT/s to 128 GT/s alongside the introduction of bundled port support. CXL memory modules can be attached via the PICe slot and scalable through switches. CXL-attached memory exhibits bandwidth, latency, and capacity characteristics between the DRAM and SSD.
CXL introduces a new layer into the memory hierarchy for LLM serving by relieving the KV cache pressure through shared, high-capacity memory pools. This helps extend the available memory beyond high-speed GPU HBM. A CXL-based architecture can add large pools of memory that are accessible to the compute infrastructure.
Figure 4: Memory Hierarchy [Image Source]
Through memory tiering, high-priority and frequently accessed data stays in GPU HBM, while lower-frequency data or larger KV cache segments offloaded from HBM are stored in CXL-attached memory. This frees the HBM for active sequences and further computations.
CXL-attached memory modules address concerns related to inefficient remote memory access. As stated in the CXL Consortiumโs AI memory wall blog, data that is not local to the HBM might need to traverse a long and inefficient path of GPU โ DRAM โ NIC โ Storage Server โ NIC โ DRAM โ GPU in a multi-GPU inference setup. Additional hops through the DRAM, Network Interface Cards (NICs), network card, etc. have to be covered before the data returns to the GPU.
In large-scale LLM deployment, even slight inefficiencies in managing prefill and decode KV caches compound into multi-second latency penalties and unutilized compute cycles. Every hop adds the following bottlenecks:
- Higher latency
- Network overheads
- Lower GPU utilization
- Poor GPU scalability
- Longer TTFT
- High data-movement resulting in increased energy costs
So how does CXL mitigate the above problems? Though the exact storage path is architecture dependent, at a high-level CXL memory can act as an additional memory tier between the local memory and storage: GPU HBM โ CPU DRAM โ CXL pooled memory โ NVMe/storage. This helps with efficient KV cache scaling and improved memory pooling and disaggregation, as hot KV cache data resides in the GPU HBM for faster access while overflow cache spills into the CXL memory pool.
In summary, CXL minimizes both the physical latency and latency overhead associated with expanding memory capacity, enabling pooled memory resources to be shared dynamically and seamlessly across GPU clusters and inference tasks.
Use cases for CXL in AI infrastructure
As per CXL consortium, CXL memory pooling is already moving beyond the labs and able to benefit across real-world data center and AI environments. CXL memory pools are helping organizations accelerate data-sensitive AI workloads and reduce Total Cost of Ownership (TCO) by improving resource utilization and reducing latency.
Here are some of the major use cases for CXL-attached memory in LLM serving and AI infrastructure:
LLM Inference and KV Cache Scaling
As seen so far, CXL memory pools can very well complement fast GPU HBM, particularly for spilled and infrequently used KV cache data. It provides the additional memory tier that is instrumental in extending the available capacity beyond HBM and CPU DRAM.
CXL memory augments GPU VRAM for KV cache storage, which helps accelerate token decoding and reduce TTFT for LLM serving.
Memory Disaggregation
Historically, running large-scale AI workloads when memory capacity was insufficient required interconnecting additional GPUs or GPU-enabled server nodes across networks.
Since every AI application has varied demands for compute and memory resources, the approach of allocating resources with fixed compute-to-memory ratios results in underutilized or wasted resources. This is particularly the case when one AI workload needs additional storage whereas the other server has unused memory.
Figure 5: CXL Memory Example – Panmnesiaโs CXL IP [Image Source]
To address the challenges associated with unutilized (or under-utilized) resources, CXL-based solutions can be leveraged whereby GPUs (or other AI accelerators) can be placed based on compute demands. This approach also resolves memory insufficiency by adding high-density memory devices (i.e., CXL-Memory Expanders) or memory nodes.
Furthermore, CXL prevents resource inefficiency by fully disaggregating memory from compute nodes and making pooled memory available as a shared asset.
AI Infrastructure
Instead of relying on the traditional strategy of provisioning additional GPUs or GPU-dense servers merely to expand memory capacity, AI (or cloud GPU) infrastructure companies must shift toward modern approaches like CXL memory pooling and sharing to solve resource underutilization.
Figure 6: Usage of CXL Memory Pooling [Image Source]
As seen in the above diagram, the core USP of CXL memory pooling is making more memory available without requiring every GPU to carry all of that capacity locally. For example, Samsung Semiconductorโs evaluation shows that CXL-based memory pooling can provide both substantial memory expansion and near-DRAM performance for KV cache offloading workloads.
Shared Memory Pools for Multi-Tenant Inference
In shared cloud AI environments, diverse inference workloads demand varying amounts of memory. Rather than statically locking memory capacity to specific GPU servers, CXL memory pooling enables dynamic memory allocation across multiple tenants as needed.
The usage of pooled CXL memory not only improves resource and memory utilisation but it also allows workloads with larger KV caches or higher concurrency access additional memory capacity when required.
Decoupling memory capacity from individual compute nodes allows infrastructure providers to size memory according to aggregate workload demands rather than over-provisioning for peak usage on each server.
Conclusion
GPU HBM, the memory closest to the GPU, is finite and highly expensive. Also, DRAM prices are up more than 90%, while even the cost of NVMe has surged. And these prices wonโt likely come down soon. These factors make memory a primary barrier to real-time AT inference (also termed the AI memory wall).
As far as AI/GPU infrastructure providers are concerned, overcoming this memory bottleneck is critical for improving GPU utilization and lowering the cost/token.
Compute Express Link (CXL) memory pooling and sharing can help bridge this gap by extending the memory capacity beyond costly HBM and slower DRAM. As CXL technology advances, pooled memory architectures will serve as essential infrastructure for next-generation AI data centers, driving greater operational efficiency, scalability, and resource flexibility.
