Quick Answer
Production GPU utilization cannot be determined from one dashboard percentage. Teams should evaluate GPU activity alongside Model FLOPs Utilization, tokens or samples processed per second, memory bandwidth, latency, idle time, successful workload progress and cost per output. A GPU can report nearly 100% activity while still delivering poor throughput or inefficient compute usage.
An AI team sees 95% GPU utilization on its dashboard. Yet, Training runs still miss deadlines, inference costs keep rising, and adding more GPUs produces only modest throughput gains.
The GPUs appear busy, but data-loading pauses, memory stalls, synchronization delays, and underfilled batches are consuming valuable GPU-hours.
This is the difference between GPU activity and productive utilization. To understand whether your infrastructure is efficient, you must look beyond GPU-Util and measure useful workload output, latency, MFU, failed GPU-hours, and cost per result.
What Does GPU Utilization Actually Measure?
GPU utilization is a time-based activity metric, not a compute-depth metric. It tells you that a kernel was running, but not how efficiently that kernel used the hardware.
According to NVIDIA System Management Interface documentation, GPU utilization is the percentage of a sampling period during which one or more kernels were executing. Depending on the GPU product, the sampling period may range from approximately one-sixth of a second to one second.
Therefore, 100% GPU-Util does not mean 100% compute efficiency. Short or underfilled kernels can maintain activity without fully using streaming multiprocessors, Tensor pipelines, or available memory bandwidth.
GPU memory usage differs from compute utilization. Frame-buffer memory usage shows how much GPU memory is allocated, while NVIDIA’s memory-utilization metric measures periods when device memory is being read or written.
Production utilization is better understood through four layers:
| Utilization layer | Primary Questions | Example Metric | Common source of waste |
|---|---|---|---|
| Provisioned | How much GPU capacity is available? | Total GPU-hours | Overprovisioning |
| Allocated | How much capacity is assigned to workloads? | Allocated GPU-hours | Scheduling and fragmentation |
| Hardware | How effectively is the GPU operating? | SM, Tensor pipe, DRAM, PCIe, NVLink activity | Memory or communication stalls |
| Application | How efficiently is the model training? | MFU (Model FLOPs Utilization) | Suboptimal batch size or architecture |
| Productive | What useful result is being delivered? | Tokens, samples, requests, or progress per dollar | Poor throughput or failed work |
The productive layer matters most. A GPU creates value when it advances a workload at the required performance, reliability, quality, and cost, not simply when it is allocated.
Why Can High GPU-Util Still Mean Poor Efficiency?
High GPU-Util can coexist with poor efficiency because the metric records activity rather than useful computation.
How can underfilled kernels keep a GPU busy?
Small kernels may execute repeatedly without exposing enough parallel work to fill the GPU. The activity percentage remains high even though much of the available processing capacity is unused.
How can memory stalls reduce useful compute?
A kernel can remain active while waiting for data from GPU memory. NVIDIA’s DCGM documentation notes that active warps may still be stalled on memory requests.
High SM activity must therefore be evaluated alongside Tensor activity, DRAM activity, achieved throughput, and workload output.
How can communication resemble useful work?
Distributed training introduces collective communication, barriers, validation calls, and synchronization. These operations may keep parts of the GPU system active without advancing model computation at the expected rate.
Model FLOPs Utilization, or MFU, attempts to measure training efficiency by comparing achieved model FLOPs with theoretical GPU throughput, though FLOP-accounting errors can distort results.
A May 2026 preprint, Instant GPU Efficiency Visibility at Fleet Scale, studied 608 production H100 training jobs and reported mean MFU of 25.1%, with a standard deviation of 10.9 percentage points. The study itself identified two framework-level FLOP-calculation errors, finding only 0.78 correlation with application-level MFU.
The finding applies to that commercial cluster and is not a universal H100 benchmark.
In one case, removing a distributed-debug configuration increased the study’s Overall FLOP Utilization metric by 2.5 times. The study does not establish that end-to-end training throughput increased by exactly the same amount.
Where Does GPU Capacity Disappear in Production?
GPU capacity often disappears while the device waits for another part of the system.
Is the GPU waiting for data?
Slow data loaders, tokenization, image decoding, augmentation, storage access, and host-to-device transfers can leave the GPU waiting between batches.
A common symptom is a sawtooth pattern in which GPU activity rises when a batch arrives and falls while the next batch is prepared.
Is the workload memory-bound?
High DRAM activity combined with low Tensor activity or achieved FLOPs may indicate that data movement, rather than computation, is the limiting resource.
Potential causes include poor memory-access patterns, CPU offloading, memory fragmentation, excessive intermediate tensors, and low arithmetic intensity.
Is distributed communication delaying progress?
All-reduce operations, network congestion, poor GPU placement, stragglers, and pipeline bubbles can force faster workers to wait for slower ones.
A cluster-wide average may look acceptable while individual GPUs spend substantial time idle.
Is scheduling stranding capacity?
Fleet efficiency can fall when GPUs remain idle between jobs, multi-GPU workloads wait for enough devices to become available, or jobs are assigned to unsuitable GPU types.
This is especially relevant in Kubernetes environments where allocation, autoscaling, and workload placement affect fleet utilization separately from within-job efficiency.
A vendor-published Cast AI 2026 study reported average GPU utilization of 5% across the non-optimized Kubernetes clusters it analyzed. This is a fleet-level observation from Cast AI’s dataset, not a universal benchmark or an MFU measurement.
Are checkpointing and failures consuming GPU-hours?
A 2026 study, The Energy Cost of Execution-Idle in GPU Clusters, analyzed 11,791 long-running jobs. Execution-idle accounted for 19.7% of in-execution time and 10.7% of in-execution energy in the measured academic cluster.
These percentages exclude deep-idle periods and should not be interpreted as shares of all provisioned cluster time or total data-center energy.
A separate 504-GPU B200 production-cluster report found that checkpoint restart loading averaged 21.5% of the storage configuration’s maximum read-bandwidth value. Checkpoint-save bursts averaged 16% of its maximum write-bandwidth value.
Those values were configuration sizing references, not independently measured storage peaks or percentages of GPU capacity.
How Do Training and Inference Use GPU Capacity Differently?
Training generally prioritizes sustained model progress. Inference must balance throughput with latency and variable demand.
| Objective | Training | Inference |
|---|---|---|
| Primary output | Tokens or samples processed | Requests or tokens served |
| Efficiency metrics | MFU, step time, scaling efficiency | Throughput, concurrency, cost per token |
| Critical target | Time to convergence | TTFT and P95/P99 latency |
| Common waste | Data stalls, communication, checkpoints | Low concurrency, bursty traffic, KV-cache limits |
| Saturation risk | Poor scaling efficiency | Queueing and tail-latency increases |
The execution-idle preprint found that its academic online-serving workloads spent 61% of in-execution time in execution-idle, accounting for 48% of in-execution energy. Training and batch inference showed lower exposure.
These findings are workload-specific. They should not be treated as expected values for every production service.
For training, prioritize useful progress per GPU-hour and time to convergence. For inference, maximize throughput without violating latency, reliability, or quality targets.
Talk to an Expert about balancing throughput, latency, concurrency and capacity headroom.
Which Metrics Reveal Useful GPU Work?
No single percentage can describe fleet use, hardware efficiency, workload performance, reliability, and financial value.
| Metric group | Recommended measurements |
|---|---|
| Fleet | Provisioned GPU-hours, allocated GPU-hours, queue time, stranded capacity, and idle time between jobs |
| Hardware | GPU-Util, SM activity, Tensor activity, DRAM activity, memory bandwidth, clocks, power, PCIe, and NVLink traffic |
| Training | Tokens or samples per second, step time, MFU, scaling efficiency, goodput, and time to convergence |
| Inference | Successful requests per second, input and output tokens per second, TTFT, ITL, P95/P99 latency, error rate, queue depth, and KV-cache usage |
| Financial | Cost per successful GPU-hour, cost per completed run, cost per million tokens, and failed-work cost |
Useful formulas include:
Allocation rate = Allocated GPU-hours ÷ provisioned GPU-hours
Productive GPU-hour rate = GPU-hours contributing to successful work ÷ paid GPU-hours
Scaling efficiency = Throughput on N GPUs ÷ (N × throughput on one GPU)
Cost per million tokens = Total compute, storage, networking, and platform cost ÷ measured token volume × 1,000,000
The token calculation should state whether it covers input tokens, output tokens, or both.
Can hardware counters replace application metrics?
No. Hardware counters are diagnostic signals, not substitutes for workload output and cost measurements.
The Overall FLOP Utilization preprintreported a correlation of 0.78 with application-level MFU after excluding 82 jobs affected by two identified FLOP-calculation errors.
In one documented production case, the metric helped identify serialized validation operations caused by a distributed-debug configuration. Removing the configuration improved the study’s OFU metric by 2.5 times. The paper did not claim that end-to-end training throughput increased by exactly the same amount.
How Can You Diagnose GPU Underutilization?
Start with useful workload output, not the GPU dashboard.
- Define throughput, latency, reliability, quality, and cost targets
- Measure useful output per GPU-hour
- Compare fleet, job, node, and individual-GPU results
- Inspect compute, transfer, communication, and idle timelines
- Identify the first constrained resource
- Change one variable at a time
- Validate the result against the original SLO
| Observed pattern | Likely issue | Investigate first |
|---|---|---|
| Low GPU-Util and high CPU activity | Input bottleneck | Data loaders and preprocessing |
| High GPU-Util and low MFU | Inefficient or memory-bound kernels | MFU and SM/Tensor/DRAM (training); tokens/sec, TTFT, p99, queue depth (inference) |
| Uneven per-GPU activity | Stragglers or placement | Per-rank timelines and topology |
| Pauses around checkpoints | Storage path | Queues, RPCs, and staging |
| High throughput and poor P99 latency | Excessive saturation | Batching and queue depth |
| Idle GPUs between jobs | Scheduling mismatch | Allocation and queue metrics |
How Can Teams Improve Utilization Without Hurting SLOs?
You should improve limited resources rather than forcing every GPU to remain saturated.
Improve training efficiency
- Tune microbatch size and gradient accumulation.
- Increase data-loader concurrency.
- Cache or prefetch input data.
- Use appropriate reduced precision.
- Overlap communication and computation.
- Improve node and GPU locality.
- Tune checkpoint frequency and recovery.
- Match GPU memory and interconnect to the model.
Improve inference efficiency
- Use dynamic or continuous batching.
- Tune concurrency against latency targets.
- Monitor queue depth and P99 latency.
- Improve KV-cache management.
- Separate interactive and batch traffic.
- Autoscale using demand and latency signals.
- Maintain deliberate capacity headroom.
Improve fleet efficiency
- Reduce idle time between jobs.
- Use workload-specific node pools.
- Consolidate smaller workloads where appropriate.
- Track failed and interrupted GPU-hours.
- Use spot capacity for fault-tolerant workloads.
- Compare cost per useful result before changing GPU type.
How Can the Right Infrastructure Support Productive GPU Utilization?
Infrastructure selection should follow the diagnosis rather than precede it.
AceCloud’s GPU Kubernetes cluster offering includes on-demand GPU options using GPUs such as the H100, A100, L40S, and RTX A6000, along with managed Kubernetes, GPU monitoring, custom node pools, autoscaling, and migration support.
| Diagnosed problem | Relevant platform capability |
|---|---|
| Idle capacity between workloads | GPU autoscaling with advanced metrics (cache, latency, throughput) |
| Operational Kubernetes overhead | Managed control plane |
| Poor workload-to-GPU fit | Multiple GPU profiles |
| Poor workload-to-GPU fit | Monitoring and observability |
| Migration friction | Workload migration assistance |
| Storage or networking bottlenecks | Integrated infrastructure assessment |
The appropriate GPU and cluster configuration should be selected according to measured compute, memory, interconnect, latency, reliability, and cost requirements.
What Should You Optimize Instead of 100% GPU Utilization?
The right goal is not 100% GPU-Util.
Training teams should optimize useful model progress, time to convergence, reliability, and cost per successful run. Inference teams should maximize completed work while protecting TTFT, tail latency, quality, and availability. At the fleet level, measure how much paid capacity contributes to successful output.
Once you identify whether the limiting factor is compute, memory, data, communication, scheduling, hardware health, or demand, you can improve the bottleneck instead of simply adding more GPUs.
Book a Free Consultation or Talk to an Expert to determine how much productive GPU capacity your workloads are actually achieving.
Frequently Asked Questions
There is no universal target. A good result meets throughput, latency, reliability, quality, and cost requirements.
It reports kernel activity during a sampling period, not how fully the GPU produced useful workload output.
Common causes include slow data loading, small batches, memory bottlenecks, synchronization, checkpointing, network delays, stragglers, and failures.
No. Frame-buffer usage shows allocated memory. Memory utilization measures read or write activity. Neither is equivalent to compute efficiency.
MFU offers a stronger view of model-compute efficiency. Combine it with throughput, step time, scaling efficiency, convergence, and failed GPU-hours.
Not necessarily. Permanent saturation may increase queueing and P99 latency. Interactive services often need capacity headroom.
Fleet utilization measures provisioned or active capacity across an environment. MFU measures achieved model FLOPs relative to theoretical throughput within a training workload.
Subtract GPU-hours contributing to successful output from total paid GPU-hours, then multiply the difference by the effective hourly cost. Include failures, repeated work, and delays.
