Why is GPU utilization low, and how can you identify whether the bottleneck is compute, network, storage, or data loading?

Low GPU utilization usually means the accelerator is waiting for something. The bottleneck may be compute, network communication, storage throughput, CPU side data preparation, workload configuration, or scheduling. The fastest way to find it is to put those signals on the same timeline and look for the layer that changes when GPU activity drops.

Looking at one GPU utilization chart is rarely enough. A GPU can be perfectly healthy while a training job is blocked on data loading or collective communication. The useful question is not simply, "Why is the GPU at 40 percent?" It is, "What was the GPU waiting for during the other 60 percent?"

What does low GPU utilization actually mean?

Low GPU utilization means the accelerator was not continuously executing useful GPU work during the measurement interval.

That does not automatically mean the GPU is too slow, misconfigured, or faulty. It only tells you that the device was not busy all the time.

A training step usually has several stages. The CPU may prepare a batch. Storage may read samples. Data may move into host memory and then device memory. Multiple workers may synchronize. The GPU executes kernels. The process repeats.

If any upstream stage cannot keep the next batch ready, the GPU waits.

If distributed workers reach a synchronization point at different times, faster workers wait.

If a job is allocated a GPU but does not have enough work to use it, the GPU waits.

This is why utilization should be treated as a starting signal, not a diagnosis.

NVIDIA Data Center GPU Manager exposes telemetry such as utilization, clocks, power, memory related fields, PCIe activity, ECC events, and other accelerator statistics depending on hardware and driver support. Those metrics are useful, but they become much more valuable when correlated with the rest of the workload path.

For the resource allocation side, GPU resource pooling and scheduling in Kubernetes explains how queueing, resource classes, and allocation can create or reduce idle capacity.

How do you tell whether the bottleneck is compute?

A compute bottleneck is likely when the GPU stays busy and the job is still slower than expected.

That sounds obvious, but it separates a very different problem from low utilization caused by waiting.

If GPU utilization remains high, GPU memory is well used, clocks are operating normally, and the workload is continuously executing kernels, the accelerator may simply be the limiting resource for the workload.

Then look at the workload shape.

Is the model too large for one device?
Would more accelerators reduce step time?
Is the workload using the expected precision mode?
Are kernels running efficiently?
Is the card being throttled by temperature or power?
Is the job sharing the device with another workload?

Hardware health matters here. A card can report high utilization and still perform below expectation if it is repeatedly slowing clocks or encountering errors.

This is where card level health should be part of performance analysis. The source operating model connects utilization with temperature, power, ECC status, and clock behavior so a degraded card is not mistaken for a normal capacity limit.

If you operate mixed accelerators, how to manage multi vendor GPUs and NPUs covers the need to normalize health and capability before comparing devices.

How do you identify a network bottleneck?

A network bottleneck becomes likely when distributed workers spend more time waiting for communication and GPU utilization falls around the same periods.

Distributed training creates heavy communication between workers. The exact pattern depends on the framework and model, but the operational signals are similar.

Watch network throughput, latency, packet loss, retransmission, interface errors, congestion indicators, and the timing of collective operations where available.

Then align those signals with GPU utilization.

If GPU utilization falls at 10:17 and network retransmission rises at the same moment, that relationship deserves attention.

If one node shows a consistently slower link while the other nodes remain healthy, that node can hold back the entire job.

If all nodes show similar network pressure, the fabric may be saturated.

The strongest diagnosis comes from relationships, not just charts. The platform should know which network ports carry traffic for which compute nodes and which nodes belong to the affected job.

Without that mapping, a network team sees a congested port and an AI team sees a slow training job, but neither can prove that the two events are connected.

How do you identify a storage bottleneck?

A storage bottleneck becomes likely when the accelerator waits for data while storage latency rises, throughput falls, or the input pipeline cannot supply batches fast enough.

Monitor storage throughput, IOPS, latency, queue depth where available, and the read pattern of the training workload.

Check checkpoints separately from dataset reads. A job can run normally during most steps and then stall during large checkpoint writes.

Capacity and performance should also be separated.

A storage pool can have plenty of free capacity and still be too slow for the workload.

The same applies to network attached storage. The storage array may be healthy while the network path between compute and storage is constrained.

This is why storage should be viewed as part of the data supply chain rather than as an isolated infrastructure service.

A useful diagnostic timeline contains GPU utilization, storage throughput, storage latency, and training step time. If storage latency spikes first and GPU utilization drops immediately after, you have a much stronger hypothesis than "the GPUs look idle."

How do you identify a data loading bottleneck?

A data loading bottleneck becomes likely when the CPU side input pipeline cannot prepare and transfer batches as fast as the GPU consumes them.

PyTorch documents that DataLoader can use multiple worker processes and provides options such as num_workers and pinned memory to improve data loading throughput. The exact settings depend on the workload, storage system, CPU capacity, batch size, and transformation logic.

The important part is measurement.

Watch CPU utilization, per worker CPU pressure, host memory, disk or storage reads, batch preparation time, and GPU idle gaps.

If CPU usage is saturated while GPU utilization is low, preprocessing may be the limiting stage.

If storage is fast but batch preparation is slow, expensive transformations may be happening on the CPU.

If batches are prepared quickly but transfer to the GPU is delayed, inspect host to device transfer behavior.

Data loading problems are common because they can look exactly like insufficient GPU work from the accelerator side.

A useful test is to reduce the complexity of the input pipeline temporarily. If GPU utilization rises sharply with synthetic or preloaded data, the bottleneck is probably upstream of the GPU.

How can you tell whether the problem is scheduling instead of performance?

A scheduling problem is likely when GPUs are available in aggregate but workloads cannot obtain the right resource shape at the right time.

Imagine a cluster with eight free GPUs spread across several nodes. A distributed job may require eight GPUs on nodes that meet a topology or memory requirement. Total free capacity looks sufficient, but usable capacity for that job is not.

Quota can create the same symptom. The hardware is free, but the project has reached its allocation limit.

Fragmentation can also matter. Small workloads can occupy parts of several devices or nodes in a way that prevents a large job from fitting.

In these cases, GPU utilization may be low across the data center even though queued jobs are waiting.

That is not a compute performance problem. It is a resource allocation problem.

The queue should therefore show the reason a job is waiting, such as quota exhausted, no matching accelerator class, insufficient contiguous topology, health exclusion, or resource fragmentation.

Why should compute, network, and storage be plotted on the same timeline?

A shared timeline lets you see which layer changed first.

Suppose GPU utilization falls from 85 percent to 30 percent.

If storage latency rises 20 seconds earlier, storage is a strong candidate.

If storage stays normal but network retransmission rises sharply at the same time, investigate the training fabric.

If both network and storage remain stable while CPU side batch time increases, investigate data loading.

If all infrastructure metrics remain normal but the GPU shows clock slowdown or ECC events, investigate compute health.

This method does not magically prove causality, but it reduces guesswork and gives you a sequence of evidence.

The source material repeatedly uses this principle: align compute, network, and storage on one time axis, then annotate the likely bottleneck domain and verify the effect after a change.

That last step matters. A diagnosis is stronger when the metric improves after the suspected constraint is removed.

What metrics should you collect first?

Start with the smallest set that can distinguish waiting from working.

For the GPU, collect utilization, memory use, temperature, power, clock state, and hardware errors where supported.

For the network, collect throughput, latency, loss, retransmission, interface errors, and link status.

For storage, collect throughput, IOPS, latency, and capacity pressure.

For data loading, collect CPU utilization, worker activity, batch preparation time, and input throughput.

For the scheduler, collect allocated versus available accelerators, queue time, queue reason, quota state, and fragmentation indicators.

For the workload, collect step time, job state, checkpoint duration, and failure events.

Then make sure all of these data sources use consistent timestamps.

A platform example that connects these layers is Sensaka. The product name matters less than the operating principle: performance diagnosis should follow the workload from data supply through network, compute, scheduling, and service output.

What is the best troubleshooting order?

Start with the symptom, then eliminate layers in the order that is fastest to verify.

First, confirm that utilization is actually low for the workload period you care about.

Second, check card health so you do not spend an hour tuning software around a degraded device.

Third, check whether the job is actually running continuously or spending time queued, paused, or synchronizing.

Fourth, align network and storage metrics with the utilization drop.

Fifth, inspect CPU side data loading.

Only after those checks should you start changing model code or adding GPUs.

If the GPUs are waiting on storage, more GPUs make the waste larger. If the job is blocked by quota, kernel tuning will not help. If one card is degraded, changing the DataLoader will not fix it.

The most useful judgment is therefore simple: treat low GPU utilization as a cross domain operations problem until the evidence proves that the GPU itself is the limiting resource.