How do RDMA, RoCE, InfiniBand, network packet loss, and storage performance affect AI training performance?

RDMA, RoCE, InfiniBand, network quality, and storage performance affect AI training because distributed training is a data movement problem as much as a compute problem. GPUs can be healthy and available while spending substantial time waiting for other workers or for data to arrive from storage.

The practical rule is to analyze compute, network, and storage on the same timeline. A falling GPU utilization curve does not prove a GPU problem. It may be the visible symptom of packet loss, communication delay, storage latency, or slow data delivery elsewhere in the training chain.

What is RDMA?

Remote Direct Memory Access, or RDMA, is a data transfer model that allows memory-to-memory communication with reduced CPU and kernel networking involvement.

NVIDIA's Network Operator documentation describes RDMA as memory-to-memory transfer that bypasses the CPU and kernel networking stack, with support for InfiniBand and RoCE environments.

That matters because distributed training repeatedly moves data between workers.

If the CPU has to handle every part of that movement through a conventional networking stack, communication overhead can become significant at scale.

RDMA helps make the communication path more direct.

It does not make the network immune to congestion, bad configuration, physical errors, or topology problems.

RDMA is therefore a transport capability, not a guarantee that training will be fast.

What is InfiniBand?

InfiniBand is a high-performance, low-latency, RDMA-capable networking technology used in HPC and large-scale AI environments.

NVIDIA's DGX SuperPOD reference documentation describes InfiniBand as a high-performance, low-latency, RDMA-capable network technology for compute environments.

In an AI cluster, InfiniBand can provide the scale-out fabric connecting nodes that participate in distributed training.

The network needs more than high headline bandwidth.

The operational team still needs to monitor ports, link state, errors, congestion, latency, throughput, topology, and the relationship between fabric events and jobs.

A network can be "up" and still slow training.

That difference between availability and performance is critical.

What is RoCE?

RDMA over Converged Ethernet, or RoCE, carries RDMA traffic over Ethernet.

That lets organizations build RDMA-capable AI fabrics on Ethernet infrastructure when the design, adapters, switches, configuration, and operations support the required behavior.

RoCE and InfiniBand should not be reduced to a simple "Ethernet versus non-Ethernet" buying decision.

The real operational questions are:

Can the fabric provide the bandwidth the workload needs?

Can it control congestion?

Can it avoid or recover from packet loss in the way the deployment expects?

Can operators see the relevant NIC and switch telemetry?

Can the topology support the scheduler's placement decisions?

Can the team troubleshoot the fabric during a training slowdown?

The answers matter more than the acronym.

What does NCCL do in distributed training?

The NVIDIA Collective Communications Library, or NCCL, provides optimized collective communication primitives for multi-GPU and multi-node workloads.

NCCL includes operations such as all-reduce, all-gather, reduce-scatter, broadcast, and point-to-point communication patterns.

These collective operations are central to many forms of distributed deep learning.

During data parallel training, for example, workers often need to exchange or reduce gradients.

That means the slowest communication path can hold back the training step.

The GPU may finish its local computation and then wait for the collective operation to complete.

From the GPU chart, that waiting can appear as lower utilization.

From the network chart, it may appear as congestion, retransmission, or an imbalance across links.

The two views need to meet.

How does network packet loss affect training?

Packet loss can increase communication time and make distributed workers wait longer for each other.

The exact effect depends on the network technology, transport, collective library, congestion control, and workload.

Operationally, the important point is that a small network problem can affect a much larger set of GPUs.

If one node participates in an eight-node collective and its network path becomes slow, the other workers can wait at synchronization points.

That creates a multiplier effect.

One bad path can waste compute on healthy nodes.

Monitor loss and error counters at the NIC and switch layers where available.

Also monitor retransmission or congestion indicators appropriate to the fabric.

Then map the affected ports to the physical nodes and training job.

A packet-loss alarm with no job context tells the network team that something happened.

A packet-loss alarm tied to a 256-GPU training job tells the operations team why it matters.

How does latency affect AI training?

Latency affects training when workloads perform frequent communication or synchronization operations that cannot be hidden behind useful computation.

High bandwidth is important for moving large amounts of data.

Low and consistent latency is important for frequent coordination.

The balance depends on workload shape.

Large all-reduce operations may emphasize bandwidth.

Smaller frequent operations can be more sensitive to latency.

Tail latency also matters.

If most nodes communicate quickly but one path periodically becomes much slower, distributed steps can wait for that slow participant.

This is why percentile and outlier behavior can matter more than a simple average.

The same idea applies to storage latency.

Averages hide the pauses that make GPUs wait.

How does storage performance affect training?

Storage affects training through dataset reads, metadata access, model loading, checkpoint writes, and checkpoint restores.

The useful storage metrics are not only capacity.

Track throughput, IOPS, latency, queue behavior where available, and job-level read or write activity.

A large sequential dataset workload may care strongly about throughput.

A workload touching many small files may become sensitive to metadata and IOPS.

Checkpointing can create large bursts of writes.

Recovery can create large reads.

If multiple jobs checkpoint at the same time, the storage system may be healthy overall but still create enough contention to slow training.

That is why storage and scheduler behavior should also be connected.

The system can sometimes reduce contention by staggering checkpoint activity or understanding which workloads share a storage pool.

Can storage cause low GPU utilization?

Yes. Slow storage can leave GPUs waiting for the next batch.

The easiest way to test the hypothesis is correlation.

Plot GPU utilization and training step time.

Add storage throughput and latency.

Add CPU-side data loading if available.

If GPU utilization falls while storage latency rises and the next batch arrives later, storage becomes a strong candidate.

If storage metrics stay normal but network communication time increases, look at the fabric.

If both are normal and CPU-side data preparation is saturated, look at data loading.

This cross-domain method is also explained in why GPU utilization can be low.

The point is to avoid changing GPU settings before proving the accelerator is the constraint.

Why should compute, network, and storage use the same timeline?

A common timeline lets operators see sequence.

Suppose a job slows at 11:42:15.

At 11:42:10, one switch port begins reporting errors.

At 11:42:12, collective communication duration increases.

At 11:42:15, GPU utilization drops.

At 11:42:18, training step time doubles.

That sequence is much stronger evidence than four separate dashboards.

The same method works for storage.

A rise in storage latency followed by data-loader delay followed by GPU idle time creates a traceable chain.

Time synchronization therefore becomes an operational requirement.

Switches, NICs, servers, storage systems, schedulers, and application metrics need timestamps that can be compared accurately enough for the diagnosis.

What should be monitored on the training network?

Monitor the signals that can explain availability, capacity, and degradation.

Useful areas include:

Link state and speed
Throughput
Packet or symbol errors where exposed
Loss indicators
Retransmission or retry behavior
Congestion telemetry
Latency
NIC health
Optics health where relevant
Port utilization
Topology
Collective communication duration where available

The exact counters differ by fabric.

Do not build one hard-coded metric list and assume it works across InfiniBand and RoCE.

Normalize the operational meaning while preserving fabric-specific detail.

For example, both fabrics need a concept of link health, but the underlying counters may differ.

How should topology affect scheduling?

Topology should influence placement when communication cost matters.

Eight GPUs on one server are not equivalent to eight GPUs spread across distant nodes if the workload expects tight communication.

Likewise, a group of nodes sharing a fast fabric domain may be a better placement than the same number of nodes reached through a less favorable path.

The scheduler should use topology as a constraint behind the resource request.

Users should not need to memorize switch names.

They should be able to request a training resource class, and the platform should satisfy its topology requirements.

This is covered more deeply in GPU resource pooling and scheduling in Kubernetes.

How do you distinguish a network bottleneck from a storage bottleneck?

Use workload phase and correlated metrics.

If the slowdown happens during model synchronization or collective operations and network indicators change at the same time, investigate the network.

If the slowdown happens while reading data or writing checkpoints and storage latency changes, investigate storage.

If both change together, check whether the storage path shares the same network fabric or switches.

A storage problem can actually be a network problem when the storage is remote.

Do not stop at organizational boundaries.

The storage team may own the array while the network team owns the path, but the training job experiences one data supply chain.

What does good operations look like?

Good operations can move from a slow training job to the constrained domain quickly and show the evidence.

The operator should be able to answer:

Which GPUs are in the job?

Which nodes host them?

Which network paths connect them?

Which storage system supplies the data?

What changed when performance dropped?

Is the bottleneck compute, network, storage, or data loading?

What happened after tuning?

A platform example that correlates these domains is Sensaka.

If I were troubleshooting a distributed training slowdown, I would not start by asking which infrastructure team owns the problem. I would start with one synchronized timeline across GPU activity, collective communication, NIC and switch telemetry, storage throughput, storage latency, and training step time. The first domain whose behavior changes before the slowdown is where I would investigate first.