How to Reduce GPU Fragmentation When Scheduling AI Workloads

GPU fragmentation can be reduced by managing accelerators as structured resource pools instead of a flat card count. The source scheduling model uses standard resource specifications, heterogeneous pooling, whole-card and sliced resources, visible queue reasons, health-aware scheduling, priority and preemption, and resource reclamation. Together, those controls help the platform preserve useful resource shapes instead of leaving many individually free cards that cannot satisfy real jobs.

Fragmentation is therefore a scheduling and capacity problem. The total number of free GPUs can look healthy while the deployable capacity for one workload class is much smaller.

What is GPU fragmentation?

GPU fragmentation is the loss of usable capacity caused by free resources being split into shapes that do not match workload requirements. The source AI infrastructure material describes resource fragmentation as one of the reasons nominal capacity is compressed.

A simple example: eight GPUs are free, and a new training job requires eight compatible GPUs under one topology requirement. The eight free cards are spread across several nodes in a way that does not satisfy the request. The total free count is eight, but the deployable count for that job is zero, and that gap is fragmentation.

Why is total free GPU count misleading?

AI workloads request more than a number. A request can depend on:

Accelerator model
Memory size
Whole card or slice
Number of cards
Node placement
Topology
Health state
Tenant quota
Priority

The source scheduling layer explicitly manages multiple vendors and models and presents standardized resource specifications to users, so "one free GPU" is an incomplete unit of supply. A free low-memory card is not equivalent to a free high-memory card for every job. A degraded card should not be treated as normal supply, and a card split into smaller slices may not satisfy a full-card training task. Fragmentation appears when the remaining supply does not match the requested shape.

How do standard resource specifications help?

Standard resource specifications reduce unnecessary variation in how users request accelerators. The source scheduling model converts heterogeneous GPU and NPU resources into standard resource offerings. Instead of asking users to select arbitrary physical cards, the platform can provide approved resource classes, which makes demand easier to understand and schedule.

For example, the environment may have separate standard specifications for:

Full accelerator training
Multi-card training
Shared inference
Development

The source materials do not prescribe the exact class names or sizes. The operational value is that standard shapes cut down on one-off requests that are difficult to fit into the pool.

How do whole-card and sliced resources affect fragmentation?

The source scheduler supports both whole-card and sliced resource forms, and both can improve utilization when they are matched to the workload.

A lightweight inference workload may not need an entire accelerator. If it occupies a full card anyway, the unused portion is effectively unavailable to other work, so a sliced resource can improve efficiency where the supported hardware and workload permit it.

The opposite problem can also occur. If a card has been split into slices and a later training job requires the full device, the current allocation may prevent the job from starting. Slicing should therefore be governed by workload class and scheduling policy, with the aim of creating resource shapes that match demand rather than slicing everything.

Why should queue reasons be visible?

Queue reasons tell the operations team why demand cannot be satisfied. The source scheduler explicitly requires the waiting reason to be visible item by item, and that is one of the most important controls for fragmentation.

A queued job can be waiting because quota is exhausted, no matching accelerator specification is free, the required topology is unavailable, healthy cards are insufficient, a higher-priority task has the resource, or fragmentation prevents the requested shape. Each of those causes needs a different action. Without the queue reason, operators may see free GPUs and assume the scheduler is broken; with it, the capacity problem becomes explainable.

How does topology create fragmentation?

Topology creates fragmentation when the job requires resources that are close enough or connected in a specific way, but the free cards are distributed across incompatible locations.

The source infrastructure model emphasizes the training network and east-west communication, and it notes that one project may request an entire cluster group, so physical and network placement can matter. Eight cards scattered across several unsuitable nodes may be less useful than eight cards available in the required topology domain. The scheduler should therefore preserve topology when placing smaller jobs if future larger jobs depend on that topology.

The source does not prescribe a specific packing algorithm. It does support the requirement that scheduling consider the resource shape and training environment.

How does health-aware scheduling reduce fragmentation?

It prevents degraded cards from appearing as usable capacity. The source scheduler uses card-level health as an input and isolates degraded cards before allocation, which improves the accuracy of the capacity model.

Without health-aware scheduling, the system can appear to have enough free resources while one of those cards is not safe for production work. A job may then be assigned to the bad card and fail, causing repeated restarts and even more disruption. The source design therefore treats health as a scheduling prerequisite, and capacity should mean healthy capacity.

For the health model, how enterprises can monitor GPU health, ECC errors, temperature, power consumption, and degraded accelerator cards explains how a card moves from healthy to degraded and back into service.

How can priority and preemption help?

Priority and preemption can help the scheduler make better use of scarce resources when competing workloads have different importance. The source scheduling layer supports:

Priority
Preemption policy
Checkpoint protection
Fault isolation
Resume from checkpoint

The combination matters because preemption alone can waste work. If a lower-priority training job is interrupted, the platform should protect its progress where the workload supports checkpointing. The resource can then be released for higher-priority work, and the interrupted job can resume later. This adds flexibility without treating every allocation as permanent.

How can resource reclamation reduce fragmentation?

Resources need to return to the pool promptly when a workload finishes, fails or releases them. The source scheduling model presents a full flow from pool to task execution and includes task and resource bindings.

If completed or failed tasks leave stale allocations behind, the platform's free-capacity view becomes wrong, which creates artificial fragmentation. The scheduler, task state and metering state should therefore agree on whether the resource is still occupied. A card that is physically free but logically allocated cannot serve the next workload, so reconciliation matters.

How can repacking help?

Repacking means moving or restructuring smaller workloads so larger compatible resource blocks become available.

The source materials support fragmentation identification and optimization recommendations, but they do not specify a particular repacking algorithm. What the source does support is this: the platform can identify fragmentation and recommend optimization, and actual movement should follow the scheduling and workflow controls supported by the environment.

For interruptible work, checkpoint and rescheduling make repacking more practical. For latency-sensitive inference, moving the workload may have different service implications, so the optimization should consider workload type and service requirement.

How should fragmentation be measured?

The source does not define one universal fragmentation formula, so the metric should be tied to the workload classes the environment actually serves. A practical approach grounded in the source is to compare total free accelerators with deployable accelerators for each standard resource specification.

If the pool has 20 free cards but can satisfy only one 8-card training request because the rest are scattered across incompatible shapes, the difference is operationally meaningful fragmentation. The metric should therefore be aware of resource classes, since one global percentage can hide which jobs are affected.

How should full-card and sliced demand be balanced?

Use observed demand and service needs. The source supports both resource forms but does not state a universal split, so operations should monitor:

How many full-card jobs arrive
How many sliced workloads arrive
Queue time by specification
Idle rate by resource type
Fragmentation by resource type

If shared inference demand is high, more slicing may improve utilization. If large training demand is frequent, preserving enough full-card or multi-card capacity may matter more. The balance should come from real queue and utilization data.

How does quota affect fragmentation?

Quota can make free capacity unavailable to a specific tenant or project even when the hardware is technically idle. The source scheduler includes tenant quota and priority in the allocation model.

This is policy rather than physical fragmentation, but users experience something similar: GPUs appear free and the job stays queued. The platform should show whether the cause is quota or resource shape, so the operations team doesn't try to solve a policy problem with hardware expansion.

How does fragmentation affect cost?

Fragmentation lowers the amount of billable or useful service that installed hardware can produce. The source cockpit tracks:

Utilization
Idle rate
Card hours
Unit Token cost

If hardware sits free because it cannot satisfy queued demand, the organization owns capacity that is not being converted into service, and effective unit cost can rise. Fragmentation should therefore appear in both scheduling and operations views, because it is an economic problem as well as a capacity problem.

How can the AI assistant help with fragmentation?

The source AI assistant explicitly includes fragmentation identification and recommendations. A user can ask why resources are idle or why a job is waiting, and the assistant can combine:

Pool inventory
Resource specification
Queue state
Health
Quota
Topology

to explain the likely cause and then recommend a supported optimization. The assistant should not change allocations by itself if the action requires approval; the source keeps the same boundary for AI recommendations and infrastructure changes.

What should a fragmentation dashboard show?

A useful view can show:

Free cards by model
Free cards by resource specification
Whole-card capacity
Sliced capacity
Queued demand by specification
Queue reason
Healthy versus degraded cards
Fragmentation warning
Idle rate
Resource reclamation state
Optimization recommendation

From there, the operator should be able to drill into the workloads causing or affected by the fragmentation.

For the broader idle-resource view, how IT teams identify idle, underutilized, and overcommitted infrastructure resources explains how fragmentation fits into the wider resource-optimization model.

A platform example that combines heterogeneous pooling, standard specifications, queue visibility, health-aware scheduling, and fragmentation recommendations is Sensaka.

If I were managing a GPU pool, I would stop using total free-card count as the main availability KPI and report deployable capacity by the standard workload shapes users actually request. Fragmentation becomes visible as soon as those two numbers stop matching.