GPU Resource Pooling and Scheduling in Kubernetes

GPU resource pooling in Kubernetes means turning a set of physical accelerators into managed capacity that workloads can request through policy instead of selecting individual servers by hand. Kubernetes provides the basic device scheduling layer, while a complete AI scheduling model usually adds resource classes, job queues, quotas, topology rules, health awareness, and resource reclamation.

The key distinction is between "Kubernetes can see a GPU" and "the organization can operate a GPU pool." The first is a device integration problem. The second is a resource management problem.

How does Kubernetes see a GPU?

Kubernetes sees specialized devices through integrations that advertise those devices to the kubelet and scheduler.

The traditional mechanism is the Device Plugin framework. A vendor plugin discovers supported devices on a node and reports the corresponding resource capacity. A Pod can request that resource, and Kubernetes schedules the Pod to a node that has enough available capacity.

Kubernetes documentation lists GPU scheduling support through device plugins and describes the device plugin framework as stable since Kubernetes 1.26.

Newer clusters can also use Dynamic Resource Allocation.

DRA gives workloads a richer way to request devices and lets drivers describe device properties and allocation behavior. As of Kubernetes 1.35, the main DRA feature is documented as stable and enabled by default.

The important practical point is that Kubernetes still relies on vendor drivers and device integration.

It does not discover every accelerator in existence without help.

For a heterogeneous environment, read how to manage multi vendor GPUs and NPUs before designing the pool.

What is a GPU resource pool?

A GPU resource pool is a logical supply of accelerator capacity managed under common rules.

The physical cards may sit in different nodes. They may have different models. Some may be full cards. Some may be partitioned. Some may belong to a training cluster while others are reserved for inference.

The resource pool creates a management layer above individual devices.

Instead of telling a user, "Run on server 27, GPU 3," the platform can offer a resource class such as "four high memory training GPUs."

The scheduler then chooses eligible devices.

A useful pool tracks at least five things:

What resources exist
Which resources are available
Which resources are healthy
Which workloads are allowed to use them
How resources are reclaimed after use

Without those controls, "pooling" is mostly a dashboard label.

What is the difference between device scheduling and job scheduling?

Device scheduling decides where a Pod can run. Job scheduling decides when a larger workload should be admitted and how scarce capacity is shared between competing jobs.

This distinction matters for AI.

A distributed training job may need many Pods and many GPUs at the same time. Starting only part of the job may be useless.

A platform may also need tenant quotas, queue priorities, borrowing rules, and preemption.

Kubernetes core scheduling is focused on placing Pods.

Kueue is a Kubernetes native project designed for job queueing and resource admission. It can manage quotas and sharing for batch, HPC, and AI workloads, and it supports resource flavors for different hardware types.

That means a common architecture is:

Kubernetes handles node and Pod scheduling.
Vendor plugins or DRA drivers expose accelerator devices.
Kueue or another batch scheduler handles queueing and quota admission.
The operations layer contributes health, topology, cost, and business context.

Those layers should cooperate rather than duplicate each other.

How should GPU resource classes be designed?

GPU resource classes should describe capability that matters to the workload, not just the marketing name of the card.

A useful class might encode memory size, accelerator family, count, topology, sharing mode, CPU, and system memory.

For example, a platform can define separate classes for large distributed training, smaller training, dedicated inference, shared inference, and development.

Do not create too many classes.

If users face 40 nearly identical options, the abstraction has failed.

At the same time, do not hide a constraint that affects workload success. If one model only runs on a specific software stack, the class needs to preserve that compatibility.

Resource flavors are useful here.

Kueue can represent different versions of a resource, such as different GPU models, and can assign quotas to those flavors.

DRA can also make device properties part of the allocation model.

The result is a user interface that is simpler than the hardware inventory without pretending the hardware is identical.

How do quotas work for GPU pools?

Quotas limit how much scarce accelerator capacity a tenant or team can consume.

A static quota reserves a defined amount for a team.

Shared quota models can let teams borrow unused capacity from each other. Fair sharing can prevent one team from consuming all spare resources indefinitely.

Kueue supports nominal quota, borrowing, lending, and preemption policies across ClusterQueues and cohorts.

This is useful because GPU demand is often bursty.

A research team may need large capacity for a few hours, then little for the rest of the day. Strictly partitioning every GPU by department can create idle hardware.

Borrowing makes the pool more efficient, but it needs clear reclaim rules.

If Team A borrows Team B's idle capacity and Team B suddenly needs it back, the scheduler needs a policy for what happens next.

That is where priority and preemption become operationally important.

How should preemption work for training jobs?

Preemption should protect useful work before reclaiming the resource.

For a short stateless inference task, restarting may be inexpensive.

For a training job that has run for hours, abrupt eviction can waste a large amount of compute unless the workload has a recent checkpoint.

A better flow is:

Identify the lower priority workload.
Trigger or verify a checkpoint where supported.
Wait for the checkpoint to complete.
Evict or suspend the workload.
Release the GPUs.
Later resume from the checkpoint.

This is not purely a Kubernetes scheduler function. The training framework, workload controller, storage system, and queue manager may all participate.

That is why "support preemption" is not enough as a requirement.

Ask what happens to the work already performed.

What is health aware GPU scheduling?

Health aware scheduling removes or deprioritizes accelerators that are technically visible but operationally unreliable.

The scheduler should not wait for a card to fail completely.

A degraded card may show increasing ECC errors, unstable temperature, repeated resets, abnormal clock behavior, or driver failures.

The operations system can convert those observations into a health state.

The scheduling layer can then exclude the card from new allocations or mark the node unschedulable.

This creates a useful loop:

Hardware monitoring detects degradation.
The resource model marks the device.
The scheduler avoids it.
Existing workloads are protected or moved.
A repair workflow is opened.
The device returns to the pool only after validation.

This is one reason an AI data center needs deeper hardware visibility than a generic container cluster.

How do GPU partitions fit into the pool?

GPU partitions let one physical accelerator be exposed as multiple smaller resources, but the isolation model depends on the partitioning technology.

NVIDIA Multi Instance GPU, on supported hardware, partitions a physical GPU into isolated instances with dedicated compute and memory resources.

Time sharing is different. It allows multiple workloads to use a GPU over time, but it does not provide the same hardware partitioning model.

From a scheduling perspective, these should be separate resource classes.

A user requesting a dedicated full GPU should not accidentally receive a time shared resource.

A user requesting a small inference slice should not reserve an entire large card if the hardware can be partitioned safely.

Pool efficiency improves when the resource shape matches the workload.

Why does topology matter for multi GPU jobs?

Topology matters because communication between GPUs can become a major part of distributed training time.

Two sets of eight GPUs may look identical by count but behave differently if one set has better local interconnect or a better network path.

Topology aware scheduling tries to place tightly coupled workloads where communication is efficient.

That can mean keeping a multi GPU job within the same node, the same high speed fabric domain, or another defined topology boundary.

Kueue also provides topology aware scheduling capabilities for workloads that need placement based on data center topology.

The operational point is not to make every user learn the topology.

The scheduler should use topology as a constraint behind the resource class.

How do you know why a GPU job is waiting?

A good queue should expose the specific reason a workload is waiting.

"Pending" is not enough.

Useful reasons include:

Tenant quota exhausted
No matching accelerator flavor available
Enough total GPUs exist, but not enough are available in the required topology
The requested full card class is exhausted
A node is blocked because of health
A higher priority workload has reserved the capacity
An admission check is waiting for capacity provisioning

This information changes operator behavior.

If the problem is quota, adding hardware will not fix it. If the problem is fragmentation, buying more GPUs may not be necessary. If the problem is a health exclusion, the repair workflow should be the priority.

Queue transparency is part of the product, not just a debugging feature.

How should resources be reclaimed?

GPU resources should be reclaimed automatically when the workload finishes, times out, or is deliberately suspended.

Orphaned allocations are expensive.

A complete system should reconcile the scheduler state, Pod state, device allocation, and metering state.

If a job says it is finished but a resource remains allocated, the pool capacity is wrong.

If a Pod disappears but the metering system keeps charging the project, the bill is wrong.

If a node fails and the resource is never removed from available capacity, the next job may be assigned to a dead device.

Resource reclamation is therefore both a scheduling function and a data consistency function.

What should I build first?

Build the resource truth first, then the scheduling policy.

You need reliable device discovery, health state, node relationships, resource classes, and workload bindings before sophisticated queueing becomes trustworthy.

Then add quotas, sharing, topology rules, checkpoint aware preemption, metering, and optimization.

A platform example that connects those layers is Sensaka.

If I were operating a small homogeneous cluster, I would keep the model simple and use native Kubernetes GPU scheduling with clear quotas. If I were operating a multi tenant, multi model, heterogeneous AI cluster, I would add an explicit resource pool, job admission layer, health aware scheduling, and cost accounting. Complexity is justified when it protects scarce accelerators from being idle, fragmented, or assigned to the wrong workload.