What are GPU quotas, reservations, priorities, and preemption, and when should each be used?

GPU quotas, reservations, priorities, and preemption solve four different scheduling problems. Quotas limit how much a tenant or project can consume. Reservations protect capacity for planned work. Priorities decide which workload should receive scarce capacity first. Preemption allows a higher-priority workload to reclaim resources from lower-priority work when the policy permits it.

The source scheduling model supports tenant quotas, task queues, priority, preemption, checkpoint protection, resource reclamation, and visible queue reasons. The important operating principle is to use these controls together instead of expecting one mechanism to solve every capacity problem.

What is a GPU quota?

A GPU quota sets the resource boundary for a tenant, project, or other approved ownership scope.

The source scheduling design describes tenant quotas as a hard constraint that prevents disorderly competition for expensive accelerator resources.

The source Q&A also says quotas can be defined using several dimensions, including:

GPU quantity
Accelerator memory
CPU and memory
Concurrent tasks
Monthly accelerator card hours

The exact quota model depends on the organization's management policy.

A quota is useful when several teams share one infrastructure pool and the platform needs to stop one team from consuming all available capacity.

It is also useful for budget control.

A project may be technically allowed to run on the shared cluster but limited to a defined amount of capacity or monthly consumption.

When should GPU quotas be used?

Use quotas when you need a hard boundary between tenants, projects, or teams.

Typical situations include:

Multiple departments share one GPU cluster.

External or internal tenants need isolated resource limits.

Projects have approved capacity budgets.

A development team should not consume production capacity without authorization.

Monthly accelerator usage must remain within an approved budget.

Quotas are also helpful when demand is unpredictable.

Without a limit, a burst of submitted jobs from one project can fill the queue and consume the entire resource pool.

The source model therefore puts quota validation before queue scheduling.

A task first has to be allowed to request the resource.

Only then should priority and scheduling policy decide when it receives the resource.

What is a GPU reservation?

A reservation protects capacity for a planned workload, project, service, or time window.

The source materials explicitly distinguish reserved capacity from free and deployable capacity in planning, although they do not prescribe one universal reservation product or algorithm.

Operationally, a reservation means some capacity is committed before it is actively consumed.

For example:

A critical training project needs 32 GPUs tomorrow night.

A production inference service needs guaranteed headroom for peak traffic.

A hardware validation program requires a specific accelerator class during a maintenance window.

The reserved capacity should not appear as generally free to other workloads if using it would jeopardize the commitment.

That makes reservation different from quota.

Quota says, "You may consume up to this amount."

Reservation says, "This amount is being held for this approved purpose."

When should reservations be used?

Use reservations when the cost of not having capacity at the required time is high.

Reservations make sense for:

Scheduled training runs
Critical production inference
Known business events
Migration windows
Large model evaluation
Hardware validation
Capacity committed to a customer or tenant

Do not reserve capacity unnecessarily.

Reserved but unused GPUs can become expensive idle capacity.

The operations dashboard should therefore show reserved capacity separately from:

Installed capacity
Allocated capacity
Free capacity
Deployable capacity

This makes intentional protection visible instead of misclassifying every reserved card as waste.

What is GPU priority?

Priority determines the order in which competing workloads should be considered for resource allocation.

The source scheduling model supports task queues with priority and makes queue reasons visible.

A higher-priority job can move ahead of a lower-priority job when both are waiting for the same constrained resource.

Priority is useful because not every workload has the same business importance.

Examples:

Production inference may be higher priority than experimental fine tuning.

A customer-facing recovery task may be higher priority than a development benchmark.

A deadline-sensitive training job may be higher priority than a background batch job.

Priority does not necessarily interrupt work that is already running.

That is the difference between priority and preemption.

When should priority be used without preemption?

Use priority alone when waiting order matters but interruption is too risky or expensive.

This is common for long-running training tasks.

Suppose two jobs need the same accelerator class.

Job A is already running.

Job B arrives with higher business priority.

If Job A cannot checkpoint safely, interrupting it may lose many hours of compute.

A priority-only policy can let Job A finish while placing Job B ahead of other queued work.

This gives the organization a business-aware queue without creating unnecessary training loss.

The source scheduling design explicitly notes that high-priority preemption must be balanced against protection of running tasks.

That is why priority and preemption should be separate controls.

What is GPU preemption?

Preemption allows the scheduler to reclaim resources from one running workload so another workload can use them.

The source scheduling model supports high-priority preemption and links it with checkpoint protection and graceful task recovery.

Preemption is useful when capacity is scarce and some workloads are explicitly allowed to be interrupted.

For example:

A low-priority batch training task is running.

A high-priority incident-recovery workload arrives.

The lower-priority task supports checkpointing.

The platform saves or uses the latest checkpoint, stops the lower-priority task, releases the GPUs, and allocates them to the higher-priority task.

Later, the interrupted task can resume.

Preemption therefore depends heavily on workload recoverability.

When should GPU preemption be used?

Use preemption when all three conditions are true.

First, the higher-priority workload has a meaningful reason to start sooner.

Second, the lower-priority workload is approved for interruption.

Third, the interrupted work can be protected well enough that the cost is acceptable.

Good candidates include:

Interruptible batch jobs
Jobs with frequent checkpoints
Development workloads
Lower service tiers
Flexible background processing

The source design specifically recommends distinguishing preemptible and non-preemptible tasks.

That distinction should be part of the workload submission policy.

Do not decide it for the first time during an incident.

When should preemption be avoided?

Avoid or heavily restrict preemption for workloads where interruption creates disproportionate loss.

Examples include:

Long-running training with no valid checkpoint
Critical production inference with no redundancy
Stateful workloads that cannot restart safely
Jobs where rescheduling changes an approved experimental condition
Tasks with external dependencies that cannot be reconstructed

The source Q&A warns that high-priority preemption can cause training loss and recommends checkpoint and graceful termination mechanisms.

The key lesson is that high priority does not automatically mean safe to preempt.

Priority expresses importance.

Preemptibility expresses recoverability and policy.

Both are needed.

How do quotas and priorities work together?

Quota decides whether the workload is entitled to request the resource.

Priority decides its place among eligible workloads.

Example:

Project A quota: 16 GPUs.

Project B quota: 16 GPUs.

Project A is already using all 16.

A new Project A job asks for 8 more.

The job can be rejected or queued because the project quota is exhausted.

Its priority does not override the quota unless the organization explicitly defines such a policy.

Now imagine both projects are within quota and are waiting for the same GPU class.

Priority can determine which job is served first.

Keeping those controls separate makes scheduling explainable.

How do reservations and quotas work together?

Reservations protect a known amount of future or guaranteed capacity, while quotas limit overall consumption.

A project can have both.

Example:

Project A quota: 64 GPUs.

Project A reservation for tonight: 32 GPUs.

The project can consume up to 64 according to policy, but 32 are specifically protected for the planned run.

The source materials do not define a universal interaction rule between reservation and quota.

That relationship should therefore be specified in the enterprise scheduling policy.

The important operating requirement is that reservations remain visible so supposedly free capacity is not double-promised.

How do reservations and priorities work together?

A reservation should normally protect the committed resource independently of ordinary queue priority.

Otherwise the reservation is not a real guarantee.

However, enterprises may define emergency override policies.

The source materials do not prescribe such an override model.

The safer source-grounded design is to make the policy explicit and auditable.

If a reserved pool is overridden, the platform should record:

Who authorized it
Which workload used the capacity
Which reservation was affected
What service risk was accepted

This is especially important when the reservation belongs to another tenant or customer commitment.

How does checkpointing change preemption policy?

Checkpointing reduces the cost of interruption.

The source scheduler includes checkpoint protection, fault isolation, and resume from checkpoint.

A workload with a recent checkpoint can release its resources with less lost work.

That makes it a better preemption candidate.

A workload whose last checkpoint is six hours old is much more expensive to interrupt.

The scheduling policy can therefore consider checkpoint state in addition to static priority.

The source does not define a specific scoring algorithm.

The supported principle is that running-task protection and checkpoint capability should influence preemption decisions.

For failure recovery, how AI infrastructure can automatically recover training jobs after a GPU or server failure explains the same checkpoint mechanism from the resilience side.

How should queue reasons be shown?

The source design says queue reasons should be visible item by item.

That is essential when quotas, reservations, priorities, and fragmentation all exist at the same time.

A job can be waiting because:

Quota exhausted
Matching GPU type unavailable
Reserved capacity unavailable to this project
Higher-priority work is ahead
Required topology is unavailable
Healthy capacity is insufficient
Resource fragmentation prevents placement

The user should see the actual reason.

Otherwise the same queue can look like a capacity problem when it is really a policy problem.

How do health checks affect these policies?

The source scheduling architecture includes card-level health as an input to resource allocation.

That means quota and reservation numbers should refer to usable, healthy resources rather than installed cards only.

If a tenant has a reservation for eight GPUs but one reserved card becomes degraded, the platform needs to identify the shortfall and find suitable replacement capacity where policy allows.

A degraded card should not remain schedulable merely because a reservation expects it.

Health-aware scheduling makes the promises more realistic.

For the card-level model, how enterprises can monitor GPU health, ECC errors, temperature, power consumption, and degraded accelerator cards explains how health state should affect scheduling.

How should enterprises choose between the four controls?

Use the control that matches the problem.

Use quota for entitlement and consumption boundaries.

Use reservation for guaranteed or planned capacity.

Use priority for waiting order.

Use preemption for controlled interruption when urgent work must reclaim capacity.

Do not use priority as a substitute for quota.

Do not use reservations for every workload.

Do not enable preemption without a recovery plan.

Do not assume a large quota guarantees immediate capacity.

A platform example that combines tenant quotas, priority, preemption, health-aware scheduling, and resource recovery is Sensaka.

If I were defining the scheduling policy, I would make every submitted workload declare four things: who owns it, what quota applies, what priority it has, and whether it can be preempted safely. Reservations can then be added for workloads that need guaranteed capacity at a specific time. That produces a scheduling model users can understand and operations can audit.