AI and GPU Workloads on Kubernetes: Use Expensive Accelerators Without Losing Control

Kubernetes can provide a common platform for AI inference, training, data preparation and batch scoring. It does not automatically make accelerator workloads efficient, portable, fairly scheduled, cost effective, or recoverable.

The objective is reliable business output from a controlled amount of accelerator capacity — the business metric should be cost per valid outcome, not cost per device-hour.

Use CPUs when latency is acceptable, throughput is modest, or the model is small — use accelerators only when measurement shows a material improvement in latency, throughput, or cost per request. A GPU can reduce elapsed time while increasing hourly cost.

Choose dedicated devices when performance and isolation matter more than utilization. Do not describe a time-shared GPU as equivalent to an exclusive one.

Models may load from a container image (reproducible, but slow pulls for large models), object storage (independent lifecycle, but download time and credentials to manage), or a PersistentVolume (fast local access, but storage attachment and backup dependencies).

Readiness should confirm the model is loaded and a sample inference succeeds — not merely that the HTTP process started. Liveness should never depend on a temporarily unavailable model registry or optional telemetry, or a shared outage can trigger a fleet-wide restart loop.

KServe provides Kubernetes custom resources and controllers for serving predictive and generative AI models, with deployment modes that integrate with serverless or high-density serving components. It can simplify serving, but it also introduces controllers and CRDs that must be backed up and recovered like any other platform dependency.

The HorizontalPodAutoscaler can scale supported workloads from resource, custom or external metrics — useful inference signals include queue depth, concurrent requests, tokens per second and accelerator saturation.

Scale-to-zero can cut cost for infrequently used models but introduces cold-start latency — evaluate model size, download time and customer timeout before using it for anything with a strict first-response deadline.

Kubeflow Trainer is a Kubernetes-native platform for distributed training and fine-tuning across frameworks including PyTorch, JAX and XGBoost. It doesn't remove the need to define checkpoint frequency, dataset access, retry behaviour, or accelerator topology.

Store checkpoints (model weights, optimizer state, training step, random state) outside the Pod filesystem — object storage, shared filesystem, or a managed training repository — and test restarting from a checkpoint after Pod, node and cluster loss, not just after a clean shutdown.

AI workloads may expose business data, customer prompts, training datasets, model weights and API credentials. A user who can create an arbitrary Pod with access to the model PVC or object-store identity may be able to copy the model — review effective privilege, not just resource names.

Monitor four layers: cluster (Pending Pods, scheduler latency), accelerator (GPU utilization, memory, temperature, errors), model server (queue depth, latency, token rate), and business (valid prediction rate, conversion, cost per request).

Useful unit costs include cost per 1,000 inferences, cost per million tokens, and cost per training run. Idle GPU capacity may be justified by latency objectives or failure reserve — label justified reserve separately from accidental waste, since usage-only chargeback can undervalue capacity reserved for availability.

Protect Kubernetes resources, serving/training CRDs, model registry metadata, model weights, training checkpoints, evaluation results, container images and Secrets. Model artefacts should normally exist outside the source cluster — a PersistentVolume containing the only model copy creates unnecessary recovery risk.

Some models can provide degraded CPU inference during accelerator failure — evaluate and load-test this fallback rather than assuming it works. See the disaster recovery plan guide for the broader recovery framework.