How to Manage Multi Vendor GPUs and NPUs

The clean way to manage GPUs and NPUs from multiple vendors is to separate the problem into layers: physical discovery, health normalization, resource abstraction, scheduling, and workload telemetry. Do not pretend every accelerator is identical. Give users a standard way to request capability while keeping vendor and model differences visible to the platform.

Kubernetes can help with device exposure and scheduling, but it does not remove the need for vendor drivers, device plugins or DRA drivers, firmware management, health collection, and operational policies. The management layer still has to know what each card is, whether it is healthy, and which workloads can actually run on it.

Why is multi vendor accelerator management difficult?

Multi vendor accelerator management is difficult because the devices differ at more than the hardware name.

They can have different drivers, runtimes, firmware, monitoring interfaces, memory sizes, precision support, partitioning models, topology requirements, and framework compatibility.

Even health metrics are inconsistent.

One vendor may expose memory temperature directly. Another may expose only a board temperature. ECC terminology can differ. Power metrics can come from different interfaces. Firmware fields can change between versions.

This means the first mistake is trying to solve heterogeneity only at the dashboard level.

A single page that lists every card is useful, but it does not make the devices operationally interchangeable.

You need a common data model underneath.

What should the common accelerator inventory contain?

A useful accelerator inventory should identify each physical card, its host, its capabilities, its software dependencies, and its health.

At minimum, keep the vendor, model, serial number or unique device identifier, host node, slot or bus location, memory capacity, firmware, driver relationship, health state, temperature, power, and allocation state where those values are available.

For scheduling, add operational attributes such as whether the card is full size or partitioned, whether it belongs to a particular topology domain, whether it is reserved, and whether it is considered degraded.

For governance, link the card to the physical server, rack, data center, owner, maintenance status, and workload history.

This is where in band and out of band monitoring work together.

The BMC is useful for server level hardware state. Vendor runtime libraries or Agents may be required for card level utilization and memory metrics. Kubernetes provides the workload binding. The resource inventory has to join those views.

How does Kubernetes expose GPUs and other accelerators?

Kubernetes exposes specialized hardware through vendor integrations such as device plugins, and newer clusters can also use Dynamic Resource Allocation.

The device plugin framework lets a vendor advertise devices as node resources to the kubelet. Kubernetes can then schedule a Pod that requests that resource onto a node that has available capacity.

This model is widely used for GPUs.

Dynamic Resource Allocation, usually shortened to DRA, adds a more expressive way to request devices and device properties. As of Kubernetes 1.35, the main DRA capability is stable and enabled by default according to the Kubernetes documentation.

That matters for heterogeneous infrastructure because a workload can express more than a simple count of generic devices.

However, Kubernetes still needs a working driver and vendor integration.

DRA does not make an unsupported accelerator supported. It gives the scheduler and workload a better resource contract.

Should users request a vendor specific GPU name?

Users should request vendor specific resources only when the workload truly depends on that vendor or model.

For many platform users, the better interface is a standard specification.

For example, an organization might define resource classes such as:

Large training, 8 full accelerators with high memory
Standard training, 4 full accelerators
Inference, 1 full accelerator
Shared inference, partitioned accelerator
Development, CPU only

Those classes are examples, not universal standards.

The point is that the business user asks for the capability they need, while the platform maps the request to hardware that satisfies it.

If a workload requires a specific CUDA version, a specific model architecture, or a particular memory size, that requirement should remain explicit.

Abstraction should reduce unnecessary hardware knowledge. It should not hide compatibility constraints.

How do you decide whether two accelerator models belong in the same pool?

Two accelerator models belong in the same logical pool only if the workloads assigned to that pool can run correctly and predictably on both.

Check software compatibility first.

Can the same framework and model run on both? Are the necessary drivers and libraries available? Does the workload require a vendor specific runtime?

Then check hardware capability.

Memory capacity is often a hard constraint. A model that requires more memory than one card provides cannot be scheduled there simply because both devices are accelerators.

Topology can matter for multi card training. Interconnect characteristics can change training performance significantly.

Performance differences matter for SLOs and cost. If two models deliver very different throughput, treating them as one undifferentiated resource can make queue time and completion estimates misleading.

The best abstraction usually keeps the user interface simple but preserves resource flavors underneath.

Kueue, for example, supports ResourceFlavor objects and quotas that can represent different GPU models or other resource variations.

How should health affect scheduling?

Health should be a scheduling input, not only an alarm after the job starts.

A card can be technically online but still be a poor scheduling target.

Examples include increasing ECC errors, repeated resets, persistent high temperature, degraded clock behavior, or recurring driver failures.

If the management layer marks such a card as degraded, the scheduler should avoid assigning new work to it.

The exact action depends on risk.

A warning may only reduce scheduling priority. A more serious condition may mark the node unschedulable. A confirmed failure may trigger task migration, checkpoint recovery, and a repair work order.

This is one of the strongest reasons to connect hardware management with the scheduler.

If hardware monitoring lives in one tool and scheduling in another with no shared health state, the scheduler can keep placing jobs on a card that operations already know is unreliable.

How do quotas work in a heterogeneous pool?

Quotas should control both how much a team can consume and which resource classes it is allowed to consume.

A simple count of "10 GPUs" may be misleading if the environment contains very different accelerator models.

One approach is to set quotas by resource flavor or specification. A team may receive a quota for a particular training class and a separate quota for inference.

Another approach is a credit model where different resource types consume different amounts of a shared budget.

Kueue supports cluster level quotas, resource flavors, borrowing, lending, and fair sharing. That gives platform teams a Kubernetes native way to govern batch and AI workloads across teams.

Whatever method you choose, keep the billing and quota definitions aligned.

If the scheduler counts one resource one way and the cost system counts it another way, users will dispute the results.

How do full cards, partitions, and sharing fit together?

Full card allocation, hardware partitioning, and time sharing are different operating models and should be exposed as different resource types.

Full card allocation provides the clearest isolation. It is often preferred for large training jobs or workloads with predictable performance requirements.

Hardware partitioning, such as NVIDIA Multi Instance GPU on supported hardware, divides a physical GPU into isolated instances with dedicated compute and memory resources.

Time sharing lets multiple workloads use the same physical GPU over time. It can improve utilization for lighter workloads, but performance isolation is different from hardware partitioning.

Do not label all three simply as "shared GPU."

Users need to understand what performance and isolation they are buying.

The deeper scheduling mechanics are covered in GPU resource pooling and scheduling in Kubernetes.

How should multi vendor monitoring be designed?

Multi vendor monitoring should normalize common concepts while preserving vendor specific detail.

A common health schema might include status, temperature, power, memory usage, error count, utilization, and firmware.

Under that common layer, retain the raw vendor fields.

This is important for troubleshooting. A normalized "warning" is useful for dashboards, but an engineer may need the original sensor name, error code, or vendor event.

Collection frequency also needs thought.

Critical failure indicators may deserve faster polling or event driven collection. Slow changing inventory fields such as serial numbers do not.

The system should also make collection failures visible. If a driver upgrade breaks a metric exporter, the platform should show that the metric source is unavailable rather than silently reporting an old value.

Can one platform manage both GPU and NPU resources?

Yes, one platform can manage both if it treats them as accelerator resources with vendor specific adapters under a common resource model.

The common layer can cover identity, health, capacity, allocation, workload binding, quota, and cost.

The adapter layer handles vendor specific drivers, APIs, metrics, and deployment logic.

That is the scalable architecture.

Trying to force every vendor into one hard coded interface usually creates a brittle system. A plugin or adapter model makes it easier to add new accelerator families without redesigning the entire platform.

A platform example using this general model is Sensaka.

If I were building a heterogeneous accelerator environment, I would not start with a "single pane of glass" project. I would start with a trustworthy card inventory and a resource classification model. Once the platform can answer what each card is, whether it is healthy, what it can run, and who is using it, the unified view becomes useful instead of cosmetic.