How Multiple Teams and Tenants Can Securely Share GPU Clusters

Multiple teams or tenants can securely share expensive GPU infrastructure by separating identity, permissions, resource entitlement, workload placement, service access, metering, and audit. The source design treats multi-tenancy as both a people problem and a compute problem: tenants have members and permissions, and they also have projects, quotas, tasks, model services, Token usage, and cost boundaries.

The goal is to share the expensive infrastructure without turning the environment into one unrestricted resource pool. A user should receive the capacity their project is allowed to use, see only the objects they are authorized to see, and leave a traceable record when they submit, change, or consume services.

What does multi-tenant GPU sharing need to isolate?

The source model isolates tenants across several operational layers. For people, that means tenant members, project members, roles, and permissions. For compute, it covers resource quotas, task ownership, resource pools, and scheduling. Model services include models, knowledge bases, applications, and access keys, and consumption covers accelerator card hours, Token usage, and cost.

Compute isolation alone is not enough. Two teams can have separate GPU quotas and still have a data-governance problem if one can see the other's model, work order, or knowledge base, so the tenant boundary has to travel through the full operating chain.

Why should tenants and projects both exist?

A tenant is a broader ownership and isolation boundary, and a project is a more specific working and accounting unit inside that tenant. The source Q&A describes multi-tenancy as involving both people and compute, including members, permissions, projects, quotas, tasks, model services, Token usage, and cost isolation.

This allows an organization to group several projects under one tenant while still tracking consumption and responsibility at project level. For example, a Research Department tenant might contain Project A for foundation model fine tuning and Project B for vision inference. The tenant provides the organizational boundary, and the project provides the workload and cost boundary.

How should role permissions work?

Use least privilege. The source governance model includes organizational hierarchy, role permission matrices, tenant membership, project membership, and separate authorization for sensitive operations, so a user should receive only the permissions required for their role.

For example, a tenant administrator can manage members and some project settings inside the tenant, while a project user can submit workloads inside the project quota. An operations engineer can manage infrastructure health, and a security administrator can manage sensitive policies. A vendor account can access only approved devices or work orders for a limited period.

The source Q&A explicitly says a tenant administrator can manage members, projects, and some permissions inside the platform administrator's authorized boundary but cannot cross into other tenants or global resources. That is the correct multi-tenant principle.

How should GPU quotas enforce fairness?

Quota should be checked before the workload enters normal scheduling competition. The source scheduler treats tenant quota as a hard constraint, which prevents one tenant from submitting enough work to consume the whole pool simply because its tasks have been waiting longer.

According to the source Q&A, quota can be based on GPU count, accelerator memory, CPU and memory, concurrent jobs, or monthly card hours, and the organization can choose the dimensions that match its operating model. A hard capacity quota protects infrastructure, and a usage quota can also support budget control.

How should priorities work across tenants?

Priority should be set by policy, whatever order users happen to submit in. The source scheduler supports queue priority and preemption, so an enterprise can distinguish production from development, critical customer service from background batch, and urgent incident work from flexible research.

Priority needs governance, however. If every tenant can mark every job "highest priority," the control has no value. The source does not prescribe a detailed priority-administration policy. A practical design grounded in the source is to bind high-priority levels to roles, project classes, or approved workload types and preserve the decision in the audit history.

Should teams share whole GPUs or GPU slices?

The source scheduling design supports both whole-card and sliced resources, and the choice depends on the workload and supported hardware. Whole-card allocation is appropriate when the task requires exclusive accelerator resources, predictable performance, or a full device. Sliced resources can improve efficiency for lighter workloads where supported and where the required isolation behavior is acceptable.

The source also identifies single-card multi-container isolation and shared-card interference as technical challenges, so shared GPU use should not be treated as automatically equivalent to dedicated allocation. Monitor the shared workload behavior, keep the resource form visible, and use standard resource specifications so users understand what they are requesting.

How should heterogeneous GPUs be shared?

Normalize them into standard resource specifications while preserving the actual model and health underneath. The source platform supports heterogeneous pooling across multiple accelerator vendors and models, and users can request a standard resource class instead of manually selecting individual physical cards. This makes sharing easier because the platform can manage card type, card count, CPU, memory, and whole or sliced form.

The source also notes that multi-vendor scheduling is technically difficult because drivers, runtimes, and device plugins differ. The standard resource layer should therefore simplify the request while retaining compatibility rules, without pretending every card is identical.

How should card health protect tenants?

Health should be checked before allocation. The source scheduling design explicitly isolates degraded accelerator cards before tasks are assigned, which matters even more in a shared environment. A tenant should not receive a card simply because it is technically visible and currently free.

If the card is degraded, the platform should remove it from normal scheduling or apply the approved health policy. That prevents one tenant from becoming the unlucky recipient of known-bad capacity, and it improves fairness because usable supply is measured from healthy resources instead of installed inventory.

How should workload bindings be tracked?

The source container-and-GPU model keeps two-way mappings between accelerator cards and containers and preserves a binding timeline for audit. The platform can therefore answer which workload is using a GPU, which GPU a container is using, which tenant owns the workload, and when the binding started and ended.

Those relationships are essential for shared infrastructure because they support incident impact analysis, cost allocation, security review, resource reclamation, and historical audit. If a hardware error occurs, operations can identify the exact tenant and task affected without exposing unrelated tenant data.

How should model-service access be isolated?

The source model-service layer separates models, knowledge bases, applications, keys, quotas, and usage by tenant and project. API keys are bound to project tags so Token consumption can be attributed correctly, which gives the service layer its own access boundary.

A tenant should not be able to call another tenant's private model service or retrieve its knowledge base simply because both use the same underlying GPU cluster. The compute pool can be shared while service and data ownership remain isolated.

How should knowledge bases be isolated?

The source RAG design explicitly says knowledge-base permissions are isolated by tenant, which is a critical security control. Two tenants can use the same approved model while retrieving from different enterprise knowledge, because the model is shared infrastructure and the knowledge base remains tenant-specific.

This separation is one of the benefits of decoupling the model and knowledge lifecycles. The same principle should apply to operational documents and work orders, so users retrieve only the knowledge their role and tenant allow.

How should usage be metered?

The source metering model attributes consumption by tenant, project, model, and accelerator type, and it tracks accelerator card hours and Token usage. That gives the organization a clear answer to the basic shared-infrastructure question of who used what.

Without metering, shared GPUs can become politically difficult to manage because every team believes another team is consuming the expensive capacity. Metering turns the discussion into data.

For the detailed allocation model, how companies can measure the cost of AI infrastructure by GPU hour, Token, project, tenant, or model explains how ownership and usage dimensions connect.

How should cost visibility work?

Every tenant or project should be able to see its own consumption inside the authorized scope, and the source operations model supports project and tenant cost reports. Organizations don't have to bill teams internally: some enterprises may use the figures only for showback, while others use them for chargeback.

The main security requirement is that one tenant's detailed cost and workload data should not be exposed to another tenant without authorization. Aggregated management reporting can exist at the platform level for administrators.

How should audit work across tenants?

The source governance model requires traceability by person, time, target object, and action. For shared GPU operations, the audited actions include resource requests, quota changes, priority changes, workload submission, preemption, manual resource release, model-service publication, API-key management, permission changes, and other sensitive operations.

The audit record should retain the acting user or service identity. This is especially important for tenant administrators, who may have substantial control inside the tenant while still being unable to modify global resource policy. The audit history shows whether that boundary was respected.

For the scheduling-policy side of shared capacity, what GPU quotas, reservations, priorities, and preemption mean and when each should be used explains how entitlement and urgency should remain separate controls.

How should vendor and support access be handled?

The source Q&A recommends restricted vendor accounts with limited work-order scope and time-based permissions, and the pattern works for shared infrastructure too. A hardware vendor may need access to one failed GPU node, but that should not automatically extend to other tenant workloads, model services, knowledge bases, project usage, or global administration.

Temporary, scoped access reduces unnecessary exposure, and every login and operation should remain auditable.

How should one tenant's failure avoid affecting others?

Use resource boundaries, health isolation, topology awareness, and controlled scheduling. If one card becomes degraded, isolate it before new allocation. If one job creates abnormal contention on a shared card, the platform should detect the interference and apply the supported resource policy. If a node fails, identify the affected workloads and reschedule only where the approved recovery policy allows it.

The source design also tracks relationships from GPU to container to application, which helps contain the incident to the affected tenant or service.

What should a tenant-facing view show?

A tenant or project view can show the authorized resource quota, current allocation, available quota, running and queued tasks with queue reasons, model services, Token consumption, card hours, cost, and current alarms affecting owned resources. It should not show infrastructure or tenant data outside the authorized boundary.

A platform example that combines tenant quotas, project isolation, resource scheduling, Token attribution, and least-privilege access is Sensaka.

If I were designing secure GPU sharing, I would make one principle central: the physical accelerator pool can be shared, while ownership, permissions, workload identity, service access, usage data, and audit stay explicit at every layer. With that in place, a shared cluster works as a multi-tenant platform instead of a common machine room.