How can companies measure the cost of AI infrastructure by GPU hour, Token, project, tenant, or model?

Companies can measure AI infrastructure cost by combining resource metering and service metering under the same ownership model. Accelerator hours measure how long expensive compute is allocated, Token metering measures model-service consumption, and project, tenant, and model identifiers explain who used the resources and what produced the cost.

The difficult part is allocation, not arithmetic. If the scheduler, API gateway, energy system, and billing system use different project names or time windows, the same workload can produce several conflicting cost numbers. Cost accounting therefore needs one shared definition of resource, owner, usage period, and unit.

What is a GPU hour?

A GPU hour is one accelerator card assigned or consumed for one hour according to the metering policy.

A simple calculation is:

GPU hours = Number of GPUs × Allocation time in hours

If a training job uses 8 GPUs for 3 hours:

8 × 3 = 24 GPU hours

This metric measures resource time.

It does not automatically measure useful work.

A job can consume 24 GPU hours while the average utilization is only 40 percent.

That distinction is important because allocated capacity still has economic value even when the workload is inefficient.

The cost system should therefore keep GPU hours and utilization as separate metrics.

GPU hours answer, "How much accelerator capacity did this workload occupy?"

Utilization answers, "How busy was the accelerator while it was occupied?"

Combining both makes idle cost visible.

Should GPU hours use allocated time or active time?

Use the definition that matches your business question, and label it clearly.

Allocated GPU hours count the full period in which a card is reserved for the workload.

Active GPU hours count only periods that meet a defined activity rule.

For internal capacity planning, allocated hours are often more useful because the scheduler could not give the same card to another tenant during that period.

For performance analysis, active time can reveal how much of the reservation actually executed useful work.

For billing, the commercial model decides.

If a tenant reserves a dedicated GPU for ten hours, charging only for moments above a utilization threshold may not reflect the capacity held for that tenant.

A useful reporting model can show both:

Allocated GPU hours
Estimated active GPU hours
Idle rate
Average utilization

That separates capacity consumption from efficiency.

How do you calculate cost per GPU hour?

Cost per GPU hour is the cost allocated to accelerator capacity divided by the GPU hours in the same period.

A simple formula is:

Cost per GPU hour = Allocated accelerator-related cost / GPU hours

The numerator can be narrow or broad.

A narrow model may include only hardware depreciation and electricity.

A broader model may include:

Accelerator depreciation
Server chassis cost
Power
Cooling overhead
Rack or colocation cost
Network
Storage
Software
Operations labor
Maintenance

There is no universal correct numerator.

The requirement is consistency.

If you call the result "fully loaded cost per GPU hour," define every cost category included.

If you call it "energy cost per GPU hour," include energy cost only.

Do not compare two unit costs that use different boundaries.

How does Token metering fit into infrastructure cost?

Token metering connects model-service output to the infrastructure that produced it.

A model-serving platform can record input and output Tokens by project, tenant, API key, model, and time period.

The infrastructure layer can record accelerator hours, energy, and other resource cost during that same period.

Then the system can calculate:

Cost per 1,000 Tokens
Cost per 1 million Tokens
Energy cost per Token
Infrastructure cost per Token
Cost by model
Cost by project
Cost by tenant

This turns AI cost from an equipment number into a service-unit number.

For the service architecture behind those records, what MaaS is and how model repositories, inference instances, API gateways, and Token metering work together explains the full chain.

How do you calculate cost per Token?

Cost per Token is the cost attributed to a model service divided by the Token volume attributed to the same service and time period.

For example:

A model service is allocated $600 of infrastructure cost during one day.

It processes 300 million billable Tokens.

$600 / 300,000,000 = $0.000002 per Token

That can also be reported as:

$2 per 1 million Tokens

This example is illustrative. Your result depends entirely on the cost boundary and Token definition.

A production system should define:

Input versus output Token treatment
Failed requests
Retries
Cached results
Shared-instance allocation
Time zone
Billing period
Model version
Project and tenant ownership

The number is only useful when the definition is stable.

Should input and output Tokens be costed separately?

They can be, especially when output generation has materially different compute behavior from input processing.

But separate pricing is a business decision, not a requirement for infrastructure operations.

At minimum, record input and output Tokens separately where the serving system exposes them.

You can always combine them later.

You cannot separate them later if you never recorded the distinction.

Keeping both also helps explain cost changes.

Suppose request count stays flat but average output length doubles.

Total Token volume rises even though the number of calls is unchanged.

That can increase inference compute and cost.

A request-only dashboard would hide the cause.

How do you allocate cost by project?

Allocate cost by project by linking every resource reservation and service call to a stable project identifier.

For training, the scheduler or job submission process should require the project identity.

For inference, the API key, service route, tenant, or application identity should map to a project.

For dedicated resources, allocation is straightforward.

For shared resources, use a documented allocation rule.

Examples include:

GPU reservation time
Actual execution time
Token volume
Request volume
Reserved capacity share
Measured utilization share

Choose the rule that best reflects how the shared resource is consumed.

Then keep it consistent across periods.

The platform should also preserve the raw usage records so finance or operations can explain the result.

How do you allocate cost by tenant?

Tenant allocation works the same way, but the tenant becomes the primary ownership dimension.

A tenant may contain several projects.

That means the data model should support hierarchy.

For example:

Tenant A
Project A1
Project A2
Model Service A
Training Job A3

Costs can then be reported at each level without duplicating usage.

If Project A1 consumes 100 GPU hours and Project A2 consumes 50, the tenant total is 150.

The same model works for Token usage.

This is why one shared ownership model is essential.

Do not let the scheduler use department names while the gateway uses account IDs and finance uses another project code with no mapping between them.

How do you allocate cost by model?

Model-level cost requires a relationship between the model version, its inference instances, and the compute they consume.

For dedicated model instances, this is straightforward.

If Model X occupies four GPUs for ten hours, those 40 GPU hours can be attributed directly.

Shared serving is harder.

If several models share one accelerator, the platform needs an allocation rule based on execution time, Token volume, reserved share, or another measurable driver.

Model-level cost should also include service quality context.

A cheaper model is not automatically better if it fails the required SLO or produces unacceptable output quality.

Cost is one operating dimension.

Service quality is another.

The useful comparison is cost at the required level of quality.

How should energy be included?

Energy should be included using measured accelerator or server power where available, with a clear rule for facility overhead.

At accelerator level, integrate power over time to calculate kWh.

At server level, use server metering where available.

If you want to estimate facility-attributed energy, you can apply a documented facility overhead factor such as a relevant PUE measurement, but label it as an allocation method.

Then apply the electricity tariff for the matching time period.

This matters when electricity pricing changes by time of day.

A flexible training job that runs off peak may have a lower energy cost even if it uses the same number of GPU hours.

For the formulas, how a data center can calculate PUE, WUE, GPU energy consumption, and energy cost per Token provides the detailed method.

How do you measure idle cost?

Idle cost is the cost of allocated or owned capacity that is not producing the intended amount of useful work under the organization's definition.

Start by defining idle.

A completely unallocated card is one type.

A card allocated to a job at 5 percent utilization is another.

A card waiting on storage is another.

A fragmented pool with free cards that cannot satisfy any queued job is another.

These causes should not be grouped into one number and blamed on users.

Classify idle reason where possible.

Examples:

No demand
Quota restriction
Fragmentation
Network bottleneck
Storage bottleneck
Data loading
Degraded hardware
Oversized resource request

Then calculate the accelerator hours and energy associated with each cause.

That turns an idle-rate KPI into an optimization backlog.

How should shared infrastructure cost be allocated?

Shared cost should use a stable allocation driver that is understandable and hard to manipulate.

Network and storage are common examples.

Possible allocation drivers include bandwidth consumed, storage capacity reserved, I/O volume, GPU hours, project share, or a fixed base charge plus variable usage.

There is no single best method.

The question is whether the allocation is decision-useful.

If the storage cost is tiny relative to GPU cost, an overly complex IOPS-based allocation may create more administrative effort than value.

If storage is a major cost and certain projects consume disproportionate bandwidth, a more detailed rule may be justified.

Start simple.

Increase precision only when the result changes a real decision.

How should cost data connect to quotas?

Cost and quota definitions should share the same resource model.

If a tenant has a quota for eight high-memory GPUs, the metering system should identify those same resources consistently.

If the quota system treats a partitioned GPU as one resource while billing treats it as a fraction, document the conversion.

Misaligned definitions create disputes.

The source operating model calls this out directly: accelerator hours, energy consumption, and Token metering need aligned allocation definitions so different modules do not produce inconsistent figures.

This is a data-governance issue as much as a billing issue.

What should an AI cost dashboard show?

A useful cost dashboard should show cost, usage, efficiency, and ownership together.

At minimum:

Total accelerator hours
Available versus allocated capacity
Idle rate
Cost per GPU hour
Token volume
Cost per Token
Energy consumption
Project ranking
Tenant ranking
Model ranking
Resource type or accelerator model
Trend over time

Every summary number should support drill-down to the usage records that produced it.

A platform example using one shared metering model across projects, tenants, models, and accelerator resources is Sensaka.

If I were building AI cost accounting, I would start with one rule: every GPU hour and every Token must carry an owner and a time interval. Once that is true, project, tenant, and model reporting becomes a grouping problem. Without it, even a beautiful cost dashboard is mostly an estimate.