What is MaaS, and how do model repositories, inference instances, API gateways, and Token metering work together?

MaaS, or Model as a Service, turns an AI model into an operable service that applications can call through a controlled interface. The operating chain is straightforward: the model repository manages the model asset, inference instances run the model, the API gateway exposes and controls access, and Token metering records how the service is consumed.

The important shift is from "we have a model file" to "we operate a production service." Once a model goes live, you need version control, deployment rules, service health, routing, quotas, access keys, usage records, rollback, and cost attribution. MaaS is the layer that connects those activities.

What is MaaS?

Model as a Service is a way to provide AI models as managed, callable services rather than asking every user or application team to deploy and operate the model independently.

A MaaS platform usually sits between the underlying compute infrastructure and the applications that consume AI capabilities.

Below it are GPU or NPU resources, operating systems, containers, schedulers, storage, and networks.

Inside it are model assets, deployment templates, inference instances, service routing, usage controls, and metering.

Above it are applications, agents, business systems, developers, and users.

A useful MaaS layer therefore needs to answer several questions at once.

Which model version is approved for production?

Where is it running?

How much compute is assigned to it?

Which endpoint should applications call?

Who is allowed to call it?

How much traffic is it receiving?

How many input and output Tokens are being processed?

What is the success rate?

What does the service cost?

If those questions are split across unrelated systems, operating the model becomes harder than simply deploying it.

What does a model repository do?

A model repository manages the model as a controlled asset.

At minimum, it should record the model name, version, source, status, owner, and the information required to deploy it.

A stronger repository can also store evaluation results and use them as a go-live criterion.

That creates an important separation between "a model exists" and "a model is approved for service."

The source operating model also treats models and knowledge bases as separate objects. That is useful because one model can serve several knowledge bases, and one knowledge base may be tested against several models.

Keeping them separate avoids hard-coding the retrieval layer into the model asset itself.

The repository should also preserve version history.

If model version 2.3 is deployed and later causes quality or latency problems, the operations team needs to know which earlier version can be restored.

This is where repository management connects directly to deployment and rollback.

What is an inference instance?

An inference instance is a running deployment of a model that accepts requests and produces outputs.

The instance consumes compute resources.

It may run on one accelerator, several accelerators, a partitioned accelerator, or CPU resources depending on the model and service design.

The important point is that the model asset and the running instance are different objects.

One model version can have several inference instances.

Those instances can serve different environments, tenants, regions, performance tiers, or traffic segments.

Each instance should have operational state such as:

Running or stopped
Assigned resource specification
Model version
Replica count
Endpoint state
Health status
Current load
Success rate
Latency
Token throughput

That distinction is necessary for safe operations.

If one instance becomes unhealthy, you may route traffic away from it without removing the model from the repository.

If you deploy a new version, you can create new instances before sending production traffic to them.

How do deployment templates fit into MaaS?

Deployment templates make model delivery repeatable.

Instead of configuring every inference service by hand, the platform can define a reusable template that specifies the required compute, runtime, container image, model version, replica count, network exposure, scaling policy, and other operational settings.

Templates reduce configuration drift.

They also create a clear review point.

An organization can approve a production template for a particular model family and restrict changes to controlled fields.

This becomes especially useful when different models need different resource shapes.

A small embedding model may run efficiently on a shared accelerator.

A large language model may require several high-memory accelerators.

A template lets the user request the service without manually choosing individual devices.

The underlying scheduler still has to find compatible hardware.

For the scheduling side of that process, GPU resource pooling and scheduling in Kubernetes explains how resource classes, quotas, topology, and health can affect placement.

How do canary release and rollback work for model services?

Canary release sends a limited portion of traffic to a new model or deployment before full rollout.

The purpose is to reduce the impact of a bad release.

For example, a new model version can start with a small traffic share while the platform watches success rate, latency, Token throughput, and application feedback.

If the new version behaves normally, the traffic share can increase.

If it behaves poorly, traffic can return to the previous version.

Rollback is therefore not only a model-repository action.

It involves the service gateway and the running inference instances.

The repository knows the approved versions.

The deployment layer starts the required instances.

The gateway controls which version receives traffic.

The monitoring layer measures the effect.

The workflow records the change.

That closed loop is what makes model delivery an operations capability.

Why does MaaS need an API gateway?

The API gateway gives applications one managed entry point to model services.

Without a gateway, every application may need to know the address and behavior of each individual model instance.

That creates tight coupling.

If a model instance moves, scales, or is replaced, application configuration may have to change.

A gateway hides that internal topology.

It can expose a stable service endpoint and route requests to the appropriate model or instance behind it.

The gateway can also enforce operational controls such as:

API key validation
Tenant identification
Rate limits
Quotas
Routing rules
Model selection
Fallback policies
Graceful degradation
Request logging
Invocation audit

This is where model switching can become an operations action instead of an application-development change.

If Model A becomes unavailable, the gateway may route an approved class of requests to Model B without requiring every calling application to be redeployed.

How does multi-model routing work?

Multi-model routing uses request attributes and policy to choose the model or instance that should handle a call.

A simple route can be based on endpoint.

A more advanced route can consider tenant, requested model, project policy, traffic class, model health, cost, or fallback rules.

The important requirement is explainability.

The platform should be able to show why a request went to a particular model.

That matters for troubleshooting and billing.

If two model versions have different cost or quality characteristics, silent routing changes can create confusing usage reports.

Routing policy therefore belongs in the same operational chain as metering and audit.

What does graceful degradation mean for MaaS?

Graceful degradation means keeping a useful level of service when the preferred model or capacity is unavailable.

The exact behavior depends on the application.

A request may be routed to a smaller approved model.

A lower service tier may accept slower latency.

A non-critical feature may be disabled.

A queue may temporarily absorb traffic.

The gateway can support those policies, but the business rule has to come from the service owner.

Not every model is interchangeable.

A fallback model may have different output quality, context limits, safety behavior, or cost.

The platform should therefore treat fallback as an explicit policy, not as a random routing decision.

What is Token metering?

Token metering records the amount of model input and output processed through the service.

The source operating model treats a Token as the metering unit for model input and output.

A useful usage record should identify more than the Token count.

It should connect the consumption to:

Time
Project
Tenant
API key or caller identity
Model
Model version
Inference service
Input Tokens
Output Tokens
Request result
Latency
Billing or cost rule

That turns raw usage into operational information.

You can answer which project consumed the most Tokens, which model created the highest service cost, and whether a usage spike came from more requests or larger prompts.

The same records can feed quota and cost reporting.

How does Token metering connect to billing?

Token metering can support billing or internal chargeback when the metering definition matches the commercial or accounting definition.

That last condition matters.

If the gateway counts Tokens one way while the billing system uses another definition, users will dispute the numbers.

Define whether billing counts input Tokens, output Tokens, or both.

Define how failed calls are treated.

Define how cached results are handled if the service supports them.

Define how retries are counted.

Define the time zone and billing period.

Then use the same definition across usage dashboards, invoices, project reports, and cost analysis.

For the broader cost model, how companies can measure AI infrastructure cost by GPU hour, Token, project, tenant, or model explains how service consumption and infrastructure consumption can be combined.

How do inference instances and Token metering connect?

The relationship between inference instance and Token usage allows the platform to compare service output with the compute resources that produced it.

Suppose two instances serve the same model.

Instance A uses four accelerators and produces 20 million Tokens during an hour.

Instance B uses four similar accelerators and produces 12 million Tokens during the same hour.

That does not prove Instance B is inefficient, because request mix and output length may differ.

But it gives the operations team a starting point.

Add latency, success rate, request size, power, accelerator utilization, and model version.

Now the platform can investigate why two similar deployments produce different results.

That is the value of connecting service metering with infrastructure telemetry.

How should MaaS handle scaling?

MaaS should scale inference capacity based on measurable service demand and resource availability.

Possible signals include request concurrency, queue depth, latency, Token throughput, accelerator utilization, or other workload-specific metrics.

Kubernetes Horizontal Pod Autoscaling can adjust workload replicas based on resource or custom metrics when the application architecture supports it.

Model serving adds another constraint: a new replica may take time to load model weights and become ready.

That means scaling policy should account for startup time.

If demand rises faster than instances can start, the service may need reserved headroom.

If demand falls, the platform should avoid keeping expensive accelerators allocated without need.

Scaling therefore sits between service SLOs and cost control.

What should a MaaS operations dashboard show?

A useful MaaS dashboard should show service supply, demand, quality, and cost together.

For supply, show models, approved versions, inference instances, assigned accelerators, and available capacity.

For demand, show calls, input Tokens, output Tokens, concurrency, and traffic by tenant or project.

For quality, show success rate, latency, time to first Token where relevant, and SLO status.

For cost, show accelerator hours, energy where available, and unit Token cost.

Every top-level number should support drill-down.

If Token success rate drops, the operator should be able to find the affected model and instance.

If cost rises, the operator should be able to identify the project, tenant, model, and compute behind the increase.

A platform example built around this connected service chain is Sensaka.

If I were evaluating MaaS architecture, I would ask one question: can the platform trace a production API call all the way from the caller and gateway to the model version, inference instance, underlying compute, Token record, and audit trail? If the answer is yes, MaaS is functioning as an operating layer. If the answer is no, it is probably still a collection of model-serving components.