How do SRE metrics such as MTTD, MTTR, SLO, error budgets, and burn rate apply to AI and infrastructure operations?

SRE metrics turn AI and infrastructure reliability into something teams can measure and govern. MTTD tells you how quickly problems are detected, MTTR tells you how quickly service is restored, SLOs define the reliability target, error budgets quantify how much unreliability is acceptable, and burn rate shows how fast that budget is being consumed.

For AI infrastructure, these metrics need to cover more than application uptime. They can apply to training success, inference success, Token throughput, time to first Token, accelerator availability, failure recovery, and other critical paths that affect the ability to turn compute into a stable service.

What is MTTD?

Mean Time to Detect, or MTTD, measures how long it takes the operations system or team to detect an incident after it begins.

A simple formula is:

MTTD = Sum of detection delays / Number of incidents

The hard part is defining incident start.

For a hardware event, the start may be the first ECC error.

For a service incident, it may be the first request that violates the service indicator.

For a cooling issue, it may be the first threshold breach.

Pick one rule for each incident class and use it consistently.

A low MTTD means problems become visible quickly.

A high MTTD means failures can affect workloads for too long before anyone knows.

Automatic detection can reduce MTTD, but only if alarms are meaningful.

An alarm storm that nobody trusts may technically detect the problem in seconds while the human response remains slow.

That is why MTTD should be read alongside alarm quality and incident consolidation.

What is MTTR?

Mean Time to Repair, Recover, Resolve, or Restore is commonly abbreviated MTTR, but organizations use the second word differently.

Define yours.

For infrastructure operations, the most useful version is often time to restore service.

A simple formula is:

MTTR = Sum of restoration durations / Number of incidents

Again, define the start and end timestamps.

Does the clock start at failure or at detection?

Does it stop when the device is repaired or when the user-facing service is restored?

Those are different metrics.

For an AI training incident, the service may be restored when the job resumes from checkpoint on healthy resources, even if the failed GPU is repaired later.

That is often a better operations metric because it measures how long productive work was interrupted.

Hardware repair time can be tracked separately.

How do MTTD and MTTR apply to GPU failures?

For a GPU failure, MTTD measures how quickly the platform identifies that the card or node has become unhealthy.

MTTR measures how quickly the affected workload returns to a healthy state under the defined recovery process.

A good fault-tolerance loop can reduce MTTR by:

Detecting the bad card
Stopping new scheduling to it
Finding the affected job
Selecting a valid checkpoint
Allocating healthy replacement resources
Restarting the job
Validating progress

This is why hardware health and scheduler integration matter.

If the team needs 20 minutes to identify which job uses the card, that relationship lookup becomes part of MTTR.

If a checkpoint is six hours old, the service may restore quickly but the business still loses six hours of compute.

That lost work should be tracked separately.

The article on how AI infrastructure can automatically recover training jobs after a GPU or server failure covers this recovery loop in detail.

What is an SLO?

A Service Level Objective, or SLO, is a target value for a service level indicator over a defined period.

Google's SRE guidance treats SLOs as targets for service reliability based on measurable service level indicators.

For an inference service, possible indicators include:

Request success rate
Token success rate
Latency
Time to first Token
Token throughput
Timeout rate

For a training service, possible indicators include:

Job start success
Task completion success
Queue time
Checkpoint success
Recovery time
Resource availability

For infrastructure, possible indicators include:

Accelerator availability
Network availability
Storage latency
Provisioning success
Hardware repair response

Do not create SLOs for every metric.

Choose the indicators that represent user-visible or business-important reliability.

How should an AI service SLO be defined?

Define the service, the indicator, the target, the measurement window, and the population of events.

For example:

Service: production inference endpoint
Indicator: successful requests / eligible requests
Target: 99.9 percent
Window: rolling 30 days
Exclusions: explicitly defined maintenance or non-billable test traffic

The exact number is a business decision.

The method is the important part.

Two teams can both claim a 99.9 percent SLO while counting different requests.

That makes comparison meaningless.

Keep the definition exportable and versioned.

The source operating model also supports service tiers such as Gold, Silver, and Bronze so different models can have different thresholds.

That can be useful when one service requires strict latency while another is best effort.

What is an error budget?

An error budget is the amount of unreliability allowed by the SLO during the measurement window.

If the SLO is 99.9 percent success, the error budget is the remaining 0.1 percent under that definition.

For a request-based SLO:

Error budget = Total eligible events × Allowed failure fraction

If there are 1,000,000 eligible requests and the SLO allows 0.1 percent failure:

1,000,000 × 0.001 = 1,000 allowed failed requests

That does not mean failure is desirable.

It gives the organization a quantitative boundary for balancing reliability and change velocity.

If the budget is healthy, the team has room for normal delivery risk.

If the budget is exhausted, reliability work should take priority.

The source SRE model uses exactly this principle: when the budget crosses the threshold, release activity can be frozen until reliability debt is repaid.

What does burn rate mean?

Burn rate measures how quickly the service is consuming its error budget compared with the rate that would consume the budget evenly across the full window.

A burn rate of 1 means the service is consuming the budget at the expected average pace that would exhaust it exactly at the end of the period.

A burn rate greater than 1 means the budget is being consumed faster.

A very high burn rate means the service can exhaust its allowance quickly if the condition continues.

Burn rate is useful because raw error counts can be misleading.

Ten failures in one minute may be severe for a low-volume service.

The same ten failures may be negligible for a very high-volume service.

Burn rate relates the failures to the SLO and remaining budget.

Why use multiple burn-rate windows?

Multiple windows help detect both fast incidents and slow reliability degradation.

Google's SRE Workbook describes multiwindow, multi-burn-rate alerting as a way to detect meaningful SLO consumption while controlling alert noise.

A short window is sensitive to rapid failure.

A longer window confirms that the condition is persistent enough to matter.

You can also define separate fast-burn and slow-burn alert paths.

Fast burn means the budget is disappearing quickly and requires urgent response.

Slow burn means the service is degrading over a longer period and may need planned reliability work.

The source operating model uses this same concept with multi-window, multi-burn-rate alerts.

The objective is not to generate more alerts.

It is to generate alerts when the reliability objective is genuinely at risk.

How should SLOs apply to Token services?

Token services can use indicators that reflect both availability and quality of delivery.

The source material lists measures such as Token success rate, throughput, time to first Token, per-Token latency, and timeout rate.

A model service can therefore have several SLOs.

For example:

Request success rate
Token generation success
Time to first Token
Streaming continuity
Overall latency
Timeout rate

Do not put all indicators into one composite score if they represent different user experiences.

A service can have excellent success rate and terrible latency.

The user still sees a bad service.

Separate SLOs make the failure mode visible.

How should SRE apply to training services?

Training SRE should focus on the reliability of the job lifecycle.

Useful indicators include:

Queue admission time
Job start success
Job completion success
Checkpoint success
Recovery success
Mean lost compute after failure
Resource availability
Repeated failure rate

A training job that fails after 18 hours and restarts from a 17-hour-old checkpoint is technically recovered but operationally expensive.

That lost compute should be visible.

Likewise, a scheduler can be highly available while users wait hours for the requested resource class.

Service reliability is the experience of obtaining useful compute, not simply the uptime of the scheduler process.

What is an automatic-remediation ratio?

Automatic-remediation ratio measures the share of eligible incidents resolved through approved automated actions without manual execution.

A simple definition is:

Automatic-remediation ratio = Automatically resolved eligible incidents / Total eligible incidents

Define "eligible."

High-risk incidents should not be counted as automation failures if policy intentionally requires human approval.

The source SRE model separates incidents by risk tier.

Known transient events can self-recover.

Controlled-risk incidents can run approved scripts with rollback.

Risk-bearing changes require workflow approval and canary execution.

That makes the automation ratio meaningful because it measures automation inside the approved boundary.

What is toil reduction?

Toil reduction measures how much repetitive manual operational work has been removed or shortened.

Possible indicators include:

Manual interventions per incident
Operator minutes per incident
Number of repeated actions automated
Work orders completed without manual command execution
Time spent on routine inspection
Repeated incident recurrence

Toil is not simply "work people dislike."

Some manual work is valuable because it requires judgment.

The goal is to automate repetitive, predictable actions so engineers can focus on diagnosis, capacity, reliability, and improvement.

Measure the time actually saved.

Otherwise "automation" can become a feature count with no operational benefit.

How should incident reviews connect to SRE metrics?

Post-incident review should turn individual failures into changes in the reliability system.

The review should preserve:

Timeline
Root cause
Contributing factors
Detection gap
Recovery gap
Amount of service impact
Amount of lost compute
SLO impact
Error-budget consumption
Improvement actions
Owner
Due date

If MTTD was poor, improve detection.

If MTTR was poor, improve runbooks, ownership, checkpoints, or automation.

If the same failure keeps consuming the error budget, fix the underlying reliability issue instead of improving only the response process.

The review should feed the knowledge base so future incidents benefit from what was learned.

How do error budgets affect release decisions?

Error budgets provide a measurable release gate.

If the service is comfortably within its SLO and has budget remaining, normal release activity can continue according to policy.

If the budget is nearly exhausted, high-risk changes can be slowed.

If the budget is exhausted, the team can freeze non-essential releases and prioritize reliability work.

This changes the discussion from opinion to evidence.

Product teams can see the reliability constraint.

Operations teams can show the consumption trend.

Management can see the forecast exhaustion date.

That is the management value of an error budget.

How does AIOps support SRE?

AIOps can reduce MTTD by detecting and correlating failures faster, and it can reduce MTTR by identifying likely causes and recommending or executing approved remediation.

But SRE provides the measurement framework that tells you whether those improvements are real.

If an AIOps system claims faster diagnosis, compare MTTD before and after.

If automated recovery is added, compare MTTR and recurrence.

If alarm correlation improves, compare operator workload and false incident volume.

If reliability deteriorates while automation increases, the automation is not successful.

The relationship between the two is covered in how AIOps reduces alarm noise, identifies root causes, and determines business impact.

What should an SRE dashboard show?

A useful SRE dashboard should show reliability performance, budget state, incident risk, and operational effort.

At minimum:

MTTD
MTTR
SLO attainment
Remaining error budget
Burn rate
Forecast budget exhaustion
Incident count
Risk tier
Automatic-remediation ratio
Toil trend
Repeated incidents
Post-incident action status

A platform example applying these measures from infrastructure through Token services is Sensaka.

If I were introducing SRE into AI infrastructure, I would start with three critical services, not 100 metrics. Define one or two SLOs for each, measure MTTD and MTTR, create a 30-day error budget, and review every incident that burns a meaningful share of it. Once the definitions are trusted, expand the system.