Proton Outage: What a Cooling Failure Exposed

Proton's August 27, 2026 outage is a useful reminder that redundancy can fail in ways the failover design does not model cleanly. A total cooling failure in its Frankfurt data center pushed room temperature from about 21.8°C to 51.9°C in less than half an hour, killed equipment one component at a time, and forced engineers to choose between service recovery and saving hardware.

Proton published a detailed incident report the next day, so this is one of those outages where we do not need to guess from user reports. The timeline gives specific temperatures, failure modes, database decisions, and root cause information. Those details make the incident more useful than a generic lesson about "having redundancy."

What actually caused the Proton outage?

Proton says the root cause was a total failure of the cooling system in its Frankfurt data center. The later investigation traced that failure to an air-filter replacement performed on both redundant air compressors that powered the cooling system.

The maintenance was performed by the data center operator during the night without prior notice to Proton, according to the incident report. Proton also says the operator failed to communicate the cooling failure when it occurred, reducing the response time available to the on-call team.

This is the first important lesson. Two redundant components can still share a maintenance process, an operator, a procedure, or a human decision. If both are touched at the same time, the design may be electrically redundant while the operation is not.

In our Data Center and AIOps section, this is the kind of dependency that matters more than a diagram with two boxes labelled A and B.

How fast did the room become dangerous?

The main room went from approximately 21.8°C to 51.9°C in less than half an hour, and some probes reported air temperatures of 60°C. Proton says servers and network equipment began failing one by one as the temperature rose.

The speed surprised even an organisation that had planned for data center failure. Proton says that higher server power density has changed the thermal timeline, and what once might have taken three or four hours to become critical went critical in about 20 minutes in this incident.

That is a major operational change. High-density CPUs and GPUs put more heat into the same room and rack volume. When active cooling disappears, the thermal buffer can be much shorter than older runbooks assume.

A failover plan that expects an hour of stable hardware after cooling loss can be structurally wrong if the actual window is measured in minutes.

Why didn't redundancy prevent the outage?

Proton had redundant systems and enough capacity to survive a complete data center failure, but the actual failure was partial, progressive, and messy. Random components were dying as the room heated, which did not match the clean "site available or site unavailable" state that automated failover logic handles best.

The user-facing incident escalated when both the primary and backup network switch on a critical rack failed. That rack contained several primary database copies.

Most Proton systems can fail over automatically, but primary database failovers require human supervision. Proton says this is intended to reduce the risk of split-brain conditions, where copies diverge after connectivity problems and become difficult to reconcile safely.

That design choice is reasonable. It also means recovery speed depends on engineers having enough time and enough trustworthy information to select the right replica.

Why was the Frankfurt versus Zurich decision difficult?

The team had to decide whether to fail databases over to surviving replicas in Frankfurt or to replicas in Zurich. Frankfurt would normally provide lower latency and less disruption, but a replica inside the same overheating building could fail moments later.

At 00:45 CEST, cooling had been restored and temperatures were dropping. Proton decided to use Frankfurt replicas where they were still alive and Zurich where no Frankfurt replica remained, partly to avoid making larger traffic-flow changes during an already unstable incident.

That did not restore everything immediately. Many network cards had reached 105°C, compared with a stated normal operating temperature of 45°C. The cards entered thermal-protection mode and required a cold system reset.

Proton's security restrictions around out-of-band management also meant additional staff had to be involved in recovery. That is another useful tradeoff: tighter administrative access can reduce security risk while making emergency recovery slower if the access path is not designed for a site incident.

Did Proton lose data?

Proton says no emails were lost. Email delivery in both directions was delayed, and some noncritical systems such as push notifications and payment processing took longer to recover.

Most services were back online for most users by around 01:30 CEST, with some remaining systems recovering around 02:00 CEST. The work continued after the visible outage because database primaries were split between Zurich and Frankfurt and some systems were operating with reduced redundancy or performance.

This distinction matters in incident reporting. "Service restored" does not mean "infrastructure normal." A platform can be serving users while the engineering team is still one failure away from another incident.

The same logic applies to backup. A successful backup job is not the same as a proven recovery path. Our Proxmox Backup Server guide focuses on that separation because recovery state is what matters under pressure.

What does this outage teach about cooling redundancy?

Cooling redundancy has to include maintenance and operating procedure, not just duplicate equipment. If two compressors are redundant but both receive the same maintenance action at the same time, the maintenance process creates a shared failure domain.

A useful review should therefore ask who can take both cooling paths out of service, which alarms reach the customer directly, whether thermal telemetry is visible independently of the facility operator, and what automatic workload actions start when temperature crosses defined thresholds.

You should also know the thermal survival time of the current hardware density. Do not inherit a number from an older facility design. Measure or model what happens with today's rack power.

Finally, define what "complete site failure" looks like to automation. Proton's problem was that the site was failing gradually. Progressive degradation can be harder than a clean power cut because each decision has to account for components that are alive now but may not stay alive.

What would I test after reading Proton's postmortem?

I would test a cooling-loss scenario that assumes only 20 minutes before equipment becomes unreliable. That forces the team to decide what gets shut down, what fails over first, which databases require human approval, and whether the remote site really has enough capacity.

I would also review shared maintenance dependencies. Ask whether redundant power, cooling, network, or storage paths can be worked on simultaneously by one vendor or one change ticket. If the answer is yes, the redundancy claim needs an operational qualifier.

Then test out-of-band access during the same exercise. Security controls should stay strong, but emergency access cannot depend on waking one specific person who happens to know the path.

The opposite approach, relying on automatic full-site failover alone, is acceptable only when the application can genuinely treat a site as a single binary failure domain. Proton's incident shows why high-density infrastructure often fails in a less convenient shape.