Proton Outage: What a Cooling Failure Exposed

Proton's August 27, 2026 outage is a good reminder that redundancy can fail in ways the failover design does not model cleanly. A total cooling failure in its Frankfurt data center pushed room temperature from about 21.8°C to 51.9°C in less than half an hour, killed equipment one component at a time, and forced engineers to choose between service recovery and saving hardware.

Proton published a detailed incident report the next day, so for once we do not need to guess from user reports. The timeline gives specific temperatures, failure modes, database decisions, and root cause information, which teaches a lot more than a generic lesson about "having redundancy."

What actually caused the Proton outage?

Proton says the root cause was a total failure of the cooling system in its Frankfurt data center. The later investigation traced that failure to an air-filter replacement performed on both redundant air compressors that powered the cooling system.

According to the incident report, the data center operator performed the maintenance during the night without prior notice to Proton. Proton also says the operator failed to communicate the cooling failure when it occurred, which cut into the response time available to the on-call team.

The first lesson is that two redundant components can still share a maintenance process, an operator, a procedure, or a human decision. If both are touched at the same time, the design may be electrically redundant while the operation is not.

In our Data Center and AIOps section, this is the kind of dependency that matters more than a diagram with two boxes labelled A and B.

How fast did the room become dangerous?

The main room went from approximately 21.8°C to 51.9°C in less than half an hour, and some probes reported air temperatures of 60°C. Proton says servers and network equipment began failing one by one as the temperature rose.

The speed surprised even an organisation that had planned for data center failure. Proton says higher server power density has changed the thermal timeline: what once might have taken three or four hours to become critical went critical in about 20 minutes in this incident.

That is a big operational change. High-density CPUs and GPUs put more heat into the same room and rack volume, so when active cooling disappears, the thermal buffer can be much shorter than older runbooks assume. A failover plan that expects an hour of stable hardware after cooling loss can be structurally wrong if the actual window is measured in minutes.

Why didn't redundancy prevent the outage?

Proton had redundant systems and enough capacity to survive a complete data center failure, but the actual failure was partial, progressive, and messy. Random components were dying as the room heated, which did not match the clean "site available or site unavailable" state that automated failover logic handles best.

The user-facing incident escalated when both the primary and backup network switch on a critical rack failed. That rack contained several primary database copies.

Most Proton systems can fail over automatically, but primary database failovers require human supervision. Proton says this is meant to reduce the risk of split-brain conditions, where copies diverge after connectivity problems and become difficult to reconcile safely.

That design choice is reasonable, and it also means recovery speed depends on engineers having enough time and enough trustworthy information to select the right replica.

Why was the Frankfurt versus Zurich decision difficult?

The team had to decide whether to fail databases over to surviving replicas in Frankfurt or to replicas in Zurich. Frankfurt would normally give lower latency and less disruption, but a replica inside the same overheating building could fail moments later.

At 00:45 CEST, cooling had been restored and temperatures were dropping. Proton decided to use Frankfurt replicas where they were still alive and Zurich where no Frankfurt replica remained, partly to avoid larger traffic-flow changes during an already unstable incident.

That did not restore everything immediately. Many network cards had reached 105°C, compared with a stated normal operating temperature of 45°C. The cards entered thermal-protection mode and required a cold system reset.

Proton's security restrictions around out-of-band management also meant additional staff had to be involved in recovery. That is another tradeoff worth noting: tighter administrative access can reduce security risk while making emergency recovery slower if the access path is not designed for a site incident.

Did Proton lose data?

Proton says no emails were lost. Email delivery in both directions was delayed, and some noncritical systems such as push notifications and payment processing took longer to recover.

Most services were back online for most users by around 01:30 CEST, with some remaining systems recovering around 02:00 CEST. Work continued after the visible outage, because database primaries were split between Zurich and Frankfurt and some systems were running with reduced redundancy or performance.

Incident reports should keep those two states apart. A platform can be serving users again while its infrastructure is still abnormal and the engineering team is one failure away from another incident.

Backup works the same way. A successful backup job proves that data was written, and only a tested recovery path proves you can get it back. Our Proxmox Backup Server guide focuses on that separation because recovery state is what matters under pressure.

What does this outage teach about cooling redundancy?

Cooling redundancy has to cover maintenance and operating procedure as well as duplicate equipment. If two compressors are redundant but both receive the same maintenance action at the same time, the maintenance process creates a shared failure domain.

A useful review should ask who can take both cooling paths out of service, which alarms reach the customer directly, whether thermal telemetry is visible independently of the facility operator, and what automatic workload actions start when temperature crosses defined thresholds.

You should also know the thermal survival time of the current hardware density. Do not inherit a number from an older facility design; measure or model what happens with today's rack power.

Finally, define what "complete site failure" looks like to automation. Proton's site was failing gradually, and progressive degradation can be harder than a clean power cut because each decision has to account for components that are alive now but may not stay alive.

What would I test after reading Proton's postmortem?

I would test a cooling-loss scenario that assumes only 20 minutes before equipment becomes unreliable. That forces the team to decide what gets shut down, what fails over first, which databases require human approval, and whether the remote site really has enough capacity.

I would also review shared maintenance dependencies. Ask whether redundant power, cooling, network, or storage paths can be worked on simultaneously by one vendor or one change ticket. If they can, the redundancy claim needs an operational qualifier.

Then test out-of-band access during the same exercise. Security controls should stay strong, but emergency access cannot depend on waking one specific person who happens to know the path.

Relying on automatic full-site failover alone is acceptable only when the application can genuinely treat a site as a single binary failure domain. Proton's incident shows why high-density infrastructure often fails in a less convenient shape.