Why a Three-Node HA Cluster Panics When Two Nodes Go Down

The setup that looks perfect on paper

A small virtualization cluster can feel like the sweet spot for reliability: three nodes, shared storage over iSCSI or NFS, and high availability configured so virtual machines can move smoothly between hosts. Maintenance mode works beautifully. You put one node into maintenance, live migration kicks in, workloads move away, and the host reboots peacefully.

Everything behaves exactly as advertised. That's why the situation becomes so confusing when an administrator tries something seemingly harmless. Two hosts go into maintenance mode, and all the virtual machines migrate successfully to the third node. Nothing is overloaded, and the hardware is powerful enough to run every workload on a single machine.

Then both maintenance nodes reboot, and suddenly the final node starts behaving like something has gone terribly wrong. Services panic, and sometimes the host even reboots. What looked like a routine maintenance window suddenly feels like the cluster is fighting back.

For anyone watching the console scroll past error messages, the question becomes immediate and frustrating: why can't the last node simply keep running the VMs?

The cluster's paranoia kicks in

The answer lies in a rule that governs most high-availability clusters: majority consensus.

Distributed systems don't assume they understand what's happening in the network. They assume the opposite. Networks fail, links drop, switches glitch, and packets disappear. Because of that uncertainty, clusters rely on quorum, the ability for a majority of nodes to confirm that the cluster state is valid. In a three-node cluster, that majority is two nodes.

The moment two nodes disappear, the remaining host loses quorum. From the cluster's perspective, it doesn't know if the other machines are actually offline or if it's the one isolated from the rest of the system. One engineer summarized it in a blunt way: when two nodes vanish, the last one assumes it might be the problem.

That suspicion triggers protective behavior. Instead of continuing to run workloads blindly, the cluster halts operations to avoid something far worse than downtime.

The disaster everyone is trying to avoid

The reason behind this strict rule is split brain, which cluster engineers fear more than almost anything else.

Imagine the cluster allowed the final node to continue running every VM without question. Now imagine the original scenario wasn't a planned reboot but a network partition, where one node loses connectivity to the other two. If the isolated node keeps running workloads while the other nodes assume it's dead, the cluster might start those same virtual machines elsewhere, and two identical instances of the same VM exist at the same time.

Databases write conflicting data. Filesystems become corrupted. Transactions diverge in ways that are almost impossible to repair afterward. Someone described the result as "two of every VM running," which usually means a very long day for whoever maintains the infrastructure.

To prevent that scenario, cluster software follows a conservative rule: if a node can't confirm it belongs to the majority, it refuses to keep running services, even if the administrator knows the situation is safe.

Maintenance mode doesn't change the math

Many people are surprised that maintenance mode doesn't alter this behavior. It feels like it should, since placing nodes into maintenance mode clearly tells the cluster that those machines are intentionally leaving the workload pool.

Maintenance mode is about scheduling and migration, though. It doesn't override the consensus system that governs cluster safety. Once two nodes stop communicating, the remaining host still sees a loss of quorum. The cluster logic doesn't care that the nodes were intentionally rebooted; it simply observes that communication with the majority has disappeared. A distributed system can only act on the signals it receives, and it has no way to see intent.

The VMware comparison that always comes up

Whenever this situation appears, comparisons with other virtualization platforms surface quickly.

Some administrators point to VMware's datastore heartbeating system. In that model, hosts can write information directly to shared storage, allowing other nodes to confirm whether a VM is truly running somewhere else. That approach gives the cluster another signal beyond network communication.

It's not a universal solution, though. Different platforms make different design choices about how cluster safety should work. Some rely heavily on storage signals, while others prioritize network consensus between nodes.

One comment summed it up with a sligh