When a Cluster Loses Its Mind: The Harsh Reality Behind Three-Node High Availability

# When a Cluster Loses Its Mind: The Harsh Reality Behind Three-Node High Availability ## The Setup That Looks Perfect on Paper A small virtualization cluster can feel like the sweet spot for reliability. Three nodes. Shared storage over iSCSI or NFS. High availability configured so virtual machines can move seamlessly between hosts. Maintenance mode works beautifully: you put one node into maintenance, live migration kicks in, workloads move away, and the host reboots peacefully. Everything behaves exactly as advertised. That’s why the situation becomes so confusing when an administrator tries something seemingly harmless. Two hosts go into maintenance mode. All the virtual machines migrate successfully to the third node. Nothing is overloaded. The hardware is powerful enough to run every workload on a single machine. Then both maintenance nodes reboot. And suddenly the final node starts behaving like something has gone terribly wrong. Services panic. Sometimes the host even reboots. What looked like a routine maintenance window suddenly feels like the cluster is fighting back. For anyone watching the console scroll past error messages, the question becomes immediate and frustrating: why can’t the last node simply keep running the VMs? ## The Cluster’s Paranoia Kicks In The answer lies in a rule that quietly governs most high-availability clusters: **majority consensus**. Distributed systems don’t assume they understand what’s happening in the network. They assume the opposite. Networks fail, links drop, switches glitch, and packets disappear. Because of that uncertainty, clusters rely on quorum — the ability for a majority of nodes to confirm that the cluster state is valid. In a three-node cluster, that majority is two nodes. The moment two nodes disappear, the remaining host loses quorum. From the cluster’s perspective, it doesn’t know if the other machines are actually offline or if it’s the one isolated from the rest of the system. One engineer summarized it in a blunt way: when two nodes vanish, the last one assumes *it might be the problem*. That suspicion triggers protective behavior. Instead of continuing to run workloads blindly, the cluster halts operations to avoid something far worse than downtime. ## The Disaster Everyone Is Trying to Avoid The reason behind this strict rule is something cluster engineers fear more than almost anything else: **split brain**. Imagine the cluster allowed the final node to continue running every VM without question. Now imagine the original scenario wasn’t a planned reboot but a network partition. One node loses connectivity to the other two. If the isolated node keeps running workloads while the other nodes assume it’s dead, the cluster might start those same virtual machines elsewhere. Suddenly two identical instances of the same VM exist at the same time. Databases write conflicting data. Filesystems become corrupted. Transactions diverge in ways that are almost impossible to repair afterward. Someone described the result as “two of every VM running,” which usually means a very long day for whoever maintains the infrastructure. To prevent that scenario, cluster software follows a conservative rule: if a node can’t confirm it belongs to the majority, it refuses to keep running services. Even if the administrator knows the situation is safe. ## Maintenance Mode Doesn’t Change the Math One detail that surprises many people is that maintenance mode doesn’t alter this behavior. It feels like it should. After all, placing nodes into maintenance mode clearly tells the cluster that those machines are intentionally leaving the workload pool. But maintenance mode is about scheduling and migration. It doesn’t override the fundamental consensus system that governs cluster safety. Once two nodes stop communicating, the remaining host still sees a loss of quorum. The cluster logic doesn’t care that the nodes were intentionally rebooted. It simply observes that communication with the majority has disappeared. In distributed systems, intent is invisible. Only signals matter. ## The VMware Comparison That Always Comes Up Whenever this situation appears, comparisons with other virtualization platforms surface quickly. Some administrators point to VMware’s datastore heartbeating system. In that model, hosts can write information directly to shared storage, allowing other nodes to confirm whether a VM is truly running somewhere else. That approach gives the cluster another signal beyond network communication. But it’s not a universal solution. Different platforms make different design choices about how cluster safety should work. Some rely heavily on storage signals. Others prioritize network consensus between nodes. One comment summed it up with a sligh