The Day a Three-Node Cluster Refused to Trust Itself
When "high availability" suddenly feels fragile
Running a three-node virtualization cluster usually brings a comforting sense of redundancy. You reboot one node and the other two keep everything alive, while maintenance mode migrates virtual machines smoothly. Workloads slide between hosts like a well-choreographed dance while administrators sip coffee and watch dashboards.
Then someone tries something that feels perfectly reasonable: reboot two nodes at the same time after migrating all the VMs to the third host. Suddenly the cluster behaves like it's having an existential crisis.
The last node starts panicking. Services restart, and sometimes the host even reboots. Instead of calmly running every workload in the cluster, the final machine refuses to trust the situation. To the administrator watching from the console, it looks absurd, because the hardware is powerful enough to run everything, the VMs are already migrated, and nothing is technically overloaded.
So why does the cluster revolt the moment two nodes disappear?
The invisible rule: majority always wins
One explanation comes from people who've lived inside distributed systems for years. They point to a rule that sits beneath most high-availability clusters: majority rule.
In a three-node setup, the cluster expects at least two nodes to agree about reality. That agreement is called quorum, and once quorum is lost, the system no longer trusts its own state.
One engineer explained it bluntly: when two nodes disappear, the remaining node assumes the problem might actually be itself. From its perspective, the other machines may not have vanished at all, and it might simply be isolated.
This sounds paranoid, but the paranoia exists for a reason. Distributed systems treat isolation as dangerous because it can lead to something worse than downtime, which is split brain.
The nightmare scenario engineers fear
Imagine the cluster behaves the way administrators initially expect. Two nodes go offline, and the last node keeps running every VM without complaint.
Now imagine a different situation. Instead of a planned reboot, the network link between nodes fails, and the cluster splits into two groups that can't see each other. If both sides believe they're the legitimate cluster, they might start running the same virtual machines simultaneously.
That's how you end up with duplicate databases writing to the same storage, two copies of a VM processing transactions, and filesystems corrupted beyond repair. One admin described the outcome in colorful terms: suddenly you have two of every VM running, and the day goes downhill fast.
To prevent that nightmare, cluster software follows a strict rule. If a node can't confirm it's part of the majority, it refuses to run workloads, even if that means shutting down services that were perfectly healthy seconds earlier. From the system's perspective, stopping everything is the safer option.
Why maintenance mode doesn't solve it
Many administrators assume maintenance mode should change the equation. After all, placing nodes in maintenance mode migrates workloads and signals the cluster that those machines are intentionally stepping away. If the cluster knows two nodes are deliberately offline, the reasoning goes, the remaining host should keep running the VMs.
The cluster doesn't see it that way. Maintenance mode handles workload migration and scheduling decisions, and it doesn't change the quorum rules governing cluster safety. Once two nodes disappear from communication, the remaining node still loses majority agreement.
From the software's perspective, there's no reliable proof the other hosts are truly offline and not simply unreachable. Distributed systems operate on one harsh assumption: networks lie all the time.
The VMware comparison that always appears
Whenever this behavior surprises someone, comparisons to other virtualization platforms appear almost instantly.
In some environments, administrators are used to mechanisms like datastore heartbeating or storage locks. Those systems allow hosts to verify whether another machine still owns a VM or has lost access to shared storage.
That approach creates a different style of failure detection. Instead of relying primarily on cluster quorum, the system looks at shared storage signals to determine whether a VM is still alive elsewhere.
Some engineers prefer this model, and others see it as trading one set of risks for another. As one commenter pointed out, every platform makes different design choices about where trust comes from, whether that's network consensus, storage locks, or some hybrid combination, and none of them are perfect.
The blunt simplicity of Proxmox's philosophy
The design philosophy behind many open-source clusters is almost stubbornly simple. If a node can't confirm it's part of the majority, it assumes something dangerous might be happening and shuts things down, without clever guessing or optimistic assumptions.
This approach puts consistency and data safety above convenience, so administrators must follow strict operational patterns. Reboot nodes one at a time, maintain quorum, and avoid situations where the cluster loses its voting majority.
Some people find that frustrating, because the system seems to ignore obvious intentions during maintenance windows. From a distributed-systems perspective, though, the cluster is behaving exactly as designed and protecting data from scenarios humans often underestimate.
The hidden cost of small clusters
This situation also shows a limit of small clusters that's easy to forget. A three-node cluster can only tolerate one failure at a time, and the moment two nodes disappear, quorum is gone.
That limitation leads to the advice that inevitably surfaces in conversations like this: if you want the cluster to survive losing two nodes, you need more nodes. It sounds almost dismissive, but mathematically it's accurate. Larger clusters increase the number of failures that can occur without breaking majority consensus. A five-node cluster can lose two nodes and still maintain quorum, and a seven-node cluster can lose three. With three nodes, there's simply no margin.
The moment the design finally makes sense
For administrators encountering this behavior for the first time, the system feels overly cautious. They know exactly what they're doing, they deliberately placed nodes into maintenance mode, and the scenario is controlled.
Distributed systems don't rely on human certainty, though. They rely on rules that survive chaotic conditions like broken switches, network partitions, crashed hosts, and corrupted memory. Those rules often look stubborn during routine maintenance, and they exist because someone, somewhere, once lost an entire cluster to split brain.
So the next time a three-node cluster refuses to keep running after two nodes disappear, what looks like panic is the cluster declining to guess about a state it can't confirm.