Ceph 2 OSD Cluster: Why 3 MONs Don't Make It HA

You can get a Proxmox cluster with three Ceph MONs and only two OSDs working, and it still makes a poor high-availability storage design.

It's easy to see how people end up here. Three MONs give Ceph monitor quorum, so the control plane looks healthy. Two OSDs can hold two copies of the data, so the pool looks healthy too while everything is online. Add a separate Ceph network and the diagram starts to look redundant. Then one storage node fails, and the gap between MON quorum and data redundancy starts to hurt.

Proxmox defaults Ceph pools to three replicas, with at least two available for I/O. A two-OSD design can't meet that normal 3/2 policy. Dropping the pool to two replicas makes the design fit the hardware, but it throws away much of the failure margin that makes Ceph worth running. Getting Ceph to start is easy; the thing to ask is what's left after something breaks.

Do three MONs make a two-OSD Ceph cluster highly available?

No, because MON quorum and OSD redundancy solve different problems. A Ceph Monitor, or MON, maintains the cluster maps and takes part in consensus about cluster state. You want an odd number of them so a majority survives one MON failure. An OSD, or Object Storage Daemon, stores the actual data.

If three MONs are alive but a pool has lost too many OSD replicas, quorum can't conjure up another copy of the missing data. The monitors will simply agree that the storage is degraded or unavailable.

Most of the confusion around this small-cluster design came from mixing those two up. The proposed setup had three Proxmox nodes and three MONs, with OSDs on only two of them. The third machine was intentionally weaker and would run hardware-bound workloads, such as Home Assistant with a Zigbee device.

A Ceph client node with no OSD is fine. A Proxmox node doesn't need its own OSD to use an RBD pool; it reaches Ceph storage over the public network like any other client. The weak point is that the whole data set lives in only two places, and the third node lacking an OSD has nothing to do with that.

Why does Proxmox default Ceph pools to size 3 and min_size 2?

Three copies give a meaningful failure margin and still allow I/O while one replica is temporarily unavailable.

In current Proxmox Ceph configuration, the pool defaults are:

• size = 3
• min_size = 2

size is the number of replicas Ceph tries to keep for each object. min_size is the minimum number of replicas that must be available before the pool accepts I/O.

A healthy 3/2 pool keeps three copies. If an OSD or host fails and one copy disappears, two remain, and the pool keeps working while Ceph restores redundancy.

With only two OSDs you can't have three distinct replicas, which leaves two obvious options, and neither behaves like normal 3/2 Ceph.

The first is size=2, min_size=2. Both copies are now required for I/O, so losing one of the two storage OSDs can stop the pool accepting I/O, because only one replica is left.

The second is size=2, min_size=1. The pool keeps going on one surviving copy. That sounds more available until you realize your distributed storage is now serving data from its only remaining replica. One more disk problem, a bit error, an operator mistake, or any other failure can turn a degraded pool into lost data.

Proxmox staff have repeatedly warned against treating 2/1 as the normal answer. In a Proxmox forum discussion, staff called 2/1 risky because a failure on the remaining copy can leave you with no good replica, and another staff member recommended sticking with the default 3/2 unless you have a special requirement. Keeping I/O flowing and keeping data safe are separate goals, and 2/1 trades away some of the second to keep the first.

Is the network actually the biggest problem?

Not in this design. The OSD count is the first architectural problem, and the proposed network could add a second.

Proxmox recommends at least 10 Gbps dedicated to Ceph traffic. Its documentation specifically warns that Ceph recovery traffic can interfere with the latency-sensitive Corosync cluster network and may cost you cluster quorum. The design discussion included putting Ceph public traffic on the same network as Corosync. That can work in a lab, but I wouldn't build it that way on purpose if I cared about failure behavior.

Corosync needs little bandwidth but is very sensitive to latency and jitter. Ceph gets noisy at exactly the wrong moments, when the infrastructure is already stressed by an OSD failure, recovery, backfill, or rebalancing. A quiet cluster can look perfect in testing. The test that matters is pulling an OSD while guests are generating I/O. If Ceph recovery floods the path Corosync depends on, a storage failure starts affecting cluster membership too, and two independent problems turn into one big outage.

For more on planning small clusters, the Mr.PlanB Proxmox hub covers storage, clustering, migration, and failure design together, so you aren't tuning each one as an isolated setting.

What are the Ceph public and cluster networks actually for?

Calling the public network "the MON network" misses most of what it does. Ceph clients talk directly to OSDs over the public network. MONs are reachable there too, but more importantly, VM storage traffic from Proxmox nodes reaches Ceph over this front-side network.

The optional cluster network is for something else. When you configure one, Ceph uses it for OSD heartbeats, object replication, and recovery traffic between OSDs.

So a Proxmox node with no OSD still needs a good path to the Ceph public network if it will run VMs stored on the Ceph pool. Running a MON doesn't mean it has to take part in OSD replication. Someone in the discussion made exactly this correction: the third node can use the Ceph pool without an OSD, because access to the pool isn't limited to nodes that physically store replicas.

This is also why the slowest client link matters. If one compute node has only 1GbE to the Ceph public network while the storage nodes have 2.5GbE or 10GbE, VMs on that compute node are limited by the slower connection. That doesn't drag every Ceph link in the cluster down to 1Gbps, but workloads on the slow node can feel far worse than those on better-connected nodes.

Can Ceph work over 1GbE or 2.5GbE?

Yes, though working at all and working well during recovery are very different bars.

One administrator who had run a similar two-OSD, three-node setup called it functional but unpleasant. They said it felt roughly like running VMs off very slow flash storage, with high latency and throughput about what you'd expect from a saturated 1GbE link. That's one person's experience rather than a general Ceph benchmark, but the underlying limit is real.

Ceph clients talk directly to OSDs, the OSDs then replicate, and recovery and backfill can add another large stream on top. The network carries a lot more than the original guest write. The Ceph project notes that OSD replication puts extra load on the storage network. Its current networking guidance says a single public network is enough for many deployments, especially with fast networking, and that a separate cluster network helps when traffic is heavy.

Proxmox is more prescriptive for hyperconverged deployments and recommends at least 10 Gbps for Ceph. With modern NVMe, even 10GbE can be the bottleneck.

So 2.5GbE is perfectly reasonable for a Ceph learning lab. It's much harder to justify for storage that's supposed to stay fast and predictable when things fail.

What happens if one of the two OSD nodes fails?

It depends on the pool's replica settings, and neither outcome is great. With two replicas and min_size=2, losing one copy can block I/O because the pool drops below its required minimum. You're protected from writing to a single copy, but the service isn't highly available.

With two replicas and min_size=1, the surviving OSD keeps serving I/O. The service stays up, but the affected data may now depend on a single replica, which is the exact state distributed storage is meant to avoid.

Three MONs don't change either case. Monitor quorum can be perfectly healthy while the pool is one hardware event away from losing data. The most upvoted technical reply to the design summed it up: it can work, but it's not a good idea.

Would adding a third OSD change the design?

Yes. A third OSD in a third failure domain changes the failure model far more than another MON would.

With three storage hosts, Ceph can place three replicas on three hosts and use the normal 3/2 policy. A host can then disappear with two copies still in place. That's much closer to what people usually mean by Ceph high availability.

The catch is that the third machine in this design was deliberately weaker, and that's where hardware symmetry comes in. Ceph performs best when OSD capacity and performance are reasonably balanced across hosts. A much slower storage node can affect recovery, placement, and how the whole pool feels. You can build asymmetric Ceph, but pushing the weaker machine into the storage layer isn't always better than choosing a simpler storage model.

Is ZFS replication the better choice for a three-node homelab?

Often, yes, especially when only two nodes are suitable as storage servers.

Several experienced users in the discussion recommended stepping back from Ceph and using local ZFS with Proxmox storage replication. That deserves more attention than it usually gets, because ZFS replication can be part of a Proxmox HA design. Proxmox can asynchronously replicate guest disks from local ZFS storage to another node, and if the source node fails, HA can restart the guest from the latest replicated copy on the target.

The tradeoff is RPO. Replication runs on a schedule, so a recovered VM can lose changes made after the last successful run. The default pvesr schedule is every 15 minutes, and you can make it more aggressive if the hardware and network can keep up.

Ceph aims to keep shared storage replicated continuously. ZFS replication accepts a recovery-point window in exchange for much simpler infrastructure. In a home cluster where some VMs are tied to specific nodes anyway, that can be the more sensible trade.

One contributor said they use Ceph at work but ZFS replication at home, because Ceph's performance and capacity overhead was hard to justify in a three-node, six-OSD homelab. Their point was matching the architecture to the scale, which is a narrower claim than ZFS being "better than Ceph."

What about LINSTOR for only two storage nodes?

LINSTOR is worth knowing about, though it swaps one kind of distributed-storage complexity for another.

A few participants suggested LINSTOR with DRBD, which can replicate block storage synchronously across a small number of nodes. One user described a three-node Proxmox environment where only two nodes held the replicated storage and said it worked well. That fits a topology where two machines are strong storage nodes and the third mostly provides compute or quorum.

LINSTOR brings its own operational model, though: DRBD concepts, controllers, satellites, and support considerations. If you specifically want to learn distributed storage, that can be interesting. If you just want to survive a host reboot, local ZFS replication is usually the smaller mental leap.

What should you build if the goal is learning Ceph?

Build Ceph, and call it what it is: a lab. Running a two-OSD cluster on purpose to learn how MONs, MGRs, OSDs, pools, CRUSH, RBD, recovery, and network paths behave is fine. An imperfect setup can teach you more than a clean production recipe. Just don't mistake a successful install for proof that the architecture holds up, and test the ugly cases:

1. Pull one OSD.
2. Watch what happens to I/O.
3. Check whether the pool becomes degraded, inactive, or continues on one copy.
4. Saturate the Ceph network.
5. Watch Corosync latency at the same time.
6. Move a VM onto the slowest compute node.
7. Measure guest disk latency.
8. Restore the failed OSD and observe recovery traffic.
9. Test a complete host outage.
10. Restore a VM from backup after deliberately destroying the lab copy.

A Proxmox Health Check can help find host and cluster-side weaknesses, but for Ceph the most useful test is still failure under load.

A working two-OSD Ceph cluster says little about the design

What makes the original architecture interesting is that almost every piece of it is valid on its own. Three MONs are sensible. A node without an OSD can still use the Ceph pool. A separate cluster network can carry OSD replication, two OSDs can store two replicas, and Ceph does run on 1GbE or 2.5GbE. Put all of those true statements together and you still get a system with a weak failure model.

For a real three-node Ceph design, aim for three storage failure domains, keep the normal 3/2 replica policy unless you have a well-understood reason to change it, and give Ceph enough bandwidth that recovery doesn't threaten Corosync. If your hardware can't do that yet, ZFS replication may simply be the architecture that fits the cluster you own.