Ceph on 2.5GbE Works, but Everything Around It Is the Problem

The hardware looked close to ideal for a compact Proxmox cluster: two Lenovo M920q systems, one M720q, 32 GB of RAM in each, a small SATA boot SSD, and a 1 TB NVMe drive for storage. A much stronger Dell tower was still running every VM and LXC. The new cluster had to survive a node failure without recovery eating a weekend.

The storage question is where the tidy little setup got complicated. Ceph looked like the obvious answer, with shared distributed storage, automatic redundancy, and better use of multiple nodes. The trouble was what came along with it: network upgrades, more writes, less usable capacity, possible consumer SSD wear, cooling, extra NICs, maybe a dedicated switch, maybe 10GbE after all. At that point you're designing infrastructure, which is a bigger job than setting up virtualization.

The 2.5GbE question is almost a distraction

The loudest question was whether Ceph could work on 2.5GbE. People running small Ceph environments said yes.

One user reported running Ceph over 2.5GbE without trouble. Another described a mini PC cluster on bonded 2.5GbE links that reached roughly 750 MB/s peak writes and close to 900 MB/s reads in that particular setup. Someone else said a home Ceph deployment had been acceptable even on 1GbE. None of those anecdotes proves 2.5GbE is ideal, but they do show that Ceph runs fine below 10GbE.

The more useful argument was about what happens when the cluster is busy, rebuilding, or recovering from a failure. Ceph constantly moves replicated data between nodes, so the storage network carries more than a normal VM workload. A link that feels fine during light homelab use gets slow fast when a disk fails and the rebuild starts.

That's why several commenters still pushed toward 10GbE. One suggested dual port Mellanox cards in a direct ring between the three Lenovo nodes. Another was already running Intel X520 or X550 adapters in similar Tiny systems. The appeal was practical: used 10GbE hardware stops the network from being the first bottleneck you hit, and migrations get faster too.

The official Proxmox guidance is more conservative than the homelab anecdotes. Proxmox recommends at least 10 Gbps dedicated to Ceph traffic and warns that recovery traffic can interfere with latency sensitive cluster communication if everything shares one network. For NVMe based storage, it notes that a single fast device can already exceed what 10GbE can comfortably carry. So 2.5GbE is possible, but it still isn't the most important design decision here.

Ceph still costs the storage you thought ZFS was wasting

The original hesitation about ZFS replication was capacity. Replicating VM and LXC storage across nodes looked expensive because the copies take space. Part of Ceph's appeal was that it seemed like a smarter way to pool the three 1 TB NVMe drives.

Several commenters pointed out that Ceph replication makes copies too. A common three replica configuration stores data three times, so a small cluster can end up with roughly one third of raw capacity for unique data. One user said it bluntly: Ceph was not going to create more usable space. Another compared the options and landed in the same place. ZFS replication and Ceph both trade capacity for copies, with different architectures underneath.

That changes the decision a lot. With ZFS replication, you don't have to replicate every guest to every node. Several people argued that only the important VMs and containers need it, and less critical workloads can rely on backups or manual recovery. A small homelab can then spend its storage where it counts.

Ceph appeals for other reasons. Storage is distributed and available across the cluster, which is much closer to real hyperconverged infrastructure. A disk failure doesn't necessarily take the whole compute node out of service, and the storage layer can rebuild redundancy on the remaining devices, which is a powerful feature.

Three small systems with one main NVMe each, though, sit very close to the minimum shape of a Ceph cluster, and several experienced commenters saw that as the weak point. One called three nodes technically workable but with little room for failure. Another said Ceph on three nodes made sense as a learning opportunity but felt like overkill if the goal was keeping home services up.

If you're building the lab to learn Ceph, the inconvenience is part of what you're learning. If you're building it to keep DNS, Home Assistant, storage, or family services online, success looks different.

The SSD problem is nastier than the network problem

The thread got more interesting when it moved from Ethernet speed to SSD endurance. The proposed nodes used consumer style NVMe drives. Several people warned right away that Ceph can be rough on consumer flash because distributed storage writes a lot. One commenter had used consumer SSDs in a small Ceph environment and said they wore out in about six months, after which they were replaced with Intel data center SSDs. That's one person's experience and doesn't predict lifespan in general, but a story like that should change how you judge a cheap homelab build.

Another user with almost the same Lenovo cluster said this exact limitation was why those machines ended up unused. Their replacement platform could hold a boot device plus two dedicated Ceph drives. Others recommended enterprise SSD models, which brought up a size problem: some inexpensive enterprise NVMe devices use longer form factors that don't fit easily inside Tiny PCs.

From there the 2.5GbE plan keeps growing. A PCIe riser can take a 10GbE NIC. The WiFi slot can sometimes take another NVMe device through an adapter, and some builders moved the Proxmox boot drive into that slot to free up SATA or NVMe capacity. Others removed SSD casings, added small fans, printed custom shrouds, or used SATA power adapters to keep hot network cards under control. All of that works, though it turns a neat three node cluster into a miniature engineering project.

Heat came up again and again. The machines are small, the rack is dense, and 10GbE adapters run hot. One builder with a similar M920 based setup added a dedicated 40 mm fan for the NIC. Another commenter said throttling and heat had been a real problem for them. The original rack already had an open back and a 120 mm fan.

By this point the question is how many extra compromises you'll accept before this stops being the simple HA setup you wanted. Whether Ceph can run on 2.5GbE matters less.

ZFS replication and PBS keep winning the practicality argument

The strongest pushback against Ceph came from people asking what problem actually needed solving, and none of them claimed Ceph was bad.

The stated priority was VM and LXC recovery after a node failure, and several commenters thought ZFS replication covered that. Proxmox storage replication sends guest volumes to another node incrementally, and HA can be combined with replication, although you can lose data written between the last replication and the failure. That tradeoff matters, but it's much easier to reason about than distributed storage across three tiny machines.

One user said a similar three mini PC setup with local ZFS and replication worked well. Another recommended starting with ZFS because only the important workloads need replicating, which is a far more economical use of 1 TB drives than treating every byte as critical.

Someone else had trusted ZFS based HA, tested node failures successfully, and later had a VM disk disappear after an HA event. A backup saved them. It's one anecdote and doesn't establish a general ZFS failure pattern, but it's a reminder that replication doesn't replace backup.

That's where Proxmox Backup Server kept coming up. The administrator already had PBS running as a VM, with backups stored on a UNAS Pro, and planned to move PBS to bare metal. Several commenters thought that was enough for a home environment: spread workloads across the compute nodes, keep good backups, accept that a failed node may mean a restore, and don't spend a surprising amount of money and time to avoid a few hours of downtime.

It's less exciting than Ceph, and when downtime costs you inconvenience and not revenue, it may also be the more sensible design.

For more Proxmox planning topics, the Mr.PlanB Proxmox hub covers clustering, storage, and migration decisions from the same practical angle.

The best homelab architecture depends on what you want to learn

Most of the disagreement came down to what a homelab is supposed to optimize.

One group wanted real distributed storage and high availability. For them, 10GbE NICs, direct attached cables, extra airflow, enterprise SSDs, and more storage devices were part of building it properly, and Ceph was the whole point.

A second group cared more about services staying up than about the infrastructure itself. Their answer was ZFS replication for selected guests, PBS for recovery, and much less hardware. If a node dies, you restore what matters, since a few hours of downtime costs less than redesigning every Tiny PC around Ceph.

Then there was the middle ground: run Ceph on 2.5GbE now, see how it behaves, and upgrade when the limits actually show up. Several people had already shown that small Ceph deployments can work this way, as long as you accept the bottleneck, rebuild time, SSD wear, and thermal limits as part of the experiment.

My read of the whole debate is that 2.5GbE doesn't make this design risky. The risk comes from wanting distributed storage, high availability, maximum capacity, low cost, consumer hardware, and minimal complexity all from the same three tiny machines, and you won't get all of those.

In a learning lab, working through those compromises can be fun. For infrastructure that has to run every day without attention, the boring option usually holds up better.