Proxmox, Power Failures, and the One Time initramfs Quietly Ruined My Week

Power outages are supposed to be boring. Lights go out, lights come back on, and maybe your microwave clock blinks 12:00 like it's 1998 again. In a homelab, though, they can turn into something much more dramatic — the kind where your server boots to a black screen and a cryptic error message that feels way too calm for the situation. That's exactly how this week started. After a brief power cut at home, my Proxmox server came back… sort of. The BIOS loaded. GRUB showed up. The kernel started. And then everything stopped with a message that basically said, "root filesystem not mounted." No shell. No login. No friendly recovery prompt. Just a system that absolutely refused to go any further. If you run Proxmox at home, this story might feel uncomfortably familiar. ## The False Comfort of "It Was Working Yesterday" One of the most frustrating parts of boot failures is how unfair they feel. Nothing changed. No updates. No new disks. No late-night tinkering. The server shut off because the power went out, and now it wouldn't come back. That's it. Naturally, the first instinct was denial. Reboot again. Same error. Try an older kernel from the GRUB menu. Same result. At this point, you start bargaining with the machine. Maybe it just needs one more restart. Maybe the error will magically fix itself. It won't. When Linux tells you it can't mount the root filesystem, it's not being dramatic. Something genuinely important is missing, broken, or unreadable. And once you hit that point, there's no graceful way forward without rolling up your sleeves. ## Boot Errors That Tell You Almost Nothing "Root fs not mounted" sounds clear until you realize how many things it could mean. - Is the disk dead? - Is the filesystem corrupted? - Did the UUID change? - Is the kernel missing drivers? - Is initramfs broken? The system doesn't care which one it is, and it's not going to help you narrow it down. That part is on you. The first real step was booting from a live environment. Any Linux live ISO will do, but the Proxmox installer itself works fine for recovery. Once you're in, you can finally see what the machine sees. Drives were there. Partitions looked normal. ZFS pools imported cleanly. No obvious filesystem corruption. That ruled out the nightmare scenarios. So why couldn't the system boot? ## initramfs: Small File, Huge Responsibility If you've never had to think about initramfs before, congratulations. That means it's been quietly doing its job. initramfs is a tiny, temporary filesystem loaded into memory during early boot. Its entire purpose is to load the drivers and modules needed so the kernel can find and mount your real root filesystem. If it's missing a storage driver, a filesystem module, or anything critical, the kernel hits a wall and gives up. Power loss during updates is one of the easiest ways to break it. And once initramfs is broken, the symptoms look exactly like a dead system — even if your disks are perfectly fine. ## The "This Should Be Easy" Fix That Wasn't In theory, the fix is simple: regenerate initramfs. From a chroot or recovery shell, you run the usual command to rebuild it for the installed kernels. On most systems, that's the end of the story. Except this time, it wasn't. Every attempt to rebuild initramfs failed. Not with a helpful error, either. Just enough output to tell me something was wrong, followed by a hard stop. At first glance, it looked like another broken dependency or a half-installed package. This is where things got interesting. ## The Package You Forgot You Ever Installed Months earlier, I had temporarily installed an old NVIDIA GPU in the server. It was a quick experiment, nothing serious. I installed a helper package to make GPU virtualization easier, tested it, and eventually removed the card. What I didn't remove, apparently, was everything else. That NVIDIA-related helper package had quietly stuck around. And now, during initramfs generation, it was failing hard because the hardware it expected simply wasn't there anymore. Under normal circumstances, you'd never notice. The system booted fine. Updates worked. Life went on. Until the power went out. ## When Old Experiments Come Back to Haunt You This is the part that hurts the most. The failure wasn't caused by something I did recently. It wasn't a risky update or a rushed config change. It was technical debt. The kind you forget about because it never complains. initramfs doesn't care how old a package is. If it's hooked into the boot process and it fails, your entire system goes down with it. Once the NVIDIA packages were fully removed — not just the obvious ones, but all the helpers and dependencies — the rebuild finally worked. initramfs regenerated cleanly. Kernels rebuilt without errors. On the next reboot, the system came back like nothing had ever happened. ## Why Graceful Shutdowns Rarely Do This It's worth saying out loud: a normal reboot or shutdown almost never causes this kind of problem. Modern filesystems are good at protecting themselves. initramfs updates are usually atomic. But power loss doesn't care about timing. If the system is in the middle of updating kernels, rebuilding initramfs, or touching bootloader files when the power drops, you're rolling the dice. Most of the time, you win. Sometimes, you really don't. ## Lessons Learned the Hard Way A few takeaways stand out after the dust settles. First, initramfs deserves more respect than it gets. It's small, invisible, and absolutely essential. When it breaks, the system doesn't limp — it just stops. Second, old packages matter. Especially ones that hook into low-level parts of the system. If you're done experimenting, clean up properly. Your future self will thank you. Third, recovery is often possible. Boot failures feel catastrophic, but most of the time the data is fine. The OS just needs help getting out of its own way. And finally: yes, get a UPS. A short power outage shouldn't be enough to ruin your week, but without battery backup, it absolutely can. Pairing a UPS with proper shutdown signaling is one of those boring investments that only feels exciting the day it saves you. ## The Quiet Kind of Failure That Teaches You the Most There's something uniquely frustrating about a problem that doesn't announce itself until everything stops working. No warnings. No degraded mode. Just a dead-silent server after a flicker of the lights. But those are also the failures that teach you the most about how your system actually works. Not the glossy parts. The early boot steps. The tiny files that hold everything together. initramfs ruined my week, sure. But it also earned a permanent place on the mental checklist. Next time the power goes out, at least I'll know where to look first.