How a Power Failure and a Broken initramfs Took Down My Proxmox

Power outages are supposed to be boring. The lights go out, the lights come back on, and maybe your microwave clock blinks 12:00 like it's 1998 again. In a homelab, though, they can get much more dramatic, like when your server boots to a black screen and a cryptic error message that sounds way too calm for the situation.

That's exactly how this week started.

After a brief power cut at home, my Proxmox server came back… sort of. The BIOS loaded, GRUB showed up, and the kernel started. Then everything stopped with a message that basically said, "root filesystem not mounted." There was no shell, no login, and no friendly recovery prompt, just a system that flatly refused to go any further.

If you run Proxmox at home, this story might feel uncomfortably familiar.

The false comfort of "it was working yesterday"

One of the most frustrating parts of boot failures is how unfair they feel. Nothing had changed: no updates, no new disks, no late-night tinkering. The server shut off because the power went out, and now it wouldn't come back.

Naturally, my first instinct was denial. I rebooted again and got the same error. I tried an older kernel from the GRUB menu, with the same result. At that point you start bargaining with the machine. Maybe it just needs one more restart, or maybe the error will fix itself.

It won't. When Linux tells you it can't mount the root filesystem, it isn't exaggerating. Something genuinely important is missing, broken, or unreadable, and once you hit that point, there's no graceful way forward without rolling up your sleeves.

Boot errors that tell you almost nothing

"Root fs not mounted" sounds clear until you realize how many things it could mean.

• Is the disk dead?
• Is the filesystem corrupted?
• Did the UUID change?
• Is the kernel missing drivers?
• Is initramfs broken?

The system doesn't care which one it is, and it won't help you narrow it down. That part is on you.

The first real step was booting from a live environment. Any Linux live ISO will do, but the Proxmox installer itself works fine for recovery, and once you're in, you can finally see what the machine sees.

The drives were there and the partitions looked normal. ZFS pools imported cleanly, and there was no obvious filesystem corruption, which ruled out the nightmare scenarios. So why couldn't the system boot?

initramfs: small file, huge responsibility

If you've never had to think about initramfs before, congratulations. It means it's been doing its job without complaint.

initramfs is a tiny, temporary filesystem loaded into memory during early boot. Its entire purpose is to load the drivers and modules the kernel needs to find and mount your real root filesystem. If it's missing a storage driver, a filesystem module, or anything else critical, the kernel hits a wall and gives up.

Power loss during updates is one of the easiest ways to break it. And once initramfs is broken, the symptoms look exactly like a dead system, even if your disks are perfectly fine.

The "this should be easy" fix that wasn't

In theory, the fix is simple: regenerate initramfs. From a chroot or recovery shell, you run the usual command to rebuild it for the installed kernels, and on most systems that's the end of the story.

This time it wasn't. Every attempt to rebuild initramfs failed, and not with a helpful error either. I got just enough output to tell me something was wrong, followed by a hard stop. At first glance it looked like another broken dependency or a half-installed package.

This is where things got interesting.

The package you forgot you ever installed

Months earlier, I had temporarily installed an old NVIDIA GPU in the server. It was a quick experiment, nothing serious. I installed a helper package to make GPU virtualization easier, tested it, and eventually removed the card.

What I apparently didn't remove was everything else. That NVIDIA-related helper package had stuck around unnoticed, and now, during initramfs generation, it was failing hard because the hardware it expected simply wasn't there anymore.

Under normal circumstances, you'd never notice. The system booted fine, updates worked, and life went on, right up until the power went out.

When old experiments come back to haunt you

This is the part that hurts the most. The failure didn't come from anything I did recently, like a risky update or a rushed config change. It came from technical debt, the kind you forget about because it never complains.

initramfs doesn't care how old a package is. If the package is hooked into the boot process and it fails, your entire system goes down with it.

Once the NVIDIA packages were fully removed, including all the helpers and dependencies along with the obvious ones, the rebuild finally worked. initramfs regenerated cleanly and the kernels rebuilt without errors. On the next reboot, the system came back like nothing had ever happened.

Why graceful shutdowns rarely do this

It's worth saying out loud that a normal reboot or shutdown almost never causes this kind of problem. Modern filesystems are good at protecting themselves, and initramfs updates are usually atomic.

Power loss doesn't care about timing, though. If the system is in the middle of updating kernels, rebuilding initramfs, or touching bootloader files when the power drops, you're rolling the dice. Most of the time you win. Sometimes you really don't.

Lessons learned the hard way

A few takeaways stand out after the dust settles.

First, initramfs deserves more respect than it gets. It's small, invisible, and absolutely essential, and when it breaks, the system doesn't limp along. It just stops.

Second, old packages matter, especially ones that hook into low-level parts of the system. If you're done experimenting, clean up properly. Your future self will thank you.

Third, recovery is often possible. Boot failures feel catastrophic, but most of the time the data is fine and the OS just needs help getting out of its own way.

And finally: yes, get a UPS. A short power outage shouldn't be enough to ruin your week, but without battery backup it absolutely can. Pairing a UPS with proper shutdown signaling is one of those boring investments that only feels exciting the day it saves you.

The silent failures teach you the most

There's something uniquely frustrating about a problem that doesn't announce itself until everything stops working. You get no warnings and no degraded mode, just a dead-silent server after a flicker of the lights.

Those are also the failures that teach you the most about how your system actually works: the early boot steps and the tiny files that hold everything together, far from the glossy parts.

initramfs ruined my week, sure, but it also earned a permanent place on my mental checklist. Next time the power goes out, at least I'll know where to look first.