Proxmox UPS Shutdown: Why Cluster Order Matters

A UPS shutdown for one Proxmox host is simple: detect low battery, stop the guests, power off the machine. A cluster is harder, because HA may try to restart guests on nodes that are already getting ready to shut down, Ceph may lose enough OSDs to block I/O, and a backup server may use a completely different API permission model.

PVE-UPS is trying to handle those problems in its recent updates. The project was already a small LXC or Docker appliance that watches a UPS over SNMP or through an existing Network UPS Tools server and shuts Proxmox hosts down through the API. It needs no agent on each node, and it stays in dry-run mode until the administrator explicitly arms it.

The newer releases add Proxmox Backup Server targets, multiple webhooks, stronger self-tests, and beta cluster-aware shutdown logic. The cluster logic is the big change, because once a cluster is involved, the order in which things shut down is part of your storage and HA design.

Why can HA make a power outage worse?

Proxmox HA exists to recover services when a node disappears. During a deliberate cluster shutdown, that recovery is exactly what you don't want.

Picture a three-node cluster on battery. Node A hits its shutdown threshold and powers off. If HA is still armed, the cluster can read that as a failure and start relocating or restarting services on Nodes B and C, which are running on the same limited battery and may be minutes from their own shutdown. The cluster ends up doing more work when it should be shedding load, so the new PVE-UPS cluster mode can use Proxmox's disarm-ha mechanism before the first node goes down.

The current Proxmox HA documentation describes a disarmed state in which watchdogs are released cluster-wide and automatic fencing, failover, and recovery stop. Proxmox warns explicitly that services aren't protected while HA is disarmed, so the window should be short. For a controlled shutdown that's what you want, since the nodes are going away on purpose and the cluster shouldn't try to recover from it.

PVE-UPS labels this cluster preparation as beta and requires Proxmox VE 9.2 or newer for disarm-ha. On older 8.x environments it does a reduced cluster preparation and doesn't pretend the same HA behavior is available. I like that, because a UPS automation tool shouldn't silently assume a cluster feature exists when the outage is already underway.

For the wider HA and storage context, see the Mr.PlanB Proxmox hub. How a cluster behaves in a power failure depends heavily on how it was designed in the first place.

Why does Ceph change the shutdown sequence?

In a Ceph HCI cluster, storage can become unavailable before the last Proxmox node shuts down.

Proxmox's own Ceph shutdown documentation opens with a rule that sounds obvious and is easy to break in automation: stop all Ceph clients first. In a hyper-converged Proxmox environment, the clients are mostly the VMs and containers using RBD or CephFS. Only once they've stopped accessing Ceph should you prepare the storage cluster for shutdown. Proxmox then recommends checking Ceph health, setting the noout flag so OSDs aren't treated as permanently lost during the planned shutdown, and powering down nodes in a controlled order.

A generic per-node shutdown can get this wrong, so PVE-UPS has a separate hyper-converged Ceph option. Its sequence follows the dependency chain:

1. Disarm HA.
2. Stop every guest in the cluster.
3. Apply the Ceph maintenance preparation.
4. Shut down the nodes in the configured order.

Step 2 is where a simpler approach goes wrong. If Node A just stops its own guests and powers off while VMs keep running on Nodes B and C, the remaining Ceph pool can drop below the replica count needed for I/O, and those surviving VMs can block on storage. Node B is then trying to shut down guests whose virtual disks have stopped responding, and the shutdown can stall while the UPS keeps draining. That's a much nastier failure than one host ignoring a poweroff command.

Why not keep one HA node running as long as possible?

Sometimes that's the right strategy. One administrator described a different outage plan: when UPS runtime drops to about half, shut down two of the three HA nodes and move every guest to the remaining one. Keep that machine running until only a few minutes of battery are left, then shut it down too.

That's a perfectly reasonable design when the workload fits on one node and the storage still works after the other nodes are gone. It isn't automatically safe for a Ceph HCI cluster, because compute HA and distributed storage have different dependencies. If the last node relies on Ceph OSDs that went down with the first two nodes, keeping it online gains you nothing: the VMs may still be running as far as the CPU is concerned, but their storage is gone.

With external NFS, iSCSI, local ZFS replication, or some other storage layout without Ceph, consolidation can make a lot more sense. Shutting the whole cluster down as a unit should be a policy you choose for your setup, and PVE-UPS makes it configurable, which is the right call.

What changed for Proxmox Backup Server?

PVE-UPS can now treat Proxmox Backup Server as a proper shutdown target of its own, instead of pretending PBS is another PVE node.

The APIs look similar, but the permission models differ. PVE uses Sys.PowerMgmt for host shutdown permissions. PBS uses Sys.PowerManagement, and its API paths and token authorization are different. That's why entering a PBS system as if it were a PVE host could fail with what looked like an invalid API token. Each host entry now has a type, so mixed PVE and PBS environments can be described correctly.

There's a security tradeoff here too. The current PVE-UPS documentation notes that PBS doesn't offer a narrow shutdown-only permission like PVE does. The documented PBS setup grants an Admin role scoped to /system/status, and the project warns production users to think carefully before giving one appliance access to both the virtualization hosts and the backup server.

That's good advice, since you run a backup server so you have an independent way to recover. If the same management appliance and management network can control both production hosts and backups, a compromise or mistake reaches further. In a production design, a separate shutdown path or management segment for PBS may be the better choice.

Why did API token privilege separation confuse users?

A working API connection doesn't prove the token is allowed to shut down the host.

One early user set up the dedicated power-management role correctly and still got a warning that Sys.PowerMgmt could not be confirmed. The cause was Proxmox API token privilege separation. With privilege separation enabled, a token needs its own ACL entries and doesn't simply inherit everything granted to its parent user. The PVE-UPS setup gives the power role to the dedicated user and creates the shutdown token with privilege separation disabled, so the token inherits the user's permission.

The relevant command uses --privsep 0, and the developer later confirmed this is intentional for the documented setup. You want to hit this kind of problem while commissioning the tool, well before the UPS is on battery, and the stronger self-tests in the newer release are meant to catch it.

What does the new self-test actually catch?

It checks the details that would otherwise only fail during a real outage, and the Proxmox node name is one example. A config might have the wrong capitalization or an added domain suffix, and a basic connection test could still pass until the shutdown call targeted a node name the API didn't recognize. The new self-test checks the configured name against the API.

It also checks cluster state once per cluster, runs automatically after the appliance re-arms, and has a manual "Run self-test now" option.

Next to cluster shutdown sequencing these look like small features, but they may matter more in practice. Administrators hope UPS automation almost never runs for real, so configuration mistakes can sit unnoticed for months or years, and the longer the gap between real events, the more you depend on self-tests.

One user later reported that the tool shut down their servers successfully during the first blackout they'd had in years. It's one anecdote, but it shows the problem well: an emergency workflow can go unused for a very long time and then suddenly be the only thing that matters.

Why do multiple webhooks matter during a blackout?

"The shutdown started" is only one of the events you might want to hear about. PVE-UPS now supports several webhook targets instead of one endpoint. Each gets its own format, severity filter, test function, and optional authorization header. Built-in formats cover Slack, Discord, ntfy, Microsoft Teams, plain JSON, and plain text, and there's a custom template option for anything else.

A failed webhook also shows up in the event log, on the target card, and on the dashboard. That's how infrastructure automation should work, since a notification system that silently stops sending is almost worse than none, because you keep trusting it.

During a power event you may want different severities to go to different places. "UPS on battery" might go to a chat channel, while "cluster shutdown has begun" or "host shutdown failed" may need a more urgent route. The project's notification settings are now fine-grained enough for that.

Should PVE-UPS run inside the cluster it shuts down?

It can, but where you put it matters. The project usually runs as an unprivileged LXC on one of the protected Proxmox nodes, and its shutdown order leaves the appliance's own host for last.

In a Ceph cluster there's one more constraint: the PVE-UPS guest shouldn't depend on the Ceph storage it's shutting down. The current project documentation says the installer rejects a Ceph-backed root filesystem unless the administrator overrides the check. That's sensible: if the appliance's disk lives on Ceph and Ceph drops below min_size mid-shutdown, the appliance running the shutdown can freeze before it finishes. So in a Ceph HCI environment, put the PVE-UPS guest on local storage.

What should you test before trusting any UPS automation?

Don't wait for a real power outage. Start in dry-run mode and prove every integration path:

• Confirm the UPS readings are correct.
• Confirm battery percentage and runtime thresholds are realistic under actual load.
• Test every PVE API token.
• Test the PBS path separately if you use it.
• Test every webhook.
• Run the self-test.
• Confirm the configured node names match the API.
• If you use HA, verify that the cluster reaches the expected disarmed state.
• If you use Ceph, test guest shutdown and maintenance preparation in a lab or maintenance window.

Then run a controlled power-loss exercise. Unplug the UPS from utility power with the servers still connected and watch the whole sequence. Seeing the first shutdown command go out tells you little, so verify that guests stop, Ceph stays responsive until the clients are gone, HA doesn't restart workloads, hosts shut down in the expected order, and the UPS still has enough runtime left for the slowest case.

A Proxmox Health Check can find many cluster-side weaknesses before that exercise, but you still have to test the shutdown path itself.

Reading the battery is the easy part of UPS automation

Compared with coordinating a clustered shutdown, UPS monitoring is easy. SNMP can report runtime and NUT can expose battery state. The hard part is deciding what happens next when the infrastructure has HA, distributed storage, backup servers, and several hosts with different dependencies.

PVE-UPS is getting interesting because its newer releases are starting to handle that orchestration. PBS support fixes a real API mismatch, multiple webhooks make failures easier to see, and better self-tests catch configuration errors sooner. The cluster-aware beta is the more ambitious piece.

A Proxmox cluster is more than three independent servers that happen to share a GUI. In a power failure, HA state, guest state, Ceph health, storage availability, and shutdown order all affect each other. If the sequence is wrong, the cluster can end up fighting its own shutdown. If it's right, the UPS does what you bought it for and gives everything enough time to stop cleanly.