22 Proxmox Nodes Hacked: What the Recovery Missed

A 22-node Proxmox cluster was reportedly compromised after an end-of-life Proxmox VE 7.4 environment stayed reachable from the internet on port 8006. The breach was bad enough, but I find what happened afterwards more instructive: rebuilding the hosts left some footholds in place, and the attackers came back once service was restored.

Everyone will take away "patch, and don't expose hypervisor management to the internet," and they should. The less obvious point is that recovering a clustered system after root compromise involves a lot more than restoring files from backup. A compromised kernel can lie to your recovery tools, and shared cluster state can keep attacker credentials alive through a node rebuild. Guest agents turn hypervisor root into guest root, and backups can save the platform while the control plane stays contaminated. In this incident all of those failures lined up.

What happened to the 22-node Proxmox cluster?

Netfront's published incident report says at least 33 external addresses got root across its 22-node Proxmox VE cluster between September 1 and September 2, 2026.

The affected zone was still on Proxmox VE 7.4, which reached end of life in July 2024. Port 8006 was reachable from the internet because customers used the Proxmox interface to manage their virtual machines.

The entry point was CVE-2023-54391. It affects libpve-access-control versions from 7.0-7 up to, but not including, 8.0.4. An attacker who can reach the login API can abuse the two-factor challenge flow to get a valid ticket for any enabled account that has no second factor configured.

The vulnerable code path had already disappeared from newer Proxmox packages in July 2023. Nobody recognized the security impact or documented it publicly until September 2026.

Netfront says its first anomalous request arrived about 25 minutes after the Proxmox advisory went public, and the first confirmed root shell showed up later that evening. From there things sped up, and the logs reportedly showed several independent sources sweeping the whole fleet. One address opened exactly one shell on each of the 22 nodes in just over a minute, and another racked up nearly 2,000 shell sessions across the cluster over a day.

The report says outright that this didn't look like one coordinated operator: once an easy authentication bypass is public, a vulnerable management plane can pull in several unrelated attackers at once.

Why did compromising one Proxmox node threaten all 22?

A Proxmox cluster shares sensitive control-plane state between nodes by design.

Proxmox uses pmxcfs, its distributed cluster file system, mounted at /etc/pve. It replicates cluster configuration so every node sees the same users, VM definitions, firewall configuration, certificates, HA settings, and other cluster-wide state. One of the protected files is /etc/pve/priv/authorized_keys, which Proxmox uses for SSH authentication between cluster members.

Netfront argues this made fleet-wide spread much easier once attackers had root on a host. Cluster members need a trusted way to talk to each other and share configuration, so that alone isn't a design flaw, but it changes how you respond. Root on one clustered hypervisor is a different situation from root on one standalone Linux server, because every trust relationship tying that node to the rest of the cluster is now in play.

If you're designing or reviewing a cluster, the Mr.PlanB Proxmox hub covers how networking, storage, HA, and cluster state interact, which is more useful than looking at each host on its own.

Why did the hosts look like failing hardware?

The incident report says an eBPF rootkit made the compromised nodes behave so badly that the team first went chasing hardware problems. Symptoms included load averages between 30 and 160 on otherwise idle systems, commands stopping unexpectedly, and tools such as top crashing.

Package verification didn't turn up the cause right away, because the legitimate binaries hadn't necessarily been replaced. The interception was happening inside the kernel. The rootkit reportedly hooked system calls to hide processes and files, block inspection, protect its own processes, and alter what user-space tools saw. Netfront tied parts of it to public eBPF proof-of-concept code and did not describe it as a sophisticated private rootkit.

I find that part uncomfortable. Attackers don't have to invent their stealth techniques anymore, because public research code makes it cheap to build something that's hard to investigate from inside the host. You don't need to start scanning every busy server for exotic rootkits, but you do need to stop trusting what a host tells you about itself once kernel-level compromise is plausible.

How did the attackers reach customer VMs?

Netfront says the attackers used the QEMU Guest Agent from the compromised Proxmox hosts, and I'd put that near the top of the list of details worth remembering.

The guest agent handles communication between host and guest. Proxmox uses it for guest shutdown, filesystem freeze and thaw, IP reporting, and command execution. After a host compromise, that convenience hands the attacker a lot of power.

According to the report, API logs showed commands running inside guest VMs through agent/exec, with output collected through agent/exec-status. No guest password was needed, because the hypervisor already had the authority to request those actions through the agent. Guests with the QEMU Guest Agent enabled were affected, and guests without it weren't reached this way.

I wouldn't read that as advice to disable the guest agent everywhere, since it does legitimate operational work. Still, hypervisor root sits in an extremely privileged position over every guest, and strong guest passwords don't form a real security boundary against a fully compromised hypervisor. Protect the management plane with that in mind.

Why did the first restore fail?

The team restored the filesystem while the compromised kernel was still running. Netfront says its first recovery attempt used:

rsync -aHAX --delete

to restore clean data over an infected host. The command completed, and the rootkit survived it.

The report explains why. The rootkit intercepted directory enumeration, and rsync --delete has to list the destination's files to work out what to remove. If the kernel hides a malicious file from that listing, rsync never knows the file is there, so it doesn't delete it. A perfectly clean backup can still produce a failed restore, because a compromised operating system is telling the recovery tool what exists on the destination.

Netfront changed the process: remove the rootkit, reboot, restore, reboot again, then validate. Hosts that wouldn't boot reliably were rebuilt from external media, where the compromised kernel wasn't running at all. After deep host compromise, that external path is the safer model. If you have reason to think the kernel has been subverted, be suspicious of any restore performed from inside it.

Why did attackers get back in after the rebuild?

The hosts were rebuilt, but the shared cluster control plane still held credentials the attackers had created. For me this is the most useful part of the story.

Netfront says it rotated passwords, SSH keys, and the contents of parts of /etc/pve/priv/. At first it didn't fully enumerate API tokens or check every account against an authoritative list of identities that should exist. That gap mattered because /etc/pve is the live Proxmox cluster file system, replicated across nodes, so rebuilding one host does nothing about malicious state the cluster keeps copying back.

The company's later audit reportedly found 42 API tokens associated with root@pam and ten rogue accounts named to look like legitimate ones. On September 7, after the platform was back in service, credentials that had survived the rebuild were used again.

A clean node can rejoin a dirty control plane, so when a clustered platform is compromised, recovery needs its own audit of shared configuration, credentials, tokens, ACLs, automation accounts, certificates, SSH trust, and anything else replicated outside the host filesystem.

A Proxmox Health Check can catch ordinary configuration problems, but a post-compromise review needs a far more aggressive identity and trust audit.

What did the backup design get right?

Of everything in the design, the separation between production and backup infrastructure held up best.

Netfront says its backup systems pulled data from the production hosts, so production never pushed into or administered the backup destination. The affected zone therefore had no credentials that could delete its remote recovery copies. The company also kept cross-zone copies in separate buildings.

That matches Proxmox Backup Server guidance. Proxmox recommends off-site copies and restrictive backup permissions, specifically warns against giving backup clients delete privileges, and recommends periodic restore tests instead of assuming successful backup jobs prove you can recover.

Replication is not automatically a backup. If a compromised production system can authenticate to the copy, change it, and delete it, the attacker may be able to wipe out production and recovery data together. A backup worth having sits behind a trust boundary production can't casually cross, so recovery, while still hard work, starts from something you can trust.

Was the real mistake running Proxmox VE 7.4?

It was one of several mistakes. The community reaction was unusually consistent on two points: the environment ran an EOL hypervisor release, and its management interface was publicly reachable. Each raises the risk, and together they multiply it.

People in the discussion also raised a fair point about why production teams hesitate on major hypervisor upgrades. Infrastructure that's stable, well understood, and serving customers can't be upgraded casually. Upgrading immediately without testing is risky too, so you need a patching process that lets you test major upgrades before the current version turns into an unsupported emergency.

A lab or representative mirror environment, documented rollback procedures, maintenance windows, hardware compatibility checks, guest testing, and staged node upgrades all cost time. So does rebuilding 22 compromised hypervisors. Skipping the upgrade process means paying that cost later, on a schedule the organization no longer controls.

Should Proxmox port 8006 ever be internet-facing?

Avoid direct exposure unless you have an unusually strong reason and a compensating security architecture.

The Proxmox interface is the management plane for the hypervisor, and nothing about it is low privilege. That's why the sharpest criticism in the discussion went to exposing port 8006 directly to untrusted networks, and it was not aimed at Proxmox itself. Better options include a management VLAN, VPN, dedicated jump host, identity-aware proxy, tightly restricted firewall rules, and MFA.

Customer self-service makes this harder, because external users may legitimately need some management functions. They still don't need direct network access to the entire hypervisor control plane. A separate customer portal using narrowly scoped Proxmox API permissions is one option, and an access proxy with strong identity controls is another.

Whichever you pick, assume the authentication code can have bugs, and design so that a single authentication bypass isn't enough to get root on every hypervisor.

What should a Proxmox recovery checklist include after root compromise?

The incident points to a checklist much broader than "restore from backup":

• Rebuild from trusted media when kernel compromise is possible.
• Treat every node in the trust domain as suspect.
• Audit /etc/pve as a separate shared control plane.
• Enumerate every user and every API token.
• Reconcile ACLs against a known-good design.
• Rotate passwords, SSH keys, certificates, and service credentials.
• Review guest-agent activity, and inspect guest systems if the compromised hypervisor could execute inside them.
• Verify backups from an environment the compromised hosts couldn't alter.
• Forward logs off-cluster so attackers can't erase the only useful history.
• Test the management interface from the public internet and confirm it isn't reachable unless it's intentionally protected.

Above all, don't limit the search to the persistence mechanism you've already seen. Netfront's report describes dozens of planted tokens and accounts, many of them never used, and a dormant backdoor is still a backdoor.

An impressive rebuild that should not have been needed

The recovery work in this incident has real engineering value. The investigation traced strange host behavior to an eBPF rootkit, and the team found a failure mode in in-place rsync recovery. They also recovered hosts that wouldn't boot, kept usable copies thanks to the cross-zone backup architecture, and later reconstructed attacker activity from logs in considerable detail.

Still, heroic recovery is a poor security strategy. Twenty-two EOL hypervisors with a public management plane let one authentication bypass become a fleet-wide incident, and restoring the hosts without fully auditing shared cluster credentials opened a second door.

The better setup is boring: supported software, restricted management access, MFA, independent backups, off-cluster logging, tested upgrades, a complete identity inventory, and recovery procedures that assume a compromised host may lie. None of that makes a good war story, and that's fine.