OpenZFS Offline Dedup: Is ZFS Dedup Finally Practical?

OpenZFS just gained a much more practical way to deduplicate existing files on Linux: upstream master now implements FIDEDUPERANGE using ZFS block cloning. That means tools such as duperemove and bees can find identical file ranges after the data is written and ask ZFS to share those blocks without paying the traditional memory cost of keeping every candidate in the ZFS deduplication table.

I checked the actual OpenZFS commit rather than relying on the forum headline. The change was committed on August 20, 2026 and is newer than stable OpenZFS 2.4.2, which was released in May, so this is an upstream feature to watch and test, not something I would assume is already available on a production TrueNAS, Proxmox, Debian, or Ubuntu system.

What exactly changed in OpenZFS deduplication?

OpenZFS now implements the Linux FIDEDUPERANGE interface by comparing file ranges and using ZFS block cloning when the ranges are byte-for-byte identical. Before this change, ZFS returned an unsupported operation response for this interface.

FIDEDUPERANGE is a Linux filesystem interface that lets a userspace deduplication tool say, in effect, “these two ranges look identical, can the filesystem verify that and make them share storage?” The filesystem remains responsible for verifying that the data matches before changing storage references.

The OpenZFS implementation does that verification inside ZFS. Where block pointers and strong checksums are enough to prove equality, it can avoid unnecessary reads. Where they are not enough, it reads and compares the data. If the ranges match, it uses the existing block cloning machinery so the two logical ranges reference shared physical blocks.

That is different from making an ordinary copy. The files remain separate logical files. If one later changes, copy on write behavior preserves the other.

This is a natural extension of ZFS block cloning and a useful topic for anyone following ZFS and storage design, because it changes when you can decide to deduplicate data.

How is offline dedup different from normal ZFS dedup?

Offline dedup happens after data has already been written, while traditional ZFS dedup participates in the write path and tracks dedup candidates in a deduplication table. That timing difference changes the resource tradeoff.

Traditional inline ZFS dedup can save space immediately when matching blocks arrive, but it has a reputation for expensive metadata. The deduplication table, usually called the DDT, must track block fingerprints and references. Large DDTs can consume significant memory and create performance pressure when they do not fit comfortably in RAM and cache.

The new FIDEDUPERANGE path does not need to populate the traditional DDT for every deduplicated block. The upstream commit explicitly describes the goal as sharing matching ranges through block cloning without the memory cost of the dedup table.

Offline dedup shifts cost into scanning. A tool such as duperemove or bees has to identify candidate duplicates, then ask the filesystem to confirm and share them. That consumes CPU, metadata IO, and read bandwidth at the time you run the scan rather than adding the same style of global lookup cost to every incoming write.

Neither model is universally better. If your workload continuously writes duplicate backup blocks and immediate space reduction is essential, inline dedup can still make sense. If you have a large file tree where duplicates accumulate over time and you can schedule scanning during quieter periods, offline dedup is much easier to reason about.

Does this mean ZFS dedup no longer needs lots of RAM?

This new offline dedup path avoids the traditional DDT memory cost for the ranges it deduplicates, but it does not make all ZFS deduplication memory free. If you enable normal dedup=on, you are still using the traditional or Fast Dedup machinery and its metadata structures.

That distinction is the most important part of the story. “OpenZFS fixed dedup RAM usage” would be a misleading headline. What OpenZFS added is another path to space sharing that does not require the same always-on DDT design.

The broader Fast Dedup project is also improving the traditional system. OpenZFS has added features such as DDT quotas, prefetching, a log design, and pruning work so inline dedup can be bounded and managed more predictably. Those improvements and offline dedup solve related problems from different directions.

I would think of the choices like this:

• Compression remains the default space saving feature I would enable almost everywhere.
• Block cloning saves space when software intentionally creates reflink style copies.
• Offline dedup finds duplicate data later and converts identical ranges into shared blocks.
• Inline dedup tries to detect duplicate blocks as they are written.

Once you separate those four mechanisms, the ZFS design becomes much easier to discuss.

Can I use OpenZFS offline dedup today?

Not safely from the assumption that your current stable package already includes it. The relevant commit landed in upstream OpenZFS master on August 20, 2026, while OpenZFS 2.4.2 was released on May 12, 2026.

As of August 21, that means a normal system running the stable 2.4.2 release should be assumed not to have this commit unless its vendor has backported it. Distribution packaging can also lag upstream, and appliance products often qualify filesystem changes on their own schedules.

Before testing, check the exact OpenZFS build provided by your operating system or appliance and read its release notes. On a Proxmox host, that means the Proxmox kernel and ZFS packages matter more than the existence of a GitHub commit. On TrueNAS, the TrueNAS release train matters. On Debian or Ubuntu, the distribution package version matters.

If you want to experiment with upstream code, use disposable data first. Filesystem features deserve a higher bar than application features because a crash can affect an entire pool, not just one process.

Anyone running ZFS under virtualization should also keep the storage change separate from hypervisor upgrades. If you are already planning a Proxmox VE 8 to 9 upgrade, I would not introduce experimental OpenZFS master code in the same window.

How does FIDEDUPERANGE stay safe if a file changes during dedup?

The OpenZFS implementation compares and clones ranges under locking so the bytes that are shared are the bytes that were actually compared. That is essential because an offline tool can identify candidates in userspace, but only the filesystem can safely decide whether the on-disk ranges are still identical at the moment of deduplication.

The commit also handles ranges that are already shared. In that case, OpenZFS can report the range as deduplicated without needlessly rewriting block references.

Another good design choice is that the dedup operation does not change file content, so it does not need to behave like a normal write. The implementation avoids treating the operation as content modification simply because storage references become shared.

This is why filesystem native support matters. A userspace tool should not try to fake deduplication by rewriting files behind applications. It should identify candidates and ask the filesystem to perform the safe reference operation.

Which workloads could benefit most from offline ZFS dedup?

Offline ZFS dedup is most attractive where duplicate data already exists in ordinary files and scanning can be scheduled outside the critical write path. Backup repositories, repeated VM image exports, ISO collections, user home directories, software build trees, and content archives are obvious candidates to test.

The actual savings will depend on duplication at the block and range level, file block size, compression, encryption behavior, and how the scanner finds candidates. A directory full of files with similar names is not necessarily deduplicable. A directory full of byte-identical VM templates probably is.

Backups are interesting because modern backup applications often perform their own chunking, compression, and deduplication. Running filesystem dedup below an already deduplicated repository may save little while adding scan IO. The right test is not “does offline dedup work?” but “how many physical bytes does it save on this repository after accounting for scan cost?”

I would measure four things in a lab: bytes saved, scan duration, read IO generated, and the effect on later writes to deduplicated files. If the tool saves 3 percent of capacity while reading 40 TB every night, that is probably a bad trade. If it saves 35 percent of a mostly static archive during a weekly low priority scan, that is much more interesting.

Is this finally the ZFS dedup feature most users should use?

For many users who have avoided dedup=on because they did not want a large always-on DDT, offline dedup is the first ZFS dedup model I would seriously consider testing. It matches the way administrators already think about maintenance jobs: find waste, reclaim it, then return the system to normal work.

I would still keep compression as the default and leave inline dedup off unless the workload proves it deserves the complexity. The new offline path is attractive precisely because it gives you another option without forcing the write path to solve deduplication for every block forever.

The limiting factor today is availability. The code is merged upstream, but stable OpenZFS 2.4.2 predates it. I would watch for the first stable release or vendor build that includes the commit, test duperemove or bees against a copy of real data, and make the decision from measured savings.

That is the point where ZFS dedup becomes more interesting to ordinary storage admins. The question changes from “can I afford to turn dedup on for this entire dataset?” to “is this duplicate data worth cleaning up after the fact?”