OpenZFS Offline Dedup: Is ZFS Dedup Finally Practical?

OpenZFS just gained a much more practical way to deduplicate existing files on Linux: upstream master now implements FIDEDUPERANGE using ZFS block cloning. That means tools such as duperemove and bees can find identical file ranges after the data is written and ask ZFS to share those blocks without paying the traditional memory cost of keeping every candidate in the ZFS deduplication table.

I checked the actual OpenZFS commit instead of relying on the forum headline. The change was committed on August 20, 2026 and is newer than stable OpenZFS 2.4.2, which was released in May. For now it is an upstream feature to watch and test, and I would not assume it is already available on a production TrueNAS, Proxmox, Debian, or Ubuntu system.

What exactly changed in OpenZFS deduplication?

OpenZFS now implements the Linux FIDEDUPERANGE interface by comparing file ranges and using ZFS block cloning when the ranges are byte-for-byte identical. Before this change, ZFS returned an unsupported operation response for this interface.

FIDEDUPERANGE is a Linux filesystem interface that lets a userspace deduplication tool say, in effect, "these two ranges look identical, can the filesystem verify that and make them share storage?" The filesystem remains responsible for verifying that the data matches before changing storage references.

The OpenZFS implementation does that verification inside ZFS. Where block pointers and strong checksums are enough to prove equality, it can avoid unnecessary reads. Where they are not enough, it reads and compares the data. If the ranges match, it uses the existing block cloning machinery so the two logical ranges reference shared physical blocks.

The result differs from an ordinary copy: the files remain separate logical files, and if one later changes, copy on write behavior preserves the other.

This is a natural extension of ZFS block cloning and a useful topic for anyone following ZFS and storage design, because it changes when you can decide to deduplicate data.

How is offline dedup different from normal ZFS dedup?

Offline dedup happens after data has already been written, while traditional ZFS dedup participates in the write path and tracks dedup candidates in a deduplication table. That timing difference changes the resource tradeoff.

Traditional inline ZFS dedup can save space immediately when matching blocks arrive, but it has a reputation for expensive metadata. The deduplication table, usually called the DDT, must track block fingerprints and references. Large DDTs can consume significant memory and create performance pressure when they do not fit comfortably in RAM and cache.

The new FIDEDUPERANGE path does not need to populate the traditional DDT for every deduplicated block. The upstream commit explicitly describes the goal as sharing matching ranges through block cloning without the memory cost of the dedup table.

Offline dedup shifts cost into scanning. A tool such as duperemove or bees has to identify candidate duplicates, then ask the filesystem to confirm and share them. That consumes CPU, metadata IO, and read bandwidth at the time you run the scan rather than adding the same style of global lookup cost to every incoming write.

Neither model is better in every case. If your workload continuously writes duplicate backup blocks and immediate space reduction is essential, inline dedup can still make sense. If you have a large file tree where duplicates accumulate over time and you can schedule scanning during quieter periods, offline dedup is much easier to reason about.

Does this mean ZFS dedup no longer needs lots of RAM?

This new offline dedup path avoids the traditional DDT memory cost for the ranges it deduplicates, but it does not make all ZFS deduplication memory free. If you enable normal dedup=on, you are still using the traditional or Fast Dedup machinery and its metadata structures.

That distinction matters more than anything else here, and "OpenZFS fixed dedup RAM usage" would be a misleading headline. OpenZFS added another path to space sharing that does not require the same always-on DDT design.

The broader Fast Dedup project is also improving the traditional system. OpenZFS has added features such as DDT quotas, prefetching, a log design, and pruning work so inline dedup can be bounded and managed more predictably. Those improvements and offline dedup solve related problems from different directions.

I would think of the choices like this:

• Compression remains the default space saving feature I would enable almost everywhere.
• Block cloning saves space when software intentionally creates reflink style copies.
• Offline dedup finds duplicate data later and converts identical ranges into shared blocks.
• Inline dedup tries to detect duplicate blocks as they are written.

Once you separate those four mechanisms, the ZFS design becomes much easier to discuss.

Can I use OpenZFS offline dedup today?

You should not assume your current stable package already includes it. The relevant commit landed in upstream OpenZFS master on August 20, 2026, while OpenZFS 2.4.2 was released on May 12, 2026.

As of August 21, that means a normal system running the stable 2.4.2 release should be assumed not to have this commit unless its vendor has backported it. Distribution packaging can also lag upstream, and appliance products often qualify filesystem changes on their own schedules.

Before testing, check the exact OpenZFS build provided by your operating system or appliance and read its release notes. On a Proxmox host, that means the Proxmox kernel and ZFS packages matter more than the existence of a GitHub commit. On TrueNAS, the TrueNAS release train matters. On Debian or Ubuntu, the distribution package version matters.

If you want to experiment with upstream code, use disposable data first. Filesystem features deserve a higher bar than application features because a crash can affect an entire pool instead of one process.

Anyone running ZFS under virtualization should also keep the storage change separate from hypervisor upgrades. If you are already planning a Proxmox VE 8 to 9 upgrade, I would not introduce experimental OpenZFS master code in the same window.

How does FIDEDUPERANGE stay safe if a file changes during dedup?

The OpenZFS implementation compares and clones ranges under locking so the bytes that are shared are the bytes that were actually compared. This matters because an offline tool can identify candidates in userspace, but only the filesystem can safely decide whether the on-disk ranges are still identical at the moment of deduplication.

The commit also handles ranges that are already shared. In that case, OpenZFS can report the range as deduplicated without needlessly rewriting block references.

I also like that the dedup operation does not change file content, so it does not need to behave like a normal write, and the implementation avoids treating it as content modification just because storage references become shared.

That is why filesystem native support matters. A userspace tool should identify candidates and ask the filesystem to perform the safe reference operation instead of faking deduplication by rewriting files behind applications.

Which workloads could benefit most from offline ZFS dedup?

Offline ZFS dedup is most attractive where duplicate data already exists in ordinary files and scanning can be scheduled outside the critical write path. Backup repositories, repeated VM image exports, ISO collections, user home directories, software build trees, and content archives are obvious candidates to test.

The actual savings will depend on duplication at the block and range level, file block size, compression, encryption behavior, and how the scanner finds candidates. A directory full of files with similar names is not necessarily deduplicable. A directory full of byte-identical VM templates probably is.

Backups are interesting because modern backup applications often perform their own chunking, compression, and deduplication. Running filesystem dedup below an already deduplicated repository may save little while adding scan IO. The useful test is how many physical bytes it saves on this repository after accounting for scan cost.

I would measure four things in a lab: bytes saved, scan duration, read IO generated, and the effect on later writes to deduplicated files. If the tool saves 3 percent of capacity while reading 40 TB every night, that is probably a bad trade. If it saves 35 percent of a mostly static archive during a weekly low priority scan, that is much more interesting.

Is this finally the ZFS dedup feature most users should use?

For many users who have avoided dedup=on because they did not want a large always-on DDT, offline dedup is the first ZFS dedup model I would seriously consider testing. It matches the way administrators already think about maintenance jobs: find waste, reclaim it, then return the system to normal work.

I would still keep compression as the default and leave inline dedup off unless the workload proves it deserves the complexity. The new offline path is attractive precisely because it gives you another option without forcing the write path to solve deduplication for every block forever.

Availability is what holds it back today. The code is merged upstream, but stable OpenZFS 2.4.2 predates it. I would watch for the first stable release or vendor build that includes the commit, test duperemove or bees against a copy of real data, and make the decision from measured savings.

At that point ZFS dedup becomes more interesting to ordinary storage admins, because the question changes from "can I afford to turn dedup on for this entire dataset?" to "is this duplicate data worth cleaning up after the fact?"