ZFS Configuration Guide for Optimal Performance
ZFS (Zettabyte File System) combines the roles of a traditional filesystem and a volume manager. This guide covers how to install, configure, tune, and maintain ZFS for good performance in your homelab.
Why ZFS Hates Hardware RAID
What makes ZFS different?
Built-in checksumming detects and corrects silent data corruption, and snapshots give you instant, space-efficient point-in-time copies. RAID-Z is software RAID with better performance than traditional hardware RAID. ZFS also offers transparent compression to save space, deduplication to eliminate duplicate data blocks, and the ARC cache, an intelligent caching system for better performance.
System requirements
Minimum requirements
At minimum you need 8GB of RAM (16GB+ recommended), any 64-bit processor, and at least 2 drives for redundancy.
Recommended configuration
Plan for 1GB of RAM per TB of storage, or 5GB per TB if you use deduplication. A multi-core processor helps performance, and the disks should be enterprise-grade drives with power loss protection.
Installing ZFS
Ubuntu/Debian
sudo apt update
sudo apt install zfsutils-linux
Proxmox
ZFS is included by default on Proxmox, so you can check the version right away:
zfs version
Creating ZFS pools
Single disk (no redundancy)
A single-disk pool has no redundancy, so it is not recommended for production data.
sudo zpool create tank /dev/sdb
Mirror (RAID-1)
A mirror uses two disks, and each holds a complete copy of the data:
sudo zpool create tank mirror /dev/sdb /dev/sdc
RAID-Z1 (similar to RAID-5)
RAID-Z1 needs a minimum of 3 disks and survives 1 disk failure:
sudo zpool create tank raidz1 /dev/sdb /dev/sdc /dev/sdd
RAID-Z2 (similar to RAID-6)
RAID-Z2 needs a minimum of 4 disks and survives 2 disk failures:
sudo zpool create tank raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sde
RAID-Z3
RAID-Z3 needs a minimum of 5 disks and survives 3 disk failures:
sudo zpool create tank raidz3 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
ZFS pool recommendations
Home and small office
With 3 to 4 drives, use RAID-Z1. With 5 to 8 drives, use RAID-Z2. With 9 or more drives, consider RAID-Z3 or multiple RAID-Z2 vdevs.
Enterprise and critical data
Always use RAID-Z2 or RAID-Z3, add hot spares, and use enterprise drives.
Optimizing ZFS performance
ARC cache tuning
To limit the ARC cache size, set the maximum and minimum in /etc/modprobe.d/zfs.conf:
# Limit ARC to 8GB
options zfs zfs_arc_max=8589934592
# Minimum ARC size
options zfs zfs_arc_min=4294967296
Then rebuild the initramfs and reboot to apply the changes:
sudo update-initramfs -u
sudo reboot
Enable compression
LZ4 compression is fast and saves space. Enable it on the pool with:
sudo zfs set compression=lz4 tank
Configure record size
For databases (small random I/O), use a small record size:
sudo zfs set recordsize=16K tank/database
For media files (large sequential I/O), use a large one:
sudo zfs set recordsize=1M tank/media
Disable access time updates
Turning off access time updates improves performance. Set it on the pool with:
sudo zfs set atime=off tank
Enable ZFS auto-trim
If the pool is built on SSDs, enable auto-trim on it:
sudo zpool set autotrim=on tank
Working with datasets
Create datasets
sudo zfs create tank/documents
sudo zfs create tank/photos
sudo zfs create tank/videos
Set quotas
sudo zfs set quota=100G tank/documents
Set reservations
sudo zfs set reservation=50G tank/database
List datasets
zfs list
Snapshots
Create a snapshot
sudo zfs snapshot tank/documents@backup-2024-01-15
List snapshots
zfs list -t snapshot
Roll back to a snapshot
sudo zfs rollback tank/documents@backup-2024-01-15
Delete a snapshot
sudo zfs destroy tank/documents@backup-2024-01-15
Automated snapshots
Install Sanoid for automated snapshot management of your datasets:
sudo apt install sanoid
Then define your datasets and a retention template in /etc/sanoid/sanoid.conf:
[tank/documents]
use_template = production
recursive = yes
[template_production]
frequently = 0
hourly = 24
daily = 7
monthly = 12
yearly = 0
autosnap = yes
autoprune = yes
Send and receive (replication)
Send a snapshot to a file
sudo zfs send tank/documents@backup | gzip > backup.gz
Receive a snapshot from a file
gunzip -c backup.gz | sudo zfs receive tank/restore
Incremental replication
Start with an initial full send of the dataset to the backup server:
sudo zfs send tank/documents@initial | ssh user@backup-server \
sudo zfs receive backup/documents
Subsequent incremental sends transfer only the changes since the earlier snapshot:
sudo zfs send -i @initial tank/documents@today | ssh user@backup-server \
sudo zfs receive backup/documents
Monitoring ZFS
Pool status
sudo zpool status
Pool I/O statistics
sudo zpool iostat -v 2
ARC statistics
cat /proc/spl/kstat/zfs/arcstats
You can also use arc_summary for a more readable report of the same statistics:
sudo arc_summary
Dataset usage
zfs list -o name,used,avail,refer,mountpoint
Check for errors
sudo zpool status -v
Maintenance
Scrub the pool
Regular scrubs verify data integrity. Start one on the pool with:
sudo zpool scrub tank
While the scrub runs, you can check its progress with the status command:
sudo zpool status
To schedule monthly scrubs automatically, add this line to your crontab:
# Run on the first Sunday of each month at 2 AM
0 2 1-7 * * [ "$(date +\%u)" = "7" ] && /sbin/zpool scrub tank
Replace a failed drive
sudo zpool replace tank /dev/sdb /dev/sdf
After replacing the drive, monitor the rebuild until it finishes:
watch -n 5 zpool status
Add a hot spare
sudo zpool add tank spare /dev/sdg
Troubleshooting
Pool won't import
If a pool won't import normally, you can force the import:
sudo zpool import -f tank
Degraded pool
For a degraded pool, start by checking the detailed status:
sudo zpool status -v
If the errors were transient, clear them from the pool status:
sudo zpool clear tank
Out of space
When a pool runs out of space, check usage per dataset and snapshot:
zfs list -o name,used,avail,refer
zfs list -t snapshot
Then delete old snapshots that you no longer need to free space:
sudo zfs destroy tank/documents@old-snapshot
Performance issues
For performance issues, first check the ARC hit rate in the summary:
sudo arc_summary | grep "Hit Rate"
Then check how fragmented the pool is with the list command:
sudo zpool list -o name,fragmentation
Best practices
Do's
- Use ECC RAM for data integrity.
- Run regular scrubs (monthly).
- Enable compression (lz4).
- Take frequent snapshots.
- Test your backups.
- Monitor pool health.
- Use whole disks instead of partitions.
Don'ts
- Never exceed 80% pool capacity.
- Don't use RAID-Z with different sized drives.
- Avoid deduplication unless you have 5GB RAM per TB.
- Don't expand RAID-Z vdevs (not supported).
- Never use consumer drives without testing.
- Don't ignore scrub errors.
Advanced features
Encryption
To create an encrypted dataset protected by a passphrase, run:
sudo zfs create -o encryption=aes-256-gcm \
-o keyformat=passphrase tank/secure
Deduplication
Deduplication requires significant RAM. Enable it on a dataset with:
sudo zfs set dedup=on tank/vms
NFS shares
sudo zfs set sharenfs=on tank/share
sudo zfs set sharenfs="rw=@192.168.1.0/24" tank/share
SMB shares
sudo zfs set sharesmb=on tank/share
Performance benchmarks
Test sequential write speed by writing a large test file with dd:
dd if=/dev/zero of=/tank/testfile bs=1M count=10000
Test sequential read speed by reading the same test file back:
dd if=/tank/testfile of=/dev/null bs=1M
For more comprehensive testing, install fio and run a sequential write job:
sudo apt install fio
fio --name=seqwrite --rw=write --bs=1M --size=10G \
--filename=/tank/testfile
Conclusion
ZFS brings enterprise-grade storage features to your homelab, and with proper configuration and maintenance it delivers strong data integrity and performance. Use RAID-Z2 for important data, enable LZ4 compression, schedule regular scrubs, take frequent snapshots, monitor pool health, and never exceed 80% capacity.