Which ZFS to use to after the death of OpenSolaris?

on

Update 2018-07-18

The landscape for ZFS (and copy-on-write filesystems in general) has changed a great deal since this was published over seven years ago in 2011. Though, not necessarily in the way anticipated back then. (Won't it be interesting to see what the next 7 years hold!)

Some of the options in the table below are dead projects. Others are still relevant but hold less prominence and/or promise as they did back then.

A quick overview of options as of 2018, in no particular order:

  • OpenZFS has been an ongoing, active umbrella project since 2013.
    • It provides a common ZFS codebase for ZFS on Linux, FreeBSD ZFS, OpenZFS on MacOS, and Illumoms ZFS.
    • On most or all of those platforms (notably Linux and various BSDs), it is widely regarded as stable, robust, and enterprise-ready. On all platforms, it operates in kernel space.
    • On Linux:
      • It is available as a Loadable Kernel Module (LKM).
      • ZFS-on-Linux has been included in Ubuntu's precompiled apt repository since version 16.04 LTS (April 2016). It was available before that via convenient PPT on 12.04 LTS (at some time after initial release), though not always smoothly. (By 14.04 LTS, it all worked fairly seamlessly and reliably.)
        • Since it is technically not distributed with Ubuntu, the problems of ZFS' GPL-incompatible CDDL license are skirted.
  • Btrfs has, arguably, disappointed at least on the enterprise front. While it has useful features that ZFS doesn't (such as the ability to shrink arrays and volumes), it still is regarded as too risky by many mission-critical enterprise data architects, and has yet to land as the long-term default filesystem for any major Linux distribution. Development is still active and ongoing.
  • ReFS: Microsoft released ReFS in 2012, as a hopeful enterprise (if not also consumer-level) replacement to NTFS. While that hasn't happened yet (as of writing), it includes similar but non-overlapping features compared to ZFS and Btrfs (such as copy-on-write design). Compared to ZFS and Btrfs, ReFS (as of writing) does not support CoW snapshots, full end-to-end checksumming, and on-the-fly compression. However development is ongoing. It may eventually support some or all of those differential features (and more), and may eventually replace NTFS. But not being open-source, it is unlikely to be available to non-Microsoft platforms.
  • Bcachefs is a promising filesystem in progress, based on the stable core of the cachefs project. The developers' goals are a high-performance filesystem that provides copy-on-write, "free" CoW snapshots, end-to-end checksumming, intelligent SSD-caching, on-the-fly compression and encryption, and ZFS/Btrfs-like scalability. The promise is a GPL-licensed filesystem that might offer the best features of ZFS and Btrfs. Time will tell if it matures.

Original 2011 content

Why ZFS?

You probably are not reading this to discover what ZFS is (short story: its a file system), so I'll skip that. You probably also understand the benefits, but I'll summarize them anyway, briefly:

  • Not available in any mainstream modern filesystem (as of writing):
    • End-to-end data integrity, through on-disk checksumming and automatic healing (if redundancy exists).
      • With exponentially growing disk capacities, "silent corruption" is approaching a mathematical certainty with current filesystems.
      • End-to-end data integrity is becoming a baseline requirement for any data you wish to remain valid over time, yet few filesystems address (as of writing).
      • Helps prevent or eliminate "bit rot" (a form of silent corruption).
      • Helps prevent or eliminate data corruption by any hardware component in the chain.
  • Still somewhat rare in modern filesystems (as of writing):
    • Copy-On-Write transactional model (for more reliable on-disk writes and provides foundation for other advanced features).
    • Double-parity RAID-Z2 (like RAID-6) and triple-parity RAID-Z3, to combat the growing near-mathematical likelihood of a second drive failing while resilvering from a first failed drive in RAID-5 configurations.
    • Elimination of the RAID-5 power-failure "Write Hole" for striped parity pools.
    • Software RAID implimentation (essentially required for end-to-end data integrity) with performance rivaling or exceeding modern hardware RAID hardware.
    • Can have redundancy for automatic bitrot healing even without mirrors or RAID-Zx, with the "copies" property.
    • Fine-grained snapshot capability.
    • Can stream snapshots to any format, medium, and/or destination.
    • Read/write snapshot clones.
    • All-in-one block device, array, and pool management—with only two basic commands. (The classic storage stack is flattened with redundancy between layers removed.)
    • Pools can be safely grown by replacing disks in an array with larger models, and/or by adding arrays new arrays.
      • Neither arrays nor pools can be shrunk (as of writing; usually not a concern in most use cases).
    • Advanced two-stage, SSD-aware read-caching model.
    • Transparent in-line block-level pool-wide deduplication.
    • Transparent in-line compression.
    • Transparent in-line encryption.

Which ZFS?

For those requiring open-source solutions (whether on principle, cost, or both), there used to be only one real answer to this question: OpenSolaris. Sun Microsystems made history when they open-sourced Solaris. While alternatives to ZFS on [Open]Solaris started springing up, OpenSolaris (where ZFS was born) was still the de-facto (and arguably best) way to get ZFS.

That story began to change when it seemed increasingly likely that Oracle would purchase Sun Microsystems, and [Open]Solaris along with it. The fear (not completely unfounded) was that Oracle would kill the "Open" in OpenSolaris, in spite of the fact that Sun's own CDDL license would make that difficult. The fruits of that purchase have been revealed: OpenSolaris has become Oracle Solaris Express 11. It both confirms and refutes the fears. Without going into the details of how and when Oracle plans to release the CDDL code to the community, suffice to say it is less than ideal, and not very "open" at all. (In addition, it seems commonly expected that entirely proprietary code will start to replace existing CDDL'ed code).

Integrating ZFS into an operating system is not just a technical challenge, it can also be a legal one. The original engineers of ZFS themselves at Sun allegedly wanted a license model that was explicitly incompatible with the common GPL license of the Linux kernel. Sun created the "CDDL" open-source license, based on the Mozilla license, which was already well-known to be incompatible with GPL.

However, that does not mean it is impossible to include ZFS in a non-Solaris kernel. For starters, the BSD license is not incompatible with CDDL. Furthermore, the incompatibility with GPL only applies to code included in the kernel. But it can be run in user-space (e.g. FUSE), or could be installed as a Loadable Kernel Module not distributed with the kernel itself.

Availability of ZFS and ZFS-like advanced file systems, as of 2011-02-09

Stable ? Platform based on ZFS ver ZPool ver Pros Cons Notes
yes Oracle Solaris 11 Express 2010.11 ON 151a 31 • Has the most recent implementation of ZFS (as of writing).
• Easiest to install and administer.
• GUI integration is more polished and useful than FreeBSD+Gnome, NexentaCP+Gnome, and OpenSolaris.
• Free (as in beer).
• /boot is on ZFS snapshot system, integrated with GUI tools.
 • Not free (as in speech).
• Free (as in beer) license is limited to developers.
• Limited driver support.
• Primitive GUI.
• Administrative commands are very different than BSD or Linux.
• Primitive package manager.
• Limited package repository.
yes Nexenta Core Platform v3.0.1 ON 134b 26 • Solid text-mode installer.
• Third party utilities available for ZFS web interface.
 • Has nothing to do with Ubuntu, in spite of product positioning.
• Gnome can be installed, but the result is very primitive and arguably not worth the effort.
• Limited number of ported GNU commands winds up confusing Solaris users/admins. (GNU binaries can be disabled.)
 • The Nexenta community ported the Debian apt-get package manager, and several Linux-style binary packages are available through it.
• In spite of some community confusion, native linux binaries will not run.
• See this post for clarification on Nexenta/Ubuntu.
• Plans to move base from ON 134b, to Illumos in future. (Which will make it less reliant on Oracle, and better able to permanently fork from Oracle Solaris.)
yes FreeBSD v8.1 (BSD) 14 • Most liberal/free licensing model.
• Solid implementation of ZFS (as old as OpenSolaris 2009.06 (as of writing), under active development).
 • Ancient installer.
• Manual tinkering required for bootable root on ZFS (single or mirrored).
• No deduplication with supplied ZFS version.
 • Very much an "old-school" POSIX OS.
• GUI-less installation by default.
• Able to acheive very small installation footprint.
• Manual tinkering of config files, init scripts, and daemons required for most configuration/administration.
yes Sun OpenSolaris 2009.06 ON 111b 14 • Arguably "The Last Free Solaris".
• Solid installer.
• GUI (Gnome) slightly more polished than FreeBSD+Gnome or NexentaCP+Gnome.
 • No longer maintained.
• Limited driver support.
• Administrative commands very different than BSD or Linux.
• Primitive package manager.
• Converting root pool to a mirror requires manual tinkering.
• Limited package repository.
• Cryptic package names.
• No deduplication.
 • If you are satisfied with ZPool v14 (same as latest stable FreeBSD v8.1), then it's hard to go wrong with this.
• Extended ACL model is completely broken and unusable for production (classic model is fine).
• Because ZFS kernel-level CIFS/SMB service relies on broken ACL model, you must use add-on Samba server (which is solid) instead, for Windows share server.
yes ZFS-FUSE for Linux, Mac OS X, and BSD FUSE cross-platform user-space filesystem API 23 • Solid (as of writing), in spite of getting off to a rough start.
• Perhaps inarguably the easiest to install and go (including installing a modern, user-friendly Linux distro).
• Able to acheive very small installation footprint with choice of a minimalist Linux distro.
 • Does not perform as well as in-kernel variants.
yes KQ Infotech ZFS Linux Loadable Kernel Module (Linux) 28 • Linux Loadable kernel module should in theory outperform ZFS-FUSE for Linux.
• Claims to be "GA-quality".
 • Targeted to the enterprise with paid support. License terms are not clear on their website, but would have to also be CDDL (free and open-source).
no Btrfs (Linux) n/a n/a • Covers many of the major benefits of ZFS (particularly COW, pooling, end-to-end checksumming, integrated management, writable snapshots).
• Pools can be grown and shrunk (ZFS can only grow).
• GPL license compatible with Linux kernel (vs GPL-incompatible CDDL license of ZFS).
• Already included in Linux kernel (for testing only).
 • Not yet enterprise-ready (as of writing).
 • Very different implementation approach to features that ZFS.
• Current build (as of writing) does not include deduplication, but it is planned in some form.
no Illumos ON 147 22 • The Illumos project aims to replace Oracle's closed-source drivers, encryption code, and other binaries with open-source, while maintaining binary compatibility. This should reduce the risk of another sudden "shutdown" of a Solaris-based project.
• The project plans to be the base platform for several future distributions (including OpenIndiana, Nexenta, Schillix, Belinix, etc.).
 • Project homepage does not show many updates or much recent activity (as of writing).
no Schillix Illumos 22 • A convenient way to get a binary dev build of Illumos, without compiling it yourself.
no OpenIndiana ON 147 28 • Aims to stay true to the OpenSolaris experience. • Plans to move base from ON 147, to Illumos in future.
no LLNL ZFS Linux Loadable Kernel Module (Linux) 28 • Linux Loadable kernel module should in theory outperform ZFS-FUSE for Linux. • Little available information (as of writing). • Lawrence Livermore National Laboratory port of ZFS under contract by the US Department of Energy. Like the KQ Infotech LKM, it is a Linux Loadable Kernel Module distributed separately, thus doesn't violate Sun/Oracle's GPL-incompatible CDDL license.
no Mac ZFS (Mac OS X) 2 8 • Old version of ZFS. • A Mac OS X (Darwin) Loadable Kernel Module.
? Milax ON 128a 20 • Small footprint, low overhead. • No deduplication in this version of ZFS. • Targeted as a minimalist installation.
? EON ZFS Storage ON 134 (?) 26 • Can install to and boot from thumbdrive. • Runs from RAM.
• Minimalist installation.