Difference between revisions of "Feature Flags"

From OpenZFS
Jump to navigation Jump to search
(→‎Feature Flags Implementation: Add FreeBSD -CURRENT, and fix link to mm@'s site)
(→‎Feature flags implementation: Colored no red, centered text, added a few feature flags and newer ZoL versions)
(8 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Originally the ZFS on-disk format was versioned with a single number that was increased whenever a new on-disk format change was introduced. This worked well when a single entity controlled the development of ZFS; however, in the more distributed development model of OpenZFS a single version number is not ideal. Every OpenZFS implementation would need to agree on every change to the on-disk format.
ZFS on-disk formats were originally versioned with a single number, which increased whenever the format changed. The numbered approach was suitable when development of ZFS was driven by a single organisation.  


One of the first OpenZFS projects was a new versioning system called "feature flags" that tags on-disk format changes with unique names. The system supports both completely independent format changes, as well as format changes that depend on each other. A pool's on-disk format is portable between OpenZFS implementations as long as all of the feature flags in use by the pool are supported by both implementations.
For distributed development of OpenZFS, version numbering was unsuitable. Any change to the number would have required agreement, across all implementations, of each change to the on-disk format.  


For more details see these [http://blog.delphix.com/csiden/files/2012/01/ZFS_Feature_Flags.pdf slides (Jan 2012)] and [http://illumos.org/man/5/zpool-features <tt>zpool-features(5)</tt>] (illumos) or [http://www.freebsd.org/cgi/man.cgi?query=zpool-features&sektion=7&manpath=FreeBSD+11-current <tt>zpool-features(7)</tt>] (FreeBSD).
OpenZFS feature flags – an alternative to traditional version numbering – allow '''a uniquely named ''pool property'' for each change to the on-disk format'''. This approach supports:  


For greatest compatibility, features that are exclusive (when enabled) should be periodically ported to all other distributions.
* format changes that are independent
* format changes that depend on each other.  


== Feature Flags Implementation ==
== Compatibility ==
The current state of OpenZFS feature flags implementation is in the following table.
 
Where all ''features'' that are used by a pool are supported by multiple implementations of OpenZFS, the on-disk format is portable across those implementations.
 
Features that are exclusive when enabled should be periodically ported to all distributions.
 
== Reference materials ==
 
[http://web.archive.org/web/20160419064650/http://blog.delphix.com/csiden/files/2012/01/ZFS_Feature_Flags.pdf ''ZFS Feature Flags''] (Christopher Siden, 2012-01, in the Internet Archive Wayback Machine) in particular: "… Legacy version numbers still exist for pool versions 1-28 …".
 
[http://illumos.org/man/5/zpool-features <tt>zpool-features</tt>](5) – illumos
 
[https://www.freebsd.org/cgi/man.cgi?query=zpool-features&sektion=7&manpath=FreeBSD <tt>zpool-features</tt>](7) – FreeBSD
 
== Feature flags implementation ==


{| class="wikitable"
{| class="wikitable"
!Feature flag
!Feature flag
!colspan=4|FreeBSD
!colspan=4|FreeBSD
!ZFS on Linux
!colspan=3|ZFS on Linux
!OpenZFS OSX
!OpenZFS OSX
!OmniOS
!OmniOS
|-
|-
|
|
| 8.4
| 10.3
| 9.3
| 11.0
| 10.1
| stable/11
| r283009
| r313645
| 0.6.4
| 0.6.5
| 1.3.1
| 0.7
| git ([https://github.com/zfsonlinux/zfs/commit/94370f59 94370f59])
| 1.6.1
| r151014
| r151014
|-
|-
!async_destroy
!async_destroy
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!empty_bpobj
!empty_bpobj
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!lz4_compress
!lz4_compress
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!multi_vdev_crash_dump
!multi_vdev_crash_dump
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|no
|style="background-color:lightsalmon; text-align: center;"|no
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!spacemap_histogram
!spacemap_histogram
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!enabled_txg
!enabled_txg
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!hole_birth
!hole_birth
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!extensible_dataset
!extensible_dataset
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!embedded_data
!embedded_data
|no
|style="background-color:lightgreen; text-align: center;"|yes
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!bookmarks
!bookmarks
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!filesystem_limits
!filesystem_limits
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|no
|style="background-color:lightgreen; text-align: center;"|yes
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
|-
!large_blocks
!large_blocks
|no
|style="background-color:lightgreen; text-align: center;"|yes
|no
|style="background-color:lightgreen; text-align: center;"|yes
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|no
|style="background-color:lightgreen; text-align: center;"|yes
|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|-
!sha512
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|??
|-
!skein
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|??
|-
!edonr
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|??
|??
|-
!userobj_acccounting
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|??
|??
|style="background-color:lightgreen; text-align: center;"|yes
|style="background-color:lightgreen; text-align: center;"|yes
|??
|??
|-
!device_removal
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|??
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightgreen; text-align: center;"|yes
|??
|??
|-
!zpool_checkpoint
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|??
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightgreen; text-align: center;"|yes
|??
|??
|-
!spacemap_v2
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|??
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightgreen; text-align: center;"|yes
|??
|??
|-
!encryption
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|??
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightgreen; text-align: center;"|yes
|??
|??
|-
!zstd
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|style="background-color:lightsalmon; text-align: center;"|no
|-
|}
|}


Line 139: Line 279:
* yes = implemented
* yes = implemented


Table source: [http://blog.vx.sk/archives/44-OpenZFS-Feature-Flags-Compatibility-Matrix.html ''OpenZFS Feature Flags Compatibility Matrix''] at blog.vx.sk
Source: [http://blog.vx.sk/archives/44-OpenZFS-Feature-Flags-Compatibility-Matrix.html ''OpenZFS Feature Flags Compatibility Matrix - VX weblog'']

Revision as of 19:21, 7 July 2018

ZFS on-disk formats were originally versioned with a single number, which increased whenever the format changed. The numbered approach was suitable when development of ZFS was driven by a single organisation.

For distributed development of OpenZFS, version numbering was unsuitable. Any change to the number would have required agreement, across all implementations, of each change to the on-disk format.

OpenZFS feature flags – an alternative to traditional version numbering – allow a uniquely named pool property for each change to the on-disk format. This approach supports:

  • format changes that are independent
  • format changes that depend on each other.

Compatibility

Where all features that are used by a pool are supported by multiple implementations of OpenZFS, the on-disk format is portable across those implementations.

Features that are exclusive when enabled should be periodically ported to all distributions.

Reference materials

ZFS Feature Flags (Christopher Siden, 2012-01, in the Internet Archive Wayback Machine) in particular: "… Legacy version numbers still exist for pool versions 1-28 …".

zpool-features(5) – illumos

zpool-features(7) – FreeBSD

Feature flags implementation

Feature flag FreeBSD ZFS on Linux OpenZFS OSX OmniOS
10.3 11.0 stable/11 r313645 0.6.5 0.7 git (94370f59) 1.6.1 r151014
async_destroy yes yes yes yes yes yes yes yes yes
empty_bpobj yes yes yes yes yes yes yes yes yes
lz4_compress yes yes yes yes yes yes yes yes yes
multi_vdev_crash_dump yes yes yes yes no yes yes yes yes
spacemap_histogram yes yes yes yes yes yes yes yes yes
enabled_txg yes yes yes yes yes yes yes yes yes
hole_birth yes yes yes yes yes yes yes yes yes
extensible_dataset yes yes yes yes yes yes yes yes yes
embedded_data yes yes yes yes yes yes yes yes yes
bookmarks yes yes yes yes yes yes yes yes yes
filesystem_limits yes yes yes yes yes yes yes yes yes
large_blocks yes yes yes yes yes yes yes yes yes
sha512 no yes yes yes no yes yes yes ??
skein no yes yes yes no yes yes yes ??
edonr no no no no no yes yes ?? ??
userobj_acccounting no no no ?? ?? yes yes ?? ??
device_removal no no no ?? no no yes ?? ??
zpool_checkpoint no no no ?? no no yes ?? ??
spacemap_v2 no no no ?? no no yes ?? ??
encryption no no no ?? no no yes ?? ??
zstd no no no no no no no no no

Table legend

  • no = not implemented
  • yes = implemented

Source: OpenZFS Feature Flags Compatibility Matrix - VX weblog