github.com/tompao/docker@v1.9.1/docs/userguide/storagedriver/zfs-driver.md (about)

     1  <!--[metadata]>
     2  +++
     3  title = "ZFS storage in practice"
     4  description = "Learn how to optimize your use of ZFS driver."
     5  keywords = ["container, storage, driver, ZFS "]
     6  [menu.main]
     7  parent = "mn_storage_docker"
     8  +++
     9  <![end-metadata]-->
    10  
    11  # Docker and ZFS in practice
    12  
    13  ZFS is a next generation filesystem that supports many advanced storage technologies such as volume management, snapshots, checksumming, compression and deduplication, replication and more.
    14  
    15  It was created by Sun Microsystems (now Oracle Corporation) and is open sourced under the CDDL license. Due to licensing incompatibilities between the CDDL and GPL, ZFS cannot be shipped as part of the mainline Linux kernel. However, the ZFS On Linux (ZoL) project provides an out-of-tree kernel module and userspace tools which can be installed separately.
    16  
    17  The ZFS on Linux (ZoL) port is healthy and maturing. However, at this point in time it is not recommended to use the `zfs` Docker storage driver for production use unless you have substantial experience with ZFS on Linux.
    18  
    19  > **Note:** There is also a FUSE implementation of ZFS on the Linux platform. This should work with Docker but is not recommended. The native ZFS driver (ZoL) is more tested, more performant, and is more widely used. The remainder of this document will relate to the native ZoL port.
    20  
    21  
    22  ## Image layering and sharing with ZFS
    23  
    24  The Docker `zfs` storage driver makes extensive use of three ZFS datasets:
    25  
    26  - filesystems
    27  - snapshots
    28  - clones
    29  
    30  ZFS filesystems are thinly provisioned and have space allocated to them from a ZFS pool (zpool) via allocate on demand operations. Snapshots and clones are space-efficient point-in-time copies of ZFS filesystems. Snapshots are read-only. Clones are read-write. Clones can only be created from snapshots. This simple relationship is shown in the diagram below.
    31  
    32  ![](images/zfs_clones.jpg)
    33  
    34  The solid line in the diagram shows the process flow for creating a clone. Step 1 creates a snapshot of the filesystem, and step two creates the clone from the snapshot. The dashed line shows the relationship between the clone and the filesystem, via the snapshot. All three ZFS datasets draw space form the same underlying zpool.
    35  
    36  On Docker hosts using the `zfs` storage driver, the base layer of an image is a ZFS filesystem. Each child layer is a ZFS clone based on a ZFS snapshot of the layer below it. A container is a ZFS clone based on a ZFS Snapshot of the top layer of the image it's created from. All ZFS datasets draw their space from a common zpool. The diagram below shows how this is put together with a running container based on a two-layer image.
    37  
    38  ![](images/zfs_zpool.jpg)
    39  
    40  The following process explains how images are layered and containers created. The process is based on the diagram above.
    41  
    42  1. The base layer of the image exists on the Docker host as a ZFS filesystem.
    43  
    44      This filesystem consumes space from the zpool used to create the Docker host's local storage area at `/var/lib/docker`.
    45  
    46  2. Additional image layers are clones of the dataset hosting the image layer directly below it.
    47  
    48      In the diagram, "Layer 1" is added by making a ZFS snapshot of the base layer and then creating a clone from that snapshot. The clone is writable and consumes space on-demand from the zpool. The snapshot is read-only, maintaining the base layer as an immutable object.
    49  
    50  3. When the container is launched, a read-write layer is added above the image.
    51  
    52      In the diagram above, the container's read-write layer is created by making a snapshot of the top layer of the image (Layer 1) and creating a clone from that snapshot.
    53  
    54      As changes are made to the container, space is allocated to it from the zpool via allocate-on-demand operations. By default, ZFS will allocate space in blocks of 128K.
    55  
    56  This process of creating child layers and containers from *read-only* snapshots allows images to be maintained as immutable objects.
    57  
    58  ## Container reads and writes with ZFS
    59  
    60  Container reads with the `zfs` storage driver are very simple. A newly launched container is based on a ZFS clone. This clone initially shares all of its data with the dataset it was created from. This means that read operations with the `zfs` storage driver are fast &ndash; even if the data being read was copied into the container yet. This sharing of data blocks is shown in the diagram below.
    61  
    62  ![](images/zpool_blocks.jpg)
    63  
    64  Writing new data to a container is accomplished via an allocate-on-demand operation. Every time a new area of the container needs writing to, a new block is allocated from the zpool. This means that containers consume additional space as new data is written to them. New space is allocated to the container (ZFS Clone) from the underlying zpool.
    65  
    66  Updating *existing data* in a container is accomplished by allocating new blocks to the containers clone and storing the changed data in those new blocks. The original are unchanged, allowing the underlying image dataset to remain immutable. This is the same as writing to a normal ZFS filesystem and is an implementation of copy-on-write semantics.
    67  
    68  ## Configure Docker with the ZFS storage driver
    69  
    70  The `zfs` storage driver is only supported on a Docker host where `/var/lib/docker` is mounted as a ZFS filesystem. This section shows you how to install and configure native ZFS on Linux (ZoL) on an Ubuntu 14.04 system.
    71  
    72  ### Prerequisites
    73  
    74  If you have already used the Docker daemon on your Docker host and have images you want to keep, `push` them Docker Hub or your private Docker Trusted Registry before attempting this procedure.
    75  
    76  Stop the Docker daemon. Then, ensure that you have a spare block device at `/dev/xvdb`. The device identifier may be be different in your environment and you should substitute your own values throughout the procedure.
    77  
    78  ### Install Zfs on Ubuntu 14.04 LTS
    79  
    80  1. If it is running, stop the Docker `daemon`.
    81  
    82  1. Install `the software-properties-common` package.
    83  
    84      This is required for the `add-apt-repository` command.
    85  
    86          $ sudo apt-get install software-properties-common
    87          Reading package lists... Done
    88          Building dependency tree
    89          <output truncated>
    90  
    91  2. Add the `zfs-native` package archive.
    92  
    93          $ sudo add-apt-repository ppa:zfs-native/stable
    94           The native ZFS filesystem for Linux. Install the ubuntu-zfs package.
    95          <output truncated>
    96          gpg: key F6B0FC61: public key "Launchpad PPA for Native ZFS for Linux" imported
    97          gpg: Total number processed: 1
    98          gpg:               imported: 1  (RSA: 1)
    99          OK
   100  
   101  3. Get the latest package lists for all registered repositories and package archives.
   102  
   103          $ sudo apt-get update
   104          Ign http://us-west-2.ec2.archive.ubuntu.com trusty InRelease
   105          Get:1 http://us-west-2.ec2.archive.ubuntu.com trusty-updates InRelease [64.4 kB]
   106          <output truncated>
   107          Fetched 10.3 MB in 4s (2,370 kB/s)
   108          Reading package lists... Done
   109  
   110  4. Install the `ubuntu-zfs` package.
   111  
   112          $ sudo apt-get install -y ubuntu-zfs
   113          Reading package lists... Done
   114          Building dependency tree
   115          <output truncated>
   116  
   117  5. Load the `zfs` module.
   118  
   119          $ sudo modprobe zfs
   120  
   121  6. Verify that it loaded correctly.
   122  
   123          $ lsmod | grep zfs
   124          zfs                  2768247  0
   125          zunicode              331170  1 zfs
   126          zcommon                55411  1 zfs
   127          znvpair                89086  2 zfs,zcommon
   128          spl                    96378  3 zfs,zcommon,znvpair
   129          zavl                   15236  1 zfs
   130  
   131  ## Configure ZFS for Docker
   132  
   133  Once ZFS is installed and loaded, you're ready to configure ZFS for Docker.
   134  
   135  
   136  1. Create a new `zpool`.
   137  
   138          $ sudo zpool create -f zpool-docker /dev/xvdb
   139  
   140    The command creates the `zpool` and gives it the name "zpool-docker". The name is arbitrary.
   141  
   142  2. Check that the `zpool` exists.
   143  
   144          $ sudo zfs list
   145          NAME            USED  AVAIL    REFER  MOUNTPOINT
   146          zpool-docker    55K   3.84G    19K    /zpool-docker
   147  
   148  3. Create and mount a new ZFS filesystem to `/var/lib/docker`.
   149  
   150          $ sudo zfs create -o mountpoint=/var/lib/docker zpool-docker/docker
   151  
   152  4. Check that the previous step worked.
   153  
   154          $ sudo zfs list -t all
   155          NAME                 USED  AVAIL  REFER  MOUNTPOINT
   156          zpool-docker         93.5K  3.84G    19K  /zpool-docker
   157          zpool-docker/docker  19K    3.84G    19K  /var/lib/docker
   158  
   159    Now that you have a ZFS filesystem mounted to `/var/lib/docker`, the daemon should automatically load with the `zfs` storage driver.
   160  
   161  5. Start the Docker daemon.
   162  
   163          $ sudo service docker start
   164          docker start/running, process 2315
   165  
   166      The procedure for starting the Docker daemon may differ depending on the
   167      Linux distribution you are using. It is possible to force the Docker daemon
   168      to start with the `zfs` storage driver by passing the `--storage-driver=zfs`
   169      flag to the `docker daemon` command, or to the `DOCKER_OPTS` line in the
   170      Docker config file.
   171  
   172  6. Verify that the daemon is using the `zfs` storage driver.
   173  
   174          $ sudo docker info
   175          Containers: 0
   176          Images: 0
   177          Storage Driver: zfs
   178           Zpool: zpool-docker
   179           Zpool Health: ONLINE
   180           Parent Dataset: zpool-docker/docker
   181           Space Used By Parent: 27648
   182           Space Available: 4128139776
   183           Parent Quota: no
   184           Compression: off
   185          Execution Driver: native-0.2
   186          [...]
   187  
   188    The output of the command above shows that the Docker daemon is using the
   189    `zfs` storage driver and that the parent dataset is the `zpool-docker/docker`
   190    filesystem created earlier.
   191  
   192  Your Docker host is now using ZFS to store to manage its images and containers.
   193  
   194  ## ZFS and Docker performance
   195  
   196  There are several factors that influence the performance of Docker using the `zfs` storage driver.
   197  
   198  - **Memory**. Memory has a major impact on ZFS performance. This goes back to the fact that ZFS was originally designed for use on big Sun Solaris servers with large amounts of memory. Keep this in mind when sizing your Docker hosts.
   199  
   200  - **ZFS Features**. Using ZFS features, such as deduplication, can significantly increase the amount
   201  of memory ZFS uses. For memory consumption and performance reasons it is
   202  recommended to turn off ZFS deduplication. However, deduplication at other
   203  layers in the stack (such as SAN or NAS arrays) can still be used as these do
   204  not impact ZFS memory usage and performance. If using SAN, NAS or other hardware
   205  RAID technologies you should continue to follow existing best practices for
   206  using them with ZFS.
   207  
   208  * **ZFS Caching**. ZFS caches disk blocks in a memory structure called the adaptive replacement cache (ARC). The *Single Copy ARC* feature of ZFS allows a single cached copy of a block to be shared by multiple clones of a filesystem. This means that multiple running containers can share a single copy of cached block. This means that ZFS is a good option for PaaS and other high density use cases.
   209  
   210  - **Fragmentation**. Fragmentation is a natural byproduct of copy-on-write filesystems like ZFS. However, ZFS writes in 128K blocks and allocates *slabs* (multiple 128K blocks) to CoW operations in an attempt to reduce fragmentation. The ZFS intent log (ZIL) and the coalescing of writes (delayed writes) also help to reduce fragmentation. 
   211  
   212  - **Use the native ZFS driver for Linux**. Although the Docker `zfs` storage driver supports the ZFS FUSE implementation, it is not recommended when high performance is required. The native ZFS on Linux driver tends to perform better than the FUSE implementation.
   213  
   214  The following generic performance best practices also apply to ZFS.
   215  
   216  - **Use of SSD**. For best performance it is always a good idea to use fast storage media such as solid state devices (SSD). However, if you only have a limited amount of SSD storage available it is recommended to place the ZIL on SSD.
   217  
   218  - **Use Data Volumes**. Data volumes provide the best and most predictable performance. This is because they bypass the storage driver and do not incur any of the potential overheads introduced by thin provisioning and copy-on-write. For this reason, you may want to place heavy write workloads on data volumes.