github.com/docker/docker-ce@v17.12.1-ce-rc2+incompatible/components/cli/docs/reference/commandline/dockerd.md (about)

     1  ---
     2  title: "dockerd"
     3  aliases: ["/engine/reference/commandline/daemon/"]
     4  description: "The daemon command description and usage"
     5  keywords: "container, daemon, runtime"
     6  ---
     7  
     8  <!-- This file is maintained within the docker/cli GitHub
     9       repository at https://github.com/docker/cli/. Make all
    10       pull requests against that repo. If you see this file in
    11       another repository, consider it read-only there, as it will
    12       periodically be overwritten by the definitive file. Pull
    13       requests which include edits to this file in other repositories
    14       will be rejected.
    15  -->
    16  
    17  # daemon
    18  
    19  ```markdown
    20  Usage:	dockerd COMMAND
    21  
    22  A self-sufficient runtime for containers.
    23  
    24  Options:
    25        --add-runtime runtime                   Register an additional OCI compatible runtime (default [])
    26        --allow-nondistributable-artifacts list Push nondistributable artifacts to specified registries (default [])
    27        --api-cors-header string                Set CORS headers in the Engine API
    28        --authorization-plugin list             Authorization plugins to load (default [])
    29        --bip string                            Specify network bridge IP
    30    -b, --bridge string                         Attach containers to a network bridge
    31        --cgroup-parent string                  Set parent cgroup for all containers
    32        --cluster-advertise string              Address or interface name to advertise
    33        --cluster-store string                  URL of the distributed storage backend
    34        --cluster-store-opt map                 Set cluster store options (default map[])
    35        --config-file string                    Daemon configuration file (default "/etc/docker/daemon.json")
    36        --containerd string                     Path to containerd socket
    37        --cpu-rt-period int                     Limit the CPU real-time period in microseconds
    38        --cpu-rt-runtime int                    Limit the CPU real-time runtime in microseconds
    39        --data-root string                      Root directory of persistent Docker state (default "/var/lib/docker")
    40    -D, --debug                                 Enable debug mode
    41        --default-gateway ip                    Container default gateway IPv4 address
    42        --default-gateway-v6 ip                 Container default gateway IPv6 address
    43        --default-runtime string                Default OCI runtime for containers (default "runc")
    44        --default-ulimit ulimit                 Default ulimits for containers (default [])
    45        --dns list                              DNS server to use (default [])
    46        --dns-opt list                          DNS options to use (default [])
    47        --dns-search list                       DNS search domains to use (default [])
    48        --exec-opt list                         Runtime execution options (default [])
    49        --exec-root string                      Root directory for execution state files (default "/var/run/docker")
    50        --experimental                          Enable experimental features
    51        --fixed-cidr string                     IPv4 subnet for fixed IPs
    52        --fixed-cidr-v6 string                  IPv6 subnet for fixed IPs
    53    -G, --group string                          Group for the unix socket (default "docker")
    54        --help                                  Print usage
    55    -H, --host list                             Daemon socket(s) to connect to (default [])
    56        --icc                                   Enable inter-container communication (default true)
    57        --init                                  Run an init in the container to forward signals and reap processes
    58        --init-path string                      Path to the docker-init binary
    59        --insecure-registry list                Enable insecure registry communication (default [])
    60        --ip ip                                 Default IP when binding container ports (default 0.0.0.0)
    61        --ip-forward                            Enable net.ipv4.ip_forward (default true)
    62        --ip-masq                               Enable IP masquerading (default true)
    63        --iptables                              Enable addition of iptables rules (default true)
    64        --ipv6                                  Enable IPv6 networking
    65        --label list                            Set key=value labels to the daemon (default [])
    66        --live-restore                          Enable live restore of docker when containers are still running
    67        --log-driver string                     Default driver for container logs (default "json-file")
    68    -l, --log-level string                      Set the logging level ("debug", "info", "warn", "error", "fatal") (default "info")
    69        --log-opt map                           Default log driver options for containers (default map[])
    70        --max-concurrent-downloads int          Set the max concurrent downloads for each pull (default 3)
    71        --max-concurrent-uploads int            Set the max concurrent uploads for each push (default 5)
    72        --metrics-addr string                   Set default address and port to serve the metrics api on
    73        --mtu int                               Set the containers network MTU
    74        --node-generic-resources list           Advertise user-defined resource
    75        --no-new-privileges                     Set no-new-privileges by default for new containers
    76        --oom-score-adjust int                  Set the oom_score_adj for the daemon (default -500)
    77    -p, --pidfile string                        Path to use for daemon PID file (default "/var/run/docker.pid")
    78        --raw-logs                              Full timestamps without ANSI coloring
    79        --registry-mirror list                  Preferred Docker registry mirror (default [])
    80        --seccomp-profile string                Path to seccomp profile
    81        --selinux-enabled                       Enable selinux support
    82        --shutdown-timeout int                  Set the default shutdown timeout (default 15)
    83    -s, --storage-driver string                 Storage driver to use
    84        --storage-opt list                      Storage driver options (default [])
    85        --swarm-default-advertise-addr string   Set default address or interface for swarm advertised address
    86        --tls                                   Use TLS; implied by --tlsverify
    87        --tlscacert string                      Trust certs signed only by this CA (default "~/.docker/ca.pem")
    88        --tlscert string                        Path to TLS certificate file (default "~/.docker/cert.pem")
    89        --tlskey string                         Path to TLS key file (default ~/.docker/key.pem")
    90        --tlsverify                             Use TLS and verify the remote
    91        --userland-proxy                        Use userland proxy for loopback traffic (default true)
    92        --userland-proxy-path string            Path to the userland proxy binary
    93        --userns-remap string                   User/Group setting for user namespaces
    94    -v, --version                               Print version information and quit
    95  ```
    96  
    97  Options with [] may be specified multiple times.
    98  
    99  ## Description
   100  
   101  `dockerd` is the persistent process that manages containers. Docker
   102  uses different binaries for the daemon and client. To run the daemon you
   103  type `dockerd`.
   104  
   105  To run the daemon with debug output, use `dockerd -D` or add `"debug": true` to
   106  the `daemon.json` file.
   107  
   108  > **Note**: In Docker 1.13 and higher, enable experimental features by starting
   109  > `dockerd` with the `--experimental` flag or adding `"experimental": true` to the
   110  > `daemon.json` file. In earlier Docker versions, a different build was required
   111  > to enable experimental features.
   112  
   113  ## Examples
   114  
   115  ### Daemon socket option
   116  
   117  The Docker daemon can listen for [Docker Engine API](../api/)
   118  requests via three different types of Socket: `unix`, `tcp`, and `fd`.
   119  
   120  By default, a `unix` domain socket (or IPC socket) is created at
   121  `/var/run/docker.sock`, requiring either `root` permission, or `docker` group
   122  membership.
   123  
   124  If you need to access the Docker daemon remotely, you need to enable the `tcp`
   125  Socket. Beware that the default setup provides un-encrypted and
   126  un-authenticated direct access to the Docker daemon - and should be secured
   127  either using the [built in HTTPS encrypted socket](https://docs.docker.com/engine/security/https/), or by
   128  putting a secure web proxy in front of it. You can listen on port `2375` on all
   129  network interfaces with `-H tcp://0.0.0.0:2375`, or on a particular network
   130  interface using its IP address: `-H tcp://192.168.59.103:2375`. It is
   131  conventional to use port `2375` for un-encrypted, and port `2376` for encrypted
   132  communication with the daemon.
   133  
   134  > **Note**: If you're using an HTTPS encrypted socket, keep in mind that only
   135  > TLS1.0 and greater are supported. Protocols SSLv3 and under are not
   136  > supported anymore for security reasons.
   137  
   138  On Systemd based systems, you can communicate with the daemon via
   139  [Systemd socket activation](http://0pointer.de/blog/projects/socket-activation.html),
   140  use `dockerd -H fd://`. Using `fd://` will work perfectly for most setups but
   141  you can also specify individual sockets: `dockerd -H fd://3`. If the
   142  specified socket activated files aren't found, then Docker will exit. You can
   143  find examples of using Systemd socket activation with Docker and Systemd in the
   144  [Docker source tree](https://github.com/docker/docker/tree/master/contrib/init/systemd/).
   145  
   146  You can configure the Docker daemon to listen to multiple sockets at the same
   147  time using multiple `-H` options:
   148  
   149  ```bash
   150  # listen using the default unix socket, and on 2 specific IP addresses on this host.
   151  
   152  $ sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2
   153  ```
   154  
   155  The Docker client will honor the `DOCKER_HOST` environment variable to set the
   156  `-H` flag for the client. Use **one** of the following commands:
   157  
   158  ```bash
   159  $ docker -H tcp://0.0.0.0:2375 ps
   160  ```
   161  
   162  ```bash
   163  $ export DOCKER_HOST="tcp://0.0.0.0:2375"
   164  
   165  $ docker ps
   166  ```
   167  
   168  Setting the `DOCKER_TLS_VERIFY` environment variable to any value other than
   169  the empty string is equivalent to setting the `--tlsverify` flag. The following
   170  are equivalent:
   171  
   172  ```bash
   173  $ docker --tlsverify ps
   174  # or
   175  $ export DOCKER_TLS_VERIFY=1
   176  $ docker ps
   177  ```
   178  
   179  The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`
   180  environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes
   181  precedence over `HTTP_PROXY`.
   182  
   183  #### Bind Docker to another host/port or a Unix socket
   184  
   185  > **Warning**:
   186  > Changing the default `docker` daemon binding to a
   187  > TCP port or Unix *docker* user group will increase your security risks
   188  > by allowing non-root users to gain *root* access on the host. Make sure
   189  > you control access to `docker`. If you are binding
   190  > to a TCP port, anyone with access to that port has full Docker access;
   191  > so it is not advisable on an open network.
   192  
   193  With `-H` it is possible to make the Docker daemon to listen on a
   194  specific IP and port. By default, it will listen on
   195  `unix:///var/run/docker.sock` to allow only local connections by the
   196  *root* user. You *could* set it to `0.0.0.0:2375` or a specific host IP
   197  to give access to everybody, but that is **not recommended** because
   198  then it is trivial for someone to gain root access to the host where the
   199  daemon is running.
   200  
   201  Similarly, the Docker client can use `-H` to connect to a custom port.
   202  The Docker client will default to connecting to `unix:///var/run/docker.sock`
   203  on Linux, and `tcp://127.0.0.1:2376` on Windows.
   204  
   205  `-H` accepts host and port assignment in the following format:
   206  
   207      tcp://[host]:[port][path] or unix://path
   208  
   209  For example:
   210  
   211  -   `tcp://` -> TCP connection to `127.0.0.1` on either port `2376` when TLS encryption
   212      is on, or port `2375` when communication is in plain text.
   213  -   `tcp://host:2375` -> TCP connection on
   214      host:2375
   215  -   `tcp://host:2375/path` -> TCP connection on
   216      host:2375 and prepend path to all requests
   217  -   `unix://path/to/socket` -> Unix socket located
   218      at `path/to/socket`
   219  
   220  `-H`, when empty, will default to the same value as
   221  when no `-H` was passed in.
   222  
   223  `-H` also accepts short form for TCP bindings: `host:` or `host:port` or `:port`
   224  
   225  Run Docker in daemon mode:
   226  
   227  ```bash
   228  $ sudo <path to>/dockerd -H 0.0.0.0:5555 &
   229  ```
   230  
   231  Download an `ubuntu` image:
   232  
   233  ```bash
   234  $ docker -H :5555 pull ubuntu
   235  ```
   236  
   237  You can use multiple `-H`, for example, if you want to listen on both
   238  TCP and a Unix socket
   239  
   240  ```bash
   241  # Run docker in daemon mode
   242  $ sudo <path to>/dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock &
   243  # Download an ubuntu image, use default Unix socket
   244  $ docker pull ubuntu
   245  # OR use the TCP port
   246  $ docker -H tcp://127.0.0.1:2375 pull ubuntu
   247  ```
   248  
   249  ### Daemon storage-driver
   250  
   251  On Linux, the Docker daemon has support for several different image layer storage
   252  drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay` and `overlay2`.
   253  
   254  The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that
   255  is unlikely to be merged into the main kernel. These are also known to cause
   256  some serious kernel crashes. However `aufs` allows containers to share
   257  executable and shared library memory, so is a useful choice when running
   258  thousands of containers with the same program or libraries.
   259  
   260  The `devicemapper` driver uses thin provisioning and Copy on Write (CoW)
   261  snapshots. For each devicemapper graph location – typically
   262  `/var/lib/docker/devicemapper` – a thin pool is created based on two block
   263  devices, one for data and one for metadata. By default, these block devices
   264  are created automatically by using loopback mounts of automatically created
   265  sparse files. Refer to [Devicemapper options](#devicemapper-options) below
   266  for a way how to customize this setup.
   267  [~jpetazzo/Resizing Docker containers with the Device Mapper plugin](http://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/)
   268  article explains how to tune your existing setup without the use of options.
   269  
   270  The `btrfs` driver is very fast for `docker build` - but like `devicemapper`
   271  does not share executable memory between devices. Use
   272  `dockerd -s btrfs -g /mnt/btrfs_partition`.
   273  
   274  The `zfs` driver is probably not as fast as `btrfs` but has a longer track record
   275  on stability. Thanks to `Single Copy ARC` shared blocks between clones will be
   276  cached only once. Use `dockerd -s zfs`. To select a different zfs filesystem
   277  set `zfs.fsname` option as described in [ZFS options](#zfs-options).
   278  
   279  The `overlay` is a very fast union filesystem. It is now merged in the main
   280  Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). `overlay`
   281  also supports page cache sharing, this means multiple containers accessing
   282  the same file can share a single page cache entry (or entries), it makes
   283  `overlay` as efficient with memory as `aufs` driver. Call
   284  `dockerd -s overlay` to use it.
   285  
   286  > **Note**: As promising as `overlay` is, the feature is still quite young and
   287  > should not be used in production. Most notably, using `overlay` can cause
   288  > excessive inode consumption (especially as the number of images grows), as
   289  > well as > being incompatible with the use of RPMs.
   290  
   291  The `overlay2` uses the same fast union filesystem but takes advantage of
   292  [additional features](https://lkml.org/lkml/2015/2/11/106) added in Linux
   293  kernel 4.0 to avoid excessive inode consumption. Call `dockerd -s overlay2`
   294  to use it.
   295  
   296  > **Note**: Both `overlay` and `overlay2` are currently unsupported on `btrfs`
   297  > or any Copy on Write filesystem and should only be used over `ext4` partitions.
   298  
   299  On Windows, the Docker daemon supports a single image layer storage driver
   300  depending on the image platform: `windowsfilter` for Windows images, and
   301  `lcow` for Linux containers on Windows.
   302  
   303  ### Options per storage driver
   304  
   305  Particular storage-driver can be configured with options specified with
   306  `--storage-opt` flags. Options for `devicemapper` are prefixed with `dm`,
   307  options for `zfs` start with `zfs`, options for `btrfs` start with `btrfs`
   308  and options for `lcow` start with `lcow`.
   309  
   310  #### Devicemapper options
   311  
   312  This is an example of the configuration file for devicemapper on Linux:
   313  
   314  ```json
   315  {
   316    "storage-driver": "devicemapper",
   317    "storage-opts": [
   318      "dm.thinpooldev=/dev/mapper/thin-pool",
   319      "dm.use_deferred_deletion=true",
   320      "dm.use_deferred_removal=true"
   321    ]
   322  }
   323  ```
   324  
   325  ##### `dm.thinpooldev`
   326  
   327  Specifies a custom block storage device to use for the thin pool.
   328  
   329  If using a block device for device mapper storage, it is best to use `lvm`
   330  to create and manage the thin-pool volume. This volume is then handed to Docker
   331  to exclusively create snapshot volumes needed for images and containers.
   332  
   333  Managing the thin-pool outside of Engine makes for the most feature-rich
   334  method of having Docker utilize device mapper thin provisioning as the
   335  backing storage for Docker containers. The highlights of the lvm-based
   336  thin-pool management feature include: automatic or interactive thin-pool
   337  resize support, dynamically changing thin-pool features, automatic thinp
   338  metadata checking when lvm activates the thin-pool, etc.
   339  
   340  As a fallback if no thin pool is provided, loopback files are
   341  created. Loopback is very slow, but can be used without any
   342  pre-configuration of storage. It is strongly recommended that you do
   343  not use loopback in production. Ensure your Engine daemon has a
   344  `--storage-opt dm.thinpooldev` argument provided.
   345  
   346  ###### Example:
   347  
   348  ```bash
   349  $ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool
   350  ```
   351  
   352  ##### `dm.directlvm_device`
   353  
   354  As an alternative to providing a thin pool as above, Docker can setup a block
   355  device for you.
   356  
   357  ###### Example:
   358  
   359  ```bash
   360  $ sudo dockerd --storage-opt dm.directlvm_device=/dev/xvdf
   361  ```
   362  
   363  ##### `dm.thinp_percent`
   364  
   365  Sets the percentage of passed in block device to use for storage.
   366  
   367  ###### Example:
   368  
   369  ```bash
   370  $ sudo dockerd --storage-opt dm.thinp_percent=95
   371  ```
   372  
   373  ##### `dm.thinp_metapercent`
   374  
   375  Sets the percentage of the passed in block device to use for metadata storage.
   376  
   377  ###### Example:
   378  
   379  ```bash
   380  $ sudo dockerd --storage-opt dm.thinp_metapercent=1
   381  ```
   382  
   383  ##### `dm.thinp_autoextend_threshold`
   384  
   385  Sets the value of the percentage of space used before `lvm` attempts to
   386  autoextend the available space [100 = disabled]
   387  
   388  ###### Example:
   389  
   390  ```bash
   391  $ sudo dockerd --storage-opt dm.thinp_autoextend_threshold=80
   392  ```
   393  
   394  ##### `dm.thinp_autoextend_percent`
   395  
   396  Sets the value percentage value to increase the thin pool by when `lvm`
   397  attempts to autoextend the available space [100 = disabled]
   398  
   399  ###### Example:
   400  
   401  ```bash
   402  $ sudo dockerd --storage-opt dm.thinp_autoextend_percent=20
   403  ```
   404  
   405  
   406  ##### `dm.basesize`
   407  
   408  Specifies the size to use when creating the base device, which limits the
   409  size of images and containers. The default value is 10G. Note, thin devices
   410  are inherently "sparse", so a 10G device which is mostly empty doesn't use
   411  10 GB of space on the pool. However, the filesystem will use more space for
   412  the empty case the larger the device is.
   413  
   414  The base device size can be increased at daemon restart which will allow
   415  all future images and containers (based on those new images) to be of the
   416  new base device size.
   417  
   418  ###### Examples
   419  
   420  ```bash
   421  $ sudo dockerd --storage-opt dm.basesize=50G
   422  ```
   423  
   424  This will increase the base device size to 50G. The Docker daemon will throw an
   425  error if existing base device size is larger than 50G. A user can use
   426  this option to expand the base device size however shrinking is not permitted.
   427  
   428  This value affects the system-wide "base" empty filesystem
   429  that may already be initialized and inherited by pulled images. Typically,
   430  a change to this value requires additional steps to take effect:
   431  
   432   ```bash
   433  $ sudo service docker stop
   434  
   435  $ sudo rm -rf /var/lib/docker
   436  
   437  $ sudo service docker start
   438  ```
   439  
   440  
   441  ##### `dm.loopdatasize`
   442  
   443  > **Note**: This option configures devicemapper loopback, which should not
   444  > be used in production.
   445  
   446  Specifies the size to use when creating the loopback file for the
   447  "data" device which is used for the thin pool. The default size is
   448  100G. The file is sparse, so it will not initially take up this
   449  much space.
   450  
   451  ###### Example
   452  
   453  ```bash
   454  $ sudo dockerd --storage-opt dm.loopdatasize=200G
   455  ```
   456  
   457  ##### `dm.loopmetadatasize`
   458  
   459  > **Note**: This option configures devicemapper loopback, which should not
   460  > be used in production.
   461  
   462  Specifies the size to use when creating the loopback file for the
   463  "metadata" device which is used for the thin pool. The default size
   464  is 2G. The file is sparse, so it will not initially take up
   465  this much space.
   466  
   467  ###### Example
   468  
   469  ```bash
   470  $ sudo dockerd --storage-opt dm.loopmetadatasize=4G
   471  ```
   472  
   473  ##### `dm.fs`
   474  
   475  Specifies the filesystem type to use for the base device. The supported
   476  options are "ext4" and "xfs". The default is "xfs"
   477  
   478  ###### Example
   479  
   480  ```bash
   481  $ sudo dockerd --storage-opt dm.fs=ext4
   482  ```
   483  
   484  ##### `dm.mkfsarg`
   485  
   486  Specifies extra mkfs arguments to be used when creating the base device.
   487  
   488  ###### Example
   489  
   490  ```bash
   491  $ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal"
   492  ```
   493  
   494  ##### `dm.mountopt`
   495  
   496  Specifies extra mount options used when mounting the thin devices.
   497  
   498  ###### Example
   499  
   500  ```bash
   501  $ sudo dockerd --storage-opt dm.mountopt=nodiscard
   502  ```
   503  
   504  ##### `dm.datadev`
   505  
   506  (Deprecated, use `dm.thinpooldev`)
   507  
   508  Specifies a custom blockdevice to use for data for the thin pool.
   509  
   510  If using a block device for device mapper storage, ideally both `datadev` and
   511  `metadatadev` should be specified to completely avoid using the loopback
   512  device.
   513  
   514  ###### Example
   515  
   516  ```bash
   517  $ sudo dockerd \
   518        --storage-opt dm.datadev=/dev/sdb1 \
   519        --storage-opt dm.metadatadev=/dev/sdc1
   520  ```
   521  
   522  ##### `dm.metadatadev`
   523  
   524  (Deprecated, use `dm.thinpooldev`)
   525  
   526  Specifies a custom blockdevice to use for metadata for the thin pool.
   527  
   528  For best performance the metadata should be on a different spindle than the
   529  data, or even better on an SSD.
   530  
   531  If setting up a new metadata pool it is required to be valid. This can be
   532  achieved by zeroing the first 4k to indicate empty metadata, like this:
   533  
   534  ```bash
   535  $ dd if=/dev/zero of=$metadata_dev bs=4096 count=1
   536  ```
   537  
   538  ###### Example
   539  
   540  ```bash
   541  $ sudo dockerd \
   542        --storage-opt dm.datadev=/dev/sdb1 \
   543        --storage-opt dm.metadatadev=/dev/sdc1
   544  ```
   545  
   546  ##### `dm.blocksize`
   547  
   548  Specifies a custom blocksize to use for the thin pool. The default
   549  blocksize is 64K.
   550  
   551  ###### Example
   552  
   553  ```bash
   554  $ sudo dockerd --storage-opt dm.blocksize=512K
   555  ```
   556  
   557  ##### `dm.blkdiscard`
   558  
   559  Enables or disables the use of `blkdiscard` when removing devicemapper
   560  devices. This is enabled by default (only) if using loopback devices and is
   561  required to resparsify the loopback file on image/container removal.
   562  
   563  Disabling this on loopback can lead to *much* faster container removal
   564  times, but will make the space used in `/var/lib/docker` directory not be
   565  returned to the system for other use when containers are removed.
   566  
   567  ###### Examples
   568  
   569  ```bash
   570  $ sudo dockerd --storage-opt dm.blkdiscard=false
   571  ```
   572  
   573  ##### `dm.override_udev_sync_check`
   574  
   575  Overrides the `udev` synchronization checks between `devicemapper` and `udev`.
   576  `udev` is the device manager for the Linux kernel.
   577  
   578  To view the `udev` sync support of a Docker daemon that is using the
   579  `devicemapper` driver, run:
   580  
   581  ```bash
   582  $ docker info
   583  [...]
   584  Udev Sync Supported: true
   585  [...]
   586  ```
   587  
   588  When `udev` sync support is `true`, then `devicemapper` and udev can
   589  coordinate the activation and deactivation of devices for containers.
   590  
   591  When `udev` sync support is `false`, a race condition occurs between
   592  the`devicemapper` and `udev` during create and cleanup. The race condition
   593  results in errors and failures. (For information on these failures, see
   594  [docker#4036](https://github.com/docker/docker/issues/4036))
   595  
   596  To allow the `docker` daemon to start, regardless of `udev` sync not being
   597  supported, set `dm.override_udev_sync_check` to true:
   598  
   599  ```bash
   600  $ sudo dockerd --storage-opt dm.override_udev_sync_check=true
   601  ```
   602  
   603  When this value is `true`, the  `devicemapper` continues and simply warns
   604  you the errors are happening.
   605  
   606  > **Note**: The ideal is to pursue a `docker` daemon and environment that does
   607  > support synchronizing with `udev`. For further discussion on this
   608  > topic, see [docker#4036](https://github.com/docker/docker/issues/4036).
   609  > Otherwise, set this flag for migrating existing Docker daemons to
   610  > a daemon with a supported environment.
   611  
   612  ##### `dm.use_deferred_removal`
   613  
   614  Enables use of deferred device removal if `libdm` and the kernel driver
   615  support the mechanism.
   616  
   617  Deferred device removal means that if device is busy when devices are
   618  being removed/deactivated, then a deferred removal is scheduled on
   619  device. And devices automatically go away when last user of the device
   620  exits.
   621  
   622  For example, when a container exits, its associated thin device is removed.
   623  If that device has leaked into some other mount namespace and can't be
   624  removed, the container exit still succeeds and this option causes the
   625  system to schedule the device for deferred removal. It does not wait in a
   626  loop trying to remove a busy device.
   627  
   628  ###### Example
   629  
   630  ```bash
   631  $ sudo dockerd --storage-opt dm.use_deferred_removal=true
   632  ```
   633  
   634  ##### `dm.use_deferred_deletion`
   635  
   636  Enables use of deferred device deletion for thin pool devices. By default,
   637  thin pool device deletion is synchronous. Before a container is deleted,
   638  the Docker daemon removes any associated devices. If the storage driver
   639  can not remove a device, the container deletion fails and daemon returns.
   640  
   641  ```none
   642  Error deleting container: Error response from daemon: Cannot destroy container
   643  ```
   644  
   645  To avoid this failure, enable both deferred device deletion and deferred
   646  device removal on the daemon.
   647  
   648  ```bash
   649  $ sudo dockerd \
   650        --storage-opt dm.use_deferred_deletion=true \
   651        --storage-opt dm.use_deferred_removal=true
   652  ```
   653  
   654  With these two options enabled, if a device is busy when the driver is
   655  deleting a container, the driver marks the device as deleted. Later, when
   656  the device isn't in use, the driver deletes it.
   657  
   658  In general it should be safe to enable this option by default. It will help
   659  when unintentional leaking of mount point happens across multiple mount
   660  namespaces.
   661  
   662  ##### `dm.min_free_space`
   663  
   664  Specifies the min free space percent in a thin pool require for new device
   665  creation to succeed. This check applies to both free data space as well
   666  as free metadata space. Valid values are from 0% - 99%. Value 0% disables
   667  free space checking logic. If user does not specify a value for this option,
   668  the Engine uses a default value of 10%.
   669  
   670  Whenever a new a thin pool device is created (during `docker pull` or during
   671  container creation), the Engine checks if the minimum free space is
   672  available. If sufficient space is unavailable, then device creation fails
   673  and any relevant `docker` operation fails.
   674  
   675  To recover from this error, you must create more free space in the thin pool
   676  to recover from the error. You can create free space by deleting some images
   677  and containers from the thin pool. You can also add more storage to the thin
   678  pool.
   679  
   680  To add more space to a LVM (logical volume management) thin pool, just add
   681  more storage to the volume group container thin pool; this should automatically
   682  resolve any errors. If your configuration uses loop devices, then stop the
   683  Engine daemon, grow the size of loop files and restart the daemon to resolve
   684  the issue.
   685  
   686  ###### Example
   687  
   688  ```bash
   689  $ sudo dockerd --storage-opt dm.min_free_space=10%
   690  ```
   691  
   692  ##### `dm.xfs_nospace_max_retries`
   693  
   694  Specifies the maximum number of retries XFS should attempt to complete
   695  IO when ENOSPC (no space) error is returned by underlying storage device.
   696  
   697  By default XFS retries infinitely for IO to finish and this can result
   698  in unkillable process. To change this behavior one can set
   699  xfs_nospace_max_retries to say 0 and XFS will not retry IO after getting
   700  ENOSPC and will shutdown filesystem.
   701  
   702  ###### Example
   703  
   704  ```bash
   705  $ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0
   706  ```
   707  
   708  ##### `dm.libdm_log_level`
   709  
   710  Specifies the maxmimum `libdm` log level that will be forwarded to the
   711  `dockerd` log (as specified by `--log-level`). This option is primarily
   712  intended for debugging problems involving `libdm`. Using values other than the
   713  defaults may cause false-positive warnings to be logged.
   714  
   715  Values specified must fall within the range of valid `libdm` log levels. At the
   716  time of writing, the following is the list of `libdm` log levels as well as
   717  their corresponding levels when output by `dockerd`.
   718  
   719  | `libdm` Level | Value | `--log-level` |
   720  | ------------- | -----:| ------------- |
   721  | `_LOG_FATAL`  |     2 | error         |
   722  | `_LOG_ERR`    |     3 | error         |
   723  | `_LOG_WARN`   |     4 | warn          |
   724  | `_LOG_NOTICE` |     5 | info          |
   725  | `_LOG_INFO`   |     6 | info          |
   726  | `_LOG_DEBUG`  |     7 | debug         |
   727  
   728  ###### Example
   729  
   730  ```bash
   731  $ sudo dockerd \
   732        --log-level debug \
   733        --storage-opt dm.libdm_log_level=7
   734  ```
   735  
   736  #### ZFS options
   737  
   738  ##### `zfs.fsname`
   739  
   740  Set zfs filesystem under which docker will create its own datasets.
   741  By default docker will pick up the zfs filesystem where docker graph
   742  (`/var/lib/docker`) is located.
   743  
   744  ###### Example
   745  
   746  ```bash
   747  $ sudo dockerd -s zfs --storage-opt zfs.fsname=zroot/docker
   748  ```
   749  
   750  #### Btrfs options
   751  
   752  ##### `btrfs.min_space`
   753  
   754  Specifies the minimum size to use when creating the subvolume which is used
   755  for containers. If user uses disk quota for btrfs when creating or running
   756  a container with **--storage-opt size** option, docker should ensure the
   757  **size** cannot be smaller than **btrfs.min_space**.
   758  
   759  ###### Example
   760  
   761  ```bash
   762  $ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G
   763  ```
   764  
   765  #### Overlay2 options
   766  
   767  ##### `overlay2.override_kernel_check`
   768  
   769  Overrides the Linux kernel version check allowing overlay2. Support for
   770  specifying multiple lower directories needed by overlay2 was added to the
   771  Linux kernel in 4.0.0. However, some older kernel versions may be patched
   772  to add multiple lower directory support for OverlayFS. This option should
   773  only be used after verifying this support exists in the kernel. Applying
   774  this option on a kernel without this support will cause failures on mount.
   775  
   776  ##### `overlay2.size`
   777  
   778  Sets the default max size of the container. It is supported only when the
   779  backing fs is `xfs` and mounted with `pquota` mount option. Under these
   780  conditions the user can pass any size less then the backing fs size.
   781  
   782  ###### Example
   783  
   784  ```bash
   785  $ sudo dockerd -s overlay2 --storage-opt overlay2.size=1G
   786  ```
   787  
   788  
   789  #### Windowsfilter options
   790  
   791  ##### `size`
   792  
   793  Specifies the size to use when creating the sandbox which is used for containers.
   794  Defaults to 20G.
   795  
   796  ###### Example
   797  
   798  ```PowerShell
   799  C:\> dockerd --storage-opt size=40G
   800  ```
   801  
   802  #### LCOW (Linux Containers on Windows) options
   803  
   804  ##### `lcow.globalmode`
   805  
   806  Specifies whether the daemon instantiates utility VM instances as required 
   807  (recommended and default if omitted), or uses single global utility VM (better
   808  performance, but has security implications and not recommended for production
   809  deployments).
   810  
   811  ###### Example
   812  
   813  ```PowerShell
   814  C:\> dockerd --storage-opt lcow.globalmode=false
   815  ```
   816  
   817  ##### `lcow.kirdpath`
   818  
   819  Specifies the folder path to the location of a pair of kernel and initrd files
   820  used for booting a utility VM. Defaults to `%ProgramFiles%\Linux Containers`.
   821  
   822  ###### Example
   823  
   824  ```PowerShell
   825  C:\> dockerd --storage-opt lcow.kirdpath=c:\path\to\files
   826  ```
   827  
   828  ##### `lcow.kernel`
   829  
   830  Specifies the filename of a kernel file located in the `lcow.kirdpath` path.
   831  Defaults to `bootx64.efi`.
   832  
   833  ###### Example
   834  
   835  ```PowerShell
   836  C:\> dockerd --storage-opt lcow.kernel=kernel.efi
   837  ```
   838  
   839  ##### `lcow.initrd`
   840  
   841  Specifies the filename of an initrd file located in the `lcow.kirdpath` path.
   842  Defaults to `initrd.img`.
   843  
   844  ###### Example
   845  
   846  ```PowerShell
   847  C:\> dockerd --storage-opt lcow.initrd=myinitrd.img
   848  ```
   849  
   850  ##### `lcow.bootparameters`
   851  
   852  Specifies additional boot parameters for booting utility VMs when in kernel/
   853  initrd mode. Ignored if the utility VM is booting from VHD. These settings
   854  are kernel specific.
   855  
   856  ###### Example
   857  
   858  ```PowerShell
   859  C:\> dockerd --storage-opt "lcow.bootparameters='option=value'"
   860  ```
   861  
   862  ##### `lcow.vhdx`
   863  
   864  Specifies a custom VHDX to boot a utility VM, as an alternate to kernel
   865  and initrd booting. Defaults to `uvm.vhdx` under `lcow.kirdpath`.
   866  
   867  ###### Example
   868  
   869  ```PowerShell
   870  C:\> dockerd --storage-opt lcow.vhdx=custom.vhdx
   871  ```
   872  
   873  ##### `lcow.timeout`
   874  
   875  Specifies the timeout for utility VM operations in seconds. Defaults
   876  to 300.
   877  
   878  ###### Example
   879  
   880  ```PowerShell
   881  C:\> dockerd --storage-opt lcow.timeout=240
   882  ```
   883  
   884  ##### `lcow.sandboxsize`
   885  
   886  Specifies the size in GB to use when creating the sandbox which is used for
   887  containers. Defaults to 20. Cannot be less than 20.
   888  
   889  ###### Example
   890  
   891  ```PowerShell
   892  C:\> dockerd --storage-opt lcow.sandboxsize=40
   893  ```
   894  
   895  ### Docker runtime execution options
   896  
   897  The Docker daemon relies on a
   898  [OCI](https://github.com/opencontainers/runtime-spec) compliant runtime
   899  (invoked via the `containerd` daemon) as its interface to the Linux
   900  kernel `namespaces`, `cgroups`, and `SELinux`.
   901  
   902  By default, the Docker daemon automatically starts `containerd`. If you want to
   903  control `containerd` startup, manually start `containerd` and pass the path to
   904  the `containerd` socket using the `--containerd` flag. For example:
   905  
   906  ```bash
   907  $ sudo dockerd --containerd /var/run/dev/docker-containerd.sock
   908  ```
   909  
   910  Runtimes can be registered with the daemon either via the
   911  configuration file or using the `--add-runtime` command line argument.
   912  
   913  The following is an example adding 2 runtimes via the configuration:
   914  
   915  ```json
   916  {
   917  	"default-runtime": "runc",
   918  	"runtimes": {
   919  		"runc": {
   920  			"path": "runc"
   921  		},
   922  		"custom": {
   923  			"path": "/usr/local/bin/my-runc-replacement",
   924  			"runtimeArgs": [
   925  				"--debug"
   926  			]
   927  		}
   928  	}
   929  }
   930  ```
   931  
   932  This is the same example via the command line:
   933  
   934  ```bash
   935  $ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement
   936  ```
   937  
   938  > **Note**: Defining runtime arguments via the command line is not supported.
   939  
   940  #### Options for the runtime
   941  
   942  You can configure the runtime using options specified
   943  with the `--exec-opt` flag. All the flag's options have the `native` prefix. A
   944  single `native.cgroupdriver` option is available.
   945  
   946  The `native.cgroupdriver` option specifies the management of the container's
   947  cgroups. You can only specify `cgroupfs` or `systemd`. If you specify
   948  `systemd` and it is not available, the system errors out. If you omit the
   949  `native.cgroupdriver` option,` cgroupfs` is used.
   950  
   951  This example sets the `cgroupdriver` to `systemd`:
   952  
   953  ```bash
   954  $ sudo dockerd --exec-opt native.cgroupdriver=systemd
   955  ```
   956  
   957  Setting this option applies to all containers the daemon launches.
   958  
   959  Also Windows Container makes use of `--exec-opt` for special purpose. Docker user
   960  can specify default container isolation technology with this, for example:
   961  
   962  ```console
   963  > dockerd --exec-opt isolation=hyperv
   964  ```
   965  
   966  Will make `hyperv` the default isolation technology on Windows. If no isolation
   967  value is specified on daemon start, on Windows client, the default is
   968  `hyperv`, and on Windows server, the default is `process`.
   969  
   970  ### Daemon DNS options
   971  
   972  To set the DNS server for all Docker containers, use:
   973  
   974  ```bash
   975  $ sudo dockerd --dns 8.8.8.8
   976  ```
   977  
   978  To set the DNS search domain for all Docker containers, use:
   979  
   980  ```bash
   981  $ sudo dockerd --dns-search example.com
   982  ```
   983  
   984  ### Allow push of nondistributable artifacts
   985  
   986  Some images (e.g., Windows base images) contain artifacts whose distribution is
   987  restricted by license. When these images are pushed to a registry, restricted
   988  artifacts are not included.
   989  
   990  To override this behavior for specific registries, use the
   991  `--allow-nondistributable-artifacts` option in one of the following forms:
   992  
   993  * `--allow-nondistributable-artifacts myregistry:5000` tells the Docker daemon
   994    to push nondistributable artifacts to myregistry:5000.
   995  * `--allow-nondistributable-artifacts 10.1.0.0/16` tells the Docker daemon to
   996    push nondistributable artifacts to all registries whose resolved IP address
   997    is within the subnet described by the CIDR syntax.
   998  
   999  This option can be used multiple times.
  1000  
  1001  This option is useful when pushing images containing nondistributable artifacts
  1002  to a registry on an air-gapped network so hosts on that network can pull the
  1003  images without connecting to another server.
  1004  
  1005  > **Warning**: Nondistributable artifacts typically have restrictions on how
  1006  > and where they can be distributed and shared. Only use this feature to push
  1007  > artifacts to private registries and ensure that you are in compliance with
  1008  > any terms that cover redistributing nondistributable artifacts.
  1009  
  1010  ### Insecure registries
  1011  
  1012  Docker considers a private registry either secure or insecure. In the rest of
  1013  this section, *registry* is used for *private registry*, and `myregistry:5000`
  1014  is a placeholder example for a private registry.
  1015  
  1016  A secure registry uses TLS and a copy of its CA certificate is placed on the
  1017  Docker host at `/etc/docker/certs.d/myregistry:5000/ca.crt`. An insecure
  1018  registry is either not using TLS (i.e., listening on plain text HTTP), or is
  1019  using TLS with a CA certificate not known by the Docker daemon. The latter can
  1020  happen when the certificate was not found under
  1021  `/etc/docker/certs.d/myregistry:5000/`, or if the certificate verification
  1022  failed (i.e., wrong CA).
  1023  
  1024  By default, Docker assumes all, but local (see local registries below),
  1025  registries are secure. Communicating with an insecure registry is not possible
  1026  if Docker assumes that registry is secure. In order to communicate with an
  1027  insecure registry, the Docker daemon requires `--insecure-registry` in one of
  1028  the following two forms:
  1029  
  1030  * `--insecure-registry myregistry:5000` tells the Docker daemon that
  1031    myregistry:5000 should be considered insecure.
  1032  * `--insecure-registry 10.1.0.0/16` tells the Docker daemon that all registries
  1033    whose domain resolve to an IP address is part of the subnet described by the
  1034    CIDR syntax, should be considered insecure.
  1035  
  1036  The flag can be used multiple times to allow multiple registries to be marked
  1037  as insecure.
  1038  
  1039  If an insecure registry is not marked as insecure, `docker pull`,
  1040  `docker push`, and `docker search` will result in an error message prompting
  1041  the user to either secure or pass the `--insecure-registry` flag to the Docker
  1042  daemon as described above.
  1043  
  1044  Local registries, whose IP address falls in the 127.0.0.0/8 range, are
  1045  automatically marked as insecure as of Docker 1.3.2. It is not recommended to
  1046  rely on this, as it may change in the future.
  1047  
  1048  Enabling `--insecure-registry`, i.e., allowing un-encrypted and/or untrusted
  1049  communication, can be useful when running a local registry.  However,
  1050  because its use creates security vulnerabilities it should ONLY be enabled for
  1051  testing purposes.  For increased security, users should add their CA to their
  1052  system's list of trusted CAs instead of enabling `--insecure-registry`.
  1053  
  1054  #### Legacy Registries
  1055  
  1056  Starting with Docker 17.12, operations against registries supporting only the 
  1057  legacy v1 protocol are no longer supported. Specifically, the daemon will not
  1058  attempt `push`, `pull` and `login` to v1 registries. The exception to this is
  1059  `search` which can still be performed on v1 registries.
  1060  
  1061  The `disable-legacy-registry` configuration option has been removed and, when
  1062  used, will produce an error on daemon startup.
  1063  
  1064  
  1065  ### Running a Docker daemon behind an HTTPS_PROXY
  1066  
  1067  When running inside a LAN that uses an `HTTPS` proxy, the Docker Hub
  1068  certificates will be replaced by the proxy's certificates. These certificates
  1069  need to be added to your Docker host's configuration:
  1070  
  1071  1. Install the `ca-certificates` package for your distribution
  1072  2. Ask your network admin for the proxy's CA certificate and append them to
  1073     `/etc/pki/tls/certs/ca-bundle.crt`
  1074  3. Then start your Docker daemon with `HTTPS_PROXY=http://username:password@proxy:port/ dockerd`.
  1075     The `username:` and `password@` are optional - and are only needed if your
  1076     proxy is set up to require authentication.
  1077  
  1078  This will only add the proxy and authentication to the Docker daemon's requests -
  1079  your `docker build`s and running containers will need extra configuration to
  1080  use the proxy
  1081  
  1082  ### Default `ulimit` settings
  1083  
  1084  `--default-ulimit` allows you to set the default `ulimit` options to use for
  1085  all containers. It takes the same options as `--ulimit` for `docker run`. If
  1086  these defaults are not set, `ulimit` settings will be inherited, if not set on
  1087  `docker run`, from the Docker daemon. Any `--ulimit` options passed to
  1088  `docker run` will overwrite these defaults.
  1089  
  1090  Be careful setting `nproc` with the `ulimit` flag as `nproc` is designed by Linux to
  1091  set the maximum number of processes available to a user, not to a container. For details
  1092  please check the [run](run.md) reference.
  1093  
  1094  ### Node discovery
  1095  
  1096  The `--cluster-advertise` option specifies the `host:port` or `interface:port`
  1097  combination that this particular daemon instance should use when advertising
  1098  itself to the cluster. The daemon is reached by remote hosts through this value.
  1099  If you  specify an interface, make sure it includes the IP address of the actual
  1100  Docker host. For Engine installation created through `docker-machine`, the
  1101  interface is typically `eth1`.
  1102  
  1103  The daemon uses [libkv](https://github.com/docker/libkv/) to advertise
  1104  the node within the cluster. Some key-value backends support mutual
  1105  TLS. To configure the client TLS settings used by the daemon can be configured
  1106  using the `--cluster-store-opt` flag, specifying the paths to PEM encoded
  1107  files. For example:
  1108  
  1109  ```bash
  1110  $ sudo dockerd \
  1111      --cluster-advertise 192.168.1.2:2376 \
  1112      --cluster-store etcd://192.168.1.2:2379 \
  1113      --cluster-store-opt kv.cacertfile=/path/to/ca.pem \
  1114      --cluster-store-opt kv.certfile=/path/to/cert.pem \
  1115      --cluster-store-opt kv.keyfile=/path/to/key.pem
  1116  ```
  1117  
  1118  The currently supported cluster store options are:
  1119  
  1120  | Option                | Description                                                                                                                                                                                                                   |
  1121  |:----------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
  1122  | `discovery.heartbeat` | Specifies the heartbeat timer in seconds which is used by the daemon as a `keepalive` mechanism to make sure discovery module treats the node as alive in the cluster. If not configured, the default value is 20 seconds.    |
  1123  | `discovery.ttl`       | Specifies the TTL (time-to-live) in seconds which is used by the discovery module to timeout a node if a valid heartbeat is not received within the configured ttl value. If not configured, the default value is 60 seconds. |
  1124  | `kv.cacertfile`       | Specifies the path to a local file with PEM encoded CA certificates to trust.                                                                                                                                                 |
  1125  | `kv.certfile`         | Specifies the path to a local file with a PEM encoded certificate. This certificate is used as the client cert for communication with the Key/Value store.                                                                    |
  1126  | `kv.keyfile`          | Specifies the path to a local file with a PEM encoded private key. This private key is used as the client key for communication with the Key/Value store.                                                                     |
  1127  | `kv.path`             | Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'.                                                                                                                            |
  1128  
  1129  ### Access authorization
  1130  
  1131  Docker's access authorization can be extended by authorization plugins that your
  1132  organization can purchase or build themselves. You can install one or more
  1133  authorization plugins when you start the Docker `daemon` using the
  1134  `--authorization-plugin=PLUGIN_ID` option.
  1135  
  1136  ```bash
  1137  $ sudo dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,...
  1138  ```
  1139  
  1140  The `PLUGIN_ID` value is either the plugin's name or a path to its specification
  1141  file. The plugin's implementation determines whether you can specify a name or
  1142  path. Consult with your Docker administrator to get information about the
  1143  plugins available to you.
  1144  
  1145  Once a plugin is installed, requests made to the `daemon` through the
  1146  command line or Docker's Engine API are allowed or denied by the plugin.
  1147  If you have multiple plugins installed, each plugin, in order, must
  1148  allow the request for it to complete.
  1149  
  1150  For information about how to create an authorization plugin, see [authorization
  1151  plugin](../../extend/plugins_authorization.md) section in the Docker extend section of this documentation.
  1152  
  1153  
  1154  ### Daemon user namespace options
  1155  
  1156  The Linux kernel
  1157  [user namespace support](http://man7.org/linux/man-pages/man7/user_namespaces.7.html)
  1158  provides additional security by enabling a process, and therefore a container,
  1159  to have a unique range of user and group IDs which are outside the traditional
  1160  user and group range utilized by the host system. Potentially the most important
  1161  security improvement is that, by default, container processes running as the
  1162  `root` user will have expected administrative privilege (with some restrictions)
  1163  inside the container but will effectively be mapped to an unprivileged `uid` on
  1164  the host.
  1165  
  1166  For details about how to use this feature, as well as limitations, see
  1167  [Isolate containers with a user namespace](https://docs.docker.com/engine/security/userns-remap/).
  1168  
  1169  ### Miscellaneous options
  1170  
  1171  IP masquerading uses address translation to allow containers without a public
  1172  IP to talk to other machines on the Internet. This may interfere with some
  1173  network topologies and can be disabled with `--ip-masq=false`.
  1174  
  1175  Docker supports softlinks for the Docker data directory (`/var/lib/docker`) and
  1176  for `/var/lib/docker/tmp`. The `DOCKER_TMPDIR` and the data directory can be
  1177  set like this:
  1178  
  1179      DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1
  1180      # or
  1181      export DOCKER_TMPDIR=/mnt/disk2/tmp
  1182      /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1
  1183  
  1184  #### Default cgroup parent
  1185  
  1186  The `--cgroup-parent` option allows you to set the default cgroup parent
  1187  to use for containers. If this option is not set, it defaults to `/docker` for
  1188  fs cgroup driver and `system.slice` for systemd cgroup driver.
  1189  
  1190  If the cgroup has a leading forward slash (`/`), the cgroup is created
  1191  under the root cgroup, otherwise the cgroup is created under the daemon
  1192  cgroup.
  1193  
  1194  Assuming the daemon is running in cgroup `daemoncgroup`,
  1195  `--cgroup-parent=/foobar` creates a cgroup in
  1196  `/sys/fs/cgroup/memory/foobar`, whereas using `--cgroup-parent=foobar`
  1197  creates the cgroup in `/sys/fs/cgroup/memory/daemoncgroup/foobar`
  1198  
  1199  The systemd cgroup driver has different rules for `--cgroup-parent`. Systemd
  1200  represents hierarchy by slice and the name of the slice encodes the location in
  1201  the tree. So `--cgroup-parent` for systemd cgroups should be a slice name. A
  1202  name can consist of a dash-separated series of names, which describes the path
  1203  to the slice from the root slice. For example, `--cgroup-parent=user-a-b.slice`
  1204  means the memory cgroup for the container is created in
  1205  `/sys/fs/cgroup/memory/user.slice/user-a.slice/user-a-b.slice/docker-<id>.scope`.
  1206  
  1207  This setting can also be set per container, using the `--cgroup-parent`
  1208  option on `docker create` and `docker run`, and takes precedence over
  1209  the `--cgroup-parent` option on the daemon.
  1210  
  1211  #### Daemon metrics
  1212  
  1213  The `--metrics-addr` option takes a tcp address to serve the metrics API.
  1214  This feature is still experimental, therefore, the daemon must be running in experimental
  1215  mode for this feature to work.
  1216  
  1217  To serve the metrics API on localhost:1337 you would specify `--metrics-addr 127.0.0.1:1337`
  1218  allowing you to make requests on the API at `127.0.0.1:1337/metrics` to receive metrics in the
  1219  [prometheus](https://prometheus.io/docs/instrumenting/exposition_formats/) format.
  1220  
  1221  If you are running a prometheus server you can add this address to your scrape configs
  1222  to have prometheus collect metrics on Docker.  For more information
  1223  on prometheus you can view the website [here](https://prometheus.io/).
  1224  
  1225  ```none
  1226  scrape_configs:
  1227    - job_name: 'docker'
  1228      static_configs:
  1229        - targets: ['127.0.0.1:1337']
  1230  ```
  1231  
  1232  Please note that this feature is still marked as experimental as metrics and metric
  1233  names could change while this feature is still in experimental.  Please provide
  1234  feedback on what you would like to see collected in the API.
  1235  
  1236  #### Node Generic Resources
  1237  
  1238  The `--node-generic-resources` option takes a list of key-value
  1239  pair (`key=value`) that allows you to advertise user defined resources
  1240  in a swarm cluster.
  1241  
  1242  The current expected use case is to advertise NVIDIA GPUs so that services
  1243  requesting `NVIDIA-GPU=[0-16]` can land on a node that has enough GPUs for
  1244  the task to run.
  1245  
  1246  Example of usage:
  1247  ```json
  1248  {
  1249  	"node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"]
  1250  }
  1251  ```
  1252  
  1253  ### Daemon configuration file
  1254  
  1255  The `--config-file` option allows you to set any configuration option
  1256  for the daemon in a JSON format. This file uses the same flag names as keys,
  1257  except for flags that allow several entries, where it uses the plural
  1258  of the flag name, e.g., `labels` for the `label` flag.
  1259  
  1260  The options set in the configuration file must not conflict with options set
  1261  via flags. The docker daemon fails to start if an option is duplicated between
  1262  the file and the flags, regardless their value. We do this to avoid
  1263  silently ignore changes introduced in configuration reloads.
  1264  For example, the daemon fails to start if you set daemon labels
  1265  in the configuration file and also set daemon labels via the `--label` flag.
  1266  Options that are not present in the file are ignored when the daemon starts.
  1267  
  1268  ##### On Linux
  1269  
  1270  The default location of the configuration file on Linux is
  1271  `/etc/docker/daemon.json`. The `--config-file` flag can be used to specify a
  1272   non-default location.
  1273  
  1274  This is a full example of the allowed configuration options on Linux:
  1275  
  1276  ```json
  1277  {
  1278  	"authorization-plugins": [],
  1279  	"data-root": "",
  1280  	"dns": [],
  1281  	"dns-opts": [],
  1282  	"dns-search": [],
  1283  	"exec-opts": [],
  1284  	"exec-root": "",
  1285  	"experimental": false,
  1286  	"storage-driver": "",
  1287  	"storage-opts": [],
  1288  	"labels": [],
  1289  	"live-restore": true,
  1290  	"log-driver": "",
  1291  	"log-opts": {},
  1292  	"mtu": 0,
  1293  	"pidfile": "",
  1294  	"cluster-store": "",
  1295  	"cluster-store-opts": {},
  1296  	"cluster-advertise": "",
  1297  	"max-concurrent-downloads": 3,
  1298  	"max-concurrent-uploads": 5,
  1299  	"default-shm-size": "64M",
  1300  	"shutdown-timeout": 15,
  1301  	"debug": true,
  1302  	"hosts": [],
  1303  	"log-level": "",
  1304  	"tls": true,
  1305  	"tlsverify": true,
  1306  	"tlscacert": "",
  1307  	"tlscert": "",
  1308  	"tlskey": "",
  1309  	"swarm-default-advertise-addr": "",
  1310  	"api-cors-header": "",
  1311  	"selinux-enabled": false,
  1312  	"userns-remap": "",
  1313  	"group": "",
  1314  	"cgroup-parent": "",
  1315  	"default-ulimits": {},
  1316  	"init": false,
  1317  	"init-path": "/usr/libexec/docker-init",
  1318  	"ipv6": false,
  1319  	"iptables": false,
  1320  	"ip-forward": false,
  1321  	"ip-masq": false,
  1322  	"userland-proxy": false,
  1323  	"userland-proxy-path": "/usr/libexec/docker-proxy",
  1324  	"ip": "0.0.0.0",
  1325  	"bridge": "",
  1326  	"bip": "",
  1327  	"fixed-cidr": "",
  1328  	"fixed-cidr-v6": "",
  1329  	"default-gateway": "",
  1330  	"default-gateway-v6": "",
  1331  	"icc": false,
  1332  	"raw-logs": false,
  1333  	"allow-nondistributable-artifacts": [],
  1334  	"registry-mirrors": [],
  1335  	"seccomp-profile": "",
  1336  	"insecure-registries": [],
  1337  	"no-new-privileges": false,
  1338  	"default-runtime": "runc",
  1339  	"oom-score-adjust": -500,
  1340  	"node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"],
  1341  	"runtimes": {
  1342  		"cc-runtime": {
  1343  			"path": "/usr/bin/cc-runtime"
  1344  		},
  1345  		"custom": {
  1346  			"path": "/usr/local/bin/my-runc-replacement",
  1347  			"runtimeArgs": [
  1348  				"--debug"
  1349  			]
  1350  		}
  1351  	}
  1352  }
  1353  ```
  1354  
  1355  > **Note:** You cannot set options in `daemon.json` that have already been set on
  1356  > daemon startup as a flag.
  1357  > On systems that use `systemd` to start the Docker daemon, `-H` is already set, so
  1358  > you cannot use the `hosts` key in `daemon.json` to add listening addresses.
  1359  > See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for how
  1360  > to accomplish this task with a systemd drop-in file.
  1361  
  1362  ##### On Windows
  1363  
  1364  The default location of the configuration file on Windows is
  1365   `%programdata%\docker\config\daemon.json`. The `--config-file` flag can be
  1366   used to specify a non-default location.
  1367  
  1368  This is a full example of the allowed configuration options on Windows:
  1369  
  1370  ```json
  1371  {
  1372      "authorization-plugins": [],
  1373      "data-root": "",
  1374      "dns": [],
  1375      "dns-opts": [],
  1376      "dns-search": [],
  1377      "exec-opts": [],
  1378      "experimental": false,
  1379      "storage-driver": "",
  1380      "storage-opts": [],
  1381      "labels": [],
  1382      "log-driver": "",
  1383      "mtu": 0,
  1384      "pidfile": "",
  1385      "cluster-store": "",
  1386      "cluster-advertise": "",
  1387      "max-concurrent-downloads": 3,
  1388      "max-concurrent-uploads": 5,
  1389      "shutdown-timeout": 15,
  1390      "debug": true,
  1391      "hosts": [],
  1392      "log-level": "",
  1393      "tlsverify": true,
  1394      "tlscacert": "",
  1395      "tlscert": "",
  1396      "tlskey": "",
  1397      "swarm-default-advertise-addr": "",
  1398      "group": "",
  1399      "default-ulimits": {},
  1400      "bridge": "",
  1401      "fixed-cidr": "",
  1402      "raw-logs": false,
  1403      "allow-nondistributable-artifacts": [],
  1404      "registry-mirrors": [],
  1405      "insecure-registries": []
  1406  }
  1407  ```
  1408  
  1409  #### Configuration reload behavior
  1410  
  1411  Some options can be reconfigured when the daemon is running without requiring
  1412  to restart the process. We use the `SIGHUP` signal in Linux to reload, and a global event
  1413  in Windows with the key `Global\docker-daemon-config-$PID`. The options can
  1414  be modified in the configuration file but still will check for conflicts with
  1415  the provided flags. The daemon fails to reconfigure itself
  1416  if there are conflicts, but it won't stop execution.
  1417  
  1418  The list of currently supported options that can be reconfigured is this:
  1419  
  1420  - `debug`: it changes the daemon to debug mode when set to true.
  1421  - `cluster-store`: it reloads the discovery store with the new address.
  1422  - `cluster-store-opts`: it uses the new options to reload the discovery store.
  1423  - `cluster-advertise`: it modifies the address advertised after reloading.
  1424  - `labels`: it replaces the daemon labels with a new set of labels.
  1425  - `live-restore`: Enables [keeping containers alive during daemon downtime](https://docs.docker.com/engine/admin/live-restore/).
  1426  - `max-concurrent-downloads`: it updates the max concurrent downloads for each pull.
  1427  - `max-concurrent-uploads`: it updates the max concurrent uploads for each push.
  1428  - `default-runtime`: it updates the runtime to be used if not is
  1429    specified at container creation. It defaults to "default" which is
  1430    the runtime shipped with the official docker packages.
  1431  - `runtimes`: it updates the list of available OCI runtimes that can
  1432    be used to run containers
  1433  - `authorization-plugin`: specifies the authorization plugins to use.
  1434  - `allow-nondistributable-artifacts`: Replaces the set of registries to which the daemon will push nondistributable artifacts with a new set of registries.
  1435  - `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config.
  1436  - `registry-mirrors`: it replaces the daemon registry mirrors with a new set of registry mirrors. If some existing registry mirrors in daemon's configuration are not in newly reloaded registry mirrors, these existing ones will be removed from daemon's config.
  1437  
  1438  Updating and reloading the cluster configurations such as `--cluster-store`,
  1439  `--cluster-advertise` and `--cluster-store-opts` will take effect only if
  1440  these configurations were not previously configured. If `--cluster-store`
  1441  has been provided in flags and `cluster-advertise` not, `cluster-advertise`
  1442  can be added in the configuration file without accompanied by `--cluster-store`.
  1443  Configuration reload will log a warning message if it detects a change in
  1444  previously configured cluster configurations.
  1445  
  1446  
  1447  ### Run multiple daemons
  1448  
  1449  > **Note:** Running multiple daemons on a single host is considered as "experimental". The user should be aware of
  1450  > unsolved problems. This solution may not work properly in some cases. Solutions are currently under development
  1451  > and will be delivered in the near future.
  1452  
  1453  This section describes how to run multiple Docker daemons on a single host. To
  1454  run multiple daemons, you must configure each daemon so that it does not
  1455  conflict with other daemons on the same host. You can set these options either
  1456  by providing them as flags, or by using a [daemon configuration file](#daemon-configuration-file).
  1457  
  1458  The following daemon options must be configured for each daemon:
  1459  
  1460  ```none
  1461  -b, --bridge=                          Attach containers to a network bridge
  1462  --exec-root=/var/run/docker            Root of the Docker execdriver
  1463  --data-root=/var/lib/docker            Root of persisted Docker data
  1464  -p, --pidfile=/var/run/docker.pid      Path to use for daemon PID file
  1465  -H, --host=[]                          Daemon socket(s) to connect to
  1466  --iptables=true                        Enable addition of iptables rules
  1467  --config-file=/etc/docker/daemon.json  Daemon configuration file
  1468  --tlscacert="~/.docker/ca.pem"         Trust certs signed only by this CA
  1469  --tlscert="~/.docker/cert.pem"         Path to TLS certificate file
  1470  --tlskey="~/.docker/key.pem"           Path to TLS key file
  1471  ```
  1472  
  1473  When your daemons use different values for these flags, you can run them on the same host without any problems.
  1474  It is very important to properly understand the meaning of those options and to use them correctly.
  1475  
  1476  - The `-b, --bridge=` flag is set to `docker0` as default bridge network. It is created automatically when you install Docker.
  1477  If you are not using the default, you must create and configure the bridge manually or just set it to 'none': `--bridge=none`
  1478  - `--exec-root` is the path where the container state is stored. The default value is `/var/run/docker`. Specify the path for
  1479  your running daemon here.
  1480  - `--data-root` is the path where persisted data such as images, volumes, and
  1481  cluster state are stored. The default value is `/var/lib/docker`. To avoid any
  1482  conflict with other daemons, set this parameter separately for each daemon.
  1483  - `-p, --pidfile=/var/run/docker.pid` is the path where the process ID of the daemon is stored. Specify the path for your
  1484  pid file here.
  1485  - `--host=[]` specifies where the Docker daemon will listen for client connections. If unspecified, it defaults to `/var/run/docker.sock`.
  1486  -  `--iptables=false` prevents the Docker daemon from adding iptables rules. If
  1487  multiple daemons manage iptables rules, they may overwrite rules set by another
  1488  daemon. Be aware that disabling this option requires you to manually add
  1489  iptables rules to expose container ports. If you prevent Docker from adding
  1490  iptables rules, Docker will also not add IP masquerading rules, even if you set
  1491  `--ip-masq` to `true`. Without IP masquerading rules, Docker containers will not be
  1492  able to connect to external hosts or the internet when using network other than
  1493  default bridge.
  1494  - `--config-file=/etc/docker/daemon.json` is the path where configuration file is stored. You can use it instead of
  1495  daemon flags. Specify the path for each daemon.
  1496  - `--tls*` Docker daemon supports `--tlsverify` mode that enforces encrypted and authenticated remote connections.
  1497  The `--tls*` options enable use of specific certificates for individual daemons.
  1498  
  1499  Example script for a separate “bootstrap” instance of the Docker daemon without network:
  1500  
  1501  ```bash
  1502  $ sudo dockerd \
  1503          -H unix:///var/run/docker-bootstrap.sock \
  1504          -p /var/run/docker-bootstrap.pid \
  1505          --iptables=false \
  1506          --ip-masq=false \
  1507          --bridge=none \
  1508          --data-root=/var/lib/docker-bootstrap \
  1509          --exec-root=/var/run/docker-bootstrap
  1510  ```