github.com/sld880311/docker@v0.0.0-20200524143708-d5593973a475/docs/reference/commandline/dockerd.md (about)

     1  ---
     2  title: "dockerd"
     3  aliases: ["/engine/reference/commandline/daemon/"]
     4  description: "The daemon command description and usage"
     5  keywords: "container, daemon, runtime"
     6  ---
     7  
     8  <!-- This file is maintained within the docker/docker Github
     9       repository at https://github.com/docker/docker/. Make all
    10       pull requests against that repo. If you see this file in
    11       another repository, consider it read-only there, as it will
    12       periodically be overwritten by the definitive file. Pull
    13       requests which include edits to this file in other repositories
    14       will be rejected.
    15  -->
    16  
    17  # daemon
    18  
    19  ```markdown
    20  Usage:	dockerd COMMAND
    21  
    22  A self-sufficient runtime for containers.
    23  
    24  Options:
    25        --add-runtime runtime                   Register an additional OCI compatible runtime (default [])
    26        --api-cors-header string                Set CORS headers in the Engine API
    27        --authorization-plugin list             Authorization plugins to load (default [])
    28        --bip string                            Specify network bridge IP
    29    -b, --bridge string                         Attach containers to a network bridge
    30        --cgroup-parent string                  Set parent cgroup for all containers
    31        --cluster-advertise string              Address or interface name to advertise
    32        --cluster-store string                  URL of the distributed storage backend
    33        --cluster-store-opt map                 Set cluster store options (default map[])
    34        --config-file string                    Daemon configuration file (default "/etc/docker/daemon.json")
    35        --containerd string                     Path to containerd socket
    36        --cpu-rt-period int                     Limit the CPU real-time period in microseconds
    37        --cpu-rt-runtime int                    Limit the CPU real-time runtime in microseconds
    38    -D, --debug                                 Enable debug mode
    39        --default-gateway ip                    Container default gateway IPv4 address
    40        --default-gateway-v6 ip                 Container default gateway IPv6 address
    41        --default-runtime string                Default OCI runtime for containers (default "runc")
    42        --default-ulimit ulimit                 Default ulimits for containers (default [])
    43        --disable-legacy-registry               Disable contacting legacy registries
    44        --dns list                              DNS server to use (default [])
    45        --dns-opt list                          DNS options to use (default [])
    46        --dns-search list                       DNS search domains to use (default [])
    47        --exec-opt list                         Runtime execution options (default [])
    48        --exec-root string                      Root directory for execution state files (default "/var/run/docker")
    49        --experimental                          Enable experimental features
    50        --fixed-cidr string                     IPv4 subnet for fixed IPs
    51        --fixed-cidr-v6 string                  IPv6 subnet for fixed IPs
    52    -g, --graph string                          Root of the Docker runtime (default "/var/lib/docker")
    53    -G, --group string                          Group for the unix socket (default "docker")
    54        --help                                  Print usage
    55    -H, --host list                             Daemon socket(s) to connect to (default [])
    56        --icc                                   Enable inter-container communication (default true)
    57        --init                                  Run an init in the container to forward signals and reap processes
    58        --init-path string                      Path to the docker-init binary
    59        --insecure-registry list                Enable insecure registry communication (default [])
    60        --ip ip                                 Default IP when binding container ports (default 0.0.0.0)
    61        --ip-forward                            Enable net.ipv4.ip_forward (default true)
    62        --ip-masq                               Enable IP masquerading (default true)
    63        --iptables                              Enable addition of iptables rules (default true)
    64        --ipv6                                  Enable IPv6 networking
    65        --label list                            Set key=value labels to the daemon (default [])
    66        --live-restore                          Enable live restore of docker when containers are still running
    67        --log-driver string                     Default driver for container logs (default "json-file")
    68    -l, --log-level string                      Set the logging level ("debug", "info", "warn", "error", "fatal") (default "info")
    69        --log-opt map                           Default log driver options for containers (default map[])
    70        --max-concurrent-downloads int          Set the max concurrent downloads for each pull (default 3)
    71        --max-concurrent-uploads int            Set the max concurrent uploads for each push (default 5)
    72        --metrics-addr string                   Set default address and port to serve the metrics api on
    73        --mtu int                               Set the containers network MTU
    74        --oom-score-adjust int                  Set the oom_score_adj for the daemon (default -500)
    75    -p, --pidfile string                        Path to use for daemon PID file (default "/var/run/docker.pid")
    76        --raw-logs                              Full timestamps without ANSI coloring
    77        --registry-mirror list                  Preferred Docker registry mirror (default [])
    78        --seccomp-profile string                Path to seccomp profile
    79        --selinux-enabled                       Enable selinux support
    80        --shutdown-timeout int                  Set the default shutdown timeout (default 15)
    81    -s, --storage-driver string                 Storage driver to use
    82        --storage-opt list                      Storage driver options (default [])
    83        --swarm-default-advertise-addr string   Set default address or interface for swarm advertised address
    84        --tls                                   Use TLS; implied by --tlsverify
    85        --tlscacert string                      Trust certs signed only by this CA (default "~/.docker/ca.pem")
    86        --tlscert string                        Path to TLS certificate file (default "~/.docker/cert.pem")
    87        --tlskey string                         Path to TLS key file (default ~/.docker/key.pem")
    88        --tlsverify                             Use TLS and verify the remote
    89        --userland-proxy                        Use userland proxy for loopback traffic (default true)
    90        --userland-proxy-path string            Path to the userland proxy binary
    91        --userns-remap string                   User/Group setting for user namespaces
    92    -v, --version                               Print version information and quit
    93  ```
    94  
    95  Options with [] may be specified multiple times.
    96  
    97  ## Description
    98  
    99  `dockerd` is the persistent process that manages containers. Docker
   100  uses different binaries for the daemon and client. To run the daemon you
   101  type `dockerd`.
   102  
   103  To run the daemon with debug output, use `dockerd -D` or add `debug: true` to
   104  the `daemon.json` file.
   105  
   106  > **Note**: In Docker 1.13 and higher, enable experimental features by starting
   107  > `dockerd` with the `--experimental` flag or adding `experimental: true` to the
   108  > `daemon.json` file. In earlier Docker versions, a different build was required
   109  > to enable experimental features.
   110  
   111  ## Examples
   112  
   113  ### Daemon socket option
   114  
   115  The Docker daemon can listen for [Docker Engine API](../api/)
   116  requests via three different types of Socket: `unix`, `tcp`, and `fd`.
   117  
   118  By default, a `unix` domain socket (or IPC socket) is created at
   119  `/var/run/docker.sock`, requiring either `root` permission, or `docker` group
   120  membership.
   121  
   122  If you need to access the Docker daemon remotely, you need to enable the `tcp`
   123  Socket. Beware that the default setup provides un-encrypted and
   124  un-authenticated direct access to the Docker daemon - and should be secured
   125  either using the [built in HTTPS encrypted socket](https://docs.docker.com/engine/security/https/), or by
   126  putting a secure web proxy in front of it. You can listen on port `2375` on all
   127  network interfaces with `-H tcp://0.0.0.0:2375`, or on a particular network
   128  interface using its IP address: `-H tcp://192.168.59.103:2375`. It is
   129  conventional to use port `2375` for un-encrypted, and port `2376` for encrypted
   130  communication with the daemon.
   131  
   132  > **Note**: If you're using an HTTPS encrypted socket, keep in mind that only
   133  > TLS1.0 and greater are supported. Protocols SSLv3 and under are not
   134  > supported anymore for security reasons.
   135  
   136  On Systemd based systems, you can communicate with the daemon via
   137  [Systemd socket activation](http://0pointer.de/blog/projects/socket-activation.html),
   138  use `dockerd -H fd://`. Using `fd://` will work perfectly for most setups but
   139  you can also specify individual sockets: `dockerd -H fd://3`. If the
   140  specified socket activated files aren't found, then Docker will exit. You can
   141  find examples of using Systemd socket activation with Docker and Systemd in the
   142  [Docker source tree](https://github.com/docker/docker/tree/master/contrib/init/systemd/).
   143  
   144  You can configure the Docker daemon to listen to multiple sockets at the same
   145  time using multiple `-H` options:
   146  
   147  ```bash
   148  # listen using the default unix socket, and on 2 specific IP addresses on this host.
   149  
   150  $ sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2
   151  ```
   152  
   153  The Docker client will honor the `DOCKER_HOST` environment variable to set the
   154  `-H` flag for the client. Use **one** of the following commands:
   155  
   156  ```bash
   157  $ docker -H tcp://0.0.0.0:2375 ps
   158  ```
   159  
   160  ```bash
   161  $ export DOCKER_HOST="tcp://0.0.0.0:2375"
   162  
   163  $ docker ps
   164  ```
   165  
   166  Setting the `DOCKER_TLS_VERIFY` environment variable to any value other than
   167  the empty string is equivalent to setting the `--tlsverify` flag. The following
   168  are equivalent:
   169  
   170  ```bash
   171  $ docker --tlsverify ps
   172  # or
   173  $ export DOCKER_TLS_VERIFY=1
   174  $ docker ps
   175  ```
   176  
   177  The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`
   178  environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes
   179  precedence over `HTTP_PROXY`.
   180  
   181  #### Bind Docker to another host/port or a Unix socket
   182  
   183  > **Warning**:
   184  > Changing the default `docker` daemon binding to a
   185  > TCP port or Unix *docker* user group will increase your security risks
   186  > by allowing non-root users to gain *root* access on the host. Make sure
   187  > you control access to `docker`. If you are binding
   188  > to a TCP port, anyone with access to that port has full Docker access;
   189  > so it is not advisable on an open network.
   190  
   191  With `-H` it is possible to make the Docker daemon to listen on a
   192  specific IP and port. By default, it will listen on
   193  `unix:///var/run/docker.sock` to allow only local connections by the
   194  *root* user. You *could* set it to `0.0.0.0:2375` or a specific host IP
   195  to give access to everybody, but that is **not recommended** because
   196  then it is trivial for someone to gain root access to the host where the
   197  daemon is running.
   198  
   199  Similarly, the Docker client can use `-H` to connect to a custom port.
   200  The Docker client will default to connecting to `unix:///var/run/docker.sock`
   201  on Linux, and `tcp://127.0.0.1:2376` on Windows.
   202  
   203  `-H` accepts host and port assignment in the following format:
   204  
   205      tcp://[host]:[port][path] or unix://path
   206  
   207  For example:
   208  
   209  -   `tcp://` -> TCP connection to `127.0.0.1` on either port `2376` when TLS encryption
   210      is on, or port `2375` when communication is in plain text.
   211  -   `tcp://host:2375` -> TCP connection on
   212      host:2375
   213  -   `tcp://host:2375/path` -> TCP connection on
   214      host:2375 and prepend path to all requests
   215  -   `unix://path/to/socket` -> Unix socket located
   216      at `path/to/socket`
   217  
   218  `-H`, when empty, will default to the same value as
   219  when no `-H` was passed in.
   220  
   221  `-H` also accepts short form for TCP bindings: `host:` or `host:port` or `:port`
   222  
   223  Run Docker in daemon mode:
   224  
   225  ```bash
   226  $ sudo <path to>/dockerd -H 0.0.0.0:5555 &
   227  ```
   228  
   229  Download an `ubuntu` image:
   230  
   231  ```bash
   232  $ docker -H :5555 pull ubuntu
   233  ```
   234  
   235  You can use multiple `-H`, for example, if you want to listen on both
   236  TCP and a Unix socket
   237  
   238  ```bash
   239  # Run docker in daemon mode
   240  $ sudo <path to>/dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock &
   241  # Download an ubuntu image, use default Unix socket
   242  $ docker pull ubuntu
   243  # OR use the TCP port
   244  $ docker -H tcp://127.0.0.1:2375 pull ubuntu
   245  ```
   246  
   247  ### Daemon storage-driver
   248  
   249  The Docker daemon has support for several different image layer storage
   250  drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay` and `overlay2`.
   251  
   252  The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that
   253  is unlikely to be merged into the main kernel. These are also known to cause
   254  some serious kernel crashes. However `aufs` allows containers to share
   255  executable and shared library memory, so is a useful choice when running
   256  thousands of containers with the same program or libraries.
   257  
   258  The `devicemapper` driver uses thin provisioning and Copy on Write (CoW)
   259  snapshots. For each devicemapper graph location – typically
   260  `/var/lib/docker/devicemapper` – a thin pool is created based on two block
   261  devices, one for data and one for metadata. By default, these block devices
   262  are created automatically by using loopback mounts of automatically created
   263  sparse files. Refer to [Storage driver options](#storage-driver-options) below
   264  for a way how to customize this setup.
   265  [~jpetazzo/Resizing Docker containers with the Device Mapper plugin](http://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/)
   266  article explains how to tune your existing setup without the use of options.
   267  
   268  The `btrfs` driver is very fast for `docker build` - but like `devicemapper`
   269  does not share executable memory between devices. Use
   270  `dockerd -s btrfs -g /mnt/btrfs_partition`.
   271  
   272  The `zfs` driver is probably not as fast as `btrfs` but has a longer track record
   273  on stability. Thanks to `Single Copy ARC` shared blocks between clones will be
   274  cached only once. Use `dockerd -s zfs`. To select a different zfs filesystem
   275  set `zfs.fsname` option as described in [Storage driver options](#storage-driver-options).
   276  
   277  The `overlay` is a very fast union filesystem. It is now merged in the main
   278  Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). `overlay`
   279  also supports page cache sharing, this means multiple containers accessing
   280  the same file can share a single page cache entry (or entries), it makes
   281  `overlay` as efficient with memory as `aufs` driver. Call
   282  `dockerd -s overlay` to use it.
   283  
   284  > **Note**: As promising as `overlay` is, the feature is still quite young and
   285  > should not be used in production. Most notably, using `overlay` can cause
   286  > excessive inode consumption (especially as the number of images grows), as
   287  > well as > being incompatible with the use of RPMs.
   288  
   289  The `overlay2` uses the same fast union filesystem but takes advantage of
   290  [additional features](https://lkml.org/lkml/2015/2/11/106) added in Linux
   291  kernel 4.0 to avoid excessive inode consumption. Call `dockerd -s overlay2`
   292  to use it.
   293  
   294  > **Note**: Both `overlay` and `overlay2` are currently unsupported on `btrfs`
   295  > or any Copy on Write filesystem and should only be used over `ext4` partitions.
   296  
   297  ### Options per storage driver
   298  
   299  Particular storage-driver can be configured with options specified with
   300  `--storage-opt` flags. Options for `devicemapper` are prefixed with `dm`,
   301  options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`.
   302  
   303  #### Devicemapper options
   304  
   305  ##### `dm.thinpooldev`
   306  
   307  Specifies a custom block storage device to use for the thin pool.
   308  
   309  If using a block device for device mapper storage, it is best to use `lvm`
   310  to create and manage the thin-pool volume. This volume is then handed to Docker
   311  to exclusively create snapshot volumes needed for images and containers.
   312  
   313  Managing the thin-pool outside of Engine makes for the most feature-rich
   314  method of having Docker utilize device mapper thin provisioning as the
   315  backing storage for Docker containers. The highlights of the lvm-based
   316  thin-pool management feature include: automatic or interactive thin-pool
   317  resize support, dynamically changing thin-pool features, automatic thinp
   318  metadata checking when lvm activates the thin-pool, etc.
   319  
   320  As a fallback if no thin pool is provided, loopback files are
   321  created. Loopback is very slow, but can be used without any
   322  pre-configuration of storage. It is strongly recommended that you do
   323  not use loopback in production. Ensure your Engine daemon has a
   324  `--storage-opt dm.thinpooldev` argument provided.
   325  
   326  ###### Example:
   327  
   328  ```bash
   329  $ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool
   330  ```
   331  
   332  ##### `dm.basesize`
   333  
   334  Specifies the size to use when creating the base device, which limits the
   335  size of images and containers. The default value is 10G. Note, thin devices
   336  are inherently "sparse", so a 10G device which is mostly empty doesn't use
   337  10 GB of space on the pool. However, the filesystem will use more space for
   338  the empty case the larger the device is.
   339  
   340  The base device size can be increased at daemon restart which will allow
   341  all future images and containers (based on those new images) to be of the
   342  new base device size.
   343  
   344  ###### Examples
   345  
   346  ```bash
   347  $ sudo dockerd --storage-opt dm.basesize=50G
   348  ```
   349  
   350  This will increase the base device size to 50G. The Docker daemon will throw an
   351  error if existing base device size is larger than 50G. A user can use
   352  this option to expand the base device size however shrinking is not permitted.
   353  
   354  This value affects the system-wide "base" empty filesystem
   355  that may already be initialized and inherited by pulled images. Typically,
   356  a change to this value requires additional steps to take effect:
   357  
   358   ```bash
   359  $ sudo service docker stop
   360  
   361  $ sudo rm -rf /var/lib/docker
   362  
   363  $ sudo service docker start
   364  ```
   365  
   366  
   367  ##### `dm.loopdatasize`
   368  
   369  > **Note**: This option configures devicemapper loopback, which should not
   370  > be used in production.
   371  
   372  Specifies the size to use when creating the loopback file for the
   373  "data" device which is used for the thin pool. The default size is
   374  100G. The file is sparse, so it will not initially take up this
   375  much space.
   376  
   377  ###### Example
   378  
   379  ```bash
   380  $ sudo dockerd --storage-opt dm.loopdatasize=200G
   381  ```
   382  
   383  ##### `dm.loopmetadatasize`
   384  
   385  > **Note**: This option configures devicemapper loopback, which should not
   386  > be used in production.
   387  
   388  Specifies the size to use when creating the loopback file for the
   389  "metadata" device which is used for the thin pool. The default size
   390  is 2G. The file is sparse, so it will not initially take up
   391  this much space.
   392  
   393  ###### Example
   394  
   395  ```bash
   396  $ sudo dockerd --storage-opt dm.loopmetadatasize=4G
   397  ```
   398  
   399  ##### `dm.fs`
   400  
   401  Specifies the filesystem type to use for the base device. The supported
   402  options are "ext4" and "xfs". The default is "xfs"
   403  
   404  ###### Example
   405  
   406  ```bash
   407  $ sudo dockerd --storage-opt dm.fs=ext4
   408  ```
   409  
   410  ##### `dm.mkfsarg`
   411  
   412  Specifies extra mkfs arguments to be used when creating the base device.
   413  
   414  ###### Example
   415  
   416  ```bash
   417  $ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal"
   418  ```
   419  
   420  ##### `dm.mountopt`
   421  
   422  Specifies extra mount options used when mounting the thin devices.
   423  
   424  ###### Example
   425  
   426  ```bash
   427  $ sudo dockerd --storage-opt dm.mountopt=nodiscard
   428  ```
   429  
   430  ##### `dm.datadev`
   431  
   432  (Deprecated, use `dm.thinpooldev`)
   433  
   434  Specifies a custom blockdevice to use for data for the thin pool.
   435  
   436  If using a block device for device mapper storage, ideally both `datadev` and
   437  `metadatadev` should be specified to completely avoid using the loopback
   438  device.
   439  
   440  ###### Example
   441  
   442  ```bash
   443  $ sudo dockerd \
   444        --storage-opt dm.datadev=/dev/sdb1 \
   445        --storage-opt dm.metadatadev=/dev/sdc1
   446  ```
   447  
   448  ##### `dm.metadatadev`
   449  
   450  (Deprecated, use `dm.thinpooldev`)
   451  
   452  Specifies a custom blockdevice to use for metadata for the thin pool.
   453  
   454  For best performance the metadata should be on a different spindle than the
   455  data, or even better on an SSD.
   456  
   457  If setting up a new metadata pool it is required to be valid. This can be
   458  achieved by zeroing the first 4k to indicate empty metadata, like this:
   459  
   460  ```bash
   461  $ dd if=/dev/zero of=$metadata_dev bs=4096 count=1
   462  ```
   463  
   464  ###### Example
   465  
   466  ```bash
   467  $ sudo dockerd \
   468        --storage-opt dm.datadev=/dev/sdb1 \
   469        --storage-opt dm.metadatadev=/dev/sdc1
   470  ```
   471  
   472  ##### `dm.blocksize`
   473  
   474  Specifies a custom blocksize to use for the thin pool. The default
   475  blocksize is 64K.
   476  
   477  ###### Example
   478  
   479  ```bash
   480  $ sudo dockerd --storage-opt dm.blocksize=512K
   481  ```
   482  
   483  ##### `dm.blkdiscard`
   484  
   485  Enables or disables the use of `blkdiscard` when removing devicemapper
   486  devices. This is enabled by default (only) if using loopback devices and is
   487  required to resparsify the loopback file on image/container removal.
   488  
   489  Disabling this on loopback can lead to *much* faster container removal
   490  times, but will make the space used in `/var/lib/docker` directory not be
   491  returned to the system for other use when containers are removed.
   492  
   493  ###### Examples
   494  
   495  ```bash
   496  $ sudo dockerd --storage-opt dm.blkdiscard=false
   497  ```
   498  
   499  ##### `dm.override_udev_sync_check`
   500  
   501  Overrides the `udev` synchronization checks between `devicemapper` and `udev`.
   502  `udev` is the device manager for the Linux kernel.
   503  
   504  To view the `udev` sync support of a Docker daemon that is using the
   505  `devicemapper` driver, run:
   506  
   507  ```bash
   508  $ docker info
   509  [...]
   510  Udev Sync Supported: true
   511  [...]
   512  ```
   513  
   514  When `udev` sync support is `true`, then `devicemapper` and udev can
   515  coordinate the activation and deactivation of devices for containers.
   516  
   517  When `udev` sync support is `false`, a race condition occurs between
   518  the`devicemapper` and `udev` during create and cleanup. The race condition
   519  results in errors and failures. (For information on these failures, see
   520  [docker#4036](https://github.com/docker/docker/issues/4036))
   521  
   522  To allow the `docker` daemon to start, regardless of `udev` sync not being
   523  supported, set `dm.override_udev_sync_check` to true:
   524  
   525  ```bash
   526  $ sudo dockerd --storage-opt dm.override_udev_sync_check=true
   527  ```
   528  
   529  When this value is `true`, the  `devicemapper` continues and simply warns
   530  you the errors are happening.
   531  
   532  > **Note**: The ideal is to pursue a `docker` daemon and environment that does
   533  > support synchronizing with `udev`. For further discussion on this
   534  > topic, see [docker#4036](https://github.com/docker/docker/issues/4036).
   535  > Otherwise, set this flag for migrating existing Docker daemons to
   536  > a daemon with a supported environment.
   537  
   538  ##### `dm.use_deferred_removal`
   539  
   540  Enables use of deferred device removal if `libdm` and the kernel driver
   541  support the mechanism.
   542  
   543  Deferred device removal means that if device is busy when devices are
   544  being removed/deactivated, then a deferred removal is scheduled on
   545  device. And devices automatically go away when last user of the device
   546  exits.
   547  
   548  For example, when a container exits, its associated thin device is removed.
   549  If that device has leaked into some other mount namespace and can't be
   550  removed, the container exit still succeeds and this option causes the
   551  system to schedule the device for deferred removal. It does not wait in a
   552  loop trying to remove a busy device.
   553  
   554  ###### Example
   555  
   556  ```bash
   557  $ sudo dockerd --storage-opt dm.use_deferred_removal=true
   558  ```
   559  
   560  ##### `dm.use_deferred_deletion`
   561  
   562  Enables use of deferred device deletion for thin pool devices. By default,
   563  thin pool device deletion is synchronous. Before a container is deleted,
   564  the Docker daemon removes any associated devices. If the storage driver
   565  can not remove a device, the container deletion fails and daemon returns.
   566  
   567  ```none
   568  Error deleting container: Error response from daemon: Cannot destroy container
   569  ```
   570  
   571  To avoid this failure, enable both deferred device deletion and deferred
   572  device removal on the daemon.
   573  
   574  ```bash
   575  $ sudo dockerd \
   576        --storage-opt dm.use_deferred_deletion=true \
   577        --storage-opt dm.use_deferred_removal=true
   578  ```
   579  
   580  With these two options enabled, if a device is busy when the driver is
   581  deleting a container, the driver marks the device as deleted. Later, when
   582  the device isn't in use, the driver deletes it.
   583  
   584  In general it should be safe to enable this option by default. It will help
   585  when unintentional leaking of mount point happens across multiple mount
   586  namespaces.
   587  
   588  ##### `dm.min_free_space`
   589  
   590  Specifies the min free space percent in a thin pool require for new device
   591  creation to succeed. This check applies to both free data space as well
   592  as free metadata space. Valid values are from 0% - 99%. Value 0% disables
   593  free space checking logic. If user does not specify a value for this option,
   594  the Engine uses a default value of 10%.
   595  
   596  Whenever a new a thin pool device is created (during `docker pull` or during
   597  container creation), the Engine checks if the minimum free space is
   598  available. If sufficient space is unavailable, then device creation fails
   599  and any relevant `docker` operation fails.
   600  
   601  To recover from this error, you must create more free space in the thin pool
   602  to recover from the error. You can create free space by deleting some images
   603  and containers from the thin pool. You can also add more storage to the thin
   604  pool.
   605  
   606  To add more space to a LVM (logical volume management) thin pool, just add
   607  more storage to the volume group container thin pool; this should automatically
   608  resolve any errors. If your configuration uses loop devices, then stop the
   609  Engine daemon, grow the size of loop files and restart the daemon to resolve
   610  the issue.
   611  
   612  ###### Example
   613  
   614  ```bash
   615  $ sudo dockerd --storage-opt dm.min_free_space=10%
   616  ```
   617  
   618  ##### `dm.xfs_nospace_max_retries`
   619  
   620  Specifies the maximum number of retries XFS should attempt to complete
   621  IO when ENOSPC (no space) error is returned by underlying storage device.
   622  
   623  By default XFS retries infinitely for IO to finish and this can result
   624  in unkillable process. To change this behavior one can set
   625  xfs_nospace_max_retries to say 0 and XFS will not retry IO after getting
   626  ENOSPC and will shutdown filesystem.
   627  
   628  ###### Example
   629  
   630  ```bash
   631  $ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0
   632  ```
   633  
   634  #### ZFS options
   635  
   636  ##### `zfs.fsname`
   637  
   638  Set zfs filesystem under which docker will create its own datasets.
   639  By default docker will pick up the zfs filesystem where docker graph
   640  (`/var/lib/docker`) is located.
   641  
   642  ###### Example
   643  
   644  ```bash
   645  $ sudo dockerd -s zfs --storage-opt zfs.fsname=zroot/docker
   646  ```
   647  
   648  #### Btrfs options
   649  
   650  ##### `btrfs.min_space`
   651  
   652  Specifies the minimum size to use when creating the subvolume which is used
   653  for containers. If user uses disk quota for btrfs when creating or running
   654  a container with **--storage-opt size** option, docker should ensure the
   655  **size** cannot be smaller than **btrfs.min_space**.
   656  
   657  ###### Example
   658  
   659  ```bash
   660  $ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G
   661  ```
   662  
   663  #### Overlay2 options
   664  
   665  ##### `overlay2.override_kernel_check`
   666  
   667  Overrides the Linux kernel version check allowing overlay2. Support for
   668  specifying multiple lower directories needed by overlay2 was added to the
   669  Linux kernel in 4.0.0. However, some older kernel versions may be patched
   670  to add multiple lower directory support for OverlayFS. This option should
   671  only be used after verifying this support exists in the kernel. Applying
   672  this option on a kernel without this support will cause failures on mount.
   673  
   674  ### Docker runtime execution options
   675  
   676  The Docker daemon relies on a
   677  [OCI](https://github.com/opencontainers/runtime-spec) compliant runtime
   678  (invoked via the `containerd` daemon) as its interface to the Linux
   679  kernel `namespaces`, `cgroups`, and `SELinux`.
   680  
   681  By default, the Docker daemon automatically starts `containerd`. If you want to
   682  control `containerd` startup, manually start `containerd` and pass the path to
   683  the `containerd` socket using the `--containerd` flag. For example:
   684  
   685  ```bash
   686  $ sudo dockerd --containerd /var/run/dev/docker-containerd.sock
   687  ```
   688  
   689  Runtimes can be registered with the daemon either via the
   690  configuration file or using the `--add-runtime` command line argument.
   691  
   692  The following is an example adding 2 runtimes via the configuration:
   693  
   694  ```json
   695  {
   696  	"default-runtime": "runc",
   697  	"runtimes": {
   698  		"runc": {
   699  			"path": "runc"
   700  		},
   701  		"custom": {
   702  			"path": "/usr/local/bin/my-runc-replacement",
   703  			"runtimeArgs": [
   704  				"--debug"
   705  			]
   706  		}
   707  	}
   708  }
   709  ```
   710  
   711  This is the same example via the command line:
   712  
   713  ```bash
   714  $ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement
   715  ```
   716  
   717  > **Note**: Defining runtime arguments via the command line is not supported.
   718  
   719  #### Options for the runtime
   720  
   721  You can configure the runtime using options specified
   722  with the `--exec-opt` flag. All the flag's options have the `native` prefix. A
   723  single `native.cgroupdriver` option is available.
   724  
   725  The `native.cgroupdriver` option specifies the management of the container's
   726  cgroups. You can specify only specify `cgroupfs` or `systemd`. If you specify
   727  `systemd` and it is not available, the system errors out. If you omit the
   728  `native.cgroupdriver` option,` cgroupfs` is used.
   729  
   730  This example sets the `cgroupdriver` to `systemd`:
   731  
   732  ```bash
   733  $ sudo dockerd --exec-opt native.cgroupdriver=systemd
   734  ```
   735  
   736  Setting this option applies to all containers the daemon launches.
   737  
   738  Also Windows Container makes use of `--exec-opt` for special purpose. Docker user
   739  can specify default container isolation technology with this, for example:
   740  
   741  ```bash
   742  $ sudo dockerd --exec-opt isolation=hyperv
   743  ```
   744  
   745  Will make `hyperv` the default isolation technology on Windows. If no isolation
   746  value is specified on daemon start, on Windows client, the default is
   747  `hyperv`, and on Windows server, the default is `process`.
   748  
   749  #### Daemon DNS options
   750  
   751  To set the DNS server for all Docker containers, use:
   752  
   753  ```bash
   754  $ sudo dockerd --dns 8.8.8.8
   755  ```
   756  
   757  
   758  To set the DNS search domain for all Docker containers, use:
   759  
   760  ```bash
   761  $ sudo dockerd --dns-search example.com
   762  ```
   763  
   764  #### Insecure registries
   765  
   766  Docker considers a private registry either secure or insecure. In the rest of
   767  this section, *registry* is used for *private registry*, and `myregistry:5000`
   768  is a placeholder example for a private registry.
   769  
   770  A secure registry uses TLS and a copy of its CA certificate is placed on the
   771  Docker host at `/etc/docker/certs.d/myregistry:5000/ca.crt`. An insecure
   772  registry is either not using TLS (i.e., listening on plain text HTTP), or is
   773  using TLS with a CA certificate not known by the Docker daemon. The latter can
   774  happen when the certificate was not found under
   775  `/etc/docker/certs.d/myregistry:5000/`, or if the certificate verification
   776  failed (i.e., wrong CA).
   777  
   778  By default, Docker assumes all, but local (see local registries below),
   779  registries are secure. Communicating with an insecure registry is not possible
   780  if Docker assumes that registry is secure. In order to communicate with an
   781  insecure registry, the Docker daemon requires `--insecure-registry` in one of
   782  the following two forms:
   783  
   784  * `--insecure-registry myregistry:5000` tells the Docker daemon that
   785    myregistry:5000 should be considered insecure.
   786  * `--insecure-registry 10.1.0.0/16` tells the Docker daemon that all registries
   787    whose domain resolve to an IP address is part of the subnet described by the
   788    CIDR syntax, should be considered insecure.
   789  
   790  The flag can be used multiple times to allow multiple registries to be marked
   791  as insecure.
   792  
   793  If an insecure registry is not marked as insecure, `docker pull`,
   794  `docker push`, and `docker search` will result in an error message prompting
   795  the user to either secure or pass the `--insecure-registry` flag to the Docker
   796  daemon as described above.
   797  
   798  Local registries, whose IP address falls in the 127.0.0.0/8 range, are
   799  automatically marked as insecure as of Docker 1.3.2. It is not recommended to
   800  rely on this, as it may change in the future.
   801  
   802  Enabling `--insecure-registry`, i.e., allowing un-encrypted and/or untrusted
   803  communication, can be useful when running a local registry.  However,
   804  because its use creates security vulnerabilities it should ONLY be enabled for
   805  testing purposes.  For increased security, users should add their CA to their
   806  system's list of trusted CAs instead of enabling `--insecure-registry`.
   807  
   808  ##### Legacy Registries
   809  
   810  Enabling `--disable-legacy-registry` forces a docker daemon to only interact with registries which support the V2 protocol.  Specifically, the daemon will not attempt `push`, `pull` and `login` to v1 registries.  The exception to this is `search` which can still be performed on v1 registries.
   811  
   812  #### Running a Docker daemon behind an HTTPS_PROXY
   813  
   814  When running inside a LAN that uses an `HTTPS` proxy, the Docker Hub
   815  certificates will be replaced by the proxy's certificates. These certificates
   816  need to be added to your Docker host's configuration:
   817  
   818  1. Install the `ca-certificates` package for your distribution
   819  2. Ask your network admin for the proxy's CA certificate and append them to
   820     `/etc/pki/tls/certs/ca-bundle.crt`
   821  3. Then start your Docker daemon with `HTTPS_PROXY=http://username:password@proxy:port/ dockerd`.
   822     The `username:` and `password@` are optional - and are only needed if your
   823     proxy is set up to require authentication.
   824  
   825  This will only add the proxy and authentication to the Docker daemon's requests -
   826  your `docker build`s and running containers will need extra configuration to
   827  use the proxy
   828  
   829  #### Default `ulimit` settings
   830  
   831  `--default-ulimit` allows you to set the default `ulimit` options to use for
   832  all containers. It takes the same options as `--ulimit` for `docker run`. If
   833  these defaults are not set, `ulimit` settings will be inherited, if not set on
   834  `docker run`, from the Docker daemon. Any `--ulimit` options passed to
   835  `docker run` will overwrite these defaults.
   836  
   837  Be careful setting `nproc` with the `ulimit` flag as `nproc` is designed by Linux to
   838  set the maximum number of processes available to a user, not to a container. For details
   839  please check the [run](run.md) reference.
   840  
   841  #### Node discovery
   842  
   843  The `--cluster-advertise` option specifies the `host:port` or `interface:port`
   844  combination that this particular daemon instance should use when advertising
   845  itself to the cluster. The daemon is reached by remote hosts through this value.
   846  If you  specify an interface, make sure it includes the IP address of the actual
   847  Docker host. For Engine installation created through `docker-machine`, the
   848  interface is typically `eth1`.
   849  
   850  The daemon uses [libkv](https://github.com/docker/libkv/) to advertise
   851  the node within the cluster. Some key-value backends support mutual
   852  TLS. To configure the client TLS settings used by the daemon can be configured
   853  using the `--cluster-store-opt` flag, specifying the paths to PEM encoded
   854  files. For example:
   855  
   856  ```bash
   857  $ sudo dockerd \
   858      --cluster-advertise 192.168.1.2:2376 \
   859      --cluster-store etcd://192.168.1.2:2379 \
   860      --cluster-store-opt kv.cacertfile=/path/to/ca.pem \
   861      --cluster-store-opt kv.certfile=/path/to/cert.pem \
   862      --cluster-store-opt kv.keyfile=/path/to/key.pem
   863  ```
   864  
   865  The currently supported cluster store options are:
   866  
   867  | Option                | Description |
   868  |-----------------------|-------------|
   869  | `discovery.heartbeat` | Specifies the heartbeat timer in seconds which is used by the daemon as a `keepalive` mechanism to make sure discovery module treats the node as alive in the cluster. If not configured, the default value is 20 seconds. |
   870  | `discovery.ttl`       | Specifies the TTL (time-to-live) in seconds which is used by the discovery module to timeout a node if a valid heartbeat is not received within the configured ttl value. If not configured, the default value is 60 seconds. |
   871  | `kv.cacertfile`       | Specifies the path to a local file with PEM encoded CA certificates to trust. |
   872  | `kv.certfile`         | Specifies the path to a local file with a PEM encoded certificate. This certificate is used as the client cert for communication with the Key/Value store. |
   873  | `kv.keyfile`          | Specifies the path to a local file with a PEM encoded private key. This private key is used as the client key for communication with the Key/Value store. |
   874  | `kv.path`             | Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. |
   875  
   876  #### Access authorization
   877  
   878  Docker's access authorization can be extended by authorization plugins that your
   879  organization can purchase or build themselves. You can install one or more
   880  authorization plugins when you start the Docker `daemon` using the
   881  `--authorization-plugin=PLUGIN_ID` option.
   882  
   883  ```bash
   884  $ sudo dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,...
   885  ```
   886  
   887  The `PLUGIN_ID` value is either the plugin's name or a path to its specification
   888  file. The plugin's implementation determines whether you can specify a name or
   889  path. Consult with your Docker administrator to get information about the
   890  plugins available to you.
   891  
   892  Once a plugin is installed, requests made to the `daemon` through the command
   893  line or Docker's Engine API are allowed or denied by the plugin.  If you have
   894  multiple plugins installed, at least one must allow the request for it to
   895  complete.
   896  
   897  For information about how to create an authorization plugin, see [authorization
   898  plugin](../../extend/plugins_authorization.md) section in the Docker extend section of this documentation.
   899  
   900  
   901  #### Daemon user namespace options
   902  
   903  The Linux kernel [user namespace support](http://man7.org/linux/man-pages/man7/user_namespaces.7.html) provides additional security by enabling
   904  a process, and therefore a container, to have a unique range of user and
   905  group IDs which are outside the traditional user and group range utilized by
   906  the host system. Potentially the most important security improvement is that,
   907  by default, container processes running as the `root` user will have expected
   908  administrative privilege (with some restrictions) inside the container but will
   909  effectively be mapped to an unprivileged `uid` on the host.
   910  
   911  When user namespace support is enabled, Docker creates a single daemon-wide mapping
   912  for all containers running on the same engine instance. The mappings will
   913  utilize the existing subordinate user and group ID feature available on all modern
   914  Linux distributions.
   915  The [`/etc/subuid`](http://man7.org/linux/man-pages/man5/subuid.5.html) and
   916  [`/etc/subgid`](http://man7.org/linux/man-pages/man5/subgid.5.html) files will be
   917  read for the user, and optional group, specified to the `--userns-remap`
   918  parameter.  If you do not wish to specify your own user and/or group, you can
   919  provide `default` as the value to this flag, and a user will be created on your behalf
   920  and provided subordinate uid and gid ranges. This default user will be named
   921  `dockremap`, and entries will be created for it in `/etc/passwd` and
   922  `/etc/group` using your distro's standard user and group creation tools.
   923  
   924  > **Note**: The single mapping per-daemon restriction is in place for now
   925  > because Docker shares image layers from its local cache across all
   926  > containers running on the engine instance.  Since file ownership must be
   927  > the same for all containers sharing the same layer content, the decision
   928  > was made to map the file ownership on `docker pull` to the daemon's user and
   929  > group mappings so that there is no delay for running containers once the
   930  > content is downloaded. This design preserves the same performance for `docker
   931  > pull`, `docker push`, and container startup as users expect with
   932  > user namespaces disabled.
   933  
   934  ##### Start the daemon with user namespaces enabled
   935  
   936  To enable user namespace support, start the daemon with the
   937  `--userns-remap` flag, which accepts values in the following formats:
   938  
   939   - uid
   940   - uid:gid
   941   - username
   942   - username:groupname
   943  
   944  If numeric IDs are provided, translation back to valid user or group names
   945  will occur so that the subordinate uid and gid information can be read, given
   946  these resources are name-based, not id-based.  If the numeric ID information
   947  provided does not exist as entries in `/etc/passwd` or `/etc/group`, daemon
   948  startup will fail with an error message.
   949  
   950  **Example: starting with default Docker user management:**
   951  
   952  ```bash
   953  $ sudo dockerd --userns-remap=default
   954  ```
   955  
   956  When `default` is provided, Docker will create - or find the existing - user and group
   957  named `dockremap`. If the user is created, and the Linux distribution has
   958  appropriate support, the `/etc/subuid` and `/etc/subgid` files will be populated
   959  with a contiguous 65536 length range of subordinate user and group IDs, starting
   960  at an offset based on prior entries in those files.  For example, Ubuntu will
   961  create the following range, based on an existing user named `user1` already owning
   962  the first 65536 range:
   963  
   964  ```bash
   965  $ cat /etc/subuid
   966  user1:100000:65536
   967  dockremap:165536:65536
   968  ```
   969  
   970  If you have a preferred/self-managed user with subordinate ID mappings already
   971  configured, you can provide that username or uid to the `--userns-remap` flag.
   972  If you have a group that doesn't match the username, you may provide the `gid`
   973  or group name as well; otherwise the username will be used as the group name
   974  when querying the system for the subordinate group ID range.
   975  
   976  The output of `docker info` can be used to determine if the daemon is running
   977  with user namespaces enabled or not. If the daemon is configured with user
   978  namespaces, the Security Options entry in the response will list "userns" as
   979  one of the enabled security features.
   980  
   981  ##### Behavior differences when user namespaces are enabled
   982  
   983  When you start the Docker daemon with `--userns-remap`, Docker segregates the graph directory
   984  where the images are stored by adding an extra directory with a name corresponding to the
   985  remapped UID and GID. For example, if the remapped UID and GID begin with `165536`, all
   986  images and containers running with that remap setting are located in `/var/lib/docker/165536.165536`
   987  instead of `/var/lib/docker/`.
   988  
   989  In addition, the files and directories within the new directory, which correspond to
   990  images and container layers, are also owned by the new UID and GID. To set the ownership
   991  correctly, you need to re-pull the images and restart the containers after starting the
   992  daemon with `--userns-remap`.
   993  
   994  ##### Detailed information on `subuid`/`subgid` ranges
   995  
   996  Given potential advanced use of the subordinate ID ranges by power users, the
   997  following paragraphs define how the Docker daemon currently uses the range entries
   998  found within the subordinate range files.
   999  
  1000  The simplest case is that only one contiguous range is defined for the
  1001  provided user or group. In this case, Docker will use that entire contiguous
  1002  range for the mapping of host uids and gids to the container process.  This
  1003  means that the first ID in the range will be the remapped root user, and the
  1004  IDs above that initial ID will map host ID 1 through the end of the range.
  1005  
  1006  From the example `/etc/subuid` content shown above, the remapped root
  1007  user would be uid 165536.
  1008  
  1009  If the system administrator has set up multiple ranges for a single user or
  1010  group, the Docker daemon will read all the available ranges and use the
  1011  following algorithm to create the mapping ranges:
  1012  
  1013  1. The range segments found for the particular user will be sorted by *start ID* ascending.
  1014  2. Map segments will be created from each range in increasing value with a length matching the length of each segment. Therefore the range segment with the lowest numeric starting value will be equal to the remapped root, and continue up through host uid/gid equal to the range segment length. As an example, if the lowest segment starts at ID 1000 and has a length of 100, then a map of 1000 -> 0 (the remapped root) up through 1100 -> 100 will be created from this segment. If the next segment starts at ID 10000, then the next map will start with mapping 10000 -> 101 up to the length of this second segment. This will continue until no more segments are found in the subordinate files for this user.
  1015  3. If more than five range segments exist for a single user, only the first five will be utilized, matching the kernel's limitation of only five entries in `/proc/self/uid_map` and `proc/self/gid_map`.
  1016  
  1017  ##### Disable user namespace for a container
  1018  
  1019  If you enable user namespaces on the daemon, all containers are started
  1020  with user namespaces enabled. In some situations you might want to disable
  1021  this feature for a container, for example, to start a privileged container (see
  1022  [user namespace known restrictions](#user-namespace-known-restrictions)).
  1023  To enable those advanced features for a specific container use `--userns=host`
  1024  in the `run/exec/create` command.
  1025  This option will completely disable user namespace mapping for the container's user.
  1026  
  1027  ##### User namespace known restrictions
  1028  
  1029  The following standard Docker features are currently incompatible when
  1030  running a Docker daemon with user namespaces enabled:
  1031  
  1032   - sharing PID or NET namespaces with the host (`--pid=host` or `--net=host`)
  1033   - Using `--privileged` mode flag on `docker run` (unless also specifying `--userns=host`)
  1034  
  1035  In general, user namespaces are an advanced feature and will require
  1036  coordination with other capabilities. For example, if volumes are mounted from
  1037  the host, file ownership will have to be pre-arranged if the user or
  1038  administrator wishes the containers to have expected access to the volume
  1039  contents. Note that when using external volume or graph driver plugins, those
  1040  external software programs must be made aware of user and group mapping ranges
  1041  if they are to work seamlessly with user namespace support.
  1042  
  1043  Finally, while the `root` user inside a user namespaced container process has
  1044  many of the expected admin privileges that go along with being the superuser, the
  1045  Linux kernel has restrictions based on internal knowledge that this is a user namespaced
  1046  process. The most notable restriction that we are aware of at this time is the
  1047  inability to use `mknod`. Permission will be denied for device creation even as
  1048  container `root` inside a user namespace.
  1049  
  1050  ### Miscellaneous options
  1051  
  1052  IP masquerading uses address translation to allow containers without a public
  1053  IP to talk to other machines on the Internet. This may interfere with some
  1054  network topologies and can be disabled with `--ip-masq=false`.
  1055  
  1056  Docker supports softlinks for the Docker data directory (`/var/lib/docker`) and
  1057  for `/var/lib/docker/tmp`. The `DOCKER_TMPDIR` and the data directory can be
  1058  set like this:
  1059  
  1060      DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1
  1061      # or
  1062      export DOCKER_TMPDIR=/mnt/disk2/tmp
  1063      /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1
  1064  
  1065  #### Default cgroup parent
  1066  
  1067  The `--cgroup-parent` option allows you to set the default cgroup parent
  1068  to use for containers. If this option is not set, it defaults to `/docker` for
  1069  fs cgroup driver and `system.slice` for systemd cgroup driver.
  1070  
  1071  If the cgroup has a leading forward slash (`/`), the cgroup is created
  1072  under the root cgroup, otherwise the cgroup is created under the daemon
  1073  cgroup.
  1074  
  1075  Assuming the daemon is running in cgroup `daemoncgroup`,
  1076  `--cgroup-parent=/foobar` creates a cgroup in
  1077  `/sys/fs/cgroup/memory/foobar`, whereas using `--cgroup-parent=foobar`
  1078  creates the cgroup in `/sys/fs/cgroup/memory/daemoncgroup/foobar`
  1079  
  1080  The systemd cgroup driver has different rules for `--cgroup-parent`. Systemd
  1081  represents hierarchy by slice and the name of the slice encodes the location in
  1082  the tree. So `--cgroup-parent` for systemd cgroups should be a slice name. A
  1083  name can consist of a dash-separated series of names, which describes the path
  1084  to the slice from the root slice. For example, `--cgroup-parent=user-a-b.slice`
  1085  means the memory cgroup for the container is created in
  1086  `/sys/fs/cgroup/memory/user.slice/user-a.slice/user-a-b.slice/docker-<id>.scope`.
  1087  
  1088  This setting can also be set per container, using the `--cgroup-parent`
  1089  option on `docker create` and `docker run`, and takes precedence over
  1090  the `--cgroup-parent` option on the daemon.
  1091  
  1092  #### Daemon metrics
  1093  
  1094  The `--metrics-addr` option takes a tcp address to serve the metrics API.
  1095  This feature is still experimental, therefore, the daemon must be running in experimental
  1096  mode for this feature to work.
  1097  
  1098  To serve the metrics API on localhost:1337 you would specify `--metrics-addr 127.0.0.1:1337`
  1099  allowing you to make requests on the API at `127.0.0.1:1337/metrics` to receive metrics in the
  1100  [prometheus](https://prometheus.io/docs/instrumenting/exposition_formats/) format.
  1101  
  1102  If you are running a prometheus server you can add this address to your scrape configs
  1103  to have prometheus collect metrics on Docker.  For more information
  1104  on prometheus you can view the website [here](https://prometheus.io/).
  1105  
  1106  ```none
  1107  scrape_configs:
  1108    - job_name: 'docker'
  1109      static_configs:
  1110        - targets: ['127.0.0.1:1337']
  1111  ```
  1112  
  1113  Please note that this feature is still marked as experimental as metrics and metric
  1114  names could change while this feature is still in experimental.  Please provide
  1115  feedback on what you would like to see collected in the API.
  1116  
  1117  #### Daemon configuration file
  1118  
  1119  The `--config-file` option allows you to set any configuration option
  1120  for the daemon in a JSON format. This file uses the same flag names as keys,
  1121  except for flags that allow several entries, where it uses the plural
  1122  of the flag name, e.g., `labels` for the `label` flag.
  1123  
  1124  The options set in the configuration file must not conflict with options set
  1125  via flags. The docker daemon fails to start if an option is duplicated between
  1126  the file and the flags, regardless their value. We do this to avoid
  1127  silently ignore changes introduced in configuration reloads.
  1128  For example, the daemon fails to start if you set daemon labels
  1129  in the configuration file and also set daemon labels via the `--label` flag.
  1130  Options that are not present in the file are ignored when the daemon starts.
  1131  
  1132  ##### On Linux
  1133  
  1134  The default location of the configuration file on Linux is
  1135  `/etc/docker/daemon.json`. The `--config-file` flag can be used to specify a
  1136   non-default location.
  1137  
  1138  This is a full example of the allowed configuration options on Linux:
  1139  
  1140  ```json
  1141  {
  1142  	"authorization-plugins": [],
  1143  	"dns": [],
  1144  	"dns-opts": [],
  1145  	"dns-search": [],
  1146  	"exec-opts": [],
  1147  	"exec-root": "",
  1148  	"experimental": false,
  1149  	"storage-driver": "",
  1150  	"storage-opts": [],
  1151  	"labels": [],
  1152  	"live-restore": true,
  1153  	"log-driver": "",
  1154  	"log-opts": {},
  1155  	"mtu": 0,
  1156  	"pidfile": "",
  1157  	"graph": "",
  1158  	"cluster-store": "",
  1159  	"cluster-store-opts": {},
  1160  	"cluster-advertise": "",
  1161  	"max-concurrent-downloads": 3,
  1162  	"max-concurrent-uploads": 5,
  1163  	"shutdown-timeout": 15,
  1164  	"debug": true,
  1165  	"hosts": [],
  1166  	"log-level": "",
  1167  	"tls": true,
  1168  	"tlsverify": true,
  1169  	"tlscacert": "",
  1170  	"tlscert": "",
  1171  	"tlskey": "",
  1172  	"swarm-default-advertise-addr": "",
  1173  	"api-cors-header": "",
  1174  	"selinux-enabled": false,
  1175  	"userns-remap": "",
  1176  	"group": "",
  1177  	"cgroup-parent": "",
  1178  	"default-ulimits": {},
  1179  	"init": false,
  1180  	"init-path": "/usr/libexec/docker-init",
  1181  	"ipv6": false,
  1182  	"iptables": false,
  1183  	"ip-forward": false,
  1184  	"ip-masq": false,
  1185  	"userland-proxy": false,
  1186  	"userland-proxy-path": "/usr/libexec/docker-proxy",
  1187  	"ip": "0.0.0.0",
  1188  	"bridge": "",
  1189  	"bip": "",
  1190  	"fixed-cidr": "",
  1191  	"fixed-cidr-v6": "",
  1192  	"default-gateway": "",
  1193  	"default-gateway-v6": "",
  1194  	"icc": false,
  1195  	"raw-logs": false,
  1196  	"registry-mirrors": [],
  1197  	"seccomp-profile": "",
  1198  	"insecure-registries": [],
  1199  	"disable-legacy-registry": false,
  1200  	"default-runtime": "runc",
  1201  	"oom-score-adjust": -500,
  1202  	"runtimes": {
  1203  		"runc": {
  1204  			"path": "runc"
  1205  		},
  1206  		"custom": {
  1207  			"path": "/usr/local/bin/my-runc-replacement",
  1208  			"runtimeArgs": [
  1209  				"--debug"
  1210  			]
  1211  		}
  1212  	}
  1213  }
  1214  ```
  1215  
  1216  > **Note:** You cannot set options in `daemon.json` that have already been set on
  1217  > daemon startup as a flag.
  1218  > On systems that use `systemd` to start the Docker daemon, `-H` is already set, so
  1219  > you cannot use the `hosts` key in `daemon.json` to add listening addresses.
  1220  > See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for how
  1221  > to accomplish this task with a systemd drop-in file.
  1222  
  1223  ##### On Windows
  1224  
  1225  The default location of the configuration file on Windows is
  1226   `%programdata%\docker\config\daemon.json`. The `--config-file` flag can be
  1227   used to specify a non-default location.
  1228  
  1229  This is a full example of the allowed configuration options on Windows:
  1230  
  1231  ```json
  1232  {
  1233      "authorization-plugins": [],
  1234      "dns": [],
  1235      "dns-opts": [],
  1236      "dns-search": [],
  1237      "exec-opts": [],
  1238      "experimental": false,
  1239      "storage-driver": "",
  1240      "storage-opts": [],
  1241      "labels": [],
  1242      "log-driver": "",
  1243      "mtu": 0,
  1244      "pidfile": "",
  1245      "graph": "",
  1246      "cluster-store": "",
  1247      "cluster-advertise": "",
  1248      "max-concurrent-downloads": 3,
  1249      "max-concurrent-uploads": 5,
  1250      "shutdown-timeout": 15,
  1251      "debug": true,
  1252      "hosts": [],
  1253      "log-level": "",
  1254      "tlsverify": true,
  1255      "tlscacert": "",
  1256      "tlscert": "",
  1257      "tlskey": "",
  1258      "swarm-default-advertise-addr": "",
  1259      "group": "",
  1260      "default-ulimits": {},
  1261      "bridge": "",
  1262      "fixed-cidr": "",
  1263      "raw-logs": false,
  1264      "registry-mirrors": [],
  1265      "insecure-registries": [],
  1266      "disable-legacy-registry": false
  1267  }
  1268  ```
  1269  
  1270  #### Configuration reload behavior
  1271  
  1272  Some options can be reconfigured when the daemon is running without requiring
  1273  to restart the process. We use the `SIGHUP` signal in Linux to reload, and a global event
  1274  in Windows with the key `Global\docker-daemon-config-$PID`. The options can
  1275  be modified in the configuration file but still will check for conflicts with
  1276  the provided flags. The daemon fails to reconfigure itself
  1277  if there are conflicts, but it won't stop execution.
  1278  
  1279  The list of currently supported options that can be reconfigured is this:
  1280  
  1281  - `debug`: it changes the daemon to debug mode when set to true.
  1282  - `cluster-store`: it reloads the discovery store with the new address.
  1283  - `cluster-store-opts`: it uses the new options to reload the discovery store.
  1284  - `cluster-advertise`: it modifies the address advertised after reloading.
  1285  - `labels`: it replaces the daemon labels with a new set of labels.
  1286  - `live-restore`: Enables [keeping containers alive during daemon downtime](https://docs.docker.com/engine/admin/live-restore/).
  1287  - `max-concurrent-downloads`: it updates the max concurrent downloads for each pull.
  1288  - `max-concurrent-uploads`: it updates the max concurrent uploads for each push.
  1289  - `default-runtime`: it updates the runtime to be used if not is
  1290    specified at container creation. It defaults to "default" which is
  1291    the runtime shipped with the official docker packages.
  1292  - `runtimes`: it updates the list of available OCI runtimes that can
  1293    be used to run containers
  1294  - `authorization-plugin`: specifies the authorization plugins to use.
  1295  - `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config.
  1296  
  1297  Updating and reloading the cluster configurations such as `--cluster-store`,
  1298  `--cluster-advertise` and `--cluster-store-opts` will take effect only if
  1299  these configurations were not previously configured. If `--cluster-store`
  1300  has been provided in flags and `cluster-advertise` not, `cluster-advertise`
  1301  can be added in the configuration file without accompanied by `--cluster-store`.
  1302  Configuration reload will log a warning message if it detects a change in
  1303  previously configured cluster configurations.
  1304  
  1305  
  1306  ### Run multiple daemons
  1307  
  1308  > **Note:** Running multiple daemons on a single host is considered as "experimental". The user should be aware of
  1309  > unsolved problems. This solution may not work properly in some cases. Solutions are currently under development
  1310  > and will be delivered in the near future.
  1311  
  1312  This section describes how to run multiple Docker daemons on a single host. To
  1313  run multiple daemons, you must configure each daemon so that it does not
  1314  conflict with other daemons on the same host. You can set these options either
  1315  by providing them as flags, or by using a [daemon configuration file](#daemon-configuration-file).
  1316  
  1317  The following daemon options must be configured for each daemon:
  1318  
  1319  ```none
  1320  -b, --bridge=                          Attach containers to a network bridge
  1321  --exec-root=/var/run/docker            Root of the Docker execdriver
  1322  -g, --graph=/var/lib/docker            Root of the Docker runtime
  1323  -p, --pidfile=/var/run/docker.pid      Path to use for daemon PID file
  1324  -H, --host=[]                          Daemon socket(s) to connect to
  1325  --iptables=true                        Enable addition of iptables rules
  1326  --config-file=/etc/docker/daemon.json  Daemon configuration file
  1327  --tlscacert="~/.docker/ca.pem"         Trust certs signed only by this CA
  1328  --tlscert="~/.docker/cert.pem"         Path to TLS certificate file
  1329  --tlskey="~/.docker/key.pem"           Path to TLS key file
  1330  ```
  1331  
  1332  When your daemons use different values for these flags, you can run them on the same host without any problems.
  1333  It is very important to properly understand the meaning of those options and to use them correctly.
  1334  
  1335  - The `-b, --bridge=` flag is set to `docker0` as default bridge network. It is created automatically when you install Docker.
  1336  If you are not using the default, you must create and configure the bridge manually or just set it to 'none': `--bridge=none`
  1337  - `--exec-root` is the path where the container state is stored. The default value is `/var/run/docker`. Specify the path for
  1338  your running daemon here.
  1339  - `--graph` is the path where images are stored. The default value is `/var/lib/docker`. To avoid any conflict with other daemons
  1340  set this parameter separately for each daemon.
  1341  - `-p, --pidfile=/var/run/docker.pid` is the path where the process ID of the daemon is stored. Specify the path for your
  1342  pid file here.
  1343  - `--host=[]` specifies where the Docker daemon will listen for client connections. If unspecified, it defaults to `/var/run/docker.sock`.
  1344  -  `--iptables=false` prevents the Docker daemon from adding iptables rules. If
  1345  multiple daemons manage iptables rules, they may overwrite rules set by another
  1346  daemon. Be aware that disabling this option requires you to manually add
  1347  iptables rules to expose container ports. If you prevent Docker from adding
  1348  iptables rules, Docker will also not add IP masquerading rules, even if you set
  1349  `--ip-masq` to `true`. Without IP masquerading rules, Docker containers will not be
  1350  able to connect to external hosts or the internet when using network other than
  1351  default bridge.
  1352  - `--config-file=/etc/docker/daemon.json` is the path where configuration file is stored. You can use it instead of
  1353  daemon flags. Specify the path for each daemon.
  1354  - `--tls*` Docker daemon supports `--tlsverify` mode that enforces encrypted and authenticated remote connections.
  1355  The `--tls*` options enable use of specific certificates for individual daemons.
  1356  
  1357  Example script for a separate “bootstrap” instance of the Docker daemon without network:
  1358  
  1359  ```bash
  1360  $ sudo dockerd \
  1361          -H unix:///var/run/docker-bootstrap.sock \
  1362          -p /var/run/docker-bootstrap.pid \
  1363          --iptables=false \
  1364          --ip-masq=false \
  1365          --bridge=none \
  1366          --graph=/var/lib/docker-bootstrap \
  1367          --exec-root=/var/run/docker-bootstrap
  1368  ```