github.com/cspotcode/docker-cli@v20.10.0-rc1.0.20201201121459-3faad7acc5b8+incompatible/docs/reference/commandline/dockerd.md (about) 1 --- 2 title: "dockerd" 3 description: "The daemon command description and usage" 4 keywords: "container, daemon, runtime" 5 redirect_from: 6 - /engine/reference/commandline/daemon/ 7 --- 8 9 <!-- This file is maintained within the docker/cli GitHub 10 repository at https://github.com/docker/cli/. Make all 11 pull requests against that repo. If you see this file in 12 another repository, consider it read-only there, as it will 13 periodically be overwritten by the definitive file. Pull 14 requests which include edits to this file in other repositories 15 will be rejected. 16 --> 17 18 # daemon 19 20 ```markdown 21 Usage: dockerd COMMAND 22 23 A self-sufficient runtime for containers. 24 25 Options: 26 --add-runtime runtime Register an additional OCI compatible runtime (default []) 27 --allow-nondistributable-artifacts list Allow push of nondistributable artifacts to registry 28 --api-cors-header string Set CORS headers in the Engine API 29 --authorization-plugin list Authorization plugins to load 30 --bip string Specify network bridge IP 31 -b, --bridge string Attach containers to a network bridge 32 --cgroup-parent string Set parent cgroup for all containers 33 --config-file string Daemon configuration file (default "/etc/docker/daemon.json") 34 --containerd string containerd grpc address 35 --containerd-namespace string Containerd namespace to use (default "moby") 36 --containerd-plugins-namespace string Containerd namespace to use for plugins (default "plugins.moby") 37 --cpu-rt-period int Limit the CPU real-time period in microseconds for the 38 parent cgroup for all containers 39 --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds for the 40 parent cgroup for all containers 41 --cri-containerd start containerd with cri 42 --data-root string Root directory of persistent Docker state (default "/var/lib/docker") 43 -D, --debug Enable debug mode 44 --default-address-pool pool-options Default address pools for node specific local networks 45 --default-cgroupns-mode string Default mode for containers cgroup namespace ("host" | "private") (default "host") 46 --default-gateway ip Container default gateway IPv4 address 47 --default-gateway-v6 ip Container default gateway IPv6 address 48 --default-ipc-mode string Default mode for containers ipc ("shareable" | "private") (default "private") 49 --default-runtime string Default OCI runtime for containers (default "runc") 50 --default-shm-size bytes Default shm size for containers (default 64MiB) 51 --default-ulimit ulimit Default ulimits for containers (default []) 52 --dns list DNS server to use 53 --dns-opt list DNS options to use 54 --dns-search list DNS search domains to use 55 --exec-opt list Runtime execution options 56 --exec-root string Root directory for execution state files (default "/var/run/docker") 57 --experimental Enable experimental features 58 --fixed-cidr string IPv4 subnet for fixed IPs 59 --fixed-cidr-v6 string IPv6 subnet for fixed IPs 60 -G, --group string Group for the unix socket (default "docker") 61 --help Print usage 62 -H, --host list Daemon socket(s) to connect to 63 --host-gateway-ip ip IP address that the special 'host-gateway' string in --add-host resolves to. 64 Defaults to the IP address of the default bridge 65 --icc Enable inter-container communication (default true) 66 --init Run an init in the container to forward signals and reap processes 67 --init-path string Path to the docker-init binary 68 --insecure-registry list Enable insecure registry communication 69 --ip ip Default IP when binding container ports (default 0.0.0.0) 70 --ip-forward Enable net.ipv4.ip_forward (default true) 71 --ip-masq Enable IP masquerading (default true) 72 --iptables Enable addition of iptables rules (default true) 73 --ipv6 Enable IPv6 networking 74 --label list Set key=value labels to the daemon 75 --live-restore Enable live restore of docker when containers are still running 76 --log-driver string Default driver for container logs (default "json-file") 77 -l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info") 78 --log-opt map Default log driver options for containers (default map[]) 79 --max-concurrent-downloads int Set the max concurrent downloads for each pull (default 3) 80 --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) 81 --max-download-attempts int Set the max download attempts for each pull (default 5) 82 --metrics-addr string Set default address and port to serve the metrics api on 83 --mtu int Set the containers network MTU 84 --network-control-plane-mtu int Network Control plane MTU (default 1500) 85 --no-new-privileges Set no-new-privileges by default for new containers 86 --node-generic-resource list Advertise user-defined resource 87 --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) 88 -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") 89 --raw-logs Full timestamps without ANSI coloring 90 --registry-mirror list Preferred Docker registry mirror 91 --rootless Enable rootless mode; typically used with RootlessKit 92 --seccomp-profile string Path to seccomp profile 93 --selinux-enabled Enable selinux support 94 --shutdown-timeout int Set the default shutdown timeout (default 15) 95 -s, --storage-driver string Storage driver to use 96 --storage-opt list Storage driver options 97 --swarm-default-advertise-addr string Set default address or interface for swarm advertised address 98 --tls Use TLS; implied by --tlsverify 99 --tlscacert string Trust certs signed only by this CA (default "~/.docker/ca.pem") 100 --tlscert string Path to TLS certificate file (default "~/.docker/cert.pem") 101 --tlskey string Path to TLS key file (default "~/.docker/key.pem") 102 --tlsverify Use TLS and verify the remote 103 --userland-proxy Use userland proxy for loopback traffic (default true) 104 --userland-proxy-path string Path to the userland proxy binary 105 --userns-remap string User/Group setting for user namespaces 106 -v, --version Print version information and quit 107 ``` 108 109 Options with [] may be specified multiple times. 110 111 ## Description 112 113 `dockerd` is the persistent process that manages containers. Docker 114 uses different binaries for the daemon and client. To run the daemon you 115 type `dockerd`. 116 117 To run the daemon with debug output, use `dockerd -D` or add `"debug": true` to 118 the `daemon.json` file. 119 120 > **Enabling experimental features** 121 > 122 > Enable experimental features by starting `dockerd` with the `--experimental` 123 > flag or adding `"experimental": true` to the `daemon.json` file. 124 125 ### Environment variables 126 127 For easy reference, the following list of environment variables are supported 128 by the `dockerd` command line: 129 130 * `DOCKER_DRIVER` The graph driver to use. 131 * `DOCKER_NOWARN_KERNEL_VERSION` Prevent warnings that your Linux kernel is 132 unsuitable for Docker. 133 * `DOCKER_RAMDISK` If set this will disable 'pivot_root'. 134 * `DOCKER_TMPDIR` Location for temporary Docker files. 135 * `MOBY_DISABLE_PIGZ` Do not use [`unpigz`](https://linux.die.net/man/1/pigz) to 136 decompress layers in parallel when pulling images, even if it is installed. 137 138 ## Examples 139 140 ### Daemon socket option 141 142 The Docker daemon can listen for [Docker Engine API](https://docs.docker.com/engine/api/) 143 requests via three different types of Socket: `unix`, `tcp`, and `fd`. 144 145 By default, a `unix` domain socket (or IPC socket) is created at 146 `/var/run/docker.sock`, requiring either `root` permission, or `docker` group 147 membership. 148 149 If you need to access the Docker daemon remotely, you need to enable the `tcp` 150 Socket. Beware that the default setup provides un-encrypted and 151 un-authenticated direct access to the Docker daemon - and should be secured 152 either using the [built in HTTPS encrypted socket](https://docs.docker.com/engine/security/https/), or by 153 putting a secure web proxy in front of it. You can listen on port `2375` on all 154 network interfaces with `-H tcp://0.0.0.0:2375`, or on a particular network 155 interface using its IP address: `-H tcp://192.168.59.103:2375`. It is 156 conventional to use port `2375` for un-encrypted, and port `2376` for encrypted 157 communication with the daemon. 158 159 > **Note** 160 > 161 > If you're using an HTTPS encrypted socket, keep in mind that only 162 > TLS1.0 and greater are supported. Protocols SSLv3 and under are not 163 > supported anymore for security reasons. 164 165 On Systemd based systems, you can communicate with the daemon via 166 [Systemd socket activation](http://0pointer.de/blog/projects/socket-activation.html), 167 use `dockerd -H fd://`. Using `fd://` will work perfectly for most setups but 168 you can also specify individual sockets: `dockerd -H fd://3`. If the 169 specified socket activated files aren't found, then Docker will exit. You can 170 find examples of using Systemd socket activation with Docker and Systemd in the 171 [Docker source tree](https://github.com/docker/docker/tree/master/contrib/init/systemd/). 172 173 You can configure the Docker daemon to listen to multiple sockets at the same 174 time using multiple `-H` options: 175 176 ```bash 177 # listen using the default unix socket, and on 2 specific IP addresses on this host. 178 179 $ sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2 180 ``` 181 182 The Docker client will honor the `DOCKER_HOST` environment variable to set the 183 `-H` flag for the client. Use **one** of the following commands: 184 185 ```bash 186 $ docker -H tcp://0.0.0.0:2375 ps 187 ``` 188 189 ```bash 190 $ export DOCKER_HOST="tcp://0.0.0.0:2375" 191 192 $ docker ps 193 ``` 194 195 Setting the `DOCKER_TLS_VERIFY` environment variable to any value other than 196 the empty string is equivalent to setting the `--tlsverify` flag. The following 197 are equivalent: 198 199 ```bash 200 $ docker --tlsverify ps 201 # or 202 $ export DOCKER_TLS_VERIFY=1 203 $ docker ps 204 ``` 205 206 The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` 207 environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes 208 precedence over `HTTP_PROXY`. 209 210 The Docker client supports connecting to a remote daemon via SSH: 211 212 ``` 213 $ docker -H ssh://me@example.com:22 ps 214 $ docker -H ssh://me@example.com ps 215 $ docker -H ssh://example.com ps 216 ``` 217 218 To use SSH connection, you need to set up `ssh` so that it can reach the 219 remote host with public key authentication. Password authentication is not 220 supported. If your key is protected with passphrase, you need to set up 221 `ssh-agent`. 222 223 #### Bind Docker to another host/port or a Unix socket 224 225 > **Warning** 226 > 227 > Changing the default `docker` daemon binding to a 228 > TCP port or Unix *docker* user group will increase your security risks 229 > by allowing non-root users to gain *root* access on the host. Make sure 230 > you control access to `docker`. If you are binding 231 > to a TCP port, anyone with access to that port has full Docker access; 232 > so it is not advisable on an open network. 233 {: .warning :} 234 235 With `-H` it is possible to make the Docker daemon to listen on a 236 specific IP and port. By default, it will listen on 237 `unix:///var/run/docker.sock` to allow only local connections by the 238 *root* user. You *could* set it to `0.0.0.0:2375` or a specific host IP 239 to give access to everybody, but that is **not recommended** because 240 then it is trivial for someone to gain root access to the host where the 241 daemon is running. 242 243 Similarly, the Docker client can use `-H` to connect to a custom port. 244 The Docker client will default to connecting to `unix:///var/run/docker.sock` 245 on Linux, and `tcp://127.0.0.1:2376` on Windows. 246 247 `-H` accepts host and port assignment in the following format: 248 249 tcp://[host]:[port][path] or unix://path 250 251 For example: 252 253 - `tcp://` -> TCP connection to `127.0.0.1` on either port `2376` when TLS encryption 254 is on, or port `2375` when communication is in plain text. 255 - `tcp://host:2375` -> TCP connection on 256 host:2375 257 - `tcp://host:2375/path` -> TCP connection on 258 host:2375 and prepend path to all requests 259 - `unix://path/to/socket` -> Unix socket located 260 at `path/to/socket` 261 262 `-H`, when empty, will default to the same value as 263 when no `-H` was passed in. 264 265 `-H` also accepts short form for TCP bindings: `host:` or `host:port` or `:port` 266 267 Run Docker in daemon mode: 268 269 ```bash 270 $ sudo <path to>/dockerd -H 0.0.0.0:5555 & 271 ``` 272 273 Download an `ubuntu` image: 274 275 ```bash 276 $ docker -H :5555 pull ubuntu 277 ``` 278 279 You can use multiple `-H`, for example, if you want to listen on both 280 TCP and a Unix socket 281 282 ```bash 283 # Run docker in daemon mode 284 $ sudo <path to>/dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock & 285 # Download an ubuntu image, use default Unix socket 286 $ docker pull ubuntu 287 # OR use the TCP port 288 $ docker -H tcp://127.0.0.1:2375 pull ubuntu 289 ``` 290 291 ### Daemon storage-driver 292 293 On Linux, the Docker daemon has support for several different image layer storage 294 drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay`, `overlay2`, and `fuse-overlayfs`. 295 296 The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that 297 is unlikely to be merged into the main kernel. These are also known to cause 298 some serious kernel crashes. However `aufs` allows containers to share 299 executable and shared library memory, so is a useful choice when running 300 thousands of containers with the same program or libraries. 301 302 The `devicemapper` driver uses thin provisioning and Copy on Write (CoW) 303 snapshots. For each devicemapper graph location – typically 304 `/var/lib/docker/devicemapper` – a thin pool is created based on two block 305 devices, one for data and one for metadata. By default, these block devices 306 are created automatically by using loopback mounts of automatically created 307 sparse files. Refer to [Devicemapper options](#devicemapper-options) below 308 for a way how to customize this setup. 309 [~jpetazzo/Resizing Docker containers with the Device Mapper plugin](http://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/) 310 article explains how to tune your existing setup without the use of options. 311 312 The `btrfs` driver is very fast for `docker build` - but like `devicemapper` 313 does not share executable memory between devices. Use 314 `dockerd -s btrfs -g /mnt/btrfs_partition`. 315 316 The `zfs` driver is probably not as fast as `btrfs` but has a longer track record 317 on stability. Thanks to `Single Copy ARC` shared blocks between clones will be 318 cached only once. Use `dockerd -s zfs`. To select a different zfs filesystem 319 set `zfs.fsname` option as described in [ZFS options](#zfs-options). 320 321 The `overlay` is a very fast union filesystem. It is now merged in the main 322 Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). `overlay` 323 also supports page cache sharing, this means multiple containers accessing 324 the same file can share a single page cache entry (or entries), it makes 325 `overlay` as efficient with memory as `aufs` driver. Call `dockerd -s overlay` 326 to use it. 327 328 The `overlay2` uses the same fast union filesystem but takes advantage of 329 [additional features](https://lkml.org/lkml/2015/2/11/106) added in Linux 330 kernel 4.0 to avoid excessive inode consumption. Call `dockerd -s overlay2` 331 to use it. 332 333 > **Note** 334 > 335 > The `overlay` storage driver can cause excessive inode consumption (especially 336 > as the number of images grows). We recommend using the `overlay2` storage 337 > driver instead. 338 339 340 > **Note** 341 > 342 > Both `overlay` and `overlay2` are currently unsupported on `btrfs` 343 > or any Copy on Write filesystem and should only be used over `ext4` partitions. 344 345 The `fuse-overlayfs` driver is similar to `overlay2` but works in userspace. 346 The `fuse-overlayfs` driver is expected to be used for [Rootless mode](https://docs.docker.com/engine/security/rootless/). 347 348 On Windows, the Docker daemon supports a single image layer storage driver 349 depending on the image platform: `windowsfilter` for Windows images, and 350 `lcow` for Linux containers on Windows. 351 352 ### Options per storage driver 353 354 Particular storage-driver can be configured with options specified with 355 `--storage-opt` flags. Options for `devicemapper` are prefixed with `dm`, 356 options for `zfs` start with `zfs`, options for `btrfs` start with `btrfs` 357 and options for `lcow` start with `lcow`. 358 359 #### Devicemapper options 360 361 This is an example of the configuration file for devicemapper on Linux: 362 363 ```json 364 { 365 "storage-driver": "devicemapper", 366 "storage-opts": [ 367 "dm.thinpooldev=/dev/mapper/thin-pool", 368 "dm.use_deferred_deletion=true", 369 "dm.use_deferred_removal=true" 370 ] 371 } 372 ``` 373 374 ##### `dm.thinpooldev` 375 376 Specifies a custom block storage device to use for the thin pool. 377 378 If using a block device for device mapper storage, it is best to use `lvm` 379 to create and manage the thin-pool volume. This volume is then handed to Docker 380 to exclusively create snapshot volumes needed for images and containers. 381 382 Managing the thin-pool outside of Engine makes for the most feature-rich 383 method of having Docker utilize device mapper thin provisioning as the 384 backing storage for Docker containers. The highlights of the lvm-based 385 thin-pool management feature include: automatic or interactive thin-pool 386 resize support, dynamically changing thin-pool features, automatic thinp 387 metadata checking when lvm activates the thin-pool, etc. 388 389 As a fallback if no thin pool is provided, loopback files are 390 created. Loopback is very slow, but can be used without any 391 pre-configuration of storage. It is strongly recommended that you do 392 not use loopback in production. Ensure your Engine daemon has a 393 `--storage-opt dm.thinpooldev` argument provided. 394 395 ###### Example: 396 397 ```bash 398 $ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool 399 ``` 400 401 ##### `dm.directlvm_device` 402 403 As an alternative to providing a thin pool as above, Docker can setup a block 404 device for you. 405 406 ###### Example: 407 408 ```bash 409 $ sudo dockerd --storage-opt dm.directlvm_device=/dev/xvdf 410 ``` 411 412 ##### `dm.thinp_percent` 413 414 Sets the percentage of passed in block device to use for storage. 415 416 ###### Example: 417 418 ```bash 419 $ sudo dockerd --storage-opt dm.thinp_percent=95 420 ``` 421 422 ##### `dm.thinp_metapercent` 423 424 Sets the percentage of the passed in block device to use for metadata storage. 425 426 ###### Example: 427 428 ```bash 429 $ sudo dockerd --storage-opt dm.thinp_metapercent=1 430 ``` 431 432 ##### `dm.thinp_autoextend_threshold` 433 434 Sets the value of the percentage of space used before `lvm` attempts to 435 autoextend the available space [100 = disabled] 436 437 ###### Example: 438 439 ```bash 440 $ sudo dockerd --storage-opt dm.thinp_autoextend_threshold=80 441 ``` 442 443 ##### `dm.thinp_autoextend_percent` 444 445 Sets the value percentage value to increase the thin pool by when `lvm` 446 attempts to autoextend the available space [100 = disabled] 447 448 ###### Example: 449 450 ```bash 451 $ sudo dockerd --storage-opt dm.thinp_autoextend_percent=20 452 ``` 453 454 455 ##### `dm.basesize` 456 457 Specifies the size to use when creating the base device, which limits the 458 size of images and containers. The default value is 10G. Note, thin devices 459 are inherently "sparse", so a 10G device which is mostly empty doesn't use 460 10 GB of space on the pool. However, the filesystem will use more space for 461 the empty case the larger the device is. 462 463 The base device size can be increased at daemon restart which will allow 464 all future images and containers (based on those new images) to be of the 465 new base device size. 466 467 ###### Examples 468 469 ```bash 470 $ sudo dockerd --storage-opt dm.basesize=50G 471 ``` 472 473 This will increase the base device size to 50G. The Docker daemon will throw an 474 error if existing base device size is larger than 50G. A user can use 475 this option to expand the base device size however shrinking is not permitted. 476 477 This value affects the system-wide "base" empty filesystem 478 that may already be initialized and inherited by pulled images. Typically, 479 a change to this value requires additional steps to take effect: 480 481 ```bash 482 $ sudo service docker stop 483 484 $ sudo rm -rf /var/lib/docker 485 486 $ sudo service docker start 487 ``` 488 489 490 ##### `dm.loopdatasize` 491 492 > **Note** 493 > 494 > This option configures devicemapper loopback, which should not 495 > be used in production. 496 497 Specifies the size to use when creating the loopback file for the 498 "data" device which is used for the thin pool. The default size is 499 100G. The file is sparse, so it will not initially take up this 500 much space. 501 502 ###### Example 503 504 ```bash 505 $ sudo dockerd --storage-opt dm.loopdatasize=200G 506 ``` 507 508 ##### `dm.loopmetadatasize` 509 510 > **Note** 511 > 512 > This option configures devicemapper loopback, which should not 513 > be used in production. 514 515 Specifies the size to use when creating the loopback file for the 516 "metadata" device which is used for the thin pool. The default size 517 is 2G. The file is sparse, so it will not initially take up 518 this much space. 519 520 ###### Example 521 522 ```bash 523 $ sudo dockerd --storage-opt dm.loopmetadatasize=4G 524 ``` 525 526 ##### `dm.fs` 527 528 Specifies the filesystem type to use for the base device. The supported 529 options are "ext4" and "xfs". The default is "xfs" 530 531 ###### Example 532 533 ```bash 534 $ sudo dockerd --storage-opt dm.fs=ext4 535 ``` 536 537 ##### `dm.mkfsarg` 538 539 Specifies extra mkfs arguments to be used when creating the base device. 540 541 ###### Example 542 543 ```bash 544 $ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" 545 ``` 546 547 ##### `dm.mountopt` 548 549 Specifies extra mount options used when mounting the thin devices. 550 551 ###### Example 552 553 ```bash 554 $ sudo dockerd --storage-opt dm.mountopt=nodiscard 555 ``` 556 557 ##### `dm.datadev` 558 559 (Deprecated, use `dm.thinpooldev`) 560 561 Specifies a custom blockdevice to use for data for the thin pool. 562 563 If using a block device for device mapper storage, ideally both `datadev` and 564 `metadatadev` should be specified to completely avoid using the loopback 565 device. 566 567 ###### Example 568 569 ```bash 570 $ sudo dockerd \ 571 --storage-opt dm.datadev=/dev/sdb1 \ 572 --storage-opt dm.metadatadev=/dev/sdc1 573 ``` 574 575 ##### `dm.metadatadev` 576 577 (Deprecated, use `dm.thinpooldev`) 578 579 Specifies a custom blockdevice to use for metadata for the thin pool. 580 581 For best performance the metadata should be on a different spindle than the 582 data, or even better on an SSD. 583 584 If setting up a new metadata pool it is required to be valid. This can be 585 achieved by zeroing the first 4k to indicate empty metadata, like this: 586 587 ```bash 588 $ dd if=/dev/zero of=$metadata_dev bs=4096 count=1 589 ``` 590 591 ###### Example 592 593 ```bash 594 $ sudo dockerd \ 595 --storage-opt dm.datadev=/dev/sdb1 \ 596 --storage-opt dm.metadatadev=/dev/sdc1 597 ``` 598 599 ##### `dm.blocksize` 600 601 Specifies a custom blocksize to use for the thin pool. The default 602 blocksize is 64K. 603 604 ###### Example 605 606 ```bash 607 $ sudo dockerd --storage-opt dm.blocksize=512K 608 ``` 609 610 ##### `dm.blkdiscard` 611 612 Enables or disables the use of `blkdiscard` when removing devicemapper 613 devices. This is enabled by default (only) if using loopback devices and is 614 required to resparsify the loopback file on image/container removal. 615 616 Disabling this on loopback can lead to *much* faster container removal 617 times, but will make the space used in `/var/lib/docker` directory not be 618 returned to the system for other use when containers are removed. 619 620 ###### Examples 621 622 ```bash 623 $ sudo dockerd --storage-opt dm.blkdiscard=false 624 ``` 625 626 ##### `dm.override_udev_sync_check` 627 628 Overrides the `udev` synchronization checks between `devicemapper` and `udev`. 629 `udev` is the device manager for the Linux kernel. 630 631 To view the `udev` sync support of a Docker daemon that is using the 632 `devicemapper` driver, run: 633 634 ```bash 635 $ docker info 636 [...] 637 Udev Sync Supported: true 638 [...] 639 ``` 640 641 When `udev` sync support is `true`, then `devicemapper` and udev can 642 coordinate the activation and deactivation of devices for containers. 643 644 When `udev` sync support is `false`, a race condition occurs between 645 the`devicemapper` and `udev` during create and cleanup. The race condition 646 results in errors and failures. (For information on these failures, see 647 [docker#4036](https://github.com/docker/docker/issues/4036)) 648 649 To allow the `docker` daemon to start, regardless of `udev` sync not being 650 supported, set `dm.override_udev_sync_check` to true: 651 652 ```bash 653 $ sudo dockerd --storage-opt dm.override_udev_sync_check=true 654 ``` 655 656 When this value is `true`, the `devicemapper` continues and simply warns 657 you the errors are happening. 658 659 > **Note** 660 > 661 > The ideal is to pursue a `docker` daemon and environment that does 662 > support synchronizing with `udev`. For further discussion on this 663 > topic, see [docker#4036](https://github.com/docker/docker/issues/4036). 664 > Otherwise, set this flag for migrating existing Docker daemons to 665 > a daemon with a supported environment. 666 667 ##### `dm.use_deferred_removal` 668 669 Enables use of deferred device removal if `libdm` and the kernel driver 670 support the mechanism. 671 672 Deferred device removal means that if device is busy when devices are 673 being removed/deactivated, then a deferred removal is scheduled on 674 device. And devices automatically go away when last user of the device 675 exits. 676 677 For example, when a container exits, its associated thin device is removed. 678 If that device has leaked into some other mount namespace and can't be 679 removed, the container exit still succeeds and this option causes the 680 system to schedule the device for deferred removal. It does not wait in a 681 loop trying to remove a busy device. 682 683 ###### Example 684 685 ```bash 686 $ sudo dockerd --storage-opt dm.use_deferred_removal=true 687 ``` 688 689 ##### `dm.use_deferred_deletion` 690 691 Enables use of deferred device deletion for thin pool devices. By default, 692 thin pool device deletion is synchronous. Before a container is deleted, 693 the Docker daemon removes any associated devices. If the storage driver 694 can not remove a device, the container deletion fails and daemon returns. 695 696 ```console 697 Error deleting container: Error response from daemon: Cannot destroy container 698 ``` 699 700 To avoid this failure, enable both deferred device deletion and deferred 701 device removal on the daemon. 702 703 ```bash 704 $ sudo dockerd \ 705 --storage-opt dm.use_deferred_deletion=true \ 706 --storage-opt dm.use_deferred_removal=true 707 ``` 708 709 With these two options enabled, if a device is busy when the driver is 710 deleting a container, the driver marks the device as deleted. Later, when 711 the device isn't in use, the driver deletes it. 712 713 In general it should be safe to enable this option by default. It will help 714 when unintentional leaking of mount point happens across multiple mount 715 namespaces. 716 717 ##### `dm.min_free_space` 718 719 Specifies the min free space percent in a thin pool require for new device 720 creation to succeed. This check applies to both free data space as well 721 as free metadata space. Valid values are from 0% - 99%. Value 0% disables 722 free space checking logic. If user does not specify a value for this option, 723 the Engine uses a default value of 10%. 724 725 Whenever a new a thin pool device is created (during `docker pull` or during 726 container creation), the Engine checks if the minimum free space is 727 available. If sufficient space is unavailable, then device creation fails 728 and any relevant `docker` operation fails. 729 730 To recover from this error, you must create more free space in the thin pool 731 to recover from the error. You can create free space by deleting some images 732 and containers from the thin pool. You can also add more storage to the thin 733 pool. 734 735 To add more space to a LVM (logical volume management) thin pool, just add 736 more storage to the volume group container thin pool; this should automatically 737 resolve any errors. If your configuration uses loop devices, then stop the 738 Engine daemon, grow the size of loop files and restart the daemon to resolve 739 the issue. 740 741 ###### Example 742 743 ```bash 744 $ sudo dockerd --storage-opt dm.min_free_space=10% 745 ``` 746 747 ##### `dm.xfs_nospace_max_retries` 748 749 Specifies the maximum number of retries XFS should attempt to complete 750 IO when ENOSPC (no space) error is returned by underlying storage device. 751 752 By default XFS retries infinitely for IO to finish and this can result 753 in unkillable process. To change this behavior one can set 754 xfs_nospace_max_retries to say 0 and XFS will not retry IO after getting 755 ENOSPC and will shutdown filesystem. 756 757 ###### Example 758 759 ```bash 760 $ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0 761 ``` 762 763 ##### `dm.libdm_log_level` 764 765 Specifies the maxmimum `libdm` log level that will be forwarded to the 766 `dockerd` log (as specified by `--log-level`). This option is primarily 767 intended for debugging problems involving `libdm`. Using values other than the 768 defaults may cause false-positive warnings to be logged. 769 770 Values specified must fall within the range of valid `libdm` log levels. At the 771 time of writing, the following is the list of `libdm` log levels as well as 772 their corresponding levels when output by `dockerd`. 773 774 | `libdm` Level | Value | `--log-level` | 775 | ------------- | -----:| ------------- | 776 | `_LOG_FATAL` | 2 | error | 777 | `_LOG_ERR` | 3 | error | 778 | `_LOG_WARN` | 4 | warn | 779 | `_LOG_NOTICE` | 5 | info | 780 | `_LOG_INFO` | 6 | info | 781 | `_LOG_DEBUG` | 7 | debug | 782 783 ###### Example 784 785 ```bash 786 $ sudo dockerd \ 787 --log-level debug \ 788 --storage-opt dm.libdm_log_level=7 789 ``` 790 791 #### ZFS options 792 793 ##### `zfs.fsname` 794 795 Set zfs filesystem under which docker will create its own datasets. 796 By default docker will pick up the zfs filesystem where docker graph 797 (`/var/lib/docker`) is located. 798 799 ###### Example 800 801 ```bash 802 $ sudo dockerd -s zfs --storage-opt zfs.fsname=zroot/docker 803 ``` 804 805 #### Btrfs options 806 807 ##### `btrfs.min_space` 808 809 Specifies the minimum size to use when creating the subvolume which is used 810 for containers. If user uses disk quota for btrfs when creating or running 811 a container with **--storage-opt size** option, docker should ensure the 812 **size** cannot be smaller than **btrfs.min_space**. 813 814 ###### Example 815 816 ```bash 817 $ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G 818 ``` 819 820 #### Overlay2 options 821 822 ##### `overlay2.override_kernel_check` 823 824 Overrides the Linux kernel version check allowing overlay2. Support for 825 specifying multiple lower directories needed by overlay2 was added to the 826 Linux kernel in 4.0.0. However, some older kernel versions may be patched 827 to add multiple lower directory support for OverlayFS. This option should 828 only be used after verifying this support exists in the kernel. Applying 829 this option on a kernel without this support will cause failures on mount. 830 831 ##### `overlay2.size` 832 833 Sets the default max size of the container. It is supported only when the 834 backing fs is `xfs` and mounted with `pquota` mount option. Under these 835 conditions the user can pass any size less then the backing fs size. 836 837 ###### Example 838 839 ```bash 840 $ sudo dockerd -s overlay2 --storage-opt overlay2.size=1G 841 ``` 842 843 844 #### Windowsfilter options 845 846 ##### `size` 847 848 Specifies the size to use when creating the sandbox which is used for containers. 849 Defaults to 20G. 850 851 ###### Example 852 853 ```powershell 854 C:\> dockerd --storage-opt size=40G 855 ``` 856 857 #### LCOW (Linux Containers on Windows) options 858 859 ##### `lcow.globalmode` 860 861 Specifies whether the daemon instantiates utility VM instances as required 862 (recommended and default if omitted), or uses single global utility VM (better 863 performance, but has security implications and not recommended for production 864 deployments). 865 866 ###### Example 867 868 ```powershell 869 C:\> dockerd --storage-opt lcow.globalmode=false 870 ``` 871 872 ##### `lcow.kirdpath` 873 874 Specifies the folder path to the location of a pair of kernel and initrd files 875 used for booting a utility VM. Defaults to `%ProgramFiles%\Linux Containers`. 876 877 ###### Example 878 879 ```powershell 880 C:\> dockerd --storage-opt lcow.kirdpath=c:\path\to\files 881 ``` 882 883 ##### `lcow.kernel` 884 885 Specifies the filename of a kernel file located in the `lcow.kirdpath` path. 886 Defaults to `bootx64.efi`. 887 888 ###### Example 889 890 ```powershell 891 C:\> dockerd --storage-opt lcow.kernel=kernel.efi 892 ``` 893 894 ##### `lcow.initrd` 895 896 Specifies the filename of an initrd file located in the `lcow.kirdpath` path. 897 Defaults to `initrd.img`. 898 899 ###### Example 900 901 ```powershell 902 C:\> dockerd --storage-opt lcow.initrd=myinitrd.img 903 ``` 904 905 ##### `lcow.bootparameters` 906 907 Specifies additional boot parameters for booting utility VMs when in kernel/ 908 initrd mode. Ignored if the utility VM is booting from VHD. These settings 909 are kernel specific. 910 911 ###### Example 912 913 ```powershell 914 C:\> dockerd --storage-opt "lcow.bootparameters='option=value'" 915 ``` 916 917 ##### `lcow.vhdx` 918 919 Specifies a custom VHDX to boot a utility VM, as an alternate to kernel 920 and initrd booting. Defaults to `uvm.vhdx` under `lcow.kirdpath`. 921 922 ###### Example 923 924 ```powershell 925 C:\> dockerd --storage-opt lcow.vhdx=custom.vhdx 926 ``` 927 928 ##### `lcow.timeout` 929 930 Specifies the timeout for utility VM operations in seconds. Defaults 931 to 300. 932 933 ###### Example 934 935 ```powershell 936 C:\> dockerd --storage-opt lcow.timeout=240 937 ``` 938 939 ##### `lcow.sandboxsize` 940 941 Specifies the size in GB to use when creating the sandbox which is used for 942 containers. Defaults to 20. Cannot be less than 20. 943 944 ###### Example 945 946 ```powershell 947 C:\> dockerd --storage-opt lcow.sandboxsize=40 948 ``` 949 950 ### Docker runtime execution options 951 952 The Docker daemon relies on a 953 [OCI](https://github.com/opencontainers/runtime-spec) compliant runtime 954 (invoked via the `containerd` daemon) as its interface to the Linux 955 kernel `namespaces`, `cgroups`, and `SELinux`. 956 957 By default, the Docker daemon automatically starts `containerd`. If you want to 958 control `containerd` startup, manually start `containerd` and pass the path to 959 the `containerd` socket using the `--containerd` flag. For example: 960 961 ```bash 962 $ sudo dockerd --containerd /var/run/dev/docker-containerd.sock 963 ``` 964 965 Runtimes can be registered with the daemon either via the 966 configuration file or using the `--add-runtime` command line argument. 967 968 The following is an example adding 2 runtimes via the configuration: 969 970 ```json 971 { 972 "default-runtime": "runc", 973 "runtimes": { 974 "custom": { 975 "path": "/usr/local/bin/my-runc-replacement", 976 "runtimeArgs": [ 977 "--debug" 978 ] 979 }, 980 "runc": { 981 "path": "runc" 982 } 983 } 984 } 985 ``` 986 987 This is the same example via the command line: 988 989 ```bash 990 $ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement 991 ``` 992 993 > **Note** 994 > 995 > Defining runtime arguments via the command line is not supported. 996 997 #### Options for the runtime 998 999 You can configure the runtime using options specified 1000 with the `--exec-opt` flag. All the flag's options have the `native` prefix. A 1001 single `native.cgroupdriver` option is available. 1002 1003 The `native.cgroupdriver` option specifies the management of the container's 1004 cgroups. You can only specify `cgroupfs` or `systemd`. If you specify 1005 `systemd` and it is not available, the system errors out. If you omit the 1006 `native.cgroupdriver` option,` cgroupfs` is used on cgroup v1 hosts, `systemd` 1007 is used on cgroup v2 hosts with systemd available. 1008 1009 This example sets the `cgroupdriver` to `systemd`: 1010 1011 ```bash 1012 $ sudo dockerd --exec-opt native.cgroupdriver=systemd 1013 ``` 1014 1015 Setting this option applies to all containers the daemon launches. 1016 1017 Also Windows Container makes use of `--exec-opt` for special purpose. Docker user 1018 can specify default container isolation technology with this, for example: 1019 1020 ```console 1021 > dockerd --exec-opt isolation=hyperv 1022 ``` 1023 1024 Will make `hyperv` the default isolation technology on Windows. If no isolation 1025 value is specified on daemon start, on Windows client, the default is 1026 `hyperv`, and on Windows server, the default is `process`. 1027 1028 ### Daemon DNS options 1029 1030 To set the DNS server for all Docker containers, use: 1031 1032 ```bash 1033 $ sudo dockerd --dns 8.8.8.8 1034 ``` 1035 1036 To set the DNS search domain for all Docker containers, use: 1037 1038 ```bash 1039 $ sudo dockerd --dns-search example.com 1040 ``` 1041 1042 ### Allow push of nondistributable artifacts 1043 1044 Some images (e.g., Windows base images) contain artifacts whose distribution is 1045 restricted by license. When these images are pushed to a registry, restricted 1046 artifacts are not included. 1047 1048 To override this behavior for specific registries, use the 1049 `--allow-nondistributable-artifacts` option in one of the following forms: 1050 1051 * `--allow-nondistributable-artifacts myregistry:5000` tells the Docker daemon 1052 to push nondistributable artifacts to myregistry:5000. 1053 * `--allow-nondistributable-artifacts 10.1.0.0/16` tells the Docker daemon to 1054 push nondistributable artifacts to all registries whose resolved IP address 1055 is within the subnet described by the CIDR syntax. 1056 1057 This option can be used multiple times. 1058 1059 This option is useful when pushing images containing nondistributable artifacts 1060 to a registry on an air-gapped network so hosts on that network can pull the 1061 images without connecting to another server. 1062 1063 > **Warning**: Nondistributable artifacts typically have restrictions on how 1064 > and where they can be distributed and shared. Only use this feature to push 1065 > artifacts to private registries and ensure that you are in compliance with 1066 > any terms that cover redistributing nondistributable artifacts. 1067 1068 ### Insecure registries 1069 1070 Docker considers a private registry either secure or insecure. In the rest of 1071 this section, *registry* is used for *private registry*, and `myregistry:5000` 1072 is a placeholder example for a private registry. 1073 1074 A secure registry uses TLS and a copy of its CA certificate is placed on the 1075 Docker host at `/etc/docker/certs.d/myregistry:5000/ca.crt`. An insecure 1076 registry is either not using TLS (i.e., listening on plain text HTTP), or is 1077 using TLS with a CA certificate not known by the Docker daemon. The latter can 1078 happen when the certificate was not found under 1079 `/etc/docker/certs.d/myregistry:5000/`, or if the certificate verification 1080 failed (i.e., wrong CA). 1081 1082 By default, Docker assumes all, but local (see local registries below), 1083 registries are secure. Communicating with an insecure registry is not possible 1084 if Docker assumes that registry is secure. In order to communicate with an 1085 insecure registry, the Docker daemon requires `--insecure-registry` in one of 1086 the following two forms: 1087 1088 * `--insecure-registry myregistry:5000` tells the Docker daemon that 1089 myregistry:5000 should be considered insecure. 1090 * `--insecure-registry 10.1.0.0/16` tells the Docker daemon that all registries 1091 whose domain resolve to an IP address is part of the subnet described by the 1092 CIDR syntax, should be considered insecure. 1093 1094 The flag can be used multiple times to allow multiple registries to be marked 1095 as insecure. 1096 1097 If an insecure registry is not marked as insecure, `docker pull`, 1098 `docker push`, and `docker search` will result in an error message prompting 1099 the user to either secure or pass the `--insecure-registry` flag to the Docker 1100 daemon as described above. 1101 1102 Local registries, whose IP address falls in the 127.0.0.0/8 range, are 1103 automatically marked as insecure as of Docker 1.3.2. It is not recommended to 1104 rely on this, as it may change in the future. 1105 1106 Enabling `--insecure-registry`, i.e., allowing un-encrypted and/or untrusted 1107 communication, can be useful when running a local registry. However, 1108 because its use creates security vulnerabilities it should ONLY be enabled for 1109 testing purposes. For increased security, users should add their CA to their 1110 system's list of trusted CAs instead of enabling `--insecure-registry`. 1111 1112 #### Legacy Registries 1113 1114 Operations against registries supporting only the legacy v1 protocol are no longer 1115 supported. Specifically, the daemon will not attempt `push`, `pull` and `login` 1116 to v1 registries. The exception to this is `search` which can still be performed 1117 on v1 registries. 1118 1119 1120 ### Running a Docker daemon behind an HTTPS_PROXY 1121 1122 When running inside a LAN that uses an `HTTPS` proxy, the Docker Hub 1123 certificates will be replaced by the proxy's certificates. These certificates 1124 need to be added to your Docker host's configuration: 1125 1126 1. Install the `ca-certificates` package for your distribution 1127 2. Ask your network admin for the proxy's CA certificate and append them to 1128 `/etc/pki/tls/certs/ca-bundle.crt` 1129 3. Then start your Docker daemon with `HTTPS_PROXY=http://username:password@proxy:port/ dockerd`. 1130 The `username:` and `password@` are optional - and are only needed if your 1131 proxy is set up to require authentication. 1132 1133 This will only add the proxy and authentication to the Docker daemon's requests - 1134 your `docker build`s and running containers will need extra configuration to 1135 use the proxy 1136 1137 ### Default `ulimit` settings 1138 1139 `--default-ulimit` allows you to set the default `ulimit` options to use for 1140 all containers. It takes the same options as `--ulimit` for `docker run`. If 1141 these defaults are not set, `ulimit` settings will be inherited, if not set on 1142 `docker run`, from the Docker daemon. Any `--ulimit` options passed to 1143 `docker run` will overwrite these defaults. 1144 1145 Be careful setting `nproc` with the `ulimit` flag as `nproc` is designed by Linux to 1146 set the maximum number of processes available to a user, not to a container. For details 1147 please check the [run](run.md) reference. 1148 1149 ### Node discovery 1150 1151 The `--cluster-advertise` option specifies the `host:port` or `interface:port` 1152 combination that this particular daemon instance should use when advertising 1153 itself to the cluster. The daemon is reached by remote hosts through this value. 1154 If you specify an interface, make sure it includes the IP address of the actual 1155 Docker host. For Engine installation created through `docker-machine`, the 1156 interface is typically `eth1`. 1157 1158 The daemon uses [libkv](https://github.com/docker/libkv/) to advertise 1159 the node within the cluster. Some key-value backends support mutual 1160 TLS. To configure the client TLS settings used by the daemon can be configured 1161 using the `--cluster-store-opt` flag, specifying the paths to PEM encoded 1162 files. For example: 1163 1164 ```bash 1165 $ sudo dockerd \ 1166 --cluster-advertise 192.168.1.2:2376 \ 1167 --cluster-store etcd://192.168.1.2:2379 \ 1168 --cluster-store-opt kv.cacertfile=/path/to/ca.pem \ 1169 --cluster-store-opt kv.certfile=/path/to/cert.pem \ 1170 --cluster-store-opt kv.keyfile=/path/to/key.pem 1171 ``` 1172 1173 The currently supported cluster store options are: 1174 1175 | Option | Description | 1176 |:----------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 1177 | `discovery.heartbeat` | Specifies the heartbeat timer in seconds which is used by the daemon as a `keepalive` mechanism to make sure discovery module treats the node as alive in the cluster. If not configured, the default value is 20 seconds. | 1178 | `discovery.ttl` | Specifies the TTL (time-to-live) in seconds which is used by the discovery module to timeout a node if a valid heartbeat is not received within the configured ttl value. If not configured, the default value is 60 seconds. | 1179 | `kv.cacertfile` | Specifies the path to a local file with PEM encoded CA certificates to trust. | 1180 | `kv.certfile` | Specifies the path to a local file with a PEM encoded certificate. This certificate is used as the client cert for communication with the Key/Value store. | 1181 | `kv.keyfile` | Specifies the path to a local file with a PEM encoded private key. This private key is used as the client key for communication with the Key/Value store. | 1182 | `kv.path` | Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. | 1183 1184 ### Access authorization 1185 1186 Docker's access authorization can be extended by authorization plugins that your 1187 organization can purchase or build themselves. You can install one or more 1188 authorization plugins when you start the Docker `daemon` using the 1189 `--authorization-plugin=PLUGIN_ID` option. 1190 1191 ```bash 1192 $ sudo dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... 1193 ``` 1194 1195 The `PLUGIN_ID` value is either the plugin's name or a path to its specification 1196 file. The plugin's implementation determines whether you can specify a name or 1197 path. Consult with your Docker administrator to get information about the 1198 plugins available to you. 1199 1200 Once a plugin is installed, requests made to the `daemon` through the 1201 command line or Docker's Engine API are allowed or denied by the plugin. 1202 If you have multiple plugins installed, each plugin, in order, must 1203 allow the request for it to complete. 1204 1205 For information about how to create an authorization plugin, see [authorization 1206 plugin](../../extend/plugins_authorization.md) section in the Docker extend section of this documentation. 1207 1208 1209 ### Daemon user namespace options 1210 1211 The Linux kernel 1212 [user namespace support](http://man7.org/linux/man-pages/man7/user_namespaces.7.html) 1213 provides additional security by enabling a process, and therefore a container, 1214 to have a unique range of user and group IDs which are outside the traditional 1215 user and group range utilized by the host system. Potentially the most important 1216 security improvement is that, by default, container processes running as the 1217 `root` user will have expected administrative privilege (with some restrictions) 1218 inside the container but will effectively be mapped to an unprivileged `uid` on 1219 the host. 1220 1221 For details about how to use this feature, as well as limitations, see 1222 [Isolate containers with a user namespace](https://docs.docker.com/engine/security/userns-remap/). 1223 1224 ### Miscellaneous options 1225 1226 IP masquerading uses address translation to allow containers without a public 1227 IP to talk to other machines on the Internet. This may interfere with some 1228 network topologies and can be disabled with `--ip-masq=false`. 1229 1230 Docker supports softlinks for the Docker data directory (`/var/lib/docker`) and 1231 for `/var/lib/docker/tmp`. The `DOCKER_TMPDIR` and the data directory can be 1232 set like this: 1233 1234 DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 1235 # or 1236 export DOCKER_TMPDIR=/mnt/disk2/tmp 1237 /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 1238 1239 #### Default cgroup parent 1240 1241 The `--cgroup-parent` option allows you to set the default cgroup parent 1242 to use for containers. If this option is not set, it defaults to `/docker` for 1243 fs cgroup driver and `system.slice` for systemd cgroup driver. 1244 1245 If the cgroup has a leading forward slash (`/`), the cgroup is created 1246 under the root cgroup, otherwise the cgroup is created under the daemon 1247 cgroup. 1248 1249 Assuming the daemon is running in cgroup `daemoncgroup`, 1250 `--cgroup-parent=/foobar` creates a cgroup in 1251 `/sys/fs/cgroup/memory/foobar`, whereas using `--cgroup-parent=foobar` 1252 creates the cgroup in `/sys/fs/cgroup/memory/daemoncgroup/foobar` 1253 1254 The systemd cgroup driver has different rules for `--cgroup-parent`. Systemd 1255 represents hierarchy by slice and the name of the slice encodes the location in 1256 the tree. So `--cgroup-parent` for systemd cgroups should be a slice name. A 1257 name can consist of a dash-separated series of names, which describes the path 1258 to the slice from the root slice. For example, `--cgroup-parent=user-a-b.slice` 1259 means the memory cgroup for the container is created in 1260 `/sys/fs/cgroup/memory/user.slice/user-a.slice/user-a-b.slice/docker-<id>.scope`. 1261 1262 This setting can also be set per container, using the `--cgroup-parent` 1263 option on `docker create` and `docker run`, and takes precedence over 1264 the `--cgroup-parent` option on the daemon. 1265 1266 #### Daemon metrics 1267 1268 The `--metrics-addr` option takes a tcp address to serve the metrics API. 1269 This feature is still experimental, therefore, the daemon must be running in experimental 1270 mode for this feature to work. 1271 1272 To serve the metrics API on `localhost:9323` you would specify `--metrics-addr 127.0.0.1:9323`, 1273 allowing you to make requests on the API at `127.0.0.1:9323/metrics` to receive metrics in the 1274 [prometheus](https://prometheus.io/docs/instrumenting/exposition_formats/) format. 1275 1276 Port `9323` is the [default port associated with Docker 1277 metrics](https://github.com/prometheus/prometheus/wiki/Default-port-allocations) 1278 to avoid collisions with other prometheus exporters and services. 1279 1280 If you are running a prometheus server you can add this address to your scrape configs 1281 to have prometheus collect metrics on Docker. For more information 1282 on prometheus refer to the [prometheus website](https://prometheus.io/). 1283 1284 ```yaml 1285 scrape_configs: 1286 - job_name: 'docker' 1287 static_configs: 1288 - targets: ['127.0.0.1:9323'] 1289 ``` 1290 1291 Please note that this feature is still marked as experimental as metrics and metric 1292 names could change while this feature is still in experimental. Please provide 1293 feedback on what you would like to see collected in the API. 1294 1295 #### Node Generic Resources 1296 1297 The `--node-generic-resources` option takes a list of key-value 1298 pair (`key=value`) that allows you to advertise user defined resources 1299 in a swarm cluster. 1300 1301 The current expected use case is to advertise NVIDIA GPUs so that services 1302 requesting `NVIDIA-GPU=[0-16]` can land on a node that has enough GPUs for 1303 the task to run. 1304 1305 Example of usage: 1306 ```json 1307 { 1308 "node-generic-resources": [ 1309 "NVIDIA-GPU=UUID1", 1310 "NVIDIA-GPU=UUID2" 1311 ] 1312 } 1313 ``` 1314 1315 ### Daemon configuration file 1316 1317 The `--config-file` option allows you to set any configuration option 1318 for the daemon in a JSON format. This file uses the same flag names as keys, 1319 except for flags that allow several entries, where it uses the plural 1320 of the flag name, e.g., `labels` for the `label` flag. 1321 1322 The options set in the configuration file must not conflict with options set 1323 via flags. The docker daemon fails to start if an option is duplicated between 1324 the file and the flags, regardless their value. We do this to avoid 1325 silently ignore changes introduced in configuration reloads. 1326 For example, the daemon fails to start if you set daemon labels 1327 in the configuration file and also set daemon labels via the `--label` flag. 1328 Options that are not present in the file are ignored when the daemon starts. 1329 1330 ##### On Linux 1331 1332 The default location of the configuration file on Linux is 1333 `/etc/docker/daemon.json`. The `--config-file` flag can be used to specify a 1334 non-default location. 1335 1336 This is a full example of the allowed configuration options on Linux: 1337 1338 ```json 1339 { 1340 "allow-nondistributable-artifacts": [], 1341 "api-cors-header": "", 1342 "authorization-plugins": [], 1343 "bip": "", 1344 "bridge": "", 1345 "cgroup-parent": "", 1346 "cluster-advertise": "", 1347 "cluster-store": "", 1348 "cluster-store-opts": {}, 1349 "containerd": "/run/containerd/containerd.sock", 1350 "containerd-namespace": "docker", 1351 "containerd-plugin-namespace": "docker-plugins", 1352 "data-root": "", 1353 "debug": true, 1354 "default-address-pools": [ 1355 { 1356 "base": "172.80.0.0/16", 1357 "size": 24 1358 }, 1359 { 1360 "base": "172.90.0.0/16", 1361 "size": 24 1362 } 1363 ], 1364 "default-cgroupns-mode": "private", 1365 "default-gateway": "", 1366 "default-gateway-v6": "", 1367 "default-runtime": "runc", 1368 "default-shm-size": "64M", 1369 "default-ulimits": { 1370 "nofile": { 1371 "Hard": 64000, 1372 "Name": "nofile", 1373 "Soft": 64000 1374 } 1375 }, 1376 "dns": [], 1377 "dns-opts": [], 1378 "dns-search": [], 1379 "exec-opts": [], 1380 "exec-root": "", 1381 "experimental": false, 1382 "features": {}, 1383 "fixed-cidr": "", 1384 "fixed-cidr-v6": "", 1385 "group": "", 1386 "hosts": [], 1387 "icc": false, 1388 "init": false, 1389 "init-path": "/usr/libexec/docker-init", 1390 "insecure-registries": [], 1391 "ip": "0.0.0.0", 1392 "ip-forward": false, 1393 "ip-masq": false, 1394 "iptables": false, 1395 "ipv6": false, 1396 "labels": [], 1397 "live-restore": true, 1398 "log-driver": "json-file", 1399 "log-level": "", 1400 "log-opts": { 1401 "env": "os,customer", 1402 "labels": "somelabel", 1403 "max-file": "5", 1404 "max-size": "10m" 1405 }, 1406 "max-concurrent-downloads": 3, 1407 "max-concurrent-uploads": 5, 1408 "max-download-attempts": 5, 1409 "mtu": 0, 1410 "no-new-privileges": false, 1411 "node-generic-resources": [ 1412 "NVIDIA-GPU=UUID1", 1413 "NVIDIA-GPU=UUID2" 1414 ], 1415 "oom-score-adjust": -500, 1416 "pidfile": "", 1417 "raw-logs": false, 1418 "registry-mirrors": [], 1419 "runtimes": { 1420 "cc-runtime": { 1421 "path": "/usr/bin/cc-runtime" 1422 }, 1423 "custom": { 1424 "path": "/usr/local/bin/my-runc-replacement", 1425 "runtimeArgs": [ 1426 "--debug" 1427 ] 1428 } 1429 }, 1430 "seccomp-profile": "", 1431 "selinux-enabled": false, 1432 "shutdown-timeout": 15, 1433 "storage-driver": "", 1434 "storage-opts": [], 1435 "swarm-default-advertise-addr": "", 1436 "tls": true, 1437 "tlscacert": "", 1438 "tlscert": "", 1439 "tlskey": "", 1440 "tlsverify": true, 1441 "userland-proxy": false, 1442 "userland-proxy-path": "/usr/libexec/docker-proxy", 1443 "userns-remap": "" 1444 } 1445 ``` 1446 1447 > **Note:** 1448 > 1449 > You cannot set options in `daemon.json` that have already been set on 1450 > daemon startup as a flag. 1451 > On systems that use `systemd` to start the Docker daemon, `-H` is already set, so 1452 > you cannot use the `hosts` key in `daemon.json` to add listening addresses. 1453 > See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for how 1454 > to accomplish this task with a systemd drop-in file. 1455 1456 ##### On Windows 1457 1458 The default location of the configuration file on Windows is 1459 `%programdata%\docker\config\daemon.json`. The `--config-file` flag can be 1460 used to specify a non-default location. 1461 1462 This is a full example of the allowed configuration options on Windows: 1463 1464 ```json 1465 { 1466 "allow-nondistributable-artifacts": [], 1467 "authorization-plugins": [], 1468 "bridge": "", 1469 "cluster-advertise": "", 1470 "cluster-store": "", 1471 "containerd": "\\\\.\\pipe\\containerd-containerd", 1472 "containerd-namespace": "docker", 1473 "containerd-plugin-namespace": "docker-plugins", 1474 "data-root": "", 1475 "debug": true, 1476 "default-ulimits": {}, 1477 "dns": [], 1478 "dns-opts": [], 1479 "dns-search": [], 1480 "exec-opts": [], 1481 "experimental": false, 1482 "features": {}, 1483 "fixed-cidr": "", 1484 "group": "", 1485 "hosts": [], 1486 "insecure-registries": [], 1487 "labels": [], 1488 "log-driver": "", 1489 "log-level": "", 1490 "max-concurrent-downloads": 3, 1491 "max-concurrent-uploads": 5, 1492 "max-download-attempts": 5, 1493 "mtu": 0, 1494 "pidfile": "", 1495 "raw-logs": false, 1496 "registry-mirrors": [], 1497 "shutdown-timeout": 15, 1498 "storage-driver": "", 1499 "storage-opts": [], 1500 "swarm-default-advertise-addr": "", 1501 "tlscacert": "", 1502 "tlscert": "", 1503 "tlskey": "", 1504 "tlsverify": true 1505 } 1506 ``` 1507 1508 #### Feature options 1509 The optional field `features` in `daemon.json` allows users to enable or disable specific 1510 daemon features. For example, `{"features":{"buildkit": true}}` enables `buildkit` as the 1511 default docker image builder. 1512 1513 The list of currently supported feature options: 1514 - `buildkit`: It enables `buildkit` as default builder when set to `true` or disables it by 1515 `false`. Note that if this option is not explicitly set in the daemon config file, then it 1516 is up to the cli to determine which builder to invoke. 1517 1518 #### Configuration reload behavior 1519 1520 Some options can be reconfigured when the daemon is running without requiring 1521 to restart the process. We use the `SIGHUP` signal in Linux to reload, and a global event 1522 in Windows with the key `Global\docker-daemon-config-$PID`. The options can 1523 be modified in the configuration file but still will check for conflicts with 1524 the provided flags. The daemon fails to reconfigure itself 1525 if there are conflicts, but it won't stop execution. 1526 1527 The list of currently supported options that can be reconfigured is this: 1528 1529 - `debug`: it changes the daemon to debug mode when set to true. 1530 - `cluster-store`: it reloads the discovery store with the new address. 1531 - `cluster-store-opts`: it uses the new options to reload the discovery store. 1532 - `cluster-advertise`: it modifies the address advertised after reloading. 1533 - `labels`: it replaces the daemon labels with a new set of labels. 1534 - `live-restore`: Enables [keeping containers alive during daemon downtime](https://docs.docker.com/config/containers/live-restore/). 1535 - `max-concurrent-downloads`: it updates the max concurrent downloads for each pull. 1536 - `max-concurrent-uploads`: it updates the max concurrent uploads for each push. 1537 - `max-download-attempts`: it updates the max download attempts for each pull. 1538 - `default-runtime`: it updates the runtime to be used if not is 1539 specified at container creation. It defaults to "default" which is 1540 the runtime shipped with the official docker packages. 1541 - `runtimes`: it updates the list of available OCI runtimes that can 1542 be used to run containers. 1543 - `authorization-plugin`: it specifies the authorization plugins to use. 1544 - `allow-nondistributable-artifacts`: Replaces the set of registries to which the daemon will push nondistributable artifacts with a new set of registries. 1545 - `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config. 1546 - `registry-mirrors`: it replaces the daemon registry mirrors with a new set of registry mirrors. If some existing registry mirrors in daemon's configuration are not in newly reloaded registry mirrors, these existing ones will be removed from daemon's config. 1547 - `shutdown-timeout`: it replaces the daemon's existing configuration timeout with a new timeout for shutting down all containers. 1548 - `features`: it explicitly enables or disables specific features. 1549 1550 Updating and reloading the cluster configurations such as `--cluster-store`, 1551 `--cluster-advertise` and `--cluster-store-opts` will take effect only if 1552 these configurations were not previously configured. If `--cluster-store` 1553 has been provided in flags and `cluster-advertise` not, `cluster-advertise` 1554 can be added in the configuration file without accompanied by `--cluster-store`. 1555 Configuration reload will log a warning message if it detects a change in 1556 previously configured cluster configurations. 1557 1558 1559 ### Run multiple daemons 1560 1561 > **Note:** Running multiple daemons on a single host is considered as "experimental". The user should be aware of 1562 > unsolved problems. This solution may not work properly in some cases. Solutions are currently under development 1563 > and will be delivered in the near future. 1564 1565 This section describes how to run multiple Docker daemons on a single host. To 1566 run multiple daemons, you must configure each daemon so that it does not 1567 conflict with other daemons on the same host. You can set these options either 1568 by providing them as flags, or by using a [daemon configuration file](#daemon-configuration-file). 1569 1570 The following daemon options must be configured for each daemon: 1571 1572 ```console 1573 -b, --bridge= Attach containers to a network bridge 1574 --exec-root=/var/run/docker Root of the Docker execdriver 1575 --data-root=/var/lib/docker Root of persisted Docker data 1576 -p, --pidfile=/var/run/docker.pid Path to use for daemon PID file 1577 -H, --host=[] Daemon socket(s) to connect to 1578 --iptables=true Enable addition of iptables rules 1579 --config-file=/etc/docker/daemon.json Daemon configuration file 1580 --tlscacert="~/.docker/ca.pem" Trust certs signed only by this CA 1581 --tlscert="~/.docker/cert.pem" Path to TLS certificate file 1582 --tlskey="~/.docker/key.pem" Path to TLS key file 1583 ``` 1584 1585 When your daemons use different values for these flags, you can run them on the same host without any problems. 1586 It is very important to properly understand the meaning of those options and to use them correctly. 1587 1588 - The `-b, --bridge=` flag is set to `docker0` as default bridge network. It is created automatically when you install Docker. 1589 If you are not using the default, you must create and configure the bridge manually or just set it to 'none': `--bridge=none` 1590 - `--exec-root` is the path where the container state is stored. The default value is `/var/run/docker`. Specify the path for 1591 your running daemon here. 1592 - `--data-root` is the path where persisted data such as images, volumes, and 1593 cluster state are stored. The default value is `/var/lib/docker`. To avoid any 1594 conflict with other daemons, set this parameter separately for each daemon. 1595 - `-p, --pidfile=/var/run/docker.pid` is the path where the process ID of the daemon is stored. Specify the path for your 1596 pid file here. 1597 - `--host=[]` specifies where the Docker daemon will listen for client connections. If unspecified, it defaults to `/var/run/docker.sock`. 1598 - `--iptables=false` prevents the Docker daemon from adding iptables rules. If 1599 multiple daemons manage iptables rules, they may overwrite rules set by another 1600 daemon. Be aware that disabling this option requires you to manually add 1601 iptables rules to expose container ports. If you prevent Docker from adding 1602 iptables rules, Docker will also not add IP masquerading rules, even if you set 1603 `--ip-masq` to `true`. Without IP masquerading rules, Docker containers will not be 1604 able to connect to external hosts or the internet when using network other than 1605 default bridge. 1606 - `--config-file=/etc/docker/daemon.json` is the path where configuration file is stored. You can use it instead of 1607 daemon flags. Specify the path for each daemon. 1608 - `--tls*` Docker daemon supports `--tlsverify` mode that enforces encrypted and authenticated remote connections. 1609 The `--tls*` options enable use of specific certificates for individual daemons. 1610 1611 Example script for a separate “bootstrap” instance of the Docker daemon without network: 1612 1613 ```bash 1614 $ sudo dockerd \ 1615 -H unix:///var/run/docker-bootstrap.sock \ 1616 -p /var/run/docker-bootstrap.pid \ 1617 --iptables=false \ 1618 --ip-masq=false \ 1619 --bridge=none \ 1620 --data-root=/var/lib/docker-bootstrap \ 1621 --exec-root=/var/run/docker-bootstrap 1622 ```