github.com/adityamillind98/nomad@v0.11.8/website/pages/docs/drivers/docker.mdx (about) 1 --- 2 layout: docs 3 page_title: 'Drivers: Docker' 4 sidebar_title: Docker 5 description: The Docker task driver is used to run Docker based tasks. 6 --- 7 8 # Docker Driver 9 10 Name: `docker` 11 12 The `docker` driver provides a first-class Docker workflow on Nomad. The Docker 13 driver handles downloading containers, mapping ports, and starting, watching, 14 and cleaning up after containers. 15 16 ## Task Configuration 17 18 ```hcl 19 task "webservice" { 20 driver = "docker" 21 22 config { 23 image = "redis:3.2" 24 labels { 25 group = "webservice-cache" 26 } 27 } 28 } 29 ``` 30 31 The `docker` driver supports the following configuration in the job spec. Only 32 `image` is required. 33 34 - `image` - The Docker image to run. The image may include a tag or custom URL 35 and should include `https://` if required. By default it will be fetched from 36 Docker Hub. If the tag is omitted or equal to `latest` the driver will always 37 try to pull the image. If the image to be pulled exists in a registry that 38 requires authentication credentials must be provided to Nomad. Please see the 39 [Authentication section](#authentication). 40 41 ```hcl 42 config { 43 image = "https://hub.docker.internal/redis:3.2" 44 } 45 ``` 46 47 - `args` - (Optional) A list of arguments to the optional `command`. If no 48 `command` is specified, the arguments are passed directly to the container. 49 References to environment variables or any [interpretable Nomad 50 variables](/docs/runtime/interpolation) will be interpreted before 51 launching the task. For example: 52 53 ```hcl 54 config { 55 args = [ 56 "-bind", "${NOMAD_PORT_http}", 57 "${nomad.datacenter}", 58 "${MY_ENV}", 59 "${meta.foo}", 60 ] 61 } 62 ``` 63 64 - `auth` - (Optional) Provide authentication for a private registry (see below). 65 66 - `auth_soft_fail` `(bool: false)` - Don't fail the task on an auth failure. 67 Attempt to continue without auth. 68 69 - `command` - (Optional) The command to run when starting the container. 70 71 ```hcl 72 config { 73 command = "my-command" 74 } 75 ``` 76 77 - `dns_search_domains` - (Optional) A list of DNS search domains for the container 78 to use. 79 80 - `dns_options` - (Optional) A list of DNS options for the container to use. 81 82 - `dns_servers` - (Optional) A list of DNS servers for the container to use 83 (e.g. ["8.8.8.8", "8.8.4.4"]). Requires Docker v1.10 or greater. 84 85 - `entrypoint` - (Optional) A string list overriding the image's entrypoint. 86 87 - `extra_hosts` - (Optional) A list of hosts, given as host:IP, to be added to 88 `/etc/hosts`. 89 90 - `force_pull` - (Optional) `true` or `false` (default). Always pull most recent image 91 instead of using existing local image. Should be set to `true` if repository tags 92 are mutable. If image's tag is `latest` or omitted, the image will always be pulled 93 regardless of this setting. 94 95 - `hostname` - (Optional) The hostname to assign to the container. When 96 launching more than one of a task (using `count`) with this option set, every 97 container the task starts will have the same hostname. 98 99 - `interactive` - (Optional) `true` or `false` (default). Keep STDIN open on 100 the container. 101 102 - `sysctl` - (Optional) A key-value map of sysctl configurations to set to the 103 containers on start. 104 105 ```hcl 106 config { 107 sysctl { 108 net.core.somaxconn = "16384" 109 } 110 } 111 ``` 112 113 - `ulimit` - (Optional) A key-value map of ulimit configurations to set to the 114 containers on start. 115 116 ```hcl 117 config { 118 ulimit { 119 nproc = "4242" 120 nofile = "2048:4096" 121 } 122 } 123 ``` 124 125 - `privileged` - (Optional) `true` or `false` (default). Privileged mode gives 126 the container access to devices on the host. Note that this also requires the 127 nomad agent and docker daemon to be configured to allow privileged 128 containers. 129 130 - `ipc_mode` - (Optional) The IPC mode to be used for the container. The default 131 is `none` for a private IPC namespace. Other values are `host` for sharing 132 the host IPC namespace or the name or id of an existing container. Note that 133 it is not possible to refer to Docker containers started by Nomad since their 134 names are not known in advance. Note that setting this option also requires the 135 Nomad agent to be configured to allow privileged containers. 136 137 - `ipv4_address` - (Optional) The IPv4 address to be used for the container when 138 using user defined networks. Requires Docker 1.13 or greater. 139 140 - `ipv6_address` - (Optional) The IPv6 address to be used for the container when 141 using user defined networks. Requires Docker 1.13 or greater. 142 143 - `labels` - (Optional) A key-value map of labels to set to the containers on 144 start. 145 146 ```hcl 147 config { 148 labels { 149 foo = "bar" 150 zip = "zap" 151 } 152 } 153 ``` 154 155 - `load` - (Optional) Load an image from a `tar` archive file instead of from a 156 remote repository. Equivalent to the `docker load -i <filename>` command. 157 158 ```hcl 159 artifact { 160 source = "http://path.to/redis.tar" 161 } 162 config { 163 load = "redis.tar" 164 image = "redis" 165 } 166 ``` 167 168 - `logging` - (Optional) A key-value map of Docker logging options. 169 Defaults to `json-file` with log rotation (`max-file=2` and `max-size=2m`). 170 171 ```hcl 172 config { 173 logging { 174 type = "fluentd" 175 config { 176 fluentd-address = "localhost:24224" 177 tag = "your_tag" 178 } 179 } 180 } 181 ``` 182 183 - `mac_address` - (Optional) The MAC address for the container to use (e.g. 184 "02:68:b3:29:da:98"). 185 186 - `memory_hard_limit` - (Optional) The maximum allowable amount of memory used 187 (megabytes) by the container. If set, the [`memory`](/docs/job-specification/resources#memory) 188 parameter of the task resource configuration becomes a soft limit passed to the 189 docker driver as [`--memory_reservation`](https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory), 190 and `memory_hard_limit` is passed as the [`--memory`](https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory) 191 hard limit. When the host is under memory pressure, the behavior of soft limit 192 activation is governed by the [Kernel](https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt). 193 194 - `network_aliases` - (Optional) A list of network-scoped aliases, provide a way for a 195 container to be discovered by an alternate name by any other container within 196 the scope of a particular network. Network-scoped alias is supported only for 197 containers in user defined networks 198 199 ```hcl 200 config { 201 network_mode = "user-network" 202 network_aliases = [ 203 "${NOMAD_TASK_NAME}", 204 "${NOMAD_TASK_NAME}-${NOMAD_ALLOC_INDEX}" 205 ] 206 } 207 ``` 208 209 - `network_mode` - (Optional) The network mode to be used for the container. In 210 order to support userspace networking plugins in Docker 1.9 this accepts any 211 value. The default is `bridge` for all operating systems but Windows, which 212 defaults to `nat`. Other networking modes may not work without additional 213 configuration on the host (which is outside the scope of Nomad). Valid values 214 pre-docker 1.9 are `default`, `bridge`, `host`, `none`, or `container:name`. 215 216 - `pid_mode` - (Optional) `host` or not set (default). Set to `host` to share 217 the PID namespace with the host. Note that this also requires the Nomad agent 218 to be configured to allow privileged containers. 219 See below for more details. 220 221 - `port_map` - (Optional) A key-value map of port labels (see below). 222 223 - `security_opt` - (Optional) A list of string flags to pass directly to 224 [`--security-opt`](https://docs.docker.com/engine/reference/run/#security-configuration). 225 For example: 226 227 ```hcl 228 config { 229 security_opt = [ 230 "credentialspec=file://gmsaUser.json", 231 ] 232 } 233 ``` 234 235 - `shm_size` - (Optional) The size (bytes) of /dev/shm for the container. 236 237 - `storage_opt` - (Optional) A key-value map of storage options set to the containers on start. 238 This overrides the [host dockerd configuration](https://docs.docker.com/engine/reference/commandline/dockerd/#options-per-storage-driver). 239 For example: 240 241 ```hcl 242 config { 243 storage_opt = { 244 size = "40G" 245 } 246 } 247 ``` 248 249 - `SSL` - (Optional) If this is set to true, Nomad uses SSL to talk to the 250 repository. The default value is `true`. **Deprecated as of 0.5.3** 251 252 - `tty` - (Optional) `true` or `false` (default). Allocate a pseudo-TTY for the 253 container. 254 255 - `uts_mode` - (Optional) `host` or not set (default). Set to `host` to share 256 the UTS namespace with the host. Note that this also requires the Nomad agent 257 to be configured to allow privileged containers. 258 259 - `userns_mode` - (Optional) `host` or not set (default). Set to `host` to use 260 the host's user namespace when user namespace remapping is enabled on the 261 docker daemon. 262 263 - `volumes` - (Optional) A list of `host_path:container_path` strings to bind 264 host paths to container paths. Mounting host paths outside of the allocation 265 directory can be disabled on clients by setting the `docker.volumes.enabled` 266 option set to false. This will limit volumes to directories that exist inside 267 the allocation directory. We recommend using [`mounts`](#mounts) if you wish 268 to have more control over volume definitions. 269 270 ```hcl 271 config { 272 volumes = [ 273 # Use absolute paths to mount arbitrary paths on the host 274 "/path/on/host:/path/in/container", 275 276 # Use relative paths to rebind paths already in the allocation dir 277 "relative/to/task:/also/in/container" 278 ] 279 } 280 ``` 281 282 - `volume_driver` - (Optional) The name of the volume driver used to mount 283 volumes. Must be used along with `volumes`. If `volume_driver` is omitted, 284 then relative paths will be mounted from inside the allocation dir. If a 285 `"local"` or other driver is used, then they may be named volumes instead. 286 If `docker.volumes.enabled` is false then volume drivers and paths outside the 287 allocation directory are disallowed. 288 289 ```hcl 290 config { 291 volumes = [ 292 # Use named volume created outside nomad. 293 "name-of-the-volume:/path/in/container" 294 ] 295 # Name of the Docker Volume Driver used by the container 296 volume_driver = "pxd" 297 } 298 ``` 299 300 - `work_dir` - (Optional) The working directory inside the container. 301 302 - `mounts` - (Optional) A list of 303 [mounts](https://docs.docker.com/engine/reference/commandline/service_create/#add-bind-mounts-or-volumes) 304 to be mounted into the container. Volume, bind, and tmpfs type mounts are supported. 305 306 ```hcl 307 config { 308 mounts = [ 309 # sample volume mount 310 { 311 type = "volume" 312 target = "/path/in/container" 313 source = "name-of-volume" 314 readonly = false 315 volume_options { 316 no_copy = false 317 labels { 318 foo = "bar" 319 } 320 driver_config { 321 name = "pxd" 322 options = { 323 foo = "bar" 324 } 325 } 326 } 327 }, 328 # sample bind mount 329 { 330 type = "bind" 331 target = "/path/in/container" 332 source = "/path/in/host" 333 readonly = false 334 bind_options { 335 propagation = "rshared" 336 } 337 }, 338 # sample tmpfs mount 339 { 340 type = "tmpfs" 341 target = "/path/in/container" 342 readonly = false 343 tmpfs_options { 344 size = 100000 # size in bytes 345 } 346 } 347 ] 348 } 349 ``` 350 351 - `devices` - (Optional) A list of 352 [devices](https://docs.docker.com/engine/reference/commandline/run/#add-host-device-to-container-device) 353 to be exposed the container. `host_path` is the only required field. By default, the container will be able to 354 `read`, `write` and `mknod` these devices. Use the optional `cgroup_permissions` field to restrict permissions. 355 356 ```hcl 357 config { 358 devices = [ 359 { 360 host_path = "/dev/sda1" 361 container_path = "/dev/xvdc" 362 cgroup_permissions = "r" 363 }, 364 { 365 host_path = "/dev/sda2" 366 container_path = "/dev/xvdd" 367 } 368 ] 369 } 370 ``` 371 372 - `cap_add` - (Optional) A list of Linux capabilities as strings to pass directly to 373 [`--cap-add`](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). 374 Effective capabilities (computed from `cap_add` and `cap_drop`) have to match the configured whitelist. 375 The whitelist can be customized using the [`allow_caps`](#plugin_caps) plugin option key in the client node's configuration. 376 For example: 377 378 ```hcl 379 config { 380 cap_add = [ 381 "SYS_TIME", 382 ] 383 } 384 ``` 385 386 - `cap_drop` - (Optional) A list of Linux capabilities as strings to pass directly to 387 [`--cap-drop`](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). 388 Effective capabilities (computed from `cap_add` and `cap_drop`) have to match the configured whitelist. 389 The whitelist can be customized using the [`allow_caps`](#plugin_caps) plugin option key in the client node's configuration. 390 For example: 391 392 ```hcl 393 config { 394 cap_drop = [ 395 "MKNOD", 396 ] 397 } 398 ``` 399 400 - `cpu_hard_limit` - (Optional) `true` or `false` (default). Use hard CPU 401 limiting instead of soft limiting. By default this is `false` which means 402 soft limiting is used and containers are able to burst above their CPU limit 403 when there is idle capacity. 404 405 - `cpu_cfs_period` - (Optional) An integer value that specifies the duration in microseconds of the period 406 during which the CPU usage quota is measured. The default is 100000 (0.1 second) and the maximum allowed 407 value is 1000000 (1 second). See [here](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-cpu#sect-cfs) 408 for more details. 409 410 - `advertise_ipv6_address` - (Optional) `true` or `false` (default). Use the container's 411 IPv6 address (GlobalIPv6Address in Docker) when registering services and checks. 412 See [IPv6 Docker containers](/docs/job-specification/service#IPv6 Docker containers) for details. 413 414 - `readonly_rootfs` - (Optional) `true` or `false` (default). Mount 415 the container's filesystem as read only. 416 417 - `runtime` - (Optional) A string representing a configured runtime to pass to docker. 418 This is equivalent to the `--runtime` argument in the docker CLI 419 For example, to use gVisor: 420 421 ```hcl 422 config { 423 # gVisor runtime is runsc 424 runtime = "runsc" 425 } 426 ``` 427 428 - `pids_limit` - (Optional) An integer value that specifies the pid limit for 429 the container. Defaults to unlimited. 430 431 Additionally, the docker driver supports customization of the container's user through the task's [`user` option](/docs/job-specification/task#user). 432 433 ### Container Name 434 435 Nomad creates a container after pulling an image. Containers are named 436 `{taskName}-{allocId}`. This is necessary in order to place more than one 437 container from the same task on a host (e.g. with count > 1). This also means 438 that each container's name is unique across the cluster. 439 440 This is not configurable. 441 442 ### Authentication 443 444 If you want to pull from a private repo (for example on dockerhub or quay.io), 445 you will need to specify credentials in your job via: 446 447 - the `auth` option in the task config. 448 449 - by storing explicit repository credentials or by specifying Docker 450 `credHelpers` in a file and setting the auth [config](#plugin_auth_file) 451 value on the client in the plugin options. 452 453 - by specifying an auth [helper](#plugin_auth_helper) on the client in the 454 plugin options. 455 456 The `auth` object supports the following keys: 457 458 - `username` - (Optional) The account username. 459 460 - `password` - (Optional) The account password. 461 462 - `email` - (Optional) The account email. 463 464 - `server_address` - (Optional) The server domain/IP without the protocol. 465 Docker Hub is used by default. 466 467 Example task-config: 468 469 ```hcl 470 task "example" { 471 driver = "docker" 472 473 config { 474 image = "secret/service" 475 476 auth { 477 username = "dockerhub_user" 478 password = "dockerhub_password" 479 } 480 } 481 } 482 ``` 483 484 Example docker-config, using two helper scripts in \$PATH, 485 "docker-credential-ecr" and "docker-credential-vault": 486 487 ```json 488 { 489 "auths": { 490 "internal.repo": { 491 "auth": "`echo -n '<username>:<password>' | base64 -w0`" 492 } 493 }, 494 "credHelpers": { 495 "<XYZ>.dkr.ecr.<region>.amazonaws.com": "ecr-login" 496 }, 497 "credsStore": "secretservice" 498 } 499 ``` 500 501 Example agent configuration, using a helper script "docker-credential-ecr" in 502 \$PATH 503 504 ```hcl 505 client { 506 enabled = true 507 } 508 509 plugin "docker" { 510 config { 511 auth { 512 # Nomad will prepend "docker-credential-" to the helper value and call 513 # that script name. 514 helper = "ecr" 515 } 516 } 517 } 518 ``` 519 520 !> **Be Careful!** At this time these credentials are stored in Nomad in plain 521 text. Secrets management will be added in a later release. 522 523 ## Networking 524 525 Docker supports a variety of networking configurations, including using host 526 interfaces, SDNs, etc. Nomad uses `bridged` networking by default, like Docker. 527 528 You can specify other networking options, including custom networking plugins 529 in Docker 1.9. **You may need to perform additional configuration on the host 530 in order to make these work.** This additional configuration is outside the 531 scope of Nomad. 532 533 ### Allocating Ports 534 535 You can allocate ports to your task using the port syntax described on the 536 [networking page](/docs/job-specification/network). Here is a recap: 537 538 ```hcl 539 task "example" { 540 driver = "docker" 541 542 resources { 543 network { 544 port "http" {} 545 port "https" {} 546 } 547 } 548 } 549 ``` 550 551 ### Forwarding and Exposing Ports 552 553 A Docker container typically specifies which port a service will listen on by 554 specifying the `EXPOSE` directive in the `Dockerfile`. 555 556 Because dynamic ports will not match the ports exposed in your Dockerfile, 557 Nomad will automatically expose all of the ports it allocates to your 558 container. 559 560 These ports will be identified via environment variables. For example: 561 562 ```hcl 563 port "http" {} 564 ``` 565 566 If Nomad allocates port `23332` to your task for `http`, `23332` will be 567 automatically exposed and forwarded to your container, and the driver will set 568 an environment variable `NOMAD_PORT_http` with the value `23332` that you can 569 read inside your container. 570 571 This provides an easy way to use the `host` networking option for better 572 performance. 573 574 ### Using the Port Map 575 576 If you prefer to use the traditional port-mapping method, you can specify the 577 `port_map` option in your job specification. It looks like this: 578 579 ```hcl 580 task "example" { 581 driver = "docker" 582 583 config { 584 image = "redis" 585 586 port_map { 587 redis = 6379 588 } 589 } 590 591 resources { 592 network { 593 mbits = 20 594 port "redis" {} 595 } 596 } 597 } 598 ``` 599 600 If Nomad allocates port `23332` to your task, the Docker driver will 601 automatically setup the port mapping from `23332` on the host to `6379` in your 602 container, so it will just work! 603 604 Note that by default this only works with `bridged` networking mode. It may 605 also work with custom networking plugins which implement the same API for 606 expose and port forwarding. 607 608 ### Advertising Container IPs 609 610 _New in Nomad 0.6._ 611 612 When using network plugins like `weave` that assign containers a routable IP 613 address, that address will automatically be used in any `service` 614 advertisements for the task. You may override what address is advertised by 615 using the `address_mode` parameter on a `service`. See 616 [service](/docs/job-specification/service) for details. 617 618 ### Networking Protocols 619 620 The Docker driver configures ports on both the `tcp` and `udp` protocols. 621 622 This is not configurable. 623 624 ### Other Networking Modes 625 626 Some networking modes like `container` or `none` will require coordination 627 outside of Nomad. First-class support for these options may be improved later 628 through Nomad plugins or dynamic job configuration. 629 630 ## Client Requirements 631 632 Nomad requires Docker to be installed and running on the host alongside the 633 Nomad agent. Nomad was developed against Docker `1.8.2` and `1.9`. 634 635 By default Nomad communicates with the Docker daemon using the daemon's Unix 636 socket. Nomad will need to be able to read/write to this socket. If you do not 637 run Nomad as root, make sure you add the Nomad user to the Docker group so 638 Nomad can communicate with the Docker daemon. 639 640 For example, on Ubuntu you can use the `usermod` command to add the `vagrant` 641 user to the `docker` group so you can run Nomad without root: 642 643 ```shell-session 644 $ sudo usermod -G docker -a vagrant 645 ``` 646 647 For the best performance and security features you should use recent versions 648 of the Linux Kernel and Docker daemon. 649 650 If you would like to change any of the options related to the `docker` driver on 651 a Nomad client, you can modify them with the [plugin stanza][plugin-stanza] syntax. Below is an example of a configuration (many of the values are the default). See the next section for more information on the options. 652 653 ```hcl 654 plugin "docker" { 655 config { 656 endpoint = "unix:///var/run/docker.sock" 657 658 auth { 659 config = "/etc/docker-auth.json" 660 helper = "docker-credential-aws" 661 } 662 663 tls { 664 cert = "/etc/nomad/nomad.pub" 665 key = "/etc/nomad/nomad.pem" 666 ca = "/etc/nomad/nomad.cert" 667 } 668 669 gc { 670 image = true 671 image_delay = "3m" 672 container = true 673 674 dangling_containers { 675 enabled = true 676 dry_run = false 677 period = "5m" 678 creation_grace = "5m" 679 } 680 } 681 682 volumes { 683 enabled = true 684 selinuxlabel = "z" 685 } 686 687 allow_privileged = false 688 allow_caps = ["CHOWN", "NET_RAW"] 689 690 # allow_caps can also be set to "ALL" 691 # allow_caps = ["ALL"] 692 } 693 } 694 ``` 695 696 ## Plugin Options 697 698 - `endpoint` - If using a non-standard socket, HTTP or another location, or if 699 TLS is being used, docker.endpoint must be set. If unset, Nomad will attempt 700 to instantiate a Docker client using the DOCKER_HOST environment variable and 701 then fall back to the default listen address for the given operating system. 702 Defaults to unix:///var/run/docker.sock on Unix platforms and 703 npipe:////./pipe/docker_engine for Windows. 704 705 - `allow_privileged` - Defaults to `false`. Changing this to true will allow 706 containers to use privileged mode, which gives the containers full access to 707 the host's devices. Note that you must set a similar setting on the Docker 708 daemon for this to work. 709 710 - `pull_activity_timeout` - Defaults to `2m`. If Nomad receives no communication 711 from the Docker engine during an image pull within this timeframe, Nomad will 712 timeout the request that initiated the pull command. (Minimum of `1m`) 713 714 - `allow_caps`<a id="plugin_caps"></a> - A list of allowed Linux capabilities. 715 Defaults to 716 "CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP, 717 NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE", which is the list of 718 capabilities allowed by docker by default, as defined here. Allows the 719 operator to control which capabilities can be obtained by tasks using cap_add 720 and cap_drop options. Supports the value "ALL" as a shortcut for whitelisting 721 all capabilities. 722 723 - `allow_runtimes` - defaults to `["runc", "nvidia"]` - A list of the allowed 724 docker runtimes a task may use. 725 726 - `auth` stanza: 727 728 - `config`<a id="plugin_auth_file"></a> - Allows an operator to specify a 729 JSON file which is in the dockercfg format containing authentication 730 information for a private registry, from either (in order) `auths`, 731 `credHelpers` or `credsStore`. 732 - `helper`<a id="plugin_auth_helper"></a> - Allows an operator to specify a 733 [credsStore](https://docs.docker.com/engine/reference/commandline/login/#credential-helper-protocol) 734 -like script on \$PATH to lookup authentication information from external 735 sources. The script's name must begin with `docker-credential-` and this 736 option should include only the basename of the script, not the path. 737 738 - `tls` stanza: 739 740 - `cert` - Path to the server's certificate file (`.pem`). Specify this 741 along with `key` and `ca` to use a TLS client to connect to the docker 742 daemon. `endpoint` must also be specified or this setting will be ignored. 743 - `key` - Path to the client's private key (`.pem`). Specify this along with 744 `cert` and `ca` to use a TLS client to connect to the docker daemon. 745 `endpoint` must also be specified or this setting will be ignored. 746 - `ca` - Path to the server's CA file (`.pem`). Specify this along with 747 `cert` and `key` to use a TLS client to connect to the docker daemon. 748 `endpoint` must also be specified or this setting will be ignored. 749 750 - `disable_log_collection` - Defaults to `false`. Setting this to true will 751 disable Nomad logs collection of Docker tasks. If you don't rely on nomad log 752 capabilities and exclusively use host based log aggregation, you may consider 753 this option to disable nomad log collection overhead. 754 755 - `gc` stanza: 756 757 - `image` - Defaults to `true`. Changing this to `false` will prevent Nomad 758 from removing images from stopped tasks. 759 - `image_delay` - A time duration, as [defined 760 here](https://golang.org/pkg/time/#ParseDuration), that defaults to `3m`. 761 The delay controls how long Nomad will wait between an image being unused 762 and deleting it. If a tasks is received that uses the same image within 763 the delay, the image will be reused. 764 - `container` - Defaults to `true`. This option can be used to disable Nomad 765 from removing a container when the task exits. Under a name conflict, 766 Nomad may still remove the dead container. 767 - `dangling_containers` stanza for controlling dangling container detection 768 and cleanup: 769 770 - `enabled` - Defaults to `true`. Enables dangling container handling. 771 - `dry_run` - Defaults to `false`. Only log dangling containers without 772 cleaning them up. 773 - `period` - Defaults to `"5m"`. A time duration that controls interval 774 between Nomad scans for dangling containers. 775 - `creation_grace` - Defaults to `"5m"`. Grace period after a container is 776 created during which the GC ignores it. Only used to prevent the GC from 777 removing newly created containers before they are registered with the 778 GC. Should not need adjusting higher but may be adjusted lower to GC 779 more aggressively. 780 781 - `volumes` stanza: 782 783 - `enabled` - Defaults to `true`. Allows tasks to bind host paths 784 (`volumes`) inside their container and use volume drivers 785 (`volume_driver`). Binding relative paths is always allowed and will be 786 resolved relative to the allocation's directory. 787 - `selinuxlabel` - Allows the operator to set a SELinux label to the 788 allocation and task local bind-mounts to containers. If used with 789 `docker.volumes.enabled` set to false, the labels will still be applied to 790 the standard binds in the container. 791 792 - `infra_image` - This is the Docker image to use when creating the parent 793 container necessary when sharing network namespaces between tasks. Defaults 794 to "gcr.io/google_containers/pause-amd64:3.0". 795 796 ## Client Configuration 797 798 ~> Note: client configuration options will soon be deprecated. Please use 799 [plugin options][plugin-options] instead. See the [plugin stanza][plugin-stanza] 800 documentation for more information. 801 802 The `docker` driver has the following [client configuration 803 options](/docs/configuration/client#options): 804 805 - `docker.endpoint` - If using a non-standard socket, HTTP or another location, 806 or if TLS is being used, `docker.endpoint` must be set. If unset, Nomad will 807 attempt to instantiate a Docker client using the `DOCKER_HOST` environment 808 variable and then fall back to the default listen address for the given 809 operating system. Defaults to `unix:///var/run/docker.sock` on Unix platforms 810 and `npipe:////./pipe/docker_engine` for Windows. 811 812 - `docker.auth.config` <a id="auth_file"></a>- Allows an operator to specify a 813 JSON file which is in the dockercfg format containing authentication 814 information for a private registry, from either (in order) `auths`, 815 `credHelpers` or `credsStore`. 816 817 - `docker.auth.helper` <a id="auth_helper"></a>- Allows an operator to specify a 818 [credsStore](https://docs.docker.com/engine/reference/commandline/login/#credential-helper-protocol) 819 -like script on \$PATH to lookup authentication information from external 820 sources. The script's name must begin with `docker-credential-` and this 821 option should include only the basename of the script, not the path. 822 823 - `docker.tls.cert` - Path to the server's certificate file (`.pem`). Specify 824 this along with `docker.tls.key` and `docker.tls.ca` to use a TLS client to 825 connect to the docker daemon. `docker.endpoint` must also be specified or this 826 setting will be ignored. 827 828 - `docker.tls.key` - Path to the client's private key (`.pem`). Specify this 829 along with `docker.tls.cert` and `docker.tls.ca` to use a TLS client to 830 connect to the docker daemon. `docker.endpoint` must also be specified or this 831 setting will be ignored. 832 833 - `docker.tls.ca` - Path to the server's CA file (`.pem`). Specify this along 834 with `docker.tls.cert` and `docker.tls.key` to use a TLS client to connect to 835 the docker daemon. `docker.endpoint` must also be specified or this setting 836 will be ignored. 837 838 - `docker.cleanup.image` Defaults to `true`. Changing this to `false` will 839 prevent Nomad from removing images from stopped tasks. 840 841 - `docker.cleanup.image.delay` A time duration, as [defined 842 here](https://golang.org/pkg/time/#ParseDuration), that defaults to `3m`. The 843 delay controls how long Nomad will wait between an image being unused and 844 deleting it. If a tasks is received that uses the same image within the delay, 845 the image will be reused. 846 847 - `docker.volumes.enabled`: Defaults to `true`. Allows tasks to bind host paths 848 (`volumes`) inside their container and use volume drivers (`volume_driver`). 849 Binding relative paths is always allowed and will be resolved relative to the 850 allocation's directory. 851 852 - `docker.volumes.selinuxlabel`: Allows the operator to set a SELinux label to 853 the allocation and task local bind-mounts to containers. If used with 854 `docker.volumes.enabled` set to false, the labels will still be applied to the 855 standard binds in the container. 856 857 - `docker.privileged.enabled` Defaults to `false`. Changing this to `true` will 858 allow containers to use `privileged` mode, which gives the containers full 859 access to the host's devices. Note that you must set a similar setting on the 860 Docker daemon for this to work. 861 862 - `docker.caps.whitelist`: A list of allowed Linux capabilities. Defaults to 863 `"CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP, SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE"`, which is the list of 864 capabilities allowed by docker by default, as [defined 865 here](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). 866 Allows the operator to control which capabilities can be obtained by tasks 867 using `cap_add` and `cap_drop` options. Supports the value `"ALL"` as a 868 shortcut for whitelisting all capabilities. 869 870 - `docker.cleanup.container`: Defaults to `true`. This option can be used to 871 disable Nomad from removing a container when the task exits. Under a name 872 conflict, Nomad may still remove the dead container. 873 874 - `docker.nvidia_runtime`: Defaults to `nvidia`. This option allows operators to select the runtime that should be used in order to expose Nvidia GPUs to the container. 875 876 Note: When testing or using the `-dev` flag you can use `DOCKER_HOST`, 877 `DOCKER_TLS_VERIFY`, and `DOCKER_CERT_PATH` to customize Nomad's behavior. If 878 `docker.endpoint` is set Nomad will **only** read client configuration from the 879 config file. 880 881 An example is given below: 882 883 ```hcl 884 client { 885 options { 886 "docker.cleanup.image" = "false" 887 } 888 } 889 ``` 890 891 ## Client Attributes 892 893 The `docker` driver will set the following client attributes: 894 895 - `driver.docker` - This will be set to "1", indicating the driver is 896 available. 897 - `driver.docker.bridge_ip` - The IP of the Docker bridge network if one 898 exists. 899 - `driver.docker.version` - This will be set to version of the docker server. 900 901 Here is an example of using these properties in a job file: 902 903 ```hcl 904 job "docs" { 905 # Require docker version higher than 1.2. 906 constraint { 907 attribute = "${driver.docker.version}" 908 operator = ">" 909 version = "1.2" 910 } 911 } 912 ``` 913 914 ## Resource Isolation 915 916 ### CPU 917 918 Nomad limits containers' CPU based on CPU shares. CPU shares allow containers 919 to burst past their CPU limits. CPU limits will only be imposed when there is 920 contention for resources. When the host is under load your process may be 921 throttled to stabilize QoS depending on how many shares it has. You can see how 922 many CPU shares are available to your process by reading `NOMAD_CPU_LIMIT`. 923 1000 shares are approximately equal to 1 GHz. 924 925 Please keep the implications of CPU shares in mind when you load test workloads 926 on Nomad. 927 928 ### Memory 929 930 Nomad limits containers' memory usage based on total virtual memory. This means 931 that containers scheduled by Nomad cannot use swap. This is to ensure that a 932 swappy process does not degrade performance for other workloads on the same 933 host. 934 935 Since memory is not an elastic resource, you will need to make sure your 936 container does not exceed the amount of memory allocated to it, or it will be 937 terminated or crash when it tries to malloc. A process can inspect its memory 938 limit by reading `NOMAD_MEMORY_LIMIT`, but will need to track its own memory 939 usage. Memory limit is expressed in megabytes so 1024 = 1 GB. 940 941 ### IO 942 943 Nomad's Docker integration does not currently provide QoS around network or 944 filesystem IO. These will be added in a later release. 945 946 ### Security 947 948 Docker provides resource isolation by way of 949 [cgroups and namespaces](https://docs.docker.com/introduction/understanding-docker/#the-underlying-technology). 950 Containers essentially have a virtual file system all to themselves. If you 951 need a higher degree of isolation between processes for security or other 952 reasons, it is recommended to use full virtualization like 953 [QEMU](/docs/drivers/qemu). 954 955 ## Caveats 956 957 ### Dangling Containers 958 959 Nomad 0.10.2 introduces a detector and a reaper for dangling Docker containers, 960 containers that Nomad starts yet does not manage or track. Though rare, they 961 lead to unexpectedly running services, potentially with stale versions. 962 963 When Docker daemon becomes unavailable as Nomad starts a task, it is possible 964 for Docker to successfully start the container but return a 500 error code from 965 the API call. In such cases, Nomad retries and eventually aims to kill such 966 containers. However, if the Docker Engine remains unhealthy, subsequent retries 967 and stop attempts may still fail, and the started container becomes a dangling 968 container that Nomad no longer manages. 969 970 The newly added reaper periodically scans for such containers. It only targets 971 containers with a `com.hashicorp.nomad.allocation_id` label, or match Nomad's 972 conventions for naming and bind-mounts (i.e. `/alloc`, `/secrets`, `local`). 973 Containers that don't match Nomad container patterns are left untouched. 974 975 Operators can run the reaper in a dry-run mode, where it only logs dangling 976 container ids without killing them, or disable it by setting the 977 `gc.dangling_containers` config stanza. 978 979 ### Docker for Windows 980 981 Docker for Windows only supports running Windows containers. Because Docker for 982 Windows is relatively new and rapidly evolving you may want to consult the 983 [list of relevant issues on GitHub][winissues]. 984 985 [winissues]: https://github.com/hashicorp/nomad/issues?q=is%3Aopen+is%3Aissue+label%3Adriver%2Fdocker+label%3Aplatform-windows 986 [plugin-options]: #plugin-options 987 [plugin-stanza]: /docs/configuration/plugin