github.com/taylorchu/nomad@v0.5.3-rc1.0.20170407200202-db11e7dd7b55/website/source/docs/drivers/docker.html.md (about) 1 --- 2 layout: "docs" 3 page_title: "Drivers: Docker" 4 sidebar_current: "docs-drivers-docker" 5 description: |- 6 The Docker task driver is used to run Docker based tasks. 7 --- 8 9 # Docker Driver 10 11 Name: `docker` 12 13 The `docker` driver provides a first-class Docker workflow on Nomad. The Docker 14 driver handles downloading containers, mapping ports, and starting, watching, 15 and cleaning up after containers. 16 17 ## Task Configuration 18 19 ```hcl 20 task "webservice" { 21 driver = "docker" 22 23 config { 24 image = "redis:3.2" 25 labels { 26 group = "webservice-cache" 27 } 28 } 29 } 30 ``` 31 32 The `docker` driver supports the following configuration in the job spec: 33 34 * `image` - The Docker image to run. The image may include a tag or custom URL 35 and should include `https://` if required. By default it will be fetched from 36 Docker Hub. If the tag is omitted or equal to `latest` the driver will always 37 try to pull the image. If the image to be pulled exists in a registry that 38 requires authentication credentials must be provided to Nomad. Please see the 39 [Authentication section](#authentication). 40 41 ```hcl 42 config { 43 image = "https://hub.docker.internal/redis:3.2" 44 } 45 ``` 46 47 * `load` - (Optional) A list of paths to image archive files. If 48 this key is not specified, Nomad assumes the `image` is hosted on a repository 49 and attempts to pull the image. The `artifact` blocks can be specified to 50 download each of the archive files. The equivalent of `docker load -i path` 51 would be run on each of the archive files. 52 53 ```hcl 54 artifact { 55 source = "http://path.to/redis.tar" 56 } 57 config { 58 load = ["redis.tar"] 59 image = "redis" 60 } 61 ``` 62 63 * `command` - (Optional) The command to run when starting the container. 64 65 ```hcl 66 config { 67 command = "my-command" 68 } 69 ``` 70 71 * `args` - (Optional) A list of arguments to the optional `command`. If no 72 `command` is specified, the args are passed directly to the container. 73 References to environment variables or any [interpretable Nomad 74 variables](/docs/runtime/interpolation.html) will be interpreted before 75 launching the task. For example: 76 77 ```hcl 78 config { 79 args = [ 80 "-bind", "${NOMAD_PORT_http}", 81 "${nomad.datacenter}", 82 "${MY_ENV}", 83 "${meta.foo}", 84 ] 85 } 86 ``` 87 88 * `labels` - (Optional) A key-value map of labels to set to the containers on 89 start. 90 91 ```hcl 92 config { 93 labels { 94 foo = "bar" 95 zip = "zap" 96 } 97 } 98 ``` 99 100 * `privileged` - (Optional) `true` or `false` (default). Privileged mode gives 101 the container access to devices on the host. Note that this also requires the 102 nomad agent and docker daemon to be configured to allow privileged 103 containers. 104 105 * `ipc_mode` - (Optional) The IPC mode to be used for the container. The default 106 is `none` for a private IPC namespace. Other values are `host` for sharing 107 the host IPC namespace or the name or id of an existing container. Note that 108 it is not possible to refer to Docker containers started by Nomad since their 109 names are not known in advance. Note that setting this option also requires the 110 Nomad agent to be configured to allow privileged containers. 111 112 * `pid_mode` - (Optional) `host` or not set (default). Set to `host` to share 113 the PID namespace with the host. Note that this also requires the Nomad agent 114 to be configured to allow privileged containers. 115 116 * `uts_mode` - (Optional) `host` or not set (default). Set to `host` to share 117 the UTS namespace with the host. Note that this also requires the Nomad agent 118 to be configured to allow privileged containers. 119 120 * `userns_mode` - (Optional) `host` or not set (default). Set to `host` to use 121 the host's user namespace when user namespace remapping is enabled on the 122 docker daemon. 123 124 * `network_mode` - (Optional) The network mode to be used for the container. In 125 order to support userspace networking plugins in Docker 1.9 this accepts any 126 value. The default is `bridge` for all operating systems but Windows, which 127 defaults to `nat`. Other networking modes may not work without additional 128 configuration on the host (which is outside the scope of Nomad). Valid values 129 pre-docker 1.9 are `default`, `bridge`, `host`, `none`, or `container:name`. 130 See below for more details. 131 132 * `network_aliases` - (Optional) A list of network-scoped aliases, provide a way for a 133 container to be discovered by an alternate name by any other container within 134 the scope of a particular network. Network-scoped alias is supported only for 135 containers in user defined networks 136 137 ```hcl 138 config { 139 network_mode = "user-network" 140 network_aliases = [ 141 "${NOMAD_TASK_NAME}", 142 "${NOMAD_TASK_NAME}-${NOMAD_ALLOC_INDEX}" 143 ] 144 } 145 ``` 146 147 * `ipv4_address` - (Optional) The IPv4 address to be used for the container when 148 using user defined networks. Requires docker 1.13.0 or greater. 149 150 * `ipv6_address` - (Optional) The IPv6 address to be used for the container when 151 using user defined networks. Requires docker 1.13.0 or greater. 152 153 * `hostname` - (Optional) The hostname to assign to the container. When 154 launching more than one of a task (using `count`) with this option set, every 155 container the task starts will have the same hostname. 156 157 * `dns_servers` - (Optional) A list of DNS servers for the container to use 158 (e.g. ["8.8.8.8", "8.8.4.4"]). *Docker API v1.10 and above only* 159 160 * `dns_search_domains` - (Optional) A list of DNS search domains for the container 161 to use. 162 163 * `SSL` - (Optional) If this is set to true, Nomad uses SSL to talk to the 164 repository. The default value is `true`. **Deprecated as of 0.5.3** 165 166 * `port_map` - (Optional) A key-value map of port labels (see below). 167 168 * `auth` - (Optional) Provide authentication for a private registry (see below). 169 170 * `tty` - (Optional) `true` or `false` (default). Allocate a pseudo-TTY for the 171 container. 172 173 * `interactive` - (Optional) `true` or `false` (default). Keep STDIN open on 174 the container. 175 176 * `force_pull` - (Optional) `true` or `false` (default). Always pull latest image 177 instead of using existing local image. Should be set to `true` if repository tags 178 are mutable. 179 180 * `shm_size` - (Optional) The size (bytes) of /dev/shm for the container. 181 182 * `logging` - (Optional) A key-value map of Docker logging options. The default 183 value is `syslog`. 184 185 ```hcl 186 config { 187 logging { 188 type = "fluentd" 189 config { 190 fluentd-address = "localhost:24224" 191 tag = "your_tag" 192 } 193 } 194 } 195 ``` 196 197 * `volumes` - (Optional) A list of `host_path:container_path` strings to bind 198 host paths to container paths. Mounting host paths outside of the allocation 199 directory can be disabled on clients by setting the `docker.volumes.enabled` 200 option set to false. This will limit volumes to directories that exist inside 201 the allocation directory. 202 203 ```hcl 204 config { 205 volumes = [ 206 # Use absolute paths to mount arbitrary paths on the host 207 "/path/on/host:/path/in/container", 208 209 # Use relative paths to rebind paths already in the allocation dir 210 "relative/to/task:/also/in/container" 211 ] 212 } 213 ``` 214 215 * `volume_driver` - (Optional) The name of the volume driver used to mount 216 volumes. Must be used along with `volumes`. 217 Using a `volume_driver` also allows to use `volumes` with a named volume as 218 well as absolute paths. If `docker.volumes.enabled` is false then volume 219 drivers are disallowed. 220 221 ```hcl 222 config { 223 volumes = [ 224 # Use named volume created outside nomad. 225 "name-of-the-volume:/path/in/container" 226 ] 227 # Name of the Docker Volume Driver used by the container 228 volume_driver = "flocker" 229 } 230 ``` 231 232 * `work_dir` - (Optional) The working directory inside the container. 233 234 ### Container Name 235 236 Nomad creates a container after pulling an image. Containers are named 237 `{taskName}-{allocId}`. This is necessary in order to place more than one 238 container from the same task on a host (e.g. with count > 1). This also means 239 that each container's name is unique across the cluster. 240 241 This is not configurable. 242 243 ### Authentication 244 245 If you want to pull from a private repo (for example on dockerhub or quay.io), 246 you will need to specify credentials in your job via the `auth` option or by 247 storing the credentials in a file and setting the 248 [docker.auth.config](#auth_file) value on the client. 249 250 The `auth` object supports the following keys: 251 252 * `username` - (Optional) The account username. 253 254 * `password` - (Optional) The account password. 255 256 * `email` - (Optional) The account email. 257 258 * `server_address` - (Optional) The server domain/IP without the protocol. 259 Docker Hub is used by default. 260 261 Example: 262 263 ```hcl 264 task "example" { 265 driver = "docker" 266 267 config { 268 image = "secret/service" 269 270 auth { 271 username = "dockerhub_user" 272 password = "dockerhub_password" 273 } 274 } 275 } 276 ``` 277 278 !> **Be Careful!** At this time these credentials are stored in Nomad in plain 279 text. Secrets management will be added in a later release. 280 281 ## Networking 282 283 Docker supports a variety of networking configurations, including using host 284 interfaces, SDNs, etc. Nomad uses `bridged` networking by default, like Docker. 285 286 You can specify other networking options, including custom networking plugins 287 in Docker 1.9. **You may need to perform additional configuration on the host 288 in order to make these work.** This additional configuration is outside the 289 scope of Nomad. 290 291 ### Allocating Ports 292 293 You can allocate ports to your task using the port syntax described on the 294 [networking page](/docs/job-specification/network.html). Here is a recap: 295 296 ```hcl 297 task "example" { 298 driver = "docker" 299 300 resources { 301 network { 302 port "http" {} 303 port "https" {} 304 } 305 } 306 } 307 ``` 308 309 ### Forwarding and Exposing Ports 310 311 A Docker container typically specifies which port a service will listen on by 312 specifying the `EXPOSE` directive in the `Dockerfile`. 313 314 Because dynamic ports will not match the ports exposed in your Dockerfile, 315 Nomad will automatically expose all of the ports it allocates to your 316 container. 317 318 These ports will be identified via environment variables. For example: 319 320 ``` 321 port "http" {} 322 ``` 323 324 If Nomad allocates port `23332` to your task for `http`, `23332` will be 325 automatically exposed and forwarded to your container, and the driver will set 326 an environment variable `NOMAD_PORT_http` with the value `23332` that you can 327 read inside your container. 328 329 This provides an easy way to use the `host` networking option for better 330 performance. 331 332 ### Using the Port Map 333 334 If you prefer to use the traditional port-mapping method, you can specify the 335 `port_map` option in your job specification. It looks like this: 336 337 ```hcl 338 task "example" { 339 driver = "docker" 340 341 config { 342 image = "redis" 343 344 port_map { 345 redis = 6379 346 } 347 } 348 349 resources { 350 network { 351 mbits = 20 352 port "redis" {} 353 } 354 } 355 } 356 ``` 357 358 If Nomad allocates port `23332` to your task, the Docker driver will 359 automatically setup the port mapping from `23332` on the host to `6379` in your 360 container, so it will just work! 361 362 Note that by default this only works with `bridged` networking mode. It may 363 also work with custom networking plugins which implement the same API for 364 expose and port forwarding. 365 366 ### Networking Protocols 367 368 The Docker driver configures ports on both the `tcp` and `udp` protocols. 369 370 This is not configurable. 371 372 ### Other Networking Modes 373 374 Some networking modes like `container` or `none` will require coordination 375 outside of Nomad. First-class support for these options may be improved later 376 through Nomad plugins or dynamic job configuration. 377 378 ## Client Requirements 379 380 Nomad requires Docker to be installed and running on the host alongside the 381 Nomad agent. Nomad was developed against Docker `1.8.2` and `1.9`. 382 383 By default Nomad communicates with the Docker daemon using the daemon's unix 384 socket. Nomad will need to be able to read/write to this socket. If you do not 385 run Nomad as root, make sure you add the Nomad user to the Docker group so 386 Nomad can communicate with the Docker daemon. 387 388 For example, on Ubuntu you can use the `usermod` command to add the `vagrant` 389 user to the `docker` group so you can run Nomad without root: 390 391 sudo usermod -G docker -a vagrant 392 393 For the best performance and security features you should use recent versions 394 of the Linux Kernel and Docker daemon. 395 396 ## Client Configuration 397 398 The `docker` driver has the following [client configuration 399 options](/docs/agent/configuration/client.html#options): 400 401 * `docker.endpoint` - Defaults to `unix:///var/run/docker.sock`. You will need 402 to customize this if you use a non-standard socket (HTTP or another 403 location). 404 405 * `docker.auth.config` <a id="auth_file"></a>- Allows an operator to specify a 406 JSON file which is in the dockercfg format containing authentication 407 information for a private registry. 408 409 * `docker.tls.cert` - Path to the server's certificate file (`.pem`). Specify 410 this along with `docker.tls.key` and `docker.tls.ca` to use a TLS client to 411 connect to the docker daemon. `docker.endpoint` must also be specified or 412 this setting will be ignored. 413 414 * `docker.tls.key` - Path to the client's private key (`.pem`). Specify this 415 along with `docker.tls.cert` and `docker.tls.ca` to use a TLS client to 416 connect to the docker daemon. `docker.endpoint` must also be specified or 417 this setting will be ignored. 418 419 * `docker.tls.ca` - Path to the server's CA file (`.pem`). Specify this along 420 with `docker.tls.cert` and `docker.tls.key` to use a TLS client to connect to 421 the docker daemon. `docker.endpoint` must also be specified or this setting 422 will be ignored. 423 424 * `docker.cleanup.image` Defaults to `true`. Changing this to `false` will 425 prevent Nomad from removing images from stopped tasks. 426 427 * `docker.cleanup.image.delay` A time duration that defaults to `3m`. The delay 428 controls how long Nomad will wait between an image being unused and deleting 429 it. If a tasks is received that uses the same image within the delay, the 430 image will be reused. 431 432 * `docker.volumes.enabled`: Defaults to `true`. Allows tasks to bind host paths 433 (`volumes`) inside their container and use volume drivers (`volume_driver`). 434 Binding relative paths is always allowed and will be resolved relative to the 435 allocation's directory. 436 437 * `docker.volumes.selinuxlabel`: Allows the operator to set a SELinux 438 label to the allocation and task local bind-mounts to containers. If used 439 with `docker.volumes.enabled` set to false, the labels will still be applied 440 to the standard binds in the container. 441 442 * `docker.privileged.enabled` Defaults to `false`. Changing this to `true` will 443 allow containers to use `privileged` mode, which gives the containers full 444 access to the host's devices. Note that you must set a similar setting on the 445 Docker daemon for this to work. 446 447 Note: When testing or using the `-dev` flag you can use `DOCKER_HOST`, 448 `DOCKER_TLS_VERIFY`, and `DOCKER_CERT_PATH` to customize Nomad's behavior. If 449 `docker.endpoint` is set Nomad will **only** read client configuration from the 450 config file. 451 452 An example is given below: 453 454 ```hcl 455 client { 456 options { 457 "docker.cleanup.image" = "false" 458 } 459 } 460 ``` 461 462 ## Client Attributes 463 464 The `docker` driver will set the following client attributes: 465 466 * `driver.docker` - This will be set to "1", indicating the driver is 467 available. 468 * `driver.docker.version` - This will be set to version of the docker server. 469 470 Here is an example of using these properties in a job file: 471 472 ```hcl 473 job "docs" { 474 # Require docker version higher than 1.2. 475 constraint { 476 attribute = "${driver.docker.version}" 477 operator = ">" 478 version = "1.2" 479 } 480 } 481 ``` 482 483 ## Resource Isolation 484 485 ### CPU 486 487 Nomad limits containers' CPU based on CPU shares. CPU shares allow containers 488 to burst past their CPU limits. CPU limits will only be imposed when there is 489 contention for resources. When the host is under load your process may be 490 throttled to stabilize QOS depending on how many shares it has. You can see how 491 many CPU shares are available to your process by reading `NOMAD_CPU_LIMIT`. 492 1000 shares are approximately equal to 1Ghz. 493 494 Please keep the implications of CPU shares in mind when you load test workloads 495 on Nomad. 496 497 ### Memory 498 499 Nomad limits containers' memory usage based on total virtual memory. This means 500 that containers scheduled by Nomad cannot use swap. This is to ensure that a 501 swappy process does not degrade performance for other workloads on the same 502 host. 503 504 Since memory is not an elastic resource, you will need to make sure your 505 container does not exceed the amount of memory allocated to it, or it will be 506 terminated or crash when it tries to malloc. A process can inspect its memory 507 limit by reading `NOMAD_MEMORY_LIMIT`, but will need to track its own memory 508 usage. Memory limit is expressed in megabytes so 1024 = 1Gb. 509 510 ### IO 511 512 Nomad's Docker integration does not currently provide QOS around network or 513 filesystem IO. These will be added in a later release. 514 515 ### Security 516 517 Docker provides resource isolation by way of 518 [cgroups and namespaces](https://docs.docker.com/introduction/understanding-docker/#the-underlying-technology). 519 Containers essentially have a virtual file system all to themselves. If you 520 need a higher degree of isolation between processes for security or other 521 reasons, it is recommended to use full virtualization like 522 [QEMU](/docs/drivers/qemu.html). 523 524 ## Docker For Mac Caveats 525 526 Docker For Mac runs docker inside a small VM and then allows access to parts of 527 the host filesystem into that VM. At present, nomad uses a syslog server bound to 528 a unix socket within a path that both the host and the VM can access to forward 529 log messages back to nomad. But at present, Docker For Mac does not work for 530 unix domain sockets (https://github.com/docker/for-mac/issues/483) in one of 531 these shared paths. 532 533 As a result, using nomad with the docker driver on OS X/macOS will work, but no 534 logs will be available to nomad. Users must use the native docker facilities to 535 examine the logs of any jobs running under docker. 536 537 In the future, we will resolve this issue, one way or another.