github.com/smintz/nomad@v0.8.3/website/source/docs/vault-integration/index.html.md (about) 1 --- 2 layout: "docs" 3 page_title: "Vault Integration" 4 sidebar_current: "docs-vault-integration" 5 description: |- 6 Learn how to integrate with HashiCorp Vault and retrieve Vault tokens for 7 tasks. 8 --- 9 10 # Vault Integration 11 12 Many workloads require access to tokens, passwords, certificates, API keys, and 13 other secrets. To enable secure, auditable and easy access to your secrets, 14 Nomad integrates with HashiCorp's [Vault][]. Nomad servers and clients 15 coordinate with Vault to derive a Vault token that has access to only the Vault 16 policies the tasks needs. Nomad clients make the token available to the task and 17 handle the tokens renewal. Further, Nomad's [`template` block][template] can 18 retrieve secrets from Vault making it easier than ever to secure your 19 infrastructure. 20 21 Note that in order to use Vault with Nomad, you will need to configure and 22 install Vault separately from Nomad. Nomad does not run Vault for you. 23 24 ## Vault Configuration 25 26 To use the Vault integration, Nomad servers must be provided a Vault token. This 27 token can either be a root token or a periodic token with permissions to create 28 from a token role. The root token is the easiest way to get started, but we 29 recommend a token role based token for production installations. Nomad servers 30 will renew the token automatically. **Note that the Nomad clients do not need to 31 be provided with a Vault token.** 32 33 ### Root Token Integration 34 35 If Nomad is given a [root 36 token](https://www.vaultproject.io/docs/concepts/tokens.html#root-tokens), no 37 further configuration is needed as Nomad can derive a token for jobs using any 38 Vault policies. 39 40 ### Token Role based Integration 41 42 Vault's [Token Authentication Backend][auth] supports a concept called "roles". 43 Token roles allow policies to be grouped together and token creation to be 44 delegated to a trusted service such as Nomad. By creating a token role, the set 45 of policies that tasks managed by Nomad can access may be limited compared to 46 giving Nomad a root token. Token roles allow both white-list and blacklist 47 management of policies accessible to the role. 48 49 To configure Nomad and Vault to create tokens against a role, the following must 50 occur: 51 52 1. Create a "nomad-server" policy used by Nomad to create and manage tokens. 53 54 2. Create a Vault token role with the configuration described below. 55 56 3. Configure Nomad to use the created token role. 57 58 4. Give Nomad servers a periodic token with the "nomad-server" policy created 59 above. 60 61 #### Required Vault Policies 62 63 The token Nomad receives must have the capabilities listed below. An explanation 64 for the use of each capability is given. 65 66 ```hcl 67 # Allow creating tokens under "nomad-cluster" token role. The token role name 68 # should be updated if "nomad-cluster" is not used. 69 path "auth/token/create/nomad-cluster" { 70 capabilities = ["update"] 71 } 72 73 # Allow looking up "nomad-cluster" token role. The token role name should be 74 # updated if "nomad-cluster" is not used. 75 path "auth/token/roles/nomad-cluster" { 76 capabilities = ["read"] 77 } 78 79 # Allow looking up the token passed to Nomad to validate # the token has the 80 # proper capabilities. This is provided by the "default" policy. 81 path "auth/token/lookup-self" { 82 capabilities = ["read"] 83 } 84 85 # Allow looking up incoming tokens to validate they have permissions to access 86 # the tokens they are requesting. This is only required if 87 # `allow_unauthenticated` is set to false. 88 path "auth/token/lookup" { 89 capabilities = ["update"] 90 } 91 92 # Allow revoking tokens that should no longer exist. This allows revoking 93 # tokens for dead tasks. 94 path "auth/token/revoke-accessor" { 95 capabilities = ["update"] 96 } 97 98 # Allow checking the capabilities of our own token. This is used to validate the 99 # token upon startup. 100 path "sys/capabilities-self" { 101 capabilities = ["update"] 102 } 103 104 # Allow our own token to be renewed. 105 path "auth/token/renew-self" { 106 capabilities = ["update"] 107 } 108 ``` 109 110 The above [`nomad-server` policy](/data/vault/nomad-server-policy.hcl) is 111 available for download. Below is an example of writing this policy to Vault: 112 113 ``` 114 # Download the policy 115 $ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L 116 117 # Write the policy to Vault 118 $ vault policy write nomad-server nomad-server-policy.hcl 119 ``` 120 121 #### Vault Token Role Configuration 122 123 A Vault token role must be created for use by Nomad. The token role can be used 124 to manage what Vault policies are accessible by jobs submitted to Nomad. The 125 policies can be managed as a whitelist by using `allowed_policies` in the token 126 role definition or as a blacklist by using `disallowed_policies`. 127 128 If using `allowed_policies`, tasks may only request Vault policies that are in 129 the list. If `disallowed_policies` is used, task may request any policy that is 130 not in the `disallowed_policies` list. There are trade-offs to both approaches 131 but generally it is easier to use the blacklist approach and add policies that 132 you would not like tasks to have access to into the `disallowed_policies` list. 133 134 An example token role definition is given below: 135 136 ```json 137 { 138 "disallowed_policies": "nomad-server", 139 "explicit_max_ttl": 0, 140 "name": "nomad-cluster", 141 "orphan": true, 142 "period": 259200, 143 "renewable": true 144 } 145 ``` 146 147 148 ##### Token Role Requirements 149 150 Nomad checks that token role has an appropriate configuration for use by the 151 cluster. Fields that are checked are documented below as well as descriptions of 152 the important fields. See Vault's [Token Authentication Backend][auth] 153 documentation for all possible fields and more complete documentation. 154 155 * `allowed_policies` - Specifies the list of allowed policies as a 156 comma-separated string. This list should contain all policies that jobs running 157 under Nomad should have access to. 158 159 * `disallowed_policies` - Specifies the list of disallowed policies as a 160 comma-separated string. This list should contain all policies that jobs running 161 under Nomad should **not** have access to. The policy created above that 162 grants Nomad the ability to generate tokens from the token role should be 163 included in list of disallowed policies. This prevents tokens created by 164 Nomad from generating new tokens with different policies than those granted 165 by Nomad. 166 167 A regression occurred in Vault 0.6.4 when validating token creation using a 168 token role with `disallowed_policies` such that it is not usable with 169 Nomad. This will be remedied in 0.6.5 and does not effect earlier versions 170 of Vault. 171 172 * `explicit_max_ttl` - Specifies the max TTL of a token. **Must be set to `0`** to 173 allow periodic tokens. 174 175 * `name` - Specifies the name of the policy. We recommend using the name 176 `nomad-cluster`. If a different name is chosen, replace the token role in the 177 above policy. 178 179 * `orphan` - Specifies whether tokens created against this token role will be 180 orphaned and have no parents. Nomad does not enforce the value of this field 181 but understanding the implications of each value is important. 182 183 If set to false, all tokens will be revoked when the Vault token given to 184 Nomad expires. This makes it easy to revoke all tokens generated by Nomad but 185 forces all Nomad servers to use the same Vault token, even through upgrades of 186 Nomad servers. If the Vault token that was given to Nomad and used to generate 187 a tasks token expires, the token used by the task will also be revoked which 188 is not ideal. 189 190 When set to true, the tokens generated for tasks will not be revoked when 191 Nomad's token is revoked. However Nomad will still revoke tokens when the 192 allocation is no longer running, minimizing the lifetime of any task's token. 193 With orphaned enabled, each Nomad server may also use a unique Vault token, 194 making bootstrapping and upgrading simpler. As such, **setting `orphan = true` 195 is the recommended setting**. 196 197 * `period` - Specifies the length the TTL is extended by each renewal in 198 seconds. It is suggested to set this value on the order of magnitude of 3 days 199 (259200 seconds) to avoid a large renewal request rate to Vault. **Must be set 200 to a positive value**. 201 202 * `renewable` - Specifies whether created tokens are renewable. **Must be set to 203 `true`**. This allows Nomad to renew tokens for tasks. 204 205 The above [`nomad-cluster` token role](/data/vault/nomad-cluster-role.json) is 206 available for download. Below is an example of writing this role to Vault: 207 208 ``` 209 # Download the token role 210 $ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L 211 212 # Create the token role with Vault 213 $ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json 214 ``` 215 216 217 #### Example Configuration 218 219 To make getting started easy, the basic [`nomad-server` 220 policy](/data/vault/nomad-server-policy.hcl) and 221 [`nomad-cluster` role](/data/vault/nomad-cluster-role.json) described above are 222 available for download. 223 224 The below example assumes Vault is accessible, unsealed and the operator has 225 appropriate permissions. 226 227 ```shell 228 # Download the policy and token role 229 $ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L 230 $ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L 231 232 # Write the policy to Vault 233 $ vault policy write nomad-server nomad-server-policy.hcl 234 235 # Create the token role with Vault 236 $ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json 237 ``` 238 239 #### Retrieving the Token Role based Token 240 241 After the token role is created, a token suitable for the Nomad servers may be 242 retrieved by issuing the following Vault command: 243 244 ``` 245 $ vault token create -policy nomad-server -period 72h -orphan 246 Key Value 247 --- ----- 248 token f02f01c2-c0d1-7cb7-6b88-8a14fada58c0 249 token_accessor 8cb7fcb3-9a4f-6fbf-0efc-83092bb0cb1c 250 token_duration 259200s 251 token_renewable true 252 token_policies [default nomad-server] 253 ``` 254 255 The `-orphan` flag is included when generating the Nomad server token above to 256 prevent revocation of the token when its parent expires. Vault typically 257 creates tokens with a parent-child relationship. When an ancestor token is 258 revoked, all of its descendant tokens and their associated leases are revoked 259 as well. 260 261 When generating Nomad's Vault token, we need to ensure that revocation of the 262 parent token does not revoke Nomad's token. To prevent this behavior we 263 specify the `-orphan` flag when we create the Nomad's Vault token. All 264 other tokens generated by Nomad for jobs will be generated using the policy 265 default of `orphan = false`. 266 267 More information about creating orphan tokens can be found in 268 [Vault's Token Hierarchies and Orphan Tokens documentation][tokenhierarchy]. 269 270 The token can then be set in the server configuration's 271 [`vault` stanza][config], as a command-line flag, or via an environment 272 variable. 273 274 ``` 275 $ VAULT_TOKEN=f02f01c2-c0d1-7cb7-6b88-8a14fada58c0 nomad agent -config /path/to/config 276 ``` 277 278 An example of what may be contained in the configuration is shown below. For 279 complete documentation please see the [Nomad agent Vault integration][config] 280 configuration. 281 282 ```hcl 283 vault { 284 enabled = true 285 ca_path = "/etc/certs/ca" 286 cert_file = "/var/certs/vault.crt" 287 key_file = "/var/certs/vault.key" 288 address = "https://vault.service.consul:8200" 289 create_from_role = "nomad-cluster" 290 } 291 ``` 292 293 ## Agent Configuration 294 295 To enable Vault integration, please see the [Nomad agent Vault 296 integration][config] configuration. 297 298 ## Vault Definition Syntax 299 300 To configure a job to retrieve Vault tokens, please see the [`vault` job 301 specification documentation][vault-spec]. 302 303 ## Troubleshooting 304 305 ### Invalid Vault token 306 307 Upon startup, Nomad will attempt to connect to the specified Vault server. Nomad 308 will lookup the passed token and if the token is from a token role, the token 309 role will be validated. Nomad will not shutdown if given an invalid Vault token, 310 but will log the reasons the token is invalid and disable Vault integration. 311 312 ### Permission Denied errors 313 314 If you are using a Vault version less than 0.7.1 with a Nomad version greater than or equal to 0.6.1, you will need to update your task's policy (listed in [the `vault` stanza of the job specification][vault-spec]) to add the following: 315 316 ``` 317 path "sys/leases/renew" { 318 capabilities = ["update"] 319 } 320 ``` 321 322 This is included in Vault's "default" policy beginning with Vault 0.7.1 and is relied upon by Nomad's Vault integration beginning with Nomad 0.6.1. If you're using a newer Nomad version with an older Vault version, your default policy may not automatically include this and you will see "permission denied" errors in your Nomad logs similar to the following: 323 324 ``` 325 Code: 403. Errors: 326 URL: PUT https://vault:8200/v1/sys/leases/renew 327 * permission denied 328 ``` 329 330 ## Assumptions 331 332 - Vault 0.6.2 or later is needed. 333 334 [auth]: https://www.vaultproject.io/docs/auth/token.html "Vault Authentication Backend" 335 [config]: /docs/agent/configuration/vault.html "Nomad Vault Configuration Block" 336 [createfromrole]: /docs/agent/configuration/vault.html#create_from_role "Nomad vault create_from_role Configuration Flag" 337 [template]: /docs/job-specification/template.html "Nomad template Job Specification" 338 [vault]: https://www.vaultproject.io/ "Vault by HashiCorp" 339 [vault-spec]: /docs/job-specification/vault.html "Nomad Vault Job Specification" 340 [tokenhierarchy]: https://www.vaultproject.io/docs/concepts/tokens.html#token-hierarchies-and-orphan-tokens "Vault Tokens - Token Hierarchies and Orphan Tokens"