github.com/ferranbt/nomad@v0.9.3-0.20190607002617-85c449b7667c/website/source/guides/security/acl.html.markdown (about) 1 --- 2 layout: "guides" 3 page_title: "Access Control" 4 sidebar_current: "guides-security-acl" 5 description: |- 6 Nomad provides an optional Access Control List (ACL) system which can be used to control 7 access to data and APIs. The ACL is Capability-based, relying on tokens which are 8 associated with policies to determine which fine grained rules can be applied. 9 --- 10 11 # Access Control 12 13 Nomad provides an optional Access Control List (ACL) system which can be used to control access to data and APIs. The ACL is [Capability-based](https://en.wikipedia.org/wiki/Capability-based_security), relying on tokens which are associated with policies to determine which fine grained rules can be applied. Nomad's capability based ACL system is very similar to the design of [AWS IAM](https://aws.amazon.com/iam/). 14 15 # ACL System Overview 16 17 The ACL system is designed to be easy to use and fast to enforce while providing administrative insight. At the highest level, there are three major components to the ACL system: 18 19  20 21 * **ACL Policies**. No permissions are granted by default, making Nomad a default-deny or whitelist system. Policies allow a set of capabilities or actions to be granted or whitelisted. For example, a "readonly" policy might only grant the ability to list and inspect running jobs, but not to submit new ones. 22 23 * **ACL Tokens**. Requests to Nomad are authenticated by using bearer token. Each ACL token has a public Accessor ID which is used to name a token, and a Secret ID which is used to make requests to Nomad. The Secret ID is provided using a request header (`X-Nomad-Token`) and is used to authenticate the caller. Tokens are either `management` or `client` types. The `management` tokens are effectively "root" in the system, and can perform any operation. The `client` tokens are associated with one or more ACL policies which grant specific capabilities. 24 25 * **Capabilities**. Capabilities are the set of actions that can be performed. This includes listing jobs, submitting jobs, querying nodes, etc. A `management` token is granted all capabilities, while `client` tokens are granted specific capabilities via ACL Policies. The full set of capabilities is discussed below in the rule specifications. 26 27 ### ACL Policies 28 29 An ACL policy is a named set of rules. Each policy must have a unique name, an optional description, and a rule set. 30 A client ACL token can be associated with multiple policies, and a request is allowed if _any_ of the associated policies grant the capability. 31 Management tokens cannot be associated with policies because they are granted all capabilities. 32 33 The special `anonymous` policy can be defined to grant capabilities to requests which are made anonymously. An anonymous request is a request made to Nomad without the `X-Nomad-Token` header specified. This can be used to allow anonymous users to list jobs and view their status, while requiring authenticated requests to submit new jobs or modify existing jobs. By default, there is no `anonymous` policy set meaning all anonymous requests are denied. 34 35 ### ACL Tokens 36 37 ACL tokens are used to authenticate requests and determine if the caller is authorized to perform an action. Each ACL token has a public Accessor ID which is used to identify the token, a Secret ID which is used to make requests to Nomad, and an optional human readable name. All `client` type tokens are associated with one or more policies, and can perform an action if any associated policy allows it. Tokens can be associated with policies which do not exist, which are the equivalent of granting no capabilities. The `management` type tokens cannot be associated with policies, but can perform any action. 38 39 When ACL tokens are created, they can be optionally marked as `Global`. This causes them to be created in the authoritative region and replicated to all other regions. Otherwise, tokens are created locally in the region the request was made and not replicated. Local tokens cannot be used for cross-region requests since they are not replicated between regions. 40 41 ### Capabilities and Scope 42 43 The following table summarizes the ACL Rules that are available for constructing policy rules: 44 45 | Policy | Scope | 46 | ---------- | -------------------------------------------- | 47 | [namespace](#namespace-rules) | Job related operations by namespace | 48 | [agent](#agent-rules) | Utility operations in the Agent API | 49 | [node](#node-rules) | Node-level catalog operations | 50 | [operator](#operator-rules) | Cluster-level operations in the Operator API | 51 | [quota](#quota-rules) | Quota specification related operations | 52 53 Constructing rules from these policies is covered in detail in the Rule Specification section below. 54 55 ### Multi-Region Configuration 56 57 Nomad supports multi-datacenter and multi-region configurations. A single region is able to service multiple datacenters, and all servers in a region replicate their state between each other. In a multi-region configuration, there is a set of servers per region. Each region operates independently and is loosely coupled to allow jobs to be scheduled in any region and requests to flow transparently to the correct region. 58 59 When ACLs are enabled, Nomad depends on an "authoritative region" to act as a single source of truth for ACL policies and global ACL tokens. The authoritative region is configured in the [`server` stanza](/docs/configuration/server.html) of agents, and all regions must share a single authoritative source. Any ACL policies or global ACL tokens are created in the authoritative region first. All other regions replicate ACL policies and global ACL tokens to act as local mirrors. This allows policies to be administered centrally, and for enforcement to be local to each region for low latency. 60 61 Global ACL tokens are used to allow cross region requests. Standard ACL tokens are created in a single target region and not replicated. This means if a request takes place between regions, global tokens must be used so that both regions will have the token registered. 62 63 # Configuring ACLs 64 65 ACLs are not enabled by default, and must be enabled. Clients and Servers need to set `enabled` in the [`acl` stanza](/docs/configuration/acl.html). This enables the [ACL Policy](/api/acl-policies.html) and [ACL Token](/api/acl-tokens.html) APIs, as well as endpoint enforcement. 66 67 For multi-region configurations, all servers must be configured to use a single [authoritative region](/docs/configuration/server.html#authoritative_region). The authoritative region is responsible for managing ACL policies and global tokens. Servers in other regions will replicate policies and global tokens to act as a mirror, and must have their [`replication_token`](/docs/configuration/acl.html#replication_token) configured. 68 69 # Bootstrapping ACLs 70 71 Bootstrapping ACLs on a new cluster requires a few steps, outlined below: 72 73 ### Enable ACLs on Nomad Servers 74 75 The APIs needed to manage policies and tokens are not enabled until ACLs are enabled. To begin, we need to enable the ACLs on the servers. If a multi-region setup is used, the authoritative region should be enabled first. For each server: 76 77 1. Set `enabled = true` in the [`acl` stanza](/docs/configuration/acl.html#enabled). 78 1. Set `authoritative_region` in the [`server` stanza](/docs/configuration/server.html#authoritative_region). 79 1. For servers outside the authoritative region, set `replication_token` in the [`acl` stanza](/docs/configuration/acl.html#replication_token). Replication tokens should be `management` type tokens which are either created in the authoritative region, or created as Global tokens. 80 1. Restart the Nomad server to pick up the new configuration. 81 82 Please take care to restart the servers one at a time, and ensure each server has joined and is operating correctly before restarting another. 83 84 ### Generate the initial token 85 86 Once the ACL system is enabled, we need to generate our initial token. This first token is used to bootstrap the system and care should be taken not to lose it. Once the ACL system is enabled, we use the [Bootstrap CLI](/docs/commands/acl/bootstrap.html): 87 88 ```text 89 $ nomad acl bootstrap 90 Accessor ID = 5b7fd453-d3f7-6814-81dc-fcfe6daedea5 91 Secret ID = 9184ec35-65d4-9258-61e3-0c066d0a45c5 92 Name = Bootstrap Token 93 Type = management 94 Global = true 95 Policies = n/a 96 Create Time = 2017-09-11 17:38:10.999089612 +0000 UTC 97 Create Index = 7 98 Modify Index = 7 99 ``` 100 101 Once the initial bootstrap is performed, it cannot be performed again unless [reset](#reseting-acl-bootstrap). Make sure to save this AccessorID and SecretID. 102 The bootstrap token is a `management` type token, meaning it can perform any operation. It should be used to setup the ACL policies and create additional ACL tokens. The bootstrap token can be deleted and is like any other token, so care should be taken to not revoke all management tokens. 103 104 ### Enable ACLs on Nomad Clients 105 106 To enforce client endpoints, we need to enable ACLs on clients as well. This is simpler than servers, and we just need to set `enabled = true` in the [`acl` stanza](/docs/configuration/acl.html). Once configured, we need to restart the client for the change. 107 108 109 ### Set an Anonymous Policy (Optional) 110 111 The ACL system uses a whitelist or default-deny model. This means by default no permissions are granted. 112 For clients making requests without ACL tokens, we may want to grant some basic level of access. This is done by setting rules 113 on the special "anonymous" policy. This policy is applied to any requests made without a token. 114 115 To permit anonymous users to read, we can setup the following policy: 116 117 ```text 118 # Store our token secret ID 119 $ export NOMAD_TOKEN="BOOTSTRAP_SECRET_ID" 120 121 # Write out the payload 122 $ cat > payload.json <<EOF 123 { 124 "Name": "anonymous", 125 "Description": "Allow read-only access for anonymous requests", 126 "Rules": " 127 namespace \"default\" { 128 policy = \"read\" 129 } 130 agent { 131 policy = \"read\" 132 } 133 node { 134 policy = \"read\" 135 } 136 " 137 } 138 EOF 139 140 # Install the policy 141 $ curl --request POST \ 142 --data @payload.json \ 143 -H "X-Nomad-Token: $NOMAD_TOKEN" \ 144 https://localhost:4646/v1/acl/policy/anonymous 145 146 # Verify anonymous request works 147 $ curl https://localhost:4646/v1/jobs 148 ``` 149 150 # Rule Specification 151 152 A core part of the ACL system is the rule language which is used to describe the policy that must be enforced. 153 We make use of the [HashiCorp Configuration Language (HCL)](https://github.com/hashicorp/hcl/) to specify rules. 154 This language is human readable and interoperable with JSON making it easy to machine-generate. Policies can contain any number of rules. 155 156 Policies typically have several dispositions: 157 158 * `read`: allow the resource to be read but not modified 159 * `write`: allow the resource to be read and modified 160 * `deny`: do not allow the resource to be read or modified. Deny takes precedence when multiple policies are associated with a token. 161 162 Specification in the HCL format looks like: 163 164 ```text 165 # Allow read only access to the default namespace 166 namespace "default" { 167 policy = "read" 168 } 169 170 # Allow writing to the `foo` namespace 171 namespace "foo" { 172 policy = "write" 173 } 174 175 agent { 176 policy = "read" 177 } 178 179 node { 180 policy = "read" 181 } 182 183 quota { 184 policy = "read" 185 } 186 ``` 187 188 This is equivalent to the following JSON input: 189 190 ```json 191 { 192 "namespace": { 193 "default": { 194 "policy": "read" 195 }, 196 "foo": { 197 "policy": "write" 198 } 199 }, 200 "agent": { 201 "policy": "read" 202 }, 203 "node": { 204 "policy": "read" 205 }, 206 "quota": { 207 "policy": "read" 208 } 209 } 210 ``` 211 212 The [ACL Policy](/api/acl-policies.html) API allows either HCL or JSON to be used to define the content of the rules section. 213 214 ### Namespace Rules 215 216 The `namespace` policy controls access to a namespace, including the [Jobs API](/api/jobs.html), [Deployments API](/api/deployments.html), [Allocations API](/api/allocations.html), and [Evaluations API](/api/evaluations.html). 217 218 ``` 219 namespace "default" { 220 policy = "write" 221 } 222 223 namespace "sensitive" { 224 policy = "read" 225 } 226 ``` 227 228 Namespace rules are keyed by the namespace name they apply to. When no namespace is specified, the "default" namespace is the one used. For example, the above policy grants write access to the default namespace, and read access to the sensitive namespace. In addition to the coarse grained `policy` specification, the `namespace` stanza allows setting a more fine grained list of `capabilities`. This includes: 229 230 * `deny` - When multiple policies are associated with a token, deny will take precedence and prevent any capabilities. 231 * `list-jobs` - Allows listing the jobs and seeing coarse grain status. 232 * `read-job` - Allows inspecting a job and seeing fine grain status. 233 * `submit-job` - Allows jobs to be submitted or modified. 234 * `dispatch-job` - Allows jobs to be dispatched 235 * `read-logs` - Allows the logs associated with a job to be viewed. 236 * `read-fs` - Allows the filesystem of allocations associated to be viewed. 237 * `sentinel-override` - Allows soft mandatory policies to be overridden. 238 239 The coarse grained policy dispositions are shorthand for the fine grained capabilities: 240 241 * `deny` policy - ["deny"] 242 * `read` policy - ["list-jobs", "read-job"] 243 * `write` policy - ["list-jobs", "read-job", "submit-job", "read-logs", "read-fs", "dispatch-job"] 244 245 When both the policy short hand and a capabilities list are provided, the capabilities are merged: 246 247 ``` 248 # Allow reading jobs and submitting jobs, without allowing access 249 # to view log output or inspect the filesystem 250 namespace "default" { 251 policy = "read" 252 capabilities = ["submit-job"] 253 } 254 ``` 255 256 Namespaces definitions may also include globs, allowing a single policy definition to apply to a set of namespaces. For example, the below policy allows read access to most production namespaces, but allows write access to the "production-api" namespace, and rejects any access to the "production-web" namespace. 257 258 ``` 259 namespace "production-*" { 260 policy = "read" 261 } 262 263 namespace "production-api" { 264 policy = "write" 265 } 266 267 namespace "production-web" { 268 policy = "deny" 269 } 270 ``` 271 272 Namespaces are matched to their policies first by performing a lookup on any _exact match_, before falling back to performing a glob based lookup. When looking up namespaces by glob, the matching policy with the greatest number of matched characters will be chosen. For example: 273 274 ``` 275 namespace "*-web" { 276 policy = "deny" 277 } 278 279 namespace "*" { 280 policy = "write" 281 } 282 ``` 283 284 Will evaluate to deny for `production-web`, because it is 9 characters different from the `"*-web"` rule, but 13 characters different from the `"*"` rule. 285 286 ### Node Rules 287 288 The `node` policy controls access to the [Node API](/api/nodes.html) such as listing nodes or triggering a node drain. 289 Node rules are specified for all nodes using the `node` key: 290 291 ``` 292 node { 293 policy = "read" 294 } 295 ``` 296 297 There's only one node policy allowed per rule set, and its value is set to one of the policy dispositions. 298 299 ### Agent Rules 300 301 The `agent` policy controls access to the utility operations in the [Agent API](/api/agent.html), such as join and leave. 302 Agent rules are specified for all agents using the `agent` key: 303 304 ``` 305 agent { 306 policy = "write" 307 } 308 ``` 309 310 There's only one agent policy allowed per rule set, and its value is set to one of the policy dispositions. 311 312 313 ### Operator Rules 314 315 The `operator` policy controls access to the [Operator API](/api/operator.html). Operator rules look like: 316 317 ``` 318 operator { 319 policy = "read" 320 } 321 ``` 322 323 There's only one operator policy allowed per rule set, and its value is set to one of the policy dispositions. In the example above, the token could be used to query the operator endpoints for diagnostic purposes but not make any changes. 324 325 ### Quota Rules 326 327 The `quota` policy controls access to the quota specification operations in the [Quota API](/api/quotas.html), such as quota creation and deletion. 328 Quota rules are specified for all quotas using the `quota` key: 329 330 ``` 331 quota { 332 policy = "write" 333 } 334 ``` 335 336 There's only one quota policy allowed per rule set, and its value is set to one of the policy dispositions. 337 338 # Advanced Topics 339 340 ### Outages and Multi-Region Replication 341 342 The ACL system takes some steps to ensure operation during outages. Clients nodes maintain a limited 343 cache of ACL tokens and ACL policies that have recently or frequently been used, associated with a time-to-live (TTL). 344 345 When the region servers are unavailable, the clients will automatically ignore the cache TTL, 346 and extend the cache until the outage has recovered. For any policies or tokens that are not cached, 347 they will be treated as missing and denied access until the outage has been resolved. 348 349 Nomad servers have all the policies and tokens locally and can continue serving requests even if 350 quorum is lost. The tokens and policies may become stale during this period as data is not actively 351 replicating, but will be automatically fixed when the outage has been resolved. 352 353 In a multi-region setup, there is a single authoritative region which is the source of truth for 354 ACL policies and global ACL tokens. All other regions asynchronously replicate from the authoritative 355 region. When replication is interrupted, the existing data is used for request processing and may 356 become stale. When the authoritative region is reachable, replication will resume and repair any 357 inconsistency. 358 359 ### Resetting ACL Bootstrap 360 361 If all management tokens are lost, it is possible to reset the ACL bootstrap so that it can be performed again. 362 First, we need to determine the reset index, this can be done by calling the reset endpoint: 363 364 ``` 365 $ nomad acl bootstrap 366 367 Error bootstrapping: Unexpected response code: 500 (ACL bootstrap already done (reset index: 7)) 368 ``` 369 370 Here we can see the `reset index`. To reset the ACL system, we create the 371 `acl-bootstrap-reset` file in the data directory of the **leader** node: 372 373 ``` 374 $ echo 7 >> /nomad-data-dir/server/acl-bootstrap-reset 375 ``` 376 377 With the reset key setup, we can bootstrap like normal: 378 379 ``` 380 $ nomad acl bootstrap 381 Accessor ID = 52d3353d-d7b9-d945-0591-1af608732b76 382 Secret ID = 4b0a41ca-6d32-1853-e64b-de0d347e4525 383 Name = Bootstrap Token 384 Type = management 385 Global = true 386 Policies = n/a 387 Create Time = 2017-09-11 18:38:11.929089612 +0000 UTC 388 Create Index = 11 389 Modify Index = 11 390 ``` 391 392 If we attempt to bootstrap again, we will get a mismatch on the reset index: 393 394 ``` 395 $ nomad acl bootstrap 396 397 Error bootstrapping: Unexpected response code: 500 (Invalid bootstrap reset index (specified 7, reset index: 11)) 398 ``` 399 400 This is because the reset file is in place, but with the incorrect index. 401 The reset file can be deleted, but Nomad will not reset the bootstrap until the index is corrected. 402 403 ## Vault Integration 404 HashiCorp Vault has a secret backend for generating short-lived Nomad tokens. As Vault has a number of 405 authentication backends, it could provide a workflow where a user or orchestration system authenticates 406 using an pre-existing identity service (LDAP, Okta, Amazon IAM, etc.) in order to obtain a short-lived 407 Nomad token. 408 409 ~> HashiCorp Vault is a standalone product with its own set of deployment and 410 configuration best practices. Please review [Vault's 411 documentation](https://www.vaultproject.io/docs/index.html) before deploying it 412 in production. 413 414 For evaluation purposes, a Vault server in "dev" mode can be used. 415 416 ``` 417 $ vault server -dev 418 ==> Vault server configuration: 419 420 Cgo: disabled 421 Cluster Address: https://127.0.0.1:8201 422 Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", tls: "disabled") 423 Log Level: info 424 Mlock: supported: false, enabled: false 425 Redirect Address: http://127.0.0.1:8200 426 Storage: inmem 427 Version: Vault v0.8.3 428 Version Sha: a393b20cb6d96c73e52eb5af776c892b8107a45d 429 430 ==> WARNING: Dev mode is enabled! 431 432 In this mode, Vault is completely in-memory and unsealed. 433 Vault is configured to only have a single unseal key. The root 434 token has already been authenticated with the CLI, so you can 435 immediately begin using the Vault CLI. 436 437 The only step you need to take is to set the following 438 environment variables: 439 440 export VAULT_ADDR='http://127.0.0.1:8200' 441 442 The unseal key and root token are reproduced below in case you 443 want to seal/unseal the Vault or play with authentication. 444 445 Unseal Key: YzFfPgnLl9R1f6bLU7tGqi/PIDhDaAV/tlNDMV5Rrq0= 446 Root Token: f84b587e-5882-bba1-a3f0-d1a3d90ca105 447 ``` 448 449 ### Pre-requisites 450 - Nomad ACL system bootstrapped. 451 - A management token (the bootstrap token can be used, but for production 452 systems it's recommended to have a separate token) 453 - A set of policies created in Nomad 454 - An unsealed Vault server (Vault running in `dev` mode is unsealed 455 automatically upon startup) 456 - Vault must be version 0.9.3 or later to have the Nomad plugin 457 458 ### Configuration 459 Mount the [`nomad`][nomad_backend] secret backend in Vault: 460 461 ``` 462 $ vault mount nomad 463 Successfully mounted 'nomad' at 'nomad'! 464 ``` 465 466 Configure access with Nomad's address and management token: 467 468 ``` 469 $ vault write nomad/config/access \ 470 address=http://127.0.0.1:4646 \ 471 token=adf4238a-882b-9ddc-4a9d-5b6758e4159e 472 Success! Data written to: nomad/config/access 473 ``` 474 475 Vault secret backends have the concept of roles, which are configuration units that group one or more 476 Vault policies to a potential identity attribute, (e.g. LDAP Group membership). The name of the role 477 is specified on the path, while the mapping to policies is done by naming them in a comma separated list, 478 for example: 479 480 ``` 481 $ vault write nomad/role/role-name policies=policyone,policytwo 482 Success! Data written to: nomad/role/role-name 483 ``` 484 485 Similarly, to create management tokens, or global tokens: 486 487 ``` 488 $ vault write nomad/role/role-name type=management global=true 489 Success! Data written to: nomad/role/role-name 490 ``` 491 492 Create a Vault policy to allow different identities to get tokens associated with a particular 493 role: 494 495 ``` 496 $ echo 'path "nomad/creds/role-name" { 497 capabilities = ["read"] 498 }' | vault policy write nomad-user-policy - 499 Policy 'nomad-user-policy' written. 500 ``` 501 502 If you have an existing authentication backend (like LDAP), follow the relevant instructions to create 503 a role available on the [Authentication backends page](https://www.vaultproject.io/docs/auth/index.html). 504 Otherwise, for testing purposes, a Vault token can be generated associated with the policy: 505 506 ``` 507 $ vault token create -policy=nomad-user-policy 508 Key Value 509 --- ----- 510 token deedfa83-99b5-34a1-278d-e8fb76809a5b 511 token_accessor fd185371-7d80-8011-4f45-1bb3af2c2733 512 token_duration 768h0m0s 513 token_renewable true 514 token_policies [default nomad-user-policy] 515 ``` 516 517 Finally obtain a Nomad Token using the existing Vault Token: 518 519 ``` 520 $ vault read nomad/creds/role-name 521 Key Value 522 --- ----- 523 lease_id nomad/creds/role-name/6fb22e25-0cd1-b4c9-494e-aba330c317b9 524 lease_duration 768h0m0s 525 lease_renewable true 526 accessor_id 10b8fb49-7024-2126-8683-ab355b581db2 527 secret_id 8898d19c-e5b3-35e4-649e-4153d63fbea9 528 ``` 529 530 Verify that the token is created correctly in Nomad, looking it up by its accessor: 531 532 ``` 533 $ nomad acl token info 10b8fb49-7024-2126-8683-ab355b581db2 534 Accessor ID = 10b8fb49-7024-2126-8683-ab355b581db2 535 Secret ID = 8898d19c-e5b3-35e4-649e-4153d63fbea9 536 Name = Vault test root 1507307164169530060 537 Type = management 538 Global = true 539 Policies = n/a 540 Create Time = 2017-10-06 16:26:04.170633207 +0000 UTC 541 Create Index = 228 542 Modify Index = 228 543 ``` 544 545 Any user or process with access to Vault can now obtain short lived Nomad Tokens in order to 546 carry out operations, thus centralising the access to Nomad tokens. 547 548 549 [nomad_backend]: https://www.vaultproject.io/docs/secrets/nomad/index.html