github.com/smintz/nomad@v0.8.3/website/source/guides/quotas.html.md (about) 1 --- 2 layout: "guides" 3 page_title: "Resource Quotas" 4 sidebar_current: "guides-quotas" 5 description: |- 6 Nomad Enterprise provides support for resource quotas, which allow operators 7 to restrict the aggregate resource usage of namespaces. 8 --- 9 10 # Resource Quotas 11 12 [Nomad Enterprise](https://www.hashicorp.com/products/nomad/) provides support 13 for resource quotas, which allow operators to restrict the aggregate resource 14 usage of namespaces. 15 16 ~> **Enterprise Only!** This functionality only exists in Nomad Enterprise. 17 This is not present in the open source version of Nomad. 18 19 ## Use Case 20 21 When many teams or users are sharing Nomad clusters, there is the concern that a 22 single user could use more than their fair share of resources. Resource quotas 23 provide a mechanism for cluster administrators to restrict the resources that a 24 [namespace](/guides/namespaces.html) has access to. 25 26 ## Quotas Objects 27 28 Quota specifications are first class objects in Nomad. A quota specification 29 has a unique name, an optional human readable description and a set of quota 30 limits. The quota limits defines the allowed resource usage within a region. 31 32 Quota objects are shareable among namespaces. This allows an operator to define 33 higher level quota specifications, such as a `prod-api` quota, and multiple 34 namespaces can apply the `prod-api` quota specification. 35 36 When a quota specification is attached to a namespace, all resource usage by 37 jobs in the namespaces are accounted toward the quota limits. If the resource is 38 exhausted, allocations with the namespaces will be queued until resources become 39 available by either other jobs finishing or the quota being expanded. 40 41 ## Working with Quotas 42 43 For specific details about working with quotas, see the [quotas 44 commands](/docs/commands/quota.html) and [HTTP API](/api/quotas.html) 45 documentation. 46 47 ### Creating quotas: 48 49 Resource quotas can be interacted with using the `nomad quota` subcommand. To 50 get started with creating a quota specification, run `nomad quota init` which 51 produces an example quota specification: 52 53 ``` 54 $ nomad quota init 55 Example quota specification written to spec.hcl 56 57 $ cat spec.hcl 58 name = "default-quota" 59 description = "Limit the shared default namespace" 60 61 # Create a limit for the global region. Additional limits may 62 # be specified in-order to limit other regions. 63 limit { 64 region = "global" 65 region_limit { 66 cpu = 2500 67 memory = 1000 68 } 69 } 70 ``` 71 72 A quota specification is composed of one or more resource limits. Each limit 73 applies to a particular Nomad region. Within the limit object, operators can 74 specify the allowed CPU and memory usage. 75 76 To create the particular quota, it is as simple as running: 77 78 ``` 79 $ nomad quota apply spec.hcl 80 Successfully applied quota specification "default-quota"! 81 82 $ nomad quota list 83 Name Description 84 default-quota Limit the shared default namespace 85 api-prod Production instances of backend API servers 86 api-qa QA instances of backend API servers 87 web-prod Production instances of webservers 88 web-qa QA instances of webservers 89 ``` 90 91 ### Attaching Quotas to Namespaces 92 93 In order for a quota to be enforced, we have to attach the quota specification 94 to a namespace. This can be done using the `nomad namespace apply` command. 95 We could add the quota specification we just created to the `default` namespace 96 as follows: 97 98 ``` 99 $ nomad namespace apply -quota default-quota default 100 Successfully applied namespace "default"! 101 ``` 102 103 ### Viewing Quotas 104 105 Lets now run a job in the default namespace now that we have attached a quota: 106 107 ``` 108 $ nomad job init 109 Example job file written to example.nomad 110 111 $ nomad job run -detach example.nomad 112 Job registration successful 113 Evaluation ID: 985a1df8-0221-b891-5dc1-4d31ad4e2dc3 114 115 $ nomad quota status default-quota 116 Name = default-quota 117 Description = Limit the shared default namespace 118 Limits = 1 119 120 Quota Limits 121 Region CPU Usage Memory Usage 122 global 500 / 2500 256 / 1000 123 ``` 124 125 We can see the newly created job is accounted against the quota specification 126 since it is being run in a namespace that has attached the quota. Now let us 127 scale up the job from `count = 1` to `count = 4`: 128 129 ``` 130 # Change count 131 $ nomad job run -detach example.nomad 132 Job registration successful 133 Evaluation ID: ce8e1941-0189-b866-3dc4-7cd92dc38a69 134 135 $ nomad status example 136 ID = example 137 Name = example 138 Submit Date = 10/16/17 10:51:32 PDT 139 Type = service 140 Priority = 50 141 Datacenters = dc1 142 Status = running 143 Periodic = false 144 Parameterized = false 145 146 Summary 147 Task Group Queued Starting Running Failed Complete Lost 148 cache 1 0 3 0 0 0 149 150 Placement Failure 151 Task Group "cache": 152 * Quota limit hit "memory exhausted (1024 needed > 1000 limit)" 153 154 Latest Deployment 155 ID = 7cd98a69 156 Status = running 157 Description = Deployment is running 158 159 Deployed 160 Task Group Desired Placed Healthy Unhealthy 161 cache 4 3 0 0 162 163 Allocations 164 ID Node ID Task Group Version Desired Status Created At 165 6d735236 81f72d90 cache 1 run running 10/16/17 10:51:32 PDT 166 ce8e1941 81f72d90 cache 1 run running 10/16/17 10:51:32 PDT 167 9b8e185e 81f72d90 cache 1 run running 10/16/17 10:51:24 PDT 168 ``` 169 170 Here we can see Nomad created two more allocations but did not place the fourth 171 allocation since that would cause the quota to be oversubscribed on memory. 172 173 ### ACLs 174 175 Access to quotas can be restricted using [ACLs](/guides/acl.html). As an 176 example we could create an ACL policy that allows read-only access to quotas. 177 178 ``` 179 # Allow read only access to quotas. 180 quota { 181 policy = "read" 182 } 183 ``` 184 185 Creating or modifying quotas should typically be guarded by ACLs such that users 186 can not bypass enforcement by simply increasing or removing the quota 187 specification. 188 189 ## Resource Limits 190 191 When specifying resource limits the following enforcement behaviors are defined: 192 193 * `limit < 0`: A limit less than zero disallows any access to the resource. 194 195 * `limit == 0`: A limit of zero allows unlimited access to the resource. 196 197 * `limit > 0`: A limit greater than zero enforces that the consumption is less 198 than or equal to the given limit. 199 200 ## Federation 201 202 Nomad makes working with quotas in a federated cluster simple by replicating 203 quota specifications from the [authoritative Nomad 204 region](/docs/agent/configuration/server.html#authoritative_region). This allows 205 operators to interact with a single cluster but create quota specifications that 206 apply to all Nomad clusters. 207 208 As an example, we can create a quota specification that applies to two regions: 209 210 ``` 211 name = "federated-example" 212 description = "A single quota spec effecting multiple regions" 213 214 # Create a limits for two regions 215 limit { 216 region = "europe" 217 region_limit { 218 cpu = 20000 219 memory = 10000 220 } 221 } 222 223 limit { 224 region = "asia" 225 region_limit { 226 cpu = 10000 227 memory = 5000 228 } 229 } 230 ``` 231 232 If we apply this, and attach it to a namespace with jobs in each region, we can 233 see how the enforcement applies across federated clusters. 234 235 ``` 236 $ nomad quota apply spec.hcl 237 Successfully applied quota specification "federated-example"! 238 239 $ nomad quota status federated-example 240 Name = federated-example 241 Description = A single quota spec effecting multiple regions 242 Limits = 2 243 244 Quota Limits 245 Region CPU Usage Memory Usage 246 asia 2500 / 10000 1000 / 5000 247 europe 8800 / 20000 6000 / 10000 248 ```