github.com/anth0d/nomad@v0.0.0-20221214183521-ae3a0a2cad06/website/content/docs/operations/metrics-reference.mdx (about) 1 --- 2 layout: docs 3 page_title: Metrics Reference 4 description: Learn about the different metrics available in Nomad. 5 --- 6 7 # Metrics Reference 8 9 The Nomad agent collects various runtime metrics about the performance of 10 different libraries and subsystems. These metrics are aggregated on a ten 11 second interval and are retained for one minute. 12 13 This data can be accessed via an HTTP endpoint or via sending a signal to the 14 Nomad process. This data is available via HTTP at `/metrics`. See 15 [Metrics](/api-docs/metrics) for more information. 16 17 To view this data via sending a signal to the Nomad process: on Unix, 18 this is `USR1` while on Windows it is `BREAK`. Once Nomad receives the signal, 19 it will dump the current telemetry information to the agent's `stderr`. 20 21 This telemetry information can be used for debugging or otherwise 22 getting a better view of what Nomad is doing. 23 24 Telemetry information can be streamed to both [statsite](https://github.com/armon/statsite) 25 as well as statsd based on providing the appropriate configuration options. 26 27 To configure the telemetry output please see the [agent 28 configuration](/docs/configuration/telemetry). 29 30 Below is sample output of a telemetry dump: 31 32 ```text 33 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.nomad.broker.total_blocked': 0.000 34 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.nomad.plan.queue_depth': 0.000 35 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.runtime.malloc_count': 7568.000 36 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.runtime.total_gc_runs': 8.000 37 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.nomad.broker.total_ready': 0.000 38 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.runtime.num_goroutines': 56.000 39 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.runtime.sys_bytes': 3999992.000 40 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.runtime.heap_objects': 4135.000 41 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.nomad.heartbeat.active': 1.000 42 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.nomad.broker.total_unacked': 0.000 43 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.nomad.broker.total_waiting': 0.000 44 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.runtime.alloc_bytes': 634056.000 45 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.runtime.free_count': 3433.000 46 [2015-09-17 16:59:40 -0700 PDT][G] 'nomad.runtime.total_gc_pause_ns': 6572135.000 47 [2015-09-17 16:59:40 -0700 PDT][C] 'nomad.memberlist.msg.alive': Count: 1 Sum: 1.000 48 [2015-09-17 16:59:40 -0700 PDT][C] 'nomad.serf.member.join': Count: 1 Sum: 1.000 49 [2015-09-17 16:59:40 -0700 PDT][C] 'nomad.raft.barrier': Count: 1 Sum: 1.000 50 [2015-09-17 16:59:40 -0700 PDT][C] 'nomad.raft.apply': Count: 1 Sum: 1.000 51 [2015-09-17 16:59:40 -0700 PDT][C] 'nomad.nomad.rpc.query': Count: 2 Sum: 2.000 52 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.serf.queue.Query': Count: 6 Sum: 0.000 53 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.nomad.fsm.register_node': Count: 1 Sum: 1.296 54 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.serf.queue.Intent': Count: 6 Sum: 0.000 55 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.runtime.gc_pause_ns': Count: 8 Min: 126492.000 Mean: 821516.875 Max: 3126670.000 Stddev: 1139250.294 Sum: 6572135.000 56 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.raft.leader.dispatchLog': Count: 3 Min: 0.007 Mean: 0.018 Max: 0.039 Stddev: 0.018 Sum: 0.054 57 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.nomad.leader.reconcileMember': Count: 1 Sum: 0.007 58 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.nomad.leader.reconcile': Count: 1 Sum: 0.025 59 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.raft.fsm.apply': Count: 1 Sum: 1.306 60 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.nomad.client.get_allocs': Count: 1 Sum: 0.110 61 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.nomad.worker.dequeue_eval': Count: 29 Min: 0.003 Mean: 363.426 Max: 503.377 Stddev: 228.126 Sum: 10539.354 62 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.serf.queue.Event': Count: 6 Sum: 0.000 63 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.raft.commitTime': Count: 3 Min: 0.013 Mean: 0.037 Max: 0.079 Stddev: 0.037 Sum: 0.110 64 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.nomad.leader.barrier': Count: 1 Sum: 0.071 65 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.nomad.client.register': Count: 1 Sum: 1.626 66 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.nomad.eval.dequeue': Count: 21 Min: 500.610 Mean: 501.753 Max: 503.361 Stddev: 1.030 Sum: 10536.813 67 [2015-09-17 16:59:40 -0700 PDT][S] 'nomad.memberlist.gossip': Count: 12 Min: 0.009 Mean: 0.017 Max: 0.025 Stddev: 0.005 Sum: 0.204 68 ``` 69 70 ### Metric Types 71 72 | Type | Description | Quantiles | 73 | ------- | ------------------------------------------------------------------------------------------------------------------- | --------- | 74 | Gauge | Gauge types report an absolute number at the end of the aggregation interval | false | 75 | Counter | Counts are incremented and flushed at the end of the aggregation interval and then are reset to zero | true | 76 | Timer | Timers measure the time to complete a task and will include quantiles, means, standard deviation, etc per interval. | true | 77 78 ### Tagged Metrics 79 80 Nomad emits metrics in a tagged format. Each metric can support more than one 81 tag, meaning that it is possible to do a match over metrics for datapoints 82 such as a particular datacenter, and return all metrics with this tag. Nomad 83 supports labels for namespaces as well. 84 85 ## Key Metrics 86 87 The metrics in the table below are the most important metrics for monitoring 88 the overall health of a Nomad cluster. 89 90 When telemetry is being streamed to statsite or statsd, `interval` in the 91 table below is defined to be their flush interval. Otherwise, the interval can 92 be assumed to be 10 seconds when retrieving metrics using the above described 93 signals. 94 95 | Metrics | Description | Unit | Type | 96 | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------------------------------ | ------- | 97 | `nomad.runtime.alloc_bytes` | Memory utilization | # of bytes | Gauge | 98 | `nomad.runtime.heap_objects` | Number of objects on the heap. General memory pressure indicator | # of heap objects | Gauge | 99 | `nomad.runtime.num_goroutines` | Number of goroutines and general load pressure indicator | # of goroutines | Gauge | 100 | `nomad.nomad.broker.total_blocked` | Evaluations that are blocked until an existing evaluation for the same job completes | # of evaluations | Gauge | 101 | `nomad.nomad.broker.total_ready` | Number of evaluations ready to be processed | # of evaluations | Gauge | 102 | `nomad.nomad.broker.total_unacked` | Evaluations dispatched for processing but incomplete | # of evaluations | Gauge | 103 | `nomad.nomad.heartbeat.active` | Number of active heartbeat timers. Each timer represents a Nomad Client connection | # of heartbeat timers | Gauge | 104 | `nomad.nomad.heartbeat.invalidate` | The length of time it takes to invalidate a Nomad Client due to failed heartbeats | ms / Heartbeat Invalidation | Timer | 105 | `nomad.nomad.plan.evaluate` | Time to validate a scheduler Plan. Higher values cause lower scheduling throughput. Similar to `nomad.plan.submit` but does not include RPC time or time in the Plan Queue | ms / Plan Evaluation | Timer | 106 | `nomad.nomad.plan.node_rejected` | Number of times a node has had a plan rejected. A node with a high rate of rejections may have an underlying issue causing it to be unschedulable. Refer to [this link][s_port_plan_failure] for more information | # of rejected plans | Counter | 107 | `nomad.nomad.plan.queue_depth` | Number of scheduler Plans waiting to be evaluated | # of plans | Gauge | 108 | `nomad.nomad.plan.submit` | Time to submit a scheduler Plan. Higher values cause lower scheduling throughput | ms / Plan Submit | Timer | 109 | `nomad.nomad.rpc.query` | Number of RPC queries | RPC Queries / `interval` | Counter | 110 | `nomad.nomad.rpc.request_error` | Number of RPC requests being handled that result in an error | RPC Errors / `interval` | Counter | 111 | `nomad.nomad.rpc.request` | Number of RPC requests being handled | RPC Requests / `interval` | Counter | 112 | `nomad.nomad.vault.token_last_renewal` | Time since last successful Vault token renewal | Milliseconds | Gauge | 113 | `nomad.nomad.vault.token_next_renewal` | Time until next Vault token renewal attempt | Milliseconds | Gauge | 114 | `nomad.nomad.worker.invoke_scheduler.<type>` | Time to run the scheduler of the given type | ms / Scheduler Run | Timer | 115 | `nomad.nomad.worker.wait_for_index` | Time waiting for Raft log replication from leader. High delays result in lower scheduling throughput | ms / Raft Index Wait | Timer | 116 | `nomad.raft.apply` | Number of Raft transactions | Raft transactions / `interval` | Counter | 117 | `nomad.raft.leader.lastContact` | Time since last contact to leader. General indicator of Raft latency | ms / Leader Contact | Timer | 118 | `nomad.raft.replication.appendEntries` | Raft transaction commit time | ms / Raft Log Append | Timer | 119 | `nomad.license.expiration_time_epoch` | Time as epoch (seconds since Jan 1 1970) at which license will expire | Seconds | Gauge | 120 121 ## Client Metrics 122 123 The Nomad client emits metrics related to the resource usage of the allocations 124 and tasks running on it and the node itself. Operators have to explicitly turn 125 on publishing host and allocation metrics. Publishing allocation and host 126 metrics can be turned on by setting the value of `publish_allocation_metrics` 127 `publish_node_metrics` to `true`. 128 129 By default the collection interval is 1 second but it can be changed by the 130 changing the value of the `collection_interval` key in the `telemetry` 131 configuration block. 132 133 Please see the [agent configuration](/docs/configuration/telemetry) 134 page for more details. 135 136 Additional labels are emitted for [parameterized](/docs/job-specification/parameterized) and 137 [periodic](/docs/job-specification/parameterized) jobs. Nomad 138 emits the parent job id as a new label `parent_id`. Also, the labels `dispatch_id` 139 and `periodic_id` are emitted, containing the ID of the specific invocation of the 140 parameterized or periodic job respectively. For example, a dispatch job with the id 141 `myjob/dispatch-1312323423423`, will have the following labels. 142 143 | Label | Value | 144 | ----------- | ------------------------------ | 145 | job | `myjob/dispatch-1312323423423` | 146 | parent_id | `myjob` | 147 | dispatch_id | `1312323423423` | 148 149 ## Host Metrics 150 151 Nomad will emit [tagged metrics][tagged-metrics], in the below format: 152 153 | Metric | Description | Unit | Type | Labels | 154 | --------------------------------------- | ----------------------------------------------------------------------------------- | ---------- | ----- | ------------------------------------------------------------------------------------- | 155 | `nomad.client.allocated.cpu` | Total amount of CPU shares the scheduler has allocated to tasks | Mhz | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 156 | `nomad.client.allocated.memory` | Total amount of memory the scheduler has allocated to tasks | Megabytes | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 157 | `nomad.client.allocated_disk` | Total amount of disk space the scheduler has allocated to tasks | Megabytes | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 158 | `nomad.client.allocations.blocked` | Number of allocations waiting for previous versions to exit | Integer | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 159 | `nomad.client.allocations.migrating` | Number of allocations migrating data from previous versions (see [`sticky`][sticky])| Integer | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 160 | `nomad.client.allocations.pending` | Number of allocations pending (received by the client but not yet running) | Integer | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 161 | `nomad.client.allocations.running` | Number of allocations running | Integer | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 162 | `nomad.client.allocations.start` | Number of allocations starting | Integer | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 163 | `nomad.client.allocations.terminal` | Number of allocations terminal | Integer | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 164 | `nomad.client.allocs.oom_killed` | Number of allocations OOM killed | Integer | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 165 | `nomad.client.host.cpu.idle` | CPU utilization in idle state | Percentage | Gauge | cpu, datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 166 | `nomad.client.host.cpu.system` | CPU utilization in system space | Percentage | Gauge | cpu, datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 167 | `nomad.client.host.cpu.total` | Total CPU utilization | Percentage | Gauge | cpu, datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 168 | `nomad.client.host.cpu.user` | CPU utilization in user space | Percentage | Gauge | cpu, datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 169 | `nomad.client.host.disk.available` | Amount of space which is available | Bytes | Gauge | datacenter, disk, host, node_class, node_id, node_scheduling_eligibility, node_status | 170 | `nomad.client.host.disk.inodes_percent` | Disk space consumed by the inodes | Percentage | Gauge | datacenter, disk, host, node_class, node_id, node_scheduling_eligibility, node_status | 171 | `nomad.client.host.disk.size` | Total size of the device | Bytes | Gauge | datacenter, disk, host, node_class, node_id, node_scheduling_eligibility, node_status | 172 | `nomad.client.host.disk.used_percent` | Percentage of disk space used | Percentage | Gauge | datacenter, disk, host, node_class, node_id, node_scheduling_eligibility, node_status | 173 | `nomad.client.host.disk.used` | Amount of space which has been used | Bytes | Gauge | datacenter, disk, host, node_class, node_id, node_scheduling_eligibility, node_status | 174 | `nomad.client.host.memory.available` | Total amount of memory available to processes which includes free and cached memory | Bytes | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 175 | `nomad.client.host.memory.free` | Amount of memory which is free | Bytes | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 176 | `nomad.client.host.memory.total` | Total amount of physical memory on the node | Bytes | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 177 | `nomad.client.host.memory.used` | Amount of memory used by processes | Bytes | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 178 | `nomad.client.unallocated.cpu` | Total amount of CPU shares free for the scheduler to allocate to tasks | Mhz | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 179 | `nomad.client.unallocated.disk` | Total amount of disk space free for the scheduler to allocate to tasks | Megabytes | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 180 | `nomad.client.unallocated.memory` | Total amount of memory free for the scheduler to allocate to tasks | Megabytes | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 181 | `nomad.client.uptime` | Uptime of the host running the Nomad client | Seconds | Gauge | datacenter, host, node_class, node_id, node_scheduling_eligibility, node_status | 182 183 ## Allocation Metrics 184 185 The following metrics are emitted for each allocation if allocation metrics 186 are enabled. Note that allocation metrics available may be dependent on the 187 task driver; not all task drivers can provide all metrics. 188 189 | Metric | Description | Unit | Type | Labels | 190 | --------------------------------------------- | ----------------------------------------------------------------- | ----------- | ----- | ------------------------------------------------ | 191 | `nomad.client.allocs.cpu.allocated` | Total CPU resources allocated by the task across all cores | MHz | Gauge | alloc_id, host, job, namespace, task, task_group | 192 | `nomad.client.allocs.cpu.system` | Total CPU resources consumed by the task in system space | Percentage | Gauge | alloc_id, host, job, namespace, task, task_group | 193 | `nomad.client.allocs.cpu.throttled_periods` | Total number of CPU periods that the task was throttled | Nanoseconds | Gauge | alloc_id, host, job, namespace, task, task_group | 194 | `nomad.client.allocs.cpu.throttled_time` | Total time that the task was throttled | Nanoseconds | Gauge | alloc_id, host, job, namespace, task, task_group | 195 | `nomad.client.allocs.cpu.total_percent` | Total CPU resources consumed by the task across all cores | Percentage | Gauge | alloc_id, host, job, namespace, task, task_group | 196 | `nomad.client.allocs.cpu.total_ticks` | CPU ticks consumed by the process in the last collection interval | Integer | Gauge | alloc_id, host, job, namespace, task, task_group | 197 | `nomad.client.allocs.cpu.user` | Total CPU resources consumed by the task in the user space | Percentage | Gauge | alloc_id, host, job, namespace, task, task_group | 198 | `nomad.client.allocs.memory.allocated` | Amount of memory allocated by the task | Bytes | Gauge | alloc_id, host, job, namespace, task, task_group | 199 | `nomad.client.allocs.memory.cache` | Amount of memory cached by the task | Bytes | Gauge | alloc_id, host, job, namespace, task, task_group | 200 | `nomad.client.allocs.memory.kernel_max_usage` | Maximum amount of memory ever used by the kernel for this task | Bytes | Gauge | alloc_id, host, job, namespace, task, task_group | 201 | `nomad.client.allocs.memory.kernel_usage` | Amount of memory used by the kernel for this task | Bytes | Gauge | alloc_id, host, job, namespace, task, task_group | 202 | `nomad.client.allocs.memory.max_usage` | Maximum amount of memory ever used by the task | Bytes | Gauge | alloc_id, host, job, namespace, task, task_group | 203 | `nomad.client.allocs.memory.rss` | Amount of RSS memory consumed by the task | Bytes | Gauge | alloc_id, host, job, namespace, task, task_group | 204 | `nomad.client.allocs.memory.swap` | Amount of memory swapped by the task | Bytes | Gauge | alloc_id, host, job, namespace, task, task_group | 205 | `nomad.client.allocs.memory.usage` | Total amount of memory used by the task | Bytes | Gauge | alloc_id, host, job, namespace, task, task_group | 206 207 ## Job Summary Metrics 208 209 Job summary metrics are emitted by the Nomad leader server. 210 211 | Metric | Description | Unit | Type | Labels | 212 | ---------------------------------- | ---------------------------------------- | ------- | ----- | -------------------------------- | 213 | `nomad.nomad.job_summary.complete` | Number of complete allocations for a job | Integer | Gauge | host, job, namespace, task_group | 214 | `nomad.nomad.job_summary.failed` | Number of failed allocations for a job | Integer | Gauge | host, job, namespace, task_group | 215 | `nomad.nomad.job_summary.lost` | Number of lost allocations for a job | Integer | Gauge | host, job, namespace, task_group | 216 | `nomad.nomad.job_summary.unknown` | Number of unknown allocations for a job | Integer | Gauge | host, job, namespace, task_group | 217 | `nomad.nomad.job_summary.queued` | Number of queued allocations for a job | Integer | Gauge | host, job, namespace, task_group | 218 | `nomad.nomad.job_summary.running` | Number of running allocations for a job | Integer | Gauge | host, job, namespace, task_group | 219 | `nomad.nomad.job_summary.starting` | Number of starting allocations for a job | Integer | Gauge | host, job, namespace, task_group | 220 221 ## Job Status Metrics 222 223 Job status metrics are emitted by the Nomad leader server. 224 225 | Metric | Description | Unit | Type | Labels | 226 | -------------------------------- | ---------------------- | ------- | ----- | ------ | 227 | `nomad.nomad.job_status.dead` | Number of dead jobs | Integer | Gauge | host | 228 | `nomad.nomad.job_status.pending` | Number of pending jobs | Integer | Gauge | host | 229 | `nomad.nomad.job_status.running` | Number of running jobs | Integer | Gauge | host | 230 231 ## Server Metrics 232 233 The following table includes metrics for overall cluster health in addition to 234 those listed in [Key Metrics](#key-metrics) above. 235 236 | Metric | Description | Unit | Type | Labels | 237 |------------------------------------------------------|--------------------------------------------------------------------------------|----------------------|---------|---------------------------------------------------------| 238 | `nomad.memberlist.gossip` | Time elapsed to broadcast gossip messages | Nanoseconds | Summary | host | 239 | `nomad.nomad.acl.bootstrap` | Time elapsed for `ACL.Bootstrap` RPC call | Nanoseconds | Summary | host | 240 | `nomad.nomad.acl.delete_policies` | Time elapsed for `ACL.DeletePolicies` RPC call | Nanoseconds | Summary | host | 241 | `nomad.nomad.acl.delete_tokens` | Time elapsed for `ACL.DeleteTokens` RPC call | Nanoseconds | Summary | host | 242 | `nomad.nomad.acl.get_policies` | Time elapsed for `ACL.GetPolicies` RPC call | Nanoseconds | Summary | host | 243 | `nomad.nomad.acl.get_policy` | Time elapsed for `ACL.GetPolicy` RPC call | Nanoseconds | Summary | host | 244 | `nomad.nomad.acl.get_token` | Time elapsed for `ACL.GetToken` RPC call | Nanoseconds | Summary | host | 245 | `nomad.nomad.acl.get_tokens` | Time elapsed for `ACL.GetTokens` RPC call | Nanoseconds | Summary | host | 246 | `nomad.nomad.acl.list_policies` | Time elapsed for `ACL.ListPolicies` RPC call | Nanoseconds | Summary | host | 247 | `nomad.nomad.acl.list_tokens` | Time elapsed for `ACL.ListTokens` RPC call | Nanoseconds | Summary | host | 248 | `nomad.nomad.acl.resolve_token` | Time elapsed for `ACL.ResolveToken` RPC call | Nanoseconds | Summary | host | 249 | `nomad.nomad.acl.upsert_policies` | Time elapsed for `ACL.UpsertPolicies` RPC call | Nanoseconds | Summary | host | 250 | `nomad.nomad.acl.upsert_tokens` | Time elapsed for `ACL.UpsertTokens` RPC call | Nanoseconds | Summary | host | 251 | `nomad.nomad.alloc.exec` | Time elapsed to establish alloc exec | Nanoseconds | Summary | host | 252 | `nomad.nomad.alloc.get_alloc` | Time elapsed for `Alloc.GetAlloc` RPC call | Nanoseconds | Summary | host | 253 | `nomad.nomad.alloc.get_allocs` | Time elapsed for `Alloc.GetAllocs` RPC call | Nanoseconds | Summary | host | 254 | `nomad.nomad.alloc.list` | Time elapsed for `Alloc.List` RPC call | Nanoseconds | Summary | host | 255 | `nomad.nomad.alloc.stop` | Time elapsed for `Alloc.Stop` RPC call | Nanoseconds | Summary | host | 256 | `nomad.nomad.alloc.update_desired_transition` | Time elapsed for `Alloc.UpdateDesiredTransition` RPC call | Nanoseconds | Summary | host | 257 | `nomad.nomad.blocked_evals.cpu` | Amount of CPU shares requested by blocked evals | Integer | Gauge | datacenter, host, node_class | 258 | `nomad.nomad.blocked_evals.memory` | Amount of memory requested by blocked evals | Integer | Gauge | datacenter, host, node_class | 259 | `nomad.nomad.blocked_evals.job.cpu` | Amount of CPU shares requested by blocked evals of a job | Integer | Gauge | host, job, namespace | 260 | `nomad.nomad.blocked_evals.job.memory` | Amount of memory requested by blocked evals of a job | Integer | Gauge | host, job, namespace | 261 | `nomad.nomad.blocked_evals.total_blocked` | Count of evals in the blocked state for any reason (cluster resource exhaustion or quota limtis) | Integer | Gauge | host | 262 | `nomad.nomad.blocked_evals.total_escaped` | Count of evals that have escaped computed node classes. This indicates a scheduler optimization was skipped and is not usually a source of concern. | Integer | Gauge | host | 263 | `nomad.nomad.blocked_evals.total_quota_limit` | Count of blocked evals due to quota limits (the resources for these jobs are *not* counted in other blocked_evals metrics) | Integer | Gauge | host | 264 | `nomad.nomad.broker.batch_ready` | Count of batch evals ready to be scheduled | Integer | Gauge | host | 265 | `nomad.nomad.broker.batch_unacked` | Count of unacknowledged batch evals | Integer | Gauge | host | 266 | `nomad.nomad.broker.eval_waiting` | Time elapsed with evaluation waiting to be enqueued | Nanoseconds | Gauge | eval_id, job, namespace | 267 | `nomad.nomad.broker.service_ready` | Count of service evals ready to be scheduled | Integer | Gauge | host | 268 | `nomad.nomad.broker.service_unacked` | Count of unacknowledged service evals | Integer | Gauge | host | 269 | `nomad.nomad.broker.system_ready` | Count of system evals ready to be scheduled | Integer | Gauge | host | 270 | `nomad.nomad.broker.system_unacked` | Count of unacknowledged system evals | Integer | Gauge | host | 271 | `nomad.nomad.broker.total_ready` | Count of evals in the ready state | Integer | Gauge | host | 272 | `nomad.nomad.broker.total_waiting` | Count of evals waiting to be enqueued | Integer | Gauge | host | 273 | `nomad.nomad.client.batch_deregister` | Time elapsed for `Node.BatchDeregister` RPC call | Nanoseconds | Summary | host | 274 | `nomad.nomad.client.deregister` | Time elapsed for `Node.Deregister` RPC call | Nanoseconds | Summary | host | 275 | `nomad.nomad.client.derive_si_token` | Time elapsed for `Node.DeriveSIToken` RPC call | Nanoseconds | Summary | host | 276 | `nomad.nomad.client.derive_vault_token` | Time elapsed for `Node.DeriveVaultToken` RPC call | Nanoseconds | Summary | host | 277 | `nomad.nomad.client.emit_events` | Time elapsed for `Node.EmitEvents` RPC call | Nanoseconds | Summary | host | 278 | `nomad.nomad.client.evaluate` | Time elapsed for `Node.Evaluate` RPC call | Nanoseconds | Summary | host | 279 | `nomad.nomad.client.get_allocs` | Time elapsed for `Node.GetAllocs` RPC call | Nanoseconds | Summary | host | 280 | `nomad.nomad.client.get_client_allocs` | Time elapsed for `Node.GetClientAllocs` RPC call | Nanoseconds | Summary | host | 281 | `nomad.nomad.client.get_node` | Time elapsed for `Node.GetNode` RPC call | Nanoseconds | Summary | host | 282 | `nomad.nomad.client.list` | Time elapsed for `Node.List` RPC call | Nanoseconds | Summary | host | 283 | `nomad.nomad.client.register` | Time elapsed for `Node.Register` RPC call | Nanoseconds | Summary | host | 284 | `nomad.nomad.client.stats` | Time elapsed for `Client.Stats` RPC call | Nanoseconds | Summary | host | 285 | `nomad.nomad.client.update_alloc` | Time elapsed for `Node.UpdateAlloc` RPC call | Nanoseconds | Summary | host | 286 | `nomad.nomad.client.update_drain` | Time elapsed for `Node.UpdateDrain` RPC call | Nanoseconds | Summary | host | 287 | `nomad.nomad.client.update_eligibility` | Time elapsed for `Node.UpdateEligibility` RPC call | Nanoseconds | Summary | host | 288 | `nomad.nomad.client.update_status` | Time elapsed for `Node.UpdateStatus` RPC call | Nanoseconds | Summary | host | 289 | `nomad.nomad.client_allocations.garbage_collect_all` | Time elapsed for `ClientAllocations.GarbageCollectAll` RPC call | Nanoseconds | Summary | host | 290 | `nomad.nomad.client_allocations.garbage_collect` | Time elapsed for `ClientAllocations.GarbageCollect` RPC call | Nanoseconds | Summary | host | 291 | `nomad.nomad.client_allocations.restart` | Time elapsed for `ClientAllocations.Restart` RPC call | Nanoseconds | Summary | host | 292 | `nomad.nomad.client_allocations.signal` | Time elapsed for `ClientAllocations.Signal` RPC call | Nanoseconds | Summary | host | 293 | `nomad.nomad.client_allocations.stats` | Time elapsed for `ClientAllocations.Stats` RPC call | Nanoseconds | Summary | host | 294 | `nomad.nomad.client_csi_controller.attach_volume` | Time elapsed for `Controller.AttachVolume` RPC call | Nanoseconds | Summary | host | 295 | `nomad.nomad.client_csi_controller.detach_volume` | Time elapsed for `Controller.DetachVolume` RPC call | Nanoseconds | Summary | host | 296 | `nomad.nomad.client_csi_controller.validate_volume` | Time elapsed for `Controller.ValidateVolume` RPC call | Nanoseconds | Summary | host | 297 | `nomad.nomad.client_csi_node.detach_volume` | Time elapsed for `Node.DetachVolume` RPC call | Nanoseconds | Summary | host | 298 | `nomad.nomad.deployment.allocations` | Time elapsed for `Deployment.Allocations` RPC call | Nanoseconds | Summary | host | 299 | `nomad.nomad.deployment.cancel` | Time elapsed for `Deployment.Cancel` RPC call | Nanoseconds | Summary | host | 300 | `nomad.nomad.deployment.fail` | Time elapsed for `Deployment.Fail` RPC call | Nanoseconds | Summary | host | 301 | `nomad.nomad.deployment.get_deployment` | Time elapsed for `Deployment.GetDeployment` RPC call | Nanoseconds | Summary | host | 302 | `nomad.nomad.deployment.list` | Time elapsed for `Deployment.List` RPC call | Nanoseconds | Summary | host | 303 | `nomad.nomad.deployment.pause` | Time elapsed for `Deployment.Pause` RPC call | Nanoseconds | Summary | host | 304 | `nomad.nomad.deployment.promote` | Time elapsed for `Deployment.Promote` RPC call | Nanoseconds | Summary | host | 305 | `nomad.nomad.deployment.reap` | Time elapsed for `Deployment.Reap` RPC call | Nanoseconds | Summary | host | 306 | `nomad.nomad.deployment.run` | Time elapsed for `Deployment.Run` RPC call | Nanoseconds | Summary | host | 307 | `nomad.nomad.deployment.set_alloc_health` | Time elapsed for `Deployment.SetAllocHealth` RPC call | Nanoseconds | Summary | host | 308 | `nomad.nomad.deployment.unblock` | Time elapsed for `Deployment.Unblock` RPC call | Nanoseconds | Summary | host | 309 | `nomad.nomad.eval.ack` | Time elapsed for `Eval.Ack` RPC call | Nanoseconds | Summary | host | 310 | `nomad.nomad.eval.allocations` | Time elapsed for `Eval.Allocations` RPC call | Nanoseconds | Summary | host | 311 | `nomad.nomad.eval.create` | Time elapsed for `Eval.Create` RPC call | Nanoseconds | Summary | host | 312 | `nomad.nomad.eval.dequeue` | Time elapsed for `Eval.Dequeue` RPC call | Nanoseconds | Summary | host | 313 | `nomad.nomad.eval.get_eval` | Time elapsed for `Eval.GetEval` RPC call | Nanoseconds | Summary | host | 314 | `nomad.nomad.eval.list` | Time elapsed for `Eval.List` RPC call | Nanoseconds | Summary | host | 315 | `nomad.nomad.eval.nack` | Time elapsed for `Eval.Nack` RPC call | Nanoseconds | Summary | host | 316 | `nomad.nomad.eval.reap` | Time elapsed for `Eval.Reap` RPC call | Nanoseconds | Summary | host | 317 | `nomad.nomad.eval.reblock` | Time elapsed for `Eval.Reblock` RPC call | Nanoseconds | Summary | host | 318 | `nomad.nomad.eval.update` | Time elapsed for `Eval.Update` RPC call | Nanoseconds | Summary | host | 319 | `nomad.nomad.file_system.list` | Time elapsed for `FileSystem.List` RPC call | Nanoseconds | Summary | host | 320 | `nomad.nomad.file_system.logs` | Time elapsed to establish `FileSystem.Logs` RPC | Nanoseconds | Summary | host | 321 | `nomad.nomad.file_system.stat` | Time elapsed for `FileSystem.Stat` RPC call | Nanoseconds | Summary | host | 322 | `nomad.nomad.file_system.stream` | Time elapsed to establish `FileSystem.Stream` RPC | Nanoseconds | Summary | host | 323 | `nomad.nomad.fsm.alloc_client_update` | Time elapsed to apply `AllocClientUpdate` raft entry | Nanoseconds | Summary | host | 324 | `nomad.nomad.fsm.alloc_update_desired_transition` | Time elapsed to apply `AllocUpdateDesiredTransition` raft entry | Nanoseconds | Summary | host | 325 | `nomad.nomad.fsm.apply_acl_policy_delete` | Time elapsed to apply `ApplyACLPolicyDelete` raft entry | Nanoseconds | Summary | host | 326 | `nomad.nomad.fsm.apply_acl_policy_upsert` | Time elapsed to apply `ApplyACLPolicyUpsert` raft entry | Nanoseconds | Summary | host | 327 | `nomad.nomad.fsm.apply_acl_token_bootstrap` | Time elapsed to apply `ApplyACLTokenBootstrap` raft entry | Nanoseconds | Summary | host | 328 | `nomad.nomad.fsm.apply_acl_token_delete` | Time elapsed to apply `ApplyACLTokenDelete` raft entry | Nanoseconds | Summary | host | 329 | `nomad.nomad.fsm.apply_acl_token_upsert` | Time elapsed to apply `ApplyACLTokenUpsert` raft entry | Nanoseconds | Summary | host | 330 | `nomad.nomad.fsm.apply_csi_plugin_delete` | Time elapsed to apply `ApplyCSIPluginDelete` raft entry | Nanoseconds | Summary | host | 331 | `nomad.nomad.fsm.apply_csi_volume_batch_claim` | Time elapsed to apply `ApplyCSIVolumeBatchClaim` raft entry | Nanoseconds | Summary | host | 332 | `nomad.nomad.fsm.apply_csi_volume_claim` | Time elapsed to apply `ApplyCSIVolumeClaim` raft entry | Nanoseconds | Summary | host | 333 | `nomad.nomad.fsm.apply_csi_volume_deregister` | Time elapsed to apply `ApplyCSIVolumeDeregister` raft entry | Nanoseconds | Summary | host | 334 | `nomad.nomad.fsm.apply_csi_volume_register` | Time elapsed to apply `ApplyCSIVolumeRegister` raft entry | Nanoseconds | Summary | host | 335 | `nomad.nomad.fsm.apply_deployment_alloc_health` | Time elapsed to apply `ApplyDeploymentAllocHealth` raft entry | Nanoseconds | Summary | host | 336 | `nomad.nomad.fsm.apply_deployment_delete` | Time elapsed to apply `ApplyDeploymentDelete` raft entry | Nanoseconds | Summary | host | 337 | `nomad.nomad.fsm.apply_deployment_promotion` | Time elapsed to apply `ApplyDeploymentPromotion` raft entry | Nanoseconds | Summary | host | 338 | `nomad.nomad.fsm.apply_deployment_status_update` | Time elapsed to apply `ApplyDeploymentStatusUpdate` raft entry | Nanoseconds | Summary | host | 339 | `nomad.nomad.fsm.apply_job_stability` | Time elapsed to apply `ApplyJobStability` raft entry | Nanoseconds | Summary | host | 340 | `nomad.nomad.fsm.apply_namespace_delete` | Time elapsed to apply `ApplyNamespaceDelete` raft entry | Nanoseconds | Summary | host | 341 | `nomad.nomad.fsm.apply_namespace_upsert` | Time elapsed to apply `ApplyNamespaceUpsert` raft entry | Nanoseconds | Summary | host | 342 | `nomad.nomad.fsm.apply_plan_results` | Time elapsed to apply `ApplyPlanResults` raft entry | Nanoseconds | Summary | host | 343 | `nomad.nomad.fsm.apply_scheduler_config` | Time elapsed to apply `ApplySchedulerConfig` raft entry | Nanoseconds | Summary | host | 344 | `nomad.nomad.fsm.autopilot` | Time elapsed to apply `Autopilot` raft entry | Nanoseconds | Summary | host | 345 | `nomad.nomad.fsm.batch_deregister_job` | Time elapsed to apply `BatchDeregisterJob` raft entry | Nanoseconds | Summary | host | 346 | `nomad.nomad.fsm.batch_deregister_node` | Time elapsed to apply `BatchDeregisterNode` raft entry | Nanoseconds | Summary | host | 347 | `nomad.nomad.fsm.batch_node_drain_update` | Time elapsed to apply `BatchNodeDrainUpdate` raft entry | Nanoseconds | Summary | host | 348 | `nomad.nomad.fsm.cluster_meta` | Time elapsed to apply `ClusterMeta` raft entry | Nanoseconds | Summary | host | 349 | `nomad.nomad.fsm.delete_eval` | Time elapsed to apply `DeleteEval` raft entry | Nanoseconds | Summary | host | 350 | `nomad.nomad.fsm.deregister_job` | Time elapsed to apply `DeregisterJob` raft entry | Nanoseconds | Summary | host | 351 | `nomad.nomad.fsm.deregister_node` | Time elapsed to apply `DeregisterNode` raft entry | Nanoseconds | Summary | host | 352 | `nomad.nomad.fsm.deregister_si_accessor` | Time elapsed to apply `DeregisterSITokenAccessor` raft entry | Nanoseconds | Summary | host | 353 | `nomad.nomad.fsm.deregister_vault_accessor` | Time elapsed to apply `DeregisterVaultAccessor` raft entry | Nanoseconds | Summary | host | 354 | `nomad.nomad.fsm.node_drain_update` | Time elapsed to apply `NodeDrainUpdate` raft entry | Nanoseconds | Summary | host | 355 | `nomad.nomad.fsm.node_eligibility_update` | Time elapsed to apply `NodeEligibilityUpdate` raft entry | Nanoseconds | Summary | host | 356 | `nomad.nomad.fsm.node_status_update` | Time elapsed to apply `NodeStatusUpdate` raft entry | Nanoseconds | Summary | host | 357 | `nomad.nomad.fsm.persist` | Time elapsed to apply `Persist` raft entry | Nanoseconds | Summary | host | 358 | `nomad.nomad.fsm.register_job` | Time elapsed to apply `RegisterJob` raft entry | Nanoseconds | Summary | host | 359 | `nomad.nomad.fsm.register_node` | Time elapsed to apply `RegisterNode` raft entry | Nanoseconds | Summary | host | 360 | `nomad.nomad.fsm.update_eval` | Time elapsed to apply `UpdateEval` raft entry | Nanoseconds | Summary | host | 361 | `nomad.nomad.fsm.upsert_node_events` | Time elapsed to apply `UpsertNodeEvents` raft entry | Nanoseconds | Summary | host | 362 | `nomad.nomad.fsm.upsert_scaling_event` | Time elapsed to apply `UpsertScalingEvent` raft entry | Nanoseconds | Summary | host | 363 | `nomad.nomad.fsm.upsert_si_accessor` | Time elapsed to apply `UpsertSITokenAccessors` raft entry | Nanoseconds | Summary | host | 364 | `nomad.nomad.fsm.upsert_vault_accessor` | Time elapsed to apply `UpsertVaultAccessor` raft entry | Nanoseconds | Summary | host | 365 | `nomad.nomad.job.allocations` | Time elapsed for `Job.Allocations` RPC call | Nanoseconds | Summary | host | 366 | `nomad.nomad.job.batch_deregister` | Time elapsed for `Job.BatchDeregister` RPC call | Nanoseconds | Summary | host | 367 | `nomad.nomad.job.deployments` | Time elapsed for `Job.Deployments` RPC call | Nanoseconds | Summary | host | 368 | `nomad.nomad.job.deregister` | Time elapsed for `Job.Deregister` RPC call | Nanoseconds | Summary | host | 369 | `nomad.nomad.job.dispatch` | Time elapsed for `Job.Dispatch` RPC call | Nanoseconds | Summary | host | 370 | `nomad.nomad.job.evaluate` | Time elapsed for `Job.Evaluate` RPC call | Nanoseconds | Summary | host | 371 | `nomad.nomad.job.evaluations` | Time elapsed for `Job.Evaluations` RPC call | Nanoseconds | Summary | host | 372 | `nomad.nomad.job.get_job_versions` | Time elapsed for `Job.GetJobVersions` RPC call | Nanoseconds | Summary | host | 373 | `nomad.nomad.job.get_job` | Time elapsed for `Job.GetJob` RPC call | Nanoseconds | Summary | host | 374 | `nomad.nomad.job.latest_deployment` | Time elapsed for `Job.LatestDeployment` RPC call | Nanoseconds | Summary | host | 375 | `nomad.nomad.job.list` | Time elapsed for `Job.List` RPC call | Nanoseconds | Summary | host | 376 | `nomad.nomad.job.plan` | Time elapsed for `Job.Plan` RPC call | Nanoseconds | Summary | host | 377 | `nomad.nomad.job.register` | Time elapsed for `Job.Register` RPC call | Nanoseconds | Summary | host | 378 | `nomad.nomad.job.revert` | Time elapsed for `Job.Revert` RPC call | Nanoseconds | Summary | host | 379 | `nomad.nomad.job.scale_status` | Time elapsed for `Job.ScaleStatus` RPC call | Nanoseconds | Summary | host | 380 | `nomad.nomad.job.scale` | Time elapsed for `Job.Scale` RPC call | Nanoseconds | Summary | host | 381 | `nomad.nomad.job.stable` | Time elapsed for `Job.Stable` RPC call | Nanoseconds | Summary | host | 382 | `nomad.nomad.job.validate` | Time elapsed for `Job.Validate` RPC call | Nanoseconds | Summary | host | 383 | `nomad.nomad.job_summary.get_job_summary` | Time elapsed for `Job.Summary` RPC call | Nanoseconds | Summary | host | 384 | `nomad.nomad.leader.barrier` | Time elapsed to establish a raft barrier during leader transition | Nanoseconds | Summary | host | 385 | `nomad.nomad.leader.reconcileMember` | Time elapsed to reconcile a serf peer with state store | Nanoseconds | Summary | host | 386 | `nomad.nomad.leader.reconcile` | Time elapsed to reconcile all serf peers with state store | Nanoseconds | Summary | host | 387 | `nomad.nomad.namespace.delete_namespaces` | Time elapsed for `Namespace.DeleteNamespaces` | Nanoseconds | Summary | host | 388 | `nomad.nomad.namespace.get_namespace` | Time elapsed for `Namespace.GetNamespace` | Nanoseconds | Summary | host | 389 | `nomad.nomad.namespace.get_namespaces` | Time elapsed for `Namespace.GetNamespaces` | Nanoseconds | Summary | host | 390 | `nomad.nomad.namespace.list_namespace` | Time elapsed for `Namespace.ListNamespaces` | Nanoseconds | Summary | host | 391 | `nomad.nomad.namespace.upsert_namespaces` | Time elapsed for `Namespace.UpsertNamespaces` | Nanoseconds | Summary | host | 392 | `nomad.nomad.periodic.force` | Time elapsed for `Periodic.Force` RPC call | Nanoseconds | Summary | host | 393 | `nomad.nomad.plan.apply` | Time elapsed to apply a plan | Nanoseconds | Summary | host | 394 | `nomad.nomad.plan.evaluate` | Time elapsed to evaluate a plan | Nanoseconds | Summary | host | 395 | `nomad.nomad.plan.node_rejected` | Number of times a node has had a plan rejected | Integer | Counter | host, node_id | 396 | `nomad.nomad.plan.rejection_tracker.node_score` | Number of times a node has had a plan rejected within the tracker window | Integer | Gauge | host, node_id | 397 | `nomad.nomad.plan.queue_depth` | Count of evals in the plan queue | Integer | Gauge | host | 398 | `nomad.nomad.plan.submit` | Time elapsed for `Plan.Submit` RPC call | Nanoseconds | Summary | host | 399 | `nomad.nomad.plan.wait_for_index` | Time elapsed that planner waits for the raft index of the plan to be processed | Nanoseconds | Summary | host | 400 | `nomad.nomad.plugin.delete` | Time elapsed for `CSIPlugin.Delete` RPC call | Nanoseconds | Summary | host | 401 | `nomad.nomad.plugin.get` | Time elapsed for `CSIPlugin.Get` RPC call | Nanoseconds | Summary | host | 402 | `nomad.nomad.plugin.list` | Time elapsed for `CSIPlugin.List` RPC call | Nanoseconds | Summary | host | 403 | `nomad.nomad.scaling.get_policy` | Time elapsed for `Scaling.GetPolicy` RPC call | Nanoseconds | Summary | host | 404 | `nomad.nomad.scaling.list_policies` | Time elapsed for `Scaling.ListPolicies` RPC call | Nanoseconds | Summary | host | 405 | `nomad.nomad.search.prefix_search` | Time elapsed for `Search.PrefixSearch` RPC call | Nanoseconds | Summary | host | 406 | `nomad.nomad.vault.create_token` | Time elapsed to create Vault token | Nanoseconds | Gauge | host | 407 | `nomad.nomad.vault.distributed_tokens_revoked` | Count of revoked tokens | Integer | Gauge | host | 408 | `nomad.nomad.vault.lookup_token` | Time elapsed to lookup Vault token | Nanoseconds | Gauge | host | 409 | `nomad.nomad.vault.renew_failed` | Count of failed attempts to renew Vault token | Integer | Gauge | host | 410 | `nomad.nomad.vault.renew` | Time elapsed to renew Vault token | Nanoseconds | Gauge | host | 411 | `nomad.nomad.vault.revoke_tokens` | Time elapsed to revoke Vault tokens | Nanoseconds | Gauge | host | 412 | `nomad.nomad.vault.token_last_renewal` | Time since last successful Vault token renewal | Milliseconds | Gauge | host | 413 | `nomad.nomad.vault.token_next_renewal` | Time until next Vault token renewal attempt | Milliseconds | Gauge | host | 414 | `nomad.nomad.vault.token_ttl` | Time to live for Vault token | Milliseconds | Gauge | host | 415 | `nomad.nomad.vault.undistributed_tokens_abandoned` | Count of abandoned tokens | Integer | Gauge | host | 416 | `nomad.nomad.volume.claim` | Time elapsed for `CSIVolume.Claim` RPC call | Nanoseconds | Summary | host | 417 | `nomad.nomad.volume.deregister` | Time elapsed for `CSIVolume.Deregister` RPC call | Nanoseconds | Summary | host | 418 | `nomad.nomad.volume.get` | Time elapsed for `CSIVolume.Get` RPC call | Nanoseconds | Summary | host | 419 | `nomad.nomad.volume.list` | Time elapsed for `CSIVolume.List` RPC call | Nanoseconds | Summary | host | 420 | `nomad.nomad.volume.register` | Time elapsed for `CSIVolume.Register` RPC call | Nanoseconds | Summary | host | 421 | `nomad.nomad.volume.unpublish` | Time elapsed for `CSIVolume.Unpublish` RPC call | Nanoseconds | Summary | host | 422 | `nomad.nomad.worker.create_eval` | Time elapsed for worker to create an eval | Nanoseconds | Summary | host | 423 | `nomad.nomad.worker.dequeue_eval` | Time elapsed for worker to dequeue an eval | Nanoseconds | Summary | host | 424 | `nomad.nomad.worker.invoke_scheduler_service` | Time elapsed for worker to invoke the scheduler | Nanoseconds | Summary | host | 425 | `nomad.nomad.worker.send_ack` | Time elapsed for worker to send acknowledgement | Nanoseconds | Summary | host | 426 | `nomad.nomad.worker.submit_plan` | Time elapsed for worker to submit plan | Nanoseconds | Summary | host | 427 | `nomad.nomad.worker.update_eval` | Time elapsed for worker to submit updated eval | Nanoseconds | Summary | host | 428 | `nomad.nomad.worker.wait_for_index` | Time elapsed that worker waits for the raft index of the eval to be processed | Nanoseconds | Summary | host | 429 | `nomad.raft.appliedIndex` | Current index applied to FSM | Integer | Gauge | host | 430 | `nomad.raft.barrier` | Count of blocking raft API calls | Integer | Counter | host | 431 | `nomad.raft.commitNumLogs` | Count of logs enqueued | Integer | Gauge | host | 432 | `nomad.raft.commitTime` | Time elapsed to commit writes | Nanoseconds | Summary | host | 433 | `nomad.raft.fsm.apply` | Time elapsed to apply write to FSM | Nanoseconds | Summary | host | 434 | `nomad.raft.fsm.enqueue` | Time elapsed to enqueue write to FSM | Nanoseconds | Summary | host | 435 | `nomad.raft.lastIndex` | Most recent index seen | Integer | Gauge | host | 436 | `nomad.raft.leader.dispatchLog` | Time elapsed to write log, mark in flight, and start replication | Nanoseconds | Summary | host | 437 | `nomad.raft.leader.dispatchNumLogs` | Count of logs dispatched | Integer | Gauge | host | 438 | `nomad.raft.replication.appendEntries` | Raft transaction commit time | ms / Raft Log Append | Timer | | 439 | `nomad.raft.state.candidate` | Count of entering candidate state | Integer | Gauge | host | 440 | `nomad.raft.state.follower` | Count of entering follower state | Integer | Gauge | host | 441 | `nomad.raft.state.leader` | Count of entering leader state | Integer | Gauge | host | 442 | `nomad.raft.transition.heartbeat_timeout` | Count of failing to heartbeat and starting election | Integer | Gauge | host | 443 | `nomad.raft.transition.leader_lease_timeout` | Count of stepping down as leader after losing quorum | Integer | Gauge | host | 444 | `nomad.runtime.free_count` | Count of objects freed from heap by go runtime GC | Integer | Gauge | host | 445 | `nomad.runtime.gc_pause_ns` | Go runtime GC pause times | Nanoseconds | Summary | host | 446 | `nomad.runtime.sys_bytes` | Go runtime GC metadata size | # of bytes | Gauge | host | 447 | `nomad.runtime.total_gc_pause_ns` | Total elapsed go runtime GC pause times | Nanoseconds | Gauge | host | 448 | `nomad.runtime.total_gc_runs` | Count of go runtime GC runs | Integer | Gauge | host | 449 | `nomad.serf.queue.Event` | Count of memberlist events received | Integer | Summary | host | 450 | `nomad.serf.queue.Intent` | Count of memberlist changes | Integer | Summary | host | 451 | `nomad.serf.queue.Query` | Count of memberlist queries | Integer | Summary | host | 452 | `nomad.scheduler.allocs.rescheduled.attempted` | Count of attempts to reschedule an allocation | Integer | Count | alloc_id, job, namespace, task_group | 453 | `nomad.scheduler.allocs.rescheduled.limit` | Maximum number of attempts to reschedule an allocation | Integer | Count | alloc_id, job, namespace, task_group | 454 | `nomad.scheduler.allocs.rescheduled.wait_until` | Time that a rescheduled allocation will be delayed | Float | Gauge | alloc_id, job, namespace, task_group, follow_up_eval_id | 455 | `nomad.state.snapshotIndex` | Current snapshot index | Integer | Gauge | host | 456 457 ## Raft BoltDB Metrics 458 459 Raft database metrics are emitted by the `raft-boltdb` library. 460 461 | Metric | Description | Unit | Type | 462 | ----------------------------------------- | ----------------------------------------- | ----------- | ------- | 463 | `nomad.raft.boltdb.numFreePages` | Number of free pages | Integer | Gauge | 464 | `nomad.raft.boltdb.numPendingPages` | Number of pending pages | Integer | Gauge | 465 | `nomad.raft.boltdb.freePageBytes` | Number of free page bytes | Integer | Gauge | 466 | `nomad.raft.boltdb.freelistBytes` | Number of freelist bytes | Integer | Gauge | 467 | `nomad.raft.boltdb.totalReadTxn` | Count of total read transactions | Integer | Counter | 468 | `nomad.raft.boltdb.openReadTxn` | Number of current open read transactions | Integer | Gauge | 469 | `nomad.raft.boltdb.txstats.pageCount` | Number of pages in use | Integer | Gauge | 470 | `nomad.raft.boltdb.txstats.pageAlloc` | Number of page allocations | Integer | Gauge | 471 | `nomad.raft.boltdb.txstats.cursorCount` | Count of total database cursors | Integer | Counter | 472 | `nomad.raft.boltdb.txstats.nodeCount` | Count of total database nodes | Integer | Counter | 473 | `nomad.raft.boltdb.txstats.nodeDeref` | Count of total database node dereferences | Integer | Counter | 474 | `nomad.raft.boltdb.txstats.rebalance` | Count of total rebalance operations | Integer | Counter | 475 | `nomad.raft.boltdb.txstats.rebalanceTime` | Sample of rebalance operation times | Nanoseconds | Summary | 476 | `nomad.raft.boltdb.txstats.split` | Count of total split operations | Integer | Counter | 477 | `nomad.raft.boltdb.txstats.spill` | Count of total spill operations | Integer | Counter | 478 | `nomad.raft.boltdb.txstats.spillTime` | Sample of spill operation times | Nanoseconds | Summary | 479 | `nomad.raft.boltdb.txstats.write` | Count of total write operations | Integer | Counter | 480 | `nomad.raft.boltdb.txstats.writeTime` | Sample of write operation times | Nanoseconds | Summary | 481 482 ## Agent Metrics 483 484 Agent metrics are emitted by all Nomad agents running in either client or server mode. 485 486 | Metric | Description | Unit | Type | 487 | ----------------------------------------- | ----------------------------------------------------- | ----------- | ------- | 488 | `nomad.agent.http.exceeded` | Count of HTTP connections exceeding concurrency limit | Integer | Counter | 489 490 [tagged-metrics]: /docs/operations/metrics-reference#tagged-metrics 491 [sticky]: /docs/job-specification/ephemeral_disk#sticky 492 [s_port_plan_failure]: /s/port-plan-failure