github.com/minio/minio@v0.0.0-20240328213742-3f72439b8a27/docs/metrics/v3.md (about) 1 # Metrics Version 3 2 3 In metrics version 3, all metrics are available under the endpoint: 4 5 ``` 6 /minio/metrics/v3 7 ``` 8 9 however, a specific path under this is required. 10 11 Metrics are organized into groups at paths **relative** to the top-level endpoint above. 12 13 ## Metrics Request Handling 14 15 Each endpoint below can be queried at different intervals as needed via a scrape configuration in Prometheus or a compatible metrics collection tool. 16 17 For ease of configuration, each (non-empty) parent of the path serves all metric endpoints that are at descendant paths. For example, to query all system metrics one needs to only scrape `/minio/metrics/v3/system/`. 18 19 Some metrics are bucket specific. These will have a `/bucket` component in their path. As the number of buckets can be large, the metrics scrape operation needs to be provided with a specific list of buckets via the `bucket` query parameter. Only metrics for the given buckets will be returned (with the bucket label set). For example to query API metrics for buckets `test1` and `test2`, make a scrape request to `/minio/metrics/v3/api/bucket?buckets=test1,test2`. 20 21 Instead of a metrics scrape, it is also possible to list the metrics that would be returned by a path. This is done by adding a `?list` query parameter. The MinIO server will then list all possible metrics that could be returned. During an actual metrics scrape, only available metrics are returned - not all of them. With the `list` query parameter, the output format can be selected - just set the request `Content-Type` to `application/json` for JSON output, or `text/plain` for a simple markdown formatted table. The latter is the default. 22 23 ## Request, System and Cluster Metrics 24 25 At a high level metrics are grouped into three categories, listed in the following sub-sections. The path in each of the tables is relative to the top-level endpoint. 26 27 ### Request metrics 28 29 These are metrics about requests served by the (current) node. 30 31 | Path | Description | 32 |-----------------|--------------------------------------------------| 33 | `/api/requests` | Metrics over all requests | 34 | `/api/bucket` | Metrics over all requests split by bucket labels | 35 | | | 36 37 38 ### System metrics 39 40 These are metrics about the minio process and the node. 41 42 | Path | Description | 43 |-----------------------------|---------------------------------------------------| 44 | `/system/drive` | Metrics about drives on the system | 45 | `/system/network/internode` | Metrics about internode requests made by the node | 46 | `/system/process` | Standard process metrics | 47 | `/system/go` | Standard Go lang metrics | 48 | | | 49 50 ### Cluster metrics 51 52 These present metrics about the whole MinIO cluster. 53 54 | Path | Description | 55 |--------------------------|-----------------------------| 56 | `/cluster/health` | Cluster health metrics | 57 | `/cluster/usage/objects` | Object statistics | 58 | `/cluster/usage/buckets` | Object statistics by bucket | 59 | `/cluster/erasure-set` | Erasure set metrics | 60 | | | 61 62 ## Metrics Listing 63 64 Each of the following sub-sections list metrics returned by each of the endpoints. 65 66 The standard metrics groups for ProcessCollector and GoCollector are not shown below. 67 68 ### `/api/requests` 69 70 | Name | Type | Help | Labels | 71 |------------------------------------------------|-----------|---------------------------------------------------------|----------------------------------| 72 | `minio_api_requests_rejected_auth_total` | `counter` | Total number of requests rejected for auth failure | `type,pool_index,server` | 73 | `minio_api_requests_rejected_header_total` | `counter` | Total number of requests rejected for invalid header | `type,pool_index,server` | 74 | `minio_api_requests_rejected_timestamp_total` | `counter` | Total number of requests rejected for invalid timestamp | `type,pool_index,server` | 75 | `minio_api_requests_rejected_invalid_total` | `counter` | Total number of invalid requests | `type,pool_index,server` | 76 | `minio_api_requests_waiting_total` | `gauge` | Total number of requests in the waiting queue | `type,pool_index,server` | 77 | `minio_api_requests_incoming_total` | `gauge` | Total number of incoming requests | `type,pool_index,server` | 78 | `minio_api_requests_inflight_total` | `gauge` | Total number of requests currently in flight | `name,type,pool_index,server` | 79 | `minio_api_requests_total` | `counter` | Total number of requests | `name,type,pool_index,server` | 80 | `minio_api_requests_errors_total` | `counter` | Total number of requests with (4xx and 5xx) errors | `name,type,pool_index,server` | 81 | `minio_api_requests_5xx_errors_total` | `counter` | Total number of requests with 5xx errors | `name,type,pool_index,server` | 82 | `minio_api_requests_4xx_errors_total` | `counter` | Total number of requests with 4xx errors | `name,type,pool_index,server` | 83 | `minio_api_requests_canceled_total` | `counter` | Total number of requests canceled by the client | `name,type,pool_index,server` | 84 | `minio_api_requests_ttfb_seconds_distribution` | `counter` | Distribution of time to first byte across API calls | `name,type,le,pool_index,server` | 85 | `minio_api_requests_traffic_sent_bytes` | `counter` | Total number of bytes sent | `type,pool_index,server` | 86 | `minio_api_requests_traffic_received_bytes` | `counter` | Total number of bytes received | `type,pool_index,server` | 87 88 ### `/api/bucket` 89 90 | Name | Type | Help | Labels | 91 |----------------------------------------------|-----------|------------------------------------------------------------------|-----------------------------------------| 92 | `minio_api_bucket_traffic_received_bytes` | `counter` | Total number of bytes sent for a bucket | `bucket,type,server,pool_index` | 93 | `minio_api_bucket_traffic_sent_bytes` | `counter` | Total number of bytes received for a bucket | `bucket,type,server,pool_index` | 94 | `minio_api_bucket_inflight_total` | `gauge` | Total number of requests currently in flight for a bucket | `bucket,name,type,server,pool_index` | 95 | `minio_api_bucket_total` | `counter` | Total number of requests for a bucket | `bucket,name,type,server,pool_index` | 96 | `minio_api_bucket_canceled_total` | `counter` | Total number of requests canceled by the client for a bucket | `bucket,name,type,server,pool_index` | 97 | `minio_api_bucket_4xx_errors_total` | `counter` | Total number of requests with 4xx errors for a bucket | `bucket,name,type,server,pool_index` | 98 | `minio_api_bucket_5xx_errors_total` | `counter` | Total number of requests with 5xx errors for a bucket | `bucket,name,type,server,pool_index` | 99 | `minio_api_bucket_ttfb_seconds_distribution` | `counter` | Distribution of time to first byte across API calls for a bucket | `bucket,name,le,type,server,pool_index` | 100 101 ### `/system/drive` 102 103 | Name | Type | Help | Labels | 104 |------------------------------------------------|-----------|-----------------------------------------------------------------------------------|-----------------------------------------------------| 105 | `minio_system_drive_used_bytes` | `gauge` | Total storage used on a drive in bytes | `drive,set_index,drive_index,pool_index,server` | 106 | `minio_system_drive_free_bytes` | `gauge` | Total storage free on a drive in bytes | `drive,set_index,drive_index,pool_index,server` | 107 | `minio_system_drive_total_bytes` | `gauge` | Total storage available on a drive in bytes | `drive,set_index,drive_index,pool_index,server` | 108 | `minio_system_drive_free_inodes` | `gauge` | Total free inodes on a drive | `drive,set_index,drive_index,pool_index,server` | 109 | `minio_system_drive_timeout_errors_total` | `counter` | Total timeout errors on a drive | `drive,set_index,drive_index,pool_index,server` | 110 | `minio_system_drive_availability_errors_total` | `counter` | Total availability errors (I/O errors, permission denied and timeouts) on a drive | `drive,set_index,drive_index,pool_index,server` | 111 | `minio_system_drive_waiting_io` | `gauge` | Total waiting I/O operations on a drive | `drive,set_index,drive_index,pool_index,server` | 112 | `minio_system_drive_api_latency_micros` | `gauge` | Average last minute latency in µs for drive API storage operations | `drive,api,set_index,drive_index,pool_index,server` | 113 | `minio_system_drive_offline_count` | `gauge` | Count of offline drives | `pool_index,server` | 114 | `minio_system_drive_online_count` | `gauge` | Count of online drives | `pool_index,server` | 115 | `minio_system_drive_count` | `gauge` | Count of all drives | `pool_index,server` | 116 117 ### `/system/network/internode` 118 119 | Name | Type | Help | Labels | 120 |------------------------------------------------------|-----------|----------------------------------------------------------|---------------------| 121 | `minio_system_network_internode_errors_total` | `counter` | Total number of failed internode calls | `server,pool_index` | 122 | `minio_system_network_internode_dial_errors_total` | `counter` | Total number of internode TCP dial timeouts and errors | `server,pool_index` | 123 | `minio_system_network_internode_dial_avg_time_nanos` | `gauge` | Average dial time of internodes TCP calls in nanoseconds | `server,pool_index` | 124 | `minio_system_network_internode_sent_bytes_total` | `counter` | Total number of bytes sent to other peer nodes | `server,pool_index` | 125 | `minio_system_network_internode_recv_bytes_total` | `counter` | Total number of bytes received from other peer nodes | `server,pool_index` | 126 127 ### `/cluster/health` 128 129 | Name | Type | Help | Labels | 130 |----------------------------------------------------|---------|------------------------------------------------|--------| 131 | `minio_cluster_health_drives_offline_count` | `gauge` | Count of offline drives in the cluster | | 132 | `minio_cluster_health_drives_online_count` | `gauge` | Count of online drives in the cluster | | 133 | `minio_cluster_health_drives_count` | `gauge` | Count of all drives in the cluster | | 134 | `minio_cluster_health_nodes_offline_count` | `gauge` | Count of offline nodes in the cluster | | 135 | `minio_cluster_health_nodes_online_count` | `gauge` | Count of online nodes in the cluster | | 136 | `minio_cluster_health_capacity_raw_total_bytes` | `gauge` | Total cluster raw storage capacity in bytes | | 137 | `minio_cluster_health_capacity_raw_free_bytes` | `gauge` | Total cluster raw storage free in bytes | | 138 | `minio_cluster_health_capacity_usable_total_bytes` | `gauge` | Total cluster usable storage capacity in bytes | | 139 | `minio_cluster_health_capacity_usable_free_bytes` | `gauge` | Total cluster usable storage free in bytes | | 140 141 142 ### `/cluster/usage/objects` 143 144 | Name | Type | Help | Labels | 145 |----------------------------------------------------------|---------|----------------------------------------------------------------|---------| 146 | `minio_cluster_usage_objects_since_last_update_seconds` | `gauge` | Time since last update of usage metrics in seconds | | 147 | `minio_cluster_usage_objects_total_bytes` | `gauge` | Total cluster usage in bytes | | 148 | `minio_cluster_usage_objects_count` | `gauge` | Total cluster objects count | | 149 | `minio_cluster_usage_objects_versions_count` | `gauge` | Total cluster object versions (including delete markers) count | | 150 | `minio_cluster_usage_objects_delete_markers_count` | `gauge` | Total cluster delete markers count | | 151 | `minio_cluster_usage_objects_buckets_count` | `gauge` | Total cluster buckets count | | 152 | `minio_cluster_usage_objects_size_distribution` | `gauge` | Cluster object size distribution | `range` | 153 | `minio_cluster_usage_objects_version_count_distribution` | `gauge` | Cluster object version count distribution | `range` | 154 155 ### `/cluster/usage/buckets` 156 157 | Name | Type | Help | Labels | 158 |-----------------------------------------------------------------|---------|------------------------------------------------------------------|----------------| 159 | `minio_cluster_usage_buckets_since_last_update_seconds` | `gauge` | Time since last update of usage metrics in seconds | | 160 | `minio_cluster_usage_buckets_total_bytes` | `gauge` | Total bucket size in bytes | `bucket` | 161 | `minio_cluster_usage_buckets_objects_count` | `gauge` | Total objects count in bucket | `bucket` | 162 | `minio_cluster_usage_buckets_versions_count` | `gauge` | Total object versions (including delete markers) count in bucket | `bucket` | 163 | `minio_cluster_usage_buckets_delete_markers_count` | `gauge` | Total delete markers count in bucket | `bucket` | 164 | `minio_cluster_usage_buckets_quota_total_bytes` | `gauge` | Total bucket quota in bytes | `bucket` | 165 | `minio_cluster_usage_buckets_object_size_distribution` | `gauge` | Bucket object size distribution | `range,bucket` | 166 | `minio_cluster_usage_buckets_object_version_count_distribution` | `gauge` | Bucket object version count distribution | `range,bucket` | 167 168 ### `/cluster/erasure-set` 169 170 | Name | Type | Help | Labels | 171 |--------------------------------------------------|---------|---------------------------------------------------------------|------------------| 172 | `minio_cluster_erasure_set_overall_write_quorum` | `gauge` | Overall write quorum across pools and sets | | 173 | `minio_cluster_erasure_set_overall_health` | `gauge` | Overall health across pools and sets (1=healthy, 0=unhealthy) | | 174 | `minio_cluster_erasure_set_read_quorum` | `gauge` | Read quorum for the erasure set in a pool | `pool_id,set_id` | 175 | `minio_cluster_erasure_set_write_quorum` | `gauge` | Write quorum for the erasure set in a pool | `pool_id,set_id` | 176 | `minio_cluster_erasure_set_online_drives_count` | `gauge` | Count of online drives in the erasure set in a pool | `pool_id,set_id` | 177 | `minio_cluster_erasure_set_healing_drives_count` | `gauge` | Count of healing drives in the erasure set in a pool | `pool_id,set_id` | 178 | `minio_cluster_erasure_set_health` | `gauge` | Health of the erasure set in a pool (1=healthy, 0=unhealthy) | `pool_id,set_id` |