github.com/thanos-io/thanos@v0.32.5/docs/components/query.md (about) 1 # Querier/Query 2 3 The `thanos query` command (also known as "Querier") implements the [Prometheus HTTP v1 API](https://prometheus.io/docs/prometheus/latest/querying/api/) to query data in a Thanos cluster via PromQL. 4 5 In short, it gathers the data needed to evaluate the query from underlying [StoreAPIs](https://github.com/thanos-io/thanos/blob/main/pkg/store/storepb/rpc.proto), evaluates the query and returns the result. 6 7 Querier is fully stateless and horizontally scalable. 8 9 Example command to run Querier: 10 11 ```bash 12 thanos query \ 13 --http-address "0.0.0.0:9090" \ 14 --endpoint "<store-api>:<grpc-port>" \ 15 --endpoint "<store-api2>:<grpc-port>" 16 ``` 17 18 ## Querier use cases, why do I need this component? 19 20 Thanos Querier essentially allows to aggregate and optionally deduplicate multiple metrics backends under single Prometheus Query endpoint. 21 22 ### Global View 23 24 Since for Querier "a backend" is anything that implements gRPC StoreAPI we can aggregate data from any number of the different storages like: 25 26 * Prometheus (see [Sidecar](sidecar.md)) 27 * Object Storage (see [Store Gateway](store.md)) 28 * Global alerting/recording rules evaluations (see [Ruler](rule.md)) 29 * Metrics received from Prometheus remote write streams (see [Receiver](receive.md)) 30 * Another Querier (you can stack Queriers on top of each other) 31 * Non-Prometheus systems! 32 * e.g [OpenTSDB](../integrations.md#opentsdb-as-storeapi) 33 34 Thanks to that, you can run queries (manually, from Grafana or via Alerting rule) that aggregate metrics from mix of those sources. 35 36 Some examples: 37 38 * `sum(cpu_used{cluster=~"cluster-(eu1|eu2|eu3|us1|us2|us3)", job="service1"})` that will give you sum of CPU used inside all listed clusters for service `service1`. This will work even if those clusters runs multiple Prometheus servers each. Querier will know which data sources to query. 39 40 * In single cluster you shard Prometheus functionally or have different Prometheus instances for different tenants. You can spin up Querier to have access to both within single Query evaluation. 41 42 ### Run-time deduplication of HA groups 43 44 Prometheus is stateful and does not allow replicating its database. This means that increasing high availability by running multiple Prometheus replicas is not very easy to use. Simple load balancing will not work as for example after some crash, replica might be up but querying such replica will result in small gap during the period it was down. You have a second replica that maybe was up, but it could be down in other moment (e.g rolling restart), so load balancing on top of those is not working well. 45 46 Thanos Querier instead pulls the data from both replicas, and deduplicate those signals, filling the gaps if any, transparently to the Querier consumer. 47 48 ## Metric Query Flow Overview 49 50 <img src="../img/querier.svg" class="img-fluid" alt="querier-steps"/> 51 52 Overall QueryAPI exposed by Thanos is guaranteed to be compatible with [Prometheus 2.x. API](https://prometheus.io/docs/prometheus/latest/querying/api/). The above diagram shows what Querier does for each Prometheus query request. 53 54 See [here](../service-discovery.md) on how to connect Querier with desired StoreAPIs. 55 56 ### Deduplication 57 58 The query layer can deduplicate series that were collected from high-availability pairs of data sources such as Prometheus. A fixed single or multiple replica labels must be chosen for the entire cluster and can then be passed to query nodes on startup. 59 60 Two or more series that are only distinguished by the given replica label, will be merged into a single time series. This also hides gaps in collection of a single data source. 61 62 ### An example with a single replica labels: 63 64 * Prometheus + sidecar "A": `cluster=1,env=2,replica=A` 65 * Prometheus + sidecar "B": `cluster=1,env=2,replica=B` 66 * Prometheus + sidecar "A" in different cluster: `cluster=2,env=2,replica=A` 67 68 If we configure Querier like this: 69 70 ``` 71 thanos query \ 72 --http-address "0.0.0.0:9090" \ 73 --query.replica-label "replica" \ 74 --endpoint "<store-api>:<grpc-port>" \ 75 --endpoint "<store-api2>:<grpc-port>" \ 76 ``` 77 78 And we query for metric `up{job="prometheus",env="2"}` with this option we will get 2 results: 79 80 * `up{job="prometheus",env="2",cluster="1"} 1` 81 * `up{job="prometheus",env="2",cluster="2"} 1` 82 83 WITHOUT this replica flag (deduplication turned off), we will get 3 results: 84 85 * `up{job="prometheus",env="2",cluster="1",replica="A"} 1` 86 * `up{job="prometheus",env="2",cluster="1",replica="B"} 1` 87 * `up{job="prometheus",env="2",cluster="2",replica="A"} 1` 88 89 ### The same output will be present for this example with multiple replica labels: 90 91 * Prometheus + sidecar "A": `cluster=1,env=2,replica=A,replicaX=A` 92 * Prometheus + sidecar "B": `cluster=1,env=2,replica=B,replicaX=B` 93 * Prometheus + sidecar "A" in different cluster: `cluster=2,env=2,replica=A,replicaX=A` 94 95 ``` 96 thanos query \ 97 --http-address "0.0.0.0:9090" \ 98 --query.replica-label "replica" \ 99 --query.replica-label "replicaX" \ 100 --endpoint "<store-api>:<grpc-port>" \ 101 --endpoint "<store-api2>:<grpc-port>" \ 102 ``` 103 104 This logic can also be controlled via parameter on QueryAPI. More details below. 105 106 ## Experimental PromQL Engine 107 108 By default, Thanos querier comes with standard Prometheus PromQL engine. However, when `--query.promql-engine=thanos` is specified, Thanos will use [experimental Thanos PromQL engine](http://github.com/thanos-community/promql-engine) which is a drop-in, efficient implementation of PromQL engine with query planner and optimizers. 109 110 To learn more, see [the introduction talk](https://youtu.be/pjkWzDVxWk4?t=3609) from [the PromConEU 2022](https://promcon.io/2022-munich/talks/opening-pandoras-box-redesigning/). 111 112 This feature is still **experimental** given active development. All queries should be supported due to bulit-in fallback to old PromQL if something is not yet implemented. 113 114 For new engine bugs/issues, please use https://github.com/thanos-community/promql-engine GitHub issues. 115 116 ## Query API Overview 117 118 As mentioned, Query API exposed by Thanos is guaranteed to be compatible with [Prometheus 2.x. API](https://prometheus.io/docs/prometheus/latest/querying/api/). However for additional Thanos features on top of Prometheus, Thanos adds: 119 120 * partial response behaviour 121 * several additional parameters listed below 122 * custom response fields. 123 124 Let's walk through all of those extensions: 125 126 ### Partial Response 127 128 QueryAPI and StoreAPI has additional behaviour controlled via query parameter called [PartialResponseStrategy](../../pkg/store/storepb/rpc.pb.go). 129 130 This parameter controls tradeoff between accuracy and availability. 131 132 Partial response is a potentially missed result within query against QueryAPI or StoreAPI. This can happen if one of StoreAPIs is returning error or timeout whereas couple of others returns success. It does not mean you are missing data, you might lucky enough that you actually get the correct data as the broken StoreAPI did not have anything for your query. 133 134 If partial response happen QueryAPI returns human readable warnings explained [here](#custom-response-fields). 135 136 Now it supports two strategies: 137 * "warn" 138 * "abort" (default) 139 140 NOTE: Having a warning does not necessarily mean partial response (e.g no store matched query warning). 141 142 Querier also allows to configure different timeouts: 143 144 * `--query.timeout` 145 * `--store.response-timeout` 146 147 If you prefer availability over accuracy you can set tighter timeout to underlying StoreAPI than overall query timeout. If partial response strategy is NOT `abort`, this will "ignore" slower StoreAPIs producing just warning with 200 status code response. 148 149 ### Deduplication replica labels. 150 151 | HTTP URL/FORM parameter | Type | Default | Example | 152 |-------------------------|------------|----------------------------------------------|-------------------------------------------------| 153 | `replicaLabels` | `[]string` | `query.replica-label` flag (default: empty). | `replicaLabels=replicaA&replicaLabels=replicaB` | 154 | | | | | 155 156 This overwrites the `query.replica-label` cli flag to allow dynamic replica labels at query time. 157 158 ### Deduplication Enabled 159 160 | HTTP URL/FORM parameter | Type | Default | Example | 161 |-------------------------|-----------|-----------------------------------------------------------------|----------------------------------------| 162 | `dedup` | `Boolean` | True, but effect depends on `query.replica` configuration flag. | `1, t, T, TRUE, true, True` for "True" | 163 | | | | | 164 165 This controls if query results should be deduplicated using the replica labels. 166 167 ### Auto downsampling 168 169 | HTTP URL/FORM parameter | Type | Default | Example | 170 |-------------------------|----------------------------------------|--------------------------------------------------------------------------|---------| 171 | `max_source_resolution` | `Float64/time.Duration/model.Duration` | `step / 5` or `0` if `query.auto-downsampling` is false (default: False) | `5m` | 172 | | | | | 173 174 Max source resolution is max resolution in seconds we want to use for data we query for. 175 176 Available options: 177 178 * `auto` - Select downsample resolution automatically based on the query. 179 * `0` - Only use raw data. 180 * `5m` - Use max 5m downsampling. 181 * `1h` - Use max 1h downsampling. 182 183 ### Partial Response Strategy 184 185 // TODO(bwplotka): Update. This will change to "strategy" soon as [PartialResponseStrategy enum here](../../pkg/store/storepb/rpc.proto) 186 187 | HTTP URL/FORM parameter | Type | Default | Example | 188 |-------------------------|-----------|-----------------------------------------------|----------------------------------------| 189 | `partial_response` | `Boolean` | `query.partial-response` flag (default: True) | `1, t, T, TRUE, true, True` for "True" | 190 | | | | | 191 192 If true, then all storeAPIs that will be unavailable (and thus return no data) will not cause query to fail, but instead return warning. 193 194 ### Custom Response Fields 195 196 Any additional field does not break compatibility, however there is no guarantee that Grafana or any other client will understand those. 197 198 Currently Thanos UI exposed by Thanos understands 199 200 ```go 201 type queryData struct { 202 ResultType promql.ValueType `json:"resultType"` 203 Result promql.Value `json:"result"` 204 205 // Additional Thanos Response field. 206 Warnings []error `json:"warnings,omitempty"` 207 } 208 ``` 209 210 Additional field is `Warnings` that contains every error that occurred that is assumed non critical. `partial_response` option controls if storeAPI unavailability is considered critical. 211 212 ### Concurrent Selects 213 214 Thanos Querier has the ability to perform concurrent select request per query. It dissects given PromQL statement and executes selectors concurrently against the discovered StoreAPIs. The maximum number of concurrent requests are being made per query is controlled by `query.max-concurrent-select` flag. Keep in mind that the maximum number of concurrent queries that are handled by querier is controlled by `query.max-concurrent`. Please consider implications of combined value while tuning the querier. 215 216 ### Store filtering 217 218 It's possible to provide a set of matchers to the Querier api to select specific stores to be used during the query using the `storeMatch[]` parameter. It is useful when debugging a slow/broken store. It uses the same format as the matcher of [Prometheus' federate api](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers). Note that at the moment the querier only supports the `__address__` which contain the address of the store as it is shown on the `/stores` endpoint of the UI. 219 220 Example: 221 222 ``` 223 - targets: 224 - prometheus-foo.thanos-sidecar:10901 225 - prometheus-bar.thanos-sidecar:10901 226 ``` 227 228 ``` 229 http://localhost:10901/api/v1/query?query=up&dedup=true&partial_response=true&storeMatch[]={__address__=~"prometheus-foo.*"} 230 ``` 231 232 Will only return metrics from `prometheus-foo.thanos-sidecar:10901` 233 234 ## Expose UI on a sub-path 235 236 It is possible to expose thanos-query UI and optionally API on a sub-path. The sub-path can be defined either statically or dynamically via an HTTP header. Static path prefix definition follows the pattern used in Prometheus, where `web.route-prefix` option defines HTTP request path prefix (endpoints prefix) and `web.external-prefix` prefixes the URLs in HTML code and the HTTP redirect responses. 237 238 Additionally, Thanos supports dynamic prefix configuration, which [is not yet implemented by Prometheus](https://github.com/prometheus/prometheus/issues/3156). Dynamic prefixing simplifies setup when `thanos query` is exposed on a sub-path behind a reverse proxy, for example, via a Kubernetes ingress controller [Traefik](https://docs.traefik.io/routing/routers/) or [nginx](https://github.com/kubernetes/ingress-nginx/pull/1805). If `PathPrefixStrip: /some-path` option or `traefik.frontend.rule.type: PathPrefixStrip` Kubernetes Ingress annotation is set, then `Traefik` writes the stripped prefix into X-Forwarded-Prefix header. Then, `thanos query --web.prefix-header=X-Forwarded-Prefix` will serve correct HTTP redirects and links prefixed by the stripped path. 239 240 ## File SD 241 242 `--store.sd-files` flag provides a path to a JSON or YAML formatted file, which contains a list of targets in [Prometheus target format](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config). 243 244 Example file SD file in YAML: 245 246 ``` 247 - targets: 248 - prometheus-0.thanos-sidecar:10901 249 - prometheus-1.thanos-sidecar:10901 250 - thanos-store:10901 251 - thanos-short-store:10901 252 - thanos-rule:10901 253 - targets: 254 - prometheus-0.thanos-sidecar.infra:10901 255 - prometheus-1.thanos-sidecar.infra:10901 256 - thanos-store.infra:10901 257 ``` 258 259 ## Active Query Tracking 260 261 `--query.active-query-path` is an option which allows the user to specify a directory which will contain a `queries.active` file to track active queries. To enable this feature, the user has to specify a directory other than "", since that is skipped being the default. 262 263 ## Flags 264 265 ```$ mdox-exec="thanos query --help" 266 usage: thanos query [<flags>] 267 268 Query node exposing PromQL enabled Query API with data retrieved from multiple 269 store nodes. 270 271 Flags: 272 --alert.query-url=ALERT.QUERY-URL 273 The external Thanos Query URL that would be set 274 in all alerts 'Source' field. 275 --enable-feature= ... Comma separated experimental feature names 276 to enable.The current list of features is 277 query-pushdown. 278 --endpoint=<endpoint> ... Addresses of statically configured Thanos 279 API servers (repeatable). The scheme may be 280 prefixed with 'dns+' or 'dnssrv+' to detect 281 Thanos API servers through respective DNS 282 lookups. 283 --endpoint-group=<endpoint-group> ... 284 Experimental: DNS name of statically configured 285 Thanos API server groups (repeatable). Targets 286 resolved from the DNS name will be queried in 287 a round-robin, instead of a fanout manner. 288 This flag should be used when connecting a 289 Thanos Query to HA groups of Thanos components. 290 --endpoint-group-strict=<endpoint-group-strict> ... 291 Experimental: DNS name of statically configured 292 Thanos API server groups (repeatable) that are 293 always used, even if the health check fails. 294 --endpoint-strict=<staticendpoint> ... 295 Addresses of only statically configured Thanos 296 API servers that are always used, even if 297 the health check fails. Useful if you have a 298 caching layer on top. 299 --grpc-address="0.0.0.0:10901" 300 Listen ip:port address for gRPC endpoints 301 (StoreAPI). Make sure this address is routable 302 from other components. 303 --grpc-client-server-name="" 304 Server name to verify the hostname on 305 the returned gRPC certificates. See 306 https://tools.ietf.org/html/rfc4366#section-3.1 307 --grpc-client-tls-ca="" TLS CA Certificates to use to verify gRPC 308 servers 309 --grpc-client-tls-cert="" TLS Certificates to use to identify this client 310 to the server 311 --grpc-client-tls-key="" TLS Key for the client's certificate 312 --grpc-client-tls-secure Use TLS when talking to the gRPC server 313 --grpc-client-tls-skip-verify 314 Disable TLS certificate verification i.e self 315 signed, signed by fake CA 316 --grpc-compression=none Compression algorithm to use for gRPC requests 317 to other clients. Must be one of: snappy, none 318 --grpc-grace-period=2m Time to wait after an interrupt received for 319 GRPC Server. 320 --grpc-server-max-connection-age=60m 321 The grpc server max connection age. This 322 controls how often to re-establish connections 323 and redo TLS handshakes. 324 --grpc-server-tls-cert="" TLS Certificate for gRPC server, leave blank to 325 disable TLS 326 --grpc-server-tls-client-ca="" 327 TLS CA to verify clients against. If no 328 client CA is specified, there is no client 329 verification on server side. (tls.NoClientCert) 330 --grpc-server-tls-key="" TLS Key for the gRPC server, leave blank to 331 disable TLS 332 -h, --help Show context-sensitive help (also try 333 --help-long and --help-man). 334 --http-address="0.0.0.0:10902" 335 Listen host:port for HTTP endpoints. 336 --http-grace-period=2m Time to wait after an interrupt received for 337 HTTP Server. 338 --http.config="" [EXPERIMENTAL] Path to the configuration file 339 that can enable TLS or authentication for all 340 HTTP endpoints. 341 --log.format=logfmt Log format to use. Possible options: logfmt or 342 json. 343 --log.level=info Log filtering level. 344 --log.request.decision= Deprecation Warning - This flag would 345 be soon deprecated, and replaced with 346 `request.logging-config`. Request Logging 347 for logging the start and end of requests. By 348 default this flag is disabled. LogFinishCall: 349 Logs the finish call of the requests. 350 LogStartAndFinishCall: Logs the start and 351 finish call of the requests. NoLogCall: Disable 352 request logging. 353 --query.active-query-path="" 354 Directory to log currently active queries in 355 the queries.active file. 356 --query.auto-downsampling Enable automatic adjustment (step / 5) to what 357 source of data should be used in store gateways 358 if no max_source_resolution param is specified. 359 --query.conn-metric.label=external_labels... ... 360 Optional selection of query connection metric 361 labels to be collected from endpoint set 362 --query.default-evaluation-interval=1m 363 Set default evaluation interval for sub 364 queries. 365 --query.default-step=1s Set default step for range queries. Default 366 step is only used when step is not set in UI. 367 In such cases, Thanos UI will use default 368 step to calculate resolution (resolution 369 = max(rangeSeconds / 250, defaultStep)). 370 This will not work from Grafana, but Grafana 371 has __step variable which can be used. 372 --query.lookback-delta=QUERY.LOOKBACK-DELTA 373 The maximum lookback duration for retrieving 374 metrics during expression evaluations. 375 PromQL always evaluates the query for the 376 certain timestamp (query range timestamps are 377 deduced by step). Since scrape intervals might 378 be different, PromQL looks back for given 379 amount of time to get latest sample. If it 380 exceeds the maximum lookback delta it assumes 381 series is stale and returns none (a gap). 382 This is why lookback delta should be set to at 383 least 2 times of the slowest scrape interval. 384 If unset it will use the promql default of 5m. 385 --query.max-concurrent=20 Maximum number of queries processed 386 concurrently by query node. 387 --query.max-concurrent-select=4 388 Maximum number of select requests made 389 concurrently per a query. 390 --query.metadata.default-time-range=0s 391 The default metadata time range duration for 392 retrieving labels through Labels and Series API 393 when the range parameters are not specified. 394 The zero value means range covers the time 395 since the beginning. 396 --query.partial-response Enable partial response for queries if 397 no partial_response param is specified. 398 --no-query.partial-response for disabling. 399 --query.promql-engine=prometheus 400 Default PromQL engine to use. 401 --query.replica-label=QUERY.REPLICA-LABEL ... 402 Labels to treat as a replica indicator along 403 which data is deduplicated. Still you will 404 be able to query without deduplication using 405 'dedup=false' parameter. Data includes time 406 series, recording rules, and alerting rules. 407 --query.telemetry.request-duration-seconds-quantiles=0.1... ... 408 The quantiles for exporting metrics about the 409 request duration quantiles. 410 --query.telemetry.request-samples-quantiles=100... ... 411 The quantiles for exporting metrics about the 412 samples count quantiles. 413 --query.telemetry.request-series-seconds-quantiles=10... ... 414 The quantiles for exporting metrics about the 415 series count quantiles. 416 --query.timeout=2m Maximum time to process query by query node. 417 --request.logging-config=<content> 418 Alternative to 'request.logging-config-file' 419 flag (mutually exclusive). Content 420 of YAML file with request logging 421 configuration. See format details: 422 https://thanos.io/tip/thanos/logging.md/#configuration 423 --request.logging-config-file=<file-path> 424 Path to YAML file with request logging 425 configuration. See format details: 426 https://thanos.io/tip/thanos/logging.md/#configuration 427 --selector-label=<name>="<value>" ... 428 Query selector labels that will be exposed in 429 info endpoint (repeated). 430 --store=<store> ... Deprecation Warning - This flag is deprecated 431 and replaced with `endpoint`. Addresses of 432 statically configured store API servers 433 (repeatable). The scheme may be prefixed with 434 'dns+' or 'dnssrv+' to detect store API servers 435 through respective DNS lookups. 436 --store-strict=<staticstore> ... 437 Deprecation Warning - This flag is deprecated 438 and replaced with `endpoint-strict`. Addresses 439 of only statically configured store API servers 440 that are always used, even if the health check 441 fails. Useful if you have a caching layer on 442 top. 443 --store.limits.request-samples=0 444 The maximum samples allowed for a single 445 Series request, The Series call fails if 446 this limit is exceeded. 0 means no limit. 447 NOTE: For efficiency the limit is internally 448 implemented as 'chunks limit' considering each 449 chunk contains a maximum of 120 samples. 450 --store.limits.request-series=0 451 The maximum series allowed for a single Series 452 request. The Series call fails if this limit is 453 exceeded. 0 means no limit. 454 --store.response-timeout=0ms 455 If a Store doesn't send any data in this 456 specified duration then a Store will be ignored 457 and partial data will be returned if it's 458 enabled. 0 disables timeout. 459 --store.sd-dns-interval=30s 460 Interval between DNS resolutions. 461 --store.sd-files=<path> ... 462 Path to files that contain addresses of store 463 API servers. The path can be a glob pattern 464 (repeatable). 465 --store.sd-interval=5m Refresh interval to re-read file SD files. 466 It is used as a resync fallback. 467 --store.unhealthy-timeout=5m 468 Timeout before an unhealthy store is cleaned 469 from the store UI page. 470 --tracing.config=<content> 471 Alternative to 'tracing.config-file' flag 472 (mutually exclusive). Content of YAML file 473 with tracing configuration. See format details: 474 https://thanos.io/tip/thanos/tracing.md/#configuration 475 --tracing.config-file=<file-path> 476 Path to YAML file with tracing 477 configuration. See format details: 478 https://thanos.io/tip/thanos/tracing.md/#configuration 479 --version Show application version. 480 --web.disable-cors Whether to disable CORS headers to be set by 481 Thanos. By default Thanos sets CORS headers to 482 be allowed by all. 483 --web.external-prefix="" Static prefix for all HTML links and 484 redirect URLs in the UI query web interface. 485 Actual endpoints are still served on / or the 486 web.route-prefix. This allows thanos UI to be 487 served behind a reverse proxy that strips a URL 488 sub-path. 489 --web.prefix-header="" Name of HTTP request header used for dynamic 490 prefixing of UI links and redirects. 491 This option is ignored if web.external-prefix 492 argument is set. Security risk: enable 493 this option only if a reverse proxy in 494 front of thanos is resetting the header. 495 The --web.prefix-header=X-Forwarded-Prefix 496 option can be useful, for example, if Thanos 497 UI is served via Traefik reverse proxy with 498 PathPrefixStrip option enabled, which sends the 499 stripped prefix value in X-Forwarded-Prefix 500 header. This allows thanos UI to be served on a 501 sub-path. 502 --web.route-prefix="" Prefix for API and UI endpoints. This allows 503 thanos UI to be served on a sub-path. 504 Defaults to the value of --web.external-prefix. 505 This option is analogous to --web.route-prefix 506 of Prometheus. 507 508 ``` 509 510 ## Exported metrics 511 512 Thanos Query also exports metrics about its own performance. You can find a list with these metrics below. 513 514 **Disclaimer**: this list is incomplete. The remaining metrics will be added over time. 515 516 | Name | Type | Labels | Description | 517 |-----------------------------------------|-----------|-------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| 518 | grpc_client_handled_total | Counter | grpc_code, grpc_method, grpc_service, grpc_type | Number of gRPC client requests handled by this query instance (including errors) | 519 | grpc_server_handled_total | Counter | grpc_code, grpc_method, grpc_service, grpc_type | Number of gRPC server requests handled by this query instance (including errors) | 520 | thanos_store_api_query_duration_seconds | Histogram | samples_le, series_le | Duration of the Thanos Store API select phase for a query according to the amount of samples and series selected. |