github.com/muhammadn/cortex@v1.9.1-0.20220510110439-46bb7000d03d/docs/blocks-storage/querier.md (about) 1 --- 2 title: "Querier" 3 linkTitle: "Querier" 4 weight: 2 5 slug: querier 6 --- 7 8 <!-- DO NOT EDIT THIS FILE - This file has been automatically generated from its .template --> 9 10 The **querier** service handles queries using the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language. The querier service is used both by the chunks and blocks storage, and the [general architecture documentation](../architecture.md#querier) applies to the blocks storage too, except for the differences described in this document. 11 12 The querier is **stateless**. 13 14 ## How it works 15 16 The querier needs to have an almost up-to-date view over the entire storage bucket, in order to find the right blocks to lookup at query time. The querier can keep the bucket view updated in to two different ways: 17 18 1. Periodically scanning the bucket (default) 19 2. Periodically downloading the [bucket index](./bucket-index.md) 20 21 ### Bucket index disabled (default) 22 23 At startup, **queriers** iterate over the entire storage bucket to discover all tenants blocks and download the `meta.json` for each block. During this initial bucket scanning phase, a querier is not ready to handle incoming queries yet and its `/ready` readiness probe endpoint will fail. 24 25 While running, queriers periodically iterate over the storage bucket to discover new tenants and recently uploaded blocks. Queriers do **not** download any content from blocks except a small `meta.json` file containing the block's metadata (including the minimum and maximum timestamp of samples within the block). 26 27 Queriers use the metadata to compute the list of blocks that need to be queried at query time and fetch matching series from the [store-gateway](./store-gateway.md) instances holding the required blocks. 28 29 ### Bucket index enabled 30 31 When [bucket index](./bucket-index.md) is enabled, queriers lazily download the bucket index upon the first query received for a given tenant, cache it in memory and periodically keep it update. The bucket index contains the list of blocks and block deletion marks of a tenant, which is later used during the query execution to find the set of blocks that need to be queried for the given query. 32 33 Given the bucket index removes the need to scan the bucket, it brings few benefits: 34 35 1. The querier is expected to be ready shortly after startup. 36 2. Lower volume of API calls to object storage. 37 38 ### Anatomy of a query request 39 40 When a querier receives a query range request, it contains the following parameters: 41 42 - `query`: the PromQL query expression itself (e.g. `rate(node_cpu_seconds_total[1m])`) 43 - `start`: the start time 44 - `end`: the end time 45 - `step`: the query resolution (e.g. `30` to have 1 resulting data point every 30s) 46 47 Given a query, the querier analyzes the `start` and `end` time range to compute a list of all known blocks containing at least 1 sample within this time range. Given the list of blocks, the querier then computes a list of store-gateway instances holding these blocks and sends a request to each matching store-gateway instance asking to fetch all the samples for the series matching the `query` within the `start` and `end` time range. 48 49 The request sent to each store-gateway contains the list of block IDs that are expected to be queried, and the response sent back by the store-gateway to the querier contains the list of block IDs that were actually queried. This list may be a subset of the requested blocks, for example due to recent blocks resharding event (ie. last few seconds). The querier runs a consistency check on responses received from the store-gateways to ensure all expected blocks have been queried; if not, the querier retries to fetch samples from missing blocks from different store-gateways (if the `-store-gateway.sharding-ring.replication-factor` is greater than `1`) and if the consistency check fails after all retries, the query execution fails as well (correctness is always guaranteed). 50 51 If the query time range covers a period within `-querier.query-ingesters-within` duration, the querier also sends the request to all ingesters, in order to fetch samples that have not been uploaded to the long-term storage yet. 52 53 Once all samples have been fetched from both store-gateways and ingesters, the querier proceeds with running the PromQL engine to execute the query and send back the result to the client. 54 55 ### How queriers connect to store-gateway 56 57 Queriers need to discover store-gateways in order to connect to them at query time. The service discovery mechanism used depends whether blocks sharding is enabled in the store-gateways. 58 59 When blocks sharding is **enabled**, queriers need to access to the store-gateways hash ring and thus queriers need to be configured with the same `-store-gateway.sharding-ring.*` flags (or their respective YAML config options) store-gateways have been configured. 60 61 When blocks sharding is **disabled**, queriers need the `-querier.store-gateway-addresses` CLI flag (or its respective YAML config option) being set to a comma separated list of store-gateway addresses in [DNS Service Discovery format]((../configuration/arguments.md#dns-service-discovery). Queriers will evenly balance the requests to query blocks across the resolved addresses. 62 63 ## Caching 64 65 The querier supports the following caches: 66 67 - [Metadata cache](#metadata-cache) 68 69 Caching is optional, but **highly recommended** in a production environment. Please also check out the [production tips](./production-tips.md#caching) for more information about configuring the cache. 70 71 ### Metadata cache 72 73 [Store-gateway](./store-gateway.md) and querier can use memcached for caching bucket metadata: 74 75 - List of tenants 76 - List of blocks per tenant 77 - Block's `meta.json` content 78 - Block's `deletion-mark.json` existence and content 79 - Tenant's `bucket-index.json.gz` content 80 81 Using the metadata cache can significantly reduce the number of API calls to object storage and protects from linearly scale the number of these API calls with the number of querier and store-gateway instances (because the bucket is periodically scanned and synched by each querier and store-gateway). 82 83 To enable metadata cache, please set `-blocks-storage.bucket-store.metadata-cache.backend`. Only `memcached` backend is supported currently. Memcached client has additional configuration available via flags with `-blocks-storage.bucket-store.metadata-cache.memcached.*` prefix. 84 85 Additional options for configuring metadata cache have `-blocks-storage.bucket-store.metadata-cache.*` prefix. By configuring TTL to zero or negative value, caching of given item type is disabled. 86 87 _The same memcached backend cluster should be shared between store-gateways and queriers._ 88 89 ## Querier configuration 90 91 This section described the querier configuration. For the general Cortex configuration and references to common config blocks, please refer to the [configuration documentation](../configuration/config-file-reference.md). 92 93 ### `querier_config` 94 95 The `querier_config` configures the Cortex querier. 96 97 ```yaml 98 querier: 99 # The maximum number of concurrent queries. 100 # CLI flag: -querier.max-concurrent 101 [max_concurrent: <int> | default = 20] 102 103 # The timeout for a query. 104 # CLI flag: -querier.timeout 105 [timeout: <duration> | default = 2m] 106 107 # Use iterators to execute query, as opposed to fully materialising the series 108 # in memory. 109 # CLI flag: -querier.iterators 110 [iterators: <boolean> | default = false] 111 112 # Use batch iterators to execute query, as opposed to fully materialising the 113 # series in memory. Takes precedent over the -querier.iterators flag. 114 # CLI flag: -querier.batch-iterators 115 [batch_iterators: <boolean> | default = true] 116 117 # Use streaming RPCs to query ingester. 118 # CLI flag: -querier.ingester-streaming 119 [ingester_streaming: <boolean> | default = true] 120 121 # Maximum number of samples a single query can load into memory. 122 # CLI flag: -querier.max-samples 123 [max_samples: <int> | default = 50000000] 124 125 # Maximum lookback beyond which queries are not sent to ingester. 0 means all 126 # queries are sent to ingester. 127 # CLI flag: -querier.query-ingesters-within 128 [query_ingesters_within: <duration> | default = 0s] 129 130 # Query long-term store for series, label values and label names APIs. Works 131 # only with blocks engine. 132 # CLI flag: -querier.query-store-for-labels-enabled 133 [query_store_for_labels_enabled: <boolean> | default = false] 134 135 # Enable the @ modifier in PromQL. 136 # CLI flag: -querier.at-modifier-enabled 137 [at_modifier_enabled: <boolean> | default = false] 138 139 # The time after which a metric should be queried from storage and not just 140 # ingesters. 0 means all queries are sent to store. When running the blocks 141 # storage, if this option is enabled, the time range of the query sent to the 142 # store will be manipulated to ensure the query end is not more recent than 143 # 'now - query-store-after'. 144 # CLI flag: -querier.query-store-after 145 [query_store_after: <duration> | default = 0s] 146 147 # Maximum duration into the future you can query. 0 to disable. 148 # CLI flag: -querier.max-query-into-future 149 [max_query_into_future: <duration> | default = 10m] 150 151 # The default evaluation interval or step size for subqueries. 152 # CLI flag: -querier.default-evaluation-interval 153 [default_evaluation_interval: <duration> | default = 1m] 154 155 # Active query tracker monitors active queries, and writes them to the file in 156 # given directory. If Cortex discovers any queries in this log during startup, 157 # it will log them to the log file. Setting to empty value disables active 158 # query tracker, which also disables -querier.max-concurrent option. 159 # CLI flag: -querier.active-query-tracker-dir 160 [active_query_tracker_dir: <string> | default = "./active-query-tracker"] 161 162 # Time since the last sample after which a time series is considered stale and 163 # ignored by expression evaluations. 164 # CLI flag: -querier.lookback-delta 165 [lookback_delta: <duration> | default = 5m] 166 167 # Comma separated list of store-gateway addresses in DNS Service Discovery 168 # format. This option should be set when using the blocks storage and the 169 # store-gateway sharding is disabled (when enabled, the store-gateway 170 # instances form a ring and addresses are picked from the ring). 171 # CLI flag: -querier.store-gateway-addresses 172 [store_gateway_addresses: <string> | default = ""] 173 174 store_gateway_client: 175 # Enable TLS for gRPC client connecting to store-gateway. 176 # CLI flag: -querier.store-gateway-client.tls-enabled 177 [tls_enabled: <boolean> | default = false] 178 179 # Path to the client certificate file, which will be used for authenticating 180 # with the server. Also requires the key path to be configured. 181 # CLI flag: -querier.store-gateway-client.tls-cert-path 182 [tls_cert_path: <string> | default = ""] 183 184 # Path to the key file for the client certificate. Also requires the client 185 # certificate to be configured. 186 # CLI flag: -querier.store-gateway-client.tls-key-path 187 [tls_key_path: <string> | default = ""] 188 189 # Path to the CA certificates file to validate server certificate against. 190 # If not set, the host's root CA certificates are used. 191 # CLI flag: -querier.store-gateway-client.tls-ca-path 192 [tls_ca_path: <string> | default = ""] 193 194 # Override the expected name on the server certificate. 195 # CLI flag: -querier.store-gateway-client.tls-server-name 196 [tls_server_name: <string> | default = ""] 197 198 # Skip validating server certificate. 199 # CLI flag: -querier.store-gateway-client.tls-insecure-skip-verify 200 [tls_insecure_skip_verify: <boolean> | default = false] 201 202 # Second store engine to use for querying. Empty = disabled. 203 # CLI flag: -querier.second-store-engine 204 [second_store_engine: <string> | default = ""] 205 206 # If specified, second store is only used for queries before this timestamp. 207 # Default value 0 means secondary store is always queried. 208 # CLI flag: -querier.use-second-store-before-time 209 [use_second_store_before_time: <time> | default = 0] 210 211 # When distributor's sharding strategy is shuffle-sharding and this setting is 212 # > 0, queriers fetch in-memory series from the minimum set of required 213 # ingesters, selecting only ingesters which may have received series since 214 # 'now - lookback period'. The lookback period should be greater or equal than 215 # the configured 'query store after' and 'query ingesters within'. If this 216 # setting is 0, queriers always query all ingesters (ingesters shuffle 217 # sharding on read path is disabled). 218 # CLI flag: -querier.shuffle-sharding-ingesters-lookback-period 219 [shuffle_sharding_ingesters_lookback_period: <duration> | default = 0s] 220 ``` 221 222 ### `blocks_storage_config` 223 224 The `blocks_storage_config` configures the blocks storage. 225 226 ```yaml 227 blocks_storage: 228 # Backend storage to use. Supported backends are: s3, gcs, azure, swift, 229 # filesystem. 230 # CLI flag: -blocks-storage.backend 231 [backend: <string> | default = "s3"] 232 233 s3: 234 # The S3 bucket endpoint. It could be an AWS S3 endpoint listed at 235 # https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an 236 # S3-compatible service in hostname:port format. 237 # CLI flag: -blocks-storage.s3.endpoint 238 [endpoint: <string> | default = ""] 239 240 # S3 region. If unset, the client will issue a S3 GetBucketLocation API call 241 # to autodetect it. 242 # CLI flag: -blocks-storage.s3.region 243 [region: <string> | default = ""] 244 245 # S3 bucket name 246 # CLI flag: -blocks-storage.s3.bucket-name 247 [bucket_name: <string> | default = ""] 248 249 # S3 secret access key 250 # CLI flag: -blocks-storage.s3.secret-access-key 251 [secret_access_key: <string> | default = ""] 252 253 # S3 access key ID 254 # CLI flag: -blocks-storage.s3.access-key-id 255 [access_key_id: <string> | default = ""] 256 257 # If enabled, use http:// for the S3 endpoint instead of https://. This 258 # could be useful in local dev/test environments while using an 259 # S3-compatible backend storage, like Minio. 260 # CLI flag: -blocks-storage.s3.insecure 261 [insecure: <boolean> | default = false] 262 263 # The signature version to use for authenticating against S3. Supported 264 # values are: v4, v2. 265 # CLI flag: -blocks-storage.s3.signature-version 266 [signature_version: <string> | default = "v4"] 267 268 # The s3_sse_config configures the S3 server-side encryption. 269 # The CLI flags prefix for this block config is: blocks-storage 270 [sse: <s3_sse_config>] 271 272 http: 273 # The time an idle connection will remain idle before closing. 274 # CLI flag: -blocks-storage.s3.http.idle-conn-timeout 275 [idle_conn_timeout: <duration> | default = 1m30s] 276 277 # The amount of time the client will wait for a servers response headers. 278 # CLI flag: -blocks-storage.s3.http.response-header-timeout 279 [response_header_timeout: <duration> | default = 2m] 280 281 # If the client connects to S3 via HTTPS and this option is enabled, the 282 # client will accept any certificate and hostname. 283 # CLI flag: -blocks-storage.s3.http.insecure-skip-verify 284 [insecure_skip_verify: <boolean> | default = false] 285 286 # Maximum time to wait for a TLS handshake. 0 means no limit. 287 # CLI flag: -blocks-storage.s3.tls-handshake-timeout 288 [tls_handshake_timeout: <duration> | default = 10s] 289 290 # The time to wait for a server's first response headers after fully 291 # writing the request headers if the request has an Expect header. 0 to 292 # send the request body immediately. 293 # CLI flag: -blocks-storage.s3.expect-continue-timeout 294 [expect_continue_timeout: <duration> | default = 1s] 295 296 # Maximum number of idle (keep-alive) connections across all hosts. 0 297 # means no limit. 298 # CLI flag: -blocks-storage.s3.max-idle-connections 299 [max_idle_connections: <int> | default = 100] 300 301 # Maximum number of idle (keep-alive) connections to keep per-host. If 0, 302 # a built-in default value is used. 303 # CLI flag: -blocks-storage.s3.max-idle-connections-per-host 304 [max_idle_connections_per_host: <int> | default = 100] 305 306 # Maximum number of connections per host. 0 means no limit. 307 # CLI flag: -blocks-storage.s3.max-connections-per-host 308 [max_connections_per_host: <int> | default = 0] 309 310 gcs: 311 # GCS bucket name 312 # CLI flag: -blocks-storage.gcs.bucket-name 313 [bucket_name: <string> | default = ""] 314 315 # JSON representing either a Google Developers Console 316 # client_credentials.json file or a Google Developers service account key 317 # file. If empty, fallback to Google default logic. 318 # CLI flag: -blocks-storage.gcs.service-account 319 [service_account: <string> | default = ""] 320 321 azure: 322 # Azure storage account name 323 # CLI flag: -blocks-storage.azure.account-name 324 [account_name: <string> | default = ""] 325 326 # Azure storage account key 327 # CLI flag: -blocks-storage.azure.account-key 328 [account_key: <string> | default = ""] 329 330 # Azure storage container name 331 # CLI flag: -blocks-storage.azure.container-name 332 [container_name: <string> | default = ""] 333 334 # Azure storage endpoint suffix without schema. The account name will be 335 # prefixed to this value to create the FQDN 336 # CLI flag: -blocks-storage.azure.endpoint-suffix 337 [endpoint_suffix: <string> | default = ""] 338 339 # Number of retries for recoverable errors 340 # CLI flag: -blocks-storage.azure.max-retries 341 [max_retries: <int> | default = 20] 342 343 swift: 344 # OpenStack Swift authentication API version. 0 to autodetect. 345 # CLI flag: -blocks-storage.swift.auth-version 346 [auth_version: <int> | default = 0] 347 348 # OpenStack Swift authentication URL 349 # CLI flag: -blocks-storage.swift.auth-url 350 [auth_url: <string> | default = ""] 351 352 # OpenStack Swift username. 353 # CLI flag: -blocks-storage.swift.username 354 [username: <string> | default = ""] 355 356 # OpenStack Swift user's domain name. 357 # CLI flag: -blocks-storage.swift.user-domain-name 358 [user_domain_name: <string> | default = ""] 359 360 # OpenStack Swift user's domain ID. 361 # CLI flag: -blocks-storage.swift.user-domain-id 362 [user_domain_id: <string> | default = ""] 363 364 # OpenStack Swift user ID. 365 # CLI flag: -blocks-storage.swift.user-id 366 [user_id: <string> | default = ""] 367 368 # OpenStack Swift API key. 369 # CLI flag: -blocks-storage.swift.password 370 [password: <string> | default = ""] 371 372 # OpenStack Swift user's domain ID. 373 # CLI flag: -blocks-storage.swift.domain-id 374 [domain_id: <string> | default = ""] 375 376 # OpenStack Swift user's domain name. 377 # CLI flag: -blocks-storage.swift.domain-name 378 [domain_name: <string> | default = ""] 379 380 # OpenStack Swift project ID (v2,v3 auth only). 381 # CLI flag: -blocks-storage.swift.project-id 382 [project_id: <string> | default = ""] 383 384 # OpenStack Swift project name (v2,v3 auth only). 385 # CLI flag: -blocks-storage.swift.project-name 386 [project_name: <string> | default = ""] 387 388 # ID of the OpenStack Swift project's domain (v3 auth only), only needed if 389 # it differs the from user domain. 390 # CLI flag: -blocks-storage.swift.project-domain-id 391 [project_domain_id: <string> | default = ""] 392 393 # Name of the OpenStack Swift project's domain (v3 auth only), only needed 394 # if it differs from the user domain. 395 # CLI flag: -blocks-storage.swift.project-domain-name 396 [project_domain_name: <string> | default = ""] 397 398 # OpenStack Swift Region to use (v2,v3 auth only). 399 # CLI flag: -blocks-storage.swift.region-name 400 [region_name: <string> | default = ""] 401 402 # Name of the OpenStack Swift container to put chunks in. 403 # CLI flag: -blocks-storage.swift.container-name 404 [container_name: <string> | default = ""] 405 406 # Max retries on requests error. 407 # CLI flag: -blocks-storage.swift.max-retries 408 [max_retries: <int> | default = 3] 409 410 # Time after which a connection attempt is aborted. 411 # CLI flag: -blocks-storage.swift.connect-timeout 412 [connect_timeout: <duration> | default = 10s] 413 414 # Time after which an idle request is aborted. The timeout watchdog is reset 415 # each time some data is received, so the timeout triggers after X time no 416 # data is received on a request. 417 # CLI flag: -blocks-storage.swift.request-timeout 418 [request_timeout: <duration> | default = 5s] 419 420 filesystem: 421 # Local filesystem storage directory. 422 # CLI flag: -blocks-storage.filesystem.dir 423 [dir: <string> | default = ""] 424 425 # This configures how the querier and store-gateway discover and synchronize 426 # blocks stored in the bucket. 427 bucket_store: 428 # Directory to store synchronized TSDB index headers. 429 # CLI flag: -blocks-storage.bucket-store.sync-dir 430 [sync_dir: <string> | default = "tsdb-sync"] 431 432 # How frequently to scan the bucket, or to refresh the bucket index (if 433 # enabled), in order to look for changes (new blocks shipped by ingesters 434 # and blocks deleted by retention or compaction). 435 # CLI flag: -blocks-storage.bucket-store.sync-interval 436 [sync_interval: <duration> | default = 15m] 437 438 # Max number of concurrent queries to execute against the long-term storage. 439 # The limit is shared across all tenants. 440 # CLI flag: -blocks-storage.bucket-store.max-concurrent 441 [max_concurrent: <int> | default = 100] 442 443 # Maximum number of concurrent tenants synching blocks. 444 # CLI flag: -blocks-storage.bucket-store.tenant-sync-concurrency 445 [tenant_sync_concurrency: <int> | default = 10] 446 447 # Maximum number of concurrent blocks synching per tenant. 448 # CLI flag: -blocks-storage.bucket-store.block-sync-concurrency 449 [block_sync_concurrency: <int> | default = 20] 450 451 # Number of Go routines to use when syncing block meta files from object 452 # storage per tenant. 453 # CLI flag: -blocks-storage.bucket-store.meta-sync-concurrency 454 [meta_sync_concurrency: <int> | default = 20] 455 456 # Minimum age of a block before it's being read. Set it to safe value (e.g 457 # 30m) if your object storage is eventually consistent. GCS and S3 are 458 # (roughly) strongly consistent. 459 # CLI flag: -blocks-storage.bucket-store.consistency-delay 460 [consistency_delay: <duration> | default = 0s] 461 462 index_cache: 463 # The index cache backend type. Supported values: inmemory, memcached. 464 # CLI flag: -blocks-storage.bucket-store.index-cache.backend 465 [backend: <string> | default = "inmemory"] 466 467 inmemory: 468 # Maximum size in bytes of in-memory index cache used to speed up blocks 469 # index lookups (shared between all tenants). 470 # CLI flag: -blocks-storage.bucket-store.index-cache.inmemory.max-size-bytes 471 [max_size_bytes: <int> | default = 1073741824] 472 473 memcached: 474 # Comma separated list of memcached addresses. Supported prefixes are: 475 # dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV 476 # query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup 477 # made after that). 478 # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.addresses 479 [addresses: <string> | default = ""] 480 481 # The socket read/write timeout. 482 # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.timeout 483 [timeout: <duration> | default = 100ms] 484 485 # The maximum number of idle connections that will be maintained per 486 # address. 487 # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections 488 [max_idle_connections: <int> | default = 16] 489 490 # The maximum number of concurrent asynchronous operations can occur. 491 # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-async-concurrency 492 [max_async_concurrency: <int> | default = 50] 493 494 # The maximum number of enqueued asynchronous operations allowed. 495 # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-async-buffer-size 496 [max_async_buffer_size: <int> | default = 10000] 497 498 # The maximum number of concurrent connections running get operations. 499 # If set to 0, concurrency is unlimited. 500 # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency 501 [max_get_multi_concurrency: <int> | default = 100] 502 503 # The maximum number of keys a single underlying get operation should 504 # run. If more keys are specified, internally keys are split into 505 # multiple batches and fetched concurrently, honoring the max 506 # concurrency. If set to 0, the max batch size is unlimited. 507 # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-batch-size 508 [max_get_multi_batch_size: <int> | default = 0] 509 510 # The maximum size of an item stored in memcached. Bigger items are not 511 # stored. If set to 0, no maximum size is enforced. 512 # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-item-size 513 [max_item_size: <int> | default = 1048576] 514 515 # Use memcached auto-discovery mechanism provided by some cloud provider 516 # like GCP and AWS 517 # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.auto-discovery 518 [auto_discovery: <boolean> | default = false] 519 520 chunks_cache: 521 # Backend for chunks cache, if not empty. Supported values: memcached. 522 # CLI flag: -blocks-storage.bucket-store.chunks-cache.backend 523 [backend: <string> | default = ""] 524 525 memcached: 526 # Comma separated list of memcached addresses. Supported prefixes are: 527 # dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV 528 # query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup 529 # made after that). 530 # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.addresses 531 [addresses: <string> | default = ""] 532 533 # The socket read/write timeout. 534 # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.timeout 535 [timeout: <duration> | default = 100ms] 536 537 # The maximum number of idle connections that will be maintained per 538 # address. 539 # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-idle-connections 540 [max_idle_connections: <int> | default = 16] 541 542 # The maximum number of concurrent asynchronous operations can occur. 543 # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-async-concurrency 544 [max_async_concurrency: <int> | default = 50] 545 546 # The maximum number of enqueued asynchronous operations allowed. 547 # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-async-buffer-size 548 [max_async_buffer_size: <int> | default = 10000] 549 550 # The maximum number of concurrent connections running get operations. 551 # If set to 0, concurrency is unlimited. 552 # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-get-multi-concurrency 553 [max_get_multi_concurrency: <int> | default = 100] 554 555 # The maximum number of keys a single underlying get operation should 556 # run. If more keys are specified, internally keys are split into 557 # multiple batches and fetched concurrently, honoring the max 558 # concurrency. If set to 0, the max batch size is unlimited. 559 # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-get-multi-batch-size 560 [max_get_multi_batch_size: <int> | default = 0] 561 562 # The maximum size of an item stored in memcached. Bigger items are not 563 # stored. If set to 0, no maximum size is enforced. 564 # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-item-size 565 [max_item_size: <int> | default = 1048576] 566 567 # Use memcached auto-discovery mechanism provided by some cloud provider 568 # like GCP and AWS 569 # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.auto-discovery 570 [auto_discovery: <boolean> | default = false] 571 572 # Size of each subrange that bucket object is split into for better 573 # caching. 574 # CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-size 575 [subrange_size: <int> | default = 16000] 576 577 # Maximum number of sub-GetRange requests that a single GetRange request 578 # can be split into when fetching chunks. Zero or negative value = 579 # unlimited number of sub-requests. 580 # CLI flag: -blocks-storage.bucket-store.chunks-cache.max-get-range-requests 581 [max_get_range_requests: <int> | default = 3] 582 583 # TTL for caching object attributes for chunks. 584 # CLI flag: -blocks-storage.bucket-store.chunks-cache.attributes-ttl 585 [attributes_ttl: <duration> | default = 168h] 586 587 # TTL for caching individual chunks subranges. 588 # CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-ttl 589 [subrange_ttl: <duration> | default = 24h] 590 591 metadata_cache: 592 # Backend for metadata cache, if not empty. Supported values: memcached. 593 # CLI flag: -blocks-storage.bucket-store.metadata-cache.backend 594 [backend: <string> | default = ""] 595 596 memcached: 597 # Comma separated list of memcached addresses. Supported prefixes are: 598 # dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV 599 # query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup 600 # made after that). 601 # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.addresses 602 [addresses: <string> | default = ""] 603 604 # The socket read/write timeout. 605 # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.timeout 606 [timeout: <duration> | default = 100ms] 607 608 # The maximum number of idle connections that will be maintained per 609 # address. 610 # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-idle-connections 611 [max_idle_connections: <int> | default = 16] 612 613 # The maximum number of concurrent asynchronous operations can occur. 614 # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-async-concurrency 615 [max_async_concurrency: <int> | default = 50] 616 617 # The maximum number of enqueued asynchronous operations allowed. 618 # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-async-buffer-size 619 [max_async_buffer_size: <int> | default = 10000] 620 621 # The maximum number of concurrent connections running get operations. 622 # If set to 0, concurrency is unlimited. 623 # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-get-multi-concurrency 624 [max_get_multi_concurrency: <int> | default = 100] 625 626 # The maximum number of keys a single underlying get operation should 627 # run. If more keys are specified, internally keys are split into 628 # multiple batches and fetched concurrently, honoring the max 629 # concurrency. If set to 0, the max batch size is unlimited. 630 # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-get-multi-batch-size 631 [max_get_multi_batch_size: <int> | default = 0] 632 633 # The maximum size of an item stored in memcached. Bigger items are not 634 # stored. If set to 0, no maximum size is enforced. 635 # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-item-size 636 [max_item_size: <int> | default = 1048576] 637 638 # Use memcached auto-discovery mechanism provided by some cloud provider 639 # like GCP and AWS 640 # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.auto-discovery 641 [auto_discovery: <boolean> | default = false] 642 643 # How long to cache list of tenants in the bucket. 644 # CLI flag: -blocks-storage.bucket-store.metadata-cache.tenants-list-ttl 645 [tenants_list_ttl: <duration> | default = 15m] 646 647 # How long to cache list of blocks for each tenant. 648 # CLI flag: -blocks-storage.bucket-store.metadata-cache.tenant-blocks-list-ttl 649 [tenant_blocks_list_ttl: <duration> | default = 5m] 650 651 # How long to cache list of chunks for a block. 652 # CLI flag: -blocks-storage.bucket-store.metadata-cache.chunks-list-ttl 653 [chunks_list_ttl: <duration> | default = 24h] 654 655 # How long to cache information that block metafile exists. Also used for 656 # user deletion mark file. 657 # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-exists-ttl 658 [metafile_exists_ttl: <duration> | default = 2h] 659 660 # How long to cache information that block metafile doesn't exist. Also 661 # used for user deletion mark file. 662 # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-doesnt-exist-ttl 663 [metafile_doesnt_exist_ttl: <duration> | default = 5m] 664 665 # How long to cache content of the metafile. 666 # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-content-ttl 667 [metafile_content_ttl: <duration> | default = 24h] 668 669 # Maximum size of metafile content to cache in bytes. Caching will be 670 # skipped if the content exceeds this size. This is useful to avoid 671 # network round trip for large content if the configured caching backend 672 # has an hard limit on cached items size (in this case, you should set 673 # this limit to the same limit in the caching backend). 674 # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-max-size-bytes 675 [metafile_max_size_bytes: <int> | default = 1048576] 676 677 # How long to cache attributes of the block metafile. 678 # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-attributes-ttl 679 [metafile_attributes_ttl: <duration> | default = 168h] 680 681 # How long to cache attributes of the block index. 682 # CLI flag: -blocks-storage.bucket-store.metadata-cache.block-index-attributes-ttl 683 [block_index_attributes_ttl: <duration> | default = 168h] 684 685 # How long to cache content of the bucket index. 686 # CLI flag: -blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl 687 [bucket_index_content_ttl: <duration> | default = 5m] 688 689 # Maximum size of bucket index content to cache in bytes. Caching will be 690 # skipped if the content exceeds this size. This is useful to avoid 691 # network round trip for large content if the configured caching backend 692 # has an hard limit on cached items size (in this case, you should set 693 # this limit to the same limit in the caching backend). 694 # CLI flag: -blocks-storage.bucket-store.metadata-cache.bucket-index-max-size-bytes 695 [bucket_index_max_size_bytes: <int> | default = 1048576] 696 697 # Duration after which the blocks marked for deletion will be filtered out 698 # while fetching blocks. The idea of ignore-deletion-marks-delay is to 699 # ignore blocks that are marked for deletion with some delay. This ensures 700 # store can still serve blocks that are meant to be deleted but do not have 701 # a replacement yet. Default is 6h, half of the default value for 702 # -compactor.deletion-delay. 703 # CLI flag: -blocks-storage.bucket-store.ignore-deletion-marks-delay 704 [ignore_deletion_mark_delay: <duration> | default = 6h] 705 706 bucket_index: 707 # True to enable querier and store-gateway to discover blocks in the 708 # storage via bucket index instead of bucket scanning. 709 # CLI flag: -blocks-storage.bucket-store.bucket-index.enabled 710 [enabled: <boolean> | default = false] 711 712 # How frequently a bucket index, which previously failed to load, should 713 # be tried to load again. This option is used only by querier. 714 # CLI flag: -blocks-storage.bucket-store.bucket-index.update-on-error-interval 715 [update_on_error_interval: <duration> | default = 1m] 716 717 # How long a unused bucket index should be cached. Once this timeout 718 # expires, the unused bucket index is removed from the in-memory cache. 719 # This option is used only by querier. 720 # CLI flag: -blocks-storage.bucket-store.bucket-index.idle-timeout 721 [idle_timeout: <duration> | default = 1h] 722 723 # The maximum allowed age of a bucket index (last updated) before queries 724 # start failing because the bucket index is too old. The bucket index is 725 # periodically updated by the compactor, while this check is enforced in 726 # the querier (at query time). 727 # CLI flag: -blocks-storage.bucket-store.bucket-index.max-stale-period 728 [max_stale_period: <duration> | default = 1h] 729 730 # Max size - in bytes - of a chunks pool, used to reduce memory allocations. 731 # The pool is shared across all tenants. 0 to disable the limit. 732 # CLI flag: -blocks-storage.bucket-store.max-chunk-pool-bytes 733 [max_chunk_pool_bytes: <int> | default = 2147483648] 734 735 # If enabled, store-gateway will lazy load an index-header only once 736 # required by a query. 737 # CLI flag: -blocks-storage.bucket-store.index-header-lazy-loading-enabled 738 [index_header_lazy_loading_enabled: <boolean> | default = false] 739 740 # If index-header lazy loading is enabled and this setting is > 0, the 741 # store-gateway will offload unused index-headers after 'idle timeout' 742 # inactivity. 743 # CLI flag: -blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout 744 [index_header_lazy_loading_idle_timeout: <duration> | default = 20m] 745 746 tsdb: 747 # Local directory to store TSDBs in the ingesters. 748 # CLI flag: -blocks-storage.tsdb.dir 749 [dir: <string> | default = "tsdb"] 750 751 # TSDB blocks range period. 752 # CLI flag: -blocks-storage.tsdb.block-ranges-period 753 [block_ranges_period: <list of duration> | default = 2h0m0s] 754 755 # TSDB blocks retention in the ingester before a block is removed. This 756 # should be larger than the block_ranges_period and large enough to give 757 # store-gateways and queriers enough time to discover newly uploaded blocks. 758 # CLI flag: -blocks-storage.tsdb.retention-period 759 [retention_period: <duration> | default = 6h] 760 761 # How frequently the TSDB blocks are scanned and new ones are shipped to the 762 # storage. 0 means shipping is disabled. 763 # CLI flag: -blocks-storage.tsdb.ship-interval 764 [ship_interval: <duration> | default = 1m] 765 766 # Maximum number of tenants concurrently shipping blocks to the storage. 767 # CLI flag: -blocks-storage.tsdb.ship-concurrency 768 [ship_concurrency: <int> | default = 10] 769 770 # How frequently does Cortex try to compact TSDB head. Block is only created 771 # if data covers smallest block range. Must be greater than 0 and max 5 772 # minutes. 773 # CLI flag: -blocks-storage.tsdb.head-compaction-interval 774 [head_compaction_interval: <duration> | default = 1m] 775 776 # Maximum number of tenants concurrently compacting TSDB head into a new 777 # block 778 # CLI flag: -blocks-storage.tsdb.head-compaction-concurrency 779 [head_compaction_concurrency: <int> | default = 5] 780 781 # If TSDB head is idle for this duration, it is compacted. Note that up to 782 # 25% jitter is added to the value to avoid ingesters compacting 783 # concurrently. 0 means disabled. 784 # CLI flag: -blocks-storage.tsdb.head-compaction-idle-timeout 785 [head_compaction_idle_timeout: <duration> | default = 1h] 786 787 # The write buffer size used by the head chunks mapper. Lower values reduce 788 # memory utilisation on clusters with a large number of tenants at the cost 789 # of increased disk I/O operations. 790 # CLI flag: -blocks-storage.tsdb.head-chunks-write-buffer-size-bytes 791 [head_chunks_write_buffer_size_bytes: <int> | default = 4194304] 792 793 # The number of shards of series to use in TSDB (must be a power of 2). 794 # Reducing this will decrease memory footprint, but can negatively impact 795 # performance. 796 # CLI flag: -blocks-storage.tsdb.stripe-size 797 [stripe_size: <int> | default = 16384] 798 799 # True to enable TSDB WAL compression. 800 # CLI flag: -blocks-storage.tsdb.wal-compression-enabled 801 [wal_compression_enabled: <boolean> | default = false] 802 803 # TSDB WAL segments files max size (bytes). 804 # CLI flag: -blocks-storage.tsdb.wal-segment-size-bytes 805 [wal_segment_size_bytes: <int> | default = 134217728] 806 807 # True to flush blocks to storage on shutdown. If false, incomplete blocks 808 # will be reused after restart. 809 # CLI flag: -blocks-storage.tsdb.flush-blocks-on-shutdown 810 [flush_blocks_on_shutdown: <boolean> | default = false] 811 812 # If TSDB has not received any data for this duration, and all blocks from 813 # TSDB have been shipped, TSDB is closed and deleted from local disk. If set 814 # to positive value, this value should be equal or higher than 815 # -querier.query-ingesters-within flag to make sure that TSDB is not closed 816 # prematurely, which could cause partial query results. 0 or negative value 817 # disables closing of idle TSDB. 818 # CLI flag: -blocks-storage.tsdb.close-idle-tsdb-timeout 819 [close_idle_tsdb_timeout: <duration> | default = 0s] 820 821 # limit the number of concurrently opening TSDB's on startup 822 # CLI flag: -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup 823 [max_tsdb_opening_concurrency_on_startup: <int> | default = 10] 824 825 # Enables support for exemplars in TSDB and sets the maximum number that 826 # will be stored. 0 or less means disabled. 827 # CLI flag: -blocks-storage.tsdb.max-exemplars 828 [max_exemplars: <int> | default = 0] 829 ```