github.com/muhammadn/cortex@v1.9.1-0.20220510110439-46bb7000d03d/docs/blocks-storage/store-gateway.md (about)

     1  ---
     2  title: "Store-gateway"
     3  linkTitle: "Store-gateway"
     4  weight: 3
     5  slug: store-gateway
     6  ---
     7  
     8  <!-- DO NOT EDIT THIS FILE - This file has been automatically generated from its .template -->
     9  
    10  The store-gateway is the Cortex service responsible to query series from blocks. The store-gateway is required when running the Cortex blocks storage.
    11  
    12  The store-gateway is **semi-stateful**.
    13  
    14  ## How it works
    15  
    16  The store-gateway needs to have an almost up-to-date view over the storage bucket, in order to discover blocks belonging to their shard. The store-gateway can keep the bucket view updated in to two different ways:
    17  
    18  1. Periodically scanning the bucket (default)
    19  2. Periodically downloading the [bucket index](./bucket-index.md)
    20  
    21  ### Bucket index disabled (default)
    22  
    23  At startup **store-gateways** iterate over the entire storage bucket to discover blocks for all tenants and download the `meta.json` and index-header for each block. During this initial bucket synchronization phase, the store-gateway `/ready` readiness probe endpoint will fail.
    24  
    25  While running, store-gateways periodically rescan the storage bucket to discover new blocks (uploaded by the ingesters and [compactor](./compactor.md)) and blocks marked for deletion or fully deleted since the last scan (as a result of compaction). The frequency at which this occurs is configured via `-blocks-storage.bucket-store.sync-interval`.
    26  
    27  The blocks chunks and the entire index are never fully downloaded by the store-gateway. The index-header is stored to the local disk, in order to avoid to re-download it on subsequent restarts of a store-gateway. For this reason, it's recommended - but not required - to run the store-gateway with a persistent disk. For example, if you're running the Cortex cluster in Kubernetes, you may use a StatefulSet with a persistent volume claim for the store-gateways.
    28  
    29  _For more information about the index-header, please refer to [Binary index-header documentation](./binary-index-header.md)._
    30  
    31  ### Bucket index enabled
    32  
    33  When bucket index is enabled, the overall workflow is the same but, instead of iterating over the bucket objects, the store-gateway fetch the [bucket index](./bucket-index.md) for each tenant belonging to their shard in order to discover each tenant's blocks and block deletion marks.
    34  
    35  _For more information about the bucket index, please refer to [bucket index documentation](./bucket-index.md)._
    36  
    37  ## Blocks sharding and replication
    38  
    39  The store-gateway optionally supports blocks sharding. Sharding can be used to horizontally scale blocks in a large cluster without hitting any vertical scalability limit.
    40  
    41  When sharding is enabled, store-gateway instances builds an [hash ring](../architecture.md#the-hash-ring) and blocks get sharded and replicated across the pool of store-gateway instances registered within the ring.
    42  
    43  Store-gateways continuously monitor the ring state and whenever the ring topology changes (e.g. a new instance has been added/removed or gets healthy/unhealthy) each store-gateway instance resync the blocks assigned to its shard, based on the block ID hash matching the token ranges assigned to the instance itself within the ring.
    44  
    45  For each block belonging to a store-gateway shard, the store-gateway loads its `meta.json`, the `deletion-mark.json` and the index-header. Once a block is loaded on the store-gateway, it's ready to be queried by queriers. When the querier queries blocks through a store-gateway, the response will contain the list of actually queried block IDs. If a querier tries to query a block which has not been loaded by a store-gateway, the querier will either retry on a different store-gateway (if blocks replication is enabled) or fail the query.
    46  
    47  Blocks can be replicated across multiple store-gateway instances based on a replication factor configured via `-store-gateway.sharding-ring.replication-factor`. The blocks replication is used to protect from query failures caused by some blocks not loaded by any store-gateway instance at a given time like, for example, in the event of a store-gateway failure or while restarting a store-gateway instance (e.g. during a rolling update).
    48  
    49  This feature can be enabled via `-store-gateway.sharding-enabled=true` and requires the backend [hash ring](../architecture.md#the-hash-ring) to be configured via `-store-gateway.sharding-ring.*` flags (or their respective YAML config options).
    50  
    51  ### Sharding strategies
    52  
    53  The store-gateway supports two sharding strategies:
    54  
    55  - `default`
    56  - `shuffle-sharding`
    57  
    58  The **`default`** sharding strategy spreads the blocks of each tenant across all store-gateway instances. It's the easiest form of sharding supported, but doesn't provide any workload isolation between different tenants.
    59  
    60  The **`shuffle-sharding`** strategy spreads the blocks of a tenant across a subset of store-gateway instances. This way, the number of store-gateway instances loading blocks of a single tenant is limited and the blast radius of any issue that could be introduced by the tenant's workload is limited to its shard instances.
    61  
    62  The shuffle sharding strategy can be enabled via `-store-gateway.sharding-strategy=shuffle-sharding` and requires the `-store-gateway.tenant-shard-size` flag (or their respective YAML config options) to be set to the default shard size, which is the default number of store-gateway instances each tenant should be sharded to. The shard size can then be overridden on a per-tenant basis setting the `store_gateway_tenant_shard_size` in the limits overrides.
    63  
    64  _Please check out the [shuffle sharding documentation](../guides/shuffle-sharding.md) for more information about how it works._
    65  
    66  ### Auto-forget
    67  
    68  When a store-gateway instance cleanly shutdowns, it automatically unregisters itself from the ring. However, in the event of a crash or node failure, the instance will not be unregistered from the ring, potentially leaving a spurious entry in the ring forever.
    69  
    70  To protect from this, when an healthy store-gateway instance finds another instance in the ring which is unhealthy for more than 10 times the configured `-store-gateway.sharding-ring.heartbeat-timeout`, the healthy instance forcibly removes the unhealthy one from the ring.
    71  
    72  This feature is called **auto-forget** and is built into the store-gateway.
    73  
    74  ### Zone-awareness
    75  
    76  The store-gateway replication optionally supports [zone-awareness](../guides/zone-replication.md). When zone-aware replication is enabled and the blocks replication factor is > 1, each block is guaranteed to be replicated across store-gateway instances running in different availability zones.
    77  
    78  **To enable** the zone-aware replication for the store-gateways you should:
    79  
    80  1. Configure the availability zone for each store-gateway via the `-store-gateway.sharding-ring.instance-availability-zone` CLI flag (or its respective YAML config option)
    81  2. Enable blocks zone-aware replication via the `-store-gateway.sharding-ring.zone-awareness-enabled` CLI flag (or its respective YAML config option). Please be aware this configuration option should be set to store-gateways, queriers and rulers.
    82  3. Rollout store-gateways, queriers and rulers to apply the new configuration
    83  
    84  ### Waiting for stable ring at startup
    85  
    86  In the event of a cluster cold start or scale up of 2+ store-gateway instances at the same time we may end up in a situation where each new store-gateway instance starts at a slightly different time and thus each one runs the initial blocks sync based on a different state of the ring. For example, in case of a cold start, the first store-gateway joining the ring may load all blocks since the sharding logic runs based on the current state of the ring, which is 1 single store-gateway.
    87  
    88  To reduce the likelihood this could happen, the store-gateway waits for a stable ring at startup. A ring is considered stable if no instance is added/removed to the ring for at least `-store-gateway.sharding-ring.wait-stability-min-duration`. If the ring keep getting changed after `-store-gateway.sharding-ring.wait-stability-max-duration`, the store-gateway will stop waiting for a stable ring and will proceed starting up normally.
    89  
    90  To disable this waiting logic, you can start the store-gateway with `-store-gateway.sharding-ring.wait-stability-min-duration=0`.
    91  
    92  ## Blocks index-header
    93  
    94  The [index-header](./binary-index-header.md) is a subset of the block index which the store-gateway downloads from the object storage and keeps on the local disk in order to speed up queries.
    95  
    96  At startup, the store-gateway downloads the index-header of each block belonging to its shard. A store-gateway is not ready until this initial index-header download is completed. Moreover, while running, the store-gateway periodically looks for newly uploaded blocks in the storage and downloads the index-header for the blocks belonging to its shard.
    97  
    98  ### Index-header lazy loading
    99  
   100  By default, each index-header is memory mapped by the store-gateway right after downloading it. In a cluster with a large number of blocks, each store-gateway may have a large amount of memory mapped index-headers, regardless how frequently they're used at query time.
   101  
   102  Cortex supports a configuration option `-blocks-storage.bucket-store.index-header-lazy-loading-enabled=true` to enable index-header lazy loading. When enabled, index-headers will be memory mapped only once required by a query and will be automatically released after `-blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout` time of inactivity.
   103  
   104  ## Caching
   105  
   106  The store-gateway supports the following caches:
   107  
   108  - [Index cache](#index-cache)
   109  - [Chunks cache](#chunks-cache)
   110  - [Metadata cache](#metadata-cache)
   111  
   112  Caching is optional, but **highly recommended** in a production environment. Please also check out the [production tips](./production-tips.md#caching) for more information about configuring the cache.
   113  
   114  ### Index cache
   115  
   116  The store-gateway can use a cache to speed up lookups of postings and series from TSDB blocks indexes. Two backends are supported:
   117  
   118  - `inmemory`
   119  - `memcached`
   120  
   121  #### In-memory index cache
   122  
   123  The `inmemory` index cache is **enabled by default** and its max size can be configured through the flag `-blocks-storage.bucket-store.index-cache.inmemory.max-size-bytes` (or config file). The trade-off of using the in-memory index cache is:
   124  
   125  - Pros: zero latency
   126  - Cons: increased store-gateway memory usage, not shared across multiple store-gateway replicas (when sharding is disabled or replication factor > 1)
   127  
   128  #### Memcached index cache
   129  
   130  The `memcached` index cache allows to use [Memcached](https://memcached.org/) as cache backend. This cache backend is configured using `-blocks-storage.bucket-store.index-cache.backend=memcached` and requires the Memcached server(s) addresses via `-blocks-storage.bucket-store.index-cache.memcached.addresses` (or config file). The addresses are resolved using the [DNS service provider](../configuration/arguments.md#dns-service-discovery).
   131  
   132  The trade-off of using the Memcached index cache is:
   133  
   134  - Pros: can scale beyond a single node memory (Memcached cluster), shared across multiple store-gateway instances
   135  - Cons: higher latency in the cache round trip compared to the in-memory one
   136  
   137  The Memcached client uses a jump hash algorithm to shard cached entries across a cluster of Memcached servers. For this reason, you should make sure memcached servers are **not** behind any kind of load balancer and their address is configured so that servers are added/removed to the end of the list whenever a scale up/down occurs.
   138  
   139  For example, if you're running Memcached in Kubernetes, you may:
   140  
   141  1. Deploy your Memcached cluster using a [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/)
   142  2. Create an [headless service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services) for Memcached StatefulSet
   143  3. Configure the Cortex's Memcached client address using the `dnssrvnoa+` [service discovery](../configuration/arguments.md#dns-service-discovery)
   144  
   145  ### Chunks cache
   146  
   147  Store-gateway can also use a cache for storing chunks fetched from the storage. Chunks contain actual samples, and can be reused if user query hits the same series for the same time range.
   148  
   149  To enable chunks cache, please set `-blocks-storage.bucket-store.chunks-cache.backend`. Chunks can currently only be stored into Memcached cache. Memcached client can be configured via flags with `-blocks-storage.bucket-store.chunks-cache.memcached.*` prefix.
   150  
   151  There are additional low-level options for configuring chunks cache. Please refer to other flags with `-blocks-storage.bucket-store.chunks-cache.*` prefix.
   152  
   153  ### Metadata cache
   154  
   155  Store-gateway and [querier](./querier.md) can use memcached for caching bucket metadata:
   156  
   157  - List of tenants
   158  - List of blocks per tenant
   159  - Block's `meta.json` content
   160  - Block's `deletion-mark.json` existence and content
   161  - Tenant's `bucket-index.json.gz` content
   162  
   163  Using the metadata cache can significantly reduce the number of API calls to object storage and protects from linearly scale the number of these API calls with the number of querier and store-gateway instances (because the bucket is periodically scanned and synched by each querier and store-gateway).
   164  
   165  To enable metadata cache, please set `-blocks-storage.bucket-store.metadata-cache.backend`. Only `memcached` backend is supported currently. Memcached client has additional configuration available via flags with `-blocks-storage.bucket-store.metadata-cache.memcached.*` prefix.
   166  
   167  Additional options for configuring metadata cache have `-blocks-storage.bucket-store.metadata-cache.*` prefix. By configuring TTL to zero or negative value, caching of given item type is disabled.
   168  
   169  _The same memcached backend cluster should be shared between store-gateways and queriers._
   170  
   171  ## Store-gateway HTTP endpoints
   172  
   173  - `GET /store-gateway/ring`<br />
   174    Displays the status of the store-gateways ring, including the tokens owned by each store-gateway and an option to remove (forget) instances from the ring.
   175  
   176  ## Store-gateway configuration
   177  
   178  This section described the store-gateway configuration. For the general Cortex configuration and references to common config blocks, please refer to the [configuration documentation](../configuration/config-file-reference.md).
   179  
   180  ### `store_gateway_config`
   181  
   182  The `store_gateway_config` configures the store-gateway service used by the blocks storage.
   183  
   184  ```yaml
   185  store_gateway:
   186    # Shard blocks across multiple store gateway instances. This option needs be
   187    # set both on the store-gateway and querier when running in microservices
   188    # mode.
   189    # CLI flag: -store-gateway.sharding-enabled
   190    [sharding_enabled: <boolean> | default = false]
   191  
   192    # The hash ring configuration. This option is required only if blocks sharding
   193    # is enabled.
   194    sharding_ring:
   195      # The key-value store used to share the hash ring across multiple instances.
   196      # This option needs be set both on the store-gateway and querier when
   197      # running in microservices mode.
   198      kvstore:
   199        # Backend storage to use for the ring. Supported values are: consul, etcd,
   200        # inmemory, memberlist, multi.
   201        # CLI flag: -store-gateway.sharding-ring.store
   202        [store: <string> | default = "consul"]
   203  
   204        # The prefix for the keys in the store. Should end with a /.
   205        # CLI flag: -store-gateway.sharding-ring.prefix
   206        [prefix: <string> | default = "collectors/"]
   207  
   208        # The consul_config configures the consul client.
   209        # The CLI flags prefix for this block config is:
   210        # store-gateway.sharding-ring
   211        [consul: <consul_config>]
   212  
   213        # The etcd_config configures the etcd client.
   214        # The CLI flags prefix for this block config is:
   215        # store-gateway.sharding-ring
   216        [etcd: <etcd_config>]
   217  
   218        multi:
   219          # Primary backend storage used by multi-client.
   220          # CLI flag: -store-gateway.sharding-ring.multi.primary
   221          [primary: <string> | default = ""]
   222  
   223          # Secondary backend storage used by multi-client.
   224          # CLI flag: -store-gateway.sharding-ring.multi.secondary
   225          [secondary: <string> | default = ""]
   226  
   227          # Mirror writes to secondary store.
   228          # CLI flag: -store-gateway.sharding-ring.multi.mirror-enabled
   229          [mirror_enabled: <boolean> | default = false]
   230  
   231          # Timeout for storing value to secondary store.
   232          # CLI flag: -store-gateway.sharding-ring.multi.mirror-timeout
   233          [mirror_timeout: <duration> | default = 2s]
   234  
   235      # Period at which to heartbeat to the ring. 0 = disabled.
   236      # CLI flag: -store-gateway.sharding-ring.heartbeat-period
   237      [heartbeat_period: <duration> | default = 15s]
   238  
   239      # The heartbeat timeout after which store gateways are considered unhealthy
   240      # within the ring. 0 = never (timeout disabled). This option needs be set
   241      # both on the store-gateway and querier when running in microservices mode.
   242      # CLI flag: -store-gateway.sharding-ring.heartbeat-timeout
   243      [heartbeat_timeout: <duration> | default = 1m]
   244  
   245      # The replication factor to use when sharding blocks. This option needs be
   246      # set both on the store-gateway and querier when running in microservices
   247      # mode.
   248      # CLI flag: -store-gateway.sharding-ring.replication-factor
   249      [replication_factor: <int> | default = 3]
   250  
   251      # File path where tokens are stored. If empty, tokens are not stored at
   252      # shutdown and restored at startup.
   253      # CLI flag: -store-gateway.sharding-ring.tokens-file-path
   254      [tokens_file_path: <string> | default = ""]
   255  
   256      # True to enable zone-awareness and replicate blocks across different
   257      # availability zones.
   258      # CLI flag: -store-gateway.sharding-ring.zone-awareness-enabled
   259      [zone_awareness_enabled: <boolean> | default = false]
   260  
   261      # Minimum time to wait for ring stability at startup. 0 to disable.
   262      # CLI flag: -store-gateway.sharding-ring.wait-stability-min-duration
   263      [wait_stability_min_duration: <duration> | default = 1m]
   264  
   265      # Maximum time to wait for ring stability at startup. If the store-gateway
   266      # ring keeps changing after this period of time, the store-gateway will
   267      # start anyway.
   268      # CLI flag: -store-gateway.sharding-ring.wait-stability-max-duration
   269      [wait_stability_max_duration: <duration> | default = 5m]
   270  
   271      # Name of network interface to read address from.
   272      # CLI flag: -store-gateway.sharding-ring.instance-interface-names
   273      [instance_interface_names: <list of string> | default = [eth0 en0]]
   274  
   275      # The availability zone where this instance is running. Required if
   276      # zone-awareness is enabled.
   277      # CLI flag: -store-gateway.sharding-ring.instance-availability-zone
   278      [instance_availability_zone: <string> | default = ""]
   279  
   280    # The sharding strategy to use. Supported values are: default,
   281    # shuffle-sharding.
   282    # CLI flag: -store-gateway.sharding-strategy
   283    [sharding_strategy: <string> | default = "default"]
   284  ```
   285  
   286  ### `blocks_storage_config`
   287  
   288  The `blocks_storage_config` configures the blocks storage.
   289  
   290  ```yaml
   291  blocks_storage:
   292    # Backend storage to use. Supported backends are: s3, gcs, azure, swift,
   293    # filesystem.
   294    # CLI flag: -blocks-storage.backend
   295    [backend: <string> | default = "s3"]
   296  
   297    s3:
   298      # The S3 bucket endpoint. It could be an AWS S3 endpoint listed at
   299      # https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an
   300      # S3-compatible service in hostname:port format.
   301      # CLI flag: -blocks-storage.s3.endpoint
   302      [endpoint: <string> | default = ""]
   303  
   304      # S3 region. If unset, the client will issue a S3 GetBucketLocation API call
   305      # to autodetect it.
   306      # CLI flag: -blocks-storage.s3.region
   307      [region: <string> | default = ""]
   308  
   309      # S3 bucket name
   310      # CLI flag: -blocks-storage.s3.bucket-name
   311      [bucket_name: <string> | default = ""]
   312  
   313      # S3 secret access key
   314      # CLI flag: -blocks-storage.s3.secret-access-key
   315      [secret_access_key: <string> | default = ""]
   316  
   317      # S3 access key ID
   318      # CLI flag: -blocks-storage.s3.access-key-id
   319      [access_key_id: <string> | default = ""]
   320  
   321      # If enabled, use http:// for the S3 endpoint instead of https://. This
   322      # could be useful in local dev/test environments while using an
   323      # S3-compatible backend storage, like Minio.
   324      # CLI flag: -blocks-storage.s3.insecure
   325      [insecure: <boolean> | default = false]
   326  
   327      # The signature version to use for authenticating against S3. Supported
   328      # values are: v4, v2.
   329      # CLI flag: -blocks-storage.s3.signature-version
   330      [signature_version: <string> | default = "v4"]
   331  
   332      # The s3_sse_config configures the S3 server-side encryption.
   333      # The CLI flags prefix for this block config is: blocks-storage
   334      [sse: <s3_sse_config>]
   335  
   336      http:
   337        # The time an idle connection will remain idle before closing.
   338        # CLI flag: -blocks-storage.s3.http.idle-conn-timeout
   339        [idle_conn_timeout: <duration> | default = 1m30s]
   340  
   341        # The amount of time the client will wait for a servers response headers.
   342        # CLI flag: -blocks-storage.s3.http.response-header-timeout
   343        [response_header_timeout: <duration> | default = 2m]
   344  
   345        # If the client connects to S3 via HTTPS and this option is enabled, the
   346        # client will accept any certificate and hostname.
   347        # CLI flag: -blocks-storage.s3.http.insecure-skip-verify
   348        [insecure_skip_verify: <boolean> | default = false]
   349  
   350        # Maximum time to wait for a TLS handshake. 0 means no limit.
   351        # CLI flag: -blocks-storage.s3.tls-handshake-timeout
   352        [tls_handshake_timeout: <duration> | default = 10s]
   353  
   354        # The time to wait for a server's first response headers after fully
   355        # writing the request headers if the request has an Expect header. 0 to
   356        # send the request body immediately.
   357        # CLI flag: -blocks-storage.s3.expect-continue-timeout
   358        [expect_continue_timeout: <duration> | default = 1s]
   359  
   360        # Maximum number of idle (keep-alive) connections across all hosts. 0
   361        # means no limit.
   362        # CLI flag: -blocks-storage.s3.max-idle-connections
   363        [max_idle_connections: <int> | default = 100]
   364  
   365        # Maximum number of idle (keep-alive) connections to keep per-host. If 0,
   366        # a built-in default value is used.
   367        # CLI flag: -blocks-storage.s3.max-idle-connections-per-host
   368        [max_idle_connections_per_host: <int> | default = 100]
   369  
   370        # Maximum number of connections per host. 0 means no limit.
   371        # CLI flag: -blocks-storage.s3.max-connections-per-host
   372        [max_connections_per_host: <int> | default = 0]
   373  
   374    gcs:
   375      # GCS bucket name
   376      # CLI flag: -blocks-storage.gcs.bucket-name
   377      [bucket_name: <string> | default = ""]
   378  
   379      # JSON representing either a Google Developers Console
   380      # client_credentials.json file or a Google Developers service account key
   381      # file. If empty, fallback to Google default logic.
   382      # CLI flag: -blocks-storage.gcs.service-account
   383      [service_account: <string> | default = ""]
   384  
   385    azure:
   386      # Azure storage account name
   387      # CLI flag: -blocks-storage.azure.account-name
   388      [account_name: <string> | default = ""]
   389  
   390      # Azure storage account key
   391      # CLI flag: -blocks-storage.azure.account-key
   392      [account_key: <string> | default = ""]
   393  
   394      # Azure storage container name
   395      # CLI flag: -blocks-storage.azure.container-name
   396      [container_name: <string> | default = ""]
   397  
   398      # Azure storage endpoint suffix without schema. The account name will be
   399      # prefixed to this value to create the FQDN
   400      # CLI flag: -blocks-storage.azure.endpoint-suffix
   401      [endpoint_suffix: <string> | default = ""]
   402  
   403      # Number of retries for recoverable errors
   404      # CLI flag: -blocks-storage.azure.max-retries
   405      [max_retries: <int> | default = 20]
   406  
   407    swift:
   408      # OpenStack Swift authentication API version. 0 to autodetect.
   409      # CLI flag: -blocks-storage.swift.auth-version
   410      [auth_version: <int> | default = 0]
   411  
   412      # OpenStack Swift authentication URL
   413      # CLI flag: -blocks-storage.swift.auth-url
   414      [auth_url: <string> | default = ""]
   415  
   416      # OpenStack Swift username.
   417      # CLI flag: -blocks-storage.swift.username
   418      [username: <string> | default = ""]
   419  
   420      # OpenStack Swift user's domain name.
   421      # CLI flag: -blocks-storage.swift.user-domain-name
   422      [user_domain_name: <string> | default = ""]
   423  
   424      # OpenStack Swift user's domain ID.
   425      # CLI flag: -blocks-storage.swift.user-domain-id
   426      [user_domain_id: <string> | default = ""]
   427  
   428      # OpenStack Swift user ID.
   429      # CLI flag: -blocks-storage.swift.user-id
   430      [user_id: <string> | default = ""]
   431  
   432      # OpenStack Swift API key.
   433      # CLI flag: -blocks-storage.swift.password
   434      [password: <string> | default = ""]
   435  
   436      # OpenStack Swift user's domain ID.
   437      # CLI flag: -blocks-storage.swift.domain-id
   438      [domain_id: <string> | default = ""]
   439  
   440      # OpenStack Swift user's domain name.
   441      # CLI flag: -blocks-storage.swift.domain-name
   442      [domain_name: <string> | default = ""]
   443  
   444      # OpenStack Swift project ID (v2,v3 auth only).
   445      # CLI flag: -blocks-storage.swift.project-id
   446      [project_id: <string> | default = ""]
   447  
   448      # OpenStack Swift project name (v2,v3 auth only).
   449      # CLI flag: -blocks-storage.swift.project-name
   450      [project_name: <string> | default = ""]
   451  
   452      # ID of the OpenStack Swift project's domain (v3 auth only), only needed if
   453      # it differs the from user domain.
   454      # CLI flag: -blocks-storage.swift.project-domain-id
   455      [project_domain_id: <string> | default = ""]
   456  
   457      # Name of the OpenStack Swift project's domain (v3 auth only), only needed
   458      # if it differs from the user domain.
   459      # CLI flag: -blocks-storage.swift.project-domain-name
   460      [project_domain_name: <string> | default = ""]
   461  
   462      # OpenStack Swift Region to use (v2,v3 auth only).
   463      # CLI flag: -blocks-storage.swift.region-name
   464      [region_name: <string> | default = ""]
   465  
   466      # Name of the OpenStack Swift container to put chunks in.
   467      # CLI flag: -blocks-storage.swift.container-name
   468      [container_name: <string> | default = ""]
   469  
   470      # Max retries on requests error.
   471      # CLI flag: -blocks-storage.swift.max-retries
   472      [max_retries: <int> | default = 3]
   473  
   474      # Time after which a connection attempt is aborted.
   475      # CLI flag: -blocks-storage.swift.connect-timeout
   476      [connect_timeout: <duration> | default = 10s]
   477  
   478      # Time after which an idle request is aborted. The timeout watchdog is reset
   479      # each time some data is received, so the timeout triggers after X time no
   480      # data is received on a request.
   481      # CLI flag: -blocks-storage.swift.request-timeout
   482      [request_timeout: <duration> | default = 5s]
   483  
   484    filesystem:
   485      # Local filesystem storage directory.
   486      # CLI flag: -blocks-storage.filesystem.dir
   487      [dir: <string> | default = ""]
   488  
   489    # This configures how the querier and store-gateway discover and synchronize
   490    # blocks stored in the bucket.
   491    bucket_store:
   492      # Directory to store synchronized TSDB index headers.
   493      # CLI flag: -blocks-storage.bucket-store.sync-dir
   494      [sync_dir: <string> | default = "tsdb-sync"]
   495  
   496      # How frequently to scan the bucket, or to refresh the bucket index (if
   497      # enabled), in order to look for changes (new blocks shipped by ingesters
   498      # and blocks deleted by retention or compaction).
   499      # CLI flag: -blocks-storage.bucket-store.sync-interval
   500      [sync_interval: <duration> | default = 15m]
   501  
   502      # Max number of concurrent queries to execute against the long-term storage.
   503      # The limit is shared across all tenants.
   504      # CLI flag: -blocks-storage.bucket-store.max-concurrent
   505      [max_concurrent: <int> | default = 100]
   506  
   507      # Maximum number of concurrent tenants synching blocks.
   508      # CLI flag: -blocks-storage.bucket-store.tenant-sync-concurrency
   509      [tenant_sync_concurrency: <int> | default = 10]
   510  
   511      # Maximum number of concurrent blocks synching per tenant.
   512      # CLI flag: -blocks-storage.bucket-store.block-sync-concurrency
   513      [block_sync_concurrency: <int> | default = 20]
   514  
   515      # Number of Go routines to use when syncing block meta files from object
   516      # storage per tenant.
   517      # CLI flag: -blocks-storage.bucket-store.meta-sync-concurrency
   518      [meta_sync_concurrency: <int> | default = 20]
   519  
   520      # Minimum age of a block before it's being read. Set it to safe value (e.g
   521      # 30m) if your object storage is eventually consistent. GCS and S3 are
   522      # (roughly) strongly consistent.
   523      # CLI flag: -blocks-storage.bucket-store.consistency-delay
   524      [consistency_delay: <duration> | default = 0s]
   525  
   526      index_cache:
   527        # The index cache backend type. Supported values: inmemory, memcached.
   528        # CLI flag: -blocks-storage.bucket-store.index-cache.backend
   529        [backend: <string> | default = "inmemory"]
   530  
   531        inmemory:
   532          # Maximum size in bytes of in-memory index cache used to speed up blocks
   533          # index lookups (shared between all tenants).
   534          # CLI flag: -blocks-storage.bucket-store.index-cache.inmemory.max-size-bytes
   535          [max_size_bytes: <int> | default = 1073741824]
   536  
   537        memcached:
   538          # Comma separated list of memcached addresses. Supported prefixes are:
   539          # dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV
   540          # query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup
   541          # made after that).
   542          # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.addresses
   543          [addresses: <string> | default = ""]
   544  
   545          # The socket read/write timeout.
   546          # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.timeout
   547          [timeout: <duration> | default = 100ms]
   548  
   549          # The maximum number of idle connections that will be maintained per
   550          # address.
   551          # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections
   552          [max_idle_connections: <int> | default = 16]
   553  
   554          # The maximum number of concurrent asynchronous operations can occur.
   555          # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-async-concurrency
   556          [max_async_concurrency: <int> | default = 50]
   557  
   558          # The maximum number of enqueued asynchronous operations allowed.
   559          # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-async-buffer-size
   560          [max_async_buffer_size: <int> | default = 10000]
   561  
   562          # The maximum number of concurrent connections running get operations.
   563          # If set to 0, concurrency is unlimited.
   564          # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency
   565          [max_get_multi_concurrency: <int> | default = 100]
   566  
   567          # The maximum number of keys a single underlying get operation should
   568          # run. If more keys are specified, internally keys are split into
   569          # multiple batches and fetched concurrently, honoring the max
   570          # concurrency. If set to 0, the max batch size is unlimited.
   571          # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-batch-size
   572          [max_get_multi_batch_size: <int> | default = 0]
   573  
   574          # The maximum size of an item stored in memcached. Bigger items are not
   575          # stored. If set to 0, no maximum size is enforced.
   576          # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-item-size
   577          [max_item_size: <int> | default = 1048576]
   578  
   579          # Use memcached auto-discovery mechanism provided by some cloud provider
   580          # like GCP and AWS
   581          # CLI flag: -blocks-storage.bucket-store.index-cache.memcached.auto-discovery
   582          [auto_discovery: <boolean> | default = false]
   583  
   584      chunks_cache:
   585        # Backend for chunks cache, if not empty. Supported values: memcached.
   586        # CLI flag: -blocks-storage.bucket-store.chunks-cache.backend
   587        [backend: <string> | default = ""]
   588  
   589        memcached:
   590          # Comma separated list of memcached addresses. Supported prefixes are:
   591          # dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV
   592          # query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup
   593          # made after that).
   594          # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.addresses
   595          [addresses: <string> | default = ""]
   596  
   597          # The socket read/write timeout.
   598          # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.timeout
   599          [timeout: <duration> | default = 100ms]
   600  
   601          # The maximum number of idle connections that will be maintained per
   602          # address.
   603          # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-idle-connections
   604          [max_idle_connections: <int> | default = 16]
   605  
   606          # The maximum number of concurrent asynchronous operations can occur.
   607          # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-async-concurrency
   608          [max_async_concurrency: <int> | default = 50]
   609  
   610          # The maximum number of enqueued asynchronous operations allowed.
   611          # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-async-buffer-size
   612          [max_async_buffer_size: <int> | default = 10000]
   613  
   614          # The maximum number of concurrent connections running get operations.
   615          # If set to 0, concurrency is unlimited.
   616          # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-get-multi-concurrency
   617          [max_get_multi_concurrency: <int> | default = 100]
   618  
   619          # The maximum number of keys a single underlying get operation should
   620          # run. If more keys are specified, internally keys are split into
   621          # multiple batches and fetched concurrently, honoring the max
   622          # concurrency. If set to 0, the max batch size is unlimited.
   623          # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-get-multi-batch-size
   624          [max_get_multi_batch_size: <int> | default = 0]
   625  
   626          # The maximum size of an item stored in memcached. Bigger items are not
   627          # stored. If set to 0, no maximum size is enforced.
   628          # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-item-size
   629          [max_item_size: <int> | default = 1048576]
   630  
   631          # Use memcached auto-discovery mechanism provided by some cloud provider
   632          # like GCP and AWS
   633          # CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.auto-discovery
   634          [auto_discovery: <boolean> | default = false]
   635  
   636        # Size of each subrange that bucket object is split into for better
   637        # caching.
   638        # CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-size
   639        [subrange_size: <int> | default = 16000]
   640  
   641        # Maximum number of sub-GetRange requests that a single GetRange request
   642        # can be split into when fetching chunks. Zero or negative value =
   643        # unlimited number of sub-requests.
   644        # CLI flag: -blocks-storage.bucket-store.chunks-cache.max-get-range-requests
   645        [max_get_range_requests: <int> | default = 3]
   646  
   647        # TTL for caching object attributes for chunks.
   648        # CLI flag: -blocks-storage.bucket-store.chunks-cache.attributes-ttl
   649        [attributes_ttl: <duration> | default = 168h]
   650  
   651        # TTL for caching individual chunks subranges.
   652        # CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-ttl
   653        [subrange_ttl: <duration> | default = 24h]
   654  
   655      metadata_cache:
   656        # Backend for metadata cache, if not empty. Supported values: memcached.
   657        # CLI flag: -blocks-storage.bucket-store.metadata-cache.backend
   658        [backend: <string> | default = ""]
   659  
   660        memcached:
   661          # Comma separated list of memcached addresses. Supported prefixes are:
   662          # dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV
   663          # query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup
   664          # made after that).
   665          # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.addresses
   666          [addresses: <string> | default = ""]
   667  
   668          # The socket read/write timeout.
   669          # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.timeout
   670          [timeout: <duration> | default = 100ms]
   671  
   672          # The maximum number of idle connections that will be maintained per
   673          # address.
   674          # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-idle-connections
   675          [max_idle_connections: <int> | default = 16]
   676  
   677          # The maximum number of concurrent asynchronous operations can occur.
   678          # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-async-concurrency
   679          [max_async_concurrency: <int> | default = 50]
   680  
   681          # The maximum number of enqueued asynchronous operations allowed.
   682          # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-async-buffer-size
   683          [max_async_buffer_size: <int> | default = 10000]
   684  
   685          # The maximum number of concurrent connections running get operations.
   686          # If set to 0, concurrency is unlimited.
   687          # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-get-multi-concurrency
   688          [max_get_multi_concurrency: <int> | default = 100]
   689  
   690          # The maximum number of keys a single underlying get operation should
   691          # run. If more keys are specified, internally keys are split into
   692          # multiple batches and fetched concurrently, honoring the max
   693          # concurrency. If set to 0, the max batch size is unlimited.
   694          # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-get-multi-batch-size
   695          [max_get_multi_batch_size: <int> | default = 0]
   696  
   697          # The maximum size of an item stored in memcached. Bigger items are not
   698          # stored. If set to 0, no maximum size is enforced.
   699          # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-item-size
   700          [max_item_size: <int> | default = 1048576]
   701  
   702          # Use memcached auto-discovery mechanism provided by some cloud provider
   703          # like GCP and AWS
   704          # CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.auto-discovery
   705          [auto_discovery: <boolean> | default = false]
   706  
   707        # How long to cache list of tenants in the bucket.
   708        # CLI flag: -blocks-storage.bucket-store.metadata-cache.tenants-list-ttl
   709        [tenants_list_ttl: <duration> | default = 15m]
   710  
   711        # How long to cache list of blocks for each tenant.
   712        # CLI flag: -blocks-storage.bucket-store.metadata-cache.tenant-blocks-list-ttl
   713        [tenant_blocks_list_ttl: <duration> | default = 5m]
   714  
   715        # How long to cache list of chunks for a block.
   716        # CLI flag: -blocks-storage.bucket-store.metadata-cache.chunks-list-ttl
   717        [chunks_list_ttl: <duration> | default = 24h]
   718  
   719        # How long to cache information that block metafile exists. Also used for
   720        # user deletion mark file.
   721        # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-exists-ttl
   722        [metafile_exists_ttl: <duration> | default = 2h]
   723  
   724        # How long to cache information that block metafile doesn't exist. Also
   725        # used for user deletion mark file.
   726        # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-doesnt-exist-ttl
   727        [metafile_doesnt_exist_ttl: <duration> | default = 5m]
   728  
   729        # How long to cache content of the metafile.
   730        # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-content-ttl
   731        [metafile_content_ttl: <duration> | default = 24h]
   732  
   733        # Maximum size of metafile content to cache in bytes. Caching will be
   734        # skipped if the content exceeds this size. This is useful to avoid
   735        # network round trip for large content if the configured caching backend
   736        # has an hard limit on cached items size (in this case, you should set
   737        # this limit to the same limit in the caching backend).
   738        # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-max-size-bytes
   739        [metafile_max_size_bytes: <int> | default = 1048576]
   740  
   741        # How long to cache attributes of the block metafile.
   742        # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-attributes-ttl
   743        [metafile_attributes_ttl: <duration> | default = 168h]
   744  
   745        # How long to cache attributes of the block index.
   746        # CLI flag: -blocks-storage.bucket-store.metadata-cache.block-index-attributes-ttl
   747        [block_index_attributes_ttl: <duration> | default = 168h]
   748  
   749        # How long to cache content of the bucket index.
   750        # CLI flag: -blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl
   751        [bucket_index_content_ttl: <duration> | default = 5m]
   752  
   753        # Maximum size of bucket index content to cache in bytes. Caching will be
   754        # skipped if the content exceeds this size. This is useful to avoid
   755        # network round trip for large content if the configured caching backend
   756        # has an hard limit on cached items size (in this case, you should set
   757        # this limit to the same limit in the caching backend).
   758        # CLI flag: -blocks-storage.bucket-store.metadata-cache.bucket-index-max-size-bytes
   759        [bucket_index_max_size_bytes: <int> | default = 1048576]
   760  
   761      # Duration after which the blocks marked for deletion will be filtered out
   762      # while fetching blocks. The idea of ignore-deletion-marks-delay is to
   763      # ignore blocks that are marked for deletion with some delay. This ensures
   764      # store can still serve blocks that are meant to be deleted but do not have
   765      # a replacement yet. Default is 6h, half of the default value for
   766      # -compactor.deletion-delay.
   767      # CLI flag: -blocks-storage.bucket-store.ignore-deletion-marks-delay
   768      [ignore_deletion_mark_delay: <duration> | default = 6h]
   769  
   770      bucket_index:
   771        # True to enable querier and store-gateway to discover blocks in the
   772        # storage via bucket index instead of bucket scanning.
   773        # CLI flag: -blocks-storage.bucket-store.bucket-index.enabled
   774        [enabled: <boolean> | default = false]
   775  
   776        # How frequently a bucket index, which previously failed to load, should
   777        # be tried to load again. This option is used only by querier.
   778        # CLI flag: -blocks-storage.bucket-store.bucket-index.update-on-error-interval
   779        [update_on_error_interval: <duration> | default = 1m]
   780  
   781        # How long a unused bucket index should be cached. Once this timeout
   782        # expires, the unused bucket index is removed from the in-memory cache.
   783        # This option is used only by querier.
   784        # CLI flag: -blocks-storage.bucket-store.bucket-index.idle-timeout
   785        [idle_timeout: <duration> | default = 1h]
   786  
   787        # The maximum allowed age of a bucket index (last updated) before queries
   788        # start failing because the bucket index is too old. The bucket index is
   789        # periodically updated by the compactor, while this check is enforced in
   790        # the querier (at query time).
   791        # CLI flag: -blocks-storage.bucket-store.bucket-index.max-stale-period
   792        [max_stale_period: <duration> | default = 1h]
   793  
   794      # Max size - in bytes - of a chunks pool, used to reduce memory allocations.
   795      # The pool is shared across all tenants. 0 to disable the limit.
   796      # CLI flag: -blocks-storage.bucket-store.max-chunk-pool-bytes
   797      [max_chunk_pool_bytes: <int> | default = 2147483648]
   798  
   799      # If enabled, store-gateway will lazy load an index-header only once
   800      # required by a query.
   801      # CLI flag: -blocks-storage.bucket-store.index-header-lazy-loading-enabled
   802      [index_header_lazy_loading_enabled: <boolean> | default = false]
   803  
   804      # If index-header lazy loading is enabled and this setting is > 0, the
   805      # store-gateway will offload unused index-headers after 'idle timeout'
   806      # inactivity.
   807      # CLI flag: -blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout
   808      [index_header_lazy_loading_idle_timeout: <duration> | default = 20m]
   809  
   810    tsdb:
   811      # Local directory to store TSDBs in the ingesters.
   812      # CLI flag: -blocks-storage.tsdb.dir
   813      [dir: <string> | default = "tsdb"]
   814  
   815      # TSDB blocks range period.
   816      # CLI flag: -blocks-storage.tsdb.block-ranges-period
   817      [block_ranges_period: <list of duration> | default = 2h0m0s]
   818  
   819      # TSDB blocks retention in the ingester before a block is removed. This
   820      # should be larger than the block_ranges_period and large enough to give
   821      # store-gateways and queriers enough time to discover newly uploaded blocks.
   822      # CLI flag: -blocks-storage.tsdb.retention-period
   823      [retention_period: <duration> | default = 6h]
   824  
   825      # How frequently the TSDB blocks are scanned and new ones are shipped to the
   826      # storage. 0 means shipping is disabled.
   827      # CLI flag: -blocks-storage.tsdb.ship-interval
   828      [ship_interval: <duration> | default = 1m]
   829  
   830      # Maximum number of tenants concurrently shipping blocks to the storage.
   831      # CLI flag: -blocks-storage.tsdb.ship-concurrency
   832      [ship_concurrency: <int> | default = 10]
   833  
   834      # How frequently does Cortex try to compact TSDB head. Block is only created
   835      # if data covers smallest block range. Must be greater than 0 and max 5
   836      # minutes.
   837      # CLI flag: -blocks-storage.tsdb.head-compaction-interval
   838      [head_compaction_interval: <duration> | default = 1m]
   839  
   840      # Maximum number of tenants concurrently compacting TSDB head into a new
   841      # block
   842      # CLI flag: -blocks-storage.tsdb.head-compaction-concurrency
   843      [head_compaction_concurrency: <int> | default = 5]
   844  
   845      # If TSDB head is idle for this duration, it is compacted. Note that up to
   846      # 25% jitter is added to the value to avoid ingesters compacting
   847      # concurrently. 0 means disabled.
   848      # CLI flag: -blocks-storage.tsdb.head-compaction-idle-timeout
   849      [head_compaction_idle_timeout: <duration> | default = 1h]
   850  
   851      # The write buffer size used by the head chunks mapper. Lower values reduce
   852      # memory utilisation on clusters with a large number of tenants at the cost
   853      # of increased disk I/O operations.
   854      # CLI flag: -blocks-storage.tsdb.head-chunks-write-buffer-size-bytes
   855      [head_chunks_write_buffer_size_bytes: <int> | default = 4194304]
   856  
   857      # The number of shards of series to use in TSDB (must be a power of 2).
   858      # Reducing this will decrease memory footprint, but can negatively impact
   859      # performance.
   860      # CLI flag: -blocks-storage.tsdb.stripe-size
   861      [stripe_size: <int> | default = 16384]
   862  
   863      # True to enable TSDB WAL compression.
   864      # CLI flag: -blocks-storage.tsdb.wal-compression-enabled
   865      [wal_compression_enabled: <boolean> | default = false]
   866  
   867      # TSDB WAL segments files max size (bytes).
   868      # CLI flag: -blocks-storage.tsdb.wal-segment-size-bytes
   869      [wal_segment_size_bytes: <int> | default = 134217728]
   870  
   871      # True to flush blocks to storage on shutdown. If false, incomplete blocks
   872      # will be reused after restart.
   873      # CLI flag: -blocks-storage.tsdb.flush-blocks-on-shutdown
   874      [flush_blocks_on_shutdown: <boolean> | default = false]
   875  
   876      # If TSDB has not received any data for this duration, and all blocks from
   877      # TSDB have been shipped, TSDB is closed and deleted from local disk. If set
   878      # to positive value, this value should be equal or higher than
   879      # -querier.query-ingesters-within flag to make sure that TSDB is not closed
   880      # prematurely, which could cause partial query results. 0 or negative value
   881      # disables closing of idle TSDB.
   882      # CLI flag: -blocks-storage.tsdb.close-idle-tsdb-timeout
   883      [close_idle_tsdb_timeout: <duration> | default = 0s]
   884  
   885      # limit the number of concurrently opening TSDB's on startup
   886      # CLI flag: -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup
   887      [max_tsdb_opening_concurrency_on_startup: <int> | default = 10]
   888  
   889      # Enables support for exemplars in TSDB and sets the maximum number that
   890      # will be stored. 0 or less means disabled.
   891      # CLI flag: -blocks-storage.tsdb.max-exemplars
   892      [max_exemplars: <int> | default = 0]
   893  ```