github.com/muhammadn/cortex@v1.9.1-0.20220510110439-46bb7000d03d/docs/guides/overrides-exporter.md (about)

     1  ---
     2  title: "Overrides Exporter"
     3  linkTitle: "Overrides Exporter"
     4  weight: 10
     5  slug: overrides-exporter
     6  ---
     7  
     8  Since Cortex is a multi-tenant system, it supports applying limits to each tenant to prevent
     9  any single one from using too many resources. In order to help operators understand how close
    10  to their limits tenants are, the `overrides-exporter` module can expose limits as Prometheus metrics.
    11  
    12  ## Context
    13  
    14  To update configuration without restarting, Cortex allows operators to supply a `runtime_config`
    15  file that will be periodically reloaded. This file can be specified under the `runtime_config` section
    16  of the main [configuration file](../configuration/arguments.md#runtime-configuration-file) or using the `-runtime-config.file`
    17  command line flag. This file is used to apply tenant-specific limits.
    18  
    19  ## Example
    20  
    21  The `overrides-exporter` is not enabled by default, it must be explicitly enabled. We recommend
    22  only running a single instance of it in your cluster due to the cardinality of the metrics
    23  emitted.
    24  
    25  With a `runtime.yaml` file given below
    26  
    27  [embedmd]:# (./overrides-exporter-runtime.yaml)
    28  ```yaml
    29  # file: runtime.yaml
    30  # In this example, we're overriding ingestion limits for a single tenant.
    31  overrides:
    32    "user1":
    33      ingestion_burst_size: 350000
    34      ingestion_rate: 350000
    35      max_global_series_per_metric: 300000
    36      max_global_series_per_user: 300000
    37      max_series_per_metric: 0
    38      max_series_per_user: 0
    39      max_samples_per_query: 100000
    40      max_series_per_query: 100000
    41  ```
    42  
    43  The `overrides-exporter` is configured to run as follows
    44  
    45  ```
    46  cortex -target overrides-exporter -runtime-config.file runtime.yaml -server.http-listen-port=8080
    47  ```
    48  
    49  After the `overrides-exporter` starts, you can to use `curl` to inspect the tenant overrides.
    50  
    51  ```text
    52  curl -s http://localhost:8080/metrics | grep cortex_overrides
    53  # HELP cortex_overrides Resource limit overrides applied to tenants
    54  # TYPE cortex_overrides gauge
    55  cortex_overrides{limit_name="ingestion_burst_size",user="user1"} 350000
    56  cortex_overrides{limit_name="ingestion_rate",user="user1"} 350000
    57  cortex_overrides{limit_name="max_global_series_per_metric",user="user1"} 300000
    58  cortex_overrides{limit_name="max_global_series_per_user",user="user1"} 300000
    59  cortex_overrides{limit_name="max_local_series_per_metric",user="user1"} 0
    60  cortex_overrides{limit_name="max_local_series_per_user",user="user1"} 0
    61  cortex_overrides{limit_name="max_samples_per_query",user="user1"} 100000
    62  cortex_overrides{limit_name="max_series_per_query",user="user1"} 100000
    63  ```
    64  
    65  With these metrics, you can set up alerts to know when tenants are close to hitting their limits
    66  before they exceed them.