github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/docs/sources/operations/overrides-exporter.md (about)

     1  ---
     2  title: "Overrides Exporter"
     3  weight: 20
     4  ---
     5  
     6  Loki is a multi-tenant system that supports applying limits to each tenant as a mechanism for resource management. The `overrides-exporter` module exposes these limits as Prometheus metrics in order to help operators better understand tenant behavior.
     7  
     8  ## Context
     9  
    10  Configuration updates to tenant limits can be applied to Loki without restart via the [`runtime_config`](../configuration/#runtime-configuration-file) feature.
    11  
    12  ## Example
    13  
    14  The `overrides-exporter` module is disabled by default. We recommend running a single instance per cluster to avoid issues with metric cardinality. The `overrides-exporter` creates one metric for every scalar field in the limits configuration under the metric `loki_overrides_defaults` with the default value for that field after loading the Loki configuration. It also exposes another metric for _every_ differing field for _every_ tenant.
    15  
    16  Using an example `runtime.yaml`:
    17  
    18  ```yaml
    19  overrides:
    20    "tenant_1":
    21      ingestion_rate_mb: 10
    22      max_streams_per_user: 100000
    23      max_chunks_per_query: 100000
    24  ```
    25  
    26  Launch an instance of the `overrides-exporter`:
    27  
    28  ```shell
    29  loki -target=overrides-exporter -runtime-config.file=runtime.yaml -config.file=basic_schema_config.yaml -server.http-listen-port=8080
    30  ```
    31  
    32  To inspect the tenant limit overrides:
    33  
    34  ```shell
    35  $ curl -sq localhost:8080/metrics | grep override
    36  # HELP loki_overrides Resource limit overrides applied to tenants
    37  # TYPE loki_overrides gauge
    38  loki_overrides{limit_name="ingestion_rate_mb",user="tenant_1"} 10
    39  loki_overrides{limit_name="max_chunks_per_query",user="tenant_1"} 100000
    40  loki_overrides{limit_name="max_streams_per_user",user="tenant_1"} 100000
    41  # HELP loki_overrides_defaults Default values for resource limit overrides applied to tenants
    42  # TYPE loki_overrides_defaults gauge
    43  loki_overrides_defaults{limit_name="cardinality_limit"} 100000
    44  loki_overrides_defaults{limit_name="creation_grace_period"} 6e+11
    45  loki_overrides_defaults{limit_name="ingestion_burst_size_mb"} 6
    46  loki_overrides_defaults{limit_name="ingestion_rate_mb"} 4
    47  loki_overrides_defaults{limit_name="max_cache_freshness_per_query"} 6e+10
    48  loki_overrides_defaults{limit_name="max_chunks_per_query"} 2e+06
    49  loki_overrides_defaults{limit_name="max_concurrent_tail_requests"} 10
    50  loki_overrides_defaults{limit_name="max_entries_limit_per_query"} 5000
    51  loki_overrides_defaults{limit_name="max_global_streams_per_user"} 5000
    52  loki_overrides_defaults{limit_name="max_label_name_length"} 1024
    53  loki_overrides_defaults{limit_name="max_label_names_per_series"} 30
    54  loki_overrides_defaults{limit_name="max_label_value_length"} 2048
    55  loki_overrides_defaults{limit_name="max_line_size"} 0
    56  loki_overrides_defaults{limit_name="max_queriers_per_tenant"} 0
    57  loki_overrides_defaults{limit_name="max_query_length"} 2.5956e+15
    58  loki_overrides_defaults{limit_name="max_query_lookback"} 0
    59  loki_overrides_defaults{limit_name="max_query_parallelism"} 32
    60  loki_overrides_defaults{limit_name="max_query_series"} 500
    61  loki_overrides_defaults{limit_name="max_streams_matchers_per_query"} 1000
    62  loki_overrides_defaults{limit_name="max_streams_per_user"} 0
    63  loki_overrides_defaults{limit_name="min_sharding_lookback"} 0
    64  loki_overrides_defaults{limit_name="per_stream_rate_limit"} 3.145728e+06
    65  loki_overrides_defaults{limit_name="per_stream_rate_limit_burst"} 1.572864e+07
    66  loki_overrides_defaults{limit_name="per_tenant_override_period"} 1e+10
    67  loki_overrides_defaults{limit_name="reject_old_samples_max_age"} 1.2096e+15
    68  loki_overrides_defaults{limit_name="retention_period"} 2.6784e+15
    69  loki_overrides_defaults{limit_name="ruler_evaluation_delay_duration"} 0
    70  loki_overrides_defaults{limit_name="ruler_max_rule_groups_per_tenant"} 0
    71  loki_overrides_defaults{limit_name="ruler_max_rules_per_rule_group"} 0
    72  loki_overrides_defaults{limit_name="ruler_remote_write_queue_batch_send_deadline"} 0
    73  loki_overrides_defaults{limit_name="ruler_remote_write_queue_capacity"} 0
    74  loki_overrides_defaults{limit_name="ruler_remote_write_queue_max_backoff"} 0
    75  loki_overrides_defaults{limit_name="ruler_remote_write_queue_max_samples_per_send"} 0
    76  loki_overrides_defaults{limit_name="ruler_remote_write_queue_max_shards"} 0
    77  loki_overrides_defaults{limit_name="ruler_remote_write_queue_min_backoff"} 0
    78  loki_overrides_defaults{limit_name="ruler_remote_write_queue_min_shards"} 0
    79  loki_overrides_defaults{limit_name="ruler_remote_write_timeout"} 0
    80  loki_overrides_defaults{limit_name="split_queries_by_interval"} 0
    81  ```
    82  
    83  Alerts can be created based on these metrics to inform operators when tenants are close to hitting their limits allowing for increases to be applied before the tenant limits are exceeded.