github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/docs/sources/operations/recording-rules.md (about)

     1  ---
     2  title: Recording Rules
     3  ---
     4  
     5  # Recording Rules
     6  
     7  Recording rules are evaluated by the `ruler` component. Each `ruler` acts as its own `querier`, in the sense that it
     8  executes queries against the store without using the `query-frontend` or `querier` components. It will respect all query
     9  [limits](https://grafana.com/docs/loki/latest/configuration/#limits_config) put in place for the `querier`.
    10  
    11  Loki's implementation of recording rules largely reuses Prometheus' code.
    12  
    13  Samples generated by recording rules are sent to Prometheus using Prometheus' **remote-write** feature.
    14  
    15  ## Write-Ahead Log (WAL)
    16  
    17  All samples generated by recording rules are written to a WAL. The WAL's main benefit is that it persists the samples
    18  generated by recording rules to disk, which means that if your `ruler` crashes, you won't lose any data.
    19  We are trading off extra memory usage and slower start-up times for this functionality.
    20  
    21  A WAL is created per tenant; this is done to prevent cross-tenant interactions. If all samples were to be written
    22  to a single WAL, this would increase the chances that one tenant could cause data-loss for others. A typical scenario here
    23  is that Prometheus will, for example, reject a remote-write request with 100 samples if just 1 of those samples is invalid in some way.
    24  
    25  ### Start-up
    26  
    27  When the `ruler` starts up, it will load the WALs for the tenants who have recording rules. These WAL files are stored
    28  on disk and are loaded into memory.
    29  
    30  Note: WALs are loaded one at a time upon start-up. This is a current limitation of the Loki ruler.
    31  For this reason, it is adviseable that the number of rule groups serviced by a ruler be kept to a reasonable size, since
    32  _no rule evaluation occurs while WAL replay is in progress (this includes alerting rules)_.
    33  
    34  ### Truncation
    35  
    36  WAL files are regularly truncated to reduce their size on disk.
    37  [This guide](https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/#wal-truncation-and-checkpointing)
    38  from one of the Prometheus maintainers (Ganesh Vernekar) gives an excellent overview of the truncation, checkpointing,
    39  and replaying of the WAL.
    40  
    41  ### Cleaner
    42  
    43  <span style="background-color:#f3f973;">WAL Cleaner is an experimental feature.</span>
    44  
    45  The WAL Cleaner watches for abandoned WALs (tenants who no longer have recording rules associated) and deletes them.
    46  Enable this feature only if you are running into storage concerns with WALs that are too large. WALs should not grow
    47  excessively large due to truncation.
    48  
    49  ## Scaling
    50  
    51  See Mimir's guide for [configuring Grafana Mimir hash rings](https://grafana.com/docs/mimir/latest/operators-guide/configuring/configuring-hash-rings/) for scaling the ruler using a ring.
    52  
    53  Note: the `ruler` shards by rule _group_, not by individual rules. This is an artifact of the fact that Prometheus
    54  recording rules need to run in order since one recording rule can reuse another - but this is not possible in Loki.
    55  
    56  ## Deployment
    57  
    58  The `ruler` needs to persist its WAL files to disk, and it incurs a bit of a start-up cost by reading these WALs into memory.
    59  As such, it is recommended that you try to minimize churn of individual `ruler` instances since rule evaluation is blocked
    60  while the WALs are being read from disk.
    61  
    62  ### Kubernetes
    63  
    64  It is recommended that you run the `rulers` using `StatefulSets`. The `ruler` will write its WAL files to persistent storage,
    65  so a `Persistent Volume` should be utilised.
    66  
    67  ## Remote-Write
    68  
    69  ### Per-Tenant Limits
    70  
    71  Remote-write can be configured at a global level in the base configuration, and certain parameters tuned specifically on
    72  a per-tenant basis. Most of the configuration options [defined here](../../configuration/#ruler)
    73  have [override options](../../configuration/#limits_config) (which can be also applied at runtime!).
    74  
    75  ### Tuning
    76  
    77  Remote-write can be tuned if the default configuration is insufficient (see [Failure Modes](#failure-modes) below).
    78  
    79  There is a [guide](https://prometheus.io/docs/practices/remote_write/) on the Prometheus website, all of which applies to Loki, too.
    80  
    81  ## Observability
    82  
    83  Since Loki reuses the Prometheus code for recording rules and WALs, it also gains all of Prometheus' observability.
    84  
    85  Prometheus exposes a number of metrics for its WAL implementation, and these have all been prefixed with `loki_ruler_wal_`.
    86  
    87  For example: `prometheus_remote_storage_bytes_total` → `loki_ruler_wal_prometheus_remote_storage_bytes_total`
    88  
    89  Additional metrics are exposed, also with the prefix `loki_ruler_wal_`. All per-tenant metrics contain a `tenant`
    90  label, so be aware that cardinality could begin to be a concern if the number of tenants grows sufficiently large.
    91  
    92  Some key metrics to note are:
    93  - `loki_ruler_wal_appender_ready`: whether a WAL appender is ready to accept samples (1) or not (0)
    94  - `loki_ruler_wal_prometheus_remote_storage_samples_total`: number of samples sent per tenant to remote storage
    95  - `loki_ruler_wal_prometheus_remote_storage_samples...`
    96    - `loki_ruler_wal_prometheus_remote_storage_samples_pending_total`: samples buffered in memory, waiting to be sent to remote storage
    97    - `loki_ruler_wal_prometheus_remote_storage_samples_failed_total`: samples that failed when sent to remote storage
    98    - `loki_ruler_wal_prometheus_remote_storage_samples_dropped_total`: samples dropped by relabel configurations
    99    - `loki_ruler_wal_prometheus_remote_storage_samples_retried_total`: samples re-resent to remote storage
   100  - `loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds`: highest timestamp of sample appended to WAL
   101  - `loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds`: highest timestamp of sample sent to remote storage.
   102  
   103  We've created a basic [dashboard in our loki-mixin](https://github.com/grafana/loki/tree/main/production/loki-mixin/dashboards/recording-rules.libsonnet)
   104  which you can use to administer recording rules.
   105  
   106  ## Failure Modes
   107  
   108  ### Remote-Write Lagging
   109  
   110  Remote-write can lag behind for many reasons:
   111  
   112  1. Remote-write storage (Prometheus) is temporarily unavailable
   113  2. A tenant is producing samples too quickly from a recording rule
   114  3. Remote-write is tuned too low, creating backpressure
   115  
   116  It can be determined by subtracting
   117  `loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds` from
   118  `loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds`.
   119  
   120  In case 1, the `ruler` will continue to retry sending these samples until the remote storage becomes available again. Be
   121  aware that if the remote storage is down for longer than `ruler.wal.max-age`, data loss may occur after truncation occurs.
   122  
   123  In cases 2 & 3, you should consider [tuning](#tuning) remote-write appropriately.
   124  
   125  Further reading: see [this blog post](https://grafana.com/blog/2021/04/12/how-to-troubleshoot-remote-write-issues-in-prometheus/)
   126  by Prometheus maintainer Callum Styan.
   127  
   128  ### Appender Not Ready
   129  
   130  Each tenant's WAL has an "appender" internally; this appender is used to _append_ samples to the WAL. The appender is marked
   131  as _not ready_ until the WAL replay is complete upon startup. If the WAL is corrupted for some reason, or is taking a long
   132  time to replay, you can determine this by alerting on `loki_ruler_wal_appender_ready < 1`.
   133  
   134  ### Corrupt WAL
   135  
   136  If a disk fails or the `ruler` does not terminate correctly, there's a chance one or more tenant WALs can become corrupted.
   137  A mechanism exists for automatically repairing the WAL, but this cannot handle every conceivable scenario. In this case,
   138  the `loki_ruler_wal_corruptions_repair_failed_total` metric will be incremented.
   139  
   140  ### Found another failure mode?
   141  
   142  Please open an [issue](https://github.com/grafana/loki/issues) and tell us about it!
   143