github.com/muhammadn/cortex@v1.9.1-0.20220510110439-46bb7000d03d/docs/operations/requests-mirroring-to-secondary-cluster.md (about)

     1  ---
     2  title: "Requests mirroring to secondary cluster"
     3  linkTitle: "Requests mirroring to secondary cluster"
     4  weight: 4
     5  slug: requests-mirroring-to-secondary-cluster
     6  ---
     7  
     8  Requests mirroring (or shadowing) is a technique you can use to mirror requests from a primary Cortex cluster to a secondary one.
     9  
    10  For example, requests mirroring can be used when you need to setup a testing Cortex cluster receiving the same series ingested by a primary one without having control over Prometheus remote write config (if you do, then configuring two remote write entries in Prometheus would be the preferred option).
    11  
    12  ## Mirroring with Envoy proxy
    13  
    14  [Envoy proxy](https://www.envoyproxy.io/) can be used to mirror HTTP requests to a secondary upstream cluster. From a network path perspective, you should run Envoy in front of both clusters distributors, letting Envoy to proxy requests to the primary Cortex cluster and mirror them to a secondary cluster in background. The performances and availability of the secondary cluster have no impact on the requests to the primary one. The response to the client will always be the one from the primary one. In this sense, the requests from Envoy to the secondary cluster are "fire and forget".
    15  
    16  ### Example Envoy config
    17  
    18  The following Envoy configuration shows an example with two Cortex clusters. Envoy will listen on port `9900` and will proxies all requests to `cortex-primary:80`, mirroring it to `cortex-secondary:80` too.
    19  
    20  [embedmd]:# (./requests-mirroring-envoy.yaml)
    21  ```yaml
    22  admin:
    23    # No access logs.
    24    access_log_path: /dev/null
    25    address:
    26      socket_address: { address: 0.0.0.0, port_value: 9901 }
    27  
    28  static_resources:
    29    listeners:
    30      - name: cortex_listener
    31        address:
    32          socket_address: { address: 0.0.0.0, port_value: 9900 }
    33        filter_chains:
    34          - filters:
    35              - name: envoy.http_connection_manager
    36                config:
    37                  stat_prefix: cortex_ingress
    38                  route_config:
    39                    name: all_routes
    40                    virtual_hosts:
    41                      - name: all_hosts
    42                        domains: ["*"]
    43                        routes:
    44                          - match: { prefix: "/" }
    45                            route:
    46                              cluster: cortex_primary
    47  
    48                              # Specifies the upstream timeout. This spans between the point at which the entire downstream
    49                              # request has been processed and when the upstream response has been completely processed.
    50                              timeout: 15s
    51  
    52                              # Specifies the cluster that requests will be mirrored to. The performances and availability of
    53                              # the secondary cluster have no impact on the requests to the primary one. The response to the
    54                              # client will always be the one from the primary one. The requests from Envoy to the secondary
    55                              # cluster are "fire and forget".
    56                              request_mirror_policies:
    57                                - cluster: cortex_secondary
    58                  http_filters:
    59                    - name: envoy.router
    60    clusters:
    61      - name: cortex_primary
    62        type: STRICT_DNS
    63        connect_timeout: 1s
    64        hosts: [{ socket_address: { address: cortex-primary, port_value: 80 }}]
    65        dns_refresh_rate: 5s
    66      - name: cortex_secondary
    67        type: STRICT_DNS
    68        connect_timeout: 1s
    69        hosts: [{ socket_address: { address: cortex-secondary, port_value: 80 }}]
    70        dns_refresh_rate: 5s
    71  ```