github.com/muhammadn/cortex@v1.9.1-0.20220510110439-46bb7000d03d/docs/operations/query-tee.md (about)

     1  ---
     2  title: "Query Tee (service)"
     3  linkTitle: "Query Tee (service)"
     4  weight: 3
     5  slug: query-tee
     6  ---
     7  
     8  The `query-tee` is a standalone service which can be used for testing purposes to compare the query performances of 2+ backend systems (ie. Cortex clusters) ingesting the same exact series.
     9  
    10  This service exposes Prometheus-compatible read API endpoints and, for each received request, performs the request against all backends tracking the response time of each backend and then sends back to the client one of the received responses.
    11  
    12  ## How to run it
    13  
    14  You can run `query-tee` in two ways:
    15  
    16  - Build it from sources
    17    ```
    18    go run ./cmd/query-tee -help
    19    ```
    20  - Run it via the provided Docker image
    21    ```
    22    docker run quay.io/cortexproject/query-tee -help
    23    ```
    24  
    25  The service requires at least 1 backend endpoint (but 2 are required in order to compare performances) configured as comma-separated HTTP(S) URLs via the CLI flag **`-backend.endpoints`**. For each incoming request, `query-tee` will clone the request and send it to each backend, tracking performance metrics for each backend before sending back the response to the client.
    26  
    27  ## How it works
    28  
    29  ### API endpoints
    30  
    31  The following Prometheus API endpoints are supported by `query-tee`:
    32  
    33  - `/api/v1/query` (GET)
    34  - `/api/v1/query_range` (GET)
    35  - `/api/v1/labels` (GET)
    36  - `/api/v1/label/{name}/values` (GET)
    37  - `/api/v1/series` (GET)
    38  - `/api/v1/metadata` (GET)
    39  - `/api/v1/alerts` (GET)
    40  - `/api/v1/rules` (GET)
    41  
    42  #### Pass-through requests
    43  
    44  `query-tee` supports acting as a transparent proxy for requests to routes not matching any of the documented API endpoints above.
    45  When enabled, those requests are passed on to just the configured preferred backend.
    46  To activate this feature it requires setting `-proxy.passthrough-non-registered-routes=true` flag and configuring a preferred backend.
    47  
    48  ### Authentication
    49  
    50  `query-tee` supports [HTTP basic authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication). It allows either to configure username and password in the backend URL, to forward the request auth to the backend or merge the two.
    51  
    52  The request sent from the `query-tee` to the backend includes HTTP basic authentication when one of the following conditions are met:
    53  
    54  - If the endpoint URL has username and password, `query-tee` uses it.
    55  - If the endpoint URL has username only, `query-tee` keeps the username and inject the password received in the incoming request (if any).
    56  - If the endpoint URL has no username and no password, `query-tee` forwards the incoming request basic authentication (if any).
    57  
    58  ### Backend response selection
    59  
    60  `query-tee` allows to configure a preferred backend from which picking the response to send back to the client. The preferred backend can be configured via the CLI flag `-backend.preferred=<hostname>`, setting it to the hostname of the preferred backend.
    61  
    62  When a preferred backend **is set**, `query-tee` sends back to the client:
    63  
    64  - The preferred backend response if the status code is 2xx or 4xx
    65  - Otherwise, the first received 2xx or 4xx response if at least a backend succeeded
    66  - Otherwise, the first received response
    67  
    68  When a preferred backend **is not set**, `query-tee` sends back to the client:
    69  
    70  - The first received 2xx or 4xx response if at least a backend succeeded
    71  - Otherwise, the first received response
    72  
    73  _Note: from the `query-tee` perspective, a backend request is considered successful even if the status code is 4xx because it generally means the error is due to an invalid request and not to a backend issue._
    74  
    75  ### Backend results comparison
    76  
    77  `query-tee` allows to optionally enable the query results comparison between two backends. The results comparison can be enabled via the CLI flag `-proxy.compare-responses=true` and requires exactly two configured backends with a preferred one.
    78  
    79  When the comparison is enabled, the `query-tee` compares the response received from the two configured backends and logs a message for each query whose results don't match, as well as keeps track of the number of successful and failed comparison through the metric `cortex_querytee_responses_compared_total`.
    80  
    81  Floating point sample values are compared with a small tolerance that can be configured via `-proxy.value-comparison-tolerance`. This prevents false positives due to differences in floating point values _rounding_ introduced by the non deterministic series ordering within the Prometheus PromQL engine.
    82  
    83  ### Slow backends
    84  
    85  `query-tee` sends back to the client the first viable response as soon as available, without waiting to receive a response from all backends.
    86  
    87  ### Exported metrics
    88  
    89  `query-tee` exposes the following Prometheus metrics on the port configured via the CLI flag `-server.metrics-port`:
    90  
    91  ```bash
    92  # HELP cortex_querytee_request_duration_seconds Time (in seconds) spent serving HTTP requests.
    93  # TYPE cortex_querytee_request_duration_seconds histogram
    94  cortex_querytee_request_duration_seconds_bucket{backend="<hostname>",method="<method>",route="<route>",status_code="<status>",le="<bucket>"}
    95  cortex_querytee_request_duration_seconds_sum{backend="<hostname>",method="<method>",route="<route>",status_code="<status>"}
    96  cortex_querytee_request_duration_seconds_count{backend="<hostname>",method="<method>",route="<route>",status_code="<status>"}
    97  
    98  # HELP cortex_querytee_responses_total Total number of responses sent back to the client by the selected backend.
    99  # TYPE cortex_querytee_responses_total counter
   100  cortex_querytee_responses_total{backend="<hostname>",method="<method>",route="<route>"}
   101  
   102  # HELP cortex_querytee_responses_compared_total Total number of responses compared per route name by result.
   103  # TYPE cortex_querytee_responses_compared_total counter
   104  cortex_querytee_responses_compared_total{route="<route>",result="<success|fail>"}
   105  ```