github.com/thanos-io/thanos@v0.32.5/docs/components/sidecar.md (about)

     1  # Sidecar
     2  
     3  The `thanos sidecar` command runs a component that gets deployed along with a Prometheus instance. This allows sidecar to optionally upload metrics to object storage and allow [Queriers](query.md) to query Prometheus data with common, efficient StoreAPI.
     4  
     5  In details:
     6  
     7  * It implements Thanos' Store API on top of Prometheus' remote-read API. This allows [Queriers](query.md) to treat Prometheus servers as yet another source of time series data without directly talking to its APIs.
     8  * Optionally, the sidecar uploads TSDB blocks to an object storage bucket as Prometheus produces them every 2 hours. This allows Prometheus servers to be run with relatively low retention while their historic data is made durable and queryable via object storage.
     9  
    10    NOTE: This still does NOT mean that Prometheus can be fully stateless, because if it crashes and restarts you will lose ~2 hours of metrics, so persistent disk for Prometheus is highly recommended. The closest to stateless you can get is using remote write (which Thanos supports, see [Receiver](receive.md). Remote write has other risks and consequences, and still if crashed you lose in positive case seconds of metrics data, so persistent disk is recommended in all cases.
    11  
    12  * Optionally Thanos sidecar is able to watch Prometheus rules and configuration, decompress and substitute environment variables if needed and ping Prometheus to reload them. Read more about this in [here](#reloader-configuration)
    13  
    14  Prometheus servers connected to the Thanos cluster via the sidecar are subject to a few limitations and recommendations for safe operations:
    15  
    16  * The recommended Prometheus version is 2.2.1 or greater (including newest releases). This is due to Prometheus instability in previous versions as well as lack of `flags` endpoint.
    17  * (!) The Prometheus `external_labels` section of the Prometheus configuration file has unique labels in the overall Thanos system. Those external labels will be used by the sidecar and then Thanos in many places. See [external labels](../storage.md#external-labels) docs.
    18  * The `--web.enable-admin-api` flag is enabled to support sidecar to get metadata from Prometheus like external labels.
    19  * The `--web.enable-lifecycle` flag is enabled if you want to use sidecar reloading features (`--reload.*` flags).
    20  
    21  If you choose to use the sidecar to also upload data to object storage:
    22  
    23  * Must specify object storage (`--objstore.*` flags)
    24  * It only uploads uncompacted Prometheus blocks. For compacted blocks, see [Upload compacted blocks](#upload-compacted-blocks).
    25  * The `--storage.tsdb.min-block-duration` and `--storage.tsdb.max-block-duration` must be set to equal values to disable local compaction in order to use Thanos sidecar upload, otherwise leave local compaction on if sidecar just exposes StoreAPI and your retention is normal. The default of `2h` is recommended. Mentioned parameters set to equal values disable the internal Prometheus compaction, which is needed to avoid the corruption of uploaded data when Thanos compactor does its job, this is critical for data consistency and should not be ignored if you plan to use Thanos compactor. Even though you set mentioned parameters equal, you might observe Prometheus internal metric `prometheus_tsdb_compactions_total` being incremented, don't be confused by that: Prometheus writes initial head block to filesystem via its internal compaction mechanism, but if you have followed recommendations - data won't be modified by Prometheus before the sidecar uploads it. Thanos sidecar will also check sanity of the flags set to Prometheus on the startup and log errors or warning if they have been configured improperly (#838).
    26  * The retention of Prometheus is recommended to not be lower than three times of the min block duration, so 6 hours. This achieves resilience in the face of connectivity issues to the object storage since all local data will remain available within the Thanos cluster. If connectivity gets restored the backlog of blocks gets uploaded to the object storage.
    27  
    28  ## Reloader Configuration
    29  
    30  Thanos can watch changes in Prometheus configuration and refresh Prometheus configuration if `--web.enable-lifecycle` enabled.
    31  
    32  You can configure watching for changes in directory via `--reloader.rule-dir=DIR_NAME` flag.
    33  
    34  Thanos sidecar can watch `--reloader.config-file=CONFIG_FILE` configuration file, replace environment variables found in there in `$(VARIABLE)` format, and produce generated config in `--reloader.config-envsubst-file=OUT_CONFIG_FILE` file.
    35  
    36  ## Example basic deployment
    37  
    38  ```bash
    39  prometheus \
    40    --storage.tsdb.max-block-duration=2h \
    41    --storage.tsdb.min-block-duration=2h \
    42    --web.enable-lifecycle
    43  ```
    44  
    45  ```bash
    46  thanos sidecar \
    47      --tsdb.path        "/path/to/prometheus/data/dir" \
    48      --prometheus.url   "http://localhost:9090" \
    49      --objstore.config-file  "bucket.yml"
    50  ```
    51  
    52  The example content of `bucket.yml`:
    53  
    54  ```yaml mdox-exec="go run scripts/cfggen/main.go --name=gcs.Config"
    55  type: GCS
    56  config:
    57    bucket: ""
    58    service_account: ""
    59  prefix: ""
    60  ```
    61  
    62  ## Upload compacted blocks
    63  
    64  If you want to migrate from a pure Prometheus setup to Thanos and have to keep the historical data, you can use the flag `--shipper.upload-compacted`. This will also upload blocks that were compacted by Prometheus. Values greater than 1 in the `compaction.level` field of a Prometheus block’s `meta.json` file indicate level of compaction.
    65  
    66  To use this, the Prometheus compaction needs to be disabled. This can be done by setting the following flags for Prometheus:
    67  
    68  - `--storage.tsdb.min-block-duration=2h`
    69  - `--storage.tsdb.max-block-duration=2h`
    70  
    71  ## Flags
    72  
    73  ```$ mdox-exec="thanos sidecar --help"
    74  usage: thanos sidecar [<flags>]
    75  
    76  Sidecar for Prometheus server.
    77  
    78  Flags:
    79        --grpc-address="0.0.0.0:10901"
    80                                   Listen ip:port address for gRPC endpoints
    81                                   (StoreAPI). Make sure this address is routable
    82                                   from other components.
    83        --grpc-grace-period=2m     Time to wait after an interrupt received for
    84                                   GRPC Server.
    85        --grpc-server-max-connection-age=60m
    86                                   The grpc server max connection age. This
    87                                   controls how often to re-establish connections
    88                                   and redo TLS handshakes.
    89        --grpc-server-tls-cert=""  TLS Certificate for gRPC server, leave blank to
    90                                   disable TLS
    91        --grpc-server-tls-client-ca=""
    92                                   TLS CA to verify clients against. If no
    93                                   client CA is specified, there is no client
    94                                   verification on server side. (tls.NoClientCert)
    95        --grpc-server-tls-key=""   TLS Key for the gRPC server, leave blank to
    96                                   disable TLS
    97        --hash-func=               Specify which hash function to use when
    98                                   calculating the hashes of produced files.
    99                                   If no function has been specified, it does not
   100                                   happen. This permits avoiding downloading some
   101                                   files twice albeit at some performance cost.
   102                                   Possible values are: "", "SHA256".
   103    -h, --help                     Show context-sensitive help (also try
   104                                   --help-long and --help-man).
   105        --http-address="0.0.0.0:10902"
   106                                   Listen host:port for HTTP endpoints.
   107        --http-grace-period=2m     Time to wait after an interrupt received for
   108                                   HTTP Server.
   109        --http.config=""           [EXPERIMENTAL] Path to the configuration file
   110                                   that can enable TLS or authentication for all
   111                                   HTTP endpoints.
   112        --log.format=logfmt        Log format to use. Possible options: logfmt or
   113                                   json.
   114        --log.level=info           Log filtering level.
   115        --min-time=0000-01-01T00:00:00Z
   116                                   Start of time range limit to serve. Thanos
   117                                   sidecar will serve only metrics, which happened
   118                                   later than this value. Option can be a constant
   119                                   time in RFC3339 format or time duration
   120                                   relative to current time, such as -1d or 2h45m.
   121                                   Valid duration units are ms, s, m, h, d, w, y.
   122        --objstore.config=<content>
   123                                   Alternative to 'objstore.config-file'
   124                                   flag (mutually exclusive). Content of
   125                                   YAML file that contains object store
   126                                   configuration. See format details:
   127                                   https://thanos.io/tip/thanos/storage.md/#configuration
   128        --objstore.config-file=<file-path>
   129                                   Path to YAML file that contains object
   130                                   store configuration. See format details:
   131                                   https://thanos.io/tip/thanos/storage.md/#configuration
   132        --prometheus.get_config_interval=30s
   133                                   How often to get Prometheus config
   134        --prometheus.get_config_timeout=5s
   135                                   Timeout for getting Prometheus config
   136        --prometheus.http-client=<content>
   137                                   Alternative to 'prometheus.http-client-file'
   138                                   flag (mutually exclusive). Content
   139                                   of YAML file or string with http
   140                                   client configs. See Format details:
   141                                   https://thanos.io/tip/components/sidecar.md/#configuration.
   142        --prometheus.http-client-file=<file-path>
   143                                   Path to YAML file or string with http
   144                                   client configs. See Format details:
   145                                   https://thanos.io/tip/components/sidecar.md/#configuration.
   146        --prometheus.ready_timeout=10m
   147                                   Maximum time to wait for the Prometheus
   148                                   instance to start up
   149        --prometheus.url=http://localhost:9090
   150                                   URL at which to reach Prometheus's API.
   151                                   For better performance use local network.
   152        --reloader.config-envsubst-file=""
   153                                   Output file for environment variable
   154                                   substituted config file.
   155        --reloader.config-file=""  Config file watched by the reloader.
   156        --reloader.retry-interval=5s
   157                                   Controls how often reloader retries config
   158                                   reload in case of error.
   159        --reloader.rule-dir=RELOADER.RULE-DIR ...
   160                                   Rule directories for the reloader to refresh
   161                                   (repeated field).
   162        --reloader.watch-interval=3m
   163                                   Controls how often reloader re-reads config and
   164                                   rules.
   165        --request.logging-config=<content>
   166                                   Alternative to 'request.logging-config-file'
   167                                   flag (mutually exclusive). Content
   168                                   of YAML file with request logging
   169                                   configuration. See format details:
   170                                   https://thanos.io/tip/thanos/logging.md/#configuration
   171        --request.logging-config-file=<file-path>
   172                                   Path to YAML file with request logging
   173                                   configuration. See format details:
   174                                   https://thanos.io/tip/thanos/logging.md/#configuration
   175        --shipper.upload-compacted
   176                                   If true shipper will try to upload compacted
   177                                   blocks as well. Useful for migration purposes.
   178                                   Works only if compaction is disabled on
   179                                   Prometheus. Do it once and then disable the
   180                                   flag when done.
   181        --store.limits.request-samples=0
   182                                   The maximum samples allowed for a single
   183                                   Series request, The Series call fails if
   184                                   this limit is exceeded. 0 means no limit.
   185                                   NOTE: For efficiency the limit is internally
   186                                   implemented as 'chunks limit' considering each
   187                                   chunk contains a maximum of 120 samples.
   188        --store.limits.request-series=0
   189                                   The maximum series allowed for a single Series
   190                                   request. The Series call fails if this limit is
   191                                   exceeded. 0 means no limit.
   192        --tracing.config=<content>
   193                                   Alternative to 'tracing.config-file' flag
   194                                   (mutually exclusive). Content of YAML file
   195                                   with tracing configuration. See format details:
   196                                   https://thanos.io/tip/thanos/tracing.md/#configuration
   197        --tracing.config-file=<file-path>
   198                                   Path to YAML file with tracing
   199                                   configuration. See format details:
   200                                   https://thanos.io/tip/thanos/tracing.md/#configuration
   201        --tsdb.path="./data"       Data directory of TSDB.
   202        --version                  Show application version.
   203  
   204  ```
   205  
   206  ## Configuration
   207  
   208  ### Prometheus HTTP client
   209  
   210  You can configure the Prometheus HTTP client for Thanos sidecar with YAML, either by passing the YAML content directly to the `--prometheus.http-client` flag, or by passing the YAML file path to the `--prometheus.http-client-file` flag.
   211  
   212  The configuration format is the following:
   213  
   214  ```yaml
   215  basic_auth:
   216    username: ""
   217    password: ""
   218    password_file: ""
   219  bearer_token: ""
   220  bearer_token_file: ""
   221  proxy_url: ""
   222  tls_config:
   223    ca_file: ""
   224    cert_file: ""
   225    key_file: ""
   226    server_name: ""
   227    insecure_skip_verify: false
   228  transport_config:
   229    max_idle_conns: 0
   230    max_idle_conns_per_host: 0
   231    idle_conn_timeout: 0
   232    response_header_timeout: 0
   233    expect_continue_timeout: 0
   234    max_conns_per_host: 0
   235    disable_compression: false
   236    tls_handshake_timeout: 0
   237  ```