github.com/thanos-io/thanos@v0.32.5/docs/components/tools.md (about)

     1  # Tools
     2  
     3  The `thanos tools` subcommand of Thanos is a set of additional CLI, short-living tools that are meant to be ran for development or debugging purposes.
     4  
     5  All commands added as tools should land in `tools.go` or file with `tools_` prefix.
     6  
     7  ## Flags
     8  
     9  ```$ mdox-exec="thanos tools --help"
    10  usage: thanos tools <command> [<args> ...]
    11  
    12  Tools utility commands
    13  
    14  Flags:
    15    -h, --help               Show context-sensitive help (also try --help-long and
    16                             --help-man).
    17        --log.format=logfmt  Log format to use. Possible options: logfmt or json.
    18        --log.level=info     Log filtering level.
    19        --tracing.config=<content>
    20                             Alternative to 'tracing.config-file' flag
    21                             (mutually exclusive). Content of YAML file
    22                             with tracing configuration. See format details:
    23                             https://thanos.io/tip/thanos/tracing.md/#configuration
    24        --tracing.config-file=<file-path>
    25                             Path to YAML file with tracing
    26                             configuration. See format details:
    27                             https://thanos.io/tip/thanos/tracing.md/#configuration
    28        --version            Show application version.
    29  
    30  Subcommands:
    31    tools bucket verify [<flags>]
    32      Verify all blocks in the bucket against specified issues. NOTE: Depending on
    33      issue this might take time and will need downloading all specified blocks to
    34      disk.
    35  
    36    tools bucket ls [<flags>]
    37      List all blocks in the bucket.
    38  
    39    tools bucket inspect [<flags>]
    40      Inspect all blocks in the bucket in detailed, table-like way.
    41  
    42    tools bucket web [<flags>]
    43      Web interface for remote storage bucket.
    44  
    45    tools bucket replicate [<flags>]
    46      Replicate data from one object storage to another. NOTE: Currently it works
    47      only with Thanos blocks (meta.json has to have Thanos metadata).
    48  
    49    tools bucket downsample [<flags>]
    50      Continuously downsamples blocks in an object store bucket.
    51  
    52    tools bucket cleanup [<flags>]
    53      Cleans up all blocks marked for deletion.
    54  
    55    tools bucket mark --id=ID --marker=MARKER [<flags>]
    56      Mark block for deletion or no-compact in a safe way. NOTE: If the compactor
    57      is currently running compacting same block, this operation would be
    58      potentially a noop.
    59  
    60    tools bucket rewrite --id=ID [<flags>]
    61      Rewrite chosen blocks in the bucket, while deleting or modifying
    62      series Resulted block has modified stats in meta.json. Additionally
    63      compaction.sources are altered to not confuse readers of meta.json.
    64      Instead thanos.rewrite section is added with useful info like old sources
    65      and deletion requests. NOTE: It's recommended to turn off compactor while
    66      doing this operation. If the compactor is running and touching exactly same
    67      block that is being rewritten, the resulted rewritten block might only cause
    68      overlap (mitigated by marking overlapping block manually for deletion) and
    69      the data you wanted to rewrite could already part of bigger block.
    70  
    71      Use FILESYSTEM type of bucket to rewrite block on disk (suitable for vanilla
    72      Prometheus) After rewrite, it's caller responsibility to delete or mark
    73      source block for deletion to avoid overlaps. WARNING: This procedure is
    74      *IRREVERSIBLE* after certain time (delete delay), so do backup your blocks
    75      first.
    76  
    77    tools bucket retention [<flags>]
    78      Retention applies retention policies on the given bucket. Please make sure
    79      no compactor is running on the same bucket at the same time.
    80  
    81    tools rules-check --rules=RULES
    82      Check if the rule files are valid or not.
    83  
    84  
    85  ```
    86  
    87  ## Bucket
    88  
    89  The `thanos tools bucket` subcommand of Thanos is a set of commands to inspect data in object storage buckets. It is normally run as a standalone command to aid with troubleshooting.
    90  
    91  Example:
    92  
    93  ```bash
    94  thanos tools bucket verify --objstore.config-file=bucket.yml
    95  ```
    96  
    97  The content of `bucket.yml`:
    98  
    99  ```yaml mdox-exec="go run scripts/cfggen/main.go --name=gcs.Config"
   100  type: GCS
   101  config:
   102    bucket: ""
   103    service_account: ""
   104  prefix: ""
   105  ```
   106  
   107  Bucket can be extended to add more subcommands that will be helpful when working with object storage buckets by adding a new command within [`/cmd/thanos/tools_bucket.go`](../../cmd/thanos/tools_bucket.go)  .
   108  
   109  ```$ mdox-exec="thanos tools bucket --help"
   110  usage: thanos tools bucket [<flags>] <command> [<args> ...]
   111  
   112  Bucket utility commands
   113  
   114  Flags:
   115    -h, --help               Show context-sensitive help (also try --help-long and
   116                             --help-man).
   117        --log.format=logfmt  Log format to use. Possible options: logfmt or json.
   118        --log.level=info     Log filtering level.
   119        --objstore.config=<content>
   120                             Alternative to 'objstore.config-file' flag (mutually
   121                             exclusive). Content of YAML file that contains
   122                             object store configuration. See format details:
   123                             https://thanos.io/tip/thanos/storage.md/#configuration
   124        --objstore.config-file=<file-path>
   125                             Path to YAML file that contains object
   126                             store configuration. See format details:
   127                             https://thanos.io/tip/thanos/storage.md/#configuration
   128        --tracing.config=<content>
   129                             Alternative to 'tracing.config-file' flag
   130                             (mutually exclusive). Content of YAML file
   131                             with tracing configuration. See format details:
   132                             https://thanos.io/tip/thanos/tracing.md/#configuration
   133        --tracing.config-file=<file-path>
   134                             Path to YAML file with tracing
   135                             configuration. See format details:
   136                             https://thanos.io/tip/thanos/tracing.md/#configuration
   137        --version            Show application version.
   138  
   139  Subcommands:
   140    tools bucket verify [<flags>]
   141      Verify all blocks in the bucket against specified issues. NOTE: Depending on
   142      issue this might take time and will need downloading all specified blocks to
   143      disk.
   144  
   145    tools bucket ls [<flags>]
   146      List all blocks in the bucket.
   147  
   148    tools bucket inspect [<flags>]
   149      Inspect all blocks in the bucket in detailed, table-like way.
   150  
   151    tools bucket web [<flags>]
   152      Web interface for remote storage bucket.
   153  
   154    tools bucket replicate [<flags>]
   155      Replicate data from one object storage to another. NOTE: Currently it works
   156      only with Thanos blocks (meta.json has to have Thanos metadata).
   157  
   158    tools bucket downsample [<flags>]
   159      Continuously downsamples blocks in an object store bucket.
   160  
   161    tools bucket cleanup [<flags>]
   162      Cleans up all blocks marked for deletion.
   163  
   164    tools bucket mark --id=ID --marker=MARKER [<flags>]
   165      Mark block for deletion or no-compact in a safe way. NOTE: If the compactor
   166      is currently running compacting same block, this operation would be
   167      potentially a noop.
   168  
   169    tools bucket rewrite --id=ID [<flags>]
   170      Rewrite chosen blocks in the bucket, while deleting or modifying
   171      series Resulted block has modified stats in meta.json. Additionally
   172      compaction.sources are altered to not confuse readers of meta.json.
   173      Instead thanos.rewrite section is added with useful info like old sources
   174      and deletion requests. NOTE: It's recommended to turn off compactor while
   175      doing this operation. If the compactor is running and touching exactly same
   176      block that is being rewritten, the resulted rewritten block might only cause
   177      overlap (mitigated by marking overlapping block manually for deletion) and
   178      the data you wanted to rewrite could already part of bigger block.
   179  
   180      Use FILESYSTEM type of bucket to rewrite block on disk (suitable for vanilla
   181      Prometheus) After rewrite, it's caller responsibility to delete or mark
   182      source block for deletion to avoid overlaps. WARNING: This procedure is
   183      *IRREVERSIBLE* after certain time (delete delay), so do backup your blocks
   184      first.
   185  
   186    tools bucket retention [<flags>]
   187      Retention applies retention policies on the given bucket. Please make sure
   188      no compactor is running on the same bucket at the same time.
   189  
   190  
   191  ```
   192  
   193  ### Bucket Web
   194  
   195  `tools bucket web` is used to inspect bucket blocks in form of interactive web UI.
   196  
   197  This will start local webserver that will periodically update the view with given refresh.
   198  
   199  <img src="../img/bucket-web.jpg" class="img-fluid" alt="web"/>
   200  
   201  Example:
   202  
   203  ```
   204  thanos tools bucket web --objstore.config-file="..."
   205  ```
   206  
   207  ```$ mdox-exec="thanos tools bucket web --help"
   208  usage: thanos tools bucket web [<flags>]
   209  
   210  Web interface for remote storage bucket.
   211  
   212  Flags:
   213    -h, --help                    Show context-sensitive help (also try
   214                                  --help-long and --help-man).
   215        --http-address="0.0.0.0:10902"
   216                                  Listen host:port for HTTP endpoints.
   217        --http-grace-period=2m    Time to wait after an interrupt received for
   218                                  HTTP Server.
   219        --http.config=""          [EXPERIMENTAL] Path to the configuration file
   220                                  that can enable TLS or authentication for all
   221                                  HTTP endpoints.
   222        --label=LABEL             External block label to use as group title
   223        --log.format=logfmt       Log format to use. Possible options: logfmt or
   224                                  json.
   225        --log.level=info          Log filtering level.
   226        --max-time=9999-12-31T23:59:59Z
   227                                  End of time range limit to serve. Thanos
   228                                  tool bucket web will serve only blocks,
   229                                  which happened earlier than this value. Option
   230                                  can be a constant time in RFC3339 format or time
   231                                  duration relative to current time, such as -1d
   232                                  or 2h45m. Valid duration units are ms, s, m, h,
   233                                  d, w, y.
   234        --min-time=0000-01-01T00:00:00Z
   235                                  Start of time range limit to serve. Thanos
   236                                  tool bucket web will serve only blocks, which
   237                                  happened later than this value. Option can be a
   238                                  constant time in RFC3339 format or time duration
   239                                  relative to current time, such as -1d or 2h45m.
   240                                  Valid duration units are ms, s, m, h, d, w, y.
   241        --objstore.config=<content>
   242                                  Alternative to 'objstore.config-file'
   243                                  flag (mutually exclusive). Content of
   244                                  YAML file that contains object store
   245                                  configuration. See format details:
   246                                  https://thanos.io/tip/thanos/storage.md/#configuration
   247        --objstore.config-file=<file-path>
   248                                  Path to YAML file that contains object
   249                                  store configuration. See format details:
   250                                  https://thanos.io/tip/thanos/storage.md/#configuration
   251        --refresh=30m             Refresh interval to download metadata from
   252                                  remote storage
   253        --selector.relabel-config=<content>
   254                                  Alternative to 'selector.relabel-config-file'
   255                                  flag (mutually exclusive). Content of
   256                                  YAML file that contains relabeling
   257                                  configuration that allows selecting
   258                                  blocks. It follows native Prometheus
   259                                  relabel-config syntax. See format details:
   260                                  https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
   261        --selector.relabel-config-file=<file-path>
   262                                  Path to YAML file that contains relabeling
   263                                  configuration that allows selecting
   264                                  blocks. It follows native Prometheus
   265                                  relabel-config syntax. See format details:
   266                                  https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
   267        --timeout=5m              Timeout to download metadata from remote storage
   268        --tracing.config=<content>
   269                                  Alternative to 'tracing.config-file' flag
   270                                  (mutually exclusive). Content of YAML file
   271                                  with tracing configuration. See format details:
   272                                  https://thanos.io/tip/thanos/tracing.md/#configuration
   273        --tracing.config-file=<file-path>
   274                                  Path to YAML file with tracing
   275                                  configuration. See format details:
   276                                  https://thanos.io/tip/thanos/tracing.md/#configuration
   277        --version                 Show application version.
   278        --web.disable-cors        Whether to disable CORS headers to be set by
   279                                  Thanos. By default Thanos sets CORS headers to
   280                                  be allowed by all.
   281        --web.external-prefix=""  Static prefix for all HTML links and redirect
   282                                  URLs in the bucket web UI interface.
   283                                  Actual endpoints are still served on / or the
   284                                  web.route-prefix. This allows thanos bucket
   285                                  web UI to be served behind a reverse proxy that
   286                                  strips a URL sub-path.
   287        --web.prefix-header=""    Name of HTTP request header used for dynamic
   288                                  prefixing of UI links and redirects.
   289                                  This option is ignored if web.external-prefix
   290                                  argument is set. Security risk: enable
   291                                  this option only if a reverse proxy in
   292                                  front of thanos is resetting the header.
   293                                  The --web.prefix-header=X-Forwarded-Prefix
   294                                  option can be useful, for example, if Thanos
   295                                  UI is served via Traefik reverse proxy with
   296                                  PathPrefixStrip option enabled, which sends the
   297                                  stripped prefix value in X-Forwarded-Prefix
   298                                  header. This allows thanos UI to be served on a
   299                                  sub-path.
   300        --web.route-prefix=""     Prefix for API and UI endpoints. This allows
   301                                  thanos UI to be served on a sub-path.
   302                                  Defaults to the value of --web.external-prefix.
   303                                  This option is analogous to --web.route-prefix
   304                                  of Prometheus.
   305  
   306  ```
   307  
   308  ### Bucket Verify
   309  
   310  `tools bucket verify` is used to verify and optionally repair blocks within the specified bucket.
   311  
   312  Example:
   313  
   314  ```
   315  thanos tools bucket verify --objstore.config-file="..."
   316  ```
   317  
   318  When using the `--repair` option, make sure that the compactor job is disabled first.
   319  
   320  ```$ mdox-exec="thanos tools bucket verify --help"
   321  usage: thanos tools bucket verify [<flags>]
   322  
   323  Verify all blocks in the bucket against specified issues. NOTE: Depending on
   324  issue this might take time and will need downloading all specified blocks to
   325  disk.
   326  
   327  Flags:
   328        --delete-delay=0s    Duration after which blocks marked for deletion
   329                             would be deleted permanently from source bucket by
   330                             compactor component. If delete-delay is non zero,
   331                             blocks will be marked for deletion and compactor
   332                             component is required to delete blocks from source
   333                             bucket. If delete-delay is 0, blocks will be deleted
   334                             straight away. Use this if you want to get rid of
   335                             or move the block immediately. Note that deleting
   336                             blocks immediately can cause query failures, if store
   337                             gateway still has the block loaded, or compactor is
   338                             ignoring the deletion because it's compacting the
   339                             block at the same time.
   340    -h, --help               Show context-sensitive help (also try --help-long and
   341                             --help-man).
   342        --id=ID ...          Block IDs to verify (and optionally repair) only.
   343                             If none is specified, all blocks will be verified.
   344                             Repeated field
   345    -i, --issues=index_known_issues... ...
   346                             Issues to verify (and optionally repair). Possible
   347                             issue to verify, without repair: [overlapped_blocks];
   348                             Possible issue to verify and repair:
   349                             [index_known_issues duplicated_compaction]
   350        --log.format=logfmt  Log format to use. Possible options: logfmt or json.
   351        --log.level=info     Log filtering level.
   352        --objstore-backup.config=<content>
   353                             Alternative to 'objstore-backup.config-file'
   354                             flag (mutually exclusive). Content of YAML
   355                             file that contains object store-backup
   356                             configuration. See format details:
   357                             https://thanos.io/tip/thanos/storage.md/#configuration
   358                             Used for repair logic to backup blocks before
   359                             removal.
   360        --objstore-backup.config-file=<file-path>
   361                             Path to YAML file that contains object
   362                             store-backup configuration. See format details:
   363                             https://thanos.io/tip/thanos/storage.md/#configuration
   364                             Used for repair logic to backup blocks before
   365                             removal.
   366        --objstore.config=<content>
   367                             Alternative to 'objstore.config-file' flag (mutually
   368                             exclusive). Content of YAML file that contains
   369                             object store configuration. See format details:
   370                             https://thanos.io/tip/thanos/storage.md/#configuration
   371        --objstore.config-file=<file-path>
   372                             Path to YAML file that contains object
   373                             store configuration. See format details:
   374                             https://thanos.io/tip/thanos/storage.md/#configuration
   375    -r, --repair             Attempt to repair blocks for which issues were
   376                             detected
   377        --tracing.config=<content>
   378                             Alternative to 'tracing.config-file' flag
   379                             (mutually exclusive). Content of YAML file
   380                             with tracing configuration. See format details:
   381                             https://thanos.io/tip/thanos/tracing.md/#configuration
   382        --tracing.config-file=<file-path>
   383                             Path to YAML file with tracing
   384                             configuration. See format details:
   385                             https://thanos.io/tip/thanos/tracing.md/#configuration
   386        --version            Show application version.
   387  
   388  ```
   389  
   390  ### Bucket ls
   391  
   392  `tools bucket ls` is used to list all blocks in the specified bucket.
   393  
   394  Example:
   395  
   396  ```
   397  thanos tools bucket ls -o json --objstore.config-file="..."
   398  ```
   399  
   400  ```$ mdox-exec="thanos tools bucket ls --help"
   401  usage: thanos tools bucket ls [<flags>]
   402  
   403  List all blocks in the bucket.
   404  
   405  Flags:
   406        --exclude-delete     Exclude blocks marked for deletion.
   407    -h, --help               Show context-sensitive help (also try --help-long and
   408                             --help-man).
   409        --log.format=logfmt  Log format to use. Possible options: logfmt or json.
   410        --log.level=info     Log filtering level.
   411        --objstore.config=<content>
   412                             Alternative to 'objstore.config-file' flag (mutually
   413                             exclusive). Content of YAML file that contains
   414                             object store configuration. See format details:
   415                             https://thanos.io/tip/thanos/storage.md/#configuration
   416        --objstore.config-file=<file-path>
   417                             Path to YAML file that contains object
   418                             store configuration. See format details:
   419                             https://thanos.io/tip/thanos/storage.md/#configuration
   420    -o, --output=""          Optional format in which to print each block's
   421                             information. Options are 'json', 'wide' or a custom
   422                             template.
   423        --tracing.config=<content>
   424                             Alternative to 'tracing.config-file' flag
   425                             (mutually exclusive). Content of YAML file
   426                             with tracing configuration. See format details:
   427                             https://thanos.io/tip/thanos/tracing.md/#configuration
   428        --tracing.config-file=<file-path>
   429                             Path to YAML file with tracing
   430                             configuration. See format details:
   431                             https://thanos.io/tip/thanos/tracing.md/#configuration
   432        --version            Show application version.
   433  
   434  ```
   435  
   436  ### Bucket inspect
   437  
   438  `tools bucket inspect` is used to inspect buckets in a detailed way using stdout in ASCII table format.
   439  
   440  Example:
   441  
   442  ```
   443  thanos tools bucket inspect -l environment=\"prod\" --objstore.config-file="..."
   444  ```
   445  
   446  ```$ mdox-exec="thanos tools bucket inspect --help"
   447  usage: thanos tools bucket inspect [<flags>]
   448  
   449  Inspect all blocks in the bucket in detailed, table-like way.
   450  
   451  Flags:
   452    -h, --help                 Show context-sensitive help (also try --help-long
   453                               and --help-man).
   454        --log.format=logfmt    Log format to use. Possible options: logfmt or
   455                               json.
   456        --log.level=info       Log filtering level.
   457        --objstore.config=<content>
   458                               Alternative to 'objstore.config-file'
   459                               flag (mutually exclusive). Content of
   460                               YAML file that contains object store
   461                               configuration. See format details:
   462                               https://thanos.io/tip/thanos/storage.md/#configuration
   463        --objstore.config-file=<file-path>
   464                               Path to YAML file that contains object
   465                               store configuration. See format details:
   466                               https://thanos.io/tip/thanos/storage.md/#configuration
   467        --output=table         Output format for result. Currently supports table,
   468                               cvs, tsv.
   469    -l, --selector=<name>=\"<value>\" ...
   470                               Selects blocks based on label, e.g. '-l
   471                               key1=\"value1\" -l key2=\"value2\"'. All key value
   472                               pairs must match.
   473        --sort-by=FROM... ...  Sort by columns. It's also possible to sort by
   474                               multiple columns, e.g. '--sort-by FROM --sort-by
   475                               UNTIL'. I.e., if the 'FROM' value is equal the rows
   476                               are then further sorted by the 'UNTIL' value.
   477        --timeout=5m           Timeout to download metadata from remote storage
   478        --tracing.config=<content>
   479                               Alternative to 'tracing.config-file' flag
   480                               (mutually exclusive). Content of YAML file
   481                               with tracing configuration. See format details:
   482                               https://thanos.io/tip/thanos/tracing.md/#configuration
   483        --tracing.config-file=<file-path>
   484                               Path to YAML file with tracing
   485                               configuration. See format details:
   486                               https://thanos.io/tip/thanos/tracing.md/#configuration
   487        --version              Show application version.
   488  
   489  ```
   490  
   491  ### Bucket replicate
   492  
   493  `bucket tools replicate` is used to replicate buckets from one object storage to another.
   494  
   495  NOTE: Currently it works only with Thanos blocks (meta.json has to have Thanos metadata).
   496  
   497  Example:
   498  
   499  ```
   500  thanos tools bucket replicate --objstore.config-file="..." --objstore-to.config="..."
   501  ```
   502  
   503  ```$ mdox-exec="thanos tools bucket replicate --help"
   504  usage: thanos tools bucket replicate [<flags>]
   505  
   506  Replicate data from one object storage to another. NOTE: Currently it works only
   507  with Thanos blocks (meta.json has to have Thanos metadata).
   508  
   509  Flags:
   510        --compaction=COMPACTION ...
   511                                Only blocks with these compaction levels
   512                                will be replicated. Repeated flag. Overrides
   513                                compaction-min and compaction-max if set.
   514        --compaction-max=4      Only blocks up to a maximum of this compaction
   515                                level will be replicated.
   516        --compaction-min=1      Only blocks with at least this compaction level
   517                                will be replicated.
   518    -h, --help                  Show context-sensitive help (also try --help-long
   519                                and --help-man).
   520        --http-address="0.0.0.0:10902"
   521                                Listen host:port for HTTP endpoints.
   522        --http-grace-period=2m  Time to wait after an interrupt received for HTTP
   523                                Server.
   524        --http.config=""        [EXPERIMENTAL] Path to the configuration file
   525                                that can enable TLS or authentication for all HTTP
   526                                endpoints.
   527        --id=ID ...             Block to be replicated to the destination bucket.
   528                                IDs will be used to match blocks and other
   529                                matchers will be ignored. When specified, this
   530                                command will be run only once after successful
   531                                replication. Repeated field
   532        --ignore-marked-for-deletion
   533                                Do not replicate blocks that have deletion mark.
   534        --log.format=logfmt     Log format to use. Possible options: logfmt or
   535                                json.
   536        --log.level=info        Log filtering level.
   537        --matcher=MATCHER       blocks whose external labels match this matcher
   538                                will be replicated. All Prometheus matchers are
   539                                supported, including =, !=, =~ and !~.
   540        --max-time=9999-12-31T23:59:59Z
   541                                End of time range limit to replicate. Thanos
   542                                Replicate will replicate only metrics, which
   543                                happened earlier than this value. Option can be a
   544                                constant time in RFC3339 format or time duration
   545                                relative to current time, such as -1d or 2h45m.
   546                                Valid duration units are ms, s, m, h, d, w, y.
   547        --min-time=0000-01-01T00:00:00Z
   548                                Start of time range limit to replicate. Thanos
   549                                Replicate will replicate only metrics, which
   550                                happened later than this value. Option can be a
   551                                constant time in RFC3339 format or time duration
   552                                relative to current time, such as -1d or 2h45m.
   553                                Valid duration units are ms, s, m, h, d, w, y.
   554        --objstore-to.config=<content>
   555                                Alternative to 'objstore-to.config-file'
   556                                flag (mutually exclusive). Content of
   557                                YAML file that contains object store-to
   558                                configuration. See format details:
   559                                https://thanos.io/tip/thanos/storage.md/#configuration
   560                                The object storage which replicate data to.
   561        --objstore-to.config-file=<file-path>
   562                                Path to YAML file that contains object
   563                                store-to configuration. See format details:
   564                                https://thanos.io/tip/thanos/storage.md/#configuration
   565                                The object storage which replicate data to.
   566        --objstore.config=<content>
   567                                Alternative to 'objstore.config-file'
   568                                flag (mutually exclusive). Content of
   569                                YAML file that contains object store
   570                                configuration. See format details:
   571                                https://thanos.io/tip/thanos/storage.md/#configuration
   572        --objstore.config-file=<file-path>
   573                                Path to YAML file that contains object
   574                                store configuration. See format details:
   575                                https://thanos.io/tip/thanos/storage.md/#configuration
   576        --resolution=0s... ...  Only blocks with these resolutions will be
   577                                replicated. Repeated flag.
   578        --single-run            Run replication only one time, then exit.
   579        --tracing.config=<content>
   580                                Alternative to 'tracing.config-file' flag
   581                                (mutually exclusive). Content of YAML file
   582                                with tracing configuration. See format details:
   583                                https://thanos.io/tip/thanos/tracing.md/#configuration
   584        --tracing.config-file=<file-path>
   585                                Path to YAML file with tracing
   586                                configuration. See format details:
   587                                https://thanos.io/tip/thanos/tracing.md/#configuration
   588        --version               Show application version.
   589  
   590  ```
   591  
   592  ### Bucket downsample
   593  
   594  `tools bucket downsample` is used to downsample blocks in an object store bucket as a service. It implements the downsample API on top of historical data in an object storage bucket.
   595  
   596  ```bash
   597  thanos tools bucket downsample \
   598      --data-dir        "/local/state/data/dir" \
   599      --objstore.config-file "bucket.yml"
   600  ```
   601  
   602  The content of `bucket.yml`:
   603  
   604  ```yaml mdox-exec="go run scripts/cfggen/main.go --name=gcs.Config"
   605  type: GCS
   606  config:
   607    bucket: ""
   608    service_account: ""
   609  prefix: ""
   610  ```
   611  
   612  ```$ mdox-exec="thanos tools bucket downsample --help"
   613  usage: thanos tools bucket downsample [<flags>]
   614  
   615  Continuously downsamples blocks in an object store bucket.
   616  
   617  Flags:
   618        --block-files-concurrency=1
   619                                Number of goroutines to use when
   620                                fetching/uploading block files from object
   621                                storage.
   622        --data-dir="./data"     Data directory in which to cache blocks and
   623                                process downsamplings.
   624        --downsample.concurrency=1
   625                                Number of goroutines to use when downsampling
   626                                blocks.
   627        --hash-func=            Specify which hash function to use when
   628                                calculating the hashes of produced files. If no
   629                                function has been specified, it does not happen.
   630                                This permits avoiding downloading some files twice
   631                                albeit at some performance cost. Possible values
   632                                are: "", "SHA256".
   633    -h, --help                  Show context-sensitive help (also try --help-long
   634                                and --help-man).
   635        --http-address="0.0.0.0:10902"
   636                                Listen host:port for HTTP endpoints.
   637        --http-grace-period=2m  Time to wait after an interrupt received for HTTP
   638                                Server.
   639        --http.config=""        [EXPERIMENTAL] Path to the configuration file
   640                                that can enable TLS or authentication for all HTTP
   641                                endpoints.
   642        --log.format=logfmt     Log format to use. Possible options: logfmt or
   643                                json.
   644        --log.level=info        Log filtering level.
   645        --objstore.config=<content>
   646                                Alternative to 'objstore.config-file'
   647                                flag (mutually exclusive). Content of
   648                                YAML file that contains object store
   649                                configuration. See format details:
   650                                https://thanos.io/tip/thanos/storage.md/#configuration
   651        --objstore.config-file=<file-path>
   652                                Path to YAML file that contains object
   653                                store configuration. See format details:
   654                                https://thanos.io/tip/thanos/storage.md/#configuration
   655        --tracing.config=<content>
   656                                Alternative to 'tracing.config-file' flag
   657                                (mutually exclusive). Content of YAML file
   658                                with tracing configuration. See format details:
   659                                https://thanos.io/tip/thanos/tracing.md/#configuration
   660        --tracing.config-file=<file-path>
   661                                Path to YAML file with tracing
   662                                configuration. See format details:
   663                                https://thanos.io/tip/thanos/tracing.md/#configuration
   664        --version               Show application version.
   665        --wait-interval=5m      Wait interval between downsample runs.
   666  
   667  ```
   668  
   669  ### Bucket mark
   670  
   671  `tools bucket mark` can be used to manually mark block for deletion.
   672  
   673  NOTE: If the [Compactor](compact.md) is currently running and compacting exactly same block, this operation would be potentially a noop."
   674  
   675  ```bash
   676  thanos tools bucket mark \
   677      --id "01C8320GCGEWBZF51Q46TTQEH9" --id "01C8J352831FXGZQMN2NTJ08DY"
   678      --objstore.config-file "bucket.yml"
   679  ```
   680  
   681  The example content of `bucket.yml`:
   682  
   683  ```yaml mdox-exec="go run scripts/cfggen/main.go --name=gcs.Config"
   684  type: GCS
   685  config:
   686    bucket: ""
   687    service_account: ""
   688  prefix: ""
   689  ```
   690  
   691  ```$ mdox-exec="thanos tools bucket mark --help"
   692  usage: thanos tools bucket mark --id=ID --marker=MARKER [<flags>]
   693  
   694  Mark block for deletion or no-compact in a safe way. NOTE: If the compactor is
   695  currently running compacting same block, this operation would be potentially a
   696  noop.
   697  
   698  Flags:
   699        --details=DETAILS    Human readable details to be put into marker.
   700    -h, --help               Show context-sensitive help (also try --help-long and
   701                             --help-man).
   702        --id=ID ...          ID (ULID) of the blocks to be marked for deletion
   703                             (repeated flag)
   704        --log.format=logfmt  Log format to use. Possible options: logfmt or json.
   705        --log.level=info     Log filtering level.
   706        --marker=MARKER      Marker to be put.
   707        --objstore.config=<content>
   708                             Alternative to 'objstore.config-file' flag (mutually
   709                             exclusive). Content of YAML file that contains
   710                             object store configuration. See format details:
   711                             https://thanos.io/tip/thanos/storage.md/#configuration
   712        --objstore.config-file=<file-path>
   713                             Path to YAML file that contains object
   714                             store configuration. See format details:
   715                             https://thanos.io/tip/thanos/storage.md/#configuration
   716        --remove             Remove the marker.
   717        --tracing.config=<content>
   718                             Alternative to 'tracing.config-file' flag
   719                             (mutually exclusive). Content of YAML file
   720                             with tracing configuration. See format details:
   721                             https://thanos.io/tip/thanos/tracing.md/#configuration
   722        --tracing.config-file=<file-path>
   723                             Path to YAML file with tracing
   724                             configuration. See format details:
   725                             https://thanos.io/tip/thanos/tracing.md/#configuration
   726        --version            Show application version.
   727  
   728  ```
   729  
   730  ### Bucket Rewrite
   731  
   732  `tools bucket rewrite` rewrites chosen blocks in the bucket, while deleting or modifying series.
   733  
   734  For example we can remove all non counters from the block you have on your disk (e.g in Prometheus dir):
   735  
   736  ```bash
   737  thanos tools bucket rewrite --no-dry-run \
   738    --id 01DN3SK96XDAEKRB1AN30AAW6E \
   739    --objstore.config "
   740  type: FILESYSTEM
   741  config:
   742    directory: <local dir>
   743  " \
   744    --rewrite.to-delete-config "
   745  - matchers: \"{__name__!~\\\".*total\\\"}\"
   746  "
   747  ```
   748  
   749  By default, rewrite also produces `change.log` in the tmp local dir. Look for log message like:
   750  
   751  ```
   752  ts=2020-11-09T00:40:13.703322181Z caller=level.go:63 level=info msg="changelog will be available" file=/tmp/thanos-rewrite/01EPN74E401ZD2SQXS4SRY6DZX/change.log`
   753  ```
   754  
   755  ```$ mdox-exec="thanos tools bucket rewrite --help"
   756  usage: thanos tools bucket rewrite --id=ID [<flags>]
   757  
   758  Rewrite chosen blocks in the bucket, while deleting or modifying series Resulted
   759  block has modified stats in meta.json. Additionally compaction.sources are
   760  altered to not confuse readers of meta.json. Instead thanos.rewrite section
   761  is added with useful info like old sources and deletion requests. NOTE: It's
   762  recommended to turn off compactor while doing this operation. If the compactor
   763  is running and touching exactly same block that is being rewritten, the resulted
   764  rewritten block might only cause overlap (mitigated by marking overlapping block
   765  manually for deletion) and the data you wanted to rewrite could already part of
   766  bigger block.
   767  
   768  Use FILESYSTEM type of bucket to rewrite block on disk (suitable for vanilla
   769  Prometheus) After rewrite, it's caller responsibility to delete or mark source
   770  block for deletion to avoid overlaps. WARNING: This procedure is *IRREVERSIBLE*
   771  after certain time (delete delay), so do backup your blocks first.
   772  
   773  Flags:
   774        --delete-blocks           Whether to delete the original blocks after
   775                                  rewriting blocks successfully. Available in non
   776                                  dry-run mode only.
   777        --dry-run                 Prints the series changes instead of doing them.
   778                                  Defaults to true, for user to double check. (:
   779                                  Pass --no-dry-run to skip this.
   780        --hash-func=              Specify which hash function to use when
   781                                  calculating the hashes of produced files.
   782                                  If no function has been specified, it does not
   783                                  happen. This permits avoiding downloading some
   784                                  files twice albeit at some performance cost.
   785                                  Possible values are: "", "SHA256".
   786    -h, --help                    Show context-sensitive help (also try
   787                                  --help-long and --help-man).
   788        --id=ID ...               ID (ULID) of the blocks for rewrite (repeated
   789                                  flag).
   790        --log.format=logfmt       Log format to use. Possible options: logfmt or
   791                                  json.
   792        --log.level=info          Log filtering level.
   793        --objstore.config=<content>
   794                                  Alternative to 'objstore.config-file'
   795                                  flag (mutually exclusive). Content of
   796                                  YAML file that contains object store
   797                                  configuration. See format details:
   798                                  https://thanos.io/tip/thanos/storage.md/#configuration
   799        --objstore.config-file=<file-path>
   800                                  Path to YAML file that contains object
   801                                  store configuration. See format details:
   802                                  https://thanos.io/tip/thanos/storage.md/#configuration
   803        --prom-blocks             If specified, we assume the blocks to be
   804                                  uploaded are only used with Prometheus so we
   805                                  don't check external labels in this case.
   806        --rewrite.add-change-log  If specified, all modifications are written to
   807                                  new block directory. Disable if latency is to
   808                                  high.
   809        --rewrite.to-delete-config=<content>
   810                                  Alternative to 'rewrite.to-delete-config-file'
   811                                  flag (mutually exclusive). Content of YAML file
   812                                  that contains []metadata.DeletionRequest that
   813                                  will be applied to blocks
   814        --rewrite.to-delete-config-file=<file-path>
   815                                  Path to YAML file that contains
   816                                  []metadata.DeletionRequest that will be applied
   817                                  to blocks
   818        --rewrite.to-relabel-config=<content>
   819                                  Alternative to 'rewrite.to-relabel-config-file'
   820                                  flag (mutually exclusive). Content of YAML
   821                                  file that contains relabel configs that will be
   822                                  applied to blocks
   823        --rewrite.to-relabel-config-file=<file-path>
   824                                  Path to YAML file that contains relabel configs
   825                                  that will be applied to blocks
   826        --tmp.dir="/tmp/thanos-rewrite"
   827                                  Working directory for temporary files
   828        --tracing.config=<content>
   829                                  Alternative to 'tracing.config-file' flag
   830                                  (mutually exclusive). Content of YAML file
   831                                  with tracing configuration. See format details:
   832                                  https://thanos.io/tip/thanos/tracing.md/#configuration
   833        --tracing.config-file=<file-path>
   834                                  Path to YAML file with tracing
   835                                  configuration. See format details:
   836                                  https://thanos.io/tip/thanos/tracing.md/#configuration
   837        --version                 Show application version.
   838  
   839  ```
   840  
   841  ## Rules-check
   842  
   843  The `tools rules-check` subcommand contains tools for validation of Prometheus rules.
   844  
   845  This is allowing to check the rules with the same validation as is used by the Thanos Ruler node.
   846  
   847  NOTE: The check is equivalent to the `promtool check rules` with addition of Thanos Ruler extended rules file syntax, which includes `partial_response_strategy` field which `promtool` does not allow.
   848  
   849  If the check fails the command fails with exit code `1`, otherwise `0`.
   850  
   851  Example:
   852  
   853  ```
   854  ./thanos tools rules-check --rules cmd/thanos/testdata/rules-files/*.yaml
   855  ```
   856  
   857  ```$ mdox-exec="thanos tools rules-check --help"
   858  usage: thanos tools rules-check --rules=RULES
   859  
   860  Check if the rule files are valid or not.
   861  
   862  Flags:
   863    -h, --help               Show context-sensitive help (also try --help-long and
   864                             --help-man).
   865        --log.format=logfmt  Log format to use. Possible options: logfmt or json.
   866        --log.level=info     Log filtering level.
   867        --rules=RULES ...    The rule files glob to check (repeated).
   868        --tracing.config=<content>
   869                             Alternative to 'tracing.config-file' flag
   870                             (mutually exclusive). Content of YAML file
   871                             with tracing configuration. See format details:
   872                             https://thanos.io/tip/thanos/tracing.md/#configuration
   873        --tracing.config-file=<file-path>
   874                             Path to YAML file with tracing
   875                             configuration. See format details:
   876                             https://thanos.io/tip/thanos/tracing.md/#configuration
   877        --version            Show application version.
   878  
   879  ```
   880  
   881  #### Probes
   882  
   883  - The downsample service exposes two endpoints for probing:
   884    - `/-/healthy` starts as soon as the initial setup is completed.
   885    - `/-/ready` starts after all the bootstrapping completed (e.g object store bucket connection) and ready to serve traffic.
   886  
   887  > NOTE: Metric endpoint starts immediately so, make sure you set up readiness probe on designated HTTP `/-/ready` path.