github.com/thanos-io/thanos@v0.32.5/docs/sharding.md (about)

     1  # Sharding
     2  
     3  # Background
     4  
     5  Currently, all components that read from object store assume that all the operations and functionality should be done based on **all** the available blocks that are present in the certain bucket's root directory.
     6  
     7  This is in most cases totally fine, however with time and allowance of storing blocks from multiple `Sources` into the same bucket, the number of objects in a bucket can grow drastically.
     8  
     9  This means that with time you might want to scale out certain components e.g:
    10  
    11  ## Compactor
    12  
    13  Larger number of objects does not matter much, however compactor has to scale (CPU, network, disk, memory) with number of Sources pushing blocks to the object storage.
    14  
    15  ## Store Gateway
    16  
    17  Queries against store gateway which are touching large number of blocks (no matter from what Sources) might be expensive, so it can scale with number of blocks.
    18  
    19  # Relabelling
    20  
    21  Similar to [promtail](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#relabel_configs) this config will follow native [Prometheus relabel-config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) syntax.
    22  
    23  Now, thanos only support following relabel actions:
    24  
    25  * keep
    26  
    27  * drop
    28  
    29  * hashmod
    30    * `external labels` for all components
    31  
    32    * `__block_id` for store gateway, see this [example](https://github.com/observatorium/configuration/blob/bf1304b0d7bce2ae3fefa80412bb358f9aa176fb/environments/openshift/manifests/observatorium-template.yaml#L1514-L1521)
    33  
    34  The relabel config defines filtering process done on **every** synchronization with object storage.
    35  
    36  We will allow potentially manipulating with several of inputs:
    37  
    38  * External labels:
    39    * `<name>`
    40  
    41  Output:
    42  
    43  * If output is empty, drop block.
    44  
    45  By default, on empty relabel-config, all external labels are assumed.
    46  
    47  Example usages would be:
    48  
    49  * Drop blocks which contains external labels cluster=A
    50  
    51  ```yaml
    52  - action: drop
    53    regex: "A"
    54    source_labels:
    55    - cluster
    56  ```
    57  
    58  * Keep only blocks which contains external labels cluster=A
    59  
    60  ```yaml
    61  - action: keep
    62    regex: "A"
    63    source_labels:
    64    - cluster
    65  ```
    66  
    67  We can shard by adjusting which labels should be included in the blocks.
    68  
    69  # Time Partitioning
    70  
    71  For store gateway, we can specify `--min-time` and `--max-time` flags to filter for what blocks store gateway should be responsible for.
    72  
    73  More details can refer to "Time based partitioning" chapter in [Store gateway](components/store.md).