github.com/grafana/pyroscope@v1.18.0/pkg/compactor/SPLIT_MERGE_COMPACTOR.md (about)

     1  # Split and Merge Compactor
     2  
     3  ## How the grouper works
     4  
     5  Given the list of all blocks in the storage:
     6  
     7  - Group blocks by resolution + external labels (excluding shard ID). For each group:
     8    - Group blocks by compactable time ranges (eg. 2h, 6h, 12h, 24h)
     9      - Given the time range, if the time range is the "smallest one" (eg. 2h) AND there are non-sharded blocks, then we should split the block
    10        - Example: TR=2h
    11          - Case 1: all the blocks for the 2h time range have the "shard ID" label in the meta.json.
    12            In this case, we shouldn't run the split stage anymore.
    13          - Case 2: there is at least 1 block for the 2h time range NOT having "shard ID" in the meta.json.
    14            In this case, we should run the split stage on 2h blocks without the "shard ID".
    15        - Horizontal sharding
    16          - Each compactor will take decisions without a central coordination
    17          - Each compactor will run the planning for ALL blocks
    18          - Each compactor will only execute "jobs" belonging to its shard
    19          - Splitting job (each job consists of a set of blocks which will be merged together calling CompactWithSplitting())
    20            - Which blocks should belong to a specific job?
    21              - We don't want all blocks to be compacted together, otherwise doesn't scale
    22              - We use the configured number of output shards to determine how many concurrent jobs we want to run to split/shard blocks
    23              - For each block to split, we add it to the job with index `hash(blockID) % numShards`
    24              - Output: up to `numShards` jobs (each job contain the set of blocks to be merged together when running CompactWithSplitting())
    25            - How can we know if the compactor instance should process a job or not?
    26              - A job is owned if `hash(tenant + stage + time range + shard ID)` belongs to the compactor tokens
    27        - If the previous check has not produced any job AT ALL it means all the blocks for the "smallest time range" are already split
    28          (or there are no blocks at all), so we can proceed with the merging stage:
    29          - Group blocks by "shard ID" (taking in account the case a block doesn't have the shard ID)
    30          - Create a job for each group that contains 2+ blocks (because they need to be merged together)
    31          - Add the job to the list of "valid jobs to execute" only if for the job shard ID there are no other
    32            jobs already in the list of "valid jobs" overlapping its time range
    33  - Execute the jobs that belong to the compactor instance
    34    - Loop through jobs and filter out the jobs hashes not belonging to the compactor tokens