github.com/thanos-io/thanos@v0.32.5/docs/proposals-accepted/202209-receive-tenant-external-labels.md (about)

     1  ---
     2  type: proposal
     3  title: Allow statically specifying tenant-specific external labels in Receivers
     4  status: accepted
     5  owner: haanhvu
     6  menu: proposals-accepted
     7  ---
     8  
     9  ## 1 Related links/tickets
    10  
    11  https://github.com/thanos-io/thanos/issues/5434
    12  
    13  ## 2 Why
    14  
    15  We would like to do cross-tenant activities like grouping tenants' blocks or querying tenants that share the same attributes. Tenant's external labels can help us do those.
    16  
    17  ## 3 Pitfalls of the current solution
    18  
    19  Currently, we can only add external labels to Receiver itself, not to each tenant in the Receiver. So we can't do cross-tenant activities like grouping tenants' blocks or querying tenants that share the same attributes.
    20  
    21  ## 4 Goals
    22  
    23  * Allow users to statically add arbitrary tenants’ external labels in the easiest way possible
    24  * Allow users to statically change arbitrary tenants’ external labels in the easiest way possible
    25  * Changes in tenants’ external labels are handled correctly
    26  * Backward compatibility (e.g., with Receiver’s external labels) is assured
    27  * Tenants’ external labels are handled separately in RouterIngestor, RouterOnly, and IngestorOnly modes
    28  
    29  ## 5 Non-goals
    30  
    31  * Logically split RouterOnly and IngestorOnly modes (Issue [#5643](https://github.com/thanos-io/thanos/issues/5643)): If RouterOnly and IngestorOnly modes are logically split, implementing tenants’ external labels in RouterOnly and IngestorOnly modes would be less challenging. However, fixing this issue will not be a goal of this proposal, because it's not directly related to tenants' external labels. Regardless, fixing this issue before implementing tenants’ external labels in RouterOnly and IngestorOnly modes would be the best-case scenario.
    32  * Dynamically extract tenants' external labels from time series' data: This proposal only covers statically specifying tenants' external labels. Dynamically receiving and extracting tenants' external labels from time series' data will be added as a follow-up to this proposal.
    33  
    34  ## 6 Audience
    35  
    36  Users who are admin personas and need to perform admin operations on Thanos for multiple tenants
    37  
    38  ## 7 How
    39  
    40  In the hashring config, there will be new field `external_labels`. Something like this:
    41  
    42  ```
    43      [
    44          {
    45              "hashring": "tenant-a-b",
    46              "endpoints": ["127.0.0.1:10901"],
    47              "tenants": ["tenant-a, tenant-b"]
    48              "external_labels": ["key1=value1", "key2=value2", "key3=value3"]
    49          },
    50      ]
    51  ```
    52  
    53  In Receivers' MultiTSDB, external labels will be extended to each corresponding tenant's label set when the tenant's TSDB is started.
    54  
    55  Next thing we have to do is handling changes for tenants' external labels. That is, whenever users make any changes to tenants' external labels, Receivers' MultiTSDB will update those changes in each corresponding tenant's label set.
    56  
    57  We will handle the cases of hard tenancy first. Once tenants' external labels can be handled in those cases, we will move to soft tenancy cases.
    58  
    59  Tenants’ external labels will be first implemented in RouterIngestor, since this is the most commonly used mode.
    60  
    61  After that, we can implement tenants’ external labels in RouterOnly and IngestorOnly modes. As stated above, the best-case scenario would be logically splitting RouterOnly and IngestorOnl (Issue [#5643](https://github.com/thanos-io/thanos/issues/5643)) before implement tenants’ external labels in each.
    62  
    63  For the tests, the foremost ones are testing defining one or multiple tenants’ external labels correctly, handling changes in tenants’ external labels correctly, backward compatibility with Receiver’s external labels, and shipper detecting and uploading tenants’ external labels correctly to block storage. We may add more tests in the future but currently these are the most important ones to do first.
    64  
    65  ## 8 Implementation plan
    66  
    67  * Add a new `external_labels` field in the hashring config
    68  * Allow MultiTSDB to extend external labels to each corresponding tenant's label set
    69  * Allow MultiTSDB to update each tenant's label set whenever its external labels change
    70  * Handle external labels in soft tenancy cases
    71  * Implement tenants’ external labels in RouterOnly
    72  * Implement tenants’ external labels in IngestorOnly
    73  
    74  ### 9 Test plan
    75  
    76  * Defining one or multiple tenants’ external labels correctly
    77  * Handling changes in tenants’ external labels correctly
    78  * Backward compatibility with Receiver’s external labels
    79  * Shipper detecting and uploading tenants’ external labels correctly to block storage
    80  
    81  ### 10 Follow-up
    82  
    83  * Dynamically extract tenants' external labels from time series' data: Once statically specifying tenants' external labels have been implemented and tested successfully and completely, we can think of implementing dynamically receiving and extracting tenants' external labels from time series' data.
    84  * Automatically making use of tenants' external labels: We can think of the most useful use cases with tenants' external labels and whether we should automate any of those use cases. One typical case is automatically grouping new blocks based on tenants' external labels.
    85  
    86  (Both of these are Ben's ideas expressed [here](https://github.com/thanos-io/thanos/pull/5720#pullrequestreview-1167923565).)