github.com/thanos-io/thanos@v0.32.5/docs/proposals-accepted/202012-receive-split.md (about) 1 --- 2 type: proposal 3 title: Thanos Routing Receive and Ingesting Receive 4 status: accepted 5 menu: proposals-accepted 6 --- 7 8 ### Related Tickets 9 10 * Hashring improvements: https://github.com/thanos-io/thanos/issues/3141 11 * Previous proposal implementations: https://github.com/thanos-io/thanos/pull/3845, https://github.com/thanos-io/thanos/pull/3580 12 * Distribution idea: https://github.com/thanos-io/thanos/pull/2675 13 14 ### Summary 15 16 This document describes the motivation and design of running **Receiver** in a stateless mode that does not have capabilities to store samples, it only routes remote write to further receivers based on hashring. 17 18 This allows setting optional deployment model were only ***routing receivers*** are using hashring files and does the routing and replication. That allows ***ingesting receivers*** to not handle any routing or hashring, only receiving multi tenant writes. 19 20 ### Motivation 21 22 [@squat](https://github.com/squat): 23 24 > Currently, any change to the hashring configuration file will trigger all Thanos Receive nodes to flush their multi-TSDBs, causing them to enter an unready state until the flush is complete. This unavailability during a flush allows for a clear state transition, however it can result in downtimes on the order of five minutes for every configuration change. Moreover, during configuration changes, the hashring goes through an even longer period of partial unreadiness, where some nodes begin and finish flushing before and after others. During this partial unreadiness, the hashring can expect high internal request failure rates, which cause clients to retry their requests, resulting in even higher load. Therefore, when the hashring configuration is changed due to automatic horizontal scaling of a set of Thanos Receivers, the system can expect higher than normal resource utilization, which can create a positive feedback loop that continuously scales the hashring. 25 26 ### Goals 27 28 * Reduce downtime of the ingestion logic in Thanos Receiver. 29 30 ### Proposal 31 32 We propose allowing to run Thanos Receiver in a mode that only forwards/replicates remote write (distributor mode) apart from the ingesting mode (default mode). You can enable that mode by simply *not specifying*: 33 34 ```yaml 35 --receive.local-endpoint=RECEIVE.LOCAL-ENDPOINT 36 Endpoint of local receive node. Used to 37 identify the local node in the hashring 38 configuration. 39 ``` 40 41 We can call this mode a "**Routing Receiver**". Similarly, we can skip specifying any hashring to Thanos Receiver (`--receive.hashrings-file=<path>`), explicitly purposing it only for ingesting. We can call this mode "**Ingesting Receiver**". 42 43 User can also mix all of those two modes for various federated hashrings etc. So instead of what we had before: 44 45  46 47 We have: 48 49  50 51 This allows us to (optionally) model deployment in a way that avoid expensive re-configuration of the stateful ingesting Receivers after the hashring configuration file has changed. 52 53 In comparison to previous proposal (as mentioned in [alternatives](#previous-proposal-separate-receive-route-command) we have big advantages: 54 55 1. We can *reduce number of components* in Thanos system, we can reuse similar component flags and documentation. Users has to learn about one less command and in result Thanos design is much more approachable. Less components mean less maintenance, code and other implicit duties: Separate changelogs, issue confusions, boilerplates, etc. 56 2. Allow consistent pattern with Query. We don't have separate StoreAPI component for proxying, we have that baked into Querier. This has been proven to be flexible and understandable, so I would like to propose similar pattern in Receiver. 57 3. This is more future proof for potential advanced cases like *chain of routers -> receivers -> routers -> receivers* for federated writes, so ***trees with depth n***. 58 59 ### Plan 60 61 * Receiver without `--receive.hashrings` does not forward or replicate requests, **it routes straight to multi-tsdb**. 62 * Receiver without ` --Receiver.local-endpoint` will assume that no storage is needed, **so will skip creating any resources for multi TSDB**. 63 * Add changes to the documentation (it's simplistic now). Mention two modes. 64 65 ### Alternative Solutions 66 67 #### Previous Proposal: Separate receive-route command 68 69 1. Split the Receiver component into **receive-route** and **receiver** (and ensure ease of resharding events). 70 2. Evaluate any effects on performance by simulating scenarios and collecting and analyzing metrics. 71 3. Use ***consistent hashing*** to avoid reshuffling time series after resharding events. The exact consistent hashing mechanism to be used needs some further research. 72 4. **Migration**: We document how the new architecture can be set up to have the same general deployment of the old architecture. (We run router and Receiver on the same node). 73 74 This potentially makes the receiver more difficult to operate and understandable for Thanos users. I would argue this is however much harder in overall Thanos deployment. Otherwise, this option is exactly the same. 75 76 #### Flag for current Receiver: --receive-route 77 78 Idea would be similar same as in [Proposal](#proposal), but there will be explicit flag to turn off local storage capabilities. 79 80 I think we can have much more understandable logic if *we simply not* configure hashring for **ingesting receivers** and not configure local hashring endpoint to notify that such Receiver instance will never store anything.