github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/broker.md (about) 1 --- 2 title: broker 3 type: output 4 status: stable 5 categories: ["Utility"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/broker.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 19 Allows you to route messages to multiple child outputs using a range of 20 brokering [patterns](#patterns). 21 22 23 <Tabs defaultValue="common" values={[ 24 { label: 'Common', value: 'common', }, 25 { label: 'Advanced', value: 'advanced', }, 26 ]}> 27 28 <TabItem value="common"> 29 30 ```yaml 31 # Common config fields, showing default values 32 output: 33 label: "" 34 broker: 35 pattern: fan_out 36 outputs: [] 37 batching: 38 count: 0 39 byte_size: 0 40 period: "" 41 check: "" 42 ``` 43 44 </TabItem> 45 <TabItem value="advanced"> 46 47 ```yaml 48 # All config fields, showing default values 49 output: 50 label: "" 51 broker: 52 copies: 1 53 pattern: fan_out 54 max_in_flight: 1 55 outputs: [] 56 batching: 57 count: 0 58 byte_size: 0 59 period: "" 60 check: "" 61 processors: [] 62 ``` 63 64 </TabItem> 65 </Tabs> 66 67 [Processors](/docs/components/processors/about) can be listed to apply across 68 individual outputs or all outputs: 69 70 ```yaml 71 output: 72 broker: 73 pattern: fan_out 74 outputs: 75 - resource: foo 76 - resource: bar 77 # Processors only applied to messages sent to bar. 78 processors: 79 - resource: bar_processor 80 81 # Processors applied to messages sent to all brokered outputs. 82 processors: 83 - resource: general_processor 84 ``` 85 86 ## Fields 87 88 ### `copies` 89 90 The number of copies of each configured output to spawn. 91 92 93 Type: `int` 94 Default: `1` 95 96 ### `pattern` 97 98 The brokering pattern to use. 99 100 101 Type: `string` 102 Default: `"fan_out"` 103 Options: `fan_out`, `fan_out_sequential`, `round_robin`, `greedy`. 104 105 ### `max_in_flight` 106 107 The maximum number of parallel message batches to have in flight at any given time. Note that if a child output has a higher `max_in_flight` then the switch output will automatically match it, therefore this value is the minimum `max_in_flight` to set in cases where the child values can't be inferred (such as when using resource outputs as children). Only relevant for `fan_out`, `fan_out_sequential` brokers. 108 109 110 Type: `int` 111 Default: `1` 112 113 ### `outputs` 114 115 A list of child outputs to broker. 116 117 118 Type: `array` 119 Default: `[]` 120 121 ### `batching` 122 123 Allows you to configure a [batching policy](/docs/configuration/batching). 124 125 126 Type: `object` 127 128 ```yaml 129 # Examples 130 131 batching: 132 byte_size: 5000 133 count: 0 134 period: 1s 135 136 batching: 137 count: 10 138 period: 1s 139 140 batching: 141 check: this.contains("END BATCH") 142 count: 0 143 period: 1m 144 ``` 145 146 ### `batching.count` 147 148 A number of messages at which the batch should be flushed. If `0` disables count based batching. 149 150 151 Type: `int` 152 Default: `0` 153 154 ### `batching.byte_size` 155 156 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 157 158 159 Type: `int` 160 Default: `0` 161 162 ### `batching.period` 163 164 A period in which an incomplete batch should be flushed regardless of its size. 165 166 167 Type: `string` 168 Default: `""` 169 170 ```yaml 171 # Examples 172 173 period: 1s 174 175 period: 1m 176 177 period: 500ms 178 ``` 179 180 ### `batching.check` 181 182 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 183 184 185 Type: `string` 186 Default: `""` 187 188 ```yaml 189 # Examples 190 191 check: this.type == "end_of_transaction" 192 ``` 193 194 ### `batching.processors` 195 196 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 197 198 199 Type: `array` 200 Default: `[]` 201 202 ```yaml 203 # Examples 204 205 processors: 206 - archive: 207 format: lines 208 209 processors: 210 - archive: 211 format: json_array 212 213 processors: 214 - merge_json: {} 215 ``` 216 217 ## Patterns 218 219 The broker pattern determines the way in which messages are allocated and can be 220 chosen from the following: 221 222 ### `fan_out` 223 224 With the fan out pattern all outputs will be sent every message that passes 225 through Benthos in parallel. 226 227 If an output applies back pressure it will block all subsequent messages, and if 228 an output fails to send a message it will be retried continuously until 229 completion or service shut down. 230 231 Sometimes it is useful to disable the back pressure or retries of certain fan 232 out outputs and instead drop messages that have failed or were blocked. In this 233 case you can wrap outputs with a [`drop_on` output](/docs/components/outputs/drop_on). 234 235 ### `fan_out_sequential` 236 237 Similar to the fan out pattern except outputs are written to sequentially, 238 meaning an output is only written to once the preceding output has confirmed 239 receipt of the same message. 240 241 ### `round_robin` 242 243 With the round robin pattern each message will be assigned a single output 244 following their order. If an output applies back pressure it will block all 245 subsequent messages. If an output fails to send a message then the message will 246 be re-attempted with the next input, and so on. 247 248 ### `greedy` 249 250 The greedy pattern results in higher output throughput at the cost of 251 potentially disproportionate message allocations to those outputs. Each message 252 is sent to a single output, which is determined by allowing outputs to claim 253 messages as soon as they are able to process them. This results in certain 254 faster outputs potentially processing more messages at the cost of slower 255 outputs. 256