github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/redis_streams.md (about) 1 --- 2 title: redis_streams 3 type: output 4 status: stable 5 categories: ["Services"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/redis_streams.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 19 Pushes messages to a Redis (v5.0+) Stream (which is created if it doesn't 20 already exist) using the XADD command. 21 22 23 <Tabs defaultValue="common" values={[ 24 { label: 'Common', value: 'common', }, 25 { label: 'Advanced', value: 'advanced', }, 26 ]}> 27 28 <TabItem value="common"> 29 30 ```yaml 31 # Common config fields, showing default values 32 output: 33 label: "" 34 redis_streams: 35 url: tcp://localhost:6379 36 stream: benthos_stream 37 body_key: body 38 max_length: 0 39 max_in_flight: 1 40 metadata: 41 exclude_prefixes: [] 42 batching: 43 count: 0 44 byte_size: 0 45 period: "" 46 check: "" 47 ``` 48 49 </TabItem> 50 <TabItem value="advanced"> 51 52 ```yaml 53 # All config fields, showing default values 54 output: 55 label: "" 56 redis_streams: 57 url: tcp://localhost:6379 58 kind: simple 59 master: "" 60 tls: 61 enabled: false 62 skip_cert_verify: false 63 enable_renegotiation: false 64 root_cas: "" 65 root_cas_file: "" 66 client_certs: [] 67 stream: benthos_stream 68 body_key: body 69 max_length: 0 70 max_in_flight: 1 71 metadata: 72 exclude_prefixes: [] 73 batching: 74 count: 0 75 byte_size: 0 76 period: "" 77 check: "" 78 processors: [] 79 ``` 80 81 </TabItem> 82 </Tabs> 83 84 It's possible to specify a maximum length of the target stream by setting it to 85 a value greater than 0, in which case this cap is applied only when Redis is 86 able to remove a whole macro node, for efficiency. 87 88 Redis stream entries are key/value pairs, as such it is necessary to specify the 89 key to be set to the body of the message. All metadata fields of the message 90 will also be set as key/value pairs, if there is a key collision between 91 a metadata item and the body then the body takes precedence. 92 93 ## Performance 94 95 This output benefits from sending multiple messages in flight in parallel for 96 improved performance. You can tune the max number of in flight messages with the 97 field `max_in_flight`. 98 99 This output benefits from sending messages as a batch for improved performance. 100 Batches can be formed at both the input and output level. You can find out more 101 [in this doc](/docs/configuration/batching). 102 103 ## Fields 104 105 ### `url` 106 107 The URL of the target Redis server. Database is optional and is supplied as the URL path. The scheme `tcp` is equivalent to `redis`. 108 109 110 Type: `string` 111 Default: `"tcp://localhost:6379"` 112 113 ```yaml 114 # Examples 115 116 url: :6397 117 118 url: localhost:6397 119 120 url: redis://localhost:6379 121 122 url: redis://:foopassword@redisplace:6379 123 124 url: redis://localhost:6379/1 125 126 url: redis://localhost:6379/1,redis://localhost:6380/1 127 ``` 128 129 ### `kind` 130 131 Specifies a simple, cluster-aware, or failover-aware redis client. 132 133 134 Type: `string` 135 Default: `"simple"` 136 137 ```yaml 138 # Examples 139 140 kind: simple 141 142 kind: cluster 143 144 kind: failover 145 ``` 146 147 ### `master` 148 149 Name of the redis master when `kind` is `failover` 150 151 152 Type: `string` 153 Default: `""` 154 155 ```yaml 156 # Examples 157 158 master: mymaster 159 ``` 160 161 ### `tls` 162 163 Custom TLS settings can be used to override system defaults. 164 165 **Troubleshooting** 166 167 Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you're using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. 168 169 170 Type: `object` 171 172 ### `tls.enabled` 173 174 Whether custom TLS settings are enabled. 175 176 177 Type: `bool` 178 Default: `false` 179 180 ### `tls.skip_cert_verify` 181 182 Whether to skip server side certificate verification. 183 184 185 Type: `bool` 186 Default: `false` 187 188 ### `tls.enable_renegotiation` 189 190 Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you're seeing the error message `local error: tls: no renegotiation`. 191 192 193 Type: `bool` 194 Default: `false` 195 Requires version 3.45.0 or newer 196 197 ### `tls.root_cas` 198 199 An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 200 201 202 Type: `string` 203 Default: `""` 204 205 ```yaml 206 # Examples 207 208 root_cas: |- 209 -----BEGIN CERTIFICATE----- 210 ... 211 -----END CERTIFICATE----- 212 ``` 213 214 ### `tls.root_cas_file` 215 216 An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 217 218 219 Type: `string` 220 Default: `""` 221 222 ```yaml 223 # Examples 224 225 root_cas_file: ./root_cas.pem 226 ``` 227 228 ### `tls.client_certs` 229 230 A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. 231 232 233 Type: `array` 234 Default: `[]` 235 236 ```yaml 237 # Examples 238 239 client_certs: 240 - cert: foo 241 key: bar 242 243 client_certs: 244 - cert_file: ./example.pem 245 key_file: ./example.key 246 ``` 247 248 ### `tls.client_certs[].cert` 249 250 A plain text certificate to use. 251 252 253 Type: `string` 254 Default: `""` 255 256 ### `tls.client_certs[].key` 257 258 A plain text certificate key to use. 259 260 261 Type: `string` 262 Default: `""` 263 264 ### `tls.client_certs[].cert_file` 265 266 The path to a certificate to use. 267 268 269 Type: `string` 270 Default: `""` 271 272 ### `tls.client_certs[].key_file` 273 274 The path of a certificate key to use. 275 276 277 Type: `string` 278 Default: `""` 279 280 ### `stream` 281 282 The stream to add messages to. 283 284 285 Type: `string` 286 Default: `"benthos_stream"` 287 288 ### `body_key` 289 290 A key to set the raw body of the message to. 291 292 293 Type: `string` 294 Default: `"body"` 295 296 ### `max_length` 297 298 When greater than zero enforces a rough cap on the length of the target stream. 299 300 301 Type: `int` 302 Default: `0` 303 304 ### `max_in_flight` 305 306 The maximum number of messages to have in flight at a given time. Increase this to improve throughput. 307 308 309 Type: `int` 310 Default: `1` 311 312 ### `metadata` 313 314 Specify criteria for which metadata values are included in the message body. 315 316 317 Type: `object` 318 319 ### `metadata.exclude_prefixes` 320 321 Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. 322 323 324 Type: `array` 325 Default: `[]` 326 327 ### `batching` 328 329 Allows you to configure a [batching policy](/docs/configuration/batching). 330 331 332 Type: `object` 333 334 ```yaml 335 # Examples 336 337 batching: 338 byte_size: 5000 339 count: 0 340 period: 1s 341 342 batching: 343 count: 10 344 period: 1s 345 346 batching: 347 check: this.contains("END BATCH") 348 count: 0 349 period: 1m 350 ``` 351 352 ### `batching.count` 353 354 A number of messages at which the batch should be flushed. If `0` disables count based batching. 355 356 357 Type: `int` 358 Default: `0` 359 360 ### `batching.byte_size` 361 362 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 363 364 365 Type: `int` 366 Default: `0` 367 368 ### `batching.period` 369 370 A period in which an incomplete batch should be flushed regardless of its size. 371 372 373 Type: `string` 374 Default: `""` 375 376 ```yaml 377 # Examples 378 379 period: 1s 380 381 period: 1m 382 383 period: 500ms 384 ``` 385 386 ### `batching.check` 387 388 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 389 390 391 Type: `string` 392 Default: `""` 393 394 ```yaml 395 # Examples 396 397 check: this.type == "end_of_transaction" 398 ``` 399 400 ### `batching.processors` 401 402 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 403 404 405 Type: `array` 406 Default: `[]` 407 408 ```yaml 409 # Examples 410 411 processors: 412 - archive: 413 format: lines 414 415 processors: 416 - archive: 417 format: json_array 418 419 processors: 420 - merge_json: {} 421 ``` 422 423