github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/kafka_franz.md (about) 1 --- 2 title: kafka_franz 3 type: output 4 status: experimental 5 categories: ["Services"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/kafka_franz.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 :::caution EXPERIMENTAL 19 This component is experimental and therefore subject to change or removal outside of major version releases. 20 ::: 21 An alternative Kafka output using the [Franz Kafka client library](https://github.com/twmb/franz-go). 22 23 Introduced in version 3.61.0. 24 25 26 <Tabs defaultValue="common" values={[ 27 { label: 'Common', value: 'common', }, 28 { label: 'Advanced', value: 'advanced', }, 29 ]}> 30 31 <TabItem value="common"> 32 33 ```yaml 34 # Common config fields, showing default values 35 output: 36 label: "" 37 kafka_franz: 38 seed_brokers: [] 39 topic: "" 40 key: "" 41 metadata: 42 include_prefixes: [] 43 include_patterns: [] 44 max_in_flight: 10 45 batching: 46 count: 0 47 byte_size: 0 48 period: "" 49 check: "" 50 ``` 51 52 </TabItem> 53 <TabItem value="advanced"> 54 55 ```yaml 56 # All config fields, showing default values 57 output: 58 label: "" 59 kafka_franz: 60 seed_brokers: [] 61 topic: "" 62 key: "" 63 partitioner: "" 64 metadata: 65 include_prefixes: [] 66 include_patterns: [] 67 max_in_flight: 10 68 batching: 69 count: 0 70 byte_size: 0 71 period: "" 72 check: "" 73 processors: [] 74 max_message_bytes: 1MB 75 compression: "" 76 tls: 77 enabled: false 78 skip_cert_verify: false 79 enable_renegotiation: false 80 root_cas: "" 81 root_cas_file: "" 82 client_certs: [] 83 sasl: [] 84 ``` 85 86 </TabItem> 87 </Tabs> 88 89 Writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating it back to the input. 90 91 This output is new and experimental, and the existing `kafka` input is not going anywhere, but here's some reasons why it might be worth trying this one out: 92 93 - You like shiny new stuff 94 - You are experiencing issues with the existing `kafka` output 95 - Someone told you to 96 97 98 ## Fields 99 100 ### `seed_brokers` 101 102 A list of broker addresses to connect to in order to establish connections. If an item of the list contains commas it will be expanded into multiple addresses. 103 104 105 Type: `array` 106 107 ```yaml 108 # Examples 109 110 seed_brokers: 111 - localhost:9092 112 113 seed_brokers: 114 - foo:9092 115 - bar:9092 116 117 seed_brokers: 118 - foo:9092,bar:9092 119 ``` 120 121 ### `topic` 122 123 A topic to write messages to. 124 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 125 126 127 Type: `string` 128 129 ### `key` 130 131 An optional key to populate for each message. 132 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 133 134 135 Type: `string` 136 137 ### `partitioner` 138 139 Override the default murmur2 hashing partitioner. 140 141 142 Type: `string` 143 144 | Option | Summary | 145 |---|---| 146 | `least_backup` | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | 147 | `round_robin` | Round-robin's messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | 148 149 150 ### `metadata` 151 152 Determine which (if any) metadata values should be added to messages as headers. 153 154 155 Type: `object` 156 157 ### `metadata.include_prefixes` 158 159 Provide a list of explicit metadata key prefixes to match against. 160 161 162 Type: `array` 163 164 ```yaml 165 # Examples 166 167 include_prefixes: 168 - foo_ 169 - bar_ 170 171 include_prefixes: 172 - kafka_ 173 174 include_prefixes: 175 - content- 176 ``` 177 178 ### `metadata.include_patterns` 179 180 Provide a list of explicit metadata key regular expression (re2) patterns to match against. 181 182 183 Type: `array` 184 185 ```yaml 186 # Examples 187 188 include_patterns: 189 - .* 190 191 include_patterns: 192 - _timestamp_unix$ 193 ``` 194 195 ### `max_in_flight` 196 197 The maximum number of batches to be sending in parallel at any given time. 198 199 200 Type: `int` 201 Default: `10` 202 203 ### `batching` 204 205 Allows you to configure a [batching policy](/docs/configuration/batching). 206 207 208 Type: `object` 209 210 ```yaml 211 # Examples 212 213 batching: 214 byte_size: 5000 215 count: 0 216 period: 1s 217 218 batching: 219 count: 10 220 period: 1s 221 222 batching: 223 check: this.contains("END BATCH") 224 count: 0 225 period: 1m 226 ``` 227 228 ### `batching.count` 229 230 A number of messages at which the batch should be flushed. If `0` disables count based batching. 231 232 233 Type: `int` 234 Default: `0` 235 236 ### `batching.byte_size` 237 238 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 239 240 241 Type: `int` 242 Default: `0` 243 244 ### `batching.period` 245 246 A period in which an incomplete batch should be flushed regardless of its size. 247 248 249 Type: `string` 250 Default: `""` 251 252 ```yaml 253 # Examples 254 255 period: 1s 256 257 period: 1m 258 259 period: 500ms 260 ``` 261 262 ### `batching.check` 263 264 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 265 266 267 Type: `string` 268 Default: `""` 269 270 ```yaml 271 # Examples 272 273 check: this.type == "end_of_transaction" 274 ``` 275 276 ### `batching.processors` 277 278 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 279 280 281 Type: `array` 282 283 ```yaml 284 # Examples 285 286 processors: 287 - archive: 288 format: lines 289 290 processors: 291 - archive: 292 format: json_array 293 294 processors: 295 - merge_json: {} 296 ``` 297 298 ### `max_message_bytes` 299 300 The maximum space in bytes than an individual message may take, messages larger than this value will be rejected. This field corresponds to Kafka's `max.message.bytes`. 301 302 303 Type: `string` 304 Default: `"1MB"` 305 306 ```yaml 307 # Examples 308 309 max_message_bytes: 100MB 310 311 max_message_bytes: 50mib 312 ``` 313 314 ### `compression` 315 316 Optionally set an explicit compression type. The default preference is to use snappy when the broker supports it, and fall back to none if not. 317 318 319 Type: `string` 320 Options: `lz4`, `snappy`, `gzip`, `none`, `zstd`. 321 322 ### `tls` 323 324 Custom TLS settings can be used to override system defaults. 325 326 327 Type: `object` 328 329 ### `tls.enabled` 330 331 Whether custom TLS settings are enabled. 332 333 334 Type: `bool` 335 Default: `false` 336 337 ### `tls.skip_cert_verify` 338 339 Whether to skip server side certificate verification. 340 341 342 Type: `bool` 343 Default: `false` 344 345 ### `tls.enable_renegotiation` 346 347 Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you're seeing the error message `local error: tls: no renegotiation`. 348 349 350 Type: `bool` 351 Default: `false` 352 Requires version 3.45.0 or newer 353 354 ### `tls.root_cas` 355 356 An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 357 358 359 Type: `string` 360 Default: `""` 361 362 ```yaml 363 # Examples 364 365 root_cas: |- 366 -----BEGIN CERTIFICATE----- 367 ... 368 -----END CERTIFICATE----- 369 ``` 370 371 ### `tls.root_cas_file` 372 373 An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 374 375 376 Type: `string` 377 Default: `""` 378 379 ```yaml 380 # Examples 381 382 root_cas_file: ./root_cas.pem 383 ``` 384 385 ### `tls.client_certs` 386 387 A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. 388 389 390 Type: `array` 391 392 ```yaml 393 # Examples 394 395 client_certs: 396 - cert: foo 397 key: bar 398 399 client_certs: 400 - cert_file: ./example.pem 401 key_file: ./example.key 402 ``` 403 404 ### `tls.client_certs[].cert` 405 406 A plain text certificate to use. 407 408 409 Type: `string` 410 Default: `""` 411 412 ### `tls.client_certs[].key` 413 414 A plain text certificate key to use. 415 416 417 Type: `string` 418 Default: `""` 419 420 ### `tls.client_certs[].cert_file` 421 422 The path to a certificate to use. 423 424 425 Type: `string` 426 Default: `""` 427 428 ### `tls.client_certs[].key_file` 429 430 The path of a certificate key to use. 431 432 433 Type: `string` 434 Default: `""` 435 436 ### `sasl` 437 438 Specify one or more methods of SASL authentication. SASL is tried in order; if the broker supports the first mechanism, all connections will use that mechanism. If the first mechanism fails, the client will pick the first supported mechanism. If the broker does not support any client mechanisms, connections will fail. 439 440 441 Type: `array` 442 443 ```yaml 444 # Examples 445 446 sasl: 447 - mechanism: SCRAM-SHA-512 448 password: bar 449 username: foo 450 ``` 451 452 ### `sasl[].mechanism` 453 454 The SASL mechanism to use. 455 456 457 Type: `string` 458 459 | Option | Summary | 460 |---|---| 461 | `OAUTHBEARER` | OAuth Bearer based authentication. | 462 | `PLAIN` | Plain text authentication. | 463 | `SCRAM-SHA-256` | SCRAM based authentication as specified in RFC5802. | 464 | `SCRAM-SHA-512` | SCRAM based authentication as specified in RFC5802. | 465 466 467 ### `sasl[].username` 468 469 A username to provide for PLAIN or SCRAM-* authentication. 470 471 472 Type: `string` 473 Default: `""` 474 475 ### `sasl[].password` 476 477 A password to provide for PLAIN or SCRAM-* authentication. 478 479 480 Type: `string` 481 Default: `""` 482 483 ### `sasl[].token` 484 485 The token to use for a single session's OAUTHBEARER authentication. 486 487 488 Type: `string` 489 Default: `""` 490 491 ### `sasl[].extensions` 492 493 Key/value pairs to add to OAUTHBEARER authentication requests. 494 495 496 Type: `object` 497 498