github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/kafka.md (about) 1 --- 2 title: kafka 3 type: output 4 status: stable 5 categories: ["Services"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/kafka.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 19 The kafka output type writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating it back to the input. 20 21 22 <Tabs defaultValue="common" values={[ 23 { label: 'Common', value: 'common', }, 24 { label: 'Advanced', value: 'advanced', }, 25 ]}> 26 27 <TabItem value="common"> 28 29 ```yaml 30 # Common config fields, showing default values 31 output: 32 label: "" 33 kafka: 34 addresses: 35 - localhost:9092 36 topic: benthos_stream 37 client_id: benthos_kafka_output 38 target_version: 1.0.0 39 key: "" 40 partitioner: fnv1a_hash 41 compression: none 42 static_headers: {} 43 metadata: 44 exclude_prefixes: [] 45 max_in_flight: 1 46 batching: 47 count: 0 48 byte_size: 0 49 period: "" 50 check: "" 51 ``` 52 53 </TabItem> 54 <TabItem value="advanced"> 55 56 ```yaml 57 # All config fields, showing default values 58 output: 59 label: "" 60 kafka: 61 addresses: 62 - localhost:9092 63 tls: 64 enabled: false 65 skip_cert_verify: false 66 enable_renegotiation: false 67 root_cas: "" 68 root_cas_file: "" 69 client_certs: [] 70 sasl: 71 mechanism: "" 72 user: "" 73 password: "" 74 access_token: "" 75 token_cache: "" 76 token_key: "" 77 topic: benthos_stream 78 client_id: benthos_kafka_output 79 target_version: 1.0.0 80 rack_id: "" 81 key: "" 82 partitioner: fnv1a_hash 83 partition: "" 84 compression: none 85 static_headers: {} 86 metadata: 87 exclude_prefixes: [] 88 inject_tracing_map: "" 89 max_in_flight: 1 90 ack_replicas: false 91 max_msg_bytes: 1000000 92 timeout: 5s 93 retry_as_batch: false 94 batching: 95 count: 0 96 byte_size: 0 97 period: "" 98 check: "" 99 processors: [] 100 max_retries: 0 101 backoff: 102 initial_interval: 3s 103 max_interval: 10s 104 max_elapsed_time: 30s 105 ``` 106 107 </TabItem> 108 </Tabs> 109 110 The config field `ack_replicas` determines whether we wait for acknowledgement from all replicas or just a single broker. 111 112 Both the `key` and `topic` fields can be dynamically set using function interpolations described [here](/docs/configuration/interpolation#bloblang-queries). 113 114 [Metadata](/docs/configuration/metadata) will be added to each message sent as headers (version 0.11+), but can be restricted using the field [`metadata`](#metadata). 115 116 ### Strict Ordering and Retries 117 118 When strict ordering is required for messages written to topic partitions it is important to ensure that both the field `max_in_flight` is set to `1` and that the field `retry_as_batch` is set to `true`. 119 120 You must also ensure that failed batches are never rerouted back to the same output. This can be done by setting the field `max_retries` to `0` and `backoff.max_elapsed_time` to empty, which will apply back pressure indefinitely until the batch is sent successfully. 121 122 However, this also means that manual intervention will eventually be required in cases where the batch cannot be sent due to configuration problems such as an incorrect `max_msg_bytes` estimate. A less strict but automated alternative would be to route failed batches to a dead letter queue using a [`fallback` broker](/docs/components/outputs/fallback), but this would allow subsequent batches to be delivered in the meantime whilst those failed batches are dealt with. 123 124 ### Troubleshooting 125 126 If you're seeing issues writing to or reading from Kafka with this component then it's worth trying out the newer [`kafka_franz` output](/docs/components/outputs/kafka_franz). 127 128 - I'm seeing logs that report `Failed to connect to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`, but the brokers are definitely reachable. 129 130 Unfortunately this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double check your authentication configuration and also ensure that you have [enabled TLS](#tlsenabled) if applicable. 131 132 ## Performance 133 134 This output benefits from sending multiple messages in flight in parallel for 135 improved performance. You can tune the max number of in flight messages with the 136 field `max_in_flight`. 137 138 This output benefits from sending messages as a batch for improved performance. 139 Batches can be formed at both the input and output level. You can find out more 140 [in this doc](/docs/configuration/batching). 141 142 ## Fields 143 144 ### `addresses` 145 146 A list of broker addresses to connect to. If an item of the list contains commas it will be expanded into multiple addresses. 147 148 149 Type: `array` 150 Default: `["localhost:9092"]` 151 152 ```yaml 153 # Examples 154 155 addresses: 156 - localhost:9092 157 158 addresses: 159 - localhost:9041,localhost:9042 160 161 addresses: 162 - localhost:9041 163 - localhost:9042 164 ``` 165 166 ### `tls` 167 168 Custom TLS settings can be used to override system defaults. 169 170 171 Type: `object` 172 173 ### `tls.enabled` 174 175 Whether custom TLS settings are enabled. 176 177 178 Type: `bool` 179 Default: `false` 180 181 ### `tls.skip_cert_verify` 182 183 Whether to skip server side certificate verification. 184 185 186 Type: `bool` 187 Default: `false` 188 189 ### `tls.enable_renegotiation` 190 191 Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you're seeing the error message `local error: tls: no renegotiation`. 192 193 194 Type: `bool` 195 Default: `false` 196 Requires version 3.45.0 or newer 197 198 ### `tls.root_cas` 199 200 An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 201 202 203 Type: `string` 204 Default: `""` 205 206 ```yaml 207 # Examples 208 209 root_cas: |- 210 -----BEGIN CERTIFICATE----- 211 ... 212 -----END CERTIFICATE----- 213 ``` 214 215 ### `tls.root_cas_file` 216 217 An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 218 219 220 Type: `string` 221 Default: `""` 222 223 ```yaml 224 # Examples 225 226 root_cas_file: ./root_cas.pem 227 ``` 228 229 ### `tls.client_certs` 230 231 A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. 232 233 234 Type: `array` 235 Default: `[]` 236 237 ```yaml 238 # Examples 239 240 client_certs: 241 - cert: foo 242 key: bar 243 244 client_certs: 245 - cert_file: ./example.pem 246 key_file: ./example.key 247 ``` 248 249 ### `tls.client_certs[].cert` 250 251 A plain text certificate to use. 252 253 254 Type: `string` 255 Default: `""` 256 257 ### `tls.client_certs[].key` 258 259 A plain text certificate key to use. 260 261 262 Type: `string` 263 Default: `""` 264 265 ### `tls.client_certs[].cert_file` 266 267 The path to a certificate to use. 268 269 270 Type: `string` 271 Default: `""` 272 273 ### `tls.client_certs[].key_file` 274 275 The path of a certificate key to use. 276 277 278 Type: `string` 279 Default: `""` 280 281 ### `sasl` 282 283 Enables SASL authentication. 284 285 286 Type: `object` 287 288 ### `sasl.mechanism` 289 290 The SASL authentication mechanism, if left empty SASL authentication is not used. Warning: SCRAM based methods within Benthos have not received a security audit. 291 292 293 Type: `string` 294 Default: `""` 295 296 | Option | Summary | 297 |---|---| 298 | `PLAIN` | Plain text authentication. NOTE: When using plain text auth it is extremely likely that you'll also need to [enable TLS](#tlsenabled). | 299 | `OAUTHBEARER` | OAuth Bearer based authentication. | 300 | `SCRAM-SHA-256` | Authentication using the SCRAM-SHA-256 mechanism. | 301 | `SCRAM-SHA-512` | Authentication using the SCRAM-SHA-512 mechanism. | 302 303 304 ### `sasl.user` 305 306 A `PLAIN` username. It is recommended that you use environment variables to populate this field. 307 308 309 Type: `string` 310 Default: `""` 311 312 ```yaml 313 # Examples 314 315 user: ${USER} 316 ``` 317 318 ### `sasl.password` 319 320 A `PLAIN` password. It is recommended that you use environment variables to populate this field. 321 322 323 Type: `string` 324 Default: `""` 325 326 ```yaml 327 # Examples 328 329 password: ${PASSWORD} 330 ``` 331 332 ### `sasl.access_token` 333 334 A static `OAUTHBEARER` access token 335 336 337 Type: `string` 338 Default: `""` 339 340 ### `sasl.token_cache` 341 342 Instead of using a static `access_token` allows you to query a [`cache`](/docs/components/caches/about) resource to fetch `OAUTHBEARER` tokens from 343 344 345 Type: `string` 346 Default: `""` 347 348 ### `sasl.token_key` 349 350 Required when using a `token_cache`, the key to query the cache with for tokens. 351 352 353 Type: `string` 354 Default: `""` 355 356 ### `topic` 357 358 The topic to publish messages to. 359 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 360 361 362 Type: `string` 363 Default: `"benthos_stream"` 364 365 ### `client_id` 366 367 An identifier for the client connection. 368 369 370 Type: `string` 371 Default: `"benthos_kafka_output"` 372 373 ### `target_version` 374 375 The version of the Kafka protocol to use. This limits the capabilities used by the client and should ideally match the version of your brokers. 376 377 378 Type: `string` 379 Default: `"1.0.0"` 380 381 ### `rack_id` 382 383 A rack identifier for this client. 384 385 386 Type: `string` 387 Default: `""` 388 389 ### `key` 390 391 The key to publish messages with. 392 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 393 394 395 Type: `string` 396 Default: `""` 397 398 ### `partitioner` 399 400 The partitioning algorithm to use. 401 402 403 Type: `string` 404 Default: `"fnv1a_hash"` 405 Options: `fnv1a_hash`, `murmur2_hash`, `random`, `round_robin`, `manual`. 406 407 ### `partition` 408 409 The manually-specified partition to publish messages to, relevant only when the field `partitioner` is set to `manual`. Must be able to parse as a 32-bit integer. 410 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 411 412 413 Type: `string` 414 Default: `""` 415 416 ### `compression` 417 418 The compression algorithm to use. 419 420 421 Type: `string` 422 Default: `"none"` 423 Options: `none`, `snappy`, `lz4`, `gzip`, `zstd`. 424 425 ### `static_headers` 426 427 An optional map of static headers that should be added to messages in addition to metadata. 428 429 430 Type: `object` 431 Default: `{}` 432 433 ```yaml 434 # Examples 435 436 static_headers: 437 first-static-header: value-1 438 second-static-header: value-2 439 ``` 440 441 ### `metadata` 442 443 Specify criteria for which metadata values are sent with messages as headers. 444 445 446 Type: `object` 447 448 ### `metadata.exclude_prefixes` 449 450 Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. 451 452 453 Type: `array` 454 Default: `[]` 455 456 ### `inject_tracing_map` 457 458 EXPERIMENTAL: A [Bloblang mapping](/docs/guides/bloblang/about) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. 459 460 461 Type: `string` 462 Default: `""` 463 Requires version 3.45.0 or newer 464 465 ```yaml 466 # Examples 467 468 inject_tracing_map: meta = meta().merge(this) 469 470 inject_tracing_map: root.meta.span = this 471 ``` 472 473 ### `max_in_flight` 474 475 The maximum number of parallel message batches to have in flight at any given time. 476 477 478 Type: `int` 479 Default: `1` 480 481 ### `ack_replicas` 482 483 Ensure that messages have been copied across all replicas before acknowledging receipt. 484 485 486 Type: `bool` 487 Default: `false` 488 489 ### `max_msg_bytes` 490 491 The maximum size in bytes of messages sent to the target topic. 492 493 494 Type: `int` 495 Default: `1000000` 496 497 ### `timeout` 498 499 The maximum period of time to wait for message sends before abandoning the request and retrying. 500 501 502 Type: `string` 503 Default: `"5s"` 504 505 ### `retry_as_batch` 506 507 When enabled forces an entire batch of messages to be retried if any individual message fails on a send, otherwise only the individual messages that failed are retried. Disabling this helps to reduce message duplicates during intermittent errors, but also makes it impossible to guarantee strict ordering of messages. 508 509 510 Type: `bool` 511 Default: `false` 512 513 ### `batching` 514 515 Allows you to configure a [batching policy](/docs/configuration/batching). 516 517 518 Type: `object` 519 520 ```yaml 521 # Examples 522 523 batching: 524 byte_size: 5000 525 count: 0 526 period: 1s 527 528 batching: 529 count: 10 530 period: 1s 531 532 batching: 533 check: this.contains("END BATCH") 534 count: 0 535 period: 1m 536 ``` 537 538 ### `batching.count` 539 540 A number of messages at which the batch should be flushed. If `0` disables count based batching. 541 542 543 Type: `int` 544 Default: `0` 545 546 ### `batching.byte_size` 547 548 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 549 550 551 Type: `int` 552 Default: `0` 553 554 ### `batching.period` 555 556 A period in which an incomplete batch should be flushed regardless of its size. 557 558 559 Type: `string` 560 Default: `""` 561 562 ```yaml 563 # Examples 564 565 period: 1s 566 567 period: 1m 568 569 period: 500ms 570 ``` 571 572 ### `batching.check` 573 574 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 575 576 577 Type: `string` 578 Default: `""` 579 580 ```yaml 581 # Examples 582 583 check: this.type == "end_of_transaction" 584 ``` 585 586 ### `batching.processors` 587 588 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 589 590 591 Type: `array` 592 Default: `[]` 593 594 ```yaml 595 # Examples 596 597 processors: 598 - archive: 599 format: lines 600 601 processors: 602 - archive: 603 format: json_array 604 605 processors: 606 - merge_json: {} 607 ``` 608 609 ### `max_retries` 610 611 The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. 612 613 614 Type: `int` 615 Default: `0` 616 617 ### `backoff` 618 619 Control time intervals between retry attempts. 620 621 622 Type: `object` 623 624 ### `backoff.initial_interval` 625 626 The initial period to wait between retry attempts. 627 628 629 Type: `string` 630 Default: `"3s"` 631 632 ### `backoff.max_interval` 633 634 The maximum period to wait between retry attempts. 635 636 637 Type: `string` 638 Default: `"10s"` 639 640 ### `backoff.max_elapsed_time` 641 642 The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. 643 644 645 Type: `string` 646 Default: `"30s"` 647 648