github.com/Jeffail/benthos/v3@v3.65.0/CHANGELOG.md (about) 1 Changelog 2 ========= 3 4 All notable changes to this project will be documented in this file. 5 6 ## Unreleased 7 8 ## 3.65.0 - 2022-03-07 9 10 ### Added 11 12 - New `sql_raw` processor and output. 13 14 ### Fixed 15 16 - Corrected a case where nested `parallel` processors that result in emptied batches (all messages filtered) would propagate an unack rather than an acknowledgement. 17 18 ### Changed 19 20 - The `sql` processor and output are no longer marked as deprecated and will therefore not be removed in V4. This change was made in order to provide more time to migrate to the new `sql_raw` processor and output. 21 22 ## 3.64.0 - 2022-02-23 23 24 ### Added 25 26 - Field `nack_reject_patterns` added to the `amqp_0_9` input. 27 - New experimental `mongodb` input. 28 - Field `cast` added to the `xml` processor and `parse_xml` bloblang method. 29 - New experimental `gcp_bigquery_select` processor. 30 - New `assign` bloblang method. 31 - The `protobuf` processor now supports `Any` fields in protobuf definitions. 32 - The `azure_queue_storage` input field `queue_name` now supports interpolation functions. 33 34 ### Fixed 35 36 - Fixed an issue where manually clearing errors within a `catch` processor would result in subsequent processors in the block being skipped. 37 - The `cassandra` output should now automatically match `float` columns. 38 - Fixed an issue where the `elasticsearch` output would collapse batched messages of matching ID rather than send as individual items. 39 - Running streams mode with `--no-api` no longer removes the `/ready` endpoint. 40 41 ### Changed 42 43 - The `throttle` processor has now been marked as deprecated. 44 45 ## 3.63.0 - 2022-02-08 46 47 ### Added 48 49 - Field `cors` added to the `http_server` input and output, for supporting CORS requests when custom servers are used. 50 - Field `server_side_encryption` added to the `aws_s3` output. 51 - Field `use_histogram_timing` and `histogram_buckets` added to the `prometheus` metrics exporter. 52 - New duration string and back off field types added to plugin config builders. 53 - Experimental field `multipart` added to the `http_client` output. 54 - Codec `regex` added to inputs. 55 - Field `timeout` added to the `cassandra` output. 56 - New experimental `gcp_bigquery_select` input. 57 - Field `ack_wait` added to the `nats_jetstream` input. 58 59 ### Changed 60 61 - The old map-style resource config fields (`resources.processors.<name>`, etc) are now marked as deprecated. Use the newer list based fields (`processor_resources`, etc) instead. 62 63 ### Fixed 64 65 - The `generate` input now supports zeroed duration strings (`0s`, etc) for unbounded document creation. 66 - The `aws_dynamodb_partiql` processor no longer ignores the `endpoint` field. 67 - Corrected duplicate detection for custom cache implementations. 68 - Fixed panic caused by invalid bounds in the `range` function. 69 - Resource config files imported now allow (and ignore) a `tests` field. 70 - Fixed an issue where the `aws_kinesis` input would fail to back off during unyielding read attempts. 71 - Fixed a linting error with `zmq4` input/output `urls` fields that was incorrectly expecting a string. 72 73 ## 3.62.0 - 2022-01-21 74 75 ### Added 76 77 - Field `sync` added to the `gcp_pubsub` input. 78 - New input, processor, and output config field types added to the plugin APIs. 79 - Added new experimental `parquet` processor. 80 - New Bloblang method `format_json`. 81 - Field `collection` in `mongodb` processor and output now supports interpolation functions. 82 - Field `output_raw` added to the `jq` processor. 83 - The lambda distribution now supports a `BENTHOS_CONFIG_PATH` environment variable for specifying a custom config path. 84 - Field `metadata` added to `http` and `http_client` components. 85 - Field `ordering_key` added to the `gcp_pubsub` output. 86 - A suite of new experimental `geoip_` methods have been added. 87 - Added flag `--deprecated` to the `benthos lint` subcommand for detecting deprecated fields. 88 89 ### Changed 90 91 - The `sql` processor and output have been marked deprecated in favour of the newer `sql_insert`, `sql_select` alternatives. 92 93 ### Fixed 94 95 - The input codec `chunked` is no longer capped by the packet size of the incoming streams. 96 - The `schema_registry_decode` and `schema_registry_encode` processors now honour trailing slashes in the `url` field. 97 - Processors configured within `pipeline.processors` now share processors across threads rather than clone them. 98 - Go API: Errors returned from input/output plugin `Close` methods no longer cause shutdown to block. 99 - The `pulsar` output should now follow authentication configuration. 100 - Fixed an issue where the `aws_sqs` output might occasionally retry a failed message send with an invalid empty message body. 101 102 ## 3.61.0 - 2021-12-28 103 104 ### Added 105 106 - Field `json_marshal_mode` added to the MongoDB processor. 107 - Fields `extract_headers.include_prefixes` and `extract_headers.include_patterns` added to the `http_client` input and output and to the `http` processor. 108 - Fields `sync_response.metadata_headers.include_prefixes` and `sync_response.metadata_headers.include_patterns` added to the `http_server` input. 109 - The `http_client` input and output and the `http` processor field `copy_response_headers` has been deprecated in favour of the `extract_headers` functionality. 110 - Added new cli flag `--no-api` for the `streams` subcommand to disable the REST API. 111 - New experimental `kafka_franz` input and output. 112 - Added new Bloblang function `ksuid`. 113 - All `codec` input fields now support custom csv delimiters. 114 115 ### Fixed 116 117 - Streams mode paths now resolve glob patterns in all cases. 118 - Prevented the `nats` input from error logging when acknowledgments can't be fulfilled due to the lack of message replies. 119 - Fixed an issue where GCP inputs and outputs could terminate requests early due to a cancelled client context. 120 - Prevented more parsing errors in Bloblang mappings with windows style line endings. 121 122 ## 3.60.1 - 2021-12-03 123 124 ### Fixed 125 126 - Fixed an issue where the `mongodb` output would incorrectly report upsert not allowed on valid operators. 127 128 ## 3.60.0 - 2021-12-01 129 130 ### Added 131 132 - The `pulsar` input and output now support `oauth2` and `token` authentication mechanisms. 133 - The `pulsar` input now enriches messages with more metadata. 134 - Fields `message_group_id`, `message_deduplication_id`, and `metadata` added to the `aws_sns` output. 135 - Field `upsert` added to the `mongodb` processor and output. 136 137 ### Fixed 138 139 - The `schema_registry_encode` and `schema_registry_decode` processors now honour path prefixes included in the `url` field. 140 - The `mqtt` input and output `keepalive` field is now interpreted as seconds, previously it was being erroneously interpreted as nanoseconds. 141 - The header `Content-Type` in the field `http_server.sync_response.headers` is now detected in a case insensitive way when populating multipart message encoding types. 142 - The `nats_jetstream` input and outputs should now honour `auth.*` config fields. 143 144 ## 3.59.0 - 2021-11-22 145 146 ### Added 147 148 - New Bloblang method `parse_duration_iso8601` for parsing ISO-8601 duration strings into an integer. 149 - The `nats` input now supports metadata from headers when supported. 150 - Field `headers` added to the `nats` output. 151 - Go API: Optional field definitions added for config specs. 152 - New (experimental) `sql_select` input. 153 - New (experimental) `sql_select` and `sql_insert` processors, which will supersede the existing `sql` processor. 154 - New (experimental) `sql_insert` output, which will supersede the existing `sql` output. 155 - Field `retained_interpolated` added to the `mqtt` output. 156 - Bloblang now allows optional carriage returns before line feeds at line endings. 157 - New CLI flag `-w`/`-watcher` added for automatically detecting and applying configuration file changes. 158 - Field `avro_raw_json` added to the `schema_registry_encode` processor. 159 - New (experimental) `msgpack` processor. 160 - New `parse_msgpack` and `format_msgpack` Bloblang methods. 161 162 ### Fixed 163 164 - Fixed an issue where the `azure_table_storage` output would attempt to send >100 size batches (and fail). 165 - Fixed an issue in the `subprocess` input where saturated stdout streams could become corrupted. 166 167 ## 3.58.0 - 2021-11-02 168 169 ### Added 170 171 - `amqp_0_9` components now support TLS EXTERNAL auth. 172 - Field `urls` added to the `amqp_0_9` input and output. 173 - New experimental `schema_registry_encode` processor. 174 - Field `write_timeout` added to the `mqtt` output, and field `connect_timeout` added to both the input and output. 175 - The `websocket` input and output now support custom `tls` configuration. 176 - New output broker type `fallback` added as a drop-in replacement for the now deprecated `try` broker. 177 178 ### Fixed 179 180 - Removed a performance bottleneck when consuming a large quantity of small files with the `file` input. 181 182 ## 3.57.0 - 2021-10-14 183 184 ### Added 185 186 - Go API: New config field types `StringMap`, `IntList`, and `IntMap`. 187 - The `http_client` input, output and processor now include the response body in request error logs for more context. 188 - Field `dynamic_client_id_suffix` added to the `mqtt` input and output. 189 190 ### Fixed 191 192 - Corrected an issue where the `sftp` input could consume duplicate documents before shutting down when ran in batch mode. 193 194 ## 3.56.0 - 2021-09-22 195 196 ### Added 197 198 - Fields `cache_control`, `content_disposition`, `content_language` and `website_redirect_location` added to the `aws_s3` output. 199 - Field `cors.enabled` and `cors.allowed_origins` added to the server wide `http` config. 200 - For Kafka components the config now supports the `rack_id` field which may contain a rack identifier for the Kafka client. 201 - Allow mapping imports in Bloblang environments to be disabled. 202 - Go API: Isolated Bloblang environments are now honored by all components. 203 - Go API: The stream builder now evaluates environment variable interpolations. 204 - Field `unsafe_dynamic_query` added to the `sql` processor. 205 - The `kafka` output now supports `zstd` compression. 206 207 ### Fixed 208 209 - The `test` subcommand now expands resource glob patterns (`benthos -r "./foo/*.yaml" test ./...`). 210 - The Bloblang equality operator now returns `false` when comparing non-null values with `null` rather than a mismatched types error. 211 212 ## 3.55.0 - 2021-09-08 213 214 ### Added 215 216 - New experimental `gcp_bigquery` output. 217 - Go API: It's now possible to parse a config spec directly with `ParseYAML`. 218 - Bloblang methods and functions now support named parameters. 219 - Field `args_mapping` added to the `cassandra` output. 220 - For NATS, NATS Streaming and Jetstream components the config now supports specifying either `nkey_file` or `user_credentials_file` to configure authentication. 221 222 ## 3.54.0 - 2021-09-01 223 224 ### Added 225 226 - The `mqtt` input and output now support sending a last will, configuring a keep alive timeout, and setting retained out output messages. 227 - Go API: New stream builder `AddBatchProducerFunc` and `AddBatchConsumerFunc` methods. 228 - Field `gzip_compression` added to the `elasticsearch` output. 229 - The `redis_streams` input now supports creating the stream with the `MKSTREAM` command (enabled by default). 230 - The `kafka` output now supports manual partition allocation using interpolation functions in the field `partition`. 231 232 ### Fixed 233 234 - The bloblang method `contains` now correctly compares numerical values in arrays and objects. 235 236 ## 3.53.0 - 2021-08-19 237 238 ### Added 239 240 - Go API: Added ability to create and register `BatchBuffer` plugins. 241 - New `system_window` buffer for processing message windows (sliding or tumbling) following the system clock. 242 - Field `root_cas` added to all TLS configuration blocks. 243 - The `sftp` input and output now support key based authentication. 244 - New Bloblang function `nanoid`. 245 - The `gcp_cloud_storage` output now supports custom collision behaviour with the field `collision_mode`. 246 - Field `priority` added to the `amqp_0_9` output. 247 - Operator `keys` added to the `redis` processor. 248 - The `http_client` input when configured in stream mode now allows message body interpolation functions within the URL and header parameters. 249 250 ### Fixed 251 252 - Fixed a panic that would occur when executing a pipeline where processor or input resources reference rate limits. 253 254 ## 3.52.0 - 2021-08-02 255 256 ### Added 257 258 - The `elasticsearch` output now supports delete, update and index operations. 259 - Go API: Added ability to create and register `BatchInput` plugins. 260 261 ### Fixed 262 263 - Prevented the `http_server` input from blocking graceful pipeline termination indefinitely. 264 - Removed annoying nil error log from HTTP client components when parsing responses. 265 266 ## 3.51.0 - 2021-07-26 267 268 ### Added 269 270 - The `redis_streams`, `redis_pubsub` and `redis_list` outputs now all support batching for higher throughput. 271 - The `amqp_1` input and output now support passing and receiving metadata as annotations. 272 - Config unit test definitions can now use files for both the input and expected output. 273 - Field `track_properties` added to the `azure_queue_storage` input for enriching messages with properties such as the message backlog. 274 - Go API: The new plugin APIs, available at `./public/service`, are considered stable. 275 - The streams mode API now uses the setting `http.read_timeout` for timing out stream CRUD endpoints. 276 277 ### Fixed 278 279 - The Bloblang function `random_int` now only resolves dynamic arguments once during the lifetime of the mapping. Documentation has been updated in order to clarify the behaviour with dynamic arguments. 280 - Fixed an issue where plugins registered would return `failed to obtain docs for X type Y` linting errors. 281 - HTTP client components are now more permissive regarding invalid Content-Type headers. 282 283 ## 3.50.0 - 2021-07-19 284 285 ### Added 286 287 - New CLI flag `--set` (`-s`) for overriding arbitrary fields in a config. E.g. `-s input.type=http_server` would override the config setting the input type to `http_server`. 288 - Unit test definitions now support mocking components. 289 290 ## 3.49.0 - 2021-07-12 291 292 ### Added 293 294 - The `nats` input now supports acks. 295 - The `memory` and `file` cache types now expose metrics akin to other caches. 296 297 ### Fixed 298 299 - The `switch` output when `retry_until_success` is set to `false` will now provide granular nacks to pre-batched messages. 300 - The URL printed in error messages when HTTP client components fail should now show interpolated values as they were interpreted. 301 - Go Plugins API V2: Batched processors should now show in tracing, and no longer complain about spans being closed more than once. 302 303 ## 3.48.0 - 2021-06-25 304 305 ### Added 306 307 - Algorithm `lz4` added to the `compress` and `decompress` processors. 308 - New experimental `aws_dynamodb_partiql` processor. 309 - Go Plugins API: new run opt `OptUseContext` for an extra shutdown mechanism. 310 311 ### Fixed 312 313 - Fixed an issue here the `http_client` would prematurely drop connections when configured with `stream.enabled` set to `true`. 314 - Prevented closed output brokers from leaving child outputs running when they've failed to establish a connection. 315 - Fixed metrics prefixes in streams mode for nested components. 316 317 ## 3.47.0 - 2021-06-16 318 319 ### Added 320 321 - CLI flag `max-token-length` added to the `blobl` subcommand. 322 - Go Plugins API: Plugin components can now be configured seamlessly like native components, meaning the namespace `plugin` is no longer required and configuration fields can be placed within the namespace of the plugin itself. Note that the old style (within `plugin`) is still supported. 323 - The `http_client` input fields `url` and `headers` now support interpolation functions that access metadata and contents of the last received message. 324 - Rate limit resources now emit `checked`, `limited` and `error` metrics. 325 - A new experimental plugins API is available for early adopters, and can be found at `./public/x/service`. 326 - A new experimental template system is available for early adopters, examples can be found in `./template`. 327 - New beta Bloblang method `bloblang` for executing dynamic mappings. 328 - All `http` components now support a beta `jwt` authentication mechanism. 329 - New experimental `schema_registry_decode` processor. 330 - New Bloblang method `parse_duration` for parsing duration strings into an integer. 331 - New experimental `twitter_search` input. 332 - New field `args_mapping` added to the `sql` processor and output for mapping explicitly typed arguments. 333 - Added format `csv` to the `unarchive` processor. 334 - The `redis` processor now supports `incrby` operations. 335 - New experimental `discord` input and output. 336 - The `http_server` input now adds a metadata field `http_server_verb`. 337 - New Bloblang methods `parse_yaml` and `format_yaml`. 338 - CLI flag `env-file` added to Benthos for parsing dotenv files. 339 - New `mssql` SQL driver for the `sql` processor and output. 340 - New POST endpoint `/resources/{type}/{id}` added to Benthos streams mode for dynamically mutating resource configs. 341 342 ### Changed 343 344 - Go Plugins API: The Bloblang `ArgSpec` now returns a public error type `ArgError`. 345 - Components that support glob paths (`file`, `csv`, etc) now also support super globs (double asterisk). 346 - The `aws_kinesis` input is now stable. 347 - The `gcp_cloud_storage` input and output are now beta. 348 - The `kinesis` input is now deprecated. 349 - Go Plugins API: the minimum version of Go required is now 1.16. 350 351 ### Fixed 352 353 - Fixed a rare panic caused when executing a `workflow` resource processor that references `branch` resources across parallel threads. 354 - The `mqtt` input with multiple topics now works with brokers that would previously error on multiple subscriptions. 355 - Fixed initialisation of components configured as resources that reference other resources, where under certain circumstances the components would fail to obtain a true reference to the target resource. This fix makes it so that resources are accessed only when used, which will also make it possible to introduce dynamic resources in future. 356 - The streams mode endpoint `/streams/{id}/stats` should now work again provided the default manager is used. 357 358 ## 3.46.1 - 2021-05-19 359 360 ### Fixed 361 362 - The `branch` processor now writes error logs when the request or result map fails. 363 - The `branch` processor (and `workflow` by proxy) now allow errors to be mapped into the branch using `error()` in the `request_map`. 364 - Added a linting rule that warns against having a `reject` output under a `switch` broker without `retry_until_success` disabled. 365 - Prevented a panic or variable corruption that could occur when a Bloblang mapping is executed by parallel threads. 366 367 ## 3.46.0 - 2021-05-06 368 369 ### Added 370 371 - The `create` subcommand now supports a `--small`/`-s` flag that reduces the output down to only core components and common fields. 372 - Go Plugins API: Added method `Overlay` to the public Bloblang package. 373 - The `http_server` input now adds path parameters (`/{foo}/{bar}`) to the metadata of ingested messages. 374 - The `stdout` output now has a `codec` field. 375 - New Bloblang methods `format_timestamp_strftime` and `parse_timestamp_strptime`. 376 - New experimental `nats_jetstream` input and output. 377 378 ### Fixed 379 380 - Go Plugins API: Bloblang method and function plugins now automatically resolve dynamic arguments. 381 382 ## 3.45.1 - 2021-04-27 383 384 ### Fixed 385 386 - Fixed a regression where the `http_client` input with an empty `payload` would crash with a `url` containing interpolation functions. 387 - Broker output types (`broker`, `try`, `switch`) now automatically match the highest `max_in_flight` of their children. The field `max_in_flight` can still be manually set in order to enforce a minimum value for when inference isn't possible, such as with dynamic output resources. 388 389 ## 3.45.0 - 2021-04-23 390 391 ### Added 392 393 - Experimental `azure_renew_lock` field added to the `amqp_1` input. 394 - New beta `root_meta` function. 395 - Field `dequeue_visibility_timeout` added to the `azure_queue_storage` input. 396 - Field `max_in_flight` added to the `azure_queue_storage` output. 397 - New beta Bloblang methods `format_timestamp_unix` and `format_timestamp_unix_nano`. 398 - New Bloblang methods `reverse` and `index_of`. 399 - Experimental `extract_tracing_map` field added to the `kafka` input. 400 - Experimental `inject_tracing_map` field added to the `kafka` output. 401 - Field `oauth2.scopes` added to HTTP components. 402 - The `mqtt` input and output now support TLS. 403 - Field `enable_renegotiation` added to `tls` configurations. 404 - Bloblang `if` expressions now support an arbitrary number of `else if` blocks. 405 406 ### Fixed 407 408 - The `checkpoint_limit` field for the `kafka` input now works according to explicit messages in flight rather than the actual offset. This means it now works as expected with compacted topics. 409 - The `aws_kinesis` input should now automatically recover when the shard iterator has expired. 410 - Corrected an issue where messages prefixed with valid JSON documents or values were being decoded in truncated form when the remainder was invalid. 411 412 ### Changed 413 414 - The following beta components have been promoted to stable: 415 + `ristretto` cache 416 + `csv` and `generate` inputs 417 + `reject` output 418 + `branch`, `jq` and `workflow` processors 419 420 ## 3.44.1 - 2021-04-15 421 422 ### Fixed 423 424 - Fixed an issue where the `kafka` input with partition balancing wasn't committing offsets. 425 426 ## 3.44.0 - 2021-04-09 427 428 ### Added 429 430 - The `http_server` input now provides a metadata field `http_server_request_path`. 431 - New methods `sort_by` and `key_values` added to Bloblang. 432 433 ### Fixed 434 435 - Glob patterns for various components no longer resolve to bad paths in the absence of matches. 436 - Fixed an issue where acknowledgements from the `azure_queue_storage` input would timeout prematurely, resulting in duplicated message delivery. 437 - Unit test definitions no longer have implicit test cases when omitted. 438 439 ## 3.43.1 - 2021-04-05 440 441 ### Fixed 442 443 - Vastly improved Bloblang mapping errors. 444 - The `azure_blob_storage` input will now gracefully terminate if the client credentials become invalid. 445 - Prevented the experimental `gcp_cloud_storage` input from closing early during large file consumption. 446 447 ## 3.43.0 - 2021-03-31 448 449 ### New 450 451 - New (experimental) Apache Pulsar input and output. 452 - Field `codec` added to the `socket` output. 453 - New Bloblang method `map_each_key`. 454 - General config linting improvements. 455 - Bloblang mappings and interpolated fields within configs are now compile checked during linting. 456 - New output level `metadata.exclude_prefixes` config field for restricting metadata values sent to the following outputs: `kafka`, `aws_s3`, `amqp_0_9`, `redis_streams`, `aws_sqs`, `gcp_pubsub`. 457 - All NATS components now have `tls` support. 458 - Bloblang now supports context capture in query lambdas. 459 - New subcommand `benthos blobl server` that hosts a Bloblang editor web application. 460 - New (experimental) `mongodb` output, cache and processor. 461 - New (experimental) `gcp_cloud_storage` input and output. 462 - Field `batch_as_multipart` added to the `http_client` output. 463 - Inputs, outputs, processors, caches and rate limits now have a component level config field `label`, which sets the metrics and logging prefix. 464 - Resources can now be declared in the new `<component>_resources` fields at the root of config files, the old `resources.<component>s.<label>` style is still valid for backwards compatibility reasons. 465 - Bloblang mappings now support importing the entirety of a map from a path using `from "<path>"` syntax. 466 467 ### Fixed 468 469 - Corrected ack behaviour for the beta `azure_queue_storage` input. 470 - Bloblang compressed arithmetic expressions with field names (`foo+bar`) now correctly parse. 471 - Fixed throughput issues with the `aws_sqs` input. 472 - Prevented using the `root` keyword within Bloblang queries, returning an error message explaining alternative options. Eventually `root` references within queries will be fully supported and so returning clear errors messages is a temporary fix. 473 - Increased the offset commit API version used by the `kafka` input to v0.8.2 when consuming explicit partitions. 474 475 ### Changed 476 477 - Go API: Component implementations now require explicit import from `./public/components/all` in order to be invokable. This should be done automatically at all plugin and custom build entry points. If, however, you notice that your builds have begun complaining that known components do not exist then you will need to explicitly import the package with `_ "github.com/Jeffail/benthos/v3/public/components/all"`, if this is the case then please report it as an issue so that it can be dealt with. 478 479 ## 3.42.1 - 2021-03-26 480 481 ### Fixed 482 483 - Fixed a potential pipeline stall that would occur when non-batched outputs receive message batches. 484 485 ## 3.42.0 - 2021-02-22 486 487 ### New 488 489 - New `azure_queue_storage` input. 490 - All inputs with a `codec` field now support multipart. 491 - New `codec` field added to the `http_client`, `socket`, `socket_server` and `stdin` inputs. 492 - The `kafka` input now allows an empty consumer group for operating without stored offsets. 493 - The `kafka` input now supports partition ranges. 494 495 ### Fixed 496 497 - The bloblang `encode` method algorithm `ascii85` no longer returns an error when the input is misaligned. 498 499 ## 3.41.1 - 2021-02-15 500 501 ### Fixed 502 503 - The `catch` method now properly executes dynamic argument functions. 504 505 ## 3.41.0 - 2021-02-15 506 507 ### New 508 509 - New `http` fields `cert_file` and `key_file`, which when specified enforce HTTPS for the general Benthos server. 510 - Bloblang method `catch` now supports `deleted()` as an argument. 511 512 ### Fixed 513 514 - Fixed an issue with custom labels becoming stagnant with the `influxdb` metrics type. 515 - Fixed a potential unhandled error when writing to the `azure_queue_storage` output. 516 517 ## 3.40.0 - 2021-02-08 518 519 ### New 520 521 - Experimental `sharded_join` fields added to the `sequence` input. 522 - Added a new API for writing Bloblang plugins in Go at [`./public/bloblang`](https://pkg.go.dev/github.com/Jeffail/benthos/v3/public/bloblang). 523 - Field `fields_mapping` added to the `log` processor. 524 525 ### Fixed 526 527 - Prevented pre-existing errors from failing/aborting branch execution in the `branch` and `workflow` processors. 528 - Fixed `subprocess` processor message corruption with codecs `length_prefixed_uint32_be` and `netstring`. 529 530 ### Changed 531 532 - The `bloblang` input has been renamed to `generate`. This change is backwards compatible and `bloblang` will still be recognized until the next major version release. 533 - Bloblang more often preserves integer precision in arithmetic operations. 534 535 ## 3.39.0 - 2021-02-01 536 537 ### New 538 539 - Field `key` in output `redis_list` now supports interpolation functions. 540 - Field `tags` added to output `aws_s3`. 541 - New experimental `sftp` input and output. 542 - New input codec `chunker`. 543 - New field `import_paths` added to the `protobuf` processor, replaces the now deprecated `import_path` field. 544 - Added format `concatenate` to the `archive` processor. 545 546 ### Changed 547 548 - The `aws_lambda` processor now adds a metadata field `lambda_function_error` to messages when the function invocation suffers a runtime error. 549 550 ### Fixed 551 552 - Fixed an issue with the `azure_blob_storage` output where `blob_type` set to `APPEND` could result in send failures. 553 - Fixed a potential panic when shutting down a `socket_server` input with messages in flight. 554 - The `switch` processor now correctly flags errors on messages that cause a check to throw an error. 555 556 ## 3.38.0 - 2021-01-18 557 558 ### New 559 560 - New bloblang method `bytes`. 561 - The bloblang method `index` now works on byte arrays. 562 - Field `branch_resources` added to the `workflow` processor. 563 - Field `storage_sas_token` added to the `azure_blob_storage` input and output. 564 - The bloblang method `hash` and the `hash` processor now support `md5`. 565 - Field `collector_url` added to the `jaeger` tracer. 566 - The bloblang method `strip_html` now allows you to specify a list of allowed elements. 567 - New bloblang method `parse_xml`. 568 - New bloblang method `replace_many`. 569 - New bloblang methods `filepath_split` and `filepath_join`. 570 571 ### Changed 572 573 - The `cassandra` outputs `backoff.max_elapsed_time` field was unused and has been hidden from docs. 574 575 ## 3.37.0 - 2021-01-06 576 577 ### New 578 579 - Field `content_type` and `content_encoding` added to the `amqp_0_9` output. 580 - Batching fields added to the `hdfs` output. 581 - Field `codec_send` and `codec_recv` added to the `subprocess` processor. 582 - Methods `min`, `max`, `abs`, `log`, `log10` and `ceil` added to Bloblang. 583 - Added field `pattern_paths` to the `grok` processor. 584 - The `grok` processor now supports dots within field names for nested values. 585 - New `drop_on` output. 586 587 ### Fixed 588 589 - The `xml` processor now supports non UTF-8 encoding schemes. 590 591 ### Changed 592 593 - The `drop_on_error` output has been deprecated in favour of the new `drop_on` output. 594 595 ## 3.36.0 - 2020-12-24 596 597 ### New 598 599 - New `influxdb` metrics target. 600 - New `azure_blob_storage` input. 601 - New `azure_queue_storage` output. 602 - The `bloblang` input field `interval` now supports cron expressions. 603 - New beta `aws_kinesis` and `aws_sqs` inputs. 604 - The `bool` bloblang method now supports a wider range of string values. 605 - New `reject` output type for conditionally rejecting messages. 606 - All Redis components now support clustering and fail-over patterns. 607 - The `compress` and `decompress` processors now support snappy. 608 609 ### Fixed 610 611 - Fixed a panic on startup when using `if` statements within a `workflow` branch request or response map. 612 - The `meta` bloblang function error messages now include the name of the required value. 613 - Config unit tests now report processor errors when checks fail. 614 - Environment variable interpolations now allow dots within the variable name. 615 616 ### Changed 617 618 - The experimental `aws_s3` input is now marked as beta. 619 - The beta `kinesis_balanced` input is now deprecated. 620 - All Azure components have been renamed to include the prefix `azure_`, e.g. `blob_storage` is now `azure_blob_storage`. The old names can still be used for backwards compatibility. 621 - All AWS components have been renamed to include the prefix `aws_`, e.g. `s3` is now `aws_s3`. The old names can still be used for backwards compatibility. 622 623 ## 3.35.0 - 2020-12-07 624 625 ### New 626 627 - New field `retry_as_batch` added to the `kafka` output to assist in ensuring message ordering through retries. 628 - Field `delay_period` added to the experimental `aws_s3` input. 629 - Added service options for adding API middlewares and specify TLS options for plugin builds. 630 - Method `not_empty` added to Bloblang. 631 - New `bloblang` predicate type added to unit tests. 632 - Unit test case field `target_processors` now allows you to optionally specify a target file. 633 - Basic auth support added to the `prometheus` metrics pusher. 634 635 ### Changed 636 637 - Unit tests that define environment variables that are run serially (`parallel: false`) will retain those environment variables during execution, as opposed to only at config parse time. 638 - Lambda distributions now look for config files relative to the binary location, allowing you to deploy configs from the same zip as the binary. 639 640 ### Fixed 641 642 - Add `Content-Type` headers in streams API responses. 643 - Field `delete_objects` is now respected by the experimental `aws_s3` input. 644 - Fixed a case where resource processors couldn't access rate limit resources. 645 - Input files that are valid according to the codec but empty now trigger acknowledgements. 646 - Mapping `deleted()` within Bloblang object and array literals now correctly omits the values. 647 648 ## 3.34.0 - 2020-11-20 649 650 ### New 651 652 - New field `format` added to `logger` supporting `json` and `logfmt`. 653 - The `file` input now provides the metadata field `path` on payloads. 654 655 ### Fixed 656 657 - The `output.sent` metric now properly represents the number of individual messages sent even after archiving batches. 658 - Fixed a case where metric processors in streams mode pipelines and dynamic components would hang. 659 - Sync responses of >1 payloads should now get a correct rfc1341 multipart header. 660 - The `cassandra` output now correctly marshals float and double values. 661 - The `nanomsg` input with a `SUB` socket no longer attempts to set invalid timeout. 662 663 ## 3.33.0 - 2020-11-16 664 665 ### Added 666 667 - Added field `codec` to the `file` output. 668 - The `file` output now supports dynamic file paths. 669 - Added field `ttl` to the `cache` processor and output. 670 - New `sql` output, which is similar to the `sql` processor and currently supports Clickhouse, PostgreSQL and MySQL. 671 - The `kafka` input now supports multiple topics, topic partition balancing, and checkpointing. 672 - New `cassandra` output. 673 - Field `allowed_verbs` added to the `http_server` input and output. 674 - New bloblang function `now`, and method `parse_timestamp`. 675 - New bloblang methods `floor` and `round`. 676 - The bloblang method `format_timestamp` now supports strings in ISO 8601 format as well as unix epochs with decimal precision up to nanoseconds. 677 678 ## Changed 679 680 - The `files` output has been deprecated as its behaviour is now covered by `file`. 681 - The `kafka_balanced` input has now been deprecated as its functionality has been added to the `kafka` input. 682 - The `cloudwatch` metrics aggregator is now considered stable. 683 - The `sequence` input is now considered stable. 684 - The `switch` processor no longer permits cases with no processors. 685 686 ## Fixed 687 688 - Fixed the `tar` and `tar-gzip` input codecs in experimental inputs. 689 - Fixed a crash that could occur when referencing contextual fields within interpolation functions. 690 - The `noop` processor can now be inferred with an empty object (`noop: {}`). 691 - Fixed potential message corruption with the `file` input when using the `lines` codec. 692 693 ## 3.32.0 - 2020-10-29 694 695 ### Added 696 697 - The `csv` input now supports glob patterns in file paths. 698 - The `file` input now supports multiple paths, glob patterns, and a range of codecs. 699 - New experimental `aws_s3` input. 700 - All `redis` components now support TLS. 701 - The `-r` cli flag now supports glob patterns. 702 703 ### Fixed 704 705 - Bloblang literals, including method and function arguments, can now be mutated without brackets regardless of where they appear. 706 - Bloblang maps now work when running bloblang with the `blobl` subcommand. 707 708 ### Changed 709 710 - The `ristretto` cache no longer forces retries on get commands, and the retry fields have been changed in order to reflect this behaviour. 711 - The `files` input has been deprecated as its behaviour is now covered by `file`. 712 - Numbers within JSON documents are now parsed in a way that preserves precision even in cases where the number does not fit a 64-bit signed integer or float. When arithmetic is applied to those numbers (either in Bloblang or by other means) the number is converted (and precision lost) at that point based on the operation itself. 713 714 This change means that string coercion on large numbers (e.g. `root.foo = this.large_int.string()`) should now preserve the original form. However, if you are using plugins that interact with JSON message payloads you must ensure that your plugins are able to process the [`json.Number`](https://golang.org/pkg/encoding/json/#Number) type. 715 716 This change should otherwise not alter the behaviour of your configs, but if you notice odd side effects you can disable this feature by setting the environment variable `BENTHOS_USE_NUMBER` to `false` (`BENTHOS_USE_NUMBER=false benthos -c ./config.yaml`). Please [raise an issue](https://github.com/Jeffail/benthos/issues/new) if this is the case so that it can be looked into. 717 718 ## 3.31.0 - 2020-10-15 719 720 ### Added 721 722 - New input `subprocess`. 723 - New output `subprocess`. 724 - Field `auto_ack` added to the `amqp_0_9` input. 725 - Metric labels can be renamed for `prometheus` and `cloudwatch` metrics components using `path_mapping` by assigning meta fields. 726 727 ### Fixed 728 729 - Metrics labels registered using the `rename` metrics component are now sorted before registering, fixing incorrect values that could potentially be seen when renaming multiple metrics to the same name. 730 731 ## 3.30.0 - 2020-10-06 732 733 ### Added 734 735 - OAuth 2.0 using the client credentials token flow is now supported by the `http_client` input and output, and the `http` processor. 736 - Method `format_timestamp` added to Bloblang. 737 - Methods `re_find_object` and `re_find_all_object` added to Bloblang. 738 - Field `connection_string` added to the Azure `blob_storage` and `table_storage` outputs. 739 - Field `public_access_level` added to the Azure `blob_storage` output. 740 - Bloblang now supports trailing commas in object and array literals and function and method parameters. 741 742 ### Fixed 743 744 - The `amqp_1` input and output now re-establish connections to brokers on any unknown error. 745 - Batching components now more efficiently attempt a final flush of data during graceful shutdown. 746 - The `dynamic` output is now more flexible with removing outputs, and should no longer block the API as aggressively. 747 748 ## 3.29.0 - 2020-09-21 749 750 ### Added 751 752 - New cli flag `log.level` for overriding the configured logging level. 753 - New integration test suite (much more dapper and also a bit more swanky than the last). 754 755 ### Changed 756 757 - The default value for `batching.count` fields is now zero, which means adding a non-count based batching mechanism without also explicitly overriding `count` no longer incorrectly caps batches at one message. This change is backwards compatible in that working batching configs will not change in behaviour. However, a broken batching config will now behave as expected. 758 759 ### Fixed 760 761 - Improved Bloblang parser error messages for function and method parameters. 762 763 ## 3.28.0 - 2020-09-14 764 765 ### Added 766 767 - New methods `any`, `all` and `json_schema` added to Bloblang. 768 - New function `file` added to Bloblang. 769 - The `switch` output can now route batched messages individually (when using the new `cases` field). 770 - The `switch` processor now routes batched messages individually (when using the new `cases` field). 771 - The `workflow` processor can now reference resource configured `branch` processors. 772 - The `metric` processor now has a field `name` that replaces the now deprecated field `path`. When used the processor now applies to all messages of a batch and the name of the metric is now absolute, without being prefixed by a path generated based on its position within the config. 773 - New field `check` added to `group_by` processor children, which now replaces the old `condition` field. 774 - New field `check` added to `while` processor, which now replaces the old `condition` field. 775 - New field `check` added to `read_until` input, which now replaces the old `condition` field. 776 777 ### Changed 778 779 - The `bloblang` input with an interval configured now emits the first message straight away. 780 781 ## 3.27.0 - 2020-09-07 782 783 ### Added 784 785 - New function `range` added to Bloblang. 786 - New beta `jq` processor. 787 - New driver `clickhouse` added to the `sql` processor. 788 789 ### Changed 790 791 - New field `data_source_name` replaces `dsn` for the `sql` processor, and when using this field each message of a batch is processed individually. When using the field `dsn` the behaviour remains unchanged for backwards compatibility. 792 793 ### Fixed 794 795 - Eliminated situations where an `amqp_0_9` or `amqp_1` component would abandon a connection reset due to partial errors. 796 - The Bloblang parser now allows naked negation of queries. 797 - The `cache` processor interpolations for `key` and `value` now cross-batch reference messages before processing. 798 799 ## 3.26.0 - 2020-08-30 800 801 ### Added 802 803 - New Bloblang methods `not_null` and `filter`. 804 - New Bloblang function `env`. 805 - New field `path_mapping` added to all metrics types. 806 - Field `max_in_flight` added to the `dynamic` output. 807 - The `workflow` processor has been updated to use `branch` processors with the new field `branches`, these changes are backwards compatible with the now deprecated `stages` field. 808 809 ### Changed 810 811 - The `rename`, `whitelist` and `blacklist` metrics types are now deprecated, and the `path_mapping` field should be used instead. 812 - The `conditional`, `process_map` and `process_dag` processors are now deprecated and are superseded by the `switch`, `branch` and `workflow` processors respectively. 813 814 ### Fixed 815 816 - Fixed `http` processor error log messages that would print incorrect URLs. 817 - The `http_server` input now emits `latency` metrics. 818 - Fixed a panic that could occur during the shutdown of an `http_server` input serving a backlog of requests. 819 - Explicit component types (`type: foo`) are now checked by the config linter. 820 - The `amqp_1` input and output should now reconnect automatically after an unexpected link detach. 821 822 ## 3.25.0 - 2020-08-16 823 824 ### Added 825 826 - Improved parser error messages with the `blobl` subcommand. 827 - Added flag `file` to the `blobl` subcommand. 828 - New Bloblang method `parse_timestamp_unix`. 829 - New beta `protobuf` processor. 830 - New beta `branch` processor. 831 - Batching fields added to `s3` output. 832 833 ### Changed 834 835 - The `http` processor field `max_parallel` has been deprecated in favour of rate limits, and the fields within `request` have been moved to the root of the `http` namespace. This change is backwards compatible and `http.request` fields will still be recognized until the next major version release. 836 - The `process_field` processor is now deprecated, and `branch` should be used instead. 837 838 ### Fixed 839 840 - Wholesale metadata mappings (`meta = {"foo":"bar"}`) in Bloblang now correctly clear pre-existing fields. 841 842 ## 3.24.1 - 2020-08-03 843 844 ### Fixed 845 846 - Prevented an issue where batched outputs would terminate at start up. Fixes a regression introduced in v3.24.0. 847 848 ## 3.24.0 - 2020-08-02 849 850 ### Added 851 852 - Endpoint `/ready` added to streams mode API. 853 - Azure `table_storage` output now supports batched sends. 854 - All HTTP components are now able to configure a proxy URL. 855 - New `ristretto` cache. 856 - Field `shards` added to `memory` cache. 857 858 ### Fixed 859 860 - Batch error handling and retry logic has been improved for the `kafka` and `dynamodb` outputs. 861 - Bloblang now allows non-matching not-equals comparisons, allowing `foo != null` expressions. 862 863 ### Changed 864 865 - Condition `check_interpolation` has been deprecated. 866 867 ## 3.23.0 - 2020-07-26 868 869 ### Added 870 871 - Path segments in Bloblang mapping targets can now be quote-escaped. 872 - New beta `sequence` input, for sequentially chaining inputs. 873 - New beta `csv` input for consuming CSV files. 874 - New beta Azure `table_storage` output. 875 - New `parse_csv` Bloblang method. 876 - New `throw` Bloblang function. 877 - The `slice` Bloblang method now supports negative low and high arguments. 878 879 ### Fixed 880 881 - Manual `mqtt` connection handling for both the input and output. This should fix some cases where connections were dropped and never recovered. 882 - Fixed Bloblang error where calls to a `.get` method would return `null` after the first query. 883 - The `for_each` processor no longer interlaces child processors during split processing. 884 885 ## 3.22.0 - 2020-07-19 886 887 ### Added 888 889 - Added TLS fields to `elasticsearch` output. 890 - New Bloblang methods `encrypt_aes` and `decrypt_aes` added. 891 - New field `static_headers` added to the `kafka` output. 892 - New field `enabled` added to the `http` config section. 893 - Experimental CLI flag `-resources` added for specifying files containing extra resources. 894 895 ### Fixed 896 897 - The `amqp_0_9` now resolves `type` and `key` fields per message of a batch. 898 899 ## 3.21.0 - 2020-07-12 900 901 ### Added 902 903 - New beta `bloblang` input for generating documents. 904 - New beta Azure `blob_storage` output. 905 - Field `sync_response.status` added to `http_server` input. 906 - New Bloblang `errored` function. 907 908 ### Fixed 909 910 - The `json_schema` processor no longer lower cases fields within error messages. 911 - The `dynamodb` cache no longer creates warning logs for get misses. 912 913 ## 3.20.0 - 2020-07-05 914 915 ### Added 916 917 - SASL config fields added to `amqp_1` input and output. 918 - The `lint` subcommand now supports triple dot wildcard paths: `./foo/...`. 919 - The `test` subcommand now supports tests defined within the target config file being tested. 920 921 ### Fixed 922 923 - Bloblang boolean operands now short circuit. 924 925 ## 3.19.0 - 2020-06-28 926 927 ### Added 928 929 - Fields `strict_mode` and `max_in_flight` added to the `switch` output. 930 - New beta `amqp_1` input and output added. 931 932 ## 3.18.0 - 2020-06-14 933 934 ### Added 935 936 - Field `drop_empty_bodies` added to the `http_client` input. 937 938 ### Fixed 939 940 - Fixed deleting and skipping maps with the `blobl` subcommand. 941 942 ## 3.17.0 - 2020-06-07 943 944 ### Added 945 946 - New field `type` added to the `amqp_0_9` output. 947 - New bloblang methods `explode` and `without`. 948 949 ### Fixed 950 951 - Message functions such as `json` and `content` now work correctly when executing bloblang with the `blobl` sub command. 952 953 ## 3.16.0 - 2020-05-31 954 955 ### Added 956 957 - New bloblang methods `type`, `join`, `unique`, `escape_html`, `unescape_html`, `re_find_all` and `re_find_all_submatch`. 958 - Bloblang `sort` method now allows custom sorting functions. 959 - Bloblang now supports `if` expressions. 960 - Bloblang now allows joining strings with the `+` operator. 961 - Bloblang now supports multiline strings with triple quotes. 962 963 ### Changed 964 965 - The `xml` processor is now less strict with XML parsing, allowing unrecognised escape sequences to be passed through unchanged. 966 967 ### Fixed 968 969 - The bloblang method `map_each` now respects `Nothing` mapping by copying the underlying value unchanged. 970 - It's now possible to reference resource inputs and outputs in streams mode. 971 - Fixed a problem with compiling old interpolation functions with arguments containing colons (i.e. `${!timestamp_utc:2006-01-02T15:04:05.000Z}`) 972 973 ## 3.15.0 - 2020-05-24 974 975 ### Added 976 977 - Flag `log` added to `test` sub command to allow logging during tests. 978 - New subcommand `blobl` added for convenient mapping over the command line. 979 - Lots of new bloblang methods. 980 981 ### Fixed 982 983 - The `redis_streams` input no longer incorrectly copies message data into a metadata field. 984 985 ### Changed 986 987 - Bloblang is no longer considered beta. Therefore, no breaking changes will be introduced outside of a major version release. 988 989 ## 3.14.0 - 2020-05-17 990 991 ### Added 992 993 - New `ascii85` and `z85` options have been added to the `encode` and `decode` processors. 994 995 ### Bloblang BETA Changes 996 997 - The `meta` function no longer reflects changes made within the map itself. 998 - Extracting data from other messages of a batch using `from` no longer reflects changes made within a map. 999 - Meta assignments are no longer allowed within named maps. 1000 - Assigning `deleted()` to `root` now filters out a message entirely. 1001 - Lots of new methods and goodies. 1002 1003 ## 3.13.0 - 2020-05-10 1004 1005 ### Added 1006 1007 - New HMAC algorithms added to `hash` processor. 1008 - New beta `bloblang` processor. 1009 - New beta `bloblang` condition. 1010 1011 ### Fixed 1012 1013 - Prevented a crash that might occur with high-concurrent access of `http_server` metrics with labels. 1014 - The `http_client` output now respects the `copy_response_headers` field. 1015 1016 ## 3.12.0 - 2020-04-19 1017 1018 ### Added 1019 1020 - Vastly improved function interpolations, including better batch handling and arithmetic operators. 1021 - The `gcp_pubsub` output now supports function interpolation on the field `topic`. 1022 - New `contains_any` and `contains_any_cs` operators added to the `text` condition. 1023 - Support for input and output `resource` types. 1024 - The `broker` and `switch` output types now allow async messages and batching within child outputs. 1025 - Field `schema_path` added to the `avro` processor. 1026 - The `redis` cache, `redis_list` inputs and outputs now support selecting a database with the URL path. 1027 - New field `max_in_flight` added to the `broker` output. 1028 1029 ### Changed 1030 1031 - Benthos now runs in strict mode, but this can be disabled with `--chilled`. 1032 - The Benthos CLI has been revamped, the old flags are still supported but are deprecated. 1033 - The `http_server` input now accepts requests without a content-type header. 1034 1035 ### Fixed 1036 1037 - Outputs that resolve function interpolations now correctly resolve the `batch_size` function. 1038 - The `kinesis_balanced` input now correctly establishes connections. 1039 - Fixed an auth transport issue with the `gcp_pubsub` input and output. 1040 1041 ## 3.11.0 - 2020-03-08 1042 1043 ### Added 1044 1045 - Format `syslog_rfc3164` added to the `parse_log` processor. 1046 - New `multilevel` cache. 1047 - New `json_append`, `json_type` and `json_length` functions added to the `awk` processor. 1048 - New `flatten` operator added to the `json` processor. 1049 1050 ### Changed 1051 1052 - Processors that fail now set the opentracing tag `error` to `true`. 1053 1054 ### Fixed 1055 1056 - Kafka connectors now correctly set username and password for all SASL strategies. 1057 1058 ## 3.10.0 - 2020-02-05 1059 1060 ### Added 1061 1062 - Field `delete_files` added to `files` input. 1063 - TLS fields added to `nsq` input and output. 1064 - Field `processors` added to batching fields to easily accommodate aggregations and archiving of batched messages. 1065 - New `parse_log` processor. 1066 - New `json` condition. 1067 - Operators `flatten_array`, `fold_number_array` and `fold_string_array` added to `json` processor. 1068 1069 ### Changed 1070 1071 - The `redis_streams` input no longer flushes >1 fetched messages as a batch. 1072 1073 ### Fixed 1074 1075 - Re-enabled Kafka connections using SASL without TLS. 1076 1077 ## 3.9.0 - 2020-01-27 1078 1079 ### Added 1080 1081 - New `socket`, `socket_server` inputs. 1082 - New `socket` output. 1083 - Kafka connectors now support SASL using `OAUTHBEARER`, `SCRAM-SHA-256`, `SCRAM-SHA-512` mechanisms. 1084 - Experimental support for AWS CloudWatch metrics. 1085 1086 ### Changed 1087 1088 - The `tcp`, `tcp_server` and `udp_server` inputs have been deprecated and moved into the `socket` and `socket_server` inputs respectively. 1089 - The `udp` and `tcp` outputs have been deprecated and moved into the `socket` output. 1090 1091 ### Fixed 1092 1093 - The `subprocess` processor now correctly flags errors that occur. 1094 1095 ## 3.8.0 - 2020-01-17 1096 1097 ### Added 1098 1099 - New field `max_in_flight` added to the following outputs: 1100 + `amqp_0_9` 1101 + `cache` 1102 + `dynamodb` 1103 + `elasticsearch` 1104 + `gcp_pubsub` 1105 + `hdfs` 1106 + `http_client` 1107 + `kafka` 1108 + `kinesis` 1109 + `kinesis_firehose` 1110 + `mqtt` 1111 + `nanomsg` 1112 + `nats` 1113 + `nats_stream` 1114 + `nsq` 1115 + `redis_hash` 1116 + `redis_list` 1117 + `redis_pubsub` 1118 + `redis_streams` 1119 + `s3` 1120 + `sns` 1121 + `sqs` 1122 - Batching fields added to the following outputs: 1123 + `dynamodb` 1124 + `elasticsearch` 1125 + `http_client` 1126 + `kafka` 1127 + `kinesis` 1128 + `kinesis_firehose` 1129 + `sqs` 1130 - More TRACE level logs added throughout the pipeline. 1131 - Operator `delete` added to `cache` processor. 1132 - Operator `explode` added to `json` processor. 1133 - Field `storage_class` added to `s3` output. 1134 - Format `json_map` added to `unarchive` processor. 1135 1136 ### Fixed 1137 1138 - Function interpolated strings within the `json` processor `value` field are now correctly unicode escaped. 1139 - Retry intervals for `kafka` output have been tuned to prevent circuit breaker throttling. 1140 1141 ## 3.7.0 - 2019-12-21 1142 1143 ### Added 1144 1145 - New `try` output, which is a drop-in replacement for a `broker` with the `try` pattern. 1146 - Field `successful_on` added to the `http` processor. 1147 - The `statsd` metrics type now supports Datadog or InfluxDB tagging. 1148 - Field `sync_response.headers` added to `http_server` input. 1149 - New `sync_response` processor. 1150 - Field `partitioner` added to the `kafka` output. 1151 1152 ### Changed 1153 1154 - The `http` processor now gracefully handles empty responses. 1155 1156 ### Fixed 1157 1158 - The `kafka` input should now correctly recover from coordinator failures during an offset commit. 1159 - Attributes permitted by the `sqs` output should now have parity with real limitations. 1160 1161 ## 3.6.1 - 2019-12-05 1162 1163 ### Fixed 1164 1165 - Batching using an input `broker` now works with only one child input configured. 1166 - The `zmq4` input now correctly supports broker based batching. 1167 1168 ## 3.6.0 - 2019-12-03 1169 1170 ### Added 1171 1172 - New `workflow` processor. 1173 - New `resource` processor. 1174 - Processors can now be registered within the `resources` section of a config. 1175 1176 ### Changed 1177 1178 - The `mqtt` output field `topic` field now supports interpolation functions. 1179 1180 ### Fixed 1181 1182 - The `kafka` output no longer attempts to send headers on old versions of the protocol. 1183 1184 ## 3.5.0 - 2019-11-26 1185 1186 ### Added 1187 1188 - New `regexp_expand` operator added to the `text` processor. 1189 - New `json_schema` processor. 1190 1191 ## 3.4.0 - 2019-11-12 1192 1193 ### Added 1194 1195 - New `amqp_0_9` output which replaces the now deprecated `amqp` output. 1196 - The `broker` output now supports batching. 1197 1198 ### Fixed 1199 1200 - The `memory` buffer now allows parallel processing of batched payloads. 1201 - Version and date information should now be correctly displayed in archive distributions. 1202 1203 ## 3.3.1 - 2019-10-21 1204 1205 ### Fixed 1206 1207 - The `s3` input now correctly unescapes bucket keys when streaming from SQS. 1208 1209 ## 3.3.0 - 2019-10-20 1210 1211 ### Added 1212 1213 - Field `sqs_endpoint` added to the `s3` input. 1214 - Field `kms_key_id` added to the `s3` output. 1215 - Operator `delete` added to `metadata` processor. 1216 - New experimental metrics aggregator `stdout`. 1217 - Field `ack_wait` added to `nats_stream` input. 1218 - New `batching` field added to `broker` input for batching merged streams. 1219 - Field `healthcheck` added to `elasticsearch` output. 1220 - New `json_schema` condition. 1221 1222 ### Changed 1223 1224 - Experimental `kafka_cg` input has been removed. 1225 - The `kafka_balanced` inputs underlying implementation has been replaced with the `kafka_cg` one. 1226 - All inputs have been updated to automatically utilise >1 processing threads, with the exception of `kafka` and `kinesis`. 1227 1228 ## 3.2.0 - 2019-09-27 1229 1230 ### Added 1231 1232 - New `is` operator added to `text` condition. 1233 - New config unit test condition `content_matches`. 1234 - Field `init_values` added to the `memory` cache. 1235 - New `split` operator added to `json` processor. 1236 - Fields `user` and `password` added to `mqtt` input and output. 1237 - New experimental `amqp_0_9` input. 1238 1239 ### Changed 1240 1241 - Linting is now disabled for the environment var config shipped with docker images, this should prevent the log spam on start up. 1242 - Go API: Experimental `reader.Async` component methods renamed. 1243 1244 ## 3.1.1 - 2019-09-23 1245 1246 ### Fixed 1247 1248 - Prevented `kafka_cg` input lock up after batch policy period trigger with no backlog. 1249 1250 ## 3.1.0 - 2019-09-23 1251 1252 ### Added 1253 1254 - New `redis` processor. 1255 - New `kinesis_firehose` output. 1256 - New experimental `kafka_cg` input. 1257 - Go API: The `metrics.Local` aggregator now supports labels. 1258 1259 ### Fixed 1260 1261 - The `json` processor no longer removes content moved from a path to the same path. 1262 1263 ## 3.0.0 - 2019-09-17 1264 1265 This is a major version release, for more information and guidance on how to migrate please refer to [https://benthos.dev/docs/guides/migration/v3](https://www.benthos.dev/docs/guides/migration/v3). 1266 1267 ### Added 1268 1269 - The `json` processor now allows you to `move` from either a root source or to a root destination. 1270 - Added interpolation to the `metadata` processor `key` field. 1271 - Granular retry fields added to `kafka` output. 1272 1273 ### Changed 1274 1275 - Go modules are now fully supported, imports must now include the major version (e.g. `github.com/Jeffail/benthos/v3`). 1276 - Removed deprecated `mmap_file` buffer. 1277 - Removed deprecated (and undocumented) metrics paths. 1278 - Moved field `prefix` from root of `metrics` into relevant child components. 1279 - Names of `process_dag` stages must now match the regexp `[a-zA-Z0-9_-]+`. 1280 - Go API: buffer constructors now take a `types.Manager` argument in parity with other components. 1281 - JSON dot paths within the following components have been updated to allow array-based operations: 1282 + `awk` processor 1283 + `json` processor 1284 + `process_field` processor 1285 + `process_map` processor 1286 + `check_field` condition 1287 + `json_field` function interpolation 1288 + `s3` input 1289 + `dynamodb` output 1290 1291 ### Fixed 1292 1293 - The `sqs` output no longer attempts to send invalid attributes with payloads from metadata. 1294 - During graceful shutdown Benthos now scales the attempt to propagate acks for sent messages with the overall system shutdown period. 1295 1296 ## 2.15.1 - 2019-09-10 1297 1298 ### Fixed 1299 1300 - The `s3` and `sqs` inputs should now correctly log handles and codes from failed SQS message deletes and visibility timeout changes. 1301 1302 ## 2.15.0 - 2019-09-03 1303 1304 ### Added 1305 1306 - New `message_group_id` and `message_deduplication_id` fields added to `sqs` output for supporting FIFO queues. 1307 1308 ## 2.14.0 - 2019-08-29 1309 1310 ### Added 1311 1312 - Metadata field `gcp_pubsub_publish_time_unix` added to `gcp_pubsub` input. 1313 - New `tcp` and `tcp_server` inputs. 1314 - New `udp_server` input. 1315 - New `tcp` and `udp` outputs. 1316 - Metric paths `output.batch.bytes` and `output.batch.latency` added. 1317 - New `rate_limit` processor. 1318 1319 ### Fixed 1320 1321 - The `json` processor now correctly stores parsed `value` JSON when using `set` on the root path. 1322 1323 ## 2.13.0 - 2019-08-27 1324 1325 ### Added 1326 1327 - The `sqs` input now adds some message attributes as metadata. 1328 - Added field `delete_message` to `sqs` input. 1329 - The `sqs` output now sends metadata as message attributes. 1330 - New `batch_policy` field added to `memory` buffer. 1331 - New `xml` processor. 1332 1333 ### Fixed 1334 1335 - The `prometheus` metrics exporter adds quantiles back to timing metrics. 1336 1337 ## 2.12.2 - 2019-08-19 1338 1339 ### Fixed 1340 1341 - Capped slices from lines reader are now enforced. 1342 - The `json` processor now correctly honours a `null` value. 1343 1344 ## 2.12.1 - 2019-08-16 1345 1346 ### Changed 1347 1348 - Disabled `kinesis_balanced` input for WASM builds. 1349 1350 ## 2.12.0 - 2019-08-16 1351 1352 ### Added 1353 1354 - Field `codec` added to `process_field` processor. 1355 - Removed experimental status from sync responses components, which are now considered stable. 1356 - Field `pattern_definitions` added to `grok` processor. 1357 1358 ### Changed 1359 1360 - Simplified serverless lambda main function body for improving plugin documentation. 1361 1362 ### Fixed 1363 1364 - Fixed a bug where the `prepend` and `append` operators of the `text` processor could result in invalid messages when consuming line-based inputs. 1365 1366 ## 2.11.2 - 2019-08-06 1367 1368 ### Added 1369 1370 - Field `clean_session` added to `mqtt` input. 1371 - The `http_server` input now adds request query parameters to messages as metadata. 1372 1373 ## 2.11.1 - 2019-08-05 1374 1375 ### Fixed 1376 1377 - Prevent concurrent access race condition on nested parallel `process_map` processors. 1378 1379 ## 2.11.0 - 2019-08-03 1380 1381 ### Added 1382 1383 - New beta input `kinesis_balanced`. 1384 - Field `profile` added to AWS components credentials config. 1385 1386 ## 2.10.0 - 2019-07-29 1387 1388 ### Added 1389 1390 - Improved error messages attached to payloads that fail `process_dag`. post mappings. 1391 - New `redis_hash` output. 1392 - New `sns` output. 1393 1394 ## 2.9.3 - 2019-07-18 1395 1396 ### Added 1397 1398 - Allow extracting metric `rename` submatches into labels. 1399 - Field `use_patterns` added to `redis_pubsub` input for subscribing to channels using glob-style patterns. 1400 1401 ## 2.9.2 - 2019-07-17 1402 1403 ### Changed 1404 1405 - Go API: It's now possible to specify a custom config unit test file path suffix. 1406 1407 ## 2.9.1 - 2019-07-15 1408 1409 ### Added 1410 1411 - New rate limit and websocket message fields added to `http_server` input. 1412 - The `http` processor now optionally copies headers from response into resulting message metadata. 1413 - The `http` processor now sets a `http_status_code` metadata value into resulting messages (provided one is received.) 1414 1415 ### Changed 1416 1417 - Go API: Removed experimental `Block` functions from the cache and rate limit packages. 1418 1419 ## 2.9.0 - 2019-07-12 1420 1421 ### Added 1422 1423 - New (experimental) command flags `--test` and `--gen-test` added. 1424 - All http client components output now set a metric `request_timeout`. 1425 1426 ## 2.8.6 - 2019-07-10 1427 1428 ### Added 1429 1430 - All errors caught by processors should now be accessible via the `${!error}` interpolation function, rather than just flagged as `true`. 1431 1432 ### Fixed 1433 1434 - The `process_field` processor now propagates metadata to the original payload with the `result_type` set to discard. This allows proper error propagation. 1435 1436 ## 2.8.5 - 2019-07-03 1437 1438 ### Added 1439 1440 - Field `max_buffer` added to `subprocess` processor. 1441 1442 ### Fixed 1443 1444 - The `subprocess` processor now correctly logs and recovers subprocess pipeline related errors (such as exceeding buffer limits.) 1445 1446 ## 2.8.4 - 2019-07-02 1447 1448 ### Added 1449 1450 - New `json_delete` function added to the `awk` processor. 1451 1452 ### Fixed 1453 1454 - SQS output now correctly waits between retry attempts and escapes error loops during shutdown. 1455 1456 ## 2.8.3 - 2019-06-28 1457 1458 ### Added 1459 1460 - Go API: Add `RunWithOpts` opt `OptOverrideConfigDefaults`. 1461 1462 ### Fixed 1463 1464 - The `filter` and `filter_parts` config sections now correctly marshall when printing with `--all`. 1465 1466 ## 2.8.2 - 2019-06-28 1467 1468 ### Added 1469 1470 - Go API: A new service method `RunWithOpts` has been added in order to accomodate service customisations with opt funcs. 1471 1472 ## 2.8.1 - 2019-06-28 1473 1474 - New interpolation function `error`. 1475 1476 ## 2.8.0 - 2019-06-24 1477 1478 ### Added 1479 1480 - New `number` condition. 1481 - New `number` processor. 1482 - New `avro` processor. 1483 - Operator `enum` added to `text` condition. 1484 - Field `result_type` added to `process_field` processor for marshalling results into non-string types. 1485 - Go API: Plugin APIs now allow nil config constructors. 1486 - Registering plugins automatically adds plugin documentation flags to the main Benthos service. 1487 1488 ## 2.7.0 - 2019-06-20 1489 1490 ### Added 1491 1492 - Output `http_client` is now able to propagate responses from each request back to inputs supporting sync responses. 1493 - Added support for Gzip compression to `http_server` output sync responses. 1494 - New `check_interpolation` condition. 1495 1496 ## 2.6.0 - 2019-06-18 1497 1498 ### Added 1499 1500 - New `sync_response` output type, with experimental support added to the `http_server` input. 1501 - SASL authentication fields added to all Kafka components. 1502 1503 ## 2.5.0 - 2019-06-14 1504 1505 ### Added 1506 1507 - The `s3` input now sets `s3_content_encoding` metadata (when not using the download manager.) 1508 - New trace logging for the `rename`, `blacklist` and `whitelist` metric components to assist with debugging. 1509 1510 ## 2.4.0 - 2019-06-06 1511 1512 ### Added 1513 1514 - Ability to combine sync and async responses in serverless distributions. 1515 1516 ### Changed 1517 1518 - The `insert_part`, `merge_json` and `unarchive` processors now propagate message contexts. 1519 1520 ## 2.3.2 - 2019-06-05 1521 1522 ### Fixed 1523 1524 - JSON processors no longer escape `&`, `<`, and `>` characters by default. 1525 1526 ## 2.3.1 - 2019-06-04 1527 1528 ### Fixed 1529 1530 - The `http` processor now preserves message metadata and contexts. 1531 - Any `http` components that create requests with messages containing empty bodies now correctly function in WASM. 1532 1533 ## 2.3.0 - 2019-06-04 1534 1535 ### Added 1536 1537 - New `fetch_buffer_cap` field for `kafka` and `kafka_balanced` inputs. 1538 - Input `gcp_pubsub` now has the field `max_batch_count`. 1539 1540 ### Changed 1541 1542 - Reduced allocations under most JSON related processors. 1543 - Streams mode API now logs linting errors. 1544 1545 ## 2.2.4 - 2019-06-02 1546 1547 ### Added 1548 1549 - New interpolation function `batch_size`. 1550 1551 ## 2.2.3 - 2019-05-31 1552 1553 ### Fixed 1554 1555 - Output `elasticsearch` no longer reports index not found errors on connect. 1556 1557 ## 2.2.2 - 2019-05-30 1558 1559 ### Fixed 1560 1561 - Input reader no longer overrides message contexts for opentracing spans. 1562 1563 ## 2.2.1 - 2019-05-29 1564 1565 ### Fixed 1566 1567 - Improved construction error messages for `broker` and `switch` input and outputs. 1568 1569 ### Changed 1570 1571 - Plugins that don't use a configuration structure can now return nil in their sanitise functions in order to have the plugin section omitted. 1572 1573 ## 2.2.0 - 2019-05-22 1574 1575 ### Added 1576 1577 - The `kafka` and `kafka_balanced` inputs now set a `kafka_lag` metadata field to incoming messages. 1578 - The `awk` processor now has a variety of typed `json_set` functions `json_set_int`, `json_set_float` and `json_set_bool`. 1579 - Go API: Add experimental function for blocking cache and ratelimit constructors. 1580 1581 ### Fixed 1582 1583 - The `json` processor now defaults to an executable operator (clean). 1584 1585 ## 2.1.3 - 2019-05-20 1586 1587 ### Added 1588 1589 - Add experimental function for blocking processor constructors. 1590 1591 ## 2.1.2 - 2019-05-20 1592 1593 ### Added 1594 1595 - Core service logic has been moved into new package `service`, making it easier to maintain plugin builds that match upstream Benthos. 1596 1597 ## 2.1.1 - 2019-05-17 1598 1599 ### Added 1600 1601 - Experimental support for WASM builds. 1602 1603 ## 2.1.0 - 2019-05-16 1604 1605 ### Added 1606 1607 - Config linting now reports line numbers. 1608 - Config interpolations now support escaping. 1609 1610 ## 2.0.0 - 2019-05-14 1611 1612 ### Added 1613 1614 - API for creating `cache` implementations. 1615 - API for creating `rate_limit` implementations. 1616 1617 ### Changed 1618 1619 This is a major version released due to a series of minor breaking changes, you can read the [full migration guide here](https://www.benthos.dev/docs/guides/migration/v2). 1620 1621 #### Configuration 1622 1623 - Benthos now attempts to infer the `type` of config sections whenever the field is omitted, for more information please read this overview: [Concise Configuration](https://www.benthos.dev/docs/configuration/about#concise-configuration). 1624 - Field `unsubscribe_on_close` of the `nats_stream` input is now `false` by default. 1625 1626 #### Service 1627 1628 - The following commandline flags have been removed: `swap-envs`, `plugins-dir`, `list-input-plugins`, `list-output-plugins`, `list-processor-plugins`, `list-condition-plugins`. 1629 1630 #### Go API 1631 1632 - Package `github.com/Jeffail/benthos/lib/processor/condition` changed to `github.com/Jeffail/benthos/lib/condition`. 1633 - Interface `types.Cache` now has `types.Closable` embedded. 1634 - Interface `types.RateLimit` now has `types.Closable` embedded. 1635 - Add method `GetPlugin` to interface `types.Manager`. 1636 - Add method `WithFields` to interface `log.Modular`. 1637 1638 ## 1.20.4 - 2019-05-13 1639 1640 ### Fixed 1641 1642 - Ensure `process_batch` processor gets normalised correctly. 1643 1644 ## 1.20.3 - 2019-05-11 1645 1646 ### Added 1647 1648 - New `for_each` processor with the same behaviour as `process_batch`, `process_batch` is now considered an alias for `for_each`. 1649 1650 ## 1.20.2 - 2019-05-10 1651 1652 ### Changed 1653 1654 - The `sql` processor now executes across the batch, documentation updated to clarify. 1655 1656 ## 1.20.1 - 2019-05-10 1657 1658 ### Fixed 1659 1660 - Corrected `result_codec` field in `sql` processor config. 1661 1662 ## 1.20.0 - 2019-05-10 1663 1664 ### Added 1665 1666 - New `sql` processor. 1667 1668 ### Fixed 1669 1670 - Using `json_map_columns` with the `dynamodb` output should now correctly store `null` and array values within the target JSON structure. 1671 1672 ## 1.19.2 - 2019-05-09 1673 1674 ### Added 1675 1676 - New `encode` and `decode` scheme `hex`. 1677 1678 ### Fixed 1679 1680 - Fixed potential panic when attempting an invalid HTTP client configuration. 1681 1682 ## 1.19.1 - 2019-05-08 1683 1684 ### Fixed 1685 1686 - Benthos in streams mode no longer tries to load directory `/benthos/streams` by default. 1687 1688 ## 1.19.0 - 2019-05-07 1689 1690 ### Added 1691 1692 - Field `json_map_columns` added to `dynamodb` output. 1693 1694 ## 1.18.0 - 2019-05-06 1695 1696 ### Added 1697 1698 - JSON references are now supported in configuration files. 1699 1700 ## 1.17.0 - 2019-05-04 1701 1702 ### Added 1703 1704 - The `hash` processor now supports `sha1`. 1705 - Field `force_path_style_urls` added to `s3` components. 1706 - Field `content_type` of the `s3` output is now interpolated. 1707 - Field `content_encoding` added to `s3` output. 1708 1709 ### Fixed 1710 1711 - The `benthos-lambda` distribution now correctly returns all message parts in synchronous execution. 1712 1713 ### Changed 1714 1715 - Docker builds now use a locally cached `vendor` for dependencies. 1716 - All `s3` components no longer default to enforcing path style URLs. 1717 1718 ## 1.16.0 - 2019-04-30 1719 1720 ### Added 1721 1722 - New output `drop_on_error`. 1723 - Field `retry_until_success` added to `switch` output. 1724 1725 ### Fixed 1726 1727 - Improved error and output logging for `subprocess` processor when the process exits unexpectedly. 1728 1729 ## 1.15.0 - 2019-04-26 1730 1731 ### Changed 1732 1733 - The main docker image is now based on busybox. 1734 - Lint rule added for `batch` processors outside of the input section. 1735 1736 ## 1.14.3 - 2019-04-25 1737 1738 ### Fixed 1739 1740 - Removed potential `benthos-lambda` panic on shut down. 1741 1742 ## 1.14.2 - 2019-04-25 1743 1744 ### Fixed 1745 1746 - The `redis` cache no longer incorrectly returns a "key not found" error instead of connection errors. 1747 1748 ## 1.14.1 - 2019-04-24 1749 1750 ### Changed 1751 1752 - Changed docker tag format from `vX.Y.Z` to `X.Y.Z`. 1753 1754 ## 1.14.0 - 2019-04-24 1755 1756 ### Added 1757 1758 - Output `broker` pattern `fan_out_sequential`. 1759 - Output type `drop` for dropping all messages. 1760 - New interpolation function `timestamp_utc`. 1761 1762 ## 1.13.0 - 2019-04-22 1763 1764 ### Added 1765 1766 - New `benthos-lambda` distribution for running Benthos as a lambda function. 1767 1768 ## 1.12.0 - 2019-04-21 1769 1770 ### Added 1771 1772 - New `s3` cache implementation. 1773 - New `file` cache implementation. 1774 - Operators `quote` and `unquote` added to the `text` processor. 1775 - Configs sent via the streams mode HTTP API are now interpolated with environment variable substitutes. 1776 1777 ### Changed 1778 1779 - All AWS `s3` components now enforce path style syntax for bucket URLs. This improves compatibility with third party endpoints. 1780 1781 ## 1.11.0 - 2019-04-12 1782 1783 ### Added 1784 1785 - New `parallel` processor. 1786 1787 ### Fixed 1788 1789 - The `dynamodb` cache `get` call now correctly reports key not found versus general request error. 1790 1791 ## 1.10.10 - 2019-04-10 1792 1793 ### Added 1794 1795 - New `sqs_bucket_path` field added to `s3` input. 1796 1797 ### Fixed 1798 1799 - The `sqs` input now rejects messages that fail by resetting the visibility timeout. 1800 - The `sqs` input no longer fails to delete consumed messages when the batch contains duplicate message IDs. 1801 1802 ## 1.10.9 - 2019-04-05 1803 1804 ### Fixed 1805 1806 - The `metric` processor no longer mixes label keys when processing across parallel pipelines. 1807 1808 ## 1.10.8 - 2019-04-03 1809 1810 ### Added 1811 1812 - Comma separated `kafka` and `kafka_balanced` address and topic values are now trimmed for whitespace. 1813 1814 ## 1.10.6 - 2019-04-02 1815 1816 ### Added 1817 1818 - Field `max_processing_period` added to `kafka` and `kafka_balanced` inputs. 1819 1820 ### Fixed 1821 1822 - Compaction intervals are now respected by the `memory` cache type. 1823 1824 ## 1.10.5 - 2019-03-29 1825 1826 ### Fixed 1827 1828 - Improved `kafka_balanced` consumer group connection behaviour. 1829 1830 ## 1.10.4 - 2019-03-29 1831 1832 ### Added 1833 1834 - More `kafka_balanced` input config fields for consumer group timeouts. 1835 1836 ## 1.10.3 - 2019-03-28 1837 1838 ### Added 1839 1840 - New config interpolation function `uuid_v4`. 1841 1842 ## 1.10.2 - 2019-03-21 1843 1844 ### Fixed 1845 1846 - The `while` processor now correctly checks conditions against the first batch of the result of last processor loop. 1847 1848 ## 1.10.1 - 2019-03-19 1849 1850 ### Added 1851 1852 - Field `max_loops` added to `while` processor. 1853 1854 ## 1.10.0 - 2019-03-18 1855 1856 ### Added 1857 1858 - New `while` processor. 1859 1860 ## 1.9.0 - 2019-03-17 1861 1862 ### Added 1863 1864 - New `cache` processor. 1865 - New `all` condition. 1866 - New `any` condition. 1867 1868 ## 1.8.0 - 2019-03-14 1869 1870 ### Added 1871 1872 - Function interpolation for field `subject` added to `nats` output. 1873 1874 ### Changed 1875 1876 - Switched underlying `kafka_balanced` implementation to sarama consumer. 1877 1878 ## 1.7.10 - 2019-03-11 1879 1880 ### Fixed 1881 1882 - Always allow acknowledgement flush during graceful termination. 1883 1884 ## 1.7.9 - 2019-03-08 1885 1886 ### Fixed 1887 1888 - Removed unnecessary subscription check from `gcp_pubsub` input. 1889 1890 ## 1.7.7 - 2019-03-08 1891 1892 ### Added 1893 1894 - New field `fields` added to `log` processor for structured log output. 1895 1896 ## 1.7.3 - 2019-03-05 1897 1898 ### Added 1899 1900 - Function interpolation for field `channel` added to `redis_pubsub` output. 1901 1902 ## 1.7.2 - 2019-03-01 1903 1904 ### Added 1905 1906 - Field `byte_size` added to `split` processor. 1907 1908 ## 1.7.1 - 2019-02-27 1909 1910 ### Fixed 1911 1912 - Field `dependencies` of children of the `process_dag` processor now correctly parsed from config files. 1913 1914 ## 1.7.0 - 2019-02-26 1915 1916 ### Added 1917 1918 - Field `push_job_name` added to `prometheus` metrics type. 1919 - New `rename` metrics target. 1920 1921 ### Fixed 1922 1923 - Removed potential race condition in `process_dag` with raw bytes conditions. 1924 1925 ## 1.6.1 - 2019-02-21 1926 1927 ### Added 1928 1929 - Field `max_batch_count` added to `s3` input. 1930 - Field `max_number_of_messages` added to `sqs` input. 1931 1932 ## 1.6.0 - 2019-02-20 1933 1934 ### Added 1935 1936 - New `blacklist` metrics target. 1937 - New `whitelist` metrics target. 1938 - Initial support for opentracing, including a new `tracer` root component. 1939 - Improved generated metrics documentation and config examples. 1940 - The `nats_stream` input now has a field `unsubscribe_on_close` that when disabled allows durable subscription offsets to persist even when all connections are closed. 1941 - Metadata field `nats_stream_sequence` added to `nats_stream` input. 1942 1943 ## 1.5.1 - 2019-02-11 1944 1945 ### Fixed 1946 1947 - The `subprocess` processor no longer sends unexpected empty lines when messages end with a line break. 1948 1949 ## 1.5.0 - 2019-02-07 1950 1951 ### Added 1952 1953 - New `switch` processor. 1954 1955 ### Fixed 1956 1957 - Printing configs now sanitises resource sections. 1958 1959 ## 1.4.1 - 2019-02-04 1960 1961 ### Fixed 1962 1963 - The `headers` field in `http` configs now detects and applies `host` keys. 1964 1965 ## 1.4.0 - 2019-02-04 1966 1967 ### Added 1968 1969 - New `json_documents` format added to the `unarchive` processor. 1970 - Field `push_interval` added to the `prometheus` metrics type. 1971 1972 ## 1.3.2 - 2019-01-31 1973 1974 ### Fixed 1975 1976 - Brokers now correctly parse configs containing plugin types as children. 1977 1978 ## 1.3.1 - 2019-01-30 1979 1980 ### Fixed 1981 1982 - Output broker types now correctly allocates nested processors for `fan_out` and `try` patterns. 1983 - JSON formatted loggers now correctly escape error messages with line breaks. 1984 1985 ## 1.3.0 - 2019-01-29 1986 1987 ### Added 1988 1989 - Improved error logging for `s3` input download failures. 1990 - More metadata fields copied to messages from the `s3` input. 1991 - Field `push_url` added to the `prometheus` metrics target. 1992 1993 ## 1.2.1 - 2019-01-28 1994 1995 ### Added 1996 1997 - Resources (including plugins) that implement `Closable` are now shutdown cleanly. 1998 1999 ## 1.2.0 - 2019-01-28 2000 2001 ### Added 2002 2003 - New `json_array` format added to the `archive` and `unarchive` processors. 2004 - Preliminary support added to the resource manager API to allow arbitrary shared resource plugins. 2005 2006 ## 1.1.4 - 2019-01-23 2007 2008 ### Fixed 2009 2010 - The `s3` input now caps and iterates batched SQS deletes. 2011 2012 ## 1.1.3 - 2019-01-22 2013 2014 ### Fixed 2015 2016 - The `archive` processor now interpolates the `path` per message of the batch. 2017 2018 ## 1.1.2 - 2019-01-21 2019 2020 ### Fixed 2021 2022 - Fixed environment variable interpolation when combined with embedded function interpolations. 2023 - Fixed break down metric indexes for input and output brokers. 2024 2025 ## 1.1.0 - 2019-01-17 2026 2027 ### Added 2028 2029 - Input `s3` can now toggle the use of a download manager, switching off now downloads metadata from the target file. 2030 - Output `s3` now writes metadata to the uploaded file. 2031 - Operator `unescape_url_query` added to `text` processor. 2032 2033 ### Fixed 2034 2035 - The `nats_steam` input and output now actively attempt to recover stale connections. 2036 - The `awk` processor prints errors and flags failure when the program exits with a non-zero status. 2037 2038 ## 1.0.2 - 2019-01-07 2039 2040 ### Fixed 2041 2042 - The `subprocess` processor now attempts to read all flushed stderr output from a process when it fails. 2043 2044 ## 1.0.1 - 2019-01-05 2045 2046 ### Added 2047 2048 - Function `print_log` added to `awk` processor. 2049 2050 ### Fixed 2051 2052 - The `awk` processor function `json_get` no longer returns string values with quotes. 2053 2054 ## 1.0.0 - 2019-01-01 2055 2056 ### Changed 2057 2058 - Processor `awk` codecs changed. 2059 2060 ## 0.42.4 - 2018-12-31 2061 2062 ### Changed 2063 2064 - Output type `sqs` now supports batched message sends. 2065 2066 ## 0.42.3 - 2018-12-28 2067 2068 ### Added 2069 2070 - Functions `json_get` and `json_set` added to `awk` processor. 2071 2072 ## 0.42.1 - 2018-12-20 2073 2074 ### Added 2075 2076 - Functions `timestamp_format`, `timestamp_format_nano`, `metadata_get` and `metadata_set` added to `awk` processor. 2077 2078 ## 0.42.0 - 2018-12-19 2079 2080 ### Added 2081 2082 - New `sleep` processor. 2083 - New `awk` processor. 2084 2085 ### Changed 2086 2087 - Converted all integer based time period fields to string based, e.g. `timeout_ms: 5000` would now be `timeout: 5s`. This will may potentially be disruptive but the `--strict` flag should catch all deprecated fields in an existing config. 2088 2089 ## 0.41.0 - 2018-12-12 2090 2091 ### Changed 2092 2093 - Renamed `max_batch_size` to `max_batch_count` for consistency. 2094 2095 ## 0.40.2 - 2018-12-12 2096 2097 ### Added 2098 2099 - New `max_batch_size` field added to `kafka`, `kafka_balanced` and `amqp` inputs. This provides a mechanism for creating message batches optimistically. 2100 2101 ## 0.40.0 - 2018-12-10 2102 2103 ### Added 2104 2105 - New `subprocess` processor. 2106 2107 ### Changed 2108 2109 - API: The `types.Processor` interface has been changed in order to add lifetime cleanup methods (added `CloseAsync` and `WaitForClose`). For the overwhelming majority of processors these functions will be no-ops. 2110 - More consistent `condition` metrics. 2111 2112 ## 0.39.2 - 2018-12-07 2113 2114 ### Added 2115 2116 - New `try` and `catch` processors for improved processor error handling. 2117 2118 ## 0.39.1 - 2018-12-07 2119 2120 ### Added 2121 2122 - All processors now attach error flags. 2123 - S3 input is now more flexible with SNS triggered SQS events. 2124 2125 ### Changed 2126 2127 - Processor metrics have been made more consistent. 2128 2129 ## 0.39.0 - 2018-12-05 2130 2131 ### Added 2132 2133 - New endpoint `/ready` that returns 200 when both the input and output components are connected, otherwise 503. This is intended to be used as a readiness probe. 2134 2135 ### Changed 2136 2137 - Large simplifications to all metrics paths. 2138 - Fully removed the previously deprecated `combine` processor. 2139 - Input and output plugins updated to support new connection health checks. 2140 2141 ## 0.38.10 - 2018-12-04 2142 2143 ### Added 2144 2145 - Field `role_external_id` added to all S3 credential configs. 2146 - New `processor_failed` condition and improved processor error handling which can be read about [here](./docs/error_handling.md) 2147 2148 ## 0.38.8 - 2018-11-29 2149 2150 ### Added 2151 2152 - New `content_type` field for the `s3` output. 2153 2154 ## 0.38.6 - 2018-11-28 2155 2156 ### Added 2157 2158 - New `group_by_value` processor. 2159 2160 ## 0.38.5 - 2018-11-27 2161 2162 ### Added 2163 2164 - Lint errors are logged (level INFO) during normal Benthos operation. 2165 - New `--strict` command flag which causes Benthos to abort when linting errors are found in a config file. 2166 2167 ## 0.38.4 - 2018-11-26 2168 2169 ### Added 2170 2171 - New `--lint` command flag for linting config files. 2172 2173 ## 0.38.1 - 2018-11-23 2174 2175 ### Changed 2176 2177 - The `s3` output now attempts to batch uploads. 2178 - The `s3` input now exposes errors in deleting SQS messages during acks. 2179 2180 ## 0.38.0 - 2018-11-22 2181 2182 ### Changed 2183 2184 - Resource based conditions no longer benefit from cached results. In practice this optimisation was easy to lose in config and difficult to maintain. 2185 2186 ## 0.37.4 - 2018-11-22 2187 2188 ### Added 2189 2190 - Metadata is now sent to `kafka` outputs. 2191 - New `max_inflight` field added to the `nats_stream` input. 2192 2193 ### Fixed 2194 2195 - Fixed relative path trimming for streams from file directories. 2196 2197 ## 0.37.2 - 2018-11-15 2198 2199 ### Fixed 2200 2201 - The `dynamodb` cache and output types now set TTL columns as unix timestamps. 2202 2203 ## 0.37.1 - 2018-11-13 2204 2205 ### Added 2206 2207 - New `escape_url_query` operator for the `text` processor. 2208 2209 ## 0.37.0 - 2018-11-09 2210 2211 ### Changed 2212 2213 - Removed submatch indexes in the `text` processor `find_regexp` operator and added documentation for expanding submatches in the `replace_regexp` operator. 2214 2215 ## 0.36.4 - 2018-11-09 2216 2217 ### Added 2218 2219 - Allow submatch indexes in the `find_regexp` operator for the `text` processor. 2220 2221 ## 0.36.3 - 2018-11-08 2222 2223 ### Added 2224 2225 - New `find_regexp` operator for the `text` processor. 2226 2227 ## 0.36.1 - 2018-11-07 2228 2229 ### Added 2230 2231 - New `aws` fields to the `elasticsearch` output to allow AWS authentication. 2232 2233 ## 0.36.0 - 2018-11-06 2234 2235 ### Added 2236 2237 - Add max-outstanding fields to `gcp_pubsub` input. 2238 - Add new `dynamodb` output. 2239 2240 ### Changed 2241 2242 - The `s3` output now calculates `path` field function interpolations per message of a batch. 2243 2244 ## 0.35.1 - 2018-10-31 2245 2246 ### Added 2247 2248 - New `set` operator for the `text` processor. 2249 2250 ## 0.35.0 - 2018-10-30 2251 2252 ### Added 2253 2254 - New `cache` output type. 2255 2256 ## 0.34.13 - 2018-10-29 2257 2258 ### Added 2259 2260 - New `group_by` processor. 2261 - Add bulk send support to `elasticsearch` output. 2262 2263 ## 0.34.8 - 2018-10-10 2264 2265 ### Added 2266 2267 - New `content` interpolation function. 2268 2269 ## 0.34.7 - 2018-10-04 2270 2271 ### Added 2272 2273 - New `redis` cache type. 2274 2275 ## 0.34.5 - 2018-10-02 2276 2277 ### Changed 2278 2279 - The `process_map` processor now allows map target path overrides when a target is the parent of another target. 2280 2281 ## 0.34.4 - 2018-10-02 2282 2283 ### Added 2284 2285 - Field `pipeline` and `sniff` added to the `elasticsearch` output. 2286 - Operators `to_lower` and `to_upper` added to the `text` processor. 2287 2288 ## 0.34.3 - 2018-09-29 2289 2290 ### Added 2291 2292 - Field `endpoint` added to all AWS types. 2293 2294 ## 0.34.2 - 2018-09-27 2295 2296 ### Changed 2297 2298 - Allow `log` config field `static_fields` to be fully overridden. 2299 2300 ## 0.34.0 - 2018-09-27 2301 2302 ### Added 2303 2304 - New `process_dag` processor. 2305 - New `static_fields` map added to log config for setting static log fields. 2306 2307 ### Changed 2308 2309 - JSON log field containing component path moved from `@service` to `component`. 2310 2311 ## 0.33.0 - 2018-09-22 2312 2313 ### Added 2314 2315 - New `gcp_pubsub` input and outputs. 2316 - New `log` processor. 2317 - New `lambda` processor. 2318 2319 ## 0.32.0 - 2018-09-18 2320 2321 ### Added 2322 2323 - New `process_batch` processor. 2324 - Added `count` field to `batch` processor. 2325 - Metrics for `kinesis` output throttles. 2326 2327 ### Changed 2328 2329 - The `combine` processor is now considered DEPRECATED, please use the `batch` processor instead. 2330 - The `batch` processor field `byte_size` is now set at 0 (and therefore ignored) by default. A log warning has been added in case anyone was relying on the default. 2331 2332 ## 0.31.4 - 2018-09-16 2333 2334 ### Added 2335 2336 - New `rate_limit` resource with a `local` type. 2337 - Field `rate_limit` added to `http` based processors, inputs and outputs. 2338 2339 ## 0.31.2 - 2018-09-14 2340 2341 ### Added 2342 2343 - New `prefetch_count` field added to `nats` input. 2344 2345 ## 0.31.0 - 2018-09-11 2346 2347 ### Added 2348 2349 - New `bounds_check` condition type. 2350 - New `check_field` condition type. 2351 - New `queue` field added to `nats` input. 2352 - Function interpolation for the `topic` field of the `nsq` output. 2353 2354 ### Changed 2355 2356 - The `nats` input now defaults to joining a queue. 2357 2358 ## 0.30.1 - 2018-09-06 2359 2360 ### Changed 2361 2362 - The redundant `nsq` output field `max_in_flight` has been removed. 2363 - The `files` output now interpolates paths per message part of a batch. 2364 2365 ## 0.30.0 - 2018-09-06 2366 2367 ### Added 2368 2369 - New `hdfs` input and output. 2370 - New `switch` output. 2371 - New `enum` and `has_prefix` operators for the `metadata` condition. 2372 - Ability to set `tls` client certificate fields directly. 2373 2374 ## 0.29.0 - 2018-09-02 2375 2376 ### Added 2377 2378 - New `retry` output. 2379 - Added `regex_partial` and `regex_exact` operators to the `metadata` condition. 2380 2381 ### Changed 2382 2383 - The `kinesis` output field `retries` has been renamed `max_retries` in order to expose the difference in its zero value behaviour (endless retries) versus other `retry` fields (zero retries). 2384 2385 ## 0.28.0 - 2018-09-01 2386 2387 ### Added 2388 2389 - New `endpoint` field added to `kinesis` input. 2390 - New `dynamodb` cache type. 2391 2392 ## 0.27.0 - 2018-08-30 2393 2394 ### Added 2395 2396 - Function interpolation for the `topic` field of the `kafka` output. 2397 - New `target_version` field for the `kafka_balanced` input. 2398 - TLS config fields for client certificates. 2399 2400 ### Changed 2401 2402 - TLS config field `cas_file` has been renamed `root_cas_file`. 2403 2404 ## 0.26.3 - 2018-08-29 2405 2406 ### Added 2407 2408 - New `zip` option for the `archive` and `unarchive` processors. 2409 2410 ### Changed 2411 2412 - The `kinesis` output type now supports batched sends and per message interpolation. 2413 2414 ## 0.26.2 - 2018-08-27 2415 2416 ### Added 2417 2418 - New `metric` processor. 2419 2420 ## 0.26.1 - 2018-08-26 2421 2422 ### Added 2423 2424 - New `redis_streams` input and output. 2425 2426 ## 0.26.0 - 2018-08-25 2427 2428 ### Added 2429 2430 - New `kinesis` input and output. 2431 2432 ## 0.25.0 - 2018-08-22 2433 2434 ### Added 2435 2436 - The `index` field of the `elasticsearch` output can now be dynamically set using function interpolation. 2437 - New `hash` processor. 2438 2439 ### Changed 2440 2441 - API: The `metrics.Type` interface has been changed in order to add labels. 2442 2443 ## 0.24.0 - 2018-08-17 2444 2445 ### Changed 2446 2447 - Significant restructuring of `amqp` inputs and outputs. These changes should be backwards compatible for existing pipelines, but changes the way in which queues, exchanges and bindings are declared using these types. 2448 2449 ## 0.23.17 - 2018-08-17 2450 2451 ### Added 2452 2453 - New durable fields for `amqp` input and output types. 2454 2455 ## 0.23.15 - 2018-08-16 2456 2457 ### Changed 2458 2459 - Improved statsd client with better cached aggregation. 2460 2461 ## 0.23.14 - 2018-08-16 2462 2463 ### Added 2464 2465 - New `tls` fields for `amqp` input and output types. 2466 2467 ## 0.23.12 - 2018-08-14 2468 2469 ### Added 2470 2471 - New `type` field for `elasticsearch` output. 2472 2473 ## 0.23.9 - 2018-08-10 2474 2475 ### Added 2476 2477 - New `throttle` processor. 2478 2479 ## 0.23.6 - 2018-08-09 2480 2481 ### Added 2482 2483 - New `less_than` and `greater_than` operators for `metadata` condition. 2484 2485 ## 0.23.4 - 2018-08-09 2486 2487 ### Added 2488 2489 - New `metadata` condition type. 2490 - More metadata fields for `kafka` input. 2491 - Field `commit_period_ms` for `kafka` and `kafka_balanced` inputs for specifying a commit period. 2492 2493 ## 0.23.1 - 2018-08-06 2494 2495 ### Added 2496 2497 - New `retries` field to `s3` input, to cap the number of download attempts made on the same bucket item. 2498 - Added metadata based mechanism to detect final message from a `read_until` input. 2499 - Added field to `split` processor for specifying target batch sizes. 2500 2501 ## 0.23.0 - 2018-08-06 2502 2503 ### Added 2504 2505 - Metadata fields are now per message part within a batch. 2506 - New `metadata_json_object` function interpolation to return a JSON object of metadata key/value pairs. 2507 2508 ### Changed 2509 2510 - The `metadata` function interpolation now allows part indexing and no longer returns a JSON object when no key is specified, this behaviour can now be done using the `metadata_json_object` function. 2511 2512 ## 0.22.0 - 2018-08-03 2513 2514 ### Added 2515 2516 - Fields for the `http` processor to enable parallel requests from message batches. 2517 2518 ### Changed 2519 2520 - Broker level output processors are now applied _before_ the individual output processors. 2521 - The `dynamic` input and output HTTP paths for CRUD operations are now `/inputs/{input_id}` and `/outputs/{output_id}` respectively. 2522 - Removed deprecated `amazon_s3`, `amazon_sqs` and `scalability_protocols` input and output types. 2523 - Removed deprecated `json_fields` field from the `dedupe` processor. 2524 2525 ## 0.21.0 - 2018-07-31 2526 2527 ### Added 2528 2529 - Add conditions to `process_map` processor. 2530 2531 ### Changed 2532 2533 - TLS config fields have been cleaned up for multiple types. This affects the `kafka`, `kafka_balanced` and `http_client` input and output types, as well as the `http` processor type. 2534 2535 ## 0.20.8 - 2018-07-30 2536 2537 ### Added 2538 2539 - New `delete_all` and `delete_prefix` operators for `metadata` processor. 2540 - More metadata fields extracted from the AMQP input. 2541 - HTTP clients now support function interpolation on the URL and header values, this includes the `http_client` input and output as well as the `http` processor. 2542 2543 ## 0.20.7 - 2018-07-27 2544 2545 ### Added 2546 2547 - New `key` field added to the `dedupe` processor, allowing you to deduplicate using function interpolation. This deprecates the `json_paths` array field. 2548 2549 ## 0.20.6 - 2018-07-27 2550 2551 ### Added 2552 2553 - New `s3` and `sqs` input and output types, these replace the now deprecated `amazon_s3` and `amazon_sqs` types respectively, which will eventually be removed. 2554 - New `nanomsg` input and output types, these replace the now deprecated `scalability_protocols` types, which will eventually be removed. 2555 2556 ## 0.20.5 - 2018-07-27 2557 2558 ### Added 2559 2560 - Metadata fields are now collected from MQTT input. 2561 - AMQP output writes all metadata as headers. 2562 - AMQP output field `key` now supports function interpolation. 2563 2564 ## 0.20.1 - 2018-07-26 2565 2566 ### Added 2567 2568 - New `metadata` processor and configuration interpolation function. 2569 2570 ## 0.20.0 - 2018-07-26 2571 2572 ### Added 2573 2574 - New config interpolator function `json_field` for extracting parts of a JSON message into a config value. 2575 2576 ### Changed 2577 2578 - Log level config field no longer stutters, `logger.log_level` is now `logger.level`. 2579 2580 ## 0.19.1 - 2018-07-25 2581 2582 ### Added 2583 2584 - Ability to create batches via conditions on message payloads in the `batch` processor. 2585 - New `--examples` flag for generating specific examples from Benthos. 2586 2587 ## 0.19.0 - 2018-07-23 2588 2589 ### Added 2590 2591 - New `text` processor. 2592 2593 ### Changed 2594 2595 - Processor `process_map` replaced field `strict_premapping` with `premap_optional`. 2596 2597 ## 0.18.0 - 2018-07-20 2598 2599 ### Added 2600 2601 - New `process_field` processor. 2602 - New `process_map` processor. 2603 2604 ### Changed 2605 2606 - Removed mapping fields from the `http` processor, this behaviour has been put into the new `process_map` processor instead. 2607 2608 ## 0.17.0 - 2018-07-17 2609 2610 ### Changed 2611 2612 - Renamed `content` condition type to `text` in order to clarify its purpose. 2613 2614 ## 0.16.4 - 2018-07-17 2615 2616 ### Added 2617 2618 - Latency metrics for caches. 2619 - TLS options for `kafka` and `kafka_partitions` inputs and outputs. 2620 2621 ### Changed 2622 2623 - Metrics for items configured within the `resources` section are now namespaced under their identifier. 2624 2625 ## 0.16.3 - 2018-07-16 2626 2627 ### Added 2628 2629 - New `copy` and `move` operators for the `json` processor. 2630 2631 ## 0.16.2 - 2018-07-12 2632 2633 ### Added 2634 2635 - Metrics for recording `http` request latencies. 2636 2637 ## 0.16.0 - 2018-07-09 2638 2639 ### Changed 2640 2641 - Improved and rearranged fields for `http_client` input and output. 2642 2643 ## 0.15.5 - 2018-07-08 2644 2645 ### Added 2646 2647 - More compression and decompression targets. 2648 - New `lines` option for archive/unarchive processors. 2649 - New `encode` and `decode` processors. 2650 - New `period_ms` field for the `batch` processor. 2651 - New `clean` operator for the `json` processor. 2652 2653 ## 0.15.4 - 2018-07-04 2654 2655 ### Added 2656 2657 - New `http` processor, where payloads can be sent to arbitrary HTTP endpoints and the result constructed into a new payload. 2658 - New `inproc` inputs and outputs for linking streams together. 2659 2660 ## 0.15.3 - 2018-07-03 2661 2662 ### Added 2663 2664 - New streams endpoint `/streams/{id}/stats` for obtaining JSON metrics for a stream. 2665 2666 ### Changed 2667 2668 - Allow comma separated topics for `kafka_balanced`. 2669 2670 ## 0.15.0 - 2018-06-28 2671 2672 ### Added 2673 2674 - Support for PATCH verb on the streams mode `/streams/{id}` endpoint. 2675 2676 ### Changed 2677 2678 - Sweeping changes were made to the environment variable configuration file. This file is now auto generated along with its supporting document. This change will impact the docker image. 2679 2680 ## 0.14.7 - 2018-06-24 2681 2682 ### Added 2683 2684 - New `filter_parts` processor for filtering individual parts of a message batch. 2685 - New field `open_message` for `websocket` input. 2686 2687 ### Changed 2688 2689 - No longer setting default input processor. 2690 2691 ## 0.14.6 - 2018-06-21 2692 2693 ### Added 2694 2695 - New `root_path` field for service wide `http` config. 2696 2697 ## 0.14.5 - 2018-06-21 2698 2699 ### Added 2700 2701 - New `regexp_exact` and `regexp_partial` content condition operators. 2702 2703 ## 0.14.4 - 2018-06-19 2704 2705 ## Changed 2706 2707 - The `statsd` metrics target will now periodically report connection errors. 2708 2709 ## 0.14.2 - 2018-06-18 2710 2711 ## Changed 2712 2713 - The `json` processor will now `append` array values in expanded form. 2714 2715 ## 0.14.0 - 2018-06-15 2716 2717 ### Added 2718 2719 - More granular config options in the `http_client` output for controlling retry logic. 2720 - New `try` pattern for the output `broker` type, which can be used in order to configure fallback outputs. 2721 - New `json` processor, this replaces `delete_json`, `select_json`, `set_json`. 2722 2723 ### Changed 2724 2725 - The `streams` API endpoints have been changed to become more "RESTy". 2726 - Removed the `delete_json`, `select_json` and `set_json` processors, please use the `json` processor instead. 2727 2728 ## 0.13.5 - 2018-06-10 2729 2730 ### Added 2731 2732 - New `grok` processor for creating structured objects from unstructured data. 2733 2734 ## 0.13.4 - 2018-06-08 2735 2736 ### Added 2737 2738 - New `files` input type for reading multiple files as discrete messages. 2739 2740 ### Changed 2741 2742 - Increase default `max_buffer` for `stdin`, `file` and `http_client` inputs. 2743 - Command flags `--print-yaml` and `--print-json` changed to provide sanitised outputs unless accompanied by new `--all` flag. 2744 2745 ### Removed 2746 2747 - Badger based buffer option has been removed. 2748 2749 ## 0.13.3 - 2018-06-06 2750 2751 ### Added 2752 2753 - New metrics wrapper for more basic interface implementations. 2754 - New `delete_json` processor. 2755 - New field `else_processors` for `conditional` processor. 2756 2757 ## 0.13.2 - 2018-06-03 2758 2759 ### Added 2760 2761 - New websocket endpoint for `http_server` input. 2762 - New websocket endpoint for `http_server` output. 2763 - New `websocket` input type. 2764 - New `websocket` output type. 2765 2766 ## 0.13.1 - 2018-06-02 2767 2768 ### Added 2769 2770 - Goreleaser config for generating release packages. 2771 2772 ### Changed 2773 2774 - Back to using Scratch as base for Docker image, instead taking ca-certificates from the build image. 2775 2776 ## 0.13.0 - 2018-06-02 2777 2778 ### Added 2779 2780 - New `batch` processor for combining payloads up to a number of bytes. 2781 - New `conditional` processor, allows you to configure a chain of processors to only be run if the payload passes a `condition`. 2782 - New `--stream` mode features: 2783 + POST verb for `/streams` path now supported. 2784 + New `--streams-dir` flag for parsing a directory of stream configs. 2785 2786 ### Changed 2787 2788 - The `condition` processor has been renamed `filter`. 2789 - The `custom_delimiter` fields in any line reader types `file`, `stdin`, `stdout`, etc have been renamed `delimiter`, where the behaviour is the same. 2790 - Now using Alpine as base for Docker image, includes ca-certificates.