github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/kinesis_firehose.md (about) 1 --- 2 title: kinesis_firehose 3 type: output 4 status: deprecated 5 categories: ["Services","AWS"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/kinesis_firehose.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 :::warning DEPRECATED 19 This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). 20 ::: 21 22 Sends messages to a Kinesis Firehose delivery stream. 23 24 25 <Tabs defaultValue="common" values={[ 26 { label: 'Common', value: 'common', }, 27 { label: 'Advanced', value: 'advanced', }, 28 ]}> 29 30 <TabItem value="common"> 31 32 ```yaml 33 # Common config fields, showing default values 34 output: 35 label: "" 36 kinesis_firehose: 37 stream: "" 38 max_in_flight: 1 39 batching: 40 count: 0 41 byte_size: 0 42 period: "" 43 check: "" 44 region: eu-west-1 45 ``` 46 47 </TabItem> 48 <TabItem value="advanced"> 49 50 ```yaml 51 # All config fields, showing default values 52 output: 53 label: "" 54 kinesis_firehose: 55 stream: "" 56 max_in_flight: 1 57 batching: 58 count: 0 59 byte_size: 0 60 period: "" 61 check: "" 62 processors: [] 63 region: eu-west-1 64 endpoint: "" 65 credentials: 66 profile: "" 67 id: "" 68 secret: "" 69 token: "" 70 role: "" 71 role_external_id: "" 72 max_retries: 0 73 backoff: 74 initial_interval: 1s 75 max_interval: 5s 76 max_elapsed_time: 30s 77 ``` 78 79 </TabItem> 80 </Tabs> 81 82 ## Alternatives 83 84 This output has been renamed to [`aws_kinesis_firehose`](/docs/components/outputs/aws_kinesis_firehose). 85 86 ### Credentials 87 88 By default Benthos will use a shared credentials file when connecting to AWS 89 services. It's also possible to set them explicitly at the component level, 90 allowing you to transfer data across accounts. You can find out more 91 [in this document](/docs/guides/cloud/aws). 92 93 ## Performance 94 95 This output benefits from sending multiple messages in flight in parallel for 96 improved performance. You can tune the max number of in flight messages with the 97 field `max_in_flight`. 98 99 This output benefits from sending messages as a batch for improved performance. 100 Batches can be formed at both the input and output level. You can find out more 101 [in this doc](/docs/configuration/batching). 102 103 ## Fields 104 105 ### `stream` 106 107 The stream to publish messages to. 108 109 110 Type: `string` 111 Default: `""` 112 113 ### `max_in_flight` 114 115 The maximum number of messages to have in flight at a given time. Increase this to improve throughput. 116 117 118 Type: `int` 119 Default: `1` 120 121 ### `batching` 122 123 Allows you to configure a [batching policy](/docs/configuration/batching). 124 125 126 Type: `object` 127 128 ```yaml 129 # Examples 130 131 batching: 132 byte_size: 5000 133 count: 0 134 period: 1s 135 136 batching: 137 count: 10 138 period: 1s 139 140 batching: 141 check: this.contains("END BATCH") 142 count: 0 143 period: 1m 144 ``` 145 146 ### `batching.count` 147 148 A number of messages at which the batch should be flushed. If `0` disables count based batching. 149 150 151 Type: `int` 152 Default: `0` 153 154 ### `batching.byte_size` 155 156 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 157 158 159 Type: `int` 160 Default: `0` 161 162 ### `batching.period` 163 164 A period in which an incomplete batch should be flushed regardless of its size. 165 166 167 Type: `string` 168 Default: `""` 169 170 ```yaml 171 # Examples 172 173 period: 1s 174 175 period: 1m 176 177 period: 500ms 178 ``` 179 180 ### `batching.check` 181 182 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 183 184 185 Type: `string` 186 Default: `""` 187 188 ```yaml 189 # Examples 190 191 check: this.type == "end_of_transaction" 192 ``` 193 194 ### `batching.processors` 195 196 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 197 198 199 Type: `array` 200 Default: `[]` 201 202 ```yaml 203 # Examples 204 205 processors: 206 - archive: 207 format: lines 208 209 processors: 210 - archive: 211 format: json_array 212 213 processors: 214 - merge_json: {} 215 ``` 216 217 ### `region` 218 219 The AWS region to target. 220 221 222 Type: `string` 223 Default: `"eu-west-1"` 224 225 ### `endpoint` 226 227 Allows you to specify a custom endpoint for the AWS API. 228 229 230 Type: `string` 231 Default: `""` 232 233 ### `credentials` 234 235 Optional manual configuration of AWS credentials to use. More information can be found [in this document](/docs/guides/cloud/aws). 236 237 238 Type: `object` 239 240 ### `credentials.profile` 241 242 A profile from `~/.aws/credentials` to use. 243 244 245 Type: `string` 246 Default: `""` 247 248 ### `credentials.id` 249 250 The ID of credentials to use. 251 252 253 Type: `string` 254 Default: `""` 255 256 ### `credentials.secret` 257 258 The secret for the credentials being used. 259 260 261 Type: `string` 262 Default: `""` 263 264 ### `credentials.token` 265 266 The token for the credentials being used, required when using short term credentials. 267 268 269 Type: `string` 270 Default: `""` 271 272 ### `credentials.role` 273 274 A role ARN to assume. 275 276 277 Type: `string` 278 Default: `""` 279 280 ### `credentials.role_external_id` 281 282 An external ID to provide when assuming a role. 283 284 285 Type: `string` 286 Default: `""` 287 288 ### `max_retries` 289 290 The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. 291 292 293 Type: `int` 294 Default: `0` 295 296 ### `backoff` 297 298 Control time intervals between retry attempts. 299 300 301 Type: `object` 302 303 ### `backoff.initial_interval` 304 305 The initial period to wait between retry attempts. 306 307 308 Type: `string` 309 Default: `"1s"` 310 311 ### `backoff.max_interval` 312 313 The maximum period to wait between retry attempts. 314 315 316 Type: `string` 317 Default: `"5s"` 318 319 ### `backoff.max_elapsed_time` 320 321 The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. 322 323 324 Type: `string` 325 Default: `"30s"` 326 327