github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/aws_kinesis_firehose.md (about) 1 --- 2 title: aws_kinesis_firehose 3 type: output 4 status: stable 5 categories: ["Services","AWS"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/aws_kinesis_firehose.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 19 Sends messages to a Kinesis Firehose delivery stream. 20 21 Introduced in version 3.36.0. 22 23 24 <Tabs defaultValue="common" values={[ 25 { label: 'Common', value: 'common', }, 26 { label: 'Advanced', value: 'advanced', }, 27 ]}> 28 29 <TabItem value="common"> 30 31 ```yaml 32 # Common config fields, showing default values 33 output: 34 label: "" 35 aws_kinesis_firehose: 36 stream: "" 37 max_in_flight: 1 38 batching: 39 count: 0 40 byte_size: 0 41 period: "" 42 check: "" 43 region: eu-west-1 44 ``` 45 46 </TabItem> 47 <TabItem value="advanced"> 48 49 ```yaml 50 # All config fields, showing default values 51 output: 52 label: "" 53 aws_kinesis_firehose: 54 stream: "" 55 max_in_flight: 1 56 batching: 57 count: 0 58 byte_size: 0 59 period: "" 60 check: "" 61 processors: [] 62 region: eu-west-1 63 endpoint: "" 64 credentials: 65 profile: "" 66 id: "" 67 secret: "" 68 token: "" 69 role: "" 70 role_external_id: "" 71 max_retries: 0 72 backoff: 73 initial_interval: 1s 74 max_interval: 5s 75 max_elapsed_time: 30s 76 ``` 77 78 </TabItem> 79 </Tabs> 80 81 ### Credentials 82 83 By default Benthos will use a shared credentials file when connecting to AWS 84 services. It's also possible to set them explicitly at the component level, 85 allowing you to transfer data across accounts. You can find out more 86 [in this document](/docs/guides/cloud/aws). 87 88 ## Performance 89 90 This output benefits from sending multiple messages in flight in parallel for 91 improved performance. You can tune the max number of in flight messages with the 92 field `max_in_flight`. 93 94 This output benefits from sending messages as a batch for improved performance. 95 Batches can be formed at both the input and output level. You can find out more 96 [in this doc](/docs/configuration/batching). 97 98 ## Fields 99 100 ### `stream` 101 102 The stream to publish messages to. 103 104 105 Type: `string` 106 Default: `""` 107 108 ### `max_in_flight` 109 110 The maximum number of messages to have in flight at a given time. Increase this to improve throughput. 111 112 113 Type: `int` 114 Default: `1` 115 116 ### `batching` 117 118 Allows you to configure a [batching policy](/docs/configuration/batching). 119 120 121 Type: `object` 122 123 ```yaml 124 # Examples 125 126 batching: 127 byte_size: 5000 128 count: 0 129 period: 1s 130 131 batching: 132 count: 10 133 period: 1s 134 135 batching: 136 check: this.contains("END BATCH") 137 count: 0 138 period: 1m 139 ``` 140 141 ### `batching.count` 142 143 A number of messages at which the batch should be flushed. If `0` disables count based batching. 144 145 146 Type: `int` 147 Default: `0` 148 149 ### `batching.byte_size` 150 151 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 152 153 154 Type: `int` 155 Default: `0` 156 157 ### `batching.period` 158 159 A period in which an incomplete batch should be flushed regardless of its size. 160 161 162 Type: `string` 163 Default: `""` 164 165 ```yaml 166 # Examples 167 168 period: 1s 169 170 period: 1m 171 172 period: 500ms 173 ``` 174 175 ### `batching.check` 176 177 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 178 179 180 Type: `string` 181 Default: `""` 182 183 ```yaml 184 # Examples 185 186 check: this.type == "end_of_transaction" 187 ``` 188 189 ### `batching.processors` 190 191 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 192 193 194 Type: `array` 195 Default: `[]` 196 197 ```yaml 198 # Examples 199 200 processors: 201 - archive: 202 format: lines 203 204 processors: 205 - archive: 206 format: json_array 207 208 processors: 209 - merge_json: {} 210 ``` 211 212 ### `region` 213 214 The AWS region to target. 215 216 217 Type: `string` 218 Default: `"eu-west-1"` 219 220 ### `endpoint` 221 222 Allows you to specify a custom endpoint for the AWS API. 223 224 225 Type: `string` 226 Default: `""` 227 228 ### `credentials` 229 230 Optional manual configuration of AWS credentials to use. More information can be found [in this document](/docs/guides/cloud/aws). 231 232 233 Type: `object` 234 235 ### `credentials.profile` 236 237 A profile from `~/.aws/credentials` to use. 238 239 240 Type: `string` 241 Default: `""` 242 243 ### `credentials.id` 244 245 The ID of credentials to use. 246 247 248 Type: `string` 249 Default: `""` 250 251 ### `credentials.secret` 252 253 The secret for the credentials being used. 254 255 256 Type: `string` 257 Default: `""` 258 259 ### `credentials.token` 260 261 The token for the credentials being used, required when using short term credentials. 262 263 264 Type: `string` 265 Default: `""` 266 267 ### `credentials.role` 268 269 A role ARN to assume. 270 271 272 Type: `string` 273 Default: `""` 274 275 ### `credentials.role_external_id` 276 277 An external ID to provide when assuming a role. 278 279 280 Type: `string` 281 Default: `""` 282 283 ### `max_retries` 284 285 The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. 286 287 288 Type: `int` 289 Default: `0` 290 291 ### `backoff` 292 293 Control time intervals between retry attempts. 294 295 296 Type: `object` 297 298 ### `backoff.initial_interval` 299 300 The initial period to wait between retry attempts. 301 302 303 Type: `string` 304 Default: `"1s"` 305 306 ### `backoff.max_interval` 307 308 The maximum period to wait between retry attempts. 309 310 311 Type: `string` 312 Default: `"5s"` 313 314 ### `backoff.max_elapsed_time` 315 316 The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. 317 318 319 Type: `string` 320 Default: `"30s"` 321 322