github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/s3.md (about) 1 --- 2 title: s3 3 type: output 4 status: deprecated 5 categories: ["Services","AWS"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/s3.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 :::warning DEPRECATED 19 This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). 20 ::: 21 22 Sends message parts as objects to an Amazon S3 bucket. Each object is uploaded 23 with the path specified with the `path` field. 24 25 26 <Tabs defaultValue="common" values={[ 27 { label: 'Common', value: 'common', }, 28 { label: 'Advanced', value: 'advanced', }, 29 ]}> 30 31 <TabItem value="common"> 32 33 ```yaml 34 # Common config fields, showing default values 35 output: 36 label: "" 37 s3: 38 bucket: "" 39 path: ${!count("files")}-${!timestamp_unix_nano()}.txt 40 tags: {} 41 content_type: application/octet-stream 42 metadata: 43 exclude_prefixes: [] 44 max_in_flight: 1 45 batching: 46 count: 0 47 byte_size: 0 48 period: "" 49 check: "" 50 region: eu-west-1 51 ``` 52 53 </TabItem> 54 <TabItem value="advanced"> 55 56 ```yaml 57 # All config fields, showing default values 58 output: 59 label: "" 60 s3: 61 bucket: "" 62 path: ${!count("files")}-${!timestamp_unix_nano()}.txt 63 tags: {} 64 content_type: application/octet-stream 65 content_encoding: "" 66 cache_control: "" 67 content_disposition: "" 68 content_language: "" 69 website_redirect_location: "" 70 metadata: 71 exclude_prefixes: [] 72 storage_class: STANDARD 73 kms_key_id: "" 74 server_side_encryption: "" 75 force_path_style_urls: false 76 max_in_flight: 1 77 timeout: 5s 78 batching: 79 count: 0 80 byte_size: 0 81 period: "" 82 check: "" 83 processors: [] 84 region: eu-west-1 85 endpoint: "" 86 credentials: 87 profile: "" 88 id: "" 89 secret: "" 90 token: "" 91 role: "" 92 role_external_id: "" 93 ``` 94 95 </TabItem> 96 </Tabs> 97 98 ## Alternatives 99 100 This output has been renamed to [`aws_s3`](/docs/components/outputs/aws_s3). 101 102 In order to have a different path for each object you should use function 103 interpolations described [here](/docs/configuration/interpolation#bloblang-queries), which are 104 calculated per message of a batch. 105 106 ### Metadata 107 108 Metadata fields on messages will be sent as headers, in order to mutate these values (or remove them) check out the [metadata docs](/docs/configuration/metadata). 109 110 ### Tags 111 112 The tags field allows you to specify key/value pairs to attach to objects as tags, where the values support 113 [interpolation functions](/docs/configuration/interpolation#bloblang-queries): 114 115 ```yaml 116 output: 117 aws_s3: 118 bucket: TODO 119 path: ${!count("files")}-${!timestamp_unix_nano()}.tar.gz 120 tags: 121 Key1: Value1 122 Timestamp: ${!meta("Timestamp")} 123 ``` 124 125 ### Credentials 126 127 By default Benthos will use a shared credentials file when connecting to AWS 128 services. It's also possible to set them explicitly at the component level, 129 allowing you to transfer data across accounts. You can find out more 130 [in this document](/docs/guides/cloud/aws). 131 132 ### Batching 133 134 It's common to want to upload messages to S3 as batched archives, the easiest 135 way to do this is to batch your messages at the output level and join the batch 136 of messages with an 137 [`archive`](/docs/components/processors/archive) and/or 138 [`compress`](/docs/components/processors/compress) processor. 139 140 For example, if we wished to upload messages as a .tar.gz archive of documents 141 we could achieve that with the following config: 142 143 ```yaml 144 output: 145 s3: 146 bucket: TODO 147 path: ${!count("files")}-${!timestamp_unix_nano()}.tar.gz 148 batching: 149 count: 100 150 period: 10s 151 processors: 152 - archive: 153 format: tar 154 - compress: 155 algorithm: gzip 156 ``` 157 158 Alternatively, if we wished to upload JSON documents as a single large document 159 containing an array of objects we can do that with: 160 161 ```yaml 162 output: 163 s3: 164 bucket: TODO 165 path: ${!count("files")}-${!timestamp_unix_nano()}.json 166 batching: 167 count: 100 168 processors: 169 - archive: 170 format: json_array 171 ``` 172 173 ## Performance 174 175 This output benefits from sending multiple messages in flight in parallel for 176 improved performance. You can tune the max number of in flight messages with the 177 field `max_in_flight`. 178 179 ## Fields 180 181 ### `bucket` 182 183 The bucket to upload messages to. 184 185 186 Type: `string` 187 Default: `""` 188 189 ### `path` 190 191 The path of each message to upload. 192 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 193 194 195 Type: `string` 196 Default: `"${!count(\"files\")}-${!timestamp_unix_nano()}.txt"` 197 198 ```yaml 199 # Examples 200 201 path: ${!count("files")}-${!timestamp_unix_nano()}.txt 202 203 path: ${!meta("kafka_key")}.json 204 205 path: ${!json("doc.namespace")}/${!json("doc.id")}.json 206 ``` 207 208 ### `tags` 209 210 Key/value pairs to store with the object as tags. 211 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 212 213 214 Type: `object` 215 Default: `{}` 216 217 ```yaml 218 # Examples 219 220 tags: 221 Key1: Value1 222 Timestamp: ${!meta("Timestamp")} 223 ``` 224 225 ### `content_type` 226 227 The content type to set for each object. 228 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 229 230 231 Type: `string` 232 Default: `"application/octet-stream"` 233 234 ### `content_encoding` 235 236 An optional content encoding to set for each object. 237 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 238 239 240 Type: `string` 241 Default: `""` 242 243 ### `cache_control` 244 245 The cache control to set for each object. 246 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 247 248 249 Type: `string` 250 Default: `""` 251 252 ### `content_disposition` 253 254 The content disposition to set for each object. 255 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 256 257 258 Type: `string` 259 Default: `""` 260 261 ### `content_language` 262 263 The content language to set for each object. 264 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 265 266 267 Type: `string` 268 Default: `""` 269 270 ### `website_redirect_location` 271 272 The website redirect location to set for each object. 273 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 274 275 276 Type: `string` 277 Default: `""` 278 279 ### `metadata` 280 281 Specify criteria for which metadata values are attached to objects as headers. 282 283 284 Type: `object` 285 286 ### `metadata.exclude_prefixes` 287 288 Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. 289 290 291 Type: `array` 292 Default: `[]` 293 294 ### `storage_class` 295 296 The storage class to set for each object. 297 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 298 299 300 Type: `string` 301 Default: `"STANDARD"` 302 Options: `STANDARD`, `REDUCED_REDUNDANCY`, `GLACIER`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `DEEP_ARCHIVE`. 303 304 ### `kms_key_id` 305 306 An optional server side encryption key. 307 308 309 Type: `string` 310 Default: `""` 311 312 ### `server_side_encryption` 313 314 An optional server side encryption algorithm. 315 316 317 Type: `string` 318 Default: `""` 319 320 ### `force_path_style_urls` 321 322 Forces the client API to use path style URLs, which helps when connecting to custom endpoints. 323 324 325 Type: `bool` 326 Default: `false` 327 328 ### `max_in_flight` 329 330 The maximum number of messages to have in flight at a given time. Increase this to improve throughput. 331 332 333 Type: `int` 334 Default: `1` 335 336 ### `timeout` 337 338 The maximum period to wait on an upload before abandoning it and reattempting. 339 340 341 Type: `string` 342 Default: `"5s"` 343 344 ### `batching` 345 346 Allows you to configure a [batching policy](/docs/configuration/batching). 347 348 349 Type: `object` 350 351 ```yaml 352 # Examples 353 354 batching: 355 byte_size: 5000 356 count: 0 357 period: 1s 358 359 batching: 360 count: 10 361 period: 1s 362 363 batching: 364 check: this.contains("END BATCH") 365 count: 0 366 period: 1m 367 ``` 368 369 ### `batching.count` 370 371 A number of messages at which the batch should be flushed. If `0` disables count based batching. 372 373 374 Type: `int` 375 Default: `0` 376 377 ### `batching.byte_size` 378 379 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 380 381 382 Type: `int` 383 Default: `0` 384 385 ### `batching.period` 386 387 A period in which an incomplete batch should be flushed regardless of its size. 388 389 390 Type: `string` 391 Default: `""` 392 393 ```yaml 394 # Examples 395 396 period: 1s 397 398 period: 1m 399 400 period: 500ms 401 ``` 402 403 ### `batching.check` 404 405 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 406 407 408 Type: `string` 409 Default: `""` 410 411 ```yaml 412 # Examples 413 414 check: this.type == "end_of_transaction" 415 ``` 416 417 ### `batching.processors` 418 419 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 420 421 422 Type: `array` 423 Default: `[]` 424 425 ```yaml 426 # Examples 427 428 processors: 429 - archive: 430 format: lines 431 432 processors: 433 - archive: 434 format: json_array 435 436 processors: 437 - merge_json: {} 438 ``` 439 440 ### `region` 441 442 The AWS region to target. 443 444 445 Type: `string` 446 Default: `"eu-west-1"` 447 448 ### `endpoint` 449 450 Allows you to specify a custom endpoint for the AWS API. 451 452 453 Type: `string` 454 Default: `""` 455 456 ### `credentials` 457 458 Optional manual configuration of AWS credentials to use. More information can be found [in this document](/docs/guides/cloud/aws). 459 460 461 Type: `object` 462 463 ### `credentials.profile` 464 465 A profile from `~/.aws/credentials` to use. 466 467 468 Type: `string` 469 Default: `""` 470 471 ### `credentials.id` 472 473 The ID of credentials to use. 474 475 476 Type: `string` 477 Default: `""` 478 479 ### `credentials.secret` 480 481 The secret for the credentials being used. 482 483 484 Type: `string` 485 Default: `""` 486 487 ### `credentials.token` 488 489 The token for the credentials being used, required when using short term credentials. 490 491 492 Type: `string` 493 Default: `""` 494 495 ### `credentials.role` 496 497 A role ARN to assume. 498 499 500 Type: `string` 501 Default: `""` 502 503 ### `credentials.role_external_id` 504 505 An external ID to provide when assuming a role. 506 507 508 Type: `string` 509 Default: `""` 510 511