github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/inputs/azure_blob_storage.md (about) 1 --- 2 title: azure_blob_storage 3 type: input 4 status: beta 5 categories: ["Services","Azure"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/input/azure_blob_storage.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 :::caution BETA 19 This component is mostly stable but breaking changes could still be made outside of major version releases if a fundamental problem with the component is found. 20 ::: 21 22 Downloads objects within an Azure Blob Storage container, optionally filtered by 23 a prefix. 24 25 Introduced in version 3.36.0. 26 27 28 <Tabs defaultValue="common" values={[ 29 { label: 'Common', value: 'common', }, 30 { label: 'Advanced', value: 'advanced', }, 31 ]}> 32 33 <TabItem value="common"> 34 35 ```yaml 36 # Common config fields, showing default values 37 input: 38 label: "" 39 azure_blob_storage: 40 storage_account: "" 41 storage_access_key: "" 42 storage_sas_token: "" 43 storage_connection_string: "" 44 container: "" 45 prefix: "" 46 codec: all-bytes 47 ``` 48 49 </TabItem> 50 <TabItem value="advanced"> 51 52 ```yaml 53 # All config fields, showing default values 54 input: 55 label: "" 56 azure_blob_storage: 57 storage_account: "" 58 storage_access_key: "" 59 storage_sas_token: "" 60 storage_connection_string: "" 61 container: "" 62 prefix: "" 63 codec: all-bytes 64 delete_objects: false 65 ``` 66 67 </TabItem> 68 </Tabs> 69 70 Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix. 71 72 ## Downloading Large Files 73 74 When downloading large files it's often necessary to process it in streamed parts in order to avoid loading the entire file in memory at a given time. In order to do this a [`codec`](#codec) can be specified that determines how to break the input into smaller individual messages. 75 76 ## Metadata 77 78 This input adds the following metadata fields to each message: 79 80 ``` 81 - blob_storage_key 82 - blob_storage_container 83 - blob_storage_last_modified 84 - blob_storage_last_modified_unix 85 - blob_storage_content_type 86 - blob_storage_content_encoding 87 - All user defined metadata 88 ``` 89 90 You can access these metadata fields using [function interpolation](/docs/configuration/interpolation#metadata). 91 92 ## Fields 93 94 ### `storage_account` 95 96 The storage account to download blobs from. This field is ignored if `storage_connection_string` is set. 97 98 99 Type: `string` 100 Default: `""` 101 102 ### `storage_access_key` 103 104 The storage account access key. This field is ignored if `storage_connection_string` is set. 105 106 107 Type: `string` 108 Default: `""` 109 110 ### `storage_sas_token` 111 112 The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. 113 114 115 Type: `string` 116 Default: `""` 117 Requires version 3.38.0 or newer 118 119 ### `storage_connection_string` 120 121 A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. 122 123 124 Type: `string` 125 Default: `""` 126 127 ### `container` 128 129 The name of the container from which to download blobs. 130 131 132 Type: `string` 133 Default: `""` 134 135 ### `prefix` 136 137 An optional path prefix, if set only objects with the prefix are consumed. 138 139 140 Type: `string` 141 Default: `""` 142 143 ### `codec` 144 145 The way in which the bytes of a data source should be converted into discrete messages, codecs are useful for specifying how large files or contiunous streams of data might be processed in small chunks rather than loading it all in memory. It's possible to consume lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. Codecs can be chained with `/`, for example a gzip compressed CSV file can be consumed with the codec `gzip/csv`. 146 147 148 Type: `string` 149 Default: `"all-bytes"` 150 151 | Option | Summary | 152 |---|---| 153 | `auto` | EXPERIMENTAL: Attempts to derive a codec for each file based on information such as the extension. For example, a .tar.gz file would be consumed with the `gzip/tar` codec. Defaults to all-bytes. | 154 | `all-bytes` | Consume the entire file as a single binary message. | 155 | `chunker:x` | Consume the file in chunks of a given number of bytes. | 156 | `csv` | Consume structured rows as comma separated values, the first row must be a header row. | 157 | `csv:x` | Consume structured rows as values separated by a custom delimiter, the first row must be a header row. The custom delimiter must be a single character, e.g. the codec `"csv:\t"` would consume a tab delimited file. | 158 | `delim:x` | Consume the file in segments divided by a custom delimiter. | 159 | `gzip` | Decompress a gzip file, this codec should precede another codec, e.g. `gzip/all-bytes`, `gzip/tar`, `gzip/csv`, etc. | 160 | `lines` | Consume the file in segments divided by linebreaks. | 161 | `multipart` | Consumes the output of another codec and batches messages together. A batch ends when an empty message is consumed. For example, the codec `lines/multipart` could be used to consume multipart messages where an empty line indicates the end of each batch. | 162 | `regex:(?m)^\d\d:\d\d:\d\d` | Consume the file in segments divided by regular expression. | 163 | `tar` | Parse the file as a tar archive, and consume each file of the archive as a message. | 164 165 166 ```yaml 167 # Examples 168 169 codec: lines 170 171 codec: "delim:\t" 172 173 codec: delim:foobar 174 175 codec: gzip/csv 176 ``` 177 178 ### `delete_objects` 179 180 Whether to delete downloaded objects from the blob once they are processed. 181 182 183 Type: `bool` 184 Default: `false` 185 186