github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/azure_table_storage.md (about) 1 --- 2 title: azure_table_storage 3 type: output 4 status: beta 5 categories: ["Services","Azure"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/azure_table_storage.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 :::caution BETA 19 This component is mostly stable but breaking changes could still be made outside of major version releases if a fundamental problem with the component is found. 20 ::: 21 22 Stores message parts in an Azure Table Storage table. 23 24 Introduced in version 3.36.0. 25 26 27 <Tabs defaultValue="common" values={[ 28 { label: 'Common', value: 'common', }, 29 { label: 'Advanced', value: 'advanced', }, 30 ]}> 31 32 <TabItem value="common"> 33 34 ```yaml 35 # Common config fields, showing default values 36 output: 37 label: "" 38 azure_table_storage: 39 storage_account: "" 40 storage_access_key: "" 41 storage_connection_string: "" 42 table_name: "" 43 partition_key: "" 44 row_key: "" 45 properties: {} 46 max_in_flight: 1 47 batching: 48 count: 0 49 byte_size: 0 50 period: "" 51 check: "" 52 ``` 53 54 </TabItem> 55 <TabItem value="advanced"> 56 57 ```yaml 58 # All config fields, showing default values 59 output: 60 label: "" 61 azure_table_storage: 62 storage_account: "" 63 storage_access_key: "" 64 storage_connection_string: "" 65 table_name: "" 66 partition_key: "" 67 row_key: "" 68 properties: {} 69 insert_type: INSERT 70 max_in_flight: 1 71 timeout: 5s 72 batching: 73 count: 0 74 byte_size: 0 75 period: "" 76 check: "" 77 processors: [] 78 ``` 79 80 </TabItem> 81 </Tabs> 82 83 Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. 84 85 In order to set the `table_name`, `partition_key` and `row_key` 86 you can use function interpolations described [here](/docs/configuration/interpolation#bloblang-queries), which are 87 calculated per message of a batch. 88 89 If the `properties` are not set in the config, all the `json` fields 90 are marshaled and stored in the table, which will be created if it does not exist. 91 92 The `object` and `array` fields are marshaled as strings. e.g.: 93 94 The JSON message: 95 ```json 96 { 97 "foo": 55, 98 "bar": { 99 "baz": "a", 100 "bez": "b" 101 }, 102 "diz": ["a", "b"] 103 } 104 ``` 105 106 Will store in the table the following properties: 107 ```yml 108 foo: '55' 109 bar: '{ "baz": "a", "bez": "b" }' 110 diz: '["a", "b"]' 111 ``` 112 113 It's also possible to use function interpolations to get or transform the properties values, e.g.: 114 115 ```yml 116 properties: 117 device: '${! json("device") }' 118 timestamp: '${! json("timestamp") }' 119 ``` 120 121 ## Performance 122 123 This output benefits from sending multiple messages in flight in parallel for 124 improved performance. You can tune the max number of in flight messages with the 125 field `max_in_flight`. 126 127 This output benefits from sending messages as a batch for improved performance. 128 Batches can be formed at both the input and output level. You can find out more 129 [in this doc](/docs/configuration/batching). 130 131 ## Fields 132 133 ### `storage_account` 134 135 The storage account to upload messages to. This field is ignored if `storage_connection_string` is set. 136 137 138 Type: `string` 139 Default: `""` 140 141 ### `storage_access_key` 142 143 The storage account access key. This field is ignored if `storage_connection_string` is set. 144 145 146 Type: `string` 147 Default: `""` 148 149 ### `storage_connection_string` 150 151 A storage account connection string. This field is required if `storage_account` and `storage_access_key` are not set. 152 153 154 Type: `string` 155 Default: `""` 156 157 ### `table_name` 158 159 The table to store messages into. 160 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 161 162 163 Type: `string` 164 Default: `""` 165 166 ```yaml 167 # Examples 168 169 table_name: ${!meta("kafka_topic")} 170 ``` 171 172 ### `partition_key` 173 174 The partition key. 175 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 176 177 178 Type: `string` 179 Default: `""` 180 181 ```yaml 182 # Examples 183 184 partition_key: ${!json("date")} 185 ``` 186 187 ### `row_key` 188 189 The row key. 190 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 191 192 193 Type: `string` 194 Default: `""` 195 196 ```yaml 197 # Examples 198 199 row_key: ${!json("device")}-${!uuid_v4()} 200 ``` 201 202 ### `properties` 203 204 A map of properties to store into the table. 205 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 206 207 208 Type: `object` 209 Default: `{}` 210 211 ### `insert_type` 212 213 Type of insert operation 214 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 215 216 217 Type: `string` 218 Default: `"INSERT"` 219 Options: `INSERT`, `INSERT_MERGE`, `INSERT_REPLACE`. 220 221 ### `max_in_flight` 222 223 The maximum number of messages to have in flight at a given time. Increase this to improve throughput. 224 225 226 Type: `int` 227 Default: `1` 228 229 ### `timeout` 230 231 The maximum period to wait on an upload before abandoning it and reattempting. 232 233 234 Type: `string` 235 Default: `"5s"` 236 237 ### `batching` 238 239 Allows you to configure a [batching policy](/docs/configuration/batching). 240 241 242 Type: `object` 243 244 ```yaml 245 # Examples 246 247 batching: 248 byte_size: 5000 249 count: 0 250 period: 1s 251 252 batching: 253 count: 10 254 period: 1s 255 256 batching: 257 check: this.contains("END BATCH") 258 count: 0 259 period: 1m 260 ``` 261 262 ### `batching.count` 263 264 A number of messages at which the batch should be flushed. If `0` disables count based batching. 265 266 267 Type: `int` 268 Default: `0` 269 270 ### `batching.byte_size` 271 272 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 273 274 275 Type: `int` 276 Default: `0` 277 278 ### `batching.period` 279 280 A period in which an incomplete batch should be flushed regardless of its size. 281 282 283 Type: `string` 284 Default: `""` 285 286 ```yaml 287 # Examples 288 289 period: 1s 290 291 period: 1m 292 293 period: 500ms 294 ``` 295 296 ### `batching.check` 297 298 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 299 300 301 Type: `string` 302 Default: `""` 303 304 ```yaml 305 # Examples 306 307 check: this.type == "end_of_transaction" 308 ``` 309 310 ### `batching.processors` 311 312 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 313 314 315 Type: `array` 316 Default: `[]` 317 318 ```yaml 319 # Examples 320 321 processors: 322 - archive: 323 format: lines 324 325 processors: 326 - archive: 327 format: json_array 328 329 processors: 330 - merge_json: {} 331 ``` 332 333