github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/dynamodb.md (about) 1 --- 2 title: dynamodb 3 type: output 4 status: deprecated 5 categories: ["Services","AWS"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/dynamodb.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 :::warning DEPRECATED 19 This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). 20 ::: 21 22 Inserts items into a DynamoDB table. 23 24 25 <Tabs defaultValue="common" values={[ 26 { label: 'Common', value: 'common', }, 27 { label: 'Advanced', value: 'advanced', }, 28 ]}> 29 30 <TabItem value="common"> 31 32 ```yaml 33 # Common config fields, showing default values 34 output: 35 label: "" 36 dynamodb: 37 table: "" 38 string_columns: {} 39 json_map_columns: {} 40 max_in_flight: 1 41 batching: 42 count: 0 43 byte_size: 0 44 period: "" 45 check: "" 46 region: eu-west-1 47 ``` 48 49 </TabItem> 50 <TabItem value="advanced"> 51 52 ```yaml 53 # All config fields, showing default values 54 output: 55 label: "" 56 dynamodb: 57 table: "" 58 string_columns: {} 59 json_map_columns: {} 60 ttl: "" 61 ttl_key: "" 62 max_in_flight: 1 63 batching: 64 count: 0 65 byte_size: 0 66 period: "" 67 check: "" 68 processors: [] 69 region: eu-west-1 70 endpoint: "" 71 credentials: 72 profile: "" 73 id: "" 74 secret: "" 75 token: "" 76 role: "" 77 role_external_id: "" 78 max_retries: 3 79 backoff: 80 initial_interval: 1s 81 max_interval: 5s 82 max_elapsed_time: 30s 83 ``` 84 85 </TabItem> 86 </Tabs> 87 88 ## Alternatives 89 90 This output has been renamed to [`aws_dynamodb`](/docs/components/outputs/aws_dynamodb). 91 92 The field `string_columns` is a map of column names to string values, 93 where the values are 94 [function interpolated](/docs/configuration/interpolation#bloblang-queries) per message of a 95 batch. This allows you to populate string columns of an item by extracting 96 fields within the document payload or metadata like follows: 97 98 ```yml 99 string_columns: 100 id: ${!json("id")} 101 title: ${!json("body.title")} 102 topic: ${!meta("kafka_topic")} 103 full_content: ${!content()} 104 ``` 105 106 The field `json_map_columns` is a map of column names to json paths, 107 where the [dot path](/docs/configuration/field_paths) is extracted from each document and 108 converted into a map value. Both an empty path and the path `.` are 109 interpreted as the root of the document. This allows you to populate map columns 110 of an item like follows: 111 112 ```yml 113 json_map_columns: 114 user: path.to.user 115 whole_document: . 116 ``` 117 118 A column name can be empty: 119 120 ```yml 121 json_map_columns: 122 "": . 123 ``` 124 125 In which case the top level document fields will be written at the root of the 126 item, potentially overwriting previously defined column values. If a path is not 127 found within a document the column will not be populated. 128 129 ### Credentials 130 131 By default Benthos will use a shared credentials file when connecting to AWS 132 services. It's also possible to set them explicitly at the component level, 133 allowing you to transfer data across accounts. You can find out more 134 [in this document](/docs/guides/cloud/aws). 135 136 ## Performance 137 138 This output benefits from sending multiple messages in flight in parallel for 139 improved performance. You can tune the max number of in flight messages with the 140 field `max_in_flight`. 141 142 This output benefits from sending messages as a batch for improved performance. 143 Batches can be formed at both the input and output level. You can find out more 144 [in this doc](/docs/configuration/batching). 145 146 ## Fields 147 148 ### `table` 149 150 The table to store messages in. 151 152 153 Type: `string` 154 Default: `""` 155 156 ### `string_columns` 157 158 A map of column keys to string values to store. 159 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 160 161 162 Type: `object` 163 Default: `{}` 164 165 ```yaml 166 # Examples 167 168 string_columns: 169 full_content: ${!content()} 170 id: ${!json("id")} 171 title: ${!json("body.title")} 172 topic: ${!meta("kafka_topic")} 173 ``` 174 175 ### `json_map_columns` 176 177 A map of column keys to [field paths](/docs/configuration/field_paths) pointing to value data within messages. 178 179 180 Type: `object` 181 Default: `{}` 182 183 ```yaml 184 # Examples 185 186 json_map_columns: 187 user: path.to.user 188 whole_document: . 189 190 json_map_columns: 191 "": . 192 ``` 193 194 ### `ttl` 195 196 An optional TTL to set for items, calculated from the moment the message is sent. 197 198 199 Type: `string` 200 Default: `""` 201 202 ### `ttl_key` 203 204 The column key to place the TTL value within. 205 206 207 Type: `string` 208 Default: `""` 209 210 ### `max_in_flight` 211 212 The maximum number of messages to have in flight at a given time. Increase this to improve throughput. 213 214 215 Type: `int` 216 Default: `1` 217 218 ### `batching` 219 220 Allows you to configure a [batching policy](/docs/configuration/batching). 221 222 223 Type: `object` 224 225 ```yaml 226 # Examples 227 228 batching: 229 byte_size: 5000 230 count: 0 231 period: 1s 232 233 batching: 234 count: 10 235 period: 1s 236 237 batching: 238 check: this.contains("END BATCH") 239 count: 0 240 period: 1m 241 ``` 242 243 ### `batching.count` 244 245 A number of messages at which the batch should be flushed. If `0` disables count based batching. 246 247 248 Type: `int` 249 Default: `0` 250 251 ### `batching.byte_size` 252 253 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 254 255 256 Type: `int` 257 Default: `0` 258 259 ### `batching.period` 260 261 A period in which an incomplete batch should be flushed regardless of its size. 262 263 264 Type: `string` 265 Default: `""` 266 267 ```yaml 268 # Examples 269 270 period: 1s 271 272 period: 1m 273 274 period: 500ms 275 ``` 276 277 ### `batching.check` 278 279 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 280 281 282 Type: `string` 283 Default: `""` 284 285 ```yaml 286 # Examples 287 288 check: this.type == "end_of_transaction" 289 ``` 290 291 ### `batching.processors` 292 293 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 294 295 296 Type: `array` 297 Default: `[]` 298 299 ```yaml 300 # Examples 301 302 processors: 303 - archive: 304 format: lines 305 306 processors: 307 - archive: 308 format: json_array 309 310 processors: 311 - merge_json: {} 312 ``` 313 314 ### `region` 315 316 The AWS region to target. 317 318 319 Type: `string` 320 Default: `"eu-west-1"` 321 322 ### `endpoint` 323 324 Allows you to specify a custom endpoint for the AWS API. 325 326 327 Type: `string` 328 Default: `""` 329 330 ### `credentials` 331 332 Optional manual configuration of AWS credentials to use. More information can be found [in this document](/docs/guides/cloud/aws). 333 334 335 Type: `object` 336 337 ### `credentials.profile` 338 339 A profile from `~/.aws/credentials` to use. 340 341 342 Type: `string` 343 Default: `""` 344 345 ### `credentials.id` 346 347 The ID of credentials to use. 348 349 350 Type: `string` 351 Default: `""` 352 353 ### `credentials.secret` 354 355 The secret for the credentials being used. 356 357 358 Type: `string` 359 Default: `""` 360 361 ### `credentials.token` 362 363 The token for the credentials being used, required when using short term credentials. 364 365 366 Type: `string` 367 Default: `""` 368 369 ### `credentials.role` 370 371 A role ARN to assume. 372 373 374 Type: `string` 375 Default: `""` 376 377 ### `credentials.role_external_id` 378 379 An external ID to provide when assuming a role. 380 381 382 Type: `string` 383 Default: `""` 384 385 ### `max_retries` 386 387 The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. 388 389 390 Type: `int` 391 Default: `3` 392 393 ### `backoff` 394 395 Control time intervals between retry attempts. 396 397 398 Type: `object` 399 400 ### `backoff.initial_interval` 401 402 The initial period to wait between retry attempts. 403 404 405 Type: `string` 406 Default: `"1s"` 407 408 ### `backoff.max_interval` 409 410 The maximum period to wait between retry attempts. 411 412 413 Type: `string` 414 Default: `"5s"` 415 416 ### `backoff.max_elapsed_time` 417 418 The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. 419 420 421 Type: `string` 422 Default: `"30s"` 423 424