github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/sql_insert.md (about) 1 --- 2 title: sql_insert 3 type: output 4 status: stable 5 categories: ["Services"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/sql_insert.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 Inserts a row into an SQL database for each message. 19 20 Introduced in version 3.59.0. 21 22 23 <Tabs defaultValue="common" values={[ 24 { label: 'Common', value: 'common', }, 25 { label: 'Advanced', value: 'advanced', }, 26 ]}> 27 28 <TabItem value="common"> 29 30 ```yaml 31 # Common config fields, showing default values 32 output: 33 label: "" 34 sql_insert: 35 driver: "" 36 dsn: "" 37 table: "" 38 columns: [] 39 args_mapping: "" 40 max_in_flight: 64 41 batching: 42 count: 0 43 byte_size: 0 44 period: "" 45 check: "" 46 ``` 47 48 </TabItem> 49 <TabItem value="advanced"> 50 51 ```yaml 52 # All config fields, showing default values 53 output: 54 label: "" 55 sql_insert: 56 driver: "" 57 dsn: "" 58 table: "" 59 columns: [] 60 args_mapping: "" 61 prefix: "" 62 suffix: "" 63 max_in_flight: 64 64 batching: 65 count: 0 66 byte_size: 0 67 period: "" 68 check: "" 69 processors: [] 70 ``` 71 72 </TabItem> 73 </Tabs> 74 75 ## Examples 76 77 <Tabs defaultValue="Table Insert (MySQL)" values={[ 78 { label: 'Table Insert (MySQL)', value: 'Table Insert (MySQL)', }, 79 ]}> 80 81 <TabItem value="Table Insert (MySQL)"> 82 83 84 Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: 85 86 ```yaml 87 output: 88 sql_insert: 89 driver: mysql 90 dsn: foouser:foopassword@tcp(localhost:3306)/foodb 91 table: footable 92 columns: [ id, name, topic ] 93 args_mapping: | 94 root = [ 95 this.user.id, 96 this.user.name, 97 meta("kafka_topic"), 98 ] 99 ``` 100 101 </TabItem> 102 </Tabs> 103 104 ## Fields 105 106 ### `driver` 107 108 A database [driver](#drivers) to use. 109 110 111 Type: `string` 112 Options: `mysql`, `postgres`, `clickhouse`, `mssql`. 113 114 ### `dsn` 115 116 A Data Source Name to identify the target database. 117 118 #### Drivers 119 120 The following is a list of supported drivers and their respective DSN formats: 121 122 | Driver | Data Source Name Format | 123 |---|---| 124 | `clickhouse` | [`tcp://[netloc][:port][?param1=value1&...¶mN=valueN]`](https://github.com/ClickHouse/clickhouse-go#dsn) 125 | `mysql` | `[username[:password]@][protocol[(address)]]/dbname[?param1=value1&...¶mN=valueN]` | 126 | `postgres` | `postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&...]` | 127 | `mssql` | `sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&...]` | 128 129 Please note that the `postgres` driver enforces SSL by default, you 130 can override this with the parameter `sslmode=disable` if required. 131 132 133 Type: `string` 134 135 ```yaml 136 # Examples 137 138 dsn: tcp://host1:9000?username=user&password=qwerty&database=clicks&read_timeout=10&write_timeout=20&alt_hosts=host2:9000,host3:9000 139 140 dsn: foouser:foopassword@tcp(localhost:3306)/foodb 141 142 dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable 143 ``` 144 145 ### `table` 146 147 The table to insert to. 148 149 150 Type: `string` 151 152 ```yaml 153 # Examples 154 155 table: foo 156 ``` 157 158 ### `columns` 159 160 A list of columns to insert. 161 162 163 Type: `array` 164 165 ```yaml 166 # Examples 167 168 columns: 169 - foo 170 - bar 171 - baz 172 ``` 173 174 ### `args_mapping` 175 176 A [Bloblang mapping](/docs/guides/bloblang/about) which should evaluate to an array of values matching in size to the number of columns specified. 177 178 179 Type: `string` 180 181 ```yaml 182 # Examples 183 184 args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] 185 186 args_mapping: root = [ meta("user.id") ] 187 ``` 188 189 ### `prefix` 190 191 An optional prefix to prepend to the insert query (before INSERT). 192 193 194 Type: `string` 195 196 ### `suffix` 197 198 An optional suffix to append to the insert query. 199 200 201 Type: `string` 202 203 ```yaml 204 # Examples 205 206 suffix: ON CONFLICT (name) DO NOTHING 207 ``` 208 209 ### `max_in_flight` 210 211 The maximum number of inserts to run in parallel. 212 213 214 Type: `int` 215 Default: `64` 216 217 ### `batching` 218 219 Allows you to configure a [batching policy](/docs/configuration/batching). 220 221 222 Type: `object` 223 224 ```yaml 225 # Examples 226 227 batching: 228 byte_size: 5000 229 count: 0 230 period: 1s 231 232 batching: 233 count: 10 234 period: 1s 235 236 batching: 237 check: this.contains("END BATCH") 238 count: 0 239 period: 1m 240 ``` 241 242 ### `batching.count` 243 244 A number of messages at which the batch should be flushed. If `0` disables count based batching. 245 246 247 Type: `int` 248 Default: `0` 249 250 ### `batching.byte_size` 251 252 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 253 254 255 Type: `int` 256 Default: `0` 257 258 ### `batching.period` 259 260 A period in which an incomplete batch should be flushed regardless of its size. 261 262 263 Type: `string` 264 Default: `""` 265 266 ```yaml 267 # Examples 268 269 period: 1s 270 271 period: 1m 272 273 period: 500ms 274 ``` 275 276 ### `batching.check` 277 278 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 279 280 281 Type: `string` 282 Default: `""` 283 284 ```yaml 285 # Examples 286 287 check: this.type == "end_of_transaction" 288 ``` 289 290 ### `batching.processors` 291 292 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 293 294 295 Type: `array` 296 297 ```yaml 298 # Examples 299 300 processors: 301 - archive: 302 format: lines 303 304 processors: 305 - archive: 306 format: json_array 307 308 processors: 309 - merge_json: {} 310 ``` 311 312