github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/processors/http.md (about) 1 --- 2 title: http 3 type: processor 4 status: stable 5 categories: ["Integration"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/processor/http.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 19 Performs an HTTP request using a message batch as the request body, and replaces 20 the original message parts with the body of the response. 21 22 23 <Tabs defaultValue="common" values={[ 24 { label: 'Common', value: 'common', }, 25 { label: 'Advanced', value: 'advanced', }, 26 ]}> 27 28 <TabItem value="common"> 29 30 ```yaml 31 # Common config fields, showing default values 32 label: "" 33 http: 34 url: http://localhost:4195/post 35 verb: POST 36 headers: 37 Content-Type: application/octet-stream 38 rate_limit: "" 39 timeout: 5s 40 parallel: false 41 ``` 42 43 </TabItem> 44 <TabItem value="advanced"> 45 46 ```yaml 47 # All config fields, showing default values 48 label: "" 49 http: 50 url: http://localhost:4195/post 51 verb: POST 52 headers: 53 Content-Type: application/octet-stream 54 metadata: 55 include_prefixes: [] 56 include_patterns: [] 57 oauth: 58 enabled: false 59 consumer_key: "" 60 consumer_secret: "" 61 access_token: "" 62 access_token_secret: "" 63 request_url: "" 64 oauth2: 65 enabled: false 66 client_key: "" 67 client_secret: "" 68 token_url: "" 69 scopes: [] 70 jwt: 71 enabled: false 72 private_key_file: "" 73 signing_method: "" 74 claims: {} 75 basic_auth: 76 enabled: false 77 username: "" 78 password: "" 79 tls: 80 enabled: false 81 skip_cert_verify: false 82 enable_renegotiation: false 83 root_cas: "" 84 root_cas_file: "" 85 client_certs: [] 86 extract_headers: 87 include_prefixes: [] 88 include_patterns: [] 89 rate_limit: "" 90 timeout: 5s 91 retry_period: 1s 92 max_retry_backoff: 300s 93 retries: 3 94 backoff_on: 95 - 429 96 drop_on: [] 97 successful_on: [] 98 proxy_url: "" 99 parallel: false 100 ``` 101 102 </TabItem> 103 </Tabs> 104 105 If a processed message batch contains more than one message they will be sent in 106 a single request as a [multipart message](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). 107 Alternatively, message batches can be sent in parallel by setting the field 108 `parallel` to `true`. 109 110 The `rate_limit` field can be used to specify a rate limit 111 [resource](/docs/components/rate_limits/about) to cap the rate of requests 112 across all parallel components service wide. 113 114 The URL and header values of this type can be dynamically set using function 115 interpolations described [here](/docs/configuration/interpolation#bloblang-queries). 116 117 In order to map or encode the payload to a specific request body, and map the 118 response back into the original payload instead of replacing it entirely, you 119 can use the [`branch` processor](/docs/components/processors/branch). 120 121 ## Response Codes 122 123 Benthos considers any response code between 200 and 299 inclusive to indicate a 124 successful response, you can add more success status codes with the field 125 `successful_on`. 126 127 When a request returns a response code within the `backoff_on` field 128 it will be retried after increasing intervals. 129 130 When a request returns a response code within the `drop_on` field it 131 will not be reattempted and is immediately considered a failed request. 132 133 ## Adding Metadata 134 135 If the request returns an error response code this processor sets a metadata 136 field `http_status_code` on the resulting message. 137 138 Use the field `extract_headers` to specify rules for which other 139 headers should be copied into the resulting message from the response. 140 141 ## Error Handling 142 143 When all retry attempts for a message are exhausted the processor cancels the 144 attempt. These failed messages will continue through the pipeline unchanged, but 145 can be dropped or placed in a dead letter queue according to your config, you 146 can read about these patterns [here](/docs/configuration/error_handling). 147 148 ## Examples 149 150 <Tabs defaultValue="Branched Request" values={[ 151 { label: 'Branched Request', value: 'Branched Request', }, 152 ]}> 153 154 <TabItem value="Branched Request"> 155 156 157 This example uses a [`branch` processor](/docs/components/processors/branch/) to strip the request message into an empty body, grab an HTTP payload, and place the result back into the original message at the path `repo.status`: 158 159 ```yaml 160 pipeline: 161 processors: 162 - branch: 163 request_map: 'root = ""' 164 processors: 165 - http: 166 url: https://hub.docker.com/v2/repositories/jeffail/benthos 167 verb: GET 168 result_map: 'root.repo.status = this' 169 ``` 170 171 </TabItem> 172 </Tabs> 173 174 ## Fields 175 176 ### `url` 177 178 The URL to connect to. 179 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 180 181 182 Type: `string` 183 Default: `"http://localhost:4195/post"` 184 185 ### `verb` 186 187 A verb to connect with 188 189 190 Type: `string` 191 Default: `"POST"` 192 193 ```yaml 194 # Examples 195 196 verb: POST 197 198 verb: GET 199 200 verb: DELETE 201 ``` 202 203 ### `headers` 204 205 A map of headers to add to the request. 206 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 207 208 209 Type: `object` 210 Default: `{"Content-Type":"application/octet-stream"}` 211 212 ```yaml 213 # Examples 214 215 headers: 216 Content-Type: application/octet-stream 217 ``` 218 219 ### `metadata` 220 221 Specify optional matching rules to determine which metadata keys should be added to the HTTP request as headers. 222 223 224 Type: `object` 225 226 ### `metadata.include_prefixes` 227 228 Provide a list of explicit metadata key prefixes to match against. 229 230 231 Type: `array` 232 Default: `[]` 233 234 ```yaml 235 # Examples 236 237 include_prefixes: 238 - foo_ 239 - bar_ 240 241 include_prefixes: 242 - kafka_ 243 244 include_prefixes: 245 - content- 246 ``` 247 248 ### `metadata.include_patterns` 249 250 Provide a list of explicit metadata key regular expression (re2) patterns to match against. 251 252 253 Type: `array` 254 Default: `[]` 255 256 ```yaml 257 # Examples 258 259 include_patterns: 260 - .* 261 262 include_patterns: 263 - _timestamp_unix$ 264 ``` 265 266 ### `oauth` 267 268 Allows you to specify open authentication via OAuth version 1. 269 270 271 Type: `object` 272 273 ### `oauth.enabled` 274 275 Whether to use OAuth version 1 in requests. 276 277 278 Type: `bool` 279 Default: `false` 280 281 ### `oauth.consumer_key` 282 283 A value used to identify the client to the service provider. 284 285 286 Type: `string` 287 Default: `""` 288 289 ### `oauth.consumer_secret` 290 291 A secret used to establish ownership of the consumer key. 292 293 294 Type: `string` 295 Default: `""` 296 297 ### `oauth.access_token` 298 299 A value used to gain access to the protected resources on behalf of the user. 300 301 302 Type: `string` 303 Default: `""` 304 305 ### `oauth.access_token_secret` 306 307 A secret provided in order to establish ownership of a given access token. 308 309 310 Type: `string` 311 Default: `""` 312 313 ### `oauth.request_url` 314 315 The URL of the OAuth provider. 316 317 318 Type: `string` 319 Default: `""` 320 321 ### `oauth2` 322 323 Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. 324 325 326 Type: `object` 327 328 ### `oauth2.enabled` 329 330 Whether to use OAuth version 2 in requests. 331 332 333 Type: `bool` 334 Default: `false` 335 336 ### `oauth2.client_key` 337 338 A value used to identify the client to the token provider. 339 340 341 Type: `string` 342 Default: `""` 343 344 ### `oauth2.client_secret` 345 346 A secret used to establish ownership of the client key. 347 348 349 Type: `string` 350 Default: `""` 351 352 ### `oauth2.token_url` 353 354 The URL of the token provider. 355 356 357 Type: `string` 358 Default: `""` 359 360 ### `oauth2.scopes` 361 362 A list of optional requested permissions. 363 364 365 Type: `array` 366 Default: `[]` 367 Requires version 3.45.0 or newer 368 369 ### `jwt` 370 371 BETA: Allows you to specify JWT authentication. 372 373 374 Type: `object` 375 376 ### `jwt.enabled` 377 378 Whether to use JWT authentication in requests. 379 380 381 Type: `bool` 382 Default: `false` 383 384 ### `jwt.private_key_file` 385 386 A file with the PEM encoded via PKCS1 or PKCS8 as private key. 387 388 389 Type: `string` 390 Default: `""` 391 392 ### `jwt.signing_method` 393 394 A method used to sign the token such as RS256, RS384 or RS512. 395 396 397 Type: `string` 398 Default: `""` 399 400 ### `jwt.claims` 401 402 A value used to identify the claims that issued the JWT. 403 404 405 Type: `object` 406 Default: `{}` 407 408 ### `basic_auth` 409 410 Allows you to specify basic authentication. 411 412 413 Type: `object` 414 415 ### `basic_auth.enabled` 416 417 Whether to use basic authentication in requests. 418 419 420 Type: `bool` 421 Default: `false` 422 423 ### `basic_auth.username` 424 425 A username to authenticate as. 426 427 428 Type: `string` 429 Default: `""` 430 431 ### `basic_auth.password` 432 433 A password to authenticate with. 434 435 436 Type: `string` 437 Default: `""` 438 439 ### `tls` 440 441 Custom TLS settings can be used to override system defaults. 442 443 444 Type: `object` 445 446 ### `tls.enabled` 447 448 Whether custom TLS settings are enabled. 449 450 451 Type: `bool` 452 Default: `false` 453 454 ### `tls.skip_cert_verify` 455 456 Whether to skip server side certificate verification. 457 458 459 Type: `bool` 460 Default: `false` 461 462 ### `tls.enable_renegotiation` 463 464 Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you're seeing the error message `local error: tls: no renegotiation`. 465 466 467 Type: `bool` 468 Default: `false` 469 Requires version 3.45.0 or newer 470 471 ### `tls.root_cas` 472 473 An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 474 475 476 Type: `string` 477 Default: `""` 478 479 ```yaml 480 # Examples 481 482 root_cas: |- 483 -----BEGIN CERTIFICATE----- 484 ... 485 -----END CERTIFICATE----- 486 ``` 487 488 ### `tls.root_cas_file` 489 490 An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 491 492 493 Type: `string` 494 Default: `""` 495 496 ```yaml 497 # Examples 498 499 root_cas_file: ./root_cas.pem 500 ``` 501 502 ### `tls.client_certs` 503 504 A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. 505 506 507 Type: `array` 508 Default: `[]` 509 510 ```yaml 511 # Examples 512 513 client_certs: 514 - cert: foo 515 key: bar 516 517 client_certs: 518 - cert_file: ./example.pem 519 key_file: ./example.key 520 ``` 521 522 ### `tls.client_certs[].cert` 523 524 A plain text certificate to use. 525 526 527 Type: `string` 528 Default: `""` 529 530 ### `tls.client_certs[].key` 531 532 A plain text certificate key to use. 533 534 535 Type: `string` 536 Default: `""` 537 538 ### `tls.client_certs[].cert_file` 539 540 The path to a certificate to use. 541 542 543 Type: `string` 544 Default: `""` 545 546 ### `tls.client_certs[].key_file` 547 548 The path of a certificate key to use. 549 550 551 Type: `string` 552 Default: `""` 553 554 ### `extract_headers` 555 556 Specify which response headers should be added to resulting messages as metadata. Header keys are lowercased before matching, so ensure that your patterns target lowercased versions of the header keys that you expect. 557 558 559 Type: `object` 560 561 ### `extract_headers.include_prefixes` 562 563 Provide a list of explicit metadata key prefixes to match against. 564 565 566 Type: `array` 567 Default: `[]` 568 569 ```yaml 570 # Examples 571 572 include_prefixes: 573 - foo_ 574 - bar_ 575 576 include_prefixes: 577 - kafka_ 578 579 include_prefixes: 580 - content- 581 ``` 582 583 ### `extract_headers.include_patterns` 584 585 Provide a list of explicit metadata key regular expression (re2) patterns to match against. 586 587 588 Type: `array` 589 Default: `[]` 590 591 ```yaml 592 # Examples 593 594 include_patterns: 595 - .* 596 597 include_patterns: 598 - _timestamp_unix$ 599 ``` 600 601 ### `rate_limit` 602 603 An optional [rate limit](/docs/components/rate_limits/about) to throttle requests by. 604 605 606 Type: `string` 607 Default: `""` 608 609 ### `timeout` 610 611 A static timeout to apply to requests. 612 613 614 Type: `string` 615 Default: `"5s"` 616 617 ### `retry_period` 618 619 The base period to wait between failed requests. 620 621 622 Type: `string` 623 Default: `"1s"` 624 625 ### `max_retry_backoff` 626 627 The maximum period to wait between failed requests. 628 629 630 Type: `string` 631 Default: `"300s"` 632 633 ### `retries` 634 635 The maximum number of retry attempts to make. 636 637 638 Type: `int` 639 Default: `3` 640 641 ### `backoff_on` 642 643 A list of status codes whereby the request should be considered to have failed and retries should be attempted, but the period between them should be increased gradually. 644 645 646 Type: `array` 647 Default: `[429]` 648 649 ### `drop_on` 650 651 A list of status codes whereby the request should be considered to have failed but retries should not be attempted. This is useful for preventing wasted retries for requests that will never succeed. Note that with these status codes the _request_ is dropped, but _message_ that caused the request will not be dropped. 652 653 654 Type: `array` 655 Default: `[]` 656 657 ### `successful_on` 658 659 A list of status codes whereby the attempt should be considered successful, this is useful for dropping requests that return non-2XX codes indicating that the message has been dealt with, such as a 303 See Other or a 409 Conflict. All 2XX codes are considered successful unless they are present within `backoff_on` or `drop_on`, regardless of this field. 660 661 662 Type: `array` 663 Default: `[]` 664 665 ### `proxy_url` 666 667 An optional HTTP proxy URL. 668 669 670 Type: `string` 671 Default: `""` 672 673 ### `parallel` 674 675 When processing batched messages, whether to send messages of the batch in parallel, otherwise they are sent within a single request. 676 677 678 Type: `bool` 679 Default: `false` 680 681