github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/inputs/http_client.md (about) 1 --- 2 title: http_client 3 type: input 4 status: stable 5 categories: ["Network"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/input/http_client.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 19 Connects to a server and continuously performs requests for a single message. 20 21 22 <Tabs defaultValue="common" values={[ 23 { label: 'Common', value: 'common', }, 24 { label: 'Advanced', value: 'advanced', }, 25 ]}> 26 27 <TabItem value="common"> 28 29 ```yaml 30 # Common config fields, showing default values 31 input: 32 label: "" 33 http_client: 34 url: http://localhost:4195/get 35 verb: GET 36 headers: 37 Content-Type: application/octet-stream 38 rate_limit: "" 39 timeout: 5s 40 payload: "" 41 stream: 42 enabled: false 43 reconnect: true 44 codec: lines 45 ``` 46 47 </TabItem> 48 <TabItem value="advanced"> 49 50 ```yaml 51 # All config fields, showing default values 52 input: 53 label: "" 54 http_client: 55 url: http://localhost:4195/get 56 verb: GET 57 headers: 58 Content-Type: application/octet-stream 59 metadata: 60 include_prefixes: [] 61 include_patterns: [] 62 oauth: 63 enabled: false 64 consumer_key: "" 65 consumer_secret: "" 66 access_token: "" 67 access_token_secret: "" 68 request_url: "" 69 oauth2: 70 enabled: false 71 client_key: "" 72 client_secret: "" 73 token_url: "" 74 scopes: [] 75 jwt: 76 enabled: false 77 private_key_file: "" 78 signing_method: "" 79 claims: {} 80 basic_auth: 81 enabled: false 82 username: "" 83 password: "" 84 tls: 85 enabled: false 86 skip_cert_verify: false 87 enable_renegotiation: false 88 root_cas: "" 89 root_cas_file: "" 90 client_certs: [] 91 extract_headers: 92 include_prefixes: [] 93 include_patterns: [] 94 rate_limit: "" 95 timeout: 5s 96 retry_period: 1s 97 max_retry_backoff: 300s 98 retries: 3 99 backoff_on: 100 - 429 101 drop_on: [] 102 successful_on: [] 103 proxy_url: "" 104 payload: "" 105 drop_empty_bodies: true 106 stream: 107 enabled: false 108 reconnect: true 109 codec: lines 110 max_buffer: 1000000 111 ``` 112 113 </TabItem> 114 </Tabs> 115 116 The URL and header values of this type can be dynamically set using function 117 interpolations described [here](/docs/configuration/interpolation#bloblang-queries). 118 119 ### Streaming 120 121 If you enable streaming then Benthos will consume the body of the response as a continuous stream of data, breaking messages out following a chosen codec. This allows you to consume APIs that provide long lived streamed data feeds (such as Twitter). 122 123 ### Pagination 124 125 This input supports interpolation functions in the `url` and `headers` fields where data from the previous successfully consumed message (if there was one) can be referenced. This can be used in order to support basic levels of pagination. However, in cases where pagination depends on logic it is recommended that you use an [`http` processor](/docs/components/processors/http) instead, often combined with a [`generate` input](/docs/components/inputs/generate) in order to schedule the processor. 126 127 ## Examples 128 129 <Tabs defaultValue="Basic Pagination" values={[ 130 { label: 'Basic Pagination', value: 'Basic Pagination', }, 131 ]}> 132 133 <TabItem value="Basic Pagination"> 134 135 Interpolation functions within the `url` and `headers` fields can be used to reference the previously consumed message, which allows simple pagination. 136 137 ```yaml 138 input: 139 http_client: 140 url: >- 141 https://api.example.com/search?query=allmyfoos&start_time=${! ( 142 (timestamp_unix()-300).format_timestamp("2006-01-02T15:04:05Z","UTC").escape_url_query() 143 ) }${! ("&next_token="+this.meta.next_token.not_null()) | "" } 144 verb: GET 145 rate_limit: foo_searches 146 oauth2: 147 enabled: true 148 token_url: https://api.example.com/oauth2/token 149 client_key: "${EXAMPLE_KEY}" 150 client_secret: "${EXAMPLE_SECRET}" 151 152 rate_limit_resources: 153 - label: foo_searches 154 local: 155 count: 1 156 interval: 30s 157 ``` 158 159 </TabItem> 160 </Tabs> 161 162 ## Fields 163 164 ### `url` 165 166 The URL to connect to. 167 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 168 169 170 Type: `string` 171 Default: `"http://localhost:4195/get"` 172 173 ### `verb` 174 175 A verb to connect with 176 177 178 Type: `string` 179 Default: `"GET"` 180 181 ```yaml 182 # Examples 183 184 verb: POST 185 186 verb: GET 187 188 verb: DELETE 189 ``` 190 191 ### `headers` 192 193 A map of headers to add to the request. 194 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 195 196 197 Type: `object` 198 Default: `{"Content-Type":"application/octet-stream"}` 199 200 ```yaml 201 # Examples 202 203 headers: 204 Content-Type: application/octet-stream 205 ``` 206 207 ### `metadata` 208 209 Specify optional matching rules to determine which metadata keys should be added to the HTTP request as headers. 210 211 212 Type: `object` 213 214 ### `metadata.include_prefixes` 215 216 Provide a list of explicit metadata key prefixes to match against. 217 218 219 Type: `array` 220 Default: `[]` 221 222 ```yaml 223 # Examples 224 225 include_prefixes: 226 - foo_ 227 - bar_ 228 229 include_prefixes: 230 - kafka_ 231 232 include_prefixes: 233 - content- 234 ``` 235 236 ### `metadata.include_patterns` 237 238 Provide a list of explicit metadata key regular expression (re2) patterns to match against. 239 240 241 Type: `array` 242 Default: `[]` 243 244 ```yaml 245 # Examples 246 247 include_patterns: 248 - .* 249 250 include_patterns: 251 - _timestamp_unix$ 252 ``` 253 254 ### `oauth` 255 256 Allows you to specify open authentication via OAuth version 1. 257 258 259 Type: `object` 260 261 ### `oauth.enabled` 262 263 Whether to use OAuth version 1 in requests. 264 265 266 Type: `bool` 267 Default: `false` 268 269 ### `oauth.consumer_key` 270 271 A value used to identify the client to the service provider. 272 273 274 Type: `string` 275 Default: `""` 276 277 ### `oauth.consumer_secret` 278 279 A secret used to establish ownership of the consumer key. 280 281 282 Type: `string` 283 Default: `""` 284 285 ### `oauth.access_token` 286 287 A value used to gain access to the protected resources on behalf of the user. 288 289 290 Type: `string` 291 Default: `""` 292 293 ### `oauth.access_token_secret` 294 295 A secret provided in order to establish ownership of a given access token. 296 297 298 Type: `string` 299 Default: `""` 300 301 ### `oauth.request_url` 302 303 The URL of the OAuth provider. 304 305 306 Type: `string` 307 Default: `""` 308 309 ### `oauth2` 310 311 Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. 312 313 314 Type: `object` 315 316 ### `oauth2.enabled` 317 318 Whether to use OAuth version 2 in requests. 319 320 321 Type: `bool` 322 Default: `false` 323 324 ### `oauth2.client_key` 325 326 A value used to identify the client to the token provider. 327 328 329 Type: `string` 330 Default: `""` 331 332 ### `oauth2.client_secret` 333 334 A secret used to establish ownership of the client key. 335 336 337 Type: `string` 338 Default: `""` 339 340 ### `oauth2.token_url` 341 342 The URL of the token provider. 343 344 345 Type: `string` 346 Default: `""` 347 348 ### `oauth2.scopes` 349 350 A list of optional requested permissions. 351 352 353 Type: `array` 354 Default: `[]` 355 Requires version 3.45.0 or newer 356 357 ### `jwt` 358 359 BETA: Allows you to specify JWT authentication. 360 361 362 Type: `object` 363 364 ### `jwt.enabled` 365 366 Whether to use JWT authentication in requests. 367 368 369 Type: `bool` 370 Default: `false` 371 372 ### `jwt.private_key_file` 373 374 A file with the PEM encoded via PKCS1 or PKCS8 as private key. 375 376 377 Type: `string` 378 Default: `""` 379 380 ### `jwt.signing_method` 381 382 A method used to sign the token such as RS256, RS384 or RS512. 383 384 385 Type: `string` 386 Default: `""` 387 388 ### `jwt.claims` 389 390 A value used to identify the claims that issued the JWT. 391 392 393 Type: `object` 394 Default: `{}` 395 396 ### `basic_auth` 397 398 Allows you to specify basic authentication. 399 400 401 Type: `object` 402 403 ### `basic_auth.enabled` 404 405 Whether to use basic authentication in requests. 406 407 408 Type: `bool` 409 Default: `false` 410 411 ### `basic_auth.username` 412 413 A username to authenticate as. 414 415 416 Type: `string` 417 Default: `""` 418 419 ### `basic_auth.password` 420 421 A password to authenticate with. 422 423 424 Type: `string` 425 Default: `""` 426 427 ### `tls` 428 429 Custom TLS settings can be used to override system defaults. 430 431 432 Type: `object` 433 434 ### `tls.enabled` 435 436 Whether custom TLS settings are enabled. 437 438 439 Type: `bool` 440 Default: `false` 441 442 ### `tls.skip_cert_verify` 443 444 Whether to skip server side certificate verification. 445 446 447 Type: `bool` 448 Default: `false` 449 450 ### `tls.enable_renegotiation` 451 452 Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you're seeing the error message `local error: tls: no renegotiation`. 453 454 455 Type: `bool` 456 Default: `false` 457 Requires version 3.45.0 or newer 458 459 ### `tls.root_cas` 460 461 An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 462 463 464 Type: `string` 465 Default: `""` 466 467 ```yaml 468 # Examples 469 470 root_cas: |- 471 -----BEGIN CERTIFICATE----- 472 ... 473 -----END CERTIFICATE----- 474 ``` 475 476 ### `tls.root_cas_file` 477 478 An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 479 480 481 Type: `string` 482 Default: `""` 483 484 ```yaml 485 # Examples 486 487 root_cas_file: ./root_cas.pem 488 ``` 489 490 ### `tls.client_certs` 491 492 A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. 493 494 495 Type: `array` 496 Default: `[]` 497 498 ```yaml 499 # Examples 500 501 client_certs: 502 - cert: foo 503 key: bar 504 505 client_certs: 506 - cert_file: ./example.pem 507 key_file: ./example.key 508 ``` 509 510 ### `tls.client_certs[].cert` 511 512 A plain text certificate to use. 513 514 515 Type: `string` 516 Default: `""` 517 518 ### `tls.client_certs[].key` 519 520 A plain text certificate key to use. 521 522 523 Type: `string` 524 Default: `""` 525 526 ### `tls.client_certs[].cert_file` 527 528 The path to a certificate to use. 529 530 531 Type: `string` 532 Default: `""` 533 534 ### `tls.client_certs[].key_file` 535 536 The path of a certificate key to use. 537 538 539 Type: `string` 540 Default: `""` 541 542 ### `extract_headers` 543 544 Specify which response headers should be added to resulting messages as metadata. Header keys are lowercased before matching, so ensure that your patterns target lowercased versions of the header keys that you expect. 545 546 547 Type: `object` 548 549 ### `extract_headers.include_prefixes` 550 551 Provide a list of explicit metadata key prefixes to match against. 552 553 554 Type: `array` 555 Default: `[]` 556 557 ```yaml 558 # Examples 559 560 include_prefixes: 561 - foo_ 562 - bar_ 563 564 include_prefixes: 565 - kafka_ 566 567 include_prefixes: 568 - content- 569 ``` 570 571 ### `extract_headers.include_patterns` 572 573 Provide a list of explicit metadata key regular expression (re2) patterns to match against. 574 575 576 Type: `array` 577 Default: `[]` 578 579 ```yaml 580 # Examples 581 582 include_patterns: 583 - .* 584 585 include_patterns: 586 - _timestamp_unix$ 587 ``` 588 589 ### `rate_limit` 590 591 An optional [rate limit](/docs/components/rate_limits/about) to throttle requests by. 592 593 594 Type: `string` 595 Default: `""` 596 597 ### `timeout` 598 599 A static timeout to apply to requests. 600 601 602 Type: `string` 603 Default: `"5s"` 604 605 ### `retry_period` 606 607 The base period to wait between failed requests. 608 609 610 Type: `string` 611 Default: `"1s"` 612 613 ### `max_retry_backoff` 614 615 The maximum period to wait between failed requests. 616 617 618 Type: `string` 619 Default: `"300s"` 620 621 ### `retries` 622 623 The maximum number of retry attempts to make. 624 625 626 Type: `int` 627 Default: `3` 628 629 ### `backoff_on` 630 631 A list of status codes whereby the request should be considered to have failed and retries should be attempted, but the period between them should be increased gradually. 632 633 634 Type: `array` 635 Default: `[429]` 636 637 ### `drop_on` 638 639 A list of status codes whereby the request should be considered to have failed but retries should not be attempted. This is useful for preventing wasted retries for requests that will never succeed. Note that with these status codes the _request_ is dropped, but _message_ that caused the request will not be dropped. 640 641 642 Type: `array` 643 Default: `[]` 644 645 ### `successful_on` 646 647 A list of status codes whereby the attempt should be considered successful, this is useful for dropping requests that return non-2XX codes indicating that the message has been dealt with, such as a 303 See Other or a 409 Conflict. All 2XX codes are considered successful unless they are present within `backoff_on` or `drop_on`, regardless of this field. 648 649 650 Type: `array` 651 Default: `[]` 652 653 ### `proxy_url` 654 655 An optional HTTP proxy URL. 656 657 658 Type: `string` 659 Default: `""` 660 661 ### `payload` 662 663 An optional payload to deliver for each request. 664 665 666 Type: `string` 667 Default: `""` 668 669 ### `drop_empty_bodies` 670 671 Whether empty payloads received from the target server should be dropped. 672 673 674 Type: `bool` 675 Default: `true` 676 677 ### `stream` 678 679 Allows you to set streaming mode, where requests are kept open and messages are processed line-by-line. 680 681 682 Type: `object` 683 684 ### `stream.enabled` 685 686 Enables streaming mode. 687 688 689 Type: `bool` 690 Default: `false` 691 692 ### `stream.reconnect` 693 694 Sets whether to re-establish the connection once it is lost. 695 696 697 Type: `bool` 698 Default: `true` 699 700 ### `stream.codec` 701 702 The way in which the bytes of a continuous stream are converted into messages. It's possible to consume lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. It's not necessary to add gzip in the codec when the response headers specify it as it will be decompressed automatically. 703 704 705 Type: `string` 706 Default: `"lines"` 707 Requires version 3.42.0 or newer 708 709 | Option | Summary | 710 |---|---| 711 | `auto` | EXPERIMENTAL: Attempts to derive a codec for each file based on information such as the extension. For example, a .tar.gz file would be consumed with the `gzip/tar` codec. Defaults to all-bytes. | 712 | `all-bytes` | Consume the entire file as a single binary message. | 713 | `chunker:x` | Consume the file in chunks of a given number of bytes. | 714 | `csv` | Consume structured rows as comma separated values, the first row must be a header row. | 715 | `csv:x` | Consume structured rows as values separated by a custom delimiter, the first row must be a header row. The custom delimiter must be a single character, e.g. the codec `"csv:\t"` would consume a tab delimited file. | 716 | `delim:x` | Consume the file in segments divided by a custom delimiter. | 717 | `gzip` | Decompress a gzip file, this codec should precede another codec, e.g. `gzip/all-bytes`, `gzip/tar`, `gzip/csv`, etc. | 718 | `lines` | Consume the file in segments divided by linebreaks. | 719 | `multipart` | Consumes the output of another codec and batches messages together. A batch ends when an empty message is consumed. For example, the codec `lines/multipart` could be used to consume multipart messages where an empty line indicates the end of each batch. | 720 | `regex:(?m)^\d\d:\d\d:\d\d` | Consume the file in segments divided by regular expression. | 721 | `tar` | Parse the file as a tar archive, and consume each file of the archive as a message. | 722 723 724 ```yaml 725 # Examples 726 727 codec: lines 728 729 codec: "delim:\t" 730 731 codec: delim:foobar 732 733 codec: csv 734 ``` 735 736 ### `stream.max_buffer` 737 738 Must be larger than the largest line of the stream. 739 740 741 Type: `int` 742 Default: `1000000` 743 744