github.com/Jeffail/benthos/v3@v3.65.0/website/docs/guides/streams_mode/using_config_files.md (about) 1 --- 2 title: Streams Via Config Files 3 --- 4 5 When running Benthos in `streams` mode it's possible to create streams with their own static configurations, simply list one or more files after the `streams` subcommand: 6 7 ```sh 8 benthos streams ./foo.yaml ./configs/*.yaml 9 ``` 10 11 ## Resources 12 13 A stream configuration should only include the base stream component fields (`input`, `buffer`, `pipeline`, `output`), and therefore should NOT include any [resources][resources]. Instead, define resources separately and import them using the `-r`/`--resources` flag: 14 15 ```sh 16 benthos -r "./resources/prod/*.yaml" streams ./stream_configs/*.yaml 17 ``` 18 19 ## Walkthrough 20 21 Make a directory of stream configs: 22 23 ``` bash 24 $ mkdir ./streams 25 26 $ cat > ./streams/foo.yaml <<EOF 27 input: 28 http_server: {} 29 pipeline: 30 threads: 4 31 processors: 32 - bloblang: 'root = {"id": this.user.id, "content": this.body.content}' 33 output: 34 http_server: {} 35 EOF 36 37 $ cat > ./streams/bar.yaml <<EOF 38 input: 39 kafka: 40 addresses: 41 - localhost:9092 42 topics: 43 - my_topic 44 pipeline: 45 threads: 1 46 processors: 47 - bloblang: 'root = this.uppercase()' 48 output: 49 elasticsearch: 50 urls: 51 - http://localhost:9200 52 EOF 53 ``` 54 55 Run Benthos in streams mode, pointing to our directory of streams: 56 57 ``` bash 58 $ benthos streams ./streams/*.yaml 59 ``` 60 61 On a separate terminal you can query the set of streams loaded: 62 63 ``` bash 64 $ curl http://localhost:4195/streams | jq '.' 65 { 66 "bar": { 67 "active": true, 68 "uptime": 19.381001424, 69 "uptime_str": "19.381001552s" 70 }, 71 "foo": { 72 "active": true, 73 "uptime": 19.380582951, 74 "uptime_str": "19.380583306s" 75 } 76 } 77 ``` 78 79 You can also query a specific stream to see the loaded configuration: 80 81 ``` bash 82 $ curl http://localhost:4195/streams/foo | jq '.' 83 { 84 "active": true, 85 "uptime": 69.334717193, 86 "uptime_str": "1m9.334717193s", 87 "config": { 88 "input": { 89 "http_server": { 90 "address": "", 91 "cert_file": "", 92 "key_file": "", 93 "path": "/post", 94 "timeout": "5s" 95 } 96 }, 97 "buffer": { 98 "memory": { 99 "limit": 10000000 100 } 101 }, 102 "pipeline": { 103 "processors": [ 104 { 105 "bloblang": "root = {\"id\": this.user.id, \"content\": this.body.content}", 106 } 107 ], 108 "threads": 4 109 }, 110 "output": { 111 "http_server": { 112 "address": "", 113 "cert_file": "", 114 "key_file": "", 115 "path": "/get", 116 "stream_path": "/get/stream", 117 "timeout": "5s" 118 } 119 } 120 } 121 } 122 ``` 123 124 You can then send data to the stream via it's namespaced URL: 125 126 ``` 127 $ curl http://localhost:4195/foo/post -d '{"user":{"id":"foo"},"body":{"content":"bar"}}' 128 ``` 129 130 There are other endpoints [in the REST API][rest-api] for creating, updating and deleting streams. 131 132 [rest-api]: /docs/guides/streams_mode/using_rest_api 133 [interpolation]: /docs/configuration/interpolation 134 [resources]: /docs/configuration/resources