github.com/Jeffail/benthos/v3@v3.65.0/README.md (about) 1 ![Benthos](icon.png "Benthos") 2 3 [![godoc for Jeffail/benthos][godoc-badge]][godoc-url] 4 [![goreportcard for Jeffail/benthos][goreport-badge]][goreport-url] 5 [![Build Status][actions-badge]][actions-url] 6 [![Discord invite][discord-badge]][discord-url] 7 [![Docs site][website-badge]][website-url] 8 9 Benthos is a high performance and resilient stream processor, able to connect various [sources][inputs] and [sinks][outputs] in a range of brokering patterns and perform [hydration, enrichments, transformations and filters][processors] on payloads. 10 11 It comes with a [powerful mapping language][bloblang-about], is easy to deploy and monitor, and ready to drop into your pipeline either as a static binary, docker image, or [serverless function][serverless], making it cloud native as heck. 12 13 Benthos is fully declarative, with stream pipelines defined in a single config file, allowing you to specify connectors and a list of processing stages: 14 15 ```yaml 16 input: 17 gcp_pubsub: 18 project: foo 19 subscription: bar 20 21 pipeline: 22 processors: 23 - bloblang: | 24 root.message = this 25 root.meta.link_count = this.links.length() 26 root.user.age = this.user.age.number() 27 28 output: 29 redis_streams: 30 url: tcp://TODO:6379 31 stream: baz 32 max_in_flight: 20 33 ``` 34 35 ### Delivery Guarantees 36 37 Yep, we got 'em. Benthos implements transaction based resiliency with back pressure. When connecting to at-least-once sources and sinks it guarantees at-least-once delivery without needing to persist messages during transit. 38 39 ## Supported Sources & Sinks 40 41 Apache Pulsar, AWS (DynamoDB, Kinesis, S3, SQS, SNS), Azure (Blob storage, Queue storage, Table storage), Cassandra, Elasticsearch, File, GCP (Pub/Sub, Cloud storage), HDFS, HTTP (server and client, including websockets), Kafka, Memcached, MQTT, Nanomsg, NATS, NATS JetStream, NATS Streaming, NSQ, AMQP 0.91 (RabbitMQ), AMQP 1, Redis (streams, list, pubsub, hashes), MongoDB, SQL (MySQL, PostgreSQL, Clickhouse, MSSQL), Stdin/Stdout, TCP & UDP, sockets and ZMQ4. 42 43 Connectors are being added constantly, if something you want is missing then [open an issue](https://github.com/Jeffail/benthos/issues/new). 44 45 ## Documentation 46 47 If you want to dive fully into Benthos then don't waste your time in this dump, check out the [documentation site][general-docs]. 48 49 For guidance on how to configure more advanced stream processing concepts such as stream joins, enrichment workflows, etc, check out the [cookbooks section.][cookbooks] 50 51 For guidance on building your own custom plugins in Go check out [the public APIs.][godoc-url] 52 53 ## Install 54 55 Grab a binary for your OS from [here.][releases] Or use this script: 56 57 ```shell 58 curl -Lsf https://sh.benthos.dev | bash 59 ``` 60 61 Or pull the docker image: 62 63 ```shell 64 docker pull jeffail/benthos 65 ``` 66 67 Benthos can also be installed via Homebrew: 68 69 ```shell 70 brew install benthos 71 ``` 72 73 For more information check out the [getting started guide][getting-started]. 74 75 ## Run 76 77 ```shell 78 benthos -c ./config.yaml 79 ``` 80 81 Or, with docker: 82 83 ```shell 84 # Using a config file 85 docker run --rm -v /path/to/your/config.yaml:/benthos.yaml jeffail/benthos 86 87 # Using a series of -s flags 88 docker run --rm -p 4195:4195 jeffail/benthos \ 89 -s "input.type=http_server" \ 90 -s "output.type=kafka" \ 91 -s "output.kafka.addresses=kafka-server:9092" \ 92 -s "output.kafka.topic=benthos_topic" 93 ``` 94 95 ## Monitoring 96 97 ### Health Checks 98 99 Benthos serves two HTTP endpoints for health checks: 100 - `/ping` can be used as a liveness probe as it always returns a 200. 101 - `/ready` can be used as a readiness probe as it serves a 200 only when both the input and output are connected, otherwise a 503 is returned. 102 103 ### Metrics 104 105 Benthos [exposes lots of metrics][metrics] either to Statsd, Prometheus or for debugging purposes an HTTP endpoint that returns a JSON formatted object. 106 107 ### Tracing 108 109 Benthos also [emits tracing events][tracers] to a tracer of your choice (currently only [Jaeger][jaeger] is supported) which can be used to visualise the processors within a pipeline. 110 111 ## Configuration 112 113 Benthos provides lots of tools for making configuration discovery, debugging and organisation easy. You can [read about them here][config-doc]. 114 115 ## Build 116 117 Build with Go (1.16 or later): 118 119 ```shell 120 git clone git@github.com:Jeffail/benthos 121 cd benthos 122 make 123 ``` 124 125 ## Lint 126 127 Benthos uses [golangci-lint][golangci-lint] for linting, which you can install with: 128 129 ```shell 130 curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin 131 ``` 132 133 And then run it with `make lint`. 134 135 ### Plugins 136 137 It's pretty easy to write your own custom plugins for Benthos in Go, for information check out [the API docs][godoc-url], and for inspiration there's an [example repo][plugin-repo] demonstrating a variety of plugin implementations. 138 139 ### Docker Builds 140 141 There's a multi-stage `Dockerfile` for creating a Benthos docker image which results in a minimal image from scratch. You can build it with: 142 143 ```shell 144 make docker 145 ``` 146 147 Then use the image: 148 149 ```shell 150 docker run --rm \ 151 -v /path/to/your/benthos.yaml:/config.yaml \ 152 -v /tmp/data:/data \ 153 -p 4195:4195 \ 154 benthos -c /config.yaml 155 ``` 156 157 ### ZMQ4 Support 158 159 Benthos supports ZMQ4 for both data input and output. To add this you need to install libzmq4 and use the compile time flag when building Benthos: 160 161 ```shell 162 make TAGS=ZMQ4 163 ``` 164 165 Or to build a docker image using CGO, which includes ZMQ: 166 167 ```shell 168 make docker-cgo 169 ``` 170 171 ## Contributing 172 173 Contributions are welcome, please [read the guidelines](CONTRIBUTING.md), come and chat (links are on the [community page][community]), and watch your back. 174 175 [inputs]: https://www.benthos.dev/docs/components/inputs/about 176 [processors]: https://www.benthos.dev/docs/components/processors/about 177 [outputs]: https://www.benthos.dev/docs/components/outputs/about 178 [metrics]: https://www.benthos.dev/docs/components/metrics/about 179 [tracers]: https://www.benthos.dev/docs/components/tracers/about 180 [config-interp]: https://www.benthos.dev/docs/configuration/interpolation 181 [streams-api]: https://www.benthos.dev/docs/guides/streams_mode/streams_api 182 [streams-mode]: https://www.benthos.dev/docs/guides/streams_mode/about 183 [general-docs]: https://www.benthos.dev/docs/about 184 [bloblang-about]: https://www.benthos.dev/docs/guides/bloblang/about 185 [config-doc]: https://www.benthos.dev/docs/configuration/about 186 [serverless]: https://www.benthos.dev/docs/guides/serverless/about 187 [cookbooks]: https://www.benthos.dev/cookbooks 188 [releases]: https://github.com/Jeffail/benthos/releases 189 [plugin-repo]: https://github.com/benthosdev/benthos-plugin-example 190 [getting-started]: https://www.benthos.dev/docs/guides/getting_started 191 192 [godoc-badge]: https://pkg.go.dev/badge/github.com/Jeffail/benthos/v3/public 193 [godoc-url]: https://pkg.go.dev/github.com/Jeffail/benthos/v3/public 194 [goreport-badge]: https://goreportcard.com/badge/github.com/Jeffail/benthos 195 [goreport-url]: https://goreportcard.com/report/Jeffail/benthos 196 [actions-badge]: https://github.com/Jeffail/benthos/actions/workflows/test.yml/badge.svg 197 [actions-url]: https://github.com/Jeffail/benthos/actions/workflows/test.yml 198 [discord-badge]: https://img.shields.io/badge/Discord-Come%20and%20chill-blue 199 [discord-url]: https://discord.com/invite/6VaWjzP 200 [website-badge]: https://img.shields.io/badge/Docs-Learn%20more-ffc7c7 201 [website-url]: https://www.benthos.dev 202 203 [community]: https://www.benthos.dev/community 204 205 [golangci-lint]: https://golangci-lint.run/ 206 [jaeger]: https://www.jaegertracing.io/