github.com/segmentio/kafka-go@v0.4.48-0.20240318174348-3f6244eb34fd/docker_compose_versions/README.md (about)

     1  # Bitnami Kafka
     2  
     3  This document outlines how to create a docker-compose file for a specific Bitnami Kafka version.
     4  
     5  
     6  ## Steps to create docker-compose
     7  
     8  - Refer to [docker-hub Bitnami Kafka tags](https://hub.docker.com/r/bitnami/kafka/tags) and sort by NEWEST to locate the image preferred, for example: `2.7.0`
     9  - There is documentation in the (main branch)[https://github.com/bitnami/containers/blob/main/bitnami/kafka/README.md] for environment config setup information. Refer to the `Notable Changes` section.
    10  - Sometimes there is a need to understand how the set up is being done. To locate the appropriate Kafka release in the repo [bitnami/containers](https://github.com/bitnami/containers), go through the [kafka commit history](https://github.com/bitnami/containers/commits/main/bitnami/kafka).
    11  - Once a commit is located, Refer to README.md, Dockerfile, entrypoint and various init scripts to understand the environment variables to config server.properties mapping conventions. Alternatively, you can spin up the required Kafka image and refer the mapping inside the container.
    12  - Ensure you follow the environment variable conventions in your docker-compose. Without proper environment variables, the Kafka cluster cannot start or can start with undesired configs. For example, Since Kafka version 2.3, all server.properties docker-compose environment configs start with `KAFKA_CFG_<config_with_underscore>`
    13  - Older versions of Bitnami Kafka have different conventions and limited docker-compose environment variables exposed for configs needed in server.properties
    14  
    15  
    16  In kafka-go, for all the test cases to succeed, Kafka cluster should have following server.properties along with a relevant kafka_jaas.conf mentioned in the KAFKA_OPTS. Goal is to ensure that the docker-compose file generates below server.properties.
    17  
    18  
    19  server.properties
    20  ```
    21  advertised.host.name=localhost
    22  advertised.listeners=PLAINTEXT://localhost:9092,SASL_PLAINTEXT://localhost:9093
    23  advertised.port=9092
    24  auto.create.topics.enable=true
    25  broker.id=1
    26  delete.topic.enable=true
    27  group.initial.rebalance.delay.ms=0
    28  listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
    29  log.dirs=/kafka/kafka-logs-1d5951569d78
    30  log.retention.check.interval.ms=300000
    31  log.retention.hours=168
    32  log.segment.bytes=1073741824
    33  message.max.bytes=200000000
    34  num.io.threads=8
    35  num.network.threads=3
    36  num.partitions=1
    37  num.recovery.threads.per.data.dir=1
    38  offsets.topic.replication.factor=1
    39  port=9092
    40  sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
    41  socket.receive.buffer.bytes=102400
    42  socket.request.max.bytes=104857600
    43  socket.send.buffer.bytes=102400
    44  transaction.state.log.min.isr=1
    45  transaction.state.log.replication.factor=1
    46  zookeeper.connect=zookeeper:2181
    47  zookeeper.connection.timeout.ms=6000
    48  ```
    49  
    50  
    51  ## run docker-compose and test cases
    52  
    53  run docker-compose
    54  ```
    55  # docker-compose -f ./docker_compose_versions/docker-compose-<kafka_version>.yml up -d
    56  ```
    57  
    58  
    59  run test cases
    60  ```
    61  # go clean -cache; KAFKA_SKIP_NETTEST=1 KAFKA_VERSION=<a.b.c> go test -race -cover ./...;
    62  ```
    63  
    64  
    65  ## Various Bitnami Kafka version issues observed in circleci
    66  
    67  
    68  ### Kafka v101, v111, v201, v211 and v221
    69  
    70  
    71  In kafka-go repo, all the tests require sasl.enabled.mechanisms as PLAIN,SCRAM-SHA-256,SCRAM-SHA-512 for the Kafka cluster.
    72  
    73  
    74  It has been observed for Kafka v101, v111, v201, v211 and v221 which are used in the circleci for build have issues with SCRAM.
    75  
    76  
    77  There is no way to override the config sasl.enabled.mechanisms causing Kafka cluster to start up as PLAIN.
    78  
    79  
    80  There has been some attempts made to override sasl.enabled.mechanisms 
    81  - Modified entrypoint in docker-compose to append the server.properties with relevant configs sasl.enabled.mechanisms before running entrypoint.sh. This resulted in failures for Kafka v101, v111, v201, v211 and v221. Once Kafka server starts, server.properties gets appended with default value of sasl.enabled.mechanisms  there by cluster to start with out PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
    82  - Mounted a docker-compose volume for server.propeties. However, This also resulted in failures for Kafka v101, v111, v201, v211 and v221. Once Kafka server starts, server.properties gets appended with default value of sasl.enabled.mechanisms there by cluster to start with out PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
    83  
    84  
    85  NOTE: 
    86  - Kafka v101, v111, v201, v211 and v221 have no docker-compose files since we need SCRAM for kafka-go test cases to succeed. 
    87  - There is no Bitnami Kafka image for v222 hence testing has been performed on v221
    88  
    89  
    90  ### Kafka v231
    91  
    92  In Bitnami Kafka v2.3, all server.properties docker-compose environment configs start with `KAFKA_CFG_<config_with_underscore>`. However, it is not picking the custom populated kafka_jaas.conf.
    93  
    94  
    95  After a lot of debugging, it has been noticed that there aren't enough privileges to create the kafka_jaas.conf. Hence the environment variables below need to be added in docker-compose to generate the kafka_jaas.conf. This issue is not noticed after kafka v2.3
    96  
    97  
    98  ```
    99  KAFKA_INTER_BROKER_USER: adminplain
   100  KAFKA_INTER_BROKER_PASSWORD: admin-secret
   101  KAFKA_BROKER_USER: adminplain
   102  KAFKA_BROKER_PASSWORD: admin-secret
   103  ```
   104  
   105  There is a docker-compose file `docker-compose-231.yml` in the folder `kafka-go/docker_compose_versions` for reference.
   106  
   107  
   108  ## References
   109  
   110  
   111  For user reference, please find the some of the older kafka versions commits from the [kafka commit history](https://github.com/bitnami/containers/commits/main/bitnami/kafka). For Kafka versions with no commit history, data is populated with the latest version available for the tag.
   112  
   113  
   114  ### Kafka v010: docker-compose reference: `kafka-go/docker_compose_versions/docker-compose-010.yml`
   115  - [tag](https://hub.docker.com/r/bitnami/kafka/tags?page=1&ordering=last_updated&name=0.10.2.1)
   116  - [kafka commit](https://github.com/bitnami/containers/tree/c4240f0525916a418245c7ef46d9534a7a212c92/bitnami/kafka)
   117  
   118  
   119  ### Kafka v011: docker-compose reference: `kafka-go/docker_compose_versions/docker-compose-011.yml`
   120  - [tag](https://hub.docker.com/r/bitnami/kafka/tags?page=1&ordering=last_updated&name=0.11.0)
   121  - [kafka commit](https://github.com/bitnami/containers/tree/7724adf655e4ca9aac69d606d41ad329ef31eeca/bitnami/kafka)
   122  
   123  
   124  ### Kafka v101: docker-compose reference: N/A
   125  - [tag](https://hub.docker.com/r/bitnami/kafka/tags?page=1&ordering=last_updated&name=1.0.1)
   126  - [kafka commit](https://github.com/bitnami/containers/tree/44cc8f4c43ead6edebd3758c8df878f4f9da82c2/bitnami/kafka)
   127  
   128  
   129  ### Kafka v111: docker-compose reference: N/A
   130  - [tag](https://hub.docker.com/r/bitnami/kafka/tags?page=1&ordering=last_updated&name=1.1.1)
   131  - [kafka commit](https://github.com/bitnami/containers/tree/cb593dc98c2eb7a39f2792641e741d395dbe50e7/bitnami/kafka)
   132  
   133  
   134  ### Kafka v201: docker-compose reference: N/A
   135  - [tag](https://hub.docker.com/r/bitnami/kafka/tags?page=1&ordering=last_updated&name=2.0.1)
   136  - [kafka commit](https://github.com/bitnami/containers/tree/9ff8763df265c87c8b59f8d7ff0cf69299d636c9/bitnami/kafka)
   137  
   138  
   139  ### Kafka v211: docker-compose reference: N/A
   140  - [tag](https://hub.docker.com/r/bitnami/kafka/tags?page=1&ordering=last_updated&name=2.1.1)
   141  - [kafka commit](https://github.com/bitnami/containers/tree/d3a9d40afc2b7e7de53486538a63084c1a565d43/bitnami/kafka)
   142  
   143  
   144  ### Kafka v221: docker-compose reference: N/A
   145  - [tag](https://hub.docker.com/r/bitnami/kafka/tags?page=1&ordering=last_updated&name=2.2.1)
   146  - [kafka commit](https://github.com/bitnami/containers/tree/f132ef830d1ba9b78392ec4619174b4640c276c9/bitnami/kafka)
   147  
   148  
   149  ### Kafka v231: docker-compose reference: `kafka-go/docker_compose_versions/docker-compose-231.yml`
   150  - [tag](https://hub.docker.com/r/bitnami/kafka/tags?page=1&ordering=last_updated&name=2.3.1)
   151  - [kafka commit](https://github.com/bitnami/containers/tree/ae572036b5281456b0086345fec0bdb74f7cf3a3/bitnami/kafka)
   152