github.com/1aal/kubeblocks@v0.0.0-20231107070852-e1c03e598921/docs/user_docs/kubeblocks-for-kafka/cluster-management/create-a-kafka-cluster.md (about) 1 --- 2 title: Create a cluster for Kafka 3 description: Guide for cluster creation for kafka 4 keywords: [kafka, cluster, management] 5 sidebar_position: 1 6 sidebar_label: Create 7 --- 8 9 # Create a Kafka cluster 10 11 This document shows how to create and connect to a Kafka cluster. 12 13 ## Before you start 14 15 * [Install kbcli](./../../installation/install-with-kbcli/install-kbcli.md). 16 * Install KubeBlocks: You can install KubeBlocks by [kbcli](./../../installation/install-with-kbcli/install-kubeblocks-with-kbcli.md) or by [Helm](./../../installation/install-with-helm/install-kubeblocks-with-helm.md). 17 * Make sure kafka addon is enabled with `kbcli addon list`. 18 19 ```bash 20 kbcli addon list 21 > 22 NAME TYPE STATUS EXTRAS AUTO-INSTALL INSTALLABLE-SELECTOR 23 ... 24 kafka Helm Enabled true 25 ... 26 ``` 27 28 :::note 29 30 * KubeBlocks integrates Kafka v3.3.2, running it in KRaft mode. 31 * You are not recommended to use kraft cluster in combined mode in production environment. 32 * The controller number suggested ranges from 3 to 5, out of complexity and availability. 33 34 ::: 35 ## Create a Kafka cluster 36 37 <Tabs> 38 <TabItem value="using kbcli" label="Using kbcli" default> 39 40 The cluster creation command is simply `kbcli cluster create`. Further, you can customize your cluster resources as demanded by using the `--set` flag. 41 42 ```bash 43 kbcli cluster create kafka 44 ``` 45 46 See the table below for detailed descriptions of customizable parameters, setting the `--termination-policy` is necessary, and you are strongly recommended to turn on the monitor and enable all logs. 47 48 📎 Table 1. kbcli cluster create flags description 49 50 | Option | Description | 51 |---------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 52 | --mode='combined' | Mode for Kafka kraft cluster, 'combined' is combined Kafka controller and broker,'separated' is broker and controller running independently. Legal values [combined, separated] | 53 | --replicas=1 | The number of Kafka broker replicas for combined mode. In combined mode, this number also refers to the number of the kraft controller. Valid value range[1,3,5] | 54 | --broker-replicas=1 | The number of Kafka broker replicas for separated mode. | 55 | --controller-replicas=1 | The number of Kafka controller replicas for separated mode. In separated mode, this number refers to the number of kraft controller. Valid value range [1,3,5] | 56 | --termination-policy='Delete' | The termination policy of cluster. Legal values [DoNotTerminate, Halt, Delete, WipeOut]. <br />- DoNotTerminate: DoNotTerminate blocks the delete operation. <br />- Halt: Halt deletes workload resources such as statefulset, deployment workloads but keeps PVCs. <br />- Delete: Delete is based on Halt and deletes PVCs. - WipeOut: WipeOut is based on Delete and wipes out all volume snapshots and snapshot data from backup storage location. | 57 | --storage-enable=false | Enable storage for Kafka. | 58 | --host-network-accessible=false | Specify whether the cluster can be accessed from within the VPC. | 59 | --publicly-accessible=false | Specify whether the cluster can be accessed from the public internet. | 60 | --broker-heap='-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64' | Kafka broker's jvm heap setting. | 61 | --controller-heap='-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64' | Kafka controller's jvm heap setting for separated mode. The setting takes effect only when mode='separated'. | 62 | --cpu=1 | CPU cores. | 63 | --memory=1 | Memory, the unit is Gi. | 64 | --storage=20 | Data Storage size, the unit is Gi. | 65 | --storage-class='' | The StorageClass for Kafka Data Storage. | 66 | --meta-storage=5 | Metadata Storage size, the unit is Gi. | 67 | --meta-storage-class='' | The StorageClass for Kafka Metadata Storage. | 68 | --monitor-enable=false | Enable monitor for Kafka. | 69 | --monitor-replicas=1 | The number of Kafka monitor replicas. | 70 | --sasl-enable=false | Enable authentication using SASL/PLAIN for Kafka. <br /> -server: admin/kubeblocks <br /> -client: client/kubeblocks <br /> built-in jaas file stores on /tools/client-ssl.properties | 71 </TabItem> 72 73 <TabItem value="using kubectl" label="Using kubectl" default> 74 75 * Create a Kafka cluster in combined mode. 76 77 ```bash 78 # create kafka in combined mode 79 kubectl apply -f - <<EOF 80 apiVersion: apps.kubeblocks.io/v1alpha1 81 kind: Cluster 82 metadata: 83 name: kafka-combined 84 namespace: default 85 spec: 86 affinity: 87 podAntiAffinity: Preferred 88 tenancy: SharedNode 89 topologyKeys: 90 - kubernetes.io/hostname 91 clusterDefinitionRef: kafka 92 clusterVersionRef: kafka-3.3.2 93 componentSpecs: 94 - componentDefRef: kafka-server 95 monitor: false 96 name: broker 97 noCreatePDB: false 98 replicas: 1 99 resources: 100 limits: 101 cpu: "0.5" 102 memory: 0.5Gi 103 requests: 104 cpu: "0.5" 105 memory: 0.5Gi 106 serviceAccountName: kb-kafka-sa 107 terminationPolicy: Delete 108 EOF 109 ``` 110 111 * Create a Kafka cluster in separated mode. 112 113 ```bash 114 # Create kafka cluster in separated mode 115 kubectl apply -f - <<EOF 116 apiVersion: apps.kubeblocks.io/v1alpha1 117 kind: Cluster 118 metadata: 119 name: kafka-separated 120 namespace: default 121 spec: 122 affinity: 123 podAntiAffinity: Preferred 124 tenancy: SharedNode 125 topologyKeys: 126 - kubernetes.io/hostname 127 clusterDefinitionRef: kafka 128 clusterVersionRef: kafka-3.3.2 129 componentSpecs: 130 - componentDefRef: controller 131 monitor: false 132 name: controller 133 noCreatePDB: false 134 replicas: 1 135 resources: 136 limits: 137 cpu: "0.5" 138 memory: 0.5Gi 139 requests: 140 cpu: "0.5" 141 memory: 0.5Gi 142 serviceAccountName: kb-kafka-sa 143 tls: false 144 - componentDefRef: kafka-broker 145 monitor: false 146 name: broker 147 noCreatePDB: false 148 replicas: 1 149 resources: 150 limits: 151 cpu: "0.5" 152 memory: 0.5Gi 153 requests: 154 cpu: "0.5" 155 memory: 0.5Gi 156 serviceAccountName: kb-kafka-sa 157 tls: false 158 terminationPolicy: Delete 159 EOF 160 ``` 161 162 </TabItem> 163 164 </Tabs>