github.com/spotahome/redis-operator@v1.2.4/README.md (about) 1 # redis-operator 2 3 [](https://github.com/spotahome/redis-operator) 4 [](https://goreportcard.com/report/github.com/spotahome/redis-operator) 5 6 Redis Operator creates/configures/manages redis-failovers atop Kubernetes. 7 8 ## Requirements 9 10 Kubernetes version: 1.21 or higher 11 Redis version: 6 or higher 12 13 Redis operator is being tested against kubernetes 1.22 1.23 1.24 and redis 6 14 All dependencies have been vendored, so there's no need to any additional download. 15 16 ## Operator deployment on Kubernetes 17 18 In order to create Redis failovers inside a Kubernetes cluster, the operator has to be deployed. 19 It can be done with plain old [deployment](example/operator), using [Kustomize](manifests/kustomize) or with the provided [Helm chart](charts/redisoperator). 20 21 ### Using the Helm chart 22 23 From the root folder of the project, execute the following: 24 25 ``` 26 helm repo add redis-operator https://spotahome.github.io/redis-operator 27 helm repo update 28 helm install redis-operator redis-operator/redis-operator 29 ``` 30 31 #### Update helm chart 32 33 Helm chart only manage the creation of CRD in the first install. In order to update the CRD you will need to apply directly. 34 35 ``` 36 REDIS_OPERATOR_VERSION=v1.2.4 37 kubectl replace -f https://raw.githubusercontent.com/spotahome/redis-operator/${REDIS_OPERATOR_VERSION}/manifests/databases.spotahome.com_redisfailovers.yaml 38 ``` 39 40 ``` 41 helm upgrade redis-operator redis-operator/redis-operator 42 ``` 43 ### Using kubectl 44 45 To create the operator, you can directly create it with kubectl: 46 47 ``` 48 REDIS_OPERATOR_VERSION=v1.2.4 49 kubectl create -f https://raw.githubusercontent.com/spotahome/redis-operator/${REDIS_OPERATOR_VERSION}/manifests/databases.spotahome.com_redisfailovers.yaml 50 kubectl apply -f https://raw.githubusercontent.com/spotahome/redis-operator/${REDIS_OPERATOR_VERSION}/example/operator/all-redis-operator-resources.yaml 51 ``` 52 53 This will create a deployment named `redisoperator`. 54 55 ### Using kustomize 56 57 The kustomize setup included in this repo is highly customizable using [components](https://kubectl.docs.kubernetes.io/guides/config_management/components/), 58 but it also comes with a few presets (in the form of overlays) supporting the most common use cases. 59 60 To install the operator with default settings and every necessary resource (including RBAC, service account, default resource limits, etc), install the `default` overlay: 61 62 ```shell 63 kustomize build github.com/spotahome/redis-operator/manifests/kustomize/overlays/default 64 ``` 65 66 If you would like to customize RBAC or the service account used, you can install the `minimal` overlay. 67 68 Finally, you can install the `full` overlay if you want everything this operator has to offer, including Prometheus ServiceMonitor resources. 69 70 It's always a good practice to pin the version of the operator in your configuration to make sure you are not surprised by changes on the latest development branch: 71 72 ```shell 73 kustomize build github.com/spotahome/redis-operator/manifests/kustomize/overlays/default?ref=v1.2.4 74 ``` 75 76 You can easily create your own config by creating a `kustomization.yaml` file 77 (for example to apply custom resource limits, to add custom labels or to customize the namespace): 78 79 ```yaml 80 apiVersion: kustomize.config.k8s.io/v1beta1 81 kind: Kustomization 82 83 namespace: redis-operator 84 85 commonLabels: 86 foo: bar 87 88 resources: 89 - github.com/spotahome/redis-operator/manifests/kustomize/overlays/full 90 ``` 91 92 Take a look at the manifests inside [manifests/kustomize](manifests/kustomize) for more details. 93 94 ## Usage 95 96 Once the operator is deployed inside a Kubernetes cluster, a new API will be accesible, so you'll be able to create, update and delete redisfailovers. 97 98 In order to deploy a new redis-failover a [specification](example/redisfailover/basic.yaml) has to be created: 99 100 ``` 101 REDIS_OPERATOR_VERSION=v1.2.4 102 kubectl create -f https://raw.githubusercontent.com/spotahome/redis-operator/${REDIS_OPERATOR_VERSION}/example/redisfailover/basic.yaml 103 ``` 104 105 This redis-failover will be managed by the operator, resulting in the following elements created inside Kubernetes: 106 107 - `rfr-<NAME>`: Redis configmap 108 - `rfr-<NAME>`: Redis statefulset 109 - `rfr-<NAME>`: Redis service (if redis-exporter is enabled) 110 - `rfs-<NAME>`: Sentinel configmap 111 - `rfs-<NAME>`: Sentinel deployment 112 - `rfs-<NAME>`: Sentinel service 113 114 **NOTE**: `NAME` is the named provided when creating the RedisFailover. 115 **IMPORTANT**: the name of the redis-failover to be created cannot be longer that 48 characters, due to prepend of redis/sentinel identification and statefulset limitation. 116 117 ### Persistence 118 119 The operator has the ability of add persistence to Redis data. By default an `emptyDir` will be used, so the data is not saved. 120 121 In order to have persistence, a `PersistentVolumeClaim` usage is allowed. The full [PVC definition has to be added](example/redisfailover/persistent-storage.yaml) to the Redis Failover Spec under the `Storage` section. 122 123 **IMPORTANT**: By default, the persistent volume claims will be deleted when the Redis Failover is. If this is not the expected usage, a `keepAfterDeletion` flag can be added under the `storage` section of Redis. [An example is given](example/redisfailover/persistent-storage-no-pvc-deletion.yaml). 124 125 ### NodeAffinity and Tolerations 126 127 You can use NodeAffinity and Tolerations to deploy Pods to isolated groups of Nodes. Examples are given for [node affinity](example/redisfailover/node-affinity.yaml), [pod anti affinity](example/redisfailover/pod-anti-affinity.yaml) and [tolerations](example/redisfailover/tolerations.yaml). 128 129 ## Topology Spread Contraints 130 131 You can use the `topologySpreadContraints` to ensure the pods of a type(redis or sentinel) are evenly distributed across zones/nodes. Examples are for using [topology spread constraints](example/redisfailover/topology-spread-contraints.yaml). Further document on how `topologySpreadConstraints` work could be found [here](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/). 132 133 ### Custom configurations 134 135 It is possible to configure both Redis and Sentinel. This is done with the `customConfig` option inside their spec. It is a list of configurations and their values. Example are given in the [custom config example file](example/redisfailover/custom-config.yaml). 136 137 In order to have the ability of this configurations to be changed "on the fly", without the need of reload the redis/sentinel processes, the operator will apply them with calls to the redises/sentinels, using `config set` or `sentinel set mymaster` respectively. Because of this, **no changes on the configmaps** will appear regarding this custom configurations and the entries of `customConfig` from Redis spec will not be written on `redis.conf` file. To verify the actual Redis configuration use [`redis-cli CONFIG GET *`](https://redis.io/commands/config-get). 138 139 **Important**: in the Sentinel options, there are some "conversions" to be made: 140 141 - Configuration on the `sentinel.conf`: `sentinel down-after-milliseconds mymaster 2000` 142 - Configuration on the `configOptions`: `down-after-milliseconds 2000` 143 144 **Important 2**: do **NOT** change the options used for control the redis/sentinel such as `port`, `bind`, `dir`, etc. 145 146 ### Custom shutdown script 147 148 By default, a custom shutdown file is given. This file makes redis to `SAVE` it's data, and in the case that redis is master, it'll call sentinel to ask for a failover. 149 150 This behavior is configurable, creating a configmap and indicating to use it. An example about how to use this option can be found on the [shutdown example file](example/redisfailover/custom-shutdown.yaml). 151 152 **Important**: the configmap has to be in the same namespace. The configmap has to have a `shutdown.sh` data, containing the script. 153 154 ### Custom SecurityContext 155 156 By default Kubernetes will run containers as the user specified in the Dockerfile (or the root user if not specified), this is not always desirable. 157 If you need the containers to run as a specific user (or provide any other PodSecurityContext options) then you can specify a custom `securityContext` in the 158 `redisfailover` object. See the [SecurityContext example file](example/redisfailover/security-context.yaml) for an example. You can visit kubernetes documentation for detailed docs about [security context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) 159 160 ### Custom containerSecurityContext at container level 161 162 By default Kubernetes will run containers with default docker capabilities for exemple, this is not always desirable. 163 If you need the containers to run with specific capabilities or with read only root file system (or provide any other securityContext options) then you can specify a custom `containerSecurityContext` in the 164 `redisfailover` object. See the [ContainerSecurityContext example file](example/redisfailover/container-security-context.yaml) for an example. Keys available under containerSecurityContext are detailed [here](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#securitycontext-v1-core) 165 166 ### Custom command 167 168 By default, redis and sentinel will be called with the basic command, giving the configuration file: 169 170 - Redis: `redis-server /redis/redis.conf` 171 - Sentinel: `redis-server /redis/sentinel.conf --sentinel` 172 173 If necessary, this command can be changed with the `command` option inside redis/sentinel spec. An example can be found in the [custom command example file](example/redisfailover/custom-command.yaml). 174 175 ### Custom Priority Class 176 In order to use a custom Kubernetes [Priority Class](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass) for Redis and/or Sentinel pods, you can set the `priorityClassName` in the redis/sentinel spec, this attribute has no default and depends on the specific cluster configuration. **Note:** the operator doesn't create the referenced `Priority Class` resource. 177 178 ### Custom Service Account 179 In order to use a custom Kubernetes [Service Account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) for Redis and/or Sentinel pods, you can set the `serviceAccountName` in the redis/sentinel spec, if not specified the `default` Service Account will be used. **Note:** the operator doesn't create the referenced `Service Account` resource. 180 181 ### Custom Pod Annotations 182 By default, no pod annotations will be applied to Redis nor Sentinel pods. 183 184 In order to apply custom pod Annotations, you can provide the `podAnnotations` option inside redis/sentinel spec. An example can be found in the [custom annotations example file](example/redisfailover/custom-annotations.yaml). 185 ### Custom Service Annotations 186 By default, no service annotations will be applied to the Redis nor Sentinel services. 187 188 In order to apply custom service Annotations, you can provide the `serviceAnnotations` option inside redis/sentinel spec. An example can be found in the [custom annotations example file](example/redisfailover/custom-annotations.yaml). 189 190 ### Control of label propagation. 191 By default the operator will propagate all labels on the CRD down to the resources that it creates. This can be problematic if the 192 labels on the CRD are not fully under your own control (for example: being deployed by a gitops operator) 193 as a change to a labels value can fail on immutable resources such as PodDisruptionBudgets. To control what labels the operator propagates 194 to resource is creates you can modify the labelWhitelist option in the spec. 195 196 By default specifying no whitelist or an empty whitelist will cause all labels to still be copied as not to break backwards compatibility. 197 198 Items in the array should be regular expressions, see [here](example/redisfailover/control-label-propagation.yaml) as an example of how they can be used and 199 [here](https://github.com/google/re2/wiki/Syntax) for a syntax reference. 200 201 The whitelist can also be used as a form of blacklist by specifying a regular expression that will not match any label. 202 203 NOTE: The operator will always add the labels it requires for operation to resources. These are the following: 204 ``` 205 app.kubernetes.io/component 206 app.kubernetes.io/managed-by 207 app.kubernetes.io/name 208 app.kubernetes.io/part-of 209 redisfailovers.databases.spotahome.com/name 210 ``` 211 212 213 ### ExtraVolumes and ExtraVolumeMounts 214 215 If the user choose to have extra volumes creates and mounted, he could use the `extraVolumes` and `extraVolumeMounts`, in `spec.redis` of the CRD. This allows users to mount the extra configurations, or secrets to be used. A typical use case for this might be 216 - Secrets that sidecars might use to backup of RDBs 217 - Extra users and their secrets and acls that could used the initContainers to create multiple users 218 - Extra Configurations that could merge on top the existing configurations 219 - To pass failover scripts for addition for additional operations 220 221 ``` 222 --- 223 apiVersion: v1 224 kind: Secret 225 metadata: 226 name: foo 227 namespace: exm 228 type: Opaque 229 stringData: 230 password: MWYyZDFlMmU2N2Rm 231 --- 232 apiVersion: databases.spotahome.com/v1 233 kind: RedisFailover 234 metadata: 235 name: foo 236 namespace: exm 237 spec: 238 sentinel: 239 replicas: 3 240 extraVolumes: 241 - name: foo 242 secret: 243 secretName: foo 244 optional: false 245 extraVolumeMounts: 246 - name: foo 247 mountPath: "/etc/foo" 248 readOnly: true 249 redis: 250 replicas: 3 251 extraVolumes: 252 - name: foo 253 secret: 254 secretName: foo 255 optional: false 256 extraVolumeMounts: 257 - name: foo 258 mountPath: "/etc/foo" 259 readOnly: true 260 ``` 261 262 263 264 ## Connection to the created Redis Failovers 265 266 In order to connect to the redis-failover and use it, a [Sentinel-ready](https://redis.io/topics/sentinel-clients) library has to be used. This will connect through the Sentinel service to the Redis node working as a master. 267 The connection parameters are the following: 268 269 ``` 270 url: rfs-<NAME> 271 port: 26379 272 master-name: mymaster 273 ``` 274 275 ### Enabling redis auth 276 277 To enable auth create a secret with a password field: 278 279 ``` 280 echo -n "pass" > password 281 kubectl create secret generic redis-auth --from-file=password 282 283 ## example config 284 apiVersion: databases.spotahome.com/v1 285 kind: RedisFailover 286 metadata: 287 name: redisfailover 288 spec: 289 sentinel: 290 replicas: 3 291 redis: 292 replicas: 1 293 auth: 294 secretPath: redis-auth 295 ``` 296 You need to set secretPath as the secret name which is created before. 297 298 ### Bootstrapping from pre-existing Redis Instance(s) 299 If you are wanting to migrate off of a pre-existing Redis instance, you can provide a `bootstrapNode` to your `RedisFailover` resource spec. 300 301 This `bootstrapNode` can be configured as follows: 302 | Key | Type | Description | Example File | 303 |:--------------:|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------| 304 | host | **required** | The IP of the target Redis address or the ClusterIP of a pre-existing Kubernetes Service targeting Redis pods | [bootstrapping.yaml](example/redisfailover/bootstrapping.yaml) | 305 | port | _optional_ | The Port that the target Redis address is listening to. Defaults to `6379`. | [bootstrapping-with-port.yaml](example/redisfailover/bootstrapping-with-port.yaml) | 306 | allowSentinels | _optional_ | Allow the Operator to also create the specified Sentinel resources and point them to the target Node/Port. By default, the Sentinel resources will **not** be created when bootstrapping. | [bootstrapping-with-sentinels.yaml](example/redisfailover/bootstrapping-with-sentinels.yaml) | 307 308 #### What is Bootstrapping? 309 When a `bootstrapNode` is provided, the Operator will always set all of the defined Redis instances to replicate from the provided `bootstrapNode` host value. 310 This allows for defining a `RedisFailover` that replicates from an existing Redis instance to ease cutover from one instance to another. 311 312 **Note: Redis instance will always be configured with `replica-priority 0`. This means that these Redis instances can _never_ be promoted to a `master`.** 313 314 Depending on the configuration provided, the Operator will launch the `RedisFailover` in two bootstrapping states: without sentinels and with sentinels. 315 316 #### Default Bootstrapping Mode (Without Sentinels) 317 By default, if the `RedisFailover` resource defines a valid `bootstrapNode`, **only the redis instances will be created**. 318 This allows for ease of bootstrapping from an existing `RedisFailover` instance without the Sentinels intermingling with each other. 319 320 #### Bootstrapping With Sentinels 321 When `allowSentinels` is provided, the Operator will also create the defined Sentinel resources. These sentinels will be configured to point to the provided 322 `bootstrapNode` as their monitored master. 323 324 ### Default versions 325 326 The image versions deployed by the operator can be found on the [defaults file](api/redisfailover/v1/defaults.go). 327 ## Cleanup 328 329 ### Operator and CRD 330 331 If you want to delete the operator from your Kubernetes cluster, the operator deployment should be deleted. 332 333 Also, the CRD has to be deleted. Deleting CRD automatically wil delete all redis failover custom resources and their managed resources: 334 335 ``` 336 kubectl delete crd redisfailovers.databases.spotahome.com 337 ``` 338 339 ### Single Redis Failover 340 341 Thanks to Kubernetes' `OwnerReference`, all the objects created from a redis-failover will be deleted after the custom resource is. 342 343 ``` 344 kubectl delete redisfailover <NAME> 345 ``` 346 347 ## Docker Images 348 349 ### Redis Operator 350 351 [](https://quay.io/repository/spotahome/redis-operator) 352 ## Documentation 353 354 For the code documentation, you can lookup on the [GoDoc](https://godoc.org/github.com/spotahome/redis-operator). 355 356 Also, you can check more deeply information on the [docs folder](docs).