sigs.k8s.io/cluster-api@v1.7.1/docs/book/src/tasks/external-etcd.md (about)

     1  # Support for external etcd
     2  
     3  Cluster API Bootstrap Provider Kubeadm supports using an external etcd cluster for your workload Kubernetes clusters.
     4  
     5  ## ⚠️ Warnings ⚠️
     6  
     7  Before getting started you should be aware of the expectations that come with using an external etcd cluster.
     8  
     9  * Cluster API is unable to manage any aspect of the external etcd cluster.
    10  * Depending on how you configure your etcd nodes you may incur additional cloud costs in data transfer.
    11      * As an example, cross availability zone traffic can cost money on cloud providers. You don't have to deploy etcd across availability zones, but if you do please be aware of the costs.
    12  
    13  ## Getting started
    14  
    15  To use this, you will need to create an etcd cluster and generate an apiserver-etcd-client certificate and private key. This behaviour can be tested using [`kubeadm`](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/) and [`etcdadm`](https://github.com/kubernetes-sigs/etcdadm).
    16  
    17  ### Setting up etcd with kubeadm
    18  
    19  CA certificates are required to setup etcd cluster. If you already have a CA then the CA's `crt` and `key` must be copied to `/etc/kubernetes/pki/etcd/ca.crt` and `/etc/kubernetes/pki/etcd/ca.key`.
    20  
    21  If you do not already have a CA then run command `kubeadm init phase certs etcd-ca`. This creates two files:
    22  
    23  * `/etc/kubernetes/pki/etcd/ca.crt`
    24  * `/etc/kubernetes/pki/etcd/ca.key`
    25  
    26  This certificate and private key are used to sign etcd server and peer certificates as well as other client certificates (like the apiserver-etcd-client certificate or the etcd-healthcheck-client certificate). More information on how to setup external etcd with kubeadm can be found [here](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/#setting-up-the-cluster).
    27  
    28  Once the etcd cluster is setup, you will need the following files from the etcd cluster:
    29  
    30  1. `/etc/kubernetes/pki/apiserver-etcd-client.crt` and `/etc/kubernetes/pki/apiserver-etcd-client.key`
    31  2. `/etc/kubernetes/pki/etcd/ca.crt`
    32  
    33  You'll use these files to create the necessary Secrets on the management cluster (see the "Creating the required Secrets" section).
    34  
    35  ### Setting up etcd with etcdadm (Alpha)
    36  
    37  `etcdadm` creates the CA if one does not exist, uses it to sign its server and peer certificates, and finally to sign the API server etcd client certificate. The CA's `crt` and `key` generated using `etcdadm` are stored in `/etc/etcd/pki/ca.crt` and `/etc/etcd/pki/ca.key`. `etcdadm` also generates a certificate for the API server etcd client; the certificate and private key are found at `/etc/etcd/pki/apiserver-etcd-client.crt` and `/etc/etcd/pki/apiserver-etcd-client.key`, respectively.
    38  
    39  Once the etcd cluster has been bootstrapped using `etcdadm`, you will need the following files from the etcd cluster:
    40  
    41  1. `/etc/etcd/pki/apiserver-etcd-client.crt` and `/etc/etcd/pki/apiserver-etcd-client.key`
    42  2. `/etc/etcd/pki/etcd/ca.crt`
    43  
    44  You'll use these files in the next section to create the necessary Secrets on the management cluster.
    45  
    46  ## Creating the required Secrets
    47  
    48  Regardless of the method used to bootstrap the etcd cluster, you will need to use the certificates copied from the etcd cluster to create some [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-using-kubectl-create-secret) on the management cluster.
    49  
    50  In the commands below to create the Secrets, substitute `$CLUSTER_NAME` with the name of the workload cluster to be created by CAPI, and substitute `$CLUSTER_NAMESPACE` with the name of the namespace where the workload cluster will be created. The namespace can be omitted if the workload cluster will be created in the default namespace.
    51  
    52  First, you will need to create a Secret containing the API server etcd client certificate and key. This command assumes the certificate and private key are in the current directory; adjust your command accordingly if they are not:
    53  
    54  ```bash
    55  # Kubernetes API server etcd client certificate and key
    56  kubectl create secret tls $CLUSTER_NAME-apiserver-etcd-client \
    57    --cert apiserver-etcd-client.crt \
    58    --key apiserver-etcd-client.key \
    59    --namespace $CLUSTER_NAMESPACE
    60  ```
    61  
    62  Next, create a Secret for the etcd cluster's CA certificate. The `kubectl create secret tls` command requires _both_ a certificate and a key, but the key isn't needed by CAPI. Instead, use the `kubectl create secret generic` command, and note that the file containing the CA certificate **must** be named `tls.crt`:
    63  
    64  ```bash
    65  # Etcd's CA crt file to validate the generated client certificates
    66  kubectl create secret generic $CLUSTER_NAME-etcd \
    67    --from-file tls.crt \
    68    --namespace $CLUSTER_NAMESPACE
    69  ```
    70  
    71  **Note:** The above commands will base64 encode the certificate/key files by default.
    72  
    73  Alternatively you can base64 encode the files and put them in two secrets. The secrets must be formatted as follows and the cert material must be base64 encoded:
    74  
    75  ```yaml
    76  # Kubernetes APIServer etcd client certificate
    77  kind: Secret
    78  apiVersion: v1
    79  metadata:
    80    name: $CLUSTER_NAME-apiserver-etcd-client
    81    namespace: $CLUSTER_NAMESPACE
    82  data:
    83    tls.crt: |
    84      LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCRENDQWV5Z0F3SUJBZ0lJZFlkclZUMzV0
    85      NW93RFFZSktvWklodmNOQVFFTEJRQXdEekVOTUFzR0ExVUUKQXhNRVpYUmpaREFlRncweE9UQTVN
    86      ...
    87    tls.key: |
    88      LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBdlFlTzVKOE5j
    89      VCtDeGRubFR3alpuQ3YwRzByY0tETklhZzlSdFdrZ1p4MEcxVm1yClA4Zy9BRkhXVHdxSTUrNi81
    90      ...
    91  ```
    92  
    93  ```yaml
    94  # Etcd's CA crt file to validate the generated client certificates
    95  kind: Secret
    96  apiVersion: v1
    97  metadata:
    98    name: $CLUSTER_NAME-etcd
    99    namespace: $CLUSTER_NAMESPACE
   100  data:
   101    tls.crt: |
   102      LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBRENDQWVpZ0F3SUJBZ0lJRDNrVVczaDIy
   103      K013RFFZSktvWklodmNOQVFFTEJRQXdEekVOTUFzR0ExVUUKQXhNRVpYUmpaREFlRncweE9UQTVN
   104      ...
   105  ```
   106  
   107  The Secrets must be created _before_ creating the workload cluster.
   108  
   109  ## Configuring CABPK
   110  
   111  Once the Secrets are in place on the management cluster, the rest of the process leverages standard `kubeadm` configuration. Configure your ClusterConfiguration for the workload cluster as follows:
   112  
   113  ```yaml
   114  apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
   115  kind: KubeadmConfig
   116  metadata:
   117    name: CLUSTER_NAME-controlplane-0
   118    namespace: CLUSTER_NAMESPACE
   119  spec:
   120    ... # initConfiguration goes here
   121    clusterConfiguration:
   122      etcd:
   123        external:
   124          endpoints:
   125            - https://10.0.0.230:2379
   126          caFile: /etc/kubernetes/pki/etcd/ca.crt
   127          certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
   128          keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
   129      ... # other clusterConfiguration goes here
   130  ```
   131  
   132  Create your workload cluster as normal. The new workload cluster should use the configured external etcd nodes instead of creating co-located etcd Pods on the control plane nodes.
   133  
   134  ## Additional Notes/Caveats
   135  
   136  * Depending on the provider, additional changes to the workload cluster's manifest may be necessary to ensure the new CAPI-managed nodes have connectivity to the existing etcd nodes. For example, on AWS you will need to leverage the `additionalSecurityGroups` field on the AWSMachine and/or AWSMachineTemplate objects to add the CAPI-managed nodes to a security group that has connectivity to the existing etcd cluster. Other mechanisms exist for other providers.