sigs.k8s.io/cluster-api-provider-aws@v1.5.5/docs/book/src/topics/external-cloud-provider-with-ebs-csi-driver.md (about)

     1  # External AWS Cloud Provider and AWS CSI Driver
     2  
     3  ## Overview
     4  The support for in-tree cloud providers and the CSI drivers is coming to an end and CAPA supports various upgrade paths
     5  to use [external cloud provider (Cloud Controller Manager - CCM) ](https://github.com/kubernetes/cloud-provider-aws) and external CSI drivers.
     6  This document explains how to create a CAPA cluster with external CSI/CCM plugins and how to upgrade existing clusters that rely on in-tree providers.
     7  
     8  
     9  ## Creating clusters with external CSI/CCM and validating
    10  For clusters that will use external CCM, `cloud-provider: external` flag needs to be set in KubeadmConfig resources in both `KubeadmControlPlane` and `MachineDeployment` resources.
    11  
    12      clusterConfiguration:
    13        apiServer:
    14          extraArgs:
    15            cloud-provider: external
    16        controllerManager:
    17          extraArgs:
    18            cloud-provider: external
    19      initConfiguration:
    20        nodeRegistration:
    21          kubeletExtraArgs:
    22            cloud-provider: external
    23      joinConfiguration:
    24        nodeRegistration:
    25          kubeletExtraArgs:
    26            cloud-provider: external
    27  
    28  
    29  External CCM and EBS CSI driver can be installed manually or using ClusterResourceSets (CRS) onto the CAPA workload cluster.
    30  To install them with CRS, create a CRS resource on the management cluster with labels, for example `csi: external` and `ccm: external` labels. 
    31  Then, when creating `Cluster` objects for workload clusters that should have this CSR applied, create them with matching labels `csi: external` and `ccm: external` for CSI and CCM, respectively.
    32  
    33  Manifests for installing the AWS CCM and the AWS EBS CSI driver are available from their respective 
    34  GitHub repositories (see [here for the AWS CCM](https://github.com/kubernetes/cloud-provider-aws) and 
    35  [here for the AWS EBS CSI driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver)).
    36  
    37  An example of a workload cluster manifest with labels assigned for matching to a CRS can be found 
    38  [here](https://github.com/kubernetes-sigs/cluster-api-provider-aws/tree/main/templates/cluster-template-external-cloud-provider.yaml).
    39  
    40  ### Verifying dynamically provisioned volumes with CSI driver
    41  Once you have the cluster with external CCM and CSI controller running successfully, you can test the CSI driver functioning with following steps after switching to workload cluster:
    42  1. Create a service (say,`nginx`)
    43  ```yaml
    44  apiVersion: v1
    45  kind: Service
    46  metadata:
    47    name: nginx-svc
    48    namespace: default
    49  spec:
    50    clusterIP: None
    51    ports:
    52      - name: nginx-web
    53        port: 80
    54    selector:
    55      app: nginx
    56  ```
    57  2. Create a storageclass and statefulset for the service created above with the persistent volume assigned to the storageclass:
    58  ```yaml
    59  kind: StorageClass
    60  apiVersion: storage.k8s.io/v1
    61  metadata:
    62    name: aws-ebs-volumes
    63  provisioner: ebs.csi.aws.com
    64  volumeBindingMode: WaitForFirstConsumer
    65  parameters:
    66    csi.storage.k8s.io/fstype: xfs
    67    type: io1
    68    iopsPerGB: "100"
    69  allowedTopologies:
    70    - matchLabelExpressions:
    71        - key: topology.ebs.csi.aws.com/zone
    72          values:
    73            - us-east-1a
    74  ---
    75  apiVersion: apps/v1
    76  kind: StatefulSet
    77  metadata:
    78    name: nginx-statefulset
    79  spec:
    80    serviceName: "nginx-svc"
    81    replicas: 2
    82    selector:
    83      matchLabels:
    84        app: nginx
    85    template:
    86      metadata:
    87        labels:
    88          app: nginx
    89      spec:
    90        containers:
    91          - name: nginx
    92            image: registry.k8s.io/nginx-slim:0.8
    93            ports:
    94              - name: nginx-web
    95                containerPort: 80
    96            volumeMounts:
    97              - name: nginx-volumes
    98                mountPath: /usr/share/nginx/html
    99        volumes:
   100          - name: nginx-volumes
   101            persistentVolumeClaim:
   102              claimName: nginx-volumes
   103    volumeClaimTemplates:
   104      - metadata:
   105          name: nginx-volumes
   106        spec:
   107          storageClassName: "aws-ebs-volumes"
   108          accessModes: [ "ReadWriteOnce" ]
   109          resources:
   110            requests:
   111              storage: 4Gi
   112   ```
   113  3. Once you apply the above manifest, the EBS volumes will be created and attached to the worker nodes.
   114  
   115  >**IMPORTANT WARNING:** The CRDs from the AWS EBS CSI driver and AWS external cloud provider gives issue while installing the respective controllers on the AWS Cluster, it doesn't allow statefulsets to create the volume on existing EC2 instance.
   116  > We need the CSI controller deployment and CCM pinned to the control plane which has right permissions to create, attach 
   117  > and mount the volumes to EC2 instances. To achieve this, you should add the node affinity rules to the CSI driver controller deployment and CCM DaemonSet manifests.
   118  > ```yaml
   119  > tolerations:
   120  > - key: node-role.kubernetes.io/master
   121  >   effect: NoSchedule
   122  > - effect: NoSchedule
   123  >   key: node-role.kubernetes.io/control-plane 
   124  > affinity:
   125  >   nodeAffinity:
   126  >   requiredDuringSchedulingIgnoredDuringExecution:
   127  >     nodeSelectorTerms:
   128  >       - matchExpressions:
   129  >           - key: node-role.kubernetes.io/control-plane
   130  >             operator: Exists
   131  >       - matchExpressions:
   132  >           - key: node-role.kubernetes.io/master
   133  >             operator: Exists
   134  >```
   135   
   136  
   137  ## Validated upgrade paths for existing clusters
   138  
   139  From Kubernetes 1.23 onwards, `CSIMigrationAWS` flag is enabled by default, which requires the installation of [external CSI driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver), unless `CSIMigrationAWS` is disabled by the user.
   140  For installing external CSI/CCM in the upgraded cluster, CRS can be used, see the section above for details.
   141  
   142  CCM and CSI do not need to be migrated to use external plugins at the same time, 
   143  external CSI drivers works with in-tree CCM (Warning: using in-tree CSI with external CCM does not work).
   144  
   145  **Following 3 upgrade paths are validated:**
   146  - Scenario 1: During upgrade to v1.23.x, disabling `CSIMigrationAWS` flag and keep using in-tree CCM and CSI.
   147  - Scenario 2: During upgrade to v1.23.x, enabling `CSIMigrationAWS` flag and using in-tree CCM with external CSI.
   148  - Scenario 3: During upgrade to v1.23.x, enabling `CSIMigrationAWS` flag and using external CCM and CSI.
   149  
   150  
   151  |                             | CSI      | CCM      | feature-gate CSIMigrationAWS | external-cloud-volume-plugin |
   152  |-----------------------------|----------|----------|------------------------------|------------------------------|
   153  | **Scenario 1**              |          |          |                              |                              |
   154  | From Kubernetes  < v1.23    | in-tree  | in-tree  | off                          | NA                           |
   155  | To Kubernetes    >= v1.23   | in-tree  | in-tree  | off                          | NA                           |
   156  | **Scenario 2**              |          |          |                              |                              |
   157  | From Kubernetes  < v1.23    | in-tree  | in-tree  | off                          | NA                           |
   158  | To Kubernetes    >= v1.23   | external | in-tree  | on                           | NA                           |
   159  | **Scenario 3**              |          |          |                              |                              |
   160  | From Kubernetes  < v1.23    | in-tree  | in-tree  | off                          | NA                           |
   161  | To Kubernetes    >= v1.23   | external | external | on                           | aws                          |
   162  
   163  
   164  **KubeadmConfig in the upgraded cluster for scenario 1:**
   165  
   166      clusterConfiguration:
   167        apiServer:
   168          extraArgs:
   169            cloud-provider: aws
   170        controllerManager:
   171          extraArgs:
   172            cloud-provider: aws
   173            feature-gates: CSIMigrationAWS=false
   174      initConfiguration:
   175        nodeRegistration:
   176          kubeletExtraArgs:
   177            cloud-provider: aws
   178            feature-gates: CSIMigrationAWS=false
   179          name: '{{ ds.meta_data.local_hostname }}'
   180      joinConfiguration:
   181        nodeRegistration:
   182          kubeletExtraArgs:
   183            cloud-provider: aws
   184            feature-gates: CSIMigrationAWS=false
   185  
   186  **KubeadmConfig in the upgraded cluster for scenario 2:**
   187  
   188  When `CSIMigrationAWS=true`, installed external CSI driver will be used while relying on in-tree CCM.
   189  
   190      clusterConfiguration:
   191        apiServer:
   192          extraArgs:
   193            cloud-provider: aws
   194            feature-gates: CSIMigrationAWS=true   // Set only if Kubernetes version < 1.23.x, otherwise this flag is enabled by default.
   195        controllerManager:
   196          extraArgs:
   197            cloud-provider: aws
   198            feature-gates: CSIMigrationAWS=true   // Set only if Kubernetes version < 1.23.x, otherwise this flag is enabled by default.
   199      initConfiguration:
   200        nodeRegistration:
   201          kubeletExtraArgs:
   202            cloud-provider: aws
   203            feature-gates: CSIMigrationAWS=true   // Set only if Kubernetes version < 1.23.x, otherwise this flag is enabled by default.
   204      joinConfiguration:
   205        nodeRegistration:
   206          kubeletExtraArgs:
   207            cloud-provider: aws
   208            feature-gates: CSIMigrationAWS=true   // Set only if Kubernetes version < 1.23.x, otherwise this flag is enabled by default.
   209  
   210  **KubeadmConfig in the upgraded cluster for scenario 3:**
   211  
   212  `external-cloud-volume-plugin` flag needs to be set for old Kubelets to keep talking to in-tree CCM and upgrade fails without this is set.
   213  
   214  
   215      clusterConfiguration:
   216        apiServer:
   217          extraArgs:
   218            cloud-provider: external
   219        controllerManager:
   220          extraArgs:
   221            cloud-provider: external
   222            external-cloud-volume-plugin: aws
   223      initConfiguration:
   224        nodeRegistration:
   225          kubeletExtraArgs:
   226            cloud-provider: external
   227      joinConfiguration:
   228        nodeRegistration:
   229          kubeletExtraArgs:
   230            cloud-provider: external