sigs.k8s.io/cluster-api-provider-aws@v1.5.5/docs/book/src/topics/bring-your-own-aws-infrastructure.md (about)

     1  # Bring Your Own AWS Infrastructure
     2  
     3  Normally, Cluster API will create infrastructure on AWS when standing up a new workload cluster. However, it is possible to have Cluster API re-use external AWS infrastructure instead of creating its own infrastructure. 
     4  
     5  There are two possible ways to do this:
     6  * By consuming existing AWS infrastructure
     7  * By using externally managed AWS infrastructure
     8  > **IMPORTANT NOTE**: This externally managed AWS infrastructure should not be confused with EKS-managed clusters.
     9  
    10  Follow the instructions below to configure Cluster API to consume existing AWS infrastructure.
    11  
    12  ## Consuming Existing AWS Infrastructure
    13  
    14  ### Overview
    15  
    16  CAPA supports using existing AWS resources while creating AWS Clusters which gives flexibility to the users to bring their own existing resources into the cluster instead of creating new resources again.
    17  
    18  Follow the instructions below to configure Cluster API to consume existing AWS infrastructure.
    19  
    20  ### Prerequisites
    21  
    22  In order to have Cluster API consume existing AWS infrastructure, you will need to have already created the following resources:
    23  
    24  * A VPC
    25  * One or more private subnets (subnets that do not have a route to an Internet gateway)
    26  * A NAT gateway for each private subnet, along with associated Elastic IP addresses (only needed if the nodes require access to the Internet, i.e. pulling public images)
    27    * A public subnet in the same Availability Zone (AZ) for each private subnet (this is required for NAT gateways to function properly)
    28  * An Internet gateway for all public subnets (only required if the workload cluster is set to use an Internet facing load balancer or one or more NAT gateways exist in the VPC)
    29  * Route table associations that provide connectivity to the Internet through a NAT gateway (for private subnets) or the Internet gateway (for public subnets)
    30  * VPC endpoints for `ec2`, `elasticloadbalancing`, `secretsmanager` an `autoscaling` (if using MachinePools) when the private Subnets do not have a NAT gateway
    31  
    32  You will need the ID of the VPC and subnet IDs that Cluster API should use. This information is available via the AWS Management Console or the AWS CLI.
    33  
    34  Note that there is no need to create an Elastic Load Balancer (ELB), security groups, or EC2 instances; Cluster API will take care of these items.
    35  
    36  If you want to use existing security groups, these can be specified and new ones will not be created.
    37  
    38  If you want to use an existing control load load balancer, specify its name.
    39  
    40  ### Tagging AWS Resources
    41  
    42  Cluster API itself does tag AWS resources it creates. The `sigs.k8s.io/cluster-api-provider-aws/cluster/<cluster-name>` (where `<cluster-name>` matches the `metadata.name` field of the Cluster object) tag, with a value of `owned`, tells Cluster API that it has ownership of the resource. In this case, Cluster API will modify and manage the lifecycle of the resource.
    43  
    44  When consuming existing AWS infrastructure, the Cluster API AWS provider does not require any tags to be present. The absence of the tags on an AWS resource indicates to Cluster API that it should not modify the resource or attempt to manage the lifecycle of the resource.
    45  
    46  However, the built-in Kubernetes AWS cloud provider _does_ require certain tags in order to function properly. Specifically, all subnets where Kubernetes nodes reside should have the `kubernetes.io/cluster/<cluster-name>` tag present. Private subnets should also have the `kubernetes.io/role/internal-elb` tag with a value of 1, and public subnets should have the `kubernetes.io/role/elb` tag with a value of 1. These latter two tags help the cloud provider understand which subnets to use when creating load balancers.
    47  
    48  Finally, if the controller manager isn't started with the `--configure-cloud-routes: "false"` parameter, the route table(s) will also need the `kubernetes.io/cluster/<cluster-name>` tag. (This parameter can be added by customizing the `KubeadmConfigSpec` object of the `KubeadmControlPlane` object.)
    49  
    50  ### Configuring the AWSCluster Specification
    51  
    52  Specifying existing infrastructure for Cluster API to use takes place in the specification for the AWSCluster object. Specifically, you will need to add an entry with the VPC ID and the IDs of all applicable subnets into the `network` field. Here is an example:
    53  
    54  For EC2
    55  ```yaml
    56  apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    57  kind: AWSCluster
    58  ```
    59  For EKS
    60  ```yaml
    61  apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    62  kind: AWSManagedControlPlane
    63  ```
    64  
    65  ```yaml
    66  spec:
    67    network:
    68      vpc:
    69        id: vpc-0425c335226437144
    70      subnets:
    71      - id: subnet-0261219d564bb0dc5
    72      - id: subnet-0fdcccba78668e013
    73  ```
    74  
    75  When you use `kubectl apply` to apply the Cluster and AWSCluster specifications to the management cluster, Cluster API will use the specified VPC ID and subnet IDs, and will not create a new VPC, new subnets, or other associated resources. It _will_, however, create a new ELB and new security groups.
    76  
    77  ### Placing EC2 Instances in Specific AZs
    78  
    79  To distribute EC2 instances across multiple AZs, you can add information to the Machine specification. This is optional and only necessary if control over AZ placement is desired.
    80  
    81  To tell Cluster API that an EC2 instance should be placed in a particular AZ but allow Cluster API to select which subnet in that AZ can be used, add this to the Machine specification:
    82  
    83  ```yaml
    84  spec:
    85    failureDomain: "us-west-2a"
    86  ```
    87  
    88  If using a MachineDeployment, specify AZ placement like so:
    89  
    90  ```yaml
    91  spec:
    92    template:
    93      spec:
    94        failureDomain: "us-west-2b"
    95  ```
    96  
    97  Note that all replicas within a MachineDeployment will reside in the same AZ.
    98  
    99  ### Placing EC2 Instances in Specific Subnets
   100  
   101  To specify that an EC2 instance should be placed in a specific subnet, add this to the AWSMachine specification:
   102  
   103  ```yaml
   104  spec:
   105    subnet:
   106      id: subnet-0a3507a5ad2c5c8c3
   107  ```
   108  
   109  When using MachineDeployments, users can control subnet selection by adding information to the AWSMachineTemplate associated with that MachineDeployment, like this:
   110  
   111  ```yaml
   112  spec:
   113    template:
   114      spec:
   115        subnet:
   116          id: subnet-0a3507a5ad2c5c8c3
   117  ```
   118  
   119  Users may either specify `failureDomain` on the Machine or MachineDeployment objects, _or_ users may explicitly specify subnet IDs on the AWSMachine or AWSMachineTemplate objects. If both are specified, the subnet ID is used and the `failureDomain` is ignored.
   120  
   121  ### Security Groups
   122  
   123  To use existing security groups for instances for a cluster, add this to the AWSCluster specification:
   124  
   125  ```yaml
   126  spec:
   127    network:
   128      securityGroupOverrides:
   129        bastion: sg-0350a3507a5ad2c5c8c3
   130        controlplane: sg-0350a3507a5ad2c5c8c3
   131        apiserver-lb: sg-0200a3507a5ad2c5c8c3
   132        node: sg-04e870a3507a5ad2c5c8c3
   133        lb: sg-00a3507a5ad2c5c8c3
   134  ```
   135  
   136  Any additional security groups specified in an AWSMachineTemplate will be applied in addition to these overriden security groups.
   137  
   138  To specify additional security groups for the control plane load balancer for a cluster, add this to the AWSCluster specification:
   139  
   140  ```yaml
   141  spec:
   142    controlPlaneLoadBalancer:
   143      AdditionalsecurityGroups:
   144      - sg-0200a3507a5ad2c5c8c3
   145      - ...
   146  ```
   147  
   148  ### Control Plane Load Balancer
   149  
   150  The cluster control plane is accessed through a Classic ELB. By default, Cluster API creates the Classic ELB. To use an existing Classic ELB, add its name to the AWSCluster specification:
   151  
   152  ```yaml
   153  spec:
   154    controlPlaneLoadBalancer:
   155      name: my-classic-elb-name
   156  ```
   157  
   158  As control plane instances are added or removed, Cluster API will register and deregister them, respectively, with the Classic ELB.
   159  
   160  > **WARNING:** Using an existing Classic ELB is an advanced feature. **If you use an existing Classic ELB, you must correctly configure it, and attach subnets to it.**
   161  > 
   162  >An incorrectly configured Classic ELB can easily lead to a non-functional cluster. We strongly recommend you let Cluster API create the Classic ELB.
   163  
   164  ### Caveats/Notes
   165  
   166  * When both public and private subnets are available in an AZ, CAPI will choose the private subnet in the AZ over the public subnet for placing EC2 instances.
   167  * If you configure CAPI to use existing infrastructure as outlined above, CAPI will _not_ create an SSH bastion host. Combined with the previous bullet, this means you must make sure you have established some form of connectivity to the instances that CAPI will create.
   168  
   169  ## Using Externally managed AWS Clusters
   170  
   171  ### Overview
   172  
   173  Alternatively, CAPA supports externally managed cluster infrastructure which is useful for scenarios where a different persona is managing the cluster infrastructure out-of-band(external system) while still wanting to use CAPI for automated machine management.
   174  Users can make use of existing AWSCluster CRDs in their externally managed clusters.
   175  
   176  ### How to use externally managed clusters?
   177  
   178  Users have to use `cluster.x-k8s.io/managed-by: "<name-of-system>"` annotation to depict that AWS resources are managed externally. If CAPA controllers come across this annotation in any of the AWS resources while reconciliation, then it will ignore the resource and not perform any reconciliation(including creating/modifying any of the AWS resources, or it's status).
   179  
   180  A predicate `ResourceIsNotExternallyManaged` is exposed by Cluster API which allows CAPA controllers to differentiate between externally managed vs CAPA managed resources. For example:
   181  ```go
   182  c, err := ctrl.NewControllerManagedBy(mgr).
   183          For(&providerv1.InfraCluster{}).
   184          Watches(...).
   185          WithOptions(options).
   186          WithEventFilter(predicates.ResourceIsNotExternallyManaged(ctrl.LoggerFrom(ctx))).
   187          Build(r)
   188  if err != nil {
   189  	return errors.Wrap(err, "failed setting up with a controller manager")
   190  }
   191  ```
   192  The external system must provide all required fields within the spec of the AWSCluster and must adhere to the CAPI provider contract and set the AWSCluster status to be ready when it is appropriate to do so.
   193  
   194  > **IMPORTANT NOTE**: Users should take care of skipping reconciliation in external controllers within mapping function while enqueuing requests. For example:
   195  > ```go
   196  > err := c.Watch(
   197  >		&source.Kind{Type: &infrav1.AWSCluster{}},
   198  >		handler.EnqueueRequestsFromMapFunc(func(a client.Object) []reconcile.Request {
   199  >		   if annotations.IsExternallyManaged(awsCluster) {
   200  >			    log.Info("AWSCluster is externally managed, skipping mapping.")
   201  >			    return nil
   202  >		   }
   203  >            return []reconcile.Request{
   204  >	           {
   205  >		         NamespacedName: client.ObjectKey{Namespace: c.Namespace, Name: c.Spec.InfrastructureRef.Name},
   206  >	           },
   207  >	        }}))
   208  > if err != nil {
   209  >    // handle it
   210  > }
   211  > ```
   212  
   213  ### Caveats
   214  Once the user has created externally managed AWSCluster, it is not allowed to convert it to CAPA managed cluster. However, converting from managed to externally managed is allowed.
   215  
   216  User should only use this feature if their cluster infrastructure lifecycle management has constraints that the reference implementation does not support. See [user stories](https://github.com/kubernetes-sigs/cluster-api/blob/10d89ceca938e4d3d94a1d1c2b60515bcdf39829/docs/proposals/20210203-externally-managed-cluster-infrastructure.md#user-stories) for more details.