sigs.k8s.io/cluster-api-provider-azure@v1.14.3/docs/book/src/topics/custom-vnet.md (about)

     1  # Custom Virtual Networks
     2  
     3  ## Pre-existing vnet and subnets
     4  
     5  To deploy a cluster using a pre-existing vnet, modify the `AzureCluster` spec to include the name and resource group of the existing vnet as follows, as well as the control plane and node subnets as follows:
     6  
     7  ```yaml
     8  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
     9  kind: AzureCluster
    10  metadata:
    11    name: cluster-byo-vnet
    12    namespace: default
    13  spec:
    14    location: southcentralus
    15    networkSpec:
    16      vnet:
    17        resourceGroup: custom-vnet
    18        name: my-vnet
    19      subnets:
    20        - name: my-control-plane-subnet
    21          role: control-plane
    22          securityGroup:
    23            name: my-control-plane-nsg
    24        - name: my-node-subnet
    25          role: node
    26          routeTable:
    27            name: my-node-routetable
    28          securityGroup:
    29            name: my-node-nsg
    30    resourceGroup: cluster-byo-vnet
    31    ```
    32  
    33  When providing a vnet, it is required to also provide the two subnets that should be used for control planes and nodes.
    34  
    35  If providing an existing vnet and subnets with existing network security groups, make sure that the control plane security group allows inbound to port 6443, as port 6443 is used by kubeadm to bootstrap the control planes. Alternatively, you can [provide a custom control plane endpoint](https://github.com/kubernetes-sigs/cluster-api-bootstrap-provider-kubeadm#kubeadmconfig-objects) in the `KubeadmConfig` spec.
    36  
    37  The pre-existing vnet can be in the same resource group or a different resource group in the same subscription as the target cluster. When deleting the `AzureCluster`, the vnet and resource group will only be deleted if they are "managed" by capz, ie. they were created during cluster deployment. Pre-existing vnets and resource groups will *not* be deleted.
    38  
    39  ## Virtual Network Peering
    40  
    41  Alternatively, pre-existing vnets can be peered with a cluster's newly created vnets by specifying each vnet by name and resource group.
    42  
    43  ```yaml
    44  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    45  kind: AzureCluster
    46  metadata:
    47    name: cluster-vnet-peering
    48    namespace: default
    49  spec:
    50    location: southcentralus
    51    networkSpec:
    52      vnet:
    53        name: my-vnet
    54        cidrBlocks:
    55          - 10.255.0.0/16
    56        peerings:
    57        - resourceGroup: vnet-peering-rg
    58          remoteVnetName: existing-vnet-1
    59        - resourceGroup: vnet-peering-rg
    60          remoteVnetName: existing-vnet-2
    61      subnets:
    62        - name: my-subnet-cp
    63          role: control-plane
    64          cidrBlocks:
    65            - 10.255.0.0/24
    66        - name: my-subnet-node
    67          role: node
    68          cidrBlocks:
    69            - 10.255.1.0/24
    70    resourceGroup: cluster-vnet-peering
    71    ```
    72  
    73  Currently, only virtual networks on the same subscription can be peered. Also, note that when creating workload clusters with internal load balancers, the management cluster must be in the same VNet or a peered VNet. See [here](https://capz.sigs.k8s.io/topics/api-server-endpoint.html#warning) for more details.
    74  
    75  ## Custom Network Spec
    76  
    77  It is also possible to customize the vnet to be created without providing an already existing vnet. To do so, simply modify the `AzureCluster` `NetworkSpec` as desired. Here is an illustrative example of a cluster with a customized vnet address space (CIDR) and customized subnets:
    78  
    79  ```yaml
    80  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    81  kind: AzureCluster
    82  metadata:
    83    name: cluster-example
    84    namespace: default
    85  spec:
    86    location: southcentralus
    87    networkSpec:
    88      vnet:
    89        name: my-vnet
    90        cidrBlocks:
    91          - 10.0.0.0/16
    92      subnets:
    93        - name: my-subnet-cp
    94          role: control-plane
    95          cidrBlocks:
    96            - 10.0.1.0/24
    97        - name: my-subnet-node
    98          role: node
    99          cidrBlocks:
   100            - 10.0.2.0/24
   101    resourceGroup: cluster-example
   102  ```
   103  
   104  If no CIDR block is provided, `10.0.0.0/8` will be used by default, with default internal LB private IP `10.0.0.100`.
   105  
   106  ### Custom Security Rules
   107  
   108  <aside class="note">
   109  
   110  <h1> Note </h1>
   111  
   112  Security Rules were previously known as `ingressRule` in v1alpha3.
   113  
   114  </aside>
   115  
   116  Security rules can also be customized as part of the subnet specification in a custom network spec.
   117  Note that ingress rules for the Kubernetes API Server port (default 6443) and SSH (22) are automatically added to the controlplane subnet only if security rules aren't specified.
   118  It is the responsibility of the user to supply those rules themselves if using custom rules.
   119  
   120  Here is an illustrative example of customizing rules that builds on the one above by adding an egress rule to the control plane nodes:
   121  
   122  ```yaml
   123  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
   124  kind: AzureCluster
   125  metadata:
   126    name: cluster-example
   127    namespace: default
   128  spec:
   129    location: southcentralus
   130    networkSpec:
   131      vnet:
   132        name: my-vnet
   133        cidrBlocks:
   134          - 10.0.0.0/16
   135      subnets:
   136        - name: my-subnet-cp
   137          role: control-plane
   138          cidrBlocks:
   139            - 10.0.1.0/24
   140          securityGroup:
   141            name: my-subnet-cp-nsg
   142            securityRules:
   143              - name: "allow_ssh"
   144                description: "allow SSH"
   145                direction: "Inbound"
   146                priority: 2200
   147                protocol: "*"
   148                destination: "*"
   149                destinationPorts: "22"
   150                source: "*"
   151                sourcePorts: "*"
   152                action: "Allow"
   153              - name: "allow_apiserver"
   154                description: "Allow K8s API Server"
   155                direction: "Inbound"
   156                priority: 2201
   157                protocol: "*"
   158                destination: "*"
   159                destinationPorts: "6443"
   160                source: "*"
   161                sourcePorts: "*"
   162                action: "Allow"
   163              - name: "allow_port_50000"
   164                description: "allow port 50000"
   165                direction: "Outbound"
   166                priority: 2202
   167                protocol: "Tcp"
   168                destination: "*"
   169                destinationPorts: "50000"
   170                source: "*"
   171                sourcePorts: "*"
   172                action: "Allow"
   173        - name: my-subnet-node
   174          role: node
   175          cidrBlocks:
   176            - 10.0.2.0/24
   177    resourceGroup: cluster-example
   178  ```
   179  
   180  ### Virtual Network service endpoints
   181  
   182  Sometimes it's desirable to use [Virtual Network service endpoints](https://learn.microsoft.com/azure/virtual-network/virtual-network-service-endpoints-overview) to establish secure and direct connectivity to Azure services from your subnet(s). Service Endpoints are configured on a per-subnet basis. Vnets managed by either `AzureCluster` or `AzureManagedControlPlane` can have `serviceEndpoints` optionally set on each subnet.
   183  
   184  ```yaml
   185  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
   186  kind: AzureCluster
   187  metadata:
   188    name: cluster-example
   189    namespace: default
   190  spec:
   191    location: southcentralus
   192    networkSpec:
   193      vnet:
   194        name: my-vnet
   195        cidrBlocks:
   196          - 10.0.0.0/16
   197      subnets:
   198        - name: my-subnet-cp
   199          role: control-plane
   200          cidrBlocks:
   201            - 10.0.1.0/24
   202          serviceEndpoints:
   203            - service: Microsoft.AzureActiveDirectory
   204              locations: ["*"]
   205        - name: my-subnet-node
   206          role: node
   207          cidrBlocks:
   208            - 10.0.2.0/24
   209          serviceEndpoints:
   210            - service: Microsoft.AzureActiveDirectory
   211              locations: ["*"]
   212            - service: Microsoft.Storage
   213              locations: ["southcentralus"]
   214    resourceGroup: cluster-example
   215  ```
   216  
   217  ### Private Endpoints
   218  
   219  A [Private Endpoint](https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overview) is a network interface that uses
   220  a private IP address from your virtual network. This network interface connects you privately and securely to a service that's powered
   221  by [Azure Private Link](https://learn.microsoft.com/en-us/azure/private-link/private-link-overview). Azure Private Link enables you
   222  to access Azure PaaS Services (for example, Azure Storage and SQL Database) and Azure hosted customer-owned/partner services over a
   223  private endpoint in your virtual network.
   224  
   225   Private Endpoints are configured on a per-subnet basis. Vnets managed by either
   226  `AzureCluster`, `AzureClusterTemplates` or `AzureManagedControlPlane` can have `privateEndpoints` optionally set on each subnet.
   227  
   228  - `AzureCluster` example:
   229  
   230  ```yaml
   231  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
   232  kind: AzureCluster
   233  metadata:
   234    name: cluster-example
   235    namespace: default
   236  spec:
   237    location: eastus2
   238    resourceGroup: cluster-example
   239    networkSpec:
   240      vnet:
   241        name: my-vnet
   242        cidrBlocks:
   243          - 10.0.0.0/16
   244      subnets:
   245        - name: my-subnet-cp
   246          role: control-plane
   247          cidrBlocks:
   248            - 10.0.1.0/24
   249        - name: my-subnet-node
   250          role: node
   251          cidrBlocks:
   252            - 10.0.2.0/24
   253          privateEndpoints:
   254           - name: my-pe
   255             privateLinkServiceConnections:
   256             - privateLinkServiceID: /subscriptions/<Subscription ID>/resourceGroups/<Remote Resource Group Name>/providers/Microsoft.Network/privateLinkServices/<Private Link Service Name>
   257  ```
   258  
   259  - `AzureManagedControlPlane` example:
   260  
   261  ```yaml
   262  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
   263  kind: AzureManagedControlPlane
   264  metadata:
   265    name: cluster-example
   266    namespace: default
   267  spec:
   268    version: v1.25.2
   269    sshPublicKey: ""
   270    identityRef:
   271      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
   272      kind: AzureClusterIdentity
   273      name: cluster-identity
   274    location: eastus2
   275    resourceGroupName: cluster-example
   276    virtualNetwork:
   277      name: my-vnet
   278      cidrBlock: 10.0.0.0/16
   279      subnet:
   280        cidrBlock: 10.0.2.0/24
   281        name: my-subnet
   282        privateEndpoints:
   283        - name: my-pe
   284          customNetworkInterfaceName: nic-my-pe # optional
   285          applicationSecurityGroups: # optional
   286          - <ASG ID>
   287          privateIPAddresses: # optional
   288          - 10.0.2.10
   289          location: eastus2 # optional
   290          privateLinkServiceConnections:
   291          - name: my-pls # optional
   292            privateLinkServiceID: /subscriptions/<Subscription ID>/resourceGroups/<Remote Resource Group Name>/providers/Microsoft.Storage/storageAccounts/<Name>
   293            groupIds:
   294            - "blob"
   295  ```
   296  
   297  ### Custom subnets
   298  
   299  Sometimes it's desirable to use different subnets for different node pools.
   300  Several subnets can be specified in the `networkSpec` to be later referenced by name from other CR's like `AzureMachine` or `AzureMachinePool`.
   301  When more than one `node` subnet is specified, the `subnetName` field in those other CR's becomes mandatory because the controllers wouldn't know which subnet to use.
   302  
   303  The subnet used for the control plane must use the role `control-plane` while the subnets for the worker nodes must use the role `node`.
   304  
   305  
   306  ```yaml
   307  ---
   308  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
   309  kind: AzureCluster
   310  metadata:
   311    name: cluster-example
   312    namespace: default
   313  spec:
   314    location: southcentralus
   315    networkSpec:
   316      subnets:
   317      - name: control-plane-subnet
   318        role: control-plane
   319      - name: subnet-mp-1
   320        role: node
   321      - name: subnet-mp-2
   322        role: node
   323      vnet:
   324        name: my-vnet
   325        cidrBlocks:
   326          - 10.0.0.0/16
   327    resourceGroup: cluster-example
   328  ---
   329  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
   330  kind: AzureMachinePool
   331  metadata:
   332    name: mp1
   333    namespace: default
   334  spec:
   335    location: southcentralus
   336    strategy:
   337      rollingUpdate:
   338        deletePolicy: Oldest
   339        maxSurge: 25%
   340        maxUnavailable: 1
   341      type: RollingUpdate
   342    template:
   343      osDisk:
   344        diskSizeGB: 30
   345        managedDisk:
   346          storageAccountType: Premium_LRS
   347        osType: Linux
   348      sshPublicKey: ${YOUR_SSH_PUB_KEY}
   349      subnetName: subnet-mp-1
   350      vmSize: Standard_B2s
   351  ---
   352  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
   353  kind: AzureMachinePool
   354  metadata:
   355    name: mp2
   356    namespace: default
   357  spec:
   358    location: southcentralus
   359    strategy:
   360      rollingUpdate:
   361        deletePolicy: Oldest
   362        maxSurge: 25%
   363        maxUnavailable: 1
   364      type: RollingUpdate
   365    template:
   366      osDisk:
   367        diskSizeGB: 30
   368        managedDisk:
   369          storageAccountType: Premium_LRS
   370        osType: Linux
   371      sshPublicKey: ${YOUR_SSH_PUB_KEY}
   372      subnetName: subnet-mp-2
   373      vmSize: Standard_B2s
   374  ```
   375  
   376  If you don't specify any `node` subnets, one subnet with role `node` will be created and added to the `networkSpec` definition.