github.com/cilium/cilium@v1.16.2/Documentation/network/concepts/ipam/multi-pool.rst (about)

     1  .. only:: not (epub or latex or html)
     2  
     3      WARNING: You are looking at unreleased Cilium documentation.
     4      Please use the official rendered version released here:
     5      http://docs.cilium.io
     6  
     7  .. _ipam_crd_multi_pool:
     8  
     9  Multi-Pool (Beta)
    10  #################
    11  
    12  .. include:: ../../../beta.rst
    13  
    14  The Multi-Pool IPAM mode supports allocating PodCIDRs from multiple different IPAM pools, depending
    15  on properties of the workload defined by the user, e.g. annotations.
    16  
    17  Architecture
    18  ************
    19  
    20  .. image:: multi-pool.png
    21      :align: center
    22  
    23  When running in the Multi-Pool IPAM mode, Cilium will use the ``ipam.cilium.io/ip-pool`` annotation
    24  on pods and namespaces to determine the IPAM pool from which a pod's IP is allocated from.
    25  
    26    1. If there is an ``ipam.cilium.io/ip-pool=A`` annotation on the pod itself, Cilium will
    27       allocate the pod's IP from the pool named ``A``.
    28    2. If there is no annotation on the pod, but the namespace of the pod has an
    29       ``ipam.cilium.io/ip-pool=B`` annotation, Cilium will
    30       allocate the pod's IP from the pool named ``B``.
    31    3. If neither the pod nor the namespace have a ``ipam.cilium.io/ip-pool`` annotation,
    32       the pod's IP will be allocated from the pool named ``default``.
    33  
    34  The annotation is only considered when a pod is created. Changing the ``ip-pool``
    35  annotation on an already running pod has no effect.
    36  
    37  The ``CiliumNode`` resource is extended with an additional ``spec.ipam.pools`` section:
    38  
    39  ``spec.ipam.pools.requested``
    40    List of IPAM pool requests for this node. Each entry specifies the pool and the number of
    41    requested IP addresses. This field is owned and written to by the Cilium agent running on
    42    the particular node. It is read by the Cilium operator to fulfill the requests.
    43  
    44  ``spec.ipam.pools.allocated``
    45    List of CIDRs allocated to this node and the pool they were allocated from.
    46    Cilium operator adds new PodCIDRs to this field. Cilium agent removes PodCIDRs
    47    it has released and is no longer using.
    48  
    49  IP pools are managed using the cluster-wide ``CiliumPodIPPool`` custom resource.
    50  Each ``CiliumPodIPPool`` contains the cluster-wide CIDR from which per-node
    51  PodCIDRs are allocated:
    52  
    53  .. code-block:: yaml
    54  
    55    apiVersion: cilium.io/v2alpha1
    56    kind: CiliumPodIPPool
    57    metadata:
    58      name: green-pool
    59    spec:
    60      ipv4:
    61        cidrs:
    62          - 10.20.0.0/16
    63          - 10.30.0.0/16
    64        maskSize: 24
    65      ipv6:
    66        cidrs:
    67          - fd00::/104
    68        maskSize: 120
    69  
    70  New pools can be added at run-time. The list of CIDRs in each pool can also be
    71  extended at run-time. In-use CIDRs may not be removed from an existing pool, and
    72  existing pools may not be deleted if they are still in use by a Cilium node.
    73  The mask size of a pool is immutable and the same for all nodes. Neither restriction
    74  is enforced until :gh-issue:`26966` is resolved.
    75  
    76  Configuration
    77  *************
    78  
    79  Multi-Pool IPAM can be enabled using the ``ipam.mode=multi-pool`` Helm value.
    80  To have the Cilium operator automatically create ``CiliumPodIPPools`` custom
    81  resources at startup, use the ``ipam.operator.autoCreateCiliumPodIPPools`` Helm
    82  value. It contains a map which follows the ``CiliumPodIPPools`` CRD schema
    83  described above.
    84  
    85  .. code-block:: yaml
    86  
    87    ipam:
    88      mode: multi-pool
    89      operator:
    90        autoCreateCiliumPodIPPools:
    91          default:
    92            ipv4:
    93              cidrs:
    94                - 10.10.0.0/16
    95              maskSize: 24
    96          other:
    97            ipv4:
    98              cidrs:
    99                - 10.20.0.0/16
   100              maskSize: 24
   101  
   102  .. note::
   103  
   104    For a practical tutorial on how to enable this mode in Cilium, see
   105    :ref:`gsg_ipam_crd_multi_pool`.
   106  
   107  Allocation Parameters
   108  ---------------------
   109  
   110  Cilium agent can be configured to pre-allocate IPs from each pool. This behavior
   111  can be controlled using the ``ipam-multi-pool-pre-allocation`` flag. It
   112  contains a key-value map of the form ``<pool-name>=<preAllocIPs>`` where
   113  ``preAllocIPs`` specifies how many IPs are to be pre-allocated to the local
   114  node. The same number of IPs are pre-allocated for each address family. This
   115  means that a pool which contains both IPv4 and IPv6 CIDRs will pre-allocate
   116  ``preAllocIPs`` IPv4 addresses and ``preAllocIPs`` IPv6 addresses.
   117  
   118  The flag defaults to ``default=8``, which means it will pre-allocate 8 IPs from
   119  the ``default`` pool. All other pools which do not have an entry in the
   120  ``ipam-multi-pool-pre-allocation`` map are assumed to have a ``preAllocIPs`` of
   121  zero, i.e. no IPs are pre-allocated for that pool.
   122  
   123  Depending on the number of in-use IPs and the number of pending IP allocation
   124  requests, Cilium agent might pre-allocate more than ``preAllocIPs`` IPs. The
   125  formula Cilium agent uses to compute the absolute number of needed IPs from each
   126  pool is:
   127  
   128  .. code-block:: go
   129  
   130    neededIPs = roundUp(inUseIPs + pendingIPs + preAllocIPs, preAllocIPs)
   131  
   132  Where ``inUseIPs`` is the number of IPs that are currently in use,
   133  ``pendingIPs`` number of IPs that have a pending pod (i.e. pods which have been
   134  scheduled on the node, but not yet received an IP), and ``preAllocIPs`` is the
   135  minimum number of IPs that we want to pre-allocate as a buffer, i.e. the value
   136  taken from the ``ipam-multi-pool-pre-allocation`` map.
   137  
   138  Routing to Allocated PodCIDRs
   139  -----------------------------
   140  
   141  PodCIDRs allocated from ``CiliumPodIPPools`` can be announced to the network by the
   142  :ref:`bgp_control_plane` (:ref:`bgp_control_plane_multipool_ipam`). Alternatively,
   143  the ``autoDirectNodeRoutes`` Helm option can be used to enable automatic routing
   144  between nodes on a L2 network.
   145  
   146   .. _ipam_crd_multi_pool_limitations:
   147  
   148  Limitations
   149  ***********
   150  
   151  Multi-Pool IPAM is a preview feature. The following limitations apply to Cilium running in
   152  Multi-Pool IPAM mode:
   153  
   154  .. warning::
   155     - Tunnel mode is not supported. Multi-Pool IPAM may only be used in direct routing mode.
   156     - Transparent encryption is only supported with WireGuard and cannot be used with IPsec.
   157     - IPAM pools with overlapping CIDRs are not supported. Each pod IP must be
   158       unique in the cluster due the way Cilium determines the security identity
   159       of endpoints by way of the IPCache.
   160     - Multi-Pool IPAM does not support local node routes (``enable-local-node-route``) and
   161       requires the use of per-endpoint routes (see :ref:`native_routing`) instead.
   162     - iptables-based masquerading requires ``egressMasqueradeInterfaces`` to be set
   163       (see masquerading :ref:`masq_modes` and :gh-issue:`22273` for details).
   164       Alternatively, eBPF-based masquerading is fully supported and may be used instead.
   165       Note that if the used IPAM pools do not belong to a common native-routing CIDR,
   166       you may want to use ``ip-masq-agent``, which allows multiple disjunct non-masquerading
   167       CIDRs to be defined. See :ref:`concepts_masquerading` for details on how to use the
   168       ``ip-masq-agent`` feature.