github.com/cilium/cilium@v1.16.2/Documentation/network/kubernetes/ciliumendpointslice.rst (about)

     1  .. only:: not (epub or latex or html)
     2  
     3      WARNING: You are looking at unreleased Cilium documentation.
     4      Please use the official rendered version released here:
     5      https://docs.cilium.io
     6  
     7  .. _CiliumEndpointSlice:
     8  
     9  ***************************
    10  CiliumEndpointSlice
    11  ***************************
    12  
    13  .. note::
    14      This is a beta feature. Please provide feedback and file a GitHub issue
    15      if you experience any problems.
    16  
    17      The tasks needed for graduating this feature "Stable" are documented
    18      in :gh-issue:`31904`.
    19  
    20  This document describes CiliumEndpointSlices (CES), which enable batching of
    21  CiliumEndpoint (CEP) objects in the cluster to achieve better scalability.
    22  
    23  When enabled, Cilium Operator watches CEP objects and groups/batches slim versions
    24  of them into CES objects. Cilium Agent watches CES objects to learn about
    25  remote endpoints in this mode. API-server stress due to remote endpoint info
    26  propagation should be reduced in this case, allowing for better scalability,
    27  at the cost of potentially longer delay before identities of new endpoints are
    28  recognized throughout the cluster.
    29  
    30  .. note::
    31  
    32     CiliumEndpointSlice is a concept that is specific to Cilium and is not
    33     related to `Kubernetes' EndpointSlice`_. Although the names are similar, and
    34     even though the concept of slices in each feature brings similar
    35     improvements for scalability, they address different problems.
    36  
    37     Kubernetes' Endpoints and EndpointSlices allow Cilium to make load-balancing
    38     decisions for a particular Service object; Kubernetes' EndpointSlices offer
    39     a scalable way to track Service back-ends within a cluster.
    40  
    41     By contrast, CiliumEndpoints and CiliumEndpointSlices are used to make
    42     network routing and policy decisions. So CiliumEndpointSlices focus on
    43     tracking Pods, batching CEPs to reduce the number of updates to propagate
    44     through the API-server on large clusters.
    45  
    46     Enabling one does not affect the other.
    47  
    48  .. _Kubernetes' EndpointSlice: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
    49  
    50  Deploy Cilium with CES
    51  =======================
    52  
    53  CES are disabled by default. This section describes the steps necessary for enabling them.
    54  
    55  Pre-Requisites
    56  ~~~~~~~~~~~~~~
    57  
    58  * Make sure that CEPs are enabled (the ``--disable-endpoint-crd`` flag is not set to ``true``)
    59  * Make sure you are not relying on the Egress Gateway which is not compatible with CES (see Egress Gateway :ref:`egress-gateway-incompatible-features`)
    60  
    61  Migration Procedure
    62  ~~~~~~~~~~~~~~~~~~~
    63  In order to minimize endpoint propagation delays, it is recommended to upgrade the Operator first,
    64  let it create all CES objects, and then upgrade the Agents afterwards.
    65  
    66  #. Enable CES on the Operator by setting the ``ciliumEndpointSlice.enabled`` value to ``true`` in your Helm chart or
    67     by directly setting the ``--enable-cilium-endpoint-slice`` flag to ``true`` on the Operator. Re-deploy the Operator.
    68  
    69  #. Once the Operator is running, verify that the ``CiliumEndpointSlice`` CRD has been successfully registered:
    70  
    71     .. code-block:: shell-session
    72  
    73        $ kubectl get crd ciliumendpointslices.cilium.io
    74        NAME                                         CREATED AT
    75        ciliumendpointslices.cilium.io               2021-11-05T05:41:28Z
    76  
    77  #. Verify that the Operator has started creating CES objects:
    78  
    79     .. code-block:: shell-session
    80  
    81        $ kubectl get ces
    82        NAME                  AGE
    83        ces-2fvynpvzn-4ncg9   1m17s
    84        ces-2jyqj8pfl-tdfm8   1m20s
    85  
    86  #. Let the Operator create CES objects for all existing CEPs in the cluster. This may take some time, depending on the
    87     size of the cluster. You can monitor the progress by checking the rate of CES object creation in the cluster, for example by
    88     looking at the ``apiserver_storage_objects`` Kubernetes metric or by looking at ``ciliumendpointslices`` resource
    89     creation requests in Kubernetes Audit Logs. You can also monitor the metrics emitted by the Operator, such as ``cilium_operator_ces_sync_total``. All CES-related metrics are documented in the :ref:`ces_metrics` section of the metric documentation.
    90  
    91  #. Once the metrics have stabilized (in other words, when the Operator has created CES objects for all existing CEPs), upgrade the
    92     Cilium Agents on all nodes by setting the ``--enable-cilium-endpoint-slice`` flag to ``true`` and re-deploying them.
    93  
    94  
    95  Configuration Options
    96  =====================
    97  
    98  Several options are available to adjust the performance and behavior of the CES feature:
    99  
   100  * You can configure the way CEPs are batched into CES by changing the maximum number of CEPs in a
   101    CES (``--ces-max-cilium-endpoints-per-ces``) or by changing the way CEPs are grouped into CES (``--ces-slice-mode``).
   102    Right now two modes are supported: ``cesSliceModeIdentity`` which groups CEPs based on :ref:`security_identities`
   103    and ``cesSliceModeFCFS`` which groups CEPs on a "First Come, First Served" basis.
   104  
   105  * You can also fine-tune rate-limiting settings for the Operator communications with the API-server. Refer to the ``--ces-*`` flags for the ``cilium-operator`` binary.
   106  
   107  Known Issues and Workarounds
   108  ============================
   109  
   110  Potential Race Condition when Identity of an Existing Endpoint Changes
   111  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   112  When there's an identity change for any existing resource without the pods being re-created
   113  (this can happen when the namespace labels change), in a very unlikely situation, the endpoints that
   114  undergo this change might experience connection disruption.
   115  
   116  Root cause for this potential disruption is that when identity of CEPs
   117  change, the operator will try to re-group/re-batch them into a different
   118  set of CESs. This breaks the atomic operation of an UPGRADE into that of
   119  an DELETE and an ADD. If the agent gets the DELETE (from old CES) first,
   120  it will remove the corresponding CEP's information from the ipcache,
   121  resulting in traffic to/from said CEP with an UNKNOWN identity.
   122  
   123  In current implementation, Cilium adds a delay (default: 1s) before sending
   124  out the DELETE event. This should greatly reduce the probability of
   125  connection disruption in most cases.