github.com/cilium/cilium@v1.16.2/Documentation/network/ebpf/maps.rst (about)

     1  .. only:: not (epub or latex or html)
     2  
     3      WARNING: You are looking at unreleased Cilium documentation.
     4      Please use the official rendered version released here:
     5      https://docs.cilium.io
     6  
     7  .. _bpf_map_limitations:
     8  
     9  eBPF Maps
    10  =========
    11  
    12  All BPF maps are created with upper capacity limits. Insertion beyond the limit
    13  will fail and thus limits the scalability of the datapath. The following table
    14  shows the default values of the maps. Each limit can be bumped in the source
    15  code. Configuration options will be added on request if demand arises.
    16  
    17  ======================== ================ =============== =====================================================
    18  Map Name                 Scope            Default Limit   Scale Implications
    19  ======================== ================ =============== =====================================================
    20  Auth                     node             512k            Max 512k authenticated relations per node
    21  Connection Tracking      node or endpoint 1M TCP/256k UDP Max 1M concurrent TCP connections, max 256k expected UDP answers
    22  NAT                      node             512k            Max 512k NAT entries
    23  Neighbor Table           node             512k            Max 512k neighbor entries
    24  Endpoints                node             64k             Max 64k local endpoints + host IPs per node
    25  IP cache                 node             512k            Max 256k endpoints (IPv4+IPv6), max 512k endpoints (IPv4 or IPv6) across all clusters
    26  Load Balancer            node             64k             Max 64k cumulative backends across all services across all clusters
    27  Policy                   endpoint         16k             Max 16k allowed identity + port + protocol pairs for specific endpoint
    28  Proxy Map                node             512k            Max 512k concurrent redirected TCP connections to proxy
    29  Tunnel                   node             64k             Max 32k nodes (IPv4+IPv6) or 64k nodes (IPv4 or IPv6) across all clusters
    30  IPv4 Fragmentation       node             8k              Max 8k fragmented datagrams in flight simultaneously on the node
    31  Session Affinity         node             64k             Max 64k affinities from different clients
    32  IPv4 Masq                node             16k             Max 16k IPv4 cidrs used by BPF-based ip-masq-agent
    33  IPv6 Masq                node             16k             Max 16k IPv6 cidrs used by BPF-based ip-masq-agent
    34  Service Source Ranges    node             64k             Max 64k cumulative LB source ranges across all services
    35  Egress Policy            endpoint         16k             Max 16k endpoints across all destination CIDRs across all clusters 
    36  Node                     node             16k             Max 16k distinct node IPs (IPv4 & IPv6) across all clusters.
    37  ======================== ================ =============== =====================================================
    38  
    39  For some BPF maps, the upper capacity limit can be overridden using command
    40  line options for ``cilium-agent``. A given capacity can be set using
    41  ``--bpf-auth-map-max``, ``--bpf-ct-global-tcp-max``, ``--bpf-ct-global-any-max``,
    42  ``--bpf-nat-global-max``, ``--bpf-neigh-global-max``, ``--bpf-policy-map-max``,
    43  ``--bpf-fragments-map-max`` and ``--bpf-lb-map-max``.
    44  
    45  .. Note::
    46  
    47     In case the ``--bpf-ct-global-tcp-max`` and/or ``--bpf-ct-global-any-max``
    48     are specified, the NAT table size (``--bpf-nat-global-max``) must not exceed
    49     2/3 of the combined CT table size (TCP + UDP). This will automatically be set
    50     if either ``--bpf-nat-global-max`` is not explicitly set or if dynamic BPF
    51     map sizing is used (see below).
    52  
    53  Using the ``--bpf-map-dynamic-size-ratio`` flag, the upper capacity limits of
    54  several large BPF maps are determined at agent startup based on the given ratio
    55  of the total system memory. For example, a given ratio of 0.0025 leads to 0.25%
    56  of the total system memory to be used for these maps.
    57  
    58  This flag affects the following BPF maps that consume most memory in the system:
    59  ``cilium_ct_{4,6}_global``, ``cilium_ct_{4,6}_any``,
    60  ``cilium_nodeport_neigh{4,6}``, ``cilium_snat_v{4,6}_external`` and
    61  ``cilium_lb{4,6}_reverse_sk``.
    62  
    63  ``kube-proxy`` sets as the maximum number entries in the linux's connection
    64  tracking table based on the number of cores the machine has. ``kube-proxy`` has
    65  a default of ``32768`` maximum entries per core with a minimum of ``131072``
    66  entries regardless of the number of cores the machine has.
    67  
    68  Cilium has its own connection tracking tables as BPF Maps and the number of
    69  entries of such maps is calculated based on the amount of total memory in the
    70  node with a minimum of ``131072`` entries regardless the amount of memory the
    71  machine has.
    72  
    73  The following table presents the value that ``kube-proxy`` and Cilium sets for
    74  their own connection tracking tables when Cilium is configured with
    75  ``--bpf-map-dynamic-size-ratio: 0.0025``.
    76  
    77  +------+--------------+-----------------------+-------------------+
    78  | vCPU | Memory (GiB) | Kube-proxy CT entries | Cilium CT entries |
    79  +------+--------------+-----------------------+-------------------+
    80  |    1 |         3.75 |                131072 |            131072 |
    81  +------+--------------+-----------------------+-------------------+
    82  |    2 |          7.5 |                131072 |            131072 |
    83  +------+--------------+-----------------------+-------------------+
    84  |    4 |           15 |                131072 |            131072 |
    85  +------+--------------+-----------------------+-------------------+
    86  |    8 |           30 |                262144 |            284560 |
    87  +------+--------------+-----------------------+-------------------+
    88  |   16 |           60 |                524288 |            569120 |
    89  +------+--------------+-----------------------+-------------------+
    90  |   32 |          120 |               1048576 |           1138240 |
    91  +------+--------------+-----------------------+-------------------+
    92  |   64 |          240 |               2097152 |           2276480 |
    93  +------+--------------+-----------------------+-------------------+
    94  |   96 |          360 |               3145728 |           4552960 |
    95  +------+--------------+-----------------------+-------------------+