github.com/cilium/cilium@v1.16.2/Documentation/network/servicemesh/ingress-reference.rst (about)

     1  Reference
     2  #########
     3  
     4  How Cilium Ingress and Gateway API differ from other Ingress controllers
     5  ************************************************************************
     6  
     7  One of the biggest differences between Cilium's Ingress and Gateway API support
     8  and other Ingress controllers is how closely tied the implementation is to the
     9  CNI. For Cilium, Ingress and Gateway API are part of the networking stack,
    10  and so behave in a different way to other Ingress or Gateway API controllers
    11  (even other Ingress or Gateway API controllers running in a Cilium cluster).
    12  
    13  Other Ingress or Gateway API controllers are generally installed as a Deployment
    14  or Daemonset in the cluster, and exposed via a Loadbalancer Service or similar (which Cilium
    15  can, of course, enable).
    16  
    17  Cilium's Ingress and Gateway API config is exposed with a Loadbalancer or NodePort
    18  service, or optionally can be exposed on the Host network also. But in all of
    19  these cases, when traffic arrives at the Service's port, eBPF code intercepts
    20  the traffic and transparently forwards it to Envoy (using the TPROXY kernel facility).
    21  
    22  This affects things like client IP visibility, which works differently for Cilium's
    23  Ingress and Gateway API support to other Ingress controllers.
    24  
    25  It also allows Cilium's Network Policy engine to apply CiliumNetworkPolicy to
    26  traffic bound for and traffic coming from an Ingress.
    27  
    28  Cilium's ingress config and CiliumNetworkPolicy
    29  ***********************************************
    30  
    31  Ingress and Gateway API traffic bound to backend services via Cilium passes through a
    32  per-node Envoy proxy.
    33  
    34  The per-node Envoy proxy has special code that allows it to interact with the
    35  eBPF policy engine, and do policy lookups on traffic. This allows Envoy to be
    36  a Network Policy enforcement point, both for Ingress (and Gateway API) traffic,
    37  and also for east-west traffic via GAMMA or L7 Traffic Management.
    38  
    39  However, for ingress config, there's also an additional step. Traffic that arrives at
    40  Envoy *for Ingress or Gateway API* is assigned the special ``ingress`` identity
    41  in Cilium's Policy engine.
    42  
    43  Traffic coming from outside the cluster is usually assigned the ``world`` identity
    44  (unless there are IP CIDR policies in the cluster). This means that there are
    45  actually *two* logical Policy enforcement points in Cilium Ingress - before traffic
    46  arrives at the ``ingress`` identity, and after, when it is about to exit the
    47  per-node Envoy.
    48  
    49  .. image:: /images/ingress-policy.png
    50      :align: center
    51  
    52  This means that, when applying Network Policy to a cluster, it's important to
    53  ensure that both steps are allowed, and that traffic is allowed from ``world`` to
    54  ``ingress``, and from ``ingress`` to identities in the cluster (like the
    55  ``productpage`` identity in the image above).
    56  
    57  Please see the :ref:`gs_ingress_and_network_policy` for more details for Ingress,
    58  although the same principles also apply for Gateway API.
    59  
    60  Source IP Visibility
    61  ********************
    62  
    63  .. Note::
    64  
    65      By default, source IP visibility for Cilium ingress config, both Ingress
    66      and Gateway API, should *just work* on most installations. Read this section
    67      for more information on requirements and relevant settings.
    68  
    69  Having a backend be able to deduce what IP address the actual request came from
    70  is important for most applications.
    71  
    72  By default, Cilium's Envoy instances are configured to append the visible source
    73  address of incoming HTTP connections to the ``X-Forwarded-For`` header, using the
    74  usual rules. That is, by default Cilium sets the number of trusted hops to ``0``,
    75  indicating that Envoy should use the address the connection is opened from, rather
    76  than a value inside the ``X-Forwarded-For`` list. Increasing this count will
    77  have Envoy use the ``n`` th value from the list, counting from the right.
    78  
    79  Envoy will also set the ``X-Envoy-External-Address`` header to the trusted client
    80  address, whatever that turns out to be, based on ``X-Forwarded-For``.
    81  
    82  .. Note::
    83      
    84      Backends using Cilium ingress (whether via Ingress or Gateway API) should
    85      just see the ``X-Forwarded-For`` and ``X-Envoy-External-Address`` headers (which
    86      are handled transparently by many HTTP libraries).
    87  
    88  ``externalTrafficPolicy`` for Loadbalancer or NodePort Services
    89  ===============================================================
    90  
    91  Cilium's ingress support (both for Ingress and Gateway API) often uses a Loadbalancer
    92  or NodePort Service to expose the Envoy Daemonset.
    93  
    94  In these cases, the Service object has one field that is particularly relevant
    95  to Client IP visibility - the ``externalTrafficPolicy`` field.
    96  
    97  It has two relevant settings:
    98  
    99  - ``Local``: Nodes will only route traffic to Pods running on the local node, 
   100    *without masquerading the source IP*. Because of this, in clusters that use
   101    ``kube-proxy``, this is the only way to ensure source IP visibility. Part of
   102    the contract for ``externalTrafficPolicy`` local is also that the node will
   103    open a port (the ``healthCheckNodePort``, automatically set by Kubernetes when
   104    ``externalTrafficPolicy: Local`` is set), and requests to
   105    ``http://<nodeIP>:<healthCheckNodePort>/healthz`` will return 200 on nodes that
   106    have local pods running, and non-200 on nodes that don't. Cilium implements this
   107    for general Loadbalancer Services, but it's a bit different for Cilium ingress
   108    config (both Ingress and Gateway API).
   109  - ``Cluster``: Node will route to all endpoints across the cluster evenly. This
   110    has a couple of other effects: Firstly, upstream loadbalancers will expect to
   111    be able to send traffic to any node and have it end up at a backend Pod, and
   112    the node *may* masquerade the source IP. This means that in many cases,
   113    ``externalTrafficPolicy: Cluster`` may mean that the backend pod does *not* see
   114    the source IP.
   115  
   116  In Cilium's case, all ingress traffic bound for a Service that exposes Envoy is
   117  *always* going to the local node, and is *always* forwarded to Envoy using the
   118  Linux Kernel TPROXY function, which transparently forwards packets to the backend.
   119  
   120  This means that for Cilium ingress config, for both Ingress and Gateway API, things
   121  work a little differently in both ``externalTrafficPolicy`` cases.
   122  
   123  .. Note::
   124  
   125      In *both* ``externalTrafficPolicy`` cases, traffic will arrive at any node
   126      in the cluster, and be forwarded to *Envoy* **while keeping the source IP intact**.
   127  
   128  Also, for any Services that exposes Cilium's Envoy, Cilium will ensure that
   129  when ``externalTrafficPolicy: Local`` is set, every node in the cluster will
   130  pass the ``healthCheckNodePort`` check, so that external load balancers will
   131  forward correctly.
   132  
   133  However, for Cilium's ingress config, both Ingress and Gateway API, **it is not
   134  necessary** to configure ``externalTrafficPolicy: Local`` to keep the source IP
   135  visible to the backend pod (via the ``X-Forwarded-For`` and ``X-Envoy-External-Address``
   136  fields).
   137  
   138  TLS Passthrough and source IP visibility
   139  ========================================
   140  
   141  Both Ingress and Gateway API support TLS Passthrough configuration (via annotation
   142  for Ingress, and the TLSRoute resource for Gateway API). This configuration allows
   143  multiple TLS Passthrough backends to share the same TLS port on a loadbalancer,
   144  with Envoy inspecting the Server Name Indicator (SNI) field of the TLS handshake,
   145  and using that to forward the TLS stream to a backend.
   146  
   147  However, this poses problems for source IP visibility, because Envoy is doing a
   148  TCP Proxy of the TLS stream.
   149  
   150  What happens is that the TLS traffic arrives at Envoy, terminating a TCP stream,
   151  Envoy inspects the client hello to find the SNI, picks a backend to forward to,
   152  then starts a new TCP stream and forwards the TLS traffic inside the downstream
   153  (outside)  packets to the upstream (the backend).
   154  
   155  Because it's a new TCP stream, as far as the backends are concerned, the source
   156  IP is Envoy (which is often the Node IP, depending on your Cilium config).
   157  
   158  .. Note::
   159  
   160      When doing TLS Passthrough, backends will see Cilium Envoy's IP address
   161      as the source of the forwarded TLS streams.