github.com/datadog/cilium@v1.6.12/Documentation/concepts/networking.rst (about)

     1  .. only:: not (epub or latex or html)
     2  
     3      WARNING: You are looking at unreleased Cilium documentation.
     4      Please use the official rendered version released here:
     5      http://docs.cilium.io
     6  
     7  .. _arch_ip_connectivity:
     8  .. _multi host networking:
     9  
    10  *********************
    11  Multi Host Networking
    12  *********************
    13  
    14  Cilium is in full control over both ends of the connection for connections
    15  inside the cluster. It can thus transmit state and security context information
    16  between two container hosts by embedding the information in encapsulation
    17  headers or even unused bits of the IPv6 packet header. This allows Cilium to
    18  transmit the security context of where the packet originates, which allows
    19  tracing back which container labels are assigned to the origin container.
    20  
    21  .. note::
    22  
    23     As the packet headers contain security sensitive information, it is highly
    24     recommended to either encrypt all traffic or run Cilium in a trusted network
    25     environment.
    26  
    27  Cilium keeps the networking concept as simple as possible. There are two
    28  networking models to choose from.
    29  
    30  - :ref:`arch_overlay`
    31  - :ref:`arch_direct_routing`
    32  
    33  Regardless of the option chosen, the container itself has no awareness of the
    34  underlying network it runs on; it only contains a default route which points to
    35  the IP address of the cluster node. Given the removal of the routing cache in
    36  the Linux kernel, this reduces the amount of state to keep in the per
    37  connection flow cache (TCP metrics), which allows to terminate millions of
    38  connections in each container.
    39  
    40  .. _arch_overlay:
    41  
    42  Overlay Network Mode
    43  ====================
    44  
    45  When no configuration is provided, Cilium automatically runs in this mode.
    46  
    47  In this mode, all cluster nodes form a mesh of tunnels using the UDP based
    48  encapsulation protocols `VXLAN` or `Geneve`. All container-to-container network
    49  traffic is routed through these tunnels. This mode has several major
    50  advantages:
    51  
    52  - **Simplicity:** The network which connects the cluster nodes does not need to
    53    be made aware of the *cluster prefix*. Cluster nodes can spawn multiple
    54    routing or link-layer domains. The topology of the underlying network is
    55    irrelevant as long as cluster nodes can reach each other using IP/UDP.
    56  
    57  - **Auto-configuration:** When running together with an orchestration system
    58    such as Kubernetes, the list of all nodes in the cluster including their
    59    associated allocation prefix node is made available to each agent
    60    automatically. This means that if Kubernetes is being run with the
    61    ``--allocate-node-cidrs`` option, Cilium can form an overlay network
    62    automatically without any configuration by the user. New nodes joining the
    63    cluster will automatically be incorporated into the mesh.
    64  
    65  - **Identity transfer:** Encapsulation protocols allow for the carrying of
    66    arbitrary metadata along with the network packet. Cilium makes use of this
    67    ability to transfer metadata such as the source security identity and
    68    load balancing state to perform direct-server-return.
    69  
    70  .. _arch_direct_routing:
    71  
    72  Direct / Native Routing Mode
    73  ============================
    74  
    75  .. note:: This is an advanced networking mode which requires the underlying
    76            network to be made aware of container IPs. You can enable this mode
    77            by running Cilium with the option ``--tunnel disabled``.
    78  
    79  In direct routing mode, Cilium will hand all packets which are not addressed
    80  for another local endpoint to the routing subsystem of the Linux kernel. This
    81  means that the packet will be routed as if a local process would have emitted
    82  the packet. As a result, the network connecting the cluster nodes must be aware
    83  that each of the node IP prefixes are reachable by using the node's primary IP
    84  address as an L3 next hop address.
    85  
    86  Cilium automatically enables IP forwarding in Linux when direct mode is
    87  configured, but it is up to the container cluster administrator to ensure that
    88  each routing element in the underlying network has a route that describes each
    89  node IP as the IP next hop for the corresponding node prefix.
    90  
    91  This is typically achieved using two methods:
    92  
    93  - Operation of a routing protocol such as OSPF or BGP via routing daemon such
    94    as Zebra, bird, bgpd. The routing protocols will announce the *node allocation
    95    prefix* via the node's IP to all other nodes.
    96  
    97  - Use of the cloud provider's routing functionality. Refer to the documentation
    98    of your cloud provider for additional details  (e.g,. `AWS VPC Route Tables`_
    99    or `GCE Routes`_). These APIs can be used to associate each node prefix with
   100    the appropriate next hop IP each time a container node is added to the
   101    cluster.  If you are running Kubernetes with the ``--cloud-provider`` in
   102    combination with the ``--allocate-node-cidrs`` option then this is configured
   103    automatically for IPv4 prefixes.
   104  
   105  .. note:: Use of direct routing mode with advanced policy use cases such as
   106            L7 policies is currently beta. Please provide feedback and file a
   107            GitHub issue if you experience any problems.
   108  
   109  
   110  .. _AWS VPC Route Tables: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Route_Tables.html
   111  .. _GCE Routes: https://cloud.google.com/compute/docs/reference/latest/routes
   112  
   113  There are two possible approaches to performing network forwarding for
   114  container-to-container traffic:
   115  
   116  .. _Cluster Mesh:
   117  
   118  Cluster Mesh
   119  ============
   120  
   121  Cluster mesh extends the networking datapath across multiple clusters. It
   122  allows endpoints in all connected clusters to communicate while providing full
   123  policy enforcement. Load-balancing is available via Kubernetes annotations.
   124  
   125  See :ref:`gs_clustermesh` for instructions on how to set up cluster mesh.
   126  
   127  Container Communication with External Hosts
   128  ===========================================
   129  
   130  Container communication with the outside world has two primary modes:
   131  
   132   * Containers exposing API services for consumption by hosts outside of the
   133     container cluster.
   134  
   135   * Containers making outgoing connections.  Examples include connecting to
   136     3rd-party API services like Twilio or Stripe as well as accessing private
   137     APIs that are hosted elsewhere in your enterprise datacenter or cloud
   138     deployment.
   139  
   140  In the :ref:`arch_direct_routing` mode described before, if container IP
   141  addresses are routable outside of the container cluster, communication with
   142  external hosts requires little more than enabling L3 forwarding on each of the
   143  Linux nodes.
   144  
   145  .. _concepts_external_access:
   146  
   147  External Network Connectivity
   148  =============================
   149  
   150  If the destination of a packet lies outside of the cluster, Cilium will
   151  delegate routing to the routing subsystem of the cluster node to use the
   152  default route which is installed on the node of the cluster.
   153  
   154  As the IP addresses used for the **cluster prefix** are typically allocated
   155  from RFC1918 private address blocks and are not publicly routable. Cilium will
   156  automatically masquerade the source IP address of all traffic that is leaving
   157  the cluster. This behavior can be disabled by running ``cilium-agent`` with
   158  the option ``--masquerade=false``.
   159  
   160  Public Endpoint Exposure
   161  ========================
   162  
   163  In direct routing mode, *endpoint* IPs can be publicly routable IPs and no
   164  additional action needs to be taken.
   165  
   166  In overlay mode, *endpoints* that are accepting inbound connections from
   167  cluster external clients likely want to be exposed via some kind of
   168  load-balancing layer. Such a load-balancer will have a public external address
   169  that is not part of the Cilium network.  This can be achieved by having a
   170  load-balancer container that both has a public IP on an externally reachable
   171  network and a private IP on a Cilium network.  However, many container
   172  orchestration frameworks, like Kubernetes, have built in abstractions to handle
   173  this "ingress" load-balancing capability, which achieve the same effect that
   174  Cilium handles forwarding and security only for ''internal'' traffic between
   175  different services.
   176