github.com/cilium/cilium@v1.16.2/Documentation/internals/hubble.rst (about)

     1  .. only:: not (epub or latex or html)
     2  
     3      WARNING: You are looking at unreleased Cilium documentation.
     4      Please use the official rendered version released here:
     5      https://docs.cilium.io
     6  
     7  .. _hubble_internals:
     8  
     9  ****************
    10  Hubble internals
    11  ****************
    12  
    13  .. note:: This documentation section is targeted at developers who are
    14            interested in contributing to Hubble. For this purpose, it describes
    15            Hubble internals.
    16  
    17  .. note:: This documentation covers the Hubble server (sometimes referred as
    18            "Hubble embedded") and Hubble Relay components but does not cover the
    19            Hubble UI and CLI.
    20  
    21  Hubble builds on top of Cilium and eBPF to enable deep visibility into the
    22  communication and behavior of services as well as the networking infrastructure
    23  in a completely transparent manner. One of the design goals of Hubble is to
    24  achieve all of this at large scale.
    25  
    26  Hubble's server component is embedded into the Cilium agent in order to achieve
    27  high performance with low-overhead. The gRPC services offered by Hubble server
    28  may be consumed locally via a Unix domain socket or, more typically, through
    29  Hubble Relay. Hubble Relay is a standalone component which is aware of all
    30  Hubble instances and offers full cluster visibility by connecting to their
    31  respective gRPC APIs. This capability is usually referred to as multi-node.
    32  Hubble Relay's main goal is to offer a rich API that can be safely exposed and
    33  consumed by the Hubble UI and CLI.
    34  
    35  Hubble Architecture
    36  ===================
    37  
    38  Hubble exposes gRPC services from the Cilium process that allows clients to
    39  receive flows and other type of data.
    40  
    41  Hubble server
    42  -------------
    43  
    44  The Hubble server component implements two gRPC services. The **Observer
    45  service** which may optionally be exposed via a TCP socket in addition to a
    46  local Unix domain socket and the **Peer service**, which is served on both
    47  as well as being exposed as a Kubernetes Service when enabled via TCP.
    48  
    49  The Observer service
    50  ^^^^^^^^^^^^^^^^^^^^
    51  
    52  The Observer service is the principal service. It provides four RPC endpoints:
    53  ``GetFlows``, ``GetNodes``, ``GetNamespaces``  and ``ServerStatus``.
    54  
    55  * ``GetNodes`` returns a list of metrics and other information related to each Hubble instance
    56  * ``ServerStatus`` returns a summary the information in ``GetNodes``
    57  * ``GetNamespaces`` returns a list of namespaces that had network flows within the last one hour
    58  * ``GetFlows`` returns a stream of flow related events
    59  
    60  Using ``GetFlows``, callers get a stream of payloads. Request parameters allow
    61  callers to specify filters in the form of allow lists and deny lists to allow
    62  for fine-grained filtering of data.
    63  
    64  In order to answer ``GetFlows`` requests, Hubble stores monitoring events from
    65  Cilium's event monitor into a user-space ring buffer structure. Monitoring
    66  events are obtained by registering a new listener on Cilium monitor. The
    67  ring buffer is capable of storing a configurable amount of events in memory.
    68  Events are continuously consumed, overriding older ones once the ring buffer is
    69  full.
    70  
    71  Additionally, the Observer service also provides the ``GetAgentEvents`` and
    72  ``GetDebugEvents`` RPC endpoints to expose data about the Cilium agent events
    73  and Cilium datapath debug events, respectively. Both are similar to ``GetFlows``
    74  except they do not implement filtering capabilities.
    75  
    76  .. image:: ./../images/hubble_getflows.png
    77  
    78  For efficiency, the internal buffer length is a bit mask of ones + 1. The most
    79  significant bit of this bit mask is the same position of the most significant
    80  bit position of 'n'. In other terms, the internal buffer size is always a power
    81  of 2 with 1 slot reserved for the writer. In effect, from a user perspective,
    82  the ring buffer capacity is one less than a power of 2. As the ring buffer is a
    83  hot code path, it has been designed to not employ any locking mechanisms and
    84  uses atomic operations instead. While this approach has performance benefits,
    85  it also has the downsides of being a complex component.
    86  
    87  Due to its complex nature, the ring buffer is typically accessed via a ring
    88  reader that abstracts the complexity of this data structure for reading. The
    89  ring reader allows reading one event at the time with 'previous' and 'next'
    90  methods but also implements a follow mode where events are continuously read as
    91  they are written to the ring buffer.
    92  
    93  The Peer service
    94  ^^^^^^^^^^^^^^^^
    95  
    96  The Peer service sends information about Hubble peers in the cluster in a
    97  stream. When the ``Notify`` method is called, it reports information about all
    98  the peers in the cluster and subsequently sends information about peers that
    99  are updated, added, or removed from the cluster. Thus, it allows the caller to
   100  keep track of all Hubble instances and query their respective gRPC services.
   101  
   102  This service is exposed as a Kubernetes Service and is primarily used by Hubble
   103  Relay in order to have a cluster-wide view of all Hubble instances.
   104  
   105  The Peer service obtains peer change notifications by subscribing to Cilium's
   106  node manager. To this end, it internally defines a handler that implements
   107  Cilium's datapath node handler interface.
   108  
   109  .. _hubble_relay:
   110  
   111  Hubble Relay
   112  ------------
   113  
   114  Hubble Relay is the Hubble component that brings multi-node support. It
   115  leverages the Peer service to obtain information about Hubble instances and
   116  consume their gRPC API in order to provide a more rich API that covers events
   117  from across the entire cluster (or even multiple clusters in a ClusterMesh
   118  scenario).
   119  
   120  Hubble Relay was first introduced as a technology preview with the release of
   121  Cilium v1.8 and was declared stable with the release of Cilium v1.9.
   122  
   123  Hubble Relay implements the Observer service for multi-node. To that end, it
   124  maintains a persistent connection with every Hubble peer in a cluster with a
   125  peer manager. This component provides callers with the list of peers. Callers
   126  may report when a peer is unreachable, in which case the peer manager will
   127  attempt to reconnect.
   128  
   129  As Hubble Relay connects to every node in a cluster, the Hubble server
   130  instances must make their API available (by default on port 4244). By default,
   131  Hubble server endpoints are secured using mutual TLS (mTLS) when exposed on a
   132  TCP port in order to limit access to Hubble Relay only.