github.com/cilium/cilium@v1.16.2/Documentation/security/network/proxy/envoy.rst (about)

     1  .. only:: not (epub or latex or html)
     2  
     3      WARNING: You are looking at unreleased Cilium documentation.
     4      Please use the official rendered version released here:
     5      https://docs.cilium.io
     6  
     7  .. _envoy:
     8  
     9  =====
    10  Envoy
    11  =====
    12  
    13  Envoy proxy shipped with Cilium is built with minimal Envoy extensions and custom policy enforcement filters.
    14  Cilium uses this minimal distribution as its host proxy for enforcing HTTP and other L7 policies as specified in network policies
    15  for the cluster. Cilium proxy is distributed within the Cilium images.
    16  
    17  For more information on the version compatibility matrix, see `Cilium Proxy documentation <https://github.com/cilium/proxy#version-compatibility-matrix>`_.
    18  
    19  ***********************
    20  Deployment as DaemonSet
    21  ***********************
    22  
    23  Background
    24  ==========
    25  
    26  When Cilium L7 functionality (Ingress, Gateway API, Network Policies with L7 functionality, L7 Protocol Visibility)
    27  is enabled or installed in a Kubernetes cluster, the Cilium agent starts an Envoy proxy as separate process within
    28  the Cilium agent pod.
    29  
    30  That Envoy proxy instance becomes responsible for proxying all matching L7 requests on that node.
    31  As a result, L7 traffic targeted by policies depends on the availability of the Cilium agent pod.
    32  
    33  Alternatively, it's possible to deploy the Envoy proxy as independently life-cycled DaemonSet called ``cilium-envoy``
    34  instead of running it from within the Cilium Agent Pod.
    35  
    36  The communication between Cilium agent and Envoy proxy takes place via UNIX domain sockets in both deployment modes.
    37  Be that streaming the access logs (e.g. L7 Protocol Visibility), updating the configuration via
    38  `xDS <https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol>`_ or accessing the admin interface.
    39  
    40  Enable and configure Envoy DaemonSet
    41  ====================================
    42  
    43  To enable the dedicated Envoy proxy DaemonSet, install Cilium with the Helm value ``envoy.enabled`` set to ``true``.
    44  
    45  Please see the :ref:`helm_reference` (keys with ``envoy.*``) for detailed information on how to configure the Envoy proxy DaemonSet.
    46  
    47  Potential Benefits
    48  ==================
    49  
    50  - Cilium Agent restarts (e.g. for upgrades) without impacts for the live traffic proxied via Envoy.
    51  - Envoy patch release upgrades without impacts for the Cilium Agent.
    52  - Separate CPU and memory limits for Envoy and Cilium Agent for performance isolation.
    53  - Envoy application log not mixed with the one of the Cilium Agent.
    54  - Dedicated health probes for the Envoy proxy.
    55  - Explicit deployment of Envoy proxy during Cilium installation (compared to on demand in the embedded mode).
    56  
    57  .. admonition:: Video
    58   :class: attention
    59  
    60    If you'd like to see Cilium Envoy in action, check out `eCHO episode 127: Cilium & Envoy <https://www.youtube.com/watch?v=HEwruycGbCU>`__.
    61  
    62  Known Limitations
    63  =================
    64  
    65  - Due to Pod-to-Pod communication with the Cilium Agent via UNIX domain sockets, Envoy DaemonSet isn't supported with SELinux enabled on the host. This is the default for Red Hat OpenShift.
    66  
    67  *************
    68  Go Extensions
    69  *************
    70  
    71  .. note:: This feature is currently in beta phase.
    72  
    73  .. note:: The Go extensions proxylib framework is residing in cilium/proxy repository.
    74  
    75  This is a guide for developers who are interested in writing a Go extension to the 
    76  Envoy proxy as part of Cilium.   
    77  
    78  .. image:: _static/proxylib_logical_flow.png
    79  
    80  As depicted above, this framework allows a developer to write a small amount of Go
    81  code (green box) focused on parsing a new API protocol, and this Go code is able to  
    82  take full advantage of Cilium features including high-performance redirection to/from Envoy, 
    83  rich L7-aware policy language
    84  and access logging, and visibility into encrypted traffic via kTLS (coming soon!).
    85  In sum, you as the developer need only worry about the logic of parsing the protocol, 
    86  and Cilium + Envoy + eBPF do the heavy-lifting.  
    87  
    88  This guide uses simple examples based on a hypothetical "r2d2" protocol 
    89  (see `proxylib/r2d2/r2d2parser.go <https://github.com/cilium/proxy/blob/main/proxylib/r2d2/r2d2parser.go>`_)
    90  that might be used to talk to a simple protocol droid a long time ago in a galaxy far, far away.   
    91  But it also points to other real protocols like Memcached and Cassandra that already exist in the cilium/proxylib 
    92  directory.  
    93  
    94  Step 1: Decide on a Basic Policy Model
    95  ======================================
    96  
    97  To get started, take some time to think about what it means to provide protocol-aware security
    98  in the context of your chosen protocol.   Most protocols follow a common pattern of a client 
    99  who performs an ''operation'' on a ''resource''.   For example: 
   100  
   101  - A standard RESTful HTTP request has a GET/POST/PUT/DELETE methods (operation) and URLs (resource).
   102  - A database protocol like MySQL has SELECT/INSERT/UPDATE/DELETE actions (operation) on a combined database + table name (resource).   
   103  - A queueing protocol like Kafka has produce/consume (operation) on a particular queue (resource).    
   104  
   105  A common policy model is to allow the user to whitelist certain operations on one or more resources.   
   106  In some cases, the resources need to support regexes to avoid explicit matching on variable content 
   107  like ids (e.g., /users/<uuid> would match /users/.*) 
   108  
   109  In our examples, the ''r2d2'' example, we'll use a basic set of operations (READ/WRITE/HALT/RESET). 
   110  The READ and WRITE commands also support a 'filename' resource, while HALT and RESET have no resource.  
   111  
   112  Step 2: Understand Protocol, Encoding, Framing and Types
   113  ========================================================
   114  
   115  Next, get your head wrapped around how a protocol looks terms of the raw data, as this is what you'll be parsing. 
   116  
   117  Try looking for official definitions of the protocol or API.   Official docs will not only help you quickly 
   118  learn how the protocol works, but will also help you by documenting tricky corner cases that wouldn't be 
   119  obvious just from regular use of the protocol.   For example, here are example specs for 
   120  `Redis Protocol <https://redis.io/topics/protocol>`_ , `Cassandra Protocol <https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v4.spec>`_,  
   121  and `AWS SQS <https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Welcome.html>`_ .  
   122  
   123  These specs help you understand protocol aspects like: 
   124  
   125  - **encoding / framing** : how to recognize the beginning/end of individual requests/replies within a TCP stream. 
   126    This typically involves reading a header that encodes the overall request length, though some simple 
   127    protocols use a delimiter like ''\r\n\'' to separate messages.  
   128  
   129  - **request/reply fields** : for most protocols, you will need to parse out fields at various offsets
   130    into the request data in order to extract security-relevant values for visibility + filtering.  In some cases, access
   131    control requires filtering requests from clients to servers, but in some cases, parsing replies will also be required
   132    if reply data is required to understand future requests (e.g., prepared-statements in database protocols).  
   133  
   134  - **message flow** : specs often describe various dependencies between different requests.  Basic protocols tend to 
   135    follow a simple serial request/reply model, but more advanced protocols will support pipelining (i.e., sending 
   136    multiple requests before any replies have been received).  
   137  
   138  - **protocol errors** : when a Cilium proxy denies a request based on policy, it should return a protocol-specific
   139    error to the client (e.g., in HTTP, a proxy should return a ''403 Access Denied'' error).  Looking at the protocol
   140    spec will typically indicate how you should return an equivalent ''Access Denied'' error.    
   141    
   142  Sometimes, the protocol spec does not give you a full sense of the set of commands that can be sent over the protocol.  In that 
   143  case, looking at higher-level user documentation can fill in some of these knowledge gaps.  Here are examples for 
   144  `Redis Commands <https://redis.io/commands>`_ and `Cassandra CQL Commands <https://docs.datastax.com/en/archived/cql/3.1/cql/cql_reference/cqlCommandsTOC.html>`_ .
   145   
   146  Another great trick is to use `Wireshark <https://www.wireshark.org>`_  to capture raw packet data between
   147  a client and server.   For many protocols, the `Wireshark Sample Captures <https://wiki.wireshark.org/SampleCaptures>`_ 
   148  has already saved captures for us.  Otherwise, you can easily use tcpdump to capture a file.  For example, for 
   149  MySQL traffic on port 3306, you could run the following in a container running the MySQL client or server: 
   150  “tcpdump -s 0 port 3306 -w mysql.pcap”.  `More Info <https://linuxexplore.com/2012/06/07/use-tcpdump-to-capture-in-a-pcap-file-wireshark-dump/>`_    
   151  
   152  In our example r2d2 protocol, we'll keep the spec as simple as possible.  It is a text-only based protocol, 
   153  with each request being a line terminated by ''\r\n''.  A request starts with a case-insensitive string 
   154  command ("READ","WRITE","HALT","RESET").   If the command is "READ" or "WRITE", the command must be followed
   155  by a space, and a non-empty filename that contains only non whitespace ASCII characters.  
   156  
   157  Step 3: Search for Existing Parser Code / Libraries
   158  ===================================================
   159  
   160  Look for open source Go library/code that can help.    
   161  Is there existing open source Go code that parse your protocol that you can leverage, 
   162  either directly as library or a motivating example?  For example, the `tidwall/recon library 
   163  <https://github.com/tidwall/redcon>`_ parses Redis in Go, and `Vitess 
   164  <https://github.com/vitessio/vitess>`_ parses MySQL in Go.   `Wireshark dissectors 
   165  <https://github.com/boundary/wireshark/tree/master/epan/dissectors>`_ also has a wealth of 
   166  protocol parsers written in C that can serve as useful guidance.    Note:  finding client-only 
   167  protocol parsing code is typically less helpful than finding a proxy implementation, or a full 
   168  parser library.   This is because the set of requests a client parsers is typically the inverse
   169  set of the requests a Cilium proxy needs to parse, since the proxy mimics the server rather than 
   170  the client.   Still, viewing a Go client can give you a general idea of how to parse the 
   171  general serialization format of the protocol.  
   172  
   173  Step 4: Follow the Cilium Developer Guide
   174  =========================================
   175  
   176  It is easiest to start Cilium development by following the :ref:`dev_guide`
   177  
   178  After cloning cilium/proxy repo:
   179  
   180  .. code-block:: shell-session
   181  
   182      $ cd proxy
   183      $ vagrant up
   184      $ cd proxylib
   185  
   186  While this dev VM is running, you can open additional terminals to the cilium/proxy dev VM
   187  by running ``vagrant ssh`` from within the cilium/proxy source directory.
   188  
   189  
   190  Step 5: Create New Proxy Skeleton 
   191  =================================
   192  
   193  From inside the proxylib directory, copy the rd2d directory and rename the files. 
   194  Replace ''newproto'' with your protocol: 
   195  
   196  .. code-block:: shell-session
   197  
   198      $ mkdir newproto
   199      $ cd newproto
   200      $ cp ../r2d2/r2d2parser.go newproto.go
   201      $ cp ../r2d2/r2d2parser_test.go newproto_test.go
   202  
   203  
   204  Within both newproto.go and newproto_test.go update references to r2d2 with
   205  your protocol name.   Search for both ''r2d2'' and ''R2D2''.  
   206  
   207  Also, edit proxylib.go and add the following import line: 
   208  
   209  :: 
   210  
   211         _ "github.com/cilium/proxy/proxylib/newproto"
   212  
   213  
   214  Step 6: Update OnData Method 
   215  ============================
   216  
   217  Implementing a parser requires you as the developer to implement three primary functions,
   218  shown as blue in the diagram below.   We will cover OnData() in this section, and 
   219  the other functions in section `Step 9:  Add Policy Loading and Matching`_.  
   220  
   221  .. image:: _static/proxylib_key_functions.png
   222  
   223  The beating heart of your parsing is implementing the onData function.  You can think of any 
   224  proxy as have two data streams, one in the request direction (i.e., client to server) and one in 
   225  the reply direction (i.e., server to client).   OnData is called when there is data to process, 
   226  and the value of the boolean 'reply' parameter indicates the direction of the stream for a given 
   227  call to OnData.   The data passed to OnData is a slice of byte slices (i.e., an array of byte arrays).  
   228  
   229  The return values of the OnData function tell the Go framework tell how data in the stream
   230  should be processed, with four primary outcomes:  
   231  
   232  - **PASS x** :  The next x bytes in the data stream passed to OnData represent a request/reply that should be
   233    passed on to the server/client.   The common case here is that this is a request that should be 
   234    allowed by policy, or that no policy is applied.  Note: x bytes may be less than the total amount
   235    of data passed to OnData, in which case the remaining bytes will still be in the data stream when
   236    onData is invoked next.  x bytes may also be more than the data that has been passed to OnData. 
   237    For example, in the case of a protocol where the parser filters only on values in a protocol header, 
   238    it is often possible to make a filtering decision, and then pass (or drop) the size of the full 
   239    request/reply without having the entire request passed to Go.  
   240  
   241  - **MORE x** :  The buffers passed to OnData to do not represent all of the data required to frame and
   242    filter the request/reply.  Instead, the parser 
   243    needs to see at least x additional bytes beyond the current data to make a decision.  
   244    In some cases, the full request must be read to understand framing and filtering, but in others a decision 
   245    can be made simply by reading a protocol header.   When parsing data, be defensive, and recognize that it is technically possible that 
   246    data arrives one byte at a time. Two common scenarios exist here:
   247  
   248    - **Text-based Protocols** : For text-based protocols
   249      that use a delimiter like "\r\n", it is common to simply check if the delimiter exists, and return 
   250      MORE 1 if it does not, as technically one more character could result in the delimiter being present.
   251      See the sample r2d2 parser as a basic example of this.    
   252  
   253    - **Binary-based protocols** : Many binary protocols  
   254      have a fixed header length, which containers a field that then indicates the remaining length
   255      of the request.  In the binary case, first check to make sure a full header is received.  Typically
   256      the header will indicate both the full request length (i.e., framing), as well as the request type, 
   257      which indicates how much of the full request must be read in order to perform filtering (in many cases, this is less than 
   258      the full request).  A binary parser will typically return MORE if the data passed to OnData is less than 
   259      the header length.   After reading a full header, the simple approach is for the parser to return MORE to wait 
   260      for the full request to be received and parsed  (see the existing CassandraParser as an example).
   261      However, as an optimization, the parser can attempt to only 
   262      request the minimum number of bytes required beyond the header to make a policy decision, and then PASS or DROP
   263      the remaining bytes without requiring them to be passed to the Go parser. 
   264  
   265  - **DROP x** :  Remove the first x bytes from the data stream passed to OnData, as they represent a request/reply
   266    that should not be forwarded to the client or server based on policy.  Don't worry about making onData return 
   267    a drop right away, as we'll return to DROP in a later step below.  
   268  
   269  - **ERROR y** : The connection contains data that does not match the protocol spec, and prevents you from further 
   270    parsing the data stream.   The framework will terminate the connection.   An example would be a request length
   271    that falls outside the min/max specified by the protocol spec, or values for a field that fall outside the values
   272    indicated by the spec (e.g., wrong versions, unknown commands).  If you are still able to properly frame the 
   273    requests, you can also choose to simply drop the request and return a protocol error (e.g., similar to an 
   274    ''HTTP 400 Bad Request'' error.   But in all cases, you should write your parser defensively, such that you 
   275    never forward a request that you do not understand, as such a request could become an avenue for subverting 
   276    the intended security visibility and filtering policies.  See proxylib/types.h for the set of valid error codes.   
   277  
   278  See proxylib/proxylib/parserfactory.go for the official OnData interface definition.   
   279  
   280  Keep it simple, and work iteratively.  Start out just getting the framing right.  Can you write a parser that just 
   281  prints out the length and contents of a request, and then PASS each request with no policy enforcement?   
   282  
   283  One simple trick is to comment out the r2d2 parsing logic in OnData, but leave it in the file as a reference, as your protocol will likely
   284  require similar code as we add more functionality below.  
   285  
   286  Step 7: Use Unit Testing To Drive Development
   287  =============================================
   288  
   289  Use unit tests to drive your development.    Its tempting to want to first test your parser by firing up a
   290  client and server and developing on the fly.   But in our experience you’ll iterate faster by using the 
   291  great unit test framework created along with the Go proxy framework.   This framework lets you pass
   292  in an example set of requests as byte arrays to a CheckOnDataOK method, which are passed to the parser's OnData method.
   293  CheckOnDataOK takes a set of expected return values, and compares them to the actual return values from OnData 
   294  processing the byte arrays.  
   295  
   296  Take some time to look at the unit tests for the r2d2 parser, and then for more complex parsers like Cassandra
   297  and Memcached.   For simple text-based protocols, you can simply write ASCII strings to represent protocol messages, 
   298  and convert them to []byte arrays and pass them to CheckOnDataOK.   For binary protocols, one can either create 
   299  byte arrays directly, or use a mechanism to convert a hex string to byte[] array using a helper function like 
   300  hexData in cassandra/cassandraparser_test.go
   301  
   302  A great way to get the exact data to pass in is to copy the data from the Wireshark captures mentioned
   303  above in Step #2.   You can see the full application layer data streams in Wireshark by right-clicking
   304  on a packet and selecting “Follow As… TCP Stream”.  If the protocol is text-based, you can copy the data 
   305  as ASCII (see r2d2/r2d2parser_test.go as an example of this).   For binary data, it can be easier to instead 
   306  select “raw” in the drop-down, and use a basic utility to convert from ascii strings to binary raw data (see 
   307  cassandra/cassandraparser_test.go for an example of this). 
   308  
   309  To run the unit tests, go to proxylib/newproto and run: 
   310  
   311  .. code-block:: shell-session
   312  
   313    $ go test
   314  
   315  This will build the latest version of your parser and unit test files and run the unit tests.   
   316  
   317  Step 8: Add More Advanced Parsing
   318  =================================
   319  
   320  Thinking back to step #1, what are the critical fields to parse out of the request in order to 
   321  understand the “operation” and “resource” of each request.  Can you print those out for each request?
   322  
   323  Use the unit test framework to pass in increasingly complex requests, and confirm that the parser prints out the right values, and that the 
   324  unit tests are properly slicing the datastream into requests and parsing out the required fields. 
   325  
   326  A couple scenarios to make sure your parser handles properly via unit tests: 
   327  
   328  - data chunks that are less than a full request (return MORE) 
   329  - requests that are spread across multiple data chunks. (return MORE ,then PASS) 
   330  - multiple requests that are bundled into a single data chunk (return PASS, then another PASS)
   331  - rejection of malformed requests (return ERROR). 
   332  
   333  For certain advanced cases, it is required for a parser to store state across requests. 
   334  In this case, data can be stored using data structures that
   335  are included as part of the main parser struct.  See CassandraParser in cassandra/cassandraparser.go as an example 
   336  of how the parser uses a string to store the current 'keyspace' in use, and uses Go maps to keep 
   337  state required for handling prepared queries.   
   338  
   339  Step 9:  Add Policy Loading and Matching
   340  ========================================
   341  
   342  Once you have the parsing of most protocol messages ironed out, its time to start enforcing policy. 
   343  
   344  First, create a Go object that will represent a single rule in the policy language. For example,
   345  this is the rule for the r2d2 protocol, which performs exact match on the command string, and a regex
   346  on the filename:  
   347  
   348  .. code-block:: go
   349  
   350      type R2d2Rule struct {
   351         cmdExact   string
   352         fileRegexCompiled *regexp.Regexp
   353      }
   354  
   355  There are two key methods to update: 
   356  
   357  - Matches :   This function implements the basic logic of comparing data from a single request 
   358    against a single policy rule, and return true if that rule matches (i.e., allows) that request.  
   359  
   360  - <NewProto>RuleParser : Reads key value pairs from policy, validates those entries, and stores
   361    them as a <NewProto>Rule object.   
   362  
   363  See r2d2/r2d2parser.go for examples of both functions for the r2d2 protocol.  
   364  
   365  You'll also need to update OnData to call p.connection.Matches(), and if this function return false, 
   366  return DROP for a request.  Note: despite the similar names between the Matches() function you 
   367  create in your newprotoparser.go and p.connection.Matches(), do not confuse
   368  the two.  Your OnData function should always call p.connection.Matches() rather than invoking your
   369  own Matches() directly, as p.connection.Matches()
   370  calls the parser's Matches() function only on the subset of L7 rules that apply for the given 
   371  Cilium source identity for this particular connection.  
   372  
   373  Once you add the logic to call Matches() and return DROP in OnData, you will need to update
   374  unit tests to have policies that allow the traffic you expect to be passed.   The following 
   375  is an example of how r2d2/r2d2parser_test.go adds an allow-all policy for a given test: 
   376  
   377  .. code-block:: go
   378  
   379      s.ins.CheckInsertPolicyText(c, "1", []string{`
   380          name: "cp1"
   381          policy: 2
   382          ingress_per_port_policies: <
   383            port: 80
   384            rules: <
   385              l7_proto: "r2d2"
   386            >
   387          >
   388          `})
   389  
   390  The following is an example of a policy that would allow READ commands with a file 
   391  regex of ".*": 
   392  
   393  .. code-block:: go
   394  
   395      s.ins.CheckInsertPolicyText(c, "1", []string{`
   396          name: "cp2"
   397          policy: 2
   398          ingress_per_port_policies: <
   399            port: 80
   400            rules: <
   401              l7_proto: "r2d2"
   402              l7_rules: <
   403              rule: <
   404                key: "cmd"
   405                value: "READ"
   406              >
   407              rule: <
   408                key: "file"
   409                value: ".*"
   410              >
   411                >
   412              >
   413            >
   414          >
   415          `})
   416  
   417  
   418  Step 10: Inject Error Response
   419  ==============================
   420  
   421  Simply dropping the request from the request data stream prevents the request from reaching the server, but it would 
   422  leave the client hanging, waiting for a response that would never come since the server did not see the request. 
   423  
   424  Instead, the proxy should return an application-layer reply indicating that access was denied, similar to how
   425  an HTTP proxy would return a ''403 Access Denied'' error.  Look back at the protocol spec discussed in Step 2 to 
   426  understand what an access denied message looks like for this protocol, and use the p.connection.Inject() method 
   427  to send this error reply back to the client.   See r2d2/r2d2parser.go for an example. 
   428  
   429  .. code-block:: go
   430  
   431      p.connection.Inject(true, []byte("ERROR\r\n"))
   432  
   433  Note:  p.connection.Inject() will inject the data it is passed into the reply datastream.  In order for the client 
   434  to parse this data correctly, it must be injected at a proper framing boundary (i.e., in between other reply messages
   435  that may be in the reply data stream).  If the client is following a basic serial request/reply model per connection, this is 
   436  essentially guaranteed as at the time of a request that is denied, there are no other replies potentially in the 
   437  reply datastream.   But if the protocol supports pipelining (i.e., multiple requests in flight) replies must be properly 
   438  framed and PASSed on a per request basis, and the timing of the call to p.connection.Inject() must be controlled
   439  such that the client will properly match the Error response with the correct request.   See the Memcached parser
   440  as an example of how to accomplish this.  
   441  
   442  Step 11: Add Access Logging
   443  ===========================
   444  
   445  Cilium also has the notion of an ''Access Log'', which records each request handled by the proxy 
   446  and indicates whether the request was allowed or denied.  
   447  
   448  A call to ``p.connection.Log()`` implements access logging. See the OnData function in r2d2/r2d2parser.go 
   449  as an example: 
   450  
   451  .. code-block:: go
   452  
   453        p.connection.Log(access_log_entry_type,
   454          &cilium.LogEntry_GenericL7{
   455              &cilium.L7LogEntry{
   456                  Proto: "r2d2",
   457                  Fields: map[string]string{
   458                      "cmd":  reqData.cmd,
   459                      "file": reqData.file,
   460                  },
   461              },
   462        })  
   463  
   464  Step 12: Manual Testing
   465  =======================
   466  
   467  Find the standard docker container for running the protocol server.  Often the same image also has a CLI client that you can use as a client. 
   468  
   469  Start both a server and client container running in the cilium dev VM, and attach them to the already created “cilium-net”.  For example, with Cassandra, we run:
   470  
   471  .. code-block:: shell-session
   472  
   473      docker run --name cass-server -l id=cass-server -d --net cilium-net cassandra
   474  
   475      docker run --name cass-client -l id=cass-client -d --net cilium-net cassandra sh -c 'sleep 3000' 
   476   
   477  
   478  Note that we run both containers with labels that will make it easy to refer to these containers in a cilium 
   479  network policy.   Note that we have the client container run the sleep command, as we will use 'docker exec' to 
   480  access the client CLI.  
   481  
   482  Use ``cilium-dbg endpoint list`` to identify the IP address of the protocol server.  
   483  
   484  .. code-block:: shell-session
   485  
   486    $ cilium-dbg endpoint list
   487    ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
   488               ENFORCEMENT        ENFORCEMENT                                                                                     
   489    2987       Disabled           Disabled          31423      container:id=cass-server      f00d::a0b:0:0:bab    10.11.51.247    ready   
   490    27333      Disabled           Disabled          4          reserved:health               f00d::a0b:0:0:6ac5   10.11.92.46     ready   
   491    50923      Disabled           Disabled          18253      container:id=cass-client      f00d::a0b:0:0:c6eb   10.11.175.191   ready 
   492  
   493  One can then invoke the client CLI using that server IP address (10.11.51.247 in the above example):
   494  
   495  .. code-block:: shell-session
   496  
   497      docker exec -it cass-client sh -c 'cqlsh 10.11.51.247 -e "select * from system.local"'
   498  
   499  Note that in the above example, ingress policy is not enforced for the Cassandra server endpoint, so no data will flow through the
   500  Cassandra parser.  A simple ''allow all'' L7 Cassandra policy can be used to send all data to the Cassandra server through the 
   501  Go Cassandra parser.  This policy has a single empty rule, which matches all requests.  An allow all policy looks like: 
   502  
   503  .. code-block:: json
   504  
   505    [ { 
   506      "endpointSelector": {"matchLabels":{"id":"cass-server"}}, 
   507      "ingress": [ {
   508  	  "toPorts": [{
   509  		  "ports": [{"port": "9042", "protocol": "TCP"}],
   510              		"rules": {
   511                  		"l7proto": "cassandra",
   512                  		"l7": [{}]
   513              		}
   514  		}]
   515  	  } ] 
   516    }]
   517  
   518  
   519  A policy can be imported into cilium using ``cilium policy import``, after which another call to ``cilium-dbg endpoint list``
   520  confirms that ingress policy is now in place on the server.  If the above policy was saved to a file cass-allow-all.json, 
   521  one would run: 
   522  
   523  .. code-block:: shell-session
   524  
   525      $ cilium-dbg policy import cass-allow-all.json
   526      Revision: 1
   527      $ cilium-dbg endpoint list
   528      ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
   529                 ENFORCEMENT        ENFORCEMENT                                                                                     
   530      2987       Enabled            Disabled          31423      container:id=cass-server      f00d::a0b:0:0:bab    10.11.51.247    ready   
   531      27333      Disabled           Disabled          4          reserved:health               f00d::a0b:0:0:6ac5   10.11.92.46     ready   
   532      50923      Disabled           Disabled          18253      container:id=cass-client      f00d::a0b:0:0:c6eb   10.11.175.191   ready 
   533  
   534  Note that policy is now showing as ''Enabled'' for the Cassandra server on ingress. 
   535  
   536  To remove this or any other policy, run: 
   537  
   538  .. code-block:: shell-session
   539  
   540      $ cilium-dbg policy delete --all 
   541  
   542  To install a new policy, first delete, and then run ``cilium policy import`` again.  For example, the following policy would allow
   543  select statements on a specific set of tables to this Cassandra server, but deny all other queries. 
   544  
   545  .. code-block:: json
   546  
   547    [ {
   548      "endpointSelector": {"matchLabels":{"id":"cass-server"}},
   549      "ingress": [ {
   550            "toPorts": [{
   551                    "ports": [{"port": "9042", "protocol": "TCP"}],
   552                          "rules": {
   553                                  "l7proto": "cassandra",
   554                                  "l7": [
   555                                         { "query_action" : "select", "query_table": "^system.*"},
   556                                         { "query_action" : "select", "query_table" : "^posts_db.posts$"}
   557  
   558                                  ]}
   559                          }]
   560           }]
   561    } ]
   562  
   563  When performing manual testing, remember that each time you change your Go proxy code, you must
   564  re-run ``make`` and ``sudo make install`` and then restart the cilium-agent process.  If the only changes
   565  you have made since last compiling cilium are in your cilium/proxylib directory, you can safely 
   566  just run ``make`` and ``sudo make install``  in that directory, which saves time.  
   567  For example: 
   568  
   569  .. code-block:: shell-session
   570  
   571    $ cd proxylib  // only safe is this is the only directory that has changed
   572    $ make  
   573      <snip> 
   574    $ sudo make install 
   575      <snip> 
   576  
   577  If you rebase or other files change, you need to run both commands from the top level directory.  
   578  
   579  Cilium agent default to running as a service in the development VM.  However, the default options do not include 
   580  the ``--debug-verbose=flow`` flag, which is critical to getting visibility in troubleshooting Go proxy frameworks. 
   581  So it is easiest to stop the cilium service and run the cilium-agent directly as a command in a terminal window, 
   582  and adding the ``--debug-verbose=flow`` flag. 
   583  
   584  .. code-block:: shell-session
   585  
   586    $ sudo service cilium stop 
   587    
   588    $ sudo /usr/bin/cilium-agent --debug --ipv4-range 10.11.0.0/16 --kvstore-opt consul.address=192.168.60.11:8500 --kvstore consul -t vxlan --fixed-identity-mapping=128=kv-store --fixed-identity-mapping=129=kube-dns --debug-verbose=flow
   589  
   590  
   591  Step 13: Add Runtime Tests
   592  ==========================
   593  
   594  Before submitting this change to the Cilium community, it is recommended that you add runtime tests that will run as
   595  part of Cilium's continuous integration testing.   Usually these runtime test can be based on the same container 
   596  images and test commands you used for manual testing.   
   597  
   598  The best approach for adding runtime tests is typically to start out by copying-and-pasting an existing L7 protocol runtime
   599  test and then updating it to run the container images and CLI commands specific to the new protocol.   
   600  See cilium/test/runtime/cassandra.go as an example that matches the use of Cassandra described above in the manual testing
   601  section.   Note that the json policy files used by the runtime tests are stored in cilium/test/runtime/manifests, and 
   602  the Cassandra example policies in those directories are easy to use as a based for similar policies you may create for your
   603  new protocol.  
   604  
   605  Step 14: Review Spec for Corner Cases
   606  =====================================
   607  
   608  Many protocols have advanced features or corner cases that will not manifest themselves as part of basic testing.   
   609  Once you have written a first rev of the parser, it is a good idea to go back and review the protocol's spec or list of 
   610  commands to see what if any aspects may fall outside the scope of your initial parser.    
   611  For example, corner cases like the handling of empty or nil lists may not show up in your testing, but may cause your
   612  parser to fail.   Add more unit tests to cover these corner cases.  
   613  It is OK for the first rev of your parser not to handle all types of requests, or to have a simplified policy structure 
   614  in terms of which fields can be matched.   However, it is 
   615  important to know what aspects of the protocol you are not parsing, and ensure that it does not lead to any security concerns. 
   616  For example, failing to parse prepared statements in a database protocol and instead just passing PREPARE and EXECUTE
   617  commands through would lead to gaping security whole that would render your other filtering meaningless in the face of
   618  a sophisticated attacker.   
   619  
   620  Step 15: Write Docs or Getting Started Guide (optional) 
   621  =======================================================
   622  
   623  At a minimum, the policy examples included as part of the runtime tests serve
   624  as basic documentation of the policy and its expected behavior.  But we also 
   625  encourage adding more user friendly examples and documentation, for example, 
   626  Getting Started Guides.  ``cilium/Documentation/gettingstarted/cassandra.rst`` is
   627  a good example to follow.   Also be sure to update ``Documentation/gettingstarted/index.rst``
   628  with a link to this new getting started guide. 
   629  
   630  With that, you are ready to post this change for feedback from the Cilium community. Congrats!