github.com/imran-kn/cilium-fork@v1.6.9/Documentation/envoy/extensions.rst (about)

     1  .. only:: not (epub or latex or html)
     2  
     3      WARNING: You are looking at unreleased Cilium documentation.
     4      Please use the official rendered version released here:
     5      http://docs.cilium.io
     6  
     7  *******************
     8  Envoy Go Extensions
     9  *******************
    10  
    11  .. note:: This feature is currently in beta phase.
    12  
    13  This is a guide for developers who are interested in writing a Go extension to the 
    14  Envoy proxy as part of Cilium.   
    15  
    16  .. image:: images/proxylib_logical_flow.png
    17  
    18  As depicted above, this framework allows a developer to write a small amount of Go
    19  code (green box) focused on parsing a new API protocol, and this Go code is able to  
    20  take full advantage of Cilium features including high-performance redirection to/from Envoy, 
    21  rich L7-aware policy language
    22  and access logging, and visibility into encrypted traffic traffic via kTLS (coming soon!).  
    23  In sum, you as the developer need only worry about the logic of parsing the protocol, 
    24  and Cilium + Envoy + BPF do the heavy-lifting.  
    25  
    26  This guide uses simple examples based on a hypothetical "r2d2" protocol 
    27  (see `proxylib/r2d2/r2d2parser.go <https://github.com/cilium/cilium/blob/master/proxylib/r2d2/r2d2parser.go>`_) 
    28  that might be used to talk to a simple protocol droid a long time ago in a galaxy far, far away.   
    29  But it also points to other real protocols like Memcached and Cassandra that already exist in the cilium/proxylib 
    30  directory.  
    31  
    32  Step 1: Decide on a Basic Policy Model
    33  ======================================
    34  
    35  To get started, take some time to think about what it means to provide protocol-aware security
    36  in the context of your chosen protocol.   Most protocols follow a common pattern of a client 
    37  who performs an ''operation'' on a ''resource''.   For example: 
    38  
    39  - A standard RESTful HTTP request has a GET/POST/PUT/DELETE methods (operation) and URLs (resource).
    40  - A database protocol like MySQL has SELECT/INSERT/UPDATE/DELETE actions (operation) on a combined database + table name (resource).   
    41  - A queueing protocol like Kafka has produce/consume (operation) on a particular queue (resource).    
    42  
    43  A common policy model is to allow the user to whitelist certain operations on one or more resources.   
    44  In some cases, the resources need to support regexes to avoid explicit matching on variable content 
    45  like ids (e.g., /users/<uuid> would match /users/.*) 
    46  
    47  In our examples, the ''r2d2'' example, we'll use a basic set of operations (READ/WRITE/HALT/RESET). 
    48  The READ and WRITE commands also support a 'filename' resource, while HALT and RESET have no resource.  
    49  
    50  Step 2: Understand Protocol, Encoding, Framing and Types
    51  ========================================================
    52  
    53  Next, get your head wrapped around how a protocol looks terms of the raw data, as this is what you'll be parsing. 
    54  
    55  Try looking for official definitions of the protocol or API.   Official docs will not only help you quickly 
    56  learn how the protocol works, but will also help you by documenting tricky corner cases that wouldn't be 
    57  obvious just from regular use of the protocol.   For example, here are example specs for 
    58  `Redis Protocol <https://redis.io/topics/protocol>`_ , `Cassandra Protocol <https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v4.spec>`_,  
    59  and `AWS SQS <https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Welcome.html>`_ .  
    60  
    61  These specs help you understand protocol aspects like: 
    62  
    63  - **encoding / framing** : how to recognize the beginning/end of individual requests/replies within a TCP stream. 
    64    This typically involves reading a header that encodes the overall request length, though some simple 
    65    protocols use a delimiter like ''\r\n\'' to separate messages.  
    66  
    67  - **request/reply fields** : for most protocols, you will need to parse out fields at various offsets
    68    into the request data in order to extract security-relevant values for visibility + filtering.  In some cases, access
    69    control requires filtering requests from clients to servers, but in some cases, parsing replies will also be required
    70    if reply data is required to understand future requests (e.g., prepared-statements in database protocols).  
    71  
    72  - **message flow** : specs often describe various dependencies between different requests.  Basic protocols tend to 
    73    follow a simple serial request/reply model, but more advanced protocols will support pipelining (i.e., sending 
    74    multiple requests before any replies have been received).  
    75  
    76  - **protocol errors** : when a Cilium proxy denies a request based on policy, it should return a protocol-specific
    77    error to the client (e.g., in HTTP, a proxy should return a ''403 Access Denied'' error).  Looking at the protocol
    78    spec will typically indicate how you should return an equivalent ''Access Denied'' error.    
    79    
    80  Sometimes, the protocol spec does not give you a full sense of the set of commands that can be sent over the protocol.  In that 
    81  case, looking at higher-level user documentation can fill in some of these knowledge gaps.  Here are examples for 
    82  `Redis Commands <https://redis.io/commands>`_ and `Cassandra CQL Commands  <https://docs.datastax.com/en/cql/3.1/cql/cql_reference/cqlCommandsTOC.html>`_ .  
    83   
    84  Another great trick is to use `Wireshark <https://www.wireshark.org>`_  to capture raw packet data between
    85  a client and server.   For many protocols, the `Wireshark Sample Captures <https://wiki.wireshark.org/SampleCaptures>`_ 
    86  has already saved captures for us.  Otherwise, you can easily use tcpdump to capture a file.  For example, for 
    87  MySQL traffic on port 3306, you could run the following in a container running the MySQL client or server: 
    88  “tcpdump -s 0 port 3306 -w mysql.pcap”.  `More Info <https://linuxexplore.com/2012/06/07/use-tcpdump-to-capture-in-a-pcap-file-wireshark-dump/>`_    
    89  
    90  In our example r2d2 protocol, we'll keep the spec as simple as possible.  It is a text-only based protocol, 
    91  with each request being a line terminated by ''\r\n''.  A request starts with a case-insensitive string 
    92  command ("READ","WRITE","HALT","RESET").   If the command is "READ" or "WRITE", the command must be followed
    93  by a space, and a non-empty filename that contains only non whitespace ASCII characters.  
    94  
    95  Step 3: Search for Existing Parser Code / Libraries
    96  ===================================================
    97  
    98  Look for open source Go library/code that can help.    
    99  Is there existing open source Go code that parse your protocol that you can leverage, 
   100  either directly as library or a motivating example?  For example, the `tidwall/recon library 
   101  <https://github.com/tidwall/redcon>`_ parses Redis in Go, and `Vitess 
   102  <https://github.com/vitessio/vitess>`_ parses MySQL in Go.   `Wireshark dissectors 
   103  <https://github.com/boundary/wireshark/tree/master/epan/dissectors>`_ also has a wealth of 
   104  protocol parsers written in C that can serve as useful guidance.    Note:  finding client-only 
   105  protocol parsing code is typically less helpful than finding a proxy implementation, or a full 
   106  parser library.   This is because the set of requests a client parsers is typically the inverse
   107  set of the requests a Cilium proxy needs to parse, since the proxy mimics the server rather than 
   108  the client.   Still, viewing a Go client can give you a general idea of how to parse the 
   109  general serialization format of the protocol.  
   110  
   111  Step 4: Follow the Cilium Developer Guide
   112  =========================================
   113  
   114  It is easiest to start Cilium development by following the :ref:`dev_guide`
   115  
   116  After cloning Cilium: 
   117  
   118  ::
   119  
   120      $ cd cilium 
   121      $ contrib/vagrant/start.sh 
   122      $ cd proxylib
   123  
   124  While this dev VM is running, you can open additional terminals to the Cilium dev VM
   125  by running ''vagrant ssh'' from within the cilium source directory.  
   126  
   127  
   128  Step 5: Create New Proxy Skeleton 
   129  =================================
   130  
   131  From inside the proxylib directory, copy the rd2d directory and rename the files. 
   132  Replace ''newproto'' with your protocol: 
   133  
   134  :: 
   135  
   136      $ mkdir newproto
   137      $ cd newproto
   138      $ cp ../r2d2/r2d2parser.go newproto.go
   139      $ cp ../r2d2/r2d2parser_test.go newproto_test.go
   140  
   141  
   142  Within both newproto.go and newproto_test.go update references to r2d2 with
   143  your protocol name.   Search for both ''r2d2'' and ''R2D2''.  
   144  
   145  Also, edit proxylib.go and add the following import line: 
   146  
   147  :: 
   148  
   149         _ "github.com/cilium/cilium/proxylib/newproto"
   150  
   151  
   152  Step 6: Update OnData Method 
   153  ============================
   154  
   155  Implementing a parser requires you as the developer to implement three primary functions,
   156  shown as blue in the diagram below.   We will cover OnData() in this section, and 
   157  the other functions in section `Step 9:  Add Policy Loading and Matching`_.  
   158  
   159  .. image:: images/proxylib_key_functions.png
   160  
   161  The beating heart of your parsing is implementing the onData function.  You can think of any 
   162  proxy as have two data streams, one in the request direction (i.e., client to server) and one in 
   163  the reply direction (i.e., server to client).   OnData is called when there is data to process, 
   164  and the value of the boolean 'reply' parameter indicates the direction of the stream for a given 
   165  call to OnData.   The data passed to OnData is a slice of byte slices (i.e., an array of byte arrays).  
   166  
   167  The return values of the OnData function tell the Go framework tell how data in the stream
   168  should be processed, with four primary outcomes:  
   169  
   170  - **PASS x** :  The next x bytes in the data stream passed to OnData represent a request/reply that should be
   171    passed on to the server/client.   The common case here is that this is a request that should be 
   172    allowed by policy, or that no policy is applied.  Note: x bytes may be less than the total amount
   173    of data passed to OnData, in which case the remaining bytes will still be in the data stream when
   174    onData is invoked next.  x bytes may also be more than the data that has been passed to OnData. 
   175    For example, in the case of a protocol where the parser filters only on values in a protocol header, 
   176    it is often possible to make a filtering decision, and then pass (or drop) the size of the full 
   177    request/reply without having the entire request passed to Go.  
   178  
   179  - **MORE x** :  The buffers passed to OnData to do not represent all of the data required to frame and
   180    filter the request/reply.  Instead, the parser 
   181    needs to see at least x additional bytes beyond the current data to make a decision.  
   182    In some cases, the full request must be read to understand framing and filtering, but in others a decision 
   183    can be made simply by reading a protocol header.   When parsing data, be defensive, and recognize that it is technically possible that 
   184    data arrives one byte byte at a time.   Two common scenarios exist here:  
   185  
   186    - **Text-based Protocols** : For text-based protocols
   187      that use a delimiter like "\r\n", it is common to simply check if the delimiter exists, and return 
   188      MORE 1 if it does not, as technically one more character could result in the delimiter being present.
   189      See the sample r2d2 parser as a basic example of this.    
   190  
   191    - **Binary-based protocols** : Many binary protocols  
   192      have a fixed header length, which containers a field that then indicates the remaining length
   193      of the request.  In the binary case, first check to make sure a full header is received.  Typically
   194      the header will indicate both the full request length (i.e., framing), as well as the request type, 
   195      which indicates how much of the full request must be read in order to perform filtering (in many cases, this is less than 
   196      the full request).  A binary parser will typically return MORE if the data passed to OnData is less than 
   197      the header length.   After reading a full header, the simple approach is for the parser to return MORE to wait 
   198      for the full request to be received and parsed  (see the existing CassandraParser as an example).
   199      However, as an optimization, the parser can attempt to only 
   200      request the minimum number of bytes required beyond the header to make a policy decision, and then PASS or DROP
   201      the remaining bytes without requiring them to be passed to the Go parser. 
   202  
   203  - **DROP x** :  Remove the first x bytes from the data stream passed to OnData, as they represent a request/reply
   204    that should not be forwarded to the client or server based on policy.  Don't worry about making onData return 
   205    a drop right away, as we'll return to DROP in a later step below.  
   206  
   207  - **ERROR y** : The connection contains data that does not match the protocol spec, and prevents you from further 
   208    parsing the data stream.   The framework will terminate the connection.   An example would be a request length
   209    that falls outside the min/max specified by the protocol spec, or values for a field that fall outside the values
   210    indicated by the spec (e.g., wrong versions, unknown commands).  If you are still able to properly frame the 
   211    requests, you can also choose to simply drop the request and return a protocol error (e.g., similar to an 
   212    ''HTTP 400 Bad Request'' error.   But in all cases, you should write your parser defensively, such that you 
   213    never forward a request that you do not understand, as such a request could become an avenue for subverting 
   214    the intended security visibility and filtering policies.  See proxylib/types.h for the set of valid error codes.   
   215  
   216  See proxylib/proxylib/parserfactory.go for the official OnData interface definition.   
   217  
   218  Keep it simple, and work iteratively.  Start out just getting the framing right.  Can you write a parser that just 
   219  prints out the length and contents of a request, and then PASS each request with no policy enforcement?   
   220  
   221  One simple trick is to comment out the r2d2 parsing logic in OnData, but leave it in the file as a reference, as your protocol will likely
   222  require similar code as we add more functionality below.  
   223  
   224  Step 7: Use Unit Testing To Drive Development
   225  =============================================
   226  
   227  Use unit tests to drive your development.    Its tempting to want to first test your parser by firing up a
   228  client and server and developing on the fly.   But in our experience you’ll iterate faster by using the 
   229  great unit test framework created along with the Go proxy framework.   This framework lets you pass
   230  in an example set of requests as byte arrays to a CheckOnDataOK method, which are passed to the parser's OnData method.
   231  CheckOnDataOK takes a set of expected return values, and compares them to the actual return values from OnData 
   232  processing the byte arrays.  
   233  
   234  Take some time to look at the unit tests for the r2d2 parser, and then for more complex parsers like Cassandra
   235  and Memcached.   For simple text-based protocols, you can simply write ASCII strings to represent protocol messages, 
   236  and convert them to []byte arrays and pass them to CheckOnDataOK.   For binary protocols, one can either create 
   237  byte arrays directly, or use a mechanism to convert a hex string to byte[] array using a helper function like 
   238  hexData in cassandra/cassandraparser_test.go
   239  
   240  A great way to get the exact data to pass in is to copy the data from the Wireshark captures mentioned
   241  above in Step #2.   You can see the full application layer data streams in Wireshark by right-clicking
   242  on a packet and selecting “Follow As… TCP Stream”.  If the protocol is text-based, you can copy the data 
   243  as ASCII (see r2d2/r2d2parser_test.go as an example of this).   For binary data, it can be easier to instead 
   244  select “raw” in the drop-down, and use a basic utility to convert from ascii strings to binary raw data (see 
   245  cassandra/cassandraparser_test.go for an example of this). 
   246  
   247  To run the unit tests, go to proxylib/newproto and run: 
   248  
   249  :: 
   250  
   251    $ go test
   252  
   253  This will build the latest version of your parser and unit test files and run the unit tests.   
   254  
   255  Step 8: Add More Advanced Parsing
   256  =================================
   257  
   258  Thinking back to step #1, what are the critical fields to parse out of the request in order to 
   259  understand the “operation” and “resource” of each request.  Can you print those out for each request?
   260  
   261  Use the unit test framework to pass in increasingly complex requests, and confirm that the parser prints out the right values, and that the 
   262  unit tests are properly slicing the datastream into requests and parsing out the required fields. 
   263  
   264  A couple scenarios to make sure your parser handles properly via unit tests: 
   265  
   266  - data chunks that are less than a full request (return MORE) 
   267  - requests that are spread across multiple data chunks. (return MORE ,then PASS) 
   268  - multiple requests that are bundled into a single data chunk (return PASS, then another PASS)
   269  - rejection of malformed requests (return ERROR). 
   270  
   271  For certain advanced cases, it is required for a parser to store state across requests. 
   272  In this case, data can be stored using data structures that
   273  are included as part of the main parser struct.  See CassandraParser in cassandra/cassandraparser.go as an example 
   274  of how the parser uses a string to store the current 'keyspace' in use, and uses Go maps to keep 
   275  state required for handling prepared queries.   
   276  
   277  Step 9:  Add Policy Loading and Matching
   278  ========================================
   279  
   280  Once you have the parsing of most protocol messages ironed out, its time to start enforcing policy. 
   281  
   282  First, create a Go object that will represent a single rule in the policy language. For example,
   283  this is the rule for the r2d2 protocol, which performs exact match on the command string, and a regex
   284  on the filename:  
   285  
   286  :: 
   287  
   288   type R2d2Rule struct {
   289      cmdExact   string
   290      fileRegexCompiled *regexp.Regexp
   291   }
   292  
   293  There are two key methods to update: 
   294  
   295  - Matches :   This function implements the basic logic of comparing data from a single request 
   296    against a single policy rule, and return true if that rule matches (i.e., allows) that request.  
   297  
   298  - <NewProto>RuleParser : Reads key value pairs from policy, validates those entries, and stores
   299    them as a <NewProto>Rule object.   
   300  
   301  See r2d2/r2d2parser.go for examples of both functions for the r2d2 protocol.  
   302  
   303  You'll also need to update OnData to call p.connection.Matches(), and if this function return false, 
   304  return DROP for a request.  Note: despite the similar names between the Matches() function you 
   305  create in your newprotoparser.go and p.connection.Matches(), do not confuse
   306  the two.  Your OnData function should always call p.connection.Matches() rather than invoking your
   307  own Matches() directly, as p.connection.Matches()
   308  calls the parser's Matches() function only on the subset of L7 rules that apply for the given 
   309  Cilium source identity for this particular connection.  
   310  
   311  Once you add the logic to call Matches() and return DROP in OnData, you will need to update
   312  unit tests to have policies that allow the traffic you expect to be passed.   The following 
   313  is an example of how r2d2/r2d2parser_test.go adds an allow-all policy for a given test: 
   314  
   315  :: 
   316  
   317      s.ins.CheckInsertPolicyText(c, "1", []string{`
   318          name: "cp1"
   319          policy: 2
   320          ingress_per_port_policies: <
   321            port: 80
   322            rules: <
   323              l7_proto: "r2d2"
   324            >
   325          >
   326          `})
   327  
   328  The following is an example of a policy that would allow READ commands with a file 
   329  regex of ".*": 
   330  
   331  :: 
   332  
   333      s.ins.CheckInsertPolicyText(c, "1", []string{`
   334          name: "cp2"
   335          policy: 2
   336          ingress_per_port_policies: <
   337            port: 80
   338            rules: <
   339              l7_proto: "r2d2"
   340              l7_rules: <
   341              rule: <
   342                key: "cmd"
   343                value: "READ"
   344              >
   345              rule: <
   346                key: "file"
   347                value: ".*"
   348              >
   349                >
   350              >
   351            >
   352          >
   353          `})
   354  
   355  
   356  Step 10: Inject Error Response
   357  ==============================
   358  
   359  Simply dropping the request from the request data stream prevents the request from reaching the server, but it would 
   360  leave the client hanging, waiting for a response that would never come since the server did not see the request. 
   361  
   362  Instead, the proxy should return an application-layer reply indicating that access was denied, similar to how
   363  an HTTP proxy would return a ''403 Access Denied'' error.  Look back at the protocol spec discussed in Step 2 to 
   364  understand what an access denied message looks like for this protocol, and use the p.connection.Inject() method 
   365  to send this error reply back to the client.   See r2d2/r2d2parser.go for an example. 
   366  
   367  :: 
   368  
   369      p.connection.Inject(true, []byte("ERROR\r\n"))
   370  
   371  Note:  p.connection.Inject() will inject the data it is passed into the reply datastream.  In order for the client 
   372  to parse this data correctly, it must be injected at a proper framing boundary (i.e., in between other reply messages
   373  that may be in the reply data stream).  If the client is following a basic serial request/reply model per connection, this is 
   374  essentially guaranteed as at the time of a request that is denied, there are no other replies potentially in the 
   375  reply datastream.   But if the protocol supports pipelining (i.e., multiple requests in flight) replies must be properly 
   376  framed and PASSed on a per request basis, and the timing of the call to p.connection.Inject() must be controlled
   377  such that the client will properly match the Error response with the correct request.   See the Memcached parser
   378  as an example of how to accomplish this.  
   379  
   380  Step 11: Add Access Logging
   381  ===========================
   382  
   383  Cilium also has the notion of an ''Access Log'', which records each request handled by the proxy 
   384  and indicates whether the request was allowed or denied.  
   385  
   386  A call to ''p.connection.Log()'' implements access logging. See the OnData function in r2d2/r2d2parser.go 
   387  as an example: 
   388  
   389  :: 
   390  
   391        p.connection.Log(access_log_entry_type,
   392          &cilium.LogEntry_GenericL7{
   393              &cilium.L7LogEntry{
   394                  Proto: "r2d2",
   395                  Fields: map[string]string{
   396                      "cmd":  reqData.cmd,
   397                      "file": reqData.file,
   398                  },
   399              },
   400        })  
   401  
   402  Step 12: Manual Testing
   403  =======================
   404  
   405  Find the standard docker container for running the protocol server.  Often the same image also has a CLI client that you can use as a client. 
   406  
   407  Start both a server and client container running in the cilium dev VM, and attach them to the already created “cilium-net”.  For example, with Cassandra, we run:
   408  
   409  :: 
   410  
   411      docker run --name cass-server -l id=cass-server -d --net cilium-net cassandra
   412  
   413      docker run --name cass-client -l id=cass-client -d --net cilium-net cassandra sh -c 'sleep 3000' 
   414   
   415  
   416  Note that we run both containers with labels that will make it easy to refer to these containers in a cilium 
   417  network policy.   Note that we have the client container run the sleep command, as we will use 'docker exec' to 
   418  access the client CLI.  
   419  
   420  Use ''cilium endpoint list'' to identify the IP address of the protocol server.  
   421  
   422  :: 
   423  
   424    $ cilium endpoint list
   425    ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
   426               ENFORCEMENT        ENFORCEMENT                                                                                     
   427    2987       Disabled           Disabled          31423      container:id=cass-server      f00d::a0b:0:0:bab    10.11.51.247    ready   
   428    27333      Disabled           Disabled          4          reserved:health               f00d::a0b:0:0:6ac5   10.11.92.46     ready   
   429    50923      Disabled           Disabled          18253      container:id=cass-client      f00d::a0b:0:0:c6eb   10.11.175.191   ready 
   430  
   431  One can then invoke the client CLI using that server IP address (10.11.51.247 in the above example):
   432  
   433  :: 
   434  
   435   docker exec -it cass-client sh -c 'cqlsh 10.11.51.247 -e "select * from system.local"'
   436  
   437  Note that in the above example, ingress policy is not enforced for the Cassandra server endpoint, so no data will flow through the
   438  Cassandra parser.  A simple ''allow all'' L7 Cassandra policy can be used to send all data to the Cassandra server through the 
   439  Go Cassandra parser.  This policy has a single empty rule, which matches all requests.  An allow all policy looks like: 
   440  
   441  :: 
   442  
   443    [ { 
   444      "endpointSelector": {"matchLabels":{"id":"cass-server"}}, 
   445      "ingress": [ {
   446  	  "toPorts": [{
   447  		  "ports": [{"port": "9042", "protocol": "TCP"}],
   448              		"rules": {
   449                  		"l7proto": "cassandra",
   450                  		"l7": [{}]
   451              		}
   452  		}]
   453  	  } ] 
   454    }]
   455  
   456  
   457  A policy can be imported into cilium using ''cilium policy import'', after which another call to ''cilium endpoint list''
   458  confirms that ingress policy is now in place on the server.  If the above policy was saved to a file cass-allow-all.json, 
   459  one would run: 
   460  
   461  :: 
   462  
   463      $ cilium policy import cass-allow-all.json
   464      Revision: 1
   465      $ cilium endpoint list
   466      ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
   467                 ENFORCEMENT        ENFORCEMENT                                                                                     
   468      2987       Enabled            Disabled          31423      container:id=cass-server      f00d::a0b:0:0:bab    10.11.51.247    ready   
   469      27333      Disabled           Disabled          4          reserved:health               f00d::a0b:0:0:6ac5   10.11.92.46     ready   
   470      50923      Disabled           Disabled          18253      container:id=cass-client      f00d::a0b:0:0:c6eb   10.11.175.191   ready 
   471  
   472  Note that policy is now showing as ''Enabled'' for the Cassandra server on ingress. 
   473  
   474  To remove this or any other policy, run: 
   475  
   476  :: 
   477  
   478      $ cilium policy delete --all 
   479  
   480  To install a new policy, first delete, and then run ''cilium policy import'' again.  For example, the following policy would allow
   481  select statements on a specific set of tables to this Cassandra server, but deny all other queries. 
   482  
   483  :: 
   484  
   485    [ {
   486      "endpointSelector": {"matchLabels":{"id":"cass-server"}},
   487      "ingress": [ {
   488            "toPorts": [{
   489                    "ports": [{"port": "9042", "protocol": "TCP"}],
   490                          "rules": {
   491                                  "l7proto": "cassandra",
   492                                  "l7": [
   493                                         { "query_action" : "select", "query_table": "^system.*"},
   494                                         { "query_action" : "select", "query_table" : "^posts_db.posts$"}
   495  
   496                                  ]}
   497                          }]
   498           }]
   499    } ]
   500  
   501  When performing manual testing, remember that each time you change your Go proxy code, you must
   502  re-run ''make'' and ''sudo make install'' and then restart the cilium-agent process.  If the only changes
   503  you have made since last compiling cilium are in your cilium/proxylib directory, you can safely 
   504  just run ''make'' and ''sudo make install''  in that directory, which saves time.  
   505  For example: 
   506  
   507  :: 
   508  
   509    $ cd proxylib  // only safe is this is the only directory that has changed
   510    $ make  
   511      <snip> 
   512    $ sudo make install 
   513      <snip> 
   514  
   515  If you rebase or other files change, you need to run both commands from the top level directory.  
   516  
   517  Cilium agent default to running as a service in the development VM.  However, the default options do not include 
   518  the ''--debug-verbose=flow'' flag, which is critical to getting visibility in troubleshooting Go proxy frameworks. 
   519  So it is easiest to stop the cilium service and run the cilium-agent directly as a command in a terminal window, 
   520  and adding the ''--debug-verbose=flow'' flag. 
   521  
   522  :: 
   523  
   524    $ sudo service cilium stop 
   525    
   526    $ sudo /usr/bin/cilium-agent --debug --auto-direct-node-routes --ipv4-range 10.11.0.0/16 --kvstore-opt consul.address=192.168.33.11:8500 --kvstore consul --container-runtime=docker --container-runtime-endpoint=unix:///var/run/docker.sock -t vxlan --fixed-identity-mapping=128=kv-store --fixed-identity-mapping=129=kube-dns --debug-verbose=flow --access-log=/var/log/cilium-access.log
   527  
   528  
   529  The cilium access log is accessible from within the developer VM at ''/var/log/cilium-access.log''
   530  
   531  
   532  Step 13: Add Runtime Tests
   533  ==========================
   534  
   535  Before submitting this change to the Cilium community, it is recommended that you add runtime tests that will run as
   536  part of Cilium's continuous integration testing.   Usually these runtime test can be based on the same container 
   537  images and test commands you used for manual testing.   
   538  
   539  The best approach for adding runtime tests is typically to start out by copying-and-pasting an existing L7 protocol runtime
   540  test and then updating it to run the container images and CLI commands specific to the new protocol.   
   541  See cilium/test/runtime/cassandra.go as an example that matches the use of Cassandra described above in the manual testing
   542  section.   Note that the json policy files used by the runtime tests are stored in cilium/test/runtime/manifests, and 
   543  the Cassandra example policies in those directories are easy to use as a based for similar policies you may create for your
   544  new protocol.  
   545  
   546  Step 14: Review Spec for Corner Cases
   547  =====================================
   548  
   549  Many protocols have advanced features or corner cases that will not manifest themselves as part of basic testing.   
   550  Once you have written a first rev of the parser, it is a good idea to go back and review the protocol's spec or list of 
   551  commands to see what if any aspects may fall outside the scope of your initial parser.    
   552  For example, corner cases like the handling of empty or nil lists may not show up in your testing, but may cause your
   553  parser to fail.   Add more unit tests to cover these corner cases.  
   554  It is OK for the first rev of your parser not to handle all types of requests, or to have a simplified policy structure 
   555  in terms of which fields can be matched.   However, it is 
   556  important to know what aspects of the protocol you are not parsing, and ensure that it does not lead to any security concerns. 
   557  For example, failing to parse prepared statements in a database protocol and instead just passing PREPARE and EXECUTE
   558  commands through would lead to gaping security whole that would render your other filtering meaningless in the face of
   559  a sophisticated attacker.   
   560  
   561  Step 15: Write Docs or Getting Started Guide (optional) 
   562  =======================================================
   563  
   564  At a minimum, the policy examples included as part of the runtime tests serve
   565  as basic documentation of the policy and its expected behavior.  But we also 
   566  encourage adding more user friendly examples and documentation, for example, 
   567  Getting Started Guides.  cilium/Documentation/gettingstarted/cassandra.rst is
   568  a good example to follow.   Also be sure to update Documentation/gettingstarted/index.rst
   569  with a link to this new getting started guide. 
   570  
   571  With that, you are ready to post this change for feedback from the Cilium community.  Congrats!