github.com/cilium/cilium@v1.16.2/Documentation/security/network/proxy/envoy.rst (about) 1 .. only:: not (epub or latex or html) 2 3 WARNING: You are looking at unreleased Cilium documentation. 4 Please use the official rendered version released here: 5 https://docs.cilium.io 6 7 .. _envoy: 8 9 ===== 10 Envoy 11 ===== 12 13 Envoy proxy shipped with Cilium is built with minimal Envoy extensions and custom policy enforcement filters. 14 Cilium uses this minimal distribution as its host proxy for enforcing HTTP and other L7 policies as specified in network policies 15 for the cluster. Cilium proxy is distributed within the Cilium images. 16 17 For more information on the version compatibility matrix, see `Cilium Proxy documentation <https://github.com/cilium/proxy#version-compatibility-matrix>`_. 18 19 *********************** 20 Deployment as DaemonSet 21 *********************** 22 23 Background 24 ========== 25 26 When Cilium L7 functionality (Ingress, Gateway API, Network Policies with L7 functionality, L7 Protocol Visibility) 27 is enabled or installed in a Kubernetes cluster, the Cilium agent starts an Envoy proxy as separate process within 28 the Cilium agent pod. 29 30 That Envoy proxy instance becomes responsible for proxying all matching L7 requests on that node. 31 As a result, L7 traffic targeted by policies depends on the availability of the Cilium agent pod. 32 33 Alternatively, it's possible to deploy the Envoy proxy as independently life-cycled DaemonSet called ``cilium-envoy`` 34 instead of running it from within the Cilium Agent Pod. 35 36 The communication between Cilium agent and Envoy proxy takes place via UNIX domain sockets in both deployment modes. 37 Be that streaming the access logs (e.g. L7 Protocol Visibility), updating the configuration via 38 `xDS <https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol>`_ or accessing the admin interface. 39 40 Enable and configure Envoy DaemonSet 41 ==================================== 42 43 To enable the dedicated Envoy proxy DaemonSet, install Cilium with the Helm value ``envoy.enabled`` set to ``true``. 44 45 Please see the :ref:`helm_reference` (keys with ``envoy.*``) for detailed information on how to configure the Envoy proxy DaemonSet. 46 47 Potential Benefits 48 ================== 49 50 - Cilium Agent restarts (e.g. for upgrades) without impacts for the live traffic proxied via Envoy. 51 - Envoy patch release upgrades without impacts for the Cilium Agent. 52 - Separate CPU and memory limits for Envoy and Cilium Agent for performance isolation. 53 - Envoy application log not mixed with the one of the Cilium Agent. 54 - Dedicated health probes for the Envoy proxy. 55 - Explicit deployment of Envoy proxy during Cilium installation (compared to on demand in the embedded mode). 56 57 .. admonition:: Video 58 :class: attention 59 60 If you'd like to see Cilium Envoy in action, check out `eCHO episode 127: Cilium & Envoy <https://www.youtube.com/watch?v=HEwruycGbCU>`__. 61 62 Known Limitations 63 ================= 64 65 - Due to Pod-to-Pod communication with the Cilium Agent via UNIX domain sockets, Envoy DaemonSet isn't supported with SELinux enabled on the host. This is the default for Red Hat OpenShift. 66 67 ************* 68 Go Extensions 69 ************* 70 71 .. note:: This feature is currently in beta phase. 72 73 .. note:: The Go extensions proxylib framework is residing in cilium/proxy repository. 74 75 This is a guide for developers who are interested in writing a Go extension to the 76 Envoy proxy as part of Cilium. 77 78 .. image:: _static/proxylib_logical_flow.png 79 80 As depicted above, this framework allows a developer to write a small amount of Go 81 code (green box) focused on parsing a new API protocol, and this Go code is able to 82 take full advantage of Cilium features including high-performance redirection to/from Envoy, 83 rich L7-aware policy language 84 and access logging, and visibility into encrypted traffic via kTLS (coming soon!). 85 In sum, you as the developer need only worry about the logic of parsing the protocol, 86 and Cilium + Envoy + eBPF do the heavy-lifting. 87 88 This guide uses simple examples based on a hypothetical "r2d2" protocol 89 (see `proxylib/r2d2/r2d2parser.go <https://github.com/cilium/proxy/blob/main/proxylib/r2d2/r2d2parser.go>`_) 90 that might be used to talk to a simple protocol droid a long time ago in a galaxy far, far away. 91 But it also points to other real protocols like Memcached and Cassandra that already exist in the cilium/proxylib 92 directory. 93 94 Step 1: Decide on a Basic Policy Model 95 ====================================== 96 97 To get started, take some time to think about what it means to provide protocol-aware security 98 in the context of your chosen protocol. Most protocols follow a common pattern of a client 99 who performs an ''operation'' on a ''resource''. For example: 100 101 - A standard RESTful HTTP request has a GET/POST/PUT/DELETE methods (operation) and URLs (resource). 102 - A database protocol like MySQL has SELECT/INSERT/UPDATE/DELETE actions (operation) on a combined database + table name (resource). 103 - A queueing protocol like Kafka has produce/consume (operation) on a particular queue (resource). 104 105 A common policy model is to allow the user to whitelist certain operations on one or more resources. 106 In some cases, the resources need to support regexes to avoid explicit matching on variable content 107 like ids (e.g., /users/<uuid> would match /users/.*) 108 109 In our examples, the ''r2d2'' example, we'll use a basic set of operations (READ/WRITE/HALT/RESET). 110 The READ and WRITE commands also support a 'filename' resource, while HALT and RESET have no resource. 111 112 Step 2: Understand Protocol, Encoding, Framing and Types 113 ======================================================== 114 115 Next, get your head wrapped around how a protocol looks terms of the raw data, as this is what you'll be parsing. 116 117 Try looking for official definitions of the protocol or API. Official docs will not only help you quickly 118 learn how the protocol works, but will also help you by documenting tricky corner cases that wouldn't be 119 obvious just from regular use of the protocol. For example, here are example specs for 120 `Redis Protocol <https://redis.io/topics/protocol>`_ , `Cassandra Protocol <https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v4.spec>`_, 121 and `AWS SQS <https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Welcome.html>`_ . 122 123 These specs help you understand protocol aspects like: 124 125 - **encoding / framing** : how to recognize the beginning/end of individual requests/replies within a TCP stream. 126 This typically involves reading a header that encodes the overall request length, though some simple 127 protocols use a delimiter like ''\r\n\'' to separate messages. 128 129 - **request/reply fields** : for most protocols, you will need to parse out fields at various offsets 130 into the request data in order to extract security-relevant values for visibility + filtering. In some cases, access 131 control requires filtering requests from clients to servers, but in some cases, parsing replies will also be required 132 if reply data is required to understand future requests (e.g., prepared-statements in database protocols). 133 134 - **message flow** : specs often describe various dependencies between different requests. Basic protocols tend to 135 follow a simple serial request/reply model, but more advanced protocols will support pipelining (i.e., sending 136 multiple requests before any replies have been received). 137 138 - **protocol errors** : when a Cilium proxy denies a request based on policy, it should return a protocol-specific 139 error to the client (e.g., in HTTP, a proxy should return a ''403 Access Denied'' error). Looking at the protocol 140 spec will typically indicate how you should return an equivalent ''Access Denied'' error. 141 142 Sometimes, the protocol spec does not give you a full sense of the set of commands that can be sent over the protocol. In that 143 case, looking at higher-level user documentation can fill in some of these knowledge gaps. Here are examples for 144 `Redis Commands <https://redis.io/commands>`_ and `Cassandra CQL Commands <https://docs.datastax.com/en/archived/cql/3.1/cql/cql_reference/cqlCommandsTOC.html>`_ . 145 146 Another great trick is to use `Wireshark <https://www.wireshark.org>`_ to capture raw packet data between 147 a client and server. For many protocols, the `Wireshark Sample Captures <https://wiki.wireshark.org/SampleCaptures>`_ 148 has already saved captures for us. Otherwise, you can easily use tcpdump to capture a file. For example, for 149 MySQL traffic on port 3306, you could run the following in a container running the MySQL client or server: 150 “tcpdump -s 0 port 3306 -w mysql.pcap”. `More Info <https://linuxexplore.com/2012/06/07/use-tcpdump-to-capture-in-a-pcap-file-wireshark-dump/>`_ 151 152 In our example r2d2 protocol, we'll keep the spec as simple as possible. It is a text-only based protocol, 153 with each request being a line terminated by ''\r\n''. A request starts with a case-insensitive string 154 command ("READ","WRITE","HALT","RESET"). If the command is "READ" or "WRITE", the command must be followed 155 by a space, and a non-empty filename that contains only non whitespace ASCII characters. 156 157 Step 3: Search for Existing Parser Code / Libraries 158 =================================================== 159 160 Look for open source Go library/code that can help. 161 Is there existing open source Go code that parse your protocol that you can leverage, 162 either directly as library or a motivating example? For example, the `tidwall/recon library 163 <https://github.com/tidwall/redcon>`_ parses Redis in Go, and `Vitess 164 <https://github.com/vitessio/vitess>`_ parses MySQL in Go. `Wireshark dissectors 165 <https://github.com/boundary/wireshark/tree/master/epan/dissectors>`_ also has a wealth of 166 protocol parsers written in C that can serve as useful guidance. Note: finding client-only 167 protocol parsing code is typically less helpful than finding a proxy implementation, or a full 168 parser library. This is because the set of requests a client parsers is typically the inverse 169 set of the requests a Cilium proxy needs to parse, since the proxy mimics the server rather than 170 the client. Still, viewing a Go client can give you a general idea of how to parse the 171 general serialization format of the protocol. 172 173 Step 4: Follow the Cilium Developer Guide 174 ========================================= 175 176 It is easiest to start Cilium development by following the :ref:`dev_guide` 177 178 After cloning cilium/proxy repo: 179 180 .. code-block:: shell-session 181 182 $ cd proxy 183 $ vagrant up 184 $ cd proxylib 185 186 While this dev VM is running, you can open additional terminals to the cilium/proxy dev VM 187 by running ``vagrant ssh`` from within the cilium/proxy source directory. 188 189 190 Step 5: Create New Proxy Skeleton 191 ================================= 192 193 From inside the proxylib directory, copy the rd2d directory and rename the files. 194 Replace ''newproto'' with your protocol: 195 196 .. code-block:: shell-session 197 198 $ mkdir newproto 199 $ cd newproto 200 $ cp ../r2d2/r2d2parser.go newproto.go 201 $ cp ../r2d2/r2d2parser_test.go newproto_test.go 202 203 204 Within both newproto.go and newproto_test.go update references to r2d2 with 205 your protocol name. Search for both ''r2d2'' and ''R2D2''. 206 207 Also, edit proxylib.go and add the following import line: 208 209 :: 210 211 _ "github.com/cilium/proxy/proxylib/newproto" 212 213 214 Step 6: Update OnData Method 215 ============================ 216 217 Implementing a parser requires you as the developer to implement three primary functions, 218 shown as blue in the diagram below. We will cover OnData() in this section, and 219 the other functions in section `Step 9: Add Policy Loading and Matching`_. 220 221 .. image:: _static/proxylib_key_functions.png 222 223 The beating heart of your parsing is implementing the onData function. You can think of any 224 proxy as have two data streams, one in the request direction (i.e., client to server) and one in 225 the reply direction (i.e., server to client). OnData is called when there is data to process, 226 and the value of the boolean 'reply' parameter indicates the direction of the stream for a given 227 call to OnData. The data passed to OnData is a slice of byte slices (i.e., an array of byte arrays). 228 229 The return values of the OnData function tell the Go framework tell how data in the stream 230 should be processed, with four primary outcomes: 231 232 - **PASS x** : The next x bytes in the data stream passed to OnData represent a request/reply that should be 233 passed on to the server/client. The common case here is that this is a request that should be 234 allowed by policy, or that no policy is applied. Note: x bytes may be less than the total amount 235 of data passed to OnData, in which case the remaining bytes will still be in the data stream when 236 onData is invoked next. x bytes may also be more than the data that has been passed to OnData. 237 For example, in the case of a protocol where the parser filters only on values in a protocol header, 238 it is often possible to make a filtering decision, and then pass (or drop) the size of the full 239 request/reply without having the entire request passed to Go. 240 241 - **MORE x** : The buffers passed to OnData to do not represent all of the data required to frame and 242 filter the request/reply. Instead, the parser 243 needs to see at least x additional bytes beyond the current data to make a decision. 244 In some cases, the full request must be read to understand framing and filtering, but in others a decision 245 can be made simply by reading a protocol header. When parsing data, be defensive, and recognize that it is technically possible that 246 data arrives one byte at a time. Two common scenarios exist here: 247 248 - **Text-based Protocols** : For text-based protocols 249 that use a delimiter like "\r\n", it is common to simply check if the delimiter exists, and return 250 MORE 1 if it does not, as technically one more character could result in the delimiter being present. 251 See the sample r2d2 parser as a basic example of this. 252 253 - **Binary-based protocols** : Many binary protocols 254 have a fixed header length, which containers a field that then indicates the remaining length 255 of the request. In the binary case, first check to make sure a full header is received. Typically 256 the header will indicate both the full request length (i.e., framing), as well as the request type, 257 which indicates how much of the full request must be read in order to perform filtering (in many cases, this is less than 258 the full request). A binary parser will typically return MORE if the data passed to OnData is less than 259 the header length. After reading a full header, the simple approach is for the parser to return MORE to wait 260 for the full request to be received and parsed (see the existing CassandraParser as an example). 261 However, as an optimization, the parser can attempt to only 262 request the minimum number of bytes required beyond the header to make a policy decision, and then PASS or DROP 263 the remaining bytes without requiring them to be passed to the Go parser. 264 265 - **DROP x** : Remove the first x bytes from the data stream passed to OnData, as they represent a request/reply 266 that should not be forwarded to the client or server based on policy. Don't worry about making onData return 267 a drop right away, as we'll return to DROP in a later step below. 268 269 - **ERROR y** : The connection contains data that does not match the protocol spec, and prevents you from further 270 parsing the data stream. The framework will terminate the connection. An example would be a request length 271 that falls outside the min/max specified by the protocol spec, or values for a field that fall outside the values 272 indicated by the spec (e.g., wrong versions, unknown commands). If you are still able to properly frame the 273 requests, you can also choose to simply drop the request and return a protocol error (e.g., similar to an 274 ''HTTP 400 Bad Request'' error. But in all cases, you should write your parser defensively, such that you 275 never forward a request that you do not understand, as such a request could become an avenue for subverting 276 the intended security visibility and filtering policies. See proxylib/types.h for the set of valid error codes. 277 278 See proxylib/proxylib/parserfactory.go for the official OnData interface definition. 279 280 Keep it simple, and work iteratively. Start out just getting the framing right. Can you write a parser that just 281 prints out the length and contents of a request, and then PASS each request with no policy enforcement? 282 283 One simple trick is to comment out the r2d2 parsing logic in OnData, but leave it in the file as a reference, as your protocol will likely 284 require similar code as we add more functionality below. 285 286 Step 7: Use Unit Testing To Drive Development 287 ============================================= 288 289 Use unit tests to drive your development. Its tempting to want to first test your parser by firing up a 290 client and server and developing on the fly. But in our experience you’ll iterate faster by using the 291 great unit test framework created along with the Go proxy framework. This framework lets you pass 292 in an example set of requests as byte arrays to a CheckOnDataOK method, which are passed to the parser's OnData method. 293 CheckOnDataOK takes a set of expected return values, and compares them to the actual return values from OnData 294 processing the byte arrays. 295 296 Take some time to look at the unit tests for the r2d2 parser, and then for more complex parsers like Cassandra 297 and Memcached. For simple text-based protocols, you can simply write ASCII strings to represent protocol messages, 298 and convert them to []byte arrays and pass them to CheckOnDataOK. For binary protocols, one can either create 299 byte arrays directly, or use a mechanism to convert a hex string to byte[] array using a helper function like 300 hexData in cassandra/cassandraparser_test.go 301 302 A great way to get the exact data to pass in is to copy the data from the Wireshark captures mentioned 303 above in Step #2. You can see the full application layer data streams in Wireshark by right-clicking 304 on a packet and selecting “Follow As… TCP Stream”. If the protocol is text-based, you can copy the data 305 as ASCII (see r2d2/r2d2parser_test.go as an example of this). For binary data, it can be easier to instead 306 select “raw” in the drop-down, and use a basic utility to convert from ascii strings to binary raw data (see 307 cassandra/cassandraparser_test.go for an example of this). 308 309 To run the unit tests, go to proxylib/newproto and run: 310 311 .. code-block:: shell-session 312 313 $ go test 314 315 This will build the latest version of your parser and unit test files and run the unit tests. 316 317 Step 8: Add More Advanced Parsing 318 ================================= 319 320 Thinking back to step #1, what are the critical fields to parse out of the request in order to 321 understand the “operation” and “resource” of each request. Can you print those out for each request? 322 323 Use the unit test framework to pass in increasingly complex requests, and confirm that the parser prints out the right values, and that the 324 unit tests are properly slicing the datastream into requests and parsing out the required fields. 325 326 A couple scenarios to make sure your parser handles properly via unit tests: 327 328 - data chunks that are less than a full request (return MORE) 329 - requests that are spread across multiple data chunks. (return MORE ,then PASS) 330 - multiple requests that are bundled into a single data chunk (return PASS, then another PASS) 331 - rejection of malformed requests (return ERROR). 332 333 For certain advanced cases, it is required for a parser to store state across requests. 334 In this case, data can be stored using data structures that 335 are included as part of the main parser struct. See CassandraParser in cassandra/cassandraparser.go as an example 336 of how the parser uses a string to store the current 'keyspace' in use, and uses Go maps to keep 337 state required for handling prepared queries. 338 339 Step 9: Add Policy Loading and Matching 340 ======================================== 341 342 Once you have the parsing of most protocol messages ironed out, its time to start enforcing policy. 343 344 First, create a Go object that will represent a single rule in the policy language. For example, 345 this is the rule for the r2d2 protocol, which performs exact match on the command string, and a regex 346 on the filename: 347 348 .. code-block:: go 349 350 type R2d2Rule struct { 351 cmdExact string 352 fileRegexCompiled *regexp.Regexp 353 } 354 355 There are two key methods to update: 356 357 - Matches : This function implements the basic logic of comparing data from a single request 358 against a single policy rule, and return true if that rule matches (i.e., allows) that request. 359 360 - <NewProto>RuleParser : Reads key value pairs from policy, validates those entries, and stores 361 them as a <NewProto>Rule object. 362 363 See r2d2/r2d2parser.go for examples of both functions for the r2d2 protocol. 364 365 You'll also need to update OnData to call p.connection.Matches(), and if this function return false, 366 return DROP for a request. Note: despite the similar names between the Matches() function you 367 create in your newprotoparser.go and p.connection.Matches(), do not confuse 368 the two. Your OnData function should always call p.connection.Matches() rather than invoking your 369 own Matches() directly, as p.connection.Matches() 370 calls the parser's Matches() function only on the subset of L7 rules that apply for the given 371 Cilium source identity for this particular connection. 372 373 Once you add the logic to call Matches() and return DROP in OnData, you will need to update 374 unit tests to have policies that allow the traffic you expect to be passed. The following 375 is an example of how r2d2/r2d2parser_test.go adds an allow-all policy for a given test: 376 377 .. code-block:: go 378 379 s.ins.CheckInsertPolicyText(c, "1", []string{` 380 name: "cp1" 381 policy: 2 382 ingress_per_port_policies: < 383 port: 80 384 rules: < 385 l7_proto: "r2d2" 386 > 387 > 388 `}) 389 390 The following is an example of a policy that would allow READ commands with a file 391 regex of ".*": 392 393 .. code-block:: go 394 395 s.ins.CheckInsertPolicyText(c, "1", []string{` 396 name: "cp2" 397 policy: 2 398 ingress_per_port_policies: < 399 port: 80 400 rules: < 401 l7_proto: "r2d2" 402 l7_rules: < 403 rule: < 404 key: "cmd" 405 value: "READ" 406 > 407 rule: < 408 key: "file" 409 value: ".*" 410 > 411 > 412 > 413 > 414 > 415 `}) 416 417 418 Step 10: Inject Error Response 419 ============================== 420 421 Simply dropping the request from the request data stream prevents the request from reaching the server, but it would 422 leave the client hanging, waiting for a response that would never come since the server did not see the request. 423 424 Instead, the proxy should return an application-layer reply indicating that access was denied, similar to how 425 an HTTP proxy would return a ''403 Access Denied'' error. Look back at the protocol spec discussed in Step 2 to 426 understand what an access denied message looks like for this protocol, and use the p.connection.Inject() method 427 to send this error reply back to the client. See r2d2/r2d2parser.go for an example. 428 429 .. code-block:: go 430 431 p.connection.Inject(true, []byte("ERROR\r\n")) 432 433 Note: p.connection.Inject() will inject the data it is passed into the reply datastream. In order for the client 434 to parse this data correctly, it must be injected at a proper framing boundary (i.e., in between other reply messages 435 that may be in the reply data stream). If the client is following a basic serial request/reply model per connection, this is 436 essentially guaranteed as at the time of a request that is denied, there are no other replies potentially in the 437 reply datastream. But if the protocol supports pipelining (i.e., multiple requests in flight) replies must be properly 438 framed and PASSed on a per request basis, and the timing of the call to p.connection.Inject() must be controlled 439 such that the client will properly match the Error response with the correct request. See the Memcached parser 440 as an example of how to accomplish this. 441 442 Step 11: Add Access Logging 443 =========================== 444 445 Cilium also has the notion of an ''Access Log'', which records each request handled by the proxy 446 and indicates whether the request was allowed or denied. 447 448 A call to ``p.connection.Log()`` implements access logging. See the OnData function in r2d2/r2d2parser.go 449 as an example: 450 451 .. code-block:: go 452 453 p.connection.Log(access_log_entry_type, 454 &cilium.LogEntry_GenericL7{ 455 &cilium.L7LogEntry{ 456 Proto: "r2d2", 457 Fields: map[string]string{ 458 "cmd": reqData.cmd, 459 "file": reqData.file, 460 }, 461 }, 462 }) 463 464 Step 12: Manual Testing 465 ======================= 466 467 Find the standard docker container for running the protocol server. Often the same image also has a CLI client that you can use as a client. 468 469 Start both a server and client container running in the cilium dev VM, and attach them to the already created “cilium-net”. For example, with Cassandra, we run: 470 471 .. code-block:: shell-session 472 473 docker run --name cass-server -l id=cass-server -d --net cilium-net cassandra 474 475 docker run --name cass-client -l id=cass-client -d --net cilium-net cassandra sh -c 'sleep 3000' 476 477 478 Note that we run both containers with labels that will make it easy to refer to these containers in a cilium 479 network policy. Note that we have the client container run the sleep command, as we will use 'docker exec' to 480 access the client CLI. 481 482 Use ``cilium-dbg endpoint list`` to identify the IP address of the protocol server. 483 484 .. code-block:: shell-session 485 486 $ cilium-dbg endpoint list 487 ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS 488 ENFORCEMENT ENFORCEMENT 489 2987 Disabled Disabled 31423 container:id=cass-server f00d::a0b:0:0:bab 10.11.51.247 ready 490 27333 Disabled Disabled 4 reserved:health f00d::a0b:0:0:6ac5 10.11.92.46 ready 491 50923 Disabled Disabled 18253 container:id=cass-client f00d::a0b:0:0:c6eb 10.11.175.191 ready 492 493 One can then invoke the client CLI using that server IP address (10.11.51.247 in the above example): 494 495 .. code-block:: shell-session 496 497 docker exec -it cass-client sh -c 'cqlsh 10.11.51.247 -e "select * from system.local"' 498 499 Note that in the above example, ingress policy is not enforced for the Cassandra server endpoint, so no data will flow through the 500 Cassandra parser. A simple ''allow all'' L7 Cassandra policy can be used to send all data to the Cassandra server through the 501 Go Cassandra parser. This policy has a single empty rule, which matches all requests. An allow all policy looks like: 502 503 .. code-block:: json 504 505 [ { 506 "endpointSelector": {"matchLabels":{"id":"cass-server"}}, 507 "ingress": [ { 508 "toPorts": [{ 509 "ports": [{"port": "9042", "protocol": "TCP"}], 510 "rules": { 511 "l7proto": "cassandra", 512 "l7": [{}] 513 } 514 }] 515 } ] 516 }] 517 518 519 A policy can be imported into cilium using ``cilium policy import``, after which another call to ``cilium-dbg endpoint list`` 520 confirms that ingress policy is now in place on the server. If the above policy was saved to a file cass-allow-all.json, 521 one would run: 522 523 .. code-block:: shell-session 524 525 $ cilium-dbg policy import cass-allow-all.json 526 Revision: 1 527 $ cilium-dbg endpoint list 528 ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS 529 ENFORCEMENT ENFORCEMENT 530 2987 Enabled Disabled 31423 container:id=cass-server f00d::a0b:0:0:bab 10.11.51.247 ready 531 27333 Disabled Disabled 4 reserved:health f00d::a0b:0:0:6ac5 10.11.92.46 ready 532 50923 Disabled Disabled 18253 container:id=cass-client f00d::a0b:0:0:c6eb 10.11.175.191 ready 533 534 Note that policy is now showing as ''Enabled'' for the Cassandra server on ingress. 535 536 To remove this or any other policy, run: 537 538 .. code-block:: shell-session 539 540 $ cilium-dbg policy delete --all 541 542 To install a new policy, first delete, and then run ``cilium policy import`` again. For example, the following policy would allow 543 select statements on a specific set of tables to this Cassandra server, but deny all other queries. 544 545 .. code-block:: json 546 547 [ { 548 "endpointSelector": {"matchLabels":{"id":"cass-server"}}, 549 "ingress": [ { 550 "toPorts": [{ 551 "ports": [{"port": "9042", "protocol": "TCP"}], 552 "rules": { 553 "l7proto": "cassandra", 554 "l7": [ 555 { "query_action" : "select", "query_table": "^system.*"}, 556 { "query_action" : "select", "query_table" : "^posts_db.posts$"} 557 558 ]} 559 }] 560 }] 561 } ] 562 563 When performing manual testing, remember that each time you change your Go proxy code, you must 564 re-run ``make`` and ``sudo make install`` and then restart the cilium-agent process. If the only changes 565 you have made since last compiling cilium are in your cilium/proxylib directory, you can safely 566 just run ``make`` and ``sudo make install`` in that directory, which saves time. 567 For example: 568 569 .. code-block:: shell-session 570 571 $ cd proxylib // only safe is this is the only directory that has changed 572 $ make 573 <snip> 574 $ sudo make install 575 <snip> 576 577 If you rebase or other files change, you need to run both commands from the top level directory. 578 579 Cilium agent default to running as a service in the development VM. However, the default options do not include 580 the ``--debug-verbose=flow`` flag, which is critical to getting visibility in troubleshooting Go proxy frameworks. 581 So it is easiest to stop the cilium service and run the cilium-agent directly as a command in a terminal window, 582 and adding the ``--debug-verbose=flow`` flag. 583 584 .. code-block:: shell-session 585 586 $ sudo service cilium stop 587 588 $ sudo /usr/bin/cilium-agent --debug --ipv4-range 10.11.0.0/16 --kvstore-opt consul.address=192.168.60.11:8500 --kvstore consul -t vxlan --fixed-identity-mapping=128=kv-store --fixed-identity-mapping=129=kube-dns --debug-verbose=flow 589 590 591 Step 13: Add Runtime Tests 592 ========================== 593 594 Before submitting this change to the Cilium community, it is recommended that you add runtime tests that will run as 595 part of Cilium's continuous integration testing. Usually these runtime test can be based on the same container 596 images and test commands you used for manual testing. 597 598 The best approach for adding runtime tests is typically to start out by copying-and-pasting an existing L7 protocol runtime 599 test and then updating it to run the container images and CLI commands specific to the new protocol. 600 See cilium/test/runtime/cassandra.go as an example that matches the use of Cassandra described above in the manual testing 601 section. Note that the json policy files used by the runtime tests are stored in cilium/test/runtime/manifests, and 602 the Cassandra example policies in those directories are easy to use as a based for similar policies you may create for your 603 new protocol. 604 605 Step 14: Review Spec for Corner Cases 606 ===================================== 607 608 Many protocols have advanced features or corner cases that will not manifest themselves as part of basic testing. 609 Once you have written a first rev of the parser, it is a good idea to go back and review the protocol's spec or list of 610 commands to see what if any aspects may fall outside the scope of your initial parser. 611 For example, corner cases like the handling of empty or nil lists may not show up in your testing, but may cause your 612 parser to fail. Add more unit tests to cover these corner cases. 613 It is OK for the first rev of your parser not to handle all types of requests, or to have a simplified policy structure 614 in terms of which fields can be matched. However, it is 615 important to know what aspects of the protocol you are not parsing, and ensure that it does not lead to any security concerns. 616 For example, failing to parse prepared statements in a database protocol and instead just passing PREPARE and EXECUTE 617 commands through would lead to gaping security whole that would render your other filtering meaningless in the face of 618 a sophisticated attacker. 619 620 Step 15: Write Docs or Getting Started Guide (optional) 621 ======================================================= 622 623 At a minimum, the policy examples included as part of the runtime tests serve 624 as basic documentation of the policy and its expected behavior. But we also 625 encourage adding more user friendly examples and documentation, for example, 626 Getting Started Guides. ``cilium/Documentation/gettingstarted/cassandra.rst`` is 627 a good example to follow. Also be sure to update ``Documentation/gettingstarted/index.rst`` 628 with a link to this new getting started guide. 629 630 With that, you are ready to post this change for feedback from the Cilium community. Congrats!