gvisor.dev/gvisor@v0.0.0-20240520182842-f9d4d51c7e0f/test/packetimpact/README.md (about)

     1  # Packetimpact
     2  
     3  ## What is packetimpact?
     4  
     5  Packetimpact is a tool for platform-independent network testing. It is heavily
     6  inspired by [packetdrill](https://github.com/google/packetdrill). It creates two
     7  network namespaces. One is for the test bench, which operates the test. The
     8  other is for the device-under-test (DUT), which is the software being tested.
     9  The test bench communicates over the network with the DUT to check correctness
    10  of the network.
    11  
    12  ### Goals
    13  
    14  Packetimpact aims to provide:
    15  
    16  *   A **multi-platform** solution that can test both Linux and gVisor.
    17  *   **Conciseness** on par with packetdrill scripts.
    18  *   **Control-flow** like for loops, conditionals, and variables.
    19  *   **Flexibility** to specify every byte in a packet or use multiple sockets.
    20  
    21  ## How to run packetimpact tests?
    22  
    23  Run a test, e.g. `fin_wait2_timeout`, against Linux:
    24  
    25  ```bash
    26  $ bazel test //test/packetimpact/tests:fin_wait2_timeout_native_test
    27  ```
    28  
    29  Run the same test, but against gVisor:
    30  
    31  ```bash
    32  $ bazel test //test/packetimpact/tests:fin_wait2_timeout_netstack_test
    33  ```
    34  
    35  ## When to use packetimpact?
    36  
    37  There are a few ways to write networking tests for gVisor currently:
    38  
    39  *   [Go unit tests](https://github.com/google/gvisor/tree/master/pkg/tcpip)
    40  *   [syscall tests](https://github.com/google/gvisor/tree/master/test/syscalls/linux)
    41  *   [packetdrill tests](https://github.com/google/gvisor/tree/master/test/packetdrill)
    42  *   packetimpact tests
    43  
    44  The right choice depends on the needs of the test.
    45  
    46  Feature        | Go unit test | syscall test | packetdrill | packetimpact
    47  -------------- | ------------ | ------------ | ----------- | ------------
    48  Multi-platform | no           | **YES**      | **YES**     | **YES**
    49  Concise        | no           | somewhat     | somewhat    | **VERY**
    50  Control-flow   | **YES**      | **YES**      | no          | **YES**
    51  Flexible       | **VERY**     | no           | somewhat    | **VERY**
    52  
    53  ### Go unit tests
    54  
    55  If the test depends on the internals of gVisor and doesn't need to run on Linux
    56  or other platforms for comparison purposes, a Go unit test can be appropriate.
    57  They can observe internals of gVisor networking. The downside is that they are
    58  **not concise** and **not multi-platform**. If you require insight on gVisor
    59  internals, this is the right choice.
    60  
    61  ### Syscall tests
    62  
    63  Syscall tests are **multi-platform** but cannot examine the internals of gVisor
    64  networking. They are **concise**. They can use **control-flow** structures like
    65  conditionals, for loops, and variables. However, they are limited to only what
    66  the POSIX interface provides so they are **not flexible**. For example, you
    67  would have difficulty writing a syscall test that intentionally sends a bad IP
    68  checksum. Or if you did write that test with raw sockets, it would be very
    69  **verbose** to write a test that intentionally send wrong checksums, wrong
    70  protocols, wrong sequence numbers, etc.
    71  
    72  ### Packetdrill tests
    73  
    74  Packetdrill tests are **multi-platform** and can run against both Linux and
    75  gVisor. They are **concise** and use a special packetdrill scripting language.
    76  They are **more flexible** than a syscall test in that they can send packets
    77  that a syscall test would have difficulty sending, like a packet with a
    78  calcuated ACK number. But they are also somewhat limimted in flexibiilty in that
    79  they can't do tests with multiple sockets. They have **no control-flow** ability
    80  like variables or conditionals. For example, it isn't possible to send a packet
    81  that depends on the window size of a previous packet because the packetdrill
    82  language can't express that. Nor could you branch based on whether or not the
    83  other side supports window scaling, for example.
    84  
    85  ### Packetimpact tests
    86  
    87  Packetimpact tests are similar to Packetdrill tests except that they are written
    88  in Go instead of the packetdrill scripting language. That gives them all the
    89  **control-flow** abilities of Go (loops, functions, variables, etc). They are
    90  **multi-platform** in the same way as packetdrill tests but even more
    91  **flexible** because Go is more expressive than the scripting language of
    92  packetdrill. However, Go is **not as concise** as the packetdrill language. Many
    93  design decisions below are made to mitigate that.
    94  
    95  ## How it works
    96  
    97  ```
    98       Testbench                           Device-Under-Test (DUT)
    99      +-------------------+               +------------------------+
   100      |                   |   TEST NET    |                        |
   101      | rawsockets.go <-->| <===========> | <---+                  |
   102      |           ^       |               |     |                  |
   103      |           |       |               |     |                  |
   104      |           v       |               |     |                  |
   105      |     unittest      |               |     |                  |
   106      |           ^       |               |     |                  |
   107      |           |       |               |     |                  |
   108      |           v       |               |     v                  |
   109      |         dut.go <========gRPC========> posix server         |
   110      |                   |  CONTROL NET  |                        |
   111      +-------------------+               +------------------------+
   112  ```
   113  
   114  Two network namespaces are created by the test runner, one for the testbench and
   115  the other for the device under test (DUT). The runner connects the two
   116  namespaces with a control veth pair and test veth pair. It also does some other
   117  tasks like waiting until the DUT is ready before starting the test and
   118  installing iptables rules so that RST won't be generated for TCP segments from
   119  the DUT that the kernel has no knowledge about.
   120  
   121  ### DUT
   122  
   123  The DUT namespace runs a program called the "posix_server". The posix_server is
   124  written in c++ for maximum portability. Its job is to receive directions from
   125  the test bench on what actions to take. For this, the posix_server does three
   126  steps in a loop:
   127  
   128  1.  Listen for a request from the test bench.
   129  2.  Execute a command.
   130  3.  Send the response back to the test bench.
   131  
   132  The requests and responses are
   133  [protobufs](https://developers.google.com/protocol-buffers) and the
   134  communication is done with [gRPC](https://grpc.io/). The commands run are
   135  [POSIX socket commands](https://en.wikipedia.org/wiki/Berkeley_sockets#Socket_API_functions),
   136  with the inputs and outputs converted into protobuf requests and responses. All
   137  communication is on the control network, so that the test network is unaffected
   138  by extra packets.
   139  
   140  For example, this is the request and response pair to call
   141  [`socket()`](http://man7.org/linux/man-pages/man2/socket.2.html):
   142  
   143  ```protocol-buffer
   144  message SocketRequest {
   145    int32 domain = 1;
   146    int32 type = 2;
   147    int32 protocol = 3;
   148  }
   149  
   150  message SocketResponse {
   151    int32 fd = 1;
   152    int32 errno_ = 2;
   153  }
   154  ```
   155  
   156  ##### Alternatives considered
   157  
   158  *   We could have use JSON for communication instead. It would have been a
   159      lighter-touch than protobuf but protobuf handles all the data type and has
   160      strict typing to prevent a class of errors. The test bench could be written
   161      in other languages, too.
   162  *   Instead of mimicking the POSIX interfaces, arguments could have had a more
   163      natural form, like the `bind()` getting a string IP address instead of bytes
   164      in a `sockaddr_t`. However, conforming to the existing structures keeps more
   165      of the complexity in Go and keeps the posix_server simpler and thus more
   166      likely to compile everywhere.
   167  
   168  ### Test Bench
   169  
   170  The test bench does most of the work in a test. It is a Go program that compiles
   171  on the host and is run inside the test bench's namespace. It is a regular
   172  [go unit test](https://golang.org/pkg/testing/) that imports the test bench
   173  framework. The test bench framework is based on three basic utilities:
   174  
   175  *   Commanding the DUT to run POSIX commands and return responses.
   176  *   Sending raw packets to the DUT on the test network.
   177  *   Listening for raw packets from the DUT on the test network.
   178  
   179  #### DUT commands
   180  
   181  To keep the interface to the DUT consistent and easy-to-use, each POSIX command
   182  supported by the posix_server is wrapped in functions with signatures similar to
   183  the ones in the [Go unix package](https://godoc.org/golang.org/x/sys/unix). This
   184  way all the details of endianess and (un)marshalling of go structs such as
   185  [unix.Timeval](https://godoc.org/golang.org/x/sys/unix#Timeval) is handled in
   186  one place. This also makes it straight-forward to convert tests that use `unix.`
   187  or `syscall.` calls to `dut.` calls.
   188  
   189  For example, creating a connection to the DUT and commanding it to make a socket
   190  looks like this:
   191  
   192  ```go
   193  dut := testbench.NewDut(t)
   194  fd, err := dut.SocketWithErrno(unix.AF_INET, unix.SOCK_STREAM, unix.IPPROTO_IP)
   195  if fd < 0 {
   196    t.Fatalf(...)
   197  }
   198  ```
   199  
   200  Because the usual case is to fail the test when the DUT fails to create a
   201  socket, there is a concise version of each of the `...WithErrno` functions that
   202  does that:
   203  
   204  ```go
   205  dut := testbench.NewDut(t)
   206  fd := dut.Socket(unix.AF_INET, unix.SOCK_STREAM, unix.IPPROTO_IP)
   207  ```
   208  
   209  The DUT and other structs in the code store a `*testing.T` so that they can
   210  provide versions of functions that call `t.Fatalf(...)`. This helps keep tests
   211  concise.
   212  
   213  ##### Alternatives considered
   214  
   215  *   Instead of mimicking the `unix.` go interface, we could have invented a more
   216      natural one, like using `float64` instead of `Timeval`. However, using the
   217      same function signatures that `unix.` has makes it easier to convert code to
   218      `dut.`. Also, using an existing interface ensures that we don't invent an
   219      interface that isn't extensible. For example, if we invented a function for
   220      `bind()` that didn't support IPv6 and later we had to add a second `bind6()`
   221      function.
   222  
   223  #### Sending/Receiving Raw Packets
   224  
   225  The framework wraps POSIX sockets for sending and receiving raw frames. Both
   226  send and receive are synchronous commands.
   227  [SO_RCVTIMEO](http://man7.org/linux/man-pages/man7/socket.7.html) is used to set
   228  a timeout on the receive commands. For ease of use, these are wrapped in an
   229  `Injector` and a `Sniffer`. They have functions:
   230  
   231  ```go
   232  func (s *Sniffer) Recv(timeout time.Duration) []byte {...}
   233  func (i *Injector) Send(b []byte) {...}
   234  ```
   235  
   236  ##### Alternatives considered
   237  
   238  *   [gopacket](https://github.com/google/gopacket) pcap has raw socket support
   239      but requires cgo. cgo is not guaranteed to be portable from the host to the
   240      container and in practice, the container doesn't recognize binaries built on
   241      the host if they use cgo. Packetimpact used to be based on docker, so the
   242      library was not adopted, now we can start to consider using the library.
   243  *   Both gVisor and gopacket have the ability to read and write pcap files
   244      without cgo but that is insufficient here because we can't just replay pcap
   245      files, we need a more dynamic solution.
   246  *   The sniffer and injector can't share a socket because they need to be bound
   247      differently.
   248  *   Sniffing could have been done asynchronously with channels, obviating the
   249      need for `SO_RCVTIMEO`. But that would introduce asynchronous complication.
   250      `SO_RCVTIMEO` is well supported on the test bench.
   251  
   252  #### `Layer` struct
   253  
   254  A large part of packetimpact tests is creating packets to send and comparing
   255  received packets against expectations. To keep tests concise, it is useful to be
   256  able to specify just the important parts of packets that need to be set. For
   257  example, sending a packet with default values except for TCP Flags. And for
   258  packets received, it's useful to be able to compare just the necessary parts of
   259  received packets and ignore the rest.
   260  
   261  To aid in both of those, Go structs with optional fields are created for each
   262  encapsulation type, such as IPv4, TCP, and Ethernet. This is inspired by
   263  [scapy](https://scapy.readthedocs.io/en/latest/). For example, here is the
   264  struct for Ethernet:
   265  
   266  ```go
   267  type Ether struct {
   268    LayerBase
   269    SrcAddr *tcpip.LinkAddress
   270    DstAddr *tcpip.LinkAddress
   271    Type    *tcpip.NetworkProtocolNumber
   272  }
   273  ```
   274  
   275  Each struct has the same fields as those in the
   276  [gVisor headers](https://github.com/google/gvisor/tree/master/pkg/tcpip/header)
   277  but with a pointer for each field that may be `nil`.
   278  
   279  ##### Alternatives considered
   280  
   281  *   Just use []byte like gVisor headers do. The drawback is that it makes the
   282      tests more verbose.
   283      *   For example, there would be no way to call `Send(myBytes)` concisely and
   284          indicate if the checksum should be calculated automatically versus
   285          overridden. The only way would be to add lines to the test to calculate
   286          it before each Send, which is wordy. Or make multiple versions of Send:
   287          one that checksums IP, one that doesn't, one that checksums TCP, one
   288          that does both, etc. That would be many combinations.
   289      *   Filtering inputs would become verbose. Either:
   290      *   large conditionals that need to be repeated many places:
   291          `h[FlagOffset] == SYN && h[LengthOffset:LengthOffset+2] == ...` or
   292      *   Many functions, one per field, like: `filterByFlag(myBytes, SYN)`,
   293          `filterByLength(myBytes, 20)`, `filterByNextProto(myBytes, 0x8000)`,
   294          etc.
   295      *   Using pointers allows us to combine `Layer`s with reflection. So the
   296          default `Layers` can be overridden by a `Layers` with just the TCP
   297          conection's src/dst which can be overridden by one with just a test
   298          specific TCP window size.
   299      *   It's a proven way to separate the details of a packet from the byte
   300          format as shown by scapy's success.
   301  *   Use packetgo. It's more general than parsing packets with gVisor. However:
   302      *   packetgo doesn't have optional fields so many of the above problems
   303          still apply.
   304      *   It would be yet another dependency.
   305      *   It's not as well known to engineers that are already writing gVisor
   306          code.
   307      *   It might be a good candidate for replacing the parsing of packets into
   308          `Layer`s if all that parsing turns out to be more work than parsing by
   309          packetgo and converting *that* to `Layer`. packetgo has easier to use
   310          getters for the layers. This could be done later in a way that doesn't
   311          break tests.
   312  
   313  #### `Layer` methods
   314  
   315  The `Layer` structs provide a way to partially specify an encapsulation. They
   316  also need methods for using those partially specified encapsulation, for example
   317  to marshal them to bytes or compare them. For those, each encapsulation
   318  implements the `Layer` interface:
   319  
   320  ```go
   321  // Layer is the interface that all encapsulations must implement.
   322  //
   323  // A Layer is an encapsulation in a packet, such as TCP, IPv4, IPv6, etc. A
   324  // Layer contains all the fields of the encapsulation. Each field is a pointer
   325  // and may be nil.
   326  type Layer interface {
   327      // toBytes converts the Layer into bytes. In places where the Layer's field
   328      // isn't nil, the value that is pointed to is used. When the field is nil, a
   329      // reasonable default for the Layer is used. For example, "64" for IPv4 TTL
   330      // and a calculated checksum for TCP or IP. Some layers require information
   331      // from the previous or next layers in order to compute a default, such as
   332      // TCP's checksum or Ethernet's type, so each Layer has a doubly-linked list
   333      // to the layer's neighbors.
   334      toBytes() ([]byte, error)
   335  
   336      // match checks if the current Layer matches the provided Layer. If either
   337      // Layer has a nil in a given field, that field is considered matching.
   338      // Otherwise, the values pointed to by the fields must match.
   339      match(Layer) bool
   340  
   341      // length in bytes of the current encapsulation
   342      length() int
   343  
   344      // next gets a pointer to the encapsulated Layer.
   345      next() Layer
   346  
   347      // prev gets a pointer to the Layer encapsulating this one.
   348      prev() Layer
   349  
   350      // setNext sets the pointer to the encapsulated Layer.
   351      setNext(Layer)
   352  
   353      // setPrev sets the pointer to the Layer encapsulating this one.
   354      setPrev(Layer)
   355  }
   356  ```
   357  
   358  The `next` and `prev` make up a link listed so that each layer can get at the
   359  information in the layer around it. This is necessary for some protocols, like
   360  TCP that needs the layer before and payload after to compute the checksum. Any
   361  sequence of `Layer` structs is valid so long as the parser and `toBytes`
   362  functions can map from type to protool number and vice-versa. When the mapping
   363  fails, an error is emitted explaining what functionality is missing. The
   364  solution is either to fix the ordering or implement the missing protocol.
   365  
   366  For each `Layer` there is also a parsing function. For example, this one is for
   367  Ethernet:
   368  
   369  ```
   370  func ParseEther(b []byte) (Layers, error)
   371  ```
   372  
   373  The parsing function converts bytes received on the wire into a `Layer`
   374  (actually `Layers`, see below) which has no `nil`s in it. By using
   375  `match(Layer)` to compare against another `Layer` that *does* have `nil`s in it,
   376  the received bytes can be partially compared. The `nil`s behave as
   377  "don't-cares".
   378  
   379  ##### Alternatives considered
   380  
   381  *   Matching against `[]byte` instead of converting to `Layer` first.
   382      *   The downside is that it precludes the use of a `cmp.Equal` one-liner to
   383          do comparisons.
   384      *   It creates confusion in the code to deal with both representations at
   385          different times. For example, is the checksum calculated on `[]byte` or
   386          `Layer` when sending? What about when checking received packets?
   387  
   388  #### `Layers`
   389  
   390  ```
   391  type Layers []Layer
   392  
   393  func (ls *Layers) match(other Layers) bool {...}
   394  func (ls *Layers) toBytes() ([]byte, error) {...}
   395  ```
   396  
   397  `Layers` is an array of `Layer`. It represents a stack of encapsulations, such
   398  as `Layers{Ether{},IPv4{},TCP{},Payload{}}`. It also has `toBytes()` and
   399  `match(Layers)`, like `Layer`. The parse functions above actually return
   400  `Layers` and not `Layer` because they know about the headers below and
   401  sequentially call each parser on the remaining, encapsulated bytes.
   402  
   403  All this leads to the ability to write concise packet processing. For example:
   404  
   405  ```go
   406  etherType := 0x8000
   407  flags = uint8(header.TCPFlagSyn|header.TCPFlagAck)
   408  toMatch := Layers{Ether{Type: &etherType}, IPv4{}, TCP{Flags: &flags}}
   409  for {
   410    recvBytes := sniffer.Recv(time.Second)
   411    if recvBytes == nil {
   412      println("Got no packet for 1 second")
   413    }
   414    gotPacket, err := ParseEther(recvBytes)
   415    if err == nil && toMatch.match(gotPacket) {
   416      println("Got a TCP/IPv4/Eth packet with SYNACK")
   417    }
   418  }
   419  ```
   420  
   421  ##### Alternatives considered
   422  
   423  *   Don't use previous and next pointers.
   424      *   Each layer may need to be able to interrogate the layers around it, like
   425          for computing the next protocol number or total length. So *some*
   426          mechanism is needed for a `Layer` to see neighboring layers.
   427      *   We could pass the entire array `Layers` to the `toBytes()` function.
   428          Passing an array to a method that includes in the array the function
   429          receiver itself seems wrong.
   430  
   431  #### `layerState`
   432  
   433  `Layers` represents the different headers of a packet but a connection includes
   434  more state. For example, a TCP connection needs to keep track of the next
   435  expected sequence number and also the next sequence number to send. This is
   436  stored in a `layerState` struct. This is the `layerState` for TCP:
   437  
   438  ```go
   439  // tcpState maintains state about a TCP connection.
   440  type tcpState struct {
   441      out, in                   TCP
   442      localSeqNum, remoteSeqNum *seqnum.Value
   443      synAck                    *TCP
   444      portPickerFD              int
   445      finSent                   bool
   446  }
   447  ```
   448  
   449  The next sequence numbers for each side of the connection are stored. `out` and
   450  `in` have defaults for the TCP header, such as the expected source and
   451  destination ports for outgoing packets and incoming packets.
   452  
   453  ##### `layerState` interface
   454  
   455  ```go
   456  // layerState stores the state of a layer of a connection.
   457  type layerState interface {
   458      // outgoing returns an outgoing layer to be sent in a frame.
   459      outgoing() Layer
   460  
   461      // incoming creates an expected Layer for comparing against a received Layer.
   462      // Because the expectation can depend on values in the received Layer, it is
   463      // an input to incoming. For example, the ACK number needs to be checked in a
   464      // TCP packet but only if the ACK flag is set in the received packet.
   465      incoming(received Layer) Layer
   466  
   467      // sent updates the layerState based on the Layer that was sent. The input is
   468      // a Layer with all prev and next pointers populated so that the entire frame
   469      // as it was sent is available.
   470      sent(sent Layer) error
   471  
   472      // received updates the layerState based on a Layer that is received. The
   473      // input is a Layer with all prev and next pointers populated so that the
   474      // entire frame as it was received is available.
   475      received(received Layer) error
   476  
   477      // close frees associated resources held by the LayerState.
   478      close() error
   479  }
   480  ```
   481  
   482  `outgoing` generates the default Layer for an outgoing packet. For TCP, this
   483  would be a `TCP` with the source and destination ports populated. Because they
   484  are static, they are stored inside the `out` member of `tcpState`. However, the
   485  sequence numbers change frequently so the outgoing sequence number is stored in
   486  the `localSeqNum` and put into the output of outgoing for each call.
   487  
   488  `incoming` does the same functions for packets that arrive but instead of
   489  generating a packet to send, it generates an expect packet for filtering packets
   490  that arrive. For example, if a `TCP` header arrives with the wrong ports, it can
   491  be ignored as belonging to a different connection. `incoming` needs the received
   492  header itself as an input because the filter may depend on the input. For
   493  example, the expected sequence number depends on the flags in the TCP header.
   494  
   495  `sent` and `received` are run for each header that is actually sent or received
   496  and used to update the internal state. `incoming` and `outgoing` should *not* be
   497  used for these purpose. For example, `incoming` is called on every packet that
   498  arrives but only packets that match ought to actually update the state.
   499  `outgoing` is called to created outgoing packets and those packets are always
   500  sent, so unlike `incoming`/`received`, there is one `outgoing` call for each
   501  `sent` call.
   502  
   503  `close` cleans up after the layerState. For example, TCP and UDP need to keep a
   504  port reserved and then release it.
   505  
   506  #### Connections
   507  
   508  Using `layerState` above, we can create connections.
   509  
   510  ```go
   511  // Connection holds a collection of layer states for maintaining a connection
   512  // along with sockets for sniffer and injecting packets.
   513  type Connection struct {
   514      layerStates []layerState
   515      injector    Injector
   516      sniffer     Sniffer
   517      t           *testing.T
   518  }
   519  ```
   520  
   521  The connection stores an array of `layerState` in the order that the headers
   522  should be present in the frame to send. For example, Ether then IPv4 then TCP.
   523  The injector and sniffer are for writing and reading frames. A `*testing.T` is
   524  stored so that internal errors can be reported directly without code in the unit
   525  test.
   526  
   527  The `Connection` has some useful functions:
   528  
   529  ```go
   530  // Close frees associated resources held by the Connection.
   531  func (conn *Connection) Close() {...}
   532  // CreateFrame builds a frame for the connection with layer overriding defaults
   533  // of the innermost layer and additionalLayers added after it.
   534  func (conn *Connection) CreateFrame(layer Layer, additionalLayers ...Layer) Layers {...}
   535  // SendFrame sends a frame on the wire and updates the state of all layers.
   536  func (conn *Connection) SendFrame(frame Layers) {...}
   537  // Send a packet with reasonable defaults. Potentially override the final layer
   538  // in the connection with the provided layer and add additionLayers.
   539  func (conn *Connection) Send(layer Layer, additionalLayers ...Layer) {...}
   540  // Expect a frame with the final layerStates layer matching the provided Layer
   541  // within the timeout specified. If it doesn't arrive in time, it returns nil.
   542  func (conn *Connection) Expect(layer Layer, timeout time.Duration) (Layer, error) {...}
   543  // ExpectFrame expects a frame that matches the provided Layers within the
   544  // timeout specified. If it doesn't arrive in time, it returns nil.
   545  func (conn *Connection) ExpectFrame(layers Layers, timeout time.Duration) (Layers, error) {...}
   546  // Drain drains the sniffer's receive buffer by receiving packets until there's
   547  // nothing else to receive.
   548  func (conn *Connection) Drain() {...}
   549  ```
   550  
   551  `CreateFrame` uses the `[]layerState` to create a frame to send. The first
   552  argument is for overriding defaults in the last header of the frame, because
   553  this is the most common need. For a TCPIPv4 connection, this would be the TCP
   554  header. Optional additionalLayers can be specified to add to the frame being
   555  created, such as a `Payload` for `TCP`.
   556  
   557  `SendFrame` sends the frame to the DUT. It is combined with `CreateFrame` to
   558  make `Send`. For unittests with basic sending needs, `Send` can be used. If more
   559  control is needed over the frame, it can be made with `CreateFrame`, modified in
   560  the unit test, and then sent with `SendFrame`.
   561  
   562  On the receiving side, there is `Expect` and `ExpectFrame`. Like with the
   563  sending side, there are two forms of each function, one for just the last header
   564  and one for the whole frame. The expect functions use the `[]layerState` to
   565  create a template for the expected incoming frame. That frame is then overridden
   566  by the values in the first argument. Finally, a loop starts sniffing packets on
   567  the wire for frames. If a matching frame is found before the timeout, it is
   568  returned without error. If not, nil is returned and the error contains text of
   569  all the received frames that didn't match. Exactly one of the outputs will be
   570  non-nil, even if no frames are received at all.
   571  
   572  `Drain` sniffs and discards all the frames that have yet to be received. A
   573  common way to write a test is:
   574  
   575  ```go
   576  conn.Drain() // Discard all outstanding frames.
   577  conn.Send(...) // Send a frame with overrides.
   578  // Now expect a frame with a certain header and fail if it doesn't arrive.
   579  if _, err := conn.Expect(...); err != nil { t.Fatal(...) }
   580  ```
   581  
   582  Or for a test where we want to check that no frame arrives:
   583  
   584  ```go
   585  if gotOne, _ := conn.Expect(...); gotOne != nil { t.Fatal(...) }
   586  ```
   587  
   588  #### Specializing `Connection`
   589  
   590  Because there are some common combinations of `layerState` into `Connection`,
   591  they are defined:
   592  
   593  ```go
   594  // TCPIPv4 maintains the state for all the layers in a TCP/IPv4 connection.
   595  type TCPIPv4 Connection
   596  // UDPIPv4 maintains the state for all the layers in a UDP/IPv4 connection.
   597  type UDPIPv4 Connection
   598  ```
   599  
   600  Each has a `NewXxx` function to create a new connection with reasonable
   601  defaults. They also have functions that call the underlying `Connection`
   602  functions but with specialization and tighter type-checking. For example:
   603  
   604  ```go
   605  func (conn *TCPIPv4) Send(tcp TCP, additionalLayers ...Layer) {
   606      (*Connection)(conn).Send(&tcp, additionalLayers...)
   607  }
   608  func (conn *TCPIPv4) Drain() {
   609      conn.sniffer.Drain()
   610  }
   611  ```
   612  
   613  They may also have some accessors to get or set the internal state of the
   614  connection:
   615  
   616  ```go
   617  func (conn *TCPIPv4) state() *tcpState {
   618      state, ok := conn.layerStates[len(conn.layerStates)-1].(*tcpState)
   619      if !ok {
   620          conn.t.Fatalf("expected final state of %v to be tcpState", conn.layerStates)
   621      }
   622      return state
   623  }
   624  func (conn *TCPIPv4) RemoteSeqNum() *seqnum.Value {
   625      return conn.state().remoteSeqNum
   626  }
   627  func (conn *TCPIPv4) LocalSeqNum() *seqnum.Value {
   628      return conn.state().localSeqNum
   629  }
   630  ```
   631  
   632  Unittests will in practice use these functions and not the functions on
   633  `Connection`. For example, `NewTCPIPv4()` and then call `Send` on that rather
   634  than cast is to a `Connection` and call `Send` on that cast result.
   635  
   636  ##### Alternatives considered
   637  
   638  *   Instead of storing `outgoing` and `incoming`, store values.
   639      *   There would be many more things to store instead, like `localMac`,
   640          `remoteMac`, `localIP`, `remoteIP`, `localPort`, and `remotePort`.
   641      *   Construction of a packet would be many lines to copy each of these
   642          values into a `[]byte`. And there would be slight variations needed for
   643          each encapsulation stack, like TCPIPv6 and ARP.
   644      *   Filtering incoming packets would be a long sequence:
   645      *   Compare the MACs, then
   646      *   Parse the next header, then
   647      *   Compare the IPs, then
   648      *   Parse the next header, then
   649      *   Compare the TCP ports. Instead it's all just one call to
   650          `cmp.Equal(...)`, for all sequences.
   651      *   A TCPIPv6 connection could share most of the code. Only the type of the
   652          IP addresses are different. The types of `outgoing` and `incoming` would
   653          be remain `Layers`.
   654      *   An ARP connection could share all the Ethernet parts. The IP `Layer`
   655          could be factored out of `outgoing`. After that, the IPv4 and IPv6
   656          connections could implement one interface and a single TCP struct could
   657          have either network protocol through composition.
   658  
   659  ## Putting it all together
   660  
   661  Here's what te start of a packetimpact unit test looks like. This test creates a
   662  TCP connection with the DUT. There are added comments for explanation in this
   663  document but a real test might not include them in order to stay even more
   664  concise.
   665  
   666  ```go
   667  func TestMyTcpTest(t *testing.T) {
   668    // Prepare a DUT for communication.
   669    dut := testbench.NewDUT(t)
   670  
   671    // This does:
   672    //   dut.Socket()
   673    //   dut.Bind()
   674    //   dut.Getsockname() to learn the new port number
   675    //   dut.Listen()
   676    listenFD, remotePort := dut.CreateListener(unix.SOCK_STREAM, unix.IPPROTO_TCP, 1)
   677    defer dut.Close(listenFD) // Tell the DUT to close the socket at the end of the test.
   678  
   679    // Monitor a new TCP connection with sniffer, injector, sequence number tracking,
   680    // and reasonable outgoing and incoming packet field default IPs, MACs, and port numbers.
   681    conn := testbench.NewTCPIPv4(t, dut, remotePort)
   682  
   683    // Perform a 3-way handshake: send SYN, expect SYNACK, send ACK.
   684    conn.Handshake()
   685  
   686    // Tell the DUT to accept the new connection.
   687    acceptFD := dut.Accept(acceptFd)
   688  }
   689  ```
   690  
   691  ### Adding a new packetimpact test
   692  
   693  *   Create a go test in the [tests directory](tests/)
   694  *   Add a `packetimpact_testbench` rule in [BUILD](tests/BUILD)
   695  *   Add the test into the `ALL_TESTS` list in [defs.bzl](runner/defs.bzl),
   696      otherwise you will see an error message complaining about a missing test.
   697  
   698  ## Other notes
   699  
   700  *   The time between receiving a SYN-ACK and replying with an ACK in `Handshake`
   701      is about 3ms. This is much slower than the native unix response, which is
   702      about 0.3ms. Packetdrill gets closer to 0.3ms. For tests where timing is
   703      crucial, packetdrill is faster and more precise.