github.com/SagerNet/gvisor@v0.0.0-20210707092255-7731c139d75c/test/packetimpact/README.md (about)

     1  # Packetimpact
     2  
     3  ## What is packetimpact?
     4  
     5  Packetimpact is a tool for platform-independent network testing. It is heavily
     6  inspired by [packetdrill](https://github.com/google/packetdrill). It creates two
     7  docker containers connected by a network. One is for the test bench, which
     8  operates the test. The other is for the device-under-test (DUT), which is the
     9  software being tested. The test bench communicates over the network with the DUT
    10  to check correctness of the network.
    11  
    12  ### Goals
    13  
    14  Packetimpact aims to provide:
    15  
    16  *   A **multi-platform** solution that can test both Linux and gVisor.
    17  *   **Conciseness** on par with packetdrill scripts.
    18  *   **Control-flow** like for loops, conditionals, and variables.
    19  *   **Flexibilty** to specify every byte in a packet or use multiple sockets.
    20  
    21  ## How to run packetimpact tests?
    22  
    23  Build the test container image by running the following at the root of the
    24  repository:
    25  
    26  ```bash
    27  $ make load-packetimpact
    28  ```
    29  
    30  Run a test, e.g. `fin_wait2_timeout`, against Linux:
    31  
    32  ```bash
    33  $ bazel test //test/packetimpact/tests:fin_wait2_timeout_native_test
    34  ```
    35  
    36  Run the same test, but against gVisor:
    37  
    38  ```bash
    39  $ bazel test //test/packetimpact/tests:fin_wait2_timeout_netstack_test
    40  ```
    41  
    42  ## When to use packetimpact?
    43  
    44  There are a few ways to write networking tests for gVisor currently:
    45  
    46  *   [Go unit tests](https://github.com/google/gvisor/tree/master/pkg/tcpip)
    47  *   [syscall tests](https://github.com/google/gvisor/tree/master/test/syscalls/linux)
    48  *   [packetdrill tests](https://github.com/google/gvisor/tree/master/test/packetdrill)
    49  *   packetimpact tests
    50  
    51  The right choice depends on the needs of the test.
    52  
    53  Feature        | Go unit test | syscall test | packetdrill | packetimpact
    54  -------------- | ------------ | ------------ | ----------- | ------------
    55  Multi-platform | no           | **YES**      | **YES**     | **YES**
    56  Concise        | no           | somewhat     | somewhat    | **VERY**
    57  Control-flow   | **YES**      | **YES**      | no          | **YES**
    58  Flexible       | **VERY**     | no           | somewhat    | **VERY**
    59  
    60  ### Go unit tests
    61  
    62  If the test depends on the internals of gVisor and doesn't need to run on Linux
    63  or other platforms for comparison purposes, a Go unit test can be appropriate.
    64  They can observe internals of gVisor networking. The downside is that they are
    65  **not concise** and **not multi-platform**. If you require insight on gVisor
    66  internals, this is the right choice.
    67  
    68  ### Syscall tests
    69  
    70  Syscall tests are **multi-platform** but cannot examine the internals of gVisor
    71  networking. They are **concise**. They can use **control-flow** structures like
    72  conditionals, for loops, and variables. However, they are limited to only what
    73  the POSIX interface provides so they are **not flexible**. For example, you
    74  would have difficulty writing a syscall test that intentionally sends a bad IP
    75  checksum. Or if you did write that test with raw sockets, it would be very
    76  **verbose** to write a test that intentionally send wrong checksums, wrong
    77  protocols, wrong sequence numbers, etc.
    78  
    79  ### Packetdrill tests
    80  
    81  Packetdrill tests are **multi-platform** and can run against both Linux and
    82  gVisor. They are **concise** and use a special packetdrill scripting language.
    83  They are **more flexible** than a syscall test in that they can send packets
    84  that a syscall test would have difficulty sending, like a packet with a
    85  calcuated ACK number. But they are also somewhat limimted in flexibiilty in that
    86  they can't do tests with multiple sockets. They have **no control-flow** ability
    87  like variables or conditionals. For example, it isn't possible to send a packet
    88  that depends on the window size of a previous packet because the packetdrill
    89  language can't express that. Nor could you branch based on whether or not the
    90  other side supports window scaling, for example.
    91  
    92  ### Packetimpact tests
    93  
    94  Packetimpact tests are similar to Packetdrill tests except that they are written
    95  in Go instead of the packetdrill scripting language. That gives them all the
    96  **control-flow** abilities of Go (loops, functions, variables, etc). They are
    97  **multi-platform** in the same way as packetdrill tests but even more
    98  **flexible** because Go is more expressive than the scripting language of
    99  packetdrill. However, Go is **not as concise** as the packetdrill language. Many
   100  design decisions below are made to mitigate that.
   101  
   102  ## How it works
   103  
   104  ```
   105       Testbench                           Device-Under-Test (DUT)
   106      +-------------------+               +------------------------+
   107      |                   |   TEST NET    |                        |
   108      | rawsockets.go <-->| <===========> | <---+                  |
   109      |           ^       |               |     |                  |
   110      |           |       |               |     |                  |
   111      |           v       |               |     |                  |
   112      |     unittest      |               |     |                  |
   113      |           ^       |               |     |                  |
   114      |           |       |               |     |                  |
   115      |           v       |               |     v                  |
   116      |         dut.go <========gRPC========> posix server         |
   117      |                   |  CONTROL NET  |                        |
   118      +-------------------+               +------------------------+
   119  ```
   120  
   121  Two docker containers are created by a "runner" script, one for the testbench
   122  and the other for the device under test (DUT). The script connects the two
   123  containers with a control network and test network. It also does some other
   124  tasks like waiting until the DUT is ready before starting the test and disabling
   125  Linux networking that would interfere with the test bench.
   126  
   127  ### DUT
   128  
   129  The DUT container runs a program called the "posix_server". The posix_server is
   130  written in c++ for maximum portability. It is compiled on the host. The script
   131  that starts the containers copies it into the DUT's container and runs it. It's
   132  job is to receive directions from the test bench on what actions to take. For
   133  this, the posix_server does three steps in a loop:
   134  
   135  1.  Listen for a request from the test bench.
   136  2.  Execute a command.
   137  3.  Send the response back to the test bench.
   138  
   139  The requests and responses are
   140  [protobufs](https://developers.google.com/protocol-buffers) and the
   141  communication is done with [gRPC](https://grpc.io/). The commands run are
   142  [POSIX socket commands](https://en.wikipedia.org/wiki/Berkeley_sockets#Socket_API_functions),
   143  with the inputs and outputs converted into protobuf requests and responses. All
   144  communication is on the control network, so that the test network is unaffected
   145  by extra packets.
   146  
   147  For example, this is the request and response pair to call
   148  [`socket()`](http://man7.org/linux/man-pages/man2/socket.2.html):
   149  
   150  ```protocol-buffer
   151  message SocketRequest {
   152    int32 domain = 1;
   153    int32 type = 2;
   154    int32 protocol = 3;
   155  }
   156  
   157  message SocketResponse {
   158    int32 fd = 1;
   159    int32 errno_ = 2;
   160  }
   161  ```
   162  
   163  ##### Alternatives considered
   164  
   165  *   We could have use JSON for communication instead. It would have been a
   166      lighter-touch than protobuf but protobuf handles all the data type and has
   167      strict typing to prevent a class of errors. The test bench could be written
   168      in other languages, too.
   169  *   Instead of mimicking the POSIX interfaces, arguments could have had a more
   170      natural form, like the `bind()` getting a string IP address instead of bytes
   171      in a `sockaddr_t`. However, conforming to the existing structures keeps more
   172      of the complexity in Go and keeps the posix_server simpler and thus more
   173      likely to compile everywhere.
   174  
   175  ### Test Bench
   176  
   177  The test bench does most of the work in a test. It is a Go program that compiles
   178  on the host and is copied by the script into test bench's container. It is a
   179  regular [go unit test](https://golang.org/pkg/testing/) that imports the test
   180  bench framework. The test bench framwork is based on three basic utilities:
   181  
   182  *   Commanding the DUT to run POSIX commands and return responses.
   183  *   Sending raw packets to the DUT on the test network.
   184  *   Listening for raw packets from the DUT on the test network.
   185  
   186  #### DUT commands
   187  
   188  To keep the interface to the DUT consistent and easy-to-use, each POSIX command
   189  supported by the posix_server is wrapped in functions with signatures similar to
   190  the ones in the [Go unix package](https://godoc.org/golang.org/x/sys/unix). This
   191  way all the details of endianess and (un)marshalling of go structs such as
   192  [unix.Timeval](https://godoc.org/golang.org/x/sys/unix#Timeval) is handled in
   193  one place. This also makes it straight-forward to convert tests that use `unix.`
   194  or `syscall.` calls to `dut.` calls.
   195  
   196  For example, creating a connection to the DUT and commanding it to make a socket
   197  looks like this:
   198  
   199  ```go
   200  dut := testbench.NewDut(t)
   201  fd, err := dut.SocketWithErrno(unix.AF_INET, unix.SOCK_STREAM, unix.IPPROTO_IP)
   202  if fd < 0 {
   203    t.Fatalf(...)
   204  }
   205  ```
   206  
   207  Because the usual case is to fail the test when the DUT fails to create a
   208  socket, there is a concise version of each of the `...WithErrno` functions that
   209  does that:
   210  
   211  ```go
   212  dut := testbench.NewDut(t)
   213  fd := dut.Socket(unix.AF_INET, unix.SOCK_STREAM, unix.IPPROTO_IP)
   214  ```
   215  
   216  The DUT and other structs in the code store a `*testing.T` so that they can
   217  provide versions of functions that call `t.Fatalf(...)`. This helps keep tests
   218  concise.
   219  
   220  ##### Alternatives considered
   221  
   222  *   Instead of mimicking the `unix.` go interface, we could have invented a more
   223      natural one, like using `float64` instead of `Timeval`. However, using the
   224      same function signatures that `unix.` has makes it easier to convert code to
   225      `dut.`. Also, using an existing interface ensures that we don't invent an
   226      interface that isn't extensible. For example, if we invented a function for
   227      `bind()` that didn't support IPv6 and later we had to add a second `bind6()`
   228      function.
   229  
   230  #### Sending/Receiving Raw Packets
   231  
   232  The framework wraps POSIX sockets for sending and receiving raw frames. Both
   233  send and receive are synchronous commands.
   234  [SO_RCVTIMEO](http://man7.org/linux/man-pages/man7/socket.7.html) is used to set
   235  a timeout on the receive commands. For ease of use, these are wrapped in an
   236  `Injector` and a `Sniffer`. They have functions:
   237  
   238  ```go
   239  func (s *Sniffer) Recv(timeout time.Duration) []byte {...}
   240  func (i *Injector) Send(b []byte) {...}
   241  ```
   242  
   243  ##### Alternatives considered
   244  
   245  *   [gopacket](https://github.com/google/gopacket) pcap has raw socket support
   246      but requires cgo. cgo is not guaranteed to be portable from the host to the
   247      container and in practice, the container doesn't recognize binaries built on
   248      the host if they use cgo.
   249  *   Both gVisor and gopacket have the ability to read and write pcap files
   250      without cgo but that is insufficient here because we can't just replay pcap
   251      files, we need a more dynamic solution.
   252  *   The sniffer and injector can't share a socket because they need to be bound
   253      differently.
   254  *   Sniffing could have been done asynchronously with channels, obviating the
   255      need for `SO_RCVTIMEO`. But that would introduce asynchronous complication.
   256      `SO_RCVTIMEO` is well supported on the test bench.
   257  
   258  #### `Layer` struct
   259  
   260  A large part of packetimpact tests is creating packets to send and comparing
   261  received packets against expectations. To keep tests concise, it is useful to be
   262  able to specify just the important parts of packets that need to be set. For
   263  example, sending a packet with default values except for TCP Flags. And for
   264  packets received, it's useful to be able to compare just the necessary parts of
   265  received packets and ignore the rest.
   266  
   267  To aid in both of those, Go structs with optional fields are created for each
   268  encapsulation type, such as IPv4, TCP, and Ethernet. This is inspired by
   269  [scapy](https://scapy.readthedocs.io/en/latest/). For example, here is the
   270  struct for Ethernet:
   271  
   272  ```go
   273  type Ether struct {
   274    LayerBase
   275    SrcAddr *tcpip.LinkAddress
   276    DstAddr *tcpip.LinkAddress
   277    Type    *tcpip.NetworkProtocolNumber
   278  }
   279  ```
   280  
   281  Each struct has the same fields as those in the
   282  [gVisor headers](https://github.com/google/gvisor/tree/master/pkg/tcpip/header)
   283  but with a pointer for each field that may be `nil`.
   284  
   285  ##### Alternatives considered
   286  
   287  *   Just use []byte like gVisor headers do. The drawback is that it makes the
   288      tests more verbose.
   289      *   For example, there would be no way to call `Send(myBytes)` concisely and
   290          indicate if the checksum should be calculated automatically versus
   291          overridden. The only way would be to add lines to the test to calculate
   292          it before each Send, which is wordy. Or make multiple versions of Send:
   293          one that checksums IP, one that doesn't, one that checksums TCP, one
   294          that does both, etc. That would be many combinations.
   295      *   Filtering inputs would become verbose. Either:
   296      *   large conditionals that need to be repeated many places:
   297          `h[FlagOffset] == SYN && h[LengthOffset:LengthOffset+2] == ...` or
   298      *   Many functions, one per field, like: `filterByFlag(myBytes, SYN)`,
   299          `filterByLength(myBytes, 20)`, `filterByNextProto(myBytes, 0x8000)`,
   300          etc.
   301      *   Using pointers allows us to combine `Layer`s with reflection. So the
   302          default `Layers` can be overridden by a `Layers` with just the TCP
   303          conection's src/dst which can be overridden by one with just a test
   304          specific TCP window size.
   305      *   It's a proven way to separate the details of a packet from the byte
   306          format as shown by scapy's success.
   307  *   Use packetgo. It's more general than parsing packets with gVisor. However:
   308      *   packetgo doesn't have optional fields so many of the above problems
   309          still apply.
   310      *   It would be yet another dependency.
   311      *   It's not as well known to engineers that are already writing gVisor
   312          code.
   313      *   It might be a good candidate for replacing the parsing of packets into
   314          `Layer`s if all that parsing turns out to be more work than parsing by
   315          packetgo and converting *that* to `Layer`. packetgo has easier to use
   316          getters for the layers. This could be done later in a way that doesn't
   317          break tests.
   318  
   319  #### `Layer` methods
   320  
   321  The `Layer` structs provide a way to partially specify an encapsulation. They
   322  also need methods for using those partially specified encapsulation, for example
   323  to marshal them to bytes or compare them. For those, each encapsulation
   324  implements the `Layer` interface:
   325  
   326  ```go
   327  // Layer is the interface that all encapsulations must implement.
   328  //
   329  // A Layer is an encapsulation in a packet, such as TCP, IPv4, IPv6, etc. A
   330  // Layer contains all the fields of the encapsulation. Each field is a pointer
   331  // and may be nil.
   332  type Layer interface {
   333      // toBytes converts the Layer into bytes. In places where the Layer's field
   334      // isn't nil, the value that is pointed to is used. When the field is nil, a
   335      // reasonable default for the Layer is used. For example, "64" for IPv4 TTL
   336      // and a calculated checksum for TCP or IP. Some layers require information
   337      // from the previous or next layers in order to compute a default, such as
   338      // TCP's checksum or Ethernet's type, so each Layer has a doubly-linked list
   339      // to the layer's neighbors.
   340      toBytes() ([]byte, error)
   341  
   342      // match checks if the current Layer matches the provided Layer. If either
   343      // Layer has a nil in a given field, that field is considered matching.
   344      // Otherwise, the values pointed to by the fields must match.
   345      match(Layer) bool
   346  
   347      // length in bytes of the current encapsulation
   348      length() int
   349  
   350      // next gets a pointer to the encapsulated Layer.
   351      next() Layer
   352  
   353      // prev gets a pointer to the Layer encapsulating this one.
   354      prev() Layer
   355  
   356      // setNext sets the pointer to the encapsulated Layer.
   357      setNext(Layer)
   358  
   359      // setPrev sets the pointer to the Layer encapsulating this one.
   360      setPrev(Layer)
   361  }
   362  ```
   363  
   364  The `next` and `prev` make up a link listed so that each layer can get at the
   365  information in the layer around it. This is necessary for some protocols, like
   366  TCP that needs the layer before and payload after to compute the checksum. Any
   367  sequence of `Layer` structs is valid so long as the parser and `toBytes`
   368  functions can map from type to protool number and vice-versa. When the mapping
   369  fails, an error is emitted explaining what functionality is missing. The
   370  solution is either to fix the ordering or implement the missing protocol.
   371  
   372  For each `Layer` there is also a parsing function. For example, this one is for
   373  Ethernet:
   374  
   375  ```
   376  func ParseEther(b []byte) (Layers, error)
   377  ```
   378  
   379  The parsing function converts bytes received on the wire into a `Layer`
   380  (actually `Layers`, see below) which has no `nil`s in it. By using
   381  `match(Layer)` to compare against another `Layer` that *does* have `nil`s in it,
   382  the received bytes can be partially compared. The `nil`s behave as
   383  "don't-cares".
   384  
   385  ##### Alternatives considered
   386  
   387  *   Matching against `[]byte` instead of converting to `Layer` first.
   388      *   The downside is that it precludes the use of a `cmp.Equal` one-liner to
   389          do comparisons.
   390      *   It creates confusion in the code to deal with both representations at
   391          different times. For example, is the checksum calculated on `[]byte` or
   392          `Layer` when sending? What about when checking received packets?
   393  
   394  #### `Layers`
   395  
   396  ```
   397  type Layers []Layer
   398  
   399  func (ls *Layers) match(other Layers) bool {...}
   400  func (ls *Layers) toBytes() ([]byte, error) {...}
   401  ```
   402  
   403  `Layers` is an array of `Layer`. It represents a stack of encapsulations, such
   404  as `Layers{Ether{},IPv4{},TCP{},Payload{}}`. It also has `toBytes()` and
   405  `match(Layers)`, like `Layer`. The parse functions above actually return
   406  `Layers` and not `Layer` because they know about the headers below and
   407  sequentially call each parser on the remaining, encapsulated bytes.
   408  
   409  All this leads to the ability to write concise packet processing. For example:
   410  
   411  ```go
   412  etherType := 0x8000
   413  flags = uint8(header.TCPFlagSyn|header.TCPFlagAck)
   414  toMatch := Layers{Ether{Type: &etherType}, IPv4{}, TCP{Flags: &flags}}
   415  for {
   416    recvBytes := sniffer.Recv(time.Second)
   417    if recvBytes == nil {
   418      println("Got no packet for 1 second")
   419    }
   420    gotPacket, err := ParseEther(recvBytes)
   421    if err == nil && toMatch.match(gotPacket) {
   422      println("Got a TCP/IPv4/Eth packet with SYNACK")
   423    }
   424  }
   425  ```
   426  
   427  ##### Alternatives considered
   428  
   429  *   Don't use previous and next pointers.
   430      *   Each layer may need to be able to interrogate the layers around it, like
   431          for computing the next protocol number or total length. So *some*
   432          mechanism is needed for a `Layer` to see neighboring layers.
   433      *   We could pass the entire array `Layers` to the `toBytes()` function.
   434          Passing an array to a method that includes in the array the function
   435          receiver itself seems wrong.
   436  
   437  #### `layerState`
   438  
   439  `Layers` represents the different headers of a packet but a connection includes
   440  more state. For example, a TCP connection needs to keep track of the next
   441  expected sequence number and also the next sequence number to send. This is
   442  stored in a `layerState` struct. This is the `layerState` for TCP:
   443  
   444  ```go
   445  // tcpState maintains state about a TCP connection.
   446  type tcpState struct {
   447      out, in                   TCP
   448      localSeqNum, remoteSeqNum *seqnum.Value
   449      synAck                    *TCP
   450      portPickerFD              int
   451      finSent                   bool
   452  }
   453  ```
   454  
   455  The next sequence numbers for each side of the connection are stored. `out` and
   456  `in` have defaults for the TCP header, such as the expected source and
   457  destination ports for outgoing packets and incoming packets.
   458  
   459  ##### `layerState` interface
   460  
   461  ```go
   462  // layerState stores the state of a layer of a connection.
   463  type layerState interface {
   464      // outgoing returns an outgoing layer to be sent in a frame.
   465      outgoing() Layer
   466  
   467      // incoming creates an expected Layer for comparing against a received Layer.
   468      // Because the expectation can depend on values in the received Layer, it is
   469      // an input to incoming. For example, the ACK number needs to be checked in a
   470      // TCP packet but only if the ACK flag is set in the received packet.
   471      incoming(received Layer) Layer
   472  
   473      // sent updates the layerState based on the Layer that was sent. The input is
   474      // a Layer with all prev and next pointers populated so that the entire frame
   475      // as it was sent is available.
   476      sent(sent Layer) error
   477  
   478      // received updates the layerState based on a Layer that is receieved. The
   479      // input is a Layer with all prev and next pointers populated so that the
   480      // entire frame as it was receieved is available.
   481      received(received Layer) error
   482  
   483      // close frees associated resources held by the LayerState.
   484      close() error
   485  }
   486  ```
   487  
   488  `outgoing` generates the default Layer for an outgoing packet. For TCP, this
   489  would be a `TCP` with the source and destination ports populated. Because they
   490  are static, they are stored inside the `out` member of `tcpState`. However, the
   491  sequence numbers change frequently so the outgoing sequence number is stored in
   492  the `localSeqNum` and put into the output of outgoing for each call.
   493  
   494  `incoming` does the same functions for packets that arrive but instead of
   495  generating a packet to send, it generates an expect packet for filtering packets
   496  that arrive. For example, if a `TCP` header arrives with the wrong ports, it can
   497  be ignored as belonging to a different connection. `incoming` needs the received
   498  header itself as an input because the filter may depend on the input. For
   499  example, the expected sequence number depends on the flags in the TCP header.
   500  
   501  `sent` and `received` are run for each header that is actually sent or received
   502  and used to update the internal state. `incoming` and `outgoing` should *not* be
   503  used for these purpose. For example, `incoming` is called on every packet that
   504  arrives but only packets that match ought to actually update the state.
   505  `outgoing` is called to created outgoing packets and those packets are always
   506  sent, so unlike `incoming`/`received`, there is one `outgoing` call for each
   507  `sent` call.
   508  
   509  `close` cleans up after the layerState. For example, TCP and UDP need to keep a
   510  port reserved and then release it.
   511  
   512  #### Connections
   513  
   514  Using `layerState` above, we can create connections.
   515  
   516  ```go
   517  // Connection holds a collection of layer states for maintaining a connection
   518  // along with sockets for sniffer and injecting packets.
   519  type Connection struct {
   520      layerStates []layerState
   521      injector    Injector
   522      sniffer     Sniffer
   523      t           *testing.T
   524  }
   525  ```
   526  
   527  The connection stores an array of `layerState` in the order that the headers
   528  should be present in the frame to send. For example, Ether then IPv4 then TCP.
   529  The injector and sniffer are for writing and reading frames. A `*testing.T` is
   530  stored so that internal errors can be reported directly without code in the unit
   531  test.
   532  
   533  The `Connection` has some useful functions:
   534  
   535  ```go
   536  // Close frees associated resources held by the Connection.
   537  func (conn *Connection) Close() {...}
   538  // CreateFrame builds a frame for the connection with layer overriding defaults
   539  // of the innermost layer and additionalLayers added after it.
   540  func (conn *Connection) CreateFrame(layer Layer, additionalLayers ...Layer) Layers {...}
   541  // SendFrame sends a frame on the wire and updates the state of all layers.
   542  func (conn *Connection) SendFrame(frame Layers) {...}
   543  // Send a packet with reasonable defaults. Potentially override the final layer
   544  // in the connection with the provided layer and add additionLayers.
   545  func (conn *Connection) Send(layer Layer, additionalLayers ...Layer) {...}
   546  // Expect a frame with the final layerStates layer matching the provided Layer
   547  // within the timeout specified. If it doesn't arrive in time, it returns nil.
   548  func (conn *Connection) Expect(layer Layer, timeout time.Duration) (Layer, error) {...}
   549  // ExpectFrame expects a frame that matches the provided Layers within the
   550  // timeout specified. If it doesn't arrive in time, it returns nil.
   551  func (conn *Connection) ExpectFrame(layers Layers, timeout time.Duration) (Layers, error) {...}
   552  // Drain drains the sniffer's receive buffer by receiving packets until there's
   553  // nothing else to receive.
   554  func (conn *Connection) Drain() {...}
   555  ```
   556  
   557  `CreateFrame` uses the `[]layerState` to create a frame to send. The first
   558  argument is for overriding defaults in the last header of the frame, because
   559  this is the most common need. For a TCPIPv4 connection, this would be the TCP
   560  header. Optional additionalLayers can be specified to add to the frame being
   561  created, such as a `Payload` for `TCP`.
   562  
   563  `SendFrame` sends the frame to the DUT. It is combined with `CreateFrame` to
   564  make `Send`. For unittests with basic sending needs, `Send` can be used. If more
   565  control is needed over the frame, it can be made with `CreateFrame`, modified in
   566  the unit test, and then sent with `SendFrame`.
   567  
   568  On the receiving side, there is `Expect` and `ExpectFrame`. Like with the
   569  sending side, there are two forms of each function, one for just the last header
   570  and one for the whole frame. The expect functions use the `[]layerState` to
   571  create a template for the expected incoming frame. That frame is then overridden
   572  by the values in the first argument. Finally, a loop starts sniffing packets on
   573  the wire for frames. If a matching frame is found before the timeout, it is
   574  returned without error. If not, nil is returned and the error contains text of
   575  all the received frames that didn't match. Exactly one of the outputs will be
   576  non-nil, even if no frames are received at all.
   577  
   578  `Drain` sniffs and discards all the frames that have yet to be received. A
   579  common way to write a test is:
   580  
   581  ```go
   582  conn.Drain() // Discard all outstanding frames.
   583  conn.Send(...) // Send a frame with overrides.
   584  // Now expect a frame with a certain header and fail if it doesn't arrive.
   585  if _, err := conn.Expect(...); err != nil { t.Fatal(...) }
   586  ```
   587  
   588  Or for a test where we want to check that no frame arrives:
   589  
   590  ```go
   591  if gotOne, _ := conn.Expect(...); gotOne != nil { t.Fatal(...) }
   592  ```
   593  
   594  #### Specializing `Connection`
   595  
   596  Because there are some common combinations of `layerState` into `Connection`,
   597  they are defined:
   598  
   599  ```go
   600  // TCPIPv4 maintains the state for all the layers in a TCP/IPv4 connection.
   601  type TCPIPv4 Connection
   602  // UDPIPv4 maintains the state for all the layers in a UDP/IPv4 connection.
   603  type UDPIPv4 Connection
   604  ```
   605  
   606  Each has a `NewXxx` function to create a new connection with reasonable
   607  defaults. They also have functions that call the underlying `Connection`
   608  functions but with specialization and tighter type-checking. For example:
   609  
   610  ```go
   611  func (conn *TCPIPv4) Send(tcp TCP, additionalLayers ...Layer) {
   612      (*Connection)(conn).Send(&tcp, additionalLayers...)
   613  }
   614  func (conn *TCPIPv4) Drain() {
   615      conn.sniffer.Drain()
   616  }
   617  ```
   618  
   619  They may also have some accessors to get or set the internal state of the
   620  connection:
   621  
   622  ```go
   623  func (conn *TCPIPv4) state() *tcpState {
   624      state, ok := conn.layerStates[len(conn.layerStates)-1].(*tcpState)
   625      if !ok {
   626          conn.t.Fatalf("expected final state of %v to be tcpState", conn.layerStates)
   627      }
   628      return state
   629  }
   630  func (conn *TCPIPv4) RemoteSeqNum() *seqnum.Value {
   631      return conn.state().remoteSeqNum
   632  }
   633  func (conn *TCPIPv4) LocalSeqNum() *seqnum.Value {
   634      return conn.state().localSeqNum
   635  }
   636  ```
   637  
   638  Unittests will in practice use these functions and not the functions on
   639  `Connection`. For example, `NewTCPIPv4()` and then call `Send` on that rather
   640  than cast is to a `Connection` and call `Send` on that cast result.
   641  
   642  ##### Alternatives considered
   643  
   644  *   Instead of storing `outgoing` and `incoming`, store values.
   645      *   There would be many more things to store instead, like `localMac`,
   646          `remoteMac`, `localIP`, `remoteIP`, `localPort`, and `remotePort`.
   647      *   Construction of a packet would be many lines to copy each of these
   648          values into a `[]byte`. And there would be slight variations needed for
   649          each encapsulation stack, like TCPIPv6 and ARP.
   650      *   Filtering incoming packets would be a long sequence:
   651      *   Compare the MACs, then
   652      *   Parse the next header, then
   653      *   Compare the IPs, then
   654      *   Parse the next header, then
   655      *   Compare the TCP ports. Instead it's all just one call to
   656          `cmp.Equal(...)`, for all sequences.
   657      *   A TCPIPv6 connection could share most of the code. Only the type of the
   658          IP addresses are different. The types of `outgoing` and `incoming` would
   659          be remain `Layers`.
   660      *   An ARP connection could share all the Ethernet parts. The IP `Layer`
   661          could be factored out of `outgoing`. After that, the IPv4 and IPv6
   662          connections could implement one interface and a single TCP struct could
   663          have either network protocol through composition.
   664  
   665  ## Putting it all together
   666  
   667  Here's what te start of a packetimpact unit test looks like. This test creates a
   668  TCP connection with the DUT. There are added comments for explanation in this
   669  document but a real test might not include them in order to stay even more
   670  concise.
   671  
   672  ```go
   673  func TestMyTcpTest(t *testing.T) {
   674    // Prepare a DUT for communication.
   675    dut := testbench.NewDUT(t)
   676  
   677    // This does:
   678    //   dut.Socket()
   679    //   dut.Bind()
   680    //   dut.Getsockname() to learn the new port number
   681    //   dut.Listen()
   682    listenFD, remotePort := dut.CreateListener(unix.SOCK_STREAM, unix.IPPROTO_TCP, 1)
   683    defer dut.Close(listenFD) // Tell the DUT to close the socket at the end of the test.
   684  
   685    // Monitor a new TCP connection with sniffer, injector, sequence number tracking,
   686    // and reasonable outgoing and incoming packet field default IPs, MACs, and port numbers.
   687    conn := testbench.NewTCPIPv4(t, dut, remotePort)
   688  
   689    // Perform a 3-way handshake: send SYN, expect SYNACK, send ACK.
   690    conn.Handshake()
   691  
   692    // Tell the DUT to accept the new connection.
   693    acceptFD := dut.Accept(acceptFd)
   694  }
   695  ```
   696  
   697  ### Adding a new packetimpact test
   698  
   699  *   Create a go test in the [tests directory](tests/)
   700  *   Add a `packetimpact_testbench` rule in [BUILD](tests/BUILD)
   701  *   Add the test into the `ALL_TESTS` list in [defs.bzl](runner/defs.bzl),
   702      otherwise you will see an error message complaining about a missing test.
   703  
   704  ## Other notes
   705  
   706  *   The time between receiving a SYN-ACK and replying with an ACK in `Handshake`
   707      is about 3ms. This is much slower than the native unix response, which is
   708      about 0.3ms. Packetdrill gets closer to 0.3ms. For tests where timing is
   709      crucial, packetdrill is faster and more precise.