github.com/coreos/mantle@v0.13.0/kola/README.md (about)

     1  # Adding Tests
     2  
     3  ## Quick Start
     4  
     5  1. Fork and clone the [`mantle` repository](https://github.com/coreos/mantle/)
     6  2. Move into `kola/tests/` and look for the package your test would best fit
     7  3. Edit the file and add your test(s), ensuring that you register your new test(s) in the packages `init()`
     8  4. Commit, push, and PR your result
     9  
    10  ### Example
    11  
    12  Say we wanted to add a simple [noop](https://en.wikipedia.org/wiki/NOP_(code)) test in the `podman` test package. If we follow the above instructions it would look like this:
    13  
    14  ```
    15  $ git clone git@github.com:$GITHUB_USERNAME/mantle.git
    16  <snip/>
    17  $ pushd kola/tests/
    18  $ $EDITOR podman/podman.go  # Add the test
    19  // Test: I'm a NOOP!
    20  func podmanNOOP(c cluster.TestCluster) {
    21      // NOOP!
    22  }
    23  $ $EDITOR podman/podman.go # Register the test in the init
    24  func init() {
    25      register.Register(&register.Test{
    26          Run:         podmanNOOP,
    27          ClusterSize: 1,
    28          Name:        `podman.noop`,
    29          Distros:     []string{"rhcos"},
    30      })
    31  <snip/>
    32  $ popd
    33  $ ./build kola
    34  # Check and ensure the test is there
    35  $ ./bin/kola list | grep podman
    36  podman.base                                     [all]                                   [all]   [rhcos]
    37  podman.network                                  [all]                                   [all]   [rhcos]
    38  podman.noop                                     [all]                                   [all]   [rhcos]
    39  podman.workflow                                 [all]                                   [all]   [rhcos]
    40  # Run your test and see what happens
    41  $ sudo ./bin/kola run -b rhcos --qemu-image `pwd`/rhcos-410.8.20190502.0-qemu.qcow2 podman.noop
    42  === RUN   podman.noop
    43  --- PASS: podman.noop (21.08s)
    44  PASS, output in _kola_temp/qemu-2019-05-08-1535-16606
    45  # git add/commit/push...
    46  # Open PR to get the test added!
    47  ```
    48  
    49  ## Grouping Tests
    50  
    51  Sometimes it makes sense to group tests together under a specific package, especially when these tests are related and require the same test parameters. For `kola` it only takes a forwarding function to do testing groups. This forwarding function should take `cluster.TestCluster` as it's only input, and execute running other tests with `cluster.TestCluster.Run()`.
    52  
    53  It is worth noting that the tests within the group are executed sequentially and on the same machine. As such, it is not recommended to group tests which modify the system state.
    54  
    55  Additionally, the FailFast flag can be enabled during the test registration to skip any remaining steps after a failure has occurred.
    56  
    57  Continuing with the look at the `podman` package we can see that `podman.base` is registered like so:
    58  
    59  ```golang
    60      register.Register(&register.Test{
    61              Run:         podmanBaseTest,
    62              ClusterSize: 1,
    63              Name:        `podman.base`,
    64              Distros:     []string{"rhcos"},
    65      })
    66  ```
    67  
    68  If we look at `podmanBaseTest` it becomes very obvious that it's not a test of it's own, but a group of tests.
    69  
    70  ```go
    71  func podmanBaseTest(c cluster.TestCluster) {
    72          c.Run("info", podmanInfo)
    73          c.Run("resources", podmanResources)
    74          c.Run("network", podmanNetworksReliably)
    75  }
    76  ```
    77  
    78  ## Adding New Packages
    79  
    80  If you need to add a new testing package there are few steps that must be done.
    81  
    82  1. Create a new directory in `kola/tests/` which is descriptive of what will be tested.
    83  2. Add at least one file in the new directory with it's package the same name as it's directory name
    84  3. Edit the kola/registry/registry.go file to include your new package
    85  4. Add and register your new tests
    86  
    87  As an example, let's say you want to add a new test package called `foo`.
    88  
    89  1. First create `kola/tests/foo/`
    90  2. Then `echo "package foo" > kola/tests/foo/foo.go`
    91  3. Next, edit `kola/registry/registry.go` and add this to the imports `_ "github.com/coreos/mantle/kola/tests/foo"`
    92  
    93  ```golang
    94  package registry
    95  
    96  // Tests imported for registration side effects. These make up the OS test suite and is explicitly imported from the main package.
    97  import (
    98          _ "github.com/coreos/mantle/kola/tests/coretest"
    99          _ "github.com/coreos/mantle/kola/tests/crio"
   100          _ "github.com/coreos/mantle/kola/tests/docker"
   101          _ "github.com/coreos/mantle/kola/tests/etcd"
   102          _ "github.com/coreos/mantle/kola/tests/foo"
   103  <snip/>
   104  ```
   105  
   106  4. Lastly, use $EDITOR on `kola/tests/foo/foo.go` adding in new test groups and tests.
   107  
   108  ## Full Example
   109  
   110  ### File: kola/tests/foo/foo.go
   111  ```golang
   112  // Copyright 2019 Red Hat, Inc.
   113  //
   114  // Licensed under the Apache License, Version 2.0 (the "License");
   115  // you may not use this file except in compliance with the License.
   116  // You may obtain a copy of the License at
   117  //
   118  //     http://www.apache.org/licenses/LICENSE-2.0
   119  //
   120  // Unless required by applicable law or agreed to in writing, software
   121  // distributed under the License is distributed on an "AS IS" BASIS,
   122  // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   123  // See the License for the specific language governing permissions and
   124  // limitations under the License.
   125  
   126  package foo
   127  
   128  import (
   129          "github.com/coreos/mantle/kola/cluster"
   130          "github.com/coreos/mantle/kola/register"
   131  )
   132  
   133  // init runs when the package is imported and takes care of registering tests
   134  func init() {
   135      register.Register(&register.Test{ // See: https://godoc.org/github.com/coreos/mantle/kola/register#Test
   136              Run:         exampleTestGroup,
   137              ClusterSize: 1,
   138              Name:        `example.example`,
   139              Flags:       []register.Flag{}, // See: https://godoc.org/github.com/coreos/mantle/kola/register#Flag
   140              Distros:     []string{"rhcos"},
   141              FailFast:    true,
   142      })
   143  }
   144  
   145  // exampleTestGroup groups all of the example.example tests together
   146  func exampleTestGroup(c cluster.TestCluster) {
   147      c.Run("test1", exampleTestOne)
   148      c.Run("test2", exampleTestTwo)
   149  }
   150  
   151  // The first example test (and it does nothing!)
   152  func exampleTestOne(c cluster.TestCluster) {
   153      // NOOP!
   154  }
   155  
   156  // The second example test and it makes sure os-release has content
   157  func exampleTestTwo(c cluster.TestCluster) {
   158      // Get the first machine in the cluster
   159      m := c.Machines()[0]
   160      osrelease := c.MustSSH(m, `cat /etc/os-release`)
   161      if string(osrelease) == "" {
   162          c.Errorf("/etc/os-release was empty. Expected content.")
   163      }
   164  }
   165  ```
   166  
   167  ### File: kola/registry/registry.go
   168  
   169  ```golang
   170  package registry
   171  
   172  // Tests imported for registration side effects. These make up the OS test suite and is explicitly imported from the main package.
   173  import (
   174          _ "github.com/coreos/mantle/kola/tests/coretest"
   175          _ "github.com/coreos/mantle/kola/tests/crio"
   176          _ "github.com/coreos/mantle/kola/tests/docker"
   177          _ "github.com/coreos/mantle/kola/tests/etcd"
   178          _ "github.com/coreos/mantle/kola/tests/flannel"
   179          _ "github.com/coreos/mantle/kola/tests/foo"
   180          _ "github.com/coreos/mantle/kola/tests/ignition"
   181          _ "github.com/coreos/mantle/kola/tests/kubernetes"
   182          _ "github.com/coreos/mantle/kola/tests/locksmith"
   183          _ "github.com/coreos/mantle/kola/tests/metadata"
   184          _ "github.com/coreos/mantle/kola/tests/misc"
   185          _ "github.com/coreos/mantle/kola/tests/ostree"
   186          _ "github.com/coreos/mantle/kola/tests/packages"
   187          _ "github.com/coreos/mantle/kola/tests/podman"
   188          _ "github.com/coreos/mantle/kola/tests/rkt"
   189          _ "github.com/coreos/mantle/kola/tests/rpmostree"
   190          _ "github.com/coreos/mantle/kola/tests/systemd"
   191          _ "github.com/coreos/mantle/kola/tests/torcx"
   192          _ "github.com/coreos/mantle/kola/tests/update"
   193  )
   194  ```