github.com/containers/podman/v5@v5.1.0-rc1/test/README.md (about)

     1  ![PODMAN logo](https://raw.githubusercontent.com/containers/common/main/logos/podman-logo-full-vert.png)
     2  # Test utils
     3  Test utils provide common functions and structs for testing. It includes two structs:
     4  * `PodmanTest`: Handle the *podman* command and other global resources like temporary
     5  directory. It provides basic methods, like checking podman image and pod status. Test
     6  suites should create their owner test *struct* as a composite of `PodmanTest`, and their
     7  owner PodmanMakeOptions().
     8  
     9  * `PodmanSession`: Store execution session data and related *methods*. Such like get command
    10  output and so on. It can be used directly in the test suite, only embed it to your owner
    11  session struct if you need expend it.
    12  
    13  ## Unittest for test/utils
    14  To ensure neither *tests* nor *utils* break, There are unit-tests for each *functions* and
    15  *structs* in `test/utils`. When you adding functions or structs to this *package*, please
    16  update both unit-tests for it and this documentation.
    17  
    18  ### Run unit test for test/utils
    19  Run unit test for test/utils.
    20  
    21  ```
    22  make localunit
    23  ```
    24  
    25  ## Structure of the test utils and test suites
    26  The test *utils* package is at the same level of test suites. Each test suites also have their
    27  owner common functions and structs stored in `libpod_suite_test.go`.
    28  
    29  # Ginkgo test framework
    30  [Ginkgo](https://github.com/onsi/ginkgo) is a BDD testing framework. This allows
    31  us to use native Golang to perform our tests and there is a strong affiliation
    32  between Ginkgo and the Go test framework.
    33  
    34  ## Installing dependencies
    35  Some external binaries are required to successfully run the tests.
    36  The test currently depend on:
    37   - normal podman runtime dependencies
    38   - coreutils
    39   - ncat
    40   - gzip
    41   - xz
    42   - htpasswd
    43   - iproute2
    44   - iptables
    45   - util-linux
    46   - tar
    47   - docker
    48   - systemd/systemctl
    49  
    50  Most of these are only required for a few tests so it is not a big issue if not everything is installed. Only a few test should fail then.
    51  
    52  ### Installing ginkgo
    53  Build ginkgo and install it under ./test/tools/build/ginkgo with the following command:
    54  ```
    55  make .install.ginkgo
    56  ```
    57  
    58  # Integration Tests
    59  Test suite for integration test for podman command line. It has its own structs:
    60  * `PodmanTestIntegration`: Integration test *struct* as a composite of `PodmanTest`. It
    61  set up the global options for *podman* command to ignore the environment influence from
    62  different test system.
    63  
    64  * `PodmanSessionIntegration`: This *struct* has it own *methods* for checking command
    65  output with given format JSON by using *structs* defined in inspect package.
    66  
    67  ## Running the integration tests
    68  You can run the entire suite of integration tests with the following command:
    69  
    70  ```
    71  make localintegration
    72  ```
    73  
    74  To run the remote tests use:
    75  ```
    76  make remoteintegration
    77  ```
    78  
    79  ### Test variables
    80  
    81  Some test only work as rootless while others only work as root. So to test everything
    82  you should make sure to run the make command above as normal user and root.
    83  
    84  The following environment variables are supported by the test setup:
    85   - `PODMAN_BINARY`: path to the podman binary, defaults to `bin/podman` in the repository root.
    86   - `PODMAN_REMOTE_BINARY`: path to the podman-remote binary, defaults to `bin/podman-remote` in the repository root.
    87   - `QUADLET_BINARY`: path to the quadlet binary, defaults to `bin/quadlet` in the repository root.
    88   - `CONMON_BINARY`: path to th conmon binary, defaults to `/usr/libexec/podman/conmon`.
    89   - `OCI_RUNTIME`: which oci runtime to use, defaults to `crun`.
    90   - `NETWORK_BACKEND`: the network backend, either `netavark` (default) or `cni`.
    91   - `PODMAN_DB`: the database backend `sqlite` (default) or `boltdb`.
    92   - `PODMAN_TEST_IMAGE_CACHE_DIR`: path were the container images should be cached, defaults to `/tmp`.
    93  
    94  ### Running a single file of integration tests
    95  You can run a single file of integration tests using the go test command:
    96  
    97  ```
    98  make localintegration FOCUS_FILE=your_test.go
    99  ```
   100  
   101  `FOCUS_FILE` file maps to ginkgo's `--focus-file` option, see the ginkgo
   102  [docs](https://onsi.github.io/ginkgo/#location-based-filtering) for the accepted syntax.
   103  
   104  For remote tests use the `remoteintegration` Makefile target instead.
   105  
   106  ### Running a single integration test
   107  Before running the test suite, you have to declare which test you want run in the test
   108  file itself. Consider the following actual test:
   109  ```
   110  It("podman inspect bogus pod", func() {
   111  		session := podmanTest.Podman([]string{"pod", "inspect", "foobar"})
   112  		session.WaitWithDefaultTimeout()
   113  		Expect(session).To(ExitWithError(125, "no such pod foobar"))
   114  	})
   115  ```
   116  
   117  To mark this as the test you want run, you simply change the *It* description to *FIt*. Please note how
   118  both the `F` and `I` are capitalized. Also see the ginkgo [docs](https://onsi.github.io/ginkgo/#focused-specs).
   119  
   120  *Note*: Be sure you remove the `F` from the tests before committing your changes or you will skip all tests
   121  in that file except the one with the `FIt` denotation.
   122  
   123  Alternatively you can use the `FOCUS` option which maps to `--focus`, again see the ginkgo
   124  [docs](https://onsi.github.io/ginkgo/#description-based-filtering) for more info about the syntax.
   125  ```
   126  make localintegration FOCUS="podman inspect bogus pod"
   127  ```
   128  
   129  ### Controlling Ginkgo parameters
   130  You can control some of the parameters passed to Ginkgo
   131  
   132  - Disable parallel tests by setting `GINKGO_PARALLEL=n`
   133  - Set flake retry count (default 3) to one by setting `GINKGO_FLAKE_ATTEMPTS=1`
   134  - Produce colorful tests report by setting `GINKGO_NO_COLOR=n`
   135  
   136  # System tests
   137  System tests are used for testing the *podman* CLI in the context of a complete system. It
   138  requires that *podman*, all dependencies, and configurations are in place.  The intention of
   139  system testing is to match as closely as possible with real-world user/developer use-cases
   140  and environments. The orchestration of the environments and tests is left to external
   141  tooling.
   142  
   143  System tests use Bash Automated Testing System (`bats`) as a testing framework.
   144  Install it via your package manager or get latest stable version
   145  [directly from the repository](https://github.com/bats-core/bats-core), e.g.:
   146  
   147  ```
   148  mkdir -p ~/tools/bats
   149  git clone --single-branch --branch v1.1.0 https://github.com/bats-core/bats-core.git ~/tools/bats
   150  ```
   151  
   152  Make sure that `bats` binary (`bin/bats` in the repository) is in your `PATH`, if not - add it:
   153  
   154  ```
   155  PATH=$PATH:~/tools/bats/bin
   156  ```
   157  
   158  System tests also rely on `jq`, `socat`, `nmap`, and other tools. For a
   159  comprehensive list, see the `%package tests` section in the
   160  [fedora specfile](https://src.fedoraproject.org/rpms/podman/blob/main/f/podman.spec).
   161  
   162  ## Running system tests
   163  When `bats` is installed and is in your `PATH`, you can run the test suite with following command:
   164  
   165  ```
   166  make localsystem
   167  ```
   168  
   169  ## Running the system tests in a more controlled way
   170  If you would like to run a subset of the system tests, or configure the environment (e.g. root vs rootless, local vs remote),
   171  use `hack/bats`.
   172  
   173  For usage run:
   174  ```
   175  hack/bats --help
   176  ```
   177  
   178  ## Contributing to system tests
   179  
   180  Please see [the TODO list of needed workflows/tests](system/TODO.md).