github.com/containers/podman/v5@v5.1.0-rc1/test/README.md (about) 1 ![PODMAN logo](https://raw.githubusercontent.com/containers/common/main/logos/podman-logo-full-vert.png) 2 # Test utils 3 Test utils provide common functions and structs for testing. It includes two structs: 4 * `PodmanTest`: Handle the *podman* command and other global resources like temporary 5 directory. It provides basic methods, like checking podman image and pod status. Test 6 suites should create their owner test *struct* as a composite of `PodmanTest`, and their 7 owner PodmanMakeOptions(). 8 9 * `PodmanSession`: Store execution session data and related *methods*. Such like get command 10 output and so on. It can be used directly in the test suite, only embed it to your owner 11 session struct if you need expend it. 12 13 ## Unittest for test/utils 14 To ensure neither *tests* nor *utils* break, There are unit-tests for each *functions* and 15 *structs* in `test/utils`. When you adding functions or structs to this *package*, please 16 update both unit-tests for it and this documentation. 17 18 ### Run unit test for test/utils 19 Run unit test for test/utils. 20 21 ``` 22 make localunit 23 ``` 24 25 ## Structure of the test utils and test suites 26 The test *utils* package is at the same level of test suites. Each test suites also have their 27 owner common functions and structs stored in `libpod_suite_test.go`. 28 29 # Ginkgo test framework 30 [Ginkgo](https://github.com/onsi/ginkgo) is a BDD testing framework. This allows 31 us to use native Golang to perform our tests and there is a strong affiliation 32 between Ginkgo and the Go test framework. 33 34 ## Installing dependencies 35 Some external binaries are required to successfully run the tests. 36 The test currently depend on: 37 - normal podman runtime dependencies 38 - coreutils 39 - ncat 40 - gzip 41 - xz 42 - htpasswd 43 - iproute2 44 - iptables 45 - util-linux 46 - tar 47 - docker 48 - systemd/systemctl 49 50 Most of these are only required for a few tests so it is not a big issue if not everything is installed. Only a few test should fail then. 51 52 ### Installing ginkgo 53 Build ginkgo and install it under ./test/tools/build/ginkgo with the following command: 54 ``` 55 make .install.ginkgo 56 ``` 57 58 # Integration Tests 59 Test suite for integration test for podman command line. It has its own structs: 60 * `PodmanTestIntegration`: Integration test *struct* as a composite of `PodmanTest`. It 61 set up the global options for *podman* command to ignore the environment influence from 62 different test system. 63 64 * `PodmanSessionIntegration`: This *struct* has it own *methods* for checking command 65 output with given format JSON by using *structs* defined in inspect package. 66 67 ## Running the integration tests 68 You can run the entire suite of integration tests with the following command: 69 70 ``` 71 make localintegration 72 ``` 73 74 To run the remote tests use: 75 ``` 76 make remoteintegration 77 ``` 78 79 ### Test variables 80 81 Some test only work as rootless while others only work as root. So to test everything 82 you should make sure to run the make command above as normal user and root. 83 84 The following environment variables are supported by the test setup: 85 - `PODMAN_BINARY`: path to the podman binary, defaults to `bin/podman` in the repository root. 86 - `PODMAN_REMOTE_BINARY`: path to the podman-remote binary, defaults to `bin/podman-remote` in the repository root. 87 - `QUADLET_BINARY`: path to the quadlet binary, defaults to `bin/quadlet` in the repository root. 88 - `CONMON_BINARY`: path to th conmon binary, defaults to `/usr/libexec/podman/conmon`. 89 - `OCI_RUNTIME`: which oci runtime to use, defaults to `crun`. 90 - `NETWORK_BACKEND`: the network backend, either `netavark` (default) or `cni`. 91 - `PODMAN_DB`: the database backend `sqlite` (default) or `boltdb`. 92 - `PODMAN_TEST_IMAGE_CACHE_DIR`: path were the container images should be cached, defaults to `/tmp`. 93 94 ### Running a single file of integration tests 95 You can run a single file of integration tests using the go test command: 96 97 ``` 98 make localintegration FOCUS_FILE=your_test.go 99 ``` 100 101 `FOCUS_FILE` file maps to ginkgo's `--focus-file` option, see the ginkgo 102 [docs](https://onsi.github.io/ginkgo/#location-based-filtering) for the accepted syntax. 103 104 For remote tests use the `remoteintegration` Makefile target instead. 105 106 ### Running a single integration test 107 Before running the test suite, you have to declare which test you want run in the test 108 file itself. Consider the following actual test: 109 ``` 110 It("podman inspect bogus pod", func() { 111 session := podmanTest.Podman([]string{"pod", "inspect", "foobar"}) 112 session.WaitWithDefaultTimeout() 113 Expect(session).To(ExitWithError(125, "no such pod foobar")) 114 }) 115 ``` 116 117 To mark this as the test you want run, you simply change the *It* description to *FIt*. Please note how 118 both the `F` and `I` are capitalized. Also see the ginkgo [docs](https://onsi.github.io/ginkgo/#focused-specs). 119 120 *Note*: Be sure you remove the `F` from the tests before committing your changes or you will skip all tests 121 in that file except the one with the `FIt` denotation. 122 123 Alternatively you can use the `FOCUS` option which maps to `--focus`, again see the ginkgo 124 [docs](https://onsi.github.io/ginkgo/#description-based-filtering) for more info about the syntax. 125 ``` 126 make localintegration FOCUS="podman inspect bogus pod" 127 ``` 128 129 ### Controlling Ginkgo parameters 130 You can control some of the parameters passed to Ginkgo 131 132 - Disable parallel tests by setting `GINKGO_PARALLEL=n` 133 - Set flake retry count (default 3) to one by setting `GINKGO_FLAKE_ATTEMPTS=1` 134 - Produce colorful tests report by setting `GINKGO_NO_COLOR=n` 135 136 # System tests 137 System tests are used for testing the *podman* CLI in the context of a complete system. It 138 requires that *podman*, all dependencies, and configurations are in place. The intention of 139 system testing is to match as closely as possible with real-world user/developer use-cases 140 and environments. The orchestration of the environments and tests is left to external 141 tooling. 142 143 System tests use Bash Automated Testing System (`bats`) as a testing framework. 144 Install it via your package manager or get latest stable version 145 [directly from the repository](https://github.com/bats-core/bats-core), e.g.: 146 147 ``` 148 mkdir -p ~/tools/bats 149 git clone --single-branch --branch v1.1.0 https://github.com/bats-core/bats-core.git ~/tools/bats 150 ``` 151 152 Make sure that `bats` binary (`bin/bats` in the repository) is in your `PATH`, if not - add it: 153 154 ``` 155 PATH=$PATH:~/tools/bats/bin 156 ``` 157 158 System tests also rely on `jq`, `socat`, `nmap`, and other tools. For a 159 comprehensive list, see the `%package tests` section in the 160 [fedora specfile](https://src.fedoraproject.org/rpms/podman/blob/main/f/podman.spec). 161 162 ## Running system tests 163 When `bats` is installed and is in your `PATH`, you can run the test suite with following command: 164 165 ``` 166 make localsystem 167 ``` 168 169 ## Running the system tests in a more controlled way 170 If you would like to run a subset of the system tests, or configure the environment (e.g. root vs rootless, local vs remote), 171 use `hack/bats`. 172 173 For usage run: 174 ``` 175 hack/bats --help 176 ``` 177 178 ## Contributing to system tests 179 180 Please see [the TODO list of needed workflows/tests](system/TODO.md).