github.com/kata-containers/tests@v0.0.0-20240307153542-772105b56064/README.md (about) 1 # Kata Containers tests 2 3 This repository contains various types of tests and utilities (called 4 "content" from now on) for testing the [Kata Containers](https://github.com/kata-containers) 5 code repositories. 6 7 ## Getting the code 8 9 ``` 10 $ go get -d github.com/kata-containers/tests 11 ``` 12 13 ## Test Content 14 15 We provide several tests to ensure Kata-Containers run on different scenarios 16 and with different container managers. 17 18 1. Integration tests to ensure compatibility with: 19 - [Kubernetes](https://github.com/kata-containers/tests/tree/main/integration/kubernetes) 20 - [Containerd](https://github.com/kata-containers/tests/tree/main/integration/containerd) 21 2. [Stability tests](./stability) 22 3. [Metrics](https://github.com/kata-containers/tests/tree/main/metrics) 23 4. [VFIO](https://github.com/kata-containers/tests/tree/main/functional/vfio) 24 25 ## CI Content 26 27 This repository contains a [number of scripts](/.ci) 28 that run from under a "CI" (Continuous Integration) system. 29 30 ### Centralised scripts 31 32 The CI scripts in this repository are used to test changes to the content of 33 this repository. These scripts are also used by the other Kata Containers code 34 repositories. 35 36 The advantages of this approach are: 37 38 - Functionality is defined once. 39 - Easy to make changes affecting all code repositories centrally. 40 41 - Assurance that all the code repositories are tested in this same way. 42 43 CI scripts also provide a convenient way for other Kata repositories to 44 install software. The preferred way to use these scripts is to invoke `make` 45 with the corresponding `install-` target. For example, to install CRI-O you 46 would use: 47 48 ``` 49 $ make -C <path-to-this-repo> install-crio 50 ``` 51 52 Use `make list-install-targets` to retrieve all the available install targets. 53 54 ### CI setup 55 56 > **WARNING:** 57 > 58 > The CI scripts perform a lot of setup before running content under a 59 > CI. Some of this setup runs as the `root` user and **could break your developer's 60 > system**. See [Developer Mode](#developer-mode). 61 62 ### Controlling the CI 63 64 #### GitHub Actions 65 66 Kata Containers uses GitHub Actions in the [Kata Containers](https://github.com/kata-containers/kata-containers) repos. 67 All those actions, apart from the one to test `kata-deploy`, are automatically triggered when 68 a pull request is submitted. The trigger phrase for testing kata-deploy is `/test_kata_deploy`. 69 70 #### Jenkins 71 72 The Jenkins configuration and most documentation is kept in the [CI repository](https://github.com/kata-containers/ci). 73 Jenkins is setup to trigger a CI run on all the slaves/nodes when a `/test` comment is added to a pull request. However, 74 there are some specific comments that are defined for specific CI slaves/nodes which are defined in the Jenkins 75 `config.xml` files in the `<triggerPhase>` XML element in the [CI repository](https://github.com/kata-containers/ci). 76 77 #### Specific Jenkins job triggers 78 79 Some jobs like a particular distro, feature or architecture can be triggered individually, the specific job triggers information 80 can be found in the [Community repository](https://github.com/kata-containers/community/wiki/Controlling-the-CI). 81 82 ### Detecting a CI system 83 84 The strategy to check if the tests are running under a CI system is to see 85 if the `CI` variable is set to the value `true`. For example, in shell syntax: 86 87 ``` 88 if [ "$CI" = true ]; then 89 : # Assumed to be running in a CI environment 90 else 91 : # Assumed to NOT be running in a CI environment 92 fi 93 ``` 94 95 ### Display verbose failure details 96 97 By default, when scripts in this repository fail, they display an 98 error message to `stderr` and return `1` (failure) to the shell. 99 However, when [called by the CI](#detecting-a-ci-system), the scripts 100 will also dump a full stack trace and some environment details to `stderr`. 101 102 To force this behaviour outside of a CI environment, ensure the variable below is set: 103 104 ```bash 105 export KATA_TEST_VERBOSE=true 106 ``` 107 108 ### Breaking Compatibility 109 110 In case the patch you submit breaks the CI because it needs to be tested 111 together with a patch from another `kata-containers` repository, you have to 112 specify which repository and which pull request it depends on. 113 114 Using a simple tag `Depends-on:` in your commit message will allow the CI to 115 run properly. Notice that this tag is parsed from the latest commit of the 116 pull request. 117 118 For example: 119 120 ``` 121 Subsystem: Change summary 122 123 Detailed explanation of your changes. 124 125 Fixes: #nnn 126 127 Depends-on:github.com/kata-containers/kata-containers#999 128 129 Signed-off-by: <contributor@foo.com> 130 131 ``` 132 133 In this example, we tell the CI to fetch the pull request 999 from the `kata-containers` 134 repository and use that rather than the `main` branch when testing the changes 135 contained in this pull request. 136 137 ## CLI tools 138 139 This repository contains a number of [command line tools](cmd). They are used 140 by the [CI](#ci-content) tests but may be useful for user to run stand alone. 141 142 ## Developer Mode 143 144 Developers need a way to run as much test content as possible locally, but as 145 explained in [CI Setup](#ci-setup), running *all* the content in this 146 repository could be dangerous. 147 148 The recommended approach to resolve this issue is to set the following variable 149 to any non-blank value **before using *any* content from this repository**: 150 151 ``` 152 export KATA_DEV_MODE=true 153 ``` 154 155 Setting this variable has the following effects: 156 157 - Disables content that might not be safe for developers to run locally. 158 - Ignores the effect of the `CI` variable being set (for extra safety). 159 160 You should be aware that setting this variable provides a safe *subset* of 161 functionality; it is still possible that PRs raised for code repositories will 162 still fail under the automated CI systems since those systems are running all 163 possible tests. 164 165 ## Write a new Unit Test 166 167 See the [unit test advice documentation](https://github.com/kata-containers/kata-containers/blob/main/docs/Unit-Test-Advice.md). 168 169 ## Run the Kata Containers tests 170 171 ### Requirements to run Kata Containers tests 172 173 You need to install the following to run Kata Containers tests: 174 175 - [golang](https://golang.org/dl) 176 177 To view the versions of go known to work, see the `golang` entry in the 178 [versions database](https://github.com/kata-containers/kata-containers/blob/main/versions.yaml). 179 180 - `make`. 181 182 ### Prepare an environment 183 184 The recommended method to set up Kata Containers is to use the official and latest 185 stable release. You can find the official documentation to do this in the 186 [Kata Containers installation user guides](https://github.com/kata-containers/kata-containers/blob/main/docs/install/README.md). 187 188 To try the latest commits of Kata use the CI scripts, which build and install from the 189 `kata-containers` repositories, with the following steps: 190 191 > **Warning:** This may replace/delete packages and configuration that you already have. 192 > Please use these steps only on a testing environment. 193 194 Add the `$GOPATH/bin` directory to the PATH: 195 ``` 196 $ export PATH=${GOPATH}/bin:${PATH} 197 ``` 198 199 Clone the `kata-container/tests` repository: 200 ``` 201 $ go get -d github.com/kata-containers/tests 202 ``` 203 204 Go to the tests repo directory: 205 ``` 206 $ cd $GOPATH/src/github.com/kata-containers/tests 207 ``` 208 209 Execute the setup script: 210 ``` 211 $ export CI=true 212 $ export CI_JOB=CRI_CONTAINERD_K8S 213 $ .ci/setup.sh 214 ``` 215 In this case we are exporting the environment variables for the CRI_CONTAINERD_K8S Jenkins Job for more information 216 of which CI_JOB needs to be used see the following https://github.com/kata-containers/tests/blob/main/.ci/ci_job_flags.sh. 217 218 > **Limitation:** If the script fails for a reason and it is re-executed, it will execute 219 all steps from the beginning and not from the failed step. 220 221 ### Run the tests 222 223 If you have already installed the Kata Containers packages and a container 224 manager (i.e. Kubernetes), and you want to execute the content 225 for all the tests, run the following: 226 227 ``` 228 $ export RUNTIME=kata-runtime 229 $ export KATA_DEV_MODE=true 230 $ sudo -E PATH=$PATH make test 231 ``` 232 233 You can also execute a single test suite. For example, if you want to execute 234 the Kubernetes integration tests, run the following: 235 ``` 236 $ sudo -E PATH=$PATH make kubernetes 237 ``` 238 239 A list of available test suite `make` targets can be found by running the 240 following: 241 242 ``` 243 $ make help 244 ``` 245 246 ### Running subsets of tests 247 248 Individual tests or subsets of tests can be selected to be run. The method of 249 test selection depends on which type of test framework the test is written 250 with. Most of the Kata Containers test suites are written 251 using [Bats](https://github.com/sstephenson/bats) files. 252 253 #### Running Bats based tests 254 255 The Bats based tests are shell scripts, starting with the line: 256 257 ```sh 258 #!/usr/bin/env bats 259 ``` 260 261 This allows the Bats files to be executed directly. Before executing the file, 262 ensure you have Bats installed. The Bats files should be executed 263 from the root directory of the tests repository to ensure they can locate all other 264 necessary components. An example of how a Bats test is run from the `Makefile` 265 looks like: 266 267 ```makefile 268 kubernetes: 269 bash -f .ci/install_bats.sh 270 bash -f integration/kubernetes/run_kubernetes_tests.sh 271 ``` 272 273 ## Metrics tests 274 See the [metrics documentation](metrics). 275 276 ## Kata Admission controller webhook 277 See the [webhook documentation](kata-webhook). 278 279 ## Using Vagrant to test your code changes 280 281 It is strongly recommended that you test your changes locally before opening 282 a pull request as this can save people's time and CI resources. Because 283 testing Kata Containers involve complex build and setup instructions, scripts 284 on the `.ci` directory are created to ease and provide a reproducible process; but they 285 are meant to run on CI environments that can be discarded after use. Therefore, 286 developers have noticed dangerous side effects from running those scripts on a workstation 287 or development environment. 288 289 That said, we provide in this repository a `Vagrantfile` which allows developers to use 290 the [vagrant](https://www.vagrantup.com) tool to create a VM with the setup as close as 291 as possible to the environments where CI jobs will run the tests. Thus, allowing to 292 reproduce a CI job locally. 293 294 Your workstation must be capable of running VMs with: 295 * 8GB of system memory 296 * ~45GB and ~20GB of disk space for the VM images (Fedora and Ubuntu, respectively) on 297 the Libvirt's storage pool 298 299 Currently it supports the creation of *Fedora (32 and 35)* and *Ubuntu 20.04* VM, as shown on the table 300 below. The `Vagrantfile` was tested on Fedora 33 and Ubuntu 20.04 hosts, and it is 301 [known to fail](https://github.com/kata-containers/tests/issues/3942) the boot of Fedora VM on 302 Ubuntu host. If you have the need of testing on a different guest or it fails to work 303 on your host's distro then please [open an issue](https://github.com/kata-containers/tests/issues/new/choose) 304 to let us know. 305 306 |Host | Fedora 32 guest | Ubuntu 20.04 guest | 307 | --- | --- | --- | 308 | Fedora 33 | Yes | Yes | 309 | Ubuntu 20.04 | No | Yes | 310 311 Besides having vagrant installed in your host, it is needed the [vagrant libvirt plug-in](https://github.com/vagrant-libvirt/vagrant-libvirt) (Libvirt is the provider currently used), QEMU and `rsync` (needed to copy files between 312 the host and guest). 313 314 For example, to install the required software on Fedora host: 315 ```sh 316 $ sudo dnf install -y qemu-kvm libvirt vagrant vagrant-libvirt rsync 317 ``` 318 > **Note**: ensure that you don't have Kata Container's built QEMU overwritten 319 > the distro's in your host, otherwise Vagrant will not work. 320 321 Use the `vagrant up [fedora|ubuntu]` command to bring up the VM. Vagrant is going to 322 pull (unless cached) the base VM image, provision it and then bootstrap the 323 Kata Containers environment (essentially by sourcing environment variables 324 and running the `.ci/setup.sh` script). For example: 325 326 ```sh 327 $ cd ${GOPATH}/src/github.com/kata-containers/tests 328 $ vagrant up fedora 329 ``` 330 331 The following repositories are automatically copied to the guest: 332 * `${GOPATH}/src/github.com/kata-containers/tests` 333 * `${GOPATH}/src/github.com/kata-containers/kata-containers` 334 335 If you want to reproduce a specific CI job, ensure that you have the `CI_JOB` 336 environment variable exported on your host environment *before* you run 337 `vagrant up`. For the possible `CI_JOB` values, see the `.ci/ci_job_flags.sh` 338 file. For example, the following will setup the VM to run CRI-O + Kubernetes 339 job: 340 ```sh 341 $ cd $GOPATH/src/github.com/kata-containers/tests 342 $ export CI_JOB="CRIO_K8S" 343 $ vagrant up fedora 344 ``` 345 346 At this point, if everything went well, you have a fully functional environment 347 with Kata Containers built and installed. To connect in the VM and run the tests: 348 ```sh 349 $ vagrant ssh fedora 350 $ .ci/run.sh 351 ``` 352 353 In theory you could export `CI_JOB` with a different value and re-provision the 354 same VM (`vagrant provision [fedora|ubuntu]`), however this is not recommended because 355 our CI scripts are meant for a single-shot execution. So if you need to run a different 356 job locally, you should destroy the VM with the `vagrant destroy [fedora|ubuntu]` command 357 then start the process again. 358 359 The Vagrant configuration sometimes can get into inconsistent state. That may happen, for 360 instance, when the domain on Libvirt was created by the framework but it thinks the box 361 is not initialized yet. Also you may want to stop using Vagrant and you want to simply 362 wipe out all Vagrant control files and resources from your workstation. For those purposes you 363 should consider using the `.ci/vagrant-cleaner.sh` script; run `.ci/vagrant-cleaner.sh -h` for 364 further information.