github.com/dmaizel/tests@v0.0.0-20210728163746-cae6a2d9cee8/README.md (about) 1 # Kata Containers tests 2 3 * [Kata Containers tests](#kata-containers-tests) 4 * [Getting the code](#getting-the-code) 5 * [Test Content](#test-content) 6 * [CI Content](#ci-content) 7 * [Centralised scripts](#centralised-scripts) 8 * [CI setup](#ci-setup) 9 * [Controlling the CI](#controlling-the-ci) 10 * [Detecting a CI system](#detecting-a-ci-system) 11 * [Breaking Compatibility](#breaking-compatibility) 12 * [CLI tools](#cli-tools) 13 * [Developer Mode](#developer-mode) 14 * [Write a new Unit Test](#write-a-new-unit-test) 15 * [Run the Kata Containers tests](#run-the-kata-containers-tests) 16 * [Requirements to run Kata Containers tests](#requirements-to-run-kata-containers-tests) 17 * [Prepare an environment](#prepare-an-environment) 18 * [Run the tests](#run-the-tests) 19 * [Running subsets of tests](#running-subsets-of-tests) 20 * [Metrics tests](#metrics-tests) 21 * [Kata Admission controller webhook](#kata-admission-controller-webhook) 22 * [Using Vagrant to test your code changes](#using-vagrant-to-test-your-code-changes) 23 24 This repository contains various types of tests and utilities (called 25 "content" from now on) for testing the [Kata Containers](https://github.com/kata-containers) 26 code repositories. 27 28 ## Getting the code 29 30 ``` 31 $ go get -d github.com/kata-containers/tests 32 ``` 33 34 ## Test Content 35 36 We provide several tests to ensure Kata-Containers run on different scenarios 37 and with different container managers. 38 39 1. Integration tests to ensure compatibility with: 40 - [Kubernetes](https://github.com/kata-containers/tests/tree/main/integration/kubernetes) 41 - [Containerd](https://github.com/kata-containers/tests/tree/main/integration/containerd) 42 2. [Stability tests](https://github.com/kata-containers/tests/tree/main/integration/stability) 43 3. [Metrics](https://github.com/kata-containers/tests/tree/main/metrics) 44 4. [VFIO](https://github.com/kata-containers/tests/tree/main/functional/vfio) 45 46 ## CI Content 47 48 This repository contains a [number of scripts](/.ci) 49 that run from under a "CI" (Continuous Integration) system. 50 51 ### Centralised scripts 52 53 The CI scripts in this repository are used to test changes to the content of 54 this repository. These scripts are also used by the other Kata Containers code 55 repositories. 56 57 The advantages of this approach are: 58 59 - Functionality is defined once. 60 - Easy to make changes affecting all code repositories centrally. 61 62 - Assurance that all the code repositories are tested in this same way. 63 64 CI scripts also provide a convenient way for other Kata repositories to 65 install software. The preferred way to use these scripts is to invoke `make` 66 with the corresponding `install-` target. For example, to install CRI-O you 67 would use: 68 69 ``` 70 $ make -C <path-to-this-repo> install-crio 71 ``` 72 73 Use `make list-install-targets` to retrieve all the available install targets. 74 75 ### CI setup 76 77 > **WARNING:** 78 > 79 > The CI scripts perform a lot of setup before running content under a 80 > CI. Some of this setup runs as the `root` user and **could break your developer's 81 > system**. See [Developer Mode](#developer-mode). 82 83 ### Controlling the CI 84 85 #### GitHub Actions 86 87 Kata Containers uses GitHub Actions in the [Kata Containers](https://github.com/kata-containers/kata-containers) repos. 88 All those actions, apart from the one to test `kata-deploy`, are automatically triggered when 89 a pull request is submitted. The trigger phrase for testing kata-deploy is `/test_kata_deploy`. 90 91 #### Jenkins 92 93 The Jenkins configuration and most documentation is kept in the [CI repository](https://github.com/kata-containers/ci). 94 Jenkins is setup to trigger a CI run on all the slaves/nodes when a `/test` comment is added to a pull request. However, 95 there are some specific comments that are defined for specific CI slaves/nodes which are defined in the Jenkins 96 `config.xml` files in the `<triggerPhase>` XML element in the [CI repository](https://github.com/kata-containers/ci). 97 98 #### Specific Jenkins job triggers 99 100 Some jobs like a particular distro, feature or architecture can be triggered individually, the specific job triggers information 101 can be found in the [Community repository](https://github.com/kata-containers/community/wiki/Controlling-the-CI). 102 103 ### Detecting a CI system 104 105 The strategy to check if the tests are running under a CI system is to see 106 if the `CI` variable is set to the value `true`. For example, in shell syntax: 107 108 ``` 109 if [ "$CI" = true ]; then 110 : # Assumed to be running in a CI environment 111 else 112 : # Assumed to NOT be running in a CI environment 113 fi 114 ``` 115 116 ### Breaking Compatibility 117 118 In case the patch you submit breaks the CI because it needs to be tested 119 together with a patch from another `kata-containers` repository, you have to 120 specify which repository and which pull request it depends on. 121 122 Using a simple tag `Depends-on:` in your commit message will allow the CI to 123 run properly. Notice that this tag is parsed from the latest commit of the 124 pull request. 125 126 For example: 127 128 ``` 129 Subsystem: Change summary 130 131 Detailed explanation of your changes. 132 133 Fixes: #nnn 134 135 Depends-on:github.com/kata-containers/kata-containers#999 136 137 Signed-off-by: <contributor@foo.com> 138 139 ``` 140 141 In this example, we tell the CI to fetch the pull request 999 from the `kata-containers` 142 repository and use that rather than the `main` branch when testing the changes 143 contained in this pull request. 144 145 ## CLI tools 146 147 This repository contains a number of [command line tools](cmd). They are used 148 by the [CI](#ci-content) tests but may be useful for user to run stand alone. 149 150 ## Developer Mode 151 152 Developers need a way to run as much test content as possible locally, but as 153 explained in [CI Setup](#ci-setup), running *all* the content in this 154 repository could be dangerous. 155 156 The recommended approach to resolve this issue is to set the following variable 157 to any non-blank value **before using *any* content from this repository**: 158 159 ``` 160 export KATA_DEV_MODE=true 161 ``` 162 163 Setting this variable has the following effects: 164 165 - Disables content that might not be safe for developers to run locally. 166 - Ignores the effect of the `CI` variable being set (for extra safety). 167 168 You should be aware that setting this variable provides a safe *subset* of 169 functionality; it is still possible that PRs raised for code repositories will 170 still fail under the automated CI systems since those systems are running all 171 possible tests. 172 173 ## Write a new Unit Test 174 175 See the [unit test advice documentation](Unit-Test-Advice.md). 176 177 ## Run the Kata Containers tests 178 179 ### Requirements to run Kata Containers tests 180 181 You need to install the following to run Kata Containers tests: 182 183 - [golang](https://golang.org/dl) 184 185 To view the versions of go known to work, see the `golang` entry in the 186 [versions database](https://github.com/kata-containers/kata-containers/blob/main/versions.yaml). 187 188 - `make`. 189 190 ### Prepare an environment 191 192 The recommended method to set up Kata Containers is to use the official and latest 193 stable release. You can find the official documentation to do this in the 194 [Kata Containers installation user guides](https://github.com/kata-containers/documentation/blob/master/install/README.md). 195 196 To try the latest commits of Kata use the CI scripts, which build and install from the 197 `kata-containers` repositories, with the following steps: 198 199 > **Warning:** This may replace/delete packages and configuration that you already have. 200 > Please use these steps only on a testing environment. 201 202 Add the `$GOPATH/bin` directory to the PATH: 203 ``` 204 $ export PATH=${GOPATH}/bin:${PATH} 205 ``` 206 207 Clone the `kata-container/tests` repository: 208 ``` 209 $ go get -d github.com/kata-containers/tests 210 ``` 211 212 Go to the tests repo directory: 213 ``` 214 $ cd $GOPATH/src/github.com/kata-containers/tests 215 ``` 216 217 Execute the setup script: 218 ``` 219 $ .ci/setup.sh 220 ``` 221 > **Limitation:** If the script fails for a reason and it is re-executed, it will execute 222 all steps from the beginning and not from the failed step. 223 224 ### Run the tests 225 226 If you have already installed the Kata Containers packages and a container 227 manager (i.e. Kubernetes), and you want to execute the content 228 for all the tests, run the following: 229 230 ``` 231 $ export RUNTIME=kata-runtime 232 $ export KATA_DEV_MODE=true 233 $ sudo -E PATH=$PATH make test 234 ``` 235 236 You can also execute a single test suite. For example, if you want to execute 237 the Kubernetes integration tests, run the following: 238 ``` 239 $ sudo -E PATH=$PATH make kubernetes 240 ``` 241 242 A list of available test suite `make` targets can be found by running the 243 following: 244 245 ``` 246 $ make help 247 ``` 248 249 ### Running subsets of tests 250 251 Individual tests or subsets of tests can be selected to be run. The method of 252 test selection depends on which type of test framework the test is written 253 with. Most of the Kata Containers test suites are written 254 using [Bats](https://github.com/sstephenson/bats) files. 255 256 #### Running Bats based tests 257 258 The Bats based tests are shell scripts, starting with the line: 259 260 ```sh 261 #!/usr/bin/env bats 262 ``` 263 264 This allows the Bats files to be executed directly. Before executing the file, 265 ensure you have Bats installed. The Bats files should be executed 266 from the root directory of the tests repository to ensure they can locate all other 267 necessary components. An example of how a Bats test is run from the `Makefile` 268 looks like: 269 270 ```makefile 271 kubernetes: 272 bash -f .ci/install_bats.sh 273 bash -f integration/kubernetes/run_kubernetes_tests.sh 274 ``` 275 276 ## Metrics tests 277 See the [metrics documentation](metrics). 278 279 ## Kata Admission controller webhook 280 See the [webhook documentation](kata-webhook). 281 282 ## Using Vagrant to test your code changes 283 284 It is strongly recommended that you test your changes locally before opening 285 a pull request as this can save people's time and CI resources. Because 286 testing Kata Containers involve complex build and setup instructions, scripts 287 on the `.ci` directory are created to ease and provide a reproducible process; but they 288 are meant to run on CI environments that can be discarded after use. Therefore, 289 developers have noticed dangerous side effects from running those scripts on a workstation 290 or development environment. 291 292 That said, we provide in this repository a `Vagrantfile` which allows developers to use 293 the [vagrant](https://www.vagrantup.com) tool to create a VM with the setup as close as 294 as possible to the environments where CI jobs will run the tests. Thus, allowing to 295 reproduce a CI job locally. 296 297 Currently it is only able to create a *Fedora 32* or *Ubuntu 20.04* VM. And your workstation 298 must be capable of running VMs with: 299 * 8GB of system memory 300 * ~45GB and ~20GB of disk space for the VM images (Fedora and Ubuntu, respectively) on 301 the Libvirt's storage pool 302 303 Besides having vagrant installed in your host, it is needed the [vagrant libvirt plug-in](https://github.com/vagrant-libvirt/vagrant-libvirt) (Libvirt is the provider currently used), QEMU and `rsync` (needed to copy files between 304 the host and guest). 305 306 For example, to install the required software on Fedora host: 307 ```sh 308 $ sudo dnf install -y qemu-kvm libvirt vagrant vagrant-libvirt rsync 309 ``` 310 > **Note**: ensure that you don't have Kata Container's built QEMU overwritten 311 > the distro's in your host, otherwise Vagrant will not work. 312 313 Use the `vagrant up [fedora|ubuntu]` command to bring up the VM. Vagrant is going to 314 pull (unless cached) the base VM image, provision it and then bootstrap the 315 Kata Containers environment (essentially by sourcing environment variables 316 and running the `.ci/setup.sh` script). For example: 317 318 ```sh 319 $ cd ${GOPATH}/src/github.com/kata-containers/tests 320 $ vagrant up fedora 321 ``` 322 323 The following repositories are automatically copied to the guest: 324 * `${GOPATH}/src/github.com/kata-containers/tests` 325 * `${GOPATH}/src/github.com/kata-containers/kata-containers` 326 327 If you want to reproduce a specific CI job, ensure that you have the `CI_JOB` 328 environment variable exported on your host environment *before* you run 329 `vagrant up`. For the possible `CI_JOB` values, see the `.ci/ci_job_flags.sh` 330 file. For example, the following will setup the VM to run CRI-O + Kubernetes 331 job: 332 ```sh 333 $ cd $GOPATH/src/github.com/kata-containers/tests 334 $ export CI_JOB="CRIO_K8S" 335 $ vagrant up fedora 336 ``` 337 338 At this point, if everything went well, you have a fully functional environment 339 with Kata Containers built and installed. To connect in the VM and run the tests: 340 ```sh 341 $ vagrant ssh fedora 342 $ .ci/run.sh 343 ``` 344 345 In theory you could export `CI_JOB` with a different value and re-provision the 346 same VM (`vagrant provision [fedora|ubuntu]`), however this is not recommended because 347 our CI scripts are meant for a single-shot execution. So if you need to run a different 348 job locally, you should destroy the VM with the `vagrant destroy [fedora|ubuntu]` command 349 then start the process again.