k8s.io/kubernetes@v1.29.3/test/images/README.md (about) 1 # Kubernetes test images 2 3 ## Overview 4 5 All the images found here are used in Kubernetes tests that ensure its features and functionality. 6 The images are built and published as manifest lists, allowing multiarch and cross platform support. 7 8 This guide will provide information on how to: make changes to images, bump their version, build the 9 new images, test the changes made, promote the newly built staging images. 10 11 12 ## Prerequisites 13 14 In order to build the docker test images, a Linux node is required. The node will require `make`, 15 `docker (version 19.03.0 or newer)`, and ``docker buildx``, which will be used to build multiarch 16 images, as well as Windows images. In order to properly build multi-arch and Windows images, some 17 initialization is required (in CI this is done in [cloudbuild.yaml](cloudbuild.yaml)): 18 19 ```shell 20 docker run --rm --privileged multiarch/qemu-user-static --reset -p yes 21 docker buildx create --name img-builder --use 22 docker buildx inspect --bootstrap 23 ``` 24 25 The node must be able to push the images to the desired container registry, make sure you are 26 authenticated with the registry you're pushing to. 27 28 ## Updating the tests images 29 30 There are several thousands of tests in Kubernetes E2E testing. Not all of them are being run on 31 new PRs, and thus, not all images are used, especially those that are not used by Conformance tests. 32 33 So, in order to prevent regressions in the images and failing jobs, any changes made to the image 34 itself or its binaries will require the image's version to be bumped. In the case of a regression 35 which cannot be immediately resolved, the image version used in E2E tests will be reverted to the 36 last known stable version. 37 38 Most tests used in E2E testing suite use the `agnhost` image. It contains several subcommands with 39 different [functionalities](agnhost/README.md) used to validate different Kubernetes behaviors. If 40 a new functionality needs testing, consider adding an `agnhost` subcommand for it first, before 41 creating an entirely separate test image. 42 43 The general process of making updates to the images is as follows: 44 45 1. [Making changes to an image](#making-changes-to-an-image) 46 2. [Building the image](#building-images) 47 3. [Testing your changes](#testing-images) 48 4. [Promoting your changes](#promoting-images) 49 50 After going through these steps your image will be used in the e2e tests. There are some additional considerations for 51 [completely new images](#creating-and-promoting-new-images) and [Windows images](#windows-test-images-considerations). 52 53 ### Creating and promoting new images 54 55 If you intend to add an entirely different image and have it automatically built by the Image Builder 56 and used in E2E tests, you will also have to define the postsubmit prow job for it. This can easily 57 be done by running [this script](https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/k8s-staging-e2e-test-images.sh) 58 in `kubernetes/test-infra`. 59 60 ### Windows test images considerations 61 62 Ideally, the same `Dockerfile` can be used to build both Windows and Linux images. However, that isn't 63 always possible. If a different `Dockerfile` is needed for an image, it should be named `Dockerfile_windows`. 64 When building, `image-util.sh` will first check for this file name when building Windows images. 65 66 The building process uses `docker buildx` to build both Windows and Linux images, but there are a few 67 limitations when it comes to the Windows images: 68 69 - The Dockerfile can have multiple stages, including Windows and Linux stages for the same image, but 70 the Windows stage cannot have any `RUN` commands (see the agnhost's `Dockerfile_windows` as an example). 71 - The Windows stage cannot have any `WORKDIR` commands due to a bug (https://github.com/docker/buildx/issues/378) 72 - When copying Windows symlink files to a Windows image, `docker buildx` changes the symlink target, 73 prepending `Files\` to them (https://github.com/docker/buildx/issues/373) (for example, the symlink 74 target `C:\bin\busybox.exe` becomes `Files\C:\bin\busybox.exe`). This can be avoided by having symlink 75 targets with relative paths and having the target duplicated (for example, the symlink target 76 `busybox.exe` becomes `Files\busybox.exe` when copied, so the binary `C:\bin\Files\busybox.exe` 77 should exist in order for the symlink to be used correctly). See the busybox's `Dockerfile_windows` as 78 an example. 79 - `docker buildx` overwrites the image's PATH environment variable to a Linux PATH environment variable, 80 which won't work properly on Windows. See https://github.com/moby/buildkit/issues/1560 81 - The base image for all the Windows images is nanoserver, which is ~10 times smaller than Windows Servercore. 82 Most binaries added to the image will work out of the box, but some will not due to missing dependencies 83 (**attention**: the image will still build successfully, even if the added binaries will not work). 84 For example, `coredns.exe` requires `netapi32.dll`, which cannot be found on a nanoserver image, but 85 we can copy it from a servercore image (see the agnhost image's `Dockerfile_windows` file as an example). 86 A good rule of thumb is to use 64-bit applications instead of 32-bit as they have fewer dependencies. 87 You can determine what dependencies are missing by running `procmon.exe` on the container's host 88 (make sure that process isolation is used, not Hyper-V isolation). 89 [This](https://stefanscherer.github.io/find-dependencies-in-windows-containers/) is a useful guide on how to use `procmon.exe`. 90 91 Because of the docker buildx limitations regarding the `RUN` commands mentioned above, we need to use a 92 Windows helper image in order to properly build the Windows images without requiring any Windows nodes in 93 the regular image building process. This helper image can be found in at `e2eteam/powershell-helper:6.2.7`. 94 It can be used by anyone, but if you need to build your own, you can read more about it [here](windows/README.md). 95 96 For Windows, in order to spawn process-isolated containers, the container OS version should closely match 97 the host OS version. For this reason, we build test images for different Windows OS Versions: 1809 (Windows Server 2019) 98 and ltsc2022 (Windows Server 2022). In order to add support for a new Windows OS version, a new entry for that OS version will have 99 to be first added to the `windows-servercore-cache` and `busybox` images, followed by the rest of the images. 100 These images are then used by the rest of the E2E test images as a cache / base image. 101 102 ## Making changes to an image 103 104 Make updates to the functionality of the images required for your test case and update the version number. 105 106 The version can easily be bumped by modifying the file `test/images/${IMAGE_NAME}/VERSION`, which will 107 be used when building the image. 108 109 Some test images (`agnhost`) are used as bases for other images (`kitten`, `nautilus`). If the parent 110 image's `VERSION` has been bumped, also bump the version in the children's `BASEIMAGE` files in order 111 for base image changes to be reflected in the child images as well. 112 113 Keep in mind that the Kubernetes CI will not run with the image changes you've made until promoted. It is a good idea 114 to build the image and push it to your own registry first, and run some tests that are using that image. Continue with the steps below to build, test and promote the changes. 115 116 ## Building images 117 118 The images are built through `make`. Since some images (`agnhost`) are used as a base for other images, 119 it is recommended to build them first, if needed. 120 121 An image can be built by simply running the command: 122 123 ```bash 124 make all WHAT=agnhost 125 ``` 126 127 To build AND push an image, the following command can be used: 128 129 ```bash 130 make all-push WHAT=agnhost 131 ``` 132 133 By default, the images will be tagged and pushed under the `registry.k8s.io/e2e-test-images` 134 registry. That can changed by running this command instead: 135 136 ```bash 137 REGISTRY=foo_registry make all-push WHAT=agnhost 138 ``` 139 140 *NOTE* (for test `gcr.io` image publishers): Some tests (e.g.: `should serve a basic image on each replica with a private image`) 141 require the `agnhost` image to be published in an authenticated repo as well: 142 143 ```bash 144 REGISTRY=registry.k8s.io/e2e-test-images make all-push WHAT=agnhost 145 REGISTRY=gcr.io/k8s-authenticated-test make all-push WHAT=agnhost 146 ``` 147 148 Additionally, `WHAT=all-conformance` can be used to build / push the images most commonly used in E2E 149 Conformance tests. 150 151 152 ## Testing images 153 154 Once the image has been built and pushed to an accessible registry, you can run the tests using that image 155 by having the environment variable `KUBE_TEST_REPO_LIST` set before running the tests that are using the 156 image: 157 158 ```bash 159 export KUBE_TEST_REPO_LIST=/path/to/repo_list.yaml 160 ``` 161 162 `repo_list.yaml` is a configuration file used by the E2E tests, in which you can set alternative registries 163 to pull the images from. Sample file: 164 165 ```yaml 166 promoterE2eRegistry: your-awesome-registry 167 gcRegistry: your-awesome-registry 168 sampleRegistry: your-awesome-registry 169 ``` 170 171 Keep in mind that some tests are using multiple images, so it is a good idea to also build and push those images. 172 173 Finally, make sure to bump the image version used in E2E testing by modifying the file `test/utils/image/manifest.go`, and recompile afterwards: 174 175 ```bash 176 ./build/run.sh make WHAT=test/e2e/e2e.test 177 ``` 178 179 After all the above has been done, run the [desired tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md) to make sure your changes work. 180 181 ### Promoting Images 182 183 Now that you have made the changes and tested locally, you are ready to share those changes. This is a multi step process: 184 185 1. In the same pull request containing your proposed changes, bump the version of the image in question. Each test image has 186 a VERSION file in its directory. For example, the agnhost image's VERSION file is in `test/images/agnhost/VERSION`. 187 2. After the pull request has been approved and merged, an **automatic** postsubmit 188 job will then be triggered which will build the images that were changed. For example, if a change was 189 made in `test/images/agnhost`, then the job [post-kubernetes-push-e2e-agnhost-test-images]( 190 https://testgrid.k8s.io/sig-testing-images#post-kubernetes-push-e2e-agnhost-test-images) 191 will be triggered. The postsubmit job will push the images to the `gcr.io/k8s-staging-e2e-test-images` registry. You can use the image 192 from the staging registry to do more testing if required. All the postsubmit jobs and their logs for all the images can be seen in 193 [testgrid](https://testgrid.k8s.io/sig-testing-images) which is helpful for troubleshooting. Note that these images are not the same as used by the e2e jobs and still need to be promoted to the final registry. 194 1. The next step is to promote the image to the `registry.k8s.io/e2e-test-images` registry by adding a line in 195 [kubernetes/k8s.io](https://github.com/kubernetes/k8s.io/blob/main/registry.k8s.io/images/k8s-staging-e2e-test-images/images.yaml). See this [pull request](https://github.com/kubernetes/k8s.io/pull/1804) for an example You will need the image manifest list's digest, which can be obtained by using [manifest-tool](https://github.com/estesp/manifest-tool): 196 197 ```bash 198 manifest-tool inspect --raw gcr.io/k8s-staging-e2e-test-images/${IMAGE_NAME}:${VERSION} | jq '.[0].Digest' 199 ``` 200 1. Finally you can open a pull request to update the e2e tests to use the newly promoted image by updating the image tag in [test/utils/image/manifest.go](../utils/image/manifest.go) file. See this [pull request](https://github.com/kubernetes/kubernetes/pull/100383/files) for an example. 201 202 You have now gone through the entire process and your changes will be used by the e2e tests. 203 204 ## Known issues and workarounds 205 206 `docker manifest create` fails due to permission denied on `/etc/docker/certs.d/gcr.io` (https://github.com/docker/for-linux/issues/396). This issue can be resolved by running: 207 208 ```bash 209 sudo chmod o+x /etc/docker 210 ``` 211 212 A few images have been mirrored from dockerhub into the `gcr.io/k8s-staging-e2e-test-images` registry 213 (`busybox`,`httpd`, `httpd-new`, `nginx`, `nginx-new`, `perl`), and they 214 only have a noop Dockerfile. However, due to an [issue](https://github.com/kubernetes/test-infra/issues/20884), 215 the same SHA cannot be pushed twice. A small change to them is required in order to generate a new SHA, 216 which can then be pushed and promoted.