github.com/yrj2011/jx-test-infra@v0.0.0-20190529031832-7a2065ee98eb/kubetest/README.md (about) 1 # Kubetest 2 3 Kubetest is the interface for launching and running e2e tests. 4 5 See the contributor documentation for information about [e2e testing]. 6 7 Kubetest sits between [bootstrap.py] and various parts of the e2e lifecycle. 8 9 The `bootstrap.py` library is a nominal/optional part of [prow]. 10 This library is responsible for: 11 * checking out each repository correctly 12 * starting kubetest (or whatever the test binary is for other jobs) 13 * uploading the test result (including artifacts) to gcs 14 15 The e2e lifecycle may: 16 * `--build` kubernetes, 17 * `--stage` this build to gcs, 18 * `--extract` a staged build from gcs, 19 * turn `--up` a new cluster using various `--deployment` strategies, 20 * `--test` this cluster using [ginkgo] `--test_args` 21 * `--dump` logs to a local folder, and finally 22 * turn `--down` the cluster after completing testing, 23 * `--timeout` after a particular duration (allowing extra time to clean up). 24 25 Note that developers frequently use `kubetest` by calling `go run hack/e2e.go` 26 in the `kubernetes/kubernetes` repository. This `hack/e2e.go` program is a 27 wrapper around updating `kubetest` (at most once a day) before calling it. 28 29 If you're making a change to kubetest, watch the canary jobs after it has merged; 30 these run the latest images of `kubekins-e2e`, and thus don't have to wait 31 for a version bump in the prow config. The canary jobs are used to give early 32 signal on whether new kubetest features are working as intended, before bumping 33 the image for general e2e tests. 34 35 ## Installation 36 37 Please run `go get -u k8s.io/test-infra/kubetest` to install kubetest. 38 39 Common alternatives: 40 ``` 41 go run hack/e2e.go # from kubernetes/kubernetes 42 go install k8s.io/test-infra/kubetest # if you check out test-infra 43 bazel run //kubetest # use bazel to build and run 44 ``` 45 46 ### Releases 47 48 Right now `kubetest` is expected to run at head, regardless of the version of 49 kubernetes being targeted. 50 51 Most e2e images, such as [kubekins-e2e] and [kubekins-e2e-prow] compile the 52 latest version of kubetest whenever the image is updated (most updates to these 53 images are done in order to update kubetest). 54 55 56 ## Build 57 58 If `PWD` is in the `kubernetes/kubernetes` directory `--build` will build 59 whatever changes you have made into a quick release. 60 61 Control the details of the `--build=bazel` by appending one of the build modes 62 (see help for current list). 63 64 ### Stage a build 65 66 It is inefficient for every job to rebuild the same version. Instead our CI 67 system defines build jobs which `--stage` the build somewhere on GCS. Some 68 providers such as GKE require a staged build, whereas others like GCE allow you 69 to `scp` over the binaries directly to each node. 70 71 72 ### Extract a build 73 74 Aside from the build jobs, most of our CI systems `--extract` a prebuilt 75 version. This saves a bunch of time compiling. 76 77 The most common options are either a specific version`--extract=v1.7.0-beta.1`, 78 a release `--extract=release/stable` or `--extract=ci/latest-1.8`. 79 80 Note that you can extract 1 or 2 versions. Using 2 versions is useful for skew 81 and upgrade testing. 82 83 See [extract_k8s.go] for further details. 84 85 86 ## Cluster-lifecycle 87 88 There are various ways to deploy kubernetes. Choose a strategy with the 89 `--deployment` flag (for example `--deployment=kops` or `--deployment=kubernetes-anywhere`). 90 See `kubetest --help` for a full list of options. 91 92 ### Up 93 94 The `--up` flag will tell `kubetest` to turn up a new cluster for you. 95 96 It will first attempt to tear down an old instance of the same cluster. 97 98 Currently requires a complicated set of flags and environment variables 99 such as `--gcp-project`, `--federation`, etc. 100 101 We are in the process of converting all environment variables into flags. See 102 the current set of flag options with `kubetest -h`. 103 104 #### Save/load credentials 105 106 The `--save` flag tells kubetest to upload your cluster credentials onto gcs 107 somewhere. Later calling `kubetest --save` without an `--up` flag tells kubetest 108 to load these credentials instead of turning up a new cluster. 109 110 111 #### Dynamic project selection 112 113 Most e2e jobs assume control of a GCP project (see leaks section below). 114 115 If `kubetest` is running inside a pod then it will attempt to talk to [boskos] 116 to dynamically reserve a project. 117 118 This makes it easier for developers to add and remove jobs. With boskos they no 119 longer need to worry about creating, provisioning, naming, etc a project for 120 this new job. 121 122 See the boskos docs for more details. 123 124 ### Dump logs 125 126 The `--dump` flag tells `kubetest` to try and collect logs if there is a 127 problem. Typically this means master and node logs. 128 129 Collecting these logs may take a long time. This typically involves sshing to 130 each node, searching for and downloading any relevant logs. 131 132 There is also a `--logexporter-gcs-path` option which tells `kubetest` to run a 133 container on each node which uploads logs directly to GCS. This dramatically 134 reduces time required to dump logs, especially for scalability tests. 135 136 ### Down 137 138 The `--down` flag tells `kubetest` to clear up the cluster after finishing. 139 140 Kubetest will try its best to tear down the cluster in spite of problems such as 141 failing `--up`, `--test`, etc. The `--timeout` also includes some buffer to 142 allow time for `--down` to clean up. 143 144 #### Leaks 145 146 The `--check-leaked-resources` option tells kubetest to look for any extra `GCP` 147 resources after tearing down the cluster. 148 149 The expectation is that any resources created by kubernetes will be cleaned up 150 during cluster teardown. 151 152 This logic may be buggy so this options takes a snapshot of the resources at 153 various points in time (start, after cluster up, after testing, after cluster 154 down) and ensures that there are no resources present after down that weren't 155 already present at the start. 156 157 158 ## Testing 159 160 Most testing uses ginkgo but there are other options available. 161 162 ### Ginkgo 163 164 The `--test` flag tells `kubetest` to run the `test.e2e` binary built/extracted 165 from the `kubernetes/kubernetes` repo. 166 167 Typically jobs also include a `--test_args=--ginkgo.focus=FOO --ginkgo.skip=BAR` 168 flag to filter down to a particular set of interesting tests. 169 170 ### Upgrade, skew, kubemark, federation 171 172 You can also run `--kubemark` or `--federation` tests instead of the standard 173 tests. 174 175 Tests can use `--skew` and `--upgrade_args` if they provided multiple 176 `--extract` flags (or manually created a `kubernetes/kubernetes_skew` directory 177 as a sibling to `kubernetes/kubernetes`). This will cause tests to run from the 178 skew directory, potentially to upgrade/downgrade kubernetes to another version. 179 180 A simple example is:<br> 181 `kubetest --up --check-version-skew=false --extract=v1.8.0-beta.1 182 --extract=v1.7.5 --upgrade_args=--ginkgo.focus="Feature:SomeUpgrade" 183 --gcp-project=google.com:some-project` 184 185 The command runs all (and only) upgrade tests tagged with `Feature:SomeUpgrade` 186 label on GCE. The command downloads `v1.7.5` and `v1.8.0-beta.1` releases, 187 unzips downloaded files, and runs the tests to upgrade the cluster from `v1.7.5` 188 to `v1.8.0-beta.1`. You will be able to find 2 new directories named `kubernetes` 189 and `kubernetes_skew` at current directory. `kubernetes` is the directory 190 corresponding to release indicated by first `--extract` flag, while `kubernetes_skew` 191 corresponds to second flag. 192 193 Note that order of the 2 `--extract` flags matters: `--extract=v2 --extract=v1` means 194 upgrading from v1 to v2. The command does not run other e2e tests after completing 195 the upgrade tests. If you want to run the e2e tests, specify also `--test` and 196 `--test_args` flags. 197 198 Tips: CI upgrade tests listed at `jobs/config.json` show flags used in the real CI 199 test environment, which is a good source to learn more about how the flags are used. 200 201 ### Staging 202 203 If you want to create a new release with your own changes, you have to upload built 204 manifests to gcs. The command is very similar: 205 `kubetest --build --stage=gs://some/path/to/v1.8.0-beta.1 --check-version-skew=false 206 --extract=gs://some/path/to/v1.8.0-beta.1 --extract=v1.7.5 207 --upgrade_args=--ginkgo.focus="Feature:SomeUpgrade" 208 --gcp-project=google.com:some-project` 209 210 If you already have release of a specific version, you do not need to fetch the 211 release again. For instance, if you have `v1.7.5` release and its directory is at the 212 right path, the command below does the same as above: 213 `kubetest --build --stage=gs://some/path/to/v1.8.0-beta.1 --check-version-skew=false 214 --extract=gs://some/path/to/v1.8.0-beta.1 215 --upgrade_args=--ginkgo.focus="Feature:SomeUpgrade" 216 --gcp-project=google.com:some-project` 217 218 [bootstrap.py]: /jenkins/bootstrap.py 219 [boskos]: /boskos 220 [e2e testing]: https://git.k8s.io/community/contributors/devel/e2e-tests.md 221 [extract_k8s.go]: /kubetest/extract_k8s.go 222 [ginkgo]: https://github.com/onsi/ginkgo 223 [kubekins-e2e]: /images/kubekins-e2e 224 [kubekins-e2e-prow]: /images/e2e-prow 225 [prow]: /prow