github.com/shashidharatd/test-infra@v0.0.0-20171006011030-71304e1ca560/kubetest/README.md (about) 1 Kubetest is the interface for launching and running e2e tests. 2 3 See the contributor documentation for information about [e2e testing]. 4 5 Kubetest sits between [bootstrap.py] and various parts of the e2e lifecycle. 6 7 The `bootstrap.py` library is a nominal/optional part of [prow]. 8 This library is responsible for: 9 * checking out each repository correctly 10 * starting kubetest (or whatever the test binary is for other jobs) 11 * uploading the test result (including artifacts) to gcs 12 13 The e2e lifecycle may: 14 * `--build` kubernetes, 15 * `--stage` this build to gcs, 16 * `--extract` a staged build from gcs, 17 * turn `--up` a new cluster using various `--deployment` strategies, 18 * `--test` this cluster using [ginkgo] `--test_args` 19 * `--dump` logs to a local folder, and finally 20 * turn `--down` the cluster after completing testing, 21 * `--timeout` after a particular duration (allowing extra time to clean up). 22 23 Note that developers frequently use `kubetest` by calling `go run hack/e2e.go` 24 in the `kubernetes/kubernetes` repository. This `hack/e2e.go` program is a 25 wrapper around updating `kubetest` (at most once a day) before calling it. 26 27 ## Installation 28 29 Please run `go get -u k8s.io/test-infra/kubetest` to install kubetest. 30 31 Common alternatives: 32 ``` 33 go run hack/e2e.go # from kubernetes/kubernetes 34 go install k8s.io/test-infra/kubetest # if you check out test-infra 35 bazel run //kubetest # use bazel to build and run 36 ``` 37 38 ### Releases 39 40 Right now `kubetest` is expected to run at head, regardless of the version of 41 kubernetes being targeted. 42 43 Most e2e images, such as [kubekins-e2e] and [kubekins-e2e-prow] compile the 44 latest version of kubetest whenever the image is updated (most updates to these 45 images are done in order to update kubetest). 46 47 48 ## Build 49 50 If `PWD` is in the `kubernetes/kubernetes` directory `--build` will build 51 whatever changes you have made into a quick release. 52 53 Control the details of the `--build=bazel` by appending one of the build modes 54 (see help for current list). 55 56 ### Stage a build 57 58 It is inefficient for every job to rebuild the same version. Instead our CI 59 system defines build jobs which `--stage` the build somewhere on GCS. Some 60 providers such as GKE require a staged build, whereas others like GCE allow you 61 to `scp` over the binaries directly to each node. 62 63 64 ### Extract a build 65 66 Aside from the build jobs, most of our CI systems `--extract` a prebuilt 67 version. This saves a bunch of time compiling. 68 69 The most common options are either a specific version`--extract=v1.7.0-beta.1`, 70 a release `--extract=release/stable` or `--extract=ci/latest-1.8`. 71 72 Note that you can extract 1 or 2 versions. Using 2 versions is useful for skew 73 and upgrade testing. 74 75 See [extract.go] for further details. 76 77 78 ## Cluster-lifecycle 79 80 There are various ways to deploy kubernetes. Choose a strategy with the 81 `--deployment=kubernetes-anywhere` flag. See kubetest help for current list of 82 options. 83 84 ### Up 85 86 The `--up` flag will tell `kubetest` to turn up a new cluster for you. 87 88 It will first attempt to tear down an old instance of the same cluster. 89 90 Currently requires a complicated set of flags and environment variables 91 such as `--gcp-project`, `--federation`, etc. 92 93 We are in the process of converting all environment variables into flags. See 94 the current set of flag options with `kubetest -h`. 95 96 #### Save/load credentials 97 98 The `--save` flag tells kubetest to upload your cluster credentials onto gcs 99 somewhere. Later calling `kubetest --save` without an `--up` flag tells kubetest 100 to load these credentials instead of turning up a new cluster. 101 102 103 #### Dynamic project selection 104 105 Most e2e jobs assume control of a GCP project (see leaks section below). 106 107 If `kubetest` is running inside a pod then it will attempt to talk to [boskos] 108 to dynamically reserve a project. 109 110 This makes it easier for developers to add and remove jobs. With boskos they no 111 longer need to worry about creating, provisioning, naming, etc a project for 112 this new job. 113 114 See the boskos docs for more details. 115 116 ### Dump logs 117 118 The `--dump` flag tells `kubetest` to try and collect logs if there is a 119 problem. Typically this means master and node logs. 120 121 Collecting these logs may take a long time. This typically involves sshing to 122 each node, searching for and downloading any relevant logs. 123 124 There is also a `--logexporter-gcs-path` option which tells `kubetest` to run a 125 container on each node which uploads logs directly to GCS. This dramatically 126 reduces time required to dump logs, especially for scalability tests. 127 128 ### Down 129 130 The `--down` flag tells `kubetest` to clear up the cluster after finishing. 131 132 Kubetest will try its best to tear down the cluster in spite of problems such as 133 failing `--up`, `--test`, etc. The `--timeout` also includes some buffer to 134 allow time for `--down` to clean up. 135 136 #### Leaks 137 138 The `--check-leaked-resources` option tells kubetest to look for any extra `GCP` 139 resources after tearing down the cluster. 140 141 The expectation is that any resources created by kubernetes will be cleaned up 142 during cluster teardown. 143 144 This logic may be buggy so this options takes a snapshot of the resources at 145 various points in time (start, after cluster up, after testing, after cluster 146 down) and ensures that there are no resources present after down that weren't 147 already present at the start. 148 149 150 ## Testing 151 152 Most testing uses ginkgo but there are other options available. 153 154 ### Ginkgo 155 156 The `--test` flag tells `kubetest` to run the `test.e2e` binary built/extracted 157 from the `kubernetes/kubernetes` repo. 158 159 Typically jobs also include a `--test_args=--ginkgo.focus=FOO --ginkgo.skip=BAR` 160 flag to filter down to a particular set of interesting tests. 161 162 ### Upgrade, skew, kubemark, federation 163 164 You can also run `--kubemark` or `--federation` tests instead of the standard 165 tests. 166 167 Tests can use `--skew` and `--upgrade_args` if they provided multiple 168 `--extract` flags (or manually created a `kubernetes/kubernetes_skew` directory 169 as a sibling to `kubernetes/kubernetes`. This will cause tests to run from the 170 skew directory, potentially to upgrade/downgrade kubernetes to another version. 171 172 A simple example is:<br> 173 `kubetest --up --check-version-skew=false --extract=v1.8.0-beta.1 174 --extract=v1.7.5 --upgrade_args=--ginkgo.focus="Feature:SomeUpgrade" 175 --gcp-project=google.com:some-project -v` 176 177 The command runs all (and only) upgrade tests tagged with `Feature:SomeUpgrade` 178 label on GCE. The command downloads `v1.7.5` and `v1.8.0-beta.1` releases, 179 unzips downloaded files, and runs the tests to upgrade the cluster from `v1.7.5` 180 to `v1.8.0-beta.1`. You will be able to find 2 new directories named `kubernetes` 181 and `kubernetes_skew` at current directory. `kubernetes` is the directory 182 corresponding to release indicated by first `--extract` flag, while `kubernetes_skew` 183 corresponds to second flag. 184 185 Note that order of the 2 `--extract` flags matters: `--extract=v2 --extract=v1` means 186 upgrading from v1 to v2. The command does not run other e2e tests after completing 187 the upgrade tests. If you want to run the e2e tests, specify also `--test` and 188 `--test_args` flags. 189 190 Tips: CI upgrade tests listed at `jobs/config.json` show flags used in the real CI 191 test environment, which is a good source to learn more about how the flags are used. 192 193 ### Staging 194 195 If you want to create a new release with your own changes, you have to upload built 196 manifests to gcs. The command is very similar: 197 `kubetest --build --stage=gs://some/path/to/v1.8.0-beta.1 --check-version-skew=false 198 --extract=gs://some/path/to/v1.8.0-beta.1 --extract=v1.7.5 199 --upgrade_args=--ginkgo.focus="Feature:SomeUpgrade" 200 --gcp-project=google.com:some-project -v` 201 202 If you already have release of a specific version, you do not need to fetch the 203 release again. For instance, if you have `v1.7.5` release and its directory is at the 204 right path, the command below does the same as above: 205 `kubetest --build --stage=gs://some/path/to/v1.8.0-beta.1 --check-version-skew=false 206 --extract=gs://some/path/to/v1.8.0-beta.1 207 --upgrade_args=--ginkgo.focus="Feature:SomeUpgrade" 208 --gcp-project=google.com:some-project -v` 209 210 [bootstrap.py]: /jenkins/bootstrap.py 211 [boskos]: /boskos 212 [e2e testing]: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md 213 [extract.go]: /kubetest/extract.go 214 [ginkgo]: https://github.com/onsi/ginkgo 215 [kubekins-e2e]: /jenkins/e2e-image 216 [kubekins-e2e-prow]: /images/e2e-prow 217 [prow]: /prow