github.com/shashidharatd/test-infra@v0.0.0-20171006011030-71304e1ca560/kubetest/README.md (about)

     1  Kubetest is the interface for launching and running e2e tests.
     2  
     3  See the contributor documentation for information about [e2e testing].
     4  
     5  Kubetest sits between [bootstrap.py] and various parts of the e2e lifecycle.
     6  
     7  The `bootstrap.py` library is a nominal/optional part of [prow].
     8  This library is responsible for:
     9  * checking out each repository correctly
    10  * starting kubetest (or whatever the test binary is for other jobs)
    11  * uploading the test result (including artifacts) to gcs
    12  
    13  The e2e lifecycle may:
    14  * `--build` kubernetes,
    15  * `--stage` this build to gcs,
    16  * `--extract` a staged build from gcs,
    17  * turn `--up` a new cluster using various `--deployment` strategies,
    18  * `--test` this cluster using [ginkgo] `--test_args`
    19  * `--dump` logs to a local folder, and finally
    20  * turn `--down` the cluster after completing testing,
    21  * `--timeout` after a particular duration (allowing extra time to clean up).
    22  
    23  Note that developers frequently use `kubetest` by calling `go run hack/e2e.go`
    24  in the `kubernetes/kubernetes` repository. This `hack/e2e.go` program is a
    25  wrapper around updating `kubetest` (at most once a day) before calling it.
    26  
    27  ## Installation
    28  
    29  Please run `go get -u k8s.io/test-infra/kubetest` to install kubetest.
    30  
    31  Common alternatives:
    32  ```
    33  go run hack/e2e.go  # from kubernetes/kubernetes
    34  go install k8s.io/test-infra/kubetest  # if you check out test-infra
    35  bazel run //kubetest  # use bazel to build and run
    36  ```
    37  
    38  ### Releases
    39  
    40  Right now `kubetest` is expected to run at head, regardless of the version of
    41  kubernetes being targeted.
    42  
    43  Most e2e images, such as [kubekins-e2e] and [kubekins-e2e-prow] compile the
    44  latest version of kubetest whenever the image is updated (most updates to these
    45  images are done in order to update kubetest).
    46  
    47  
    48  ## Build
    49  
    50  If `PWD` is in the `kubernetes/kubernetes` directory `--build` will build
    51  whatever changes you have made into a quick release.
    52  
    53  Control the details of the `--build=bazel` by appending one of the build modes
    54  (see help for current list).
    55  
    56  ### Stage a build
    57  
    58  It is inefficient for every job to rebuild the same version. Instead our CI
    59  system defines build jobs which `--stage` the build somewhere on GCS. Some
    60  providers such as GKE require a staged build, whereas others like GCE allow you
    61  to `scp` over the binaries directly to each node.
    62  
    63  
    64  ### Extract a build
    65  
    66  Aside from the build jobs, most of our CI systems `--extract` a prebuilt
    67  version. This saves a bunch of time compiling.
    68  
    69  The most common options are either a specific version`--extract=v1.7.0-beta.1`,
    70  a release `--extract=release/stable` or `--extract=ci/latest-1.8`.
    71  
    72  Note that you can extract 1 or 2 versions. Using 2 versions is useful for skew
    73  and upgrade testing.
    74  
    75  See [extract.go] for further details.
    76  
    77  
    78  ## Cluster-lifecycle
    79  
    80  There are various ways to deploy kubernetes. Choose a strategy with the
    81  `--deployment=kubernetes-anywhere` flag. See kubetest help for current list of
    82  options.
    83  
    84  ### Up
    85  
    86  The `--up` flag will tell `kubetest` to turn up a new cluster for you.
    87  
    88  It will first attempt to tear down an old instance of the same cluster.
    89  
    90  Currently requires a complicated set of flags and environment variables
    91  such as `--gcp-project`, `--federation`, etc.
    92  
    93  We are in the process of converting all environment variables into flags. See
    94  the current set of flag options with `kubetest -h`.
    95  
    96  #### Save/load credentials
    97  
    98  The `--save` flag tells kubetest to upload your cluster credentials onto gcs
    99  somewhere. Later calling `kubetest --save` without an `--up` flag tells kubetest
   100  to load these credentials instead of turning up a new cluster.
   101  
   102  
   103  #### Dynamic project selection
   104  
   105  Most e2e jobs assume control of a GCP project (see leaks section below).
   106  
   107  If `kubetest` is running inside a pod then it will attempt to talk to [boskos]
   108  to dynamically reserve a project.
   109  
   110  This makes it easier for developers to add and remove jobs. With boskos they no
   111  longer need to worry about creating, provisioning, naming, etc a project for
   112  this new job.
   113  
   114  See the boskos docs for more details.
   115  
   116  ### Dump logs
   117  
   118  The `--dump` flag tells `kubetest` to try and collect logs if there is a
   119  problem. Typically this means master and node logs.
   120  
   121  Collecting these logs may take a long time. This typically involves sshing to
   122  each node, searching for and downloading any relevant logs.
   123  
   124  There is also a `--logexporter-gcs-path` option which tells `kubetest` to run a
   125  container on each node which uploads logs directly to GCS. This dramatically
   126  reduces time required to dump logs, especially for scalability tests.
   127  
   128  ### Down
   129  
   130  The `--down` flag tells `kubetest` to clear up the cluster after finishing.
   131  
   132  Kubetest will try its best to tear down the cluster in spite of problems such as
   133  failing `--up`, `--test`, etc. The `--timeout` also includes some buffer to
   134  allow time for `--down` to clean up.
   135  
   136  #### Leaks
   137  
   138  The `--check-leaked-resources` option tells kubetest to look for any extra `GCP`
   139  resources after tearing down the cluster.
   140  
   141  The expectation is that any resources created by kubernetes will be cleaned up
   142  during cluster teardown.
   143  
   144  This logic may be buggy so this options takes a snapshot of the resources at
   145  various points in time (start, after cluster up, after testing, after cluster
   146  down) and ensures that there are no resources present after down that weren't
   147  already present at the start.
   148  
   149  
   150  ## Testing
   151  
   152  Most testing uses ginkgo but there are other options available.
   153  
   154  ### Ginkgo
   155  
   156  The `--test` flag tells `kubetest` to run the `test.e2e` binary built/extracted
   157  from the `kubernetes/kubernetes` repo.
   158  
   159  Typically jobs also include a `--test_args=--ginkgo.focus=FOO --ginkgo.skip=BAR`
   160  flag to filter down to a particular set of interesting tests.
   161  
   162  ### Upgrade, skew, kubemark, federation
   163  
   164  You can also run `--kubemark` or `--federation` tests instead of the standard
   165  tests.
   166  
   167  Tests can use `--skew` and `--upgrade_args` if they provided multiple
   168  `--extract` flags (or manually created a `kubernetes/kubernetes_skew` directory
   169  as a sibling to `kubernetes/kubernetes`. This will cause tests to run from the
   170  skew directory, potentially to upgrade/downgrade kubernetes to another version.
   171  
   172  A simple example is:<br>
   173  `kubetest --up --check-version-skew=false --extract=v1.8.0-beta.1 
   174  --extract=v1.7.5 --upgrade_args=--ginkgo.focus="Feature:SomeUpgrade" 
   175  --gcp-project=google.com:some-project -v`
   176  
   177  The command runs all (and only) upgrade tests tagged with `Feature:SomeUpgrade` 
   178  label on GCE. The command downloads `v1.7.5` and `v1.8.0-beta.1` releases,
   179  unzips downloaded files, and runs the tests to upgrade the cluster from `v1.7.5` 
   180  to `v1.8.0-beta.1`. You will be able to find 2 new directories named `kubernetes`
   181  and `kubernetes_skew` at current directory. `kubernetes` is the directory
   182  corresponding to release indicated by first `--extract` flag, while `kubernetes_skew` 
   183  corresponds to second flag.
   184  
   185  Note that order of the 2 `--extract` flags matters: `--extract=v2 --extract=v1` means
   186  upgrading from v1 to v2. The command does not run other e2e tests after completing 
   187  the upgrade tests. If you want to run the e2e tests, specify also `--test` and 
   188  `--test_args` flags.
   189  
   190  Tips: CI upgrade tests listed at `jobs/config.json` show flags used in the real CI 
   191  test environment, which is a good source to learn more about how the flags are used.
   192  
   193  ### Staging
   194  
   195  If you want to create a new release with your own changes, you have to upload built 
   196  manifests to gcs. The command is very similar:
   197  `kubetest --build --stage=gs://some/path/to/v1.8.0-beta.1  --check-version-skew=false 
   198  --extract=gs://some/path/to/v1.8.0-beta.1  --extract=v1.7.5 
   199  --upgrade_args=--ginkgo.focus="Feature:SomeUpgrade" 
   200  --gcp-project=google.com:some-project -v`
   201  
   202  If you already have release of a specific version, you do not need to fetch the 
   203  release again. For instance, if you have `v1.7.5` release and its directory is at the 
   204  right path, the command below does the same as above:
   205  `kubetest --build --stage=gs://some/path/to/v1.8.0-beta.1  --check-version-skew=false 
   206  --extract=gs://some/path/to/v1.8.0-beta.1 
   207  --upgrade_args=--ginkgo.focus="Feature:SomeUpgrade" 
   208  --gcp-project=google.com:some-project -v`
   209  
   210  [bootstrap.py]: /jenkins/bootstrap.py
   211  [boskos]: /boskos
   212  [e2e testing]: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md
   213  [extract.go]: /kubetest/extract.go
   214  [ginkgo]: https://github.com/onsi/ginkgo
   215  [kubekins-e2e]: /jenkins/e2e-image
   216  [kubekins-e2e-prow]: /images/e2e-prow
   217  [prow]: /prow