github.com/abayer/test-infra@v0.0.5/README.md (about) 1 # Kubernetes Test Infrastructure 2 3 [](https://travis-ci.org/kubernetes/test-infra) [](https://goreportcard.com/report/github.com/kubernetes/test-infra) [](https://godoc.org/github.com/kubernetes/test-infra) 4 5 The test-infra repository contains a collection of tools for testing Kubernetes 6 and displaying Kubernetes tests results. See also [CONTRIBUTING.md](CONTRIBUTING.md). 7 8 See the [architecture diagram](docs/architecture.svg) for an overview of how 9 the different services interact. 10 11 ## Viewing test results 12 13 * The [Kubernetes TestGrid](https://k8s-testgrid.appspot.com/) shows historical test results 14 - Configure your own testgrid dashboard at [testgrid/config.yaml](testgrid/config.yaml) 15 - [Gubernator](https://k8s-gubernator.appspot.com/) formats the output of each run 16 * [PR Dashboard](https://k8s-gubernator.appspot.com/pr) finds PRs that need your attention 17 * [Prow](https://prow.k8s.io) schedules testing and updates issues 18 - Prow responds to GitHub events, timers and [manual commands](https://go.k8s.io/bot-commands) 19 given in GitHub comments. 20 - The [prow dashboard](https://prow.k8s.io/) shows what it is currently testing 21 - Configure prow to run new tests at [config/jobs](config/jobs) 22 * [Triage Dashboard](https://go.k8s.io/triage) aggregates failures 23 - Triage clusters together similar failures 24 - Search for test failures across jobs 25 - Filter down failures in a specific regex of tests and/or jobs 26 * [Velodrome metrics](http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1) track job and test health. 27 - [Kettle](kettle) does collection, [metrics](metrics) does reporting, and [velodrome](velodrome) is the frontend. 28 29 ## Automated testing 30 31 Test anything with the following pattern: 32 33 ``` 34 git clone https://github.com/kubernetes/test-infra 35 test-infra/jenkins/bootstrap.py --job=J --repo=R --service-account=S.json --upload=gs://B 36 ``` 37 38 The `--job=J` flag specifies what test job to run. 39 The `--repo=R` (or `--bare`) flag controls what we check out from git. 40 41 Anyone can reconfigure our CI system with a test-infra PR that updates the 42 appropriate files. Detailed instructions follow: 43 44 ### E2E Testing 45 46 Our e2e testing uses [kubetest](/kubetest) to build/deploy/test kubernetes 47 clusters on various providers. Please see those documents for additional details 48 about this tool as well as e2e testing generally. 49 50 ### Create a new job 51 52 Create a PR in this repo to add/update/remove a job or suite. Specifically 53 you'll need to do the following: 54 * Add the job to the appropriate section in [`config/jobs`](config/jobs) 55 - Directory Structure: 56 - In general for jobs for github.com/org/repo use config/jobs/org/repo/filename.yaml 57 - For Kubernetes repos we also allow config/jobs/kubernetes/sig-foo/filename.yaml 58 - We use basename of the config name as a key in the prow configmap, so the name of your config file need to be unique across the config subdir 59 - Type of jobs: 60 - Presubmit jobs run on unmerged code in PRs 61 - Postsubmit jobs run after merging code 62 - Periodic job run on a timed basis 63 - You can find more prowjob definitions at [how-to-add-new-jobs](prow#how-to-add-new-jobs) 64 - Scenario args: (if you are using [bootstrap.py](jenkins/bootstrap.py) instead of [podutils](prow/pod-utilities.md)) 65 - [Scenarios](scenarios) are python wrappers used by our entry point script [bootstrap.py](jenkins/bootstrap.py). 66 - You can append scenario/kubetest args inline in your prowjob definition, example: 67 ```yaml 68 - name: foo-repo-test 69 interval: 1h 70 agent: kubernetes 71 spec: 72 containers: 73 - image: gcr.io/k8s-testimages/kubekins-e2e:latest-master 74 args: 75 - --repo=github.com/org/repo 76 - --timeout=90 77 - --scenario=execute 78 - -- 79 - make 80 - test 81 ``` 82 83 * Add the job name to the `test_groups` list in [`testgrid/config.yaml`](testgrid/config.yaml) 84 - Also the group to at least one `dashboard_tab` 85 86 The configs need to be sorted and kubernetes must be in sync with the security repo, or else presubmit will fail. 87 You can run the script below to keep them valid: 88 ``` 89 hack/update-config.sh 90 ``` 91 92 NOTE: `kubernetes/kubernetes` and `kubernetes-security/kubernetes` must have matching presubmits. 93 94 Please test the job on your local workstation before creating a PR: 95 ``` 96 mkdir /tmp/whatever && cd /tmp/whatever 97 $GOPATH/src/k8s.io/test-infra/jenkins/bootstrap.py \ 98 --job=J \ # aka your new job 99 --repo=R1 --repo=R2 \ # what repos to check out 100 --service-account ~/S.json # the service account to use to launch GCE/GKE clusters 101 # Note: create a service account at the cloud console for the project J uses 102 ``` 103 104 #### Release branch jobs & Image validation jobs 105 106 Release branch jobs and image validation jobs are defined in [test_config.yaml](experiment/test_config.yaml). 107 We test different master/node image versions against multiple k8s branches on different features. 108 109 Those jobs are using channel based versions, current supported testing map is: 110 - k8s-dev : master 111 - k8s-beta : release-1.11 112 - k8s-stable1 : release-1.10 113 - k8s-stable2 : release-1.9 114 - k8s-stable3 : release-1.8 115 116 Our build job will generate a ci/(channel-name) file pointer in gcs. 117 118 After you update [test_config.yaml](experiment/test_config.yaml), please run 119 120 ``` 121 bazel run //experiment:generate_tests -- --yaml-config-path=experiment/test_config.yaml 122 ``` 123 124 to regenerate the job configs. 125 126 We are moving towards making more jobs to fit into the generated config. 127 128 129 Presubmit will tell you if you forget to do any of this correctly. 130 131 Merge your PR and [@k8s-ci-robot] will deploy your change automatically. 132 133 ### Update an existing job 134 135 Largely similar to creating a new job, except you can just modify the existing 136 entries rather than adding new ones. 137 138 Update what a job does by editing its definition in [`config/jobs`](config/jobs). 139 140 Update where the job appears on testgrid by changing [`testgrid/config.yaml`]. 141 142 ### Delete a job 143 144 The reverse of creating a new job: delete the appropriate entries in 145 [`config/jobs`] and [`testgrid/config.yaml`]. 146 147 Merge your PR and [@k8s-ci-robot] will deploy your change automatically. 148 149 ## Building and testing the test-infra 150 151 We use [Bazel](https://www.bazel.io/) to build and test the code in this repo. 152 The commands `bazel build //...` and `bazel test //...` should be all you need 153 for most cases. If you modify Go code, run `./hack/update-bazel.sh` to keep 154 `BUILD.bazel` files up-to-date. 155 156 ## Contributing Test Results 157 158 The Kubernetes project encourages organizations to contribute execution of e2e 159 test jobs for a variety of platforms (e.g., Azure, rktnetes). For information about 160 how to contribute test results, see [Contributing Test Results](docs/contributing-test-results.md). 161 162 ## Other Docs 163 164 * [kubernetes/test-infra dependency management](docs/dep.md) 165 166 167 [`config/jobs`]: /config/jobs 168 [`testgrid/config.yaml`]: /testgrid/config.yaml 169 [test-infra oncall]: https://go.k8s.io/oncall 170 [@k8s-ci-robot]: (https://github.com/k8s-ci-robot)