github.com/shashidharatd/test-infra@v0.0.0-20171006011030-71304e1ca560/docs/federated_testing.md (about) 1 # Federated Testing 2 3 The quality of Kubernetes depends on exercising the software on a 4 variety of platforms. The Kubernetes project welcomes contributions of 5 test results from organizations that execute e2e test jobs. The 6 [24-hour test history 7 dashboard](http://storage.googleapis.com/kubernetes-test-history/static/index.html) 8 displays a summary of all results for the past day. 9 [TestGrid](https://k8s-testgrid.appspot.com/) shows the latest results. 10 11 We are actively working on improving this process, which means that this 12 document may not be kept exactly up-to-date. Feel free to file an issue against 13 this repo when you run into problems. 14 15 16 ## Overview 17 18 To contribute test results for a platform, an organization executes the 19 e2e test cases and uploads the test results to a Google Cloud Storage 20 (GCS) bucket. The test-infra repo contains scripts in `jenkins/` to run the 21 tests and to upload the results. 22 23 The various dashboards will read test results from these buckets. The remainder 24 of this document explains how to contribute test results. 25 26 27 ### Run the e2e tests and upload results 28 29 Setup a periodic test job that will run at least once per day. The test 30 job runs e2e tests and uploads the results. 31 32 1. Create a Google Cloud Storage bucket that will store the test 33 results. The bucket should have public read access. The GCS URL 34 must be stored in an environment variable named 35 `JENKINS_GCS_LOGS_PATH`. For example, Google's Jenkins server stores 36 results to `gs://kubernetes-jenkins/logs/`. 37 2. Run `jenkins/dockerized-e2e-runner.sh` from the test-infra repo. 38 There are several environment variables that affect the behavior of this 39 script. See [e2e-runner 40 Environment Variables](#e2e-runner-environment-variables). This will run 41 the tests from within a docker container. If this won't work for you, you 42 can try running `jenkins/e2e-image/e2e-runner.sh`. 43 3. Run `JENKINS_BUILD_FINISHED={SUCCESS|UNSTABLE|FAILURE|ABORTED} 44 hack/jenkins/upload-to-gcs.sh` from the kubernetes repo to upload the results 45 to the GCS bucket. If running from a Jenkins job, it is recommend to perform 46 this step as a post-build step. See 47 `jenkins/job-configs/kubernetes-jenkins/global.yaml` for an example. 48 49 50 ### Include results in test history 51 52 Collect results from federated test jobs by adding the Google Cloud Storage 53 bucket to [`buckets.yaml`](/buckets.yaml). This will also allow Gubernator to 54 track your jobs. To add buckets to the TestGrid, open an issue against this 55 repo. 56 57 ### e2e-runner Environment Variables 58 59 The following environment variables should be set appropriately before 60 running the `e2e-runner.sh` script: 61 62 - WORKSPACE * 63 - JOB_NAME * 64 - BUILD_NUMBER * 65 - GINKGO_TEST_ARGS 66 - JENKINS_GCS_LOGS_PATH 67 - JENKINS_USE_EXISTING_BINARIES (optional. Set to "y" for locally built binaries.) 68 - E2E_TEST="true" 69 70 \* If using Jenkins to execute the e2e test job, then these environment 71 variables may be set by Jenkins. Otherwise, they must be set before 72 executing `e2e-runner.sh`. See above for how they affect how results 73 are uploaded to a GCS bucket. 74 75 Test results are stored in 76 `gs://$JENKINS_GCS_LOGS_PATH/$JOB_NAME/$BUILD_NUMBER`. Every run should upload 77 `started.json`, `finished.json`, and `build-log.txt`, and can optionally upload 78 JUnit XML to the `artifacts/` directory. See the [Gubernator README](./../gubernator/README.md#job-artifact-gcs-layout) 79 for more information. 80 81 If the platform should be setup or torn down by `e2e-runner.sh`, then 82 optionally set: 83 84 - E2E_UP="true" 85 - E2E_DOWN="true" 86