github.com/shashidharatd/test-infra@v0.0.0-20171006011030-71304e1ca560/docs/federated_testing.md (about)

     1  # Federated Testing
     2  
     3  The quality of Kubernetes depends on exercising the software on a
     4  variety of platforms.  The Kubernetes project welcomes contributions of
     5  test results from organizations that execute e2e test jobs.  The
     6  [24-hour test history
     7  dashboard](http://storage.googleapis.com/kubernetes-test-history/static/index.html)
     8  displays a summary of all results for the past day.
     9  [TestGrid](https://k8s-testgrid.appspot.com/) shows the latest results.
    10  
    11  We are actively working on improving this process, which means that this
    12  document may not be kept exactly up-to-date. Feel free to file an issue against
    13  this repo when you run into problems.
    14  
    15  
    16  ## Overview
    17  
    18  To contribute test results for a platform, an organization executes the
    19  e2e test cases and uploads the test results to a Google Cloud Storage
    20  (GCS) bucket.  The test-infra repo contains scripts in `jenkins/` to run the
    21  tests and to upload the results.
    22  
    23  The various dashboards will read test results from these buckets. The remainder
    24  of this document explains how to contribute test results.
    25  
    26  
    27  ### Run the e2e tests and upload results
    28  
    29  Setup a periodic test job that will run at least once per day.  The test
    30  job runs e2e tests and uploads the results.
    31  
    32  1. Create a Google Cloud Storage bucket that will store the test
    33     results.  The bucket should have public read access.  The GCS URL
    34     must be stored in an environment variable named
    35     `JENKINS_GCS_LOGS_PATH`.  For example, Google's Jenkins server stores
    36     results to `gs://kubernetes-jenkins/logs/`.
    37  2. Run `jenkins/dockerized-e2e-runner.sh` from the test-infra repo.
    38     There are several environment variables that affect the behavior of this
    39     script.  See [e2e-runner
    40     Environment Variables](#e2e-runner-environment-variables). This will run
    41     the tests from within a docker container. If this won't work for you, you
    42     can try running `jenkins/e2e-image/e2e-runner.sh`.
    43  3. Run `JENKINS_BUILD_FINISHED={SUCCESS|UNSTABLE|FAILURE|ABORTED}
    44     hack/jenkins/upload-to-gcs.sh` from the kubernetes repo to upload the results
    45     to the GCS bucket.  If running from a Jenkins job, it is recommend to perform
    46     this step as a post-build step.  See
    47     `jenkins/job-configs/kubernetes-jenkins/global.yaml` for an example.
    48  
    49  
    50  ### Include results in test history
    51  
    52  Collect results from federated test jobs by adding the Google Cloud Storage
    53  bucket to [`buckets.yaml`](/buckets.yaml). This will also allow Gubernator to
    54  track your jobs. To add buckets to the TestGrid, open an issue against this
    55  repo.
    56  
    57  ### e2e-runner Environment Variables
    58  
    59  The following environment variables should be set appropriately before
    60  running the `e2e-runner.sh` script:
    61  
    62  - WORKSPACE *
    63  - JOB_NAME *
    64  - BUILD_NUMBER *
    65  - GINKGO_TEST_ARGS
    66  - JENKINS_GCS_LOGS_PATH
    67  - JENKINS_USE_EXISTING_BINARIES (optional. Set to "y" for locally built binaries.)
    68  - E2E_TEST="true"
    69  
    70  \* If using Jenkins to execute the e2e test job, then these environment
    71     variables may be set by Jenkins.  Otherwise, they must be set before
    72     executing `e2e-runner.sh`.  See above for how they affect how results
    73     are uploaded to a GCS bucket.
    74  
    75  Test results are stored in
    76  `gs://$JENKINS_GCS_LOGS_PATH/$JOB_NAME/$BUILD_NUMBER`. Every run should upload
    77  `started.json`, `finished.json`, and `build-log.txt`, and can optionally upload
    78  JUnit XML to the `artifacts/` directory. See the [Gubernator README](./../gubernator/README.md#job-artifact-gcs-layout)
    79  for more information.
    80  
    81  If the platform should be setup or torn down by `e2e-runner.sh`, then
    82  optionally set:
    83  
    84  - E2E_UP="true"
    85  - E2E_DOWN="true"
    86