k8s.io/test-infra@v0.0.0-20240520184403-27c6b4c223d8/testgrid/conformance/README.md (about)

     1  # Display Conformance Tests with Testgrid
     2  
     3  This directory contains tooling for uploading [Kubernetes Conformance test](https://github.com/cncf/k8s-conformance) results for display / monitoring on [TestGrid](../README.md), a tool used heavily by [the core kubernetes project](https://github.com/kubernetes/kubernetes) to monitor test results, particularly as part of the release process.
     4  
     5  Federated conformance test results are hosted on the TestGrid [conformance dashboards](https://testgrid.k8s.io/conformance-all), including the "all" dashboard, and specific sub-dashboards, see [the TestGrid README](../README.md#dashboards) for details on dashboards. Generally we are aiming to have a dashboard here for each provider E.g. "[conformance-cloud-provider-openstack](https://testgrid.k8s.io/conformance-cloud-provider-openstack)" as well as [a cross-vendor dashboard](https://testgrid.k8s.io/conformance-all) to track project wide conformance.
     6  
     7  All Kubernetes cluster providers are invited to post results from their conformance test jobs and results from reliable continuous integration against the release branches may even be used as a signal by the Kubernetes release team in the [release-blocking dashboards](https://testgrid.k8s.io/sig-release-master-blocking).
     8  
     9  The release team has caught actual conformance test regressions using these dashboards just in the first month or so of setting up GCE / OpenStack conformance on TestGrid, and had them fixed before the Kubernetes 1.11 release.
    10  
    11  For the original design doc and further details on the motivation please see [design.md](./design.md).
    12  
    13  ## Usage Guide
    14  
    15  1. First you will need to set up a publicly readable GCS bucket per [contributing test results](https://github.com/kubernetes/test-infra/blob/master/docs/contributing-test-results.md#contributing-test-results) to host your jobs' results.
    16  If you cannot or do not want to set up a GCS bucket and only wish to post conformance test results, please file an issue in https://github.com/kubernetes/k8s.io for a staging bucket, referencing https://github.com/kubernetes/k8s.io/pull/501.
    17  
    18  2. Make a PR to test-infra adding your bucket to the TestGrid config (again see [contributing test results](https://github.com/kubernetes/test-infra/blob/master/docs/contributing-test-results.md#contributing-test-results)).
    19  
    20  - See The following PR from setting up the initial OpenStack bucket: [#7670](https://github.com/kubernetes/test-infra/pull/7670)
    21  
    22  3. Setup a job in your CI system to run the conformance tests. To use [`upload_e2e.py`](./upload_e2e.py) the job environment must have `python` (v3.X) and `gcloud` / `gsutil` commands. For the gcloud CLI see [Installing the Google Cloud SDK](https://cloud.google.com/sdk/downloads).
    23  
    24  This job will need to:
    25     - `a)` setup a cluster from the kubernetes release / branch you want to test
    26     - `b)` run the conformance tests
    27     - `c)` obtain the JUnit .xml results and ginkgo (e2e test runner) log output
    28     - `d)` upload the results to the GCS bucket
    29  
    30  Setting up the test cluster in `a)` is provider specific and not currently covered here.
    31  
    32  For running the conformance tests and obtaining the result files (`b)` and `c)`) you have the following options:
    33  
    34   - follow [the official conformance testing guide's instructions](https://github.com/cncf/k8s-conformance/blob/master/instructions.md#running) to run and obtain the result files
    35  
    36   - or use [kubetest](https://github.com/kubernetes/test-infra/tree/master/kubetest):
    37     - cd to a Kubernetes source tree (git clone) for the release you wish to test, using something like:
    38     ```sh
    39     git clone https://github.com/kubernetes/kubernetes.git && cd kubernetes && git checkout release-1.11
    40     ```
    41     - run `make all WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl"` to build the test binaries for Kubernetes 1.24 and earlier.
    42     - run `make all WHAT="test/e2e/e2e.test github.com/onsi/ginkgo/v2/ginkgo cmd/kubectl"` to build the test binaries for Kubernetes 1.25 and later.
    43     - make sure `kubectl` / `$KUBECONFIG` is authed to your cluster
    44     - run [kubetest](https://github.com/kubernetes/test-infra/tree/master/kubetest) with:
    45      ```sh
    46      export KUBERNETES_CONFORMANCE_TEST=y
    47      kubetest --provider=skeleton \
    48               --test \
    49               --test_args="--ginkgo.focus=\[Conformance\]" \
    50               --dump=./_artifacts | tee ./e2e.log
    51      ```
    52     - You can then find the log file and JUnit at `./e2e.log` and `./_artifacts/junit_01.xml` respectively.
    53  
    54   - or use the [Sonobuouy CLI](https://github.com/heptio/sonobuoy#using-the-cli) to run the tests and then obtain a "snapshot" with the official instructions [when run locally](https://github.com/heptio/sonobuoy#download-and-run). You can then get the e2e log and JUnit from the snapshot (see the [plugins section](https://github.com/heptio/sonobuoy/blob/master/site/docs/master/snapshot.md#plugins) of the snapshot documentation)
    55  
    56  For uploading the results (`d)`) you can use the tooling provided here (or build your own mimicking it), to use `upload_e2e.py` provide the following required flags:
    57  - `--junit` -- The path to the JUnit result file(s): `--junit=/path/to/junit/result/file/junit_01.xml`
    58    - note that this flag accepts [glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)), E.g. `--junit=./artifacts/junit_*.xml`
    59  
    60  - `--log` -- The path to the ginkgo log file / test output:
    61   `--log=/path/to/e2e.log`
    62  
    63  - `--bucket` -- The upload prefix, which should include the GCS bucket as well as the job name, E.g. `gs://k8s-conformance-openstack/periodic-logs/ci-cloud-provider-openstack-acceptance-test-e2e-conformance-stable-branch-v1.11/` like: `--bucket=gs://your-bucket/your-job`
    64  
    65  You can optionally also provide:
    66  - `--key-file` -- A Google Cloud [service account](https://cloud.google.com/iam/docs/service-accounts) keyfile, used to automatically authenticate to GCS. Otherwise you will need to authenticate with [`gcloud auth`](https://cloud.google.com/sdk/gcloud/reference/auth/) in some other part of your CI to use this tool.
    67  Specify like:
    68  `--key-file=/path/to-key-file.json`
    69    - If you are using a GKE EngProd provided bucket, we've provided you with this file, otherwise see [Create and Manage Service Accounts](https://cloud.google.com/iam/docs/managing-service-accounts), [Create and manage Service Account Keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys), and [Cloud Storage IAM Roles](https://cloud.google.com/storage/docs/access-control/iam-roles) for docs on setting up your own service account with upload access to the bucket and creating a credentials file for it.
    70  
    71  - `--year` -- The year in which the logfile was produced, otherwise the current year on the host machine is assumed when parsing timestamps for the job's start / finish time. E.g. `--year=2018`
    72  
    73  - `--metadata` -- A JSON dict of metadata key-value pairs that can be displayed in custom TestGrid column headers. E.g. `--metadata='{"version": 52e0b2617ffec85d467f96de34d47e9bb407f880"}'`.
    74    - For more details please see [metadata for finished.json](https://github.com/kubernetes/test-infra/tree/master/gubernator#job-artifact-gcs-layout) and custom [column headers in TestGrid](https://github.com/kubernetes/test-infra/blob/master/testgrid/README.md#column-headers).
    75