github.com/yrj2011/jx-test-infra@v0.0.0-20190529031832-7a2065ee98eb/docs/tests/node-e2e-ci.md (about)

     1  # Running Kubernetes node e2e Conformance tests on RHEL
     2  
     3  Currently in Red Hat, we have two periodic Jenkins jobs running Kubernetes node e2e Conformance tests on RHEL.
     4  The first job runs the tests over standard `Kubelet` binary.
     5  The second one runs the same set of tests over containerized `Kubelet`.
     6  Both jobs are run over VM instances provisioned in AWS.
     7  
     8  The document describes the actions that were needed to implement the jobs.
     9  There are three steps to consider:
    10  
    11  1. [Running node e2e tests locally](#running-node-e2e-tests-locally)
    12  2. [Uploading test results to GCS bucket](#uploading-test-results-to-gcs-bucket)
    13  3. [Publishing test results in the TestGrid](#publishing-test-results-in-the-testgrid)
    14  
    15  The `Kubernetes` git repository already provides most of the code needed to run the node e2e tests.
    16  Thus, all the effort reduces to running `Makefile` with a set of relevant parameters.
    17  With [#56250](https://github.com/kubernetes/kubernetes/pull/56250)
    18  merged we are able to run the tests over containerized `Kubelet` as well.
    19  
    20  Once all the tests are finished, the test results are expected to be published
    21  into a GCS bucket. At the same time, the GCS bucket needs to be registered
    22  in the `TestGrid` so the results can be shared with the upstream community
    23  (and block a new release of `Kubernetes` in case the tests fail and are required not to fail).
    24  
    25  General upstream documentation on adding a new e2e tests is available at
    26  [contributing-test-results.md](../contributing-test-results.md).
    27  
    28  ## Running node e2e tests locally
    29  
    30  ### Standard Kubelet
    31  
    32  It's enough to run the following command from the root `Kubernetes` repository
    33  directory:
    34  
    35  ```sh
    36  KUBELET_FLAGS="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/"
    37  make test-e2e-node TEST_ARGS="--kubelet-flags=\"${KUBELET_FLAGS}\"" \
    38      FOCUS="Conformance"
    39  ```
    40  
    41  The command builds all the necessary binaries and runs the node e2e test suite.
    42  The RHEL requires the ``--cgroup-driver=systemd`` flag to be set.
    43  
    44  ### Containerized Kubelet
    45  
    46  In this variant there are two steps required to do:
    47  
    48  1. build `hyperkube` docker image
    49  2. tell the node e2e tests to run the containerized variant with the `hyperkube` image
    50  
    51  **Build hyperkube docker image**
    52  
    53  Before the docker image can be created, the `hyperkube` binary needs to be built.
    54  Assuming the `GOPATH` is properly set and the required `golang` version installed,
    55  it's just enough to run:
    56  
    57  ```sh
    58  cd $GOPATH/src/k8s.io/kubernetes
    59  make WHAT="cmd/hyperkube"
    60  ```
    61  
    62  Based on your architecture and OS, the `hyperkube` binary can be written
    63  under `_output/local/bin/linux/amd64/` directory.
    64  Once written, you can build the docker image by running:
    65  
    66  ```sh
    67  cd $GOPATH/src/k8s.io/kubernetes/cluster/images/hyperkube
    68  export REGISTRY=registry.access.redhat.com
    69  # Expected location of the hyperkube binary
    70  export HYPERKUBE_BIN="_output/local/bin/linux/amd64/hyperkube"
    71  # Either latest or the current git commit
    72  IMAGE_TAG=$(git describe --abbrev=0)
    73  make build VERSION=${IMAGE_TAG} ARCH=amd64 BASEIMAGE=rhel7
    74  ```
    75  
    76  The docker image tag is set to reflect the current commit in the `Kubernetes`
    77  repository. The `rhel7` image is used as the base docker image.
    78  Once run, image with the `hyperkube-amd64` name is built.
    79  
    80  **Running `Conformance` tests**
    81  
    82  Once the `hyperkube` docker image is built, the node e2e tests over containerized
    83  `Kubelet` can be run via:
    84  
    85  ```sh
    86  IMAGE_TAG=$(git describe --abbrev=0)
    87  HYPERKUBE_IMAGE="registry.access.redhat.com/hyperkube-amd64:${IMAGE_TAG}"
    88  # --cgroups-per-qos=true no longer available
    89  KUBELET_FLAGS="--cgroup-driver=systemd --cgroup-root=/"
    90  
    91  # --kubelet-containerized and --hyperkube-image introduced
    92  # by https://github.com/kubernetes/kubernetes/pull/56250
    93  make test-e2e-node TEST_ARGS="--kubelet-containerized=true \
    94      --hyperkube-image=\"${HYPERKUBE_IMAGE}\" \
    95      --kubelet-flags=\"${KUBELET_FLAGS}\"" \
    96      FOCUS="Conformance"
    97  ```
    98  
    99  ## Uploading test results to GCS bucket
   100  
   101  First step is to get a GCS bucket, either to create new or use existing one.
   102  Content of the bucket must be made publicly available (see https://cloud.google.com/storage/docs/access-control/making-data-public).
   103  For periodic jobs the expected GCS path is in the following form (see [gcs bucket layout](https://github.com/kubernetes/test-infra/blob/master/gubernator/README.md#gcs-bucket-layout) description):
   104  
   105  ```sh
   106  gs://kubernetes-github-redhat/logs/${JOB_NAME}/${BUILD_NUMBER}/
   107  ```
   108  
   109  The `TestGrid` then expects the following content of each build:
   110  
   111  * started.json
   112  
   113    **Example**:
   114    ```json
   115    {
   116      "node": "ip-172-18-0-237.ec2.internal",
   117      "timestamp": 1511906201,
   118      "repos": {
   119        "k8s.io/kubernetes": "master"
   120      },
   121      "version": "v1.10.0-alpha.0.684+51033c4dec6e00",
   122      "repo-version": "v1.10.0-alpha.0.684+51033c4dec6e00"
   123    }
   124    ```
   125  
   126  * finished.json
   127  
   128    **Example**:
   129    ```json
   130    {
   131      "timestamp": 1511907565,
   132      "version": "v1.10.0-alpha.0.684+51033c4dec6e00",
   133      "result": "SUCCESS",
   134      "passed": true,
   135      "job-version": "v1.10.0-alpha.0.684+51033c4dec6e00",
   136      "metadata": {
   137        "repo": "k8s.io/kubernetes",
   138        "repos": {
   139          "k8s.io/kubernetes": "master"
   140        },
   141        "repo-commit": "51033c4dec6e00cbbb550fcc09940efc54e54f79",
   142        "version": "v1.10.0-alpha.0.684+51033c4dec6e00",
   143        "job-version": "v1.10.0-alpha.0.684+51033c4dec6e00"
   144      }
   145    }
   146    ```
   147  
   148  * build-log.txt
   149  * artifacts
   150  
   151    Directory that provides additional information about a build. E.g.
   152    * junit files
   153    * logs of individual nodes
   154    * metadata
   155  
   156  Official description of the individual files and their content is described by [job artifacts gcs layout](https://github.com/kubernetes/test-infra/blob/master/gubernator/README.md#job-artifact-gcs-layout). You can check a real example with more data at https://console.cloud.google.com/storage/browser/kubernetes-jenkins/logs/ci-cri-containerd-node-e2e/2600.
   157  
   158  ## Publishing test results in TestGrid
   159  
   160  To have the [TestGrid](https://k8s-testgrid.appspot.com/) consume the new build results, one needs to extend the TestGrid
   161  configuration file at https://github.com/kubernetes/test-infra/blob/master/testgrid/config/config.yaml.
   162  
   163  The header of the file describes what needs to be done to add new build.
   164  The current jobs have been added through https://github.com/kubernetes/test-infra/pull/5693 PR.
   165  
   166  Once the PR is merged, one has to wait up to 30 minutes until the GCS bucket processing is run, the job results are processed and available in the TestGrid.
   167  
   168  ## Publishing test results in BigQuery
   169  
   170  Add the bucket to the list of GCS buckets at [/kettle/buckets.yaml]. Results will be updated daily, and appear in the [/kettle/README.md] BigQuery tables.