k8s.io/test-infra@v0.0.0-20240520184403-27c6b4c223d8/robots/coverage/docs/design.md (about)

     1  # Overview
     2  
     3  This code coverage tool calculates per file, per package and overall coverage on target directories. It generates the following artifacts:
     4  
     5    - code coverage profile, which is produced by "[go test -coverprofile](https://blog.golang.org/cover)" and contains all block level code coverage data
     6    - XML file that stores file-level and package-level code coverage, formatted to be readable by TestGrid
     7  
     8  The tool has two major modes of operation, based on whether it is running in post-submit or pre-submit workflow.
     9  Post-submit workflow runs on a specific commit on a branch and is typically triggered when commits are pushed to a branch (e.g. when a PR merges).
    10  Pre-submit workflow runs on the merge commit for a pull request and is typically triggered by PR creation or update.
    11  
    12  The tool performs the following additional operations when running in pre-submit mode:
    13  
    14    - after running code coverage on target directories, it compares the new result with the one stored by the post-submit workflow and generate coverage difference.
    15    - it reports coverage changes (calculated above) to the pull request as a comment by a robot github account.
    16    - it uses go tools to generate line by line coverage and stores the result in html, with a link as part of the robot comment mentioned above.
    17    - it can be configured to return with a non-zero status if coverage falls below threshold.
    18  
    19  Note that pre-submit mode can not be used without the post-submit mode, because post-submit results are consumed as a baseline by the pre-submit mode.
    20  
    21  ## Users
    22  
    23  The pre-submit mode is intended for a developer to see the impact on code coverage of his/her commit.
    24  
    25  The post-submit mode, provides input for TestGrid - TestGrid is for repo managers and/or test infra team to monitor code coverage stats over time.
    26  
    27  ## Programming Language Supported
    28  
    29  The code coverage tool only collect code coverage for Go files
    30  
    31  # Design of Test Coverage Tool
    32  
    33  The tool takes input from three sources:
    34  
    35  1. Target directory
    36  
    37      - It runs [test coverage profiling](https://blog.golang.org/cover) on target repository.
    38        - target directory can be passed as flags when running the binary. E.g "--cov-target=./pkg/"
    39      - .gitattribute file
    40        - it uses git attribute to filter files (see latter section on workflows for details)
    41  
    42  2. (In pre-submit workflow only) It reads previously stored post-submit code coverage profile from gcs bucket. The profile serves as a base of comparison for determining the pull request's coverage delta.
    43  
    44  3. Variables passed through flags. Here is a list of these variables.
    45  
    46      |flag       |meaning                            |sample value |
    47      | --------- | --------------------------------- | ----------- |
    48      |artifacts  |local directory to dump artifacts  |./artifacts  |
    49      |cov-target |target directories to run coverage |./pkg1 ./pkg2|
    50      |cov-threshold-percentage|coverage threshold in percentage|85 |
    51      |profile-name|file name for code coverage profile|coverage_profile.txt|
    52      |postsubmit-gcs-bucket|gcs bucket that stores code coverage profile in post-submit run|knative-prow|
    53      |postsubmit-job-name|job name in gcs bucket that stores code coverage profile in post-submit run|post-knative-serving-go-coverage|
    54  
    55  Note that the last two flags are for pre-submit mode only. The last three flags are used to locate the code coverage profile in GCS bucket.
    56  
    57  Here is the step-by-step description of the pre-submit and post-submit workflows
    58  
    59  ## Post-submit workflow
    60  
    61  The tool produces & stores coverage profile for later presubmit jobs to compare against; in addition, it produces per-file and per-package coverage result as input for [TestGrid](https://github.com/kubernetes/test-infra/tree/master/testgrid). Testgrid can use the data produced here to display coverage trend in a tabular or graphical way.
    62  
    63  1. Generate coverage profile. Completion marker generated upon successful run. Both stored in artifacts directory.
    64      - Completion marker is used by later pre-submit job when searching for a healthy and complete
    65      code coverage profile in the post-submit jobs
    66      - Successfully generated coverage profile may be used as the basis of comparison for coverage change, in pre-submit workflow
    67  2. Read, filter, and summarizes data from coverage profile and store per-file coverage data
    68      - filter based on git attribute to ignore files with the following git attributes
    69        - linguist-generated
    70        - coverage-excluded
    71  
    72        An example of how these git attribute is used can be found [here](https://github.com/knative/serving/blob/master/.gitattributes)
    73      - Stores in the XML format, that is used by TestGrid, and dump it in artifacts directory
    74        - The XML should be a valid JUnit XML file. See [JUnit XML format](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html)
    75        - In addition to being a valid JUnit XML file, the file needs to have the properties specified [here](https://github.com/kubernetes/test-infra/tree/master/gubernator#gcs-layout) to be readable by TestGrid
    76        - For each file that has a coverage level lower than the threshold, the corresponding entry in the XML should have a \<failure\> tag
    77  
    78  ## Pre-submit workflow
    79  
    80  Runs code coverage tool to report coverage change from a PR
    81  
    82  1. Generate coverage profile in artifacts directory;
    83  2. Read, filter, and summarizes data from coverage profile and store per-file coverage data (Same as the corresponding step in post-submit);
    84  3. Calculate coverage changes. Compare the coverage file generated in this cycle against the most recent successful post-submit build. Coverage file for post-submit commits were generated in post-submit workflow and stored in gcs bucket;
    85  4. Use PR data from github, git-attributes, as well as coverage change data calculated above, to produce a list of files that we care about in the line-by-line coverage report. produce line by line coverage html and add link to covbot report. Note that covbot is the robot github account used to report code coverage change results. See Covbot section for more details;
    86  5. If any file in this commit has a coverage change, let covbot post presubmit coverage on github, under that conversation of the PR.
    87    - The covbot comment should have the following information on each file with a coverage change
    88      - file name
    89      - old coverage (coverage before any change in the PR)
    90      - new coverage (coverage after applied all changes in the PR)
    91      - change the coverage
    92    - After each new posting, any previous posting by covbot should be removed
    93  
    94  ## Locally running presubmit and post-submit workflows
    95  
    96  Both workflows may be triggered locally in command line, as long as all the required flags are
    97  supplied correctly. In addition, the following env var needs to be set:
    98  
    99  - JOB_TYPE (one of 'presubmit', 'postsubmit', 'local-presubmit')
   100  
   101  use 'local-presubmit' will run the presubmit workflow without posting result on github PR
   102  
   103  ## Covbot
   104  
   105  As mentioned in the presubmit workflow section, covbot is the short name for the robot github
   106  account used to report code coverage change results. It can be created as a regular github
   107  account - it does not need to be named covbot as that name is already taken on GitHub. It only need a comment access to the repo it need to be run on. If the repo is private, it also need read access.
   108  
   109  After the robot account is created, download the github token and supply the path to the token
   110  file to code coverage binary, as the value for parameter "github-token"
   111  
   112  # Usage with container based CI/CD system
   113  
   114  We pack the test coverage feature in a container, that is triggered to run by a CI/CD system such as [prow](https://github.com/kubernetes/test-infra/tree/master/prow), in response to GitHub events such as pulls and merges.
   115  
   116  Here is [an example of a dockerfile](https://github.com/kubernetes/test-infra/blob/a1e910ae6811a1821ad98fa28e6fad03972a8c20/coverage/Dockerfile) using [Docker](https://www.docker.com/).
   117  Here is [an example of a Makefile](https://github.com/kubernetes/test-infra/blob/a1e910ae6811a1821ad98fa28e6fad03972a8c20/coverage/Makefile) that builds and pushes the docker image on [Google Container Registry](https://cloud.google.com/container-registry/).
   118  
   119  ## Usage with prow
   120  
   121  Prow is tested working well with this Code Coverage tool. It's usage is described below
   122  
   123  - Prow can be used as the system to handle GitHub events mentioned in the two workflows.
   124  - Prow, in both workflows, supplies the flags and secrets for the binary, clones the repository, and uploads logs & artifacts to GCS bucket.
   125  
   126    - The pre-submit prow job is triggered by any new commit to a PR. At the end of the binary run, it can return a pass or fail status context to GitHub. [Tide](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide) can use that status to block PR with low coverage.
   127  
   128    - The post-submit prow job is triggered by merge events to the base branch.
   129  
   130  ### Prow Job Specification
   131  
   132  Here is an example of a pre-submit prow job spec that runs the coverage tool in a container (the container build file can be found [here](https://github.com/kubernetes/test-infra/blob/a1e910ae6811a1821ad98fa28e6fad03972a8c20/coverage/Makefile)). The args section includes all the arguments for the binary of the tool.
   133  
   134  ```yaml
   135    - name: pull-knative-serving-go-coverage
   136      labels:
   137        preset-service-account: "true"
   138      always_run: true
   139      optional: true
   140      decorate: true
   141      clone_uri: "git@github.com:knative/serving.git"
   142      spec:
   143        containers:
   144        - image: gcr.io/knative-tests/test-infra/coverage:latest
   145          imagePullPolicy: Always
   146          command:
   147          - "/coverage"
   148          args:
   149          - "--postsubmit-gcs-bucket=knative-prow"
   150          - "--postsubmit-job-name=post-knative-serving-go-coverage"
   151          - "--profile-name=coverage_profile.txt"
   152          - "--artifacts=$(ARTIFACTS)" # folder to store artifacts, such as the coverage profile
   153          - "--cov-target=./pkg/" # target directory of test coverage
   154          - "--cov-threshold-percentage=50" # minimum level of acceptable presubmit coverage on a per-file level
   155          - "--github-token=/etc/github-token/token"
   156        env:
   157        - name: GOOGLE_APPLICATION_CREDENTIALS
   158          value: /etc/service-account/service-account.json
   159        volumes:
   160        - name: github-token
   161          secret:
   162            secretName: covbot-token
   163        - name: service
   164          secret:
   165            secretName: service-account
   166        volumeMounts:
   167        - name: github-token
   168          mountPath: /etc/github-token
   169          readOnly: true
   170        - name: service
   171          mountPath: /etc/service-account
   172          readOnly: true
   173  ```