github.com/munnerz/test-infra@v0.0.0-20190108210205-ce3d181dc989/robots/coverage/docs/design.md (about)

     1  # Overview
     2  This code coverage tool calculates per file, per package and overall coverage on target directories. It generates the following artifacts
     3    - code coverage profile, which is produced by "[go test -coverprofile](https://blog.golang.org/cover)" and contains all block level code coverage data
     4    - XML file that stores file-level and package-level code coverage, formatted to be readable by TestGrid
     5  
     6  The tool has two major modes of operation, based on whether it is running in post-submit or pre-submit workflow. 
     7  Post-submit workflow runs on a specific commit on a branch and is typically triggered when commits are pushed to a branch (e.g. when a PR merges). 
     8  Pre-submit workflow runs on the merge commit for a pull request and is typically triggered by PR creation or update.
     9  
    10  The tool performs the following additional operations when running in pre-submit mode 
    11    - after running code coverage on target directories, it compares the new result with the one stored by 
    12    the post-submit workflow and generate coverage difference. 
    13    - it reports coverage changes (calculated above) to the pull request as a comment by a robot github account. 
    14    - it uses go tools to generate line by line coverage and stores the result in html, 
    15    with a link as part of the robot comment mentioned above.
    16    - it can be configured to return with a non-zero status if coverage falls below threshold.
    17  
    18  Note that pre-submit mode can not be used without the post-submit mode, because post-submit results are consumed as a baseline by the pre-submit mode.
    19  
    20  ## Users
    21  The pre-submit mode is intended for a developer to see the impact on code coverage of his/her commit. 
    22  
    23  The post-submit mode, provides input for TestGrid - TestGrid is for repo managers and/or test infra team to monitor code coverage stats over time.
    24  
    25  ## Programming Language Supported
    26  The code coverage tool only collect code coverage for Go files
    27  
    28  
    29  # Design of Test Coverage Tool
    30  The tool takes input from three sources
    31  1. Target directory
    32    - It runs [test coverage profiling](https://blog.golang.org/cover) on target repository. 
    33      - target directory can be passed as flags when running the binary. E.g "--cov-target=./pkg/"
    34    - .gitattribute file
    35      - it uses git attribute to filter files (see latter section on workflows for details)
    36  2. (In pre-submit workflow only) It reads previously stored post-submit code coverage profile from gcs bucket. The profile
    37  serves as a base of comparison for determining the pull request's coverage delta.
    38  3. Variables passed through flags. Here is a list of these variables.
    39  
    40      |flag       |meaning                            |sample value |
    41      | --------- | --------------------------------- | ----------- |
    42      |artifacts  |local directory to dump artifacts  |./artifacts  |
    43      |cov-target |target directories to run coverage |./pkg1 ./pkg2|
    44      |cov-threshold-percentage|coverage threshold in percentage|85 |
    45      |profile-name|file name for code coverage profile|coverage_profile.txt|
    46      |postsubmit-gcs-bucket|gcs bucket that stores code coverage profile in post-submit run|knative-prow|
    47      |postsubmit-job-name|job name in gcs bucket that stores code coverage profile in post-submit run|post-knative-serving-go-coverage|
    48      
    49  Note that the last two flags are for pre-submit mode only. The last three flags are used to locate the code coverage profile in GCS bucket.
    50  
    51  Here is the step-by-step description of the pre-submit and post-submit workflows
    52  
    53  ## Post-submit workflow
    54  The tool produces & stores coverage profile for later presubmit jobs to compare against; in addition, it produces per-file and per-package coverage result as input for [TestGrid](https://github.com/kubernetes/test-infra/tree/master/testgrid). Testgrid can use the data produced here to display coverage trend in a tabular or graphical way. 
    55  
    56  1. Generate coverage profile. Completion marker generated upon successful run. Both stored
    57   in artifacts directory.
    58      - Completion marker is used by later pre-submit job when searching for a healthy and complete 
    59      code coverage profile in the post-submit jobs
    60      - Successfully generated coverage profile may be used as the basis of comparison for coverage change, in pre-submit workflow
    61  2. Read, filter, and summarizes data from coverage profile and store per-file coverage data
    62      - filter based on git attribute to ignore files with the following git attributes
    63        - linguist-generated
    64        - coverage-excluded
    65        
    66        An example of how these git attribute is used can be found [here](https://github.com/knative/serving/blob/master/.gitattributes)
    67      - Stores in the XML format, that is used by TestGrid, and dump it in artifacts directory
    68        - The XML should be a valid JUnit XML file. See 
    69    [JUnit XML format](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html)
    70        - In addition to being a valid jUnit XML file, the file needs to have the properties specified [here](https://github.com/kubernetes/test-infra/tree/master/gubernator#gcs-layout) to be readable by TestGrid 
    71        - For each file that has a coverage level lower than the threshold, the corresponding entry in the XML should have a \<failure\> tag
    72  
    73  ## Pre-submit workflow
    74  Runs code coverage tool to report coverage change from a PR
    75  
    76  1. Generate coverage profile in artifacts directory
    77  2. Read, filter, and summarizes data from coverage profile and store per-file coverage data (Same as the corresponding step in post-submit)
    78  3. Calculate coverage changes. Compare the coverage file generated in this cycle against the most
    79   recent successful post-submit build. Coverage file for post-submit commits were generated in 
    80   post-submit workflow and stored in gcs bucket
    81  4. Use PR data from github, git-attributes, as well as coverage change data calculated above, to 
    82  produce a list of files that we care about in the line-by-line coverage report. produce line by 
    83  line coverage html and add link to covbot report. Note that covbot is the robot github account 
    84  used to report code coverage change results. See Covbot section for more details.
    85  5. If any file in this commit has a coverage change, let covbot post presubmit coverage on github, under that conversation of the PR. 
    86    - The covbot comment should have the following information on each file with a coverage change
    87      - file name
    88      - old coverage (coverage before any change in the PR)
    89      - new coverage (coverage after applied all changes in the PR)
    90      - change the coverage
    91    - After each new posting, any previous posting by covbot should be removed
    92  
    93  ## Locally running presubmit and post-submit workflows
    94  Both workflows may be triggered locally in command line, as long as all the required flags are 
    95  supplied correctly. In addition, the following env var needs to be set:
    96  - JOB_TYPE (one of 'presubmit', 'postsubmit', 'local-presubmit')
    97  
    98  use 'local-presubmit' will run the presubmit workflow without posting result on github PR
    99    
   100  ## Covbot
   101  As mentioned in the presubmit workflow section, covbot is the short name for the robot github 
   102  account used to report code coverage change results. It can be created as a regular github 
   103  account - it does not need to be named covbot as that name is already taken on Github. It only need a 
   104  comment access to the repo it need to be run on. If the repo is private, it also need read access. 
   105    
   106  After the robot account is created, download the github token and supply the path to the token 
   107  file to code coverage binary, as the value for parameter "github-token"
   108  
   109  # Usage with container based CI/CD system
   110  We pack the test coverage feature in a container, that is triggered to run by a CI/CD system such as [prow](https://github.com/kubernetes/test-infra/tree/master/prow), in response to Github events such as pulls and merges.
   111  
   112  Here is [an example of a dockerfile](https://github.com/kubernetes/test-infra/blob/a1e910ae6811a1821ad98fa28e6fad03972a8c20/coverage/Dockerfile) using [Docker](https://www.docker.com/). 
   113  Here is [an example of a Makefile](https://github.com/kubernetes/test-infra/blob/a1e910ae6811a1821ad98fa28e6fad03972a8c20/coverage/Makefile) that builds and pushes the docker image on [Google Container Registry](https://cloud.google.com/container-registry/).
   114  
   115  ## Usage with prow
   116  Prow is tested working well with this Code Coverage tool. It's usage is described below
   117  
   118  - Prow can be used as the system to handle Github events mentioned in the two workflows. 
   119  - Prow, in both workflows, supplies the flags and secrets for the binary, clones the repository, and uploads logs & artifacts to GCS bucket.
   120  
   121    - The pre-submit prow job is triggered by any new commit to a PR. At the end of the binary run, it can return a pass or fail status context to Github. [Tide](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide) can use that status to block PR with low coverage.
   122  
   123    - The post-submit prow job is triggered by merge events to the base branch.
   124  
   125  ### Prow Job Specification
   126  Here is an example of a pre-submit prow job spec that runs the coverage tool in a container (the container build file can be found [here](https://github.com/kubernetes/test-infra/blob/a1e910ae6811a1821ad98fa28e6fad03972a8c20/coverage/Makefile)). The args section includes all the arguments for the binary of the tool. 
   127  ```
   128    - name: pull-knative-serving-go-coverage
   129      labels:
   130        preset-service-account: "true"
   131      always_run: true
   132      optional: true
   133      decorate: true
   134      clone_uri: "git@github.com:knative/serving.git"
   135      spec:
   136        containers:
   137        - image: gcr.io/knative-tests/test-infra/coverage:latest
   138          imagePullPolicy: Always
   139          command:
   140          - "/coverage"
   141          args:
   142          - "--postsubmit-gcs-bucket=knative-prow"
   143          - "--postsubmit-job-name=post-knative-serving-go-coverage"
   144          - "--profile-name=coverage_profile.txt"
   145          - "--artifacts=$(ARTIFACTS)" # folder to store artifacts, such as the coverage profile
   146          - "--cov-target=./pkg/" # target directory of test coverage
   147          - "--cov-threshold-percentage=50" # minimum level of acceptable presubmit coverage on a per-file level
   148          - "--github-token=/etc/github-token/token"
   149        env:
   150        - name: GOOGLE_APPLICATION_CREDENTIALS
   151          value: /etc/service-account/service-account.json
   152        volumes:
   153        - name: github-token
   154          secret:
   155            secretName: covbot-token
   156        - name: service
   157          secret:
   158            secretName: service-account
   159        volumeMounts:
   160        - name: github-token
   161          mountPath: /etc/github-token
   162          readOnly: true
   163        - name: service
   164          mountPath: /etc/service-account
   165          readOnly: true
   166  ```