github.com/secure-build/gitlab-runner@v12.5.0+incompatible/docs/development/reviewing-gitlab-runner.md (about)

     1  # Reviewing GitLab Runner
     2  
     3  This document contains rules and suggestions for GitLab Runner project reviewers.
     4  
     5  ## Reviewing tests coverage reports
     6  
     7  In GitLab Runner project, we have a lot of code. Unfortunately, the code coverage is not comprehensive.
     8  Currently (early 2019), the coverage is on the level of ~55%.
     9  
    10  While adding tests to a legacy code is a hard task, we should ensure that new code that is being
    11  added to the project has good tests coverage. Code reviewers are encouraged to look on the
    12  coverage reports and ensure new code is covered.
    13  
    14  We should aim for as much test coverage for new code as possible. Defining the level of
    15  required coverage for a specific change is left for the reviewer judgment. Sometimes 100% coverage
    16  will be something simple to achieve. Sometimes adding code with only 20% of the coverage will be
    17  realistic and will ensure that the most important things are being tested. Dear reviewer - chose wisely :)
    18  
    19  Getting back to the technical details...
    20  
    21  Runner's CI Pipeline helps us here and provides the coverage reports in HTML format, for tests
    22  executed in regular (`count`) and race (`atomic`) modes.
    23  
    24  There are two places where test coverage reports can be seen. For:
    25  
    26  - Contributions made directly to <https://gitlab.com/gitlab-org/gitlab-runner> project, changes merged to `master`
    27    branch and for all tagged releases.
    28  - Community contributions and contributions made directly to <https://gitlab.com/gitlab-org/gitlab-runner> project.
    29  
    30  ### Test coverage report from S3
    31  
    32  This report has a long-term life but, because it uses the `gitlab-runners-download` S3 bucket, it's available
    33  only for contributions made directly to <https://gitlab.com/gitlab-org/gitlab-runner>. It is also available
    34  for all jobs started from `master` branch (so mostly Merge Requests merges) and for all tagged releases.
    35  
    36  To open the report:
    37  
    38  1. Find the Pipeline related to the change that we want to review. It may be the latest Pipeline for the
    39     Merge Requests or a Pipeline for the tag. For example, we can look at this one:
    40     <https://gitlab.com/gitlab-org/gitlab-runner/pipelines/48686952>, which released the `v11.8.0` version of Runner.
    41  
    42  1. In the pipeline, find the `stable S3` (for tagged releases), `bleeding edge S3` (for `master` and RC tagged releases),
    43     or `development S3` (for regular commits) job which should be present at the `release` stage. In our example
    44     pipeline, it will be: <https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/165757556>.
    45  
    46  1. At the end of the job's log, we should see a line like:
    47  
    48     ```
    49     ==> Download index file: https://gitlab-runner-downloads.s3.amazonaws.com/latest/index.html
    50     ```
    51  
    52     Because when this job was triggered, and `v11.8.0` was also the `latest` release, we see  a link to the
    53     `latest` version bucket. The problem with `latest` is that the content there changes when
    54     new stable/patch versions are released.
    55  
    56     Each pipeline also creates a deployment for a specific reference (a branch name
    57     or a tag name). Several lines above we can see:
    58  
    59     ```
    60     ==> Download index file: https://gitlab-runner-downloads.s3.amazonaws.com/v11.8.0/index.html
    61     ```
    62  
    63     This URL points to a bucket, that should not be changed in the future. For a `bleeding edge S3` started
    64     from a `master` branch, the URL should look like <https://gitlab-runner-downloads.s3.amazonaws.com/master/index.html>
    65     (which obviously also changes over time) and for the one started from a RC tag, it should look
    66     like <https://gitlab-runner-downloads.s3.amazonaws.com/v11.8.0-rc1/index.html>. For the `development S3` job, started
    67     from a regular commit (mostly tracked within a Merge Request), the URL should look like
    68     <https://gitlab-runner-downloads.s3.amazonaws.com/mask-trace/index.html>. In this case the `mask-trace` is the
    69     name of the branch, which was used as Merge Request source.
    70  
    71  1. Open the S3 link gathered from the job's log. Following our example, let's open the
    72     <https://gitlab-runner-downloads.s3.amazonaws.com/v11.8.0/index.html> one. We can see here several files that
    73     are published as part of the release. We're interested in the content of the `coverage/` directory.
    74  
    75     In this directory, we can see three files with `.race.` as part of the filename, and three similar files
    76     but with `.regular.` as part of the filename. The files are tracking output of `go test` command executed
    77     with coverage options. The `.race.` files contain sources and reports for tests started with `-race` flag,
    78     while the `.regular.` files are sources and reports for tests started without this option.
    79  
    80     For those who are interested in details, the `-race` tests are using `atomic` coverage mode, while the standard
    81     tests are using `count` coverage mode.
    82  
    83     For our case, the `coverage/coverprofile.regular.html` file is what we should look at. `.race.` tests can fail
    84     in race condition situations (this is why we're executing them) and currently we have several of them that
    85     are constantly failing. This means that the coverage profile may not be full.
    86  
    87     The `.regular.` tests, instead, should give us the full overview of what's tested inside of our code. To inspect them:
    88  
    89  1. Open wanted report HTML page. As stated above, `coverage/coverprofile.regular.html` is what we're interested
    90     in, so using our initial example we should open the <https://gitlab-runner-downloads.s3.amazonaws.com/v11.8.0/coverage/coverprofile.regular.html#file0>
    91     file.
    92  
    93  1. At this moment, we can see a file browser showing test coverage details. In the drop-down select at the top,
    94     we can now start choosing files related to the reviewed modification and check how the coverage is changing.
    95  
    96  ### Test coverage report from job artifact
    97  
    98  As written above, reports hosted on S3 buckets are available only for pipelines started directly
    99  from <https://gitlab.com/gitlab-org/gitlab-runner> project. But many of the contributions that the reviewers
   100  are handling are contributions coming from community forks.
   101  
   102  In this case, we have the same two types of reports - `.regular.` and `.race.` - generated in exactly same
   103  way. The only difference is the place where they can be found and their lifespan. Reports are
   104  saved as job artifacts so they can be next passed to the release stage). There is a 7 day expiration
   105  time set on them. So when reviewing a change that executed its pipeline more than a week before, the report
   106  will be unavailable. But, a new pipeline execution, even without changes in the code, will resolve the problem.
   107  
   108  To open the report:
   109  
   110  1. Find the pipeline related to the
   111     change that will be reviewed. For example, we can use <https://gitlab.com/gitlab-org/gitlab-runner/pipelines/50600305>.
   112     It's a branch started from `master` at <https://gitlab.com/gitlab-org/gitlab-runner>. Normally we could just look
   113     for the S3 deployment, but all pipelines started inside of <https://gitlab.com/gitlab-org/gitlab-runner> have
   114     also stored the reports as jobs artifacts. In this case, I still have a time to click **keep**, so the future
   115     readers of this documentation will be able to see the artifacts. ;)
   116  
   117  1. In the pipeline, find the `test coverage report` job in the `coverage` stage. In our example it's available
   118     at <https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/172824578>.
   119  
   120  1. On the job's page, let's use the **Browse** button (on the right panel) to open Artifacts Browser. The browser
   121     for our example job is available at <https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/172824578/artifacts/browse>.
   122  
   123  1. In the browser, we need to navigate to the `out/coverage/` directory
   124     (<https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/172824578/artifacts/browse/out/coverage/>). This directory
   125     will contain the same six files - three with `.race.` and three similar with `.regular.` as it was described
   126     in the S3 strategy.
   127  
   128     For change reviewing, we're mostly interested in looking at the HTML report for `.regular.` type - the
   129     `coverprofile.regular.html` file. As we can see, all files are visible as external links, so for our
   130     example we'll open <https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/172824578/artifacts/file/out/coverage/coverprofile.regular.html>
   131     link, which will next redirect us to <https://gitlab-org.gitlab.io/-/gitlab-runner/-/jobs/172824578/artifacts/out/coverage/coverprofile.regular.html>
   132     where the report is being presented.
   133  
   134  1. At this moment, we can see the same file browser with coverage details as we seen with the S3 source.
   135     We can do the same. The only difference is that it will disappear in maximum of 7 days.
   136  
   137  ### Summary
   138  
   139  Dear reviewer, you've got your sword. Now go fight with the dragons!