github.com/nya3jp/tast@v0.0.0-20230601000426-85c8e4d83a9b/docs/test_validation.md (about)

     1  # Tast Test Validation
     2  
     3  Reviewing test code can be time-consuming, so it's desirable to automate the
     4  identification of common problems as much as possible. This document describes
     5  and compares different approaches that are available.
     6  
     7  [TOC]
     8  
     9  ## Considerations
    10  
    11  Validation methods have different benefits and downsides, several of which are
    12  listed here.
    13  
    14  ### Overridability
    15  
    16  Some validation processes can be manually overridden by developers. This is
    17  useful in the case of false positives, but can be problematic if the check is
    18  trying to defend against severe issues that would also affect other tests.
    19  
    20  ### Using source code vs. compiled code
    21  
    22  Validation processes can inspect either the original test source code or
    23  behavior at runtime. Both techniques are important:
    24  
    25  *   Source code validation is needed to detect coding style issues.
    26  *   Runtime checks are needed to detect issues that can't be easily detected
    27      from source code, e.g. bad values in struct fields that are initialized by
    28      expressions or function calls.
    29  
    30  ### Ease of writing
    31  
    32  Some validation code is easier to write than other code. In particular,
    33  inspecting [ASTs] produced by parsing source code is a lot of work.
    34  
    35  [ASTs]: https://en.wikipedia.org/wiki/Abstract_syntax_tree
    36  
    37  ### Early warning
    38  
    39  It's best if a validation system notifies test authors about issues before
    40  they've uploaded their code. If the checks are only performed when the test is
    41  compiled and executed in the commit queue, developers may need to wait hours to
    42  learn about issues.
    43  
    44  ## Validation methods
    45  
    46  This section describes different validation methods that are in use in light of
    47  the above considerations.
    48  
    49  | Validation method    | Overridable | Uses source | Easy to write | Early warning |
    50  |----------------------|:-----------:|:-----------:|:-------------:|:-------------:|
    51  | `gofmt` / `goimport` |             |      ✓      |      n/a      |       ✓       |
    52  | `go vet`             |             |      ✓      |      n/a      |               |
    53  | `tast-lint`          |      ✓      |      ✓      |               |       ✓       |
    54  | Test registration    |             |             |       ✓       |       ✓       |
    55  | Unit tests           |             |             |       ✓       |               |
    56  
    57  ### gofmt and goimport
    58  
    59  [gofmt] is used to format Go code and verify that it is syntactically valid.
    60  It's typically configured to run automatically within developers' text editors
    61  (sometimes via [goimport], which corrects and sorts `import` statements), so it
    62  generally provides instant feedback. `tast-lint` (described below) also runs
    63  `gofmt` when a change is uploaded for review.
    64  
    65  [gofmt]: https://golang.org/cmd/gofmt/
    66  [goimport]: https://godoc.org/golang.org/x/tools/cmd/goimports
    67  
    68  ### go vet
    69  
    70  The [go vet] command is part of the official Go distribution:
    71  
    72  > Vet examines Go source code and reports suspicious constructs, such as Printf
    73  > calls whose arguments do not align with the format string. Vet uses heuristics
    74  > that do not guarantee all reports are genuine problems, but it can find errors
    75  > not caught by the compilers.
    76  
    77  `go vet` is minimally configurable; we pass the `-printf.funcs` flag to enable
    78  checking of additional `printf`-like functions in addition to the
    79  `unusedresult.funcs` flag but otherwise use default settings.
    80  
    81  `go vet` currently runs as part of the `src_test` stage when building test
    82  bundle packages (e.g. `tast-local-tests-cros`). If developers don't manually
    83  emerge the test bundle package with`FEATURES=test` or run `fast_build.sh -C`,
    84  they may not learn about issues until their change is compiled in the Pre-Commit
    85  Queue. Running `go vet` earlier (e.g. when uploading the change) seems
    86  challenging since it expects to be able to resolve imports and therefore can't
    87  be easily run outside the chroot. See [issue 888259] for further discussion, and
    88  see also [Modifying Tast] for information about `fast_build.sh`.
    89  
    90  [go vet]: https://golang.org/cmd/vet/
    91  [issue 888259]: https://crbug.com/888259
    92  [Modifying Tast]: modifying_tast.md
    93  
    94  ### tast-lint
    95  
    96  [tast-lint] inspects modified source files. In addition to running `gofmt` and
    97  `goimports`, it performs additional Tast-specific checks to identify problems
    98  like disallowed cross-test dependencies and forbidden imports and function
    99  calls. `tast-lint` is compiled and executed by [run_lint.sh], which is executed
   100  by [PRESUBMIT.cfg] when a change is uploaded for review.
   101  
   102  [tast-lint]: https://chromium.googlesource.com/chromiumos/platform/tast/+/refs/heads/main/cmd/tast-lint
   103  [run_lint.sh]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/tools/run_lint.sh
   104  [PRESUBMIT.cfg]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/PRESUBMIT.cfg
   105  
   106  ### Test registration
   107  
   108  Some test metadata (as specified via the [testing.Test] struct) is validated at
   109  runtime when tests are registered via [testing.AddTest]. This validation is
   110  performed by the `instantiate` function in [test_instance.go]. Validated metadata
   111  includes test names, data paths, and timeouts. If a test contains bad data, an
   112  error is reported and no tests are executed. As such, test authors typically
   113  notice problems while trying to run their tests locally.
   114  
   115  Registration errors in local tests in the `cros` bundle are also caught by a
   116  unit test in [main_test.go].
   117  
   118  [testing.Test]: https://godoc.org/chromium.googlesource.com/chromiumos/platform/tast.git/src/go.chromium.org/tast/core/testing#Test
   119  [testing.AddTest]: https://godoc.org/chromium.googlesource.com/chromiumos/platform/tast.git/src/go.chromium.org/tast/core/testing#AddTest
   120  [test_instance.go]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/src/go.chromium.org/tast/core/internal/testing/test_instance.go
   121  [main_test.go]: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/HEAD/src/go.chromium.org/tast-tests/cros/local/bundles/cros/main_test.go
   122  
   123  ### Unit tests
   124  
   125  Some validation is test-specific. For example, all [ARC] tests should depend on
   126  the `android` and `chrome` [software features] and have a sufficiently-long
   127  timeout. These sorts of checks can be performed by a unit test that inspects
   128  test metadata after tests are registered. For the above ARC example, see
   129  [registration_test.go] and the [testcheck] package.
   130  
   131  You can use `fast_build.sh` to run unit tests in the `tast` and `tast-tests`
   132  repositories. The script's usage is described in the [Modifying Tast] document.
   133  
   134  [ARC]: https://developer.android.com/topic/arc/
   135  [software features]: test_dependencies.md
   136  [registration_test.go]: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/HEAD/src/go.chromium.org/tast-tests/cros/local/bundles/cros/arc/registration_test.go
   137  [testcheck]: https://godoc.org/chromium.googlesource.com/chromiumos/platform/tast.git/src/go.chromium.org/tast/core/testing/testcheck