github.com/nya3jp/tast@v0.0.0-20230601000426-85c8e4d83a9b/docs/design_principles.md (about)

     1  # Tast Design Principles (go/tast-design)
     2  
     3  This document lists principles that guided [Tast's design] and that should be
     4  kept in mind when considering future changes.
     5  
     6  [TOC]
     7  
     8  ## Running tests should be fast.
     9  
    10  Iteratively running tests (regardless of whether one is adding or modifying
    11  tests, trying to reproduce a failure, or verifying that a system change has
    12  fixed a failing test) should be fast from the perspective of a developer waiting
    13  for the command to complete.
    14  
    15  *   Operations should not take additional time to complete due to the choice of
    16      programming language.
    17  *   Building and deploying tests and associated data must be fast. `emerge` adds
    18      ten seconds or more of overhead even when building a C++ program with no
    19      dependencies and an empty `main()` function and shouldn't be part of the
    20      build/deploy/run cycle in its present form.
    21  *   The test system's overhead should be minimized. Nothing should be copied to
    22      the DUT when the test hasn't changed. All communication with the DUT should
    23      happen over a single persistent SSH connection, and round trips should be
    24      minimized on the critical path — otherwise, network latency kills
    25      performance when running tests on a DUT in a different geographical
    26      location.
    27  *   Developers shouldn't need to edit tests on-device in order to iterate
    28      quickly. ChromeOS systems do not make for pleasant development
    29      environments. (They may support pleasant development environments within
    30      VMs, but that doesn't help for testing.)
    31  *   Running a test shouldn't result in code being compiled. If a test needs
    32      additional executables to be installed on the DUT, then those executables
    33      should already be present in the system image. We already have a packaging
    34      system; use it.
    35  *   Information about a test (e.g. its inclusion in a suite) should be available
    36      without needing to evaluate hundreds of scripts. Don't emulate a declarative
    37      language using an imperative interpreted language.
    38  
    39  ## Tests should yield consistent results.
    40  
    41  *   Minimize the number of moving pieces when a test is run on a DUT. The
    42      framework, and tests themselves, should do everything in their power to
    43      avoid operations that might fail. Avoid runtime dependencies on external
    44      resources like databases, websites, and other network services.
    45  
    46  ## Adding or modifying a test should be easy.
    47  
    48  *   Minimize boilerplate. For example, test names shouldn't appear repeatedly in
    49      the source (e.g. directory names, control files, filenames, test
    50      implementations, `.ebuild` files). We'd frown if we saw the same lengthy
    51      string constant repeated five or more times in a C++ program. In cases where
    52      repetition is unavoidable, there should be automatic checks that the names
    53      are consistent in all locations.
    54  *   Developers shouldn't need to know the specifics of how the test system is
    55      integrated into ChromeOS. In the common case, they shouldn't need to edit
    56      `.ebuild` files when adding a test, run `cros_workon` when making changes,
    57      or set USE flags or build and deploy packages to run tests.
    58  
    59  ## Test results should be easy to interpret.
    60  
    61  *   A given run's output directory should be structured in a way that is easy to
    62      navigate.
    63  *   Logs must be easy to read. The default log level should include messages
    64      that describe what's happening at any given time (e.g. no radio silence
    65      while the test is running on a remote host: see [issue 715865]), but no
    66      non-fatal warnings and errors. A separate log file should be written with
    67      full verbose output, and it should be trivial for both machines and humans
    68      to find the overall pass/fail status of all tests and the verbose output
    69      from an individual test.
    70  *   Errors should be passed back to the top level of the test and logged there.
    71      When fatal errors are reported from deep in support libraries, test results
    72      are often difficult to interpret due to the lack of context present in the
    73      errors.
    74  *   Detailed timing information should be written in a format readable by both
    75      humans and machines to make it easy to see why a test run was slow and track
    76      long-term performance trends.
    77  *   System log information generated by the DUT while tests were running should
    78      be captured. It should be easy to compare timestamps in test results to
    79      timestamps from the DUT's system logs, even in the presence of clock skew.
    80  
    81  ## The test framework, and tests themselves, should be maintainable.
    82  
    83  *   The framework should focus on running tests. Tasks like allocating DUTs and
    84      scheduling tests on them, reimaging or repairing DUTs, and displaying and
    85      archiving test results belong elsewhere.
    86  *   There should be a clear separation between code that's used by tests and
    87      code that runs on developers' workstations or bots to deploy and run tests.
    88  *   Avoid magic. Code that spells out what it's doing is easier to debug than
    89      code that relies on action at a distance (e.g. overriding `__getattr__` or
    90      using `setattr` to dynamically set attributes in Python). Make code easy to
    91      trace unless there's an extremely compelling reason to do something fancy.
    92  *   Avoid making test libraries ornate. Nobody wants to puzzle their way through
    93      complicated object hierarchies while trying to debug a failing test.
    94  *   The code that supports tests must itself be thoroughly covered by unit
    95      tests.
    96  *   Make it easy to disable a broken test until it can be fixed by its owners.
    97  
    98  ## The test system should have opinions about the right way to write tests.
    99  
   100  Don't overwhelm developers with choices.
   101  
   102  *   Keep logging simple. There should be one way to report test failures and one
   103      way to log informative messages. Don't permit non-fatal "warning"-level
   104      errors, as nobody does anything about them and they end up permanently
   105      cluttering logs.
   106  *   Tests should be straightforward to read. Instead of distributing work across
   107      superclasses and overridden methods with non-obvious semantics (e.g.
   108      `initialize()`, `setup()`, `warmup()`, `run_once()`, `postprocess()`),
   109      implement each test in a single function, with initialization appearing at
   110      the beginning and teardown happening at exit (per language affordances).
   111  
   112  [Tast's design]: overview.md
   113  [issue 715865]: http://crbug.com/715865