github.com/nya3jp/tast@v0.0.0-20230601000426-85c8e4d83a9b/docs/test_attributes.md (about)

     1  # Tast Test Attributes (go/tast-attr)
     2  
     3  Tests may specify attributes via the `Attr` field in [testing.Test]. Attributes
     4  are free-form strings, but this document enumerates well-known attributes with
     5  established meanings.
     6  
     7  ## Manually-added attributes
     8  
     9  The following attributes may be added to control how tests are run and how their
    10  results are interpreted.
    11  
    12  Zero or more `group:*` attributes can be set to a test to assign it to
    13  *groups*. A group is a collection of tests having similar purposes or
    14  characteristics, often (but not necessarily) scheduled together.
    15  In automated testing, tests to run are usually selected by groups.
    16  If a test belongs to no group, it has no attribute `group:*` assigned, that
    17  means it will be [disabled] in automated testing.
    18  
    19  Some groups define *extra attributes* which annotate tests with extra
    20  information. They can be set if a test belongs to corresponding groups.
    21  
    22  Below is the list of most popular groups and their associated extra attributes:
    23  
    24  *   `group:mainline` - The default group for functional tests. Tests having
    25      this attribute are called *mainline* tests. A mainline test falls under
    26      exactly one of the two categories: *informational* if it has `informational`
    27      attribute; otherwise it is *critical*.
    28      Failures in critical tests justify rejecting or reverting the responsible
    29      change, while failures in informational tests are ignored.
    30      All informational mainline tests are supposed to be promoted to critical
    31      tests.
    32       * `group:criticalstaging` - This group will be used to indicate a test is intended on
    33         going into "mainline" critical testing. This group will be run on all boards/models;
    34         on ToT only.  Tests can only remain in this group long enough to gather signal (10 days),
    35         after which the owner must promote them into mainline only, or back into
    36         informational. If no owner action is taken after a 4 day grace period, they
    37         will be moved into informational.
    38       * `informational`: This group indicate a test is not critical.
    39  *   `group:appcompat` - A group of ARC app compatibility tests.
    40      Below are its sub-attribute:
    41       * `appcompat_release`: A group of ARC app compatibility tests for release testing.
    42       * `appcompat_smoke`: A group of ARC app compatibility tests for smoke testing.
    43       * `appcompat_top_apps`: A group of ARC app compatibility tests for top apps testing.
    44  *   `group:arc` - A group of ARC Functional tests.
    45      Below are its sub-attribute:
    46       * `arc_playstore`: A group of ARC tests which verify playstore
    47       functionality after updating the playstore to nightly build.
    48       * `arc_core`: A group of ARC tests which verify ARC Core functionality.
    49       * `arc_chromeos_vm`: A group of ARC tests which run ARC functionality on ChromeOS VM.
    50  *   `group:crosbolt` - Test failures are ignored and the test's performance data
    51      are uploaded to [crosbolt]. When you add this attribute, you also need to
    52      add one of `crosbolt_perbuild`, `crosbolt_nightly` or `crosbolt_weekly`.
    53  *   `group:wificell` - Tests that will run on the [wificell] fixture. Typically
    54      those tests require special hardware (APs, ...) only available in those
    55      fixtures. Some WiFi-specific functional tests that do not technically
    56      require a [wificell] are also made part of this group for consistency to
    57      simplify the validation of the whole WiFi stack.
    58      Its sub-attributes can be classified into two types:
    59      *  Role (required): specify the role of the test:
    60         *  `wificell_func`: verify basic WiFi functionalities nightly.
    61         *  `wificell_suspend`: verify basic WiFi behavior related to
    62            suspend/resume nightly.
    63         *  `wificell_cq`: Similar to wificell_func, but triggered by CLs that
    64            touch specific code paths.
    65         *  `wificell_perf`: measure WiFi performance.
    66         *  `wificell_stress`: Stress test the WiFi functionalities.
    67         *  `wificell_mtbf`: measure Mean Time Between Failures (MTBF).
    68      *  Stability (optional): if `wificell_unstable` is present, the test is yet
    69         to be verified as stable; otherwise, the test is stable.
    70  *   `group:wificell_roam` - Tests that depends on [grover] (Googlers only) fixture to run.
    71      Subattributes that specify role of the test (required):
    72      *  `wificell_roam_func`: verify basic WiFi roaming functionalities.
    73      *  `wificell_roam_perf`: measure WiFi roaming performance.
    74  *   `group:labqual` - Tests that must pass for devices to go to a low-touch lab.
    75  *   `group:mtp` - Tests that will run on the `mtpWithAndroid` fixture. Typically
    76      those tests require special hardware (Android Phone Connected to DUT) and setup
    77      available in the fixture.
    78  *   `group:cuj` - A group of CUJ tests. Tests having this attribute will have their
    79      performance data collected and sent to TPS dashboard.
    80      Below are its sub-attributes:
    81      *  `cuj_experimental`: A group of CUJ tests which are experimental and only run
    82         on a selected subset of models.
    83  See [attr.go] for the full list of valid attributes.
    84  
    85  ## Automatically-added attributes
    86  
    87  Several attributes are also added automatically:
    88  
    89  *   `bundle:<bundle>` - Test's [bundle], e.g. `cros` (automatically added).
    90  *   `dep:<dependency>` - Test [software dependency] (automatically added).
    91  *   `name:<category.Test>` - Test's full [name] (automatically added).
    92  
    93  ## Using attributes
    94  
    95  See the [Running Tests] document for information about using attributes to
    96  select which tests to run.
    97  
    98  [testing.Test]: https://godoc.org/chromium.googlesource.com/chromiumos/platform/tast.git/src/go.chromium.org/tast/core/testing#Test
    99  [crosbolt]: https://crosbolt.teams.x20web.corp.google.com/prod/crosbolt/index.html
   100  [wificell]: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/main/docs/wificell.md
   101  [grover]: https://docs.google.com/document/d/1klnkcEpbG6_0BKeLXEN9ST13-w8gcNYG1xH80dlvt7U/edit# (Googlers only)
   102  [attr.go]: https://chromium.googlesource.com/chromiumos/platform/tast/+/refs/heads/main/src/go.chromium.org/tast/core/internal/testing/attr.go
   103  [bundle]: overview.md#Test-bundles
   104  [software dependency]: test_dependencies.md
   105  [name]: writing_tests.md#Test-names
   106  [Running Tests]: running_tests.md
   107  [disabled]: writing_tests.md#Disabling-tests