github.com/status-im/status-go@v1.1.0/_docs/policies/tests.md (about)

     1  # `status-go` Test Policy
     2  
     3  - [Creating Tests](#creating-tests)
     4  - [Flaky Tests](#flaky-tests)
     5  
     6  ## Creating Tests
     7  
     8  - All new functionality MUST be introduced with tests that:
     9    - Prove that the functionality performs as described
    10    - Can be falsified
    11    - Are resistant to fuzzing
    12  - All new `integration tests` MUST BE validated via a minimum of 1000 tests.
    13    - This can be achieved using the `-count` or `-test.count` flag with the test command eg: `-count 1000` / `-test.count 1000`
    14    - Where the CI can not support this work flow automatically, the developer MUST perform validation tests via local testing.
    15      - `TODO` Add link to issue for CI automation of validation test runs of new `integration tests`.
    16    - Ensuring that the test passes consistently every time gives confidence that the test is not flaky.
    17  
    18  ## Flaky Tests
    19  
    20  Flaky tests are defined as tests that fail intermittently.
    21  
    22  - All flaky tests / failing tests MUST be resolved.
    23  - No flaky tests may be introduced into the codebase.
    24  
    25  ### Steps to resolving or reporting flaky tests
    26  
    27  #### Is it me?
    28  Determine who caused the flaky test.
    29  
    30  - Is a new test you’ve written flaky or failing?
    31    - It was you.
    32    - You MUST fix the test before merge is acceptable.
    33  - Has an existing test become flaky?
    34    - Check rerun reports. `TODO` add link to rerun reports
    35      - If the test does not appear in https://github.com/status-im/status-go/labels/E%3AFlaky%20Test or in the last three nightly test runs, it is most likely that the flakiness was introduced by your changes and needs to be addressed before proceeding with the merge.
    36      - Else the test is already documented as a flaky test (appears in the GitHub issues or in the nightly test runs), proceed to below.
    37  
    38  ```mermaid
    39  flowchart TB
    40      A([PR ready for merge]) --> B{Have any test failed?}
    41      B -->|No| C[🎉 Proceed with merge 🪄]
    42      B -->|Yes| D{
    43          Is the failing test introduced
    44          or altered by this PR?
    45      }
    46      D -->|No| E[Check rerun reports.]
    47      D -->|Yes| F[
    48          It is likely your changes introduced the flakiness.
    49          You MUST fix the test before merge is acceptable.
    50      ]
    51      F --> A
    52      E --> G{Does the test appear in `E:Flaky Test` issues<br/> or in the last three nightly test runs?<br/>}
    53      G -->|Yes| I[The flakiness needs reporting]
    54      G -->|No| F
    55      I --> J([Proceed to Reporting flow])
    56  ```
    57  
    58  #### Reporting Flaky Tests
    59  If an old test fails and/or seems flaky either locally or in CI, you MUST report the event.
    60  - Check the `status-go` GitHub repo issues for the test name(s) failing.
    61  - If the test appears in the list of flaky test issues
    62    - If the issue is open
    63      - Add a comment to the issue
    64      - Detail that you have experienced the test being flaky and in what context (local vs CI, link to the PR or branch).
    65    - If the issue is closed
    66      - Reopen the issue OR create a new issue referencing the previous issue
    67        - Either is fine, use your best judgement in this case.
    68      - Detail that you have experienced the test being flaky and in what context (local vs CI, link to the PR or branch).
    69  - If the test does not appear in the list of flaky test issues
    70    - create a new issue
    71      - The issue title should include the flaky test name
    72      - The issue should use the https://github.com/status-im/status-go/labels/E%3AFlaky%20Test label
    73    - Detail that you have experienced the test being flaky and in what context (local vs CI, link to the PR or branch).
    74  
    75  ```mermaid
    76  flowchart TB
    77      A([Ready to report a flaky test]) --> B[Check the `status-go` GitHub repo<br/>issues for the test name failing.]
    78      B --> C{Does the test appear in<br/>the list of `E: Flaky Test` issues?}
    79      C -->|No| D[
    80  	    Create a new issue
    81        - The issue title should include the flaky test name
    82        - The issue should use the `E:Flaky Test` label
    83      ]
    84      D --> E[
    85  	    Detail which test is flaky and in what context:
    86  	    local vs CI, link to the PR or branch.
    87      ]
    88      E --> J
    89      C -->|Yes| F{Is the issue open?}
    90      F -->|No| G((Either))
    91      H --> E
    92      G --> I[Reopen the issue]
    93      G --> D
    94      I --> H
    95      F -->|Yes| H[Add a comment to the issue]
    96      J([End])
    97  ```