github.com/containers/podman/v5@v5.1.0-rc1/test/system/README.md (about)

     1  Quick overview of podman system tests. The idea is to use BATS,
     2  but with a framework for making it easy to add new tests and to
     3  debug failures.
     4  
     5  Quick Start
     6  ===========
     7  
     8  Look at [030-run.bats](030-run.bats) for a simple but packed example.
     9  This introduces the basic set of helper functions:
    10  
    11  * `setup` (implicit) - resets container storage so there's
    12  one and only one (standard) image, and no running containers.
    13  
    14  * `parse_table` - you can define tables of inputs and expected results,
    15  then read those in a `while` loop. This makes it easy to add new tests.
    16  Because bash is not a programming language, the caller of `parse_table`
    17  sometimes needs to massage the returned values; `015-run.bats` offers
    18  examples of how to deal with the more typical such issues.
    19  
    20  * `run_podman` - runs command defined in `$PODMAN` (default: 'podman'
    21  but could also be './bin/podman' or 'podman-remote'), with a timeout.
    22  Checks its exit status.
    23  
    24  * `is` - compare actual vs expected output. Emits a useful diagnostic
    25  on failure.
    26  
    27  * `die` - output a properly-formatted message to stderr, and fail test
    28  
    29  * `skip_if_rootless` - if rootless, skip this test with a helpful message.
    30  
    31  * `skip_if_remote` - like the above, but skip if testing `podman-remote`
    32  
    33  * `random_string` - returns a pseudorandom alphanumeric string
    34  
    35  Test files are of the form `NNN-name.bats` where NNN is a three-digit
    36  number. Please preserve this convention, it simplifies viewing the
    37  directory and understanding test order. In particular, `00x` tests
    38  should be reserved for a first-pass fail-fast subset of tests:
    39  
    40      bats test/system/00*.bats || exit 1
    41      bats test/system
    42  
    43  ...the goal being to provide quick feedback on catastrophic failures
    44  without having to wait for the entire test suite.
    45  
    46  
    47  Running tests
    48  =============
    49  To run the tests locally in your sandbox, you can use one of these methods:
    50  * make;PODMAN=./bin/podman bats ./test/system/070-build.bats # runs just the specified test
    51  * make;PODMAN=./bin/podman bats ./test/system                # runs all
    52  * make;PODMAN=./bin/podman NETWORK_BACKEND=netavark bats ./test/system  # Assert & enable netavark testing
    53  
    54  To test as root:
    55  *  $ PODMAN=./bin/podman sudo --preserve-env=PODMAN bats test/system
    56  
    57  Analyzing test failures
    58  =======================
    59  
    60  The top priority for this scheme is to make it easy to diagnose
    61  what went wrong. To that end, `podman_run` always logs all invoked
    62  commands, their output and exit codes. In a normal run you will never
    63  see this, but BATS will display it on failure. The goal here is to
    64  give you everything you need to diagnose without having to rerun tests.
    65  
    66  The `is` comparison function is designed to emit useful diagnostics,
    67  in particular, the actual and expected strings. Please do not use
    68  the horrible BATS standard of `[ x = y ]`; that's nearly useless
    69  for tracking down failures.
    70  
    71  If the above are not enough to help you track down a failure:
    72  
    73  
    74  Debugging tests
    75  ---------------
    76  
    77  Some functions have `dprint` statements. To see the output of these,
    78  set `PODMAN_TEST_DEBUG="funcname"` where `funcname` is the name of
    79  the function or perhaps just a substring.
    80  
    81  
    82  Requirements
    83  ============
    84  
    85  - bats
    86  - jq
    87  - skopeo
    88  - nmap-ncat
    89  - httpd-tools
    90  - openssl
    91  - socat
    92  - buildah
    93  - gnupg
    94  
    95  
    96  Further Details
    97  ===============
    98  
    99  TBD. For now, look in [helpers.bash](helpers.bash); each helper function
   100  has (what are intended to be) helpful header comments. For even more
   101  examples, see and/or run `helpers.t`; that's a regression test
   102  and provides a thorough set of examples of how the helpers work.