github.com/containers/podman/v5@v5.1.0-rc1/test/upgrade/README.md (about)

     1  Background
     2  ==========
     3  
     4  For years we've been needing a way to test podman upgrades; this
     5  became much more critical on December 7, 2020, when Matt disclosed
     6  a bug he had found over the weekend
     7  ([#8613](https://github.com/containers/podman/issues/8613))
     8  in which reuse of a previously-defined field name would
     9  result in fatal JSON decode failures if current-podman were
    10  to try reading containers created with podman <= 1.8 (FIXME: confirm)
    11  
    12  Upgrade testing is a daunting problem; but in the December 12
    13  Cabal meeting Dan suggested using podman-in-podman. This PR
    14  is the result of fleshing out that idea.
    15  
    16  Overview
    17  ========
    18  
    19  The BATS script in this directory fetches and runs an old-podman
    20  container image from quay.io/podman, uses it to create and run
    21  a number of containers, then uses new-podman to interact with
    22  those containers.
    23  
    24  As of 2024-02-05 the available old-podman versions are:
    25  
    26  ```console
    27  $ bin/podman search --list-tags --limit=400 quay.io/podman/stable | awk '$2 ~ /^v/ { print $2}' | sort | column -c 75
    28  v1.4.2  v1.9.1  v3.2.0  v3.4.0  v4.1.0  v4.3.1  v4.5.1  v4.8
    29  v1.4.4  v2.0.2  v3.2.1  v3.4.1  v4.1.1  v4.4    v4.6    v4.8.0
    30  v1.5.0  v2.0.6  v3.2.2  v3.4.2  v4.2    v4.4.1  v4.6.1  v4.8.1
    31  v1.5.1  v2.1.1  v3.2.3  v3.4.4  v4.2.0  v4.4.2  v4.6.2  v4.8.2
    32  v1.6    v2.2.1  v3.3.0  v3.4.7  v4.2.1  v4.4.4  v4.7    v4.8.3
    33  v1.6.2  v3      v3.3.1  v4      v4.3    v4.5    v4.7.0  v4.9
    34  v1.9.0  v3.1.2  v3.4    v4.1    v4.3.0  v4.5.0  v4.7.2  v4.9.0
    35  ```
    36  
    37  Test invocation is:
    38  ```console
    39  $ sudo env PODMAN=bin/podman PODMAN_UPGRADE_FROM=v4.1.0 PODMAN_UPGRADE_TEST_DEBUG= bats test/upgrade
    40  ```
    41  (Path assumes you're cd'ed to top-level podman repo). `PODMAN_UPGRADE_FROM`
    42  can be any of the versions above. `PODMAN_UPGRADE_TEST_DEBUG` is empty
    43  here, but listed so you can set it `=1` and leave the podman_parent
    44  container running. Interacting with this container is left as an
    45  exercise for the reader.
    46  
    47  The script will pull the given podman image, invoke it with a scratch
    48  root directory, and have it do a small set of podman stuff (pull an
    49  image, create/run some containers). This podman process stays running
    50  because if it exits, it kills containers running inside the container.
    51  
    52  We then invoke the current (host-installed) podman, using the same
    53  scratch root directory, and perform operations on those images and
    54  containers. Most of those operations are done in individual @tests.
    55  
    56  The goal is to have this upgrade test run in CI, iterating over a
    57  loop of known old versions. This list would need to be hand-maintained
    58  and updated on new releases. There might also need to be extra
    59  configuration defined, such as per-version commands (see below).
    60  
    61  Findings
    62  ========
    63  
    64  Well, first, `v1.6.2` won't work on default f32/f33: the image
    65  does not include `crun`, so it can't work at all:
    66  
    67      ERRO[0000] oci runtime "runc" does not support CGroups V2: use system migrate to mitigate
    68  
    69  I realize that it's kind of stupid not to test 1.6, since that's
    70  precisely the test that would've caught #8613 early, but I just
    71  don't think it's worth the hassle of setting up cgroupsv1 VMs.
    72  
    73  For posterity, in an earlier incantation of this script I tried
    74  booting f32 into cgroupsv1 and ran into the following warnings
    75  when running new-podman on old-containers:
    76  ```
    77  ERRO[0000] error joining network namespace for container 322b66d94640e31b2e6921565445cf0dade4ec13cabc16ee5f29292bdc038341: error retrieving network namespace at /var/run/netns/cni-577e2289-2c05-2e28-3c3d-002a5596e7da: failed to Statfs "/var/run/netns/cni-577e2289
    78  ```
    79  
    80  Where To Go From Here
    81  =====================
    82  
    83  * Tests are still (2021-02-23) incomplete, with several failing outright.
    84    See FIXMEs in the code.
    85  
    86  * Figuring out how/if to run rootless. I think this is possible, perhaps
    87    even necessary, but will be tricky to get right because of home-directory
    88    mounting.
    89  
    90  * Figuring out how/if to run variations with different config files
    91    (e.g. running OLD-PODMAN that creates a user libpod.conf, tweaking
    92    that in the test, then running NEW-PODMAN upgrade tests)