github.com/drud/ddev@v1.21.5-alpha1.0.20230226034409-94fcc4b94453/docs/content/developers/release-management.md (about)

     1  # Release Management & Docker Images
     2  
     3  ## Release process and tools
     4  
     5  * [Goreleaser Pro](https://goreleaser.com/pro/) is used to do the actual releasing using [.goreleaser.yml](https://github.com/drud/ddev/blob/master/.goreleaser.yml). Goreleaser Pro is a licensed product that requires a license key, which is in the GitHub Workflow configuration and is available in LastPass to DDEV maintainers who need it.
     6  * The [Master Build/Release GitHub Action](https://github.com/drud/ddev/blob/master/.github/workflows/master-build.yml) does the actual running of the goreleaser actions and provides the needed secrets.
     7  
     8  ## GitHub Actions Required Secrets
     9  
    10  ### How to add new people to these accounts
    11  
    12  * AUR is Arch Linux User Repository. `ddev-bin` is at `https://aur.archlinux.org/packages/ddev-bin`. The current maintainer of this is @cweagans, who can add co-maintainers.
    13  * The [chocolatey](https://community.chocolatey.org/packages/ddev/) package. Additional maintainers can be added at [link](https://community.chocolatey.org/packages/ddev/1.21.1/ManagePackageOwners); they could then create tokens to push it.
    14  * GitHub requires write access to this repository, either via permissions on the repository or on the org.
    15  * Apple signing and notarization requires access to the Localdev Foundation group on `https://developer.apple.com`. It's easy enough to add additional people.
    16  * Windows signing is an awkward process that requires a dongle. When the current signing certificate expires we definitely want the simpler approach.
    17  * Discord
    18  * Docker
    19  
    20  ### Actual secrets required
    21  
    22  <!-- markdown-link-check-disable-next-line -->
    23  The following “Repository secret” environment variables must be added to <https://github.com/drud/ddev/settings/secrets/actions>:
    24  
    25  * `AUR_SSH_PRIVATE_KEY`: Private ssh key for the `ddev-releaser` user. This must be processed into a single line, for example, `perl -p -e 's/\n/<SPLIT>/' ~/.ssh/id_rsa_ddev_releaser| pbcopy`.
    26  * `CHOCOLATEY_API_KEY`: API key for Chocolatey.
    27  * `DDEV_GITHUB_TOKEN`: GitHub token that gives access to create releases and push to the Homebrew repositories.
    28  * `DDEV_MACOS_APP_PASSWORD`: Password used for notarization, see [signing_tools](https://github.com/drud/signing_tools).
    29  * `DDEV_MACOS_SIGNING_PASSWORD`: Password for the macOS signing key, see [signing_tools](https://github.com/drud/signing_tools).
    30  * `DDEV_WINDOWS_SIGNING_PASSWORD`: Windows signing password.
    31  * `SegmentKey`: Key that enables Segment reporting.
    32  * `FURY_ACCOUNT`: [Gemfury](https://gemfury.com) account that receives package pushes.
    33  * `FURY_TOKEN`: Push token assigned to the above Gemfury account.
    34  * `AUR_STABLE_GIT_URL`: The Git URL for AUR stable (normally `ddev-bin`), for example `ssh://aur@aur.archlinux.org/ddev-bin.git`.
    35  * `AUR_EDGE_GIT_URL`: The Git URL for AUR edge (normally `ddev-edge-bin`), for example `ssh://aur@aur.archlinux.org/ddev-edge-bin.git`.
    36  
    37  ## Creating a Release
    38  
    39  !!!tip "This is completely automated now, so nothing needs to be done unless something goes wrong."
    40  
    41  ### Prerelease Tasks
    42  
    43  * Create and execute a test plan.
    44  * Make sure [`version-history.md`](https://github.com/drud/ddev/blob/master/version-history.md) is up to date.
    45  * Make sure the Docker images are all tagged and pushed.
    46  * Make sure [`pkg/versionconstants/versionconstants.go`](https://github.com/drud/ddev/blob/master/pkg/versionconstants/versionconstants.go) is all set to point to the new images and tests have been run.
    47  
    48  ### Actual Release Creation
    49  
    50  1. Create a [release](https://github.com/drud/ddev/releases) for the new version using the GitHub UI. It should be “prerelease” if it’s an edge release.
    51  2. Use the “Auto-generate release notes” option to get the commit list, then edit to add all the other necessary info.
    52  3. Verify that Homebrew (Linux and macOS) and Chocolatey and AUR are working correctly with the right versions.
    53  
    54  ## Pushing Docker Images with the GitHub Actions Workflow
    55  
    56  The easiest way to push Docker images is to use the GitHub Actions workflow, especially if the code for the image is already in the [drud/ddev](https://github.com/drud/ddev) repository.
    57  
    58  ### Prerelease tasks
    59  
    60  1. Push the new version of drud/ddev-php-base
    61  2. Update drud/ddev-webserver to use the new version of drud/ddev-php-base and push it with the proper tag
    62  3. Make sure the `version-history.md` file is up to date.
    63  4. Make sure the docker images for the new tag are all tagged and pushed.
    64  5. Make sure the pkg/version/version.go is all set to point to the new images (and tests have been run)
    65  6. Make sure you're about to create the right release tag.
    66  
    67  ### Actual release creation
    68  
    69  1. Create a release for the new version using the GitHub UI. It should be "prerelease" if it's an edge release. Use the right tag!
    70  2. Use the "Auto-generate release notes" option to get the commit list, then edit to add all the other necessary info.
    71  3. Verify that homebrew (linux and macOS) and Chocolatey and AUR are working correctly with the right versions
    72  
    73  ## Pushing docker images with the GitHub Actions workflow
    74  
    75  The easiest way to push docker images is to use the GitHub Actions workflow, especially if the code for the image is already in the ddev repo.
    76  
    77  <!-- markdown-link-check-disable-next-line -->
    78  You can push an image at <https://github.com/drud/ddev/actions/workflows/push-tagged-image.yml>
    79  
    80  <!-- markdown-link-check-disable-next-line -->
    81  If you need to push from a forked PR, you’ll have to do this from your fork (for example, `https://github.com/drud/rfay/actions/workflows/push-tagged-image.yml`), and you’ll have to specify the branch on the fork. This requires setting the `DOCKERHUB_TOKEN` and `DOCKERHUB_USERNAME` secrets on the forked PR, for example `https://github.com/rfay/ddev/settings/secrets/actions`.
    82  
    83  * Visit `https://github.com/drud/ddev/actions/workflows/push-tagged-image.yml`.
    84  * Click the “Push tagged image” workflow on the left side of the page.
    85  * Click the “Run workflow” button in the blue section above the workflow runs.
    86  * Choose the branch to build from (usually `master`).
    87  * Enter the image (`ddev-webserver`, `ddev-dbserver`, `ddev-php-base`, etc.).
    88  * Enter the tag that will be used in `pkg/version/version.go`.
    89  
    90  ## Pushing Docker Images Manually
    91  
    92  While it’s more error-prone, images can be pushed from the command line:
    93  
    94  1. `docker login` with a user that has push privileges.
    95  2. `docker buildx create --name ddev-builder-multi --use` or if it already exists, `docker buildx use ddev-builder-multi`.
    96  3. `cd containers/<image>`.
    97  4. Before pushing `ddev-webserver`, make sure you’ve pushed a version of `ddev-php-base` and updated `ddev-webserver`’s Dockerfile to use that as a base.
    98  5. `make push VERSION=<release_version> DOCKER_ARGS=--no-cache` for most of the images. For `ddev-dbserver` it’s `make PUSH=true VERSION=<release_version> DOCKER_ARGS=--no-cache`. There’s a [push-all.sh](https://github.com/drud/ddev/blob/master/containers/push-all.sh) script to update all of them, but it takes forever.
    99  
   100  ## Maintaining `ddev-dbserver` MySQL 5.7 & 8.0 ARM64 Images
   101  
   102  Sadly, there are no ARM64 Docker images for MySQL 5.7 and 8.0, so we have our own process to maintain [drud/mysql-arm64-images](https://github.com/drud/mysql-arm64-images) and [drud/xtrabackup-build](https://github.com/drud/xtrabackup-build) images for DDEV.
   103  
   104  * `drud/mysql:5.7` uses Ubuntu 18.04 as the base image, and Ubuntu 18.04 ARM64 has `mysql-server` 5.7 in it, so we can install.
   105  * `drud/mysql:8.0` uses Ubuntu 20.04 as the base image, and Ubuntu 20.04 ARM64 has `mysql-server` 8.0 in it, so we can install it from packages.
   106  * Unfortunately, the [`ddev snapshot`](../users/usage/commands.md#snapshot) command depends on `xtrabackup` 8.0 being installed for `mysql:8.0`. There are no ARM64 packages or binaries provided by Percona for `xtrabackup`, so we build it from source with [drud/xtrabackup-build](https://github.com/drud/xtrabackup-build). **There’s a catch, however:** `xtrabackup`’s development cycle lags behind `mysql:8.0`’s development cycle, so you can’t build a usable `drud/mysql:8.0` image until there’s an `xtrabackup` version released. Further, when Ubuntu bumps `mysql-server-8.0` to a new version, there’s no way to use the old one. So the only time that you can maintain `drud/mysql:8.0` is when Ubuntu 20.04 has the same version that’s released for `percona-xtrabackup`. (In the case at this writeup, I was finally able to build `percona-xtrabackup` 8.0.28, and the same day Ubuntu bumped its packages to 8.0.29, meaning that it was unusable.)
   107  * To build percona-xtrabackup, follow the instructions on [drud/xtrabackup-build](https://github.com/drud/xtrabackup-build). You just create a release with the release of Percona xtrabackup, for example `8.0.29-21`. When that succeeds, then there is an upstream xtrabackup to be used in the drud/mysql:8.0 build.
   108  * To build `drud/mysql` (both 5.7 and 8.0) ARM64 images, follow the instructions on [drud/mysql-arm64-images](https://github.com/drud/mysql-arm64-images). After the various files are updated, you can push a new release and the proper images will be pushed.
   109  * After building a new set of `drud/mysql` images, you’ll need to push `drud/ddev-dbserver` with new tags. Make sure to update the [drud/`ddev-dbserver` Makefile](https://github.com/drud/ddev/blob/master/containers/ddev-dbserver/Makefile) to set the explicit version of the upstream `mysql:8.0` (for example, 8.0.29, if you’ve succeeded in getting 8.0.29 for `percona-xtrabackup` and `mysql:8.0`).
   110  
   111  ## Actual Release Docker Image Updates
   112  
   113  We don’t actually build every image for every point release. If there have been no changes to `ddev-router` or `ddev-ssh-agent`, for example, we only usually push those and update `pkg/version/version.go` on major releases.
   114  
   115  But here are the steps for building:
   116  
   117  1. The `drud/ddev-php-base` image must be updated as necessary with a new tag before pushing `ddev-webserver`. You can do this using the [process above](#pushing-docker-images-with-the-github-actions-workflow).
   118  2. The `drud/ddev-webserver` Dockerfile must `FROM drud/ddev-php-base:<tag>` before building/pushing `ddev-webserver`. But then it can be pushed using either the GitHub Actions or the manual technique.
   119  3. If you’re bumping `ddev-dbserver` 8.0 minor release, follow the upstream [Maintaining ddev-dbserver MySQL 5.7 & 8.0 ARM64 Images](#maintaining-ddev-dbserver-mysql-57--80-arm64-images) instructions.
   120  4. Update `pkg/version/version.go` with the correct versions for the new images, and run all the tests.
   121  
   122  ## Manually Updating Homebrew Formulas
   123  
   124  Homebrew formulas normally update with the release process, so nothing needs to be done.
   125  
   126  If you have to temporarily update the Homebrew formulas, you can do that with a commit to <https://github.com/drud/homebrew-ddev> and <https://github.com/drud/homebrew-ddev-edge>. The bottles and checksums for macOS (High Sierra) and x86_64_linux are built and pushed to the release page automatically by the release build process (see [bump_homebrew.sh](https://github.com/drud/ddev/blob/master/.ci-scripts/bump_homebrew.sh)). Test `brew upgrade ddev` both on macOS and Linux and make sure DDEV is the right version and behaves well.
   127  
   128  ## Manually Updating Chocolatey
   129  
   130  Normally the release process does okay with pushing to Chocolatey, but at times a failure can happen and it’s not worth doing the whole release process again.
   131  
   132  Note that if an existing approved release is being updated you have to have a new version. So for example, if `v1.21.3` failed, you'll need to work with `v1.21.3.1`, so `make chocolatey VERSION=v1.21.3.1` below.
   133  
   134  * Open up Gitpod, <https://gitpod.io/#https://github.com/drud/ddev> and
   135  
   136  ```bash
   137  cd /workspace/ddev
   138  git checkout <tag>
   139  sudo apt-get update && sudo apt-get install -y nsis
   140  sudo .ci-scripts/nsis_setup.sh /usr/share/nsis
   141  ```
   142  
   143  * Edit the checksum in `tools/chocolateyinstall.ps1` to match the released checksum of the `ddev-windows-installer` in `checksums.txt` of the release that is being repaired, for example, for `v1.21.3` this would be the checksum for `ddev_windows_installer.v1.21.3.exe` in [v1.21.3 checksums.txt](https://github.com/drud/ddev/releases/download/v1.21.3/checksums.txt).
   144  * Edit `url64` in `tools/chocolateyinstall.ps1` to be the intended actual DDEV download version - edit the version where it appears and edit the GitHub org. For example, if the actual version of DDEV to be downloaded is `v1.21.3` then put that there.
   145  
   146  ```bash
   147  make chocolatey VERSION=<tag>
   148  export CHOCOLATEY_API_KEY=key33333
   149  cd .gotmp/bin/windows_amd64/chocolatey
   150  docker run --rm -v $PWD:/tmp/chocolatey -w /tmp/chocolatey linuturk/mono-choco push -s https://push.chocolatey.org/ --api-key "${CHOCOLATEY_API_KEY}"
   151  
   152  ```
   153  
   154  ## Manually Updating AUR Repository
   155  
   156  The AUR repository normally updates with the release process, so nothing needs to be done.
   157  
   158  However, you can manually publish the release to [the DDEV AUR repository](https://aur.archlinux.org/packages/ddev-bin/). The README.md in the AUR git repo (`ssh://aur@aur.archlinux.org/ddev-bin.git` or `https://aur.archlinux.org/ddev-bin.git`) has instructions on how to update, including how to do it with a Docker container, so it doesn’t have to be done on an ArchLinux or Manjaro VM.
   159  
   160  ## Manually Signing the Windows Installer
   161  
   162  !!!tip "This is done by the release process, but the manual process is documented here."
   163  
   164  This is done automatically by the release build on a dedicated Windows test runner (GitHub Actions runner) named `testbot-asus-win10pro`. You would need to do this process manually on that build machine or install the fob on another machine.
   165  
   166  **After rebooting this machine, sometimes an automated reboot, the password for the security fob has to be re-entered or Windows signing will fail. We do this by opening up `tb-win11-06` using Chrome Remote Desktop (or manually physically opening it), opening Git Bash, and `cd ~/tmp && signtool sign gsudo.exe`. There happens to be a `gsudo.exe` there but it doesn’t matter what you sign—the idea is to pop up the GUI where you enter the password (which is in LastPass).**
   167  
   168  ### Basic Instructions
   169  
   170  1. Install the suggested [Windows SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/). Only the signing component is required.
   171  2. Add the path of the kit binaries to the Windows system PATH, `C:/Program Files (x86)/Windows Kits/10/bin/10.0.22621.0/x64/`.
   172  3. The keyfob and Safenet Authentication Client must be installed. The best documentation for the Safenet software is at <https://support.globalsign.com/ssl/ssl-certificates-installation/safenet-drivers>. You must configure the advanced client settings to “Enable single logon” or it will require the password on each run.
   173  4. After `make windows_install` the `ddev-windows-installer.exe` will be in `.ddev/bin/windows_amd64/ddev_windows_installer.exe` and you can sign it with `signtool sign ddev-windows-installer.exe`.
   174  5. If you need to install the GitHub self-hosted Windows runner, do it with the instructions in project settings → Actions → Runners.
   175  6. Currently the `actions/cache` runner does not work out of the box on Windows, so you have to install tar and zstd as described in [this issue](https://github.com/actions/cache/issues/580#issuecomment-1165839728).
   176  
   177  !!!tip "We shouldn’t use this high-security keyfob approach to signing on the next go-around with the certs."
   178      It’s way too difficult to manage, and the Safenet software is atrocious.