github.com/letsencrypt/boulder@v0.20251208.0/docs/release.md (about)

     1  # Boulder Release Process
     2  
     3  A description and demonstration of the full process for tagging a normal weekly
     4  release and a hotfix release.
     5  
     6  Once a release is tagged, it will be generally deployed to
     7  [staging](https://letsencrypt.org/docs/staging-environment/) and then to
     8  [production](https://acme-v02.api.letsencrypt.org/) over the next few days.
     9  
    10  ## Goals
    11  
    12  1. All development, including reverts and hotfixes needed to patch a broken
    13     release, happens on the `main` branch of this repository. Code is never
    14     deployed without being reviewed and merged here first, and code is never
    15     landed on a release branch that isn't landed on `main` first.
    16  
    17  2. Doing a normal release requires approximately zero thought. It Just Works.
    18  
    19  3. Doing a hotfix release differs as little as possible from the normal release
    20     process.
    21  
    22  ## Release Schedule
    23  
    24  Boulder developers make a new release at the beginning of each week, typically
    25  around 10am PST **Monday**. Operations deploys the new release to the [staging
    26  environment](https://letsencrypt.org/docs/staging-environment/) on **Tuesday**,
    27  typically by 2pm PST. If there have been no issues discovered with the release
    28  from its time in staging, then on **Thursday** the operations team deploys the
    29  release to the production environment.
    30  
    31  Holidays, unexpected bugs, and other resource constraints may affect the above
    32  schedule and result in staging or production updates being skipped. It should be
    33  considered a guideline for normal releases but not a strict contract.
    34  
    35  ## Release Structure
    36  
    37  As of 2025-06-30, releases are tagged with a tag of the form `v0.YYYYMMDD.N`, where
    38  the `YYYYMMDD` is the date that the initial release is cut (usually the Monday
    39  of the current week), and `N` is an integer indicating the hotfix number,
    40  starting at `0`. For example, a regular release might be `v0.20250707.0`, and
    41  the first hotfix for that release would be `v0.20250707.1`.
    42  
    43  Historically, releases were tagged with the form `release-YYYY-MM-DD[x]`, where
    44  `[x]` was an optional lowercase letter suffix for hotfixes. For example, the
    45  second hotfix release (i.e. third release overall) in the third week of
    46  January 2022 was [`release-2022-01-18b`](https://github.com/letsencrypt/boulder/releases/tag/release-2022-01-18b).
    47  
    48  All release tags are signed with a key associated with a Boulder developer. Tag
    49  signatures are automatically verified by GitHub using the public keys that
    50  developer has uploaded, and are additionally checked before being built and
    51  deployed to our staging and production environments. Note that, due to how Git
    52  works, in order for a tag to be signed it must also have a message; we set the
    53  tag message to just be a slightly more readable version of the tag name.
    54  
    55  ## Making a Release
    56  
    57  ### Prerequisites
    58  
    59  * You must have a GPG key with signing capability:
    60    * [Checking for existing GPG keys](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/checking-for-existing-gpg-keys)
    61  
    62  * If you don't have a GPG key with signing capability, create one:
    63    * [Generating a new local GPG key](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/generating-a-new-gpg-key)
    64    * [Generating a new Yubikey GPG key](https://support.yubico.com/hc/en-us/articles/360013790259-Using-Your-YubiKey-with-OpenPGP)
    65  
    66  * The signing GPG key must be added to your GitHub account:
    67    * [Adding a new GPG key to your GitHub
    68      account](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/adding-a-new-gpg-key-to-your-github-account)
    69  
    70  * `git` *may* need to be configured to call the correct GPG binary:
    71    * The default: `git config --global gpg.program gpg` is correct for most Linux platforms
    72    * On macOS and some Linux platforms: `git config --global gpg.program gpg2` is correct
    73  
    74  * `git` must be configured to use the correct GPG key:
    75    * [Telling Git about your GPG key](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/telling-git-about-your-signing-key)
    76  
    77  * Understand the [process for signing tags](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/signing-tags)
    78  
    79  ### Regular Releases
    80  
    81  Simply create a signed tag. The `tools/release/tag` tool will automatically
    82  determine the correct tag name based on the current date.
    83  
    84  ```sh
    85  go run github.com/letsencrypt/boulder/tools/release/tag@main
    86  ```
    87  
    88  This will print the newly-created tag and instructions on how to push it after
    89  you are satisfied that it is correct. Alternately you can run the command with
    90  the `-push` flag to push the resulting tag automatically.
    91  
    92  ### Hotfix Releases
    93  
    94  Sometimes it is necessary to create a new release which looks like a prior
    95  release but with one or more additional commits added. This is usually the case
    96  when we discover a critical bug in the currently-deployed version that needs to
    97  be fixed, but we don't want to include other changes that have already been
    98  merged to `main` since the currently-deployed release was tagged.
    99  
   100  In this situation, we create a new hotfix release branch starting at the point
   101  of the previous release tag. We then use the normal GitHub PR and code-review
   102  process to copy the necessary fix(es) from `main` (where they must already be
   103  merged) to the release branch. Finally we create a new release tag at the tip
   104  of the release branch instead of the tip of main.
   105  
   106  To create the new release branch, substitute the name of the release tag which you want to use as the starting point into this command:
   107  
   108  ```sh
   109  go run github.com/letsencrypt/boulder/tools/release/branch@main v0.YYYYMMDD.0
   110  ```
   111  
   112  This will create a release branch named `release-branch-v0.YYYYMMDD`. When all necessary PRs have been merged into that branch, create the new tag by substituting the branch name into this command:
   113  
   114  ```sh
   115  go run github.com/letsencrypt/boulder/tools/release/tag@main release-branch-v0.YYYYMMDD
   116  ```
   117  
   118  ## Deploying Releases
   119  
   120  When doing a release, SRE's tooling will check that:
   121  
   122  1. GitHub shows that tests have passed for the commit at the planned release
   123     tag.
   124  
   125  2. The planned release tag is an ancestor of the current `main` on GitHub, or
   126     the planned release tag is equal to the head of a branch named
   127     `release-branch-XXX`, and all commits between `main` and the head of that
   128     branch are cherry-picks of commits which landed on `main` following the
   129     normal review process.
   130  
   131  These checks ensure that all deployed code has been properly reviewed and tested before reaching production environments.