github.com/KYVENetwork/cometbft/v38@v38.0.3/RELEASES.md (about) 1 # Releases 2 3 CometBFT uses modified [semantic versioning](https://semver.org/) with each 4 release following a `vX.Y.Z` format. CometBFT is currently on major version 0 5 and uses the minor version to signal breaking changes. The `main` branch is 6 used for active development and thus it is not advisable to build against it. 7 8 The latest changes are always initially merged into `main`. Releases are 9 specified using tags and are built from long-lived "backport" branches that are 10 cut from `main` when the release process begins. Each release "line" (e.g. 11 0.34 or 0.33) has its own long-lived backport branch, and the backport branches 12 have names like `v0.34.x` or `v0.33.x` (literally, `x`; it is not a placeholder 13 in this case). CometBFT only maintains the last two releases at a time (the 14 oldest release is predominantly just security patches). 15 16 ## Backporting 17 18 As non-breaking changes land on `main`, they should also be backported to 19 these backport branches. 20 21 We use Mergify's [backport feature](https://mergify.io/features/backports) to 22 automatically backport to the needed branch. There should be a label for any 23 backport branch that you'll be targeting. To notify the bot to backport a pull 24 request, mark the pull request with the label corresponding to the correct 25 backport branch. For example, to backport to v0.38.x, add the label 26 `S:backport-to-v0.38.x`. Once the original pull request is merged, the bot will 27 try to cherry-pick the pull request to the backport branch. If the bot fails to 28 backport, it will open a pull request. The author of the original pull request 29 is responsible for solving the conflicts and merging the pull request. 30 31 ### Creating a backport branch 32 33 If this is the first release candidate for a minor version release, e.g. 34 v0.25.0, you get to have the honor of creating the backport branch! 35 36 Note that, after creating the backport branch, you'll also need to update the 37 tags on `main` so that `go mod` is able to order the branches correctly. You 38 should tag `main` with a "dev" tag that is "greater than" the backport 39 branches tags. Otherwise, `go mod` does not 'know' whether commits on `main` 40 come before or after the release. 41 42 In the following example, we'll assume that we're making a backport branch for 43 the 0.38.x line. 44 45 1. Start on `main` 46 47 2. Ensure that there is a [branch protection 48 rule](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/managing-a-branch-protection-rule) for the 49 branch you are about to create (you will need admin access to the repository 50 in order to do this). 51 52 3. Create and push the backport branch: 53 54 ```sh 55 git checkout -b v0.38.x 56 git push origin v0.38.x 57 ``` 58 59 4. Create a PR to update the documentation directory for the backport branch. 60 61 We rewrite any URLs pointing to `main` to point to the backport branch, 62 so that generated documentation will link to the correct versions of files 63 elsewhere in the repository. The following files are to be excluded from this 64 search: 65 66 * [`README.md`](./README.md) 67 * [`CHANGELOG.md`](./CHANGELOG.md) 68 * [`UPGRADING.md`](./UPGRADING.md) 69 70 The following links are to always point to `main`, regardless of where they 71 occur in the codebase: 72 73 * `https://github.com/KYVENetwork/cometbft/v38/blob/main/LICENSE` 74 75 Be sure to search for all of the following links and replace `main` with your 76 corresponding branch label or version (e.g. `v0.38.x` or `v0.38`): 77 78 * `github.com/KYVENetwork/cometbft/v38/blob/main` -> 79 `github.com/KYVENetwork/cometbft/v38/blob/v0.38.x` 80 * `github.com/KYVENetwork/cometbft/v38/tree/main` -> 81 `github.com/KYVENetwork/cometbft/v38/tree/v0.38.x` 82 * `docs.cometbft.com/main` -> `docs.cometbft.com/v0.38` 83 84 Once you have updated all of the relevant documentation: 85 86 ```sh 87 # Create and push the PR. 88 git checkout -b update-docs-v038x 89 git commit -m "Update docs for v0.38.x backport branch." 90 git push -u origin update-docs-v038x 91 ``` 92 93 Be sure to merge this PR before making other changes on the newly-created 94 backport branch. 95 96 5. Ensure that the RPC docs' `version` field in `rpc/openapi/openapi.yaml` has 97 been updated from `main` to the backport branch version. 98 99 6. Prepare the [CometBFT documentation 100 repository](https://github.com/cometbft/cometbft-docs) to build the release 101 branch's version by updating the 102 [VERSIONS](https://github.com/cometbft/cometbft-docs/blob/main/VERSIONS) 103 file. 104 105 After doing these steps, go back to `main` and do the following: 106 107 1. Create a new workflow to run e2e nightlies for the new backport branch. (See 108 [e2e-nightly-main.yml][e2e] for an example.) 109 110 2. Add a new section to the Mergify config (`.github/mergify.yml`) to enable the 111 backport bot to work on this branch, and add a corresponding `backport-to-v0.38.x` 112 [label](https://github.com/KYVENetwork/cometbft/v38/labels) so the bot can be triggered. 113 114 3. Add a new section to the Dependabot config (`.github/dependabot.yml`) to 115 enable automatic update of Go dependencies on this branch. Copy and edit one 116 of the existing branch configurations to set the correct `target-branch`. 117 118 [e2e]: https://github.com/KYVENetwork/cometbft/v38/blob/main/.github/workflows/e2e-nightly-main.yml 119 120 ## Pre-releases 121 122 Before creating an official release, especially a minor release, we may want to 123 create an alpha or beta version, or release candidate (RC) for our friends and 124 partners to test out. We use git tags to create pre-releases, and we build them 125 off of backport branches, for example: 126 127 * `v0.38.0-alpha.1` - The first alpha release of `v0.38.0`. Subsequent alpha 128 releases will be numbered `v0.38.0-alpha.2`, `v0.38.0-alpha.3`, etc. 129 130 Alpha releases are to be considered the _most_ unstable of pre-releases, and 131 are most likely not yet properly QA'd. These are made available to allow early 132 adopters to start integrating and testing new functionality before we're done 133 with QA. 134 135 * `v0.38.0-beta.1` - The first beta release of `v0.38.0`. Subsequent beta 136 releases will be numbered `v0.38.0-beta.2`, `v0.38.0-beta.3`, etc. 137 138 Beta releases can be considered more stable than alpha releases in that we 139 will have QA'd them better than alpha releases, but there still may be 140 minor breaking API changes if users have strong demands for such changes. 141 142 * `v0.38.0-rc1` - The first release candidate (RC) of `v0.38.0`. Subsequent RCs 143 will be numbered `v0.38.0-rc2`, `v0.38.0-rc3`, etc. 144 145 RCs are considered more stable than beta releases in that we will have 146 completed our QA on them. APIs will most likely be stable at this point. The 147 difference between an RC and a release is that there may still be small 148 changes (bug fixes, features) that may make their way into the series before 149 cutting a final release. 150 151 (Note that branches and tags _cannot_ have the same names, so it's important 152 that these branches have distinct names from the tags/release names.) 153 154 If this is the first pre-release for a minor release, you'll have to make a new 155 backport branch (see above). Otherwise: 156 157 1. Start from the backport branch (e.g. `v0.38.x`). 158 2. Run the integration tests and the E2E nightlies 159 (which can be triggered from the GitHub UI; 160 e.g., <https://github.com/KYVENetwork/cometbft/v38/actions/workflows/e2e-manual.yml>). 161 3. Prepare the pre-release documentation: 162 * Build the changelog with [unclog] _without_ doing an unclog release, and 163 commit the built changelog. This ensures that all changelog entries appear 164 under an "Unreleased" heading in the pre-release's changelog. The changes 165 are only considered officially "released" once we cut a regular (final) 166 release. 167 * Ensure that `UPGRADING.md` is up-to-date and includes notes on any breaking 168 changes or other upgrading flows. 169 4. Prepare the versioning: 170 * Bump TMVersionDefault version in `version.go` 171 * Bump P2P and block protocol versions in `version.go`, if necessary. 172 Check the changelog for breaking changes in these components. 173 * Bump ABCI protocol version in `version.go`, if necessary 174 5. Open a PR with these changes against the backport branch. 175 6. Once these changes have landed on the backport branch, be sure to pull them back down locally. 176 7. Once you have the changes locally, create the new tag, specifying a name and a tag "message": 177 `git tag -a v0.38.0-rc1 -m "Release Candidate v0.38.0-rc1` 178 8. Push the tag back up to origin: 179 `git push origin v0.38.0-rc1` 180 Now the tag should be available on the repo's releases page. 181 9. Future pre-releases will continue to be built off of this branch. 182 183 ## Minor release 184 185 This minor release process assumes that this release was preceded by release 186 candidates. If there were no release candidates, begin by creating a backport 187 branch, as described above. 188 189 Before performing these steps, be sure the 190 [Minor Release Checklist](#minor-release-checklist) has been completed. 191 192 1. Start on the backport branch (e.g. `v0.38.x`) 193 2. Run integration tests (`make test_integrations`) and the e2e nightlies. 194 3. Prepare the release: 195 * Do a [release][unclog-release] with [unclog] for the desired version, 196 ensuring that you write up a good summary of the major highlights of the 197 release that users would be interested in. 198 * Build the changelog using unclog, and commit the built changelog. 199 * Ensure that `UPGRADING.md` is up-to-date and includes notes on any breaking changes 200 or other upgrading flows. 201 * Bump TMVersionDefault version in `version.go` 202 * Bump P2P and block protocol versions in `version.go`, if necessary 203 * Bump ABCI protocol version in `version.go`, if necessary 204 4. Open a PR with these changes against the backport branch. 205 5. Once these changes are on the backport branch, push a tag with prepared release details. 206 This will trigger the actual release `v0.38.0`. 207 * `git tag -a v0.38.0 -m 'Release v0.38.0'` 208 * `git push origin v0.38.0` 209 6. Make sure that `main` is updated with the latest `CHANGELOG.md`, `CHANGELOG_PENDING.md`, and `UPGRADING.md`. 210 211 ## Patch release 212 213 Patch releases are done differently from minor releases: They are built off of 214 long-lived backport branches, rather than from main. As non-breaking changes 215 land on `main`, they should also be backported into these backport branches. 216 217 Patch releases don't have release candidates by default, although any tricky 218 changes may merit a release candidate. 219 220 To create a patch release: 221 222 1. Checkout the long-lived backport branch: `git checkout v0.38.x` 223 2. Run integration tests (`make test_integrations`) and the nightlies. 224 3. Check out a new branch and prepare the release: 225 * Do a [release][unclog-release] with [unclog] for the desired version, 226 ensuring that you write up a good summary of the major highlights of the 227 release that users would be interested in. 228 * Build the changelog using unclog, and commit the built changelog. 229 * Bump the TMDefaultVersion in `version.go` 230 * Bump the ABCI version number, if necessary. (Note that ABCI follows semver, 231 and that ABCI versions are the only versions which can change during patch 232 releases, and only field additions are valid patch changes.) 233 4. Open a PR with these changes that will land them back on `v0.38.x` 234 5. Once this change has landed on the backport branch, make sure to pull it locally, then push a tag. 235 * `git tag -a v0.38.1 -m 'Release v0.38.1'` 236 * `git push origin v0.38.1` 237 6. Create a pull request back to main with the CHANGELOG & version changes from the latest release. 238 * Remove all `R:patch` labels from the pull requests that were included in the release. 239 * Do not merge the backport branch into main. 240 241 ## Minor Release Checklist 242 243 The following set of steps are performed on all releases that increment the 244 _minor_ version, e.g. v0.25 to v0.26. These steps ensure that CometBFT is well 245 tested, stable, and suitable for adoption by the various diverse projects that 246 rely on CometBFT. 247 248 ### Feature Freeze 249 250 Ahead of any minor version release of CometBFT, the software enters 'Feature 251 Freeze' for at least two weeks. A feature freeze means that _no_ new features 252 are added to the code being prepared for release. No code changes should be made 253 to the code being released that do not directly improve pressing issues of code 254 quality. The following must not be merged during a feature freeze: 255 256 * Refactors that are not related to specific bug fixes. 257 * Dependency upgrades. 258 * New test code that does not test a discovered regression. 259 * New features of any kind. 260 * Documentation or spec improvements that are not related to the newly developed 261 code. 262 263 This period directly follows the creation of the [backport 264 branch](#creating-a-backport-branch). The CometBFT team instead directs all 265 attention to ensuring that the existing code is stable and reliable. Broken 266 tests are fixed, flakey-tests are remedied, end-to-end test failures are 267 thoroughly diagnosed and all efforts of the team are aimed at improving the 268 quality of the code. During this period, the upgrade harness tests are run 269 repeatedly and a variety of in-house testnets are run to ensure CometBFT 270 functions at the scale it will be used by application developers and node 271 operators. 272 273 ### Nightly End-To-End Tests 274 275 The CometBFT team maintains [a set of end-to-end 276 tests](https://github.com/KYVENetwork/cometbft/v38/blob/main/test/e2e/README.md#L1) 277 that run each night on the latest commit of the project and on the code in the 278 tip of each supported backport branch. These tests start a network of 279 containerized CometBFT processes and run automated checks that the network 280 functions as expected in both stable and unstable conditions. During the feature 281 freeze, these tests are run nightly and must pass consistently for a release of 282 CometBFT to be considered stable. 283 284 ### Upgrade Harness 285 286 The CometBFT team is creating an upgrade test harness to exercise the workflow 287 of stopping an instance of CometBFT running one version of the software and 288 starting up the same application running the next version. To support upgrade 289 testing, we will add the ability to terminate the CometBFT process at specific 290 pre-defined points in its execution so that we can verify upgrades work in a 291 representative sample of stop conditions. 292 293 ### Large Scale Testnets 294 295 The CometBFT end-to-end tests run a small network (~10s of nodes) to exercise 296 basic consensus interactions. Real world deployments of CometBFT often have 297 over a hundred nodes just in the validator set, with many others acting as full 298 nodes and sentry nodes. To gain more assurance before a release, we will also 299 run larger-scale test networks to shake out emergent behaviors at scale. 300 301 Large-scale test networks are run on a set of virtual machines (VMs). Each VM is 302 equipped with 4 Gigabytes of RAM and 2 CPU cores. The network runs a very simple 303 key-value store application. The application adds artificial delays to different 304 ABCI calls to simulate a slow application. Each testnet is briefly run with no 305 load being generated to collect a baseline performance. Once baseline is 306 captured, a consistent load is applied across the network. This load takes the 307 form of 10% of the running nodes all receiving a consistent stream of two 308 hundred transactions per minute each. 309 310 During each test net, the following metrics are monitored and collected on each 311 node: 312 313 * Consensus rounds per height 314 * Maximum connected peers, Minimum connected peers, Rate of change of peer connections 315 * Memory resident set size 316 * CPU utilization 317 * Blocks produced per minute 318 * Seconds for each step of consensus (Propose, Prevote, Precommit, Commit) 319 * Latency to receive block proposals 320 321 For these tests we intentionally target low-powered host machines (with low core 322 counts and limited memory) to ensure we observe similar kinds of resource contention 323 and limitation that real-world deployments of CometBFT experience in production. 324 325 #### 200 Node Testnet 326 327 To test the stability and performance of CometBFT in a real world scenario, 328 a 200 node test network is run. The network comprises 5 seed nodes, 175 329 validators and 20 non-validating full nodes. All nodes begin by dialing 330 a subset of the seed nodes to discover peers. The network is run for several 331 days, with metrics being collected continuously. In cases of changes to performance 332 critical systems, testnets of larger sizes should be considered. 333 334 #### Rotating Node Testnet 335 336 Real-world deployments of CometBFT frequently see new nodes arrive and old 337 nodes exit the network. The rotating node testnet ensures that CometBFT is 338 able to handle this reliably. In this test, a network with 10 validators and 339 3 seed nodes is started. A rolling set of 25 full nodes are started and each 340 connects to the network by dialing one of the seed nodes. Once the node is able 341 to blocksync to the head of the chain and begins producing blocks using 342 consensus it is stopped. Once stopped, a new node is started and 343 takes its place. This network is run for several days. 344 345 #### Vote-extension Testnet 346 347 CometBFT v0.38.0 introduced **vote-extensions**, which are added as the name suggests, to precommit votes sent by validators. 348 The Vote-extension Testnet is used to determine how vote-extensions affect the performance of CometBFT, under various settings. 349 The application used in the experiment is the same used on the (#200-node-testnet), but is configured differently to gauge de effects of varying vote extension sizes. 350 In the (#200-node-testnet) the application extends pre-commit votes with a 64 bit number encoded with variable compression. 351 In the Vote-extension Testnet, pre-commit votes are extended with a non-compressed extension of configurable size. 352 Experiments are run with multiple sizes to determine their impact and, for comparison sake, we include a run with the same settings as in the (#200-node-testnet). 353 354 The testnet consists of 175 validators, 20 non-validator full-nodes, and 5 seed nodes. 355 All 195 full-nodes begin by dialing a subset of the seed nodes to discover peers. 356 Once all full-nodes are started, a 5 minute period is waited before starting an experiment. 357 For each experiment, the load generators issue requests at a constant rate during 150 seconds, then wait for 5 minutes to allow the system to quiesce, then repeat the load generation; the load generation step is repeated 5 times for each experiment. 358 359 #### Network Partition Testnet 360 361 CometBFT is expected to recover from network partitions. A partition where no 362 subset of the nodes is left with the super-majority of the stake is expected to 363 stop making blocks. Upon alleviation of the partition, the network is expected 364 to once again become fully connected and capable of producing blocks. The 365 network partition testnet ensures that CometBFT is able to handle this 366 reliably at scale. In this test, a network with 100 validators and 95 full 367 nodes is started. All validators have equal stake. Once the network is 368 producing blocks, a set of firewall rules is deployed to create a partitioned 369 network with 50% of the stake on one side and 50% on the other. Once the 370 network stops producing blocks, the firewall rules are removed and the nodes 371 are monitored to ensure they reconnect and that the network again begins 372 producing blocks. 373 374 #### Absent Stake Testnet 375 376 CometBFT networks often run with _some_ portion of the voting power offline. 377 The absent stake testnet ensures that large networks are able to handle this 378 reliably. A set of 150 validator nodes and three seed nodes is started. The set 379 of 150 validators is configured to only possess a cumulative stake of 67% of 380 the total stake. The remaining 33% of the stake is configured to belong to 381 a validator that is never actually run in the test network. The network is run 382 for multiple days, ensuring that it is able to produce blocks without issue. 383 384 [unclog]: https://github.com/informalsystems/unclog 385 [unclog-release]: https://github.com/informalsystems/unclog#releasing-a-new-versions-change-set