k8s.io/kubernetes@v1.31.0-alpha.0.0.20240520171757-56147500dadc/CHANGELOG/CHANGELOG-1.29.md (about) 1 <!-- BEGIN MUNGE: GENERATED_TOC --> 2 3 - [v1.29.5](#v1295) 4 - [Downloads for v1.29.5](#downloads-for-v1295) 5 - [Source Code](#source-code) 6 - [Client Binaries](#client-binaries) 7 - [Server Binaries](#server-binaries) 8 - [Node Binaries](#node-binaries) 9 - [Container Images](#container-images) 10 - [Changelog since v1.29.4](#changelog-since-v1294) 11 - [Changes by Kind](#changes-by-kind) 12 - [Bug or Regression](#bug-or-regression) 13 - [Other (Cleanup or Flake)](#other-cleanup-or-flake) 14 - [Dependencies](#dependencies) 15 - [Added](#added) 16 - [Changed](#changed) 17 - [Removed](#removed) 18 - [v1.29.4](#v1294) 19 - [Downloads for v1.29.4](#downloads-for-v1294) 20 - [Source Code](#source-code-1) 21 - [Client Binaries](#client-binaries-1) 22 - [Server Binaries](#server-binaries-1) 23 - [Node Binaries](#node-binaries-1) 24 - [Container Images](#container-images-1) 25 - [Changelog since v1.29.3](#changelog-since-v1293) 26 - [Important Security Information](#important-security-information) 27 - [CVE-2024-3177: Bypassing mountable secrets policy imposed by the ServiceAccount admission plugin](#cve-2024-3177-bypassing-mountable-secrets-policy-imposed-by-the-serviceaccount-admission-plugin) 28 - [Changes by Kind](#changes-by-kind-1) 29 - [Feature](#feature) 30 - [Bug or Regression](#bug-or-regression-1) 31 - [Dependencies](#dependencies-1) 32 - [Added](#added-1) 33 - [Changed](#changed-1) 34 - [Removed](#removed-1) 35 - [v1.29.3](#v1293) 36 - [Downloads for v1.29.3](#downloads-for-v1293) 37 - [Source Code](#source-code-2) 38 - [Client Binaries](#client-binaries-2) 39 - [Server Binaries](#server-binaries-2) 40 - [Node Binaries](#node-binaries-2) 41 - [Container Images](#container-images-2) 42 - [Changelog since v1.29.2](#changelog-since-v1292) 43 - [Changes by Kind](#changes-by-kind-2) 44 - [Feature](#feature-1) 45 - [Bug or Regression](#bug-or-regression-2) 46 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-1) 47 - [Dependencies](#dependencies-2) 48 - [Added](#added-2) 49 - [Changed](#changed-2) 50 - [Removed](#removed-2) 51 - [v1.29.2](#v1292) 52 - [Downloads for v1.29.2](#downloads-for-v1292) 53 - [Source Code](#source-code-3) 54 - [Client Binaries](#client-binaries-3) 55 - [Server Binaries](#server-binaries-3) 56 - [Node Binaries](#node-binaries-3) 57 - [Container Images](#container-images-3) 58 - [Changelog since v1.29.1](#changelog-since-v1291) 59 - [Changes by Kind](#changes-by-kind-3) 60 - [Feature](#feature-2) 61 - [Bug or Regression](#bug-or-regression-3) 62 - [Dependencies](#dependencies-3) 63 - [Added](#added-3) 64 - [Changed](#changed-3) 65 - [Removed](#removed-3) 66 - [v1.29.1](#v1291) 67 - [Downloads for v1.29.1](#downloads-for-v1291) 68 - [Source Code](#source-code-4) 69 - [Client Binaries](#client-binaries-4) 70 - [Server Binaries](#server-binaries-4) 71 - [Node Binaries](#node-binaries-4) 72 - [Container Images](#container-images-4) 73 - [Changelog since v1.29.0](#changelog-since-v1290) 74 - [Changes by Kind](#changes-by-kind-4) 75 - [API Change](#api-change) 76 - [Feature](#feature-3) 77 - [Bug or Regression](#bug-or-regression-4) 78 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-2) 79 - [Dependencies](#dependencies-4) 80 - [Added](#added-4) 81 - [Changed](#changed-4) 82 - [Removed](#removed-4) 83 - [v1.29.0](#v1290) 84 - [Downloads for v1.29.0](#downloads-for-v1290) 85 - [Source Code](#source-code-5) 86 - [Client Binaries](#client-binaries-5) 87 - [Server Binaries](#server-binaries-5) 88 - [Node Binaries](#node-binaries-5) 89 - [Container Images](#container-images-5) 90 - [Changelog since v1.28.0](#changelog-since-v1280) 91 - [Urgent Upgrade Notes](#urgent-upgrade-notes) 92 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade) 93 - [Changes by Kind](#changes-by-kind-5) 94 - [Deprecation](#deprecation) 95 - [API Change](#api-change-1) 96 - [Feature](#feature-4) 97 - [Documentation](#documentation) 98 - [Failing Test](#failing-test) 99 - [Bug or Regression](#bug-or-regression-5) 100 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-3) 101 - [Dependencies](#dependencies-5) 102 - [Added](#added-5) 103 - [Changed](#changed-5) 104 - [Removed](#removed-5) 105 - [v1.29.0-rc.2](#v1290-rc2) 106 - [Downloads for v1.29.0-rc.2](#downloads-for-v1290-rc2) 107 - [Source Code](#source-code-6) 108 - [Client Binaries](#client-binaries-6) 109 - [Server Binaries](#server-binaries-6) 110 - [Node Binaries](#node-binaries-6) 111 - [Container Images](#container-images-6) 112 - [Changelog since v1.29.0-rc.1](#changelog-since-v1290-rc1) 113 - [Changes by Kind](#changes-by-kind-6) 114 - [Feature](#feature-5) 115 - [Dependencies](#dependencies-6) 116 - [Added](#added-6) 117 - [Changed](#changed-6) 118 - [Removed](#removed-6) 119 - [v1.29.0-rc.1](#v1290-rc1) 120 - [Downloads for v1.29.0-rc.1](#downloads-for-v1290-rc1) 121 - [Source Code](#source-code-7) 122 - [Client Binaries](#client-binaries-7) 123 - [Server Binaries](#server-binaries-7) 124 - [Node Binaries](#node-binaries-7) 125 - [Container Images](#container-images-7) 126 - [Changelog since v1.29.0-rc.0](#changelog-since-v1290-rc0) 127 - [Dependencies](#dependencies-7) 128 - [Added](#added-7) 129 - [Changed](#changed-7) 130 - [Removed](#removed-7) 131 - [v1.29.0-rc.0](#v1290-rc0) 132 - [Downloads for v1.29.0-rc.0](#downloads-for-v1290-rc0) 133 - [Source Code](#source-code-8) 134 - [Client Binaries](#client-binaries-8) 135 - [Server Binaries](#server-binaries-8) 136 - [Node Binaries](#node-binaries-8) 137 - [Container Images](#container-images-8) 138 - [Changelog since v1.29.0-alpha.3](#changelog-since-v1290-alpha3) 139 - [Changes by Kind](#changes-by-kind-7) 140 - [API Change](#api-change-2) 141 - [Feature](#feature-6) 142 - [Bug or Regression](#bug-or-regression-6) 143 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-4) 144 - [Dependencies](#dependencies-8) 145 - [Added](#added-8) 146 - [Changed](#changed-8) 147 - [Removed](#removed-8) 148 - [v1.29.0-alpha.3](#v1290-alpha3) 149 - [Downloads for v1.29.0-alpha.3](#downloads-for-v1290-alpha3) 150 - [Source Code](#source-code-9) 151 - [Client Binaries](#client-binaries-9) 152 - [Server Binaries](#server-binaries-9) 153 - [Node Binaries](#node-binaries-9) 154 - [Container Images](#container-images-9) 155 - [Changelog since v1.29.0-alpha.2](#changelog-since-v1290-alpha2) 156 - [Urgent Upgrade Notes](#urgent-upgrade-notes-1) 157 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-1) 158 - [Changes by Kind](#changes-by-kind-8) 159 - [Deprecation](#deprecation-1) 160 - [API Change](#api-change-3) 161 - [Feature](#feature-7) 162 - [Documentation](#documentation-1) 163 - [Failing Test](#failing-test-1) 164 - [Bug or Regression](#bug-or-regression-7) 165 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-5) 166 - [Dependencies](#dependencies-9) 167 - [Added](#added-9) 168 - [Changed](#changed-9) 169 - [Removed](#removed-9) 170 - [v1.29.0-alpha.2](#v1290-alpha2) 171 - [Downloads for v1.29.0-alpha.2](#downloads-for-v1290-alpha2) 172 - [Source Code](#source-code-10) 173 - [Client Binaries](#client-binaries-10) 174 - [Server Binaries](#server-binaries-10) 175 - [Node Binaries](#node-binaries-10) 176 - [Container Images](#container-images-10) 177 - [Changelog since v1.29.0-alpha.1](#changelog-since-v1290-alpha1) 178 - [Changes by Kind](#changes-by-kind-9) 179 - [Feature](#feature-8) 180 - [Failing Test](#failing-test-2) 181 - [Bug or Regression](#bug-or-regression-8) 182 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-6) 183 - [Dependencies](#dependencies-10) 184 - [Added](#added-10) 185 - [Changed](#changed-10) 186 - [Removed](#removed-10) 187 - [v1.29.0-alpha.1](#v1290-alpha1) 188 - [Downloads for v1.29.0-alpha.1](#downloads-for-v1290-alpha1) 189 - [Source Code](#source-code-11) 190 - [Client Binaries](#client-binaries-11) 191 - [Server Binaries](#server-binaries-11) 192 - [Node Binaries](#node-binaries-11) 193 - [Container Images](#container-images-11) 194 - [Changelog since v1.28.0](#changelog-since-v1280-1) 195 - [Changes by Kind](#changes-by-kind-10) 196 - [Deprecation](#deprecation-2) 197 - [API Change](#api-change-4) 198 - [Feature](#feature-9) 199 - [Documentation](#documentation-2) 200 - [Failing Test](#failing-test-3) 201 - [Bug or Regression](#bug-or-regression-9) 202 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-7) 203 - [Dependencies](#dependencies-11) 204 - [Added](#added-11) 205 - [Changed](#changed-11) 206 - [Removed](#removed-11) 207 208 <!-- END MUNGE: GENERATED_TOC --> 209 210 # v1.29.5 211 212 213 ## Downloads for v1.29.5 214 215 216 217 ### Source Code 218 219 filename | sha512 hash 220 -------- | ----------- 221 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes.tar.gz) | c08b09b3fb97968f9e6c59bccf20215cdbb61ebfee641bb8bae75262876eb855d7758017d7e5bf95987e6f728a28fa43da6a4025f532a9b2795349fba67dba6e 222 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-src.tar.gz) | ea8fd3c4b89ba2dcd0897124df9d9bbbe7d816a2d2e3d7a9a4dcb803f7f6aaaabe1dc9602cb37f698de3e846ad95ec0c4a8143b2a8214c1561f2753f165cc347 223 224 ### Client Binaries 225 226 filename | sha512 hash 227 -------- | ----------- 228 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-darwin-amd64.tar.gz) | 6bd763b2f5fa0e212551eb520a9e1f5f4e3ad8ed5b621b17d48a5f09b7f338ee98d8a6edaec182cd45de7279e56ddf1b93dd117512e3f44eb94624b34f819b70 229 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-darwin-arm64.tar.gz) | 89571830f9f957b941c87a6f7073fa7942a4e13cf5b3001bd6433811c022fc3b514d30bfc9d35ce0683ce1d9bc66a0aee8416556187e6fe5d58fab3ab98e061a 230 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-linux-386.tar.gz) | b0c225ae6e17a193a0532129e5b98f19a29dfc2d0075505738744bf265c53dc9b3219bbd0688ef75994f27fa3d0108182b9916de69a8f13c308ed91624dfc9c8 231 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-linux-amd64.tar.gz) | 8c03f8aa1cfe90b15f5ef590ef2224100a42aba7518049d4a65a10f2898b6c6354a66d2c3a2d24e06d5a34ff1dffaed8073fc3918768374a499e70ecc0fce90e 232 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-linux-arm.tar.gz) | 64e6ad92f21537e3191d8fe53d05a47074fd3b2a32d0bacdb001f74a65a8c55c92a83dc04b5151db3b62df9c81846a730a0f08b30e177a35fc92b8069cae94cf 233 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-linux-arm64.tar.gz) | 56c2a700ec996c1310747b7576ba155d018bf9496d131ec2dc69fe332fbd5c2e4b0ff7d6f3bd6dc621c5f6b067e5a540abc8a6450188fc50bd0152381d1d436e 234 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-linux-ppc64le.tar.gz) | d70cf260be09a1268c6c49b16ecc34ce875cdd9ddb544b9b4f445f08635b0bbb0039a0a287a38c757294cd2cb57a5b297e9eebd8192cd678656498b43bedd725 235 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-linux-s390x.tar.gz) | 7b373ac1336bda248b98bac9d92a44cb584436d3fb0b7092eda1510142c858f3ea91b73d3a64c556d60a0f2549d3a8da64f3263ae371b2b60fd73ecc917eb59e 236 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-windows-386.tar.gz) | fe58b678675a4b7de5e810cd7206de09d2f384fbad0abff8f74ee51ab0e3f8d0686dfb6ff1d556bfa4fbdc0a38f475d41447aea51041a5e6d99363444242d94c 237 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-windows-amd64.tar.gz) | b791403b1ae104afa945d1988bfde62330ff33661478b472a47410f4a5faca243ea790060be327a7946e7a45cf60d23204e6a7ce0809f0d29ce79ee5ffab5fe3 238 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-client-windows-arm64.tar.gz) | d29bd253bbd8a9c20924651cb9025de1266968ffd835c4af46d5ff0c867e454fd0f4995563add753c157fc4486b8b7a25d5fed439f7f13b8c0d14f98764cb269 239 240 ### Server Binaries 241 242 filename | sha512 hash 243 -------- | ----------- 244 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-server-linux-amd64.tar.gz) | 05a392b9882a33ee84bf3fc31ea54900e6f9ee48dd18a637fcf92bcd3c7e61052edce85da32a9da1cb530b471907ea16d903907de8ed85d7059acb89161d1f87 245 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-server-linux-arm64.tar.gz) | c8e2d743e4ed55ae6a8aded892b8786812bd9d682cc581e5b12d8211c974984b45d663026f06c0faee3aec97cf9aa84b1a19f09efe61b57168c3d7d2c5e1ba51 246 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-server-linux-ppc64le.tar.gz) | bb8f40ff65b879f9465ae76c9e235b963b39b2633af5307074c5008d6f6a27647cb645a27dfc5d970e69acd9c21c3d22d1e3fae8626e42c91edda1ba75b629a1 247 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-server-linux-s390x.tar.gz) | 6688da58a52bf0d6e9b20eceade272e69c3b4880f777da30ff6f6a1857b59bd90da7d5bdfbe4cd76d45768457152d3265375defabcd777a040c98a5f22f9097f 248 249 ### Node Binaries 250 251 filename | sha512 hash 252 -------- | ----------- 253 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-node-linux-amd64.tar.gz) | 44ca6907a7bd8b14db8ab15d6acd6da4e8c3717f94e83f57e4cb1efabef102e1085b25b118a5423420ee56c4e9445ded8b4b5465a151cadcf1b73a57b8aa3691 254 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-node-linux-arm64.tar.gz) | 75606598d442c09fc815da8704c6059c3128e93fb3d95f77884df05187ffe99da2ca3fde5c13cdd6b43eb2d046dcd320e8f7e786000058fbcb21c8fa8350eacd 255 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-node-linux-ppc64le.tar.gz) | 1c1319087a6d79bce42bc2ce9d179457378bc8a602fbd6acfb02daaec22da523bd0a6d5c8570da969ec518f5c3aebd846894a4cdcbd35b1800a95713b55c636d 256 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-node-linux-s390x.tar.gz) | 8a4e06779a0c89f9f9fd9a11d950865466414dbcdfdfc1d5309e6a8fa5dc190fdc5bedf8a04b2ac282c60939a70d1fa04485ed235e39386c6a7fba131ab70c8d 257 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.5/kubernetes-node-windows-amd64.tar.gz) | c193fa370cedbbf775db2e584461b3338bd5260b3822debd04ceac9b32936ebe11b8c46392299da05fc9b4d2bcb98980acb34c192b42aa452cf06369df14184f 258 259 ### Container Images 260 261 All container images are available as manifest lists and support the described 262 architectures. It is also possible to pull a specific architecture directly by 263 adding the "-$ARCH" suffix to the container image name. 264 265 name | architectures 266 ---- | ------------- 267 [registry.k8s.io/conformance:v1.29.5](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x) 268 [registry.k8s.io/kube-apiserver:v1.29.5](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x) 269 [registry.k8s.io/kube-controller-manager:v1.29.5](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x) 270 [registry.k8s.io/kube-proxy:v1.29.5](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x) 271 [registry.k8s.io/kube-scheduler:v1.29.5](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x) 272 [registry.k8s.io/kubectl:v1.29.5](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x) 273 274 ## Changelog since v1.29.4 275 276 ## Changes by Kind 277 278 ### Bug or Regression 279 280 - Fixed a bug that a pod may remain unscheduled if any PreFilter plugin returns nodes that do not exist. ([#124559](https://github.com/kubernetes/kubernetes/pull/124559), [@chengjoey](https://github.com/chengjoey)) [SIG Scheduling and Testing] 281 - Fixes a 1.29.0 regression that introduced a possible data race that could cause panics in kube-controller-manager and kube-scheduler ([#124518](https://github.com/kubernetes/kubernetes/pull/124518), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Scheduling] 282 - Kubeadm: fix a bug when using external CA mode and trying to upgrade to 1.29 using "kubeadm upgrade apply". Show a warning that kubeadm cannot sign the new "super-admin.conf" as the host does not have a CA and show some instructions on how to manually migrate to the separate "admin.conf" and "super-admin.conf" kubeconfig files. ([#124682](https://github.com/kubernetes/kubernetes/pull/124682), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 283 284 ### Other (Cleanup or Flake) 285 286 - For apiserver_storage_size_bytes metric, we are renaming the label for etcd to be "storage_cluster_id" instead of "cluster" to to reduce conflict and be very specific. ([#124294](https://github.com/kubernetes/kubernetes/pull/124294), [@dims](https://github.com/dims)) [SIG API Machinery, Instrumentation and Testing] 287 288 ## Dependencies 289 290 ### Added 291 _Nothing has changed._ 292 293 ### Changed 294 _Nothing has changed._ 295 296 ### Removed 297 _Nothing has changed._ 298 299 300 301 # v1.29.4 302 303 304 ## Downloads for v1.29.4 305 306 307 308 ### Source Code 309 310 filename | sha512 hash 311 -------- | ----------- 312 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes.tar.gz) | 837cc6ab833228e387e787bdb1508d74bbf79c380ac71fed7acaf9e239f3f2fcbe3fcfb9a9e41711620ec21e6cc3a5984148dc80515f37a6fabb02e50a82a29c 313 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-src.tar.gz) | 716c6fc59d8dfed72ed45dfa5535dff3bae3bd3bd9f8641c2068d76c06c21c7ebc8ba0626374312b4a20285277cb8ea4df446199caae9cc0d992346c9dc09479 314 315 ### Client Binaries 316 317 filename | sha512 hash 318 -------- | ----------- 319 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-darwin-amd64.tar.gz) | 01506990cf76344fb12207e3e88a7c38a926ad8ccffc00b0ddcfeff9a5312b01438ef8c813e877e4b856cf1cc3f52dada7cd687a487797168a3436b66c64fc9b 320 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-darwin-arm64.tar.gz) | f77fd94e97f1ecda0715f930d34ed85789f0eb0db6e3aacb35e7ebfbf101efe365169deff98c6b6d3d92b8a18d311032504adfe43e8c7f9fd4a42de2352de389 321 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-linux-386.tar.gz) | c665a345c445878120a05d2c9755a80109423e333bace0423b7208a3fea91c019e3b33c4bcb0761d52a0b5d17258249881b4dd1a5a9a584ae04e4887d3e34b96 322 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-linux-amd64.tar.gz) | c13235bd929eaaf4d0eaaa9ba883e95ce27a402ca7256c634e20a027fbf72db8834de8ea2ca7238e1fe92859e0edc7384a1cec7fbe2b7a5adf07b2e5cf99b04f 323 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-linux-arm.tar.gz) | 9c348edc150340219f4b9b8bfd17e1747df9884f9407126c1585c3a817fec5561d5d02ddaed0317ac515bf6142cc7530fcac9e735f60f92390d05e5517c7d166 324 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-linux-arm64.tar.gz) | 614cd5b5881c583505d089c09c221e4a06da0dc8b5ac70b3d93d7e2a58c8b439446a646d0bb53396c2a48535808503daa6aa1a37f43affe22176c2211fdc2cc4 325 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-linux-ppc64le.tar.gz) | 3c17be398175d0f882a0c1894c05d04fb564a4bd01ac95cbc5dab4902f7827425ff00d3fcee1fda22a31f81888effefff5a9705e266e44aa1a6c8be9ca42f0ca 326 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-linux-s390x.tar.gz) | 0a202fee4e78fffc1a25538529a9751dd7d421f75244cf6739332f606bfbf5ae455519f1f5b4378e7f22b4d8e1104f3eeb1acd37739e213b437db78f429dbc49 327 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-windows-386.tar.gz) | afa38bee4b8d09347a5d4c1b4fde74d337d42efc411d62b336c841b2bc7bf39d6355557a169132ff69ce5d239a01cd59298a061437580c168400399b0a6c71d9 328 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-windows-amd64.tar.gz) | 653c737e582a43dcb7f8475c61bbffbd892b363dbb015d30c7be9413839ffa2564a97ceeea3e273fd0c46025e06e8828a4eedd0ff7983ad848e5647bb03f2249 329 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-client-windows-arm64.tar.gz) | b779f64dac14f3d01b2565f093c27e71c793e0b0b2ff491419730d96ec5782152d7266c68f9b892264752022bb6ef600079649df9ab30e7a329bae2835a43803 330 331 ### Server Binaries 332 333 filename | sha512 hash 334 -------- | ----------- 335 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-server-linux-amd64.tar.gz) | 8554f4e2c828df8e2a3aafceea941712ca079f0710ac1390a1bde32e09fe9a588c217462a2666e0d542c42f1e42881fc975751ce7d5b81c9f88aed0c8302f6dc 336 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-server-linux-arm64.tar.gz) | 850c1233ac2b267964ed840b6f4c4d51a961d8e86b1bf78ae4bc50e31e8255f04fdbd50c91d0937b9ce197dcac30fcc3df5be1b537a6a962b9e984a4754e74bf 337 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-server-linux-ppc64le.tar.gz) | 45e1abd9ee2efd0acac599ff3fe360835f1d02cf1075ba6499d41a5f41bd98e96868fc9d3555c41a535a88a3450f64fb7552d6b9f6a281837451b059c8c3695c 338 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-server-linux-s390x.tar.gz) | 2c924db2f1a6ba83d8364d373d79714d6f7f2697fcefb3f9ca3b89ff46f194ae87b81d3567f2e77bbe9f490e68468c64cf86f0e952a7c49efc87600b92bc9a36 339 340 ### Node Binaries 341 342 filename | sha512 hash 343 -------- | ----------- 344 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-node-linux-amd64.tar.gz) | bf5eb2f4ef8e215941d98ca62bc4901fcad7f68dda153e1f077f9688b1e3f273d3b25bc48167cefd0a1f1ce1e0a3525d71bf7aa37b7fa8b8f639af35df0233bd 345 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-node-linux-arm64.tar.gz) | d983706b1975d3b8c3cb1dae833efc178337df2c2abdd835588a0be6dbaef55c27e1df993e8623b87f56f7f7a8f6de34390e8a085911644fa8af2d49f47512e0 346 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-node-linux-ppc64le.tar.gz) | 2d0358f1c7b6bc8146a7ec386b44e037dc4dd46b17584a276dc473ee57f74d0f5895b001217cef086785fa12482236fcfa96e005bc46c43fd42edd6e332f6b7f 347 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-node-linux-s390x.tar.gz) | be7a9cf871c0255df63e16f21005ecf19ad8702a3b4be6483ed158aa8a85ad5ed8719ddabaf85110a2e05ec777fb7426967926be8a7d3a3ee51b557c181b78b9 348 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.4/kubernetes-node-windows-amd64.tar.gz) | b4a632a37f76b2486e4ded68928edaa159b41d9f428c396710ca67582040439df0f11dfeeb39d4ba3eef02186ffe937eae08cd9d5fdad84bcf43a682d1226d13 349 350 ### Container Images 351 352 All container images are available as manifest lists and support the described 353 architectures. It is also possible to pull a specific architecture directly by 354 adding the "-$ARCH" suffix to the container image name. 355 356 name | architectures 357 ---- | ------------- 358 [registry.k8s.io/conformance:v1.29.4](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x) 359 [registry.k8s.io/kube-apiserver:v1.29.4](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x) 360 [registry.k8s.io/kube-controller-manager:v1.29.4](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x) 361 [registry.k8s.io/kube-proxy:v1.29.4](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x) 362 [registry.k8s.io/kube-scheduler:v1.29.4](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x) 363 [registry.k8s.io/kubectl:v1.29.4](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x) 364 365 ## Changelog since v1.29.3 366 367 ## Important Security Information 368 369 This release contains changes that address the following vulnerabilities: 370 371 ### CVE-2024-3177: Bypassing mountable secrets policy imposed by the ServiceAccount admission plugin 372 373 A security issue was discovered in Kubernetes where users may be able to launch containers that bypass the mountable secrets policy enforced by the ServiceAccount admission plugin when using containers, init containers, and ephemeral containers with the envFrom field populated. 374 375 **Affected Versions**: 376 - kube-apiserver v1.29.0 - v1.29.3 377 - kube-apiserver v1.28.0 - v1.28.8 378 - kube-apiserver <= v1.27.12 379 380 **Fixed Versions**: 381 - kube-apiserver v1.29.4 382 - kube-apiserver v1.28.9 383 - kube-apiserver v1.27.13 384 385 This vulnerability was reported by tha3e1vl. 386 387 388 **CVSS Rating:** Low (2.7) [CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:L/I:N/A:N](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:L/I:N/A:N) 389 390 ## Changes by Kind 391 392 ### Feature 393 394 - Kubernetes is now built with go 1.21.9 395 - update debian-base to bookworm-v1.0.2 ([#124197](https://github.com/kubernetes/kubernetes/pull/124197), [@cpanato](https://github.com/cpanato)) [SIG API Machinery, Architecture, Cloud Provider, Release, Storage and Testing] 396 397 ### Bug or Regression 398 399 - Fix pod restart after node reboot when NewVolumeManagerReconstruction feature gate is enabled and SELinuxMountReadWriteOncePod disabled ([#124140](https://github.com/kubernetes/kubernetes/pull/124140), [@bertinatto](https://github.com/bertinatto)) [SIG Node] 400 - Golang.org/x/net is bumped to v0.23.0 to address CVE-2023-45288 ([#124180](https://github.com/kubernetes/kubernetes/pull/124180), [@MadhavJivrajani](https://github.com/MadhavJivrajani)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 401 - Kube-apiserver: fixes a 1.27+ regression in watch stability by serving watch requests without a resourceVersion from the watch cache by default, as in <1.27 (disabling the change in #115096 by default). This mitigates the impact of an etcd watch bug (https://github.com/etcd-io/etcd/pull/17555). If the 1.27 change in #115096 to serve these requests from underlying storage is still desired despite the impact on watch stability, it can be re-enabled with a `WatchFromStorageWithoutResourceVersion` feature gate. ([#123973](https://github.com/kubernetes/kubernetes/pull/123973), [@serathius](https://github.com/serathius)) [SIG API Machinery] 402 - Kubeadm: fix panic in the command "kubeadm certs check-expiration" when "/etc/kubernetes/pki" exists but cannot be read. ([#124124](https://github.com/kubernetes/kubernetes/pull/124124), [@carlory](https://github.com/carlory)) [SIG Cluster Lifecycle] 403 - NONE ([#124327](https://github.com/kubernetes/kubernetes/pull/124327), [@ritazh](https://github.com/ritazh)) [SIG Auth] 404 - OpenAPI V2 will no longer publish aggregated apiserver OpenAPI for group-versions not matching the APIService specified group version ([#123624](https://github.com/kubernetes/kubernetes/pull/123624), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing] 405 406 ## Dependencies 407 408 ### Added 409 _Nothing has changed._ 410 411 ### Changed 412 - golang.org/x/crypto: v0.16.0 → v0.21.0 413 - golang.org/x/net: v0.19.0 → v0.23.0 414 - golang.org/x/sys: v0.15.0 → v0.18.0 415 - golang.org/x/term: v0.15.0 → v0.18.0 416 417 ### Removed 418 _Nothing has changed._ 419 420 421 422 # v1.29.3 423 424 425 ## Downloads for v1.29.3 426 427 428 429 ### Source Code 430 431 filename | sha512 hash 432 -------- | ----------- 433 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes.tar.gz) | 4b168808633fe4333fa36d749f385df0a35b79b3246cf06c6a9216a8892b26139b2519b57bfaa6b075cf9099fe527526e7918e34fc0d70d3383db456283ca149 434 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-src.tar.gz) | fa2994aaf691d34c60745925686efd5ce85b180a591f0892087af3c831fb0703ac01ae80f0dde3b0b414d57419aa9a3956d26500755674a1a6d6e68d31b4a84f 435 436 ### Client Binaries 437 438 filename | sha512 hash 439 -------- | ----------- 440 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-darwin-amd64.tar.gz) | dbe82475b73cedefeb6cd296c38a49321c311a2b6dd893c2b4f4ffd608ff757629d20280dfa41e87dc8afdef0a733c734a4f583d2e883ced26767ce74e62032c 441 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-darwin-arm64.tar.gz) | 2dfbc1cd16547d8b58c5fdf12c46df6c3b90547ee27ed8d78de7668d4ad4cd6d75d6439a898358701fa41d87c0b21ba2fcb9363196edf327b2ea2956c1fef499 442 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-linux-386.tar.gz) | 8b86ed080cc9008f1dd479b7d69ba7c20ec89f0a4fc85c07dabc6a15102f877d5d4fc0aa50da0f5c177d1660adac3d24771725ac4f0c88788085f0e340927cfe 443 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-linux-amd64.tar.gz) | c9cc7ab9e3aa776f2daab3a9e10ee78d57d0c081ef43f8032de36a61c6425ba527d5df92611b058672be0975a6b97ad3f3a169e282c26275d2c0e59e1f9b1173 444 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-linux-arm.tar.gz) | 2c9c9da97b615f51aef33366a1be5ab3ccfadae8c023f81227d42d7b60df9d7735e403becd1f929178f72f7e568336317c7b76700faeced59fb5266a6982ca62 445 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-linux-arm64.tar.gz) | f66d3734ce43f54d2536f1e5ae20aa85e8602b86aad995573c852d6f03bc50907bd84bdd42d7cb14f246ab27ed0dc840502a540a0bf312b7faf9712053b0f775 446 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-linux-ppc64le.tar.gz) | 080a42f50891ebfcf4ec4588e8af6f5ccbed318825a31840a0219f511f84170eecd73b03460bf8284bdb7796015be2ba793dfd8017c96d41dc1cf41cfd9947ce 447 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-linux-s390x.tar.gz) | 3458ba376bad3bf098b4553007c9438c7bd47dc9b7be9d5e6fc05a81be3c3861781f385829be3e6072ded56f14d509fa6e11862802c7113f48fa26bda532622d 448 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-windows-386.tar.gz) | 225a6b79b442bcbe421317823c98ec3a7fd772ecd2e8758e37eee39e87da318585718dfe62583dfae0c2fcfeccf2290bef29b03029e14735a26fdded3b1febc9 449 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-windows-amd64.tar.gz) | fcfc410f59ff6dc053f7f7786cd486bcd534a419c6db022c0069c22b58d6bbf8ca5c4f58c3823b7bdc4d501dd886e86efeec04fb3469f938b0c8fe0fa545aece 450 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-client-windows-arm64.tar.gz) | 70100f18c229ecb675dfb459ca64b0545ff03d83c25a04d46e3a445d990649d80e4fb273bd04ef76f81ad06c412d8f2ade53d920b11d64fdfc7d0ab616b53649 451 452 ### Server Binaries 453 454 filename | sha512 hash 455 -------- | ----------- 456 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-server-linux-amd64.tar.gz) | 91349b3d1e2d769b4d727531a4c33a4a596dcadeaee617e339f543ca24892acf3930a8e45a810c012ece4fcd4d376700fd3810aef6cb40955a8c7b996789f514 457 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-server-linux-arm64.tar.gz) | 659e03e65bec126fac64ddc277d906afeefce6024a80fb6642990aaf588e54e6cd83edfa0f6e30a4570ded1cbc2c58be8f854ef8cf5a40a6e2d28e93288caf6f 458 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-server-linux-ppc64le.tar.gz) | f735fcdf243599ae8a70069cf3285577683f7a3f1890681da53c88cf50f973107695213401910c9849d6065a02934b8dc20704566edb2d85f019db099c6d0d04 459 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-server-linux-s390x.tar.gz) | 4888ccc80f171512c4ffb1cb3e61d5255c63aa331a5418830a28d7ad9360960e880666b9281a19e54c767636bc5c4eb66c253edab82fe8a3259df54fe885a1e2 460 461 ### Node Binaries 462 463 filename | sha512 hash 464 -------- | ----------- 465 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-node-linux-amd64.tar.gz) | 704a3039a9b203dee4f74b219162d8295dd85e16ed385fb8106a14e75f768623adf2da3083991625709ac403bd5785ea683f1451dbb58637b8b53be85627b12f 466 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-node-linux-arm64.tar.gz) | 794fcc502fe98b0196e06db1cd66f0700fe650d7ef423b85b4ecd60bf0397ab754ec3aebab79eb1b8dd42dc068a8522a9cc0a851bea977bd04801451ea653a25 467 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-node-linux-ppc64le.tar.gz) | 01f8bde765edec9bbaba7602570fbf1c2cb401185e2f61e855b1a4cfa1cdb3a91c6bf91111fccb9c3c3c425fa5ccd3d6edbf5ef9e62b643f7abca7b9be7d4539 468 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-node-linux-s390x.tar.gz) | 48573b3da001cb96113e8cd1fbfdaf017f8d00485848dfc7ff09b123ba26012ba0ac7ae27494619c1815dee6e621d1d55a86331ddd011baca6612df42f2d1fab 469 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.3/kubernetes-node-windows-amd64.tar.gz) | a1f66a9eac3b6af9138559a1630559ffedec93b2777d83886bcda36679fb453a26bd6e0fcac0ce9db89ff5dfb5cc4841fc927ed14dbd6d00b1fb12041a1b1d8c 470 471 ### Container Images 472 473 All container images are available as manifest lists and support the described 474 architectures. It is also possible to pull a specific architecture directly by 475 adding the "-$ARCH" suffix to the container image name. 476 477 name | architectures 478 ---- | ------------- 479 [registry.k8s.io/conformance:v1.29.3](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x) 480 [registry.k8s.io/kube-apiserver:v1.29.3](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x) 481 [registry.k8s.io/kube-controller-manager:v1.29.3](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x) 482 [registry.k8s.io/kube-proxy:v1.29.3](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x) 483 [registry.k8s.io/kube-scheduler:v1.29.3](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x) 484 [registry.k8s.io/kubectl:v1.29.3](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x) 485 486 ## Changelog since v1.29.2 487 488 ## Changes by Kind 489 490 ### Feature 491 492 - Kubernetes is now built with go 1.21.8 493 - update distroless-iptables to v0.4.6 ([#123773](https://github.com/kubernetes/kubernetes/pull/123773), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 494 495 ### Bug or Regression 496 497 - Fix error when trying to expand a volume that does not require node expansion ([#123055](https://github.com/kubernetes/kubernetes/pull/123055), [@gnufied](https://github.com/gnufied)) [SIG Node and Storage] 498 - Fixed a bug that an init container with containerRestartPolicy with `Always` cannot update its state from terminated to non-terminated for the pod with restartPolicy with `Never` or `OnFailure`. ([#123709](https://github.com/kubernetes/kubernetes/pull/123709), [@gjkim42](https://github.com/gjkim42)) [SIG Apps] 499 - Fixed cleanup of Pod volume mounts when a file was used as a subpath. ([#123052](https://github.com/kubernetes/kubernetes/pull/123052), [@jsafrane](https://github.com/jsafrane)) [SIG Node] 500 - Fixed the disruption controller's PDB status synchronization to maintain all PDB conditions during an update. ([#122056](https://github.com/kubernetes/kubernetes/pull/122056), [@dhenkel92](https://github.com/dhenkel92)) [SIG Apps] 501 - Fixes an issue calculating total CPU usage reported for Windows nodes ([#122999](https://github.com/kubernetes/kubernetes/pull/122999), [@marosset](https://github.com/marosset)) [SIG Node and Windows] 502 - Prevent watch cache starvation by moving its watch to separate RPC and add a SeparateCacheWatchRPC feature flag to disable this behavior ([#123693](https://github.com/kubernetes/kubernetes/pull/123693), [@mengqiy](https://github.com/mengqiy)) [SIG API Machinery] 503 - Restore --verify-only function in code generation wrappers. ([#123261](https://github.com/kubernetes/kubernetes/pull/123261), [@skitt](https://github.com/skitt)) [SIG API Machinery] 504 - Updates google.golang.org/protobuf to v1.33.0 to resolve CVE-2024-24786 ([#123763](https://github.com/kubernetes/kubernetes/pull/123763), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 505 506 ### Other (Cleanup or Flake) 507 508 - Etcd: Update to version 3.5.12 ([#123188](https://github.com/kubernetes/kubernetes/pull/123188), [@bzsuni](https://github.com/bzsuni)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing] 509 510 ## Dependencies 511 512 ### Added 513 _Nothing has changed._ 514 515 ### Changed 516 - github.com/golang/protobuf: [v1.5.3 → v1.5.4](https://github.com/golang/protobuf/compare/v1.5.3...v1.5.4) 517 - google.golang.org/protobuf: v1.31.0 → v1.33.0 518 519 ### Removed 520 _Nothing has changed._ 521 522 523 524 # v1.29.2 525 526 527 ## Downloads for v1.29.2 528 529 530 531 ### Source Code 532 533 filename | sha512 hash 534 -------- | ----------- 535 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes.tar.gz) | 5eec915a8064dc3323cec80db18d213bf6e8c7e805b01a071681e0d4c0af4785d6b8b63c8cd77077dac14347ea7410384ac4bd866db6f83e26808482fe357583 536 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-src.tar.gz) | 3ced73b388123240a2228506ffb678aea0983b2d152978255ce7e0488c5776749319170187365e3998fb9d92619abe46256036d02824fc75e59e246e27ba85de 537 538 ### Client Binaries 539 540 filename | sha512 hash 541 -------- | ----------- 542 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-darwin-amd64.tar.gz) | 3d7e9b86c93d4bdd1a32c30967fca3fa6fb932aab7582d8a5c7389b9f7a4646e4b39a7c478ae360d2ccdd3b52932b3c7f7a0094557693b046b141c556599e154 543 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-darwin-arm64.tar.gz) | a64501130b2521003f130c11d2022996795157969b234b0558bc014fb6416ec1e78e777c7c3cbb05eb65481aeeec1ed35646ed8245513af0ed3a2c88b901aa3c 544 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-linux-386.tar.gz) | bbb8c6649600866bbfbce27d9068bbed3542639ee403d86d3ecd54a69e3925ed56a0bb37c1a772a30f3057f817d89eb46768a5c72c64ef3a5e101f2465a2109f 545 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-linux-amd64.tar.gz) | 1526534f31ece4247a61eb794ed31fe2289e14387424c08cfe63770e7479df477e69836b068154f67beadbd0e039c540c73912becb03394d43302f839cad4dad 546 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-linux-arm.tar.gz) | f9fd808955cd98c29f357c439c536e980bd4a1cf04da33e0d305343fbbba2ce905b4de3e0359a0f4e01202c439e8ce16a44b49594cb5c92fda6caf506a1a1948 547 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-linux-arm64.tar.gz) | c5fe23c425e151307461bc807a64b539cb3b61b9688122bbc79a428d099a907ea11a4313b2f5eae4b70a190c6f296daffdcf199c81793a747d200d67d182c9ab 548 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-linux-ppc64le.tar.gz) | c923fd2e50f7eca933580d572d709b7330fd1d8d8ab325c45f0054ece390416682d6dd7c335a991d5f0b2a01b084e521dbab94cdb20893b29561314d0be91778 549 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-linux-s390x.tar.gz) | be407bd63812a2fef7d49f8a92a93d1ece357a455c12a96bc5ec4880a5210fe54e726d52e004395c240967cc25a8ce5a1b91071bb437296793b2db2c38eb4753 550 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-windows-386.tar.gz) | 32be8d5d05ca1ca789e71afb0671d33e232f5fc328f8dce507bde365395d589182e18bccc3b58c81c2538104e890c1c411a7b53762fb8547e2a125c9af7ad3cc 551 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-windows-amd64.tar.gz) | 002c0a681e3f766504b3479706978e028929287df7f3ea9af8a2294efecea61366f28c0524aee07261ba6d462b8c71635d4a3ec9dbb9c3a02e7fa64128dd5a3c 552 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-client-windows-arm64.tar.gz) | e74b551a1b94845bfe0c4f83463d97d9c0a523f18577d6be29a54d888df3618ea446e9801fafad07cb0f4c07fdc79559e95c4f0beb8863aae267986a7c23f143 553 554 ### Server Binaries 555 556 filename | sha512 hash 557 -------- | ----------- 558 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-server-linux-amd64.tar.gz) | d5575da7f28a5284d4ffb40ca1b597213e03c381e161c1ec2bdadd7fe0532d62f41c758443ecefed70f484fb770e0bac53218f0a429587ac983469a39e56979b 559 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-server-linux-arm64.tar.gz) | ed2bb79cc50679f2238d6247b2119b18764274160fa6f555ba7a112dc3f13c9870f1aea7eca3a572e81ca562f14d001f03307db785debe0b96ea026bc37ad456 560 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-server-linux-ppc64le.tar.gz) | 406687513b2d1b15cfbc87812918bc9f72e4899497d4f03d95f8cb9c510822591fd9c19f333e5eb14b2063be6a93f1e1ca99b2d371bd835f178dcf25f999e881 561 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-server-linux-s390x.tar.gz) | 2b7002771db07dfe220d4f9fec089e2a43a9216f9c608fea67a1f1fafa8ea828ebc019d8e89a82f8cae370d821d762252f02547e521451d2271885b70a542f77 562 563 ### Node Binaries 564 565 filename | sha512 hash 566 -------- | ----------- 567 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-node-linux-amd64.tar.gz) | 7b4ae81efc8a8bf0ece659c83dc6de0d4730b97d917e6b4c9f15fba9155422ee847bbd4464e241b9689afcda80ff2753d73e9663de167947f81084fac15b6e66 568 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-node-linux-arm64.tar.gz) | 29cde35274d9514eb5a8a252a9af57ac2f8a99f90341f707a061afea7478b83d541e1d1e2edf42efdeea9d0fe529d10b49129895c1c4fd201cf59aa1e90538d1 569 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-node-linux-ppc64le.tar.gz) | 5bdf77b98e171807c3ae05041a15b6574e09a6734e571d67c8c3b2e131b6722eb29879b82f2c95be7f6246d154bb6019d4f60f522f5b2f2e255127e000fc8aab 570 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-node-linux-s390x.tar.gz) | c45393ec1bc1853bd37374b67dde574b889ca849be662e2820f3c7d0b0ee962f317cc5d616827ae5b7215bdd38b08f2761cdc00f9c15182d33df085c726a66e7 571 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.2/kubernetes-node-windows-amd64.tar.gz) | 42b380ba122fc525146fa55032edeb6e6be74dd58f48ab4a8715bb66c1e031924f599f2e47cd30c36901ae2df591d736e46ff5fded43e533c71ced3f4bdd2230 572 573 ### Container Images 574 575 All container images are available as manifest lists and support the described 576 architectures. It is also possible to pull a specific architecture directly by 577 adding the "-$ARCH" suffix to the container image name. 578 579 name | architectures 580 ---- | ------------- 581 [registry.k8s.io/conformance:v1.29.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 582 [registry.k8s.io/kube-apiserver:v1.29.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 583 [registry.k8s.io/kube-controller-manager:v1.29.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 584 [registry.k8s.io/kube-proxy:v1.29.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 585 [registry.k8s.io/kube-scheduler:v1.29.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 586 [registry.k8s.io/kubectl:v1.29.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 587 588 ## Changelog since v1.29.1 589 590 ## Changes by Kind 591 592 ### Feature 593 594 - Add process_start_time_seconds to /metrics/slis endpoint of all components ([#122750](https://github.com/kubernetes/kubernetes/pull/122750), [@richabanker](https://github.com/richabanker)) [SIG Architecture, Instrumentation and Testing] 595 - Kubernetes is now built with go 1.21.7 596 - update setcap/debian-base to bookworm-v1.0.1 597 - update distroless-iptables to v0.4.5 ([#123218](https://github.com/kubernetes/kubernetes/pull/123218), [@cpanato](https://github.com/cpanato)) [SIG API Machinery, Architecture, Cloud Provider, Release, Storage and Testing] 598 599 ### Bug or Regression 600 601 - Fix deprecated version for pod_scheduling_duration_seconds that caused the metric to be hidden by default in 1.29. ([#123042](https://github.com/kubernetes/kubernetes/pull/123042), [@alculquicondor](https://github.com/alculquicondor)) [SIG Instrumentation and Scheduling] 602 - Fixed a bug in ValidatingAdmissionPolicy that caused policies which were using CRD parameters to fail to synchronize ([#123080](https://github.com/kubernetes/kubernetes/pull/123080), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing] 603 - Fixes a 1.29 regression in "kubeadm init" that caused a user-specified --kubeconfig file to be ignored. ([#122792](https://github.com/kubernetes/kubernetes/pull/122792), [@avorima](https://github.com/avorima)) [SIG Cluster Lifecycle] 604 - Fixes a race condition in the iptables mode of kube-proxy in 1.27 and later 605 that could result in some updates getting lost (e.g., when a service gets a 606 new endpoint, the rules for the new endpoint might not be added until 607 much later). ([#122756](https://github.com/kubernetes/kubernetes/pull/122756), [@hakman](https://github.com/hakman)) [SIG Network] 608 - If a pvc has an empty storageClassName, persistentvolume controller won't try to assign a default StorageClass ([#122704](https://github.com/kubernetes/kubernetes/pull/122704), [@carlory](https://github.com/carlory)) [SIG Apps and Storage] 609 - Kubeadm: do not upload kubelet patch configuration into `kube-system/kubelet-config` ConfigMap ([#123108](https://github.com/kubernetes/kubernetes/pull/123108), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 610 - Kubeadm: fix a bug where the --rootfs global flag does not work with "kubeadm upgrade node" for control plane nodes. ([#123096](https://github.com/kubernetes/kubernetes/pull/123096), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 611 612 ## Dependencies 613 614 ### Added 615 _Nothing has changed._ 616 617 ### Changed 618 _Nothing has changed._ 619 620 ### Removed 621 _Nothing has changed._ 622 623 624 625 # v1.29.1 626 627 628 ## Downloads for v1.29.1 629 630 631 632 ### Source Code 633 634 filename | sha512 hash 635 -------- | ----------- 636 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes.tar.gz) | e622256168b70f1c5968ba39673e8b403c898fb13a1710f4a674f666fccc1cf9414a31518810b4e7fbfad20a12264a37afafe8534ab6eb452cce588d2b92ceb0 637 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-src.tar.gz) | f229ea55c8afa9a165b245081738adfbcfa5ed41be2be9b2ed76ed0a789378614417e02109eb5717f7973a4b21a649bf18a99fbad16e05f9496e0b03ac785576 638 639 ### Client Binaries 640 641 filename | sha512 hash 642 -------- | ----------- 643 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-darwin-amd64.tar.gz) | 0e2b49ed96d24511a315453ee7325787217f8f4f403a8dd18a4d0642e5941ca9865f48b7293b39dade110de2e05f595f026e00cd1dfa639545590a60381c8980 644 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-darwin-arm64.tar.gz) | cabc9436ac37537c2ec1adf3e28df6461eaa98933cc20cb9862e8c7c72a1ea2201e5a7373d6f9242b381eadfbdf335455466b1c68a013c511486fc7c88e0782b 645 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-linux-386.tar.gz) | c25b56a20cdf64a9f250c8419d393f5808b78b5a5f3f9979cabc098f2874538f48231852aaa4bc1f88bf0abae9ef9b383c2c1a86c745da6ce173fe148a392bcf 646 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-linux-amd64.tar.gz) | 9142b014be0c14bd9965be6b55aa24db24c4bb82a4ae01ea39752998cca7b0e0c29403853bb461c2accf57177943803a314d278bb64e4e6896840742f3d347f1 647 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-linux-arm.tar.gz) | e1e43e744c263dc4cc89dc2347a3d9a8bffae3727a7d4d7386a4800164fc0b6acaed460fcd820832a4d37ad9051ec185498bd1c37ca1e440253853b62dcc85c3 648 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-linux-arm64.tar.gz) | 29a7770797f749941cd7147cd6ed5b6f836afbaea2d9f63351fb86b992af13867a08a4cf3ba8ff539b6db5c069ea4f0274a2bc85fd5301fca821817e404496dc 649 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-linux-ppc64le.tar.gz) | 96fa7e1b88b5053600ee885f3b7afab64ac72bc73332c4162bc770549b66b05ae26235c98ed11aeb6ff6e72b64d995b8aeeae725d3916322dc19c6612277bdb7 650 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-linux-s390x.tar.gz) | 5f942fd006d26cfc5350f51f97300e81f63e8c058230c91ec23479f57356d05cd51fc4466fa9a10d9b79b7fa12549a7aa7a87f6e025ecd0fc62b7744e5574303 651 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-windows-386.tar.gz) | c099b667667f50ab7b2639e8d19d655ced0caea77c4ad95b952fb604d453033218e2a2e44ee3c0e60845f3ebabd38dc3c0616761cf3382e6d9d8906614469e0c 652 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-windows-amd64.tar.gz) | e24da5de0c6c7d0865ad434270d019e5c5490078d86f490481b6f34069ddb7892bff2df1e13f6333569d3529859ac0e88553a652397b235b25ff426bfdbe4626 653 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-client-windows-arm64.tar.gz) | 4566eb1946642ded9e2ce3a79085947e8b712f534354ef1f24d80b6c48b9202c331e8d9260895f2f332ff1970bb634cbc30428df1089ff4f3dea52b4e8d211a8 654 655 ### Server Binaries 656 657 filename | sha512 hash 658 -------- | ----------- 659 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-server-linux-amd64.tar.gz) | b01f8ef7160045363b017e79f1b3c2c5b54a67aa3426c0e5c45562be88bd72a3cb5e72043764684e10e255ecbb19d697765755031aa3c1b49c4e4c12122b3077 660 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-server-linux-arm64.tar.gz) | 48b78293820ab61f5723d590787c888df51c134539cc224f1bcc2d63e78e9fe2f39e9f23192f47aa31456802d5981cd86fa56522ebfb7d63a5914d54c7682684 661 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-server-linux-ppc64le.tar.gz) | 93e8608acef77428dfecb38546b07bb2a9e9da998e674fc106e687f10387263135d081da10bbafc5dbd91f467620f999ab154975d20dc26088b1d9f2559a9582 662 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-server-linux-s390x.tar.gz) | 20458da5a5dd67e89a768037f408a440a26173afd2844696c658b768fd93a7fa4e39167d93bd8d804ccb562f13ef43b4cba65d0edbe55479d3335c5f68e1f12a 663 664 ### Node Binaries 665 666 filename | sha512 hash 667 -------- | ----------- 668 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-node-linux-amd64.tar.gz) | 89adcf2782d42b43bea9841366051c88267b30f32c32d4570fd6f2a0fe4d2fae7c79fa704be2135c60129d9f1fa7d7cf17fc1e0401b7a489175a872d6bdacc35 669 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-node-linux-arm64.tar.gz) | 15c542d2588981b13c98be42030b64e4e430e3879dd440184d5b50c3764272c8337f2416effee2697a1af2ab1617545b7375665db6cf05e6cb3a7f6e0a12f354 670 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-node-linux-ppc64le.tar.gz) | e57203fb80f7c433242b0000c7610daff2583d5c3e7b787aac2c2c15e7118cb77482ecf2b4586bd0105a7a8b875d84397e229848e41196be546b162c7332fb8d 671 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-node-linux-s390x.tar.gz) | 6eaa7087be6a0258e9bf03196667c11cccf447433b92f8239e035d4ee66d9cffbca1dc6ca9e35888d892b936148aa6972afea158eac29900fca3a1aecdcbabc1 672 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.1/kubernetes-node-windows-amd64.tar.gz) | ee54d0fc79b70cdb31d3a557cc170c61fa621956efa52c32c5d915dfa271691b4db60dce7e6efc84156dafab218a75feb76f0e8fd3aede2e80606ef6986d0348 673 674 ### Container Images 675 676 All container images are available as manifest lists and support the described 677 architectures. It is also possible to pull a specific architecture directly by 678 adding the "-$ARCH" suffix to the container image name. 679 680 name | architectures 681 ---- | ------------- 682 [registry.k8s.io/conformance:v1.29.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 683 [registry.k8s.io/kube-apiserver:v1.29.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 684 [registry.k8s.io/kube-controller-manager:v1.29.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 685 [registry.k8s.io/kube-proxy:v1.29.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 686 [registry.k8s.io/kube-scheduler:v1.29.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 687 [registry.k8s.io/kubectl:v1.29.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 688 689 ## Changelog since v1.29.0 690 691 ## Changes by Kind 692 693 ### API Change 694 695 - Fixes accidental enablement of the new alpha `optionalOldSelf` API field in CustomResourceDefinition validation rules, which should only be allowed to be set when the CRDValidationRatcheting feature gate is enabled. Existing CustomResourceDefinition objects which have the field set will retain it on update, but new CustomResourceDefinition objects will not be permitted to set the field while the CRDValidationRatcheting feature gate is disabled. ([#122343](https://github.com/kubernetes/kubernetes/pull/122343), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery] 696 697 ### Feature 698 699 - Kubernetes is now built with Go 1.21.6 ([#122711](https://github.com/kubernetes/kubernetes/pull/122711), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 700 701 ### Bug or Regression 702 703 - Allow deletion of pods that use raw block volumes on node reboot ([#122211](https://github.com/kubernetes/kubernetes/pull/122211), [@gnufied](https://github.com/gnufied)) [SIG Node and Storage] 704 - Fix an issue where kubectl apply could panic when imported as a library ([#122559](https://github.com/kubernetes/kubernetes/pull/122559), [@Jefftree](https://github.com/Jefftree)) [SIG CLI] 705 - Fix: Mount point may become local without calling NodePublishVolume after node rebooting. ([#119923](https://github.com/kubernetes/kubernetes/pull/119923), [@cvvz](https://github.com/cvvz)) [SIG Node and Storage] 706 - Fixed a regression since 1.24 in the scheduling framework when overriding MultiPoint plugins (e.g. default plugins). 707 The incorrect loop logic might lead to a plugin being loaded multiple times, consequently preventing any Pod from being scheduled, which is unexpected. ([#122366](https://github.com/kubernetes/kubernetes/pull/122366), [@caohe](https://github.com/caohe)) [SIG Scheduling] 708 - Fixed migration of in-tree vSphere volumes to the CSI driver. ([#122341](https://github.com/kubernetes/kubernetes/pull/122341), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] 709 - QueueingHint implementation for NodeAffinity is reverted because we found potential scenarios where events that make Pods schedulable could be missed. ([#122327](https://github.com/kubernetes/kubernetes/pull/122327), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 710 - QueueingHint implementation for NodeUnschedulable is reverted because we found potential scenarios where events that make Pods schedulable could be missed. ([#122326](https://github.com/kubernetes/kubernetes/pull/122326), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 711 712 ### Other (Cleanup or Flake) 713 714 - Reverts the EventedPLEG feature (beta, but disabled by default) back to alpha for a known issue ([#122718](https://github.com/kubernetes/kubernetes/pull/122718), [@pacoxu](https://github.com/pacoxu)) [SIG Node] 715 716 ## Dependencies 717 718 ### Added 719 _Nothing has changed._ 720 721 ### Changed 722 - golang.org/x/crypto: v0.14.0 → v0.16.0 723 - golang.org/x/mod: v0.12.0 → v0.14.0 724 - golang.org/x/net: v0.17.0 → v0.19.0 725 - golang.org/x/sync: v0.3.0 → v0.5.0 726 - golang.org/x/sys: v0.13.0 → v0.15.0 727 - golang.org/x/term: v0.13.0 → v0.15.0 728 - golang.org/x/text: v0.13.0 → v0.14.0 729 - golang.org/x/tools: v0.12.0 → v0.16.1 730 731 ### Removed 732 _Nothing has changed._ 733 734 735 736 # v1.29.0 737 738 [Documentation](https://docs.k8s.io) 739 740 ## Downloads for v1.29.0 741 742 ### Source Code 743 744 filename | sha512 hash 745 -------- | ----------- 746 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes.tar.gz) | `f07879916d7c4c7f8059ff9fd3c0006ce9bceb540874e183268a2bf2936df2632c4a3878a613cf2d695a80796e6c3eb52de5e3d83a73c91cb9a0bb5627091bae` 747 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-src.tar.gz) | `a37a7927224785625e9863c1e2dcbc88943593d003b8d126fee63770e6b8eff122004d0f80e1301de34e8a2d6ce208ec6fa55cad3bbe8631b92e5469f45bd00d` 748 749 ### Client Binaries 750 751 filename | sha512 hash 752 -------- | ----------- 753 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-darwin-amd64.tar.gz) | `22da1d2a217a8de91c1a8c393d17eb5ca81e243a1a3e509f3a40fb91d623670ace4ee87a09218a184aaa2eec4ca9c5478b992b8c6f136c568767d6e9dea493bf` 754 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-darwin-arm64.tar.gz) | `cbc0cafecae18a50f98aaa8b508b1808a50b7a477638dc8699830a9dae7ffa83641f9fdb9f53616b32ebc8df84835fc847ea252c5ebe647c7d3462029a63b7a0` 755 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-linux-386.tar.gz) | `f7ace756a3b6c56f2620d0ea6236fb94328c0a928094e4be7fbb78990a5771e8628bd93eac34017f3c33505c0248e8a64f933724a5fec6b322cf54dc30901985` 756 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-linux-amd64.tar.gz) | `6ff15bed6030c47e2ce90723500f08fa9968413f5b858456d4395bc67ab529b0b523ad0521e03be37664965e2fa588680aa0a5180054bc5cb3bafeef1497029b` 757 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-linux-arm.tar.gz) | `bafe1ca945c41ae671029d5398e564bac0753400ee3a50dc0b4979284c0a905e8c77575d8b64b303e9c776d09c919d27f1f99847390d4e2e1c43be826a8dc1a4` 758 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-linux-arm64.tar.gz) | `f3bca520625eaf6e6dd9af4cc709ff20bfce4da298a03e0be8835013a95fe0d6a25693d7702a4739c9477f9d49d2492d739718245ff91716fff90f60279ff376` 759 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-linux-ppc64le.tar.gz) | `e6ea574272cefe9fd6e8eea2bddd89e1d67d0cb560089813e7429f3fb6d98be0c6601f33c8a0b2364d3becfb93c0904c171096ed6cafc4071e08851566d70d82` 760 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-linux-s390x.tar.gz) | `b67dd572d84382e3f713d56bfb371de379807dca52cc4a1e082d6f4720a12770354ef2c9eac93bfc73bc0ea5f4be293db3b6c03328b94a797c2da17b9c40d9f3` 761 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-windows-386.tar.gz) | `0cf4b665f46e36616452916d744367b0ae2238098705b32de79559d06ea551173ab95190a26e87bebc03e67a75dc6a65699be3ef3db12aef82f32b66fd5afb0e` 762 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-windows-amd64.tar.gz) | `69cbe2b3942ba7d9c66e99f819adca94a9c7b420ad72cfd74407954c23ad70a4e7e76296824c4899f88232cabffe08d364c96af83bdaa538f29fa1303bcda2fa` 763 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-client-windows-arm64.tar.gz) | `44b0d1a7904bc2bf754abecb9b43a9efdc7cf700ab18f2564d95d98b4e38fe6d91f066943db7105baea964f86d77ade3b1acd57c7aaf1cdf689660f0d4422960` 764 765 ### Server Binaries 766 767 filename | sha512 hash 768 -------- | ----------- 769 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-server-linux-amd64.tar.gz) | `651a8bf34acb6d61c39cc67ae23d9ef18204f95b309561d31f49da26c0c6a1b7585e7d7c2ac2f1522b2c326470a4e1ec9aa0dcf3bb1f66e1a41e6a2286e0aa5f` 770 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-server-linux-arm64.tar.gz) | `7f1f58b05c923d860f2daa6d31906faf834584b1560f4eda01ba5499338d07a7f183030ab625557b1f5df50a5f0ea30d97d487e2571c85260e5b88fc3519cd43` 771 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-server-linux-ppc64le.tar.gz) | `3ca2af4a7d68c0d84ef65e69190daeb2392946c87c6b8e84ff8d5cf917c979f0778fc00040d4b471e71b8474ca57ac8fdf786f006260d4403b53f59a203a48f1` 772 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-server-linux-s390x.tar.gz) | `dfa172456f98210e614a9a538b9027ba211cc19f6eec22a42d5e89ce12d7f5e7e58dfd3229bb974ecba31ffafdf1a5361aef18b9610a45614a181918d87500db` 773 774 ### Node Binaries 775 776 filename | sha512 hash 777 -------- | ----------- 778 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-node-linux-amd64.tar.gz) | `8057197e9354e2e0f48aab18c0ce87e4ea39c1682cfd4c491c2bc83f8881787b09cb0c9b9f4d7bef8fbe53cc4056f5381745dbfde7f7474bb76a2358b8b3953e` 779 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-node-linux-arm64.tar.gz) | `70d086c71f6258b1667bcb1efe60c15810b5b76848fdf26781c5a90efb8a78030e9ffb230bb0fd52d994f02b13c0b558c8e8ad3a42b601a0f9440a71cf91be2d` 780 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-node-linux-ppc64le.tar.gz) | `2740f6ac0dfeebbe4ba8804b43ec5968997d9137de9a9432861c3e71e614cb84b309da31bde3554f896f829a570c21b833f0af241659ad326fa753a80f185ec4` 781 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-node-linux-s390x.tar.gz) | `9877d5a6cc84569efe30256ba5e8095f38bfa0b11c28892499a12b577b467b516880a33022d88f65263c7ffa2a9a3687ef52cb85fa611e95b14ae0c5b7a79c5c` 782 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0/kubernetes-node-windows-amd64.tar.gz) | `66b264de5e810bff31c4cf7cc575c3c57fed491fa4e21de7035dad76127e17d5fc88aff9f65277adf0826b255bf9b983f61c91bff2f8386d950f87509db6ec6b` 783 784 ### Container Images 785 786 All container images are available as manifest lists and support the described 787 architectures. It is also possible to pull a specific architecture directly by 788 adding the "-$ARCH" suffix to the container image name. 789 name | architectures 790 ---- | ------------- 791 [registry.k8s.io/conformance:v1.29.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 792 [registry.k8s.io/kube-apiserver:v1.29.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 793 [registry.k8s.io/kube-controller-manager:v1.29.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 794 [registry.k8s.io/kube-proxy:v1.29.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 795 [registry.k8s.io/kube-scheduler:v1.29.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 796 [registry.k8s.io/kubectl:v1.29.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 797 798 ## Changelog since v1.28.0 799 800 ## Urgent Upgrade Notes 801 802 ### (No, really, you MUST read this before you upgrade) 803 804 - Stopped accepting component configuration for `kube-proxy` and `kubelet` during `kubeadm upgrade plan --config`. This was a legacy behavior that was not well supported for upgrades and could be used only at the plan stage to determine if the configuration for these components stored in the cluster needs manual version migration. In the future, `kubeadm` will attempt alternative component config migration approaches. ([#120788](https://github.com/kubernetes/kubernetes/pull/120788), [@chendave](https://github.com/chendave)) 805 - `kubeadm`: a separate "super-admin.conf" file is now deployed. The User in `admin.conf` is now bound to a new RBAC Group `kubeadm:cluster-admins` that has `cluster-admin` `ClusterRole` access. The User in `super-admin.conf` is now bound to the `system:masters` built-in super-powers / break-glass Group that can bypass RBAC. Before this change, the default `admin.conf` was bound to `system:masters` Group, which was undesired. Executing `kubeadm init phase kubeconfig all` or just `kubeadm init` will now generate the new `super-admin.conf` file. The cluster admin can then decide to keep the file present on a node host or move it to a safe location. `kubadm certs renew` will renew the certificate in `super-admin.conf` to one year if the file exists; if it does not exist a "MISSING" note will be printed. `kubeadm upgrade apply` for this release will migrate this particular node to the two file setup. Subsequent kubeadm releases will continue to optionally renew the certificate in `super-admin.conf` if the file exists on disk and if renew on upgrade is not disabled. `kubeadm join --control-plane` will now generate only an `admin.conf` file that has the less privileged User. ([#121305](https://github.com/kubernetes/kubernetes/pull/121305), [@neolit123](https://github.com/neolit123)) 806 807 ## Changes by Kind 808 809 ### Deprecation 810 811 - #### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: 812 813 <!-- 814 This section can be blank if this pull request does not require a release note. 815 816 When adding links which point to resources within git repositories, like 817 KEPs or supporting documentation, please reference a specific commit and avoid 818 linking directly to the master branch. This ensures that links reference a 819 specific point in time, rather than a document that may change over time. 820 821 See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files 822 823 Please use the following format for linking documentation: 824 - [KEP]: <link> 825 - [Usage]: <link> 826 - [Other doc]: <link> 827 --> ([#119495](https://github.com/kubernetes/kubernetes/pull/119495), [@bzsuni](https://github.com/bzsuni)) [SIG API Machinery] 828 - Creation of new `CronJob` objects containing `TZ` or `CRON_TZ` in `.spec.schedule`, accidentally enabled in `v1.22`, is now disallowed. Use the `.spec.timeZone` field instead, supported in `v1.25+` clusters in default configurations. See https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#unsupported-timezone-specification for more information. ([#116252](https://github.com/kubernetes/kubernetes/pull/116252), [@soltysh](https://github.com/soltysh)) 829 - Removed the networking `alpha` API `ClusterCIDR`. ([#121229](https://github.com/kubernetes/kubernetes/pull/121229), [@aojea](https://github.com/aojea)) 830 831 ### API Change 832 833 - '`kube-apiserver`: adds `--authentication-config` flag for reading `AuthenticationConfiguration` 834 files. `--authentication-config` flag is mutually exclusive with the existing `--oidc-*` 835 flags.' ([#119142](https://github.com/kubernetes/kubernetes/pull/119142), [@aramase](https://github.com/aramase)) 836 - '`kube-scheduler` component config (`KubeSchedulerConfiguration`) `kubescheduler.config.k8s.io/v1beta3` 837 is removed in `v1.29`. Migrated `kube-scheduler` configuration files to `kubescheduler.config.k8s.io/v1`.' ([#119994](https://github.com/kubernetes/kubernetes/pull/119994), [@SataQiu](https://github.com/SataQiu)) 838 - A new sleep action for the `PreStop` lifecycle hook was added, allowing containers to pause for a specified duration before termination. ([#119026](https://github.com/kubernetes/kubernetes/pull/119026), [@AxeZhan](https://github.com/AxeZhan)) 839 - Added CEL expressions to `v1alpha1 AuthenticationConfiguration`. ([#121078](https://github.com/kubernetes/kubernetes/pull/121078), [@aramase](https://github.com/aramase)) 840 - Added Windows support for InPlace Pod Vertical Scaling feature. ([#112599](https://github.com/kubernetes/kubernetes/pull/112599), [@fabi200123](https://github.com/fabi200123)) [SIG Autoscaling, Node, Scalability, Scheduling and Windows] 841 - Added `ImageMaximumGCAge` field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. ([#121275](https://github.com/kubernetes/kubernetes/pull/121275), [@haircommander](https://github.com/haircommander)) 842 - Added `UserNamespacesPodSecurityStandards` feature gate to enable user namespace support for Pod Security Standards. 843 Enabling this feature will modify all Pod Security Standard rules to allow setting: `spec[.*].securityContext.[runAsNonRoot,runAsUser]`. 844 This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled. 845 The feature gate will not graduate or be enabled by default in future Kubernetes releases. ([#118760](https://github.com/kubernetes/kubernetes/pull/118760), [@saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Auth, Node and Release] 846 - Added `optionalOldSelf` to `x-kubernetes-validations` to support ratcheting CRD schema constraints. ([#121034](https://github.com/kubernetes/kubernetes/pull/121034), [@alexzielenski](https://github.com/alexzielenski)) 847 - Added a new `ServiceCIDR` type that allows to dynamically configure the cluster range used to allocate `Service ClusterIPs` addresses. ([#116516](https://github.com/kubernetes/kubernetes/pull/116516), [@aojea](https://github.com/aojea)) 848 - Added a new `ipMode` field to the `.status` of Services where `type` is set to `LoadBalancer`. 849 The new field is behind the `LoadBalancerIPMode` feature gate. ([#119937](https://github.com/kubernetes/kubernetes/pull/119937), [@RyanAoh](https://github.com/RyanAoh)) [SIG API Machinery, Apps, Cloud Provider, Network and Testing] 850 - Added options for configuring `nf_conntrack_udp_timeout`, and `nf_conntrack_udp_timeout_stream` variables of netfilter conntrack subsystem. ([#120808](https://github.com/kubernetes/kubernetes/pull/120808), [@aroradaman](https://github.com/aroradaman)) 851 - Added support for CEL expressions to `v1alpha1 AuthorizationConfiguration` webhook `matchConditions`. ([#121223](https://github.com/kubernetes/kubernetes/pull/121223), [@ritazh](https://github.com/ritazh)) 852 - Added support for projecting `certificates.k8s.io/v1alpha1` ClusterTrustBundle objects into pods. ([#113374](https://github.com/kubernetes/kubernetes/pull/113374), [@ahmedtd](https://github.com/ahmedtd)) 853 - Added the `DisableNodeKubeProxyVersion` feature gate. If `DisableNodeKubeProxyVersion` is enabled, the `kubeProxyVersion` field is not set. ([#120954](https://github.com/kubernetes/kubernetes/pull/120954), [@HirazawaUi](https://github.com/HirazawaUi)) 854 - Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps. 855 The incorrect cost was evident when the result of a function was used in subsequent operations. ([#119800](https://github.com/kubernetes/kubernetes/pull/119800), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider] 856 - Fixed the API comments for the Job `Ready` field in status. ([#121765](https://github.com/kubernetes/kubernetes/pull/121765), [@mimowo](https://github.com/mimowo)) 857 - Fixed the API comments for the `FailIndex` Job pod failure policy action. ([#121764](https://github.com/kubernetes/kubernetes/pull/121764), [@mimowo](https://github.com/mimowo)) 858 - Go API: the `ResourceRequirements` struct was replaced with `VolumeResourceRequirements` for use with volumes. ([#118653](https://github.com/kubernetes/kubernetes/pull/118653), [@pohly](https://github.com/pohly)) 859 - Graduated `Job BackoffLimitPerIndex` feature to `beta`. ([#121356](https://github.com/kubernetes/kubernetes/pull/121356), [@mimowo](https://github.com/mimowo)) 860 - Marked the `onPodConditions` field as optional in `Job`'s pod failure policy. ([#120204](https://github.com/kubernetes/kubernetes/pull/120204), [@mimowo](https://github.com/mimowo)) 861 - Promoted `PodReadyToStartContainers` condition to `beta`. ([#119659](https://github.com/kubernetes/kubernetes/pull/119659), [@kannon92](https://github.com/kannon92)) 862 - The `flowcontrol.apiserver.k8s.io/v1beta3` `FlowSchema` and `PriorityLevelConfiguration` APIs has been promoted to `flowcontrol.apiserver.k8s.io/v1`, with the following changes: 863 - `PriorityLevelConfiguration`: the `.spec.limited.nominalConcurrencyShares` field defaults to `30` only if the field is omitted (v1beta3 also defaulted an explicit `0` value to `30`). Specifying an explicit `0` value is not allowed in the `v1` version in v1.29 to ensure compatibility with `v1.28` API servers. In `v1.30`, explicit `0` values will be allowed in this field in the `v1` API. 864 The `flowcontrol.apiserver.k8s.io/v1beta3` APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the `v1` APIs. Transition clients and manifests to use the `v1` APIs before upgrading to `v1.32`. ([#121089](https://github.com/kubernetes/kubernetes/pull/121089), [@tkashem](https://github.com/tkashem)) 865 - The `kube-proxy` command-line documentation was updated to clarify that 866 `--bind-address` does not actually have anything to do with binding to an 867 address, and you probably don't actually want to be using it. ([#120274](https://github.com/kubernetes/kubernetes/pull/120274), [@danwinship](https://github.com/danwinship)) 868 - The `kube-scheduler` `selectorSpread` plugin has been removed, please use the `podTopologySpread` plugin instead. ([#117720](https://github.com/kubernetes/kubernetes/pull/117720), [@kerthcet](https://github.com/kerthcet)) 869 - The `matchLabelKeys/mismatchLabelKeys` feature is introduced to the hard/soft `PodAffinity/PodAntiAffinity`. ([#116065](https://github.com/kubernetes/kubernetes/pull/116065), [@sanposhiho](https://github.com/sanposhiho)) 870 - When updating a CRD, per-expression cost limit check are now skipped for `x-kubernetes-validations` rules of versions that are not mutated. ([#121460](https://github.com/kubernetes/kubernetes/pull/121460), [@jiahuif](https://github.com/jiahuif)) 871 - `CSINodeExpandSecret` feature has been promoted to `GA` in this release and is enabled 872 by default. The CSI drivers can make use of the `secretRef` values passed in `NodeExpansion` 873 request optionally sent by the CSI Client from this release onwards. ([#121303](https://github.com/kubernetes/kubernetes/pull/121303), [@humblec](https://github.com/humblec)) 874 - `NodeStageVolume` calls will now be retried if the CSI node driver is not running. ([#120330](https://github.com/kubernetes/kubernetes/pull/120330), [@rohitssingh](https://github.com/rohitssingh)) 875 - `PersistentVolumeLastPhaseTransitionTime` is now beta and enabled by default. ([#120627](https://github.com/kubernetes/kubernetes/pull/120627), [@RomanBednar](https://github.com/RomanBednar)) 876 - `ValidatingAdmissionPolicy` type checking now supports CRDs and API extensions types. ([#119109](https://github.com/kubernetes/kubernetes/pull/119109), [@jiahuif](https://github.com/jiahuif)) 877 - `kube-apiserver`: added `--authorization-config` flag for reading a configuration file containing an `apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration` object. The `--authorization-config` flag is mutually exclusive with `--authorization-modes` and `--authorization-webhook-*` flags. The `alpha` `StructuredAuthorizationConfiguration` feature flag must be enabled for `--authorization-config` to be specified. ([#120154](https://github.com/kubernetes/kubernetes/pull/120154), [@palnabarun](https://github.com/palnabarun)) 878 - `kube-proxy` now has a new nftables-based mode, available by running 879 880 `kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables` 881 882 This is currently an alpha-level feature and while it probably will not 883 eat your data, it may nibble at it a bit. (It passes e2e testing but has 884 not yet seen real-world use.) 885 886 At this point it should be functionally mostly identical to the iptables 887 mode, except that it does not (and will not) support Service NodePorts on 888 127.0.0.1. (Also note that there are currently no command-line arguments 889 for the nftables-specific config; you will need to use a config file if 890 you want to set the equivalent of any of the `--iptables-xxx` options.) 891 892 As this code is still very new, it has not been heavily optimized yet; 893 while it is expected to _eventually_ have better performance than the 894 iptables backend, very little performance testing has been done so far. ([#121046](https://github.com/kubernetes/kubernetes/pull/121046), [@danwinship](https://github.com/danwinship)) 895 - `kube-proxy`: Added an option/flag for configuring the `nf_conntrack_tcp_be_liberal` sysctl (in the kernel's netfilter conntrack subsystem). When enabled, `kube-proxy` will not install the `DROP` rule for invalid conntrack states, which currently breaks users of asymmetric routing. ([#120354](https://github.com/kubernetes/kubernetes/pull/120354), [@aroradaman](https://github.com/aroradaman)) 896 897 ### Feature 898 899 - #### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: 900 901 <!-- 902 This section can be blank if this pull request does not require a release note. 903 904 When adding links which point to resources within git repositories, like 905 KEPs or supporting documentation, please reference a specific commit and avoid 906 linking directly to the master branch. This ensures that links reference a 907 specific point in time, rather than a document that may change over time. 908 909 See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files 910 911 Please use the following format for linking documentation: 912 - [KEP]: <link> 913 - [Usage]: <link> 914 - [Other doc]: <link> 915 --> ([#119517](https://github.com/kubernetes/kubernetes/pull/119517), [@sanposhiho](https://github.com/sanposhiho)) [SIG Node, Scheduling and Testing] 916 - '`kubeadm`: added validation to verify that the `CertificateKey` is a valid hex 917 encoded AES key.' ([#120064](https://github.com/kubernetes/kubernetes/pull/120064), [@SataQiu](https://github.com/SataQiu)) 918 - A customizable `OrderedScoreFuncs()` function was introduced. Out-of-tree plugins 919 that used the scheduler's preemption interface could implement this function 920 for custom preemption preferences or return nil to keep the current behavior. ([#121867](https://github.com/kubernetes/kubernetes/pull/121867), [@lianghao208](https://github.com/lianghao208)) 921 - Added `apiextensions_apiserver_update_ratcheting_time` metric for tracking time taken during requests by feature `CRDValidationRatcheting`. ([#121462](https://github.com/kubernetes/kubernetes/pull/121462), [@alexzielenski](https://github.com/alexzielenski)) 922 - Added `apiserver_envelope_encryption_dek_cache_filled` to measure number of records in data encryption key (DEK) cache. ([#119878](https://github.com/kubernetes/kubernetes/pull/119878), [@ritazh](https://github.com/ritazh)) 923 - Added `apiserver_watch_list_duration_seconds` metrics which will measure response latency distribution in seconds for watchlist requests broken by group, version, resource and scope. ([#120490](https://github.com/kubernetes/kubernetes/pull/120490), [@p0lyn0mial](https://github.com/p0lyn0mial)) 924 - Added `job_pods_creation_total` metrics for tracking Pods created by the Job controller labeled by events which triggered the Pod creation. ([#121481](https://github.com/kubernetes/kubernetes/pull/121481), [@dejanzele](https://github.com/dejanzele)) 925 - Added `kubectl node drain` helper callbacks `OnPodDeletionOrEvictionStarted` 926 and `OnPodDeletionOrEvictionFailed`; people extending `kubectl` can use these 927 new callbacks for more granularity. Deprecated the `OnPodDeletedOrEvicted` 928 node drain helper callback. ([#117502](https://github.com/kubernetes/kubernetes/pull/117502), [@adilGhaffarDev](https://github.com/adilGhaffarDev)) 929 - Added a new `--init-only` command line flag to `kube-proxy`. Setting the flag makes `kube-proxy` perform its initial configuration that requires privileged mode, and then exit. The `--init-only` mode is intended to be executed in a privileged init container, so that the main container may run with a stricter `securityContext`. ([#120864](https://github.com/kubernetes/kubernetes/pull/120864), [@uablrek](https://github.com/uablrek)) [SIG Network and Scalability] 930 - Added a new scheduler metric, `pod_scheduling_sli_duration_seconds`, and started the deprecation for `pod_scheduling_duration_seconds`. ([#119049](https://github.com/kubernetes/kubernetes/pull/119049), [@helayoty](https://github.com/helayoty)) 931 - Added a return value to `QueueingHint` to indicate an error. If `QueueingHint` returns an error, 932 the scheduler logs it and treats the event as a `QueueAfterBackoff` so that 933 the Pod won't be stuck in the unschedulable pod pool. ([#119290](https://github.com/kubernetes/kubernetes/pull/119290), [@carlory](https://github.com/carlory)) 934 - Added apiserver identity to the following metrics: 935 `apiserver_envelope_encryption_key_id_hash_total`, `apiserver_envelope_encryption_key_id_hash_last_timestamp_seconds`, `apiserver_envelope_encryption_key_id_hash_status_last_timestamp_seconds`, `apiserver_encryption_config_controller_automatic_reload_failures_total`, `apiserver_encryption_config_controller_automatic_reload_success_total`, `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds`. 936 937 Fixed bug to surface events for the following metrics: `apiserver_encryption_config_controller_automatic_reload_failures_total`, `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds`, `apiserver_encryption_config_controller_automatic_reload_success_total`. ([#120438](https://github.com/kubernetes/kubernetes/pull/120438), [@ritazh](https://github.com/ritazh)) 938 - Added container filesystem to the `ImageFsInfoResponse`. ([#120914](https://github.com/kubernetes/kubernetes/pull/120914), [@kannon92](https://github.com/kannon92)) 939 - Added multiplication functionality to `Quantity`. ([#117411](https://github.com/kubernetes/kubernetes/pull/117411), [@tenzen-y](https://github.com/tenzen-y)) 940 - Added new feature gate called `RuntimeClassInImageCriApi` to address `kubelet` changes needed for KEP 4216. 941 Noteable changes: 942 1. Populate new `RuntimeHandler` field in CRI's `ImageSpec` struct during image pulls from container runtimes. 943 2. Pass `runtimeHandler` field in `RemoveImage()` call to container runtime in `kubelet`'s image garbage collection. ([#121456](https://github.com/kubernetes/kubernetes/pull/121456), [@kiashok](https://github.com/kiashok)) 944 - Added support for split image filesystem in kubelet. ([#120616](https://github.com/kubernetes/kubernetes/pull/120616), [@kannon92](https://github.com/kannon92)) 945 - Bumped `cel-go` to `v0.17.7` and introduced set `ext` library with new options. ([#121577](https://github.com/kubernetes/kubernetes/pull/121577), [@cici37](https://github.com/cici37)) 946 - Bumped `distroless-iptables` to `0.3.2` based on Go `1.21.1`. ([#120527](https://github.com/kubernetes/kubernetes/pull/120527), [@cpanato](https://github.com/cpanato)) 947 - Bumped `distroless-iptables` to `0.3.3` based on Go `1.21.2`. ([#121073](https://github.com/kubernetes/kubernetes/pull/121073), [@cpanato](https://github.com/cpanato)) 948 - Bumped `distroless-iptables` to `0.4.1` based on Go `1.21.3`. ([#121216](https://github.com/kubernetes/kubernetes/pull/121216), [@cpanato](https://github.com/cpanato)) 949 - Bumped distroless-iptables to 0.4.1 based on Go `1.21.3`. ([#121871](https://github.com/kubernetes/kubernetes/pull/121871), [@cpanato](https://github.com/cpanato)) 950 - CEL can now correctly handle a CRD `openAPIV3Schema` that has neither `Properties` nor `AdditionalProperties`. ([#121459](https://github.com/kubernetes/kubernetes/pull/121459), [@jiahuif](https://github.com/jiahuif)) 951 - CEL cost estimator no longer treats enums as unbounded strings when determining its length. Instead, the length is set to the longest possible enum value. ([#121085](https://github.com/kubernetes/kubernetes/pull/121085), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery] 952 - CRI: image pull per runtime class is now supported. ([#121121](https://github.com/kubernetes/kubernetes/pull/121121), [@kiashok](https://github.com/kiashok)) 953 - Certain `requestBody` parameters in the OpenAPI `v3` are now correctly marked as required. ([#120735](https://github.com/kubernetes/kubernetes/pull/120735), [@Jefftree](https://github.com/Jefftree)) 954 - Changed `kubectl help` to display basic details for subcommands from plugins. ([#116752](https://github.com/kubernetes/kubernetes/pull/116752), [@xvzf](https://github.com/xvzf)) 955 - Changed the `KMSv2KDF` feature gate to be enabled by default. ([#120433](https://github.com/kubernetes/kubernetes/pull/120433), [@enj](https://github.com/enj)) [SIG API Machinery, Auth and Testing] 956 - Client-side apply will now use OpenAPI `v3` by default. ([#120707](https://github.com/kubernetes/kubernetes/pull/120707), [@Jefftree](https://github.com/Jefftree)) 957 - Decoding etcd's response now respects the timeout context. ([#121614](https://github.com/kubernetes/kubernetes/pull/121614), [@HirazawaUi](https://github.com/HirazawaUi)) 958 - Decoupled `TaintManager` from `NodeLifeCycleController` (KEP-3902). ([#119208](https://github.com/kubernetes/kubernetes/pull/119208), [@atosatto](https://github.com/atosatto)) 959 - Enabled traces for KMSv2 encrypt/decrypt operations. ([#121095](https://github.com/kubernetes/kubernetes/pull/121095), [@aramase](https://github.com/aramase)) 960 - Fixed `kube-proxy` panicking on exit when the `Node` object changed its `PodCIDR`. ([#120375](https://github.com/kubernetes/kubernetes/pull/120375), [@pegasas](https://github.com/pegasas)) 961 - Fixed bugs in handling of server-side apply, create, and update API requests for objects containing duplicate items in keyed lists. 962 - A `create` or `update` API request with duplicate items in a keyed list no longer wipes out managedFields. Examples include env var entries with the same name, or port entries with the same containerPort in a pod spec. 963 - A server-side apply request that makes unrelated changes to an object which has duplicate items in a keyed list no longer fails, and leaves the existing duplicate items as-is. 964 - A server-side apply request that changes an object which has duplicate items in a keyed list, and modifies the duplicated item removes the duplicates and replaces them with the single item contained in the server-side apply request. ([#121575](https://github.com/kubernetes/kubernetes/pull/121575), [@apelisse](https://github.com/apelisse)) 965 - Fixed overriding default `KubeletConfig` fields in drop-in configs if not set. ([#121193](https://github.com/kubernetes/kubernetes/pull/121193), [@sohankunkerkar](https://github.com/sohankunkerkar)) 966 - Graduated API List chunking (aka pagination) feature to `stable`. ([#119503](https://github.com/kubernetes/kubernetes/pull/119503), [@wojtek-t](https://github.com/wojtek-t)) 967 - Graduated the `ReadWriteOncePod` feature gate to `GA`. ([#121077](https://github.com/kubernetes/kubernetes/pull/121077), [@chrishenzie](https://github.com/chrishenzie)) 968 - Graduated the following kubelet resource metrics to **general availability**: 969 - `container_cpu_usage_seconds_total` 970 - `container_memory_working_set_bytes` 971 - `container_start_time_seconds` 972 - `node_cpu_usage_seconds_total` 973 - `node_memory_working_set_bytes` 974 - `pod_cpu_usage_seconds_total` 975 - `pod_memory_working_set_bytes` 976 - `resource_scrape_error` 977 978 Deprecated (renamed) `scrape_error` in favor of `resource_scrape_error` ([#116897](https://github.com/kubernetes/kubernetes/pull/116897), [@Richabanker](https://github.com/Richabanker)) [SIG Architecture, Instrumentation, Node and Testing] 979 - Implemented API for streaming for the `etcd` store implementation. 980 When `sendInitialEvents ListOption` is set together with `watch=true`, it begins the watch stream with synthetic init events followed by a synthetic `Bookmark`, after which the server continues streaming events. ([#119557](https://github.com/kubernetes/kubernetes/pull/119557), [@p0lyn0mial](https://github.com/p0lyn0mial)) 981 - Improved memory usage of `kube-scheduler` by dropping the `.metadata.managedFields` field that `kube-scheduler` doesn't require. ([#119556](https://github.com/kubernetes/kubernetes/pull/119556), [@linxiulei](https://github.com/linxiulei)) 982 - In a scheduler with `Permit` plugins, when a Pod is rejected during `WaitOnPermit`, the scheduler records the plugin. 983 The scheduler will use the record to honor cluster events and queueing `hints registered` for the plugin, to inform whether to retry the pod. ([#119785](https://github.com/kubernetes/kubernetes/pull/119785), [@sanposhiho](https://github.com/sanposhiho)) 984 - In-tree cloud providers are now switched off by default. Please use `DisableCloudProviders` and `DisableKubeletCloudCredentialProvider` feature flags if you still need this functionality. ([#117503](https://github.com/kubernetes/kubernetes/pull/117503), [@dims](https://github.com/dims)) 985 - Introduced new apiserver metric `apiserver_flowcontrol_current_inqueue_seats`. This metric is analogous to `apiserver_flowcontrol_current_inqueue_requests`, but tracks the total number of seats, as each request can take more than one seat. ([#119385](https://github.com/kubernetes/kubernetes/pull/119385), [@andrewsykim](https://github.com/andrewsykim)) 986 - Introduced the `job_finished_indexes_total` metric for the `BackoffLimitPerIndex` feature. ([#121292](https://github.com/kubernetes/kubernetes/pull/121292), [@mimowo](https://github.com/mimowo)) 987 - Kubeadm: supported updating certificate organization during `kubeadm certs renew` operation. ([#121841](https://github.com/kubernetes/kubernetes/pull/121841), [@SataQiu](https://github.com/SataQiu)) 988 - Kubernetes is now built with Go `1.21.0`. ([#118996](https://github.com/kubernetes/kubernetes/pull/118996), [@cpanato](https://github.com/cpanato)) 989 - Kubernetes is now built with Go `1.21.1`. ([#120493](https://github.com/kubernetes/kubernetes/pull/120493), [@cpanato](https://github.com/cpanato)) 990 - Kubernetes is now built with Go `1.21.2`. ([#121021](https://github.com/kubernetes/kubernetes/pull/121021), [@cpanato](https://github.com/cpanato)) 991 - Kubernetes is now built with Go `1.21.4`. ([#121808](https://github.com/kubernetes/kubernetes/pull/121808), [@cpanato](https://github.com/cpanato)) 992 - Kubernetes is now built with Go `v1.21.3`. ([#121149](https://github.com/kubernetes/kubernetes/pull/121149), [@cpanato](https://github.com/cpanato)) 993 - List of metric labels can now be configured by supplying a manifest using the `--allow-metric-labels-manifest` flag. ([#118299](https://github.com/kubernetes/kubernetes/pull/118299), [@rexagod](https://github.com/rexagod)) 994 - Listed the pods using `<PVC>` as an ephemeral storage volume in "Used by:" part of the output of `kubectl describe pvc <PVC>` command. ([#120427](https://github.com/kubernetes/kubernetes/pull/120427), [@MaGaroo](https://github.com/MaGaroo)) 995 - Migrated the `nodevolumelimits` scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116884](https://github.com/kubernetes/kubernetes/pull/116884), [@mengjiao-liu](https://github.com/mengjiao-liu)) 996 - Migrated the `volumebinding scheduler plugins` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116803](https://github.com/kubernetes/kubernetes/pull/116803), [@mengjiao-liu](https://github.com/mengjiao-liu)) 997 - Priority and Fairness feature is now `stable`, the feature gate will be removed in `v1.31`. ([#121638](https://github.com/kubernetes/kubernetes/pull/121638), [@tkashem](https://github.com/tkashem)) 998 - Promoted `PodHostIPs` condition to `beta`. ([#120257](https://github.com/kubernetes/kubernetes/pull/120257), [@wzshiming](https://github.com/wzshiming)) 999 - Promoted `PodHostIPs` condition to `beta`. ([#121477](https://github.com/kubernetes/kubernetes/pull/121477), [@wzshiming](https://github.com/wzshiming)) 1000 - Promoted `PodReplacementPolicy` to `beta`. ([#121491](https://github.com/kubernetes/kubernetes/pull/121491), [@dejanzele](https://github.com/dejanzele)) 1001 - Promoted `ServiceNodePortStaticSubrange` to stable and lock to default. ([#120233](https://github.com/kubernetes/kubernetes/pull/120233), [@xuzhenglun](https://github.com/xuzhenglun)) 1002 - Promoted plugin subcommand resolution feature to `beta`. ([#120663](https://github.com/kubernetes/kubernetes/pull/120663), [@ardaguclu](https://github.com/ardaguclu)) 1003 - Removed `/livez` livezchecks for KMS v1 and v2 to ensure KMS health does not cause `kube-apiserver` restart. KMS health checks are still in place as a healthz and readiness checks. ([#120583](https://github.com/kubernetes/kubernetes/pull/120583), [@ritazh](https://github.com/ritazh)) 1004 - Restartable init containers resource in pod autoscaler are now calculated. ([#120001](https://github.com/kubernetes/kubernetes/pull/120001), [@qingwave](https://github.com/qingwave)) 1005 - Sidecar termination is now serialized and each sidecar container will receive a `SIGTERM` after all main containers and later starting sidecar containers have terminated. ([#120620](https://github.com/kubernetes/kubernetes/pull/120620), [@tzneal](https://github.com/tzneal)) 1006 - The CRD validation rule with feature gate `CustomResourceValidationExpressions` was promoted to `GA`. ([#121373](https://github.com/kubernetes/kubernetes/pull/121373), [@cici37](https://github.com/cici37)) 1007 - The KMSv2 features with feature gates `KMSv2` and `KMSv2KDF` are promoted to `GA`. The `KMSv1` feature gate is now disabled by default. ([#121485](https://github.com/kubernetes/kubernetes/pull/121485), [@ritazh](https://github.com/ritazh)) 1008 - The `--interactive` flag in `kubectl delete` is now visible to all users by default. ([#120416](https://github.com/kubernetes/kubernetes/pull/120416), [@ardaguclu](https://github.com/ardaguclu)) 1009 - The `CloudDualStackNodeIPs` feature is now `beta`, meaning that when using 1010 an external cloud provider that has been updated to support the feature, 1011 you can pass comma-separated dual-stack `--node-ips` to `kubelet` and have 1012 the cloud provider take both IPs into account. ([#120275](https://github.com/kubernetes/kubernetes/pull/120275), [@danwinship](https://github.com/danwinship)) 1013 - The `Dockerfile` for the kubectl image has been updated with the addition of a specific base image and essential utilities (bash and jq). ([#119592](https://github.com/kubernetes/kubernetes/pull/119592), [@rayandas](https://github.com/rayandas)) 1014 - The `SidecarContainers` feature has graduated to `beta` and is enabled by default. ([#121579](https://github.com/kubernetes/kubernetes/pull/121579), [@gjkim42](https://github.com/gjkim42)) 1015 - The `kube-apiserver` will now expose four new metrics to inform about errors on the clusterIP and nodePort allocation logic. ([#120843](https://github.com/kubernetes/kubernetes/pull/120843), [@aojea](https://github.com/aojea)) 1016 - The `volume_zone` plugin will consider `beta` labels as `GA` labels during the scheduling 1017 process. Therefore, if the values of the labels are the same, PVs with `beta` labels 1018 can also be scheduled to nodes with `GA` labels. ([#118923](https://github.com/kubernetes/kubernetes/pull/118923), [@AxeZhan](https://github.com/AxeZhan)) 1019 - Updated the generic apiserver library to produce an error if a new API server is configured with support for a data format other than JSON, YAML, or Protobuf. ([#121325](https://github.com/kubernetes/kubernetes/pull/121325), [@benluddy](https://github.com/benluddy)) [SIG API Machinery] 1020 - Use of secret-based service account tokens now adds an `authentication.k8s.io/legacy-token-autogenerated-secret` or `authentication.k8s.io/legacy-token-manual-secret` audit annotation containing the name of the secret used. ([#118598](https://github.com/kubernetes/kubernetes/pull/118598), [@yuanchen8911](https://github.com/yuanchen8911)) [SIG Auth, Instrumentation and Testing] 1021 - `--sync-frequency` will not affect the update interval of volumes that use `ConfigMaps` 1022 or `Secrets` when the `configMapAndSecretChangeDetectionStrategy` is set to `Cache`. 1023 The update interval is only affected by `node.alpha.kubernetes.io/ttl` node annotation." ([#120255](https://github.com/kubernetes/kubernetes/pull/120255), [@likakuli](https://github.com/likakuli)) 1024 - `CRDValidationRatcheting`: added support for ratcheting `x-kubernetes-validations` in schema. ([#121016](https://github.com/kubernetes/kubernetes/pull/121016), [@alexzielenski](https://github.com/alexzielenski)) 1025 - `DevicePluginCDIDevices` feature has been graduated to `beta` and enabled by default in the kubelet. ([#121254](https://github.com/kubernetes/kubernetes/pull/121254), [@bart0sh](https://github.com/bart0sh)) 1026 - `ValidatingAdmissionPolicy` now preserves types of composition variables, and raises type-related errors early. ([#121001](https://github.com/kubernetes/kubernetes/pull/121001), [@jiahuif](https://github.com/jiahuif)) 1027 - `cluster/gce`: added webhook to replace `PersistentVolumeLabel` admission controller. ([#121628](https://github.com/kubernetes/kubernetes/pull/121628), [@andrewsykim](https://github.com/andrewsykim)) 1028 - `dra`: the scheduler plugin now avoids additional scheduling attempts in some cases by falling back to SSA after a conflict. ([#120534](https://github.com/kubernetes/kubernetes/pull/120534), [@pohly](https://github.com/pohly)) 1029 - `etcd`: image is now based on `v3.5.9`. ([#121567](https://github.com/kubernetes/kubernetes/pull/121567), [@mzaian](https://github.com/mzaian)) 1030 - `kube-apiserver` added: 1031 - `alpha` support (guarded by the `ServiceAccountTokenJTI` feature gate) for adding a `jti` (JWT ID) claim to service account tokens it issues, adding an `authentication.kubernetes.io/credential-id` audit annotation in audit logs when the tokens are issued, and `authentication.kubernetes.io/credential-id` entry in the extra user info when the token is used to authenticate. 1032 - `alpha` support (guarded by the `ServiceAccountTokenPodNodeInfo` feature gate) for including the node name (and uid, if the node exists) as additional claims in service account tokens it issues which are bound to pods, and `authentication.kubernetes.io/node-name` and `authentication.kubernetes.io/node-uid` extra user info when the token is used to authenticate. 1033 - `alpha` support (guarded by the `ServiceAccountTokenNodeBinding` feature gate) for allowing `TokenRequests` that bind tokens directly to nodes, and (guarded by the ServiceAccountTokenNodeBindingValidation feature gate) for validating the node name and uid still exist when the token is used. ([#120780](https://github.com/kubernetes/kubernetes/pull/120780), [@munnerz](https://github.com/munnerz)) 1034 - `kube-apiserver` updated: 1035 - Updated encryption configuration file watch logic from using inotify watch to polling at an interval of every minute. Moved the logic to polling because there are variations on file changes (like symlink swapping of directories that contain the encryption config) that the file watch logic would fail to detect. Polling at a set interval prevents any such issues. Note that there is no guarantee on how quickly the API server will process the encryption config. The `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds` metric must be used to determine when the new config becomes effective.([#121310](https://github.com/kubernetes/kubernetes/pull/121310), [@nilekhc](https://github.com/nilekhc)) 1036 - `kube-controller-manager`: The `LegacyServiceAccountTokenCleanUp` feature gate is now `beta` and enabled by default. When enabled, legacy auto-generated service account token secrets are auto-labeled with a `kubernetes.io/legacy-token-invalid-since` label if the credentials have not been used in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year), **and** are referenced from the `.secrets` list of a ServiceAccount object, **and** are not referenced from pods. This label causes the authentication layer to reject use of the credentials. After being labeled as invalid, if the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year) passes without the credential being used, the secret is automatically deleted. Secrets labeled as invalid which have not been auto-deleted yet can be re-activated by removing the `kubernetes.io/legacy-token-invalid-since` label. ([#120682](https://github.com/kubernetes/kubernetes/pull/120682), [@yt2985](https://github.com/yt2985)) 1037 - `kube-proxy` will only install the `DROP` rules for invalid `conntrack` states if 1038 the `nf_conntrack_tcp_be_liberal` is not set. ([#120412](https://github.com/kubernetes/kubernetes/pull/120412), [@aojea](https://github.com/aojea)) 1039 - `kube-scheduler` implemented scheduling hints for the `NodeUnschedulable` plugin. 1040 The scheduling hints allow the scheduler to only retry scheduling a `Pod` 1041 that was previously rejected by the `NodeSchedulable` plugin if a new `Node` or a `Node` update sets `.spec.unschedulable` to false. ([#119396](https://github.com/kubernetes/kubernetes/pull/119396), [@wackxu](https://github.com/wackxu)) 1042 - `kube-scheduler` implements scheduling hints for the `NodeAffinity` plugin. 1043 The scheduling hints allow the scheduler to only retry scheduling a `Pod` 1044 that was previously rejected by the `NodeAffinity` plugin if a new `Node` or a `Node` update matches the `Pod`'s node affinity. ([#119155](https://github.com/kubernetes/kubernetes/pull/119155), [@carlory](https://github.com/carlory)) 1045 - `kubeadm`: promoted feature gate `EtcdLearnerMode` to `beta`. Learner mode for 1046 joining `etcd` members is now enabled by default. ([#120228](https://github.com/kubernetes/kubernetes/pull/120228), [@pacoxu](https://github.com/pacoxu)) 1047 - `kubeadm`: turned on feature gate `MergeCLIArgumentsWithConfig` to merge the config from flag and config file, otherwise, if the flag `--ignore-preflight-errors` is set from the CLI, then the value from config file will be ignored. ([#119946](https://github.com/kubernetes/kubernetes/pull/119946), [@chendave](https://github.com/chendave)) 1048 - `kubeadm`: will now allow deploying a kubelet that is 3 versions older than the version of `kubeadm` (N-3). This aligns with the recent change made by SIG Architecture that extends the support skew between the control plane and kubelets. Tolerate this new kubelet skew for the commands `init`, `join` and `upgrade`. Note that if the `kubeadm` user applies a control plane version that is older than the `kubeadm` version (N-1 maximum) then the skew between the kubelet and control plane would become a maximum of N-2. ([#120825](https://github.com/kubernetes/kubernetes/pull/120825), [@pacoxu](https://github.com/pacoxu)) 1049 - `kubelet` , when using `--cloud-provider=external`, will now initialize the node addresses with the value of `--node-ip` , if it exists, or waits for the cloud provider to assign the addresses. ([#121028](https://github.com/kubernetes/kubernetes/pull/121028), [@aojea](https://github.com/aojea)) 1050 - `kubelet` allows pods to use the `net.ipv4.tcp_fin_timeout`, “net.ipv4.tcp_keepalive_intvl” 1051 and “net.ipv4.tcp_keepalive_probes“ sysctl by default; Pod Security Admission 1052 allows this sysctl in `v1.29+` versions of the baseline and restricted policies. ([#121240](https://github.com/kubernetes/kubernetes/pull/121240), [@HirazawaUi](https://github.com/HirazawaUi)) 1053 - `kubelet` now allows pods to use the `net.ipv4.tcp_keepalive_time` sysctl by default 1054 and the minimal kernel version is 4.5; Pod Security Admission allows this sysctl 1055 in `v1.29+` versions of the baseline and restricted policies. ([#118846](https://github.com/kubernetes/kubernetes/pull/118846), [@cyclinder](https://github.com/cyclinder)) 1056 - `kubelet` now emits a metric for end-to-end pod startup latency, including image pull. ([#121041](https://github.com/kubernetes/kubernetes/pull/121041), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) 1057 - `kubelet` now exposes latency metrics of different stages of the node startup. ([#118568](https://github.com/kubernetes/kubernetes/pull/118568), [@qiutongs](https://github.com/qiutongs)) 1058 1059 ### Documentation 1060 1061 - Added descriptions and examples for the situation of using `kubectl rollout restart` without specifying a particular deployment. ([#120118](https://github.com/kubernetes/kubernetes/pull/120118), [@Ithrael](https://github.com/Ithrael)) 1062 - When the kubelet fails to assign CPUs to a Pod because there less available CPUs than the Pod requests, the error message changed from 1063 `not enough cpus available to satisfy request` to `not enough cpus available to satisfy request: <num_requested> requested, only <num_available> available`. ([#121059](https://github.com/kubernetes/kubernetes/pull/121059), [@matte21](https://github.com/matte21)) 1064 1065 ### Failing Test 1066 1067 - Added mock framework support for unit tests for Windows in `kubeproxy`. ([#120105](https://github.com/kubernetes/kubernetes/pull/120105), [@princepereira](https://github.com/princepereira)) 1068 - DRA: when the scheduler had to deallocate a claim after a node became unsuitable for a pod, it might have needed more attempts than really necessary. This was fixed by first disabling allocations. ([#120428](https://github.com/kubernetes/kubernetes/pull/120428), [@pohly](https://github.com/pohly)) 1069 - E2e framework: retrying after intermittent `apiserver` failures was fixed in `WaitForPodsResponding` ([#120559](https://github.com/kubernetes/kubernetes/pull/120559), [@pohly](https://github.com/pohly)) 1070 - KCM specific args can be passed with `/cluster` script, without affecting CCM. New variable name: `KUBE_CONTROLLER_MANAGER_TEST_ARGS`. ([#120524](https://github.com/kubernetes/kubernetes/pull/120524), [@jprzychodzen](https://github.com/jprzychodzen)) [SIG Cloud Provider] 1071 - `k8s.io/dynamic-resource-allocation`: DRA drivers updating to this release are compatible with Kubernetes `v1.27` and `v1.28`. ([#120868](https://github.com/kubernetes/kubernetes/pull/120868), [@pohly](https://github.com/pohly)) 1072 1073 ### Bug or Regression 1074 1075 - '`kubeadm`: printing the default component configs for `reset` and `join` is now 1076 unsupported.' ([#119346](https://github.com/kubernetes/kubernetes/pull/119346), [@chendave](https://github.com/chendave)) 1077 - '`kubeadm`: removed `system:masters` organization from `etcd/healthcheck-client` 1078 certificate.' ([#119859](https://github.com/kubernetes/kubernetes/pull/119859), [@SataQiu](https://github.com/SataQiu)) 1079 - Added `CAP_NET_RAW` to netadmin debug profile and removed privileges when debugging nodes. ([#118647](https://github.com/kubernetes/kubernetes/pull/118647), [@mochizuki875](https://github.com/mochizuki875)) 1080 - Added a check on a user attempting to create a static pod via the `kubelet` without specifying a name. They will now get a visible validation error. ([#119522](https://github.com/kubernetes/kubernetes/pull/119522), [@YTGhost](https://github.com/YTGhost)) 1081 - Added a redundant process to remove tracking finalizers from Pods that belong to Jobs. The process kicks in after the control plane marks a Job as finished. ([#119944](https://github.com/kubernetes/kubernetes/pull/119944), [@Sharpz7](https://github.com/Sharpz7)) 1082 - Added more accurate requeueing in scheduling queue for Pods rejected by the temporal failure (e.g., temporal failure on `kube-apiserver`). ([#119105](https://github.com/kubernetes/kubernetes/pull/119105), [@sanposhiho](https://github.com/sanposhiho)) 1083 - Allowed specifying `ExternalTrafficPolicy` for `Services` with `ExternalIPs`. ([#119150](https://github.com/kubernetes/kubernetes/pull/119150), [@tnqn](https://github.com/tnqn)) 1084 - Changed kubelet logs from `error` to `info` for uncached partitions when using CRI stats provider. ([#100448](https://github.com/kubernetes/kubernetes/pull/100448), [@saschagrunert](https://github.com/saschagrunert)) 1085 - Empty values are no longer assigned to undefined resources (CPU or memory) when storing the resources allocated to the pod in checkpoint. ([#117615](https://github.com/kubernetes/kubernetes/pull/117615), [@aheng-ch](https://github.com/aheng-ch)) 1086 - Fixed CEL estimated cost of `replace()` to handle a zero length replacement string correctly. 1087 Previously this would cause the estimated cost to be higher than it should be. ([#120097](https://github.com/kubernetes/kubernetes/pull/120097), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery] 1088 - Fixed OpenAPI v3 not being cleaned up after deleting `APIServices`. ([#120108](https://github.com/kubernetes/kubernetes/pull/120108), [@tnqn](https://github.com/tnqn)) 1089 - Fixed [121094](https://github.com/kubernetes/kubernetes/issues/121094) by re-introducing the readiness predicate for `externalTrafficPolicy: Local` services. ([#121116](https://github.com/kubernetes/kubernetes/pull/121116), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) 1090 - Fixed `kubectl events` not filtering events by `GroupVersion` for resources with a full name. ([#120119](https://github.com/kubernetes/kubernetes/pull/120119), [@Ithrael](https://github.com/Ithrael)) 1091 - Fixed `systemLogQuery` service name matching. ([#120678](https://github.com/kubernetes/kubernetes/pull/120678), [@rothgar](https://github.com/rothgar)) 1092 - Fixed a `1.27` scheduling regression that `PostFilter` plugin may not function if previous `PreFilter` plugins return `Skip`. ([#119769](https://github.com/kubernetes/kubernetes/pull/119769), [@Huang-Wei](https://github.com/Huang-Wei)) 1093 - Fixed a `v1.26` regression scheduling bug by ensuring that preemption is skipped when a `PreFilter` plugin returns `UnschedulableAndUnresolvable`. ([#119778](https://github.com/kubernetes/kubernetes/pull/119778), [@sanposhiho](https://github.com/sanposhiho)) 1094 - Fixed a `v1.28.0` regression where `kube-controller-manager` can crash when `StatefulSet` with `Parallel` policy and PVC labels are scaled up. ([#121142](https://github.com/kubernetes/kubernetes/pull/121142), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) 1095 - Fixed a `v1.28` regression around restarting init containers in the right order relative to normal containers. ([#120281](https://github.com/kubernetes/kubernetes/pull/120281), [@gjkim42](https://github.com/gjkim42)) 1096 - Fixed a `v1.28` regression handling negative index json patches. ([#120327](https://github.com/kubernetes/kubernetes/pull/120327), [@liggitt](https://github.com/liggitt)) 1097 - Fixed a `v1.28` regression in scheduler: a pod with concurrent events could incorrectly get moved to the unschedulable queue where it could get stuck until the next periodic purging after 5 minutes, if there was no other event for it. ([#120413](https://github.com/kubernetes/kubernetes/pull/120413), [@pohly](https://github.com/pohly)) 1098 - Fixed a bug around restarting init containers in the right order relative to normal containers with `SidecarContainers` feature enabled. ([#120269](https://github.com/kubernetes/kubernetes/pull/120269), [@gjkim42](https://github.com/gjkim42)) 1099 - Fixed a bug in the cronjob controller where already created jobs might be missing from the status. ([#120649](https://github.com/kubernetes/kubernetes/pull/120649), [@andrewsykim](https://github.com/andrewsykim)) 1100 - Fixed a bug where `Services` using finalizers may hold onto `ClusterIP` and/or `NodePort` allocated resources for longer than expected if the finalizer is removed using the status subresource. ([#120623](https://github.com/kubernetes/kubernetes/pull/120623), [@aojea](https://github.com/aojea)) 1101 - Fixed a bug where an API group's path was not unregistered from the API server's root paths when the group was deleted. ([#121283](https://github.com/kubernetes/kubernetes/pull/121283), [@tnqn](https://github.com/tnqn)) [SIG API Machinery and Testing] 1102 - Fixed a bug where containers would not start on `cgroupv2` systems where `swap` is disabled. ([#120784](https://github.com/kubernetes/kubernetes/pull/120784), [@elezar](https://github.com/elezar)) 1103 - Fixed a bug where the CPU set allocated to an init container, with containerRestartPolicy of `Always`, were erroneously reused by a regular container. ([#119447](https://github.com/kubernetes/kubernetes/pull/119447), [@gjkim42](https://github.com/gjkim42)) [SIG Node and Testing] 1104 - Fixed a bug where the device resources allocated to an init container, with `containerRestartPolicy` of `Always`, were erroneously reused by a regular container. ([#120461](https://github.com/kubernetes/kubernetes/pull/120461), [@gjkim42](https://github.com/gjkim42)) 1105 - Fixed a bug where the memory resources allocated to an init container, with containerRestartPolicy of `Always`, were erroneously reused by a regular container. ([#120715](https://github.com/kubernetes/kubernetes/pull/120715), [@gjkim42](https://github.com/gjkim42)) [SIG Node] 1106 - Fixed a concurrent map access in `TopologyCache`'s `HasPopulatedHints` method. ([#118189](https://github.com/kubernetes/kubernetes/pull/118189), [@Miciah](https://github.com/Miciah)) 1107 - Fixed a regression (`CLIENTSET_PKG: unbound variable`) when invoking deprecated `generate-groups.sh` script. ([#120877](https://github.com/kubernetes/kubernetes/pull/120877), [@soltysh](https://github.com/soltysh)) 1108 - Fixed a regression in `kube-proxy` where it might refuse to start if given 1109 single-stack `IPv6` configuration options on a node that has both `IPv4` and 1110 `IPv6` IPs. ([#121008](https://github.com/kubernetes/kubernetes/pull/121008), [@danwinship](https://github.com/danwinship)) 1111 - Fixed a regression in default configurations, which enabled `PodDisruptionConditions` 1112 by default, that prevented the control plane's pod garbage collector from deleting 1113 pods that contained duplicated field keys (environmental variables with repeated keys or 1114 container ports). ([#121103](https://github.com/kubernetes/kubernetes/pull/121103), [@mimowo](https://github.com/mimowo)) 1115 - Fixed a regression in the default `v1.27` configurations in `kube-apiserver`: fixed the `AggregatedDiscoveryEndpoint` feature (`beta` in `v1.27+`) to successfully fetch discovery information from aggregated API servers that do not check `Accept` headers when serving the `/apis` endpoint. ([#119870](https://github.com/kubernetes/kubernetes/pull/119870), [@Jefftree](https://github.com/Jefftree)) 1116 - Fixed a regression in the kubelet's behavior while creating a container when the `EventedPLEG` feature gate is enabled. ([#120942](https://github.com/kubernetes/kubernetes/pull/120942), [@sairameshv](https://github.com/sairameshv)) 1117 - Fixed a regression since `v1.27.0` in the scheduler framework when running score plugins. 1118 The `skippedScorePlugins` number might be greater than `enabledScorePlugins`, 1119 so when initializing a slice the `cap(len(skippedScorePlugins) - len(enabledScorePlugins))` is negative, 1120 which is not allowed. ([#121632](https://github.com/kubernetes/kubernetes/pull/121632), [@kerthcet](https://github.com/kerthcet)) 1121 - Fixed a situation when, sometimes, the scheduler incorrectly placed a pod in the `unschedulable` queue instead of the `backoff` queue. This happened when some plugin previously declared the pod as `unschedulable` and then in a later attempt encounters some other error. Scheduling of that pod then got delayed by up to five minutes, after which periodic flushing moved the pod back into the `active` queue. ([#120334](https://github.com/kubernetes/kubernetes/pull/120334), [@pohly](https://github.com/pohly)) 1122 - Fixed an issue related to not draining all the pods in a namespace when an empty selector, i.e., "{}," is specified in a Pod Disruption Budget (PDB). ([#119732](https://github.com/kubernetes/kubernetes/pull/119732), [@sairameshv](https://github.com/sairameshv)) 1123 - Fixed an issue where `StatefulSet` might not restart a pod after eviction or node failure. ([#120398](https://github.com/kubernetes/kubernetes/pull/120398), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) 1124 - Fixed an issue where a `CronJob` could fail to clean up Jobs when the `ResourceQuota` for `Jobs` had been reached. ([#119776](https://github.com/kubernetes/kubernetes/pull/119776), [@ASverdlov](https://github.com/ASverdlov)) 1125 - Fixed an issue where a `StatefulSet` might not restart a pod after eviction or node failure. ([#121389](https://github.com/kubernetes/kubernetes/pull/121389), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) 1126 - Fixed an issue with the `garbagecollection` controller registering duplicate event handlers if discovery requests failed. ([#117992](https://github.com/kubernetes/kubernetes/pull/117992), [@liggitt](https://github.com/liggitt)) 1127 - Fixed attaching volumes after detach errors. Now volumes that failed to detach are not treated as attached. Kubernetes will make sure they are fully attached before they can be used by pods. ([#120595](https://github.com/kubernetes/kubernetes/pull/120595), [@jsafrane](https://github.com/jsafrane)) 1128 - Fixed bug that kubelet resource metric `container_start_time_seconds` had timestamp equal to container start time. ([#120518](https://github.com/kubernetes/kubernetes/pull/120518), [@saschagrunert](https://github.com/saschagrunert)) [SIG Instrumentation, Node and Testing] 1129 - Fixed inconsistency in the calculation of number of nodes that have an image, which affect the scoring in the `ImageLocality` plugin. ([#116938](https://github.com/kubernetes/kubernetes/pull/116938), [@olderTaoist](https://github.com/olderTaoist)) 1130 - Fixed issue with incremental id generation for `loadbalancer` and `endpoint` in `kubeproxy` mock test framework. ([#120723](https://github.com/kubernetes/kubernetes/pull/120723), [@princepereira](https://github.com/princepereira)) 1131 - Fixed panic in Job controller when `podRecreationPolicy: Failed` is used, and the number of terminating pods exceeds parallelism. ([#121147](https://github.com/kubernetes/kubernetes/pull/121147), [@kannon92](https://github.com/kannon92)) 1132 - Fixed regression with adding aggregated `APIservices` panicking and affected health check introduced in release `v1.28.0`. ([#120814](https://github.com/kubernetes/kubernetes/pull/120814), [@Jefftree](https://github.com/Jefftree)) 1133 - Fixed some invalid and unimportant log calls. ([#121249](https://github.com/kubernetes/kubernetes/pull/121249), [@pohly](https://github.com/pohly)) [SIG Cloud Provider, Cluster Lifecycle and Testing] 1134 - Fixed stale SMB mount issue when SMB file share is deleted and then unmounted. ([#121851](https://github.com/kubernetes/kubernetes/pull/121851), [@andyzhangx](https://github.com/andyzhangx)) 1135 - Fixed the bug where images that were pinned by the container runtime could be garbage collected by `kubelet`. ([#119986](https://github.com/kubernetes/kubernetes/pull/119986), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) 1136 - Fixed the bug where kubelet couldn't output logs after log file rotated when `kubectl logs POD_NAME -f` is running. ([#115702](https://github.com/kubernetes/kubernetes/pull/115702), [@xyz-li](https://github.com/xyz-li)) 1137 - Fixed the calculation of the requeue time in the cronjob controller, resulting in proper handling of failed/stuck jobs. ([#121327](https://github.com/kubernetes/kubernetes/pull/121327), [@soltysh](https://github.com/soltysh)) 1138 - Fixed the issue where pod with ordinal number lower than the rolling partitioning number was being deleted. It was coming up with updated image. ([#120731](https://github.com/kubernetes/kubernetes/pull/120731), [@adilGhaffarDev](https://github.com/adilGhaffarDev)) 1139 - Fixed tracking of terminating Pods in the Job status. The field was not updated unless there were other changes to apply. ([#121342](https://github.com/kubernetes/kubernetes/pull/121342), [@dejanzele](https://github.com/dejanzele)) 1140 - Forbidden sysctls for pod sharing the respective namespaces with the host are now checked when creating or updating pods without such sysctls. ([#118705](https://github.com/kubernetes/kubernetes/pull/118705), [@pacoxu](https://github.com/pacoxu)) 1141 - If a watch with the `progressNotify` option set is to be created, and the registry hasn't provided a `newFunc`, return an error. ([#120212](https://github.com/kubernetes/kubernetes/pull/120212), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery] 1142 - Improved handling of jsonpath expressions for `kubectl wait --for`. It is now possible to use simple filter expressions which match on a field's content. ([#118748](https://github.com/kubernetes/kubernetes/pull/118748), [@andreaskaris](https://github.com/andreaskaris)) 1143 - In the `wait.PollUntilContextTimeout` function, if `immediate` is true, the condition will now be invoked before waiting, guaranteeing that the condition is invoked at least once and then wait a interval before executing again. ([#119762](https://github.com/kubernetes/kubernetes/pull/119762), [@AxeZhan](https://github.com/AxeZhan)) 1144 - Incorporating feedback on PR #119341 ([#120087](https://github.com/kubernetes/kubernetes/pull/120087), [@divyasri537](https://github.com/divyasri537)) [SIG API Machinery] 1145 - KCCM: fixed transient node addition and removal caused by #121090 while syncing load balancers on large clusters with a lot of churn. ([#121091](https://github.com/kubernetes/kubernetes/pull/121091), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) 1146 - Kubeadm: changed the "system:masters" Group in the apiserver-kubelet-client.crt certificate Subject to be "kubeadm:cluster-admins" which is a less privileged Group. ([#121837](https://github.com/kubernetes/kubernetes/pull/121837), [@neolit123](https://github.com/neolit123)) 1147 - Metric buckets for `pod_start_duration_seconds` were changed to `{0.5, 1, 2, 3, 4, 5, 6, 8, 10, 20, 30, 45, 60, 120, 180, 240, 300, 360, 480, 600, 900, 1200, 1800, 2700, 3600}`. ([#120680](https://github.com/kubernetes/kubernetes/pull/120680), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) 1148 - Mitigated http/2 DOS vulnerabilities for `CVE-2023-44487` and `CVE-2023-39325` for the API server when the client is unauthenticated. The mitigation may be disabled by setting the `UnauthenticatedHTTP2DOSMitigation` feature gate to `false` (it is enabled by default). An API server fronted by an L7 load balancer that already mitigates these http/2 attacks may choose to disable the kube-apiserver mitigation to avoid disrupting load balancer -> kube-apiserver connections if http/2 requests from multiple clients share the same backend connection. An API server on a private network may opt to disable the kube-apiserver mitigation to prevent performance regressions for unauthenticated clients. Authenticated requests rely on the fix in golang.org/x/net `v0.17.0` alone. https://issue.k8s.io/121197 tracks further mitigation of http/2 attacks by authenticated clients. ([#121120](https://github.com/kubernetes/kubernetes/pull/121120), [@enj](https://github.com/enj)) 1149 - No-op and GC related updates to cluster trust bundles no longer require attest authorization when the `ClusterTrustBundleAttest` plugin is enabled. ([#120779](https://github.com/kubernetes/kubernetes/pull/120779), [@enj](https://github.com/enj)) 1150 - Registered metric `apiserver_request_body_size_bytes` to track the size distribution of requests by `resource` and `verb`. ([#120474](https://github.com/kubernetes/kubernetes/pull/120474), [@YaoC](https://github.com/YaoC)) [SIG API Machinery and Instrumentation] 1151 - Revised the logic for `DaemonSet` rolling update to exclude nodes if scheduling constraints are not met. This eliminates the problem of rolling updates to a `DaemonSet` getting stuck around tolerations. ([#119317](https://github.com/kubernetes/kubernetes/pull/119317), [@mochizuki875](https://github.com/mochizuki875)) 1152 - Scheduler: in 1.29 pre-releases, enabling contextual logging slowed down pod scheduling. ([#121715](https://github.com/kubernetes/kubernetes/pull/121715), [@pohly](https://github.com/pohly)) [SIG Instrumentation and Scheduling] 1153 - Service Controller: will now update load balancer hosts after node's `ProviderID` is 1154 updated. ([#120492](https://github.com/kubernetes/kubernetes/pull/120492), [@cezarygerard](https://github.com/cezarygerard)) 1155 - Setting the `status.loadBalancer` of a Service whose `spec.type` is not `LoadBalancer` was previously allowed, but any update to the `metadata` or `spec` would wipe that field. Setting this field is no longer permitted unless `spec.type` is `LoadBalancer`. In the very unlikely event that this has unexpected impact, you can enable the `AllowServiceLBStatusOnNonLB` feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you need it. ([#119789](https://github.com/kubernetes/kubernetes/pull/119789), [@thockin](https://github.com/thockin)) 1156 - The `--bind-address` parameter in kube-proxy is misleading, no port is opened with this address. Instead it is translated internally to "nodeIP". The nodeIPs for both families are now taken from the Node object if `--bind-address` is unspecified or set to the "any" address (0.0.0.0 or ::). It is recommended to leave `--bind-address` unspecified, and in particular avoid to set it to localhost (127.0.0.1 or ::1) ([#119525](https://github.com/kubernetes/kubernetes/pull/119525), [@uablrek](https://github.com/uablrek)) [SIG Network and Scalability] 1157 - Updated `kube-openapi` to remove invalid defaults: OpenAPI spec no longer includes default of `{}` for certain fields where it did not make sense. ([#120757](https://github.com/kubernetes/kubernetes/pull/120757), [@alexzielenski](https://github.com/alexzielenski)) 1158 - Updated the CRI-O socket path, so users who configure kubelet to use a location like `/run/crio/crio.sock` don't see strange behaviour from CRI stats provider. ([#118704](https://github.com/kubernetes/kubernetes/pull/118704), [@dgl](https://github.com/dgl)) 1159 - Volume attach or publish operation will not fail at `kubelet` if target path directory already exists on the node. ([#119735](https://github.com/kubernetes/kubernetes/pull/119735), [@akankshapanse](https://github.com/akankshapanse)) 1160 - `cluster-bootstrap`: improved the security of the functions responsible for generation and validation of bootstrap tokens. ([#120400](https://github.com/kubernetes/kubernetes/pull/120400), [@neolit123](https://github.com/neolit123)) 1161 - `etcd`: updated to `v3.5.10`. ([#121566](https://github.com/kubernetes/kubernetes/pull/121566), [@mzaian](https://github.com/mzaian)) 1162 - `k8s.io/dynamic-resource-allocation/controller:` `UnsuitableNodes` can now handle a mix of allocated and unallocated claims correctly. ([#120338](https://github.com/kubernetes/kubernetes/pull/120338), [@pohly](https://github.com/pohly)) 1163 - `k8s.io/dynamic-resource-allocation/controller`: `ResourceClaimParameters` and `ResourceClassParameters` validation errors are now visible on `ResourceClaim`, `ResourceClass` and `Pod`. ([#121065](https://github.com/kubernetes/kubernetes/pull/121065), [@byako](https://github.com/byako)) 1164 - `k8s.io/dynamic-resource-allocation`: can now handle a `selected` node which isn't listed 1165 as `potential` node. ([#120871](https://github.com/kubernetes/kubernetes/pull/120871), [@pohly](https://github.com/pohly)) 1166 - `kube-proxy` now reports its health more accurately in dual-stack clusters when there are problems with only one IP family. ([#118146](https://github.com/kubernetes/kubernetes/pull/118146), [@aroradaman](https://github.com/aroradaman)) 1167 - `kubeadm`: Fixed the bug where it always did CRI detection when `--config` was passed, even if it is not required by the subcommand. ([#120828](https://github.com/kubernetes/kubernetes/pull/120828), [@SataQiu](https://github.com/SataQiu)) 1168 - `kubeadm`: fixed `nil` pointer when `etcd` member is already removed. ([#119753](https://github.com/kubernetes/kubernetes/pull/119753), [@pacoxu](https://github.com/pacoxu)) 1169 - `kubeadm`: fixed the bug where `--image-repository` flag is missing for some init 1170 phase sub-commands. ([#120072](https://github.com/kubernetes/kubernetes/pull/120072), [@SataQiu](https://github.com/SataQiu)) 1171 - `kubeadm`: improved the logic that checks whether a `systemd` service exists. ([#120514](https://github.com/kubernetes/kubernetes/pull/120514), [@fengxsong](https://github.com/fengxsong)) 1172 - `kubeadm`: will now use universal deserializer to decode static pod. ([#120549](https://github.com/kubernetes/kubernetes/pull/120549), [@pacoxu](https://github.com/pacoxu)) 1173 - `kubectl prune v2`: Switched annotation from `contains-group-resources` to `contains-group-kinds`, 1174 because this is what we defined in the KEP and is clearer to end-users. Although the functionality is 1175 in `alpha`, we will recognize the prior annotation. This migration support will be removed in `beta`/`GA`. ([#118942](https://github.com/kubernetes/kubernetes/pull/118942), [@justinsb](https://github.com/justinsb)) 1176 - `kubectl` will not print events if `--show-events=false` argument is passed to 1177 describe PVC subcommand. ([#120380](https://github.com/kubernetes/kubernetes/pull/120380), [@MaGaroo](https://github.com/MaGaroo)) 1178 - `scheduler`: Fixed missing field `apiVersion` from events reported by the taint 1179 manager. ([#114095](https://github.com/kubernetes/kubernetes/pull/114095), [@aimuz](https://github.com/aimuz)) 1180 1181 ### Other (Cleanup or Flake) 1182 1183 - Added automatic download of the CNI binary in `local-up-cluster.sh`, facilitating local debugging. ([#120312](https://github.com/kubernetes/kubernetes/pull/120312), [@HirazawaUi](https://github.com/HirazawaUi)) 1184 - Added context to `caches populated` log messages. ([#119796](https://github.com/kubernetes/kubernetes/pull/119796), [@sttts](https://github.com/sttts)) 1185 - Changed behavior of `kube-proxy` by allowing to set `sysctl` values lower than the existing one. ([#120448](https://github.com/kubernetes/kubernetes/pull/120448), [@aroradaman](https://github.com/aroradaman)) 1186 - Cleaned up `kube-apiserver` HTTP logs for impersonated requests. ([#119795](https://github.com/kubernetes/kubernetes/pull/119795), [@sttts](https://github.com/sttts)) 1187 - Deprecated the `--cloud-provider` and `--cloud-config` CLI parameters in kube-apiserver. 1188 These parameters will be removed in a future release. ([#120903](https://github.com/kubernetes/kubernetes/pull/120903), [@dims](https://github.com/dims)) [SIG API Machinery] 1189 - Dynamic resource allocation: will now avoid creating a new gRPC connection for every call of prepare/unprepare resource(s). ([#118619](https://github.com/kubernetes/kubernetes/pull/118619), [@TommyStarK](https://github.com/TommyStarK)) 1190 - E2E storage tests: setting test tags like `[Slow]` via the `DriverInfo.FeatureTag` field is no longer supported. ([#121391](https://github.com/kubernetes/kubernetes/pull/121391), [@pohly](https://github.com/pohly)) 1191 - Fixed an issue where the `vsphere` cloud provider would not trust a certificate if: 1192 - The issuer of the certificate was unknown (`x509.UnknownAuthorityError`) 1193 - The requested name did not match the set of authorized names (`x509.HostnameError`) 1194 - The error surfaced after attempting a connection contained one of the substrings: "certificate is not trusted" or "certificate signed by unknown authority". ([#120736](https://github.com/kubernetes/kubernetes/pull/120736), [@MadhavJivrajani](https://github.com/MadhavJivrajani)) 1195 - Fixed bug where `Adding GroupVersion` log line was constantly repeated without any group version changes. ([#119825](https://github.com/kubernetes/kubernetes/pull/119825), [@Jefftree](https://github.com/Jefftree)) 1196 - Generated `ResourceClaim` names are now more readable because of an additional hyphen before the random suffix (`<pod name>-<claim name>-<random suffix>`). ([#120336](https://github.com/kubernetes/kubernetes/pull/120336), [@pohly](https://github.com/pohly)) 1197 - Graduated `JobReadyPods` to `stable`. The feature gate can no longer be disabled. ([#121302](https://github.com/kubernetes/kubernetes/pull/121302), [@stuton](https://github.com/stuton)) 1198 - Improved memory usage of `kube-controller-manager` by dropping the `.metadata.managedFields` field that `kube-controller-manager` doesn't require. ([#118455](https://github.com/kubernetes/kubernetes/pull/118455), [@linxiulei](https://github.com/linxiulei)) 1199 - Lower and upper case feature flag values are now allowed, but the name still has to match. ([#121441](https://github.com/kubernetes/kubernetes/pull/121441), [@soltysh](https://github.com/soltysh)) 1200 - Makefile and scripts now respect `GOTOOLCHAIN` and otherwise ensure `./.go-version` is used. ([#120279](https://github.com/kubernetes/kubernetes/pull/120279), [@BenTheElder](https://github.com/BenTheElder)) 1201 - Migrated the remainder of the scheduler to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#120933](https://github.com/kubernetes/kubernetes/pull/120933), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation, Scheduling and Testing] 1202 - Optimized `NodeUnschedulable` Filter to avoid unnecessary calculations. ([#119399](https://github.com/kubernetes/kubernetes/pull/119399), [@wackxu](https://github.com/wackxu)) 1203 - Previous versions of Kubernetes on Google Cloud required that workloads (e.g. Deployments, DaemonSets, etc.) which used `PersistentDisk` volumes were using them in read-only mode. This validation provided very little value at relatively host implementation cost, and will no longer be validated. If this is a problem for a specific use-case, please set the `SkipReadOnlyValidationGCE` gate to false to re-enable the validation, and file a Kubernetes bug with details. ([#121083](https://github.com/kubernetes/kubernetes/pull/121083), [@thockin](https://github.com/thockin)) 1204 - Previously, the pod name and namespace were eliminated in the event log message. This PR attempts to add the preemptor pod UID in the preemption event message logs for easier debugging and safer transparency. ([#119971](https://github.com/kubernetes/kubernetes/pull/119971), [@kwakubiney](https://github.com/kwakubiney)) [SIG Scheduling] 1205 - Promoted to conformance a test that verified that `Services` only forward traffic on the port and protocol specified. ([#120069](https://github.com/kubernetes/kubernetes/pull/120069), [@aojea](https://github.com/aojea)) 1206 - Removed `GA` feature gate about `CSIMigrationvSphere`. ([#121291](https://github.com/kubernetes/kubernetes/pull/121291), [@bzsuni](https://github.com/bzsuni)) 1207 - Removed `GA` feature gate about `ProbeTerminationGracePeriod`. ([#121257](https://github.com/kubernetes/kubernetes/pull/121257), [@bzsuni](https://github.com/bzsuni)) 1208 - Removed `GA` feature gate for `JobTrackingWithFinalizers` in `v1.28`. ([#119100](https://github.com/kubernetes/kubernetes/pull/119100), [@bzsuni](https://github.com/bzsuni)) 1209 - Removed `GA`ed feature gate `TopologyManager`. ([#121252](https://github.com/kubernetes/kubernetes/pull/121252), [@tukwila](https://github.com/tukwila)) 1210 - Removed `GA`ed feature gates `OpenAPIV3`. ([#121255](https://github.com/kubernetes/kubernetes/pull/121255), [@tukwila](https://github.com/tukwila)) 1211 - Removed `GA`ed feature gates `SeccompDefault`. ([#121246](https://github.com/kubernetes/kubernetes/pull/121246), [@tukwila](https://github.com/tukwila)) 1212 - Removed ephemeral container legacy server support for the server versions prior to `1.22`. ([#119537](https://github.com/kubernetes/kubernetes/pull/119537), [@ardaguclu](https://github.com/ardaguclu)) 1213 - Removed the `CronJobTimeZone` feature gate (the feature is stable and always enabled) 1214 - Removed the `JobMutableNodeSchedulingDirectives` feature gate (the feature is stable and always enabled) 1215 - Removed the `LegacyServiceAccountTokenNoAutoGeneration` feature gate (the feature is stable and always enabled) ([#120192](https://github.com/kubernetes/kubernetes/pull/120192), [@SataQiu](https://github.com/SataQiu)) [SIG Apps, Auth and Scheduling] 1216 - Removed the `DownwardAPIHugePages` feature gate (the feature is stable and always enabled) ([#120249](https://github.com/kubernetes/kubernetes/pull/120249), [@pacoxu](https://github.com/pacoxu)) [SIG Apps and Node] 1217 - Removed the `GRPCContainerProbe` feature gate (the feature is stable and always enabled). ([#120248](https://github.com/kubernetes/kubernetes/pull/120248), [@pacoxu](https://github.com/pacoxu)) 1218 - Renamed `apiserver_request_body_sizes` metric to `apiserver_request_body_size_bytes`. ([#120503](https://github.com/kubernetes/kubernetes/pull/120503), [@dgrisonnet](https://github.com/dgrisonnet)) 1219 - Set the resolution for the `job_controller_job_sync_duration_seconds` metric from `4ms` to `1min`. ([#120577](https://github.com/kubernetes/kubernetes/pull/120577), [@alculquicondor](https://github.com/alculquicondor)) 1220 - The `horizontalpodautoscaling` and `clusterrole-aggregation` controllers now assume the `autoscaling/v1` and `rbac.authorization.k8s.io/v1` APIs are available. If you disable those APIs and do not want to run those controllers, exclude them by passing `--controllers=-horizontalpodautoscaling` or `--controllers=-clusterrole-aggregation` to `kube-controller-manager`. ([#117977](https://github.com/kubernetes/kubernetes/pull/117977), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Cloud Provider] 1221 - The metrics controlled by the `ComponentSLIs` feature-gate and served at `/metrics/slis` are now GA and unconditionally enabled. The feature-gate will be removed in `v1.31`. ([#120574](https://github.com/kubernetes/kubernetes/pull/120574), [@logicalhan](https://github.com/logicalhan)) 1222 - Updated CNI plugins to `v1.3.0`. ([#119969](https://github.com/kubernetes/kubernetes/pull/119969), [@saschagrunert](https://github.com/saschagrunert)) 1223 - Updated `cri-tools` to `v1.28.0`. ([#119933](https://github.com/kubernetes/kubernetes/pull/119933), [@saschagrunert](https://github.com/saschagrunert)) 1224 - Updated `distroless-iptables` to use `registry.k8s.io/build-image/distroless-iptables:v0.3.1`. ([#120352](https://github.com/kubernetes/kubernetes/pull/120352), [@saschagrunert](https://github.com/saschagrunert)) 1225 - Updated runc to `1.1.10`. ([#121739](https://github.com/kubernetes/kubernetes/pull/121739), [@ty-dc](https://github.com/ty-dc)) 1226 - Upgraded `coredns` to `v1.11.1`. ([#120116](https://github.com/kubernetes/kubernetes/pull/120116), [@tukwila](https://github.com/tukwila)) 1227 - `EnqueueExtensions` from plugins other than `PreEnqueue`, `PreFilter`, `Filter`, `Reserve` and `Permit` are now ignored. 1228 It reduces the number of kinds of cluster events the scheduler needs to subscribe/handle. ([#121571](https://github.com/kubernetes/kubernetes/pull/121571), [@sanposhiho](https://github.com/sanposhiho)) 1229 - `GetPodQOS(pod *core.Pod)` function now returns the stored value from `PodStatus.QOSClass`, if set. To compute/evaluate the value of `QOSClass` from scratch, `ComputePodQOS(pod*core.Pod)` must be used. ([#119665](https://github.com/kubernetes/kubernetes/pull/119665), [@vinaykul](https://github.com/vinaykul)) 1230 - `RetroactiveDefaultStorageClass` feature gate that graduated to GA in `v1.28` and was unconditionally enabled has been removed in `v1.29`. ([#120861](https://github.com/kubernetes/kubernetes/pull/120861), [@RomanBednar](https://github.com/RomanBednar)) 1231 - `Statefulset` now waits for new replicas in tests when removing `.start.ordinal`. ([#119761](https://github.com/kubernetes/kubernetes/pull/119761), [@soltysh](https://github.com/soltysh)) 1232 - `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` objects are 1233 persisted in `etcd` using the `v1beta1` version. Either remove alpha objects, or disable the 1234 alpha `ValidatingAdmissionPolicy` feature in a `v1.27` server before upgrading to a 1235 `v1.28` server with the beta feature and API enabled. ([#120018](https://github.com/kubernetes/kubernetes/pull/120018), [@liggitt](https://github.com/liggitt)) 1236 - `client-go`: `k8s.io/client-go/tools` events and record packages now have new APIs for specifying a context and logger. ([#120729](https://github.com/kubernetes/kubernetes/pull/120729), [@pohly](https://github.com/pohly)) 1237 - `kube-controller-manager` help now includes controllers behind a feature gate in `--controllers` flag. ([#120371](https://github.com/kubernetes/kubernetes/pull/120371), [@atiratree](https://github.com/atiratree)) 1238 - `kubeadm`: removed `system:masters` organization from `apiserver-etcd-client` 1239 certificate. ([#120521](https://github.com/kubernetes/kubernetes/pull/120521), [@SataQiu](https://github.com/SataQiu)) 1240 - `kubeadm`: removed leftover disclaimer that could be seen in the `kubeadm init phase certs` command help screen, since the "certs" phase of "init" is no longer alpha. ([#121172](https://github.com/kubernetes/kubernetes/pull/121172), [@SataQiu](https://github.com/SataQiu)) 1241 - `kubeadm`: updated warning message when swap space is detected. When swap is 1242 active on Linux, `kubeadm` explains that swap is supported for cgroup v2 only and 1243 is beta but disabled by default. ([#120198](https://github.com/kubernetes/kubernetes/pull/120198), [@pacoxu](https://github.com/pacoxu)) 1244 - `kubectl` will not support the `/swagger-2.0.0.pb-v1` endpoint that has been long deprecated. ([#119410](https://github.com/kubernetes/kubernetes/pull/119410), [@Jefftree](https://github.com/Jefftree)) 1245 - `scheduler`: handling of unschedulable pods because a `ResourceClass` is missing 1246 is a bit more efficient and no longer relies on periodic retries. ([#120213](https://github.com/kubernetes/kubernetes/pull/120213), [@pohly](https://github.com/pohly)) 1247 1248 ## Dependencies 1249 1250 ### Added 1251 - cloud.google.com/go/dataproc/v2: v2.0.1 1252 - github.com/danwinship/knftables: [v0.0.13](https://github.com/danwinship/knftables/tree/v0.0.13) 1253 - github.com/distribution/reference: [v0.5.0](https://github.com/distribution/reference/tree/v0.5.0) 1254 - github.com/google/s2a-go: [v0.1.7](https://github.com/google/s2a-go/tree/v0.1.7) 1255 - google.golang.org/genproto/googleapis/bytestream: e85fd2c 1256 1257 ### Changed 1258 - cloud.google.com/go/accessapproval: v1.6.0 → v1.7.1 1259 - cloud.google.com/go/accesscontextmanager: v1.7.0 → v1.8.1 1260 - cloud.google.com/go/aiplatform: v1.37.0 → v1.48.0 1261 - cloud.google.com/go/analytics: v0.19.0 → v0.21.3 1262 - cloud.google.com/go/apigateway: v1.5.0 → v1.6.1 1263 - cloud.google.com/go/apigeeconnect: v1.5.0 → v1.6.1 1264 - cloud.google.com/go/apigeeregistry: v0.6.0 → v0.7.1 1265 - cloud.google.com/go/appengine: v1.7.1 → v1.8.1 1266 - cloud.google.com/go/area120: v0.7.1 → v0.8.1 1267 - cloud.google.com/go/artifactregistry: v1.13.0 → v1.14.1 1268 - cloud.google.com/go/asset: v1.13.0 → v1.14.1 1269 - cloud.google.com/go/assuredworkloads: v1.10.0 → v1.11.1 1270 - cloud.google.com/go/automl: v1.12.0 → v1.13.1 1271 - cloud.google.com/go/baremetalsolution: v0.5.0 → v1.1.1 1272 - cloud.google.com/go/batch: v0.7.0 → v1.3.1 1273 - cloud.google.com/go/beyondcorp: v0.5.0 → v1.0.0 1274 - cloud.google.com/go/bigquery: v1.50.0 → v1.53.0 1275 - cloud.google.com/go/billing: v1.13.0 → v1.16.0 1276 - cloud.google.com/go/binaryauthorization: v1.5.0 → v1.6.1 1277 - cloud.google.com/go/certificatemanager: v1.6.0 → v1.7.1 1278 - cloud.google.com/go/channel: v1.12.0 → v1.16.0 1279 - cloud.google.com/go/cloudbuild: v1.9.0 → v1.13.0 1280 - cloud.google.com/go/clouddms: v1.5.0 → v1.6.1 1281 - cloud.google.com/go/cloudtasks: v1.10.0 → v1.12.1 1282 - cloud.google.com/go/compute: v1.19.0 → v1.23.0 1283 - cloud.google.com/go/contactcenterinsights: v1.6.0 → v1.10.0 1284 - cloud.google.com/go/container: v1.15.0 → v1.24.0 1285 - cloud.google.com/go/containeranalysis: v0.9.0 → v0.10.1 1286 - cloud.google.com/go/datacatalog: v1.13.0 → v1.16.0 1287 - cloud.google.com/go/dataflow: v0.8.0 → v0.9.1 1288 - cloud.google.com/go/dataform: v0.7.0 → v0.8.1 1289 - cloud.google.com/go/datafusion: v1.6.0 → v1.7.1 1290 - cloud.google.com/go/datalabeling: v0.7.0 → v0.8.1 1291 - cloud.google.com/go/dataplex: v1.6.0 → v1.9.0 1292 - cloud.google.com/go/dataqna: v0.7.0 → v0.8.1 1293 - cloud.google.com/go/datastore: v1.11.0 → v1.13.0 1294 - cloud.google.com/go/datastream: v1.7.0 → v1.10.0 1295 - cloud.google.com/go/deploy: v1.8.0 → v1.13.0 1296 - cloud.google.com/go/dialogflow: v1.32.0 → v1.40.0 1297 - cloud.google.com/go/dlp: v1.9.0 → v1.10.1 1298 - cloud.google.com/go/documentai: v1.18.0 → v1.22.0 1299 - cloud.google.com/go/domains: v0.8.0 → v0.9.1 1300 - cloud.google.com/go/edgecontainer: v1.0.0 → v1.1.1 1301 - cloud.google.com/go/essentialcontacts: v1.5.0 → v1.6.2 1302 - cloud.google.com/go/eventarc: v1.11.0 → v1.13.0 1303 - cloud.google.com/go/filestore: v1.6.0 → v1.7.1 1304 - cloud.google.com/go/firestore: v1.9.0 → v1.11.0 1305 - cloud.google.com/go/functions: v1.13.0 → v1.15.1 1306 - cloud.google.com/go/gkebackup: v0.4.0 → v1.3.0 1307 - cloud.google.com/go/gkeconnect: v0.7.0 → v0.8.1 1308 - cloud.google.com/go/gkehub: v0.12.0 → v0.14.1 1309 - cloud.google.com/go/gkemulticloud: v0.5.0 → v1.0.0 1310 - cloud.google.com/go/gsuiteaddons: v1.5.0 → v1.6.1 1311 - cloud.google.com/go/iam: v0.13.0 → v1.1.1 1312 - cloud.google.com/go/iap: v1.7.1 → v1.8.1 1313 - cloud.google.com/go/ids: v1.3.0 → v1.4.1 1314 - cloud.google.com/go/iot: v1.6.0 → v1.7.1 1315 - cloud.google.com/go/kms: v1.10.1 → v1.15.0 1316 - cloud.google.com/go/language: v1.9.0 → v1.10.1 1317 - cloud.google.com/go/lifesciences: v0.8.0 → v0.9.1 1318 - cloud.google.com/go/longrunning: v0.4.1 → v0.5.1 1319 - cloud.google.com/go/managedidentities: v1.5.0 → v1.6.1 1320 - cloud.google.com/go/maps: v0.7.0 → v1.4.0 1321 - cloud.google.com/go/mediatranslation: v0.7.0 → v0.8.1 1322 - cloud.google.com/go/memcache: v1.9.0 → v1.10.1 1323 - cloud.google.com/go/metastore: v1.10.0 → v1.12.0 1324 - cloud.google.com/go/monitoring: v1.13.0 → v1.15.1 1325 - cloud.google.com/go/networkconnectivity: v1.11.0 → v1.12.1 1326 - cloud.google.com/go/networkmanagement: v1.6.0 → v1.8.0 1327 - cloud.google.com/go/networksecurity: v0.8.0 → v0.9.1 1328 - cloud.google.com/go/notebooks: v1.8.0 → v1.9.1 1329 - cloud.google.com/go/optimization: v1.3.1 → v1.4.1 1330 - cloud.google.com/go/orchestration: v1.6.0 → v1.8.1 1331 - cloud.google.com/go/orgpolicy: v1.10.0 → v1.11.1 1332 - cloud.google.com/go/osconfig: v1.11.0 → v1.12.1 1333 - cloud.google.com/go/oslogin: v1.9.0 → v1.10.1 1334 - cloud.google.com/go/phishingprotection: v0.7.0 → v0.8.1 1335 - cloud.google.com/go/policytroubleshooter: v1.6.0 → v1.8.0 1336 - cloud.google.com/go/privatecatalog: v0.8.0 → v0.9.1 1337 - cloud.google.com/go/pubsub: v1.30.0 → v1.33.0 1338 - cloud.google.com/go/pubsublite: v1.7.0 → v1.8.1 1339 - cloud.google.com/go/recaptchaenterprise/v2: v2.7.0 → v2.7.2 1340 - cloud.google.com/go/recommendationengine: v0.7.0 → v0.8.1 1341 - cloud.google.com/go/recommender: v1.9.0 → v1.10.1 1342 - cloud.google.com/go/redis: v1.11.0 → v1.13.1 1343 - cloud.google.com/go/resourcemanager: v1.7.0 → v1.9.1 1344 - cloud.google.com/go/resourcesettings: v1.5.0 → v1.6.1 1345 - cloud.google.com/go/retail: v1.12.0 → v1.14.1 1346 - cloud.google.com/go/run: v0.9.0 → v1.2.0 1347 - cloud.google.com/go/scheduler: v1.9.0 → v1.10.1 1348 - cloud.google.com/go/secretmanager: v1.10.0 → v1.11.1 1349 - cloud.google.com/go/security: v1.13.0 → v1.15.1 1350 - cloud.google.com/go/securitycenter: v1.19.0 → v1.23.0 1351 - cloud.google.com/go/servicedirectory: v1.9.0 → v1.11.0 1352 - cloud.google.com/go/shell: v1.6.0 → v1.7.1 1353 - cloud.google.com/go/spanner: v1.45.0 → v1.47.0 1354 - cloud.google.com/go/speech: v1.15.0 → v1.19.0 1355 - cloud.google.com/go/storagetransfer: v1.8.0 → v1.10.0 1356 - cloud.google.com/go/talent: v1.5.0 → v1.6.2 1357 - cloud.google.com/go/texttospeech: v1.6.0 → v1.7.1 1358 - cloud.google.com/go/tpu: v1.5.0 → v1.6.1 1359 - cloud.google.com/go/trace: v1.9.0 → v1.10.1 1360 - cloud.google.com/go/translate: v1.7.0 → v1.8.2 1361 - cloud.google.com/go/video: v1.15.0 → v1.19.0 1362 - cloud.google.com/go/videointelligence: v1.10.0 → v1.11.1 1363 - cloud.google.com/go/vision/v2: v2.7.0 → v2.7.2 1364 - cloud.google.com/go/vmmigration: v1.6.0 → v1.7.1 1365 - cloud.google.com/go/vmwareengine: v0.3.0 → v1.0.0 1366 - cloud.google.com/go/vpcaccess: v1.6.0 → v1.7.1 1367 - cloud.google.com/go/webrisk: v1.8.0 → v1.9.1 1368 - cloud.google.com/go/websecurityscanner: v1.5.0 → v1.6.1 1369 - cloud.google.com/go/workflows: v1.10.0 → v1.11.1 1370 - cloud.google.com/go: v0.110.0 → v0.110.6 1371 - github.com/alecthomas/template: [fb15b89 → a0175ee](https://github.com/alecthomas/template/compare/fb15b89...a0175ee) 1372 - github.com/cncf/xds/go: [06c439d → e9ce688](https://github.com/cncf/xds/go/compare/06c439d...e9ce688) 1373 - github.com/coredns/corefile-migration: [v1.0.20 → v1.0.21](https://github.com/coredns/corefile-migration/compare/v1.0.20...v1.0.21) 1374 - github.com/cyphar/filepath-securejoin: [v0.2.3 → v0.2.4](https://github.com/cyphar/filepath-securejoin/compare/v0.2.3...v0.2.4) 1375 - github.com/docker/docker: [v20.10.21+incompatible → v20.10.24+incompatible](https://github.com/docker/docker/compare/v20.10.21...v20.10.24) 1376 - github.com/emicklei/go-restful/v3: [v3.9.0 → v3.11.0](https://github.com/emicklei/go-restful/v3/compare/v3.9.0...v3.11.0) 1377 - github.com/envoyproxy/go-control-plane: [v0.10.3 → v0.11.1](https://github.com/envoyproxy/go-control-plane/compare/v0.10.3...v0.11.1) 1378 - github.com/envoyproxy/protoc-gen-validate: [v0.9.1 → v1.0.2](https://github.com/envoyproxy/protoc-gen-validate/compare/v0.9.1...v1.0.2) 1379 - github.com/evanphx/json-patch: [v5.6.0+incompatible → v4.12.0+incompatible](https://github.com/evanphx/json-patch/compare/v5.6.0...v4.12.0) 1380 - github.com/fsnotify/fsnotify: [v1.6.0 → v1.7.0](https://github.com/fsnotify/fsnotify/compare/v1.6.0...v1.7.0) 1381 - github.com/go-logr/logr: [v1.2.4 → v1.3.0](https://github.com/go-logr/logr/compare/v1.2.4...v1.3.0) 1382 - github.com/godbus/dbus/v5: [v5.0.6 → v5.1.0](https://github.com/godbus/dbus/v5/compare/v5.0.6...v5.1.0) 1383 - github.com/golang/glog: [v1.0.0 → v1.1.0](https://github.com/golang/glog/compare/v1.0.0...v1.1.0) 1384 - github.com/google/cadvisor: [v0.47.3 → v0.48.1](https://github.com/google/cadvisor/compare/v0.47.3...v0.48.1) 1385 - github.com/google/cel-go: [v0.16.0 → v0.17.7](https://github.com/google/cel-go/compare/v0.16.0...v0.17.7) 1386 - github.com/google/go-cmp: [v0.5.9 → v0.6.0](https://github.com/google/go-cmp/compare/v0.5.9...v0.6.0) 1387 - github.com/googleapis/gax-go/v2: [v2.7.1 → v2.11.0](https://github.com/googleapis/gax-go/v2/compare/v2.7.1...v2.11.0) 1388 - github.com/gorilla/websocket: [v1.4.2 → v1.5.0](https://github.com/gorilla/websocket/compare/v1.4.2...v1.5.0) 1389 - github.com/grpc-ecosystem/grpc-gateway/v2: [v2.7.0 → v2.16.0](https://github.com/grpc-ecosystem/grpc-gateway/v2/compare/v2.7.0...v2.16.0) 1390 - github.com/ishidawataru/sctp: [7c296d4 → 7ff4192](https://github.com/ishidawataru/sctp/compare/7c296d4...7ff4192) 1391 - github.com/konsorten/go-windows-terminal-sequences: [v1.0.3 → v1.0.1](https://github.com/konsorten/go-windows-terminal-sequences/compare/v1.0.3...v1.0.1) 1392 - github.com/mrunalp/fileutils: [v0.5.0 → v0.5.1](https://github.com/mrunalp/fileutils/compare/v0.5.0...v0.5.1) 1393 - github.com/onsi/ginkgo/v2: [v2.9.4 → v2.13.0](https://github.com/onsi/ginkgo/v2/compare/v2.9.4...v2.13.0) 1394 - github.com/onsi/gomega: [v1.27.6 → v1.29.0](https://github.com/onsi/gomega/compare/v1.27.6...v1.29.0) 1395 - github.com/opencontainers/runc: [v1.1.7 → v1.1.10](https://github.com/opencontainers/runc/compare/v1.1.7...v1.1.10) 1396 - github.com/opencontainers/selinux: [v1.10.0 → v1.11.0](https://github.com/opencontainers/selinux/compare/v1.10.0...v1.11.0) 1397 - github.com/spf13/afero: [v1.2.2 → v1.1.2](https://github.com/spf13/afero/compare/v1.2.2...v1.1.2) 1398 - github.com/stretchr/testify: [v1.8.2 → v1.8.4](https://github.com/stretchr/testify/compare/v1.8.2...v1.8.4) 1399 - github.com/vmware/govmomi: [v0.30.0 → v0.30.6](https://github.com/vmware/govmomi/compare/v0.30.0...v0.30.6) 1400 - go.etcd.io/bbolt: v1.3.7 → v1.3.8 1401 - go.etcd.io/etcd/api/v3: v3.5.9 → v3.5.10 1402 - go.etcd.io/etcd/client/pkg/v3: v3.5.9 → v3.5.10 1403 - go.etcd.io/etcd/client/v2: v2.305.9 → v2.305.10 1404 - go.etcd.io/etcd/client/v3: v3.5.9 → v3.5.10 1405 - go.etcd.io/etcd/pkg/v3: v3.5.9 → v3.5.10 1406 - go.etcd.io/etcd/raft/v3: v3.5.9 → v3.5.10 1407 - go.etcd.io/etcd/server/v3: v3.5.9 → v3.5.10 1408 - go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful: v0.35.0 → v0.42.0 1409 - go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc: v0.35.0 → v0.42.0 1410 - go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.35.1 → v0.44.0 1411 - go.opentelemetry.io/contrib/propagators/b3: v1.10.0 → v1.17.0 1412 - go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc: v1.10.0 → v1.19.0 1413 - go.opentelemetry.io/otel/exporters/otlp/otlptrace: v1.10.0 → v1.19.0 1414 - go.opentelemetry.io/otel/metric: v0.31.0 → v1.19.0 1415 - go.opentelemetry.io/otel/sdk: v1.10.0 → v1.19.0 1416 - go.opentelemetry.io/otel/trace: v1.10.0 → v1.19.0 1417 - go.opentelemetry.io/otel: v1.10.0 → v1.19.0 1418 - go.opentelemetry.io/proto/otlp: v0.19.0 → v1.0.0 1419 - golang.org/x/crypto: v0.11.0 → v0.14.0 1420 - golang.org/x/mod: v0.10.0 → v0.12.0 1421 - golang.org/x/net: v0.13.0 → v0.17.0 1422 - golang.org/x/oauth2: v0.8.0 → v0.10.0 1423 - golang.org/x/sync: v0.2.0 → v0.3.0 1424 - golang.org/x/sys: v0.10.0 → v0.13.0 1425 - golang.org/x/term: v0.10.0 → v0.13.0 1426 - golang.org/x/text: v0.11.0 → v0.13.0 1427 - golang.org/x/tools: v0.8.0 → v0.12.0 1428 - google.golang.org/api: v0.114.0 → v0.126.0 1429 - google.golang.org/genproto/googleapis/api: dd9d682 → 23370e0 1430 - google.golang.org/genproto/googleapis/rpc: 28d5490 → b8732ec 1431 - google.golang.org/genproto: 0005af6 → f966b18 1432 - google.golang.org/grpc: v1.54.0 → v1.58.3 1433 - google.golang.org/protobuf: v1.30.0 → v1.31.0 1434 - k8s.io/gengo: c0856e2 → 9cce18d 1435 - k8s.io/klog/v2: v2.100.1 → v2.110.1 1436 - k8s.io/kube-openapi: 2695361 → 2dd684a 1437 - k8s.io/utils: d93618c → 3b25d92 1438 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.1.2 → v0.28.0 1439 - sigs.k8s.io/structured-merge-diff/v4: v4.2.3 → v4.4.1 1440 1441 ### Removed 1442 - cloud.google.com/go/dataproc: v1.12.0 1443 - cloud.google.com/go/gaming: v1.9.0 1444 - github.com/blang/semver: [v3.5.1+incompatible](https://github.com/blang/semver/tree/v3.5.1) 1445 - github.com/jmespath/go-jmespath/internal/testify: [v1.5.1](https://github.com/jmespath/go-jmespath/internal/testify/tree/v1.5.1) 1446 - go.opentelemetry.io/otel/exporters/otlp/internal/retry: v1.10.0 1447 1448 1449 1450 # v1.29.0-rc.2 1451 1452 1453 ## Downloads for v1.29.0-rc.2 1454 1455 1456 1457 ### Source Code 1458 1459 filename | sha512 hash 1460 -------- | ----------- 1461 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes.tar.gz) | daaabe57e2da16a076380072bb0e4a178400f045bb82d44d338efa90641bb4e35b590764d9ab4f365219634149588526da57d1aaabdb1ed805ee0ccd9aed63b6 1462 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-src.tar.gz) | c4a3ea15db8a7d0696f2ef4a2f3d1e65b89a931074043957fa59be2bb0fac04b9967e8eff1037b4c649fcdd34a3ad2b717d129b2ce2f45691675bbef95710833 1463 1464 ### Client Binaries 1465 1466 filename | sha512 hash 1467 -------- | ----------- 1468 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-darwin-amd64.tar.gz) | 3b22ba3aa4f778f0086e739999f81f8ca040f5a5b9b88a8e71c9cd94dc728ee9090f0c388a0020f458a6c13716c614b37e1be11b7e5402cc00380595ce3be21f 1469 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-darwin-arm64.tar.gz) | ed2d2f866f28b5f157b1ca39ae2cb832cc934c2383f8cb90b5e79781a4835579dd99ebe2f34245c4a764a49ebcb343499fded8f180ac1a084acbfa2bfc38ef37 1470 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-linux-386.tar.gz) | cdf10ab26e223742a882d19a36a932a3230a061e026514fd3cbe3d763f7de50372903e04ea9b85c2554980310d8fa8b8c140bb623e491a6c646af18499e14354 1471 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-linux-amd64.tar.gz) | 7aa3da03d393b9da31ec3fd0c5c9694ab3a3e1bc7340108375238a5b132da8955db318559f2d6f7968f141802373fa3462a980932e5bb9d59b553816fc1bc2d4 1472 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-linux-arm.tar.gz) | c24722f8f20f86a842f04e047041c631975ee4a8800da9cee1ff5415055842eb381b7c918b7da2c7421369d718b13e182008e08c2f5f2a5c5a1782c481851ddf 1473 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-linux-arm64.tar.gz) | 1d4df5d1bb6fd5fdb8b9dc3b0bb7c8b7f3c155dab019222d9611706d5140206ac9a7eda6673a2a912e1738f02f8707184e4e711d4f82f5680a8098ce59ad9f74 1474 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-linux-ppc64le.tar.gz) | c7187e7834f690958ea2b3f7aeac987f6efcd76b859487b391972a25daf347c87fc60e9e9b9a440a2c4df529c8f467b94790e6f8992cdbbac87f73d1373f03c0 1475 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-linux-s390x.tar.gz) | e758559d18ef1510d50fa57177e2023ffc4bc19d3a4e302e984bef35701e9a47894e7c7d80589742bd24a337d75b35e83faf1c097202401deee01e0f5fe31829 1476 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-windows-386.tar.gz) | 0ef788e96ea786b0d62de7dcc1315800a4106b367381e02731ca384bc89397aa7a1de5b678a0532df590c1f8d448206a236623ed729bc5c30b0e63317aec6a6c 1477 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-windows-amd64.tar.gz) | 719c3c1f9b7beb199bd9d0a0b5c85d99b76bd462397a3d4b37fe8b5970e30e69e26f30815f97bb9a4049b28c7aaf0b384d1378a14e8273774d65f3549cbd3083 1478 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-client-windows-arm64.tar.gz) | 24916863e604c14939ccd1574f754306215d603ff2efdd4dd00fa667923932aff88f8046fc8a4b7dea47f97ea5bfc53d4193fb293fa48fdefb4cd65c301e32ab 1479 1480 ### Server Binaries 1481 1482 filename | sha512 hash 1483 -------- | ----------- 1484 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-server-linux-amd64.tar.gz) | 88e7a746f69b3070908fc32213d6046a10c5df139be566f42b135fb344f9681f7d527e5d8b65214f3a5abc4564962d4d1f8391d6ce62ca3620e538bb0e836055 1485 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-server-linux-arm64.tar.gz) | 69b9bf487dabf7ae466cb9cf0e569d04908c359dea2772ed2a0daa0c5be2d1389d09cac3eebc7c350af0fd0e0f30e2cfc86c4ede423a0cd8d00f62909e13fac4 1486 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-server-linux-ppc64le.tar.gz) | 5925a756d4ab2be13141391c7c18888f2b7337aa3db05ac0cb0ed25ad66a6d290b38731e389b829571bb5fcb95b6b9f5b4d054058818adad66d1f39e45fa9356 1487 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-server-linux-s390x.tar.gz) | 82e4aa613bffc8658e8a10c53269a7977219fd38b985fb8a4a4df78a8fa876521c1925ab927a74737e3a57c415963c13717720d106b6f086294c717bdc5f02c9 1488 1489 ### Node Binaries 1490 1491 filename | sha512 hash 1492 -------- | ----------- 1493 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-node-linux-amd64.tar.gz) | fd7be4f743a641f22c04d32749825d509b6c9ff095207997db40dc39cb1647bd03f47b71287339a02f303b2903ffe6bd0910bd051f20f44c03e7d853c2e794b5 1494 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-node-linux-arm64.tar.gz) | 12454bf94d86282cbf04db188f504aed6d8c6eeda8f32b0001152aa7ed52c65c1ec0db90c99077b30987d40a4ccd6c707dcda1ce6be84c746f4c972af0b277f4 1495 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-node-linux-ppc64le.tar.gz) | 1eb0d47ca00231df825daf2c39b203c438a7c1b1d9ec1e7bcefd63c16e89771316d9a2cf66c89ad6653e70fed3ccc38a650e83be106f2738c5ae97bbc055de7f 1496 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-node-linux-s390x.tar.gz) | ff7f4da7e71859ebea3a70410575f7ffa52c476f4f6d76958366ca5e4ec7b315fdd4b5b8cbd48faaa800ffcfb64f9af85b6605582480242ae336c3fea84f8733 1497 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.2/kubernetes-node-windows-amd64.tar.gz) | 8fc83a3735d163866e370df3c6bccbe15d2ce478b95bc1fe71b199952db68c95f31d8967514b0930c2ad7bc2734c2a4ab5d8cdc738611365215f7d88a19bc2d6 1498 1499 ### Container Images 1500 1501 All container images are available as manifest lists and support the described 1502 architectures. It is also possible to pull a specific architecture directly by 1503 adding the "-$ARCH" suffix to the container image name. 1504 1505 name | architectures 1506 ---- | ------------- 1507 [registry.k8s.io/conformance:v1.29.0-rc.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 1508 [registry.k8s.io/kube-apiserver:v1.29.0-rc.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 1509 [registry.k8s.io/kube-controller-manager:v1.29.0-rc.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 1510 [registry.k8s.io/kube-proxy:v1.29.0-rc.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 1511 [registry.k8s.io/kube-scheduler:v1.29.0-rc.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 1512 [registry.k8s.io/kubectl:v1.29.0-rc.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 1513 1514 ## Changelog since v1.29.0-rc.1 1515 1516 ## Changes by Kind 1517 1518 ### Feature 1519 1520 - Bump distroless-iptables to v0.4.3 ([#122206](https://github.com/kubernetes/kubernetes/pull/122206), [@xmudrii](https://github.com/xmudrii)) [SIG Release and Testing] 1521 - Kubernetes is now built with Go 1.21.5 ([#122201](https://github.com/kubernetes/kubernetes/pull/122201), [@xmudrii](https://github.com/xmudrii)) [SIG Release and Testing] 1522 1523 ## Dependencies 1524 1525 ### Added 1526 _Nothing has changed._ 1527 1528 ### Changed 1529 _Nothing has changed._ 1530 1531 ### Removed 1532 _Nothing has changed._ 1533 1534 1535 1536 # v1.29.0-rc.1 1537 1538 1539 ## Downloads for v1.29.0-rc.1 1540 1541 1542 1543 ### Source Code 1544 1545 filename | sha512 hash 1546 -------- | ----------- 1547 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes.tar.gz) | e44677de7af6634c31b86672dc6755d97ae145fd0497229c08b156d11dcdbc922f15c715fd878b585d21a6c7dd10fde0b43135f0b6f7e77a9f957f2280a32018 1548 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-src.tar.gz) | 63e197478e315a64dae6282c1e4ce2b672f0a3941bea9920094b703d44a09aaed74228a7c29fbec4051a4cb832f6e791d54f5e76e006aada1216a502c6d2e744 1549 1550 ### Client Binaries 1551 1552 filename | sha512 hash 1553 -------- | ----------- 1554 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-darwin-amd64.tar.gz) | 167d931f6b9540b9fbd8501e3dd9d2ea032ead96f57cf4a929020fff3c4efb0456e3bc5eee2a8436670589ef18b48018dabf57081ff13393542007e9d0c8cb72 1555 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-darwin-arm64.tar.gz) | 6623d8b08b69beea8832a1130c50624f248464a01fec4fce720700ccebdd3ed440af664f5beb49a294557db3c4ff7a8fecfd3cf48f9dabcc48b9bda2d791c08e 1556 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-linux-386.tar.gz) | 11f6c0d6f0954938c4217436536a67713c403b1f3c2b988d26944374fab6f70c385cb9356e6aa51b480fdcd07539de4667be2e72fa8128ba792a607bb388254f 1557 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-linux-amd64.tar.gz) | ee7605531629a2e320f299ef49bfb87566b73245f16dee79a40b824d637bd97ca8bd7f81a177320c2e4bbb82acd7f5d840359dd79c3062c2136e0b5f04eeb90e 1558 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-linux-arm.tar.gz) | baa1af4d932c3d36ff084f2fc4c7676e76a2f0e4c4c746495f932c56a583b4390376b0b631b13f6dc3b03bd874d84c20f82e71d2483940b37ea439bc7d21dadf 1559 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-linux-arm64.tar.gz) | 26a20dfbeca7abb73a73f1cfb5337b4af71bb3d2810d053f9d94e4a3709d282e515a542f02502a7e86adf3b32ac52e7b434d1604fa9664729682d083a55b314d 1560 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-linux-ppc64le.tar.gz) | eca5ca3028b64ea44138b08831e998ac85bd054785099366e295b88b8b568c455a54f4fd110b568208169b9a3aef918c7f6caf8e05f9e73f85c26f973a589e2d 1561 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-linux-s390x.tar.gz) | 5bcde8b36b8dc1d3ed83337322fc260311238e9f067301838c5002bb0dc63153f82c2602d8c9a28c1ae5dd85b0eecc4d3e8c31dcf75f4653c6b9bebd3a564321 1562 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-windows-386.tar.gz) | d183c3183bfac0878377eaa8adf00e6ecdb3f252ed47180b8f9231d757b44c30931620b04da52c3470a1a278fa5a76e99f0b7c587b696f517caec5ff16103480 1563 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-windows-amd64.tar.gz) | 7a51ed89ad5f850bfc94e4175294d944eae9628c281fea2c18939417f84c438b82246d262e645bea9fd257deb60b51c05b1c1ebd321b35aafeb87d1b4f83ebe6 1564 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-client-windows-arm64.tar.gz) | 2efc1fd75461ed5e0bceba78681804567e731a545a801b28e8291d2f3cc8e2c8c22d3e418888a666dc1e40754681acbcbfc64fc81772088b8524fde7c55e7e3a 1565 1566 ### Server Binaries 1567 1568 filename | sha512 hash 1569 -------- | ----------- 1570 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-server-linux-amd64.tar.gz) | 28eeda8ab821891ca445b4a884ae70028146f6d4264e364a7ca88c819ea80bd95653c7f538af6c5f660921f140026f3d61c469afb0109c2f5111245592439fd7 1571 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-server-linux-arm64.tar.gz) | d786685de63f060910181f0de66511fe8f8f0fa66f00ee0c76957246a62e3de3208ec009908c6700ff83dc7c5e5c24f3c1c3118d06b07568fcb004190d1e5fe6 1572 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-server-linux-ppc64le.tar.gz) | b9d14fdcf282b4f9e523d81f71cd82d64bfb9bf9ba4affb7d6dbcfa8191f9b0d238f7091c65c1f1d516c6bcaa13f68affdd3bfda1ec759f35d505284a87494e1 1573 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-server-linux-s390x.tar.gz) | f84c315e3d3ab6e3124106783e43b18d407d5c4ef09910641ae51c034b550fba0581515181ae4d355eea5b75eb0688460a891141c47d13be902e06156908d26d 1574 1575 ### Node Binaries 1576 1577 filename | sha512 hash 1578 -------- | ----------- 1579 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-node-linux-amd64.tar.gz) | b2a974c505878c757bc643c64cf24208ed92bfe7943f6e9e50d215f027e4fc2f4a578d3975721f53ff6a06e6560dec362887a236052213784222695c916a59fb 1580 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-node-linux-arm64.tar.gz) | 7f7b6a3bcdef051ae0126a811d02b327f6f45b13050934dd4d90219a23efd5aeabb945e040a9aa57a24a4151222b558a4e2a996bad5e5f7ef508c26e6cca85ee 1581 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-node-linux-ppc64le.tar.gz) | 6245a8645a6b52c2bd40e1104d4fbba015bf583f50478b2686ecf0c3c8fdc05c6729da76dc6301ca0f2f0d63f1a5df84a0871fb4793fc3aa9e8e2b6cdae37d34 1582 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-node-linux-s390x.tar.gz) | fe530554696b84db43372ca48e95a01c46cf3a09dd51fb569b1792a341827a428bfbb04c3bab59d6aea095687eafe144534a0d98df8852d469e8bdc69a8d2d1a 1583 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.1/kubernetes-node-windows-amd64.tar.gz) | 0a80378b4037f8d325fdbdc065ce5fd841b0b66242d64ec5c6b8ad6af0e430fc020d4fb0d624909b0725761d2c249a8f75c91eae22af03cefe4a4338b57d3d29 1584 1585 ### Container Images 1586 1587 All container images are available as manifest lists and support the described 1588 architectures. It is also possible to pull a specific architecture directly by 1589 adding the "-$ARCH" suffix to the container image name. 1590 1591 name | architectures 1592 ---- | ------------- 1593 [registry.k8s.io/conformance:v1.29.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 1594 [registry.k8s.io/kube-apiserver:v1.29.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 1595 [registry.k8s.io/kube-controller-manager:v1.29.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 1596 [registry.k8s.io/kube-proxy:v1.29.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 1597 [registry.k8s.io/kube-scheduler:v1.29.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 1598 [registry.k8s.io/kubectl:v1.29.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 1599 1600 ## Changelog since v1.29.0-rc.0 1601 1602 ## Dependencies 1603 1604 ### Added 1605 _Nothing has changed._ 1606 1607 ### Changed 1608 _Nothing has changed._ 1609 1610 ### Removed 1611 _Nothing has changed._ 1612 1613 1614 1615 # v1.29.0-rc.0 1616 1617 1618 ## Downloads for v1.29.0-rc.0 1619 1620 1621 1622 ### Source Code 1623 1624 filename | sha512 hash 1625 -------- | ----------- 1626 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes.tar.gz) | 6a5b027a35b96d1cf8495efce0f9f518499b94e63e1d11058876d1b364d0bba42ccedac4612082771eb38cd54be0d8868a808de05c7e9077b8644f15a5c6f413 1627 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-src.tar.gz) | d92897e5e28a14f0fbd3f03e9016e9c86f30bf097c4e709e6dba74b1a9897ce016e3c3a44aed9d5f851af1f5d5bd0ea2240efe8d8d12d7893b7f9cff66caff55 1628 1629 ### Client Binaries 1630 1631 filename | sha512 hash 1632 -------- | ----------- 1633 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-darwin-amd64.tar.gz) | a4e8dd4e65158024a46843701ed24082eefde5d407c6d6a191b7b7f690413ea65c5422ba578e2813cd6624ac7327174554d879dbbbb324b56fbfe99892eb8d80 1634 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-darwin-arm64.tar.gz) | ef14378eaa3a35a34ab5e9b06c9856ff46165bbee2a4efc1b8512de47e8f584449d94155665978eca6264e23f131e31d072f9333117c11a3e92aadeea367b8e5 1635 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-linux-386.tar.gz) | 686a8b69525e8e1494cdc890e8023ba60f86e41ceb28cb5df7e33f152ecc3ac8c62b0b1d24fa6c8198278a9d585bfd8962d058daf7f27dfa658580598b45cafe 1636 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-linux-amd64.tar.gz) | 7ebe8d866f8fd1dccd6761be0ad5096cb861e5fb20bdea0ac65a3d63230d9a7d47df16a6933fcf4069cf819ab90d12eaf87ec53873eacd88c3feab009e85e430 1637 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-linux-arm.tar.gz) | f8a336b48c27819f979336fff3ffa7eeb5512330f3eafe7a3b85ea65a4d94b213ed20d4d35d6fe3f92cb557037a051023eb47fd6e6dcaf3e0a6fe88a5c6cd632 1638 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-linux-arm64.tar.gz) | a72602ac48b13c6a97883c34170fd64095539d4f9a3900367ff628a195aa931c27d7c9582f864c669332bbc58b4883d5e41bc65d5ac83337bdf7066e538deceb 1639 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-linux-ppc64le.tar.gz) | 002b2e685758ad6fa2a18d7706a335249f55a786a4315d3f2cab8e34d38a01302af91063742729de664c7ab06bd656b388166f82010af36e43e931e8ddd93752 1640 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-linux-s390x.tar.gz) | 21da21e1f7ba24b6967b5e22abb62e1c1691cd7cc15eb5ecd9777fa51d788a7a132f31c04306d3a59e5cee96bb58b9d1838630de1bcbf168cabe8f4afb514501 1641 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-windows-386.tar.gz) | 21494c5fe65e6a9aaf2f7f11996219155ed85a4f54d048b64df05de1adbd925af40ee51d4119801333143364902b9805cefceafac8d407f62eef1e7f07b686ee 1642 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-windows-amd64.tar.gz) | eaaddeac2e0a69a618f606574044eec8b41f4c3d4f6cf0045e4456ad57d44c865d1f183b6e0929f6913e28febe67b178662dfce3e40395c6d97180985b4fb48d 1643 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-client-windows-arm64.tar.gz) | dad6a73bf2530c0c2f58b8e77956ec444b6795c9882b0f2b960998fbd9e22720fc6fff114af3b0ad10655e9e1d627f70bc6f67fdd388d0e995aeb9bd4bf9bea2 1644 1645 ### Server Binaries 1646 1647 filename | sha512 hash 1648 -------- | ----------- 1649 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-server-linux-amd64.tar.gz) | c64651213144ef4696fa11da0ec93c6fd7540798bfc28df8e69ee8bdb35dbc7114ee043cd38dc86c75a3dbff5e45ed4474be22ce74b8c4b3206030a10cd20f8d 1650 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-server-linux-arm64.tar.gz) | 0724ce02d551d72f39c7ac6b29c78dcaeae7878126b33cde7a949d0b9be0b35b3977f5494ab48f02a382bf83a70a8ad035f4962b0644a6fedc084068b525ddf9 1651 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-server-linux-ppc64le.tar.gz) | 7f527bb02e046308b2720a99d8f6ac13e1daee23b44e77603b75aa5569a9b4baf29a7b19f3076219ff94f296ce8316fdaadb9cabaf1c58173a7e3719e94f3917 1652 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-server-linux-s390x.tar.gz) | 0285e04f2834bdbb66b46193f54724e6f9264ff992b10dbaa3694abbab297f5e1f4e95ade14f7dcb41f856d9e3a292f1af16f3ed59e2b02961451973d4972f1d 1653 1654 ### Node Binaries 1655 1656 filename | sha512 hash 1657 -------- | ----------- 1658 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-node-linux-amd64.tar.gz) | b4c5e4a0e818eb9f88128e2a051591b4955a858e400489d04b75cdfb68eb3a7d004ced839c2916bf5ca885d7ae496fb68c0620b2c3352cb5435c435756b0a70a 1659 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-node-linux-arm64.tar.gz) | e525decd637860b9621ec7ec8c42913c419bb81577a1a359e752e7628507b6e9b1a82889ef0ba17ca975aa8630edba12a87d38b75ea3f9e213493873036b92c8 1660 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-node-linux-ppc64le.tar.gz) | 4282286b775a5bdaab753c911fa0f351476d89070a34569bedb104cb2c56a408d125d44c4895cc28fcb8cd5c12585f3cbecccfe045880b546766202113a703b1 1661 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-node-linux-s390x.tar.gz) | af88dac8622e10e336e5d79f9d4511de3eceed384da210dda83223c7b6582133acaa7d8f6b361cf213a4ca3ea51379bd828816c991ed7b4c62cf6fd9830f0c30 1662 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-rc.0/kubernetes-node-windows-amd64.tar.gz) | 7b322df6a7e9e0b0b881f99d7ef76b3eda0f856345eb888efa62daf1f3638f88a630fc40626800075db90215c68a956fec7ce274381e06a173d494e4b03b4f49 1663 1664 ### Container Images 1665 1666 All container images are available as manifest lists and support the described 1667 architectures. It is also possible to pull a specific architecture directly by 1668 adding the "-$ARCH" suffix to the container image name. 1669 1670 name | architectures 1671 ---- | ------------- 1672 [registry.k8s.io/conformance:v1.29.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 1673 [registry.k8s.io/kube-apiserver:v1.29.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 1674 [registry.k8s.io/kube-controller-manager:v1.29.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 1675 [registry.k8s.io/kube-proxy:v1.29.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 1676 [registry.k8s.io/kube-scheduler:v1.29.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 1677 [registry.k8s.io/kubectl:v1.29.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 1678 1679 ## Changelog since v1.29.0-alpha.3 1680 1681 ## Changes by Kind 1682 1683 ### API Change 1684 1685 - Added support for projecting certificates.k8s.io/v1alpha1 ClusterTrustBundle objects into pods. ([#113374](https://github.com/kubernetes/kubernetes/pull/113374), [@ahmedtd](https://github.com/ahmedtd)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing] 1686 - Adds `optionalOldSelf` to `x-kubernetes-validations` to support ratcheting CRD schema constraints ([#121034](https://github.com/kubernetes/kubernetes/pull/121034), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery] 1687 - Fix API comment for the Job Ready field in status ([#121765](https://github.com/kubernetes/kubernetes/pull/121765), [@mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps] 1688 - Fix API comments for the FailIndex Job pod failure policy action. ([#121764](https://github.com/kubernetes/kubernetes/pull/121764), [@mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps] 1689 1690 ### Feature 1691 1692 - A customizable OrderedScoreFuncs() function is introduced. Out-of-tree plugins that use scheduler's preemption interface can implement this function for custom preemption preferences, or return nil to keep current behavior. ([#121867](https://github.com/kubernetes/kubernetes/pull/121867), [@lianghao208](https://github.com/lianghao208)) [SIG Scheduling] 1693 - Bump distroless-iptables to 0.4.1 based on Go 1.21.3 ([#121871](https://github.com/kubernetes/kubernetes/pull/121871), [@cpanato](https://github.com/cpanato)) [SIG Testing] 1694 - Fix overriding default KubeletConfig fields in drop-in configs if not set ([#121193](https://github.com/kubernetes/kubernetes/pull/121193), [@sohankunkerkar](https://github.com/sohankunkerkar)) [SIG Node and Testing] 1695 - KEP-4191- add support for split image filesystem in kubelet ([#120616](https://github.com/kubernetes/kubernetes/pull/120616), [@kannon92](https://github.com/kannon92)) [SIG Node and Testing] 1696 - Kubeadm: support updating certificate organization during 'kubeadm certs renew' ([#121841](https://github.com/kubernetes/kubernetes/pull/121841), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 1697 - Kubernetes is now built with Go 1.21.4 ([#121808](https://github.com/kubernetes/kubernetes/pull/121808), [@cpanato](https://github.com/cpanato)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Testing] 1698 1699 ### Bug or Regression 1700 1701 - Fix: statle smb mount issue when smb file share is deleted and then unmount ([#121851](https://github.com/kubernetes/kubernetes/pull/121851), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage] 1702 - KCCM: fix transient node addition + removal caused by #121090 while syncing load balancers on large clusters with a lot of churn ([#121091](https://github.com/kubernetes/kubernetes/pull/121091), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider, Network and Testing] 1703 - Kubeadm: change the "system:masters" Group in the apiserver-kubelet-client.crt certificate Subject to be "kubeadm:cluster-admins" which is a less privileged Group. ([#121837](https://github.com/kubernetes/kubernetes/pull/121837), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 1704 - Scheduler: in 1.29 pre-releases, enabling contextual logging slowed down pod scheduling. ([#121715](https://github.com/kubernetes/kubernetes/pull/121715), [@pohly](https://github.com/pohly)) [SIG Instrumentation and Scheduling] 1705 1706 ### Other (Cleanup or Flake) 1707 1708 - Update runc to 1.1.10 ([#121739](https://github.com/kubernetes/kubernetes/pull/121739), [@ty-dc](https://github.com/ty-dc)) [SIG Architecture and Node] 1709 1710 ## Dependencies 1711 1712 ### Added 1713 _Nothing has changed._ 1714 1715 ### Changed 1716 - github.com/mrunalp/fileutils: [v0.5.0 → v0.5.1](https://github.com/mrunalp/fileutils/compare/v0.5.0...v0.5.1) 1717 - github.com/opencontainers/runc: [v1.1.9 → v1.1.10](https://github.com/opencontainers/runc/compare/v1.1.9...v1.1.10) 1718 1719 ### Removed 1720 _Nothing has changed._ 1721 1722 1723 1724 # v1.29.0-alpha.3 1725 1726 1727 ## Downloads for v1.29.0-alpha.3 1728 1729 1730 1731 ### Source Code 1732 1733 filename | sha512 hash 1734 -------- | ----------- 1735 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes.tar.gz) | 998a680aee880601d65c14cf43a8ace13aacb3d693ac2f32c40ddc5c0a567fd4cc5627f397bf5612ed83d6b37ef568260f2700d46592bbc74174e155bf8f0606 1736 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-src.tar.gz) | ca46836dabd989a8dc6ee61032ab7f73747a5e2ef3bc11437e4036d95cbfbb9574f647b1672a098625729b62f1ff663726fcaf2dc3ea472e7b27d6b373d8afa9 1737 1738 ### Client Binaries 1739 1740 filename | sha512 hash 1741 -------- | ----------- 1742 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-darwin-amd64.tar.gz) | b82008b54b2a90e3640e786782cc20cf3a7d6a5011974f6710d418770541b53edb7d9d4ccd9489d4d81fcf7df7db38a3766db19898c86381b6fcfd7b261bc06a 1743 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-darwin-arm64.tar.gz) | b389eece6ea7ba07fdff76a6acdf36e77ed81e474277b62ef40b91ccc0d00c37f6f7c1194cacb14df844b1ea4dc66895b1c73bfedba570c71b72c6ab9a697861 1744 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-linux-386.tar.gz) | 5fa044082a1d2fb9d0b428fd2ba913196b4891f4c0571a7061ef1b6fee19ff820ff2b67506edad27fe0ddc735630d7398f66160836620188a527a8f3dbeb6b09 1745 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-linux-amd64.tar.gz) | 782d262f696e9b706de195870e5589fe3a0c4c11698574709668d4f60fcfe3cfb0137e86fb2d43a27a297bf88c29552216d30ac72255b4a757525f0b7e2385a1 1746 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-linux-arm.tar.gz) | cd9038cd3fa938aac9a0b462f7c6822d031f4e05e2529df378b4069f2d69d362236e5fc6e464d20cc42549f84f283f302b6cb33eb4a128ab91dcaa1cf04552e8 1747 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-linux-arm64.tar.gz) | d0fdf61def1be6c3b9e5259c13e8dbd764af44ed3dcdeb83c6a7d6cfe87b2293cbd88ceaad0aac87b448fe4766635b6b9eca40bc1a302f717d1bc0e26dac60ed 1748 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-linux-ppc64le.tar.gz) | c8b148404eecdff20939f0bec92024be58cb9629802c3768085834998e97b82e87f37443a867dfbb73e9922aada038d308ef02ec52a078aefb1f76360220c77b 1749 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-linux-s390x.tar.gz) | fb7070ef9d610fae614eadf9ec7fcfe68958143010b709586f0339309336e87f06d03ff8df5108b340ea063082e7b1d393b519ee6a4b4ed302427fd66e896295 1750 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-windows-386.tar.gz) | c0021a7668504a0a2be408cb2d1754bd20fc9afeeeb31dfb11f11787eb7047540c544888058300a83662dab845c726c7001562ee712b4c1d485ff0c3a88827f9 1751 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-windows-amd64.tar.gz) | 5678ab6523345ec38ba49a81c945223112228e75b82d0b459f0c9d6c37d3a0c93af4b82f05226e2d1110c1315ae5ed7ed3ad0bb085afad7f376b9715fe8fee75 1752 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-client-windows-arm64.tar.gz) | 2d7ac1add995683b29396fbbc06b15bf81ca62338ba0ed6d4738283752560fa167d6b9ffc44aeb4ed9b1bae6bc2ed8fe1d6346436c34a2acd3a5467ac68041bb 1753 1754 ### Server Binaries 1755 1756 filename | sha512 hash 1757 -------- | ----------- 1758 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-server-linux-amd64.tar.gz) | a948e26c77fb7cef3c50543c0b92c1ec1085516c4f16f9cd6695a02e1d88e44d5cd1f6fcc13bda48a580a26d024d1665ca2a960b37b31f0a7f0186e941a21e7e 1759 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-server-linux-arm64.tar.gz) | 14538ca02dc149a57c1cb30206b281248f9a84b024dad777e0326a9c6dc6c74228211e1a49f0480e2b7f825b12891d176489bcafcfab0fa05b18acc33c5044f8 1760 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-server-linux-ppc64le.tar.gz) | d347d6072f5a4c6c14ddb9418eadcc075824fa2dd15e49bfb79ee3fab7b2cc0efdd18021d7baed9024a4b83b4f9b800cedaeb8fa3917bfd47c4e5935146fc9fc 1761 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-server-linux-s390x.tar.gz) | 210ed1f933ba611cc3a828382ad15b02d2e35e74e99baed7077c21708249c5a561963e25f2582562773f1ea8f3eccb89b9fa25ca58da6eec9a516652efd432a5 1762 1763 ### Node Binaries 1764 1765 filename | sha512 hash 1766 -------- | ----------- 1767 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-node-linux-amd64.tar.gz) | 6d9b9e382a3137a2622a4631162b5c6a0c0c709fe95b76a7d5af610aec2a292b2f5a0b3378ddf8243450d774f6c1cd2ca16cbf240aed7109d819cd366b7abda9 1768 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-node-linux-arm64.tar.gz) | 0951c701155c914a0578dab9c8d584d32e8260ca923e1efccd0739db2065bf3d37d5d1b6584bc38f7873e20d164e57603e79abfecfbbe89e5386b6d7738d521b 1769 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-node-linux-ppc64le.tar.gz) | afbbedb58bd8344608e1fe047666914874419aef7f31c057a992e0dc24acae6151a7b0c53c2cfc8144ab8e0e914ee8b3a2f11adbb3791fb3b412172ade67439d 1770 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-node-linux-s390x.tar.gz) | 10c73d669dde0841078e5cee9158fa1a551c8bfe668d07beadf316386f815979f8729b89a0bed7e9e76350e82f6fd94d204187b9c3fe6e2bc1aabb2e580fee87 1771 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.3/kubernetes-node-windows-amd64.tar.gz) | c94d9f4979aeebfae9e66e029ea99ef6e349209f641d853032f87bb9cb646e885b995a512be3eef8e8cf2def76418e70086a03c00121e22c082b67b45562a6c5 1772 1773 ### Container Images 1774 1775 All container images are available as manifest lists and support the described 1776 architectures. It is also possible to pull a specific architecture directly by 1777 adding the "-$ARCH" suffix to the container image name. 1778 1779 name | architectures 1780 ---- | ------------- 1781 [registry.k8s.io/conformance:v1.29.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 1782 [registry.k8s.io/kube-apiserver:v1.29.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 1783 [registry.k8s.io/kube-controller-manager:v1.29.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 1784 [registry.k8s.io/kube-proxy:v1.29.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 1785 [registry.k8s.io/kube-scheduler:v1.29.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 1786 [registry.k8s.io/kubectl:v1.29.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 1787 1788 ## Changelog since v1.29.0-alpha.2 1789 1790 ## Urgent Upgrade Notes 1791 1792 ### (No, really, you MUST read this before you upgrade) 1793 1794 - Kubeadm: deploy a separate "super-admin.conf" file. The User in "admin.conf" is now bound to a new RBAC Group "kubeadm:cluster-admins" that have "cluster-admin" ClusterRole access. The User in "super-admin.conf" is bound to the "system:masters" built-in super-powers / break-glass Group that can bypass RBAC. Before this change the default "admin.conf" was bound to "system:masters" Group which was undesired. Executing "kubeadm init phase kubeconfig all" or just "kubeadm init" will now generate the new "super-admin.conf" file. The cluster admin can then decide to keep the file present on a node host or move it to a safe location. "kubadm certs renew" will renew the certificate in "super-admin.conf" to one year if the file exists. If it does not exist a "MISSING" note will be printed. "kubeadm upgrade apply" for this release will migrate this particular node to the two file setup. Subsequent kubeadm releases will continue to optionally renew the certificate in "super-admin.conf" if the file exists on disk and if renew on upgrade is not disabled. "kubeadm join --control-plane" will now generate only an "admin.conf" file that has the less privileged User. ([#121305](https://github.com/kubernetes/kubernetes/pull/121305), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 1795 - Stop accepting component configuration for kube-proxy and kubelet during `kubeadm upgrade plan --config`. This is a legacy behavior that is not well supported for upgrades and can be used only at the plan stage to determine if the configuration for these components stored in the cluster needs manual version migration. In the future, kubeadm will attempt alternative component config migration approaches. ([#120788](https://github.com/kubernetes/kubernetes/pull/120788), [@chendave](https://github.com/chendave)) [SIG Cluster Lifecycle] 1796 1797 ## Changes by Kind 1798 1799 ### Deprecation 1800 1801 - Creation of new CronJob objects containing `TZ` or `CRON_TZ` in `.spec.schedule`, accidentally enabled in 1.22, is now disallowed. Use the `.spec.timeZone` field instead, supported in 1.25+ clusters in default configurations. See https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#unsupported-timezone-specification for more information. ([#116252](https://github.com/kubernetes/kubernetes/pull/116252), [@soltysh](https://github.com/soltysh)) [SIG Apps] 1802 - Remove the networking alpha API ClusterCIDR ([#121229](https://github.com/kubernetes/kubernetes/pull/121229), [@aojea](https://github.com/aojea)) [SIG Apps, CLI, Cloud Provider, Network and Testing] 1803 1804 ### API Change 1805 1806 - A new sleep action for the PreStop lifecycle hook is added, allowing containers to pause for a specified duration before termination. ([#119026](https://github.com/kubernetes/kubernetes/pull/119026), [@AxeZhan](https://github.com/AxeZhan)) [SIG API Machinery, Apps, Node and Testing] 1807 - Add ImageMaximumGCAge field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. ([#121275](https://github.com/kubernetes/kubernetes/pull/121275), [@haircommander](https://github.com/haircommander)) [SIG API Machinery and Node] 1808 - Add a new ServiceCIDR type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs addresses ([#116516](https://github.com/kubernetes/kubernetes/pull/116516), [@aojea](https://github.com/aojea)) [SIG API Machinery, Apps, Auth, CLI, Network and Testing] 1809 - Add the DisableNodeKubeProxyVersion feature gate. If DisableNodeKubeProxyVersion is enabled, the kubeProxyVersion field is not set. ([#120954](https://github.com/kubernetes/kubernetes/pull/120954), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery, Apps and Node] 1810 - Added Windows support for InPlace Pod Vertical Scaling feature. ([#112599](https://github.com/kubernetes/kubernetes/pull/112599), [@fabi200123](https://github.com/fabi200123)) [SIG Autoscaling, Node, Scalability, Scheduling and Windows] 1811 - Added `UserNamespacesPodSecurityStandards` feature gate to enable user namespace support for Pod Security Standards. 1812 Enabling this feature will modify all Pod Security Standard rules to allow setting: `spec[.*].securityContext.[runAsNonRoot,runAsUser]`. 1813 This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled. 1814 The feature gate will not graduate or be enabled by default in future Kubernetes releases. ([#118760](https://github.com/kubernetes/kubernetes/pull/118760), [@saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Auth, Node and Release] 1815 - Added options for configuring nf_conntrack_udp_timeout, and nf_conntrack_udp_timeout_stream variables of netfilter conntrack subsystem. ([#120808](https://github.com/kubernetes/kubernetes/pull/120808), [@aroradaman](https://github.com/aroradaman)) [SIG API Machinery and Network] 1816 - Adds CEL expressions to v1alpha1 AuthenticationConfiguration. ([#121078](https://github.com/kubernetes/kubernetes/pull/121078), [@aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing] 1817 - Adds support for CEL expressions to v1alpha1 AuthorizationConfiguration webhook matchConditions. ([#121223](https://github.com/kubernetes/kubernetes/pull/121223), [@ritazh](https://github.com/ritazh)) [SIG API Machinery and Auth] 1818 - CSINodeExpandSecret feature has been promoted to GA in this release and enabled by default. The CSI drivers can make use of the `secretRef` values passed in NodeExpansion request optionally sent by the CSI Client from this release onwards. ([#121303](https://github.com/kubernetes/kubernetes/pull/121303), [@humblec](https://github.com/humblec)) [SIG API Machinery, Apps and Storage] 1819 - Graduate Job BackoffLimitPerIndex feature to Beta ([#121356](https://github.com/kubernetes/kubernetes/pull/121356), [@mimowo](https://github.com/mimowo)) [SIG Apps] 1820 - Kube-apiserver: adds --authorization-config flag for reading a configuration file containing an apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration object. --authorization-config flag is mutually exclusive with --authorization-modes and --authorization-webhook-* flags. The alpha StructuredAuthorizationConfiguration feature flag must be enabled for --authorization-config to be specified. ([#120154](https://github.com/kubernetes/kubernetes/pull/120154), [@palnabarun](https://github.com/palnabarun)) [SIG API Machinery, Auth and Testing] 1821 - Kube-proxy now has a new nftables-based mode, available by running 1822 1823 kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables 1824 1825 This is currently an alpha-level feature and while it probably will not 1826 eat your data, it may nibble at it a bit. (It passes e2e testing but has 1827 not yet seen real-world use.) 1828 1829 At this point it should be functionally mostly identical to the iptables 1830 mode, except that it does not (and will not) support Service NodePorts on 1831 127.0.0.1. (Also note that there are currently no command-line arguments 1832 for the nftables-specific config; you will need to use a config file if 1833 you want to set the equivalent of any of the `--iptables-xxx` options.) 1834 1835 As this code is still very new, it has not been heavily optimized yet; 1836 while it is expected to _eventually_ have better performance than the 1837 iptables backend, very little performance testing has been done so far. ([#121046](https://github.com/kubernetes/kubernetes/pull/121046), [@danwinship](https://github.com/danwinship)) [SIG API Machinery and Network] 1838 - Kube-proxy: Added an option/flag for configuring the `nf_conntrack_tcp_be_liberal` sysctl (in the kernel's netfilter conntrack subsystem). When enabled, kube-proxy will not install the DROP rule for invalid conntrack states, which currently breaks users of asymmetric routing. ([#120354](https://github.com/kubernetes/kubernetes/pull/120354), [@aroradaman](https://github.com/aroradaman)) [SIG API Machinery and Network] 1839 - PersistentVolumeLastPhaseTransitionTime is now beta, enabled by default. ([#120627](https://github.com/kubernetes/kubernetes/pull/120627), [@RomanBednar](https://github.com/RomanBednar)) [SIG Storage] 1840 - Promote PodReadyToStartContainers condition to beta. ([#119659](https://github.com/kubernetes/kubernetes/pull/119659), [@kannon92](https://github.com/kannon92)) [SIG Node and Testing] 1841 - The flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema and PriorityLevelConfiguration APIs has been promoted to flowcontrol.apiserver.k8s.io/v1, with the following changes: 1842 - PriorityLevelConfiguration: the `.spec.limited.nominalConcurrencyShares` field defaults to `30` only if the field is omitted (v1beta3 also defaulted an explicit `0` value to `30`). Specifying an explicit `0` value is not allowed in the `v1` version in v1.29 to ensure compatibility with 1.28 API servers. In v1.30, explicit `0` values will be allowed in this field in the `v1` API. 1843 The flowcontrol.apiserver.k8s.io/v1beta3 APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the `v1` APIs. Transition clients and manifests to use the `v1` APIs before upgrading to v1.32. ([#121089](https://github.com/kubernetes/kubernetes/pull/121089), [@tkashem](https://github.com/tkashem)) [SIG API Machinery and Testing] 1844 - The kube-proxy command-line documentation was updated to clarify that 1845 `--bind-address` does not actually have anything to do with binding to an 1846 address, and you probably don't actually want to be using it. ([#120274](https://github.com/kubernetes/kubernetes/pull/120274), [@danwinship](https://github.com/danwinship)) [SIG Network] 1847 - The matchLabelKeys/mismatchLabelKeys feature is introduced to the hard/soft PodAffinity/PodAntiAffinity. ([#116065](https://github.com/kubernetes/kubernetes/pull/116065), [@sanposhiho](https://github.com/sanposhiho)) [SIG API Machinery, Apps, Cloud Provider, Scheduling and Testing] 1848 - ValidatingAdmissionPolicy Type Checking now supports CRDs and API extensions types. ([#119109](https://github.com/kubernetes/kubernetes/pull/119109), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery, Apps, Auth and Testing] 1849 - When updating a CRD, per-expression cost limit check is skipped for x-kubernetes-validations rules of versions that are not mutated. ([#121460](https://github.com/kubernetes/kubernetes/pull/121460), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery] 1850 1851 ### Feature 1852 1853 - #### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: 1854 1855 <!-- 1856 This section can be blank if this pull request does not require a release note. 1857 1858 When adding links which point to resources within git repositories, like 1859 KEPs or supporting documentation, please reference a specific commit and avoid 1860 linking directly to the master branch. This ensures that links reference a 1861 specific point in time, rather than a document that may change over time. 1862 1863 See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files 1864 1865 Please use the following format for linking documentation: 1866 - [KEP]: <link> 1867 - [Usage]: <link> 1868 - [Other doc]: <link> 1869 --> ([#119517](https://github.com/kubernetes/kubernetes/pull/119517), [@sanposhiho](https://github.com/sanposhiho)) [SIG Node, Scheduling and Testing] 1870 - --interactive flag in kubectl delete will be visible to all users by default. ([#120416](https://github.com/kubernetes/kubernetes/pull/120416), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI and Testing] 1871 - Add container filesystem to the ImageFsInfoResponse. ([#120914](https://github.com/kubernetes/kubernetes/pull/120914), [@kannon92](https://github.com/kannon92)) [SIG Node and Testing] 1872 - Add job_pods_creation_total metrics for tracking Pods created by the Job controller labeled by events which triggered the Pod creation ([#121481](https://github.com/kubernetes/kubernetes/pull/121481), [@dejanzele](https://github.com/dejanzele)) [SIG Apps and Testing] 1873 - Add multiplication functionality to Quantity. ([#117411](https://github.com/kubernetes/kubernetes/pull/117411), [@tenzen-y](https://github.com/tenzen-y)) [SIG API Machinery] 1874 - Added a new `--init-only` command line flag to `kube-proxy`. Setting the flag makes `kube-proxy` perform its initial configuration that requires privileged mode, and then exit. The `--init-only` mode is intended to be executed in a privileged init container, so that the main container may run with a stricter `securityContext`. ([#120864](https://github.com/kubernetes/kubernetes/pull/120864), [@uablrek](https://github.com/uablrek)) [SIG Network and Scalability] 1875 - Added new feature gate called "RuntimeClassInImageCriApi" to address kubelet changes needed for KEP 4216. 1876 Noteable changes: 1877 1. Populate new RuntimeHandler field in CRI's ImageSpec struct during image pulls from container runtimes. 1878 2. Pass runtimeHandler field in RemoveImage() call to container runtime in kubelet's image garbage collection ([#121456](https://github.com/kubernetes/kubernetes/pull/121456), [@kiashok](https://github.com/kiashok)) [SIG Node and Windows] 1879 - Adds `apiextensions_apiserver_update_ratcheting_time` metric for tracking time taken during requests by feature `CRDValidationRatcheting` ([#121462](https://github.com/kubernetes/kubernetes/pull/121462), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery] 1880 - Bump cel-go to v0.17.7 and introduce set ext library with new options. ([#121577](https://github.com/kubernetes/kubernetes/pull/121577), [@cici37](https://github.com/cici37)) [SIG API Machinery, Auth and Cloud Provider] 1881 - Bump distroless-iptables to 0.4.1 based on Go 1.21.3 ([#121216](https://github.com/kubernetes/kubernetes/pull/121216), [@cpanato](https://github.com/cpanato)) [SIG Testing] 1882 - CEL can now correctly handle a CRD openAPIV3Schema that has neither Properties nor AdditionalProperties. ([#121459](https://github.com/kubernetes/kubernetes/pull/121459), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery and Testing] 1883 - CEL cost estimator no longer treats enums as unbounded strings when determining its length. Instead, the length is set to the longest possible enum value. ([#121085](https://github.com/kubernetes/kubernetes/pull/121085), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery] 1884 - CRDValidationRatcheting: Adds support for ratcheting `x-kubernetes-validations` in schema ([#121016](https://github.com/kubernetes/kubernetes/pull/121016), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery] 1885 - CRI: support image pull per runtime class ([#121121](https://github.com/kubernetes/kubernetes/pull/121121), [@kiashok](https://github.com/kiashok)) [SIG Node and Windows] 1886 - Calculate restartable init containers resource in pod autoscaler ([#120001](https://github.com/kubernetes/kubernetes/pull/120001), [@qingwave](https://github.com/qingwave)) [SIG Apps and Autoscaling] 1887 - Certain requestBody params in the OpenAPI v3 are correctly marked as required ([#120735](https://github.com/kubernetes/kubernetes/pull/120735), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 1888 - Client-side apply will use OpenAPI V3 by default ([#120707](https://github.com/kubernetes/kubernetes/pull/120707), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery and CLI] 1889 - Cluster/gce: add webhook to replace PersistentVolumeLabel admission controller ([#121628](https://github.com/kubernetes/kubernetes/pull/121628), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider] 1890 - Decouple TaintManager from NodeLifeCycleController (KEP-3902) ([#119208](https://github.com/kubernetes/kubernetes/pull/119208), [@atosatto](https://github.com/atosatto)) [SIG API Machinery, Apps, Instrumentation, Node, Scheduling and Testing] 1891 - DevicePluginCDIDevices feature has been graduated to Beta and enabled by default in the Kubelet ([#121254](https://github.com/kubernetes/kubernetes/pull/121254), [@bart0sh](https://github.com/bart0sh)) [SIG Node] 1892 - Dra: the scheduler plugin avoids additional scheduling attempts in some cases by falling back to SSA after a conflict ([#120534](https://github.com/kubernetes/kubernetes/pull/120534), [@pohly](https://github.com/pohly)) [SIG Node, Scheduling and Testing] 1893 - Enable traces for KMSv2 encrypt/decrypt operations. ([#121095](https://github.com/kubernetes/kubernetes/pull/121095), [@aramase](https://github.com/aramase)) [SIG API Machinery, Architecture, Auth, Instrumentation and Testing] 1894 - Etcd: build image for v3.5.9 ([#121567](https://github.com/kubernetes/kubernetes/pull/121567), [@mzaian](https://github.com/mzaian)) [SIG API Machinery] 1895 - Fixes bugs in handling of server-side apply, create, and update API requests for objects containing duplicate items in keyed lists. 1896 - A `create` or `update` API request with duplicate items in a keyed list no longer wipes out managedFields. Examples include env var entries with the same name, or port entries with the same containerPort in a pod spec. 1897 - A server-side apply request that makes unrelated changes to an object which has duplicate items in a keyed list no longer fails, and leaves the existing duplicate items as-is. 1898 - A server-side apply request that changes an object which has duplicate items in a keyed list, and modifies the duplicated item removes the duplicates and replaces them with the single item contained in the server-side apply request. ([#121575](https://github.com/kubernetes/kubernetes/pull/121575), [@apelisse](https://github.com/apelisse)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing] 1899 - Graduate the `ReadWriteOncePod` feature gate to GA ([#121077](https://github.com/kubernetes/kubernetes/pull/121077), [@chrishenzie](https://github.com/chrishenzie)) [SIG Apps, Node, Scheduling, Storage and Testing] 1900 - Introduce the job_finished_indexes_total metric for BackoffLimitPerIndex feature ([#121292](https://github.com/kubernetes/kubernetes/pull/121292), [@mimowo](https://github.com/mimowo)) [SIG Apps and Testing] 1901 - KEP-4191- add support for split image filesystem in kubelet ([#120616](https://github.com/kubernetes/kubernetes/pull/120616), [@kannon92](https://github.com/kannon92)) [SIG Node and Testing] 1902 - Kube-apiserver adds alpha support (guarded by the ServiceAccountTokenJTI feature gate) for adding a `jti` (JWT ID) claim to service account tokens it issues, adding an `authentication.kubernetes.io/credential-id` audit annotation in audit logs when the tokens are issued, and `authentication.kubernetes.io/credential-id` entry in the extra user info when the token is used to authenticate. 1903 - kube-apiserver adds alpha support (guarded by the ServiceAccountTokenPodNodeInfo feature gate) for including the node name (and uid, if the node exists) as additional claims in service account tokens it issues which are bound to pods, and `authentication.kubernetes.io/node-name` and `authentication.kubernetes.io/node-uid` extra user info when the token is used to authenticate. 1904 - kube-apiserver adds alpha support (guarded by the ServiceAccountTokenNodeBinding feature gate) for allowing TokenRequests that bind tokens directly to nodes, and (guarded by the ServiceAccountTokenNodeBindingValidation feature gate) for validating the node name and uid still exist when the token is used. ([#120780](https://github.com/kubernetes/kubernetes/pull/120780), [@munnerz](https://github.com/munnerz)) [SIG API Machinery, Apps, Auth, CLI and Testing] 1905 - Kube-controller-manager: The `LegacyServiceAccountTokenCleanUp` feature gate is now beta and enabled by default. When enabled, legacy auto-generated service account token secrets are auto-labeled with a `kubernetes.io/legacy-token-invalid-since` label if the credentials have not been used in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year), **and** are referenced from the `.secrets` list of a ServiceAccount object, **and** are not referenced from pods. This label causes the authentication layer to reject use of the credentials. After being labeled as invalid, if the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year) passes without the credential being used, the secret is automatically deleted. Secrets labeled as invalid which have not been auto-deleted yet can be re-activated by removing the `kubernetes.io/legacy-token-invalid-since` label. ([#120682](https://github.com/kubernetes/kubernetes/pull/120682), [@yt2985](https://github.com/yt2985)) [SIG Apps, Auth and Testing] 1906 - Kube-scheduler implements scheduling hints for the NodeAffinity plugin. 1907 The scheduling hints allow the scheduler to only retry scheduling a Pod 1908 that was previously rejected by the NodeAffinity plugin if a new Node or a Node update matches the Pod's node affinity. ([#119155](https://github.com/kubernetes/kubernetes/pull/119155), [@carlory](https://github.com/carlory)) [SIG Scheduling] 1909 - Kubeadm: Turn on FeatureGate `MergeCLIArgumentsWithConfig` to merge the config from flag and config file, otherwise, If the flag `--ignore-preflight-errors` is set from CLI, then the value from config file will be ignored. ([#119946](https://github.com/kubernetes/kubernetes/pull/119946), [@chendave](https://github.com/chendave)) [SIG Cluster Lifecycle] 1910 - Kubeadm: allow deploying a kubelet that is 3 versions older than the version of kubeadm (N-3). This aligns with the recent change made by SIG Architecture that extends the support skew between the control plane and kubelets. Tolerate this new kubelet skew for the commands "init", "join" and "upgrade". Note that if the kubeadm user applies a control plane version that is older than the kubeadm version (N-1 maximum) then the skew between the kubelet and control plane would become a maximum of N-2. ([#120825](https://github.com/kubernetes/kubernetes/pull/120825), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 1911 - Kubelet allows pods to use the `net.ipv4.tcp_fin_timeout` , “net.ipv4.tcp_keepalive_intvl” and “net.ipv4.tcp_keepalive_probes“ sysctl by default; Pod Security admission allows this sysctl in v1.29+ versions of the baseline and restricted policies. ([#121240](https://github.com/kubernetes/kubernetes/pull/121240), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG Auth and Node] 1912 - Kubelet allows pods to use the `net.ipv4.tcp_keepalive_time` sysctl by default and the minimal kernel version is 4.5; Pod Security admission allows this sysctl in v1.29+ versions of the baseline and restricted policies. ([#118846](https://github.com/kubernetes/kubernetes/pull/118846), [@cyclinder](https://github.com/cyclinder)) [SIG Auth, Network and Node] 1913 - Kubelet emits a metric for end-to-end pod startup latency including image pull. ([#121041](https://github.com/kubernetes/kubernetes/pull/121041), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG Node] 1914 - Kubernetes is now built with Go 1.21.3 ([#121149](https://github.com/kubernetes/kubernetes/pull/121149), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 1915 - Make decoding etcd's response respect the timeout context. ([#121614](https://github.com/kubernetes/kubernetes/pull/121614), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery] 1916 - Priority and Fairness feature is stable in 1.29, the feature gate will be removed in 1.31 ([#121638](https://github.com/kubernetes/kubernetes/pull/121638), [@tkashem](https://github.com/tkashem)) [SIG API Machinery and Testing] 1917 - Promote PodHostIPs condition to beta. ([#120257](https://github.com/kubernetes/kubernetes/pull/120257), [@wzshiming](https://github.com/wzshiming)) [SIG Network, Node and Testing] 1918 - Promote PodHostIPs condition to beta. ([#121477](https://github.com/kubernetes/kubernetes/pull/121477), [@wzshiming](https://github.com/wzshiming)) [SIG Network and Testing] 1919 - Promote PodReplacementPolicy to beta. ([#121491](https://github.com/kubernetes/kubernetes/pull/121491), [@dejanzele](https://github.com/dejanzele)) [SIG Apps and Testing] 1920 - Promotes plugin subcommand resolution feature to beta ([#120663](https://github.com/kubernetes/kubernetes/pull/120663), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI and Testing] 1921 - Sidecar termination is now serialized and each sidecar container will receive a SIGTERM after all main containers and later starting sidecar containers have terminated. ([#120620](https://github.com/kubernetes/kubernetes/pull/120620), [@tzneal](https://github.com/tzneal)) [SIG Node and Testing] 1922 - The CRD validation rule with feature gate `CustomResourceValidationExpressions` is promoted to GA. ([#121373](https://github.com/kubernetes/kubernetes/pull/121373), [@cici37](https://github.com/cici37)) [SIG API Machinery and Testing] 1923 - The KMSv2 feature with feature gates `KMSv2` and `KMSv2KDF` are promoted to GA. The `KMSv1` feature gate is now disabled by default. ([#121485](https://github.com/kubernetes/kubernetes/pull/121485), [@ritazh](https://github.com/ritazh)) [SIG API Machinery, Auth and Testing] 1924 - The SidecarContainers feature has graduated to beta and is enabled by default. ([#121579](https://github.com/kubernetes/kubernetes/pull/121579), [@gjkim42](https://github.com/gjkim42)) [SIG Node] 1925 - Updated the generic apiserver library to produce an error if a new API server is configured with support for a data format other than JSON, YAML, or Protobuf. ([#121325](https://github.com/kubernetes/kubernetes/pull/121325), [@benluddy](https://github.com/benluddy)) [SIG API Machinery] 1926 - ValidatingAdmissionPolicy now preserves types of composition variables, and raise type-related errors early. ([#121001](https://github.com/kubernetes/kubernetes/pull/121001), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery and Testing] 1927 1928 ### Documentation 1929 1930 - When the Kubelet fails to assign CPUs to a Pod because there less available CPUs than the Pod requests, the error message changed from 1931 "not enough cpus available to satisfy request" to "not enough cpus available to satisfy request: <num_requested> requested, only <num_available> available". ([#121059](https://github.com/kubernetes/kubernetes/pull/121059), [@matte21](https://github.com/matte21)) [SIG Node] 1932 1933 ### Failing Test 1934 1935 - K8s.io/dynamic-resource-allocation: DRA drivers updating to this release are compatible with Kubernetes 1.27 and 1.28. ([#120868](https://github.com/kubernetes/kubernetes/pull/120868), [@pohly](https://github.com/pohly)) [SIG Node] 1936 1937 ### Bug or Regression 1938 1939 - Add CAP_NET_RAW to netadmin debug profile and remove privileges when debugging nodes ([#118647](https://github.com/kubernetes/kubernetes/pull/118647), [@mochizuki875](https://github.com/mochizuki875)) [SIG CLI and Testing] 1940 - Add a check: if a user attempts to create a static pod via the kubelet without specifying a name, they will get a visible validation error. ([#119522](https://github.com/kubernetes/kubernetes/pull/119522), [@YTGhost](https://github.com/YTGhost)) [SIG Node] 1941 - Bugfix: OpenAPI spec no longer includes default of `{}` for certain fields where it did not make sense ([#120757](https://github.com/kubernetes/kubernetes/pull/120757), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 1942 - Changed kubelet logs from error to info for uncached partitions when using CRI stats provider ([#100448](https://github.com/kubernetes/kubernetes/pull/100448), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node] 1943 - Do not assign an empty value to the resource (CPU or memory) that not defined when stores the resources allocated to the pod in checkpoint ([#117615](https://github.com/kubernetes/kubernetes/pull/117615), [@aheng-ch](https://github.com/aheng-ch)) [SIG Node] 1944 - Etcd: Update to v3.5.10 ([#121566](https://github.com/kubernetes/kubernetes/pull/121566), [@mzaian](https://github.com/mzaian)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing] 1945 - Fix 121094 by re-introducing the readiness predicate for externalTrafficPolicy: Local services. ([#121116](https://github.com/kubernetes/kubernetes/pull/121116), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network] 1946 - Fix panic in Job controller when podRecreationPolicy: Failed is used, and the number of terminating pods exceeds parallelism. ([#121147](https://github.com/kubernetes/kubernetes/pull/121147), [@kannon92](https://github.com/kannon92)) [SIG Apps] 1947 - Fix systemLogQuery service name matching ([#120678](https://github.com/kubernetes/kubernetes/pull/120678), [@rothgar](https://github.com/rothgar)) [SIG Node] 1948 - Fixed a 1.28.0 regression where kube-controller-manager can crash when StatefulSet with Parallel policy and PVC labels is scaled up. ([#121142](https://github.com/kubernetes/kubernetes/pull/121142), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) [SIG Apps] 1949 - Fixed a bug around restarting init containers in the right order relative to normal containers with SidecarContainers feature enabled. ([#120269](https://github.com/kubernetes/kubernetes/pull/120269), [@gjkim42](https://github.com/gjkim42)) [SIG Node and Testing] 1950 - Fixed a bug where an API group's path was not unregistered from the API server's root paths when the group was deleted. ([#121283](https://github.com/kubernetes/kubernetes/pull/121283), [@tnqn](https://github.com/tnqn)) [SIG API Machinery and Testing] 1951 - Fixed a bug where the CPU set allocated to an init container, with containerRestartPolicy of `Always`, were erroneously reused by a regular container. ([#119447](https://github.com/kubernetes/kubernetes/pull/119447), [@gjkim42](https://github.com/gjkim42)) [SIG Node and Testing] 1952 - Fixed a bug where the device resources allocated to an init container, with containerRestartPolicy of `Always`, were erroneously reused by a regular container. ([#120461](https://github.com/kubernetes/kubernetes/pull/120461), [@gjkim42](https://github.com/gjkim42)) [SIG Node and Testing] 1953 - Fixed a bug where the memory resources allocated to an init container, with containerRestartPolicy of `Always`, were erroneously reused by a regular container. ([#120715](https://github.com/kubernetes/kubernetes/pull/120715), [@gjkim42](https://github.com/gjkim42)) [SIG Node] 1954 - Fixed a regression in default configurations, which enabled PodDisruptionConditions by default, 1955 that prevented the control plane's pod garbage collector from deleting pods that contained duplicated field keys (env. variables with repeated keys or container ports). ([#121103](https://github.com/kubernetes/kubernetes/pull/121103), [@mimowo](https://github.com/mimowo)) [SIG Apps, Auth, Node, Scheduling and Testing] 1956 - Fixed a regression in the Kubelet's behavior while creating a container when the `EventedPLEG` feature gate is enabled ([#120942](https://github.com/kubernetes/kubernetes/pull/120942), [@sairameshv](https://github.com/sairameshv)) [SIG Node] 1957 - Fixed a regression since 1.27.0 in scheduler framework when running score plugins. 1958 The `skippedScorePlugins` number might be greater than `enabledScorePlugins`, 1959 so when initializing a slice the cap(len(skippedScorePlugins) - len(enabledScorePlugins)) is negative, 1960 which is not allowed. ([#121632](https://github.com/kubernetes/kubernetes/pull/121632), [@kerthcet](https://github.com/kerthcet)) [SIG Scheduling] 1961 - Fixed bug that kubelet resource metric `container_start_time_seconds` had timestamp equal to container start time. ([#120518](https://github.com/kubernetes/kubernetes/pull/120518), [@saschagrunert](https://github.com/saschagrunert)) [SIG Instrumentation, Node and Testing] 1962 - Fixed inconsistency in the calculation of number of nodes that have an image, which affect the scoring in the ImageLocality plugin ([#116938](https://github.com/kubernetes/kubernetes/pull/116938), [@olderTaoist](https://github.com/olderTaoist)) [SIG Scheduling] 1963 - Fixed some invalid and unimportant log calls. ([#121249](https://github.com/kubernetes/kubernetes/pull/121249), [@pohly](https://github.com/pohly)) [SIG Cloud Provider, Cluster Lifecycle and Testing] 1964 - Fixed the bug that kubelet could't output logs after log file rotated when kubectl logs POD_NAME -f is running. ([#115702](https://github.com/kubernetes/kubernetes/pull/115702), [@xyz-li](https://github.com/xyz-li)) [SIG Node] 1965 - Fixed the issue where pod with ordinal number lower than the rolling partitioning number was being deleted it was coming up with updated image. ([#120731](https://github.com/kubernetes/kubernetes/pull/120731), [@adilGhaffarDev](https://github.com/adilGhaffarDev)) [SIG Apps and Testing] 1966 - Fixed tracking of terminating Pods in the Job status. The field was not updated unless there were other changes to apply ([#121342](https://github.com/kubernetes/kubernetes/pull/121342), [@dejanzele](https://github.com/dejanzele)) [SIG Apps and Testing] 1967 - Fixes an issue where StatefulSet might not restart a pod after eviction or node failure. ([#121389](https://github.com/kubernetes/kubernetes/pull/121389), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) [SIG Apps and Testing] 1968 - Fixes calculating the requeue time in the cronjob controller, which results in properly handling failed/stuck jobs ([#121327](https://github.com/kubernetes/kubernetes/pull/121327), [@soltysh](https://github.com/soltysh)) [SIG Apps] 1969 - Forbid sysctls for pod sharing the respective namespaces with the host when creating and update pod without such sysctls ([#118705](https://github.com/kubernetes/kubernetes/pull/118705), [@pacoxu](https://github.com/pacoxu)) [SIG Apps and Node] 1970 - K8s.io/dynamic-resource-allocation/controller: ResourceClaimParameters and ResourceClassParameters validation errors were not visible on ResourceClaim, ResourceClass and Pod. ([#121065](https://github.com/kubernetes/kubernetes/pull/121065), [@byako](https://github.com/byako)) [SIG Node] 1971 - Kube-proxy now reports its health more accurately in dual-stack clusters when there are problems with only one IP family. ([#118146](https://github.com/kubernetes/kubernetes/pull/118146), [@aroradaman](https://github.com/aroradaman)) [SIG Network and Windows] 1972 - Metric buckets for pod_start_duration_seconds are changed to {0.5, 1, 2, 3, 4, 5, 6, 8, 10, 20, 30, 45, 60, 120, 180, 240, 300, 360, 480, 600, 900, 1200, 1800, 2700, 3600} ([#120680](https://github.com/kubernetes/kubernetes/pull/120680), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG Instrumentation and Node] 1973 - Mitigates http/2 DOS vulnerabilities for CVE-2023-44487 and CVE-2023-39325 for the API server when the client is unauthenticated. The mitigation may be disabled by setting the `UnauthenticatedHTTP2DOSMitigation` feature gate to `false` (it is enabled by default). An API server fronted by an L7 load balancer that already mitigates these http/2 attacks may choose to disable the kube-apiserver mitigation to avoid disrupting load balancer → kube-apiserver connections if http/2 requests from multiple clients share the same backend connection. An API server on a private network may opt to disable the kube-apiserver mitigation to prevent performance regressions for unauthenticated clients. Authenticated requests rely on the fix in golang.org/x/net v0.17.0 alone. https://issue.k8s.io/121197 tracks further mitigation of http/2 attacks by authenticated clients. ([#121120](https://github.com/kubernetes/kubernetes/pull/121120), [@enj](https://github.com/enj)) [SIG API Machinery] 1974 - Registered metric `apiserver_request_body_size_bytes` to track the size distribution of requests by `resource` and `verb`. ([#120474](https://github.com/kubernetes/kubernetes/pull/120474), [@YaoC](https://github.com/YaoC)) [SIG API Machinery and Instrumentation] 1975 - Update the CRI-O socket path, so users who configure kubelet to use a location like `/run/crio/crio.sock` don't see strange behaviour from CRI stats provider. ([#118704](https://github.com/kubernetes/kubernetes/pull/118704), [@dgl](https://github.com/dgl)) [SIG Node] 1976 - Wait.PollUntilContextTimeout function, if immediate is true, the condition will be invoked before waiting and guarantees that the condition is invoked at least once and then wait a interval before executing again. ([#119762](https://github.com/kubernetes/kubernetes/pull/119762), [@AxeZhan](https://github.com/AxeZhan)) [SIG API Machinery] 1977 1978 ### Other (Cleanup or Flake) 1979 1980 - Allow using lower and upper case feature flag value, the name has to match still ([#121441](https://github.com/kubernetes/kubernetes/pull/121441), [@soltysh](https://github.com/soltysh)) [SIG CLI] 1981 - E2E storage tests: setting test tags like `[Slow]` via the DriverInfo.FeatureTag field is no longer supported. ([#121391](https://github.com/kubernetes/kubernetes/pull/121391), [@pohly](https://github.com/pohly)) [SIG Storage and Testing] 1982 - EnqueueExtensions from plugins other than PreEnqueue, PreFilter, Filter, Reserve and Permit are ignored. 1983 It reduces the number of kinds of cluster events the scheduler needs to subscribe/handle. ([#121571](https://github.com/kubernetes/kubernetes/pull/121571), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 1984 - GetPodQOS(pod *core.Pod) function now returns the stored value from PodStatus.QOSClass, if set. To compute/evaluate the value of QOSClass from scratch, ComputePodQOS(pod *core.Pod) must be used. ([#119665](https://github.com/kubernetes/kubernetes/pull/119665), [@vinaykul](https://github.com/vinaykul)) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing] 1985 - Graduate JobReadyPods to stable. The feature gate can no longer be disabled. ([#121302](https://github.com/kubernetes/kubernetes/pull/121302), [@stuton](https://github.com/stuton)) [SIG Apps and Testing] 1986 - Kube-controller-manager's help will include controllers behind a feature gate in `--controllers` flag ([#120371](https://github.com/kubernetes/kubernetes/pull/120371), [@atiratree](https://github.com/atiratree)) [SIG API Machinery] 1987 - Kubeadm: remove leftover ALPHA disclaimer that can be seen in the "kubeadm init phase certs" command help screen. The "certs" phase of "init" is not ALPHA. ([#121172](https://github.com/kubernetes/kubernetes/pull/121172), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 1988 - Migrated the remainder of the scheduler to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#120933](https://github.com/kubernetes/kubernetes/pull/120933), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation, Scheduling and Testing] 1989 - Previous versions of Kubernetes on Google Cloud required that workloads (e.g. Deployments, DaemonSets, etc.) which used PersistentDisk volumes were using them in read-only mode. This validation provided very little value at relatively host implementation cost, and will no longer be validated. If this is a problem for a specific use-case, please set the `SkipReadOnlyValidationGCE` gate to false to re-enable the validation, and file a kubernetes bug with details. ([#121083](https://github.com/kubernetes/kubernetes/pull/121083), [@thockin](https://github.com/thockin)) [SIG Apps] 1990 - Remove GA featuregate about CSIMigrationvSphere in 1.29 ([#121291](https://github.com/kubernetes/kubernetes/pull/121291), [@bzsuni](https://github.com/bzsuni)) [SIG API Machinery, Node and Storage] 1991 - Remove GA featuregate about ProbeTerminationGracePeriod in 1.29 ([#121257](https://github.com/kubernetes/kubernetes/pull/121257), [@bzsuni](https://github.com/bzsuni)) [SIG Node and Testing] 1992 - Remove GA featuregate for JobTrackingWithFinalizers in 1.28 ([#119100](https://github.com/kubernetes/kubernetes/pull/119100), [@bzsuni](https://github.com/bzsuni)) [SIG Apps] 1993 - Remove GAed feature gates OpenAPIV3 ([#121255](https://github.com/kubernetes/kubernetes/pull/121255), [@tukwila](https://github.com/tukwila)) [SIG API Machinery and Testing] 1994 - Remove GAed feature gates SeccompDefault ([#121246](https://github.com/kubernetes/kubernetes/pull/121246), [@tukwila](https://github.com/tukwila)) [SIG Node] 1995 - Remove GAed feature gates TopologyManager ([#121252](https://github.com/kubernetes/kubernetes/pull/121252), [@tukwila](https://github.com/tukwila)) [SIG Node] 1996 - Removed the `CronJobTimeZone` feature gate (the feature is stable and always enabled) 1997 - Removed the `JobMutableNodeSchedulingDirectives` feature gate (the feature is stable and always enabled) 1998 - Removed the `LegacyServiceAccountTokenNoAutoGeneration` feature gate (the feature is stable and always enabled) ([#120192](https://github.com/kubernetes/kubernetes/pull/120192), [@SataQiu](https://github.com/SataQiu)) [SIG Apps, Auth and Scheduling] 1999 - Removed the `DownwardAPIHugePages` feature gate (the feature is stable and always enabled) ([#120249](https://github.com/kubernetes/kubernetes/pull/120249), [@pacoxu](https://github.com/pacoxu)) [SIG Apps and Node] 2000 - Removed the `GRPCContainerProbe` feature gate (the feature is stable and always enabled) ([#120248](https://github.com/kubernetes/kubernetes/pull/120248), [@pacoxu](https://github.com/pacoxu)) [SIG API Machinery, CLI and Node] 2001 - Rename apiserver_request_body_sizes metric to apiserver_request_body_size_bytes ([#120503](https://github.com/kubernetes/kubernetes/pull/120503), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG API Machinery] 2002 - RetroactiveDefaultStorageClass feature gate that graduated to GA in 1.28 and was unconditionally enabled has been removed in v1.29. ([#120861](https://github.com/kubernetes/kubernetes/pull/120861), [@RomanBednar](https://github.com/RomanBednar)) [SIG Storage] 2003 2004 ## Dependencies 2005 2006 ### Added 2007 - cloud.google.com/go/dataproc/v2: v2.0.1 2008 - github.com/danwinship/knftables: [v0.0.13](https://github.com/danwinship/knftables/tree/v0.0.13) 2009 - github.com/google/s2a-go: [v0.1.7](https://github.com/google/s2a-go/tree/v0.1.7) 2010 - google.golang.org/genproto/googleapis/bytestream: e85fd2c 2011 2012 ### Changed 2013 - cloud.google.com/go/accessapproval: v1.6.0 → v1.7.1 2014 - cloud.google.com/go/accesscontextmanager: v1.7.0 → v1.8.1 2015 - cloud.google.com/go/aiplatform: v1.37.0 → v1.48.0 2016 - cloud.google.com/go/analytics: v0.19.0 → v0.21.3 2017 - cloud.google.com/go/apigateway: v1.5.0 → v1.6.1 2018 - cloud.google.com/go/apigeeconnect: v1.5.0 → v1.6.1 2019 - cloud.google.com/go/apigeeregistry: v0.6.0 → v0.7.1 2020 - cloud.google.com/go/appengine: v1.7.1 → v1.8.1 2021 - cloud.google.com/go/area120: v0.7.1 → v0.8.1 2022 - cloud.google.com/go/artifactregistry: v1.13.0 → v1.14.1 2023 - cloud.google.com/go/asset: v1.13.0 → v1.14.1 2024 - cloud.google.com/go/assuredworkloads: v1.10.0 → v1.11.1 2025 - cloud.google.com/go/automl: v1.12.0 → v1.13.1 2026 - cloud.google.com/go/baremetalsolution: v0.5.0 → v1.1.1 2027 - cloud.google.com/go/batch: v0.7.0 → v1.3.1 2028 - cloud.google.com/go/beyondcorp: v0.5.0 → v1.0.0 2029 - cloud.google.com/go/bigquery: v1.50.0 → v1.53.0 2030 - cloud.google.com/go/billing: v1.13.0 → v1.16.0 2031 - cloud.google.com/go/binaryauthorization: v1.5.0 → v1.6.1 2032 - cloud.google.com/go/certificatemanager: v1.6.0 → v1.7.1 2033 - cloud.google.com/go/channel: v1.12.0 → v1.16.0 2034 - cloud.google.com/go/cloudbuild: v1.9.0 → v1.13.0 2035 - cloud.google.com/go/clouddms: v1.5.0 → v1.6.1 2036 - cloud.google.com/go/cloudtasks: v1.10.0 → v1.12.1 2037 - cloud.google.com/go/compute: v1.19.0 → v1.23.0 2038 - cloud.google.com/go/contactcenterinsights: v1.6.0 → v1.10.0 2039 - cloud.google.com/go/container: v1.15.0 → v1.24.0 2040 - cloud.google.com/go/containeranalysis: v0.9.0 → v0.10.1 2041 - cloud.google.com/go/datacatalog: v1.13.0 → v1.16.0 2042 - cloud.google.com/go/dataflow: v0.8.0 → v0.9.1 2043 - cloud.google.com/go/dataform: v0.7.0 → v0.8.1 2044 - cloud.google.com/go/datafusion: v1.6.0 → v1.7.1 2045 - cloud.google.com/go/datalabeling: v0.7.0 → v0.8.1 2046 - cloud.google.com/go/dataplex: v1.6.0 → v1.9.0 2047 - cloud.google.com/go/dataqna: v0.7.0 → v0.8.1 2048 - cloud.google.com/go/datastore: v1.11.0 → v1.13.0 2049 - cloud.google.com/go/datastream: v1.7.0 → v1.10.0 2050 - cloud.google.com/go/deploy: v1.8.0 → v1.13.0 2051 - cloud.google.com/go/dialogflow: v1.32.0 → v1.40.0 2052 - cloud.google.com/go/dlp: v1.9.0 → v1.10.1 2053 - cloud.google.com/go/documentai: v1.18.0 → v1.22.0 2054 - cloud.google.com/go/domains: v0.8.0 → v0.9.1 2055 - cloud.google.com/go/edgecontainer: v1.0.0 → v1.1.1 2056 - cloud.google.com/go/essentialcontacts: v1.5.0 → v1.6.2 2057 - cloud.google.com/go/eventarc: v1.11.0 → v1.13.0 2058 - cloud.google.com/go/filestore: v1.6.0 → v1.7.1 2059 - cloud.google.com/go/firestore: v1.9.0 → v1.11.0 2060 - cloud.google.com/go/functions: v1.13.0 → v1.15.1 2061 - cloud.google.com/go/gkebackup: v0.4.0 → v1.3.0 2062 - cloud.google.com/go/gkeconnect: v0.7.0 → v0.8.1 2063 - cloud.google.com/go/gkehub: v0.12.0 → v0.14.1 2064 - cloud.google.com/go/gkemulticloud: v0.5.0 → v1.0.0 2065 - cloud.google.com/go/gsuiteaddons: v1.5.0 → v1.6.1 2066 - cloud.google.com/go/iam: v0.13.0 → v1.1.1 2067 - cloud.google.com/go/iap: v1.7.1 → v1.8.1 2068 - cloud.google.com/go/ids: v1.3.0 → v1.4.1 2069 - cloud.google.com/go/iot: v1.6.0 → v1.7.1 2070 - cloud.google.com/go/kms: v1.10.1 → v1.15.0 2071 - cloud.google.com/go/language: v1.9.0 → v1.10.1 2072 - cloud.google.com/go/lifesciences: v0.8.0 → v0.9.1 2073 - cloud.google.com/go/longrunning: v0.4.1 → v0.5.1 2074 - cloud.google.com/go/managedidentities: v1.5.0 → v1.6.1 2075 - cloud.google.com/go/maps: v0.7.0 → v1.4.0 2076 - cloud.google.com/go/mediatranslation: v0.7.0 → v0.8.1 2077 - cloud.google.com/go/memcache: v1.9.0 → v1.10.1 2078 - cloud.google.com/go/metastore: v1.10.0 → v1.12.0 2079 - cloud.google.com/go/monitoring: v1.13.0 → v1.15.1 2080 - cloud.google.com/go/networkconnectivity: v1.11.0 → v1.12.1 2081 - cloud.google.com/go/networkmanagement: v1.6.0 → v1.8.0 2082 - cloud.google.com/go/networksecurity: v0.8.0 → v0.9.1 2083 - cloud.google.com/go/notebooks: v1.8.0 → v1.9.1 2084 - cloud.google.com/go/optimization: v1.3.1 → v1.4.1 2085 - cloud.google.com/go/orchestration: v1.6.0 → v1.8.1 2086 - cloud.google.com/go/orgpolicy: v1.10.0 → v1.11.1 2087 - cloud.google.com/go/osconfig: v1.11.0 → v1.12.1 2088 - cloud.google.com/go/oslogin: v1.9.0 → v1.10.1 2089 - cloud.google.com/go/phishingprotection: v0.7.0 → v0.8.1 2090 - cloud.google.com/go/policytroubleshooter: v1.6.0 → v1.8.0 2091 - cloud.google.com/go/privatecatalog: v0.8.0 → v0.9.1 2092 - cloud.google.com/go/pubsub: v1.30.0 → v1.33.0 2093 - cloud.google.com/go/pubsublite: v1.7.0 → v1.8.1 2094 - cloud.google.com/go/recaptchaenterprise/v2: v2.7.0 → v2.7.2 2095 - cloud.google.com/go/recommendationengine: v0.7.0 → v0.8.1 2096 - cloud.google.com/go/recommender: v1.9.0 → v1.10.1 2097 - cloud.google.com/go/redis: v1.11.0 → v1.13.1 2098 - cloud.google.com/go/resourcemanager: v1.7.0 → v1.9.1 2099 - cloud.google.com/go/resourcesettings: v1.5.0 → v1.6.1 2100 - cloud.google.com/go/retail: v1.12.0 → v1.14.1 2101 - cloud.google.com/go/run: v0.9.0 → v1.2.0 2102 - cloud.google.com/go/scheduler: v1.9.0 → v1.10.1 2103 - cloud.google.com/go/secretmanager: v1.10.0 → v1.11.1 2104 - cloud.google.com/go/security: v1.13.0 → v1.15.1 2105 - cloud.google.com/go/securitycenter: v1.19.0 → v1.23.0 2106 - cloud.google.com/go/servicedirectory: v1.9.0 → v1.11.0 2107 - cloud.google.com/go/shell: v1.6.0 → v1.7.1 2108 - cloud.google.com/go/spanner: v1.45.0 → v1.47.0 2109 - cloud.google.com/go/speech: v1.15.0 → v1.19.0 2110 - cloud.google.com/go/storagetransfer: v1.8.0 → v1.10.0 2111 - cloud.google.com/go/talent: v1.5.0 → v1.6.2 2112 - cloud.google.com/go/texttospeech: v1.6.0 → v1.7.1 2113 - cloud.google.com/go/tpu: v1.5.0 → v1.6.1 2114 - cloud.google.com/go/trace: v1.9.0 → v1.10.1 2115 - cloud.google.com/go/translate: v1.7.0 → v1.8.2 2116 - cloud.google.com/go/video: v1.15.0 → v1.19.0 2117 - cloud.google.com/go/videointelligence: v1.10.0 → v1.11.1 2118 - cloud.google.com/go/vision/v2: v2.7.0 → v2.7.2 2119 - cloud.google.com/go/vmmigration: v1.6.0 → v1.7.1 2120 - cloud.google.com/go/vmwareengine: v0.3.0 → v1.0.0 2121 - cloud.google.com/go/vpcaccess: v1.6.0 → v1.7.1 2122 - cloud.google.com/go/webrisk: v1.8.0 → v1.9.1 2123 - cloud.google.com/go/websecurityscanner: v1.5.0 → v1.6.1 2124 - cloud.google.com/go/workflows: v1.10.0 → v1.11.1 2125 - cloud.google.com/go: v0.110.0 → v0.110.6 2126 - github.com/alecthomas/template: [fb15b89 → a0175ee](https://github.com/alecthomas/template/compare/fb15b89...a0175ee) 2127 - github.com/cncf/xds/go: [06c439d → e9ce688](https://github.com/cncf/xds/go/compare/06c439d...e9ce688) 2128 - github.com/cyphar/filepath-securejoin: [v0.2.3 → v0.2.4](https://github.com/cyphar/filepath-securejoin/compare/v0.2.3...v0.2.4) 2129 - github.com/docker/distribution: [v2.8.1+incompatible → v2.8.2+incompatible](https://github.com/docker/distribution/compare/v2.8.1...v2.8.2) 2130 - github.com/docker/docker: [v20.10.21+incompatible → v20.10.24+incompatible](https://github.com/docker/docker/compare/v20.10.21...v20.10.24) 2131 - github.com/envoyproxy/go-control-plane: [v0.10.3 → v0.11.1](https://github.com/envoyproxy/go-control-plane/compare/v0.10.3...v0.11.1) 2132 - github.com/envoyproxy/protoc-gen-validate: [v0.9.1 → v1.0.2](https://github.com/envoyproxy/protoc-gen-validate/compare/v0.9.1...v1.0.2) 2133 - github.com/fsnotify/fsnotify: [v1.6.0 → v1.7.0](https://github.com/fsnotify/fsnotify/compare/v1.6.0...v1.7.0) 2134 - github.com/go-logr/logr: [v1.2.4 → v1.3.0](https://github.com/go-logr/logr/compare/v1.2.4...v1.3.0) 2135 - github.com/godbus/dbus/v5: [v5.0.6 → v5.1.0](https://github.com/godbus/dbus/v5/compare/v5.0.6...v5.1.0) 2136 - github.com/golang/glog: [v1.0.0 → v1.1.0](https://github.com/golang/glog/compare/v1.0.0...v1.1.0) 2137 - github.com/google/cadvisor: [v0.47.3 → v0.48.1](https://github.com/google/cadvisor/compare/v0.47.3...v0.48.1) 2138 - github.com/google/cel-go: [v0.17.6 → v0.17.7](https://github.com/google/cel-go/compare/v0.17.6...v0.17.7) 2139 - github.com/google/go-cmp: [v0.5.9 → v0.6.0](https://github.com/google/go-cmp/compare/v0.5.9...v0.6.0) 2140 - github.com/googleapis/gax-go/v2: [v2.7.1 → v2.11.0](https://github.com/googleapis/gax-go/v2/compare/v2.7.1...v2.11.0) 2141 - github.com/grpc-ecosystem/grpc-gateway/v2: [v2.7.0 → v2.16.0](https://github.com/grpc-ecosystem/grpc-gateway/v2/compare/v2.7.0...v2.16.0) 2142 - github.com/ishidawataru/sctp: [7c296d4 → 7ff4192](https://github.com/ishidawataru/sctp/compare/7c296d4...7ff4192) 2143 - github.com/konsorten/go-windows-terminal-sequences: [v1.0.3 → v1.0.1](https://github.com/konsorten/go-windows-terminal-sequences/compare/v1.0.3...v1.0.1) 2144 - github.com/onsi/gomega: [v1.28.0 → v1.29.0](https://github.com/onsi/gomega/compare/v1.28.0...v1.29.0) 2145 - github.com/spf13/afero: [v1.2.2 → v1.1.2](https://github.com/spf13/afero/compare/v1.2.2...v1.1.2) 2146 - github.com/stretchr/testify: [v1.8.2 → v1.8.4](https://github.com/stretchr/testify/compare/v1.8.2...v1.8.4) 2147 - go.etcd.io/bbolt: v1.3.7 → v1.3.8 2148 - go.etcd.io/etcd/api/v3: v3.5.9 → v3.5.10 2149 - go.etcd.io/etcd/client/pkg/v3: v3.5.9 → v3.5.10 2150 - go.etcd.io/etcd/client/v2: v2.305.9 → v2.305.10 2151 - go.etcd.io/etcd/client/v3: v3.5.9 → v3.5.10 2152 - go.etcd.io/etcd/pkg/v3: v3.5.9 → v3.5.10 2153 - go.etcd.io/etcd/raft/v3: v3.5.9 → v3.5.10 2154 - go.etcd.io/etcd/server/v3: v3.5.9 → v3.5.10 2155 - go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful: v0.35.0 → v0.42.0 2156 - go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc: v0.35.0 → v0.42.0 2157 - go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.35.1 → v0.44.0 2158 - go.opentelemetry.io/contrib/propagators/b3: v1.10.0 → v1.17.0 2159 - go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc: v1.10.0 → v1.19.0 2160 - go.opentelemetry.io/otel/exporters/otlp/otlptrace: v1.10.0 → v1.19.0 2161 - go.opentelemetry.io/otel/metric: v0.31.0 → v1.19.0 2162 - go.opentelemetry.io/otel/sdk: v1.10.0 → v1.19.0 2163 - go.opentelemetry.io/otel/trace: v1.10.0 → v1.19.0 2164 - go.opentelemetry.io/otel: v1.10.0 → v1.19.0 2165 - go.opentelemetry.io/proto/otlp: v0.19.0 → v1.0.0 2166 - golang.org/x/crypto: v0.12.0 → v0.14.0 2167 - golang.org/x/net: v0.14.0 → v0.17.0 2168 - golang.org/x/oauth2: v0.8.0 → v0.10.0 2169 - golang.org/x/sys: v0.12.0 → v0.13.0 2170 - golang.org/x/term: v0.11.0 → v0.13.0 2171 - golang.org/x/text: v0.12.0 → v0.13.0 2172 - google.golang.org/api: v0.114.0 → v0.126.0 2173 - google.golang.org/genproto/googleapis/api: dd9d682 → 23370e0 2174 - google.golang.org/genproto/googleapis/rpc: 28d5490 → b8732ec 2175 - google.golang.org/genproto: 0005af6 → f966b18 2176 - google.golang.org/grpc: v1.54.0 → v1.58.3 2177 - k8s.io/klog/v2: v2.100.1 → v2.110.1 2178 - k8s.io/kube-openapi: d090da1 → 2dd684a 2179 - sigs.k8s.io/structured-merge-diff/v4: v4.3.0 → v4.4.1 2180 2181 ### Removed 2182 - cloud.google.com/go/dataproc: v1.12.0 2183 - cloud.google.com/go/gaming: v1.9.0 2184 - github.com/blang/semver: [v3.5.1+incompatible](https://github.com/blang/semver/tree/v3.5.1) 2185 - github.com/jmespath/go-jmespath/internal/testify: [v1.5.1](https://github.com/jmespath/go-jmespath/internal/testify/tree/v1.5.1) 2186 - go.opentelemetry.io/otel/exporters/otlp/internal/retry: v1.10.0 2187 2188 2189 2190 # v1.29.0-alpha.2 2191 2192 2193 ## Downloads for v1.29.0-alpha.2 2194 2195 2196 2197 ### Source Code 2198 2199 filename | sha512 hash 2200 -------- | ----------- 2201 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes.tar.gz) | 138f47b2c53030e171d368d382c911048ce5d8387450e5e6717f09ac8cf6289b6c878046912130d58d7814509bbc45dbc19d6ee4f24404321ea18b24ebab2a36 2202 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-src.tar.gz) | 73ab06309d6f6cbcb8a417c068367b670a04dcbe90574a7906201dd70b9c322cd052818114b746a4d61b7bce6115ae547eaafc955c41053898a315c968db2f36 2203 2204 ### Client Binaries 2205 2206 filename | sha512 hash 2207 -------- | ----------- 2208 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-darwin-amd64.tar.gz) | c9604fbb9e848a4b3dc85ee2836f74b4ccd321e4c72d22b2d4558eb0f0c3833bff35d0c36602c13c5c5c79e9233fda874bfa85433291ab3484cf61c9012ee515 2209 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-darwin-arm64.tar.gz) | fed42ecbfc20b5f63ac48bbb9b73abc4b72aca76ac8bdd51b9ea6af053b1fc6a8e63b5e11f9d14c4814f03b49531da2536f1342cda2da03514c44ccf05c311b0 2210 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-linux-386.tar.gz) | 93c61229d7b07a476296b5b800c853c8e984101d5077fc19a195673f7543e7d2eb2599311c1846c91ef1f7ae29c3e05b6f41b873e92a3429563e3d83900050da 2211 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-linux-amd64.tar.gz) | 4260b49733f6b0967c504e2246b455b2348b487e84f7a019fda8b4a87d43d27a03e7ed55b505764c14f2079c4c3d71c68d77f981b604e13e7210680f45ee66e3 2212 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-linux-arm.tar.gz) | 4e837fd2f55cbb5f93cdf60235511a85635485962f00e0378a95a7ff846eb86b7bf053203ab353b294131b2e2663d0e783dae79c18601d4d66f98a6e5152e96e 2213 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-linux-arm64.tar.gz) | 6f3954d2adc289879984d18c2605110a7d5f0a5f6366233c25adf3a742f8dc1183e8a4d4747de8077af1045a259b150e0e86b27e10d683aa8decdc760ac6279b 2214 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-linux-ppc64le.tar.gz) | 741b76827ff9e810e490d8698eb7620826a16e978e5c7744a1fa0e65124690cfc9601e7f1c8f50e77f25185ba3176789ddcb7d5caaddde66436c31658bacde1d 2215 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-linux-s390x.tar.gz) | 0c635883e2f9caca03bcf3b42ba0b479f44c8cc2a3d5dd425b0fee278f3e884bef0e897fe51cbf00bb0bc061371805d9f9cbccf839477671f92e078c04728735 2216 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-windows-386.tar.gz) | ebddbb358fd2d817908069eb66744dc62cae56ad470b1e36c6ebd0d2284e79ae5b9a5f8a86fef365f30b34e14093827ad736814241014f597e2ac88788102cf4 2217 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-windows-amd64.tar.gz) | 01a451a809cd45e7916a3e982e2b94d372accab9dfe20667e95c10d56f9194b997721c0c219ff7ff97828b6466108eec6e57dcb33e3e3b0c5f770af1514a9f1a 2218 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-client-windows-arm64.tar.gz) | 473ba648ffde41fd5b63374cc1595eb43b873808c6b0cc5e939628937f3f7fb36dba4b7c7c8ef03408d557442094ec22e12c03f40be137f9cc99761b4cc1a1f8 2219 2220 ### Server Binaries 2221 2222 filename | sha512 hash 2223 -------- | ----------- 2224 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-server-linux-amd64.tar.gz) | c3f7abcee3fdcf6f311b5de0bfe037318e646641c1ce311950d920252623cca285d1f1cef0e2d936c0f981edc1c725897a42aa9e03b77fe5f76f1090665d967f 2225 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-server-linux-arm64.tar.gz) | 17614842df6bb528434b8b063b1d1c3efc8e4eff9cbc182f049d811f68e08514026fbb616199a3dee97e62ce2fd1eda0b9778d8e74040e645c482cfe6a18a8b4 2226 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-server-linux-ppc64le.tar.gz) | 2f818035ef199a7745e24d2ce86abf6c52e351d7922885e264c5d07db3e0f21048c32db85f3044e01443abd87a45f92df52fda44e8df05000754b03f34132f2f 2227 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-server-linux-s390x.tar.gz) | 96a34c768f347f23c46f990a8f6ddf3127b13f7a183453b92eb7bc27ce896767f31b38317a6ae5a11f2d4b459ec9564385f8abe61082a4165928edfee0c9765e 2228 2229 ### Node Binaries 2230 2231 filename | sha512 hash 2232 -------- | ----------- 2233 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-node-linux-amd64.tar.gz) | 66845cf86e32c19be9d339417a4772b9bcf51b2bf4d1ef5acc2e9eb006bbd19b3c036aa3721b3d8fe08b6fb82284ba25a6ecb5eb7b84f657cc968224d028f22c 2234 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-node-linux-arm64.tar.gz) | 98902ee33242f9e78091433115804d54eafde24903a3515f0300f60c0273c7c0494666c221ce418d79e715f8ecf654f0edabc5b69765da26f83a812e963b5afb 2235 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-node-linux-ppc64le.tar.gz) | 82f1213b5942c5c1576afadb4b066dfa1427c7709adf6ba636b9a52dfdb1b20f62b1cc0436b265e714fbee08c71d8786295d2439c10cc05bd58b2ab2a87611d4 2236 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-node-linux-s390x.tar.gz) | 7cb8cb65195c5dd63329d02907cdbb0f5473066606c108f4516570f449623f93b1ca822d5a00fad063ec8630e956fa53a0ab530a8487bccb01810943847d4942 2237 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.2/kubernetes-node-windows-amd64.tar.gz) | 1222e2d7dbaf7920e1ba927231cc7e275641cf0939be1520632353df6219bbcb3b49515d084e7f2320a2ff59b2de9fee252d8f5e9c48d7509f1174c6cb357b66 2238 2239 ### Container Images 2240 2241 All container images are available as manifest lists and support the described 2242 architectures. It is also possible to pull a specific architecture directly by 2243 adding the "-$ARCH" suffix to the container image name. 2244 2245 name | architectures 2246 ---- | ------------- 2247 [registry.k8s.io/conformance:v1.29.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 2248 [registry.k8s.io/kube-apiserver:v1.29.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 2249 [registry.k8s.io/kube-controller-manager:v1.29.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 2250 [registry.k8s.io/kube-proxy:v1.29.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 2251 [registry.k8s.io/kube-scheduler:v1.29.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 2252 [registry.k8s.io/kubectl:v1.29.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 2253 2254 ## Changelog since v1.29.0-alpha.1 2255 2256 ## Changes by Kind 2257 2258 ### Feature 2259 2260 - Adds `apiserver_watch_list_duration_seconds` metrics. Which will measure response latency distribution in seconds for watch list requests broken by group, version, resource and scope ([#120490](https://github.com/kubernetes/kubernetes/pull/120490), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery and Instrumentation] 2261 - Allow-list of metric labels can be configured by supplying a manifest using the --allow-metric-labels-manifest flag ([#118299](https://github.com/kubernetes/kubernetes/pull/118299), [@rexagod](https://github.com/rexagod)) [SIG Architecture and Instrumentation] 2262 - Bump distroless-iptables to 0.3.3 based on Go 1.21.2 ([#121073](https://github.com/kubernetes/kubernetes/pull/121073), [@cpanato](https://github.com/cpanato)) [SIG Testing] 2263 - Implements API for streaming for the etcd store implementation 2264 2265 When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. ([#119557](https://github.com/kubernetes/kubernetes/pull/119557), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery] 2266 - Kubelet, when using cloud provider external, initializes temporary the node addresses using the --node-ip flag values if set, until the cloud provider overrides it. ([#121028](https://github.com/kubernetes/kubernetes/pull/121028), [@aojea](https://github.com/aojea)) [SIG Cloud Provider and Node] 2267 - Kubernetes is now built with Go 1.21.2 ([#121021](https://github.com/kubernetes/kubernetes/pull/121021), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 2268 - Migrated the volumebinding scheduler plugins to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116803](https://github.com/kubernetes/kubernetes/pull/116803), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation, Scheduling and Storage] 2269 - The kube-apiserver exposes four new metrics to inform about errors on the clusterIP and nodePort allocation logic ([#120843](https://github.com/kubernetes/kubernetes/pull/120843), [@aojea](https://github.com/aojea)) [SIG Instrumentation and Network] 2270 2271 ### Failing Test 2272 2273 - K8s.io/dynamic-resource-allocation: DRA drivers updating to this release are compatible with Kubernetes 1.27 and 1.28. ([#120868](https://github.com/kubernetes/kubernetes/pull/120868), [@pohly](https://github.com/pohly)) [SIG Node] 2274 2275 ### Bug or Regression 2276 2277 - Cluster-bootstrap: improve the security of the functions responsible for generation and validation of bootstrap tokens ([#120400](https://github.com/kubernetes/kubernetes/pull/120400), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle and Security] 2278 - Do not fail volume attach or publish operation at kubelet if target path directory already exists on the node. ([#119735](https://github.com/kubernetes/kubernetes/pull/119735), [@akankshapanse](https://github.com/akankshapanse)) [SIG Storage] 2279 - Fix regression with adding aggregated apiservices panicking and affected health check introduced in release v1.28.0 ([#120814](https://github.com/kubernetes/kubernetes/pull/120814), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing] 2280 - Fixed a bug where containers would not start on cgroupv2 systems where swap is disabled. ([#120784](https://github.com/kubernetes/kubernetes/pull/120784), [@elezar](https://github.com/elezar)) [SIG Node] 2281 - Fixed a regression in kube-proxy where it might refuse to start if given 2282 single-stack IPv6 configuration options on a node that has both IPv4 and 2283 IPv6 IPs. ([#121008](https://github.com/kubernetes/kubernetes/pull/121008), [@danwinship](https://github.com/danwinship)) [SIG Network] 2284 - Fixed attaching volumes after detach errors. Now volumes that failed to detach are not treated as attached, Kubernetes will make sure they are fully attached before they can be used by pods. ([#120595](https://github.com/kubernetes/kubernetes/pull/120595), [@jsafrane](https://github.com/jsafrane)) [SIG Apps and Storage] 2285 - Fixes a regression (CLIENTSET_PKG: unbound variable) when invoking deprecated generate-groups.sh script ([#120877](https://github.com/kubernetes/kubernetes/pull/120877), [@soltysh](https://github.com/soltysh)) [SIG API Machinery] 2286 - K8s.io/dynamic-resource-allocation/controller: UnsuitableNodes did not handle a mix of allocated and unallocated claims correctly. ([#120338](https://github.com/kubernetes/kubernetes/pull/120338), [@pohly](https://github.com/pohly)) [SIG Node] 2287 - K8s.io/dynamic-resource-allocation: handle a selected node which isn't listed as potential node ([#120871](https://github.com/kubernetes/kubernetes/pull/120871), [@pohly](https://github.com/pohly)) [SIG Node] 2288 - Kubeadm: fix the bug that kubeadm always do CRI detection when --config is passed even if it is not required by the subcommand ([#120828](https://github.com/kubernetes/kubernetes/pull/120828), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2289 2290 ### Other (Cleanup or Flake) 2291 2292 - Client-go: k8s.io/client-go/tools events and record packages have new APIs for specifying a context and logger ([#120729](https://github.com/kubernetes/kubernetes/pull/120729), [@pohly](https://github.com/pohly)) [SIG API Machinery and Instrumentation] 2293 - Deprecated the `--cloud-provider` and `--cloud-config` CLI parameters in kube-apiserver. 2294 These parameters will be removed in a future release. ([#120903](https://github.com/kubernetes/kubernetes/pull/120903), [@dims](https://github.com/dims)) [SIG API Machinery] 2295 2296 ## Dependencies 2297 2298 ### Added 2299 _Nothing has changed._ 2300 2301 ### Changed 2302 - github.com/emicklei/go-restful/v3: [v3.9.0 → v3.11.0](https://github.com/emicklei/go-restful/v3/compare/v3.9.0...v3.11.0) 2303 - github.com/onsi/ginkgo/v2: [v2.9.4 → v2.13.0](https://github.com/onsi/ginkgo/v2/compare/v2.9.4...v2.13.0) 2304 - github.com/onsi/gomega: [v1.27.6 → v1.28.0](https://github.com/onsi/gomega/compare/v1.27.6...v1.28.0) 2305 - golang.org/x/crypto: v0.11.0 → v0.12.0 2306 - golang.org/x/mod: v0.10.0 → v0.12.0 2307 - golang.org/x/net: v0.13.0 → v0.14.0 2308 - golang.org/x/sync: v0.2.0 → v0.3.0 2309 - golang.org/x/sys: v0.10.0 → v0.12.0 2310 - golang.org/x/term: v0.10.0 → v0.11.0 2311 - golang.org/x/text: v0.11.0 → v0.12.0 2312 - golang.org/x/tools: v0.8.0 → v0.12.0 2313 2314 ### Removed 2315 _Nothing has changed._ 2316 2317 2318 2319 # v1.29.0-alpha.1 2320 2321 2322 ## Downloads for v1.29.0-alpha.1 2323 2324 2325 2326 ### Source Code 2327 2328 filename | sha512 hash 2329 -------- | ----------- 2330 [kubernetes.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes.tar.gz) | 107062e8da7c416206f18b4376e9e0c2ca97b37c720a047f2bc6cf8a1bdc2b41e84defd0a29794d9562f3957932c0786a5647450b41d2850a9b328826bb3248d 2331 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-src.tar.gz) | 8182774faa5547f496642fdad7e2617a4d07d75af8ddf85fb8246087ddffab596528ffde29500adc9945d4e263fce766927ed81396a11f88876b3fa76628a371 2332 2333 ### Client Binaries 2334 2335 filename | sha512 hash 2336 -------- | ----------- 2337 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-darwin-amd64.tar.gz) | ac9a08cd98af5eb27f8dde895510db536098dd52ee89682e7f103c793cb99cddcd992e3a349d526854caaa27970aa1ef964db4cc27d1009576fb604bf0c1cdf1 2338 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-darwin-arm64.tar.gz) | 28744076618dcd7eca4175726d7f3ac67fe94f08f1b6ca4373b134a6402c0f5203f1146d79a211443c751b2f2825df3507166fc3c5e40a55d545c3e5d2a48e56 2339 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-linux-386.tar.gz) | 0207a2571b6d0e6e55f36af9d2ed27f31eacfb23f2f54dd2eb8fbc38ef5b033edb24fb9a5ece7e7020fd921a9c841fff435512d12421bfa13294cc9c297eb877 2340 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-linux-amd64.tar.gz) | 57fc39ba259ae61b88c23fd136904395abc23c44f4b4db3e2922827ec7e6def92bc77364de3e2f6b54b27bb4b5e42e9cf4d1c0aa6d12c4a5a17788d9f996d9ad 2341 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-linux-arm.tar.gz) | 53a54d3fbda46162139a90616d708727c23d3aae0a2618197df5ac443ac3d49980a62034e3f2514f1a1622e4ce5f6e821d2124a61a9e63ce6d29268b33292949 2342 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-linux-arm64.tar.gz) | ee3ca4626c802168db71ad55c1d8b45c03ec774c146dd6da245e5bb26bf7fd6728a477f1ad0c5094967a0423f94e35e4458c6716f3abe005e8fc55ae354174cf 2343 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-linux-ppc64le.tar.gz) | 60cd35076dd4afb9005349003031fa9f1802a2a120fbbe842d6fd061a1bca39baabcbb18fb4b6610a5ca626fc64e1d780c7aadb203d674697905489187a415ce 2344 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-linux-s390x.tar.gz) | 68fdd0fc35dfd6fae0d25d7834270c94b16ae860fccc4253e7c347ce165d10cadc190e8b320fd2c4afd508afc6c10f246b8a5f0148ca1b1d56f7b2843cc39d30 2345 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-windows-386.tar.gz) | 0c5d3dbfaaffa81726945510c972cc15895ea87bcd43b798675465fdadaa4d2d9597cb4fc6baee9ee719c919d1f46a9390c15cb0da60250f41eb4fcc3337b337 2346 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-windows-amd64.tar.gz) | 2e519867cbc793ea1c1e45f040de81b49c70b9b42fac072ac5cac36e8de71f0dddd0c64354631bcb2b3af36a0f377333c0cd885c2df36ef8cd7e6c8fd5628aa4 2347 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-client-windows-arm64.tar.gz) | 1a80cad80c1c9f753a38e6c951b771b0df820455141f40ba44e227f6acc81b59454f8dbff12e83c61bf647eaa1ff98944930969a99c96a087a35921f4e6ac968 2348 2349 ### Server Binaries 2350 2351 filename | sha512 hash 2352 -------- | ----------- 2353 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-server-linux-amd64.tar.gz) | c74a3f7bdd16095fb366b4313e50984f2ee7cb99c77ad2bcccea066756ce6e0fc45f4528b79c8cb7e6370430ee2d03fa6bc10ca87a59d8684a59e1ebd3524afd 2354 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-server-linux-arm64.tar.gz) | b6844b5769fd5687525dcedca42c7bb036f6acad65d3de3c8cda46dbbe0ac23c289fdb7fbf15f1c37184498d6a1fb018e41e1c97ded4581f045ad2039e3ddec2 2355 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-server-linux-ppc64le.tar.gz) | a15eb2db4821454974920a987bb1e73bc4ee638b845b07f35cab55dcf482c142d3cdaed347bfa0452d5311b3d9152463a3dae1d176b6101ed081ec594e0d526c 2356 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-server-linux-s390x.tar.gz) | 60e24d8b4902821b436b5adebd6594ef0db79802d64787a1424aa6536873e2d749dfc6ebc2eb81db3240c925500a3e927ee7385188f866c28123736459e19b7b 2357 2358 ### Node Binaries 2359 2360 filename | sha512 hash 2361 -------- | ----------- 2362 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-node-linux-amd64.tar.gz) | 44832c7b90c88e7ca70737bad8d50ee8ba434ee7a94940f9d45beda9e9aadc7e2c973b65fcb986216229796a5807dae2470dbcf1ade5c075d86011eefe21509b 2363 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-node-linux-arm64.tar.gz) | a13862d9bae0ff358377afc60f5222490a8e6bb7197d4a7d568edd4f150348f7a3dc7342129cd2d5c5353d2d43349b97c854df3e8886a8d52aedb95c634e3b5a 2364 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-node-linux-ppc64le.tar.gz) | 57348f82bb4db8c230d8dffdef513ed75d7b267b226a5d15b3deb9783f8ed56fe40f8ce018ab34c28f9f8210b2e41b0f55d185dcdbaf912dd57e2ea78f8d3c53 2365 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-node-linux-s390x.tar.gz) | 2013eb4746e818cf336e0fee37650df98c19876030397803abce9531730eb0b95e6284f5a2abdd2b97090a67d07fd7a9c74c84fc7b4b83f0bce04a6dc9ad2555 2366 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.29.0-alpha.1/kubernetes-node-windows-amd64.tar.gz) | 3a4d63e2117cdbebc655e674bb017e246c263e893fc0ca3e8dc0091d6d9f96c9f0756c0fa8b45ba461502ae432f908ea922c21378b82ff3990b271f42eedc138 2367 2368 ### Container Images 2369 2370 All container images are available as manifest lists and support the described 2371 architectures. It is also possible to pull a specific architecture directly by 2372 adding the "-$ARCH" suffix to the container image name. 2373 2374 name | architectures 2375 ---- | ------------- 2376 [registry.k8s.io/conformance:v1.29.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 2377 [registry.k8s.io/kube-apiserver:v1.29.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 2378 [registry.k8s.io/kube-controller-manager:v1.29.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 2379 [registry.k8s.io/kube-proxy:v1.29.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 2380 [registry.k8s.io/kube-scheduler:v1.29.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 2381 [registry.k8s.io/kubectl:v1.29.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 2382 2383 ## Changelog since v1.28.0 2384 2385 ## Changes by Kind 2386 2387 ### Deprecation 2388 2389 - #### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: 2390 2391 <!-- 2392 This section can be blank if this pull request does not require a release note. 2393 2394 When adding links which point to resources within git repositories, like 2395 KEPs or supporting documentation, please reference a specific commit and avoid 2396 linking directly to the master branch. This ensures that links reference a 2397 specific point in time, rather than a document that may change over time. 2398 2399 See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files 2400 2401 Please use the following format for linking documentation: 2402 - [KEP]: <link> 2403 - [Usage]: <link> 2404 - [Other doc]: <link> 2405 --> ([#119495](https://github.com/kubernetes/kubernetes/pull/119495), [@bzsuni](https://github.com/bzsuni)) [SIG API Machinery] 2406 2407 ### API Change 2408 2409 - Added a new `ipMode` field to the `.status` of Services where `type` is set to `LoadBalancer`. 2410 The new field is behind the `LoadBalancerIPMode` feature gate. ([#119937](https://github.com/kubernetes/kubernetes/pull/119937), [@RyanAoh](https://github.com/RyanAoh)) [SIG API Machinery, Apps, Cloud Provider, Network and Testing] 2411 - Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps. 2412 The incorrect cost was evident when the result of a function was used in subsequent operations. ([#119800](https://github.com/kubernetes/kubernetes/pull/119800), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider] 2413 - Go API: the ResourceRequirements struct needs to be replaced with VolumeResourceRequirements for use with volumes. ([#118653](https://github.com/kubernetes/kubernetes/pull/118653), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling, Storage and Testing] 2414 - Kube-apiserver: adds --authentication-config flag for reading AuthenticationConfiguration files. --authentication-config flag is mutually exclusive with the existing --oidc-* flags. ([#119142](https://github.com/kubernetes/kubernetes/pull/119142), [@aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing] 2415 - Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta3 is removed in v1.29. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. ([#119994](https://github.com/kubernetes/kubernetes/pull/119994), [@SataQiu](https://github.com/SataQiu)) [SIG Scheduling and Testing] 2416 - Mark the onPodConditions field as optional in Job's pod failure policy. ([#120204](https://github.com/kubernetes/kubernetes/pull/120204), [@mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps] 2417 - Retry NodeStageVolume calls if CSI node driver is not running ([#120330](https://github.com/kubernetes/kubernetes/pull/120330), [@rohitssingh](https://github.com/rohitssingh)) [SIG Apps, Storage and Testing] 2418 - The kube-scheduler `selectorSpread` plugin has been removed, please use the `podTopologySpread` plugin instead. ([#117720](https://github.com/kubernetes/kubernetes/pull/117720), [@kerthcet](https://github.com/kerthcet)) [SIG Scheduling] 2419 2420 ### Feature 2421 2422 - --sync-frequency will not affect the update interval of volumes that use ConfigMaps or Secrets when the configMapAndSecretChangeDetectionStrategy is set to Cache. The update interval is only affected by node.alpha.kubernetes.io/ttl node annotation." ([#120255](https://github.com/kubernetes/kubernetes/pull/120255), [@likakuli](https://github.com/likakuli)) [SIG Node] 2423 - Add a new scheduler metric, `pod_scheduling_sli_duration_seconds`, and start the deprecation for `pod_scheduling_duration_seconds`. ([#119049](https://github.com/kubernetes/kubernetes/pull/119049), [@helayoty](https://github.com/helayoty)) [SIG Instrumentation, Scheduling and Testing] 2424 - Added apiserver_envelope_encryption_dek_cache_filled to measure number of records in data encryption key(DEK) cache. ([#119878](https://github.com/kubernetes/kubernetes/pull/119878), [@ritazh](https://github.com/ritazh)) [SIG API Machinery and Auth] 2425 - Added kubectl node drain helper callbacks `OnPodDeletionOrEvictionStarted` and `OnPodDeletionOrEvictionFailed`; people extending `kubectl` can use these new callbacks for more granularity. 2426 - Deprecated the `OnPodDeletedOrEvicted` node drain helper callback. ([#117502](https://github.com/kubernetes/kubernetes/pull/117502), [@adilGhaffarDev](https://github.com/adilGhaffarDev)) [SIG CLI] 2427 - Adding apiserver identity to the following metrics: 2428 apiserver_envelope_encryption_key_id_hash_total, apiserver_envelope_encryption_key_id_hash_last_timestamp_seconds, apiserver_envelope_encryption_key_id_hash_status_last_timestamp_seconds, apiserver_encryption_config_controller_automatic_reload_failures_total, apiserver_encryption_config_controller_automatic_reload_success_total, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds 2429 2430 Fix bug to surface events for the following metrics: apiserver_encryption_config_controller_automatic_reload_failures_total, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds, apiserver_encryption_config_controller_automatic_reload_success_total ([#120438](https://github.com/kubernetes/kubernetes/pull/120438), [@ritazh](https://github.com/ritazh)) [SIG API Machinery, Auth, Instrumentation and Testing] 2431 - Bump distroless-iptables to 0.3.2 based on Go 1.21.1 ([#120527](https://github.com/kubernetes/kubernetes/pull/120527), [@cpanato](https://github.com/cpanato)) [SIG Testing] 2432 - Changed `kubectl help` to display basic details for subcommands from plugins ([#116752](https://github.com/kubernetes/kubernetes/pull/116752), [@xvzf](https://github.com/xvzf)) [SIG CLI] 2433 - Changed the `KMSv2KDF` feature gate to be enabled by default. ([#120433](https://github.com/kubernetes/kubernetes/pull/120433), [@enj](https://github.com/enj)) [SIG API Machinery, Auth and Testing] 2434 - Graduated the following kubelet resource metrics to **general availability**: 2435 - `container_cpu_usage_seconds_total` 2436 - `container_memory_working_set_bytes` 2437 - `container_start_time_seconds` 2438 - `node_cpu_usage_seconds_total` 2439 - `node_memory_working_set_bytes` 2440 - `pod_cpu_usage_seconds_total` 2441 - `pod_memory_working_set_bytes` 2442 - `resource_scrape_error` 2443 2444 Deprecated (renamed) `scrape_error` in favor of `resource_scrape_error` ([#116897](https://github.com/kubernetes/kubernetes/pull/116897), [@Richabanker](https://github.com/Richabanker)) [SIG Architecture, Instrumentation, Node and Testing] 2445 - Graduation API List chunking (aka pagination) feature to stable ([#119503](https://github.com/kubernetes/kubernetes/pull/119503), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery, Cloud Provider and Testing] 2446 - Implements API for streaming for the etcd store implementation 2447 2448 When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. ([#119557](https://github.com/kubernetes/kubernetes/pull/119557), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery] 2449 - Improve memory usage of kube-scheduler by dropping the `.metadata.managedFields` field that kube-scheduler doesn't require. ([#119556](https://github.com/kubernetes/kubernetes/pull/119556), [@linxiulei](https://github.com/linxiulei)) [SIG Scheduling] 2450 - In a scheduler with Permit plugins, when a Pod is rejected during WaitOnPermit, the scheduler records the plugin. 2451 The scheduler will use the record to honor cluster events and queueing hints registered for the plugin, to inform whether to retry the pod. ([#119785](https://github.com/kubernetes/kubernetes/pull/119785), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling and Testing] 2452 - In tree cloud providers are now switched off by default. Please use DisableCloudProviders and DisableKubeletCloudCredentialProvider feature flags if you still need this functionality. ([#117503](https://github.com/kubernetes/kubernetes/pull/117503), [@dims](https://github.com/dims)) [SIG API Machinery, Cloud Provider and Testing] 2453 - Introduce new apiserver metric apiserver_flowcontrol_current_inqueue_seats. This metric is analogous to `apiserver_flowcontrol_current_inqueue_requests` but tracks totals seats as each request can take more than 1 seat. ([#119385](https://github.com/kubernetes/kubernetes/pull/119385), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery] 2454 - Kube-proxy don't panic on exit when the Node object changes its PodCIDR ([#120375](https://github.com/kubernetes/kubernetes/pull/120375), [@pegasas](https://github.com/pegasas)) [SIG Network] 2455 - Kube-proxy will only install the DROP rules for invalid conntrack states if the nf_conntrack_tcp_be_liberal is not set. ([#120412](https://github.com/kubernetes/kubernetes/pull/120412), [@aojea](https://github.com/aojea)) [SIG Network] 2456 - Kubeadm: add validation to verify that the CertificateKey is a valid hex encoded AES key ([#120064](https://github.com/kubernetes/kubernetes/pull/120064), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2457 - Kubeadm: promoted feature gate `EtcdLearnerMode` to beta. Learner mode for joining etcd members is now enabled by default. ([#120228](https://github.com/kubernetes/kubernetes/pull/120228), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 2458 - Kubelet exposes latency metrics of different stages of the node startup. ([#118568](https://github.com/kubernetes/kubernetes/pull/118568), [@qiutongs](https://github.com/qiutongs)) [SIG Instrumentation, Node and Scalability] 2459 - Kubernetes is now built with Go 1.21.1 ([#120493](https://github.com/kubernetes/kubernetes/pull/120493), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 2460 - Kubernetes is now built with go 1.21.0 ([#118996](https://github.com/kubernetes/kubernetes/pull/118996), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 2461 - List the pods using <PVC> as an ephemeral storage volume in "Used by:" part of the output of `kubectl describe pvc <PVC>` command. ([#120427](https://github.com/kubernetes/kubernetes/pull/120427), [@MaGaroo](https://github.com/MaGaroo)) [SIG CLI] 2462 - Migrated the nodevolumelimits scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116884](https://github.com/kubernetes/kubernetes/pull/116884), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation, Node, Scheduling, Storage and Testing] 2463 - Promote ServiceNodePortStaticSubrange to stable and lock to default ([#120233](https://github.com/kubernetes/kubernetes/pull/120233), [@xuzhenglun](https://github.com/xuzhenglun)) [SIG Network] 2464 - QueueingHint got error in its returning value. If QueueingHint returns error, the scheduler logs the error and treats the event as QueueAfterBackoff so that the Pod wouldn't be stuck in the unschedulable pod pool. ([#119290](https://github.com/kubernetes/kubernetes/pull/119290), [@carlory](https://github.com/carlory)) [SIG Node, Scheduling and Testing] 2465 - Remove /livez livezchecks for KMS v1 and v2 to ensure KMS health does not cause kube-apiserver restart. KMS health checks are still in place as a healthz and readiness checks. ([#120583](https://github.com/kubernetes/kubernetes/pull/120583), [@ritazh](https://github.com/ritazh)) [SIG API Machinery, Auth and Testing] 2466 - The CloudDualStackNodeIPs feature is now beta, meaning that when using 2467 an external cloud provider that has been updated to support the feature, 2468 you can pass comma-separated dual-stack `--node-ips` to kubelet and have 2469 the cloud provider take both IPs into account. ([#120275](https://github.com/kubernetes/kubernetes/pull/120275), [@danwinship](https://github.com/danwinship)) [SIG API Machinery, Cloud Provider and Network] 2470 - The Dockerfile for the kubectl image has been updated with the addition of a specific base image and essential utilities (bash and jq). ([#119592](https://github.com/kubernetes/kubernetes/pull/119592), [@rayandas](https://github.com/rayandas)) [SIG CLI, Node, Release and Testing] 2471 - Use of secret-based service account tokens now adds an `authentication.k8s.io/legacy-token-autogenerated-secret` or `authentication.k8s.io/legacy-token-manual-secret` audit annotation containing the name of the secret used. ([#118598](https://github.com/kubernetes/kubernetes/pull/118598), [@yuanchen8911](https://github.com/yuanchen8911)) [SIG Auth, Instrumentation and Testing] 2472 - Volume_zone plugin will consider beta labels as GA labels during the scheduling process.Therefore, if the values of the labels are the same, PVs with beta labels can also be scheduled to nodes with GA labels. ([#118923](https://github.com/kubernetes/kubernetes/pull/118923), [@AxeZhan](https://github.com/AxeZhan)) [SIG Scheduling] 2473 2474 ### Documentation 2475 2476 - Added descriptions and examples for the situation of using kubectl rollout restart without specifying a particular deployment. ([#120118](https://github.com/kubernetes/kubernetes/pull/120118), [@Ithrael](https://github.com/Ithrael)) [SIG CLI] 2477 2478 ### Failing Test 2479 2480 - DRA: when the scheduler has to deallocate a claim after a node became unsuitable for a pod, it might have needed more attempts than really necessary. ([#120428](https://github.com/kubernetes/kubernetes/pull/120428), [@pohly](https://github.com/pohly)) [SIG Node and Scheduling] 2481 - E2e framework: retrying after intermittent apiserver failures was fixed in WaitForPodsResponding ([#120559](https://github.com/kubernetes/kubernetes/pull/120559), [@pohly](https://github.com/pohly)) [SIG Testing] 2482 - KCM specific args can be passed with `/cluster` script, without affecting CCM. New variable name: `KUBE_CONTROLLER_MANAGER_TEST_ARGS`. ([#120524](https://github.com/kubernetes/kubernetes/pull/120524), [@jprzychodzen](https://github.com/jprzychodzen)) [SIG Cloud Provider] 2483 - This contains the modified windows kubeproxy testcases with mock implementation ([#120105](https://github.com/kubernetes/kubernetes/pull/120105), [@princepereira](https://github.com/princepereira)) [SIG Network and Windows] 2484 2485 ### Bug or Regression 2486 2487 - Added a redundant process to remove tracking finalizers from Pods that belong to Jobs. The process kicks in after the control plane marks a Job as finished ([#119944](https://github.com/kubernetes/kubernetes/pull/119944), [@Sharpz7](https://github.com/Sharpz7)) [SIG Apps] 2488 - Allow specifying ExternalTrafficPolicy for Services with ExternalIPs. ([#119150](https://github.com/kubernetes/kubernetes/pull/119150), [@tnqn](https://github.com/tnqn)) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Release and Testing] 2489 - Exclude nodes from daemonset rolling update if the scheduling constraints are not met. This eliminates the problem of rolling update stuck of daemonset with tolerations. ([#119317](https://github.com/kubernetes/kubernetes/pull/119317), [@mochizuki875](https://github.com/mochizuki875)) [SIG Apps and Testing] 2490 - Fix OpenAPI v3 not being cleaned up after deleting APIServices ([#120108](https://github.com/kubernetes/kubernetes/pull/120108), [@tnqn](https://github.com/tnqn)) [SIG API Machinery and Testing] 2491 - Fix a 1.28 regression in scheduler: a pod with concurrent events could incorrectly get moved to the unschedulable queue where it could got stuck until the next periodic purging after 5 minutes if there was no other event for it. ([#120413](https://github.com/kubernetes/kubernetes/pull/120413), [@pohly](https://github.com/pohly)) [SIG Scheduling] 2492 - Fix a bug in cronjob controller where already created jobs may be missing from the status. ([#120649](https://github.com/kubernetes/kubernetes/pull/120649), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps] 2493 - Fix a concurrent map access in TopologyCache's `HasPopulatedHints` method. ([#118189](https://github.com/kubernetes/kubernetes/pull/118189), [@Miciah](https://github.com/Miciah)) [SIG Apps and Network] 2494 - Fix kubectl events doesn't filter events by GroupVersion for resource with full name. ([#120119](https://github.com/kubernetes/kubernetes/pull/120119), [@Ithrael](https://github.com/Ithrael)) [SIG CLI and Testing] 2495 - Fixed CEL estimated cost of `replace()` to handle a zero length replacement string correctly. 2496 Previously this would cause the estimated cost to be higher than it should be. ([#120097](https://github.com/kubernetes/kubernetes/pull/120097), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery] 2497 - Fixed a 1.26 regression scheduling bug by ensuring that preemption is skipped when a PreFilter plugin returns `UnschedulableAndUnresolvable` ([#119778](https://github.com/kubernetes/kubernetes/pull/119778), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling and Testing] 2498 - Fixed a 1.27 scheduling regression that PostFilter plugin may not function if previous PreFilter plugins return Skip ([#119769](https://github.com/kubernetes/kubernetes/pull/119769), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling and Testing] 2499 - Fixed a 1.28 regression around restarting init containers in the right order relative to normal containers ([#120281](https://github.com/kubernetes/kubernetes/pull/120281), [@gjkim42](https://github.com/gjkim42)) [SIG Node and Testing] 2500 - Fixed a regression in default 1.27 configurations in kube-apiserver: fixed the AggregatedDiscoveryEndpoint feature (beta in 1.27+) to successfully fetch discovery information from aggregated API servers that do not check `Accept` headers when serving the `/apis` endpoint ([#119870](https://github.com/kubernetes/kubernetes/pull/119870), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery] 2501 - Fixed an issue where a CronJob could fail to clean up Jobs when the ResourceQuota for Jobs had been reached. ([#119776](https://github.com/kubernetes/kubernetes/pull/119776), [@ASverdlov](https://github.com/ASverdlov)) [SIG Apps] 2502 - Fixes a 1.28 regression handling negative index json patches ([#120327](https://github.com/kubernetes/kubernetes/pull/120327), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 2503 - Fixes a bug where Services using finalizers may hold onto ClusterIP and/or NodePort allocated resources for longer than expected if the finalizer is removed using the status subresource ([#120623](https://github.com/kubernetes/kubernetes/pull/120623), [@aojea](https://github.com/aojea)) [SIG Network and Testing] 2504 - Fixes an issue where StatefulSet might not restart a pod after eviction or node failure. ([#120398](https://github.com/kubernetes/kubernetes/pull/120398), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) [SIG Apps] 2505 - Fixes an issue with the garbagecollection controller registering duplicate event handlers if discovery requests fail. ([#117992](https://github.com/kubernetes/kubernetes/pull/117992), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Apps] 2506 - Fixes the bug when images pinned by the container runtime can be garbage collected by kubelet ([#119986](https://github.com/kubernetes/kubernetes/pull/119986), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG Node] 2507 - Fixing issue with incremental id generation for loadbalancer and endpoint in Kubeproxy mock test framework. ([#120723](https://github.com/kubernetes/kubernetes/pull/120723), [@princepereira](https://github.com/princepereira)) [SIG Network and Windows] 2508 - If a watch with the `progressNotify` option set is to be created, and the registry hasn't provided a `newFunc`, return an error. ([#120212](https://github.com/kubernetes/kubernetes/pull/120212), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery] 2509 - Improved handling of jsonpath expressions for kubectl wait --for. It is now possible to use simple filter expressions which match on a field's content. ([#118748](https://github.com/kubernetes/kubernetes/pull/118748), [@andreaskaris](https://github.com/andreaskaris)) [SIG CLI and Testing] 2510 - Incorporating feedback on PR #119341 ([#120087](https://github.com/kubernetes/kubernetes/pull/120087), [@divyasri537](https://github.com/divyasri537)) [SIG API Machinery] 2511 - Kubeadm: Use universal deserializer to decode static pod. ([#120549](https://github.com/kubernetes/kubernetes/pull/120549), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 2512 - Kubeadm: fix nil pointer when etcd member is already removed ([#119753](https://github.com/kubernetes/kubernetes/pull/119753), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 2513 - Kubeadm: fix the bug that `--image-repository` flag is missing for some init phase sub-commands ([#120072](https://github.com/kubernetes/kubernetes/pull/120072), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2514 - Kubeadm: improve the logic that checks whether a systemd service exists. ([#120514](https://github.com/kubernetes/kubernetes/pull/120514), [@fengxsong](https://github.com/fengxsong)) [SIG Cluster Lifecycle] 2515 - Kubeadm: print the default component configs for `reset` and `join` is now not supported ([#119346](https://github.com/kubernetes/kubernetes/pull/119346), [@chendave](https://github.com/chendave)) [SIG Cluster Lifecycle] 2516 - Kubeadm: remove 'system:masters' organization from etcd/healthcheck-client certificate. ([#119859](https://github.com/kubernetes/kubernetes/pull/119859), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2517 - Kubectl prune v2: Switch annotation from `contains-group-resources` to `contains-group-kinds`, 2518 because this is what we defined in the KEP and is clearer to end-users. Although the functionality is 2519 in alpha, we will recognize the prior annotation; this migration support will be removed in beta/GA. ([#118942](https://github.com/kubernetes/kubernetes/pull/118942), [@justinsb](https://github.com/justinsb)) [SIG CLI] 2520 - Kubectl will not print events if --show-events=false argument is passed to describe PVC subcommand. ([#120380](https://github.com/kubernetes/kubernetes/pull/120380), [@MaGaroo](https://github.com/MaGaroo)) [SIG CLI] 2521 - More accurate requeueing in scheduling queue for Pods rejected by the temporal failure (e.g., temporal failure on kube-apiserver.) ([#119105](https://github.com/kubernetes/kubernetes/pull/119105), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling and Testing] 2522 - No-op and GC related updates to cluster trust bundles no longer require attest authorization when the ClusterTrustBundleAttest plugin is enabled. ([#120779](https://github.com/kubernetes/kubernetes/pull/120779), [@enj](https://github.com/enj)) [SIG Auth] 2523 - Reintroduce resourcequota.NewMonitor constructor for other consumers ([#120777](https://github.com/kubernetes/kubernetes/pull/120777), [@atiratree](https://github.com/atiratree)) [SIG Apps] 2524 - Scheduler: Fix field apiVersion is missing from events reported from taint manager ([#114095](https://github.com/kubernetes/kubernetes/pull/114095), [@aimuz](https://github.com/aimuz)) [SIG Apps, Node and Scheduling] 2525 - Service Controller: update load balancer hosts after node's ProviderID is updated ([#120492](https://github.com/kubernetes/kubernetes/pull/120492), [@cezarygerard](https://github.com/cezarygerard)) [SIG Cloud Provider and Network] 2526 - Setting the `status.loadBalancer` of a Service whose `spec.type` is not `"LoadBalancer"` was previously allowed, but any update to the `metadata` or `spec` would wipe that field. Setting this field is no longer permitted unless `spec.type` is `"LoadBalancer"`. In the very unlikely event that this has unexpected impact, you can enable the `AllowServiceLBStatusOnNonLB` feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you need it. ([#119789](https://github.com/kubernetes/kubernetes/pull/119789), [@thockin](https://github.com/thockin)) [SIG Apps and Testing] 2527 - Sometimes, the scheduler incorrectly placed a pod in the "unschedulable" queue instead of the "backoff" queue. This happened when some plugin previously declared the pod as "unschedulable" and then in a later attempt encounters some other error. Scheduling of that pod then got delayed by up to five minutes, after which periodic flushing moved the pod back into the "active" queue. ([#120334](https://github.com/kubernetes/kubernetes/pull/120334), [@pohly](https://github.com/pohly)) [SIG Scheduling] 2528 - The `--bind-address` parameter in kube-proxy is misleading, no port is opened with this address. Instead it is translated internally to "nodeIP". The nodeIPs for both families are now taken from the Node object if `--bind-address` is unspecified or set to the "any" address (0.0.0.0 or ::). It is recommended to leave `--bind-address` unspecified, and in particular avoid to set it to localhost (127.0.0.1 or ::1) ([#119525](https://github.com/kubernetes/kubernetes/pull/119525), [@uablrek](https://github.com/uablrek)) [SIG Network and Scalability] 2529 2530 ### Other (Cleanup or Flake) 2531 2532 - Add context to "caches populated" log messages. ([#119796](https://github.com/kubernetes/kubernetes/pull/119796), [@sttts](https://github.com/sttts)) [SIG API Machinery] 2533 - Add download the cni binary for the corresponding arch in local-up-cluster.sh ([#120312](https://github.com/kubernetes/kubernetes/pull/120312), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG Network and Node] 2534 - Changes behavior of kube-proxy by allowing to set sysctl values lower than the existing one. ([#120448](https://github.com/kubernetes/kubernetes/pull/120448), [@aroradaman](https://github.com/aroradaman)) [SIG Network] 2535 - Clean up kube-apiserver http logs for impersonated requests. ([#119795](https://github.com/kubernetes/kubernetes/pull/119795), [@sttts](https://github.com/sttts)) [SIG API Machinery] 2536 - Dynamic resource allocation: avoid creating a new gRPC connection for every call of prepare/unprepare resource(s) ([#118619](https://github.com/kubernetes/kubernetes/pull/118619), [@TommyStarK](https://github.com/TommyStarK)) [SIG Node] 2537 - Fixes an issue where the vsphere cloud provider will not trust a certificate if: 2538 - The issuer of the certificate is unknown (x509.UnknownAuthorityError) 2539 - The requested name does not match the set of authorized names (x509.HostnameError) 2540 - The error surfaced after attempting a connection contains one of the substrings: "certificate is not trusted" or "certificate signed by unknown authority" ([#120736](https://github.com/kubernetes/kubernetes/pull/120736), [@MadhavJivrajani](https://github.com/MadhavJivrajani)) [SIG Architecture and Cloud Provider] 2541 - Fixes bug where Adding GroupVersion log line is constantly repeated without any group version changes ([#119825](https://github.com/kubernetes/kubernetes/pull/119825), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery] 2542 - Generated ResourceClaim names are now more readable because of an additional hyphen before the random suffix (`<pod name>-<claim name>-<random suffix>` ). ([#120336](https://github.com/kubernetes/kubernetes/pull/120336), [@pohly](https://github.com/pohly)) [SIG Apps and Node] 2543 - Improve memory usage of kube-controller-manager by dropping the `.metadata.managedFields` field that kube-controller-manager doesn't require. ([#118455](https://github.com/kubernetes/kubernetes/pull/118455), [@linxiulei](https://github.com/linxiulei)) [SIG API Machinery and Cloud Provider] 2544 - Kubeadm: remove 'system:masters' organization from apiserver-etcd-client certificate ([#120521](https://github.com/kubernetes/kubernetes/pull/120521), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2545 - Kubeadm: updated warning message when swap space is detected. When swap is active on Linux, kubeadm explains that swap is supported for cgroup v2 only and is beta but disabled by default. ([#120198](https://github.com/kubernetes/kubernetes/pull/120198), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 2546 - Makefile and scripts now respect GOTOOLCHAIN and otherwise ensure ./.go-version is used ([#120279](https://github.com/kubernetes/kubernetes/pull/120279), [@BenTheElder](https://github.com/BenTheElder)) [SIG Release] 2547 - Optimized NodeUnschedulable Filter to avoid unnecessary calculations ([#119399](https://github.com/kubernetes/kubernetes/pull/119399), [@wackxu](https://github.com/wackxu)) [SIG Scheduling] 2548 - Previously, the pod name and namespace were eliminated in the event log message. This PR attempts to add the preemptor pod UID in the preemption event message logs for easier debugging and safer transparency. ([#119971](https://github.com/kubernetes/kubernetes/pull/119971), [@kwakubiney](https://github.com/kwakubiney)) [SIG Scheduling] 2549 - Promote to conformance a test that verify that Services only forward traffic on the port and protocol specified. ([#120069](https://github.com/kubernetes/kubernetes/pull/120069), [@aojea](https://github.com/aojea)) [SIG Architecture, Network and Testing] 2550 - Remove ephemeral container legacy server support for the server versions prior to 1.22 ([#119537](https://github.com/kubernetes/kubernetes/pull/119537), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI] 2551 - Scheduler: handling of unschedulable pods because a ResourceClass is missing is a bit more efficient and no longer relies on periodic retries ([#120213](https://github.com/kubernetes/kubernetes/pull/120213), [@pohly](https://github.com/pohly)) [SIG Node, Scheduling and Testing] 2552 - Set the resolution for the job_controller_job_sync_duration_seconds metric from 4ms to 1min ([#120577](https://github.com/kubernetes/kubernetes/pull/120577), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and Instrumentation] 2553 - Statefulset should wait for new replicas in tests when removing .start.ordinal ([#119761](https://github.com/kubernetes/kubernetes/pull/119761), [@soltysh](https://github.com/soltysh)) [SIG Apps and Testing] 2554 - The `horizontalpodautoscaling` and `clusterrole-aggregation` controllers now assume the `autoscaling/v1` and `rbac.authorization.k8s.io/v1` APIs are available. If you disable those APIs and do not want to run those controllers, exclude them by passing `--controllers=-horizontalpodautoscaling` or `--controllers=-clusterrole-aggregation` to `kube-controller-manager`. ([#117977](https://github.com/kubernetes/kubernetes/pull/117977), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Cloud Provider] 2555 - The metrics controlled by the ComponentSLIs feature-gate and served at /metrics/slis are now GA and unconditionally enabled. The feature-gate will be removed in 1.31. ([#120574](https://github.com/kubernetes/kubernetes/pull/120574), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation, Network, Node and Scheduling] 2556 - Updated CNI plugins to v1.3.0. ([#119969](https://github.com/kubernetes/kubernetes/pull/119969), [@saschagrunert](https://github.com/saschagrunert)) [SIG Cloud Provider, Node and Testing] 2557 - Updated cri-tools to v1.28.0. ([#119933](https://github.com/kubernetes/kubernetes/pull/119933), [@saschagrunert](https://github.com/saschagrunert)) [SIG Cloud Provider] 2558 - Updated distroless-iptables to use registry.k8s.io/build-image/distroless-iptables:v0.3.1 ([#120352](https://github.com/kubernetes/kubernetes/pull/120352), [@saschagrunert](https://github.com/saschagrunert)) [SIG Release and Testing] 2559 - Upgrade coredns to v1.11.1 ([#120116](https://github.com/kubernetes/kubernetes/pull/120116), [@tukwila](https://github.com/tukwila)) [SIG Cloud Provider and Cluster Lifecycle] 2560 - ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding objects are persisted in etcd using the v1beta1 version. Remove alpha objects or disable the alpha ValidatingAdmissionPolicy feature in a 1.27 server before upgrading to a 1.28 server with the beta feature and API enabled. ([#120018](https://github.com/kubernetes/kubernetes/pull/120018), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing] 2561 - Yes, kubectl will not support the "/swagger-2.0.0.pb-v1" endpoint that has been long deprecated ([#119410](https://github.com/kubernetes/kubernetes/pull/119410), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery] 2562 2563 ## Dependencies 2564 2565 ### Added 2566 - github.com/distribution/reference: [v0.5.0](https://github.com/distribution/reference/tree/v0.5.0) 2567 2568 ### Changed 2569 - github.com/coredns/corefile-migration: [v1.0.20 → v1.0.21](https://github.com/coredns/corefile-migration/compare/v1.0.20...v1.0.21) 2570 - github.com/docker/distribution: [v2.8.2+incompatible → v2.8.1+incompatible](https://github.com/docker/distribution/compare/v2.8.2...v2.8.1) 2571 - github.com/evanphx/json-patch: [v5.6.0+incompatible → v4.12.0+incompatible](https://github.com/evanphx/json-patch/compare/v5.6.0...v4.12.0) 2572 - github.com/google/cel-go: [v0.16.0 → v0.17.6](https://github.com/google/cel-go/compare/v0.16.0...v0.17.6) 2573 - github.com/gorilla/websocket: [v1.4.2 → v1.5.0](https://github.com/gorilla/websocket/compare/v1.4.2...v1.5.0) 2574 - github.com/opencontainers/runc: [v1.1.7 → v1.1.9](https://github.com/opencontainers/runc/compare/v1.1.7...v1.1.9) 2575 - github.com/opencontainers/selinux: [v1.10.0 → v1.11.0](https://github.com/opencontainers/selinux/compare/v1.10.0...v1.11.0) 2576 - github.com/vmware/govmomi: [v0.30.0 → v0.30.6](https://github.com/vmware/govmomi/compare/v0.30.0...v0.30.6) 2577 - google.golang.org/protobuf: v1.30.0 → v1.31.0 2578 - k8s.io/gengo: c0856e2 → 9cce18d 2579 - k8s.io/kube-openapi: 2695361 → d090da1 2580 - k8s.io/utils: d93618c → 3b25d92 2581 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.1.2 → v0.28.0 2582 - sigs.k8s.io/structured-merge-diff/v4: v4.2.3 → v4.3.0 2583 2584 ### Removed 2585 _Nothing has changed._