k8s.io/kubernetes@v1.31.0-alpha.0.0.20240520171757-56147500dadc/CHANGELOG/CHANGELOG-1.28.md (about) 1 <!-- BEGIN MUNGE: GENERATED_TOC --> 2 3 - [v1.28.10](#v12810) 4 - [Downloads for v1.28.10](#downloads-for-v12810) 5 - [Source Code](#source-code) 6 - [Client Binaries](#client-binaries) 7 - [Server Binaries](#server-binaries) 8 - [Node Binaries](#node-binaries) 9 - [Container Images](#container-images) 10 - [Changelog since v1.28.9](#changelog-since-v1289) 11 - [Changes by Kind](#changes-by-kind) 12 - [Bug or Regression](#bug-or-regression) 13 - [Other (Cleanup or Flake)](#other-cleanup-or-flake) 14 - [Dependencies](#dependencies) 15 - [Added](#added) 16 - [Changed](#changed) 17 - [Removed](#removed) 18 - [v1.28.9](#v1289) 19 - [Downloads for v1.28.9](#downloads-for-v1289) 20 - [Source Code](#source-code-1) 21 - [Client Binaries](#client-binaries-1) 22 - [Server Binaries](#server-binaries-1) 23 - [Node Binaries](#node-binaries-1) 24 - [Container Images](#container-images-1) 25 - [Changelog since v1.28.8](#changelog-since-v1288) 26 - [Important Security Information](#important-security-information) 27 - [CVE-2024-3177: Bypassing mountable secrets policy imposed by the ServiceAccount admission plugin](#cve-2024-3177-bypassing-mountable-secrets-policy-imposed-by-the-serviceaccount-admission-plugin) 28 - [Changes by Kind](#changes-by-kind-1) 29 - [Feature](#feature) 30 - [Bug or Regression](#bug-or-regression-1) 31 - [Dependencies](#dependencies-1) 32 - [Added](#added-1) 33 - [Changed](#changed-1) 34 - [Removed](#removed-1) 35 - [v1.28.8](#v1288) 36 - [Downloads for v1.28.8](#downloads-for-v1288) 37 - [Source Code](#source-code-2) 38 - [Client Binaries](#client-binaries-2) 39 - [Server Binaries](#server-binaries-2) 40 - [Node Binaries](#node-binaries-2) 41 - [Container Images](#container-images-2) 42 - [Changelog since v1.28.7](#changelog-since-v1287) 43 - [Changes by Kind](#changes-by-kind-2) 44 - [Feature](#feature-1) 45 - [Bug or Regression](#bug-or-regression-2) 46 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-1) 47 - [Dependencies](#dependencies-2) 48 - [Added](#added-2) 49 - [Changed](#changed-2) 50 - [Removed](#removed-2) 51 - [v1.28.7](#v1287) 52 - [Downloads for v1.28.7](#downloads-for-v1287) 53 - [Source Code](#source-code-3) 54 - [Client Binaries](#client-binaries-3) 55 - [Server Binaries](#server-binaries-3) 56 - [Node Binaries](#node-binaries-3) 57 - [Container Images](#container-images-3) 58 - [Changelog since v1.28.6](#changelog-since-v1286) 59 - [Changes by Kind](#changes-by-kind-3) 60 - [Feature](#feature-2) 61 - [Bug or Regression](#bug-or-regression-3) 62 - [Dependencies](#dependencies-3) 63 - [Added](#added-3) 64 - [Changed](#changed-3) 65 - [Removed](#removed-3) 66 - [v1.28.6](#v1286) 67 - [Downloads for v1.28.6](#downloads-for-v1286) 68 - [Source Code](#source-code-4) 69 - [Client Binaries](#client-binaries-4) 70 - [Server Binaries](#server-binaries-4) 71 - [Node Binaries](#node-binaries-4) 72 - [Container Images](#container-images-4) 73 - [Changelog since v1.28.5](#changelog-since-v1285) 74 - [Changes by Kind](#changes-by-kind-4) 75 - [Feature](#feature-3) 76 - [Bug or Regression](#bug-or-regression-4) 77 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-2) 78 - [Dependencies](#dependencies-4) 79 - [Added](#added-4) 80 - [Changed](#changed-4) 81 - [Removed](#removed-4) 82 - [v1.28.5](#v1285) 83 - [Downloads for v1.28.5](#downloads-for-v1285) 84 - [Source Code](#source-code-5) 85 - [Client Binaries](#client-binaries-5) 86 - [Server Binaries](#server-binaries-5) 87 - [Node Binaries](#node-binaries-5) 88 - [Container Images](#container-images-5) 89 - [Changelog since v1.28.4](#changelog-since-v1284) 90 - [Changes by Kind](#changes-by-kind-5) 91 - [Feature](#feature-4) 92 - [Bug or Regression](#bug-or-regression-5) 93 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-3) 94 - [Dependencies](#dependencies-5) 95 - [Added](#added-5) 96 - [Changed](#changed-5) 97 - [Removed](#removed-5) 98 - [v1.28.4](#v1284) 99 - [Downloads for v1.28.4](#downloads-for-v1284) 100 - [Source Code](#source-code-6) 101 - [Client Binaries](#client-binaries-6) 102 - [Server Binaries](#server-binaries-6) 103 - [Node Binaries](#node-binaries-6) 104 - [Container Images](#container-images-6) 105 - [Changelog since v1.28.3](#changelog-since-v1283) 106 - [Important Security Information](#important-security-information-1) 107 - [CVE-2023-5528: Insufficient input sanitization in in-tree storage plugin leads to privilege escalation on Windows nodes](#cve-2023-5528-insufficient-input-sanitization-in-in-tree-storage-plugin-leads-to-privilege-escalation-on-windows-nodes) 108 - [Changes by Kind](#changes-by-kind-6) 109 - [API Change](#api-change) 110 - [Feature](#feature-5) 111 - [Bug or Regression](#bug-or-regression-6) 112 - [Dependencies](#dependencies-6) 113 - [Added](#added-6) 114 - [Changed](#changed-6) 115 - [Removed](#removed-6) 116 - [v1.28.3](#v1283) 117 - [Downloads for v1.28.3](#downloads-for-v1283) 118 - [Source Code](#source-code-7) 119 - [Client Binaries](#client-binaries-7) 120 - [Server Binaries](#server-binaries-7) 121 - [Node Binaries](#node-binaries-7) 122 - [Container Images](#container-images-7) 123 - [Changelog since v1.28.2](#changelog-since-v1282) 124 - [Changes by Kind](#changes-by-kind-7) 125 - [Feature](#feature-6) 126 - [Failing Test](#failing-test) 127 - [Bug or Regression](#bug-or-regression-7) 128 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-4) 129 - [Dependencies](#dependencies-7) 130 - [Added](#added-7) 131 - [Changed](#changed-7) 132 - [Removed](#removed-7) 133 - [v1.28.2](#v1282) 134 - [Downloads for v1.28.2](#downloads-for-v1282) 135 - [Source Code](#source-code-8) 136 - [Client Binaries](#client-binaries-8) 137 - [Server Binaries](#server-binaries-8) 138 - [Node Binaries](#node-binaries-8) 139 - [Container Images](#container-images-8) 140 - [Changelog since v1.28.1](#changelog-since-v1281) 141 - [Changes by Kind](#changes-by-kind-8) 142 - [API Change](#api-change-1) 143 - [Feature](#feature-7) 144 - [Bug or Regression](#bug-or-regression-8) 145 - [Dependencies](#dependencies-8) 146 - [Added](#added-8) 147 - [Changed](#changed-8) 148 - [Removed](#removed-8) 149 - [v1.28.1](#v1281) 150 - [Downloads for v1.28.1](#downloads-for-v1281) 151 - [Source Code](#source-code-9) 152 - [Client Binaries](#client-binaries-9) 153 - [Server Binaries](#server-binaries-9) 154 - [Node Binaries](#node-binaries-9) 155 - [Container Images](#container-images-9) 156 - [Changelog since v1.28.0](#changelog-since-v1280) 157 - [Important Security Information](#important-security-information-2) 158 - [CVE-2023-3955: Insufficient input sanitization on Windows nodes leads to privilege escalation](#cve-2023-3955-insufficient-input-sanitization-on-windows-nodes-leads-to-privilege-escalation) 159 - [CVE-2023-3676: Insufficient input sanitization on Windows nodes leads to privilege escalation](#cve-2023-3676-insufficient-input-sanitization-on-windows-nodes-leads-to-privilege-escalation) 160 - [Changes by Kind](#changes-by-kind-9) 161 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-5) 162 - [Dependencies](#dependencies-9) 163 - [Added](#added-9) 164 - [Changed](#changed-9) 165 - [Removed](#removed-9) 166 - [v1.28.0](#v1280) 167 - [Downloads for v1.28.0](#downloads-for-v1280) 168 - [Source Code](#source-code-10) 169 - [Client Binaries](#client-binaries-10) 170 - [Server Binaries](#server-binaries-10) 171 - [Node Binaries](#node-binaries-10) 172 - [Container Images](#container-images-10) 173 - [Changelog since v1.27.0](#changelog-since-v1270) 174 - [Urgent Upgrade Notes](#urgent-upgrade-notes) 175 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade) 176 - [Changes by Kind](#changes-by-kind-10) 177 - [Deprecation](#deprecation) 178 - [API Change](#api-change-2) 179 - [Feature](#feature-8) 180 - [Documentation](#documentation) 181 - [Failing Test](#failing-test-1) 182 - [Bug or Regression](#bug-or-regression-9) 183 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-6) 184 - [Dependencies](#dependencies-10) 185 - [Added](#added-10) 186 - [Changed](#changed-10) 187 - [Removed](#removed-10) 188 - [v1.28.0-rc.1](#v1280-rc1) 189 - [Downloads for v1.28.0-rc.1](#downloads-for-v1280-rc1) 190 - [Source Code](#source-code-11) 191 - [Client Binaries](#client-binaries-11) 192 - [Server Binaries](#server-binaries-11) 193 - [Node Binaries](#node-binaries-11) 194 - [Container Images](#container-images-11) 195 - [Changelog since v1.28.0-rc.0](#changelog-since-v1280-rc0) 196 - [Changes by Kind](#changes-by-kind-11) 197 - [API Change](#api-change-3) 198 - [Feature](#feature-9) 199 - [Bug or Regression](#bug-or-regression-10) 200 - [Dependencies](#dependencies-11) 201 - [Added](#added-11) 202 - [Changed](#changed-11) 203 - [Removed](#removed-11) 204 - [v1.28.0-rc.0](#v1280-rc0) 205 - [Downloads for v1.28.0-rc.0](#downloads-for-v1280-rc0) 206 - [Source Code](#source-code-12) 207 - [Client Binaries](#client-binaries-12) 208 - [Server Binaries](#server-binaries-12) 209 - [Node Binaries](#node-binaries-12) 210 - [Container Images](#container-images-12) 211 - [Changelog since v1.28.0-beta.0](#changelog-since-v1280-beta0) 212 - [Changes by Kind](#changes-by-kind-12) 213 - [API Change](#api-change-4) 214 - [Feature](#feature-10) 215 - [Dependencies](#dependencies-12) 216 - [Added](#added-12) 217 - [Changed](#changed-12) 218 - [Removed](#removed-12) 219 - [v1.28.0-beta.0](#v1280-beta0) 220 - [Downloads for v1.28.0-beta.0](#downloads-for-v1280-beta0) 221 - [Source Code](#source-code-13) 222 - [Client Binaries](#client-binaries-13) 223 - [Server Binaries](#server-binaries-13) 224 - [Node Binaries](#node-binaries-13) 225 - [Container Images](#container-images-13) 226 - [Changelog since v1.28.0-alpha.4](#changelog-since-v1280-alpha4) 227 - [Changes by Kind](#changes-by-kind-13) 228 - [Deprecation](#deprecation-1) 229 - [API Change](#api-change-5) 230 - [Feature](#feature-11) 231 - [Failing Test](#failing-test-2) 232 - [Bug or Regression](#bug-or-regression-11) 233 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-7) 234 - [Dependencies](#dependencies-13) 235 - [Added](#added-13) 236 - [Changed](#changed-13) 237 - [Removed](#removed-13) 238 - [v1.28.0-alpha.4](#v1280-alpha4) 239 - [Downloads for v1.28.0-alpha.4](#downloads-for-v1280-alpha4) 240 - [Source Code](#source-code-14) 241 - [Client Binaries](#client-binaries-14) 242 - [Server Binaries](#server-binaries-14) 243 - [Node Binaries](#node-binaries-14) 244 - [Container Images](#container-images-14) 245 - [Changelog since v1.28.0-alpha.3](#changelog-since-v1280-alpha3) 246 - [Urgent Upgrade Notes](#urgent-upgrade-notes-1) 247 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-1) 248 - [Changes by Kind](#changes-by-kind-14) 249 - [Deprecation](#deprecation-2) 250 - [API Change](#api-change-6) 251 - [Feature](#feature-12) 252 - [Bug or Regression](#bug-or-regression-12) 253 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-8) 254 - [Dependencies](#dependencies-14) 255 - [Added](#added-14) 256 - [Changed](#changed-14) 257 - [Removed](#removed-14) 258 - [v1.28.0-alpha.3](#v1280-alpha3) 259 - [Downloads for v1.28.0-alpha.3](#downloads-for-v1280-alpha3) 260 - [Source Code](#source-code-15) 261 - [Client Binaries](#client-binaries-15) 262 - [Server Binaries](#server-binaries-15) 263 - [Node Binaries](#node-binaries-15) 264 - [Container Images](#container-images-15) 265 - [Changelog since v1.28.0-alpha.2](#changelog-since-v1280-alpha2) 266 - [Changes by Kind](#changes-by-kind-15) 267 - [Deprecation](#deprecation-3) 268 - [API Change](#api-change-7) 269 - [Feature](#feature-13) 270 - [Bug or Regression](#bug-or-regression-13) 271 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-9) 272 - [Dependencies](#dependencies-15) 273 - [Added](#added-15) 274 - [Changed](#changed-15) 275 - [Removed](#removed-15) 276 - [v1.28.0-alpha.2](#v1280-alpha2) 277 - [Downloads for v1.28.0-alpha.2](#downloads-for-v1280-alpha2) 278 - [Source Code](#source-code-16) 279 - [Client Binaries](#client-binaries-16) 280 - [Server Binaries](#server-binaries-16) 281 - [Node Binaries](#node-binaries-16) 282 - [Container Images](#container-images-16) 283 - [Changelog since v1.28.0-alpha.1](#changelog-since-v1280-alpha1) 284 - [Urgent Upgrade Notes](#urgent-upgrade-notes-2) 285 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-2) 286 - [Changes by Kind](#changes-by-kind-16) 287 - [Feature](#feature-14) 288 - [Bug or Regression](#bug-or-regression-14) 289 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-10) 290 - [Dependencies](#dependencies-16) 291 - [Added](#added-16) 292 - [Changed](#changed-16) 293 - [Removed](#removed-16) 294 - [v1.28.0-alpha.1](#v1280-alpha1) 295 - [Downloads for v1.28.0-alpha.1](#downloads-for-v1280-alpha1) 296 - [Source Code](#source-code-17) 297 - [Client Binaries](#client-binaries-17) 298 - [Server Binaries](#server-binaries-17) 299 - [Node Binaries](#node-binaries-17) 300 - [Container Images](#container-images-17) 301 - [Changelog since v1.27.0](#changelog-since-v1270-1) 302 - [Changes by Kind](#changes-by-kind-17) 303 - [Deprecation](#deprecation-4) 304 - [API Change](#api-change-8) 305 - [Feature](#feature-15) 306 - [Documentation](#documentation-1) 307 - [Failing Test](#failing-test-3) 308 - [Bug or Regression](#bug-or-regression-15) 309 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-11) 310 - [Dependencies](#dependencies-17) 311 - [Added](#added-17) 312 - [Changed](#changed-17) 313 - [Removed](#removed-17) 314 315 <!-- END MUNGE: GENERATED_TOC --> 316 317 # v1.28.10 318 319 320 ## Downloads for v1.28.10 321 322 323 324 ### Source Code 325 326 filename | sha512 hash 327 -------- | ----------- 328 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes.tar.gz) | 364ba90386eae32daf66266d3e225a1ae03f83934554c8a928c44c53833d971f57a0fbc6336ba9b6fd5756cb0824fe984ad8c2575182775551f70b229ad192f4 329 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-src.tar.gz) | 7930797f68229f1ca6e0992abd7a2d5408e81683065a8b75120281b3252b6f735ae802ed763dad8d04e25896c266c0fe4c55318868c14cb5422133c83b445ef1 330 331 ### Client Binaries 332 333 filename | sha512 hash 334 -------- | ----------- 335 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-darwin-amd64.tar.gz) | 49c9916b0c396f25605ed0e8038d6c0655002af2318f49f405abcf658b307c1a8815fb73436365d038e4fdb99ded00ac327f31b7d0ca5d18aca02c38ed8898b6 336 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-darwin-arm64.tar.gz) | 7232583f1c8efe9c41c7131fff4cba59280413b77905c5ba871313f551e647f73c1f7d9fbc0837b321331cd6ae6b275dc4d2518441f6929463dc720e6fc07b11 337 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-386.tar.gz) | d9e8999f7bdc53f0211f78e24c548f04b02de75a1dede8a1f91ec53166c262baf5def6d32eccd48daef9116c8425cb8b0a9a0ffe083221a6deaba2ed3eb181e6 338 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-amd64.tar.gz) | bbd684de821e294e4f39c9a577669a711ca7f1b47251dd5cae02dab62a4bc7347545b02acf61e97e0b4bcfc025710f186816a35470bdce6184a686c09216ea90 339 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-arm.tar.gz) | 84415f3bf03dae5ea72b5e02ef35cb657e564045b1cbedb8885b2b6006dca697530074aef28be946c36962a34c0ab105f56838afafc817d41572c70330237daa 340 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-arm64.tar.gz) | 196621add12081126edfda3a272109e8e2dcbb057732350086938b96b1cf32ae115b117b038da25565e3e3f9d22e05a123797115078e002470e4b91a6ae6ad26 341 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-ppc64le.tar.gz) | 62df1b45d8cd2b25f7cecd2308af7090aaadf2991044b0fa7a0d1a7b46b084a8194f5520f8b32be795e3fa6b19cd4fa1e983e3d0e916295634611a2f5cc36067 342 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-s390x.tar.gz) | 2bdf394382f78c454c73e4c3b0f8edc2904edbae994675ae4909edd9772e77e74a26d0e37dbf5cd5541c88066b5cd1e0eff5cc96154ece57132a2546fe08ec7b 343 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-windows-386.tar.gz) | d92d0afbb1de03bed99865393ee6a5c57c2cae4bd89edd197c117d031acde6c9a7e62646ec8d3756dfee3d592e624597565cad800e77348049fc787c8887144d 344 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-windows-amd64.tar.gz) | 5f9ee48813f2b098adf47c4ddb384596e757581d13b5384768bb353eb3520a83447f54e4c826628dabbea4541419516b6771f923bbc8db345a2017d0b7b7bfec 345 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-windows-arm64.tar.gz) | 0cbeaa95d1d5cf2cae2ea38bf6d71ea5a63ec7fc364c4066be12c27533dc2ad699ef79fe7ba30eda293c1eb5098426cecb04ca98d7234b89be6a989aff9449a2 346 347 ### Server Binaries 348 349 filename | sha512 hash 350 -------- | ----------- 351 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-server-linux-amd64.tar.gz) | 06245a0aea642bc38baa1549f4aaf51079252fc0f0b1346a90aadf5c9c260f1e6df1ad81fb6b00f81380270864e6a192e848cb88923a14016f8e4c7ffd097a32 352 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-server-linux-arm64.tar.gz) | 4b704f3e8099e783fd317583f32bb2be8c2916f719fbba6ba2760eddf62834819e737eb4bd960b6711a0279e7622d94c165aca67df531ce69a3813f7351fcbf6 353 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-server-linux-ppc64le.tar.gz) | 384287b9de5d37c4f6c3913751326ac8f6cc5d80a104aec190de10bd0762638308c9df058b2ca7e2d2205dfac9db7d8e418b1110b8bde6fd030bc024030eb7e9 354 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-server-linux-s390x.tar.gz) | 8dc966852a3518e59651b0e9d3e1a1af7ce6dd8a40b5cabcb8fca57275b35f792293a12c652eeb6f55d20869589753928973367fda3caf6b659c9aa1927b72ae 355 356 ### Node Binaries 357 358 filename | sha512 hash 359 -------- | ----------- 360 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-linux-amd64.tar.gz) | bffb3266c0c11c4831879ed25443ee5c1ec14b3c444140527119a887c6122f4a9a9829e5746e309e8f415a68efaeb948987f89601fde734f8c5fc889c5d0f27e 361 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-linux-arm64.tar.gz) | b6d179291f827f91b73d4cba1131f3af23000714a121292c3daaa79d15006c9d71749c1de8484b5d4ae8f6ad352c460e704f0a1be7f62d471403341a90826f03 362 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-linux-ppc64le.tar.gz) | 05605bb0ed75a6b53cc8d6d009c9705d41596b680d7b209bfc3154897da580a03b8ba46820a56de86d7f1e1b2d0d981d6efbfc279a216aa9cfccea318a41658c 363 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-linux-s390x.tar.gz) | a43f1feeadef4384f18e8aa045c3996cefa9c7680c37e93364aba3071610908712fa3bf82cd8770c6f26b62ca8705cd162ebac623c9bdd53cbf6f83694aabb94 364 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-windows-amd64.tar.gz) | 2280896ca4e762e2ffb363e94ebf881c54b0b4a7b36ee21fe305963646295683bc88aab8b831092736a5480753055cb8d7ce780571901d24ccbb99df0fde9948 365 366 ### Container Images 367 368 All container images are available as manifest lists and support the described 369 architectures. It is also possible to pull a specific architecture directly by 370 adding the "-$ARCH" suffix to the container image name. 371 372 name | architectures 373 ---- | ------------- 374 [registry.k8s.io/conformance:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x) 375 [registry.k8s.io/kube-apiserver:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x) 376 [registry.k8s.io/kube-controller-manager:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x) 377 [registry.k8s.io/kube-proxy:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x) 378 [registry.k8s.io/kube-scheduler:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x) 379 [registry.k8s.io/kubectl:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x) 380 381 ## Changelog since v1.28.9 382 383 ## Changes by Kind 384 385 ### Bug or Regression 386 387 - Fixed PersistentolumeLabel providing wrong topology labels to Azure Disk PersistentVolumes when the external Azure cloud provider is used. ([#124528](https://github.com/kubernetes/kubernetes/pull/124528), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider] 388 - Fixed a bug that a pod may remain unscheduled if any PreFilter plugin returns nodes that do not exist. ([#124764](https://github.com/kubernetes/kubernetes/pull/124764), [@chengjoey](https://github.com/chengjoey)) [SIG Scheduling and Testing] 389 390 ### Other (Cleanup or Flake) 391 392 - For apiserver_storage_size_bytes metric, we are renaming the label for etcd to be "storage_cluster_id" instead of "cluster" to to reduce conflict and be very specific. ([#124293](https://github.com/kubernetes/kubernetes/pull/124293), [@dims](https://github.com/dims)) [SIG API Machinery, Instrumentation and Testing] 393 394 ## Dependencies 395 396 ### Added 397 _Nothing has changed._ 398 399 ### Changed 400 _Nothing has changed._ 401 402 ### Removed 403 _Nothing has changed._ 404 405 406 407 # v1.28.9 408 409 410 ## Downloads for v1.28.9 411 412 413 414 ### Source Code 415 416 filename | sha512 hash 417 -------- | ----------- 418 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes.tar.gz) | 6445c7b17f50f2244f1fb39a64662db10252ec6c054379ac1119f7c0ee96b1a97aae1d1f663164e1eff89f9d6c3b3089d81702e85e8c4fed7f835bf53db1070e 419 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-src.tar.gz) | ba7ae8b833ebc21f384dd36e5efe61b12c082342314097542da0326fc19a4d54a3cd84848be60c85bf3675718eb213216d503ca8f088084e2d77b92cc1848c6a 420 421 ### Client Binaries 422 423 filename | sha512 hash 424 -------- | ----------- 425 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-darwin-amd64.tar.gz) | 90d5663170f8bedca8c95bd71653fcb1a2e1c2a7d86b765f8c46de2531447c034560900fd9a31596b4fc2606485c0923b0496902ae9c2c1e43572243596be924 426 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-darwin-arm64.tar.gz) | 207efb9097bef48895f6e03a9c3054b376d31a9f649f31b2c5bab18a26571dc5713f1a23bfe8cd546eabdc21765c219fab313e6a26866f58fef3647052ce6ca5 427 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-386.tar.gz) | 366bc0ca6b8b6e6887a57f3b75b21da78d8688dc7c3adefdf5370eda7a49ad0251c41d06ef68e71e7841c1307678425200f9b2ccd55def3749994b0f23ca542f 428 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-amd64.tar.gz) | 5142ad0fa9d709d28e481d22442550eb5806c376382990c5e8637f7846275841bfa59ace19dc5f6276b563003ef5d7d49b06ec223ba352fc040d17a351085336 429 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-arm.tar.gz) | b77f309567bd3d828499dd7332ec485257df8a8cbc0d4d65f822c68466c2a2d07bab79317f5474826a73950955bc8af9491da215f05bbdd0d53b9367c9b53062 430 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-arm64.tar.gz) | d89b89fad313764ee3b7aa71e0b87651961e1d5485bab40cc3c0af00e9e422ffe8245501baee4c465e7cfdeb446721a28d075ce53f726ab38cdfa5aff554ef8e 431 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-ppc64le.tar.gz) | dfddba1e6db1702b8b80df9bdeead04cd72db47f84d615adc2090c851543d981b3cc9970e68e832ea73d13015ac8113ddef7247828b30da906e75129bb56a17c 432 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-s390x.tar.gz) | 0957b71eba14a1728accd1e917e81b2cc95cdb4523a519b21fe82cb15885dedffbe49df2dca75059ba7590243c557d1948339a5fd10b67f141fd066606b35b57 433 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-windows-386.tar.gz) | 12179c49f2fa31970edc3b00232b69d431500200a2f3945a3fb4ce04d458c825b1e214f9ced1c7bb06777b79ca19f86444a1bee5f2f35b60f4b3407f4fa861d7 434 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-windows-amd64.tar.gz) | 6666378dbc9a43f62bfd69ef81993c4463ef1c8862dda1b40b5b18a90a81cfb2c26f19e15d8e3e019ab1ac8140cc15e11cd5d308c172f949264df69ee335047e 435 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-windows-arm64.tar.gz) | c74973f02e46c6c21a50b9b08c7211e475a8b29ad375feb84d5c36a9b8716052f5daec77e7a6b138b045ff88701740f357c7654bc48a339debae88036bc8ae0f 436 437 ### Server Binaries 438 439 filename | sha512 hash 440 -------- | ----------- 441 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-server-linux-amd64.tar.gz) | 9672e1921f858b3e77d85d8d915ba634b6693aa65ea223fc1eba0ca97e893dc391f691b8a35fb9c17b0f07ff0f6f37cae99164c2510e36a5fad6a3cdcf33a140 442 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-server-linux-arm64.tar.gz) | 668f237dbb96fa6b50f40c5452ea02c9db19c8fc07e73818d447f72b854f4864e6fcb119529439c51aab9e3233559eca7cfbe0d65d6f733f2281380c08c8a3cd 443 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-server-linux-ppc64le.tar.gz) | 222aefa46ce11f3345f72e1ff058da797d2fc3ccd08e5a9a8d4438f7b0262e4abb87cb6d7d719b30105a574fc5e61c9378f6fd1ccb16cfe7ccf5db5e8e0f8299 444 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-server-linux-s390x.tar.gz) | 74d30a38d00843d4b90906aff0aeee067effb64f6e3e244b6fd730a016de39c5754b20127b12239ada4f23099c6f1bcb2a619b084e8a3a5478b7e9fb8465e4c8 445 446 ### Node Binaries 447 448 filename | sha512 hash 449 -------- | ----------- 450 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-linux-amd64.tar.gz) | c215d09bd69bb71ecfc81d6a4605e16c68fae940ab62880b9e3a60e84805897c1bbd29fb98fbc3908629809b7396fe6c00765d6e59c44c85666bf70371aa6b4f 451 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-linux-arm64.tar.gz) | d19a290d769491fe1d97cb416aa481bdfb7d4831a4ceec35abc90d5035f8cd529fb3e4653b1ec71cf8d0a38ce10d6e1e9d054bdee7d243cc01fcb44de94ddabe 452 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-linux-ppc64le.tar.gz) | ab00c8323ec13870a270beb0d172b8c3371c69b234a422979ac5acb68349f46dc87a65fd734305ea940985e46d60a4c4a4a2886076a31ca4a67660506582076d 453 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-linux-s390x.tar.gz) | 60dcdb46e9a0b35505e06725eac88b590e62a97ad978573dfc98392a57538ff0d1c8ec15449cc8f47747c97bd29bda75302599443f7f969eb99eba9cbd78c27e 454 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-windows-amd64.tar.gz) | 7c2d2f8fdefae24583de5eecbb61165196508f91db57ec7d03eb9c61c02f61b440b1472e0d7619dbd81d94c2aefe04613db59e25e12cd4792cec86aace76d3d4 455 456 ### Container Images 457 458 All container images are available as manifest lists and support the described 459 architectures. It is also possible to pull a specific architecture directly by 460 adding the "-$ARCH" suffix to the container image name. 461 462 name | architectures 463 ---- | ------------- 464 [registry.k8s.io/conformance:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x) 465 [registry.k8s.io/kube-apiserver:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x) 466 [registry.k8s.io/kube-controller-manager:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x) 467 [registry.k8s.io/kube-proxy:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x) 468 [registry.k8s.io/kube-scheduler:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x) 469 [registry.k8s.io/kubectl:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x) 470 471 ## Changelog since v1.28.8 472 473 ## Important Security Information 474 475 This release contains changes that address the following vulnerabilities: 476 477 ### CVE-2024-3177: Bypassing mountable secrets policy imposed by the ServiceAccount admission plugin 478 479 A security issue was discovered in Kubernetes where users may be able to launch containers that bypass the mountable secrets policy enforced by the ServiceAccount admission plugin when using containers, init containers, and ephemeral containers with the envFrom field populated. 480 481 **Affected Versions**: 482 - kube-apiserver v1.29.0 - v1.29.3 483 - kube-apiserver v1.28.0 - v1.28.8 484 - kube-apiserver <= v1.27.12 485 486 **Fixed Versions**: 487 - kube-apiserver v1.29.4 488 - kube-apiserver v1.28.9 489 - kube-apiserver v1.27.13 490 491 This vulnerability was reported by tha3e1vl. 492 493 494 **CVSS Rating:** Low (2.7) [CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:L/I:N/A:N](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:L/I:N/A:N) 495 496 ## Changes by Kind 497 498 ### Feature 499 500 - Kubernetes is now built with go 1.21.9 501 - update debian-base/set-cap to bookworm-v1.0.2 ([#124198](https://github.com/kubernetes/kubernetes/pull/124198), [@cpanato](https://github.com/cpanato)) [SIG API Machinery, Architecture, Release and Testing] 502 503 ### Bug or Regression 504 505 - Fix pod restart after node reboot when NewVolumeManagerReconstruction feature gate is enabled and SELinuxMountReadWriteOncePod disabled ([#124141](https://github.com/kubernetes/kubernetes/pull/124141), [@bertinatto](https://github.com/bertinatto)) [SIG Node] 506 - Golang.org/x/net is bumped to v0.23.0 to address CVE-2023-45288 ([#124179](https://github.com/kubernetes/kubernetes/pull/124179), [@MadhavJivrajani](https://github.com/MadhavJivrajani)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 507 - Kube-apiserver: fixes a 1.27+ regression in watch stability by serving watch requests without a resourceVersion from the watch cache by default, as in <1.27 (disabling the change in #115096 by default). This mitigates the impact of an etcd watch bug (https://github.com/etcd-io/etcd/pull/17555). If the 1.27 change in #115096 to serve these requests from underlying storage is still desired despite the impact on watch stability, it can be re-enabled with a `WatchFromStorageWithoutResourceVersion` feature gate. ([#124006](https://github.com/kubernetes/kubernetes/pull/124006), [@serathius](https://github.com/serathius)) [SIG API Machinery] 508 - Kubeadm: fix panic in the command "kubeadm certs check-expiration" when "/etc/kubernetes/pki" exists but cannot be read. ([#124124](https://github.com/kubernetes/kubernetes/pull/124124), [@carlory](https://github.com/carlory)) [SIG Cluster Lifecycle] 509 - NONE ([#124326](https://github.com/kubernetes/kubernetes/pull/124326), [@ritazh](https://github.com/ritazh)) [SIG Auth] 510 - OpenAPI V2 will no longer publish aggregated apiserver OpenAPI for group-versions not matching the APIService specified group version ([#123625](https://github.com/kubernetes/kubernetes/pull/123625), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing] 511 512 ## Dependencies 513 514 ### Added 515 _Nothing has changed._ 516 517 ### Changed 518 - golang.org/x/crypto: v0.16.0 → v0.21.0 519 - golang.org/x/net: v0.19.0 → v0.23.0 520 - golang.org/x/sys: v0.15.0 → v0.18.0 521 - golang.org/x/term: v0.15.0 → v0.18.0 522 523 ### Removed 524 _Nothing has changed._ 525 526 527 528 # v1.28.8 529 530 531 ## Downloads for v1.28.8 532 533 534 535 ### Source Code 536 537 filename | sha512 hash 538 -------- | ----------- 539 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes.tar.gz) | 66998a96c8af53a8249708f5b25e77aae10fcf07e863d8c80145759dc7813d944a4bbd1b3a2b3676f33d2daf89d94bfe79475efe77694bb39b804dd8946405e8 540 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-src.tar.gz) | 02177b6c0517146bec036ddbad8575f742bbf8b328a604bc30f5e16398dee22f192b6840370d9e345492f919de98a0978c55ea0890587cb91afe77109a7efe8c 541 542 ### Client Binaries 543 544 filename | sha512 hash 545 -------- | ----------- 546 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-darwin-amd64.tar.gz) | d5cde8002471c38a70417e708c6d355498e770de513dd8cbc5364036498eed01d2a2b69266a91ad132bfaab4fa0082986af7e3d9907d906459093eb30797bd84 547 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-darwin-arm64.tar.gz) | 03895f6d43f07cd83225100ac633830bf96a4a7f8bf1cf4175ab11bb4c98696efefa67ff1ea6333d00fe46bd8dc329dcb5a5355f97a616ad3666d9b21d8d28e7 548 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-386.tar.gz) | 072894bceb8dbc4b443be095d0446d33b5ab235d312a4a571afa81411880bb882a352d4b023e5edf96be7fbb66f367a37056ce0cb5b2c3f54e767cfe45d111bd 549 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-amd64.tar.gz) | 0812400f8285dc6ead6d25b332ebe3355edd1619b092d1af2445c578553b09a435af7627322eb50d07d0f2d82fd29f4441ceb5ba0b4c4bb990db2ebe197d3e0b 550 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-arm.tar.gz) | f08d1a33a84145f2493b0d4d85a85a2331d3753b93836166faa41e9222eca1cfdf0e3da624654d64a810bf16159ac2524d4be789ee85d6255ea1a3fde12b712f 551 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-arm64.tar.gz) | 1ae4bd65c4f3483797c9b7dd7e1e8727b22a10dc09c877a9c2d468f615d1cdc915ec9f372832025fa7153cdb8e302d239738671a32a3ec715e2b7aa6d307dbd1 552 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-ppc64le.tar.gz) | daf0b71a340f7afe81cd0c49d42f9a6e259fcb0b5bcb60f7b0ddfe0137d7a5b9a80569aa38b1e6eb59d094b2bd69d64487d7df8021fd5720ec216ddb615e37d9 553 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-s390x.tar.gz) | 61e81b992aea5ecbfb395b9ff26b66a209530dc3155d3275ff977833d4d8798972571a2b01cf6eee8389b75a9d03aee555c8c211114aba601e1864f86b8006da 554 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-windows-386.tar.gz) | 5c8890d079280deb489905e9bf8b1ef5df9e1267eda9dd0c691bca8f1eb2395629a5bea4d4e2d9d71b89af422a515e226aab50724ee673f298e15aa311926738 555 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-windows-amd64.tar.gz) | 63a47818e2d45ffa0e598bc7c0915e140121fedb7cde578fd00332118401bbf495aaee811da11f0cd9075e58a94fb1eda40936a7c12fbfae99e06a4bd04ed0fd 556 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-windows-arm64.tar.gz) | 287a83b2f19f15ebef9488d65c7509a0423345873671a748f77ea71acdae1d44cb83c7d638677e248907eff14e4fa6612947e7b7f63c1eaabaecf5d607f3f49b 557 558 ### Server Binaries 559 560 filename | sha512 hash 561 -------- | ----------- 562 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-server-linux-amd64.tar.gz) | 3cb91a04cff7cc54d40e3cdeb9bd2b6ba57a54223de1aac82db33afcac47b7ed25a7267984937a7fb99e9eac23080a1879ec392978def7d70d327cff208eea2e 563 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-server-linux-arm64.tar.gz) | 6bd7d6654ab5454cada91454189b250c693264b0fe66dd2bcb7a80d7969785ecfecf90a8b550d1e9e576316c4275bbbd91a445819ec20a710eddc3c611fec3b4 564 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-server-linux-ppc64le.tar.gz) | 12058f01fcfa5c450c95f5f00149d095db7703bdb3736a6ed3fd33e8af3c60b702cfc4ae90cc79bbe0bb60de8470f9969eed4135343d527e15f852c3a986ab69 565 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-server-linux-s390x.tar.gz) | 3b37e695260f4305f916dfe53a4d6235ca4c4a559e0006e2c38e1832132e7b65c231f5e7daa6d7507f6d1be303430bd38f8425e1623a10389724c576054b4f7c 566 567 ### Node Binaries 568 569 filename | sha512 hash 570 -------- | ----------- 571 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-linux-amd64.tar.gz) | 8739961e2657ac011ff9112d85d1958a6fe6f6c8d358ae5e882c526f40d49f95ebf3b90e35473fdd50981f952951714a2d0ed722efbae886586a36e33fb7d79f 572 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-linux-arm64.tar.gz) | 85a5d9971e6ed12e51f274259dc39223599c42a0f81f546f19ac5ccc60caae46e2feb850d6fd5659aed6e2411127109488833d270ff42344dbaa6b30ffecf511 573 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-linux-ppc64le.tar.gz) | 11e284538091e43cd06c5ecf1c038f75242225d06ea95bd0c395bef2f8964a1037ee5e89bf74892d011e2a61d86f7d16aafb691e43b22feaa189fd53c26f2639 574 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-linux-s390x.tar.gz) | e89d208a8349e5d5208473bbf1aa67285039d18ff87b0d32555baec13c29ffee529a7a8fed36363fa9adc9825568c98d35a5fd8bab74803df3c1c1c1846e8fd1 575 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-windows-amd64.tar.gz) | 1dfa21cab0fc5dba81752f541801eab2d35da875b4a4688fd0b0b7d864a328f7fd60fdfb0488cdb314238102719aa9ba8bc7e7c539c466ff43bf556c1714b080 576 577 ### Container Images 578 579 All container images are available as manifest lists and support the described 580 architectures. It is also possible to pull a specific architecture directly by 581 adding the "-$ARCH" suffix to the container image name. 582 583 name | architectures 584 ---- | ------------- 585 [registry.k8s.io/conformance:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x) 586 [registry.k8s.io/kube-apiserver:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x) 587 [registry.k8s.io/kube-controller-manager:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x) 588 [registry.k8s.io/kube-proxy:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x) 589 [registry.k8s.io/kube-scheduler:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x) 590 [registry.k8s.io/kubectl:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x) 591 592 ## Changelog since v1.28.7 593 594 ## Changes by Kind 595 596 ### Feature 597 598 - Kubernetes is now built with go 1.21.8 599 - update distroless-iptables to v0.4.6 ([#123772](https://github.com/kubernetes/kubernetes/pull/123772), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 600 601 ### Bug or Regression 602 603 - Fix error when trying to expand a volume that does not require node expansion ([#123055](https://github.com/kubernetes/kubernetes/pull/123055), [@gnufied](https://github.com/gnufied)) [SIG Node and Storage] 604 - Fixed a bug that an init container with containerRestartPolicy with `Always` cannot update its state from terminated to non-terminated for the pod with restartPolicy with `Never` or `OnFailure`. ([#123710](https://github.com/kubernetes/kubernetes/pull/123710), [@gjkim42](https://github.com/gjkim42)) [SIG Apps] 605 - Fixed cleanup of Pod volume mounts when a file was used as a subpath. ([#123052](https://github.com/kubernetes/kubernetes/pull/123052), [@jsafrane](https://github.com/jsafrane)) [SIG Node] 606 - Fixed the disruption controller's PDB status synchronization to maintain all PDB conditions during an update. ([#122056](https://github.com/kubernetes/kubernetes/pull/122056), [@dhenkel92](https://github.com/dhenkel92)) [SIG Apps] 607 - Fixes an issue calculating total CPU usage reported for Windows nodes ([#122999](https://github.com/kubernetes/kubernetes/pull/122999), [@marosset](https://github.com/marosset)) [SIG Node and Windows] 608 - Prevent watch cache starvation by moving its watch to separate RPC and add a SeparateCacheWatchRPC feature flag to disable this behavior ([#123694](https://github.com/kubernetes/kubernetes/pull/123694), [@mengqiy](https://github.com/mengqiy)) [SIG API Machinery] 609 - Restore --verify-only function in code generation wrappers. ([#123261](https://github.com/kubernetes/kubernetes/pull/123261), [@skitt](https://github.com/skitt)) [SIG API Machinery] 610 - Updates google.golang.org/protobuf to v1.33.0 to resolve CVE-2024-24786 ([#123764](https://github.com/kubernetes/kubernetes/pull/123764), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 611 612 ### Other (Cleanup or Flake) 613 614 - Build etcd image v3.5.12 ([#123069](https://github.com/kubernetes/kubernetes/pull/123069), [@bzsuni](https://github.com/bzsuni)) [SIG API Machinery and Etcd] 615 616 ## Dependencies 617 618 ### Added 619 _Nothing has changed._ 620 621 ### Changed 622 - github.com/golang/protobuf: [v1.5.3 → v1.5.4](https://github.com/golang/protobuf/compare/v1.5.3...v1.5.4) 623 - google.golang.org/protobuf: v1.31.0 → v1.33.0 624 625 ### Removed 626 _Nothing has changed._ 627 628 629 630 # v1.28.7 631 632 633 ## Downloads for v1.28.7 634 635 636 637 ### Source Code 638 639 filename | sha512 hash 640 -------- | ----------- 641 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes.tar.gz) | 084b28ba78ad4cc6da8422535222ee2232a6fef0f5f7eb5cd0aa1fd6b53433c4a4b2efc0002020c6cfae91ff8c194e0e0a55c876f3e563d33ff8ae4dcfaccb30 642 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-src.tar.gz) | d20000cd269a569013c5cac3761e03f5c2f99094f57b43e0f531a752b355fc7348efbf93ce75185659a2b2a5be65fed5cfa9948cdaab9d359f571899eab74530 643 644 ### Client Binaries 645 646 filename | sha512 hash 647 -------- | ----------- 648 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-darwin-amd64.tar.gz) | be563c6e071585fcfb0d627ee8e2c1cc2ab0375e0e6b3ca5fab758dc085825a399169e1e309d008c853a493e3a1b899ff7bbfd491db082f7becf98d80a31fe87 649 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-darwin-arm64.tar.gz) | 49d1ef441b7de249ca988608f865d9d9d3e565c437888371ac2901cee6e98f607602cf8183f54bbf9ac77bd0d50bebca151e172b4c764ae0f6e0523bdddc0111 650 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-386.tar.gz) | 693f2fc3288f9697c831b9d52c507277c3ef38afe8136492755c1928e489c49cd7fa78c482c91421199769b979d19c014faf36a7c558ecc6eed245fbd03c63cb 651 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-amd64.tar.gz) | 8dc1da1b4f447c0d3d196da83f9560d685f311c2f3a10ca2560af8f328dd1b390547b9ca49a3754e92be4b77b086e3db2a888d8f69afb696d7e6f0b5d5958bf9 652 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-arm.tar.gz) | a00d191feb238cbd845c8f0c641891d043b182e45c75466af8111bbae51e8e81233545fb431149d23b0525976037474421125031b9b05c9730c983f0ce804937 653 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-arm64.tar.gz) | 340105ecd45b0c7904552849310e6e681f3e209583d4e54298ba7a2f2f8ec205acd5b17e119e0cf4c58bb993c1527610a4ff30c7b584c2f0239a93cdb6ba3357 654 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-ppc64le.tar.gz) | eaa9ff433b487b7f42d38e322b8f901cca30f4d22ead2cdb9c48aa13c44c48ba4059c1913ef72be9b812a7792d58c0b6197be87d64a29b6bc0135cce8f43578a 655 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-s390x.tar.gz) | a0d7471236cba0ac9644916e09b92d42f2c884ea277e2b368061b68e47d6c357fc0cc79c7aee79a19862168072b49bd9a238c3db59bc1f8afd4d0cf76e7ea725 656 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-windows-386.tar.gz) | 24ff9a4ce3715d293a2895e178fe9acc2ac393ec089e75eec343f541bf48b7d03c7e43d0ca09fbe8694914ed2f05f8d86e5d113bff1433f9744a06b931a7cef4 657 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-windows-amd64.tar.gz) | da69beeb042b50279e3185d9b92f9738a8082c40f073e5b34b585ae6f11268d69277104ff17f572cfb5a0065ec86bb18e2882eab63fd0c9d9aee18c45402a5e6 658 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-windows-arm64.tar.gz) | 1e3b2e84638e75c70984f1366d24cc68f9eee0a9ee37008f22686173ca886ce538562252537753f929a4d242a91134efeb90d75164074a202b90845394337cd0 659 660 ### Server Binaries 661 662 filename | sha512 hash 663 -------- | ----------- 664 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-server-linux-amd64.tar.gz) | ea566c66916cb6b892dec2151850bae766a5593e3a4c1cc39b1ff945fffd843611ea980a01cc9dea4bd69bc1a1fefc6e31284937e8d88c19f7d8e183917bdf8f 665 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-server-linux-arm64.tar.gz) | 157b2ccde60b2a197e707b504affea22717f32fa01b7727bf7bd8e2709c728eb09e9dc918b11beb01908298f1eba2a6bbf8939e12dc23987031bedc6d6b8ed96 666 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-server-linux-ppc64le.tar.gz) | 8921192dfb3e8fc375cf0e0dd572799f4a0fb59ffe3722092779fadf04d3a00aaba5d478965bf613a0dc6be156474e615cd480f695931c7300e24851f993e833 667 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-server-linux-s390x.tar.gz) | f5b7b47bf826157e847cac806104f36848483682926ad3c168cbd273c3a573e582fa686b36c9016d4e593957672566c20d2f8f24df30f1f2863ebbb2d60c96b0 668 669 ### Node Binaries 670 671 filename | sha512 hash 672 -------- | ----------- 673 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-linux-amd64.tar.gz) | e1cfaf7e2a8f264fdbf6424d50732c8806f2a3a64cf3ba34f06a48fbfcfdcb6c255d9873285361fabef4d8290eee373e5933ec67cf1145b00042ee547e65ba8f 674 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-linux-arm64.tar.gz) | 984f5d37ef39c21358d3e96c27e8bf5e451b41cb268cb81c65fb3a54e0ff55284e8e9ba668f99a70daa4ab1cdaa54f67f6c421ff283948906551529c372f67b5 675 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-linux-ppc64le.tar.gz) | d03ae5a3952246dbba92093e8a38d0ccf46fa345717e248427d49bca04ea66b1985a2bbdc442aad87e81d8f619cfe060191cf7d7ec97f91a8590d98dda538f2c 676 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-linux-s390x.tar.gz) | 94420b4279f825bc6e412d51c6076c07fa44a06c55f6bea8a5a028a762b581d87dc94ea0d20369f66410d6340d4623f9c1cac0b68d27bbc921b80f28de39a0e2 677 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-windows-amd64.tar.gz) | 7ae84dffdacb29d67f9a5d655e96e68e0adb5d8bcb329e5dea6c9a603a3cc5b72825077eb535ba6a9a34691c79524ef2ee7745f0b15be97f7e97d8ddd0cfd14e 678 679 ### Container Images 680 681 All container images are available as manifest lists and support the described 682 architectures. It is also possible to pull a specific architecture directly by 683 adding the "-$ARCH" suffix to the container image name. 684 685 name | architectures 686 ---- | ------------- 687 [registry.k8s.io/conformance:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 688 [registry.k8s.io/kube-apiserver:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 689 [registry.k8s.io/kube-controller-manager:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 690 [registry.k8s.io/kube-proxy:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 691 [registry.k8s.io/kube-scheduler:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 692 [registry.k8s.io/kubectl:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 693 694 ## Changelog since v1.28.6 695 696 ## Changes by Kind 697 698 ### Feature 699 700 - Add process_start_time_seconds to /metrics/slis endpoint of all components ([#122750](https://github.com/kubernetes/kubernetes/pull/122750), [@richabanker](https://github.com/richabanker)) [SIG Architecture, Instrumentation and Testing] 701 - Kubernetes is now built with go 1.21.7 702 - update setcap/debian-base to bookworm-v1.0.1 703 - update distroless-iptables to v0.4.5 ([#123228](https://github.com/kubernetes/kubernetes/pull/123228), [@cpanato](https://github.com/cpanato)) [SIG API Machinery, Architecture, Release and Testing] 704 705 ### Bug or Regression 706 707 - Fixes a race condition in the iptables mode of kube-proxy in 1.27 and later 708 that could result in some updates getting lost (e.g., when a service gets a 709 new endpoint, the rules for the new endpoint might not be added until 710 much later). ([#122757](https://github.com/kubernetes/kubernetes/pull/122757), [@hakman](https://github.com/hakman)) [SIG Network] 711 - If a pvc has an empty storageClassName, persistentvolume controller won't try to assign a default StorageClass ([#122704](https://github.com/kubernetes/kubernetes/pull/122704), [@carlory](https://github.com/carlory)) [SIG Apps and Storage] 712 - Kubeadm: do not upload kubelet patch configuration into `kube-system/kubelet-config` ConfigMap ([#123107](https://github.com/kubernetes/kubernetes/pull/123107), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 713 - Kubeadm: fix a bug where the --rootfs global flag does not work with "kubeadm upgrade node" for control plane nodes. ([#123097](https://github.com/kubernetes/kubernetes/pull/123097), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 714 715 ## Dependencies 716 717 ### Added 718 _Nothing has changed._ 719 720 ### Changed 721 _Nothing has changed._ 722 723 ### Removed 724 _Nothing has changed._ 725 726 727 728 # v1.28.6 729 730 731 ## Downloads for v1.28.6 732 733 734 735 ### Source Code 736 737 filename | sha512 hash 738 -------- | ----------- 739 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes.tar.gz) | 63f60784a96277b7e06524123d483534d65bffa5416cabbe6b48659bbd4f901ffc8cbf9708c0edf2bd37eb09ea2cc5e77813de7a1f274bc5a7f467e949ce0fd8 740 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-src.tar.gz) | 33410e98750825ce3472a0b4add5e5b2ec42c3735c405d045fffe8b71e0345ef3a335cda142a896b49190e0f667d33db909f851dae00059d406d1b067e645bfe 741 742 ### Client Binaries 743 744 filename | sha512 hash 745 -------- | ----------- 746 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-darwin-amd64.tar.gz) | c02f0d2c96bd7d62f699f21437b2b49966ce1fd67442cc847b3e95a3233144983627e3d69cc20d0250b08f197481e5d9a4f49988174957a9ebc7bc3178cd64e8 747 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-darwin-arm64.tar.gz) | fc4aa1681b2897e69a0b2924a96ac82b46443b98b5cc26f5536b648509735dc21f09d29b68506153c05c915ad519979d6e751023b46fa4e2baf1b90dbc1bddc7 748 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-386.tar.gz) | d46bdc7005c941baaecc5f770b348ab1d80701b4255b34cd4cf887bc722bdb4826524d95d3c7da6052e7506e9f8ffc40c2674436177f7771227cdfbcc1c1827f 749 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-amd64.tar.gz) | b81a92ff7ead2de98f86b3da3976917cce45e576c6368c564f1dbe20a2e0cb99fa51da3aa5cc16805d57abe856f82cf3318c1425d6d05557d18b856e8550a06d 750 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-arm.tar.gz) | 9ce9c439328a93cf4c69236ca6fea2834cd31bcb3752e535be15ec63cc79f4bb1645a79bc9cb900b8d97ff65920cda9b95de95b81af3b70e3d436b17827d4af9 751 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-arm64.tar.gz) | 8618c45c20ff4b0c839731e645c9f081363f25ef3b8e1436aa7643db8cbf03749a20d22ca0cd3e98cf679b74eee7cb2d3ba9b80d0d817dcb41dac6322fdd670e 752 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-ppc64le.tar.gz) | 38ef6b5c0287f267059f71ede0c22f30fa2291e5d1a18ad63cb540de0df51a87b66f3aa04567208e4ce4b298f99d2de72e6010101b3075e8896b5c5189e77bfc 753 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-s390x.tar.gz) | 94b46b7bbfadeaba05ec086cc7031be41d697b8377ddab419177ab39518ed59b7f2f8e6ee12256bd2582030788ac126009d82f409f5db5d8c51bbf6e3a8f4f3e 754 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-windows-386.tar.gz) | 57b55ba7dd596dedfb09343ce5f8ea14783511bf30e7b462c7df06f27abc4f1cff2d479c56d7de1d3022567edcfb7c923465f9e651759ef429f15acb595b9530 755 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-windows-amd64.tar.gz) | 84d90c155b9ffc1748f225a9c47a04a48cdbb8106c9fcaa33d0893bdd5590da089a81943754012bb932d4ac5eb645696eb589d60e8747f561ecf319bcadb220a 756 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-windows-arm64.tar.gz) | c5d6d1081bbb0410faba25fca0d21c8a9417df958b953c23ec41c31266f43ef92aa34d8b838e351442d956089ea387abf6b09a4279a2a91e37c592f45427434e 757 758 ### Server Binaries 759 760 filename | sha512 hash 761 -------- | ----------- 762 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-server-linux-amd64.tar.gz) | f01b166c4581ff933972be0c82ee168dee26724598141a5855bf26a663c52771e3476a1999e65680940b5436871e887432f93525d83924b0f39a706e8e14a5b7 763 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-server-linux-arm64.tar.gz) | ffbb00d02764f301bdf4e5e840443a1146c729c379462194a24e9596cb66f87469373cc6df9293f7a36cc246b90a1dd9e637c8ed6fb657ddda9d69c40e70c809 764 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-server-linux-ppc64le.tar.gz) | 1d6aa422e06174018ae86af755e80fa2b1b4f3c114f051183bdd04479fc47a17769e34d6a09525567e1afa577069895716637f805b5af15ac32bd80ee79b48fe 765 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-server-linux-s390x.tar.gz) | 256104b0d9cf882fd8702f7b0a2c20533bd4ee7035db03caa8eeb2b54c0aa70742eb7c934b6d9e8def7108fe991e10a5d0447e02f25ef65f2feadfa13e5d3de6 766 767 ### Node Binaries 768 769 filename | sha512 hash 770 -------- | ----------- 771 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-linux-amd64.tar.gz) | f71b2b1193fc1316f66203f1c8d862ebfff59783c90a582d06553273b25ae70761eb971e35bbc45164a38cd5177196644c06174d2d6a3db108c292bd47d165ab 772 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-linux-arm64.tar.gz) | 86ea03b352369f5c0d24002bec322640ec69364031eea827667a4436c2357d2e35ca92cd3489d59fe045c9492bae76336d086e575db2d96f5db668e5010a608b 773 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-linux-ppc64le.tar.gz) | 0ce43b5f3e1df59e3f4e5673a16e23314e2c4f3d30639b965120cafebe709b812401c4db3397e9785bc46c3aac0ad92d9b315cce084cf42814437f8ad9767525 774 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-linux-s390x.tar.gz) | dac0e90c901513f9006cc5ecc363c8241e097e32ee17ef792bcd698b3c64394ecf32b861484fa7365d5bb4c5cb6fa61e7526a890e392302be53d2f49758e642e 775 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-windows-amd64.tar.gz) | ed00414a085ae0c986040c97d3c1d2631e353795e420d13feba1ced16b9977a17caaa24f9e563adfbd9128a60f5cebc206ae304ca4dfd928219fd9cc71b20219 776 777 ### Container Images 778 779 All container images are available as manifest lists and support the described 780 architectures. It is also possible to pull a specific architecture directly by 781 adding the "-$ARCH" suffix to the container image name. 782 783 name | architectures 784 ---- | ------------- 785 [registry.k8s.io/conformance:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 786 [registry.k8s.io/kube-apiserver:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 787 [registry.k8s.io/kube-controller-manager:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 788 [registry.k8s.io/kube-proxy:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 789 [registry.k8s.io/kube-scheduler:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 790 [registry.k8s.io/kubectl:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 791 792 ## Changelog since v1.28.5 793 794 ## Changes by Kind 795 796 ### Feature 797 798 - Kubernetes is now built with Go 1.20.13 ([#122712](https://github.com/kubernetes/kubernetes/pull/122712), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 799 800 ### Bug or Regression 801 802 - Allow deletion of pods that use raw block volumes on node reboot ([#122211](https://github.com/kubernetes/kubernetes/pull/122211), [@gnufied](https://github.com/gnufied)) [SIG Node and Storage] 803 - Etcd: Update to version 3.5.10 ([#121805](https://github.com/kubernetes/kubernetes/pull/121805), [@mzaian](https://github.com/mzaian)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing] 804 - Fix: Mount point may become local without calling NodePublishVolume after node rebooting. ([#119923](https://github.com/kubernetes/kubernetes/pull/119923), [@cvvz](https://github.com/cvvz)) [SIG Node and Storage] 805 - Fixed a regression since 1.24 in the scheduling framework when overriding MultiPoint plugins (e.g. default plugins). 806 The incorrect loop logic might lead to a plugin being loaded multiple times, consequently preventing any Pod from being scheduled, which is unexpected. ([#122368](https://github.com/kubernetes/kubernetes/pull/122368), [@caohe](https://github.com/caohe)) [SIG Scheduling] 807 808 ### Other (Cleanup or Flake) 809 810 - Reverts the EventedPLEG feature (beta, but disabled by default) back to alpha for a known issue ([#122719](https://github.com/kubernetes/kubernetes/pull/122719), [@pacoxu](https://github.com/pacoxu)) [SIG Node] 811 812 ## Dependencies 813 814 ### Added 815 _Nothing has changed._ 816 817 ### Changed 818 - golang.org/x/crypto: v0.14.0 → v0.16.0 819 - golang.org/x/mod: v0.10.0 → v0.14.0 820 - golang.org/x/net: v0.17.0 → v0.19.0 821 - golang.org/x/sync: v0.2.0 → v0.5.0 822 - golang.org/x/sys: v0.13.0 → v0.15.0 823 - golang.org/x/term: v0.13.0 → v0.15.0 824 - golang.org/x/text: v0.13.0 → v0.14.0 825 - golang.org/x/tools: v0.8.0 → v0.16.1 826 827 ### Removed 828 _Nothing has changed._ 829 830 831 832 # v1.28.5 833 834 835 ## Downloads for v1.28.5 836 837 838 839 ### Source Code 840 841 filename | sha512 hash 842 -------- | ----------- 843 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes.tar.gz) | 62f0f175e544a31e685d1371c5a06af158be6a539aa0f33c2adb0765c995592e8182a29da3f92a326b98d2ec09b3202d63a73289bbfcb17cbb1e6e6961329c34 844 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-src.tar.gz) | 18f493ac18499c4e0e50b72bc3f01670e849d02f8b4bc70b9deefbbabb6119314105196d51d4fdcd9de2c66054ffacbcceb41ad19581e3d2a2d893ef2d97201d 845 846 ### Client Binaries 847 848 filename | sha512 hash 849 -------- | ----------- 850 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-darwin-amd64.tar.gz) | 4167663b3bf355a710cf84abd72df0c40a99c230c90711cf6d374d606022eea5d93137b1139530696a4678c3714bb61cfe5ac2fc599c7b4a4ad79486e7427bf9 851 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-darwin-arm64.tar.gz) | f41cb16a674312206d6a5ae0db9670f5873e1ab946b9bae65afc5899a6b4624af12b1c4a6eb08330cc7147572b6a6d097d864244a308fb2a55a5099c013fc9d5 852 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-386.tar.gz) | 87f205d969374cf790a6bbef32d9a76edf83babdb9a33e50009e25be4281968d89340d24144bfa1ab73bb589fcf86fef604e567d6405c30deb2fe84a0246ef27 853 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-amd64.tar.gz) | 9cd61a97b37cb27cc565f5a2cebd6086b86148c5759eb0e6a0c03e7be4b701bec407c46a65633c51a00a7aa74733c2fdd082b9da3382d38525e2e5b8dbb11b77 854 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-arm.tar.gz) | cc79b49bf4478032bcd086167959141648e7805173636f8779a0a0242db2a5cbdb96131ec2eaa823db89c022a0e4b88cfd0a292a99ceebca3bbbe56c20410c57 855 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-arm64.tar.gz) | 84f4074804e49764b7f911d7f23c6814df18f47cddc4d56d0b951cec12a82d27602744a73888314473639051bc4f1318cbd53ecd79be009f8d01d9ab4f71748e 856 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-ppc64le.tar.gz) | 31f0dea0b968e9b0df3199943a7684795db1bf9168f2c65eaf6b2c77932eb2f87c7ccc1e2988c087c13d908671fcec6def9d28b7c08e23b7943c96785abadd74 857 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-s390x.tar.gz) | 71216d98b1f2f663d47814af5893456a8e39639af2d571eca23bd70e670b9809c81d6845a681d9fb4609943a51b9058126b3ed9f0c9c6ce363cdb9835ca21041 858 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-windows-386.tar.gz) | d5c04c16229332ceeb77cc61bd8e8508f886f713c8ad97674ca16039033e8695a048216835bc633456c561748b3c571d4a0c6e6eb34ff542935c78536a9e268e 859 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-windows-amd64.tar.gz) | c7d453bc83d88c0ae2db9561e14172a4de1fc572ffc09699528a363b835e61a2426e7fa403afe549c6f437cd64cd789354cf64232b66bcfe507019b0106ada29 860 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-windows-arm64.tar.gz) | 09045b94d4efc0d184eb3dcab2a162908838109d7c2661202c6aefb5490275e99e79b994400a495c8264ff65b6b0423bf6e0a84eee579aac872abcda7d60a0aa 861 862 ### Server Binaries 863 864 filename | sha512 hash 865 -------- | ----------- 866 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-server-linux-amd64.tar.gz) | 22c6e2f42fcd5f7b9509803388c75013e6cafad815673ada33ed112bcd2798b5973ac518975479a1b0a4c7d0ea85b647273e9d8bf3296373839259592c7c7bc1 867 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-server-linux-arm64.tar.gz) | 162c982de71d05d09e2396de6121fab39f4cca72b75e69a6d43b1cbc14d852711c6ac0ed3458c96f1e34fe7af222038355b9389cc814877d60b9dce113649d24 868 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-server-linux-ppc64le.tar.gz) | 1bebd732647659ad57b25d2cff977e3c6a9f2cc38752eafd6c007ce174b3eafeb0a2a9ed6b1838df794a43bd782bc0ea65245df2058fab1b11e43204aaaa1cdb 869 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-server-linux-s390x.tar.gz) | 5c2254c6ff0c779cc42234c66d0ccfb12a6c5e81bc1c82feb476cf375fa54f0304a34372d9af83b778495ab94dc560886b8d3f44d1fb6768727ea33960c61ff4 870 871 ### Node Binaries 872 873 filename | sha512 hash 874 -------- | ----------- 875 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-linux-amd64.tar.gz) | 07aeddf94fb7172646a83a6304319568c5ab897f172cacf7fcdd0f5d7978fa85d9a32848d7b342dd27ddb1ffd9a38eb3d0cbb020fb93c8651cd28b30c5d631d0 876 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-linux-arm64.tar.gz) | ca26eb4f25ecd2ee0d4cd9d7a3813baf350fe3e5ab38ec02b1d06db8481cbf0b04f016fc15af8af7ddf789e6f42daa9ab09531d983020bf04b2f2fdd338cf15f 877 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-linux-ppc64le.tar.gz) | f850f0449e50ea3d8655edd9cdbb6447dce15d01919b023edf3ce910b03bf09f04327cb22707d931ac4391a6910a83b461f4b5b57f0e80ecf4c130861298df98 878 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-linux-s390x.tar.gz) | dff3115be5882177dcc774f3f277c64cf0658ad35d23a2f2323e2a952df69fb644b57cf26b2b5ccf84a1cd56ad80762494be236522c266fdce2790aa48671a70 879 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-windows-amd64.tar.gz) | c78383266ed43429600acc8a46bcc6c77437af7c758b9c530716d34d33f2bd9667a13491324b8b3f733e2ea63fc6047ab639f8126956291a2fc8124a37b024cd 880 881 ### Container Images 882 883 All container images are available as manifest lists and support the described 884 architectures. It is also possible to pull a specific architecture directly by 885 adding the "-$ARCH" suffix to the container image name. 886 887 name | architectures 888 ---- | ------------- 889 [registry.k8s.io/conformance:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 890 [registry.k8s.io/kube-apiserver:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 891 [registry.k8s.io/kube-controller-manager:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 892 [registry.k8s.io/kube-proxy:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 893 [registry.k8s.io/kube-scheduler:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 894 [registry.k8s.io/kubectl:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 895 896 ## Changelog since v1.28.4 897 898 ## Changes by Kind 899 900 ### Feature 901 902 - Kubernetes is now built with Go 1.20.12 ([#122216](https://github.com/kubernetes/kubernetes/pull/122216), [@xmudrii](https://github.com/xmudrii)) [SIG Release and Testing] 903 904 ### Bug or Regression 905 906 - Fix panic if there are more terminating pods than active pods ([#122267](https://github.com/kubernetes/kubernetes/pull/122267), [@kannon92](https://github.com/kannon92)) [SIG Apps] 907 - Fix: statle smb mount issue when smb file share is deleted and then unmount ([#121851](https://github.com/kubernetes/kubernetes/pull/121851), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage] 908 - Fixed a regression since 1.27.0 in scheduler framework when running score plugins. 909 The `skippedScorePlugins` number might be greater than `enabledScorePlugins`, 910 so when initializing a slice the cap(len(skippedScorePlugins) - len(enabledScorePlugins)) is negative, 911 which is not allowed. ([#121667](https://github.com/kubernetes/kubernetes/pull/121667), [@kerthcet](https://github.com/kerthcet)) [SIG Scheduling] 912 - Fixes a kube-apiserver log volume regression bug in default 1.27 configurations (introduced in 1.26, activated by the AggregatedDiscoveryEndpoint feature enablement in 1.27) ([#122096](https://github.com/kubernetes/kubernetes/pull/122096), [@ritazh](https://github.com/ritazh)) [SIG API Machinery] 913 - Fixes a regression in kube-scheduler memory use in default 1.28 configurations by moving the SchedulerQueueingHints feature gate back to disabled by default. ([#122291](https://github.com/kubernetes/kubernetes/pull/122291), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 914 - Fixes an issue where StatefulSet might not restart a pod after eviction or node failure. ([#121389](https://github.com/kubernetes/kubernetes/pull/121389), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) [SIG Apps and Testing] 915 - The scheduling queue didn't notice any extenders' failures, it could miss some cluster events, 916 and it could end up Pods rejected by Extenders stuck in unschedulable pod pool in 5min in the worst-case scenario. 917 Now, the scheduling queue notices extenders' failures and requeue Pods rejected by Extenders appropriately. ([#122045](https://github.com/kubernetes/kubernetes/pull/122045), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 918 919 ### Other (Cleanup or Flake) 920 921 - Bump distroless-iptables to 0.2.8 based on Go 1.20.11 ([#121976](https://github.com/kubernetes/kubernetes/pull/121976), [@cpanato](https://github.com/cpanato)) [SIG Testing] 922 - Makefile and scripts now respect GOTOOLCHAIN and otherwise ensure ./.go-version is used ([#122075](https://github.com/kubernetes/kubernetes/pull/122075), [@BenTheElder](https://github.com/BenTheElder)) [SIG Release and Testing] 923 924 ## Dependencies 925 926 ### Added 927 _Nothing has changed._ 928 929 ### Changed 930 - github.com/cyphar/filepath-securejoin: [v0.2.3 → v0.2.4](https://github.com/cyphar/filepath-securejoin/compare/v0.2.3...v0.2.4) 931 932 ### Removed 933 _Nothing has changed._ 934 935 936 937 # v1.28.4 938 939 940 ## Downloads for v1.28.4 941 942 943 944 ### Source Code 945 946 filename | sha512 hash 947 -------- | ----------- 948 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes.tar.gz) | 9b1aa58395d4fe0efd75382dc4fac3c3203570f80f71e5a4e354983b597d4af442475bbcc65a7fccb2c2faa874954b69d09122c84e5ee78caa2ddc4cd8b82b26 949 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-src.tar.gz) | 70b929f5ab11b0bb14d0e7870a2f4e3b4b1c5d52016cec4560198a339fe3363c6e9de0ebc50b643ba12d569bd527737141c36d8abccb559596e772624a8219cb 950 951 ### Client Binaries 952 953 filename | sha512 hash 954 -------- | ----------- 955 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-darwin-amd64.tar.gz) | ed701ec7c67260bfb5cb910ef890732d171e72f3abff552d321a014ae59f1f45fa1339949c0711a9e82626eed5d55916489c8e339381e120c4a9b63970b8b3c0 956 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-darwin-arm64.tar.gz) | 43b6c1daa9b5597281536a4875f6f63258111f1a6fa1a38c41f95d07309991dba4f2eccf2321db5ea126d0f51ace2ed390ed10919ce7280f134078aebfc54d1a 957 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-386.tar.gz) | fe0299567d47edf8625ea84b12233e07559cacae0fc3797afb249c86171b803125d2c34c15bdbc4a25a05646c8480bf9943b8dd9a2583cbedf70bccef309113f 958 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-amd64.tar.gz) | 8da3b845e44ecbf94f2766d18dcccd2cd55f5645038c5ca50ae2163989cec5a330fc6c4b55780e4986f2b619edacd21c0236a5637b9623cb452b97b590a2f483 959 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-arm.tar.gz) | 0e51666c37cc3e151ec67457a9d51820d6b7a2fe52d6bcb6c187b8a7461d64e6279c762ce34a272baa05c0feae47859df390e2aceb6f8396ba471a103456b9b5 960 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-arm64.tar.gz) | 57293627a6f4b2bd45db5101f81a8b8d6e4986f5a11d24f4f782c87b5867b8b8015654b8355dd51292cb663b1719d34282259a514d566993fd9c16db95768a7b 961 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-ppc64le.tar.gz) | 789fc636d69e58c41fd2fd83f07ed66bc96ea2e929114d5b7fcf2522a3a5c61b49c56986ac866d093224f5ef6878e9dfb6ffbe8674df5228b869864ae140680b 962 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-s390x.tar.gz) | 99f3cd73dbfdb0bcade8552c5adc050b67a416edf68f35bde0e62fd93fcc88d160fa416d23ee72f0ec2d7a375f990d218c4fe8b69ebc0923bf68c4cbd992ab10 963 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-windows-386.tar.gz) | 9a60ead2cc4333138936a561a6d07671f8b17b865733d98f8d64cd91f0551897adfe1a1ca660dc2bf474375128ecdc4654983eb63285be50656b59a54ba7bf77 964 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-windows-amd64.tar.gz) | ba174234eba338f563a4ba6fb66d12db62df272d5fdf2c68fd12d7660c88036f5c35881705f07e4b0bad58dad2d49465643c40547d14a718f5a2098468e233d0 965 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-windows-arm64.tar.gz) | 7e830b5aadef11109ff3bfb24d57455ab9292f51e37f8c60d84c09ccd9adf4db2cd338c7402d8c03242db5d96877275dddf56f45531392ac7f7071d85bb1ec7c 966 967 ### Server Binaries 968 969 filename | sha512 hash 970 -------- | ----------- 971 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-server-linux-amd64.tar.gz) | c7d7016860ff44c15322040a5764d4acbde32ffebeb84802b9be820b4be22d9e0a1f8c2ee4547dccebaf133acc22f624d46be0de567ac5f98eb97303bbd5d7a9 972 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-server-linux-arm64.tar.gz) | 4f1d2299f0d0ca52ff4a793d41b8bcb4c50fc9fe6584559a92f88e91966fc6d47b78cff407076e9169d6a592cddbadebeb2348c7c96192eb5fba0f71818a3752 973 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-server-linux-ppc64le.tar.gz) | 1f86ee121c0f91f7cd0e0006fea01248b2a3afd49c748da99a85774a5c8dc0b98a2a4b3668186cb59fd77321287b8a745e6feb2d29dfc0a178795361b5b8a4ff 974 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-server-linux-s390x.tar.gz) | 98be6cfd43643ac86771ce36856925e7827cb57bec9482a33c23b6ee04c6a208e9f359027f762a825c8b62163dbf1786ec64ff51dd47b495d6f09cce6423d0e1 975 976 ### Node Binaries 977 978 filename | sha512 hash 979 -------- | ----------- 980 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-linux-amd64.tar.gz) | 845afdaa584d917cd29d46963bc337560eecb59c97b9ce0f664e3bd9ebe9c803fb239fcd6af69b852c2f553480d09b2b0a34db109d9fb8ea315a0df0fbbce0d0 981 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-linux-arm64.tar.gz) | cf17d5cce0519fff510a4599e014f1972e3e34d7f2f453c75d556296cdffc550271c76c1742ced1b5890ffd934c49c6aee67b769c0f758c914a7e40a402ccd3a 982 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-linux-ppc64le.tar.gz) | 8338669995d7179807a3cd82f6788830907440d19d91f0d5d8572f8a84a2fda101f878548fa2332736ee7e09c8d70af7402a195b54019a15ff95850751420edc 983 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-linux-s390x.tar.gz) | 65e0742ca66a62ab971fc11d5e5c19ea776fedea9f3648b5dac7689005f951ec63dc0455ef2d004aeada094ba1922d0c827e6af678f2ddb68134f1a936daf258 984 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-windows-amd64.tar.gz) | a147c116fc4d6c49e624a2ea0a59bdebbe61e9fa7f28743cd543459f61b6f22f7c77d0f4926a46e3d116fc30c39b2df89395d4da44f5c6a443bda35392139f51 985 986 ### Container Images 987 988 All container images are available as manifest lists and support the described 989 architectures. It is also possible to pull a specific architecture directly by 990 adding the "-$ARCH" suffix to the container image name. 991 992 name | architectures 993 ---- | ------------- 994 [registry.k8s.io/conformance:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 995 [registry.k8s.io/kube-apiserver:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 996 [registry.k8s.io/kube-controller-manager:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 997 [registry.k8s.io/kube-proxy:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 998 [registry.k8s.io/kube-scheduler:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 999 [registry.k8s.io/kubectl:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 1000 1001 ## Changelog since v1.28.3 1002 1003 ## Important Security Information 1004 1005 This release contains changes that address the following vulnerabilities: 1006 1007 ### CVE-2023-5528: Insufficient input sanitization in in-tree storage plugin leads to privilege escalation on Windows nodes 1008 1009 A security issue was discovered in Kubernetes where a user that can create pods and persistent volumes on Windows nodes may be able to escalate to admin privileges on those nodes. Kubernetes clusters are only affected if they are using an in-tree storage plugin for Windows nodes. 1010 1011 **Affected Versions**: 1012 - kubelet >= v1.8.0 1013 1014 **Fixed Versions**: 1015 - kubelet v1.28.4 1016 - kubelet v1.27.8 1017 - kubelet v1.26.11 1018 - kubelet v1.25.16 1019 1020 This vulnerability was reported by Tomer Peled @tomerpeled92" 1021 1022 1023 **CVSS Rating:** High (7.2) [CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H) 1024 1025 ## Changes by Kind 1026 1027 ### API Change 1028 1029 - Retry NodeStageVolume calls if CSI node driver is not running ([#120330](https://github.com/kubernetes/kubernetes/pull/120330), [@rohitssingh](https://github.com/rohitssingh)) [SIG Apps, Storage and Testing] 1030 1031 ### Feature 1032 1033 - Kubernetes is now built with Go 1.20.11 ([#121812](https://github.com/kubernetes/kubernetes/pull/121812), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 1034 1035 ### Bug or Regression 1036 1037 - Fix 121094 by re-introducing the readiness predicate for externalTrafficPolicy: Local services. ([#121116](https://github.com/kubernetes/kubernetes/pull/121116), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network] 1038 - Fixed a regression in default configurations, which enabled PodDisruptionConditions by default, 1039 that prevented the control plane's pod garbage collector from deleting pods that contained duplicated field keys (env. variables with repeated keys or container ports). ([#121379](https://github.com/kubernetes/kubernetes/pull/121379), [@mimowo](https://github.com/mimowo)) [SIG Apps, Auth, Node, Scheduling and Testing] 1040 - Fixed the issue where pod with ordinal number lower than the rolling partitioning number was being deleted it was coming up with updated image. ([#120731](https://github.com/kubernetes/kubernetes/pull/120731), [@adilGhaffarDev](https://github.com/adilGhaffarDev)) [SIG Apps and Testing] 1041 - Fixes calculating the requeue time in the cronjob controller, which results in properly handling failed/stuck jobs ([#121327](https://github.com/kubernetes/kubernetes/pull/121327), [@soltysh](https://github.com/soltysh)) [SIG Apps] 1042 - Service Controller: update load balancer hosts after node's ProviderID is updated ([#120492](https://github.com/kubernetes/kubernetes/pull/120492), [@cezarygerard](https://github.com/cezarygerard)) [SIG Cloud Provider and Network] 1043 1044 ## Dependencies 1045 1046 ### Added 1047 _Nothing has changed._ 1048 1049 ### Changed 1050 - cloud.google.com/go/compute: v1.19.0 → v1.19.1 1051 - github.com/cncf/xds/go: [06c439d → e9ce688](https://github.com/cncf/xds/go/compare/06c439d...e9ce688) 1052 - github.com/envoyproxy/go-control-plane: [v0.10.3 → 9239064](https://github.com/envoyproxy/go-control-plane/compare/v0.10.3...9239064) 1053 - github.com/envoyproxy/protoc-gen-validate: [v0.9.1 → v0.10.1](https://github.com/envoyproxy/protoc-gen-validate/compare/v0.9.1...v0.10.1) 1054 - github.com/golang/glog: [v1.0.0 → v1.1.0](https://github.com/golang/glog/compare/v1.0.0...v1.1.0) 1055 - google.golang.org/grpc: v1.54.0 → v1.56.3 1056 - google.golang.org/protobuf: v1.30.0 → v1.31.0 1057 1058 ### Removed 1059 _Nothing has changed._ 1060 1061 1062 1063 # v1.28.3 1064 1065 1066 ## Downloads for v1.28.3 1067 1068 1069 1070 ### Source Code 1071 1072 filename | sha512 hash 1073 -------- | ----------- 1074 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes.tar.gz) | 98fd6d3713e8708e7664adf7e9fcae73b570ec0e45b40aa9e8344eb9301b5b82c103e263347bf6996813ef6c8df302727754b955c20afa1c69f3784c0a2432d5 1075 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-src.tar.gz) | 1568c2f0464dd4c3c99e636dcd8ff6ec7716ae0c7e2c6bcb0b98cf30006f282bc011a2296a449026886f84ff7d37963f59e4cc5afdf45ec8d392b7d71a738f55 1076 1077 ### Client Binaries 1078 1079 filename | sha512 hash 1080 -------- | ----------- 1081 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-darwin-amd64.tar.gz) | a49da64f8408cd91e082ef199daf5f1d84460620a78c8f9a65ee0b1905a02b4f4ade2abe95e342291c4ea341be2dccb53cdd9b7f05ee79c33772c786f36e116f 1082 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-darwin-arm64.tar.gz) | 5565934425a12c8a38e2270839624dcf617346ceca07c2b5f8fda25940c6361b6ec948babb2d02d855edb2fadaa57c12856a8f7fc67a34a606710486b326a4ce 1083 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-386.tar.gz) | 58f0a7342903350e25acca29ffb59851fff47c49e66a4d5f27e73b49baa570596741dc1989a53f0a84361d5dcc1f41a3bb3bf8369ee7c7ac85275056bd17e59b 1084 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-amd64.tar.gz) | 0d5e1d09eb0008a67b1d59aa63e8b6e7e7230ba1ab32cdac8a722188d166f5dc9008b595947c42aff8a410596ece0a4346cd19ac9ab3a2913cee0eaab127b238 1085 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-arm.tar.gz) | eae7aa6f40b94dd3d098f3a5c788e1b9dc3051a055b6bac64a602c9ebfcc70645231269c0322abf94f9fd65348c16c0be78385d324e80e00ef4843b0f82cd49f 1086 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-arm64.tar.gz) | 95192e41d73e4b0585b8fd54e4c79c92c1ed9d37b80dfa8c8d3a1e289b5a7c32d67de4f5fce193e5f0fc82867f7c6f73d75fdc516ca437c236930eff90106088 1087 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-ppc64le.tar.gz) | 3e4289893cb3f8492d99b77d09eddaa3d55ee2cbf6a70c7ff1c9ce38f6b744c62c545a55d01d43d716876c648f628044945e90c361a82cc1a008ae808b29c92b 1088 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-s390x.tar.gz) | 8a2b4a3180752612dd636c26853e9b23fb7a58a8b631f6f29172c3002ebdf7086d7def21a187ced179fda4d08a613390dee1a0dae46e5d136c9cb8813da54049 1089 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-windows-386.tar.gz) | 57c0c4dfffe7e81d5144ee672d200e8204aab635701b2418e6cdb6eb130a65bbe08f7651dfdad7ae047818a33fb37aad2b7d6bfbb2d853c35108f1462c3cfa27 1090 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-windows-amd64.tar.gz) | 3607e1ce781dcd636de8bc7f470257675c6e333bb6a56948016463b2581b2ed4bad1f7b19edf5e0fccaa767d4cc57fdc196f0ce18182001901f0084bd8c5b98b 1091 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-windows-arm64.tar.gz) | a1fcacceccf712c752521a505e14aed113c75f40b690293e9f5411c5bfaeab9946cd2f067cdf7c4e5f57407c104feb9f4fa6a74c763653c460b89cae4a0d317e 1092 1093 ### Server Binaries 1094 1095 filename | sha512 hash 1096 -------- | ----------- 1097 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-server-linux-amd64.tar.gz) | cca2f7a1aa100c2ccc789536f27015848a45c7261523e605a2dcd0d49a06db85320706725c7f34cb9e90402f6d3349798a6d62c160e6811acb7c5bccd54aaff2 1098 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-server-linux-arm64.tar.gz) | b5b2705a45d0ce2bf7bc3b2a5854796497b5b88a77aabae162fd6fb9e20c1fa71ba620c3183d4098a4f2f9f029406e3c5a36c9d21ac2471e51029fbee984c3db 1099 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-server-linux-ppc64le.tar.gz) | 9b8637025481cb4774491634b62d03f94e152728710fe2e3b08a5f895d1d2902ec287f73a07ef441e16c5af8f3fc20659eec478f12f816948d2229123378a0f2 1100 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-server-linux-s390x.tar.gz) | 78b2eb80422e23e7124f54103b83438d1ef18f25dd7c0d598f42f2aac5982f49165fee7da5a0d407ac81dda11161bc6a676cb4ce39fdcd5b00347662848f7428 1101 1102 ### Node Binaries 1103 1104 filename | sha512 hash 1105 -------- | ----------- 1106 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-linux-amd64.tar.gz) | fa04b8aaaa0c8cf9eec44f5a7b623547ab4821db45c6cd8c877b2eb0b6419c5f5ee5f2181af5bad9d1017811e1ea7b78362e1d0d6ba455e5c0cf899f2ce7d996 1107 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-linux-arm64.tar.gz) | f18549315c58a86b4ec96eb04ea3dc10e3c01f9b835f721d8d20de4053e345e2789d7a4c211422a10f0f2b0ec3f12535766f61aabf174f400454434d4534c649 1108 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-linux-ppc64le.tar.gz) | 03bbf9a6053a42fef41048470cd4e8956bb0cd6b3c407ce86f5192f2a2c95c97ad9c9823d7e257882bf4cf2489b5225a3f49e7948ff7f1e758d4084faef0c1b8 1109 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-linux-s390x.tar.gz) | 2495e2810f763174102884d25e932d2dcd6984adf1bfe6733837c23174b20ee8fe47d6a9961ccc86fe233a161544d153a462e71badf6acc8d4f89513d82bcd37 1110 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-windows-amd64.tar.gz) | 0e29716b0e16bd3aa3e85d85fb6ddc0de895a7ad4b0d6aacc5503c93b12f67139138f92b2b0dbf91001f32a3c9ac31a0f9116cfff3701d3546ac204f58791c0a 1111 1112 ### Container Images 1113 1114 All container images are available as manifest lists and support the described 1115 architectures. It is also possible to pull a specific architecture directly by 1116 adding the "-$ARCH" suffix to the container image name. 1117 1118 name | architectures 1119 ---- | ------------- 1120 [registry.k8s.io/conformance:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 1121 [registry.k8s.io/kube-apiserver:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 1122 [registry.k8s.io/kube-controller-manager:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 1123 [registry.k8s.io/kube-proxy:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 1124 [registry.k8s.io/kube-scheduler:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 1125 [registry.k8s.io/kubectl:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 1126 1127 ## Changelog since v1.28.2 1128 1129 ## Changes by Kind 1130 1131 ### Feature 1132 1133 - Kubernetes is now built with Go 1.20.10 ([#121153](https://github.com/kubernetes/kubernetes/pull/121153), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 1134 - Kubernetes is now built with Go 1.20.9 ([#121025](https://github.com/kubernetes/kubernetes/pull/121025), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 1135 1136 ### Failing Test 1137 1138 - E2e framework: retrying after intermittent apiserver failures was fixed in WaitForPodsResponding ([#120559](https://github.com/kubernetes/kubernetes/pull/120559), [@pohly](https://github.com/pohly)) [SIG Testing] 1139 1140 ### Bug or Regression 1141 1142 - Adds an opt-in mitigation for http/2 DOS vulnerabilities for CVE-2023-44487 and CVE-2023-39325 for the API server when the client is unauthenticated. The mitigation may be enabled by setting the `UnauthenticatedHTTP2DOSMitigation` feature gate to `true` (it is disabled by default). An API server fronted by an L7 load balancer that already mitigates these http/2 attacks may choose not to enable the kube-apiserver mitigation to avoid disrupting load balancer → kube-apiserver connections if http/2 requests from multiple clients share the same backend connection. An API server on a private network may choose not to enable the kube-apiserver mitigation to prevent performance regressions for unauthenticated clients. Authenticated requests rely on the fix in golang.org/x/net v0.17.0 alone. https://issue.k8s.io/121197 tracks further mitigation of http/2 attacks by authenticated clients. ([#121196](https://github.com/kubernetes/kubernetes/pull/121196), [@enj](https://github.com/enj)) [SIG API Machinery] 1143 - Fix 1.28.0 regression where adding aggregated APIService objects could cause apiserver to panic and affect the health check ([#121040](https://github.com/kubernetes/kubernetes/pull/121040), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing] 1144 - Fix a bug in cronjob controller where already created jobs may be missing from the status. ([#120649](https://github.com/kubernetes/kubernetes/pull/120649), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps] 1145 - Fixed a 1.28.0 regression where kube-controller-manager can crash when StatefulSet with Parallel policy and PVC labels is scaled up. ([#121184](https://github.com/kubernetes/kubernetes/pull/121184), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) [SIG Apps] 1146 - Fixed a bug where containers would not start on cgroupv2 systems where swap is disabled. ([#120924](https://github.com/kubernetes/kubernetes/pull/120924), [@klueska](https://github.com/klueska)) [SIG Node] 1147 - Fixed a regression in kube-proxy where it might refuse to start if given 1148 single-stack IPv6 configuration options on a node that has both IPv4 and 1149 IPv6 IPs. ([#121008](https://github.com/kubernetes/kubernetes/pull/121008), [@danwinship](https://github.com/danwinship)) [SIG Network] 1150 - Fixed an issue to not drain all the pods in a namespace when an empty-selector i.e. "{}" is specified in a Pod Disruption Budget (PDB) ([#121131](https://github.com/kubernetes/kubernetes/pull/121131), [@sairameshv](https://github.com/sairameshv)) [SIG Apps] 1151 - Fixed attaching volumes after detach errors. Now volumes that failed to detach are not treated as attached, Kubernetes will make sure they are fully attached before they can be used by pods. ([#120595](https://github.com/kubernetes/kubernetes/pull/120595), [@jsafrane](https://github.com/jsafrane)) [SIG Apps and Storage] 1152 - Fixed bug to surface events for the following metrics: apiserver_encryption_config_controller_automatic_reload_failures_total, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds, apiserver_encryption_config_controller_automatic_reload_success_total ([#120544](https://github.com/kubernetes/kubernetes/pull/120544), [@ritazh](https://github.com/ritazh)) [SIG API Machinery, Auth and Testing] 1153 - Fixes a bug where Services using finalizers may hold onto ClusterIP and/or NodePort allocated resources for longer than expected if the finalizer is removed using the status subresource ([#120654](https://github.com/kubernetes/kubernetes/pull/120654), [@aojea](https://github.com/aojea)) [SIG Testing] 1154 - Revised the logic for DaemonSet rolling update to exclude nodes if scheduling constraints are not met. 1155 This eliminates the problem of rolling updates to a DaemonSet getting stuck around tolerations. ([#120785](https://github.com/kubernetes/kubernetes/pull/120785), [@mochizuki875](https://github.com/mochizuki875)) [SIG Apps and Testing] 1156 - Sometimes, the scheduler incorrectly placed a pod in the "unschedulable" queue instead of the "backoff" queue. This happened when some plugin previously declared the pod as "unschedulable" and then in a later attempt encounters some other error. Scheduling of that pod then got delayed by up to five minutes, after which periodic flushing moved the pod back into the "active" queue. ([#120334](https://github.com/kubernetes/kubernetes/pull/120334), [@pohly](https://github.com/pohly)) [SIG Scheduling] 1157 1158 ### Other (Cleanup or Flake) 1159 1160 - Fixes an issue where the vsphere cloud provider will not trust a certificate if: 1161 - The issuer of the certificate is unknown (x509.UnknownAuthorityError) 1162 - The requested name does not match the set of authorized names (x509.HostnameError) 1163 - The error surfaced after attempting a connection contains one of the substrings: "certificate is not trusted" or "certificate signed by unknown authority" ([#120768](https://github.com/kubernetes/kubernetes/pull/120768), [@MadhavJivrajani](https://github.com/MadhavJivrajani)) [SIG Architecture and Cloud Provider] 1164 - Set the resolution for the job_controller_job_sync_duration_seconds metric from 4ms to 1min ([#120667](https://github.com/kubernetes/kubernetes/pull/120667), [@mimowo](https://github.com/mimowo)) [SIG Apps and Instrumentation] 1165 1166 ## Dependencies 1167 1168 ### Added 1169 _Nothing has changed._ 1170 1171 ### Changed 1172 - github.com/vmware/govmomi: [v0.30.0 → v0.30.6](https://github.com/vmware/govmomi/compare/v0.30.0...v0.30.6) 1173 - golang.org/x/crypto: v0.11.0 → v0.14.0 1174 - golang.org/x/net: v0.13.0 → v0.17.0 1175 - golang.org/x/sys: v0.10.0 → v0.13.0 1176 - golang.org/x/term: v0.10.0 → v0.13.0 1177 - golang.org/x/text: v0.11.0 → v0.13.0 1178 1179 ### Removed 1180 _Nothing has changed._ 1181 1182 1183 1184 # v1.28.2 1185 1186 1187 ## Downloads for v1.28.2 1188 1189 1190 1191 ### Source Code 1192 1193 filename | sha512 hash 1194 -------- | ----------- 1195 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes.tar.gz) | f6e13f2632697aab3ce6230d777240dd3d9c23b65eba7ff7d1df5d330e4dd926f8f439b77d823f8d08f44ddcd7eeca476af6d83eaa29cf623e86f2e4f315074a 1196 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-src.tar.gz) | 3c21536962251eb199e4b0f42379cdfa172e826b10a28d8946df23bb8bae5e12d09647448d1f7a9c7146166178dd38398dee308dcc1e604000be908e1e0bbe89 1197 1198 ### Client Binaries 1199 1200 filename | sha512 hash 1201 -------- | ----------- 1202 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-darwin-amd64.tar.gz) | b85d0f2da76708a934cb8cecdf08a2d7c146c8f8209f49deab82b01c15842cf3c0631e01977af20230e69d478dfb21b5bf6acf9fc985d9ae27d1126f7a9f1112 1203 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-darwin-arm64.tar.gz) | 0e556a34b3c659c45a9368b8b0d709831bdfa6562adb48dd5c924085806f3e1b76d4ba3b5dd719bc2d126f1bd640ddc94ddbd37515168a4de1a358b4605e031a 1204 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-386.tar.gz) | a091434bc89f762655fb76e301c9287297ab48079118eb045589d6ef246fcda307a1799732178c568ea8c64211b5bdae3ba7a836bdd60d1c61bb7dcb7b7ee324 1205 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-amd64.tar.gz) | ab534cd06d8cc89d1288590cfae98415facaa7db2f481d8f6be0a20574d2990cc55348cf98386c34df7788aa80ff018fc844816a2b605bcb350a82d752738fdc 1206 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-arm.tar.gz) | 31fb2570bd4dff5ad9f6525e33fa80847ee35d2804a1c81af8ce27855a1b4d8267bea3b522ac90bd49bc5c6a9a9fd9388a14899ebd48baac214a98644f357e02 1207 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-arm64.tar.gz) | 2304f6888752dd22e898526df091b66aae85835690e922a8e017d57e077dd1c8fcdfca16fc5aba94e9fb51ad800832305a574aee24c6cfa5d37278c8e28a144e 1208 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-ppc64le.tar.gz) | 5dbb4fefd197b14ccdc3b82d088b0e09987b1a8afbf47e03abb6707c455716658ed95d7e2dce7e5d7e12981febd53eebd0ed6296d282d2ce317c191a2c8116e9 1209 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-s390x.tar.gz) | 455e436244ff306604eb5d8a230a24186799ef5c462b7f278bb2d62e36245639db7a0d89241682dc201351a4c648ba788acdd7ac73a486cd6db426e1079ed87b 1210 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-windows-386.tar.gz) | 96a4cf768fb59ad626a383bddcc9bab433f9d309ba3b06a8c9d927799a5ee6c4645412d22a190255c9a3e7104bf6d914f5976fcdd39876bdf1598c72dabb68a7 1211 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-windows-amd64.tar.gz) | e7075cf3f0103edd22962fdb5a9adb4f80249a4adc8309794fc15f2f3d8e934feee6adc47c1724cc3a1c497d9cc0d4afbe0d66511042670cd2f1da1d82c70ba3 1212 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-windows-arm64.tar.gz) | d230193bc1f73f6834f0a9e919b673b7fcd645343773d9cf05b33ce95d81d5ed6f8efa1620e7099e8291beab47a2f3e058dde25a96fd0873b2de2ab28e7c3b7d 1213 1214 ### Server Binaries 1215 1216 filename | sha512 hash 1217 -------- | ----------- 1218 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-server-linux-amd64.tar.gz) | cd54f2f48733806208d3b1585ed307bebf91893e250a1eb3e18355c9a9e6d1f75a70966cb66165be2fbb8566e5368b14f66f63a8929fbceca30aa73bfd491441 1219 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-server-linux-arm64.tar.gz) | e879568fb40ac54b897ac52e39aa3077278f9ea502c4b7d639a247d503be85db623c264cc30300016ac1651db2c93bc82420decb5e904af396f54e40d3aa33ff 1220 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-server-linux-ppc64le.tar.gz) | 212ef04a6d443f239fa6ecb34df5f4fc93f172dab5a5d2931a0554eafdd3901e8504f6e8918712c44f843f3be55adb83c241839d60a9de1d4b6988035f47dc6d 1221 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-server-linux-s390x.tar.gz) | b7efe45fbb811ee09e0c8daec0b608546f065c389e2b480c0d5178ea778690f11fc3d880a84baba2331bb30042898a616b5a4f7934456543aaa9b3fc0169f923 1222 1223 ### Node Binaries 1224 1225 filename | sha512 hash 1226 -------- | ----------- 1227 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-linux-amd64.tar.gz) | adafa3beb4525d898a602ec87f1d2b35ad89f71f6b89aad81ee0cdff97a2916ef323a21b5007d61bb0453e12d3058ad21861184049eb22c26b86be4dc268e7d9 1228 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-linux-arm64.tar.gz) | c9ece93728bc2712004d508c37e692add5cc358437ee0b209ef062b81db137cccf85bf9f4560c6436c0b83193a4ba23fa28e06dd432281b946f235aa30e88842 1229 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-linux-ppc64le.tar.gz) | 85918f6235563c10d58ae18e0a1918011492c4f812a753ba185447b6080aae88ccf94c6008519694fb9fca4b2a9cc02dec6ae91f1ca55a489b383d85c8f7fa9c 1230 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-linux-s390x.tar.gz) | ea59ee997d3df3d89405be4943dbe8f76cc8066e98173db30f0f83ad06a9c60c73217004f837f14310bf17228fd0c4e2d2917632ddda3cc3e483c353edc38745 1231 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-windows-amd64.tar.gz) | b8125003fcd8fa7e89d7cbc49f634cc01cd8fdf198a3325f22e5bb3743923e03e82c3d4a7545b32b0da82c950b8d887997e502839359027f194de4a0a4774654 1232 1233 ### Container Images 1234 1235 All container images are available as manifest lists and support the described 1236 architectures. It is also possible to pull a specific architecture directly by 1237 adding the "-$ARCH" suffix to the container image name. 1238 1239 name | architectures 1240 ---- | ------------- 1241 [registry.k8s.io/conformance:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 1242 [registry.k8s.io/kube-apiserver:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 1243 [registry.k8s.io/kube-controller-manager:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 1244 [registry.k8s.io/kube-proxy:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 1245 [registry.k8s.io/kube-scheduler:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 1246 [registry.k8s.io/kubectl:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 1247 1248 ## Changelog since v1.28.1 1249 1250 ## Changes by Kind 1251 1252 ### API Change 1253 1254 - Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps. 1255 The incorrect cost was evident when the result of a function was used in subsequent operations. ([#119807](https://github.com/kubernetes/kubernetes/pull/119807), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider] 1256 - Mark Job onPodConditions as optional in pod failure policy ([#120208](https://github.com/kubernetes/kubernetes/pull/120208), [@mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps] 1257 1258 ### Feature 1259 1260 - Kubernetes is now built with Go 1.20.8 ([#120495](https://github.com/kubernetes/kubernetes/pull/120495), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing] 1261 1262 ### Bug or Regression 1263 1264 - Fix OpenAPI v3 not being cleaned up after deleting APIServices ([#120108](https://github.com/kubernetes/kubernetes/pull/120108), [@tnqn](https://github.com/tnqn)) [SIG API Machinery and Testing] 1265 - Fix a 1.28 regression in scheduler: a pod with concurrent events could incorrectly get moved to the unschedulable queue where it could got stuck until the next periodic purging after 5 minutes if there was no other event for it. ([#120445](https://github.com/kubernetes/kubernetes/pull/120445), [@pohly](https://github.com/pohly)) [SIG Scheduling] 1266 - Fix a concurrent map access in TopologyCache's `HasPopulatedHints` method. ([#120372](https://github.com/kubernetes/kubernetes/pull/120372), [@Miciah](https://github.com/Miciah)) [SIG Network] 1267 - Fixed a 1.26 regression scheduling bug by ensuring that preemption is skipped when a PreFilter plugin returns `UnschedulableAndUnresolvable` ([#119951](https://github.com/kubernetes/kubernetes/pull/119951), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 1268 - Fixed a 1.27 scheduling regression that PostFilter plugin may not function if previous PreFilter plugins return Skip ([#119942](https://github.com/kubernetes/kubernetes/pull/119942), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling and Testing] 1269 - Fixed a 1.28 regression around restarting init containers in the right order relative to normal containers ([#120440](https://github.com/kubernetes/kubernetes/pull/120440), [@gjkim42](https://github.com/gjkim42)) [SIG Node and Testing] 1270 - Fixed a regression in default 1.27 configurations in kube-apiserver: fixed the AggregatedDiscoveryEndpoint feature (beta in 1.27+) to successfully fetch discovery information from aggregated API servers that do not check `Accept` headers when serving the `/apis` endpoint ([#120359](https://github.com/kubernetes/kubernetes/pull/120359), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery] 1271 - Fixes a 1.28 regression handling negative index json patches ([#120329](https://github.com/kubernetes/kubernetes/pull/120329), [@liggitt](https://github.com/liggitt)) [SIG API Machinery] 1272 - Fixes a bug where images pinned by the container runtime can be garbage collected by kubelet. ([#120053](https://github.com/kubernetes/kubernetes/pull/120053), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG Node] 1273 - Ignore context canceled from validate and mutate webhook ([#120155](https://github.com/kubernetes/kubernetes/pull/120155), [@divyasri537](https://github.com/divyasri537)) [SIG API Machinery] 1274 - Kubeadm: fix nil pointer when etcd member is already removed ([#120010](https://github.com/kubernetes/kubernetes/pull/120010), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 1275 1276 ## Dependencies 1277 1278 ### Added 1279 _Nothing has changed._ 1280 1281 ### Changed 1282 - github.com/evanphx/json-patch: [v5.6.0+incompatible → v4.12.0+incompatible](https://github.com/evanphx/json-patch/compare/v5.6.0...v4.12.0) 1283 - github.com/google/cel-go: [v0.16.0 → v0.16.1](https://github.com/google/cel-go/compare/v0.16.0...v0.16.1) 1284 1285 ### Removed 1286 _Nothing has changed._ 1287 1288 1289 1290 # v1.28.1 1291 1292 1293 ## Downloads for v1.28.1 1294 1295 1296 1297 ### Source Code 1298 1299 filename | sha512 hash 1300 -------- | ----------- 1301 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes.tar.gz) | 87829907851a0f75bdca725d89c1290ca898fb92c0c6271fc6ddda8c06361bb0131306700b5c7c96d9f083b223e61e3d4cc55479de21cd4de64ab942ce2f91e2 1302 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-src.tar.gz) | 52297b3ef9082933d55973bddd54249dccb0b3aeb6ca131d80a35d6c60c70711042b6373d66fb1bf9e79046a4a366060a31faef0033ae5e29f14df8e5f1c6f87 1303 1304 ### Client Binaries 1305 1306 filename | sha512 hash 1307 -------- | ----------- 1308 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-darwin-amd64.tar.gz) | 2a0bdb8dffb4d710ee01ca48437da9f80f8568efb4c9ac7a13b201d307e9f68698b2c102b8fcd2f59e4082cd330229ac5febfb6e99dea2a90c2aa93aa176d720 1309 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-darwin-arm64.tar.gz) | 7824c1907510fda1a91870f55adad9ff4d5e7b01d2cfb9805e7b8c56c24aaa4b6aefd3ad5f374eb6f97056142d5d5ea9ed552347d293789eafe3bc56fcd17326 1310 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-386.tar.gz) | ef812e222d771bb29209d3a66e4168c0f03922b3cc8585151833ce304f30b124e97cec27fa2ae5abe990bf901f27db81835c817e70ae6738d2b58e863854e996 1311 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-amd64.tar.gz) | b5e9823886c8c26c22078cf5cd233612f38240e5ceb3c7bc5c032fbbfee59f6a631b53aa541bf8afc2eba496f5d0476357d1738cf771aaa95661c83d91372b51 1312 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-arm.tar.gz) | fae089315283125f5a30103260319daefaf092aa488175000ad19ff55bb90b62f9b7d3b9fe446b3103087f06fd353d96f521a8f8f33cd2cf3e0887183ff3087f 1313 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-arm64.tar.gz) | a46645ca155fb3a98159678b92f0075a14ea731a0e0feebc1f2f243e2b50f7413272969aabd6feb7a68f1d64de66fb330e389edc326c8ec90457b3c7c9b25783 1314 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-ppc64le.tar.gz) | 45259cabe085abba6d0255bdecd6890c89a098a7dcc8f65013acdaf1471a013d4671c04b047f36c160b7623e476266c375e0c87e3e1d3ba666c9f27a184bfc78 1315 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-s390x.tar.gz) | 72f8a4f15745bec71328bd3ec346cf886b0a56bc409bc0c756828bfa74e6e52e7bf444a40da133a9de899df60c2e44082fd60e5532d9f1fef8c948827e8ef51e 1316 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-windows-386.tar.gz) | 7a2b3d717f586b14720c97642a2f7a8bbb6720f874a4dbb22a8f450598017e3247ed1c1e861993a489fedb10ebd782aa074182061323b0c3509ace10534b9a3f 1317 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-windows-amd64.tar.gz) | ebc759a5164fbe9c292db46c9ed2070d6162a2ba7bc2d246d2538d5d9322ca08d309507e428b954616e0c4fe21db32671300970bb0fbd8286e85461d9dcc4487 1318 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-windows-arm64.tar.gz) | a77c2ed43148fe914d483862481dbe70ef459bf56d1c4b7a75e4146b67be41b5d54756b94414ad6013c31a13affe13594600462551a64bc0ce82883b8b44cc35 1319 1320 ### Server Binaries 1321 1322 filename | sha512 hash 1323 -------- | ----------- 1324 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-server-linux-amd64.tar.gz) | 5727d958896b8032667111da019b0ea39ed1040f966c6758180f5f42656c5c78965cd61c162d64ae9df830fcfaddf61c1d05433430facb20fe2b96e4e1a4716f 1325 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-server-linux-arm64.tar.gz) | ab3df8c2d10a91ab155b2c615b4ae95650d949294503a3863fd93181417287e50a4b1b3641e48e6f729c2fa666b4d62e620234841c5047ed9789d567cb600a37 1326 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-server-linux-ppc64le.tar.gz) | 4a8a86e10b90944fbb0a80f55b99bba77ff5e82806cb11a13286c9e434523eee3723960ad9615a5d44e74e693041575624f2e82132c17441d9faa4bd21170a59 1327 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-server-linux-s390x.tar.gz) | 12ebaeba75861087d058cbf65b2853d54a802d61408f7fd05480d0f00ebb06240d00b84cbfc445f66478f359deb65a265e0108f7e4f1f82dc664c12be82a17f9 1328 1329 ### Node Binaries 1330 1331 filename | sha512 hash 1332 -------- | ----------- 1333 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-linux-amd64.tar.gz) | d736a6abfeec797a48944caf926c737fb67e9cb2fa20913b4d5c2b1b12b2aca550a82bb4a20836f15fe8c4054e5fe7aebd3f380845054eaad9774fd68296ac48 1334 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-linux-arm64.tar.gz) | c1f7984b4f173c98bc0020348848eee50a42193fb49803229bc79efafc89ea4e9d23398aea96f393265e6ba2705b0c0edb1372b0052f6567b455a12798ec9f4e 1335 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-linux-ppc64le.tar.gz) | 94f4fe0cc6185d8ee361b04c49abdbd768863877f54bb641eb02c787f8d6011ad6dfc074e496c1a9004f9fa3143e1d546f3904059ded0866d9d4ab9eca0ab670 1336 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-linux-s390x.tar.gz) | e81b885ea5eb00628f67715f72bcd4b9d5d143b0bef4f9a40f592c3ee7ffefb01a35241e4801c3a3e1e353944eb91797c9346e834410478345a781e8b8ae1b40 1337 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-windows-amd64.tar.gz) | 0b049a819495249820c64f7b728b6abf31548c977a501cd4a9a60ab60bed35c688677db443002735033dbb3a2f6ae13842b07b81c7f72eb3d13f2f900f4d421a 1338 1339 ### Container Images 1340 1341 All container images are available as manifest lists and support the described 1342 architectures. It is also possible to pull a specific architecture directly by 1343 adding the "-$ARCH" suffix to the container image name. 1344 1345 name | architectures 1346 ---- | ------------- 1347 [registry.k8s.io/conformance:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 1348 [registry.k8s.io/kube-apiserver:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 1349 [registry.k8s.io/kube-controller-manager:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 1350 [registry.k8s.io/kube-proxy:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 1351 [registry.k8s.io/kube-scheduler:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 1352 [registry.k8s.io/kubectl:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 1353 1354 ## Changelog since v1.28.0 1355 1356 ## Important Security Information 1357 1358 This release contains changes that address the following vulnerabilities: 1359 1360 ### CVE-2023-3955: Insufficient input sanitization on Windows nodes leads to privilege escalation 1361 1362 A security issue was discovered in Kubernetes where a user that can create pods on Windows nodes may be able to escalate to admin privileges on those nodes. Kubernetes clusters are only affected if they include Windows nodes. 1363 1364 **Affected Versions**: 1365 - kubelet <= v1.28.0 1366 - kubelet <= v1.27.4 1367 - kubelet <= v1.26.7 1368 - kubelet <= v1.25.12 1369 - kubelet <= v1.24.16 1370 1371 **Fixed Versions**: 1372 - kubelet v1.28.1 1373 - kubelet v1.27.5 1374 - kubelet v1.26.8 1375 - kubelet v1.25.13 1376 - kubelet v1.24.17 1377 1378 This vulnerability was discovered by James Sturtevant @jsturtevant and Mark Rossetti @marosset during the process of fixing CVE-2023-3676 (that original CVE was reported by Tomer Peled @tomerpeled92) 1379 1380 1381 **CVSS Rating:** High (8.8) [CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H) 1382 1383 1384 ### CVE-2023-3676: Insufficient input sanitization on Windows nodes leads to privilege escalation 1385 1386 A security issue was discovered in Kubernetes where a user that can create pods on Windows nodes may be able to escalate to admin privileges on those nodes. Kubernetes clusters are only affected if they include Windows nodes. 1387 1388 **Affected Versions**: 1389 - kubelet <= v1.28.0 1390 - kubelet <= v1.27.4 1391 - kubelet <= v1.26.7 1392 - kubelet <= v1.25.12 1393 - kubelet <= v1.24.16 1394 1395 **Fixed Versions**: 1396 - kubelet v1.28.1 1397 - kubelet v1.27.5 1398 - kubelet v1.26.8 1399 - kubelet v1.25.13 1400 - kubelet v1.24.17 1401 1402 This vulnerability was reported by Tomer Peled @tomerpeled92 1403 1404 1405 **CVSS Rating:** High (8.8) [CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H) 1406 1407 ## Changes by Kind 1408 1409 ### Other (Cleanup or Flake) 1410 1411 - Fixes ability to build 1.28 without network access ([#119982](https://github.com/kubernetes/kubernetes/pull/119982), [@liggitt](https://github.com/liggitt)) [SIG Testing] 1412 1413 ## Dependencies 1414 1415 ### Added 1416 _Nothing has changed._ 1417 1418 ### Changed 1419 _Nothing has changed._ 1420 1421 ### Removed 1422 _Nothing has changed._ 1423 1424 1425 1426 # v1.28.0 1427 1428 [Documentation](https://docs.k8s.io) 1429 1430 ## Downloads for v1.28.0 1431 1432 ### Source Code 1433 1434 filename | sha512 hash 1435 -------- | ----------- 1436 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes.tar.gz) | `9aaf7cc004d09297dc7bbc1f0149d6424d85717b0f31658997bf9a1eee7343ad1ede25e506e1b85956f6b08393d5c7b58e59de860c2f880d97544fd79dfae9da` 1437 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-src.tar.gz) | `8e9071210316caac1762535d9437a1e7dcbf644ae8852f4d5babd19a743810c3a2ff2b30f009ba47d28a50e7f5691e56f9b273444bc387dbe95be56c71f2c9d4` 1438 1439 ### Client Binaries 1440 1441 filename | sha512 hash 1442 -------- | ----------- 1443 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-darwin-amd64.tar.gz) | `e9282dc96a73431ed50ef0c515be2a7953b4a243d42ce817e282006aa0431f4f3909971701c4847ea2dd9f268de13fbca40424eff6316f7697faebc2bc0fcff6` 1444 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-darwin-arm64.tar.gz) | `d0c24710717979494d32e9e518435e0803c297a369a1e5022fb522df6a2d4cf91112ad2a3e583abc85080c8ead9644dbb42a6387518a834e5f3d93ca097d0977` 1445 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-386.tar.gz) | `8f55c1fa60e50c40a81551ab2d6fdf9327a57c445e281105f426ff686395097219766869edef1935b299feb58ab01c9612a1efe3cd1ef06bdc09fc2f93f2ba3a` 1446 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-amd64.tar.gz) | `0506d82a49794089137e0f240deb3bbaf48be6e61b5b242af02a0d2a0f94fb1df55fcb87570cf40f9abec6b2d6bf11d40fa5d66ab0829ea43169448bda2609d1` 1447 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-arm.tar.gz) | `3db12e083be8f7a7555a391311f78064a0ffa51f5cfdb6509ee71f33ee5bf56d986e687c5eb39e01c9def7b2154b0298a41c0b960c1fc76228b99c39546529c9` 1448 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-arm64.tar.gz) | `c907332aa6b38ecd82ed7c16741d80e8f23857b49ebff05bdb89692ab286955e03537cc2aba631e932c56bcbcc2aafee2a9b69c5bb6496b869d31b771dc93759` 1449 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-ppc64le.tar.gz) | `fccda39ca81d171ff8bf30a27596ea6e13a6fca7113c3b46f1c2915d50a367b96b6db2e7d8e27fd76c5b3b00f3d447b1da4d1a70fbaf652a7b2b2c4aae71853f` 1450 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-s390x.tar.gz) | `f177677ba4c7e59901ba5cbc10d23384e69cabbbe7f924d0bf0267534eefca4ed0b855ad7193533f5034b080d9894278393b9012b008dd17056d9684aa36e7d3` 1451 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-windows-386.tar.gz) | `bc3cc07e5101c1d1b4484f28b748b03083ad8b1a74c51d67b7c9d628c65a9db07d20f0695f458508a88a259d1f4396b2008d898476716998a32d74dd84901320` 1452 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-windows-amd64.tar.gz) | `6969e153438cfc3165530562b4bb4cb620588a8b59f1f28bb6a369d7e4ba5f636613d3cab2925d1b00385239ac82bb46bffa00cbbadc3aed1ab54ed620909de2` 1453 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-windows-arm64.tar.gz) | `3955501ba210e0af25df1431e8b9f641b6d746d0fb882192a24b1d2b18b55ddb48e0c3ab086a0f6e0bf2156a38e11b979f78d5d7b898e24dc34e10f1d558abb0` 1454 1455 ### Server Binaries 1456 1457 filename | sha512 hash 1458 -------- | ----------- 1459 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-server-linux-amd64.tar.gz) | `36476c4c64f23b3a48d6e79885ddc764102eefd2ab25b1d721386bdfc15c954d7940ab905615bec357dc5530610ecf7b8e640790206cb5a8da8312cea46db97a` 1460 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-server-linux-arm64.tar.gz) | `42c17eb2229f4210521bb78ff7712de9ac4a3e483b503e4a4f9a889a4aea3015890360242d88e9f2dbcb3d4d645bbfc37b31a5d2f151023b594392c23d1b0154` 1461 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-server-linux-ppc64le.tar.gz) | `e8929e915e61cb9380f6e48abf21f6dfd00b52d7afc1867f8999f55fdf8c60404397dcec51f6feda7bd281fdb15035516a3187162349320460574b14c2a63f25` 1462 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-server-linux-s390x.tar.gz) | `88bce78171ff9b12796ccc3e6781a09afb05b7412dcf154aa1b9a9da151a01de07e8ef3a35e305822a67fbf0dfd482bc7320d0b7a39c6384f74139153b0644f8` 1463 1464 ### Node Binaries 1465 1466 filename | sha512 hash 1467 -------- | ----------- 1468 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-linux-amd64.tar.gz) | `fed9ba533e54a4dd6cf26788c27d4f41534ee4f6cf22ee75b183afc45764273e8ac008f06297608342797bc9463c82603947800dce37155424489e20987d3dea` 1469 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-linux-arm64.tar.gz) | `a500906699bc25a8c0825fd38e92b1ce5fddbb3bfc09561e21cfd04bab0cefe6430231b9f347835c8e1d06ffb926b72ae272c119eb72d64580b318d7fcacad20` 1470 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-linux-ppc64le.tar.gz) | `16bb9a4762fdb4f5140cf518a93d812ddb04c08cc98f0447d1c540d290648a8a050d2d6133e244b40645bb25813d149a96a313de5af178ed30a5dab2919fa845` 1471 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-linux-s390x.tar.gz) | `fb2cf3f819f8c3329fbfc13588a8b206bb16e3b4e351680ed03e3a74cc34b42341743f8f913941e25ed3ca2d7779bf331f31f30821787b1f8cb916f58f183ab4` 1472 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-windows-amd64.tar.gz) | `60e964a33f10cf0b361f50199aa19f0d89dd82073b31377e7d61b6fb761ef779cc3577bf407edb448c104333185d25eba8d8953e917f52cb62b504ea85121ac4` 1473 1474 ### Container Images 1475 1476 All container images are available as manifest lists and support the described 1477 architectures. It is also possible to pull a specific architecture directly by 1478 adding the "-$ARCH" suffix to the container image name. 1479 name | architectures 1480 ---- | ------------- 1481 [registry.k8s.io/conformance:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 1482 [registry.k8s.io/kube-apiserver:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 1483 [registry.k8s.io/kube-controller-manager:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 1484 [registry.k8s.io/kube-proxy:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 1485 [registry.k8s.io/kube-scheduler:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 1486 [registry.k8s.io/kubectl:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 1487 1488 ## Changelog since v1.27.0 1489 1490 ## Urgent Upgrade Notes 1491 1492 ### (No, really, you MUST read this before you upgrade) 1493 1494 - Action required for the custom scheduler plugin developers. 1495 Here's the breaking change in `EnqueueExtension` in the scheduling framework. 1496 The `EventsToRegister` in `EnqueueExtension` changed the return value from `ClusterEvent` to `ClusterEventWithHint`. `ClusterEventWithHint` allows each plugin to filter out more useless events via the callback function named `QueueingHintFn`. 1497 When the scheduling queue receives a cluster event, before moving each Pod from unschedulable pod pool to activeQ/backoffQ, it will call QueueingHintFn of plugins that rejected each Pod in the previous scheduling cycle. 1498 Depending on the value returned from QueueingHintFn, the scheduling queue changes how it queues each Pod: 1499 - if more than one QueueingHintFn returns QueueImmediately, it queues Pod to activeQ. 1500 - If no QueueingHintFn returns QueueImmediately and more than one plugin returns QueueAfterBackoff, it queues Pod to backoffQ if Pod is backing off, or to activeQ if Pod's backoff has already finished. 1501 - If all QueueingHintFn return QueueSkip, it puts this pod back to the unschedulable pod pool 1502 1503 Having appropriate QueueingHintFn contributes to reducing useless retries and thus improves the overall scheduler's performance. 1504 1505 **How can I migrate?** 1506 1507 For backward compatibility, nil `QueueingHintFn` is treated as always returning QueueAfterBackoff. 1508 So, if you want to just keep the existing behavior, you can register `ClusterEventWithHint` with no `QueueingHintFn` in it. 1509 But, registering appropriate `QueueingHintFn` is, of course, better from a scheduling performance perspective. ([#118551](https://github.com/kubernetes/kubernetes/pull/118551), [@sanposhiho](https://github.com/sanposhiho)) [SIG Node, Scheduling, Storage and Testing] 1510 - CephFS volume plugin (`kubernetes.io/cephfs`) has been deprecated in this release and will be removed in a subsequent release. The alternative is to use the CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes cluster. ([#118143](https://github.com/kubernetes/kubernetes/pull/118143), [@humblec](https://github.com/humblec)) 1511 - Deprecated support for CSI migration of `Ceph RBD volumes`. Users who were relying on Kubernetes' ability 1512 to migrate to an out-of-tree storage driver should complete that migration before the support for it is removed. ([#118303](https://github.com/kubernetes/kubernetes/pull/118303), [@carlory](https://github.com/carlory)) 1513 - RBD volume plugin (`kubernetes.io/rbd`) has been deprecated in this release 1514 and will be removed in a subsequent release. Alternative is to use RBD CSI driver 1515 (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. ([#118552](https://github.com/kubernetes/kubernetes/pull/118552), [@humblec](https://github.com/humblec)) 1516 1517 ## Changes by Kind 1518 1519 ### Deprecation 1520 1521 - Changed `kubectl version` default output to be identical to what `kubectl version --short` printed, 1522 and removed `--short` flag entirely. ([#116720](https://github.com/kubernetes/kubernetes/pull/116720), [@soltysh](https://github.com/soltysh)) 1523 - Kube-controller-manager deprecate `--volume-host-cidr-denylist` and `--volume-host-allow-local-loopback` flags. ([#118128](https://github.com/kubernetes/kubernetes/pull/118128), [@carlory](https://github.com/carlory)) [SIG API Machinery, Apps, Network, Node, Storage and Testing] 1524 - Kubelet: The `--azure-container-registry-config` flag has been deprecated and will be removed in a future release, please use `--image-credential-provider-config` and `--image-credential-provider-bin-dir` to setup acr credential provider instead. ([#118596](https://github.com/kubernetes/kubernetes/pull/118596), [@SataQiu](https://github.com/SataQiu)) [SIG Node] 1525 - Removed tracking annotation from validation and defaulting. ([#117633](https://github.com/kubernetes/kubernetes/pull/117633), [@kannon92](https://github.com/kannon92)) 1526 - Removed withdrawn feature `NetworkPolicyStatus`. ([#115843](https://github.com/kubernetes/kubernetes/pull/115843), [@rikatz](https://github.com/rikatz)) 1527 - The deprecated flag `--lock-object-namespace` and `--lock-object-name` have been removed from kube-scheduler. Please use `--leader-elect-resource-namespace` and `--leader-elect-resource-name` or ComponentConfig instead to configure those parameters. ([#119130](https://github.com/kubernetes/kubernetes/pull/119130), [@SataQiu](https://github.com/SataQiu)) [SIG Scheduling] 1528 - `KMSv1` is deprecated and will only receive security updates going forward. Use `KMSv2` instead. In a future release, Set `--feature-gates=KMSv1=true` to use the deprecated KMSv1 feature. ([#119007](https://github.com/kubernetes/kubernetes/pull/119007), [@aramase](https://github.com/aramase)) 1529 1530 ### API Change 1531 1532 - A CDIDevice field is included in the Device Plugin's `ContainerAllocateResponse`. This field maps to the CDIDevice field in the CRI protocol. ([#118254](https://github.com/kubernetes/kubernetes/pull/118254), [@elezar](https://github.com/elezar)) [SIG Node and Testing] 1533 - ACTION_REQUIRED 1534 When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. ([#118420](https://github.com/kubernetes/kubernetes/pull/118420), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps] 1535 - Added `ServedVersions` field to `StorageVersion` API. ([#118386](https://github.com/kubernetes/kubernetes/pull/118386), [@Richabanker](https://github.com/Richabanker)) 1536 - Added `IP mode` field to loadbalancer status ingress. ([#118895](https://github.com/kubernetes/kubernetes/pull/118895), [@RyanAoh](https://github.com/RyanAoh)) 1537 - Added `podReplacementPolicy` and terminating field to job api. ([#119301](https://github.com/kubernetes/kubernetes/pull/119301), [@kannon92](https://github.com/kannon92)) 1538 - Added a new `namespaceParamRef` field to `admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy`. ([#119215](https://github.com/kubernetes/kubernetes/pull/119215), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing] 1539 - Added a warning that TLS 1.3 ciphers are not configurable. ([#115399](https://github.com/kubernetes/kubernetes/pull/115399), [@3u13r](https://github.com/3u13r)) [SIG API Machinery and Node] 1540 - Added error handling for seccomp localhost configurations that do not properly set a `localhostProfile`. ([#117020](https://github.com/kubernetes/kubernetes/pull/117020), [@cji](https://github.com/cji)) 1541 - Added fields `reason` and `fieldPath` into CRD validation rules to allow users to specify reason and field path when validation failed. ([#118041](https://github.com/kubernetes/kubernetes/pull/118041), [@cici37](https://github.com/cici37)) [SIG API Machinery] 1542 - Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a `namespaceObject` 1543 variable with expressions. ([#118267](https://github.com/kubernetes/kubernetes/pull/118267), [@cici37](https://github.com/cici37)) [SIG API Machinery and Testing] 1544 - Added new `CRDValidationRatcheting` alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. ([#118990](https://github.com/kubernetes/kubernetes/pull/118990), [@alexzielenski](https://github.com/alexzielenski)) 1545 - Added new annotation `batch.kubernetes.io/cronjob-scheduled-timestamp` to Job objects scheduled from CronJobs. ([#118137](https://github.com/kubernetes/kubernetes/pull/118137), [@helayoty](https://github.com/helayoty)) 1546 - Added new config option `delayCacheUntilActive` to `KubeSchedulerConfiguration` that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in `kube-scheduler` ([#115754](https://github.com/kubernetes/kubernetes/pull/115754), [@linxiulei](https://github.com/linxiulei)) [SIG API Machinery and Scheduling] 1547 - Changed how KMS v2 encryption at rest can generate data encryption keys. 1548 When you enable the `KMSv2KDF` feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. ([#118828](https://github.com/kubernetes/kubernetes/pull/118828), [@enj](https://github.com/enj)) 1549 - Exposed `rest.DefaultServerUrlFor` function. ([#118055](https://github.com/kubernetes/kubernetes/pull/118055), [@timofurrer](https://github.com/timofurrer)) 1550 - Extended the Job API for alpha version of `BackoffLimitPerIndex`. ([#119294](https://github.com/kubernetes/kubernetes/pull/119294), [@mimowo](https://github.com/mimowo)) 1551 - Graduated `AdmissionWebhookMatchCondition` feature to beta. ([#119380](https://github.com/kubernetes/kubernetes/pull/119380), [@a-hilaly](https://github.com/a-hilaly)) 1552 - If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via `memory.oom.group` . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. ([#117793](https://github.com/kubernetes/kubernetes/pull/117793), [@tzneal](https://github.com/tzneal)) [SIG Apps, Node and Testing] 1553 - In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . ([#118782](https://github.com/kubernetes/kubernetes/pull/118782), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 1554 - Indexed Job pods now have the pod completion index set as a pod label. ([#118883](https://github.com/kubernetes/kubernetes/pull/118883), [@danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps] 1555 - Kube-proxy: added `--logging-format` flag to support structured logging. ([#117800](https://github.com/kubernetes/kubernetes/pull/117800), [@cyclinder](https://github.com/cyclinder)) 1556 - NodeVolumeLimits implement the `PreFilter` extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. ([#115398](https://github.com/kubernetes/kubernetes/pull/115398), [@tangwz](https://github.com/tangwz)) [SIG Scheduling] 1557 - PersistentVolumes have a new `LastPhaseTransitionTime` field which holds a timestamp of when the volume last transitioned its phase. ([#116469](https://github.com/kubernetes/kubernetes/pull/116469), [@RomanBednar](https://github.com/RomanBednar)) 1558 - Pods which set `hostNetwork: true` and declare ports, get the `hostPort` field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now `hostPort` will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. ([#117696](https://github.com/kubernetes/kubernetes/pull/117696), [@thockin](https://github.com/thockin)) [SIG Apps] 1559 - Promoted API groups `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` to `v1beta1`. ([#118644](https://github.com/kubernetes/kubernetes/pull/118644), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps and Testing] 1560 - Promoted the feature gate `ValidtaingAdmissionPolicy` to beta, and it is turned off by default. ([#119409](https://github.com/kubernetes/kubernetes/pull/119409), [@alexzielenski](https://github.com/alexzielenski)) 1561 - Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to `BETA` stability. ([#119264](https://github.com/kubernetes/kubernetes/pull/119264), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation] 1562 - Removed `resizeStatus` enum from `pvc.Status` and replaced with `AllocatedResourceStatus`. ([#116335](https://github.com/kubernetes/kubernetes/pull/116335), [@gnufied](https://github.com/gnufied)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing] 1563 - Removed `WindowsHostProcessContainers` feature-gate. ([#117570](https://github.com/kubernetes/kubernetes/pull/117570), [@marosset](https://github.com/marosset)) [SIG API Machinery, Apps, Auth, Node and Windows] 1564 - Revised the comment about the feature-gate level for `PodFailurePolicy` from alpha to beta. ([#117802](https://github.com/kubernetes/kubernetes/pull/117802), [@kerthcet](https://github.com/kerthcet)) [SIG API Machinery and Apps] 1565 - StatefulSet pods now have the pod index set as a pod label `statefulset.kubernetes.io/pod-index`. ([#119232](https://github.com/kubernetes/kubernetes/pull/119232), [@danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps] 1566 - Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver ([#117740](https://github.com/kubernetes/kubernetes/pull/117740), [@Richabanker](https://github.com/Richabanker)) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing] 1567 - Supported `BackoffLimitPerIndex` in Jobs. ([#118009](https://github.com/kubernetes/kubernetes/pull/118009), [@mimowo](https://github.com/mimowo)) 1568 - The `IPTablesOwnershipCleanup` feature (KEP-3178) is now GA; kubelet no longer 1569 creates the `KUBE-MARK-DROP` chain (which has been unused for several releases) 1570 or the `KUBE-MARK-MASQ` chain (which is now only created by kube-proxy). ([#119374](https://github.com/kubernetes/kubernetes/pull/119374), [@danwinship](https://github.com/danwinship)) 1571 - The `SelfSubjectReview` API is promoted to `authentication.k8s.io/v1` and the `kubectl auth whoami` command is GA. ([#117713](https://github.com/kubernetes/kubernetes/pull/117713), [@nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Architecture, Auth, CLI and Testing] 1572 - The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still `<pod>-<claim name>`, but a random suffix will avoid name collisions. ([#117351](https://github.com/kubernetes/kubernetes/pull/117351), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing] 1573 - The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. ([#116429](https://github.com/kubernetes/kubernetes/pull/116429), [@gjkim42](https://github.com/gjkim42)) [SIG API Machinery, Apps, Node, Scheduling and Testing] 1574 - Updated the comment about the feature-gate level for `PodFailurePolicy` from alpha to beta ([#118278](https://github.com/kubernetes/kubernetes/pull/118278), [@mimowo](https://github.com/mimowo)) 1575 - `client-go`: Improved memory use of reflector caches when watching large numbers 1576 of objects which do not change frequently. ([#113362](https://github.com/kubernetes/kubernetes/pull/113362), [@sxllwx](https://github.com/sxllwx)) 1577 - `component-base/logs` is now stricter about not applying configurations multiple 1578 times and will return an error when that is attempted. Can be overridden by binaries 1579 which need to do that. ([#117108](https://github.com/kubernetes/kubernetes/pull/117108), [@pohly](https://github.com/pohly)) 1580 - `kube-controller-manager`: The `LegacyServiceAccountTokenCleanUp` feature gate 1581 is now available as alpha (off by default). When enabled, the `legacy-service-account-token-cleaner` 1582 controller loop removes service account token secrets that have not been used 1583 in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting 1584 to one year), **and are** referenced from the `.secrets` list of a ServiceAccount 1585 object, **and are not** referenced from pods. ([#115554](https://github.com/kubernetes/kubernetes/pull/115554), [@yt2985](https://github.com/yt2985)) 1586 - `kube-scheduler` component config (KubeSchedulerConfiguration) `kubescheduler.config.k8s.io/v1beta2` 1587 is removed in `v1.28`. Migrate `kube-scheduler` configuration files to `kubescheduler.config.k8s.io/v1`. ([#117649](https://github.com/kubernetes/kubernetes/pull/117649), [@SataQiu](https://github.com/SataQiu)) 1588 1589 ### Feature 1590 1591 - A ValidatingAdmissionPolicy now has its `messageExpression` field checked against resolved types. ([#119209](https://github.com/kubernetes/kubernetes/pull/119209), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery] 1592 - Added '--concurrent-cron-job-syncs' flag for `kube-controller-manager` to set the number of workers for cron job controller. ([#117550](https://github.com/kubernetes/kubernetes/pull/117550), [@borgerli](https://github.com/borgerli)) 1593 - Added '--concurrent-job-syncs' flag for `kube-controller-manager` to set the number of job controller workers. ([#117138](https://github.com/kubernetes/kubernetes/pull/117138), [@tosi3k](https://github.com/tosi3k)) 1594 - Added `--concurrency` flag to configure the concurrency of `kubectl diff` execution, defaults to 1. ([#118810](https://github.com/kubernetes/kubernetes/pull/118810), [@brancz](https://github.com/brancz)) 1595 - Added `ConsistentListFromCache` feature gate that allows apiserver to serve consistent lists from cache. ([#118508](https://github.com/kubernetes/kubernetes/pull/118508), [@serathius](https://github.com/serathius)) 1596 - Added `DisruptionTarget` condition to the pod preempted by kubelet to make room for a critical pod. ([#117586](https://github.com/kubernetes/kubernetes/pull/117586), [@mimowo](https://github.com/mimowo)) 1597 - Added `apiserver_admission_match_condition_evaluation_seconds` and `apiserver_admission_match_condition_exclusions_total` metrics. ([#119311](https://github.com/kubernetes/kubernetes/pull/119311), [@ivelichkovich](https://github.com/ivelichkovich)) 1598 - Added a container image for `kubectl` at `registry.k8s.io/kubectl` across the same architectures as other images (linux/amd64 linux/arm64 linux/s390x linux/ppc64le) ([#116672](https://github.com/kubernetes/kubernetes/pull/116672), [@dims](https://github.com/dims)) [SIG Architecture and Release] 1599 - Added a new command line argument `--interactive` to kubectl. The new command line argument lets a user confirm deletion requests per resource interactively. ([#114530](https://github.com/kubernetes/kubernetes/pull/114530), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI and Testing] 1600 - Added a new feature gate, `SchedulerQueueingHints` (enabled by default). 1601 The new feature gate activates a framework for fine-grained filtering of events related to scheduler plugins. 1602 In this release, no default scheduling plugins make use of the hinting framework, so you should not expect any behavior changes. ([#119328](https://github.com/kubernetes/kubernetes/pull/119328), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 1603 - Added full cgroup v2 swap support for both `Limited` and `Unlimited` swap. 1604 1605 When `LimitedSwap` is enabled the swap limit would be automatically calculated for 1606 Burstable QoS pods. For Best-Effort/Guaranteed QoS pods, swap would be disabled. 1607 1608 Containers with memory requests equal to their memory limits also won't have 1609 swap access, and it is a way to opt-out of swap for a single container. 1610 1611 The formula for the swap limit for Burstable QoS pods is: 1612 `(<memory-request>/<node-memory-capacity>)*<node-swap-capacity>`. 1613 1614 Support for `cgroup v1` is removed. ([#118764](https://github.com/kubernetes/kubernetes/pull/118764), [@iholder101](https://github.com/iholder101)) 1615 - Added handling for pods in podgc for `PodReplacementPolicy` or `PodDisruption`. ([#118772](https://github.com/kubernetes/kubernetes/pull/118772), [@kannon92](https://github.com/kannon92)) 1616 - Added reason to metric `attachdetach_controller_forced_detaches` in the attach detach controller. ([#119185](https://github.com/kubernetes/kubernetes/pull/119185), [@xing-yang](https://github.com/xing-yang)) 1617 - Added support for pod `hostNetwork` field selector ([#110477](https://github.com/kubernetes/kubernetes/pull/110477), [@halfcrazy](https://github.com/halfcrazy)) [SIG Apps and Node] 1618 - Added swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). ([#118865](https://github.com/kubernetes/kubernetes/pull/118865), [@iholder101](https://github.com/iholder101)) 1619 - Added the implementation for `PodRecreationPolicy` to wait for the creation of pods once the existing ones are fully terminated. ([#117015](https://github.com/kubernetes/kubernetes/pull/117015), [@kannon92](https://github.com/kannon92)) 1620 - Allow to monitor client-go DNS resolver latencies via `rest_client_dns_resolution_duration_seconds` Prometheus metric. ([#115357](https://github.com/kubernetes/kubernetes/pull/115357), [@mfojtik](https://github.com/mfojtik)) 1621 - Apiserver adds two new metrics `etcd_requests_total` and `etcd_request_errors_total` that allow users to monitor requests to etcd storage, split by operation and resource type. ([#117222](https://github.com/kubernetes/kubernetes/pull/117222), [@iyear](https://github.com/iyear)) [SIG API Machinery] 1622 - Bumped `distroless-iptables` to 0.2.6 based on Go 1.20.6. ([#119365](https://github.com/kubernetes/kubernetes/pull/119365), [@xmudrii](https://github.com/xmudrii)) 1623 - Bumped metrics-server to `v0.6.3`. ([#117120](https://github.com/kubernetes/kubernetes/pull/117120), [@dgrisonnet](https://github.com/dgrisonnet)) 1624 - CEL authorizer checks no longer raise runtime errors. Calls to "check" will always return a decision object and the authorization error (if any) can be accessed within expressions using the new decision methods "errored" and "error". ([#118804](https://github.com/kubernetes/kubernetes/pull/118804), [@benluddy](https://github.com/benluddy)) [SIG API Machinery] 1625 - CRI: exposed commit memory bytes in container stats specific to Windows ([#119238](https://github.com/kubernetes/kubernetes/pull/119238), [@kiashok](https://github.com/kiashok)) 1626 - Client-go now exposes two new metrics to monitor the client-go logic that 1627 generate http.Transports for the clients. 1628 1629 - `rest_client_transport_cache_entries` is a gauge metric 1630 with the number of existing entries in the internal cache 1631 1632 - `rest_client_transport_create_calls_total` is a counter 1633 that increments each time a new transport is created, storing 1634 the result of the operation needed to generate it: hit, miss 1635 or uncacheable. ([#117295](https://github.com/kubernetes/kubernetes/pull/117295), [@aojea](https://github.com/aojea)) 1636 - Cloud controller manager's node controller now emits timing metrics for initial `Node` synchronization. These metrics measure the delay between the creation of a new `Node` and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's `--concurrent-node-syncs` flag. ([#119241](https://github.com/kubernetes/kubernetes/pull/119241), [@cartermckinnon](https://github.com/cartermckinnon)) [SIG Cloud Provider and Instrumentation] 1637 - Dynamic resource allocation: when a claim uses "wait for first consumer" allocation (the default), then it will now get deallocated after it was used by a pod. That ensures that the next pod isn't affected by previous scheduling decision and that resources are not kept allocated unless really needed. If keeping a claim allocated is desired, use "immediate allocation." ([#118936](https://github.com/kubernetes/kubernetes/pull/118936), [@pohly](https://github.com/pohly)) 1638 - Enabled use of pods with volumes and user namespaces. The feature gate was renamed from `UserNamespacesStatelessPodsSupport` to `UserNamespacesSupport`. ([#118691](https://github.com/kubernetes/kubernetes/pull/118691), [@giuseppe](https://github.com/giuseppe)) 1639 - External credential provider plugins will now have their standard error output logged by kubelet upon failures. ([#117448](https://github.com/kubernetes/kubernetes/pull/117448), [@cartermckinnon](https://github.com/cartermckinnon)) 1640 - Faster scheduling when `ResourceClaims` are involved. ([#119078](https://github.com/kubernetes/kubernetes/pull/119078), [@pohly](https://github.com/pohly)) 1641 - Fixed the alpha `CloudDualStackNodeIPs` feature. ([#118329](https://github.com/kubernetes/kubernetes/pull/118329), [@danwinship](https://github.com/danwinship)) 1642 - Graduated the `LegacyServiceAccountTokenTracking` feature gate to GA. The usage of auto-generated secret-based service account token now produces warnings, and relevant Secrets are labeled with a last-used timestamp (label key `kubernetes.io/legacy-token-last-used`). ([#117591](https://github.com/kubernetes/kubernetes/pull/117591), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth and Testing] 1643 - Graduated the `ProbeTerminationGracePeriod` feature gate to GA. ([#114307](https://github.com/kubernetes/kubernetes/pull/114307), [@rphillips](https://github.com/rphillips)) 1644 - Hashing of KeyID in Logs 1645 1646 This release adds a feature to hash the `KeyID` values in the logs. The `KeyID` values are sensitive information that should not be exposed in plain text in the logs. By hashing the `KeyID` values, we can protect the confidentiality of the data while still being able to log the necessary information. ([#118988](https://github.com/kubernetes/kubernetes/pull/118988), [@nilekhc](https://github.com/nilekhc)) [SIG API Machinery, Auth and Testing] 1647 - Implemented alpha support for a drop-in kubelet configuration directory. ([#119390](https://github.com/kubernetes/kubernetes/pull/119390), [@sohankunkerkar](https://github.com/sohankunkerkar)) 1648 - In the course of admitting a single request, the ValidatingAdmissionPolicy plugin will perform no more than one authorization check per unique authorizer expression. All evaluations of identical authorizer expressions will produce the same decision. ([#116443](https://github.com/kubernetes/kubernetes/pull/116443), [@benluddy](https://github.com/benluddy)) [SIG API Machinery and Testing] 1649 - Introduce support for CEL optionals (see [CEL spec proposal 246](https://github.com/google/cel-spec/wiki/proposal-246)). 1650 This feature will not be fully enabled until a future Kubernetes release (likely to be v1.29), but is added in v1.28 to enable 1651 safe rollback on downgrade. ([#118339](https://github.com/kubernetes/kubernetes/pull/118339), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth, Cloud Provider and Testing] 1652 - Kube-controller-manager: the dynamic resource controller steps in when a pod got created such that the scheduler ignores it (i.e. spec.nodeName is set) and then takes care of triggering delayed resource claim allocation and/or reserving a claim for the pod. ([#118209](https://github.com/kubernetes/kubernetes/pull/118209), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node and Testing] 1653 - Kube-proxy handles Terminating EndpointSlices conditions and enables zero downtime deployments for Services with ExternalTrafficPolicy=Local author: @andrewsykim ([#117718](https://github.com/kubernetes/kubernetes/pull/117718), [@aojea](https://github.com/aojea)) [SIG Network, Testing and Windows] 1654 - Kube-proxy service health returns http header `X-Load-Balancing-Endpoint-Weight` with number of local endpoints. The same information is still available in response body JSON `payload.LocalEndpoints`. ([#118999](https://github.com/kubernetes/kubernetes/pull/118999), [@cezarygerard](https://github.com/cezarygerard)) 1655 - Kubelet: plugins for dynamic resource allocation may use the `v1alpha3` API instead of v1alpha2 if they want to do prepare/unprepare operations in batches. ([#119012](https://github.com/kubernetes/kubernetes/pull/119012), [@pohly](https://github.com/pohly)) 1656 - Kubelet: security of dynamic resource allocation was enhanced by limiting node access to those objects that are needed on the node. ([#116254](https://github.com/kubernetes/kubernetes/pull/116254), [@pohly](https://github.com/pohly)) [SIG Auth and Testing] 1657 - Kubelet: un-deprecated `--provider-id` flag. ([#116530](https://github.com/kubernetes/kubernetes/pull/116530), [@pacoxu](https://github.com/pacoxu)) 1658 - Kubernetes is now built with Go `1.20.4`. ([#117744](https://github.com/kubernetes/kubernetes/pull/117744), [@xmudrii](https://github.com/xmudrii)) [SIG Release and Testing] 1659 - Kubernetes is now built with Go `1.20.5`. ([#118507](https://github.com/kubernetes/kubernetes/pull/118507), [@jeremyrickard](https://github.com/jeremyrickard)) 1660 - Kubernetes is now built with Go `1.20.6`. ([#119324](https://github.com/kubernetes/kubernetes/pull/119324), [@xmudrii](https://github.com/xmudrii)) 1661 - Metric `scheduler_scheduler_goroutines` is removed. Use `scheduler_goroutines` instead. ([#117727](https://github.com/kubernetes/kubernetes/pull/117727), [@kerthcet](https://github.com/kerthcet)) [SIG Scheduling] 1662 - Migrated `pkg/controller/endpoint` to contextual logging. ([#116755](https://github.com/kubernetes/kubernetes/pull/116755), [@my-git9](https://github.com/my-git9)) 1663 - Migrated `pkg/scheduler/framework/preemption` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116835](https://github.com/kubernetes/kubernetes/pull/116835), [@mengjiao-liu](https://github.com/mengjiao-liu)) 1664 - Migrated `pod-security-admission` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#114471](https://github.com/kubernetes/kubernetes/pull/114471), [@Namanl2001](https://github.com/Namanl2001)) [SIG Apps and Auth] 1665 - Migrated controller functions to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116930](https://github.com/kubernetes/kubernetes/pull/116930), [@fatsheep9146](https://github.com/fatsheep9146)) [SIG API Machinery, Apps, Network, Node, Storage and Testing] 1666 - Migrated the Job controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116910](https://github.com/kubernetes/kubernetes/pull/116910), [@fatsheep9146](https://github.com/fatsheep9146)) [SIG API Machinery, Apps and Testing] 1667 - Migrated the `EndpointSlice` and `EndpointSliceMirroring` controllers (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#115295](https://github.com/kubernetes/kubernetes/pull/115295), [@Namanl2001](https://github.com/Namanl2001)) [SIG API Machinery, Apps, Network and Testing] 1668 - Migrated the certificate controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#113994](https://github.com/kubernetes/kubernetes/pull/113994), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG API Machinery, Apps, Auth, Instrumentation and Testing] 1669 - Migrated the noderesources scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116748](https://github.com/kubernetes/kubernetes/pull/116748), [@mengjiao-liu](https://github.com/mengjiao-liu)) 1670 - Migrated the podtopologyspread scheduler plugins to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116797](https://github.com/kubernetes/kubernetes/pull/116797), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 1671 - Moved `non-graceful node` shutdown to GA. ([#118228](https://github.com/kubernetes/kubernetes/pull/118228), [@carlory](https://github.com/carlory)) 1672 - New CEL Library functions to support Kubernetes Quantities. ([#118803](https://github.com/kubernetes/kubernetes/pull/118803), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery] 1673 - New Metrics Added for Encryption Configuration Controller 1674 1675 This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include: 1676 1677 - `apiserver_encryption_config_controller_automatic_reload_failures_total`: Total number of failed automatic reloads of encryption configuration. 1678 - `apiserver_encryption_config_controller_automatic_reload_success_total`: Total number of successful automatic reloads of encryption configuration. 1679 - `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds`: Timestamp of the last successful or failed automatic reload of encryption configuration. 1680 1681 These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. ([#119008](https://github.com/kubernetes/kubernetes/pull/119008), [@nilekhc](https://github.com/nilekhc)) 1682 - New staging repo has been created for the `EndpointSlice` reconciler. ([#118953](https://github.com/kubernetes/kubernetes/pull/118953), [@mskrocki](https://github.com/mskrocki)) 1683 - Promoted `ServiceNodePortStaticSubrange` feature gate to beta, and it will be enabled by default. ([#117877](https://github.com/kubernetes/kubernetes/pull/117877), [@xuzhenglun](https://github.com/xuzhenglun)) 1684 - Promoted the following apiserver flowcontrol metrics to Beta: 1685 1686 - apiserver_flowcontrol_request_wait_duration_seconds 1687 - apiserver_flowcontrol_current_executing_seats 1688 - apiserver_flowcontrol_nominal_limit_seats 1689 - apiserver_flowcontrol_rejected_requests_total 1690 - apiserver_flowcontrol_dispatched_requests_total 1691 - apiserver_flowcontrol_current_inqueue_requests 1692 - apiserver_flowcontrol_current_executing_requests ([#119110](https://github.com/kubernetes/kubernetes/pull/119110), [@andrewsykim](https://github.com/andrewsykim)) 1693 - Renamed `PodHasNetwork` to `PodReadyToStartContainers`. ([#117702](https://github.com/kubernetes/kubernetes/pull/117702), [@kannon92](https://github.com/kannon92)) [SIG Node and Testing] 1694 - Replaced `apiserver_storage_db_total_size_in_bytes` with `apiserver_storage_size_bytes` metric. ([#118812](https://github.com/kubernetes/kubernetes/pull/118812), [@serathius](https://github.com/serathius)) 1695 - Scheduler now waits for handlers to finish syncing before the scheduling cycles start. ([#116729](https://github.com/kubernetes/kubernetes/pull/116729), [@AxeZhan](https://github.com/AxeZhan)) 1696 - Set metrics-server's metric-resolution to 15s. ([#117121](https://github.com/kubernetes/kubernetes/pull/117121), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG Cloud Provider and Instrumentation] 1697 - SubjectAccessReview requests sent to webhook authorizers now default `spec.resourceAttributes.version` to `*` if unset. ([#116937](https://github.com/kubernetes/kubernetes/pull/116937), [@AxeZhan](https://github.com/AxeZhan)) [SIG Apps and Auth] 1698 - Supported specifying a custom retry period for cloud `load-balancer` operations. ([#94021](https://github.com/kubernetes/kubernetes/pull/94021), [@timoreimann](https://github.com/timoreimann)) 1699 - The "value" part in the `wait --for=jsonpath='{expression}'[=value]` is now 1700 optional. If the value is not provided i.e., the command looks like `wait --for=jsonpath='{expression}'` 1701 then the wait condition is interpreted as matched when the expression returns 1702 *any* single JSON value like object or a literal. ([#118160](https://github.com/kubernetes/kubernetes/pull/118160), [@minherz](https://github.com/minherz)) 1703 - The Kubernetes apiserver now emits a warning message for Pods with a null labelSelector in podAffinity or topologySpreadConstraints. The null labelSelector means "match none". Using it in podAffinity or topologySpreadConstraint could lead to unintended behavior. ([#117025](https://github.com/kubernetes/kubernetes/pull/117025), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 1704 - The `AdvancedAuditing` feature gate that graduated to GA in `v1.12` (and was unconditionally 1705 enabled) has been removed. ([#118763](https://github.com/kubernetes/kubernetes/pull/118763), [@Shubham82](https://github.com/Shubham82)) 1706 - The `ExpandedDNSConfig` feature has graduated to GA. 'ExpandedDNSConfig' feature was locked to default value and will be removed in v1.30. If you were setting this feature gate explicitly, please remove it now. ([#116741](https://github.com/kubernetes/kubernetes/pull/116741), [@gjkim42](https://github.com/gjkim42)) [SIG Apps, Network and Node] 1707 - The apiserver debug endpoint `/debug/api_priority_and_fairness/dump_requests` has been extended to dump executing requests as well as queued ones. A column for StartTime has been added to the returned table, with the queued requests having a StartTime of "0001-01-01T00:00:00Z". The executing requests have a RequestIndexInQueue of -1, and the QueueIndex is also -1 for priority levels without queues. ([#119009](https://github.com/kubernetes/kubernetes/pull/119009), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 1708 - The helping message of commands which have sub-commands is now clearer and more instructive. It will show the full command instead of `kubectl <command> --help ...` 1709 1710 Changed `kubectl create secret --help` description. There will be a short introduction to the three secret types and clearer guidance on how to use the command. ([#117930](https://github.com/kubernetes/kubernetes/pull/117930), [@LronDC](https://github.com/LronDC)) 1711 - The scheduler skips the `InterPodAffinity` Score plugin when nothing to do with the Pod. 1712 It will affect some metrics values related to the InterPodAffinity Score plugin. ([#117794](https://github.com/kubernetes/kubernetes/pull/117794), [@utam0k](https://github.com/utam0k)) [SIG Scheduling] 1713 - The scheduler skips the `PodTopologySpread` Filter plugin if no spread constraints. 1714 It will affect some metrics values related to the `PodTopologySpread` Filter plugin. ([#117683](https://github.com/kubernetes/kubernetes/pull/117683), [@utam0k](https://github.com/utam0k)) 1715 - The scheduler skips the `PodTopologySpread` Score plugin when nothing to do with the Pod. 1716 It will affect some metrics values related to the PodTopologySpread Score plugin. ([#118608](https://github.com/kubernetes/kubernetes/pull/118608), [@utam0k](https://github.com/utam0k)) 1717 - The short names `vwc` and `mwc` were introduced for the resources `validatingwebhookconfigurations` and `mutatingwebhookconfigurations`. ([#117535](https://github.com/kubernetes/kubernetes/pull/117535), [@hysyeah](https://github.com/hysyeah)) 1718 - Updated etcd image to `3.5.9-0`. ([#117999](https://github.com/kubernetes/kubernetes/pull/117999), [@kkkkun](https://github.com/kkkkun)) [SIG API Machinery] 1719 - Updated cAdvisor to `v0.47.2` and fixed metrics in `cri-o` when a container restarts. ([#118774](https://github.com/kubernetes/kubernetes/pull/118774), [@harche](https://github.com/harche)) 1720 - Updated distroless I-tables to use registry.k8s.io/build-image/distroless-iptables:v0.2.5 ([#118541](https://github.com/kubernetes/kubernetes/pull/118541), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG Testing] 1721 - Updated distroless iptables to use released image `registry.k8s.io/build-image/distroless-iptables:v0.2.4` ([#117746](https://github.com/kubernetes/kubernetes/pull/117746), [@xmudrii](https://github.com/xmudrii)) [SIG Testing] 1722 - Updated the scheduler interface and cache methods to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116849](https://github.com/kubernetes/kubernetes/pull/116849), [@mengjiao-liu](https://github.com/mengjiao-liu)) 1723 - ValidatingAdmissionPolicy type checking now correctly handles `authorizer` variable. ([#118540](https://github.com/kubernetes/kubernetes/pull/118540), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery] 1724 - When a pod is done or not going to run, then `ResourceClaims` for it can be reused by other pods or deleted. ([#118817](https://github.com/kubernetes/kubernetes/pull/118817), [@pohly](https://github.com/pohly)) 1725 - With the `KubeletCgroupDriverFromCRI` feature gate enabled and sufficiently new version of a container 1726 runtime, kubelet automatically detects the cgroup driver config from the container runtime, eliminating 1727 the need to specify the `cgroupDriver` configuration option (or `--cgroup-driver` flag) of kubelet. ([#118770](https://github.com/kubernetes/kubernetes/pull/118770), [@marquiz](https://github.com/marquiz)) 1728 - [Kube-proxy]: Implemented connection draining for terminating nodes. ([#116470](https://github.com/kubernetes/kubernetes/pull/116470), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) 1729 - `--version=v1.X.Y...` can now be used to set the prerelease and buildID portions of the version reported by components ([#117688](https://github.com/kubernetes/kubernetes/pull/117688), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Architecture and Release] 1730 - `RetroactiveDefaultStorageClass` feature made stable and enabled by default. ([#118102](https://github.com/kubernetes/kubernetes/pull/118102), [@RomanBednar](https://github.com/RomanBednar)) 1731 - `TopologyManagerPolicyOptions` feature-flag is promoted to beta and enabled by default. ([#118816](https://github.com/kubernetes/kubernetes/pull/118816), [@PiotrProkop](https://github.com/PiotrProkop)) 1732 - `force_delete_pods_total` and `force_delete_pod_errors_total` metrics count all pod deletion behaviors. ([#118480](https://github.com/kubernetes/kubernetes/pull/118480), [@carlory](https://github.com/carlory)) 1733 - `klog` text output now uses JSON as encoding for structs, maps and slices. ([#117687](https://github.com/kubernetes/kubernetes/pull/117687), [@pohly](https://github.com/pohly)) 1734 - `kube-proxy` in iptables mode will now have separate `sync_full_proxy_rules_duration_seconds`\nand 1735 `sync_partial_proxy_rules_duration_seconds` (in addition to the existing\n`sync_proxy_rules_duration_seconds`), 1736 giving better information about the duration of each \nsync type, rather than 1737 only giving a weighted average of the two sync types together. ([#117787](https://github.com/kubernetes/kubernetes/pull/117787), [@danwinship](https://github.com/danwinship)) 1738 - `kubeadm`: added a new "kubeadm config validate" command that can be used to 1739 validate any input config file. Use the `--config` flag to pass a config file 1740 to it. See the command `--help` screen for more information. As a result of adding 1741 this new command, enhance the validation capabilities of the existing "kubeadm 1742 config migrate" command. For both commands unknown APIs or fields will throw errors. ([#118013](https://github.com/kubernetes/kubernetes/pull/118013), [@neolit123](https://github.com/neolit123)) 1743 - `kubeadm`: added the `--allow-experimental-api` flag to "kubeadm config migrate/validate" commands. It can be used to migrate or validate WIP/experimental APIs in the future. ([#118866](https://github.com/kubernetes/kubernetes/pull/118866), [@neolit123](https://github.com/neolit123)) 1744 - `kubeadm`: generate CA certificates with a start time that is offset 5 1745 minutes in the past relative to the current system time to workaround cases of 1746 clock desync. ([#118922](https://github.com/kubernetes/kubernetes/pull/118922), [@champtar](https://github.com/champtar)) 1747 - `plugin_evaluation_total` metric supports prescore/score extension point. 1748 The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming pod. ([#118025](https://github.com/kubernetes/kubernetes/pull/118025), [@AxeZhan](https://github.com/AxeZhan)) 1749 1750 ### Documentation 1751 1752 - Enhanced clarity in error messaging when waiting for volume creation ([#118262](https://github.com/kubernetes/kubernetes/pull/118262), [@torredil](https://github.com/torredil)) [SIG Apps and Storage] 1753 1754 ### Failing Test 1755 1756 - Allowed Azure Disk e2es to use newer topology labels if available from nodes. ([#117216](https://github.com/kubernetes/kubernetes/pull/117216), [@gnufied](https://github.com/gnufied)) 1757 - Fixed nil pointer in test AfterEach volumeperf.go for sidecar release. ([#117368](https://github.com/kubernetes/kubernetes/pull/117368), [@sunnylovestiramisu](https://github.com/sunnylovestiramisu)) 1758 - Switched back to `debian-base` instead of distroless for conformance image. ([#119422](https://github.com/kubernetes/kubernetes/pull/119422), [@saschagrunert](https://github.com/saschagrunert)) 1759 1760 ### Bug or Regression 1761 1762 - Added a new event `FailedToRetrieveImagePullSecret` which will be generated when a pod references an `ImagePullSecret` that doesn't exist. ([#117927](https://github.com/kubernetes/kubernetes/pull/117927), [@kaisoz](https://github.com/kaisoz)) [SIG Node] 1763 - Added additional validation for endpoint IP configuration while iterating through queried endpoint list. ([#116749](https://github.com/kubernetes/kubernetes/pull/116749), [@princepereira](https://github.com/princepereira)) 1764 - Added warning for dup ports update/patching in pod's container ports and service ports. ([#113245](https://github.com/kubernetes/kubernetes/pull/113245), [@pacoxu](https://github.com/pacoxu)) 1765 - As in Kubernetes `v1.26` and `v1.27`, resource claims do not get prepared by `kubelet` when no container uses them. This was changed accidentally in [v1.28.0-alpha.1](https://github.com/kubernetes/kubernetes/releases/tag/v1.28.0-alpha.1). ([#118786](https://github.com/kubernetes/kubernetes/pull/118786), [@pohly](https://github.com/pohly)) 1766 - Bumped cadvisor version to `v0.47.3`. ([#119225](https://github.com/kubernetes/kubernetes/pull/119225), [@iholder101](https://github.com/iholder101)) 1767 - CI job `ci-kubernetes-node-arm64-ubuntu-serial` will test node e2e on arm64, `use-dockerized-build` and `target-build-arch` are required to run this job. ([#118567](https://github.com/kubernetes/kubernetes/pull/118567), [@chendave](https://github.com/chendave)) 1768 - CVE-2023-27561 CVE-2023-25809 CVE-2023-28642: Bump fix runc v1.1.4 -> v1.1.5 ([#117095](https://github.com/kubernetes/kubernetes/pull/117095), [@PushkarJ](https://github.com/PushkarJ)) [SIG Architecture, Node and Security] 1769 - Code blocks in `kubectl {$COMMAND}--help` will move right by 3 indentation. ([#118029](https://github.com/kubernetes/kubernetes/pull/118029), [@ardaguclu](https://github.com/ardaguclu)) 1770 - Compute the backoff delay more accurately for deleted pods ([#118413](https://github.com/kubernetes/kubernetes/pull/118413), [@mimowo](https://github.com/mimowo)) [SIG Apps] 1771 - Declare Job as finished only after removing all Pod finalizers to avoid orphan Pods. ([#119159](https://github.com/kubernetes/kubernetes/pull/119159), [@alculquicondor](https://github.com/alculquicondor)) 1772 - During device plugin allocation, resources requested by the pod can only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated. If these conditions are not sattsfied, the pod would fail with `UnexpectedAdmissionError` error. ([#116376](https://github.com/kubernetes/kubernetes/pull/116376), [@swatisehgal](https://github.com/swatisehgal)) [SIG Node and Testing] 1773 - Dynamic Resource Allocation: logged an error and submitted an event when `Kubelet` failed to prepare dynamic resources. ([#118578](https://github.com/kubernetes/kubernetes/pull/118578), [@bart0sh](https://github.com/bart0sh)) 1774 - Ensure Job status updates are batched by 1s. This fixes an unlikely scenario when a sequence of immediately 1775 completing pods could trigger a sequence of non-batched Job status updates. ([#118470](https://github.com/kubernetes/kubernetes/pull/118470), [@mimowo](https://github.com/mimowo)) [SIG Apps] 1776 - Faster `StatefulSet` creation when `Parallel` mode is enabled. ([#117865](https://github.com/kubernetes/kubernetes/pull/117865), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) 1777 - Fixed a data race in TopologyCache when `AddHints` and `SetNodes` are called concurrently. ([#117249](https://github.com/kubernetes/kubernetes/pull/117249), [@tnqn](https://github.com/tnqn)) [SIG Apps and Network] 1778 - Fixed a race condition in `kube-proxy` when using LocalModeNodeCIDR, to avoid dropping Services traffic if the object node is recreated when `kube-proxy` is starting. ([#118499](https://github.com/kubernetes/kubernetes/pull/118499), [@aojea](https://github.com/aojea)) 1779 - Fixed bug where `listOfStrings.join()` in CEL expressions resulted in an unexpected internal error. ([#117593](https://github.com/kubernetes/kubernetes/pull/117593), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery] 1780 - Fixed incorrect calculation for ResourceQuota with PriorityClass as its scope. ([#117677](https://github.com/kubernetes/kubernetes/pull/117677), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG API Machinery] 1781 - Fix: After a Node is down and take some time to get back to up again, the mount point of the evicted Pods cannot be cleaned up successfully. (#111933) Meanwhile Kubelet will print the log `Orphaned pod "xxx" found, but error not a directory occurred when trying to remove the volumes dir` every 2 seconds. (#105536) ([#116134](https://github.com/kubernetes/kubernetes/pull/116134), [@cvvz](https://github.com/cvvz)) [SIG Node and Storage] 1782 - Fix: The volume is not detached after the pod and PVC objects are deleted. ([#116138](https://github.com/kubernetes/kubernetes/pull/116138), [@cvvz](https://github.com/cvvz)) [SIG Storage] 1783 - Fixed Cronjob `status.lastSuccessfulTime` not populated by a manually triggered ([#118530](https://github.com/kubernetes/kubernetes/pull/118530), [@carlory](https://github.com/carlory)) 1784 - Fixed Topology Aware Hints not working when the `topology.kubernetes.io/zone` label is added after Node creation. ([#117245](https://github.com/kubernetes/kubernetes/pull/117245), [@tnqn](https://github.com/tnqn)) 1785 - Fixed `creationTimestamp: null` causing unnecessary writes to etcd. ([#116865](https://github.com/kubernetes/kubernetes/pull/116865), [@alexzielenski](https://github.com/alexzielenski)) 1786 - Fixed `vSphere` cloud provider not to skip detach volumes from nodes at `kube-controller-startup`. ([#117243](https://github.com/kubernetes/kubernetes/pull/117243), [@jsafrane](https://github.com/jsafrane)) 1787 - Fixed a bug at `kube-apiserver` start where `APIService` objects for custom resources could be deleted and recreated. ([#118104](https://github.com/kubernetes/kubernetes/pull/118104), [@liggitt](https://github.com/liggitt)) 1788 - Fixed a bug that unintentionally overrides custom Accept headers in http (live-/readiness)-probes if the header is in lowercase. ([#114606](https://github.com/kubernetes/kubernetes/pull/114606), [@tuunit](https://github.com/tuunit)) 1789 - Fixed a bug where `kubectl port-forward`, when used with a Deployment, could connect to a terminating pod even when a running pod is also available. ([#119256](https://github.com/kubernetes/kubernetes/pull/119256), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] 1790 - Fixed a bug where pv recycler failed to scrub volume with too many files in the directory due to hitting ARG_MAX limit with rm command (#117189). ([#117283](https://github.com/kubernetes/kubernetes/pull/117283), [@defo89](https://github.com/defo89)) [SIG Cloud Provider and Storage] 1791 - Fixed a memory leak in the Kubernetes API server that occurs during APIService processing. ([#117258](https://github.com/kubernetes/kubernetes/pull/117258), [@enj](https://github.com/enj)) [SIG API Machinery] 1792 - Fixed a race condition between `Run()` and `SetTransform()` and `SetWatchErrorHandler()` in shared informers. ([#117870](https://github.com/kubernetes/kubernetes/pull/117870), [@howardjohn](https://github.com/howardjohn)) [SIG API Machinery] 1793 - Fixed a race condition serving `OpenAPI` content ([#117705](https://github.com/kubernetes/kubernetes/pull/117705), [@Jefftree](https://github.com/Jefftree)) 1794 - Fixed a regression in `1.27.0` that resulted in `missing metadata in converted object` errors when modifying objects for multi-version custom resource definitions with a conversion strategy of `None`. ([#117301](https://github.com/kubernetes/kubernetes/pull/117301), [@ncdc](https://github.com/ncdc)) 1795 - Fixed a regression in `kubectl` and `client-go` discovery when configured with a server URL other than the root of a server ([#117495](https://github.com/kubernetes/kubernetes/pull/117495), [@ardaguclu](https://github.com/ardaguclu)) 1796 - Fixed an issue where the API server did not send impersonated UID to authentication webhooks. ([#116681](https://github.com/kubernetes/kubernetes/pull/116681), [@stlaz](https://github.com/stlaz)) [SIG API Machinery and Auth] 1797 - Fixed bug that caused a resource to include patch directives when using strategic merge patch against a non-existent field. ([#117568](https://github.com/kubernetes/kubernetes/pull/117568), [@alexzielenski](https://github.com/alexzielenski)) 1798 - Fixed bug to correctly report `ErrRegistryUnavailable` on pulling container images for remote CRI runtimes. ([#117612](https://github.com/kubernetes/kubernetes/pull/117612), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node] 1799 - Fixed bug where explain was not properly respecting jsonpaths. ([#115694](https://github.com/kubernetes/kubernetes/pull/115694), [@mpuckett159](https://github.com/mpuckett159)) 1800 - Fixed bug where using the $deleteFromPrimitiveList directive in a strategic merge patch of certain fields would remove the other values from the list instead of the values specified. ([#110472](https://github.com/kubernetes/kubernetes/pull/110472), [@brianpursley](https://github.com/brianpursley)) [SIG API Machinery] 1801 - Fixed component status calling etcd health endpoint over http which exposed kubernetes to the risk of complete watch starvation and is inconsistent with other etcd probing done by `kube-apiserver`. ([#118460](https://github.com/kubernetes/kubernetes/pull/118460), [@serathius](https://github.com/serathius)) 1802 - Fixed computing backoff delay when using Job pod failure policy, by including in the backoff delay calculation pod failures ignored from the backoffLimit counter. ([#119434](https://github.com/kubernetes/kubernetes/pull/119434), [@mimowo](https://github.com/mimowo)) 1803 - Fixed cronjob controller handling of complex schedules, like `30 6-16/4 * * 1-5`, for example. ([#118724](https://github.com/kubernetes/kubernetes/pull/118724), [@soltysh](https://github.com/soltysh)) 1804 - Fixed deletion of non-admissible pods that are deleted during Kubelet restart. ([#118497](https://github.com/kubernetes/kubernetes/pull/118497), [@mimowo](https://github.com/mimowo)) 1805 - Fixed issue where `kubectl-convert` would fail when encountering resources that could not be converted to the specified api version. New behavior is to warn the user of the failed conversions and continue to convert the remaining resources. ([#117002](https://github.com/kubernetes/kubernetes/pull/117002), [@gxwilkerson33](https://github.com/gxwilkerson33)) 1806 - Fixed issue where there was no response or error from kubectl rollout status when there were no resources of specified kind. ([#117884](https://github.com/kubernetes/kubernetes/pull/117884), [@gxwilkerson33](https://github.com/gxwilkerson33)) [SIG CLI] 1807 - Fixed kubelet startup getting stuck with `NewVolumeManagerReconstruction` feature enabled and a CSI volume present in /var/lib/kubelet/pods. ([#117804](https://github.com/kubernetes/kubernetes/pull/117804), [@jsafrane](https://github.com/jsafrane)) [SIG Node and Storage] 1808 - Fixed performance regression in scheduler caused by frequent metric lookup on critical code path. ([#117594](https://github.com/kubernetes/kubernetes/pull/117594), [@tosi3k](https://github.com/tosi3k)) 1809 - Fixed restricted debug profile. ([#117543](https://github.com/kubernetes/kubernetes/pull/117543), [@mochizuki875](https://github.com/mochizuki875)) 1810 - Fixed the `preStop` hook. This will now block the pod termination grace period. ([#115835](https://github.com/kubernetes/kubernetes/pull/115835), [@HirazawaUi](https://github.com/HirazawaUi)) 1811 - Fixed the discoverability of `apiregistration.k8s.io` in `openapi/v3` ([#118879](https://github.com/kubernetes/kubernetes/pull/118879), [@atiratree](https://github.com/atiratree)) 1812 - If `kubeadm reset` finds no etcd member ID for the peer it removes during the `remove-etcd-member` phase, it continues immediately to other phases, instead of retrying the phase for up to 3 minutes before continuing. ([#117724](https://github.com/kubernetes/kubernetes/pull/117724), [@dlipovetsky](https://github.com/dlipovetsky)) [SIG Cluster Lifecycle] 1813 - Improved exponential backoff in Reflector, significantly reducing the load on Kubernetes apiserver in case of throttling of requests. ([#118132](https://github.com/kubernetes/kubernetes/pull/118132), [@marseel](https://github.com/marseel)) [SIG API Machinery and Scalability] 1814 - Known issue: fixed that the PreEnqueue plugins aren't executed for Pods proceeding to activeQ through backoffQ. ([#117194](https://github.com/kubernetes/kubernetes/pull/117194), [@sanposhiho](https://github.com/sanposhiho)) [SIG Release and Scheduling] 1815 - Kubeadm: the limitation that the `ignorePreflightErrors` field can not be set to `all` in kubeadm config file has been removed. ([#119351](https://github.com/kubernetes/kubernetes/pull/119351), [@SataQiu](https://github.com/SataQiu)) 1816 - Kubelet terminates pods correctly upon restart, fixing an issue where pods may have not been fully terminated if the kubelet was restarted during pod termination. ([#117019](https://github.com/kubernetes/kubernetes/pull/117019), [@bobbypage](https://github.com/bobbypage)) [SIG Node and Testing] 1817 - Kubelet will now skip pod resource checks when the request is zero. ([#116408](https://github.com/kubernetes/kubernetes/pull/116408), [@ChenLingPeng](https://github.com/ChenLingPeng)) 1818 - Number of errors reported to the metric `storage_operation_duration_seconds_count` for emptyDir decreased significantly because previously one error was reported for each projected volume created. ([#117022](https://github.com/kubernetes/kubernetes/pull/117022), [@mpatlasov](https://github.com/mpatlasov)) [SIG Storage] 1819 - Pod termination will be faster when the pod has a missing volume reference. ([#117412](https://github.com/kubernetes/kubernetes/pull/117412), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node and Testing] 1820 - Recording timing traces had a race condition. Impact in practice was probably low. ([#117139](https://github.com/kubernetes/kubernetes/pull/117139), [@pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 1821 - Reduced CPU and memory consumption of `kube-apiserver` if OpenAPI V2 will not be accessed by any client. Also improved performance of the apiserver on installation of many CRDs. ([#118212](https://github.com/kubernetes/kubernetes/pull/118212), [@Jefftree](https://github.com/Jefftree)) 1822 - Removed leading zeros from the etcd member ID in kubeadm log messages. ([#117919](https://github.com/kubernetes/kubernetes/pull/117919), [@dlipovetsky](https://github.com/dlipovetsky)) [SIG Cluster Lifecycle] 1823 - Resolves a spurious "Unknown discovery response content-type" error in client-go discovery requests by tolerating extra content-type parameters in API responses ([#117571](https://github.com/kubernetes/kubernetes/pull/117571), [@seans3](https://github.com/seans3)) [SIG API Machinery] 1824 - Reverted NewVolumeManagerReconstruction and SELinuxMountReadWriteOncePod feature gates to disabled by default to resolve a regression of volume reconstruction on kubelet/node restart ([#117751](https://github.com/kubernetes/kubernetes/pull/117751), [@liggitt](https://github.com/liggitt)) [SIG Storage] 1825 - Setting a mirror pod's phase to Succeeded or Failed can prevent the corresponding static pod from restarting due mutation of a Kubelet cache. ([#116482](https://github.com/kubernetes/kubernetes/pull/116482), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node] 1826 - Show a warning when `volume.beta.kubernetes.io/storage-class` annotation is used in pv or pvc ([#117036](https://github.com/kubernetes/kubernetes/pull/117036), [@haoruan](https://github.com/haoruan)) [SIG Storage] 1827 - Static pods were taking extra time to be restarted after being updated. Static pods that are waiting to restart were not correctly counted in `kubelet_working_pods`. ([#116995](https://github.com/kubernetes/kubernetes/pull/116995), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node] 1828 - The Daemonset controller creates replacements for terminal Pods, which can appear during VM preemptions or when using Pod finalizers. ([#118716](https://github.com/kubernetes/kubernetes/pull/118716), [@alculquicondor](https://github.com/alculquicondor)) 1829 - The `pod_scheduling_duration_seconds` metrics won't consider the time when a pod fails `PreEnqueue` (like being gated). ([#118049](https://github.com/kubernetes/kubernetes/pull/118049), [@helayoty](https://github.com/helayoty)) 1830 - The kube-proxy `sync_proxy_rules_iptables_total` metric has now reverted back 1831 to its pre-1.27 behavior of tracking the total number of iptables rules that 1832 kube-proxy is responsible for, rather than only counting the number of rules 1833 that it re-synced on the last sync. The new `sync_proxy_rules_iptables_last` 1834 metric now gives the latter number. ([#119140](https://github.com/kubernetes/kubernetes/pull/119140), [@danwinship](https://github.com/danwinship)) [SIG Network] 1835 - The metric `apiserver_flowcontrol_request_concurrency_limit` has been deprecated and will be removed in a future release. It is a duplicate of `apiserver_flowcontrol_nominal_limit_seats` (introduced in release 1.26) but has an outdated name and had an outdated HELP string. ([#118959](https://github.com/kubernetes/kubernetes/pull/118959), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 1836 - Updated `etcd` version to `3.5.8`. ([#117335](https://github.com/kubernetes/kubernetes/pull/117335), [@kkkkun](https://github.com/kkkkun)) 1837 - Updated apiserver metric `request_filter_duration_seconds` to include a 10s, 15s and 30s bucket. 1838 Updated apiserver metric `request_wait_duration_seconds` to include a 15s bucket. ([#118945](https://github.com/kubernetes/kubernetes/pull/118945), [@andrewsykim](https://github.com/andrewsykim)) 1839 - Updated kube-apiserver's priority & fairness work estimator such that 'max seats' is MIN(0.15 x nominalCL, nominalCL / handSize) 1840 1841 This fixes a bug where clients with requests using hand size x max seats greater than the nominal concurrency limit can starve other requests in the same priority level. ([#118601](https://github.com/kubernetes/kubernetes/pull/118601), [@andrewsykim](https://github.com/andrewsykim)) 1842 - Updated static pods are restarted 2s faster by correcting a safe but non-optimal ordering bug. ([#116690](https://github.com/kubernetes/kubernetes/pull/116690), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node] 1843 - Users will no longer see an error for failed events caused due to terminating namespace. ([#114849](https://github.com/kubernetes/kubernetes/pull/114849), [@padlar](https://github.com/padlar)) [SIG API Machinery] 1844 - [Dual-stack] Fixed `generateAPIPodStatus()` of kubelet handling Secondary IP. hostIPs order may not be consistent. If secondary IP is before primary one, current logic adds primary IP twice into `PodIPs`, which leads to error: "may specify no more than one IP for each IP family". ([#116879](https://github.com/kubernetes/kubernetes/pull/116879), [@lzhecheng](https://github.com/lzhecheng)) 1845 - [KCCM] service controller: change the cloud controller manager to make `providerID` a predicate when synchronizing nodes. This change allows load balancer integrations to ensure that the `providerID` is set when configuring 1846 load balancers and targets. ([#117388](https://github.com/kubernetes/kubernetes/pull/117388), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network] 1847 - `kube-apiserver` will now always remove its endpoint from Kubernetes service during 1848 graceful shutdown (even if it's the only/last one). ([#116685](https://github.com/kubernetes/kubernetes/pull/116685), [@nayihz](https://github.com/nayihz)) 1849 - `kubeadm:` fixed a bug where the static pod changes detection logic is inconsistent 1850 with kubelet. ([#118069](https://github.com/kubernetes/kubernetes/pull/118069), [@SataQiu](https://github.com/SataQiu)) 1851 - `kubeadm`: `crictl pull` should use `-i` to set the image service endpoint. ([#117835](https://github.com/kubernetes/kubernetes/pull/117835), [@pacoxu](https://github.com/pacoxu)) 1852 - `kubeadm`: fixed a bug where file copy(backup) could not be executed correctly 1853 on Windows platform during upgrade. ([#117861](https://github.com/kubernetes/kubernetes/pull/117861), [@SataQiu](https://github.com/SataQiu)) 1854 - `kubeadm`: speedup init by 0s or 20s. kubelet-start phase is now after etcd 1855 and control-plane phases, removing a race condition between kubelet looking for 1856 static pod manifests and kubeadm writing them. ([#117984](https://github.com/kubernetes/kubernetes/pull/117984), [@champtar](https://github.com/champtar)) 1857 - `kubeadm`: will now throw warnings instead of errors for deprecated feature gates. ([#118270](https://github.com/kubernetes/kubernetes/pull/118270), [@pacoxu](https://github.com/pacoxu)) 1858 - `kubectl events --for` will also support fully qualified names such as replicasets.apps, 1859 etc. ([#117034](https://github.com/kubernetes/kubernetes/pull/117034), [@ardaguclu](https://github.com/ardaguclu)) 1860 - `kubectl explain` should correctly work for all resources. ([#118876](https://github.com/kubernetes/kubernetes/pull/118876), [@atiratree](https://github.com/atiratree)) 1861 - `kubectl expose` supports the creation of different protocol services on the same port. ([#114909](https://github.com/kubernetes/kubernetes/pull/114909), [@aimuz](https://github.com/aimuz)) 1862 - `kubelet` will ensure /etc/hosts file is mode 0644 regardless of umask. ([#113209](https://github.com/kubernetes/kubernetes/pull/113209), [@luozhiwenn](https://github.com/luozhiwenn)) 1863 - `kubelet`: print sorted volumes message in events. ([#117079](https://github.com/kubernetes/kubernetes/pull/117079), [@qingwave](https://github.com/qingwave)) 1864 - `wait.PollUntilContextTimeout` function, if immediate is true, the condition 1865 will be invoked before waiting and guarantees that the condition is invoked at 1866 least once, regardless of whether the context has been cancelled. ([#118686](https://github.com/kubernetes/kubernetes/pull/118686), [@aojea](https://github.com/aojea)) 1867 1868 ### Other (Cleanup or Flake) 1869 1870 - A `v2-level` info log will be added, which will output the details of the pod being preempted, including victim and preemptor. ([#117214](https://github.com/kubernetes/kubernetes/pull/117214), [@HirazawaUi](https://github.com/HirazawaUi)) 1871 - Allowed container runtimes to use `ErrSignatureValidationFailed` as possible image pull failure. ([#117717](https://github.com/kubernetes/kubernetes/pull/117717), [@saschagrunert](https://github.com/saschagrunert)) 1872 - Deprecated `genericclioptions.IOStreams` and used `genericiooptions.IOStreams`. ([#117102](https://github.com/kubernetes/kubernetes/pull/117102), [@ardaguclu](https://github.com/ardaguclu)) 1873 - E2e framework: the `node-role.kubernetes.io/master` taint has been removed from the default value of `--non-blocking-taints` flag. You may need to set `--non-blocking-taints` explicitly if the cluster to be tested has nodes with the deprecated `node-role.kubernetes.io/master` taint. ([#118510](https://github.com/kubernetes/kubernetes/pull/118510), [@SataQiu](https://github.com/SataQiu)) [SIG Testing] 1874 - Enabled the `node-local` kubelet podresources API endpoint on windows, alongside unix. ([#115133](https://github.com/kubernetes/kubernetes/pull/115133), [@ffromani](https://github.com/ffromani)) 1875 - Fixed dra e2e image build on non-amd64 architectures. ([#117912](https://github.com/kubernetes/kubernetes/pull/117912), [@bart0sh](https://github.com/bart0sh)) [SIG Node and Testing] 1876 - Kube-apiserver adds two new alpha metrics `conversion_webhook_request_total` and `conversion_webhook_duration_seconds` that allow users to monitor requests to CRD conversion webhooks, split by result, and failure_type (In case of failure). ([#118292](https://github.com/kubernetes/kubernetes/pull/118292), [@cchapla](https://github.com/cchapla)) [SIG API Machinery, Architecture and Instrumentation] 1877 - Kube-proxy will now warn at startup if the configuration seems inconsistent 1878 with respect to IP families. (For example, if you have an IPv4 node IP, but 1879 `--cluster-cidr` is IPv6.) ([#119003](https://github.com/kubernetes/kubernetes/pull/119003), [@danwinship](https://github.com/danwinship)) [SIG Network] 1880 - Kube-proxy: removed log warning about not using config file. ([#118115](https://github.com/kubernetes/kubernetes/pull/118115), [@TommyStarK](https://github.com/TommyStarK)) [SIG Network] 1881 - Made Job controller batching of syncJob invocations enabled unconditionally (it was conditional on JobReadyPods feature before). 1882 Also, Job controller's constants for default backoff and maximal backoff are lowered down to 1s (from 10s) and 1min (from 6min), respectively. These constants are used to determine the backoff delay for the next Job controller sync in case of a request failure. ([#118615](https://github.com/kubernetes/kubernetes/pull/118615), [@mimowo](https://github.com/mimowo)) [SIG Apps and Testing] 1883 - Marked the feature gate `ExperimentalHostUserNamespaceDefaulting` as deprecated. 1884 Enabling the feature gate already had no effect; the deprecation allows for removing the feature gate in a future release. ([#116723](https://github.com/kubernetes/kubernetes/pull/116723), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Node] 1885 - Migrated `pkg/scheduler/framework/runtime` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116842](https://github.com/kubernetes/kubernetes/pull/116842), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 1886 - Migrated the disruption controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#119147](https://github.com/kubernetes/kubernetes/pull/119147), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG API Machinery, Apps, Instrumentation and Testing] 1887 - Migrated the interpodaffinity scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116635](https://github.com/kubernetes/kubernetes/pull/116635), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 1888 - Migrated the podgc controller and some other remaining log calls within `kube-controller-manager` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). `kube-controller-manager` is now converted completely. ([#119250](https://github.com/kubernetes/kubernetes/pull/119250), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Cloud Provider, Instrumentation, Network, Storage and Testing] 1889 - Migrated the volumezone scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116829](https://github.com/kubernetes/kubernetes/pull/116829), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 1890 - Moved `k8s.io/kubernetes/pkg/kubelet/cri/streaming` package to `k8s.io/kubelet/pkg/cri/streaming`. ([#118253](https://github.com/kubernetes/kubernetes/pull/118253), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node, Release and Security] 1891 - OpenAPI proto deserializations should use `gnostic-models` instead of the gnostic library. ([#118384](https://github.com/kubernetes/kubernetes/pull/118384), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node, Storage and Testing] 1892 - Projects which use k8s.io/code-generator and invoke `generate-groups` or `generate-internal-groups.sh` have a new, simpler script (`kube_codegen.sh`) they can use. The old scripts are deprecated but remain intact. ([#117262](https://github.com/kubernetes/kubernetes/pull/117262), [@thockin](https://github.com/thockin)) [SIG API Machinery and Instrumentation] 1893 - Promoted `kubernetes_healthcheck` and `kubernetes_healthchecks_total` to `BETA` stability level. ([#118986](https://github.com/kubernetes/kubernetes/pull/118986), [@logicalhan](https://github.com/logicalhan)) 1894 - Reduced delay when processing jobs after a transient API error. ([#118759](https://github.com/kubernetes/kubernetes/pull/118759), [@mimowo](https://github.com/mimowo)) 1895 - Removed GA'ed feature gate `DelegateFSGroupToCSIDriver`. ([#117655](https://github.com/kubernetes/kubernetes/pull/117655), [@carlory](https://github.com/carlory)) 1896 - Removed GA'ed feature gate `DevicePlugins`. ([#117656](https://github.com/kubernetes/kubernetes/pull/117656), [@carlory](https://github.com/carlory)) 1897 - Removed GA'ed feature gate `KubeletCredentialProviders`. ([#116901](https://github.com/kubernetes/kubernetes/pull/116901), [@pacoxu](https://github.com/pacoxu)) 1898 - Removed GA'ed feature gates: `MixedProtocolLBService`, `ServiceInternalTrafficPolicy`, 1899 `ServiceIPStaticSubrange`, and `EndpointSliceTerminatingCondition`. ([#117237](https://github.com/kubernetes/kubernetes/pull/117237), [@yulng](https://github.com/yulng)) 1900 - Removed `KUBECTL_EXPLAIN_OPENAPIV3` which is already redundant. ([#119286](https://github.com/kubernetes/kubernetes/pull/119286), [@ardaguclu](https://github.com/ardaguclu)) 1901 - Removed the deprecated `azureFile` in-tree storage plugin. ([#118236](https://github.com/kubernetes/kubernetes/pull/118236), [@andyzhangx](https://github.com/andyzhangx)) 1902 - Revised `OpenAPI v2` fetching for CustomResourceDefinitions. CRDs are now aggregated lazily, 1903 which improves resource usage during installation of many CRDs. As a result, the first request 1904 to fetch the OpenAPI may be slower. ([#118808](https://github.com/kubernetes/kubernetes/pull/118808), [@Jefftree](https://github.com/Jefftree)) 1905 - Shrank the `OpenAPI v2` spec by more than 50%, especially for less CPU resource consumption. ([#118204](https://github.com/kubernetes/kubernetes/pull/118204), [@sttts](https://github.com/sttts)) 1906 - Structured logging of `NamespacedName` was inconsistent with `klog.KObj`. Now both will use lower case field names and namespace is optional. ([#117238](https://github.com/kubernetes/kubernetes/pull/117238), [@pohly](https://github.com/pohly)) 1907 - The `GetAllocatableResources` podresources API endpoint is now GA. ([#118973](https://github.com/kubernetes/kubernetes/pull/118973), [@ffromani](https://github.com/ffromani)) 1908 - The `NetworkPolicyLegacy` test suite (deprecated in `v1.21`) has now officially been removed in favor of the new table driven e2e tests. ([#118915](https://github.com/kubernetes/kubernetes/pull/118915), [@astoycos](https://github.com/astoycos)) 1909 - The `generate_groups.sh` and `generate_internal_groups.sh` scripts from the `k8s.io/code-generator` repo are deprecated (but still work) in favor of `kube_codegen.sh` in that same repo. Projects which use the old scripts are encouraged to look at adopting the new one. ([#117897](https://github.com/kubernetes/kubernetes/pull/117897), [@thockin](https://github.com/thockin)) [SIG API Machinery] 1910 - The feature gate `CSIStorageCapacity` have been removed and must no longer be referenced in `--feature-gates` flags. ([#118018](https://github.com/kubernetes/kubernetes/pull/118018), [@humblec](https://github.com/humblec)) 1911 - The feature gates `CSIMigrationGCE` is graduated to GA and were unconditionally enabled have been removed in `v1.25`, and the entire `gcepd` package has been removed. ([#117055](https://github.com/kubernetes/kubernetes/pull/117055), [@cyclinder](https://github.com/cyclinder)) 1912 - The feature gates `DisableAcceleratorUsageMetrics` and `PodSecurity` that graduated to GA and were unconditionally enabled have been removed in v1.28. ([#114068](https://github.com/kubernetes/kubernetes/pull/114068), [@cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Node, Scheduling and Storage] 1913 - The kubelet podresources endpoint is GA and always enabled. ([#116525](https://github.com/kubernetes/kubernetes/pull/116525), [@ffromani](https://github.com/ffromani)) [SIG Node] 1914 - The metric `apiserver_flowcontrol_current_executing_seats` has been introduced as a duplicate of `apiserver_flowcontrol_request_concurrency_in_use` because the latter has a confusing name and will be removed in a later release. ([#118960](https://github.com/kubernetes/kubernetes/pull/118960), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 1915 - Updated `Cluster Autosaler` to version 1.26.1. ([#116526](https://github.com/kubernetes/kubernetes/pull/116526), [@pacoxu](https://github.com/pacoxu)) [SIG Autoscaling and Cloud Provider] 1916 - Updated `cri-tools` to `v1.27.0`. ([#117545](https://github.com/kubernetes/kubernetes/pull/117545), [@saschagrunert](https://github.com/saschagrunert)) 1917 - Updated `setcap` image to debian bookworm v1.0.0. ([#119247](https://github.com/kubernetes/kubernetes/pull/119247), [@saschagrunert](https://github.com/saschagrunert)) 1918 - Updated `cri-tools` to `v1.26.1`. ([#116649](https://github.com/kubernetes/kubernetes/pull/116649), [@saschagrunert](https://github.com/saschagrunert)) [SIG Architecture and Release] 1919 - Updated debian-base image to `bookworm-v1.0.0`. ([#119095](https://github.com/kubernetes/kubernetes/pull/119095), [@saschagrunert](https://github.com/saschagrunert)) 1920 - Use table-driven test for `TestPerPodSchedulingMetrics`. ([#118842](https://github.com/kubernetes/kubernetes/pull/118842), [@helayoty](https://github.com/helayoty)) 1921 - When retrieving event resources, the `reportingController` and `reportingInstance` fields in the event will contain values. ([#116506](https://github.com/kubernetes/kubernetes/pull/116506), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery and Instrumentation] 1922 - [KCCM] drop filtering nodes for the providerID when syncing load balancers, but have changes to the field trigger a re-sync of load balancers. This should ensure that cloud providers which don't specify providerID, can still use the service controller implementation to provision load balancers. ([#117602](https://github.com/kubernetes/kubernetes/pull/117602), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network] 1923 - `kube-apiserver` added two new metrics `authorization_attempts_total` and `authorization_duration_seconds` 1924 that allow users to monitor requests to authorization webhooks, split by result. ([#117211](https://github.com/kubernetes/kubernetes/pull/117211), [@HirazawaUi](https://github.com/HirazawaUi)) 1925 - `kube-apiserver`: Improved memory use when performing `GetList` on the cache. ([#116327](https://github.com/kubernetes/kubernetes/pull/116327), [@sxllwx](https://github.com/sxllwx)) 1926 - `kube-controller-manager` and `cloud-controller-manager` have changed the 1927 name of controllers that can be turned on/off that are passed to the `--controllers` 1928 flag (e.g., `pod-garbage-collector-controller`). The old names (eg `podgc`) are 1929 also accepted and aliased to the new names. ([#115813](https://github.com/kubernetes/kubernetes/pull/115813), [@atiratree](https://github.com/atiratree)) 1930 - `kubeadm`: Introduced a new feature gate `UpgradeAddonsBeforeControlPlane` to 1931 fix a kube-proxy skew policy misalignment. Its default value is `false`. Upgrade 1932 of the CoreDNS and kube-proxy addons will now trigger after all the control plane 1933 instances have been upgraded, unless the fearure gate is set to true. This feature 1934 gate will be removed in a future release. ([#117660](https://github.com/kubernetes/kubernetes/pull/117660), [@pacoxu](https://github.com/pacoxu)) 1935 1936 1937 ## Dependencies 1938 1939 ### Added 1940 - cloud.google.com/go/accessapproval: v1.6.0 1941 - cloud.google.com/go/accesscontextmanager: v1.7.0 1942 - cloud.google.com/go/aiplatform: v1.37.0 1943 - cloud.google.com/go/analytics: v0.19.0 1944 - cloud.google.com/go/apigateway: v1.5.0 1945 - cloud.google.com/go/apigeeconnect: v1.5.0 1946 - cloud.google.com/go/apigeeregistry: v0.6.0 1947 - cloud.google.com/go/appengine: v1.7.1 1948 - cloud.google.com/go/area120: v0.7.1 1949 - cloud.google.com/go/artifactregistry: v1.13.0 1950 - cloud.google.com/go/asset: v1.13.0 1951 - cloud.google.com/go/assuredworkloads: v1.10.0 1952 - cloud.google.com/go/automl: v1.12.0 1953 - cloud.google.com/go/baremetalsolution: v0.5.0 1954 - cloud.google.com/go/batch: v0.7.0 1955 - cloud.google.com/go/beyondcorp: v0.5.0 1956 - cloud.google.com/go/billing: v1.13.0 1957 - cloud.google.com/go/binaryauthorization: v1.5.0 1958 - cloud.google.com/go/certificatemanager: v1.6.0 1959 - cloud.google.com/go/channel: v1.12.0 1960 - cloud.google.com/go/cloudbuild: v1.9.0 1961 - cloud.google.com/go/clouddms: v1.5.0 1962 - cloud.google.com/go/cloudtasks: v1.10.0 1963 - cloud.google.com/go/compute/metadata: v0.2.3 1964 - cloud.google.com/go/compute: v1.19.0 1965 - cloud.google.com/go/contactcenterinsights: v1.6.0 1966 - cloud.google.com/go/container: v1.15.0 1967 - cloud.google.com/go/containeranalysis: v0.9.0 1968 - cloud.google.com/go/datacatalog: v1.13.0 1969 - cloud.google.com/go/dataflow: v0.8.0 1970 - cloud.google.com/go/dataform: v0.7.0 1971 - cloud.google.com/go/datafusion: v1.6.0 1972 - cloud.google.com/go/datalabeling: v0.7.0 1973 - cloud.google.com/go/dataplex: v1.6.0 1974 - cloud.google.com/go/dataproc: v1.12.0 1975 - cloud.google.com/go/dataqna: v0.7.0 1976 - cloud.google.com/go/datastream: v1.7.0 1977 - cloud.google.com/go/deploy: v1.8.0 1978 - cloud.google.com/go/dialogflow: v1.32.0 1979 - cloud.google.com/go/dlp: v1.9.0 1980 - cloud.google.com/go/documentai: v1.18.0 1981 - cloud.google.com/go/domains: v0.8.0 1982 - cloud.google.com/go/edgecontainer: v1.0.0 1983 - cloud.google.com/go/errorreporting: v0.3.0 1984 - cloud.google.com/go/essentialcontacts: v1.5.0 1985 - cloud.google.com/go/eventarc: v1.11.0 1986 - cloud.google.com/go/filestore: v1.6.0 1987 - cloud.google.com/go/functions: v1.13.0 1988 - cloud.google.com/go/gaming: v1.9.0 1989 - cloud.google.com/go/gkebackup: v0.4.0 1990 - cloud.google.com/go/gkeconnect: v0.7.0 1991 - cloud.google.com/go/gkehub: v0.12.0 1992 - cloud.google.com/go/gkemulticloud: v0.5.0 1993 - cloud.google.com/go/gsuiteaddons: v1.5.0 1994 - cloud.google.com/go/iam: v0.13.0 1995 - cloud.google.com/go/iap: v1.7.1 1996 - cloud.google.com/go/ids: v1.3.0 1997 - cloud.google.com/go/iot: v1.6.0 1998 - cloud.google.com/go/kms: v1.10.1 1999 - cloud.google.com/go/language: v1.9.0 2000 - cloud.google.com/go/lifesciences: v0.8.0 2001 - cloud.google.com/go/logging: v1.7.0 2002 - cloud.google.com/go/longrunning: v0.4.1 2003 - cloud.google.com/go/managedidentities: v1.5.0 2004 - cloud.google.com/go/maps: v0.7.0 2005 - cloud.google.com/go/mediatranslation: v0.7.0 2006 - cloud.google.com/go/memcache: v1.9.0 2007 - cloud.google.com/go/metastore: v1.10.0 2008 - cloud.google.com/go/monitoring: v1.13.0 2009 - cloud.google.com/go/networkconnectivity: v1.11.0 2010 - cloud.google.com/go/networkmanagement: v1.6.0 2011 - cloud.google.com/go/networksecurity: v0.8.0 2012 - cloud.google.com/go/notebooks: v1.8.0 2013 - cloud.google.com/go/optimization: v1.3.1 2014 - cloud.google.com/go/orchestration: v1.6.0 2015 - cloud.google.com/go/orgpolicy: v1.10.0 2016 - cloud.google.com/go/osconfig: v1.11.0 2017 - cloud.google.com/go/oslogin: v1.9.0 2018 - cloud.google.com/go/phishingprotection: v0.7.0 2019 - cloud.google.com/go/policytroubleshooter: v1.6.0 2020 - cloud.google.com/go/privatecatalog: v0.8.0 2021 - cloud.google.com/go/pubsublite: v1.7.0 2022 - cloud.google.com/go/recaptchaenterprise/v2: v2.7.0 2023 - cloud.google.com/go/recommendationengine: v0.7.0 2024 - cloud.google.com/go/recommender: v1.9.0 2025 - cloud.google.com/go/redis: v1.11.0 2026 - cloud.google.com/go/resourcemanager: v1.7.0 2027 - cloud.google.com/go/resourcesettings: v1.5.0 2028 - cloud.google.com/go/retail: v1.12.0 2029 - cloud.google.com/go/run: v0.9.0 2030 - cloud.google.com/go/scheduler: v1.9.0 2031 - cloud.google.com/go/secretmanager: v1.10.0 2032 - cloud.google.com/go/security: v1.13.0 2033 - cloud.google.com/go/securitycenter: v1.19.0 2034 - cloud.google.com/go/servicedirectory: v1.9.0 2035 - cloud.google.com/go/shell: v1.6.0 2036 - cloud.google.com/go/spanner: v1.45.0 2037 - cloud.google.com/go/speech: v1.15.0 2038 - cloud.google.com/go/storagetransfer: v1.8.0 2039 - cloud.google.com/go/talent: v1.5.0 2040 - cloud.google.com/go/texttospeech: v1.6.0 2041 - cloud.google.com/go/tpu: v1.5.0 2042 - cloud.google.com/go/trace: v1.9.0 2043 - cloud.google.com/go/translate: v1.7.0 2044 - cloud.google.com/go/video: v1.15.0 2045 - cloud.google.com/go/videointelligence: v1.10.0 2046 - cloud.google.com/go/vision/v2: v2.7.0 2047 - cloud.google.com/go/vmmigration: v1.6.0 2048 - cloud.google.com/go/vmwareengine: v0.3.0 2049 - cloud.google.com/go/vpcaccess: v1.6.0 2050 - cloud.google.com/go/webrisk: v1.8.0 2051 - cloud.google.com/go/websecurityscanner: v1.5.0 2052 - cloud.google.com/go/workflows: v1.10.0 2053 - github.com/alecthomas/kingpin/v2: [v2.3.2](https://github.com/alecthomas/kingpin/v2/tree/v2.3.2) 2054 - github.com/antlr/antlr4/runtime/Go/antlr/v4: [8188dc5](https://github.com/antlr/antlr4/runtime/Go/antlr/v4/tree/8188dc5) 2055 - github.com/google/gnostic-models: [v0.6.8](https://github.com/google/gnostic-models/tree/v0.6.8) 2056 - github.com/googleapis/enterprise-certificate-proxy: [v0.2.3](https://github.com/googleapis/enterprise-certificate-proxy/tree/v0.2.3) 2057 - github.com/xhit/go-str2duration/v2: [v2.1.0](https://github.com/xhit/go-str2duration/v2/tree/v2.1.0) 2058 - go.etcd.io/gofail: v0.1.0 2059 - google.golang.org/genproto/googleapis/api: dd9d682 2060 - google.golang.org/genproto/googleapis/rpc: 28d5490 2061 2062 ### Changed 2063 - cloud.google.com/go/bigquery: v1.8.0 → v1.50.0 2064 - cloud.google.com/go/datastore: v1.1.0 → v1.11.0 2065 - cloud.google.com/go/firestore: v1.1.0 → v1.9.0 2066 - cloud.google.com/go/pubsub: v1.3.1 → v1.30.0 2067 - cloud.google.com/go: v0.97.0 → v0.110.0 2068 - github.com/Azure/azure-sdk-for-go: [v55.0.0+incompatible → v68.0.0+incompatible](https://github.com/Azure/azure-sdk-for-go/compare/v55.0.0...v68.0.0) 2069 - github.com/Azure/go-autorest/autorest/adal: [v0.9.20 → v0.9.23](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.9.20...v0.9.23) 2070 - github.com/Azure/go-autorest/autorest/validation: [v0.1.0 → v0.3.1](https://github.com/Azure/go-autorest/autorest/validation/compare/v0.1.0...v0.3.1) 2071 - github.com/Azure/go-autorest/autorest: [v0.11.27 → v0.11.29](https://github.com/Azure/go-autorest/autorest/compare/v0.11.27...v0.11.29) 2072 - github.com/Microsoft/go-winio: [v0.4.17 → v0.6.0](https://github.com/Microsoft/go-winio/compare/v0.4.17...v0.6.0) 2073 - github.com/alecthomas/units: [f65c72e → b94a6e3](https://github.com/alecthomas/units/compare/f65c72e...b94a6e3) 2074 - github.com/cenkalti/backoff/v4: [v4.1.3 → v4.2.1](https://github.com/cenkalti/backoff/v4/compare/v4.1.3...v4.2.1) 2075 - github.com/census-instrumentation/opencensus-proto: [v0.2.1 → v0.4.1](https://github.com/census-instrumentation/opencensus-proto/compare/v0.2.1...v0.4.1) 2076 - github.com/cespare/xxhash/v2: [v2.1.2 → v2.2.0](https://github.com/cespare/xxhash/v2/compare/v2.1.2...v2.2.0) 2077 - github.com/cilium/ebpf: [v0.7.0 → v0.9.1](https://github.com/cilium/ebpf/compare/v0.7.0...v0.9.1) 2078 - github.com/cncf/udpa/go: [04548b0 → c52dc94](https://github.com/cncf/udpa/go/compare/04548b0...c52dc94) 2079 - github.com/cncf/xds/go: [cb28da3 → 06c439d](https://github.com/cncf/xds/go/compare/cb28da3...06c439d) 2080 - github.com/cockroachdb/datadriven: [bf6692d → v1.0.2](https://github.com/cockroachdb/datadriven/compare/bf6692d...v1.0.2) 2081 - github.com/container-storage-interface/spec: [v1.7.0 → v1.8.0](https://github.com/container-storage-interface/spec/compare/v1.7.0...v1.8.0) 2082 - github.com/containerd/cgroups: [v1.0.1 → v1.1.0](https://github.com/containerd/cgroups/compare/v1.0.1...v1.1.0) 2083 - github.com/containerd/ttrpc: [v1.1.0 → v1.2.2](https://github.com/containerd/ttrpc/compare/v1.1.0...v1.2.2) 2084 - github.com/coredns/caddy: [v1.1.0 → v1.1.1](https://github.com/coredns/caddy/compare/v1.1.0...v1.1.1) 2085 - github.com/coreos/go-oidc: [v2.1.0+incompatible → v2.2.1+incompatible](https://github.com/coreos/go-oidc/compare/v2.1.0...v2.2.1) 2086 - github.com/coreos/go-semver: [v0.3.0 → v0.3.1](https://github.com/coreos/go-semver/compare/v0.3.0...v0.3.1) 2087 - github.com/coreos/go-systemd/v22: [v22.4.0 → v22.5.0](https://github.com/coreos/go-systemd/v22/compare/v22.4.0...v22.5.0) 2088 - github.com/docker/distribution: [v2.8.1+incompatible → v2.8.2+incompatible](https://github.com/docker/distribution/compare/v2.8.1...v2.8.2) 2089 - github.com/dustin/go-humanize: [v1.0.0 → v1.0.1](https://github.com/dustin/go-humanize/compare/v1.0.0...v1.0.1) 2090 - github.com/envoyproxy/go-control-plane: [49ff273 → v0.10.3](https://github.com/envoyproxy/go-control-plane/compare/49ff273...v0.10.3) 2091 - github.com/envoyproxy/protoc-gen-validate: [v0.1.0 → v0.9.1](https://github.com/envoyproxy/protoc-gen-validate/compare/v0.1.0...v0.9.1) 2092 - github.com/evanphx/json-patch: [v4.12.0+incompatible → v5.6.0+incompatible](https://github.com/evanphx/json-patch/compare/v4.12.0...v5.6.0) 2093 - github.com/frankban/quicktest: [v1.11.3 → v1.14.0](https://github.com/frankban/quicktest/compare/v1.11.3...v1.14.0) 2094 - github.com/fvbommel/sortorder: [v1.0.1 → v1.1.0](https://github.com/fvbommel/sortorder/compare/v1.0.1...v1.1.0) 2095 - github.com/go-kit/log: [v0.2.0 → v0.2.1](https://github.com/go-kit/log/compare/v0.2.0...v0.2.1) 2096 - github.com/go-logr/logr: [v1.2.3 → v1.2.4](https://github.com/go-logr/logr/compare/v1.2.3...v1.2.4) 2097 - github.com/go-openapi/jsonreference: [v0.20.1 → v0.20.2](https://github.com/go-openapi/jsonreference/compare/v0.20.1...v0.20.2) 2098 - github.com/go-task/slim-sprig: [348f09d → 52ccab3](https://github.com/go-task/slim-sprig/compare/348f09d...52ccab3) 2099 - github.com/gofrs/uuid: [v4.0.0+incompatible → v4.4.0+incompatible](https://github.com/gofrs/uuid/compare/v4.0.0...v4.4.0) 2100 - github.com/golang-jwt/jwt/v4: [v4.4.2 → v4.5.0](https://github.com/golang-jwt/jwt/v4/compare/v4.4.2...v4.5.0) 2101 - github.com/google/cadvisor: [v0.47.1 → v0.47.3](https://github.com/google/cadvisor/compare/v0.47.1...v0.47.3) 2102 - github.com/google/cel-go: [v0.12.6 → v0.16.0](https://github.com/google/cel-go/compare/v0.12.6...v0.16.0) 2103 - github.com/google/gofuzz: [v1.1.0 → v1.2.0](https://github.com/google/gofuzz/compare/v1.1.0...v1.2.0) 2104 - github.com/googleapis/gax-go/v2: [v2.1.1 → v2.7.1](https://github.com/googleapis/gax-go/v2/compare/v2.1.1...v2.7.1) 2105 - github.com/inconshreveable/mousetrap: [v1.0.1 → v1.1.0](https://github.com/inconshreveable/mousetrap/compare/v1.0.1...v1.1.0) 2106 - github.com/kr/pretty: [v0.3.0 → v0.3.1](https://github.com/kr/pretty/compare/v0.3.0...v0.3.1) 2107 - github.com/matttproud/golang_protobuf_extensions: [v1.0.2 → v1.0.4](https://github.com/matttproud/golang_protobuf_extensions/compare/v1.0.2...v1.0.4) 2108 - github.com/mitchellh/go-wordwrap: [v1.0.0 → v1.0.1](https://github.com/mitchellh/go-wordwrap/compare/v1.0.0...v1.0.1) 2109 - github.com/mitchellh/mapstructure: [v1.4.1 → v1.1.2](https://github.com/mitchellh/mapstructure/compare/v1.4.1...v1.1.2) 2110 - github.com/onsi/ginkgo/v2: [v2.9.1 → v2.9.4](https://github.com/onsi/ginkgo/v2/compare/v2.9.1...v2.9.4) 2111 - github.com/onsi/gomega: [v1.27.4 → v1.27.6](https://github.com/onsi/gomega/compare/v1.27.4...v1.27.6) 2112 - github.com/opencontainers/runc: [v1.1.4 → v1.1.7](https://github.com/opencontainers/runc/compare/v1.1.4...v1.1.7) 2113 - github.com/prometheus/client_golang: [v1.14.0 → v1.16.0](https://github.com/prometheus/client_golang/compare/v1.14.0...v1.16.0) 2114 - github.com/prometheus/client_model: [v0.3.0 → v0.4.0](https://github.com/prometheus/client_model/compare/v0.3.0...v0.4.0) 2115 - github.com/prometheus/common: [v0.37.0 → v0.44.0](https://github.com/prometheus/common/compare/v0.37.0...v0.44.0) 2116 - github.com/prometheus/procfs: [v0.8.0 → v0.10.1](https://github.com/prometheus/procfs/compare/v0.8.0...v0.10.1) 2117 - github.com/seccomp/libseccomp-golang: [f33da4d → v0.10.0](https://github.com/seccomp/libseccomp-golang/compare/f33da4d...v0.10.0) 2118 - github.com/spf13/cobra: [v1.6.0 → v1.7.0](https://github.com/spf13/cobra/compare/v1.6.0...v1.7.0) 2119 - github.com/stretchr/testify: [v1.8.1 → v1.8.2](https://github.com/stretchr/testify/compare/v1.8.1...v1.8.2) 2120 - github.com/vishvananda/netns: [v0.0.2 → v0.0.4](https://github.com/vishvananda/netns/compare/v0.0.2...v0.0.4) 2121 - github.com/xlab/treeprint: [v1.1.0 → v1.2.0](https://github.com/xlab/treeprint/compare/v1.1.0...v1.2.0) 2122 - go.etcd.io/bbolt: v1.3.6 → v1.3.7 2123 - go.etcd.io/etcd/api/v3: v3.5.7 → v3.5.9 2124 - go.etcd.io/etcd/client/pkg/v3: v3.5.7 → v3.5.9 2125 - go.etcd.io/etcd/client/v2: v2.305.7 → v2.305.9 2126 - go.etcd.io/etcd/client/v3: v3.5.7 → v3.5.9 2127 - go.etcd.io/etcd/pkg/v3: v3.5.7 → v3.5.9 2128 - go.etcd.io/etcd/raft/v3: v3.5.7 → v3.5.9 2129 - go.etcd.io/etcd/server/v3: v3.5.7 → v3.5.9 2130 - go.opencensus.io: v0.23.0 → v0.24.0 2131 - go.starlark.net: 8dd3e2e → a134d8f 2132 - go.uber.org/atomic: v1.7.0 → v1.10.0 2133 - go.uber.org/multierr: v1.6.0 → v1.11.0 2134 - golang.org/x/crypto: v0.1.0 → v0.11.0 2135 - golang.org/x/exp: 6cc2880 → a9213ee 2136 - golang.org/x/mod: v0.9.0 → v0.10.0 2137 - golang.org/x/net: v0.8.0 → v0.13.0 2138 - golang.org/x/oauth2: ee48083 → v0.8.0 2139 - golang.org/x/sync: v0.1.0 → v0.2.0 2140 - golang.org/x/sys: v0.6.0 → v0.10.0 2141 - golang.org/x/term: v0.6.0 → v0.10.0 2142 - golang.org/x/text: v0.8.0 → v0.11.0 2143 - golang.org/x/time: 90d013b → v0.3.0 2144 - golang.org/x/tools: v0.7.0 → v0.8.0 2145 - google.golang.org/api: v0.60.0 → v0.114.0 2146 - google.golang.org/genproto: c8bf987 → 0005af6 2147 - google.golang.org/grpc: v1.51.0 → v1.54.0 2148 - google.golang.org/protobuf: v1.28.1 → v1.30.0 2149 - gopkg.in/gcfg.v1: v1.2.0 → v1.2.3 2150 - gopkg.in/natefinch/lumberjack.v2: v2.0.0 → v2.2.1 2151 - gopkg.in/warnings.v0: v0.1.1 → v0.1.2 2152 - k8s.io/klog/v2: v2.90.1 → v2.100.1 2153 - k8s.io/kube-openapi: 15aac26 → 2695361 2154 - k8s.io/utils: a36077c → d93618c 2155 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.1.1 → v0.1.2 2156 - sigs.k8s.io/kustomize/api: v0.13.2 → 6ce0bf3 2157 - sigs.k8s.io/kustomize/cmd/config: v0.11.1 → v0.11.2 2158 - sigs.k8s.io/kustomize/kustomize/v5: v5.0.1 → 6ce0bf3 2159 - sigs.k8s.io/kustomize/kyaml: v0.14.1 → 6ce0bf3 2160 2161 ### Removed 2162 - github.com/antlr/antlr4/runtime/Go/antlr: [v1.4.10](https://github.com/antlr/antlr4/runtime/Go/antlr/tree/v1.4.10) 2163 - github.com/certifi/gocertifi: [2c3bb06](https://github.com/certifi/gocertifi/tree/2c3bb06) 2164 - github.com/cockroachdb/errors: [v1.2.4](https://github.com/cockroachdb/errors/tree/v1.2.4) 2165 - github.com/cockroachdb/logtags: [eb05cc2](https://github.com/cockroachdb/logtags/tree/eb05cc2) 2166 - github.com/docopt/docopt-go: [ee0de3b](https://github.com/docopt/docopt-go/tree/ee0de3b) 2167 - github.com/getsentry/raven-go: [v0.2.0](https://github.com/getsentry/raven-go/tree/v0.2.0) 2168 - github.com/google/gnostic: [v0.5.7-v3refs](https://github.com/google/gnostic/tree/v0.5.7-v3refs) 2169 2170 2171 2172 # v1.28.0-rc.1 2173 2174 2175 ## Downloads for v1.28.0-rc.1 2176 2177 2178 2179 ### Source Code 2180 2181 filename | sha512 hash 2182 -------- | ----------- 2183 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes.tar.gz) | efc4ec914eb4e1147cdbadb9a5dccc4608a983ba6308c85d4c2e8e1c984f35c12e04b027d0a0f6e07c2371fae9aa4879b4831158e7cfe77887da7e20778e717b 2184 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-src.tar.gz) | cfdd470979b447dab1678c03bd9bc7745ef7d6907a043d5180e96494d4a5b91b4d8a08b09726e15cda4668437cc296528df646f5f58c870af8134312cf8851ba 2185 2186 ### Client Binaries 2187 2188 filename | sha512 hash 2189 -------- | ----------- 2190 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-darwin-amd64.tar.gz) | ca12f940ef00fc38152dae75d8f817b03be4d59c7a8d64a80a515fbbf78d526a2b98311efbc9a8d34361b70ba0b07156cff7fbb6c19dc503c7c16e0dfc8e3ec0 2191 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-darwin-arm64.tar.gz) | 9d1d0aaedd698a4c5ad5ed2903e8492e52e6f16f858b8d186129edb5c9b199d4352e443c8aba67f58db7fcf950162ffbb4f7211570a0c4be684656ad5ada42bf 2192 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-386.tar.gz) | 59e1c0692821eb64b59147baf43985226ae2223fff66981a2a13d5d60d5e102faf7c5cecfa5b8ec1c5a48c9a9fed35223f8ebbc93ac972ea949f2a3096f64672 2193 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-amd64.tar.gz) | a348ad33f936e075083e2ffec4405a726984df8c522e10d34ad65b64eee53902fb6483903581c30b047872fd130cf24f6cdc193458fcf7d5774364bf78c1c982 2194 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-arm.tar.gz) | 3069b3000445218a98b7d11bd196cbfb84aed16dcbd16adff88935bb01a87f8da29cc4824de4c8af7ddc2050134e2c3467408218fc7209700c0e1c0aec2d3ced 2195 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-arm64.tar.gz) | d59ebea7b7d78ee1ef59317073a5a4b9e513b9f43026ffc1b7745211e8cd9de738a05d1fa2c29501d3cee24252732c3348f109f2b7c1e7425a4eff46cf1b4654 2196 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-ppc64le.tar.gz) | 2490266ee44469c56bd85f86774668bc9a6d6f2d4f21ddeb95da8eb638f743501e9ed3646c817db0c746730f93b381b6c5ae46d25d9dbadc78d3ca8f89eccfd6 2197 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-s390x.tar.gz) | a843bc8df85594ab5fca9e1e17997e14cdf8e45ef2e74165222b6dc8d26a8f85d85a972a5c3e1740f3eb6d3647b81e3dfb66787cdac6dcd42a59c2f5507f6031 2198 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-windows-386.tar.gz) | 1ce7d2802cb9ebbc61f68ae3a8380056fc039e9959ea999f3d7da1254b33225809932a9679b2a0f96ca4adad73aa4aedba3ec9f20182899f62ff59133e48f4d1 2199 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-windows-amd64.tar.gz) | 85986399b437aa48d276301a7a06b17e4004d5423dec2faf699a3a377dd28f3e734b0655848168407fb25b6898389daf45f0ad695519e1f3f31586e9a8586531 2200 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-windows-arm64.tar.gz) | ac18cf32f0273470841024a762d7ded78993128bada439340d0f4c604af6d7001971f3075437e65471ab7ae89d15fd82f2689b6d47da681dcc8779c277a9cea5 2201 2202 ### Server Binaries 2203 2204 filename | sha512 hash 2205 -------- | ----------- 2206 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-server-linux-amd64.tar.gz) | 6b5ebfe42050e48f108da43275cdfb0b5ec652867d12a632bf5ed4b00482efd2470184028bf94b36a1f05c5a70ad1057f334483461f9212bdb48dcac6b169600 2207 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-server-linux-arm64.tar.gz) | 09b1a44182ab30a4740b703cae5f46ce4cb4af9716ed1f2cb2a4dc59c9baa2eaa402b01167b04c6801b550035ba9f939d4d1209689363daffe870dd2f44e4528 2208 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-server-linux-ppc64le.tar.gz) | 09ba87bfb42f7f710b446a588d2047fe6aea26df171aacc3157c3fa4c9e718856ad3efc45b0050d35a9153e94d5da81c632ddaec71663d30c5d43284292b305f 2209 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-server-linux-s390x.tar.gz) | 42ae7f05cb279e57ef89570b1596759cc771663ceb72f358a9119c91b981b99335b46a887f59f8a8727303366c3111bd4696817343ddbee3ee02811bd6022e4c 2210 2211 ### Node Binaries 2212 2213 filename | sha512 hash 2214 -------- | ----------- 2215 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-linux-amd64.tar.gz) | 4a0c7ae5ce52698087eaca1ec108ca5c1333ddc607a5fcb63d5e65cde17e3c8e64037905da02656e4a663037be1b00441754af4563c5eed1ec8ab57bf692c4ed 2216 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-linux-arm64.tar.gz) | aba8ddbc9cd9cf0e2fd5eaac8bee2490135c31fcd43751dd5e4438eca813372104e99f34517acfdd2abddf3a28cffef4cc42eb9bfecd76b50d89adff5675f32b 2217 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-linux-ppc64le.tar.gz) | d16d79c2f5680d31aa6fcc659de210632d0ed761b6c4c067ab5976fe41360a9cc7a75cad545e04831bf9d2b8669523dd9cf4c756337c5328ff10a8d61a5301bc 2218 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-linux-s390x.tar.gz) | f7bb37de732b35db011f1d2f52ac461c7f912de39bb16c13a638b2dcd160876c6d1e278d36fdee07d8598b69b30e33f7c9bc980b6b25651e4b74cf3517514371 2219 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-windows-amd64.tar.gz) | 86114fbfa8980e678bddb3d01290b5b3158d92ffdc92b970d1a224fff1f7914415c9adb3b663f5f036d5abb0aad95c96c1a819b309e157aba9392a6c77e65ff5 2220 2221 ### Container Images 2222 2223 All container images are available as manifest lists and support the described 2224 architectures. It is also possible to pull a specific architecture directly by 2225 adding the "-$ARCH" suffix to the container image name. 2226 2227 name | architectures 2228 ---- | ------------- 2229 [registry.k8s.io/conformance:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 2230 [registry.k8s.io/kube-apiserver:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 2231 [registry.k8s.io/kube-controller-manager:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 2232 [registry.k8s.io/kube-proxy:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 2233 [registry.k8s.io/kube-scheduler:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 2234 [registry.k8s.io/kubectl:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 2235 2236 ## Changelog since v1.28.0-rc.0 2237 2238 ## Changes by Kind 2239 2240 ### API Change 2241 2242 - Aggregated discovery now returns `responseKind: {}` for resources which are missing group/version/kind information, to ensure compatibility with v0.26.0-v0.26.3 clients. ([#119835](https://github.com/kubernetes/kubernetes/pull/119835), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing] 2243 - Fix CustomResourceDefinition status.storedVersions validation error messages. ([#119653](https://github.com/kubernetes/kubernetes/pull/119653), [@sttts](https://github.com/sttts)) [SIG API Machinery] 2244 - Kube-proxy in Kubernetes >= 1.28 up until v1.28.0-beta.0 ignored the `-v` command line flag when combined with `--config`. ([#119867](https://github.com/kubernetes/kubernetes/pull/119867), [@pohly](https://github.com/pohly)) [SIG Network] 2245 2246 ### Feature 2247 2248 - Bump distroless-iptables to 0.2.7 based on Go 1.20.7 ([#119818](https://github.com/kubernetes/kubernetes/pull/119818), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG Testing] 2249 - Kubernetes is now built with Go 1.20.7 ([#119804](https://github.com/kubernetes/kubernetes/pull/119804), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG Release and Testing] 2250 2251 ### Bug or Regression 2252 2253 - Fixes issue https://github.com/kubernetes-sigs/cloud-provider-azure/issues/4230 and removes the additional filtering on `NotReady` nodes by the azure cloud provider code ([#119128](https://github.com/kubernetes/kubernetes/pull/119128), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider] 2254 - Kube-scheduler: Fine-grained tracking of events (introduced in 1.28) suffered from a data race when binding fails. ([#119729](https://github.com/kubernetes/kubernetes/pull/119729), [@pohly](https://github.com/pohly)) [SIG Scheduling] 2255 - Revert kubelet prober metrics `pod` tag to include actual pod name ([#118549](https://github.com/kubernetes/kubernetes/pull/118549), [@a7i](https://github.com/a7i)) [SIG Node] 2256 - When the cluster size is small and the scheduler doesn't get unscheduled Pods frequently, the scheduler doesn't try to reschedule Pods in some cases. ([#119784](https://github.com/kubernetes/kubernetes/pull/119784), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling and Testing] 2257 2258 ## Dependencies 2259 2260 ### Added 2261 _Nothing has changed._ 2262 2263 ### Changed 2264 - golang.org/x/net: v0.12.0 → v0.13.0 2265 2266 ### Removed 2267 _Nothing has changed._ 2268 2269 2270 2271 # v1.28.0-rc.0 2272 2273 2274 ## Downloads for v1.28.0-rc.0 2275 2276 2277 2278 ### Source Code 2279 2280 filename | sha512 hash 2281 -------- | ----------- 2282 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes.tar.gz) | bd3feaf924371be8fe3e2130c6cac5fd5fd3c90d42be383e076c16160c95ec48668b5b330e0742d562a3b0eae18eda71bab76dff5e2aebad61513c2be6b251b6 2283 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-src.tar.gz) | 9f3fbb4c624124bf82473028e9c21a123f525e9dc8a224ede4f00fbf0630ae812ba58d56be69aa45a2b039da1deb4ce9052061b40699945f9fb88bb59fcb3977 2284 2285 ### Client Binaries 2286 2287 filename | sha512 hash 2288 -------- | ----------- 2289 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-darwin-amd64.tar.gz) | fabc769eef82c242d905cee81d7e876280adeffa95208ddd4d1c0de69e82d775afef984cc9b3b375ee2072e8dceb846da105c76c94e6d323f9778c9e9c0b49ba 2290 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-darwin-arm64.tar.gz) | 0a321eaa366d73feab93bab77e56dbd8ee2fde298766a8f7e37b98f6053c4e86ae77f64561bd083cd5f3b4f61d727de013d6ef74a0bd0c35b7afb8cb110a063c 2291 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-386.tar.gz) | a8720f87b0eafefca413547e3f603660d147b9198eabac03ac59a57a70fc011c48daaef4cf1d63a974578e8b20f98e6f3dca7997f6feaa009944e16ec47ea8c9 2292 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-amd64.tar.gz) | 918dc97380ebe56c16de8131d35f9ddc21b2196f8b6b0b24361fa4a23a2cf56c75edf4555eccbad3453663c007bd51d147e0a589f933e0759410879e2aeadfd8 2293 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-arm.tar.gz) | c1f87fff085884632cb1b60c72f4cf168079bff4150270e67855b1211a1240e3252e6791dfb61672a2ffbe4314b360b917657608b3c65d661871852ba84e8ca3 2294 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-arm64.tar.gz) | 2f33cb523472e162dc206edee4e33903a4550d3f73adbe327c7c34f8084b44dd0ed6b6b28c9d85eca4e6e2a1d1124c9dbc0d8a4a14abc7b810a7e4085f5e97bc 2295 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-ppc64le.tar.gz) | 9edc3618d12e4480ccb448e928017bd9c3f1e8ad16bc83cddea4a73c81ab2d7a5085bbedf2a0324a9377d17faa7168e2d5c27a7de5bed8e07809b1227c4b9079 2296 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-s390x.tar.gz) | 53a31035114aaa7d837dcdc7663a9ee91e6d49d549eea6c7337a6f3a4325b34a6931e65ce471a758bed152a92adb434581084e2810b16c8143582501b48e4363 2297 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-windows-386.tar.gz) | 8158c3947c5838fef84b6427fb27e6cf2375344f6d27bd1a2d0563d3a242bc445278968bcf36b7657a4db7322b2a9d5aad028480c6bb5fbf2faf3a2dddad931e 2298 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-windows-amd64.tar.gz) | aa809425d557fc9323748a0484f9b59f9f6e089ab6256da5690014efbda9a9c1a96110a3511f930e4e2714315005bd803ea059b1a5221a825f109a69d6c60967 2299 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-windows-arm64.tar.gz) | f72acfa3ac61cbf7fc2b612a9c8c9d19de42488752120e6b4d69fafa35beda00bfbe9dc839430b3987ff5e9737930d0a9fae867ab35824f0a0eef47b6496404d 2300 2301 ### Server Binaries 2302 2303 filename | sha512 hash 2304 -------- | ----------- 2305 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-server-linux-amd64.tar.gz) | a3ca2b91e204f6bcf84e1d54412ca6c908a1e02a48b8022da821bfd4fe136b565a980da3fef2270a34ca637ff4ff306cd3b09760556602db15a7b8a3dfead0d1 2306 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-server-linux-arm64.tar.gz) | 40e6aea2c1fa0f9f6c135363991216d80b927c6aabcf30fc3288512c38fd75b8fff868770209cde8016a2a33dee7ad862709840c7a040f6203240b06bc2e5c5d 2307 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-server-linux-ppc64le.tar.gz) | da41491b39fb8a1a2f7619f14ff1165ff0d467bef82348e71bfd31d481678dff27b03a01d3fa2deccdf04227a0fe7c9593d8d7b9e7745070afcb53dfd70b2bc6 2308 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-server-linux-s390x.tar.gz) | 6deb9a3625f510cdd5dea4ced9648fe563595cddfd2f9223e4acbf3fe74cd2b4283011984105f79794731e81b7b3725a16c964b70f25a4bad76b60b75f54372d 2309 2310 ### Node Binaries 2311 2312 filename | sha512 hash 2313 -------- | ----------- 2314 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-linux-amd64.tar.gz) | d4178a311495afe6e9d4a5c4fa9b1d0d17c7a0918305b782406ea5f5fbfcd8c4f60073e02b9aff3c37eb9f4fd331844177e48650b4e489b0a5430da5e00a33a2 2315 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-linux-arm64.tar.gz) | 8ee5e87248caca033fafea7ea1470fcc282cc402f6591d9120fd87c4e533bade19e125c1a840d1ebf503fb0eda21096a047571ffde79a6b0263494799bdf042f 2316 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-linux-ppc64le.tar.gz) | 022f4804d408788d426176f5dda8046005084261c426766476b429e2d62f60e22509d66e9199350ba501b17f6e6cb72e7e4bd2581b950c85e5c5a05efcb0139c 2317 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-linux-s390x.tar.gz) | d9e95ce6e0f886d8a89df8d53f2533c9be2ef733be25a7455cfc89028ee5ed5b7fbdfb81f7c3e452b405dc8b3d4c252f74039d300a0fd01c7b75e7b5f6c0c551 2318 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-windows-amd64.tar.gz) | 6bd4809f6b8d1bea8da4565ddda294d39eb3ab424dd47e01a5919e974b33316c64fc21bc8b2c62a05f14db71621951ecc82c5988da11b8190a6e1ee2eec79cd5 2319 2320 ### Container Images 2321 2322 All container images are available as manifest lists and support the described 2323 architectures. It is also possible to pull a specific architecture directly by 2324 adding the "-$ARCH" suffix to the container image name. 2325 2326 name | architectures 2327 ---- | ------------- 2328 [registry.k8s.io/conformance:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 2329 [registry.k8s.io/kube-apiserver:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 2330 [registry.k8s.io/kube-controller-manager:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 2331 [registry.k8s.io/kube-proxy:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 2332 [registry.k8s.io/kube-scheduler:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 2333 [registry.k8s.io/kubectl:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 2334 2335 ## Changelog since v1.28.0-beta.0 2336 2337 ## Changes by Kind 2338 2339 ### API Change 2340 2341 - PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. ([#116469](https://github.com/kubernetes/kubernetes/pull/116469), [@RomanBednar](https://github.com/RomanBednar)) [SIG API Machinery, Apps, Auth, Node, Release, Storage and Testing] 2342 - Promoted API groups `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` to `v1beta1`. ([#118644](https://github.com/kubernetes/kubernetes/pull/118644), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps and Testing] 2343 - Promoted the feature gate `ValidtaingAdmissionPolicy` to beta and it is turned off by default. ([#119409](https://github.com/kubernetes/kubernetes/pull/119409), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps, Auth, Instrumentation, Node, Release, Storage and Testing] 2344 - Changed how KMS v2 encryption at rest can generate data encryption keys. When you enable the `KMSv2KDF` feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. ([#118828](https://github.com/kubernetes/kubernetes/pull/118828), [@enj](https://github.com/enj)) [SIG API Machinery, Auth and Testing] 2345 2346 ### Feature 2347 2348 - Add implementation for PodRecreationPolicy to wait for creation of pods once the existing ones are fully terminated. ([#117015](https://github.com/kubernetes/kubernetes/pull/117015), [@kannon92](https://github.com/kannon92)) [SIG API Machinery, Apps and Testing] 2349 2350 ## Dependencies 2351 2352 ### Added 2353 _Nothing has changed._ 2354 2355 ### Changed 2356 _Nothing has changed._ 2357 2358 ### Removed 2359 _Nothing has changed._ 2360 2361 2362 2363 # v1.28.0-beta.0 2364 2365 2366 ## Downloads for v1.28.0-beta.0 2367 2368 2369 2370 ### Source Code 2371 2372 filename | sha512 hash 2373 -------- | ----------- 2374 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes.tar.gz) | 5a4575157380ff5dd66fd87e7045f2f57ed0db59513bfef57ee768a7a98f855faa06503a7480e77cdf5128fe66461a6c91f0705f8148347f903342f45b65f8da 2375 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-src.tar.gz) | 8efebf779daf168929dd2798d2e52750d09339a17256091b4315b1de82167b26388725a60bc3525468b1a23656932faf1c45ba6957df0bd2b3f48bafc8b62138 2376 2377 ### Client Binaries 2378 2379 filename | sha512 hash 2380 -------- | ----------- 2381 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-darwin-amd64.tar.gz) | 956bd780cb36815ae8969e345d4f6004740167f3e5e3a1d1b1deda254ac2b167371b7c9e79497bd01e3f11d2e2f8e1c35f8fdc3114f08324470635cff1efab20 2382 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-darwin-arm64.tar.gz) | 68ebd65edb40e23c5e70b6666fea34a774ccd66313dc884e28a8032cfd8c166c1a2dc66c635a61b0d568fe825b208bfd32a040e3eded536f71617acac625f3ea 2383 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-386.tar.gz) | 0291c01019e47af1631f3f90f46aa92e4572301fe2411586fb2d69291fdf113a8ff78531f51530d05c6113e28e0e69e23f1d2e2143f832f6b8f77a133b09e493 2384 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-amd64.tar.gz) | bd9a98bd35ed9bb113399e50456da7008629e060381a7f6b9071fd7bcd498cb252da2824376662a4d4d4cae637feb90f3768414751e7ec0339c6c9711f6aceda 2385 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-arm.tar.gz) | 311b3c1a42d68fa337f6e8b2290817196c97024535ee94f2180971e09ca78c9037b41fca48c1eda1fa75079631ea8805f0c6a173e35fe4a9762d13bf3bdf1c58 2386 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-arm64.tar.gz) | 039491fddc63e279821ff5287cada111b5c94dc0389d697adc2503f77905a74cda1a67dd52d4b42be0141896d04cfb18f3dfc0c387620d9a9fb3a7f2aa5b399e 2387 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-ppc64le.tar.gz) | 70e4011dce6777c511a69872a1e015b3a2137cea100cd6b988946ea227bd05827f4223a44c9d24433043cc414d3746465603dc4de2e84128e7689712dd29b00d 2388 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-s390x.tar.gz) | 51ea51fbcdcf7f2c9c4c97b6f28ceb42753e52ccf69318c05befa77b94d9c17e871061b4d03dbc12632663efed161424b3dba97ac6df46cf27d6deb1a0c011fb 2389 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-windows-386.tar.gz) | e069121289664d53a6570d90ac2d66911d18cf8c47461f5e8e2cd52ddd651a0e119327d23455b9c273e979378379d27e5cf102c7f8b8c98871c9ef9c7e790e52 2390 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-windows-amd64.tar.gz) | f19a06ad641a282b9895673a0628cc937222b53d9b852fd7543de01294d403100d0bcd4659fbf73bf8cdc55a8e3c7f494991db2b9f4d8bc63446b7810232d3aa 2391 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-windows-arm64.tar.gz) | 7bd6a6d6ae7b487aa35c162532e9b59e184aac0fb6b65c61b2af06217007f78aeb5d0bed039983c534358152155739a6e30f533488c49a13d3c81d54eea6b8c6 2392 2393 ### Server Binaries 2394 2395 filename | sha512 hash 2396 -------- | ----------- 2397 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-server-linux-amd64.tar.gz) | 7cd83dc6e7013ef8638965fb3e30f8ba122f045d987029da345c185662bea824fc6a3fae34ff549c457638daf703833312893180538552d194fcc7f4fb0642fd 2398 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-server-linux-arm64.tar.gz) | 07e9304f6864f4334b9710ad1a130044c9b1349bf9e47e5c9857b688322abe5babf45a95cdbad3f2650b5447c11864edc3d50fa86de5d485e84730260efbdc8e 2399 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-server-linux-ppc64le.tar.gz) | 84f7c2b7021bd136d0c9013b349d63102433f7485552a0f19cb61f4630f256aef1f99a54f9da14b5d6c242778488c5539e0c358b9e421aafaf746ce783773e9b 2400 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-server-linux-s390x.tar.gz) | 80263d622d219ae687ea24d9b552c5e6fcb920edc6adc73fe5c742cbb34db08a045f52babb0e6c5acfab98616b9f9f2c87150db9a04f5799836b8c8fd0709f31 2401 2402 ### Node Binaries 2403 2404 filename | sha512 hash 2405 -------- | ----------- 2406 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-linux-amd64.tar.gz) | bb24b87e1971fde06882d4a2b91b2663ee9daad1e1f8f31f457eaac235b26466ec2413d947d06803b8fc9c356e56e77f7ed31b1f021eae0fdb3df426bc610717 2407 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-linux-arm64.tar.gz) | 91aa0771cf8f6615655aeda2484d967f53dbe10cee7ac724be23570d5ca60e3fe11e354cd8b715b882ba20534dab67fee505cb2cd6df1c90d124f778eadff67b 2408 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-linux-ppc64le.tar.gz) | e30e0efdd8ee02b3ee9c4da1e106a41ffdff7606401f44647e6bd03ebf4ef1900bf7c7d5a2382412a0e2b4bd7013e04e3baa08637d4dbe2b2993fcdab7e2378d 2409 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-linux-s390x.tar.gz) | e011016917d19bc5b84a76899a3eb6d7e2a6bf270e2d799ba77a9c90daabaf2055655ed0b6a62f2b9e5edfbd4b902e6e4a2408d2dc5c63a19a706d3f838a3864 2410 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-windows-amd64.tar.gz) | b9f365607de7112b2e62458462f1261905b1731d40c125db228bbc5aa48cc108872552db3c873702df0607bc0eb1996a1dca16cd27526f002bf1842e2eefc4ef 2411 2412 ### Container Images 2413 2414 All container images are available as manifest lists and support the described 2415 architectures. It is also possible to pull a specific architecture directly by 2416 adding the "-$ARCH" suffix to the container image name. 2417 2418 name | architectures 2419 ---- | ------------- 2420 [registry.k8s.io/conformance:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 2421 [registry.k8s.io/kube-apiserver:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 2422 [registry.k8s.io/kube-controller-manager:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 2423 [registry.k8s.io/kube-proxy:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 2424 [registry.k8s.io/kube-scheduler:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 2425 [registry.k8s.io/kubectl:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 2426 2427 ## Changelog since v1.28.0-alpha.4 2428 2429 ## Changes by Kind 2430 2431 ### Deprecation 2432 2433 - Changed `kubectl version` default output to be identical to what `kubectl version --short` printed, 2434 and remove `--short` flag entirely. ([#116720](https://github.com/kubernetes/kubernetes/pull/116720), [@soltysh](https://github.com/soltysh)) [SIG CLI and Testing] 2435 - Deprecated support for CSI migration of Ceph RBD volumes. 2436 2437 Users who were relying on Kubernetes' ability to migrate to an out-of-tree storage driver should complete 2438 that migration before the support for it is removed. ([#118303](https://github.com/kubernetes/kubernetes/pull/118303), [@carlory](https://github.com/carlory)) [SIG Storage] 2439 - KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. In the future, set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. ([#119007](https://github.com/kubernetes/kubernetes/pull/119007), [@aramase](https://github.com/aramase)) [SIG API Machinery and Auth] 2440 - The deprecated flag `--lock-object-namespace` and `--lock-object-name` have been removed from kube-scheduler. Please use `--leader-elect-resource-namespace` and `--leader-elect-resource-name` or ComponentConfig instead to configure those parameters. ([#119130](https://github.com/kubernetes/kubernetes/pull/119130), [@SataQiu](https://github.com/SataQiu)) [SIG Scheduling] 2441 2442 ### API Change 2443 2444 - A CDIDevice field is includes in the Device Plugin's `ContainerAllocateResponse`. This field maps to the CDIDevice field in the CRI protocol. ([#118254](https://github.com/kubernetes/kubernetes/pull/118254), [@elezar](https://github.com/elezar)) [SIG Node and Testing] 2445 - Add new annotation `batch.kubernetes.io/cronjob-scheduled-timestamp` to Job objects scheduled from CronJobs. ([#118137](https://github.com/kubernetes/kubernetes/pull/118137), [@helayoty](https://github.com/helayoty)) [SIG Apps] 2446 - Add podReplacementPolicy and terminating field to job api ([#119301](https://github.com/kubernetes/kubernetes/pull/119301), [@kannon92](https://github.com/kannon92)) [SIG API Machinery and Apps] 2447 - Added fields `reason` and `fieldPath` into CRD validation rules to allow users to specify reason and field path when validation failed. ([#118041](https://github.com/kubernetes/kubernetes/pull/118041), [@cici37](https://github.com/cici37)) [SIG API Machinery] 2448 - Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a `namespaceObject` 2449 variable with expressions. ([#118267](https://github.com/kubernetes/kubernetes/pull/118267), [@cici37](https://github.com/cici37)) [SIG API Machinery and Testing] 2450 - Adds new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. ([#118990](https://github.com/kubernetes/kubernetes/pull/118990), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 2451 - Adds new namespaceParamRef to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy ([#119215](https://github.com/kubernetes/kubernetes/pull/119215), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing] 2452 - Extend the Job API for alpha version of BackoffLimitPerIndex ([#119294](https://github.com/kubernetes/kubernetes/pull/119294), [@mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps] 2453 - Graduate `AdmissionWebhookMatchCondition` feature to beta ([#119380](https://github.com/kubernetes/kubernetes/pull/119380), [@a-hilaly](https://github.com/a-hilaly)) [SIG API Machinery] 2454 - In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . ([#118782](https://github.com/kubernetes/kubernetes/pull/118782), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 2455 - Indexed Job pods now have the pod completion index set as a pod label. ([#118883](https://github.com/kubernetes/kubernetes/pull/118883), [@danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps] 2456 - Kube-proxy: add '--logging-format' flag to support structured logging ([#117800](https://github.com/kubernetes/kubernetes/pull/117800), [@cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Architecture, Instrumentation and Network] 2457 - Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to `BETA` stability. ([#119264](https://github.com/kubernetes/kubernetes/pull/119264), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation] 2458 - Removed `resizeStatus` enum from `pvc.Status` and replaced with `AllocatedResourceStatus` ([#116335](https://github.com/kubernetes/kubernetes/pull/116335), [@gnufied](https://github.com/gnufied)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing] 2459 - StatefulSet pods now have the pod index set as a pod label `statefulset.kubernetes.io/pod-index`. ([#119232](https://github.com/kubernetes/kubernetes/pull/119232), [@danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps] 2460 - Support BackoffLimitPerIndex in Jobs ([#118009](https://github.com/kubernetes/kubernetes/pull/118009), [@mimowo](https://github.com/mimowo)) [SIG API Machinery, Apps and Testing] 2461 - Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver ([#117740](https://github.com/kubernetes/kubernetes/pull/117740), [@Richabanker](https://github.com/Richabanker)) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing] 2462 - The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer 2463 creates the KUBE-MARK-DROP chain (which has been unused for several releases) 2464 or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). ([#119374](https://github.com/kubernetes/kubernetes/pull/119374), [@danwinship](https://github.com/danwinship)) [SIG API Machinery, Network and Node] 2465 - The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still `<pod>-<claim name>`, but a random suffix will avoid name collisions. ([#117351](https://github.com/kubernetes/kubernetes/pull/117351), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing] 2466 - The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. ([#116429](https://github.com/kubernetes/kubernetes/pull/116429), [@gjkim42](https://github.com/gjkim42)) [SIG API Machinery, Apps, Node, Scheduling and Testing] 2467 2468 ### Feature 2469 2470 - A ValidatingAdmissionPolicy now has its `messageExpression` field checked against resolved types. ([#119209](https://github.com/kubernetes/kubernetes/pull/119209), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery] 2471 - Add ConsistentListFromCache feature gate that allows apiserver to serve consistent lists from cache ([#118508](https://github.com/kubernetes/kubernetes/pull/118508), [@serathius](https://github.com/serathius)) [SIG API Machinery, Instrumentation and Testing] 2472 - Add full cgroup v2 swap support for both Limited and Unlimited swap. 2473 2474 When LimitedSwap is enabled the swap limit would be automatically calculated for 2475 Burstable QoS pods. For Best-Effort / Guaranteed QoS pods, swap would be disabled. 2476 2477 Containers with memory requests equal to their memory limits also won't have 2478 swap access, and it is a way to opt-out of swap for a single container. 2479 2480 The formula for the swap limit for Burstable QoS pods is: 2481 `(<memory-request>/<node-memory-capacity>)*<node-swap-capacity>`. 2482 2483 Support for cgroup v1 is removed. ([#118764](https://github.com/kubernetes/kubernetes/pull/118764), [@iholder101](https://github.com/iholder101)) [SIG Node and Testing] 2484 - Add handling for pods in podgc for PodReplacementPolicy or PodDisruption ([#118772](https://github.com/kubernetes/kubernetes/pull/118772), [@kannon92](https://github.com/kannon92)) [SIG Apps and Testing] 2485 - Add reason to metric `attachdetach_controller_forced_detaches` in the attach detach controller. ([#119185](https://github.com/kubernetes/kubernetes/pull/119185), [@xing-yang](https://github.com/xing-yang)) [SIG Apps and Storage] 2486 - Add swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). ([#118865](https://github.com/kubernetes/kubernetes/pull/118865), [@iholder101](https://github.com/iholder101)) [SIG Node and Testing] 2487 - Added a new command line argument `--interactive` to kubectl. The new command line argument lets a user confirm deletion requests per resource interactively. ([#114530](https://github.com/kubernetes/kubernetes/pull/114530), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI and Testing] 2488 - Added a new feature gate, `SchedulerQueueingHints` (enabled by default). 2489 The new feature gate activates a framework for fine-grained filtering of events related to scheduler plugins. 2490 In this release, no default scheduling plugins make use of the hinting framework, so you should not expect any behavior changes. ([#119328](https://github.com/kubernetes/kubernetes/pull/119328), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 2491 - Adds apiserver_admission_match_condition_evaluation_seconds and apiserver_admission_match_condition_exclusions_total metrics ([#119311](https://github.com/kubernetes/kubernetes/pull/119311), [@ivelichkovich](https://github.com/ivelichkovich)) [SIG API Machinery] 2492 - Bump distroless-iptables to 0.2.6 based on Go 1.20.6 ([#119365](https://github.com/kubernetes/kubernetes/pull/119365), [@xmudrii](https://github.com/xmudrii)) [SIG Testing] 2493 - CEL authorizer checks no longer raise runtime errors. Calls to "check" will always return a decision object and the authorization error (if any) can be accessed within expressions using the new decision methods "errored" and "error". ([#118804](https://github.com/kubernetes/kubernetes/pull/118804), [@benluddy](https://github.com/benluddy)) [SIG API Machinery] 2494 - CRI: expose commit memory bytes in container stats specific to Windows ([#119238](https://github.com/kubernetes/kubernetes/pull/119238), [@kiashok](https://github.com/kiashok)) [SIG Node and Windows] 2495 - Cloud controller manager's node controller now emits timing metrics for initial `Node` synchronization. These metrics measure the delay between the creation of a new `Node` and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's `--concurrent-node-syncs` flag. ([#119241](https://github.com/kubernetes/kubernetes/pull/119241), [@cartermckinnon](https://github.com/cartermckinnon)) [SIG Cloud Provider and Instrumentation] 2496 - Faster scheduling when ResourceClaims are involved ([#119078](https://github.com/kubernetes/kubernetes/pull/119078), [@pohly](https://github.com/pohly)) [SIG Node and Scheduling] 2497 - Graduate the `ProbeTerminationGracePeriod` feature gate to GA ([#114307](https://github.com/kubernetes/kubernetes/pull/114307), [@rphillips](https://github.com/rphillips)) [SIG Apps and Node] 2498 - Hashing of KeyID in Logs 2499 2500 This release adds a feature to hash the `KeyID` values in the logs. The `KeyID` values are sensitive information that should not be exposed in plain text in the logs. By hashing the `KeyID` values, we can protect the confidentiality of the data while still being able to log the necessary information. ([#118988](https://github.com/kubernetes/kubernetes/pull/118988), [@nilekhc](https://github.com/nilekhc)) [SIG API Machinery, Auth and Testing] 2501 - Implement alpha support for a drop-in kubelet configuration directory ([#119390](https://github.com/kubernetes/kubernetes/pull/119390), [@sohankunkerkar](https://github.com/sohankunkerkar)) [SIG Node] 2502 - In the course of admitting a single request, the ValidatingAdmissionPolicy plugin will perform no more than one authorization check per unique authorizer expression. All evaluations of identical authorizer expressions will produce the same decision. ([#116443](https://github.com/kubernetes/kubernetes/pull/116443), [@benluddy](https://github.com/benluddy)) [SIG API Machinery and Testing] 2503 - Kube-controller-manager: the dynamic resource controller steps in when a pod got created such that the scheduler ignores it (i.e. spec.nodeName is set) and then takes care of triggering delayed resource claim allocation and/or reserving a claim for the pod. ([#118209](https://github.com/kubernetes/kubernetes/pull/118209), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node and Testing] 2504 - Kube-proxy service health returns http header "X-Load-Balancing-Endpoint-Weight" with number of local endpoints. The same information is still available in response body JSON payload.LocalEndpoints. ([#118999](https://github.com/kubernetes/kubernetes/pull/118999), [@cezarygerard](https://github.com/cezarygerard)) [SIG Network] 2505 - Kubelet: plugins for dynamic resource allocation may use the v1alpha3 API instead of v1alpha2 if they want to do prepare/unprepare operations in batches. ([#119012](https://github.com/kubernetes/kubernetes/pull/119012), [@pohly](https://github.com/pohly)) [SIG Node and Testing] 2506 - Kubelet: security of dynamic resource allocation was enhanced by limiting node access to those objects that are needed on the node. ([#116254](https://github.com/kubernetes/kubernetes/pull/116254), [@pohly](https://github.com/pohly)) [SIG Auth and Testing] 2507 - Kubernetes is now built with Go 1.20.6 ([#119324](https://github.com/kubernetes/kubernetes/pull/119324), [@xmudrii](https://github.com/xmudrii)) [SIG API Machinery, Auth, Cloud Provider, Release and Testing] 2508 - Migrate `pkg/controller/endpoint` to contextual logging ([#116755](https://github.com/kubernetes/kubernetes/pull/116755), [@my-git9](https://github.com/my-git9)) [SIG Apps, Instrumentation and Network] 2509 - Migrated the `EndpointSlice` and `EndpointSliceMirroring` controllers (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#115295](https://github.com/kubernetes/kubernetes/pull/115295), [@Namanl2001](https://github.com/Namanl2001)) [SIG API Machinery, Apps, Network and Testing] 2510 - Move non-graceful node shutdown to GA. ([#118228](https://github.com/kubernetes/kubernetes/pull/118228), [@carlory](https://github.com/carlory)) [SIG Apps, Storage and Testing] 2511 - New CEL Library functions to support Kubernetes Quantities. ([#118803](https://github.com/kubernetes/kubernetes/pull/118803), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery] 2512 - New Metrics Added for Encryption Configuration Controller 2513 2514 This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include: 2515 2516 - `apiserver_encryption_config_controller_automatic_reload_failures_total`: Total number of failed automatic reloads of encryption configuration. 2517 - `apiserver_encryption_config_controller_automatic_reload_success_total`: Total number of successful automatic reloads of encryption configuration. 2518 - `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds`: Timestamp of the last successful or failed automatic reload of encryption configuration. 2519 2520 These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. ([#119008](https://github.com/kubernetes/kubernetes/pull/119008), [@nilekhc](https://github.com/nilekhc)) [SIG API Machinery, Auth and Instrumentation] 2521 - New staging repo has been created for the EndpointSlice reconciler. ([#118953](https://github.com/kubernetes/kubernetes/pull/118953), [@mskrocki](https://github.com/mskrocki)) [SIG Apps, Network and Release] 2522 - Promote the following apiserver flowcontrol metrics to Beta: 2523 2524 apiserver_flowcontrol_request_wait_duration_seconds 2525 apiserver_flowcontrol_current_executing_seats 2526 apiserver_flowcontrol_nominal_limit_seats 2527 apiserver_flowcontrol_rejected_requests_total 2528 apiserver_flowcontrol_dispatched_requests_total 2529 apiserver_flowcontrol_current_inqueue_requests 2530 apiserver_flowcontrol_current_executing_requests ([#119110](https://github.com/kubernetes/kubernetes/pull/119110), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery and Instrumentation] 2531 - Replace `apiserver_storage_db_total_size_in_bytes` with `apiserver_storage_size_bytes` metric ([#118812](https://github.com/kubernetes/kubernetes/pull/118812), [@serathius](https://github.com/serathius)) [SIG API Machinery, Instrumentation and Testing] 2532 - The apiserver debug endpoint `/debug/api_priority_and_fairness/dump_requests` has been extended to dump executing requests as well as queued ones. A column for StartTime has been added to the returned table, with the queued requests having a StartTime of "0001-01-01T00:00:00Z". The executing requests have a RequestIndexInQueue of -1, and the QueueIndex is also -1 for priority levels without queues. ([#119009](https://github.com/kubernetes/kubernetes/pull/119009), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 2533 - The scheduler skips the PodTopologySpread Score plugin when nothing to do with the Pod. 2534 It will affect some metrics values related to the PodTopologySpread Score plugin. ([#118608](https://github.com/kubernetes/kubernetes/pull/118608), [@utam0k](https://github.com/utam0k)) [SIG Scheduling] 2535 - TopologyManagerPolicyOptions feature-flag is promoted to beta and enabled by default. ([#118816](https://github.com/kubernetes/kubernetes/pull/118816), [@PiotrProkop](https://github.com/PiotrProkop)) [SIG Node] 2536 - Update kube-apiserver's priority & fairness work estimator such that 'max seats' is MIN(0.15 x nominalCL, nomincalCL / handSize) ([#118601](https://github.com/kubernetes/kubernetes/pull/118601), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery] 2537 - ValidatingAdmissionPolicy type checking now correctly handles `authorizer` variable. ([#118540](https://github.com/kubernetes/kubernetes/pull/118540), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery] 2538 - With the KubeletCgroupDriverFromCRI feature gate enabled and sufficiently new version of a container 2539 runtime, kubelet automatically detects the cgroup driver config from the container runtime, eliminating 2540 the need to specify the `cgroupDriver` configuration option (or --cgroup-driver` flag) of kubelet. ([#118770](https://github.com/kubernetes/kubernetes/pull/118770), [@marquiz](https://github.com/marquiz)) [SIG Node] 2541 - [Kube-proxy]: implement connection draining for terminating nodes, KEP-3836 ([#116470](https://github.com/kubernetes/kubernetes/pull/116470), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Network] 2542 - `force_delete_pods_total ` and `force_delete_pod_errors_total ` metrics count all pod deletion behaviors. ([#118480](https://github.com/kubernetes/kubernetes/pull/118480), [@carlory](https://github.com/carlory)) [SIG Apps] 2543 2544 ### Failing Test 2545 2546 - Switched back to debian-base instead of distroless for conformance image. ([#119422](https://github.com/kubernetes/kubernetes/pull/119422), [@saschagrunert](https://github.com/saschagrunert)) [SIG Architecture, Release and Testing] 2547 2548 ### Bug or Regression 2549 2550 - Add warning for dup ports update/patching in pod's container ports and service ports ([#113245](https://github.com/kubernetes/kubernetes/pull/113245), [@pacoxu](https://github.com/pacoxu)) [SIG Network] 2551 - Bump cadvisor version to v0.47.3 ([#119225](https://github.com/kubernetes/kubernetes/pull/119225), [@iholder101](https://github.com/iholder101)) [SIG Node and Testing] 2552 - Dynamic Resource Allocation: log a error and submit an event when Kubelet fails to prepare dynamic resources ([#118578](https://github.com/kubernetes/kubernetes/pull/118578), [@bart0sh](https://github.com/bart0sh)) [SIG Node] 2553 - Fix computing backoff delay when using Job pod failure policy, by including in the backoff delay calculation pod failures ignored from the backoffLimit counter ([#119434](https://github.com/kubernetes/kubernetes/pull/119434), [@mimowo](https://github.com/mimowo)) [SIG Apps] 2554 - Fix discoverability of apiregistration.k8s.io in openapi/v3 ([#118879](https://github.com/kubernetes/kubernetes/pull/118879), [@atiratree](https://github.com/atiratree)) [SIG API Machinery] 2555 - Fixed a bug where `kubectl port-forward`, when used with a Deployment, could connect to a terminating pod even when a running pod is also available. ([#119256](https://github.com/kubernetes/kubernetes/pull/119256), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] 2556 - Fixed kubelet startup getting stuck with `NewVolumeManagerReconstruction` feature enabled and a CSI volume present in /var/lib/kubelet/pods. ([#117804](https://github.com/kubernetes/kubernetes/pull/117804), [@jsafrane](https://github.com/jsafrane)) [SIG Node and Storage] 2557 - Kubeadm: the limitation that the 'ignorePreflightErrors' field can not be set to 'all' in kubeadm config file has been removed ([#119351](https://github.com/kubernetes/kubernetes/pull/119351), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2558 - Only declare Job as finished after removing all Pod finalizers to avoid orphan Pods ([#119159](https://github.com/kubernetes/kubernetes/pull/119159), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and Testing] 2559 - Reduces CPU and memory consumption of kube-apiserver if OpenAPI V2 is not accessed by any client. Also improves performance of the apiserver on installation of many CRDs. ([#118212](https://github.com/kubernetes/kubernetes/pull/118212), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 2560 - The kube-proxy `sync_proxy_rules_iptables_total` metric has now reverted back 2561 to its pre-1.27 behavior of tracking the total number of iptables rules that 2562 kube-proxy is responsible for, rather than only counting the number of rules 2563 that it re-synced on the last sync. The new `sync_proxy_rules_iptables_last` 2564 metric now gives the latter number. ([#119140](https://github.com/kubernetes/kubernetes/pull/119140), [@danwinship](https://github.com/danwinship)) [SIG Network] 2565 - The metric `apiserver_flowcontrol_request_concurrency_limit` has been deprecated and will be removed in a future release. It is a duplicate of `apiserver_flowcontrol_nominal_limit_seats` (introduced in release 1.26) but has an outdated name and had an outdated HELP string. ([#118959](https://github.com/kubernetes/kubernetes/pull/118959), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 2566 - [Dual-stack] Fix generateAPIPodStatus() of kubelet handling Secondary IP. hostIPs order may not be be consistent. If secondary IP is before primary one, current logic adds primary IP twice into PodIPs, which leads to error: "may specify no more than one IP for each IP family". ([#116879](https://github.com/kubernetes/kubernetes/pull/116879), [@lzhecheng](https://github.com/lzhecheng)) [SIG Node] 2567 2568 ### Other (Cleanup or Flake) 2569 2570 - Migrated the disruption controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#119147](https://github.com/kubernetes/kubernetes/pull/119147), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG API Machinery, Apps, Instrumentation and Testing] 2571 - Migrated the podgc controller and some other remaining log calls within `kube-controller-manager` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). `kube-controller-manager` is now converted completely. ([#119250](https://github.com/kubernetes/kubernetes/pull/119250), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Cloud Provider, Instrumentation, Network, Storage and Testing] 2572 - Remove KUBECTL_EXPLAIN_OPENAPIV3 which is already redundant ([#119286](https://github.com/kubernetes/kubernetes/pull/119286), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI] 2573 - Revised OpenAPI v2 fetching for CustomResourceDefinitions. CRDs are now aggregated lazily, 2574 which improves resource usage during installation of many CRDs. As a result, the first request 2575 to fetch the OpenAPI may be slower. ([#118808](https://github.com/kubernetes/kubernetes/pull/118808), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing] 2576 - Shrink the OpenAPI v2 spec by more than 50%, especially for less CPU resource consumption. ([#118204](https://github.com/kubernetes/kubernetes/pull/118204), [@sttts](https://github.com/sttts)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 2577 - The GetAllocatableResources podresources API endpoint is now GA ([#118973](https://github.com/kubernetes/kubernetes/pull/118973), [@ffromani](https://github.com/ffromani)) [SIG Node and Testing] 2578 - Updated debian-base image to bookworm-v1.0.0. ([#119095](https://github.com/kubernetes/kubernetes/pull/119095), [@saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Architecture, Release and Testing] 2579 - Updated setcap image to debian bookworm v1.0.0. ([#119247](https://github.com/kubernetes/kubernetes/pull/119247), [@saschagrunert](https://github.com/saschagrunert)) [SIG Release] 2580 2581 ## Dependencies 2582 2583 ### Added 2584 - github.com/xhit/go-str2duration/v2: [v2.1.0](https://github.com/xhit/go-str2duration/v2/tree/v2.1.0) 2585 2586 ### Changed 2587 - github.com/alecthomas/kingpin/v2: [v2.3.1 → v2.3.2](https://github.com/alecthomas/kingpin/v2/compare/v2.3.1...v2.3.2) 2588 - github.com/google/cadvisor: [v0.47.2 → v0.47.3](https://github.com/google/cadvisor/compare/v0.47.2...v0.47.3) 2589 - github.com/prometheus/client_model: [v0.3.0 → v0.4.0](https://github.com/prometheus/client_model/compare/v0.3.0...v0.4.0) 2590 - github.com/prometheus/common: [v0.42.0 → v0.44.0](https://github.com/prometheus/common/compare/v0.42.0...v0.44.0) 2591 - github.com/rogpeppe/go-internal: [v1.6.1 → v1.10.0](https://github.com/rogpeppe/go-internal/compare/v1.6.1...v1.10.0) 2592 - golang.org/x/crypto: v0.6.0 → v0.11.0 2593 - golang.org/x/net: v0.9.0 → v0.12.0 2594 - golang.org/x/oauth2: v0.6.0 → v0.8.0 2595 - golang.org/x/sys: v0.8.0 → v0.10.0 2596 - golang.org/x/term: v0.7.0 → v0.10.0 2597 - golang.org/x/text: v0.9.0 → v0.11.0 2598 - k8s.io/kube-openapi: 7562a10 → 2695361 2599 2600 ### Removed 2601 - github.com/xhit/go-str2duration: [v1.2.0](https://github.com/xhit/go-str2duration/tree/v1.2.0) 2602 2603 2604 2605 # v1.28.0-alpha.4 2606 2607 2608 ## Downloads for v1.28.0-alpha.4 2609 2610 2611 2612 ### Source Code 2613 2614 filename | sha512 hash 2615 -------- | ----------- 2616 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes.tar.gz) | 05a404e2a5a526cb4713a9cab1cfcadb03cbeb065663a8ccec9c7eaf60277e1c69bea422716fc3b805ca569effb036b2d88adc752409b4f6103f10111f620736 2617 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-src.tar.gz) | 0707c72499098c2eb8ca3cffd1baf0cb4da553ded8acbf6bc1b725461484a75ba5baf277ccbe318cdb5df0c970cb31bf8afe3df0130acdd23c35b8a2fbc8a15f 2618 2619 ### Client Binaries 2620 2621 filename | sha512 hash 2622 -------- | ----------- 2623 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-darwin-amd64.tar.gz) | 5fc07afae149003d53d8bc72f9d3bbf578efd7ec7c36fda46a436774f4471cd457317dada967fc3596d369783219bdca1974d62f47c09fb8b2d158a78d48aebe 2624 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-darwin-arm64.tar.gz) | a025567c86d8af69e34d589f36f079eeff85105e342047f5d74b1ba749b9b857d19ffadd280910fd58926dfae54eb7eb8203009ac96362877a05e3cb88c49e4b 2625 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-386.tar.gz) | b17a16d8f3ce78e92b2988e726a9c818f0f0f36b8ae22809f4db2568c1746b585888820cb3cb276d00b76781e75bb10d1e9a19887d438ffee58c435775f114f0 2626 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-amd64.tar.gz) | 8133aaf1c3a964d32666fa7616917235ca62ed57b879915cae3acc4867db46e35b127cc302d1a3fa7fff143ade6f73c0c1667b45eb1debef052b2b69f5c407f1 2627 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-arm.tar.gz) | 84f30f2d113f9003207e547dcb4c3467e17b93b05dade5c6b8cd847bd040e7b21574c1d75d923d8f1d3906a4793ab8a78ab477cb16ccd72a98221c0edd394ce9 2628 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-arm64.tar.gz) | e7cad9b40187afa63168e40ec96128d4f2469115049cd0642e3a255d25b325c662fd99c1866dd6798a634d2de179493be9c05de11372f86f6d31329b24b8c283 2629 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-ppc64le.tar.gz) | 3597df23599a6f9da7886601c0ff7e49fc23b0817b6463aa426e7191f23955f772986494d9b8926b9b9dcb1f7f2f75054739b3d25a893f4d65e3f58d567eeb2b 2630 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-s390x.tar.gz) | a5630f732ef831722c778484742368a3768276bd4e443bcedcbd2c02b1164265e7a70fb55b6e7560558b0a3d4eaca3cbc7c7ded436c19024e6826224d73b4ef6 2631 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-windows-386.tar.gz) | 0be37194c9d1fc75877346eb1ab1f612286068732558d59f862c1901c3217b91a7e758f41aaba2308142aab4170bb4e5f4e7291fde7717062a2fb4ca91b159bf 2632 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-windows-amd64.tar.gz) | f53a4216ac8e959b40d10257bc01044b1c1e430c5da02b61b6ed5184f0acb7317d19f36946e60de0d168d45d71d404bd615a2beb9b70b6495b2db0182fde375c 2633 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-windows-arm64.tar.gz) | 0bce42d19642c6dd794ce7944274b0dd997df5984fa8a22f72f07551f00533f8f018bd209994c69615ca33d4f8a41a873b364e5c3d996b2edccf849f94621236 2634 2635 ### Server Binaries 2636 2637 filename | sha512 hash 2638 -------- | ----------- 2639 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-server-linux-amd64.tar.gz) | 593409f6e3accafbd448da6d6b775cc1f85cc4b787acb7b7580e94221a528c6b805e73d5fe16fc36a9c2838da6bad3928b18c3771ec95c5bbd0efd19d404d8e2 2640 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-server-linux-arm64.tar.gz) | 3e8f26b51f85b61ad249f54255077a8f4ffaf80c55935cfa2f490f56eea112eb2df569882a7f486cd19371e41c1f65c43aa2bffeece3e35269b67c19ff9e7ae4 2641 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-server-linux-ppc64le.tar.gz) | 3e4d8f4ebed1632878a78936e62331973fa57e8b394c79c262f98316a81f460b6bd8ee4cf9dd74d77df289ea2cf3716e58a431a5f52c610c916a7f45cee80bdd 2642 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-server-linux-s390x.tar.gz) | 6aa22b0fa568b70dd9c34408dfb70b60d09b2fd65429671e23d0becfc83aac75d082818c21737105e4e2485752fd9e5d5f1d92e8dbbc46b257d269237155a85a 2643 2644 ### Node Binaries 2645 2646 filename | sha512 hash 2647 -------- | ----------- 2648 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-linux-amd64.tar.gz) | 7f0a902031857e8ec8189dd37c52e788c36e02c5e19439ec24602822e95466b008064419961f728f3c02b661dfc23d89fcc0bca15b085f9ed3d001b4cd94adb8 2649 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-linux-arm64.tar.gz) | fa564b5fe5d69f2c31b453da9024ab505adfb62373722fe3d6d3852eb13e938d271f5a90542982ea9ea4fd2182f67a720be0b8c77f2e7353a6bbda3ff16e34a5 2650 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-linux-ppc64le.tar.gz) | 14ea08bf2ad7f177de8f59b4ac44bd7ba451a9b6493c1b6a3fbc00337e2c7865824b790e30d3a4dbb6cbce9ecbfa62a4e12fb7da04049e2bdf718d273131fff2 2651 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-linux-s390x.tar.gz) | a1119b19f271a5ddb90b71f4b3e844b3d46889340349fcbf297c2ffaae253538e303535f7b180a1107d189a7fc66fdf3f029da90db761f1ca5faf52aedfd0c64 2652 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-windows-amd64.tar.gz) | a3b03111ff946bb21722d1f140cfa483b9692381aaadc3cf7726d2e53b717b573887f3a541ab42cb3244a95f023817d2fefd5c4314d1b5fe30ecd68643709295 2653 2654 ### Container Images 2655 2656 All container images are available as manifest lists and support the described 2657 architectures. It is also possible to pull a specific architecture directly by 2658 adding the "-$ARCH" suffix to the container image name. 2659 2660 name | architectures 2661 ---- | ------------- 2662 [registry.k8s.io/conformance:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 2663 [registry.k8s.io/kube-apiserver:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 2664 [registry.k8s.io/kube-controller-manager:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 2665 [registry.k8s.io/kube-proxy:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 2666 [registry.k8s.io/kube-scheduler:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 2667 [registry.k8s.io/kubectl:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 2668 2669 ## Changelog since v1.28.0-alpha.3 2670 2671 ## Urgent Upgrade Notes 2672 2673 ### (No, really, you MUST read this before you upgrade) 2674 2675 - Action required for the custom scheduler plugin developers. 2676 Here's the breaking change in `EnqueueExtension` in the scheduling framework. 2677 The `EventsToRegister` in `EnqueueExtension` changed the return value from `ClusterEvent` to `ClusterEventWithHint`. `ClusterEventWithHint` allows each plugin to filter out more useless events via the callback function named `QueueingHintFn`. 2678 When the scheduling queue receives a cluster event, before moving each Pod from unschedulable pod pool to activeQ/backoffQ, it will call QueueingHintFn of plugins that rejected each Pod in the previous scheduling cycle. 2679 Depending on the value returned from QueueingHintFn, the scheduling queue changes how it queues each Pod: 2680 - if more than one QueueingHintFn returns QueueImmediately, it queues Pod to activeQ. 2681 - If no QueueingHintFn returns QueueImmediately and more than one plugin returns QueueAfterBackoff, it queues Pod to backoffQ if Pod is backing off, or to activeQ if Pod's backoff has already finished. 2682 - If all QueueingHintFn return QueueSkip, it puts this pod back to the unschedulable pod pool 2683 2684 Having appropriate QueueingHintFn contributes to reducing useless retries and thus improves the overall scheduler's performance. 2685 2686 **How can I migrate?** 2687 2688 For backward compatibility, nil `QueueingHintFn` is treated as always returning QueueAfterBackoff. 2689 So, if you want to just keep the existing behavior, you can register `ClusterEventWithHint` with no `QueueingHintFn` in it. 2690 But, registering appropriate `QueueingHintFn` is, of course, better from a scheduling performance perspective. ([#118551](https://github.com/kubernetes/kubernetes/pull/118551), [@sanposhiho](https://github.com/sanposhiho)) [SIG Node, Scheduling, Storage and Testing] 2691 - RBD volume plugin ( `kubernetes.io/rbd`) has been deprecated in this release and will be removed in a subsequent release. Alternative is to use RBD CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. ([#118552](https://github.com/kubernetes/kubernetes/pull/118552), [@humblec](https://github.com/humblec)) [SIG Storage] 2692 2693 ## Changes by Kind 2694 2695 ### Deprecation 2696 2697 - KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. Set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. ([#119007](https://github.com/kubernetes/kubernetes/pull/119007), [@aramase](https://github.com/aramase)) [SIG API Machinery and Auth] 2698 2699 ### API Change 2700 2701 - Add ServedVersions field to StorageVersion API ([#118386](https://github.com/kubernetes/kubernetes/pull/118386), [@Richabanker](https://github.com/Richabanker)) [SIG API Machinery and Testing] 2702 - Component-base/logs is now more strict about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. ([#117108](https://github.com/kubernetes/kubernetes/pull/117108), [@pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation, Scheduling and Testing] 2703 2704 ### Feature 2705 2706 - "plugin_evaluation_total" metric supports prescore/score extension point. 2707 The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming Pod. ([#118025](https://github.com/kubernetes/kubernetes/pull/118025), [@AxeZhan](https://github.com/AxeZhan)) [SIG Scheduling] 2708 - Add `--concurrency` flag to configure the concurrency of `kubectl diff` execution, defaults to 1 ([#118810](https://github.com/kubernetes/kubernetes/pull/118810), [@brancz](https://github.com/brancz)) [SIG CLI] 2709 - AdvancedAuditing feature gate that graduated to GA in 1.12 and was unconditionally enabled has been removed in v1.28. ([#118763](https://github.com/kubernetes/kubernetes/pull/118763), [@Shubham82](https://github.com/Shubham82)) [SIG API Machinery and Auth] 2710 - Allow to monitor client-go DNS resolver latencies via `rest_client_dns_resolution_duration_seconds` Prometheus metric ([#115357](https://github.com/kubernetes/kubernetes/pull/115357), [@mfojtik](https://github.com/mfojtik)) [SIG API Machinery, Architecture and Instrumentation] 2711 - Dynamic resource allocation: when a claim uses "wait for first consumer" allocation (the default), then it will now get deallocated after it was used by a pod. That ensures that the next pod isn't affected by previous scheduling decision and that resources are not kept allocated unless really needed. If keeping a claim allocated is desired, use "immediate allocation". ([#118936](https://github.com/kubernetes/kubernetes/pull/118936), [@pohly](https://github.com/pohly)) [SIG Apps, Node and Testing] 2712 - Kubeadm: add the --allow-experimental-api flag to "kubeadm config migrate/validate" commands. It can be used to migrate or validate WIP / experimental APIs in the future. ([#118866](https://github.com/kubernetes/kubernetes/pull/118866), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 2713 - Kubeadm: generate CA certificates with a start time that is offset 5 minutes in the past relative to the current system time to workaround cases of clock desync. 2714 client-go: allow to set NotBefore in NewSelfSignedCACert() ([#118922](https://github.com/kubernetes/kubernetes/pull/118922), [@champtar](https://github.com/champtar)) [SIG API Machinery, Auth and Cluster Lifecycle] 2715 - Migrated controller functions to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116930](https://github.com/kubernetes/kubernetes/pull/116930), [@fatsheep9146](https://github.com/fatsheep9146)) [SIG API Machinery, Apps, Network, Node, Storage and Testing] 2716 - Migrated the certificate controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#113994](https://github.com/kubernetes/kubernetes/pull/113994), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG API Machinery, Apps, Auth, Instrumentation and Testing] 2717 - Now it is possible to use pods with volumes and user namespaces. The feature gate was renamed from UserNamespacesStatelessPodsSupport to UserNamespacesSupport ([#118691](https://github.com/kubernetes/kubernetes/pull/118691), [@giuseppe](https://github.com/giuseppe)) [SIG Apps, Node and Testing] 2718 - RetroactiveDefaultStorageClass feature is stable and enabled by default. ([#118102](https://github.com/kubernetes/kubernetes/pull/118102), [@RomanBednar](https://github.com/RomanBednar)) [SIG Apps, Storage and Testing] 2719 - Schedular now waits for handlers to finish syncing before the scheduling cycles start. ([#116729](https://github.com/kubernetes/kubernetes/pull/116729), [@AxeZhan](https://github.com/AxeZhan)) [SIG Apps, Scheduling and Testing] 2720 - The "value" part in the `wait --for=jsonpath='{expression}'[=value]` is now optional. If the value is not provided i.e. the command looks like `wait --for=jsonpath='{expression}'` then the wait condition is interpreted as matched when the expression returns *any* single JSON value like object or a literal. ([#118160](https://github.com/kubernetes/kubernetes/pull/118160), [@minherz](https://github.com/minherz)) [SIG CLI and Testing] 2721 - Updated cAdvisor to v0.47.2 - Fix metrics in cri-o when a container restarts ([#118774](https://github.com/kubernetes/kubernetes/pull/118774), [@harche](https://github.com/harche)) [SIG Node] 2722 - When a pod is done or not going to run, then ResourceClaims for it can be reused by other pods or deleted. ([#118817](https://github.com/kubernetes/kubernetes/pull/118817), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node and Testing] 2723 2724 ### Bug or Regression 2725 2726 - Added a new event `FailedToRetrieveImagePullSecret` which will be generated when a pod references an `ImagePullSecret` that doesn't exist. ([#117927](https://github.com/kubernetes/kubernetes/pull/117927), [@kaisoz](https://github.com/kaisoz)) [SIG Node] 2727 - As in Kubernetes 1.26 and 1.27, resource claims do not get prepared by kubelet when no container uses them. This was changed accidentally in [v1.28.0-alpha.1](https://github.com/kubernetes/kubernetes/releases/tag/v1.28.0-alpha.1). ([#118786](https://github.com/kubernetes/kubernetes/pull/118786), [@pohly](https://github.com/pohly)) [SIG Node and Testing] 2728 - Faster StatefulSet creation when `Parallel` mode is enabled. ([#117865](https://github.com/kubernetes/kubernetes/pull/117865), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) [SIG Apps] 2729 - Fix cronjob controller handling of complex schedules, like "30 6-16/4 * * 1-5", for example ([#118724](https://github.com/kubernetes/kubernetes/pull/118724), [@soltysh](https://github.com/soltysh)) [SIG Apps] 2730 - Fix deletion of non-admissible pods that are deleted during Kubelet restart ([#118497](https://github.com/kubernetes/kubernetes/pull/118497), [@mimowo](https://github.com/mimowo)) [SIG Node and Testing] 2731 - Fix discoverability of apiregistration.k8s.io in openapi/v3 ([#118879](https://github.com/kubernetes/kubernetes/pull/118879), [@atiratree](https://github.com/atiratree)) [SIG API Machinery] 2732 - Kubectl explain should correctly work for all resources ([#118876](https://github.com/kubernetes/kubernetes/pull/118876), [@atiratree](https://github.com/atiratree)) [SIG CLI] 2733 - Kubectl expose supports the creation of different protocol service on the same port ([#114909](https://github.com/kubernetes/kubernetes/pull/114909), [@aimuz](https://github.com/aimuz)) [SIG CLI] 2734 - The Daemonset controller creates replacements for terminal Pods, which can appear during VM preemptions or when using Pod finalizers ([#118716](https://github.com/kubernetes/kubernetes/pull/118716), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps, Node and Testing] 2735 - The `pod_scheduling_duration_seconds` metrics won't consider the time when a Pod fails PreEnqueue (like being gated). ([#118049](https://github.com/kubernetes/kubernetes/pull/118049), [@helayoty](https://github.com/helayoty)) [SIG Scheduling] 2736 - Update apiserver metric request_filter_duration_seconds to include a 10s, 15s and 30s bucket. 2737 - Update apiserver metric request_wait_duration_seconds to include a 15s bucket. ([#118945](https://github.com/kubernetes/kubernetes/pull/118945), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery, Instrumentation and Testing] 2738 - Users will no longer see an error for failed events caused due to terminating namespace. ([#114849](https://github.com/kubernetes/kubernetes/pull/114849), [@padlar](https://github.com/padlar)) [SIG API Machinery] 2739 - Wait.PollUntilContextTimeout function, if immediate is true, the condition will be invoked before waiting and guarantees that the condition is invoked at least once, regardless of whether the context has been cancelled. ([#118686](https://github.com/kubernetes/kubernetes/pull/118686), [@aojea](https://github.com/aojea)) [SIG API Machinery] 2740 2741 ### Other (Cleanup or Flake) 2742 2743 - Kube-controller-manager and cloud-controller-manager have changed the name of controllers that can be turned off/on that are passed to the `--controllers` flag (eg `pod-garbage-collector-controller` ). The old names (eg `podgc`) are also accepted and aliased to the new names ([#115813](https://github.com/kubernetes/kubernetes/pull/115813), [@atiratree](https://github.com/atiratree)) [SIG API Machinery and Cloud Provider] 2744 - Kube-proxy will now warn at startup if the configuration seems inconsistent 2745 with respect to IP families. (For example, if you have an IPv4 node IP, but 2746 `--cluster-cidr` is IPv6.) ([#119003](https://github.com/kubernetes/kubernetes/pull/119003), [@danwinship](https://github.com/danwinship)) [SIG Network] 2747 - Promote `kubernetes_healthcheck` and `kubernetes_healthchecks_total` to `BETA` stability level. ([#118986](https://github.com/kubernetes/kubernetes/pull/118986), [@logicalhan](https://github.com/logicalhan)) [SIG Architecture, Instrumentation and Testing] 2748 - Reduce delay when processing jobs after a transient API error ([#118759](https://github.com/kubernetes/kubernetes/pull/118759), [@mimowo](https://github.com/mimowo)) [SIG Apps] 2749 - The NetworkPolicyLegacy test suite (deprecated in v1.21) has now officially been removed in favor of the new table driven e2e tests. ([#118915](https://github.com/kubernetes/kubernetes/pull/118915), [@astoycos](https://github.com/astoycos)) [SIG Network and Testing] 2750 - The feature gates `CSIMigrationGCE` is graduated to GA and were unconditionally enabled have been removed in v1.25, and the entire gcepd package has been removed. ([#117055](https://github.com/kubernetes/kubernetes/pull/117055), [@cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Node, Scheduling and Storage] 2751 - The metric `apiserver_flowcontrol_current_executing_seats` has been introduced as a duplicate of `apiserver_flowcontrol_request_concurrency_in_use` because the latter has a confusing name and will be removed in a later release. ([#118960](https://github.com/kubernetes/kubernetes/pull/118960), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 2752 - Use table-driven test for TestPerPodSchedulingMetrics ([#118842](https://github.com/kubernetes/kubernetes/pull/118842), [@helayoty](https://github.com/helayoty)) [SIG Scheduling] 2753 2754 ## Dependencies 2755 2756 ### Added 2757 _Nothing has changed._ 2758 2759 ### Changed 2760 - github.com/google/cadvisor: [v0.47.1 → v0.47.2](https://github.com/google/cadvisor/compare/v0.47.1...v0.47.2) 2761 2762 ### Removed 2763 _Nothing has changed._ 2764 2765 2766 2767 # v1.28.0-alpha.3 2768 2769 2770 ## Downloads for v1.28.0-alpha.3 2771 2772 2773 2774 ### Source Code 2775 2776 filename | sha512 hash 2777 -------- | ----------- 2778 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes.tar.gz) | 1479e19873837e09f08f4c14d4b7587b1bc40d0b0d3214637311b63068301d34a63663f5d13b8ec62c81095a30eef1e8589633c630fe613eb825eb4afa0ddeb9 2779 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-src.tar.gz) | 1f5d3486f15574220d3c5a5d7c7a2b7412347418650deeda326eb513ba2bb43c197e089999756eee09ac4b55dcafc18eef96c6e15e20fcc9b91183a35f224cc1 2780 2781 ### Client Binaries 2782 2783 filename | sha512 hash 2784 -------- | ----------- 2785 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-darwin-amd64.tar.gz) | 4e5139d5a5a98343ded5b506451c3b9b052803a2e2ebb4e2328e17edbbde56dde749407ef8fc816283a1a1b1f80939a76b7c64a09a9496d4448ec47fe34cd95b 2786 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-darwin-arm64.tar.gz) | c58fdf32d3f1b411e10a7ede56115020c7a4c50912a899f06cf94d2f06a5e24a21df1deda6eb0e87f70d88afec46186f64dd18bb1b26b94b24b01059aef88b2c 2787 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-386.tar.gz) | 26d3d2663de49063d02c682557dce616f5630b69a1ca219024ccd3405081193732bee42d24d9e5f5cf9fcd214da50defd64c141775a6aea372dbe3e9793e4547 2788 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-amd64.tar.gz) | f9c29ea075c4b842c39587e911b83e2c798d6d69ed5f4601ce17ce3c33bb9db8f511b3d2149dfe3d86576763e92e0b6eb6d3aa1b12bc04868fb6c8a013a2fdd1 2789 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-arm.tar.gz) | be984144ebe8c2f3156b3d13aaebd362543399f4c91f8f18b1936999327468ccb5af7d95723280000b627e2f81fb119d6e175cc305391ffd83b3b632eb3c30df 2790 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-arm64.tar.gz) | bfc44ed472351dee434984e49be2cc9959df2f98048beeb1ad85a50bfe833a5396bb3f5c7f81ced2048de269bd60f6d4cf55944c0d44d288fb858b5552354389 2791 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-ppc64le.tar.gz) | d7ee2633c172442a04d317d4d935b55aa892131f716308b78d0f7b3cb5d2bfa9069f7b55ec837e4ff9a260717313e7fc4c7ca4a931f9f2b36768a9c54593ba45 2792 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-s390x.tar.gz) | 4621ed7a77990430cbb03a14f85780f059a408a567c26c14eb7c7127300f4e1ac1b2013206f00a92e35da9f1142909745262ad43a8026a9ebe31cf423dbb3b90 2793 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-windows-386.tar.gz) | 6526de9f4ccd6f036152d73ebbf3a2b626ca7291c85d5b46977e8a8643ff80616665f4bd8b009310cc887f7d5e1827045a0b2489dd19621338e727e18cca4097 2794 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-windows-amd64.tar.gz) | 784e8d07b1db0331f454faac34b279bffba16c11f98ba25aa54f64eb6a0d9d55f0e8221828aa4e6e4d0d6bfb66a3c22a0d8a66d6287b37a33890a960d25fc54e 2795 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-windows-arm64.tar.gz) | 523d2809d4ff4616ffd94e0868ac15ee10f87bee65760741b09ba2a04626e3ce226a5fc44a1bddb32835403ee374a6c12afa74ec0d2b9441304a11d923f43cee 2796 2797 ### Server Binaries 2798 2799 filename | sha512 hash 2800 -------- | ----------- 2801 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-server-linux-amd64.tar.gz) | 5c2716137752b514fe7a8b23291cc7a661984deaf642d111e65690e928ecfbb6b5168b1f043847f3d4689cafbbdf5bb6fb5c97d2fd6f5f83c5b6cc701a4f388f 2802 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-server-linux-arm64.tar.gz) | 814e4ec6b46960e90ccce6ef1c234666f372b2b41583ead530b4163b829c4bb9001df822daf55493e54ad749076ab5391ebde7261d73f4e524f6cace402d49c3 2803 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-server-linux-ppc64le.tar.gz) | 8a7749282ba4d3df877097dc07843112e815879ee911379d02ed33d5aab59cb60ffd27127ebd94879bef45d9534470d6aa7e48d71ecb455e3055c2dbb169fcdb 2804 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-server-linux-s390x.tar.gz) | 031a55294a06e5773d3c277dcab6b1e32c5e6409448015282275cb62e5bafb6a10845f0de3d37d501067f3f4176f48d04cafdd4eb96f6f7e30c7274ce3adaec6 2805 2806 ### Node Binaries 2807 2808 filename | sha512 hash 2809 -------- | ----------- 2810 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-linux-amd64.tar.gz) | 488d426c09f92a6f30d77352af7154b9fda8758a946566b31d855a4c30f82ccdb071d40708f96e78f9bf5ca961be27fa67ecc833101818e43da19bc967c9eb28 2811 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-linux-arm64.tar.gz) | e5f6914bd6e393915ea814401c09f4b0accc3454087091e60b01ea7c87aa64edd1a0af954cd1a90a3dc83c5559772d1ba5a8ff04134638e911d972947e3e94a0 2812 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-linux-ppc64le.tar.gz) | 459453ad83c9cdd2568e43ed988ec2635b5413b435e587e8a6394feb7d49c550622c491f337c48f90ad32cc02b7ef47e24069f09737817a1c37f1a8930d6c5c2 2813 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-linux-s390x.tar.gz) | e5a6ff5bb58275720c18b6181ffb9d135dcfaf05a229f87787538775e0280543db43e7218168f0a2de8a74acca6899dccd1cce88780b086ce6a96fb3e0368870 2814 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-windows-amd64.tar.gz) | 7cc51923e34cfeb00681e7c2d26dff5c442b39d0000f11ad30ec5c843d80847a0353b3b4e6e5a228ad5cae7f33dfc0d4c38bf4b843b98b8115d28c3dc683dd9e 2815 2816 ### Container Images 2817 2818 All container images are available as manifest lists and support the described 2819 architectures. It is also possible to pull a specific architecture directly by 2820 adding the "-$ARCH" suffix to the container image name. 2821 2822 name | architectures 2823 ---- | ------------- 2824 [registry.k8s.io/conformance:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 2825 [registry.k8s.io/kube-apiserver:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 2826 [registry.k8s.io/kube-controller-manager:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 2827 [registry.k8s.io/kube-proxy:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 2828 [registry.k8s.io/kube-scheduler:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 2829 [registry.k8s.io/kubectl:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 2830 2831 ## Changelog since v1.28.0-alpha.2 2832 2833 ## Changes by Kind 2834 2835 ### Deprecation 2836 2837 - Kube-controller-manager deprecate `--volume-host-cidr-denylist` and `--volume-host-allow-local-loopback` flags. ([#118128](https://github.com/kubernetes/kubernetes/pull/118128), [@carlory](https://github.com/carlory)) [SIG API Machinery, Apps, Network, Node, Storage and Testing] 2838 - Kubelet: The `--azure-container-registry-config` flag has been deprecated and will be removed in a future release, please use `--image-credential-provider-config` and `--image-credential-provider-bin-dir` to setup acr credential provider instead. ([#118596](https://github.com/kubernetes/kubernetes/pull/118596), [@SataQiu](https://github.com/SataQiu)) [SIG Node] 2839 2840 ### API Change 2841 2842 - ACTION_REQUIRED 2843 When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. ([#118420](https://github.com/kubernetes/kubernetes/pull/118420), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps] 2844 - Expose rest.DefaultServerUrlFor function ([#118055](https://github.com/kubernetes/kubernetes/pull/118055), [@timofurrer](https://github.com/timofurrer)) [SIG API Machinery] 2845 - If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via `memory.oom.group` . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. ([#117793](https://github.com/kubernetes/kubernetes/pull/117793), [@tzneal](https://github.com/tzneal)) [SIG Apps, Node and Testing] 2846 - Update the comment about the feature-gate level for PodFailurePolicy from alpha to beta ([#118278](https://github.com/kubernetes/kubernetes/pull/118278), [@mimowo](https://github.com/mimowo)) [SIG Apps] 2847 2848 ### Feature 2849 2850 - Add '--concurrent-cron-job-syncs' flag for kube-controller-manager to set the number of workers for cron job controller ([#117550](https://github.com/kubernetes/kubernetes/pull/117550), [@borgerli](https://github.com/borgerli)) [SIG Apps] 2851 - Client-go: make generated CA certificates valid 1 hour in the past (NewSelfSignedCACert). Applies to CA certificates and other certificates generated by kubeadm. ([#118631](https://github.com/kubernetes/kubernetes/pull/118631), [@champtar](https://github.com/champtar)) [SIG Auth] 2852 - Fixes the alpha `CloudDualStackNodeIPs` feature. ([#118329](https://github.com/kubernetes/kubernetes/pull/118329), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] 2853 - Kubelet: un-deprecate --provider-id flag ([#116530](https://github.com/kubernetes/kubernetes/pull/116530), [@pacoxu](https://github.com/pacoxu)) [SIG Node] 2854 - Migrated the Job controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116910](https://github.com/kubernetes/kubernetes/pull/116910), [@fatsheep9146](https://github.com/fatsheep9146)) [SIG API Machinery, Apps and Testing] 2855 - Rename PodHasNetwork to PodReadyToStartContainers ([#117702](https://github.com/kubernetes/kubernetes/pull/117702), [@kannon92](https://github.com/kannon92)) [SIG Node and Testing] 2856 2857 ### Bug or Regression 2858 2859 - CI job `ci-kubernetes-node-arm64-ubuntu-serial` will test node e2e on arm64, `use-dockerized-build` and `target-build-arch` are required to run this job. ([#118567](https://github.com/kubernetes/kubernetes/pull/118567), [@chendave](https://github.com/chendave)) [SIG Node and Testing] 2860 - Fix Cronjob status.lastSuccessfulTime not populated by a manually triggered job ([#118530](https://github.com/kubernetes/kubernetes/pull/118530), [@carlory](https://github.com/carlory)) [SIG CLI] 2861 - Fix component status calling etcd health endpoint over http which exposed kubernetes to the risk of complete watch starvation and is inconsistent with other etcd probing done by kube-apiserver. ([#118460](https://github.com/kubernetes/kubernetes/pull/118460), [@serathius](https://github.com/serathius)) [SIG API Machinery] 2862 - Fixed the preStop hook will block the pod termination grace period ([#115835](https://github.com/kubernetes/kubernetes/pull/115835), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG Node and Testing] 2863 - Users will no longer see an error for failed events caused due to terminating namespace. ([#114849](https://github.com/kubernetes/kubernetes/pull/114849), [@padlar](https://github.com/padlar)) [SIG API Machinery] 2864 2865 ### Other (Cleanup or Flake) 2866 2867 - Kube-proxy: remove log warning about not using config file ([#118115](https://github.com/kubernetes/kubernetes/pull/118115), [@TommyStarK](https://github.com/TommyStarK)) [SIG Network] 2868 - Make Job controller batching of syncJob invocations enabled unconditionally (it was conditional on JobReadyPods feature before). 2869 2870 Also, Job controller's constants for default backoff and maximal backoff are lowered down to 1s (from 10s) and 1min (from 6min), respectively. These constants are used to determine the backoff delay for the next Job controller sync in case of a request failure. ([#118615](https://github.com/kubernetes/kubernetes/pull/118615), [@mimowo](https://github.com/mimowo)) [SIG Apps and Testing] 2871 - Migrated the interpodaffinity scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116635](https://github.com/kubernetes/kubernetes/pull/116635), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 2872 2873 ## Dependencies 2874 2875 ### Added 2876 - github.com/alecthomas/kingpin/v2: [v2.3.1](https://github.com/alecthomas/kingpin/v2/tree/v2.3.1) 2877 - github.com/xhit/go-str2duration: [v1.2.0](https://github.com/xhit/go-str2duration/tree/v1.2.0) 2878 2879 ### Changed 2880 - github.com/alecthomas/units: [f65c72e → b94a6e3](https://github.com/alecthomas/units/compare/f65c72e...b94a6e3) 2881 - github.com/go-kit/log: [v0.2.0 → v0.2.1](https://github.com/go-kit/log/compare/v0.2.0...v0.2.1) 2882 - github.com/kr/pretty: [v0.3.0 → v0.3.1](https://github.com/kr/pretty/compare/v0.3.0...v0.3.1) 2883 - github.com/matttproud/golang_protobuf_extensions: [v1.0.2 → v1.0.4](https://github.com/matttproud/golang_protobuf_extensions/compare/v1.0.2...v1.0.4) 2884 - github.com/prometheus/client_golang: [v1.14.0 → v1.16.0](https://github.com/prometheus/client_golang/compare/v1.14.0...v1.16.0) 2885 - github.com/prometheus/common: [v0.37.0 → v0.42.0](https://github.com/prometheus/common/compare/v0.37.0...v0.42.0) 2886 - github.com/prometheus/procfs: [v0.8.0 → v0.10.1](https://github.com/prometheus/procfs/compare/v0.8.0...v0.10.1) 2887 - golang.org/x/sync: v0.1.0 → v0.2.0 2888 2889 ### Removed 2890 _Nothing has changed._ 2891 2892 2893 2894 # v1.28.0-alpha.2 2895 2896 2897 ## Downloads for v1.28.0-alpha.2 2898 2899 2900 2901 ### Source Code 2902 2903 filename | sha512 hash 2904 -------- | ----------- 2905 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes.tar.gz) | 3ef803c3df0a722e4fdaa045fa7aca8c39892916b0788e0c3216747397fe4e6e8fc02483c7e1a7280ed9f2716324c067cd806037dbf9b635e268f8b62db43841 2906 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-src.tar.gz) | c7594dfdefc92fabd6ffeef08fb98e6686c6bbf8dc952197e6629838e645c83d4b7d375e078dc3f61f6cdbd425ca68eb88f0ffdbb1e3d2e034af9efeb8f7f34b 2907 2908 ### Client Binaries 2909 2910 filename | sha512 hash 2911 -------- | ----------- 2912 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-darwin-amd64.tar.gz) | 66e80a10b94d81ca08bd3eb94168afa429c3bc2b036e80bd55e4d0c93b4eb9e645d575ff520e6961368d2559e9bec786fbf115d0d902af7a3b0eae43213fb9e2 2913 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-darwin-arm64.tar.gz) | 0654d9ec7234a042d08db30cf01b69f66889451219a57a9e8a1dda2e11e7767e0ea1a171c2f16957bc6fedf1777bea455a03f453b9b06046d1588824ddb72627 2914 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-386.tar.gz) | cdb3fdaacad12e8910f7b3dc26cd4bb684fcfbf8e9d0e594a1dda3da8ca804a9f69d0d030893b15cdd4c0daf613b3efdf8c2be54aaa8406764aaa17550df2fb8 2915 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-amd64.tar.gz) | ceb64baa175b6444f719311f76b5de7cc1cfcad13650d3a3dc50fa5fbb6f5c724aa7fc63aa4df27fc1e3b58c419a3d1b3aaa75fb415c3709bd4f4c7253b4d99a 2916 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-arm.tar.gz) | c9a9f307fe69ce45a1f31ff8f68a158c08de202676952fc243cba4bca4f66f020fc68c7f36191d731ac536f9ff96a0e8c75a7ecc2884eac59468b92af7f72c5c 2917 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-arm64.tar.gz) | b5ee31bfb316559a9bec2bf7c5c2e555d1cfc07a307e1bfd2f4022b3a9988f2b6ae2e1189079a81dac0d3cc46d8982e25eafb2aaad5500be1c872313427e3544 2918 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-ppc64le.tar.gz) | d91e713944e21cd7729933fcb13118a3aae1a59e9809f7c8c23c047a55ceb9428459758b27d1496be3909d5097cdf5bcdc9089f6dbbfca6154a763ede0fc38d1 2919 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-s390x.tar.gz) | b34d9111b967d295092a57c9838f97400f526c9ad8086c562e090b3e92d171159dcece2576dfd0345c59c3adab06f5b01ff51a840e5997764b5bdc15df44c00f 2920 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-windows-386.tar.gz) | 82dc590e72c866e561617aed045b4377b90e4eb42574e998c71fe287202acf37ff0a3f1d55545d3465b2410108e0b4a4ab921a0fd749a99efc36723fe945c1f6 2921 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-windows-amd64.tar.gz) | a7aba1527252e942bb3a88df21a0c7a6c9f9fdbfbb1e5b2e6e28c431a413fbb51d1e50682f05a353fc6f997ed7b5e7b229f08aea0d12b92631c1b8152b029fce 2922 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-windows-arm64.tar.gz) | 728574089c01b253a1e81da19680ffaf58a88780653583d1af35ad68ee50c05cbbc21a7ca1b33c293d6db0eaa1de04988b8540467f3b4dfa22e48d5ca7a93604 2923 2924 ### Server Binaries 2925 2926 filename | sha512 hash 2927 -------- | ----------- 2928 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-server-linux-amd64.tar.gz) | fe373323bd6940fc1801d04f3fd1f574a85d103ce6526f58bc894cf0bbbbe387a26f4a4f248249aec36f88bc15c1eceb67b85af9dc876b139bad245b0551f219 2929 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-server-linux-arm64.tar.gz) | 4e76b003da0dedc9b457cb47fbcb4e8f719144aced497b6ebaaf9c515c8125a72e72c3a059757de3cdfc177dd4e3b18368ba47fd2fefb9cb2e4bc6a23c73f802 2930 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-server-linux-ppc64le.tar.gz) | c17f57f13d1393d4cf32a4ac0c122b6ebe400d88a26b3bd2eae9c6b4b4ed9f26a23ab99032847fb758bc4cbfa09a4dc7b843603d81659d0b79d42953efeca15b 2931 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-server-linux-s390x.tar.gz) | 36ce4946d3879c1e4b5e6e928be8da0eb9f146c96633d1bcb7d6541fa158f5a176d8174fba113d17bacff0db2debcedad73aed2d4054e3cc65bbf29256e62942 2932 2933 ### Node Binaries 2934 2935 filename | sha512 hash 2936 -------- | ----------- 2937 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-linux-amd64.tar.gz) | 34274b82a809fea6bdbfc1602dc66929efa71da0b3e510880a7b498253aa379a5ddc1fb1875666d2e6b5bbdd6473a7471a6b80e46526d57358ae9f97ec0e1904 2938 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-linux-arm64.tar.gz) | 610dc57e34222170c57b8c31ae15392cb960c2bff750f57acf0925d1597b54326c2f78b610c0f79f9762ba67a3af077f985b0ed8ef275e0d81794775b6448d58 2939 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-linux-ppc64le.tar.gz) | 2b65b45735680dab34e462f45c34c64c809bd8e80b2a45297446b1489dc974a54f8cd11364192e090e98edd987b4d7fb81b81d439987b1131d64a943d94ebb8f 2940 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-linux-s390x.tar.gz) | b3cba42fa9532bb9c7d80e31b1be8cb904bad1d32d6f3c651a690be1435d7f8d610bb203c89c420e96997a11a51b3e46f05fe626d2818fcc1bd3cef7d5a15b92 2941 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-windows-amd64.tar.gz) | 0b00eac88f2220dbb7f6243f9d4de41789bf3f38c14eec1c44bd48b9a067a43ef4cc9c468dc5869598d9f5ba06828b8bc5b7ea5eaa42faabaf584a8bdcb9ea29 2942 2943 ### Container Images 2944 2945 All container images are available as manifest lists and support the described 2946 architectures. It is also possible to pull a specific architecture directly by 2947 adding the "-$ARCH" suffix to the container image name. 2948 2949 name | architectures 2950 ---- | ------------- 2951 [registry.k8s.io/conformance:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 2952 [registry.k8s.io/kube-apiserver:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 2953 [registry.k8s.io/kube-controller-manager:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 2954 [registry.k8s.io/kube-proxy:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 2955 [registry.k8s.io/kube-scheduler:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 2956 [registry.k8s.io/kubectl:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 2957 2958 ## Changelog since v1.28.0-alpha.1 2959 2960 ## Urgent Upgrade Notes 2961 2962 ### (No, really, you MUST read this before you upgrade) 2963 2964 - CephFS volume plugin ( `kubernetes.io/cephfs`) has been deprecated in this release and will be removed in a subsequent release. Alternative is to use CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. ([#118143](https://github.com/kubernetes/kubernetes/pull/118143), [@humblec](https://github.com/humblec)) [SIG Storage] 2965 2966 ## Changes by Kind 2967 2968 ### Feature 2969 2970 - Introduce support for CEL optionals (see [CEL spec proposal 246](https://github.com/google/cel-spec/wiki/proposal-246)). 2971 This feature will not be fully enabled until a future Kubernetes release (likely to be v1.29), but is added in v1.28 to enable 2972 safe rollback on downgrade. ([#118339](https://github.com/kubernetes/kubernetes/pull/118339), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth, Cloud Provider and Testing] 2973 - Kubernetes is now built with Go 1.20.5 ([#118507](https://github.com/kubernetes/kubernetes/pull/118507), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Testing] 2974 - Promote ServiceNodePortStaticSubrange to beta and it will be enabled by default ([#117877](https://github.com/kubernetes/kubernetes/pull/117877), [@xuzhenglun](https://github.com/xuzhenglun)) [SIG Network] 2975 - The `ExpandedDNSConfig` feature has graduated to GA. 'ExpandedDNSConfig' feature was locked to default value and will be removed in v1.30. If you were setting this feature gate explicitly, please remove it now. ([#116741](https://github.com/kubernetes/kubernetes/pull/116741), [@gjkim42](https://github.com/gjkim42)) [SIG Apps, Network and Node] 2976 - The helping message of commands which have sub-commands is now clearer and more instructive. It will show the full command instead of 'kubectl <command> --help ...' 2977 2978 Changed 'kubectl create secret --help' description. There will be a short introduction to the three secret types and clearer guidance on how to use the command. ([#117930](https://github.com/kubernetes/kubernetes/pull/117930), [@LronDC](https://github.com/LronDC)) [SIG CLI and Testing] 2979 - Updated distroless I-tables to use registry.k8s.io/build-image/distroless-iptables:v0.2.5 ([#118541](https://github.com/kubernetes/kubernetes/pull/118541), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG Testing] 2980 2981 ### Bug or Regression 2982 2983 - Compute the backoff delay more accurately for deleted pods ([#118413](https://github.com/kubernetes/kubernetes/pull/118413), [@mimowo](https://github.com/mimowo)) [SIG Apps] 2984 - Ensure Job status updates are batched by 1s. This fixes an unlikely scenario when a sequence of immediately 2985 completing pods could trigger a sequence of non-batched Job status updates. ([#118470](https://github.com/kubernetes/kubernetes/pull/118470), [@mimowo](https://github.com/mimowo)) [SIG Apps] 2986 - Fix a race condition in kube-proxy when using LocalModeNodeCIDR to avoid dropping Services traffic if the object node is recreated when kube-proxy is starting ([#118499](https://github.com/kubernetes/kubernetes/pull/118499), [@aojea](https://github.com/aojea)) [SIG Network] 2987 - Fixed a race condition between `Run()` and `SetTransform()` and `SetWatchErrorHandler()` in shared informers. ([#117870](https://github.com/kubernetes/kubernetes/pull/117870), [@howardjohn](https://github.com/howardjohn)) [SIG API Machinery] 2988 - Fixes bug where explain was not properly respecting jsonpaths ([#115694](https://github.com/kubernetes/kubernetes/pull/115694), [@mpuckett159](https://github.com/mpuckett159)) [SIG CLI] 2989 - Kubelet: print sorted volumes message in events ([#117079](https://github.com/kubernetes/kubernetes/pull/117079), [@qingwave](https://github.com/qingwave)) [SIG Node] 2990 2991 ### Other (Cleanup or Flake) 2992 2993 - E2e framework: the `node-role.kubernetes.io/master` taint has been removed from the default value of `--non-blocking-taints` flag. You may need to set `--non-blocking-taints` explicitly if the cluster to be tested has nodes with the deprecated `node-role.kubernetes.io/master` taint. ([#118510](https://github.com/kubernetes/kubernetes/pull/118510), [@SataQiu](https://github.com/SataQiu)) [SIG Testing] 2994 - Kube-apiserver adds two new alpha metrics `conversion_webhook_request_total` and `conversion_webhook_duration_seconds` that allow users to monitor requests to CRD conversion webhooks, split by result, and failure_type (In case of failure). ([#118292](https://github.com/kubernetes/kubernetes/pull/118292), [@cchapla](https://github.com/cchapla)) [SIG API Machinery, Architecture and Instrumentation] 2995 - Moved `k8s.io/kubernetes/pkg/kubelet/cri/streaming` package to `k8s.io/kubelet/pkg/cri/streaming`. ([#118253](https://github.com/kubernetes/kubernetes/pull/118253), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node, Release and Security] 2996 - OpenAPI proto deserializations should use gnostic-models instead of the gnostic library ([#118384](https://github.com/kubernetes/kubernetes/pull/118384), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node, Storage and Testing] 2997 - [KCCM] drop filtering nodes for the providerID when syncing load balancers, but have changes to the field trigger a re-sync of load balancers. This should ensure that cloud providers which don't specify providerID, can still use the service controller implementation to provision load balancers. ([#117602](https://github.com/kubernetes/kubernetes/pull/117602), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network] 2998 2999 ## Dependencies 3000 3001 ### Added 3002 - github.com/antlr/antlr4/runtime/Go/antlr/v4: [8188dc5](https://github.com/antlr/antlr4/runtime/Go/antlr/v4/tree/8188dc5) 3003 - github.com/google/gnostic-models: [v0.6.8](https://github.com/google/gnostic-models/tree/v0.6.8) 3004 3005 ### Changed 3006 - github.com/dustin/go-humanize: [v1.0.0 → v1.0.1](https://github.com/dustin/go-humanize/compare/v1.0.0...v1.0.1) 3007 - github.com/evanphx/json-patch: [v4.12.0+incompatible → v5.6.0+incompatible](https://github.com/evanphx/json-patch/compare/v4.12.0...v5.6.0) 3008 - github.com/go-openapi/jsonreference: [v0.20.1 → v0.20.2](https://github.com/go-openapi/jsonreference/compare/v0.20.1...v0.20.2) 3009 - github.com/google/cel-go: [v0.12.6 → v0.16.0](https://github.com/google/cel-go/compare/v0.12.6...v0.16.0) 3010 - github.com/mitchellh/mapstructure: [v1.4.1 → v1.1.2](https://github.com/mitchellh/mapstructure/compare/v1.4.1...v1.1.2) 3011 - go.starlark.net: 8dd3e2e → a134d8f 3012 - golang.org/x/exp: 6cc2880 → a9213ee 3013 - golang.org/x/sys: v0.7.0 → v0.8.0 3014 - k8s.io/kube-openapi: 7828149 → 7562a10 3015 - sigs.k8s.io/kustomize/api: v0.13.2 → 6ce0bf3 3016 - sigs.k8s.io/kustomize/cmd/config: v0.11.1 → v0.11.2 3017 - sigs.k8s.io/kustomize/kustomize/v5: v5.0.1 → 6ce0bf3 3018 - sigs.k8s.io/kustomize/kyaml: v0.14.1 → 6ce0bf3 3019 3020 ### Removed 3021 - github.com/antlr/antlr4/runtime/Go/antlr: [v1.4.10](https://github.com/antlr/antlr4/runtime/Go/antlr/tree/v1.4.10) 3022 - github.com/docopt/docopt-go: [ee0de3b](https://github.com/docopt/docopt-go/tree/ee0de3b) 3023 - github.com/google/gnostic: [v0.5.7-v3refs](https://github.com/google/gnostic/tree/v0.5.7-v3refs) 3024 3025 3026 3027 # v1.28.0-alpha.1 3028 3029 3030 ## Downloads for v1.28.0-alpha.1 3031 3032 3033 3034 ### Source Code 3035 3036 filename | sha512 hash 3037 -------- | ----------- 3038 [kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes.tar.gz) | 65d841f778b00a04a13f3e722753704d4164f8590c2b0aca9cbb9bf85822be5343205ead8c71f9502d8b22fc84d80804fed5edc665662b0405bb0efa65fec808 3039 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-src.tar.gz) | 82fbe3f389b922cc635a896fa6c3e8cc342e4ca70003ca5491c7b3eb2e38065349e270da9c0deb0e541271978ade247ff3a420806a51d035a5a850262e41baa9 3040 3041 ### Client Binaries 3042 3043 filename | sha512 hash 3044 -------- | ----------- 3045 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-darwin-amd64.tar.gz) | c5be770467a8617221021255a22a970a72ccee3672b1973fb31c65b1de02767d014a8e9058f710f0d9b402f2b056fd17ed216cb1d6126f9738efb16f88e184c0 3046 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-darwin-arm64.tar.gz) | a194b07e23b8cee142080361394e0db7f3fb0488c16eeef3059dfb178f4cef6e124ad31c511a516058b8f82a6ab0f0194183714016ebd88e3060368528405e2c 3047 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-386.tar.gz) | ada349bd3f76b5572467a8fad504c26a223eeb50ad7677287b39db434adb5a59d2ceadd1922712f99878153f20fa8b0cd2b30a16e8e178a41c6ac747b55ee79c 3048 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-amd64.tar.gz) | c729d419e53a006996f5e583e0fa9a541ea7d2df7dc875dae729c63cd8222f10121908750c48ff34942fcbdf6456ed977bef86c4b979202fab120de0a7a42fc9 3049 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-arm.tar.gz) | 6bf4a115b4f4b7b21d193fe44f99c5b019e9f2097e831bd44958de6e63bd8068a70a9cfa535dc18dca23c0c4461195e8a62c8f1cd9faff7f5bb3c7b1b13ad604 3050 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-arm64.tar.gz) | 536101d9f50bf71e66e35781e0ca729156227405225986198276a43d2cf32aa2cbae32f0743bcb967701309ea3bd19e9e9f6150e532a2d251440f18ca8afbd16 3051 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-ppc64le.tar.gz) | a8dd8c0aaa7dce825f982edbff1ecd57671643e2725390c60b43450118abf2dd3594f306af6cbbd2df1aa146a0b21d0576c1b6e8e1dd2b50190702d1e879ad3a 3052 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-s390x.tar.gz) | b6ab35eb6c55536f91c4c0ae32b8db3462426fea11a4cce3e06581129995b42c4acdd16674e357d92280dae5ab9f50bcb6b8d5052d65c0a06b9c21fbb646e830 3053 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-windows-386.tar.gz) | 362c2f7f7327775a75b0c6cc2e3e372475d7d9291ad5f7c224632e037fe181b149d6def98dbd034d8ba73d3bac335a7788fbaa08df924e05c9ed9844fa75135f 3054 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-windows-amd64.tar.gz) | 75297a5c9f7d8f39f640d97bf4ece9a78b2226103d6b66865dcf6752375bf76b9d3e3d4b13efb291275621e7b1e4858eaa36f469ac73495bba43dfca2b900085 3055 [kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-windows-arm64.tar.gz) | e79cddab0abb31ef7f17855d9b14799fc7a66247c3aa71eed01231d40cb5caa7dad08082904fd18cc126cef1d3a7c2f42b8a8994e7ab40271eb0d8baa1a42f74 3056 3057 ### Server Binaries 3058 3059 filename | sha512 hash 3060 -------- | ----------- 3061 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-server-linux-amd64.tar.gz) | 64b5c5e1502fbe6a21ff6cde999408ff83f1d3b1088fbe05d720f90e5f0a9193b5ba1b1aaaee65e6ec1354e63e60d29c55a90535f79624f4526dea96295ad48d 3062 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-server-linux-arm64.tar.gz) | 26519f8406e2900b00a22d4e03260701ded84ddba0730f25a794f5b4bfcba452ab1c321f32fe30a7e2bf748fc93cf05fe81b2fdec7fa86af1e9f882428179f85 3063 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-server-linux-ppc64le.tar.gz) | c66df63d33607d8a3f2ae57ca80e4134b423bd8448ee3ecd72936f0c5973d027ab27f92481fc83e41b4b929cdae4be3865477e59f316dc102e19aa79e52afe6b 3064 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-server-linux-s390x.tar.gz) | 5a6c30cdf7f24b2ab906cf1a27f07bb7e5fafef100942b33320c2e8445b7934c2663ae7b7cc47f8aec173c1788ace9576144df357bef83e3d7a42e827f1a7c94 3065 3066 ### Node Binaries 3067 3068 filename | sha512 hash 3069 -------- | ----------- 3070 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-linux-amd64.tar.gz) | 95ce88f26c3809f268e8b83122dc4d0685e7b31f44dedad3b1360edd76c921e2a6e0c9077c136fea078299f4451280fbf49c9f956fc30339db752e5aa0e73367 3071 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-linux-arm64.tar.gz) | bf36de0876bab1b08e1268dd5602d5af46e99a9939e8befcb9d6fea91d04fc67438d136ae28503c3342dcff63e9849b2ca81b00c29627a9a477fcaed5e4f3443 3072 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-linux-ppc64le.tar.gz) | 649b49fe2319a9fd149d08665bdbe3c825f21bb96d4695dbb4fadad367e027f000272326217194f8319cb074ee6f15dc9b6bf4c0ff4dfcda08003680b39faebf 3073 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-linux-s390x.tar.gz) | 2b0c9466e9d42576d1bae61b2141e41521cfb0ae2c13ff3b59ea8abec124a44601c76a3e9e0a6283b6c74e9fee27d420b131238811f4dd4bdee789247b44941c 3074 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-windows-amd64.tar.gz) | a26243c3e7bab5180b5ff44139dfcecb6975326fdc6dec9b71f5dfccd89889710bcfadcde5c5a0c9ef03378396729e9b2763b38d6b67840239cb144981b98317 3075 3076 ### Container Images 3077 3078 All container images are available as manifest lists and support the described 3079 architectures. It is also possible to pull a specific architecture directly by 3080 adding the "-$ARCH" suffix to the container image name. 3081 3082 name | architectures 3083 ---- | ------------- 3084 [registry.k8s.io/conformance:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 3085 [registry.k8s.io/kube-apiserver:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 3086 [registry.k8s.io/kube-controller-manager:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 3087 [registry.k8s.io/kube-proxy:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 3088 [registry.k8s.io/kube-scheduler:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 3089 [registry.k8s.io/kubectl:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x) 3090 3091 ## Changelog since v1.27.0 3092 3093 ## Changes by Kind 3094 3095 ### Deprecation 3096 3097 - Remove tracking annotation from validation and defaulting ([#117633](https://github.com/kubernetes/kubernetes/pull/117633), [@kannon92](https://github.com/kannon92)) [SIG Apps] 3098 - Remove withdrawn feature NetworkPolicyStatus ([#115843](https://github.com/kubernetes/kubernetes/pull/115843), [@rikatz](https://github.com/rikatz)) [SIG API Machinery, Apps, Architecture, Network and Testing] 3099 3100 ### API Change 3101 3102 - Added a warning that TLS 1.3 ciphers are not configurable. ([#115399](https://github.com/kubernetes/kubernetes/pull/115399), [@3u13r](https://github.com/3u13r)) [SIG API Machinery and Node] 3103 - Added error handling for seccomp localhost configurations that do not properly set a localhostProfile ([#117020](https://github.com/kubernetes/kubernetes/pull/117020), [@cji](https://github.com/cji)) [SIG API Machinery and Node] 3104 - Added new config option `delayCacheUntilActive` to `KubeSchedulerConfiguration` that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in `kube-scheduler` ([#115754](https://github.com/kubernetes/kubernetes/pull/115754), [@linxiulei](https://github.com/linxiulei)) [SIG API Machinery and Scheduling] 3105 - Client-go: Improved memory use of reflector caches when watching large numbers of objects which do not change frequently ([#113362](https://github.com/kubernetes/kubernetes/pull/113362), [@sxllwx](https://github.com/sxllwx)) [SIG API Machinery] 3106 - Kube-controller-manager: The `LegacyServiceAccountTokenCleanUp` feature gate is now available as alpha (off by default). When enabled, the `legacy-service-account-token-cleaner` controller loop removes service account token secrets that have not been used in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year), **and are** referenced from the `.secrets` list of a ServiceAccount object, **and are not** referenced from pods. ([#115554](https://github.com/kubernetes/kubernetes/pull/115554), [@yt2985](https://github.com/yt2985)) [SIG API Machinery, Apps, Auth, Release and Testing] 3107 - Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta2 is removed in v1.28. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. ([#117649](https://github.com/kubernetes/kubernetes/pull/117649), [@SataQiu](https://github.com/SataQiu)) [SIG API Machinery, Scheduling and Testing] 3108 - NodeVolumeLimits implement the PreFilter extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. ([#115398](https://github.com/kubernetes/kubernetes/pull/115398), [@tangwz](https://github.com/tangwz)) [SIG Scheduling] 3109 - Pods which set `hostNetwork: true` and declare ports get the `hostPort` field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now `hostPort` will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. ([#117696](https://github.com/kubernetes/kubernetes/pull/117696), [@thockin](https://github.com/thockin)) [SIG Apps] 3110 - Removing WindowsHostProcessContainers feature-gate ([#117570](https://github.com/kubernetes/kubernetes/pull/117570), [@marosset](https://github.com/marosset)) [SIG API Machinery, Apps, Auth, Node and Windows] 3111 - Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta ([#117802](https://github.com/kubernetes/kubernetes/pull/117802), [@kerthcet](https://github.com/kerthcet)) [SIG API Machinery and Apps] 3112 - The `SelfSubjectReview` API is promoted to `authentication.k8s.io/v1` and the `kubectl auth whoami` command is GA. ([#117713](https://github.com/kubernetes/kubernetes/pull/117713), [@nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Architecture, Auth, CLI and Testing] 3113 3114 ### Feature 3115 3116 - Add '--concurrent-job-syncs' flag for kube-controller-manager to set the number of job controller workers ([#117138](https://github.com/kubernetes/kubernetes/pull/117138), [@tosi3k](https://github.com/tosi3k)) [SIG API Machinery and CLI] 3117 - Add DisruptionTarget condition to the pod preempted by Kubelet to make room for a critical pod ([#117586](https://github.com/kubernetes/kubernetes/pull/117586), [@mimowo](https://github.com/mimowo)) [SIG Node and Testing] 3118 - Added a container image for `kubectl` at `registry.k8s.io/kubectl` across the same architectures as other images (linux/amd64 linux/arm64 linux/s390x linux/ppc64le) ([#116672](https://github.com/kubernetes/kubernetes/pull/116672), [@dims](https://github.com/dims)) [SIG Architecture and Release] 3119 - Added support for pod `hostNetwork` field selector ([#110477](https://github.com/kubernetes/kubernetes/pull/110477), [@halfcrazy](https://github.com/halfcrazy)) [SIG Apps and Node] 3120 - Apiserver adds two new metrics `etcd_requests_total` and `etcd_request_errors_total` that allow users to monitor requests to etcd storage, split by operation and resource type. ([#117222](https://github.com/kubernetes/kubernetes/pull/117222), [@iyear](https://github.com/iyear)) [SIG API Machinery] 3121 - Bump metrics-server to v0.6.3. ([#117120](https://github.com/kubernetes/kubernetes/pull/117120), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG Cloud Provider and Instrumentation] 3122 - Client-go exposes two new metrics to monitor the client-go logic that 3123 generate http.Transports for the clients. 3124 3125 - rest_client_transport_cache_entries is a gauge metric 3126 with the number of existin entries in the internal cache 3127 3128 - rest_client_transport_create_calls_total is a counter 3129 that increments each time a new transport is created, storing 3130 the result of the operation needed to generate it: hit, miss 3131 or uncacheable ([#117295](https://github.com/kubernetes/kubernetes/pull/117295), [@aojea](https://github.com/aojea)) [SIG API Machinery, Architecture, Instrumentation, Network, Node and Testing] 3132 - External credential provider plugins now have their standard error output logged by kubelet upon failures. ([#117448](https://github.com/kubernetes/kubernetes/pull/117448), [@cartermckinnon](https://github.com/cartermckinnon)) [SIG Node] 3133 - Graduated the `LegacyServiceAccountTokenTracking` feature gate to GA. The usage of auto-generated secret-based service account token now produces warnings, and relevant Secrets are labeled with a last-used timestamp (label key `kubernetes.io/legacy-token-last-used`). ([#117591](https://github.com/kubernetes/kubernetes/pull/117591), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth and Testing] 3134 - Klog text output now uses JSON as encoding for structs, maps and slices. ([#117687](https://github.com/kubernetes/kubernetes/pull/117687), [@pohly](https://github.com/pohly)) [SIG Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 3135 - Kube-proxy handles Terminating EndpointSlices conditions and enables zero downtime deployments for Services with ExternalTrafficPolicy=Local author: @andrewsykim ([#117718](https://github.com/kubernetes/kubernetes/pull/117718), [@aojea](https://github.com/aojea)) [SIG Network, Testing and Windows] 3136 - Kube-proxy in iptables mode now has separate `sync_full_proxy_rules_duration_seconds` 3137 and `sync_partial_proxy_rules_duration_seconds` (in addition to the existing 3138 `sync_proxy_rules_duration_seconds`), to give better information about how long 3139 each sync type is taking, rather than only giving a weighted average of the two 3140 sync types together. ([#117787](https://github.com/kubernetes/kubernetes/pull/117787), [@danwinship](https://github.com/danwinship)) [SIG Network] 3141 - Kubeadm: add `--feature-gates` flag for `kubeadm upgrade node` ([#118316](https://github.com/kubernetes/kubernetes/pull/118316), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 3142 - Kubeadm: add a new "kubeadm config validate" command that can be used to validate any input config file. Use the --config flag to pass a config file to it. See the command --help screen for more information. As a result of adding this new command, enhance the validation capabilities of the existing "kubeadm config migrate" command. For both commands unknown APIs or fields will throw errors. ([#118013](https://github.com/kubernetes/kubernetes/pull/118013), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 3143 - Kubernetes is now built with Go 1.20.4 ([#117744](https://github.com/kubernetes/kubernetes/pull/117744), [@xmudrii](https://github.com/xmudrii)) [SIG Release and Testing] 3144 - Metric `scheduler_scheduler_goroutines` is removed. Use `scheduler_goroutines` instead. ([#117727](https://github.com/kubernetes/kubernetes/pull/117727), [@kerthcet](https://github.com/kerthcet)) [SIG Scheduling] 3145 - Migrated `pkg/scheduler/framework/preemption` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116835](https://github.com/kubernetes/kubernetes/pull/116835), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 3146 - Migrated `pod-security-admission` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#114471](https://github.com/kubernetes/kubernetes/pull/114471), [@Namanl2001](https://github.com/Namanl2001)) [SIG Apps and Auth] 3147 - Migrated the noderesources scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116748](https://github.com/kubernetes/kubernetes/pull/116748), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 3148 - Migrated the podtopologyspread scheduler plugins to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116797](https://github.com/kubernetes/kubernetes/pull/116797), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 3149 - Set metrics-server's metric-resolution to 15s ([#117121](https://github.com/kubernetes/kubernetes/pull/117121), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG Cloud Provider and Instrumentation] 3150 - SubjectAccessReview requests sent to webhook authorizers now default `spec.resourceAttributes.version` to `*` if unset. ([#116937](https://github.com/kubernetes/kubernetes/pull/116937), [@AxeZhan](https://github.com/AxeZhan)) [SIG Apps and Auth] 3151 - Support specifying a custom retry period for cloud load-balancer operations ([#94021](https://github.com/kubernetes/kubernetes/pull/94021), [@timoreimann](https://github.com/timoreimann)) [SIG API Machinery, Cloud Provider and Network] 3152 - The Kubernetes apiserver now emits a warning message for Pods with a null labelSelector in podAffinity or topologySpreadConstraints. The null labelSelector means "match none". Using it in podAffinity or topologySpreadConstraint could lead to unintended behavior. ([#117025](https://github.com/kubernetes/kubernetes/pull/117025), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling] 3153 - The scheduler skips the InterPodAffinity Score plugin when nothing to do with the Pod. 3154 It will affect some metrics values related to the InterPodAffinity Score plugin. ([#117794](https://github.com/kubernetes/kubernetes/pull/117794), [@utam0k](https://github.com/utam0k)) [SIG Scheduling] 3155 - The scheduler skips the PodTopologySpread Filter plugin if no spread constraints. 3156 It will affect some metrics values related to the PodTopologySpread Filter plugin. ([#117683](https://github.com/kubernetes/kubernetes/pull/117683), [@utam0k](https://github.com/utam0k)) [SIG Scheduling] 3157 - The short names vwc and mwc were introduced for the resources validatingwebhookconfigurations and mutatingwebhookconfigurations. ([#117535](https://github.com/kubernetes/kubernetes/pull/117535), [@hysyeah](https://github.com/hysyeah)) [SIG API Machinery] 3158 - Update etcd image to 3.5.9-0 ([#117999](https://github.com/kubernetes/kubernetes/pull/117999), [@kkkkun](https://github.com/kkkkun)) [SIG API Machinery] 3159 - Update the scheduler interface and cache methods to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116849](https://github.com/kubernetes/kubernetes/pull/116849), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Apps, Instrumentation, Scheduling and Testing] 3160 - Updated distroless iptables to use released image `registry.k8s.io/build-image/distroless-iptables:v0.2.4` ([#117746](https://github.com/kubernetes/kubernetes/pull/117746), [@xmudrii](https://github.com/xmudrii)) [SIG Testing] 3161 - `--version=v1.X.Y...` can now be used to set the prerelease and buildID portions of the version reported by components ([#117688](https://github.com/kubernetes/kubernetes/pull/117688), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Architecture and Release] 3162 3163 ### Documentation 3164 3165 - Enhanced clarity in error messaging when waiting for volume creation ([#118262](https://github.com/kubernetes/kubernetes/pull/118262), [@torredil](https://github.com/torredil)) [SIG Apps and Storage] 3166 3167 ### Failing Test 3168 3169 - Allow Azure Disk e2es to use newer topology labels if available from nodes ([#117216](https://github.com/kubernetes/kubernetes/pull/117216), [@gnufied](https://github.com/gnufied)) [SIG Storage and Testing] 3170 - Fix nil pointer in test AfterEach volumeperf.go for sidecar release ([#117368](https://github.com/kubernetes/kubernetes/pull/117368), [@sunnylovestiramisu](https://github.com/sunnylovestiramisu)) [SIG Storage and Testing] 3171 3172 ### Bug or Regression 3173 3174 - CVE-2023-27561 CVE-2023-25809 CVE-2023-28642: Bump fix runc v1.1.4 -> v1.1.5 ([#117095](https://github.com/kubernetes/kubernetes/pull/117095), [@PushkarJ](https://github.com/PushkarJ)) [SIG Architecture, Node and Security] 3175 - Code blocks in kubectl {$COMMAND}--help will move right by 3 indentation. ([#118029](https://github.com/kubernetes/kubernetes/pull/118029), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI] 3176 - During device plugin allocation, resources requested by the pod can only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated. If these conditions are not sattsfied, the pod would fail with `UnexpectedAdmissionError` error. ([#116376](https://github.com/kubernetes/kubernetes/pull/116376), [@swatisehgal](https://github.com/swatisehgal)) [SIG Node and Testing] 3177 - Fix Topology Aware Hints not working when the `topology.kubernetes.io/zone` label is added after Node creation ([#117245](https://github.com/kubernetes/kubernetes/pull/117245), [@tnqn](https://github.com/tnqn)) [SIG Apps and Network] 3178 - Fix a data race in TopologyCache when `AddHints` and `SetNodes` are called concurrently ([#117249](https://github.com/kubernetes/kubernetes/pull/117249), [@tnqn](https://github.com/tnqn)) [SIG Apps and Network] 3179 - Fix bug where `listOfStrings.join()` in CEL expressions resulted in an unexpected internal error. ([#117593](https://github.com/kubernetes/kubernetes/pull/117593), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery] 3180 - Fix incorrect calculation for ResourceQuota with PriorityClass as its scope. ([#117677](https://github.com/kubernetes/kubernetes/pull/117677), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG API Machinery] 3181 - Fix performance regression in scheduler caused by frequent metric lookup on critical code path. ([#117594](https://github.com/kubernetes/kubernetes/pull/117594), [@tosi3k](https://github.com/tosi3k)) [SIG Scheduling] 3182 - Fix restricted debug profile. ([#117543](https://github.com/kubernetes/kubernetes/pull/117543), [@mochizuki875](https://github.com/mochizuki875)) [SIG CLI and Testing] 3183 - Fix: After a Node is down and take some time to get back to up again, the mount point of the evicted Pods cannot be cleaned up successfully. (#111933) Meanwhile Kubelet will print the log `Orphaned pod "xxx" found, but error not a directory occurred when trying to remove the volumes dir` every 2 seconds. (#105536) ([#116134](https://github.com/kubernetes/kubernetes/pull/116134), [@cvvz](https://github.com/cvvz)) [SIG Node and Storage] 3184 - Fix: the volume is not detached after the pod and PVC objects are deleted ([#116138](https://github.com/kubernetes/kubernetes/pull/116138), [@cvvz](https://github.com/cvvz)) [SIG Storage] 3185 - Fixed a bug that unintentionally overrides your custom Accept headers in http (live-/readiness)-probes if the header is in lower casing ([#114606](https://github.com/kubernetes/kubernetes/pull/114606), [@tuunit](https://github.com/tuunit)) [SIG Network and Node] 3186 - Fixed a bug where pv recycler failed to scrub volume with too many files in the directory due to hitting ARG_MAX limit with rm command (#117189). ([#117283](https://github.com/kubernetes/kubernetes/pull/117283), [@defo89](https://github.com/defo89)) [SIG Cloud Provider and Storage] 3187 - Fixed a memory leak in the Kubernetes API server that occurs during APIService processing. ([#117258](https://github.com/kubernetes/kubernetes/pull/117258), [@enj](https://github.com/enj)) [SIG API Machinery] 3188 - Fixed an issue where the API server did not send impersonated UID to authentication webhooks. ([#116681](https://github.com/kubernetes/kubernetes/pull/116681), [@stlaz](https://github.com/stlaz)) [SIG API Machinery and Auth] 3189 - Fixed bug to correctly report `ErrRegistryUnavailable` on pulling container images for remote CRI runtimes. ([#117612](https://github.com/kubernetes/kubernetes/pull/117612), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node] 3190 - Fixed bug where using the $deleteFromPrimitiveList directive in a strategic merge patch of certain fields would remove the other values from the list instead of the values specified. ([#110472](https://github.com/kubernetes/kubernetes/pull/110472), [@brianpursley](https://github.com/brianpursley)) [SIG API Machinery] 3191 - Fixed issue where kubectl-convert would fail when encountering resources that could not be converted to the specified api version. New behavior is to warn the user of the failed conversions and continue to convert the remaining resources. ([#117002](https://github.com/kubernetes/kubernetes/pull/117002), [@gxwilkerson33](https://github.com/gxwilkerson33)) [SIG CLI and Testing] 3192 - Fixed issue where there was no response or error from kubectl rollout status when there were no resources of specified kind. ([#117884](https://github.com/kubernetes/kubernetes/pull/117884), [@gxwilkerson33](https://github.com/gxwilkerson33)) [SIG CLI] 3193 - Fixed vSphere cloud provider not to skip detach volumes from nodes at kube-controller-startup. ([#117243](https://github.com/kubernetes/kubernetes/pull/117243), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider] 3194 - Fixes a bug at kube-apiserver start where APIService objects for custom resources could be deleted and recreated. ([#118104](https://github.com/kubernetes/kubernetes/pull/118104), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing] 3195 - Fixes a race condition serving OpenAPI content ([#117705](https://github.com/kubernetes/kubernetes/pull/117705), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation and Node] 3196 - Fixes a regression in 1.27.0 that resulted in "missing metadata in converted object" errors when modifying objects for multi-version custom resource definitions with a conversion strategy of `None`. ([#117301](https://github.com/kubernetes/kubernetes/pull/117301), [@ncdc](https://github.com/ncdc)) [SIG API Machinery] 3197 - Fixes a regression in kubectl and client-go discovery when configured with a server URL other than the root of a server. ([#117495](https://github.com/kubernetes/kubernetes/pull/117495), [@ardaguclu](https://github.com/ardaguclu)) [SIG API Machinery] 3198 - Fixes bug that caused a resource to include patch directives when using strategic merge patch against a non-existent field ([#117568](https://github.com/kubernetes/kubernetes/pull/117568), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing] 3199 - Fixes creationTimestamp: null causing unnecessary writes to etcd ([#116865](https://github.com/kubernetes/kubernetes/pull/116865), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing] 3200 - If `kubeadm reset` finds no etcd member ID for the peer it removes during the `remove-etcd-member` phase, it continues immediately to other phases, instead of retrying the phase for up to 3 minutes before continuing. ([#117724](https://github.com/kubernetes/kubernetes/pull/117724), [@dlipovetsky](https://github.com/dlipovetsky)) [SIG Cluster Lifecycle] 3201 - Improved exponential backoff in Reflector, significantly reducing the load on Kubernetes apiserver in case of throttling of requests. ([#118132](https://github.com/kubernetes/kubernetes/pull/118132), [@marseel](https://github.com/marseel)) [SIG API Machinery and Scalability] 3202 - Known issue: fixed that the PreEnqueue plugins aren't executed for Pods proceeding to activeQ through backoffQ. ([#117194](https://github.com/kubernetes/kubernetes/pull/117194), [@sanposhiho](https://github.com/sanposhiho)) [SIG Release and Scheduling] 3203 - Kube-apiserver always removes its endpoint from kubernetes service during graceful shutdown (even if it's the only/last one) ([#116685](https://github.com/kubernetes/kubernetes/pull/116685), [@czybjtu](https://github.com/czybjtu)) [SIG API Machinery] 3204 - Kubeadm: crictl pull should use `-i` to set the image service endpoint ([#117835](https://github.com/kubernetes/kubernetes/pull/117835), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 3205 - Kubeadm: fix a bug where file copy(backup) could not be executed correctly on Windows platform during upgrade ([#117861](https://github.com/kubernetes/kubernetes/pull/117861), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 3206 - Kubeadm: fix a bug where the static pod changes detection logic is inconsistent with kubelet ([#118069](https://github.com/kubernetes/kubernetes/pull/118069), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 3207 - Kubeadm: speedup init by 0s or 20s. kubelet-start phase is now after etcd and control-plane phases, removing a race condition between kubelet looking for static pod manifests and kubeadm writing them. ([#117984](https://github.com/kubernetes/kubernetes/pull/117984), [@champtar](https://github.com/champtar)) [SIG Cluster Lifecycle] 3208 - Kubeadm: throw warnings instead of errors for deprecated feature gates ([#118270](https://github.com/kubernetes/kubernetes/pull/118270), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 3209 - Kubectl events --for will also support fully qualified names such as replicasets.apps, etc. ([#117034](https://github.com/kubernetes/kubernetes/pull/117034), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI and Testing] 3210 - Kubelet now skips pod resource checks when the request is zero. ([#116408](https://github.com/kubernetes/kubernetes/pull/116408), [@ChenLingPeng](https://github.com/ChenLingPeng)) [SIG Scheduling] 3211 - Kubelet terminates pods correctly upon restart, fixing an issue where pods may have not been fully terminated if the kubelet was restarted during pod termination. ([#117019](https://github.com/kubernetes/kubernetes/pull/117019), [@bobbypage](https://github.com/bobbypage)) [SIG Node and Testing] 3212 - Kubelet will ensure /etc/hosts file is mode 0644 regardless of umask. ([#113209](https://github.com/kubernetes/kubernetes/pull/113209), [@luozhiwenn](https://github.com/luozhiwenn)) [SIG Node] 3213 - Number of errors reported to the metric `storage_operation_duration_seconds_count` for emptyDir decreased significantly because previously one error was reported for each projected volume created. ([#117022](https://github.com/kubernetes/kubernetes/pull/117022), [@mpatlasov](https://github.com/mpatlasov)) [SIG Storage] 3214 - Pod termination will be faster when the pod has a missing volume reference. ([#117412](https://github.com/kubernetes/kubernetes/pull/117412), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node and Testing] 3215 - Recording timing traces had a race condition. Impact in practice was probably low. ([#117139](https://github.com/kubernetes/kubernetes/pull/117139), [@pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] 3216 - Removed leading zeros from the etcd member ID in kubeadm log messages. ([#117919](https://github.com/kubernetes/kubernetes/pull/117919), [@dlipovetsky](https://github.com/dlipovetsky)) [SIG Cluster Lifecycle] 3217 - Resolves a spurious "Unknown discovery response content-type" error in client-go discovery requests by tolerating extra content-type parameters in API responses ([#117571](https://github.com/kubernetes/kubernetes/pull/117571), [@seans3](https://github.com/seans3)) [SIG API Machinery] 3218 - Reverted NewVolumeManagerReconstruction and SELinuxMountReadWriteOncePod feature gates to disabled by default to resolve a regression of volume reconstruction on kubelet/node restart ([#117751](https://github.com/kubernetes/kubernetes/pull/117751), [@liggitt](https://github.com/liggitt)) [SIG Storage] 3219 - Setting a mirror pod's phase to Succeeded or Failed can prevent the corresponding static pod from restarting due mutation of a Kubelet cache. ([#116482](https://github.com/kubernetes/kubernetes/pull/116482), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node] 3220 - Show a warning when `volume.beta.kubernetes.io/storage-class` annotation is used in pv or pvc ([#117036](https://github.com/kubernetes/kubernetes/pull/117036), [@haoruan](https://github.com/haoruan)) [SIG Storage] 3221 - Static pods were taking extra time to be restarted after being updated. Static pods that are waiting to restart were not correctly counted in `kubelet_working_pods`. ([#116995](https://github.com/kubernetes/kubernetes/pull/116995), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node] 3222 - This PR adds additional validation for endpoint ip configuration while iterating through queried endpoint list. ([#116749](https://github.com/kubernetes/kubernetes/pull/116749), [@princepereira](https://github.com/princepereira)) [SIG Network and Windows] 3223 - Update etcd version to 3.5.8 ([#117335](https://github.com/kubernetes/kubernetes/pull/117335), [@kkkkun](https://github.com/kkkkun)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing] 3224 - Updated static pods are restarted 2s faster by correcting a safe but non-optimal ordering bug. ([#116690](https://github.com/kubernetes/kubernetes/pull/116690), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node] 3225 - [KCCM] service controller: change the cloud controller manager to make `providerID` a predicate when synchronizing nodes. This change allows load balancer integrations to ensure that the `providerID` is set when configuring 3226 load balancers and targets. ([#117388](https://github.com/kubernetes/kubernetes/pull/117388), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network] 3227 3228 ### Other (Cleanup or Flake) 3229 3230 - A v2-level info log will be added, which will output the details of the pod being preempted, including victim and preemptor ([#117214](https://github.com/kubernetes/kubernetes/pull/117214), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG Scheduling] 3231 - Allow container runtimes to use `ErrSignatureValidationFailed` as possible image pull failure. ([#117717](https://github.com/kubernetes/kubernetes/pull/117717), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node] 3232 - Deprecate genericclioptions.IOStreams and use genericiooptions.IOStreams ([#117102](https://github.com/kubernetes/kubernetes/pull/117102), [@ardaguclu](https://github.com/ardaguclu)) [SIG Auth, CLI and Release] 3233 - Enables the node-local kubelet podresources API endpoint on windows, alongside unix. ([#115133](https://github.com/kubernetes/kubernetes/pull/115133), [@ffromani](https://github.com/ffromani)) [SIG Cloud Provider, Node, Testing and Windows] 3234 - Fixed dra e2e image build on non-amd64 architectures ([#117912](https://github.com/kubernetes/kubernetes/pull/117912), [@bart0sh](https://github.com/bart0sh)) [SIG Node and Testing] 3235 - Kube-apiserver adds two new metrics `authorization_attempts_total` and `authorization_duration_seconds` that allow users to monitor requests to authorization webhooks, split by result. ([#117211](https://github.com/kubernetes/kubernetes/pull/117211), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery, Auth and Instrumentation] 3236 - Kubeadm: introduce a new feature gate UpgradeAddonsBeforeControlPlane to fix a kube-proxy skew policy misalignment. Its default value is `false`. Upgrade of the CoreDNS and kube-proxy addons will now trigger after all the control plane instances have been upgraded, unless the fearure gate is set to true. This feature gate will be removed in a future release. ([#117660](https://github.com/kubernetes/kubernetes/pull/117660), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 3237 - Marked the feature gate `ExperimentalHostUserNamespaceDefaulting` as deprecated. 3238 Enabling the feature gate already had no effect; the deprecation allows for removing the feature gate in a future release. ([#116723](https://github.com/kubernetes/kubernetes/pull/116723), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Node] 3239 - Migrated `pkg/scheduler/framework/runtime` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116842](https://github.com/kubernetes/kubernetes/pull/116842), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 3240 - Migrated the volumezone scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116829](https://github.com/kubernetes/kubernetes/pull/116829), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling] 3241 - Projects which use k8s.io/code-generator and invoke `generate-groups` or `generate-internal-groups.sh` have a new, simpler script (`kube_codegen.sh`) they can use. The old scripts are deprecated but remain intact. ([#117262](https://github.com/kubernetes/kubernetes/pull/117262), [@thockin](https://github.com/thockin)) [SIG API Machinery and Instrumentation] 3242 - Remove GAed feature gate DelegateFSGroupToCSIDriver ([#117655](https://github.com/kubernetes/kubernetes/pull/117655), [@carlory](https://github.com/carlory)) [SIG Storage] 3243 - Remove GAed feature gate DevicePlugins ([#117656](https://github.com/kubernetes/kubernetes/pull/117656), [@carlory](https://github.com/carlory)) [SIG Node] 3244 - Remove GAed feature gate KubeletCredentialProviders ([#116901](https://github.com/kubernetes/kubernetes/pull/116901), [@pacoxu](https://github.com/pacoxu)) [SIG Cloud Provider, Node and Testing] 3245 - Remove GAed feature gates: MixedProtocolLBService, ServiceInternalTrafficPolicy, ServiceIPStaticSubrange, and EndpointSliceTerminatingCondition ([#117237](https://github.com/kubernetes/kubernetes/pull/117237), [@yulng](https://github.com/yulng)) [SIG Network] 3246 - Removed the deprecated `azureFile` in-tree storage plugin ([#118236](https://github.com/kubernetes/kubernetes/pull/118236), [@andyzhangx](https://github.com/andyzhangx)) [SIG API Machinery, Cloud Provider, Node and Storage] 3247 - Structured logging of NamespacedName was inconsistent with klog.KObj. Now both use lower case field names and namespace is optional. ([#117238](https://github.com/kubernetes/kubernetes/pull/117238), [@pohly](https://github.com/pohly)) [SIG API Machinery, Architecture and Instrumentation] 3248 - The `generate_groups.sh` and `generate_internal_groups.sh` scripts from the k8s.io/code-generator repo are deprecated (but still work) in favor of `kube_codegen.sh` in that same repo. Projects which use the old scripts are encouraged to look at adopting the new one. ([#117897](https://github.com/kubernetes/kubernetes/pull/117897), [@thockin](https://github.com/thockin)) [SIG API Machinery] 3249 - The feature gate CSIStorageCapacity have been removed and must no longer be referenced in `--feature-gates` flags ([#118018](https://github.com/kubernetes/kubernetes/pull/118018), [@humblec](https://github.com/humblec)) [SIG Storage] 3250 - The feature gates `DisableAcceleratorUsageMetrics` and `PodSecurity` that graduated to GA and were unconditionally enabled have been removed in v1.28 ([#114068](https://github.com/kubernetes/kubernetes/pull/114068), [@cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Node, Scheduling and Storage] 3251 - The kubelet podresources endpoint is GA and always enabled ([#116525](https://github.com/kubernetes/kubernetes/pull/116525), [@ffromani](https://github.com/ffromani)) [SIG Node] 3252 - Updated Cluster Autosaler to version 1.26.1 ([#116526](https://github.com/kubernetes/kubernetes/pull/116526), [@pacoxu](https://github.com/pacoxu)) [SIG Autoscaling and Cloud Provider] 3253 - Updated cri-tools to v1.26.1. ([#116649](https://github.com/kubernetes/kubernetes/pull/116649), [@saschagrunert](https://github.com/saschagrunert)) [SIG Architecture and Release] 3254 - Updated cri-tools to v1.27.0 ([#117545](https://github.com/kubernetes/kubernetes/pull/117545), [@saschagrunert](https://github.com/saschagrunert)) [SIG Cloud Provider and Node] 3255 - When retrieving event resources, the reportingController and reportingInstance fields in the event will contain values. ([#116506](https://github.com/kubernetes/kubernetes/pull/116506), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery and Instrumentation] 3256 3257 ## Dependencies 3258 3259 ### Added 3260 - cloud.google.com/go/accessapproval: v1.6.0 3261 - cloud.google.com/go/accesscontextmanager: v1.7.0 3262 - cloud.google.com/go/aiplatform: v1.37.0 3263 - cloud.google.com/go/analytics: v0.19.0 3264 - cloud.google.com/go/apigateway: v1.5.0 3265 - cloud.google.com/go/apigeeconnect: v1.5.0 3266 - cloud.google.com/go/apigeeregistry: v0.6.0 3267 - cloud.google.com/go/appengine: v1.7.1 3268 - cloud.google.com/go/area120: v0.7.1 3269 - cloud.google.com/go/artifactregistry: v1.13.0 3270 - cloud.google.com/go/asset: v1.13.0 3271 - cloud.google.com/go/assuredworkloads: v1.10.0 3272 - cloud.google.com/go/automl: v1.12.0 3273 - cloud.google.com/go/baremetalsolution: v0.5.0 3274 - cloud.google.com/go/batch: v0.7.0 3275 - cloud.google.com/go/beyondcorp: v0.5.0 3276 - cloud.google.com/go/billing: v1.13.0 3277 - cloud.google.com/go/binaryauthorization: v1.5.0 3278 - cloud.google.com/go/certificatemanager: v1.6.0 3279 - cloud.google.com/go/channel: v1.12.0 3280 - cloud.google.com/go/cloudbuild: v1.9.0 3281 - cloud.google.com/go/clouddms: v1.5.0 3282 - cloud.google.com/go/cloudtasks: v1.10.0 3283 - cloud.google.com/go/compute/metadata: v0.2.3 3284 - cloud.google.com/go/compute: v1.19.0 3285 - cloud.google.com/go/contactcenterinsights: v1.6.0 3286 - cloud.google.com/go/container: v1.15.0 3287 - cloud.google.com/go/containeranalysis: v0.9.0 3288 - cloud.google.com/go/datacatalog: v1.13.0 3289 - cloud.google.com/go/dataflow: v0.8.0 3290 - cloud.google.com/go/dataform: v0.7.0 3291 - cloud.google.com/go/datafusion: v1.6.0 3292 - cloud.google.com/go/datalabeling: v0.7.0 3293 - cloud.google.com/go/dataplex: v1.6.0 3294 - cloud.google.com/go/dataproc: v1.12.0 3295 - cloud.google.com/go/dataqna: v0.7.0 3296 - cloud.google.com/go/datastream: v1.7.0 3297 - cloud.google.com/go/deploy: v1.8.0 3298 - cloud.google.com/go/dialogflow: v1.32.0 3299 - cloud.google.com/go/dlp: v1.9.0 3300 - cloud.google.com/go/documentai: v1.18.0 3301 - cloud.google.com/go/domains: v0.8.0 3302 - cloud.google.com/go/edgecontainer: v1.0.0 3303 - cloud.google.com/go/errorreporting: v0.3.0 3304 - cloud.google.com/go/essentialcontacts: v1.5.0 3305 - cloud.google.com/go/eventarc: v1.11.0 3306 - cloud.google.com/go/filestore: v1.6.0 3307 - cloud.google.com/go/functions: v1.13.0 3308 - cloud.google.com/go/gaming: v1.9.0 3309 - cloud.google.com/go/gkebackup: v0.4.0 3310 - cloud.google.com/go/gkeconnect: v0.7.0 3311 - cloud.google.com/go/gkehub: v0.12.0 3312 - cloud.google.com/go/gkemulticloud: v0.5.0 3313 - cloud.google.com/go/gsuiteaddons: v1.5.0 3314 - cloud.google.com/go/iam: v0.13.0 3315 - cloud.google.com/go/iap: v1.7.1 3316 - cloud.google.com/go/ids: v1.3.0 3317 - cloud.google.com/go/iot: v1.6.0 3318 - cloud.google.com/go/kms: v1.10.1 3319 - cloud.google.com/go/language: v1.9.0 3320 - cloud.google.com/go/lifesciences: v0.8.0 3321 - cloud.google.com/go/logging: v1.7.0 3322 - cloud.google.com/go/longrunning: v0.4.1 3323 - cloud.google.com/go/managedidentities: v1.5.0 3324 - cloud.google.com/go/maps: v0.7.0 3325 - cloud.google.com/go/mediatranslation: v0.7.0 3326 - cloud.google.com/go/memcache: v1.9.0 3327 - cloud.google.com/go/metastore: v1.10.0 3328 - cloud.google.com/go/monitoring: v1.13.0 3329 - cloud.google.com/go/networkconnectivity: v1.11.0 3330 - cloud.google.com/go/networkmanagement: v1.6.0 3331 - cloud.google.com/go/networksecurity: v0.8.0 3332 - cloud.google.com/go/notebooks: v1.8.0 3333 - cloud.google.com/go/optimization: v1.3.1 3334 - cloud.google.com/go/orchestration: v1.6.0 3335 - cloud.google.com/go/orgpolicy: v1.10.0 3336 - cloud.google.com/go/osconfig: v1.11.0 3337 - cloud.google.com/go/oslogin: v1.9.0 3338 - cloud.google.com/go/phishingprotection: v0.7.0 3339 - cloud.google.com/go/policytroubleshooter: v1.6.0 3340 - cloud.google.com/go/privatecatalog: v0.8.0 3341 - cloud.google.com/go/pubsublite: v1.7.0 3342 - cloud.google.com/go/recaptchaenterprise/v2: v2.7.0 3343 - cloud.google.com/go/recommendationengine: v0.7.0 3344 - cloud.google.com/go/recommender: v1.9.0 3345 - cloud.google.com/go/redis: v1.11.0 3346 - cloud.google.com/go/resourcemanager: v1.7.0 3347 - cloud.google.com/go/resourcesettings: v1.5.0 3348 - cloud.google.com/go/retail: v1.12.0 3349 - cloud.google.com/go/run: v0.9.0 3350 - cloud.google.com/go/scheduler: v1.9.0 3351 - cloud.google.com/go/secretmanager: v1.10.0 3352 - cloud.google.com/go/security: v1.13.0 3353 - cloud.google.com/go/securitycenter: v1.19.0 3354 - cloud.google.com/go/servicedirectory: v1.9.0 3355 - cloud.google.com/go/shell: v1.6.0 3356 - cloud.google.com/go/spanner: v1.45.0 3357 - cloud.google.com/go/speech: v1.15.0 3358 - cloud.google.com/go/storagetransfer: v1.8.0 3359 - cloud.google.com/go/talent: v1.5.0 3360 - cloud.google.com/go/texttospeech: v1.6.0 3361 - cloud.google.com/go/tpu: v1.5.0 3362 - cloud.google.com/go/trace: v1.9.0 3363 - cloud.google.com/go/translate: v1.7.0 3364 - cloud.google.com/go/video: v1.15.0 3365 - cloud.google.com/go/videointelligence: v1.10.0 3366 - cloud.google.com/go/vision/v2: v2.7.0 3367 - cloud.google.com/go/vmmigration: v1.6.0 3368 - cloud.google.com/go/vmwareengine: v0.3.0 3369 - cloud.google.com/go/vpcaccess: v1.6.0 3370 - cloud.google.com/go/webrisk: v1.8.0 3371 - cloud.google.com/go/websecurityscanner: v1.5.0 3372 - cloud.google.com/go/workflows: v1.10.0 3373 - github.com/googleapis/enterprise-certificate-proxy: [v0.2.3](https://github.com/googleapis/enterprise-certificate-proxy/tree/v0.2.3) 3374 - go.etcd.io/gofail: v0.1.0 3375 - google.golang.org/genproto/googleapis/api: dd9d682 3376 - google.golang.org/genproto/googleapis/rpc: 28d5490 3377 3378 ### Changed 3379 - cloud.google.com/go/bigquery: v1.8.0 → v1.50.0 3380 - cloud.google.com/go/datastore: v1.1.0 → v1.11.0 3381 - cloud.google.com/go/firestore: v1.1.0 → v1.9.0 3382 - cloud.google.com/go/pubsub: v1.3.1 → v1.30.0 3383 - cloud.google.com/go: v0.97.0 → v0.110.0 3384 - github.com/Azure/azure-sdk-for-go: [v55.0.0+incompatible → v68.0.0+incompatible](https://github.com/Azure/azure-sdk-for-go/compare/v55.0.0...v68.0.0) 3385 - github.com/Azure/go-autorest/autorest/adal: [v0.9.20 → v0.9.23](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.9.20...v0.9.23) 3386 - github.com/Azure/go-autorest/autorest/validation: [v0.1.0 → v0.3.1](https://github.com/Azure/go-autorest/autorest/validation/compare/v0.1.0...v0.3.1) 3387 - github.com/Azure/go-autorest/autorest: [v0.11.27 → v0.11.29](https://github.com/Azure/go-autorest/autorest/compare/v0.11.27...v0.11.29) 3388 - github.com/Microsoft/go-winio: [v0.4.17 → v0.6.0](https://github.com/Microsoft/go-winio/compare/v0.4.17...v0.6.0) 3389 - github.com/cenkalti/backoff/v4: [v4.1.3 → v4.2.1](https://github.com/cenkalti/backoff/v4/compare/v4.1.3...v4.2.1) 3390 - github.com/census-instrumentation/opencensus-proto: [v0.2.1 → v0.4.1](https://github.com/census-instrumentation/opencensus-proto/compare/v0.2.1...v0.4.1) 3391 - github.com/cespare/xxhash/v2: [v2.1.2 → v2.2.0](https://github.com/cespare/xxhash/v2/compare/v2.1.2...v2.2.0) 3392 - github.com/cilium/ebpf: [v0.7.0 → v0.9.1](https://github.com/cilium/ebpf/compare/v0.7.0...v0.9.1) 3393 - github.com/cncf/udpa/go: [04548b0 → c52dc94](https://github.com/cncf/udpa/go/compare/04548b0...c52dc94) 3394 - github.com/cncf/xds/go: [cb28da3 → 06c439d](https://github.com/cncf/xds/go/compare/cb28da3...06c439d) 3395 - github.com/cockroachdb/datadriven: [bf6692d → v1.0.2](https://github.com/cockroachdb/datadriven/compare/bf6692d...v1.0.2) 3396 - github.com/container-storage-interface/spec: [v1.7.0 → v1.8.0](https://github.com/container-storage-interface/spec/compare/v1.7.0...v1.8.0) 3397 - github.com/containerd/cgroups: [v1.0.1 → v1.1.0](https://github.com/containerd/cgroups/compare/v1.0.1...v1.1.0) 3398 - github.com/containerd/ttrpc: [v1.1.0 → v1.2.2](https://github.com/containerd/ttrpc/compare/v1.1.0...v1.2.2) 3399 - github.com/coredns/caddy: [v1.1.0 → v1.1.1](https://github.com/coredns/caddy/compare/v1.1.0...v1.1.1) 3400 - github.com/coreos/go-oidc: [v2.1.0+incompatible → v2.2.1+incompatible](https://github.com/coreos/go-oidc/compare/v2.1.0...v2.2.1) 3401 - github.com/coreos/go-semver: [v0.3.0 → v0.3.1](https://github.com/coreos/go-semver/compare/v0.3.0...v0.3.1) 3402 - github.com/coreos/go-systemd/v22: [v22.4.0 → v22.5.0](https://github.com/coreos/go-systemd/v22/compare/v22.4.0...v22.5.0) 3403 - github.com/docker/distribution: [v2.8.1+incompatible → v2.8.2+incompatible](https://github.com/docker/distribution/compare/v2.8.1...v2.8.2) 3404 - github.com/envoyproxy/go-control-plane: [49ff273 → v0.10.3](https://github.com/envoyproxy/go-control-plane/compare/49ff273...v0.10.3) 3405 - github.com/envoyproxy/protoc-gen-validate: [v0.1.0 → v0.9.1](https://github.com/envoyproxy/protoc-gen-validate/compare/v0.1.0...v0.9.1) 3406 - github.com/frankban/quicktest: [v1.11.3 → v1.14.0](https://github.com/frankban/quicktest/compare/v1.11.3...v1.14.0) 3407 - github.com/fvbommel/sortorder: [v1.0.1 → v1.1.0](https://github.com/fvbommel/sortorder/compare/v1.0.1...v1.1.0) 3408 - github.com/go-logr/logr: [v1.2.3 → v1.2.4](https://github.com/go-logr/logr/compare/v1.2.3...v1.2.4) 3409 - github.com/go-task/slim-sprig: [348f09d → 52ccab3](https://github.com/go-task/slim-sprig/compare/348f09d...52ccab3) 3410 - github.com/gofrs/uuid: [v4.0.0+incompatible → v4.4.0+incompatible](https://github.com/gofrs/uuid/compare/v4.0.0...v4.4.0) 3411 - github.com/golang-jwt/jwt/v4: [v4.4.2 → v4.5.0](https://github.com/golang-jwt/jwt/v4/compare/v4.4.2...v4.5.0) 3412 - github.com/google/gofuzz: [v1.1.0 → v1.2.0](https://github.com/google/gofuzz/compare/v1.1.0...v1.2.0) 3413 - github.com/googleapis/gax-go/v2: [v2.1.1 → v2.7.1](https://github.com/googleapis/gax-go/v2/compare/v2.1.1...v2.7.1) 3414 - github.com/inconshreveable/mousetrap: [v1.0.1 → v1.1.0](https://github.com/inconshreveable/mousetrap/compare/v1.0.1...v1.1.0) 3415 - github.com/mitchellh/go-wordwrap: [v1.0.0 → v1.0.1](https://github.com/mitchellh/go-wordwrap/compare/v1.0.0...v1.0.1) 3416 - github.com/onsi/ginkgo/v2: [v2.9.1 → v2.9.4](https://github.com/onsi/ginkgo/v2/compare/v2.9.1...v2.9.4) 3417 - github.com/onsi/gomega: [v1.27.4 → v1.27.6](https://github.com/onsi/gomega/compare/v1.27.4...v1.27.6) 3418 - github.com/opencontainers/runc: [v1.1.4 → v1.1.7](https://github.com/opencontainers/runc/compare/v1.1.4...v1.1.7) 3419 - github.com/rogpeppe/go-internal: [v1.10.0 → v1.6.1](https://github.com/rogpeppe/go-internal/compare/v1.10.0...v1.6.1) 3420 - github.com/seccomp/libseccomp-golang: [f33da4d → v0.10.0](https://github.com/seccomp/libseccomp-golang/compare/f33da4d...v0.10.0) 3421 - github.com/spf13/cobra: [v1.6.0 → v1.7.0](https://github.com/spf13/cobra/compare/v1.6.0...v1.7.0) 3422 - github.com/stretchr/testify: [v1.8.1 → v1.8.2](https://github.com/stretchr/testify/compare/v1.8.1...v1.8.2) 3423 - github.com/vishvananda/netns: [v0.0.2 → v0.0.4](https://github.com/vishvananda/netns/compare/v0.0.2...v0.0.4) 3424 - github.com/xlab/treeprint: [v1.1.0 → v1.2.0](https://github.com/xlab/treeprint/compare/v1.1.0...v1.2.0) 3425 - go.etcd.io/bbolt: v1.3.6 → v1.3.7 3426 - go.etcd.io/etcd/api/v3: v3.5.7 → v3.5.9 3427 - go.etcd.io/etcd/client/pkg/v3: v3.5.7 → v3.5.9 3428 - go.etcd.io/etcd/client/v2: v2.305.7 → v2.305.9 3429 - go.etcd.io/etcd/client/v3: v3.5.7 → v3.5.9 3430 - go.etcd.io/etcd/pkg/v3: v3.5.7 → v3.5.9 3431 - go.etcd.io/etcd/raft/v3: v3.5.7 → v3.5.9 3432 - go.etcd.io/etcd/server/v3: v3.5.7 → v3.5.9 3433 - go.opencensus.io: v0.23.0 → v0.24.0 3434 - go.uber.org/atomic: v1.7.0 → v1.10.0 3435 - go.uber.org/multierr: v1.6.0 → v1.11.0 3436 - golang.org/x/crypto: v0.1.0 → v0.6.0 3437 - golang.org/x/mod: v0.9.0 → v0.10.0 3438 - golang.org/x/net: v0.8.0 → v0.9.0 3439 - golang.org/x/oauth2: ee48083 → v0.6.0 3440 - golang.org/x/sys: v0.6.0 → v0.7.0 3441 - golang.org/x/term: v0.6.0 → v0.7.0 3442 - golang.org/x/text: v0.8.0 → v0.9.0 3443 - golang.org/x/time: 90d013b → v0.3.0 3444 - golang.org/x/tools: v0.7.0 → v0.8.0 3445 - google.golang.org/api: v0.60.0 → v0.114.0 3446 - google.golang.org/genproto: c8bf987 → 0005af6 3447 - google.golang.org/grpc: v1.51.0 → v1.54.0 3448 - google.golang.org/protobuf: v1.28.1 → v1.30.0 3449 - gopkg.in/gcfg.v1: v1.2.0 → v1.2.3 3450 - gopkg.in/natefinch/lumberjack.v2: v2.0.0 → v2.2.1 3451 - gopkg.in/warnings.v0: v0.1.1 → v0.1.2 3452 - k8s.io/klog/v2: v2.90.1 → v2.100.1 3453 - k8s.io/kube-openapi: 15aac26 → 7828149 3454 - k8s.io/utils: a36077c → d93618c 3455 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.1.1 → v0.1.2 3456 3457 ### Removed 3458 - github.com/certifi/gocertifi: [2c3bb06](https://github.com/certifi/gocertifi/tree/2c3bb06) 3459 - github.com/cockroachdb/errors: [v1.2.4](https://github.com/cockroachdb/errors/tree/v1.2.4) 3460 - github.com/cockroachdb/logtags: [eb05cc2](https://github.com/cockroachdb/logtags/tree/eb05cc2) 3461 - github.com/getsentry/raven-go: [v0.2.0](https://github.com/getsentry/raven-go/tree/v0.2.0)