k8s.io/kubernetes@v1.31.0-alpha.0.0.20240520171757-56147500dadc/CHANGELOG/CHANGELOG-1.21.md (about) 1 <!-- BEGIN MUNGE: GENERATED_TOC --> 2 3 - [v1.21.14](#v12114) 4 - [Downloads for v1.21.14](#downloads-for-v12114) 5 - [Source Code](#source-code) 6 - [Client Binaries](#client-binaries) 7 - [Server Binaries](#server-binaries) 8 - [Node Binaries](#node-binaries) 9 - [Container Images](#container-images) 10 - [Changelog since v1.21.13](#changelog-since-v12113) 11 - [Dependencies](#dependencies) 12 - [Added](#added) 13 - [Changed](#changed) 14 - [Removed](#removed) 15 - [v1.21.13](#v12113) 16 - [Downloads for v1.21.13](#downloads-for-v12113) 17 - [Source Code](#source-code-1) 18 - [Client Binaries](#client-binaries-1) 19 - [Server Binaries](#server-binaries-1) 20 - [Node Binaries](#node-binaries-1) 21 - [Container Images](#container-images-1) 22 - [Changelog since v1.21.12](#changelog-since-v12112) 23 - [Changes by Kind](#changes-by-kind) 24 - [Bug or Regression](#bug-or-regression) 25 - [Dependencies](#dependencies-1) 26 - [Added](#added-1) 27 - [Changed](#changed-1) 28 - [Removed](#removed-1) 29 - [v1.21.12](#v12112) 30 - [Downloads for v1.21.12](#downloads-for-v12112) 31 - [Source Code](#source-code-2) 32 - [Client Binaries](#client-binaries-2) 33 - [Server Binaries](#server-binaries-2) 34 - [Node Binaries](#node-binaries-2) 35 - [Container Images](#container-images-2) 36 - [Changelog since v1.21.11](#changelog-since-v12111) 37 - [Changes by Kind](#changes-by-kind-1) 38 - [Bug or Regression](#bug-or-regression-1) 39 - [Dependencies](#dependencies-2) 40 - [Added](#added-2) 41 - [Changed](#changed-2) 42 - [Removed](#removed-2) 43 - [v1.21.11](#v12111) 44 - [Downloads for v1.21.11](#downloads-for-v12111) 45 - [Source Code](#source-code-3) 46 - [Client Binaries](#client-binaries-3) 47 - [Server Binaries](#server-binaries-3) 48 - [Node Binaries](#node-binaries-3) 49 - [Container Images](#container-images-3) 50 - [Changelog since v1.21.10](#changelog-since-v12110) 51 - [Changes by Kind](#changes-by-kind-2) 52 - [API Change](#api-change) 53 - [Feature](#feature) 54 - [Bug or Regression](#bug-or-regression-2) 55 - [Dependencies](#dependencies-3) 56 - [Added](#added-3) 57 - [Changed](#changed-3) 58 - [Removed](#removed-3) 59 - [v1.21.10](#v12110) 60 - [Downloads for v1.21.10](#downloads-for-v12110) 61 - [Source Code](#source-code-4) 62 - [Client Binaries](#client-binaries-4) 63 - [Server Binaries](#server-binaries-4) 64 - [Node Binaries](#node-binaries-4) 65 - [Container Images](#container-images-4) 66 - [Changelog since v1.21.9](#changelog-since-v1219) 67 - [Changes by Kind](#changes-by-kind-3) 68 - [Feature](#feature-1) 69 - [Bug or Regression](#bug-or-regression-3) 70 - [Dependencies](#dependencies-4) 71 - [Added](#added-4) 72 - [Changed](#changed-4) 73 - [Removed](#removed-4) 74 - [v1.21.9](#v1219) 75 - [Downloads for v1.21.9](#downloads-for-v1219) 76 - [Source Code](#source-code-5) 77 - [Client Binaries](#client-binaries-5) 78 - [Server Binaries](#server-binaries-5) 79 - [Node Binaries](#node-binaries-5) 80 - [Changelog since v1.21.8](#changelog-since-v1218) 81 - [Changes by Kind](#changes-by-kind-4) 82 - [Feature](#feature-2) 83 - [Bug or Regression](#bug-or-regression-4) 84 - [Other (Cleanup or Flake)](#other-cleanup-or-flake) 85 - [Dependencies](#dependencies-5) 86 - [Added](#added-5) 87 - [Changed](#changed-5) 88 - [Removed](#removed-5) 89 - [v1.21.8](#v1218) 90 - [Downloads for v1.21.8](#downloads-for-v1218) 91 - [Source Code](#source-code-6) 92 - [Client Binaries](#client-binaries-6) 93 - [Server Binaries](#server-binaries-6) 94 - [Node Binaries](#node-binaries-6) 95 - [Changelog since v1.21.7](#changelog-since-v1217) 96 - [Changes by Kind](#changes-by-kind-5) 97 - [Feature](#feature-3) 98 - [Bug or Regression](#bug-or-regression-5) 99 - [Dependencies](#dependencies-6) 100 - [Added](#added-6) 101 - [Changed](#changed-6) 102 - [Removed](#removed-6) 103 - [v1.21.7](#v1217) 104 - [Downloads for v1.21.7](#downloads-for-v1217) 105 - [Source Code](#source-code-7) 106 - [Client Binaries](#client-binaries-7) 107 - [Server Binaries](#server-binaries-7) 108 - [Node Binaries](#node-binaries-7) 109 - [Changelog since v1.21.6](#changelog-since-v1216) 110 - [Changes by Kind](#changes-by-kind-6) 111 - [Feature](#feature-4) 112 - [Failing Test](#failing-test) 113 - [Bug or Regression](#bug-or-regression-6) 114 - [Dependencies](#dependencies-7) 115 - [Added](#added-7) 116 - [Changed](#changed-7) 117 - [Removed](#removed-7) 118 - [v1.21.6](#v1216) 119 - [Downloads for v1.21.6](#downloads-for-v1216) 120 - [Source Code](#source-code-8) 121 - [Client Binaries](#client-binaries-8) 122 - [Server Binaries](#server-binaries-8) 123 - [Node Binaries](#node-binaries-8) 124 - [Changelog since v1.21.5](#changelog-since-v1215) 125 - [Changes by Kind](#changes-by-kind-7) 126 - [API Change](#api-change-1) 127 - [Feature](#feature-5) 128 - [Failing Test](#failing-test-1) 129 - [Bug or Regression](#bug-or-regression-7) 130 - [Dependencies](#dependencies-8) 131 - [Added](#added-8) 132 - [Changed](#changed-8) 133 - [Removed](#removed-8) 134 - [v1.21.5](#v1215) 135 - [Downloads for v1.21.5](#downloads-for-v1215) 136 - [Source Code](#source-code-9) 137 - [Client Binaries](#client-binaries-9) 138 - [Server Binaries](#server-binaries-9) 139 - [Node Binaries](#node-binaries-9) 140 - [Changelog since v1.21.4](#changelog-since-v1214) 141 - [Important Security Information](#important-security-information) 142 - [CVE-2021-25741: Symlink Exchange Can Allow Host Filesystem Access](#cve-2021-25741-symlink-exchange-can-allow-host-filesystem-access) 143 - [Changes by Kind](#changes-by-kind-8) 144 - [Feature](#feature-6) 145 - [Bug or Regression](#bug-or-regression-8) 146 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-1) 147 - [Dependencies](#dependencies-9) 148 - [Added](#added-9) 149 - [Changed](#changed-9) 150 - [Removed](#removed-9) 151 - [v1.21.4](#v1214) 152 - [Downloads for v1.21.4](#downloads-for-v1214) 153 - [Source Code](#source-code-10) 154 - [Client Binaries](#client-binaries-10) 155 - [Server Binaries](#server-binaries-10) 156 - [Node Binaries](#node-binaries-10) 157 - [Changelog since v1.21.3](#changelog-since-v1213) 158 - [Changes by Kind](#changes-by-kind-9) 159 - [Feature](#feature-7) 160 - [Bug or Regression](#bug-or-regression-9) 161 - [Dependencies](#dependencies-10) 162 - [Added](#added-10) 163 - [Changed](#changed-10) 164 - [Removed](#removed-10) 165 - [v1.21.3](#v1213) 166 - [Downloads for v1.21.3](#downloads-for-v1213) 167 - [Source Code](#source-code-11) 168 - [Client Binaries](#client-binaries-11) 169 - [Server Binaries](#server-binaries-11) 170 - [Node Binaries](#node-binaries-11) 171 - [Changelog since v1.21.2](#changelog-since-v1212) 172 - [Changes by Kind](#changes-by-kind-10) 173 - [Feature](#feature-8) 174 - [Bug or Regression](#bug-or-regression-10) 175 - [Dependencies](#dependencies-11) 176 - [Added](#added-11) 177 - [Changed](#changed-11) 178 - [Removed](#removed-11) 179 - [v1.21.2](#v1212) 180 - [Downloads for v1.21.2](#downloads-for-v1212) 181 - [Source Code](#source-code-12) 182 - [Client Binaries](#client-binaries-12) 183 - [Server Binaries](#server-binaries-12) 184 - [Node Binaries](#node-binaries-12) 185 - [Changelog since v1.21.1](#changelog-since-v1211) 186 - [Changes by Kind](#changes-by-kind-11) 187 - [Feature](#feature-9) 188 - [Failing Test](#failing-test-2) 189 - [Bug or Regression](#bug-or-regression-11) 190 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-2) 191 - [Dependencies](#dependencies-12) 192 - [Added](#added-12) 193 - [Changed](#changed-12) 194 - [Removed](#removed-12) 195 - [v1.21.1](#v1211) 196 - [Downloads for v1.21.1](#downloads-for-v1211) 197 - [Source Code](#source-code-13) 198 - [Client Binaries](#client-binaries-13) 199 - [Server Binaries](#server-binaries-13) 200 - [Node Binaries](#node-binaries-13) 201 - [Changelog since v1.21.0](#changelog-since-v1210) 202 - [Changes by Kind](#changes-by-kind-12) 203 - [API Change](#api-change-2) 204 - [Feature](#feature-10) 205 - [Failing Test](#failing-test-3) 206 - [Bug or Regression](#bug-or-regression-12) 207 - [Dependencies](#dependencies-13) 208 - [Added](#added-13) 209 - [Changed](#changed-13) 210 - [Removed](#removed-13) 211 - [v1.21.0](#v1210) 212 - [Downloads for v1.21.0](#downloads-for-v1210) 213 - [Source Code](#source-code-14) 214 - [Client Binaries](#client-binaries-14) 215 - [Server Binaries](#server-binaries-14) 216 - [Node Binaries](#node-binaries-14) 217 - [Changelog since v1.20.0](#changelog-since-v1200) 218 - [What's New (Major Themes)](#whats-new-major-themes) 219 - [Deprecation of PodSecurityPolicy](#deprecation-of-podsecuritypolicy) 220 - [Kubernetes API Reference Documentation](#kubernetes-api-reference-documentation) 221 - [Kustomize Updates in Kubectl](#kustomize-updates-in-kubectl) 222 - [Default Container Annotation](#default-container-annotation) 223 - [Immutable Secrets and ConfigMaps](#immutable-secrets-and-configmaps) 224 - [Structured Logging in Kubelet](#structured-logging-in-kubelet) 225 - [Storage Capacity Tracking](#storage-capacity-tracking) 226 - [Generic Ephemeral Volumes](#generic-ephemeral-volumes) 227 - [CSI Service Account Token](#csi-service-account-token) 228 - [CSI Health Monitoring](#csi-health-monitoring) 229 - [Known Issues](#known-issues) 230 - [<code>TopologyAwareHints</code> feature falls back to default behavior](#topologyawarehints-feature-falls-back-to-default-behavior) 231 - [Urgent Upgrade Notes](#urgent-upgrade-notes) 232 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade) 233 - [Changes by Kind](#changes-by-kind-13) 234 - [Deprecation](#deprecation) 235 - [API Change](#api-change-3) 236 - [Feature](#feature-11) 237 - [Documentation](#documentation) 238 - [Failing Test](#failing-test-4) 239 - [Bug or Regression](#bug-or-regression-13) 240 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-3) 241 - [Uncategorized](#uncategorized) 242 - [Dependencies](#dependencies-14) 243 - [Added](#added-14) 244 - [Changed](#changed-14) 245 - [Removed](#removed-14) 246 - [v1.21.0-rc.0](#v1210-rc0) 247 - [Downloads for v1.21.0-rc.0](#downloads-for-v1210-rc0) 248 - [Source Code](#source-code-15) 249 - [Client binaries](#client-binaries-15) 250 - [Server binaries](#server-binaries-15) 251 - [Node binaries](#node-binaries-15) 252 - [Changelog since v1.21.0-beta.1](#changelog-since-v1210-beta1) 253 - [Urgent Upgrade Notes](#urgent-upgrade-notes-1) 254 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-1) 255 - [Changes by Kind](#changes-by-kind-14) 256 - [API Change](#api-change-4) 257 - [Feature](#feature-12) 258 - [Bug or Regression](#bug-or-regression-14) 259 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-4) 260 - [Dependencies](#dependencies-15) 261 - [Added](#added-15) 262 - [Changed](#changed-15) 263 - [Removed](#removed-15) 264 - [v1.21.0-beta.1](#v1210-beta1) 265 - [Downloads for v1.21.0-beta.1](#downloads-for-v1210-beta1) 266 - [Source Code](#source-code-16) 267 - [Client binaries](#client-binaries-16) 268 - [Server binaries](#server-binaries-16) 269 - [Node binaries](#node-binaries-16) 270 - [Changelog since v1.21.0-beta.0](#changelog-since-v1210-beta0) 271 - [Urgent Upgrade Notes](#urgent-upgrade-notes-2) 272 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-2) 273 - [Changes by Kind](#changes-by-kind-15) 274 - [Deprecation](#deprecation-1) 275 - [API Change](#api-change-5) 276 - [Feature](#feature-13) 277 - [Bug or Regression](#bug-or-regression-15) 278 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-5) 279 - [Uncategorized](#uncategorized-1) 280 - [Dependencies](#dependencies-16) 281 - [Added](#added-16) 282 - [Changed](#changed-16) 283 - [Removed](#removed-16) 284 - [v1.21.0-beta.0](#v1210-beta0) 285 - [Downloads for v1.21.0-beta.0](#downloads-for-v1210-beta0) 286 - [Source Code](#source-code-17) 287 - [Client binaries](#client-binaries-17) 288 - [Server binaries](#server-binaries-17) 289 - [Node binaries](#node-binaries-17) 290 - [Changelog since v1.21.0-alpha.3](#changelog-since-v1210-alpha3) 291 - [Urgent Upgrade Notes](#urgent-upgrade-notes-3) 292 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-3) 293 - [Changes by Kind](#changes-by-kind-16) 294 - [Deprecation](#deprecation-2) 295 - [API Change](#api-change-6) 296 - [Feature](#feature-14) 297 - [Documentation](#documentation-1) 298 - [Failing Test](#failing-test-5) 299 - [Bug or Regression](#bug-or-regression-16) 300 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-6) 301 - [Dependencies](#dependencies-17) 302 - [Added](#added-17) 303 - [Changed](#changed-17) 304 - [Removed](#removed-17) 305 - [v1.21.0-alpha.3](#v1210-alpha3) 306 - [Downloads for v1.21.0-alpha.3](#downloads-for-v1210-alpha3) 307 - [Source Code](#source-code-18) 308 - [Client binaries](#client-binaries-18) 309 - [Server binaries](#server-binaries-18) 310 - [Node binaries](#node-binaries-18) 311 - [Changelog since v1.21.0-alpha.2](#changelog-since-v1210-alpha2) 312 - [Urgent Upgrade Notes](#urgent-upgrade-notes-4) 313 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-4) 314 - [Changes by Kind](#changes-by-kind-17) 315 - [API Change](#api-change-7) 316 - [Feature](#feature-15) 317 - [Documentation](#documentation-2) 318 - [Failing Test](#failing-test-6) 319 - [Bug or Regression](#bug-or-regression-17) 320 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-7) 321 - [Dependencies](#dependencies-18) 322 - [Added](#added-18) 323 - [Changed](#changed-18) 324 - [Removed](#removed-18) 325 - [v1.21.0-alpha.2](#v1210-alpha2) 326 - [Downloads for v1.21.0-alpha.2](#downloads-for-v1210-alpha2) 327 - [Source Code](#source-code-19) 328 - [Client binaries](#client-binaries-19) 329 - [Server binaries](#server-binaries-19) 330 - [Node binaries](#node-binaries-19) 331 - [Changelog since v1.21.0-alpha.1](#changelog-since-v1210-alpha1) 332 - [Urgent Upgrade Notes](#urgent-upgrade-notes-5) 333 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-5) 334 - [Changes by Kind](#changes-by-kind-18) 335 - [Deprecation](#deprecation-3) 336 - [API Change](#api-change-8) 337 - [Documentation](#documentation-3) 338 - [Bug or Regression](#bug-or-regression-18) 339 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-8) 340 - [Dependencies](#dependencies-19) 341 - [Added](#added-19) 342 - [Changed](#changed-19) 343 - [Removed](#removed-19) 344 - [v1.21.0-alpha.1](#v1210-alpha1) 345 - [Downloads for v1.21.0-alpha.1](#downloads-for-v1210-alpha1) 346 - [Source Code](#source-code-20) 347 - [Client binaries](#client-binaries-20) 348 - [Server binaries](#server-binaries-20) 349 - [Node binaries](#node-binaries-20) 350 - [Changelog since v1.20.0](#changelog-since-v1200-1) 351 - [Urgent Upgrade Notes](#urgent-upgrade-notes-6) 352 - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-6) 353 - [Changes by Kind](#changes-by-kind-19) 354 - [Deprecation](#deprecation-4) 355 - [API Change](#api-change-9) 356 - [Feature](#feature-16) 357 - [Bug or Regression](#bug-or-regression-19) 358 - [Other (Cleanup or Flake)](#other-cleanup-or-flake-9) 359 - [Uncategorized](#uncategorized-2) 360 - [Dependencies](#dependencies-20) 361 - [Added](#added-20) 362 - [Changed](#changed-20) 363 - [Removed](#removed-20) 364 365 <!-- END MUNGE: GENERATED_TOC --> 366 367 # v1.21.14 368 369 370 ## Downloads for v1.21.14 371 372 373 374 ### Source Code 375 376 filename | sha512 hash 377 -------- | ----------- 378 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes.tar.gz) | d1f2bdf2e7f18fe019156955b5361d6ea1ebb92039696d20d475dc7bb4ae0a74058d4fd64a5aa783d6332d16ca19196c71f88d4c6e2bbe52173fc11aa864f0c2 379 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-src.tar.gz) | 47cfe98ce5189e70e9147a8df9cd6b2cb3d3048ec0b91843dd2bc21506f662a42ec26eb7f7de671b863f9bd5ca2835524527ca94aca2d8b6be0aef4c4f8d9531 380 381 ### Client Binaries 382 383 filename | sha512 hash 384 -------- | ----------- 385 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-darwin-amd64.tar.gz) | a59b3d33bf77f196b351b1582ef9c5293dba8a6b358b8f75a4dc857e5ce39d106d4f4435c80ce71e1f60b4c0ed361c444ff81836b9a0ad0b72a7e576d7b31488 386 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-darwin-arm64.tar.gz) | a1cd396ef0b0a6881cc68608c61c1736629326e80dc709d21745783295d06bebeb4239ac7d8fc0aad2dfad6ade3908340803fb0c9543f29a704e2b3d8d9e331a 387 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-linux-386.tar.gz) | da2a71c57f6dda6ca7d2b6ee7c243b791d2f9e514a96b4b35c528c873d2e7839fc24409680107808816b269d38142eebf3db1d20acebf8d7eab31ff186e95178 388 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-linux-amd64.tar.gz) | 52a98cc64abeea4187391cbf0ad5bdd69b6920c2b29b8f9afad194441e642fb8f252e14a91c095ef1e85a23e5bb587916bd319566b6e8d1e03be5505400f44b4 389 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-linux-arm.tar.gz) | fb204c3494bb7acf59ea0dc2b6f94c5a7d2b7e6d69c05b7a7e77b3fc438e574e9ac5f8720a4ae8f2f660822262aa8c4723bfe269c49ed641031f8259210e5b48 390 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-linux-arm64.tar.gz) | ed613592035b45c4f4571eed2b739c837def4287aacc7c37665e596484aa6dbd299224ef12ec329cfa8a655e20b6bee6f6fc3204bc79b92d832171073ddde191 391 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-linux-ppc64le.tar.gz) | 17c40aa73dd78d38d53ac53b9f517ec7c728de9aa7e3c60eb6000024c21570bb1fa547356a53281625b1703670a0ee7f75285b0ec3b59dfd52060a690431ba97 392 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-linux-s390x.tar.gz) | 1816d3b5d032ac7a46077c3a876dc8f54aa71124ef00ef36f01809b39c31d66d3e0eabfad0b5a2e21e9a3235ce301c67c9a9ad1cd5bc08f8874033c418eb5b29 393 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-windows-386.tar.gz) | bf07323bddf1386538971e7b3f926e748b9599bc70de51739b15f9da76321f4d05b94e803434f42ae5568aecdcf1e26d62527884832d3d2590988fed8f23e143 394 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-client-windows-amd64.tar.gz) | 68071753c65a6483b71927c2ed2e8eeadfbfb1116fac06fccd067fc6182bd588e88afd79c97cc1b4b8a82c0ed6bed1873015024ae3b11694c82f142aa33bafd7 395 396 ### Server Binaries 397 398 filename | sha512 hash 399 -------- | ----------- 400 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-server-linux-amd64.tar.gz) | 2ce7c05a8a6a53c8dc3cf792ad4b6d99c552fa2819303d021a36c4f1a0818e1719ea1d7a9064eb5e1fa990977cd5222f557822e6c558f3cd5d36bbd5063067fb 401 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-server-linux-arm.tar.gz) | 92df18a0824966e6b930e9a5a91425d8570c566b18438fccbb3a59c5dce6675a489739c051657ed7b29fda395b46bede788f06d9c3252e0f489c2a932006b7e1 402 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-server-linux-arm64.tar.gz) | 85d76454446c32d571b1c0c7c517a50991c508df0a069e8188169d6f10e7466f47ceaf09b2500efb9745bb65314c52c7d88705090261e4213aa1ca64eb51809f 403 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-server-linux-ppc64le.tar.gz) | 95f96a9e79633b0b358cd2fe71946ac9110c0f5ec1534a7dc7ce7603c576315782f5675748a3acba0fbe157531a8392a8e97f54c7790fce1dc7a187989c5ee0a 404 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-server-linux-s390x.tar.gz) | d97c46a38394822381eea3c9d7ce041b89246e0b7ece7849bf5af627ec2e2146e2828804dd329330a59d5cbbc45d3a906a400fc0f58c2c4de4b5bb17cae28bdf 405 406 ### Node Binaries 407 408 filename | sha512 hash 409 -------- | ----------- 410 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-node-linux-amd64.tar.gz) | 35a28acfdbf15265b4d463ecf86cb7bd9c1261fbb2a273d83971e5080050048a4c970dd1daa341c9682d667040ff32be27ec9c6c2e36f51b3d32e673007255d1 411 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-node-linux-arm.tar.gz) | 538514ada79b8cf167acdf79a2c472001dc69859de96fc429c7c727769262831e257ac0dd4dc547270594c17c4e685a2a27826eea79c74831ef7460dd11fcf4c 412 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-node-linux-arm64.tar.gz) | 25bd250a333058dd544ae55e80716ee30dd4fa5159a56b5675c9be86df2eb87697c768adfd14ad558a7a0dc73238bc4a5978afb2e83f3574a83eef4f8dff7431 413 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-node-linux-ppc64le.tar.gz) | 75cf26950ac41d8c4c305a9987cbfd8e4a5d510e11dfe37b66dde4600b6d28add9a6cff11b6a2adf2aa622d625cdb9f0ad77655b4e7a8dda44da2dd72e5b4a27 414 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-node-linux-s390x.tar.gz) | 1e6014fc5e83b6c5794c1d0e555bd62729af49cfd72b539ac931e3be8d45250124d320082957f8e4d7fdd614031e84d887d630ae31aa2ba565b5e4c83a257345 415 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.14/kubernetes-node-windows-amd64.tar.gz) | e18f2ecc27026cdc59c019e4a9011f2d9740945f949ff3b40b768e870fb1d2da378391de5c08b070589d87aebe1284d215010397d984a31e1e1fc37fec697c65 416 417 ### Container Images 418 419 All container images are available as manifest lists and support the described 420 architectures. It is also possible to pull a specific architecture directly by 421 adding the "-$ARCH" suffix to the container image name. 422 423 name | architectures 424 ---- | ------------- 425 [k8s.gcr.io/conformance:v1.21.14](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 426 [k8s.gcr.io/kube-apiserver:v1.21.14](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 427 [k8s.gcr.io/kube-controller-manager:v1.21.14](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 428 [k8s.gcr.io/kube-proxy:v1.21.14](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 429 [k8s.gcr.io/kube-scheduler:v1.21.14](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 430 431 ## Changelog since v1.21.13 432 433 ## Dependencies 434 435 ### Added 436 _Nothing has changed._ 437 438 ### Changed 439 _Nothing has changed._ 440 441 ### Removed 442 _Nothing has changed._ 443 444 445 446 # v1.21.13 447 448 449 ## Downloads for v1.21.13 450 451 452 453 ### Source Code 454 455 filename | sha512 hash 456 -------- | ----------- 457 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes.tar.gz) | 3d545a8ce6cf0bc089a90f8256513eb39a7a76c64bda61a5a58a87bde5496435e8eccb006f1b766c4aaff500819f5da390c4d58efa5cf3b2bfb4f6b0db153282 458 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-src.tar.gz) | 5578d3364c005df89dbb5c151df96b0d41254f095e044c0aa7d4f0c0d0217378f24c3bef51dc7a2886a92433ed641fc1ed55d010520f49f67edf3811bdb9eac7 459 460 ### Client Binaries 461 462 filename | sha512 hash 463 -------- | ----------- 464 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-darwin-amd64.tar.gz) | 01d10c9ce3e5ee131e77b98677a0a2362ab1934c9a802770ecebc945a8bda1fad56441c5ea9d5756ad551658ef340cb02a6708914255da50755c85a246b385ca 465 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-darwin-arm64.tar.gz) | 0ce09b21b1ddef2c9f72f58bf4b69bd02141be9a2385704033c5eed16f8493ee2cd46f02e7d35ecdd25854a188bdefa1106d79b668552ae43ec4e78ea7156c3e 466 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-linux-386.tar.gz) | ef94422047e0fc41abf7012b87af9a1c93edc551d5653d042bdaea9ef5d392ec1de2c264bd713b2ea13ac175d969c0eb4651c398b29e7fef65e4abb19b997517 467 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-linux-amd64.tar.gz) | a1968909cf21a36ff3ac00c6c8c3ade78d81ed1bc81298569da6d57712365fc313bbae469d0ab9662a716af28f6b113d0a8d1146ef75ed86b1964b4623dca567 468 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-linux-arm.tar.gz) | de4e9788ddea6c50811a1b47f5e733b23c497ec24e81212fbdfd1b3a969fdc1b2f8171a155f8569e9686a612be9f7949b81ac97e1fcbf9199bae9b7cea30af4f 469 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-linux-arm64.tar.gz) | 1cea12e532c83c6b75adc0ad225ba6a11a2efb09413a17545203025cd674f1b327ec75a834ecd8609f45de7b4924c802798479dd5054794438124873acca6372 470 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-linux-ppc64le.tar.gz) | 93cda5556e6c6b52cec04319e67db5e97656de398a7d8bf09973a7341bbca4a70650518bfa553ff9bdd0a8860d3a499178223e52dd8139542701291e85bf7348 471 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-linux-s390x.tar.gz) | a80465b2524f782c1a8a457433ce98ca47c9b227d6966b0adec21764a9fe7243cef0ef39dc07abdf5dbbb053cdbb675772b57d58c5176f1322e1360175665973 472 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-windows-386.tar.gz) | 4971dd79a55283c79b1959d53508777260f7f8db21f2539eeac0a142b4ab8d44a2108b0170087aed69c22e06b31a4467d81745b66e1bed89c90015267e3bbc99 473 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-client-windows-amd64.tar.gz) | 988c4384cc758f41fc79e7eb59daed6a735c9aed44b417d08181846060fecdff8dd89f9f6900e73037570e5bd5431a79a4d4cc5457d587a67734295df70c0fef 474 475 ### Server Binaries 476 477 filename | sha512 hash 478 -------- | ----------- 479 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-server-linux-amd64.tar.gz) | c333e3c481dc58a6baccfd07481115ba5f42126e81097e2807894ee8e39b9f14c3c4972c70ae1262a1f23ea327aa3b07a30d2e7716ebb8e9a8d45a8a9d8c0ffc 480 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-server-linux-arm.tar.gz) | b636f96ebb88e4c7b75d44b2713a0a97861549154680bf8c939e12f5632b4c393457b2094ba76a2d2fdba1404132e472a8e3d875e3183b971123904c92e94711 481 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-server-linux-arm64.tar.gz) | 0a5265c55aeb0a04767a8e1dcf7052e7edee5d4d56d1d2bea5a4765e64a4428e79dde33d20fcda546c18a0b7270241d7b93ac5590d6da7688d92de671fc57b06 482 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-server-linux-ppc64le.tar.gz) | 320c828a5fac9d3b3f4ec13dcc1929cf2adaa371f4e502d50715fd15ea48c1e0fcb65374684562df65380b1b02c449c82153a1da546e2a78370a8cbb6cd8d4e7 483 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-server-linux-s390x.tar.gz) | 7557fc644c2987ec99ca3c4e0ecacb5b1bf96ce38e79222901e452ffd76838981ec019300bb27a28579c7ca8828955850b4adae112dabaab92cd530a5979f03f 484 485 ### Node Binaries 486 487 filename | sha512 hash 488 -------- | ----------- 489 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-node-linux-amd64.tar.gz) | 713a657896ad0c2adb4d4b6f94be203bad8a3746e927f142924aef89a10e671bb843ba4515e09ce62631454ee04eb1ab5a43abfb4091d76aab8602991c5f774c 490 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-node-linux-arm.tar.gz) | e9a68fdb3e833ecdbf82beed0f9b419b79231a0c1529bd931e83cbad0a4f6e3e6856974cbaae9c361bba64e3bdd946ce18a5b5fd0e3f9354ec75eb2a7f121837 491 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-node-linux-arm64.tar.gz) | b11ef6d97a7a7536c379ad9b500262c43986e53db6304614b1b64c2e21fa2044eb8f6606eda7a71fe41fdcee845156e4977ee48d9a45a081e85f93fb78cfe195 492 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-node-linux-ppc64le.tar.gz) | 0c210524c5f6b7291fd393a6b7c0505cf44d1cc5f663a0d4753b2a86f45698673698f413857a18af6897efd28d8bc4cc68050bd2ae8a01455758adaea2f0bee3 493 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-node-linux-s390x.tar.gz) | 6f1997e972b124763c45cb8fbf99daa0159731aae7f37d6bf7b378681605eea7e0f7a6d9643d5ddc09aee252d1164d47bd10b5186cdefea2bdec3a8e48d2c792 494 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.13/kubernetes-node-windows-amd64.tar.gz) | d2215efa5abdde58723462b20ce7114392748da722afa3cedbb89bfee2fe0ae25b3c40ffb4a6c394dfb4a91cfcf2b8c0253e7f99979e44f6af4df7311bc19c5b 495 496 ### Container Images 497 498 All container images are available as manifest lists and support the described 499 architectures. It is also possible to pull a specific architecture directly by 500 adding the "-$ARCH" suffix to the container image name. 501 502 name | architectures 503 ---- | ------------- 504 [k8s.gcr.io/conformance:v1.21.13](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 505 [k8s.gcr.io/kube-apiserver:v1.21.13](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 506 [k8s.gcr.io/kube-controller-manager:v1.21.13](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 507 [k8s.gcr.io/kube-proxy:v1.21.13](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 508 [k8s.gcr.io/kube-scheduler:v1.21.13](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 509 510 ## Changelog since v1.21.12 511 512 ## Changes by Kind 513 514 ### Bug or Regression 515 516 - Correct event registration for multiple scheduler plugins; this fixes a potential significant delay in re-queueing unschedulable pods. ([#109466](https://github.com/kubernetes/kubernetes/pull/109466), [@ahg-g](https://github.com/ahg-g)) [SIG Scheduling and Testing] 517 - Existing InTree AzureFile PVs which don't have a secret namespace defined will now work properly after enabling CSI migration - the namespace will be obtained from ClaimRef. ([#108000](https://github.com/kubernetes/kubernetes/pull/108000), [@RomanBednar](https://github.com/RomanBednar)) [SIG Cloud Provider and Storage] 518 519 ## Dependencies 520 521 ### Added 522 _Nothing has changed._ 523 524 ### Changed 525 _Nothing has changed._ 526 527 ### Removed 528 _Nothing has changed._ 529 530 531 532 # v1.21.12 533 534 535 ## Downloads for v1.21.12 536 537 538 539 ### Source Code 540 541 filename | sha512 hash 542 -------- | ----------- 543 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes.tar.gz) | 602bf6335571e9e2d2dc5de8e65ad5d326fe79493d1b583b1ba66564a2372455529e70cfea8e6a5d491faa13283cd5cc4362ca4e6a1c008498f3240d7b66c163 544 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-src.tar.gz) | 53e93df60bb450774d86772e422959513236db24ea0fda3dbf26a2371d906df807e56545545e4f02db9f3790adba2e27e2b4d71b91a72addad6785a336a441fa 545 546 ### Client Binaries 547 548 filename | sha512 hash 549 -------- | ----------- 550 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-darwin-amd64.tar.gz) | cb6d8c1d270385fd9b56201bc02a94b8f77259db234ddb664c4eb1eaf38ee648e55b5957dba6f3f36471202b78db0a74e4be5a149dda4f5fb68b47f5f8bf1e89 551 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-darwin-arm64.tar.gz) | 8664ec7c26e071cfcc3ca43e58e59864557e2843771189acf0e4207bb448c84c488450aec8c4888618c27595fc8650e57b3013e09ac725c7a454b83355e205a5 552 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-linux-386.tar.gz) | ea7515063a0bf091a880422ed19335ee952653d1f41624c4ac0acada95984e9c5f89e7a35c3ff3bf9da11a9f28bd7bf3863bc4c6b3d9e560f91c4b0ce3af5f7d 553 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-linux-amd64.tar.gz) | 8b1ee9506eeffece359c6db79b43bf227fc6effc917ba87ddd708dc6389c628fb30d37c9ba80d9af8dc5092507cc4225ef393e575c5deabb1472e2ba7ea8a913 554 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-linux-arm.tar.gz) | 0c9ae672f97071f3e2751e1bc2c56bfbe380bf25838e9bddea017daa95cbda0058bd6eb9dbcdd33f68f080fe1912ee6141c4f267232aed77904d5d75e4902795 555 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-linux-arm64.tar.gz) | c7fead900dc24463779ceb6969ab84ecee8ecbe7261389d63e0bdd84f3b98713e809ffe2125a2397dda1bdbe2160912b8f4c11abc001fa3b733a88040dbd8d92 556 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-linux-ppc64le.tar.gz) | d203d3663af0c5fbad234eeacd679333e41bc524ac4828f82c807ac57608469b634f1e94fc3827cc13a51b11a681ed5e55d4413da1429203298a1477bc56bb26 557 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-linux-s390x.tar.gz) | 0b21ca10c1cb2c555604de7c0a2f83ab2244389483a655013f7406363f61912e4819514b2c15bbbc57c5e18a2cdbb513ca9d08e1520c809338d82bd27e739eda 558 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-windows-386.tar.gz) | 785ed9a1380ca7038ff6a0a18a1d02c61c476f40091884061e1c396436f5e02bed18780872e1f3cd301f9db48ce200d3004753ccbad9519606c522edcf2375ed 559 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-client-windows-amd64.tar.gz) | c322eeb2c32ab093f1eecfa10ffff62acc1b3ca4e113ee8255b5107d33e7245b9050902ad5f6fc5db961e8aae08c2331841b5da21a0d27a903cd1f88ee45e33e 560 561 ### Server Binaries 562 563 filename | sha512 hash 564 -------- | ----------- 565 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-server-linux-amd64.tar.gz) | a4621eeb35d7328611eead353c1296bca7bd8bc6bbb5683b8e0c8717b69d70fab053903c519d80e6fed94fafee7a5226aa84414fecde06074bbd8f1afa0e84bb 566 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-server-linux-arm.tar.gz) | 185e70b026c417d15d997ae8d51d1739cc2aea6ca69187e3fb1e69a44aa01cd7e34d1cfbc412dbdfbe8e1e3b6d9dcc7a7f993c3c5a450e5da1bc21cd4fa14d24 567 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-server-linux-arm64.tar.gz) | c252d6a3c0ea949afd0fad494042be440b792a79bd15f430db5ce79440e2a8597a0b1556848be38a34ca560708099791d5b57b0b2f9442c01bb189c16243b730 568 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-server-linux-ppc64le.tar.gz) | b5aba905e843ca23519e69206fc26f8391b474e02dc07949ffdbc472dcbc2e842580731fdd95555c6abf0ef511437d7494c7bdcc16961d2c06ece41137f708d1 569 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-server-linux-s390x.tar.gz) | 2292c40944470594b6c97a3d3cb63e1de27e557359baf0c03f55664e252cce1c5a80a7e37c1c44ebaf52f4f8b68bc3441a04be24480ab57e2b93bbc47e5ef725 570 571 ### Node Binaries 572 573 filename | sha512 hash 574 -------- | ----------- 575 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-node-linux-amd64.tar.gz) | 727e873f5980492ed680c8b639c6ae02601722533e4a5b9a388ea232744c1e305ce89d1d38303875109e9ccda5d5c8bb0012935ab5c7dde74adec8d65a18cc34 576 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-node-linux-arm.tar.gz) | 5f666dcfb3142a02f97348cb4a1a9cb9d3390a621aa33dead157339ada12f79e0bf8ebf248d4fc3c0726e3ddf17bca4fb2e21b3c590691515ccf4b828142c12c 577 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-node-linux-arm64.tar.gz) | a191f5267eb2f720103807e8133d98498cf9a0cc4f2a50a305ff071df3e80a7de89e6791deb9162ce1e602bbf073fd5036e85d066b62706d6a74782bc8f055af 578 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-node-linux-ppc64le.tar.gz) | 72ae31276e53e3ca07a5c8018138fac2140aa4d1da53b6845c0165bbd49c6d35940b101c37df86762b8d7ae029aae2076b93a405912ef21732ba04db886ad2a9 579 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-node-linux-s390x.tar.gz) | 4ec572173b0b611b8d368d4781baede425da666a4c5dc9ee02ee8bac76e132745eb0cd27ea12da9094829b4a76e9de093f7fa2f508415d7b18d3d1d23e541f9e 580 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.12/kubernetes-node-windows-amd64.tar.gz) | 6f60fe987070e818c5572819ea6f1c6ab03a989b9ac53e184c6ba2488da055ab64c85d7ae29734ba979d5618ab8b4bb21e29789b2dc15de969ca3b59a45fc044 581 582 ### Container Images 583 584 All container images are available as manifest lists and support the described 585 architectures. It is also possible to pull a specific architecture directly by 586 adding the "-$ARCH" suffix to the container image name. 587 588 name | architectures 589 ---- | ------------- 590 [k8s.gcr.io/conformance:v1.21.12](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 591 [k8s.gcr.io/kube-apiserver:v1.21.12](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 592 [k8s.gcr.io/kube-controller-manager:v1.21.12](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 593 [k8s.gcr.io/kube-proxy:v1.21.12](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 594 [k8s.gcr.io/kube-scheduler:v1.21.12](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 595 596 ## Changelog since v1.21.11 597 598 ## Changes by Kind 599 600 ### Bug or Regression 601 602 - Fixes error handling in a kubectl method used in downstream packages. ([#108521](https://github.com/kubernetes/kubernetes/pull/108521), [@heybronson](https://github.com/heybronson)) [SIG CLI] 603 - Prevent kube-scheduler from nominating a Pod that was deleted or already scheduled to a node ([#109248](https://github.com/kubernetes/kubernetes/pull/109248), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] 604 605 ## Dependencies 606 607 ### Added 608 _Nothing has changed._ 609 610 ### Changed 611 - golang.org/x/crypto: 5ea612d → 5770296 612 - golang.org/x/sys: d19ff85 → 665e8c7 613 614 ### Removed 615 _Nothing has changed._ 616 617 618 619 # v1.21.11 620 621 622 ## Downloads for v1.21.11 623 624 625 626 ### Source Code 627 628 filename | sha512 hash 629 -------- | ----------- 630 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes.tar.gz) | 654dbddda1bfc8cf9ef439561f2661f948191c6eb1b502f9756416a8f1a3dbe347d26bc2ba3edac94a6e5f7b57dcf5aa70f90e881dc387094d736e30ed0abf5d 631 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-src.tar.gz) | 6f3e8b54eb06aea56fdee0b9dcd2404f4531b66d9ffde6ea977cebe6f61957276377b8f7c2e0d4c958747691169b2aca3081ab0f02e90cbf5497fe2f10fbd872 632 633 ### Client Binaries 634 635 filename | sha512 hash 636 -------- | ----------- 637 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-darwin-amd64.tar.gz) | b4d3348384346e288762f2aa57c43b6502814bae1123716bb1d7ea8e3a89bf6d50bf041e906bb65951225d62d8114fe1ccb9fc8d01a61d25bcad15b9fdcfb60a 638 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-darwin-arm64.tar.gz) | 2fad4ead5ab00bb1a71092a04c03307ea9445deb65f9d753ae892987141b074f42996e57b08f44d89fffffad86a1309ce6333b18084b800f556c1d20b186bb5b 639 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-linux-386.tar.gz) | b8114e1e877613ac6642a121ce6513e8af9b1a23f417c66f6ebfa0d91ff41a115dd28037e5aa466ce17f1c0fb94b142bdc89a8c7829db28c8e4b050ee88c9101 640 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-linux-amd64.tar.gz) | 84d128d1021167d3c58cc142115b0a6b31a50e3e78eb234648cdb6e26f42ab76143c0b5fd34f5476d963f2d09d8907031425eb0ceec529db6eab44a404b55530 641 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-linux-arm.tar.gz) | 4cd3ef06b8eb806f17eab383c8b97b943c8864e8103e9330732dfcc849cc76baaf0d5bd000be11551485010fd51fb428e8cc243a30bf5bdaf1ab5df1903027be 642 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-linux-arm64.tar.gz) | 62601d78baf10115246de1956bd0f24c30d412e910c942ebbda87157fd9eb3845c213b47075cb94c08dc301b493f5d25c3d39d0229d02a365bfa37bd0f0ee0f4 643 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-linux-ppc64le.tar.gz) | 02c20f3ffc214f4f171c9ef05efe13015077deff142f452dc9dfe224483065b85776b5c2f0356d5e22e6fdfba86973189f98263e920490f2d9ef33170333f482 644 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-linux-s390x.tar.gz) | a4a072246afdfeae41bad025a4d66acd5612dc7dba1fb97c1c8809e50e37772fb7545f31471f75f51971e5c4aac53dadf25c4c066bcf1f3e55a87d3cb1533638 645 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-windows-386.tar.gz) | 8273d3b6a975f1bac3fccfd81f0cd9b755e2313555882c34810e33c651eeeebfabc9d6930599998502dfd5a4c7679a8363dbb5884e6b0de248a3c1124124b929 646 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-client-windows-amd64.tar.gz) | 2a3729e4eae348fa8e14aaf3f7eb92794309e5e2a994e176ee61edcbc36aea25f1276716cde4b3ba656b6e05bdb2ed5c87f3e789d7ab73ec8dc3405ba43ea80a 647 648 ### Server Binaries 649 650 filename | sha512 hash 651 -------- | ----------- 652 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-server-linux-amd64.tar.gz) | 8d9a636a37af1f361c6350b7c3fbd7b1b723a5c396579ee29fe74751ab51ca5ed17a6d69d06d52b56027f405a9ae38f4cc3183f5c6e20db390205ed3354cb81a 653 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-server-linux-arm.tar.gz) | c7105dd353c0089303ccfebf9534418995f5418a080411b773e9616f042284e3b3d4b9718f5248eda7785a2725bad370d1323d899a93093a356133578383ce76 654 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-server-linux-arm64.tar.gz) | 258bdc66264a6a18007c4839ce3138904a26479dad9c7f99610416f439aa2ff0538d398fdeca323a36167e07b23abe3bd05f33a108f30ac8482f73b0cada82aa 655 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-server-linux-ppc64le.tar.gz) | 71321a7fd9c5d02b7b00e92b4c818c9b7b95d722377e94f23c88cdde2e2bbfd362fc608bbe71494d00d677e9c091ee8b6bfeef343e4f6d21574feeb42fdf998e 656 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-server-linux-s390x.tar.gz) | af037fe2028c7176622154fe97b0fa6b46034c7ffa9237eb01d4dabc61a03b9c04d4b97079ea807bf28b23cca12ba9a8a058e52176d59fabdf86e5cecf429a12 657 658 ### Node Binaries 659 660 filename | sha512 hash 661 -------- | ----------- 662 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-node-linux-amd64.tar.gz) | 9937dad3d9c8712f508c0df572faa58af1fd09e7e9c81ccd3c964e59ec95abe9c9617fe7899a41b6a2985f3d687d2a9c0d4f40e06aba62b64e2d4a1d261cf5d9 663 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-node-linux-arm.tar.gz) | a27c766eed26bc8280908232b8713811c7d84eeb428177462e370f45adad6ce89fd08caf2a781c6a253567d66c1870cfb69c4ef1e088b566a25d13b2b064211a 664 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-node-linux-arm64.tar.gz) | 362b3bf24ea715521fe6835d7949fa386d4f3baab89d8ed38351fd522fbe29b1e3b79b07ff9f76216848100ad637265715374f399ba0eab906576583481bb97b 665 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-node-linux-ppc64le.tar.gz) | 7964a70aa21835d8d56185d7a9c5e90253953c99fa21a6ad2b9cb057b96c95cab2cae517fa8e7c0d58a3daa0f325bafb37b00b4858e832493b67c31c118ff863 666 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-node-linux-s390x.tar.gz) | 31d11883546c088de577413460ff13f8a14c69ef9278ef62586302158583ea050afa8c8c7bb54b4a51142884bd03e8b8843e474c4a76b0bc327b7133a9d60d15 667 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.11/kubernetes-node-windows-amd64.tar.gz) | 8fb7b9873dc24069a2975f7afe2f146f8ce0c684ccd181fcccefb35f04cbbb153b90bf7f258c409fc2bec41a5682946a3fe6afc7a184b23353be422cd5ba1581 668 669 ### Container Images 670 671 All container images are available as manifest lists and support the described 672 architectures. It is also possible to pull a specific architecture directly by 673 adding the "-$ARCH" suffix to the container image name. 674 675 name | architectures 676 ---- | ------------- 677 [k8s.gcr.io/conformance:v1.21.11](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x) 678 [k8s.gcr.io/kube-apiserver:v1.21.11](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x) 679 [k8s.gcr.io/kube-controller-manager:v1.21.11](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x) 680 [k8s.gcr.io/kube-proxy:v1.21.11](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x) 681 [k8s.gcr.io/kube-scheduler:v1.21.11](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x) 682 683 ## Changelog since v1.21.10 684 685 ## Changes by Kind 686 687 ### API Change 688 689 - Fixes a regression in v1beta1 PodDisruptionBudget handling of "strategic merge patch"-type API requests for the `selector` field. Prior to 1.21, these requests would merge `matchLabels` content and replace `matchExpressions` content. In 1.21, patch requests touching the `selector` field started replacing the entire selector. This is consistent with server-side apply and the v1 PodDisruptionBudget behavior, but should not have been changed for v1beta1. ([#108142](https://github.com/kubernetes/kubernetes/pull/108142), [@liggitt](https://github.com/liggitt)) [SIG Auth and Testing] 690 691 ### Feature 692 693 - Kubernetes is now built with Golang 1.16.15 ([#108565](https://github.com/kubernetes/kubernetes/pull/108565), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] 694 695 ### Bug or Regression 696 697 - Bump sigs.k8s.io/apiserver-network-proxy/konnectivity-client to v0.0.30, fixing goroutine leaks in kube-apiserver. ([#108440](https://github.com/kubernetes/kubernetes/pull/108440), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery and Cloud Provider] 698 - Fixes a bug where a partial EndpointSlice update could cause node name information to be dropped from endpoints that were not updated. ([#108203](https://github.com/kubernetes/kubernetes/pull/108203), [@robscott](https://github.com/robscott)) [SIG Network] 699 - Kubelet: Fixes memory usage within the kubelet's inotify subsystem. ([#108335](https://github.com/kubernetes/kubernetes/pull/108335), [@rphillips](https://github.com/rphillips)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage] 700 701 ## Dependencies 702 703 ### Added 704 - go.uber.org/goleak: v1.1.10 705 706 ### Changed 707 - k8s.io/utils: da69540 → 6203023 708 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.27 → v0.0.30 709 710 ### Removed 711 _Nothing has changed._ 712 713 714 715 # v1.21.10 716 717 718 ## Downloads for v1.21.10 719 720 721 722 ### Source Code 723 724 filename | sha512 hash 725 -------- | ----------- 726 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes.tar.gz) | 3347bfe32a12f2bc19a43197929e4cd032d2def13e818b771fddb946a511c0889676078e3ad7c350b86ad9cc5ea297b5baa23afd20921c78e30a857c679e0189 727 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-src.tar.gz) | 8c40b76190c1b2b8645e37010fb0d966751b9600f3c9ae2ac0d431e695284b0c1626795f5783c8b5a711539e4ea62721d69b9cbd2f9ca5d1b149f67c453d46e9 728 729 ### Client Binaries 730 731 filename | sha512 hash 732 -------- | ----------- 733 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-darwin-amd64.tar.gz) | 2604fabb39dee3fbe39da9ecd8169724b7ec67e4e67ca02a452d3f053e070223d0f0168e86309080e0f8773d11f2c13274d308bd46b8b4fd25179ab08d5169bf 734 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-darwin-arm64.tar.gz) | 2a3ff3686a201247291ae1d32c1653ce3f8372f5ecd9a486bff7f2cfa52acd9af693db13341f75fa78b2a9a73723e99f5ff3372c0e2177b38745e25319e49264 735 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-linux-386.tar.gz) | 97d3dc330fe89810b5abff539c05817d61f8aa71f691f460e1b07c827152b1cb031bbbfec656d76245e1a7902a3ec6eb583f2246e581e245b14c6c48c8f0c81d 736 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-linux-amd64.tar.gz) | 6ffd76390fe0c1d1e550930d4abdfc73198e63baa2f6a4de0492c2254aee88204b820fc055bcb58573405a272b1339744c90737fd9528380e88b86e3aae10e03 737 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-linux-arm.tar.gz) | 0ef6696d1d2f6790c2ad9ea75ae574eff50ac9b32e1ba869bd7fa13fecff3dbdab237a9ce45efc380afc774c1c31d0c7c2c5bac3971a9aa185de22bf3185042a 738 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-linux-arm64.tar.gz) | 7d49c30e86fd20ef28d1e43e333fc81869d844f5ae30a17a160833b5b4a6ffa65862ab470723d6d64ed7b3ad5b3105cff3a9ae91e8314facedd8184bd9d69052 739 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-linux-ppc64le.tar.gz) | d3b8552eeb796daa6d485d26e5b961852ff8f47f9ed8de98130649d6f022a37a2f8cab538b72921873da93454b92cb429989588f7cca8ced2335373a7f26e496 740 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-linux-s390x.tar.gz) | 7f16fb933a649f7863e87d2a46f1f2d583c7205d9f81371938646eaa4e16fed8cf30d43a24752141af29bf90ddb5012156d69c75b7791800a187f48248532ee0 741 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-windows-386.tar.gz) | 0294286299f0c0e3767c64cc502a72069e0c6345e385ea88fad7031acb65d0886e6a6526a8e6b2101fb02720854ea72e63df5103eae509d2656a8045d7494e08 742 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-client-windows-amd64.tar.gz) | 7d8cf06ac466f66e0fa465934f92aa0aa4a341840a6b64ed5769e055598de8eef768efdd371d66cceaf39a5f7abbfae4fb448e15d12c8abd5a1c65c4813d3c43 743 744 ### Server Binaries 745 746 filename | sha512 hash 747 -------- | ----------- 748 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-server-linux-amd64.tar.gz) | a3b22da73e9f5b88cd9b589363492cd70c85e5fdef90ed61fe1b118a6e16d080b00bb8bbc888efd03b836bf55caf44f453c49af77f2af209de7979db42765b94 749 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-server-linux-arm.tar.gz) | c0bb83fa395b6c86e5dad69a7ef5944bd1220896918ea04a1adf7203ed7c61ef52dc520952d7471e4f6c7423785109272d44bc1196bcb345cd131fd9abf8833a 750 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-server-linux-arm64.tar.gz) | 6572f557d7b0666a6527873fd0b6b996c0ed22031337d1fefacfa176555d999d5a31fcc4d48cf9d6c250776bff6d136c95b03f7bbbf2ea5a2835fd2dd135cd9d 751 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-server-linux-ppc64le.tar.gz) | 5b2376e4eec6e22f987a0ab9b5a63602dc8dd987d87a8a71e2648e2033d484b09446ea65b291489548785d9dccc4174ef12f2ae49ab58869a6fa5ac14240a223 752 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-server-linux-s390x.tar.gz) | c454702e6de40d0ae89bf5c28839a5ceb2e11084bcbe9a4e18e291843e6f1584e4ecc4e3443f9d1779d859523785c737ed5ee264a6496f2603801f8b2de2caed 753 754 ### Node Binaries 755 756 filename | sha512 hash 757 -------- | ----------- 758 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-node-linux-amd64.tar.gz) | 2511056ad624a2779f7e17209e8b354aedb555febc2b1a71c5ebd06599644b98951114b1cb50573504dfdea8e333094a54db5338f54be8a83263c2a596a79151 759 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-node-linux-arm.tar.gz) | 8eed7587c401291bebdf2da2bf172c648abbdc950929257688fe7becf72222a250496a9e2aef1a63c32ce66a39b5d109f46930b745ec7457734b81900c278634 760 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-node-linux-arm64.tar.gz) | ea7740c957de8c2bbad1b27629eab0b5d5fb65574125e0b1ae1913a5b3fda7e91f5e2d4aae70bf9fe1145e9bf243b5c3b36b87cc41d75c7c496b35ed1ab625c8 761 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-node-linux-ppc64le.tar.gz) | 188a2193839ee2991919c7e34f5789be1f86176027c3866a189060d1600f821ad688f86053fff8d4b3b658bd81331b0754fcbdd4d2096100ff427e65fa2459a8 762 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-node-linux-s390x.tar.gz) | 7ccb6611f966366c9af02e24e19f0e5133792a4237c9462fe677ea2b5bfd9914f328cbd0c29dedc917ecde9fa90d48535dc2e7cc258bf487b41595fe414037d6 763 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.10/kubernetes-node-windows-amd64.tar.gz) | d0de58045f6930652f0aca3d21e671e4163deb29e63fabec454fb0d3e72282feb40554baecf9739f4458434e5a8562867ff33f6237c0a74a44f074f4061e4602 764 765 ### Container Images 766 767 All container images are available as manifest lists and support the described 768 architectures. It is also possible to pull a specific architecture directly by 769 adding the "-$ARCH" suffix to the container image name. 770 771 name | architectures 772 ---- | ------------- 773 k8s.gcr.io/conformance:v1.21.10 | amd64, arm, arm64, ppc64le, s390x 774 k8s.gcr.io/kube-apiserver:v1.21.10 | amd64, arm, arm64, ppc64le, s390x 775 k8s.gcr.io/kube-controller-manager:v1.21.10 | amd64, arm, arm64, ppc64le, s390x 776 k8s.gcr.io/kube-proxy:v1.21.10 | amd64, arm, arm64, ppc64le, s390x 777 k8s.gcr.io/kube-scheduler:v1.21.10 | amd64, arm, arm64, ppc64le, s390x 778 779 ## Changelog since v1.21.9 780 781 ## Changes by Kind 782 783 ### Feature 784 785 - Kubernetes is now built with Golang 1.16.13 ([#107615](https://github.com/kubernetes/kubernetes/pull/107615), [@palnabarun](https://github.com/palnabarun)) [SIG Cloud Provider, Instrumentation, Release and Testing] 786 - Kubernetes is now built with Golang 1.16.14 ([#108102](https://github.com/kubernetes/kubernetes/pull/108102), [@xmudrii](https://github.com/xmudrii)) [SIG Cloud Provider, Instrumentation, Release and Testing] 787 788 ### Bug or Regression 789 790 - An issue preventing the daemonset controller to retry on status updates that failed because of conflicts is now fixed. ([#107639](https://github.com/kubernetes/kubernetes/pull/107639), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Apps] 791 - Fix Azurefile volumeid collision issue in csi migration ([#107575](https://github.com/kubernetes/kubernetes/pull/107575), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] 792 - Fix: delete non existing Azure disk issue ([#107406](https://github.com/kubernetes/kubernetes/pull/107406), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] 793 794 ## Dependencies 795 796 ### Added 797 _Nothing has changed._ 798 799 ### Changed 800 _Nothing has changed._ 801 802 ### Removed 803 _Nothing has changed._ 804 805 806 807 # v1.21.9 808 809 810 ## Downloads for v1.21.9 811 812 ### Source Code 813 814 filename | sha512 hash 815 -------- | ----------- 816 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes.tar.gz) | 525d981f5fa61abfd52df07e49a3bbe87a11e1e654d63e601124e35fb5b82d23390601562300cb0c1f62bdacb3bf9bb73c91067ab2cb0cb47b9b9cb34aadd289 817 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-src.tar.gz) | e9f506722ab67c594b46a082e69661812eb0b2562d66204d497067c128985bce91e11c0fcb67c59901baa35bb974f5daf04bd825252be00cd9ee9d0aba76d12a 818 819 ### Client Binaries 820 821 filename | sha512 hash 822 -------- | ----------- 823 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-darwin-amd64.tar.gz) | 5598923aa727cbc897c93e15ece19db73cd8f6644280a13d8f34ccba8f19a16b1e58406cb9f54b23fa189402af8c528886487359543a995e17630f8bb4ff27cb 824 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-darwin-arm64.tar.gz) | 4f37beba73fec432d647c4113af1612e4132fb75e8c87536116acbf1fd5c53b54013d29bea7c3207ced9e27b05bc620f7c0dc17b56f3d485b9aa4612c7b5e93d 825 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-linux-386.tar.gz) | 7520d505175857e390e84494c076223a64c448b3a62411a131ae40df375ea9df176b0ba48aa80fd4989e7cb44c2b29cbb6384aac953e7569f70dc3044356ccaa 826 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-linux-amd64.tar.gz) | ebc42133afe7c54095ae72b2b4ccb98c2a450701671a584ee6659a78b091bf91f7f2d161b77824d8d6c7ab381f68ece3dbac59f9bf4f7e5b8aaec9f1c27dfed9 827 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-linux-arm.tar.gz) | 3573921054b9f620651a965fe3acb4c4dce621c657022f23e53e78fcce2a9e2f1ad006946795fc19678d4eed660252941d7bd33912bf08d514da28bbbf675073 828 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-linux-arm64.tar.gz) | 4e0b091dd44f175f603caf8ed3162707094c28c96b04b1d4dfc4f6b3f9ffac2f52d60a66651e91bf263749a2543f2847581bde3658743a5197fed018b25038e8 829 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-linux-ppc64le.tar.gz) | ca627f1f47d04cd25490f967a9c7717cf0dd77ace759f6144c85cf081d7644b2367934ab1842f9a4f4c58519cf1042e4a11dd91cb63c165f3f542e70856f9c8e 830 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-linux-s390x.tar.gz) | fe1ebe7660ede23cb1689401d71be77ff96ce3726a9fee382207e3c719d55db6d82696ab1890ee64a7f30148189be62323766a8d75140d4ed639799354f21a76 831 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-windows-386.tar.gz) | 3d44ec09edaa4481c750537fffebe1dfd0cea84039383be1b38d842f5e6e5b7853d14d58dbd05c4d9d22e615762028e47b33951ccc8e9b0bdcb284dd57b53c80 832 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-client-windows-amd64.tar.gz) | 34ec7389e0966d2579bd703ac56b5ffbf448f1c1bf48f674ee54ef1bd26dec732bee55c3e140005722f1dda97c4c8dada66712d572cb4cd784fa1fab7ba95a68 833 834 ### Server Binaries 835 836 filename | sha512 hash 837 -------- | ----------- 838 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-server-linux-amd64.tar.gz) | 574eb1cfc5b36bf93df361395212149d8afcd6ccf8a410ab3080484744f37f264cf4a96723b155bc754748b9c8a21d02c8eb00b474bf060ce96b92e0a0e99ed1 839 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-server-linux-arm.tar.gz) | 325b7dd1d14ff08bb04e8db15e20fd8e42c67b38de80f4a056931673a4bd3d2542f575adddb5c79c489bf020061ad52799c7db0fd76ae19be3edb0e17a42279c 840 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-server-linux-arm64.tar.gz) | af6f5d62c400113ec980cc135a249ae53c691a30258d28421e1e43c469ef0712c0ba36048a73c445c489d1282f8d8323f91af2b69edc8f47846a11aae4a239d3 841 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-server-linux-ppc64le.tar.gz) | e53c8b74714e32955be63342785e7a0888de9c8a540d59031489f2d5003cd388ff27abb7e5a61a097463ba88453ddcd0d43699137f6752e4812d5ba4c5a2893a 842 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-server-linux-s390x.tar.gz) | aa2ca699498a196d6cb6a7773c2e49f628e64b8f54fd28f962a030cf25f025d55438fd092b694835d9f5b6781e1f2a9cced0e48308c6bf2e95f8cc68f107a4b6 843 844 ### Node Binaries 845 846 filename | sha512 hash 847 -------- | ----------- 848 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-node-linux-amd64.tar.gz) | ecfbddd6d7e21cb1357001f9bae2adf678b382e2ef9229e573c7eaa267b556ffcce8c6928651e51f497cf40123b31cc6efbae9951d3d3872078f7dd683b8fac5 849 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-node-linux-arm.tar.gz) | 578dc334690254f00d346d443a668fbf9dfd8920ec2b4db3aeea221aa5ad21759a1862a05095c463e5135e2f2e7dc6926c683dd41bc5577bd0b0cf43d931d608 850 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-node-linux-arm64.tar.gz) | ac089d151baa9c3b6dca5825f569a071e8f6be61eca9f0cd17eedf8a0f47455fb4f12f9f662e3bb296937122a5460cc6997408a93551b641bef52d7237021f1a 851 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-node-linux-ppc64le.tar.gz) | 1642ecce783b0565e516330bdeda454ae1c617873a5d652b5b783b03c94a760a37e9a9c83ca63af9888c8fb6d3ffe2220e9403060f769802bff0f0a907567677 852 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-node-linux-s390x.tar.gz) | d50dbfc328e75ed3a253fb2bed141ed84428e9e62a0482a4407c621d0e74c647e9887a847e0577b2b3386f977a21a427dafa2eb3edc9c0480cae51f1165a7748 853 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.9/kubernetes-node-windows-amd64.tar.gz) | 29b8fc0a4e9eace791df958ff36166c5913a1fafa2113f86605b73a7926abe0dfcd686e7e7d88302c4c5324809b8dfcb46dee9c7eaefc6f9a0e0a21171b03d9e 854 855 ## Changelog since v1.21.8 856 857 ## Changes by Kind 858 859 ### Feature 860 861 - Kube-apiserver: when merging lists, Server Side Apply now prefers the order of the submitted request instead of the existing persisted object ([#107569](https://github.com/kubernetes/kubernetes/pull/107569), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Storage and Testing] 862 863 ### Bug or Regression 864 865 - An inefficient lock in EndpointSlice controller metrics cache has been reworked. Network programming latency may be significantly reduced in certain scenarios, especially in clusters with a large number of Services. ([#107169](https://github.com/kubernetes/kubernetes/pull/107169), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 866 - Client-go: fix that paged list calls with ResourceVersionMatch set would fail once paging kicked in. ([#107336](https://github.com/kubernetes/kubernetes/pull/107336), [@fasaxc](https://github.com/fasaxc)) [SIG API Machinery] 867 - Fix a panic when using invalid output format in kubectl create secret command ([#107345](https://github.com/kubernetes/kubernetes/pull/107345), [@rikatz](https://github.com/rikatz)) [SIG CLI] 868 - Fixes a rare race condition handling requests that timeout ([#107460](https://github.com/kubernetes/kubernetes/pull/107460), [@liggitt](https://github.com/liggitt)) [SIG API Machinery] 869 - Mount-utils: Detect potential stale file handle ([#107040](https://github.com/kubernetes/kubernetes/pull/107040), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage] 870 871 ### Other (Cleanup or Flake) 872 873 - Updates konnectivity-network-proxy to v0.0.27. This includes a memory leak fix for the network proxy ([#107188](https://github.com/kubernetes/kubernetes/pull/107188), [@rata](https://github.com/rata)) [SIG API Machinery and Cloud Provider] 874 875 ## Dependencies 876 877 ### Added 878 _Nothing has changed._ 879 880 ### Changed 881 - github.com/google/cadvisor: [v0.39.0 → v0.39.3](https://github.com/google/cadvisor/compare/v0.39.0...v0.39.3) 882 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.22 → v0.0.27 883 - sigs.k8s.io/structured-merge-diff/v4: v4.1.2 → v4.2.1 884 885 ### Removed 886 _Nothing has changed._ 887 888 889 890 # v1.21.8 891 892 893 ## Downloads for v1.21.8 894 895 ### Source Code 896 897 filename | sha512 hash 898 -------- | ----------- 899 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes.tar.gz) | 84e6742f2341a7288b23dae90111a377621babf1bdd9e167a57c31b577b95594fd451e9acf21663e6f365f1a7cc6c983028819036facdb35066576b341af8d62 900 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-src.tar.gz) | 93561a6fcca2bfd6491643c33f10068d4cabb2ae6304b24955202d1d97af04dc9255f2491aa74de26b4754778561009fc5f81782c18d53e72cd373cecf077ada 901 902 ### Client Binaries 903 904 filename | sha512 hash 905 -------- | ----------- 906 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-darwin-amd64.tar.gz) | 55f84b9951126f1d720b8f6ab7a91af0718ac606c96045fd947156e51d8c63a81ca3338cb00b22702a6f14da74acaa4535c9f62a47483c8518c7e661e3b17b2a 907 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-darwin-arm64.tar.gz) | bd58541ebd2112b4180d2b70bdd4beca590908f626ab1e1d2a4ac39c4a1091de88f7e0d292ac8419269ac25b0d6206f5437a0290803ff97832ccfe8136686121 908 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-linux-386.tar.gz) | 74daacebbfd85e7b429672386adf4a9504d3e447d13667c15ffe625fc0c3c9d5d70c65aa6030631d30948717a39125bd22b33bbb570ca01cd8042b45127a72de 909 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-linux-amd64.tar.gz) | 36ee8e17cd088d02eeadf8279aed5d1e64316c2cd7cbb81857a83035f8e61cec2ac34b1243863cf987f1833efa0bf31cf83d7165239ba7de3a4631fb7454c3fe 910 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-linux-arm.tar.gz) | 0f9db9b2b4b46da8bfc7ffd3c93dfc59a9a8400553ecf470cbb6d5d3fbb928ff513de2d7566151a8d4b15d306336f0028c82ff4a007ffe529541e87727e172ee 911 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-linux-arm64.tar.gz) | 275e4f1185949ee39aba61690434571ab219b07b277ad3639e09ec4bd2ebb1cd78f255e9ffe9363713707103ee3b4290471b1ea3f3e7902a5d58010245d7f941 912 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-linux-ppc64le.tar.gz) | 8a8cadb86b1d6a3aba33f596f58dd15fa91ecc7414b005c018ab0fe40159caa07455dd8ff6294bd777550944015987ef5971598f4bce05e2c6f34a5d760e8fde 913 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-linux-s390x.tar.gz) | 112e1d63bd92553f45dee1d3ff5748cb9b20ad92117b231efaa1336ff5a1f5fa53c9d6d455c1dd88c30a3557513bcc38103ba0bbf386ea1e4b0d057a01c6fd30 914 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-windows-386.tar.gz) | 808f139f0e542e95f3537ce663a3bd781f3769efdd33210030a72d70e507c5e32428ce54fef5891814e1be91fa377d9ab6d6d5f1da7f39a56fc6afb62b2996f3 915 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-client-windows-amd64.tar.gz) | 96ee2b00b0996f7cefcd8fffa68263302a6d6ee15a6ce99d92b06d912249a43eb3f952d051c8f237ff9337e4cdc496ddfbd4d6f06d64312f61089b33655991fc 916 917 ### Server Binaries 918 919 filename | sha512 hash 920 -------- | ----------- 921 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-server-linux-amd64.tar.gz) | d059b68bb9fedd24af0b70debc8074ad61c223f787fd0d6867c3e3ee19d907f0cb51dd41dbb5c82dcaee2b9383e6d5c5f48a46c9957b228705f95cb0167fbcdd 922 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-server-linux-arm.tar.gz) | 8cb9cc950c4accf39929da5f885dade95ab0b741c8d93fe61986bdd36e43c508073d6d5bac3b8bf274a5f2dfc541c6a1ea8e73529ec8e2ca1fa06eac7bf5babf 923 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-server-linux-arm64.tar.gz) | 34e1ec0dee4878eb4b0c37b0452257d9426a66558fd133ce838953eae1cf34edc21c994216d153d37c76645fb17f0392698c54b793d98e0d6d22d7c72bf39a92 924 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-server-linux-ppc64le.tar.gz) | 9262613f51ebb57db5c87d2e62960a37dea91f33bd5257796f2d001d0d96c1bf40f2679d9ece88f749215d18883c3a15ac6b7eac9c84b8f3f6efd8da73b1c42b 925 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-server-linux-s390x.tar.gz) | bd54d2af171e89b732a78494adc5f5d1e776a716eeb0ac8b910fcd780426980adb23655ee84ca49a72227c160c3445b4c77e8426df53b0fd251ec60c181806bd 926 927 ### Node Binaries 928 929 filename | sha512 hash 930 -------- | ----------- 931 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-node-linux-amd64.tar.gz) | 2f85dc09e62febd5803e1d564339c4905d647bc8ac972f66dccc1aca4b5cab4ff592e8bd342adcb88f0761b4a15f5cf6d423da6be3a1ecc2639700b7d64b3523 932 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-node-linux-arm.tar.gz) | d972321851c3d9ec10650d2df7728de98528304fc6ee2980de45e2d99a165f4cd468cacbf4e92469bcb2f2312eef2148575d239fa17244656bc8a217f7a0fd0c 933 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-node-linux-arm64.tar.gz) | 6a9d574c10dd2ffe729d28dcf147aa8e6dfaa0127be130e5b7cada4ec3e8fb241012676511c55ae4ac4bb71c385647013b4cf38027e61876a0ac51e9006b3622 934 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-node-linux-ppc64le.tar.gz) | 1bbe450cdc3df9732e9a95eb2dedf4e218db700f20c9e861eab0db944631b4fc8cbcf71c188eedfb4273fdc66349be00adc4047376decb11d77176cb295cf856 935 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-node-linux-s390x.tar.gz) | 05b23eedbb06650f2aba4d6ef0d575a7ee3dfdb4b2c0c8d5b0e29e2d0e15ee7d620190a05f2dc2fdbde1b43356f32211763fca4649cb2e84ded8223ad64e6c76 936 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.8/kubernetes-node-windows-amd64.tar.gz) | df924885dc3d2ad706c0a2f793e4e9d6104ea1f892195f972128786c35019ff2e5052448ee1bcd925ab177bbaef9afa3c64afc7f0cfa4e774a3a16fe3a01cd67 937 938 ## Changelog since v1.21.7 939 940 ## Changes by Kind 941 942 ### Feature 943 944 - Kubernetes is now built with Golang 1.16.11 ([#106839](https://github.com/kubernetes/kubernetes/pull/106839), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] 945 - Kubernetes is now built with Golang 1.16.12 ([#106983](https://github.com/kubernetes/kubernetes/pull/106983), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] 946 - Update golang.org/x/net to v0.0.0-20211209124913-491a49abca63 ([#106961](https://github.com/kubernetes/kubernetes/pull/106961), [@cpanato](https://github.com/cpanato)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage] 947 948 ### Bug or Regression 949 950 - Fix: skip instance not found when decoupling vmss from lb ([#105835](https://github.com/kubernetes/kubernetes/pull/105835), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 951 - Fixed SELinux relabeling of CSI volumes after CSI driver failure. ([#106553](https://github.com/kubernetes/kubernetes/pull/106553), [@jsafrane](https://github.com/jsafrane)) [SIG Node and Storage] 952 - Kubeadm: allow the "certs check-expiration" command to not require the existence of the cluster CA key (ca.key file) when checking the expiration of managed certificates in kubeconfig files. ([#106929](https://github.com/kubernetes/kubernetes/pull/106929), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 953 - Kubeadm: during execution of the "check expiration" command, treat the etcd CA as external if there is a missing etcd CA key file (etcd/ca.key) and perform the proper validation on certificates signed by the etcd CA. Additionally, make sure that the CA for all entries in the output table is included - for both certificates on disk and in kubeconfig files. ([#106924](https://github.com/kubernetes/kubernetes/pull/106924), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 954 - The scheduler's assumed pods have 2min instead of 30s to receive nodeName pod updates ([#106632](https://github.com/kubernetes/kubernetes/pull/106632), [@ahg-g](https://github.com/ahg-g)) [SIG Scheduling] 955 956 ## Dependencies 957 958 ### Added 959 _Nothing has changed._ 960 961 ### Changed 962 - golang.org/x/net: 3d97a24 → 491a49a 963 964 ### Removed 965 _Nothing has changed._ 966 967 968 969 # v1.21.7 970 971 972 ## Downloads for v1.21.7 973 974 ### Source Code 975 976 filename | sha512 hash 977 -------- | ----------- 978 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes.tar.gz) | 272b73d76a8d90cea95940bf5b67eb04b9b5d118734398072955b556a9c05f72b5a200b1d546503a2ba106283d7e60c0c2634c2886bcc9fc3d6c0e045c3290de 979 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-src.tar.gz) | ca3de4042795d663c1a68a2a782842904b7f95eb91b9e4c87db0c161fb510b4dcdbe7b1d8209a2bcde9f3ba7cb9ce49814483fb11bad693391017bef1bc4c1fc 980 981 ### Client Binaries 982 983 filename | sha512 hash 984 -------- | ----------- 985 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-darwin-amd64.tar.gz) | 4bc06c7b023d7a1f5b592c97b46c979e89e168d188d1bcf8999256a38dee63122a4bfd5756fc881e6447752c0098268769e125863162c3ffd36fd53d905fbe68 986 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-darwin-arm64.tar.gz) | 73a0e841f631cae0f7a84c3460187e7c8718f77d82e13af57204ba83cc40e1421ae423a30c4565234cfae5eaf7df7d8f58cdb707dcdf6b050054b69d83592c78 987 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-linux-386.tar.gz) | dca88b5c60a7e512195c29157a25b3b3702d50c24b3b90f719b1f887bd1c4284dcedc8f9358e5828fd6bc171d3797789749a6969b7294a9023472738eb4974bc 988 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-linux-amd64.tar.gz) | aafab261dae85c3a4a3e4affbf8e24d62a80c34c6fdba312cf741588825cf50a85821a8afbf98430db879c3f6a375b566854071b4235ab1c19ad2c201c357f3e 989 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-linux-arm.tar.gz) | 08441420659c960c1661fd2c4d4b68dd2bc4b51e63e14d08e28aa79d1835f302a9472ab6cf6c4dd6dd1e23a78cb03e509107c223d3c35cbdab3fe387dc36810a 990 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-linux-arm64.tar.gz) | 4f90190543a02e65bae7f21ed4ea9af870471703bbf47a5cb89e589784f467b11aa2212347656aa9c2b1652b923267d28c90df543a64fb7cd9e93d542a7dbaa6 991 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-linux-ppc64le.tar.gz) | 7e8d37ac1f5b17bb18396e5d1ed60533b8634608a5c33aaa9a477544dba7b78b6f3aaab850853a05d4da0e665608ab2bf290ede2644bc8c0d73fab2b6530979d 992 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-linux-s390x.tar.gz) | 9843d0f53f9113a70701b407dc857d15103e7161abbe56c6267935c33eb2851a79537e1b51e9defd48bd85b0177ebd3294b95759fdf1bee02725c67a2604c37b 993 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-windows-386.tar.gz) | c5785c684e604fa67250cf778bbae5ff33aa114358ac450c2cbd6ce333dc1e11213682e3f1de7dc68fdbdb860f526135c1847bfccdcaecf63fb00e042e07a057 994 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-client-windows-amd64.tar.gz) | fb65022c279fcdb5e6dc6618670126aa9ed08d641c152b62c603bd7b40840a47a1ccdf8811df08ef3b5c000bac3a0d84ad250db50688055444ce2510073a5e78 995 996 ### Server Binaries 997 998 filename | sha512 hash 999 -------- | ----------- 1000 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-server-linux-amd64.tar.gz) | 720cbc936a81f117c479456b939a8e2920e235a70c0111cb35e77acd71d9cc0da4d8fc7668ef6458177715d2b3419871ffbd64d8d5c55e34bd76f97cc1dd596d 1001 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-server-linux-arm.tar.gz) | 47481f83ac7fb801a5df8a6ed92fc91af3a001250749b5523450332d8335afd27119599e96aee37983bb424b3ccb7abf81df577b4816d36b4da68948781193f2 1002 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-server-linux-arm64.tar.gz) | 3ddbdf5e4c8274dd3a8e717454feb4c6440d10772ebadde64f37de90debf95dede8a5b005a79f881c9524bbbbac112fcdedaa7bcfbedf23642bd8720caac468c 1003 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-server-linux-ppc64le.tar.gz) | 440bd7b2972623d1347f494e6b17b8a55a26056f5f259bc31c4ec936654c9e8583a7d8eff4b8f9e458b00747d0b119a294175f59d338ca693941333577634971 1004 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-server-linux-s390x.tar.gz) | b749113d963f2b3f56025fa8d832f552be5b6ff814fe5888c91d934e5cfadbd2ce0e6b3b8042257c2d923d4628c1cafcb339650d22d099e8cd51e841fa245328 1005 1006 ### Node Binaries 1007 1008 filename | sha512 hash 1009 -------- | ----------- 1010 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-node-linux-amd64.tar.gz) | e6a7f2507f6a4b2f36523573deac0ad3d7922638ca7a0b8c0f8fda32cd512cf342a66ab041d43e7c853963479eaad6c59d4d2a96c59c8cbe18b48166cc23125e 1011 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-node-linux-arm.tar.gz) | 74512bc870bd4b678cce0951f6b62143a5863b429178c1a34f4c83126c7560632a39ad43c77cdf8f46bd75ebab32f431f04e45f49dc101d1ee8d205201eb5233 1012 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-node-linux-arm64.tar.gz) | 5d983437f08cdb402aa5204ad005df9e8e3bbac0853477d6f4d61887e5cfdcdacb3de5fd91b13a8dacf0a66c9b7f96a73ea3a3c87c6ea997dd9c9f949089a2db 1013 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-node-linux-ppc64le.tar.gz) | 63b2c9aaf54e053630690be69a33a5914445bddb2609843e4f8d3c89006bea234e77e4d0ece8e50c592142f401c0d7cc57f7a82074656c1035ed2ca267e1f58b 1014 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-node-linux-s390x.tar.gz) | 3beef616efce3bdda0dee08df850762d9f2945f08b6b377519058daaf56856325b54d892c9233dab3f371195079f110e91e668bbd049ae855790e1373f9899d2 1015 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.7/kubernetes-node-windows-amd64.tar.gz) | 989bae0ba9ce584c2906392d0629446b1cc662878339115e6136c69210d053092ca7e3114ba976976c44f06050198ff8bd0806d0c8a8b5d8c80ed988d115c87b 1016 1017 ## Changelog since v1.21.6 1018 1019 ## Changes by Kind 1020 1021 ### Feature 1022 1023 - Kubernetes is now built with Golang 1.16.10 (#106224, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing] 1024 - Update debian-base, debian-iptables, setcap images to pick up CVE fixes 1025 - Debian-base to v1.9.0 1026 - Debian-iptables to v1.6.7 1027 - setcap to v2.0.4 (#106147, @cpanato) [SIG Release and Testing] 1028 1029 ### Failing Test 1030 1031 - Fixes hostpath storage e2e tests within SELinux enabled env (#105787, @Elbehery) [SIG Testing] 1032 1033 ### Bug or Regression 1034 1035 - EndpointSlice Mirroring controller now cleans up managed EndpointSlices when a Service selector is added (#106135, @robscott) [SIG Apps, Network and Testing] 1036 - Fix a bug that `--disabled-metrics` doesn't function well. (#106391, @Huang-Wei) [SIG API Machinery, Cluster Lifecycle and Instrumentation] 1037 - Fix a panic in kubectl when creating secrets with an improper output type (#106354, @lauchokyip) [SIG CLI] 1038 - Fix concurrent map access causing panics when logging timed-out API calls. (#106113, @marseel) [SIG API Machinery] 1039 - Fixed very rare volume corruption when a pod is deleted while kubelet is offline. 1040 Retry FibreChannel devices cleanup after error to ensure FC device is detached before it can be used on another node. (#102656, @jsafrane) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage] 1041 - Support more than 100 disk mounts on Windows (#105673, @andyzhangx) [SIG Storage and Windows] 1042 1043 ## Dependencies 1044 1045 ### Added 1046 _Nothing has changed._ 1047 1048 ### Changed 1049 - k8s.io/kube-openapi: 591a79e → 3cc51fd 1050 - k8s.io/utils: 67b214c → da69540 1051 1052 ### Removed 1053 _Nothing has changed._ 1054 1055 1056 1057 # v1.21.6 1058 1059 1060 ## Downloads for v1.21.6 1061 1062 ### Source Code 1063 1064 filename | sha512 hash 1065 -------- | ----------- 1066 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes.tar.gz) | 1a88a2f9d2ab9a5d7abe16d788c0d690a217b08313e94c923911bee71c9d0520ec2c9f09bc79d52765a26ad56cf4945725d28c8c3564cbee97d563934132ceaa 1067 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-src.tar.gz) | 3816ddf61d578052c53eba635ad2f4f1e160ca267bd13cdf6d4a6db551a65c2ef62ec1542679dbf75c2befb8d347bec9ed52ac1ea9eeadda3463ba500744a1c2 1068 1069 ### Client Binaries 1070 1071 filename | sha512 hash 1072 -------- | ----------- 1073 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-darwin-amd64.tar.gz) | 5eaf7ca84cf1fa0bb41581bf99bdaa08de51e56d5345cc03c45eb5082952cf0ac9d2c3cf34c9fbd7798bf2ee9c4d27f8bcd1b7580ce9b49fba1e7861f1524502 1074 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-darwin-arm64.tar.gz) | 21bf5b81d36f97a4ee4f83f52899588738c542dfbaef21656468156e0a4e247e02c970a794089be5bc4046e5e371a967286affa103f507de0a925f99bea0695e 1075 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-linux-386.tar.gz) | 302e80b5e868c867245c4cbd110670dab5db4847f96de6199c4005d3d28f6a0945a73fddfebdea05bdab6af79da7805e74995d59a66500802584c9c9f1d65a6b 1076 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-linux-amd64.tar.gz) | 79efb1b99acecbdc605b2cd6cc5c563ed80bf0e433b444908778fbcefeae34334d254a2e4c7822bdbadf64215ecc826614956c27d5f8d7094968dc8bcc14d04f 1077 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-linux-arm.tar.gz) | 4edb5f26636d5d6e9bb463bfbca794e16a1ca1e641ad02e024f23c46381d5dad361edbfa6ff35abe7a7ddac5c745207e23900968171a6990168c74ade569747a 1078 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-linux-arm64.tar.gz) | 95857d18fb3e4f02408cdd99eade98a8275e8d5663104c235a4bee0a5dd256c8bfc57261c59d80a8d5da16df1205eeedc05b74ea6e76330dedf044183e3d3886 1079 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-linux-ppc64le.tar.gz) | 51df9a87c5bf1d746afcf13333367d9aec18d10db49ada29b8eed8e25be9714bc16fb46bc9bfc5b2afb6c218b1daf4e71bfff5339473e4993ddf35efe3676545 1080 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-linux-s390x.tar.gz) | 2f4701611b3b83326f93b510e5a0387a547984abab9f5534675278ebee5fc97832b314bd0a061ae21dbbfdd04e978004291215f52530b9e344e1df07d1830472 1081 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-windows-386.tar.gz) | ad8dc908960f61e3f0aed29d5e5a95fcce06fc9859f8dc47086d4341c044d04259334f484af2c42221824bc8cd9d68feea8f8f2dca22ff42f9df1d3be3f910bd 1082 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-client-windows-amd64.tar.gz) | afa5a32c131378dac1892a93169f6dd861111c90743c923a979731ff88898898752afe1c696eb26aa5a0cf421e0945e5a04a37389cac01087e3965f7e89fa814 1083 1084 ### Server Binaries 1085 1086 filename | sha512 hash 1087 -------- | ----------- 1088 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-server-linux-amd64.tar.gz) | 03a0243fbe69a1f11f72fd7d670862a219074dea872dcac7b9d580051ac5d005015a1bfcab239c8ca90c0ba3f22aeb98b8f315b76df308abf8dcd94d27d4d2de 1089 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-server-linux-arm.tar.gz) | 99c7563d85df914516c8f8f2d4d0118583103e1e32f31a37690d7eb24057382dd9dcf682d34f4d182d9ffb30c1b47f58ff42f89e34eb6cc6221401942e71b40d 1090 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-server-linux-arm64.tar.gz) | 7625075f3708dc0d453b0e7f63a9526cb68412ff20e75cc0f9e7f08712b3bc9a0c6f5fbcad84f941f5b8de4a023e5c7de8332c47395cec93ff092c302df806de 1091 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-server-linux-ppc64le.tar.gz) | e8428545883e5a5c5b66c50125854417f9a9f536d4b18a0b56e829ba5412839cc3cad70807d3f6843e8f2ff71b33ad6df6fe48da89ccb39bcd8f6499aaa55257 1092 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-server-linux-s390x.tar.gz) | 9c0ab0b5fe01817c6485a54f660a5b4bd452b50c412c7eccc209db1758e7b7f9e3b62f7a14d409ab4917400dd7f00ef7a87912a127dd007916b7395c1c3d8552 1093 1094 ### Node Binaries 1095 1096 filename | sha512 hash 1097 -------- | ----------- 1098 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-node-linux-amd64.tar.gz) | 1e9d5b2be8874b1f87e8d551c1020d321988652ad0fcde59836adc6d4eb18c99c44aa0fb5923eea3a1daa76eb362be5dda9bd0eafc5b012a733d6f06d6a60f84 1099 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-node-linux-arm.tar.gz) | 3586d44bc96ff850ddf9e57be5bd3ae501de9aefe2a52aff9714366e70ef2c3c9516e1ae15e1a01ff663646f6b6068c1ffd800acc38ad05c69d4c446720aa81d 1100 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-node-linux-arm64.tar.gz) | b722f001aeb89dffe52c4503a20b635e56a93dedc0434f91351d028ab31432e0de494798b3fd834344d8b76b3d40740bb791fdc0a1204763d11657e818d4544c 1101 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-node-linux-ppc64le.tar.gz) | e510932f0c1f77042df1215cde46e11e17c51c7ed16986dcb078442420c7c845e39695c6f5fb73c031ef14421d23d9567bfd043cd63d347eb5bc6b992b264487 1102 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-node-linux-s390x.tar.gz) | df6dde4f9a9a23eed9bc69e88d88edd840d8260d5f9368836ba3138943591b55fab399251695c10d90e40e65e653dfaa6a3b457dd1605c8868db684caf700f2b 1103 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.6/kubernetes-node-windows-amd64.tar.gz) | 998a1aeeb5bd119d3df5b85e867cac2e4d3ad4ffeb6de558834e87b9f67bec4380878a573769b49aa973ab8ef637b5e420b4b5ccaaee070fc62bb258ca5b535d 1104 1105 ## Changelog since v1.21.5 1106 1107 ## Changes by Kind 1108 1109 ### API Change 1110 1111 - Kube-apiserver: Fixes handling of CRD schemas containing literal null values in enums (#104989, @liggitt) [SIG API Machinery, Apps and Network] 1112 1113 ### Feature 1114 1115 - Kubernetes is now built with Golang 1.16.9 (#105672, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing] 1116 1117 ### Failing Test 1118 1119 - Backport: e2e.test now uses the secure port to retrieve metrics data to ensure compatibility with 1.21 and 1.22 (for upgrade tests) (#104328, @pohly) [SIG API Machinery, Instrumentation, Storage and Testing] 1120 1121 ### Bug or Regression 1122 1123 - Aggregate errors when putting vmss 1124 fix: consolidate logs for instance not found error (#105365, @nilo19) [SIG Cloud Provider] 1125 - Allow CSI drivers to just run offline expansion tests (#102740, @gnufied) [SIG Storage and Testing] 1126 - Bump klog to v2.9.0, fixing log lines that render as byte arrays (#105407, @ehashman) [SIG Architecture, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage] 1127 - Fix overriding logs files on reboot. (#105614, @rphillips) [SIG Node] 1128 - Fix winkernel kube-proxy to only use dual stack when host and networking supports it (#101047, @jsturtevant) [SIG Network and Windows] 1129 - Fix: ignore not a VMSS error for VMAS nodes in EnsureBackendPoolDeleted. (#105402, @ialidzhikov) [SIG Cloud Provider] 1130 - Fix: ignore the case when updating Azure tags (#104686, @nilo19) [SIG Cloud Provider] 1131 - Revert PR #102925 which introduced unexpected scheduling behavior based on balanced resource allocation (#105238, @damemi) [SIG Scheduling] 1132 - Updates golang.org/x/text to v0.3.6 to fix CVE-2020-28852 (#102601, @jonesbr17) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage] 1133 1134 ## Dependencies 1135 1136 ### Added 1137 _Nothing has changed._ 1138 1139 ### Changed 1140 - golang.org/x/text: v0.3.4 → v0.3.6 1141 - k8s.io/klog/v2: v2.8.0 → v2.9.0 1142 1143 ### Removed 1144 _Nothing has changed._ 1145 1146 1147 1148 # v1.21.5 1149 1150 1151 ## Downloads for v1.21.5 1152 1153 ### Source Code 1154 1155 filename | sha512 hash 1156 -------- | ----------- 1157 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes.tar.gz) | e0f22fb6db131eb1bb0bbd26a3bb230a42dc9c36e867d3511789a8312bed8381137ea8c72cb0dce29fff1eb1ca08179af9e7e12c2b3491be87170ea4b00da437 1158 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-src.tar.gz) | 793dcdce411d456879dd6602a420d5d42980adef3eff93e6b7c356948e028f86f7a74b69ec25fe89d03abdf98b8759f8089c9904b1d498284dc94d71fe3204cd 1159 1160 ### Client Binaries 1161 1162 filename | sha512 hash 1163 -------- | ----------- 1164 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-darwin-amd64.tar.gz) | 4d14904d69e9f50f6c44256b4942d6623e2233e45601fb17b2b58a7f6601adacd27add292f64dbe8297f81e27052b14f83f24ef4b2ba1c84344f0169d7aa24b8 1165 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-darwin-arm64.tar.gz) | c7ea9ddf73d5be842d96e904abf0b4bb3057199957c1e5f8daae3955904c58d209dc52168623cbb101438a08fa78cda785b5cbbd859a5043da125aeaf4ea7c02 1166 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-linux-386.tar.gz) | 6bffb84698f72e9606449dbb95b04e0aec8f0e91eb4fb22f26354e932bf9909e23cda1da780d9b1a264998d9765e9fe71b9b5afd500ca6d1c6443ddd249c9786 1167 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-linux-amd64.tar.gz) | 0bd3f5a4141bf3aaf8045a9ec302561bb70f6b9a7d988bc617370620d0dbadef947e1c8855cda0347d1dd1534332ee17a950cac5a8fcb78f2c3e38c62058abde 1168 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-linux-arm.tar.gz) | c91b99bf58d1e4d881a61051436afc1b2f6286c26bf2be6e9041c4f1f1239f67c315e4167f4e3c7ecb7585c272eddf3511ff1ec73e8020257c97ba9e1bd1d00d 1169 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-linux-arm64.tar.gz) | ce9ce6e9798f1ca0ecafc7217f30c721540b6882e89d5f48aaa0498a02f22cdead07514168f778d64b2a9af0294ddcd18091fe2c534c7942aee9dd1c5c0d8a7a 1170 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-linux-ppc64le.tar.gz) | 1b2604fca222592db9c6344cf848cc5740db18e170ca30ac3cb30f771dbaf0fe10ff9e3c114cc6e7ff79b5d7262303804e087d690733f3385f19b1adc8dc9f1a 1171 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-linux-s390x.tar.gz) | 63f8706e097e5c61b6a27d01066b453fe253ad35e18aff280fda17b0c9a2fa64d4c9db8167040b65c6a12cad1c66b2cf6deafda17dfab8d38ebd93bddad33770 1172 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-windows-386.tar.gz) | 38b39b8fa66e74492c7c819c206a2add852b8db19bf3fac9076b5f9dd821ccf1b653bf80dcbb23cd6d022045ea371165739f82549fc9157a6b0030e02307fd3a 1173 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-client-windows-amd64.tar.gz) | 21ace8644bad6890c256aedba7553f49df16142db8e1d28ac525eabc877e717406781792ffbeed89d1cc46f2eaf968b440c1787c38ae364aeb18cceea3453045 1174 1175 ### Server Binaries 1176 1177 filename | sha512 hash 1178 -------- | ----------- 1179 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-server-linux-amd64.tar.gz) | 4800486117fc95f5d73a58dd0709acacf1d80ffba70a765fdbba3d79cc6438d23332478696783979b88996e12ac277a05c71c7eafb2249aaaf3671f454cc3210 1180 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-server-linux-arm.tar.gz) | c577b97d1e6adbe24c65891714306cd6d053bbfde0ea5e9a5e886e3a0bcb677c86c67dae098baa2189636e3c7c18fb1da7d3019061d94bb78c5173a58dbd9923 1181 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-server-linux-arm64.tar.gz) | 91f237f7ca4211261c3c9968e56635fbd9105c8e0d5cde03e36b547de459ab172e7e57e9bf143cfb0724c4a1783825c67e299ae3a2efb90c2ef6d89d79003745 1182 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-server-linux-ppc64le.tar.gz) | f6ac5cbbb084ce9d22dabd54166260c53cb6287938a2196bd45e5315e28ff8ed56c4978510216e90653ca35d05c664dbea8f32f3ac1a4daa6b75385c07719de6 1183 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-server-linux-s390x.tar.gz) | 44afad116c8f6774a755f138e10805b6cb73c0adf9a7933b2f2b03c218d002d8669506e545277a79251edde8bfafe09f16a6e730e1f8193036bd1e2325c68041 1184 1185 ### Node Binaries 1186 1187 filename | sha512 hash 1188 -------- | ----------- 1189 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-node-linux-amd64.tar.gz) | 41bd25451f71dd714ff186db409c7dd0ec1a5f95178d14fe47c0a12f20c024dc12fd14d4766cab8258c1f15e1df096eabd41b240062a207ebf9294dbbd2c1392 1190 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-node-linux-arm.tar.gz) | 460b4bd98eab684b7315c0f5df647cd83c817703a77f31de7e67658221e404fd41139528d6980c433c7adb2ccdaa8e002328b4c844c1e4bd24e874158b3de863 1191 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-node-linux-arm64.tar.gz) | 0937053bbe03752ce2782c03c4d206c1e96796d765c70dd2e048961e2e7c405b562240b409af2b1980f1e375f72e38f6e4cdc35d3a0bc6f38680e40fdab74de9 1192 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-node-linux-ppc64le.tar.gz) | 4e102436ecd5ed8cff460c004169f266308ab4d003a3eb872fe923718dca37373915f24f257fbc38d6e1ec0b35d57a3aa2d5b53d7458b4d177eecf297029c768 1193 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-node-linux-s390x.tar.gz) | 30266d0f899b404f188b067c890fcc0aa6a0cbaa6cabf251472ea91b70ab7cb273bc39b461c66b87ba1d87e43147290254a60e3f65b1fd5e8b88b2338b30341a 1194 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.5/kubernetes-node-windows-amd64.tar.gz) | 75e478e1fbbb34d6992470769f4e61a3a73546b1481897aedeb2d1db0c71aa344284ef57f25a81bc2013993fe9f1fe76b36fc136f422809d2fce2393c0e4fa82 1195 1196 ## Changelog since v1.21.4 1197 1198 ## Important Security Information 1199 1200 This release contains changes that address the following vulnerabilities: 1201 1202 ### CVE-2021-25741: Symlink Exchange Can Allow Host Filesystem Access 1203 1204 A security issue was discovered in Kubernetes where a user may be able to 1205 create a container with subpath volume mounts to access files & 1206 directories outside of the volume, including on the host filesystem. 1207 1208 **Affected Versions**: 1209 - kubelet v1.22.0 - v1.22.1 1210 - kubelet v1.21.0 - v1.21.4 1211 - kubelet v1.20.0 - v1.20.10 1212 - kubelet <= v1.19.14 1213 1214 **Fixed Versions**: 1215 - kubelet v1.22.2 1216 - kubelet v1.21.5 1217 - kubelet v1.20.11 1218 - kubelet v1.19.15 1219 1220 This vulnerability was reported by Fabricio Voznika and Mark Wolters of Google. 1221 1222 **CVSS Rating:** High (8.8) [CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.0#CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H) 1223 1224 ## Changes by Kind 1225 1226 ### Feature 1227 1228 - Kubernetes is now built with Golang 1.16.8 ([#104906](https://github.com/kubernetes/kubernetes/pull/104906), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] 1229 1230 ### Bug or Regression 1231 1232 - Fix NodeAuthenticator tests in dualstack ([#104840](https://github.com/kubernetes/kubernetes/pull/104840), [@ardaguclu](https://github.com/ardaguclu)) [SIG Auth and Testing] 1233 - Fix: skip case sensitivity when checking Azure NSG rules 1234 fix: ensure InstanceShutdownByProviderID return false for creating Azure VMs ([#104447](https://github.com/kubernetes/kubernetes/pull/104447), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] 1235 - Fixed occasional pod cgroup freeze when using cgroup v1 and systemd driver. 1236 Fixed "failed to create container ... unit already exists" when using cgroup v1 and systemd driver. ([#104530](https://github.com/kubernetes/kubernetes/pull/104530), [@kolyshkin](https://github.com/kolyshkin)) [SIG CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing] 1237 - Kube-proxy: delete stale conntrack UDP entries for loadbalancer ingress IP. ([#104151](https://github.com/kubernetes/kubernetes/pull/104151), [@aojea](https://github.com/aojea)) [SIG Network] 1238 - Metrics changes: Fix exposed buckets of `scheduler_volume_scheduling_duration_seconds_bucket` metric ([#100720](https://github.com/kubernetes/kubernetes/pull/100720), [@dntosas](https://github.com/dntosas)) [SIG Apps, Instrumentation, Scheduling and Storage] 1239 - Pass additional flags to subpath mount to avoid flakes in certain conditions ([#104347](https://github.com/kubernetes/kubernetes/pull/104347), [@mauriciopoppe](https://github.com/mauriciopoppe)) [SIG Storage] 1240 - When using `kubectl replace` (or the equivalent API call) on a Service, the caller no longer needs to do a read-modify-write cycle to fetch the allocated values for `.spec.clusterIP` and `.spec.ports[].nodePort`. Instead the API server will automatically carry these forward from the original object when the new object does not specify them. ([#104673](https://github.com/kubernetes/kubernetes/pull/104673), [@thockin](https://github.com/thockin)) [SIG Network] 1241 1242 ### Other (Cleanup or Flake) 1243 1244 - Kube-apiserver: sets an upper-bound on the lifetime of idle keep-alive connections and time to read the headers of incoming requests ([#103958](https://github.com/kubernetes/kubernetes/pull/103958), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Node] 1245 1246 ## Dependencies 1247 1248 ### Added 1249 - github.com/bits-and-blooms/bitset: [v1.2.0](https://github.com/bits-and-blooms/bitset/tree/v1.2.0) 1250 1251 ### Changed 1252 - github.com/cilium/ebpf: [v0.5.0 → v0.6.2](https://github.com/cilium/ebpf/compare/v0.5.0...v0.6.2) 1253 - github.com/coreos/go-systemd/v22: [v22.3.1 → v22.3.2](https://github.com/coreos/go-systemd/v22/compare/v22.3.1...v22.3.2) 1254 - github.com/golang/protobuf: [v1.4.3 → v1.5.0](https://github.com/golang/protobuf/compare/v1.4.3...v1.5.0) 1255 - github.com/google/go-cmp: [v0.5.4 → v0.5.5](https://github.com/google/go-cmp/compare/v0.5.4...v0.5.5) 1256 - github.com/opencontainers/runc: [v1.0.0-rc95 → v1.0.2](https://github.com/opencontainers/runc/compare/v1.0.0-rc95...v1.0.2) 1257 - github.com/opencontainers/selinux: [v1.8.0 → v1.8.2](https://github.com/opencontainers/selinux/compare/v1.8.0...v1.8.2) 1258 - github.com/sirupsen/logrus: [v1.7.0 → v1.8.1](https://github.com/sirupsen/logrus/compare/v1.7.0...v1.8.1) 1259 - google.golang.org/protobuf: v1.25.0 → v1.26.0 1260 1261 ### Removed 1262 - github.com/willf/bitset: [v1.1.11](https://github.com/willf/bitset/tree/v1.1.11) 1263 1264 1265 1266 # v1.21.4 1267 1268 1269 ## Downloads for v1.21.4 1270 1271 ### Source Code 1272 1273 filename | sha512 hash 1274 -------- | ----------- 1275 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes.tar.gz) | e0651f30e883194bcfa59cf57dbb40f853f9e84e79472f35db75e37af09f63de1ed960694376e623e5d176dc23c064b74311cde28d32af53aac0b0d4c0649949 1276 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-src.tar.gz) | fe3b571b654b7db3bdd2b6b4d231b4fdb82cba643bce91108bacf7a0dc80ca9fa3236a269386262ade936e579ebccd71d20ca52ddff014f3e0e5b70e083002af 1277 1278 ### Client Binaries 1279 1280 filename | sha512 hash 1281 -------- | ----------- 1282 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-darwin-amd64.tar.gz) | dff004b8ee195143949854553373fe23fae0abe234a416a364ddbf37bc18c1ab895cc9089b3c9625b7c26ea184d25c4ac7c21f99204d1625f39a11e2be788d27 1283 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-darwin-arm64.tar.gz) | b452e5240c5bed55970f4e3023dcf45ba4a29f9ce93364ad23a5c00b68eb887e502aef569b3af4ee2632192073f1a155e13fc13cc73ab94540f2728f88d53737 1284 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-linux-386.tar.gz) | 3ea3f2d158f213f665784bb1d946f85dfe6dc1b484f662ad16c57114a270ecfec664a94087e4df0e20c51e867a5e43dc8cf8f68c4d2283427371319bec29f4dc 1285 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-linux-amd64.tar.gz) | ae95dc475d442f06dec8a6c071f3e1be4222dd50610f32297f5e7e0a05cd853723885ba0a25e46835c44b7f5d3e109d079cca1236294023db130eaeaed246616 1286 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-linux-arm.tar.gz) | 3aefede77e7f8c76f89d51dac0f2c858c31f1e3ec6d874847019230c9db3dacd2637b66b74e1c36cad774b5c6be387b2559cdcf174f477a11860fac4dd69f093 1287 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-linux-arm64.tar.gz) | ad8d2289c2def736fe3ade8f58288db4a6d435df6886e89f0fca70286c6d15432d1dd4c75eb865929df6f3bfc2698d7e14d4fb03873f77e1e3551d99d5a276e2 1288 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-linux-ppc64le.tar.gz) | e8ba8239a80aefe058a475009ba94cab428311a21f7d83c752b91520fe0fb464c4cd135dd0ff554129b98042c0a2b6e23f1e422b82936d6786d7d662a9de637b 1289 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-linux-s390x.tar.gz) | c0793027f484b3a69db8f38f15a87cf6a5bf71c2416afc528c264dc27ae15f8923b9ea96a508e752ae44ac1dd77703fa73967d66f5efb124e1a32972aa7875bb 1290 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-windows-386.tar.gz) | e8c12e960f7c3dc990d7a12838ec27f83bd4b56fe04753e166aa21d62fe2f1ba7d206a0a290d0430b0558629644a77138335acf3c9b583d932829b56ac9f964e 1291 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-client-windows-amd64.tar.gz) | 6155124609e3a88ad10991167c1b6727cf2785c05c261f62e895df4e7383c004ec8d3b05a02f5706114d37b9c2cf5e05067d8b67c9ec2457da1916c445c05748 1292 1293 ### Server Binaries 1294 1295 filename | sha512 hash 1296 -------- | ----------- 1297 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-server-linux-amd64.tar.gz) | 006f27ef85b038a8443e521873ea1b3c891d98a93718d374b8168da69900733822a91de29d05cdecb778348d1fca2d7b7ded5b05d67be02e5be93226db271af5 1298 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-server-linux-arm.tar.gz) | 592ef90a48a934a2968067e154c786793e409fca47a013e0958690083f97f14d51222d73c07858d828e16c1bafe33842fedb71e324848de57f5c06eef5169434 1299 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-server-linux-arm64.tar.gz) | 8f8f70605e78a491c98e6ed7609e74d29a79ba8a603f2444905b5584c82e311ca3df25c949e89058808f184930febc342faf155f58ee58d7aee216344458d5db 1300 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-server-linux-ppc64le.tar.gz) | 484f84ce8a2e7fc0a3b46c528497624a314e3024b8a30b4a61aba753ff4b302a462d318775be3b1f4001ebcd5b301fc7d34a45a9b7c9d92694bd6bea7d759c1b 1301 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-server-linux-s390x.tar.gz) | a2701033e2b667b026bc5b2386aead64ab521c4949099b4e937b6eacb5ff75aafc83afa0c04fa35579d4955e5322362048a748b38da232a33ab191f08116c0e4 1302 1303 ### Node Binaries 1304 1305 filename | sha512 hash 1306 -------- | ----------- 1307 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-node-linux-amd64.tar.gz) | de99c3ec833401de7165b16aec3893fec002e327e49b42d397bc2475c4215173b36029459d96a6e63feb285a6c54770fd9d62f423c3081fbf4d537557f4c67c8 1308 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-node-linux-arm.tar.gz) | c9665002bcb84e454f5f43e054674fa0259540bc85155bafcb63c8716a28b04cb7d31590268ac67a6e9ecc3c40e7f97b0d18006373d193b0d2c6c00ba6852e07 1309 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-node-linux-arm64.tar.gz) | 1ed7193022eebedc988fe64257c11788b6104054c6496768a6f704c98c54573f9e98ea588f09715460c1c5f6435db94b6c49d539f4dd4fcef5555ea13313f614 1310 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-node-linux-ppc64le.tar.gz) | 420353da529474f8e9856fbbc10eae343776d8121b9d53c318a1eb354e0c1ff693a4c1961ca692faef3ef152104de45f922a9c29382718648cb0c84e659a6176 1311 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-node-linux-s390x.tar.gz) | e7df41ce015229a733953a39a2f54cf46566885f403d6c091244ea0a37875d279ffdcb2fae7de9510ed9c61289bb512dcfcd7dc67bc739d75bd2c51272518d93 1312 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.4/kubernetes-node-windows-amd64.tar.gz) | c9b31c6eedb7f1ee84b8920be92a187f19ce6e5575c693f94a88fa990e9f8e53e6f4aec6b47163d2b43b78c386d79e928f82dd8bbd84aa39868edfcaa6e3063c 1313 1314 ## Changelog since v1.21.3 1315 1316 ## Changes by Kind 1317 1318 ### Feature 1319 1320 - Kubernetes is now built with Golang 1.16.7 ([#104201](https://github.com/kubernetes/kubernetes/pull/104201), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] 1321 1322 ### Bug or Regression 1323 1324 - Disable aufs module for gce clusters ([#103831](https://github.com/kubernetes/kubernetes/pull/103831), [@lizhuqi](https://github.com/lizhuqi)) [SIG Cloud Provider] 1325 - Fix kube-apiserver metric reporting for the deprecated watch path of /api/<version>/watch/... ([#104190](https://github.com/kubernetes/kubernetes/pull/104190), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Instrumentation] 1326 - Fix the code is leaking the defaulting between unrelated pod instances. ([#103284](https://github.com/kubernetes/kubernetes/pull/103284), [@kebe7jun](https://github.com/kebe7jun)) [SIG CLI] 1327 - Fix: Provide IPv6 support for internal load balancer ([#103794](https://github.com/kubernetes/kubernetes/pull/103794), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1328 - Fix: cleanup outdated routes ([#102935](https://github.com/kubernetes/kubernetes/pull/102935), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1329 - Fix: delete non existing disk issue ([#102083](https://github.com/kubernetes/kubernetes/pull/102083), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] 1330 - Fix: ignore not a VMSS error for VMAS nodes in reconcileBackendPools ([#103997](https://github.com/kubernetes/kubernetes/pull/103997), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1331 - Fix: return empty VMAS name if using standalone VM ([#103470](https://github.com/kubernetes/kubernetes/pull/103470), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1332 - Fixed a bug that scheduler extenders are not called on preemptions ([#103019](https://github.com/kubernetes/kubernetes/pull/103019), [@ordovicia](https://github.com/ordovicia)) [SIG Scheduling] 1333 - Fixes an issue cleaning up CertificateSigningRequest objects with an unparseable `status.certificate` field ([#103948](https://github.com/kubernetes/kubernetes/pull/103948), [@liggitt](https://github.com/liggitt)) [SIG Apps and Auth] 1334 - Fixes issue with websocket-based watches of Service objects not closing correctly on timeout ([#102541](https://github.com/kubernetes/kubernetes/pull/102541), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing] 1335 1336 ## Dependencies 1337 1338 ### Added 1339 _Nothing has changed._ 1340 1341 ### Changed 1342 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.19 → v0.0.22 1343 1344 ### Removed 1345 _Nothing has changed._ 1346 1347 1348 1349 # v1.21.3 1350 1351 1352 ## Downloads for v1.21.3 1353 1354 ### Source Code 1355 1356 filename | sha512 hash 1357 -------- | ----------- 1358 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes.tar.gz) | da3b444a24d29a88357f69b34029c6aa1cfd4ffb8380c4f02b56fca24ac7f04ad8832484080f83a2c142cc1aba6ef51c821b88d4eecf2bc0942c6e33470caf07 1359 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-src.tar.gz) | 2de636025d611728460db92564aec42de1fdab52b2a310a4e390a013e0f62a0fe0987b243cfb9a01d278fae5b1df93141e7b62e202b413a1c6a40a4f9d8eba85 1360 1361 ### Client Binaries 1362 1363 filename | sha512 hash 1364 -------- | ----------- 1365 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-darwin-amd64.tar.gz) | e6b5a70fd9e0cb279ef4a6890e0bde3dcac6265d125c845b00afad7118f6dd97c952b62cbcf7e28ab741a7f8dfcf24cd93d6a44594f36238bf30c5108c0b0222 1366 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-darwin-arm64.tar.gz) | 213345f9a215d6cf9d67cfbb8be948c7977e5bd2efc16a9b5443311dece5ee5883c2df6d4a342c8bddfd66e1a997ce04f2d7fdea3bb527a87888e6567e4e7a66 1367 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-linux-386.tar.gz) | caeba030682ee4bb02afb9558ec9fdf7b7d3d03cd016fdb6479b94db1c1e752721443d4f3b488ec831f9a24816b6a91be777e7b52cbf35c8c0bbd6b83bd72715 1368 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-linux-amd64.tar.gz) | a2ccab98460d80c9dcbc4c776e373c88e0921883da05610e17acf51e6f1c50a46891482cc62720647c1488e9e79986f61c90574a095a98a4e90806065089d5ef 1369 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-linux-arm.tar.gz) | c299c36ca76216b0025690f005e72ba93693a56901a38c2e176d09fd696a8b26cc1d30b9be68976adf4fe79a5a8a24efa4aa5036c76c2a7b913c64bebef48cc9 1370 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-linux-arm64.tar.gz) | d7f27dd115bca27d85ea83925a1b46fe6211ab503bc05a0c3a9558027fcaa11bddc7fef3dd1183c760fa4426a3bffeedd2527dec69473174f252168fe179f452 1371 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-linux-ppc64le.tar.gz) | 8c3058245c90f2ef3bf2f3bfa610b2be50d69110d3b6ab24cd10f0372bed488a6c9e8181827ae597d535c0de7ad388121c4098c5b2ed58e72b2f9bab45ac6429 1372 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-linux-s390x.tar.gz) | 4125c3b54e144a42dcbd08a958c057009f4d4cfd9e3be1a3310e18bd85296415018249799c6b8bb4cf0d61ccfbfc0cdc9bdeeb0f189805995f657ace43c84191 1373 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-windows-386.tar.gz) | 100db1eb541cf08cc87dc3e5b4843286202f2156cfb8af885bd9577322f3b6d34b573c12ee7c6292ee064f84ffc281b84f25dbd5b8f47a88663aff36b10759c7 1374 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-client-windows-amd64.tar.gz) | 9300a7630110f9a10b8de731b7d368bbc5b35ac9f59d4da0a3a72fcae3797c8de77a9fd16e85b9c77ebca8d4bb6cd5f2bdb3876e1558cf24f98d757c0d70a3f6 1375 1376 ### Server Binaries 1377 1378 filename | sha512 hash 1379 -------- | ----------- 1380 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-server-linux-amd64.tar.gz) | a516adcd06fcb0ace9a1cfb1519f57f4e4ca59183043a5245354403a6c42c0bbfe968cc02741d7b6f2287fc838ccd7bfa0bf19e951ea3653096dbc6ac265223c 1381 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-server-linux-arm.tar.gz) | a7da6578aa738cab1875e5d5df65454b2fa7f26bf149b655233e53718350e7d57ef64b929448a134b1aa61e7cba1c25fb845bd6a6e5978e2ba6b91d6a3d40023 1382 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-server-linux-arm64.tar.gz) | 57cd457ef8bcf90e5205e8dc63ab989ec5f669f56dce30dd9ebb1f4ccbad9001bf8f56b85311400f0bc7a3c9fdf10637ec8570bce5d7795f518a7fa88b9e52e9 1383 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-server-linux-ppc64le.tar.gz) | 3a311f17d0e90a1b93c9e2ae97068e2c6d84526d499853850e5ef0718fb99c17251420a7377f595d250df2db17adc53b4920d7fc98fff475b76fb7481acd7525 1384 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-server-linux-s390x.tar.gz) | 8c709dfabf06d7aa77dbf21af814b5d35964bb43791ef3954d1b3d7e1c0a15a7b80d1d4c96855f23ed96b14ac9872a4c07afc1e7140c1263951a75623da48195 1385 1386 ### Node Binaries 1387 1388 filename | sha512 hash 1389 -------- | ----------- 1390 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-node-linux-amd64.tar.gz) | 16802f95a9c75aa7a4643b60d19d6d1245f036a3c6b6687ee756715210c15a4b7716c76f584d0364d4abb7aae846c23427fcd61d663f31af4fc8171b354d6e56 1391 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-node-linux-arm.tar.gz) | 60a13c386ab6765acfcdfdd3ebd5bcd1d6f266719653eff17a5a78f19ab410d99c9d8b9baaa08570239dfb24d256a9da7f07ea2b78e5d8f9ae10fbc84c44cff1 1392 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-node-linux-arm64.tar.gz) | c4a6e153be21c0755b55fb7ed0c965726a9b6801ef5e0d3d574f8449f8b98137e5c6484f14daa7f626f65b8af432a8cc02b98f443c820c38671c8105fbdc381e 1393 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-node-linux-ppc64le.tar.gz) | c14e0324bf7bf75a6c20e42a7b8f5ef447f74db2cd4b365985c8456619d0c09c970beb50ca811e1665789d6ed890fcff0b7b3dc6bc3fbfad0c4c3d60412c4fcc 1394 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-node-linux-s390x.tar.gz) | 2977f7c1196e92f4efd69f51e16cf54096fa52d3fb62178f17daab2b17de9f8e6a71517c6a79859b62ba0dfee963f27e837cbd906655ff1b3e657ae9db9d6b95 1395 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.3/kubernetes-node-windows-amd64.tar.gz) | 0f9181d7c455deb7e872983998c4250de321c85277144b0ecd205d379fb2f5cdcb4a63231878dd1172aed265eeb7d83e207e4c280281ee7b92a11f7e148b3238 1396 1397 ## Changelog since v1.21.2 1398 1399 ## Changes by Kind 1400 1401 ### Feature 1402 1403 - Kubernetes is now built with Golang 1.16.6 ([#103670](https://github.com/kubernetes/kubernetes/pull/103670), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] 1404 - Updates the following images to pick up CVE fixes: 1405 - `debian` to v1.8.0 1406 - `debian-iptables` to v1.6.5 1407 - `setcap` to v2.0.3 ([#103235](https://github.com/kubernetes/kubernetes/pull/103235), [@thejoycekung](https://github.com/thejoycekung)) [SIG API Machinery, Release and Testing] 1408 1409 ### Bug or Regression 1410 1411 - Fix scoring for NodeResourcesMostAllocated and NodeResourcesBalancedAllocation plugins when nodes have containers with no requests. This was leaving to under-utilization of small nodes. ([#102925](https://github.com/kubernetes/kubernetes/pull/102925), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] 1412 - ServiceOwnsFrontendIP shouldn't report error when the public IP doesn't match ([#102516](https://github.com/kubernetes/kubernetes/pull/102516), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1413 - Switch scheduler to generate the merge patch on pod status instead of the full pod ([#103133](https://github.com/kubernetes/kubernetes/pull/103133), [@marwanad](https://github.com/marwanad)) [SIG Scheduling] 1414 - VSphere: Fix regression during attach disk if datastore is within a storage folder or datastore cluster. ([#102969](https://github.com/kubernetes/kubernetes/pull/102969), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider] 1415 1416 ## Dependencies 1417 1418 ### Added 1419 _Nothing has changed._ 1420 1421 ### Changed 1422 - sigs.k8s.io/structured-merge-diff/v4: v4.1.0 → v4.1.2 1423 1424 ### Removed 1425 _Nothing has changed._ 1426 1427 1428 1429 # v1.21.2 1430 1431 1432 ## Downloads for v1.21.2 1433 1434 ### Source Code 1435 1436 filename | sha512 hash 1437 -------- | ----------- 1438 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes.tar.gz) | 7626af740aef3ec01e50cef0eb418b4a53305259a60aaa7886087378565baf7e30865fbb40f8b26c440462c8e44649f8bb9e2cb71e4776e83ffe7c4c84daee4b 1439 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-src.tar.gz) | e9236f76593e7f7ef13f95e9c8ef41e37368f8b9b3823b74009778d30a5f794367af437556c7d79e7eb31813ab8dcf80756eab31b26ce56981c3591f6095408d 1440 1441 ### Client Binaries 1442 1443 filename | sha512 hash 1444 -------- | ----------- 1445 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-darwin-amd64.tar.gz) | 6245ebb73ac4215420f1453906ac7a63b25954dd89a40c6f99b88cede14016d106226e6284fdc6f0634016ee7d82a8cffed424eff2b4e06a77e8e1c26d4d6759 1446 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-darwin-arm64.tar.gz) | 671f571541d325acd6789db233961d6924dbf442bb9e61aa4248e324e08906dec7997d6ed4560dfdd5f438fbc2622cfd21be9085514ffa4ea45835e912fc2265 1447 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-linux-386.tar.gz) | 5885dc7269a5323406c8f7366528b824b2cb876943a2e534b57e4375f367838cf7109062f47ccb2d0ed8fda29d910d159f1002c83d85f5d8bb4cf087c8e3dd74 1448 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-linux-amd64.tar.gz) | 0d744eec49354250a43510efc4d911a4b67f3b36dc62c28e2b852e31e9d00e474d0f44287c5b89ccbafcaa91ed1ecaa12a881a25984aa6e33fdf5c55141e9349 1449 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-linux-arm.tar.gz) | 2937bfc5c0f1c8ad2d28c88394ee2781c47d19fb7cebaf00c68e6dab3448988fe748d2c7db7c67799eee9bb2d1ab3f1a57ab571f7d20f5465aff9877a5fdb5db 1450 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-linux-arm64.tar.gz) | 137d6cd5bb8110ce47b4ebf0b9725a1b303e03263c2ce8fe7069ff51aae2d209bab94bb5dc056da8bf4478ee93af8b285b5a9088509f6c085ac789678f93ed6b 1451 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-linux-ppc64le.tar.gz) | 6eafe1aeb08186d3cbf046be7a9b8456b49cae44cbdb9f265ad4f5347b81a52cb6b3ce22ec24330ae084ee668906fbe512679c89c21ac6a29bdf0eda77c5d9eb 1452 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-linux-s390x.tar.gz) | eda9716935bc8a955e0a86d2957479c53dda875dd2ae7d5d12927d6dd358be89befeaf5643e17ceeca5d0dd30d5ce6af43c17fe8af19ba47949d7d01b913dc87 1453 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-windows-386.tar.gz) | deb12050d694763b252f3208ff90392b70ed469c76b5f49b4ecb4368c317ae314e6a3f3e09f22d5d9d62a240a0838ac30a468f47638207fb934343eecb137c65 1454 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-client-windows-amd64.tar.gz) | 997ae278d08d99bb75c67dfe86d35c023682296a8e7b7b78c08ae468924c0464cd5f81674cb188133c2c44a35e58f13c2da3052227013b8c3a255c7ab02dbb0e 1455 1456 ### Server Binaries 1457 1458 filename | sha512 hash 1459 -------- | ----------- 1460 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-server-linux-amd64.tar.gz) | 2fa740e6144c68ee040d616a42b67e10989b9ef9460248c006b86aa549b99ec27527bf75cb8f11154141a8168921abc058dcdc4b86414d47683f285a04746a10 1461 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-server-linux-arm.tar.gz) | fa51f2cc2f05c19123e0f73df54859176ce314b87dc8dcf9d163c953d1716ef88e6d87b58f55e28a0aafde423fb9653f2fd93c11e707d7c35234359a6b88781e 1462 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-server-linux-arm64.tar.gz) | 897e9e5a43da6f4030b5254f13c8a54146f54883ee03726bc1228cf589c26e29d76ea38196f2afe709d413847a30e8e4a8ee6ba286750942d2f6986ead86a856 1463 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-server-linux-ppc64le.tar.gz) | 8caa368a632bdc5ffebb02ff6329a6248ab5140159c792010bd94fc6f3090bee10c7d9404b7cb6e701d2bcf8536aaef8e26d47a988e0b681f541ae03de7a1d4a 1464 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-server-linux-s390x.tar.gz) | 83fc6ee6368c1ce749e640725ea0351145118a98f9313cac5cb49dbc8380dcdd56ae7a794b86229cc79e20f8582b73f0ff8a3216ebf681fe52e099595895e390 1465 1466 ### Node Binaries 1467 1468 filename | sha512 hash 1469 -------- | ----------- 1470 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-node-linux-amd64.tar.gz) | 589c62ccdeeddc023b3a3cd8127b87ca532ab4ff481f74c876863ba89429e2dc537664d036f98fd188b63bc121fc4e08a4a35fb13c150517a27f2c35b677560e 1471 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-node-linux-arm.tar.gz) | a8374564aa2f7baeab61df3501a3b75d348815ded92b2e2c41b6f50bb630d5bac22c3d02df6ce0c2c26065c04912611660a8bd7866c2e0da3389382bda22a0dc 1472 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-node-linux-arm64.tar.gz) | ea00bd5340e745593fbc119e008e8f9feb83f1f8f2e878e4de9ec5fe6605a4068cc5527f76e336809e63a6f5c8cdd21b7c4941915d5a22d1151de3b99f3ef26c 1473 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-node-linux-ppc64le.tar.gz) | 3e1916be9226bc0410e3fb49335458193ba88ab6eba692ce1c179a4b39d017afe6ac9062562a54dec03dc805df80d30f498da7d5a53218c301d38935cfdf8651 1474 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-node-linux-s390x.tar.gz) | bcb149e814d5ad328fef82f496ed92792e147e42fd68521f02e6e25838dd101cbd72c93dd424c17ace1095f13f98436b7bf0b45ad0532698e3070c2602d8817b 1475 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.2/kubernetes-node-windows-amd64.tar.gz) | 6528327d22aa64c90356bf55a60b8183fa054070c69c81185f33936ac02487b1d1a3308c9df0e6dd4bb72b75aa927bca61b2463bb8f2b71ef82a20b73c5007b2 1476 1477 ## Changelog since v1.21.1 1478 1479 ## Changes by Kind 1480 1481 ### Feature 1482 1483 - Kubernetes is now built with Golang 1.16.5 ([#102739](https://github.com/kubernetes/kubernetes/pull/102739), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] 1484 - Update the setcap image to buster-v2.0.1 ([#102377](https://github.com/kubernetes/kubernetes/pull/102377), [@xmudrii](https://github.com/xmudrii)) [SIG Release] 1485 1486 ### Failing Test 1487 1488 - Fixes the `should receive events on concurrent watches in same order` conformance test to work properly on clusters that auto-create additional configmaps in namespaces ([#101950](https://github.com/kubernetes/kubernetes/pull/101950), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing] 1489 1490 ### Bug or Regression 1491 1492 - Added jitter factor to lease controller that better smears load on kube-apiserver over time. ([#101652](https://github.com/kubernetes/kubernetes/pull/101652), [@marseel](https://github.com/marseel)) [SIG API Machinery and Scalability] 1493 - Avoid caching the Azure VMSS instances whose network profile is nil ([#100948](https://github.com/kubernetes/kubernetes/pull/100948), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] 1494 - Azure: avoid setting cached Sku when updating VMSS and VMSS instances ([#102005](https://github.com/kubernetes/kubernetes/pull/102005), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] 1495 - Fix a bug on the endpoint slices mirroring controller where endpoint NotReadyAddresses were mirrored as Ready to the corresponding EndpointSlice ([#102683](https://github.com/kubernetes/kubernetes/pull/102683), [@aojea](https://github.com/aojea)) [SIG Apps and Network] 1496 - Fix a bug that a preemptor pod may exist as a phantom in the scheduler. ([#102498](https://github.com/kubernetes/kubernetes/pull/102498), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] 1497 - Fix removing pods from podTopologyHints mapping ([#101892](https://github.com/kubernetes/kubernetes/pull/101892), [@aheng-ch](https://github.com/aheng-ch)) [SIG Node] 1498 - Fix resource enforcement when using systemd cgroup driver ([#102147](https://github.com/kubernetes/kubernetes/pull/102147), [@kolyshkin](https://github.com/kolyshkin)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing] 1499 - Fix: avoid nil-pointer panic when checking the frontend IP configuration ([#101739](https://github.com/kubernetes/kubernetes/pull/101739), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1500 - Fix: not tagging static public IP ([#101752](https://github.com/kubernetes/kubernetes/pull/101752), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1501 - Fixed false-positive uncertain volume attachments, which led to unexpected detachment of CSI migrated volumes ([#101737](https://github.com/kubernetes/kubernetes/pull/101737), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps and Storage] 1502 - Fixed garbage collection of dangling VolumeAttachments for PersistentVolumes migrated to CSI on startup of kube-controller-manager. ([#102176](https://github.com/kubernetes/kubernetes/pull/102176), [@timebertt](https://github.com/timebertt)) [SIG Apps and Storage] 1503 - Kube-proxy log now shows the "Skipping topology aware endpoint filtering since no hints were provided for zone" warning under the right conditions ([#101857](https://github.com/kubernetes/kubernetes/pull/101857), [@dervoeti](https://github.com/dervoeti)) [SIG Network] 1504 - Kubeadm: remove the "ephemeral_storage" request from the etcd static pod that kubeadm deploys on stacked etcd control plane nodes. This request has caused sporadic failures on some setups due to a problem in the kubelet with cadvisor and the LocalStorageCapacityIsolation feature gate. See this issue for more details: https://github.com/kubernetes/kubernetes/issues/99305 ([#102673](https://github.com/kubernetes/kubernetes/pull/102673), [@jackfrancis](https://github.com/jackfrancis)) [SIG Cluster Lifecycle] 1505 - Kubeadm: when using a custom image repository for CoreDNS kubeadm now will append the "coredns" image name instead of "coredns/coredns", thus restoring the behaviour existing before the v1.21 release. Users who rely on nested folder for the coredns image should set the "clusterConfiguration.dns.imageRepository" value including the nested path name (e.g using "registry.company.xyz/coredns" will force kubeadm to use "registry.company.xyz/coredns/coredns" image). No action is needed if using the default registry (k8s.gcr.io). ([#102502](https://github.com/kubernetes/kubernetes/pull/102502), [@ykakarap](https://github.com/ykakarap)) [SIG Cluster Lifecycle] 1506 - Register/Deregister Targets in chunks for AWS TargetGroup ([#101592](https://github.com/kubernetes/kubernetes/pull/101592), [@M00nF1sh](https://github.com/M00nF1sh)) [SIG Cloud Provider] 1507 - Respect annotation size limit for server-side apply updates to the client-side apply annotation. Also, fix opt-out of this behavior by setting the client-side apply annotation to the empty string. ([#102105](https://github.com/kubernetes/kubernetes/pull/102105), [@julianvmodesto](https://github.com/julianvmodesto)) [SIG API Machinery] 1508 - Reverted the previous fix for portforward cleanup because it introduced a kubelet regression which can lead into segmentation faults. ([#102587](https://github.com/kubernetes/kubernetes/pull/102587), [@saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery and Node] 1509 1510 ### Other (Cleanup or Flake) 1511 1512 - Client-go: reduce verbosity of "Starting/Stopping reflector" messages to 3 again ([#102788](https://github.com/kubernetes/kubernetes/pull/102788), [@pohly](https://github.com/pohly)) [SIG API Machinery] 1513 - Update the Debian images to pick up CVE fixes in the base images: 1514 - Update the `debian-base` image to v1.7.0 1515 - Update the `debian-iptables` image to v1.6.1 ([#102340](https://github.com/kubernetes/kubernetes/pull/102340), [@cpanato](https://github.com/cpanato)) [SIG API Machinery and Testing] 1516 1517 ## Dependencies 1518 1519 ### Added 1520 - github.com/checkpoint-restore/go-criu/v5: [v5.0.0](https://github.com/checkpoint-restore/go-criu/v5/tree/v5.0.0) 1521 - github.com/frankban/quicktest: [v1.11.3](https://github.com/frankban/quicktest/tree/v1.11.3) 1522 1523 ### Changed 1524 - github.com/cilium/ebpf: [v0.2.0 → v0.5.0](https://github.com/cilium/ebpf/compare/v0.2.0...v0.5.0) 1525 - github.com/containerd/console: [v1.0.1 → v1.0.2](https://github.com/containerd/console/compare/v1.0.1...v1.0.2) 1526 - github.com/coreos/go-systemd/v22: [v22.1.0 → v22.3.1](https://github.com/coreos/go-systemd/v22/compare/v22.1.0...v22.3.1) 1527 - github.com/godbus/dbus/v5: [v5.0.3 → v5.0.4](https://github.com/godbus/dbus/v5/compare/v5.0.3...v5.0.4) 1528 - github.com/google/go-cmp: [v0.5.2 → v0.5.4](https://github.com/google/go-cmp/compare/v0.5.2...v0.5.4) 1529 - github.com/kr/pretty: [v0.2.0 → v0.2.1](https://github.com/kr/pretty/compare/v0.2.0...v0.2.1) 1530 - github.com/moby/sys/mountinfo: [v0.4.0 → v0.4.1](https://github.com/moby/sys/mountinfo/compare/v0.4.0...v0.4.1) 1531 - github.com/opencontainers/runc: [v1.0.0-rc93 → v1.0.0-rc95](https://github.com/opencontainers/runc/compare/v1.0.0-rc93...v1.0.0-rc95) 1532 - github.com/opencontainers/runtime-spec: [e6143ca → 1c3f411](https://github.com/opencontainers/runtime-spec/compare/e6143ca...1c3f411) 1533 - golang.org/x/sys: a50acf3 → d19ff85 1534 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.15 → v0.0.19 1535 1536 ### Removed 1537 - github.com/checkpoint-restore/go-criu/v4: [v4.1.0](https://github.com/checkpoint-restore/go-criu/v4/tree/v4.1.0) 1538 1539 1540 1541 # v1.21.1 1542 1543 1544 ## Downloads for v1.21.1 1545 1546 ### Source Code 1547 1548 filename | sha512 hash 1549 -------- | ----------- 1550 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes.tar.gz) | 39005d4e11684d732babf419e06403a67ffb4f6c78cb7cb6e8214a7a1e6171de2a302d20d72bbb9da7f2748f46fd3bcf7d7e3b4767e3c8167f536d93e5f7bd7b 1551 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-src.tar.gz) | 2d25debd4a4559ca9338c7f35682ec72d8aa5c1ebe424b9866fc9391fbbbc21dac5498b5af35c5109cbdfb5cee9a2259b99eb3086d05cff6510b9ac1407626df 1552 1553 ### Client Binaries 1554 1555 filename | sha512 hash 1556 -------- | ----------- 1557 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-darwin-amd64.tar.gz) | b5ee08ce233f80f3c9d8918347169752df49f9745792762893b3bbc5c7a36246028533a405c8a41172205e3b6d54895ddc56e9b464feab7054f8ef5a2b2b059b 1558 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-darwin-arm64.tar.gz) | 3e10c84c3cb184bf3d81bc62009e85dc9f6df00d381136617a71007c2b4758cc3879b4bb90b7e416d11d4e58d7a44eb89562640a2a75a1b369241a79068999be 1559 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-linux-386.tar.gz) | 34391ca524b35d2f970ffd2f59becb4f8131b6576976af4724212afbdb972449deca87ee6bdcd4f10c8093fd0513a5d11baa59112a29338ec3e717ad93998a7f 1560 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-linux-amd64.tar.gz) | 97050c54cd0f41e2efc954400bf65726c699b456506a0cbbb752aa79ec8662335dffa81c5373fb2e3231bab1e8ff73a3a1b5329caa2225ae7809fe1269c03f99 1561 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-linux-arm.tar.gz) | 2bb268e38004bde0ed27ac0c2a2fdbe5fe2c2a95272ef2f289c53049e24ee0b94bf4590fdfec99a1e8b72dfec917bd7a40435cbbda445fe04c9a6aad61f250a5 1562 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-linux-arm64.tar.gz) | ef67fc5795c95f3b9d8dab2476805167af5b3b84242fd5f340e73bb144a99040462093a6db53719dead419b77efd6e37660f513fcd93e870dc47f75e039dbbbb 1563 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-linux-ppc64le.tar.gz) | 701e54ee5d3f0754c88ca766c3ce624157a2e96b57a72ea8a5e350e2ed4ac8302e9940b851bbfb93904629afa80f71438f6cf8d1110b9cb47efe99690830d967 1564 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-linux-s390x.tar.gz) | d44519897d65308d47a621a5995938379bdf022bf69ed5b3468d09b04800970e61e57d03acd0d48411fa9cade6c7df638c576710f414f00b716e54933ce4244c 1565 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-windows-386.tar.gz) | a404d61d60e14182c931819512e5048f2ba3d51381a1b14fe52c54c49abc828aa32cadc4c377291d52151f5eca1b89d526389a12eed51ac353498f494b646522 1566 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-client-windows-amd64.tar.gz) | 0474acd68f2d298e772af003631d437fb4c7a445955f740389e01096910b20639a45d160122082020917fa187b754be396d64b3275cbab4aabdc451cc978a743 1567 1568 ### Server Binaries 1569 1570 filename | sha512 hash 1571 -------- | ----------- 1572 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-server-linux-amd64.tar.gz) | 9e370a48ef2a428ad3b0878d0ae0989caf156e36a2b40bf803aaddeed941c09120900eedc21e643d6e819529308e36db3da57bc887766f45ef2c8f56897a1170 1573 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-server-linux-arm.tar.gz) | 1a6e256ba0a3318b1a419aac17eaabce9df9650a70974da0b677fcc6b628f87473394dc45b0524733424d2eaf2f09c0b9ac9baee5a9247a0d1376f026c202f38 1574 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-server-linux-arm64.tar.gz) | 24b146b1befbcc845b4e95e7b24d9d9cfc06bdf77409135ce6c7c0c99f4803ee9aed7f3cc99023ec0a9069f9f9044bd5b2562b13cb5825ff4317675ef160a7e7 1575 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-server-linux-ppc64le.tar.gz) | 36708c18c4760f2766245fd8c58038d1f21d118b0ccf50183a154e5a1c299a5f04db5c6b19d295b649c68e9ed320372253272314b306eda6bd674a715033548e 1576 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-server-linux-s390x.tar.gz) | 3e9333253875306e055c00c0cced7d564c512807ada0ad499524b3a11b3fb4cfbca944c711b41ce9ebb2a631487208aa0543ed64e0d2e5e992d405a4f4c5990c 1577 1578 ### Node Binaries 1579 1580 filename | sha512 hash 1581 -------- | ----------- 1582 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-node-linux-amd64.tar.gz) | 752bfccbc40b710a6042c4d590a8bf43b14c9dbf6d1afd1f5d87bd11465b45279d0d7508227eaedf148a9ebda4fe32106acf38d7becee1e844ba045ae7e8985e 1583 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-node-linux-arm.tar.gz) | 892781b14d2750c007133cc43cb12cdd7ccce9f03b28b40f4f840484b170349900b2016f7c3797f9df7f078dbd409606cf4f339cbd981ff64d0112c651cc1725 1584 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-node-linux-arm64.tar.gz) | af95314feb5085e0a13b271b78754c617f93846d67149931d6cd8d93d816e51cbf556b146b0992e6596998f7bbd31865801689a7cc1f6ee9654f743c5949a1a1 1585 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-node-linux-ppc64le.tar.gz) | 382512bfe522544d8f537133d4c62e2f976a1e6ccca7aaecbb8d974b839fba2bda4d4392de4096c9de83cc51586fe8c8454884e5e95020f562629fb3dbd618e3 1586 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-node-linux-s390x.tar.gz) | 14060d0ce6bb640ed6af389f1f20bc0219a59d5090eaac308f3aee6e0aed1236e786e015505bbed4c293de89560757b521a1b222b2ee87f57ad090fe26a67e0a 1587 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.1/kubernetes-node-windows-amd64.tar.gz) | c72121f132107ebddf7a55a412e28f83633ffc78038c35e85dbe7eed253acf3370e56fe304b842c15a1476116f4a0a2cf1e92185b54d2176a79b6d914e93deab 1588 1589 ## Changelog since v1.21.0 1590 1591 ## Changes by Kind 1592 1593 ### API Change 1594 1595 - "Auto" is now a valid value for the `service.kubernetes.io/topology-aware-hints` annotation. ([#100728](https://github.com/kubernetes/kubernetes/pull/100728), [@robscott](https://github.com/robscott)) [SIG Apps, Instrumentation and Network] 1596 - We have added a new Priority & Fairness rule that exempts all probes (/readyz, /healthz, /livez) to prevent 1597 restarting of "healthy" kube-apiserver instance(s) by kubelet. ([#101111](https://github.com/kubernetes/kubernetes/pull/101111), [@tkashem](https://github.com/tkashem)) [SIG API Machinery] 1598 1599 ### Feature 1600 1601 - Add NeedResize function to kubernetes/mount-utils, user can call this function to determine if fs need to be resized ([#101253](https://github.com/kubernetes/kubernetes/pull/101253), [@AndyXiangLi](https://github.com/AndyXiangLi)) [SIG Storage] 1602 - Base image updates to mitigate kube-proxy and etcd container image CVEs 1603 - debian-base to buster-v1.6.0 1604 - debian-iptables to buster-v1.6.0 ([#100976](https://github.com/kubernetes/kubernetes/pull/100976), [@jindijamie](https://github.com/jindijamie)) [SIG Release and Testing] 1605 - Kubernetes is now built with Golang 1.16.3 ([#101209](https://github.com/kubernetes/kubernetes/pull/101209), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] 1606 - Kubernetes is now built with Golang 1.16.4 ([#101811](https://github.com/kubernetes/kubernetes/pull/101811), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] 1607 1608 ### Failing Test 1609 1610 - Fixed generic ephemeal volumes with OwnerReferencesPermissionEnforcement admission plugin enabled. ([#101186](https://github.com/kubernetes/kubernetes/pull/101186), [@jsafrane](https://github.com/jsafrane)) [SIG Auth and Storage] 1611 - Resolves an issue with the "ServiceAccountIssuerDiscovery should support OIDC discovery" conformance test failing on clusters which are configured with issuers outside the cluster ([#101725](https://github.com/kubernetes/kubernetes/pull/101725), [@mtaufen](https://github.com/mtaufen)) [SIG Auth and Testing] 1612 1613 ### Bug or Regression 1614 1615 - Adds node event handlers to dual stack kube-proxy implementation to fix Topology Aware Hints. ([#101054](https://github.com/kubernetes/kubernetes/pull/101054), [@robscott](https://github.com/robscott)) [SIG Network] 1616 - Azurefile: Normalize share name to not include capital letters ([#100731](https://github.com/kubernetes/kubernetes/pull/100731), [@kassarl](https://github.com/kassarl)) [SIG Cloud Provider and Storage] 1617 - EndpointSlice IP validation now matches Endpoints IP validation. ([#101084](https://github.com/kubernetes/kubernetes/pull/101084), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 1618 - Ensure service deleted when the Azure resource group has been deleted ([#100944](https://github.com/kubernetes/kubernetes/pull/100944), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] 1619 - Fix EndpointSlice describe panic when an Endpoint doesn't have zone ([#101025](https://github.com/kubernetes/kubernetes/pull/101025), [@tnqn](https://github.com/tnqn)) [SIG CLI] 1620 - Fix display of Job completion mode on kubectl describe ([#101198](https://github.com/kubernetes/kubernetes/pull/101198), [@alculquicondor](https://github.com/alculquicondor)) [SIG CLI] 1621 - Fix: azure file inline volume namespace issue in csi migration translation ([#101235](https://github.com/kubernetes/kubernetes/pull/101235), [@andyzhangx](https://github.com/andyzhangx)) [SIG Apps, Cloud Provider, Node and Storage] 1622 - Fix: set "host is down" as corrupted mount ([#101398](https://github.com/kubernetes/kubernetes/pull/101398), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] 1623 - Fixed a bug where startupProbe stopped working after a container's first restart ([#101093](https://github.com/kubernetes/kubernetes/pull/101093), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 1624 - Fixed port-forward memory leak for long-running and heavily used connections. ([#99839](https://github.com/kubernetes/kubernetes/pull/99839), [@saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery and Node] 1625 - Kubectl create service now respects namespace flag ([#101005](https://github.com/kubernetes/kubernetes/pull/101005), [@zxh326](https://github.com/zxh326)) [SIG CLI] 1626 - Kubelet: improve the performance when waiting for a synchronization of the node list with the kube-apiserver ([#99336](https://github.com/kubernetes/kubernetes/pull/99336), [@neolit123](https://github.com/neolit123)) [SIG Node] 1627 - No support endpointslice in linux userpace mode ([#101504](https://github.com/kubernetes/kubernetes/pull/101504), [@JornShen](https://github.com/JornShen)) [SIG Network] 1628 - Renames the timeout field for the DelegatingAuthenticationOptions to TokenRequestTimeout and set the timeout only for the token review client. Previously the timeout was also applied to watches making them reconnecting every 10 seconds. ([#101102](https://github.com/kubernetes/kubernetes/pull/101102), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery, Auth and Cloud Provider] 1629 - Respect ExecProbeTimeout=false for dockershim ([#101127](https://github.com/kubernetes/kubernetes/pull/101127), [@jackfrancis](https://github.com/jackfrancis)) [SIG Node and Testing] 1630 - Upgrades functionality of `kubectl kustomize` as described at https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.1.0 ([#101177](https://github.com/kubernetes/kubernetes/pull/101177), [@KnVerey](https://github.com/KnVerey)) [SIG CLI] 1631 1632 ## Dependencies 1633 1634 ### Added 1635 _Nothing has changed._ 1636 1637 ### Changed 1638 - sigs.k8s.io/kustomize/api: v0.8.5 → v0.8.8 1639 - sigs.k8s.io/kustomize/cmd/config: v0.9.7 → v0.9.10 1640 - sigs.k8s.io/kustomize/kustomize/v4: v4.0.5 → v4.1.2 1641 - sigs.k8s.io/kustomize/kyaml: v0.10.15 → v0.10.17 1642 1643 ### Removed 1644 _Nothing has changed._ 1645 1646 1647 1648 # v1.21.0 1649 1650 [Documentation](https://docs.k8s.io) 1651 1652 ## Downloads for v1.21.0 1653 1654 ### Source Code 1655 1656 filename | sha512 hash 1657 -------- | ----------- 1658 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes.tar.gz) | `19bb76a3fa5ce4b9f043b2a3a77c32365ab1fcb902d8dd6678427fb8be8f49f64a5a03dc46aaef9c7dadee05501cf83412eda46f0edacbb8fc1ed0bf5fb79142` 1659 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-src.tar.gz) | `f942e6d6c10007a6e9ce21e94df597015ae646a7bc3e515caf1a3b79f1354efb9aff59c40f2553a8e3d43fe4a01742241f5af18b69666244906ed11a22e3bc49` 1660 1661 ### Client Binaries 1662 1663 filename | sha512 hash 1664 -------- | ----------- 1665 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-darwin-amd64.tar.gz) | `be9d1440e418e5253fb8a3d8aba705ca8160746a9bd17325ad626a986b6da9f733af864155a651a32b7bca94b533b8d596005ddbe5248bdeea85db47a1b957ed` 1666 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-darwin-arm64.tar.gz) | `eed0ddc81d104bb2d41ace13f737c490423d5df4ebddc7376e45c18ed66af35933c9376b912c1c3da105945b04056f6ca0870c156bee8a307cf4189ca5eb1dd1` 1667 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-386.tar.gz) | `8a2f30c4434199762f2a96141dab4241c1cce2711bea9ea39cc63c2c5e7d31719ed7f076efac1931604e3a94578d3bbf0cfa454965708c96f3cfb91789868746` 1668 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-amd64.tar.gz) | `cd3cfa645fa31de3716f1f63506e31b73d2aa8d37bb558bb3b3e8c151f35b3d74d44e03cbd05be67e380f9a5d015aba460222afdac6677815cd99a85c2325cf0` 1669 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-arm.tar.gz) | `936042aa11cea0f6dfd2c30fc5dbe655420b34799bede036b1299a92d6831f589ca10290b73b9c9741560b603ae31e450ad024e273f2b4df5354bfac272691d8` 1670 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-arm64.tar.gz) | `42beb75364d7bf4bf526804b8a35bd0ab3e124b712e9d1f45c1b914e6be0166619b30695feb24b3eecef134991dacb9ab3597e788bd9e45cf35addddf20dd7f6` 1671 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-ppc64le.tar.gz) | `4baba2ed7046b28370eccc22e2378ae79e3ce58220d6f4f1b6791e8233bec8379e30200bb20b971456b83f2b791ea166fdfcf1ea56908bc1eea03590c0eda468` 1672 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-s390x.tar.gz) | `37fa0c4d703aef09ce68c10ef3e7362b0313c8f251ce38eea579cd18fae4023d3d2b70e0f31577cabe6958ab9cfc30e98d25a7c64e69048b423057c3cf728339` 1673 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-windows-386.tar.gz) | `6900db36c1e3340edfd6dfd8d720575a904c932d39a8a7fa36401595e971a0235bd42111dbcc1cbb77e7374e47f1380a68c637997c18f96a0d9cdc9f3714c4c9` 1674 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-windows-amd64.tar.gz) | `90de67f6f79fc63bcfdf35066e3d84501cc85433265ffad36fd1a7a428a31b446249f0644a1e97495ea8b2a08e6944df6ef30363003750339edaa2aceffe937c` 1675 1676 ### Server Binaries 1677 1678 filename | sha512 hash 1679 -------- | ----------- 1680 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-amd64.tar.gz) | `3941dcc2309ac19ec185603a79f5a086d8a198f98c04efa23f15a177e5e1f34946ea9392ba9f5d24d0d727839438f067fef1001fc6e88b27b8b01e35bbd962ca` 1681 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-arm.tar.gz) | `6507abf6c2ec2b336901dc23269f6c577ec0049b8bad3c9dd6ad63f21aa10f09bfbbfa6e064c2466d250411d3e10f8672791a9e10942e38de7bfbaf7a8bcc9da` 1682 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-arm64.tar.gz) | `5abe76f867ca6865344e957bf166b81766c049ec4eb183a8a5580c22a7f8474db1edf90fd901a5833e56128b6825811653a1d27f72fd34ce5b1287a8c10da05c` 1683 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-ppc64le.tar.gz) | `62507b182ca25396a285d91241536860e58f54fac937e97cbdf91948c83bb41be97d33277400489bf50e85164d560205540b76e94e5d519892312bdc63df1067` 1684 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-s390x.tar.gz) | `04f2a1f7d1388e4a7d7d9f597f872a3da36f26839cfed16aad6df07021c03f4dca1df06b19cfda56df09d1c2d9a13ebd0af40ca1b9b6aecfaf427ab7712d88f3` 1685 1686 ### Node Binaries 1687 1688 filename | sha512 hash 1689 -------- | ----------- 1690 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-amd64.tar.gz) | `c1831c708109c31b3878e5a9327ea4b9e546504d0b6b00f3d43db78b5dd7d5114d32ac24a9a505f9cadbe61521f0419933348d2cd309ed8cfe3987d9ca8a7e2c` 1691 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-arm.tar.gz) | `b68dd5bcfc7f9ce2781952df40c8c3a64c29701beff6ac22f042d6f31d4de220e9200b7e8272ddf608114327770acdaf3cb9a34a0a5206e784bda717ea080e0f` 1692 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-arm64.tar.gz) | `7fa84fc500c28774ed25ca34b6f7b208a2bea29d6e8379f84b9f57bd024aa8fe574418cee7ee26edd55310716d43d65ae7b9cbe11e40c995fe2eac7f66bdb423` 1693 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-ppc64le.tar.gz) | `a4278b3f8e458e9581e01f0c5ba8443303c987988ee136075a8f2f25515d70ca549fbd2e4d10eefca816c75c381d62d71494bd70c47034ab47f8315bbef4ae37` 1694 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-s390x.tar.gz) | `8de2bc6f22f232ff534b45012986eac23893581ccb6c45bd637e40dbe808ce31d5a92375c00dc578bdbadec342b6e5b70c1b9f3d3a7bb26ccfde97d71f9bf84a` 1695 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-windows-amd64.tar.gz) | `b82e94663d330cff7a117f99a7544f27d0bc92b36b5a283b3c23725d5b33e6f15e0ebf784627638f22f2d58c58c0c2b618ddfd226a64ae779693a0861475d355` 1696 1697 ## Changelog since v1.20.0 1698 1699 ## What's New (Major Themes) 1700 1701 ### Deprecation of PodSecurityPolicy 1702 1703 PSP as an admission controller resource is being deprecated. Deployed PodSecurityPolicy's will keep working until version 1.25, their target removal from the codebase. A new feature, with a working title of "PSP replacement policy", is being developed in [KEP-2579](https://features.k8s.io/2579). To learn more, read [PodSecurityPolicy Deprecation: Past, Present, and Future](https://blog.k8s.io/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/). 1704 1705 ### Kubernetes API Reference Documentation 1706 1707 The API reference is now generated with [`gen-resourcesdocs`](https://github.com/kubernetes-sigs/reference-docs/tree/c96658d89fb21037b7d00d27e6dbbe6b32375837/gen-resourcesdocs) and it is moving to [Kubernetes API](https://docs.k8s.io/reference/kubernetes-api/) 1708 1709 ### Kustomize Updates in Kubectl 1710 1711 [Kustomize](https://github.com/kubernetes-sigs/kustomize) version in kubectl had a jump from v2.0.3 to [v4.0.5](https://github.com/kubernetes/kubernetes/pull/98946). Kustomize is now treated as a library and future updates will be less sporadic. 1712 1713 ### Default Container Annotation 1714 1715 Pod with multiple containers can use `kubectl.kubernetes.io/default-container` annotation to have a container preselected for kubectl commands. More can be read in [KEP-2227](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/2227-kubectl-default-container/README.md). 1716 1717 ### Immutable Secrets and ConfigMaps 1718 1719 Immutable Secrets and ConfigMaps graduates to GA. This feature allows users to specify that the contents of a particular Secret or ConfigMap is immutable for its object lifetime. For such instances, Kubelet will not watch/poll for changes and therefore reducing apiserver load. 1720 1721 ### Structured Logging in Kubelet 1722 1723 Kubelet has adopted structured logging, thanks to community effort in accomplishing this within the release timeline. Structured logging in the project remains an ongoing effort -- for folks interested in participating, [keep an eye / chime in to the mailing list discussion](https://groups.google.com/g/kubernetes-dev/c/y4WIw-ntUR8). 1724 1725 ### Storage Capacity Tracking 1726 1727 Traditionally, the Kubernetes scheduler was based on the assumptions that additional persistent storage is available everywhere in the cluster and has infinite capacity. Topology constraints addressed the first point, but up to now pod scheduling was still done without considering that the remaining storage capacity may not be enough to start a new pod. [Storage capacity tracking](https://docs.k8s.io/concepts/storage/storage-capacity/) addresses that by adding an API for a CSI driver to report storage capacity and uses that information in the Kubernetes scheduler when choosing a node for a pod. This feature serves as a stepping stone for supporting dynamic provisioning for local volumes and other volume types that are more capacity constrained. 1728 1729 ### Generic Ephemeral Volumes 1730 1731 [Generic ephermeral volumes](https://docs.k8s.io/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes) feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod. It can be used to provide scratch storage that is different from the root disk, for example persistent memory, or a separate local disk on that node. All StorageClass parameters for volume provisioning are supported. All features supported with PersistentVolumeClaims are supported, such as storage capacity tracking, snapshots and restore, and volume resizing. 1732 1733 ### CSI Service Account Token 1734 1735 CSI Service Account Token feature moves to Beta in 1.21. This feature improves the security posture and allows CSI drivers to receive pods' [bound service account tokens](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md). This feature also provides a knob to re-publish volumes so that short-lived volumes can be refreshed. 1736 1737 ### CSI Health Monitoring 1738 1739 The CSI health monitoring feature is being released as a second Alpha in Kubernetes 1.21. This feature enables CSI Drivers to share abnormal volume conditions from the underlying storage systems with Kubernetes so that they can be reported as events on PVCs or Pods. This feature serves as a stepping stone towards programmatic detection and resolution of individual volume health issues by Kubernetes. 1740 1741 ## Known Issues 1742 1743 ### `TopologyAwareHints` feature falls back to default behavior 1744 1745 The feature gate currently falls back to the default behavior in most cases. Enabling the feature gate will add hints to `EndpointSlices`, but functional differences are only observed in non-dual stack kube-proxy implementation. This is fixed by [#101054](https://github.com/kubernetes/kubernetes/pull/101054). 1746 1747 ## Urgent Upgrade Notes 1748 1749 ### (No, really, you MUST read this before you upgrade) 1750 1751 - Kube-proxy's IPVS proxy mode no longer sets the net.ipv4.conf.all.route_localnet sysctl parameter. Nodes upgrading will have net.ipv4.conf.all.route_localnet set to 1 but new nodes will inherit the system default (usually 0). If you relied on any behavior requiring net.ipv4.conf.all.route_localnet, you must set ensure it is enabled as kube-proxy will no longer set it automatically. This change helps to further mitigate CVE-2020-8558. ([#92938](https://github.com/kubernetes/kubernetes/pull/92938), [@lbernail](https://github.com/lbernail)) [SIG Network and Release] 1752 - Kubeadm: during "init" an empty cgroupDriver value in the KubeletConfiguration is now always set to "systemd" unless the user is explicit about it. This requires existing machine setups to configure the container runtime to use the "systemd" driver. Documentation on this topic can be found here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/. When upgrading existing clusters / nodes using "kubeadm upgrade" the old cgroupDriver value is preserved, but in 1.22 this change will also apply to "upgrade". For more information on migrating to the "systemd" driver or remaining on the "cgroupfs" driver see: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/. ([#99471](https://github.com/kubernetes/kubernetes/pull/99471), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 1753 - Newly provisioned PVs by EBS plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. ([#99130](https://github.com/kubernetes/kubernetes/pull/99130), [@ayberk](https://github.com/ayberk)) [SIG Cloud Provider, Storage and Testing] 1754 - Newly provisioned PVs by OpenStack Cinder plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. ([#99719](https://github.com/kubernetes/kubernetes/pull/99719), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider and Storage] 1755 - Newly provisioned PVs by gce-pd will no longer have the beta FailureDomain label. gce-pd volume plugin will start to have GA topology label instead. ([#98700](https://github.com/kubernetes/kubernetes/pull/98700), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Cloud Provider, Storage and Testing] 1756 - OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. ([#98538](https://github.com/kubernetes/kubernetes/pull/98538), [@dims](https://github.com/dims)) [SIG Storage] 1757 - Remove alpha `CSIMigrationXXComplete` flag and add alpha `InTreePluginXXUnregister` flag. Deprecate `CSIMigrationvSphereComplete` flag and it will be removed in v1.22. ([#98243](https://github.com/kubernetes/kubernetes/pull/98243), [@Jiawei0227](https://github.com/Jiawei0227)) 1758 - Remove storage metrics `storage_operation_errors_total`, since we already have `storage_operation_status_count`.And add new field `status` for `storage_operation_duration_seconds`, so that we can know about all status storage operation latency. ([#98332](https://github.com/kubernetes/kubernetes/pull/98332), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation and Storage] 1759 - The metric `storage_operation_errors_total` is not removed, but is marked deprecated, and the metric `storage_operation_status_count` is marked deprecated. In both cases the `storage_operation_duration_seconds` metric can be used to recover equivalent counts (using `status=fail-unknown` in the case of `storage_operations_errors_total`). ([#99045](https://github.com/kubernetes/kubernetes/pull/99045), [@mattcary](https://github.com/mattcary)) 1760 - `ServiceNodeExclusion`, `NodeDisruptionExclusion` and `LegacyNodeRoleBehavior` features have been promoted to GA. `ServiceNodeExclusion` and `NodeDisruptionExclusion` are now unconditionally enabled, while `LegacyNodeRoleBehavior` is unconditionally disabled. To prevent control plane nodes from being added to load balancers automatically, upgrade users need to add "node.kubernetes.io/exclude-from-external-load-balancers" label to control plane nodes. ([#97543](https://github.com/kubernetes/kubernetes/pull/97543), [@pacoxu](https://github.com/pacoxu)) 1761 1762 ## Changes by Kind 1763 1764 ### Deprecation 1765 1766 - Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling --ignore-errors flag. ([#98203](https://github.com/kubernetes/kubernetes/pull/98203), [@yuzhiquan](https://github.com/yuzhiquan)) 1767 - Delete deprecated `service.beta.kubernetes.io/azure-load-balancer-mixed-protocols` mixed procotol annotation in favor of the MixedProtocolLBService feature ([#97096](https://github.com/kubernetes/kubernetes/pull/97096), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1768 - Deprecate the `topologyKeys` field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy. ([#96736](https://github.com/kubernetes/kubernetes/pull/96736), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps] 1769 - Kube-proxy: remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS ([#97336](https://github.com/kubernetes/kubernetes/pull/97336), [@maaoBit](https://github.com/maaoBit)) [SIG Network] 1770 - Kubeadm: deprecated command "alpha selfhosting pivot" is now removed. ([#97627](https://github.com/kubernetes/kubernetes/pull/97627), [@knight42](https://github.com/knight42)) 1771 - Kubeadm: graduate the command `kubeadm alpha kubeconfig user` to `kubeadm kubeconfig user`. The `kubeadm alpha kubeconfig user` command is deprecated now. ([#97583](https://github.com/kubernetes/kubernetes/pull/97583), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] 1772 - Kubeadm: the "kubeadm alpha certs" command is removed now, please use "kubeadm certs" instead. ([#97706](https://github.com/kubernetes/kubernetes/pull/97706), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] 1773 - Kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error. ([#99646](https://github.com/kubernetes/kubernetes/pull/99646), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] 1774 - Kubectl: The deprecated `kubectl alpha debug` command is removed. Use `kubectl debug` instead. ([#98111](https://github.com/kubernetes/kubernetes/pull/98111), [@pandaamanda](https://github.com/pandaamanda)) [SIG CLI] 1775 - Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97935](https://github.com/kubernetes/kubernetes/pull/97935), [@adeniyistephen](https://github.com/adeniyistephen)) [SIG Release and Testing] 1776 - Remove deprecated `--generator, --replicas, --service-generator, --service-overrides, --schedule` from `kubectl run` 1777 Deprecate `--serviceaccount, --hostport, --requests, --limits` in `kubectl run` ([#99732](https://github.com/kubernetes/kubernetes/pull/99732), [@soltysh](https://github.com/soltysh)) 1778 - Remove the deprecated metrics "scheduling_algorithm_preemption_evaluation_seconds" and "binding_duration_seconds", suggest to use "scheduler_framework_extension_point_duration_seconds" instead. ([#96447](https://github.com/kubernetes/kubernetes/pull/96447), [@chendave](https://github.com/chendave)) [SIG Cluster Lifecycle, Instrumentation, Scheduling and Testing] 1779 - Removing experimental windows container hyper-v support with Docker ([#97141](https://github.com/kubernetes/kubernetes/pull/97141), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] 1780 - Rename metrics `etcd_object_counts` to `apiserver_storage_object_counts` and mark it as stable. The original `etcd_object_counts` metrics name is marked as "Deprecated" and will be removed in the future. ([#99785](https://github.com/kubernetes/kubernetes/pull/99785), [@erain](https://github.com/erain)) [SIG API Machinery, Instrumentation and Testing] 1781 - The GA TokenRequest and TokenRequestProjection feature gates have been removed and are unconditionally enabled. Remove explicit use of those feature gates in CLI invocations. ([#97148](https://github.com/kubernetes/kubernetes/pull/97148), [@wawa0210](https://github.com/wawa0210)) [SIG Node] 1782 - The PodSecurityPolicy API is deprecated in 1.21, and will no longer be served starting in 1.25. ([#97171](https://github.com/kubernetes/kubernetes/pull/97171), [@deads2k](https://github.com/deads2k)) [SIG Auth and CLI] 1783 - The `batch/v2alpha1` CronJob type definitions and clients are deprecated and removed. ([#96987](https://github.com/kubernetes/kubernetes/pull/96987), [@soltysh](https://github.com/soltysh)) [SIG API Machinery, Apps, CLI and Testing] 1784 - The `export` query parameter (inconsistently supported by API resources and deprecated in v1.14) is fully removed. Requests setting this query parameter will now receive a 400 status response. ([#98312](https://github.com/kubernetes/kubernetes/pull/98312), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Auth and Testing] 1785 - `audit.k8s.io/v1beta1` and `audit.k8s.io/v1alpha1` audit policy configuration and audit events are deprecated in favor of `audit.k8s.io/v1`, available since v1.13. kube-apiserver invocations that specify alpha or beta policy configurations with `--audit-policy-file`, or explicitly request alpha or beta audit events with `--audit-log-version` / `--audit-webhook-version` must update to use `audit.k8s.io/v1` and accept `audit.k8s.io/v1` events prior to v1.24. ([#98858](https://github.com/kubernetes/kubernetes/pull/98858), [@carlory](https://github.com/carlory)) [SIG Auth] 1786 - `discovery.k8s.io/v1beta1` EndpointSlices are deprecated in favor of `discovery.k8s.io/v1`, and will no longer be served in Kubernetes v1.25. ([#100472](https://github.com/kubernetes/kubernetes/pull/100472), [@liggitt](https://github.com/liggitt)) 1787 - `diskformat` storage class parameter for in-tree vSphere volume plugin is deprecated as of v1.21 release. Please consider updating storageclass and remove `diskformat` parameter. vSphere CSI Driver does not support diskformat storageclass parameter. 1788 1789 vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3. 1790 1791 VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM's Hardware version set to at least vmx-15. 1792 1793 Multi vCenter support is deprecated as of v1.21. If you have a Kubernetes cluster spanning across multiple vCenter servers, please consider moving all k8s nodes to a single vCenter Server. vSphere CSI Driver does not support Kubernetes deployment spanning across multiple vCenter servers. 1794 1795 Support for these deprecations will be available till Kubernetes v1.24. ([#98546](https://github.com/kubernetes/kubernetes/pull/98546), [@divyenpatel](https://github.com/divyenpatel)) 1796 1797 ### API Change 1798 1799 - 1. PodAffinityTerm includes a namespaceSelector field to allow selecting eligible namespaces based on their labels. 1800 2. A new CrossNamespacePodAffinity quota scope API that allows restricting which namespaces allowed to use PodAffinityTerm with corss-namespace reference via namespaceSelector or namespaces fields. ([#98582](https://github.com/kubernetes/kubernetes/pull/98582), [@ahg-g](https://github.com/ahg-g)) [SIG API Machinery, Apps, Auth and Testing] 1801 - Add Probe-level terminationGracePeriodSeconds field ([#99375](https://github.com/kubernetes/kubernetes/pull/99375), [@ehashman](https://github.com/ehashman)) [SIG API Machinery, Apps, Node and Testing] 1802 - Added `.spec.completionMode` field to Job, with accepted values `NonIndexed` (default) and `Indexed`. This is an alpha field and is only honored by servers with the `IndexedJob` feature gate enabled. ([#98441](https://github.com/kubernetes/kubernetes/pull/98441), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and CLI] 1803 - Adds support for endPort field in NetworkPolicy ([#97058](https://github.com/kubernetes/kubernetes/pull/97058), [@rikatz](https://github.com/rikatz)) [SIG Apps and Network] 1804 - CSIServiceAccountToken graduates to Beta and enabled by default. ([#99298](https://github.com/kubernetes/kubernetes/pull/99298), [@zshihang](https://github.com/zshihang)) 1805 - Cluster admins can now turn off `/debug/pprof` and `/debug/flags/v` endpoint in kubelet by setting `enableProfilingHandler` and `enableDebugFlagsHandler` to `false` in the Kubelet configuration file. Options `enableProfilingHandler` and `enableDebugFlagsHandler` can be set to `true` only when `enableDebuggingHandlers` is also set to `true`. ([#98458](https://github.com/kubernetes/kubernetes/pull/98458), [@SaranBalaji90](https://github.com/SaranBalaji90)) 1806 - DaemonSets accept a MaxSurge integer or percent on their rolling update strategy that will launch the updated pod on nodes and wait for those pods to go ready before marking the old out-of-date pods as deleted. This allows workloads to avoid downtime during upgrades when deployed using DaemonSets. This feature is alpha and is behind the DaemonSetUpdateSurge feature gate. ([#96441](https://github.com/kubernetes/kubernetes/pull/96441), [@smarterclayton](https://github.com/smarterclayton)) [SIG Apps and Testing] 1807 - Enable SPDY pings to keep connections alive, so that `kubectl exec` and `kubectl portforward` won't be interrupted. ([#97083](https://github.com/kubernetes/kubernetes/pull/97083), [@knight42](https://github.com/knight42)) [SIG API Machinery and CLI] 1808 - FieldManager no longer owns fields that get reset before the object is persisted (e.g. "status wiping"). ([#99661](https://github.com/kubernetes/kubernetes/pull/99661), [@kevindelgado](https://github.com/kevindelgado)) [SIG API Machinery, Auth and Testing] 1809 - Fixes server-side apply for APIService resources. ([#98576](https://github.com/kubernetes/kubernetes/pull/98576), [@kevindelgado](https://github.com/kevindelgado)) 1810 - Generic ephemeral volumes are beta. ([#99643](https://github.com/kubernetes/kubernetes/pull/99643), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing] 1811 - Hugepages request values are limited to integer multiples of the page size. ([#98515](https://github.com/kubernetes/kubernetes/pull/98515), [@lala123912](https://github.com/lala123912)) [SIG Apps] 1812 - Implement the GetAvailableResources in the podresources API. ([#95734](https://github.com/kubernetes/kubernetes/pull/95734), [@fromanirh](https://github.com/fromanirh)) [SIG Instrumentation, Node and Testing] 1813 - IngressClass resource can now reference a resource in a specific namespace 1814 for implementation-specific configuration (previously only Cluster-level resources were allowed). 1815 This feature can be enabled using the IngressClassNamespacedParams feature gate. ([#99275](https://github.com/kubernetes/kubernetes/pull/99275), [@hbagdi](https://github.com/hbagdi)) 1816 - Jobs API has a new `.spec.suspend` field that can be used to suspend and resume Jobs. This is an alpha field which is only honored by servers with the `SuspendJob` feature gate enabled. ([#98727](https://github.com/kubernetes/kubernetes/pull/98727), [@adtac](https://github.com/adtac)) 1817 - Kubelet Graceful Node Shutdown feature graduates to Beta and enabled by default. ([#99735](https://github.com/kubernetes/kubernetes/pull/99735), [@bobbypage](https://github.com/bobbypage)) 1818 - Kubernetes is now built using go1.15.7 ([#98363](https://github.com/kubernetes/kubernetes/pull/98363), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Node, Release and Testing] 1819 - Namespace API objects now have a `kubernetes.io/metadata.name` label matching their metadata.name field to allow selecting any namespace by its name using a label selector. ([#96968](https://github.com/kubernetes/kubernetes/pull/96968), [@jayunit100](https://github.com/jayunit100)) [SIG API Machinery, Apps, Cloud Provider, Storage and Testing] 1820 - One new field "InternalTrafficPolicy" in Service is added. 1821 It specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. 1822 "Cluster" routes internal traffic to a Service to all endpoints. 1823 "Local" routes traffic to node-local endpoints only, and traffic is dropped if no node-local endpoints are ready. 1824 The default value is "Cluster". ([#96600](https://github.com/kubernetes/kubernetes/pull/96600), [@maplain](https://github.com/maplain)) [SIG API Machinery, Apps and Network] 1825 - PodDisruptionBudget API objects can now contain conditions in status. ([#98127](https://github.com/kubernetes/kubernetes/pull/98127), [@mortent](https://github.com/mortent)) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation] 1826 - PodSecurityPolicy only stores "generic" as allowed volume type if the GenericEphemeralVolume feature gate is enabled ([#98918](https://github.com/kubernetes/kubernetes/pull/98918), [@pohly](https://github.com/pohly)) [SIG Auth and Security] 1827 - Promote CronJobs to batch/v1 ([#99423](https://github.com/kubernetes/kubernetes/pull/99423), [@soltysh](https://github.com/soltysh)) [SIG API Machinery, Apps, CLI and Testing] 1828 - Promote Immutable Secrets/ConfigMaps feature to Stable. This allows to set `immutable` field in Secret or ConfigMap object to mark their contents as immutable. ([#97615](https://github.com/kubernetes/kubernetes/pull/97615), [@wojtek-t](https://github.com/wojtek-t)) [SIG Apps, Architecture, Node and Testing] 1829 - Remove support for building Kubernetes with bazel. ([#99561](https://github.com/kubernetes/kubernetes/pull/99561), [@BenTheElder](https://github.com/BenTheElder)) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows] 1830 - Scheduler extender filter interface now can report unresolvable failed nodes in the new field `FailedAndUnresolvableNodes` of `ExtenderFilterResult` struct. Nodes in this map will be skipped in the preemption phase. ([#92866](https://github.com/kubernetes/kubernetes/pull/92866), [@cofyc](https://github.com/cofyc)) [SIG Scheduling] 1831 - Services can specify loadBalancerClass to use a custom load balancer ([#98277](https://github.com/kubernetes/kubernetes/pull/98277), [@XudongLiuHarold](https://github.com/XudongLiuHarold)) 1832 - Storage capacity tracking (= the CSIStorageCapacity feature) graduates to Beta and enabled by default, storage.k8s.io/v1alpha1/VolumeAttachment and storage.k8s.io/v1alpha1/CSIStorageCapacity objects are deprecated ([#99641](https://github.com/kubernetes/kubernetes/pull/99641), [@pohly](https://github.com/pohly)) 1833 - Support for Indexed Job: a Job that is considered completed when Pods associated to indexes from 0 to (.spec.completions-1) have succeeded. ([#98812](https://github.com/kubernetes/kubernetes/pull/98812), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and CLI] 1834 - The BoundServiceAccountTokenVolume feature has been promoted to beta, and enabled by default. 1835 - This changes the tokens provided to containers at `/var/run/secrets/kubernetes.io/serviceaccount/token` to be time-limited, auto-refreshed, and invalidated when the containing pod is deleted. 1836 - Clients should reload the token from disk periodically (once per minute is recommended) to ensure they continue to use a valid token. `k8s.io/client-go` version v11.0.0+ and v0.15.0+ reload tokens automatically. 1837 - By default, injected tokens are given an extended lifetime so they remain valid even after a new refreshed token is provided. The metric `serviceaccount_stale_tokens_total` can be used to monitor for workloads that are depending on the extended lifetime and are continuing to use tokens even after a refreshed token is provided to the container. If that metric indicates no existing workloads are depending on extended lifetimes, injected token lifetime can be shortened to 1 hour by starting `kube-apiserver` with `--service-account-extend-token-expiration=false`. ([#95667](https://github.com/kubernetes/kubernetes/pull/95667), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth, Cluster Lifecycle and Testing] 1838 - The EndpointSlice Controllers are now GA. The `EndpointSliceController` will not populate the `deprecatedTopology` field and will only provide topology information through the `zone` and `nodeName` fields. ([#99870](https://github.com/kubernetes/kubernetes/pull/99870), [@swetharepakula](https://github.com/swetharepakula)) 1839 - The Endpoints controller will now set the `endpoints.kubernetes.io/over-capacity` annotation to "warning" when an Endpoints resource contains more than 1000 addresses. In a future release, the controller will truncate Endpoints that exceed this limit. The EndpointSlice API can be used to support significantly larger number of addresses. ([#99975](https://github.com/kubernetes/kubernetes/pull/99975), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 1840 - The PodDisruptionBudget API has been promoted to policy/v1 with no schema changes. The only functional change is that an empty selector (`{}`) written to a policy/v1 PodDisruptionBudget now selects all pods in the namespace. The behavior of the policy/v1beta1 API remains unchanged. The policy/v1beta1 PodDisruptionBudget API is deprecated and will no longer be served in 1.25+. ([#99290](https://github.com/kubernetes/kubernetes/pull/99290), [@mortent](https://github.com/mortent)) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Scheduling and Testing] 1841 - The `EndpointSlice` API is now GA. The `EndpointSlice` topology field has been removed from the GA API and will be replaced by a new per Endpoint Zone field. If the topology field was previously used, it will be converted into an annotation in the v1 Resource. The `discovery.k8s.io/v1alpha1` API is removed. ([#99662](https://github.com/kubernetes/kubernetes/pull/99662), [@swetharepakula](https://github.com/swetharepakula)) 1842 - The `controller.kubernetes.io/pod-deletion-cost` annotation can be set to offer a hint on the cost of deleting a `Pod` compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are deleted first. This is an alpha feature. ([#99163](https://github.com/kubernetes/kubernetes/pull/99163), [@ahg-g](https://github.com/ahg-g)) 1843 - The kube-apiserver now resets `managedFields` that got corrupted by a mutating admission controller. ([#98074](https://github.com/kubernetes/kubernetes/pull/98074), [@kwiesmueller](https://github.com/kwiesmueller)) 1844 - Topology Aware Hints are now available in alpha and can be enabled with the `TopologyAwareHints` feature gate. ([#99522](https://github.com/kubernetes/kubernetes/pull/99522), [@robscott](https://github.com/robscott)) [SIG API Machinery, Apps, Auth, Instrumentation, Network and Testing] 1845 - Users might specify the `kubectl.kubernetes.io/default-exec-container` annotation in a Pod to preselect container for kubectl commands. ([#97099](https://github.com/kubernetes/kubernetes/pull/97099), [@pacoxu](https://github.com/pacoxu)) [SIG CLI] 1846 1847 ### Feature 1848 1849 - A client-go metric, rest_client_exec_plugin_call_total, has been added to track total calls to client-go credential plugins. ([#98892](https://github.com/kubernetes/kubernetes/pull/98892), [@ankeesler](https://github.com/ankeesler)) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation] 1850 - A new histogram metric to track the time it took to delete a job by the `TTLAfterFinished` controller ([#98676](https://github.com/kubernetes/kubernetes/pull/98676), [@ahg-g](https://github.com/ahg-g)) 1851 - AWS cloud provider supports auto-discovering subnets without any `kubernetes.io/cluster/<clusterName>` tags. It also supports additional service annotation `service.beta.kubernetes.io/aws-load-balancer-subnets` to manually configure the subnets. ([#97431](https://github.com/kubernetes/kubernetes/pull/97431), [@kishorj](https://github.com/kishorj)) 1852 - Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling --ignore-errors flag. ([#98203](https://github.com/kubernetes/kubernetes/pull/98203), [@yuzhiquan](https://github.com/yuzhiquan)) 1853 - Add --permit-address-sharing flag to `kube-apiserver` to listen with `SO_REUSEADDR`. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoids waiting for the kernel to release socket in `TIME_WAIT` state, and hence, considerably reducing `kube-apiserver` restart times under certain conditions. ([#93861](https://github.com/kubernetes/kubernetes/pull/93861), [@sttts](https://github.com/sttts)) 1854 - Add `csi_operations_seconds` metric on kubelet that exposes CSI operations duration and status for node CSI operations. ([#98979](https://github.com/kubernetes/kubernetes/pull/98979), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Instrumentation and Storage] 1855 - Add `migrated` field into `storage_operation_duration_seconds` metric ([#99050](https://github.com/kubernetes/kubernetes/pull/99050), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps, Instrumentation and Storage] 1856 - Add flag --lease-reuse-duration-seconds for kube-apiserver to config etcd lease reuse duration. ([#97009](https://github.com/kubernetes/kubernetes/pull/97009), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery and Scalability] 1857 - Add metric etcd_lease_object_counts for kube-apiserver to observe max objects attached to a single etcd lease. ([#97480](https://github.com/kubernetes/kubernetes/pull/97480), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery, Instrumentation and Scalability] 1858 - Add support to generate client-side binaries for new darwin/arm64 platform ([#97743](https://github.com/kubernetes/kubernetes/pull/97743), [@dims](https://github.com/dims)) [SIG Release and Testing] 1859 - Added `ephemeral_volume_controller_create[_failures]_total` counters to kube-controller-manager metrics ([#99115](https://github.com/kubernetes/kubernetes/pull/99115), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation and Storage] 1860 - Added support for installing `arm64` node artifacts. ([#99242](https://github.com/kubernetes/kubernetes/pull/99242), [@liu-cong](https://github.com/liu-cong)) 1861 - Adds alpha feature `VolumeCapacityPriority` which makes the scheduler prioritize nodes based on the best matching size of statically provisioned PVs across multiple topologies. ([#96347](https://github.com/kubernetes/kubernetes/pull/96347), [@cofyc](https://github.com/cofyc)) [SIG Apps, Network, Scheduling, Storage and Testing] 1862 - Adds the ability to pass --strict-transport-security-directives to the kube-apiserver to set the HSTS header appropriately. Be sure you understand the consequences to browsers before setting this field. ([#96502](https://github.com/kubernetes/kubernetes/pull/96502), [@249043822](https://github.com/249043822)) [SIG Auth] 1863 - Adds two new metrics to cronjobs, a histogram to track the time difference when a job is created and the expected time when it should be created, as well as a gauge for the missed schedules of a cronjob ([#99341](https://github.com/kubernetes/kubernetes/pull/99341), [@alaypatel07](https://github.com/alaypatel07)) 1864 - Alpha implementation of Kubectl Command Headers: SIG CLI KEP 859 enabled when KUBECTL_COMMAND_HEADERS environment variable set on the client command line. ([#98952](https://github.com/kubernetes/kubernetes/pull/98952), [@seans3](https://github.com/seans3)) 1865 - Base-images: Update to debian-iptables:buster-v1.4.0 1866 - Uses iptables 1.8.5 1867 - base-images: Update to debian-base:buster-v1.3.0 1868 - cluster/images/etcd: Build etcd:3.4.13-2 image 1869 - Uses debian-base:buster-v1.3.0 ([#98401](https://github.com/kubernetes/kubernetes/pull/98401), [@pacoxu](https://github.com/pacoxu)) [SIG Testing] 1870 - CRIContainerLogRotation graduates to GA and unconditionally enabled. ([#99651](https://github.com/kubernetes/kubernetes/pull/99651), [@umohnani8](https://github.com/umohnani8)) 1871 - Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. ([#99385](https://github.com/kubernetes/kubernetes/pull/99385), [@YoyinZyc](https://github.com/YoyinZyc)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Release] 1872 - Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. ([#99738](https://github.com/kubernetes/kubernetes/pull/99738), [@YoyinZyc](https://github.com/YoyinZyc)) [SIG API Machinery, Cluster Lifecycle and Instrumentation] 1873 - EmptyDir memory backed volumes are sized as the minimum of pod allocatable memory on a host and an optional explicit user provided value. ([#100319](https://github.com/kubernetes/kubernetes/pull/100319), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node] 1874 - Enables Kubelet to check volume condition and log events to corresponding pods. ([#99284](https://github.com/kubernetes/kubernetes/pull/99284), [@fengzixu](https://github.com/fengzixu)) [SIG Apps, Instrumentation, Node and Storage] 1875 - EndpointSliceNodeName graduates to GA and thus will be unconditionally enabled -- NodeName will always be available in the v1beta1 API. ([#99746](https://github.com/kubernetes/kubernetes/pull/99746), [@swetharepakula](https://github.com/swetharepakula)) 1876 - Export `NewDebuggingRoundTripper` function and `DebugLevel` options in the k8s.io/client-go/transport package. ([#98324](https://github.com/kubernetes/kubernetes/pull/98324), [@atosatto](https://github.com/atosatto)) 1877 - Kube-proxy iptables: new metric sync_proxy_rules_iptables_total that exposes the number of rules programmed per table in each iteration ([#99653](https://github.com/kubernetes/kubernetes/pull/99653), [@aojea](https://github.com/aojea)) [SIG Instrumentation and Network] 1878 - Kube-scheduler now logs plugin scoring summaries at --v=4 ([#99411](https://github.com/kubernetes/kubernetes/pull/99411), [@damemi](https://github.com/damemi)) [SIG Scheduling] 1879 - Kubeadm now includes CoreDNS v1.8.0. ([#96429](https://github.com/kubernetes/kubernetes/pull/96429), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] 1880 - Kubeadm: IPv6DualStack feature gate graduates to Beta and enabled by default ([#99294](https://github.com/kubernetes/kubernetes/pull/99294), [@pacoxu](https://github.com/pacoxu)) 1881 - Kubeadm: a warning to user as ipv6 site-local is deprecated ([#99574](https://github.com/kubernetes/kubernetes/pull/99574), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle and Network] 1882 - Kubeadm: add support for certificate chain validation. When using kubeadm in external CA mode, this allows an intermediate CA to be used to sign the certificates. The intermediate CA certificate must be appended to each signed certificate for this to work correctly. ([#97266](https://github.com/kubernetes/kubernetes/pull/97266), [@robbiemcmichael](https://github.com/robbiemcmichael)) [SIG Cluster Lifecycle] 1883 - Kubeadm: amend the node kernel validation to treat CGROUP_PIDS, FAIR_GROUP_SCHED as required and CFS_BANDWIDTH, CGROUP_HUGETLB as optional ([#96378](https://github.com/kubernetes/kubernetes/pull/96378), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle and Node] 1884 - Kubeadm: apply the "node.kubernetes.io/exclude-from-external-load-balancers" label on control plane nodes during "init", "join" and "upgrade" to preserve backwards compatibility with the lagacy LB mode where nodes labeled as "master" where excluded. To opt-out you can remove the label from a node. See #97543 and the linked KEP for more details. ([#98269](https://github.com/kubernetes/kubernetes/pull/98269), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 1885 - Kubeadm: if the user has customized their image repository via the kubeadm configuration, pass the custom pause image repository and tag to the kubelet via --pod-infra-container-image not only for Docker but for all container runtimes. This flag tells the kubelet that it should not garbage collect the image. ([#99476](https://github.com/kubernetes/kubernetes/pull/99476), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 1886 - Kubeadm: perform pre-flight validation on host/node name upon `kubeadm init` and `kubeadm join`, showing warnings on non-compliant names ([#99194](https://github.com/kubernetes/kubernetes/pull/99194), [@pacoxu](https://github.com/pacoxu)) 1887 - Kubectl version changed to write a warning message to stderr if the client and server version difference exceeds the supported version skew of +/-1 minor version. ([#98250](https://github.com/kubernetes/kubernetes/pull/98250), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] 1888 - Kubectl: Add `--use-protocol-buffers` flag to kubectl top pods and nodes. ([#96655](https://github.com/kubernetes/kubernetes/pull/96655), [@serathius](https://github.com/serathius)) 1889 - Kubectl: `kubectl get` will omit managed fields by default now. Users could set `--show-managed-fields` to true to show managedFields when the output format is either `json` or `yaml`. ([#96878](https://github.com/kubernetes/kubernetes/pull/96878), [@knight42](https://github.com/knight42)) [SIG CLI and Testing] 1890 - Kubectl: a Pod can be preselected as default container using `kubectl.kubernetes.io/default-container` annotation ([#99833](https://github.com/kubernetes/kubernetes/pull/99833), [@mengjiao-liu](https://github.com/mengjiao-liu)) 1891 - Kubectl: add bash-completion for comma separated list on `kubectl get` ([#98301](https://github.com/kubernetes/kubernetes/pull/98301), [@phil9909](https://github.com/phil9909)) 1892 - Kubernetes is now built using go1.15.8 ([#98834](https://github.com/kubernetes/kubernetes/pull/98834), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] 1893 - Kubernetes is now built with Golang 1.16 ([#98572](https://github.com/kubernetes/kubernetes/pull/98572), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing] 1894 - Kubernetes is now built with Golang 1.16.1 ([#100106](https://github.com/kubernetes/kubernetes/pull/100106), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] 1895 - Metrics can now be disabled explicitly via a command line flag (i.e. '--disabled-metrics=metric1,metric2') ([#99217](https://github.com/kubernetes/kubernetes/pull/99217), [@logicalhan](https://github.com/logicalhan)) 1896 - New admission controller `DenyServiceExternalIPs` is available. Clusters which do not *need* the Service `externalIPs` feature should enable this controller and be more secure. ([#97395](https://github.com/kubernetes/kubernetes/pull/97395), [@thockin](https://github.com/thockin)) 1897 - Overall, enable the feature of `PreferNominatedNode` will improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of `PreferNominatedNode`, the pod might not be scheduled to the best candidate node in the cluster. ([#93179](https://github.com/kubernetes/kubernetes/pull/93179), [@chendave](https://github.com/chendave)) [SIG Scheduling and Testing] 1898 - Persistent Volumes formatted with the btrfs filesystem will now automatically resize when expanded. ([#99361](https://github.com/kubernetes/kubernetes/pull/99361), [@Novex](https://github.com/Novex)) [SIG Storage] 1899 - Port the devicemanager to Windows node to allow device plugins like directx ([#93285](https://github.com/kubernetes/kubernetes/pull/93285), [@aarnaud](https://github.com/aarnaud)) [SIG Node, Testing and Windows] 1900 - Removes cAdvisor JSON metrics (/stats/container, /stats/<podname>/<containername>, /stats/<namespace>/<podname>/<poduid>/<containername>) from the kubelet. ([#99236](https://github.com/kubernetes/kubernetes/pull/99236), [@pacoxu](https://github.com/pacoxu)) 1901 - Rename metrics `etcd_object_counts` to `apiserver_storage_object_counts` and mark it as stable. The original `etcd_object_counts` metrics name is marked as "Deprecated" and will be removed in the future. ([#99785](https://github.com/kubernetes/kubernetes/pull/99785), [@erain](https://github.com/erain)) [SIG API Machinery, Instrumentation and Testing] 1902 - Sysctls graduates to General Availability and thus unconditionally enabled. ([#99158](https://github.com/kubernetes/kubernetes/pull/99158), [@wgahnagl](https://github.com/wgahnagl)) 1903 - The Kubernetes pause image manifest list now contains an image for Windows Server 20H2. ([#97322](https://github.com/kubernetes/kubernetes/pull/97322), [@claudiubelu](https://github.com/claudiubelu)) [SIG Windows] 1904 - The NodeAffinity plugin implements the PreFilter extension, offering enhanced performance for Filter. ([#99213](https://github.com/kubernetes/kubernetes/pull/99213), [@AliceZhang2016](https://github.com/AliceZhang2016)) [SIG Scheduling] 1905 - The `CronJobControllerV2` feature flag graduates to Beta and set to be enabled by default. ([#98878](https://github.com/kubernetes/kubernetes/pull/98878), [@soltysh](https://github.com/soltysh)) 1906 - The `EndpointSlice` mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields are mirrored. 1907 The well-known annotation `endpoints.kubernetes.io/last-change-trigger-time` is skipped and not mirrored. ([#98116](https://github.com/kubernetes/kubernetes/pull/98116), [@aojea](https://github.com/aojea)) 1908 - The `RunAsGroup` feature has been promoted to GA in this release. ([#94641](https://github.com/kubernetes/kubernetes/pull/94641), [@krmayankk](https://github.com/krmayankk)) [SIG Auth and Node] 1909 - The `ServiceAccountIssuerDiscovery` feature has graduated to GA, and is unconditionally enabled. The `ServiceAccountIssuerDiscovery` feature-gate will be removed in 1.22. ([#98553](https://github.com/kubernetes/kubernetes/pull/98553), [@mtaufen](https://github.com/mtaufen)) [SIG API Machinery, Auth and Testing] 1910 - The `TTLAfterFinished` feature flag is now beta and enabled by default ([#98678](https://github.com/kubernetes/kubernetes/pull/98678), [@ahg-g](https://github.com/ahg-g)) 1911 - The apimachinery util/net function used to detect the bind address `ResolveBindAddress()` takes into consideration global IP addresses on loopback interfaces when 1) the host has default routes, or 2) there are no global IPs on those interfaces in order to support more complex network scenarios like BGP Unnumbered RFC 5549 ([#95790](https://github.com/kubernetes/kubernetes/pull/95790), [@aojea](https://github.com/aojea)) [SIG Network] 1912 - The feature gate `RootCAConfigMap` graduated to GA in v1.21 and therefore will be unconditionally enabled. This flag will be removed in v1.22 release. ([#98033](https://github.com/kubernetes/kubernetes/pull/98033), [@zshihang](https://github.com/zshihang)) 1913 - The pause image upgraded to `v3.4.1` in kubelet and kubeadm for both Linux and Windows. ([#98205](https://github.com/kubernetes/kubernetes/pull/98205), [@pacoxu](https://github.com/pacoxu)) 1914 - Update pause container to run as pseudo user and group `65535:65535`. This implies the release of version 3.5 of the container images. ([#97963](https://github.com/kubernetes/kubernetes/pull/97963), [@saschagrunert](https://github.com/saschagrunert)) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Security and Testing] 1915 - Update the latest validated version of Docker to 20.10 ([#98977](https://github.com/kubernetes/kubernetes/pull/98977), [@neolit123](https://github.com/neolit123)) [SIG CLI, Cluster Lifecycle and Node] 1916 - Upgrade node local dns to 1.17.0 for better IPv6 support ([#99749](https://github.com/kubernetes/kubernetes/pull/99749), [@pacoxu](https://github.com/pacoxu)) [SIG Cloud Provider and Network] 1917 - Upgrades `IPv6Dualstack` to `Beta` and turns it on by default. New clusters or existing clusters are not be affected until an actor starts adding secondary Pods and service CIDRS CLI flags as described here: [IPv4/IPv6 Dual-stack](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack) ([#98969](https://github.com/kubernetes/kubernetes/pull/98969), [@khenidak](https://github.com/khenidak)) 1918 - Users might specify the `kubectl.kubernetes.io/default-container` annotation in a Pod to preselect container for kubectl commands. ([#99581](https://github.com/kubernetes/kubernetes/pull/99581), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG CLI] 1919 - When downscaling ReplicaSets, ready and creation timestamps are compared in a logarithmic scale. ([#99212](https://github.com/kubernetes/kubernetes/pull/99212), [@damemi](https://github.com/damemi)) [SIG Apps and Testing] 1920 - When the kubelet is watching a ConfigMap or Secret purely in the context of setting environment variables 1921 for containers, only hold that watch for a defined duration before cancelling it. This change reduces the CPU 1922 and memory usage of the kube-apiserver in large clusters. ([#99393](https://github.com/kubernetes/kubernetes/pull/99393), [@chenyw1990](https://github.com/chenyw1990)) [SIG API Machinery, Node and Testing] 1923 - WindowsEndpointSliceProxying feature gate has graduated to beta and is enabled by default. This means kube-proxy will read from EndpointSlices instead of Endpoints on Windows by default. ([#99794](https://github.com/kubernetes/kubernetes/pull/99794), [@robscott](https://github.com/robscott)) [SIG Network] 1924 - `kubectl wait` ensures that observedGeneration >= generation to prevent stale state reporting. An example scenario can be found on CRD updates. ([#97408](https://github.com/kubernetes/kubernetes/pull/97408), [@KnicKnic](https://github.com/KnicKnic)) 1925 1926 ### Documentation 1927 1928 - Azure file migration graduates to beta, with CSIMigrationAzureFile flag off by default 1929 as it requires installation of AzureFile CSI Driver. Users should enable CSIMigration and 1930 CSIMigrationAzureFile features and install the [AzureFile CSI Driver](https://github.com/kubernetes-sigs/azurefile-csi-driver) 1931 to avoid disruption to existing Pod and PVC objects at that time. Azure File CSI driver does not support using same persistent 1932 volume with different fsgroups. When CSI migration is enabled for azurefile driver, such case is not supported. 1933 (there is a case we support where volume is mounted with 0777 and then it readable/writable by everyone) ([#96293](https://github.com/kubernetes/kubernetes/pull/96293), [@andyzhangx](https://github.com/andyzhangx)) 1934 - Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97935](https://github.com/kubernetes/kubernetes/pull/97935), [@adeniyistephen](https://github.com/adeniyistephen)) [SIG Release and Testing] 1935 - Set kubelet option `--volume-stats-agg-period` to negative value to disable volume calculations. ([#96675](https://github.com/kubernetes/kubernetes/pull/96675), [@pacoxu](https://github.com/pacoxu)) [SIG Node] 1936 1937 ### Failing Test 1938 1939 - Escape the special characters like `[`, `]` and ` ` that exist in vsphere windows path ([#98830](https://github.com/kubernetes/kubernetes/pull/98830), [@liyanhui1228](https://github.com/liyanhui1228)) [SIG Storage and Windows] 1940 - Kube-proxy: fix a bug on UDP `NodePort` Services where stale connection tracking entries may blackhole the traffic directed to the `NodePort` ([#98305](https://github.com/kubernetes/kubernetes/pull/98305), [@aojea](https://github.com/aojea)) 1941 - Kubelet: fixes a bug in the HostPort dockershim implementation that caused the conformance test "HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol" to fail. ([#98755](https://github.com/kubernetes/kubernetes/pull/98755), [@aojea](https://github.com/aojea)) [SIG Cloud Provider, Network and Node] 1942 1943 ### Bug or Regression 1944 1945 - AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. ([#96873](https://github.com/kubernetes/kubernetes/pull/96873), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG Node] 1946 - All data is no longer automatically deleted when a failure is detected during creation of the volume data file on a CSI volume. Now only the data file and volume path is removed. ([#96021](https://github.com/kubernetes/kubernetes/pull/96021), [@huffmanca](https://github.com/huffmanca)) 1947 - Clean ReplicaSet by revision instead of creation timestamp in deployment controller ([#97407](https://github.com/kubernetes/kubernetes/pull/97407), [@waynepeking348](https://github.com/waynepeking348)) [SIG Apps] 1948 - Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. ([#98133](https://github.com/kubernetes/kubernetes/pull/98133), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1949 - Client-go exec credential plugins will pass stdin only when interactive terminal is detected on stdin. This fixes a bug where previously it was checking if **stdout** is an interactive terminal. ([#99654](https://github.com/kubernetes/kubernetes/pull/99654), [@ankeesler](https://github.com/ankeesler)) 1950 - Cloud-controller-manager: routes controller should not depend on --allocate-node-cidrs ([#97029](https://github.com/kubernetes/kubernetes/pull/97029), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider and Testing] 1951 - Cluster Autoscaler version bump to v1.20.0 ([#97011](https://github.com/kubernetes/kubernetes/pull/97011), [@towca](https://github.com/towca)) 1952 - Creating a PVC with DataSource should fail for non-CSI plugins. ([#97086](https://github.com/kubernetes/kubernetes/pull/97086), [@xing-yang](https://github.com/xing-yang)) [SIG Apps and Storage] 1953 - EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. ([#99345](https://github.com/kubernetes/kubernetes/pull/99345), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 1954 - EndpointSlice controllers are less likely to create duplicate EndpointSlices. ([#100103](https://github.com/kubernetes/kubernetes/pull/100103), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 1955 - EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. ([#99756](https://github.com/kubernetes/kubernetes/pull/99756), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 1956 - Ensure all vSphere nodes are are tracked by volume attach-detach controller ([#96689](https://github.com/kubernetes/kubernetes/pull/96689), [@gnufied](https://github.com/gnufied)) 1957 - Ensure empty string annotations are copied over in rollbacks. ([#94858](https://github.com/kubernetes/kubernetes/pull/94858), [@waynepeking348](https://github.com/waynepeking348)) 1958 - Ensure only one LoadBalancer rule is created when HA mode is enabled ([#99825](https://github.com/kubernetes/kubernetes/pull/99825), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] 1959 - Ensure that client-go's EventBroadcaster is safe (non-racy) during shutdown. ([#95664](https://github.com/kubernetes/kubernetes/pull/95664), [@DirectXMan12](https://github.com/DirectXMan12)) [SIG API Machinery] 1960 - Explicitly pass `KUBE_BUILD_CONFORMANCE=y` in `package-tarballs` to reenable building the conformance tarballs. ([#100571](https://github.com/kubernetes/kubernetes/pull/100571), [@puerco](https://github.com/puerco)) 1961 - Fix Azure file migration e2e test failure when CSIMigration is turned on. ([#97877](https://github.com/kubernetes/kubernetes/pull/97877), [@andyzhangx](https://github.com/andyzhangx)) 1962 - Fix CSI-migrated inline EBS volumes failing to mount if their volumeID is prefixed by aws:// ([#96821](https://github.com/kubernetes/kubernetes/pull/96821), [@wongma7](https://github.com/wongma7)) [SIG Storage] 1963 - Fix CVE-2020-8555 for Gluster client connections. ([#97922](https://github.com/kubernetes/kubernetes/pull/97922), [@liggitt](https://github.com/liggitt)) [SIG Storage] 1964 - Fix NPE in ephemeral storage eviction ([#98261](https://github.com/kubernetes/kubernetes/pull/98261), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 1965 - Fix PermissionDenied issue on SMB mount for Windows ([#99550](https://github.com/kubernetes/kubernetes/pull/99550), [@andyzhangx](https://github.com/andyzhangx)) 1966 - Fix bug that would let the Horizontal Pod Autoscaler scale down despite at least one metric being unavailable/invalid ([#99514](https://github.com/kubernetes/kubernetes/pull/99514), [@mikkeloscar](https://github.com/mikkeloscar)) [SIG Apps and Autoscaling] 1967 - Fix cgroup handling for systemd with cgroup v2 ([#98365](https://github.com/kubernetes/kubernetes/pull/98365), [@odinuge](https://github.com/odinuge)) [SIG Node] 1968 - Fix counting error in service/nodeport/loadbalancer quota check ([#97451](https://github.com/kubernetes/kubernetes/pull/97451), [@pacoxu](https://github.com/pacoxu)) [SIG API Machinery, Network and Testing] 1969 - Fix errors when accessing Windows container stats for Dockershim ([#98510](https://github.com/kubernetes/kubernetes/pull/98510), [@jsturtevant](https://github.com/jsturtevant)) [SIG Node and Windows] 1970 - Fix kube-proxy container image architecture for non amd64 images. ([#98526](https://github.com/kubernetes/kubernetes/pull/98526), [@saschagrunert](https://github.com/saschagrunert)) 1971 - Fix missing cadvisor machine metrics. ([#97006](https://github.com/kubernetes/kubernetes/pull/97006), [@lingsamuel](https://github.com/lingsamuel)) [SIG Node] 1972 - Fix nil VMSS name when setting service to auto mode ([#97366](https://github.com/kubernetes/kubernetes/pull/97366), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 1973 - Fix privileged config of Pod Sandbox which was previously ignored. ([#96877](https://github.com/kubernetes/kubernetes/pull/96877), [@xeniumlee](https://github.com/xeniumlee)) 1974 - Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable ([#95269](https://github.com/kubernetes/kubernetes/pull/95269), [@SataQiu](https://github.com/SataQiu)) [SIG Node] 1975 - Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. ([#97980](https://github.com/kubernetes/kubernetes/pull/97980), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Node] 1976 - Fix to recover CSI volumes from certain dangling attachments ([#96617](https://github.com/kubernetes/kubernetes/pull/96617), [@yuga711](https://github.com/yuga711)) [SIG Apps and Storage] 1977 - Fix: azure file latency issue for metadata-heavy workloads ([#97082](https://github.com/kubernetes/kubernetes/pull/97082), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] 1978 - Fixed Cinder volume IDs on OpenStack Train ([#96673](https://github.com/kubernetes/kubernetes/pull/96673), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider] 1979 - Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. ([#97013](https://github.com/kubernetes/kubernetes/pull/97013), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] 1980 - Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. ([#97174](https://github.com/kubernetes/kubernetes/pull/97174), [@hanlins](https://github.com/hanlins)) [SIG Node] 1981 - Fixed a bug that causes smaller number of conntrack-max being used under CPU static policy. (#99225, @xh4n3) ([#99613](https://github.com/kubernetes/kubernetes/pull/99613), [@xh4n3](https://github.com/xh4n3)) [SIG Network] 1982 - Fixed a bug that on k8s nodes, when the policy of INPUT chain in filter table is not ACCEPT, healthcheck nodeport would not work. 1983 Added iptables rules to allow healthcheck nodeport traffic. ([#97824](https://github.com/kubernetes/kubernetes/pull/97824), [@hanlins](https://github.com/hanlins)) [SIG Network] 1984 - Fixed a bug that the kubelet cannot start on BtrfS. ([#98042](https://github.com/kubernetes/kubernetes/pull/98042), [@gjkim42](https://github.com/gjkim42)) [SIG Node] 1985 - Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. ([#95783](https://github.com/kubernetes/kubernetes/pull/95783), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] 1986 - Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children ([#98068](https://github.com/kubernetes/kubernetes/pull/98068), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Apps] 1987 - Fixed authentication_duration_seconds metric scope. Previously, it included whole apiserver request duration which yields inaccurate results. ([#99944](https://github.com/kubernetes/kubernetes/pull/99944), [@marseel](https://github.com/marseel)) 1988 - Fixed bug in CPUManager with race on container map access ([#97427](https://github.com/kubernetes/kubernetes/pull/97427), [@klueska](https://github.com/klueska)) [SIG Node] 1989 - Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. ([#99315](https://github.com/kubernetes/kubernetes/pull/99315), [@iwankgb](https://github.com/iwankgb)) [SIG Node] 1990 - Fixed cleanup of block devices when /var/lib/kubelet is a symlink. ([#96889](https://github.com/kubernetes/kubernetes/pull/96889), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] 1991 - Fixed no effect namespace when exposing deployment with --dry-run=client. ([#97492](https://github.com/kubernetes/kubernetes/pull/97492), [@masap](https://github.com/masap)) [SIG CLI] 1992 - Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. ([#98311](https://github.com/kubernetes/kubernetes/pull/98311), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] 1993 - Fixes a bug of identifying the correct containerd process. ([#97888](https://github.com/kubernetes/kubernetes/pull/97888), [@pacoxu](https://github.com/pacoxu)) 1994 - Fixes add-on manager leader election to use leases instead of endpoints, similar to what kube-controller-manager does in 1.20 ([#98968](https://github.com/kubernetes/kubernetes/pull/98968), [@liggitt](https://github.com/liggitt)) 1995 - Fixes connection errors when using `--volume-host-cidr-denylist` or `--volume-host-allow-local-loopback` ([#98436](https://github.com/kubernetes/kubernetes/pull/98436), [@liggitt](https://github.com/liggitt)) [SIG Network and Storage] 1996 - Fixes problem where invalid selector on `PodDisruptionBudget` leads to a nil pointer dereference that causes the Controller manager to crash loop. ([#98750](https://github.com/kubernetes/kubernetes/pull/98750), [@mortent](https://github.com/mortent)) 1997 - Fixes spurious errors about IPv6 in `kube-proxy` logs on nodes with IPv6 disabled. ([#99127](https://github.com/kubernetes/kubernetes/pull/99127), [@danwinship](https://github.com/danwinship)) 1998 - Fixing a bug where a failed node may not have the NoExecute taint set correctly ([#96876](https://github.com/kubernetes/kubernetes/pull/96876), [@howieyuen](https://github.com/howieyuen)) [SIG Apps and Node] 1999 - GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. ([#97740](https://github.com/kubernetes/kubernetes/pull/97740), [@prameshj](https://github.com/prameshj)) [SIG Cloud Provider and Network] 2000 - Ignore update pod with no new images in alwaysPullImages admission controller ([#96668](https://github.com/kubernetes/kubernetes/pull/96668), [@pacoxu](https://github.com/pacoxu)) [SIG Apps, Auth and Node] 2001 - Improve speed of vSphere PV provisioning and reduce number of API calls ([#100054](https://github.com/kubernetes/kubernetes/pull/100054), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] 2002 - KUBECTL_EXTERNAL_DIFF now accepts equal sign for additional parameters. ([#98158](https://github.com/kubernetes/kubernetes/pull/98158), [@dougsland](https://github.com/dougsland)) [SIG CLI] 2003 - Kube-apiserver: an update of a pod with a generic ephemeral volume dropped that volume if the feature had been disabled since creating the pod with such a volume ([#99446](https://github.com/kubernetes/kubernetes/pull/99446), [@pohly](https://github.com/pohly)) [SIG Apps, Node and Storage] 2004 - Kube-proxy: remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS ([#97336](https://github.com/kubernetes/kubernetes/pull/97336), [@maaoBit](https://github.com/maaoBit)) [SIG Network] 2005 - Kubeadm installs etcd v3.4.13 when creating cluster v1.19 ([#97244](https://github.com/kubernetes/kubernetes/pull/97244), [@pacoxu](https://github.com/pacoxu)) 2006 - Kubeadm: Fixes a kubeadm upgrade bug that could cause a custom CoreDNS configuration to be replaced with the default. ([#97016](https://github.com/kubernetes/kubernetes/pull/97016), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] 2007 - Kubeadm: Some text in the `kubeadm upgrade plan` output has changed. If you have scripts or other automation that parses this output, please review these changes and update your scripts to account for the new output. ([#98728](https://github.com/kubernetes/kubernetes/pull/98728), [@stmcginnis](https://github.com/stmcginnis)) [SIG Cluster Lifecycle] 2008 - Kubeadm: fix a bug in the host memory detection code on 32bit Linux platforms ([#97403](https://github.com/kubernetes/kubernetes/pull/97403), [@abelbarrera15](https://github.com/abelbarrera15)) [SIG Cluster Lifecycle] 2009 - Kubeadm: fix a bug where "kubeadm join" would not properly handle missing names for existing etcd members. ([#97372](https://github.com/kubernetes/kubernetes/pull/97372), [@ihgann](https://github.com/ihgann)) [SIG Cluster Lifecycle] 2010 - Kubeadm: fix a bug where "kubeadm upgrade" commands can fail if CoreDNS v1.8.0 is installed. ([#97919](https://github.com/kubernetes/kubernetes/pull/97919), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 2011 - Kubeadm: fix a bug where external credentials in an existing admin.conf prevented the CA certificate to be written in the cluster-info ConfigMap. ([#98882](https://github.com/kubernetes/kubernetes/pull/98882), [@kvaps](https://github.com/kvaps)) [SIG Cluster Lifecycle] 2012 - Kubeadm: get k8s CI version markers from k8s infra bucket ([#98836](https://github.com/kubernetes/kubernetes/pull/98836), [@hasheddan](https://github.com/hasheddan)) [SIG Cluster Lifecycle and Release] 2013 - Kubeadm: skip validating pod subnet against node-cidr-mask when allocate-node-cidrs is set to be false ([#98984](https://github.com/kubernetes/kubernetes/pull/98984), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2014 - Kubectl logs: `--ignore-errors` is now honored by all containers, maintaining consistency with parallelConsumeRequest behavior. ([#97686](https://github.com/kubernetes/kubernetes/pull/97686), [@wzshiming](https://github.com/wzshiming)) 2015 - Kubectl-convert: Fix `no kind "Ingress" is registered for version` error ([#97754](https://github.com/kubernetes/kubernetes/pull/97754), [@wzshiming](https://github.com/wzshiming)) 2016 - Kubectl: Fixed panic when describing an ingress backend without an API Group ([#100505](https://github.com/kubernetes/kubernetes/pull/100505), [@lauchokyip](https://github.com/lauchokyip)) [SIG CLI] 2017 - Kubelet now cleans up orphaned volume directories automatically ([#95301](https://github.com/kubernetes/kubernetes/pull/95301), [@lorenz](https://github.com/lorenz)) [SIG Node and Storage] 2018 - Kubelet.exe on Windows now checks that the process running as administrator and the executing user account is listed in the built-in administrators group. This is the equivalent to checking the process is running as uid 0. ([#96616](https://github.com/kubernetes/kubernetes/pull/96616), [@perithompson](https://github.com/perithompson)) [SIG Node and Windows] 2019 - Kubelet: Fix kubelet from panic after getting the wrong signal ([#98200](https://github.com/kubernetes/kubernetes/pull/98200), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 2020 - Kubelet: Fix repeatedly acquiring the inhibit lock ([#98088](https://github.com/kubernetes/kubernetes/pull/98088), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 2021 - Kubelet: Fixed the bug of getting the number of cpu when the number of cpu logical processors is more than 64 in windows ([#97378](https://github.com/kubernetes/kubernetes/pull/97378), [@hwdef](https://github.com/hwdef)) [SIG Node and Windows] 2022 - Limits lease to have 1000 maximum attached objects. ([#98257](https://github.com/kubernetes/kubernetes/pull/98257), [@lingsamuel](https://github.com/lingsamuel)) 2023 - Mitigate CVE-2020-8555 for kube-up using GCE by preventing local loopback folume hosts. ([#97934](https://github.com/kubernetes/kubernetes/pull/97934), [@mattcary](https://github.com/mattcary)) [SIG Cloud Provider and Storage] 2024 - On single-stack configured (IPv4 or IPv6, but not both) clusters, Services which are both headless (no clusterIP) and selectorless (empty or undefined selector) will report `ipFamilyPolicy RequireDualStack` and will have entries in `ipFamilies[]` for both IPv4 and IPv6. This is a change from alpha, but does not have any impact on the manually-specified Endpoints and EndpointSlices for the Service. ([#99555](https://github.com/kubernetes/kubernetes/pull/99555), [@thockin](https://github.com/thockin)) [SIG Apps and Network] 2025 - Performance regression #97685 has been fixed. ([#97860](https://github.com/kubernetes/kubernetes/pull/97860), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 2026 - Pod Log stats for windows now reports metrics ([#99221](https://github.com/kubernetes/kubernetes/pull/99221), [@jsturtevant](https://github.com/jsturtevant)) [SIG Node, Storage, Testing and Windows] 2027 - Pod status updates faster when reacting on probe results. The first readiness probe will be called faster when startup probes succeeded, which will make Pod status as ready faster. ([#98376](https://github.com/kubernetes/kubernetes/pull/98376), [@matthyx](https://github.com/matthyx)) 2028 - Readjust `kubelet_containers_per_pod_count` buckets to only show metrics greater than 1. ([#98169](https://github.com/kubernetes/kubernetes/pull/98169), [@wawa0210](https://github.com/wawa0210)) 2029 - Remove CSI topology from migrated in-tree gcepd volume. ([#97823](https://github.com/kubernetes/kubernetes/pull/97823), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Cloud Provider and Storage] 2030 - Requests with invalid timeout parameters in the request URL now appear in the audit log correctly. ([#96901](https://github.com/kubernetes/kubernetes/pull/96901), [@tkashem](https://github.com/tkashem)) [SIG API Machinery and Testing] 2031 - Resolve a "concurrent map read and map write" crashing error in the kubelet ([#95111](https://github.com/kubernetes/kubernetes/pull/95111), [@choury](https://github.com/choury)) [SIG Node] 2032 - Resolves spurious `Failed to list *v1.Secret` or `Failed to list *v1.ConfigMap` messages in kubelet logs. ([#99538](https://github.com/kubernetes/kubernetes/pull/99538), [@liggitt](https://github.com/liggitt)) [SIG Auth and Node] 2033 - ResourceQuota of an entity now inclusively calculate Pod overhead ([#99600](https://github.com/kubernetes/kubernetes/pull/99600), [@gjkim42](https://github.com/gjkim42)) 2034 - Return zero time (midnight on Jan. 1, 1970) instead of negative number when reporting startedAt and finishedAt of the not started or a running Pod when using `dockershim` as a runtime. ([#99585](https://github.com/kubernetes/kubernetes/pull/99585), [@Iceber](https://github.com/Iceber)) 2035 - Reverts breaking change to inline AzureFile volumes; referenced secrets are now searched for in the same namespace as the pod as in previous releases. ([#100563](https://github.com/kubernetes/kubernetes/pull/100563), [@msau42](https://github.com/msau42)) 2036 - Scores from InterPodAffinity have stronger differentiation. ([#98096](https://github.com/kubernetes/kubernetes/pull/98096), [@leileiwan](https://github.com/leileiwan)) [SIG Scheduling] 2037 - Specifying the KUBE_TEST_REPO environment variable when e2e tests are executed will instruct the test infrastructure to load that image from a location within the specified repo, using a predefined pattern. ([#93510](https://github.com/kubernetes/kubernetes/pull/93510), [@smarterclayton](https://github.com/smarterclayton)) [SIG Testing] 2038 - Static pods will be deleted gracefully. ([#98103](https://github.com/kubernetes/kubernetes/pull/98103), [@gjkim42](https://github.com/gjkim42)) [SIG Node] 2039 - Sync node status during kubelet node shutdown. 2040 Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. ([#98005](https://github.com/kubernetes/kubernetes/pull/98005), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 2041 - The calculation of pod UIDs for static pods has changed to ensure each static pod gets a unique value - this will cause all static pod containers to be recreated/restarted if an in-place kubelet upgrade from 1.20 to 1.21 is performed. Note that draining pods before upgrading the kubelet across minor versions is the supported upgrade path. ([#87461](https://github.com/kubernetes/kubernetes/pull/87461), [@bboreham](https://github.com/bboreham)) [SIG Node] 2042 - The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 ([#99795](https://github.com/kubernetes/kubernetes/pull/99795), [@robscott](https://github.com/robscott)) [SIG Network] 2043 - Truncates a message if it hits the `NoteLengthLimit` when the scheduler records an event for the pod that indicates the pod has failed to schedule. ([#98715](https://github.com/kubernetes/kubernetes/pull/98715), [@carlory](https://github.com/carlory)) 2044 - Updated k8s.gcr.io/ingress-gce-404-server-with-metrics-amd64 to a version that serves /metrics endpoint on a non-default port. ([#97621](https://github.com/kubernetes/kubernetes/pull/97621), [@vbannai](https://github.com/vbannai)) [SIG Cloud Provider] 2045 - Updates the commands ` 2046 - kubectl kustomize {arg} 2047 - kubectl apply -k {arg} 2048 `to use same code as kustomize CLI [v4.0.5](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.0.5) ([#98946](https://github.com/kubernetes/kubernetes/pull/98946), [@monopole](https://github.com/monopole)) 2049 - Use force unmount for NFS volumes if regular mount fails after 1 minute timeout ([#96844](https://github.com/kubernetes/kubernetes/pull/96844), [@gnufied](https://github.com/gnufied)) [SIG Storage] 2050 - Use network.Interface.VirtualMachine.ID to get the binded VM 2051 Skip standalone VM when reconciling LoadBalancer ([#97635](https://github.com/kubernetes/kubernetes/pull/97635), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 2052 - Using exec auth plugins with kubectl no longer results in warnings about constructing many client instances from the same exec auth config. ([#97857](https://github.com/kubernetes/kubernetes/pull/97857), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Auth] 2053 - When a CNI plugin returns dual-stack pod IPs, kubelet will now try to respect the 2054 "primary IP family" of the cluster by picking a primary pod IP of the same family 2055 as the (primary) node IP, rather than assuming that the CNI plugin returned the IPs 2056 in the order the administrator wanted (since some CNI plugins don't allow 2057 configuring this). ([#97979](https://github.com/kubernetes/kubernetes/pull/97979), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] 2058 - When dynamically provisioning Azure File volumes for a premium account, the requested size will be set to 100GB if the request is initially lower than this value to accommodate Azure File requirements. ([#99122](https://github.com/kubernetes/kubernetes/pull/99122), [@huffmanca](https://github.com/huffmanca)) [SIG Cloud Provider and Storage] 2059 - When using `Containerd` on Windows, the `C:\Windows\System32\drivers\etc\hosts` file will now be managed by kubelet. ([#83730](https://github.com/kubernetes/kubernetes/pull/83730), [@claudiubelu](https://github.com/claudiubelu)) 2060 - `VolumeBindingArgs` now allow `BindTimeoutSeconds` to be set as zero, while the value zero indicates no waiting for the checking of volume binding operation. ([#99835](https://github.com/kubernetes/kubernetes/pull/99835), [@chendave](https://github.com/chendave)) [SIG Scheduling and Storage] 2061 - `kubectl exec` and `kubectl attach` now honor the `--quiet` flag which suppresses output from the local binary that could be confused by a script with the remote command output (all non-failure output is hidden). In addition, print inline with exec and attach the list of alternate containers when we default to the first spec.container. ([#99004](https://github.com/kubernetes/kubernetes/pull/99004), [@smarterclayton](https://github.com/smarterclayton)) [SIG CLI] 2062 2063 ### Other (Cleanup or Flake) 2064 2065 - APIs for kubelet annotations and labels from `k8s.io/kubernetes/pkg/kubelet/apis` are now moved under `k8s.io/kubelet/pkg/apis/` ([#98931](https://github.com/kubernetes/kubernetes/pull/98931), [@michaelbeaumont](https://github.com/michaelbeaumont)) 2066 - Apiserver_request_duration_seconds is promoted to stable status. ([#99925](https://github.com/kubernetes/kubernetes/pull/99925), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Instrumentation and Testing] 2067 - Bump github.com/Azure/go-autorest/autorest to v0.11.12 ([#97033](https://github.com/kubernetes/kubernetes/pull/97033), [@patrickshan](https://github.com/patrickshan)) [SIG API Machinery, CLI, Cloud Provider and Cluster Lifecycle] 2068 - Clients required to use go1.15.8+ or go1.16+ if kube-apiserver has the goaway feature enabled to avoid unexpected data race condition. ([#98809](https://github.com/kubernetes/kubernetes/pull/98809), [@answer1991](https://github.com/answer1991)) 2069 - Delete deprecated `service.beta.kubernetes.io/azure-load-balancer-mixed-protocols` mixed procotol annotation in favor of the MixedProtocolLBService feature ([#97096](https://github.com/kubernetes/kubernetes/pull/97096), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 2070 - EndpointSlice generation is now incremented when labels change. ([#99750](https://github.com/kubernetes/kubernetes/pull/99750), [@robscott](https://github.com/robscott)) [SIG Network] 2071 - Featuregate AllowInsecureBackendProxy graduates to GA and unconditionally enabled. ([#99658](https://github.com/kubernetes/kubernetes/pull/99658), [@deads2k](https://github.com/deads2k)) 2072 - Increase timeout for pod lifecycle test to reach pod status=ready ([#96691](https://github.com/kubernetes/kubernetes/pull/96691), [@hh](https://github.com/hh)) 2073 - Increased `CSINodeIDMaxLength` from 128 bytes to 192 bytes. ([#98753](https://github.com/kubernetes/kubernetes/pull/98753), [@Jiawei0227](https://github.com/Jiawei0227)) 2074 - Kube-apiserver: The OIDC authenticator no longer waits 10 seconds before attempting to fetch the metadata required to verify tokens. ([#97693](https://github.com/kubernetes/kubernetes/pull/97693), [@enj](https://github.com/enj)) [SIG API Machinery and Auth] 2075 - Kube-proxy: Traffic from the cluster directed to ExternalIPs is always sent directly to the Service. ([#96296](https://github.com/kubernetes/kubernetes/pull/96296), [@aojea](https://github.com/aojea)) [SIG Network and Testing] 2076 - Kubeadm: change the default image repository for CI images from 'gcr.io/kubernetes-ci-images' to 'gcr.io/k8s-staging-ci-images' ([#97087](https://github.com/kubernetes/kubernetes/pull/97087), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2077 - Kubectl: The deprecated `kubectl alpha debug` command is removed. Use `kubectl debug` instead. ([#98111](https://github.com/kubernetes/kubernetes/pull/98111), [@pandaamanda](https://github.com/pandaamanda)) [SIG CLI] 2078 - Kubelet command line flags related to dockershim are now showing deprecation message as they will be removed along with dockershim in future release. ([#98730](https://github.com/kubernetes/kubernetes/pull/98730), [@dims](https://github.com/dims)) 2079 - Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97618](https://github.com/kubernetes/kubernetes/pull/97618), [@jherrera123](https://github.com/jherrera123)) [SIG Release and Testing] 2080 - Process start time on Windows now uses current process information ([#97491](https://github.com/kubernetes/kubernetes/pull/97491), [@jsturtevant](https://github.com/jsturtevant)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Windows] 2081 - Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object ([#98430](https://github.com/kubernetes/kubernetes/pull/98430), [@liggitt](https://github.com/liggitt)) [SIG Network and Testing] 2082 - The `AttachVolumeLimit` feature gate (GA since v1.17) has been removed and now unconditionally enabled. ([#96539](https://github.com/kubernetes/kubernetes/pull/96539), [@ialidzhikov](https://github.com/ialidzhikov)) 2083 - The `CSINodeInfo` feature gate that is GA since v1.17 is unconditionally enabled, and can no longer be specified via the `--feature-gates` argument. ([#96561](https://github.com/kubernetes/kubernetes/pull/96561), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Apps, Auth, Scheduling, Storage and Testing] 2084 - The `apiserver_request_total` metric is promoted to stable status and no longer has a content-type dimensions, so any alerts/charts which presume the existence of this will fail. This is however, unlikely to be the case since it was effectively an unbounded dimension in the first place. ([#99788](https://github.com/kubernetes/kubernetes/pull/99788), [@logicalhan](https://github.com/logicalhan)) 2085 - The default delegating authorization options now allow unauthenticated access to healthz, readyz, and livez. A system:masters user connecting to an authz delegator will not perform an authz check. ([#98325](https://github.com/kubernetes/kubernetes/pull/98325), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Auth, Cloud Provider and Scheduling] 2086 - The deprecated feature gates `CSIDriverRegistry`, `BlockVolume` and `CSIBlockVolume` are now unconditionally enabled and can no longer be specified in component invocations. ([#98021](https://github.com/kubernetes/kubernetes/pull/98021), [@gavinfish](https://github.com/gavinfish)) [SIG Storage] 2087 - The deprecated feature gates `RotateKubeletClientCertificate`, `AttachVolumeLimit`, `VolumePVCDataSource` and `EvenPodsSpread` are now unconditionally enabled and can no longer be specified in component invocations. ([#97306](https://github.com/kubernetes/kubernetes/pull/97306), [@gavinfish](https://github.com/gavinfish)) [SIG Node, Scheduling and Storage] 2088 - The e2e suite can be instructed not to wait for pods in kube-system to be ready or for all nodes to be ready by passing `--allowed-not-ready-nodes=-1` when invoking the e2e.test program. This allows callers to run subsets of the e2e suite in scenarios other than perfectly healthy clusters. ([#98781](https://github.com/kubernetes/kubernetes/pull/98781), [@smarterclayton](https://github.com/smarterclayton)) [SIG Testing] 2089 - The feature gates `WindowsGMSA` and `WindowsRunAsUserName` that are GA since v1.18 are now removed. ([#96531](https://github.com/kubernetes/kubernetes/pull/96531), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Node and Windows] 2090 - The new `-gce-zones` flag on the `e2e.test` binary instructs tests that check for information about how the cluster interacts with the cloud to limit their queries to the provided zone list. If not specified, the current behavior of asking the cloud provider for all available zones in multi zone clusters is preserved. ([#98787](https://github.com/kubernetes/kubernetes/pull/98787), [@smarterclayton](https://github.com/smarterclayton)) [SIG API Machinery, Cluster Lifecycle and Testing] 2091 - Update cri-tools to [v1.20.0](https://github.com/kubernetes-sigs/cri-tools/releases/tag/v1.20.0) ([#97967](https://github.com/kubernetes/kubernetes/pull/97967), [@rajibmitra](https://github.com/rajibmitra)) [SIG Cloud Provider] 2092 - Windows nodes on GCE will take longer to start due to dependencies installed at node creation time. ([#98284](https://github.com/kubernetes/kubernetes/pull/98284), [@pjh](https://github.com/pjh)) [SIG Cloud Provider] 2093 - `apiserver_storage_objects` (a newer version of `etcd_object_counts`) is promoted and marked as stable. ([#100082](https://github.com/kubernetes/kubernetes/pull/100082), [@logicalhan](https://github.com/logicalhan)) 2094 2095 ### Uncategorized 2096 2097 - GCE L4 Loadbalancers now handle > 5 ports in service spec correctly. ([#99595](https://github.com/kubernetes/kubernetes/pull/99595), [@prameshj](https://github.com/prameshj)) [SIG Cloud Provider] 2098 - The DownwardAPIHugePages feature is beta. Users may use the feature if all workers in their cluster are min 1.20 version. The feature will be enabled by default in all installations in 1.22. ([#99610](https://github.com/kubernetes/kubernetes/pull/99610), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node] 2099 2100 ## Dependencies 2101 2102 ### Added 2103 - github.com/go-errors/errors: [v1.0.1](https://github.com/go-errors/errors/tree/v1.0.1) 2104 - github.com/gobuffalo/here: [v0.6.0](https://github.com/gobuffalo/here/tree/v0.6.0) 2105 - github.com/google/shlex: [e7afc7f](https://github.com/google/shlex/tree/e7afc7f) 2106 - github.com/markbates/pkger: [v0.17.1](https://github.com/markbates/pkger/tree/v0.17.1) 2107 - github.com/moby/spdystream: [v0.2.0](https://github.com/moby/spdystream/tree/v0.2.0) 2108 - github.com/monochromegane/go-gitignore: [205db1a](https://github.com/monochromegane/go-gitignore/tree/205db1a) 2109 - github.com/niemeyer/pretty: [a10e7ca](https://github.com/niemeyer/pretty/tree/a10e7ca) 2110 - github.com/xlab/treeprint: [a009c39](https://github.com/xlab/treeprint/tree/a009c39) 2111 - go.starlark.net: 8dd3e2e 2112 - golang.org/x/term: 6a3ed07 2113 - sigs.k8s.io/kustomize/api: v0.8.5 2114 - sigs.k8s.io/kustomize/cmd/config: v0.9.7 2115 - sigs.k8s.io/kustomize/kustomize/v4: v4.0.5 2116 - sigs.k8s.io/kustomize/kyaml: v0.10.15 2117 2118 ### Changed 2119 - dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891 2120 - github.com/Azure/go-autorest/autorest: [v0.11.1 → v0.11.12](https://github.com/Azure/go-autorest/autorest/compare/v0.11.1...v0.11.12) 2121 - github.com/NYTimes/gziphandler: [56545f4 → v1.1.1](https://github.com/NYTimes/gziphandler/compare/56545f4...v1.1.1) 2122 - github.com/cilium/ebpf: [1c8d4c9 → v0.2.0](https://github.com/cilium/ebpf/compare/1c8d4c9...v0.2.0) 2123 - github.com/container-storage-interface/spec: [v1.2.0 → v1.3.0](https://github.com/container-storage-interface/spec/compare/v1.2.0...v1.3.0) 2124 - github.com/containerd/console: [v1.0.0 → v1.0.1](https://github.com/containerd/console/compare/v1.0.0...v1.0.1) 2125 - github.com/containerd/containerd: [v1.4.1 → v1.4.4](https://github.com/containerd/containerd/compare/v1.4.1...v1.4.4) 2126 - github.com/coredns/corefile-migration: [v1.0.10 → v1.0.11](https://github.com/coredns/corefile-migration/compare/v1.0.10...v1.0.11) 2127 - github.com/creack/pty: [v1.1.7 → v1.1.11](https://github.com/creack/pty/compare/v1.1.7...v1.1.11) 2128 - github.com/docker/docker: [bd33bbf → v20.10.2+incompatible](https://github.com/docker/docker/compare/bd33bbf...v20.10.2) 2129 - github.com/go-logr/logr: [v0.2.0 → v0.4.0](https://github.com/go-logr/logr/compare/v0.2.0...v0.4.0) 2130 - github.com/go-openapi/spec: [v0.19.3 → v0.19.5](https://github.com/go-openapi/spec/compare/v0.19.3...v0.19.5) 2131 - github.com/go-openapi/strfmt: [v0.19.3 → v0.19.5](https://github.com/go-openapi/strfmt/compare/v0.19.3...v0.19.5) 2132 - github.com/go-openapi/validate: [v0.19.5 → v0.19.8](https://github.com/go-openapi/validate/compare/v0.19.5...v0.19.8) 2133 - github.com/gogo/protobuf: [v1.3.1 → v1.3.2](https://github.com/gogo/protobuf/compare/v1.3.1...v1.3.2) 2134 - github.com/golang/mock: [v1.4.1 → v1.4.4](https://github.com/golang/mock/compare/v1.4.1...v1.4.4) 2135 - github.com/google/cadvisor: [v0.38.5 → v0.39.0](https://github.com/google/cadvisor/compare/v0.38.5...v0.39.0) 2136 - github.com/heketi/heketi: [c2e2a4a → v10.2.0+incompatible](https://github.com/heketi/heketi/compare/c2e2a4a...v10.2.0) 2137 - github.com/kisielk/errcheck: [v1.2.0 → v1.5.0](https://github.com/kisielk/errcheck/compare/v1.2.0...v1.5.0) 2138 - github.com/konsorten/go-windows-terminal-sequences: [v1.0.3 → v1.0.2](https://github.com/konsorten/go-windows-terminal-sequences/compare/v1.0.3...v1.0.2) 2139 - github.com/kr/text: [v0.1.0 → v0.2.0](https://github.com/kr/text/compare/v0.1.0...v0.2.0) 2140 - github.com/mattn/go-runewidth: [v0.0.2 → v0.0.7](https://github.com/mattn/go-runewidth/compare/v0.0.2...v0.0.7) 2141 - github.com/miekg/dns: [v1.1.4 → v1.1.35](https://github.com/miekg/dns/compare/v1.1.4...v1.1.35) 2142 - github.com/moby/sys/mountinfo: [v0.1.3 → v0.4.0](https://github.com/moby/sys/mountinfo/compare/v0.1.3...v0.4.0) 2143 - github.com/moby/term: [672ec06 → df9cb8a](https://github.com/moby/term/compare/672ec06...df9cb8a) 2144 - github.com/mrunalp/fileutils: [abd8a0e → v0.5.0](https://github.com/mrunalp/fileutils/compare/abd8a0e...v0.5.0) 2145 - github.com/olekukonko/tablewriter: [a0225b3 → v0.0.4](https://github.com/olekukonko/tablewriter/compare/a0225b3...v0.0.4) 2146 - github.com/opencontainers/runc: [v1.0.0-rc92 → v1.0.0-rc93](https://github.com/opencontainers/runc/compare/v1.0.0-rc92...v1.0.0-rc93) 2147 - github.com/opencontainers/runtime-spec: [4d89ac9 → e6143ca](https://github.com/opencontainers/runtime-spec/compare/4d89ac9...e6143ca) 2148 - github.com/opencontainers/selinux: [v1.6.0 → v1.8.0](https://github.com/opencontainers/selinux/compare/v1.6.0...v1.8.0) 2149 - github.com/sergi/go-diff: [v1.0.0 → v1.1.0](https://github.com/sergi/go-diff/compare/v1.0.0...v1.1.0) 2150 - github.com/sirupsen/logrus: [v1.6.0 → v1.7.0](https://github.com/sirupsen/logrus/compare/v1.6.0...v1.7.0) 2151 - github.com/syndtr/gocapability: [d983527 → 42c35b4](https://github.com/syndtr/gocapability/compare/d983527...42c35b4) 2152 - github.com/willf/bitset: [d5bec33 → v1.1.11](https://github.com/willf/bitset/compare/d5bec33...v1.1.11) 2153 - github.com/yuin/goldmark: [v1.1.27 → v1.2.1](https://github.com/yuin/goldmark/compare/v1.1.27...v1.2.1) 2154 - golang.org/x/crypto: 7f63de1 → 5ea612d 2155 - golang.org/x/exp: 6cc2880 → 85be41e 2156 - golang.org/x/mobile: d2bd2a2 → e6ae53a 2157 - golang.org/x/mod: v0.3.0 → ce943fd 2158 - golang.org/x/net: 69a7880 → 3d97a24 2159 - golang.org/x/sync: cd5d95a → 67f06af 2160 - golang.org/x/sys: 5cba982 → a50acf3 2161 - golang.org/x/time: 3af7569 → f8bda1e 2162 - golang.org/x/tools: c1934b7 → v0.1.0 2163 - gopkg.in/check.v1: 41f04d3 → 8fa4692 2164 - gopkg.in/yaml.v2: v2.2.8 → v2.4.0 2165 - gotest.tools/v3: v3.0.2 → v3.0.3 2166 - k8s.io/gengo: 83324d8 → b6c5ce2 2167 - k8s.io/klog/v2: v2.4.0 → v2.8.0 2168 - k8s.io/kube-openapi: d219536 → 591a79e 2169 - k8s.io/system-validators: v1.2.0 → v1.4.0 2170 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15 2171 - sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.1.0 2172 2173 ### Removed 2174 - github.com/codegangsta/negroni: [v1.0.0](https://github.com/codegangsta/negroni/tree/v1.0.0) 2175 - github.com/docker/spdystream: [449fdfc](https://github.com/docker/spdystream/tree/449fdfc) 2176 - github.com/golangplus/bytes: [45c989f](https://github.com/golangplus/bytes/tree/45c989f) 2177 - github.com/golangplus/fmt: [2a5d6d7](https://github.com/golangplus/fmt/tree/2a5d6d7) 2178 - github.com/gorilla/context: [v1.1.1](https://github.com/gorilla/context/tree/v1.1.1) 2179 - github.com/kr/pty: [v1.1.5](https://github.com/kr/pty/tree/v1.1.5) 2180 - rsc.io/quote/v3: v3.1.0 2181 - rsc.io/sampler: v1.3.0 2182 - sigs.k8s.io/kustomize: v2.0.3+incompatible 2183 2184 2185 2186 # v1.21.0-rc.0 2187 2188 2189 ## Downloads for v1.21.0-rc.0 2190 2191 ### Source Code 2192 2193 filename | sha512 hash 2194 -------- | ----------- 2195 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes.tar.gz) | ef53a41955d6f8a8d2a94636af98b55d633fb8a5081517559039e019b3dd65c9d10d4e7fa297ab88a7865d772f3eecf72e7b0eeba5e87accb4000c91da33e148 2196 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-src.tar.gz) | 9335a01b50d351776d3b8d00c07a5233844c51d307e361fa7e55a0620c1cb8b699e43eacf45ae9cafd8cbc44752e6987450c528a5bede8204706b7673000b5fc 2197 2198 ### Client binaries 2199 2200 filename | sha512 hash 2201 -------- | ----------- 2202 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-darwin-amd64.tar.gz) | 964135e43234cee275c452f5f06fb6d2bcd3cff3211a0d50fa35fff1cc4446bc5a0ac5125405dadcfb6596cb152afe29fabf7aad5b35b100e1288db890b70f8e 2203 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-darwin-arm64.tar.gz) | 50d782abaa4ded5e706b3192d87effa953ceabbd7d91e3d48b0c1fa2206a1963a909c14b923560f5d09cac2c7392edc5f38a13fbf1e9a40bc94e3afe8de10622 2204 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-386.tar.gz) | 72af5562f24184a2d7c27f95fa260470da979fbdcacce39a372f8f3add2991d7af8bc78f4e1dbe7a0f97e3f559b149b72a51491d3b13008da81872ee50f02f37 2205 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-amd64.tar.gz) | 1eddb8f6b51e005bc6f7b519d036cbe3d2f6d97dbf7d212dd933fb56354c29f222d050519115a9bcf94555aef095db7cf763469e47bb4ae3c6c07f97edf437cb 2206 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-arm.tar.gz) | 670f8ca60ea3cf0bb3262a772715e0ea735fccda6a92f3186299361dc455b304ae177d4017e0b67bbfa4a95e36f4cc3f7eb335e2a5130c93ac3fba2aff4519bf 2207 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-arm64.tar.gz) | a69a47907cff138ba393d8c87044fd95d97f3ca8f35d301b50742e2801ad7c229d99d6667971091f65825eb51854d585be0dd7421670110b1aa567e67e7ab4b3 2208 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-ppc64le.tar.gz) | b929feade94b71c81908abdcd4343b1e1e20098fd65e10d4d02585ad649d292d06f52c7ddc349efa188ce5b093e703c7aa9582c6ae5a69699adb87bbf5350243 2209 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-s390x.tar.gz) | 899d1470e412282cf289d8e24806d1a08c62ec0151f345ae3c9e497cc7bc0feab76498de4dd897d6adcdfa0c422e6b1a37e25d928669030f53457fd69d6e7df7 2210 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-windows-386.tar.gz) | 9f0bc90a269eabd06fe4f637b5172a3a6a7d3de26de0d66504c2e1f2093083c584ea39031db6075a7da7a86b98c48bed25aa88d4ac09060b38692c6a5b637078 2211 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-windows-amd64.tar.gz) | 05c8cc10188a1294b0d51d052942742a9b26411a08ec73494bf0e728a8a167e0a7863bdfc8864e76a371b584380098381805341e18b4b283b5d0cf298d5f7c7c 2212 2213 ### Server binaries 2214 2215 filename | sha512 hash 2216 -------- | ----------- 2217 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-amd64.tar.gz) | 355f278728ef7ac7eb2f5568c99c1429543c6302bbd0ed3bd0378c08116075e56ae850a49241313f078e2392702672ec6c9b70c8d97b4f2f5f4bee36828a63ba 2218 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-arm.tar.gz) | 9ac02c2825e2fd4e92f0c0f67180c67c24e32841ccbabc82284bf6293727ffecfae65e8a42b527c2a7ca482752384928eb65c2a1706144ae7819a6b3a1ab291c 2219 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-arm64.tar.gz) | eb412453da03c82a9248412c8ccf4d4baa1fbfa81edd8d4f81d28969b40a3727e18934accc68f643d253446c58ffd2623292402495480b3d4b2a837b5318b957 2220 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-ppc64le.tar.gz) | 07da2812c35bbc427ee5b4a0b601c3ae271e0d50ab0dd4c5c25399f43506fa2a187642eb9d4d2085df7b90264d48ea2f31088af87d9efa7eb2e87f91e1fdbde4 2221 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-s390x.tar.gz) | 3b79442a3d6e389c4ff105922a8e49994c0b6c088d2c501bd8c78d9f9e814902f5bb72c8f9c89380b750fda9b3a336759b9b68f11d70bef4f0e984564a95c29e 2222 2223 ### Node binaries 2224 2225 filename | sha512 hash 2226 -------- | ----------- 2227 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-amd64.tar.gz) | f12edf1faf5f07de1ebc5a8626601c12927902e10aca3f11e398637382fdf55365dbd9a0ef38858553fb7569495ae2cf68f155dd2e49b85b27d76fb599bb92e4 2228 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-arm.tar.gz) | 4fba8fc4e2102f07fb778aab597ec7231ea65c35e1aa618fe98b707b64a931237bd842c173e9120326e4d9deb983bb3917176762bba2212612bbc09d6e2105c4 2229 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-arm64.tar.gz) | a2e1be5459a8346839970faf4e7ebdb8ab9f3273e02babf1f3199b06bdb67434a2d18fcd1628cf1b989756e99d8dad6624a455b9db11d50f51f509f4df5c27da 2230 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-ppc64le.tar.gz) | 16d2c1cc295474fc49fe9a827ddd73e81bdd6b76af7074987b90250023f99b6d70bf474e204c7d556802111984fcb3a330740b150bdc7970d0e3634eb94a1665 2231 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-s390x.tar.gz) | 9dc6faa6cd007b13dfce703f3e271f80adcc4e029c90a4a9b4f2f143b9756f2893f8af3d7c2cf813f2bd6731cffd87d15d4229456c1685939f65bf467820ec6e 2232 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-windows-amd64.tar.gz) | f8bac2974c9142bfb80cd5eadeda79f79f27b78899a4e6e71809b795c708824ba442be83fdbadb98e01c3823dd8350776358258a205e851ed045572923cacba7 2233 2234 ## Changelog since v1.21.0-beta.1 2235 2236 ## Urgent Upgrade Notes 2237 2238 ### (No, really, you MUST read this before you upgrade) 2239 2240 - Migrated pkg/kubelet/cm/cpuset/cpuset.go to structured logging. Exit code changed from 255 to 1. ([#100007](https://github.com/kubernetes/kubernetes/pull/100007), [@utsavoza](https://github.com/utsavoza)) [SIG Instrumentation and Node] 2241 2242 ## Changes by Kind 2243 2244 ### API Change 2245 2246 - Add Probe-level terminationGracePeriodSeconds field ([#99375](https://github.com/kubernetes/kubernetes/pull/99375), [@ehashman](https://github.com/ehashman)) [SIG API Machinery, Apps, Node and Testing] 2247 - CSIServiceAccountToken is Beta now ([#99298](https://github.com/kubernetes/kubernetes/pull/99298), [@zshihang](https://github.com/zshihang)) [SIG Auth, Storage and Testing] 2248 - Discovery.k8s.io/v1beta1 EndpointSlices are deprecated in favor of discovery.k8s.io/v1, and will no longer be served in Kubernetes v1.25. ([#100472](https://github.com/kubernetes/kubernetes/pull/100472), [@liggitt](https://github.com/liggitt)) [SIG Network] 2249 - FieldManager no longer owns fields that get reset before the object is persisted (e.g. "status wiping"). ([#99661](https://github.com/kubernetes/kubernetes/pull/99661), [@kevindelgado](https://github.com/kevindelgado)) [SIG API Machinery, Auth and Testing] 2250 - Generic ephemeral volumes are beta. ([#99643](https://github.com/kubernetes/kubernetes/pull/99643), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing] 2251 - Implement the GetAvailableResources in the podresources API. ([#95734](https://github.com/kubernetes/kubernetes/pull/95734), [@fromanirh](https://github.com/fromanirh)) [SIG Instrumentation, Node and Testing] 2252 - The Endpoints controller will now set the `endpoints.kubernetes.io/over-capacity` annotation to "warning" when an Endpoints resource contains more than 1000 addresses. In a future release, the controller will truncate Endpoints that exceed this limit. The EndpointSlice API can be used to support significantly larger number of addresses. ([#99975](https://github.com/kubernetes/kubernetes/pull/99975), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 2253 - The PodDisruptionBudget API has been promoted to policy/v1 with no schema changes. The only functional change is that an empty selector (`{}`) written to a policy/v1 PodDisruptionBudget now selects all pods in the namespace. The behavior of the policy/v1beta1 API remains unchanged. The policy/v1beta1 PodDisruptionBudget API is deprecated and will no longer be served in 1.25+. ([#99290](https://github.com/kubernetes/kubernetes/pull/99290), [@mortent](https://github.com/mortent)) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Scheduling and Testing] 2254 - Topology Aware Hints are now available in alpha and can be enabled with the `TopologyAwareHints` feature gate. ([#99522](https://github.com/kubernetes/kubernetes/pull/99522), [@robscott](https://github.com/robscott)) [SIG API Machinery, Apps, Auth, Instrumentation, Network and Testing] 2255 2256 ### Feature 2257 2258 - Add e2e test to validate performance metrics of volume lifecycle operations ([#94334](https://github.com/kubernetes/kubernetes/pull/94334), [@RaunakShah](https://github.com/RaunakShah)) [SIG Storage and Testing] 2259 - EmptyDir memory backed volumes are sized as the minimum of pod allocatable memory on a host and an optional explicit user provided value. ([#100319](https://github.com/kubernetes/kubernetes/pull/100319), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node] 2260 - Enables Kubelet to check volume condition and log events to corresponding pods. ([#99284](https://github.com/kubernetes/kubernetes/pull/99284), [@fengzixu](https://github.com/fengzixu)) [SIG Apps, Instrumentation, Node and Storage] 2261 - Introduce a churn operator to scheduler perf testing framework. ([#98900](https://github.com/kubernetes/kubernetes/pull/98900), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling and Testing] 2262 - Kubernetes is now built with Golang 1.16.1 ([#100106](https://github.com/kubernetes/kubernetes/pull/100106), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] 2263 - Migrated pkg/kubelet/cm/devicemanager to structured logging ([#99976](https://github.com/kubernetes/kubernetes/pull/99976), [@knabben](https://github.com/knabben)) [SIG Instrumentation and Node] 2264 - Migrated pkg/kubelet/cm/memorymanager to structured logging ([#99974](https://github.com/kubernetes/kubernetes/pull/99974), [@knabben](https://github.com/knabben)) [SIG Instrumentation and Node] 2265 - Migrated pkg/kubelet/cm/topologymanager to structure logging ([#99969](https://github.com/kubernetes/kubernetes/pull/99969), [@knabben](https://github.com/knabben)) [SIG Instrumentation and Node] 2266 - Rename metrics `etcd_object_counts` to `apiserver_storage_object_counts` and mark it as stable. The original `etcd_object_counts` metrics name is marked as "Deprecated" and will be removed in the future. ([#99785](https://github.com/kubernetes/kubernetes/pull/99785), [@erain](https://github.com/erain)) [SIG API Machinery, Instrumentation and Testing] 2267 - Update pause container to run as pseudo user and group `65535:65535`. This implies the release of version 3.5 of the container images. ([#97963](https://github.com/kubernetes/kubernetes/pull/97963), [@saschagrunert](https://github.com/saschagrunert)) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Security and Testing] 2268 - Users might specify the `kubectl.kubernetes.io/default-exec-container` annotation in a Pod to preselect container for kubectl commands. ([#99833](https://github.com/kubernetes/kubernetes/pull/99833), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG CLI] 2269 2270 ### Bug or Regression 2271 2272 - Add ability to skip OpenAPI handler installation to the GenericAPIServer ([#100341](https://github.com/kubernetes/kubernetes/pull/100341), [@kevindelgado](https://github.com/kevindelgado)) [SIG API Machinery] 2273 - Count pod overhead against an entity's ResourceQuota ([#99600](https://github.com/kubernetes/kubernetes/pull/99600), [@gjkim42](https://github.com/gjkim42)) [SIG API Machinery and Node] 2274 - EndpointSlice controllers are less likely to create duplicate EndpointSlices. ([#100103](https://github.com/kubernetes/kubernetes/pull/100103), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 2275 - Ensure only one LoadBalancer rule is created when HA mode is enabled ([#99825](https://github.com/kubernetes/kubernetes/pull/99825), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] 2276 - Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. ([#95783](https://github.com/kubernetes/kubernetes/pull/95783), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] 2277 - Fixed authentication_duration_seconds metric. Previously it included whole apiserver request duration. ([#99944](https://github.com/kubernetes/kubernetes/pull/99944), [@marseel](https://github.com/marseel)) [SIG API Machinery, Instrumentation and Scalability] 2278 - Fixes issue where inline AzueFile secrets could not be accessed from the pod's namespace. ([#100563](https://github.com/kubernetes/kubernetes/pull/100563), [@msau42](https://github.com/msau42)) [SIG Storage] 2279 - Improve speed of vSphere PV provisioning and reduce number of API calls ([#100054](https://github.com/kubernetes/kubernetes/pull/100054), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] 2280 - Kubectl: Fixed panic when describing an ingress backend without an API Group ([#100505](https://github.com/kubernetes/kubernetes/pull/100505), [@lauchokyip](https://github.com/lauchokyip)) [SIG CLI] 2281 - Kubectl: fix case of age column in describe node (#96963, @bl-ue) ([#96963](https://github.com/kubernetes/kubernetes/pull/96963), [@bl-ue](https://github.com/bl-ue)) [SIG CLI] 2282 - Kubelet.exe on Windows now checks that the process running as administrator and the executing user account is listed in the built-in administrators group. This is the equivalent to checking the process is running as uid 0. ([#96616](https://github.com/kubernetes/kubernetes/pull/96616), [@perithompson](https://github.com/perithompson)) [SIG Node and Windows] 2283 - Kubelet: Fixed the bug of getting the number of cpu when the number of cpu logical processors is more than 64 in windows ([#97378](https://github.com/kubernetes/kubernetes/pull/97378), [@hwdef](https://github.com/hwdef)) [SIG Node and Windows] 2284 - Pass `KUBE_BUILD_CONFORMANCE=y` to the package-tarballs to reenable building the conformance tarballs. ([#100571](https://github.com/kubernetes/kubernetes/pull/100571), [@puerco](https://github.com/puerco)) [SIG Release] 2285 - Pod Log stats for windows now reports metrics ([#99221](https://github.com/kubernetes/kubernetes/pull/99221), [@jsturtevant](https://github.com/jsturtevant)) [SIG Node, Storage, Testing and Windows] 2286 2287 ### Other (Cleanup or Flake) 2288 2289 - A new storage E2E testsuite covers CSIStorageCapacity publishing if a driver opts into the test. ([#100537](https://github.com/kubernetes/kubernetes/pull/100537), [@pohly](https://github.com/pohly)) [SIG Storage and Testing] 2290 - Convert cmd/kubelet/app/server.go to structured logging ([#98334](https://github.com/kubernetes/kubernetes/pull/98334), [@wawa0210](https://github.com/wawa0210)) [SIG Node] 2291 - If kube-apiserver enabled goaway feature, clients required golang 1.15.8 or 1.16+ version to avoid un-expected data race issue. ([#98809](https://github.com/kubernetes/kubernetes/pull/98809), [@answer1991](https://github.com/answer1991)) [SIG API Machinery] 2292 - Increased CSINodeIDMaxLength from 128 bytes to 192 bytes. ([#98753](https://github.com/kubernetes/kubernetes/pull/98753), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps and Storage] 2293 - Migrate `pkg/kubelet/pluginmanager` to structured logging ([#99885](https://github.com/kubernetes/kubernetes/pull/99885), [@qingwave](https://github.com/qingwave)) [SIG Node] 2294 - Migrate `pkg/kubelet/preemption/preemption.go` and `pkg/kubelet/logs/container_log_manager.go` to structured logging ([#99848](https://github.com/kubernetes/kubernetes/pull/99848), [@qingwave](https://github.com/qingwave)) [SIG Node] 2295 - Migrate `pkg/kubelet/(cri)` to structured logging ([#99006](https://github.com/kubernetes/kubernetes/pull/99006), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node] 2296 - Migrate `pkg/kubelet/(node, pod)` to structured logging ([#98847](https://github.com/kubernetes/kubernetes/pull/98847), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node] 2297 - Migrate `pkg/kubelet/(volume,container)` to structured logging ([#98850](https://github.com/kubernetes/kubernetes/pull/98850), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node] 2298 - Migrate `pkg/kubelet/kubelet_node_status.go` to structured logging ([#98154](https://github.com/kubernetes/kubernetes/pull/98154), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node and Release] 2299 - Migrate `pkg/kubelet/lifecycle,oom` to structured logging ([#99479](https://github.com/kubernetes/kubernetes/pull/99479), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Node] 2300 - Migrate cmd/kubelet/+ pkg/kubelet/cadvisor/cadvisor_linux.go + pkg/kubelet/cri/remote/util/util_unix.go + pkg/kubelet/images/image_manager.go to structured logging ([#99994](https://github.com/kubernetes/kubernetes/pull/99994), [@AfrouzMashayekhi](https://github.com/AfrouzMashayekhi)) [SIG Instrumentation and Node] 2301 - Migrate pkg/kubelet/cm/container_manager_linux.go and pkg/kubelet/cm/container_manager_stub.go to structured logging ([#100001](https://github.com/kubernetes/kubernetes/pull/100001), [@shiyajuan123](https://github.com/shiyajuan123)) [SIG Instrumentation and Node] 2302 - Migrate pkg/kubelet/cm/cpumanage/{topology/togit pology.go, policy_none.go, cpu_assignment.go} to structured logging ([#100163](https://github.com/kubernetes/kubernetes/pull/100163), [@lala123912](https://github.com/lala123912)) [SIG Instrumentation and Node] 2303 - Migrate pkg/kubelet/cm/cpumanager/state to structured logging ([#99563](https://github.com/kubernetes/kubernetes/pull/99563), [@jmguzik](https://github.com/jmguzik)) [SIG Instrumentation and Node] 2304 - Migrate pkg/kubelet/config to structured logging ([#100002](https://github.com/kubernetes/kubernetes/pull/100002), [@AfrouzMashayekhi](https://github.com/AfrouzMashayekhi)) [SIG Instrumentation and Node] 2305 - Migrate pkg/kubelet/kubelet.go to structured logging ([#99861](https://github.com/kubernetes/kubernetes/pull/99861), [@navidshaikh](https://github.com/navidshaikh)) [SIG Instrumentation and Node] 2306 - Migrate pkg/kubelet/kubeletconfig to structured logging ([#100265](https://github.com/kubernetes/kubernetes/pull/100265), [@ehashman](https://github.com/ehashman)) [SIG Node] 2307 - Migrate pkg/kubelet/kuberuntime to structured logging ([#99970](https://github.com/kubernetes/kubernetes/pull/99970), [@krzysiekg](https://github.com/krzysiekg)) [SIG Instrumentation and Node] 2308 - Migrate pkg/kubelet/prober to structured logging ([#99830](https://github.com/kubernetes/kubernetes/pull/99830), [@krzysiekg](https://github.com/krzysiekg)) [SIG Instrumentation and Node] 2309 - Migrate pkg/kubelet/winstats to structured logging ([#99855](https://github.com/kubernetes/kubernetes/pull/99855), [@hexxdump](https://github.com/hexxdump)) [SIG Instrumentation and Node] 2310 - Migrate probe log messages to structured logging ([#97093](https://github.com/kubernetes/kubernetes/pull/97093), [@aldudko](https://github.com/aldudko)) [SIG Instrumentation and Node] 2311 - Migrate remaining kubelet files to structured logging ([#100196](https://github.com/kubernetes/kubernetes/pull/100196), [@ehashman](https://github.com/ehashman)) [SIG Instrumentation and Node] 2312 - `apiserver_storage_objects` (a newer version of `etcd_object_counts) is promoted and marked as stable. ([#100082](https://github.com/kubernetes/kubernetes/pull/100082), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Instrumentation and Testing] 2313 2314 ## Dependencies 2315 2316 ### Added 2317 _Nothing has changed._ 2318 2319 ### Changed 2320 - github.com/cilium/ebpf: [1c8d4c9 → v0.2.0](https://github.com/cilium/ebpf/compare/1c8d4c9...v0.2.0) 2321 - github.com/containerd/console: [v1.0.0 → v1.0.1](https://github.com/containerd/console/compare/v1.0.0...v1.0.1) 2322 - github.com/containerd/containerd: [v1.4.1 → v1.4.4](https://github.com/containerd/containerd/compare/v1.4.1...v1.4.4) 2323 - github.com/creack/pty: [v1.1.9 → v1.1.11](https://github.com/creack/pty/compare/v1.1.9...v1.1.11) 2324 - github.com/docker/docker: [bd33bbf → v20.10.2+incompatible](https://github.com/docker/docker/compare/bd33bbf...v20.10.2) 2325 - github.com/google/cadvisor: [v0.38.8 → v0.39.0](https://github.com/google/cadvisor/compare/v0.38.8...v0.39.0) 2326 - github.com/konsorten/go-windows-terminal-sequences: [v1.0.3 → v1.0.2](https://github.com/konsorten/go-windows-terminal-sequences/compare/v1.0.3...v1.0.2) 2327 - github.com/moby/sys/mountinfo: [v0.1.3 → v0.4.0](https://github.com/moby/sys/mountinfo/compare/v0.1.3...v0.4.0) 2328 - github.com/moby/term: [672ec06 → df9cb8a](https://github.com/moby/term/compare/672ec06...df9cb8a) 2329 - github.com/mrunalp/fileutils: [abd8a0e → v0.5.0](https://github.com/mrunalp/fileutils/compare/abd8a0e...v0.5.0) 2330 - github.com/opencontainers/runc: [v1.0.0-rc92 → v1.0.0-rc93](https://github.com/opencontainers/runc/compare/v1.0.0-rc92...v1.0.0-rc93) 2331 - github.com/opencontainers/runtime-spec: [4d89ac9 → e6143ca](https://github.com/opencontainers/runtime-spec/compare/4d89ac9...e6143ca) 2332 - github.com/opencontainers/selinux: [v1.6.0 → v1.8.0](https://github.com/opencontainers/selinux/compare/v1.6.0...v1.8.0) 2333 - github.com/sirupsen/logrus: [v1.6.0 → v1.7.0](https://github.com/sirupsen/logrus/compare/v1.6.0...v1.7.0) 2334 - github.com/syndtr/gocapability: [d983527 → 42c35b4](https://github.com/syndtr/gocapability/compare/d983527...42c35b4) 2335 - github.com/willf/bitset: [d5bec33 → v1.1.11](https://github.com/willf/bitset/compare/d5bec33...v1.1.11) 2336 - gotest.tools/v3: v3.0.2 → v3.0.3 2337 - k8s.io/klog/v2: v2.5.0 → v2.8.0 2338 - sigs.k8s.io/structured-merge-diff/v4: v4.0.3 → v4.1.0 2339 2340 ### Removed 2341 _Nothing has changed._ 2342 2343 2344 2345 # v1.21.0-beta.1 2346 2347 2348 ## Downloads for v1.21.0-beta.1 2349 2350 ### Source Code 2351 2352 filename | sha512 hash 2353 -------- | ----------- 2354 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes.tar.gz) | c9f4f25242e319e5d90f49d26f239a930aad69677c0f3c2387c56bb13482648a26ed234be2bfe2352508f35010e3eb6d3b127c31a9f24fa1e53ac99c38520fe4 2355 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-src.tar.gz) | 255357db8fa160cab2187658906b674a8b0d9b9a5b5f688cc7b69dc124f5da00362c6cc18ae9b80f7ddb3da6f64c2ab2f12fb9b63a4e063c7366a5375b175cda 2356 2357 ### Client binaries 2358 2359 filename | sha512 hash 2360 -------- | ----------- 2361 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-darwin-amd64.tar.gz) | 02efd389c8126456416fd2c7ea25c3cc30f612649ad91f631f068d6c0e5e539484d3763cb9a8645ad6b8077e4fcd1552a659d7516ebc4ce6828cf823b65c3016 2362 [kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-darwin-arm64.tar.gz) | ac90dcd1699d1d7ff9c8342d481f6d0d97ccdc3ec501a56dc7c9e1898a8f77f712bf66942d304bfe581b5494f13e3efa211865de88f89749780e9e26e673dbdb 2363 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-386.tar.gz) | cce5fb84cc7a1ee664f89d8ad3064307c51c044e9ddd2ae5a004939b69d3b3ef6f29acc5782e27d0c8f0d6d3d9c96e922f5d1b99d210ca3e754666d775df9f0c 2364 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-amd64.tar.gz) | 2e93bbd2e60ad7cd8fe495115e96c55b1dc8facd100a827ef9c197a732679b60cceb9ea7bf92a1f5e328c3b8adfa8d3922cbc5d8370e374f3381b83f5b877b4f 2365 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-arm.tar.gz) | 23f03b6a8fa9decce9b89a2c1bd3dae6d0b2f9e533e35a79e2c5a29326a165259677594ae83c877219a21bdb95557a284e55f4eec12954742794579c89a7d7e5 2366 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-arm64.tar.gz) | 3acf3101b46568b0ded6b90f13df0e918870d6812dc1a584903ddb8ba146484a204b9e442f863df47c7d4dab043fd9f7294c5510d3eb09004993d6d3b1e9e13c 2367 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-ppc64le.tar.gz) | f749198df69577f62872d3096138a1b8969ec6b1636eb68eb56640bf33cf5f97a11df4363462749a1c0dc3ccbb8ae76c5d66864bf1c5cf7e52599caaf498e504 2368 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-s390x.tar.gz) | 3f6c0189d59fca22cdded3a02c672ef703d17e6ab0831e173a870e14ccec436c142600e9fc35b403571b6906f2be8d18d38d33330f7caada971bbe1187b388f6 2369 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-windows-386.tar.gz) | 03d92371c425cf331c80807c0ac56f953be304fc6719057258a363d527d186d610e1d4b4d401b34128062983265c2e21f2d2389231aa66a6f5787eee78142cf6 2370 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-windows-amd64.tar.gz) | 489ece0c886a025ca3a25d28518637a5a824ea6544e7ef8778321036f13c8909a978ad4ceca966cec1e1cda99f25ca78bfd37460d1231c77436d216d43c872ad 2371 2372 ### Server binaries 2373 2374 filename | sha512 hash 2375 -------- | ----------- 2376 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-amd64.tar.gz) | 2e95cb31d5afcb6842c41d25b7d0c18dd7e65693b2d93c8aa44e5275f9c6201e1a67685c7a8ddefa334babb04cb559d26e39b6a18497695a07dc270568cae108 2377 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-arm.tar.gz) | 2927e82b98404c077196ce3968f3afd51a7576aa56d516019bd3976771c0213ba01e78da5b77478528e770da0d334e9457995fafb98820ed68b2ee34beb68856 2378 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-arm64.tar.gz) | e0f7aea3ea598214a9817bc04949389cb7e4e7b9503141a590ef48c0b681fe44a4243ebc6280752fa41aa1093149b3ee1bcef7664edb746097a342281825430b 2379 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-ppc64le.tar.gz) | c011f7eb01294e9ba5d5ced719068466f88ed595dcb8d554a36a4dd5118fb6b3d6bafe8bf89aa2d42988e69793ed777ba77b8876c6ec74f898a43cfce1f61bf4 2380 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-s390x.tar.gz) | 15f6683e7f16caab7eebead2b7c15799460abbf035a43de0b75f96b0be19908f58add98a777a0cca916230d60cf6bfe3fee92b9dcff50274b1e37c243c157969 2381 2382 ### Node binaries 2383 2384 filename | sha512 hash 2385 -------- | ----------- 2386 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-amd64.tar.gz) | ed58679561197110f366b9109f7afd62c227bfc271918ccf3eea203bb2ab6428eb5db4dd6c965f202a8a636f66da199470269b863815809b99d53d2fa47af2ea 2387 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-arm.tar.gz) | 7e6c7f1957fcdecec8fef689c5019edbc0d0c11d22dafbfef0a07121d10d8f6273644f73511bd06a9a88b04d81a940bd6645ffb5711422af64af547a45c76273 2388 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-arm64.tar.gz) | a3618f29967e7a1574917a67f0296e65780321eda484b99aa32bfd4dc9b35acdefce33da952ac52dfb509fbac5bf700cf177431fad2ab4adcab0544538939faa 2389 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-ppc64le.tar.gz) | 326d3eb521b41bdf489912177f70b8cdd7cd828bb9b3d847ed3694eb27e457f24e0a88b8e51b726eee39800a3c5a40c1b30e3a8ec4a34d8041b3d8ef05d1b749 2390 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-s390x.tar.gz) | 022d05ebaa66a0332c4fe18cdaf23d14c2c7e4d1f2af7f27baaf1eb042e6890dc3434b4ac8ba58c35d590717956f8c3458112685aff4938b94b18e263c3f4256 2391 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-windows-amd64.tar.gz) | fa691ed93f07af6bc1cf57e20a30580d6c528f88e5fea3c14f39c1820969dc5a0eb476c5b87b288593d0c086c4dd93aff6165082393283c3f46c210f9bb66d61 2392 2393 ## Changelog since v1.21.0-beta.0 2394 2395 ## Urgent Upgrade Notes 2396 2397 ### (No, really, you MUST read this before you upgrade) 2398 2399 - Kubeadm: during "init" an empty cgroupDriver value in the KubeletConfiguration is now always set to "systemd" unless the user is explicit about it. This requires existing machine setups to configure the container runtime to use the "systemd" driver. Documentation on this topic can be found here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/. When upgrading existing clusters / nodes using "kubeadm upgrade" the old cgroupDriver value is preserved, but in 1.22 this change will also apply to "upgrade". For more information on migrating to the "systemd" driver or remaining on the "cgroupfs" driver see: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/. ([#99471](https://github.com/kubernetes/kubernetes/pull/99471), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 2400 - Migrate `pkg/kubelet/(dockershim, network)` to structured logging 2401 Exit code changed from 255 to 1 ([#98939](https://github.com/kubernetes/kubernetes/pull/98939), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Network and Node] 2402 - Migrate `pkg/kubelet/certificate` to structured logging 2403 Exit code changed from 255 to 1 ([#98993](https://github.com/kubernetes/kubernetes/pull/98993), [@SataQiu](https://github.com/SataQiu)) [SIG Auth and Node] 2404 - Newly provisioned PVs by EBS plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. ([#99130](https://github.com/kubernetes/kubernetes/pull/99130), [@ayberk](https://github.com/ayberk)) [SIG Cloud Provider, Storage and Testing] 2405 - Newly provisioned PVs by OpenStack Cinder plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. ([#99719](https://github.com/kubernetes/kubernetes/pull/99719), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider and Storage] 2406 - OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. ([#98538](https://github.com/kubernetes/kubernetes/pull/98538), [@dims](https://github.com/dims)) [SIG Storage] 2407 - Package pkg/kubelet/server migrated to structured logging 2408 Exit code changed from 255 to 1 ([#99838](https://github.com/kubernetes/kubernetes/pull/99838), [@adisky](https://github.com/adisky)) [SIG Node] 2409 - Pkg/kubelet/kuberuntime/kuberuntime_manager.go migrated to structured logging 2410 Exit code changed from 255 to 1 ([#99841](https://github.com/kubernetes/kubernetes/pull/99841), [@adisky](https://github.com/adisky)) [SIG Instrumentation and Node] 2411 2412 ## Changes by Kind 2413 2414 ### Deprecation 2415 2416 - Kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error. ([#99646](https://github.com/kubernetes/kubernetes/pull/99646), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] 2417 - Remove deprecated --generator --replicas --service-generator --service-overrides --schedule from kubectl run 2418 Deprecate --serviceaccount --hostport --requests --limits in kubectl run ([#99732](https://github.com/kubernetes/kubernetes/pull/99732), [@soltysh](https://github.com/soltysh)) [SIG CLI and Testing] 2419 - `audit.k8s.io/v1beta1` and `audit.k8s.io/v1alpha1` audit policy configuration and audit events are deprecated in favor of `audit.k8s.io/v1`, available since v1.13. kube-apiserver invocations that specify alpha or beta policy configurations with `--audit-policy-file`, or explicitly request alpha or beta audit events with `--audit-log-version` / `--audit-webhook-version` must update to use `audit.k8s.io/v1` and accept `audit.k8s.io/v1` events prior to v1.24. ([#98858](https://github.com/kubernetes/kubernetes/pull/98858), [@carlory](https://github.com/carlory)) [SIG Auth] 2420 - `diskformat` stroage class parameter for in-tree vSphere volume plugin is deprecated as of v1.21 release. Please consider updating storageclass and remove `diskformat` parameter. vSphere CSI Driver does not support diskformat storageclass parameter. 2421 2422 vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3. 2423 2424 VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM's Hardware version set to at least vmx-15. 2425 2426 Multi vCenter support is deprecated as of v1.21. If you have a Kubernetes cluster spanning across multiple vCenter servers, please consider moving all k8s nodes to a single vCenter Server. vSphere CSI Driver does not support Kubernetes deployment spanning across multiple vCenter servers. 2427 2428 Support for these deprecations will be available till Kubernetes v1.24. ([#98546](https://github.com/kubernetes/kubernetes/pull/98546), [@divyenpatel](https://github.com/divyenpatel)) [SIG Cloud Provider and Storage] 2429 2430 ### API Change 2431 2432 - 1. PodAffinityTerm includes a namespaceSelector field to allow selecting eligible namespaces based on their labels. 2433 2. A new CrossNamespacePodAffinity quota scope API that allows restricting which namespaces allowed to use PodAffinityTerm with corss-namespace reference via namespaceSelector or namespaces fields. ([#98582](https://github.com/kubernetes/kubernetes/pull/98582), [@ahg-g](https://github.com/ahg-g)) [SIG API Machinery, Apps, Auth and Testing] 2434 - Add a default metadata name labels for selecting any namespace by its name. ([#96968](https://github.com/kubernetes/kubernetes/pull/96968), [@jayunit100](https://github.com/jayunit100)) [SIG API Machinery, Apps, Cloud Provider, Storage and Testing] 2435 - Added `.spec.completionMode` field to Job, with accepted values `NonIndexed` (default) and `Indexed` ([#98441](https://github.com/kubernetes/kubernetes/pull/98441), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and CLI] 2436 - Clarified NetworkPolicy policyTypes documentation ([#97216](https://github.com/kubernetes/kubernetes/pull/97216), [@joejulian](https://github.com/joejulian)) [SIG Network] 2437 - DaemonSets accept a MaxSurge integer or percent on their rolling update strategy that will launch the updated pod on nodes and wait for those pods to go ready before marking the old out-of-date pods as deleted. This allows workloads to avoid downtime during upgrades when deployed using DaemonSets. This feature is alpha and is behind the DaemonSetUpdateSurge feature gate. ([#96441](https://github.com/kubernetes/kubernetes/pull/96441), [@smarterclayton](https://github.com/smarterclayton)) [SIG Apps and Testing] 2438 - EndpointSlice API is now GA. The EndpointSlice topology field has been removed from the GA API and will be replaced by a new per Endpoint Zone field. If the topology field was previously used, it will be converted into an annotation in the v1 Resource. The discovery.k8s.io/v1alpha1 API is removed. ([#99662](https://github.com/kubernetes/kubernetes/pull/99662), [@swetharepakula](https://github.com/swetharepakula)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network and Testing] 2439 - EndpointSlice Controllers are now GA. The EndpointSlice Controller will not populate the `deprecatedTopology` field and will only provide topology information through the `zone` and `nodeName` fields. ([#99870](https://github.com/kubernetes/kubernetes/pull/99870), [@swetharepakula](https://github.com/swetharepakula)) [SIG API Machinery, Apps, Auth, Network and Testing] 2440 - IngressClass resource can now reference a resource in a specific namespace 2441 for implementation-specific configuration(previously only Cluster-level resources were allowed). 2442 This feature can be enabled using the IngressClassNamespacedParams feature gate. ([#99275](https://github.com/kubernetes/kubernetes/pull/99275), [@hbagdi](https://github.com/hbagdi)) [SIG API Machinery, CLI and Network] 2443 - Introduce conditions for PodDisruptionBudget ([#98127](https://github.com/kubernetes/kubernetes/pull/98127), [@mortent](https://github.com/mortent)) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation] 2444 - Jobs API has a new .spec.suspend field that can be used to suspend and resume Jobs ([#98727](https://github.com/kubernetes/kubernetes/pull/98727), [@adtac](https://github.com/adtac)) [SIG API Machinery, Apps, Node, Scheduling and Testing] 2445 - Kubelet Graceful Node Shutdown feature is now beta. ([#99735](https://github.com/kubernetes/kubernetes/pull/99735), [@bobbypage](https://github.com/bobbypage)) [SIG Node] 2446 - Limit the quest value of hugepage to integer multiple of page size. ([#98515](https://github.com/kubernetes/kubernetes/pull/98515), [@lala123912](https://github.com/lala123912)) [SIG Apps] 2447 - One new field "InternalTrafficPolicy" in Service is added. 2448 It specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. 2449 "Cluster" routes internal traffic to a Service to all endpoints. 2450 "Local" routes traffic to node-local endpoints only, and traffic is dropped if no node-local endpoints are ready. 2451 The default value is "Cluster". ([#96600](https://github.com/kubernetes/kubernetes/pull/96600), [@maplain](https://github.com/maplain)) [SIG API Machinery, Apps and Network] 2452 - PodSecurityPolicy only stores "generic" as allowed volume type if the GenericEphemeralVolume feature gate is enabled ([#98918](https://github.com/kubernetes/kubernetes/pull/98918), [@pohly](https://github.com/pohly)) [SIG Auth and Security] 2453 - Promote CronJobs to batch/v1 ([#99423](https://github.com/kubernetes/kubernetes/pull/99423), [@soltysh](https://github.com/soltysh)) [SIG API Machinery, Apps, CLI and Testing] 2454 - Remove support for building Kubernetes with bazel. ([#99561](https://github.com/kubernetes/kubernetes/pull/99561), [@BenTheElder](https://github.com/BenTheElder)) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows] 2455 - Setting loadBalancerClass in load balancer type of service is available with this PR. 2456 Users who want to use a custom load balancer can specify loadBalancerClass to achieve it. ([#98277](https://github.com/kubernetes/kubernetes/pull/98277), [@XudongLiuHarold](https://github.com/XudongLiuHarold)) [SIG API Machinery, Apps, Cloud Provider and Network] 2457 - Storage capacity tracking (= the CSIStorageCapacity feature) is beta, storage.k8s.io/v1alpha1/VolumeAttachment and storage.k8s.io/v1alpha1/CSIStorageCapacity objects are deprecated ([#99641](https://github.com/kubernetes/kubernetes/pull/99641), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Scheduling, Storage and Testing] 2458 - Support for Indexed Job: a Job that is considered completed when Pods associated to indexes from 0 to (.spec.completions-1) have succeeded. ([#98812](https://github.com/kubernetes/kubernetes/pull/98812), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and CLI] 2459 - The apiserver now resets managedFields that got corrupted by a mutating admission controller. ([#98074](https://github.com/kubernetes/kubernetes/pull/98074), [@kwiesmueller](https://github.com/kwiesmueller)) [SIG API Machinery and Testing] 2460 - `controller.kubernetes.io/pod-deletion-cost` annotation can be set to offer a hint on the cost of deleting a pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are deleted first. This is an alpha feature. ([#99163](https://github.com/kubernetes/kubernetes/pull/99163), [@ahg-g](https://github.com/ahg-g)) [SIG Apps] 2461 2462 ### Feature 2463 2464 - A client-go metric, rest_client_exec_plugin_call_total, has been added to track total calls to client-go credential plugins. ([#98892](https://github.com/kubernetes/kubernetes/pull/98892), [@ankeesler](https://github.com/ankeesler)) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation] 2465 - Add --use-protocol-buffers flag to kubectl top pods and nodes ([#96655](https://github.com/kubernetes/kubernetes/pull/96655), [@serathius](https://github.com/serathius)) [SIG CLI] 2466 - Add support to generate client-side binaries for new darwin/arm64 platform ([#97743](https://github.com/kubernetes/kubernetes/pull/97743), [@dims](https://github.com/dims)) [SIG Release and Testing] 2467 - Added `ephemeral_volume_controller_create[_failures]_total` counters to kube-controller-manager metrics ([#99115](https://github.com/kubernetes/kubernetes/pull/99115), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation and Storage] 2468 - Adds alpha feature `VolumeCapacityPriority` which makes the scheduler prioritize nodes based on the best matching size of statically provisioned PVs across multiple topologies. ([#96347](https://github.com/kubernetes/kubernetes/pull/96347), [@cofyc](https://github.com/cofyc)) [SIG Apps, Network, Scheduling, Storage and Testing] 2469 - Adds two new metrics to cronjobs, a histogram to track the time difference when a job is created and the expected time when it should be created, and a gauge for the missed schedules of a cronjob ([#99341](https://github.com/kubernetes/kubernetes/pull/99341), [@alaypatel07](https://github.com/alaypatel07)) [SIG Apps and Instrumentation] 2470 - Alpha implementation of Kubectl Command Headers: SIG CLI KEP 859 enabled when KUBECTL_COMMAND_HEADERS environment variable set on the client command line. 2471 - To enable: export KUBECTL_COMMAND_HEADERS=1; kubectl ... ([#98952](https://github.com/kubernetes/kubernetes/pull/98952), [@seans3](https://github.com/seans3)) [SIG API Machinery and CLI] 2472 - Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. ([#99738](https://github.com/kubernetes/kubernetes/pull/99738), [@YoyinZyc](https://github.com/YoyinZyc)) [SIG API Machinery, Cluster Lifecycle and Instrumentation] 2473 - Disruption controller only sends one event per PodDisruptionBudget if scale can't be computed ([#98128](https://github.com/kubernetes/kubernetes/pull/98128), [@mortent](https://github.com/mortent)) [SIG Apps] 2474 - EndpointSliceNodeName will always be enabled, so NodeName will always be available in the v1beta1 API. ([#99746](https://github.com/kubernetes/kubernetes/pull/99746), [@swetharepakula](https://github.com/swetharepakula)) [SIG Apps and Network] 2475 - Graduate CRIContainerLogRotation feature gate to GA. ([#99651](https://github.com/kubernetes/kubernetes/pull/99651), [@umohnani8](https://github.com/umohnani8)) [SIG Node and Testing] 2476 - Kube-proxy iptables: new metric sync_proxy_rules_iptables_total that exposes the number of rules programmed per table in each iteration ([#99653](https://github.com/kubernetes/kubernetes/pull/99653), [@aojea](https://github.com/aojea)) [SIG Instrumentation and Network] 2477 - Kube-scheduler now logs plugin scoring summaries at --v=4 ([#99411](https://github.com/kubernetes/kubernetes/pull/99411), [@damemi](https://github.com/damemi)) [SIG Scheduling] 2478 - Kubeadm: a warning to user as ipv6 site-local is deprecated ([#99574](https://github.com/kubernetes/kubernetes/pull/99574), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle and Network] 2479 - Kubeadm: apply the "node.kubernetes.io/exclude-from-external-load-balancers" label on control plane nodes during "init", "join" and "upgrade" to preserve backwards compatibility with the lagacy LB mode where nodes labeled as "master" where excluded. To opt-out you can remove the label from a node. See #97543 and the linked KEP for more details. ([#98269](https://github.com/kubernetes/kubernetes/pull/98269), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 2480 - Kubeadm: if the user has customized their image repository via the kubeadm configuration, pass the custom pause image repository and tag to the kubelet via --pod-infra-container-image not only for Docker but for all container runtimes. This flag tells the kubelet that it should not garbage collect the image. ([#99476](https://github.com/kubernetes/kubernetes/pull/99476), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 2481 - Kubeadm: promote IPv6DualStack feature gate to Beta ([#99294](https://github.com/kubernetes/kubernetes/pull/99294), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 2482 - Kubectl version changed to write a warning message to stderr if the client and server version difference exceeds the supported version skew of +/-1 minor version. ([#98250](https://github.com/kubernetes/kubernetes/pull/98250), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] 2483 - Kubernetes is now built with Golang 1.16 ([#98572](https://github.com/kubernetes/kubernetes/pull/98572), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing] 2484 - Persistent Volumes formatted with the btrfs filesystem will now automatically resize when expanded. ([#99361](https://github.com/kubernetes/kubernetes/pull/99361), [@Novex](https://github.com/Novex)) [SIG Storage] 2485 - Remove cAdvisor json metrics api collected by Kubelet ([#99236](https://github.com/kubernetes/kubernetes/pull/99236), [@pacoxu](https://github.com/pacoxu)) [SIG Node] 2486 - Sysctls is now GA and locked to default ([#99158](https://github.com/kubernetes/kubernetes/pull/99158), [@wgahnagl](https://github.com/wgahnagl)) [SIG Node] 2487 - The NodeAffinity plugin implements the PreFilter extension, offering enhanced performance for Filter. ([#99213](https://github.com/kubernetes/kubernetes/pull/99213), [@AliceZhang2016](https://github.com/AliceZhang2016)) [SIG Scheduling] 2488 - The endpointslice mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields are mirrored. 2489 The well-known annotation endpoints.kubernetes.io/last-change-trigger-time is skipped and not mirrored. ([#98116](https://github.com/kubernetes/kubernetes/pull/98116), [@aojea](https://github.com/aojea)) [SIG Apps, Network and Testing] 2490 - Update the latest validated version of Docker to 20.10 ([#98977](https://github.com/kubernetes/kubernetes/pull/98977), [@neolit123](https://github.com/neolit123)) [SIG CLI, Cluster Lifecycle and Node] 2491 - Upgrade node local dns to 1.17.0 for better IPv6 support ([#99749](https://github.com/kubernetes/kubernetes/pull/99749), [@pacoxu](https://github.com/pacoxu)) [SIG Cloud Provider and Network] 2492 - Users might specify the `kubectl.kubernetes.io/default-exec-container` annotation in a Pod to preselect container for kubectl commands. ([#99581](https://github.com/kubernetes/kubernetes/pull/99581), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG CLI] 2493 - When downscaling ReplicaSets, ready and creation timestamps are compared in a logarithmic scale. ([#99212](https://github.com/kubernetes/kubernetes/pull/99212), [@damemi](https://github.com/damemi)) [SIG Apps and Testing] 2494 - When the kubelet is watching a ConfigMap or Secret purely in the context of setting environment variables 2495 for containers, only hold that watch for a defined duration before cancelling it. This change reduces the CPU 2496 and memory usage of the kube-apiserver in large clusters. ([#99393](https://github.com/kubernetes/kubernetes/pull/99393), [@chenyw1990](https://github.com/chenyw1990)) [SIG API Machinery, Node and Testing] 2497 - WindowsEndpointSliceProxying feature gate has graduated to beta and is enabled by default. This means kube-proxy will read from EndpointSlices instead of Endpoints on Windows by default. ([#99794](https://github.com/kubernetes/kubernetes/pull/99794), [@robscott](https://github.com/robscott)) [SIG Network] 2498 2499 ### Bug or Regression 2500 2501 - Creating a PVC with DataSource should fail for non-CSI plugins. ([#97086](https://github.com/kubernetes/kubernetes/pull/97086), [@xing-yang](https://github.com/xing-yang)) [SIG Apps and Storage] 2502 - EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. ([#99345](https://github.com/kubernetes/kubernetes/pull/99345), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 2503 - EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. ([#99756](https://github.com/kubernetes/kubernetes/pull/99756), [@robscott](https://github.com/robscott)) [SIG Apps and Network] 2504 - Fix --ignore-errors does not take effect if multiple logs are printed and unfollowed ([#97686](https://github.com/kubernetes/kubernetes/pull/97686), [@wzshiming](https://github.com/wzshiming)) [SIG CLI] 2505 - Fix bug that would let the Horizontal Pod Autoscaler scale down despite at least one metric being unavailable/invalid ([#99514](https://github.com/kubernetes/kubernetes/pull/99514), [@mikkeloscar](https://github.com/mikkeloscar)) [SIG Apps and Autoscaling] 2506 - Fix cgroup handling for systemd with cgroup v2 ([#98365](https://github.com/kubernetes/kubernetes/pull/98365), [@odinuge](https://github.com/odinuge)) [SIG Node] 2507 - Fix smb mount PermissionDenied issue on Windows ([#99550](https://github.com/kubernetes/kubernetes/pull/99550), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider, Storage and Windows] 2508 - Fixed a bug that causes smaller number of conntrack-max being used under CPU static policy. (#99225, @xh4n3) ([#99613](https://github.com/kubernetes/kubernetes/pull/99613), [@xh4n3](https://github.com/xh4n3)) [SIG Network] 2509 - Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. ([#99315](https://github.com/kubernetes/kubernetes/pull/99315), [@iwankgb](https://github.com/iwankgb)) [SIG Node] 2510 - Fixes add-on manager leader election ([#98968](https://github.com/kubernetes/kubernetes/pull/98968), [@liggitt](https://github.com/liggitt)) [SIG Cloud Provider] 2511 - Improved update time of pod statuses following new probe results. ([#98376](https://github.com/kubernetes/kubernetes/pull/98376), [@matthyx](https://github.com/matthyx)) [SIG Node and Testing] 2512 - Kube-apiserver: an update of a pod with a generic ephemeral volume dropped that volume if the feature had been disabled since creating the pod with such a volume ([#99446](https://github.com/kubernetes/kubernetes/pull/99446), [@pohly](https://github.com/pohly)) [SIG Apps, Node and Storage] 2513 - Kubeadm: skip validating pod subnet against node-cidr-mask when allocate-node-cidrs is set to be false ([#98984](https://github.com/kubernetes/kubernetes/pull/98984), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2514 - On single-stack configured (IPv4 or IPv6, but not both) clusters, Services which are both headless (no clusterIP) and selectorless (empty or undefined selector) will report `ipFamilyPolicy RequireDualStack` and will have entries in `ipFamilies[]` for both IPv4 and IPv6. This is a change from alpha, but does not have any impact on the manually-specified Endpoints and EndpointSlices for the Service. ([#99555](https://github.com/kubernetes/kubernetes/pull/99555), [@thockin](https://github.com/thockin)) [SIG Apps and Network] 2515 - Resolves spurious `Failed to list *v1.Secret` or `Failed to list *v1.ConfigMap` messages in kubelet logs. ([#99538](https://github.com/kubernetes/kubernetes/pull/99538), [@liggitt](https://github.com/liggitt)) [SIG Auth and Node] 2516 - Return zero time (midnight on Jan. 1, 1970) instead of negative number when reporting startedAt and finishedAt of the not started or a running Pod when using dockershim as a runtime. ([#99585](https://github.com/kubernetes/kubernetes/pull/99585), [@Iceber](https://github.com/Iceber)) [SIG Node] 2517 - Stdin is now only passed to client-go exec credential plugins when it is detected to be an interactive terminal. Previously, it was passed to client-go exec plugins when **stdout** was detected to be an interactive terminal. ([#99654](https://github.com/kubernetes/kubernetes/pull/99654), [@ankeesler](https://github.com/ankeesler)) [SIG API Machinery and Auth] 2518 - The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 ([#99795](https://github.com/kubernetes/kubernetes/pull/99795), [@robscott](https://github.com/robscott)) [SIG Network] 2519 - Updates the commands 2520 - kubectl kustomize {arg} 2521 - kubectl apply -k {arg} 2522 to use same code as kustomize CLI v4.0.5 2523 - [v4.0.5]: https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.0.5 ([#98946](https://github.com/kubernetes/kubernetes/pull/98946), [@monopole](https://github.com/monopole)) [SIG API Machinery, Architecture, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage] 2524 - When a CNI plugin returns dual-stack pod IPs, kubelet will now try to respect the 2525 "primary IP family" of the cluster by picking a primary pod IP of the same family 2526 as the (primary) node IP, rather than assuming that the CNI plugin returned the IPs 2527 in the order the administrator wanted (since some CNI plugins don't allow 2528 configuring this). ([#97979](https://github.com/kubernetes/kubernetes/pull/97979), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] 2529 - When using Containerd on Windows, the "C:\Windows\System32\drivers\etc\hosts" file will now be managed by kubelet. ([#83730](https://github.com/kubernetes/kubernetes/pull/83730), [@claudiubelu](https://github.com/claudiubelu)) [SIG Node and Windows] 2530 - `VolumeBindingArgs` now allow `BindTimeoutSeconds` to be set as zero, while the value zero indicates no waiting for the checking of volume binding operation. ([#99835](https://github.com/kubernetes/kubernetes/pull/99835), [@chendave](https://github.com/chendave)) [SIG Scheduling and Storage] 2531 - `kubectl exec` and `kubectl attach` now honor the `--quiet` flag which suppresses output from the local binary that could be confused by a script with the remote command output (all non-failure output is hidden). In addition, print inline with exec and attach the list of alternate containers when we default to the first spec.container. ([#99004](https://github.com/kubernetes/kubernetes/pull/99004), [@smarterclayton](https://github.com/smarterclayton)) [SIG CLI] 2532 2533 ### Other (Cleanup or Flake) 2534 2535 - Apiserver_request_duration_seconds is promoted to stable status. ([#99925](https://github.com/kubernetes/kubernetes/pull/99925), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Instrumentation and Testing] 2536 - Apiserver_request_total is promoted to stable status and no longer has a content-type dimensions, so any alerts/charts which presume the existence of this will fail. This is however, unlikely to be the case since it was effectively an unbounded dimension in the first place. ([#99788](https://github.com/kubernetes/kubernetes/pull/99788), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Instrumentation and Testing] 2537 - EndpointSlice generation is now incremented when labels change. ([#99750](https://github.com/kubernetes/kubernetes/pull/99750), [@robscott](https://github.com/robscott)) [SIG Network] 2538 - Featuregate AllowInsecureBackendProxy is promoted to GA ([#99658](https://github.com/kubernetes/kubernetes/pull/99658), [@deads2k](https://github.com/deads2k)) [SIG API Machinery] 2539 - Migrate `pkg/kubelet/(eviction)` to structured logging ([#99032](https://github.com/kubernetes/kubernetes/pull/99032), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node] 2540 - Migrate deployment controller log messages to structured logging ([#97507](https://github.com/kubernetes/kubernetes/pull/97507), [@aldudko](https://github.com/aldudko)) [SIG Apps] 2541 - Migrate pkg/kubelet/cloudresource to structured logging ([#98999](https://github.com/kubernetes/kubernetes/pull/98999), [@sladyn98](https://github.com/sladyn98)) [SIG Node] 2542 - Migrate pkg/kubelet/cri/remote logs to structured logging ([#98589](https://github.com/kubernetes/kubernetes/pull/98589), [@chenyw1990](https://github.com/chenyw1990)) [SIG Node] 2543 - Migrate pkg/kubelet/kuberuntime/kuberuntime_container.go logs to structured logging ([#96973](https://github.com/kubernetes/kubernetes/pull/96973), [@chenyw1990](https://github.com/chenyw1990)) [SIG Instrumentation and Node] 2544 - Migrate pkg/kubelet/status to structured logging ([#99836](https://github.com/kubernetes/kubernetes/pull/99836), [@navidshaikh](https://github.com/navidshaikh)) [SIG Instrumentation and Node] 2545 - Migrate pkg/kubelet/token to structured logging ([#99264](https://github.com/kubernetes/kubernetes/pull/99264), [@palnabarun](https://github.com/palnabarun)) [SIG Auth, Instrumentation and Node] 2546 - Migrate pkg/kubelet/util to structured logging ([#99823](https://github.com/kubernetes/kubernetes/pull/99823), [@navidshaikh](https://github.com/navidshaikh)) [SIG Instrumentation and Node] 2547 - Migrate proxy/userspace/proxier.go logs to structured logging ([#97837](https://github.com/kubernetes/kubernetes/pull/97837), [@JornShen](https://github.com/JornShen)) [SIG Network] 2548 - Migrate some kubelet/metrics log messages to structured logging ([#98627](https://github.com/kubernetes/kubernetes/pull/98627), [@jialaijun](https://github.com/jialaijun)) [SIG Instrumentation and Node] 2549 - Process start time on Windows now uses current process information ([#97491](https://github.com/kubernetes/kubernetes/pull/97491), [@jsturtevant](https://github.com/jsturtevant)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Windows] 2550 2551 ### Uncategorized 2552 2553 - Migrate pkg/kubelet/stats to structured logging ([#99607](https://github.com/kubernetes/kubernetes/pull/99607), [@krzysiekg](https://github.com/krzysiekg)) [SIG Node] 2554 - The DownwardAPIHugePages feature is beta. Users may use the feature if all workers in their cluster are min 1.20 version. The feature will be enabled by default in all installations in 1.22. ([#99610](https://github.com/kubernetes/kubernetes/pull/99610), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node] 2555 2556 ## Dependencies 2557 2558 ### Added 2559 - github.com/go-errors/errors: [v1.0.1](https://github.com/go-errors/errors/tree/v1.0.1) 2560 - github.com/gobuffalo/here: [v0.6.0](https://github.com/gobuffalo/here/tree/v0.6.0) 2561 - github.com/google/shlex: [e7afc7f](https://github.com/google/shlex/tree/e7afc7f) 2562 - github.com/markbates/pkger: [v0.17.1](https://github.com/markbates/pkger/tree/v0.17.1) 2563 - github.com/monochromegane/go-gitignore: [205db1a](https://github.com/monochromegane/go-gitignore/tree/205db1a) 2564 - github.com/niemeyer/pretty: [a10e7ca](https://github.com/niemeyer/pretty/tree/a10e7ca) 2565 - github.com/xlab/treeprint: [a009c39](https://github.com/xlab/treeprint/tree/a009c39) 2566 - go.starlark.net: 8dd3e2e 2567 - golang.org/x/term: 6a3ed07 2568 - sigs.k8s.io/kustomize/api: v0.8.5 2569 - sigs.k8s.io/kustomize/cmd/config: v0.9.7 2570 - sigs.k8s.io/kustomize/kustomize/v4: v4.0.5 2571 - sigs.k8s.io/kustomize/kyaml: v0.10.15 2572 2573 ### Changed 2574 - dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891 2575 - github.com/creack/pty: [v1.1.7 → v1.1.9](https://github.com/creack/pty/compare/v1.1.7...v1.1.9) 2576 - github.com/go-openapi/spec: [v0.19.3 → v0.19.5](https://github.com/go-openapi/spec/compare/v0.19.3...v0.19.5) 2577 - github.com/go-openapi/strfmt: [v0.19.3 → v0.19.5](https://github.com/go-openapi/strfmt/compare/v0.19.3...v0.19.5) 2578 - github.com/go-openapi/validate: [v0.19.5 → v0.19.8](https://github.com/go-openapi/validate/compare/v0.19.5...v0.19.8) 2579 - github.com/google/cadvisor: [v0.38.7 → v0.38.8](https://github.com/google/cadvisor/compare/v0.38.7...v0.38.8) 2580 - github.com/kr/text: [v0.1.0 → v0.2.0](https://github.com/kr/text/compare/v0.1.0...v0.2.0) 2581 - github.com/mattn/go-runewidth: [v0.0.2 → v0.0.7](https://github.com/mattn/go-runewidth/compare/v0.0.2...v0.0.7) 2582 - github.com/olekukonko/tablewriter: [a0225b3 → v0.0.4](https://github.com/olekukonko/tablewriter/compare/a0225b3...v0.0.4) 2583 - github.com/sergi/go-diff: [v1.0.0 → v1.1.0](https://github.com/sergi/go-diff/compare/v1.0.0...v1.1.0) 2584 - golang.org/x/crypto: 7f63de1 → 5ea612d 2585 - golang.org/x/exp: 6cc2880 → 85be41e 2586 - golang.org/x/mobile: d2bd2a2 → e6ae53a 2587 - golang.org/x/mod: v0.3.0 → ce943fd 2588 - golang.org/x/net: 69a7880 → 3d97a24 2589 - golang.org/x/sys: 5cba982 → a50acf3 2590 - golang.org/x/time: 3af7569 → f8bda1e 2591 - golang.org/x/tools: 113979e → v0.1.0 2592 - gopkg.in/check.v1: 41f04d3 → 8fa4692 2593 - gopkg.in/yaml.v2: v2.2.8 → v2.4.0 2594 - k8s.io/kube-openapi: d219536 → 591a79e 2595 - k8s.io/system-validators: v1.3.0 → v1.4.0 2596 2597 ### Removed 2598 - github.com/codegangsta/negroni: [v1.0.0](https://github.com/codegangsta/negroni/tree/v1.0.0) 2599 - github.com/golangplus/bytes: [45c989f](https://github.com/golangplus/bytes/tree/45c989f) 2600 - github.com/golangplus/fmt: [2a5d6d7](https://github.com/golangplus/fmt/tree/2a5d6d7) 2601 - github.com/gorilla/context: [v1.1.1](https://github.com/gorilla/context/tree/v1.1.1) 2602 - github.com/kr/pty: [v1.1.5](https://github.com/kr/pty/tree/v1.1.5) 2603 - sigs.k8s.io/kustomize: v2.0.3+incompatible 2604 2605 2606 2607 # v1.21.0-beta.0 2608 2609 2610 ## Downloads for v1.21.0-beta.0 2611 2612 ### Source Code 2613 2614 filename | sha512 hash 2615 -------- | ----------- 2616 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes.tar.gz) | 69b73a03b70b0ed006e9fef3f5b9bc68f0eb8dc40db6cc04777c03a2cb83a008c783012ca186b1c48357fb192403dbcf6960f120924785e2076e215b9012d546 2617 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-src.tar.gz) | 9620fb6d37634271bdd423c09f33f3bd29e74298aa82c47dffc8cb6bd2ff44fa8987a53c53bc529db4ca96ec41503aa81cc8d0c3ac106f3b06c4720de933a8e6 2618 2619 ### Client binaries 2620 2621 filename | sha512 hash 2622 -------- | ----------- 2623 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-darwin-amd64.tar.gz) | 2a6f3fcd6b571f5ccde56b91e6e179a01899244be496dae16a2a16e0405c9437b75c6dc853b56f9a4876a7c0a60ec624ccd28400bf8fb960258263172f6860ba 2624 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-386.tar.gz) | 78fe9ad9f9a9bc043293327223f0038a2c087ca65e87187a6dcae7a24aef9565fe498d295a4639b0b90524469a04930022fcecd815d0afc742eb87ddd8eb7ef5 2625 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-amd64.tar.gz) | c025f5e5bd132355e7dd1296cf2ec752264e7f754c4d95fc34b076bd75bef2f571d30872bcb3d138ce95c592111353d275a80eb31f82c07000874b4c56282dbd 2626 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-arm.tar.gz) | 9975cd2f08fbc202575fb15ba6fc51dab23155ca4d294ebb48516a81efa51f58bab3a87d41c865103756189b554c020371d729ad42880ba788f25047ffc46910 2627 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-arm64.tar.gz) | 56a6836e24471e42e9d9a8488453f2d55598d70c8aca0a307d5116139c930c25c469fd0d1ab5060fbe88dad75a9b5209a08dc11d644af5f3ebebfbcb6c16266c 2628 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-ppc64le.tar.gz) | b6a6cc9baad0ad85ed079ee80e6d6acc905095cfb440998bbc0f553b94fa80077bd58b8692754de477517663d51161705e6e89a1b6d04aa74819800db3517722 2629 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-s390x.tar.gz) | 7b743481b340f510bf9ae28ea8ea91150aa1e8c37fe104b66d7b3aff62f5e6db3c590d2c13d14dbb5c928de31c7613372def2496075853611d10d6b5fa5b60bd 2630 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-windows-386.tar.gz) | df06c7a524ce84c1f8d7836aa960c550c88dbca0ec4854df4dd0a85b3c84b8ecbc41b54e8c4669ce28ac670659ff0fad795deb1bc539f3c3b3aa885381265f5a 2631 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-windows-amd64.tar.gz) | 4568497b684564f2a94fbea6cbfd778b891231470d9a6956c3b7a3268643d13b855c0fc5ebea5f769300cc0c7719c2c331c387f468816f182f63e515adeaa7a0 2632 2633 ### Server binaries 2634 2635 filename | sha512 hash 2636 -------- | ----------- 2637 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-amd64.tar.gz) | 42883cca2d312153baf693fc6024a295359a421e74fd70eefc927413be4e0353debe634e7cca6b9a8f7d8a0cee3717e03ba5d29a306e93139b1c2f3027535a6d 2638 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-arm.tar.gz) | e0042215e84c769ba4fc4d159ccf67b2c4a26206bfffb0ec5152723dc813ff9c1426aa0e9b963d7bfa2efb266ca43561b596b459152882ebb42102ccf60bd8eb 2639 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-arm64.tar.gz) | bfad29d43e14152cb9bc7c4df6aa77929c6eca64a294bb832215bdba9fa0ee2195a2b709c0267dc7426bb371b547ee80bb8461a8c678c9bffa0819aa7db96289 2640 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-ppc64le.tar.gz) | ca67674c01c6cebdc8160c85b449eab1a23bb0557418665246e0208543fa2eaaf97679685c7b49bee3a4300904c0399c3d762ae34dc3e279fd69ce792c4b07ff 2641 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-s390x.tar.gz) | 285352b628ec754b01b8ad4ef1427223a142d58ebcb46f6861df14d68643133b32330460b213b1ba5bc5362ff2b6dacd8e0c2d20cce6e760fa1954af8a60df8b 2642 2643 ### Node binaries 2644 2645 filename | sha512 hash 2646 -------- | ----------- 2647 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-amd64.tar.gz) | d92d9b30e7e44134a0cd9db4c01924d365991ea16b3131200b02a82cff89c8701f618cd90e7f1c65427bd4bb5f78b10d540b2262de2c143b401fa44e5b25627b 2648 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-arm.tar.gz) | 551092f23c27fdea4bb2d0547f6075892534892a96fc2be7786f82b58c93bffdb5e1c20f8f11beb8bed46c24f36d4c18ec5ac9755435489efa28e6ae775739bd 2649 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-arm64.tar.gz) | 26ae7f4163e527349b8818ee38b9ee062314ab417f307afa49c146df8f5a2bd689509b128bd4a1efd3896fd89571149a9955ada91f8ca0c2f599cd863d613c86 2650 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-ppc64le.tar.gz) | 821fa953f6cebc69d2d481e489f3e90899813d20e2eefbabbcadd019d004108e7540f741fabe60e8e7c6adbb1053ac97898bbdddec3ca19f34a71aa3312e0d4e 2651 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-s390x.tar.gz) | 22197d4f66205d5aa9de83dfddcc4f2bb3195fd7067cdb5c21e61dbeae217bc112fb7ecff8a539579b60ad92298c2b4c87b9b7c7e6ec1ee1ffa0c6e4bc4412c1 2652 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-windows-amd64.tar.gz) | 7e22e0d9603562a04dee16a513579f06b1ff6354d97d669bd68f8777ec7f89f6ef027fb23ab0445d7bba0bb689352f0cc748ce90e3f597c6ebe495464a96b860 2653 2654 ## Changelog since v1.21.0-alpha.3 2655 2656 ## Urgent Upgrade Notes 2657 2658 ### (No, really, you MUST read this before you upgrade) 2659 2660 - The metric `storage_operation_errors_total` is not removed, but is marked deprecated, and the metric `storage_operation_status_count` is marked deprecated. In both cases the storage_operation_duration_seconds metric can be used to recover equivalent counts (using `status=fail-unknown` in the case of `storage_operations_errors_total`). ([#99045](https://github.com/kubernetes/kubernetes/pull/99045), [@mattcary](https://github.com/mattcary)) [SIG Instrumentation and Storage] 2661 2662 ## Changes by Kind 2663 2664 ### Deprecation 2665 2666 - The `batch/v2alpha1` CronJob type definitions and clients are deprecated and removed. ([#96987](https://github.com/kubernetes/kubernetes/pull/96987), [@soltysh](https://github.com/soltysh)) [SIG API Machinery, Apps, CLI and Testing] 2667 2668 ### API Change 2669 2670 - Cluster admins can now turn off /debug/pprof and /debug/flags/v endpoint in kubelet by setting enableProfilingHandler and enableDebugFlagsHandler to false in their kubelet configuration file. enableProfilingHandler and enableDebugFlagsHandler can be set to true only when enableDebuggingHandlers is also set to true. ([#98458](https://github.com/kubernetes/kubernetes/pull/98458), [@SaranBalaji90](https://github.com/SaranBalaji90)) [SIG Node] 2671 - The BoundServiceAccountTokenVolume feature has been promoted to beta, and enabled by default. 2672 - This changes the tokens provided to containers at `/var/run/secrets/kubernetes.io/serviceaccount/token` to be time-limited, auto-refreshed, and invalidated when the containing pod is deleted. 2673 - Clients should reload the token from disk periodically (once per minute is recommended) to ensure they continue to use a valid token. `k8s.io/client-go` version v11.0.0+ and v0.15.0+ reload tokens automatically. 2674 - By default, injected tokens are given an extended lifetime so they remain valid even after a new refreshed token is provided. The metric `serviceaccount_stale_tokens_total` can be used to monitor for workloads that are depending on the extended lifetime and are continuing to use tokens even after a refreshed token is provided to the container. If that metric indicates no existing workloads are depending on extended lifetimes, injected token lifetime can be shortened to 1 hour by starting `kube-apiserver` with `--service-account-extend-token-expiration=false`. ([#95667](https://github.com/kubernetes/kubernetes/pull/95667), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth, Cluster Lifecycle and Testing] 2675 2676 ### Feature 2677 2678 - A new histogram metric to track the time it took to delete a job by the ttl-after-finished controller ([#98676](https://github.com/kubernetes/kubernetes/pull/98676), [@ahg-g](https://github.com/ahg-g)) [SIG Apps and Instrumentation] 2679 - AWS cloudprovider supports auto-discovering subnets without any kubernetes.io/cluster/<clusterName> tags. It also supports additional service annotation service.beta.kubernetes.io/aws-load-balancer-subnets to manually configure the subnets. ([#97431](https://github.com/kubernetes/kubernetes/pull/97431), [@kishorj](https://github.com/kishorj)) [SIG Cloud Provider] 2680 - Add `--permit-address-sharing` flag to kube-apiserver to listen with SO_REUSEADDR. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoid waiting for the kernel to release socket in TIME_WAIT state, and hence, considably reducing kube-apiserver restart times under certain conditions. ([#93861](https://github.com/kubernetes/kubernetes/pull/93861), [@sttts](https://github.com/sttts)) [SIG API Machinery] 2681 - Add `csi_operations_seconds` metric on kubelet that exposes CSI operations duration and status for node CSI operations. ([#98979](https://github.com/kubernetes/kubernetes/pull/98979), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Instrumentation and Storage] 2682 - Add `migrated` field into `storage_operation_duration_seconds` metric ([#99050](https://github.com/kubernetes/kubernetes/pull/99050), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps, Instrumentation and Storage] 2683 - Add bash-completion for comma separated list on `kubectl get` ([#98301](https://github.com/kubernetes/kubernetes/pull/98301), [@phil9909](https://github.com/phil9909)) [SIG CLI] 2684 - Added support for installing arm64 node artifacts. ([#99242](https://github.com/kubernetes/kubernetes/pull/99242), [@liu-cong](https://github.com/liu-cong)) [SIG Cloud Provider] 2685 - Feature gate RootCAConfigMap is graduated to GA in 1.21 and will be removed in 1.22. ([#98033](https://github.com/kubernetes/kubernetes/pull/98033), [@zshihang](https://github.com/zshihang)) [SIG API Machinery and Auth] 2686 - Kubeadm: during "init" and "join" perform preflight validation on the host / node name and throw warnings if a name is not compliant ([#99194](https://github.com/kubernetes/kubernetes/pull/99194), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 2687 - Kubectl: `kubectl get` will omit managed fields by default now. Users could set `--show-managed-fields` to true to show managedFields when the output format is either `json` or `yaml`. ([#96878](https://github.com/kubernetes/kubernetes/pull/96878), [@knight42](https://github.com/knight42)) [SIG CLI and Testing] 2688 - Metrics can now be disabled explicitly via a command line flag (i.e. '--disabled-metrics=bad_metric1,bad_metric2') ([#99217](https://github.com/kubernetes/kubernetes/pull/99217), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Cluster Lifecycle and Instrumentation] 2689 - TTLAfterFinished is now beta and enabled by default ([#98678](https://github.com/kubernetes/kubernetes/pull/98678), [@ahg-g](https://github.com/ahg-g)) [SIG Apps and Auth] 2690 - The `RunAsGroup` feature has been promoted to GA in this release. ([#94641](https://github.com/kubernetes/kubernetes/pull/94641), [@krmayankk](https://github.com/krmayankk)) [SIG Auth and Node] 2691 - Turn CronJobControllerV2 on by default. ([#98878](https://github.com/kubernetes/kubernetes/pull/98878), [@soltysh](https://github.com/soltysh)) [SIG Apps] 2692 - UDP protocol support for Agnhost connect subcommand ([#98639](https://github.com/kubernetes/kubernetes/pull/98639), [@knabben](https://github.com/knabben)) [SIG Testing] 2693 - Upgrades `IPv6Dualstack` to `Beta` and turns it on by default. Clusters new and existing will not be affected until user starting adding secondary pod and service cidrs cli flags as described here: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack ([#98969](https://github.com/kubernetes/kubernetes/pull/98969), [@khenidak](https://github.com/khenidak)) [SIG API Machinery, Apps, Cloud Provider, Network and Node] 2694 2695 ### Documentation 2696 2697 - Fix ALPHA stability level reference link ([#98641](https://github.com/kubernetes/kubernetes/pull/98641), [@Jeffwan](https://github.com/Jeffwan)) [SIG Auth, Cloud Provider, Instrumentation and Storage] 2698 2699 ### Failing Test 2700 2701 - Escape the special characters like `[`, `]` and ` ` that exist in vsphere windows path ([#98830](https://github.com/kubernetes/kubernetes/pull/98830), [@liyanhui1228](https://github.com/liyanhui1228)) [SIG Storage and Windows] 2702 - Kube-proxy: fix a bug on UDP NodePort Services where stale conntrack entries may blackhole the traffic directed to the NodePort. ([#98305](https://github.com/kubernetes/kubernetes/pull/98305), [@aojea](https://github.com/aojea)) [SIG Network] 2703 2704 ### Bug or Regression 2705 2706 - Add missing `--kube-api-content-type` in kubemark hollow template ([#98911](https://github.com/kubernetes/kubernetes/pull/98911), [@Jeffwan](https://github.com/Jeffwan)) [SIG Scalability and Testing] 2707 - Avoid duplicate error messages when runing kubectl edit quota ([#98201](https://github.com/kubernetes/kubernetes/pull/98201), [@pacoxu](https://github.com/pacoxu)) [SIG API Machinery and Apps] 2708 - Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. ([#98133](https://github.com/kubernetes/kubernetes/pull/98133), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 2709 - Fix errors when accessing Windows container stats for Dockershim ([#98510](https://github.com/kubernetes/kubernetes/pull/98510), [@jsturtevant](https://github.com/jsturtevant)) [SIG Node and Windows] 2710 - Fixes spurious errors about IPv6 in kube-proxy logs on nodes with IPv6 disabled. ([#99127](https://github.com/kubernetes/kubernetes/pull/99127), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] 2711 - In the method that ensures that the docker and containerd are in the correct containers with the proper OOM score set up, fixed the bug of identifying containerd process. ([#97888](https://github.com/kubernetes/kubernetes/pull/97888), [@pacoxu](https://github.com/pacoxu)) [SIG Node] 2712 - Kubelet now cleans up orphaned volume directories automatically ([#95301](https://github.com/kubernetes/kubernetes/pull/95301), [@lorenz](https://github.com/lorenz)) [SIG Node and Storage] 2713 - When dynamically provisioning Azure File volumes for a premium account, the requested size will be set to 100GB if the request is initially lower than this value to accommodate Azure File requirements. ([#99122](https://github.com/kubernetes/kubernetes/pull/99122), [@huffmanca](https://github.com/huffmanca)) [SIG Cloud Provider and Storage] 2714 2715 ### Other (Cleanup or Flake) 2716 2717 - APIs for kubelet annotations and labels from k8s.io/kubernetes/pkg/kubelet/apis are now available under k8s.io/kubelet/pkg/apis/ ([#98931](https://github.com/kubernetes/kubernetes/pull/98931), [@michaelbeaumont](https://github.com/michaelbeaumont)) [SIG Apps, Auth and Node] 2718 - Migrate `pkg/kubelet/(pod, pleg)` to structured logging ([#98990](https://github.com/kubernetes/kubernetes/pull/98990), [@gjkim42](https://github.com/gjkim42)) [SIG Instrumentation and Node] 2719 - Migrate pkg/kubelet/nodestatus to structured logging ([#99001](https://github.com/kubernetes/kubernetes/pull/99001), [@QiWang19](https://github.com/QiWang19)) [SIG Node] 2720 - Migrate pkg/kubelet/server logs to structured logging ([#98643](https://github.com/kubernetes/kubernetes/pull/98643), [@chenyw1990](https://github.com/chenyw1990)) [SIG Node] 2721 - Migrate proxy/winkernel/proxier.go logs to structured logging ([#98001](https://github.com/kubernetes/kubernetes/pull/98001), [@JornShen](https://github.com/JornShen)) [SIG Network and Windows] 2722 - Migrate scheduling_queue.go to structured logging ([#98358](https://github.com/kubernetes/kubernetes/pull/98358), [@tanjing2020](https://github.com/tanjing2020)) [SIG Scheduling] 2723 - Several flags related to the deprecated dockershim which are present in the kubelet command line are now deprecated. ([#98730](https://github.com/kubernetes/kubernetes/pull/98730), [@dims](https://github.com/dims)) [SIG Node] 2724 - The deprecated feature gates `CSIDriverRegistry`, `BlockVolume` and `CSIBlockVolume` are now unconditionally enabled and can no longer be specified in component invocations. ([#98021](https://github.com/kubernetes/kubernetes/pull/98021), [@gavinfish](https://github.com/gavinfish)) [SIG Storage] 2725 2726 ## Dependencies 2727 2728 ### Added 2729 _Nothing has changed._ 2730 2731 ### Changed 2732 - sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.0.3 2733 2734 ### Removed 2735 _Nothing has changed._ 2736 2737 2738 2739 # v1.21.0-alpha.3 2740 2741 2742 ## Downloads for v1.21.0-alpha.3 2743 2744 ### Source Code 2745 2746 filename | sha512 hash 2747 -------- | ----------- 2748 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes.tar.gz) | 704ec916a1dbd134c54184d2652671f80ae09274f9d23dbbed312944ebeccbc173e2e6b6949b38bdbbfdaf8aa032844deead5efeda1b3150f9751386d9184bc8 2749 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-src.tar.gz) | 57db9e7560cfc9c10e7059cb5faf9c4bd5eb8f9b7964f44f000a417021cf80873184b774e7c66c80d4aba84c14080c6bc335618db3d2e5f276436ae065e25408 2750 2751 ### Client binaries 2752 2753 filename | sha512 hash 2754 -------- | ----------- 2755 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-darwin-amd64.tar.gz) | e2706efda92d5cf4f8b69503bb2f7703a8754407eff7f199bb77847838070e720e5f572126c14daa4c0c03b59bb1a63c1dfdeb6e936a40eff1d5497e871e3409 2756 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-386.tar.gz) | 007bb23c576356ed0890bdfd25a0f98d552599e0ffec19fb982591183c7c1f216d8a3ffa3abf15216be12ae5c4b91fdcd48a7306a2d26b007b86a6abd553fc61 2757 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-amd64.tar.gz) | 39504b0c610348beba60e8866fff265bad58034f74504951cd894c151a248db718d10f77ebc83f2c38b2d517f8513a46325b38889eefa261ca6dbffeceba50ff 2758 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-arm.tar.gz) | 30bc2c40d0c759365422ad1651a6fb35909be771f463c5b971caf401f9209525d05256ab70c807e88628dd357c2896745eecf13eda0b748464da97d0a5ef2066 2759 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-arm64.tar.gz) | 085cdf574dc8fd33ece667130b8c45830b522a07860e03a2384283b1adea73a9652ef3dfaa566e69ee00aea1a6461608814b3ce7a3f703e4a934304f7ae12f97 2760 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-ppc64le.tar.gz) | b34b845037d83ea7b3e2d80a9ede4f889b71b17b93b1445f0d936a36e98c13ed6ada125630a68d9243a5fcd311ee37cdcc0c05da484da8488ea5060bc529dbfc 2761 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-s390x.tar.gz) | c4758adc7a404b776556efaa79655db2a70777c562145d6ea6887f3335988367a0c2fcd4383e469340f2a768b22e786951de212805ca1cb91104d41c21e0c9ce 2762 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-windows-386.tar.gz) | f51edc79702bbd1d9cb3a672852a405e11b20feeab64c5411a7e85c9af304960663eb6b23ef96e0f8c44a722fecf58cb6d700ea2c42c05b3269d8efd5ad803f2 2763 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-windows-amd64.tar.gz) | 6a3507ce4ac40a0dc7e4720538863fa15f8faf025085a032f34b8fa0f6fa4e8c26849baf649b5b32829b9182e04f82721b13950d31cf218c35be6bf1c05d6abf 2764 2765 ### Server binaries 2766 2767 filename | sha512 hash 2768 -------- | ----------- 2769 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-amd64.tar.gz) | 19181d162dfb0b30236e2bf1111000e037eece87c037ca2b24622ca94cb88db86aa4da4ca533522518b209bc9983bbfd6b880a7898e0da96b33f3f6c4690539b 2770 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-arm.tar.gz) | 42a02f9e08a78ad5da6e5fa1ab12bf1e3c967c472fdbdadbd8746586da74dc8093682ba8513ff2a5301393c47ee9021b860e88ada56b13da386ef485708e46ca 2771 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-arm64.tar.gz) | 3c8ba8eb02f70061689bd7fab7813542005efe2edc6cfc6b7aecd03ffedf0b81819ad91d69fff588e83023d595eefbfe636aa55e1856add8733bf42fff3c748f 2772 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-ppc64le.tar.gz) | cd9e6537450411c39a06fd0b5819db3d16b668d403fb3627ec32c0e32dd1c4860e942934578ca0e1d1b8e6f21f450ff81e37e0cd46ff5c5faf7847ab074aefc5 2773 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-s390x.tar.gz) | ada3f65e53bc0e0c0229694dd48c425388089d6d77111a62476d1b08f6ad1d8ab3d60b9ed7d95ac1b42c2c6be8dc0618f40679717160769743c43583d8452362 2774 2775 ### Node binaries 2776 2777 filename | sha512 hash 2778 -------- | ----------- 2779 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-amd64.tar.gz) | ae0fec6aa59e49624b55d9a11c12fdf717ddfe04bdfd4f69965d03004a34e52ee4a3e83f7b61d0c6a86f43b72c99f3decb195b39ae529ef30526d18ec5f58f83 2780 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-arm.tar.gz) | 9a48c140ab53b7ed8ecec6903988a1a474efc16d2538e5974bc9a12f0c9190be78c4f9e326bf4e982d0b7045a80b99dd0fda7e9b650663be5b89bfd991596746 2781 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-arm64.tar.gz) | 6912adbc9300344bea470d6435f7b387bfce59767078c11728ce59faf47cd3f72b41b9604fcc5cda45e9816fe939fbe2fb33e52a773e6ff2dfa9a615b4df6141 2782 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-ppc64le.tar.gz) | d66dccfe3e6ed6d81567c70703f15375a53992b3a5e2814b98c32e581b861ad95912e03ed2562415d087624c008038bb4a816611fa255442ae752968ea15856b 2783 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-s390x.tar.gz) | ad8c69a28f1fbafa3f1cb54909bfd3fc22b104bed63d7ca2b296208c9d43eb5f2943a0ff267da4c185186cdd9f7f77b315cd7f5f1bf9858c0bf42eceb9ac3c58 2784 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-windows-amd64.tar.gz) | 91d723aa848a9cb028f5bcb41090ca346fb973961521d025c4399164de2c8029b57ca2c4daca560d3c782c05265d2eb0edb0abcce6f23d3efbecf2316a54d650 2785 2786 ## Changelog since v1.21.0-alpha.2 2787 2788 ## Urgent Upgrade Notes 2789 2790 ### (No, really, you MUST read this before you upgrade) 2791 2792 - Newly provisioned PVs by gce-pd will no longer have the beta FailureDomain label. gce-pd volume plugin will start to have GA topology label instead. ([#98700](https://github.com/kubernetes/kubernetes/pull/98700), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Cloud Provider, Storage and Testing] 2793 - Remove alpha CSIMigrationXXComplete flag and add alpha InTreePluginXXUnregister flag. Deprecate CSIMigrationvSphereComplete flag and it will be removed in 1.22. ([#98243](https://github.com/kubernetes/kubernetes/pull/98243), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Node and Storage] 2794 2795 ## Changes by Kind 2796 2797 ### API Change 2798 2799 - Adds support for portRange / EndPort in Network Policy ([#97058](https://github.com/kubernetes/kubernetes/pull/97058), [@rikatz](https://github.com/rikatz)) [SIG Apps and Network] 2800 - Fixes using server-side apply with APIService resources ([#98576](https://github.com/kubernetes/kubernetes/pull/98576), [@kevindelgado](https://github.com/kevindelgado)) [SIG API Machinery, Apps and Testing] 2801 - Kubernetes is now built using go1.15.7 ([#98363](https://github.com/kubernetes/kubernetes/pull/98363), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Node, Release and Testing] 2802 - Scheduler extender filter interface now can report unresolvable failed nodes in the new field `FailedAndUnresolvableNodes` of `ExtenderFilterResult` struct. Nodes in this map will be skipped in the preemption phase. ([#92866](https://github.com/kubernetes/kubernetes/pull/92866), [@cofyc](https://github.com/cofyc)) [SIG Scheduling] 2803 2804 ### Feature 2805 2806 - A lease can only attach up to 1k objects. ([#98257](https://github.com/kubernetes/kubernetes/pull/98257), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery] 2807 - Add ignore-errors flag for drain, support none-break drain in group ([#98203](https://github.com/kubernetes/kubernetes/pull/98203), [@yuzhiquan](https://github.com/yuzhiquan)) [SIG CLI] 2808 - Base-images: Update to debian-iptables:buster-v1.4.0 2809 - Uses iptables 1.8.5 2810 - base-images: Update to debian-base:buster-v1.3.0 2811 - cluster/images/etcd: Build etcd:3.4.13-2 image 2812 - Uses debian-base:buster-v1.3.0 ([#98401](https://github.com/kubernetes/kubernetes/pull/98401), [@pacoxu](https://github.com/pacoxu)) [SIG Testing] 2813 - Export NewDebuggingRoundTripper function and DebugLevel options in the k8s.io/client-go/transport package. ([#98324](https://github.com/kubernetes/kubernetes/pull/98324), [@atosatto](https://github.com/atosatto)) [SIG API Machinery] 2814 - Kubectl wait ensures that observedGeneration >= generation if applicable ([#97408](https://github.com/kubernetes/kubernetes/pull/97408), [@KnicKnic](https://github.com/KnicKnic)) [SIG CLI] 2815 - Kubernetes is now built using go1.15.8 ([#98834](https://github.com/kubernetes/kubernetes/pull/98834), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] 2816 - New admission controller "denyserviceexternalips" is available. Clusters which do not *need* the Service "externalIPs" feature should enable this controller and be more secure. ([#97395](https://github.com/kubernetes/kubernetes/pull/97395), [@thockin](https://github.com/thockin)) [SIG API Machinery] 2817 - Overall, enable the feature of `PreferNominatedNode` will improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of `PreferNominatedNode`, the pod might not be scheduled to the best candidate node in the cluster. ([#93179](https://github.com/kubernetes/kubernetes/pull/93179), [@chendave](https://github.com/chendave)) [SIG Scheduling and Testing] 2818 - Pause image upgraded to 3.4.1 in kubelet and kubeadm for both Linux and Windows. ([#98205](https://github.com/kubernetes/kubernetes/pull/98205), [@pacoxu](https://github.com/pacoxu)) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Testing and Windows] 2819 - The `ServiceAccountIssuerDiscovery` feature has graduated to GA, and is unconditionally enabled. The `ServiceAccountIssuerDiscovery` feature-gate will be removed in 1.22. ([#98553](https://github.com/kubernetes/kubernetes/pull/98553), [@mtaufen](https://github.com/mtaufen)) [SIG API Machinery, Auth and Testing] 2820 2821 ### Documentation 2822 2823 - Feat: azure file migration go beta in 1.21. Feature gates CSIMigration to Beta (on by default) and CSIMigrationAzureFile to Beta (off by default since it requires installation of the AzureFile CSI Driver) 2824 The in-tree AzureFile plugin "kubernetes.io/azure-file" is now deprecated and will be removed in 1.23. Users should enable CSIMigration + CSIMigrationAzureFile features and install the AzureFile CSI Driver (https://github.com/kubernetes-sigs/azurefile-csi-driver) to avoid disruption to existing Pod and PVC objects at that time. 2825 Users should start using the AzureFile CSI Driver directly for any new volumes. ([#96293](https://github.com/kubernetes/kubernetes/pull/96293), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] 2826 2827 ### Failing Test 2828 2829 - Kubelet: the HostPort implementation in dockershim was not taking into consideration the HostIP field, causing that the same HostPort can not be used with different IP addresses. 2830 This bug causes the conformance test "HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol" to fail. ([#98755](https://github.com/kubernetes/kubernetes/pull/98755), [@aojea](https://github.com/aojea)) [SIG Cloud Provider, Network and Node] 2831 2832 ### Bug or Regression 2833 2834 - Fix NPE in ephemeral storage eviction ([#98261](https://github.com/kubernetes/kubernetes/pull/98261), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 2835 - Fixed a bug that on k8s nodes, when the policy of INPUT chain in filter table is not ACCEPT, healthcheck nodeport would not work. 2836 Added iptables rules to allow healthcheck nodeport traffic. ([#97824](https://github.com/kubernetes/kubernetes/pull/97824), [@hanlins](https://github.com/hanlins)) [SIG Network] 2837 - Fixed kube-proxy container image architecture for non amd64 images. ([#98526](https://github.com/kubernetes/kubernetes/pull/98526), [@saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Release and Testing] 2838 - Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. ([#98311](https://github.com/kubernetes/kubernetes/pull/98311), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] 2839 - Fixes a panic in the disruption budget controller for PDB objects with invalid selectors ([#98750](https://github.com/kubernetes/kubernetes/pull/98750), [@mortent](https://github.com/mortent)) [SIG Apps] 2840 - Fixes connection errors when using `--volume-host-cidr-denylist` or `--volume-host-allow-local-loopback` ([#98436](https://github.com/kubernetes/kubernetes/pull/98436), [@liggitt](https://github.com/liggitt)) [SIG Network and Storage] 2841 - If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400. 2842 - in cases where the client specifies a timeout in the request URL, the overall request deadline is shortened now since the deadline is setup as soon as the request is received by the apiserver. ([#96901](https://github.com/kubernetes/kubernetes/pull/96901), [@tkashem](https://github.com/tkashem)) [SIG API Machinery and Testing] 2843 - Kubeadm: Some text in the `kubeadm upgrade plan` output has changed. If you have scripts or other automation that parses this output, please review these changes and update your scripts to account for the new output. ([#98728](https://github.com/kubernetes/kubernetes/pull/98728), [@stmcginnis](https://github.com/stmcginnis)) [SIG Cluster Lifecycle] 2844 - Kubeadm: fix a bug where external credentials in an existing admin.conf prevented the CA certificate to be written in the cluster-info ConfigMap. ([#98882](https://github.com/kubernetes/kubernetes/pull/98882), [@kvaps](https://github.com/kvaps)) [SIG Cluster Lifecycle] 2845 - Kubeadm: fix bad token placeholder text in "config print *-defaults --help" ([#98839](https://github.com/kubernetes/kubernetes/pull/98839), [@Mattias-](https://github.com/Mattias-)) [SIG Cluster Lifecycle] 2846 - Kubeadm: get k8s CI version markers from k8s infra bucket ([#98836](https://github.com/kubernetes/kubernetes/pull/98836), [@hasheddan](https://github.com/hasheddan)) [SIG Cluster Lifecycle and Release] 2847 - Mitigate CVE-2020-8555 for kube-up using GCE by preventing local loopback folume hosts. ([#97934](https://github.com/kubernetes/kubernetes/pull/97934), [@mattcary](https://github.com/mattcary)) [SIG Cloud Provider and Storage] 2848 - Remove CSI topology from migrated in-tree gcepd volume. ([#97823](https://github.com/kubernetes/kubernetes/pull/97823), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Cloud Provider and Storage] 2849 - Sync node status during kubelet node shutdown. 2850 Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. ([#98005](https://github.com/kubernetes/kubernetes/pull/98005), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 2851 - Truncates a message if it hits the NoteLengthLimit when the scheduler records an event for the pod that indicates the pod has failed to schedule. ([#98715](https://github.com/kubernetes/kubernetes/pull/98715), [@carlory](https://github.com/carlory)) [SIG Scheduling] 2852 - We will no longer automatically delete all data when a failure is detected during creation of the volume data file on a CSI volume. Now we will only remove the data file and volume path. ([#96021](https://github.com/kubernetes/kubernetes/pull/96021), [@huffmanca](https://github.com/huffmanca)) [SIG Storage] 2853 2854 ### Other (Cleanup or Flake) 2855 2856 - Fix the description of command line flags that can override --config ([#98254](https://github.com/kubernetes/kubernetes/pull/98254), [@changshuchao](https://github.com/changshuchao)) [SIG Scheduling] 2857 - Migrate scheduler/taint_manager.go structured logging ([#98259](https://github.com/kubernetes/kubernetes/pull/98259), [@tanjing2020](https://github.com/tanjing2020)) [SIG Apps] 2858 - Migrate staging/src/k8s.io/apiserver/pkg/admission logs to structured logging ([#98138](https://github.com/kubernetes/kubernetes/pull/98138), [@lala123912](https://github.com/lala123912)) [SIG API Machinery] 2859 - Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object ([#98430](https://github.com/kubernetes/kubernetes/pull/98430), [@liggitt](https://github.com/liggitt)) [SIG Network and Testing] 2860 - The default delegating authorization options now allow unauthenticated access to healthz, readyz, and livez. A system:masters user connecting to an authz delegator will not perform an authz check. ([#98325](https://github.com/kubernetes/kubernetes/pull/98325), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Auth, Cloud Provider and Scheduling] 2861 - The e2e suite can be instructed not to wait for pods in kube-system to be ready or for all nodes to be ready by passing `--allowed-not-ready-nodes=-1` when invoking the e2e.test program. This allows callers to run subsets of the e2e suite in scenarios other than perfectly healthy clusters. ([#98781](https://github.com/kubernetes/kubernetes/pull/98781), [@smarterclayton](https://github.com/smarterclayton)) [SIG Testing] 2862 - The feature gates `WindowsGMSA` and `WindowsRunAsUserName` that are GA since v1.18 are now removed. ([#96531](https://github.com/kubernetes/kubernetes/pull/96531), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Node and Windows] 2863 - The new `-gce-zones` flag on the `e2e.test` binary instructs tests that check for information about how the cluster interacts with the cloud to limit their queries to the provided zone list. If not specified, the current behavior of asking the cloud provider for all available zones in multi zone clusters is preserved. ([#98787](https://github.com/kubernetes/kubernetes/pull/98787), [@smarterclayton](https://github.com/smarterclayton)) [SIG API Machinery, Cluster Lifecycle and Testing] 2864 2865 ## Dependencies 2866 2867 ### Added 2868 - github.com/moby/spdystream: [v0.2.0](https://github.com/moby/spdystream/tree/v0.2.0) 2869 2870 ### Changed 2871 - github.com/NYTimes/gziphandler: [56545f4 → v1.1.1](https://github.com/NYTimes/gziphandler/compare/56545f4...v1.1.1) 2872 - github.com/container-storage-interface/spec: [v1.2.0 → v1.3.0](https://github.com/container-storage-interface/spec/compare/v1.2.0...v1.3.0) 2873 - github.com/go-logr/logr: [v0.2.0 → v0.4.0](https://github.com/go-logr/logr/compare/v0.2.0...v0.4.0) 2874 - github.com/gogo/protobuf: [v1.3.1 → v1.3.2](https://github.com/gogo/protobuf/compare/v1.3.1...v1.3.2) 2875 - github.com/kisielk/errcheck: [v1.2.0 → v1.5.0](https://github.com/kisielk/errcheck/compare/v1.2.0...v1.5.0) 2876 - github.com/yuin/goldmark: [v1.1.27 → v1.2.1](https://github.com/yuin/goldmark/compare/v1.1.27...v1.2.1) 2877 - golang.org/x/sync: cd5d95a → 67f06af 2878 - golang.org/x/tools: c1934b7 → 113979e 2879 - k8s.io/klog/v2: v2.4.0 → v2.5.0 2880 - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15 2881 2882 ### Removed 2883 - github.com/docker/spdystream: [449fdfc](https://github.com/docker/spdystream/tree/449fdfc) 2884 2885 2886 2887 # v1.21.0-alpha.2 2888 2889 2890 ## Downloads for v1.21.0-alpha.2 2891 2892 ### Source Code 2893 2894 filename | sha512 hash 2895 -------- | ----------- 2896 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes.tar.gz) | 6836f6c8514253fe0831fd171fc4ed92eb6d9a773491c8dc82b90d171a1b10076bd6bfaea56ec1e199c5f46c273265bdb9f174f0b2d99c5af1de4c99b862329e 2897 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-src.tar.gz) | d137694804741a05ab09e5f9a418448b66aba0146c028eafce61bcd9d7c276521e345ce9223ffbc703e8172041d58dfc56a3242a4df3686f24905a4541fcd306 2898 2899 ### Client binaries 2900 2901 filename | sha512 hash 2902 -------- | ----------- 2903 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-darwin-amd64.tar.gz) | 9478b047a97717953f365c13a098feb7e3cb30a3df22e1b82aa945f2208dcc5cb90afc441ba059a3ae7aafb4ee000ec3a52dc65a8c043a5ac7255a391c875330 2904 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-386.tar.gz) | 44c8dd4b1ddfc256d35786c8abf45b0eb5f0794f5e310d2efc865748adddc50e8bf38aa71295ae8a82884cb65f2e0b9b0737b000f96fd8f2d5c19971d7c4d8e8 2905 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-amd64.tar.gz) | e1291989892769de6b978c17b8612b94da6f3b735a4d895100af622ca9ebb968c75548afea7ab00445869625dd0da3afec979e333afbb445805f5d31c1c13cc7 2906 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-arm.tar.gz) | 3c4bcb8cbe73822d68a2f62553a364e20bec56b638c71d0f58679b4f4b277d809142346f18506914e694f6122a3e0f767eab20b7b1c4dbb79e4c5089981ae0f1 2907 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-arm64.tar.gz) | 9389974a790268522e187f5ba5237f3ee4684118c7db76bc3d4164de71d8208702747ec333b204c7a78073ab42553cbbce13a1883fab4fec617e093b05fab332 2908 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-ppc64le.tar.gz) | 63399e53a083b5af3816c28ff162c9de6b64c75da4647f0d6bbaf97afdf896823cb1e556f2abac75c6516072293026d3ff9f30676fd75143ac6ca3f4d21f4327 2909 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-s390x.tar.gz) | 50898f197a9d923971ff9046c9f02779b57f7b3cea7da02f3ea9bab8c08d65a9c4a7531a2470fa14783460f52111a52b96ebf916c0a1d8215b4070e4e861c1b0 2910 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-windows-386.tar.gz) | a7743e839e1aa19f5ee20b6ee5000ac8ef9e624ac5be63bb574fad6992e4b9167193ed07e03c9bc524e88bfeed66c95341a38a03bff1b10bc9910345f33019f0 2911 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-windows-amd64.tar.gz) | 5f1d19c230bd3542866d16051808d184e9dd3e2f8c001ed4cee7b5df91f872380c2bf56a3add8c9413ead9d8c369efce2bcab4412174df9b823d3592677bf74e 2912 2913 ### Server binaries 2914 2915 filename | sha512 hash 2916 -------- | ----------- 2917 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-amd64.tar.gz) | ef2cac10febde231aeb6f131e589450c560eeaab8046b49504127a091cddc17bc518c2ad56894a6a033033ab6fc6e121b1cc23691683bc36f45fe6b1dd8e0510 2918 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-arm.tar.gz) | d11c9730307f08e80b2b8a7c64c3e9a9e43c622002e377dfe3a386f4541e24adc79a199a6f280f40298bb36793194fd44ed45defe8a3ee54a9cb1386bc26e905 2919 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-arm64.tar.gz) | 28f8c32bf98ee1add7edf5d341c3bac1afc0085f90dcbbfb8b27a92087f13e2b53c327c8935ee29bf1dc3160655b32bbe3e29d5741a8124a3848a777e7d42933 2920 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-ppc64le.tar.gz) | 99ae8d44b0de3518c27fa8bbddd2ecf053dfb789fb9d65f8a4ecf4c8331cf63d2f09a41c2bcd5573247d5f66a1b2e51944379df1715017d920d521b98589508a 2921 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-s390x.tar.gz) | f8c0e954a2dfc6845614488dadeed069cc7f3f08e33c351d7a77c6ef97867af590932e8576d12998a820a0e4d35d2eee797c764e2810f09ab1e90a5acaeaad33 2922 2923 ### Node binaries 2924 2925 filename | sha512 hash 2926 -------- | ----------- 2927 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-amd64.tar.gz) | c5456d50bfbe0d75fb150b3662ed7468a0abd3970792c447824f326894382c47bbd3a2cc5a290f691c8c09585ff6fe505ab86b4aff2b7e5ccee11b5e6354ae6c 2928 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-arm.tar.gz) | 335b5cd8672e053302fd94d932fb2fa2e48eeeb1799650b3f93acdfa635e03a8453637569ab710c46885c8317759f4c60aaaf24dca9817d9fa47500fe4a3ca53 2929 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-arm64.tar.gz) | 3ee87dbeed8ace9351ac89bdaf7274dd10b4faec3ceba0825f690ec7a2bb7eb7c634274a1065a0939eec8ff3e43f72385f058f4ec141841550109e775bc5eff9 2930 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-ppc64le.tar.gz) | 6956f965b8d719b164214ec9195fdb2c776b907fe6d2c524082f00c27872a73475927fd7d2a994045ce78f6ad2aa5aeaf1eb5514df1810d2cfe342fd4e5ce4a1 2931 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-s390x.tar.gz) | 3b643aa905c709c57083c28dd9e8ffd88cb64466cda1499da7fc54176b775003e08b9c7a07b0964064df67c8142f6f1e6c13bfc261bd65fb064049920bfa57d0 2932 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-windows-amd64.tar.gz) | b2e6d6fb0091f2541f9925018c2bdbb0138a95bab06b4c6b38abf4b7144b2575422263b78fb3c6fd09e76d90a25a8d35a6d4720dc169794d42c95aa22ecc6d5f 2933 2934 ## Changelog since v1.21.0-alpha.1 2935 2936 ## Urgent Upgrade Notes 2937 2938 ### (No, really, you MUST read this before you upgrade) 2939 2940 - Remove storage metrics `storage_operation_errors_total`, since we already have `storage_operation_status_count`.And add new field `status` for `storage_operation_duration_seconds`, so that we can know about all status storage operation latency. ([#98332](https://github.com/kubernetes/kubernetes/pull/98332), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation and Storage] 2941 2942 ## Changes by Kind 2943 2944 ### Deprecation 2945 2946 - Remove the TokenRequest and TokenRequestProjection feature gates ([#97148](https://github.com/kubernetes/kubernetes/pull/97148), [@wawa0210](https://github.com/wawa0210)) [SIG Node] 2947 - Removing experimental windows container hyper-v support with Docker ([#97141](https://github.com/kubernetes/kubernetes/pull/97141), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] 2948 - The `export` query parameter (inconsistently supported by API resources and deprecated in v1.14) is fully removed. Requests setting this query parameter will now receive a 400 status response. ([#98312](https://github.com/kubernetes/kubernetes/pull/98312), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Auth and Testing] 2949 2950 ### API Change 2951 2952 - Enable SPDY pings to keep connections alive, so that `kubectl exec` and `kubectl port-forward` won't be interrupted. ([#97083](https://github.com/kubernetes/kubernetes/pull/97083), [@knight42](https://github.com/knight42)) [SIG API Machinery and CLI] 2953 2954 ### Documentation 2955 2956 - Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97935](https://github.com/kubernetes/kubernetes/pull/97935), [@adeniyistephen](https://github.com/adeniyistephen)) [SIG Release and Testing] 2957 - Set kubelet option `--volume-stats-agg-period` to negative value to disable volume calculations. ([#96675](https://github.com/kubernetes/kubernetes/pull/96675), [@pacoxu](https://github.com/pacoxu)) [SIG Node] 2958 2959 ### Bug or Regression 2960 2961 - Clean ReplicaSet by revision instead of creation timestamp in deployment controller ([#97407](https://github.com/kubernetes/kubernetes/pull/97407), [@waynepeking348](https://github.com/waynepeking348)) [SIG Apps] 2962 - Ensure that client-go's EventBroadcaster is safe (non-racy) during shutdown. ([#95664](https://github.com/kubernetes/kubernetes/pull/95664), [@DirectXMan12](https://github.com/DirectXMan12)) [SIG API Machinery] 2963 - Fix azure file migration issue ([#97877](https://github.com/kubernetes/kubernetes/pull/97877), [@andyzhangx](https://github.com/andyzhangx)) [SIG Auth, Cloud Provider and Storage] 2964 - Fix kubelet from panic after getting the wrong signal ([#98200](https://github.com/kubernetes/kubernetes/pull/98200), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 2965 - Fix repeatedly acquire the inhibit lock ([#98088](https://github.com/kubernetes/kubernetes/pull/98088), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 2966 - Fixed a bug that the kubelet cannot start on BtrfS. ([#98042](https://github.com/kubernetes/kubernetes/pull/98042), [@gjkim42](https://github.com/gjkim42)) [SIG Node] 2967 - Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children ([#98068](https://github.com/kubernetes/kubernetes/pull/98068), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Apps] 2968 - Fixed no effect namespace when exposing deployment with --dry-run=client. ([#97492](https://github.com/kubernetes/kubernetes/pull/97492), [@masap](https://github.com/masap)) [SIG CLI] 2969 - Fixing a bug where a failed node may not have the NoExecute taint set correctly ([#96876](https://github.com/kubernetes/kubernetes/pull/96876), [@howieyuen](https://github.com/howieyuen)) [SIG Apps and Node] 2970 - Indentation of `Resource Quota` block in kubectl describe namespaces output gets correct. ([#97946](https://github.com/kubernetes/kubernetes/pull/97946), [@dty1er](https://github.com/dty1er)) [SIG CLI] 2971 - KUBECTL_EXTERNAL_DIFF now accepts equal sign for additional parameters. ([#98158](https://github.com/kubernetes/kubernetes/pull/98158), [@dougsland](https://github.com/dougsland)) [SIG CLI] 2972 - Kubeadm: fix a bug where "kubeadm join" would not properly handle missing names for existing etcd members. ([#97372](https://github.com/kubernetes/kubernetes/pull/97372), [@ihgann](https://github.com/ihgann)) [SIG Cluster Lifecycle] 2973 - Kubelet should ignore cgroup driver check on Windows node. ([#97764](https://github.com/kubernetes/kubernetes/pull/97764), [@pacoxu](https://github.com/pacoxu)) [SIG Node and Windows] 2974 - Make podTopologyHints protected by lock ([#95111](https://github.com/kubernetes/kubernetes/pull/95111), [@choury](https://github.com/choury)) [SIG Node] 2975 - Readjust kubelet_containers_per_pod_count bucket ([#98169](https://github.com/kubernetes/kubernetes/pull/98169), [@wawa0210](https://github.com/wawa0210)) [SIG Instrumentation and Node] 2976 - Scores from InterPodAffinity have stronger differentiation. ([#98096](https://github.com/kubernetes/kubernetes/pull/98096), [@leileiwan](https://github.com/leileiwan)) [SIG Scheduling] 2977 - Specifying the KUBE_TEST_REPO environment variable when e2e tests are executed will instruct the test infrastructure to load that image from a location within the specified repo, using a predefined pattern. ([#93510](https://github.com/kubernetes/kubernetes/pull/93510), [@smarterclayton](https://github.com/smarterclayton)) [SIG Testing] 2978 - Static pods will be deleted gracefully. ([#98103](https://github.com/kubernetes/kubernetes/pull/98103), [@gjkim42](https://github.com/gjkim42)) [SIG Node] 2979 - Use network.Interface.VirtualMachine.ID to get the binded VM 2980 Skip standalone VM when reconciling LoadBalancer ([#97635](https://github.com/kubernetes/kubernetes/pull/97635), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 2981 2982 ### Other (Cleanup or Flake) 2983 2984 - Kubeadm: change the default image repository for CI images from 'gcr.io/kubernetes-ci-images' to 'gcr.io/k8s-staging-ci-images' ([#97087](https://github.com/kubernetes/kubernetes/pull/97087), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 2985 - Migrate generic_scheduler.go and types.go to structured logging. ([#98134](https://github.com/kubernetes/kubernetes/pull/98134), [@tanjing2020](https://github.com/tanjing2020)) [SIG Scheduling] 2986 - Migrate proxy/winuserspace/proxier.go logs to structured logging ([#97941](https://github.com/kubernetes/kubernetes/pull/97941), [@JornShen](https://github.com/JornShen)) [SIG Network] 2987 - Migrate staging/src/k8s.io/apiserver/pkg/audit/policy/reader.go logs to structured logging. ([#98252](https://github.com/kubernetes/kubernetes/pull/98252), [@lala123912](https://github.com/lala123912)) [SIG API Machinery and Auth] 2988 - Migrate staging\src\k8s.io\apiserver\pkg\endpoints logs to structured logging ([#98093](https://github.com/kubernetes/kubernetes/pull/98093), [@lala123912](https://github.com/lala123912)) [SIG API Machinery] 2989 - Node ([#96552](https://github.com/kubernetes/kubernetes/pull/96552), [@pandaamanda](https://github.com/pandaamanda)) [SIG Apps, Cloud Provider, Node and Scheduling] 2990 - The kubectl alpha debug command was scheduled to be removed in v1.21. ([#98111](https://github.com/kubernetes/kubernetes/pull/98111), [@pandaamanda](https://github.com/pandaamanda)) [SIG CLI] 2991 - Update cri-tools to [v1.20.0](https://github.com/kubernetes-sigs/cri-tools/releases/tag/v1.20.0) ([#97967](https://github.com/kubernetes/kubernetes/pull/97967), [@rajibmitra](https://github.com/rajibmitra)) [SIG Cloud Provider] 2992 - Windows nodes on GCE will take longer to start due to dependencies installed at node creation time. ([#98284](https://github.com/kubernetes/kubernetes/pull/98284), [@pjh](https://github.com/pjh)) [SIG Cloud Provider] 2993 2994 ## Dependencies 2995 2996 ### Added 2997 _Nothing has changed._ 2998 2999 ### Changed 3000 - github.com/google/cadvisor: [v0.38.6 → v0.38.7](https://github.com/google/cadvisor/compare/v0.38.6...v0.38.7) 3001 - k8s.io/gengo: 83324d8 → b6c5ce2 3002 3003 ### Removed 3004 _Nothing has changed._ 3005 3006 3007 3008 # v1.21.0-alpha.1 3009 3010 3011 ## Downloads for v1.21.0-alpha.1 3012 3013 ### Source Code 3014 3015 filename | sha512 hash 3016 -------- | ----------- 3017 [kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes.tar.gz) | b2bacd5c3fc9f829e6269b7d2006b0c6e464ff848bb0a2a8f2fe52ad2d7c4438f099bd8be847d8d49ac6e4087f4d74d5c3a967acd798e0b0cb4d7a2bdb122997 3018 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-src.tar.gz) | 518ac5acbcf23902fb1b902b69dbf3e86deca5d8a9b5f57488a15f185176d5a109558f3e4df062366af874eca1bcd61751ee8098b0beb9bcdc025d9a1c9be693 3019 3020 ### Client binaries 3021 3022 filename | sha512 hash 3023 -------- | ----------- 3024 [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-darwin-amd64.tar.gz) | eaa7aea84a5ed954df5ec710cbeb6ec88b46465f43cb3d09aabe2f714b84a050a50bf5736089f09dbf1090f2e19b44823d656c917e3c8c877630756c3026f2b6 3025 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-386.tar.gz) | 47f74b8d46ad1779c5b0b5f15aa15d5513a504eeb6f53db4201fbe9ff8956cb986b7c1b0e9d50a99f78e9e2a7f304f3fc1cc2fa239296d9a0dd408eb6069e975 3026 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-amd64.tar.gz) | 1a148e282628b008c8abd03dd12ec177ced17584b5115d92cd33dd251e607097d42e9da8c7089bd947134b900f85eb75a4740b6a5dd580c105455b843559df39 3027 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-arm.tar.gz) | d13d2feb73bd032dc01f7e2955b98d8215a39fe1107d037a73fa1f7d06c3b93ebaa53ed4952d845c64454ef3cca533edb97132d234d50b6fb3bcbd8a8ad990eb 3028 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-arm64.tar.gz) | 8252105a17b09a78e9ad2c024e4e401a69764ac869708a071aaa06f81714c17b9e7c5b2eb8efde33f24d0b59f75c5da607d5e1e72bdf12adfbb8c829205cd1c1 3029 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-ppc64le.tar.gz) | 297a9082df4988389dc4be30eb636dff49f36f5d87047bab44745884e610f46a17ae3a08401e2cab155b7c439f38057bfd8288418215f7dd3bf6a49dbe61ea0e 3030 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-s390x.tar.gz) | 04c06490dd17cd5dccfd92bafa14acf64280ceaea370d9635f23aeb6984d1beae6d0d1d1506edc6f30f927deeb149b989d3e482b47fbe74008b371f629656e79 3031 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-windows-386.tar.gz) | ec6e9e87a7d685f8751d7e58f24f417753cff5554a7229218cb3a08195d461b2e12409344950228e9fbbc92a8a06d35dd86242da6ff1e6652ec1fae0365a88c1 3032 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-windows-amd64.tar.gz) | 51039e6221d3126b5d15e797002ae01d4f0b10789c5d2056532f27ef13f35c5a2e51be27764fda68e8303219963126559023aed9421313bec275c0827fbcaf8a 3033 3034 ### Server binaries 3035 3036 filename | sha512 hash 3037 -------- | ----------- 3038 [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-amd64.tar.gz) | 4edf820930c88716263560275e3bd7fadb8dc3700b9f8e1d266562e356e0abeb1a913f536377dab91218e3940b447d6bf1da343b85da25c2256dc4dcde5798dd 3039 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-arm.tar.gz) | b15213e53a8ab4ba512ce6ef9ad42dd197d419c61615cd23de344227fd846c90448d8f3d98e555b63ba5b565afa627cca6b7e3990ebbbba359c96f2391302df1 3040 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-arm64.tar.gz) | 5be29cca9a9358fc68351ee63e99d57dc2ffce6e42fc3345753dbbf7542ff2d770c4852424158540435fa6e097ce3afa9b13affc40c8b3b69fe8406798f8068f 3041 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-ppc64le.tar.gz) | 89fd99ab9ce85db0b94b86709932105efc883cc93959cf7ea9a39e79a4acea23064d7010eeb577450cccabe521c04b7ba47bbec212ed37edeed7cb04bad34518 3042 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-s390x.tar.gz) | 2fbc30862c77d247aa8d96ab9d1a144599505287b0033a3a2d0988958e7bb2f2e8b67f52c1fec74b4ec47d74ba22cd0f6cb5c4228acbaa72b1678d5fece0254d 3043 3044 ### Node binaries 3045 3046 filename | sha512 hash 3047 -------- | ----------- 3048 [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-amd64.tar.gz) | 95658d321a0a371c0900b401d1469d96915310afbc4e4b9b11f031438bb188513b57d5a60b5316c3b0c18f541cda6f0ac42f59a76495f8abc743a067115da23a 3049 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-arm.tar.gz) | f375acfb42aad6c65b833c270e7e3acfe9cd1d6b2441c33874e77faae263957f7acfe86f1b71f14298118595e4cc6952c7dea0c832f7f2e72428336f13034362 3050 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-arm64.tar.gz) | 43b4baccd58d74e7f48d096ab92f2bbbcdf47e30e7a3d2b56c6cc9f90002cfd4fefaac894f69bd5f9f4dbdb09a4749a77eb76b1b97d91746bd96fe94457879ab 3051 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-ppc64le.tar.gz) | e7962b522c6c7c14b9ee4c1d254d8bdd9846b2b33b0443fc9c4a41be6c40e5e6981798b720f0148f36263d5cc45d5a2bb1dd2f9ab2838e3d002e45b9bddeb7bf 3052 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-s390x.tar.gz) | 49ebc97f01829e65f7de15be00b882513c44782eaadd1b1825a227e3bd3c73cc6aca8345af05b303d8c43aa2cb944a069755b2709effb8cc22eae621d25d4ba5 3053 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-windows-amd64.tar.gz) | 6e0fd7724b09e6befbcb53b33574e97f2db089f2eee4bbf391abb7f043103a5e6e32e3014c0531b88f9a3ca88887bbc68625752c44326f98dd53adb3a6d1bed8 3054 3055 ## Changelog since v1.20.0 3056 3057 ## Urgent Upgrade Notes 3058 3059 ### (No, really, you MUST read this before you upgrade) 3060 3061 - Kube-proxy's IPVS proxy mode no longer sets the net.ipv4.conf.all.route_localnet sysctl parameter. Nodes upgrading will have net.ipv4.conf.all.route_localnet set to 1 but new nodes will inherit the system default (usually 0). If you relied on any behavior requiring net.ipv4.conf.all.route_localnet, you must set ensure it is enabled as kube-proxy will no longer set it automatically. This change helps to further mitigate CVE-2020-8558. ([#92938](https://github.com/kubernetes/kubernetes/pull/92938), [@lbernail](https://github.com/lbernail)) [SIG Network and Release] 3062 3063 ## Changes by Kind 3064 3065 ### Deprecation 3066 3067 - Deprecate the `topologyKeys` field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy. ([#96736](https://github.com/kubernetes/kubernetes/pull/96736), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps] 3068 - Kubeadm: deprecated command "alpha selfhosting pivot" is removed now. ([#97627](https://github.com/kubernetes/kubernetes/pull/97627), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] 3069 - Kubeadm: graduate the command `kubeadm alpha kubeconfig user` to `kubeadm kubeconfig user`. The `kubeadm alpha kubeconfig user` command is deprecated now. ([#97583](https://github.com/kubernetes/kubernetes/pull/97583), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] 3070 - Kubeadm: the "kubeadm alpha certs" command is removed now, please use "kubeadm certs" instead. ([#97706](https://github.com/kubernetes/kubernetes/pull/97706), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] 3071 - Remove the deprecated metrics "scheduling_algorithm_preemption_evaluation_seconds" and "binding_duration_seconds", suggest to use "scheduler_framework_extension_point_duration_seconds" instead. ([#96447](https://github.com/kubernetes/kubernetes/pull/96447), [@chendave](https://github.com/chendave)) [SIG Cluster Lifecycle, Instrumentation, Scheduling and Testing] 3072 - The PodSecurityPolicy API is deprecated in 1.21, and will no longer be served starting in 1.25. ([#97171](https://github.com/kubernetes/kubernetes/pull/97171), [@deads2k](https://github.com/deads2k)) [SIG Auth and CLI] 3073 3074 ### API Change 3075 3076 - Change the APIVersion proto name of BoundObjectRef from aPIVersion to apiVersion. ([#97379](https://github.com/kubernetes/kubernetes/pull/97379), [@kebe7jun](https://github.com/kebe7jun)) [SIG Auth] 3077 - Promote Immutable Secrets/ConfigMaps feature to Stable. 3078 This allows to set `Immutable` field in Secrets or ConfigMap object to mark their contents as immutable. ([#97615](https://github.com/kubernetes/kubernetes/pull/97615), [@wojtek-t](https://github.com/wojtek-t)) [SIG Apps, Architecture, Node and Testing] 3079 3080 ### Feature 3081 3082 - Add flag --lease-max-object-size and metric etcd_lease_object_counts for kube-apiserver to config and observe max objects attached to a single etcd lease. ([#97480](https://github.com/kubernetes/kubernetes/pull/97480), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery, Instrumentation and Scalability] 3083 - Add flag --lease-reuse-duration-seconds for kube-apiserver to config etcd lease reuse duration. ([#97009](https://github.com/kubernetes/kubernetes/pull/97009), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery and Scalability] 3084 - Adds the ability to pass --strict-transport-security-directives to the kube-apiserver to set the HSTS header appropriately. Be sure you understand the consequences to browsers before setting this field. ([#96502](https://github.com/kubernetes/kubernetes/pull/96502), [@249043822](https://github.com/249043822)) [SIG Auth] 3085 - Kubeadm now includes CoreDNS v1.8.0. ([#96429](https://github.com/kubernetes/kubernetes/pull/96429), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] 3086 - Kubeadm: add support for certificate chain validation. When using kubeadm in external CA mode, this allows an intermediate CA to be used to sign the certificates. The intermediate CA certificate must be appended to each signed certificate for this to work correctly. ([#97266](https://github.com/kubernetes/kubernetes/pull/97266), [@robbiemcmichael](https://github.com/robbiemcmichael)) [SIG Cluster Lifecycle] 3087 - Kubeadm: amend the node kernel validation to treat CGROUP_PIDS, FAIR_GROUP_SCHED as required and CFS_BANDWIDTH, CGROUP_HUGETLB as optional ([#96378](https://github.com/kubernetes/kubernetes/pull/96378), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle and Node] 3088 - The Kubernetes pause image manifest list now contains an image for Windows Server 20H2. ([#97322](https://github.com/kubernetes/kubernetes/pull/97322), [@claudiubelu](https://github.com/claudiubelu)) [SIG Windows] 3089 - The apimachinery util/net function used to detect the bind address `ResolveBindAddress()` 3090 takes into consideration global ip addresses on loopback interfaces when: 3091 - the host has default routes 3092 - there are no global IPs on those interfaces. 3093 in order to support more complex network scenarios like BGP Unnumbered RFC 5549 ([#95790](https://github.com/kubernetes/kubernetes/pull/95790), [@aojea](https://github.com/aojea)) [SIG Network] 3094 3095 ### Bug or Regression 3096 3097 - ## Changelog 3098 3099 ### General 3100 - Fix priority expander falling back to a random choice even though there is a higher priority option to choose 3101 - Clone `kubernetes/kubernetes` in `update-vendor.sh` shallowly, instead of fetching all revisions 3102 - Speed up binpacking by reducing the number of PreFilter calls (call once per pod instead of #pods*#nodes times) 3103 - Speed up finding unneeded nodes by 5x+ in very large clusters by reducing the number of PreFilter calls 3104 - Expose `--max-nodes-total` as a metric 3105 - Errors in `IncreaseSize` changed from type `apiError` to `cloudProviderError` 3106 - Make `build-in-docker` and `test-in-docker` work on Linux systems with SELinux enabled 3107 - Fix an error where existing nodes were not considered as destinations while finding place for pods in scale-down simulations 3108 - Remove redundant log lines and reduce severity around parsing kubeEnv 3109 - Don't treat nodes created by virtual kubelet as nodes from non-autoscaled node groups 3110 - Remove redundant logging around calculating node utilization 3111 - Add configurable `--network` and `--rm` flags for docker in `Makefile` 3112 - Subtract DaemonSet pods' requests from node allocatable in the denominator while computing node utilization 3113 - Include taints by condition when determining if a node is unready/still starting 3114 - Fix `update-vendor.sh` to work on OSX and zsh 3115 - Add best-effort eviction for DaemonSet pods while scaling down non-empty nodes 3116 - Add build support for ARM64 3117 3118 ### AliCloud 3119 - Add missing daemonsets and replicasets to ALI example cluster role 3120 3121 ### Apache CloudStack 3122 - Add support for Apache CloudStack 3123 3124 ### AWS 3125 - Regenerate list of EC2 instances 3126 - Fix pricing endpoint in AWS China Region 3127 3128 ### Azure 3129 - Add optional jitter on initial VMSS VM cache refresh, keep the refreshes spread over time 3130 - Serve from cache for the whole period of ongoing throttling 3131 - Fix unwanted VMSS VMs cache invalidations 3132 - Enforce setting the number of retries if cloud provider backoff is enabled 3133 - Don't update capacity if VMSS provisioning state is updating 3134 - Support allocatable resources overrides via VMSS tags 3135 - Add missing stable labels in template nodes 3136 - Proactively set instance status to deleting on node deletions 3137 3138 ### Cluster API 3139 - Migrate interaction with the API from using internal types to using Unstructured 3140 - Improve tests to work better with constrained resources 3141 - Add support for node autodiscovery 3142 - Add support for `--cloud-config` 3143 - Update group identifier to use for Cluster API annotations 3144 3145 ### Exoscale 3146 - Add support for Exoscale 3147 3148 ### GCE 3149 - Decrease the number of GCE Read Requests made while deleting nodes 3150 - Base pricing of custom instances on their instance family type 3151 - Add pricing information for missing machine types 3152 - Add pricing information for different GPU types 3153 - Ignore the new `topology.gke.io/zone` label when comparing groups 3154 - Add missing stable labels to template nodes 3155 3156 ### HuaweiCloud 3157 - Add auto scaling group support 3158 - Implement node group by AS 3159 - Implement getting desired instance number of node group 3160 - Implement increasing node group size 3161 - Implement TemplateNodeInfo 3162 - Implement caching instances 3163 3164 ### IONOS 3165 - Add support for IONOS 3166 3167 ### Kubemark 3168 - Skip non-kubemark nodes while computing node infos for node groups. 3169 3170 ### Magnum 3171 - Add Magnum support in the Cluster Autoscaler helm chart 3172 3173 ### Packet 3174 - Allow empty nodepools 3175 - Add support for multiple nodepools 3176 - Add pricing support 3177 3178 ## Image 3179 Image: `k8s.gcr.io/autoscaling/cluster-autoscaler:v1.20.0` ([#97011](https://github.com/kubernetes/kubernetes/pull/97011), [@towca](https://github.com/towca)) [SIG Cloud Provider] 3180 - AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. ([#96873](https://github.com/kubernetes/kubernetes/pull/96873), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG Node] 3181 - Add limited lines to log when having tail option ([#93920](https://github.com/kubernetes/kubernetes/pull/93920), [@zhouya0](https://github.com/zhouya0)) [SIG Node] 3182 - Avoid systemd-logind loading configuration warning ([#97950](https://github.com/kubernetes/kubernetes/pull/97950), [@wzshiming](https://github.com/wzshiming)) [SIG Node] 3183 - Cloud-controller-manager: routes controller should not depend on --allocate-node-cidrs ([#97029](https://github.com/kubernetes/kubernetes/pull/97029), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider and Testing] 3184 - Copy annotations with empty value when deployment rolls back ([#94858](https://github.com/kubernetes/kubernetes/pull/94858), [@waynepeking348](https://github.com/waynepeking348)) [SIG Apps] 3185 - Detach volumes from vSphere nodes not tracked by attach-detach controller ([#96689](https://github.com/kubernetes/kubernetes/pull/96689), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] 3186 - Fix kubectl label error when local=true is set. ([#97440](https://github.com/kubernetes/kubernetes/pull/97440), [@pandaamanda](https://github.com/pandaamanda)) [SIG CLI] 3187 - Fix Azure file share not deleted issue when the namespace is deleted ([#97417](https://github.com/kubernetes/kubernetes/pull/97417), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] 3188 - Fix CVE-2020-8555 for Gluster client connections. ([#97922](https://github.com/kubernetes/kubernetes/pull/97922), [@liggitt](https://github.com/liggitt)) [SIG Storage] 3189 - Fix counting error in service/nodeport/loadbalancer quota check ([#97451](https://github.com/kubernetes/kubernetes/pull/97451), [@pacoxu](https://github.com/pacoxu)) [SIG API Machinery, Network and Testing] 3190 - Fix kubectl-convert import known versions ([#97754](https://github.com/kubernetes/kubernetes/pull/97754), [@wzshiming](https://github.com/wzshiming)) [SIG CLI and Testing] 3191 - Fix missing cadvisor machine metrics. ([#97006](https://github.com/kubernetes/kubernetes/pull/97006), [@lingsamuel](https://github.com/lingsamuel)) [SIG Node] 3192 - Fix nil VMSS name when setting service to auto mode ([#97366](https://github.com/kubernetes/kubernetes/pull/97366), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 3193 - Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable ([#95269](https://github.com/kubernetes/kubernetes/pull/95269), [@SataQiu](https://github.com/SataQiu)) [SIG Node] 3194 - Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. ([#97980](https://github.com/kubernetes/kubernetes/pull/97980), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Node] 3195 - Fix to recover CSI volumes from certain dangling attachments ([#96617](https://github.com/kubernetes/kubernetes/pull/96617), [@yuga711](https://github.com/yuga711)) [SIG Apps and Storage] 3196 - Fix: azure file latency issue for metadata-heavy workloads ([#97082](https://github.com/kubernetes/kubernetes/pull/97082), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] 3197 - Fixed Cinder volume IDs on OpenStack Train ([#96673](https://github.com/kubernetes/kubernetes/pull/96673), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider] 3198 - Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. ([#97013](https://github.com/kubernetes/kubernetes/pull/97013), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] 3199 - Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. ([#97174](https://github.com/kubernetes/kubernetes/pull/97174), [@hanlins](https://github.com/hanlins)) [SIG Node] 3200 - Fixed bug in CPUManager with race on container map access ([#97427](https://github.com/kubernetes/kubernetes/pull/97427), [@klueska](https://github.com/klueska)) [SIG Node] 3201 - Fixed cleanup of block devices when /var/lib/kubelet is a symlink. ([#96889](https://github.com/kubernetes/kubernetes/pull/96889), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] 3202 - GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. ([#97740](https://github.com/kubernetes/kubernetes/pull/97740), [@prameshj](https://github.com/prameshj)) [SIG Cloud Provider and Network] 3203 - Ignore update pod with no new images in alwaysPullImages admission controller ([#96668](https://github.com/kubernetes/kubernetes/pull/96668), [@pacoxu](https://github.com/pacoxu)) [SIG Apps, Auth and Node] 3204 - Kubeadm now installs version 3.4.13 of etcd when creating a cluster with v1.19 ([#97244](https://github.com/kubernetes/kubernetes/pull/97244), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 3205 - Kubeadm: avoid detection of the container runtime for commands that do not need it ([#97625](https://github.com/kubernetes/kubernetes/pull/97625), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] 3206 - Kubeadm: fix a bug in the host memory detection code on 32bit Linux platforms ([#97403](https://github.com/kubernetes/kubernetes/pull/97403), [@abelbarrera15](https://github.com/abelbarrera15)) [SIG Cluster Lifecycle] 3207 - Kubeadm: fix a bug where "kubeadm upgrade" commands can fail if CoreDNS v1.8.0 is installed. ([#97919](https://github.com/kubernetes/kubernetes/pull/97919), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 3208 - Performance regression [#97685](https://github.com/kubernetes/kubernetes/issues/97685) has been fixed. ([#97860](https://github.com/kubernetes/kubernetes/pull/97860), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] 3209 - Remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS ([#97336](https://github.com/kubernetes/kubernetes/pull/97336), [@maaoBit](https://github.com/maaoBit)) [SIG Network] 3210 - The current version of the container image publicly exposed IP serving a /metrics endpoint to the Internet. The new version of the container image serves /metrics endpoint on a different port. ([#97621](https://github.com/kubernetes/kubernetes/pull/97621), [@vbannai](https://github.com/vbannai)) [SIG Cloud Provider] 3211 - Use force unmount for NFS volumes if regular mount fails after 1 minute timeout ([#96844](https://github.com/kubernetes/kubernetes/pull/96844), [@gnufied](https://github.com/gnufied)) [SIG Storage] 3212 - Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server would mean deletion of all the resources from container runtime. ([#92817](https://github.com/kubernetes/kubernetes/pull/92817), [@kmala](https://github.com/kmala)) [SIG Node] 3213 - Using exec auth plugins with kubectl no longer results in warnings about constructing many client instances from the same exec auth config. ([#97857](https://github.com/kubernetes/kubernetes/pull/97857), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Auth] 3214 - Warning about using a deprecated volume plugin is logged only once. ([#96751](https://github.com/kubernetes/kubernetes/pull/96751), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] 3215 3216 ### Other (Cleanup or Flake) 3217 3218 - Bump github.com/Azure/go-autorest/autorest to v0.11.12 ([#97033](https://github.com/kubernetes/kubernetes/pull/97033), [@patrickshan](https://github.com/patrickshan)) [SIG API Machinery, CLI, Cloud Provider and Cluster Lifecycle] 3219 - Delete deprecated mixed protocol annotation ([#97096](https://github.com/kubernetes/kubernetes/pull/97096), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] 3220 - Kube-proxy: Traffic from the cluster directed to ExternalIPs is always sent directly to the Service. ([#96296](https://github.com/kubernetes/kubernetes/pull/96296), [@aojea](https://github.com/aojea)) [SIG Network and Testing] 3221 - Kubeadm: fix a whitespace issue in the output of the "kubeadm join" command shown as the output of "kubeadm init" and "kubeadm token create --print-join-command" ([#97413](https://github.com/kubernetes/kubernetes/pull/97413), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] 3222 - Kubeadm: improve the error messaging when the user provides an invalid discovery token CA certificate hash. ([#97290](https://github.com/kubernetes/kubernetes/pull/97290), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] 3223 - Migrate log messages in pkg/scheduler/{scheduler.go,factory.go} to structured logging ([#97509](https://github.com/kubernetes/kubernetes/pull/97509), [@aldudko](https://github.com/aldudko)) [SIG Scheduling] 3224 - Migrate proxy/iptables/proxier.go logs to structured logging ([#97678](https://github.com/kubernetes/kubernetes/pull/97678), [@JornShen](https://github.com/JornShen)) [SIG Network] 3225 - Migrate some scheduler log messages to structured logging ([#97349](https://github.com/kubernetes/kubernetes/pull/97349), [@aldudko](https://github.com/aldudko)) [SIG Scheduling] 3226 - NONE ([#97167](https://github.com/kubernetes/kubernetes/pull/97167), [@geegeea](https://github.com/geegeea)) [SIG Node] 3227 - NetworkPolicy validation framework optimizations for rapidly verifying CNI's work correctly across several pods and namespaces ([#91592](https://github.com/kubernetes/kubernetes/pull/91592), [@jayunit100](https://github.com/jayunit100)) [SIG Network, Storage and Testing] 3228 - Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97618](https://github.com/kubernetes/kubernetes/pull/97618), [@jherrera123](https://github.com/jherrera123)) [SIG Release and Testing] 3229 - Scheduler plugin validation now provides all errors detected instead of the first one. ([#96745](https://github.com/kubernetes/kubernetes/pull/96745), [@lingsamuel](https://github.com/lingsamuel)) [SIG Node, Scheduling and Testing] 3230 - Storage related e2e testsuite redesign & cleanup ([#96573](https://github.com/kubernetes/kubernetes/pull/96573), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Storage and Testing] 3231 - The OIDC authenticator no longer waits 10 seconds before attempting to fetch the metadata required to verify tokens. ([#97693](https://github.com/kubernetes/kubernetes/pull/97693), [@enj](https://github.com/enj)) [SIG API Machinery and Auth] 3232 - The `AttachVolumeLimit` feature gate that is GA since v1.17 is now removed. ([#96539](https://github.com/kubernetes/kubernetes/pull/96539), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Storage] 3233 - The `CSINodeInfo` feature gate that is GA since v1.17 is unconditionally enabled, and can no longer be specified via the `--feature-gates` argument. ([#96561](https://github.com/kubernetes/kubernetes/pull/96561), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Apps, Auth, Scheduling, Storage and Testing] 3234 - The deprecated feature gates `RotateKubeletClientCertificate`, `AttachVolumeLimit`, `VolumePVCDataSource` and `EvenPodsSpread` are now unconditionally enabled and can no longer be specified in component invocations. ([#97306](https://github.com/kubernetes/kubernetes/pull/97306), [@gavinfish](https://github.com/gavinfish)) [SIG Node, Scheduling and Storage] 3235 - `ServiceNodeExclusion`, `NodeDisruptionExclusion` and `LegacyNodeRoleBehavior`(locked to false) features have been promoted to GA. 3236 To prevent control plane nodes being added to load balancers automatically, upgrade users need to add "node.kubernetes.io/exclude-from-external-load-balancers" label to control plane nodes. ([#97543](https://github.com/kubernetes/kubernetes/pull/97543), [@pacoxu](https://github.com/pacoxu)) [SIG API Machinery, Apps, Cloud Provider and Network] 3237 3238 ### Uncategorized 3239 3240 - Adding Brazilian Portuguese translation for kubectl ([#61595](https://github.com/kubernetes/kubernetes/pull/61595), [@cpanato](https://github.com/cpanato)) [SIG CLI] 3241 3242 ## Dependencies 3243 3244 ### Added 3245 _Nothing has changed._ 3246 3247 ### Changed 3248 - github.com/Azure/go-autorest/autorest: [v0.11.1 → v0.11.12](https://github.com/Azure/go-autorest/autorest/compare/v0.11.1...v0.11.12) 3249 - github.com/coredns/corefile-migration: [v1.0.10 → v1.0.11](https://github.com/coredns/corefile-migration/compare/v1.0.10...v1.0.11) 3250 - github.com/golang/mock: [v1.4.1 → v1.4.4](https://github.com/golang/mock/compare/v1.4.1...v1.4.4) 3251 - github.com/google/cadvisor: [v0.38.5 → v0.38.6](https://github.com/google/cadvisor/compare/v0.38.5...v0.38.6) 3252 - github.com/heketi/heketi: [c2e2a4a → v10.2.0+incompatible](https://github.com/heketi/heketi/compare/c2e2a4a...v10.2.0) 3253 - github.com/miekg/dns: [v1.1.4 → v1.1.35](https://github.com/miekg/dns/compare/v1.1.4...v1.1.35) 3254 - k8s.io/system-validators: v1.2.0 → v1.3.0 3255 3256 ### Removed 3257 - rsc.io/quote/v3: v3.1.0 3258 - rsc.io/sampler: v1.3.0