github.com/qsunny/k8s@v0.0.0-20220101153623-e6dca256d5bf/examples-master/staging/volumes/vsphere/README.md (about) 1 # vSphere Volume 2 3 - [Prerequisites](#prerequisites) 4 - [Examples](#examples) 5 - [Volumes](#volumes) 6 - [Persistent Volumes](#persistent-volumes) 7 - [Storage Class](#storage-class) 8 - [Storage Policy Management inside kubernetes](#storage-policy-management-inside-kubernetes) 9 - [Using existing vCenter SPBM policy](#using-existing-vcenter-spbm-policy) 10 - [Virtual SAN policy support inside Kubernetes](#virtual-san-policy-support-inside-kubernetes) 11 - [Stateful Set](#stateful-set) 12 13 ## Prerequisites 14 15 - Kubernetes with vSphere Cloud Provider configured. 16 For cloudprovider configuration please refer [vSphere getting started guide](http://kubernetes.io/docs/getting-started-guides/vsphere/). 17 18 ## Examples 19 20 ### Volumes 21 22 1. Create VMDK. 23 24 First ssh into ESX and then use following command to create vmdk, 25 26 ```shell 27 vmkfstools -c 2G /vmfs/volumes/datastore1/volumes/myDisk.vmdk 28 ``` 29 30 2. Create Pod which uses 'myDisk.vmdk'. 31 32 See example 33 34 ```yaml 35 apiVersion: v1 36 kind: Pod 37 metadata: 38 name: test-vmdk 39 spec: 40 containers: 41 - image: k8s.gcr.io/test-webserver 42 name: test-container 43 volumeMounts: 44 - mountPath: /test-vmdk 45 name: test-volume 46 volumes: 47 - name: test-volume 48 # This VMDK volume must already exist. 49 vsphereVolume: 50 volumePath: "[datastore1] volumes/myDisk" 51 fsType: ext4 52 ``` 53 54 [Download example](vsphere-volume-pod.yaml?raw=true) 55 56 Creating the pod: 57 58 ``` bash 59 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pod.yaml 60 ``` 61 62 Verify that pod is running: 63 64 ```bash 65 $ kubectl get pods test-vmdk 66 NAME READY STATUS RESTARTS AGE 67 test-vmdk 1/1 Running 0 48m 68 ``` 69 70 ### Persistent Volumes 71 72 1. Create VMDK. 73 74 First ssh into ESX and then use following command to create vmdk, 75 76 ```shell 77 vmkfstools -c 2G /vmfs/volumes/datastore1/volumes/myDisk.vmdk 78 ``` 79 80 2. Create Persistent Volume. 81 82 See example: 83 84 ```yaml 85 apiVersion: v1 86 kind: PersistentVolume 87 metadata: 88 name: pv0001 89 spec: 90 capacity: 91 storage: 2Gi 92 accessModes: 93 - ReadWriteOnce 94 persistentVolumeReclaimPolicy: Retain 95 vsphereVolume: 96 volumePath: "[datastore1] volumes/myDisk" 97 fsType: ext4 98 ``` 99 In the above example datastore1 is located in the root folder. If datastore is member of Datastore Cluster or located in sub folder, the folder path needs to be provided in the VolumePath as below. 100 ```yaml 101 vsphereVolume: 102 VolumePath: "[DatastoreCluster/datastore1] volumes/myDisk" 103 ``` 104 105 [Download example](vsphere-volume-pv.yaml?raw=true) 106 107 Creating the persistent volume: 108 109 ``` bash 110 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pv.yaml 111 ``` 112 113 Verifying persistent volume is created: 114 115 ``` bash 116 $ kubectl describe pv pv0001 117 Name: pv0001 118 Labels: <none> 119 Status: Available 120 Claim: 121 Reclaim Policy: Retain 122 Access Modes: RWO 123 Capacity: 2Gi 124 Message: 125 Source: 126 Type: vSphereVolume (a Persistent Disk resource in vSphere) 127 VolumePath: [datastore1] volumes/myDisk 128 FSType: ext4 129 No events. 130 ``` 131 132 3. Create Persistent Volume Claim. 133 134 See example: 135 136 ```yaml 137 kind: PersistentVolumeClaim 138 apiVersion: v1 139 metadata: 140 name: pvc0001 141 spec: 142 accessModes: 143 - ReadWriteOnce 144 resources: 145 requests: 146 storage: 2Gi 147 ``` 148 149 [Download example](vsphere-volume-pvc.yaml?raw=true) 150 151 Creating the persistent volume claim: 152 153 ``` bash 154 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvc.yaml 155 ``` 156 157 Verifying persistent volume claim is created: 158 159 ``` bash 160 $ kubectl describe pvc pvc0001 161 Name: pvc0001 162 Namespace: default 163 Status: Bound 164 Volume: pv0001 165 Labels: <none> 166 Capacity: 2Gi 167 Access Modes: RWO 168 No events. 169 ``` 170 171 3. Create Pod which uses Persistent Volume Claim. 172 173 See example: 174 175 ```yaml 176 apiVersion: v1 177 kind: Pod 178 metadata: 179 name: pvpod 180 spec: 181 containers: 182 - name: test-container 183 image: k8s.gcr.io/test-webserver 184 volumeMounts: 185 - name: test-volume 186 mountPath: /test-vmdk 187 volumes: 188 - name: test-volume 189 persistentVolumeClaim: 190 claimName: pvc0001 191 ``` 192 193 [Download example](vsphere-volume-pvcpod.yaml?raw=true) 194 195 Creating the pod: 196 197 ``` bash 198 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcpod.yaml 199 ``` 200 201 Verifying pod is created: 202 203 ``` bash 204 $ kubectl get pod pvpod 205 NAME READY STATUS RESTARTS AGE 206 pvpod 1/1 Running 0 48m 207 ``` 208 209 ### Storage Class 210 211 __Note: Here you don't need to create vmdk it is created for you.__ 212 1. Create Storage Class. 213 214 Example 1: 215 216 ```yaml 217 kind: StorageClass 218 apiVersion: storage.k8s.io/v1beta1 219 metadata: 220 name: fast 221 provisioner: kubernetes.io/vsphere-volume 222 parameters: 223 diskformat: zeroedthick 224 fstype: ext3 225 ``` 226 227 [Download example](vsphere-volume-sc-fast.yaml?raw=true) 228 229 You can also specify the datastore in the Storageclass as shown in example 2. The volume will be created on the datastore specified in the storage class. 230 This field is optional. If not specified as shown in example 1, the volume will be created on the datastore specified in the vsphere config file used to initialize the vSphere Cloud Provider. 231 232 Example 2: 233 234 ```yaml 235 kind: StorageClass 236 apiVersion: storage.k8s.io/v1beta1 237 metadata: 238 name: fast 239 provisioner: kubernetes.io/vsphere-volume 240 parameters: 241 diskformat: zeroedthick 242 datastore: VSANDatastore 243 ``` 244 If datastore is member of DataStore Cluster or within some sub folder, the datastore folder path needs to be provided in the datastore parameter as below. 245 246 ```yaml 247 parameters: 248 datastore: DatastoreCluster/VSANDatastore 249 ``` 250 251 [Download example](vsphere-volume-sc-with-datastore.yaml?raw=true) 252 Creating the storageclass: 253 254 ``` bash 255 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-sc-fast.yaml 256 ``` 257 258 Verifying storage class is created: 259 260 ``` bash 261 $ kubectl describe storageclass fast 262 Name: fast 263 IsDefaultClass: No 264 Annotations: <none> 265 Provisioner: kubernetes.io/vsphere-volume 266 Parameters: diskformat=zeroedthick,fstype=ext3 267 No events. 268 ``` 269 270 2. Create Persistent Volume Claim. 271 272 See example: 273 274 ```yaml 275 kind: PersistentVolumeClaim 276 apiVersion: v1 277 metadata: 278 name: pvcsc001 279 annotations: 280 volume.beta.kubernetes.io/storage-class: fast 281 spec: 282 accessModes: 283 - ReadWriteOnce 284 resources: 285 requests: 286 storage: 2Gi 287 ``` 288 289 [Download example](vsphere-volume-pvcsc.yaml?raw=true) 290 291 Creating the persistent volume claim: 292 293 ``` bash 294 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcsc.yaml 295 ``` 296 297 Verifying persistent volume claim is created: 298 299 ``` bash 300 $ kubectl describe pvc pvcsc001 301 Name: pvcsc001 302 Namespace: default 303 StorageClass: fast 304 Status: Bound 305 Volume: pvc-83295256-f8e0-11e6-8263-005056b2349c 306 Labels: <none> 307 Capacity: 2Gi 308 Access Modes: RWO 309 Events: 310 FirstSeen LastSeen Count From SubObjectPath Type Reason Message 311 --------- -------- ----- ---- ------------- -------- ------ ------- 312 1m 1m 1 persistentvolume-controller Normal ProvisioningSucceeded Successfully provisioned volume pvc-83295256-f8e0-11e6-8263-005056b2349c using kubernetes.io/vsphere-volume 313 314 ``` 315 316 Persistent Volume is automatically created and is bounded to this pvc. 317 318 Verifying persistent volume claim is created: 319 320 ``` bash 321 $ kubectl describe pv pvc-83295256-f8e0-11e6-8263-005056b2349c 322 Name: pvc-83295256-f8e0-11e6-8263-005056b2349c 323 Labels: <none> 324 StorageClass: fast 325 Status: Bound 326 Claim: default/pvcsc001 327 Reclaim Policy: Delete 328 Access Modes: RWO 329 Capacity: 2Gi 330 Message: 331 Source: 332 Type: vSphereVolume (a Persistent Disk resource in vSphere) 333 VolumePath: [datastore1] kubevols/kubernetes-dynamic-pvc-83295256-f8e0-11e6-8263-005056b2349c.vmdk 334 FSType: ext3 335 No events. 336 ``` 337 338 __Note: VMDK is created inside ```kubevols``` folder in datastore which is mentioned in 'vsphere' cloudprovider configuration. 339 The cloudprovider config is created during setup of Kubernetes cluster on vSphere.__ 340 341 3. Create Pod which uses Persistent Volume Claim with storage class. 342 343 See example: 344 345 ```yaml 346 apiVersion: v1 347 kind: Pod 348 metadata: 349 name: pvpod 350 spec: 351 containers: 352 - name: test-container 353 image: k8s.gcr.io/test-webserver 354 volumeMounts: 355 - name: test-volume 356 mountPath: /test-vmdk 357 volumes: 358 - name: test-volume 359 persistentVolumeClaim: 360 claimName: pvcsc001 361 ``` 362 363 [Download example](vsphere-volume-pvcscpod.yaml?raw=true) 364 365 Creating the pod: 366 367 ``` bash 368 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcscpod.yaml 369 ``` 370 371 Verifying pod is created: 372 373 ``` bash 374 $ kubectl get pod pvpod 375 NAME READY STATUS RESTARTS AGE 376 pvpod 1/1 Running 0 48m 377 ``` 378 379 ### Storage Policy Management inside kubernetes 380 #### Using existing vCenter SPBM policy 381 Admins can use the existing vCenter Storage Policy Based Management (SPBM) policy to configure a persistent volume with the SPBM policy. 382 __Note: Here you don't need to create persistent volume it is created for you.__ 383 384 1. Create Storage Class. 385 386 Example 1: 387 388 ```yaml 389 kind: StorageClass 390 apiVersion: storage.k8s.io/v1 391 metadata: 392 name: fast 393 provisioner: kubernetes.io/vsphere-volume 394 parameters: 395 diskformat: zeroedthick 396 storagePolicyName: gold 397 ``` 398 [Download example](vsphere-volume-spbm-policy.yaml?raw=true) 399 400 The admin specifies the SPBM policy - "gold" as part of storage class definition for dynamic volume provisioning. When a PVC is created, the persistent volume will be provisioned on a compatible datastore with maximum free space that satisfies the "gold" storage policy requirements. 401 402 Example 2: 403 404 ```yaml 405 kind: StorageClass 406 apiVersion: storage.k8s.io/v1 407 metadata: 408 name: fast 409 provisioner: kubernetes.io/vsphere-volume 410 parameters: 411 diskformat: zeroedthick 412 storagePolicyName: gold 413 datastore: VSANDatastore 414 ``` 415 [Download example](vsphere-volume-spbm-policy-with-datastore.yaml?raw=true) 416 417 The admin can also specify a custom datastore where he wants the volume to be provisioned along with the SPBM policy name. When a PVC is created, the vSphere Cloud Provider checks if the user specified datastore satisfies the "gold" storage policy requirements. If yes, it will provision the persistent volume on user specified datastore. If not, it will error out to the user that the user specified datastore is not compatible with "gold" storage policy requirements. 418 419 #### Virtual SAN policy support inside Kubernetes 420 421 Vsphere Infrastructure(VI) Admins will have the ability to specify custom Virtual SAN Storage Capabilities during dynamic volume provisioning. You can now define storage requirements, such as performance and availability, in the form of storage capabilities during dynamic volume provisioning. The storage capability requirements are converted into a Virtual SAN policy which are then pushed down to the Virtual SAN layer when a persistent volume (virtual disk) is being created. The virtual disk is distributed across the Virtual SAN datastore to meet the requirements. 422 423 The official [VSAN policy documentation](https://pubs.vmware.com/vsphere-65/index.jsp?topic=%2Fcom.vmware.vsphere.virtualsan.doc%2FGUID-08911FD3-2462-4C1C-AE81-0D4DBC8F7990.html) describes in detail about each of the individual storage capabilities that are supported by VSAN. The user can specify these storage capabilities as part of storage class definition based on his application needs. 424 425 The policy settings can be one or more of the following: 426 427 * *hostFailuresToTolerate*: represents NumberOfFailuresToTolerate 428 * *diskStripes*: represents NumberofDiskStripesPerObject 429 * *objectSpaceReservation*: represents ObjectSpaceReservation 430 * *cacheReservation*: represents FlashReadCacheReservation 431 * *iopsLimit*: represents IOPSLimitForObject 432 * *forceProvisioning*: represents if volume must be Force Provisioned 433 434 __Note: Here you don't need to create persistent volume it is created for you.__ 435 1. Create Storage Class. 436 437 Example 1: 438 439 ```yaml 440 kind: StorageClass 441 apiVersion: storage.k8s.io/v1beta1 442 metadata: 443 name: fast 444 provisioner: kubernetes.io/vsphere-volume 445 parameters: 446 diskformat: zeroedthick 447 hostFailuresToTolerate: "2" 448 cachereservation: "20" 449 ``` 450 [Download example](vsphere-volume-sc-vsancapabilities.yaml?raw=true) 451 452 Here a persistent volume will be created with the Virtual SAN capabilities - hostFailuresToTolerate to 2 and cachereservation is 20% read cache reserved for storage object. Also the persistent volume will be *zeroedthick* disk. 453 The official [VSAN policy documentation](https://pubs.vmware.com/vsphere-65/index.jsp?topic=%2Fcom.vmware.vsphere.virtualsan.doc%2FGUID-08911FD3-2462-4C1C-AE81-0D4DBC8F7990.html) describes in detail about each of the individual storage capabilities that are supported by VSAN and can be configured on the virtual disk. 454 455 You can also specify the datastore in the Storageclass as shown in example 2. The volume will be created on the datastore specified in the storage class. 456 This field is optional. If not specified as shown in example 1, the volume will be created on the datastore specified in the vsphere config file used to initialize the vSphere Cloud Provider. 457 458 Example 2: 459 460 ```yaml 461 kind: StorageClass 462 apiVersion: storage.k8s.io/v1beta1 463 metadata: 464 name: fast 465 provisioner: kubernetes.io/vsphere-volume 466 parameters: 467 diskformat: zeroedthick 468 datastore: VSANDatastore 469 hostFailuresToTolerate: "2" 470 cachereservation: "20" 471 ``` 472 473 [Download example](vsphere-volume-sc-vsancapabilities-with-datastore.yaml?raw=true) 474 475 __Note: If you do not apply a storage policy during dynamic provisioning on a VSAN datastore, it will use a default Virtual SAN policy.__ 476 477 Creating the storageclass: 478 479 ``` bash 480 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-sc-vsancapabilities.yaml 481 ``` 482 483 Verifying storage class is created: 484 485 ``` bash 486 $ kubectl describe storageclass fast 487 Name: fast 488 Annotations: <none> 489 Provisioner: kubernetes.io/vsphere-volume 490 Parameters: diskformat=zeroedthick, hostFailuresToTolerate="2", cachereservation="20" 491 No events. 492 ``` 493 494 2. Create Persistent Volume Claim. 495 496 See example: 497 498 ```yaml 499 kind: PersistentVolumeClaim 500 apiVersion: v1 501 metadata: 502 name: pvcsc-vsan 503 annotations: 504 volume.beta.kubernetes.io/storage-class: fast 505 spec: 506 accessModes: 507 - ReadWriteOnce 508 resources: 509 requests: 510 storage: 2Gi 511 ``` 512 513 [Download example](vsphere-volume-pvcsc.yaml?raw=true) 514 515 Creating the persistent volume claim: 516 517 ``` bash 518 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcsc.yaml 519 ``` 520 521 Verifying persistent volume claim is created: 522 523 ``` bash 524 $ kubectl describe pvc pvcsc-vsan 525 Name: pvcsc-vsan 526 Namespace: default 527 Status: Bound 528 Volume: pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d 529 Labels: <none> 530 Capacity: 2Gi 531 Access Modes: RWO 532 No events. 533 ``` 534 535 Persistent Volume is automatically created and is bounded to this pvc. 536 537 Verifying persistent volume claim is created: 538 539 ``` bash 540 $ kubectl describe pv pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d 541 Name: pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d 542 Labels: <none> 543 Status: Bound 544 Claim: default/pvcsc-vsan 545 Reclaim Policy: Delete 546 Access Modes: RWO 547 Capacity: 2Gi 548 Message: 549 Source: 550 Type: vSphereVolume (a Persistent Disk resource in vSphere) 551 VolumePath: [VSANDatastore] kubevols/kubernetes-dynamic-pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d.vmdk 552 FSType: ext4 553 No events. 554 ``` 555 556 __Note: VMDK is created inside ```kubevols``` folder in datastore which is mentioned in 'vsphere' cloudprovider configuration. 557 The cloudprovider config is created during setup of Kubernetes cluster on vSphere.__ 558 559 3. Create Pod which uses Persistent Volume Claim with storage class. 560 561 See example: 562 563 ```yaml 564 apiVersion: v1 565 kind: Pod 566 metadata: 567 name: pvpod 568 spec: 569 containers: 570 - name: test-container 571 image: k8s.gcr.io/test-webserver 572 volumeMounts: 573 - name: test-volume 574 mountPath: /test 575 volumes: 576 - name: test-volume 577 persistentVolumeClaim: 578 claimName: pvcsc-vsan 579 ``` 580 581 [Download example](vsphere-volume-pvcscvsanpod.yaml?raw=true) 582 583 Creating the pod: 584 585 ``` bash 586 $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcscvsanpod.yaml 587 ``` 588 589 Verifying pod is created: 590 591 ``` bash 592 $ kubectl get pod pvpod 593 NAME READY STATUS RESTARTS AGE 594 pvpod 1/1 Running 0 48m 595 ``` 596 597 ### Stateful Set 598 599 vSphere volumes can be consumed by Stateful Sets. 600 601 1. Create a storage class that will be used by the ```volumeClaimTemplates``` of a Stateful Set. 602 603 See example: 604 605 ```yaml 606 kind: StorageClass 607 apiVersion: storage.k8s.io/v1beta1 608 metadata: 609 name: thin-disk 610 provisioner: kubernetes.io/vsphere-volume 611 parameters: 612 diskformat: thin 613 ``` 614 615 [Download example](simple-storageclass.yaml) 616 617 2. Create a Stateful set that consumes storage from the Storage Class created. 618 619 See example: 620 ```yaml 621 --- 622 apiVersion: v1 623 kind: Service 624 metadata: 625 name: nginx 626 labels: 627 app: nginx 628 spec: 629 ports: 630 - port: 80 631 name: web 632 clusterIP: None 633 selector: 634 app: nginx 635 --- 636 # for k8s versions before 1.9.0 use apps/v1beta2 and before 1.8.0 use extensions/v1beta1 637 apiVersion:apps/v1 638 kind: StatefulSet 639 metadata: 640 name: web 641 labels: 642 app: nginx 643 spec: 644 serviceName: "nginx" 645 selector: 646 matchLabels: 647 app: nginx 648 replicas: 14 649 template: 650 metadata: 651 labels: 652 app: nginx 653 spec: 654 containers: 655 - name: nginx 656 image: k8s.gcr.io/nginx-slim:0.8 657 ports: 658 - containerPort: 80 659 name: web 660 volumeMounts: 661 - name: www 662 mountPath: /usr/share/nginx/html 663 volumeClaimTemplates: 664 - metadata: 665 name: www 666 annotations: 667 volume.beta.kubernetes.io/storage-class: thin-disk 668 spec: 669 accessModes: [ "ReadWriteOnce" ] 670 resources: 671 requests: 672 storage: 1Gi 673 ``` 674 This will create Persistent Volume Claims for each replica and provision a volume for each claim if an existing volume could be bound to the claim. 675 676 [Download example](simple-statefulset.yaml) 677 678 <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> 679 []() 680 <!-- END MUNGE: GENERATED_ANALYTICS -->