github.com/alibaba/sealer@v0.8.6-0.20220430115802-37a2bdaa8173/docs/test-report/release-v0.1.5.md (about) 1 # sealer release v0.1.5 test report 2 3 ## Cloud mode: 4 5 ### init: 6 7 ```shell 8 1.wget https://github.com/alibaba/sealer/releases/download/v0.1.5-rc/sealer-0.0.0-linux-amd64.tar.gz && tar zxvf sealer-0.0.0-linux-amd64.tar.gz && mv sealer /usr/local/bin/ 9 2.export 'ACCESSKEYID'='***'&&export 'ACCESSKEYSECRET'='***'&&export 'RegionID'='***' 10 3.sealer run registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.19.9 //（Cluster started successfully） 11 ``` 12 13 The default password is the Clusterfile password in the Run image. 14 15 **Check the Clusterfile：** 16 17 ```shell 18 [root@iZ2vc2ce4hs85f7asxhaipZ ~]# cat /root/.sealer/my-cluster/Clusterfile 19 apiVersion: zlink.aliyun.com/v1alpha1 20 kind: Cluster 21 metadata: 22 annotations: 23 sea.aliyun.com/ClusterEIP: 47.108.132.245 24 sea.aliyun.com/EipID: eip-2vca3x1lxu96ropt66am5 25 sea.aliyun.com/Master0ID: i-2vc2ce4hs85f7ql6dqda 26 sea.aliyun.com/Master0InternalIP: 172.16.0.194 27 sea.aliyun.com/MasterIDs: i-2vc2ce4hs85f7ql6dqdb,i-2vc2ce4hs85f7ql6dqdc,i-2vc2ce4hs85f7ql6dqda 28 sea.aliyun.com/NodeIDs: i-2vc4eamks4zejf9n16tm,i-2vc4eamks4zejf9n16tn,i-2vc4eamks4zejf9n16tl 29 sea.aliyun.com/RegionID: cn-chengdu 30 sea.aliyun.com/SecurityGroupID: sg-2vcdx512qke5iqpx1iwx 31 sea.aliyun.com/VSwitchID: vsw-2vc55ufqm05qb8zdxux0h 32 sea.aliyun.com/VpcID: vpc-2vcj16rdif2upvgeowdu6 33 sea.aliyun.com/ZoneID: cn-chengdu-a 34 creationTimestamp: null 35 name: my-cluster 36 spec: 37 certSANS: 38 - aliyun-inc.com 39 - 10.0.0.2 40 image: kubernetes:v1.19.9 41 masters: 42 count: "3" 43 cpu: "4" 44 dataDisks: 45 - "100" 46 ipList: 47 - 172.16.0.194 48 - 172.16.0.196 49 - 172.16.0.195 50 memory: "4" 51 systemDisk: "100" 52 network: 53 cniName: calico 54 interface: eth0 55 podCIDR: 100.64.0.0/10 56 svcCIDR: 10.96.0.0/22 57 nodes: 58 count: "3" 59 cpu: "4" 60 dataDisks: 61 - "100" 62 ipList: 63 - 172.16.0.198 64 - 172.16.0.199 65 - 172.16.0.197 66 memory: "4" 67 systemDisk: "100" 68 provider: ALI_CLOUD 69 ssh: 70 passwd: Seadent123 71 pk: xxx 72 pkPasswd: xxx 73 user: root 74 status: {} 75 ``` 76 77 ### scale up: 78 79 #### 1. Modify /root/sealer/mycluster/Clusterfile file Masters, Nodes Count number： 80 81 ```shell 82 masters: 83 count: "4"//Master adds one node 84 cpu: "4" 85 dataDisks: 86 - "100" 87 ipList: 88 - 172.16.0.194 89 - 172.16.0.196 90 - 172.16.0.195 91 memory: "4" 92 systemDisk: "100" 93 network: 94 cniName: calico 95 interface: eth0 96 podCIDR: 100.64.0.0/10 97 svcCIDR: 10.96.0.0/22 98 nodes: 99 count: "4"//node adds one node 100 ``` 101 102 #### 2. Execute Sealer apply-f /root/.sealer/my-cluster/Clusterfile (Successfully executed): 103 104 ```shell 105 [root@iZ2vc2ce4hs85f7asxhaipZ ~]# kubectl get nodes 106 NAME STATUS ROLES AGE VERSION 107 iz2vc1hagqsoc9ut8fp511z Ready <none> 2m50s v1.19.9 108 iz2vc2ce4hs85f7ql6dqdaz Ready master 22m v1.19.9 109 iz2vc2ce4hs85f7ql6dqdbz Ready master 20m v1.19.9 110 iz2vc2ce4hs85f7ql6dqdcz Ready master 21m v1.19.9 111 iz2vc4eamks4zejf9n16tlz Ready <none> 19m v1.19.9 112 iz2vc4eamks4zejf9n16tmz Ready <none> 19m v1.19.9 113 iz2vc4eamks4zejf9n16tnz Ready <none> 19m v1.19.9 114 iz2vcdx512qke5j0l5bksrz Ready master 3m6s v1.19.9 115 [root@iZ2vc2ce4hs85f7asxhaipZ ~]# cat /root/.sealer/my-cluster/Clusterfile 116 apiVersion: zlink.aliyun.com/v1alpha1 117 kind: Cluster 118 metadata: 119 annotations: 120 sea.aliyun.com/ClusterEIP: 47.108.132.245 121 sea.aliyun.com/EipID: eip-2vca3x1lxu96ropt66am5 122 sea.aliyun.com/Master0ID: i-2vc2ce4hs85f7ql6dqda 123 sea.aliyun.com/Master0InternalIP: 172.16.0.194 124 sea.aliyun.com/MasterIDs: i-2vc2ce4hs85f7ql6dqdb,i-2vc2ce4hs85f7ql6dqdc,i-2vc2ce4hs85f7ql6dqdai-2vcdx512qke5j0l5bksr 125 sea.aliyun.com/NodeIDs: i-2vc4eamks4zejf9n16tm,i-2vc4eamks4zejf9n16tn,i-2vc4eamks4zejf9n16tli-2vc1hagqsoc9ut8fp511 126 sea.aliyun.com/RegionID: cn-chengdu 127 sea.aliyun.com/SecurityGroupID: sg-2vcdx512qke5iqpx1iwx 128 sea.aliyun.com/VSwitchID: vsw-2vc55ufqm05qb8zdxux0h 129 sea.aliyun.com/VpcID: vpc-2vcj16rdif2upvgeowdu6 130 sea.aliyun.com/ZoneID: cn-chengdu-a 131 creationTimestamp: null 132 name: my-cluster 133 spec: 134 certSANS: 135 - aliyun-inc.com 136 - 10.0.0.2 137 image: kubernetes:v1.19.9 138 masters: 139 count: "4" 140 cpu: "4" 141 dataDisks: 142 - "100" 143 ipList: 144 - 172.16.0.194 145 - 172.16.0.196 146 - 172.16.0.195 147 - 172.16.0.200 148 memory: "4" 149 systemDisk: "100" 150 network: 151 cniName: calico 152 interface: eth0 153 podCIDR: 100.64.0.0/10 154 svcCIDR: 10.96.0.0/22 155 nodes: 156 count: "4" 157 cpu: "4" 158 dataDisks: 159 - "100" 160 ipList: 161 - 172.16.0.198 162 - 172.16.0.199 163 - 172.16.0.197 164 - 172.16.0.201 165 memory: "4" 166 systemDisk: "100" 167 provider: ALI_CLOUD 168 ssh: 169 passwd: Seadent123 170 pk: xxx 171 pkPasswd: xxx 172 user: root 173 status: {} 174 ``` 175 176 ### scale down: 177 178 #### 1. Modify /root/sealer/mycluster/Clusterfile file Masters, Nodes Count number: 179 180 ```shell 181 masters: 182 count: "3"//Masters reduced by one 183 cpu: "4" 184 dataDisks: 185 - "100" 186 ipList: 187 - 172.16.0.194 188 - 172.16.0.196 189 - 172.16.0.195//master iplist reduced by one 190 memory: "4" 191 systemDisk: "100" 192 network: 193 cniName: calico 194 interface: eth0 195 podCIDR: 100.64.0.0/10 196 svcCIDR: 10.96.0.0/22 197 nodes: 198 count: "3"//nodes reduced by one 199 cpu: "4" 200 dataDisks: 201 - "100" 202 ipList: 203 - 172.16.0.198 204 - 172.16.0.199 205 - 172.16.0.197//node iplist reduced by one 206 memory: "4" 207 systemDisk: "100" 208 ``` 209 210 #### 2. Execute sealer apply -f /root/.sealer/my-cluster/Clusterfile（Successfully executed）: 211 212 ```shell 213 [root@iZ2vc2ce4hs85f7asxhaipZ ~]# sealer apply -f /root/.sealer/my-cluster/Clusterfile 214 2021-06-03 15:03:50 [INFO] reconcile master instances success [172.16.0.194 172.16.0.196 172.16.0.195] 215 2021-06-03 15:03:51 [INFO] reconcile node instances success [172.16.0.198 172.16.0.199 172.16.0.197] 216 2021-06-03 15:03:51 [INFO] desired master 3, current master 4, desired nodes 3, current nodes 4 217 2021-06-03 15:03:51 [INFO] delete nodes [172.16.0.200 172.16.0.201] 218 2021-06-03 15:03:51 [INFO] scale the cluster success 219 [root@iZ2vc2ce4hs85f7asxhaipZ ~]# kubectl get nodes 220 NAME STATUS ROLES AGE VERSION 221 iz2vc2ce4hs85f7ql6dqdaz Ready master 39m v1.19.9 222 iz2vc2ce4hs85f7ql6dqdbz Ready master 37m v1.19.9 223 iz2vc2ce4hs85f7ql6dqdcz Ready master 38m v1.19.9 224 iz2vc4eamks4zejf9n16tlz Ready <none> 37m v1.19.9 225 iz2vc4eamks4zejf9n16tmz Ready <none> 37m v1.19.9 226 iz2vc4eamks4zejf9n16tnz Ready <none> 37m v1.19.9 227 ``` 228 229 #### 3. Check the Clusterfile 230 231 ```shell 232 [root@iZ2vc2ce4hs85f7asxhaipZ ~]# cat /root/.sealer/my-cluster/Clusterfile 233 apiVersion: zlink.aliyun.com/v1alpha1 234 kind: Cluster 235 metadata: 236 annotations: 237 ShouldBeDeleteInstancesIDs: "" 238 sea.aliyun.com/ClusterEIP: 47.108.132.245 239 sea.aliyun.com/EipID: eip-2vca3x1lxu96ropt66am5 240 sea.aliyun.com/Master0ID: i-2vc2ce4hs85f7ql6dqda 241 sea.aliyun.com/Master0InternalIP: 172.16.0.194 242 sea.aliyun.com/MasterIDs: i-2vc2ce4hs85f7ql6dqdb,i-2vc2ce4hs85f7ql6dqdc,i-2vc2ce4hs85f7ql6dqdai-2vcdx512qke5j0l5bksr 243 sea.aliyun.com/NodeIDs: i-2vc4eamks4zejf9n16tm,i-2vc4eamks4zejf9n16tn,i-2vc4eamks4zejf9n16tli-2vc1hagqsoc9ut8fp511 244 sea.aliyun.com/RegionID: cn-chengdu 245 sea.aliyun.com/SecurityGroupID: sg-2vcdx512qke5iqpx1iwx 246 sea.aliyun.com/VSwitchID: vsw-2vc55ufqm05qb8zdxux0h 247 sea.aliyun.com/VpcID: vpc-2vcj16rdif2upvgeowdu6 248 sea.aliyun.com/ZoneID: cn-chengdu-a 249 creationTimestamp: null 250 name: my-cluster 251 spec: 252 certSANS: 253 - aliyun-inc.com 254 - 10.0.0.2 255 image: kubernetes:v1.19.9 256 masters: 257 count: "3" 258 cpu: "4" 259 dataDisks: 260 - "100" 261 ipList: 262 - 172.16.0.194 263 - 172.16.0.196 264 - 172.16.0.195 265 memory: "4" 266 systemDisk: "100" 267 network: 268 cniName: calico 269 interface: eth0 270 podCIDR: 100.64.0.0/10 271 svcCIDR: 10.96.0.0/22 272 nodes: 273 count: "3" 274 cpu: "4" 275 dataDisks: 276 - "100" 277 ipList: 278 - 172.16.0.198 279 - 172.16.0.199 280 - 172.16.0.197 281 memory: "4" 282 systemDisk: "100" 283 provider: ALI_CLOUD 284 ssh: 285 passwd: Seadent123 286 pk: xxx 287 pkPasswd: xxx 288 user: root 289 status: {} 290 ``` 291 292 #### 4. Scale down node machine successfully deleted 293 294 ### 295 296 ### 297 298 ### Bare metal mode： 299 300 ### A.Executed on master0： 4C4G centos7.9 301 302 ### 1. init： 303 304 ```shell 305 1.wget https://github.com/alibaba/sealer/releases/download/v0.1.5-rc/sealer-0.0.0-linux-amd64.tar.g && tar zxvf sealer-0.0.0-linux-amd64.tar.gz && mv sealer /usr/local/bin/ 306 307 2.sealer run registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.19.9 --masters 172.16.0.165,172.16.0.166,172.16.0.167 --nodes 172.16.0.168,172.16.0.169 --passwd Sealer123 308 集群成功启动 309 [root@test-baremetal004 ~]# kubectl get nodes -A 310 NAME STATUS ROLES AGE VERSION 311 test-baremetal003 Ready master 12m v1.19.9 312 test-baremetal004 Ready master 13m v1.19.9 313 test-baremetal005 Ready master 12m v1.19.9 314 test-baremetal006 Ready <none> 11m v1.19.9 315 test-baremetal007 Ready <none> 11m v1.19.9 316 ``` 317 318 **Problem: The number of counts in /root/.sealer/my-cluster/Clusterfile does not change but does not affect the result** 319 320 **check Clusterfile：** 321 322 ```shell 323 [root@test-baremetal004 ~]# cat /root/.sealer/my-cluster/Clusterfile 324 apiVersion: zlink.aliyun.com/v1alpha1 325 kind: Cluster 326 metadata: 327 creationTimestamp: null 328 name: my-cluster 329 spec: 330 certSANS: 331 - aliyun-inc.com 332 - 10.0.0.2 333 image: registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.19.9 334 masters: 335 count: "3" 336 cpu: "4" 337 dataDisks: 338 - "100" 339 ipList: 340 - 172.16.0.165 341 - 172.16.0.166 342 - 172.16.0.167 343 ~ 344 nodes: 345 count: "3" //Count does not match the number of Iplists 346 cpu: "4" 347 dataDisks: 348 - "100" 349 ipList: 350 - 172.16.0.168 351 - 172.16.0.169 352 ~ 353 ``` 354 355 ### 2. Scale up 356 357 **Add node Masters: 172.16.0.170, 172.16.0.171; nodes: 172.16.0.172** 358 359 1. **Modify Clusterfile：** 360 361 ```shell 362 ~ 363 masters: 364 count: "3" 365 cpu: "4" 366 dataDisks: 367 - "100" 368 ipList: 369 - 172.16.0.165 370 - 172.16.0.166 371 - 172.16.0.167 372 - 172.16.0.170//masters adds one node 373 - 172.16.0.171 374 ~ 375 nodes: 376 count: "3" 377 cpu: "4" 378 dataDisks: 379 - "100" 380 ipList: 381 - 172.16.0.168 382 - 172.16.0.169 383 - 172.16.0.172//nodes adds one node 384 ``` 385 386 1. **Execute sealer apply -f /root/.sealer/my-cluster/Clusterfile （executed successfully)：** 387 388 ```shell 389 [root@test-baremetal004 ~]# kubectl get nodes -A 390 NAME STATUS ROLES AGE VERSION 391 test-baremetal001 Ready <none> 33s v1.19.9 392 test-baremetal002 Ready master 56s v1.19.9 393 test-baremetal003 Ready master 26m v1.19.9 394 test-baremetal004 Ready master 27m v1.19.9 395 test-baremetal005 Ready master 25m v1.19.9 396 test-baremetal006 Ready <none> 25m v1.19.9 397 test-baremetal007 Ready <none> 25m v1.19.9 398 test-baremetal009 Ready master 74s v1.19.9 399 ``` 400 401 ### 3. Scale down 402 403 1. **Modify Clusterfile**： 404 405 ```shell 406 masters: 407 count: "3" 408 cpu: "4" 409 dataDisks: 410 - "100" 411 ipList: 412 - 172.16.0.165 413 - 172.16.0.166 414 - 172.16.0.167 //master reduced by one 415 memory: "4" 416 systemDisk: "100" 417 network: 418 cniName: calico 419 interface: eth0 420 podCIDR: 100.64.0.0/10 421 svcCIDR: 10.96.0.0/22 422 nodes: 423 count: "3" 424 cpu: "4" 425 dataDisks: 426 - "100" 427 ipList: 428 - 172.16.0.168 429 - 172.16.0.169 //node reduced by one 430 ``` 431 432 2. **Execute sealer apply -f /root/.sealer/my-cluster/Clusterfile（executed successfully）：** 433 434 ```shell 435 [root@test-baremetal004 ~]# kubectl get nodes -A 436 NAME STATUS ROLES AGE VERSION 437 test-baremetal003 Ready master 29m v1.19.9 438 test-baremetal004 Ready master 29m v1.19.9 439 test-baremetal005 Ready master 28m v1.19.9 440 test-baremetal006 Ready <none> 27m v1.19.9 441 test-baremetal007 Ready <none> 27m v1.19.9 442 ``` 443 444 **Problem：** 445 446 **Execution Sealer delete-f Clusterfile directly after Scale Down. The two nodes deleted by Scale Down will still have the rootfs directory and the Seatutil tool ** 447 448 ```shell 449 [root@baremetal002 ~]# ssh root@172.16.0.170 450 root@172.16.0.160's password: 451 Last login: Thu Jun 3 17:08:30 2021 from 172.16.0.156 452 Welcome to Alibaba Cloud Elastic Compute Service ! 453 454 [root@iZ2vcdx512qke5k9yuwucfZ ~]# ls /var/lib/sealer/data/my-cluster/rootfs/ 455 bin cri images Kubefile README.md scripts TREE.md 456 cni etc kubeadm-join-config.yaml Metadata registry statics 457 ``` 458 459 ### 4. Delete 460 461 **Execute Sealer delete-f /root/.Sealer /my-cluster/Clusterfile (the last step reports an error, does not affect the result) :** 462 463 **All nodes were deleted successfully** 464 465 ```shell 466 /bin/sh: /var/lib/sealer/data/my-cluster/rootfs/scripts/clean.sh: No such file or directory 467 2021-06-04 15:53:54 [EROR] exec command failed Process exited with status 127 468 2021-06-04 15:53:54 [EROR] 172.16.0.165:exec /bin/sh -c /var/lib/sealer/data/my-cluster/rootfs/scripts/clean.sh failed, exec command failed 172.16.0.165 /bin/sh -c /var/lib/sealer/data/my-cluster/rootfs/scripts/clean.sh 469 2021-06-04 15:53:55 [INFO] [ssh][172.16.0.169] : rm -rf /var/lib/sealer/data/my-cluster/rootfs 470 2021-06-04 15:53:55 [INFO] [ssh][172.16.0.168] : rm -rf /var/lib/sealer/data/my-cluster/rootfs 471 2021-06-04 15:53:55 [INFO] [ssh][172.16.0.167] : rm -rf /var/lib/sealer/data/my-cluster/rootfs 472 2021-06-04 15:53:55 [INFO] [ssh][172.16.0.166] : rm -rf /var/lib/sealer/data/my-cluster/rootfs 473 2021-06-04 15:53:55 [EROR] unmountRootfs failed 474 ``` 475 476 **Error: /var/lib/sealer/data/my-cluster directory does not exist ** 477 478 **// Seautil tool still exists after delete ** 479 480 ### 5. Rerun after delete： 481 482 1. **Execute run registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.19.9 -- masters 172.16.0.165,172.16.0.166,172.16.0.167 -- nodes 172.16.0.168,172.16.0.169 --passwd (executed successfully）** 483 484 ```shell 485 [root@baremetal002 ~]# kubectl get nodes -A 486 NAME STATUS ROLES AGE VERSION 487 baremetal001 Ready <none> 46s v1.19.9 488 baremetal002 Ready master 2m21s v1.19.9 489 baremetal003 Ready master 98s v1.19.9 490 baremetal004 Ready <none> 45s v1.19.9 491 ``` 492 493 2. **Execute sealer delete -f /root/.sealer/my-cluster/Clusterfile** 494 495 ### B. Not executed in master0： 4C4G centos7.9 496 497 1. 498 499 | masters | 172.16.0.165,172.16.0.166,172.16.0.167 | 500 | ------- | -------------------------------------- | 501 | nodes | 172.16.0.168,172.16.0.169 | 502 503 **Execute on node 172.16.0.164:**： 504 505 **sealer run registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.19.9 --masters 172.16.0.165,172.16.0.166,172.16.0.167 --nodes 172.16.0.168,172.16.0.169 --passwd sealer123（executed successfully）** 506 507 ```shell 508 [root@test-baremetal008 ~]# kubectl get nodes -A 509 NAME STATUS ROLES AGE VERSION 510 test-baremetal003 Ready master 87s v1.19.9 511 test-baremetal004 Ready master 2m6s v1.19.9 512 test-baremetal005 Ready master 33s v1.19.9 513 test-baremetal006 Ready <none> 15s v1.19.9 514 test-baremetal007 Ready <none> 14s v1.19.9 515 ``` 516 517 ### 2. Scale up 518 519 **masters add 172.16.0.170，172.16.0.171** 520 521 **nodes add 172.16.0.172** 522 523 1. **Modify Clusterfile：** 524 525 ```shell 526 masters: 527 count: "3" 528 cpu: "4" 529 dataDisks: 530 - "100" 531 ipList: 532 - 172.16.0.165 533 - 172.16.0.166 534 - 172.16.0.167 535 - 172.16.0.170//masters adds two node 536 - 172.16.0.171 537 memory: "4" 538 systemDisk: "100" 539 network: 540 cniName: calico 541 interface: eth0 542 podCIDR: 100.64.0.0/10 543 svcCIDR: 10.96.0.0/22 544 nodes: 545 count: "3" 546 cpu: "4" 547 dataDisks: 548 - "100" 549 ipList: 550 - 172.16.0.168 551 - 172.16.0.169 552 - 172.16.0.172//nodes adds one node 553 ``` 554 555 2. **execute: sealer apply -f /root/.sealer/my-cluster/Clusterfile （executed successfully）** 556 557 ```shell 558 [root@test-baremetal008 ~]# kubectl get nodes -A 559 NAME STATUS ROLES AGE VERSION 560 test-baremetal001 Ready <none> 117s v1.19.9 561 test-baremetal002 Ready master 2m14s v1.19.9 562 test-baremetal003 Ready master 8m58s v1.19.9 563 test-baremetal004 Ready master 9m37s v1.19.9 564 test-baremetal005 Ready master 8m4s v1.19.9 565 test-baremetal006 Ready <none> 7m46s v1.19.9 566 test-baremetal007 Ready <none> 7m45s v1.19.9 567 test-baremetal009 Ready master 2m52s v1.19.9 568 ``` 569 570 ### 3. scale down 571 572 **Delete 172.16.0.170, 172.16.0.171 masters and 172.16.0.172 nodes** 573 574 1. **Modify Clusterfile**： 575 576 ```shell 577 masters: 578 count: "3" 579 cpu: "4" 580 dataDisks: 581 - "100" 582 ipList: 583 - 172.16.0.165 584 - 172.16.0.166 585 - 172.16.0.167 586 memory: "4" 587 systemDisk: "100" 588 network: 589 cniName: calico 590 interface: eth0 591 podCIDR: 100.64.0.0/10 592 svcCIDR: 10.96.0.0/22 593 nodes: 594 count: "3" 595 cpu: "4" 596 dataDisks: 597 - "100" 598 ipList: 599 - 172.16.0.168 600 - 172.16.0.169 601 ``` 602 603 2. **execute Sealer /root/.sealer/my-cluster/Clusterfile (Successfully executed）：** 604 605 ``` 606 [root@test-baremetal008 ~]# kubectl get nodes -A 607 NAME STATUS ROLES AGE VERSION 608 test-baremetal003 Ready master 16m v1.19.9 609 test-baremetal004 Ready master 16m v1.19.9 610 test-baremetal005 Ready master 15m v1.19.9 611 test-baremetal006 Ready <none> 14m v1.19.9 612 test-baremetal007 Ready <none> 14m v1.19.9 613 ``` 614 615 **Problem: (same as the problem in Master0 node execution scale down) :** 616 617 **The Scale Down node needs to remove rootfs as well as the Seautil tool** 618 619 ```shell 620 [root@test-baremetal008 ~]# ssh root@172.16.0.170 //Enter the node that has been scaled down 621 Welcome to Alibaba Cloud Elastic Compute Service ! 622 [root@test-baremetal009 ~]# ls /var/lib/sealer/data/my-cluster/rootfs/ 623 bin/ cri/ images/ 624 Kubefile README.md scripts/ TREE.md 625 cni/ etc/ kubeadm-join-config.yaml Metadata registry/ statics/ 626 ``` 627 628 ### 4. Delete 629 630 **execute ： sealer delete -f /root/.sealer/my-cluster/Clusterfile （Delete successfully）：** 631 632 ### 5. Rerun after delete： 633 634 **execute： sealer run registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.19.9 --masters 172.16.0.165,172.16.0.166,172.16.0.167 --nodes 172.16.0.168,172.16.0.169 --passwd Seadent23（executed successfully）** 635 636 ```shell 637 [root@test-baremetal008 ~]# kubectl get nodes -A 638 NAME STATUS ROLES AGE VERSION 639 test-baremetal003 Ready master 12m v1.19.9 640 test-baremetal004 Ready master 12m v1.19.9 641 test-baremetal005 Ready master 11m v1.19.9 642 test-baremetal006 Ready <none> 11m v1.19.9 643 test-baremetal007 Ready <none> 11m v1.19.9 644 ```