github.com/greenboxal/deis@v1.12.1/docs/managing_deis/add_remove_host.rst (about) 1 :title: Adding/Removing Hosts 2 :description: Considerations for adding or removing Deis hosts. 3 4 .. _add_remove_host: 5 6 Adding/Removing Hosts 7 ===================== 8 9 Most Deis components handle new machines just fine. Care has to be taken when removing machines from 10 the cluster, however, since the deis-store components act as the backing store for all the 11 stateful data Deis needs to function properly. 12 13 Note that these instructions follow the Ceph documentation for `removing monitors`_ and `removing OSDs`_. 14 Should these instructions differ significantly from the Ceph documentation, the Ceph documentation 15 should be followed, and a PR to update this documentation would be much appreciated. 16 17 Since Ceph uses the Paxos algorithm, it is important to always have enough monitors in the cluster 18 to be able to achieve a majority: 1:1, 2:3, 3:4, 3:5, 4:6, etc. It is always preferable to add 19 a new node to the cluster before removing an old one, if possible. 20 21 This documentation will assume a running three-node Deis cluster. 22 We will add a fourth machine to the cluster, then remove the first machine. 23 24 Inspecting health 25 ----------------- 26 27 Before we begin, we should check the state of the Ceph cluster to be sure it's healthy. 28 To do this, we use ``deis-store-admin`` - see :ref:`using-store-admin`. 29 30 .. code-block:: console 31 32 core@deis-1 ~ $ nse deis-store-admin 33 root@deis-1:/# ceph -s 34 cluster 20038e38-4108-4e79-95d4-291d0eef2949 35 health HEALTH_OK 36 monmap e3: 3 mons at {deis-1=172.17.8.100:6789/0,deis-2=172.17.8.101:6789/0,deis-3=172.17.8.102:6789/0}, election epoch 16, quorum 0,1,2 deis-1,deis-2,deis-3 37 mdsmap e10: 1/1/1 up {0=deis-2=up:active}, 2 up:standby 38 osdmap e36: 3 osds: 3 up, 3 in 39 pgmap v2096: 1344 pgs, 12 pools, 369 MB data, 448 objects 40 24198 MB used, 23659 MB / 49206 MB avail 41 1344 active+clean 42 43 We see from the ``pgmap`` that we have 1344 placement groups, all of which are ``active+clean``. This is good! 44 45 Adding a node 46 ------------- 47 48 To add a new node to your Deis cluster, simply provision a new CoreOS machine with the same 49 etcd discovery URL specified in the cloud-config file. When the new machine comes up, it will join the etcd cluster. 50 You can confirm this with ``fleetctl list-machines``. 51 52 Since the store components are global units, they will be automatically started on the new node. 53 54 Once the new machine is running, we can inspect the Ceph cluster health again: 55 56 .. code-block:: console 57 58 root@deis-1:/# ceph -s 59 cluster 20038e38-4108-4e79-95d4-291d0eef2949 60 health HEALTH_WARN 4 pgs recovering; 7 pgs recovery_wait; 31 pgs stuck unclean; recovery 325/1353 objects degraded (24.021%); clock skew detected on mon.deis-4 61 monmap e4: 4 mons at {deis-1=172.17.8.100:6789/0,deis-2=172.17.8.101:6789/0,deis-3=172.17.8.102:6789/0,deis-4=172.17.8.103:6789/0}, election epoch 20, quorum 0,1,2,3 deis-1,deis-2,deis-3,deis-4 62 mdsmap e11: 1/1/1 up {0=deis-2=up:active}, 3 up:standby 63 osdmap e40: 4 osds: 4 up, 4 in 64 pgmap v2172: 1344 pgs, 12 pools, 370 MB data, 451 objects 65 29751 MB used, 34319 MB / 65608 MB avail 66 325/1353 objects degraded (24.021%) 67 88 active 68 7 active+recovery_wait 69 1245 active+clean 70 4 active+recovering 71 recovery io 2302 kB/s, 2 objects/s 72 client io 204 B/s wr, 0 op/s 73 74 Note that we are in a ``HEALTH_WARN`` state, and we have placement groups recovering. Ceph is 75 copying data to our new node. We can query the status of this until it completes. Then, we should 76 we something like: 77 78 .. code-block:: console 79 80 root@deis-1:/# ceph -s 81 cluster 20038e38-4108-4e79-95d4-291d0eef2949 82 health HEALTH_OK 83 monmap e4: 4 mons at {deis-1=172.17.8.100:6789/0,deis-2=172.17.8.101:6789/0,deis-3=172.17.8.102:6789/0,deis-4=172.17.8.103:6789/0}, election epoch 20, quorum 0,1,2,3 deis-1,deis-2,deis-3,deis-4 84 mdsmap e11: 1/1/1 up {0=deis-2=up:active}, 3 up:standby 85 osdmap e40: 4 osds: 4 up, 4 in 86 pgmap v2216: 1344 pgs, 12 pools, 372 MB data, 453 objects 87 29749 MB used, 34324 MB / 65608 MB avail 88 1344 active+clean 89 client io 409 B/s wr, 0 op/s 90 91 We're back in a ``HEALTH_OK``, and note the following: 92 93 .. code-block:: console 94 95 monmap e4: 4 mons at {deis-1=172.17.8.100:6789/0,deis-2=172.17.8.101:6789/0,deis-3=172.17.8.102:6789/0,deis-4=172.17.8.103:6789/0}, election epoch 20, quorum 0,1,2,3 deis-1,deis-2,deis-3,deis-4 96 mdsmap e11: 1/1/1 up {0=deis-2=up:active}, 3 up:standby 97 osdmap e40: 4 osds: 4 up, 4 in 98 99 We have 4 monitors, OSDs, and metadata servers. Hooray! 100 101 .. note:: 102 103 If you have applied the `custom firewall script`_ to your cluster, you will have to run this 104 script again and reboot your nodes for iptables to remove the duplicate rules. 105 106 Removing a node 107 --------------- 108 109 When removing a node from the cluster that runs a deis-store component, you'll need to tell Ceph 110 that the store services on this host will be leaving the cluster. 111 In this example we're going to remove the first node in our cluster, deis-1. 112 That machine has an IP address of ``172.17.8.100``. 113 114 .. _removing_an_osd: 115 116 Removing an OSD 117 ~~~~~~~~~~~~~~~ 118 119 Before we can tell Ceph to remove an OSD, we need the OSD ID. We can get this from etcd: 120 121 .. code-block:: console 122 123 core@deis-2 ~ $ etcdctl get /deis/store/osds/172.17.8.100 124 2 125 126 Note: In some cases, we may not know the IP or hostname or the machine we want to remove. 127 In these cases, we can use ``ceph osd tree`` to see the current state of the cluster. 128 This will list all the OSDs in the cluster, and report which ones are down. 129 130 Now that we have the OSD's ID, let's remove it. We'll need a shell in any store container 131 on any host in the cluster (except the one we're removing). In this example, I am on ``deis-2``. 132 133 .. code-block:: console 134 135 core@deis-2 ~ $ nse deis-store-admin 136 root@deis-2:/# ceph osd out 2 137 marked out osd.2. 138 139 This instructs Ceph to start relocating placement groups on that OSD to another host. We can watch this with ``ceph -w``: 140 141 .. code-block:: console 142 143 root@deis-2:/# ceph -w 144 cluster 20038e38-4108-4e79-95d4-291d0eef2949 145 health HEALTH_WARN 4 pgs recovery_wait; 151 pgs stuck unclean; recovery 654/1365 objects degraded (47.912%); clock skew detected on mon.deis-4 146 monmap e4: 4 mons at {deis-1=172.17.8.100:6789/0,deis-2=172.17.8.101:6789/0,deis-3=172.17.8.102:6789/0,deis-4=172.17.8.103:6789/0}, election epoch 20, quorum 0,1,2,3 deis-1,deis-2,deis-3,deis-4 147 mdsmap e11: 1/1/1 up {0=deis-2=up:active}, 3 up:standby 148 osdmap e42: 4 osds: 4 up, 3 in 149 pgmap v2259: 1344 pgs, 12 pools, 373 MB data, 455 objects 150 23295 MB used, 24762 MB / 49206 MB avail 151 654/1365 objects degraded (47.912%) 152 151 active 153 4 active+recovery_wait 154 1189 active+clean 155 recovery io 1417 kB/s, 1 objects/s 156 client io 113 B/s wr, 0 op/s 157 158 2014-11-04 06:45:07.940731 mon.0 [INF] pgmap v2260: 1344 pgs: 142 active, 3 active+recovery_wait, 1199 active+clean; 373 MB data, 23301 MB used, 24757 MB / 49206 MB avail; 619/1365 objects degraded (45.348%); 1724 kB/s, 0 keys/s, 1 objects/s recovering 159 2014-11-04 06:45:17.948788 mon.0 [INF] pgmap v2261: 1344 pgs: 141 active, 4 active+recovery_wait, 1199 active+clean; 373 MB data, 23301 MB used, 24757 MB / 49206 MB avail; 82 B/s rd, 0 op/s; 619/1365 objects degraded (45.348%); 843 kB/s, 0 keys/s, 0 objects/s recovering 160 2014-11-04 06:45:18.962420 mon.0 [INF] pgmap v2262: 1344 pgs: 140 active, 5 active+recovery_wait, 1199 active+clean; 373 MB data, 23318 MB used, 24740 MB / 49206 MB avail; 371 B/s rd, 0 B/s wr, 0 op/s; 618/1365 objects degraded (45.275%); 0 B/s, 0 keys/s, 0 objects/s recovering 161 2014-11-04 06:45:23.347089 mon.0 [INF] pgmap v2263: 1344 pgs: 130 active, 5 active+recovery_wait, 1209 active+clean; 373 MB data, 23331 MB used, 24727 MB / 49206 MB avail; 379 B/s rd, 0 B/s wr, 0 op/s; 572/1365 objects degraded (41.905%); 2323 kB/s, 0 keys/s, 4 objects/s recovering 162 2014-11-04 06:45:37.970125 mon.0 [INF] pgmap v2264: 1344 pgs: 129 active, 4 active+recovery_wait, 1211 active+clean; 373 MB data, 23336 MB used, 24722 MB / 49206 MB avail; 568/1365 objects degraded (41.612%); 659 kB/s, 2 keys/s, 1 objects/s recovering 163 2014-11-04 06:45:40.006110 mon.0 [INF] pgmap v2265: 1344 pgs: 129 active, 4 active+recovery_wait, 1211 active+clean; 373 MB data, 23336 MB used, 24722 MB / 49206 MB avail; 568/1365 objects degraded (41.612%); 11 B/s, 3 keys/s, 0 objects/s recovering 164 2014-11-04 06:45:43.034215 mon.0 [INF] pgmap v2266: 1344 pgs: 129 active, 4 active+recovery_wait, 1211 active+clean; 373 MB data, 23344 MB used, 24714 MB / 49206 MB avail; 1010 B/s wr, 0 op/s; 568/1365 objects degraded (41.612%) 165 2014-11-04 06:45:44.048059 mon.0 [INF] pgmap v2267: 1344 pgs: 129 active, 4 active+recovery_wait, 1211 active+clean; 373 MB data, 23344 MB used, 24714 MB / 49206 MB avail; 1766 B/s wr, 0 op/s; 568/1365 objects degraded (41.612%) 166 2014-11-04 06:45:48.366555 mon.0 [INF] pgmap v2268: 1344 pgs: 129 active, 4 active+recovery_wait, 1211 active+clean; 373 MB data, 23345 MB used, 24713 MB / 49206 MB avail; 576 B/s wr, 0 op/s; 568/1365 objects degraded (41.612%) 167 168 Eventually, the cluster will return to a clean state and will once again report ``HEALTH_OK``. 169 Then, we can stop the daemon. Since the store units are global units, we can't target a specific 170 one to stop. Instead, we log into the host machine and instruct Docker to stop the container. 171 172 Reminder: make sure you're logged into the machine you're removing from the cluster! 173 174 .. code-block:: console 175 176 core@deis-1 ~ $ docker stop deis-store-daemon 177 deis-store-daemon 178 179 Back inside a store container on ``deis-2``, we can finally remove the OSD: 180 181 .. code-block:: console 182 183 core@deis-2 ~ $ nse deis-store-admin 184 root@deis-2:/# ceph osd crush remove osd.2 185 removed item id 2 name 'osd.2' from crush map 186 root@deis-2:/# ceph auth del osd.2 187 updated 188 root@deis-2:/# ceph osd rm 2 189 removed osd.2 190 191 For cleanup, we should remove the OSD entry from etcd: 192 193 .. code-block:: console 194 195 core@deis-2 ~ $ etcdctl rm /deis/store/osds/172.17.8.100 196 197 That's it! If we inspect the health, we see that there are now 3 osds again, and all of our placement groups are ``active+clean``. 198 199 .. code-block:: console 200 201 core@deis-2 ~ $ nse deis-store-admin 202 root@deis-2:/# ceph -s 203 cluster 20038e38-4108-4e79-95d4-291d0eef2949 204 health HEALTH_OK 205 monmap e4: 4 mons at {deis-1=172.17.8.100:6789/0,deis-2=172.17.8.101:6789/0,deis-3=172.17.8.102:6789/0,deis-4=172.17.8.103:6789/0}, election epoch 20, quorum 0,1,2,3 deis-1,deis-2,deis-3,deis-4 206 mdsmap e11: 1/1/1 up {0=deis-2=up:active}, 3 up:standby 207 osdmap e46: 3 osds: 3 up, 3 in 208 pgmap v2338: 1344 pgs, 12 pools, 375 MB data, 458 objects 209 23596 MB used, 24465 MB / 49206 MB avail 210 1344 active+clean 211 client io 326 B/s wr, 0 op/s 212 213 Removing a monitor 214 ~~~~~~~~~~~~~~~~~~ 215 216 Removing a monitor is much easier. First, we remove the etcd entry so any clients that are using Ceph won't use the monitor for connecting: 217 218 .. code-block:: console 219 220 $ etcdctl rm /deis/store/hosts/172.17.8.100 221 222 Within 5 seconds, confd will run on all store clients and remove the monitor from the ``ceph.conf`` configuration file. 223 224 Next, we stop the container: 225 226 .. code-block:: console 227 228 core@deis-1 ~ $ docker stop deis-store-monitor 229 deis-store-monitor 230 231 232 Back on another host, we can again enter a store container and then remove this monitor: 233 234 .. code-block:: console 235 236 core@deis-2 ~ $ nse deis-store-admin 237 root@deis-2:/# ceph mon remove deis-1 238 removed mon.deis-1 at 172.17.8.100:6789/0, there are now 3 monitors 239 2014-11-04 06:57:59.712934 7f04bc942700 0 monclient: hunting for new mon 240 2014-11-04 06:57:59.712934 7f04bc942700 0 monclient: hunting for new mon 241 242 Note that there may be faults that follow - this is normal to see when a Ceph client is 243 unable to communicate with a monitor. The important line is that we see ``removed mon.deis-1 at 172.17.8.100:6789/0, there are now 3 monitors``. 244 245 Finally, let's check the health of the cluster: 246 247 .. code-block:: console 248 249 root@deis-2:/# ceph -s 250 cluster 20038e38-4108-4e79-95d4-291d0eef2949 251 health HEALTH_OK 252 monmap e5: 3 mons at {deis-2=172.17.8.101:6789/0,deis-3=172.17.8.102:6789/0,deis-4=172.17.8.103:6789/0}, election epoch 26, quorum 0,1,2 deis-2,deis-3,deis-4 253 mdsmap e17: 1/1/1 up {0=deis-4=up:active}, 3 up:standby 254 osdmap e47: 3 osds: 3 up, 3 in 255 pgmap v2359: 1344 pgs, 12 pools, 375 MB data, 458 objects 256 23605 MB used, 24455 MB / 49206 MB avail 257 1344 active+clean 258 client io 816 B/s wr, 0 op/s 259 260 We're done! 261 262 Removing a metadata server 263 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 264 265 Like the daemon, we'll just stop the Docker container for the metadata service. 266 267 Reminder: make sure you're logged into the machine you're removing from the cluster! 268 269 .. code-block:: console 270 271 core@deis-1 ~ $ docker stop deis-store-metadata 272 deis-store-metadata 273 274 This is actually all that's necessary. Ceph provides a ``ceph mds rm`` command, but has no 275 documentation for it. See: http://docs.ceph.com/docs/hammer/rados/operations/control/#mds-subsystem 276 277 Removing the host from etcd 278 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 279 280 The etcd cluster still has an entry for the host we've removed, so we'll need to remove this entry. 281 This can be achieved by making a request to the etcd API. See `remove machines`_ for details. 282 283 .. _`custom firewall script`: https://github.com/deis/deis/blob/master/contrib/util/custom-firewall.sh 284 .. _`remove machines`: https://coreos.com/docs/distributed-configuration/etcd-api/#remove-machines 285 .. _`removing monitors`: http://ceph.com/docs/hammer/rados/operations/add-or-rm-mons/#removing-monitors 286 .. _`removing OSDs`: http://docs.ceph.com/docs/hammer/rados/operations/add-or-rm-osds/#removing-osds-manual 287 288 Automatic Host Removal 289 ====================== 290 291 The ``contrib/user-data.example`` provides a unit which provides some experimental logic to clean-up a Deis 292 node's Ceph and etcd membership before reboot, shutdown or halt events. Currently, the procedure will 293 also be triggered if it is manually stopped via systemctl. 294 295 The unit requires that the optional ``deis-store-admin`` component is installed. 296 297 .. code-block:: console 298 299 root@deis-1:/# deisctl install store-admin 300 root@deis-1:/# deisctl start store-admin 301 302 To enable the feature you must enable and start the unit: 303 304 .. code-block:: console 305 306 root@deis-1:/# systemctl enable graceful-deis-shutdown 307 root@deis-1:/# systemctl start graceful-deis-shutdown 308 309 The unit is now active and will now be stopped ahead of the store components, Docker and etcd. 310 The script ``/opt/bin/graceful-shutdown.sh`` will determine if the Ceph cluster is healthy, and attempt 311 to remove this node from the Deis store components if there are greater than 3 Ceph nodes in the cluster.