github.com/brahmaroutu/docker@v1.2.1-0.20160809185609-eb28dde01f16/docs/swarm/admin_guide.md (about) 1 <!--[metadata]> 2 +++ 3 aliases = [ 4 "/engine/swarm/manager-administration-guide/" 5 ] 6 title = "Swarm administration guide" 7 description = "Manager administration guide" 8 keywords = ["docker, container, swarm, manager, raft"] 9 [menu.main] 10 identifier="manager_admin_guide" 11 parent="engine_swarm" 12 weight="20" 13 +++ 14 <![end-metadata]--> 15 16 # Administer and maintain a swarm of Docker Engines 17 18 When you run a swarm of Docker Engines, **manager nodes** are the key components 19 for managing the swarm and storing the swarm state. It is important to 20 understand some key features of manager nodes in order to properly deploy and 21 maintain the swarm. 22 23 This article covers the following swarm administration tasks: 24 25 * [Using a static IP for manager node advertise address](#use-a-static-ip-for-manager-node-advertise-address) 26 * [Adding manager nodes for fault tolerance](#add-manager-nodes-for-fault-tolerance) 27 * [Distributing manager nodes](#distribute-manager-nodes) 28 * [Running manager-only nodes](#run-manager-only-nodes) 29 * [Backing up the swarm state](#back-up-the-swarm-state) 30 * [Monitoring the swarm health](#monitor-swarm-health) 31 * [Troubleshooting a manager node](#troubleshoot-a-manager-node) 32 * [Forcefully removing a node](#force-remove-a-node) 33 * [Recovering from disaster](#recover-from-disaster) 34 35 Refer to [How swarm mode nodes work](how-swarm-mode-works/nodes.md) 36 for a brief overview of Docker Swarm mode and the difference between manager and 37 worker nodes. 38 39 ## Operating manager nodes in a swarm 40 41 Swarm manager nodes use the [Raft Consensus Algorithm](raft.md) to manage the 42 swarm state. You only need to understand some general concepts of Raft in 43 order to manage a swarm. 44 45 There is no limit on the number of manager nodes. The decision about how many 46 manager nodes to implement is a trade-off between performance and 47 fault-tolerance. Adding manager nodes to a swarm makes the swarm more 48 fault-tolerant. However, additional manager nodes reduce write performance 49 because more nodes must acknowledge proposals to update the swarm state. 50 This means more network round-trip traffic. 51 52 Raft requires a majority of managers, also called a quorum, to agree on proposed 53 updates to the swarm. A quorum of managers must also agree on node additions 54 and removals. Membership operations are subject to the same constraints as state 55 replication. 56 57 ## Use a static IP for manager node advertise address 58 59 When initiating a swarm, you have to specify the `--advertise-addr` flag to 60 advertise your address to other manager nodes in the swarm. For more 61 information, see [Run Docker Engine in swarm mode](swarm-mode.md#configure-the-advertise-address). Because manager nodes are 62 meant to be a stable component of the infrastructure, you should use a *fixed 63 IP address* for the advertise address to prevent the swarm from becoming 64 unstable on machine reboot. 65 66 If the whole swarm restarts and every manager node subsequently gets a new IP 67 address, there is no way for any node to contact an existing manager. Therefore 68 the swarm is hung while nodes to contact one another at their old IP addresses. 69 70 Dynamic IP addresses are OK for worker nodes. 71 72 ## Add manager nodes for fault tolerance 73 74 You should maintain an odd number of managers in the swarm to support manager 75 node failures. Having an odd number of managers ensures that during a network 76 partition, there is a higher chance that a quorum remains available to process 77 requests if the network is partitioned into two sets. Keeping a quorum is not 78 guaranteed if you encounter more than two network partitions. 79 80 | Swarm Size | Majority | Fault Tolerance | 81 |:------------:|:----------:|:-----------------:| 82 | 1 | 1 | 0 | 83 | 2 | 2 | 0 | 84 | **3** | 2 | **1** | 85 | 4 | 3 | 1 | 86 | **5** | 3 | **2** | 87 | 6 | 4 | 2 | 88 | **7** | 4 | **3** | 89 | 8 | 5 | 3 | 90 | **9** | 5 | **4** | 91 92 For example, in a swarm with *5 nodes*, if you lose *3 nodes*, you don't have a 93 quorum. Therefore you can't add or remove nodes until you recover one of the 94 unavailable manager nodes or recover the swarm with disaster recovery 95 commands. See [Recover from disaster](#recover-from-disaster). 96 97 While it is possible to scale a swarm down to a single manager node, it is 98 impossible to demote the last manager node. This ensures you maintain access to 99 the swarm and that the swarm can still process requests. Scaling down to a 100 single manager is an unsafe operation and is not recommended. If 101 the last node leaves the swarm unexpetedly during the demote operation, the 102 swarm will become unavailable until you reboot the node or restart with 103 `--force-new-cluster`. 104 105 You manage swarm membership with the `docker swarm` and `docker node` 106 subsystems. Refer to [Add nodes to a swarm](join-nodes.md) for more information 107 on how to add worker nodes and promote a worker node to be a manager. 108 109 ## Distribute manager nodes 110 111 In addition to maintaining an odd number of manager nodes, pay attention to 112 datacenter topology when placing managers. For optimal fault-tolerance, distribute 113 manager nodes across a minimum of 3 availability-zones to support failures of an 114 entire set of machines or common maintenance scenarios. If you suffer a failure 115 in any of those zones, the swarm should maintain a quorum of manager nodes 116 available to process requests and rebalance workloads. 117 118 | Swarm manager nodes | Repartition (on 3 Availability zones) | 119 |:-------------------:|:--------------------------------------:| 120 | 3 | 1-1-1 | 121 | 5 | 2-2-1 | 122 | 7 | 3-2-2 | 123 | 9 | 3-3-3 | 124 125 ## Run manager-only nodes 126 127 By default manager nodes also act as a worker nodes. This means the scheduler 128 can assign tasks to a manager node. For small and non-critical swarms 129 assigning tasks to managers is relatively low-risk as long as you schedule 130 services using **resource constraints** for *cpu* and *memory*. 131 132 However, because manager nodes use the Raft consensus algorithm to replicate data 133 in a consistent way, they are sensitive to resource starvation. You should 134 isolate managers in your swarm from processes that might block swarm 135 operations like swarm heartbeat or leader elections. 136 137 To avoid interference with manager node operation, you can drain manager nodes 138 to make them unavailable as worker nodes: 139 140 ```bash 141 docker node update --availability drain <NODE> 142 ``` 143 144 When you drain a node, the scheduler reassigns any tasks running on the node to 145 other available worker nodes in the swarm. It also prevents the scheduler from 146 assigning tasks to the node. 147 148 ## Back up the swarm state 149 150 Docker manager nodes store the swarm state and manager logs in the following 151 directory: 152 153 ```bash 154 /var/lib/docker/swarm/raft 155 ``` 156 157 Back up the `raft` data directory often so that you can use it in case of 158 [disaster recovery](#recover-from-disaster). Then you can take the `raft` 159 directory of one of the manager nodes to restore to a new swarm. 160 161 ## Monitor swarm health 162 163 You can monitor the health of manager nodes by querying the docker `nodes` API 164 in JSON format through the `/nodes` HTTP endpoint. Refer to the [nodes API documentation](../reference/api/docker_remote_api_v1.24.md#36-nodes) 165 for more information. 166 167 From the command line, run `docker node inspect <id-node>` to query the nodes. 168 For instance, to query the reachability of the node as a manager: 169 170 ```bash 171 docker node inspect manager1 --format "{{ .ManagerStatus.Reachability }}" 172 reachable 173 ``` 174 175 To query the status of the node as a worker that accept tasks: 176 177 ```bash 178 docker node inspect manager1 --format "{{ .Status.State }}" 179 ready 180 ``` 181 182 From those commands, we can see that `manager1` is both at the status 183 `reachable` as a manager and `ready` as a worker. 184 185 An `unreachable` health status means that this particular manager node is unreachable 186 from other manager nodes. In this case you need to take action to restore the unreachable 187 manager: 188 189 - Restart the daemon and see if the manager comes back as reachable. 190 - Reboot the machine. 191 - If neither restarting or rebooting work, you should add another manager node or promote a worker to be a manager node. You also need to cleanly remove the failed node entry from the manager set with `docker node demote <NODE>` and `docker node rm <id-node>`. 192 193 Alternatively you can also get an overview of the swarm health from a manager 194 node with `docker node ls`: 195 196 ```bash 197 198 docker node ls 199 ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS 200 1mhtdwhvsgr3c26xxbnzdc3yp node05 Accepted Ready Active 201 516pacagkqp2xc3fk9t1dhjor node02 Accepted Ready Active Reachable 202 9ifojw8of78kkusuc4a6c23fx * node01 Accepted Ready Active Leader 203 ax11wdpwrrb6db3mfjydscgk7 node04 Accepted Ready Active 204 bb1nrq2cswhtbg4mrsqnlx1ck node03 Accepted Ready Active Reachable 205 di9wxgz8dtuh9d2hn089ecqkf node06 Accepted Ready Active 206 ``` 207 208 ## Troubleshoot a manager node 209 210 You should never restart a manager node by copying the `raft` directory from another node. The data directory is unique to a node ID. A node can only use a node ID once to join the swarm. The node ID space should be globally unique. 211 212 To cleanly re-join a manager node to a cluster: 213 214 1. To demote the node to a worker, run `docker node demote <NODE>`. 215 2. To remove the node from the swarm, run `docker node rm <NODE>`. 216 3. Re-join the node to the swarm with a fresh state using `docker swarm join`. 217 218 For more information on joining a manager node to a swarm, refer to 219 [Join nodes to a swarm](join-nodes.md). 220 221 ## Force remove a node 222 223 In most cases, you should shut down a node before removing it from a swarm with the `docker node rm` command. If a node becomes unreachable, unresponsive, or compromised you can forcefully remove the node without shutting it down by passing the `--force` flag. For instance, if `node9` becomes compromised: 224 225 <!-- bash hint breaks block quote --> 226 ``` 227 $ docker node rm node9 228 229 Error response from daemon: rpc error: code = 9 desc = node node9 is not down and can't be removed 230 231 $ docker node rm --force node9 232 233 Node node9 removed from swarm 234 ``` 235 236 Before you forcefully remove a manager node, you must first demote it to the 237 worker role. Make sure that you always have an odd number of manager nodes if 238 you demote or remove a manager 239 240 ## Recover from disaster 241 242 Swarm is resilient to failures and the swarm can recover from any number 243 of temporary node failures (machine reboots or crash with restart). 244 245 In a swarm of `N` managers, there must be a quorum of manager nodes greater than 246 50% of the total number of managers (or `(N/2)+1`) in order for the swarm to 247 process requests and remain available. This means the swarm can tolerate up to 248 `(N-1)/2` permanent failures beyond which requests involving swarm management 249 cannot be processed. These types of failures include data corruption or hardware 250 failures. 251 252 Even if you follow the guidelines here, it is possible that you can lose a 253 quorum of manager nodes. If you can't recover the quorum by conventional 254 means such as restarting faulty nodes, you can recover the swarm by running 255 `docker swarm init --force-new-cluster` on a manager node. 256 257 ```bash 258 # From the node to recover 259 docker swarm init --force-new-cluster --advertise-addr node01:2377 260 ``` 261 262 The `--force-new-cluster` flag puts the Docker Engine into swarm mode as a 263 manager node of a single-node swarm. It discards swarm membership information 264 that existed before the loss of the quorum but it retains data necessary to the 265 Swarm such as services, tasks and the list of worker nodes.