github.com/dkerwin/nomad@v0.3.3-0.20160525181927-74554135514b/website/source/docs/upgrade/index.html.md (about) 1 --- 2 layout: "docs" 3 page_title: "Upgrading Nomad" 4 sidebar_current: "docs-upgrade-upgrading" 5 description: |- 6 Learn how to upgrade Nomad. 7 --- 8 9 # Upgrading Nomad 10 11 This page documents how to upgrade Nomad when a new version is released. 12 13 ~> **Upgrade Warning!** Both Nomad Clients and Servers are meant to be 14 long-running processes that maintain communication with each other. Nomad 15 Servers maintain quorum with other Servers and Clients are in constant 16 communication with Servers. As such, care should be taken to properly 17 upgrade Nomad to ensure minimal service disruption. Unsafe upgrades can 18 cause a service outage. 19 20 ## Upgrade Process 21 22 For upgrades we strive to ensure backwards compatibility. For most upgrades, the 23 process is as simple as upgrading the binary and restarting the service. 24 25 Prior to starting the upgrade please check the 26 [specific version details](/docs/upgrade/upgrade-specific.html) page as some 27 version differences may require specific steps. 28 29 At a high level we complete the following steps to upgrade Nomad: 30 31 * **Add the new version** 32 * **Check cluster health** 33 * **Remove the old version** 34 * **Check cluster health** 35 * **Upgrade clients** 36 37 ### 1. Add the new version to the existing cluster 38 39 Whether you are replacing the software in place on existing systems or bringing 40 up new hosts you should make changes incrementally, verifying cluster health at 41 each step of the upgrade 42 43 On a single server, install the new version of Nomad. You can do this by 44 joining a new server to the cluster or by replacing or upgrading the binary 45 locally and restarting the service. 46 47 ### 2. Check cluster health 48 49 Monitor the Nomad logs on the remaining nodes to check the new node has entered 50 the cluster correctly. 51 52 Run `nomad agent-info` on the new server and check that the `last_log_index` is 53 of a similar value to the other nodes. This step ensures that changes have been 54 replicated to the new node. 55 56 ``` 57 ubuntu@nomad-server-10-1-1-4:~$ nomad agent-info 58 nomad 59 bootstrap = false 60 known_regions = 1 61 leader = false 62 server = true 63 raft 64 applied_index = 53460 65 commit_index = 53460 66 fsm_pending = 0 67 last_contact = 54.512216ms 68 last_log_index = 53460 69 last_log_term = 1 70 last_snapshot_index = 49511 71 last_snapshot_term = 1 72 num_peers = 2 73 ... 74 ``` 75 76 Continue with the upgrades across the Server fleet making sure to do a single Nomad 77 server at a time. You can check state of the servers and clients with the 78 `nomad server-members` and `nomad node-status` commands which indicate state of the 79 nodes. 80 81 ### 3. Remove the old versions from servers 82 83 If you are doing an in place upgrade on existing servers this step is not 84 necessary as the version was changed in place. 85 86 If you are doing an upgrade by adding new servers and removing old servers 87 from the fleet you need to ensure that the server has left the fleet safely. 88 89 1. Stop the service on the existing host 90 2. On another server issue a `nomad server-members` and check the status, if 91 the server is now in a left state you are safe to continue. 92 3. If the server is not in a left state, issue a `nomad server-force-leave <server id>` 93 to remove the server from the cluster. 94 95 Monitor the logs of the other hosts in the Nomad cluster over this period. 96 97 ### 4. Check cluster health 98 99 Use the same actions in step #2 above to confirm cluster health. 100 101 ### 5. Upgrade clients 102 103 Following the successful upgrade of the servers you can now update your 104 clients using a similar process as the servers. If you wish to gracefully 105 move tasks on a client use the `nomad node-drain <node-id>` command to 106 gracefully migrate jobs to another client in the cluster. The `node-drain` 107 command prevents new tasks from being allocated to the client and begins 108 migrating existing allocations to another client. 109 110 ## Done! 111 112 You are now running the latest Nomad version. You can verify all 113 Clients joined by running `nomad node-status` and checking all the clients 114 are in a `ready` state. 115 116