github.com/dkerwin/nomad@v0.3.3-0.20160525181927-74554135514b/website/source/docs/upgrade/index.html.md (about)

     1  ---
     2  layout: "docs"
     3  page_title: "Upgrading Nomad"
     4  sidebar_current: "docs-upgrade-upgrading"
     5  description: |-
     6    Learn how to upgrade Nomad.
     7  ---
     8  
     9  # Upgrading Nomad
    10  
    11  This page documents how to upgrade Nomad when a new version is released.
    12  
    13  ~> **Upgrade Warning!** Both Nomad Clients and Servers are meant to be 
    14  long-running processes that maintain communication with each other. Nomad 
    15  Servers maintain quorum with other Servers and Clients are in constant 
    16  communication with Servers. As such, care should be taken to properly 
    17  upgrade Nomad to ensure minimal service disruption. Unsafe upgrades can
    18  cause a service outage.
    19  
    20  ## Upgrade Process
    21  
    22  For upgrades we strive to ensure backwards compatibility. For most upgrades, the
    23  process is as simple as upgrading the binary and restarting the service.
    24  
    25  Prior to starting the upgrade please check the 
    26  [specific version details](/docs/upgrade/upgrade-specific.html) page as some 
    27  version differences may require specific steps.
    28  
    29  At a high level we complete the following steps to upgrade Nomad:
    30  
    31  * **Add the new version**
    32  * **Check cluster health**
    33  * **Remove the old version**
    34  * **Check cluster health**
    35  * **Upgrade clients**
    36  
    37  ### 1. Add the new version to the existing cluster
    38  
    39  Whether you are replacing the software in place on existing systems or bringing
    40  up new hosts you should make changes incrementally, verifying cluster health at
    41  each step of the upgrade  
    42  
    43  On a single server, install the new version of Nomad.  You can do this by
    44  joining a new server to the cluster or by replacing or upgrading the binary
    45  locally and restarting the service.
    46  
    47  ### 2. Check cluster health
    48  
    49  Monitor the Nomad logs on the remaining nodes to check the new node has entered
    50  the cluster correctly.
    51  
    52  Run `nomad agent-info` on the new server and check that the `last_log_index` is
    53  of a similar value to the other nodes.  This step ensures that changes have been
    54  replicated to the new node.
    55  
    56  ```
    57  ubuntu@nomad-server-10-1-1-4:~$ nomad agent-info
    58  nomad
    59    bootstrap = false
    60    known_regions = 1
    61    leader = false
    62    server = true
    63  raft
    64    applied_index = 53460
    65    commit_index = 53460
    66    fsm_pending = 0
    67    last_contact = 54.512216ms
    68    last_log_index = 53460
    69    last_log_term = 1
    70    last_snapshot_index = 49511
    71    last_snapshot_term = 1
    72    num_peers = 2
    73  ...
    74  ```
    75  
    76  Continue with the upgrades across the Server fleet making sure to do a single Nomad
    77  server at a time.  You can check state of the servers and clients with the
    78  `nomad server-members` and `nomad node-status` commands which indicate state of the
    79  nodes.
    80  
    81  ### 3. Remove the old versions from servers
    82  
    83  If you are doing an in place upgrade on existing servers this step is not
    84  necessary as the version was changed in place.
    85  
    86  If you are doing an upgrade by adding new servers and removing old servers
    87  from the fleet you need to ensure that the server has left the fleet safely.
    88  
    89  1. Stop the service on the existing host
    90  2. On another server issue a `nomad server-members` and check the status, if 
    91  the server is now in a left state you are safe to continue.
    92  3. If the server is not in a left state, issue a `nomad server-force-leave <server id>`
    93  to remove the server from the cluster.
    94  
    95  Monitor the logs of the other hosts in the Nomad cluster over this period.
    96  
    97  ### 4. Check cluster health
    98  
    99  Use the same actions in step #2 above to confirm cluster health.
   100  
   101  ### 5. Upgrade clients
   102  
   103  Following the successful upgrade of the servers you can now update your
   104  clients using a similar process as the servers.  If you wish to gracefully
   105  move tasks on a client use the `nomad node-drain <node-id>` command to 
   106  gracefully migrate jobs to another client in the cluster.  The `node-drain` 
   107  command prevents new tasks from being allocated to the client and begins 
   108  migrating existing allocations to another client.
   109  
   110  ## Done! 
   111  
   112  You are now running the latest Nomad version. You can verify all
   113  Clients joined by running `nomad node-status` and checking all the clients
   114  are in a `ready` state.
   115  
   116