github.com/NVIDIA/aistore@v1.3.23-0.20240517131212-7df6609be51d/docs/leave_cluster.md (about)

     1  ---
     2  layout: post
     3  title: LEAVE CLUSTER
     4  permalink: /docs/leave-cluster
     5  redirect_from:
     6   - /leave_cluster.md/
     7   - /docs/leave_cluster.md/
     8  ---
     9  
    10  **Note**: for the most recent updates on the topic of cluster membership and node lifecycle, please also check:
    11  
    12  * [Node lifecycle: maintenance mode, rebalance/rebuild, shutdown, decommission](/docs/lifecycle_node.md)
    13  
    14  Also, see related:
    15  
    16  * [Joining aistore cluster](join_cluster.md)
    17  * [Global rebalance](rebalance.md)
    18  * [CLI: `ais cluster` command](/docs/cli/cluster.md)
    19  * [Scripted integration tests](https://github.com/NVIDIA/aistore/tree/main/ais/test/scripts)
    20  
    21  ## Table of Contents
    22  
    23  - [Putting a node in maintenance](#putting-a-node-in-maintenance)
    24  - [Clearing maintenance state](#clearing-maintenance-state)
    25  - [Removing a node from a cluster](#removing-a-node-from-a-cluster)
    26  - [Interrupt node removal](#interrupt-node-removal)
    27  - [Checking removal status](#checking-removal-status)
    28  
    29  Sometimes a node has to be removed from a cluster - either temporarily (e.g., to perform maintenance tasks) or permanently. It is important to do the "removal" gracefully, via provided AIS APIs. In particular, if the node is a storage target, it contains user data that must be *rebalanced* to remaining clustered nodes.
    30  
    31  Two kinds of node removal are supported:
    32  
    33  - temporary removal, e.g. for node maintenance. In this case, the node remains in the cluster but it stops responding to client requests. Temporary removal can be done in two ways: `start-maintenance` keeps the node running, and `shutdown` stops the node;
    34  - permanent removal, e.g. node decommission. `decommission` disables a node and starts moving all its objects to other targets. When rebalancing finishes, the primary proxy automatically removes the node from the cluster. On leaving the cluster, the node erases its AIS metadata and optionally deletes all user data.
    35  
    36  ### Putting a node in maintenance
    37  
    38  ![Put a node in maintenance](images/maintenance.png)
    39  
    40  To temporarily take a node out of the cluster, put a node in maintenance (CLI). Nodes undergoing maintenance remain in the cluster map, as shown in the above diagram.
    41  
    42  In general, when storage targets leave (or join) the cluster, the current *primary* (leader) proxy transactionally creates the *next* updated version of the cluster map and synchronizes the new map across the entire cluster so that each and every node gets the new version. 
    43  This results in each AIS target starting to traverse its locally stored content, recomputing object locations, and sending at least some of the objects to their respective *new* locations.
    44  Object migration is then carried out via an intra-cluster optimized [communication mechanism](/transport/README.md) and over a separate [physical or logical network](/cmn/network.go), if provisioned.
    45  
    46  For more details about the rebalancing process, see [REBALANCE](/docs/rebalance.md).
    47  
    48  ```console
    49  $ ais cluster add-remove-nodes start-maintenance 59262t8087
    50  Node "59262t8087" is in maintenance mode
    51  Started rebalance "g1", use 'ais show job xaction g1' to monitor the progress
    52  ```
    53  
    54  Alternatively, you can shut down the node to stop all AIS services on the node after putting it in maintenance mode:
    55  
    56  ```console
    57  $ ais cluster add-remove-nodes shutdown 59262t8087
    58  Node "59262t8087" is in maintenance mode
    59  Started rebalance "g1", use 'ais show job xaction g1' to monitor the progress
    60  ```
    61  
    62  If the node is a target, after a quick preparation, the cluster will rebalance. When the rebalance finishes, it is safe to turn the node off.
    63  
    64  If the node does not contain any important data, you can skip rebalancing with `--no-rebalance`, so the node is safe to switch off shortly after putting it in maintenance:
    65  
    66  ```console
    67  $ ais cluster add-remove-nodes start-maintenance 59262t8087 --no-rebalance
    68  Node "59262t8087" is in maintenance
    69  ```
    70  
    71  ### Clearing maintenance state
    72  
    73  Once a node is put in maintenance mode, the cluster keeps it in this state until you notify the cluster that the node is ready to use.
    74  If the node was shut down, you have to restart or power it on and wait until the node registers at the primary proxy beforehand.
    75  After getting the notification, the cluster clears the maintenance state and starts rebalance:
    76  
    77  ```console
    78  $ ais cluster add-remove-nodes stop-maintenance 59262t8087
    79  Node "59262t8087" maintenance stopped
    80  Started rebalance "g3", use 'ais show job xaction g3' to monitor the progress
    81  ```
    82  
    83  To skip automatic rebalance, provide flag `--no-rebalance`.
    84  It is *recommended* to keep automatic rebalance running automatically, but these are some cases in which it is *safe to skip rebalancing*:
    85  
    86  - all buckets are empty
    87  - maintenance was started with `--no-rebalance` and no object was added or updated during maintenance
    88  - objects can be refetched from remote sources. E.g, all buckets are remote AIS, HTTP or Cloud ones. In this case, targets redownload missing objects. This can cost extra money for Cloud traffic
    89  - you are going to stop maintenance for more than 1 node. So, all nodes except the last one are back to the cluster with the flag `--no-rebalance`, and the last node starts the automatic rebalance
    90  
    91  ### Removing a node from a cluster
    92  
    93  ![Decommission a node](images/decommission.png)
    94  
    95  To completely remove the node from the cluster, decommission the node (CLI):
    96  
    97  ```console
    98  $ ais cluster add-remove-nodes decommission 59262t8087
    99  Node "59262t8087" is in maintenance
   100  Started rebalance "g1", use 'ais show job xaction g1' to monitor the progress
   101  ```
   102  
   103  When the rebalance finishes, the cluster removes the node automatically from the list.
   104  On unregistering, the node erases its AIS metadata.
   105  Skipping rebalance runs quick preparations and removes the node from the cluster immediately:
   106  
   107  ```console
   108  $ ais cluster add-remove-nodes decommission --no-rebalance 59262t8087
   109  Node "59262t8087" removed from the cluster
   110  ```
   111  
   112  Note that `decommission` cleans up all AIS metadata and stops the node. `shutdown` only stops AIS services.
   113  If the node is a target, the node will be shut down after the rebalance has finished. Otherwise, if the node is a proxy, the node will shut down immediately.
   114  
   115  ```console
   116  $ ais cluster add-remove-nodes shutdown 59262t8087
   117  Node "59262t8087" is being shut down
   118  Started rebalance "g1", use 'ais show job xaction g1' to monitor the progress
   119  ```
   120  
   121  Note that you can interrupt a node removal with rebalance skipped. If you remove a node by mistake, you have to join it manually with `ais cluster add-remove-nodes join` command.
   122  
   123  ### Interrupt node removal
   124  
   125  ![Interrupt node removal](images/decommission_abort.png)
   126  
   127  While rebalance is running, you can interrupt the node removal to get the node back to the cluster.
   128  Rebalance starts automatically when a node is a target and flag `--no-rebalance` is not set after the node is registered at the cluster and the cluster clears the node's maintenance state.
   129  
   130  ```console
   131  $ ais cluster add-remove-nodes stop-maintenance 59262t8087
   132  Node "59262t8087" maintenance stopped
   133  Started rebalance "g3", use 'ais show job xaction g3' to monitor the progress
   134  ```
   135  
   136  The node starts accepting all the requests after joining the cluster and after the cluster clears node's maintenance state. You do not have to wait until the rebalance is done.
   137  
   138  ### Checking removal status
   139  
   140  Putting a node in maintenance does NOT automatically power off the node.
   141  
   142  AIS only runs a **rebalance** when a node is in maintenance mode. Manually check the cluster health (`show cluster target`) to ensure that it is safe to turn the node off. Besides checking xaction progress, the removal status can be monitored with `ais cluster status`.
   143  
   144  In the example below it is safe to turn off the node (the column `REBALANCE` states that the rebalance has already finished and the node is labeled `maintenance`):
   145  
   146  ```console
   147  $ ais show cluster target
   148  TARGET           MEM USED %      MEM AVAIL       CAP USED %      CAP AVAIL       CPU USED %      REBALANCE    UPTIME  STATUS
   149  59262t8087       0.13%           31.28GiB        16%             2.435TiB        0.00%           finished     31m     maintenance
   150  93683t8084       0.13%           31.28GiB        16%             2.435TiB        0.12%           finished     31m     online
   151  ```
   152  
   153  For decommissioning nodes, the status looks this while the rebalance is running:
   154  
   155  ```console
   156  $ ais show cluster target
   157  TARGET           MEM USED %      MEM AVAIL       CAP USED %      CAP AVAIL       CPU USED %      REBALANCE    UPTIME  STATUS
   158  59262t8087       0.13%           31.28GiB        16%             2.435TiB        0.00%           running      31m     decommission
   159  93683t8084       0.13%           31.28GiB        16%             2.435TiB        0.12%           running      31m     online
   160  ```
   161  
   162  On finishing the rebalance, the primary proxy removes the node automatically:
   163  
   164  ```console
   165  $ ais show cluster target
   166  TARGET           MEM USED %      MEM AVAIL       CAP USED %      CAP AVAIL       CPU USED %      REBALANCE    UPTIME
   167  93683t8084       0.13%           31.28GiB        16%             2.435TiB        0.12%           running      31m
   168  ```