github.com/minio/minio@v0.0.0-20240328213742-3f72439b8a27/docs/distributed/DECOMMISSION.md (about)

     1  # Decommissioning
     2  
     3  Decommissiong is a mechanism in MinIO to drain older pools (usually with old hardware) and migrate the content from such pools to a newer pools (usually better hardware). Decommissioning spreads the data across all pools - for example, if you decommission `pool1`, all the data from `pool1` spreads across `pool2` and `pool3`.
     4  
     5  ## Features
     6  
     7  - A pool in decommission still allows READ access to all its contents, newer WRITEs will automatically be scheduled to only pools not in decommission status.
     8  - All versioned buckets maintain the same order for "versions" for each object after being decommissioned to the other pools.
     9  - A pool interrupted during the decommission process, such as for a cluster restart, resumes from where it left off.
    10  
    11  ## How to decommission a pool
    12  
    13  ```
    14  λ mc admin decommission start alias/ http://minio{1...2}/data{1...4}
    15  ```
    16  
    17  ## Status decommissioning a pool
    18  
    19  ### Decommissioning without args lists all pools
    20  
    21  ```
    22  λ mc admin decommission status alias/
    23  ┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────┐
    24  │ ID  │ Pools                           │ Capacity                         │ Status │
    25  │ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Active │
    26  │ 2nd │ http://minio{3...4}/data{1...4} │ 329 GiB (used) / 421 GiB (total) │ Active │
    27  └─────┴─────────────────────────────────┴──────────────────────────────────┴────────┘
    28  ```
    29  
    30  ### Decommissioning status
    31  
    32  ```
    33  λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
    34  Decommissioning rate at 36 MiB/sec [4 TiB/50 TiB]
    35  Started: 1 minute ago
    36  ```
    37  
    38  Once it is **Complete**
    39  
    40  ```
    41  λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
    42  Decommission of pool http://minio{1...2}/data{1...4} is complete, you may now remove it from server command line
    43  ```
    44  
    45  ### A pool not under decommissioning will throw an error
    46  
    47  ```
    48  λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
    49  ERROR: This pool is not scheduled for decommissioning currently.
    50  ```
    51  
    52  ## Canceling a decommission
    53  
    54  Stop an on-going decommission in progress, mainly used in situations when the load may be too high and you may want to schedule the decommission at a later point in time.
    55  
    56  `mc admin decommission cancel` without an argument, lists out any on-going decommission in progress.
    57  
    58  ```
    59  λ mc admin decommission cancel alias/
    60  ┌─────┬─────────────────────────────────┬──────────────────────────────────┬──────────┐
    61  │ ID  │ Pools                           │ Capacity                         │ Status   │
    62  │ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining │
    63  └─────┴─────────────────────────────────┴──────────────────────────────────┴──────────┘
    64  ```
    65  
    66  > NOTE: Canceled decommission will not make the pool active again, since we might have  potentially partial namespace on the other pools, to avoid this scenario be absolutely sure to make decommissioning a planned well thought activity. This is not to be run on a daily basis.
    67  
    68  ```
    69  λ mc admin decommission cancel alias/ http://minio{1...2}/data{1...4}
    70  ┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────────────────┐
    71  │ ID  │ Pools                           │ Capacity                         │ Status             │
    72  │ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining(Canceled) │
    73  └─────┴─────────────────────────────────┴──────────────────────────────────┴────────────────────┘
    74  ```
    75  
    76  If the decommission process fails for any reason, the status indicates failed.
    77  
    78  ```
    79  λ mc admin decommission status alias/
    80  ┌─────┬─────────────────────────────────┬──────────────────────────────────┬──────────────────┐
    81  │ ID  │ Pools                           │ Capacity                         │ Status           │
    82  │ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining(Failed) │
    83  │ 2nd │ http://minio{3...4}/data{1...4} │ 329 GiB (used) / 421 GiB (total) │ Active           │
    84  └─────┴─────────────────────────────────┴──────────────────────────────────┴──────────────────┘
    85  ```
    86  
    87  ## Restart a canceled or failed decommission
    88  
    89  ```
    90  λ mc admin decommission start alias/ http://minio{1...2}/data{1...4}
    91  ```
    92  
    93  ## When decommission is 'Complete'
    94  
    95  Once decommission is complete, it will be indicated with *Complete* status.  *Complete* means that now you can now safely remove the first pool argument from the MinIO command line.
    96  
    97  ```
    98  λ mc admin decommission status alias/
    99  ┌─────┬─────────────────────────────────┬──────────────────────────────────┬──────────┐
   100  │ ID  │ Pools                           │ Capacity                         │ Status   │
   101  │ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Complete │
   102  │ 2nd │ http://minio{3...4}/data{1...4} │ 329 GiB (used) / 421 GiB (total) │ Active   │
   103  └─────┴─────────────────────────────────┴──────────────────────────────────┴──────────┘
   104  ```
   105  
   106  - On baremetal setups, if you have `MINIO_VOLUMES="http://minio{1...2}/data{1...4} http://minio{3...4}/data{1...4}"`, you can remove the first argument `http://minio{1...2}/data{1...4}` to update your `MINIO_VOLUMES` setting, then restart all the servers in the setup in parallel using `systemctl restart minio`.
   107  
   108  - On Kubernetes setups, the statefulset specification needs to be modified by changing the command line input for the MinIO container. Once the relevant changes are done, proceed to execute `kubectl apply -f statefulset.yaml`.
   109  
   110  - On Operator based MinIO deployments, you need to modify the `tenant.yaml` specification and modify the `pools:` section from two entries to a single entry. After making relevant changes, proceed to execute `kubectl apply -f tenant.yaml`.
   111  
   112  > Without a 'Complete' status any 'Active' or 'Draining' pool(s) are not allowed to be removed once configured.
   113  
   114  ## NOTE
   115  
   116  - Empty delete markers (such as for objects with no other successor versions) do not transition to the new pool to avoid creating empty metadata on the other pool(s). If you believe transitioning empty delete markers is required, open a GitHub issue.
   117  
   118  ## TODO
   119  
   120  - Richer progress UI is not present at the moment, this will be addressed in subsequent releases. Currently however a RATE of data transfer and usage increase is displayed via `mc`.
   121  - Transitioned Hot Tier's as pooled setups are not currently supported, attempting to decommission buckets with ILM Transition will be rejected by the server. This will be supported in future releases.
   122  - Embedded Console UI does not support Decommissioning through the UI yet. This will be supported in future releases.