github.com/cockroachdb/cockroach@v20.2.0-alpha.1+incompatible/docs/RFCS/20180411_finalize_cluster_upgrade_automatically.md (about)

     1  - Feature Name: Finalize cluster upgrades automatically
     2  - Status: in-progress
     3  - Start Date: 2018 4/11
     4  - Authors: Victor Chen
     5  - RFC PR: [#24377](https://github.com/cockroachdb/cockroach/pull/24377)
     6  - Cockroach Issue:
     7  [#23912](https://github.com/cockroachdb/cockroach/issues/23912),
     8  [#22686](https://github.com/cockroachdb/cockroach/issues/22686)
     9  
    10  ## Summary
    11  
    12  We will add an auto-finalization feature to help operators finalize their
    13  cluster upgrade process without manually running the set cluster version sql
    14  command.
    15  
    16  Meanwhile, we will also add a feature that allows operators to opt out of
    17  auto-finalization to preserve the ability to downgrade.
    18  
    19  
    20  ## Motivation
    21  In the past, in order to finalize the cluster upgrade process, operators must
    22  manually run
    23  
    24  ```sql
    25  SET CLUSTER SETTING version = crdb_internal.node_executable_version();
    26  ```
    27  
    28  However, this step is easy to forget and without it, we cannot unlock
    29  backwards-incompatible features. Therefore, it is necessary to add a version
    30  check mechanism that automatically runs the finalization command once knowing
    31  all the nodes in the cluster are running the new version.
    32  
    33  Because this finalization step is irreversible, we want to allow
    34  operators to make their own decisions on whether to use this auto-finalization
    35  feature or not. For example, operators who want to opt out of auto-upgrade from
    36  `version 2.0` to its next version should run:
    37  
    38  ```sql
    39  SET CLUSTER SETTING cluster.preserve_downgrade_option = '2.0';
    40  ```
    41  
    42  Please note that the opt-out feature is on a per version basis. That is to say,
    43  if operators want to keep use manual upgrade for `version 2.1`, they have to
    44  re-run the above command with `2.0` replaced with `2.1`. Otherwise, the cluster
    45  version will still be upgraded automatically.
    46  
    47  If the operators change their mind after running the manual upgrade command,
    48  they can run the following command to change it back to auto upgrade:
    49  
    50  ```sql
    51  RESET CLUSTER SETTING cluster.preserve_downgrade_option;
    52  ```
    53  
    54  Finally, trying to set the cluster setting version when operators have already
    55  set `cluster.preserve_downgrade_option` is not allowed. Operators have to reset
    56  the `cluster.preserve_downgrade_option` to allow auto upgrade to happen.
    57  
    58  
    59  ## Detailed Design
    60  
    61  ### Add a new setting `cluster.preserve_downgrade_option`
    62  
    63  The field should be added to `CLUSTER SETTING` in the format of:
    64  `cluster.preserve_downgrade_option = 2.0`
    65  
    66  By default, the value should be `""`, an empty string, to indicate we are always
    67  using auto upgrade. We validate that the value being stored in this field matches
    68  the current cluster version before storing it.
    69  
    70  ### On node startup, have a daemon that keeps running the following logic until exit:
    71  
    72  - **IF**:
    73    - All lives nodes are running the same version as cluster version.
    74  - **THEN**:
    75    - Exit.
    76  - **ELSE IF**:
    77    - The preserve downgrade option for the current cluster version is not set.
    78    <br>**AND**
    79    - Look at `crdb_internal.gossip_nodes` and verify that all live nodes are
    80    running the new version.
    81    <br>**AND**
    82    - Pull `NodeLivenessStatus` from liveness server and verify that all
    83    non-decommissioned nodes are alive (no missing nodes).
    84  - **THEN**:
    85    - Execute the following statement until success:
    86        - Do upgrade: `SET CLUSTER SETTING version = crdb_internal.node_executable_version();`.
    87    - Exit.
    88  
    89  
    90  ## Testing
    91  
    92  ### Roachtest `upgrade.go`:
    93  
    94  Test Steps covered by `upgrade.go`
    95  (sleep between each step for a certain amount of time if necessary):
    96  
    97  1. Start a cluster of `N` nodes running `v2.0.0`.
    98  
    99  2. Perform a rolling upgrade for node `1` - `N-1`. At every iteration, check
   100  that the cluster version is not upgraded.
   101  
   102  3. Stop node `N-1`.
   103  
   104  4. Perform an upgrade for node `N`, which was running `v2.0.0`.
   105  
   106  5. Check that the cluster version is not upgraded.
   107  
   108  6. Decommission node `N-2` (decommissioned nodes should not affect auto upgrade).
   109  
   110  7. Check cannot set `cluster.preserve_downgrade_option` to any value besides
   111  `2.0`, which is the current cluster version.
   112  
   113  8. Set `cluster.preserve_downgrade_option` to be `2.0`.
   114  
   115  9. Restart node `N-1`, which was previously force stopped.
   116  
   117  10. Check that the cluster version is not upgraded.
   118  
   119  11. Check cannot set the cluster setting version until `cluster.preserve_downgrade_option`
   120  is cleared.
   121  
   122  12. Reset `cluster.preserve_downgrade_option`.
   123  
   124  13. Check that the cluster version is upgraded to new version.
   125  
   126  14. Check that `cluster.preserve_downgrade_option` has been reset.
   127  
   128  ### Unit test `TestClusterVersionUpgrade`:
   129  
   130  `TestClusterVersionUpgrade` follows similar test steps as `upgrade.go`. Since 
   131  it serves as a quick check for CI, it's not as comprehensive as the roachtest.
   132  
   133  
   134  ## UI
   135  - If the user opt out of the auto upgrade, we should have a banner that alert
   136  users when all the nodes are running a higher version than the cluster version
   137  to instruct them to manually run the upgrade command.
   138  
   139  - If all live nodes are running the newest version but some nodes are down and
   140  not decommissioned, we should have a banner to alert operators to either revive
   141  or decommission the down nodes.
   142  
   143  - Let the operator know if the auto-upgrade is disabled or not at current version.
   144    - If `cluster.preserve_downgrade_option` is equal to `cluster.version`,
   145    auto-upgrade is disabled. Otherwise it's enabled.
   146  
   147  
   148  ## Drawbacks
   149  - Race condition
   150  [#24670](https://github.com/cockroachdb/cockroach/issues/24670) (future work):
   151    - Say an operator want to upgrade a cluster of 3 nodes from version 1.1 to
   152    2.0.
   153    - As soon as the operator just replaced the last node running v1.1 with a
   154    node running v2.0, the auto-finalization will start running the set cluster
   155    version command.
   156    - However, if right before the command get executed, the operator erroneously
   157    joined another node running 1.1 version. Then the command get executed and
   158    the cluster version is set to be 2.0, with new features enabled.
   159    - But the node running 1.1 version can not perform some of the 2.0 version
   160    operations and it will cause trouble.
   161  - Unreachable Node:
   162    - We don't guarantee the upgrade will always be successful if there're
   163    unreachable nodes.
   164    - If a node is unreachable, we won't ever upgrade because it could be unsafe
   165    if the unreachable node is running in a lower version.