github.com/sealerio/sealer@v0.11.1-0.20240507115618-f4f89c5853ae/docs/design/master0-recovery.md (about)

     1  # Master0 recovery design
     2  
     3  ## Motivations
     4  
     5  Master0 plays an important role in sealer cluster, it hosts many important components such as sealer builtin registry,
     6  application images launched history and others cluster configuration files. if the master0 node is abnormal, a highly
     7  available solution is required to recover it on other master nodes.
     8  
     9  Currently, we have plans to save clusterfile to the cluster. while for built registry ,it is still deployed a single
    10  instance on master0 node. So the following document are mainly for built registry.
    11  
    12  ## Proposal
    13  
    14  ### For builtin registry
    15  
    16  * expose registry configuration to user thought clusterfile.
    17  * distribute all registry data split from cluster image registry directory to each deploy host, whether it is a cluster
    18    image or an application image.
    19  * save all registry configuration generated form sealer such as, basic auth information and https certificate.
    20  * save the boot history of each application image information such as tag,version and image id to cluster, which was
    21    launched through the sealer run-app.
    22  
    23  ## Implementation
    24  
    25  ### For configuration of builtin registry
    26  
    27  * expose below `Registry` struct to clusterfile which is contains configurations about local registry and remote
    28     registry.
    29  
    30  ```yaml
    31  type Registry struct {
    32    Domain        string        `json:"domain,omitempty"`
    33    Port          int           `json:"port,omitempty"`
    34    Username      string        `json:"username,omitempty"`
    35    Password      string        `json:"password,omitempty"`
    36    LocalRegistry LocalRegistry `json:"local_registry,omitempty"`
    37  }
    38    type LocalRegistry struct {
    39    // DeployHosts is the target host list that local registry will be deployed on.
    40    // if not set ,master0 will be the default value.
    41    DeployHosts []net.IP `json:"deploy_hosts,omitempty"`
    42    // InsecureMode indicated that whether the local registry is exposed in HTTPS.
    43    // if true sealer will generate default ssl cert.
    44    InsecureMode bool `json:"insecure_mode,omitempty,omitempty"`
    45  }
    46  ```
    47  
    48  * save registry configuration to cluster as configmap with name "sealer-registry" in namespace "kub-system".
    49  
    50  ```shell
    51  # must exist, this is core registry configuration
    52  /var/lib/sealer/data/my-cluster/rootfs/etc/registry.yml
    53  #if local registry is not booted on insecure mode
    54  /var/lib/sealer/data/my-cluster/rootfs/certs/sea.hub.crt
    55  /var/lib/sealer/data/my-cluster/rootfs/certs/sea.hub.key
    56  # if local registry is booted with basic auth.
    57  /var/lib/sealer/data/my-cluster/rootfs/etc/registry_htpasswd
    58  ```
    59  
    60  * save boot history of application image to cluster as configmap with name "sealer-apps" in namespace "kub-system".
    61  
    62  ```json
    63  {
    64    "type": "kube-installer",
    65    "applications": [
    66      {
    67        "name": "nginx",
    68        "image_id": "sha256:6bc6bc3015103b5a20d25f80cc833c0df5c5fccfdcecacfedcea296c616a534e",
    69        "type": "helm",
    70        "launch_cmd": "helm install nginx application/apps/nginx/",
    71        "version": "v1"
    72      },
    73      {
    74        "name": "dashboard",
    75        "image_id": "sha256:6bc6bc3015103b5a20d25f80cc833c0df5c5fccfdcecacfedcea296c616a534e",
    76        "type": "kube",
    77        "launch_cmd": "kubectl apply -f application/apps/dashboard/",
    78        "version": "v1"
    79      }
    80    ],
    81    "launch": {
    82      "app_names": [
    83        "nginx",
    84        "dashboard"
    85      ]
    86    }
    87  }
    88  ```
    89  
    90  ### For recovery of builtin registry
    91  
    92  Step 1: Pre-distribute all registry data to all deploy hosts
    93  
    94  * load registry configuration from cluster on new master0
    95  * dump registry configuration files from cluster to rootfs on new master0
    96  * launch builtin registry thought rootfs scripts and configuration files to recover registry service.
    97  * replace the latest registry ip address on all cluster nodes.
    98  
    99  Step 2: Post-distribute registry data on recovery host
   100  
   101  * load registry configuration from cluster on new master0
   102  * dump registry configuration files from cluster to rootfs on new master0
   103  * distribute all registry data to recovery host thought boot history loaded from cluster on new master0
   104  * launch builtin registry thought rootfs scripts and configuration files to recover registry service.
   105  * replace the latest registry ip address on all cluster nodes.
   106  
   107  Step 3: also need a sealer subcommand to do above recovery implementation
   108  
   109  ```shell
   110  sealer alpha recover master0 --host 192.168.1.100
   111  ```