github.com/openshift/installer@v1.4.17/docs/user/agent/mce-integration.md (about)

     1  # MultiCluster Engine integration
     2  
     3  This document describes how to install the multicluster engine for Kubernetes operator (mce) and how to deploy the cluster zero (hub) using the agent-based installer for Openshift.
     4  The procedure is partially automated, and it will require some manual steps after the initial cluster deployment.
     5  
     6  ## Installing while disconnected from the greater internet
     7  
     8  ### Environment Pre-requisites
     9  
    10  [Create a local image registry in your environment.](https://docs.openshift.com/container-platform/4.11/installing/disconnected_install/index.html) Make note of the DNS name and port of your local registry.
    11  
    12  Download the [oc mirror](https://github.com/openshift/oc-mirror) tool.
    13  
    14  #### Agent Installer pre-requisites
    15  
    16  For the complete pre-requisites description, please check section 1.2.1 in [[1]](#1).
    17  
    18  Create a folder (`<asset dir>`) and place a valid `install-config.yaml` and `agent-config.yaml`.
    19  
    20  ### Mirror the Openshift release and the MCE operator
    21  
    22  Using the oc mirror tool mirror the desired Openshift release (4.11+). Here is an example ImageSetConfiguration for OCP 4.11:
    23  
    24  __ocp-mce-imageset.yaml__
    25  ```
    26  kind: ImageSetConfiguration
    27  apiVersion: mirror.openshift.io/v1alpha2
    28  archiveSize: 4
    29  storageConfig:
    30    imageURL: <your-local-registry-dns-name>:<your-local-registry-port>/mirror/oc-mirror-metadata
    31    skipTLS: true
    32  mirror:
    33    platform:
    34      architectures:
    35        - "amd64"
    36      channels:
    37        - name: stable-4.11
    38          type: ocp
    39    additionalImages:
    40      - name: registry.redhat.io/ubi8/ubi:latest
    41    operators:
    42      - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.11
    43        packages:
    44          - name: multicluster-engine
    45          - name: local-storage-operator
    46  ```
    47  
    48  With this file you will be able to use this command to mirror the OCP release and the MCE and LSO operators:
    49  
    50  ```oc mirror --dest-skip-tls --config ocp-mce-imageset.yaml docker://<your-local-registry-dns-name>:<your-local-registry-port>```
    51  
    52  **Q: Why do I need the LSO operator?** A: The MCE operator is a large package and comes packaged with the infrastructure-operator as well. The infrastructure-operator needs a local storage volume to function.
    53  
    54  ### Update the mirror configurations in install-config.yaml
    55  
    56  In your __install-config.yaml__ you will need to update the registry and the certificate. The two configurations to update are `imageContentSources` and `additionalTrustBundle`.
    57  The following file snippet shows how to update the `imageContentSources` this for OCP and the MCE+LSO operators.
    58  
    59  ```
    60  imageContentSources:
    61    - source: "quay.io/openshift-release-dev/ocp-release"
    62      mirrors:
    63        - "<your-local-registry-dns-name>:<your-local-registry-port>/openshift/release-images"
    64    - source: "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
    65      mirrors:
    66        - "<your-local-registry-dns-name>:<your-local-registry-port>/openshift/release"
    67    - source: "registry.redhat.io/ubi8"
    68      mirrors:
    69        - "<your-local-registry-dns-name>:<your-local-registry-port>/ubi8"
    70    - source: "registry.redhat.io/multicluster-engine"
    71      mirrors:
    72        - "<your-local-registry-dns-name>:<your-local-registry-port>/multicluster-engine"
    73    - source: "registry.redhat.io/rhel8"
    74      mirrors:
    75        - "<your-local-registry-dns-name>:<your-local-registry-port>/rhel8"
    76    - source: "registry.redhat.io/redhat"
    77      mirrors:
    78        - "<your-local-registry-dns-name>:<your-local-registry-port>/redhat"
    79  ```
    80  
    81  **Note:** This procedure with `oc mirror` can be used to mirror any of the operators in the Red Hat Operator Indexes. After running `oc mirror` there will be a folder called `oc-mirror-workspace` with several outputs. There will be a file called __imageContentSourcePolicy.yaml__ in the most recent results folder that will identify all the mirrors you need for OCP and your selected operators.
    82  
    83  You will also need to make sure your certificate is present in the `additionalTrustBundle` field.
    84  This should look like something like this in your __install-config.yaml__:
    85  
    86  ```
    87  additionalTrustBundle: |
    88    -----BEGIN CERTIFICATE-----
    89    .
    90    .
    91    .
    92    .
    93    .
    94    .
    95    -----END CERTIFICATE-----
    96  ```
    97  
    98  Now you can generate the cluster manifests with the command:
    99  
   100  `openshift-install agent create cluster-manifests`
   101  
   102  This will update the cluster manifests to include a `mirror` folder with your mirror configuration.
   103  
   104  **From here you should be able to follow the steps for installing while connected online.**
   105  
   106  ## Installing while connected online
   107  
   108  ### Pre-requisites
   109  
   110  For the complete pre-requisites description, please check section 1.2.1 in [[1]](#1).
   111  
   112  Create a folder (`<asset dir>`) and place a valid `install-config.yaml` and `agent-config.yaml`.
   113  
   114  ### Prepare the manifests for the installation
   115  
   116  Create a subfolder named `openshift` within the `<asset dir>`. This subfolder will be used to store the extra manifests that will be applied during the installation to further customize the deployed cluster (note that the extra manifests will not be validated by the installer).
   117  
   118  #### MCE operator manifests
   119  
   120  Save the following manifests in the `<asset dir>/openshift` folder (use distinct files):
   121  
   122  __99_01_mce_namespace.yaml__
   123  ```
   124  apiVersion: v1
   125  kind: Namespace
   126  metadata:
   127    labels:
   128      openshift.io/cluster-monitoring: "true"
   129    name: multicluster-engine
   130  ```
   131  
   132  __99_02_mce_operatorgroup.yaml__
   133  ```
   134  apiVersion: operators.coreos.com/v1
   135  kind: OperatorGroup
   136  metadata:
   137    name: multicluster-engine-operatorgroup
   138    namespace: multicluster-engine
   139  spec:
   140    targetNamespaces:
   141    - multicluster-engine
   142  ```
   143  
   144  __99_03_mce_subscription.yaml__
   145  ```
   146  apiVersion: operators.coreos.com/v1alpha1
   147  kind: Subscription
   148  metadata:
   149    name: multicluster-engine
   150    namespace: multicluster-engine
   151  spec:
   152    channel: "stable-2.1"
   153    name: multicluster-engine
   154    source: redhat-operators
   155    sourceNamespace: openshift-marketplace
   156  ```
   157  
   158  #### Assisted Installer Service manifests
   159  
   160  Since distributed units (DU) will be installed via Assisted Installer Service (AIS), it will be necessary to enable it in the hub cluster (for more details, see [[2]](#2)). The AIS requires at least a couple of persistent volumes (PVs), and they could be installed via the OpenShift Local Storage operator (LSO) (see [[3]](#3)).
   161  
   162  Save the following manifests in the `<asset dir>/openshift` folder for the LSO setup (still using a separate file for each manifest):
   163  
   164  __99_04_lso_namespace.yaml__
   165  ```
   166  apiVersion: v1
   167  kind: Namespace
   168  metadata:
   169    annotations:
   170      openshift.io/cluster-monitoring: "true"
   171    name: openshift-local-storage
   172  ```
   173  
   174  __99_05_lso_operatorgroup.yaml__
   175  ```
   176  apiVersion: operators.coreos.com/v1
   177  kind: OperatorGroup
   178  metadata:
   179    name: local-operator-group
   180    namespace: openshift-local-storage
   181  spec:
   182    targetNamespaces:
   183      - openshift-local-storage
   184  ```
   185  
   186  __99_06_lso_subscription.yaml__
   187  ```
   188  apiVersion: operators.coreos.com/v1alpha1
   189  kind: Subscription
   190  metadata:
   191    name: local-storage-operator
   192    namespace: openshift-local-storage
   193  spec:
   194    installPlanApproval: Automatic
   195    name: local-storage-operator
   196    source: redhat-operators
   197    sourceNamespace: openshift-marketplace
   198  ```
   199  
   200  ### Create the agent ISO
   201  
   202  At this point your filesystem layout should look like the following:
   203  
   204  ```
   205  <asset dir>
   206      ├─ install-config.yaml
   207      ├─ agent-config.yaml
   208      └─ /openshift
   209          ├─ 99_01_mce_namespace.yaml
   210          ├─ 99_02_mce_operatorgroup.yaml
   211          ├─ 99_03_mce_subscription.yaml
   212          ├─ 99_04_lso_namespace.yaml
   213          ├─ 99_05_lso_operatorgroup.yaml
   214          └─ 99_06_lso_subscription.yaml
   215  ```
   216  
   217  Create the agent ISO by running the command:
   218  
   219  ```
   220  $ openshift-install agent create image --dir <asset dir>
   221  ```
   222  
   223  ### Cluster installation
   224  
   225  Once ready, boot the target machine using the agent ISO and wait for the installation to successfully complete. To monitor the ongoing deployment you can use this command:
   226  
   227  ```
   228  $ openshift-install agent wait-for install-complete --dir <asset dir>
   229  ```
   230  
   231  ### Hub setup
   232  
   233   As soon as the installation is completed it'd be possible then to finalize the setup to have a fully functioning hub cluster.
   234   The manifests shown in this section are meant to be applied manually. The order is relevant, and where needed the required
   235   waiting condition will also be illustrated.
   236  
   237  #### Local volumes
   238  
   239  Use the following manifest to create the required PVs that will be used by the AIS. Note that the `devicePaths` configuration
   240  should match your target machines hardware setup:
   241  
   242  __07_localvolumes.yaml__
   243   ```
   244   apiVersion: local.storage.openshift.io/v1
   245  kind: LocalVolume
   246  metadata:
   247    name: assisted-service
   248    namespace: openshift-local-storage
   249  spec:
   250    logLevel: Normal
   251    managementState: Managed
   252    storageClassDevices:
   253      - devicePaths:
   254          - /dev/vda
   255          - /dev/vdb
   256        storageClassName: assisted-service
   257        volumeMode: Filesystem
   258  ```
   259  ```
   260  $ oc apply -f 07_localvolumes.yaml
   261  ```
   262  
   263  To wait for the availability of the PVs you can use this command:
   264  
   265  ```
   266  $ oc wait localvolume -n openshift-local-storage assisted-service --for condition=Available --timeout 10m
   267  ```
   268  
   269  #### MCE
   270  
   271  Create a new multicluster engine instance by applying the following manifest:
   272  
   273  __08_mce.yaml__
   274  ```
   275  apiVersion: multicluster.openshift.io/v1
   276  kind: MultiClusterEngine
   277  metadata:
   278    name: multiclusterengine
   279  spec: {}
   280  ```
   281  ```
   282  $ oc apply -f 08_mce.yaml
   283  ```
   284  
   285  #### Enable the Assisted Installer service
   286  
   287  The AIS could be enabled through this manifest:
   288  
   289  __09_agentserviceconfig.yaml__
   290  ```
   291  apiVersion: agent-install.openshift.io/v1beta1
   292  kind: AgentServiceConfig
   293  metadata:
   294    name: agent
   295    namespace: assisted-installer
   296  spec:
   297   databaseStorage:
   298    storageClassName: assisted-service
   299    accessModes:
   300    - ReadWriteOnce
   301    resources:
   302     requests:
   303      storage: 10Gi
   304   filesystemStorage:
   305    storageClassName: assisted-service
   306    accessModes:
   307    - ReadWriteOnce
   308    resources:
   309     requests:
   310      storage: 10Gi
   311  ```
   312  ```
   313  $ oc apply -f 09_agentserviceconfig.yaml
   314  ```
   315  
   316  At this stage you can also apply the following manifest, as it will be useful when deploying spoke clusters:
   317  
   318  __10_clusterimageset.yaml__
   319  ```
   320  apiVersion: hive.openshift.io/v1
   321  kind: ClusterImageSet
   322  metadata:
   323    name: "4.12"
   324  spec:
   325    releaseImage: quay.io/openshift-release-dev/ocp-release:4.12.0-x86_64
   326  ```
   327  ```
   328  $ oc apply -f 10_clusterimageset.yaml
   329  ```
   330  
   331  #### Auto-importing the hub cluster
   332  
   333  As the last step of the finalization procedure, at this point you can import the current cluster (the one that it's hosting the mce operator and the assisted service) as the hub cluster, by applying the following manifest:
   334  
   335  __11_autoimport.yaml__
   336  ```
   337  apiVersion: cluster.open-cluster-management.io/v1
   338  kind: ManagedCluster
   339  metadata:
   340   labels:
   341     local-cluster: "true"
   342     cloud: auto-detect
   343     vendor: auto-detect
   344   name: local-cluster
   345  spec:
   346   hubAcceptsClient: true
   347  ```
   348  ```
   349  $ oc apply -f 11_autoimport.yaml
   350  ```
   351  
   352  And wait for the managed cluster to be created:
   353  
   354  ```
   355  $ oc wait -n multicluster-engine managedclusters local-cluster --for condition=ManagedClusterJoined=True --timeout 10m
   356  ```
   357  
   358  If everything goes fine, you should be able to successfully observe the new managed cluster:
   359  
   360  ```
   361  $ oc get managedcluster
   362  NAME            HUB ACCEPTED   MANAGED CLUSTER URLS             JOINED   AVAILABLE  AGE
   363  local-cluster   true           https://<your cluster url>:6443   True     True       77m
   364  ```
   365  
   366  # References
   367  
   368  * <a id="1">[1] https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.6/html/multicluster_engine/multicluster_engine_overview#installing-from-the-cli-mce
   369  * <a id="2">[2] https://docs.openshift.com/container-platform/4.9/scalability_and_performance/ztp-deploying-disconnected.html
   370  * <a id="3">[3] https://docs.openshift.com/container-platform/4.11/storage/persistent_storage/persistent-storage-local.html