github.com/openshift/installer@v1.4.17/docs/user/ovirt/install_upi.md

# Install: oVirt/RHV User-Provided Infrastructure

This User-Provisioned Infrastructure (UPI) process is based on several and
customizable steps to allow the user to integrate into an existing infrastructure.

Creating and configuring oVirt/RHV resources is the responsibility of the user
deploying OpenShift.

The OpenShift installer will still be used in several steps of the process to generate
mandatory ignition files and to monitor the installation process itself.

## Table of Contents

* [Prerequisites](#prerequisites)
  * [Ansible and oVirt roles](#ansible-and-ovirt-roles)
  * [Network Requirements](#network-requirements)
    * [Load Balancers](#load-balancers)
    * [DNS](#dns)
  * [RHCOS image](#rhcos-image)
* [Getting Ansible playbooks](#getting-ansible-playbooks)
* [Assets directory](#assets-directory)
* [Inventory explained](#inventory-explained)
  * [General section](#general-section)
  * [RHCOS section](#rhcos-section)
  * [Profiles and VMs](#profiles-and-vms)
    * [Profiles section](#profiles-section)
    * [VMs section](#vms-section)
* [Install Config](#install-config)
  * [Set compute replicas to zero](#set-compute-replicas-to-zero)
  * [Set machine network](#set-machine-network)
  * [Set platform to none](#set-platform-to-none)
* [Manifests](#manifests)
  * [Set control-plane nodes unschedulable](#set-control-plane-nodes-unschedulable)
* [Ignition configs](#ignition-configs)
* [Create templates and VMs](#create-templates-and-vms)
* [Bootstrap](#bootstrap)
* [Master nodes](#master-nodes)
* [Wait for Control Plane](#wait-for-control-plane)
* [OpenShift API](#openshift-api)
* [Retire Bootstrap](#retire-bootstrap)
* [Worker nodes](#worker-nodes)
    * [Approve CSRs](#approve-csrs)
* [Wait for Installation Complete](#wait-for-installation-complete)
* [Destroy OpenShift cluster](#destroy-openshift-cluster)

# Prerequisites
The [inventory.yml](../../../upi/ovirt/inventory.yml) file all the variables used by this installation can be customized as per
user needs.
The requirements for the UPI in terms of minimum resources are broadly the same as 
the [IPI](./install_ipi.md#minimum-resources).

- oVirt/RHV account stored in the [ovirt-config.yaml](https://github.com/openshift/installer/blob/master/docs/user/ovirt/install_ipi.md#ovirt-configyaml)
  - this file is generated by the `openshift-install` binary installer following the CLI wizard.
- Name of the oVirt/RHV cluster to use
    - contained in the [inventory.yml](../../../upi/ovirt/inventory.yml) and input in the `openshift-install`.
- A base domain name of the OpenShift cluster
    - input in the `openshift-install`.
- A name of the OpenShift cluster
    - input in the `openshift-install`.
- OpenShift Pull Secret
    - input in the `openshift-install`.
- A DNS zone
  - to configure the resolution names for the OpenShift cluster base domain.
- LoadBalancers
  - for bootstrap and control-plane machines.
  - for machines running the ingress router (usually compute nodes).

## Ansible and oVirt roles
To use the UPI process described here the following are required:

- Python3

**Note**:  
Currently most of Linux distros provides Python 3 by default.

- Ansible 

**Note for CentOS users**:  
Depending on which version  the system is running will be required [epel-release](https://fedoraproject.org/wiki/EPEL) repo enabled.

```
  $ sudo dnf install ansible
```
- python3-ovirt-engine-sdk4

```
  $ sudo dnf install python3-ovirt-engine-sdk4
```
- ovirt.image-template Ansible role (distributed as ovirt-ansible-image-template package on oVirt Manager)
- ovirt.vm-infra Ansible role (distributed as ovirt-ansible-vm-infra package on oVirt Manager)

```
  $ sudo ansible-galaxy install ovirt-ansible-vm-infra ovirt-ansible-image-template
```


To be sure to follow the UPI installation process, Ansible scripts and the binary openshift-install 
should be executed from the oVirt/RHV Manager or from a machine with access to the REST API of the 
oVirt/RHV Manager and with all the oVirt roles available (installed by default on the Manager 
machine).

## Network Requirements
The UPI installation process assumes that the user satisfies some network requirements providing them through the
existing infrastructure.
During the boot the RHCOS based machines require an IP address in `initramfs` in order to establish a network connection to get their
ignition config files.
One of the recommended ways is to use a DHCP server to manage the machines in the long-term, maybe configuring the DHCP server
itself to provide persistent IP addresses and host names to the cluster machines.

Network connectivity between machines should be configured to allow cluster components to communicate:

- Kubernetes NodePort
  Machines require connectivity to every other machine for OpenShift platform components through the port range `30000`-`32767` .

- OpenShift reserved
  Connectivity to reserved port ranges `10250`-`10259` and `9000`-`9999` should be granted on every machine.

- Machines to control-plane
  Connectivity to ports on ranges `2379`-`2380` (for etcd, peer and metrics) is required for control-plane machines and on
  port `6443` for Kubernetes API.


### Load Balancers
Before installing the OpenShift Container Platform, two load balancers (layer-4) must be provided by the user infrastructure,
one for the API and one for the Ingress Controller (to allow ingress to applications).

- Load balancer for port `6443` and `22623` on control-plane and bootstrap machines (the bootstrap can be removed after control-plane
  initialization completes).
  The `6443` must be both internal and external reachable and is needed by the Kubernetes API server.
  Port `22623` must be accessible to nodes within the cluster.

- Load balancer for port `443` and `80` for machines running the ingress router (usually worker nodes in the default configuration).
  Both ports must be accessible from within and outside the cluster.

**NOTE**: the rules above can also be set on the same load balancer server. 


### DNS
The UPI installation process requires the user to setup the existing infrastructure provided DNS to allow the correct resolution of
the main components and services

- Kubernetes API
  DNS records `api.<cluster_name>.<base_domain>` (internal and external resolution) and `api-int.<cluster_name>.<base_domain>` 
  (internal resolution) must be added to point to the Load balancer targeting the control plane machines. 

- OpenShift routes
  A DNS record `*.apps.<cluster_name>.<base_domain>` must be provided to point to the Load balancer configured to manage the
  traffic for the ingress router (ports `443` and `80` of the compute machines).

**NOTE**: the DNS records above may also point to the same IP in case you are using only one load balancer configured with the rules described
in the [previous section](#load-balancers).

## RHCOS image
This UPI installation process requires a proper RHCOS (Red Hat Enterprise Linux CoreOS) image URL to be set in the [inventory.yml](../../../upi/ovirt/inventory.yml) file.

The RHCOS images can be found [here](https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/) and you have to choose
the URL related to the `OpenStack` qcow2 image type, like in the example below

```
https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.6.0-0.nightly-2020-07-16-122837/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz
```

The version of the image should be chosen according to the OpenShift version you're about to install (in general less than or equal to the OCP
version). 
Once you have the URL set in the [inventory.yml](../../../upi/ovirt/inventory.yml) a dedicated Ansible playbook will be in charge to download the `qcow2.gz` file, uncompress it
in a specified folder and use it to create oVirt/RHV templates.

## Getting Ansible playbooks
All the Ansible playbooks used in this UPI installation process are available [here](https://github.com/openshift/installer/tree/master/upi/ovirt)
and can be downloaded with the following utility script

```sh
RELEASE="release-4.6"; \
curl -L -X GET https://api.github.com/repos/openshift/installer/contents/upi/ovirt\?ref\=${RELEASE} | 
grep 'download_url.*\.yml' | 
awk '{ print $2 }' | sed -r 's/("|",)//g' | 
xargs -n 1 curl -O
```

Different versions of the oVirt UPI playbooks can be downloaded changing the RELEASE environment variable to the desired branch 
(please be aware that this UPI work started with the `release-4.6`).

## Assets directory
Before proceeding with the installation is **required** to set an environment variable with the path (absolute or relative according to your preferences) 
of the directory in which the `openshift-install` command will put all the artifacts and that we'll also refer to in the [inventory.yml](../../../upi/ovirt/inventory.yml)`.

```sh
$ export ASSETS_DIR=./wrk
```

## Inventory Explained
This section shows an example of [inventory.yml](../../../upi/ovirt/inventory.yml), used to specify the variables needed for the UPI installation process, with a brief explanation of the sections included.

```YAML
---
all:
  vars:

    # ---
    # General section
    # ---
    ovirt_cluster: "Default" 
    ocp:
      assets_dir: "{{ lookup('env', 'ASSETS_DIR') }}"
      ovirt_config_path: "{{ lookup('env', 'HOME') }}/.ovirt/ovirt-config.yaml"

    # ---
    # RHCOS section
    # ---
    rhcos:
      image_url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest-4.6/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz"
      local_cmp_image_path: "/tmp/rhcos.qcow2.gz"
      local_image_path: "/tmp/rhcos.qcow2"

    # ---
    # Profiles section
    # ---
    control_plane:
      cluster: "{{ ovirt_cluster }}"
      memory: 16GiB
      sockets: 4
      cores: 1
      template: rhcos_tpl
      operating_system: "rhcos_x64"
      type: high_performance
      graphical_console:
        headless_mode: false
        protocol:
        - spice
        - vnc
      disks:
      - size: 120GiB
        name: os
        interface: virtio_scsi
        storage_domain: depot_nvme
      nics:
      - name: nic1
        network: lab
        profile: lab

    compute:
      cluster: "{{ ovirt_cluster }}"
      memory: 16GiB
      sockets: 4
      cores: 1
      template: worker_rhcos_tpl
      operating_system: "rhcos_x64"
      type: high_performance
      graphical_console:
        headless_mode: false
        protocol:
        - spice
        - vnc
      disks:
      - size: 120GiB
        name: os
        interface: virtio_scsi
        storage_domain: depot_nvme
      nics:
      - name: nic1
        network: lab
        profile: lab

    # ---
    # VMs section
    # ---
    vms:
    - name: "{{ metadata.infraID }}-bootstrap"
      ocp_type: bootstrap
      profile: "{{ control_plane }}"
      type: server
    - name: "{{ metadata.infraID }}-master0"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-master1"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-master2"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-worker0"
      ocp_type: worker
      profile: "{{ compute }}"
    - name: "{{ metadata.infraID }}-worker1"
      ocp_type: worker
      profile: "{{ compute }}"
    - name: "{{ metadata.infraID }}-worker2"
      ocp_type: worker
      profile: "{{ compute }}"
```

### General section
Variables in this section are mandatory and allow the user to specify 

* `ovirt_cluster`: the name of the ovirt cluster in which you'll install the OCP cluster.
* `ocp.assets_dir`: is the path of the folder in which the `openshift-install` command will put all the files built in different stages.
* `ocp.ovirt_config_path`: path of the `ovirt-config.yaml`, generated by the `openshift-install` in the [first stage](#install-config), containing the ovirt credentials 
(necessary to interact with the oVirt/RHV Manager REST API).

### RHCOS section
The `rhcos` variable contains the RHCOS public URL (`image_url`) for downloading the image in the local specified path (`local_cmp_image_path`) and
uncompressing it (in a file described by `local_image_path`)  before being able to use it.

```YAML
  rhcos:
    image_url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.6.0-0.nightly-2020-07-16-122837/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz"
    local_cmp_image_path: "/tmp/rhcos.qcow2.gz"
    local_image_path: "/tmp/rhcos.qcow2"
```

Please refer to the [specific paragraph](#rhcos-image) to learn more about which version of RHCOS image to choose and where to find it.

### Profiles and VMs
The latest and important part of the [inventory.yml](../../../upi/ovirt/inventory.yml) is related to `profiles` and `vms` definition using the capabilities
offered by the ovirt.vm-infra role, whose documentation can be found [here](#https://github.com/oVirt/ovirt-ansible-vm-infra).
In the following paragraphs we'll explain the meaning of the basic parameters from the OCP point of view.

#### Profiles section
This section is mainly composed of two variable, `control_plane` and `compute`, that define the two different profiles used respectively
for masters (and bootstrap) and workers.

```YAML
    control_plane:
      cluster: "{{ ovirt_cluster }}"
      memory: 16GiB
      sockets: 4
      cores: 1
      template: rhcos_tpl
      operating_system: "rhcos_x64"
      type: high_performance
      disks:
      - size: 120GiB
        name: os
        interface: virtio_scsi
        storage_domain: depot_nvme
      nics:
      - name: nic1
        network: lab
        profile: lab

    compute:
      cluster: "{{ ovirt_cluster }}"
      memory: 16GiB
      sockets: 4
      cores: 1
      template: worker_rhcos_tpl
      operating_system: "rhcos_x64"
      type: high_performance
      disks:
      - size: 120GiB
        name: os
        interface: virtio_scsi
        storage_domain: depot_nvme
      nics:
      - name: nic1
        network: lab
        profile: lab
```

The user can customize the parameters of both profiles according to the needs and the minimum requirements.

* `cluster`: it's already set according to the value of the variable in the [General Section](#general-section).
* `memory, sockets, cores`: mandatory parameters necessary to define the common specs of the VMs generated from this profile.
* `template`: name of the template that the virtual machine will be based on (refer to [this](#create-templates-and-vms) for further information about templates).
* `operating_system`: sets the vm OS type. With oVirt/RHV 4.4 it's mandatory to use the value `rhcos_x64` to allow the `ignition script` 
to be correctly passed to the VM. 
* `type`: it's the type that the VM will have once created.
* `disks`: in this section the specs of the disk must be set according to the basic requirements of OCP in terms of capacity and storage 
performances. It's possible to choose different storage-domains for control_plane and compute nodes.
* `nics`: defines the specs like the name of the nic and the network that the Vms will use. The virtual network interface profile can also be specified.
The MAC address will be taken from the oVirt/RHV MAC pool.

#### VMs section
In this last section of the [inventory.yml](../../../upi/ovirt/inventory.yml) there's the definition of the `vms` variable, containing all the node instance that the user plans to create
to deploy the OCP cluster (remember that there are minimum requirements in terms of number of master and worker nodes).

In the last section there's the list of all the vms that will be created and their role expressed by the `ocp_type`.

```YAML
  vms:
    - name: "{{ metadata.infraID }}-bootstrap"
      ocp_type: bootstrap
      profile: "{{ control_plane }}"
      type: server
    - name: "{{ metadata.infraID }}-master0"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-master1"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-master2"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-worker0"
      ocp_type: worker
      profile: "{{ compute }}"
    - name: "{{ metadata.infraID }}-worker1"
      ocp_type: worker
      profile: "{{ compute }}"
```

As you can see above the `vms` variable is basically defined by a list of elements each one with at least three mandatory attributes

* `name`: name of the virtual machine to create.
* `ocp_type`: the role of the virtual machine in the OCP cluster (possible values are `bootstrap`, `master`, `worker`).
* `profile`: name of the profile (`control_plane` or `compute`) from which to inherit common specs

Additional attributes can be specified to override the ones defined in the inheriting profile

* `type`: is re-defined as `server` in the `bootstrap` vm

It's also possible to use all the attributes documented in the [oVirt.vm-infra role](#https://github.com/oVirt/ovirt-ansible-vm-infra)
(e.g.: fixed MAC address for each vm that could help to assign permanent IP through a DHCP).

**Note**:
Looking at the `vms` attribute `name` setting, you can see that we are using the variable `metadata.infraID` whose value is 
obtained parsing the `metadata.json` file generated using the command `openshift-install create ignition-configs` (read more about it
[here](#ignition-configs)).
There's a specific set of Ansible tasks ([common-auth.yml](https://github.com/openshift/installer/blob/master/upi/ovirt/common-auth.yml)) included in all
the UPI playbooks that contains the code to read the `infraID` from the specific file located in the `ocp.assets_dir` 

```YAML
---
- name: include metadata.json vars
  include_vars:
    file: "{{ ocp.assets_dir }}/metadata.json"
    name: metadata
  
  ...


```

## Install config
Run the `openshift-install` to create the initial `install-config` using as assets directory the same that is specified in
the [inventory.yml](../../../upi/ovirt/inventory.yml) (`working_path`).

```sh
$ openshift-install create install-config --dir $ASSETS_DIR
? SSH Public Key /home/user/.ssh/id_dsa.pub
? Platform <ovirt>
? Engine FQDN[:PORT] [? for help] <engine.fqdn>
? Enter ovirt-engine username <admin@internal>
? Enter password <******>
? oVirt cluster <cluster>
? oVirt storage <storage>
? oVirt network <net> 
? Internal API virtual IP <172.16.0.252>
? Ingress virtual IP <172.16.0.251>
? Base Domain <example.org>
? Cluster Name <ocp4>
? Pull Secret [? for help] <********>
```

*Internal API* and *Ingress* are the IPs added following the above DNS instructions

- `api.ocp4.example.org`: 172.16.0.252
- `*.apps.ocp4.example.org`: 172.16.0.251 

*Cluster Name* (`ocp4`) and *Base Domain* (`example.org`) joint together will form the FQDN of the OCP cluster
used to expose the API interface (`https://api.ocp4.example.org:6443/`)
and the newly created applications (e.g. `https://console-openshift-console.apps.ocp4.example.org`).

You can obtain a new Pull secret from [here](https://console.redhat.com/openshift/install/pull-secret).

The result of this first step is the creation of a `install-config.yaml` in the specified assets directory:

```sh
$ tree
.
└── wrk
    └── install-config.yaml
```

File `$HOME/.ovirt/ovirt-config.yaml` was also created for you by the `openshift-install` containing all the connection
parameters needed to reach the oVirt/RHV engine and use its REST API.

**NOTE:**
Some of the parameters added during the `openshift-install` workflow, in particular the `Internal API virtual IP` and
`Ingress virtual IP`, will not be used because already configured in your infrastructure DNS (see [DNS](#dns) section).
Other parameters like `oVirt cluster`, `oVirt storage`, `oVirt network`, will be used as specified in the [inventory.yml](../../../upi/ovirt/inventory.yml)
and removed from the `install-config.yaml` with the previously mentioned `virtual IPs`, using a script reported in a 
[section below](#set-platform-to-none).

### Set compute replicas to zero
Machine API will not be used by the UPI to create nodes, we'll create compute nodes explicitly with Ansible scripts.
Therefore we'll set the number of compute nodes to zero replicas using the following python script:

```sh
$ python3 -c 'import os, yaml
path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"]
conf = yaml.safe_load(open(path))
conf["compute"][0]["replicas"] = 0
open(path, "w").write(yaml.dump(conf, default_flow_style=False))'
```

**NOTE**: All the Python snippets in this document work with both Python 3 and Python 2.

### Set machine network
OpenShift installer sets a default IP range for nodes and we need to change it according to our infrastructure.
We'll set the range to `172.16.0.0/16` (we can use the following python script for this):

```sh
$ python3 -c 'import os, yaml
path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"]
conf = yaml.safe_load(open(path))
conf["networking"]["machineNetwork"][0]["cidr"] = "172.16.0.0/16"
open(path, "w").write(yaml.dump(conf, default_flow_style=False))'
```

### Set platform to none
The UPI for oVirt/RHV is similar to the bare-metal installation process and for now we don't need the specific ovirt
platform section in the `install-config.yaml`, all the settings needed are specified in the [inventory.yml](../../../upi/ovirt/inventory.yml).
We'll remove the section 

```sh
$ python3 -c 'import os, yaml
path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"]
conf = yaml.safe_load(open(path))
platform = conf["platform"]
del platform["ovirt"]
platform["none"] = {}
open(path, "w").write(yaml.dump(conf, default_flow_style=False))'
```

## Manifests
Editing manifests is an action required for the UPI and to generate them we can use again the binary installer

```sh
$ openshift-install create manifests --dir $ASSETS_DIR
```
```sh
$ tree
.
└── wrk
    ├── manifests
    │   ├── 04-openshift-machine-config-operator.yaml
    │   ├── cluster-config.yaml
    │   ├── cluster-dns-02-config.yml
    │   ├── cluster-infrastructure-02-config.yml
    │   ├── cluster-ingress-02-config.yml
    │   ├── cluster-network-01-crd.yml
    │   ├── cluster-network-02-config.yml
    │   ├── cluster-proxy-01-config.yaml
    │   ├── cluster-scheduler-02-config.yml
    │   ├── cvo-overrides.yaml
    │   ├── kube-cloud-config.yaml
    │   ├── kube-system-configmap-root-ca.yaml
    │   ├── machine-config-server-tls-secret.yaml
    │   └── openshift-config-secret-pull-secret.yaml
    └── openshift
        ├── 99_kubeadmin-password-secret.yaml
        ├── 99_openshift-cluster-api_master-user-data-secret.yaml
        ├── 99_openshift-cluster-api_worker-user-data-secret.yaml
        ├── 99_openshift-machineconfig_99-master-ssh.yaml
        ├── 99_openshift-machineconfig_99-worker-ssh.yaml
        └── openshift-install-manifests.yaml
```

The command above will write manifests consuming the `install-config.yaml` and will show a warning message.
If you plan on reusing the `install-config.yaml` file, back it up before you generate manifests.

```sh
INFO Consuming Install Config from target directory 
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings 
```

### Set control-plane nodes unschedulable
Setting compute replicas to zero makes control-plane nodes schedulable which is something we don't want for now 
(router pods can run also on control-plane nodes but there are some Kubernetes limitation that will prevent those pods
to be reachable by the ingress load balancer).
Setting the control-plan as unschedulable means modifying the `manifests/cluster-scheduler-02-config.yml` setting
`masterSchedulable` to `False`.

```sh 
$ python3 -c 'import os, yaml
path = "%s/manifests/cluster-scheduler-02-config.yml" % os.environ["ASSETS_DIR"]
data = yaml.safe_load(open(path))
data["spec"]["mastersSchedulable"] = False
open(path, "w").write(yaml.dump(data, default_flow_style=False))'
```

## Ignition configs
The next step is the one needed to build [Ignition](https://coreos.com/ignition/docs/latest/) files from the manifests just modified.
Ignition files have to be fetched by RHCOS machine initramfs to perform configurations that will bring to a live final node.
We will use again the binary installer.

```sh 
$ openshift-install create ignition-configs --dir $ASSETS_DIR
```

```sh
$ tree
.
└── wrk
    ├── auth
    │   ├── kubeadmin-password
    │   └── kubeconfig
    ├── bootstrap.ign
    ├── master.ign
    ├── metadata.json
    └── worker.ign
```
Other than the ignition files the installer generated

- `auth` folder containing the admin credentials necessary to
  connect to the cluster via the `oc` or `kubectl` CLI utilities.
- `metadata.json` with information like the OCP cluster name, OCP cluster ID and the `infraID`
  (generated for the current running installation).

The `infraID` will be used by the UPI Ansible playbooks as prefix for the VMs created during the installation
process avoiding name clashes in case of multiple installations in the same oVirt/RHV cluster.

**Note:** certificates contained into ignition config files expire after 24 hours. You must complete the cluster installation
and keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.


## Create templates and VMs
After having checked that all our variables in the [inventory.yml](../../../upi/ovirt/inventory.yml) fit the needs, we can run the first of our Ansible provisioning
playbooks.

```sh
$ ansible-playbook -i inventory.yml create-templates-and-vms.yml
```

This playbook will use the connection parameters for the oVirt/RHV engine stored in the `$HOME/.ovirt/ovirt-config.yaml`
reading also the `metadata.json` from the assets directory.

According to the variables

```YAML
  rhcos:
    image_url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.6.0-0.nightly-2020-07-16-122837/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz"
    local_cmp_image_path: "/tmp/rhcos.qcow2.gz"
    local_image_path: "/tmp/rhcos.qcow2"
```

the RHCOS image will be downloaded (in case not already existing locally), stored locally and extracted to be uploaded on the oVirt/RHV node
and used for template creation.
The user can check the RHCOS image for the OpenShift version he want to use from [here](https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/).

Templates will be created according to the names specified in the [inventory.yml](../../../upi/ovirt/inventory.yml) for the `control_plane` and `compute` profile (in case of
different names two different templates will be created).
In case the user want to have different templates for each OCP installation in the same cluster, it is possible to customize the template name
in the [inventory.yml](../../../upi/ovirt/inventory.yml) prepending the `infraID` (like it is done for VMs' names).

```YAML
  control_plane:
    cluster: "{{ ovirt_cluster }}"
    memory: 16GiB
    sockets: 4
    cores: 1
    template: "{{ metadata.infraID }}-rhcos_tpl"
    operating_system: "rhcos_x64"
    ...
```

At the end of the execution the VMs specified will be created and left in stopped mode. This allows the user to fetch any information from the
VMs that can help configuring other infrastructure elements (e.g.: getting MAC addresses to feed a DHCP server to assign permanent IPs).

## Bootstrap

```sh
$ ansible-playbook -i inventory.yml bootstrap.yml
```
The playbook starts the bootstrap VM passing it the `bootstrap.ign` ignition file contained in the assets directory. That will allow the bootstrap
node to configure itself and be ready to serve ignition files for the master nodes.
The user can check the console inside oVirt/RHV UI or can connect to the VM via SSH.
Running the following command from inside the bootstrap VM enables them to closely monitor the bootstrap process.

```sh
$ ssh core@<boostrap.ip>
[core@ocp4-lk6b4-bootstrap ~]$ journalctl -b -f -u release-image.service -u bootkube.service
```

## Master nodes

```sh
$ ansible-playbook -i inventory.yml masters.yml
```

The `masters.yml` will start our control-plane made of three masters (but it can be customized) passing the `master.ign` ignition file to each of the VMs.
`master.ign` ignition file contains a directive that instructs masters to fetch the ignition from the URL

```sh
https://api-int.ocp4.example.org:22623/config/master
```

targeted by the Load balancer that manages the traffic on port `22623` (accessible only inside the cluster) driving it to masters and bootstrap.

## Wait for Control Plane
The user can monitor the control-plane bootstrap process with the following command:

```sh
$ openshift-install wait-for bootstrap-complete --dir $ASSETS_DIR
```

After some time the output of the command will be the following

```sh
INFO API v1.18.3+b74c5ed up
INFO Waiting up to 40m0s for bootstrapping to complete... 
```

After all the pods on master nodes and the etcd will be up and running the installer will show the following output:

```sh
INFO It is now safe to remove the bootstrap resources
```

## OpenShift API
The OpenShift API can be accessed via the `oc` or `kubectl` using the admin credentials contained in the assets directory
in the file `auth/kubeconfig`:

```sh
$ export KUBECONFIG=$ASSETS_DIR/auth/kubeconfig
$ oc get nodes
$ oc get pods -A
```
## Retire Bootstrap
After the `wait-for` command says that the bootstrap process is complete, it is possible to remove the bootstrap VM

```sh
$ ansible-playbook -i inventory.yml retire-bootstrap.yml
```

and the user can remove it also from the Load balancer directives.

## Worker nodes
```sh
$ ansible-playbook -i inventory.yml workers.yml
```

This is similar to what we did for masters but in this case workers won't automatically join the cluster, we'll need to 
approve their respective pending CSRs (Certificate Signing Requests).

### Approve CSRs
CSRs for nodes joining the cluster will need to be approved by the administrator. The following command helps to list
pending requests

```sh
$ oc get csr -A
```

Eventually one pending CSR per node will be shown

```sh
NAME        AGE    SIGNERNAME                                    REQUESTOR                                                                   CONDITION
csr-2lnxd   63m    kubernetes.io/kubelet-serving                 system:node:ocp4-lk6b4-master0.ocp4.example.org                             Approved,Issued
csr-hff4q   64m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-hsn96   60m    kubernetes.io/kubelet-serving                 system:node:ocp4-lk6b4-master2.ocp4.example.org                             Approved,Issued
csr-m724n   6m2s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-p4dz2   60m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-t9vfj   60m    kubernetes.io/kubelet-serving                 system:node:ocp4-lk6b4-master1.ocp4.example.org                             Approved,Issued
csr-tggtr   61m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-wcbrf   7m6s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
```

To filter and watch pending CSRs the following command can be used

```sh
$ watch "oc get csr -A | grep pending -i"
```
that refresh the output every two seconds
```sh
Every 2.0s: oc get csr -A | grep pending -i

csr-m724n   10m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-wcbrf   11m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
```

Every `Pending` request should be inspected

```sh
$ oc describe csr csr-m724n
Name:               csr-m724n
Labels:             <none>
Annotations:        <none>
CreationTimestamp:  Sun, 19 Jul 2020 15:59:37 +0200
Requesting User:    system:serviceaccount:openshift-machine-config-operator:node-bootstrapper
Signer:             kubernetes.io/kube-apiserver-client-kubelet
Status:             Pending
Subject:
         Common Name:    system:node:ocp4-lk6b4-worker1.ocp4.example.org
         Serial Number:  
         Organization:   system:nodes
Events:  <none>
```
and finally approved

```sh
$ oc adm certificate approve csr-m724n
```

After approving the first requests, another CSR per each worker will be issued and will need to be approved to have worker nodes not only 
joining the cluster, but also becoming `Ready` and having pods scheduled on them.

## Wait for Installation Complete
The following command can now be run to follow the installation process till it's complete:

```sh
$ openshift-install wait-for install-complete --dir $ASSETS_DIR --log-level debug
```

Eventually it will give as output

- URL to reach the OpenShift Console (web UI).
- Username and password for the admin login.


## Destroy OpenShift cluster

```ssh
$ ansible-playbook -i inventory.yml \
    retire-bootstrap.yml \
    retire-masters.yml   \
    retire-workers.yml
```

Removing DNS added records, Load balancers and any other infrastructure configuration is left to the user.