github.com/openshift/installer@v1.4.17/docs/user/customization.md (about) 1 # Cluster Customization 2 3 The OpenShift Installer allows for several different levels of customization. It is important to understand how and why each of these levels is exposed and the ramifications of making changes at each of these levels. This guide will attempt to walk through each of them and provide examples for why an administrator may want to make customizations. 4 5 Cluster customization can be broken into four major levels: OpenShift, Kubernetes, Platform, and OS. These four levels are rough abstraction layers (OpenShift being the highest layer and OS being the lowest) and fall into either the validated or unvalidated buckets. The levels within the validated bucket (OpenShift and Platform) encompass customization that is safe to perform - installation and automatic updates will succeed regardless of the changes made (to a reasonable degree). The levels within the unvalidated bucket (Kubernetes and OS) encompass customization that is not necessarily safe - after introducing changes, installation and automatic updates may not succeed. 6 7 ## OpenShift Customization 8 9 The most simple customization is exposed by the installer as an interactive series of prompts. These prompts are required and represent a high-level of customization. They are needed in order to get a running OpenShift cluster, but they aren't enough to get anything other than a vanilla deployment out of the box. Further customization is possible once the cluster has been provisioned, but isn't covered in this document as it is a "Day 2" operation. 10 11 ## Platform Customization 12 13 While the default cluster size may be sufficient for some, many will need to make alterations. This can include increasing the number of machines in the control plane, changing the type of the virtual machines that will be used (e.g. AWS instances), or adjusting the CIDR range used for the Kubernetes service network. This level of customization is exposed via the installer's `install-config.yaml`. The install-config can be accessed by running `openshift-install create install-config`. This file can then be modified as needed before running a later target. 14 15 The `install-config.yaml` generated by the installer will not have all of the available fields populated, so they may need to be manually added if they are needed. 16 17 The following `install-config.yaml` properties are available: 18 19 * `apiVersion` (required string): The API version for the `install-config.yaml` content. 20 The current version (as described in this documentation) is `v1`. 21 The installer may also support older API versions. 22 * `additionalTrustBundle` (optional string): a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store. 23 This trust bundle may also be used when [a proxy has been configured](#proxy). 24 * `additionalTrustBundlePolicy` (optional string): determines when to add the AdditionalTrustBundle to the nodes'' trusted certificate store. 25 "Proxyonly" is the default. The field can be set to following specified values. 26 "Proxyonly" : adds the AdditionalTrustBundle to nodes when http/https proxy is configured. 27 "Always" : always adds AdditionalTrustBundle. 28 * `baseDomain` (required string): The base domain to which the cluster should belong. 29 * `capabilities` (optional [capabilities](#capabilities)): Capabilities configures the installation of optional core cluster components. 30 * `controlPlane` (optional [machine-pool](#machine-pools)): The configuration for the machines that comprise the control plane. 31 * `compute` (optional array of [machine-pools](#machine-pools)): The configuration for the machines that comprise the compute nodes. 32 * `featureSet` (optional string): A feature set as defined in the [OpenShift API](https://github.com/openshift/api/blob/013a7b8bf9b3d57d0642b8f14122f881635ed5a3/config/v1/types_feature.go#L28-L43). Notably, `TechPreviewNoUpgrade` can be used to enable Installer features protected by a Tech Preview feature gate. Feature 33 sets will be applied to the cluster--not just the Installer--and may affect the supportability and upgradability of the cluster, depending on the specified 34 feature set. 35 * `fips` (optional boolean): Enables FIPS mode (default false). 36 * `imageContentSources` (optional array of objects): Sources and repositories for the release-image content. 37 Each entry in the array is an object with the following properties: 38 * `source` (required string): The repository that users refer to, e.g. in image pull specifications. 39 * `mirrors` (optional array of strings): One or more repositories that may also contain the same images. 40 * `metadata` (required object): Kubernetes resource ObjectMeta, from which only the `name` parameter is consumed. 41 * `name` (required string): The name of the cluster. 42 DNS records for the cluster are all subdomains of `{{.metadata.name}}.{{.baseDomain}}`. 43 * `networking` (optional object): The configuration for the pod network provider in the cluster. 44 * `clusterNetwork` (optional array of objects): The IP address pools for pods. 45 The default is 10.128.0.0/14 with a host prefix of /23. 46 * `cidr` (required [IP network](#ip-networks)): The IP block address pool. 47 * `hostPrefix` (required integer): The prefix size to allocate to each node from the CIDR. 48 For example, 24 would allocate 2^8=256 addresses to each node. If this field is not used by the plugin, it can be left unset. 49 * `machineNetwork` (optional array of objects): The IP address pools for machines. 50 * `cidr` (required [IP network](#ip-networks)): The IP block address pool. 51 The default is 10.0.0.0/16 for all platforms other than libvirt. 52 For libvirt, the default is 192.168.126.0/24. 53 * `networkType` (optional string): The type of network to install. 54 The default is [OVNKubernetes][ovn-kubernetes]. 55 * `serviceNetwork` (optional array of [IP networks](#ip-networks)): The IP address pools for services. 56 The default is 172.30.0.0/16. 57 * `platform` (required object): The configuration for the specific platform upon which to perform the installation. 58 * `aws` (optional object): [AWS-specific properties](aws/customization.md#cluster-scoped-properties). 59 * `baremetal` (optional object): [Baremetal IPI-specific properties](metal/customization_ipi.md). 60 * `azure` (optional object): [Azure-specific properties](azure/customization.md#cluster-scoped-properties). 61 * `openstack` (optional object): [OpenStack-specific properties](openstack/customization.md#cluster-scoped-properties). 62 * `ovirt` (optional object): [oVirt-specific properties](ovirt/customization.md#cluster-scoped-properties). 63 * `vsphere` (optional object): [vSphere-specific properties](vsphere/customization.md#cluster-scoped-properties). 64 * `proxy` (optional object): The proxy settings for the cluster. 65 If unset, the cluster will not be configured to use a proxy. 66 * `httpProxy` (optional string): The URL of the proxy for HTTP requests. 67 * `httpsProxy` (optional string): The URL of the proxy for HTTPS requests. 68 * `noProxy` (optional string): A comma-separated list of domains and [CIDRs][cidr-notation] for which the proxy should not be used. 69 * `publish` (optional string): This controls how the user facing endpoints of the cluster like the Kubernetes API, OpenShift routes etc. are exposed. 70 Valid values are `External` (the default) and `Internal`. 71 * `pullSecret` (required string): The secret to use when pulling images. 72 * `sshKey` (optional string): The public Secure Shell (SSH) key to provide access to instances. 73 74 ### Capabilities 75 76 * `baselineCapabilitySet` (optional string): Selects an initial set of optional capabilities to enable. The default value is `vCurrent` (the default). Aadditional valid values can be found [here](https://pkg.go.dev/github.com/openshift/api/config/v1#ClusterVersionCapabilitySet). 77 * `additionalEnabledCapabilities` (optional array of strings): Extends the set of managed capabilities beyond the baseline defined in `baselineCapabilitySet`. Default is an empty set. Valid values can be found [here](https://pkg.go.dev/github.com/openshift/api/config/v1#ClusterVersionCapability). 78 ### IP networks 79 80 IP networks are represented as strings using [Classless Inter-Domain Routing (CIDR) notation][cidr-notation] with a traditional IP address or network number, followed by the "/" (slash) character, followed by a decimal value between 0 and 32 that describes the number of significant bits. 81 For example, 10.0.0.0/16 represents IP addresses 10.0.0.0 through 10.0.255.255. 82 83 ### Machine pools 84 85 The following machine-pool properties are available: 86 87 * `architecture` (optional string): Determines the instruction set architecture of the machines in the pool. Currently, heterogeneous clusters are not supported, so all pools must specify the same architecture. 88 Valid values are `amd64` (the default). 89 * `hyperthreading` (optional string): Determines the mode of hyperthreading that machines in the pool will utilize. 90 Valid values are `Enabled` (the default) and `Disabled`. 91 * `name` (required string): The name of the machine pool. 92 * `platform` (optional object): Platform-specific machine-pool configuration. 93 * `aws` (optional object): [AWS-specific properties](aws/customization.md#machine-pools). 94 * `azure` (optional object): [Azure-specific properties](azure/customization.md#machine-pools). 95 * `gcp` (optional object): [GCP-specific properties](gcp/customization.md#machine-pools). 96 * `openstack` (optional object): [OpenStack-specific properties](openstack/customization.md#machine-pools). 97 * `ovirt` (optional object): [oVirt-specific properties](ovirt/customization.md#machine-pools). 98 * `vsphere` (optional object): [vSphere-specific properties](vsphere/customization.md#machine-pools). 99 * `replicas` (optional integer): The machine count for the machine pool. 100 101 ### Examples 102 103 While all complete `install-config.yaml` will contain platform-specific sections, the following example fragments demonstrate platform-agnostic options: 104 105 ### Additional trust bundle 106 107 ```yaml 108 apiVersion: v1 109 additionalTrustBundle: | 110 -----BEGIN CERTIFICATE----- 111 ...base-64-encoded, DER Certificate Authority cert... 112 -----END CERTIFICATE----- 113 114 -----BEGIN CERTIFICATE----- 115 ...base-64-encoded, DER Certificate Authority cert... 116 -----END CERTIFICATE----- 117 baseDomain: example.com 118 metadata: 119 name: test-cluster 120 platform: ... 121 pullSecret: '{"auths": ...}' 122 sshKey: ssh-ed25519 AAAA... 123 ``` 124 125 ### Custom capabilities 126 127 An example install config where the user can specify a custom list of capabilities than a default set deployed by the installer. In this example, the user requested to use `None`, an empty set, as the baseline capability set but specified that the `openshift-samples` to be installed. 128 129 ```yaml 130 apiVersion: v1 131 baseDomain: example.com 132 metadata: 133 name: test-cluster 134 platform: ... 135 capabilities: 136 baselineCapabilitySet: None 137 additionalEnabledCapabilities: 138 - openshift-samples 139 sshKey: ... 140 ``` 141 142 ### Custom machine pools 143 144 An example install config with custom machine pools to grow the size of the worker pool and disable hyperthreading: 145 146 ```yaml 147 apiVersion: v1 148 baseDomain: example.com 149 controlPlane: 150 name: master 151 hyperthreading: Disabled 152 compute: 153 - name: worker 154 hyperthreading: Disabled 155 replicas: 5 156 metadata: 157 name: test-cluster 158 platform: ... 159 pullSecret: '{"auths": ...}' 160 sshKey: ssh-ed25519 AAAA... 161 ``` 162 163 ### Custom networking 164 165 An example install config with custom networking: 166 167 ```yaml 168 apiVersion: v1 169 baseDomain: example.com 170 metadata: 171 name: test-cluster 172 networking: 173 clusterNetwork: 174 - cidr: 10.128.0.0/14 175 hostPrefix: 23 176 machineNetwork: 177 - cidr: 10.0.0.0/16 178 networkType: OpenShiftSDN 179 serviceNetwork: 180 - 172.30.0.0/16 181 platform: ... 182 pullSecret: '{"auths": ...}' 183 sshKey: ssh-ed25519 AAAA... 184 ``` 185 186 ### Image content sources 187 188 An example install config with custom image content sources: 189 190 ```yaml 191 apiVersion: v1 192 baseDomain: example.com 193 metadata: 194 name: test-cluster 195 imageContentSources: 196 - mirrors: 197 - registry.example.com/ocp4/openshift4 198 source: quay.io/openshift-release-dev/ocp-release-nightly 199 - mirrors: 200 - registry.example.com/ocp4/openshift4 201 source: quay.io/openshift-release-dev/ocp-v4.0-art-dev 202 platform: ... 203 pullSecret: '{"auths": ...}' 204 sshKey: ssh-ed25519 AAAA... 205 ``` 206 207 That configuration is compatible with mirrored releases created with `mirror` commands like: 208 209 ```console 210 $ oc adm release mirror \ 211 > --from=quay.io/openshift-release-dev/ocp-release-nightly:4.2.0-0.nightly-XXXXXX \ 212 > --to=registry.example.com/ocp4/openshift4 \ 213 > --to-release-image=registry.example.com/ocp4/openshift4:4.2.0-0.nightly-XXXXXX 214 ... 215 Success 216 Update image: registry.example.com/ocp4/openshift4:4.2.0-0.nightly-2019-09-11-114314 217 Mirror prefix: registry.example.com/ocp4/openshift4 218 219 To use the new mirrored repository to install, add the following section to the install-config.yaml: 220 221 imageContentSources: 222 - mirrors: 223 - registry.example.com/ocp4/openshift4 224 source: quay.io/openshift-release-dev/ocp-release-nightly 225 - mirrors: 226 - registry.example.com/ocp4/openshift4 227 source: quay.io/openshift-release-dev/ocp-v4.0-art-dev 228 ... 229 ``` 230 231 If your mirror(s) are signed by a certificate authority which RHCOS does not trust by default, you may also wish to configure [an additional trust bundle](#additional-trust-bundle). 232 233 ### Proxy 234 235 An example install config routing outgoing traffic through a proxy: 236 237 ```yaml 238 apiVersion: v1 239 baseDomain: example.com 240 metadata: 241 name: test-cluster 242 proxy: 243 httpsProxy: https://username:password@proxy.example.com:123/ 244 httpProxy: https://username:password@proxy.example.com:123/ 245 noProxy: 123.example.com,10.88.0.0/16 246 platform: ... 247 pullSecret: '{"auths": ...}' 248 sshKey: ssh-ed25519 AAAA... 249 ``` 250 251 If your proxy certificate is signed by a certificate authority which RHCOS does not trust by default, you may also wish to configure [an additional trust bundle](#additional-trust-bundle). 252 If `additionalTrustBundle` and at least one `proxy` setting are configured, the `cluster` [Proxy object][proxy] will be configured with [`trustedCA`][proxy-trusted-ca] referencing the additional trust bundle. 253 254 ## Kubernetes Customization (unvalidated) 255 256 In addition to customizing OpenShift and aspects of the underlying platform, the installer allows arbitrary modification to the Kubernetes objects that are injected into the cluster. Note that there is currently no validation on the modifications that are made, so it is possible that the changes will result in a non-functioning cluster. The Kubernetes manifests can be viewed and modified using the `manifests` and `manifest-templates` targets. 257 258 The `manifests` target will render the manifest templates and output the result into the asset directory. Perhaps the most common use for this target is to include additional manifests in the initial installation. These manifests could be added after the installation as a "Day 2" operation, but there may be cases where they are necessary beforehand. 259 260 The `manifest-templates` target will output the unrendered manifest templates into the asset directory. This allows modification to the templates before they have been rendered, which may be useful to users who wish to reuse the templates between cluster deployments. 261 262 ### Install Time Customization for Machine Configuration 263 264 **IMPORTANT**: 265 266 - These customizations require using the `manifests` target that does not provide compatibility guarantees. 267 268 In most cases, user applications should be run on the cluster via Kubernetes workload objects (e.g. DaemonSet, Deployment, etc). For example, DaemonSets are the most stable way to run a logging agent on all hosts. However, there may be some cases where these workloads need to be executed prior to the node joining the Kubernetes cluster. For example, a compliance mandate like "the user must run auditing tools as soon as the operating system comes up" might require a custom systemd unit for an auditing container in the Ignition config for some or all nodes. 269 270 Further, some aspects of RHEL CoreOS machines (usually kernel arguments such as `nosmt` for disabling hyperthreading) may need to be configured before user workloads land on a system. 271 272 The configuration of machines in OpenShift is controlled using `MachineConfig` objects and what configuration is applied to a machine in the OpenShift cluster is based on the [MachineConfigPool][machine-config-pool] objects. For these "day 1" cases, `MachineConfig` objects can be provided as additional manifests. 273 274 1. `openshift-install --dir $INSTALL_DIR create manifests` 275 276 2. Copy files with `MachineConfig` objects to `$INSTALL_DIR/openshift/` directory. 277 278 These custom `MachineConfig` objects are black boxes to the installer and the installer only plays the role of `oc create -f <custom-machine-config-object>` early enough into cluster bootstrap to make sure the configuration is used by the [MachineConfigOperator][machine-config-operator]. 279 280 3. `openshift-install --dir $INSTALL_DIR create cluster` 281 282 #### Control plane with no Taints 283 284 All control plane nodes by default register with a taint `node-role.kubernetes.io/master=:NoSchedule` making them unschedulable by most normal workloads. An installation that requires the control plane to boot without that taint can push a custom `MachineConfig` object with a `kubelet.service` that doesn't include the taint. 285 286 For example: 287 288 1. Run `manifests` target to create all the manifests. 289 290 ```console 291 $ mkdir no-taint-cluster 292 293 $ cp aws-install-config.yaml no-taint-cluster/install-config.yaml 294 295 $ openshift-install --dir no-taint-cluster create manifests 296 INFO Consuming "Install Config" from target directory 297 298 $ ls -l no-taint-cluster/** 299 no-taint-cluster/manifests: 300 total 68 301 -rw-r--r--. 1 xxxxx xxxxx 169 Feb 28 10:54 04-openshift-machine-config-operator.yaml 302 -rw-r--r--. 1 xxxxx xxxxx 1589 Feb 28 10:54 cluster-config.yaml 303 -rw-r--r--. 1 xxxxx xxxxx 149 Feb 28 10:54 cluster-dns-02-config.yml 304 -rw-r--r--. 1 xxxxx xxxxx 243 Feb 28 10:54 cluster-infrastructure-02-config.yml 305 -rw-r--r--. 1 xxxxx xxxxx 154 Feb 28 10:54 cluster-ingress-02-config.yml 306 -rw-r--r--. 1 xxxxx xxxxx 557 Feb 28 10:54 cluster-network-01-crd.yml 307 -rw-r--r--. 1 xxxxx xxxxx 327 Feb 28 10:54 cluster-network-02-config.yml 308 -rw-r--r--. 1 xxxxx xxxxx 264 Feb 28 10:54 cvo-overrides.yaml 309 -rw-r--r--. 1 xxxxx xxxxx 118 Feb 28 10:54 kube-cloud-config.yaml 310 -rw-r--r--. 1 xxxxx xxxxx 1304 Feb 28 10:54 kube-system-configmap-root-ca.yaml 311 -rw-r--r--. 1 xxxxx xxxxx 4030 Feb 28 10:54 machine-config-server-tls-secret.yaml 312 -rw-r--r--. 1 xxxxx xxxxx 856 Feb 28 10:54 pull.json 313 314 no-taint-cluster/openshift: 315 total 28 316 -rw-r--r--. 1 xxxxx xxxxx 293 Feb 28 10:54 99_binding-discovery.yaml 317 -rw-r--r--. 1 xxxxx xxxxx 181 Feb 28 10:54 99_kubeadmin-password-secret.yaml 318 -rw-r--r--. 1 xxxxx xxxxx 330 Feb 28 10:54 99_openshift-cluster-api_cluster.yaml 319 -rw-r--r--. 1 xxxxx xxxxx 1015 Feb 28 10:54 99_openshift-cluster-api_master-machines-0.yaml 320 -rw-r--r--. 1 xxxxx xxxxx 2655 Feb 28 10:54 99_openshift-cluster-api_master-user-data-secret.yaml 321 -rw-r--r--. 1 xxxxx xxxxx 1750 Feb 28 10:54 99_openshift-cluster-api_worker-machineset.yaml 322 -rw-r--r--. 1 xxxxx xxxxx 2655 Feb 28 10:54 99_openshift-cluster-api_worker-user-data-secret.yaml 323 ``` 324 325 2. Create a `MachineConfig` that includes `kubelet.service` that has no taints. 326 327 ```sh 328 cat > no-taint-cluster/openshift/99-master-kubelet-no-taint.yaml <<EOF 329 apiVersion: machineconfiguration.openshift.io/v1 330 kind: MachineConfig 331 metadata: 332 labels: 333 machineconfiguration.openshift.io/role: master 334 name: 02-master-kubelet 335 spec: 336 config: 337 ignition: 338 version: 3.1.0 339 systemd: 340 units: 341 - contents: | 342 [Unit] 343 Description=Kubernetes Kubelet 344 Wants=rpc-statd.service 345 346 [Service] 347 Type=notify 348 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests 349 ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state 350 EnvironmentFile=-/etc/kubernetes/kubelet-workaround 351 EnvironmentFile=-/etc/kubernetes/kubelet-env 352 353 ExecStart=/usr/bin/kubelet \ 354 --config=/etc/kubernetes/kubelet.conf \ 355 --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \ 356 --rotate-certificates \ 357 --kubeconfig=/var/lib/kubelet/kubeconfig \ 358 --container-runtime-endpoint=/var/run/crio/crio.sock \ 359 --allow-privileged \ 360 --node-labels=node-role.kubernetes.io/master \ 361 --minimum-container-ttl-duration=6m0s \ 362 --client-ca-file=/etc/kubernetes/ca.crt \ 363 --cloud-provider=aws \ 364 --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \ 365 \ 366 --anonymous-auth=false \ 367 368 Restart=always 369 RestartSec=10 370 371 [Install] 372 WantedBy=multi-user.target 373 enabled: true 374 name: kubelet.service 375 EOF 376 ``` 377 378 `machineconfiguration.openshift.io/role: master` label attaches this `MachineConfig` to the [master][master-machine-config-pool] `MachineConfigPool`. The default configuration for the `kubelet.service` includes the [taint][default-kubelet-service-taint]. 379 380 3. Run `cluster` target to create the cluster using the custom manifests. 381 382 ```console 383 $ openshift-install --dir no-taint-cluster create cluster 384 INFO Consuming "Openshift Manifests" from target directory 385 INFO Consuming "Master Machines" from target directory 386 INFO Consuming "Common Manifests" from target directory 387 INFO Creating infrastructure resources... 388 INFO Waiting up to 30m0s for the Kubernetes API at https://api.test-cluster.example.com:6443... 389 ... 390 ``` 391 392 Check that no control plane nodes registered with taints: 393 394 ```console 395 $ oc --kubeconfig no-taint-cluster/auth/kubeconfig get nodes -ojson | jq '.items[] | select(.metadata.labels."node-role.kubernetes.io/master" == "") | .spec.taints' 396 null 397 ``` 398 399 Check that the `02-master-kubelet` `MachineConfig` exists in the cluster: 400 401 ```console 402 oc --kubeconfig no-taint-cluster/auth/kubeconfig get machineconfigs 403 NAME GENERATEDBYCONTROLLER IGNITIONVERSION CREATED 404 00-master 3.11.0-744-g5b05d9d3-dirty 3.1.0 137m 405 00-master-ssh 3.11.0-744-g5b05d9d3-dirty 137m 406 00-worker 3.11.0-744-g5b05d9d3-dirty 3.1.0 137m 407 00-worker-ssh 3.11.0-744-g5b05d9d3-dirty 137m 408 01-master-container-runtime 3.11.0-744-g5b05d9d3-dirty 3.1.0 137m 409 01-master-kubelet 3.11.0-744-g5b05d9d3-dirty 3.1.0 137m 410 02-master-kubelet 3.1.0 137m 411 01-worker-container-runtime 3.11.0-744-g5b05d9d3-dirty 3.1.0 137m 412 01-worker-kubelet 3.11.0-744-g5b05d9d3-dirty 3.1.0 137m 413 99-master-3c81ffa3-3b8d-11e9-ac1e-52fdfc072182-registries 3.11.0-744-g5b05d9d3-dirty 133m 414 99-worker-3c83a226-3b8d-11e9-ac1e-52fdfc072182-registries 3.11.0-744-g5b05d9d3-dirty 133m 415 master-55491738d7cd1ad6c72891e77c35e024 3.11.0-744-g5b05d9d3-dirty 3.1.0 137m 416 worker-edab0895c59dba7a566f4b955d87d964 3.11.0-744-g5b05d9d3-dirty 3.1.0 137m 417 ``` 418 419 #### Nodes with Custom Kernel Arguments 420 421 Custom kernel arguments can be applied to through manifests as an installer operation, and can also be applied as a [MachineConfig][machine-config] as a day 2 operation. The kernel arguments are applied upon boot and will be honored by the Machine-Config-Operator from then on. 422 423 Example application of `loglevel=7` (change Linux kernel log level to KERN_DEBUG) for master nodes: 424 425 1. Run `manifests` target to create all the manifests. 426 427 ```console 428 $ mkdir log_debug_cluster 429 430 $ openshift-install --dir log_debug_cluster create manifests 431 ... 432 433 $ ls -l log_debug_cluster/openshift 434 ... 435 99_openshift-machineconfig_99-master-ssh.yaml 436 99_openshift-machineconfig_99-worker-ssh.yaml 437 ... 438 ``` 439 440 2. . Create a `MachineConfig` that adds a kernel argument to change log level: 441 442 ```sh 443 cat > log_debug_cluster/openshift/99-master-kargs-loglevel.yaml <<EOF 444 apiVersion: machineconfiguration.openshift.io/v1 445 kind: MachineConfig 446 metadata: 447 labels: 448 machineconfiguration.openshift.io/role: "master" 449 name: 99-master-kargs-loglevel 450 spec: 451 config: 452 ignition: 453 version: 3.1.0 454 kernelArguments: 455 - 'loglevel=7' 456 EOF 457 ``` 458 459 3. Run `cluster` target to create the cluster using the custom manifests. 460 461 ```console 462 $ openshift-install --dir log_debug_cluster create cluster 463 ... 464 ``` 465 466 Check that the machineconfig has the kernel arguments applied 467 468 ```console 469 $ oc --kubeconfig log_debug_cluster/auth/kubeconfig get machineconfigs 470 NAME GENERATEDBYCONTROLLER IGNITIONVERSION CREATED 471 99-master-kargs-loglevel bd846958bc95d049547164046a962054fca093df 3.1.0 26h 472 99-master-ssh bd846958bc95d049547164046a962054fca093df 3.1.0 26h 473 ... 474 rendered-master-5f4a5bd806567871be1b608474eca373 bd846958bc95d049547164046a962054fca093df 3.1.0 26h 475 476 $ oc describe machineconfig/rendered-master-5f4a5bd806567871be1b608474eca373 | grep -A 1 "Kernel Arguments" 477 Kernel Arguments: 478 loglevel=7 479 ``` 480 481 If you wish to confirm the kernel argument is indeed being applied on the system, you can `oc debug` into a node and check with `rpm-ostree kargs`. 482 483 #### Switching RHCOS host kernel using KernelType 484 485 With OCP 4.4 and onward release, it is possible to switch from traditional to Real Time (RT) kernel on RHCOS node. During install time, switching to RT kernel can be done through manifests as an installer operation. See [customizing MachineConfig](#install-time-customization-for-machine-configuration) to configure kernelType during install time. To set kernelType as day 2 operation, see [MachineConfiguration](https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfiguration.md#kernelType) doc. 486 487 Example for switching to RT kernel on worker nodes during initial cluster install: 488 489 1. Run `manifests` target to create all the manifests. 490 491 ```console 492 $ mkdir realtime_kernel 493 $ openshift-install --dir realtime_kernel create manifests 494 ``` 495 496 2. Create a `MachineConfig` that sets `kernelType` to `realtime`: 497 498 ```sh 499 cat > realtime_kernel/openshift/99-worker-kerneltype.yaml <<EOF 500 apiVersion: machineconfiguration.openshift.io/v1 501 kind: MachineConfig 502 metadata: 503 labels: 504 machineconfiguration.openshift.io/role: "worker" 505 name: 99-worker-kerneltype 506 spec: 507 config: 508 ignition: 509 version: 3.1.0 510 kernelType: realtime 511 EOF 512 ``` 513 514 3. Run `cluster` target to create the cluster using the custom manifests. 515 516 ```console 517 $ openshift-install --dir realtime_kernel create cluster 518 ``` 519 520 Check that the MachineConfig has the kernelType applied 521 522 ```console 523 $ oc --kubeconfig realtime_kernel/auth/kubeconfig get machineconfigs 524 NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 525 ... 526 99-worker-kerneltype 3.1.0 80m 527 99-worker-ssh 3.1.0 80m 528 rendered-worker-853ba9bf0337db528a857a9c7380b95a 6306be9274cd3052f5075c81fa447c7895b7b9f4 3.1.0 78m 529 ... 530 531 4. To confirm that worker node has switched to RT kernel, access one of the worker node and run `uname -a` 532 533 ```console 534 $ oc --kubeconfig realtime_kernel/auth/kubeconfig debug node/<worker_node> 535 ... 536 sh-4.2# uname -a 537 Linux <worker_node> 4.18.0-147.3.1.rt24.96.el8_1.x86_64 #1 SMP PREEMPT RT Wed Nov 27 18:29:55 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux 538 ``` 539 540 **Note:** The RT kernel lowers throughput (performance) in return for improved worst-case latency bounds. This feature is intended only for use cases that require consistent low latency. For more information, see the [Linux Foundation wiki](https://wiki.linuxfoundation.org/realtime/start) and the [RHEL RT portal](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/8/). 541 542 #### Enabling RHCOS Extensions 543 RHCOS is a minimal OCP focused OS which provides capabilities common across all the platforms. With extensions support, beginning in OCP 4.6 and onward, users can enable a limited set of additional functionality on the RHCOS nodes. In OCP 4.6 the supported extension is `usbguard`. 544 545 Extensions can be installed by creating a MachineConfig object. It can be enabled during cluster installation as well as later on. See [customizing MachineConfig](#install-time-customization-for-machine-configuration) to enable an extension during install time. For day2 install, see [MachineConfiguration](https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfiguration.md#RHCO-Extensions) doc. 546 547 Example MachineConfig to install usbguard on worker nodes: 548 549 ```yaml 550 apiVersion: machineconfiguration.openshift.io/v1 551 kind: MachineConfig 552 metadata: 553 labels: 554 machineconfiguration.openshift.io/role: worker 555 name: worker-extensions 556 spec: 557 config: 558 ignition: 559 version: 3.1.0 560 extensions: 561 - usbguard 562 ``` 563 564 ## OS Customization (unvalidated) 565 566 In rare circumstances, certain modifications to the bootstrap and other machines may be necessary. The installer provides the "ignition-configs" target, which allows arbitrary modification to the [Ignition Configs][ignition] used to boot these machines. Note that there is currently no validation on the modifications that are made, so it is possible that the changes will result in a non-functioning cluster. 567 568 An example `worker.ign` is shown below. It has been modified to increase the HTTP timeouts used when fetching the generated worker config from the cluster. This isn't likely to be useful, but it does demonstrate what is possible. 569 570 ```json ignition 571 { 572 "ignition": { 573 "version": "3.1.0", 574 "config": { 575 "merge": [{ 576 "source": "https://test-cluster-api.example.com:22623/config/worker" 577 }] 578 }, 579 "security": { 580 "tls": { 581 "certificateAuthorities": [{ 582 "source": "data:text/plain;charset=utf-8;base64,LS0tLS1CRU..." 583 }] 584 } 585 }, 586 "timeouts": { 587 "httpResponseHeaders": 120 588 } 589 }, 590 "passwd": { 591 "users": [{ 592 "name": "core", 593 "sshAuthorizedKeys": [ 594 "ssh-ed25519 AAAA..." 595 ] 596 }] 597 } 598 } 599 ``` 600 601 [cidr-notation]: https://tools.ietf.org/html/rfc4632#section-3.1 602 [default-kubelet-service-taint]: https://github.com/openshift/machine-config-operator/blob/ad4bb0cdd514cf902e04189e0f8e146dde5affb0/templates/master/01-master-kubelet/_base/units/kubelet.service.yaml#L44 603 [ignition]: https://coreos.com/ignition/docs/latest/ 604 [machine-config-operator]: https://github.com/openshift/machine-config-operator#machine-config-operator 605 [machine-config-pool]: https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfigController.md#machinepool 606 [machine-config]: https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfiguration.md 607 [master-machine-config-pool]: https://github.com/openshift/machine-config-operator/blob/master/manifests/master.machineconfigpool.yaml 608 [ovn-kubernetes]: https://github.com/openshift/ovn-kubernetes 609 [openshift-sdn]: https://github.com/openshift/sdn 610 [proxy]: https://github.com/openshift/api/blob/f2a771e1a90ceb4e65f1ca2c8b11fc1ac6a66da8/config/v1/types_proxy.go#L11 611 [proxy-trusted-ca]: https://github.com/openshift/api/blob/f2a771e1a90ceb4e65f1ca2c8b11fc1ac6a66da8/config/v1/types_proxy.go#L44-L69