github.com/openshift/installer@v1.4.17/docs/user/openstack/ovs-dpdk.md (about)

     1  # Installing a cluster on OpenStack that supports DPDK-connected compute machines
     2  
     3  If the underlying OpenStack deployment has Open vSwitch with Data Plane Development Kit (OVS-DPDK) support enabled,
     4  an OpenShift cluster can take advantage of the feature by providing access to [poll mode drivers][1].
     5  
     6  ## Pre-requisites
     7  
     8  The following steps might be required to be checked before starting the deployment of OpenShift.
     9  
    10  ### Selecting hugepages size and CPU policy for OpenShift virtual machines
    11  
    12  You require hugepages for every OCP cluster virtual machine running on the hosts with OVS-DPDK.
    13  If hugepages are not present in the flavor, the machines will get created but the network interface won't work.
    14  You must check with your OpenStack administrator the size and number of available hugepages which can be used per node.
    15  
    16  This may be accomplished in two ways:
    17  
    18  - The flavors used for the OCP are defined with appropriate values. For example:
    19  
    20      ``` bash
    21      openstack flavor create m1.xlarge.nfv --ram 16384 --disk 40 --vcpus 8
    22      openstack flavor set --property hw:cpu_policy=dedicated --property hw:mem_page_size=large m1.xlarge.nfv
    23      ```
    24  
    25      This method requires administrative access the OpenStack deployment because flavors need to be created and modified.
    26  
    27  - The image properties maybe be defined in the install-config.yaml using [platform.openstack.clusterOSImageProperties](customization.md).
    28    For example:
    29  
    30      ``` yaml
    31      platform:
    32        openstack:
    33           clusterOSImageProperties:
    34              - hw_cpu_policy: dedicated
    35              - hw_mem_page_size: large
    36      ```
    37  
    38  If using image properties, then the flavor used to deploy the virtual machines must allow the image property values selected.
    39  For example, if you attempt to deploy a virtual machine using an image with the property `hw_mem_page_size: 1GB`,
    40  but the flavor being used is set with `--property hw:mem_page_size=small`,
    41  then the deployment of the virtual machine will fail because the flavor does not allow for using a 1GB page size (large).
    42  The flavor property would need to either not be set, which is the same as `--property hw:mem_page_size=any`,
    43  or the flavor property should be set to `--property hw:mem_page_size=large`.
    44  
    45  ### Create an aggregate group (optional)
    46  
    47  If not all your OpenStack compute nodes are configured for OVS-DPDK, you must create an aggregate group and add the compatible nodes into it.
    48  
    49  Create the group, add the nodes, set the property and update the flavor:
    50  
    51  ``` bash
    52  openstack aggregate create dpdk_group
    53  openstack aggregate add host dpdk_group <compute-hosts>
    54  openstack aggregate set --property dpdk=true dpdk_group
    55  openstack flavor set <flavor> --property aggregate_instance_extra_specs:dpdk=true
    56  ```
    57  
    58  ### Enable multiqueue (optional)
    59  
    60  If you use multiqueue with OVS-DPDK, set the `hw_vif_multiqueue_enabled` property on the image used for OCP:
    61  
    62  ```
    63  openstack image set --property hw_vif_multiqueue_enabled=true <coreos_image>
    64  ```
    65  
    66  where `coreos_image` is the name of the RHCOS image to be used.
    67  This is the value later entered for `platform.openstack.clusterOSImage`.
    68  This can also be accomplished by using `platform.openstack.clusterOSImageProperties` in `install-config.yaml`:
    69  
    70  ``` yaml
    71  platform:
    72    openstack:
    73      clusterOSImageProperties:
    74        - hw_cpu_policy: dedicated
    75        - hw_mem_page_size: 1GB
    76        - hw_numa_nodes: '1'
    77        - hw_vif_multiqueue_enabled: true
    78  ```
    79  
    80  ### Create additional network to attach poll mode driver interfaces
    81  
    82  The `poll_mode_driver` network will be where the ports using poll mode drivers will be attached to.
    83  In most cases, this will be a Neutron provider network.
    84  
    85  ``` bash
    86  openstack network create dpdk_network --provider-physical-network dpdk-phy1 --provider-network-type vlan --provider-segment <VLAN-ID>
    87  openstack subnet create dpdk_subnet --network dpdk_network --subnet-range 192.0.2.0/24 --dhcp
    88  ```
    89  
    90  The ID of the network will be later used to inform the cluster which ports to attach the poll mode drivers to.
    91  
    92  ## OpenShift cluster deployment
    93  
    94  ### Deploy the cluster
    95  
    96  1. Create and edit the `install-config.yaml`.
    97  
    98      ``` bash
    99      openshift-install create install-config --log-level=debug
   100      ```
   101  
   102      Edit the `install-config.yaml`, providing the following values:
   103  
   104      - `compute.platform.openstack.additionalNetworkIDs`
   105  
   106          The ID of the Neutron network where the ports will use the poll mode drivers will be attached to.
   107  
   108  2. If required, [create the manifests](../customization.md#install-time-customization-for-machine-configuration) to bind the PMD drivers to ports on the desired nodes.
   109  
   110      Using a manifest at this stage only allows the selection of all control-plane nodes and/or all worker nodes.
   111      It is not possible to select a subgroup of workers or control-plane nodes.
   112  
   113  3. Create the cluster.
   114  
   115      ``` bash
   116      openshift-install create cluster --log-level=debug
   117      ```
   118  
   119  ### Create a performance profile
   120  
   121  To get the best performance out of the workers attached to the VFIO interface, the correct settings for hugepages and other relevant settings need to be used.
   122  The node-tuning-operator component is used to configure these settings on the desired worker nodes.
   123  
   124  The following is an example of the performance profile that should be applied.
   125  The user should provide the values for kernel args, `HUGEPAGES`, `CPU_ISOLATED` and `CPU_RESERVED` depending on their system:
   126  
   127  ``` yaml
   128  apiVersion: performance.openshift.io/v2
   129  kind: PerformanceProfile
   130  metadata:
   131    name: cnf-performanceprofile
   132  spec:
   133    additionalKernelArgs:
   134      - nmi_watchdog=0
   135      - audit=0
   136      - mce=off
   137      - processor.max_cstate=1
   138      - idle=poll
   139      - intel_idle.max_cstate=0
   140      - default_hugepagesz=1GB
   141      - hugepagesz=1G
   142      - intel_iommu=on
   143    cpu:
   144      isolated: <CPU_ISOLATED>
   145      reserved: <CPU_RESERVED>
   146    hugepages:
   147      defaultHugepagesSize: 1G
   148      pages:
   149        - count: <HUGEPAGES>
   150          node: 0
   151          size: 1G
   152    nodeSelector:
   153      node-role.kubernetes.io/worker: ''
   154    realTimeKernel:
   155      enabled: false
   156  ```
   157  
   158  ### Label your CNF workers
   159  
   160  Once the CNF workers are deployed, you must label them SR-IOV capable:
   161  
   162  ```sh
   163  oc label node <node-name> feature.node.kubernetes.io/network-sriov.capable="true"
   164  ```
   165  
   166  These workers will be used to run the CNF workloads.
   167  
   168  ### Install the SRIOV Network Operator and configure a network device
   169  
   170  You must install the SR-IOV Network Operator. To install the Operator, you will need access to an account on your OpenShift cluster that has `cluster-admin` privileges. After you log in to the account, [install the Operator][2].
   171  
   172  Then, [configure your SR-IOV network device][3]. Note that only `netFilter` needs to be used from the `nicSelector`, as we'll give the Neutron network ID used for DPDK traffic.
   173  
   174  Example of `SriovNetworkNodePolicy` named `dpdk1`:
   175  
   176  ```
   177  apiVersion: sriovnetwork.openshift.io/v1
   178  kind: SriovNetworkNodePolicy
   179  metadata:
   180    name: dpdk1
   181    namespace: openshift-sriov-network-operator
   182  spec:
   183    deviceType: vfio-pci
   184    isRdma: false
   185    nicSelector:
   186      netFilter: openstack/NetworkID:55a54d05-9ec1-4051-8adb-1b5a7be4f1b6
   187    nodeSelector:
   188      feature.node.kubernetes.io/network-sriov.capable: 'true'
   189    numVfs: 1
   190    priority: 99
   191    resourceName: dpdk1
   192  ```
   193  
   194  Note: If the network device plugged to the network is not from Intel and is from Mellanox, then `deviceType` must be set to `netdevice` and `isRdma` set to `true`. 
   195  
   196  The SR-IOV network operator will automatically discover the devices connected on that network for each worker, and make them available for use by the CNF pods later.
   197  
   198  
   199  ## Deploy a testpmd pod
   200  
   201  This pod is an example how we can create a container that uses the hugepages, the reserved CPUs and the DPDK port:
   202  
   203  ```
   204  apiVersion: v1
   205  kind: Pod
   206  metadata:
   207    name: testpmd-dpdk
   208    namespace: mynamespace
   209  spec:
   210    containers:
   211    - name: testpmd
   212      command: ["sleep", "99999"]
   213      image: registry.redhat.io/openshift4/dpdk-base-rhel8:v4.9
   214      securityContext:
   215        capabilities:
   216          add: ["IPC_LOCK","SYS_ADMIN"]
   217        privileged: true
   218        runAsUser: 0
   219      resources:
   220        requests:
   221          memory: 1000Mi
   222          hugepages-1Gi: 1Gi
   223          cpu: '2'
   224          openshift.io/dpdk1: 1
   225        limits:
   226          hugepages-1Gi: 1Gi
   227          cpu: '2'
   228          memory: 1000Mi
   229          openshift.io/dpdk1: 1
   230      volumeMounts:
   231        - mountPath: /dev/hugepages
   232          name: hugepage
   233          readOnly: False
   234    volumes:
   235    - name: hugepage
   236      emptyDir:
   237        medium: HugePages
   238  ```
   239  
   240  More examples are documented [here][4].
   241  
   242  [1]: https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html
   243  [2]: https://docs.openshift.com/container-platform/4.10/networking/hardware_networks/installing-sriov-operator.html
   244  [3]: https://docs.openshift.com/container-platform/4.10/networking/hardware_networks/configuring-sriov-device.html
   245  [4]: https://docs.openshift.com/container-platform/4.10/networking/hardware_networks/using-dpdk-and-rdma.html#example-vf-use-in-dpdk-mode-intel_using-dpdk-and-rdma