github.com/k8snetworkplumbingwg/sriov-network-operator@v1.2.1-0.20240408194816-2d2e5a45d453/doc/vdpa.md (about)

     1  # VDPA
     2  
     3  Virtual data path acceleration (vDPA) in essence is an approach to standardize 
     4  the NIC SRIOV data plane using the virtio ring layout and placing a single standard 
     5  virtio driver in the guest/pod. It’s decoupled from any vendor implementation while 
     6  adding a generic control plane and SW infrastructure to support it. Given that it’s 
     7  an abstraction layer on top of SRIOV (Single Root I/O Virtualization) it is also 
     8  future proof to support emerging technologies such as scalable IOV.
     9  
    10  Main aspects of this solution:
    11  - primary interface in the pod is configured as a virtio/vDPA device
    12  - OVS HW offload enabled
    13  - NIC configured in switchdev mode
    14  - OVN-kubernetes CNI
    15  
    16  ## Supported Ethernet controllers
    17  
    18  The following manufacturers/NICs are known to work:
    19  
    20  - Mellanox ConnectX-6 Dx
    21  
    22  ## Prerequisites
    23  
    24  - OpenVswitch installed
    25  - Network Manager installed
    26  
    27  ### Deploy SriovNetworkNodePolicy
    28  
    29  ```yaml
    30  apiVersion: sriovnetwork.openshift.io/v1
    31  kind: SriovNetworkNodePolicy
    32  metadata:
    33    name: vdpa-policy
    34    namespace: openshift-sriov-network-operator
    35  spec:
    36    nodeSelector:
    37      feature.node.kubernetes.io/network-sriov.capable: "true"
    38    resourceName: mlxnics
    39    priority: 10
    40    numVfs: 2
    41    nicSelector:
    42      vendor: "15b3"
    43      deviceID: "101d"
    44      rootDevices:
    45      - 0000:65:00.0
    46      - 0000:65:00.1
    47    deviceType: netdevice
    48    eSwitchMode: switchdev
    49    vdpaType: virtio
    50  ```
    51  
    52  ### Create NetworkAttachmentDefinition CRD with OVN-K CNI config
    53  
    54  ```yaml
    55  apiVersion: "k8s.cni.cncf.io/v1"
    56  kind: NetworkAttachmentDefinition
    57  metadata:
    58    name: ovn-kubernetes-a
    59    namespace: kube-system
    60    annotations:
    61      k8s.v1.cni.cncf.io/resourceName: openshift.io/mlxnics
    62  spec:
    63    config: '{
    64        "cniVersion": "0.3.1",
    65        "name":"ovn-kubernetes-a",
    66        "type": "ovn-k8s-cni-overlay",
    67        "ipam": {},
    68        "dns": {}
    69      }'
    70  ```
    71  
    72  ### Deploy POD with virtio/vDPA
    73  
    74  Create POD spec and request a VF
    75  
    76  ```yaml
    77  apiVersion: v1
    78  kind: Pod
    79  metadata:
    80    name: vdpa-pod1
    81    namespace: vdpa
    82    annotations:
    83      v1.multus-cni.io/default-network: kube-system/ovn-kubernetes-a
    84  spec:
    85    containers:
    86    - name: vdpa-pod
    87      image: networkstatic/iperf3
    88      imagePullPolicy: IfNotPresent
    89      securityContext:
    90        privileged: true
    91      command:
    92        - sleep
    93        - "3600"
    94  ```
    95  
    96  ## Verify vDPA is Working
    97  
    98  Run ethtool to verify the correctness of the virtio/vDPA interface in the pod:
    99  
   100  ```bash
   101  kubectl exec -it vdpa-pod1 -- ethtool -i eth0
   102  ```
   103  
   104  ```text
   105  driver: virtio_net
   106  version: 1.0.0
   107  firmware-version: 
   108  expansion-rom-version: 
   109  bus-info: vdpa:0000:65:00.4
   110  supports-statistics: yes
   111  supports-test: no
   112  supports-eeprom-access: no
   113  supports-register-dump: no
   114  supports-priv-flags: no
   115  ```
   116  
   117  Check connectivity between the pods:
   118  
   119  ```bash
   120  kubectl exec -it vdpa-pod1 -- ping <other-pod-ip-address>
   121  ```
   122  
   123  Check datapath rules are offloaded
   124  
   125  ```text
   126  ovs-appctl dpctl/dump-flows -m type=offloaded
   127  ufid:30ae9875-d107-46a1-b8cb-cdaafd89d440, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ac63a8e99747603),packet_type(ns=0/0,id=0/0),eth(src=0a:58:c0:a8:00:06,dst=0a:58:c0:a8:00:05),eth_type(0x0800),ipv4(src=192.168.0.6,dst=192.168.0.5,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=57426), packets:7, bytes:396, used:2.310s, offloaded:yes, dp:tc, actions:ct(zone=15,nat),recirc(0x440)
   128  ufid:b895ff1d-4085-4199-8b49-0b6fc9e54a29, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ac63a8e99747603),packet_type(ns=0/0,id=0/0),eth(src=0a:58:c0:a8:00:06,dst=0a:58:c0:a8:00:05),eth_type(0x0800),ipv4(src=192.168.0.6,dst=192.168.0.5,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=57428), packets:46244, bytes:3052518, used:0.260s, offloaded:yes, dp:tc, actions:ct(zone=15,nat),recirc(0x442)
   129  ufid:ea3e4350-70ac-46aa-8fba-fff5ee54dc32, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ac63a8e99747603),packet_type(ns=0/0,id=0/0),eth(src=0a:58:c0:a8:00:06,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=0/0,tclass=0/0,hlimit=0/0,frag=no), packets:0, bytes:0, used:6.270s, offloaded:yes, dp:tc, actions:drop
   130  ufid:a23365ae-26cd-4bbb-85b5-4299c05a350d, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(8630f7a90005cdd),packet_type(ns=0/0,id=0/0),eth(src=0a:58:c0:a8:00:05,dst=0a:58:c0:a8:00:06),eth_type(0x0800),ipv4(src=192.168.0.5,dst=192.168.0.6,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=5201), packets:2156610, bytes:3049431372, used:0.260s, offloaded:yes, dp:tc, actions:ct(zone=14,nat),recirc(0x43e)
   131  ```