vHost User Client Mode in Open vSwitch* with DPDK

This article describes the concept of vHost User client mode, how it can be tested, and the benefits the feature brings to Open vSwitch* (OVS) with the Data Plane Development Kit (DPDK). This article was written with OVS users in mind who wish to know more about the feature and for users who may be configuring a virtualized OVS DPDK setup that uses vHost User ports as the guest access method for virtual machines (VMs) running in QEMU.

Note: vHost User client mode in OVS with DPDK is available on both the OVS master branch and the 2.6 release branch. Users can download the OVS master branch as a zip here or the 2.6 branch as a zip here. Installation steps for the master branch can be found here. 2.6 installation steps can be found here.

vHost User Client Mode

vHost User client mode was introduced in DPDK v16.07 to address a limitation in the DPDK, whereby if the vHost User backend (DPDK-based application such as OVS with DPDK) crashes or is restarted, VMs with DPDK vHost User ports cannot re-establish connectivity with the backend and are essentially rendered useless from a networking perspective. vHost User client mode solves this problem.

The vHost User standard uses a client-server model. The server creates and manages the vHost User sockets and the client connects to the sockets created by the server. Before the introduction of this feature, the only configuration used by OVS-DPDK had it acting as the server and QEMU* acting as the client. Figure 1 shows this configuration.

Figure 1.Typical Open vSwitch* with the Data Plane Development Kit (DPDK) configuration using OVS-DPDK vHost server mode and QEMU* using vHost client mode. The direction of the arrow indicates the client connecting to the server.

With this default configuration, if OVS-DPDK was reset the sockets would be destroyed, and when relaunched, connectivity could not be re-established with the VM. With client mode, OVS-DPDK acts as the vHost client and QEMU acts as the server. In this configuration, when the switch crashes, the sockets still exist as they are managed by QEMU. This allows OVS-DPDK to relaunch and reconnect to these sockets and thus resume normal connectivity. Figure 2 shows this configuration.

Figure 2.Open vSwitch* with the Data Plane Development Kit (DPDK) configuration using OVS-DPDK vHost client mode and QEMU* using vHost server mode. The direction of the arrow indicates the client connecting to the server.

As seen in Table 1, OVS-DPDK supports this feature by offering two different types of vHost User ports. The first, dpdkvhostuser operates in the standard server mode. The second, dpdkvhostuserclient operates in client mode as the name suggests.

Table 1. Types of vHost User ports offered by Open vSwitch* with the Data Plane Development Kit (DPDK) and their respective modes

The ability to reconnect is a very useful feature; it reduces the impact of catastrophic failure of the switch as the VMs connected to the switch do not need to be rebooted upon switch failure. It also makes other tasks more lightweight; for example, maintenance tasks such as updating the switch requires rebooting the switch, but no longer requires the VMs connected to the switch to be rebooted as well.

Test Environment

The following describes how to set up an OVS-DPDK configuration with one physical dpdk port and one vHost User dpdkvhostuserclient port, and QEMU in vHost server mode. Next, the steps to demonstrate and reconnect capability are described.

Figure 3 shows the test environment configuration.

Figure 3. Open vSwitch* with the Data Plane Development Kit (DPDK) configuration using OVS-DPDK vHost client mode and QEMU* using vHost server mode. The direction of the arrows denotes the flow of traffic. The tcpdump tool is being used on the guest to monitor incoming traffic on the eth0 interface.

Table 2 shows the hardware and software components used for this setup:

Table 2. Hardware and software components

Configuration Steps

This section describes how to build OVS with DPDK as described in the installation docs.

Configure the switch as described in the Test Environment section, with one physical dpdk port and one vHost User dpdkvhostuserclient port. Add a flow that directs traffic from the physical port (1) to the vHost User port (2):

ovs-vsctl add-br br0

ovs-vsctl set Bridge br0 datapath_type=netdev

ovs-vsctl add-port br0 dpdk0

ovs-vsctl set Interface dpdk0 type=dpdk

ovs-vsctl add-port br0 vhost0

ovs-vsctl set Interface vhost0 type=dpdkvhostuserclient

ovs-ofctl add-flow br0 in_port=1,action=output:2

Set the location of the socket:

ovs-vsctl set Interface vhost0 options:vhost-server-path="/tmp/sock0"

Logs similar to the following should be printed:

VHOST_CONFIG: vhost-user client: socket created, fd: 28

VHOST_CONFIG: failed to connect to /tmp/sock0: No such file or directory

VHOST_CONFIG: /tmp/sock0: reconnecting...

Launch QEMU in server-mode:

qemu-system-x86_64 -cpu host -enable-kvm -m 4096M -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/hugepages,share=on -numa node,memdev=mem -mem-prealloc  -drive file=/images/image.qcow2 -chardev socket,id=char0,path=/tmp/sock0,server -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off –nographic

Logs similar to the following should be printed:

QEMU waiting for connection on: disconnected:unix:/tmp/sock0,server

VHOST_CONFIG: /tmp/sock0: connected</p>

The important part of the command above is to include “,server” as part of the path argument of the chardev configuration.

Once the VM has booted successfully, test the connection between dpdk0 and vhost0. Run the tcpdump tool on the vHost interface to monitor incoming traffic. Send traffic to dpdk0 (for example, using a traffic generator); it should be switched to vhost0 and captured by the tool:

[root@localhost ~]# tcpdump -i eth0

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

09:57:50.276877 IP 2.2.2.2.0 > 1.1.1.1.0: Flags [], seq 0:6, win 0, length 6

09:57:50.779559 IP 2.2.2.2.0 > 1.1.1.1.0: Flags [], seq 0:6, win 0, length 6

To test client mode reconnection, simply reset the switch and verify connectivity is re-established by continually monitoring the tcpdump instance, and verify traffic is once again switched to the VM after a brief loss of connectivity during the reset.

This is just one of many ways the reconnect capability can be tested. For instance, it can also be tested in the reverse direction by generating traffic (for example, a ping in the VM), and verifying it reaches the physical port. The DPDK pdump tool is a useful way to monitor traffic on physical ports in OVS. Instructions for configuring and using pdump with OVS can be found in the OVS-DPDK documentation mentioned earlier in this article, or in the article DPDK Pdump in Open vSwitch* with DPDK.

Conclusion

In this article, we described and showed how OVS-DPDK vHost User ports can be configured in client mode, allowing reestablishment of connectivity upon a reset of the switch. We have demonstrated one method of how to test this feature and suggested another.

Additional Information

For more details on the DPDK vHost library, refer to the DPDK documentation.

For more information on configuring vHost User in Open vSwitch, refer to INSTALL.DPDK.rst.

Have a question? Feel free to follow up with the query on the Open vSwitch discussion mailing thread.

To learn more about OVS with DPDK, check out the following videos and articles on Intel® Developer Zone and Intel® Network Builders University.

QoS Configuration and usage for Open vSwitch* with DPDK

Open vSwitch with DPDK Architectural Deep Dive

DPDK Open vSwitch: Accelerating the Path to the Guest

DPDK Pdump in Open vSwitch* with DPDK

vHost User NUMA Awareness in Open vSwitch* with DPDK

About the Author

Ciara Loftus is a network software engineer with Intel. Her work is primarily focused on accelerated software switching solutions in user space running on Intel® architecture. Her contributions to OVS with DPDK include the addition of vHost User ports, vHost User client ports, NUMA-aware vHost User, and DPDK v16.07 support.