Introduction
Intel® Ethernet Flow Director (Intel® Ethernet FD) directs Ethernet packets to the core where the packet consuming process, application, container, or microservice is running. It is a step beyond receive side scaling (RSS) in which packets are sent to different cores for interrupt processing, and then subsequently forwarded to cores on which the consuming process is running.
Intel Ethernet FD supports advanced filters that direct received packets to different queues, and enables tight control on flow in the platform. It matches flows and CPU cores where the processing application is running for flow affinity, and supports multiple parameters for flexible flow classification and load balancing. When operating in Application Targeting Routing (ATR) mode, Intel Ethernet FD is essentially the hardware offloaded version of Receive Flow Steering available on Linux* systems, and when running in this mode, Receive Packet Steering and Receive Flow Steering are disabled.
It provides the most benefit on Linux bare-metal usages (that is, not using virtual machines (VMs)) where packets are small and traffic is heavy. And because the packet processing is offloaded to the network interface card (NIC), Intel Ethernet FD could be used to avert denial-of-service attacks.
Supported Devices
Intel Ethernet FD is supported on devices that use the ixgbe driver, including the following:
- Intel® Ethernet Converged Network Adapter X520
- Intel® Ethernet Converged Network Adapter X540
- Intel® Ethernet Controller 10 Gigabit 82599 family
It is also supported on devices that use the i40e driver:
- Intel® Ethernet Controller X710 family
- Intel® Ethernet Controller XL710 family
DPDK includes support for Intel Ethernet FD on the devices listed above. See the DPDK documentation for how to use DPDK and testpmd with Intel Ethernet FD.
In order to determine whether your device supports Intel Ethernet FD, use the ethtool command with the --show-features or -k parameter on the network interface you want to use:
# ethtool --show-features <interface name> | grep ntuple
If the ntuple-filters feature is followed by off or on, Intel Ethernet FD is supported on your Ethernet adapter. However, if the ntuple-filters feature is followed by off [fixed], Intel Ethernet FD is not supported on your network interface.
Enabling Intel® Ethernet Flow Director
Driver Parameters for Devices Supported by the ixgbe Driver
On devices that are supported by the ixgbe driver, there are two parameters that can be passed-in when the driver is loaded into the kernel that will affect Intel Ethernet FD:
- FdirPballoc
- AtrSampleRate
FdirPballoc
This driver parameter specifies the packet buffer size allocated to Intel Ethernet FD. The valid range is 1–3, where 1 specifies that 64k should be allocated for the packet buffer, 2 specifies a 128k packet buffer, and 3 specifies a 256k packet buffer. If this parameter is not explicitly passed to the driver when it is loaded into the kernel, the default value is 1 for a 64k packet buffer.
AtrSampleRate
The AtrSampleRate parameter indicates how many Tx packets will be skipped before a sample is taken. The valid range is from 0 to 255. If the parameter is not passed to the driver when it is loaded into the kernel, the default value is 20, meaning that every 20th packet will be sampled to determine if a new flow should be created. Passing a value of 0 will disable ATR mode, and no samples will be taken from the Tx queues.
The above driver parameters are not supported on devices that use the i40e driver.
To enable these parameters, first unload the ixgbe module from the kernel. Note, if you are connecting to the system over ssh, this may disconnect your session:
# rmmod ixgbe
Then re-load the ixgbe driver into the kernel with the desired parameters listed above:
# modprobe ixgbe FdirPballoc=3,2,2,3 AtrSampleRate=31,63,127,255
Note that, in this example, for each parameter there are four values. This is because on my test system, I have two network adapters that are using the ixgbe driver--an Intel Ethernet Controller 10 Gigabit 82599, and an Intel® Ethernet Controller 10 Gigabit X540--each of which has two ports. The order in which the parameters are applied is in PCI Bus/Device/Function order. To determine the PCI BDF order on your system, use the following command:
# lshw -c network -businfo
Based on this system configuration, using the modprobe command above, the Intel Ethernet Controller 10 Gigabit X540-AT2 port at PCI address 00:03.0 is allocated the FdirPballoc and AtrSampleRate parameters of 3 and 31, respectively, and the Intel Ethernet Controller 10 Gigabit 82599 port at PCI address 81:00.1 is allocated the FdirPballoc and AtrSampleRate parameters of 3 and 255, respectively.
Once you have determined that your Intel branded server network adapter supports Intel Ethernet FD and you have loaded the desired parameters into the driver (on supported models), execute the following command to enable Intel Ethernet FD:
# ethtool --features enp4s0f0 ntuple on
Because the commands below only indicate which Rx queue a matched packet should be sent to, ideally an additional step should be taken to pin both Rx queues and the process, application, or container that is consuming the network traffic to the same CPU. Pinning an application/process/container to a CPU is beyond the scope of this document, but it can be done using the taskset command. Pinning IRQs to a CPU can be done using the set_irq_affinity script that is included with the freely available sources of the i40e and ixgbe drivers. See Intel Support: Drivers and Software for the latest versions of these drivers. See also the IRQ Affinity section in this tuning guide for how to set IRQ affinity.
Using Intel Ethernet Flow Director
Intel Ethernet FD can run in one of two modes: externally programmed (EP) mode, and ATR mode. Once Intel Ethernet FD is enabled as shown above, ATR mode is the default mode, provided that the driver is in multiple Tx queue mode. When running in EP mode, the user or management/orchestration software can manually set how flows are handled. In either mode, fields are intelligently selected from the packets in the Rx queues to index into the Perfect-Match filter table. For more information on how Intel Ethernet FD works, see this whitepaper.
Application Targeting Routing
In ATR mode, Intel Ethernet FD uses fields from the outgoing packets in the Tx queues to populate the 8K-entry Perfect-Match filter table. The fields that are selected depend on the packet type; for example, fields to filter TCP traffic will be different than those used to filter user diagram protocol (UDP) traffic. Intel Ethernet FD then uses the Perfect-Match filter table to intelligently route incoming traffic to the Rx queues.
To disable ATR mode and switch to EP mode, simply use the ethtool command shown under Adding Filters to manually add a filter, and the driver will automatically enter EP mode. To automatically re-enable ATR mode, use the ethtool command under Removing Filters until the Perfect-Match filter table is empty.
Externally Programmed Mode
When Intel Ethernet FD runs in EP mode, flows are manually entered by an administrator or by management/orchestration software (for example, OpenFlow*). As mentioned above, once enabled, Intel Ethernet FD automatically enters EP mode when a flow is manually entered using the ethtool command listed under Adding Filters.
Adding Filters
The following commands illustrate how to add flows/filters to Intel Ethernet FD using the -U,
-N, or --config-ntuple switch to ethtool.
To specify that all traffic from 10.23.4.6 to 10.23.4.18 be placed in queue 4, issue this command:
# ethtool --config-ntuple flow-type tcp4 src-ip 10.23.4.6 dst-ip 10.23.4.18 action 4
Note: Without the ‘loc’ parameter, the rule is placed at position 1 of the Perfect-Match filter table. If a rule is already in that position, it is overwritten.
Forwards to queue 2 all IPv4 TCP traffic from 192.168.10.1:2000 that is going to 192.168.10.2:2001, placing the filter at position 33 of the Perfect-Match filter table (and overwriting any rule currently in that position):
# ethtool --config-ntuple <interface name> flow-type tcp4 src-ip 192.168.10.1 dst-ip 192.168.10.2 src-port 2000 dst-port 2001 action 2 loc 33
Drops all UDP packets from 10.4.83.2:
# ethtool --config-ntuple flow-type udp4 src-ip 10.4.82.2 action -1
Note: The VLAN field is not a supported filter with the i40e driver (Intel Ethernet Controller XL710 and Intel Ethernet Controller X710 NICs).
For more information and options, see the ethtool man page documentation on the -U, -N, or --config-ntuple option.
Note: The Intel Ethernet Controller XL710 and the Intel Ethernet Controller X710, of the Intel® Ethernet Adapter family, provide extended cloud filter flow support for more complex cloud networks. For more information on this feature, please see the Cloud Filter Support section in this ReadMe document, or in the ReadMe document in the root folder of the i40e driver sources.
Removing Filters
In EP mode, to remove a filter from the Perfect-Match filter table, execute the following command against the appropriate interface. ‘N’ in the rule below is the numeric location in the table that contains the rule you want to delete:
# ethtool --config-ntuple <interface name> delete N
Listing Filters
To list the filters that have been manually entered in EP mode, execute the following command against the desired interface:
# ethtool --show-ntuple <interface name>
Disabling Intel Ethernet Flow Director
Disabling Intel Ethernet FD is done with this command:
# ethtool --features enp4s0f0 ntuple off
This flushes all entries from the Perfect-Filter flow table.
Conclusion
Intel Ethernet FD directs Ethernet packets to the core where the packet consuming process, application, container, or microservice is running. This functionality is a step beyond RSS, in which packets are simply sent to different cores for interrupt processing, and then subsequently forwarded to cores on which the consuming process is running. It can be explicitly programmed by administrators and control plane management software, or it can intelligently sample outgoing traffic and automatically create Perfect-Match filters for incoming packets. When operating in automatic ATR mode, Intel Ethernet FD is essentially the hardware offloaded version of Receive Flow Steering available on Linux systems.
Intel Ethernet FD can provide additional performance benefit, particularly in workloads where packets are small and traffic is heavy (for example, in Telco environments). And because it can be used to filter and drop packets at the network interface card (NIC), it could be used to avert denial-of-service attacks.
Resources
https://downloadmirror.intel.com/26556/eng/README.txt
https://downloadmirror.intel.com/26713/eng/Readme.txt
https://downloadmirror.intel.com/22919/eng/README.txt
http://dpdk.org/doc/guides/howto/flow_bifurcation.html
http://software.intel.com/en-us/videos/creating-virtual-functions-using-sr-iov
Also, view the ReadMe file found in the root directory of both the i40e and ixgbe driver sources.
Notices
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.
This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.
The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request.
Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm.
This sample source code is released under the Intel Sample Source Code License Agreement.
Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
Microsoft, Windows, and the Windows logo are trademarks, or registered trademarks of Microsoft Corporation in the United States and/or other countries.
*Other names and brands may be claimed as the property of others.
© 2017 Intel Corporation