Inline IPsec with DPDK and Intel® 82599 Network Controller

Introduction

This article looks at inline IPsec acceleration support enablement in the Data Plane Development Kit (DPDK) framework with a particular focus on the Intel® 82599 10 Gigabit Ethernet Controller series features and support.

Inline IPsec can be used to implement IPsec-aware systems that have a better latency than lookaside-assisted and accelerated hardware, providing that the algorithm supported is suitable.

This article includes background information that will highlight the differences between “lookaside” hardware acceleration for IPsec and inline IPsec (i.e., the packet flow differences and the expected differences on the application side, in performance and handling).

An example of setting up a test system, installing, and running an IPsec Gateway application will be presented.

The article assumes that the DPDK release used is 17.11 or later.

Background

The DPDK Security Library provides a framework for the management and provisioning of security protocol operations offloaded to hardware-based devices. The library defines generic APIs to create and free security sessions that can support full protocol offload as well as inline crypto operations with network interface controller (NIC) or crypto devices.

The security library, which has been included in the DPDK since release 17.11, introduces APIs and features that provide a way of adding support for the inline crypto (IPsec) acceleration already available on the Intel 82599 10 Gigabit Ethernet Controller series. The DPDK IXGBE driver was also updated with support for inline IPsec.

The Intel 82599 10 Gigabit Ethernet Controller series only supports AES-GCM 128, hence the supported protocols:

• ESP authentication only: AES-128-GMAC (128-bit key)
• ESP encryption and authentication: AES-128-GCM (128-bit key)

The IPsec Security Gateway sample application also supports this feature, which will be explained in more detail below.

Packet Flow

First, let’s look at the packet flow and handling through the system.

The packet flow depicted above shows the packet-processing stages for an incoming encrypted IPsec packet. In the first case, it is processed (decrypted and/or authenticated) using the lookaside Intel® QuickAssist Technology (Intel® QAT) hardware accelerator. In the second case, it is processed using the inline IPsec hardware processing available in the Intel 82599 10 Gigabit Ethernet Controller.

As presented above, the second stage of the processing (decryption and/or authentication) is combined with the packet receive stage by the network controller itself. The DPDK application still needs to process the encapsulation and de-capsulation of the Encapsulating Security Payload (ESP) packet.

The outgoing flow is identical except for the packet direction.

Software APIs and the Sample Application

Now, let’s look at the software APIs, requirements, and usage model. We will use code excerpts from the IPsec Security Gateway sample application, simplified for clarity, and with error handling removed.

First, we need to create a security session, beginning with the following configuration:

struct rte_security_session_conf sess_conf = {
    .action_type = RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO,
    .protocol = RTE_SECURITY_PROTOCOL_IPSEC,
    {.ipsec = {
        .spi = <SPI>,
        .salt = <Salt>,
        .options = { 0 },
        .direction = RTE_SECURITY_IPSEC_SA_DIR_INGRESS,
        .proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP,
        .mode = RTE_SECURITY_IPSEC_SA_MODE_TUNNEL,
    } },
    .crypto_xform = <xforms>,
    .userdata = NULL,
};

Next, create the security session:

struct rte_security_ctx *ctx = (struct rte_security_ctx *)
    rte_eth_dev_get_sec_ctx(<portid>);
rte_security_session *sec_session = 
    rte_security_session_create(ctx, &sess_conf, <session_pool>);

Then, create an RTE_FLOW_ACTION_TYPE_SECURITY flow:

pattern[0].type = RTE_FLOW_ITEM_TYPE_ETH;
pattern[1].type = RTE_FLOW_ITEM_TYPE_IPV4;
pattern[1].mask = &rte_flow_item_ipv4_mask;
pattern[2].type = RTE_FLOW_ITEM_TYPE_ESP;
pattern[2].spec = &esp_spec; /*contains the SPI*/
pattern[2].mask = &rte_flow_item_esp_mask;

action[0].type = RTE_FLOW_ACTION_TYPE_SECURITY;
action[0].conf = sa->sec_session;
action[1].type = RTE_FLOW_ACTION_TYPE_END;
action[1].conf = NULL;

flow = rte_flow_create(portid, <attr>, <pattern>, <action>, &<err>);

With this, the configuration is complete and the Security Association (SA) is active for incoming ESP packets that match the flow created. The offload flags in the “rte_mbuf” struct will indicate if the packet was processed inline by setting the PKT_RX_SEC_OFFLOAD flag and, if any error occurred, PKT_RX_SEC_OFFLOAD_FAILED will also be set.

For outgoing packets, the programming is similar except that the offload flag PKT_TX_SEC_OFFLOAD needs to be set by the application.

Test System Setup

In addition to DPDK and the sample application, we will use Python* scapy v2.4.0 (packet generator) and pycryptodome v3.6.0 (which provides AES-GCM support) to generate clear text and encrypted packets.

Save the following as inline_ipsec.py:

from scapy.all import *

def main(argv):
    
    payload = 'test-' * 2000
    
    sa = SecurityAssociation(ESP, spi=5, crypt_algo='AES-GCM', 
            crypt_key='\x2b\x7e\x15\x16\x28\xae\xd2\xa6\xab\xf7\x15''\x88\x09\xcf\x4f\x3d\xde\xad\xbe\xef', 
            auth_algo='NULL', auth_key=None,
            tunnel_header=IP(src='172.16.1.5', dst='172.16.2.5'))
    sa.crypt_algo.icv_size = 16
    
    try:
        opts, args = getopt.getopt(argv, 'c:i:s:e:',
                                   ['count=', 'iface=', 'size=', 'encrypt='])
    except getopt.GetoptError:
        sys.exit(2)


    maxcount = 1
    intface = "enp2s0f0"
    paysize = 64
    do_encrypt = True

    for opt, arg in opts:
        if opt in ("-c", "--count"):
            maxcount = arg
        if opt in ("-i", "--iface"):
            intface = arg
        if opt in ("-s", "--size"):
            paysize = arg
        if opt in ("-e", "--encrypt"):
            if arg == '0' or arg == 'False':
                do_encrypt = False

    p = IP(src='192.168.105.10', dst='192.168.105.10')
    p /= "|->"
    p /= payload[0:(int(paysize) - 6)]
    p /= "<-|"
    p = IP(str(p))
    
    if do_encrypt:
        e = sa.encrypt(p)
    else:
        e = p
    
    eth_e = Ether()/e
    
    sendp(eth_e, iface=intface, count=int(maxcount))
        
if __name__ == "__main__":
    exit(main(sys.argv[1:]))

The test system configuration is shown in the figure below.

The configuration uses two Intel® 82599ES 10 Gigabit Ethernet Controller dual port cards connected as shown above.

In this particular configuration, card 1 ports will be assigned to the DPDK driver, port 0 BDF 06:00.0 and port 1 BDF 06:00.1. Card 0 ports will be assigned to the kernel driver, identified as enp2s0f0 and enp2s0f1. Note, in other systems the addresses and port names may be different.

Create the IPsec sample app configuration based on this data and save it as a file called inline.cfg. For example:

#SP IPv4 rules
sp ipv4 out esp protect 1005 pri 1 dst 192.168.105.0/24 sport 0:65535 dport 0:65535

#SA rules
sa out 1005 aead_algo aes-128-gcm aead_key 2b:7e:15:16:28:ae:d2:a6:ab:f7:15:88:09:cf:4f:3d:de:ad:be:ef \
mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
port_id 1 \
type inline-crypto-offload \

sa in 5 aead_algo aes-128-gcm aead_key 2b:7e:15:16:28:ae:d2:a6:ab:f7:15:88:09:cf:4f:3d:de:ad:be:ef \
mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
port_id 1 \
type inline-crypto-offload \

#Routing rules
rt ipv4 dst 172.16.2.5/32 port 1
rt ipv4 dst 192.168.105.10/32 port 0

Then start the application as follows:

ipsec-secgw \
	-l 6,7 \
	-w 06:00.0 -w 06:00.1 \
	--log-level 8 --socket-mem 1024,0 --vdev crypto_null \
	-- -p 0xf -P -u 0x2 \
	--config="(0,0,6),(1,0,7)" -f ./inline.cfg

Note that the NULL crypto device is only present because the application needs a “true” crypto PMD (for legacy reasons).

Testing ESP decryption and forwarding of the decrypted packets

In order to test the application, send encrypted packets on port 1:

     python inline_ipsec.py -i enp2s0f1 -s 64 -c 32 -e 1

While listening on port 0:

     tcpdump -i enp2s0f0 –vvX

Decrypted packets should be observed on port 0:

10:31:00.636564 IP (tos 0x0, ttl 63, id 1, offset 0, flags [none], 
                    proto Options (0), length 84)
    192.168.105.10 > 192.168.105.10:  ip 64
	0x0000:  4500 0054 0001 0000 3f00 0100 c0a8 690a  E..T....?.....i.
	0x0010:  c0a8 690a 7c2d 3e74 6573 742d 7465 7374  ..i.|->test-test
	0x0020:  2d74 6573 742d 7465 7374 2d74 6573 742d  -test-test-test-
	0x0030:  7465 7374 2d74 6573 742d 7465 7374 2d74  test-test-test-t
	0x0040:  6573 742d 7465 7374 2d74 6573 742d 7465  est-test-test-te
	0x0050:  733c 2d7c                                s<-|

The packet flow is as follows:

1. ESP packet is received on port 1.
2. Port 1 RX queue contains ESP packet with the payload decrypted.
3. ESP decapsulation.
4. Clear text packet containing decrypted payload is transmitted on port 0.

Testing ESP encryption

In order to test the application, send clear text packets on port 0:

     python inline_ipsec.py -i enp2s0f0 -s 64 -c 32 -e 0

While listening on port 1:

     tcpdump -i enp2s0f0 –vvX

Encrypted packets should be observed on port 1:

10:37:45.622669 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], 
                    proto ESP (50), length 140)
    172.16.1.5 > 172.16.2.5: ESP(spi=0x000003ed,seq=0x20), length 120
	0x0000:  4500 008c 0000 0000 4032 1f16 ac10 0105  E.......@2......
	0x0010:  ac10 0205 0000 03ed 0000 0020 0000 0000  ................
	0x0020:  0000 0020 fd78 c8f5 7fab 4fb3 5b98 7e79  .....x....O.[.~y
	0x0030:  81b0 b4f2 d796 ccd4 f0a7 b031 bb9b 9bde  ...........1....
	0x0040:  af18 767e 5d0f 73e3 bc82 4ea3 4afb 00eb  ..v~].s...N.J...
	0x0050:  6d02 a367 7a3a c2dd 6b64 74c1 5d41 bb45  m..gz:..kdt.]A.E
	0x0060:  7ac2 c1e0 0fb8 5f73 7fcd 4304 e396 32ea  z....._s..C...2.
	0x0070:  228e 22e5 4a3e ea72 88fb 13a7 e940 9346  ".".J>.r.....@.F
	0x0080:  4451 98cf 97fd 878c 96f0 f754            DQ.........T

The packet flow is as follows:

1. Clear text packet is received on port 0.
2. ESP encapsulation.
3. Packet is placed in the TX queue on port 1.
4. Packet is encrypted by port 1 during transmission.

Conclusion

This article presented the details of the inline IPsec support available with the Intel 82599 10 Gigabit Ethernet Controller, how it works internally and how it is supported by the DPDK framework, and how to build an application that can utilize the feature.

About the Author

Radu Nicolau is a network software engineer with Intel. His work is currently focused on IPsec-related development of data plane functions and libraries. His contributions include enablement of the AES-GCM crypto algorithm in the VPP IPsec stack, IKEv2 initiator support for VPP, and inline IPsec enablement in DPDK.