A Storage Framework for Cloud Storage Developers

Introduction

Cloud storage developers face challenges with integrating the latest devices including solid-state drives, faster network infrastructure, and the latest processor technology. Each technology improves a different aspect of performance, so keeping performance balanced while optimizing for cost requires careful consideration and an understanding of both the hardware and the software driving it. To help storage OEMs and ISVs with the integration of hardware and software, Intel has created a set of drivers and a complete, end-to-end reference storage architecture, called the Storage Performance Development Kit (SPDK). This article provides a brief overview of the SPDK including how to obtain it.

SPDK: An Overview

The SPDK is all about efficiency and scalable performance. The development kit reduces both processing and development overhead, and ensures the software layer is optimized to take advantage of the performance potential of next-generation storage media, like Non-Volatile Memory Express* (NVMe) devices.

The SPDK has two key techniques to improve storage performance: running in polled mode and at user level. In terms of processing efficiency, the SPDK leverages a number of techniques to optimize the data path. Specifically, the SPDK avoids context switches generated by I/O and moves to a polled mode driver (PMD) model running in the user space. This process removes the overhead associated with interrupts and data copies between the kernel and user memory space. By implementing huge pages and allocating static memory regions for buffers, the SPDK eliminates translation look aside buffer (TLB) misses, ensuring that the data in flight remains in cache. These types of software optimizations allow the SPDK NVMe driver to be up to 10 times more efficient than the default Linux* NVMe driver and to scale to handle millions of I/Os per second (IOPS).

For developers, the SPDK provides maximum functionality with minimum encumbrance. By providing an agile and highly scalable framework, the SPDK allows developers to focus on their application instead of spending time tuning and optimizing the platform. The permissive open source license ensures a minimum barrier to adoption in both commercial and open source projects, while allowing developers to modify the code to meet their system’s needs. The code itself is well documented and straightforward to use.

The SPDK is composed of several subcomponents, interlinked and sharing the common elements of user-level and polled-mode operation. Each of these components was created to overcome a specific performance bottleneck while creating the end-to-end SPDK architecture. And each of these components can be integrated into non-SPDK architectures, allowing customers to leverage the experience and techniques used within the SPDK to accelerate their own software. For example, the User-space Network Services (UNS) library was created to overcome the performance limits of the Linux kernel TCP/IP stack. By creating a user-level, polled implementation of the TCP/IP stack, the SPDK is able to realize substantially higher IOPS performance by spending fewer processor cycles handling TCP/IP packet sorting and processing.

The front end is composed of the Data Plane Development Kit (DPDK) NIC driver and the UNS components. The DPDK provides a framework for high-performance packet processing at the NIC, providing a fast path for the data to pass from the NIC into user space. The UNS code then takes over, cracking the TCP/IP packets and forming the iSCSI commands.

At this point the processing framework takes the packet contents and translates the iSCSI commands into SCSI block-level commands. However, before it sends these commands to the back-end drivers, the SPDK presents an API framework to add customer-specific features—“special sauce”—within the SPDK framework (see the green box in the figure above). These SPDK functions include caching, deduplication and compression of data, encryption, RAID, and erasure coding calculations. Examples of these features are included with the SPDK, though these examples are solely intended to help model real-world use cases and should not be considered production-ready implementations.

Finally the data reaches the back-end drivers, where the read and writes to the physical block devices take place. The SPDK includes user-level PMDs for several storage media: NVMe devices, Linux AIO devices such as traditional spinning disks, and memory drivers for block-addressed memory applications (for example, RAMDISKS); and devices that can use the Intel® I/O Acceleration Technology (code-named Crystal Beach DMA). This suite of back-end drivers spans the spectrum of storage device performance tiers, ensuring relevance for nearly every storage application.

SPDK Component Definitions

• User-space Network Services (UNS) TCP/IP. This module processes all of the network packets from the NIC driver and passes them through a FreeBSD*-based TCP/IP stack. The UNS library (libuns) includes both a sockets-compatible interface and a more flexible interface to allow for future zero-copy operations.
• NVMe driver. This userspace driver was written to allow any NVMe device to be used via a userspace driver. This driver was created following the same practices as the rest of the SPDK; that is, it operates in a polled-mode fashion. The key advantage of this userspace driver is the incredibly low software latency—tens of nanoseconds—which minimizes the CPU overhead of driver operations. This driver is currently released as open source.
• Mem driver. This driver is essentially a RAMDISK back-end driver for the SPDK. It was developed chiefly to perform performance tests at levels beyond what NVMe devices can achieve. It processes the READ or WRITE request to the memory devices after the processing of the customer storage application.
• CBDMA driver. This is an Intel® Xeon® processor DMA engine (also known as Intel® QuickData technology (code-named Crystal Beach DMA)) back-end driver that allows for a DMA engine to be used. This driver processes the READ or WRITE request to the CBDMA devices after the processing of the customer storage application. This driver is currently released as open source.

Notes: NVMe and iSCSI drivers are the enhanced drivers that operate in the userspace but have kernel privileges to avoid context switching and interrupts and that don’t need “locks”.

• iSCSI. Once the packets have been cracked in the TCP layer, the iSCSI Protocol Data Unit (PDU) arrives in the iSCSI module, which translates the PDU into a SCSI Command Descriptor Blocks (CDB), which in turn will eventually be translated into block read or write commands. This iSCSI target identifies the read and write SCSI request and determines the capable devices. It then passes the request through the Block Device Abstraction Layer (BDAL) programmer interface (API) to allow customers to create a value add that differentiates the services including inline de-duplication, erasure code, compression, hashing, and more. An example application using the Intel® Intelligent Storage Acceleration Library functions is included with the SPDK as a reference for integration.

The table below summarizes the availability of each component:

Considerations and Licensing

The SPDK is not meant for every storage architecture. The following questions will help you to determine whether the SPDK components are a good fit for your storage environment:

Is the storage system based on Linux?

The SPDK is currently only tested and supported on Linux.

Can the system architecture incorporate lockless PMDs into its threading model?

Since PMDs continually run on their threads (instead of sleeping or ceding to the processor when unused), they have specific thread model requirements.

Does the system currently use the Data Plane Development Kit (DPDK) to handle network packet workloads?

The DPDK contains the framework for the SPDK, so customers currently using the DPDK will likely find the close integration with the SPDK very useful.

Can your license model use non-redistributable source?

All source code for the SPDK is provided free of charge.
Some portions of the SPDK (the NVMe and CBDMA Userspace PMDs) are available as open source, BSD-licensed components. They are available at https://01.org/spdk or https://github.com/spdk

Other portions (the UNS and the userspace iSCSI Target) are licensed under an Intel license at this time but won’t be available until 2016.

Note: Intel has no support obligations for this reference software. While Intel and the open source SPDK community will use commercially reasonable efforts to investigate potential errata of unmodified released software; under no circumstances will Intel have any obligation to customers with respect to providing any maintenance or support of the software.

References

Speeding Up Your Cloud Environment on Intel® Architecture

Author
Thai Le is a software engineer focusing on Cloud Computing in the Software Solution Group at Intel® Corporation.