Build Open Source Storage Stacks in User Space

Abstract:

As storage requirements increase exponentially each year, storage hardware is advancing quickly to meet the demand. With the high performance of next-generation technologies like 3D XPoint™ technology, latency will no longer be measured in milliseconds but in nanoseconds. But this rise of next-generation storage simply shifts the I/O bottlenecks from storage media to the network and software, which we predict will force development of user mode cloud storage.  While many commercial projects such as Containers*, Cepth*, Swift*, Hadoop*, and proprietary software already run in user space, development of open source building blocks for storage systems is  an area ripe for innovation.

Background

Storage performance has two main considerations: latency and throughput. Latency refers to the time delay while moving data and throughput refers to the amount of data transferring in a given time.

Storage architectures have to be optimized around the properties of their media. For instance, HDDs have three important characteristics to consider:

1. Read and write latency is substantial because of the required layers of caching - at the drive and in the storage subsystem’s buffer.
2. Reads and writes are significantly less costly when performed in a sequential pattern instead of a random pattern. 
3. I/O performance is limited by the rotational speed and number of spindles in play.
4. Mechanical drives and segregated controllers slow throughput

These hardware and architectural limitations led to additional design points to optimize storage:

1. An interrupt-based model for I/O is more efficient.
2. Use of DRAM for HDD cache reduces latency for reads and sequential writes.
3. Use of DRAM for SDDs reduces hardware failures as well as access times.
4. Reducing the number of devices between processor and storage media reduces latency.

Figure 1 – Performance improves as data moves closer to the processor

Hardware Breakthroughs

So how to achieve ever higher performance and lower latency? First look at the hardware, especially density and power requirements but also moving away from even 1 hop to the CPU.

Multiple breakthroughs on the hardware side also increased performance with less latency, more throughput, lower power requirements, and denser capacity.

• Eliminating the external controller (the move from 1 hop to 0 hop) using the integrated Non-volatile Memory Express (NVMe) interface to the processor
• Eliminating the need for power to keep data refreshed by moving to non-volatile memory
• Creation of new types of SSDs including those with Intel Optane™ Technology that can use 3D XPoint memory – which is up to 10x denser and up to 1000x faster than NAND.

Figure 2 – Storage performance latency improvement

As shown in Figure 2, the 15k HDDs have a latency of about 2000 µs, with smaller controller and software delays.  Solid-State storage eliminated mechanical delays, moving the controller into the CPU eliminated extra steps and continual software improvements all greatly reduced latency.

Reasons for Open Source User Mode Storage Stacks

Improvements have not been limited to hardware. There are multiple reasons to develop next-generation storage appliances and architectures in user space instead of using kernel mode:

Reason #1: Reliability-Availability-Serviceability (RAS) improvements

• User mode drivers can be restarted without reboot, while any kernel mode driver restart forces a reboot.
• User mode drivers allow more flexible data collection and logging.

Reason #2: Licensing ease of User Space code

• The Storage Performance Development Kit (SPDK) is licensed under the Berkeley Software Distribution License (for NVMe or CBDMA drivers) or an Intel proprietary license (for user space network stack and iSCSI target). User space has no risk of General Public License pollution.

Reason #3: Efficiency

• Technical reasons: kernel interrupt handling and context switch overhead consume processor resources, whereas the SPDK NVMe polled-mode driver architecture delivers efficient performance and allows a single processor core to handle millions of IOPS.
• Translation Look aside Buffer (TLB) entries are cleared and cache entries evicted.

Reason #4: Design simplification

• Low-latency NVM media allows for elimination of battery-backed DIMMs.
• The user space drivers can be applied directly to server-based storage and software-defined architectures.
• As physical media latency drops, the necessity of using large DRAM caches goes away for all but the most demanding workloads.

Conclusions

Enabling developers to work in user space improves performance and simplifies development for cloud storage developers so they can take full advantage of the potential performance of new NVM technologies such as Intel Optane™ technology. Intel applies the high performance packet processing framework of the open source Data Plane Development Kit (DPDK) to a storage environment with the Storage Performance Development Kit. The SPDK is a set of drivers and a complete reference storage architecture that includes Linux user space for functions that typically required context switching into the kernel.   Both DPDK and SPDK are available as open source.

References:

• The New Breakthrough Memory 3D XPoint™ Technology Delivers
• A Storage Framework For Cloud Storage Developers
• Introducing Intel Optane™ Technology – Bringing 3D XPoint™ Memory to Storage and Memory Products
• Intel SSD vs. HDD Video
• Data Plane Development Kit
• Timeline: 50 Years of Hard Drives

The author would like recognize the following individuals for their contributions to this article. Jonathan Stern, Debra Graham, Mike Pearce, Colleen Culberson, and Victoria Davis

Author
Thai Le is a software engineer and has been with Intel® Corporation for over 20 years. His areas of focus has included software automation, server power and performance analysis, and cloud computing.