Assigning physical hard drives to Intel® Xeon Phi™ coprocessors

Intel® Xeon Phi™ coprocessors are able to directly mount and use block devices that are attached to the host system. This article provides basic instructions for formatting and mounting a hard disk drive (HDD) or solid state drive (SSD) natively on the coprocessor. Benefits include direct, dedicated, and persistent storage for individual coprocessors. A large cluster could use these to store additional software packages and data, providing more available memory without impacting network traffic. In addition, this method supports creation of swap partitions.

Conversely, features such as TRIM are not supported.

Block devices are accessed by the Intel® Xeon Phi™ coprocessor using the virtblk (or virtio_blk) driver that is part of the libvirt project. More information on this driver can be found here:

http://wiki.libvirt.org/page/Virtio#Virtio

Prerequisites

All required software is included in MPSS 3.x—no additional packages are needed.

These instructions assume that MPSS is installed and the host platform and the coprocessor are online, configured, and working correctly.

The following example will assign one dedicated drive (/dev/sdb) to one coprocessor (mic0). Additional coprocessors and drives can be configured following the same instructions, altering device names as needed.

The dedicated drive will be used exclusively for the coprocessor in the host system. This drive cannot be in use by the host platform or other coprocessors. The virtblk driver also permits mounting partitions or files as virtual drives; these options are not described here. The drive will be configured for an 8 GB swap partition with the remainder available for coprocessor dedicated storage.

Host configuration

1. Partition the drive.

Any partitioning manager should work. For example, to use fdisk, type

	fdisk /dev/sdb

2. Create two primary partitions.

When complete, the partitioning scheme should be similar to this:

	   Device Boot      Start         End      Blocks  Id System
	/dev/vda1               1        7630     7813104  82 Linux swap
	/dev/vda2            7631       30533    23452672  83 Linux

3. Initialize the swap partition

	mkswap /dev/sdb1

4. Format the storage partition

	mkfs.ext3 /dev/sdb2

5. Allocate the drive to the coprocessor.

This can be done with the following command line; however, it will not be persistent between resets of the mic driver.

	echo /dev/sdb > /sys/class/mic/mic0/virtblk_file

In order to allocate the drives when MPSS is started, and make the drives available to the coprocessor when it boots, the mpss script must be modified. This allows the drive allocation between the driver check and coprocessor boot. The script should be backed-up before altering it.

Proper startup is achieved by adding drive allocation commands directly before the "$exec" command. When complete, it should look like this:

                return 0;
        fi

        echo /dev/sdb > /sys/class/mic/mic0/virtblk_file

        $exec
        RETVAL=$?
        if [ "$RETVAL" = "0" ]; then

Make certain to use overwrite “>” and not append “>>”.

Coprocessor configuration

The mount point is set up in the overlay directory and fstab must be updated to mount the drive automatically. In this example, the drive will be mounted at /store.

When a block device is allocated in the virtblk_file, the mic_virtblk kernel module is loaded automatically on the coprocessor at boot.

1. Create the mount point

	mkdir /var/mpss/mic0/store

2. Open the coprocessor fstab file for editing

	vi /var/mpss/mic0/etc/fstab

3. Add the mounts to to fstab.

The following lines need to be appended to the file. Adjust the options, dump, and fsck settings as appropriate.

	/dev/vda1           swap     swap     defaults               0 0
	/dev/vda2           /store   ext3     defaults               1 2

4. Restart the MPSS service.

Check the new drive

Check that the new drive is properly configured and recognized by the coprocessor. In some cases, the partitioning or format performed by the host may not be compatible with the coprocessor operating system. If this is the case, partition and format the drive directly from the coprocessor.

1. Log into the coprocessor

	ssh mic0

2. Activate the swap partition

	swapon -a

Testing showed that swap partitions are not enabled at boot, so they must be enabled manually or by script.

3. Check that the swap partition is active

	cat /proc/swaps

The output should produce

	Filename	Type		Size	Used	Priority
	/dev/vda1	partition	7813100 0	-1

4. Check available storage on /store

	df /store

The output should produce

	Filesystem           1K-blocks      Used Available Use% Mounted on
	/dev/vda2             23084516    176196  21735688   1% /store