In today’s data centers, big data has put greater demands on the performance of storage systems. The Lustre* file system was purposely built to provide sustained storage performance and stability on a large scale for high-performance computing (HPC) systems. Intel designed the Intel® Enterprise Edition for Lustre* software to enable fully parallel I/O throughput across clients, servers, and storage devices. Metadata and data are stored on separate servers to allow optimization of each system for different workloads. For ease of use, Intel® EE for Lustre* software includes the Intel® Manager for Lustre* software to configure, monitor, troubleshoot, and manage the storage systems. To adopt this technology with customized storage hardware, the storage developer will need to build a plugin for Intel® EE for Lustre* software. This blog will provide a high level overview of the storage plugin components which are illustrated in the lower right corner of Figure-1.
Figure 1- Intel(R) Enterprise Edition for Lustre Software components
For more information and access to the developer guide, check online and contact our partners or an Intel representative listed at:
http://www.intel.com/content/www/us/en/software/intel-enterprise-edition-for-lustre-software.html
Storage plugins are responsible for delivering information about entities (ex. Storage devices) to the manager server. The manager server web interface and command line interface (CLI) included with the Intel® Manager for Lustre* software are built on the REST API, which is accessed via HTTP. This API is available for integration with third party applications. The API is designed to minimize the lines of code required to write a plugin, minimize the duplication of effort between different plugins, and make as much of the plugin code as possible declarative in style, to minimize the need for per-plugin testing. For example, rather than having plugins procedurally check for resources that are in bad states during an update, we provide the means to declare conditions that are automatically checked. Similarly, rather than requiring explicit notification from the plugin when a resource attribute is changed, we detect assignments to attributes and schedule transmission of updates in the background.
To present device information to the manager server, a Python* module is written using the Storage Plugin API:
The objects to be reported are described by declaring a series of Python classes: Declare Resources
Certain of these objects are used to store the contact information such as IP addresses for managed devices: Declare a ScannableResource
A main plugin class is implemented to provide required hooks for initialization and teardown: Implement the Plugin
Below are the high level steps of how to build and run the plugin feature of the Intel® EE for Lustre* software. This blog is not intended to replace the Storage Plugin Developer’s Guide for Intel® Manager for Lustre* software and the REST API for Intel® Manager for Lustre* software documents. These documents are available with access to the Intel® EE for Lustre* software. See the section above on Where to Get More Information on the Storage Plugin for contact information..
A Resource may be a physical device such as a physical hard drive, or a virtual object such as a storage pool or virtual machine. In general, storage resources may inherit from Resource directly, but optionally they may inherit from a built-in resource class as a way of identifying common resource types for presentation purposes. The plugin can import attributes, identifiers, plugins, relations, resources, and statistics from chrome_core.lib.storage_plugin.api
Certain storage resources are considered ‘scannable’:
- They can be added by the administrator using the manager server user interface
- They are the resource contacted by other Plugins to learn about resources
- This resource ‘owns’ some other resources declaring a ScannableResource
A typical scannable resource is a storage controller or couplet of storage controllers.
The Resource classes are simply declarations of what resources can be reported by the plugin and what properties they will have. The plugin must be able to initialize, update, create, scan, teardown, and remove the resources. The plugin must follow standard Python* logging.
Sensible defaults are used wherever possible when presenting UI elements relating to storage resources. By default, attribute names are transformed to capitalized text. For example, `file_size`` is transformed to File size. The plugin must use a default human readable version of the class name followed by their identifier attributes.
Before running the plugin as part of a manager server instance, use validate_storage_plugin to validate it. The plugin needs to be able to run separately so that it can be stopped and started quickly.
The plugin must be able to link up resources between different plugins, or between storage controllers and Linux hosts by using the resource attributes.
The Storage Plugin allows storage developers to use their customized storage hardware running in a Lustre environment using the Intel® EE for Lustre* software. The information in this blog is to introduce the basic components of the Storage Plugin to the developer community. For more details about the storage plugin beyond the information provided in this blog, see your Intel representative.
- Developer’s Guide for Intel® Manager for Lustre* software
- REST API for Intel® Manager for Lustre* software
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.
Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm
Intel, the Intel logo, AVX2, AVX, SIMD, AES-NI, Intel® ISA-L, and Intel® Xeon® are trademarks of Intel Corporation in the U.S. and other countries.
*Other names and brands may be claimed as the property of others
Copyright© 2014 Intel Corporation. All rights reserved.
This sample source code is released under the Intel Sample Source Code License Agreement
Optimization Notice |
Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. |