Intel® MPI Library Release Notes for Linux* OS

Overview

Intel® MPI Library is a multi-fabric message passing library based on ANL* MPICH3* and OSU* MVAPICH2*.

Intel® MPI Library implements the Message Passing Interface, version 3.1 (MPI-3) specification. The library is thread-safe and provides the MPI standard compliant multi-threading support.

To receive technical support and updates, you need to register your product copy. See Technical Support below.

Product Contents

• The Intel® MPI Library Runtime Environment (RTO) contains the tools you need to run programs including scalable process management system (Hydra), supporting utilities, and shared (.so) libraries.
• The Intel® MPI Library Development Kit (SDK) includes all of the Runtime Environment components and compilation tools: compiler wrapper scripts (mpicc, mpiicc, etc.), include files and modules, static (.a) libraries, debug libraries, and test codes.

What's New

Intel® MPI Library 2018 Update 1

• Improved startup performance on many/multicore systems (I_MPI_STARTUP_MODE).
• Bug fixes.

Intel® MPI Library 2018

• Improved startup times for Hydra when using shm:ofi or shm:tmi.
• Hard finalization is now the default.
• The default fabric list is changed when Intel® Omni-Path Architecture is detected.
• Added environment variables: I_MPI_OFI_ENABLE_LMT, I_MPI_OFI_MAX_MSG_SIZE, I_MPI_{C,CXX,FC,F}FLAGS, I_MPI_LDFLAGS, I_MPI_FORT_BIND.
• Removed support for the Intel® Xeon Phi™ coprocessor (code named Knights Corner).
• I_MPI_DAPL_TRANSLATION_CACHE is now disabled by default.
• Deprecated support for the IPM statistics format.
• Documentation is now online.

Intel® MPI Library 2017 Update 2

• Added environment variables I_MPI_HARD_FINALIZE and I_MPI_MEMORY_SWAP_LOCK.

Intel® MPI Library 2017 Update 1

• PMI-2 support for SLURM*, improved SLURM support by default.
• Improved mini help and diagnostic messages, man1 pages for mpiexec.hydra, hydra_persist, and hydra_nameserver.
• Deprecations:
  • Intel® Xeon Phi™ coprocessor (code named Knights Corner) support.
  • Cross-OS launches support.
  • DAPL, TMI, and OFA fabrics support.

Intel® MPI Library 2017

• Support for the MPI-3.1 standard.
• New topology-aware collective communication algorithms (I_MPI_ADJUST family).
• Effective MCDRAM (NUMA memory) support. See the Developer Reference, section Tuning Reference > Memory Placement Policy Control for more information.
• Controls for asynchronous progress thread pinning (I_MPI_ASYNC_PROGRESS).
• Direct receive functionality for the OFI* fabric (I_MPI_OFI_DRECV).
• PMI2 protocol support (I_MPI_PMI2).
• New process startup method (I_MPI_HYDRA_PREFORK).
• Startup improvements for the SLURM* job manager (I_MPI_SLURM_EXT).
• New algorithm for MPI-IO collective read operation on the Lustre* file system (I_MPI_LUSTRE_STRIPE_AWARE).
• Debian Almquist (dash) shell support in compiler wrapper scripts and mpitune.
• Performance tuning for processors based on Intel® microarchitecture codenamed Broadwell and for Intel® Omni-Path Architecture (Intel® OPA).
• Performance tuning for Intel® Xeon Phi™ Processor and Coprocessor (code named Knights Landing) and Intel® OPA.
• OFI latency and message rate improvements.
• OFI is now the default fabric for Intel® OPA and Intel® True Scale Fabric.
• MPD process manager is removed.
• Dedicated pvfs2 ADIO driver is disabled.
• SSHM support is removed.
• Support for the Intel® microarchitectures older than the generation codenamed Sandy Bridge is deprecated.
• Documentation improvements.

Key Features

• MPI-1, MPI-2.2 and MPI-3.1 specification conformance.
• Support for Intel® Xeon Phi™ processors (formerly code named Knights Landing).
• MPICH ABI compatibility.
• Support for any combination of the following network fabrics:
  • Network fabrics supporting Intel® Omni-Path Architecture (Intel® OPA) devices, through either Tag Matching Interface (TMI) or OpenFabrics Interface* (OFI*).
  • Network fabrics with tag matching capabilities through Tag Matching Interface (TMI), such as Intel® True Scale Fabric, Infiniband*, Myrinet* and other interconnects.
  • Native InfiniBand* interface through OFED* verbs provided by Open Fabrics Alliance* (OFA*).
  • Open Fabrics Interface* (OFI*).
  • RDMA-capable network fabrics through DAPL*, such as InfiniBand* and Myrinet*.
  • Sockets, for example, TCP/IP over Ethernet*, Gigabit Ethernet*, and other interconnects.
• (SDK only) Support for Intel® 64 architecture and Intel® MIC Architecture clusters using:
  • Intel® C++/Fortran Compiler 14.0 and newer.
  • GNU* C, C++ and Fortran 95 compilers.
• (SDK only) C, C++, Fortran 77, Fortran 90, and Fortran 2008 language bindings.
• (SDK only) Dynamic or static linking.

System Requirements

Hardware Requirements

• Systems based on the Intel® 64 architecture, in particular:
  • Intel® Core™ processor family
  • Intel® Xeon® E5 v4 processor family recommended
  • Intel® Xeon® E7 v3 processor family recommended
  • 2nd Generation Intel® Xeon Phi™ Processor (formerly code named Knights Landing)
• 1 GB of RAM per core (2 GB recommended)
• 1 GB of free hard disk space

Software Requirements

• Operating systems:
  • Red Hat* Enterprise Linux* 6, 7
  • Fedora* 23, 24
  • CentOS* 6, 7
  • SUSE* Linux Enterprise Server* 11, 12
  • Ubuntu* LTS 14.04, 16.04
  • Debian* 7, 8
• (SDK only) Compilers:
  • GNU*: C, C++, Fortran 77 3.3 or newer, Fortran 95 4.4.0 or newer
  • Intel® C++/Fortran Compiler 15.0 or newer
• Debuggers:
  • Rogue Wave* Software TotalView* 6.8 or newer
  • Allinea* DDT* 1.9.2 or newer
  • GNU* Debuggers 7.4 or newer
• Batch systems:
  • Platform* LSF* 6.1 or newer
  • Altair* PBS Pro* 7.1 or newer
  • Torque* 1.2.0 or newer
  • Parallelnavi* NQS* V2.0L10 or newer
  • NetBatch* v6.x or newer
  • SLURM* 1.2.21 or newer
  • Univa* Grid Engine* 6.1 or newer
  • IBM* LoadLeveler* 4.1.1.5 or newer
  • Platform* Lava* 1.0
• Recommended InfiniBand* software:
  • OpenFabrics* Enterprise Distribution (OFED*) 1.5.4.1 or newer
  • Intel® True Scale Fabric Host Channel Adapter Host Drivers & Software (OFED) v7.2.0 or newer
  • Mellanox* OFED* 1.5.3 or newer
• Virtual environments:
  • Docker* 1.13.0
• Additional software:
  • The memory placement functionality for NUMA nodes requires the libnuma.so library and numactl utility installed. numactl should include numactl, numactl-devel and numactl-libs.

Notes for Cluster Installation

When installing the Intel® MPI Library on all the nodes of your cluster without using a shared file system, you need to establish passwordless SSH connection between the cluster nodes. This process is described in detail in the Intel® Parallel Studio XE Installation Guide (see section 2.1).

Known Issues and Limitations

• If you observe performance degradation with an MPI application that utilizes the RMA functionality, you are recommended to set I_MPI_SCALABLE_OPTIMIZATION = 0 to get a performance gain.
• The I_MPI_JOB_FAST_STARTUP variable takes effect only when shm is selected as the intra-node fabric.
• ILP64 is not supported by MPI modules for Fortran* 2008.
• In case of program termination (like signal), remove trash in the /dev/shm/ directory manually with:

  rm -r /dev/shm/shm-col-space-*

• In case of large number of simultaneously used communicators (more than 10,000) per node, it is recommended to increase the maximum numbers of memory mappings with one of the following methods:

  • echo 1048576 > /proc/sys/vm/max_map_count

  • sysctl -w vm.max_map_count=1048576

  • disable shared memory collectives by setting the variable: I_MPI_COLL_INTRANODE=pt2pt
• On some Linux* distributions Intel® MPI Library may fail for non-root users due to security limitations. This was observed on Ubuntu* 12.04, and could impact other distributions and versions as well. Two workarounds exist:
  • Enable ptrace for non-root users with:

    echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

  • Revert the Intel® MPI Library to an earlier shared memory mechanism, which is not impacted, by setting: I_MPI_SHM_LMT=shm
• Ubuntu* does not allow attaching a debugger to a non-child process. In order to use -gdb, this behavior must be disabled by setting the sysctl value in /proc/sys/kernel/yama/ptrace_scope to 0.
• Cross-OS runs using ssh from a Windows* host fail. Two workarounds exist:
  • Create a symlink on the Linux* host that looks identical to the Windows* path to pmi_proxy.
  • Start hydra_persist on the Linux* host in the background (hydra_persist &) and use -bootstrap service from the Windows* host. This requires that the Hydra service also be installed and started on the Windows* host.
• The OFA fabric and certain DAPL providers may not work or provide worthwhile performance with the Intel® Omni-Path Fabric. For better performance, try choosing the OFI or TMI fabric.
• Enabling statistics gathering may result in increased time in MPI_Finalize.
• In systems where some nodes have only Intel® True Scale Fabric or Intel® Omni-Path Fabric available, while others have both Intel® True Scale and e.g. Mellanox* HCAs, automatic fabric detection will lead to a hang or failure, as the first type of nodes will select ofi/tmi, and the second type will select dapl as the internode fabric. To avoid this, explicitly specify a fabric that is available on all the nodes.
• In order to run a mixed OS job (Linux* and Windows*), all binaries must link to the same single or multithreaded MPI library.  The single- and multithreaded libraries are incompatible with each other and should not be mixed. Note that the pre-compiled binaries for the Intel® MPI Benchmarks are inconsistent (Linux* version links to multithreaded, Windows* version links to single threaded) and as such, at least one must be rebuilt to match the other.
• Intel® MPI Library does not support using the OFA fabric over an Intel® Symmetric Communications Interface (Intel® SCI) adapter. If you are using an Intel SCI adapter, such as with Intel® Many Integrated Core Architecture, you will need to select a different fabric.
• The TMI and OFI fabrics over PSM do not support messages larger than 2³² - 1 bytes. If you have messages larger than this limit, select a different fabric.
• If a communication between two existing MPI applications is established using the process attachment mechanism, the library does not control whether the same fabric has been selected for each application. This situation may cause unexpected applications behavior. Set the I_MPI_FABRICS variable to the same values for each application to avoid this issue.
• Do not load thread-safe libraries through dlopen(3).
• Certain DAPL providers may not function properly if your application uses system(3), fork(2), vfork(2), or clone(2) system calls. Do not use these system calls or functions based upon them. For example, system(3), with OFED* DAPL provider with Linux* kernel version earlier than official version 2.6.16. Set the RDMAV_FORK_SAFE environment variable to enable the OFED workaround with compatible kernel version.
• MPI_Mprobe, MPI_Improbe, and MPI_Cancel are not supported by the TMI and OFI fabrics.
• You may get an error message at the end of a checkpoint-restart enabled application, if some of the application processes exit in the middle of taking a checkpoint image. Such an error does not impact the application and can be ignored. To avoid this error, set a larger number than before for the -checkpoint-interval option. The error message may look as follows:

  [proxy:0:0@hostname] HYDT_ckpoint_blcr_checkpoint (./tools/ckpoint/blcr/
  ckpoint_blcr.c:313): cr_poll_checkpoint failed: No such process
  [proxy:0:0@hostname] ckpoint_thread (./tools/ckpoint/ckpoint.c:559):
  blcr checkpoint returned error
  [proxy:0:0@hostname] HYDT_ckpoint_finalize (./tools/ckpoint/ckpoint.c:878)
   : Error in checkpoint thread 0x7

• Intel® MPI Library requires the presence of the /dev/shm device in the system. To avoid failures related to the inability to create a shared memory segment, make sure the /dev/shm device is set up correctly.
• Intel® MPI Library uses TCP sockets to pass stdin stream to the application. If you redirect a large file, the transfer can take long and cause the communication to hang on the remote side. To avoid this issue, pass large files to the application as command line options.
• DAPL auto provider selection mechanism and improved NUMA support require dapl-2.0.37 or newer.
• If you set I_MPI_SHM_LMT=direct, the setting has no effect if the Linux* kernel version is lower than 3.2.
• When using the Linux boot parameter isolcpus with an Intel® Xeon Phi™ processor using default MPI settings, an application launch may fail. If possible, change or remove the isolcpus Linux boot parameter. If it is not possible, you can try setting I_MPI_PIN to off.
• In some cases, collective calls over the OFA fabric may provide incorrect results. Try setting I_MPI_ADJUST_ALLGATHER to a value between 1 and 4 to resolve the issue.

Technical Support

Every purchase of an Intel® Software Development Product includes a year of support services, which provides Priority Support at our Online Service Center web site.

In order to get support you need to register your product in the Intel® Registration Center. If your product is not registered, you will not receive Priority Support.