Distributed Memory Coarray Fortran with the Intel Fortran Compiler for Linux: Essential Guide

Introduction : 
This is an essential guide to using the Coarray Fortran (CAF) feature of the Intel Fortran Composer XE 2013 SP1 and Intel Parallel Studio XE 2015, Cluster Edition for Linux on a distributed memory cluster.

Version : 
To use the distributed memory feature of Intel CAF, you must have a licensed copy of the Intel® Cluster Studio 2013 SP1 or Intel® Parallel Studio XE 2015, Cluster Edition.  The shared memory, aka single-node, CAF feature is available in ANY editon of Intel® Parallel Studio XE 2015.  The CAF feature is currently not available under Mac OS X, and the distributed memory CAF feature is only available under the Linux operating system and ONLY in the CLUSTER EDITION of Intel® Parallel Studio XE 2015 for Linux.

Configuration Set Up : 
In order to run a distributed memory Coarray Fortran application, you must have an established linux cluster and an installation of Intel MPI version 5 or greater.  BEFORE ATTEMPTING TO RUN INTEL CAF, make sure you can run a simple MPI 'hello world' program on your cluster across multiple nodes using Intel MPI.

New to Intel MPI?  Go HERE

Successful configuration and running of MPI jobs under Intel MPI is a prerequisite to using the Intel Fortran CAF feature in distributed memory mode.  In order to get support on the Intel CAF feature you will be asked to demonstrate the ability to run a 'hello world' MPI program on your cluster.  Please read the Intel MPIRelease Notes and Getting_Started documents that come with your Intel MPI installation in the <install directory>/doc/ directory.

Before running an Intel CAF application, perform the following steps:

1) set up a "machine" fle:  If your cluster hosts are fixed and you do NOT run under a batch system with PBS, set up a static hosts file.  In your home directory, create a file with the names of your cluster hostnames, one host per line optionally with the number of cores (processors) on each host, or the number of CAF images to place on each node: similar to this:
nodehostname1:4
nodehostname2:4
...
nodehostnameN:4

This says each node will run 4 MPI processes.  Syntax is <hostname>[:<number of CAF images>]

You may use any name for the file, by convention "machinefile" or "hostsfile" are probably easiest to remember and maintain.  If you are running under a batch system where the hosts are assigned dynamically, see the Intel MPI User Guide for details on host selection.

2) source Intel MPI and Intel Fortran compiler "...vars.sh" or "...vars.csh" files:  You must set up the paths to Intel MPI and Intel Fortran in your environment.  Furthermore, these should be sourced by child processes.  Thus it is recommended to perform the following source commands and/or add these to your .bashrc or .cshrc files in your home directory:

source <path to Intel MPI installation>/[ia32 | intel64]/bin/mpivars.sh
source <path to Intel Fortran installation>/bin/compilervars.sh [ia32 | intel64]
where you choose between 32 and 64 bit environments with ia32 or intel64 respectively.

3) Setup a Coarray Fortran (CAF) configuration file:  When you run a distributed memory Coarray Fortran program, the application first runs a job launcher.  The job launcher invoked to start a distributed memory CAF application will use the Intel MPI 'mpirun' command to start the job on the hosts in the cluster.  This CAF job launcher will first read the "CAF configuration file" to pick up arguments that will be passed to the Intel MPI mpiexec.hydra command.  Thus, the "CAF configuration file" is nothing more than arguments to the Intel MPI mpiexec.hydra command.  And example CAF configuration file may contain:

-genvall -genv I_MPI_FABRICS=shm:tcp -machinefile ./hostsfile -n 8 ./my_caf_prog

In the example:

• -genvall  tells the launcher that the child CAF images will inherit environment variable setting from the current shell
• -genv I_MPI_FABRICS=shm:tcp   selects the fabric(s) to use in priority order:  use shared memory within a node, TCP for remote.  See the Intel MPI User Guide for setting fabrics
• -machinefile ./hostsfile   says to find the list of cluster nodes on which to run.  For batch based systems see the Intel MPI User Guide
• -n 8   launch 8 CAF images
• ./my_caf_prog is the name of the Coarray Fortran program

There are many many commands and configurations possible in the machine file.  See the documentation on mpiexec.hydra for a complete list of possible control options.  Some other useful options to consider:

-rr       round-robin image distribution of images to nodes.  OPTIONAL. Default is to pack each node with the number of images equal to the number of cores (real cores PLUS any hyperthreaded virtual cores) one host at at time in the order specified in machine file.  Round-robin is one way to avoid using hyperthreaded cores.  With the -rr option to mpiexec, image 1 is assigned to host1 from your machine file or PBS_NODEFILE, image 2 to host2, etc. to image N on hostN, at which point the allocation cycles back to image N+1 on host1 and so on.

-perhost N    distribute images to hosts in groups of N.  OPTIONAL. This is another way to avoid hyperthreaded cores:  set N to the number of real cores on each host.  image 1..N are allocated on host1, images N+1 to N+N on host2, etc.

Building the Application : 
You are now ready to compile your Coarray Fortran application.  Create or use an existing Coarray Fortran application.  A sample Coarray Fortran 'hello world' application is included in the <compiler install dir>Samples/en_US/Fortran/coarray_samples/ directory.

The essential compiler arguments to use for distributed memory coarray applications are:

ifort -coarray=distributed -corray-config-file=<CAF config filename>

Some essential notes:  -coarray=distributed is necessary to create a distibuted memory CAF application.  This option is only available on systems with a valid Intel Cluster Edition or Cluster Studio license.  Without this license you cannot create distributed memory Coarray Fortran applications - you can, however, create and use shared memory CAF applications with any existing Intel Composer XE for Linux or Windows license.

-coarray-config-file=<CAF configuration file>   this option is used to set tell the CAF job launcher where to find the configuration file to find runtime arguments to 'mpiexec'.  This file need not exist at the time of compilation.  This file is ONLY read at job launch.  Thus, it can be changed or modified between job runs to change the number of images along with any other valid control option to 'mpiexec'.  This give the programmer a way to change the number of images and other parameters without having to recompile the application.  A reasonable name for the file may be ~/cafconfig.txt, but the name of the file and location is up to the user to decide.  One essential note:  the executable name is hard-coded in the CAF config file, so be sure that the executable name in the config file matches the name you used with the 'ifort -o <name>' option.  Also, be sure to use either the full pathname to the executable OR the current directory "dot" name, such as './a.out' or './mycafprogram.exe' as examples.

Note:  -coarray-num-images=N  compiler option is ignored for -coarray=distributed.  This option is only used by shared memory Coarray Fortran applications.  The number of images for distributed memory CAF applications is ONLY controlled at job launch by the '-n N' option in the CAF config file.

Of course, you can include any other compiler options including all optimization options.


Running the Application : 
After compiling the program, simply execute the program:   ./a.out for example.  That's it!  The CAF executable will locate your machine file and your caf config file.  The caf config file passes arguments to the mpiexec.hydra command to start up your distributed CAF program.  Host information is pulled from the machine file.

Need to change the number of images launched or the arguments to mpiexec?  Simply change the CAF config file settings in the text file.  Remember, the -coarray-config-file=  options used at compile time set the name an location for this text file.  You should use a name and location you can remember for this file, such as -coarray-config-file=~/cafconfig.txt
Then just add mpiexec options to ~/cafconfig.txt, for example
-perhost 2 -envall -n 64 ./a.out

Note:  environment variable FORT_COARRAY_NUM_IMAGES has not effect on distributed memory CAF applications.  This environment variable is only honored by a shared memory CAF image.  Only the -n option in the CAF config file is used to control the number of CAF images for a distributed memory CAF application.

Again, read the mpiexec.hydra and I_MPI_FABRICS documentation in the Intel MPI documentation set.

Known Issues or Limitations : 

Many clusters have multiple MPI implementations installed along with Intel MPI.  The PATH variable, LD_LIBRARY_PATH MUST have Intel MPI paths BEFORE any other MPI installed on your system.  Make sure to ONLY source the mpivars.sh script to set this correctly OR insure that the correct Intel MPI paths appear before other MPI paths.

Batch system notes:  In the above notes, we added the option '-envall' to the CAF config file.  This is an attempt to get your current working environment variables to be inherited by your spawned remote CAF processes.  This was done to help insure that your PATH and LD_LIBRARY_PATH contain the paths to Intel MPI and Intel Fortran AND those paths appear before other MPI and compilers on your system.  HOWEVER, some batch scheduling systems will not allow environment inheritence: in other words they will throw out your current environment variables and use defaults for these.  That is why we suggested adding the 'source <path to intel MPI>/[ia32 | intel64]/bin/mpivars.sh' to your .bashrc, .cshrc, or .bash_profile.  These dot files are invoked by each child process created, and hence, SHOULD set the PATH and LD_LIBRARY_PATH appropriately.   When in doubt, execute 'which mpiexec' interactively, or put 'echo `which mpiexec`' in your batch script to insure the Intel MPI mpiexec is being used.  Other MPI implementation 'mpiexec' commands cannot be used and will cause errors.

It is critical to insure that you can execute an Intel MPI application PRIOR to attempting to run an Intel CAF program.  Keep a simple MPI 'hello world' handy to debug your environment.  Here is a sample: