The Intel® MPI Library comes with a cache that helps to speed up the translation of memory addresses in between the MPI library and the underlying DAPL fabric. As the Intel MPI Library documentation mentions: "The cache substantially increases performance, but may lead to correctness issues in certain situations."
While the performance benefit of the cache on real world applications is not very large, starting with Intel MPI Library 2018, the RDMA translation cache has been deactivated by default.
Users of older Intel MPI Library versions may still experience issues with the cache, which could either result in numerical correctness issues or crashes in Intel MPI. Such crashes would typically look like the following:
Assertion failed in file ../../i_rtc_cache.c at line ...
Therefore, users of older Intel MPI Library versions are recommended to switch off the RDMA translation cache. This can be accomplished by setting the following Intel MPI Library environment variables:
export I_MPI_OFA_TRANSLATION_CACHE=0 export I_MPI_DAPL_TRANSLATION_CACHE=0 export I_MPI_DAPL_UD_TRANSLATION_CACHE=0
Users of the Intel MPI Library 2018 and newer will already have these settings by default.