From 6a105dd59b6e99d47340701e4b44fbea65cd62fa Mon Sep 17 00:00:00 2001 From: Vladimir Sokolovsky Date: Mon, 6 Aug 2012 15:18:53 +0300 Subject: [PATCH] Remove release notes for unsupported packages Signed-off-by: Vladimir Sokolovsky --- release_notes/mpi-selector_release_notes.txt | 43 - release_notes/mvapich2_release_notes.txt | 147 -- release_notes/mvapich_release_notes.txt | 102 - release_notes/open_mpi_release_notes.txt | 1756 ------------------ release_notes/sdp_release_notes.txt | 248 --- 5 files changed, 2296 deletions(-) delete mode 100644 release_notes/mpi-selector_release_notes.txt delete mode 100644 release_notes/mvapich2_release_notes.txt delete mode 100644 release_notes/mvapich_release_notes.txt delete mode 100644 release_notes/open_mpi_release_notes.txt delete mode 100644 release_notes/sdp_release_notes.txt diff --git a/release_notes/mpi-selector_release_notes.txt b/release_notes/mpi-selector_release_notes.txt deleted file mode 100644 index 95944dc..0000000 --- a/release_notes/mpi-selector_release_notes.txt +++ /dev/null @@ -1,43 +0,0 @@ - MPI Selector 1.0 release notes - December 2009 - ============================== - -OFED contains a simple mechanism for system administrators and end -users to select which MPI implementation they want to use. The MPI -selector functionality is not specific to any MPI implementation; it -can be used with any implementation that provides shell startup files -that correctly set the environment for that MPI. The OFED installer -will automatically add MPI selector support for each MPI that it -installs. Additional MPI's not known by the OFED installer can be -listed in the MPI selector; see the mpi-selector(1) man page for -details. - -Note that MPI selector only affects the default MPI environment for -*future* shells. Specifically, if you use MPI selector to select MPI -implementation ABC, this default selection will not take effect until -you start a new shell (e.g., logout and login again). Other packages -(such as environment modules) provide functionality that allows -changing your environment to point to a new MPI implementation in the -current shell. The MPI selector was not meant to duplicate or replace -that functionality. - -The MPI selector functionality can be invoked in one of two ways: - -1. The mpi-selector-menu command. - - This command is a simple, menu-based program that allows the - selection of the system-wide MPI (usually only settable by root) - and a per-user MPI selection. It also shows what the current - selections are. - - This command is recommended for all users. - -2. The mpi-selector command. - - This command is a CLI-equivalent of the mpi-selector-menu, - allowing for the same functionality as mpi-selector-menu but - without the interactive menus and prompts. It is suitable for - scripting. - -See the mpi-selector(1) man page for more information. - diff --git a/release_notes/mvapich2_release_notes.txt b/release_notes/mvapich2_release_notes.txt deleted file mode 100644 index 7e1339a..0000000 --- a/release_notes/mvapich2_release_notes.txt +++ /dev/null @@ -1,147 +0,0 @@ -================================================================================ - - Open Fabrics Enterprise Distribution (OFED) - MVAPICH2-1.7 in OFED 1.5.4 Release Notes - - October 2011 - - -Overview --------- - -These are the release notes for MVAPICH2-1.7. MVAPICH2 is an MPI-2 -implementation over InfiniBand, iWARP and RoCE (RDMA over Converged Ethernet) -from the Ohio State University (http://mvapich.cse.ohio-state.edu/). - - -User Guide ----------- - -For more information on using MVAPICH2-1.7, please visit the user guide at -http://mvapich.cse.ohio-state.edu/support/. - - -Software Dependencies ---------------------- - -MVAPICH2 depends on the installation of the OFED Distribution stack with OpenSM -running. The MPI module also requires an established network interface (either -InfiniBand, IPoIB, iWARP, RoCE, uDAPL, or Ethernet). BLCR support is needed if -built with fault tolerance support. Similarly, HWLOC support is needed if built -with Portable Hardware Locality feature for CPU mapping. - - -ChangeLog ---------- - -* Features and Enhancements (since MVAPICH2 1.6) - - Based on MPICH2-1.4.1p1 - - Integrated Hybrid (UD-RC/XRC) design to get best performance on - large-scale systems with reduced/constant memory footprint - - CH3 shared memory channel for standalone hosts (including laptops) - without any InfiniBand adapters - - HugePage support - - Improved intra-node shared memory communication performance - - Shared memory backed windows for One-Sided Communication - - Support for truly passive locking for intra-node RMA in shared memory and - LIMIC based windows - - Improved on-demand InfiniBand connection setup (CH3 and RoCE) - - Tuned RDMA Fast Path Buffer size to get better performance with less - memory footprint (CH3 and Nemesis) - - Support for large data transfers (>2GB) - - Integrated with enhanced LiMIC2 (v0.5.5) to support Intra-node large - message (>2GB) transfers - - Optimized Fence synchronization (with and without LIMIC2 support) - - Automatic intra-node communication parameter tuning based on platform - - Efficient connection set-up for multi-core systems - - Enhanced designs and tuning for collectives (bcast, reduce, barrier, - gather, allreduce, allgather, gatherv, allgatherv and alltoall) - - Support for shared-memory collectives for modern clusters with up to 64 - cores/node - - MPI_THREAD_SINGLE provided by default and MPI_THREAD_MULTIPLE as an - option - - Fast process migration using RDMA - - Enabling Checkpoint/Restart support in pure SMP mode - - Compact and shorthand way to specify blocks of processes on the same host - with mpirun_rsh - - Support for latest stable version of HWLOC v1.2.2 - - Enhanced mpirun_rsh design to avoid race conditions, support for - fault-tolerance functionality and improved debug messages - - Enhanced debugging config options to generate core files and back-traces - - Automatic inter-node communication parameter tuning based on platform and - adapter detection (Nemesis) - - Integrated with latest OSU Micro-benchmarks (3.4) - - Improved performance for medium sized messages (QLogic PSM interface) - - Multi-core-aware collective support (QLogic PSM interface) - - Performance optimization for QDR cards - - Support for Chelsio T4 Adapter - - Support for Ekopath Compiler - -* Bug fixes (since MVAPICH2 1.6) - - Fixes in Checkpoint/Restart and Migration support - - Fix Restart when using automatic checkpoint - - Thanks to Alexandr for reporting this - - Handling very large one-sided transfers using RDMA - - Fixes for memory leaks - - Graceful handling of unknown HCAs - - Better handling of shmem file creation errors - - Fix for a hang in intra-node transfer - - Fix for a build error with --disable-weak-symbols - - Thanks to Peter Willis for reporting this issue - - Fixes for one-sided communication with passive target synchronization - - Better handling of memory allocation and registration failures - - Fixes for compilation warnings - - Fix a bug that disallows '=' from mpirun_rsh arguments - - Handling of non-contiguous transfer in Nemesis interface - - Bug fix in gather collective when ranks are in cyclic order - - Fix for the ignore_locks bug in MPI-IO with Lustre - - Compiler preference lists reordered to avoid mixing GCC and Intel - compilers if both are found by configure - - Fix a bug in transferring very large messages (>2GB) - - Thanks to Tibor Pausz from Univ. of Frankfurt for reporting it - - Fix a hang with One-Sided Put operation - - Fix a bug in ptmalloc integration - - Avoid double-free crash with mpispawn - - Avoid crash and print an error message in mpirun_rsh when the hostfile is - empty - - Checking for error codes in PMI design - - Verify programs can link with LiMIC2 at runtime - - Fix for compilation issue when BLCR or FTB installed in non-system paths - - Fix an issue with RDMA-Migration - - Fix a hang with RDMA CM - - Fix an issue in supporting RoCE with second port on available on HCA - - Thanks to Jeffrey Konz from HP for reporting it - - Fix for a hang with passive RMA tests (QLogic PSM interface) - -Main Verification Flows ------------------------ - -In order to verify the correctness of MVAPICH2-1.7, the following tests -and parameters were run. - -Test Description -================================================================================ -Intel Intel's MPI functionality test suite -OSU Benchmarks OSU's performance tests -IMB Intel's MPI Benchmark test -mpich2 Test suite distributed with MPICH2 -NAS NAS Parallel Benchmarks (NPB3.2) -SPEC MPI2007 SPEC's benchmark suite for MPI - - -User Guide ----------- - -The MVAPICH2 team provides a very detailed guide to build, install and use -MVAPICH2 on various platforms. The latest version of the user guide can be -obtained from http://mvapich.cse.ohio-state.edu/support/ - -Mailing List ------------- - -There is a public mailing list mvapich-discuss@cse.ohio-state.edu for mvapich -users and developers to -- Ask for help and support from each other and get prompt response -- Contribute patches and enhancements - -================================================================================ diff --git a/release_notes/mvapich_release_notes.txt b/release_notes/mvapich_release_notes.txt deleted file mode 100644 index 8c872ae..0000000 --- a/release_notes/mvapich_release_notes.txt +++ /dev/null @@ -1,102 +0,0 @@ - Open Fabrics Enterprise Distribution (OFED) - OSU MPI MVAPICH-1.2.0, in OFED 1.5 Release Notes - - December 2009 - - -=============================================================================== -Table of Contents -=============================================================================== -1. Overview -2. Software Dependencies -3. New Features -4. Bug Fixes -5. Known Issues -6. Main Verification Flows - - -=============================================================================== -1. Overview -=============================================================================== -These are the release notes for OSU MPI MVAPICH-1.2.0. -OSU MPI is an MPI channel implementation over InfiniBand -by Ohio State University (OSU). - -See http://mvapich.cse.ohio-state.edu - - -=============================================================================== -2. Software Dependencies -=============================================================================== -OSU MPI depends on the installation of the OFED stack with OpenSM running. -The MPI module also requires an established network interface (either -InfiniBand IPoIB or Ethernet). - - -=============================================================================== -3. New Features ( Compared to mvapich 1.1.0 ) -=============================================================================== -MVAPICH-1.2.0 has the following additional features: -- Advanced network recovery support -- mpirun launcher improvements -- Efficient intra-node shared memory communication - support for diskless clusters -- RoCEE (RDMAoE) networks support - -=============================================================================== -4. Bug Fixes ( Compared to mvapich 1.1.0 ) -=============================================================================== -- Multiple fixes for mpirun_rsh launcher - -=============================================================================== -5. Known Issues -=============================================================================== -- Shared memory broadcast optimization is disabled by default. - -- MVAPICH MPI compiled on AMD x86_64 does not work with MVAPICH MPI compiled - on Intel X86_64 (EM64t). - Workaround: - Use "VIADEV_USE_COMPAT_MODE=1" run time option in order to enable compatibility - mode that works for AMD and Intel platform. - -- A process running MPI cannot fork after MPI_Init unless the environment - variable IBV_FORK_SAFE=1 is set to enable fork support. This support also - requires a kernel version of 2.6.16 or higher. - -- For users of Mellanox Technologies firmware fw-23108 or fw-25208 only: - MVAPICH might fail in its default configuration if your HCA is burnt with an - fw-23108 version that is earlier than 3.4.000, or with an fw-25208 version - 4.7.400 or earlier. - - NOTE: There is no issue if you chose to update firmware during Mellanox - OFED installation as newer firmware versions were burnt. - - Workaround: - Option 1 - Update the firmware. For instructions, see Mellanox Firmware Tools - (MFT) User's Manual under the docs/ folder. - Option 2 - In mvapich.conf, set VIADEV_SRQ_ENABLE=0 - -- MVAPICH may fail to run on some SLES 10 machines due to problems in resolving - the host name. - Workaround: Edit /etc/hosts and comment-out/remove the line that maps - IP address 127.0.0.2 to the system's fully qualified hostname. - - -=============================================================================== -6. Main Verification Flows -=============================================================================== -In order to verify the correctness of MVAPICH, the following tests and -parameters were run. - -Test Description -------------------------------------------------------------------- -Intel's Test suite - 1400 Intel tests -BW/LT OSU's test for bandwidth latency -IMB Intel's MPI Benchmark test -mpitest b_eff test -Presta Presta multicast test -Linpack Linpack benchmark -NAS2.3 NAS NPB2.3 tests -SuperLU SuperLU benchmark (NERSC edition) -NAMD NAMD application -CAM CAM application diff --git a/release_notes/open_mpi_release_notes.txt b/release_notes/open_mpi_release_notes.txt deleted file mode 100644 index e231b04..0000000 --- a/release_notes/open_mpi_release_notes.txt +++ /dev/null @@ -1,1756 +0,0 @@ - Open Fabrics Enterprise Distribution (OFED) - Open MPI in OFED 1.5.1 Copyrights, License, and Release Notes - - March 2010 - -Open MPI Copyrights -------------------- -Most files in this release are marked with the copyrights of the -organizations who have edited them. The copyrights below generally -reflect members of the Open MPI core team who have contributed code to -this release. The copyrights for code used under license from other -parties are included in the corresponding files. - -Copyright (c) 2004-2008 The Trustees of Indiana University and Indiana - University Research and Technology - Corporation. All rights reserved. -Copyright (c) 2004-2009 The University of Tennessee and The University - of Tennessee Research Foundation. All rights - reserved. -Copyright (c) 2004-2008 High Performance Computing Center Stuttgart, - University of Stuttgart. All rights reserved. -Copyright (c) 2004-2007 The Regents of the University of California. - All rights reserved. -Copyright (c) 2006-2009 Los Alamos National Security, LLC. All rights - reserved. -Copyright (c) 2006-2009 Cisco Systems, Inc. All rights reserved. -Copyright (c) 2006-2008 Voltaire, Inc. All rights reserved. -Copyright (c) 2006-2008 Sandia National Laboratories. All rights - reserved. -Copyright (c) 2006-2009 Sun Microsystems, Inc. All rights reserved. - Use is subject to license terms. -Copyright (c) 2006-2009 The University of Houston. All rights - reserved. -Copyright (c) 2006-2008 Myricom, Inc. All rights reserved. -Copyright (c) 2007-2008 UT-Battelle, LLC. All rights reserved. -Copyright (c) 2007-2008 IBM Corporation. All rights reserved. -Copyright (c) 1998-2005 Forschungszentrum Juelich, Juelich - Supercomputing - Centre, Federal Republic of Germany -Copyright (c) 2005-2008 ZIH, TU Dresden, Federal Republic of Germany -Copyright (c) 2007 Evergrid, Inc. All rights reserved. -Copyright (c) 2008 Institut National de Recherche en - Informatique. All rights reserved. -Copyright (c) 2007 Lawrence Livermore National Security, LLC. - All rights reserved. -Copyright (c) 2007-2010 Mellanox Technologies. All rights reserved. -Copyright (c) 2006 QLogic Corporation. All rights reserved. - -Additional copyrights may follow - -Open MPI License ----------------- -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - -- Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - -- Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer listed - in this license in the documentation and/or other materials - provided with the distribution. - -- Neither the name of the copyright holders nor the names of its - contributors may be used to endorse or promote products derived from - this software without specific prior written permission. - -The copyright holders provide no reassurances that the source code -provided does not infringe any patent, copyright, or any other -intellectual property rights of third parties. The copyright holders -disclaim any liability to any recipient for claims brought against -recipient by any third party for infringement of that parties -intellectual property rights. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -=========================================================================== - -When submitting questions and problems, be sure to include as much -extra information as possible. This web page details all the -information that we request in order to provide assistance: - - http://www.open-mpi.org/community/help/ - -The best way to report bugs, send comments, or ask questions is to -sign up on the user's and/or developer's mailing list (for user-level -and developer-level questions; when in doubt, send to the user's -list): - - users@open-mpi.org - devel@open-mpi.org - -Because of spam, only subscribers are allowed to post to these lists -(ensure that you subscribe with and post from exactly the same e-mail -address -- joe@example.com is considered different than -joe@mycomputer.example.com!). Visit these pages to subscribe to the -lists: - - http://www.open-mpi.org/mailman/listinfo.cgi/users - http://www.open-mpi.org/mailman/listinfo.cgi/devel - -Thanks for your time. - -=========================================================================== - -Much, much more information is also available in the Open MPI FAQ: - - http://www.open-mpi.org/faq/ - -=========================================================================== - -OFED-Specific Release Notes ---------------------------- - -** SLES 10 with Pathscale compiler support: - -Using the Pathscale compiler to build Open MPI on SLES10 may result in -a non-functional Open MPI installation (every Open MPI command fails). -If this problem occurs, try upgrading your Pathscale installation to -the latest maintenance release, or use a different compiler to compile -Open MPI. - -** Intel compiler support: - -Some versions of the Intel 9.1 C++ compiler suite series produce -incorrect code when used with the Open MPI C++ bindings. Symptoms of -this problem include crashing applications (e.g., segmentation -violations) and Open MPI producing errors about incorrect parameters. -Be sure to upgrade to the latest maintenance release of the Intel 9.1 -compiler to avoid these problems. - -** Installing newer versions of Open MPI after OFED is installed: - -Open MPI can be built from source after OFED is fully installed. The -source code for Open MPI can be extracted from the SRPM shipped with -OFED or downloaded from the main Open MPI web site: -http://www.open-mpi.org/. - -To compile with Open MPI from source with OFED support, fully install -the rest of OFED. If you used the default prefix for the OFED -installation (/usr), Open MPI should build with OpenFabrics support by -default. If you used a different OFED prefix, you must tell Open MPI -what it is with the "--with-openib=" switch to configure. -You can verify that Open MPI installed with OpenFabrics support by -running (the exact version numbers displayed may be different; the -important part is that the "openib" BTL is displayed): - - shell$ ompi_info | grep openib - MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.2) - -See the rest of the documentation below for other configure command -line options and installation instructions. - -** Changelog summary - -Showing versions 1.2.7 - 1.4; see the "NEWS" file in an Open MPI -distribution for the full list. - -1.4.1 (OFED version) ---- -- Update support for various OpenFabrics devices in the openib BTL's - .ini file. -- Fixing RDMA CM failure during QP creation (Ticket #2307) - -1.4.1 ---- -- Update to PLPA v1.3.2, addressing a licensing issue identified by - the Fedora project. See - https://svn.open-mpi.org/trac/plpa/changeset/262 for details. -- Add check for malformed checkpoint metadata files (Ticket #2141). -- Fix error path in ompi-checkpoint when not able to checkpoint - (Ticket #2138). -- Cleanup component release logic when selecting checkpoint/restart - enabled components (Ticket #2135). -- Fixed VT node name detection for Cray XT platforms, and fixed some - broken VT documentation files. -- Fix a possible race condition in tearing down RDMA CM-based - connections. -- Relax error checking on MPI_GRAPH_CREATE. Thanks to David Singleton - for pointing out the issue. -- Fix a shared memory "hang" problem that occurred on x86/x86_64 - platforms when used with the GNU >=4.4.x compiler series. -- Add fix for Libtool 2.2.6b's problems with the PGI 10.x compiler - suite. Inspired directly from the upstream Libtool patches that fix - the issue (but we need something working before the next Libtool - release). - -1.4 ---- - -The *only* change in the Open MPI v1.4 release (as compared to v1.3.4) -was to update the embedded version of Libtool's libltdl to address a -potential security vulnerability. Specifically: Open MPI v1.3.4 was -created with GNU Libtool 2.2.6a; Open MPI v1.4 was created with GNU -Libtool 2.2.6b. There are no other changes between Open MPI v1.3.4 -and v1.4. - - -1.3.4 ------ - -- Fix some issues in OMPI's SRPM with regard to shell_scripts_basename - and its use with mpi-selector. Thanks to Bill Johnstone for - pointing out the problem. -- Added many new MPI job process affinity options to mpirun. See the - newly-updated mpirun(1) man page for details. -- Several updates to mpirun's XML output. -- Update to fix a few Valgrind warnings with regards to the ptmalloc2 - allocator and Open MPI's use of PLPA. -- Many updates and fixes to the (non-default) "sm" collective - component (i.e., native shared memory MPI collective operations). -- Updates and fixes to some MPI_COMM_SPAWN_MULTIPLE corner cases. -- Fix some internal copying functions in Open MPI's use of PLPA. -- Correct some SLURM nodelist parsing logic that may have interfered - with large jobs. Additionally, per advice from the SLURM team, - change the environment variable that we use for obtaining the job's - allocation. -- Revert to an older, safer (but slower) communicator ID allocation - algorithm. -- Fixed minimum distance finding for OpenFabrics devices in the openib - BTL. -- Relax the parameter checking MPI_CART_CREATE a bit. -- Fix MPI_COMM_SPAWN[_MULTIPLE] to only error-check the info arguments - on the root process. Thanks to Federico Golfre Andreasi for - reporting the problem. -- Fixed some BLCR configure issues. -- Fixed a potential deadlock when the openib BTL was used with - MPI_THREAD_MULTIPLE. -- Fixed dynamic rules selection for the "tuned" coll component. -- Added a launch progress meter to mpirun (useful for large jobs; set - the orte_report_launch_progress MCA parameter to 1 to see it). -- Reduced the number of file descriptors consumed by each MPI process. -- Add new device IDs for Chelsio T3 RNICs to the openib BTL config file. -- Fix some CRS self component issues. -- Added some MCA parameters to the PSM MTL to tune its run-time - behavior. -- Fix some VT issues with MPI_BOTTOM/MPI_IN_PLACE. -- Man page updates from the Debain Open MPI package maintainers. -- Add cycle counter support for the Alpha and Sparc platforms. -- Pass visibility flags to libltdl's configure script, resulting in - those symbols being hidden. This appears to mainly solve the - problem of applications attempting to use different versions of - libltdl from that used to build Open MPI. - - -1.3.3 ------ - -- Fix a number of issues with the openib BTL (OpenFabrics) RDMA CM, - including a memory corruption bug, a shutdown deadlock, and a route - timeout. Thanks to David McMillen and Hal Rosenstock for help in - tracking down the issues. -- Change the behavior of the EXTRA_STATE parameter that is passed to - Fortran attribute callback functions: this value is now stored - internally in MPI -- it no longer references the original value - passed by MPI_*_CREATE_KEYVAL. -- Allow the overriding RFC1918 and RFC3330 for the specification of - "private" networks, thereby influencing Open MPI's TCP - "reachability" computations. -- Improve flow control issues in the sm btl, by both tweaking the - shared memory progression rules and by enabling the "sync" collective - to barrier every 1,000th collective. -- Various fixes for the IBM XL C/C++ v10.1 compiler. -- Allow explicit disabling of ptmalloc2 hooks at runtime (e.g., enable - support for Debian's builtroot system). Thanks to Manuel Prinz and - the rest of the Debian crew for helping identify and fix this issue. -- Various minor fixes for the I/O forwarding subsystem. -- Big endian iWARP fixes in the Open Fabrics RDMA CM support. -- Update support for various OpenFabrics devices in the openib BTL's - .ini file. -- Fixed undefined symbol issue with Open MPI's parallel debugger - message queue support so it can be compiled by Sun Studio compilers. -- Update MPI_SUBVERSION to 1 in the Fortran bindings. -- Fix MPI_GRAPH_CREATE Fortran 90 binding. -- Fix MPI_GROUP_COMPARE behavior with regards to MPI_IDENT. Thanks to - Geoffrey Irving for identifying the problem and supplying the fix. -- Silence gcc 4.1 compiler warnings about type punning. Thanks to - Number Cruncher for the fix. -- Added more Valgrind and other memory-cleanup fixes. Thanks to - various Open MPI users for help with these issues. -- Miscellaneous VampirTrace fixes. -- More fixes for openib credits in heavy-congestion scenarios. -- Slightly decrease the latency in the openib BTL in some conditions - (add "send immediate" support to the openib BTL). -- Ensure to allow MPI_REQUEST_GET_STATUS to accept an - MPI_STATUS_IGNORE parameter. Thanks to Shaun Jackman for the bug - report. -- Added Microsoft Windows support. See README.WINDOWS file for - details. - - -1.3.2 ------ - -- Fixed a potential infinite loop in the openib BTL that could occur - in senders in some frequent-communication scenarios. Thanks to Don - Wood for reporting the problem. -- Add a new checksum PML variation on ob1 (main MPI point-to-point - communication engine) to detect memory corruption in node-to-node - messages -- Add a new configuration option to add padding to the openib - header so the data is aligned -- Add a new configuration option to use an alternative checksum algo - when using the checksum PML -- Fixed a problem reported by multiple users on the mailing list that - the LSF support would fail to find the appropriate libraries at - run-time. -- Allow empty shell designations from getpwuid(). Thanks to Sergey - Koposov for the bug report. -- Ensure that mpirun exits with non-zero status when applications die - due to user signal. Thanks to Geoffroy Pignot for suggesting the - fix. -- Ensure that MPI_VERSION / MPI_SUBVERSION match what is returned by - MPI_GET_VERSION. Thanks to Rob Egan for reporting the error. -- Updated MPI_*KEYVAL_CREATE functions to properly handle Fortran - extra state. -- A variety of ob1 (main MPI point-to-point communication engine) bug - fixes that could have caused hangs or seg faults. -- Do not install Open MPI's signal handlers in MPI_INIT if there are - already signal handlers installed. Thanks to Kees Verstoep for - bringing the issue to our attention. -- Fix GM support to not seg fault in MPI_INIT. -- Various VampirTrace fixes. -- Various PLPA fixes. -- No longer create BTLs for invalid (TCP) devices. -- Various man page style and lint cleanups. -- Fix critical OpenFabrics-related bug noted here: - http://www.open-mpi.org/community/lists/announce/2009/03/0029.php. - Open MPI now uses a much more robust memory intercept scheme that is - quite similar to what is used by MX. The use of "-lopenmpi-malloc" - is no longer necessary, is deprecated, and is expected to disappear - in a future release. -lopenmpi-malloc will continue to work for the - duration of the Open MPI v1.3 and v1.4 series. -- Fix some OpenFabrics shutdown errors, both regarding iWARP and SRQ. -- Allow the udapl BTL to work on Solaris platforms that support - relaxed PCI ordering. -- Fix problem where the mpirun would sometimes use rsh/ssh to launch on - the localhost (instead of simply forking). -- Minor SLURM stdin fixes. -- Fix to run properly under SGE jobs. -- Scalability and latency improvements for shared memory jobs: convert - to using one message queue instead of N queues. -- Automatically size the shared-memory area (mmap file) to match - better what is needed; specifically, so that large-np jobs will start. -- Use fixed-length MPI predefined handles in order to provide ABI - compatibility between Open MPI releases. -- Fix building of the posix paffinity component to properly get the - number of processors in loosely tested environments (e.g., - FreeBSD). Thanks to Steve Kargl for reporting the issue. -- Fix --with-libnuma handling in configure. Thanks to Gus Correa for - reporting the problem. - - -1.3.1 ------ - -- Added "sync" coll component to allow users to synchronize every N - collective operations on a given communicator. -- Increased the default values of the IB and RNR timeout MCA parameters. -- Fix a compiler error noted by Mostyn Lewis with the PGI 8.0 compiler. -- Fix an error that prevented stdin from being forwarded if the - rsh launcher was in use. Thanks to Branden Moore for pointing out - the problem. -- Correct a case where the added datatype is considered as contiguous but - has gaps in the beginning. -- Fix an error that limited the number of comm_spawns that could - simultaneously be running in some environments -- Correct a corner case in OB1's GET protocol for long messages; the - error could sometimes cause MPI jobs using the openib BTL to hang. -- Fix a bunch of bugs in the IO forwarding (IOF) subsystem and add some - new options to output to files and redirect output to xterm. Thanks to - Jody Weissmann for helping test out many of the new fixes and - features. -- Fix SLURM race condition. -- Fix MPI_File_c2f(MPI_FILE_NULL) to return 0, not -1. Thanks to - Lisandro Dalcin for the bug report. -- Fix the DSO build of tm PLM. -- Various fixes for size disparity between C int's and Fortran - INTEGER's. Thanks to Christoph van Wullen for the bug report. -- Ensure that mpirun exits with a non-zero exit status when daemons or - processes abort or fail to launch. -- Various fixes to work around Intel (NetEffect) RNIC behavior. -- Various fixes for mpirun's --preload-files and --preload-binary - options. -- Fix the string name in MPI::ERRORS_THROW_EXCEPTIONS. -- Add ability to forward SIFTSTP and SIGCONT to MPI processes if you - set the MCA parameter orte_forward_job_control to 1. -- Allow the sm BTL to allocate larger amounts of shared memory if - desired (helpful for very large multi-core boxen). -- Fix a few places where we used PATH_MAX instead of OMPI_PATH_MAX, - leading to compile problems on some platforms. Thanks to Andrea Iob - for the bug report. -- Fix mca_btl_openib_warn_no_device_params_found MCA parameter; it - was accidentally being ignored. -- Fix some run-time issues with the sctp BTL. -- Ensure that RTLD_NEXT exists before trying to use it (e.g., it - doesn't exist on Cygwin). Thanks to Gustavo Seabra for reporting - the issue. -- Various fixes to VampirTrace, including fixing compile errors on - some platforms. -- Fixed missing MPI_Comm_accept.3 man page; fixed minor issue in - orterun.1 man page. Thanks to Dirk Eddelbuettel for identifying the - problem and submitting a patch. -- Implement the XML formatted output of stdout/stderr/stddiag. -- Fixed mpirun's -wdir switch to ensure that working directories for - multiple app contexts are properly handled. Thanks to Geoffroy - Pignot for reporting the problem. -- Improvements to the MPI C++ integer constants: - - Allow MPI::SEEK_* constants to be used as constants - - Allow other MPI C++ constants to be used as array sizes -- Fix minor problem with orte-restart's command line options. See - ticket #1761 for details. Thanks to Gregor Dschung for reporting - the problem. - -1.3 ---- - -- Extended the OS X 10.5.x (Leopard) workaround for a problem when - assembly code is compiled with -g[0-9]. Thanks to Barry Smith for - reporting the problem. See ticket #1701. -- Disabled MPI_REAL16 and MPI_COMPLEX32 support on platforms where the - bit representation of REAL*16 is different than that of the C type - of the same size (usually long double). Thanks to Julien Devriendt - for reporting the issue. See ticket #1603. -- Increased the size of MPI_MAX_PORT_NAME to 1024 from 36. See ticket #1533. -- Added "notify debugger on abort" feature. See tickets #1509 and #1510. - Thanks to Seppo Sahrakropi for the bug report. -- Upgraded Open MPI tarballs to use Autoconf 2.63, Automake 1.10.1, - Libtool 2.2.6a. -- Added missing MPI::Comm::Call_errhandler() function. Thanks to Dave - Goodell for bringing this to our attention. -- Increased MPI_SUBVERSION value in mpi.h to 1 (i.e., MPI 2.1). -- Changed behavior of MPI_GRAPH_CREATE, MPI_TOPO_CREATE, and several - other topology functions per MPI-2.1. -- Fix the type of the C++ constant MPI::IN_PLACE. -- Various enhancements to the openib BTL: - - Added btl_openib_if_[in|ex]clude MCA parameters for - including/excluding comma-delimited lists of HCAs and ports. - - Added RDMA CM support, includng btl_openib_cpc_[in|ex]clude MCA - parameters - - Added NUMA support to only use "near" network adapters - - Added "Bucket SRQ" (BSRQ) support to better utilize registered - memory, including btl_openib_receive_queues MCA parameter - - Added ConnectX XRC support (and integrated with BSRQ) - - Added btl_openib_ib_max_inline_data MCA parameter - - Added iWARP support - - Revamped flow control mechansisms to be more efficient - - "mpi_leave_pinned=1" is now the default when possible, - automatically improving performance for large messages when - application buffers are re-used -- Elimiated duplicated error messages when multiple MPI processes fail - with the same error. -- Added NUMA support to the shared memory BTL. -- Add Valgrind-based memory checking for MPI-semantic checks. -- Add support for some optional Fortran datatypes (MPI_LOGICAL1, - MPI_LOGICAL2, MPI_LOGICAL4 and MPI_LOGICAL8). -- Remove the use of the STL from the C++ bindings. -- Added support for Platform/LSF job launchers. Must be Platform LSF - v7.0.2 or later. -- Updated ROMIO with the version from MPICH2 1.0.7. -- Added RDMA capable one-sided component (called rdma), which - can be used with BTL components that expose a full one-sided - interface. -- Added the optional datatype MPI_REAL2. As this is added to the "end of" - predefined datatypes in the fortran header files, there will not be - any compatibility issues. -- Added Portable Linux Processor Affinity (PLPA) for Linux. -- Addition of a finer symbols export control via the visibiliy feature - offered by some compilers. -- Added checkpoint/restart process fault tolerance support. Initially - support a LAM/MPI-like protocol. -- Removed "mvapi" BTL; all InfiniBand support now uses the OpenFabrics - driver stacks ("openib" BTL). -- Added more stringent MPI API parameter checking to help user-level - debugging. -- The ptmalloc2 memory manager component is now by default built as - a standalone library named libopenmpi-malloc. Users wanting to - use leave_pinned with ptmalloc2 will now need to link the library - into their application explicitly. All other users will use the - libc-provided allocator instead of Open MPI's ptmalloc2. This change - may be overriden with the configure option enable-ptmalloc2-internal -- The leave_pinned options will now default to using mallopt on - Linux in the cases where ptmalloc2 was not linked in. mallopt - will also only be available if munmap can be intercepted (the - default whenever Open MPI is not compiled with --without-memory- - manager. -- Open MPI will now complain and refuse to use leave_pinned if - no memory intercept / mallopt option is available. -- Add option of using Perl-based wrapper compilers instead of the - C-based wrapper compilers. The Perl-based version does not - have the features of the C-based version, but does work better - in cross-compile environments. - - -1.2.9 ------ - -- Fix a segfault when using one-sided communications on some forms of derived - datatypes. Thanks to Dorian Krause for reporting the bug. See #1715. -- Fix an alignment problem affecting one-sided communications on - some architectures (e.g., SPARC64). See #1738. -- Fix compilation on Solaris when thread support is enabled in Open MPI - (e.g., when using --with-threads). See #1736. -- Correctly take into account the MTU that an OpenFabrics device port - is using. See #1722 and - https://bugs.openfabrics.org/show_bug.cgi?id=1369. -- Fix two datatype engine bugs. See #1677. - Thanks to Peter Kjellstrom for the bugreport. -- Fix the bml r2 help filename so the help message can be found. See #1623. -- Fix a compilation problem on RHEL4U3 with the PGI 32 bit compiler - caused by . See ticket #1613. -- Fix the --enable-cxx-exceptions configure option. See ticket #1607. -- Properly handle when the MX BTL cannot open an endpoint. See ticket #1621. -- Fix a double free of events on the tcp_events list. See ticket #1631. -- Fix a buffer overun in opal_free_list_grow (called by MPI_Init). - Thanks to Patrick Farrell for the bugreport and Stephan Kramer for - the bugfix. See ticket #1583. -- Fix a problem setting OPAL_PREFIX for remote sh-based shells. - See ticket #1580. - - -1.2.8 ------ - -- Tweaked one memory barrier in the openib component to be more conservative. - May fix a problem observed on PPC machines. See ticket #1532. -- Fix OpenFabrics IB partition support. See ticket #1557. -- Restore v1.1 feature that sourced .profile on remote nodes if the default - shell will not do so (e.g. /bin/sh and /bin/ksh). See ticket #1560. -- Fix segfault in MPI_Init_thread() if ompi_mpi_init() fails. See ticket #1562. -- Adjust SLURM support to first look for $SLURM_JOB_CPUS_PER_NODE instead of - the deprecated $SLURM_TASKS_PER_NODE environment variable. This change - may be *required* when using SLURM v1.2 and above. See ticket #1536. -- Fix the MPIR_Proctable to be in process rank order. See ticket #1529. -- Fix a regression introduced in 1.2.6 for the IBM eHCA. See ticket #1526. - - -1.2.7 ------ - -- Add some Sun HCA vendor IDs. See ticket #1461. -- Fixed a memory leak in MPI_Alltoallw when called from Fortran. - Thanks to Dave Grote for the bugreport. See ticket #1457. -- Only link in libutil when it is needed/desired. Thanks to - Brian Barret for diagnosing and fixing the problem. See ticket #1455. -- Update some QLogic HCA vendor IDs. See ticket #1453. -- Fix F90 binding for MPI_CART_GET. Thanks to Scott Beardsley for - bringing it to our attention. See ticket #1429. -- Remove a spurious warning message generated in/by ROMIO. See ticket #1421. -- Fix a bug where command-line MCA parameters were not overriding - MCA parameters set from environment variables. See ticket #1380. -- Fix a bug in the AMD64 atomics assembly. Thanks to Gabriele Fatigati - for the bug report and bugfix. See ticket #1351. -- Fix a gather and scatter bug on intercommunicators when the datatype - being moved is 0 bytes. See ticket #1331. -- Some more man page fixes from the Debian maintainers. - See tickets #1324 and #1329. -- Have openib BTL (OpenFabrics support) check for the presence of - /sys/class/infiniband before allowing itself to be used. This check - prevents spurious "OMPI did not find RDMA hardware!" notices on - systems that have the software drivers installed, but no - corresponding hardware. See tickets #1321 and #1305. -- Added vendor IDs for some ConnectX openib HCAs. See ticket #1311. -- Fix some RPM specfile inconsistencies. See ticket #1308. - Thanks to Jim Kusznir for noticing the problem. -- Removed an unused function prototype that caused warnings on - some systems (e.g., OS X). See ticket #1274. -- Fix a deadlock in inter-communicator scatter/gather operations. - Thanks to Martin Audet for the bug report. See ticket #1268. - -=========================================================================== - -Much, much more information is also available in the Open MPI FAQ: - - http://www.open-mpi.org/faq/ - -=========================================================================== - -General Release Notes ---------------------- - -Detailed Open MPI v1.3 Feature List: - - o Open MPI RunTime Environment (ORTE) improvements - - General robustness improvements - - Scalable job launch (we've seen ~16K processes in less than a - minute in a highly-optimized configuration) - - New process mappers - - Support for Platform/LSF environments (v7.0.2 and later) - - More flexible processing of host lists - - new mpirun cmd line options and associated functionality - - o Fault-Tolerance Features - - Asynchronous, transparent checkpoint/restart support - - Fully coordinated checkpoint/restart coordination component - - Support for the following checkpoint/restart services: - - blcr: Berkley Lab's Checkpoint/Restart - - self: Application level callbacks - - Support for the following interconnects: - - tcp - - mx - - openib - - sm - - self - - Improved Message Logging - - o MPI_THREAD_MULTIPLE support for point-to-point messaging in the - following BTLs (note that only MPI point-to-point messaging API - functions support MPI_THREAD_MULTIPLE; other API functions likely - do not): - - tcp - - sm - - mx - - elan - - self - - o Point-to-point Messaging Layer (PML) improvements - - Memory footprint reduction - - Improved latency - - Improved algorithm for multiple communication device - ("multi-rail") support - - o Numerous Open Fabrics improvements/enhancements - - Added iWARP support (including RDMA CM) - - Memory footprint and performance improvements - - "Bucket" SRQ support for better registered memory utilization - - XRC/ConnectX support - - Message coalescing - - Improved error report mechanism with Asynchronous events - - Automatic Path Migration (APM) - - Improved processor/port binding - - Infrastructure for additional wireup strategies - - mpi_leave_pinned is now enabled by default - - o uDAPL BTL enhancements - - Multi-rail support - - Subnet checking - - Interface include/exclude capabilities - - o Processor affinity - - Linux processor affinity improvements - - Core/socket <--> process mappings - - o Collectives - - Performance improvements - - Support for hierarchical collectives (must be activated - manually; see below) - - o Miscellaneous - - MPI 2.1 compliant - - Sparse process groups and communicators - - Support for Cray Compute Node Linux (CNL) - - One-sided RDMA component (BTL-level based rather than PML-level - based) - - Aggregate MCA parameter sets - - MPI handle debugging - - Many small improvements to the MPI C++ bindings - - Valgrind support - - VampirTrace support - - Updated ROMIO to the version from MPICH2 1.0.7 - - Removed the mVAPI IB stacks - - Display most error messages only once (vs. once for each - process) - - Many other small improvements and bug fixes, too numerous to - list here - -Known issues ------------- - - o There is a segfault that sometimes occurs on one of our x86_64 test - clusters when using MPI onesided communications over Myrinet MX. - Since no one else has reported this problem we are not holding - up the 1.3 release. See ticket #1757 for the details, and any - possible workarounds. - - o XGrid support is currently broken. - https://svn.open-mpi.org/trac/ompi/ticket/1777 - - o MPI_REDUCE_SCATTER does not work with counts of 0. - https://svn.open-mpi.org/trac/ompi/ticket/1559 - - o Please also see the Open MPI bug tracker for bugs beyond this release. - https://svn.open-mpi.org/trac/ompi/report - -=========================================================================== - -The following abbreviated list of release notes applies to this code -base as of this writing (10 July 2009): - -General notes -------------- - -- Open MPI includes support for a wide variety of supplemental - hardware and software package. When configuring Open MPI, you may - need to supply additional flags to the "configure" script in order - to tell Open MPI where the header files, libraries, and any other - required files are located. As such, running "configure" by itself - may not include support for all the devices (etc.) that you expect, - especially if their support headers / libraries are installed in - non-standard locations. Network interconnects are an easy example - to discuss -- Myrinet and OpenFabrics networks, for example, both - have supplemental headers and libraries that must be found before - Open MPI can build support for them. You must specify where these - files are with the appropriate options to configure. See the - listing of configure command-line switches, below, for more details. - -- The majority of Open MPI's documentation is here in this file, the - included man pages, and on the web site FAQ - (http://www.open-mpi.org/). This will eventually be supplemented - with cohesive installation and user documentation files. - -- Note that Open MPI documentation uses the word "component" - frequently; the word "plugin" is probably more familiar to most - users. As such, end users can probably completely substitute the - word "plugin" wherever you see "component" in our documentation. - For what it's worth, we use the word "component" for historical - reasons, mainly because it is part of our acronyms and internal API - functionc calls. - -- The run-time systems that are currently supported are: - - rsh / ssh - - LoadLeveler - - PBS Pro, Open PBS, Torque - - Platform LSF (v7.0.2 and later) - - SLURM - - XGrid (known to be broken in 1.3 through 1.3.2) - - Cray XT-3 and XT-4 - - Sun Grid Engine (SGE) 6.1, 6.2 and open source Grid Engine - - Microsoft Windows CCP (Microsoft Windows server 2003 and 2008) - -- Systems that have been tested are: - - Linux (various flavors/distros), 32 bit, with gcc, and Sun Studio 12 - - Linux (various flavors/distros), 64 bit (x86), with gcc, Absoft, - Intel, Portland, Pathscale, and Sun Studio 12 compilers (*) - - OS X (10.4), 32 and 64 bit (i386, PPC, PPC64, x86_64), with gcc - and Absoft compilers (*) - - Solaris 10 update 2, 3 and 4, 32 and 64 bit (SPARC, i386, x86_64), - with Sun Studio 10, 11 and 12 - - (*) Be sure to read the Compiler Notes, below. - -- Other systems have been lightly (but not fully tested): - - Other 64 bit platforms (e.g., Linux on PPC64) - - Microsoft Windows CCP (Microsoft Windows server 2003 and 2008); - more testing and support is expected later in the Open MPI v1.3.x - series. See the README.WINDOWS file. - -Compiler Notes --------------- - -- Mixing compilers from different vendors when building Open MPI - (e.g., using the C/C++ compiler from one vendor and the F77/F90 - compiler from a different vendor) has been successfully employed by - some Open MPI users (discussed on the Open MPI user's mailing list), - but such configurations are not tested and not documented. For - example, such configurations may require additional compiler / - linker flags to make Open MPI build properly. - -- Open MPI does not support the Sparc v8 CPU target, which is the - default on Sun Solaris. The v8plus (32 bit) or v9 (64 bit) - targets must be used to build Open MPI on Solaris. This can be - done by including a flag in CFLAGS, CXXFLAGS, FFLAGS, and FCFLAGS, - -xarch=v8plus for the Sun compilers, -mv8plus for GCC. - -- At least some versions of the Intel 8.1 compiler seg fault while - compiling certain Open MPI source code files. As such, it is not - supported. - -- The Intel 9.0 v20051201 compiler on IA64 platforms seems to have a - problem with optimizing the ptmalloc2 memory manager component (the - generated code will segv). As such, the ptmalloc2 component will - automatically disable itself if it detects that it is on this - platform/compiler combination. The only effect that this should - have is that the MCA parameter mpi_leave_pinned will be inoperative. - -- Early versions of the Portland Group 6.0 compiler have problems - creating the C++ MPI bindings as a shared library (e.g., v6.0-1). - Tests with later versions show that this has been fixed (e.g., - v6.0-5). - -- The Portland Group compilers prior to version 7.0 require the - "-Msignextend" compiler flag to extend the sign bit when converting - from a shorter to longer integer. This is is different than other - compilers (such as GNU). When compiling Open MPI with the Portland - compiler suite, the following flags should be passed to Open MPI's - configure script: - - shell$ ./configure CFLAGS=-Msignextend CXXFLAGS=-Msignextend \ - --with-wrapper-cflags=-Msignextend \ - --with-wrapper-cxxflags=-Msignextend ... - - This will both compile Open MPI with the proper compile flags and - also automatically add "-Msignextend" when the C and C++ MPI wrapper - compilers are used to compile user MPI applications. - -- Using the MPI C++ bindings with the Pathscale compiler is known - to fail, possibly due to Pathscale compiler issues. - -- Using the Absoft compiler to build the MPI Fortran bindings on Suse - 9.3 is known to fail due to a Libtool compatibility issue. - -- Open MPI will build bindings suitable for all common forms of - Fortran 77 compiler symbol mangling on platforms that support it - (e.g., Linux). On platforms that do not support weak symbols (e.g., - OS X), Open MPI will build Fortran 77 bindings just for the compiler - that Open MPI was configured with. - - Hence, on platforms that support it, if you configure Open MPI with - a Fortran 77 compiler that uses one symbol mangling scheme, you can - successfully compile and link MPI Fortran 77 applications with a - Fortran 77 compiler that uses a different symbol mangling scheme. - - NOTE: For platforms that support the multi-Fortran-compiler bindings - (i.e., weak symbols are supported), due to limitations in the MPI - standard and in Fortran compilers, it is not possible to hide these - differences in all cases. Specifically, the following two cases may - not be portable between different Fortran compilers: - - 1. The C constants MPI_F_STATUS_IGNORE and MPI_F_STATUSES_IGNORE - will only compare properly to Fortran applications that were - created with Fortran compilers that that use the same - name-mangling scheme as the Fortran compiler that Open MPI was - configured with. - - 2. Fortran compilers may have different values for the logical - .TRUE. constant. As such, any MPI function that uses the Fortran - LOGICAL type may only get .TRUE. values back that correspond to - the the .TRUE. value of the Fortran compiler that Open MPI was - configured with. Note that some Fortran compilers allow forcing - .TRUE. to be 1 and .FALSE. to be 0. For example, the Portland - Group compilers provide the "-Munixlogical" option, and Intel - compilers (version >= 8.) provide the "-fpscomp logicals" option. - - You can use the ompi_info command to see the Fortran compiler that - Open MPI was configured with. - -- The Fortran 90 MPI bindings can now be built in one of three sizes - using --with-mpi-f90-size=SIZE (see description below). These sizes - reflect the number of MPI functions included in the "mpi" Fortran 90 - module and therefore which functions will be subject to strict type - checking. All functions not included in the Fortran 90 module can - still be invoked from F90 applications, but will fall back to - Fortran-77 style checking (i.e., little/none). - - - trivial: Only includes F90-specific functions from MPI-2. This - means overloaded versions of MPI_SIZEOF for all the MPI-supported - F90 intrinsic types. - - - small (default): All the functions in "trivial" plus all MPI - functions that take no choice buffers (meaning buffers that are - specified by the user and are of type (void*) in the C bindings -- - generally buffers specified for message passing). Hence, - functions like MPI_COMM_RANK are included, but functions like - MPI_SEND are not. - - - medium: All the functions in "small" plus all MPI functions that - take one choice buffer (e.g., MPI_SEND, MPI_RECV, ...). All - one-choice-buffer functions have overloaded variants for each of - the MPI-supported Fortran intrinsic types up to the number of - dimensions specified by --with-f90-max-array-dim (default value is - 4). - - Increasing the size of the F90 module (in order from trivial, small, - and medium) will generally increase the length of time required to - compile user MPI applications. Specifically, "trivial"- and - "small"-sized F90 modules generally allow user MPI applications to - be compiled fairly quickly but lose type safety for all MPI - functions with choice buffers. "medium"-sized F90 modules generally - take longer to compile user applications but provide greater type - safety for MPI functions. - - Note that MPI functions with two choice buffers (e.g., MPI_GATHER) - are not currently included in Open MPI's F90 interface. Calls to - these functions will automatically fall through to Open MPI's F77 - interface. A "large" size that includes the two choice buffer MPI - functions is possible in future versions of Open MPI. - - -General Run-Time Support Notes ------------------------------- - -- The Open MPI installation must be in your PATH on all nodes (and - potentially LD_LIBRARY_PATH, if libmpi is a shared library), unless - using the --prefix or --enable-mpirun-prefix-by-default - functionality (see below). - -- LAM/MPI-like mpirun notation of "C" and "N" is not yet supported. - -- The XGrid support is experimental - see the Open MPI FAQ and this - post on the Open MPI user's mailing list for more information: - - http://www.open-mpi.org/community/lists/users/2006/01/0539.php - -- Open MPI's run-time behavior can be customized via MCA ("MPI - Component Architecture") parameters (see below for more information - on how to get/set MCA parameter values). Some MCA parameters can be - set in a way that renders Open MPI inoperable (see notes about MCA - parameters later in this file). In particular, some parameters have - required options that must be included. - - - If specified, the "btl" parameter must include the "self" - component, or Open MPI will not be able to deliver messages to the - same rank as the sender. For example: "mpirun --mca btl tcp,self - ..." - - If specified, the "btl_tcp_if_exclude" paramater must include the - loopback device ("lo" on many Linux platforms), or Open MPI will - not be able to route MPI messages using the TCP BTL. For example: - "mpirun --mca btl_tcp_if_exclude lo,eth1 ..." - -- Running on nodes with different endian and/or different datatype - sizes within a single parallel job is supported in this release. - However, Open MPI does not resize data when datatypes differ in size - (for example, sending a 4 byte MPI_DOUBLE and receiving an 8 byte - MPI_DOUBLE will fail). - - -MPI Functionality and Features ------------------------------- - -- All MPI-2.1 functionality is supported. - -- MPI_THREAD_MULTIPLE support is included, but is only lightly tested. - It likely does not work for thread-intensive applications. Note - that *only* the MPI point-to-point communication functions for the - BTL's listed above are considered thread safe. Other support - functions (e.g., MPI attributes) have not been certified as safe - when simultaneously used by multiple threads. - - Note that Open MPI's thread support is in a fairly early stage; the - above devices are likely to *work*, but the latency is likely to be - fairly high. Specifically, efforts so far have concentrated on - *correctness*, not *performance* (yet). - -- MPI_REAL16 and MPI_COMPLEX32 are only supported on platforms where a - portable C datatype can be found that matches the Fortran type - REAL*16, both in size and bit representation. - -- Asynchronous message passing progress using threads can be turned on - with the --enable-progress-threads option to configure. - Asynchronous message passing progress is only supported with devices - that support MPI_THREAD_MULTIPLE, but is only very lightly tested - (and may not provide very much performance benefit). - - -Collectives ------------ - -- The "hierarch" coll component (i.e., an implementation of MPI - collective operations) attempts to discover network layers of - latency in order to segregate individual "local" and "global" - operations as part of the overall collective operation. In this - way, network traffic can be reduced -- or possibly even minimized - (similar to MagPIe). The current "hierarch" component only - separates MPI processes into on- and off-node groups. - - Hierarch has had sufficient correctness testing, but has not - received much performance tuning. As such, hierarch is not - activated by default -- it must be enabled manually by setting its - priority level to 100: - - mpirun --mca coll_hierarch_priority 100 ... - - We would appreciate feedback from the user community about how well - hierarch works for your applications. - - -Network Support ---------------- - -- The OpenFabrics Enterprise Distribution (OFED) software package v1.0 - will not work properly with Open MPI v1.2 (and later) due to how its - Mellanox InfiniBand plugin driver is created. The problem is fixed - OFED v1.1 (and later). - -- Older mVAPI-based InfiniBand drivers (Mellanox VAPI) are no longer - supported. Please use an older version of Open MPI (1.2 series or - earlier) if you need mVAPI support. - -- The use of fork() with the openib BTL is only partially supported, - and only on Linux kernels >= v2.6.15 with libibverbs v1.1 or later - (first released as part of OFED v1.2), per restrictions imposed by - the OFED network stack. - -- There are two MPI network models available: "ob1" and "cm". "ob1" - uses BTL ("Byte Transfer Layer") components for each supported - network. "cm" uses MTL ("Matching Tranport Layer") components for - each supported network. - - - "ob1" supports a variety of networks that can be used in - combination with each other (per OS constraints; e.g., there are - reports that the GM and OpenFabrics kernel drivers do not operate - well together): - - OpenFabrics: InfiniBand and iWARP - - Loopback (send-to-self) - - Myrinet: GM and MX - - Portals - - Quadrics Elan - - Shared memory - - TCP - - SCTP - - uDAPL - - - "cm" supports a smaller number of networks (and they cannot be - used together), but may provide better better overall MPI - performance: - - Myrinet MX (not GM) - - InfiniPath PSM - - Portals - - Open MPI will, by default, choose to use "cm" when the InfiniPath - PSM MTL can be used. Otherwise, OB1 will be used and the - corresponding BTLs will be selected. Users can force the use of ob1 - or cm if desired by setting the "pml" MCA parameter at run-time: - - shell$ mpirun --mca pml ob1 ... - or - shell$ mpirun --mca pml cm ... - -- Myrinet MX support is shared between the 2 internal devices, the MTL - and the BTL. The design of the BTL interface in Open MPI assumes - that only naive one-sided communication capabilities are provided by - the low level communication layers. However, modern communication - layers such as Myrinet MX, InfiniPath PSM, or Portals, natively - implement highly-optimized two-sided communication semantics. To - leverage these capabilities, Open MPI provides the "cm" PML and - corresponding MTL components to transfer messages rather than bytes. - The MTL interface implements a shorter code path and lets the - low-level network library decide which protocol to use (depending on - issues such as message length, internal resources and other - parameters specific to the underlying interconnect). However, Open - MPI cannot currently use multiple MTL modules at once. In the case - of the MX MTL, process loopback and on-node shared memory - communications are provided by the MX library. Moreover, the - current MX MTL does not support message pipelining resulting in - lower performances in case of non-contiguous data-types. - - The "ob1" PML and BTL components use Open MPI's internal on-node - shared memory and process loopback devices for high performance. - The BTL interface allows multiple devices to be used simultaneously. - For the MX BTL it is recommended that the first segment (which is as - a threshold between the eager and the rendezvous protocol) should - always be at most 4KB, but there is no further restriction on the - size of subsequent fragments. - - The MX MTL is recommended in the common case for best performance on - 10G hardware when most of the data transfers cover contiguous memory - layouts. The MX BTL is recommended in all other cases, such as when - using multiple interconnects at the same time (including TCP), or - transferring non contiguous data-types. - - -Shared library versioning support ---------------------------------- - -Open MPI started using GNU-Libtool recommended shared library -versioning with the v1.3.3 release (where all versions were set to -0:0:0) for the main MPI libraries: libmpi, libmpi_cxx, libmpi_f77, and -libmpi_f90. - -Open MPI's other internal libraries are not [yet] versioned for deep -voodoo technical reasons. Please see -https://svn.open-mpi.org/trac/ompi/ticket/2092 for more details. - -=========================================================================== - -Building Open MPI ------------------ - -Open MPI uses a traditional configure script paired with "make" to -build. Typical installs can be of the pattern: - ---------------------------------------------------------------------------- -shell$ ./configure [...options...] -shell$ make all install ---------------------------------------------------------------------------- - -There are many available configure options (see "./configure --help" -for a full list); a summary of the more commonly used ones follows: - ---prefix= - Install Open MPI into the base directory named . Hence, - Open MPI will place its executables in /bin, its header - files in /include, its libraries in /lib, etc. - ---with-elan= - Specify the directory where the Quadrics Elan library and header - files are located. This option is generally only necessary if the - Elan headers and libraries are not in default compiler/linker - search paths. - - Elan is the support library for Quadrics-based networks. - ---with-elan-libdir= - Look in directory for the Quadrics Elan libraries. By default, Open - MPI will look in /lib and /lib64, - which covers most cases. This option is only needed for special - configurations. - ---with-gm= - Specify the directory where the GM libraries and header files are - located. This option is generally only necessary if the GM headers - and libraries are not in default compiler/linker search paths. - - GM is the support library for older Myrinet-based networks (GM has - been obsoleted by MX). - ---with-gm-libdir= - Look in directory for the GM libraries. By default, Open MPI will - look in /lib and /lib64, which covers - most cases. This option is only needed for special configurations. - ---with-mx= - Specify the directory where the MX libraries and header files are - located. This option is generally only necessary if the MX headers - and libraries are not in default compiler/linker search paths. - - MX is the support library for Myrinet-based networks. - ---with-mx-libdir= - Look in directory for the MX libraries. By default, Open MPI will - look in /lib and /lib64, which covers - most cases. This option is only needed for special configurations. - ---with-openib= - Specify the directory where the OpenFabrics (previously known as - OpenIB) libraries and header files are located. This option is - generally only necessary if the OpenFabrics headers and libraries - are not in default compiler/linker search paths. - - "OpenFabrics" refers to iWARP- and InifiniBand-based networks. - ---with-openib-libdir= - Look in directory for the OpenFabrics libraries. By default, Open - MPI will look in /lib and /lib64, which covers most cases. This option is only - needed for special configurations. - ---with-portals= - Specify the directory where the Portals libraries and header files - are located. This option is generally only necessary if the Portals - headers and libraries are not in default compiler/linker search - paths. - - Portals is the support library for Cray interconnects, but is also - available on other platforms (e.g., there is a Portals library - implemented over regular TCP). - ---with-portals-config= - Configuration to use for Portals support. The following - values are possible: "utcp", "xt3", "xt3-modex" (default: utcp). - ---with-portals-libs= - Additional libraries to link with for Portals support. - ---with-psm= - Specify the directory where the QLogic InfiniPath PSM library and - header files are located. This option is generally only necessary - if the InfiniPath headers and libraries are not in default - compiler/linker search paths. - - PSM is the support library for QLogic InfiniPath network adapters. - ---with-psm-libdir= - Look in directory for the PSM libraries. By default, Open MPI will - look in /lib and /lib64, which covers - most cases. This option is only needed for special configurations. - ---with-sctp= - Specify the directory where the SCTP libraries and header files are - located. This option is generally only necessary if the SCTP headers - and libraries are not in default compiler/linker search paths. - - SCTP is a special network stack over ethernet networks. - ---with-sctp-libdir= - Look in directory for the SCTP libraries. By default, Open MPI will - look in /lib and /lib64, which covers - most cases. This option is only needed for special configurations. - ---with-udapl= - Specify the directory where the UDAPL libraries and header files are - located. Note that UDAPL support is disabled by default on Linux; - the --with-udapl flag must be specified in order to enable it. - Specifying the directory argument is generally only necessary if the - UDAPL headers and libraries are not in default compiler/linker - search paths. - - UDAPL is the support library for high performance networks in Sun - HPC ClusterTools and on Linux OpenFabrics networks (although the - "openib" options are preferred for Linux OpenFabrics networks, not - UDAPL). - ---with-udapl-libdir= - Look in directory for the UDAPL libraries. By default, Open MPI - will look in /lib and /lib64, - which covers most cases. This option is only needed for special - configurations. - ---with-lsf= - Specify the directory where the LSF libraries and header files are - located. This option is generally only necessary if the LSF headers - and libraries are not in default compiler/linker search paths. - - LSF is a resource manager system, frequently used as a batch - scheduler in HPC systems. - - NOTE: If you are using LSF version 7.0.5, you will need to add - "LIBS=-ldl" to the configure command line. For example: - - ./configure LIBS=-ldl --with-lsf ... - - This workaround should *only* be needed for LSF 7.0.5. - ---with-lsf-libdir= - Look in directory for the LSF libraries. By default, Open MPI will - look in /lib and /lib64, which covers - most cases. This option is only needed for special configurations. - ---with-tm= - Specify the directory where the TM libraries and header files are - located. This option is generally only necessary if the TM headers - and libraries are not in default compiler/linker search paths. - - TM is the support library for the Torque and PBS Pro resource - manager systems, both of which are frequently used as a batch - scheduler in HPC systems. - ---with-sge - Specify to build support for the Sun Grid Engine (SGE) resource - manager. SGE support is disabled by default; this option must be - specified to build OMPI's SGE support. - - The Sun Grid Engine (SGE) is a resource manager system, frequently - used as a batch scheduler in HPC systems. - ---with-mpi-param_check(=value) - "value" can be one of: always, never, runtime. If --with-mpi-param - is not specified, "runtime" is the default. If --with-mpi-param - is specified with no value, "always" is used. Using - --without-mpi-param-check is equivalent to "never". - - - always: the parameters of MPI functions are always checked for - errors - - never: the parameters of MPI functions are never checked for - errors - - runtime: whether the parameters of MPI functions are checked - depends on the value of the MCA parameter mpi_param_check - (default: yes). - ---with-threads=value - Since thread support (both support for MPI_THREAD_MULTIPLE and - asynchronous progress) is only partially tested, it is disabled by - default. To enable threading, use "--with-threads=posix". This is - most useful when combined with --enable-mpi-threads and/or - --enable-progress-threads. - ---enable-mpi-threads - Allows the MPI thread level MPI_THREAD_MULTIPLE. See - --with-threads; this is currently disabled by default. - ---enable-progress-threads - Allows asynchronous progress in some transports. See - --with-threads; this is currently disabled by default. See the - above note about asynchronous progress. - ---disable-mpi-cxx - Disable building the C++ MPI bindings. Note that this does *not* - disable the C++ checks during configure; some of Open MPI's tools - are written in C++ and therefore require a C++ compiler to be built. - ---disable-mpi-cxx-seek - Disable the MPI::SEEK_* constants. Due to a problem with the MPI-2 - specification, these constants can conflict with system-level SEEK_* - constants. Open MPI attempts to work around this problem, but the - workaround may fail in some esoteric situations. The - --disable-mpi-cxx-seek switch disables Open MPI's workarounds (and - therefore the MPI::SEEK_* constants will be unavailable). - ---disable-mpi-f77 - Disable building the Fortran 77 MPI bindings. - ---disable-mpi-f90 - Disable building the Fortran 90 MPI bindings. Also related to the - --with-f90-max-array-dim and --with-mpi-f90-size options. - ---with-mpi-f90-size= - Three sizes of the MPI F90 module can be built: trivial (only a - handful of MPI-2 F90-specific functions are included in the F90 - module), small (trivial + all MPI functions that take no choice - buffers), and medium (small + all MPI functions that take 1 choice - buffer). This parameter is only used if the F90 bindings are - enabled. - ---with-f90-max-array-dim= - The F90 MPI bindings are strictly typed, even including the number of - dimensions for arrays for MPI choice buffer parameters. Open MPI - generates these bindings at compile time with a maximum number of - dimensions as specified by this parameter. The default value is 4. - ---enable-mpirun-prefix-by-default - This option forces the "mpirun" command to always behave as if - "--prefix $prefix" was present on the command line (where $prefix is - the value given to the --prefix option to configure). This prevents - most rsh/ssh-based users from needing to modify their shell startup - files to set the PATH and/or LD_LIBRARY_PATH for Open MPI on remote - nodes. Note, however, that such users may still desire to set PATH - -- perhaps even in their shell startup files -- so that executables - such as mpicc and mpirun can be found without needing to type long - path names. --enable-orterun-prefix-by-default is a synonym for - this option. - ---disable-shared - By default, libmpi is built as a shared library, and all components - are built as dynamic shared objects (DSOs). This switch disables - this default; it is really only useful when used with - --enable-static. Specifically, this option does *not* imply - --enable-static; enabling static libraries and disabling shared - libraries are two independent options. - ---enable-static - Build libmpi as a static library, and statically link in all - components. Note that this option does *not* imply - --disable-shared; enabling static libraries and disabling shared - libraries are two independent options. - ---enable-sparse-groups - Enable the usage of sparse groups. This would save memory - significantly especially if you are creating large - communicators. (Disabled by default) - ---enable-peruse - Enable the PERUSE MPI data analysis interface. - ---enable-dlopen - Build all of Open MPI's components as standalone Dynamic Shared - Objects (DSO's) that are loaded at run-time. The opposite of this - option, --disable-dlopen, causes two things: - - 1. All of Open MPI's components will be built as part of Open MPI's - normal libraries (e.g., libmpi). - 2. Open MPI will not attempt to open any DSO's at run-time. - - Note that this option does *not* imply that OMPI's libraries will be - built as static objects (e.g., libmpi.a). It only specifies the - location of OMPI's components: standalone DSOs or folded into the - Open MPI libraries. You can control whenther Open MPI's libraries - are build as static or dynamic via --enable|disable-static and - --enable|disable-shared. - ---enable-heterogeneous - Enable support for running on heterogeneous clusters (e.g., machines - with different endian representations). Heterogeneous support is - disabled by default because it imposes a minor performance penalty. - ---enable-ptmalloc2-internal - ***NOTE: This option no longer exists. - - This option was introduced in Open MPI v1.3 and was then removed in - Open MPI v1.3.2. Open MPI fundamentally changed how it uses - ptmalloc2 support in v1.3.2 such that the - --enable-ptmalloc2-internal flag was no longer necessary. It can - still harmlessly be supplied to Open MPI's configure script, but a - warning will appear about how it is an unrecognized option. - - In v1.3 and v1.3.1, Open MPI built the ptmalloc2 library as a - standalone library that users could choose to link in or not (by - adding -lopenmpi-malloc to their link command). Using this option - restored pre-v1.3 behavior of *always* forcing the user to use the - ptmalloc2 memory manager (because it is part of libmpi). - - Starting with v1.3.2, ptmalloc2 is always built into Open MPI, but - is only activated in certain scenarios. - ---with-wrapper-cflags= ---with-wrapper-cxxflags= ---with-wrapper-fflags= ---with-wrapper-fcflags= ---with-wrapper-ldflags= ---with-wrapper-libs= - Add the specified flags to the default flags that used are in Open - MPI's "wrapper" compilers (e.g., mpicc -- see below for more - information about Open MPI's wrapper compilers). By default, Open - MPI's wrapper compilers use the same compilers used to build Open - MPI and specify an absolute minimum set of additional flags that are - necessary to compile/link MPI applications. These configure options - give system administrators the ability to embed additional flags in - OMPI's wrapper compilers (which is a local policy decision). The - meanings of the different flags are: - - : Flags passed by the mpicc wrapper to the C compiler - : Flags passed by the mpic++ wrapper to the C++ compiler - : Flags passed by the mpif77 wrapper to the F77 compiler - : Flags passed by the mpif90 wrapper to the F90 compiler - : Flags passed by all the wrappers to the linker - : Flags passed by all the wrappers to the linker - - There are other ways to configure Open MPI's wrapper compiler - behavior; see the Open MPI FAQ for more information. - -There are many other options available -- see "./configure --help". - -Changing the compilers that Open MPI uses to build itself uses the -standard Autoconf mechanism of setting special environment variables -either before invoking configure or on the configure command line. -The following environment variables are recognized by configure: - -CC - C compiler to use -CFLAGS - Compile flags to pass to the C compiler -CPPFLAGS - Preprocessor flags to pass to the C compiler - -CXX - C++ compiler to use -CXXFLAGS - Compile flags to pass to the C++ compiler -CXXCPPFLAGS - Preprocessor flags to pass to the C++ compiler - -F77 - Fortran 77 compiler to use -FFLAGS - Compile flags to pass to the Fortran 77 compiler - -FC - Fortran 90 compiler to use -FCFLAGS - Compile flags to pass to the Fortran 90 compiler - -LDFLAGS - Linker flags to pass to all compilers -LIBS - Libraries to pass to all compilers (it is rarely - necessary for users to need to specify additional LIBS) - -For example: - -shell$ ./configure CC=mycc CXX=myc++ F77=myf77 F90=myf90 ... - -***Note: We generally suggest using the above command line form for - setting different compilers (vs. setting environment variables and - then invoking "./configure"). The above form will save all - variables and values in the config.log file, which makes - post-mortem analysis easier when problems occur. - -Note that you may also want to ensure that the value of -LD_LIBRARY_PATH is set appropriately (or not at all) for your build -(or whatever environment variable is relevant for your operating -system). For example, some users have been tripped up by setting to -use non-default Fortran compilers via FC / F77, but then failing to -set LD_LIBRARY_PATH to include the directory containing that -non-default Fortran compiler's support libraries. This causes Open -MPI's configure script to fail when it tries to compile / link / run -simple Fortran programs. - -It is required that the compilers specified be compile and link -compatible, meaning that object files created by one compiler must be -able to be linked with object files from the other compilers and -produce correctly functioning executables. - -Open MPI supports all the "make" targets that are provided by GNU -Automake, such as: - -all - build the entire Open MPI package -install - install Open MPI -uninstall - remove all traces of Open MPI from the $prefix -clean - clean out the build tree - -Once Open MPI has been built and installed, it is safe to run "make -clean" and/or remove the entire build tree. - -VPATH and parallel builds are fully supported. - -Generally speaking, the only thing that users need to do to use Open -MPI is ensure that /bin is in their PATH and /lib is -in their LD_LIBRARY_PATH. Users may need to ensure to set the PATH -and LD_LIBRARY_PATH in their shell setup files (e.g., .bashrc, .cshrc) -so that non-interactive rsh/ssh-based logins will be able to find the -Open MPI executables. - -=========================================================================== - -Checking Your Open MPI Installation ------------------------------------ - -The "ompi_info" command can be used to check the status of your Open -MPI installation (located in /bin/ompi_info). Running it with -no arguments provides a summary of information about your Open MPI -installation. - -Note that the ompi_info command is extremely helpful in determining -which components are installed as well as listing all the run-time -settable parameters that are available in each component (as well as -their default values). - -The following options may be helpful: - ---all Show a *lot* of information about your Open MPI - installation. ---parsable Display all the information in an easily - grep/cut/awk/sed-able format. ---param - A of "all" and a of "all" will - show all parameters to all components. Otherwise, the - parameters of all the components in a specific framework, - or just the parameters of a specific component can be - displayed by using an appropriate and/or - name. - -Changing the values of these parameters is explained in the "The -Modular Component Architecture (MCA)" section, below. - -=========================================================================== - -Compiling Open MPI Applications -------------------------------- - -Open MPI provides "wrapper" compilers that should be used for -compiling MPI applications: - -C: mpicc -C++: mpiCC (or mpic++ if your filesystem is case-insensitive) -Fortran 77: mpif77 -Fortran 90: mpif90 - -For example: - -shell$ mpicc hello_world_mpi.c -o hello_world_mpi -g -shell$ - -All the wrapper compilers do is add a variety of compiler and linker -flags to the command line and then invoke a back-end compiler. To be -specific: the wrapper compilers do not parse source code at all; they -are solely command-line manipulators, and have nothing to do with the -actual compilation or linking of programs. The end result is an MPI -executable that is properly linked to all the relevant libraries. - -Customizing the behavior of the wrapper compilers is possible (e.g., -changing the compiler [not recommended] or specifying additional -compiler/linker flags); see the Open MPI FAQ for more information. - -=========================================================================== - -Running Open MPI Applications ------------------------------ - -Open MPI supports both mpirun and mpiexec (they are exactly -equivalent). For example: - -shell$ mpirun -np 2 hello_world_mpi -or -shell$ mpiexec -np 1 hello_world_mpi : -np 1 hello_world_mpi - -are equivalent. Some of mpiexec's switches (such as -host and -arch) -are not yet functional, although they will not error if you try to use -them. - -The rsh launcher accepts a -hostfile parameter (the option -"-machinefile" is equivalent); you can specify a -hostfile parameter -indicating an standard mpirun-style hostfile (one hostname per line): - -shell$ mpirun -hostfile my_hostfile -np 2 hello_world_mpi - -If you intend to run more than one process on a node, the hostfile can -use the "slots" attribute. If "slots" is not specified, a count of 1 -is assumed. For example, using the following hostfile: - ---------------------------------------------------------------------------- -node1.example.com -node2.example.com -node3.example.com slots=2 -node4.example.com slots=4 ---------------------------------------------------------------------------- - -shell$ mpirun -hostfile my_hostfile -np 8 hello_world_mpi - -will launch MPI_COMM_WORLD rank 0 on node1, rank 1 on node2, ranks 2 -and 3 on node3, and ranks 4 through 7 on node4. - -Other starters, such as the resource manager / batch scheduling -environments, do not require hostfiles (and will ignore the hostfile -if it is supplied). They will also launch as many processes as slots -have been allocated by the scheduler if no "-np" argument has been -provided. For example, running a SLURM job with 8 processors: - -shell$ salloc -n 8 mpirun a.out - -The above command will reserve 8 processors and run 1 copy of mpirun, -which will, in turn, launch 8 copies of a.out in a single -MPI_COMM_WORLD on the processors that were allocated by SLURM. - -Note that the values of component parameters can be changed on the -mpirun / mpiexec command line. This is explained in the section -below, "The Modular Component Architecture (MCA)". - -=========================================================================== - -The Modular Component Architecture (MCA) - -The MCA is the backbone of Open MPI -- most services and functionality -are implemented through MCA components. Here is a list of all the -component frameworks in Open MPI: - ---------------------------------------------------------------------------- - -MPI component frameworks: -------------------------- - -allocator - Memory allocator -bml - BTL management layer -btl - MPI point-to-point Byte Transfer Layer, used for MPI - point-to-point messages on some types of networks -coll - MPI collective algorithms -crcp - Checkpoint/restart coordination protocol -dpm - MPI-2 dynamic process management -io - MPI-2 I/O -mpool - Memory pooling -mtl - Matching transport layer, used for MPI point-to-point - messages on some types of networks -osc - MPI-2 one-sided communications -pml - MPI point-to-point management layer -pubsub - MPI-2 publish/subscribe management -rcache - Memory registration cache -topo - MPI topology routines - -Back-end run-time environment component frameworks: ---------------------------------------------------- - -errmgr - RTE error manager -ess - RTE environment-specfic services -filem - Remote file management -grpcomm - RTE group communications -iof - I/O forwarding -notifier - System/network administrator noficiation system -odls - OpenRTE daemon local launch subsystem -oob - Out of band messaging -plm - Process lifecycle management -ras - Resource allocation system -rmaps - Resource mapping system -rml - RTE message layer -routed - Routing table for the RML -snapc - Snapshot coordination - -Miscellaneous frameworks: -------------------------- - -backtrace - Debugging call stack backtrace support -carto - Cartography (host/network mapping) support -crs - Checkpoint and restart service -installdirs - Installation directory relocation services -maffinity - Memory affinity -memchecker - Run-time memory checking -memcpy - Memopy copy support -memory - Memory management hooks -paffinity - Processor affinity -timer - High-resolution timers - ---------------------------------------------------------------------------- - -Each framework typically has one or more components that are used at -run-time. For example, the btl framework is used by the MPI layer to -send bytes across different types underlying networks. The tcp btl, -for example, sends messages across TCP-based networks; the openib btl -sends messages across OpenFabrics-based networks; the MX btl sends -messages across Myrinet networks. - -Each component typically has some tunable parameters that can be -changed at run-time. Use the ompi_info command to check a component -to see what its tunable parameters are. For example: - -shell$ ompi_info --param btl tcp - -shows all the parameters (and default values) for the tcp btl -component. - -These values can be overridden at run-time in several ways. At -run-time, the following locations are examined (in order) for new -values of parameters: - -1. /etc/openmpi-mca-params.conf - - This file is intended to set any system-wide default MCA parameter - values -- it will apply, by default, to all users who use this Open - MPI installation. The default file that is installed contains many - comments explaining its format. - -2. $HOME/.openmpi/mca-params.conf - - If this file exists, it should be in the same format as - /etc/openmpi-mca-params.conf. It is intended to provide - per-user default parameter values. - -3. environment variables of the form OMPI_MCA_ set equal to a - - - Where is the name of the parameter. For example, set the - variable named OMPI_MCA_btl_tcp_frag_size to the value 65536 - (Bourne-style shells): - - shell$ OMPI_MCA_btl_tcp_frag_size=65536 - shell$ export OMPI_MCA_btl_tcp_frag_size - -4. the mpirun command line: --mca - - Where is the name of the parameter. For example: - - shell$ mpirun --mca btl_tcp_frag_size 65536 -np 2 hello_world_mpi - -These locations are checked in order. For example, a parameter value -passed on the mpirun command line will override an environment -variable; an environment variable will override the system-wide -defaults. - -=========================================================================== - -Common Questions ----------------- - -Many common questions about building and using Open MPI are answered -on the FAQ: - - http://www.open-mpi.org/faq/ - -=========================================================================== - -Got more questions? -------------------- - -Found a bug? Got a question? Want to make a suggestion? Want to -contribute to Open MPI? Please let us know! - -When submitting questions and problems, be sure to include as much -extra information as possible. This web page details all the -information that we request in order to provide assistance: - - http://www.open-mpi.org/community/help/ - -User-level questions and comments should generally be sent to the -user's mailing list (users@open-mpi.org). Because of spam, only -subscribers are allowed to post to this list (ensure that you -subscribe with and post from *exactly* the same e-mail address -- -joe@example.com is considered different than -joe@mycomputer.example.com!). Visit this page to subscribe to the -user's list: - - http://www.open-mpi.org/mailman/listinfo.cgi/users - -Developer-level bug reports, questions, and comments should generally -be sent to the developer's mailing list (devel@open-mpi.org). Please -do not post the same question to both lists. As with the user's list, -only subscribers are allowed to post to the developer's list. Visit -the following web page to subscribe: - - http://www.open-mpi.org/mailman/listinfo.cgi/devel - -Make today an Open MPI day! diff --git a/release_notes/sdp_release_notes.txt b/release_notes/sdp_release_notes.txt deleted file mode 100644 index 8fcd5ff..0000000 --- a/release_notes/sdp_release_notes.txt +++ /dev/null @@ -1,248 +0,0 @@ - Open Fabrics Enterprise Distribution (OFED) - SDP in OFED 1.5.4 Release Notes - - December 2011 - - - -=============================================================================== -Table of Contents -=============================================================================== -1. Overview -2. Bug Fixes and Enhancements since OFED 1.5.3 -3. ZCopy -4. Known Issues -5. Verification Applications/Flows/Tests -6. Module Parameters - -=============================================================================== -1. Overview -=============================================================================== -Sockets Direct Protocol (SDP) is an InfiniBand byte-stream transport protocol -that provides TCP stream semantics. Capable of utilizing InfiniBand's advanced -protocol offload capabilities, SDP can provide lower latency, higher bandwidth, -and lower CPU utilization than IPoIB or -Ethernet running some sockets-based applications. - -=============================================================================== -2. Main Features and Changes -=============================================================================== -- Added IPv6 support -- Added support for Inline and blueflame -- Improved stability issues -- Bug fixes - -=============================================================================== -2. Bug Fixes and Enhancements since OFED 1.5.3 -=============================================================================== -* Cleanups - - Added support for 2.6.34 / 2.6.36. - -* Bug Fixes - - Fixed compilation problems on 32 bit hosts - - Do not compile in debug mode when not asked. - - Improved recovery from errors. - -* Enhancements - - more statistics in /proc/sdpstats - - added debugfs for sdp: - - sdpprf was moved from /proc to debugfs/sdp - - debugfs/ - Socket history - - -=============================================================================== -3. ZCopy -=============================================================================== -- ZCopy is enabled by default for blocks larger than 64K. ZCopy can be disabled - by setting the module paramter sdp_zcopy_thresh to zero or to any other value - by setting it to another non zero value. - -- ZCOPY mode gives good performance for large blocks with very small cpu - utilization. When in use, all messages longer than 'sdp_zcopy_thresh' bytes - in length will cause the user space buffer to be pinned and the data sent - directly from the original buffer. This results in less CPU usage and on many - systems in enhanced bandwidth. - ZCOPY is most efficient with multi stream jobs and it performs better as the - message size increases. - The default 64K value for 'sdp_zcopy_thresh' is sometimes too low for some - systems. You must experiment with your hardware to select the best value. - -- ZCOPY vs BCOPY: - ZCOPY performance is more efficient in weak cpu and multi streams, whereas - BCOPY is more efficient in single stream. - -=============================================================================== -4. Known Issues -=============================================================================== -- SDP is at beta level on Infinihost HCA family - -- Occasionally, socket bind fails when using EINVAL. Although TCP socket is binded - successfully, SDP is occupied, thus causing the socket bind failure. - See Bugzilla 2159 and Bugzilla 2160 - -- When SO_REUSEADDR is set, only a single socket can be bind to the IP_ANY and a - specific port. TCP limitation, unless one of the sockets is listening. - -- BUG 1331 - Although TCP allows connecting to IP_ANY - 0.0.0.0 - (as a destination address!), SDP does not allow connecting to the IP_ANY - and rejects the connection. - -- BUG 1444 - The setsockopt(SO_RCVBUF) is not functional in sdp socket. - To limit top system wide sdp memory usage for recv, - use the module parameter top_mem_usage. - -- Each SDP socket currently consumes up to 2 MBytes of memory. If this value - is high for your installation, it is possible to trade off performance - for lower memory utilization per socket by reducing the value of the - "rcvbuf_scale" module parameter (default: 16). - - Note: The minimum legal value for the "rcvbuf_scale" module is 1. - At this parameter value, each socket will consume approximately 128 KBytes. - -- Small message size performance is low when messages are sent by client - at a rate lower than the rate at which they are consumed by server, - and when TCP_CORK is not set. This is observed, for example, with iperf - benchmark. - Workaround: Set the TCP_CORK socket option - to ensure data is sent in at least 32K byte chunks. - -- Performance is low on 32-bit kernels, as SDP utilizes high memory - to ease memory pressure. - Workaround: Move to a 64-bit kernel if the application remains a 32-bit one. - -- By default, SDP utilizes a 2 Kbyte MTU size. This may cause PCI-X cards - using Mellanox Technologies "Infinihost" HCAs to experience low bandwidth. - Workaround: Reset the MTU size to 1K in this situation, using either of - the two methods below: - - 1. Activate the "tavor quirk" workaround in opensm: - a. Create an opensm options cache file (/var/cache/osm/opensm.opts): - > opensm --cache-options -o - b. Add the following line to /var/cache/osm/opensm.opts: - enable_quirks TRUE - c. Rerun opensm using your usual command line options to activate - the opensm quirk option. - - 2. Activate the "tavor quirk" workaround in cma: - set the tavor_quirk module parameter of the rdma_cm module to value 1 - (default: 0). - -- When waiting for RX, the driver first polls, arms interrupt and then goes to - sleep. Polling duration could be set by recv_poll module parameter. The - higher this value is, the higher the CPU utilization is, and the number of - interrupts is lower. - This should be fine tuned according to the specific environment and - application latency. - -- When using SDP over RoCE, and the peer has a card that does not support RoCE - a delay in the connection establishment may occur. - -- BUG2185 - Occasionally, accessing /proc/net/sdpstats, causes kernel - panic. - -- When using SDP with an application that has one of set-user-ID or - set-group-ID bits on, these parameters MUST be added to the libsdp.so as well. - for further information, please see the man ld.so file. - -=============================================================================== -5. Verification Applications/Flows/Tests -=============================================================================== -- ssh/sshd -- wget/netscape/firefox/apache -- netpipe -- netperf -- LTP socket tests -- iperf-2.0.2 -- ttcp -- openmpi -- openmpi + Intel MPI benchmarks -- Threaded and forking echo client server examples -- Various Java client server applications (SUN:jre, BEA:jrockit/WebLogic, GNU:gij/gcj) -- Many UNIX utilities to verify that pre-load did not harm the applications - -=============================================================================== -6. Module Parameters -=============================================================================== - -General -------- -sdp_link_layer_ib_only: - Supports only link layer of type InfiniBand. - It is useful when not using SDP over RoCE. - -sdp_debug_level: - Enables connection establishment and teardown debug tracing. - -sdp_data_debug_level: - Enables datapath debug tracing. If set to 1, it shows only packets >1. - To enable debugging of data path, compile driver with CONFIG_SDP_DEBUG_DATA. - - -recv_poll: - Enables poll receiving before arming the interrupt. Set a higher value - to decrease the number of RX interrupts. Consequently, the CPU - utilization will be higher. - -sdp_keepalive_time: - Default idle time in seconds before keepalive probe sent. - -Resources ---------- -rcvbuf_initial_size: - Receives buffer initial size in bytes. - -rcvbuf_scale: - Not in use - -top_mem_usage: - Top system wide sdp memory usage for recv (in MB). - -max_large_sockets: - Not in use - -sdp_fmr_pool_size: - Number of FMRs to allocate for pool - -sdp_fmr_dirty_wm: - Watermark to flush fmr pool - -Thresholds ----------- -sdp_inline_thresh: - Inline copy threshold. effective to new sockets only; 0=Off. - -sdp_zcopy_thresh: - Zero copy using RDMA threshold; 0=Off. - If smaller than page size, set to page size. - -Interrupt hardware moderation: ------------------------------- -sdp_rx_coal_target: - Target number of bytes to coalesce with interrupt moderation. - -sdp_rx_coal_time: - rx coal time (jiffies). - -sdp_rx_rate_low: - rx_rate low (packets/sec). - -sdp_rx_coal_time_low: - low moderation usec. - -sdp_rx_rate_high: - rx_rate high (packets/sec). - -sdp_rx_coal_time_high: - high moderation usec. - -sdp_rx_rate_thresh: - rx rate thresh (). - -sdp_sample_interval: - sample interval (jiffies). - -hw_int_mod_count: - Forced hw int moderation val. -1 for auto (packets). 0 to disable. - -hw_int_mod_usec: - Forced hw int moderation val. -1 for auto (usec). 0 to disable. -- 2.46.0