Compaq_Parallel_Software_Environment________________ Release Notes: PVM and MPI July 1999 This document contains information about the Compaq Parallel Virtual Machine (PVM) and Message Passing Interface (MPI) software. Revision/Update Information: This version supersedes the manual issued in January 1999 for DIGITAL PSE Version 1.8. Operating System Versions: PVM: Compaq Tru64 UNIX (formerly DIGITAL UNIX) V4.0a and higher MPI: Compaq Tru64 UNIX (formerly DIGITAL UNIX) V4.0a and higher Software Versions: Compaq Parallel Software Environment Version 1.9 Compaq PVM Version 1.9 Compaq MPI Version 1.9 Compaq Computer Corporation Houston, Texas ________________________________________________________________ Revision July 1999 Compaq Computer Corporation makes no representations that the use of its products in the manner described in this publication will not infringe on existing or future patent rights, nor do the descriptions contained in this publication imply the granting of licenses to make, use, or sell equipment or software in accordance with the description. Possession, use, or copying of the software described in this publication is authorized only pursuant to a valid written license from Compaq or an authorized sublicensor. While Compaq believes that the information included in this publication is correct as of the date of publication, it is subject to change without notice. © Digital Equipment Corporation 1999. All Rights Reserved. Unpublished rights reserved under the copyright laws of the United States. Compaq Tru64 UNIX (formerly DIGITAL UNIX) documentation is proprietary to and embodies the confidential technology of Compaq Computer Corporation. Possession or use of the Tru64 UNIX documentation is hereby authorized. Duplication or dissemination of the Tru64 UNIX documentation is authorized only pursuant to a written license from Compaq Computer Corporation. Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in Subparagraph (c)(1)(ii) of DFARS 252.227-7013 or in FAR 52.227-19, as applicable. COMPAQ, the Compaq logo, the DIGITAL logo, AlphaServer, DEC, DIGITAL, and TruCluster are Registered in the United States Patent and Trademark Office. Tru64 is a trademark of Compaq Computer Corporation. UNIX is a registered trademark of The Open Group in the United States and other countries. All other trademarks and registered trademarks are the property of their respective holders. This document is available on CD-ROM. This document was prepared using VAX DOCUMENT Version 2.1. _________________________________________________________________ Contents 1 PVM Release Notes 1.1 Future PVM Development........................ 1-1 1.2 Prerequisite Software......................... 1-1 1.3 Supported Platforms........................... 1-1 1.4 Installation.................................. 1-1 1.4.1 PVM Software Distributed with PSE......... 1-1 1.4.2 Patch for MEMORY CHANNEL[TM] Software Version 1.4............................... 1-2 1.5 Applications Must be Re-Linked for Version 1.9........................................... 1-2 1.6 Documentation (PostScript and HTML)........... 1-2 1.7 New in Version 1.9............................ 1-2 1.8 New in Version 1.6............................ 1-2 1.9 Known Problems................................ 1-2 1.9.1 exec Routines............................. 1-3 1.9.2 Exhausting Virtual Memory Resources....... 1-3 1.10 Miscellaneous................................. 1-4 1.10.1 pvm_exit Blocks to Wait for an Available Receiver.................................. 1-4 1.10.2 -pthread Compile-Line Argument Needed..... 1-4 1.10.3 PVM Environment Variable Defaults......... 1-4 1.11 Problems, Suggestions or Comments............. 1-4 2 MPI Release Notes 2.1 Prerequisite Software......................... 2-1 2.2 Supported Platforms........................... 2-1 2.3 Compatibility................................. 2-1 2.4 Applications Must be Re-compiled and Re-linked..................................... 2-1 2.5 Installation.................................. 2-2 2.5.1 MPI Software Distributed with PSE......... 2-2 iii 2.5.2 Patch for MEMORY CHANNEL[TM] Software Version 1.4............................... 2-2 2.6 Documentation (PostScript and HTML)........... 2-2 2.7 New in Version 1.9............................ 2-2 2.8 New in Version 1.8............................ 2-3 2.9 Known Problems................................ 2-4 2.9.1 stdout and stderr Streams are Combined.... 2-4 2.9.2 Running Under a Debugger Leaves Files in /tmp...................................... 2-4 2.9.3 A Standard Variance....................... 2-4 2.9.4 MPI_REQUEST_FREE.......................... 2-4 2.9.5 MPI_CANCEL................................ 2-4 2.9.6 exec Routines............................. 2-4 2.9.7 Exhausting Virtual Memory Resources....... 2-5 2.9.8 Error Message: "No MEMORY CHANNEL installed"................................ 2-5 2.10 Miscellaneous................................. 2-5 2.10.1 User-Level Access for MEMORY CHANNEL[TM]............................... 2-5 2.10.2 Shared Memory Segment Limit............... 2-6 2.10.3 -ump_bufs Requires a Multiple of 32....... 2-6 2.11 Problems, Suggestions or Comments............. 2-6 3 Comments, Problems, and Help 3.1 Sending Compaq Your Comments on This Product....................................... 3-1 3.2 Getting Help from Compaq...................... 3-1 3.3 Readers Comments Form-Documentation........... 3-2 Tables 1-1 Default Values for PVM Environment Variables................................. 1-4 iv 1 _________________________________________________________________ PVM Release Notes 1.1 Future PVM Development Consideration is being given to stopping further development of Compaq PVM and eventually retiring the product. Please send a note describing your PVM needs and your opinions of this proposal to pvm@ilo.dec.com. 1.2 Prerequisite Software Compaq PVM Version 1.9 requires Compaq Tru64 UNIX Version 4.0a or higher. In addition, if you need MEMORY CHANNEL[TM] support, all cluster members require Compaq TruCluster software (or TruCluster MEMORY CHANNEL[TM] software), version 1.5 (requires Tru64 UNIX V4.0d or higher) or higher. You can also use version 1.4 of the TruCluster or MEMORY CHANNEL software if a patch is installed (see Section 1.4.2). 1.3 Supported Platforms Compaq PVM is a Compaq proprietary implementation of PVM, for Alpha systems running Tru64 UNIX. Both stand-alone SMP systems and MEMORY CHANNEL[TM] clusters are supported. 1.4 Installation 1.4.1 PVM Software Distributed with PSE For convenience, Compaq PVM software is distributed as part of the Compaq Parallel Software Environment (PSE). For installation instructions, refer to the PSE installation guide. PVM Release Notes 1-1 PVM Release Notes 1.4 Installation 1.4.2 Patch for MEMORY CHANNEL[TM] Software Version 1.4 Any cluster members running version 1.4 of Compaq TruCluster software (or TruCluster MEMORY CHANNEL[TM] software) must have Patch ID TCR141-013 or its successor installed. To ensure that you are installing the latest patch, please send mail to pvm@ilo.dec.com. To obtain patches, please use your regular Compaq support channel. No patch is needed for systems running version 1.5 or higher of Compaq TruCluster software (or TruCluster MEMORY CHANNEL[TM] software). If you have any questions or concerns, please send mail to pvm@ilo.dec.com. 1.5 Applications Must be Re-Linked for Version 1.9 Programs previously linked with any previous PVM archive library, will have to be re-linked to operate within the PVM190 environment. 1.6 Documentation (PostScript and HTML) The Compaq PVM User Guide can be found in PostScript format at /usr/opt/PVM190/pvm_guide.ps. It can also be found in HTML format on the Consolidated Layered Products Documentation CD-ROM. 1.7 New in Version 1.9 Compaq PVM Version 1.9 is a bug fix release and contains no new features. 1.8 New in Version 1.6 Compaq PVM Version 1.6 now supports multiple rails of MEMORY CHANNEL[TM] for message passing between systems. 1.9 Known Problems The following problems with Compaq PVM were known at the time of release: 1-2 PVM Release Notes PVM Release Notes 1.9 Known Problems 1.9.1 exec Routines A process that has called a PVM routine may fail if it calls execl, execv, execle, execve, execlp, or execvp. The exec routine returns EWOULDBLOCK. The problem can be avoided if the process calls fork, and calls the exec routine in the child process. 1.9.2 Exhausting Virtual Memory Resources When a process that has called a PVM routine forks, the child process sometimes loses some virtual memory resources. If an application uses multiple generations of processes (parent makes PVM call then forks child, which makes a PVM call then forks, and so on), the application may run out of vm-mapentries. The number of vm-mapentries available to an application can be changed using the sysconfig(8) command. PVM Release Notes 1-3 PVM Release Notes 1.10 Miscellaneous 1.10 Miscellaneous 1.10.1 pvm_exit Blocks to Wait for an Available Receiver A PVM task can send messages before a receiver task is available to read those messages. Such messages are queued in the sender task. If the sender task calls pvm_exit before a receiver task is available to read a message then the pvm_exit will block until a receiver task becomes available. 1.10.2 -pthread Compile-Line Argument Needed The -pthread argument must be included on the compile line. 1.10.3 PVM Environment Variable Defaults The default values of the Compaq PVM environment variables are described in Table 1-1. Table_1-1_Default_Values_for_PVM_Environment_Variables_____ Environment_Variable__Default_Value________________________ PVM_BUF_SIZE 1,048,576 PVM_MC_CHAN_SIZE 32,768 PVM_SM_CHAN_SIZE______32,768_______________________________ ________________________ Note ________________________ All the "size" parameters should be multiples of 1024. ______________________________________________________ 1.11 Problems, Suggestions or Comments Any problems, suggestions or comments should be addressed to pvm@ilo.dec.com. 1-4 PVM Release Notes 2 _________________________________________________________________ MPI Release Notes 2.1 Prerequisite Software Compaq MPI Version 1.9 requires Tru64 UNIX Version 4.0a or higher. The Fortran runtime libraries are also required. In addition, if you need MEMORY CHANNEL[TM] support, all cluster members require Compaq TruCluster software (or TruCluster MEMORY CHANNEL[TM] software), version 1.5 (requires Tru64 UNIX V4.0d or higher) or higher. You can also use version 1.4 of the TruCluster or MEMORY CHANNEL software if a patch is installed (see Section 2.5.2). 2.2 Supported Platforms Compaq MPI is a Compaq proprietary implementation of MPI, for Alpha systems running Tru64 UNIX. Both stand-alone SMP systems and MEMORY CHANNEL[TM] clusters are supported. 2.3 Compatibility Compaq MPI is compatible with version 1.1.1 of MPICH from Argonne National Labs, and will run over shared memory and MEMORY CHANNEL[TM]. Applications built using Compaq MPI cannot communicate via MPI with applications built using MPICH. 2.4 Applications Must be Re-compiled and Re-linked Compaq MPI version 1.9 requires applications to be both recompiled and re-linked. Simply re-linking applications is not sufficient. MPI Release Notes 2-1 MPI Release Notes 2.5 Installation 2.5 Installation 2.5.1 MPI Software Distributed with PSE For convenience, Compaq MPI software is distributed as part of the Compaq Parallel Software Environment (PSE). For installation instructions, refer to the PSE installation guide. 2.5.2 Patch for MEMORY CHANNEL[TM] Software Version 1.4 Any cluster members running version 1.4 of Compaq TruCluster software (or TruCluster MEMORY CHANNEL[TM] software) must have Patch ID TCR141-013 or its successor installed. To ensure that you are installing the latest patch, please send mail to mpi@ilo.dec.com. To obtain patches, please use your regular Compaq support channel. No patch is needed for systems running version 1.5 or higher of Compaq TruCluster software (or TruCluster MEMORY CHANNEL[TM] software). If you have any questions or concerns, please send mail to mpi@ilo.dec.com. 2.6 Documentation (PostScript and HTML) The Compaq MPI User Guide can be found in PostScript format at /usr/opt/MPI190/mpi_guide.ps. It can also be found in HTML format on the Consolidated Layered Products Documentation CD-ROM. There is a sample MPI program and instructions in /usr /examples/mpi. Reference pages for all of the MPI routines are included in this distribution, and can be accessed with the man command. 2.7 New in Version 1.9 Compaq MPI Version 1.9 is a bug fix release and contains no new features. 2-2 MPI Release Notes MPI Release Notes 2.8 New in Version 1.8 2.8 New in Version 1.8 Included in Version 1.8 of Compaq MPI is support for V1.0.1 of the ROMIO MPI-2 package from Argonne National Laboratory. This version of ROMIO includes everything defined in the MPI-2 I/O chapter except shared file pointer functions (Section 9.4.4), split collective data access functions (Section 9.4.5), support for file interoperability (Section 9.5), I/O error handling (Section 9.7), and I/O error classes (Section 9.8). Since shared file pointer functions are not supported, the MPI_MODE_SEQUENTIAL amode to MPI_File_open is also not supported. The subarray and distributed array datatype constructor functions from MPI-2 Chapter 4 (Sections 4.14.4 and 4.14.5) have been implemented. They are useful for accessing arrays stored in files. The functions MPI_File_f2c and MPI_File_c2f (Section 4.12.4) also have been implemented. C, Fortran, and profiling interfaces are provided for all functions that have been implemented. Limitations of this version of ROMIO include the following: o The status argument is not filled in any function. Consequently, MPI_Get_count and MPI_Get_elements will not work when passed the status object from an MPI-I/O operation. o All nonblocking I/O functions use a ROMIO-defined MPIO_ Request object instead of the usual MPI_Request object. Accordingly, the two functions, MPIO_Test and MPIO_Wait, are provided to test and wait on these MPIO_Request objects. They have the same semantics as MPI_Test and MPI_Wait. int MPIO_Test(MPIO_Request *request, int *flag, MPI_Status *status); int MPIO_Wait(MPIO_Request *request, MPI_Status *status); The usual functions MPI_Test, MPI_Wait, MPI_Testany, and so forth, will not work for nonblocking I/O. o All functions return only two possible error codes - MPI_SUCCESS on success and MPI_ERR_UNKNOWN on failure. MPI Release Notes 2-3 MPI Release Notes 2.9 Known Problems 2.9 Known Problems These are the known restrictions in Compaq MPI Version 1.9: 2.9.1 stdout and stderr Streams are Combined In this release, the stdout and stderr streams from application processes are combined and are both delivered to stdout. There is no output on stderr. 2.9.2 Running Under a Debugger Leaves Files in /tmp When a debugger is used, dmpirun creates (and never deletes) a number of small files in /tmp. If you repeatedly debug large applications, you may need to delete these files to avoid filling /tmp. 2.9.3 A Standard Variance The Fortran binding for MPI_ADDRESS specifies that addresses be returned as type INTEGER. This specification is insufficient, because INTEGERs are 32 bit, whereas addresses are 64 bit on Alpha systems. In order to allow use of the MPI_ADDRESS call, this release uses INTEGER*8 as the return type. This does not affect the C binding, or its use in a C program. 2.9.4 MPI_REQUEST_FREE MPI_REQUEST_FREE does not work properly when the request is not completed (the mpich portable code has the same problem). 2.9.5 MPI_CANCEL MPI_CANCEL does not work (the MPICH portable code does not implement this). 2.9.6 exec Routines A process that has called MPI_INIT may fail if it calls execl, execv, execle, execve, execlp, or execvp. The exec routine returns EWOULDBLOCK. This problem can be avoided if the process calls fork, and calls the exec routine in the child process. 2-4 MPI Release Notes MPI Release Notes 2.9 Known Problems 2.9.7 Exhausting Virtual Memory Resources When a process that has called MPI_INIT forks, the child process sometimes loses some virtual memory resources. If an application uses multiple generations of processes (parent makes MPI calls then forks child, which makes MPI calls then forks, and so on), the application may run out of vm-mapentries. The number of vm-mapentries available to an application may be changed using the sysconfig(8) command. 2.9.8 Error Message: "No MEMORY CHANNEL installed" A call to MPI_INIT that fails with an error reporting "no MEMORY CHANNEL installed" usually indicates that the MEMORY CHANNEL[TM] patch has not been installed. For More Information: o See Section 2.5.2 2.10 Miscellaneous 2.10.1 User-Level Access for MEMORY CHANNEL[TM] If you are using MEMORY CHANNEL[TM], ensure that each machine in the cluster that you intend to use has been initialised for user-level access. You can check this by searching for a process called imc_mapper on each machine. For example, # ps a | grep imc_mapper | grep -v grep PID TTY S TIME CMD 657 ttyp2 U 0:00.01 /usr/sbin/imc_mapper If this process does not exist on any host that you intend to use as part of the cluster, you should execute the following command (as root) on each such host: # /usr/sbin/imc_init This only needs to be executed once, and has effect until the machine is next shut down. MPI Release Notes 2-5 MPI Release Notes 2.10 Miscellaneous 2.10.2 Shared Memory Segment Limit Compaq MPI uses shared memory for communication within a single host. The default system-wide maximum shared-memory segment a process can allocate is 4MB. For programs with a large number of processes this may need to be increased. This is done by editing the /etc/sysconfigtab file and adding (or modifying) the following entry: ipc: shm-max=size in bytes For this change to take effect, a reboot is necessary. 2.10.3 -ump_bufs Requires a Multiple of 32 If you specify a buffer size using the -ump_bufs option in the dmpirun command, then you must specify a buffer that is a multiple of 32. Note: you are wasting memory resources if the size you specify is not a multiple of 8192. 2.11 Problems, Suggestions or Comments Any problems, suggestions or comments should be addressed to mpi@ilo.dec.com. 2-6 MPI Release Notes 3 _________________________________________________________________ Comments, Problems, and Help This chapter gives you detailed instructions on how to submit comments on this product, report problems, and get help from the Customer Support Center (CSC). 3.1 Sending Compaq Your Comments on This Product Compaq welcomes your comments on this product and on its documentation. You can send comments to us in the following ways: o Internet electronic mail: - PVM issues: pvm@ilo.dec.com - MPI issues: mpi@ilo.dec.com o FAX: 603-884-0153 ATTN: PSE Team o A Reader's Comment Card sent to the address on the form. o A letter sent to the following address: Compaq Computer Corporation High Performance Computing Group 110 Spit Brook Road ZKO2-3/N30 Nashua, New Hampshire 03062 USA o Online questionnaire form. Print or edit the questionnaire form provided near the end of these release notes. Send the form by Internet mail, FAX, or the postal service. 3.2 Getting Help from Compaq If you have a customer support contract and have comments or questions about Compaq software, you should contact Compaq's Customer Support Center (CSC), preferably using electronic means such as DSNlink. In the United States, customers can call the CSC at (800) 354-9000. Comments, Problems, and Help 3-1 Comments, Problems, and Help 3.3 Readers Comments Form-Documentation 3.3 Readers Comments Form-Documentation Use the following form as a template for sending comments about PSE documentation. This form can be sent by Internet mail, FAX, or postal service. 3-2 Comments, Problems, and Help Comments, Problems, and Help 3.3 Readers Comments Form-Documentation --------------------------------------------------------------------- Please complete this survey and send an online version (via Internet) or a hardcopy version (via FAX or postal service) to: Internet mail: pse@hpc.pko.dec.com FAX: 603-884-0153 Postal Service: Compaq Computer Corporation High Performance Computing Group Documentation 110 Spit Brook Road ZKO2-3/N30 Nashua, New Hampshire 03062 USA Manual Title: ______________________________________________________ Order Number: ______________________________________________________ Version: ________________________________________________ _____________________________________________________________________ _____________________________________________________________________ We welcome any comments on this manual or on any PSE documentation. Your comments and suggestions help us improve the quality of our publications. 1. If you found any errors, please list them: Page Description ____ _______________________________________________________________ ____ _______________________________________________________________ ____ _______________________________________________________________ 2. How can we improve the content, usability, or otherwise improve our documentation set? _____________________________________________________________________ _____________________________________________________________________ _____________________________________________________________________ _____________________________________________________________________ Your Name/Title ____________________________________ Dept. ________ Company _____________________________________________ Date ________ Internet address or FAX number ______________________________________ Mailing address _____________________________________________________ Comments, Problems, and Help 3-3 Comments, Problems, and Help 3.3 Readers Comments Form-Documentation ___________________________________________ Phone _________________ --------------------------------------------------------------------- 3-4 Comments, Problems, and Help