DIGITAL_Parallel_Software_Environment_______________
Release Notes: PVM and MPI

January 1999

This document contains information about DIGITAL's
Parallel Virtual Machine (PVM) and Message Passing
Interface (MPI) software.

Revision/Update Information: January, 1999

Operating System Versions: PVM:
DIGITAL UNIX V4.0a
and higher
MPI:
DIGITAL UNIX V4.0a
and higher

Software Versions: DIGITAL Parallel
Software Environment
Version 1.8
DIGITAL PVM Version 1.8
DIGITAL MPI Version 1.8

Digital Equipment Corporation
Maynard, Massachusetts

________________________________________________________________
First Printed Edition, March, 1995
Revision, January 1999

Digital Equipment Corporation makes no representations
that the use of its products in the manner described in
this publication will not infringe on existing or future
patent rights, nor do the descriptions contained in this
publication imply the granting of licenses to make, use,
or sell equipment or software in accordance with the
description.

Possession, use, or copying of the software described
in this publication is authorized only pursuant to a
valid written license from Digital or an authorized
sublicensor.

DIGITAL believes the information in this publication is
accurate as of its publication date; such information
is subject to change without notice. DIGITAL is not
responsible for any inadvertent errors.

DIGITAL conducts its business in a manner that conserves
the environment and protects the safety and health of its
employees, customers, and the community.

The following are trademarks of Digital Equipment
Corporation: AdvantageCluster, Alpha AXP, AXP, Bookreader,
DEC, DEC Fortran, DECconnector, DECmcc, DECnet, DECserver,
DECstation, DECsystem, DECsupport, DECwindows, DELNI,
DEMPR, DIGITAL, GIGAswitch, POLYCENTER, ThinWire,
TURBOchannel, TruCluster, ULTRIX, VAX, VAX DOCUMENT,
VAX FORTRAN, VMS, and the DIGITAL logo.

Network File System and NFS are registered trademarks of
Sun Microsystems, Inc.

Motif is a registered trademark of Open Software
Foundation, Inc., licensed by DIGITAL; Open Software
Foundation, OSF and OSF/1 are registered trademarks of
Open Software Foundation, Inc.

PostScript is a registered trademark of Adobe Systems, Inc.

UNIX is a registered trademark in the United States and
other countries licensed exclusively through X/Open Company
Ltd.

X Window System is a trademark of Massachusetts Institute
of Technology.

All other trademarks and registered trademarks are the
property of their respective holders.

This document is available on CDROM.

This document was prepared using VAX DOCUMENT Version 2.1.

_________________________________________________________________

Contents

1 PVM Release Notes

1.1 Future PVM Development........................ 1-1
1.2 Prerequisite Software......................... 1-1
1.3 Supported Platforms........................... 1-1
1.4 Installation.................................. 1-1
1.4.1 PVM Software Distributed with PSE......... 1-1
1.4.2 Patch for MEMORY CHANNEL[TM] Software
Version 1.4............................... 1-2
1.5 Applications Must be Re-Linked for Version
1.8........................................... 1-2
1.6 Documentation (PostScript and HTML)........... 1-2
1.7 New in Version 1.8............................ 1-2
1.8 New in Version 1.6............................ 1-2
1.9 Known Problems................................ 1-2
1.9.1 exec Routines............................. 1-3
1.9.2 Exhausting Virtual Memory Resources....... 1-3
1.10 Miscellaneous................................. 1-4
1.10.1 pvm_exit Blocks to Wait for an Available
Receiver.................................. 1-4
1.10.2 -pthread Compile-Line Argument Needed..... 1-4
1.10.3 PVM Environment Variable Defaults......... 1-4
1.11 Problems, Suggestions or Comments............. 1-4

2 MPI Release Notes

2.1 Prerequisite Software......................... 2-1
2.2 Supported Platforms........................... 2-1
2.3 Compatibility................................. 2-1
2.4 Applications Must be Re-compiled and
Re-linked..................................... 2-1
2.5 Installation.................................. 2-2
2.5.1 MPI Software Distributed with PSE......... 2-2

iii

2.5.2 Patch for MEMORY CHANNEL[TM] Software
Version 1.4............................... 2-2
2.6 Documentation (PostScript and HTML)........... 2-2
2.7 New in Version 1.8............................ 2-2
2.8 Known Problems................................ 2-3
2.8.1 stdout and stderr Streams are Combined.... 2-4
2.8.2 Running Under a Debugger Leaves Files in
/tmp...................................... 2-4
2.8.3 A Standard Variance....................... 2-4
2.8.4 MPI_REQUEST_FREE.......................... 2-4
2.8.5 MPI_CANCEL................................ 2-4
2.8.6 exec Routines............................. 2-4
2.8.7 Exhausting Virtual Memory Resources....... 2-5
2.8.8 Error Message: "No MEMORY CHANNEL
installed"................................ 2-5
2.9 Miscellaneous................................. 2-5
2.9.1 User-Level Access for MEMORY
CHANNEL[TM]............................... 2-5
2.9.2 Shared Memory Segment Limit............... 2-6
2.9.3 -ump_bufs Requires a Multiple of 32....... 2-6
2.10 Problems, Suggestions or Comments............. 2-6

3 Comments, Problems, and Help

3.1 Sending DIGITAL Your Comments on This
Product....................................... 3-1
3.2 Getting Help from DIGITAL..................... 3-1
3.3 Readers Comments Form-Documentation........... 3-2

Tables

1-1 Default Values for PVM Environment
Variables................................. 1-4

1
_________________________________________________________________

PVM Release Notes

1.1 Future PVM Development

Consideration is being given to stopping further
development of DIGITAL PVM and eventually retiring the
product. Please send a note describing your PVM needs and
your opinions of this proposal to pvm@ilo.dec.com.

1.2 Prerequisite Software

DIGITAL PVM Version 1.8 requires DIGITAL UNIX Version 4.0a
or higher.

In addition, if you need MEMORY CHANNEL[TM] support,
all cluster members require DIGITAL TruCluster software
(or TruCluster MEMORY CHANNEL[TM] software), version 1.5
(requires DIGITAL UNIX V4.0d or higher) or higher. You can
also use version 1.4 of the TruCluster or MEMORY CHANNEL
software if a patch is installed (see Section 1.4.2).

1.3 Supported Platforms

DIGITAL PVM is a DIGITAL proprietary implementation of PVM,
for Alpha systems running DIGITAL UNIX. Both stand-alone
SMP systems and MEMORY CHANNEL[TM] clusters are supported.

1.4 Installation

1.4.1 PVM Software Distributed with PSE

For convenience, DIGITAL PVM software is distributed
as part of the DIGITAL Parallel Software Environment
(PSE). For installation instructions, refer to the PSE
installation guide.

PVM Release Notes 1-1

PVM Release Notes
1.4 Installation

1.4.2 Patch for MEMORY CHANNEL[TM] Software Version 1.4

Any cluster members running version 1.4 of DIGITAL
TruCluster software (or TruCluster MEMORY CHANNEL[TM]
software) must have Patch ID TCR141-013 or its successor
installed. To ensure that you are installing the latest
patch, please send mail to pvm@ilo.dec.com. To obtain
patches, please use your regular DIGITAL support channel.

No patch is needed for systems running version 1.5 or
higher of DIGITAL TruCluster software (or TruCluster MEMORY
CHANNEL[TM] software).

If you have any questions or concerns, please send mail to
pvm@ilo.dec.com.

1.5 Applications Must be Re-Linked for Version 1.8

Programs previously linked with any previous PVM archive
library, will have to be re-linked to operate within the
PVM180 environment.

1.6 Documentation (PostScript and HTML)

The DIGITAL PVM User Guide can be found in PostScript
format at /usr/opt/PVM180/pvm_guide.ps. It can also be
found in HTML format on the Consolidated Layered Products
Documentation CD-ROM.

1.7 New in Version 1.8

DIGITAL PVM Version 1.8 is a bug fix release and contains
no new features.

1.8 New in Version 1.6

DIGITAL PVM Version 1.6 now supports multiple rails of
MEMORY CHANNEL[TM] for message passing between systems.

1.9 Known Problems

The following problems with DIGITAL PVM were known at the
time of release:

1-2 PVM Release Notes

PVM Release Notes
1.9 Known Problems

1.9.1 exec Routines

A process that has called a PVM routine may fail if it
calls execl, execv, execle, execve, execlp, or execvp.
The exec routine returns EWOULDBLOCK. The problem can
be avoided if the process calls fork, and calls the exec
routine in the child process.

1.9.2 Exhausting Virtual Memory Resources

When a process that has called a PVM routine forks,
the child process sometimes loses some virtual memory
resources. If an application uses multiple generations
of processes (parent makes PVM call then forks child, which
makes a PVM call then forks, and so on), the application
may run out of vm-mapentries. The number of vm-mapentries
available to an application can be changed using the
sysconfig(8) command.

PVM Release Notes 1-3

PVM Release Notes
1.10 Miscellaneous

1.10 Miscellaneous

1.10.1 pvm_exit Blocks to Wait for an Available Receiver

A PVM task can send messages before a receiver task is
available to read those messages. Such messages are queued
in the sender task. If the sender task calls pvm_exit
before a receiver task is available to read a message
then the pvm_exit will block until a receiver task becomes
available.

1.10.2 -pthread Compile-Line Argument Needed

The -pthread argument must be included on the compile line.

1.10.3 PVM Environment Variable Defaults

The default values of the DIGITAL PVM environment variables
are described in Table 1-1.

Table_1-1_Default_Values_for_PVM_Environment_Variables_____

Environment_Variable__Default_Value________________________

PVM_BUF_SIZE 1,048,576

PVM_MC_CHAN_SIZE 32,768

PVM_SM_CHAN_SIZE______32,768_______________________________

________________________ Note ________________________

All the "size" parameters should be multiples of 1024.

______________________________________________________

1.11 Problems, Suggestions or Comments

Any problems, suggestions or comments should be addressed
to pvm@ilo.dec.com.

1-4 PVM Release Notes

2
_________________________________________________________________

MPI Release Notes

2.1 Prerequisite Software

DIGITAL MPI Version 1.8 requires DIGITAL UNIX Version 4.0a
or higher. The Fortran runtime libraries are also required.

2.2 Supported Platforms

DIGITAL MPI is a DIGITAL proprietary implementation of MPI,
for Alpha systems running DIGITAL UNIX. Both stand-alone
SMP systems and MEMORY CHANNEL[TM] clusters are supported.

2.3 Compatibility

DIGITAL MPI is compatible with version 1.1.1 of MPICH from
Argonne National Labs, and will run over shared memory and
MEMORY CHANNEL[TM]. Applications built using DIGITAL MPI
cannot communicate via MPI with applications built using
MPICH.

2.4 Applications Must be Re-compiled and Re-linked

DIGITAL MPI version 1.8 requires applications to be both
recompiled and re-linked. Simply re-linking applications is
not sufficient.

MPI Release Notes 2-1

MPI Release Notes
2.5 Installation

2.5 Installation

2.5.1 MPI Software Distributed with PSE

For convenience, DIGITAL MPI software is distributed
as part of the DIGITAL Parallel Software Environment
(PSE). For installation instructions, refer to the PSE
installation guide.

2.5.2 Patch for MEMORY CHANNEL[TM] Software Version 1.4

Any cluster members running version 1.4 of DIGITAL
TruCluster software (or TruCluster MEMORY CHANNEL[TM]
software) must have Patch ID TCR141-013 or its successor
installed. To ensure that you are installing the latest
patch, please send mail to mpi@ilo.dec.com. To obtain
patches, please use your regular DIGITAL support channel.

No patch is needed for systems running version 1.5 or
higher of DIGITAL TruCluster software (or TruCluster MEMORY
CHANNEL[TM] software).

If you have any questions or concerns, please send mail to
mpi@ilo.dec.com.

2.6 Documentation (PostScript and HTML)

The DIGITAL MPI User Guide can be found in PostScript
format at /usr/opt/MPI180/mpi_guide.ps. It can also be
found in HTML format on the Consolidated Layered Products
Documentation CD-ROM.

There is a sample MPI program and instructions in /usr
/examples/mpi.

Reference pages for all of the MPI routines are included
in this distribution, and can be accessed with the man
command.

2.7 New in Version 1.8

Included in Version 1.8 of DIGITAL MPI is support for
V1.0.1 of the ROMIO MPI-2 package from Argonne National
Laboratory.

2-2 MPI Release Notes

MPI Release Notes
2.7 New in Version 1.8

This version of ROMIO includes everything defined in the
MPI-2 I/O chapter except shared file pointer functions
(Section 9.4.4), split collective data access functions
(Section 9.4.5), support for file interoperability (Section
9.5), I/O error handling (Section 9.7), and I/O error
classes (Section 9.8). Since shared file pointer functions
are not supported, the MPI_MODE_SEQUENTIAL amode to
MPI_File_open is also not supported. The subarray and
distributed array datatype constructor functions from
MPI-2 Chapter 4 (Sections 4.14.4 and 4.14.5) have been
implemented. They are useful for accessing arrays stored
in files. The functions MPI_File_f2c and MPI_File_c2f
(Section 4.12.4) also have been implemented. C, Fortran,
and profiling interfaces are provided for all functions
that have been implemented.

Limitations of this version of ROMIO include the following:

o The status argument is not filled in any function.
Consequently, MPI_Get_count and MPI_Get_elements will
not work when passed the status object from an MPI-I/O
operation.

o All nonblocking I/O functions use a ROMIO-defined MPIO_
Request object instead of the usual MPI_Request object.
Accordingly, the two functions, MPIO_Test and MPIO_Wait,
are provided to test and wait on these MPIO_Request
objects. They have the same semantics as MPI_Test and
MPI_Wait.

int MPIO_Test(MPIO_Request *request, int *flag, MPI_Status *status);
int MPIO_Wait(MPIO_Request *request, MPI_Status *status);

The usual functions MPI_Test, MPI_Wait, MPI_Testany, and
so forth, will not work for nonblocking I/O.

o All functions return only two possible error codes -
MPI_SUCCESS on success and MPI_ERR_UNKNOWN on failure.

2.8 Known Problems

These are the known restrictions in DIGITAL MPI Version
1.8:

MPI Release Notes 2-3

MPI Release Notes
2.8 Known Problems

2.8.1 stdout and stderr Streams are Combined

In this release, the stdout and stderr streams from
application processes are combined and are both delivered
to stdout. There is no output on stderr.

2.8.2 Running Under a Debugger Leaves Files in /tmp

When a debugger is used, dmpirun creates (and never
deletes) a number of small files in /tmp. If you repeatedly
debug large applications, you may need to delete these
files to avoid filling /tmp.

2.8.3 A Standard Variance

The Fortran binding for MPI_ADDRESS specifies that
addresses be returned as type INTEGER. This specification
is insufficient, because INTEGERs are 32 bit, whereas
addresses are 64 bit on Alpha systems.

In order to allow use of the MPI_ADDRESS call, this release
uses INTEGER*8 as the return type. This does not affect the
C binding, or its use in a C program.

2.8.4 MPI_REQUEST_FREE

MPI_REQUEST_FREE does not work properly when the request
is not completed (the mpich portable code has the same
problem).

2.8.5 MPI_CANCEL

MPI_CANCEL does not work (the MPICH portable code does not
implement this).

2.8.6 exec Routines

A process that has called MPI_INIT may fail if it calls
execl, execv, execle, execve, execlp, or execvp. The exec
routine returns EWOULDBLOCK.

This problem can be avoided if the process calls fork, and
calls the exec routine in the child process.

2-4 MPI Release Notes

MPI Release Notes
2.8 Known Problems

2.8.7 Exhausting Virtual Memory Resources

When a process that has called MPI_INIT forks, the child
process sometimes loses some virtual memory resources.
If an application uses multiple generations of processes
(parent makes MPI calls then forks child, which makes MPI
calls then forks, and so on), the application may run out
of vm-mapentries. The number of vm-mapentries available
to an application may be changed using the sysconfig(8)
command.

2.8.8 Error Message: "No MEMORY CHANNEL installed"

A call to MPI_INIT that fails with an error reporting "no
MEMORY CHANNEL installed" usually indicates that the MEMORY
CHANNEL[TM] patch has not been installed.

For More Information:

o See Section 2.5.2

2.9 Miscellaneous

2.9.1 User-Level Access for MEMORY CHANNEL[TM]

If you are using MEMORY CHANNEL[TM], ensure that each
machine in the cluster that you intend to use has been
initialised for user-level access. You can check this by
searching for a process called imc_mapper on each machine.
For example,

# ps a | grep imc_mapper | grep -v grep

PID TTY S TIME CMD
657 ttyp2 U 0:00.01 /usr/sbin/imc_mapper

If this process does not exist on any host that you intend
to use as part of the cluster, you should execute the
following command (as root) on each such host:

# /usr/sbin/imc_init

This only needs to be executed once, and has effect until
the machine is next shut down.

MPI Release Notes 2-5

MPI Release Notes
2.9 Miscellaneous

2.9.2 Shared Memory Segment Limit

DIGITAL MPI uses shared memory for communication within a
single host. The default system-wide maximum shared-memory
segment a process can allocate is 4MB. For programs with
a large number of processes this may need to be increased.
This is done by editing the /etc/sysconfigtab file and
adding (or modifying) the following entry:

ipc:
shm-max=size in bytes

For this change to take effect, a reboot is necessary.

2.9.3 -ump_bufs Requires a Multiple of 32

If you specify a buffer size using the -ump_bufs option in
the dmpirun command, then you must specify a buffer that is
a multiple of 32.

Note: you are wasting memory resources if the size you
specify is not a multiple of 8192.

2.10 Problems, Suggestions or Comments

Any problems, suggestions or comments should be addressed
to mpi@ilo.dec.com.

2-6 MPI Release Notes

3
_________________________________________________________________

Comments, Problems, and Help

This chapter gives you detailed instructions on how to
submit comments on this product, report problems, and get
help from the Customer Support Center (CSC).

3.1 Sending DIGITAL Your Comments on This Product

DIGITAL welcomes your comments on this product and on its
documentation. You can send comments to us in the following
ways:

o Internet electronic mail:

- PVM issues: pvm@ilo.dec.com

- MPI issues: mpi@ilo.dec.com

o FAX: 978-493-3628 ATTN: PSE Team

o A Reader's Comment Card sent to the address on the form.

o A letter sent to the following address:

Digital Equipment Corporation
High Performance Computing Group
129 Parker Street PKO3-2/B12
Maynard, Massachusetts 01754-2195 USA

o Online questionnaire form. Print or edit the
questionnaire form provided near the end of these
release notes. Send the form by Internet mail, FAX,
or the postal service.

3.2 Getting Help from DIGITAL

If you have a customer support contract and have comments
or questions about DIGITAL Fortran software, you should
contact DIGITAL's Customer Support Center (CSC), preferably
using electronic means such as DSNlink. In the United
States, customers can call the CSC at (800) 354-9000.

Comments, Problems, and Help 3-1

Comments, Problems, and Help
3.3 Readers Comments Form-Documentation

3.3 Readers Comments Form-Documentation

Use the following form as a template for sending comments
about PSE documentation. This form can be sent by Internet
mail, FAX, or postal service.

3-2 Comments, Problems, and Help

Comments, Problems, and Help
3.3 Readers Comments Form-Documentation

---------------------------------------------------------------------
Please complete this survey and send an online version (via Internet)
or a hardcopy version (via FAX or postal service) to:

Internet mail: pse@hpc.pko.dec.com
FAX: (978) 493-3628
Postal Service: Digital Equipment Corporation,
High Performance Computing Group Documentation, PKO3-2/B12
129 Parker Street
Maynard, Massachusetts 01754-2195 USA

Manual Title: ______________________________________________________

Order Number: ______________________________________________________

Version: ________________________________________________

_____________________________________________________________________

We welcome any comments on this manual or on any PSE documentation.
Your comments and suggestions help us improve the quality of our
publications.

1. If you found any errors, please list them:

Page Description

____ _______________________________________________________________

2. How can we improve the content, usability, or otherwise improve
our documentation set?

_____________________________________________________________________

Your Name/Title ____________________________________ Dept. ________

Company _____________________________________________ Date ________

Internet address or FAX number ______________________________________

Mailing address _____________________________________________________

Comments, Problems, and Help 3-3

Comments, Problems, and Help
3.3 Readers Comments Form-Documentation

___________________________________________ Phone _________________

---------------------------------------------------------------------

3-4 Comments, Problems, and Help