Reliable_Transaction_Router_________________________ Release Notes August, 2001 Reliable Transaction Router (RTR) is fault tolerant transactional messaging middleware used to implement large, distributed applications using client/server technology. These notes describe the new features, corrections, restrictions, and known problems with workarounds for RTR Version 4.1. Software Version: Reliable Transaction Router Version 4.1 Compaq Computer Corporation Houston, Texas ________________________________________________________________ © 2001 Compaq Computer Corporation Compaq, the Compaq logo, OpenVMS, Tru64, VAX, and VMS, and the DIGITAL logo are trademarks of Compaq Information Technologies Group, L.P. Microsoft, MS-DOS, Visual C++, Windows, and Windows NT are trademarks of Microsoft Corporation. Intel is a trademark of Intel Corporation. UNIX and X/Open are trademarks of The Open Group. All other product names mentioned herein may be trademarks of their respective companies. Confidential computer software. Valid license from Compaq or authorized sublicensor required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S.Government under vendor's standard commercial license. Compaq shall not be liable for technical or editorial errors or omissions contained herein. The information in this document is provided "as is" without warranty of any kind and is subject to change without notice. The warranties for Compaq products are set forth in the express limited warranty statements accompanying such products. Nothing herein should be construed as constituting an additional warranty. This document was prepared using VAX DOCUMENT, Version 2.1. _________________________________________________________________ Contents Preface................................................... iv 1 Information Applicable to all Platforms....... 1 1.1 New Features.............................. 1 1.2 Corrections............................... 5 1.3 Restrictions.............................. 7 1.4 Known Problems with Workarounds........... 9 2 OpenVMS-Specific Information.................. 10 2.1 Restrictions.............................. 10 2.2 Known Problems with Workarounds........... 11 3 Tru64-Specific Information.................... 11 3.1 Restrictions.............................. 11 3.2 Known Problems with Workarounds........... 11 4 Windows-Specific Information.................. 11 4.1 Restrictions.............................. 12 4.2 Known Problems with Workarounds........... 12 5 Sun Solaris-Specific Information.............. 13 5.1 Restrictions.............................. 13 5.2 Known Problems with Workarounds........... 13 6 Restrictions Described in Previous Release Notes......................................... 13 iii _________________________________________________________________ Preface Purpose of these Release Notes This document provides information for Reliable Transaction Router Version 4.1. Intended Audience These Release Notes are intended for all users of Reliable Transaction Router. Please read all of this document before using the product. Related Documentation ___________________________________________________________ Document_________Content___________________________________ For all users: Reliable Provides an overview of RTR technology Transaction concepts and solutions. Router Getting Started RTR Commands Lists all RTR commands, their qualifiers Card and defaults. Reliable Describes new features, changes, and known Transaction restrictions for RTR. Router Release Notes For the system manager: iv ___________________________________________________________ Document_________Content___________________________________ Reliable Describes how to install RTR on all Transaction supported platforms. Router Installation Guide Reliable Describes how to configure, manage, and Transaction monitor RTR. Router System Manager's Manual Reliable Explains how to migrate from RTR Version 2 Transaction to later RTR versions (OpenVMS only). Router Migration Guide For the application programmer: Reliable Describes how to design application Transaction programs for use with RTR, illustrated Router with both the C and C++ interfaces. Application Design Guide Reliable Describes the object-oriented C++ Transaction interface that can be used to implement Router C++ RTR object-oriented applications. Foundation Classes Reliable Explains how to design and code RTR Transaction applications using the C programming Router C language; contains full descriptions of Application the basic RTR API calls. Programmer's Reference Manual_____________________________________________________ v Conventions The following conventions are used in this manual: ___________________________________________________________ Convention________Meaning__________________________________ italic text Italic text emphasizes important information, indicates variables, and indicates complete titles of manuals. Italic text also represents information that can vary in system messages (for example, Internal error number), command lines (for example, /PRODUCER=name), and command parameters in text. UPPERCASE TEXT Uppercase text indicates a command, the name of a routine, the name of a file, or the abbreviation for a system privilege. user input This bold typeface is used in interactive examples to indicate typed user input. system output This typeface is used in interactive and __________________code_examples_to_indicate_system_output._ vi 1 Information Applicable to all Platforms This section describes the new features, changes, and restrictions in Reliable Transaction Router, Version 4.1. o Before upgrading to RTR V4.1, recreate your journal files; for more information read the description for PTR 14-8-369 in Section 1.2, Corrections. o For a rolling upgrade, RTR nodes should be upgraded from Version 3.2 to Version 4.0 or 4.1 in the following order: 1. Upgrade router-only and mixed frontend/router nodes 2. Upgrade mixed router/backend nodes 3. Upgrade backend-only nodes 4. Upgrade frontend-only nodes at any time For more complex configurations, where a node may have combinations of facilities with different backend/router groupings, Compaq recommends that you set the recovery protocol to use the Version 3.2 algorithm. Once RTR has been upgraded on all nodes, the recovery protocol can be reset to its default value (Version 4.0 or 4.1), and RTR restarted at a convenient time on each node. This need not be a simultaneous restart on all nodes. RTR can be restarted on each node one by one after resetting the default recovery protocol, so that continuous application availability is maintained. For additional information, see Section 1.3, Restrictions. 1.1 New Features This section describes the new features available in RTR Version 4.1 for all platforms. o C++ API RTR now provides an object-oriented C++ application programming interface. It includes C++ foundation classes which enable you to implement new RTR client and server applications, or to integrate specific classes into existing applications to add additional functionality. 1 RTR concepts have been mapped to and implemented by the set of foundation classes for handling system management and the needs of business applications. o Web-based Interface With Version 4.0, the RTR system management interface is available as a Web-based interface as well as a command line interface. The web interface allows a point and click style of managing RTR from a browser. o Optimized Shadow Recovery RTR Version 4.0 implements an optimized shadow recovery protocol that reduces the time spent in shadow recovery after an extended shadow site outage. The period of time it takes for an RTR shadow configuration to get back into a state of full redundancy is much less as a result. By default, optimized shadow recovery is enabled, but can be controlled by the new command RTR SET NODE/RECOVERY, or by setting the environment variable RTR$CRM_BE_RECOVERY_PROTOCOL (or RTR_CRM_BE_RECOVERY_PROTOCOL) to either Version 3.2 or Version 4.0. In earlier versions of RTR, shadow recovery consisted of sending a recovery query from the node with the recovering partition to the node with the partition in remember state. The query response would consist of the next remembered transaction found by RTR in that node's journal. Every recoverable transaction was requested in this manner, with network messages being sent between the recovering node and the remember node. In RTR Version 4.0, a recovery query is sent by the recovering node to the remember node and results in all recoverable transactions being sent back to the recovering node and written to that node's journal. The recoverable transactions are thus copied in blocks from the remember journal to the recovering node's journal, and deleted from the remember journal. The recovery of transactions then takes place from the recovering node's local journal. The period spent in shadow recovery under Version 4.0 is the time it takes to copy the remember transactions from the remember node to the recovering node, and is generally much less than the time it would 2 take to recover the transactions one by one over the network from the remember node. o New Monitors Four new flow control monitor pictures have been added to RTR: APPDELAY, DOWNSTREAM, FLOSTALLS, and UPSTREAM. These pictures allow monitoring of flow control induced traffic stalls by traffic type, facility, direction, and application process. A new monitor picture, FASTRECOVERY, is provided that displays the number of transactions recovered during optimized shadow recovery. There are three monitor pictures to use with the web browser interface: - ORTR contains a list of server/client transaction controllers. - STCCALLS contains the history of calls for a specified server transaction controller. - CTCCALLS contains the history of calls for a specified client transaction controller. o Information has been added to the .pdf versions of some manuals available on the software kit. o 14-5-234 Pop-up menu on browser windows Pop-up menus have been added to web management pages. They are displayed when the mouse hovers over a facility name, partition name, link, or transaction. The following options are displayed: Show detail of the link Set the link The following options are also displayed for links: Manage the computer represented by the link Manage the computer in a new browser window o 14-5-235 Special RTR standby behavior for Sun Solaris This version of RTR supports the use of external scripts to complement RTR standby failover in unclustered configurations. This behavior can be enabled with the environment variable RTR_STANDBY_WITHOUT_CLUSTER. 3 When this environment variable is set, it modifies the behavior of RTR standby failover as described below. Failover When the ACTIVE node or RTR ACP goes down, the standby node begins failover. As part of its failover, the node scans its available file systems for the RTR journal of the node that failed. This scanning continues until the journal is found. External scripts can be run at this point to make the journal available. However, NFS mounts or network shares are not accepted. A simple file copy also suffices. The current ACTIVE node now has the remote journal opened and locked. Failback Since the current ACTIVE node now has the remote journal locked, when the STANDBY node is restarted its journal is not available. When the facility is created on the STANDBY node, a facility creation event is generated on the ACTIVE node and the remote journal is closed. In addition, an external script named freeremotedisk is also called. User-defined commands can be put in this script to migrate the disks back to their original owner. Once the journal is available in the STANDBY node's file system, it is automatically opened. The user defined script freeremotedisk is located in /opt/rtr/RTR400/bin. Output from this execution is sent to /rtr/freeremotedisk.LOG. Execution of this script is also logged to the RTR log file. Restrictions Whenever the RTR_STANDBY_WITHOUT_CLUSTER variable is set, Compaq recommends that you set RTR_JAM_FAILOVER_WAIT_SECS to some suitable value (for example, 20 secs). RTR polls with this frequency to find the remote journal during failover. By default, this is zero and can lead to high CPU usage. Use of this feature should be restricted to two- node clusters and each node should have both a backend and router role. Before changing these environment variables, all RTR processes must be shut down. 4 1.2 Corrections The following changes and corrections have been made in RTR Version 4.1 for all platforms. o XA A number of issues with XA support have been addressed. o New disk space requirements for all platforms: - OpenVMS - 50K blocks - Tru64 UNIX - 21MB - Windows - 15MB - Sun Solaris - 15MB o 14-1-891 SET PARTITION/RESTART_RECOVERY does not work with standbys The command SET PARTITION/RESTART_RECOVERY was previously effective only for partitions that were in REMEMBER mode. This behavior has been changed to also allow restart recovery while the partition is in an ACTIVE state (for example, after a STANDBY takeover). In environments not recognized by RTR as a cluster, a server can now force a recovery from the failed node's journal, even when RTR has decided to make the partition ACTIVE. If the partition is not in REMEMBER or ACTIVE state, the following error message is generated: %RTR_E_PRTRECSTATE, Partition must be in remember or active (non-recovery) state o 14-1-1400 Rescheduling read-only transactions Previous versions of RTR unnecessarily rescheduled some read-only transactions during a partition state change. o 14-1-1426 Netscape Navigator The web management feature is now usable with Netscape Navigator 6 as well as Netscape 4.75, but Compaq recommends that you use Internet Explorer for optimal performance. o 14-1-1506 MODIFY JOURNAL fails if journal on non-default disk 5 In previous versions of RTR, if you created a journal on a non-default disk and then tried to modify it without specifying the disk, the operation would fail. The MODIFY JOURNAL command now modifies the journal created on the disk specified with the CREATE JOURNAL command. o 14-1-1514 MODIFY JOURNAL failed to extend journal file The MODIFY JOURNAL command may be used to extend the journal size while RTR is running. MODIFY JOURNAL/BLOCKS resets the journal target size (in blocks). The value set by the /MAXIMUM_BLOCKS qualifier is automatically increased if necessary to allow the new target value to take effect. Note that neither of these commands causes immediate file size changes, but allows extension as needed. o 14-1-1547 Memory leak in DUMP JOURNAL A memory leak and other problems found in the DUMP JOURNAL command have been corrected. o 14-1-1566 Unexpected class action deleting journal records In previous versions of RTR, records could occasionally be prematurely deleted from the journal. If the record prematurely deleted was a vote record, then, in circumstances where the existence of that record was used to decide whether an in-flight transaction should be replayed or not, it was possible for the transaction to be executed more than once. This problem could also have caused the RTR_STS_COMSTAUNO status to be generated more often than expected. This problem could also have caused LOGFILENT messages to appear unexpectedly in the RTR log file, for example: %RTR-W-NOFLUSH, flush not performed - nothing to flush %CRM-W-RECORDNOTDEL, error deleting record class x0fff for entry... o 14-1-1634 Crash on shadow recovery In RTR V4.0 ECO1, shadow recovery of transactions involving multiple participants caused the RTR ACP to abort while recovering transactions on the second partition. 6 o 14-3-372 Cannot find partition name to use with SET TRANSACTION The DUMP JOURNAL command now shows the partition names used by a transaction. o 14-8-369 RTR dump journal display showing potential corruption With previous versions of RTR, servers could fail to complete local recovery and remain permanently in state LCL_REC. This was due to a problem with journal garbage collection. The frequency of occurrence of this problem depended upon the pattern of partition failure, the number of partitions, the size of the journal, and the number of journal files. The effect of the problem was to cause a transaction to partially reappear after deletion. Since any journal might contain one or more of these partially reappeared transactions, Compaq recommends that journals be recreated before using the new release of RTR. 1.3 Restrictions The following restrictions apply to all platforms for RTR Version 4.1. o Version 3.2 and Version 4.0 rolling upgrade restrictions The following restrictions apply if you upgrade an RTR environment from RTR Version 3.2 to Version 4.0 or 4.1. The RTR recovery protocol consists of messages sent from one backend, through a router to the target backend. If either of the RTR backends is upgraded to RTR Version 4.0 or 4.1, then the router that will be used for recovery must also be running RTR Version 4.0 or 4.1. This generally means that any RTR router nodes should be upgraded to RTR Version 4.0 or 4.1 before any RTR backend nodes. There is no restriction to the order in which RTR backends should be upgraded to Version 4.0 or 4.1. 7 o If your application sends large numbers of replies within one transaction, be aware that the sending RTR ACP allocates memory for each reply until the transaction is complete. The virtual memory requirements of the RTR ACP should be set accordingly, otherwise it may fail with a malloc() error. For more information on virtual memory requirements, see the RTR System Manager's Manual, Section 2.10, RTR ACP Virtual Memory Sizing. o 14-1-1639 - Journal files will get bigger but not smaller This version of RTR does not support the automatic truncation of RTR Journal files. This can cause the initial read of the RTR journal (that is, when the first facility is created on a node) to take longer than expected when the journal is large but empty. o 14-1-1665 - Facility name only allows 30 characters While RTR documentation states that facility names can be up to 31 characters long, names are limited to 30 characters. o In addition, see PTR 14-1-39 in Section 6, Restrictions Described in Previous Release Notes. o Restrictions on the RTR wakeup handler Compaq does not recommend the use of rtr_reply_to_ client, rtr_send_to_server, or rtr_broadcast_event in an RTR wakeup handler. These calls may block when they need transaction IDs or flow control, which will cause undesired behavior. Functions permitted in an rtr_set_wakeup() handler: For an unthreaded OpenVMS application in an RTR wakeup handler in an AST, Compaq does not recommend the use of rtr_reply_to_client(), rtr_send_to_server(), rtr_ broadcast_event(), or rtr_receive_message() with a non-zero timeout. These calls may block when they need transaction IDs or flow control, which will cause the whole application to hang until the wakeup completes. 8 In an RTR wakeup handler in a threaded application the same rules apply. Note that wakeups are unnecessary in a threaded paradigm, but they may be used in common code in applications that also need to run on OpenVMS. Please note that your mainline code continues to run while your wakeup is executing, so extra synchronization may be required. Also note that if the wakeup does block then it does not generally hang the whole application. In an RTR wakeup handler in a signal in an unthreaded UNIX application, no RTR API functions and only the very few asynch-safe system and library functions may be called, because the wakeup is performed in a signal handler context. An application can write to a pipe or access a volatile sig_atomic_t variable, but using malloc() and printf(), for example, will cause unexpected failures. Alternatively, on most UNIX platforms, you can compile and link the application as a threaded application with the reentrant RTR shared library -lrtr_r. For maximum portability, the wakeup handler should do the minimum necessary to wake up the mainline event loop. You should assume that mainline code and other threads might continue to run in parallel with the wakeup, especially on machines with more than one CPU. 1.4 Known Problems with Workarounds The following known problems with workarounds apply to all platforms for RTR Version 4.1. o 14-1-498 Upgrade using RTR V3.1D journal If you are upgrading to RTR Version 4.0 or 4.1 from RTR Version 3.1D, and there are recoverable transactions in the Version 3.1D journal, then you must set the recovery protocol to Version 3.2 mode before adding any facilities on any node that has access to the Version 3.1D journal. This can be done by entering the RTR SET NODE/RECOVERY=V32 command after starting the RTR ACP, or by defining the environment variable RTR_CRM_BE_RECOVERY_PROTOCOL (system-wide) to be 32, using the method appropriate for your operating system. o 14-1-1427 Web page auto-reload may fail in Netscape browser 9 RTR system management monitor web pages are designed to refresh themselves periodically by instructing the browser to initiate a page reload after an appropriate interval. This mechanism occasionally breaks down when using the Netscape browser. The problem has been seen on both Netscape Communicator Version 4.05 and Version 4.72 on Compaq Tru64 UNIX. The refresh can be restarted by manually reloading the frame. o 14-7-991 DUMP JOURNAL output may be incorrect while journal is in use The DUMP JOURNAL command could give spurious results if it is issued while the journal is in use by an RTR ACP process. The DUMP JOURNAL command now reports %RTR-W-JOUINUSE if the journal is still being accessed by an RTR ACP process. A workaround is to make a consistent snapshot copy of the journal files and dump that instead. Some filesystem types such as AdvFS allow you to clone a filesystem for backups, or you can take one disk of a mirror pair offline. Otherwise you will have to stop RTR. The journal files can then be copied to a location where they will not be found by the journal scan on the production system (not the group subdirectory of a top level rtrjnl directory), and inspected offline on another machine, or on the same machine but with RTR DUMP JOURNAL being run by a different user who has used SET MODE /GROUP to set a group different from the production system. 2 OpenVMS-Specific Information This section gives platform-specific information for the OpenVMS implementation of Reliable Transaction Router, Version 4.1. 2.1 Restrictions See PTR 14-3-323 in Section 6, Restrictions Described in Previous Release Notes. 10 2.2 Known Problems with Workarounds The following known problems with workarounds apply to the OpenVMS platform for RTR Version 4.1. o 14-1-1568 ASTs disabled by exit handler ASTs are incorrectly left disabled after the RTR exit handler is invoked. This may adversely affect subsequent processing including any exit handlers installed by the application before the first RTR API call. Such an exit handler could work around the problem by reenabling ASTs. o 14-3-409 RTR V2 command line API not covered in RTR V4 help Before uninstalling RTR V2, make a copy of the RTR V2 help file from SYS$COMMON:[SYSHLP]RTRHLP.HLB and then copy it back afterwards. You can then access it with $ help @RTRHLP 3 Tru64-Specific Information This section gives platform-specific information for the implementation of Reliable Transaction Router Version 4.1 for Tru64 UNIX. 3.1 Restrictions See Ptrs 14-1-836, 14-1-1382, and 14-9-79 in Section 6, Restrictions Described in Previous Release Notes. 3.2 Known Problems with Workarounds None. 4 Windows-Specific Information This section gives platform-specific information for the implementation of Reliable Transaction Router Version 4.1 for Windows platforms. 11 4.1 Restrictions None. 4.2 Known Problems with Workarounds The following known problems with workarounds apply to Windows platforms for RTR Version 4.1. o 14-1-997 RTR command and HTTP servers not available after logout RTR command and HTTP servers started during the course of an interactive Windows session do not survive when users log out of that session. This means that the machine is no longer accessible to the browser GUI. This difficulty can be avoided by enabling rsh access to the Windows system. The HTTP server can then be started with a remote rtr command directed at the system concerned. rsh functionality for Windows is available from third parties. A version also ships with the Microsoft NT 4.0 Server Resource Kit. o 14-1-1628 Microsoft DTC patch update required for RTR XA & SQL running on Windows NT 4.0 A bug in Microsoft's Distributed Transaction Coordinator related to the xa_recover() call not returning the full list of in-doubt transactions has been fixed by Microsoft. If you are running Microsoft DTC Version 2.0, check the version number of the msdtc.dll image. If the version number is less than 1999.6.854.0, or if you are not running DTC Version 2.0, verify that NT 4.0 Service Pack 6A is installed, and then upgrade DTC manually. The DTC upgrade, contained in file dtcsetup.exe, and the instructions are available at: ftp://ftp.microsoft.com/bussys/distapps/ MTS/Public-Fixes/usa/DTC/SvcPack/nt4_sp6 ________________________ Note ________________________ This upgrade installs/upgrades to Microsoft DTC Version 2.0, msdtc.dll image version 1999.6.854.0. 12 The upgrade to DTC version 2.0 is the minimum required for RTR to work properly with XA. ______________________________________________________ o In addition, see PTR 14-1-455 in Section 6, Restrictions Described in Previous Release Notes. 5 Sun Solaris-Specific Information This section gives platform-specific information for the implementation of Reliable Transaction Router Version V4.1 for the Sun Solaris platform. 5.1 Restrictions Reliable Transaction Router Version 4.1 requires Sun Solaris 2.6 or newer; it will not run on Solaris 2.5. 5.2 Known Problems with Workarounds None. 6 Restrictions Described in Previous Release Notes The following restrictions were described in previous release notes and still apply. o 14-1-39 Declaring exit handlers in RTR applications If an exit handler contains calls to RTR, then the exit handler must be declared after the first call to RTR. Using the RTR Version 2 or Version 3 API, if the exit handler is declared before the first call to RTR, then any call to RTR made within the exit handler will return an error. With the Version 3 API, the error status returned is RTR_STS_INVCHANNEL. With the Version 2 API, the error status returned is RTR$_INVALCH. o 14-1-103 Using rtr_set_wakeup() in a threaded program After calling rtr_set_wakeup() in a threaded program, you should also call rtr_set_wakeup(NULL) wherever your program can exit. This prevents any wakeups in other threads while the main thread is already running the RTR exit handler. Wakeups in other threads could cause a server core dump when trying to stop the server. 13 o 14-1-263 Non-English character sets are not supported for identifiers The supported character set for RTR identifiers such as facility names is ASCII, with lowercase and uppercase letters equivalent. Eight bit characters are not supported because the name might not interoperate with RTR processes at a different location or running another RTR version. o 14-1-455 Last line of batch procedure sometimes ignored On Windows platforms, the last line of a batch procedure or command file must explicitly end with , added by pressing the Enter/Return key when creating the procedure. Without the explicit , RTR ignores the line. The workaround is to add a comment to the end of the file or to explicitly add to the end of the last line of the batch procedure. o 14-1-681, 14-3-278 ACP crash in RDM subsystem With many concurrently active transactions, where each transaction sends back many very large replies, it is possible to exhaust the virtual memory requirements of RTR to store the replies for possible recovery after a link glitch. Exhausting virtual memory would cause RTR to crash on a backend node. A workaround is to reduce the number of concurrent server channels to limit the number of concurrently active transactions on each backend, or to limit the number of replies per transaction. In Compaq tests, we exceeded the RTR limits using 10 servers, each sending back 200 replies of 64KB each. o 14-1-769 System time must not be set backwards Correct operation of RTR is not guaranteed if the time is not monotonic. When performing Year 2000 testing, stop all RTR processes and remove any journalled transactions before setting the clock back. When configuring the Network Time Protocol daemon ntp, do not select or accept by default any options that may cause the system clock to be set backwards. 14 While RTR is running as a service on NT, do not allow the time to be set backwards by ntp or in login scripts. On OpenVMS systems, do not change to a different time zone, or to or from Summer Time or Daylight Savings Time, while using current versions of RTR which are built on OpenVMS V6 to run on both OpenVMS V6 and V7. On most UNIX systems, Compaq does not recommend that you change the date and time when running in multi-user mode. Adjustments are normally made by speeding up or slowing down the clock. o 14-1-836, 14-1-1382 Crash on Tru64 UNIX Versions 5.0 and 5.0A with a sendmsg error The sendmsg binary incompatibility bug that causes a crash with EINVAL when uninitialized data is found in the newly extended part of iov_len is fixed in Compaq Tru64 UNIX 5.1 and the first patch kit for Version 5.0A. To run on Tru64 UNIX 5.0A, RTR requires Patch Kit 1 BL1 -18-Jul-2000, Patch 0096.00. o 14-3-323 Creation of CSV process on OpenVMS relies on clean LOGIN.COM If running the login.com file produces an error, RTR may not run. o 14-9-79 Support of TruCluster Server 5 The /rtr directory is now a symbolic link to a target directory that is a context-dependent symbolic link (CDSL), so that it is a different physical directory on each cluster member. Before installing RTR on a TruCluster Server, you must first remove any existing /rtr directory or /rtr symbolic link, its target directory and the directory's contents on all member nodes. If you do not need the previous journal, Compaq also recommends that you remove /rtrjnl and its contents; it will then be recreated as a symbolic link to a shared directory in /var, which usually has more space than the / filesystem. Note that although the /rtrjnl directory can also be replaced by a symbolic link, the target should be an ordinary cluster-wide directory and not a member- specific CDSL, so that each member can access the 15 journal files belonging to the other. Similarly, any other devices that you may specify when creating journals should normally also refer to cluster-wide shared directories. 16