Although applications need not be directly concerned about shadowing matters, certain points must be considered when implementing performance boosting optimizations:

• Anything specific to the executing node should not be stored in the database, since this would lead to diverging copies.
• Any physical reference to the transaction which is unique to the executing server, e.g. Channel ID, system time, DB-key, etc., should not be passed back to the client for future references within its subsequent messages. This could lead to inconsistent handling when a different server is involved in shadow operations.
  This consideration is also valid for recovery of non-shadowed servers.

For more information on designing applications, see the Tolerating Site Disaster section in the Reliable Transaction Router Application Design Guide.

5.8 Server States

The current state of a server can be examined using the SHOW SERVER/FULL command. For example,

RTR> show server/full Servers: Process-id: 13340 Facility: RTR$DEFAULT_FACILITY Channel: 131073 Flags: SRV State: active(1) Low Bound: High Bound: 87 13 rcpnam: "RTR$DEFAULT_CHANNEL" User Events: 0 RTR Events: 0 Partition-Id: 16777216 Process-id: 13340 Facility: RTR$DEFAULT_FACILITY Channel: 196610 Flags: SRV State: active Low Bound: 88 13 High Bound: 0f' rcpnam: "CHAN2" User Events: 0 RTR Events: 0 Partition-Id: 16777217

1. Server state

Figure 5-1 shows the backend server states that can occur and that appear in the State: field.

Figure 5-1 Backend Server States

The current state of a client process can be examined using the SHOW CLIENT/FULL command. For example,

RTR> show client/full Clients: Process-id: 13340 Facility: RTR$DEFAULT_FACILITY Channel: 458755 Flags: CLI State: declared(1) rcpnam: "CHAN3" User Events: 255 RTR Events: 0

1. Client state

Figure 5-2 shows the client states that can occur and that appear in the State: field.

Figure 5-2 Frontend Client States

5.10 Partition States

Router partitions:

RTR> show partition/router/full Facility: RTR$DEFAULT_FACILITY State: ACTIVE(1) Low Bound: 0 High Bound: 4294967295 Failover policy: fail_to_standby Backends: node10 States: active(2) Primary Main: node10 Shadow Main:

1. Router state
2. Backend state

Backend partitions:

RTR> show partition/backend/full Partition name: RTR$DEFAULT_PARTITION_16777217 Facility: RTR$DEFAULT_FACILITY State: active(1) Low Bound: "aaaa" High Bound: "mmmm"(2) Active Servers: 0 Free Servers: 1(3) Transaction presentation: active Last Rcvy BE: Txns Active: 0 Txns Rcvrd: 0 Failover policy: fail_to_standby Key range ID: 16777217(4) Partition name: RTR$DEFAULT_PARTITION_16777218 Facility: RTR$DEFAULT_FACILITY State: active Low Bound: "nnnn" High Bound: "zzzz" Active Servers: 0 Free Servers: 1 Transaction presentation: active Last Rcvy BE: Txns Active: 0 Txns Rcvrd: 0 Failover policy: fail_to_standby Key range ID: 16777218

1. Backend server state
2. Key range for partition
3. Server application channels that are available
4. Key range or partition identification

Figure 5-3 shows the partition states that can occur and that appear in the State: field.

Figure 5-3 Router Partition States

This chapter contains information useful for analyzing performance aspects of RTR, especially in large configurations.

To manage remote nodes, you must have either proxy accounts or rsh access to them. Use RTR remote commands to manage remote nodes.

You should also add and grant operator privileges to the accounts used to manage the RTR network.

6.1 RTR Monitor Pictures

RTR supplies many monitor pictures to help you troubleshoot your application. To display a monitor picture, use the following command at the RTR prompt:

RTR> MONITOR picture-name

The following table provides suggested monitor pictures to display when you encounter problems:

For this type of failure:	Use these monitor pictures:
Most common problems	SYSTEM
Connection failures	ACCFAIL, CONNECTS, FRONTEND, LINK, NETSTAT, STALLS
Transaction sequence problems	CALLS
Channel problems	CALLS, CHANNEL, PARTIT
Quorum problems	QUORUM, ROLEQUOR
V2 interface API	V2CALLS
Journal problems	JCALLS, JOURNAL
API problems	APP2ACP, ACP2APP, REJECTS, REJHIST, ROUTERS
XA interface problems	XA
Application Problems	APP2ACP, ACP2APP, CALLS, CHANNEL, PARTIT, REJECTS, REJHIST, ROUTERS

See Chapter 7 for descriptions and examples of the monitor pictures, and Chapter 8 for the full syntax of the MONITOR command.

6.2 Enabling RTR Logging

Many problems can be better analyzed when RTR logging has been enabled.

RTR logging output can be directed to a file, for example, on RTR startup.

$ RTR SET LOG /FILE=logfile.dat

You should monitor the size of the log file; archive and purge as necessary.

6.3 Starting a Facility

When a facility is started or restarted and servers are declared, RTR recovery features require that it searches journal files of backend nodes in the facility. This allows recovery of any incomplete transactions that were in-flight when the facility last existed. However, if some of the facility's recovery information exists on a backend that is not available at startup, RTR waits for access to the journal on that backend and thus appears to "hang".

This situation can be detected by using MONITOR RECOVERY; backend nodes will be waiting for access to recovery journals. If this is the case, you may follow one of these procedures to continue the startup:

• Delete the facility and recreate it without the unavailable backend.
• Begin the startup by creating a smaller facility and using the EXTEND FACILITY and TRIM FACILITY commands.
• Force a partition to abandon recovery with a SET PARTITION command.

This section provides guidance for System Managers who are analyzing an RTR application that is not functioning correctly.

If an application using RTR hangs, use the following checklist to analyze the situation.

1. Is there a system-level problem on the node concerned, such as a full disk?
2. Has RTR been started? Is RTR running correctly?

   $ RTR SHOW RTR RTR running on node MYNODE in SYSTEM mode

3. Are the application programs running? RTR lists the processes using RTR with the following command:

   $ RTR SHOW PROCESS

   
   The user application processes should be in this list.

4. Has the application stopped?
   Use MONITOR SYSTEM to check for problems. If it indicates a problem with a subsystem, you can get additional information by monitoring that subsystem.
   Network partitioning can also be a problem; this can happen if half or fewer of the configured backends and routers are reachable. To recognize network partitioning, use the MONITOR QUORUM picture. If the number of retries keep increasing without a corresponding increase in the reason counters (CNF/RCH/QRT), you have a partitioned network.
   To check the individual links, use the MONITOR CONNECTS picture. This picture displays the link protocol for connected links, and the reason for a failed connection on any links.
5. Are the application programs running correctly? Use MONITOR CALLS to examine the state of the participating application processes.
   • Does the number of rtr_open_channel calls match the number of rtr_mt_opened messages ? If they do not match, use the MONITOR CONNECTS picture to check individual links.
   • Use MONITOR CONNECTS to make sure the connection to the router is OK.
   • Look in the RTR log file for error messages concerning any unconnected node.
   • Look at any unconnected nodes found, and determine:
     • Is RTR running?
     • Has the RTR command CREATE FACILITY been issued?
     • Are there DECnet problems, e.g., executor maximum links too low? Are the router nodes reachable?
6. Is a server waiting for an rtr_mt_accepted or rtr_mt_rejected message (in other words, has it voted, but not yet received confirmation of the outcome of the transaction)? This is most likely a problem with the application logic. Also check the database for a possible deadlock situation.
7. Is a client channel declaration not completing? Client channels need to have connectivity via a router node to at least one server channel before they get an mt_opened message. If the server is up and running, use MONITOR QUORUM and MONITOR CONNECTS to check connectivity.
8. Has a client channel called rtr_receive_message waiting for an rtr_mt_accepted or rtr_reply_to_client message and not received it within a reasonable time period? Check the application logic and the database for a deadlock.
9. Has a client channel called rtr_receive_message expecting an rtr_mt_accepted or rtr_mt_rejected message that is not forthcoming? If yes, RTR is awaiting the necessary resources for message transmission to the backend servers. Reasons could be:
   • Congestion of a network link, frontend to router or router to backend
   • Server application not correctly dequeuing messages
   • System-level problem on router or backend node
10. Use MONITOR TPS to check the transaction processing rate of each process on a system. A system's capacity is generally expressed as the throughput of the servers. If the rates are low or sporadic, contention may be the cause. For systems with throughput less than 10 tps, the MONITOR TPSLO display provides greater granularity in the associated bar graph.
    Adding server instances can often decrease applicaton throughput if transactions all access common data elements. Partitioning data so that server instances do not interfere with each other is one way to resolve database contention.
11. Use the command SHOW PARTITION/FULL to display the backlog of transactions on a server pool (partition). If the number of free servers is continually zero, the arrival rate of transactions is greater than the processing capacity of the existing server pool.
    The MONITOR QUEUES picture also shows monitor backlogs. This display shows queuing by partition. If the service time and arrival rate of transactions are large, there are not enough servers to process the load. The remedy is to start additional server instances or decrease the processing time of each transaction. Also, many transactions or messages queued can be caused by contention which is limiting the efficiency of servers.
12. Check the state of links with:

    $ RTR SHOW FACILITY /LINK

13. Check if there are sufficient concurrent application server channels to handle the transaction load; messages may have to be queued for long periods before being processed.
    Use MONITOR QUEUES to check the number of outstanding messages for each partition.
14. Check for congestion by examining the network links with the longest delays by using MONITOR TRAFFIC.
    Use the command MONITOR STALLS to determine if the network needs tuning.
    If there is no congestion, use MONITOR FLOW to discover if a link has credits for data traffic, or if the application requires more bandwidth than is available.
15. If the RTRACP dies when adding a facility (which has a backend role on the node), suspect journal file difficulties. Ensure that the journal file is not corrupted, or incompatible with the running RTR version. In the event of journal file corruption, please contact your HP support office.

Analyze the reasons why the server crashed before you restart the server. Failures that cascade could present a problem, but note that doing a restart will prevent failover.

6.6 Link Connect Failures

The following table explains the meaning of link connect failure codes:

Code	Text	Implications
NOTRECOGNISED	Node not recognized	Remote node that received the connection request does not have the local node in its RTR configuration.
REFUSED	Connection refused	Indicates one of the following conditions on the remote system: either RTR is not running, or a requested network protocol is not installed.
FACNOTDEC	Facility not declared	The requested facility is not configured on the remote node.
NODENOTCFG	Node not configured	The remote node has the local node in its configuration, though not as part of the requested facility.
ROLESMISMATCH	Roles mismatch	The remote node has the local node configured in the requested facility, but in a role other than the one requested.

Any of the above errors can occur as the result of the connection request arriving at the wrong node for any of the following reasons:

• You may have mistyped a name on either (or both) the local and remote nodes.
• DNS at either the local or remote nodes (or both) may be giving incorrect address information for the names used.
• The nodes are not in the same RTR group.
• The nodes do not support a common network protocol.
• There is a possible wildcarding error (on routers) in the facility definition.
• Wildcarding or use of the "tunnel." prefix may be necessary due to a firewall.

The following table explains the meaning of rejected transaction codes:

Code	Text	Implications
NODSTFND	No destination found	Primary and all alternate servers for a partition cannot be reached by the client application. This situation can be caused by network problems or services which have not been started or have crashed.
JNLFULL	Journal full	May occur when the RTR journal is full. Note that RTR reserves a percentage of the journal to ensure that in-progress transactions can be completed. The JNLFULL error is most likely to be seen with shadow servers running in remember mode, but can also be caused by many transactions being queued to an unresponsive server.
DLKTXRES	Deadlock detected transaction rescheduled	May occur during the commit cycle for multi-participant transactions or in extreme failover situations when the order of transactions must be corrected. This reject reason indicates that two transactions were interfering with each other. RTR rejects one branch of the offending transactions to clear the deadlock. Since this transaction branch is subsequently rescheduled by RTR, this reject can be considered informational.
TIMEOUT	Time out	Occurs if the rtr_send_to_server timeout provided by the client application expires. This reject indicates poor responsiveness by the service.

6.8 Using the Snapshot Procedure

Certain difficulties can be more easily investigated if a snapshot of the problem node is made. Make a snapshot if the application hangs, causes delays, or seems to be causing other problems.

OpenVMS

On OpenVMS systems, a snapshot is made by executing a command file:

$ @SYS$MANAGER:RTR$SNAPSHOT.COM

The output is a file named nodename _RTR_DIAGS.TMP.

Information in this file can help to determine the possible causes of a fault (OpenVMS, DECnet, RTR, environment, database, application, and so on.) The information includes numerous RTR monitor pictures, executable image versions, process states, and so on.

UNIX

On UNIX systems, make a snapshot by entering the following command on the problem node:

# rtr_snapshot.sh

The information displayed on the screen includes many RTR monitor pictures, executable image versions, process states, and other information.

Windows

To take a snapshot on Windows, click on the Snapshot icon on the RTR menu. A DOS-style window with the title "Snapshot" appears as the snapshot is taken. The file rtr_snapshot.log is created in the directory where RTR runs, for example,
C:\Program Files\HP\RTR . You can read this file with an editor such as Notepad.

Sun

To take a snapshot on Sun, use fssnap . This copies the original filesystem blocks into a file as they are changed, with some performance degradation.

6.9 Generating a Process Dump

OpenVMS Systems

Certain potential difficulties can be more easily investigated by RTR Support if a dump of the RTR ACP is available. It shows diagnostics generated if unhandled exceptions occur.

The file SYS$MANAGER:RTR$STARTUP.COM can be altered to include the definition of the logical name RTR$DUMP_DIRECTORY which specifies the device and directory where the dump is to be generated.

Since an RTR dump file typically uses about 5000 blocks, enough space should be available on the chosen disk. For a very large node installation, or a large number of links, the dump file may be up to 20,000 blocks.

To prepare for dump creation, make sure that:

• SYS$SHARE:IMGDMP has been installed
• RTR has been started from an account having CMKRNL privilege
• The account used to start RTR has write access to the directory specified by RTR$DUMP_DIRECTORY

An RTR ACP dump can be created as follows:

$ RTR RTR> SET MODE /UNSUPPORTED RTR> DEBUG ACP ^G RTR> SET MODE /NOUNSUPPORTED ^Z $

Unsupported commands should be used with care.

UNIX Systems

UNIX core files are generated with no special configuration, but their name and location may vary depending on operating system settings and how RTR is started up. The file rtr_error*.log is usually created in /rtr .

Windows Systems

On Windows systems, a process dump file can be generated by enabling the Dr. Watson post-mortem crash analyzer. This is done by entering the MS-DOS command:

(%WINDIR%\drwtsn32 -i)

The files created are %WINDIR%\DRWTSN32.LOG and %WINDIR%\USER.DMP.

These files should be included with any problem report submitted to RTR Engineering in the event of an RTR crash, along with the RTR dump file (RTR_<n>.DMP) and the RTR log file. The file rtr_error*.log is also created. Send in *.log files when reporting an error, if running RTR with logging in use.