HP OpenVMS Systems Documentation

OpenVMS Alpha System Analysis Tools Manual

Chapter 5
SDA CLUE Extension Commands

This chapter presents an overview of the SDA CLUE (Crash Log Utility Extractor) extension commands, how to display information using these commands, and how to use SDA CLUE with DOSD. This chapter also describes the SDA CLUE commands.

5.1 Overview of SDA CLUE Extensions

SDA CLUE (Crash Log Utility Extractor) commands automate the analysis of crash dumps and maintain a history of all fatal bugchecks on either a standalone or cluster system. You can use SDA CLUE commands in conjunction with SDA to collect and decode additional dump file information not readily accessible through standard SDA commands. SDA CLUE extension commands can summarize information provided by certain standard SDA commands and provide additional detail for some SDA commands. For example, SDA CLUE extension commands can quickly provide detailed extended QIO processor (XQP) summaries. You can also use SDA CLUE commands interactively on a running system to help identify performance problems.

You can use all CLUE commands when analyzing crash dumps; the only CLUE commands that are not allowed when analyzing a running system are CLUE CRASH, CLUE ERRLOG, CLUE HISTORY, and CLUE STACK.

When you reboot the system after a system failure, you automatically invoke SDA by default. To facilitate better crash dump analysis, SDA CLUE commands automatically capture and archive summary dump file information in a CLUE listing file.

A startup command procedure initiates commands that do the following:

Invoke SDA
Issue an SDA CLUE HISTORY command
Create a listing file called CLUE$nodename_ddmmyy_hhmm.LIS

The CLUE HISTORY command adds a one-line summary entry to a history file and saves the following output from SDA CLUE commands in the listing file:

Crash dump summary information
System configuration
Stack decoder
Page and swap files
Memory management statistics
Process DCL recall buffer
Active XQP processes
XQP cache header

The contents of this CLUE list file can help you analyze a system failure. If these files accumulate more space than the threshold allows (default is 5000 blocks), the oldest files are deleted until the threshold limit is reached. You can also customize this list file using the CLUE$MAX_BLOCK logical name.

For additional information on the contents of the CLUE listing file, see the reference section on CLUE HISTORY.

It is important to remember that CLUE$nodename_ddmmyy_hhmm.LIS contains only an overview of the crash dump and does not always contain enough information to determine the cause of the crash. The dump itself should always be saved using the procedures described in Section 2.2.2 and Section 2.2.3.

To inhibit the running of CLUE at system startup, define the logical CLUE$INHIBIT in the SYLOGICALS.COM file as /SYS TRUE.

5.2 Displaying Data Using SDA CLUE Commands

To invoke a CLUE command, enter the command at the SDA prompt. For example:

SDA> CLUE CONFIG

5.3 Using SDA CLUE with DOSD

DOSD (Dump Off System Disk) allows you to write the system dump file to a device other than the system disk. For SDA CLUE to be able to correctly find the dump file to be analyzed after a system crash, you need to perform the following steps:

Modify the command procedure SYS$MANAGER:SYCONFIG.COM to add the system logical name CLUE$DOSD_DEVICE to point to the device where the dump file resides. You need to supply only the physical or logical device name without a file specification.
Modify the command procedure SYS$MANAGER:SYCONFIG.COM to mount systemwide the device where the dump file resides. Otherwise, SDA CLUE cannot access and analyze the dump file.

In the following example, the dump file has been placed on device $3$DUA25, which has the label DMP$DEV. You need to add the following commands to SYS$MANAGER:SYCONFIG.COM:

$mount/system/noassist $3$dua25: dmp$dev dmp$dev
$define/system clue$dosd_device dmp$dev

5.4 Listing of SDA CLUE Extension Commands

This section describes the following SDA CLUE extension commands:

CLUE CALL_FRAME
CLUE CLEANUP
CLUE CONFIG
CLUE CRASH
CLUE ERRLOG
CLUE FRU
CLUE HISTORY
CLUE MCHK
CLUE MEMORY
CLUE PROCESS
CLUE REGISTER
CLUE SG
CLUE STACK
CLUE SYSTEM
CLUE VCC
CLUE XQP

CLUE CALL_FRAME

Displays key information, such as the PC of the caller, from the active call frames at time of the crash.

Format

CLUE CALL_FRAME [/CPU [cpu-id|ALL]
|/PROCESS [/ADDRESS=n|INDEX=n
|/IDENTIFICATION=n|process-name|ALL]]

Parameters

ALL
When used with /CPU, it requests information about all CPUs in the system. When used with /PROCESS, it requests information about all processes that exist in the system.
cpu-id
When used with /CPU, it gives the number of the CPU for which information is to be displayed. Use of the cpu-id parameter causes the CLUE CALL_FRAME command to perform an implicit SET CPU command, making the indicated CPU the current CPU for subsequent SDA commands.
process-name
When used with /PROCESS, it gives the name of the process for which information is to be displayed. Use of the process-name parameter, the /ADDRESS qualifier, the /INDEX qualifier, or the /IDENTIFICATION qualifier causes the CLUE CALL_FRAME command to perform an implicit SET PROCESS command, making the indicated process the current process for subsequent SDA commands. You can determine the names of the processes in the system by issuing a SHOW SUMMARY command.
The process-name can contain up to 15 letters and numerals, including the underscore (_) and dollar sign ($). If it contains any other characters, you must enclose the process-name in quotation marks (" ").

Qualifiers

/ADDRESS=n
Specifies the PCB address of the desired process when used with CLUE CALL_FRAME/PROCESS.
/CPU [cpu-id|ALL]
Indicates that the call frame for a CPU is required. Specify the CPU by its number or use ALL to indicate all CPUs.
/IDENTIFICATION=n
Specifies the identification of the desired process when used with CLUE CALL_FRAME/PROCESS.
/INDEX=n
Specifies the index of the desired process when used with CLUE CALL_FRAME/PROCESS.
/PROCESS [process-name|ALL]
Indicates that the call frame for a process is required. The process should be specified with either one of the qualifiers /ADDRESS, /IDENTIFICATION, or /INDEX, or by its name, or by using ALL to indicate all processes.

Description

The CLUE CALL_FRAME command displays call chain information for a process or a CPU. The process context calls work on both the running system and dump file; the CPU context calls only on dump files.
If neither /CPU nor /PROCESS is specified, the parameter (CPU-id or process-name) is ignored and the call frame for the SDA current process is displayed.

Examples

SDA>CLUE CALL/PROCESS IPCACP
Call Chain:   Process index: 000B   Process name: IPCACP  PCB: 8136EF00
-----------------------------------------------------------------------
Procedure Frame  Procedure Entry                          Return Address
---------------  ----------------------------------       ---------------------------
7FFA1CA0  Null   800C8C90  SCH$WAIT_PROC_C
7FFA1D00  Stack  800D9250  SYS$HIBER_C                    0003045C  IPCACP+0003045C
7FFA1D50  Stack  00030050  IPCACP+00030050                800D11C8  EXE$CMKRNL_C+000D8
7FFA1E60  Null   800B6120  EXE$BLDPKTSWPR_C
7FFA1E78  Null   800B6120  EXE$BLDPKTSWPR_C
7FFA1EC0  Null   80248120  NSA$CHECK_PRIVILEGE_C
7FFA1F00  Null   80084640  EXE$CMODEXECX_C
7FFA1F70  Stack  800D10F0  EXE$CMKRNL_C                   80084CC8  EXE$CMODKRNL_C+00198
7B01FAB0  Stack  00030010  IPCACP+00030010                83EA3454  SYS$IMGSTA_C+00154
7B01FB10  Stack  83EA3300  SYS$IMGSTA_C                   83D99CC4  EXE$PROC_IMGACT_C+00384
7B01FBA0  Stack  83D99BA0  EXE$PROC_IMGACT_C+00260        83D99B9C  EXE$PROC_IMGACT_C+0025C

In this example, the CLUE CALL_FRAME command displays the call frame from the process IPCACP.

SDA>CLUE CALL/CPU ALL
Call Chain:   Process index: 0000   Process name: NULL    PCB: 827377C0   (CPU 0)
---------------------------------------------------------------------------------
Procedure Frame  Procedure Entry                          Return Address
---------------  ----------------------------------       ---------------------------
8F629D28  Null   80205E00  SYS$SCS+05E00
8F629D68  Null   8020A850  SCS$REC_MSGREC_C
8F629D98  Null   914A5340  SYS$PBDRIVER+07340
8F629DB8  Null   914A4FD0  SYS$PBDRIVER+06FD0
8F629DE0  Stack  914AACF0  SYS$PBDRIVER+0CCF0             914AE5CC  SYS$PBDRIVER+105CC
8F629E50  Stack  914AE418  SYS$PBDRIVER+10418             800503B0  EXE_STD$QUEUE_FORK_C+00350
8F629F88  Null   800E95F4  SCH$WAIT_ANY_MODE_C
8F629FD0  Stack  800D0F80  SCH$IDLE_C                     800E92D0  SCH$INTERRUPT+00BB0
Call Chain:   Process index: 0000   Process name: NULL    PCB: 827377C0   (CPU 2)
---------------------------------------------------------------------------------
Procedure Frame  Procedure Entry                          Return Address
---------------  --------------------------------         ---------------------------
90FCBF88  Null   800E95F4  SCH$WAIT_ANY_MODE_C
90FCBFC8  Null   800E95F4  SCH$WAIT_ANY_MODE_C
90FCBFD0  Stack  800D0F80  SCH$IDLE_C                     800E92D0  SCH$INTERRUPT+00BB0
Call Chain:   Process index: 0000   Process name: NULL    PCB: 827377C0   (CPU 6)
---------------------------------------------------------------------------------
Procedure Frame  Procedure Entry                          Return Address
---------------  ------------------------------           ---------------------------
90FCBF88  Null   800E95FA  SCH$WAIT_ANY_MORE_c
90FD9F88  Null   800E95F4  SCH$WAIT_ANY_MODE_C
90FD9FD0  Stack  800D0F80  SCH$IDLE_C                     800E92D0  SCH$INTERRUPT+00BB0

In this example, CLUE/CPU ALL shows the call frame for all CPUs.

CLUE CLEANUP

Performs housekeeping operations to conserve disk space.

Format

CLUE CLEANUP

Parameters

None.

Qualifiers

None.

Description

CLUE CLEANUP performs housekeeping operations to conserve disk space. To avoid filling up the system disk with listing files generated by CLUE, CLUE CLEANUP is run during system startup to check the overall disk space used by all CLUE$*.LIS files.
If the CLUE$COLLECT:CLUE$*.LIS files occupy more space than the logical CLUE$MAX_BLOCKS allows, then the oldest files are deleted until the threshold is reached. If this logical name is not defined, a default value of 5,000 disk blocks is assumed. A value of zero disables housekeeping and no check on the disk space is performed.

Example


SDA> CLUE CLEANUP %CLUE-I-CLEANUP, housekeeping started... %CLUE-I-MAXBLOCK, maximum blocks allowed 5000 blocks %CLUE-I-STAT, total of 4 CLUE files, 192 blocks.

In this example, the CLUE CLEANUP command displays that the total number of blocks of disk space used by CLUE files does not exceed the maximum number of blocks allowed. No files are deleted.

CLUE CONFIG

Displays the system, memory, and device configurations.

Format

CLUE CONFIG

Parameters

None.

Qualifiers

None.

Description

CLUE CONFIG displays the system, memory, and device configurations.

Example

CLUE CRASH

Displays a crash dump summary.

Format

CLUE CRASH

Parameters

None.

Qualifiers

None.

Description

CLUE CRASH displays a crash dump summary, which includes the following items:

Bugcheck type
Current process and image
Failing PC and PS
Executive image section name and offset
General registers
Failing instructions
Exception frame, signal and mechanism arrays (if available)
CPU state information (spinlock related bugchecks only)

Example

SDA>  CLUE CRASH
Crash Time:        30-AUG-1996 13:13:46.83
Bugcheck Type:     SSRVEXCEPT, Unexpected system service exception
Node:              SWPCTX  (Standalone)
CPU Type:          DEC 3000 Model 400
VMS Version:       X6AF-FT2
Current Process:   SYSTEM
Current Image:     $31$DKB0:[SYS0.][SYSMGR]X.EXE;1
Failing PC:        00000000.00030078    SYS$K_VERSION_01+00078
Failing PS:        00000000.00000003
Module:            X
Offset:            00030078

Boot Time:         30-AUG-1996 09:06:22.00
System Uptime:               0 04:07:24.83
Crash/Primary CPU: 00/00
System/CPU Type:   0402
Saved Processes:   18
Pagesize:          8 KByte (8192 bytes)
Physical Memory:   64 MByte (8192 PFNs, contiguous memory)
Dumpfile Pagelets: 98861 blocks
Dump Flags:        olddump,writecomp,errlogcomp,dump_style
Dump Type:         raw,selective
EXE$GL_FLAGS:      poolpging,init,bugdump
Paging Files:      1 Pagefile and 1 Swapfile installed

Stack Pointers:
KSP = 00000000.7FFA1C98   ESP = 00000000.7FFA6000   SSP = 00000000.7FFAC100
USP = 00000000.7AFFBAD0

General Registers:
R0  = 00000000.00000000   R1  = 00000000.7FFA1EB8   R2  = FFFFFFFF.80D0E6C0
R3  = FFFFFFFF.80C63460   R4  = FFFFFFFF.80D12740   R5  = 00000000.000000C8
R6  = 00000000.00030038   R7  = 00000000.7FFA1FC0   R8  = 00000000.7FFAC208
R9  = 00000000.7FFAC410   R10 = 00000000.7FFAD238   R11 = 00000000.7FFCE3E0
R12 = 00000000.00000000   R13 = FFFFFFFF.80C6EB60   R14 = 00000000.00000000
R15 = 00000000.009A79FD   R16 = 00000000.000003C4   R17 = 00000000.7FFA1D40
R18 = FFFFFFFF.80C05C38   R19 = 00000000.00000000   R20 = 00000000.7FFA1F50
R21 = 00000000.00000000   R22 = 00000000.00000001   R23 = 00000000.7FFF03C8
R24 = 00000000.7FFF0040   AI  = 00000000.00000003   RA  = FFFFFFFF.82A21080
PV  = FFFFFFFF.829CF010   R28 = FFFFFFFF.8004B6DC   FP  = 00000000.7FFA1CA0
PC  = FFFFFFFF.82A210B4   PS  = 18000000.00000000

Exception Frame:
R2  = 00000000.00000003   R3  = FFFFFFFF.80C63460   R4  = FFFFFFFF.80D12740
R5  = 00000000.000000C8   R6  = 00000000.00030038   R7  = 00000000.7FFA1FC0
PC  = 00000000.00030078   PS  = 00000000.00000003

Signal Array:                            64-bit Signal Array:
Arg Count    = 00000005                  Arg Count      =          00000005
Condition    = 0000000C                  Condition      = 00000000.0000000C
Argument #2  = 00010000                  Argument #2    = 00000000.00010000
Argument #3  = 00000000                  Argument #3    = 00000000.00000000
Argument #4  = 00030078                  Argument #4    = 00000000.00030078
Argument #5  = 00000003                  Argument #5    = 00000000.00000003

Mechanism Array:
Arguments    = 0000002C                  Establisher FP = 00000000.7AFFBAD0
Flags        = 00000000                  Exception FP   = 00000000.7FFA1F00
Depth        = FFFFFFFD                  Signal Array   = 00000000.7FFA1EB8
Handler Data = 00000000.00000000         Signal64 Array = 00000000.7FFA1ED0
R0  = 00000000.00020000   R1  = 00000000.00000000   R16 = 00000000.00020004
R17 = 00000000.00010050   R18 = FFFFFFFF.FFFFFFFF   R19 = 00000000.00000000
R20 = 00000000.7FFA1F50   R21 = 00000000.00000000   R22 = 00000000.00010050
R23 = 00000000.00000000   R24 = 00000000.00010051   R25 = 00000000.00000000
R26 = FFFFFFFF.8010ACA4   R27 = 00000000.00010050   R28 = 00000000.00000000

System Registers:
Page Table Base Register (PTBR)                           00000000.00001136
Processor Base Register (PRBR)                            FFFFFFFF.80D0E000
Privileged Context Block Base (PCBB)                      00000000.003FE080
System Control Block Base (SCBB)                          00000000.000001DC
Software Interrupt Summary Register (SISR)                00000000.00000000
Address Space Number (ASN)                                00000000.0000002F
AST Summary / AST Enable (ASTSR_ASTEN)                    00000000.0000000F
Floating-Point Enable (FEN)                               00000000.00000000
Interrupt Priority Level (IPL)                            00000000.00000000
Machine Check Error Summary (MCES)                        00000000.00000000
Virtual Page Table Base Register (VPTB)                   FFFFFFFC.00000000

Failing Instruction:
SYS$K_VERSION_01+00078:         LDL             R28,(R28)

Instruction Stream (last 20 instructions):
SYS$K_VERSION_01+00028:         LDQ             R16,#X0030(R13)
SYS$K_VERSION_01+0002C:         LDQ             R27,#X0048(R13)
SYS$K_VERSION_01+00030:         LDA             R17,(R28)
SYS$K_VERSION_01+00034:         JSR             R26,(R26)
SYS$K_VERSION_01+00038:         LDQ             R26,#X0038(R13)
SYS$K_VERSION_01+0003C:         BIS             R31,SP,SP
SYS$K_VERSION_01+00040:         BIS             R31,R26,R0
SYS$K_VERSION_01+00044:         BIS             R31,FP,SP
SYS$K_VERSION_01+00048:         LDQ             R28,#X0008(SP)
SYS$K_VERSION_01+0004C:         LDQ             R13,#X0010(SP)
SYS$K_VERSION_01+00050:         LDQ             FP,#X0018(SP)
SYS$K_VERSION_01+00054:         LDA             SP,#X0020(SP)
SYS$K_VERSION_01+00058:         RET             R31,(R28)
SYS$K_VERSION_01+0005C:         BIS             R31,R31,R31
SYS$K_VERSION_01+00060:         LDA             SP,#XFFE0(SP)
SYS$K_VERSION_01+00064:         STQ             FP,#X0018(SP)
SYS$K_VERSION_01+00068:         STQ             R27,(SP)
SYS$K_VERSION_01+0006C:         BIS             R31,SP,FP
SYS$K_VERSION_01+00070:         STQ             R26,#X0010(SP)
SYS$K_VERSION_01+00074:         LDA             R28,(R31)
SYS$K_VERSION_01+00078:         LDL             R28,(R28)
SYS$K_VERSION_01+0007C:         BEQ             R28,#X000007
SYS$K_VERSION_01+00080:         LDQ             R26,#XFFE8(R27)
SYS$K_VERSION_01+00084:         BIS             R31,R26,R0
SYS$K_VERSION_01+00088:         BIS             R31,FP,SP

CLUE ERRLOG

Extracts the error log buffers from the dump file and places them into the binary file called CLUE$ERRLOG.SYS.

Format

CLUE ERRLOG [/OLD]

Parameters

None.

Qualifier

/OLD
Dumps the errorlog buffers into a file using the old errorlog format. The default action, if /OLD is not specified, is to dump the errorlog buffers in the common event header format.

Description

CLUE ERRLOG extracts the error log buffers from the dump file and places them into the binary file called CLUE$ERRLOG.SYS.
These buffers contain messages not yet written to the error log file at the time of the failure. When you analyze a failure on the same system on which it occurred, you can run the Error Log utility on the actual error log file to see these error log messages. When analyzing a failure from another system, use the CLUE ERRLOG command to create a file containing the failing system's error log messages just prior to the failure. System failures are often triggered by hardware problems, so determining what, if any, hardware errors occurred prior to the failure can help you troubleshoot a failure.
You can define the logical CLUE$ERRLOG to any file specification if you want error log information written to a file other than CLUE$ERRLOG.SYS.

Note
You need at least DECevent V2.9 to analyze the new common event header (CEH) format file. The old format file can be analyzed by ANALYZE/ERROR or any version of DECevent.

Example


SDA> CLUE ERRLOG Sequence Date Time -------- ----------- ----------- 128 11-MAY-1994 00:39:31.30 129 11-MAY-1994 00:39:32.12 130 11-MAY-1994 00:39:44.83 131 11-MAY-1994 00:44:38.97 * Crash Entry

In addition to writing the error log buffers into CLUE$ERRLOG.SYS, the CLUE ERRLOG command displays the sequence, date, and time of each error log buffer extracted from the dump file.

CLUE FRU

Outputs the Field Replacement Unit (FRU) table to a file for display by DECevent.

Format

CLUE FRU

Parameters

None.

Qualifiers

None.

Description

The FRU command extracts the FRU table into an output file (CLUE$FRU.SYS), which can then be displayed by DECevent. This command works on the running system, as well as on dump files.

CLUE HISTORY

Updates history file and generates crash dump summary output.

Format

CLUE HISTORY [/qualifier]

Parameters

None.

Qualifier

/OVERRIDE
Allows execution of this command even if the dump file has already been analyzed (DMP$V_OLDDUMP bit set).

Description

This command updates the history file pointed to by the logical name CLUE$HISTORY with a one-line entry and the major crash dump summary information. If CLUE$HISTORY is not defined, a file CLUE$HISTORY.DAT in your default directory will be created.
In addition, a listing file with summary information about the system failure is created in the directory pointed to by CLUE$COLLECT. The file name is of the form CLUE$node_ddmmyy_hhmm.LIS where the timestamp (hhmm) corresponds to the system failure time and not the time when the file was created.
The listing file contains summary information collected from the following SDA commands:

CLUE CRASH
CLUE CONFIG
CLUE MEMORY/FILES
CLUE MEMORY/STATISTIC
CLUE PROCESS/RECALL
CLUE XQP/ACTIVE

Refer to the reference section for each of these commands to see examples of the displayed information.
The logical name CLUE$FLAG controls how much information is written to the listing file.

Bit 0---Include crash dump summary
Bit 1---Include system configuration
Bit 2---Include stack decoding information
Bit 3---Include page and swap file usage
Bit 4---Include memory management statistics
Bit 5---Include process DCL recall buffer
Bit 6---Include active XQP process information
Bit 7---Include XQP cache header

If this logical name is undefined, all bits are set by default internally and all information is written to the listing file. If the value is zero, no listing file is generated. The value has to be supplied in hexadecimal form (for example, DEFINE CLUE$FLAG 81 will include the crash dump summary and the XQP cache header information).
If the logical name CLUE$SITE_PROC points to a valid and existing file, it will be executed as the final step of the CLUE HISTORY command (for example, automatic saving of the dump file during system startup). If used, this file should contain only valid SDA commands.
Refer to Chapter 2, Section 2.2.3 for more information on site-specific command files.

Contents

Index

HP OpenVMS Systems Documentation

OpenVMS Alpha System Analysis Tools Manual

Chapter 5SDA CLUE Extension Commands

5.1 Overview of SDA CLUE Extensions

5.2 Displaying Data Using SDA CLUE Commands

5.3 Using SDA CLUE with DOSD

5.4 Listing of SDA CLUE Extension Commands

CLUE CALL_FRAME

Format

Parameters

ALL

cpu-id

process-name

Qualifiers

/ADDRESS=n

/CPU [cpu-id|ALL]

/IDENTIFICATION=n

/INDEX=n

/PROCESS [process-name|ALL]

Description

Examples

CLUE CLEANUP

Format

Parameters

Qualifiers

Description

Example

CLUE CONFIG

Format

Parameters

Qualifiers

Description

CLUE CRASH

Format

Parameters

Qualifiers

Description

Example

CLUE ERRLOG

Format

Parameters

Qualifier

/OLD

Description

Example

CLUE FRU

Format

Parameters

Qualifiers

Description

CLUE HISTORY

Format

Parameters

Qualifier

/OVERRIDE

Description

Chapter 5
SDA CLUE Extension Commands