HP OpenVMS Systems

HP Integrity Virtual Machines (Integrity VM) supports OpenVMS for Integrity servers Version 8.4 is supported as a guest operating system. Integrity VM is a soft partitioning and virtualization technology within the HP Virtual Server Environment, which enables you to create multiple virtual servers or machines with shared resources within a single HP Integrity server or a hardware partition (nPartition).

Each virtual machine hosts its own "guest" operating system instance, applications, and users. On HP Integrity servers, the Integrity VM Host runs on HP-UX, while OpenVMS runs as a guest operating system.

This article provides information on some of the common tools used for Troubleshooting or Debugging OpenVMS Guests. The article describes basic configurations, common failure scenarios, and tips on how to handle them.

The following table lists the requirements for installing VM Host system for Integrity VM and OpenVMS guest on VM Host System.

This section describes the installation and verification process for Integrity VM.

For HP-UX 11iv3 March 2010 installation, see the HP-UX 11i v3 Installation and Update Guide: http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02023874/c02023874.pdf

Integrity VM Version 4.2 does not support hyperthreading. To disable the hyperthreading, use the following command:

# /usr/sbin/setboot -m off
# reboot

Or 

SHELL> cpuconfig threads off
SHELL> reset

Note: Be sure that the hyperthreads are disabled. Use the following command to check if the hyperthreads are enabled:

# getconf SC_HT_ENABLED 

1 indicates hyperthreading is enabled
0 indicates hyperthreading is disabled

Or

SHELL> cpuconfig threads 

To install Integrity VM, enter the swinstall command (assuming the kit is at /var/depots).

# swinstall -x autoreboot=true -s /var/depots/hpvm4.2.depot T2767CC 

To verify that Integrity VM is installed successfully, enter the following command:

# hpvminfo
hpvminfo: Running on an HPVM host. 

Note: To see which version of specific bundles is installed, enter the swlist command. Specific version numbers on your installation might not match with the examples described in this document.

# swlist | grep Integrity
 
T2767CC            B.04.20       Integrity VM
T8718AC            B.04.20       Integrity VM Online Migration Software
VMGuestLib         B.04.20       Integrity VM Guest Support Libraries
VMGuestSW          B.04.20       Integrity VM Guest Support Software
VMKernelSW         B.04.20       Integrity VM Kernel Software
VMProvider         B.04.20       WBEM Provider for Integrity VM

To manually start and stop Integrity VM, use the following commands:

# /sbin/init.d/hpvm start
# /sbin/init.d/hpvm stop

This section describes about the failure scenarios while installing or configuring OpenVMS guest, and provides troubleshooting tips. 

# hpvmstart –P OVMSG1
hpvmstart: HPVM currently not running.
hpvmstart: Unable to continue.

The error indicates Integrity VM is not running. To start Integrity VM, use the following command:

# /sbin/init.d/hpvm start

# hpvmmodify -P OVMSG1 -a disk:scsi::disk:/dev/rdisk/disk4

HPVM guest OVMSG1 configuration problems:
           Warning 1 on item /dev/rdisk/disk4: Devices on OpenVMS guests must use the avio_stor adapter. These problems might prevent the HPVM guest OVMSG1 from starting.
hpvmmodify: The modification process is continuing.
 

# hpvmmodify -P OVMSG1 -a network:lan::vswitch:myswitch

HPVM guest OVMSG1 configuration problems:
         Warning 1 on item myswitch: Devices on OpenVMS guests must use the avio_lan adapter.
These problems might prevent the HPVM guest OVMSG1 from starting.
hpvmmodify: The modification process is continuing.

These warnings indicate that OpenVMS guest supports only the Accelerated Virtual I/O (AVIO) interface. Integrity VM commands enables you to configure VIO devices to a guest and these devices might not give any errors during the startup. However, the VIO devices are not part of the supported configuration of a guest running on an OpenVMS operating system.

Delete the virtual switch (vswitch) and the disk with the SCSI/LAN Adapter and add the vswitch and disk with an AVIO adapter to resolve the issue as mentioned in the following section:

# hpvmmodify -P OVMSG1 -d disk:scsi::disk:/dev/rdisk/disk4
# hpvmmodify -P OVMSG1 -d network:lan::vswitch:myswitch
# hpvmmodify -P OVMSG1 -a disk:avio_stor::disk:/dev/rdisk/disk4
# hpvmmodify -P OVMSG1 -a network:avio_lan::vswitch:myswitch

Cannot open VM
/var/opt/hpvm/uuids/6b3e85bc-334b-11df-8701-00306ef445ed/vm_dev: (12) Not enough space hpvmstart: ERROR (OVMSG1): Unable to start guest 'OVMSG1'.
hpvmstart: Unable to start guest 'OVMSG1'.
hpvmstart: Unable to continue.

This error occurs on any ‘basic’ error in getting the guest software (monitor) to load or Host hardware does not support to run OpenVMS guest.

You can use the –s option with the hpvmstart command to find the configuration issue. The -s option examines the specified guest configuration and reports any errors or warnings that are preventing it from starting. The guest is not started.

For example:

# hpvmstart -s -P OVMSG1

HPVM guest OVMSG1 configuration problems:
Warning 1: OpenVMS guests not supported on this host's CPU type (Madison).
These problems might prevent the HPVM guest OVMSG1 from starting.
hpvmstart: Unable to continue.

Add virtual switch for guest OVMSG1:

# hpvmmodify –P OVMSG1 –a network:avio_lan::vswitch:Myswitch

The problem is with a virtual switch. The following section provides some information about the virtual switches:

# hpvmnet

Name     Number  State  Mode     PPA    MACAddress   IP Address

======== ====== ======= ========= ====== ============== ===============

localnet   1    Up    Shared N/A             N/A
myswitch   2    Up    Shared lan1 0x001a4b087d231 11.11.111.111

It looks like the vswitch named Myswitch does not exist (though myswitch does). Virtual switch names are case sensitive, so it is often a common mistake to switch upper and lowercase letters in a vswitch name.

• Check the log files:
• Check guest’s log and command.log for anomalies
• Check the performance of the virtual switch:
• Examine output of the hpvmnet –S <vswitch-name> -V
• Run it twice, counters are reset between execution

For example:
a. Looking at myswitch information:
# hpvmnet -V -S myswitch
Name : myswitch
.
.
.
Packets out to stream   : 5991
Packets out to guest    : 2455
Packets dropped         : 54
Broadcasts              : 4310

.
.
.

b. Run hpvmnet again to reset the counters and see if any packets are going out to stream:

# hpvmnet -V -S myswitch
Name : myswitch
.
.
.

Packets out to stream   : 0
Packets out to guest    : 235
Packets dropped         : 0
Broadcasts              : 235

.
.
.

c. The packets in and packets out to guest values should be greater than zero, this indicates the current activity on the vswitch. If it is greater then zero, then the vswitch is operational.

You can notice that nothing is going out to stream on that interface. Check the physical interface on the host. Run the hpvmnet command to find about the physical point of attachment (PPA).

# hpvmnet

Name   Number State  Mode   Name PPA  MAC Address  IPv4 Address

======== =========== ======= ======== =========== ==============

localnet 1 Up   Shared     N/A          N/A

myswitch 2 Up   Shared   lan1   0x001a4b087d23 11.11.111.111

The output indicates that the PPA is lan1.

# ifconfig lan1
lan1: flags=843<BROADCAST,RUNNING,MULTICAST>
inet 11.11.111.111 netmask fffff800 broadcast 11.11.111.255

This indicates the PPA is down. If the state is UP, the output will be:

# ifconfig lan1
lan1: flags=843<UP,BROADCAST,RUNNING,MULTICAST>
inet 11.11.111.111 netmask fffff800 broadcast 11.11.111.255

d. First restart the vswitch, because any changes to the PPA are reflected in the vswitch.
When you restart the e PPA state is automatically set to UP.

# hpvmnet -r -S myswitch
hpvmnet: Restart the vswitch 'myswitch'? [n]: y

You might see the following message while loading licenses in guest node, if it is a cluster member:

%LICENSE-I-LICIGN, 'product' 'authorization' License Ignored Not Virtual

There is a new /VIRTUAL qualifier to load Per Core License (PCL) licenses in a guest environment when the guest node is a cluster member. Modify the licenses using the /VIRTUAL qualifier and reload it. The guest node will be able join as a cluster member after reloading the licenses.

$ LICENSE MODIFY ‘product’ /VIRTUAL
$ LICENSE LOAD ‘product’

• Use Ctrl/P to generate the OpenVMS crash dump.

Or

• Use Ctrl/B to enter the virtual console. Set tunables and enter the tc command to reset the guest at CM.
  
  vMP:CM> say 0xb reco
  vMP:CM> say 256 rnum
  VMP:CM> say 1 dump
  VMP:CM> say 1 dump
  vMP:CM> tc
  
  The tc command generates an OpenVMS crash dump and the vm.core file. The vm.core file can be found at /var/opt/hpvm/guests/<guest>.
  
  Collect the INIT log after restart:
  
  vMP:CM> ed –init
  
  Alternatively, you can use the hpvmconsole command to pass commands to a guest console from the Host command line. For example, to force an INIT on a guest, use the following command:
  
  # hpvmconsole –P OVMSG1 –q –c tc –fi
  
  From the Host command line, you can save a copy of the INIT log:
  
  # hpvmconsole –P OVMSG1 –q –c 'ed –init' > /tmp/tc.log

If a Host hangs or guest crashes, output of the following form is displayed on the guest’s virtual console:

                              *** A fatal error has occurred -- VM terminated ***
            **** Dumping Guest Image ****
            **** Done with dump (nnnnnKbytes) ****

• See the HPVM monitor log, located on the VM Host in the /var/opt/hpvm/common/hpvm_mon_log.
• After the guest reboots, use the hpvmcollect command to collect the information and log files. Report the information through your support channel.
• To collect logs, use the following commands:
  
  # hpvmconsole –P OVMSG1 –q –c cl > cons.log
  # hpvmconsole –P OVMSG1 –q –c ‘ed –init’ > init.log
  # hpvmconsole –P OVMSG1 –q –c ‘ed –mca’ > mca.log

hpvmcollect

• The hpvmcollect command on the VM Host or on the guest to collect Integrity VM information that is useful in analyzing system problems.
• The hpvmcollect command collects log files, system status, device information, system and Integrity Virtual Machines configuration, guest information, and crash dumps.
• By default, the hpvmcollect command creates a directory called hpvmcollect_archive in your current directory, and copies all the Integrity VM and VM Host information.
  For example, to gather information for a guest named OVMSG1 on the VM Host, enter the following command:
  
  # hpvmcollect -P OVMSG1
  
  This command creates a directory called hpvmcollect_archive in your current directory (if it does not already exist) and then collects information about the VM Host crash dump. The information is then put into a tar file format (if there is a crash dump) or tar.gz file format (if there is no crash dump).
  
  Note: Do not modify the guest configuration before running the hpvmcollect command.

• Check the guest status, using the following command:
  
  # hpvmstatus -P OVMSG1
   

• Check the guest log from the virtual console:
  
  vMP> rec –view
   
• If any of the following message is displayed, it is an HPVM problem:
  • “Guest punishment”
  • “VMM panic”
  • “Assertion failed”
• If the log says nothing about stopping the virtual machine, look for a tombstone file in the guest machine directory:
  
  # ll /var/opt/hpvm/guests/OVMSG1/tombstone
  # ll /var/opt/hpvm/guests/OVMSG1/vm.core
   
• Collect the information using the hpvmcollect command. Report the information through your support channel.

• See the Integrity VM and OpenVMS Release Notes for known issues and limitations.
• See the Integrity VM Release Notes for the latest patch required by the Integrity VM.
• See the /var/adm/syslog/syslog.log file for the Scheduler and virtual switch issues.
  (Search for the ‘hpvmdvr’, ‘hpvmntdvr’, ‘hpvmnetd’, ‘vm_fssagt’, ‘hpvmmonlogd’, and ‘hpvmamrd’ strings.)
• See the Global Virtual Machine Monitor log file for errors.
  
  # tail –f /var/opt/hpvm/common/hpvm_mon_log
   
• To change the size of Virtual Machine Monitor log file:
  
  - Use the ch_rc command to change the file size:
  # ch_rc –a –p VMMLOGSIZE=4096
  - Kill the monitor log daemon (it respawns):
  # kill -HUP `cat /var/run/hpvmmonlogd.pid`
  (/etc/rc.config.d/hpvmconf VMMLOGSIZE=1024kb by default)
   
• See the guest log (/var/opt/hpvm/guests/<guest>/log) and guest console log(/var/opt/hpvm/guests/<guest>/console/conslog) files for errors.
• Collect other logs with the hpmvoncolse command for analysis:
  
  # hpvmconsole –P OVMSG1 –q –c cl > cons.log
  # hpvmconsole –P OVMSG1 –q –c ‘ed –init’ > init.log
  # hpvmconsole –P OVMSG1 –q –c ‘ed –mca’ > mca.log
  # hpvmconsole –P OVMSG1 –q –c ‘rec –view’ > op.log
   
• See the Virtual firmware logging, Forward Progress Log, and System Event Log
  
  # /usr/sbin/diag/contrib/slview –p 0 –f FPL
  # /usr/sbin/diag/contrib/slview –p 0 –f SEL
  # hpvmconsole -P OVMSG1
  [OVMSG1] vMP> SL
   
• Collect logs using the hpvmcollect command.

This section explains about the debugging and monitoring tools.

The default scheduler logging level is set to level 3. Messages appear in the /var/adm/syslog/syslog.log file.

For detailed scheduler logging, set log level to 7 (extremely verbose) or 6 (usually sufficient). To set the log level to 7, use the following command:

# vm_fssagt –l 7

Message description for FSS are given in the following table:

If FSS is constantly in a STRESS mode, it indicates too much system load. Occasional STRESS messages in the Scheduler logging are OK. Check the state of the Scheduler using the following command:

# hpvmstatus -S

If the state is down, use the following command to collect the logs:

# grep vm_fssagt /var/adm/syslog/syslog.log

System performance monitoring

Use the following commands and tools to view the system's performance:

• hpvmstatus
• hpvmsar
• ps and top
• glance/glanceplus

hpvmstatus

• The hpvmstatus command displays information about the operational state and the virtual hardware configuration of the virtual machines on the VM Host.

• The hpvmstatus –s command gives the information about the resource utilization within the Host.

The important options available with the hpvmstatus command are given in the following table:

Option	Displays
-e	Displays the event log for the VM Host or the specified virtual machine. The event log records all the configuration changes of a virtual machine.
-i	Prints statistics collected by the monitor. Currently, these include vCPU percentage and durations over the lifetime of the guest.
-r	Displays the CPU entitlement information for the virtual machines.
-d	Displays the devices on the specified virtual machine.
-V	Displays detailed information (verbose mode) about the virtual machines.
-s	Displays server system resources.
-S	Displays the mode the schedule is in.

hpvmsar

• The hpvmsar command is similar to the Monitor utility on an OpenVMS. The hpvmsar command displays the guest CPU utilization with statistics like Busy time, Idle time, Wait time and Host time. The field description are given in the following table:
   

  Field name	Field description
  Busy time	Counts whenever the guest runs.
  Idle time	Counts when the guest has no activity and the CPU is returned to the Host.
  Wait time	Counts when the guest is pre-empted, but has some activity pending.
  Host time	Counts when the Host is running, as seen from the guest's point of view.



• With the -a option, the hpvmsar command displays information about all the running guests.
• With the -A option, the hpvmsar command displays information about all the guests, whether they are running or not.
• Collecting data: Use the following command to collect data for 2 seconds for all the running guest:
  
  # hpvmsar –s 2 -a

ps and top

• Each VM is manifested as a UNIX process running on the VM Host, the physical resources including CPU, I/O, and so on, consumed by a given VM, can be identified by monitoring the process associated with that VM.
• These processes have the executable name hpvmapp and typically have the option –d whose argument name is the name of the VM.
  For example, the process with the command ‘hpvmapp –d OVMSG1’ corresponds to the virtual machine named ‘OVMSG1.’
• Tools such as the ps and top can be used on the VM Host to monitor a virtual machine by identifying the Process ID (PID) for a given VM. 
  For example, the PID for some VM can be identified from the output of ‘ps –fu root | grep hpvmapp’ and then used with top to identify the resources being consumed by that VM.

glance/GlancePlus

• The glance tool gives information about the system utilization including CPU, memory, disk and so on.
• It consists of two components:
• Motif-based program - "gpm"
• Character mode program -"glance"
• You can use the glance command on the Host:
  
  # glance
  
  Note: The glance tool will automatically start and update the data at fixed frequency.
   
• glance –V provides configured and current entitlements, logical and physical CPU usage, and guest uptime for all the guests.

The important options available with the glance tool are given in the following table:

hpvmdevinfo and hpvmdevmgmt

• To display the devices information and manage devices, use the hpvmdevinfo and hpvmdevmgmt commands.
• The hpvmdevinfo command displays the information about the storage devices assigned to a virtual machine. 
• Use the hpvmdevinfo command to find mappings between the virtual disks and the physical disks.
• Use the hpvmdevmgmt command to manage the devices that are associated with the VM Host and the guests.
• Check the device database. Use the following command to see if the virtual devices you created are listed. If they are not listed, you must create them again.
  
  # hpvmdevmgmt –l all
   
• To check all the devices associated with the particular guest (OVMSG1), use the following command:
  
  # hpvmdevmgmt –l gdev:depend:OVMSG1
   
• Restricted devices cannot be associated with any guest. To check all the restricted devices, use the following command:
  
  # hpvmdevmgmt –l rdev
   
• To check all the guest devices, use the following command:
  
  # hpvmdevmgmt –l gdev

dmesg

• To collect system diagnostic messages to form error log.
• To verify if the hpvmdvr and hpvmntdvr drivers are loaded with no errors:
  • The hpvmdvr driver is used for communicating between the HPVM components.
  • The hpvmntdvr driver is used for communicating between the HPVM networking components and the guest.
• The dmesg command is used with cron to produce the error log /var/adm/messages. Use the following command:
  
  # /usr/sbin/dmesg - >> /var/adm/message
  Every 10 minutes

HP Integrity Virtual Machines 4.2: Installation, Configuration, and Administration Guide: http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02023903/c02023903.pdf

HP Integrity Virtual Machine 4.2.5: Release Notes:
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02545643/c02545643.pdf

HP OpenVMS Version 8.4 Upgrade and Installation Manual:
http://h71000.www7.hp.com/doc/84final/ba322_90087/ba322_90087.pdf

HP OpenVMS Version 8.4 New Features and Documentation Overview:
http://h71000.www7.hp.com/doc/84final/6679/ba322_90088.pdf

HP OpenVMS Version 8.4 Release Notes:
http://h71000.www7.hp.com/doc/84final/6677/ba322_90089.pdf