 |
Guidelines for OpenVMS Cluster Configurations
7.7 Setting Up a Storage Controller for Use with OpenVMS
The HP storage array controllers and the manuals that provide specific
information for configuring them for use with OpenVMS follow:
- HSG60/80
HSG80 ACS Solution Software Version 8.6 for Compaq
OpenVMS, order number AA-RH4BD-TE.
This manual is available at
the following location:
ftp://ftp.compaq.com/pub/products/storageworks/techdoc/raidstorage/AA-RH4BD-TE.pdf
|
- Enterprise Virtual Array
OpenVMS Kit V2.0 for Enterprise
Virtual Array Installation and Configuration Guide, order number
AA-RR03B-TE.
This manual is available at the following location:
ftp://ftp.compaq.com/pub/products/storageworks/techdoc/enterprise/AA-RR03B-TE.pdf
|
7.7.1 Setting Up the Device Identifier for the CCL
Defining a unique device identifier for the Command Console LUN (CCL)
of the HSG and HSV is not required by OpenVMS, but it may be required
by some management tools. OpenVMS suggests that you always define a
unique device identifier since this identifier causes the creation of a
CCL device visible using the SHOW DEVICE command. Although this device
is not directly controllable on OpenVMS, you can display the multiple
paths to the storage controller using the SHOW DEVICE/FULL command, and
diagnose failed paths, as shown in the following example for $1$GGA3,
where one of the two paths has failed.
Paver> sh dev gg /mul
Device Device Error Current
Name Status Count Paths path
$1$GGA1: Online 0 2/ 2 PGA0.5000-1FE1-0011-AF08
$1$GGA3: Online 0 1/ 2 PGA0.5000-1FE1-0011-B158
$1$GGA4: Online 0 2/ 2 PGA0.5000-1FE1-0015-2C58
$1$GGA5: Online 0 2/ 2 PGA0.5000-1FE1-0015-22A8
$1$GGA6: Online 0 2/ 2 PGA0.5000-1FE1-0015-2D18
$1$GGA7: Online 0 2/ 2 PGA0.5000-1FE1-0015-2D08
$1$GGA9: Online 0 2/ 2 PGA0.5000-1FE1-0007-04E3
Paver> show dev /full $1$gga9:
Device $1$GGA9:, device type Generic SCSI device, is online, shareable, device
has multiple I/O paths.
Error count 0 Operations completed 0
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G:RWPL,W:RWPL
Reference count 0 Default buffer size 0
WWID 02000008:5000-1FE1-0007-04E0
I/O paths to device 2
Path PGA0.5000-1FE1-0007-04E3 (PAVER), primary path, current path.
Error count 0 Operations completed 0
Path PGA0.5000-1FE1-0007-04E1 (PAVER).
Error count 0 Operations completed 0
|
7.7.2 Setting Up the Device Identifier for Disk Devices
The device identifier for disks is appended to the string $1$DGA to
form the complete device name. It is essential that all disks have
unique device identifiers within a cluster. Device identifiers can be
between 0 and 32767, except a device identifier of 0 is not valid on
the HSV. Device identifiers greater than 9999 cannot be MSCP served to
other systems.
7.8 Creating a Cluster with a Shared FC System Disk
To configure nodes in an OpenVMS Cluster system, you must execute the
CLUSTER_CONFIG.COM (or CLUSTER_CONFIG_LAN.COM) command procedure. (You
can run either the full version, which provides more information about
most prompts, or the brief version.)
For the purposes of CLUSTER_CONFIG, a shared Fibre Channel (FC) bus is
treated like a shared SCSI bus, except that the allocation class
parameters do not apply to FC. The rules for setting node allocation
class and port allocation class values remain in effect when parallel
SCSI storage devices are present in a configuration that includes FC
storage devices.
To configure a new OpenVMS Cluster system, you must first enable
clustering on a single, or standalone, system. Then you can add
additional nodes to the cluster.
Example 7-5 shows how to enable clustering using brief version of
CLUSTER_CONFIG_LAN.COM on a standalone node called FCNOD1. At the end
of the procedure, FCNOD1 reboots and forms a one-node cluster.
Example 7-6 shows how to run the brief version of
CLUSTER_CONFIG_LAN.COM on FCNOD1 to add a second node, called FCNOD2,
to form a two-node cluster. At the end of the procedure, the cluster is
configured to allow FCNOD2 to boot off the same FC system disk as
FCNOD1.
The following steps are common to both examples:
- Select the default option [1] for ADD.
- Answer Yes when CLUSTER_CONFIG_LAN.COM asks whether there will be a
shared SCSI bus. SCSI in this context refers to FC as well as to
parallel SCSI.
The allocation class parameters are not affected by
the presence of FC.
- Answer No when the procedure asks whether the node will be a
satellite.
| Example 7-5 Enabling Clustering on a
Standalone FC Node |
$ @CLUSTER_CONFIG_LAN BRIEF
Cluster Configuration Procedure
Executing on an Alpha System
DECnet Phase IV is installed on this node.
The LAN, not DECnet, will be used for MOP downline loading.
This Alpha node is not currently a cluster member
MAIN MENU
1. ADD FCNOD1 to existing cluster, or form a new cluster.
2. MAKE a directory structure for a new root on a system disk.
3. DELETE a root from a system disk.
4. EXIT from this procedure.
Enter choice [1]: 1
Is the node to be a clustered node with a shared SCSI or Fibre Channel bus (Y/N)? Y
Note:
Every cluster node must have a direct connection to every other
node in the cluster. Since FCNOD1 will be a clustered node with
a shared SCSI or FC bus, and Memory Channel, CI, and DSSI are not present,
the LAN will be used for cluster communication.
Enter this cluster's group number: 511
Enter this cluster's password:
Re-enter this cluster's password for verification:
Will FCNOD1 be a boot server [Y]? Y
Verifying LAN adapters in LANACP database...
Updating LANACP LAN server process volatile and permanent databases...
Note: The LANACP LAN server process will be used by FCNOD1 for boot
serving satellites. The following LAN devices have been found:
Verifying LAN adapters in LANACP database...
LAN TYPE ADAPTER NAME SERVICE STATUS
======== ============ ==============
Ethernet EWA0 ENABLED
CAUTION: If you do not define port allocation classes later in this
procedure for shared SCSI buses, all nodes sharing a SCSI bus
must have the same non-zero ALLOCLASS value. If multiple
nodes connect to a shared SCSI bus without the same allocation
class for the bus, system booting will halt due to the error or
IO AUTOCONFIGURE after boot will keep the bus offline.
Enter a value for FCNOD1's ALLOCLASS parameter [0]: 5
Does this cluster contain a quorum disk [N]? N
Each shared SCSI bus must have a positive allocation class value. A shared
bus uses a PK adapter. A private bus may use: PK, DR, DV.
When adding a node with SCSI-based cluster communications, the shared
SCSI port allocation classes may be established in SYS$DEVICES.DAT.
Otherwise, the system's disk allocation class will apply.
A private SCSI bus need not have an entry in SYS$DEVICES.DAT. If it has an
entry, its entry may assign any legitimate port allocation class value:
n where n = a positive integer, 1 to 32767 inclusive
0 no port allocation class and disk allocation class does not apply
-1 system's disk allocation class applies (system parameter ALLOCLASS)
When modifying port allocation classes, SYS$DEVICES.DAT must be updated
for all affected nodes, and then all affected nodes must be rebooted.
The following dialog will update SYS$DEVICES.DAT on FCNOD1.
There are currently no entries in SYS$DEVICES.DAT for FCNOD1.
After the next boot, any SCSI controller on FCNOD1 will use
FCNOD1's disk allocation class.
Assign port allocation class to which adapter [RETURN for none]: PKA
Port allocation class for PKA0: 10
Port Alloclass 10 Adapter FCNOD1$PKA
Assign port allocation class to which adapter [RETURN for none]: PKB
Port allocation class for PKB0: 20
Port Alloclass 10 Adapter FCNOD1$PKA
Port Alloclass 20 Adapter FCNOD1$PKB
WARNING: FCNOD1 will be a voting cluster member. EXPECTED_VOTES for
this and every other cluster member should be adjusted at
a convenient time before a reboot. For complete instructions,
check the section on configuring a cluster in the "OpenVMS
Cluster Systems" manual.
Execute AUTOGEN to compute the SYSGEN parameters for your configuration
and reboot FCNOD1 with the new parameters. This is necessary before
FCNOD1 can become a cluster member.
Do you want to run AUTOGEN now [Y]? Y
Running AUTOGEN -- Please wait.
The system is shutting down to allow the system to boot with the
generated site-specific parameters and installed images.
The system will automatically reboot after the shutdown and the
upgrade will be complete.
|
| Example 7-6 Adding a Node to a Cluster with a
Shared FC System Disk |
$ @CLUSTER_CONFIG_LAN BRIEF
Cluster Configuration Procedure
Executing on an Alpha System
DECnet Phase IV is installed on this node.
The LAN, not DECnet, will be used for MOP downline loading.
FCNOD1 is an Alpha system and currently a member of a cluster
so the following functions can be performed:
MAIN MENU
1. ADD an Alpha node to the cluster.
2. REMOVE a node from the cluster.
3. CHANGE a cluster member's characteristics.
4. CREATE a duplicate system disk for FCNOD1.
5. MAKE a directory structure for a new root on a system disk.
6. DELETE a root from a system disk.
7. EXIT from this procedure.
Enter choice [1]: 1
This ADD function will add a new Alpha node to the cluster.
WARNING: If the node being added is a voting member, EXPECTED_VOTES for
every cluster member must be adjusted. For complete instructions
check the section on configuring a cluster in the "OpenVMS Cluster
Systems" manual.
CAUTION: If this cluster is running with multiple system disks and
common system files will be used, please, do not proceed
unless appropriate logical names are defined for cluster
common files in SYLOGICALS.COM. For instructions, refer to
the "OpenVMS Cluster Systems" manual.
Is the node to be a clustered node with a shared SCSI or Fibre Channel bus (Y/N)? Y
Will the node be a satellite [Y]? N
What is the node's SCS node name? FCNOD2
What is the node's SCSSYSTEMID number? 19.111
NOTE: 19.111 equates to an SCSSYSTEMID of 19567
Will FCNOD2 be a boot server [Y]? Y
What is the device name for FCNOD2's system root
[default DISK$V72_SSB:]? Y
What is the name of FCNOD2's system root [SYS10]?
Creating directory tree SYS10 ...
System root SYS10 created
CAUTION: If you do not define port allocation classes later in this
procedure for shared SCSI buses, all nodes sharing a SCSI bus
must have the same non-zero ALLOCLASS value. If multiple
nodes connect to a shared SCSI bus without the same allocation
class for the bus, system booting will halt due to the error or
IO AUTOCONFIGURE after boot will keep the bus offline.
Enter a value for FCNOD2's ALLOCLASS parameter [5]:
Does this cluster contain a quorum disk [N]? N
Size of pagefile for FCNOD2 [RETURN for AUTOGEN sizing]?
A temporary pagefile will be created until resizing by AUTOGEN. The
default size below is arbitrary and may or may not be appropriate.
Size of temporary pagefile [10000]?
Size of swap file for FCNOD2 [RETURN for AUTOGEN sizing]?
A temporary swap file will be created until resizing by AUTOGEN. The
default size below is arbitrary and may or may not be appropriate.
Size of temporary swap file [8000]?
Each shared SCSI bus must have a positive allocation class value. A shared
bus uses a PK adapter. A private bus may use: PK, DR, DV.
When adding a node with SCSI-based cluster communications, the shared
SCSI port allocation classes may be established in SYS$DEVICES.DAT.
Otherwise, the system's disk allocation class will apply.
A private SCSI bus need not have an entry in SYS$DEVICES.DAT. If it has an
entry, its entry may assign any legitimate port allocation class value:
n where n = a positive integer, 1 to 32767 inclusive
0 no port allocation class and disk allocation class does not apply
-1 system's disk allocation class applies (system parameter ALLOCLASS)
When modifying port allocation classes, SYS$DEVICES.DAT must be updated
for all affected nodes, and then all affected nodes must be rebooted.
The following dialog will update SYS$DEVICES.DAT on FCNOD2.
Enter [RETURN] to continue:
$20$DKA400:<VMS$COMMON.SYSEXE>SYS$DEVICES.DAT;1 contains port
allocation classes for FCNOD2. After the next boot, any SCSI
controller not assigned in SYS$DEVICES.DAT will use FCNOD2's
disk allocation class.
Assign port allocation class to which adapter [RETURN for none]: PKA
Port allocation class for PKA0: 11
Port Alloclass 11 Adapter FCNOD2$PKA
Assign port allocation class to which adapter [RETURN for none]: PKB
Port allocation class for PKB0: 20
Port Alloclass 11 Adapter FCNOD2$PKA
Port Alloclass 20 Adapter FCNOD2$PKB
Assign port allocation class to which adapter [RETURN for none]:
WARNING: FCNOD2 must be rebooted to make port allocation class
specifications in SYS$DEVICES.DAT take effect.
Will a disk local only to FCNOD2 (and not accessible at this time to FCNOD1)
be used for paging and swapping (Y/N)? N
If you specify a device other than DISK$V72_SSB: for FCNOD2's
page and swap files, this procedure will create PAGEFILE_FCNOD2.SYS
and SWAPFILE_FCNOD2.SYS in the [SYSEXE] directory on the device you
specify.
What is the device name for the page and swap files [DISK$V72_SSB:]?
%SYSGEN-I-CREATED, $20$DKA400:[SYS10.SYSEXE]PAGEFILE.SYS;1 created
%SYSGEN-I-CREATED, $20$DKA400:[SYS10.SYSEXE]SWAPFILE.SYS;1 created
The configuration procedure has completed successfully.
FCNOD2 has been configured to join the cluster.
The first time FCNOD2 boots, NETCONFIG.COM and
AUTOGEN.COM will run automatically.
|
7.9 Online Reconfiguration
The FC interconnect can be reconfigured while the hosts are running
OpenVMS. This includes the ability to:
- Add, move, or remove FC switches and HSGs.
- Add, move, or remove HSG virtual disk units.
- Change the device identifier or LUN value of the HSG virtual disk
units.
- Disconnect and reconnect FC cables. Reconnection can be to the same
or different adapters, switch ports, or HSG ports.
OpenVMS does not automatically detect most FC reconfigurations. You
must use the following procedure to safely perform an FC
reconfiguration, and to ensure that OpenVMS has adjusted its internal
data structures to match the new state:
- Dismount all disks that are involved in the reconfiguration.
- Perform the reconfiguration.
- Enter the following commands on each host that is connected to the
Fibre Channel:
SYSMAN> IO SCSI_PATH_VERIFY
SYSMAN> IO AUTOCONFIGURE
|
The purpose of the SCSI_PATH_VERIFY command is to check each FC path in
the system's IO database to determine whether the attached device has
been changed. If a device change is detected, then the FC path is
disconnected in the IO database. This allows the path to be
reconfigured for a new device by using the IO AUTOCONFIGURE command.
Note
In the current release, the SCSI_PATH_VERIFY command only operates on
FC disk devices. It does not operate on generic FC devices, such as the
HSG command console LUN (CCL). (Generic FC devices have names such as
$1$GGAnnnnn.
This means that once the CCL of an HSG has been configured by OpenVMS
with a particular device identifier, its device identifier should not
be changed.
|
7.10 HSG Host Connection Table and Devices Not Configured
When a Fibre Channel host bus adapter is connected (through a Fibre
Channel switch) to an HSG controller, the HSG controller creates an
entry in the HSG connection table. There is a separate connection for
each host bus adapter, and for each HSG port to which the adapter is
connected. (Refer to the HSG CLI command SHOW CONNECTIONS for more
information.)
Once an HSG connection exists, you can modify its parameters by using
commands that are described in the HSG Array Controller ACS
Configuration and CLI Reference Guide. Since a connection can be
modified, the HSG does not delete connection information from the table
when a host bus adapter is disconnected. Instead, when the user is done
with a connection, the user must explicitly delete the connection using
a CLI command.
The HSG controller supports a limited number of connections: ACS V8.5
allows a maximum of 64 connections and ACS V8.4 allows a maximum of 32
connections. The connection limit is the same for both single- and
dual-redundant controllers. Once the maximum number of connections is
reached, then new connections will not be made. When this happens,
OpenVMS will not configure disk devices, or certain paths to disk
devices, on the HSG.
The solution to this problem is to delete old connections that are no
longer needed. However, if your Fibre Channel fabric is large and the
number of active connections exceeds the HSG limit, then you must
reconfigure the fabric or use FC switch zoning to "hide" some
adapters from some HSG ports to reduce the number of connections.
7.11 Using Interrupt Coalescing for I/O Performance Gains
Starting with OpenVMS Alpha Version 7.3-1, interrupt coalescing is
supported for the KGPSA host adapters and is off by default. Interrupt
coalescing can improve performance in environments with high I/O work
loads by enabling the adapter to reduce the number of interrupts seen
by a host. This feature is implemented in the KGPSA firmware.
You can read and modify the current settings for interrupt coalescing
by means of the Fibre Channel Control Program (FC$CP). You must have
the CMKRNL privilege to use FC$CP.
If you specify a response count and a delay time (in milliseconds) with
FC$CP, the adapter defers interrupting the host until that number of
responses is available or until that amount of time has passed,
whichever occurs first.
Interrupt coalescing may cause a performance degradation to an
application that does synchronous I/O. If no other I/O is going through
a given KGPSA, the latency for single writes is an average of 900
microseconds longer with interrupt coalescing enabled (or higher
depending on the selected response interval).
Interrupt coalescing is set on a per KGPSA basis. You should have an
average of at least 2000 I/Os per second through a given KGPSA before
enabling interrupt coalescing.
The format of the command is:
RUN SYS$ETC:FC$CP FGx enable-value [delay][response-count]
|
In this format:
- For FGx, the valid range of x is A to Z.
- enable-value is a bit mask, with bit 1 controlling
response coalescing and bit 0 controlling interrupt coalescing. The
possible decimal values are:
1=interrupt coalescing
2=response coalescing
3=interrupt coalescing and response coalescing
- delay (in milliseconds) can range from 0 to 255 decimal.
- response-count can range from 0 to 63 decimal.
- Any negative value leaves a parameter unchanged.
- Values returned are those that are current after any changes.
OpenVMS recommends the following settings for the FC$CP command:
$ RUN SYS$ETC:FC$CP FGx 2 1 8
|
7.12 Using Fast Path in Your Configuration
Fast Path support was introduced for Fibre Channel in OpenVMS Alpha
Version 7.3 and is enabled by default. It is designed for use in a
symmetric multiprocessor system (SMP). When Fast Path is enabled, the
I/O completion processing can occur on all the processors in the SMP
system instead of only on the primary CPU. Fast Path substantially
increases the potential I/O throughput on an SMP system, and helps to
prevent the primary CPU from becoming saturated.
You can manage Fast Path programmatically using Fast Path system
services. You can also manage Fast Path with DCL commands and by using
the system parameters FAST_PATH and FAST_PATH_PORTS. For more
information about using Fast Path, refer to the VMS I/O User's Reference Volume.
7.13 FIBRE_SCAN Utility for Displaying Device Information
FIBRE_SCAN.EXE, a utility introduced in OpenVMS Alpha Version 7.3-2,
displays information about all storage devices attached to Fibre
Channel on the system; both configured and nonconfigured devices are
included. The displayed information includes such data as the Fibre
Channel target and LUN values, the vendor and product ID, device type,
port and device worldwide identifiers (WWIDs), serial number, firmware
revision level, and port login state. While the program primarily
describes disk and tape devices, some limited information is also
displayed for controller and other generic ($n$GGAn) devices.
Note
FIBRE_SCAN can be used locally on each system. It cannot be used on
systems running versions prior to OpenVMS Version 7.3-2, nor can it be
used to display devices attached to other systems in a cluster.
|
FIBRE_SCAN can be invoked in two modes:
$ MCR SYS$ETC:FIBRE_SCAN ! Scans all ports on the Fibre Channel.
$ MCR SYS$ETC:FIBRE_SCAN PGx ! Scans only port x on the Fibre Channel.
|
FIBRE_SCAN requires CMKRNL and LOG_IO privilege.
To capture the FIBRE_SCAN output in a file, use a command such as the
following before invoking FIBRE_SCAN:
$ DEFINE/USER SYS$OUTPUT xxx.log
|
FIBRE_SCAN is a display-only utility and is not capable of loading
device drivers nor otherwise configuring devices on the Fibre Channel.
To configure devices, use the SYSMAN IO AUTOCONFIGURE command.
|
