Impact of the process quota on BACKUP performance

Shriniketan Bhagwat
shriniketan.bhagwat@hp.com



Overview

The disk is the media on which the critical data and files exist. VMS BACKUP utility is used to archive these files and critical data as part of DRP.

The files present in the disk are scattered around the disk and contains one or more extents. During archive, BACKUP reads these extents and puts them into the saveset file. For maximum performance during the archival operations, random blocks from a disk must be copied to a saveset in file order with the least amount of disk head movement. Ideally, BACKUP reads the whole disk sequentially, sort the blocks by file order and then sequentially writes them to the saveset. In order to do this, BACKUP needs the working set size which is larger than the disk size. Since this is not possible, BACKUP uses as many working set pages as it is allowed to (based on the UAF parameter WSQUOTA).

This article discusses the process quotas and limits that influence the performance of BACKUP operation. This also provides guidelines to calculate appropriate values for these parameters and how these could be set using SYSGEN parameters.

Guide lines for tuning BACKUP

Below guidelines can be considered while tuning the VMS BACKUP utility.

  • Understand the mechanisms that govern the performance of BACKUP and identify probable bottlenecks in its performance on the system.
  • Understand the parameters that affect the performance of these mechanisms and determine what changes to make in process quotas and system parameters to affect BACKUP performance.
  • Know how to set the value of a particular tuning parameter relative to other key parameters and limit the amount of trial and error necessary to optimize BACKUP's performance in the environment.
  • Know how to evaluate both, the impact of tuning on BACKUP's performance and the amount of system resources still available to the BACKUP process, to achieve an optimal set of BACKUP process and system quotas for the environment.
  • Understand how the throughput capabilities of a system's hardware can limit the effectiveness of BACKUP tuning efforts.

Mechanisms that govern the performance of BACKUP

BACKUP's internal mechanisms can be divided into nine stages. Each stage governs BACKUP's performance in a different fashion. These stages are:

  1. Initializing Data Structures
  2. Scanning Directories
  3. Opening multiple Files
  4. Mapping Extents
  5. Sorting mapped extents in LBN order
  6. Issuing multiple I/O requests to read sorted extents
  7. Post processing read I/Os
  8. Issuing write I/Os
  9. Waiting for Write I/Os

Stage 1 - Initializing Data Structures

The first stage within BACKUP's internal mechanisms is initializing the data structures. Five major operations affecting performance occur during this stage:

  1. If the user authorization file (UAF) parameter WSDEFAULT is not equal to WSQUOTA, BACKUP adjusts its working set size to WSQUOTA. This step involves expanding the working set list, which may involve shifting the page tables within the process header. This is required since BACKUP needs the working set size which is larger than the disk size to read the entire disk sequentially, sort the blocks by file order and then sequentially write them to the saveset.
  2. BACKUP partitions the available working set into control region and data region. The size of the data region is determined by the values of UAF parameters FILLM and WSQUOTA and SYSGEN parameter CHANNELCNT. This partition is required to avoid gross imbalances between the number of files that BACKUP can open and the amount of buffer space into which the opened files can be read.
  3. BACKUP partitions the data region into buffers whose size is based on the value of the BACKUP qualifier /BLOCK_SIZE. Unless this parameter is set to a value between 2048 and 65,024 bytes, it defaults to 8192 bytes for tape and 32,256 bytes for disk. BACKUP adjusts any user-supplied value down to the nearest page boundary.
  4. BACKUP allocates virtual address space for the resulting data buffers and their related buffer control blocks.
  5. BACKUP creates and populates the buffer lists. From the pool of buffers previously allocated, a portion of buffers, based on the value of BACKUP qualifier /GROUP_SIZE is set aside for use as XOR buffers. /GROUP_SIZE qualifier can be set between 0 and 100. The default value is 10. BACKUP inserts the balance of the buffers into a free-buffer list for use as data buffers.

The sizing of BACKUP's data structures, particularly the data buffer pool depends upon below three sources:


  1. SYSGEN parameters WSMAX and VIRTUALPAGECNT,
  2. UAF through parameters WSQUOTA and PGFLQUOTA,
  3. BACKUP qualifiers /GROUP_SIZE and /BLOCK_SIZE.

These sources limit the memory resources with which BACKUP works during its disk read operations.


First, WSMAX and WSQUOTA limit the overall memory available to the BACKUP Process. Second, VIRTUALPAGECNT and PGFLQUOTA indirectly limit the portion of this available memory that is allocated as data buffers. Exhaustion of either of these resources during the allocation of buffer space limits the data buffer pool to the number successfully allocated before the resource exhaustion occurred. Third, the /GROUP_SIZE qualifier affects the portion of these allocated buffers available for data. Finally, the /BLOCK_SIZE qualifier can affect the overall save-set size.


Stage 2 - Scanning Directories

The second stage within BACKUP's internal mechanisms is the scanning of volume's directories. During this stage, directories are successively opened as part of the traversal of the directory tree.


In the case of a save operation where the /IMAGE or /FAST qualifier is included on the BACKUP command line, this stage is preceded by a scan of the entire index file. During this scan, BACKUP tests each valid file header against the command line's full file selection criteria. Based on this evaluation, BACKUP creates a bitmap that identifies which file IDs found in the directory file entries correspond to files that should be saved.


The UAF parameter FILLM is the primary constraint upon this stage's operation. This parameter effectively limits the number of files, or equivalently, the size of the directory subtree, processed together. The smaller value of this parameter can result in smaller read scan.


Stage 3 – Opening multiple Files

During the third stage within BACKUP's internal mechanisms, BACKUP opens qualifying files using cathedral windows that comprise multiple window control blocks (WCBs). The purpose of a cathedral window is to allow the VMS file system to hold all mapping retrieval pointers in memory simultaneously.


The SYSGEN parameter CHANNELCNT and the UAF parameters BYTLM, ENQLM, and BIOLM affect BACKUP's performance during this stage. CHANNELCNT and BYTLM limit the number of files BACKUP opens at one time. File open continues until one of these two resources exhausted. BYTLM can also indirectly limit the size of a file's window. On repeated failure to open a file because of inadequate remaining BYTLM, a file is opened with a default window whose size depends on the value of the SYSGEN parameter ACP_WINDOW. This usually results in window turns during the extent mapping stage resulting in the performance impact. ENQLM and BIOLM limit the rate of file opening.


Stage 4 - Mapping file Extents

Mapping file extents is the fourth stage within BACKUP's internal mechanisms. During this stage, BACKUP uses the extent retrieval pointers found in the file headers to map the opened files onto the data buffers. For each extent, BACKUP creates virtual virtual map (VVM) data structures to control the corresponding read I/O issued in a subsequent stage.


The values established in Stage 1 for both buffer size and count largely determines the performance during this stage. Buffer count limits the portion of opened files that are mapped. As long as free buffers remain, BACKUP continues to open files till the ability of BYTLM to handle their windows, and to map their extents. Buffer size sets an upper boundary on the extent size readable in a single I/O. Extents larger than the buffer size are split across multiple buffers. Extents smaller than the buffer size may also be split to complete nearly full buffers that have inadequate space for a complete extent. Split extents are mapped with multiple VVMs.


Stage 5 - Sorting mapped extents in LBN order

The fifth stage within BACKUP's internal mechanisms involves sorting the extent mappings as found in the VVMs. In the previous stage, BACKUP has built the list of VVMs in file-by-file and virtual block number (VBN) order. This is the order in which BACKUP saves the directory tree on its output device.


VVM count is the primary constraint on the performance of the sorting and re-linking operation. Small buffers or small extents can combine with large file and buffer counts to produce a large sort. As with most sorts, the larger the sort, the lower its efficiency. However, the larger the list of VVMs, the greater the efficiency of the disk scan. In this performance trade-off, large scans results in increased sort time, usually produce better overall BACKUP performance.


Stage 6 - Issuing multiple I/O requests to read sorted extents

The sixth stage within BACKUP's internal mechanisms is the asynchronous issuing of concurrent read I/Os. A separate read I/O request is issued for each extent or extent fragment mapped by a single VVM. These I/Os are issued in LBN order using the VVM links established in the prior stage. The result is a sequential scan of that portion of the volume containing the entire group of mapped files.


The performance of the actual data transfer from the disk is constrained by the value specified in the /IO_LOAD qualifier, the UAF parameters ASTLM and the VVM count produced during Stage 4 mapping of extents. The /IO_LOAD qualifier controls the number of simultaneous I/Os issued by the BACKUP. The default value of /IO_LOAD qualifier is 8. /IO_LOAD qualifier can be used to increase or decrease the number of simultaneous I/Os issued by the BACKUP. The /IO_LOAD qualifier value and ASTLM parameters limit the rate at which read I/Os are queued. The larger value of /IO_LOAD qualifier causes I/O subsystem flood with large number of I/Os resulting decrease in overall BACKUP performance.


Stage 7 – Post processing read I/Os

Post processing the read I/Os is the seventh stage within BACKUP's internal mechanisms. Following completion of the asynchronous read I/O operations, BACKUP waits for read I/O completions in the file-by-file VBN order in which the files were originally mapped in Stage 4. As all I/Os for a particular file completes, BACKUP closes that file. As all I/Os for a particular buffer completes, BACKUP marks that buffer as ready to be written. No explicit constraints exist which directly control BACKUP's performance during this stage.


Stage 8 - Issuing write I/Os

During the eighth stage within BACKUP's internal mechanisms, BACKUP issues write I/Os for filled data buffers. Prior to actually writing the data buffers out, BACKUP performs XOR and CRC calculations. If the XOR group count is satisfied, the XOR buffer is also written. These write I/Os may be issued concurrently with completion of read I/Os. The rate at which filled data buffers are queued for writing depends on the BACKUP command qualifiers /GROUP_SIZE and /CRC, as well as on the UAF parameters ASTLM and DIOLM.


Because write I/Os may only be issued after all CRC or XOR calculations complete, the use of the /CRC and /GROUP_SIZE qualifiers does indirectly constrain performance. Use of these qualifiers provide the extra measure of data integrity. This stage's performance is constrained by the UAF parameters ASTLM and DIOLM. Exhaustion of either of these quotas will delay BACKUP's issuing of write I/Os.


Stage 9 - Waiting for Write I/Os

The ninth stage within BACKUP's internal mechanisms involves waiting for write I/O completion. After completion of all write I/Os BACKUP release the resources consumed.


Flow Chart


Primary Tuning Parameters

In almost every stage considered, the primary constraints on BACKUP's performance are memory and I/O related resources. Tuning BACKUP is basically a process of correctly sizing BACKUP's memory and I/O resources. Five parameters are required to accomplish most of this sizing: BACKUP qualifier /BLOCK_SIZE and /IO_LOAD and UAF parameters WSQUOTA, FILLM, and BYTLM.


The memory group of primary tuning parameters consists of WSQUOTA and BACKUP's /BLOCK_SIZE qualifier. WSQUOTA controls the number of pages allowed in the BACKUP process's working set. It affects the amount of memory available for data buffers and hence is one throttle on the efficiency of the disk read scans. The /BLOCK_SIZE qualifier controls the size of each of these buffers. Therefore, it affects the efficiency of disk and tape I/Os, as well as the relative size of the resulting save set. The value of the /BLOCK_SIZE qualifier should ideally exceed the average extent size on the volume. The value of WSQUOTA should be much less than or equal to the average number of free pages in the system.


The I/O group of primary tuning parameters consists of BACKUP's /IO_LOAD qualifier and the UAF parameters FILLM, and BYTLM. FILLM controls the number of files that BACKUP can open simultaneously. It limits the number of extents mapped and, like WSQUOTA, throttles the efficiency of disk read scans. BYTLM controls the amount of nonpaged pool available for file windows, also limits the number of files opened before BACKUP performs a disk read scan. /IO_LOAD limits the number of I/O operations that can be issued simultaneously, constrains the rate at which disk read /Os are performed.


Secondary Tuning Parameters

The BACKUP's secondary tuning parameters with respect to memory and I/O related resources are SYSGEN parameters WSMAX and CHANNELCNT and UAF parameters PGFLQUOTA, BIOLM, ENQLM, and ASTLM.


The memory group of secondary tuning parameters consists of the SYSGEN parameter WSMAX, as well as the UAF parameter PGFLQUOTA. All these parameters either directly or indirectly limit the portion of WSQUOTA that is ultimately allocated as data buffers in Stage 1 of BACKUP's performance mechanisms. Hence the value of these parameters should be more than that of WSQUOTA.


The I/O group of secondary tuning parameters consists of SYSGEN parameter CHANNELCNT and UAF parameters BIOLM, ENQLM, and ASTLM. CHANNELCNT limits the number of channels BACKUP may assign to the disk in Stage 3 of its performance mechanisms and therefore, the number of files it may open. ENQLM and BIOLM limit the rate of file opening during BACKUP's Stage 2 operations. Set the value of each of these parameters above that of FILLM.


Hardware Constraints in Tuning BACKUP

The hardware constraints that affect the BACKUP performance are CPU, tape drives and disk drives.

CPU: In Stage 8 of BACKUP's internal mechanisms, the rate at which BACKUP completes its CRC calculations affects the rate at which filled data buffers are written to disk and tape. Because the speed of the CRC calculation is hardware dependent, CPU may limit BACKUP's performance during this stage.


Tape and disk drive: The rate at which the data buffers handled by the tape and disk drives also impact the BACKUP elapsed time performance.


Impact of the process quotas

The SYSGEN parameter WSMAX controls the maximum value allowed for the UAF parameter WSQUOTA which in turn controls the number of buffers BACKUP is allowed to create and use. The number of input buffers BACKUP uses is WSQUOTA divided by any BLOCK_SIZE specified on the BACKUP command line (or the default BLOCK_SIZE of 8192 bytes for magnetic tape and 32,256 bytes for disk is used). To determine how many buffers BACKUP uses, issue a BACKUP/LIST command against an existing saveset. Note the "Buffer count" field in the saveset header information.


Ideally, WSMAX and WSQUOTA should be set to the number of free pages on a system just before the BACKUP operation is initiated. The DCL command SHOW MEMORY can be used to determine how many physical pages are free. If other processes will be running during the BACKUP operation, then consider a lower value.


The SYSGEN parameter CHANNELCNT limits the maximum number of files that a process can open simultaneously. For maximum BACKUP performance, CHANNELCNT should be set to the maximum value allowed. Since CHANNELCNT is a system-wide parameter, increasing its value will increase memory used by all processes. CHANNELCNT determines the number of channel control blocks available to the process.


Below are some guide lines for setting parameters and process quotas in order to maximize BACKUP's performance:


  1. Set the process quotas WSQUOTA and WSEXTENT equal to the SYSGEN parameter WSMAX. If WSQUOTA must be reduced because system load requires it, leave WSEXTENT equal to WSMAX. Typical value is 32,768 pagelets and values over 100,000 typically result in worst performance
  2. Set the process quotas PGFLQUOTA equal to or greater than (WSQUOTA + 25,000). If PGFLQUOTA is too low, the following error can occur.
    %BACKUP-F-ALLOCMEM, error allocating virtual memory -LIB-F-INSVIRMEM, insufficient virtual memory
  3. Set the process limit FILLM equal to the SYSGEN parameter (CHANNELCNT - 20), but never higher. If FILLM is set higher than this, the following type of messages can occur:
    %BACKUP-?-NOMSG, message number
  4. This error occurs because there is no channel available to access the SYSMGTMSG message file. If FILLM is too large can also cause a BACKUP message that is accompanied by:
    -RMS-F-CHN, assign channel system service failed
    -SYSTEM-F-NOIOCHAN, no I/O channel available
  5. Set the process limit DIOLM to 100. Larger values can cause disk I/O subsystem hang or reset without a performance advantage
  6. Set the process limit ASTLM equal to or greater than DIOLM + 100, but never lower. (Note that if you set DIOLM to 32767, then you cannot set ASTLM to 32877 because its maximum value also is 32767.) If ASTLM is set too low, BACKUP will be forced into unnecessary WAIT states, thus increasing the elapsed time of the BACKUP operation.
  7. BYTLM should be set to the values equal to or greater than 256 * FILLM + 6 * DIOLM + 10,000. You can also set it to the amount of space available in nonpaged dynamic memory before the BACKUP job is initiated. Use the SHOW MEMORY/FULL DCL command and note the "Free Space" to determine the space available in nonpaged dynamic memory. If BYTLM is too low the following error message will be returned:

    %BACKUP-F-NOMSG, Message number 00A386E4
    -SYSTEM-F-EXQUOTA, exceeded quota
  8. Set the process limit BIOLM equal to the value of FILLM + 100.
  9. Set the process limit ENQLM greater than FILLM.
  10. Setting higher value for /IO_LOAD qualifier may cause the following error.

    %SYSTEM-F-EXQUOTA, process quota exceeded

NOTE: Since maximizing BACKUP performance includes setting the high process quotas, using an account dedicated to BACKUP operation is ideal. Use of these quotas during normal timesharing may adversely affect other processes.

Please refer the section "Setting Software Parameters for Efficient Backups" from "HP OpenVMS System Manager's Manual, Volume 1: Essentials" for additional details.


Some tips for troubleshooting BACKUP performance issues.

  1. If the BACKUP process consumes too much physical memory, reduce WSQUOTA using the AUTHORIZE Utility or the DCL command SET WORKING_SET. Use the SHOW PROCESS/CONTINUOUS command from another terminal to determine if BACKUP is using too much memory.
  2. If BACKUP tries to open too many files simultaneously, reduce FILLM.
  3. If BACKUP uses all the nonpaged dynamic memory, reduce BYTLM noting the dependencies with FILLM mentioned above. The SHOW MEMORY/POOL DCL command can be used to determine if BACKUP is using all the nonpaged dynamic memory.

NOTE: The aforementioned process quotas and limits can be modified using the Authorize Utility. Any modifications to a UAF record will not take effect until the process is logged out and back in. WSQUOTA and WSEXTENT can be changed immediately with the DCL command SET WORKING_SET.


Techniques for Evaluating Tuning

With the suggested parameter values defining a starting point and the hardware capabilities defining practical performance boundaries, tuning makes sense as long as key system resources remain available to BACKUP. Therefore, tuning BACKUP process requires monitoring its resource usage to determine if further tuning might yield additional performance improvement. The following procedure or techniques for gathering and evaluating BACKUP performance data can be used. The following techniques does not refer to the secondary tuning parameters, it is crucial that the values for these secondary parameters be reviewed whenever an increase in the value of a related primary tuning parameter is indicated.


Running BACKUP from a DCL command procedure provides an automated way of collecting performance statistics. The below command procedure illustrates how to collect and monitor the BACKUP performance. Invocation of this command procedure involves three command line parameters.


  1. A string of BACKUP command qualifiers, such as /IMAGE,
  2. The name of the input volume, such as $10$DKA100: and
  3. The name of the output volume together with any output file qualifiers, such as $90$MKA500:TEST.BCK /BLOCK_SIZE=32528

Refer the VMS Backup Utility Manual for further information on BACKUP's command line format, as well as it input and output specifiers.


$ SET NOON
$ SPAWN /NOWAIT /PROCESS=BACKUP_MONITOR -
MONITOR ALL_CLASSES /NODISPLAY /INTERVAL=3 /RECORD -
/FLUSH_INTERVAL=30

$ START_PROC_DIO = F$GETJPI(0,"DIRIO")
$ START_PROC_CPU = F$GETJPI(0,"CPUTIM")
$ START_PROC_FLTS = F$GETJPI(0,"PAGEFLTS")
$ START_TIME = F$TIM()
$ !
$ BACKUP 'P1 'P2 'P3
$ !
$ STOP_TIME = F$TIME()
$ STOP_PROC_DIO = F$GETJPI(0,"DIRIO")
$ STOP_PROC_CPU = F$GETJPI(0,"CPUTIM")
$ STOP_PROC_FLTS = F$GETJPI(0,"PAGEFLTS")
$ STOP BACKUP_MONITOR
$ PROC_CPU = STOP_PROC_CPU - START_PROC_CPU
$ PROC_DIO = STOP_PROC_DIO - START_PROC_DIO
$ PROC_FLTS = STOP_PROC_FLTS - START_PROC_FLTS
$ WRITE SYS$OUTPUT "CPU TIME (10 msec): ''PROC_CPU'"
$ WRITE SYS$OUTPUT "DIRECT I/O COUNT: ''PROC_DIO'"
$ WRITE SYS$OUTPUT "PAGE FAULT COUNT: ''PROC_FLTS'"
$ WRITE SYS$OUTPUT "START TIME: ''START_TIME'"
$ WRITE SYS$OUTPUT "STOP TIME: ''STOP_TIME'"
$ MONITOR ALL_CLASSES, DISK/ITEM=ALL, SCS/ITEM=ALL -
/SUMMARY=MONITOR.SUM /NODISPLAY /INPUT=MONITOR.DAT
$ EXIT


Within this procedure, DCL lexical functions can be used to gather information about the BACKUP process. F$GETJPI can collect the process absolute number of direct I/Os, CPU time, and page faults both before and after BACKUP is invoked. The lexical function F$TIME can be similarly used to calculate the elapsed time for the BACKUP operation. The difference gives BACKUP's use of these resources during its operation. VMS Monitor Utility (MONITOR) can be run concurrently with the BACKUP operation to evaluate its use of system resources. After the BACKUP operation completes, force the MONITOR process to terminate with the DCL STOP command. The raw MONITOR data collected in the file MONITOR.DAT.

Summary

Although this article describes about the BACKUP's internal mechanisms and tuning, the following two points capture the essentials regarding BACKUP performance.


First, the actual process of tuning BACKUP comprises the following four steps:


  1. Understand BACKUP's tunable mechanisms,
  2. Understand the process and system parameters with which you can tune BACKUP,
  3. Work within the tuning guidelines when setting these parameters and
  4. Know how to evaluate the impact of your tuning efforts.

Second, tuning BACKUP involves below five software parameters:


  1. /BLOCK_SIZE qualifier,
  2. /IO_LOAD qualifier,
  3. WSQUOTA,
  4. FILLM and
  5. BYTLM.

For more information