HP OpenVMS Systems

Introduction

The Oracle Rdb database engine generates platform-specific executable code subroutines at run time. On VAX systems, VAX executable instructions are generated. On Alpha systems, Alpha executable instructions are generated. When Oracle Rdb was ported to the HP Integrity platform, the ability to execute run-time created subroutines was required as well. This paper discusses background of the original interpretation implementation with a later transaction to native Itanium instruction generation.

Code Generation

When a user’s request (such as the SQL statement “SELECT * FROM CUSTOMERS WHERE CITY = ‘ESPOO’ OR CITY = ‘SALIDA’ ORDER BY LAST_NAME”) is passed to the database engine, a number of executable subroutines are created, at run-time, to perform various request-specific functions.  These functions may include copying data fields, performing null-field handling, doing data field comparisons, and so on.  This run-time request-specific code is an integral part of Oracle Rdb’s database engine and helps to provide high levels of performance. 

On VAX systems, such subroutines would contain VAX instructions (such as MOVC3, RET, MOVL and so on).  When Oracle Rdb was ported to the OpenVMS and Tru64 environments for the Alpha platform, the code generation capabilities were extended to create Alpha instructions (such as CMOV, LDA, STQ, and so on).  The logic of the subroutines, for the most part, is the same between the platforms; just the executable instructions, the register usage, and the system calling standard are different.  

The following example contains a sequence of run-time generated instructions on a VAX system.  Note the CISC architecture of the VAX computer with use of complex instructions that contain multiple operands along several addressing modes along with relatively high code density.

00FB1B18: .WORD ^M<R2,R3,R4,R5,R6,R7>
00FB1B1A: MOVL B^04(AP), R6
00FB1B1E: MOVAB @#00FB1B68, R7
00FB1B25: MOVC5 S^#00, (SP), (SP), S^#05, B^01(R7)
00FB1B2C: MOVL B^69(R6), R0
00FB1B30: ROTL S^#08, R0, R0
00FB1B34: BICL3 #FF00FF00, R0, R1
00FB1B3C: XORL2 R1, R0
00FB1B3F: ROTL S^#10, R0, R0
00FB1B43: XORB2 #80, R0
00FB1B47: XORL3 R1, R0, B^02(R7)
00FB1B4C: CLRB B^01(R7)
00FB1B4F: BBC S^#02, W^00EF(R6), 00FB1B5C
00FB1B55: XORB2 S^#01, B^01(R7)
00FB1B59: CLRL B^02(R7)
00FB1B5C: RET 

The next example contains a sample of run-time generated code on an Alpha system. Note that the Alpha is a more traditional “RISC”-style architecture where instructions are simpler, fixed size, and that the memory reference instructions either read or write memory, but do not atomically read and update memory in a single instruction.

01A060AC A00C8138 LDL R0,#XF8138(R12)
01A060B0 A02C808C LDL R1,#XF808C(R12)
01A060B4 44010400 BIS R0,R1,R0
01A060B8 F4000004 BNE R0,#X0000004 -> 1A060CC
01A060BC A00C8064 LDL R0,#XF8064(R12)
01A060C0 A02C813C LDL R1,#XF813C(R12)
01A060C4 400105A0 CMPEQ R0,R1,R0
01A060C8 F400000D BNE R0,#X000000D -> 1A06100
01A060CC A00C8110 LDL R0,#XF8110(R12)
01A060D0 A02C8090 LDL R1,#XF8090(R12)
01A060D4 44010400 BIS R0,R1,R0
01A060D8 F400000F BNE R0,#X000000F -> 1A06118
01A060DC A36B0144 LDL R27,#X00144(R11)
01A060E0 220C8118 LDA R16,#XF8118(R12)
01A060E4 223F001F LDA R17,#X0001F(R31)
01A060E8 224C8068 LDA R18,#XF8068(R12)
01A060EC 233F0003 LDA R25,#X00003(R31)
01A060F0 A75B0050 LDQ R26,#X00050(R27)
01A060F4 A77B0058 LDQ R27,#X00058(R27)
01A060F8 6B5A4000 JSR R26,(R26) "ots$strcmp_eqls"
01A060FC E4000006 BEQ R0,#X0000006 -> 1A06118
01A06100 201F0001 LDA R0,#X00001(R31)
01A06104 A75E0008 LDQ R26,#X00008(SP)
01A06108 A7BE0010 LDQ FP,#X00010(SP)
01A0610C 23DE0020 LDA SP,#X00020(SP)
01A06110 6BFA8001 RET R31,(R26),1
01A06114 00000000 CALL_PAL HALT
01A06118 47FF0400 CLR R0
01A0611C C3FFFFF9 BR #XFFFFFF9 -> 1A06104

Interpretation engine

Oracle Rdb was ported to run on the Microsoft Windows NT environment running on Intel x86 and Alpha processors (this product was, however, never released for production use). At that point in time, in order to rapidly complete the porting effort for the Intel x86 platform, an interpretation engine was created that could interpret those portions of the Alpha instruction set generated at run-time by Oracle Rdb. This approach allowed a single piece of code (the interpretation engine) to be written and, more importantly, debugged without having to change the instruction generation machinery within the Oracle Rdb database engine (which continued to generate subroutines using the Alpha instruction set).

Over time, an expanded set of “rich” instructions were added to the code generation capabilities on the Intel x86 platform. These instructions were intended to perform more complex actions as one “pseudo” instruction, replacing, in some cases, a large number of Alpha instructions in the code stream. Execution of these “rich” instructions could be more optimized as compared to individually executing long sequences of individual instructions. Approximately 150 of these “rich” instructions were eventually implemented.

Though not used on Alpha and VAX systems (the supported platforms for Oracle Rdb), this interpretation engine remained part of the Oracle Rdb source code and lay “dormant” for many years.

Itanium Emerges

With the advent of OpenVMS for the Integrity Server platform, Oracle chose to port the Rdb database engine to the Integrity Server for the OpenVMS operating system. Though native language compilers were available (primarily, in the case of Oracle Rdb, BLISS, C++, C and MACRO32), there was no immediate capability for the Oracle Rdb engine to create executable instructions for the Itanium architecture.
At this point, the interpretation engine was pressed in to service again. Most of the code had not even been compiled in over 10 years. But with a bit of effort (mostly correcting issues related to improved C compilers with enhanced detection for latent bugs), it was able to successfully execute once again. Debugging effort was required to get it working completely properly but it did prove to be a valuable tool that allowed a significantly more rapid production delivery and deployment of Oracle Rdb Release 7.2 on the Itanium platform.

Performance

Overall, we anticipated that, while the performance of the interpreted code would never be as good as a native executable code subroutine, the Integrity system, as a whole, would perform at least comparable to “equal” Alpha systems. This was the case for the vast majority of applications and systems that we analyzed. CPU, memory and IO performance tended to provide a balanced system that performs very well when running customer applications. A few applications, however, spent a significant amount of time executing the run-time generated code and these applications were, in some cases, significantly slower than we, and our customers, would have preferred.

In particular, one major customer application was generally as good or better performing on Itanium systems than on Alpha systems. But several significant queries of the application were both frequently executed and much slower once migrated from Alpha to Itanium. Analysis revealed that most of the additional CPU time was spent in the interpretation engine while running particular parts of the application.

A major effort was spent in analysis and tuning of the interpretation engine itself. This tuning yielded performance improvements of over 20% in some cases. This was, however, not nearly enough (and, regrettably, not even in the same order of magnitude required). Further analysis indicated that there was likely no way to make the interpretation model execute fast enough to meet our customers’ needs in all cases.

A New Direction

It was felt that the investment required to enhance the Oracle Rdb database engine to add another set of code generation capabilities (in addition to VAX, Alpha and “rich” instructions) for native Itanium would consume significant resources for development and testing and likely could not be completed in time for this particular customer’s production deployment schedule. There were too many locations in the code that would be required to be changed to produce instructions for yet another architecture. Our experience with the port to Alpha indicated that there would be substantial human resources required to produce and debug the resultant code.

Based on this analysis, the concept of “compiling”, at run time, a complete subroutine from a mixture of Alpha and “rich” instructions in to native Itanium executable code was born. The design that we arrived at is not dissimilar to the JAVA machinery’s “Just In Time” (also known as JIT) compiler available on many platforms: Input a stream of generic and platform-independent instructions and create platform-specific executable code which is expected to perform much better than interpreting the “pseudo” instructions.

Initial prototypes were developed to create and call an executable stream of Itanium instructions. The success of these tests supported the idea that it was viable for Oracle Rdb to be able to create native subroutines and to call such code at run time while on the Itanium platform.

High Level Design

The basic operation for what we originally called the “interp compiler” (based on the idea that this was a compiler to replace the interpretation engine) was to pass a pointer to a complete subroutine of compiled Alpha and “rich” code and then attempt to compile it completely in to a native Itanium instruction subroutine. If the compilation was successful (all instructions were able to be compiled) then a pointer to a procedure descriptor for the generated routine was returned with the low bit set (i.e., an odd value). If, however, the compilation could not be complete (if instructions were found that were not able to be compiled), the original routine address was returned (with the low bit clear as the routine had been originally allocated on a longword boundary).

Later, when the subroutine was to be called, the low bit of the routine’s address was first evaluated. If clear, the existing interpretation engine was called to execute the subroutine. If the routine address was odd (indicating that the low bit was set), the routine was called directly (after clearing the low bit) to be executed “native”.

In this way, the “interp compiler” could start small (only able to compile a few instruction types) and then grow (by adding the ability to compile more and more instructions and addressing modes and so on) all while the database engine continued to operate correctly (presumably as more and more subroutines could be compiled, execution performance would continue to improve). This made it possible to continue to execute and test Oracle Rdb while the “interp compiler” was being actively developed. Without this model it was have been a much slower process in that the “interp compiler” would have had to be entirely complete before we could even begin to test it.

Itanium Architecture

Significant attributes of the Itanium architecture that pertain to the “interp compiler” include:

• 128 bit “bundle” containing 3 instructions that will all be executed

• Multiple execution unit type combination selected via a “template” within a bundle

• Predicate registers that control if an instruction will have an effect or not

Producing code for the Itanium architecture is a fair measure more complex than, for example, code generation on the Alpha architecture. A significant set of rules and requirements are imposed in terms of which instruction type may be used in which bundle slot depending on the specified template, the use of “stops” to indicate that the results of prior instructions are required by following instructions, and so on. A larger number of registers provides major benefits in regards to having more scratch registers available for intermediate results. And the use of predicates can, in some cases, drastically reduce the number of branches taken which can, in turn, improve performance by reducing “wasted” processor cycles due to “bubbles”.

Additional steps were required after creating code. Because the Itanium instruction cache (I cache) and data cache (D cache) are not synchronized, after new executable code is created, the hardware must be notified by flushing the instruction cache for the memory addresses of the newly created code. This was accomplished most easily by calling the SYS$PAL_IMB system service specifying the starting address and the length of the generated code. The system service invalidates each I cache line as needed and ensures that the data and instruction caches are correctly synchronized prior to attempting to execute the new instructions.

Starting Simple

The Oracle Rdb “interp compiler” is implemented as a routine written in BLISS (the primary implementation language utilized by Oracle Rdb for both ease of development and resultant product performance). Generation of instructions is accomplished though a set of macros that implement primitive operations that are generally produced as one or more instructions within one or more bundles. The original project goal was to have macros that would create one instruction per bundle. Over time, more and complex macros were created to perform different functions and to create bundles with more instruction sequences to help produce faster and denser code streams.

A simple macro might, for example, produce a single ld4 instruction (to fetch 4 bytes from memory) alone in a single bundle (nop instructions would occupy the remaining two slots). Another level of complexity might be a single macro to create a pair of memory load and store instructions in a bundle. A more complex macro may implement a call sequence where output parameters are created, registers are saved, a procedure descriptor read, the routine called, and then after the call registers are restored. This sequence would require a modest number of bundles to implement.

Branches

One area of complexity is forward and backward branches within the code. The input subroutine may contain both “rich” and Alpha instructions that change the flow of control via conditional and unconditional branches. Branches are self-relative within the context of the input routine. To preserve the correct branch destination, a table is constructed that contains the address of the bundle containing the branch along with the original branch offset. Another table is maintained to associate the original instruction location along with the location of the generated code. After code generation is complete, branches are “fixed up” to adjust the destination offset to the correct destination bundle.

For performance, the Itanium architecture includes “hints” for most types of branch instructions. These hints allow a compiler to indicate additional information to the hardware in regards to how branches are expected to execute. The hardware, in turn, may use this information to predict how the flow of control is expected to operate and can allocate resources more efficiently and, ultimately, execute the whole of the code stream faster. For example, one such branch “hint” type would be “dynamic, predicted not taken”. This hint implies that the compiler expects that the conditional branch will generally not be taken but the hardware should allocate prediction resources (such a history of branches taken or not taken at this location).

Based on both performance testing and research papers evaluated, the “interp compiler” utilizes these branch hints in the generated instruction stream. Unconditional branches are specified as “static, taken”, most conditional branches are specified as “dynamic, predicted not taken”. Exceptions to these rules are backward (typically involved in a loop) branches which likely are specified as “dynamic predicted taken”.

Exception Handling

In order to allow OpenVMS exception handling mechanisms to function properly, the “interp compiler” must “register” each generated routine with the operating system. This registration includes identifying any “unwind” information specifically regarding the routine’s first and last instruction, the length of the routine’s prologue and registers and stack usage. Because the created code will be both created and executed in the processor’s executive mode, a kernel mode image exit handler is utilized to un-register the generated code during image run down. Without having such unwind information registered with the operating system, exception handling is not possible; otherwise an exception from the generated code, or code that is called by the generated code, cannot be handled and results in, depending on the mode and context, either process or image termination.

The OpenVMS calling standard uses a variant of the common Itanium standard which includes a moderately complex set of rules for representing unwind information. This scheme includes a compressed variable-length fields and a dense structure. Within the interp compiler, this “signature” information is produced at the end of executable code creation for each routine.

Simple Code Sequence Examples

The following code sequence shows the original “rich” instruction (indicated by longword address and content fields at the left part of the line) CLR_Q (clear quadword) along with its single operand followed by the generated Itanium instructions (indicated by quadword content and instruction addresses) created for the “rich” instruction.

The operation’s addressing mode is evaluated as an offset from the global register r2 (this register maps to Alpha register R12 within Oracle Rdb). The offset is created by adding 8000 to the value 0040 and then sign extending from 16 to 64 bits. Next, r0 (which is always read as the value zero) is written to the destination address, thus clearing it.
It would obviously be possible to combine these two instructions in to a single bundle. However, the construction of the interp compiler is such that the addressing steps are evaluated first and then the operation steps are produced. While it would be viable to perform a second pass to combine the instructions in to a single bundle, it has not yet been a high priority for execution optimization.

An additional concern for such optimization is that significant amounts of time could be spent in the interp compiler that could exceed the potential benefits for performance improvements of the generated code. In this case, for example, the stall caused by the memory reference will dramatically overshadow any other optimizations possible for the two instructions which still require a stop between them (as the first updates r14 which is used as input to the second).

s037E54B4 04000157 CLR_Q 
037E54B8 D0000040 dst quad*
{ .mfi
013807E80380 0000000080366190 add r14 = 3F8040, r2
000008000000 0000000080366191 nop.f 000000
000008000000 0000000080366192 nop.i 000000 ;; }
{ .mfi
008CC0E00000 00000000803661A0 st8 [r14] = r0
000008000000 00000000803661A1 nop.f 000000
000008000000 00000000803661A2 nop.i 000000 ;; }

In the next example, the MOV_Q (move quadword) “rich” instruction has two operands (source and destination address information). The source location is indicated as an offset (00000050) from a register (Alpha register R16 which is translated as Itanium register r32; the first input parameter to the routine as specified in the OpenVMS calling standard). The destination is an offset (0040) from register r2 (strictly, the offset is FFFF8040 from register R30). The interp compiler detects that both source and destination addresses are likely to be at least quadword aligned and produces a single ld8 instruction to read the source quadword and a single st8 instruction to write to the destination.

037E54C4 04000145 MOV_Q 
037E54C8 00000010 src char* (R16)
037E54CC 00000050 offset long
037E54D0 D0000040 dst quad*
{ .mfi
0108020A0380 00000000803661C0 add r14 = 0050, r32
000008000000 00000000803661C1 nop.f 000000
000008000000 00000000803661C2 nop.i 000000 }
{ .mfi
013807E803C0 00000000803661D0 add r15 = 3F8040, r2
000008000000 00000000803661D1 nop.f 000000
000008000000 00000000803661D2 nop.i 000000 ;; }
{ .mmi
0080C0E00400 00000000803661E0 ld8 r16 = [r14] ;;
008CC0F20000 00000000803661E1 st8 [r15] = r16
000008000000 00000000803661E2 nop.i 000000 ;; }

More Complex Examples

In the following example generated code, the “rich” instruction MOV_NB_BR_CLR is used to move a null bit (an indication of a database field within a row containing a value) to a byte and then branch if the bit was clear (indicating in this case that the field was not null). Note that there are 4 operands to the “rich” instruction. The interp compiler turns this “rich” instruction in to 8 Itanium instructions stored in three bundles.

The first two instructions add the offset 00CA to r32 (the first input parameter to the subroutine) and then fetch a byte from the resultant location. The next two instructions move the offset 3F8030 to r14 and then add r30 to r14 to result in the output address of the null bit.

The fifth and sixth instructions first extract the bit specified in the first operand of the rich instruction and then test the bit to determine if it is set or clear. The extr.u instruction extracts one bit from the specified position (4 in this case) and stores the result starting at bit 0 in the register r15. In the next instruction (tbit.z), the predicate register p6 will be set if the null bit is equal to zero and will be cleared if the null bit is not equal to zero.

Finally the resultant null byte is stored. If the null bit is clear (indicating that the database field has a value), a branch is to be taken. The branch displacement is a sign-extended 21-bit value indicating a number of longwords. Here, it is a forward branch of 4 longwords. In the instruction stream, if predicate p6 is true (which indicates that the null bit was not set), a relative branch is taken. Otherwise, if predicate p6 is false, the branch is not taken and execution continues at the first instruction of the following bundle.

037E546C 0400019E MOV_NB_BR_CLR 
037E5470 00000004 bitNum ubyte
037E5474 000000CA nulByt ulong
037E5478 D0000030 res ubyte*
037E547C FFE00004 brOff ulong (037E5490)
{ .mmi
01080A0943C0 00000000803660D0 add r15 = 00CA, r32 ;;
008000F003C0 00000000803660D1 ld1 r15 = [r15]
013807C60380 00000000803660D2 mov r14 = 3F8030 ;; }
{ .mii
010001E1C380 00000000803660E0 add r14 = r14, r30
00A400F103C0 00000000803660E1 extr.u r15 = r15, 04, 01 ;;
00A038F00180 00000000803660E2 tbit.z p6, p7 = r15, 00 }
{ .mfb
008C00E1E000 00000000803660F0 st1 [r14] = r15
000008000000 00000000803660F1 nop.f 000000
008400009006 00000000803660F2 (p6) br.cond.dptk.many 0000040 ;; }

Within the database environment, string operations (moving, changing and comparing) are common. The following example demonstrates the compiled code for the CMP_S “rich” instruction which is used to compare two fixed the length strings. The first operand is the number of bytes to compare. The second operand is the address of the first string and the third operand is the address of the second string. CMP_S returns either -1, 0 or 1 to the return status register (r8 which maps to Alpha R0) depending on the relationship (less than, equal, greater than) of the two strings.

The loop count application register ac.lc is used in conjunction with the br.cloop (branch counted loop) instruction to implement the main loop construct. Within the body of the loop, two bytes are fetched with a post increment of the source registers. Then the cmp.eq instruction is used to compare the values of the bytes for equality. Predicate register p7 is set if the comparison detects inequality. Un-equal values result in a branch out of the loop. Otherwise (in the case of the bytes being equal to each other), a backwards branch is taken by the br.cloop instruction to the prior bundle to fetch the next bytes.

When the bytes are known not equal, they are compared to each other with the cmp.lt instructions. If the strings are equal (when the loop executes to completion and no different bytes had been detected), r8 remains as zero. If the last bytes fetched are not equal (indicating that the loop did not complete and a difference was found), r8 is set to either -1 or 1. Note that within the final two bundles, the comparisons are done in parallel (the instructions can execute simultaneously because they do not depend on each other) and then the two moves are executed in parallel. The moves to r8 can be executed simultaneously because at most one of them will produce a result because predicates p6 and p7 are mutually exclusive – in no case will both be set. It is possible that neither is set (when the strings are equal) and r8 will remain 0.

037F22FC 0400013B CMP_S 
037F2300 0000001F srcLen uword
037F2304 D0000288 src1 byte*
037F2308 025B5950 src2 byte*
{ .mfi
000008000000 0000000080372090 nop.m 000000
000008000000 0000000080372091 nop.f 000000
00005413C000 0000000080372092 mov.i ar.lc = 1E }
{ .mfi
01382FE10380 00000000803720A0 add r14 = 3F8288, r2
000008000000 00000000803720A1 nop.f 000000
000008000000 00000000803720A2 nop.i 000000 }
{ .mlx
000008000000 00000000803720B0 nop.m 000000
000000000009 00000000803720B1 movl r15 = 00000000025B5950 ;;
00C596CA03C0 }
{ .mmi
00A000E02400 00000000803720C0 ld1 r16 = [r14], 001
00A000F02440 00000000803720C1 ld1 r17 = [r15], 001
000008000000 00000000803720C2 nop.i 000000 ;; }
{ .mbb
01C039120180 00000000803720D0 cmp.eq p6, p7 = r16, r17
008600003007 00000000803720D1 (p7) br.cond.dpnt.many 0000010
0091FFFFE140 00000000803720D2 br.cloop.sptk.few 1FFFFF0 ;; }
{ .mii
010800000200 00000000803720E0 mov r8 = r0
018001120180 00000000803720E1 cmp.lt p6, p0 = r16, r17
0180010221C0 00000000803720E2 cmp.lt p7, p0 = r17, r16 ;; }
{ .mfi
013FFFCFE206 00000000803720F0 (p6) mov r8 = 3FFFFF
000008000000 00000000803720F1 nop.f 000000
012000002207 00000000803720F2 (p7) mov r8 = 000001 ;; }

In the case of the SET_T (set text) instruction, one or more bytes of a constant value are written to memory starting at a specified location. The interp compiler attempts to optimize these memory writes by performing overlapped operations and performing as few writes as possible by promoting the size of the memory reference based on the minimum alignment of the read and write stream pointers. Two pointers are used, offset by 8 bytes, to allow multiple st8 instructions to be executed in parallel. Post-increment instruction modes are used to update the output pointers in order to avoid additional instructions that would otherwise be required in order to increment the pointers. “Tail” writes of one, two or four bytes are used to complete the sequence.

If the byte count for the fill was larger, a loop would have been generated to perform the fill. In addition, the interp compiler produces, as needed, code to perform one, two or four byte writes prior to the loop and then again after the loop in order to align the output pointer on an 8 byte boundary so that as few memory writes as possible are created.

037E5550 04000151 SET_T 
037E5554 00000020 src byte
037E5558 D0000060 dst char* #XFFFF8060(R12)
037E555C 0000001F dstLen word
{ .mfi
013807EC0380 0000000080366340 add r14 = 3F8060, r2
000008000000 0000000080366341 nop.f 000000
000008000000 0000000080366342 nop.i 000000 ;; }
{ .mlx
010800E10400 0000000080366350 add r16 = 0008, r14
008080808080 0000000080366351 movl r15 = 2020202020202020 ;;
00C2002403C0 }
{ .mmi
00ACC0E1E400 0000000080366360 st8 [r14] = r15, 010
00ACC101E400 0000000080366361 st8 [r16] = r15, 010
000008000000 0000000080366362 nop.i 000000 ;; }
{ .mmi
00ACC0E1E200 0000000080366370 st8 [r14] = r15, 008 ;;
00AC80E1E100 0000000080366371 st4 [r14] = r15, 004
000008000000 0000000080366372 nop.i 000000 ;; }
{ .mmi
00AC40E1E080 0000000080366380 st2 [r14] = r15, 002 ;;
00AC00E1E040 0000000080366381 st1 [r14] = r15, 001
000008000000 0000000080366382 nop.i 000000 ;;

Accessing Unaligned Data

Both the Alpha and Itanium systems impose a severe performance penalty when the processor attempts to perform an unaligned memory reference. An unaligned reference, for example, would be to attempt to fetch a longword (4 bytes) from a virtual address where the two lowest bits are not clear (i.e. not aligned on a 4 byte boundary). And the penalty on OpenVMS Itanium systems is significantly higher than it is on Alpha systems. Thus, avoiding alignment faults has an even greater benefit (for all processes on the system) on Itanium systems.

The “interp compiler” attempts to detect memory references that are not naturally aligned and produces a longer code sequence to perform the memory read or write operation without the overhead of an alignment fault.

For “rich” instructions, the assumption is made that register addresses are naturally aligned on quadword (8 byte) boundaries. Offset values can be then evaluated to determine if the resultant memory address is aligned or not. When an unaligned reference is predicted, a sequence of instructions can be generated to avoid the fault. For example, a load of a quadword that is located on a longword boundary an be accomplished by fetching the two longwords and then merging them together with the “mix4.r” instruction:

{ .mmi
013807CF8E40 000000008033E180 mov r57 = 3F807C ;;
000008000000 000000008033E181 nop.m 000000
01000393CE40 000000008033E182 add r57 = r30, r57 ;; }
{ .mmi
00A083908900 000000008033E190 ld4 r36 = [r57], 004 ;;
008083900380 000000008033E191 ld4 r14 = [r57]
000008000000 000000008033E192 nop.i 000000 ;; }
{ .mfi
000008000000 000000008033E1A0 nop.m 000000
000008000000 000000008033E1A1 nop.f 000000
00F88241C380 000000008033E1A2 mix4.r r14 = r14, r36 ;; }

The following sequence (adapted from analysis of code generated by the HP GEM compiler backend) is a longword store where the destination is predicted to not be naturally aligned. The least significant bit of the address (presented in r3) is tested. If it is set, the address is byte aligned and p7 is set; otherwise the address is word aligned and p6 is set. In the case of byte alignment, a single byte is stored and the address is incremented (thus aligned on a word boundary) and the output value is shifted 8 bits to the right. A word is then stored and the output is shifted right 16 bits. Finally, if the original address was word aligned, the final word is written, otherwise the final byte is written. This sequence results in either two (for word alignment) or three (for byte alignment) memory writes.

00A072000180 0611 tbit.z pr6, pr7 = r3, 0
00AC0031004E 0621 (pr7) st1 [r3] = r8, 1
00A5B882020E 0622 (pr7) shr.u r8 = r8, 8 
00AC40310080 0630 st2 [r3] = r8, 2
00A578840200 0631 shr.u r8 = r8, 16
008C40310006 0640 (pr6) st2 [r3] = r8
008C0031000E 0641 (pr7) st1 [r3] = r8

Optimizations

In some situations, the generated Itanium code sequences will execute faster than corresponding sequences on Alpha. For example, in cases of filling or comparing a relatively few bytes of memory, the code generated for Itanium includes a sequence of memory stores or fetches in-line while the Alpha code calls to the operating system routines OTS$FILL or OTS$CMP variants. The overhead of the call in some instances will be greater than the actual memory references.

In other cases, the Itanium instruction set provides instructions that perform operations that require a sequence of instructions on the Alpha platform. For example, the “mux1@rev” instruction can be used to reverse the order of bytes within a quadword. Within Oracle Rdb on the Alpha platform, this operation is accomplished in a series of independent shift and mask instructions. This byte reversal is, for example, used when constructing index keys so performance is an important consideration as this may be a commonly executed sequence.

Optimization does tend to be a repetitive and, at least based on our observations, a never-ending process. Over time, code sequences are compressed and improved with a goal of reducing latency in regards to the CPU clock rate and memory access latencies.

For example performance analysis both “by eye” and by processor cycle sampling lead to reductions in code steams by often “combining” addressing operands in to a single bundle as in the following example; initially the operands (moving the address values to r14 and r15) would have required two bundles.

06CE1B68 04000148 MOV_B 
06CE1B6C D00002D0 src byte*
06CE1B70 D00010F1 dst byte*
{ .mfi
01382FEA0380 00000000805326F0 add r14 = 3F82D0, r2
000008000000 00000000805326F1 nop.f 000000
01390FEE23C0 00000000805326F2 add r15 = 3F90F1, r2 ;; }
{ .mmi
008000E00400 0000000080532700 ld1 r16 = [r14] ;;
008C00F20000 0000000080532701 st1 [r15] = r16
000008000000 0000000080532702 nop.i 000000 ;; }

Instruction Execution Frequency

As part of a performance analysis sub-project, we created an instrumented interpretation engine that sent, via an OpenVMS mailbox, instruction execution information from all processes on a system to a separate collector process that captured instruction execution counts during a portion of an Oracle Rdb regression test run. The following table includes the top 20 instructions and the number of times each instruction was executed. In the table, the indication “RICH” indicates a “rich” instruction and “EVAX” indicates an Alpha instruction.

This data was, in turn, used as a guide for which instructions should be first considered for increased optimization by the interp compiler. The idea is that an instruction executed several times an hour has a marginal impact on performance as compared with an instruction executed thousands of times per second.

Database Performance Improvements

The creation and optimization of the Oracle Rdb “interp compiler” has been an iterative affair. Initial performance improvements from the interp compiler allowed applications running on Itanium systems to run at least as fast as on Alpha systems. Further optimizations (including reducing memory references, eliminating unneeded “stops”, avoiding alignment faults, and so on) have dramatically improved code quality and yielded even better performance. In some cases, application performance has improved by a factor of 3 due to the interp compiler generating native instructions.

And Oracle continues to measure and analyze performance of the Oracle Rdb database product family on the HP OpenVMS operating system for the Integrity Server platform. An extensive set of regression tests are continuously run in our development environment to help ensure correctness of the generated code. We are also in constant contact with our customer based to help understand their performance challenges. This input helps us decide where to focus our optimization efforts to everyone’s benefit.

Models and Examples Followed

A number of different resources were referenced in regards to code generation. In addition to the (voluminous) Intel documentation of the Itanium architecture, we also utilized the compiler machine code listings from high level language compilers on OpenVMS (for example, BLISS, C and MACRO32 which all use GEM code generator and the C++ compiler which uses an Intel code generator).

Both the OpenVMS debugger and system dump analyzer include the ability to format an instruction stream which helped significantly when we were learning the intricate details of the Itanium architecture.

The OpenVMS listings include the MACRO2000 facility which implements the MACRO32 compiler. This was used in many cases as a template for code generation for complex alpha instructions (such as ZAP and MSK). The internet also proved to be an excellent resource for example instruction streams and discussions of Itanium performance in regards to the use of the architecture.

The Intel and OpenVMS documentation was referenced extensively while we were creating the unwind information tables for generated code. And the OpenVMS calling standard manual was invaluable in regards of register usage rules.

Credit and Thanks

A large number of people devoted a great many hours to this project of developing the Oracle Rdb “Just In Time” code generator for the Itanium systems. It is not possible to remember or credit everyone who was involved. But special thanks and recognition are due to engineering members of both HP and Oracle including: John Reagan, Jeanie Leab, Guenther Froehlin, Greg Jordan, Christian Moser, Burns Fisher, Ian Smith, Martin Ramshaw, and Richard Bishop.

For more information

The Oracle Rdb web site is accessible on the internet at www.oracle.com/rdb. For more information about the Intel Itanium architecture and instruction set, visit www.intel.com. For more information about the HP OpenVMS system, visit www.hp.com and www.hp.com/go/openvms