Many thanks to :
Alan (alan_at_nabeth.cxo.dec.com)
Dr. Tom Blinn (tpb_at_doctor.zk3.dec.com)
(As usual, these guys are GREAT !)
Alan provided me with some very good clues on the
way DU handles I/O. And Dr. Tom suggested that
indexed I/O performance is probably more related to
the way COBOL allocates buckets, and how frequently
the consistency between indices and data is forced.
Below are their entire responses for the benefit of
the archives. Thanks again !
----------------------------------------------------------------------------
------------------
Original Question :
> Hello !
>
> Our development team is porting an application from VMS to DU 4.0B.
> This app does heavy reads/writes on some huge COBOL indexed files,
> and on VMS they obtained significant performance gains by fiddling with
> some RMS specific characteristics of those files, and by selecting an
> appropiate cluster size for the involved disks. We understand there are
> no such things under DU (unless you buy them from a third party).
> But we believe we can optimize this I/O performance by choosing the
> right "record length" for the files, to read multiple records on the same
> I/O
> operation.. Or, since we are using HSZ50s, maybe there's some parameter
> (like chunk size) that could help. Specifically, the files' record length
> is 256 bytes, but soon it will grow to 311 bytes.
>
> We're open to any suggestion (yeah, we already considered going
> back to VMS ;-)
>
> Many thanks in advance,
>
> Miguel Fliguer
> MINIPHONE S.A. - Buenos Aires, Argentina
> m_fliguer_at_miniphone.com.ar
----------------------------------------------------------------------------
-----------------------------
Alan's response :
To have two choices of where to put the data; on a file system
or on a raw disk.
On a raw disk, you have to use I/O sizes that are multiples
of the underlying sector size, the larger the better if high
data rates are a concern. For high request rates, smaller
is better. With smart data cacheing the right size depends
on the cache. The advantage of raw I/O is that it avoids
a data copy from the device to the kernel cache and then to
user buffer. The disadvantage the lack of file system
structure.
When the data is on a file system, reads and write go to
the buffer cache first and then the cache data is copied
to the user's buffer. You can avoid the copy by using
mmap(2) to map the file's data to the process data space.
Internally, either one operates on fixed size buffers of
data, usually 8 KB.
So, to read(2) a 311 byte record it has to read whatever
8 KB blocks of data the data happens to span, and then copy
the 311 bytes to the user buffer. Since the data is cache,
reading the next 311 byte record or one in the same 8 KB
block will avoid the read to fill the kernel buffer, but
still need the copy.
Writing is more interesting. The kernel will read the file
system block to contain the data, copy the record to the
appropriate place in the block and mark the block as being
"dirty". Then depending on a lot of things the data may
stay dirty in the cache for 10s of seconds. On UFS, the
block will only be flushed when:
1. Sync(2) is called and all dirty buffers are flushed. This
happens about every 30 seconds.
2. There are no clean buffers for reading data, so dirty buffers
are flushed.
3. The process does an fsync(2) to ensure its buffers are
flushed.
4. The process had put file into a mode where writes are
synchronous.
5. The file system is mounted so that writes are synchronous.
6. The write ends on a block boundary. When this happens
the write on that block is asynchronously started. I'm
sure if the I/O size play a part in this because I can't
find my file system sources to check.
AdvFS probably similar conditions that force flushing of
dirty data.
Your 311 byte record size is a poor size because it isn't
a convient multiple of the sector size for raw devices or
for file system blocks of 8 KB. The 256 byte size is good
for raw device because two fit in a sector. It has the
same basic problem as 311 for the file system, since it
means you'll have lots of dirty buffers in the cache. This
may not be bad, but you have to br aware of it.
----------------------------------------------------------------------------
-------------------------
Dr. Tom Blinn's response :
The physical disk read size in the current implementation of DIGITAL UNIX is
8K bytes (which is the physical page size for the system). Depending on the
file system type, data may get allocated in smaller or larger "chunks"; if
the data resides on a UFS file system, you might benefit from "tuning" it,
but it's difficult to say.
I have no personal knowledge of how indexed I/O is implemented in COBOL, and
I suspect it depends on the COBOL implementation used (Digital's COBOL or
the MicroFocus product). In either case, the indexed I/O is probably done
by runtime library code, and the actual allocation of records into buckets,
the indexing methods used, the actual points at which disk I/O is forced to
assure consistency in the indices with what's in the data parts of the file
and other things like this are surely going to have a MUCH greater impact on
performance of the application than the logical record size (number of bytes
in the 01 level data record). I know this from having done COBOL programs
as well as system programming on many systems, including OpenVMS and DIGITAL
UNIX. You need to figure out how to tune the indexed I/O processing part of
the runtime subsystem.
Tom
Dr. Thomas P. Blinn + UNIX Software Group + Compaq Computer Corporation
110 Spit Brook Road, MS ZKO3-2/U20 Nashua, New Hampshire 03062-2698
Technology Partnership Engineering Phone: (603) 884-0646
Internet: tpb_at_zk3.dec.com Digital's Easynet: alpha::tpb
Received on Thu Sep 24 1998 - 20:11:37 NZST