My original post was as follows,
-------------------------------------------------------------------------- --
" I am setting up a DS20 server (TRU64 4.0F) with a RAID Array 8000
attached using a HSZ80 SCSI controller. I have created a file domain
with a series of filesets mounted up using ADVFS to store large numbers
of small files. There will be a fileset for each month, and then a
directory beneath that for each day. Typically each daily directory will
store 90,000 to 100,000 small files in it
When copying large volumes of data to these filesets I am seeing a very
noticable degredation in performance as the copy progresses."
Several people noted that ADVFS is not very good at handling a large number files in this manner, it was suggested that this has been improved in V5.1. This upgrade is possible at the present time. Respondants also suggested converting to using UFS, and to pay particular attention to inode configuration.
The compaq guide for ADVFS notes some changes which can be made to the 'makefdmn' syntax to try and eliminate the problem.
The location for this document is as follows,
http://www.tru64unix.compaq.com/faqs/publications/base_doc/DOCUMENTATION/H TML/ADVFS_html/figures/admin-chap2.html#TofC51
Brenden Phillips suggested creating a deeper directory structure and storing files by the hour, rather than by the day. This led us to some testing where each file was stored in a sub-directory corresponding to the last two digits of the file name, so file 1223443201 goes to /2001/06/20/01, 1223443202 goes to /2001/06/20/02, and so forth. As the files are created with sequential numbers, this gives us a good even spread of files across 100 sub-directories. It was however noted that having the directory structure too deep could negate this improvement.
The performance benefits from this change have been quite staggering. An overnight job which copies the files onto the server was running until 8am each morning. It is now finishing before midnight the previous evening.
Thanks to all who responded. Looks like we have a good result on this one, with further avenues to explore if required.
Regards,
Bryan Mills.
This message is intended only for the use of the person(s) ("The intended
Recipient(s)") to whom it is addressed. It may contain information which
is privileged and confidential within the meaning of applicable law. If
you are not the intended recipient, please contact the sender as soon as
possible. The views expressed in this communication are not necessarily
those held by LYNX Express Limited.
Received on Fri Jun 22 2001 - 13:19:53 NZST