I got various and sometimes conflicting opinions, however, it appears that
500,000 files should work fine in a single ADVFS domain. The only
inconvenience is the domain's administration and longer
backup/restore/verify times, for which splitting over multiple domains is
recommended.
BMT parameters should be set high enough to accommodate the files (although
Erik Piip says that it may be already taken care of automatically in 4.0D).
Many thanks to all who replied, as follows (messages and original question
quoted below):
Dr. Tom Blinn, Robert Mulley, Erik Piip, Mohamed Nayle, James Dalton, ,
Serguei Patchkovskii, alan_at_nabeth.cxo.dec.com
ORIGINAL MESSAGE:
>Hello, managers,
>
>Trying to determine the performance impact of putting 500,000+ of 1-to-10
MB files on AdvFS filesets. The only reference from CompaqDigital I found
is at
>
>http://www.unix.digital.com/faqs/publications/base_doc/DOCUMENTATION/V40D_H
TML/AQ0R3FTE/CHXXXXXX.HTM#use-adv-fs
>
>...and it references BMT parameters for up to 800,000 files. Is setting
these to the recommended values sufficient to achieve reasonable
performance? I hear bad rumors about directory lookup times, but no
figures or explanations. Someone got a 30,000 file limit per domain
number, but no rationale for that. If splitting mountpoints
(domains+filesets), what would be good limits for file numbers, total
sizes, etc.
>
>Any help is greatly appreciated.
>
>Alex Gorbachev
>alex_at_iss-integration.com
REPLIES:
>How will the files be organized? If they're all in a single
> directory, I don't think BMT allocation tuning will help much.
> And that many files will have poor performance on a filename
> look-up. Spreading the files out over multiple directories
> should help, but if the directory structure gets too deep
> you may have a problem again. Increasing the hashtable size
> for the namei cache may help a lot. I think this is one of
> the recommendations that sys_check makes when it notices name
> lookup problems.
>
> Also, if my math is correct, 500,000 1 MB files is up around
> 500 GB of data. You won't get that in a single disk and I'm
> not sure you'd want to put it in a single domain. Using
> large arrays for the devices in the domain, if one fails the
> whole domain is in trouble. Since it is wise to spread the
> data out anyway, you may want to spread it out over multiple
> domains.
>
>
>Alex,
>
> I have 2 advfs filesets with over 500,000 files. I haven't had any
>trouble out of them since I set the BMT parameter high enough. The only
>problem is you can't adjust that value after it's created, you have to
>backup, re-create and restore. As for performance accessing a specific file
>isn't much worse then with a smaller directory. If I cat a file, I can't
>visibly tell the difference between a small directory and a >500,000
>directory. Of course doing find or ls takes a long time because of the
>number of files to search through. Backups don't take any longer then
>another directory with 1 very large file, but, pray you never have to do a
>restore. I had to restore one with around 600,000 files and it took about 8
>hours. You mileage may very, our hardware is an 8400 running 4.0B patch 9
>with mirrored disks using dual hsz70's. (I heard at DECUS that 5.0 is
>supposed to greatly improve performance on this. I can only hope it's
>true.)
>
>The 2 filesets are on different systems. They are set up one
>fileset per domain, and each domain only has one volume. The volume uses a
>hardware disk mirror composed of two physical disks. I'm about 99.995% sure
>that we have decent performance out of it because of the hardware disk
>mirror.
>
>I wish I could help you, I have sex systems running DU 4.0 E and
>NETSCAPE Proxy Server, The avarage No. of files is quiet the or larger
>but the file size are much smaller than yours.
>
>The other difference is that I have my disks connected to a RAID
>Controller, The advise that I can give is to use higher paramter than
>mentioned in the documentation (you can also check the man mkfdmn)
>
>I am using
># mkfdmn -p 131072 -x 65536 ......
>We are running our file server using Advfs file systems. Our two big
>directories, /homes and /projects
>both have over 500,000 files as of last count...
>
>One thing to note, if you are using 4.0d, you don't need to use the -x and
>-p options if you are using large nubers of files. The man pages are a
>little out of date... The BMT allocating algorithums have changed between
>4.0b and 4.0d... Our /homes domain is 50Gig. I still set aside more log
>pages (-l option) though, out of paranoia.
>
>
>bash-2.01# cd /homes
>bash-2.01# ls -lR | wc -l
> 608070
>bash-2.01# cd /homes/.tags
>bash-2.01# showfile -x M-6
>
> Id Vol PgSz Pages XtntType Segs SegSz I/O Perf File
>fffffffa.0000 1 16 37250 simple ** ** ftx 100% M-6
>
> extentMap: 1
> pageOff pageCnt vol volBlock blockCnt
> 0 2 1 32 32
> 2 37248 1 45920 595968
> extentCnt: 2
>bash-2.01#
>
>
>
>
>Also it is a good idea to do a defragment on a domain by domain basis. The
>default crontab entry is bad because it tryes to do all the domains in
>parallel, and at a default time.. Run defragment with an experation time of
>a couple of hours (-t 120), once a week, on each domain at a time (One a
>night or whatever) and you should be fine.
>
>I've had one dec person say to me if you have large file sets, keep one pet
>domain. I've followed this for the most part, execpt where I have a couple
>of small filesets that share a domain.
>
>As for performance, Advfs will be better than a similar sized UFS file
>system. If it is a very buisy file system, one thing to look at is to push
>your Advfs Transaction log onto another "faster disk", as this coule become
>a bottle neck. (We also have a sun box using UFS with abotu 1/2 as many
>files. It took over an hour to re-boot after a crash, because it was
>fsck'ing the file system so long!)
>
>Also make sure you have the latest patch kits. We are running 4.0d Patch
>Kit 3.
>
>500,000 files in one directory would slow down directory lookups
incredibly. I
>would put at most 20,000 in one directory. Split them up into
sub-directories.
>The setting of the BMT to 800,000 is not recommended it is necessary to
sustain
>that many file. It is kind of like inodes on UFS systems. If you run out of
>them you have no choice but to rebuild the domain. Make sure it is set high
>enough to accomodate the amount of files.
>You should carefully read the mkfdmn and addvol reference pages as well as
the
>AdvFS administration guide (which is accessible through the Tru64 UNIX web
>site in the documentation area, as well as orderable as a separate manual for
>V4.0x). The actual limit is around 4 billion files per fileset. However,
you
>might want to avoid creating really large domains, for the simple reason that
>it can be very difficult to administer them. For example, with 30 million
>files in a domain, it takes the better part of a day (even on a fast system)
>to verify the domain. Backups and restores go MUCH faster if you have more
>domains with fewer files.
>
>Since you are talking about well under a million files, you should be able to
>make it all work, but again, we'd encourage you to consider ways to manage
the
>file system hierarchy as multiple domains, each with a smaller number of
>files.
>
>> figures or explanations. Someone got a 30,000 file limit per domain
>> number, but no rationale for that.
>
>This is definitly not true. One of our AdvFS *filesets* has more than
>125,000 files in it; the domain containing this fileset is a little
>bit under 200,000 files. This domain is the primary data source for a
>very active NFS server and we observe no performance problems with
>it. No crashes or data corruption either in more than a year of continuous
>operation.
>
Received on Fri Jul 09 1999 - 01:09:26 NZST