Thanks to the following respondants:
"Eric Z. Ayers" <eric_at_compgen.com>
alan_at_nabeth.cxo.dec.com
bills_at_atcf.ncsc.navy.mil
Stephen Dowdy <dowdy_at_cs.colorado.edu>
Jorge Pizarro <Jorge.Pizarro_at_sonda.cl>
"C. Ray Ng" <crn_at_mail.lns.cornell.edu>
Wesley Darlington <w.darlington_at_am.qub.ac.uk>
Whom all refreshed my memory by indicating that the update daemon runs every
30 seconds.
My original message:
> Greetings -
>
> Background:
> Yesterday we added a 4GB memory board to our 8400, and are currently
> interleaving the 4GB board with two 2GB boards, and a third 2GB board
> is not interleaved. We now have 10GB RAM to feed four 440 MHz processors
> running DU v3.2G (fully patched). 2700+ processes (>500 simultaneous users).
> Primary CPU is 0% idle 90+% kernel mode; CPUs 1, 2, and 3 are averaging
> 10-20% idle. I realize we're I/O bound at peaks until we get to DU v4.0...
> We are running AdvFS (HSZ50s, hardware RAID-5, 10, 20, and 45 GB RAIDsets).
>
> Problem:
> Every 30 seconds on the dot, for the past couple of hours, since we got
> the system busy with users (all using Oracle), I see a CPU drain for
> about 10 seconds. The system is *not* running any cron, at, batch, etc...
> or any user programs that could account for this. The accuracy of every
> 30 seconds leads me to believe it is some VM (RAM, UBC, etc..) scan that
> the Operating system is doing, or some device flush.
>
> The problem (of this magnitude) started today - the first day after
> getting the system up to 10GB of RAM. We are using over 1GB of
> swap space at this time, down from 5-7GB during peak times last week.
>
> The only way I am able to identify this CPU drain is with some custom
> code of mine which does less than 30 CPU tics of work on the 8400.
> The code displays the elapsed number of tics between 0.5 second select()
> iterations, and every 30 seconds on the button, that 0.5 second select()
> timeout takes between 10 and 11 seconds to complete (virtually the same
> amount of time each 30 seconds).
>
> Question:
> Does anybody know of any 30 second Kernel timer that might could be
> causing this? I haven't (yet) run across any 30 second timer in the source
> code listings that might cause this. Anybody?! Ideas? Perhaps
> un-well-documented kernel parameters for VLM systems?
Jorge Pizarro added additional information:
|>I think that the "update" daemon is the culprit. Try killing /sbin/update
|>process but re-run it again. Some more info, we had some similar problem
|>with AS 4100 5/400, 4 CPU's, 2 GB RAM, HSZ50 with AdvFS, DU 3.2G, the
|>system was very busy, about 30 forks per second, and CPU 0 was 0% idle
|>all the time, we upgrade to DU 4.0B with last patches and the CPU use
|>was more even. Are you sure that do your load is the same that before
|>the 4GB memory upgrade? I presume that the problem is not related to
|>memory upgrade but new system load.
Alan Nabeth added additional information:
|> The classic 30 seconds timer is update, which runs sync(2)
|> which scans the buffer cache and starts a flush of all the
|> dirty buffers. The extra 4 GB of memory added the potential
|> for the UBC to use four more GB of memory. At 8 KB buffers
|> that's a lot of buffers to scan (though I think the UBC
|> arranges its memory in lists of what's dirty, etc). I
|> never heard clearly how AdvFS makes use of the UBC, specifically
|> whether data writes are synchronous or asychronous. Any UFS
|> file system could certainly be a problem, but I don't know
|> if AdvFS could be.
To which I responded directly back to him:
|>Yes, the UBC has a dirty list, (in struct vm_ubc_object, the pointer to
|>structure vu_dirtywpl of type struct vm_page stores this) which according
|>to another utility of mine (vm_mon) rarely logs a dirty page during
|>each 0.5 second scan. My utility pulls a structure which has a
|>'ubc_dirtypages' value for this, which DU may or may not be updating
|>routinely, I can't say.
|>
|>According to the source code listings, the kernel only examines
|>'ubc_dirty_limit' pages each scan, and on this system:
|>
|>(dbx) print ubc_dirty_limit
|>41648
|>
|>and then will only purge 'ubc_maxpurge' pages at a time:
|>
|>(dbx) print ubc_maxpurge
|>256
|>
|>The UBC scans are being done every 250 milliseconds with statistics
|>generated every fourth scan.
|>
|>We're only allowing the UBC to be 5-40% of memory (ubc_{min,max}percent),
|>so our change was only 2.4GB to 4GB, or less than 2GB difference.
|>Our current UBC size started the day (midnight) at 0.139GB, grew to 3.9GB
|>at 0800, then started shrinking and currently sits at 3.2GB. (We track UBC
|>and many other parameters with my sys_stat utility).
|>
|>The AdvFS uses the UBC for both metadata and filedata cache, but I also found
|>in the source listings that AdvFS CHR, BLK, and FIFO file types do not use
|>the UBC.
|>
|>AdvFS writes are both asynchronous and synchronous, depending upon the
|>action and the object doing the writing.
After I installed the memory board yesterday, (after upgrading the CPU
firmware from the v4.0 firmware CD), and booted, the update daemon had
accumulated over 4 1/2 hours of CPU time in about 24 hours of wall clock
time (with less than 4 hours of user activity) per my proc_info utility:
randy_at_glacier> proc_info -p 47 -PCR
root (47) has the following stats:
Process Creation: Sun 28 Sep 1997 11:51:46.276208
...
Current CPU if act.: 0 Kernel PreEmption Disable Flag: 0
Process Command: /sbin/update
Current command: update
PCB Registers:
PC: 0x12000b900 PS: 0x5 SP: 0xfffffffe9c34b3e0 GP: 0x0
KSP: 0xfffffffe9c34b3e0 USP: 0x11ffffbf0 PTBR: 0x10d PCC:
0xbe149a9a
Resource Usage (pid: 47): 0 00:00:00.48800 (usr) 0 04:37:01.822656 (sys)
...
I also wrote some quick process monitoring code to verify that the update
daemon was clocking most of the CPU when I noticed the CPU drain, and it
was (5 secs). While this was happening, CPU 0 went up to 93+% kernel mode
time, and CPUs 1, 2, and 3 went to 50+% idle, then they all balanced back out
when this ended.
Unfortunately this is not appearing to be strictly load related. In the
previous weeks when we were going through student registration, we saw
3000+ processes and 700 simultaneous users on this system - much higher
loads than today, though perhaps I couldn't see quite the distinct 30
second interval then as I do now, the system was a bit sluggish then.
It would appear that modifying the ubc parameters further may enter into
the realm of diminishing returns, so perhaps the only recourse is to
upgrade to DU v4.0B.
Any of my utilities mentioned by name may be found at:
ftp://raven.alaska.edu/pub/randy/{perf_mon_tools,proc_info_utils}
Randy Hayman
haymanr_at_icefog.alaska.edu
Received on Tue Sep 30 1997 - 02:06:29 NZST