DU4.0d - patch set #2 - Alpha 4100 + ESA10000
Yesterday I described a situation where a disk in a RAID5 set failed, the
hardware recovered but ADVFS panicked and took down oracle and the partition,
and I asked for insights on why ADVFS would behave this way, and whether
there were any workarounds. Of particular concern was the way ADVFS leaves
the filesystem in an odd state where it looks mounted when you do a "df",
but you can't "cd" to it, nor can you do a "ls" on it, and no files can be
accessed. Even though oracle had crashed, and the RAID5 filesystem was
completely rebuilt, we had to stop oracle, umount/remount the filesystem to
make it available.
Further examination of the logs (subsequent to my initial post) gave
the following sequence of events.
Over a period of about 1 minute:
- disk in raidset logged hard errors in the logfile
- advfs writes failed
- advfs panicked
- hardware raid (ESA10000) noticed, pulled in a hot spare
and began rebuilding.
I only received a very few replies.
- Ronny Leiahu said they had a very similar config and had seen the
same problem. He said the ADVFS/RAID5 combo is touchy, and that
the ADVFS engineers know about it, but that it is considered to
be the best config for reliability on such a large filesystem.
He also noted that it is recommended you:
- dismount advfs filesystem
- run "verify" on it
- remount it
periodically, but noted that, as with us, if you have a production
environment this isn't very viable.
- Stephen Mullin said they'd had a similar problem, gotten a firmware
fix from the HSZ group for it, the fix didn't work, and that they
were still pursuing a solution.
- Dr. Tom Blinn noted that ADVFS behaved correctly in failing the
the filesystem when it encountered hardware errors. He said:
"I would expect that with a properly configured RAID configuration, where you
have full redundancy, you should NEVER see a write error reported back up to
the higher software layers (e.g., to where it would cause an AdvFS domain
panic) UNLESS all the places where the data could/should be written failed."
but noted that perhaps this is an overly optimistic expectation of
RAID5's capability.
We are continuing to pursue this thru hardware/software support, on the
rationale that:
- As Dr. Blinn noted, ADVFS never should have seen the hardware error -
RAID5 should have kicked in first.
- If RAID5 is not capable of this kind of response now, the issue
needs to be addressed, especially in light of the fact that as
sites are building larger and larger filesystems that need both
recoverability and fast reboot, this ADVFS/RAID5 combo will become
more and more prevalent.
I also believe that ADVFS needs the ability to know that the underlying
filesystem is RAID, and to take some recovery actions based on that knowledge.
Our whole crash took about 1 minute. If ADVFS had checked back 65 seconds after
the error, it could probably have remounted the partition (which was
rebuilding)
and gone merrily on its way.
Thanks to all who replied, and any additional comments welcome.
--
Judith Reed
jreed_at_appliedtheory.com
(315) 453-2912 x335
Received on Fri Mar 12 1999 - 19:15:30 NZDT