This is an pro-active summary of a problem and the solutions to a clustering
problem I was having. Since I've seen several clustering questions here on
the list I thought I should share this to all.
Names have been changed to protect the innocent.
The cluster includes two 4100 alpha rackmount systems configured into an ASE
cluster. Each system has a public NIC and an ASE NIC. They share storage on
a RAID ARRAY 7000 though a shared SCSI bus ( with the internal termination
removed ).
They provide only one Oracle disk based service ( called "popeye" ).
Although, later I found that another "service" ( called "stock" ) was using
the /mnt1 mount point ( which is the mount point of the shared storage ).
The symptoms were one member crashing when the other member was brought
down, whether or not the member shutdown was the one running the service.
Unless both members were up, the service would come up "unassigned" and (
keep this in mind as a diagnostic tool ) could only be started after
"offlining" then "onlining" the service.
There was also a problem with the r)elocation function of asemgr. the
message returned when trying to relocate the service was that the service
couldn't be stopped.
Keep in mind that if you try to stop a service and a user or a task is using
a disk service, that the service can't umount the mount point. This can have
unpredictable or bad consequences. One way to deal with this is the "fuser"
command. This is not a graceful way to correct the problem, but it is
effective. ( i.e. fuser -k /mountpoint )
Also, while there may be a problem with a particular version of NIC, my
opinion at this point is that a mixture of versions of revs of DE500s was
really the culprit. The DE500s are really great NICs, but there are details
( which apply to any NIC ) that should be considered when setting up a
cluster ( specifically - set the "ew***_mode" for each NIC at the chevron
prompt - i.e. fast or fastFD ( or half duplex or full duplex also speed,
etc. ) )
=============================================================
At the "chevron prompt" "SHOW DEV" and visually we observed that there were
three different DE500 card versions. ( A DE500, 21140 chip with prom socket,
21140 chip without prom socket and 21143 chip with prom socket )
We replaced three of the four cards so all cards are now DE500-AA ( 21140
with prom socket ). Both systems now behaved in a stable manner and could be
shutdown without affecting the other system. I feel strongly that the
mixture of revs of DE500s was the problem, I currently have no first hand
experience with DE500-BAs in a cluster ( but that will change VERY SHORTLY,
as I have all the parts for a DS40 pair cluster and a DS20 pair cluster in
my computer room with DE500-BAs shipped as part of the configuration. I will
advise the list on the success of using the DE500-BAs. I am optimistic
regarding the outcome of using the DE500-BAs )
<<<< ALL NICs IN A CLUSTER SHOULD BE SAME MODEL AND REV >>>>>>>
Finally we addressed the relocation function.
Initially, "bluto" could relocate the "popeye" service to "sweatpea", but
"sweatpea" could not relocate the "popeye" service to "bluto". The new (
after the new NICs ) behavior was that the service no longer came up
"unassigned" when only one member of the cluster was down or shutdown.
The relocation issue was related to the "stock" process using the "/mnt1"
mount point and the stop action script issuing the "fuser -k /mnt" command
as oracle. When the "fuser -k /mnt1" command was issued as "root" ( moving
the "fuser" command to a different place in the stop action script, the
relocation works properly.
Any task or process that utilizes a cluster controlled resource must be
controlled by the cluster management system ( i.e.. asemgr - start and stop
action scripts ).
The "stock" communications process should be added to the start and stop
scripts. Currently the stop action script manages the shutdown, but
potentially could create an issue if trying to relocate the service "popeye"
from the system not running the "stock" task to the system running the
"stock" task ( this modification will take about five minutes to implement
and will prevent this from being a problem ).
<<< DON'T PERFORM TASKS THAT USE CLUSTER RESOURCES OUTSIDE OF THE ASEMGR
>>>>>
This was a "hair-pulling exercise" for me and I hope you can avoid it. Also,
disclaimer: I am very biased in favor of the COMPAQ ( formally known as DEC
clusters ) Tru64-UNIX Clusters, this clustering technology has been around
for a long time ( started in the VMS world ) and I think is the best way to
go.
Jon Vaupel Senior Project Manager
301.625.1024 EdgeMark Systems, Inc.
visit our web site at:
http://www.edgemark.com
Received on Mon Sep 13 1999 - 15:42:28 NZST