Several people wrote to suggest I send ifconfig/arp/netstat output.
I've gathered that from:
Server1 - talks to server2->server8
Server4 - talks to everyone except server8
Server8 - talks to server1->server3, not to server4->server7
I'm including it here. Details are as below - server1 and server4 send
traffic correctly out "-app2" interface - 192.168.95.192/26 - server8
receives traffic from problematic nodes through "-app2" interface
(192.168.95.192/26) but then tries to send it out "front end" -
192.168.91.0/24 - interface.
=====================================================================
server1 - alpha-s18 - reaches everyone, including server4 and server8
=====================================================================
lo0: flags=100c89<UP,LOOPBACK,NOARP,MULTICAST,SIMPLEX,NOCHECKSUM>
inet 127.0.0.1 netmask ff000000 ipmtu 4096
tu0: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.95.14 netmask ffffffc0 broadcast 192.168.95.63 ipmtu
1500
tu1: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.27.78 netmask ffffffc0 broadcast 192.168.27.127 ipmtu
1500
tu3: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.95.206 netmask ffffffc0 broadcast 192.168.95.255 ipmtu
1500
alpha-s17-app2 (192.168.95.203) at 08-00-2b-c3-48-6f
alpha-s19-app2 (192.168.95.207) at 00-06-2b-00-2a-8b
alpha-s20-app2 (192.168.95.208) at 00-06-2b-00-30-5c
alpha-s21-app2 (192.168.95.209) at 00-06-2b-00-78-5e
alpha-s22-app2 (192.168.95.210) at 00-06-2b-00-ba-3a
alpha-s23-app2 (192.168.95.211) at 00-06-2b-00-2d-1b
Route Tree for Protocol Family 2:
default 192.168.27.65 UGS 2 4462725 tu1
127.0.0.1 127.0.0.1 UHL 9 27764691 lo0
192.168.27.64/26 192.168.27.78 U 3 746352384 tu1
192.168.27.78 192.168.27.78 UHL 15 6984023 tu1
192.168.95/26 192.168.95.14 U 21 253696769 tu0
192.168.95.14 192.168.95.14 UHL 0 39855248 tu0
192.168.95.192/26 192.168.95.206 U 1 1378255 tu3
192.168.95.206 192.168.95.206 UHL 0 0 tu3
===============================================================
server4 - alpha-s20 - reaches everyone except server8
===============================================================
lo0: flags=100c89<UP,LOOPBACK,NOARP,MULTICAST,SIMPLEX,NOCHECKSUM>
inet 127.0.0.1 netmask ff000000 ipmtu 4096
tu0: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.95.15 netmask ffffffc0 broadcast 192.168.95.63 ipmtu
1500
tu1: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.27.79 netmask ffffffc0 broadcast 192.168.27.127 ipmtu
1500
tu3: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.95.208 netmask ffffffc0 broadcast 192.168.95.255 ipmtu
1500
alpha-s17-app2 (192.168.95.203) at 08-00-2b-c3-48-6f
alpha-s18-app2 (192.168.95.206) at 00-06-2b-00-2d-7b
alpha-s19-app2 (192.168.95.207) at 00-06-2b-00-2a-8b
alpha-s21-app2 (192.168.95.209) at 00-06-2b-00-78-5e
alpha-s22-app2 (192.168.95.210) at 00-06-2b-00-ba-3a
alpha-s23-app2 (192.168.95.211) at 00-06-2b-00-2d-1b
Route Tree for Protocol Family 2:
default 192.168.27.65 UGS 2 4431631 tu1
127.0.0.1 127.0.0.1 UHL 5 25940368 lo0
192.168.27.64/26 192.168.27.79 U 4 472690551 tu1
192.168.27.79 192.168.27.79 UHL 8 6983291 tu1
192.168.95/26 192.168.95.15 U 61 1632716853 tu0
192.168.95.15 192.168.95.15 UHL 0 0 tu0
192.168.95.192/26 192.168.95.208 U 1 1376443 tu3
192.168.95.208 192.168.95.208 UHL 0 12 tu3
========================================================================
====
server8 - alpha-s24 - reaches server1/server2/server3, not
server4->server7
========================================================================
====
lo0: flags=100c89<UP,LOOPBACK,NOARP,MULTICAST,SIMPLEX,NOCHECKSUM>
inet 127.0.0.1 netmask ff000000 ipmtu 4096
tu0:
flags=1c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX,OACTIVE>
inet 192.168.95.6 netmask ffffffc0 broadcast 192.168.95.63 ipmtu
1500
tu1: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.27.68 netmask ffffffc0 broadcast 192.168.27.127 ipmtu
1500
tu2: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.91.55 netmask ffffff00 broadcast 192.168.91.255 ipmtu
1500
tu3: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.27.9 netmask ffffffc0 broadcast 192.168.27.63 ipmtu
1500
tu4: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
inet 192.168.95.212 netmask ffffffc0 broadcast 192.168.95.255 ipmtu
1500
alpha-s18-app2 (192.168.95.206) at 00-06-2b-00-2d-7b
alpha-s19-app2 (192.168.95.207) at 00-06-2b-00-2a-8b
alpha-s20-app2 (192.168.95.208) at 00-06-2b-00-30-5c
alpha-s21-app2 (192.168.95.209) at 00-06-2b-00-78-5e
alpha-s22-app2 (192.168.95.210) at 00-06-2b-00-ba-3a
alpha-s23-app2 (192.168.95.211) at 00-06-2b-00-2d-1b
Route Tree for Protocol Family 2:
default 192.168.91.1 UGS 83 60046628 tu2
127.0.0.1 127.0.0.1 UHL 7 730563 lo0
192.246.157/24 192.168.27.1 UGS 0 469300 tu3
192.168.27/26 192.168.27.9 U 1 37352 tu3
192.168.27.9 192.168.27.9 UHL 0 0 tu3
192.168.27.64/26 192.168.27.68 U 11 200780825 tu1
192.168.27.68 192.168.27.68 UHL 9 623779 tu1
192.168.91/24 192.168.91.55 U 1 531795 tu2
192.168.91.55 192.168.91.55 UHL 0 0 tu2
192.168.95/26 192.168.95.6 U 11 243411281 tu0
192.168.95.6 192.168.95.6 UHL 0 104801 tu0
192.168.95.192/26 192.168.95.212 U 1 9888 tu4
192.168.95.212 192.168.95.212 UHL 0 0 tu4
======================================================================
Judith Reed
jreed_at_navisite.com
Service delivery manager, Syracuse Data Center
315-453-2912 x5835
-----Original Message-----
From: tru64-unix-managers-owner_at_ornl.gov
[mailto:tru64-unix-managers-owner_at_ornl.gov] On Behalf Of Reed, Judith
Sent: Tuesday, March 23, 2004 11:03 AM
To: tru64-unix-managers_at_ornl.gov
Subject: Question about very strange routing problem
We have a set of 8 servers, all running tru64 v5.1a, pk2. They are all
connected to multiple nets, but all share a common net as well:
Server1 server2 server3 server4 server5 server6 server7 server8
| | | | | | | |
----------------------------------------------------------------
server1/server2/server3 can all reach server8 and vice-versa.
server4/server5/server6/server7 can *NOT* reach server8, nor can server8
reach them.
server4/server5/server6/server7 can reach server1/server2/server3 and
vice-versa.
server1->server7 all know to go out the shared network to reach server8.
server8 knows to go out the shared network to reach server1->server3,
but tries to go out a different default route to reach server4->server7,
even when responding to pings/ssh/tcpdump packets coming in the shared
common network from the problematic servers.
server8 also knows the correct MAC address of *all* the servers on the
shared common network, and the correct IPs associated with those MAC
addresses.
There is nothing odd in /etc/routes or "netstat -r" output on *any* of
the servers, either the ones that *can* communicate or those who
*cannot*.
I'm baffled. If this wasn't a prod env. I'd reboot the d*mn server8, but
that's not an option. Anyone have any suggestions/insights???
Regards,
Judith Reed
jreed_at_navisite.com
Service delivery manager, Syracuse Data Center
315-453-2912 x5835
Received on Tue Mar 23 2004 - 16:47:48 NZST