Hi!
I have a problem that's really tricky.
I have a cluster consisting of two 8200 running v4.0F and ASE 1.6
The primary machine have 6 Gb of memory and four CPU:s
This is supposed to be a 24x7 production system and Compaq doen's have any
good answers.
When I have about 550 users on the system accessing a Informix/C-ISAM flat
file database system
the load on the system goes from about 4 straight up to a run-queue of 300,
over a few seconds.
As a user in Unix I don't notice that the system is slow but for the users
the system is really slow.
When I ask them to log out the system gets a normal state when about half of
the users are out, and then the system is fine.
But it doesn't happen every time and not at the same time. About 50 % of the
processes are in running state when this happens.
The CPU:s are mostly doing system time.
I got this problem last Friday and it happened again today. I haven't made
any system changes over the last 6 months.
I have no error messages in the logs.
Below I have some printouts from vmstat and trace.
ida:root> vmstat 1
Virtual Memory Statistics: (pagesize = 8192)
procs memory pages intr cpu
r w u act free wire fault cow zero react pin pout in sy cs us sy
id
152 2066 63 658K 55K 57K 1148M 218M 307M 16M 246M 937K 459 38K 5K 11
16 73
153 2066 62 658K 54K 57K 1998 34 1866 0 25 0 1K 85K237K 22
78 0
168 2047 63 658K 54K 57K 1352 0 1290 0 0 0 141 96K215K 24
76 0
144 2064 63 658K 55K 57K 4625 295 2884 0 183 0 127 84K211K 23
77 0
160 2048 63 658K 55K 57K 3585 1 2418 0 0 0 118 91K224K 24
76 0
165 2043 63 658K 55K 57K 1340 0 1284 0 0 0 109 82K241K 23
77 0
158 2050 63 658K 55K 57K 1339 0 1288 0 0 0 107 87K240K 23
77 0
158 2050 63 658K 55K 57K 2177 179 1359 0 697 0 113 91K234K 25
75 0
155 2052 63 658K 55K 57K 1342 0 1286 0 0 0 133 88K244K 22
77 0
169 2038 63 658K 55K 57K 1361 0 1289 0 2 0 121 85K239K 23
77 0
165 2034 63 657K 55K 57K 1377 0 1285 0 19 0 147 84K240K 24
76 0
177 2022 63 657K 55K 57K 1455 19 1317 0 21 0 169 89K216K 24
76 0
158 2041 63 657K 55K 57K 1301 0 1283 0 0 0 97 83K246K 24
76 0
158 2037 63 657K 55K 57K 1332 0 1281 0 0 0 94 88K238K 23
77 0
156 2039 63 657K 55K 57K 1347 0 1282 0 1 0 90 82K241K 21
78 0
143 2052 63 657K 55K 57K 1340 0 1278 0 0 0 140 83K245K 23
77 0
174 2017 63 657K 56K 57K 1354 0 1286 0 7 0 396 79K236K 22
78 0
152 2034 63 656K 56K 57K 1364 0 1278 0 1 0 853104K176K 22
78 0
166 2019 63 656K 56K 57K 2177 179 1347 0 699 0 138 94K201K 30
70 0
ida:root> vmstat -P
Total Physical Memory = 6144.00 M
= 786432 pages
Physical Memory Clusters:
start_pfn end_pfn type size_pages / size_bytes
0 256 pal 256 / 2.00M
256 786348 os 786092 / 6141.34M
786348 786432 pal 84 / 672.00k
Physical Memory Use:
start_pfn end_pfn type size_pages / size_bytes
256 278 unixtable 22 / 176.00k
278 284 scavenge 6 / 48.00k
284 852 text 568 / 4.44M
852 1006 data 154 / 1.20M
1006 1419 bss 413 / 3.23M
1419 1580 kdebug 161 / 1.26M
1580 1585 cfgmgmt 5 / 40.00k
1585 1586 locks 1 / 8.00k
1586 3362 unixtable 1776 / 13.88M
3362 3375 pmap 13 / 104.00k
3375 15926 vmtables 12551 / 98.05M
15926 786348 managed 770422 / 6018.92M
============================
Total Physical Memory Use: 786092 / 6141.34M
Managed Pages Break Down:
free pages = 78069
active pages = 35715
inactive pages = 58467
wired pages = 55746
ubc pages = 542805
==================
Total = 770802
WIRED Pages Break Down:
vm wired pages = 7801
ubc wired pages = 14626
meta data pages = 7860
malloc pages = 17084
contig pages = 140
user ptepages = 7350
kernel ptepages = 229
free ptepages = 9
==================
Total = 55099
ida:root>
This is a trace from one of the processes that are in running state.
Many of them seams to do alot of this:
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
semctl (3, 1, 8, 0x1) = 0
semctl (3, 0, 8, 0x1) = 0
semop (3, 0x140049470, 1) = 0
semop (3, 0x140049460, 1) = 0
Mvh Klas Erlandsson
Received on Mon Jun 18 2001 - 14:05:07 NZST