LSPHP5 cpu issue

[MentaL]

Hello,

I'm running a rather large vbulletin forum with VBSEO but sadly it appears I am unable to get any stability going with litespeed. My CPU will often jump up from around 16 to 100+ or more and I'm unsure as of why, assistance would be appreciated.

top - 04:39:22 up 1 day, 18:42, 1 user, load average: 100.39, 52.30, 34.55
Tasks: 247 total, 94 running, 147 sleeping, 6 stopped, 0 zombie
Cpu(s): 80.8%us, 19.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8388608k total, 6365952k used, 2022656k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24388 nobody 16 0 86488 41m 3296 R 11.8 0.5 0:03.75 lsphp5
24523 nobody 16 0 90356 44m 3248 R 11.2 0.5 0:01.39 lsphp5
24395 nobody 15 0 70196 25m 3300 S 9.4 0.3 0:03.46 lsphp5
24409 nobody 15 0 74612 29m 3304 R 9.4 0.4 0:03.41 lsphp5
24467 nobody 15 0 69712 24m 3312 S 9.4 0.3 0:01.77 lsphp5
24540 nobody 15 0 76216 31m 3236 S 9.1 0.4 0:00.47 lsphp5
24359 nobody 16 0 87248 42m 3272 R 8.5 0.5 0:01.61 lsphp5
24399 nobody 16 0 89900 44m 3524 R 8.5 0.5 0:01.86 lsphp5
24562 nobody 16 0 84980 39m 3264 R 8.5 0.5 0:00.94 lsphp5
24490 nobody 16 0 77044 32m 3252 R 8.3 0.4 0:00.83 lsphp5
24363 nobody 16 0 82348 37m 3252 S 7.7 0.5 0:01.97 lsphp5
24396 nobody 16 0 89552 44m 3272 R 7.4 0.5 0:02.19 lsphp5
24419 nobody 16 0 90480 44m 3304 R 7.4 0.5 0:01.69 lsphp5
24477 nobody 15 0 69788 25m 3304 S 7.4 0.3 0:01.83 lsphp5
24486 nobody 15 0 71476 26m 3244 S 7.4 0.3 0:00.55 lsphp5
24487 nobody 16 0 93072 47m 3460 R 7.4 0.6 0:01.69 lsphp5
24502 nobody 16 0 72812 27m 3264 R 7.4 0.3 0:01.99 lsphp5
24390 nobody 16 0 55896 11m 3268 S 7.1 0.1 0:02.70 lsphp5
24452 nobody 15 0 72808 27m 3284 S 7.1 0.3 0:02.38 lsphp5
24473 nobody 15 0 85676 40m 3248 R 7.1 0.5 0:01.90 lsphp5
24521 nobody 16 0 73844 29m 3272 S 7.1 0.4 0:01.63 lsphp5
24526 nobody 16 0 86360 37m 3264 R 7.1 0.5 0:01.40 lsphp5
24357 nobody 15 0 86796 41m 3272 S 6.8 0.5 0:00.86 lsphp5
24385 nobody 16 0 93736 48m 3592 R 6.8 0.6 0:03.49 lsphp5
24484 nobody 16 0 75944 31m 3236 R 6.8 0.4 0:00.50 lsphp5
24531 nobody 15 0 84652 39m 3244 S 6.8 0.5 0:01.30 lsphp5
24376 nobody 16 0 95528 49m 3496 R 6.5 0.6 0:03.90 lsphp5
24383 nobody 15 0 82860 37m 3276 S 6.5 0.5 0:03.27 lsphp5
24425 nobody 16 0 91256 45m 3316 R 6.5 0.6 0:02.66 lsphp5
24465 nobody 16 0 77868 33m 3312 R 6.5 0.4 0:01.54 lsphp5
24514 nobody 15 0 72248 27m 3308 S 6.5 0.3 0:01.82 lsphp5
24518 nobody 16 0 82340 37m 3300 S 6.5 0.5 0:01.90 lsphp5
24522 nobody 15 0 92808 47m 3308 S 6.5 0.6 0:01.55 lsphp5
24534 nobody 16 0 85744 40m 3264 S 6.5 0.5 0:01.20 lsphp5
24364 nobody 16 0 82648 37m 3292 R 6.2 0.5 0:03.81 lsphp5
24371 nobody 15 0 83372 38m 3236 S 6.2 0.5 0:00.79 lsphp5
24433 nobody 16 0 91820 46m 3468 R 6.2 0.6 0:00.80 lsphp5
24441 nobody 15 0 89580 42m 3796 S 6.2 0.5 0:01.04 lsphp5
24480 nobody 16 0 88000 41m 3548 S 6.2 0.5 0:01.50 lsphp5
24517 nobody 16 0 93168 43m 3388 R 6.2 0.5 0:01.39 lsphp5
24393 nobody 15 0 82688 37m 3328 S 5.9 0.5 0:03.99 lsphp5
24405 nobody 15 0 75108 30m 3272 S 5.9 0.4 0:01.17 lsphp5
24447 nobody 16 0 91820 45m 3440 R 5.9 0.6 0:00.75 lsphp5
24474 nobody 15 0 68900 24m 3256 R 5.9 0.3 0:01.40 lsphp5
24512 nobody 16 0 88920 43m 3264 R 5.9 0.5 0:01.29 lsphp5
24513 nobody 16 0 87212 42m 3244 R 5.9 0.5 0:00.79 lsphp5

Eaccelerator version; 0.9.6.1
Memcached version; 2.2.5

Max Connections = 250
PHP_LSAPI_MAX_REQUESTS=1000
PHP_LSAPI_CHILDREN=250
timeout = 60
instances = 1

Kind regards.


####

[quote="stable" top stats]
top - 04:46:17 up 1 day, 18:49, 1 user, load average: 16.80, 28.29, 30.21
Tasks: 87 total, 10 running, 70 sleeping, 6 stopped, 1 zombie
Cpu(s): 53.6%us, 9.7%sy, 2.5%ni, 31.5%id, 2.7%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8388608k total, 1163840k used, 7224768k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4037 nobody 15 0 334m 46m 15m R 73.2 0.6 0:15.01 lsphp5
4034 nobody 16 0 340m 64m 30m R 48.7 0.8 0:10.35 lsphp5
4058 nobody 16 0 332m 53m 27m R 48.7 0.7 0:08.40 lsphp5
4001 nobody 16 0 333m 42m 12m R 48.0 0.5 0:09.41 lsphp5
5523 nobody 16 0 329m 43m 18m R 47.0 0.5 0:10.26 lsphp5
1327 nobody 16 0 333m 54m 25m R 43.7 0.7 0:21.34 lsphp5
3998 nobody 15 0 337m 44m 13m R 30.8 0.5 0:12.03 lsphp5
4032 nobody 15 0 317m 29m 15m S 26.2 0.4 0:14.50 lsphp5
1326 nobody 15 0 334m 48m 19m S 23.2 0.6 0:18.46 lsphp5
4040 nobody 25 10 331m 41m 13m R 19.9 0.5 0:08.42 lsphp5
4028 nobody 15 0 329m 42m 16m S 15.9 0.5 0:09.78 lsphp5
32764 nobody 16 0 338m 47m 15m R 15.2 0.6 0:19.43 lsphp5
3572 nobody 15 0 335m 45m 13m S 8.3 0.6 0:09.93 lsphp5
21659 nobody 15 0 337m 49m 18m S 8.3 0.6 0:26.75 lsphp5
31980 root 15 0 190m 70m 508 S 7.0 0.9 59:22.05 memcached
25820 nobody 15 0 27244 11m 3664 S 3.3 0.1 0:10.36 litespeed
3568 nobody 15 0 332m 43m 16m S 1.7 0.5 0:13.18 lsphp5
4039 nobody 15 0 334m 50m 19m S 1.3 0.6 0:10.38 lsphp5
29786 nobody 15 0 334m 46m 17m S 1.3 0.6 0:35.67 lsphp5

[/quote]

 

[NiteWave]

run "top -c", you may see vbseo is consuming most of the CPU

google search "vbseo slow", can find many discussion there.

if php in non-suExec mode, APC/Xcache may be better than eAccelerator. eAccelerator 0.9.6.1 don't support user variable cache, while APC/Xcache does.

some application(vbseo maybe,not sure) may utilize user variable cache for better performance.

 

[MentaL]

Hello

[root@XXX ~]# top -c
top - 08:45:37 up 1 day, 22:49, 1 user, load average: 8.18, 33.96, 43.86
Tasks: 77 total, 6 running, 62 sleeping, 9 stopped, 0 zombie
Cpu(s): 31.5%us, 6.4%sy, 3.3%ni, 58.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8388608k total, 1079172k used, 7309436k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16378 nobody 15 0 336m 61m 31m S 68.2 0.8 0:24.27 lsphp5:/home/XXX/public_html/forum/vbseo.php
16369 nobody 15 0 329m 53m 29m S 45.6 0.7 0:21.51 lsphp5:/home/XXX/public_html/forum/vbseo.php
17434 nobody 15 0 336m 44m 12m R 42.2 0.5 0:21.46 lsphp5:/home/XXX/public_html/forum/vbseo.php
18057 nobody 16 0 319m 25m 10m R 25.6 0.3 0:02.61 lsphp5:/home/XXX/public_html/forum/vbseo.php
17418 nobody 15 0 337m 44m 13m S 22.6 0.5 0:14.86 lsphp5
18023 nobody 16 0 332m 37m 11m R 19.0 0.5 0:04.49 lsphp5:/home/XXX/public_html/forum/vbseo.php
17411 nobody 25 10 329m 40m 15m S 17.0 0.5 0:22.74 lsphp5
17432 nobody 15 0 333m 43m 14m R 16.6 0.5 0:17.30 lsphp5:/home/XXX/public_html/forum/vbseo.php
16373 nobody 15 0 333m 46m 17m S 13.6 0.6 0:20.08 lsphp5
18025 nobody 15 0 330m 36m 11m S 13.3 0.5 0:07.42 lsphp5:/home/XXX/public_html/forum/vbseo.php

I'll look into your recommendations and thanks

 

[MentaL]

Hello

I tried APC but sadly always got 503 errors with VBSEO, I tried Xcache and this seems to work good so far, load is manageable (but I wont hold my breath).

I'm using my vbulletin database cache as Memcache and vbseo's caching options as Xcache so hopefully this will assist allot.


Thanks.

/EDIT

Still having load issues;

top - 10:11:45 up 7:07, 1 user, load average: 23.08, 11.99, 6.44
Tasks: 85 total, 18 running, 61 sleeping, 6 stopped, 0 zombie
Cpu(s): 90.0%us, 9.2%sy, 0.0%ni, 0.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8388608k total, 1447976k used, 6940632k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19502 nobody 15 0 169m 54m 28m S 39.9 0.7 0:16.15 lsphp5:/home/xxx/public_html/forum/vbseo.php
18247 nobody 17 0 221m 106m 29m R 33.3 1.3 0:22.97 lsphp5:/home/xxx/public_html/forum/vbseo.php
19506 nobody 15 0 172m 54m 26m S 32.9 0.7 0:18.29 lsphp5
17945 nobody 16 0 177m 53m 20m R 28.3 0.7 0:59.55 lsphp5:/home/xxx/public_html/forum/vbseo.php
19683 nobody 16 0 171m 39m 12m S 27.3 0.5 0:07.73 lsphp5
18251 nobody 15 0 184m 70m 32m R 25.6 0.9 0:24.34 lsphp5:/home/xxx/public_html/forum/vbseo.php
18178 nobody 15 0 175m 60m 28m S 25.3 0.7 0:27.46 lsphp5:/home/xxx/public_html/forum/vbseo.php
19671 nobody 15 0 172m 40m 12m S 24.6 0.5 0:07.31 lsphp5
19509 nobody 16 0 179m 49m 14m R 23.0 0.6 0:17.01 lsphp5:/home/xxx/public_html/forum/vbseo.php
19678 nobody 16 0 180m 61m 26m R 23.0 0.8 0:07.51 lsphp5:/home/xxx/public_html/forum/vbseo.php
19681 nobody 15 0 177m 45m 13m R 22.6 0.6 0:07.85 lsphp5:/home/xxx/public_html/forum/vbseo.php
19700 nobody 16 0 179m 47m 12m R 21.6 0.6 0:04.84 lsphp5:/home/xxx/public_html/forum/vbseo.php
19575 nobody 15 0 180m 47m 12m S 21.0 0.6 0:13.38 lsphp5
19679 nobody 15 0 178m 59m 25m S 21.0 0.7 0:07.09 lsphp5
19730 nobody 15 0 175m 42m 11m R 21.0 0.5 0:01.55 lsphp5:/home/xxx/public_html/forum/vbseo.php
19675 nobody 15 0 172m 50m 22m R 20.3 0.6 0:08.58 lsphp5:/home/xxx/public_html/forum/vbseo.php
19644 nobody 15 0 174m 45m 15m R 20.0 0.6 0:08.70 lsphp5:/home/xxx/public_html/forum/vbseo.php
19682 nobody 16 0 182m 48m 13m S 20.0 0.6 0:07.19 lsphp5
18219 nobody 15 0 179m 49m 14m S 19.6 0.6 0:25.33 lsphp5
16122 nobody 15 0 179m 70m 35m R 19.3 0.9 1:37.41 lsphp5:/home/xxx/public_html/forum/vbseo.php
16120 nobody 16 0 181m 59m 24m R 18.3 0.7 1:40.05 lsphp5:/home/xxx/public_html/forum/vbseo.php
19732 nobody 16 0 176m 44m 11m R 18.0 0.5 0:02.67 lsphp5:/home/xxx/public_html/forum/vbseo.php
19716 nobody 16 0 174m 42m 12m S 17.6 0.5 0:02.17 lsphp5
18186 nobody 15 0 180m 52m 18m S 17.3 0.6 0:27.62 lsphp5
18134 nobody 15 0 179m 51m 16m S 17.0 0.6 0:36.38 lsphp5
19571 nobody 15 0 179m 50m 15m S 17.0 0.6 0:11.24 lsphp5
19680 nobody 15 0 178m 45m 13m S 16.6 0.6 0:08.11 lsphp5
19670 nobody 15 0 171m 40m 13m S 15.0 0.5 0:08.60 lsphp5:/home/xxx/public_html/forum/vbseo.php
16121 nobody 16 0 182m 69m 34m R 14.3 0.9 1:40.47 lsphp5:/home/xxx/public_html/forum/vbseo.php
19684 nobody 16 0 180m 46m 13m R 14.0 0.6 0:07.30 lsphp5:/home/xxx/public_html/forum/vbseo.p

 

[NiteWave]

not sure if this helps:
http://www.vbseo.com/f77/100-cpu-usage-vbseo-24904/
"Changed Limit URL Length (was set to 255) to 0 - immediate drop in CPU load, application now performs exactly as expected."

 

[MentaL]

That didn't alter my load at all really, my current stats and have been around this for about 12 hours now except every half hour or so it will spike up to around 50.

top - 05:51:27 up 1 day, 2:47, 1 user, load average: 4.71, 6.83, 11.24
Tasks: 58 total, 5 running, 52 sleeping, 1 stopped, 0 zombie
Cpu(s): 29.8%us, 3.8%sy, 0.0%ni, 66.1%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8388608k total, 774932k used, 7613676k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21541 nobody 18 0 174m 41m 15m S 48.5 0.5 0:06.74 lsphp5
20452 nobody 15 0 179m 44m 14m R 46.9 0.5 0:20.03 lsphp5:/home/xxx/public_html/forum/index.php
21547 nobody 17 0 179m 46m 15m S 31.2 0.6 0:05.24 lsphp5
20447 nobody 15 0 178m 46m 18m S 30.6 0.6 0:24.62 lsphp5
20426 nobody 17 0 180m 57m 26m S 29.6 0.7 0:14.38 lsphp5
20436 nobody 16 0 179m 49m 18m R 23.9 0.6 0:12.20 lsphp5:/home/xxx/public_html/forum/vbseo.php
21543 nobody 15 0 179m 41m 11m R 23.3 0.5 0:03.30 lsphp5:/home/xxx/public_html/forum/vbseo.php
20467 nobody 15 0 181m 50m 18m S 10.3 0.6 0:10.04 lsphp5
17991 nobody 17 0 182m 64m 33m S 6.0 0.8 2:12.78 lsphp5
20453 nobody 15 0 175m 42m 15m S 5.6 0.5 0:10.62 lsphp5
13825 root 18 0 84796 5832 480 S 3.3 0.1 66:56.75 /usr/local/memcached/bin/memcached -d -m 2048 -l 127.0.0.1 -p 11211 -u root -c 15000 -P /tmp/memcached.pid
9819 nobody 16 0 27556 11m 1736 S 2.3 0.1 0:52.71 litespeed (lshttpd)

 

[MentaL]

Here is a random spike;

top - 06:10:33 up 1 day, 3:06, 1 user, load average: 63.60, 42.73, 24.01
Tasks: 232 total, 7 running, 223 sleeping, 2 stopped, 0 zombie
Cpu(s): 48.8%us, 8.1%sy, 0.2%ni, 42.7%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8388608k total, 5776260k used, 2612348k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13825 root 15 0 84796 5840 480 S 34.8 0.1 68:24.58 /usr/local/memcached/bin/memcached -d -m 2048 -l 127.0.0.1 -p 11211 -u root -c 15000 -P /tmp/memcached.pid
27771 nobody 17 0 175m 33m 7404 S 34.8 0.4 0:00.18 lsphp5:/home/xxx/public_html/forum/vbseo.php
27774 nobody 16 0 174m 32m 7408 S 34.8 0.4 0:00.18 lsphp5:/home/xxx/public_html/forum/vbseo.php
27769 nobody 16 0 176m 35m 9.8m S 30.9 0.4 0:00.16 lsphp5:/home/xxx/public_html/forum/vbseo.php
27770 nobody 16 0 174m 32m 7416 S 27.1 0.4 0:00.14 lsphp5:/home/xxx/public_html/forum/vbseo.php
27773 nobody 17 0 173m 30m 6616 R 27.1 0.4 0:00.14 lsphp5:/home/xxx/public_html/forum/vbseo.php
27765 nobody 15 0 174m 32m 7580 S 25.1 0.4 0:00.16 lsphp5:/home/xxx/public_html/forum/vbseo.php
27768 nobody 17 0 174m 32m 7112 R 25.1 0.4 0:00.13 lsphp5:/home/xxx/public_html/forum/vbseo.php
27775 nobody 17 0 173m 29m 6228 R 23.2 0.4 0:00.12 lsphp5:/home/xxx/public_html/forum/vbseo.php
27766 nobody 15 0 174m 32m 7288 S 21.3 0.4 0:00.14 lsphp5:/home/xxx/public_html/forum/vbseo.php
27772 nobody 18 0 170m 26m 6076 R 19.3 0.3 0:00.10 lsphp5:/home/xxx/public_html/forum/vbseo.php
27776 nobody 18 0 169m 25m 6008 R 19.3 0.3 0:00.10 lsphp5:/home/xxx/public_html/forum/cron.php
27777 nobody 17 0 164m 20m 5400 R 15.5 0.3 0:00.08 lsphp5:/home/xxx/public_html/forum/vbseo.php
27763 nobody 15 0 177m 33m 7996 S 11.6 0.4 0:00.15 lsphp5:/home/xxx/public_html/forum/vbseo.php
24553 nobody 15 0 181m 57m 26m S 7.7 0.7 0:36.07 lsphp5:/home/xxx/public_html/forum/vbseo.php
25682 nobody 15 0 181m 48m 17m S 7.7 0.6 0:20.21 lsphp5:/home/xxx/public_html/forum/vbseo.php
25990 nobody 15 0 182m 46m 13m S 7.7 0.6 0:14.89 lsphp5:/home/xxx/public_html/forum/vbseo.php
26254 nobody 15 0 179m 43m 12m S 7.7 0.5 0:05.81 lsphp5:/home/xxx/public_html/forum/vbseo.php
27764 nobody 16 0 174m 32m 7300 S 7.7 0.4 0:00.13 lsphp5:/home/xxx/public_html/forum/vbseo.php
24447 nobody 15 0 183m 56m 23m S 5.8 0.7 0:46.80 lsphp5:/home/xxx/public_html/forum/vbseo.php
24546 nobody 15 0 176m 54m 26m S 5.8 0.7 0:30.75 lsphp5:/home/xxx/public_html/forum/vbseo.php
25667 nobody 15 0 180m 44m 12m S 5.8 0.5 0:22.08 lsphp5:/home/xxx/public_html/forum/vbseo.php
25813 nobody 15 0 175m 39m 13m S 5.8 0.5 0:22.73 lsphp5:/home/xxx/public_html/forum/vbseo.php
25971 nobody 15 0 182m 45m 14m S 5.8 0.6 0:15.92 lsphp5:/home/xxx/public_html/forum/vbseo.php
25979 nobody 15 0 182m 44m 13m S 5.8 0.5 0:15.96 lsphp5:/home/xxx/public_html/forum/vbseo.php
26078 nobody 15 0 179m 53m 23m S 5.8 0.7 0:13.64 lsphp5:/home/xxx/public_html/forum/vbseo.php
26119 nobody 15 0 180m 45m 14m S 5.8 0.6 0:10.41 lsphp5:/home/xxx/public_html/forum/vbseo.php
26135 nobody 15 0 176m 52m 25m S 5.8 0.6 0:11.13 lsphp5:/home/xxx/public_html/forum/vbseo.php
26208 nobody 15 0 179m 56m 25m S 5.8 0.7 0:05.90 lsphp5:/home/xxx/public_html/forum/vbseo.php
26246 nobody 15 0 179m 42m 11m S 5.8 0.5 0:04.94 lsphp5:/home/xxx/public_html/forum/vbseo.php
26259 nobody 15 0 175m 38m 12m S 5.8 0.5 0:03.92 lsphp5:/home/xxx/public_html/forum/vbseo.php
26261 nobody 15 0 179m 42m 11m S 5.8 0.5 0:04.04 lsphp5:/home/xxx/public_html/forum/vbseo.php
26275 nobody 15 0 179m 42m 12m S 5.8 0.5 0:04.80 lsphp5:/home/xxx/public_html/forum/vbseo.php
26280 nobody 15 0 179m 39m 9620 S 5.8 0.5 0:03.33 lsphp5:/home/xxx/public_html/forum/vbseo.php
26282 nobody 15 0 178m 41m 11m S 5.8 0.5 0:04.82 lsphp5:/home/xxx/public_html/forum/vbseo.php
26294 nobody 15 0 182m 46m 12m S 5.8 0.6 0:03.89 lsphp5:/home/xxx/public_html/forum/vbseo.php
26337 nobody 15 0 181m 42m 12m S 5.8 0.5 0:03.36 lsphp5:/home/xxx/public_html/forum/vbseo.php
26389 nobody 15 0 179m 39m 9660 S 5.8 0.5 0:03.11 lsphp5:/home/xxx/public_html/forum/vbseo.php
26390 nobody 15 0 179m 45m 14m S 5.8 0.6 0:04.12 lsphp5:/home/xxx/public_html/forum/vbseo.php
26469 nobody 15 0 179m 39m 9624 S 5.8 0.5 0:01.30 lsphp5:/home/xxx/public_html/forum/vbseo.php
27762 nobody 15 0 174m 32m 7288 S 5.8 0.4 0:00.12 lsphp5:/home/xxx/public_html/forum/vbseo.php
24452 nobody 15 0 179m 48m 17m S 3.9 0.6 0:48.70 lsphp5:/home/xxx/public_html/forum/vbseo.php
24506 nobody 15 0 177m 43m 16m S 3.9 0.5 0:40.22 lsphp5:/home/xxx/public_html/forum/vbseo.php
24543 nobody 15 0 185m 54m 18m S 3.9 0.7 0:27.57 lsphp5:/home/xxx/public_html/forum/vbseo.php
24544 nobody 15 0 181m 47m 16m S 3.9 0.6 0:32.40 lsphp5:/home/xxx/public_html/forum/vbseo.php
24552 nobody 15 0 182m 49m 18m S 3.9 0.6 0:27.81 lsphp5:/home/xxx/public_html/forum/vbseo.php
25955 nobody 15 0 184m 49m 15m S 3.9 0.6 0:13.12 lsphp5:/home/xxx/public_html/forum/vbseo.php
25988 nobody 15 0 180m 51m 19m S 3.9 0.6 0:13.21 lsphp5:/home/xxx/public_html/forum/vbseo.php
26069 nobody 15 0 179m 43m 13m S 3.9 0.5 0:11.55 lsphp5:/home/xxx/public_html/forum/vbseo.php

/EDIT

Upon checking the connections it appears that our SQL server (thats located on a different machine) is appearing to cause the spike,

[root@xxx lib]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
766 SQL.IP.ADDRESS

 

[NiteWave]

you're using memcache:
http://www.php.net/manual/en/book.memcache.php
not sure if memcached is ok and have any improvement:
http://www.php.net/manual/en/book.memcached.php

 

[MentaL]

Well it seems that the load is correcting itself...

top - 12:17:41 up 2 days, 2:06, 1 user, load average: 0.38, 0.61, 0.71
Tasks: 55 total, 4 running, 49 sleeping, 2 stopped, 0 zombie
Cpu(s): 43.4%us, 6.6%sy, 0.2%ni, 49.5%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8388608k total, 637672k used, 7750936k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13716 nobody 17 0 183m 66m 34m R 69.5 0.8 0:46.05 lsphp5
14055 nobody 16 0 186m 69m 34m R 69.5 0.9 0:20.02 lsphp5
13576 nobody 16 0 185m 71m 36m R 51.6 0.9 0:52.39 lsphp5
13578 nobody 15 0 185m 71m 37m S 41.7 0.9 0:40.28 lsphp5
14056 nobody 15 0 187m 55m 19m S 31.8 0.7 0:26.87 lsphp5
13825 root 15 0 148m 7408 484 S 7.9 0.1 128:32.45 memcached

 

[MentaL]

Meh, It's just randomly decided to play up again.

top - 13:53:52 up  3:58,  1 user,  load average: 39.48, 17.76, 11.16
Tasks: 158 total,  59 running,  84 sleeping,  15 stopped,   0 zombie
Cpu(s): 86.7%us, 13.1%sy,  0.0%ni,  0.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8388608k total,  3193808k used,  5194800k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27767 nobody    15   0  270m  44m  14m S 16.2  0.5   0:23.71 lsphp5
26029 nobody    16   0  270m  58m  31m R 10.7  0.7   0:16.62 lsphp5:/home/xxx/public_html/forum/vbseo.php
27777 nobody    16   0  276m  51m  16m R 10.7  0.6   0:18.04 lsphp5:/home/xxx/public_html/forum/vbseo.php
22220 nobody    16   0  280m  79m  43m R 10.1  1.0   0:49.18 lsphp5:/home/xxx/public_html/forum/vbseo.php
27761 nobody    15   0  273m  57m  27m S  9.8  0.7   0:18.33 lsphp5:/home/xxx/public_html/forum/vbseo.php
28355 nobody    15   0  267m  36m  11m S  9.2  0.5   0:03.88 lsphp5:/home/xxx/public_html/forum/vbseo.php
28012 nobody    15   0  267m  51m  25m S  8.5  0.6   0:11.19 lsphp5:/home/xxx/public_html/forum/vbseo.php
28593 nobody    15   0  269m  40m  11m S  8.5  0.5   0:01.73 lsphp5:/home/xxx/public_html/forum/vbseo.php
28598 nobody    15   0  268m  38m  11m S  8.2  0.5   0:01.53 lsphp5:/home/xxx/public_html/forum/vbseo.php
28023 nobody    15   0  270m  41m  14m S  7.9  0.5   0:16.75 lsphp5:/home/xxx/public_html/forum/vbseo.php
28547 nobody    16   0  270m  41m  11m R  7.9  0.5   0:01.93 lsphp5:/home/xxx/public_html/forum/vbseo.php
28549 nobody    15   0  271m  41m  11m S  7.9  0.5   0:02.12 lsphp5:/home/xxx/public_html/forum/vbseo.php
28631 nobody    15   0  267m  38m  11m S  7.9  0.5   0:01.34 lsphp5:/home/xxx/public_html/forum/vbseo.php
28528 nobody    16   0  268m  39m  11m S  7.6  0.5   0:02.52 lsphp5
28566 nobody    15   0  266m  34m 9480 S  7.6  0.4   0:01.25 lsphp5:/home/xxx/public_html/forum/vbseo.php
28588 nobody    15   0  274m  44m  11m S  7.6  0.5   0:02.09 lsphp5:/home/xxx/public_html/forum/vbseo.php
28633 nobody    16   0  263m  31m 9476 R  7.6  0.4   0:01.00 lsphp5:/home/xxx/public_html/forum/vbseo.php
28286 nobody    16   0  271m  42m  12m R  7.3  0.5   0:06.81 lsphp5:/home/xxx/public_html/forum/vbseo.php
28530 nobody    16   0  271m  41m  11m R  7.3  0.5   0:02.48 lsphp5:/home/xxx/public_html/forum/vbseo.php
26030 nobody    16   0  268m  60m  33m R  7.0  0.7   0:35.11 lsphp5:/home/xxx/public_html/forum/vbseo.php
28009 nobody    16   0  267m  40m  14m R  7.0  0.5   0:09.09 lsphp5:/home/xxx/public_html/forum/vbseo.php
28553 nobody    16   0  274m  45m  12m R  7.0  0.6   0:01.70 lsphp5:/home/xxx/public_html/forum/vbseo.php
28564 nobody    16   0  274m  44m  11m R  7.0  0.5   0:01.53 lsphp5:/home/xxx/public_html/forum/vbseo.php
28664 nobody    15   0  263m  31m 9456 R  7.0  0.4   0:00.66 lsphp5:/home/xxx/public_html/forum/vbseo.php
22172 nobody    16   0  272m  69m  40m R  6.7  0.9   1:24.27 lsphp5:/home/xxx/public_html/forum/vbseo.php
28253 nobody    16   0  271m  42m  12m R  6.7  0.5   0:06.98 lsphp5:/home/xxx/public_html/forum/vbseo.php
28531 nobody    15   0  271m  41m  11m S  6.7  0.5   0:02.10 lsphp5:/home/xxx/public_html/forum/vbseo.php
28567 nobody    15   0  267m  37m  10m S  6.7  0.5   0:01.26 lsphp5:/home/xxx/public_html/forum/vbseo.php
28666 nobody    15   0  266m  34m 9468 R  6.7  0.4   0:00.41 lsphp5:/home/xxx/public_html/forum/vbseo.php
28667 nobody    16   0  263m  31m 9604 R  6.7  0.4   0:00.65 lsphp5:/home/xxx/public_html/forum/vbseo.php

It will average between 30-100, nothing has been altered, I'm using memcached and xcache.

 

[NiteWave]

can you strace one of lsphp5 process which has run longest time, in your post, this one:
22172 nobody 16 0 272m 69m 40m R 6.7 0.9 1:24.27 lsphp5:/home/xxx/public_html/forum/vbseo.php

and see what it's doing.

 

[MentaL]

Its just outputting allot of info from vb.

fcntl(5, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(5, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
lseek(22, -4096, SEEK_CUR)              = 0
close(22)                               = 0
munmap(0x2abf6736d000, 4096)            = 0
poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "X\0\0\0\3\n\t\t\tSELECT blockid, require"..., 92) = 92
read(21, "\1\0\0\1\2[\0\0\2\3def\21xxx_database\23"..., 16384) = 221
poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "\26\1\0\0\3\n\t\t\tSELECT pf.profilefieldc"..., 282) = 282
read(21, "\1\0\0\1\25a\0\0\2\3def\21xxx_database\2"..., 16384) = 1908
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum/includes", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
lstat("/home/xxx/public_html/forum/includes/class_profileblock_blog.php", {st_mode=S_IFREG|0644, st_size=20646, ...}) = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum/includes", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
lstat("/home/xxx/public_html/forum/includes/class_profileblock_blog.php", {st_mode=S_IFREG|0644, st_size=20646, ...}) = 0
open("/home/xxx/public_html/forum/includes/class_profileblock_blog.php", O_RDONLY) = 22
fstat(22, {st_mode=S_IFREG|0644, st_size=20646, ...}) = 0
fcntl(22, F_GETFL)                      = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(22, {st_mode=S_IFREG|0644, st_size=20646, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2abf6736d000
lseek(22, 0, SEEK_CUR)                  = 0
lseek(22, 0, SEEK_SET)                  = 0
read(22, "<?php\n/*========================"..., 4096) = 4096
lseek(22, 4096, SEEK_SET)               = 4096
stat("/home/xxx/public_html/forum/includes/class_profileblock_blog.php", {st_mode=S_IFREG|0644, st_size=20646, ...}) = 0
fcntl(5, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(5, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
lseek(22, -4096, SEEK_CUR)              = 0
close(22)                               = 0
munmap(0x2abf6736d000, 4096)            = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum/includes", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
lstat("/home/xxx/public_html/forum/includes/blog_functions.php", {st_mode=S_IFREG|0644, st_size=105536, ...}) = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum/includes", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
lstat("/home/xxx/public_html/forum/includes/blog_functions.php", {st_mode=S_IFREG|0644, st_size=105536, ...}) = 0
open("/home/xxx/public_html/forum/includes/blog_functions.php", O_RDONLY) = 22
fstat(22, {st_mode=S_IFREG|0644, st_size=105536, ...}) = 0
fcntl(22, F_GETFL)                      = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(22, {st_mode=S_IFREG|0644, st_size=105536, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2abf6736d000
lseek(22, 0, SEEK_CUR)                  = 0
lseek(22, 0, SEEK_SET)                  = 0
read(22, "<?php\n/*========================"..., 4096) = 4096
lseek(22, 4096, SEEK_SET)               = 4096
stat("/home/xxx/public_html/forum/includes/blog_functions.php", {st_mode=S_IFREG|0644, st_size=105536, ...}) = 0
fcntl(3, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(3, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
lseek(22, -4096, SEEK_CUR)              = 0
close(22)                               = 0
munmap(0x2abf6736d000, 4096)            = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/xxx/public_html/forum/includes", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
lstat("/home/xxx/public_html/forum/includes/functions_xperience.php", {st_mode=S_IFREG|0644, st_size=43278, ...}) = 0
open("/home/xxx/public_html/forum/includes/functions_xperience.php", O_RDONLY) = 22
fstat(22, {st_mode=S_IFREG|0644, st_size=43278, ...}) = 0
fcntl(22, F_GETFL)                      = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(22, {st_mode=S_IFREG|0644, st_size=43278, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2abf6736d000
lseek(22, 0, SEEK_CUR)                  = 0
lseek(22, 0, SEEK_SET)                  = 0
read(22, "<?php\n/*========================"..., 4096) = 4096
lseek(22, 4096, SEEK_SET)               = 4096
stat("/home/xxx/public_html/forum/includes/functions_xperience.php", {st_mode=S_IFREG|0644, st_size=43278, ...}) = 0
fcntl(3, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(3, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
lseek(22, -4096, SEEK_CUR)              = 0
close(22)                               = 0
munmap(0x2abf6736d000, 4096)            = 0
poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "\221\0\0\0\3\n\t\tSELECT COUNT(DISTINCT(us"..., 149) = 149
read(21, "\1\0\0\1\3\37\0\0\2\3def\0\0\0\tusercount\0\f?\0\25\0"..., 16384) = 165
poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "\205\0\0\0\3\n\t\tSELECT COUNT(*) AS userc"..., 137) = 137
read(21, "\1\0\0\1\2\37\0\0\2\3def\0\0\0\tusercount\0\f?\0\25\0"..., 16384) = 127
poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "\205\0\0\0\3\n\t\tSELECT COUNT(*) AS userc"..., 137) = 137
read(21, "\1\0\0\1\2\37\0\0\2\3def\0\0\0\tusercount\0\f?\0\25\0"..., 16384) = 127
poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "\205\0\0\0\3\n\t\tSELECT COUNT(*) AS userc"..., 137) = 137
read(21, "\1\0\0\1\2\37\0\0\2\3def\0\0\0\tusercount\0\f?\0\25\0"..., 16384) = 127

Update;

It seems to be correcting itself again, allthough I made two alterations, I disabled vbulletins live topic and vboptimize, given that the optimize was already built into the configuration of vbulletin (memcached) and using xcache with vbseo , it appears to be "ok" as of this moment, but I'm sure it will go haywire shortly.

top - 05:50:47 up  1:21,  1 user,  load average: 2.40, 3.69, 9.94
Tasks:  71 total,   2 running,  60 sleeping,   9 stopped,   0 zombie
Cpu(s): 17.1%us,  3.0%sy,  1.8%ni, 78.0%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8388608k total,   991484k used,  7397124k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29766 nobody    15   0  271m  46m  16m S 54.9  0.6   0:02.71 lsphp5
17722 nobody    15   0  272m  76m  47m S 35.0  0.9   2:35.97 lsphp5
19916 nobody    16   0  277m  80m  46m R 30.0  1.0   3:08.23 lsphp5
14326 nobody    15   0  271m  70m  42m S 20.0  0.9   3:15.78 lsphp5
15846 nobody    25  10  283m  87m  47m S 17.6  1.1   4:23.42 lsphp5
20004 nobody    15   0  269m  75m  49m S 14.3  0.9   2:27.27 lsphp5
 1514 nobody    15   0 26976  10m 1580 S  1.0  0.1   1:15.07 litespeed
 9960 root      15   0 88060 3348 2584 S  0.3  0.0   0:01.46 sshd
21977 root      15   0 81580  28m  500 S  0.3  0.3   2:58.26 memcached
26252 root      15   0 12724 1260  924 R  0.3  0.0   0:00.81 top

 

[NiteWave]

poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "\221\0\0\0\3\n\t\tSELECT COUNT(DISTINCT(us"..., 149) = 149
read(21, "\1\0\0\1\3\37\0\0\2\3def\0\0\0\tusercount\0\f?\0\25\0"..., 16384) = 165
poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "\205\0\0\0\3\n\t\tSELECT COUNT(*) AS userc"..., 137) = 137
read(21, "\1\0\0\1\2\37\0\0\2\3def\0\0\0\tusercount\0\f?\0\25\0"..., 16384) = 127
poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "\205\0\0\0\3\n\t\tSELECT COUNT(*) AS userc"..., 137) = 137
read(21, "\1\0\0\1\2\37\0\0\2\3def\0\0\0\tusercount\0\f?\0\25\0"..., 16384) = 127
poll([{fd=21, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(21, "\205\0\0\0\3\n\t\tSELECT COUNT(*) AS userc"..., 137) = 137
read(21, "\1\0\0\1\2\37\0\0\2\3def\0\0\0\tusercount\0\f?\0\25\0"..., 16384) = 127

please note those lines with "Timeout", I think this may be an important clue. Many lsphp5 process are waiting something and get timeout, this will cause high load. I guess maybe waiting for responses from MySQL. most likely code bug somewhere, cause high load under some conditions.