Dead slow response from server

[dotpro]

Hi,

I've got an issue on our server and looking for some solutions, please.

I have several websites hosted on the server, running:

CPU: Dual Processor Quad Core Xeon 5520 - 2.26GHz (Nehalem) - 2 x 8MB cache w/HT
Memory: 12GB ram
hard drive type/configuration: 4x 147GB SAS 15k in RAID-10, Adaptec 5405 Card
Linux: CENTOS 5.8 x86_64 standard
Webserver: Litespeed Web Server - Enterprise Edition - Version 4.2.1
PHP version: PHP 5.3.8
MySQL version: 5.0.96

When I first installed litespeed, switching from apache, the speeds were very good, performance was good, pages loaded fast! Recently, over the past couple of months, two high traffic websites on the same server have started showing performance degradation and slowness = a lot of users complaining.

All websites used wordpress, vbulletin with pretty much same configuration and other websites are loading just fine, no hangup, no waiting for response.

I have w3 cache and cdn enabled for all of them as well... as I said, it was good but not anymore. I can't figure out what is causing two websites (one wordpress+vbulletin and other is just vbulletin forum) but the loading times are killing me!!!

I see mysql cpu high sometimes but that should effect all websites not just two, no? Plus, server load could be in single digits and still same slowness.

If I enable apache (switching back to apache) the server just stops responding, the server load goes high into 100, 150+ and then its really dead.

Any help would be appreciated, thanks

 

[NiteWave]

can you post "top -c" result to know more detail?
also if possible, screenshot of real-time stats.

how about temporarily disable w3 cache for the problem site for comparison ?

 

[dotpro]

can you post "top -c" result to know more detail?
also if possible, screenshot of real-time stats.

how about temporarily disable w3 cache for the problem site for comparison ?

Hi Nitewave,

Here is Top -c :

Tasks: 314 total, 2 running, 312 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.4%us, 40.4%sy, 0.0%ni, 49.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 12290888k total, 11081368k used, 1209520k free, 875148k buffers
Swap: 2096440k total, 147016k used, 1949424k free, 6304768k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13729 mysql 5 -10 2106m 1.8g 3780 S 738.9 15.4 5987:00 /usr/sbin/mysqld --basedir=/ --datadir=/home/mysql --user=mysql --pid-
9675 nobody 16 0 329m 101m 36m S 44.5 0.8 0:02.56 lsphp5:e/autotalk/public_html/pictures/showphoto.php
8490 nobody 15 0 278m 111m 90m S 10.3 0.9 0:38.61 lsphp5:nce/public_html/wp-content/w3tc/min/index.php
9618 nobody 15 0 329m 102m 37m S 5.3 0.9 0:03.61 lsphp5
7070 nobody 16 0 281m 118m 94m S 4.6 1.0 1:23.77 lsphp5:me/defence/public_html/pictures/showphoto.php
8777 nobody 15 0 280m 105m 80m S 4.6 0.9 0:27.27 lsphp5:nce/public_html/wp-content/w3tc/min/index.php
9689 nobody 17 0 263m 16m 7416 S 2.0 0.1 0:00.06 lsphp5:me/defence/public_html/pictures/showphoto.php
9659 root 15 0 12872 1276 812 R 0.3 0.0 0:00.04 top -c
13930 named 20 0 366m 6688 2008 S 0.3 0.1 0:51.91 /usr/sbin/named -u named
17164 nobody 0 -19 66816 20m 4312 S 0.3 0.2 0:28.14 litespeed (lshttpd)
1 root 15 0 10348 540 504 S 0.0 0.0 0:58.37 init [3]
2 root RT -5 0 0 0 S 0.0 0.0 0:00.17 [migration/0]
3 root 34 19 0 0 0 S 0.0 0.0 0:00.44 [ksoftirqd/0]
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/0]
5 root RT -5 0 0 0 S 0.0 0.0 0:50.74 [migration/1]
6 root 34 19 0 0 0 S 0.0 0.0 0:00.53 [ksoftirqd/1]
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/1]
8 root RT -5 0 0 0 S 0.0 0.0 0:40.10 [migration/2]


I have already tried disabling w3 total cache, no help. It was done during the slow period. I also tried disabling all plugins, no help either. The other website which doesn't seem to have this issue also has vbulletin, wordpress and w2 total cache.

Real time stats attached

thanks

 

[mistwang]

You should trouble shoot mysql performance. for the two heavy traffic site, the DB may be corrupted, or missing necessary index.
Please enable MySQL slow query cache and slow query log to identify the bottle neck.

 

[dotpro]

Thanks.

As of now, the websites are loading fine... I did not make any change yet.

I will look into mysql though.