Need some tips for tuning the web server

[canglan]

Hi mistwang,

I think I need some more help.

Our server has just had a huge load spike, lsphp5 was taking up all the CPU, and made the server unresponsive, couldn't even ping or SSH into the server.



The server log just before the huge spike (domains are masked):

2009-09-07 03:01:52.427 [INFO] [70.32.68.11:54866-0#APVH_xxx.xxx.xxx] connection to [/tmp/lshttpd/lsphp5.sock.582] on request #25, confirmed, 0, associated process: 5282, running: 1, error: Connection reset by peer!
2009-09-07 03:02:42.006 [INFO] [64.127.102.82:17873-0#APVH_xxx.xxx.xxx] Connection idle time: 11 while in state: 4 watching for event: 25,close!
2009-09-07 03:02:42.006 [NOTICE] [64.127.102.82:17873-0#APVH_xxx.xxx.xxx] Content len: 0, Request line:
2009-09-07 03:02:42.006 [NOTICE] [64.127.102.82:17873-0#APVH_xxx.xxx.xxx] Redirect: #1, URL: /blog/index.php
2009-09-07 03:05:14.699 [INFO] [62.25.109.195:4123-0#APVH_xxx.xxx.xxx] connection to [/tmp/lshttpd/lsphp5.sock.582] on request #1, confirmed, 0, associated process: 5282, running: 1, error: Connection reset by peer!
2009-09-07 03:07:17.530 [INFO] [209.249.84.5:51438-0#APVH_xxx.xxx.xxx] connection to [/tmp/lshttpd/lsphp5.sock.582] on request #8, confirmed, 0, associated process: 5282, running: 1, error: Connection reset by peer!
2009-09-07 03:08:19.849 [INFO] [119.107.82.136:52045-0#APVH_xxx.xxx.xxx] connection to [/tmp/lshttpd/lsphp5.sock.582] on request #15, confirmed, 0, associated process: 5282, running: 1, error: Connection reset by peer!
2009-09-07 03:09:56.439 [INFO] [195.88.32.160:2184-0#APVH_xxx.xxx.xxx] Connection idle time: 11 while in state: 4 watching for event: 25,close!
2009-09-07 03:09:56.439 [NOTICE] [195.88.32.160:2184-0#APVH_xxx.xxx.xxx] Content len: 555, Request line:
POST /wp-comments-post.php HTTP/1.1


Any ideas what could have caused such a dramatic issue?

[auser]

have you tried to disable some wordpress plugins, e.g. W3 total cache, see if any difference?

for lsphp5(pid=25210) has 320% cpu usage, can trace it to see if it's blocked by whatever system call:

~>strace -c -p 25210

run it for some seconds, then Ctrl+C, to check the output.

[canglan]

Hi auser, thanks for the tips. We are actually working with the author of W3 Total Cache to solve some potential issues.

Unfortunately we can't trace that pid because the server was completely dead and we had to switch it to rescue mode and bring it back. By then the process was no longer exist.

[canglan]

Looks like there wasn't enough memory allocated to lsphp5, I didn't realise APC shm size is counted towards the memory limit. I have increased the memory limit now and hopefully this problem will go away. :-)