1-2 minute no response from server

Discussion in 'PHP' started by inxile, Apr 12, 2011.

  1. inxile

    inxile New Member

    I'm trying to debug an issue with our server (CentOS 5.4, LiteSpeed 4.0.12 and PHP 5.3.1). Every day, as often as every 10-15 minutes, we see an unresponsive request for a web page lasting between 1-2 minutes. After that the page loads as normal. Sometimes the server is slow and continues to be for a few minutes, other times it is as fast as normal after that.

    Logs show dozens, sometimes hundreds of 'lsphp5' processes. It appears as if either the memory or processing is nearly maxed out.

    I could use some help trying to find out what exactly is causing this issue. Is it a configuration issue, or a known issue with v4.0.12?

    Any help in determining where the issue could be would be very much appreciated!
  2. NiteWave

    NiteWave Administrator

  3. inxile

    inxile New Member

    EAProc Idle

    In the console I show nearly 400 EAProc Idle. What determines this number and what settings can I adjust to keep this lower? I'm assuming this should be much lower. I can't imagine 400 idle processes is optimal?
  4. NiteWave

    NiteWave Administrator

    yes, it's not normal. something wrong of your php.

    please check if there are too many php session files in /tmp. and refer the blog post I posted last time. a system bottle neck(for example, slow /tmp partition) can cause php process fork and fork, until system resource is used up.
  5. cmanns

    cmanns New Member

    me too

    I get this randomly too.

    My system never swaps and tmp never gets more then 40% or so. However our software RAID1 is maxed, going to RAID10 w/more ram soon.

    The issue seems LiteSpeed closes the PHP Processes before XCache can even cache, so our cpu gets ate up by dozens of 5% PHP processes?
  6. mistwang

    mistwang LiteSpeed Staff

    If you use PHP suEXEC, you can consider using eAccelerator with storage path under /dev/shm.
  7. inxile

    inxile New Member

    Still having issues. Can't seem to find any outside sources causing the issues. We've upgraded the OS, LiteSpeed and PHP all to current versions.

    Is there a place that recommends config settings for LiteSpeed in a high volume PHP production environment?
  8. inxile

    inxile New Member

    There were about 3500. For a server with 4G ram, what would be the optimal settings for memory usage, connections, etc?
  9. mistwang

    mistwang LiteSpeed Staff

    when it happens, check server memory usage, CPU usage and I/O wait.
    If you have spare memory left, put php sessions to /dev/shm.
    It could be a MySQL issue, check mysql status with

    mysqladmin processlist
  10. inxile

    inxile New Member

    Yeah, MySQL has always been a minimal impact and shows no sign of being taxed.

    This is really becoming an issue and our hosting provider has been unable to find any solutions or even causes. Unfortunately this was all setup three or four people before me so I have no idea what they did that might have been different or not suggested/default.

    I noticed you offer 1 hour support, 15 hours of support and an anti-DDoS/tuning. Which or these would be best for taking a look at our setup and fine tuning it?
  11. webizen

    webizen New Member

    You can order 1 hour support when issue arises.
  12. inxile

    inxile New Member

    Here is a set of stats when an issue arose:


    Here is the monitoring stats:

    kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
    01:00:01 PM 3109568 1038472 25.04 23304 635420 1995456 101016 4.82 23360
    01:05:01 PM 3289428 858612 20.70 28836 532984 1995500 100972 4.82 23688
    01:10:09 PM 152268 3995772 96.33 10612 1402236 1499320 597152 28.48 46508
    01:15:08 PM 175720 3972320 95.76 14028 1198712 796212 1300260 62.02 338488
    01:20:08 PM 410212 3737828 90.11 14292 1149120 813132 1283340 61.21 283320

    What config settings would be best to adjust when in this kind of scenario?
  13. inxile

    inxile New Member

    high EAProc WaitQ

    We've been making a number of corrections and tweaks to the system and things have gotten better.

    However, we continue to experience a complete hang of the system from time to time... about 15-20 minutes or half hour or so. The website times out and is unresponsive for about a minute, then all is fine.

    During this time we see the following:

    EAProc WaitQ is in the mid 100's
    EAProc In Use is less than 10
    Req in Processing is slightly higher than EAProc WaitQ, mid 100's

    It appears settings are throttled a little to tight, what settings should I be looking to to adjust to relieve the EAProc WaitQ?

    Follow Up: Logs are stating the PHP suEXEC Max Conn 512 is invalid and are defaulting to 5. What are valid values?
    Last edited: May 2, 2011
  14. webizen

    webizen New Member

    PHP suEXEC Max Conn sets to 5-10 normally.

    You should adjust Max Connections in lsphp5 (in External App): bump it up by 50 at a time (environment variable PHP_LSAPI_CHILDREN needs to be set the same as Max Connections).
  15. inxile

    inxile New Member

    Currently,

    General >>
    PHP suEXEC Max Conn: 499

    lsphp5 >>
    Max Connections: 20
    PHP_LSAPI_MAX_REQUESTS=256
    PHP_LSAPI_CHILDREN=20
  16. webizen

    webizen New Member

    change to following to begin with:

    General >>
    PHP suEXEC Max Conn: 10

    lsphp5 >>
    Max Connections: 50
    PHP_LSAPI_MAX_REQUESTS=512
    PHP_LSAPI_CHILDREN=50
  17. inxile

    inxile New Member

    After about 10 minutes it happened again... a spike in WaitQ and site was unresponsive. I upped the max conn and PHP_LSAPI_CHILDREN to 100 and restarted. Took a minute for the WaitQ to die down again. Now 5 minutes later its tapped out again.

    We average about 10 million hits a month.
  18. webizen

    webizen New Member

    try PHP suEXEC Max Conn: 10 => 200.

    BTW, what's your server config (OS, RAM size, etc)?
  19. inxile

    inxile New Member

    2 7200 RPM 1 TB hard disks in a RAID-1 array, with a 3Ware 9650SE-2LP RAID controller
    8 GB of RAM
    Intel Core 2 Quad Q6600

    Litespeed Web Server Enterprise v4.0.20
    CentOS and PHP were recently updated to current
  20. mistwang

    mistwang LiteSpeed Staff

    It is likely a slow SQL query or something else blocked all PHP processed, increasing "max connections" wont work.
    You need to find out what happened from command line, check "top" output for CPU utilization, I/O wait, load, memory usage, etc.
    Check mysql slow query cache, check "mysqladmin processlist" .
    Also, check "dmesg" output.

Share This Page