Huge lsphp5 problem

Discussion in 'PHP' started by alex95bg, Aug 23, 2009.

  1. alex95bg

    alex95bg New Member

    I seem to be unlucky.. litespeed worked great
    I upgraded to the final version and recompiled php 5.2.9 with eAccelerator and Suhosin
    Everything seems to be normal.. until something very strange happens
    A huge amount of lsphp5 processes are spawned and the server load spikes to 80+

    Code:
    top - 22:48:08 up  4:36,  1 user,  load average: 71.27, 40.58, 19.89
    Tasks: 163 total,  81 running,  82 sleeping,   0 stopped,   0 zombie
    Cpu(s): 97.4%us,  1.6%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.3%hi,  0.7%si,  0.0%st
    Mem:    183412k total,   180520k used,     2892k free,     4948k buffers
    Swap:   345356k total,    65448k used,   279908k free,    75828k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                      
    16924 nobody    20   0 66200 3164 1560 R  2.0  1.7   0:04.70 lsphp5                       16564 nobody     1 -19  7484 4016 1328 S  1.3  2.2   0:30.88 lshttpd                      16617 root      20   0  2340 1172  852 R  1.3  0.6   0:12.90 top                          
    16854 nobody    20   0 66412 5220 3452 R  1.3  2.8   0:06.76 lsphp5                       16888 nobody    20   0 66600 7580 5628 R  1.3  4.1   0:09.90 lsphp5                       16891 nobody    20   0 66460 7160 5204 R  1.3  3.9   0:08.80 lsphp5                       
    16894 nobody    20   0 66368 4948 3220 R  1.3  2.7   0:05.32 lsphp5                       16897 nobody    20   0 66332 4676 2964 R  1.3  2.5   0:05.22 lsphp5                       16899 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.94 lsphp5                       
    16900 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.92 lsphp5                       16901 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.90 lsphp5                       16902 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.88 lsphp5                       
    16903 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.86 lsphp5                       16904 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.86 lsphp5                       16905 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.82 lsphp5                       
    16906 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.80 lsphp5                       16907 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.78 lsphp5                       16908 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.80 lsphp5                       
    16909 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.78 lsphp5                       16911 nobody    20   0 66200 3176 1572 R  1.3  1.7   0:04.76 lsphp5                       16912 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.74 lsphp5  
    I rebooted the PC, with no result
    So this is literally stopping the server and it spits out 503 errors after a huge delay
    I have to do this:
    Code:
    dragonheart:/usr/local/lsws/bin# ./lswsctrl stop
    [OK] lshttpd: stopped.
    dragonheart:/usr/local/lsws/bin# killall lsphp5
    dragonheart:/usr/local/lsws/bin# ./lswsctrl start
    [OK] lshttpd: pid=17156.
    dragonheart:/usr/local/lsws/bin# 
    A restart doesn't make any effect
    I havent seen this preblem never before, the code is the same, lsws has been running on this machine for morew than seven months, and almost two with this kernel (2.6.24-etchnhalf; debian 4.0r3 etch)
  2. mistwang

    mistwang LiteSpeed Staff

    You should to do one upgrade at a time, at this point it is hard tell it is a PHP issue or LSWS issue.

    What is your "Max Connections" for lsphp5? maybe you should tune it down.
  3. alex95bg

    alex95bg New Member

    I upgraded only litespeed, i used the same setup (PHP 5.2.9, suhosin, eAccelerator) before, just now eAccelerator and suhosin are different versions
    Code:
    PHP_LSAPI_MAX_REQUESTS=500
    PHP_LSAPI_CHILDREN=35
    And Max Connections is 35 :)

    The strange thing is, i tried very hard to replicate this, but i didn't succeed
    I think this happens only when the machine is very low on ram
    Yesterday there was a traffic spike, but on previous traffic spikes nothing similar happened

    Also, i can't turn on the eAccelerator optimizer, i try from control.php but it stays on no
    Code:
    Optimizer enabled  	no
    It's also enabled in php.ini
    --
    Other than this, and the solved php problem LiteSpeed is working great. I would like to thank you, lsws saved my website. It survived on apache for about a week after it's startup and apache started throwing timeouts. So when i switch to the apache backup, the machine almost dies :)

    NEW edit:
    It happened again, i killed all the processes and they spawned again
    It's ok now, let's see for how long
    Last edited: Aug 24, 2009
  4. alex95bg

    alex95bg New Member

    OK, it's happening very often :(
  5. mistwang

    mistwang LiteSpeed Staff

    You should downgrade eaccelerator and suhosin to the older version.
    eAcc 0.9.6-rc1 is in beta, and optimizer is removed if you read the release log.
  6. alex95bg

    alex95bg New Member

    How do i downgrade from the Compile PHP section? :)

    edit:
    I now looked at the server log and this doesn't seem ordinary:

    new edit:
    I setted Max Idle Time to 5
    I hope that this will kill the phantom processes
    Last edited: Aug 27, 2009
  7. alex95bg

    alex95bg New Member

    Nope, nothing works
    Theese lockups are happening even more frequently :( :confused:
  8. robfrew

    robfrew New Member

    Can you do a screenshot of your phpinfo and post it?
  9. alex95bg

    alex95bg New Member

  10. alex95bg

    alex95bg New Member

    Should i give up and downgrade litespeed?
  11. mistwang

    mistwang LiteSpeed Staff

    Please downgrade to 4.0.6 for now, you need to run the installer in 4.0.6 package.
  12. robfrew

    robfrew New Member

    Have you tried disabling eAccellerator to see if it may be the problem?
  13. alex95bg

    alex95bg New Member

    No, i haven't but i really think that it is the problem. eAccelerator gives a huge performance boost in my site and i use the eaccelerator_get and eaccelerator_put functions so i can't try it out on the live environment :(
    Though i will try with litespeed 4.0.6 because (i hope) it doesn't use the rc version of eAccelerator
    Also it will be good if we can select eAccelerator and Suhosin versions from the Compile PHP tab, having rc or beta in a production environment isn't the best thing to do :)
  14. alex95bg

    alex95bg New Member

    same thing when i downgraded to .6 but with the old php
    Now when i try to recompile php i get
    100 389 100 389 0 0 347 0 0:00:01 0:00:01 --:--:-- 347
    100 389 100 389 0 0 346 0 0:00:01 0:00:01 --:--:-- 0

    gzip: stdin: not in gzip format
    tar: Child returned status 1
    tar: Error exit delayed from previous errors
    On Extracting LSAPI

    i really hope that someone will help me
  15. mistwang

    mistwang LiteSpeed Staff

    Please the forum for work around.
  16. alex95bg

    alex95bg New Member

    Is there a work around? I didn't find one
    Maybe i will do a os change, migrating from debian to centos soon, let's hope that the issue will go away
    And now i fixed it with (a very stupid way) 1 minute cron job which counts the lsphp5 processes and if they are > 6 kills them all
  17. alex95bg

    alex95bg New Member

    I switched over to Ubuntu 8.10 Server because debian broke very bad in the middle of a traffic spike
    I watched the server for a week, everything is working great. :) i only had a few problems with the rc eaccelerator so i compiled myself the stable version and everything worked great
    i suggest you don't download the RC version on php build, it has bugs and it's a very very bad idea to pud unstable software on production servers
  18. art182

    art182 New Member

    i think i have this same problem ...

    version 4.0.9, 4.0.11 and on 4.0.6 also...

    php 5.2.11 and APC 3.1.3


    what's going on ? ( everything was good for about 1 year and now something f### )
    maybe this is problem with LSAPI or PHP ?
  19. mistwang

    mistwang LiteSpeed Staff

    It is likely a PHP/APC internal problem.
    Try turning off APC, or try eAccelerator, or upgrade APC.
    Also, if it does not solve the problem, do
    "strace -tt -T -p <pid_of_lsphp5>" and "strace -tt -T -p <pid_of_lshttpd>"

    and send us the output.
  20. art182

    art182 New Member

    oh i know whats going on now...
    there was problem with one of virtual hosts ( wordpress + spam bot + php error )

    everything ok ( 4.0.12 )
    sorry for trouble

Share This Page