[self-solved]LSWS 4.0.17 and "hanging processes"

Discussion in 'Bug Reports' started by Bono, Nov 15, 2010.

  1. Bono

    Bono New Member

    Hi, my LSWS is restarted everyday by cPanel when it is needed to process logs, as you know this is how it needs to be done according to cPanel. Lately I noticed increase in memory usage and I found out I got lot of old processes which are not killed properly by graceful restart and if I continue like this in few weeks I will ran out of memory.

    This is probably bug, so I posted this topic here and please let me know what can I do do ensure this doesn't happen anymore.

    Code:
    nobody     349  0.0  0.3 303396 16236 ?        SN   May06   0:30 lsphp5
    nobody     468  0.0  2.2 365028 92068 ?        SN   May06   0:57 lsphp5
    nobody     469  0.0  1.1 317052 46032 ?        SN   May06   1:16 lsphp5
    nobody     679  0.0  3.6 421292 151752 ?       SN   Sep03   1:56 lsphp5:/home/site/public_html/index.php
    nobody    2638  0.0  1.9 355644 81860 ?        SN   Aug22   1:33 lsphp5:/home/site/public_html/index.php
    nobody    5116  0.0  0.8 306624 37112 ?        SN   Nov13   0:46 lsphp5:/home/site/public_html/index.php
    nobody    5565  0.0  1.1 316800 48184 ?        SN   May06   2:38 lsphp5
    nobody    7045  0.0  0.7 316392 33200 ?        SN   May06   0:38 lsphp5
    nobody    8520  0.0  2.6 381748 110660 ?       SN   Jul21   0:59 lsphp5:/home/site/public_html/index.php
    nobody    9492  0.0  0.7 302512 29504 ?        SN   Oct26   0:34 lsphp5:/home/site/public_html/index.php
    nobody   10231  0.0  2.4 375232 102856 ?       SN   Nov12   1:40 lsphp5:/home/site/public_html/index.php
    nobody   11779  0.0  3.2 395612 135048 ?       SN   Nov12   2:31 lsphp5:/home/site/public_html/index.php
    nobody   12863  0.0  2.1 357752 89148 ?        SN   Jul21   1:29 lsphp5:/home/site/public_html/index.php
    nobody   12980  0.0  3.2 396644 134120 ?       SN   Nov12   2:01 lsphp5:/home/site/public_html/index.php
    nobody   14217  0.0  2.6 378288 110584 ?       SN   Nov01   2:26 lsphp5:/home/site/public_html/index.php
    nobody   14615  0.0  2.1 358444 87292 ?        SN   Jul17   1:02 lsphp5:/home/site/public_html/index.php
    nobody   15875  0.0  2.5 380048 106804 ?       SN   Nov14   0:48 lsphp5:/home/site/public_html/index.php
    nobody   15876  0.0  2.9 399508 124104 ?       SN   Nov14   1:27 lsphp5:/home/site/public_html/index.php
    nobody   18181  0.0  0.1 296248  5048 ?        SNs  12:27   0:00 lsphp5
    nobody   20869  0.0  2.6 385236 110604 ?       SN   Nov05   0:53 lsphp5:/home/site/public_html/index.php
    nobody   21666  0.0  2.4 383324 100100 ?       SN   Nov04   0:35 lsphp5:/home/site/public_html/index.php
    nobody   23373  6.1  1.0 302436 42688 ?        SN   13:05   0:38 lsphp5
    nobody   23466  8.0  0.8 302296 34716 ?        SN   13:06   0:45 lsphp5
    nobody   24060 12.0  0.7 302532 32660 ?        SN   13:10   0:39 lsphp5
    nobody   24061 11.8  0.9 306148 39756 ?        SN   13:10   0:39 lsphp5
    nobody   24069 10.2  0.8 303576 34304 ?        RN   13:10   0:33 lsphp5:/home/site/public_html/index.php
    nobody   24096  6.7  0.7 302516 30856 ?        SN   13:11   0:20 lsphp5
    nobody   24688 19.2  0.7 302260 29460 ?        SN   13:15   0:12 lsphp5
    nobody   24751 14.9  0.5 303532 24824 ?        RN   13:15   0:04 lsphp5:/home/site/public_html/index.php
    nobody   28558  0.0  2.8 384656 116840 ?       SN   Nov13   1:08 lsphp5:/home/site/public_html/index.php
    nobody   28568  0.0  2.6 383732 110628 ?       SN   Nov02   0:56 lsphp5:/home/site/public_html/index.php
    nobody   31082  0.0  2.6 374748 111984 ?       SN   Oct30   2:41 lsphp5:/home/site/public_html/index.php
    Last edited by a moderator: Nov 22, 2010
  2. webizen

    webizen New Member

    please try force reinstall 4.0.17 (LSWS Admin console -> Actions -> Version Manager).
  3. Bono

    Bono New Member

    Same thing:

    Code:
    nobody     349  0.0  0.3 303396 16236 ?        SN   May06   0:30 lsphp5
    nobody     468  0.0  2.2 365028 92068 ?        SN   May06   0:57 lsphp5
    nobody     469  0.0  1.1 317052 46032 ?        SN   May06   1:16 lsphp5
    nobody     679  0.0  3.6 421292 151752 ?       SN   Sep03   1:56 lsphp5:/home/site/public_html/index.php
    nobody    2638  0.0  1.9 355644 81860 ?        SN   Aug22   1:33 lsphp5:/home/site/public_html/index.php
    nobody    5116  0.0  0.8 306624 37112 ?        SN   Nov13   0:46 lsphp5:/home/site/public_html/index.php
    nobody    5565  0.0  1.1 316800 48184 ?        SN   May06   2:38 lsphp5
    nobody    7045  0.0  0.7 316392 33200 ?        SN   May06   0:38 lsphp5
    nobody    7245  0.0  0.1 296244  5052 ?        SNs  23:52   0:00 lsphp5
    nobody    7259 11.1  0.8 303836 35732 ?        SN   23:52   0:14 lsphp5
    nobody    7260  7.9  0.8 309472 36184 ?        SN   23:52   0:09 lsphp5
    nobody    7265  0.4  0.4 302920 17012 ?        SN   23:52   0:00 lsphp5
    nobody    7266  0.5  0.4 306240 18880 ?        SN   23:52   0:00 lsphp5
    nobody    7267  5.5  0.6 303296 26992 ?        SN   23:52   0:06 lsphp5
    nobody    7268  0.1  0.2 301144 10940 ?        SN   23:52   0:00 lsphp5
    nobody    7269  2.3  0.4 302544 19456 ?        SN   23:52   0:02 lsphp5
    nobody    7270 10.0  0.7 302656 29060 ?        SN   23:52   0:12 lsphp5
    nobody    7271 11.3  0.7 302528 32276 ?        SN   23:52   0:13 lsphp5
    nobody    8520  0.0  2.6 381748 110660 ?       SN   Jul21   0:59 lsphp5:/home/site/public_html/index.php
    nobody    9492  0.0  0.7 302512 29504 ?        SN   Oct26   0:34 lsphp5:/home/site/public_html/index.php
    nobody   10231  0.0  2.4 375232 102856 ?       SN   Nov12   1:40 lsphp5:/home/site/public_html/index.php
    nobody   11779  0.0  3.2 395612 135048 ?       SN   Nov12   2:31 lsphp5:/home/site/public_html/index.php
    nobody   12863  0.0  2.1 357752 89148 ?        SN   Jul21   1:29 lsphp5:/home/site/public_html/index.php
    nobody   12980  0.0  3.2 396644 134120 ?       SN   Nov12   2:01 lsphp5:/home/site/public_html/index.php
    nobody   14217  0.0  2.6 378288 110584 ?       SN   Nov01   2:26 lsphp5:/home/site/public_html/index.php
    nobody   14615  0.0  2.1 358444 87292 ?        SN   Jul17   1:02 lsphp5:/home/site/public_html/index.php
    nobody   15875  0.0  2.5 380048 106804 ?       SN   Nov14   0:48 lsphp5:/home/site/public_html/index.php
    nobody   15876  0.0  2.9 399508 124104 ?       SN   Nov14   1:27 lsphp5:/home/site/public_html/index.php
    nobody   20869  0.0  2.6 385236 110604 ?       SN   Nov05   0:53 lsphp5:/home/site/public_html/index.php
    nobody   21666  0.0  2.4 383324 100100 ?       SN   Nov04   0:35 lsphp5:/home/site/public_html/index.php
    nobody   28558  0.0  2.8 384656 116840 ?       SN   Nov13   1:08 lsphp5:/home/site/public_html/index.php
    nobody   28568  0.0  2.6 383732 110628 ?       SN   Nov02   0:56 lsphp5:/home/site/public_html/index.php
    nobody   31082  0.0  2.6 374748 111984 ?       SN   Oct30   2:41 lsphp5:/home/site/public_html/index.php
  4. NiteWave

    NiteWave Administrator

    what's result of pstree -p

    if lsphp5 process has no parent, then it's a runaway process, just kill it.

    you can strace the runaway process, what it is doing. a bug php script may cause dead lock and waiting for mutex etc forever.
  5. Bono

    Bono New Member

    My problem is that if I killed them, there will be soon new ones.
    Here it is:

    ─litespeed(8114)─┬─httpd(8115)
    │ └─litespeed(8116)───lsphp5(8135)─┬─lsphp5(14178)
    │ ├─lsphp5(14432)
    │ ├─lsphp5(14783)
    │ ├─lsphp5(14974)
    │ ├─lsphp5(15164)
    │ ├─lsphp5(15172)
    │ ├─lsphp5(15173)
    │ ├─lsphp5(15691)
    │ ├─lsphp5(15692)
    │ ├─lsphp5(15985)
    │ ├─lsphp5(15988)
    │ └─lsphp5(16008)
    ├─lsphp5(349)
    ├─lsphp5(468)
    ├─lsphp5(469)
    ├─lsphp5(679)
    ├─lsphp5(2638)
    ├─lsphp5(5116)
    ├─lsphp5(5565)
    ├─lsphp5(7045)
    ├─lsphp5(8520)
    ├─lsphp5(9492)
    ├─lsphp5(10231)
    ├─lsphp5(11779)
    ├─lsphp5(12863)
    ├─lsphp5(12980)
    ├─lsphp5(14217)
    ├─lsphp5(14615)
    ├─lsphp5(15875)
    ├─lsphp5(15876)
    ├─lsphp5(20869)
    ├─lsphp5(21666)
    ├─lsphp5(28558)
    ├─lsphp5(28568)
    ├─lsphp5(31082)
    ├─migration/0(2)
    ├─migration/1(5)
    ├─migration/2(8)
    ├─migration/3(11)
    ├─mingetty(5231)
    ├─mingetty(5232)
    ├─mingetty(5233)
    ├─mingetty(5234)
    ├─mingetty(5236)
    ├─mingetty(27319)
    Last edited: Nov 16, 2010
  6. NiteWave

    NiteWave Administrator

    this is the problem ... most likely it's related to bug in php script. we can login your server to investigate, in case it really a bug of lsphp.
  7. Bono

    Bono New Member

    This is trace result of this "hanging" processes:

    strace -p 9492
    Process 9492 attached - interrupt to quit
    futex(0x4a280c, FUTEX_WAIT, 2, NULL <unfinished ...>
    Process 9492 detached
    root[~]# strace -p 10231
    Process 10231 attached - interrupt to quit
    futex(0xaf8140, FUTEX_WAIT, 2, NULL <unfinished ...>
    Process 10231 detached

    And all other hanging processes to that script are the same, I could probably try to kill them and when my server will not be so busy to restart mysql also, it hasn't been restarted for 4 months.
    Last edited: Nov 16, 2010
  8. NiteWave

    NiteWave Administrator

    please check:
    lsof -p 9492
  9. Bono

    Bono New Member

    Code:
    Reply With QuoteCOMMAND  PID   USER   FD   TYPE     DEVICE     SIZE       NODE NAME
    lsphp5  9492 nobody  cwd    DIR        8,7     4096    5565132 /home/site/public_html
    lsphp5  9492 nobody  rtd    DIR        8,5     4096          2 /
    lsphp5  9492 nobody  txt    REG        8,5 16326967    1638529 /opt/lsws/fcgi-bin/lsphp-5.2.13
    lsphp5  9492 nobody  mem    REG        8,3              134404 /usr/lib/libpng12.so.0.10.0 (path inode=135327)
    lsphp5  9492 nobody  mem    REG        8,5             1900553 /lib/ld-2.5.so (path inode=1902064)
    lsphp5  9492 nobody  mem    REG        8,5             1900572 /lib/libc-2.5.so (path inode=1902211)
    lsphp5  9492 nobody  mem    REG        8,5             1900599 /lib/libdl-2.5.so (path inode=1902221)
    lsphp5  9492 nobody  mem    REG        8,5             1900602 /lib/libm-2.5.so (path inode=1902222)
    lsphp5  9492 nobody  mem    REG        8,5             1900617 /lib/libpthread-2.5.so (path inode=1900607)
    lsphp5  9492 nobody  mem    REG        8,3              135311 /usr/lib/libz.so.1.2.3 (path inode=134057)
    lsphp5  9492 nobody  mem    REG        8,5             1902190 /lib/libselinux.so.1 (path inode=1902230)
    lsphp5  9492 nobody  mem    REG        8,5             1902186 /lib/libsepol.so.1 (path inode=1902229)
    lsphp5  9492 nobody  mem    REG        8,5             1902114 /lib/librt-2.5.so (path inode=1902089)
    lsphp5  9492 nobody  mem    REG        8,5             1900699 /lib/libnsl-2.5.so (path inode=1900619)
    lsphp5  9492 nobody  mem    REG        8,5             1900755 /lib/libcrypt-2.5.so (path inode=1902226)
    lsphp5  9492 nobody  mem    REG        8,3              133701 /usr/lib/libXpm.so.4.11.0 (path inode=134010)
    lsphp5  9492 nobody  mem    REG        8,3              135331 /usr/lib/libidn.so.11.5.19 (path inode=135340)
    lsphp5  9492 nobody  mem    REG        8,3              133646 /usr/lib/libXau.so.6.0.0 (path inode=133645)
    lsphp5  9492 nobody  mem    REG        8,3              133883 /usr/lib/libX11.so.6.2.0 (path inode=133654)
    lsphp5  9492 nobody  mem    REG        8,3              133851 /usr/lib/libXdmcp.so.6.0.0 (path inode=133647)
    lsphp5  9492 nobody  mem    REG        8,5             1900744 /lib/libresolv-2.5.so (path inode=1902225)
    lsphp5  9492 nobody  mem    REG        8,5             1900766 /lib/libkeyutils-1.2.so (path inode=1900765)
    lsphp5  9492 nobody  mem    REG        8,5             1902212 /lib/libaudit.so.0.0.0 (path inode=1902231)
    lsphp5  9492 nobody  DEL    REG        8,5             1902216 /lib/libcrypto.so.0.9.8e.#prelink#.SeWKSA
    lsphp5  9492 nobody  mem    REG        8,5             1900731 /lib/libcom_err.so.2.1 (path inode=1902171)
    lsphp5  9492 nobody  DEL    REG        8,3              135324 /usr/lib/libgssapi_krb5.so.2.2.#prelink#.E2K32K
    lsphp5  9492 nobody  mem    REG        8,5             1902213 /lib/libpam.so.0.81.5 (path inode=1900552)
    lsphp5  9492 nobody  mem    REG        8,3              135316 /usr/lib/libk5crypto.so.3.1 (path inode=134089)
    lsphp5  9492 nobody  DEL    REG        8,3              134067 /usr/lib/libkrb5support.so.0.1.#prelink#.kIfM5l
    lsphp5  9492 nobody  mem    REG        8,3              134030 /usr/lib/libkrb5.so.3.3 (path inode=134395)
    lsphp5  9492 nobody  mem    REG        8,5             1902217 /lib/libssl.so.0.9.8e (path inode=1902236)
    lsphp5  9492 nobody  DEL    REG        8,3              134142 /usr/lib/libmysqlclient.so.15.0.0.#prelink#.EVX344
    lsphp5  9492 nobody  mem    REG        8,3              134047 /usr/lib/libjpeg.so.62.0.0 (path inode=134401)
    lsphp5  9492 nobody  mem    REG        8,5             1900791 /lib/libnss_dns-2.5.so (path inode=1900785)
    lsphp5  9492 nobody  DEL    REG        8,3              135313 /usr/lib/locale/locale-archive
    lsphp5  9492 nobody  DEL    REG        0,8          1361978971 /dev/zero
    lsphp5  9492 nobody  mem    REG        8,5             1900795 /lib/libnss_files-2.5.so (path inode=1902188)
    lsphp5  9492 nobody  mem    REG        8,3   333798     655548 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/xcache.so
    lsphp5  9492 nobody  mem    REG        8,5  3271198    1638455 /opt/xml2/lib/libxml2.so.2.7.6
    lsphp5  9492 nobody  mem    REG        8,5   283914    1671738 /opt/curl/lib/libcurl.so.4.2.0
    lsphp5  9492 nobody  mem    REG        8,5   221347    1704146 /opt/pcre/lib/libpcre.so.0.0.1
    lsphp5  9492 nobody  DEL    REG        0,8          1361978978 /dev/zero
    lsphp5  9492 nobody    0u  unix 0xdfb2be40          1361978964 /tmp/lshttpd/lsphp5.sock.405
    lsphp5  9492 nobody    1u  unix 0xdc840e40          1361978941 socket
    lsphp5  9492 nobody    2u  unix 0xdc840e40          1361978941 socket
    lsphp5  9492 nobody    3u   REG        8,8        0        209 /tmp/.xcache.99.1.846930886.lock (deleted)
    lsphp5  9492 nobody    4u  unix 0xe9b93740           952111171 socket
    lsphp5  9492 nobody    5u   REG        8,8        0        232 /tmp/.xcache.99.2.1681692777.lock (deleted)
    lsphp5  9492 nobody    6u   REG        8,8        0        237 /tmp/.xcache.99.3.1714636915.lock (deleted)
    lsphp5  9492 nobody    7u   REG        8,8        0        246 /tmp/.xcache.99.4.1957747793.lock (deleted)
    lsphp5  9492 nobody    8u   REG        8,8        0        247 /tmp/.xcache.99.5.424238335.lock (deleted)
    lsphp5  9492 nobody    9u  unix 0xdd235ac0          1362346562 /tmp/lshttpd/lsphp5.sock.405
    lsphp5  9492 nobody   10u  unix 0xe698cac0          1362352979 socket
  10. NiteWave

    NiteWave Administrator

    I'd give a quick guess: xcache caused the problem. can you try disable xcache and see if the issue still exist.
  11. Bono

    Bono New Member

    I can't disable xcache load would kill the box, I got millions of hits per day. I checked when 4.0.17 was released and problems started when I upgraded to 4.0.17, so I will try with 4.0.16.
  12. mistwang

    mistwang LiteSpeed Staff

    I doubt that LSWS version will affect this, it is a problem between PHP and xcache. You can use a cron job to kill those PHP process with ppid being "1".
  13. Bono

    Bono New Member

    Well, i'll test for you I have tried to kill old processes and LSWS has spawned new ones right away, mysql is also restarted just in case. Now i'm runing 4.0.16 and I'll need few days to check if new processes will be created and I will update this topic.
  14. Bono

    Bono New Member

    4.0.16 didn't made any difference I have killed all processes recompiled PHP to 5.2.14 and switched to 4.0.17 and it is fixed now. I see again process is spawned but it dies normally after few minutes. So I guess everything is back to normal.

Share This Page