[self-solved]LSWS 4.0.17 and "hanging processes"

B

Bono

Well-Known Member
#1
#1
Hi, my LSWS is restarted everyday by cPanel when it is needed to process logs, as you know this is how it needs to be done according to cPanel. Lately I noticed increase in memory usage and I found out I got lot of old processes which are not killed properly by graceful restart and if I continue like this in few weeks I will ran out of memory.

This is probably bug, so I posted this topic here and please let me know what can I do do ensure this doesn't happen anymore.

Code: 
nobody     349  0.0  0.3 303396 16236 ?        SN   May06   0:30 lsphp5
nobody     468  0.0  2.2 365028 92068 ?        SN   May06   0:57 lsphp5
nobody     469  0.0  1.1 317052 46032 ?        SN   May06   1:16 lsphp5
nobody     679  0.0  3.6 421292 151752 ?       SN   Sep03   1:56 lsphp5:/home/site/public_html/index.php
nobody    2638  0.0  1.9 355644 81860 ?        SN   Aug22   1:33 lsphp5:/home/site/public_html/index.php
nobody    5116  0.0  0.8 306624 37112 ?        SN   Nov13   0:46 lsphp5:/home/site/public_html/index.php
nobody    5565  0.0  1.1 316800 48184 ?        SN   May06   2:38 lsphp5
nobody    7045  0.0  0.7 316392 33200 ?        SN   May06   0:38 lsphp5
nobody    8520  0.0  2.6 381748 110660 ?       SN   Jul21   0:59 lsphp5:/home/site/public_html/index.php
nobody    9492  0.0  0.7 302512 29504 ?        SN   Oct26   0:34 lsphp5:/home/site/public_html/index.php
nobody   10231  0.0  2.4 375232 102856 ?       SN   Nov12   1:40 lsphp5:/home/site/public_html/index.php
nobody   11779  0.0  3.2 395612 135048 ?       SN   Nov12   2:31 lsphp5:/home/site/public_html/index.php
nobody   12863  0.0  2.1 357752 89148 ?        SN   Jul21   1:29 lsphp5:/home/site/public_html/index.php
nobody   12980  0.0  3.2 396644 134120 ?       SN   Nov12   2:01 lsphp5:/home/site/public_html/index.php
nobody   14217  0.0  2.6 378288 110584 ?       SN   Nov01   2:26 lsphp5:/home/site/public_html/index.php
nobody   14615  0.0  2.1 358444 87292 ?        SN   Jul17   1:02 lsphp5:/home/site/public_html/index.php
nobody   15875  0.0  2.5 380048 106804 ?       SN   Nov14   0:48 lsphp5:/home/site/public_html/index.php
nobody   15876  0.0  2.9 399508 124104 ?       SN   Nov14   1:27 lsphp5:/home/site/public_html/index.php
nobody   18181  0.0  0.1 296248  5048 ?        SNs  12:27   0:00 lsphp5
nobody   20869  0.0  2.6 385236 110604 ?       SN   Nov05   0:53 lsphp5:/home/site/public_html/index.php
nobody   21666  0.0  2.4 383324 100100 ?       SN   Nov04   0:35 lsphp5:/home/site/public_html/index.php
nobody   23373  6.1  1.0 302436 42688 ?        SN   13:05   0:38 lsphp5
nobody   23466  8.0  0.8 302296 34716 ?        SN   13:06   0:45 lsphp5
nobody   24060 12.0  0.7 302532 32660 ?        SN   13:10   0:39 lsphp5
nobody   24061 11.8  0.9 306148 39756 ?        SN   13:10   0:39 lsphp5
nobody   24069 10.2  0.8 303576 34304 ?        RN   13:10   0:33 lsphp5:/home/site/public_html/index.php
nobody   24096  6.7  0.7 302516 30856 ?        SN   13:11   0:20 lsphp5
nobody   24688 19.2  0.7 302260 29460 ?        SN   13:15   0:12 lsphp5
nobody   24751 14.9  0.5 303532 24824 ?        RN   13:15   0:04 lsphp5:/home/site/public_html/index.php
nobody   28558  0.0  2.8 384656 116840 ?       SN   Nov13   1:08 lsphp5:/home/site/public_html/index.php
nobody   28568  0.0  2.6 383732 110628 ?       SN   Nov02   0:56 lsphp5:/home/site/public_html/index.php
nobody   31082  0.0  2.6 374748 111984 ?       SN   Oct30   2:41 lsphp5:/home/site/public_html/index.php
 
Last edited by a moderator:
B

Bono

Well-Known Member
#3
#3
Same thing:

Code: 
nobody     349  0.0  0.3 303396 16236 ?        SN   May06   0:30 lsphp5
nobody     468  0.0  2.2 365028 92068 ?        SN   May06   0:57 lsphp5
nobody     469  0.0  1.1 317052 46032 ?        SN   May06   1:16 lsphp5
nobody     679  0.0  3.6 421292 151752 ?       SN   Sep03   1:56 lsphp5:/home/site/public_html/index.php
nobody    2638  0.0  1.9 355644 81860 ?        SN   Aug22   1:33 lsphp5:/home/site/public_html/index.php
nobody    5116  0.0  0.8 306624 37112 ?        SN   Nov13   0:46 lsphp5:/home/site/public_html/index.php
nobody    5565  0.0  1.1 316800 48184 ?        SN   May06   2:38 lsphp5
nobody    7045  0.0  0.7 316392 33200 ?        SN   May06   0:38 lsphp5
nobody    7245  0.0  0.1 296244  5052 ?        SNs  23:52   0:00 lsphp5
nobody    7259 11.1  0.8 303836 35732 ?        SN   23:52   0:14 lsphp5
nobody    7260  7.9  0.8 309472 36184 ?        SN   23:52   0:09 lsphp5
nobody    7265  0.4  0.4 302920 17012 ?        SN   23:52   0:00 lsphp5
nobody    7266  0.5  0.4 306240 18880 ?        SN   23:52   0:00 lsphp5
nobody    7267  5.5  0.6 303296 26992 ?        SN   23:52   0:06 lsphp5
nobody    7268  0.1  0.2 301144 10940 ?        SN   23:52   0:00 lsphp5
nobody    7269  2.3  0.4 302544 19456 ?        SN   23:52   0:02 lsphp5
nobody    7270 10.0  0.7 302656 29060 ?        SN   23:52   0:12 lsphp5
nobody    7271 11.3  0.7 302528 32276 ?        SN   23:52   0:13 lsphp5
nobody    8520  0.0  2.6 381748 110660 ?       SN   Jul21   0:59 lsphp5:/home/site/public_html/index.php
nobody    9492  0.0  0.7 302512 29504 ?        SN   Oct26   0:34 lsphp5:/home/site/public_html/index.php
nobody   10231  0.0  2.4 375232 102856 ?       SN   Nov12   1:40 lsphp5:/home/site/public_html/index.php
nobody   11779  0.0  3.2 395612 135048 ?       SN   Nov12   2:31 lsphp5:/home/site/public_html/index.php
nobody   12863  0.0  2.1 357752 89148 ?        SN   Jul21   1:29 lsphp5:/home/site/public_html/index.php
nobody   12980  0.0  3.2 396644 134120 ?       SN   Nov12   2:01 lsphp5:/home/site/public_html/index.php
nobody   14217  0.0  2.6 378288 110584 ?       SN   Nov01   2:26 lsphp5:/home/site/public_html/index.php
nobody   14615  0.0  2.1 358444 87292 ?        SN   Jul17   1:02 lsphp5:/home/site/public_html/index.php
nobody   15875  0.0  2.5 380048 106804 ?       SN   Nov14   0:48 lsphp5:/home/site/public_html/index.php
nobody   15876  0.0  2.9 399508 124104 ?       SN   Nov14   1:27 lsphp5:/home/site/public_html/index.php
nobody   20869  0.0  2.6 385236 110604 ?       SN   Nov05   0:53 lsphp5:/home/site/public_html/index.php
nobody   21666  0.0  2.4 383324 100100 ?       SN   Nov04   0:35 lsphp5:/home/site/public_html/index.php
nobody   28558  0.0  2.8 384656 116840 ?       SN   Nov13   1:08 lsphp5:/home/site/public_html/index.php
nobody   28568  0.0  2.6 383732 110628 ?       SN   Nov02   0:56 lsphp5:/home/site/public_html/index.php
nobody   31082  0.0  2.6 374748 111984 ?       SN   Oct30   2:41 lsphp5:/home/site/public_html/index.php
 
N

NiteWave

Administrator
#4
#4
what's result of pstree -p

if lsphp5 process has no parent, then it's a runaway process, just kill it.

you can strace the runaway process, what it is doing. a bug php script may cause dead lock and waiting for mutex etc forever.
 
B

Bono

Well-Known Member
#5
#5
My problem is that if I killed them, there will be soon new ones.
Here it is:

─litespeed(8114)─┬─httpd(8115)
│ └─litespeed(8116)───lsphp5(8135)─┬─lsphp5(14178)
│ ├─lsphp5(14432)
│ ├─lsphp5(14783)
│ ├─lsphp5(14974)
│ ├─lsphp5(15164)
│ ├─lsphp5(15172)
│ ├─lsphp5(15173)
│ ├─lsphp5(15691)
│ ├─lsphp5(15692)
│ ├─lsphp5(15985)
│ ├─lsphp5(15988)
│ └─lsphp5(16008)
├─lsphp5(349)
├─lsphp5(468)
├─lsphp5(469)
├─lsphp5(679)
├─lsphp5(2638)
├─lsphp5(5116)
├─lsphp5(5565)
├─lsphp5(7045)
├─lsphp5(8520)
├─lsphp5(9492)
├─lsphp5(10231)
├─lsphp5(11779)
├─lsphp5(12863)
├─lsphp5(12980)
├─lsphp5(14217)
├─lsphp5(14615)
├─lsphp5(15875)
├─lsphp5(15876)
├─lsphp5(20869)
├─lsphp5(21666)
├─lsphp5(28558)
├─lsphp5(28568)
├─lsphp5(31082)
├─migration/0(2)
├─migration/1(5)
├─migration/2(8)
├─migration/3(11)
├─mingetty(5231)
├─mingetty(5232)
├─mingetty(5233)
├─mingetty(5234)
├─mingetty(5236)
├─mingetty(27319)
 
Last edited:
N

NiteWave

Administrator
#6
#6
My problem is that if I killed them, there will be soon new ones.
Click to expand...
this is the problem ... most likely it's related to bug in php script. we can login your server to investigate, in case it really a bug of lsphp.
 
B

Bono

Well-Known Member
#7
#7
NiteWave said:
this is the problem ... most likely it's related to bug in php script. we can login your server to investigate, in case it really a bug of lsphp.
Click to expand...
This is trace result of this "hanging" processes:

strace -p 9492
Process 9492 attached - interrupt to quit
futex(0x4a280c, FUTEX_WAIT, 2, NULL <unfinished ...>
Process 9492 detached
root[~]# strace -p 10231
Process 10231 attached - interrupt to quit
futex(0xaf8140, FUTEX_WAIT, 2, NULL <unfinished ...>
Process 10231 detached

And all other hanging processes to that script are the same, I could probably try to kill them and when my server will not be so busy to restart mysql also, it hasn't been restarted for 4 months.
 
Last edited:
B

Bono

Well-Known Member
#9
#9
Code: 
Reply With QuoteCOMMAND  PID   USER   FD   TYPE     DEVICE     SIZE       NODE NAME
lsphp5  9492 nobody  cwd    DIR        8,7     4096    5565132 /home/site/public_html
lsphp5  9492 nobody  rtd    DIR        8,5     4096          2 /
lsphp5  9492 nobody  txt    REG        8,5 16326967    1638529 /opt/lsws/fcgi-bin/lsphp-5.2.13
lsphp5  9492 nobody  mem    REG        8,3              134404 /usr/lib/libpng12.so.0.10.0 (path inode=135327)
lsphp5  9492 nobody  mem    REG        8,5             1900553 /lib/ld-2.5.so (path inode=1902064)
lsphp5  9492 nobody  mem    REG        8,5             1900572 /lib/libc-2.5.so (path inode=1902211)
lsphp5  9492 nobody  mem    REG        8,5             1900599 /lib/libdl-2.5.so (path inode=1902221)
lsphp5  9492 nobody  mem    REG        8,5             1900602 /lib/libm-2.5.so (path inode=1902222)
lsphp5  9492 nobody  mem    REG        8,5             1900617 /lib/libpthread-2.5.so (path inode=1900607)
lsphp5  9492 nobody  mem    REG        8,3              135311 /usr/lib/libz.so.1.2.3 (path inode=134057)
lsphp5  9492 nobody  mem    REG        8,5             1902190 /lib/libselinux.so.1 (path inode=1902230)
lsphp5  9492 nobody  mem    REG        8,5             1902186 /lib/libsepol.so.1 (path inode=1902229)
lsphp5  9492 nobody  mem    REG        8,5             1902114 /lib/librt-2.5.so (path inode=1902089)
lsphp5  9492 nobody  mem    REG        8,5             1900699 /lib/libnsl-2.5.so (path inode=1900619)
lsphp5  9492 nobody  mem    REG        8,5             1900755 /lib/libcrypt-2.5.so (path inode=1902226)
lsphp5  9492 nobody  mem    REG        8,3              133701 /usr/lib/libXpm.so.4.11.0 (path inode=134010)
lsphp5  9492 nobody  mem    REG        8,3              135331 /usr/lib/libidn.so.11.5.19 (path inode=135340)
lsphp5  9492 nobody  mem    REG        8,3              133646 /usr/lib/libXau.so.6.0.0 (path inode=133645)
lsphp5  9492 nobody  mem    REG        8,3              133883 /usr/lib/libX11.so.6.2.0 (path inode=133654)
lsphp5  9492 nobody  mem    REG        8,3              133851 /usr/lib/libXdmcp.so.6.0.0 (path inode=133647)
lsphp5  9492 nobody  mem    REG        8,5             1900744 /lib/libresolv-2.5.so (path inode=1902225)
lsphp5  9492 nobody  mem    REG        8,5             1900766 /lib/libkeyutils-1.2.so (path inode=1900765)
lsphp5  9492 nobody  mem    REG        8,5             1902212 /lib/libaudit.so.0.0.0 (path inode=1902231)
lsphp5  9492 nobody  DEL    REG        8,5             1902216 /lib/libcrypto.so.0.9.8e.#prelink#.SeWKSA
lsphp5  9492 nobody  mem    REG        8,5             1900731 /lib/libcom_err.so.2.1 (path inode=1902171)
lsphp5  9492 nobody  DEL    REG        8,3              135324 /usr/lib/libgssapi_krb5.so.2.2.#prelink#.E2K32K
lsphp5  9492 nobody  mem    REG        8,5             1902213 /lib/libpam.so.0.81.5 (path inode=1900552)
lsphp5  9492 nobody  mem    REG        8,3              135316 /usr/lib/libk5crypto.so.3.1 (path inode=134089)
lsphp5  9492 nobody  DEL    REG        8,3              134067 /usr/lib/libkrb5support.so.0.1.#prelink#.kIfM5l
lsphp5  9492 nobody  mem    REG        8,3              134030 /usr/lib/libkrb5.so.3.3 (path inode=134395)
lsphp5  9492 nobody  mem    REG        8,5             1902217 /lib/libssl.so.0.9.8e (path inode=1902236)
lsphp5  9492 nobody  DEL    REG        8,3              134142 /usr/lib/libmysqlclient.so.15.0.0.#prelink#.EVX344
lsphp5  9492 nobody  mem    REG        8,3              134047 /usr/lib/libjpeg.so.62.0.0 (path inode=134401)
lsphp5  9492 nobody  mem    REG        8,5             1900791 /lib/libnss_dns-2.5.so (path inode=1900785)
lsphp5  9492 nobody  DEL    REG        8,3              135313 /usr/lib/locale/locale-archive
lsphp5  9492 nobody  DEL    REG        0,8          1361978971 /dev/zero
lsphp5  9492 nobody  mem    REG        8,5             1900795 /lib/libnss_files-2.5.so (path inode=1902188)
lsphp5  9492 nobody  mem    REG        8,3   333798     655548 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/xcache.so
lsphp5  9492 nobody  mem    REG        8,5  3271198    1638455 /opt/xml2/lib/libxml2.so.2.7.6
lsphp5  9492 nobody  mem    REG        8,5   283914    1671738 /opt/curl/lib/libcurl.so.4.2.0
lsphp5  9492 nobody  mem    REG        8,5   221347    1704146 /opt/pcre/lib/libpcre.so.0.0.1
lsphp5  9492 nobody  DEL    REG        0,8          1361978978 /dev/zero
lsphp5  9492 nobody    0u  unix 0xdfb2be40          1361978964 /tmp/lshttpd/lsphp5.sock.405
lsphp5  9492 nobody    1u  unix 0xdc840e40          1361978941 socket
lsphp5  9492 nobody    2u  unix 0xdc840e40          1361978941 socket
lsphp5  9492 nobody    3u   REG        8,8        0        209 /tmp/.xcache.99.1.846930886.lock (deleted)
lsphp5  9492 nobody    4u  unix 0xe9b93740           952111171 socket
lsphp5  9492 nobody    5u   REG        8,8        0        232 /tmp/.xcache.99.2.1681692777.lock (deleted)
lsphp5  9492 nobody    6u   REG        8,8        0        237 /tmp/.xcache.99.3.1714636915.lock (deleted)
lsphp5  9492 nobody    7u   REG        8,8        0        246 /tmp/.xcache.99.4.1957747793.lock (deleted)
lsphp5  9492 nobody    8u   REG        8,8        0        247 /tmp/.xcache.99.5.424238335.lock (deleted)
lsphp5  9492 nobody    9u  unix 0xdd235ac0          1362346562 /tmp/lshttpd/lsphp5.sock.405
lsphp5  9492 nobody   10u  unix 0xe698cac0          1362352979 socket
 
B

Bono

Well-Known Member
#11
#11
NiteWave said:
I'd give a quick guess: xcache caused the problem. can you try disable xcache and see if the issue still exist.
Click to expand...
I can't disable xcache load would kill the box, I got millions of hits per day. I checked when 4.0.17 was released and problems started when I upgraded to 4.0.17, so I will try with 4.0.16.
 
M

mistwang

LiteSpeed Staff
#12
#12
I doubt that LSWS version will affect this, it is a problem between PHP and xcache. You can use a cron job to kill those PHP process with ppid being "1".
 
B

Bono

Well-Known Member
#13
#13
mistwang said:
I doubt that LSWS version will affect this, it is a problem between PHP and xcache. You can use a cron job to kill those PHP process with ppid being "1".
Click to expand...
Well, i'll test for you I have tried to kill old processes and LSWS has spawned new ones right away, mysql is also restarted just in case. Now i'm runing 4.0.16 and I'll need few days to check if new processes will be created and I will update this topic.
 
B

Bono

Well-Known Member
#14
#14
4.0.16 didn't made any difference I have killed all processes recompiled PHP to 5.2.14 and switched to 4.0.17 and it is fixed now. I see again process is spawned but it dies normally after few minutes. So I guess everything is back to normal.
 
You must log in or register to reply here.
Top