PHP processes constantly climbing

[matt]

Not sure if this is a bug or a config error on my part, but I've been noticing that the number of lsphp processes climbs steadily and never goes down.

I ran into this when we were getting complaints about mail() not working, so I logged in and noticed that PHP was unable to talk to sendmail because there were thousands of lsphp processes running and monopolizing resources somehow.

Restarting lsws doesn't change anything. The only way I fixed it was running killall -9 lsphp and then restarting lsws. If I run a `ps aux | grep -c lsphp` then it'll show a reasonable value between 5-20, but then starts climbing again endlessly as time goes on.

Any suggestions?

 

[xing]

Two steps to isolate the problem.

1) Check php.err or stderr.log to see what type of errors php is throwing.

2) Php like to hang on network sockets. Check to make sure if you are using php to scoket to another server, that server is not overloaded or dead.

3) Check lsws error.log to see if it's auto restarting the php pool because of 503 errors. If phps die, lsws will respawn new ones which could lead to hundreds of dead phps if those php don't die correctly.

Php don't usually hang/zombify withouth throwing a bunch of errors in the logs. Check those first.

 

[mistwang]

Greetings from caribbean on world second largest ship.
I am on vacation now, can't answer all questions.
Please make sure you are using the latest LSAPI, earlier version may have this kind of problem.
Don't set "Max connections" too high, "Initial request timeout" and connection timeout should be set properly.

For the hanging PHP process, you can try "strace -p <PHP_PID>" to find out on which system call it blocked.

If those do not fix the problem, please write a simple script to monitor the number of PHP processes in a corn job, and kill them when need.

I will investigate the problem when get back on 14th.

 

[matt]

I just upgraded to 2.1.5-ent, so we'll see if the problem continues, I'll keep you updated.

 

[mistwang]

If you are using our prebuilt PHP binary, please overwrite lsws/fcgi-bin/lsphp with lsws/admin/fcgi-bin/admin_php to make sure the latest LSAPI code has been used.

 

[matt]

Yep, it's back up to 170 processes and rising.

Can I just copy admin_php to lsphp to overwrite it? To upgrade I just used the built-in automatic upgrade thingy.

 

[mistwang]

Can I just copy admin_php to lsphp to overwrite it? To upgrade I just used the built-in automatic upgrade thingy.

You cannot overwrite binary of a running process, will get "text file busy" error, have to do "rm lsws/fcgi-bin/lsphp" then "cp lsws/admin/fcgi-bin/admin_php lsws/fcgi-bin/lsphp", then restart lsws.
The automatic upgrade does not upgrade lsws/fcgi-bin/lsphp, as it is not a good idea to overwrite a user customized php binary and break things.

 

[matt]

I did that and the problem is continuing. We set up a cron script to kill the processes and restart lsws when it gets above 150 and it seems to be kicking off every 14-24 hours.

 

[mistwang]

Hi Matt,

Can you please tell me your configuration for lsphp, we will try to reproduce it.

And if possible, please do "strace -p <pid_of_a hanging_php>", post your result, it will help identify the problem as well.

Thanks,
George

 

[mistwang]

This issue should be fixed in the 2.1.6 release.