503 is back on 2.1.5 (and 2.1.4 as well)

[SyNeo]

Hi.

When running a long PHP script, the server returns the 503 error:

503 Service Unavailable
The server is temporarily busy, try again later!

Other scripts work just fine. The used environment is lshttpd 2.1.5, with PHP 5.0.4, LSAPI 1.6 and eaccelerator 0.9.7-cvs.

Any idea how to pinpoint this one?

 

[xing]

With one server on lsws 2.1.4 (forgot to upgrade to 2.1.5 on this one) + PHP 5.0.5 + eACC CVS, I also got unexplained 503 errors and backend php pool restarts in the log though no php errors to correspond to the problem.

I have disabled eACC for that server and let's see if it chokes up this time.

 

[xing]

Confirmed with 2.1.4 and php 5.0.5 minus eacc. Now will try 2.1.5 on this particular server.

 

[xing]

More than 8 hours of heavy testing on same server. Upgraded to lsws 2.1.5, same php 5.0.5...with eac disabled.

No 503 errors.

What ever is causing the sporadic restarts due to 503 with 2.1.4 is now gone, thus far, with litespeed 2.1.5.

If you system can stand a little higher load, I would suggest testing your setup with eac disabled.

 

[SyNeo]

Thanks, I will try it out. Does it means that it's a problem in the eac? It happened in the 2.1.3 version as well, btw, so it probably means it was always there?

 

[xing]

Looks like it. The price we pay for third party but free opcode cache. I tried APC 3.0.8 or whatever the latest is and that also caused some 503 errors due to dead phps so pick your poision. eAcc behave better than APC but I don't think there is much activity in the cvs tree there.

 

[mistwang]

Xing, thank you for your help.

The problem is not directly caused by LSWS, but a PHP problem, either the opcode cache or something else, LSWS is designed to recover from this kind of problem automatically when possible. So, you should receive email notifications that lsws restarts due to too many 503 errors.

To diagnose this problem, if you have a test machine, you can let lsws only start one PHP instance, then use "strace -p <pid>" to trace the PHP process when the problematic script being executed. And our error.log and stderr.log can help sometimes.

 

[SyNeo]

Hi.

I'm using LSAPI on this machine - so it means that I should strace the lsphp process?

BTW, took a peak in the log files finally. Here what they say:


error_log:
2005-11-07 11:27:12.226 [NOTICE] [config:server:epsrhpFcgi]'ProcessLimit' probably is too low, adjust the limit to: 110.
2005-11-07 11:27:12.226 [NOTICE] [config:server:epsrhpLsapi]'ProcessLimit' probably is too low, adjust the limit to: 200.


stderror:
FATAL: erealloc(): Unable to allocate 32882 bytes
[16557] EACCELERATOR: PHP unclean shutdown on opline 12 of serialize() at /var/www/lib/framework/Web/UI/TPage.php:374

FATAL: emalloc(): Unable to allocate 14136 bytes
[16656] EACCELERATOR: PHP unclean shutdown on opline -1356305782 of createSelectSql() at /usr/local/lib/php/propel/util/BasePeer.php:398

FATAL: emalloc(): Unable to allocate 6480 bytes
[16765] EACCELERATOR: PHP unclean shutdown on opline -1808403775 of appendPsTo() at /usr/local/lib/php/propel/util/Criteria.php:598

FATAL: erealloc(): Unable to allocate 65745 bytes
[20835] EACCELERATOR: PHP unclean shutdown on opline 12 of serialize() at /var/www/lib/framework/Web/UI/TPage.php:374

So it most probably means that the problem is with the EA.

 

[mistwang]

Looks like you need to increase the memory limit for lsphp, the default memory limit is in "server"->"security"->"CGI Resource Limit", each external application can set their own limits in the extapp configuration page.

The notice about ProcessLimit should not matter, if you don't want to see those notice again, increase the Process Limit for lsphp as well.