PDA

View Full Version : [solved] Max retries has been reached, 503!

track1
02-02-2011, 04:53 AM
Hello,
im testing litespeed for a large vb board, and is failing from time to time with 503 error pages. Error log shows this:

2011-02-02 13:22:50.072 NOTICE [xx.xx.xx.xx:64670-0#APVH_xxx.com] Max retries has been reached, 503!
2011-02-02 13:22:50.072 NOTICE [xx.xx.xx.xx:64670-0#APVH_xxx.com] oops! 503 Service Unavailable
2011-02-02 13:22:50.072 NOTICE [xx.xx.xx.xx:64670-0#APVH_xxx.com] Content len: 0, Request line: 'GET /a/b.php?t=2049984 HTTP/1.1'
2011-02-02 13:31:47.770 NOTICE [yy.yy.yy.yy:52497-0#APVH_xxx.com] Max retries has been reached, 503!
2011-02-02 13:31:47.770 NOTICE [yy.yy.yy.yy:52497-0#APVH_xxx.com] oops! 503 Service Unavailable
2011-02-02 13:31:47.770 NOTICE [yy.yy.yy.yy:52497-0#APVH_xxx.com] Content len: 0, Request line: 'GET /a/b.php?t=2049980 HTTP/1.1'
2011-02-02 13:31:56.074 NOTICE [yy.yy.yy.yy:52499-0#APVH_xxx.com] Max retries has been reached, 503!
2011-02-02 13:31:56.074 NOTICE [yy.yy.yy.yy:52499-0#APVH_xxx.com] oops! 503 Service Unavailable
2011-02-02 13:31:56.074 NOTICE [yy.yy.yy.yy:52499-0#APVH_xxx.com] Content len: 0, Request line: 'GET /a/b.php?t=2049998 HTTP/1.1'
2011-02-02 13:38:18.976 NOTICE [zz.zz.zz.zz:1810-0#APVH_xxx.com] Max retries has been reached, 503!
2011-02-02 13:38:18.976 NOTICE [zz.zz.zz.zz:1810-0#APVH_xxx.com] oops! 503 Service Unavailable
2011-02-02 13:38:18.976 NOTICE [zz.zz.zz.zz:1810-0#APVH_xxx.com] Content len: 0, Request line: 'GET /a/b.php?t=2049917&page=2 HTTP/1.1'
2011-02-02 13:42:19.330 NOTICE [zz.zz.zz.zz:28897-0#APVH_xxx.com] Max retries has been reached, 503!
2011-02-02 13:42:19.330 NOTICE [zz.zz.zz.zz:28897-0#APVH_xxx.com] oops! 503 Service Unavailable
2011-02-02 13:42:19.330 NOTICE [zz.zz.zz.zz:28897-0#APVH_xxx.com] Content len: 0, Request line: 'GET /a/b.php?t=2049861&page=5 HTTP/1.1'

Server: WHM/Cpanel - RH 5.2 32 bits, 4GB Ram, Dual Harpertown 5430.
Litespeed: Enterprise 4.0.19
PHP: 5.2.16, compiled for litespeed using Build Matching PHP Binary in WHM | no special modules
Others: XCache 1.3.1, tested active or inactive, same 503 errors.
Server load: (average) - Load 1.50, 1.68, 1.78

Site run under Apache 2.0 without any error.
What can i check to detect errors source, or modify to avoid these errors?
NiteWave
02-02-2011, 05:03 AM
many reason can cause 503. need check many settings. first, please check in admin console:
Server->External App->lsphp5
Memory Soft Limit (bytes)
Memory Hard Limit (bytes)

php in suExec or non-suExec mode?
track1
02-02-2011, 05:44 AM
PHP non-suExec

Max Connections 50

PHP_LSAPI_MAX_REQUESTS=500
PHP_LSAPI_CHILDREN=50

Memory Soft Limit (bytes) 900M
Memory Hard Limit (bytes) 1000M

Process Soft Limit 1000
Process Hard Limit 1000
webizen
02-02-2011, 10:19 AM
is there anything in /usr/local/apache/logs/stderr.log when 503 happens?
track1
02-02-2011, 10:33 AM
no, stderr.log file is empty
webizen
02-02-2011, 11:20 AM
For the 503 errors (correlate with timestamp and URL requested), please find out the browser info (User Agent) in access_log. This is to see whether the errors were triggered by certain browser, with cookie or not, registered user only, etc. If cookie is clear, will the error go away? Is database (mysql) on the same machine or separate one?


2011-02-02 13:42:19.330 NOTICE [zz.zz.zz.zz:28897-0#APVH_xxx.com] Max retries has been reached, 503!
2011-02-02 13:42:19.330 NOTICE [zz.zz.zz.zz:28897-0#APVH_xxx.com] oops! 503 Service Unavailable
2011-02-02 13:42:19.330 NOTICE [zz.zz.zz.zz:28897-0#APVH_xxx.com] Content len: 0, Request line: 'GET /a/b.php?t=2049861&page=5 HTTP/1.1'
track1
02-03-2011, 10:01 AM
Is running www only, mysql is running in other box.

Currently testing Litespeed at 3 web servers with same specs and running same site, with different options (default config, tunned, cache on/off...)

All get same 503 errors at times and lot of "Content len: 0, Request line: 'GET ...".

After some tests this is what i have:

- Error shows at multiple user-agents (firefox, chrome, etc).
- Logged-in, users with cookie.
- Large cookies (~4k).
track1
02-03-2011, 10:03 AM
DEBUG LOG: 503 error for user ip 123.123.123.123 attached
track1
02-03-2011, 10:23 AM
Tested LiteSpeed/4.1RC4 in one server, and same 503 problems.
track1
02-03-2011, 10:34 AM
Doing a search found other high traffic vb board related thread:

[ High Traffic vbulletin forum with 503 Errors ]
http://www.litespeedtech.com/support/forum/showthread.php?t=3801
track1
02-03-2011, 11:33 AM
after more tests, and reproducing own 503 errors with a large cookie,
found that reducing cookie size had no error.
webizen
02-03-2011, 11:49 AM
Good to hear.

The error below (from your debug log) indicates php crashed. lsapi tried to recover connection and failed (Max retries has been reached). Usually, should let PHP dump core file and find out what the root cause is.


2011-02-03 18:31:07.114 [INFO] [123.123.123.123:52532-0#APVH_site.com:lsapi] connection to [/tmp/lshttpd/lsphp5.sock.993] on req
uest #94, confirmed, 1, associated process: 27581, running: 1, error: Connection reset by peer!
2011-02-03 18:31:07.114 [DEBUG] [123.123.123.123:52532-0#APVH_site.com:lsapi] [ExtConn] close()
2011-02-03 18:31:07.114 [DEBUG] [123.123.123.123:52532-0#APVH_site.com:lsapi] HttpExtConnector::tryRecover()...
2011-02-03 18:31:07.114 [NOTICE] [123.123.123.123:52532-0#APVH_site.com:lsapi] Max retries has been reached, 503!
track1
02-03-2011, 11:54 AM
problems seems to be cookie size > 4kb
but why Apache/PHP don't crash?
mistwang
02-03-2011, 12:53 PM
If Apache/PHP crash, user only got a blank page.different symptom.
Likely a PHP internal issue as LSAPI does not deal with cookies at all.
track1
02-03-2011, 01:14 PM
yes, but apache simply does not crash (do not show a blank page with cookies over 4kb), while litespeed shows 503 error.
mistwang
02-03-2011, 01:52 PM
It is a mystery now why PHP crash with large cookie size. If you could create a simple test script to trigger it, we would like to investigate it in our lab.
xnetua
02-10-2011, 01:22 AM
Why the thread is marked as solved?
I have the same problem:

2011-02-10 11:58:39.584 [NOTICE] [213.111.228.103:51337-2#p16932:xxxxx.ru] Redirect: #1, URL: /index.php
2011-02-10 11:59:32.539 [NOTICE] [213.111.228.103:51398-0#p16932:xxxxx.ru] Max retries has been reached, 503!
2011-02-10 11:59:32.539 [NOTICE] [213.111.228.103:51398-0#p16932:xxxxx.ru] oops! 503 Service Unavailable
2011-02-10 11:59:32.539 [NOTICE] [213.111.228.103:51398-0#p16932:xxxxx.ru] Content len: 0, Request line: 'GET / HTTP/1.1'

2011-02-10 11:26:48.571 [NOTICE] [213.111.228.103:50456-0#p16932:xxxxx.ru] Max retries has been reached, 503!
2011-02-10 11:26:48.571 [NOTICE] [213.111.228.103:50456-0#p16932:xxxxx.ru] oops! 503 Service Unavailable
2011-02-10 11:26:48.571 [NOTICE] [213.111.228.103:50456-0#p16932:xxxxx.ru] Content len: 263, Request line: 'POST /wp-admin/admin-ajax.php HTTP/1.1'

2011-02-10 11:59:32.539 [INFO] [uds://usr/local/lsws/tmp/p16932_lsphp.sock] Connection refused, restart!
2011-02-10 11:59:32.539 [INFO] [uds://usr/local/lsws/tmp/p16932_lsphp.sock] Connection refused, restart!
2011-02-10 11:59:32.539 [INFO] [uds://usr/local/lsws/tmp/p16932_lsphp.sock] Connection refused, restart!
2011-02-10 11:59:32.539 [NOTICE] [213.111.228.103:51398-0#p16932:xxxxx.ru] Max retries has been reached, 503!
2011-02-10 11:59:32.539 [NOTICE] [213.111.228.103:51398-0#p16932:xxxxx.ru] oops! 503 Service Unavailable
webizen
02-10-2011, 12:20 PM
Is 503 error consistent or sporadic? If consistent, it is more of a php problem. Turn off opcode cache if any to troubleshoot. Also check stderr.log and see if there is anything when the 503 error happened.
NiteWave
02-11-2011, 12:01 AM
Why the thread is marked as solved?
need add a note here for future search:
they changed the php code to reduce the cookie size, and fixed 503 error.

503 is a general error, there are many reasons to cause 503.for example, if a php script fail to get enough memory, a 503 error may occur.

please create a new thread for better trace your issue. Will mark this thread back to "solved"