Random 503 Errors

[LiteSpeeder]

Hello,

We've been using LSWS for a few years and we are familiar with 503 errors, but random 503 errors have become too annoying recently. It occurs when the server load averages are too high. I believe it's the normal behavior to display a 503 error message when the server couldn't handle the process. The weird thing is the server load averages go higher even there is no huge traffic. This happens 3-4 times in a day and the pages can not be displayed with a 503 error message for 5 to 30 minutes.

Can you please tell me which details i should specify so you can help me?

[webizen]

check error log (e.g. /usr/local/lsws/logs/error.log or /usr/local/apache/logs/error_log) for any details when 503 happens.

[LiteSpeeder]

Retrieved 122 of total 122 log entries.
Time	Level	Message
[78.169.31.227:21101-0#APVH_******.***] File not found [/home/******/public_html/images/M_images/printButton.png]
2012-03-15 19:55:47.955	INFO	[78.169.31.227:21112-0#APVH_******.***] File not found [/home/******/public_html/images/M_images/emailButton.png]
2012-03-15 19:55:48.002	NOTICE	All pending requests are finished, stop process group lead by pid: 23383
2012-03-15 19:55:48.033	INFO	Remove pid: 23383, exitcode: 129
2012-03-15 19:55:48.882	INFO	[212.252.223.142:49558-1#APVH_******.***] File not found [/home/******/public_html/resimyukle/resimler/soylesiler.JPG]
2012-03-15 19:55:49.222	INFO	[212.252.223.142:49568-0#APVH_******.***] File not found [/home/******/public_html/resimyukle/resimler/esref_saat.JPG]
2012-03-15 19:55:50.661	INFO	[88.240.210.186:16049-0#APVH_******.***] File not found [/home/******/public_html/"http:/v2.******.***/mobile.png/"]
2012-03-15 19:55:53.078	INFO	[78.184.156.27:51417-0#APVH_******.***] File not found [/home/******/public_html/skins-1.5/common/images/magnify-clip.png]
2012-03-15 19:55:54.643	INFO	[85.106.225.38:13151-0#APVH_******.***] File not found [/home/******/public_html/resimyukle/resimler/soylesiler.JPG]
2012-03-15 19:55:54.810	INFO	[85.106.225.38:13156-0#APVH_******.***] File not found [/home/******/public_html/resimyukle/resimler/esref_saat.JPG]
2012-03-15 19:55:55.111	INFO	[212.252.223.142:49622-0#APVH_******.***] File not found [/home/******/public_html/resimyukle/resimler/soylesiler.JPG]
2012-03-15 19:55:55.125	INFO	[212.252.223.142:49623-0#APVH_******.***] File not found [/home/******/public_html/resimyukle/resimler/esref_saat.JPG]
2012-03-15 19:55:55.977	INFO	[212.253.122.13:49270-0#APVH_******.***] File not found [/home/******/public_html/images/smilies/biggrin.gif]
2012-03-15 19:56:01.298	INFO	[78.178.57.9:27374-0#APVH_******.***] File not found [/home/******/public_html/[object Object]:/www.wxdownload.info/worker/init.js]
2012-03-15 19:56:05.308	INFO	[92.44.113.31:49708-0#APVH_******.***] File not found [/home/******/public_html/"http:/v2.******.***/mobile.png/"]
2012-03-15 19:56:08.114	INFO	[78.178.57.9:27382-0#APVH_******.***] File not found [/home/******/public_html/[object Object]:/www.wxdownload.info/worker/init.js]
2012-03-15 19:56:08.636	INFO	[78.178.57.9:27395-0#APVH_******.***] File not found [/home/******/public_html/[object Object]:/www.wxdownload.info/worker/init.js]
2012-03-15 19:56:14.987	INFO	[188.3.129.54:2354-0#APVH_******.***] File not found [/home/******/public_html/"http:/v2.******.***/mobile.png/"]
2012-03-15 19:56:15.127	INFO	[176.240.137.142:1132-0#APVH_******.***] File not found [/home/******/public_html/skins-1.5/common/images/magnify-clip.png]
2012-03-15 19:56:18.723	INFO	[78.165.117.65:17086-0#APVH_******.***] File not found [/home/******/public_html/images/M_images/pdf_button.png]
2012-03-15 19:56:18.732	INFO	[78.165.117.65:17087-0#APVH_******.***] File not found [/home/******/public_html/images/M_images/printButton.png]
2012-03-15 19:56:18.740	INFO	[78.165.117.65:17088-0#APVH_******.***] File not found [/home/******/public_html/images/M_images/emailButton.png]
2012-03-15 19:56:29.371	INFO	[88.254.24.74:3440-1#APVH_******.***] File not found [/home/******/public_html/images/smilies/biggrin.gif]
2012-03-15 19:56:31.017	INFO	[88.254.24.74:3463-0#APVH_******.***] File not found [/home/******/public_html/images/smilies/biggrin.gif]
2012-03-15 19:56:31.316	INFO	[88.244.118.222:2238-0#APVH_******.***] File not found [/home/******/public_html/"http:/v2.******.***/mobile.png/"]
2012-03-15 19:56:38.053	INFO	[85.96.238.193:1123-0#APVH_******.***] connection to [/tmp/lshttpd/lsphp5.sock.699] on request #10, confirmed, 0, associated process: 24034, running: 1, error: Connection reset by peer!
2012-03-15 19:56:38.305	INFO	[95.10.206.60:13888-0#APVH_******.***] connection to [/tmp/lshttpd/lsphp5.sock.699] on request #18, confirmed, 0, associated process: 24034, running: 1, error: Connection reset by peer!
2012-03-15 19:56:47.076	INFO	[85.103.131.252:10564-0#APVH_******.***] File not found [/home/******/public_html/"http:/v2.******.***/mobile.png/"]
2012-03-15 19:56:56.658	INFO	[78.177.110.179:51115-0#APVH_******.***] File not found [/home/******/public_html/"http:/v2.******.***/ambience/buttons/printer.gif/"]
2012-03-15 19:57:00.879	INFO	[78.177.110.179:51098-0#APVH_******.***] File not found [/home/******/public_html/"http:/v2.******.***/ambience/buttons/sendtofriend.gif/"]
2012-03-15 19:57:17.032	INFO	[78.169.191.91:2266-0] Status 400: Bad HTTP protocol string: bekleniy
2012-03-15 19:57:17.538	INFO	[78.169.191.91:2267-0] Status 400: Bad HTTP protocol string: bekleniy
2012-03-15 19:57:18.900	INFO	[66.249.71.180:37132-0#APVH_******.***] Connection idle time: 61 while in state: 5 watching for event: 25,close!
2012-03-15 19:57:18.900	NOTICE	[66.249.71.180:37132-0#APVH_******.***] Content len: 0, Request line: 'GET /forum/m/320901/piramit-ile-ilgili-cozumlu-sorular/ HTTP/1.1'
2012-03-15 19:57:18.900	NOTICE	[66.249.71.180:37132-0#APVH_******.***] Redirect: #1, URL: /forum/vbseo.php
2012-03-15 19:57:18.900	INFO	[66.249.71.180:37132-0#APVH_******.***] HttpExtConnector state: 8, request body sent: 0, response body size: 0, response body sent:0, left in buffer: 0, attempts: 0.
2012-03-15 19:57:18.900	INFO	[66.249.71.180:37132-0#APVH_******.***] Lsapi connection state: 2, watching event: 25, respState: 1, m_iTotalPending:0, m_iPacketLeft:0, m_iCurRespHeader:9, req sent for 61 seconds,Total processing time: 61.
2012-03-15 19:57:19.301	INFO	[78.161.17.241:53443-0#APVH_******.***] Connection idle time: 61 while in state: 5 watching for event: 25,close!
2012-03-15 19:57:19.301	NOTICE	[78.161.17.241:53443-0#APVH_******.***] Content len: 0, Request line: 'GET /forum/soru-cevap/211882-siyasi-alanda-yapilan-inkilaplar-nelerdir.html HTTP/1.1'
2012-03-15 19:57:19.301	NOTICE	[78.161.17.241:53443-0#APVH_******.***] Redirect: #1, URL: /forum/vbseo.php
2012-03-15 19:57:19.301	INFO	[78.161.17.241:53443-0#APVH_******.***] HttpExtConnector state: 8, request body sent: 0, response body size: 0, response body sent:0, left in buffer: 0, attempts: 0.
2012-03-15 19:57:19.301	INFO	[78.161.17.241:53443-0#APVH_******.***] Lsapi connection state: 2, watching event: 25, respState: 1, m_iTotalPending:0, m_iPacketLeft:0, m_iCurRespHeader:8, req sent for 61 seconds,Total processing time: 61.
2012-03-15 19:57:19.900	INFO	[66.249.71.180:37132-0#APVH_******.***:lsapi] Connection idle time: 62 while in state: 5 watching for event: 25,close!
2012-03-15 19:57:19.900	NOTICE	[66.249.71.180:37132-0#APVH_******.***:lsapi] Content len: 0, Request line: 'GET /forum/m/320901/piramit-ile-ilgili-cozumlu-sorular/ HTTP/1.1'
2012-03-15 19:57:19.900	NOTICE	[66.249.71.180:37132-0#APVH_******.***:lsapi] Redirect: #2, URL: /500.shtml
2012-03-15 19:57:19.900	INFO	[66.249.71.180:37132-0#APVH_******.***:lsapi] HttpExtConnector state: 0, request body sent: 0, response body size: 0, response body sent:0, left in buffer: 0, attempts: 0.
2012-03-15 19:57:19.900	INFO	[66.249.71.180:37132-0#APVH_******.***:lsapi] External processor is not available.
2012-03-15 19:57:19.900	ERROR	[66.249.71.180:37132-0#APVH_******.***:lsapi] detect loop redirection: /500.shtml
2012-03-15 19:57:20.307	INFO	[78.161.17.241:53443-0#APVH_******.***:lsapi] Connection idle time: 62 while in state: 5 watching for event: 25,close!
2012-03-15 19:57:20.307	NOTICE	[78.161.17.241:53443-0#APVH_******.***:lsapi] Content len: 0, Request line: 'GET /forum/soru-cevap/211882-siyasi-alanda-yapilan-inkilaplar-nelerdir.html HTTP/1.1'
2012-03-15 19:57:20.307	NOTICE	[78.161.17.241:53443-0#APVH_******.***:lsapi] Redirect: #2, URL: /500.shtml
2012-03-15 19:57:20.307	INFO	[78.161.17.241:53443-0#APVH_******.***:lsapi] HttpExtConnector state: 0, request body sent: 0, response body size: 0, response body sent:0, left in buffer: 0, attempts: 0.
2012-03-15 19:57:20.307	INFO	[78.161.17.241:53443-0#APVH_******.***:lsapi] External processor is not available.
2012-03-15 19:57:20.307	ERROR	[78.161.17.241:53443-0#APVH_******.***:lsapi] detect loop redirection: /500.shtml

[webizen]

Originally Posted by LiteSpeeder

...
2012-03-15 19:57:20.307	INFO	[78.161.17.241:53443-0#APVH_******.***:lsapi] Connection idle time: 62 while in state: 5 watching for event: 25,close!
2012-03-15 19:57:20.307	NOTICE	[78.161.17.241:53443-0#APVH_******.***:lsapi] Content len: 0, Request line: 'GET /forum/soru-cevap/211882-siyasi-alanda-yapilan-inkilaplar-nelerdir.html HTTP/1.1'
2012-03-15 19:57:20.307	NOTICE	[78.161.17.241:53443-0#APVH_******.***:lsapi] Redirect: #2, URL: /500.shtml
2012-03-15 19:57:20.307	INFO	[78.161.17.241:53443-0#APVH_******.***:lsapi] HttpExtConnector state: 0, request body sent: 0, response body size: 0, response body sent:0, left in buffer: 0, attempts: 0.
2012-03-15 19:57:20.307	INFO	[78.161.17.241:53443-0#APVH_******.***:lsapi] External processor is not available.
2012-03-15 19:57:20.307	ERROR	[78.161.17.241:53443-0#APVH_******.***:lsapi] detect loop redirection: /500.shtml

Increase Connection Timeout (secs) (Admin CP -> Server -> Tuning) to 120

[NiteWave]

please paste the error log relating to 503.

here's an example:

2012-03-07 02:44:28.157 [NOTICE] [85.113.57.238:42742-0#domain.com] Redirect: #1, URL: /index.php
2012-03-07 03:41:47.396 [INFO] [119.63.196.62:2408-0#domain.com] connection to [/tmp/lshttpd/lsphp5.sock.226] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2012-03-07 03:41:47.410 [INFO] [119.63.196.62:2408-0#domain.com] connection to [/tmp/lshttpd/lsphp5.sock.226] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2012-03-07 03:41:47.422 [INFO] [119.63.196.62:2408-0#domain.com] connection to [/tmp/lshttpd/lsphp5.sock.226] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2012-03-07 03:41:47.422 [NOTICE] [119.63.196.62:2408-0#domain.com] Max retries has been reached, 503!
2012-03-07 03:41:47.422 [NOTICE] [119.63.196.62:2408-0#domain.com] oops! 503 Service Unavailable^M
2012-03-07 03:41:47.422 [NOTICE] [119.63.196.62:2408-0#domain.com] Content len: 0, Request line: 'GET /background HTTP/1.1'
2012-03-07 03:41:47.422 [NOTICE] [119.63.196.62:2408-0#domain.com] Redirect: #1, URL: /index.php
2012-03-07 04:59:58.562 [INFO] [157.55.17.88:63532-0#domain.com] connection to [/tmp/lshttpd/lsphp5.sock.279] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!

above is a complete error log for 503 error happening at 03:41:47, which extracted from error.log.

please check if your recent 503 error has same pattern as above: before "Max retries has been reached, 503!", there is a line like
"connection to [/tmp/lshttpd/lsphp5.sock.226] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!"

above log is extracted from a server which just upgraded to 4.1.11 recently, and all are same pattern(on request #0, confirmed, 0, associated process: -1).