408 Errors

MikeDVB · May 13, 2010

Hello,

I seem to be getting 408 errors on a few LiteSpeed powered VPSs and I'm not entirely sure as to why. The node loads/ram/ I/O etc... are all fine as are the statistics on the VPSs that are having the issues.

This is on 4.0.13 - I did try 4.0.14 previously but was getting 503 errors so I rolled back.

Code:

HTTP/1.0 408 Request Time-out
Date: Thu, 13 May 2010 10:04:43 GMT
Server: LiteSpeed
Connection: close
Cache-Control: private, no-cache, max-age=0
Pragma: no-cache
Content-Type: text/html
Content-Length: 530

Code:

<html>
<head><title> 408 Request Time-out
</title></head>
<body><h1> 408 Request Time-out
</h1>
This request takes too long to process,  it is timed out by the server. If it should not be timed out, please contact administrator of this web site  to increase 'Connection Timeout'.
<hr />
Powered By <a href='http://www.litespeedtech.com'>LiteSpeed Web Server</a><br />
<font face="Verdana, Arial, Helvetica" size=-1>LiteSpeed Technologies is not responsible for administration and contents of this web site!</font></body></html>

This happened between 6:03 AM and 6:18 AM.

sar -r

Code:

05:40:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
05:50:04 AM    243936    280352     53.47         0         0         0         0      0.00         0
06:00:01 AM    242300    281988     53.78         0         0         0         0      0.00         0
06:10:02 AM    204064    320224     61.08         0         0         0         0      0.00         0
06:20:04 AM    245820    278468     53.11         0         0         0         0      0.00         0
06:30:01 AM    268680    255608     48.75         0         0         0         0      0.00         0

sar -q

Code:

05:40:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
05:50:04 AM         2       159      1.12      0.78      0.92
06:00:01 AM         3       162      0.83      0.96      0.98
06:10:02 AM        10       163      3.28      2.82      2.01
06:20:04 AM        10       160      1.36      3.92      3.31
06:30:01 AM         2       156      0.26      0.97      2.06

sar

Code:

05:40:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
05:50:04 AM       all      7.12      0.00      1.74      0.82      0.00     90.31
06:00:01 AM       all     10.35      0.00      5.01      0.68      0.00     83.95
06:10:02 AM       all     14.67      0.00     18.03      2.87      0.00     64.42
06:20:04 AM       all      9.60      0.00     43.74      0.61      0.00     46.05
06:30:01 AM       all      7.32      0.01      1.81      1.28      0.00     89.58

Any suggestions on things I can check such as a log file, updating some settings, etc?

mistwang · May 13, 2010

What is the "Connection Timeout"
http://www.litespeedtech.com/docs/webserver/config/tuning/#connTimeout

It only happen when request takes longer than "Connection Timeout" to process. Check access see what request have 408 result if you can, see if you can reproduce it by accessing it.

Maybe it is a pike in disk I/O or something cause the server choked by that time.

MikeDVB · May 13, 2010

Here's the node I/O wait itself:

Code:

05:50:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
06:00:01 AM       all     38.88      0.21     10.33      4.75      0.00     45.83
06:10:04 AM       all     35.52      1.18     10.35     14.71      0.00     38.24
06:20:01 AM       all     35.75      0.44     10.31      6.13      0.00     47.37
06:30:01 AM       all     36.07      0.19     10.36      6.58      0.00     46.80

It did peak up to 14% but that's not terribly bad on Raid10 4x15K RPM SAS drives and not to mention it only affected this *one* VPS and not every system on the node.

I/O wait doesn't look to be the issue on the node or on the VPS itself and I've had others reporting this issue when I was able to access the site with the reported problem with no issues.

An example is one client is running a forum and people are posting about getting 408 errors while the operator of the forum as well as myself are able to browse the forms/post/search/etc all without ever receiving an issue.

I scanned the domlogs for the forum and I found 20 occurrences of the 408 errors all on various different portions of the forums.

The connection timeout is set to 15 seconds...

I do see this in the domlogs right before the 408 issues:

Code:

219.90.189.181 - - [13/May/2010:16:50:26 -0400] "GET URL HTTP/1.1" 304 0 "URL" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET
CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
LCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
C1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
.NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
.NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
.NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
LCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
CC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
 .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
T CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
 CLR 3.5.30729)"

This struck me as odd but I don't know that it would cause issues with LiteSpeed? (And yes, it did have a real request, I just changed it to "URL" to protect customer information).

MikeDVB · May 14, 2010

I'm not sure if they would cause 408 errors but I've raised the client throttling quite a bit and the connection limits.

I'm really at a loss for what else to tweak/check without any feedback from LST.

NiteWave · May 14, 2010

the log looks strange except 1st line. or any typo?

according the 408 error_log entry, check corresponding access_log entries for same IP around the time stamp, may find something unusual. for example, if above error log not typo, it looks in the client's http request, User-Agent is too long, this may trigger "Status 400: Http request header is too big, abandon!" in error_log.

MikeDVB · Jun 8, 2010

I'm still having one client who is reporting this issue - she has a VPS to herself and is running LiteSpeed.

I can see no reason for the error - we're hosting over 4,000 domains including other forums on the same version with no issues whatsoever so it's baffling.

Any ideas?

mistwang · Jun 8, 2010

Can you reproduce the error yourself?

MikeDVB · Jun 8, 2010

I can't seem to trigger it myself but I can see that it is indeed happening from the error logs.

MikeDVB · Jun 19, 2010

More reports of 408 errors, now on another server, another domain, another account.

LSWS 4.0.15 x64

Code:

2010-06-18 23:02:38.401 [INFO] [24.186.81.138:64624-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 22:56:58.804 [INFO] [24.186.81.138:61731-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 22:56:43.402 [INFO] [24.186.81.138:61659-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 22:53:49.217 [INFO] [24.186.81.138:60942-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 22:52:03.901 [INFO] [77.88.28.247:55652-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 22:46:50.801 [INFO] [24.186.81.138:57936-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 22:46:39.301 [INFO] [77.88.28.247:51371-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 22:45:51.302 [INFO] [24.186.81.138:57361-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 22:38:34.800 [INFO] [66.249.71.163:46807-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 22:37:03.323 [INFO] [24.186.81.138:65421-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 21:57:51.401 [INFO] [77.88.28.247:56477-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 21:49:29.900 [INFO] [24.186.81.138:54940-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 21:49:09.005 [INFO] [24.186.81.138:54934-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
2010-06-18 21:39:48.809 [INFO] [77.88.28.247:51613-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml]

Any suggestions on troubleshooting this?

mistwang · Jun 21, 2010

check access log or something to find out what URL causes 408, try to reproduce it.
If it is reproducible, try strace PHP process associated with the request result in 408 error, assuming it is a PHP script.

408 Errors

MikeDVB

Well-Known Member

mistwang

LiteSpeed Staff

MikeDVB

Well-Known Member

MikeDVB

Well-Known Member

NiteWave

Administrator

MikeDVB

Well-Known Member

mistwang

LiteSpeed Staff

MikeDVB

Well-Known Member

MikeDVB

Well-Known Member

mistwang

LiteSpeed Staff