408 Errors

Discussion in 'General' started by MikeDVB, May 13, 2010.

  1. MikeDVB

    MikeDVB New Member

    Hello,

    I seem to be getting 408 errors on a few LiteSpeed powered VPSs and I'm not entirely sure as to why. The node loads/ram/ I/O etc... are all fine as are the statistics on the VPSs that are having the issues.

    This is on 4.0.13 - I did try 4.0.14 previously but was getting 503 errors so I rolled back.

    Code:
    HTTP/1.0 408 Request Time-out
    Date: Thu, 13 May 2010 10:04:43 GMT
    Server: LiteSpeed
    Connection: close
    Cache-Control: private, no-cache, max-age=0
    Pragma: no-cache
    Content-Type: text/html
    Content-Length: 530
    Code:
    <html>
    <head><title> 408 Request Time-out
    </title></head>
    <body><h1> 408 Request Time-out
    </h1>
    This request takes too long to process,  it is timed out by the server. If it should not be timed out, please contact administrator of this web site  to increase 'Connection Timeout'.
    <hr />
    Powered By <a href='http://www.litespeedtech.com'>LiteSpeed Web Server</a><br />
    <font face="Verdana, Arial, Helvetica" size=-1>LiteSpeed Technologies is not responsible for administration and contents of this web site!</font></body></html>
    This happened between 6:03 AM and 6:18 AM.

    sar -r
    Code:
    05:40:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
    05:50:04 AM    243936    280352     53.47         0         0         0         0      0.00         0
    06:00:01 AM    242300    281988     53.78         0         0         0         0      0.00         0
    06:10:02 AM    204064    320224     61.08         0         0         0         0      0.00         0
    06:20:04 AM    245820    278468     53.11         0         0         0         0      0.00         0
    06:30:01 AM    268680    255608     48.75         0         0         0         0      0.00         0
    sar -q
    Code:
    05:40:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    05:50:04 AM         2       159      1.12      0.78      0.92
    06:00:01 AM         3       162      0.83      0.96      0.98
    06:10:02 AM        10       163      3.28      2.82      2.01
    06:20:04 AM        10       160      1.36      3.92      3.31
    06:30:01 AM         2       156      0.26      0.97      2.06
    sar
    Code:
    05:40:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
    05:50:04 AM       all      7.12      0.00      1.74      0.82      0.00     90.31
    06:00:01 AM       all     10.35      0.00      5.01      0.68      0.00     83.95
    06:10:02 AM       all     14.67      0.00     18.03      2.87      0.00     64.42
    06:20:04 AM       all      9.60      0.00     43.74      0.61      0.00     46.05
    06:30:01 AM       all      7.32      0.01      1.81      1.28      0.00     89.58
    Any suggestions on things I can check such as a log file, updating some settings, etc?
    Last edited: May 13, 2010
  2. mistwang

    mistwang LiteSpeed Staff

    What is the "Connection Timeout"
    http://www.litespeedtech.com/docs/webserver/config/tuning/#connTimeout

    It only happen when request takes longer than "Connection Timeout" to process. Check access see what request have 408 result if you can, see if you can reproduce it by accessing it.

    Maybe it is a pike in disk I/O or something cause the server choked by that time.
  3. MikeDVB

    MikeDVB New Member

    Here's the node I/O wait itself:
    Code:
    05:50:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
    06:00:01 AM       all     38.88      0.21     10.33      4.75      0.00     45.83
    06:10:04 AM       all     35.52      1.18     10.35     14.71      0.00     38.24
    06:20:01 AM       all     35.75      0.44     10.31      6.13      0.00     47.37
    06:30:01 AM       all     36.07      0.19     10.36      6.58      0.00     46.80
    
    It did peak up to 14% but that's not terribly bad on Raid10 4x15K RPM SAS drives and not to mention it only affected this *one* VPS and not every system on the node.

    I/O wait doesn't look to be the issue on the node or on the VPS itself and I've had others reporting this issue when I was able to access the site with the reported problem with no issues.

    An example is one client is running a forum and people are posting about getting 408 errors while the operator of the forum as well as myself are able to browse the forms/post/search/etc all without ever receiving an issue.

    I scanned the domlogs for the forum and I found 20 occurrences of the 408 errors all on various different portions of the forums.

    The connection timeout is set to 15 seconds...

    I do see this in the domlogs right before the 408 issues:

    Code:
    219.90.189.181 - - [13/May/2010:16:50:26 -0400] "GET URL HTTP/1.1" 304 0 "URL" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET
    CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    LCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    C1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    ; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    LCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    CC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
     .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    ; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    T CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
    ; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"
     CLR 3.5.30729)"
    
    This struck me as odd but I don't know that it would cause issues with LiteSpeed? (And yes, it did have a real request, I just changed it to "URL" to protect customer information).
    Last edited: May 14, 2010
  4. MikeDVB

    MikeDVB New Member

    I'm not sure if they would cause 408 errors but I've raised the client throttling quite a bit and the connection limits.

    I'm really at a loss for what else to tweak/check without any feedback from LST.
  5. NiteWave

    NiteWave Administrator

    the log looks strange except 1st line. or any typo?

    according the 408 error_log entry, check corresponding access_log entries for same IP around the time stamp, may find something unusual. for example, if above error log not typo, it looks in the client's http request, User-Agent is too long, this may trigger "Status 400: Http request header is too big, abandon!" in error_log.
  6. MikeDVB

    MikeDVB New Member

    I'm still having one client who is reporting this issue - she has a VPS to herself and is running LiteSpeed.

    I can see no reason for the error - we're hosting over 4,000 domains including other forums on the same version with no issues whatsoever so it's baffling.

    Any ideas?
  7. mistwang

    mistwang LiteSpeed Staff

    Can you reproduce the error yourself?
  8. MikeDVB

    MikeDVB New Member

    I can't seem to trigger it myself but I can see that it is indeed happening from the error logs.
  9. MikeDVB

    MikeDVB New Member

    More reports of 408 errors, now on another server, another domain, another account.

    LSWS 4.0.15 x64

    Code:
    2010-06-18 23:02:38.401 [INFO] [24.186.81.138:64624-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 22:56:58.804 [INFO] [24.186.81.138:61731-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 22:56:43.402 [INFO] [24.186.81.138:61659-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 22:53:49.217 [INFO] [24.186.81.138:60942-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 22:52:03.901 [INFO] [77.88.28.247:55652-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 22:46:50.801 [INFO] [24.186.81.138:57936-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 22:46:39.301 [INFO] [77.88.28.247:51371-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 22:45:51.302 [INFO] [24.186.81.138:57361-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 22:38:34.800 [INFO] [66.249.71.163:46807-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 22:37:03.323 [INFO] [24.186.81.138:65421-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 21:57:51.401 [INFO] [77.88.28.247:56477-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 21:49:29.900 [INFO] [24.186.81.138:54940-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 21:49:09.005 [INFO] [24.186.81.138:54934-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    2010-06-18 21:39:48.809 [INFO] [77.88.28.247:51613-0#APVH_redacted.com] File not found [/home/redacted/public_html/408.shtml] 
    Any suggestions on troubleshooting this?
  10. mistwang

    mistwang LiteSpeed Staff

    check access log or something to find out what URL causes 408, try to reproduce it.
    If it is reproducible, try strace PHP process associated with the request result in 408 error, assuming it is a PHP script.

Share This Page