Perpetual high amount of Requests until restart...

[MikeDVB]

I thought that perhaps somebody was just hammering the server but upon a restart the request in processing on the hostname dropped from 400 sustained down to 0~50ish. I am running the *latest* 4.0.10 build.

A while after restarting litespeed:

From watching it appears the "Requests in processing" just continues to go up without *ever* going down. Checking my other servers now.

 

[MikeDVB]

I'm seeing similar behavior on my other servers:

On a side note I rolled back to the August 05 LSWS 4.0.10 and now I cannot access the control panel. I do remember hearing something about changes to LSWS that would break the control panel if you rolled back to an older version but I didn't anticipate such a change in the latest 4.0.10 I guess.

 

[MikeDVB]

I re-ran the latest 4.0.10 install (didn't re-download it, just re-ran the install) and that cleared up the issue on the primary server however I am going to leave the other two as they are so George/LST can look into it if they wish.

I've opened a ticket - I am more than happy to provide you what you need to diagnose this.

 

[mistwang]

Please run the installer of earlier 4.0.10 release to roll back.
The new 4.0.10 release uses suEXEC for the web console, so it requires running the installer to fix the permissions, you may need to change the ownership of lsws/admin/tmp from "lsadm" to "nobody".

Maybe I should take a look at your server if possible. It looks like the statistic data is wrong as the number of connections are very low.

 

[MikeDVB]

Please run the installer of earlier 4.0.10 release to roll back.

I did and I couldn't tell if that "fixed" the issue due to not being able to get into the control panel.

The new 4.0.10 release uses suEXEC for the web console, so it requires running the installer to fix the permissions, you may need to change the ownership of lsws/admin/tmp from "lsadm" to "nobody".

Yeah - I just re-ran the newest installer and the issue appears to have disappeared on that server. (The issue of requests in processing I mean). I remember from speaking with Tony that you were doing something that would break backwards compatibility with older versions of the control panel so it wasn't a big surprise.

Maybe I should take a look at your server if possible. It looks like the statistic data is wrong as the number of connections are very low.

I woudl think it was just the statistical data if there weren't actual performance issues occurring that stopped the second I restarted the LSWS and that number dropped back down to near 0.

Just tell me what you need in the ticket

 

[mistwang]

That vhost is the default catch all vhost without PHP suEXEC.

We will try some trouble shoot in our lab first, if we could not reproduce it, will contact you.

408 timeout may not relate to the high number of requests in process, but the graceful restart.

 

[MikeDVB]

We will try some trouble shoot in our lab first, if we could not reproduce it, will contact you.

I'd rather you just look into it on the server that is experiencing this issue currently - I have no problem with this.

408 timeout may not relate to the high number of requests in process, but the graceful restart.

No, that was sent to me which caused me to look into it and discover this issue.

It was before the graceful restart.

 

[mistwang]

OK, the high number of requests in process has been fixed. It is just a statistical error.

Please download 4.0.10 and upgrade again. Just change the version number in the download link to get it.

The 408 error is a different issue I think. I will follow up with the ticket.

 

[MikeDVB]

408 Request Time-out
This request takes too long to process, it is timed out by the server. If it should not be timed out, please contact administrator of this web site to increase 'Connection Timeout'.

Suggestions?

Running latest 4.0.10

 

[MikeDVB]

I set up a small tester (all it does is a meta-refresh every 30 seconds and it displays the date/time using PHP)

I got this just now:

 

[mistwang]

you need to increase
http://www.litespeedtech.com/docs/webserver/config/tuning/#connTimeout

If a script indeed takes more than 30 seconds to finish, otherwise, need to trouble shoot the script why it takes so long.

 

[MikeDVB]

It's happening on page loads that are instant (or near instant).

It's nothing that is taking more than 5 seconds at most, and the server connection time-out is set to 120 seconds.

I can guarantee you nobody is waiting 120 seconds for this error to pop up.

I did do a forced restart of LSWS and that appears to have cleared up the 408's however I am a bit curious about the one page load where the php script was attempted to be downloaded instead of processed - even if very rare this is dangerous especially with config.php, configuration.php, and other such script configuration files.

I would also like to add that when the "download 408tester.php" popped up - it was more than 30 minutes after doing the LSWS restart.

 

[mistwang]

I can login take a look at the log. is it the same server as last one?
Does the 408tester.php issue can be reliably reproduced, 30 minutes after LSWS restart?

 

[mistwang]

It's happening on page loads that are instant (or near instant).

It's nothing that is taking more than 5 seconds at most, and the server connection time-out is set to 120 seconds.

Review the code, the only condition to return 408 timeout response is when there is no socket I/O activity (bytes in/out) for a connection in more than "Connection timeout (Seconds)". So, I have to experience it myself to believe it.

For the download popup, it is likely because a lshttpd crash, it wont send back the script content. So, it is a different issue to the 408 timeout.

 

[MikeDVB]

Review the code, the only condition to return 408 timeout response is when there is no socket I/O activity (bytes in/out) for a connection in more than "Connection timeout (Seconds)". So, I have to experience it myself to believe it.

It's happening on page loads (when navigating to a new page) and it is showing up *instantly* upon clicking the button or entering the URL.

For the download popup, it is likely because a lshttpd crash, it wont send back the script content. So, it is a different issue to the 408 timeout.

I hope this doesn't happen often - is this saved in a log (is it something I can grep for)?

Here is the code:

root@atlantis [/home/transfer/public_html]# cat 408tester.php
<html><head><meta http-equiv="refresh" content="30" /><title>408 Tester</title></head><body>408 Page Tester<br><?php echo date("D dS M,Y h:i a"); ?></body></htm>

 

[mistwang]

You can grep for 408.shtml in /usr/local/apache/logs/error_log.
Your code code missing a "l" in tag "</htm>" at the end.

 

[MikeDVB]

You can grep for 408.shtml in /usr/local/apache/logs/error_log.
Your code code missing a "l" in tag "</htm>" at the end.

Actually it's intact on the server I simply didn't select the whole thing when I copied/pasted and then I added the ">" not noticing that it was missing an L as well.

-rw-r--r-- 1 transfer transfer   162 Aug 21 15:52 408tester.php
root@atlantis [/home/transfer/public_html]# cat 408tester.php
<html><head><meta http-equiv="refresh" content="30" /><title>408 Tester</title></head><body>408 Page Tester<br><?php echo date("D dS M,Y h:i a"); ?></body></html>root@atlantis [/home/transfer/public_html]#

As you can see from the above the file was last modified well before my post and you can see in the screen-shot the reason that I missed the l> was that the l> was put on the next line before the bash command line.

Trust me, the issues I am having are not due to my code

The 408 issue has disappeared since I did a forced restart of LSWS but I am still getting the occasional request to download the .php file instead of the server processing it (and yes, I'm getting the contents of the PHP file).

 

[mistwang]

I am still getting the occasional request to download the .php file instead of the server processing it (and yes, I'm getting the contents of the PHP file).

Is it random? Usually it will be cached by browser.
Please try a command line tool and dump the response header as well. I usually use "lynx -mime_header <url>".

 

[mistwang]

Sometimes, the browser will cache a page having problem, make sure to clear browser cache.
If need, please submit a support ticket with temp login, we will take a look.

 

[MikeDVB]

I'm trying to recreate the issue in some form of reliable way so that I can submit it in a ticket and you guys can replicate/fix it if necessary