408 Errors - Fairly Common

[MikeDVB]

We're running LSWS 4.0.10 (latest build) on all of our production servers and we are having users that are complaining of 408 errors occasionally.

It's not after a long time, it's immediately on a page load such as when navigating from one page to the next.

Now before you say the server is over-loaded it's a Quad Core 2.4ghz 4gb ram and it is running at or below a load of 0.6~0.7 24/7 and only has around 30 accounts on it. The server is also not seeing any I/O wait because it's Raid10 4x250gb Sata2 drives and is very under-utilized.

I can assure you that this is not a script timing out or some other issue because it's happened to me on many occasions just for simple scripts and things that take no time to process at all... It isn't that you click to go to the next page, wait 10 seconds, and then the error pops up. The instant the request is sent to the server for the page it comes right back immediately with a 408 error.

[MikeDVB]

I just grepped the apache error log for 408.shtml and this is what I came up with: (I censored out domain and account names and absolute paths but you get the idea)
[quote]2009-08-25 13:47:27.450 [INFO] [94.20.29.86:1961-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 13:49:43.012 [INFO] [82.118.157.208:4321-1#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 13:52:24.318 [INFO] [82.118.157.24:1968-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 13:54:25.687 [INFO] [82.118.157.208:4812-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 13:57:12.073 [INFO] [81.17.80.65:4854-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:01:11.813 [INFO] [94.20.29.86:2456-1#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:03:14.605 [INFO] [81.17.80.65:1109-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:06:07.112 [INFO] [88.151.194.7:1316-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:09:24.284 [INFO] [85.132.55.3:1353-0#APVH_domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:11:25.553 [INFO] [217.25.18.72:2368-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:13:26.443 [INFO] [82.118.157.24:3261-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:15:57.684 [INFO] [82.118.157.24:3404-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:18:08.505 [INFO] [85.132.106.44:2437-1#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:21:19.702 [INFO] [85.132.64.154:1425-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:23:51.217 [INFO] [81.21.82.221:1269-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:27:26.664 [INFO] [80.69.50.213:1131-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:29:41.703 [INFO] [88.151.192.182:19419-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:33:15.709 [INFO] [85.132.55.3:1474-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:36:42.576 [INFO] [85.132.106.44:3108-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:38:58.013 [INFO] [217.64.29.104:1684-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:41:45.802 [INFO] [82.118.136.166:1160-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:44:56.622 [INFO] [217.14.105.224:3466-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:50:36.619 [INFO] [217.64.29.104:1868-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:54:31.900 [INFO] [80.69.50.213:1677-6#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 14:58:31.410 [INFO] [212.38.116.13:2919-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 15:01:02.506 [INFO] [212.38.116.13:3111-1#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 15:03:04.012 [INFO] [217.64.29.104:2196-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 15:07:35.402 [INFO] [217.14.105.224:4139-1#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 15:09:36.932 [INFO] [217.64.29.104:2257-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 15:11:39.185 [INFO] [217.14.105.224:4282-0#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 15:13:56.212 [INFO] [85.132.106.44:4509-1#APVH_*.domain.com] File not found [/home/domain/public_html/408.shtml]
2009-08-25 15:15:57.360 [INFO] [217.14.

[MikeDVB]

On another server:
root@atlantis [/]# cat /usr/local/apache/logs/error_log | grep 408.shtml | wc -l
232

root@boreas [/]# cat /usr/local/apache/logs/error_log | grep 408.shtml | wc -l
168

And that is just since yesterday night around 11pm - I am sure there are far more where a 408 page is actually found and as such it isn't reported in the apache error log...

Also this is happening to every account, it's not any specific script, account, user, or anything like that.

[DanEZPZ]

Just checked my logs and am seeing the same thing. One server is fairly worrying -

[~]# cat /usr/local/apache/logs/error_log | grep 408.shtml | wc -l
101487

[mistwang]

Are you using the x86_64 edition?
It might be related to a recent change, we will have a new build today, see if it help or not.

[MikeDVB]

I am using x64 where Dan is using x32.

[DanEZPZ]

root@srv1 [~]# cat /usr/local/apache/logs/error_log | grep 408.shtml | wc -l
733

root@srv8 [~]# cat /usr/local/apache/logs/error_log | grep 408.shtml | wc -l
701


Nope - I'm using the 32bit version.

[MikeDVB]

I would also like to add that I rotate my apache logs daily so that is only the amount that's occured in the last 24 hours (or less).

[mistwang]

May I login to check one of the server?

[DanEZPZ]

I'll submit a ticket now with some details.

[MikeDVB]

I'll submit a ticket now with some details.

You should have the details from the last time I contacted you George, they've not changed. Do you need them again? Otherwise you are welcome to log in and look.

[anewday]

I'm running 4.0.10 on 32-bit on the latest centos 4 cpanel.

cat /usr/local/apache/logs/error_log | grep 408.shtml | wc -l
0

but a lot of these entries ever since upgrading to 4.0

2009-08-25 06:38:38.719 [INFO] [216.245.196.203:1056-0#APVH_Default] File not found [/usr/local/apache/htdocs/w00tw00t.at.ISC.SANS.DFind:)]
2009-08-25 06:38:38.734 [INFO] [216.245.196.203:1057-0#APVH_Default] File not found [/usr/local/apache/htdocs/w00tw00t.at.ISC.SANS.DFind:)]
2009-08-25 06:38:39.590 [INFO] [216.245.196.203:1445-0#APVH_Default] File not found [/usr/local/apache/htdocs/w00tw00t.at.ISC.SANS.DFind:)]
2009-08-25 19:27:44.457 [INFO] [61.160.216.63:3358-0#APVH_Default] File not found [/usr/local/apache/htdocs/prx.php]
2009-08-25 19:27:44.508 [INFO] [61.160.216.63:3355-0#APVH_Default] File not found [/usr/local/apache/htdocs/prx.php]
2009-08-25 19:27:45.633 [INFO] [61.160.216.63:4572-0#APVH_Default] File not found [/usr/local/apache/htdocs/prx.php]
2009-08-25 19:27:45.655 [INFO] [61.160.216.63:4575-0#APVH_Default] File not found [/usr/local/apache/htdocs/prx.php]

[mistwang]

You should have the details from the last time I contacted you George, they've not changed. Do you need them again? Otherwise you are welcome to log in and look.
Tried a new build on that server. looks like it is working much better now. Monitored it for a while, only one site get 408 errors and I think it is real. Please download 4.0.10 again and upgrade your other servers and give us some feedback.

[MikeDVB]

Tried a new build on that server. looks like it is working much better now. Monitored it for a while, only one site get 408 errors and I think it is real. Please download 4.0.10 again and upgrade your other servers and give us some feedback.

Will do, I don't know if you can say publicly what the change was that you believe caused the issue but if you could update my ticket I am very curious :)

Of course if it's confidential I understand, I am just curious as I'm a developer of over 15 years myself :)