Random 404s

[matt]

On random pages I'm getting a 404 error from Litespeed, if I reload a few times the correct page is shown. These are pages that use mod_rewrite. This is all there is in the logs:

2005-12-26 22:42:47.931 [INFO] [24.6.161.136:3687-0#WP.org] File not found [/home/wporg/public_html/support/topic/52511]


This is cropping up frequently and as we're doing a new launch of WordPress and WordPress.org today it's a pretty major issue. Any ideas? This is with the latest 2.1.8.

[mistwang]

What rewrite rule is used?

Is [/home/wporg/public_html/support/topic/52511] a directory or a file?

If it happens randomly, I think it might be a problem like "running out of file descriptors".

[matt]

Here are the relevant rewrite rules.

# SUPPORT FORUMS
RewriteRule ^tags/(.*) /support/tags.php?tag=$1
RewriteRule ^support/topic/([0-9]+)/page/([0-9]+) /support/topic.php?id=$1&page=$2 [L]
RewriteRule ^support/topic/([0-9]+) /support/topic.php?id=$1 [L]
RewriteRule ^support/forum/([0-9]+)/page/([0-9]+) /support/forum.php?id=$1&page=$2 [L]
RewriteRule ^support/forum/([0-9]+) /support/forum.php?id=$1 [L]
RewriteRule ^support/profile/([0-9]+) /support/profile.php?id=$1 [L]


It works 95% of the time. How would I check file descriptors?

[mistwang]

Looks like a problem with the rewrite engine, not the problem I mentioned.

Can you please turn on the rewrite logging under the "Rewrite" tab? And send me the log messages for a request that results in 404 code (with the file not found error), grep with the client ip address can be used to find relevant log messages.

Looks like the rewrite rule has not been executed for the some request.

You can also try moving the rewrite rule form .htaccess to the vhost level.

[matt]

Long post warning. This is the 10 seconds on either side of a 404 I got, with the rewrite log at 9. The 404 is the very last line, and as you can see there is no relevant rewrite logging before it.

Something else that's notable is the 404 is the Litespeed default one, not my custom one specified in the .htaccess.

two:/opt/lsws/logs# egrep '2005-12-27 00:28:[23]' error.log.2005_12_27.1 | grep '64.132.227.126'
2005-12-27 00:28:20.139 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] strip rewrite base: '/' from URI: '/style/cool-fade.gif'
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^style/v3-head/(.*)', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(bugs|bug|bugzilla|bugreport|bugreports|messups| hiddenfeatures|crap|mistakes|eastereggs|oopsies)\. ?wordpress\.org$', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(docs|documentation|doc|codec|newbies|noonereads this)\.?wordpress\.org$', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^tags/(.*)', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^support/topic/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^support/topic/([0-9]+)', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^support/forum/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^support/forum/([0-9]+)', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^support/profile/([0-9]+)', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^support/[0-9]+/([0-9]+)/?$', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^latest.tar.gz.*', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^latest.zip.*', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^latest$', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern 'licensing/?', result: -1
2005-12-27 00:28:20.141 [INFO] [64.132.227.126:63822-2#WP.org] [REWRITE] Rule: Match 'style/cool-fade.gif' with pattern '^nightly/?', result: -1
2005-12-27 00:28:20.147 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] strip rewrite base: '/' from URI: '/style/header-logo.png'
2005-12-27 00:28:20.147 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^style/v3-head/(.*)', result: -1
2005-12-27 00:28:20.147 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.147 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(bugs|bug|bugzilla|bugreport|bugreports|messups| hiddenfeatures|crap|mistakes|eastereggs|oopsies)\. ?wordpress\.org$', result: -1
2005-12-27 00:28:20.147 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.147 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(docs|documentation|doc|codec|newbies|noonereads this)\.?wordpress\.org$', result: -1
2005-12-27 00:28:20.147 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^tags/(.*)', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^support/topic/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^support/topic/([0-9]+)', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^support/forum/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^support/forum/([0-9]+)', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^support/profile/([0-9]+)', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^support/[0-9]+/([0-9]+)/?$', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^latest.tar.gz.*', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^latest.zip.*', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^latest$', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern 'licensing/?', result: -1
2005-12-27 00:28:20.148 [INFO] [64.132.227.126:63816-1#WP.org] [REWRITE] Rule: Match 'style/header-logo.png' with pattern '^nightly/?', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] strip rewrite base: '/' from URI: '/style/v3-head/reply.png'
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match 'style/v3-head/reply.png' with pattern '^style/v3-head/(.*)', result: 2
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Source URI: 'style/v3-head/reply.png' => Result URI: '/style/wp3-heading.php?letters=reply.png'
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] replace current query string with 'letters=reply.png'
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(bugs|bug|bugzilla|bugreport|bugreports|messups| hiddenfeatures|crap|mistakes|eastereggs|oopsies)\. ?wordpress\.org$', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(docs|documentation|doc|codec|newbies|noonereads this)\.?wordpress\.org$', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^tags/(.*)', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^support/topic/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^support/topic/([0-9]+)', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^support/forum/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^support/forum/([0-9]+)', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^support/profile/([0-9]+)', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^support/[0-9]+/([0-9]+)/?$', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^latest.tar.gz.*', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^latest.zip.*', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^latest$', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern 'licensing/?', result: -1
2005-12-27 00:28:20.365 [INFO] [64.132.227.126:63822-3#WP.org] [REWRITE] Rule: Match '/style/wp3-heading.php' with pattern '^nightly/?', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] strip rewrite base: '/' from URI: '/style/doily.png'
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^style/v3-head/(.*)', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(bugs|bug|bugzilla|bugreport|bugreports|messups| hiddenfeatures|crap|mistakes|eastereggs|oopsies)\. ?wordpress\.org$', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(docs|documentation|doc|codec|newbies|noonereads this)\.?wordpress\.org$', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^tags/(.*)', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^support/topic/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^support/topic/([0-9]+)', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^support/forum/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^support/forum/([0-9]+)', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^support/profile/([0-9]+)', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^support/[0-9]+/([0-9]+)/?$', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^latest.tar.gz.*', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^latest.zip.*', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^latest$', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern 'licensing/?', result: -1
2005-12-27 00:28:20.370 [INFO] [64.132.227.126:63816-2#WP.org] [REWRITE] Rule: Match 'style/doily.png' with pattern '^nightly/?', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] strip rewrite base: '/' from URI: '/style/code-is-poetry.png'
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^style/v3-head/(.*)', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(bugs|bug|bugzilla|bugreport|bugreports|messups| hiddenfeatures|crap|mistakes|eastereggs|oopsies)\. ?wordpress\.org$', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^(.*)$', result: 2
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Cond: Match 'new.wordpress.org' with pattern '^(docs|documentation|doc|codec|newbies|noonereads this)\.?wordpress\.org$', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^tags/(.*)', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^support/topic/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^support/topic/([0-9]+)', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^support/forum/([0-9]+)/page/([0-9]+)', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^support/forum/([0-9]+)', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^support/profile/([0-9]+)', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^support/[0-9]+/([0-9]+)/?$', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^(.*)/$', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^latest.tar.gz.*', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^latest.zip.*', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^latest$', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern 'licensing/?', result: -1
2005-12-27 00:28:20.594 [INFO] [64.132.227.126:63816-3#WP.org] [REWRITE] Rule: Match 'style/code-is-poetry.png' with pattern '^nightly/?', result: -1
2005-12-27 00:28:31.841 [INFO] [64.132.227.126:63828-0#WP.org] File not found [/home/wporg/public_html/support/profile/44719]

[matt]

Something else that may be relevant, when I get a 404 I click reload over and over but the I just keep getting the 404 for 3-5 seconds.

[mistwang]

Looks like something wrong with the rewrite rules in .htaccess, rewrite rules has not been executed, Can you please try the rewrite rules at the vhost level, see if the problem dissapear. Please make sure that the full URL is used in the vhost level rewrite rule.

Only custom error pages set at vhost level is supported by LSWS.

We will investigate.

[mistwang]

Does the .htaccess file got updated frequently or the modified time of the file keeps changing?

[matt]

I've been editing it, but not really in the past few hours and the 404s have been continuing.

[mistwang]

Yes, I noticed the 404 errors as well, pretty strange.

Can you turn on the debug logging for a little while, it will help us analyzing the problem. It will generate tons of log messages for a heavily traffic web site, be careful.

You can use the toggle debug logging feature to turn it on for a little while then turn it off after reproducing the problem, please make sure that the server "log level" is set to "DEBUG". The log file can be sent to bug@litespeedtech.com .

[mistwang]

Have you tried ealier release or just 2.1.8?

[matt]

I can downgrade if you want me to try that.

[mistwang]

Yes, please.

I will try to reproduce it in our lab tomorrow, is there any special context defined for that vhost? It will be great if you can send us the xml configuration for that vhost, and the .htaccess .

I also wonder if it has anything to do with the traffic volume?

[matt]

Possibly on the traffic, we're on the front page of Digg right now and a ton of blogs.

I downgraded to 2.1.7 and still having a prob, next earliest I have is 2.1.5 which I remember having a prob with hanging PHP procs.

[matt]

I emailed all my conf files and .htaccess to your support email, let me know if anything else is needed.

[mistwang]

Is that possible that there is some process keep recreating the .htaccess, even thoug the content of .htaccess does not change? Maybe not that specific file, but other .htaccess file along the path.

As lshttpd cache the .htaccess for 10 seconds, then check for changes, reload it if the modified time was changed, if the .htaccess file is being recreated when lshttpd reload the .htaccess, lshttpd may see an incomplete .htaccess without rewrite rules, and that will be cached for about 10 seconds as well.

That's just my guess based on the symtom of the 404 errors, as when 404 errors occur, it will last about a couple seconds then back to normal and all requests to the forum will return 404. You can confirm that with the timestamp of those missing file errors in the error log.

If .htaccess do get recreated once for a while, please make sure to create a temp file and move the temp file to .htaccess, not to overwrite it directly.

I am still trying to reproduce the problem. :-)

[mistwang]

Another quick question, is there a .htacess file under "support/" folder? If yes, what is the content and does it change frequently?

[mistwang]

Hi Matt,

Please download and install http://www.litespeedtech.com/packages/2.1/lsws-2.1.9-ent-i386-linux.tar.gz, I had added some log messages to help identify the source of the problem. I think the problem is that, for some reason, lshttpd failed to open the .htaccess file, either run out of file descriptor or the file is being updated.

Anyway, please follow our howto to increase the file descriptor limit
http://www.litespeedtech.com/docs/HowTo_QA.html#qa_incfd

You can check how many file descriptors has been used with command "lsof -p <pid_of_lshttpd>".

then upgrade to 2.1.9, please watch out log messages containing "[HTAccess]" tag in the error.log. Normally, you should only see such messages at server start up or .htaccess has been changed.

[matt]

The root .htaccess is the only one, it changes pretty rarely, really only when I'm working on the site.

WordPress.org is on standard, not enterprise, though I can purchase one if needed. Will that file you pointed to still work?

[matt]

The lshttpd had 170 open files according to lsof. There are about 1500 open total on the system.

ulimit -a returns this:

two:~# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited


This seemed okay:

two:~# more /proc/sys/fs/file-max
64000

Anything else I can provide?

[mistwang]

Just change "ent" to "std" to download the new release.
If you are using the standard edition, then I think it is more likely caused by per process file descriptor limit.

You can try following if you can not reboot the server after then system wide configuration change, as root

# ulimit -n 10000
#lsws/bin/lswsctrl stop
#lsws/bin/lswsctrl start


Do not use lswsctrl restart, it won't work.

The file descriptor limit problem does not likely happen with enterprise as the default file descriptor limit for enterprise is much higher. :-)

Anyway, raise the limit see if it helps.

[matt]

Raising the limit didn't make a diff.

Running 2.1.9, here's a second of logs where the 404 thing happened to me. (The forum load.)

2005-12-28 14:50:58.047 [INFO] [141.91.240.162:22937-0#WP.org] File not found [/home/wporg/public_html/style/v3-head/forums.png]
2005-12-28 14:50:58.075 [INFO] [80.67.17.116:38120-0#WP.org] [REWRITE] strip rewrite base: '/development/' from URI: '/development/feed/'
2005-12-28 14:50:58.075 [INFO] [80.67.17.116:38120-0#WP.org] [REWRITE] Rule: Match 'feed/' with pattern '.', result: 1
2005-12-28 14:50:58.075 [INFO] [80.67.17.116:38120-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.075 [INFO] [80.67.17.116:38120-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.075 [INFO] [80.67.17.116:38120-0#WP.org] [REWRITE] Source URI: 'feed/' => Result URI: '/development/index.php'
2005-12-28 14:50:58.218 [INFO] [64.132.227.126:64223-1#WP.org] File not found [/home/wporg/public_html/support/topic/36894]
2005-12-28 14:50:58.234 [INFO] [69.27.96.169:33456-0#WP.org] [REWRITE] strip rewrite base: '/development/' from URI: '/development/feed/'
2005-12-28 14:50:58.234 [INFO] [69.27.96.169:33456-0#WP.org] [REWRITE] Rule: Match 'feed/' with pattern '.', result: 1
2005-12-28 14:50:58.234 [INFO] [69.27.96.169:33456-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.234 [INFO] [69.27.96.169:33456-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.234 [INFO] [69.27.96.169:33456-0#WP.org] [REWRITE] Source URI: 'feed/' => Result URI: '/development/index.php'
2005-12-28 14:50:58.278 [INFO] [213.139.7.91:50314-0#WP.org] [REWRITE] strip rewrite base: '/development/' from URI: '/development/feed/'
2005-12-28 14:50:58.278 [INFO] [213.139.7.91:50314-0#WP.org] [REWRITE] Rule: Match 'feed/' with pattern '.', result: 1
2005-12-28 14:50:58.278 [INFO] [213.139.7.91:50314-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.278 [INFO] [213.139.7.91:50314-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.278 [INFO] [213.139.7.91:50314-0#WP.org] [REWRITE] Source URI: 'feed/' => Result URI: '/development/index.php'
2005-12-28 14:50:58.376 [INFO] [134.58.179.8:57018-0#WP.org] File not found [/home/wporg/public_html/style/v3-head/wordpress_web_hosting.png]
2005-12-28 14:50:58.390 [INFO] [81.206.163.203:25571-0#WP.org] File not found [/home/wporg/public_html/support/topic/3282]
2005-12-28 14:50:58.399 [INFO] [86.202.232.239:49793-0#WP.org] File not found [/home/wporg/public_html/style/v3-head/plugins/links.png]
2005-12-28 14:50:58.430 [INFO] [83.237.190.110:50821-0#WP.org] File not found [/home/wporg/public_html/style/v3-head/welcome.png]
2005-12-28 14:50:58.437 [INFO] [213.54.153.40:33228-0#WP.org] File not found [/home/wporg/public_html/style/v3-head/installing_wordpress.png]
2005-12-28 14:50:58.442 [INFO] [130.208.195.18:58597-0#WP.org] [REWRITE] strip rewrite base: '/development/' from URI: '/development/feed/'
2005-12-28 14:50:58.442 [INFO] [130.208.195.18:58597-0#WP.org] [REWRITE] Rule: Match 'feed/' with pattern '.', result: 1
2005-12-28 14:50:58.442 [INFO] [130.208.195.18:58597-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.442 [INFO] [130.208.195.18:58597-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.442 [INFO] [130.208.195.18:58597-0#WP.org] [REWRITE] Source URI: 'feed/' => Result URI: '/development/index.php'
2005-12-28 14:50:58.476 [INFO] [66.249.65.13:49596-3#WP.org] File not found [/home/wporg/public_html/support/topic/21636]
2005-12-28 14:50:58.612 [INFO] [62.75.160.115:36683-0#WP.org] [REWRITE] strip rewrite base: '/development/' from URI: '/development/feed/'
2005-12-28 14:50:58.612 [INFO] [62.75.160.115:36683-0#WP.org] [REWRITE] Rule: Match 'feed/' with pattern '.', result: 1
2005-12-28 14:50:58.612 [INFO] [62.75.160.115:36683-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.612 [INFO] [62.75.160.115:36683-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed/ ) failed
2005-12-28 14:50:58.612 [INFO] [62.75.160.115:36683-0#WP.org] [REWRITE] Source URI: 'feed/' => Result URI: '/development/index.php'
2005-12-28 14:50:58.621 [INFO] [83.237.190.110:50821-1#WP.org] File not found [/home/wporg/public_html/style/v3-head/welcome_back.png]
2005-12-28 14:50:58.636 [INFO] [200.91.233.179:3445-1#WP.org] File not found [/home/wporg/public_html/latest.zip]
2005-12-28 14:50:58.689 [INFO] [80.58.46.235:36604-1#WP.org] File not found [/home/wporg/public_html/theme/connections.zip]
2005-12-28 14:50:58.749 [INFO] [59.167.66.14:23607-0#WP.org] [REWRITE] strip rewrite base: '/development/' from URI: '/development/feed'
2005-12-28 14:50:58.749 [INFO] [59.167.66.14:23607-0#WP.org] [REWRITE] Rule: Match 'feed' with pattern '.', result: 1
2005-12-28 14:50:58.749 [INFO] [59.167.66.14:23607-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed ) failed
2005-12-28 14:50:58.749 [INFO] [59.167.66.14:23607-0#WP.org] [REWRITE] stat( /home/wporg/public_html/development/feed ) failed
2005-12-28 14:50:58.749 [INFO] [59.167.66.14:23607-0#WP.org] [REWRITE] Source URI: 'feed' => Result URI: '/development/index.php'
2005-12-28 14:50:58.942 [INFO] [61.15.166.187:2862-0#WP.org] File not found [/home/wporg/public_html/style/v3-head/welcome.png]

[mistwang]

Matt,

I forgot to tell that you can turn off rewrite logging if you don't need them, so the log file will not grow very fast.

The log messages that I am interested in is the log messages with tag "HTAccess". You can email me a chunk of the error log when there is no 404, then 404 errors, then back to normal, or let me logon the admin interface temporarily to check the error log.

Thanks,

George

[mistwang]

Noticed that you already changed the URL format.


Can you please send me the full error.log of 2.1.9 for analysis, if you have one.

Thanks!

[matt]

Yeah the forum folks were about to bite my head off.

I'll upload the logs to a directory and send you a link.

[xing]

My post is not about the bug/debugging session currently in progress. But could help matt geting things running again quickly while the bug is taken care of.

Instead of rewrite rules, you can use lsws's static context and use the regex you have to bind to specific scripts.

So:

exp:^/forum/[0-9]+/ ---> /myprivate/forum.php

Of course unlike the rewrite method, using the context way requires your forum.php to parse the data out of the url.

Basically the same goal done different ways and fairly easy enough to get you off the bug on the temporary basis fairly quicky. I use the setup extensively.

[mistwang]

Matt,

Thank you for your assistance, finally, I figured out this nasty bug, a wrong variable has been used while building our htaccess cache. :oops:

Please download 2.1.9 package again. I am pretty confident that the bug has been fixed. :-)

I guess I can sleep well tonight. ;-)

Xing,

Thank you for your suggestion. The context matching way does have slightly performance advantage. :-)

Acctually, query string can be set in a matching context as well, like
exp:^/forum/([0-9]+)/(.*) ---> /myprivate/forum.php?forum=$1\&post=$2

Becareful to escape the '&', otherwise, it will be replaced by the matching URI. ;-)

[xing]

Acctually, query string can be set in a matching context as well, like
exp:^/forum/([0-9]+)/(.*) ---> /myprivate/forum.php?forum=$1\&post=$2


Nice! Didn't know I have those regex variables to work with! Intersting. Will likely think of something useful to do in the future with this.

[matt]

Since the new 2.1.9 everything seems good so far.