[mistwang]

Just as an experiment, I renamed the RAILS_ROOT/public/404.html and 500.html files and now I'm getting the following:
Code:

Status: 500 Internal Server Error Content-Type: text/html
Application error (Rails)

Color me stumped.

Looks lite rails did that, as LiteSpeed has no idea about using 404.html, and will not produce a 500 response.

Are you using rewrite rule or 404 handler to dispatch request to rails dispatcher?
I guess that the rewritten url "/system/maintance.html" has been redirected to Rails dispatcher instead of being served by web server itself. You can turn on debug logging to finit it out.
Also, you can try changing the rewrite rule from

RewriteRule ^.*$ /system/maintenance.html [L]

to

RewriteRule ^.*$ /system/maintenance.html [L,R]

to perform a redirect, maybe that can help.