A different Error 503 problem

[lanfusion]

I am new to LSWS. I installed it last night after experiencing performance issues issues with Lighttpd and Ruby on Rails. All of my sites work except http://community.lanfusion.com, a social networking site built in RoR. http://community.lanfusion.com was working fine with Lighttpd and works with Webbrick.

After following the instructions found in this thread: http://www.litespeedtech.com/support/forum/showthread.php?t=1845, I'm still having a 503 issue. Here is some vital info:

Excerpt from /usr/local/lsws/logs/error.log

2008-03-05 16:05:13.254 [INFO] [66.249.66.12:44591-0#community.lanfusion.com] connection to [uds://tmp/lshttpd/community.lanfusion.com:_.sock] on request #0, error: Connection reset by peer!
2008-03-05 16:05:17.503 [INFO] [66.249.66.12:44591-0#community.lanfusion.com] connection to [uds://tmp/lshttpd/community.lanfusion.com:_.sock] on request #0, error: Connection reset by peer!
2008-03-05 16:05:23.587 [INFO] [66.249.66.12:44591-0#community.lanfusion.com] connection to [uds://tmp/lshttpd/community.lanfusion.com:_.sock] on request #0, error: Connection reset by peer!
2008-03-05 16:05:26.819 [NOTICE] [66.249.66.12:44591-0#community.lanfusion.com] oops! 503 Service Unavailable
2008-03-05 16:05:26.819 [NOTICE] [66.249.66.12:44591-0#community.lanfusion.com] Content len: 0, Request line:
GET /topic/daily/Lookin%20for%20some%20Software?date=Feb282008 HTTP/1.1
2008-03-05 16:05:26.819 [NOTICE] [66.249.66.12:44591-0#community.lanfusion.com] Redirect: #2, URL: /dispatch.lsapi

Excerpt from stderr.log (these three lines are added each time I restart the server)

[FATAL] failed to allocate memory
[FATAL] failed to allocate memory
[FATAL] failed to allocate memory

Any help would be greatly appreciated.

Dan Matthews - LANFusion LLC - www.lanfusion.com

 

[mistwang]

set "Memory Soft/hard Limit" to "500M" or above.

 

[lanfusion]

Updated... still having the problem.

2008-03-05 16:44:37.897 [INFO] [Rails:community.lanfusion.com:/] add child process pid: 9862
2008-03-05 16:44:37.897 [INFO] [Rails:community.lanfusion.com:/] pid list size: 1
2008-03-05 16:46:14.735 [INFO] [66.249.66.12:58218-0#community.lanfusion.com] connection to [uds://tmp/lshttpd/community.lanfusion.com:_.sock] on request #0, error: Connection reset by peer!
2008-03-05 16:46:26.419 [INFO] [66.249.66.12:58218-0#community.lanfusion.com] connection to [uds://tmp/lshttpd/community.lanfusion.com:_.sock] on request #0, error: Connection reset by peer!
2008-03-05 16:46:39.416 [INFO] [66.249.66.12:58218-0#community.lanfusion.com] connection to [uds://tmp/lshttpd/community.lanfusion.com:_.sock] on request #0, error: Connection reset by peer!
2008-03-05 16:46:39.416 [NOTICE] [66.249.66.12:58218-0#community.lanfusion.com] oops! 503 Service Unavailable
2008-03-05 16:46:39.416 [NOTICE] [66.249.66.12:58218-0#community.lanfusion.com] Content len: 0, Request line:
GET /topic/classic/Posting?page=1 HTTP/1.1
2008-03-05 16:46:39.416 [NOTICE] [66.249.66.12:58218-0#community.lanfusion.com] Redirect: #2, URL: /dispatch.lsapi

 

[mistwang]

Please set the process soft/hard limit as well, maybe to ">200".

 

[lanfusion]

Now the application just hangs. After around 5 minutes I get a "500 Internal Server Error" page.

 

[mistwang]

Please check the production.log and lsws/logs/stderr.log.
If the ruby process is running, do

"strace -f -p <pid_of_ruby>"

 

[lanfusion]

nothing in the stderr.log file

this is the production log:

# Logfile created on Wed Mar 05 17:30:44 -0500 2008

DRb:RbConnError (druby://localhost:9010 - #<Errno::ECONNREFUSED: Connection refused - connect(2)>):
/usr/lib/ruby/1.8/drb/drb.rb:736:in `open'
/usr/lib/ruby/1.8/drb/drb.rb:729:in `each'
/usr/lib/ruby/1.8/drb/drb.rb:729:in `open'
/usr/lib/ruby/1.8/drb/drb.rb:1189:in `initialize'
/usr/lib/ruby/1.8/drb/drb.rb:1169:in `new'
/usr/lib/ruby/1.8/drb/drb.rb:1169:in `open'
/usr/lib/ruby/1.8/drb/drb.rb:1085:in `method_missing'
/usr/lib/ruby/1.8/drb/drb.rb:1103:in `with_friend'
/usr/lib/ruby/1.8/drb/drb.rb:1084:in `method_missing'
/usr/lib64/ruby/gems/1.8/gems/acts_as_ferret-0.4.3/lib/remote_index.rb:16:in `send'
/usr/lib64/ruby/gems/1.8/gems/acts_as_ferret-0.4.3/lib/remote_index.rb:16:in `method_missing'
/usr/lib64/ruby/gems/1.8/gems/acts_as_ferret-0.4.3/lib/act_methods.rb:189:in `acts_as_ferret'
/app/models/user.rb:31
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require'
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in `require'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:342:in `new_constants_in'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in `require'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:104:in `require_or_load'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:248:in `load_missing_constant'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:453:in `const_missing'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:465:in `const_missing'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:260:in `load_missing_constant'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:469:in `const_missing'
/app/controllers/application.rb:37:in `authorize'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:469:in `send!'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:469:in `call'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:441:in `run'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:716:in `run_before_filters'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:695:in `call_filters'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:689:in `perform_action_without_benchmark'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/rescue.rb:199:in `perform_action_without_caching'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:678:in `perform_action'
/usr/lib64/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/query_cache.rb:33:in `cache'
/usr/lib64/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/query_cache.rb:8:in `cache'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:677:in `perform_action'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in `send'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in `process_without_filters'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:685:in `process_without_session_management_support'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/session_management.rb:123:in `process'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:388:in `process'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:171:in `handle_request'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:115:in `dispatch'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:126:in `dispatch_cgi'
/usr/lib64/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:9:in `dispatch'
/usr/local/lsws/fcgi-bin/RailsRunner.rb:24

LoadError (Expected /var/www/domains/community.lanfusion.com/app/models/user.rb to define User):
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:249:in `load_missing_constant'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:453:in `const_missing'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:465:in `const_missing'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:260:in `load_missing_constant'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:469:in `const_missing'
/app/controllers/application.rb:37:in `authorize'


that's all unique... then this block of log repeats infinitely:

LoadError (Expected /var/www/domains/community.lanfusion.com/app/models/user.rb to define User):
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:249:in `load_missing_constant'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:453:in `const_missing'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:465:in `const_missing'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:260:in `load_missing_constant'
/usr/lib64/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:469:in `const_missing'
/app/controllers/application.rb:37:in `authorize'

 

[mistwang]

That's ruby on Rails errors. Your guys are RoR developer, should be able to fix them one by one.

 

[lanfusion]

Except the app works fine in webbrick and lighttpd. Only with LiteSpeed does it give me these errors.

Here it is running on the Webbrick server: http://secure1.lanfusion.net:3000/

 

[mistwang]

The difference between litespeed and others are the stripped environment variables and one ruby process fork children processes to handle all the requests.
The children process will inherit all file handles from parent. I don't know exactly what is DRb, but looks like it need to connect to a local TCP port for some services, it could be a problem when children inherit the file handle for that connection. So, you need to find a way to close the DRb connection established during initializing Rails framework before parent fork the child process.

Like what we did for the ActiveRecord connections in RailsRunner.rb

#Close all DB connections established during initialization
ActiveRecord::Base.clear_active_connections! if defined?(ActiveRecord::Base)

You need to figure out how to do that to the DRb connection.

 

[lanfusion]

Thanks for the help. I'll have my lead developer look into it.

 

[mistwang]

Please keep us posted if you find a solution.