Standard 2.2.5 + LSAPI + Rails uses up all processes

[dirkbj]

I saw this bug on earlier versions - 2.2.3 with an LSAPI Ruby module earlier to 1.11. I am still seeing this bug running 2.2.5 with the latest LSAPI Ruby module (1.11)

Here are the details:

1) OS type and Version:

Mac OS X Server 10.4.8

2) LiteSpeed Version:

2.2.5 Standard + LSAPI Ruby Module 1.11

3) If PHP related:

Not PHP related

4) All possible related info that you observed before and after, the bug occurred:

The site handles a Rails application. It runs great for days at a time, but then, for no apparant reason, it spawns a bunch of ruby threads and takes up all system resources. Our fix is to restart the server. We see no general pattern - it seems to occur when the site is busy or when it is sitting idle.

5) If you can reproduce the bug, please detail what the steps are:

Just let it run.

6) Note any errors logged in error.log and stderr.log of LiteSpeed logs directory:

No errors were logged.

7) Check /tmp/lshttpd for "core dumps"/core files:

No core dumps

[mistwang]

Maybe the ruby processes gets into a state that cannot be killed by normal SIGTERM signal.

Can you "truss" or "ktrace" those ruby processes to check the state of those hanging processes.

It would be best if we can logon to your server to check though ruby processes when it happenes.

[dirkbj]

If this were true, I believe we would see ruby processes building over time. This is not the case. The ruby process count stays generally constant over a period of days and then all-of-a-sudden grows out of control.

I will watch and try to get a trace the next time it happens.

Login access to our server, unfortunately, is not an option.

Dirk

[mistwang]

Also, please make sure the latest ruby-lsapi has been used. If you have installed it manually, then switch to gem installation, you need to remove the file installed manually.

[dirkbj]

Yes, I am and have been running the latest - installed with ruby gems.

Dirk

[admin]

What is your rails settings within litespeed? If you set instances/children value too high and a search crawler/bot comes to hammer your site on a periodic basis, it can send your server resource skyrocketing if the backend cannot support the number of simultaneous script connections.

Since we do not know the exact rail script that's causing this or detailed parameters of the incident, it would be extremely difficult to isolate/replicate on our end right now. Hopefully we can get more answers once we have some trace data if and when this happens again.

[fantasydreaming]

My similar problem to this was caused by a search using the ruby-google module... apparently http-access and http-access2 can both get stuck waiting 'forever' for a google (or anywhere, likely) response.

The solution for me was to wrap it in a Timeout::timeout(5) do() block.

[mistwang]

fantasydreaming,

Thank you for sharing your finding with us. It is really helpful.

We may improve the ruby-lsapi implementation to somehow kill those hanging processes when we got a chance.