Getting Lots of Ruby Processes
- Thread starter jp_n9
- Start date
I uninstalled the gem, just to make sure.
I have the latest version of LiteSpeed installed, and LSAPI 1.8. It has been about 7 hours since I restarted the server - and I now have 26 processes running.
Max connections is set to 10. Is there any other configuration variable somewhere that could be set incorrectly and leading to this???
Here is what is in the log:
2006-09-15 00:35:01.639 INFO [127.0.0.1:52875-0#Example] File not found [/usr/bin/DEFAULT/html/whm-server-status]
2006-09-15 00:40:02.608 INFO [127.0.0.1:53698-0#Example] File not found [/usr/bin/DEFAULT/html/whm-server-status]
2006-09-15 00:41:28.491 INFO [65.111.164.187:54922-0#VHOSTNAMEREMOVED] Index file is not available in [/home/xxxxxxx/releases/20060911024632/public/]
2006-09-15 00:45:02.298 INFO [127.0.0.1:54471-0#Example] File not found [/usr/bin/DEFAULT/html/whm-server-status]
2006-09-15 00:46:31.095 NOTICE [Rails:VHOSTNAMEREMOVED:/] stop worker processes
I have the latest version of LiteSpeed installed, and LSAPI 1.8. It has been about 7 hours since I restarted the server - and I now have 26 processes running.
Max connections is set to 10. Is there any other configuration variable somewhere that could be set incorrectly and leading to this???
Here is what is in the log:
2006-09-15 00:35:01.639 INFO [127.0.0.1:52875-0#Example] File not found [/usr/bin/DEFAULT/html/whm-server-status]
2006-09-15 00:40:02.608 INFO [127.0.0.1:53698-0#Example] File not found [/usr/bin/DEFAULT/html/whm-server-status]
2006-09-15 00:41:28.491 INFO [65.111.164.187:54922-0#VHOSTNAMEREMOVED] Index file is not available in [/home/xxxxxxx/releases/20060911024632/public/]
2006-09-15 00:45:02.298 INFO [127.0.0.1:54471-0#Example] File not found [/usr/bin/DEFAULT/html/whm-server-status]
2006-09-15 00:46:31.095 NOTICE [Rails:VHOSTNAMEREMOVED:/] stop worker processes
Last edited:
I think you are within the limit based on the Max Connections setting. It was kind of confusing for me too, but this is what I am doing now.
NOTE: EACH virtual host you set has to have its own Max Conn. setting. It is not global as far as I can tell, so if you have 3 vhosts at 10 each, you're looking at maybe 10*3*2 at the very least if the load increases , that is a LOT of ram because RoR is a frikkin Hog when it comes to real RAM usage. So, for 3 vhosts at 10 max each and a "reasonable" default behaviour of going to "twice the configured value" as described by litespeedtech, you are looking at 10 * 3 * 2 * 20mb = 1.2 GB or RES mem usage, obviously, if you have less than that available, your system will start hitting the swap, it will delay the request serving by the RoR, and that would feed the vicious cycle. pretty soon, you could start thrashing.
Unfortunately the documentation is not clear on how many processes each lsapi process can handle. If this was clear, then people won't set it to 10.. 10 *20 *3 is more than half gig of ACTUAL ram, and most people don't have that even on a VPS system.
Suggestion (to litespeedtech): Put a tool tip next to the Max Connections entry which says, "LSWS may start upto 3 times as many rails processes (at 20mb each) as this setting"
This will be a clear warning to us all that this setting can be resource intensive.
- Figure out how many processes you can afford on your vps or shared account
- Divide it by 3, and set your MAX Connections to that number.
NOTE: EACH virtual host you set has to have its own Max Conn. setting. It is not global as far as I can tell, so if you have 3 vhosts at 10 each, you're looking at maybe 10*3*2 at the very least if the load increases , that is a LOT of ram because RoR is a frikkin Hog when it comes to real RAM usage. So, for 3 vhosts at 10 max each and a "reasonable" default behaviour of going to "twice the configured value" as described by litespeedtech, you are looking at 10 * 3 * 2 * 20mb = 1.2 GB or RES mem usage, obviously, if you have less than that available, your system will start hitting the swap, it will delay the request serving by the RoR, and that would feed the vicious cycle. pretty soon, you could start thrashing.
Unfortunately the documentation is not clear on how many processes each lsapi process can handle. If this was clear, then people won't set it to 10.. 10 *20 *3 is more than half gig of ACTUAL ram, and most people don't have that even on a VPS system.
Suggestion (to litespeedtech): Put a tool tip next to the Max Connections entry which says, "LSWS may start upto 3 times as many rails processes (at 20mb each) as this setting"
This will be a clear warning to us all that this setting can be resource intensive.
Last edited:
Well if what you are saying is correct, then I'm not sure adding a tool tip is the solution.
If the "Max Connections" value is supposed to control the maximum processes created, then it should not create any more processes than the value you enter there. Maximum != Maximum * 3. It just doesn't make sense. Of course, the wording itself is already confusing, does "Connection" == "Ruby Process"?
Also, I'm not sure if that "Max Connection" value is supposed to be for the server or per Rails Application. Being that it is in the Server Tuning section - it would seem as though it should be a Server wide setting. Regardless how many Rails apps are installed, no more than the Maximum value should be started.
If the "Max Connections" value is supposed to control the maximum processes created, then it should not create any more processes than the value you enter there. Maximum != Maximum * 3. It just doesn't make sense. Of course, the wording itself is already confusing, does "Connection" == "Ruby Process"?
Also, I'm not sure if that "Max Connection" value is supposed to be for the server or per Rails Application. Being that it is in the Server Tuning section - it would seem as though it should be a Server wide setting. Regardless how many Rails apps are installed, no more than the Maximum value should be started.
Sorry for the confusion, for each connection to backend Rails application, there must be a Ruby process which is responsible for handling the request.
"Max connection" under the Rails tab is the default value for each Rails application configured, it is per application, not the maximum regardless number of rails apps. The web server does not control number of ruby processes directly, they are managed by the process manager in the first ruby process started by the server. It is not possible to coordinate number of ruby processes across multiple Rails applications.
So, if you have multiple Rails applications on your server, what you see is normal. If you have only one app, something might be wrong, and I'd like to take a look at those ruby processes running on your server.
The process manager in the first ruby process is allowed to start more children processes than the "max connections" configured, max to 2 * "max connections", when some children ruby processes stop responding to requests, and LSWS is allowed to start another group of ruby process for the same application, if LSWS think the whole group of ruby process is not working.
Check the "Initial Request Timeout" under rails tab, it should be large enough, default is "60", increase it if any request of your rails application takes longer to process than that.
Hope this will help.
"Max connection" under the Rails tab is the default value for each Rails application configured, it is per application, not the maximum regardless number of rails apps. The web server does not control number of ruby processes directly, they are managed by the process manager in the first ruby process started by the server. It is not possible to coordinate number of ruby processes across multiple Rails applications.
So, if you have multiple Rails applications on your server, what you see is normal. If you have only one app, something might be wrong, and I'd like to take a look at those ruby processes running on your server.
The process manager in the first ruby process is allowed to start more children processes than the "max connections" configured, max to 2 * "max connections", when some children ruby processes stop responding to requests, and LSWS is allowed to start another group of ruby process for the same application, if LSWS think the whole group of ruby process is not working.
Check the "Initial Request Timeout" under rails tab, it should be large enough, default is "60", increase it if any request of your rails application takes longer to process than that.
Hope this will help.
I reset the "Max Connections" under the "Rails" tab to 3. I restarted, and a few hours later I now have 12 ruby processes. This is kind of crazy.
Under the "Security" tab there is a setting "Max CGI Instances" which is left at the original 20. Could that or anything else be causing this issue?
Any other ideas?
Under the "Security" tab there is a setting "Max CGI Instances" which is left at the original 20. Could that or anything else be causing this issue?
Any other ideas?
OK, caught the bug on jp_n9's server, it only happens when ruby processes are started in suEXEC mode, user id is different from that of web server, and "max_idle_time" under "Rails" tab has been set.
When there is no request to ruby process for "max_idle_time", LSWS will try to kill this process group, but failed due to permission problem, but LSWS think the process group has been killed, when next request comes in, LSWS will start another group of ruby process, and so on.
I have updated ruby-lsapi-1.8 package to address this issue, please update your installation then and add an environment variable "LSAPI_PGRP_MAX_IDLE=xxx" under "Rails" tab to tell the ruby process to quit, the value should match "max idle time" value under rails tab.
We will fix this problem on LSWS side as well in 2.2.2, by finding a workaround for the permission problem of killing suEXEC processes, as well as setting "LSAPI_PGRP_MAX_IDLE" automatically.
When there is no request to ruby process for "max_idle_time", LSWS will try to kill this process group, but failed due to permission problem, but LSWS think the process group has been killed, when next request comes in, LSWS will start another group of ruby process, and so on.
I have updated ruby-lsapi-1.8 package to address this issue, please update your installation then and add an environment variable "LSAPI_PGRP_MAX_IDLE=xxx" under "Rails" tab to tell the ruby process to quit, the value should match "max idle time" value under rails tab.
We will fix this problem on LSWS side as well in 2.2.2, by finding a workaround for the permission problem of killing suEXEC processes, as well as setting "LSAPI_PGRP_MAX_IDLE" automatically.