Ruby Rails tuning - traffic bursts

[gwhite]

Litespeed trial Enterprise version 3.0 - 64 bit
ruby-lsapi 2.2 - gem install
Easy rails setup
single opteron server
4GB RAM
Linux 2.6 kernel

I have a system that is under fairly high load but will occasionally receive intense traffic bursts. It receives a combination of keep-alive and non keep-alive http requests. When a burst happens a large number of RailsRunner.rb processes are spawned to the point where they use so many resources that the server is essentially locked up. I have made a number of parameter changes to try and help the situation but I'd like some verification that I am moving in the right direction. Process limits and Memory limits have been increased. During regular traffic levels there is ~100 connections and ~60 rqps to litespeed in this system. All dynamic traffic, no static content. It may turn out that I simply need more hardware to deal with the load or to deal with the spikes somehow (not completely under my control). Per client throttling is not an option in this setup based on the nature of the traffic. My settings below:

Server Tuning section:
Connection Timeout - 30s
Keep-Alive Timeout - 10s
I/O Event Dispatcher - epoll

Server General:
Memory I/O Buffer - 16M

Ruby/Rails:
Max Connections 300
Environment
LSAPI_CHILDREN=300
LSAPI_MAX_REQUESTS=5000
LSAPI_MAX_IDLE=60
LSAPI_MAX_PROCESS_TIME=60
Initial Request Timeout (secs) 60
Retry Timeout (secs) 0
Connection Keepalive Timeout 60
Response Bufferring No
Back Log 50
Run On Start Up Yes
Max Idle Time 120
Priority 3
Memory Soft Limit (bytes) 1800M
Memory Hard Limit (bytes) 2000M
Process Soft Limit 380
Process Hard Limit 400

Rails Context:
Max Connections 300

[mistwang]

You do not need to set LSAPI_CHILDREN explicitly, it will be set automatically to match "Max Connections".

300 children process looks pretty high to me, set it too high may hurt the performance sometimes, and the optimal value is depends on your application, I suggest you to start from 50 and increase it little by little to find the sweet spot.

[gwhite]

So to confirm - I should remove the explicit LSAPI_CHILDREN setting, set Max Connections to 50 in the Server Ruby/Rails tab and gradually increase? Do I need to set Max Connections in the Virtual Host Rails Context to 50 as well initially? Which Max Connections setting takes precedence? Thanks

[mistwang]

Yes, remove LSAPI_CHILDREN. "Max Connections" in the virtual host rails context overrides the server level setting.

[fantasydreaming]

Even 50 connections is pretty huge. Most large sizes shouldn't use more than 10. I use 4 (which means it can raise to 6 really) for about 500k hits/day.

If your SQl queries are really slow, you'll get backlogs pretty fast, so be careful in that category. Having more children just upps your thrashing though...

[gwhite]

Yes agreed! I've taken it down to 12 and testing has improved by leaps and bounds. The application is extremely cpu intensive so I will move on to a dual processor system and work with the programmers on optimizing. Thanks

[mistwang]

That's correct.
If a application is CPU bound, you have to use more powerful CPU, if it is disk bound, add more HDs and/or use faster HD, if it is network bound upgrade the network connection.
You should always attack the real bottle neck, adding more worker processes may help to certain extent, but will not fix the problem completely.