Ruby Rails tuning - traffic bursts

Discussion in 'Ruby/Rails' started by gwhite, Mar 21, 2007.

  1. gwhite

    gwhite New Member

    Litespeed trial Enterprise version 3.0 - 64 bit
    ruby-lsapi 2.2 - gem install
    Easy rails setup
    single opteron server
    4GB RAM
    Linux 2.6 kernel

    I have a system that is under fairly high load but will occasionally receive intense traffic bursts. It receives a combination of keep-alive and non keep-alive http requests. When a burst happens a large number of RailsRunner.rb processes are spawned to the point where they use so many resources that the server is essentially locked up. I have made a number of parameter changes to try and help the situation but I'd like some verification that I am moving in the right direction. Process limits and Memory limits have been increased. During regular traffic levels there is ~100 connections and ~60 rqps to litespeed in this system. All dynamic traffic, no static content. It may turn out that I simply need more hardware to deal with the load or to deal with the spikes somehow (not completely under my control). Per client throttling is not an option in this setup based on the nature of the traffic. My settings below:

    Server Tuning section:
    Connection Timeout - 30s
    Keep-Alive Timeout - 10s
    I/O Event Dispatcher - epoll

    Server General:
    Memory I/O Buffer - 16M

    Ruby/Rails:
    Max Connections 300
    Environment
    LSAPI_CHILDREN=300
    LSAPI_MAX_REQUESTS=5000
    LSAPI_MAX_IDLE=60
    LSAPI_MAX_PROCESS_TIME=60
    Initial Request Timeout (secs) 60
    Retry Timeout (secs) 0
    Connection Keepalive Timeout 60
    Response Bufferring No
    Back Log 50
    Run On Start Up Yes
    Max Idle Time 120
    Priority 3
    Memory Soft Limit (bytes) 1800M
    Memory Hard Limit (bytes) 2000M
    Process Soft Limit 380
    Process Hard Limit 400

    Rails Context:
    Max Connections 300
  2. mistwang

    mistwang LiteSpeed Staff

    You do not need to set LSAPI_CHILDREN explicitly, it will be set automatically to match "Max Connections".

    300 children process looks pretty high to me, set it too high may hurt the performance sometimes, and the optimal value is depends on your application, I suggest you to start from 50 and increase it little by little to find the sweet spot. :)
  3. gwhite

    gwhite New Member

    So to confirm - I should remove the explicit LSAPI_CHILDREN setting, set Max Connections to 50 in the Server Ruby/Rails tab and gradually increase? Do I need to set Max Connections in the Virtual Host Rails Context to 50 as well initially? Which Max Connections setting takes precedence? Thanks
  4. mistwang

    mistwang LiteSpeed Staff

    Yes, remove LSAPI_CHILDREN. "Max Connections" in the virtual host rails context overrides the server level setting.
  5. fantasydreaming

    fantasydreaming New Member

    Even 50 connections is pretty huge. Most large sizes shouldn't use more than 10. I use 4 (which means it can raise to 6 really) for about 500k hits/day.

    If your SQl queries are really slow, you'll get backlogs pretty fast, so be careful in that category. Having more children just upps your thrashing though...
  6. gwhite

    gwhite New Member

    Yes agreed! I've taken it down to 12 and testing has improved by leaps and bounds. The application is extremely cpu intensive so I will move on to a dual processor system and work with the programmers on optimizing. Thanks
  7. mistwang

    mistwang LiteSpeed Staff

    That's correct.
    If a application is CPU bound, you have to use more powerful CPU, if it is disk bound, add more HDs and/or use faster HD, if it is network bound upgrade the network connection.
    You should always attack the real bottle neck, adding more worker processes may help to certain extent, but will not fix the problem completely.

Share This Page