killed with SIGTERM, 3-4 times a day

Discussion in 'Ruby/Rails' started by fantasydreaming, Feb 9, 2008.

  1. fantasydreaming

    fantasydreaming New Member

    Lsws seems to be sending a sigterm to my processes during processing occassionally.

    I'm using 3.3.4. I believe the latest lsapi as well.

    My backtraces show it happens sometimes during a template render, sometimes during a memcached call, and sometimes during a database requests. None of these things should take very long, so it just seems to be random.

    Load has been very low, under 0.4.

    Any ideas for debugging? I haven't gotten any complaints from people (besides the occassional "it logged me out" complaint, which I'm still trying to figure out, perhaps some cookie problem).

    Thanks for a great product!
  2. mistwang

    mistwang LiteSpeed Staff

  3. fantasydreaming

    fantasydreaming New Member

    Yes, but the backtrace and logged sigterm exceptions are showing that this is happening while the request is in procesing, and that it was a very fast request (i.e. it hasn't hung, from what I can tell). So it shouldn't be considered 'idle' since it was just given a new request 10ms previously...
  4. mistwang

    mistwang LiteSpeed Staff

    OK, we will investigate. Maybe the watchdog ruby process think the child process is idle while it is not.
  5. mistwang

    mistwang LiteSpeed Staff

    Please try adding the following environment variables under ruby tab

    LSAPI_MAX_PROCESS_TIME=10000000
    LSAPI_AVOID_FORK=1
    LSAPI_MAX_IDLE=1000000

    If those variable helps, try removing them one by one and find out which one affect this problem. Which this hint, we can locate the problem and fix it accordingly.
  6. iwarshak

    iwarshak New Member

    I am having the exact same issue. Processes getting SIGTERMed on a very fast page. Did you have any luck with this?
  7. mistwang

    mistwang LiteSpeed Staff

    Please try above environment variables, see if it helps.
    Also, please check the PID of "ruby" processes, see if the killed process is a child process. On a busy server, there should be a parent ruby process keep running.
    At least, we can narrow down which process send the SIGTERM signal, lshttpd, or the parent ruby process.
  8. iwarshak

    iwarshak New Member

    I will try that.

    I also see quite a few of these:

    [idle] connection to [uds://tmp/lshttpd/mydomain.com:_.sock] on request #81, error: Connection reset by peer!

    Could this be related to the above problem?
  9. pbengtson

    pbengtson New Member

    Specifying all three options:

    LSAPI_MAX_PROCESS_TIME=10000000
    LSAPI_AVOID_FORK=1
    LSAPI_MAX_IDLE=1000000

    makes our server crash, hard, after a while (server starts swapping then goes firmly into a downward spiral). Removing LSAPI_AVOID_FORK=1, keeping only the LSAPI_MAX_PROCESS_TIME and LSAPI_MAX_IDLE alleviates the problem for us - only a tenth of the requests result in SIGTERMs - but it doesn't remove it. Which is unacceptable; management is already talking about switching to another webserver. And we have a purchased license.

    Please advise.
  10. mistwang

    mistwang LiteSpeed Staff

    We are working on a new ruby lsapi release to address this.
  11. mistwang

    mistwang LiteSpeed Staff

    Everyone,

    Please upgrade ruby-lsapi 3.1, see if the problem is fixed or not.
  12. pbengtson

    pbengtson New Member

    Your 3.1 gem distro is b0rkinated (it is empty). Please fix ASAP.
  13. pbengtson

    pbengtson New Member

    Thanks for fixing the gem so quickly. I will post again when we have verified that it removes the SIGTERM problem.
  14. mistwang

    mistwang LiteSpeed Staff

    It takes a while for rubyforge to update all its mirrors. :)
  15. pbengtson

    pbengtson New Member

    The new ruby-lsapi 3.1 gem fixes the SIGTERM problem, at least for us (with servers under a heavy load).

    (The gem was there on rubyforge, but it was empty. Zero bytes file length.)
  16. mistwang

    mistwang LiteSpeed Staff

    Thanks for the confirmation. :)
  17. fantasydreaming

    fantasydreaming New Member

    Working great for me as well, thanks! :)
  18. mistwang

    mistwang LiteSpeed Staff

    Thanks for the update.

Share This Page