Recommendations for reactive monitoring

[nathanc]

Hey guys,
Just see if there is a recommendation for reactive monitoring. Ive used monit in the past for monitoring Mongrel webservers. Is there a preferred programs that can make sure that litespeed is always happy and healthy?

[mistwang]

You can use our caccti integration, tutorial is in our wiki.

[nathanc]

ah yes cacti. Ive been using it for a week now to graph my litespeed servers. but cacti ,correct me if im wrong, does not react, if a litespeed server goes down. Cacti is simply there to graph. Can cacti react if a litespeed server goes down?

[mistwang]

LSWS has built-in watch dog process, unless there are hardware problem or mis-configuration, LSWS will never go down.

[nathanc]

run "/opt/lsws/bin/lswsctrl reload" five times fast.
Thats crashes my litespeed servers everytime. They do not comeback on their own. I have to manually issue "/opt/lsws/bin/lswsctrl start".

[mistwang]

How often do you need to restart LSWS five times in a row within 5 seconds period? Why do you need to do that?
It could be a problem if all rails applications are set to "run on startup", so you got 20 rails frameworks to be initialized at start up.

We probably will add a feature to let user to restart each individual application instead of starting the whole server.
We could do something to make it run smoother.

[nathanc]

No I dont need to restart lsws 5 times in a row. It was simply a response to "LSWS will never go down."

I will probably use monit to make sure that the web server can recover from things like the above.

Originally Posted by mistwang

We probably will add a feature to let user to restart each individual application instead of starting the whole server.
We could do something to make it run smoother.

Thats is great! We need that feature. Right now, anytime a user changes their apps, we have to pragmatically reload each of the webservers so that the users can have their rails app restarted. So far there is a mild performance impact on all of the rails apps.

[mistwang]

No I dont need to restart lsws 5 times in a row. It was simply a response to "LSWS will never go down."

There are definitely a few other ways to break things intentionally.

[PSS]

I check if memcached is up with this small script I made (I'm not daily bash coder, feel free to improve), I run it every minute with cron. If you do the same for LS main process you can have reasonably good kickstarter in case all LS processes die.

#!/bin/sh
pid=`ps --user=nobody | grep memcached | awk '{print $1}'` 

if kill -0 $pid 2> /dev/null ; then
    exit 0
else
    echo memcached process not found, restarting...
    /usr/local/bin/memcached -u nobody -d -m 128 -l 127.0.0.3 -p 11211
    sleep 2
    pid=`ps --user=nobody | grep memcached | awk '{print $1}'`
    if kill -0 $pid 2> /dev/null ; then
      echo process $pid ok
    fi
fi
exit 0

[nathanc]

Originally Posted by PSS

I check if memcached is up with this small script I made (I'm not daily bash coder, feel free to improve), I run it every minute with cron. If you do the same for LS main process you can have reasonably good kickstarter in case all LS processes die.

#!/bin/sh
pid=`ps --user=nobody | grep memcached | awk '{print $1}'` 

if kill -0 $pid 2> /dev/null ; then
    exit 0
else
    echo memcached process not found, restarting...
    /usr/local/bin/memcached -u nobody -d -m 128 -l 127.0.0.3 -p 11211
    sleep 2
    pid=`ps --user=nobody | grep memcached | awk '{print $1}'`
    if kill -0 $pid 2> /dev/null ; then
      echo process $pid ok
    fi
fi
exit 0

Awesome scripting foo!