Litespeed restarting

Discussion in 'Bug Reports' started by nowiresltd, May 20, 2010.

  1. nowiresltd

    nowiresltd New Member

    Hi

    Cant seem to track down the reason why, but litespeed seems to be shutting down and taking approx 2 mins to come back up ever 3-5 hours (as reported by panopta), I have switched back to apache and its been 24 hours since any outage reported by panopta.

    I initially had an issue where a site in the httpd.conf had an ip not bound to the server, so i removed that but it seems to have not cured the issue.

    Where do i look to see why its restarting ?

    I really want to use litespeed, but its only a matter of time before someone notices these little outages, not to mention ruining my uptime stats :)

    Any guidance or advice will be much appreciated

    Also have it running on two other servers without issue, the only thing about this one is the SSI fix given in the forum a few posts down.
  2. mistwang

    mistwang LiteSpeed Staff

    Are you using cPanel?
    Have you checked error log for the period of outage?
  3. nowiresltd

    nowiresltd New Member

    Hi

    I have found an example log extract which i dont want to post on the forum but will send you in a PM of a the full log if you want.

    Its a mystery as to what its doing between 08:46 and 08:55, this has happend a few times, but only since I took the SSI fix. (i think)


    2010-05-21 08:46:45.632 [NOTICE] Server Restart Request via Signal...
    2010-05-21 08:46:45.632 [NOTICE] [6131] Graceful Restart...
    2010-05-21 08:46:45.632 [NOTICE] [Child: 9224] Start shutting down gracefully ...
    2010-05-21 08:46:45.637 [NOTICE] [PID: 6131]: start cgid: 21582, /usr/local/lsws/bin/lscgid


    2010-05-21 08:46:45.637 [INFO] Pass listener *:7080.
    2010-05-21 08:46:45.640 [NOTICE] [6131] Start new instance of lshttpd: 21583

    2010-05-21 08:46:45.641 [INFO] Stop listener *:7080.
    2010-05-21 08:46:46.400 [NOTICE] [Child: 9223] Start shutting down gracefully ...


    2010-05-21 08:46:47.608 [NOTICE] Your License key will expire in 11 days
    2010-05-21 08:46:47.609 [NOTICE] [21583] Loading LiteSpeed/4.0.14 Enterprise ...
    2010-05-21 08:46:47.609 [NOTICE] [ADMIN] server socket: uds://usr/local/lsws/admin/tmp/admin.sock.7848
    2010-05-21 08:46:47.609 [NOTICE] Loading configuration from /usr/local/lsws/conf/httpd_config.xml ...


    2010-05-21 08:46:47.611 [NOTICE] Recovering server socket: [*:7080]
    2010-05-21 08:46:47.612 [NOTICE] [config] chroot is disabled.
    2010-05-21 08:46:47.612 [INFO] old priority: -19, new priority: -19
    2010-05-21 08:46:47.612 [INFO] [config:server:basic] For better obscurity, server version number is hidden in the response header.
    2010-05-21 08:46:48.006 [NOTICE] [PID: 21618]: start cgid: 21635, /usr/local/lsws/bin/httpd
    2010-05-21 08:46:48.076 [WARN] [config:server:listener] No listener is available for normal virtual host!
    2010-05-21 08:46:48.076 [NOTICE] The maximum number of file descriptor limit is set to 16384.
    2010-05-21 08:46:48.264 [NOTICE] [config:server:epsr:lsphp4]'Process Limit' probably is too low, adjust the limit to: 310.
    2010-05-21 08:46:50.249 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_global.conf
    2010-05-21 08:46:50.557 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_2.conf
    2010-05-21 08:46:50.557 [INFO] Processing config file: /usr/local/apache/conf/php.conf
    2010-05-21 08:46:50.575 [INFO] Processing config file: /usr/local/apache/conf/includes/errordocument.conf
    2010-05-21 08:46:50.702 [INFO] Processing config file: /usr/local/apache/conf/modsec2.conf
    2010-05-21 08:46:50.717 [INFO] Processing config file: /usr/local/apache/conf/modsec2.user.conf
    2010-05-21 08:46:50.939 [INFO]
    2010-05-21 08:46:51.121 [INFO] Processing config file: /usr/local/apache/conf/modsec2.whitelist.conf
    2010-05-21 08:46:51.226 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
    2010-05-21 08:46:51.226 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
    2010-05-21 08:46:51.226 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
    2010-05-21 08:46:51.226 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
    2010-05-21 08:46:51.286 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
    2010-05-21 08:46:51.286 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
    2010-05-21 08:46:51.286 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
    2010-05-21 08:46:51.286 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
    2010-05-21 08:46:54.408 [NOTICE] Your License key will expire in 11 days
    2010-05-21 08:46:54.408 [NOTICE] [21734] Loading LiteSpeed/4.0.14 Enterprise ...
    2010-05-21 08:46:54.409 [NOTICE] [ADMIN] server socket: uds://usr/local/lsws/admin/tmp/admin.sock.7354
    2010-05-21 08:46:54.409 [NOTICE] Loading configuration from /usr/local/lsws/conf/httpd_config.xml ...
    2010-05-21 08:46:54.410 [NOTICE] [config] chroot is disabled.
    2010-05-21 08:46:54.410 [INFO] old priority: 18, new priority: -19
    2010-05-21 08:46:54.410 [INFO] [config:server:basic] For better obscurity, server version number is hidden in the response header.
    2010-05-21 08:46:54.411 [NOTICE] [PID: 21752]: start cgid: 21755, /usr/local/lsws/bin/httpd
    2010-05-21 08:46:54.411 [WARN] [config:server:listener] No listener is available for normal virtual host!
    2010-05-21 08:46:54.412 [NOTICE] The maximum number of file descriptor limit is set to 16384.
    2010-05-21 08:46:54.412 [NOTICE] [config:server:epsr:lsphp4]'Process Limit' probably is too low, adjust the limit to: 310.
    2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_global.conf
    2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_2.conf
    2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/php.conf
    2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/includes/errordocument.conf
    2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/modsec2.conf
    2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/modsec2.user.conf
    2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/modsec2.whitelist.conf
    2010-05-21 08:46:55.685 [NOTICE] Your License key will expire in 11 days
    2010-05-21 08:46:55.686 [NOTICE] [21770] Loading LiteSpeed/4.0.14 Enterprise ...
    2010-05-21 08:46:55.686 [NOTICE] [ADMIN] server socket: uds://usr/local/lsws/admin/tmp/admin.sock.7089
    2010-05-21 08:46:55.686 [NOTICE] Loading configuration from /usr/local/lsws/conf/httpd_config.xml ...
    2010-05-21 08:46:55.687 [NOTICE] [config] chroot is disabled.
    2010-05-21 08:46:55.687 [INFO] old priority: 18, new priority: -19
    2010-05-21 08:46:55.687 [INFO] [config:server:basic] For better obscurity, server version number is hidden in the response header.
    2010-05-21 08:46:55.689 [NOTICE] [PID: 21788]: start cgid: 21791, /usr/local/lsws/bin/httpd
    2010-05-21 08:46:55.689 [WARN] [config:server:listener] No listener is available for normal virtual host!
    2010-05-21 08:46:55.689 [NOTICE] The maximum number of file descriptor limit is set to 16384.
    2010-05-21 08:46:55.689 [NOTICE] [config:server:epsr:lsphp4]'Process Limit' probably is too low, adjust the limit to: 310.
    2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_global.conf
    2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_2.conf
    2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/php.conf
    2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/includes/errordocument.conf
    2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/modsec2.conf
    2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/modsec2.user.conf
    2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/modsec2.whitelist.conf
    2010-05-21 08:46:55.716 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
    2010-05-21 08:46:55.716 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
    2010-05-21 08:46:55.716 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
    2010-05-21 08:46:55.716 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
    2010-05-21 08:46:55.776 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
    2010-05-21 08:46:55.776 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
    2010-05-21 08:46:55.776 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
    2010-05-21 08:46:55.776 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
    2010-05-21 08:55:01.403 [NOTICE] lshttpd (21788) is ready
    2010-05-21 08:55:01.407 [NOTICE] [AutoRestarter] new child process with pid=24421 is forked!
    2010-05-21 08:55:01.409 [NOTICE] [AutoRestarter] new child process with pid=24422 is forked!
  4. mistwang

    mistwang LiteSpeed Staff

    Many vhosts are hosted by this server? Is I/O wait extremely high?
    Sometimes busy/slow disk I/O may make litepseed take some time to finish configuring all vhosts, but 8 minutes is extremely long, something is wrong.
  5. nowiresltd

    nowiresltd New Member

    There are large number of vhosts, but this is Sar output. I dont have it for the period in question.

    but during a backup the load does spike. but this has occured a few time when load hasnt been high... and 8 mins is an eternaty.

    and to quote you on the first screen you do advertise that you can double capacity of apache...

    12:00:01 AM CPU %user %nice %system %iowait %steal %idle
    12:10:01 AM all 3.05 0.06 1.94 4.87 0.00 90.09
    12:20:01 AM all 2.92 0.04 1.66 2.74 0.00 92.63
    12:30:01 AM all 2.92 0.04 1.75 3.10 0.00 92.19
    12:40:01 AM all 2.93 0.06 1.59 1.90 0.00 93.52
    12:50:01 AM all 2.82 0.05 1.52 2.15 0.00 93.47
    01:00:01 AM all 2.29 0.04 1.43 2.27 0.00 93.96
    01:10:01 AM all 2.44 0.23 1.70 6.64 0.00 88.99
    01:20:02 AM all 2.45 1.11 2.30 9.27 0.00 84.87
    01:30:01 AM all 2.08 0.74 1.79 6.35 0.00 89.03
    01:40:01 AM all 4.29 0.39 1.80 3.17 0.00 90.36
    01:50:01 AM all 2.43 0.04 1.69 3.07 0.00 92.76
    02:00:01 AM all 2.54 0.20 1.72 2.11 0.00 93.43
    02:10:01 AM all 2.54 0.06 1.87 2.66 0.00 92.87
    02:20:01 AM all 2.51 0.07 1.54 2.38 0.00 93.49
    02:30:01 AM all 3.32 0.04 1.69 2.03 0.00 92.91
    02:40:01 AM all 3.03 0.06 1.72 2.01 0.00 93.17
    02:50:01 AM all 2.13 0.04 1.30 1.62 0.00 94.91
    03:00:01 AM all 2.38 0.04 1.50 1.62 0.00 94.45
    03:10:01 AM all 2.33 0.06 1.41 1.65 0.00 94.55
    03:20:01 AM all 3.00 0.05 1.69 1.92 0.00 93.35
    03:30:01 AM all 2.55 0.04 1.35 3.47 0.00 92.60
    03:40:01 AM all 2.69 0.08 1.44 1.57 0.00 94.22

    Average: CPU %user %nice %system %iowait %steal %idle
    Average: all 2.71 0.16 1.66 3.12 0.00 92.36
  6. nowiresltd

    nowiresltd New Member

    Output from SAR

    07:20:04 AM CPU %user %nice %system %iowait %steal %idle
    07:30:01 AM all 3.83 0.05 0.81 4.40 0.00 90.91
    07:40:01 AM all 4.03 0.08 1.24 4.60 0.00 90.06
    07:50:02 AM all 2.05 0.61 0.87 6.59 0.00 89.88
    08:00:01 AM all 1.71 0.75 0.63 12.14 0.00 84.77
    08:10:01 AM all 2.04 0.06 0.81 6.32 0.00 90.76
    08:20:01 AM all 1.95 0.89 1.09 6.78 0.00 89.30
    08:30:01 AM all 1.83 0.04 0.89 4.74 0.00 92.49
    08:40:01 AM all 1.95 0.06 0.97 4.12 0.00 92.90
    08:50:01 AM all 2.49 0.04 0.68 4.72 0.00 92.07
    09:00:01 AM all 1.58 0.04 0.59 3.72 0.00 94.06
    09:10:01 AM all 1.94 0.06 0.56 2.92 0.00 94.52
    09:20:01 AM all 1.91 0.05 0.64 2.18 0.00 95.22
    09:30:01 AM all 1.92 0.04 0.58 1.77 0.00 95.69
    09:40:01 AM all 2.23 0.08 0.85 1.80 0.00 95.04
    09:50:01 AM all 1.82 0.05 0.56 1.81 0.00 95.76
    10:00:01 AM all 1.59 0.04 0.52 1.72 0.00 96.13
    10:10:01 AM all 2.01 1.30 0.69 8.52 0.00 87.48
    10:20:01 AM all 1.63 1.05 0.56 10.64 0.00 86.12
    10:30:01 AM all 1.76 1.57 0.67 8.01 0.00 87.98
    10:40:01 AM all 1.77 1.27 0.87 6.91 0.00 89.18
    10:50:02 AM all 2.34 0.04 0.53 2.85 0.00 94.23


    2010-05-22 10:00:57.551 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
    2010-05-22 10:00:57.551 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
    2010-05-22 10:02:05.918 [NOTICE] lshttpd (19641) is ready


    Resulted in 1 min 22 seconds.

    I have in the meant time Remove lsws and re installed from fresh

    And there are 1235 cpanel accounts.
  7. mistwang

    mistwang LiteSpeed Staff

    1 minutes plus might be normal for server with high I/O wait serving that amount of vhosts.

    If you strace lshttpd, you will find that most time is spending in waiting for stat() call to return.

    When LSWS restarts, the previous instance of lshttpd will continue to serve requests until the new set of lshttpd processes are ready, normally, there is no down time. However, if it takes too long, the previous instance may stop before new LSWS instance is ready.

Share This Page