Support

nowiresltd · #1 05-20-2010, 01:52 AM

Hi

Cant seem to track down the reason why, but litespeed seems to be shutting down and taking approx 2 mins to come back up ever 3-5 hours (as reported by panopta), I have switched back to apache and its been 24 hours since any outage reported by panopta.

I initially had an issue where a site in the httpd.conf had an ip not bound to the server, so i removed that but it seems to have not cured the issue.

Where do i look to see why its restarting ?

I really want to use litespeed, but its only a matter of time before someone notices these little outages, not to mention ruining my uptime stats

Any guidance or advice will be much appreciated

Also have it running on two other servers without issue, the only thing about this one is the SSI fix given in the forum a few posts down.

mistwang · #2 05-20-2010, 08:08 PM

Are you using cPanel?
Have you checked error log for the period of outage?

nowiresltd · #3 05-21-2010, 08:48 AM

Hi

I have found an example log extract which i dont want to post on the forum but will send you in a PM of a the full log if you want.

Its a mystery as to what its doing between 08:46 and 08:55, this has happend a few times, but only since I took the SSI fix. (i think)

2010-05-21 08:46:45.632 [NOTICE] Server Restart Request via Signal...
2010-05-21 08:46:45.632 [NOTICE] [6131] Graceful Restart...
2010-05-21 08:46:45.632 [NOTICE] [Child: 9224] Start shutting down gracefully ...
2010-05-21 08:46:45.637 [NOTICE] [PID: 6131]: start cgid: 21582, /usr/local/lsws/bin/lscgid

2010-05-21 08:46:45.637 [INFO] Pass listener *:7080.
2010-05-21 08:46:45.640 [NOTICE] [6131] Start new instance of lshttpd: 21583

2010-05-21 08:46:45.641 [INFO] Stop listener *:7080.
2010-05-21 08:46:46.400 [NOTICE] [Child: 9223] Start shutting down gracefully ...

2010-05-21 08:46:47.608 [NOTICE] Your License key will expire in 11 days
2010-05-21 08:46:47.609 [NOTICE] [21583] Loading LiteSpeed/4.0.14 Enterprise ...
2010-05-21 08:46:47.609 [NOTICE] [ADMIN] server socket: uds://usr/local/lsws/admin/tmp/admin.sock.7848
2010-05-21 08:46:47.609 [NOTICE] Loading configuration from /usr/local/lsws/conf/httpd_config.xml ...

2010-05-21 08:46:47.611 [NOTICE] Recovering server socket: [*:7080]
2010-05-21 08:46:47.612 [NOTICE] [config] chroot is disabled.
2010-05-21 08:46:47.612 [INFO] old priority: -19, new priority: -19
2010-05-21 08:46:47.612 [INFO] [config:server:basic] For better obscurity, server version number is hidden in the response header.
2010-05-21 08:46:48.006 [NOTICE] [PID: 21618]: start cgid: 21635, /usr/local/lsws/bin/httpd
2010-05-21 08:46:48.076 [WARN] [config:server:listener] No listener is available for normal virtual host!
2010-05-21 08:46:48.076 [NOTICE] The maximum number of file descriptor limit is set to 16384.
2010-05-21 08:46:48.264 [NOTICE] [config:server:epsr:lsphp4]'Process Limit' probably is too low, adjust the limit to: 310.
2010-05-21 08:46:50.249 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_global.conf
2010-05-21 08:46:50.557 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_2.conf
2010-05-21 08:46:50.557 [INFO] Processing config file: /usr/local/apache/conf/php.conf
2010-05-21 08:46:50.575 [INFO] Processing config file: /usr/local/apache/conf/includes/errordocument.conf
2010-05-21 08:46:50.702 [INFO] Processing config file: /usr/local/apache/conf/modsec2.conf
2010-05-21 08:46:50.717 [INFO] Processing config file: /usr/local/apache/conf/modsec2.user.conf
2010-05-21 08:46:50.939 [INFO]
2010-05-21 08:46:51.121 [INFO] Processing config file: /usr/local/apache/conf/modsec2.whitelist.conf
2010-05-21 08:46:51.226 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
2010-05-21 08:46:51.226 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
2010-05-21 08:46:51.226 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
2010-05-21 08:46:51.226 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
2010-05-21 08:46:51.286 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
2010-05-21 08:46:51.286 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
2010-05-21 08:46:51.286 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
2010-05-21 08:46:51.286 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
2010-05-21 08:46:54.408 [NOTICE] Your License key will expire in 11 days
2010-05-21 08:46:54.408 [NOTICE] [21734] Loading LiteSpeed/4.0.14 Enterprise ...
2010-05-21 08:46:54.409 [NOTICE] [ADMIN] server socket: uds://usr/local/lsws/admin/tmp/admin.sock.7354
2010-05-21 08:46:54.409 [NOTICE] Loading configuration from /usr/local/lsws/conf/httpd_config.xml ...
2010-05-21 08:46:54.410 [NOTICE] [config] chroot is disabled.
2010-05-21 08:46:54.410 [INFO] old priority: 18, new priority: -19
2010-05-21 08:46:54.410 [INFO] [config:server:basic] For better obscurity, server version number is hidden in the response header.
2010-05-21 08:46:54.411 [NOTICE] [PID: 21752]: start cgid: 21755, /usr/local/lsws/bin/httpd
2010-05-21 08:46:54.411 [WARN] [config:server:listener] No listener is available for normal virtual host!
2010-05-21 08:46:54.412 [NOTICE] The maximum number of file descriptor limit is set to 16384.
2010-05-21 08:46:54.412 [NOTICE] [config:server:epsr:lsphp4]'Process Limit' probably is too low, adjust the limit to: 310.
2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_global.conf
2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_2.conf
2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/php.conf
2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/includes/errordocument.conf
2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/modsec2.conf
2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/modsec2.user.conf
2010-05-21 08:46:54.414 [INFO] Processing config file: /usr/local/apache/conf/modsec2.whitelist.conf
2010-05-21 08:46:55.685 [NOTICE] Your License key will expire in 11 days
2010-05-21 08:46:55.686 [NOTICE] [21770] Loading LiteSpeed/4.0.14 Enterprise ...
2010-05-21 08:46:55.686 [NOTICE] [ADMIN] server socket: uds://usr/local/lsws/admin/tmp/admin.sock.7089
2010-05-21 08:46:55.686 [NOTICE] Loading configuration from /usr/local/lsws/conf/httpd_config.xml ...
2010-05-21 08:46:55.687 [NOTICE] [config] chroot is disabled.
2010-05-21 08:46:55.687 [INFO] old priority: 18, new priority: -19
2010-05-21 08:46:55.687 [INFO] [config:server:basic] For better obscurity, server version number is hidden in the response header.
2010-05-21 08:46:55.689 [NOTICE] [PID: 21788]: start cgid: 21791, /usr/local/lsws/bin/httpd
2010-05-21 08:46:55.689 [WARN] [config:server:listener] No listener is available for normal virtual host!
2010-05-21 08:46:55.689 [NOTICE] The maximum number of file descriptor limit is set to 16384.
2010-05-21 08:46:55.689 [NOTICE] [config:server:epsr:lsphp4]'Process Limit' probably is too low, adjust the limit to: 310.
2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_global.conf
2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_2.conf
2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/php.conf
2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/includes/errordocument.conf
2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/modsec2.conf
2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/modsec2.user.conf
2010-05-21 08:46:55.691 [INFO] Processing config file: /usr/local/apache/conf/modsec2.whitelist.conf
2010-05-21 08:46:55.716 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
2010-05-21 08:46:55.716 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
2010-05-21 08:46:55.716 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
2010-05-21 08:46:55.716 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
2010-05-21 08:46:55.776 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
2010-05-21 08:46:55.776 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
2010-05-21 08:46:55.776 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
2010-05-21 08:46:55.776 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
2010-05-21 08:55:01.403 [NOTICE] lshttpd (21788) is ready
2010-05-21 08:55:01.407 [NOTICE] [AutoRestarter] new child process with pid=24421 is forked!
2010-05-21 08:55:01.409 [NOTICE] [AutoRestarter] new child process with pid=24422 is forked!

mistwang · #4 05-21-2010, 03:25 PM

Many vhosts are hosted by this server? Is I/O wait extremely high?
Sometimes busy/slow disk I/O may make litepseed take some time to finish configuring all vhosts, but 8 minutes is extremely long, something is wrong.

nowiresltd · #5 05-22-2010, 01:50 AM

There are large number of vhosts, but this is Sar output. I dont have it for the period in question.

but during a backup the load does spike. but this has occured a few time when load hasnt been high... and 8 mins is an eternaty.

and to quote you on the first screen you do advertise that you can double capacity of apache...

12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 3.05 0.06 1.94 4.87 0.00 90.09
12:20:01 AM all 2.92 0.04 1.66 2.74 0.00 92.63
12:30:01 AM all 2.92 0.04 1.75 3.10 0.00 92.19
12:40:01 AM all 2.93 0.06 1.59 1.90 0.00 93.52
12:50:01 AM all 2.82 0.05 1.52 2.15 0.00 93.47
01:00:01 AM all 2.29 0.04 1.43 2.27 0.00 93.96
01:10:01 AM all 2.44 0.23 1.70 6.64 0.00 88.99
01:20:02 AM all 2.45 1.11 2.30 9.27 0.00 84.87
01:30:01 AM all 2.08 0.74 1.79 6.35 0.00 89.03
01:40:01 AM all 4.29 0.39 1.80 3.17 0.00 90.36
01:50:01 AM all 2.43 0.04 1.69 3.07 0.00 92.76
02:00:01 AM all 2.54 0.20 1.72 2.11 0.00 93.43
02:10:01 AM all 2.54 0.06 1.87 2.66 0.00 92.87
02:20:01 AM all 2.51 0.07 1.54 2.38 0.00 93.49
02:30:01 AM all 3.32 0.04 1.69 2.03 0.00 92.91
02:40:01 AM all 3.03 0.06 1.72 2.01 0.00 93.17
02:50:01 AM all 2.13 0.04 1.30 1.62 0.00 94.91
03:00:01 AM all 2.38 0.04 1.50 1.62 0.00 94.45
03:10:01 AM all 2.33 0.06 1.41 1.65 0.00 94.55
03:20:01 AM all 3.00 0.05 1.69 1.92 0.00 93.35
03:30:01 AM all 2.55 0.04 1.35 3.47 0.00 92.60
03:40:01 AM all 2.69 0.08 1.44 1.57 0.00 94.22

Average: CPU %user %nice %system %iowait %steal %idle
Average: all 2.71 0.16 1.66 3.12 0.00 92.36

nowiresltd · #6 05-22-2010, 08:54 AM

Output from SAR

07:20:04 AM CPU %user %nice %system %iowait %steal %idle
07:30:01 AM all 3.83 0.05 0.81 4.40 0.00 90.91
07:40:01 AM all 4.03 0.08 1.24 4.60 0.00 90.06
07:50:02 AM all 2.05 0.61 0.87 6.59 0.00 89.88
08:00:01 AM all 1.71 0.75 0.63 12.14 0.00 84.77
08:10:01 AM all 2.04 0.06 0.81 6.32 0.00 90.76
08:20:01 AM all 1.95 0.89 1.09 6.78 0.00 89.30
08:30:01 AM all 1.83 0.04 0.89 4.74 0.00 92.49
08:40:01 AM all 1.95 0.06 0.97 4.12 0.00 92.90
08:50:01 AM all 2.49 0.04 0.68 4.72 0.00 92.07
09:00:01 AM all 1.58 0.04 0.59 3.72 0.00 94.06
09:10:01 AM all 1.94 0.06 0.56 2.92 0.00 94.52
09:20:01 AM all 1.91 0.05 0.64 2.18 0.00 95.22
09:30:01 AM all 1.92 0.04 0.58 1.77 0.00 95.69
09:40:01 AM all 2.23 0.08 0.85 1.80 0.00 95.04
09:50:01 AM all 1.82 0.05 0.56 1.81 0.00 95.76
10:00:01 AM all 1.59 0.04 0.52 1.72 0.00 96.13
10:10:01 AM all 2.01 1.30 0.69 8.52 0.00 87.48
10:20:01 AM all 1.63 1.05 0.56 10.64 0.00 86.12
10:30:01 AM all 1.76 1.57 0.67 8.01 0.00 87.98
10:40:01 AM all 1.77 1.27 0.87 6.91 0.00 89.18
10:50:02 AM all 2.34 0.04 0.53 2.85 0.00 94.23

2010-05-22 10:00:57.551 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
2010-05-22 10:00:57.551 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
2010-05-22 10:02:05.918 [NOTICE] lshttpd (19641) is ready

Resulted in 1 min 22 seconds.

I have in the meant time Remove lsws and re installed from fresh

And there are 1235 cpanel accounts.

mistwang · #7 05-24-2010, 10:37 PM

1 minutes plus might be normal for server with high I/O wait serving that amount of vhosts.

If you strace lshttpd, you will find that most time is spending in waiting for stat() call to return.

When LSWS restarts, the previous instance of lshttpd will continue to serve requests until the new set of lshttpd processes are ready, normally, there is no down time. However, if it takes too long, the previous instance may stop before new LSWS instance is ready.