What settings for maximum speed on a large configuration ?

[Monarobase]

I'll have to check my logs, but I was wandering if the restart wasn't a symptom instead of a cause.

Could it be PHP hitting litespeed's CPU limit ?

I upped it from the default 80% to 800% yesterday but haven't had a chance to check the logs yet.

I get a 503 error when litespeed can't access PHP. Over the last 20 times in the past two days users got 503 errors I see litespeed doing a gracefull restart then a few seconds later litespeed says it tried to access PHP 3 times and didn't get an answer so it shows a 503 error message.

I'll check later today and see if the new CPU limit has solved the 503 error messages or not

[Monarobase]

Here is a section of the logs :

2013-01-23 10:10:24.000 [NOTICE] Server Restart Request via Signal...
2013-01-23 10:10:24.000 [NOTICE] [PID: 582321]: start cgid: 619120, /usr/local/lsws/bin/lscgid
2013-01-23 10:10:24.000 [INFO] Pass listener 127.0.0.1:443.
2013-01-23 10:10:24.000 [INFO] Pass listener 127.0.0.1:80.
2013-01-23 10:10:24.000 [INFO] Pass listener IP1.XX.XX.XX:443.
2013-01-23 10:10:24.000 [INFO] Pass listener IP1.XX.XX.XX:80.
2013-01-23 10:10:24.000 [INFO] Pass listener *:7080.
2013-01-23 10:10:24.001 [NOTICE] [Child: 582326] Start shutting down gracefully ...
2013-01-23 10:10:24.001 [NOTICE] [Child: 582325] Start shutting down gracefully ...
2013-01-23 10:10:24.001 [INFO] Stop listener 127.0.0.1:443.
2013-01-23 10:10:24.001 [INFO] Stop listener 127.0.0.1:80.
2013-01-23 10:10:24.001 [INFO] Stop listener IP1.XX.XX.XX:443.
2013-01-23 10:10:24.001 [INFO] Stop listener IP1.XX.XX.XX:80.
2013-01-23 10:10:24.001 [INFO] Stop listener *:7080.
2013-01-23 10:10:24.032 [NOTICE] Your License key will expire in 10 days
2013-01-23 10:10:24.032 [NOTICE] Loading LiteSpeed/4.2.1 Enterprise ...
2013-01-23 10:10:24.033 [NOTICE] [ADMIN] server socket: uds://usr/local/lsws/admin/tmp/admin.sock.7318
2013-01-23 10:10:24.033 [NOTICE] Loading configuration from /usr/local/lsws/conf/httpd_config.xml ...
2013-01-23 10:10:24.035 [NOTICE] Recovering server socket: [127.0.0.1:443]
2013-01-23 10:10:24.035 [NOTICE] Recovering server socket: [127.0.0.1:80]
2013-01-23 10:10:24.036 [NOTICE] Recovering server socket: [IP1.XX.XX.XX:443]
2013-01-23 10:10:24.036 [NOTICE] Recovering server socket: [IP1.XX.XX.XX:80]
2013-01-23 10:10:24.036 [NOTICE] Recovering server socket: [*:7080]
2013-01-23 10:10:24.036 [NOTICE] [config] chroot is disabled.
2013-01-23 10:10:24.036 [INFO] old priority: 0, new priority: 0
2013-01-23 10:10:24.036 [INFO] [config:server:basic] For better obscurity, server version number is hidden in the response header.
2013-01-23 10:10:24.039 [NOTICE] [PID: 619137]: start cgid: 619138, /usr/local/lsws/bin/httpd
2013-01-23 10:10:24.039 [NOTICE] The maximum number of file descriptor limit is set to 15000.
2013-01-23 10:10:24.055 [NOTICE] [config:server:epsr:lsphp4]'Process Limit' probably is too low, adjust the limit to: 310.
2013-01-23 10:10:24.055 [WARN] [config:server:rails] Cannot find ruby interpreter, Rails easy configuration is turned off
2013-01-23 10:10:24.055 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_global.conf
2013-01-23 10:10:24.055 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_main_2.conf
2013-01-23 10:10:24.055 [INFO] Processing config directory: /etc/httpd/conf.d
2013-01-23 10:10:24.055 [INFO] Processing config file: /etc/httpd/conf.d/000_mod_sed.conf
2013-01-23 10:10:24.055 [INFO] Processing config file: /etc/httpd/conf.d/mod_evasive.conf
2013-01-23 10:10:24.055 [INFO] Processing config file: /usr/local/apache/conf/modhostinglimits.conf
2013-01-23 10:10:24.055 [INFO] Processing config file: /usr/local/apache/conf/php.conf
2013-01-23 10:10:24.055 [INFO] Processing config file: /usr/local/apache/conf/includes/errordocument.conf
2013-01-23 10:10:24.088 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
2013-01-23 10:10:24.088 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_global.conf
2013-01-23 10:10:24.088 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
2013-01-23 10:10:24.088 [INFO] Processing config file: /usr/local/apache/conf/includes/pre_virtualhost_2.conf
2013-01-23 10:10:24.089 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
2013-01-23 10:10:24.089 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_global.conf
2013-01-23 10:10:24.089 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
2013-01-23 10:10:24.089 [INFO] Processing config file: /usr/local/apache/conf/includes/post_virtualhost_2.conf
2013-01-23 10:10:24.108 [NOTICE] litespeed (619137) is ready
2013-01-23 10:10:24.108 [NOTICE] [AutoRestarter] new child process with pid=619141 is forked!
2013-01-23 10:10:24.108 [NOTICE] [AutoRestarter] new child process with pid=619142 is forked!
2013-01-23 10:10:24.108 [INFO] Stop listener 127.0.0.1:443.
2013-01-23 10:10:24.108 [INFO] Stop listener 127.0.0.1:80.
2013-01-23 10:10:24.108 [INFO] Stop listener IP1.XX.XX.XX:443.
2013-01-23 10:10:24.108 [INFO] Stop listener IP1.XX.XX.XX:80.
2013-01-23 10:10:24.108 [INFO] Stop listener IP2.XX.XX.XX:443.
2013-01-23 10:10:24.109 [INFO] Stop listener *:7080.
2013-01-23 10:10:24.109 [INFO] Stop listener IP2.XX.XX.XX:80.
2013-01-23 10:10:24.109 [NOTICE] [Child: 619142] Setup swapping space...
2013-01-23 10:10:24.109 [NOTICE] [Child: 619141] Setup swapping space...
2013-01-23 10:10:24.109 [NOTICE] [child: 619142] Successfully change current user to nobody
2013-01-23 10:10:24.109 [NOTICE] [Child: 619142] Core dump is enabled.
2013-01-23 10:10:24.109 [NOTICE] [child: 619141] Successfully change current user to nobody
2013-01-23 10:10:24.109 [NOTICE] [Child: 619141] Core dump is enabled.
2013-01-23 10:10:24.109 [NOTICE] [Child: 619142] LiteSpeed/4.2.1 Enterprise starts successfully!
2013-01-23 10:10:24.110 [NOTICE] [Child: 619141] LiteSpeed/4.2.1 Enterprise starts successfully!
2013-01-23 10:10:24.191 [NOTICE] New litespeed process is ready, litespeed (582325) stops listeners
2013-01-23 10:10:24.191 [INFO] Stop listener 127.0.0.1:443.
2013-01-23 10:10:24.191 [INFO] Stop listener 127.0.0.1:80.
2013-01-23 10:10:24.191 [INFO] Stop listener IP1.XX.XX.XX:443.
2013-01-23 10:10:24.191 [INFO] Stop listener IP1.XX.XX.XX:80.
2013-01-23 10:10:24.191 [INFO] Stop listener *:7080.
2013-01-23 10:10:24.202 [NOTICE] New litespeed process is ready, litespeed (582326) stops listeners
2013-01-23 10:10:24.202 [INFO] Stop listener IP2.XX.XX.XX:443.
2013-01-23 10:10:24.202 [INFO] Stop listener IP2.XX.XX.XX:80.
2013-01-23 10:10:24.202 [NOTICE] [Child: 582326] Shut down successfully! 
2013-01-23 10:10:24.211 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets of process with pid=582326 !
2013-01-23 10:10:24.311 [NOTICE] [AutoRestarter] child process with pid=582326 exited with status=0!
2013-01-23 10:10:24.857 [INFO] [IPCIENT1.XX.XX.XX:34563-1#APVH_domain.tld] connection to [uds://tmp/lshttpd/lsphp5.sock.582321] on request #13, confirmed, 1, associated process: -1, running: 0, error: Connection reset by peer!
2013-01-23 10:10:24.866 [INFO] [IPCIENT1.XX.XX.XX:34563-1#APVH_domain.tld] connection to [uds://tmp/lshttpd/lsphp5.sock.582321] on request #0, confirmed, 1, associated process: -1, running: 0, error: Connection reset by peer!
2013-01-23 10:10:24.874 [INFO] [IPCIENT1.XX.XX.XX:34563-1#APVH_domain.tld] connection to [uds://tmp/lshttpd/lsphp5.sock.582321] on request #0, confirmed, 1, associated process: -1, running: 0, error: Connection reset by peer!
2013-01-23 10:10:24.874 [NOTICE] [IPCIENT1.XX.XX.XX:34563-1#APVH_domain.tld] Max retries has been reached, 503!
2013-01-23 10:10:24.874 [NOTICE] [IPCIENT1.XX.XX.XX:34563-1#APVH_domain.tld] oops! 503 Service Unavailable
2013-01-23 10:10:24.874 [NOTICE] [IPCIENT1.XX.XX.XX:34563-1#APVH_domain.tld] Content len: 0, Request line: 'GET /download/file.php?id=20060&sid=DWNURL HTTP/1.1'
2013-01-23 10:10:24.874 [INFO] [IPCIENT1.XX.XX.XX:34563-1#APVH_domain.tld] Cookie len: 84, phpbb3_igyxp_u=1; phpbb3_igyxp_k=; COOKIE_CONTENTS_1
2013-01-23 10:10:24.874 [INFO] [IPCIENT1.XX.XX.XX:34563-1#APVH_domain.tld] File not found [/home/USER1/public_html/503.shtml] 
2013-01-23 10:10:25.133 [INFO] [HTAccess] Updating configuration from [/home/USER1/public_html/.htaccess]
2013-01-23 10:10:25.200 [NOTICE] [AutoUpdate] Checking for new releases..., pid=619152
2013-01-23 10:10:25.200 [NOTICE] Start to clean up cache directory, pid=619153
2013-01-23 10:10:25.478 [INFO] [AutoUpdate] No new update.
2013-01-23 10:10:26.609 [INFO] [IPCIENT1.XX.XX.XX:34561-4#APVH_domain.tld] connection to [uds://tmp/lshttpd/lsphp5.sock.582321] on request #0, confirmed, 1, associated process: -1, running: 0, error: Connection reset by peer!
2013-01-23 10:10:26.618 [INFO] [IPCIENT1.XX.XX.XX:34561-4#APVH_domain.tld] connection to [uds://tmp/lshttpd/lsphp5.sock.582321] on request #0, confirmed, 1, associated process: -1, running: 0, error: Connection reset by peer!
2013-01-23 10:10:26.626 [INFO] [IPCIENT1.XX.XX.XX:34561-4#APVH_domain.tld] connection to [uds://tmp/lshttpd/lsphp5.sock.582321] on request #0, confirmed, 1, associated process: -1, running: 0, error: Connection reset by peer!
2013-01-23 10:10:26.626 [NOTICE] [IPCIENT1.XX.XX.XX:34561-4#APVH_domain.tld] Max retries has been reached, 503!
2013-01-23 10:10:26.626 [NOTICE] [IPCIENT1.XX.XX.XX:34561-4#APVH_domain.tld] oops! 503 Service Unavailable
2013-01-23 10:10:26.626 [NOTICE] [IPCIENT1.XX.XX.XX:34561-4#APVH_domain.tld] Content len: 0, Request line: 'GET /download/file.php?avatar=2215_1331283561.jpg HTTP/1.1'
2013-01-23 10:10:26.626 [INFO] [IPCIENT1.XX.XX.XX:34561-4#APVH_domain.tld] Cookie len: 84, phpbb3_igyxp_u=1; phpbb3_igyxp_k=; COOKIE_CONTENTS_1
2013-01-23 10:10:26.626 [INFO] [IPCIENT1.XX.XX.XX:34561-4#APVH_domain.tld] File not found [/home/USER1/public_html/503.shtml]

Every 503 error looks like this with :

First : "[NOTICE] Server Restart Request via Signal..."

Second : "[Child: XXXXX] Start shutting down gracefully ..." third the "[NOTICE] New litespeed process is ready, litespeed (582325) stops listeners"

Then "[INFO] [IPCIENT2.XX.XX.XX:34556-16#APVH_domain.tld:lsapi] connection to [uds://tmp/lshttpd/lsphp5.sock.XXXXX] on request #0, confirmed, 1, associated process: -1, running: 0, error: Connection reset by peer!"

And finaly "[NOTICE] [IPCIENT1.XX.XX.XX:34558-8#APVH_domain.tld] Max retries has been reached, 503!"

[NiteWave]

searched out a related thread:
http://www.litespeedtech.com/support...ead.php?t=6186

looks the restart not from graceful restart, but from cPanel.

[Monarobase]

I get the same 503 erros when I run a gracefull restart from the litespeed web admin pannel.

I haven't got restarts every 20 minutes like in that topic.

I can't have 503 errors every time a client adds a subdomain…

The problem here is not what is causing the gracefull restart but why PHP becomes unavailable for almost 10 seconds just after the gracefull restart.

I don't think a gracefull restart shouldn't affect PHP processes… I've got the impression that litespeed restarts then asks all PHP children to restart and during the time it takes PHP to restart all children Litespeed gives out 503 errors because it can't connect to PHP.

I've only got 5 production sites on this server, my guess is it could take longer to restart all PHP processes when there are alot more children…

Could this be a setting or a bug with LSAPI PHP Suexec in deamon mode ?

Litespeed is sending a call to a PHP process that doesn't exist anymore because it has been restarted.

Where would I look to find out if it's a bad setting ?

[Monarobase]

Here's the output of the restarts log :

[root@newns10 demo]# cat /usr/local/lsws/logs/ap_cmd_param.log | grep "Wed Jan 23"
Wed Jan 23 00:09:35 CET 2013: -k graceful
Wed Jan 23 00:20:46 CET 2013: -l
Wed Jan 23 00:20:46 CET 2013: -V
Wed Jan 23 00:20:46 CET 2013: -v
Wed Jan 23 00:20:46 CET 2013: -DSSL -t -f /usr/local/apache/conf/httpd.conf.work.yA_74YWLDRc4N0NO
Wed Jan 23 00:20:47 CET 2013: -DSSL -t -f /usr/local/apache/conf/httpd.conf.work.yA_74YWLDRc4N0NO
Wed Jan 23 01:22:16 CET 2013: -v
Wed Jan 23 01:22:17 CET 2013: -DSSL -t -f /usr/local/apache/conf/httpd.conf.work._X5VOBfoiKV2D8lh
Wed Jan 23 01:22:17 CET 2013: -DSSL -t -f /usr/local/apache/conf/httpd.conf.work._X5VOBfoiKV2D8lh
Wed Jan 23 01:25:01 CET 2013: -v
Wed Jan 23 01:25:02 CET 2013: -DSSL -t -f /usr/local/apache/conf/httpd.conf.work.458t5scsOzSeAxAh
Wed Jan 23 01:25:02 CET 2013: -DSSL -t -f /usr/local/apache/conf/httpd.conf.work.458t5scsOzSeAxAh
Wed Jan 23 01:49:47 CET 2013: -l
Wed Jan 23 01:49:47 CET 2013: -V
Wed Jan 23 01:49:48 CET 2013: -v
Wed Jan 23 01:49:48 CET 2013: -DSSL -t -f /usr/local/apache/conf/httpd.conf.work.qWUK8ZWGJS6bRV2K
Wed Jan 23 01:49:49 CET 2013: -DSSL -t -f /usr/local/apache/conf/httpd.conf.work.qWUK8ZWGJS6bRV2K
Wed Jan 23 02:06:24 CET 2013: -k graceful
Wed Jan 23 02:22:25 CET 2013: -v
Wed Jan 23 04:03:14 CET 2013: -k graceful
Wed Jan 23 06:00:04 CET 2013: -k graceful
Wed Jan 23 08:13:34 CET 2013: -k graceful
Wed Jan 23 10:10:22 CET 2013: -k graceful
Wed Jan 23 12:07:12 CET 2013: -k graceful
Wed Jan 23 13:05:59 CET 2013: -k graceful

As you can see there is nothing abnormal about the number of restarts. It's just the 503 error messages that happen a few seconds after every restart that are the problem.

[NiteWave]

have you touched following default setting ?
Server->General->Graceful Restart Timeout (secs)

[Monarobase]

I've got this :

Auto Restart : Yes
Auto Fix 503 Error : Yes
Graceful Restart Timeout (secs) : 120

The gracefull restart of litespeed is done with a few seconds. I'm pretty sure I did not touch this setting even though it says the default is 300.

I don't think I'm hitting the 120 second timeout as the whole procedure takes less than 20 seconds before PHP comes back.

[Monarobase]

Hello,

As I'm still quite new to Litespeed, is fixing this issue included in the free forum support or should I pay to have this fixed ?

What's the difference between paid and forum support ? Is it just the response time or the whole level of intervention ?

I ask this because with some companies I've tried to work with they didn't tell me when the question I asked was too complicated for free support and I had to wait a long time and insist alot before being told I had to pay to get an answer.

I don't mind it taking a few extra days by the forum although I've only got 7 more days trial and need this to be fixed before purchasing the paid licence.

Thanks

[NiteWave]

this issue is not a known issue, so I don't reply you at once.

Could this be a setting or a bug with LSAPI PHP Suexec in deamon mode ?

this looks a nice clue and I actually did simple tests in daemon mode, tried to reproduce 503 error --- but failed.

not sure if it's ok for you to run short time tests on your production server.
e.g., switch to non-daemon mode, restart, see if 503 still happen. or disable xcache temporarily and test if no 503.

another good way to strace lsphp5 process, to catch the 503 when it happens.

in debug's point of view, it's good you can easily reproduce 503 error(by graceful restart) in current environment.

also, please give a screen shot of lsphp5 settings, see if we can give any advice, which may avoid or reduce 503 error.

[Monarobase]

Hello,

I've run the tests,

I haven't been able to reproduce this with Suexec deamon deactivated and Xcache enabled, or with Suexec deamon activated and Xcache disabled.

To get a 503 error you have to be running a php instance at the right moment.

To reproduce this I run a gracefull restart and as fast as possible reload a web php using PHP multiple times.

Out of 10 tries I managed 5 503 error messages with Suexec deamon + Xcache 3.0.1.

Should I try Xcache 2 and disable varibale cache or is there a way to debug why this happens with Xcache 3.0.1 ?

It rearly makes sense to be able to allow variable cache… and it's stable so long as there isn't a gracefull restart.