What settings for maximum speed on a large configuration ?

[Monarobase]

Here's the screenshot of my PHP settings :



Thanks

[NiteWave]

the settings looks normal except
Retry Timeout(secs), usually I leave it to default 0.

I think we've got some progress by your tests. so the 503 error is narrowed down to "Suexec deamon + Xcache 3.0.1" combination only and 50% possibility to happen.

Should I try Xcache 2 and disable varibale cache or is there a way to debug why this happens with Xcache 3.0.1 ?

yes, I think this is the right path to go. since xcache looks being actively developed, you may submit a bug report to them, and they may instruct you the best way how to debug xcache.

[Monarobase]

I changed the 0 to 1s in the hope that when PHP was not accessible it would wait long enough to let PHP startup.

The 50% chance is only because I didn't reload the page at exactly the right time.

I think I need to understand exactly what is happenning before contacting xcache.

From my logs I see :
2013-01-25 00:09:46.980 [NOTICE] Server Restart Request via Signal...
2013-01-25 00:09:46.981 [NOTICE] [PID: 772009]: start cgid: 807475, /usr/local/lsws/bin/lscgid
2013-01-25 00:09:46.981 [NOTICE] [Child: 772013] Start shutting down gracefully ...
2013-01-25 00:09:46.981 [NOTICE] [Child: 772014] Start shutting down gracefully ...
[...]
2013-01-25 00:09:47.230 [NOTICE] New litespeed process is ready, litespeed (772013) stops listeners

then :

2013-01-25 00:09:47.232 [INFO] [IPADR:35426-18#APVH_domain.tld] connection to [uds://tmp/lshttpd/lsphp5.sock.772009] on request #7, confirmed, 1, associated process: -1, running: 0, error: Connection reset by peer!

multiple times

then

2013-01-25 00:10:01.397 [NOTICE] [Child: 772013] Shut down successfully!
2013-01-25 00:10:01.408 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets of process with pid=772013 !
2013-01-25 00:10:01.508 [NOTICE] [AutoRestarter] child process with pid=772013 exited with status=0!
2013-01-25 00:10:01.508 [NOTICE] [PID:772009] Server Stopped!

After that everything goes back to normal

It seems that litespped fails to load a new server but when litespeed shuts it down everything works again

[NiteWave]

during restart, please run
#pstree -p

then you'll get the idea how graceful restart works.

you'll see 2 trees which root is litespeed process. with the info of "pstree -p", at least can know 503 error is under which tree -- new or old.

[Monarobase]

Sorry for the delay.

Due to lack of time I installed Xcache 2.0.1 and have disabled the variable cache, hopeing that someone might debug and fix the issue without me reporting it…

I've got one last (quite big) problem.

I run Atomicorp ASL T-WAF in front of litespeed and sometimes I get an error saying from the T-WAF with the following message :

---
Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /XXX.html.

Reason: Error reading from remote server

Apache Server at www.XXXXXXXX.tld Port 30001
---

ASL seem to be saying it's an issue with litespeed but litespeed doesn't log anything when the error happens and ASL T-WAF doesn't log much either (except that Litespeed gave an invalid response).

Do you have any suggestions about how I could determine if it rearly is Litespeed giving a bad response or if it's an issue with the T-WAF.

The error message only happens about 20 times a day so it's very difficult to catch and almost impossible to determine if Litespeed is actually giving a blank page or a strange result or if it's the T-WAF that's not responding fast enough for litespeed or the other way around.

When I searched for these issues, alot of topics about Litespeed came up… with various other proxies, so it seems that litespeed could be for something in this issue.

[NiteWave]

since 4.2.2 is out, can you try 4.2.2 see if issue still there ?
in 4.2.2 release log:
http://www.litespeedtech.com/litespe...lease-log.html
"Improved mod_security compatibility with gotroot ruleset"

[Monarobase]

Thanks I will give it a try

We don't run the rules in litespeed but have ASL's proxy in front of litespeed.

Running their hourly updated rules seems a bit dangerous directly in litespeed as the rules are only tested in ASL's modsecurity before being released and it seems that it's almost impossible for litespeed to follow these updates.

ASL is however being helpfull on this issue. They helped me set up full modsecurity logging to try and see if there's something reported when this issue happens.

I'll let you know if/when I get some more information.

[Monarobase]

I've got an update from ASL.

ASL T-WAF runs an optimised version of Apache with Mod Security and Mod_proxy in front of litespeed. (note : this hardly adds any latency on our server, we get 15ms responses even with mod_security in T-WAF proxy).

This is what is happening :

I'm getting the following error from mod_proxy : "proxy: error reading status line from remote server"

"proxy: error reading status line from remote server" error message caused by the race condition that the backend server closed the pooled connection after the connection check by the proxy and before data sent by the proxy reached the backend

1) Client connects to the T-WAF (local reverse proxy)
2) T-WAF checks connection is live
3) Litespeed goes away
4) T-WAF proxy doesn't get the data from litespeed

Do you have any suggestions about how I could get litespeed to play well running behind a reverse proxy ?

Thanks

[NiteWave]

while no environment to experiment with this, it looks litespeed is behind apache and handle the requests passed from apache, but connection between apache and litespeed closed by litespeed in some cases.

need tune
1)lsws admin console -> Server -> Tuning -> Connection
not sure if disable keep-alive will be a solution
Max Keep-Alive Requests:0
may need increase
Connection Timeout (secs)
as well

2)lsws admin console -> Server -> Security:
disable "Per Client Throttling"(the default value)
since now only one client IP.

[Monarobase]

Thanks,

I'm not sure how to completly disable per client throttling.

In Server > Security I've got 5 sections :

File Access
Per Client Throttling
External Application Ressource
Access Denied Directories
Access Control

Inside Per Client Throttling I've got :

Static Requests/second (number >= 0)
Dynamic Requests/second (number >= 0)
Outbound Bandwidth (bytes/sec)
Inbound Bandwidth (bytes/sec)
Connection Soft Limit
Connection Hard Limit
Block Bad Request
Grace Period (sec)
Banned Period (sec)

Is it one of these settings I need to change to completly disable it ?

Thanks !