What settings for maximum speed on a large configuration ?

#21
during restart, please run
#pstree -p

then you'll get the idea how graceful restart works.

you'll see 2 trees which root is litespeed process. with the info of "pstree -p", at least can know 503 error is under which tree -- new or old.
 

Monarobase

Well-Known Member
#22
Sorry for the delay.

Due to lack of time I installed Xcache 2.0.1 and have disabled the variable cache, hopeing that someone might debug and fix the issue without me reporting it…

I've got one last (quite big) problem.

I run Atomicorp ASL T-WAF in front of litespeed and sometimes I get an error saying from the T-WAF with the following message :

---
Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /XXX.html.

Reason: Error reading from remote server

Apache Server at www.XXXXXXXX.tld Port 30001
---

ASL seem to be saying it's an issue with litespeed but litespeed doesn't log anything when the error happens and ASL T-WAF doesn't log much either (except that Litespeed gave an invalid response).

Do you have any suggestions about how I could determine if it rearly is Litespeed giving a bad response or if it's an issue with the T-WAF.

The error message only happens about 20 times a day so it's very difficult to catch and almost impossible to determine if Litespeed is actually giving a blank page or a strange result or if it's the T-WAF that's not responding fast enough for litespeed or the other way around.

When I searched for these issues, alot of topics about Litespeed came up… with various other proxies, so it seems that litespeed could be for something in this issue.
 

Monarobase

Well-Known Member
#24
Thanks I will give it a try :)

We don't run the rules in litespeed but have ASL's proxy in front of litespeed.

Running their hourly updated rules seems a bit dangerous directly in litespeed as the rules are only tested in ASL's modsecurity before being released and it seems that it's almost impossible for litespeed to follow these updates.

ASL is however being helpfull on this issue. They helped me set up full modsecurity logging to try and see if there's something reported when this issue happens.

I'll let you know if/when I get some more information.
 

Monarobase

Well-Known Member
#25
I've got an update from ASL.

ASL T-WAF runs an optimised version of Apache with Mod Security and Mod_proxy in front of litespeed. (note : this hardly adds any latency on our server, we get 15ms responses even with mod_security in T-WAF proxy).

This is what is happening :

I'm getting the following error from mod_proxy : "proxy: error reading status line from remote server"

"proxy: error reading status line from remote server" error message caused by the race condition that the backend server closed the pooled connection after the connection check by the proxy and before data sent by the proxy reached the backend
1) Client connects to the T-WAF (local reverse proxy)
2) T-WAF checks connection is live
3) Litespeed goes away
4) T-WAF proxy doesn't get the data from litespeed

Do you have any suggestions about how I could get litespeed to play well running behind a reverse proxy ?

Thanks
 

NiteWave

Administrator
#26
while no environment to experiment with this, it looks litespeed is behind apache and handle the requests passed from apache, but connection between apache and litespeed closed by litespeed in some cases.

need tune
1)lsws admin console -> Server -> Tuning -> Connection
not sure if disable keep-alive will be a solution
Max Keep-Alive Requests:0
may need increase
Connection Timeout (secs)
as well

2)lsws admin console -> Server -> Security:
disable "Per Client Throttling"(the default value)
since now only one client IP.
 

Monarobase

Well-Known Member
#27
Thanks,

I'm not sure how to completly disable per client throttling.

In Server > Security I've got 5 sections :

File Access
Per Client Throttling
External Application Ressource
Access Denied Directories
Access Control

Inside Per Client Throttling I've got :

Static Requests/second (number >= 0)
Dynamic Requests/second (number >= 0)
Outbound Bandwidth (bytes/sec)
Inbound Bandwidth (bytes/sec)
Connection Soft Limit
Connection Hard Limit
Block Bad Request
Grace Period (sec)
Banned Period (sec)

Is it one of these settings I need to change to completly disable it ?

Thanks !
 

NiteWave

Administrator
#28
Static Requests/second :0
Dynamic Requests/second :0
Outbound Bandwidth (bytes/sec):0
Inbound Bandwidth (bytes/sec):0
Connection Soft Limit: very large number
Connection Hard Limit: very large number

will disable the Throttling
 

NiteWave

Administrator
#30
1)any improvement regarding the error rate before and after the change?

2)any URL pattern or method(GET or POST) which is easier to trigger the error or just random ?

3)will the same error happen between T-WAF proxy and other web server(apache, nginx etc)?

again, it'll be much helpful if can work out a test case which can easily reproduce the error.
 

Monarobase

Well-Known Member
#31
1)any improvement regarding the error rate before and after the change?
No yesterday I got 10 errors between 0am and 7 pm
Today I got 9 errors from 0am and 7 pm

2)any URL pattern or method(GET or POST) which is easier to trigger the error or just random ?

It seems random, I don't have many POST requests so it's not likley to happen with a POST request. It happens randomly on images / html / PHP requests. (about 1 for every 1000 requests)

3)will the same error happen between T-WAF proxy and other web server(apache, nginx etc)?

An ASL developper has just told me that they have never seen this issue before and protect nginx servers without issues. Apache servers generaly integrate mod_security directly so they don't need the T-WAF.

again, it'll be much helpful if can work out a test case which can easily reproduce the error
Yes it would, but I've even tried benchmarking (siege) it without managing to create an error. It doesn't seem to happen so much when the server is under more stress.

Apache docs mention a race condition :

http://httpd.apache.org/docs/2.2/en/mod/mod_proxy_http.html

proxy-initial-not-pooled
If this variable is set no pooled connection will be reused if the client connection is an initial connection. This avoids the "proxy: error reading status line from remote server" error message caused by the race condition that the backend server closed the pooled connection after the connection check by the proxy and before data sent by the proxy reached the backend. It has to be kept in mind that setting this variable downgrades performance, especially with HTTP/1.0 clients.
 
Last edited:

Monarobase

Well-Known Member
#33
so can you set the environment variable on T-WAF ?
I'm giving it a try to see if it makes any difference. I will let you know if it removes the errors.

ASL said I should set the T-WAF (mod_security) to enforce non compressed data because this could cause issues and also prevent mod_security from analysing the data transmitted.

I've given this (disabling compression) a try but it doesn't help either (I still get errors). I think I will activate gzip compression in the T-WAF so it can analyze the data given by litespeed and see what the performance impact is.

I wil reactive keep-alive once I've finished the test with
Code:
setEnv proxy-initial-not-pooled 1
 
Last edited:

Monarobase

Well-Known Member
#34
setEnv proxy-initial-not-pooled 1

solves the problem

I've now got to test compressing with mod_deflate to see how much latency it adds. Otherwise I think I will just tell my customers they can't compress their data.
 
Top