What settings for maximum speed on a large configuration ?

Discussion in 'Install/Configuration' started by Monarobase, Jan 21, 2013.

  1. NiteWave

    NiteWave Administrator

    during restart, please run
    #pstree -p

    then you'll get the idea how graceful restart works.

    you'll see 2 trees which root is litespeed process. with the info of "pstree -p", at least can know 503 error is under which tree -- new or old.
  2. Monarobase

    Monarobase New Member

    Sorry for the delay.

    Due to lack of time I installed Xcache 2.0.1 and have disabled the variable cache, hopeing that someone might debug and fix the issue without me reporting it…

    I've got one last (quite big) problem.

    I run Atomicorp ASL T-WAF in front of litespeed and sometimes I get an error saying from the T-WAF with the following message :

    ---
    Proxy Error

    The proxy server received an invalid response from an upstream server.
    The proxy server could not handle the request GET /XXX.html.

    Reason: Error reading from remote server

    Apache Server at www.XXXXXXXX.tld Port 30001
    ---

    ASL seem to be saying it's an issue with litespeed but litespeed doesn't log anything when the error happens and ASL T-WAF doesn't log much either (except that Litespeed gave an invalid response).

    Do you have any suggestions about how I could determine if it rearly is Litespeed giving a bad response or if it's an issue with the T-WAF.

    The error message only happens about 20 times a day so it's very difficult to catch and almost impossible to determine if Litespeed is actually giving a blank page or a strange result or if it's the T-WAF that's not responding fast enough for litespeed or the other way around.

    When I searched for these issues, alot of topics about Litespeed came up… with various other proxies, so it seems that litespeed could be for something in this issue.
  3. NiteWave

    NiteWave Administrator

  4. Monarobase

    Monarobase New Member

    Thanks I will give it a try :)

    We don't run the rules in litespeed but have ASL's proxy in front of litespeed.

    Running their hourly updated rules seems a bit dangerous directly in litespeed as the rules are only tested in ASL's modsecurity before being released and it seems that it's almost impossible for litespeed to follow these updates.

    ASL is however being helpfull on this issue. They helped me set up full modsecurity logging to try and see if there's something reported when this issue happens.

    I'll let you know if/when I get some more information.
  5. Monarobase

    Monarobase New Member

    I've got an update from ASL.

    ASL T-WAF runs an optimised version of Apache with Mod Security and Mod_proxy in front of litespeed. (note : this hardly adds any latency on our server, we get 15ms responses even with mod_security in T-WAF proxy).

    This is what is happening :

    I'm getting the following error from mod_proxy : "proxy: error reading status line from remote server"

    1) Client connects to the T-WAF (local reverse proxy)
    2) T-WAF checks connection is live
    3) Litespeed goes away
    4) T-WAF proxy doesn't get the data from litespeed

    Do you have any suggestions about how I could get litespeed to play well running behind a reverse proxy ?

    Thanks
  6. NiteWave

    NiteWave Administrator

    while no environment to experiment with this, it looks litespeed is behind apache and handle the requests passed from apache, but connection between apache and litespeed closed by litespeed in some cases.

    need tune
    1)lsws admin console -> Server -> Tuning -> Connection
    not sure if disable keep-alive will be a solution
    Max Keep-Alive Requests:0
    may need increase
    Connection Timeout (secs)
    as well

    2)lsws admin console -> Server -> Security:
    disable "Per Client Throttling"(the default value)
    since now only one client IP.
  7. Monarobase

    Monarobase New Member

    Thanks,

    I'm not sure how to completly disable per client throttling.

    In Server > Security I've got 5 sections :

    File Access
    Per Client Throttling
    External Application Ressource
    Access Denied Directories
    Access Control

    Inside Per Client Throttling I've got :

    Static Requests/second (number >= 0)
    Dynamic Requests/second (number >= 0)
    Outbound Bandwidth (bytes/sec)
    Inbound Bandwidth (bytes/sec)
    Connection Soft Limit
    Connection Hard Limit
    Block Bad Request
    Grace Period (sec)
    Banned Period (sec)

    Is it one of these settings I need to change to completly disable it ?

    Thanks !
  8. NiteWave

    NiteWave Administrator

    Static Requests/second :0
    Dynamic Requests/second :0
    Outbound Bandwidth (bytes/sec):0
    Inbound Bandwidth (bytes/sec):0
    Connection Soft Limit: very large number
    Connection Hard Limit: very large number

    will disable the Throttling
  9. Monarobase

    Monarobase New Member

    Hello,

    With throttling disabled and keep-alive at zero it still happens
  10. NiteWave

    NiteWave Administrator

    1)any improvement regarding the error rate before and after the change?

    2)any URL pattern or method(GET or POST) which is easier to trigger the error or just random ?

    3)will the same error happen between T-WAF proxy and other web server(apache, nginx etc)?

    again, it'll be much helpful if can work out a test case which can easily reproduce the error.
  11. Monarobase

    Monarobase New Member

    No yesterday I got 10 errors between 0am and 7 pm
    Today I got 9 errors from 0am and 7 pm

    2)any URL pattern or method(GET or POST) which is easier to trigger the error or just random ?

    It seems random, I don't have many POST requests so it's not likley to happen with a POST request. It happens randomly on images / html / PHP requests. (about 1 for every 1000 requests)

    3)will the same error happen between T-WAF proxy and other web server(apache, nginx etc)?

    An ASL developper has just told me that they have never seen this issue before and protect nginx servers without issues. Apache servers generaly integrate mod_security directly so they don't need the T-WAF.

    Yes it would, but I've even tried benchmarking (siege) it without managing to create an error. It doesn't seem to happen so much when the server is under more stress.

    Apache docs mention a race condition :

    http://httpd.apache.org/docs/2.2/en/mod/mod_proxy_http.html

    Last edited: Feb 1, 2013
  12. NiteWave

    NiteWave Administrator

    so can you set the environment variable on T-WAF ?

    the frontend and backend litespeed better to set to keep alive
    i.e., set litespeed
    Max Keep-Alive Requests
    to default value
  13. Monarobase

    Monarobase New Member

    I'm giving it a try to see if it makes any difference. I will let you know if it removes the errors.

    ASL said I should set the T-WAF (mod_security) to enforce non compressed data because this could cause issues and also prevent mod_security from analysing the data transmitted.

    I've given this (disabling compression) a try but it doesn't help either (I still get errors). I think I will activate gzip compression in the T-WAF so it can analyze the data given by litespeed and see what the performance impact is.

    I wil reactive keep-alive once I've finished the test with
    Code:
    setEnv proxy-initial-not-pooled 1
    Last edited: Feb 2, 2013
  14. Monarobase

    Monarobase New Member

    setEnv proxy-initial-not-pooled 1

    solves the problem

    I've now got to test compressing with mod_deflate to see how much latency it adds. Otherwise I think I will just tell my customers they can't compress their data.

Share This Page