[solved] DDoS - long requests

Discussion in 'General' started by IanD, May 6, 2014.

  1. IanD

    IanD Member

    Hi,

    We are currently on the receiving end of a distributed DDoS attached (luckily not very big though).

    They have the clients 'POST'ing to a URL that we have disabled access to via htaccess so it points to a 403 page.

    But even with this, it is seriously taking resources on our web servers so I'm looking at how we can free up the resources. On the Top50 Request sorted by Request Time, it is 100% full of this:

    5.122.160.760RB542.0173862/3129436479/-1*****V2-"POST /Uxi6h HTTP/1.1"
    5.162.184.1040RB441.1166686/3373023972/3972*****V2-"POST /Uxi6h HTTP/1.1"
    5.21.246.1330RB341.5291606/93643220/20*****V2-"POST /Uxi6h HTTP/1.1"
    188.140.140.560RB311.0297158/35148320/20*****V2-"POST /Uxi6h HTTP/1.1"
    146.185.39.300RB297.0131852/13931119/19*****V2-"POST /Uxi6h HTTP/1.1"

    Some requests are still processing after 500+ seconds it seems - how can stop this? this can't be normal and I can only presume an intentional part of the attack to consume server resources.

    Is there is a setting in LiteSpeed to say if the request is not complete after say 30 seconds, kill it?

    Thanks,

    Ian
     
  2. NiteWave

    NiteWave Administrator

    I think 2 related settings are:
    lsws web admin -> Server -> General -> External Application Abort: Yes
    lsws web admin -> Server -> Tuning -> Connection Timeout (secs): default is 300.
    you can try to reduce the timeout.
     
  3. mistwang

    mistwang LiteSpeed Staff

    Try a rewrite rule like

    RewriteRule Uxi6h - [E=blockbot:1]
     
  4. IanD

    IanD Member

    Thank you for your replies.

    I have modified our rewrite to for the 'blockbot' and our Connection Timeout was already only 30 seconds, External Application Abort seems to be yes by default.

    But we are still seeing:

    ClientKaMRIn/TotalOut/TotalVHostHandlerPRequest
    5.21.227.2300RB574.6215266/4061823903/3903V2-"POST /Uxi6h HTTP/1.1"
    146.185.36.1150RB414.5394318/42468316530/-1V2-"POST /Uxi6h HTTP/1.1"
    212.215.162.1490RB413.8175014/1850476526/6526V2-"POST /Uxi6h HTTP/1.1"
    91.217.185.850RB340.4592802/89890120/20V2-"POST /Uxi6h HTTP/1.1"
    82.178.131.320RB339.6370722/3754186625/-1V2-"POST /Uxi6h HTTP/1.1"
    46.152.206.140RB328.1221646/2756955226/5226V2-"POST /Uxi6h HTTP/1.1"
    188.139.162.1430RB312.3274984/5329575225/5225V2-"POST /Uxi6h HTTP/1.1"
     
  5. mistwang

    mistwang LiteSpeed Staff

    Please check log for message like
    at INFO level, if the rewrite rule works properly, otherwise, need to make sure the rewrite rule can block those URL.
     
  6. IanD

    IanD Member

    Ah yes - sorry there was an extra space in Rewrite rule, now it is working!

    Around 8k IPs blocked in the 1st minute.
     
  7. IanD

    IanD Member

    Even though we are now blocking all the IPs in LiteSpeed that request the URL, the request queue is still full of:

    85.154.38.180RB634.2276338/7007026589/6589V2-"POST /Uxi6h HTTP/1.1"
    212.126.103.1460RB361.3731366/73748320/20V2-"POST /Uxi6h HTTP/1.1"
    5.162.162.350RB318.2596966/5992524112/4112V2-"POST /Uxi6h HTTP/1.1"
    85.154.127.160RB311.0316398/7517446585/6585V2-"POST /Uxi6h HTTP/1.1"
    82.178.131.1840RB299.2156970/35827120/20V2-"POST /Uxi6h HTTP/1.1"

    should they no longer be there now?
     
  8. NiteWave

    NiteWave Administrator

    how about the access log? does they exist in access log as well ?
     
  9. IanD

    IanD Member

    Yes:

    149.255.200.58 - - [07/May/2014:15:43:35 +0000] "POST /Uxi6h HTTP/1.1" 403 4114 "-"
    94.98.101.213 - - [07/May/2014:15:43:33 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
    185.16.26.25 - - [07/May/2014:15:42:26 +0000] "POST /Uxi6h HTTP/1.1" 200 3948 "-"
    85.154.38.210 - - [07/May/2014:15:43:10 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
    85.154.189.198 - - [07/May/2014:15:43:30 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
    37.44.37.20 - - [07/May/2014:15:43:34 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
    37.237.106.34 - - [07/May/2014:15:43:33 +0000] "POST /Uxi6h HTTP/1.0" 403 3983 "-"
    85.154.39.247 - - [07/May/2014:15:43:30 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
    5.162.165.162 - - [07/May/2014:15:43:24 +0000] "POST /Uxi6h HTTP/1.1" 403 3983 "-"

    It looks like LiteSpeed is not blocking the request until it has finished (maybe?) and therefore the requests are using up all the resources and we have some requests lasting for 600+ seconds and the IP is not in the ban list.

    I presume they are added when finally the incoming request finishes?

    The amount of data they are posting is not too big, maybe 10-100kb each request but it's having a big impact.
     
  10. IanD

    IanD Member

    Hi,

    Sorry to bring up this old thread but we've recently disabled CloudFlare and now we have exactly the same problem as before.

    Is there a way (maybe something has changed in the last 2 years!) to block a POST request to a certain URL before the POST request has finished?

    At the moment it looks like the bot posts 100Kb data to the specific URL and when has finished transferring, LiteSpeed sends a 403 header.

    Is it possible to send that header at the very beginning of the request so the server don't have to deal with all that wasted bandwidth and resources keeping the connection open while the data is transferred?

    Blocking by IP unfortunately is not feasible due to the many many 1000s of IPs being used.

    Thanks,

    Ian
     
  11. mistwang

    mistwang LiteSpeed Staff

    You can customize a mod_security rule to detect that, use "drop" action, instead of "deny", then the connection and sub-subsequent connections from the same IP will be immediately dropped.
     
  12. IanD

    IanD Member

    Works perfectly thanks - my inbound bandwidth has dropped by 50%.
     

Share This Page