Anti-DDoS Blocked IP

A

anything

Well-Known Member
#1
#1
Hi.

I've moved some of our webservers to litespeed but I've stumbled upon across two problems already.

Ive got litespeed behind varnish reverse proxy that is updating the x-forwarded-for header
litespeed is correctly extracting the x-forwarded-for IP, php reports it correctly
except this doesnt seem to work when it comes to the ddos protection

for some reason one of our websites is getting the errors:
Code: 
2011-06-07 22:58:38.996 [INFO] [xxx.xxx.xxx.xxx:57876-0] Status 400: Unexpected request body 330 bytes for request: /api!
2011-06-07 22:58:39.025 [INFO] [xxx.xxx.xxx.xxx:58771-0] Status 400: Unexpected request body 329 bytes for request: /api!
2011-06-07 22:58:39.093 [INFO] [xxx.xxx.xxx.xxx:58773-0] Status 400: Unexpected request body 326 bytes for request: /api!
2011-06-07 22:58:39.162 [INFO] [xxx.xxx.xxx.xxx:58777-0] Status 400: Unexpected request body 330 bytes for request: /api!
2011-06-07 22:58:39.231 [INFO] [xxx.xxx.xxx.xxx:58783-0] Status 400: Unexpected request body 330 bytes for request: /api!
2011-06-07 22:58:39.302 [INFO] [xxx.xxx.xxx.xxx:58785-0] Status 400: Unexpected request body 329 bytes for request: /api!
2011-06-07 22:58:39.371 [INFO] [xxx.xxx.xxx.xxx:58787-0] Status 400: Unexpected request body 329 bytes for request: /api!
2011-06-07 22:58:39.441 [INFO] [xxx.xxx.xxx.xxx:58791-0] Status 400: Unexpected request body 326 bytes for request: /api!
2011-06-07 22:58:39.535 [INFO] [xxx.xxx.xxx.xxx:58793-0] Status 400: Unexpected request body 327 bytes for request: /api!
2011-06-07 22:58:39.606 [INFO] [xxx.xxx.xxx.xxx:58800-0] Status 400: Unexpected request body 330 bytes for request: /api!
2011-06-07 22:58:39.739 [INFO] [xxx.xxx.xxx.xxx:57915-0] Status 400: Unexpected request body 329 bytes for request: /api!
and immediately following I find that the reverse proxy's ip has been blocked instead.

i've tried changing the client throttling settings. increased the max url and header sizes. added its ip to the access control list with T for trusted. but nothing seems to prevent the error.
i cannot work out what is causing the error or why. not enough information is provided.


even worse, is that because of the errors litespeed is assuming a dos attack, and is blocking it. but its blocking the wrong ip. it blocks the reverse proxy instead!
ive had to bypass varnish specifically for this site, because nobody can get to the site as soon as it errors like this.

naturally you cannot trust a dos attacker's x-forwarded-for header, so i can understand why the wrong ip might get blocked here
im also uncomfortable trusting anyone but my own proxy's x-forwarded-for header. this is quite a security problem in my opinion.
lsws needs to allow input of a list of ip's to trust to extract x-forwarded-for, and ignore it from any other ip.
this will also mean you could also trust the x-forwarded-for ip for ddos blocking.
 
Last edited:
M

mistwang

LiteSpeed Staff
#2
#2
You should get rid of varnish and use LiteSpeed cache instead, it works equally well if not much better than varnish.
You cannot expect anti-DDoS to work when you have varnish running in front of LiteSpeed. The reason for blocking reverse proxy IP is that LSWS do not even able to parse the "X-Forwarded-For" header due to the nature of "400 bad request".

It is not a bad idea to have a list of IP to trust "x-forwarded-for" header. we will do that. however, it wont help in your case.

And with reverse proxy in front, it always forward the requests from botnet no matter LSWS recognize the correct botnet IP or not, it defeat the purpose of the anti-DDoS feature.
 
A

anything

Well-Known Member
#3
#3
hi mistwang, thanks for your reply.

I just do not have enough control over litespeed cache for my liking, it has about 5 tickbox options, whereas my varnish config is 900 lines long. to to be fair theres quite a few lines of comments there, but you get my point. im sure the caching feature is quite well suited for a less complex environment.

If the headers arent corrupt will it extract the x-forwarded-for and block the correct IP? The client connection throttling features are nice, but im afraid of it blocking our proxies instead. Just in case i need to, how do i disable the anti-ddos feature?
another suggestion, would be to never block ip's that are listed as trusted in the security access control

I look forward to a feature to restrict the use of x-forwarded-for. thanks for that.


by chance i just caught the problem by watching tcpdump. the client appears to be using a custom written php script to request the api that does:
GET /api HTTP/1.0
User-Agent: ...etc...
Referer: ...etc...
Content-Type: text/xml
Host: sitename.com
Content-Length: 327

...etc...
Click to expand...
although apache was handing these bad requests fine it appears litespeed is more picky. in varnish ive been able to modify their requests as they came through the proxy and change them to a POST. but i also discovered i needed to change the content-type to application/x-www-form-urlencoded before litespeed would accept them

so in the end it was a simple corrupt http request, just very difficult to find out what it actually was and why!
 
Last edited:
K

kuffs

New Member
#5
#5
[solved] LiteSpeed blocking the load balancer

I am also having a similar problem. I am using a proxying load-balancer to balance between two hosts. Some script must be navigating the site with a slightly out-of-spec query

Code: 
HOST 1
-------------
2011-07-25 13:58:06.992	INFO	[LOAD_BALANCER_IP:47164-0] Status 400: Bad HTTP protocol string: /?type=l
2011-07-25 13:58:06.992	NOTICE	[LOAD_BALANCER_IP:47164-0] too many bad requests, block.
2011-07-25 13:58:06.992	NOTICE	[LOAD_BALANCER_IP] bot detected, close connection!

HOST 2
-------------
2011-07-25 13:43:58.187	INFO	[LOAD_BALANCER_IP:1293-0] Status 400: Bad request method: 00
2011-07-25 13:43:58.187	NOTICE	[LOAD_BALANCER_IP:1293-0] too many bad requests, block.
2011-07-25 13:43:58.187	NOTICE	[LOAD_BALANCER_IP] bot detected, close connection!
This ends up being a quite effective DoS attack when LiteSpeed ends up blocking the load balancer on both hosts.

I've already added the load balancer to the trusted IPs at the server level on both hosts.

Code: 
HOST1
-------------
LOAD_BALANCER_IPT, ALL

HOST2
-------------
LOAD_BALANCER_IPT, ALL
(IP addresses sanitized for privacy)

But it doesn't seem to work. Both hosts will eventually end up getting blocked and that results in a 500 at the site level.

I've tried setting the ban period to 0 seconds, but it gets flagged as an invalid value and instead runs at 60. I've also tried raising the grace period to an hour and it doesn't seem to have an effect. I've also tried raising the security limits on connections to a million without effect.

I realize that DDoS protection is one of the main features of your product but it is honestly not one I need. Not only because that feature ends up causing a DoS itself, but because I have other protections in place. Please advise me how to effectively disable the DDoS protection for that host.

Thanks
 
Last edited by a moderator:
W

webizen

Well-Known Member
#6
#6
Latest 4.1.2 add a safeguard to prevent Load Balancer IP from being blocked if its IP is already in trusted list.

To disable DDoS protection, set 0 for static requests/sec, dynamic request/sec, outbound bw, inbound bw in Per Client Throttling.
 
K

kuffs

New Member
#7
#7
Thank you webizen. I had forgotten to mention that we're running 4.1.1-ent. I'll give the upgrade a shot and see how it goes.

Thanks again :)
 
I

IrPr

Well-Known Member
#9
#9
webizen said:
Latest 4.1.2 add a safeguard to prevent Load Balancer IP from being blocked if its IP is already in trusted list.

To disable DDoS protection, set 0 for static requests/sec, dynamic request/sec, outbound bw, inbound bw in Per Client Throttling.
Click to expand...
where is the trusted list?
 
K

kuffs

New Member
#10
#10
IrPr said:
where is the trusted list?
Click to expand...
Configuration -> Server -> Security

Down at the bottom there is field for Access List. Add your trusted IPs or wildcards with a 'T' appended to them, as described in the help text.


webizen, thanks again for your help. Looks like 4.1.2 fixed our issues with trusted hosts being blocked anyway.
 
You must log in or register to reply here.
Top