Under DDoS need assistance

[mistwang]

Usually, the dummy bots only request one URL, so the page is cached in memory by kernel, multiple CPU license wont cause higher disk I/O.
AIO will help when the I/O wait of your server is high.
I don't think CiscoGuard really help on this, so it is better off not using it.

[anewday]

How is your server coping with the attacks now?

[Bono]

It would be cool if LiteSpeed could work with CSF in banning clients that are establishing too many connections.

[MikeDVB]

Originally Posted by anewday

How is your server coping with the attacks now?

We ended up null-routing one IP that was coming under about 15gbps of attack and we moved another client to a litespeed powered dedicated server that was receiving about 50mbps worth of SYN requests that Cisco Guard was failing to filter.

[anewday]

omg What was the load during the 50mbps sync attack and how many cpu license? I take it that lsws was able to fend it off.

[MikeDVB]

Originally Posted by anewday

omg What was the load during the 50mbps sync attack and how many cpu license? I take it that lsws was able to fend it off.

Load stayed low as I changed the site to 100% static but the issue was the kernel was ending up with a huge amount of orphaned tcp_sockets and eventually was running out of available connections.

[Bono]

Originally Posted by felosi

Install CSF and use the connection tracking feature, set ct_limit to 30 at 30 second interval. Litespeed cant do it all on its own. Also SYND is another helpful script but use only after csf has ran with ct for a while as it uses netstat and can lag your system

SYND - http://nix101.com/2007/07/21/syn-deflate/
csf - http://configserver.com/cp/csf.html

Also in lsws admin ? security

static requests -10
dynamic - 1

connection soft limit - 5
connection hard limit - 15

in my experience its best to disable smart keep alive, set keep alive timeout to 15, connection timeout to 30

Then you need to tune your tcp stack depending on how much ram you have.

Here is config I sue for servers with 8gb ram, its probably too much but ips should be getting blocked before using all this:
net.core.rmem_max = 184217728
net.ipv4.tcp_rmem = 4096 33554432 184217728
net.core.wmem_max = 184217728
net.ipv4.tcp_wmem = 4096 33554432 184217728
net.ipv4.tcp_mem = 8388608 16777216 184217728
net.core.optmem_max = 107108864
net.core.rmem_default = 10097152
net.core.rmem_default = 10097152
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_re cv = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_w ait = 60
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wa it = 30
kernel.panic=5
net.netfilter.nf_conntrack_max = 5048576
net.nf_conntrack_max = 5048576
net.ipv4.netfilter.ip_conntrack_max = 5048576

From your experience why we should turn off smart keep alive? When i'm under attack I see i'm not getting many syn requests so I cannot kill it with your tool, my main problem is TIME_WAIT, so i'm looking into to somehow fix that.

Usually my server load is around .30 but today i'm under attack so my load is pretty high.

1 CLOSING
14 ESTABLISHED
3 FIN_WAIT1
17 FIN_WAIT2
1 LAST_ACK
66 LISTEN
623 TIME_WAIT

01:05:15 up 227 days, 21:29, 1 user, load average: 2.71, 2.31, 2.14
So TIME_WAIT connections are causing load of web server.
How do you handle this kind of situations?
I have already limited dynamic requests, soft and hard limit but that doesn't help with many connections, I got 20-30 ips that are according to netstat connected more than 20-80 times in TIME_WAIT state.

[felosi]

Adjusting your time_wait timeouts in systctl as I listed above will help. But Time wait is not important, you can just set the timeout lower for that. You dont need to track it.

I am also currently soliciting developers to work on a new idea I have, what we came up with was BARF - Block Apache Request Floods - This is for http get style attacks - http://nix101.com/2009/09/04/new-bar...or-get-floods/
or just www/nix101.com/barf for direct link.

Also
1 CLOSING
14 ESTABLISHED
3 FIN_WAIT1
17 FIN_WAIT2
1 LAST_ACK

Does not look like attack to me, even with the amount of time waits you have. If your server goes down udner those conditions something is very wrong, check dmesg for packet drop errors and such

[Bono]

Originally Posted by felosi

Adjusting your time_wait timeouts in systctl as I listed above will help. But Time wait is not important, you can just set the timeout lower for that. You dont need to track it.

I am also currently soliciting developers to work on a new idea I have, what we came up with was BARF - Block Apache Request Floods - This is for http get style attacks - http://nix101.com/2009/09/04/new-bar...or-get-floods/
or just www/nix101.com/barf for direct link.

Also
1 CLOSING
14 ESTABLISHED
3 FIN_WAIT1
17 FIN_WAIT2
1 LAST_ACK

Does not look like attack to me, even with the amount of time waits you have. If your server goes down udner those conditions something is very wrong, check dmesg for packet drop errors and such

It doesn't go down, just load is higher than usual, but when my sites are busy load peaks at 12 and this is how it looks like.

49 ESTABLISHED
25 FIN_WAIT1
17 FIN_WAIT2
3 LAST_ACK
66 LISTEN
2 SYN_RECV
1058 TIME_WAIT

Later on:
1 CLOSE_WAIT
1 CLOSING
76 ESTABLISHED
8 FIN_WAIT1
48 FIN_WAIT2
3 LAST_ACK
66 LISTEN
12 SYN_RECV
1736 TIME_WAIT
Is BARF replacement for your synd script?

[MikeDVB]

In a (long awaited) update - LiteSpeed was able to keep up however it wasn't able to do 100k requests/second + the normal daily requests that would ordinarily hit the server.

I moved this one site onto it's own dedicated server with LiteSpeed and it stayed online and very responsive although it was using around 2tb of bandwidth/day