Under DDoS need assistance

Discussion in 'General' started by MikeDVB, Jul 27, 2009.

  1. MikeDVB

    MikeDVB New Member

    I am seeing a HTTP used, 9,999 connections 1 free but nearly 0 requests per second

    [​IMG]

    Suggestions, please?

    I really need to set up LSWS to handle this if at all possible.

    Attached Files:

    Last edited: Jul 27, 2009
  2. MikeDVB

    MikeDVB New Member

    Well the attacker has stopped the attack - but it would still be good information to know in the future.
  3. mistwang

    mistwang LiteSpeed Staff

  4. MikeDVB

    MikeDVB New Member

    I already did all of that mistwang, do you honestly think I would be posting here about it if I hadn't followed that already?

    I was still seeing 10,000 connections used, 0 available and 0requests/second.
  5. mistwang

    mistwang LiteSpeed Staff

    It is possible if the botnet is large enough, for example, 1000 bots, each bots make 10 connections, that will be 10,000 connections, and it still not reach the threshold to ban a IP yet.

    If it happen again, you should check your netstat output see if it is true.

    What can you do if it is true? You have to lower the threshold to make LSWS able to detect those IP abusing you server, it may cause some false alarm, but it better then being taken down completely.

    If your server has enough memory, you can raise the limit of "Max Connections" of LSWS under server tuning tab.
  6. anewday

    anewday Moderator

    Something is not right, why is it showing so much idle connections? What's the difference between idle and free?
  7. MikeDVB

    MikeDVB New Member

    The botnet was 15,000+

    Which threshold should I lower specifically?

    It's set to the 10,000 can I Go higher?
  8. anewday

    anewday Moderator

    What was the load during the attack?
  9. auser

    auser Super Moderator

    please share what's the output of:
    (1)netstat -nt|grep ESTABLISHED|wc
    (2)netstat -nt|awk '{print $5;}'|awk -F ':' '{print $1;}'|sort|uniq -c|sort -r|head

    to set, go web admin console:
    Configuration->Server->Security->Per Client Throttling:
    (for example)
    Static Requests/second:10
    Dynamic Requests/second:2
    Outbound Bandwidth (bytes/sec):4k
    Inbound Bandwidth (bytes/sec):1k
    Connection Soft Limit:5
    Connection Hard Limit:20
    Grace Period (sec):15
    Banned Period (sec):60
  10. MikeDVB

    MikeDVB New Member

    root@atlantis [/]# netstat -nt|grep ESTABLISHED|wc
    426 2556 37914

    9 88.234.114.182
    93 89.77.8.199
    9 222.127.223.74
    9 221.126.69.111
    9 203.176.146.9
    9 202.176.164.237
    9 190.229.83.224
    9 189.194.102.96
    9 189.168.54.29
    9 125.26.12.233
  11. MikeDVB

    MikeDVB New Member

    it's around 2,000 connections total, but each one is making hundreds of requests per second.

    I've blocked a lot with iptables -drop but still it's hurting
  12. auser

    auser Super Moderator

    in this case,the example setting
    Static Requests/second:10
    Dynamic Requests/second:2
    is just what you want.

    the 2nd command output didn't look right on your box. change to
    netstat -nt|awk '{print $5;}'|awk -F ':' '{print $1;}'|sort|uniq -c|sort -nr|head

    will display the top 10 IPs which have most tcp connections with your host.
  13. mistwang

    mistwang LiteSpeed Staff

    You can check /usr/local/apache/log/error_log, there will be many entries like a IP hit the connection hard limit or the soft limit.
    You can write a script to parse the list of IP that hits the hard limit, then block them with iptables.
  14. felosi

    felosi New Member

    Install CSF and use the connection tracking feature, set ct_limit to 30 at 30 second interval. Litespeed cant do it all on its own. Also SYND is another helpful script but use only after csf has ran with ct for a while as it uses netstat and can lag your system

    SYND - http://nix101.com/2007/07/21/syn-deflate/
    csf - http://configserver.com/cp/csf.html

    Also in lsws admin ? security

    static requests -10
    dynamic - 1

    connection soft limit - 5
    connection hard limit - 15

    in my experience its best to disable smart keep alive, set keep alive timeout to 15, connection timeout to 30

    Then you need to tune your tcp stack depending on how much ram you have.

    Here is config I sue for servers with 8gb ram, its probably too much but ips should be getting blocked before using all this:
    net.core.rmem_max = 184217728
    net.ipv4.tcp_rmem = 4096 33554432 184217728
    net.core.wmem_max = 184217728
    net.ipv4.tcp_wmem = 4096 33554432 184217728
    net.ipv4.tcp_mem = 8388608 16777216 184217728
    net.core.optmem_max = 107108864
    net.core.rmem_default = 10097152
    net.core.rmem_default = 10097152
    net.ipv4.tcp_max_syn_backlog = 2048
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 30
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 60
    net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 30
    kernel.panic=5
    net.netfilter.nf_conntrack_max = 5048576
    net.nf_conntrack_max = 5048576
    net.ipv4.netfilter.ip_conntrack_max = 5048576
    Last edited: Jul 28, 2009
  15. MikeDVB

    MikeDVB New Member

    Very helpful, the server actulaly has 4gb so I will see what I can do with this to tweak it for the server.

    My biggest issue is that after an attack is so large I get tossed onto CiscoGuard automatically so then there are only a few hand full of IPs that are making lots of connections (CiscoGuard Proxy IPs) - so if I start limiting connections by IP I run into issues because there is no way to differentiate between the attacker and legitimate traffic from CiscoGuard.

    Sort of a rock and a hard place, the system that is supposed to protect me makes it nearly impossible for me to handle a DDoS attack that simulates real traffic. The flow of packets is enough to trigger CiscoGuard but then CiscoGuard does nothing against a syn flood for example.
    Last edited: Jul 28, 2009
  16. MikeDVB

    MikeDVB New Member

    The server is seeing upwards of 100,000+ HTTP requests per second. Although LiteSpeed is very fast and efficient I don't know that it can handle this sort of load.

    I've done everything I can to optimize it however 10+megabits of pure HTTP requests seems to just bring it down in a hurry.
  17. anewday

    anewday Moderator

    Wow, never thought litespeed can scale this much. :D
  18. MikeDVB

    MikeDVB New Member

    It was handling upwards of 1500~2,000 requests per second pretty well but beyond that the server just begins to choke.

    I changed the site being requested to a static page and that reduced the load substantially but I am only running a 1-cpu license so I'm not sure if that is really limiting my abilities.

    I would think on a 2 or 4 cpu license it would simply cause more Disk I/O wait and more problems.
  19. anewday

    anewday Moderator

    George have said that a 2-cpu license handles ddos better than a 1-cpu license. It's better for the I/O.
  20. MikeDVB

    MikeDVB New Member

    Yeah, but is 2cpu going to make a difference over 1cpu with 100,000 requests per second?

    I would think at best doubling the CPU/Process would double it's ability to handle requests but there is the additional overhead of having a secondary process and then I/O blocking to look into.

    I am curious about AIO, but not found anything useful about it.

Share This Page