Under DDoS need assistance

Discussion in 'General' started by MikeDVB, Jul 27, 2009.

  1. mistwang

    mistwang LiteSpeed Staff

    Usually, the dummy bots only request one URL, so the page is cached in memory by kernel, multiple CPU license wont cause higher disk I/O.
    AIO will help when the I/O wait of your server is high.
    I don't think CiscoGuard really help on this, so it is better off not using it.
  2. anewday

    anewday Moderator

    How is your server coping with the attacks now?
  3. Bono

    Bono New Member

    It would be cool if LiteSpeed could work with CSF in banning clients that are establishing too many connections.
  4. MikeDVB

    MikeDVB New Member

    We ended up null-routing one IP that was coming under about 15gbps of attack and we moved another client to a litespeed powered dedicated server that was receiving about 50mbps worth of SYN requests that Cisco Guard was failing to filter.
  5. anewday

    anewday Moderator

    omg :eek: What was the load during the 50mbps sync attack and how many cpu license? I take it that lsws was able to fend it off.
    Last edited: Aug 6, 2009
  6. MikeDVB

    MikeDVB New Member

    Load stayed low as I changed the site to 100% static but the issue was the kernel was ending up with a huge amount of orphaned tcp_sockets and eventually was running out of available connections.
  7. Bono

    Bono New Member

    From your experience why we should turn off smart keep alive? When i'm under attack I see i'm not getting many syn requests so I cannot kill it with your tool, my main problem is TIME_WAIT, so i'm looking into to somehow fix that.

    Usually my server load is around .30 but today i'm under attack so my load is pretty high.

    1 CLOSING
    14 ESTABLISHED
    3 FIN_WAIT1
    17 FIN_WAIT2
    1 LAST_ACK
    66 LISTEN
    623 TIME_WAIT

    01:05:15 up 227 days, 21:29, 1 user, load average: 2.71, 2.31, 2.14
    So TIME_WAIT connections are causing load of web server.
    How do you handle this kind of situations?
    I have already limited dynamic requests, soft and hard limit but that doesn't help with many connections, I got 20-30 ips that are according to netstat connected more than 20-80 times in TIME_WAIT state.
  8. felosi

    felosi New Member

    Adjusting your time_wait timeouts in systctl as I listed above will help. But Time wait is not important, you can just set the timeout lower for that. You dont need to track it.

    I am also currently soliciting developers to work on a new idea I have, what we came up with was BARF - Block Apache Request Floods - This is for http get style attacks - http://nix101.com/2009/09/04/new-barf-update-script-for-get-floods/
    or just www/nix101.com/barf for direct link.

    Also
    1 CLOSING
    14 ESTABLISHED
    3 FIN_WAIT1
    17 FIN_WAIT2
    1 LAST_ACK

    Does not look like attack to me, even with the amount of time waits you have. If your server goes down udner those conditions something is very wrong, check dmesg for packet drop errors and such
  9. Bono

    Bono New Member

    It doesn't go down, just load is higher than usual, but when my sites are busy load peaks at 12 and this is how it looks like.

    49 ESTABLISHED
    25 FIN_WAIT1
    17 FIN_WAIT2
    3 LAST_ACK
    66 LISTEN
    2 SYN_RECV
    1058 TIME_WAIT

    Later on:
    1 CLOSE_WAIT
    1 CLOSING
    76 ESTABLISHED
    8 FIN_WAIT1
    48 FIN_WAIT2
    3 LAST_ACK
    66 LISTEN
    12 SYN_RECV
    1736 TIME_WAIT
    Is BARF replacement for your synd script?
    Last edited: Sep 8, 2009
  10. MikeDVB

    MikeDVB New Member

    In a (long awaited) update - LiteSpeed was able to keep up however it wasn't able to do 100k requests/second + the normal daily requests that would ordinarily hit the server.

    I moved this one site onto it's own dedicated server with LiteSpeed and it stayed online and very responsive although it was using around 2tb of bandwidth/day :)
  11. anewday

    anewday Moderator

    :eek: What are the specs of the quad core server?
  12. felosi

    felosi New Member

    If it is getting GET attacks you should really try BARF. It works good, you just have to manually specify the requests the attackers are made. Sometimes they will change them when they realize whats going on but I handle 5 servers with sites getting ddosed all the time and I am able to keep up with them on at least 2 sites I can think of now.

    Eventually when i find the right programmer I am going to get it programmed where it detects and blocks on ANY reptitive GET. Now that will be cool. I think you would still have to specify domain to watch for because it would just be too much for it to be checking all domlogs at once.

    With BARF, SYND, Csf w/connection tracking and litespeed with proper settings you can pretty much handle anything that gets by your network filters as long as it doesnt consume your pipe.

    But there is one bad thing about litespeed I noticed. For example for a lot of these attacking bots they was not giving user agent so we added some mod sec and rewrite rules to stop this. Now all attacking bots get a 403 yet it is totally draining my bandwidth as if it was really getting the image. I think Ill make a post about this here somewhere. Has anyone else noticed this? Also even though attacking bots are getting 403 the lsphp and lshttpd processes are going crazy. I guess where its all based on php - the error pages and all.
  13. anewday

    anewday Moderator

    What's SYND? Could you share those modsec and rewrite rules? :)
  14. MikeDVB

    MikeDVB New Member

    Just a 2.4ghz Quad with 4gb ram but the storage was network attached SCSI (so it could keep up).
  15. Bono

    Bono New Member

    This is probably related to my post
    http://www.litespeedtech.com/support/forum/showthread.php?t=3387

    Please check which process is overloading your server, on my server system cannot find files and it takes most of the resources and overloads the server.

Share This Page