Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
litespeed_wiki:config:mitigating-ddos-attacks [2018/12/13 18:40]
Jackson Zhang [If Bad requests hitting the Same URL, Block that URL]
litespeed_wiki:config:mitigating-ddos-attacks [2020/01/07 17:21] (current)
Lisa Clarke [Never set **Use Client IP in Header** to ''Yes'']
Line 1: Line 1:
-====== ​How Can I Mitigate Denial of Service (DoSand Distributed Denial of Service (DDoSAttacks======+====== ​Mitigating ​DoS and DDoS Attacks ======
  
-LiteSpeed Web Server provides several features aimed at reducing and even eliminating the impact of HTTP-level DoS and DDoS attacks. You can either use LSWS built-in features or third party ModSecurity rules such as Owasp, Atomicorp, Comodo and CloudLinux Imunify360. The following LSWS built-in configuration settings will help mitigate such attacks.+LiteSpeed Web Server provides several features aimed at reducing and even eliminating the impact of HTTP-level ​Denial of Service (DoSand Distributed Denial of Service (DDoSattacks. You can either use LSWS built-in features or third party ModSecurity rules such as Owasp, Atomicorp, Comodo and CloudLinux Imunify360. The following LSWS built-in configuration settings will help mitigate such attacks.
  
-===== Enabling LiteSpeed cache to increase server heavy traffic handling capacity ​===== +===== Enable reCAPTCHA feature ​===== 
-LiteSpeed cache is designed ​to boost site load speed and handle much more high trafficLiteSpeed cache cannot block any DDos attackhowever, it can help the server ​to handle hundreds/​Thousands ​of times more requests/​second which will definitely help to mitigate DDoS attack ​Check ​[[litespeed_wiki:​cache|here]] to learn how to enable ​lscacheFor cpanel server, you can use [[litespeed_wiki:​cpanel:​whm-plugin-lscwp-management|LiteSpeed ​cache manager]] ​to massive enable cache as one click+One of the most effective methods to mitigate DoS and DDoS attacks ​is to enable the reCAPTCHA feature. reCAPTCHA is a free service from Google that helps protect websites from spam and abuseA “CAPTCHA” is a turing test to tell human and bots apart. It is easy for humans to solvebut hard for bots and other malicious software ​to figure out. The reCAPTCHA feature is supported as of LSWS 5.4RC1 and laterPlease see [[litespeed_wiki:​config:​recaptcha|these instructions]] to enable ​reCAPTCHA. 
 +===== Enable ​LiteSpeed ​Cache ===== 
 +Enabling LiteSpeed Cache will increase the server'​s capacity ​to handle heavy traffic.
  
-===== LSWS built-in Per Client Throttling feature ​to block bad IPs =====+LiteSpeed Cache is designed ​to improve site load speed and handle increasingly high traffic. LiteSpeed Cache cannot ​block any DDos attack, however, it can help the server to handle hundreds or thousands of times more requests/​second. This definitely helps lessen the impact of DDoS attacks. [[litespeed_wiki:​cache|See these instructions]] to learn how to enable LSCache. For a cPanel server, you can use [[litespeed_wiki:​cpanel:​whm-plugin-lscwp-management|LiteSpeed Cache manager]] to mass enable LSCache in one click. ​
  
-The **Configuration > Server > Security configurations > Per Client Throttling** ​provides ​several configuration settings to limit the request, bandwidth, and connection rate per remote IP address. ​+===== Use Per-Client Throttling ===== 
 +LiteSpeed Web Server includes a built-in Per-Client Throttling feature which allows you to block bad IPs. 
 + 
 +Navigate to **Configuration > Server > Security configurations > Per Client Throttling** ​to find several configuration settings ​that you can use to limit the request, bandwidth, and connection rate per remote IP address. ​
  
 ==== Request Throttling ==== ==== Request Throttling ====
Line 15: Line 20:
 The server allows setting separate bandwidth limits for inbound and outbound traffic. The server allows setting separate bandwidth limits for inbound and outbound traffic.
   * Bandwidth numbers will be rounded up in 4KB increments.   * Bandwidth numbers will be rounded up in 4KB increments.
-  * Set to "0" ​to disable throttling.+  * Set to ''​0'' ​to disable throttling.
   * The **Outbound Bandwidth** limit allows serving more unique clients and prevents limited network bandwidth from getting used up by a small number of clients with fast network connections.   * The **Outbound Bandwidth** limit allows serving more unique clients and prevents limited network bandwidth from getting used up by a small number of clients with fast network connections.
 ==== Connection Throttling ==== ==== Connection Throttling ====
 These settings control concurrent connections coming from one client (IP address) and guard against DoS attacks. These settings control concurrent connections coming from one client (IP address) and guard against DoS attacks.
   * **Connection Hard Limit** controls how many concurrent connections are allowed from one IP address. If an IP reaches the hard connection limit, the web server will immediately close newly accepted connections from that IP address, and move on to pending connections from different IP addresses. As almost all web browsers support keep-alive/​persistent connections (multiple requests pipelined through one connection),​ the number of connections required in normal browsing is very small. Typically, one connection is enough, but some web browsers try to establish additional connections to speed up downloading. Allowing 4 to 10 connections from one IP is recommended. Less than that will probably affect normal web services.   * **Connection Hard Limit** controls how many concurrent connections are allowed from one IP address. If an IP reaches the hard connection limit, the web server will immediately close newly accepted connections from that IP address, and move on to pending connections from different IP addresses. As almost all web browsers support keep-alive/​persistent connections (multiple requests pipelined through one connection),​ the number of connections required in normal browsing is very small. Typically, one connection is enough, but some web browsers try to establish additional connections to speed up downloading. Allowing 4 to 10 connections from one IP is recommended. Less than that will probably affect normal web services.
-  * Use **Connection Soft Limit**, **Grace Period**, and **Banned Period** to spot and mitigate abusers: An IP address that stays over the soft limit for the length of the grace period will be banned for the length of time set in **Banned Period**. This is a good way to identify IPs that should be added to the **Denied List**. **Note**: The number of connections can temporarily exceed the soft limit during the grace period, as long as it is under the hard limit. After the grace period, if it is still above the soft limit, then no more connections will be allowed from that IP for duration of the banned period. +  * Use **Connection Soft Limit**, **Grace Period**, and **Banned Period** to spot and mitigate abusers: An IP address that stays over the soft limit for the length of the grace period will be banned for the length of time set in **Banned Period**. This is a good way to identify IPs that should be added to the **Denied List**. ​ 
 +   
 +**Note**: The number of connections can temporarily exceed the soft limit during the grace period, as long as it is under the hard limit. After the grace period, if it is still above the soft limit, then no more connections will be allowed from that IP for duration of the banned period.
  
 +==== Example ====
 Default Settings: Default Settings:
  
Line 30: Line 37:
 {{ :​litespeed_wiki:​config:​security-perclientthrottling-apply.png?​800 |}} {{ :​litespeed_wiki:​config:​security-perclientthrottling-apply.png?​800 |}}
  
-  ​Static Requests/​second 40 +**Static Requests/​second** = ''​40''​ 
-  Dynamic Requests/​second 2 +**Dynamic Requests/​second** = ''​2''​ 
-  Outbound Bandwidth (bytes/​sec) 0 +**Outbound Bandwidth (bytes/sec)** = ''​0''​ 
-  Inbound Bandwidth (bytes/​sec) 0 +**Inbound Bandwidth (bytes/sec)** = ''​0''​ 
-  Connection Soft Limit 15 +**Connection Soft Limit** = ''​15''​ 
-  Connection Hard Limit 20 +**Connection Hard Limit** = ''​20''​ 
-  Block Bad Request Yes +**Block Bad Request** = ''​Yes''​ 
-  Grace Period (sec) 15 +**Grace Period (sec)** = ''​15''​ 
-  Banned Period (sec) 60+**Banned Period (sec)** = ''​60''​
  
 Explanation:​ An IP that has established more than 20 connections with the web server, or has established over 15 connections of over 15 seconds (the grace period), is treated as a DoS-attacker. The server will ban the IP for 60 seconds and record a log entry in the error log file. To exclude any IP from the client throttle limits (and bypass DDoS detection), add the IP with a trailing '​T'​ (aka trusted) in **Allowed List** (**WebAdmin Console > Server > Security > Access Control**). Explanation:​ An IP that has established more than 20 connections with the web server, or has established over 15 connections of over 15 seconds (the grace period), is treated as a DoS-attacker. The server will ban the IP for 60 seconds and record a log entry in the error log file. To exclude any IP from the client throttle limits (and bypass DDoS detection), add the IP with a trailing '​T'​ (aka trusted) in **Allowed List** (**WebAdmin Console > Server > Security > Access Control**).
  
-The hard limit can be adjusted based on an attacker'​s strategy. If the botnet is not very aggressive, you will need to lower the limit to //just// below their max connection per IP, to make sure it won't affect a regular user. If they only make very few connections per IP, do not use hard limit to detect them.+The hard limit can be adjusted based on an attacker'​s strategy. If the botnet is not very aggressive, you will need to lower the limit to //just// below their max connection per IP, to make sure it won't affect a regular user. If they only make very few connections per IP, do not use the hard limit to detect them.
  
 The blocked IPs can be found in the real-time-stats report. The blocked IPs can be found in the real-time-stats report.
-===== Max Request/​Response Settings ​to reduce memory usage =====+ 
 +===== Virtual Host-Level Bandwidth Throttling ===== 
 + 
 +LiteSpeed Web Server version 5.0 introduces virtual host-level bandwidth throttling. This can be thought of as an extension of LSWS's **Per Client Throttling settings** explained as above, which allow you to control the amount of stress a single IP can put on your server. Virtual host-level bandwidth throttling allows you to customize bandwidth throttling, in Apache configs, for particular virtual hosts through ''​MaxConnPerClient <​limit_for_connections>'',''​LargeFileLimit [Type] [Minimum Size in kilobytes] [Speed in bytes/​s]'',​ ''​BandWidth [Origin] [Speed in bytes/​s]''​ and ''​MinBandWidth all -1''​. 
 +. Please check [[litespeed_wiki:​config:​vhost-level_bw_throttling|here]] for details. 
 + 
 +===== Use Max Request/​Response Settings ===== 
 +You can take advantage of the Max Request/​Response Settings to reduce memory usage. 
 Under **Configuration > Server > Tuning**: Under **Configuration > Server > Tuning**:
   * Try to set **Max Request URL Length**, **Max Request Header Size**, **Max Request Body Size**, **Max Dynamic Response Header Size**, and **Max Dynamic Response Body Size** to values that go //just// above what you need to run your site. Getting these settings trimmed down will help identify attackers and reduce the amount of memory used when you do get attacked.   * Try to set **Max Request URL Length**, **Max Request Header Size**, **Max Request Body Size**, **Max Dynamic Response Header Size**, and **Max Dynamic Response Body Size** to values that go //just// above what you need to run your site. Getting these settings trimmed down will help identify attackers and reduce the amount of memory used when you do get attacked.
Line 52: Line 67:
 {{ :​litespeed_wiki:​config:​tuning-connections.png?​800 |}} {{ :​litespeed_wiki:​config:​tuning-connections.png?​800 |}}
  
-===== Increase Max Connection Settings ​to increase the capacity ​=====+===== Increase Max Connection Settings ===== 
 +Increasing the Max Connection Settings will increase capacity and allow you to mitigate attack without limiting yourself. 
 Default: ​ Default: ​
 {{ :​litespeed_wiki:​config:​tunning-max-connections.png?​700 |}} {{ :​litespeed_wiki:​config:​tunning-max-connections.png?​700 |}}
Line 61: Line 78:
 The number of connection on port 80 doesn'​t matter. As long as the service is up, you've won! The number of connection on port 80 doesn'​t matter. As long as the service is up, you've won!
  
-===== Manually ​block known bad IPs===== +===== Manually ​Block Known Bad IPs ===== 
-If you know an attacker'​s IP, You can block it. Under **Configuration > Server > Security**: ​+If you know an attacker'​s IP, you can block it. Under **Configuration > Server > Security**: ​
   * Block IPs that abuse your web server by listing them in the **Denied List** in the Access Control table.   * Block IPs that abuse your web server by listing them in the **Denied List** in the Access Control table.
  
-===== If Bad requests hitting the Same URL, Block that URL===== +===== Manually ​Block Target URLs ===== 
-If your server is flooded by hundreds of requests from different IPs but to the same URL, you can setup rules to block access to that URL.+If your server is flooded by hundreds of requests from different IPs but to the same URL, you can set up rules to block access to that URL.
  
-For example, in the Control ​panel environment, ​ to block all access to /foo/, in /​foo/​.htaccess of the targeted domain virtual host, place the following:+For example, in a control ​panel environment, ​ to block all access to ''​/foo/''​, in the ''​/​foo/​.htaccess'' ​of the targeted domain virtual host, place the following:
  
   RewriteEngine On   RewriteEngine On
   RewriteRule .* - [L,F]   RewriteRule .* - [L,F]
  
-In LSWS native mode, you can either use rewrirule ​rules as indicated above or native context configuration:​Create a context (**Configuration > Virtual Hosts > View/Edit > Context > Add > Type** = ''​Static''​) to block access to that URL. Set **Accessible** to ''​No''​ and the context URI to match or include the URL being attacked. If the server is pounded with requests for ''/​foo/​bar.html'',​ then adding a context with **Accessible** set to ''​No''​ and the URI set to ''/​foo/​bar.html''​ will block all of those requests. You can also set the context URI to ''/​foo/''​ to block requests to //all// URLs that start with ''/​foo/''​.+In LSWS native mode, you can either use rewrite ​rules as indicated aboveor native context configuration ​like so
  
-===== ModSecurity Rules ====+  - Create a context (**Configuration > Virtual Hosts > View/Edit > Context > Add > Type** ​''​Static''​) to block access to that URL.  
 +  - Set **Accessible** to ''​No''​ and the context URI to match or include the URL being attacked.  
 +   
 +If the server is pounded with requests for ''/​foo/​bar.html'',​ then adding a context with **Accessible** set to ''​No''​ and the URI set to ''/​foo/​bar.html''​ will block all of those requests. You can also set the context URI to ''/​foo/''​ to block requests to //all// URLs that start with ''/​foo/''​. 
 + 
 +===== Use ModSecurity Rules =====
 LSWS is campatible with the most common ModSecurity Rules, such as Owasp, Atomicorp, Comodo and CloudLinux Imunify360 etc. You can enable one of them on LSWS.  ​ LSWS is campatible with the most common ModSecurity Rules, such as Owasp, Atomicorp, Comodo and CloudLinux Imunify360 etc. You can enable one of them on LSWS.  ​
  
-===== LiteSpeed'​s Advanced Anti-DDoS Setup Service ==== +===== Order LiteSpeed'​s Advanced Anti-DDoS Setup Service ==== 
-If you need assistance configuring your site to mitigate attacks, check out [[https://​store.litespeedtech.com/​store/​knowledgebase.php?​action=displayarticle&​id=125|LiteSpeed'​s Advanced Anti-DDoS Setup Service]]. LiteSpeed Denial of Service Packet Filter Setup Service will fine-tune ​the anti-DDoS configuration and set up iptables to automatically block attacking IPs detected by the web server.+If you need assistance configuring your site to mitigate attacks, check out [[https://​store.litespeedtech.com/​store/​knowledgebase.php?​action=displayarticle&​id=125|LiteSpeed'​s Advanced Anti-DDoS Setup Service]]. LiteSpeed Denial of Service Packet Filter Setup Service will fine-tune ​your anti-DDoS configuration and set up iptables to automatically block attacking IPs detected by the web server.
  
-To order LiteSpeed Advanced Anti-DDos Setup Serviceplease check [[https://​store.litespeedtech.com/​store/cart.php?​gid=5|here]]+This is sufficient for many common attack scenarios. In cases of extreme attacksthis service will not be sufficient, and only custom ​hourly support may be appropriate.
  
 +In order to determine whether the service fits your needs, we will need to identify the type of attack your site is experiencing - for example, whether it is targeting layer 4 (IP/port) or 7 (HTTP/URL), what the scale of the attack is, how many bots are in the attacking botnet, and whether you have layer 3 protection at the firewall level for synflood attacks.
 +
 +For example, LiteSpeed Advanced Anti-DDoS Setup will efficiently protect against Layer 7 HTTP and Layer 4 TCP bot attacksl, but not against Layer 3 SYN Flood attack. ​ SYN Flood send SYN packets with spoofed source IP addresses and require Layer 3 protection at the firewall level. If a TCP connection established,​ it is a Layer 4 attack, but if a TCP connection is not established,​ it is Layer 3.
 +
 +Layer 4 TCP connection floods ​ can be detected and blocked by LiteSpeed Advanced Anti-DDos Setup Layer 4 connection hard limit settings.
 +
 +For large scale attacks, server kernel level settings may need to be adjusted to handle the large amount of HTTP requests during the attack.
 +
 +Generally speaking, LiteSpeed can handle a up to 1000 bots without problem. If bot number are well over 1000, while the LiteSpeed Web Server can handle the concurrent connections,​ typically server memory or PHP execution become bottlenecks. LiteSpeed Web Server can be configured to cache the attacked page, reducing the server/PHP resources ​ and increasing the server overall capacity, but this is an example that is well beyond the scope of this service.
 +
 +When you have a front-end proxy/CDN, the Denial of Service Packet Filter Setup may not work, since it blocks attacking robots at the IP level with iptables. When there is a front proxy, it only sees the IP of the proxy, and it cannot block the proxy IP, as all traffic is coming from that IP. If you have CloudFlare Pro or a similar service already, you may not need Denial of Service Packet Filter Setup Service since they do a similar job.
 +
 +Should you need such advance support services, they may be requested through evaluation and quotation. ​
 +
 +To order LiteSpeed Advanced Anti-DDos Setup Service, please [[https://​store.litespeedtech.com/​store/​cart.php?​gid=5|visit our store]].
 +
 +===== Never set Use Client IP in Header to Yes =====
 +To restore real visitor IPs, navigate to **LiteSpeed WebAdmin Console > Configuration > General Settings** and set **Use Client IP in Header** to ''​Trusted IP Only'',​ and add your CDN such as CloudFlare IPs/subnets to the trusted list. Never set **Use Client IP in Header** to ''​Yes'',​ since clients can spoof IPs with the ''​X-Forwarded-For''​ header that is sent to CloudFlare.
 ===== Troubleshooting ===== ===== Troubleshooting =====
  
-==== Check the Banned IP and Reason====+==== Check concurrent connections ==== 
 +To check the number of concurrent TCP connections,​ run the following command: 
 +  netstat -an | grep 80 | grep ESTA | wc  
 + 
 +To check concurrent connections sorted by IP, run the following:​ 
 +  netstat -ntu | grep ESTABLISHED | awk '​{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr 
 +   
 +Please keep in mind that ''​netstat -ntu''​ will list TCP in TIME_WAIT state, which will inflate the number. For the correct concurrent TCP connections counting method, you should only count TCPs in ''​ESTABLISHED''​ state. Hence ''​grep ESTA''​ or ''​grep ESTABLISHED''​ will be required. 
 + 
 +==== Analysis of IPs from Attacker ====  
 + 
 +Bad IP's can make quick connections,​ and you end up with many ''​time_waits''​ which you won't see when just looking at established. 
 + 
 +If you don't necessarily count concurrent connections,​ and just want to analyze which IPs might be attackers, you can include ''​time_waits''​ connections. Run the command without ''​grep ESTABLISHED'',​ which gives you the ability to see what IP's just connected and dropped and may need to be blocked: 
 + 
 +   ​netstat -ntu | awk '​{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | awk '$1 >= 5 {print $0}' 
 + 
 +An attacker could make a connection, send requests to an expensive URL, wait a little while, and then close the connection. If the server does not abort the process, the backend will be used up soon, as it will keep serving requests that have been abandoned. The above command will be useful during such a situation.  
 +==== Check the Banned IP and Reason ====
 If an IP has been banned, but you don't know why, you can check it with SSH. Here is an example of a connection that was banned because it reached the hard limit. If an IP has been banned, but you don't know why, you can check it with SSH. Here is an example of a connection that was banned because it reached the hard limit.
  
-Note: Your logging level must be set //at least// to **NOTICE** in order to see the reason an IP is banned.+**Note**: Your logging level must be set //at least// to ''​NOTICE'' ​in order to see the reason an IP is banned.
  
 ===Banned IP === ===Banned IP ===
Line 95: Line 153:
 === Banned Reason=== === Banned Reason===
  
-Whenever the server adds IP to the block list, it will write a log to error log:+Whenever the server adds an IP to the block list, it will write a log to error log:
  
-  [<​IP.addr>​] bot detected for vhost [<​vhostname>​],​ reason: xxxxx, ​clase connection!+  [<​IP.addr>​] bot detected for vhost [<​vhostname>​],​ reason: xxxxx, ​close connection!
  
 For example: For example:
 <​code>​tail -f /​etc/​apache2/​logs/​error_log</​code>​ <​code>​tail -f /​etc/​apache2/​logs/​error_log</​code>​
 <​code>​ <​code>​
- ​[NOTICE] [47.22.54.182] reached per client hard connection limit: 1, close connection! + ​[NOTICE] [x.x.x.reached per client hard connection limit: 1, close connection! 
- ​[NOTICE] [47.22.54.182] bot detected for vhost [N/A], reason: OverConnHardLimit,​ close connection!+ ​[NOTICE] [x.x.x.x] bot detected for vhost [N/A], reason: OverConnHardLimit, close connection! 
 +</​code>​ 
 +or 
 +<​code>​ 
 + ​[NOTICE] [x.x.x.x] bot detected for vhost [N/A], reason: OverConnSoftLimit,​ close connection 
 +</​code>​ 
 + 
 +or 
 +<​code>​ 
 +2018-12-05 12:​18:​05.440745 [NOTICE] [x.x.x.x] bot detected for vhost [APVH_example.com:​443],​ reason: DetectByWAF, close connection!
 </​code>​ </​code>​
  
 You should be able to find out why it is added and take action accordingly. You should be able to find out why it is added and take action accordingly.
-==== mod_security====+==== ModSecurity ​====
 If the IP was banned but a record was not found in ''​error_log'',​ it's possible that IP was dropped by mod_security. If the IP was banned but a record was not found in ''​error_log'',​ it's possible that IP was dropped by mod_security.
  
 <​code>​ grep "​47.22.54.182"​ /​usr/​local/​apache/​logs/​modsec_audit.log </​code>​ <​code>​ grep "​47.22.54.182"​ /​usr/​local/​apache/​logs/​modsec_audit.log </​code>​
  
 +==== Trusted IPs ====
 +
 +If the IP address involved is in the LSWS trusted list, it shows:
 +  2018-12-05 12:​18:​05.440754 [NOTICE] [x.x.x.x] trusted, ignore!
 +
 +Whenever a mod_security with "​drop"​ action is triggered, LiteSpeed will add the IP to the blacklist. If the IP is in the trusted list, it will be ignored. As with too many blocks, please review the mod_security rule and audit_log, as LSWS will follow the rules there.
 +
 +If ModSecurity blocks a request and LSWS sees the IP as trusted, the request is still served with 403 response, but that IP won't be blacklisted. If an IP is blacklisted,​ LSWS will stop serving future requests from that IP.
 +
 +
 +Trusted IP can be either set on server level or virtual host level through .htaccess ​
 +
 +=== Set Trusted IP on Server Level ===
 +In LSWS Admin Console Server → Security → Access Control → Allowed List, you can set Trusted IP there with trailing “T”.
 +
 +=== Set Trusted IP on Virtual Host Level .htaccess === 
 +Since LSWS 5.4RC1, LSWS has virtual host trusted IP support, where you may use ''​Trusted 1.2.3.4, 5.6.7.8''​ in the Virtual Host document root .htaccess to unblock a blocked IP and make that IP trusted for that vhost. This is not the same as the **Trusted IP** configured by Admin at server level. It has no effect on bandwidth. The main effect of adding it in .htaccess is to take that IP off of the blacklist and disable WordPress Protect and reCAPTCHA when accessing that specific virtual host. 
  
 +==== Drop or Deny ====
 +What if ModSecurity does a drop (TCP FIN) rather than deny for a trusted IP? The trusted list only has an effect on the "​drop"​ action, but not on the "​deny"​ action. A trusted IP won't be added to blacklist, but trust status has no effect on other actions.
  • Admin
  • Last modified: 2018/12/13 18:40
  • by Jackson Zhang