Differences
This shows you the differences between two versions of the page.
|
litespeed_wiki:waf:avoid-faked-bots [2020/01/06 21:38] Jackson Zhang created |
litespeed_wiki:waf:avoid-faked-bots [2020/01/07 17:18] (current) Lisa Clarke Copyediting |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== How to avoid faked Google bot? ====== | + | ====== How to Avoid a Faked Google Bot ====== |
| - | A user received for some sort of attack from real Google ip ranges and Googlebot user-agent bypassing LSWS Recaptcha feature, which hit random GET /kategori/wewuj-qunuf-litiv/, etc. | + | LiteSpeed's reCAPTCHA feature is helpful for avoiding attacks, but reCAPTCHA may be bypassed when bad actors fake user-agents and IPs, and hit random URIs (like ''/kategori/wewuj-qunuf-litiv/'', as seen in the screenshot.) In particular, an attacker may pretend to be googlebot, originating from an IP within the real Google IP range. |
| {{ :litespeed_wiki:waf:google-bot-ramdom-url.png?800 |}} | {{ :litespeed_wiki:waf:google-bot-ramdom-url.png?800 |}} | ||
| - | Since those are real google bot, just update robot.txt to tell google to be more gentle crawling the site. | + | There are a few ways to deal with this problem: |
| - | Another possible solution is to customize rewrite rules testing against the user-agent, and redirect to the home page. | + | - Update ''robots.txt'' to tell Google to be more gentle crawling the site. |
| + | - Customize rewrite rules to test against the user agent, and redirect to the home page. | ||
| - | Usually, Google bot is not that aggressive. | + | Usually, googlebot is not so aggressive, so it's reasonable to assume you are under attack. If someone fakes the googlebot IP through the ''X-Forwarded-For'' header, you will need to detect and stop it. |
| - | If someone fakes google bot IP through the ''X-Forwarded-For'' header, you will need to detect and stop that. How? | + | For example, you might see these headers: |
| - | + | ||
| - | For example, you might see some headers: | + | |
| CF-IPCountry: XX | CF-IPCountry: XX | ||
| Line 23: | Line 22: | ||
| Forwarded: for=39.33.93.175 | Forwarded: for=39.33.93.175 | ||
| - | There are two ip adresses ''39.33.93.175''(faked google) and ''66.249.93.199''(real google IP). | + | There are two ip adresses ''39.33.93.175''(the faked google IP) and ''66.249.93.199''(the real google IP). |
| + | |||
| + | Configure LiteSpeed Web Server only update the client IP if the original IP is trusted. Navigate to **LiteSpeed WebAdmin Console > Configuration > General Settings** and set **Use Client IP in Header** to ''Trusted IP Only''(never set to ''Yes''), and add the google IPs/subnets to the trusted list. | ||
| - | While LiteSpeed Web Server needs to be configured to only update client IP if the original IP is trusted. In ''LiteSpeed WebAdmin Console > Configuration > General Settings'' and set ''Use Client IP in Header'' to ''Trusted IP Only''(never set to ''Yes''), and add google IPs/Subnets to the trusted list. On the other hand, Google doesn't post a public list of IP addresses for webmasters to whitelist and you can [[https://support.google.com/webmasters/answer/80553|verify googlebot IPs]] before adding them to allowed list. | + | Google doesn't post a public list of IP addresses for webmasters to whitelist, but you can [[https://support.google.com/webmasters/answer/80553|verify googlebot IPs]] before adding them to allowed list. |