Rate limiting and Google bot

W

wanah

Well-Known Member
#1
#1
Hello,

I ran into a curisous problem with Google bot a few days ago but I can't find any information about why this happened.

I disabled Request Filtering a few months ago because it was causing 403 errors with Google Bot. I didn't have any issues for some time and just recently a customer started getting 403 errors again for Google Bot.

I've got logging set to Log Level WARNING and Debug Level LOW and this does show any reason for blockages in /usr/local/apache/logs/error_log

I'm currently running a pre-release version of litespeed 4.2.4 because 4.2.3 had problems with php suexec and although you have realeased a final version yesterday I'm reluctant to upgrade because of all the issues 4.2.3 brought with it.

What should I do ? risk updating litespeed ? I'm still running LSPHP 6.2, should I upgrade to LSPHP 6.3 ?

I rearly need a stable solution, I can't beta test as this is a production server.

What could be causing 403 errors in both cPanel's domlogs and google webmaster tools.

The 403 errors have stopped for the moment (for the last 48 hours). But I can't let them start again.

One point I know is that Google bot had an issue about 10 days ago, could it have hit the "Per Client Throttling" limits. I've increased Static Requests/second to 200 and dynamic requests per second to 100 just in case…

Is Per Client Throtteling accross all accounts or just for each account ?

Sorry about all these questions but I rearly need to find a solution quickly :)
 
N

NiteWave

Administrator
#2
#2
Is Per Client Throtteling accross all accounts or just for each account ?
Click to expand...
for all accounts.

I've increased Static Requests/second to 200 and dynamic requests per second to 100 just in case ...
Click to expand...
limit static/dynamic reqs/second only slow down access from 1 IP, but won't block the IP. exceed connection soft/hard limit will cause IP being blocked.
so try to increase it.

on the other hand, the google bot looks very smart, it may adjust the frequency to access websites automatically. as long as you still see access entries in access log, you may not need worry about it.
 
Last edited:
W

wanah

Well-Known Member
#3
#3
The problem here is that it was getting 403 errors on web pages that were accessible in the web browser.

What logging level would I need to see when and why a 403 error is produced ?

I've set it to NOTICE and MIDDLE debug and hope to catch something next time it happes.
 
W

wanah

Well-Known Member
#5
#5
Code: 
66.249.72.96 - - [02/Aug/2013:03:20:16 +0200] "GET /section/page-path.html HTTP/1.1" 403 147 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
I removed our customers URL dans replaced it with "/section/page-path.html" the URL responds with a code 200 now, and google has managed to reference this page again.

I'm still trying to figure out what caused Googlebot to be given a 403 error.
 
W

wanah

Well-Known Member
#6
#6
I don't know if it's related or not but I've still got alot of these in my error logs :

Code: 
2013-08-08 10:29:48.314 [STDERR] Child process with pid: 616332 was killed by signal: 15, core dump: 0
2013-08-08 10:29:48.315 [STDERR] Child process with pid: 616337 was killed by signal: 15, core dump: 0
2013-08-08 10:29:59.292 [STDERR] Child process with pid: 616317 was killed by signal: 15, core dump: 0
Child process with pid: 616611 was killed by signal: 15, core dump: 0
2013-08-08 10:30:52.033 [STDERR] Children tracking is wrong: PID: 970855, Cur Childen: 24, count: 25, idle: 19, dying: 0
2013-08-08 10:32:21.819 [STDERR] Warning: 2013-08-08 10:32:21.819 [STDERR] /etc/magic, 0: 2013-08-08 10:32:21.819 [STDERR] using regular magic file `/usr/share/misc/magic'
2013-08-08 10:32:50.270 [STDERR] Child process with pid: 617379 was killed by signal: 15, core dump: 0
/etc/magic, 0: Warning: using regular magic file `/usr/share/misc/magic'
2013-08-08 10:37:08.168 [STDERR] /etc/magic, 0: Warning: 2013-08-08 10:37:08.168 [STDERR] using regular magic file `/usr/share/misc/magic'
Child process with pid: 619624 was killed by signal: 15, core dump: 0
2013-08-08 10:38:42.272 [STDERR] Child process with pid: 620030 was killed by signal: 15, core dump: 0
2013-08-08 10:39:37.276 [STDERR] Child process with pid: 620024 was killed by signal: 15, core dump: 0
2013-08-08 10:39:45.267 [STDERR] Child process with pid: 620279 was killed by signal: 15, core dump: 0
2013-08-08 10:40:16.283 [STDERR] Child process with pid: 620975 was killed by signal: 15, core dump: 0
Child process with pid: 621086 was killed by signal: 15, core dump: 0
2013-08-08 10:41:29.264 [STDERR] Child process with pid: 621203 was killed by signal: 15, core dump: 0
2013-08-08 10:41:30.264 [STDERR] Child process with pid: 621247 was killed by signal: 15, core dump: 0
2013-08-08 10:43:24.270 [STDERR] Child process with pid: 621423 was killed by signal: 15, core dump: 0
2013-08-08 10:43:26.269 [STDERR] Child process with pid: 621895 was killed by signal: 15, core dump: 0
2013-08-08 10:43:26.279 [STDERR] Child process with pid: 621887 was killed by signal: 15, core dump: 0
2013-08-08 10:44:12.266 [STDERR] Child process with pid: 622198 was killed by signal: 15, core dump: 0
Child process with pid: 622290 was killed by signal: 15, core dump: 0
2013-08-08 10:45:26.264 [STDERR] Child process with pid: 623029 was killed by signal: 15, core dump: 0
2013-08-08 10:45:29.265 [STDERR] Child process with pid: 623058 was killed by signal: 15, core dump: 0
2013-08-08 10:45:38.263 [STDERR] Child process with pid: 623061 was killed by signal: 15, core dump: 0
2013-08-08 10:46:44.270 [STDERR] Child process with pid: 623388 was killed by signal: 15, core dump: 0
2013-08-08 10:49:09.272 [STDERR] Child process with pid: 624077 was killed by signal: 15, core dump: 0
2013-08-08 10:49:12.266 [STDERR] Child process with pid: 624157 was killed by signal: 15, core dump: 0
2013-08-08 10:49:26.266 [STDERR] Child process with pid: 623997 was killed by signal: 15, core dump: 0
2013-08-08 10:49:26.270 [STDERR] Child process with pid: 624166 was killed by signal: 15, core dump: 0
2013-08-08 10:49:26.270 [STDERR] Child process with pid: 624214 was killed by signal: 15, core dump: 0
2013-08-08 10:49:26.270 [STDERR] Child process with pid: 624218 was killed by signal: 15, core dump: 0
2013-08-08 10:49:26.272 [STDERR] Child process with pid: 624048 was killed by signal: 15, core dump: 0
2013-08-08 10:49:26.274 [STDERR] Child process with pid: 624215 was killed by signal: 15, core dump: 0
2013-08-08 10:51:28.264 [STDERR] Child process with pid: 625399 was killed by signal: 15, core dump: 0
2013-08-08 10:51:28.265 [STDERR] Child process with pid: 625337 was killed by signal: 15, core dump: 0
2013-08-08 10:51:28.266 [STDERR] Child process with pid: 625400 was killed by signal: 15, core dump: 0
2013-08-08 10:51:28.268 [STDERR] Child process with pid: 625338 was killed by signal: 15, core dump: 0
2013-08-08 10:51:28.271 [STDERR] Child process with pid: 625117 was killed by signal: 15, core dump: 0
2013-08-08 10:53:06.269 [STDERR] Child process with pid: 624655 was killed by signal: 15, core dump: 0
2013-08-08 10:53:52.264 [STDERR] Child process with pid: 626169 was killed by signal: 15, core dump: 0
2013-08-08 10:53:52.264 [STDERR] Child process with pid: 626145 was killed by signal: 15, core dump: 0
2013-08-08 10:53:52.265 [STDERR] Child process with pid: 626040 was killed by signal: 15, core dump: 0
2013-08-08 10:53:52.266 [STDERR] Child process with pid: 626146 was killed by signal: 15, core dump: 0
2013-08-08 10:53:53.264 [STDERR] Child process with pid: 626037 was killed by signal: 15, core dump: 0
2013-08-08 10:54:05.264 [STDERR] Child process with pid: 626168 was killed by signal: 15, core dump: 0
2013-08-08 10:54:05.264 [STDERR] Child process with pid: 626221 was killed by signal: 15, core dump: 0
2013-08-08 10:54:52.266 [STDERR] Child process with pid: 626405 was killed by signal: 15, core dump: 0
2013-08-08 10:55:25.268 [STDERR] Child process with pid: 626403 was killed by signal: 15, core dump: 0
2013-08-08 10:55:29.264 [STDERR] Child process with pid: 627180 was killed by signal: 15, core dump: 0
2013-08-08 10:55:37.264 [STDERR] Child process with pid: 627125 was killed by signal: 15, core dump: 0
2013-08-08 10:55:49.277 [STDERR] Child process with pid: 626166 was killed by signal: 15, core dump: 0
2013-08-08 10:56:18.266 [STDERR] Child process with pid: 627473 was killed by signal: 15, core dump: 0
2013-08-08 10:56:56.272 [STDERR] Child process with pid: 627574 was killed by signal: 15, core dump: 0
2013-08-08 10:59:15.271 [STDERR] Child process with pid: 628382 was killed by signal: 15, core dump: 0
2013-08-08 10:59:19.266 [STDERR] Child process with pid: 628442 was killed by signal: 15, core dump: 0
2013-08-08 11:00:30.271 [STDERR] Child process with pid: 629413 was killed by signal: 15, core dump: 0
2013-08-08 11:00:34.264 [STDERR] Child process with pid: 629532 was killed by signal: 15, core dump: 0
2013-08-08 11:02:54.270 [STDERR] Child process with pid: 629865 was killed by signal: 15, core dump: 0
2013-08-08 11:03:29.280 [STDERR] Child process with pid: 630567 was killed by signal: 15, core dump: 0
2013-08-08 11:05:48.289 [STDERR] Child process with pid: 631872 was killed by signal: 15, core dump: 0
2013-08-08 11:06:58.045 [STDERR] Children tracking is wrong: PID: 970855, Cur Childen: 17, count: 18, idle: 12, dying: 0
When I had core dumps enabled the server kept crashing. Since I've disabled them it's better. I also had to disable opcode caching beacause it didn't like the child processes being killed all the time with the opcache shared memory.
 
N

NiteWave

Administrator
#8
#8
2013-08-08 11:05:48.289 [STDERR] Child process with pid: 631872 was killed by signal: 15, core dump: 0
Click to expand...
this is just information, not error.

in php suExec mode, each lsphp process will be killed if it's idle for some time. this is by design. since on a shared hosting environment, there may be thousand of accounts, thousands of websites powered by php, each lsphp process has its own uid/gid which matches user's account, so it can't be shared with other accounts. kill them frequently to lower the system load and consume less memory.
 
N

NiteWave

Administrator
#10
#10
yes, just information. please try latest 4.2.4, this info may not exist any more .. but I'm not 100% sure. if it still exist, please inform us here.
 
You must log in or register to reply here.
Top