LiteSpeed Technologies
Download Download     Blog Blog     Wiki Wiki     Forum Forum     Store     Contact Contact    

Go Back   LiteSpeed Support Forums > LiteSpeed Web Server > General > stop yahoo from spidering my site in litespeed

Reply
 
Thread Tools Display Modes
  #1  
Old 08-25-2007, 06:22 PM
hichew hichew is offline
Member
 
Join Date: Mar 2007
Location: USA
Posts: 27
Default stop yahoo from spidering my site in litespeed

is there a way in litespeed to banned / slow down yahoo spider from indexing my site?
Can someone show me how, please...

this is a quote from ppl who running vBulletin forum

Quote:
This is causing a HUGE uproar for users with online BBs's. I've seen VB and IPB users alike complaining how they suddenly now have 1200 online users when they used to have 100,and 1100 of those 1200 are Slurp spiders.

Seriously...can this get reverted back? Track the session you are spidering a site with and force the session to use a single IP. As someone else said, sure devs can code around this, but the reality is the responsibility falls on Yahoo's hands in this case.
Quote:
Stop using 100+ different IP addresses when spidering

submitted by anonymous user 1 month ago
Why is it that Yahoo! uses a different IP for every page request while spidering?

A lot of sites, such as forums, have a "guests" or "online users" count listed on their page. When Yahoo! uses 146 different IP addresses to spider 146 pages, it artificially inflates the online users count.

It's irritating.

Please use one IP address when spidering to prevent inflating online user counts.

Thanks.
Reply With Quote
  #2  
Old 08-26-2007, 04:03 PM
mistwang mistwang is offline
LiteSpeed Staff
 
Join Date: May 2003
Location: New Jersey
Posts: 7,583
You can block that subnet with server level access control.
It is hard to slow them down when they use 100+ different IP addresses.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 09:22 PM.



- Archive - Top
© Copyright 2003-2011 LiteSpeed Technologies, Inc. All rights reserved. Privacy Policy.