[solved] Give option robots.txt for each vhosts

Discussion in 'Feedback/Feature Requests' started by andych, May 26, 2013.

  1. andych

    andych New Member

    I would like to suggest that if LiteSpeed have an option for robots.txt builtin as feature just like .htaccess

    So we can adjust robots.txt for each vhosts without having one robots.txt reflects all vhosts.

    Example:
    We had 2 domains,
    We only welcome the crawlers to crawls from domain A
    We blocks crawlers to crawls domain B

    Thank you and I hope I will find solution for my case and this suggestions makes LiteSpeed even perfect web server.
    Last edited by a moderator: May 28, 2013
  2. NiteWave

    NiteWave Administrator

    can archive this by rewriterule easily(in .htaccess for example):

    RewriteCond %{HTTP_HOST} domainB
    RewriteRule robots.txt /robots_for_domainB.txt

    create a text file robots_for_domainB.txt under document root to block all requests; another robots.txt is for other domains.
  3. andych

    andych New Member

    Below are my current existing .htaccess

    In which line I should add the rules above?

    Code:
    root@www:~# cat /var/www/.htaccess
    order allow,deny
    allow from all
    
    deny from 1.2.3.4 #bad-ips
    
    Options +FollowSymlinks
    RewriteEngine On
    RewriteBase /
    # 4 lines below is to force visitors to use www subdom.
    RewriteCond %{http_host} ^main.dom
    RewriteRule ^(.*) http://www.main.dom/$1 [R=301,L]
    RewriteCond %{http_host} ^secondary.dom
    RewriteRule ^(.*) http://www.secondary.dom/$1 [R=301,L]
    
    #LSWS Cache:
    RewriteCond %{HTTP_COOKIE} !if_imloggedin=yes
    RewriteCond %{HTTP_COOKIE} !if_userid=
    RewriteCond %{HTTP_COOKIE} !if_password=
    RewriteCond %{REQUEST_METHOD} ^HEAD|GET$
    RewriteCond %{QUERY_STRING} !s=[a-fA-F0-9]{32}
    RewriteCond %{QUERY_STRING} !product=vbnexus
    RewriteCond %{REQUEST_URI} !^/(login|register|usercp|private|profile|cron|image)\.php$
    RewriteCond %{REQUEST_URI} !^/admincp
    RewriteCond %{REQUEST_URI} !^/modcp
    RewriteRule (.*) - [E=Cache-Control:max-age=300]
    
    RewriteRule ^home/$ forum.php [L,R=301]
    RewriteRule ^f([0-9]+)/$ forumdisplay.php?f=$1 [L,R=301]
    RewriteRule ^f([0-9]+)-([0-9]+)/$ forumdisplay.php?f=$1&page=$2 [L,R=301]
    RewriteRule ^t([0-9]+)/$ showthread.php?t=$1 [L,R=301]
    RewriteRule ^t([0-9]+)-([0-9]+)/$ showthread.php?t=$1&page=$2 [L,R=301]
    RewriteRule ^t([0-9]+)-print/$ showthread.php?t=$1 [L,R=301]
    RewriteRule ^blogs/u([0-9]+)-e([0-9]+)/$ entry.php?b=$2 [L,R=301]
    root@www:~# 
  4. NiteWave

    NiteWave Administrator

    can add the rules before
    #LSWS Cache
  5. andych

    andych New Member

    Hi NiteWave, thank you for confirming :)

Share This Page