Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
litespeed_wiki:cache:lscwp:configuration:enabling_the_crawler [2019/08/02 19:22] Lisa Clarke Proofreading |
litespeed_wiki:cache:lscwp:configuration:enabling_the_crawler [2019/12/09 17:18] Lisa Clarke Editing for clarity |
||
|---|---|---|---|
| Line 3: | Line 3: | ||
| These instructions apply to the WordPress LSCache crawler and other CMS LSCache crawlers where available. | These instructions apply to the WordPress LSCache crawler and other CMS LSCache crawlers where available. | ||
| - | Due to the potential of the crawler to consume considerable resources, we have put the on/off switch in the hands of the server administrators. On a control panel environment, such as cPanel, the crawler is disabled by default and can only be enabled by an admin through Apache configuration. While on LSWS native environment, the crawler is enabled by default and can be disabled on server level or virtual host level starting from LSWS 5.3.5 release. | + | Due to the potential of the crawler to consume considerable resources, we have put the on/off switch in the hands of the server administrators. In a control panel environment, such as cPanel, the crawler is disabled by default and can only be enabled by an admin through Apache configuration. In the LSWS Native environment, the crawler is //enabled// by default and can be disabled at the server level or virtual host level in LSWS v5.3.5 and above. |
| - | **NOTE: it is not recommended to turn on the crawler for shared hosting setups unless the server has enough capacity to handle it! | + | **NOTE: we do not recommend enabling the crawler for shared hosting setups unless the server has enough capacity to handle it!** |
| - | ** | + | |
| - | ===== On a shared hosting/control panel environment===== | + | ===== Shared Hosting / Control Panel Environment ===== |
| - | ==== Enabling the Crawler on a shared hosting/control panel environment === | + | |
| As of LSWS v5.1.16*, there are a few different approaches you can take to crawling on your server: | As of LSWS v5.1.16*, there are a few different approaches you can take to crawling on your server: | ||
| *You can disable it for the entire server | *You can disable it for the entire server | ||
| *You can enable it for the entire server | *You can enable it for the entire server | ||
| *You can selectively enable it for particular clients, while leaving it disabled for everyone else | *You can selectively enable it for particular clients, while leaving it disabled for everyone else | ||
| + | |||
| + | ==== Enabling the Crawler ==== | ||
| To enable the crawler in either of the second two scenarios, you need to add this “Crawler Snippet” to the appropriate configuration or include file: | To enable the crawler in either of the second two scenarios, you need to add this “Crawler Snippet” to the appropriate configuration or include file: | ||
| Line 27: | Line 27: | ||
| After you've added the Crawler Snippet in the appropriate location, you should gracefully restart the server. | After you've added the Crawler Snippet in the appropriate location, you should gracefully restart the server. | ||
| - | *If you are on v5.1.16 and having difficulty getting this to work, please force reinstall to the latest build. | + | **NOTE:** If you are on v5.1.16 and are having difficulty getting this to work, please force reinstall to the latest build. |
| ==== Limiting the Crawler ==== | ==== Limiting the Crawler ==== | ||
| Line 41: | Line 41: | ||
| SetEnv CRAWLER_USLEEP 1000 | SetEnv CRAWLER_USLEEP 1000 | ||
| SetEnv CRAWLER_LOAD_LIMIT 5.2 | SetEnv CRAWLER_LOAD_LIMIT 5.2 | ||
| + | </IfModule> | ||
| + | </code> | ||
| + | |||
| + | ==== Disabling the Crawler ==== | ||
| + | Starting from LSWS v5.3.5 or later, you may disable the crawler for an Apache virtual host, in any situation. Simply add ''CacheEngine -crawler'' to the Apache virtual host configuration, like so: | ||
| + | |||
| + | <code> | ||
| + | <IfModule LiteSpeed> | ||
| + | CacheEngine -crawler | ||
| </IfModule> | </IfModule> | ||
| </code> | </code> | ||
| Line 46: | Line 55: | ||
| ==== cPanel/WHM ==== | ==== cPanel/WHM ==== | ||
| - | === Server level === | + | === Server Level === |
| Change your working directory to: | Change your working directory to: | ||
| - | ''/usr/local/apache/conf/includes/'' for EA3 or | + | * ''/usr/local/apache/conf/includes/'' for EA3 or |
| - | ''/etc/apache2/conf.d/includes/'' for EA4. | + | * ''/etc/apache2/conf.d/includes/'' for EA4. |
| Add the Crawler Snippet and optional server variables to the ''pre_main_global.conf'' file. | Add the Crawler Snippet and optional server variables to the ''pre_main_global.conf'' file. | ||
| - | === Global virtual host level === | + | === Global Virtual Host Level === |
| Change your working directory to: | Change your working directory to: | ||
| - | ''/usr/local/apache/conf/userdata/''for EA3 or | + | * ''/usr/local/apache/conf/userdata/''for EA3 or |
| - | ''/etc/apache2/conf.d/userdata/'' for EA4 | + | * ''/etc/apache2/conf.d/userdata/'' for EA4 |
| If these directories do not exist, create them. | If these directories do not exist, create them. | ||
| Line 68: | Line 77: | ||
| /scripts/ensure_vhost_includes --all-users | /scripts/ensure_vhost_includes --all-users | ||
| - | //Note: You only need to run this command once and it will activate for all users, including new users created by WHM later. There is no need to edit the cPanel skeleton file.// | + | **Note**: You only need to run this command once and it will activate for all users, including new users created by WHM later. There is no need to edit the cPanel skeleton file. |
| - | === Individual virtual host level === | + | === Individual Virtual Host Level === |
| Change your working directory to: | Change your working directory to: | ||
| - | - For EA3: ''/usr/local/apache/conf/userdata/std/2_4/<user>/<domain>/'' | + | * For EA3: ''/usr/local/apache/conf/userdata/std/2_4/<user>/<domain>/'' |
| - | - For EA4: ''/etc/apache2/conf.d/userdata/std/2_4/<user>/<domain>/'' | + | * For EA4: ''/etc/apache2/conf.d/userdata/std/2_4/<user>/<domain>/'' |
| - | If your site support https(ssl), please also change working directory to: | + | |
| - | - For EA3: ''/usr/local/apache/conf/userdata/ssl/2_4/<user>/<domain>/'' | + | If your site supports HTTPS (SSL), please also change that working directory to: |
| - | - For EA4: ''/etc/apache2/conf.d/userdata/ssl/2_4/<user>/<domain>/'' | + | * For EA3: ''/usr/local/apache/conf/userdata/ssl/2_4/<user>/<domain>/'' |
| - | * Above example path of ''2_4'' can be other version of your apache's, e.g. 2, 2_2 | + | * For EA4: ''/etc/apache2/conf.d/userdata/ssl/2_4/<user>/<domain>/'' |
| + | |||
| + | **NOTE**: The ''2_4'' in the path is an example. You can replace it with your appropriate version, such as ''2'' or ''2_2''. | ||
| If these directories do not exist, create them. | If these directories do not exist, create them. | ||
| Line 90: | Line 101: | ||
| ==== Plesk ==== | ==== Plesk ==== | ||
| - | === Server level === | + | === Server Level === |
| Change your working directory to: | Change your working directory to: | ||
| - | ''/etc/httpd/conf.d/'' for CentOS | + | * ''/etc/httpd/conf.d/'' for CentOS |
| - | ''/etc/apache2/conf.d/'' for Debian | + | * ''/etc/apache2/conf.d/'' for Debian |
| - | ''/etc/apache2/conf-enabled'' for Ubuntu | + | * ''/etc/apache2/conf-enabled'' for Ubuntu |
| Add the Crawler Snippet and optional server variables to ''lscache.conf''. If it doesn’t exist, create it. | Add the Crawler Snippet and optional server variables to ''lscache.conf''. If it doesn’t exist, create it. | ||
| - | === Global virtual host level === | + | === Global Virtual Host Level === |
| Change your working directory to ''/usr/local/psa/admin/conf/templates/custom/domain'' | Change your working directory to ''/usr/local/psa/admin/conf/templates/custom/domain'' | ||
| + | |||
| Create it if it doesn’t exist. | Create it if it doesn’t exist. | ||
| - | Copy''/usr/local/psa/admin/conf/templates/default/domain/domainVirtualHost.php'' to this location. | + | |
| + | Copy ''/usr/local/psa/admin/conf/templates/default/domain/domainVirtualHost.php'' to this location. | ||
| Edit the file and add the Crawler Snippet and optional server variables after the ''mod_suexec.c'' block. | Edit the file and add the Crawler Snippet and optional server variables after the ''mod_suexec.c'' block. | ||
| - | Reconfigure all virtual hosts (this will regenerate new configuration files for all vhosts): | + | Reconfigure all virtual hosts (this will regenerate new configuration files for all vhosts), like so:: |
| /usr/local/psa/admin/bin/httpdmng --reconfigure-all | /usr/local/psa/admin/bin/httpdmng --reconfigure-all | ||
| - | === Individual virtual host level === | + | === Individual Virtual Host Level === |
| Change your working directory to ''/var/www/vhosts/system/<domain_name>/conf/'' | Change your working directory to ''/var/www/vhosts/system/<domain_name>/conf/'' | ||
| + | |||
| Create a file called ''vhost.conf'' if it does not already exist ( or ''vhost_ssl.conf'' for HTTPS sites). | Create a file called ''vhost.conf'' if it does not already exist ( or ''vhost_ssl.conf'' for HTTPS sites). | ||
| + | |||
| Add the Crawler Snippet and optional server variables to this file. | Add the Crawler Snippet and optional server variables to this file. | ||
| - | Reconfigure this Virtual Host (this will regenerate new configuration files for this vhost): | + | Reconfigure this Virtual Host (this will regenerate new configuration files for this vhost), like so: |
| /usr/local/psa/admin/bin/httpdmng --reconfigure-domain <domain_name> | /usr/local/psa/admin/bin/httpdmng --reconfigure-domain <domain_name> | ||
| Line 123: | Line 138: | ||
| ==== DirectAdmin ==== | ==== DirectAdmin ==== | ||
| - | === Server level === | + | === Server Level === |
| Add the Crawler Snippet and optional server variables to the ''/etc/httpd/conf/extra/httpd-includes.conf'' file. | Add the Crawler Snippet and optional server variables to the ''/etc/httpd/conf/extra/httpd-includes.conf'' file. | ||
| - | Global virtual host level | + | |
| + | === Global virtual host level === | ||
| Create a ''/usr/local/directadmin/data/templates/custom/cust_httpd.CUSTOM.2.pre'' file and add the Crawler Snippet and optional server variables to it. | Create a ''/usr/local/directadmin/data/templates/custom/cust_httpd.CUSTOM.2.pre'' file and add the Crawler Snippet and optional server variables to it. | ||
| Line 136: | Line 153: | ||
| </code> | </code> | ||
| - | ==== ''CacheEngine -crawler'' ==== | + | ===== In a LiteSpeed Native Environment ===== |
| - | Starting from LSWS 5.3.5 or later, in any situation, if you just want to ensure to disable crawler for apache virtual host, you can add ''CacheEngine -crawler'' to the Apache virtual host configuration. | + | The cache crawler is enabled by default in a LSWS Native environment. |
| - | + | ||
| - | <IfModule LiteSpeed> | + | |
| - | CacheEngine -crawler | + | |
| - | </IfModule> | + | |
| - | ''CacheEngine -crawler''(this is supported in LSWS v5.3.5 and later) in | + | To disable it at the Server Level, you will need to use LSWS 5.4 and above. There was a **Cache Features** function added to control this. |
| - | + | ||
| - | + | ||
| - | ===== In a LiteSpeed Native Environment ===== | + | |
| - | The cache crawler is enabled by default in a LSWS native environment. | + | |
| - | To disable it at the Server Level, in the LSWS WebAdmin interface, navigate to **LSWS Admin > Configuration > Server > Cache**. In **Cache Features**, check ''On'', uncheck ''Crawler'', check ''ESI'', and uncheck ''Not Set''. | + | In the LSWS WebAdmin interface, navigate to **LSWS Admin > Configuration > Server > Cache**. In **Cache Features**, check ''On'', uncheck ''Crawler'', check ''ESI'', and uncheck ''Not Set''. |
| If ''Not Set'' is checked, the other three values will be ignored and the default values will be used. (By default, all three are checked.) | If ''Not Set'' is checked, the other three values will be ignored and the default values will be used. (By default, all three are checked.) | ||
| Line 155: | Line 164: | ||
| {{:litespeed_wiki:cache:lscwp:configuration:disable-crawler-lsws-native-1.png?600|}} | {{:litespeed_wiki:cache:lscwp:configuration:disable-crawler-lsws-native-1.png?600|}} | ||
| - | To disable the cache crawler at the LSWS native Virtual Host level, you can go to **LSWS Admin > Configuration > Virtual Host > VH Name > Cache >**, and set **Cache Features** in the same manner as above. If ''Not Set'' is checked, the other three values will be ignored and the server-level configuration will be inherited. | + | To disable the cache crawler at the LSWS Native Virtual Host level, navigate to **LSWS Admin > Configuration > Virtual Host > VH Name > Cache >**, and set **Cache Features** in the same manner as above. If ''Not Set'' is checked, the other three values will be ignored and the server-level configuration will be inherited. |
| Please note: Do not set **Enable LiteMage** to ''On'', as this setting will also enable the crawler, even if ''Crawler'' is unchecked. | Please note: Do not set **Enable LiteMage** to ''On'', as this setting will also enable the crawler, even if ''Crawler'' is unchecked. | ||
| Line 170: | Line 179: | ||
| ===== Testing ===== | ===== Testing ===== | ||
| - | LiteSpeed Web server cache engine will set environment varibles for ''X-LSCACHE''. You can always check Envirment Variables through phpinfo page to see if crawler is on or not. If the crawler is not there, then it has been disabled successfully. LSWS can only disable the LiteSpeed cache plugin or LiteSpeed crawler since such LiteSpeed crawlers will check ''X_LSCACHE'' environment variable. LSWS can not stop any third party crawler from working since they don't check ''X_LSCACHE'' to act accordingly. | + | The LiteSpeed Web server cache engine will set environment varibles for ''X-LSCACHE''. You can always check Environment Variables through the ''phpinfo'' page to see if the crawler is on or not. If the crawler is not there, then it has been disabled successfully. LSWS can only disable the LiteSpeed Cache plugin or LiteSpeed crawler since such LiteSpeed crawlers will check ''X_LSCACHE'' environment variable. LSWS can not stop any third party crawler from working since they don't check ''X_LSCACHE'' to act accordingly. |
| $_SERVER['X-LSCACHE'] on,esi | $_SERVER['X-LSCACHE'] on,esi | ||