This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
Last revision Both sides next revision
litespeed_wiki:cache:no-plugin-crawler [2019/04/17 14:27]
Jackson Zhang created
litespeed_wiki:cache:no-plugin-crawler [2019/04/17 15:34]
Lisa Clarke Copyediting
Line 1: Line 1:
-====== ​Can I setup crawler ​for rewrite rule based cache? ​====== +====== ​Setting up a Crawler ​for Rewrite-Rule-Based Cache ====== 
-You may wonder"Can I setup crawler for rewrite ​ruble-based cache? ​Do you have such crawler solution ​for it if I have a sitemap for my customized application? ​+You may wonder"Can I set up a crawler for rewrite-rule-based cache? ​Does LiteSpeed ​have crawler solution ​that can be used with my customized application?​" The answer is "​Yes,"​ but there is something you should think about first:
-Normally there is no need to setup crawler for rewrite rule based cache. ​The typical ​case for rewrite rule-based cache setup is to set cache for a very short time, such as 2 minutes. ​Then cache files will be expired ​and the site will need to re-cache again. The reason to set it to a very short time is to handle the cache purge. If you set it for long timeany changes ​to that page won't be updated until the cache expired. 2 minutes could be a good timeframe meets your tolerate for the updated page not to be refreshed. ​ Of causeyou can set that cache TTL timeframe to be longer ​whichever you can are tolerant.+Normallythere is no need to set up a crawler for rewrite-rule-based cache. ​In typical ​usage, the TTL is set for a very short time, such as 2 minutes. ​Why? With rewrite rules, there is no ability to purge the cacheand so you can only cache items for a very short time, otherwise ​you run great risk of serving stale content. (Our cache plugins have more cache-management controlsincluding the ability ​to purge the cache at willwhich is why they typically have longer ​TTLs and can make good use of a crawler.)
-In some rare casesuch as user sets the cache TTL to 30 mins in the rewrite rules and still want the site to be crawled. The user has sitemap ​already for the site. How to enable crawler function within LSWS and is there any universal crawler script ​available? The answer is yes.+That saidif you have rewrite-rule-based ​cache, if you are able to set a TTL of 30 minutes or longer, ​and if you have your own sitemap, we have a universal crawler script ​for you.
-===== Enable ​"​crawler"​ on LiteSpeed server (Apache configuration) ​===== +===== Enable ​the Crawler at the Server Level ===== 
-Firstly, you will need to enable ​"crawler" ​for your LiteSpeed ​web server ​since "​crawler ​is normally disabled by default. ​In control panel environment or LiteSpeed reads apache configuration,​ you should set the following ​at apache ​configuration.+Firstly, you will need to enable ​the crawler for your LiteSpeed ​Web Server ​since it is normally disabled by default. ​Set the following ​in the Apache ​configuration:
   <​IfModule Litespeed>​   <​IfModule Litespeed>​
   CacheEngine on, crawler   CacheEngine on, crawler
   </​IfModule>​   </​IfModule>​
-Please refer the detailed steps [[litespeed_wiki:​cache:​common_installation:​shared-enable-individually|here]].+Please refer to the detailed steps [[litespeed_wiki:​cache:​common_installation:​shared-enable-individually|here]].
-===== Try a Magento 2 shell script crawler when you have a sitemap ​===== +===== Try the Script ​===== 
-Secondly, you can use any third party or your own crawler script to warm up your customized application. We do have a Magento 2 shell script crawler available as far as you have a sitemap for your site. Please refer [[https://​www.litespeedtech.com/​support/​wiki/​doku.php/​litespeed_wiki:​cache:​litemage2:​crawler|here]] for details. ​+Secondly, you can use any third party script ​or your own crawler script to warm up your customized application. We do have a Magento 2 shell script crawler available, and as long as you have a sitemap for your site, it should work for you. Please refer [[https://​www.litespeedtech.com/​support/​wiki/​doku.php/​litespeed_wiki:​cache:​litemage2:​crawler|here]] for details. ​
 ===== Troubleshooting ===== ===== Troubleshooting =====
-==== Try the above script but not working? ​==== +==== Script Not Working ​==== 
-Most likely you did not set ''​CacheEngine on, crawler''​ in apache ​configuration. Set it if you have missed it.+Most likely you did not set ''​CacheEngine on, crawler''​ in the Apache ​configuration. Set it if you have missed it.
-The server may have a domain override at ''/​etc/​hosts''​ like ''​ yourdomain.com www.yourdomain.com''​. ​It will break the crawler. Commenting it should solve the issue.+The server may have a domain override at ''/​etc/​hosts''​ like ''​ yourdomain.com www.yourdomain.com''​. ​This will break the crawler. Commenting it out should solve the issue.
-If still an issueyou can log a ticket with us to have further ​investigations. +If these steps don't solve your problemfeel free to log a ticket with us for further ​investigation.
- +
-  ​+
  • Admin
  • Last modified: 2020/09/17 20:13
  • by Lisa Clarke