Differences

This shows you the differences between two versions of the page.

--- litespeed_wiki:cache:no-plugin-crawler [2019/04/17 14:27]
Jackson Zhang created
+++ litespeed_wiki:cache:no-plugin-crawler [2019/04/17 15:34]
Lisa Clarke Copyediting
@@ Line 1: / Line 1: @@
-====== Can I setup crawler for rewrite rule based cache? ======
+====== Setting up a Crawler for Rewrite-Rule-Based Cache ======
-You may wonder: "Can I setup crawler for rewrite ruble-based cache? Do you have such crawler solution for it if I have a sitemap for my customized application?
+You may wonder, "Can I set up a crawler for rewrite-rule-based cache? Does LiteSpeed have a crawler solution that can be used with my customized application?" The answer is "Yes," but there is something you should think about first:
-Normally there is no need to setup crawler for rewrite rule based cache. The typical case for rewrite rule-based cache setup is to set cache for a very short time, such as 2 minutes. Then cache files will be expired and the site will need to re-cache again. The reason to set it to a very short time is to handle the cache purge. If you set it for a long time, any changes to that page won't be updated until the cache expired. 2 minutes could be a good timeframe meets your tolerate for the updated page not to be refreshed.  Of cause, you can set that cache TTL timeframe to be longer whichever you can are tolerant.
+Normally, there is no need to set up a crawler for rewrite-rule-based cache. In typical usage, the TTL is set for a very short time, such as 2 minutes. Why? With rewrite rules, there is no ability to purge the cache, and so you can only cache items for a very short time, otherwise you run a great risk of serving stale content. (Our cache plugins have more cache-management controls, including the ability to purge the cache at will, which is why they typically have longer TTLs and can make good use of a crawler.)
-In some rare case, such as a user sets the cache TTL to 30 mins in the rewrite rules and still want the site to be crawled. The user has sitemap already for the site. How to enable crawler function within LSWS and is there any universal crawler script available? The answer is yes.
+That said, if you have a rewrite-rule-based cache, if you are able to set a TTL of 30 minutes or longer, and if you have your own sitemap, we have a universal crawler script for you.
-===== Enable "crawler" on LiteSpeed server (Apache configuration) =====
+===== Enable the Crawler at the Server Level =====
-Firstly, you will need to enable "crawler" for your LiteSpeed web server since "crawler is normally disabled by default. In control panel environment or LiteSpeed reads apache configuration, you should set the following at apache configuration.
+Firstly, you will need to enable the crawler for your LiteSpeed Web Server since it is normally disabled by default. Set the following in the Apache configuration:
   <IfModule Litespeed>
   CacheEngine on, crawler
   </IfModule>
-Please refer the detailed steps [[litespeed_wiki:cache:common_installation:shared-enable-individually|here]].
+Please refer to the detailed steps [[litespeed_wiki:cache:common_installation:shared-enable-individually|here]].
-===== Try a Magento 2 shell script crawler when you have a sitemap =====
+===== Try the Script =====
-Secondly, you can use any third party or your own crawler script to warm up your customized application. We do have a Magento 2 shell script crawler available as far as you have a sitemap for your site. Please refer [[https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:cache:litemage2:crawler|here]] for details.
+Secondly, you can use any third party script or your own crawler script to warm up your customized application. We do have a Magento 2 shell script crawler available, and as long as you have a sitemap for your site, it should work for you. Please refer [[https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:cache:litemage2:crawler|here]] for details.
 ===== Troubleshooting =====
-==== Try the above script but not working? ====
+==== Script Not Working ====
-Most likely you did not set ''CacheEngine on, crawler'' in apache configuration. Set it if you have missed it.
+Most likely you did not set ''CacheEngine on, crawler'' in the Apache configuration. Set it if you have missed it.
-The server may have a domain override at ''/etc/hosts'' like ''127.0.0.1 yourdomain.com www.yourdomain.com''. It will break the crawler. Commenting it should solve the issue.
+The server may have a domain override at ''/etc/hosts'' like ''127.0.0.1 yourdomain.com www.yourdomain.com''. This will break the crawler. Commenting it out should solve the issue.
-If still an issue, you can log a ticket with us to have further investigations.
+If these steps don't solve your problem, feel free to log a ticket with us for further investigation.