This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
litespeed_wiki:cache:no-plugin-crawler [2019/04/17 15:34]
Lisa Clarke Copyediting
litespeed_wiki:cache:no-plugin-crawler [2020/09/17 20:13] (current)
Lisa Clarke Redirect to new Documentation Site
Line 1: Line 1:
-====== Setting up a Crawler for Rewrite-Rule-Based Cache ====== +~~REDIRECT>​https://​docs.litespeedtech.com/​lscache/noplugin/settings/~~
-You may wonder, "Can I set up a crawler for rewrite-rule-based cache? Does LiteSpeed have a crawler solution that can be used with my customized application?"​ The answer is "​Yes,"​ but there is something you should think about first: +
- +
-Normally, there is no need to set up a crawler for rewrite-rule-based cache. In typical usage, the TTL is set for a very short time, such as 2 minutes. Why? With rewrite rules, there is no ability to purge the cache, and so you can only cache items for a very short time, otherwise you run a great risk of serving stale content. (Our cache plugins have more cache-management controls, including the ability to purge the cache at will, which is why they typically have longer TTLs and can make good use of a crawler.) +
- +
-That said, if you have a rewrite-rule-based cache, if you are able to set a TTL of 30 minutes or longer, and if you have your own sitemap, we have a universal crawler script for you. +
- +
-===== Enable the Crawler at the Server Level ===== +
-Firstly, you will need to enable the crawler for your LiteSpeed Web Server since it is normally disabled by default. Set the following in the Apache configuration:​ +
-  <​IfModule Litespeed> +
-  CacheEngine on, crawler +
-  </​IfModule>​ +
- +
-Please refer to the detailed steps [[litespeed_wiki:​cache:​common_installation:​shared-enable-individually|here]]. +
- +
-===== Try the Script ===== +
-Secondly, you can use any third party script or your own crawler script to warm up your customized application. We do have a Magento 2 shell script crawler available, and as long as you have a sitemap for your site, it should work for you. Please refer [[https://www.litespeedtech.com/​support/wiki/doku.php/litespeed_wiki:​cache:​litemage2:​crawler|here]] for details.  +
- +
-===== Troubleshooting ===== +
-==== Script Not Working ==== +
-Most likely you did not set ''​CacheEngine on, crawler''​ in the Apache configuration. Set it if you have missed it. +
- +
-The server may have a domain override at ''/​etc/​hosts''​ like ''​ yourdomain.com www.yourdomain.com''​. This will break the crawler. Commenting it out should solve the issue. +
- +
-If these steps don't solve your problem, feel free to log a ticket with us for further investigation.+
  • Admin
  • Last modified: 2020/09/17 20:13
  • by Lisa Clarke