Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision | |||
litespeed_wiki:cache:no-plugin-crawler [2019/04/17 15:34] Lisa Clarke Copyediting |
litespeed_wiki:cache:no-plugin-crawler [2020/09/17 20:13] (current) Lisa Clarke Redirect to new Documentation Site |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Setting up a Crawler for Rewrite-Rule-Based Cache ====== | + | ~~REDIRECT>https://docs.litespeedtech.com/lscache/noplugin/settings/~~ |
- | You may wonder, "Can I set up a crawler for rewrite-rule-based cache? Does LiteSpeed have a crawler solution that can be used with my customized application?" The answer is "Yes," but there is something you should think about first: | + | |
- | + | ||
- | Normally, there is no need to set up a crawler for rewrite-rule-based cache. In typical usage, the TTL is set for a very short time, such as 2 minutes. Why? With rewrite rules, there is no ability to purge the cache, and so you can only cache items for a very short time, otherwise you run a great risk of serving stale content. (Our cache plugins have more cache-management controls, including the ability to purge the cache at will, which is why they typically have longer TTLs and can make good use of a crawler.) | + | |
- | + | ||
- | That said, if you have a rewrite-rule-based cache, if you are able to set a TTL of 30 minutes or longer, and if you have your own sitemap, we have a universal crawler script for you. | + | |
- | + | ||
- | ===== Enable the Crawler at the Server Level ===== | + | |
- | Firstly, you will need to enable the crawler for your LiteSpeed Web Server since it is normally disabled by default. Set the following in the Apache configuration: | + | |
- | <IfModule Litespeed> | + | |
- | CacheEngine on, crawler | + | |
- | </IfModule> | + | |
- | + | ||
- | Please refer to the detailed steps [[litespeed_wiki:cache:common_installation:shared-enable-individually|here]]. | + | |
- | + | ||
- | ===== Try the Script ===== | + | |
- | Secondly, you can use any third party script or your own crawler script to warm up your customized application. We do have a Magento 2 shell script crawler available, and as long as you have a sitemap for your site, it should work for you. Please refer [[https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:cache:litemage2:crawler|here]] for details. | + | |
- | + | ||
- | ===== Troubleshooting ===== | + | |
- | ==== Script Not Working ==== | + | |
- | Most likely you did not set ''CacheEngine on, crawler'' in the Apache configuration. Set it if you have missed it. | + | |
- | + | ||
- | The server may have a domain override at ''/etc/hosts'' like ''127.0.0.1 yourdomain.com www.yourdomain.com''. This will break the crawler. Commenting it out should solve the issue. | + | |
- | + | ||
- | If these steps don't solve your problem, feel free to log a ticket with us for further investigation. | + |