Setting up a Crawler for Rewrite-Rule-Based Cache

You may wonder, “Can I set up a crawler for rewrite-rule-based cache? Does LiteSpeed have a crawler solution that can be used with my customized application?” The answer is “Yes,” but there is something you should think about first:

Normally, there is no need to set up a crawler for rewrite-rule-based cache. In typical usage, the TTL is set for a very short time, such as 2 minutes. Why? With rewrite rules, there is no ability to purge the cache, and so you can only cache items for a very short time, otherwise you run a great risk of serving stale content. (Our cache plugins have more cache-management controls, including the ability to purge the cache at will, which is why they typically have longer TTLs and can make good use of a crawler.)

That said, if you have a rewrite-rule-based cache, if you are able to set a TTL of 30 minutes or longer, and if you have your own sitemap, we have a universal crawler script for you.

Firstly, you will need to enable the crawler for your LiteSpeed Web Server since it is normally disabled by default. Set the following in the Apache configuration:

<IfModule Litespeed>
CacheEngine on, crawler
</IfModule>

Please refer to the detailed steps here.

Secondly, you can use any third party script or your own crawler script to warm up your customized application. We do have a Magento 2 shell script crawler available, and as long as you have a sitemap for your site, it should work for you. Please refer here for details.

Most likely you did not set CacheEngine on, crawler in the Apache configuration. Set it if you have missed it.

The server may have a domain override at /etc/hosts like 127.0.0.1 yourdomain.com www.yourdomain.com. This will break the crawler. Commenting it out should solve the issue.

If these steps don't solve your problem, feel free to log a ticket with us for further investigation.