LiteSpeed Cache for Magento2: Crawler

The crawler travels through your site, refreshing pages that have expired in the cache. This makes it less likely that your visitors will encounter un-cached pages.

  1. Install and enable LiteMage Cache for Magento2
  2. Crawler Engine: The crawler must be enabled at the server level, or you will see the warning message Server crawler engine not enabled. Please checkā€¦.. If you are using a shared hosting server, please contact your hosting provider, or see our instructions.
  3. SiteMap: Prepare your site's sitemap, e.g. http://magento2.com/sitemap.xml
  1. Change the permissions so that the file is executable: chmod +x M2_crawler.sh
  2. Run the script: bash M2-crawler.sh SITE-MAP-URL

More Options

  • To get help: bash M2-crawler.sh -h
  • To change default interval request from 0.1s to custom NUM value: bash M2-crawler.sh SITE-MAP-URL -i NUM
  • To crawl with cookie set: bash M2-crawler.sh -c SITE-MAP-URL
  • To store log in /tmp/M2-crawler.log: bash M2-crawler.sh -v SITE-MAP-URL
  • To debug one URL and output on screen: bash M2-crawler.sh -d SITE-URL

Magento 2 has a builtin module for generating a sitemap and it's fast.

Enable sitemap

Navigate to Magento Admin > Stores > Settings > Configuration > Catalog > XML Sitemap

Set Generation Settings > Enabled to Yes

Configuring a Single Sitemap for All Storefronts

Navigate to Magento Admin > Marketing > Seo & Search > Sitemap

  1. Click the Add Sitemap button
  2. Enter values
    • Filename: sitemap.xml
    • Path: /
  3. Click the Save & Generate button


If all went well, a sitemap.xml file will have been generated in your Magento 2 document root.

How often do you want to re-initiate the crawling process? This depends on how long it takes to crawl your site and what you set for Public Cache TTL.

The default TTL is one day(24hr). Maybe, for example, you'd like to run the script by cronjob every 12 hours instead.

E.g. This will run twice a day, at 3:30am/15:30: 30 3/15 * * * path_to_script/M2_crawler.sh SITE-MAP-URL -m -i 0.2

Note: You can also use online crontab tool to help you to verify the time settings.

When using the browser developer tool, load a previously uncached page. You should see X-LiteSpeed-Cache: hit,litemage on the first view.

  • Admin
  • Last modified: 2019/04/04 16:12
  • by Lisa Clarke