This is an old revision of the document!


LiteSpeed Cache for Magento2: Crawler

The crawler travels through your site, refreshing pages that have expired in the cache. This makes it less likely that your visitors will encounter un-cached pages.

  1. Install and enable LiteMage Cache for Magento2
  2. Crawler Engine: The crawler must be enabled at the server level, or you will see the warning message Server crawler engine not enabled. Please check….. If you are using a shared hosting server, please contact your hosting provider, or see our instructions.
  3. SiteMap: Prepare your site's sitemap, e.g. http://magento2.com/sitemap.xml

Download from here

Change the permissions so that the file is executable: chmod +x M2_crawler.sh

Crawl Desktop&mobile share same theme

sh M2-crawler.sh SITE-MAP-URL

More Options

  • To get help: sh M2-crawler.sh -h
  • To change default interval request from 0.1s to custom NUM value: sh M2-crawler.sh SITE-MAP-URL -i NUM

The Sitemap module is build-in for generating a sitemap in Magento 2, and it's fast.

Enable sitemap

Navigate to Magento admin page → Stores → Settings → Configuration → Catalog → XML Sitemap

Set Generation Settings Enabled to Yes

Configuring a single sitemap for all storefronts

Navigate to Magento admin page → Marketing → Seo & Search → Sitemap

  1. Click Add Sitemap button
  2. Enter value
    • Filename: sitemap.xml
    • Path: /
  3. Click Save & Generate button


If all went well, a sitemap.xml file will generated in your magento 2 document root.

How often do we want to re-initiate the crawling process? This depends on how long it takes to crawl your site and what did you set for Public Cache TTL.
Default TTL is one day(24hr). Maybe you can consider to run the script by cronjob every 12 hours.
E.g. This will run twice a day, at 3:30am/15:30: 30 3/15 * * * path_to_script/M2_crawler.sh SITE-MAP-URL -m -i 0.2

Note: You can also use online crontab tool help you to verify time settings.

By using the browser developer tool, you should see X-LiteSpeed-Cache: hit,litemage at the first view

  • Admin
  • Last modified: 2018/07/27 17:58
  • by Lisa Clarke