This is an old revision of the document!
LiteSpeed Cache for Prestashop: Crawler
The crawler travels through your site, refreshing pages that have expired in the cache. This makes it less likely that your visitors will encounter un-cached pages.
Before You Begin
- Install and enable LiteSpeed Cache for Prestashop
- Crawler Engine: The crawler must be enabled at the server level, or you will see the warning message
Server crawler engine not enabled. Please check….
. If you are using a shared hosting server, please contact your hosting provider, or see our instructions. - SiteMap: Prepare your site's sitemap, e.g.
http://prestashop-123/456_sitemap.xml
How to Use the Crawler Script
Change the permissions so that the file is executable: chmod +x cachecrawler.sh
Crawl when desktop & mobile share the same theme: bash cachecrawler.sh SITE-MAP-URL
Crawl when desktop & mobile have different themes: bash cachecrawler.sh SITE-MAP-URL -m
By default, in the Prestashop cache plugin Mobile View is DISABLED. To enable mobile view, navigate to PrestaShop Admin → LiteSpeed Cache → Configuration and set Separate Mobile View to Yes
More Options
-h, --help
: Show this message and exit.-m, --with-mobile
: Crawl mobile view in addition to default view.-c, --with-cookie
: Crawl with site's cookies.-b, --black-list
: Page will be added to blacklist if HTML status error and no cache. Next run will bypass page.-g, --general-ua
: Use general user-agent instead of lscache_runner for desktop view.-i, --interval
: Change request interval.-i 0.2
changes from default 0.1 second to 0.2 seconds.-v, --verbose
: Show complete response header under/tmp/crawler.log
.-d, --debug-url
: Test one URL directly. as insh cachecrawler.sh -v -d http://example.com/test.html
.-qs,--crawl-qs
: Crawl sitemap, including URLS with query strings.-r, --report
: Display total count of crawl result.
Example commands:
- To get help:
bash cachecrawler.sh -h
- To change default interval request from 0.1s to custom NUM value:
bash cachecrawler.sh SITE-MAP-URL -i NUM
- To crawl with cookie set:
bash cachecrawler.sh -c SITE-MAP-URL
- To store log in
/tmp/crawler.log
:bash cachecrawler.sh -v SITE-MAP-URL
- To debug one URL and output on screen:
bash cachecrawler.sh -d SITE-URL
- To display total count of crawl result:
bash cachecrawler.sh -r SITE-MAP-URL
NOTE: Using multiple parameters at the same time is allowed
How to Generate a Sitemap
The Google Sitemap module is quite popular for generating a sitemap in Prestashop, and it's much faster than online generation.
Google Sitemap Module
For v1.6, Google Sitemap Module is installed by default.
For v1.7+, Google Sitemap Module needs to be installed from source first.
Download gsitemap; then change the file name to gsitemap.zip
.
Click the Configure button, you will see e.g. xxx/1_index_sitemap.xml
(This is your main SITE-MAP-URL, ).
SiteMap Online Generator
One of the popular sitemap generators is XML-Sitemaps.com After the crawl is finished. Click DOWNLOAD YOUR XML SITEMAP FILE and put it where the crawler script can access it.
Crawl Interval
How often do we want to re-initiate the crawling process? This depends on how long it takes to crawl your site and what did you set for Public Cache TTL.
Default TTL is one day(24hr). Maybe you can consider to run the script by cronjob every 12 hours.
E.g. This will run twice a day, at 3:30am/15:30: 30 3/15 * * * path_to_script/cachecrawler.sh SITE-MAP-URL -m -i 0.2
Note: You can also use online crontab tool help you to verify time settings.
How to Verify
By using the browser developer tool, you should see X-LiteSpeed-Cache: hit
at the first view for both desktop and Mobile