Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
litespeed_wiki:cache:lscps:crawler [2019/01/17 21:46]
Eric Leu
litespeed_wiki:cache:lscps:crawler [2019/10/23 17:41] (current)
Eric Leu [More Options]
Line 21: Line 21:
  
 ==== More Options==== ==== More Options====
 +  * ''​-h,​ --help'':​ Show this message and exit.
 +  * ''​-m,​ --with-mobile'':​ Crawl mobile view in addition to default view.
 +  * ''​-c,​ --with-cookie'':​ Crawl with site's cookies.
 +  * ''​-b,​ --black-list'':​ Page will be added to blacklist if HTML status error and no cache. Next run will bypass page.
 +  * ''​-g,​ --general-ua'':​ Use general user-agent instead of lscache_runner for desktop view.
 +  * ''​-i,​ --interval'':​ Change request interval. ''​-i 0.2''​ changes from default 0.1 second to 0.2 seconds.
 +  * ''​-v,​ --verbose'':​ Show complete response header under ''/​tmp/​crawler.log''​.
 +  * ''​-d,​ --debug-url'':​ Test one URL directly. as in ''​sh cachecrawler.sh -v -d http://​example.com/​test.html''​.
 +  * ''​-qs,​--crawl-qs'':​ Crawl sitemap, including URLS with query strings.
 +  * ''​-r,​ --report'':​ Display total count of crawl result.
 +
 +Example commands: ​
   * To get help: ''​bash cachecrawler.sh -h''​   * To get help: ''​bash cachecrawler.sh -h''​
   * To change default interval request from 0.1s to custom NUM value: ''​bash cachecrawler.sh SITE-MAP-URL -i NUM''​   * To change default interval request from 0.1s to custom NUM value: ''​bash cachecrawler.sh SITE-MAP-URL -i NUM''​
 +  * To crawl with cookie set: ''​bash cachecrawler.sh -c SITE-MAP-URL''​
 +  * To store log in ''/​tmp/​crawler.log'':​ ''​bash cachecrawler.sh -v SITE-MAP-URL''​
 +  * To debug one URL and output on screen: ''​bash cachecrawler.sh -d SITE-URL''​
 +  * To display total count of crawl result: ''​bash cachecrawler.sh -r SITE-MAP-URL''​
  
 +NOTE: Using multiple parameters at the same time is allowed ​
 ===== How to Generate a Sitemap===== ===== How to Generate a Sitemap=====
 The Google Sitemap module is quite popular for generating a sitemap in Prestashop, and it's much faster than online generation. ​ The Google Sitemap module is quite popular for generating a sitemap in Prestashop, and it's much faster than online generation. ​
  • Admin
  • Last modified: 2019/10/23 17:41
  • by Eric Leu