This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
litespeed_wiki:cache:litemage2:crawler [2018/07/27 18:17]
Lisa Clarke Proofreading
litespeed_wiki:cache:litemage2:crawler [2019/10/23 17:42] (current)
Eric Leu [More Options]
Line 11: Line 11:
   -[[https://​www.litespeedtech.com/​packages/​litemage2.0/​M2-crawler.sh | Download from here]]   -[[https://​www.litespeedtech.com/​packages/​litemage2.0/​M2-crawler.sh | Download from here]]
   - Change the permissions so that the file is executable: ''​chmod +x M2_crawler.sh''​   - Change the permissions so that the file is executable: ''​chmod +x M2_crawler.sh''​
-  - Run the script: ''​sh M2-crawler.sh SITE-MAP-URL''​+  - Run the script: ''​bash M2-crawler.sh SITE-MAP-URL''​
 ==== More Options==== ==== More Options====
-  * To get help: ''​sh M2-crawler.sh -h''​ +  * ''​-h,​ --help''​Show this message and exit. 
-  * To change default ​interval request from 0.1s to custom NUM value: ''​sh M2-crawler.sh ​SITE-MAP-URL -i NUM''​+  * ''​-m, --with-mobile'':​ Crawl mobile view in addition to default view. 
 +  * ''​-c, --with-cookie''​: Crawl with site's cookies. 
 +  * ''​-b,​ --black-list'':​ Page will be added to blacklist if HTML status error and no cache. Next run will bypass page. 
 +  * ''​-g,​ --general-ua'':​ Use general user-agent instead of lscache_runner for desktop view. 
 +  * ''​-i,​ --interval'':​ Change ​request ​interval. ''​-i 0.2''​ changes ​from default ​0.1 second ​to 0.2 seconds. 
 +  * ''​-v,​ --verbose''​Show complete response header under ''/​tmp/​crawler.log''​. 
 +  * ''​-d,​ --debug-url'':​ Test one URL directly. as in ''​sh M2-crawler.sh --d http://​example.com/​test.html''​. 
 +  * ''​-qs,​--crawl-qs''​: Crawl sitemap, including URLS with query strings. 
 +  * ''​-r,​ --report'':​ Display total count of crawl result.
 +Example commands: ​
 +  * To get help: ''​bash M2-crawler.sh -h''​
 +  * To change default interval request from 0.1s to custom NUM value: ''​bash M2-crawler.sh SITE-MAP-URL -i NUM''​
 +  * To crawl with cookie set: ''​bash M2-crawler.sh -c SITE-MAP-URL''​
 +  * To store log in ''/​tmp/​crawler.log'':​ ''​bash M2-crawler.sh -v SITE-MAP-URL''​
 +  * To debug one URL and output on screen: ''​bash M2-crawler.sh -d SITE-URL''​
 +  * To display total count of crawl result: ''​bash M2-crawler.sh -r SITE-MAP-URL''​
 +NOTE: Using multiple parameters at the same time is allowed ​
 ===== How to Generate a Sitemap===== ===== How to Generate a Sitemap=====
 Magento 2 has a builtin module for generating a sitemap and it's fast. Magento 2 has a builtin module for generating a sitemap and it's fast.
  • Admin
  • Last modified: 2018/07/27 18:17
  • by Lisa Clarke