Differences

This shows you the differences between two versions of the page.

--- litespeed_wiki:cache:litemage2:crawler [2019/04/04 16:12]
Lisa Clarke [More Options]
+++ litespeed_wiki:cache:litemage2:crawler [2019/10/23 16:02]
Eric Leu [More Options]
@@ Line 14: / Line 14: @@
 ==== More Options====
+  * ''-h, --help''          Show this message and exit
+  * ''-m, --with-mobile''   Crawl mobile view in addition to default view
+  * ''-c, --with-cookie''   Crawl with site's cookies
+  * ''-b, --black-list''    Page will be added to black list if html status error and no cache. Next run will bypas page
+  * ''-g, --general-ua''    Use general user-agent instead of lscache_runner for desktop view
+  * ''-i, --interval''      Change request interval. "-i 0.2" changes from default 0.1s to 0.2s
+  * ''-v, --verbose''       Show complete response header under /tmp/crawler.log
+  * ''-d, --debug-url''     Test one URL directly. "sh M2-crawler.sh -v -d http://example.com/test.html"
+  * ''-qs,--crawl-qs''      Crawl sitemap, including URLS with query strings
+  * ''-r, --report''        Display total count of crawl result
+Example command:
   * To get help: ''bash M2-crawler.sh -h''
   * To change default interval request from 0.1s to custom NUM value: ''bash M2-crawler.sh SITE-MAP-URL -i NUM''
@@ Line 19: / Line 30: @@
   * To store log in ''/tmp/M2-crawler.log'': ''bash M2-crawler.sh -v SITE-MAP-URL''
   * To debug one URL and output on screen: ''bash M2-crawler.sh -d SITE-URL''
+  * To display total count of crawl result: ''bash M2-crawler.sh -r SITE-MAP-URL''
+  * Use multiple parameters at the same time is allowed
 ===== How to Generate a Sitemap=====