Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision | Next revision Both sides next revision | ||
litespeed_wiki:cache:litemage2:crawler [2019/04/04 16:12] Lisa Clarke [More Options] |
litespeed_wiki:cache:litemage2:crawler [2019/10/23 16:02] Eric Leu [More Options] |
||
---|---|---|---|
Line 14: | Line 14: | ||
==== More Options==== | ==== More Options==== | ||
+ | * ''-h, --help'' Show this message and exit | ||
+ | * ''-m, --with-mobile'' Crawl mobile view in addition to default view | ||
+ | * ''-c, --with-cookie'' Crawl with site's cookies | ||
+ | * ''-b, --black-list'' Page will be added to black list if html status error and no cache. Next run will bypas page | ||
+ | * ''-g, --general-ua'' Use general user-agent instead of lscache_runner for desktop view | ||
+ | * ''-i, --interval'' Change request interval. "-i 0.2" changes from default 0.1s to 0.2s | ||
+ | * ''-v, --verbose'' Show complete response header under /tmp/crawler.log | ||
+ | * ''-d, --debug-url'' Test one URL directly. "sh M2-crawler.sh -v -d http://example.com/test.html" | ||
+ | * ''-qs,--crawl-qs'' Crawl sitemap, including URLS with query strings | ||
+ | * ''-r, --report'' Display total count of crawl result | ||
+ | Example command: | ||
* To get help: ''bash M2-crawler.sh -h'' | * To get help: ''bash M2-crawler.sh -h'' | ||
* To change default interval request from 0.1s to custom NUM value: ''bash M2-crawler.sh SITE-MAP-URL -i NUM'' | * To change default interval request from 0.1s to custom NUM value: ''bash M2-crawler.sh SITE-MAP-URL -i NUM'' | ||
Line 19: | Line 30: | ||
* To store log in ''/tmp/M2-crawler.log'': ''bash M2-crawler.sh -v SITE-MAP-URL'' | * To store log in ''/tmp/M2-crawler.log'': ''bash M2-crawler.sh -v SITE-MAP-URL'' | ||
* To debug one URL and output on screen: ''bash M2-crawler.sh -d SITE-URL'' | * To debug one URL and output on screen: ''bash M2-crawler.sh -d SITE-URL'' | ||
+ | * To display total count of crawl result: ''bash M2-crawler.sh -r SITE-MAP-URL'' | ||
+ | * Use multiple parameters at the same time is allowed | ||
===== How to Generate a Sitemap===== | ===== How to Generate a Sitemap===== |