Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
litespeed_wiki:cache:litemage2:crawler [2018/07/27 18:17] Lisa Clarke Proofreading |
litespeed_wiki:cache:litemage2:crawler [2019/10/23 16:22] Lisa Clarke [More Options] Copyediting |
||
|---|---|---|---|
| Line 11: | Line 11: | ||
| -[[https://www.litespeedtech.com/packages/litemage2.0/M2-crawler.sh | Download from here]] | -[[https://www.litespeedtech.com/packages/litemage2.0/M2-crawler.sh | Download from here]] | ||
| - Change the permissions so that the file is executable: ''chmod +x M2_crawler.sh'' | - Change the permissions so that the file is executable: ''chmod +x M2_crawler.sh'' | ||
| - | - Run the script: ''sh M2-crawler.sh SITE-MAP-URL'' | + | - Run the script: ''bash M2-crawler.sh SITE-MAP-URL'' |
| ==== More Options==== | ==== More Options==== | ||
| - | * To get help: ''sh M2-crawler.sh -h'' | + | * ''-h, --help'': Show this message and exit. |
| - | * To change default interval request from 0.1s to custom NUM value: ''sh M2-crawler.sh SITE-MAP-URL -i NUM'' | + | * ''-m, --with-mobile'': Crawl mobile view in addition to default view. |
| + | * ''-c, --with-cookie'': Crawl with site's cookies. | ||
| + | * ''-b, --black-list'': Page will be added to blacklist if HTML status error and no cache. Next run will bypass page. | ||
| + | * ''-g, --general-ua'': Use general user-agent instead of lscache_runner for desktop view. | ||
| + | * ''-i, --interval'': Change request interval. ''-i 0.2'' changes from default 0.1 second to 0.2 seconds. | ||
| + | * ''-v, --verbose'': Show complete response header under ''/tmp/crawler.log''. | ||
| + | * ''-d, --debug-url'': Test one URL directly. as in ''sh M2-crawler.sh -v -d http://example.com/test.html''. | ||
| + | * ''-qs,--crawl-qs'': Crawl sitemap, including URLS with query strings. | ||
| + | * ''-r, --report'': Display total count of crawl result. | ||
| + | Example commands: | ||
| + | * To get help: ''bash M2-crawler.sh -h'' | ||
| + | * To change default interval request from 0.1s to custom NUM value: ''bash M2-crawler.sh SITE-MAP-URL -i NUM'' | ||
| + | * To crawl with cookie set: ''bash M2-crawler.sh -c SITE-MAP-URL'' | ||
| + | * To store log in ''/tmp/M2-crawler.log'': ''bash M2-crawler.sh -v SITE-MAP-URL'' | ||
| + | * To debug one URL and output on screen: ''bash M2-crawler.sh -d SITE-URL'' | ||
| + | * To display total count of crawl result: ''bash M2-crawler.sh -r SITE-MAP-URL'' | ||
| + | |||
| + | NOTE: Using multiple parameters at the same time is allowed | ||
| ===== How to Generate a Sitemap===== | ===== How to Generate a Sitemap===== | ||
| Magento 2 has a builtin module for generating a sitemap and it's fast. | Magento 2 has a builtin module for generating a sitemap and it's fast. | ||