Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
litespeed_wiki:cache:litemage2:crawler [2018/07/27 18:02]
Lisa Clarke [How to Verify]
litespeed_wiki:cache:litemage2:crawler [2019/10/23 16:22]
Lisa Clarke [More Options] Copyediting
Line 9: Line 9:
  
 ===== How to Use the Crawler Script===== ===== How to Use the Crawler Script=====
-[[https://​www.litespeedtech.com/​packages/​litemage2.0/​M2-crawler.sh | Download from here]]+  -[[https://​www.litespeedtech.com/​packages/​litemage2.0/​M2-crawler.sh | Download from here]] 
 +  - Change the permissions so that the file is executable: ''​chmod +x M2_crawler.sh''​ 
 +  - Run the script: ''​bash M2-crawler.sh SITE-MAP-URL''​
  
-Change ​the permissions so that the file is executable+==== More Options==== 
-''​chmod +x M2_crawler.sh''​+  * ''​-h,​ --help'':​ Show this message and exit. 
 +  * ''​-m,​ --with-mobile'':​ Crawl mobile view in addition to default view. 
 +  * ''​-c,​ --with-cookie'':​ Crawl with site's cookies. 
 +  * ''​-b,​ --black-list'':​ Page will be added to blacklist if HTML status error and no cache. Next run will bypass page. 
 +  * ''​-g,​ --general-ua'':​ Use general user-agent instead of lscache_runner for desktop view. 
 +  * ''​-i,​ --interval'': ​Change ​request interval. ''​-i 0.2''​ changes from default 0.1 second to 0.2 seconds. 
 +  * ''​-v,​ --verbose''​Show complete response header under ''/​tmp/​crawler.log''​. 
 +  ​* ​''​-d, --debug-url'':​ Test one URL directlyas in ''​sh M2-crawler.sh -v -d http://​example.com/​test.html''​
 +  * ''​-qs,​--crawl-qs'':​ Crawl sitemap, including URLS with query strings. 
 +  * ''​-r,​ --report'':​ Display total count of crawl result.
  
-==== Crawl Desktop&​mobile share same theme==== +Example commands: ​ 
-''​sh M2-crawler.sh SITE-MAP-URL''​ +  * To get help: ''​bash M2-crawler.sh -h''​ 
- +  * To change default interval request from 0.1s to custom NUM value: ''​bash M2-crawler.sh SITE-MAP-URL -i NUM''​ 
-==== More Options==== +  * To crawl with cookie set: ''​bash M2-crawler.sh -c SITE-MAP-URL''​ 
-  * To get help: ''​sh M2-crawler.sh -h''​ +  * To store log in ''/​tmp/​M2-crawler.log'':​ ''​bash M2-crawler.sh -v SITE-MAP-URL''​ 
-  * To change default interval request from 0.1s to custom NUM value: ''​sh M2-crawler.sh SITE-MAP-URL -i NUM''​+  * To debug one URL and output on screen: ''​bash M2-crawler.sh -d SITE-URL''​ 
 +  * To display total count of crawl result: ''​bash M2-crawler.sh -r SITE-MAP-URL''​
  
 +NOTE: Using multiple parameters at the same time is allowed ​
 ===== How to Generate a Sitemap===== ===== How to Generate a Sitemap=====
-The Sitemap ​module ​is build-in ​for generating a sitemap ​in Magento 2, and it's fast. +Magento 2 has a builtin ​module for generating a sitemap and it's fast.
  
 ==== Enable sitemap ==== ==== Enable sitemap ====
-Navigate to Magento ​admin page -> Stores ​-> Settings ​-> Configuration ​-> Catalog ​-> XML Sitemap ​\\ +Navigate to **Magento ​Admin > Stores > Settings > Configuration > Catalog > XML Sitemap** 
-{{:​litespeed_wiki:​cache:​litemage2:​m2-4.png?​600|}} ​\\+{{:​litespeed_wiki:​cache:​litemage2:​m2-4.png?​600|}}
  
-Set Generation Settings Enabled to ''​Yes'' ​\\+Set **Generation Settings ​Enabled** to ''​Yes''​
 {{:​litespeed_wiki:​cache:​litemage2:​m2-5.png?​600|}} {{:​litespeed_wiki:​cache:​litemage2:​m2-5.png?​600|}}
  
-==== Configuring a single sitemap ​for all storefronts ​==== +==== Configuring a Single Sitemap ​for All Storefronts ​==== 
-Navigate to Magento ​admin page -> Marketing ​-> Seo & Search ​-> Sitemap +Navigate to **Magento ​Admin > Marketing > Seo & Search > Sitemap** 
-  - Click **Add Sitemap** button +  - Click the **Add Sitemap** button 
-  - Enter value +  - Enter values 
-    * Filename: ''​sitemap.xml''​ +    ​* **Filename**: ''​sitemap.xml''​ 
-    * Path: ''/''​ +    ​* **Path**: ''/''​ 
-  - Click **Save & Generate** button+  - Click the **Save & Generate** button
  
 {{:​litespeed_wiki:​cache:​litemage2:​m2-2.png?​600|}} \\ {{:​litespeed_wiki:​cache:​litemage2:​m2-2.png?​600|}} \\
-If all went well, a sitemap.xml file will generated in your magento ​2 document root.+If all went well, a ''​sitemap.xml'' ​file will have been generated in your Magento ​2 document root.
  
 ===== Crawl Interval ===== ===== Crawl Interval =====
Line 49: Line 62:
 E.g. This will run twice a day, at 3:​30am/​15:​30:​ ''​30 3/15 * * * path_to_script/​M2_crawler.sh SITE-MAP-URL -m -i 0.2''​ E.g. This will run twice a day, at 3:​30am/​15:​30:​ ''​30 3/15 * * * path_to_script/​M2_crawler.sh SITE-MAP-URL -m -i 0.2''​
  
-Note: You can also use [[https://​crontab.guru/​|online crontab tool]] help you to verify time settings.+Note: You can also use [[https://​crontab.guru/​|online crontab tool]] ​to help you to verify ​the time settings.
  
-===== How to Verify ===== +===== How to Verify ​the Crawler is Working ​===== 
-When using [[https://​developers.google.com/​web/​tools/​chrome-devtools/​|the browser developer tool]], ​you should see ''​X-LiteSpeed-Cache:​ hit,​litemage'' ​at the first view.+When using [[https://​developers.google.com/​web/​tools/​chrome-devtools/​|the browser developer tool]], ​load a previously uncached page. You should see ''​X-LiteSpeed-Cache:​ hit,​litemage'' ​on the first view.
  
 {{:​litespeed_wiki:​cache:​litemage2:​m2-3.png?​600|}} {{:​litespeed_wiki:​cache:​litemage2:​m2-3.png?​600|}}
  • Admin
  • Last modified: 2020/07/08 19:35
  • by Lisa Clarke