Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision | Next revision Both sides next revision | ||
litespeed_wiki:cache:lscwp:crawler [2018/07/05 12:14] qtwrk |
litespeed_wiki:cache:lscwp:crawler [2018/07/05 14:54] Lisa Clarke [Running the Crawler] Proofreading |
||
---|---|---|---|
Line 19: | Line 19: | ||
{{:litespeed_wiki:cache:lscwp:lscwp-crawler-watch.png?direct&800|}} | {{:litespeed_wiki:cache:lscwp:lscwp-crawler-watch.png?direct&800|}} | ||
+ | If you've opted to watch the crawler status, your screen will look something like the image above. The messages in the status window will vary from these, as this screenshot was grabbed from a small installation with few pages to crawl. | ||
- | + | Here's an example of a watch screen from a crawler running on a larger site: | |
- | If you've opted to watch the crawler status, your screen will look something like this. The messages in the status window will vary from these, as this screenshot was grabbed from a small installation with few pages to crawl. | + | |
{{:litespeed_wiki:cache:lscwp:troubleshooting:lscwp-crawler2.jpg|}} | {{:litespeed_wiki:cache:lscwp:troubleshooting:lscwp-crawler2.jpg|}} | ||
- | This is an explanation: | + | And here is an explanation of some of the terms: |
- | + | * ''Size'': The number of URLs in the sitemap. This example has 181. | |
- | ''Size: 181'': This means the sitemap files has 181 URLs, one URL per line. | + | * ''Crawler'': Indicates which crawler number you are watching. It's number 1 in this example. There could be multiple crawlers working, depending on your settings. |
- | + | * ''Position'': The URL number currently being fetched from the sitemap list. | |
- | ''Crawler: #1'': This means you are watching action of Crawler number 1 , there could be multiple crawlers working depends on your setting. | + | * ''Threads'': Indicates the number of threads currently being used to fetch URLs. There may be multiple threads fetching. It is smart and will adjust based on your load [[litespeed_wiki:cache:lscwp:configuration:crawler|settings]]. |
- | + | * ''Status'': Indicates the current crawler status. In this example, ''Stopped due to reset meta position'' means that the site purged or the sitemap changed while it was crawling, and as such, the crawler will restart from the top. | |
- | ''Position: 1'': This means this crawler is currently fetching the 1st URL from sitemap file. | + | |
- | + | ||
- | ''Threads: 2'': This means this is thread number 2, there could be multiple threads fetching, it is smart and will adjust based on your load [[litespeed_wiki:cache:lscwp:configuration:crawler|settings]]. | + | |
- | + | ||
- | ''Status: Stopped due to reset meta position'': This means while it's crawling, site has purged or sitemap changed, so it will restart from top. | + | |
- | + | ||
- | + | ||
If you wish to keep a particular path from being crawled, you may enter it in the **Sitemap Generation Blacklist** box and press **Save**. After the crawler has run for the first time, if it encounters any pages marked ''do-not-cache'' they will be added to this Blacklist automatically. | If you wish to keep a particular path from being crawled, you may enter it in the **Sitemap Generation Blacklist** box and press **Save**. After the crawler has run for the first time, if it encounters any pages marked ''do-not-cache'' they will be added to this Blacklist automatically. | ||