LiteSpeed Cache for WordPress: Crawler

The crawler travels through your site, refreshing pages that have expired in the cache. This makes it less likely that your visitors will encounter uncached pages.

The crawler must be enabled at the server level, or it will not work. Please contact your hosting provider, or see our instructions.

Before using any of the functions in this area, please be sure that you have configured the crawler to your liking.

1. If you don't already have a sitemap file, you can generate one here by pressing the Generate Crawler File button. (If you already have an XML sitemap, you can enter the URL on the Crawler settings tab.)
2. Set Activate to Enable to turn on the crawler.
3. If you want to watch the crawler's progress, press the Show Crawler Status button.
4. You can wait for the crawler to begin on its own according to its configuration settings, or you can press the Manually Run button and start it immediately. If the crawler is already running, this button will be ignored.

If you've opted to watch the crawler status, your screen will look something like this. The messages in the status window will vary from these, as this screenshot was grabbed from a small installation with few pages to crawl.

This is an explanation:

Size: 181: This means the sitemap files has 181 URLs, one URL per line.

Crawler: #1: This means you are watching action of Crawler number 1 , there could be multiple crawlers working depends on your setting.

Position: 1: This means this crawler is currently fetching the 1st URL from sitemap file.

Threads: 2: This means this is thread number 2, there could be multiple threads fetching, it is smart and will adjust based on your load settings.

Status: Stopped due to reset meta position: This means while it's crawling, site has purged or sitemap changed, so it will restart from top.

If you wish to keep a particular path from being crawled, you may enter it in the Sitemap Generation Blacklist box and press Save. After the crawler has run for the first time, if it encounters any pages marked do-not-cache they will be added to this Blacklist automatically.