Cache Warm Up

This is an old revision of the document!

Note: The settings in this section will take effect when the next cron runs (the default setting is 10 minutes). As a result, there’s no need to flush LiteMage cache.

Set on a per store basis. Allow LiteMage to automatically warm up the cache. Magento cron job must be enabled. Updating the cron.sh PHP_BIN setting to “PHP_BIN=/usr/local/lsws/fcgi-bin/lsphp5” will connect it to LSWS's PHP. For more information, you can check out the magento blog entry on setting up the Magento cron job.

This setting currently has 4 options:
- Store site map: Crawl the list of pages generated directly from the database backend.
- Custom defined URL list: Crawl different custom defined lists of URLs at different intervals and priorities.
- Auto collected (deep crawling) URL list: Crawl a list of pages based on visitor frequency.
- Delta list after tag purge: When a tag is purged, builds and crawls a list of the pages that were purged.
You can select and enable any combination of these options.
You can enable these types of warm up at the global level and disable them for certain stores; or you can disable them at global level and only enable the stores you want to crawl.

Global Setting. This setting tells the crawler where the LiteMage cache server is located so it can directly hit the server instead of having to go through DNS lookup.

This setting is useful if LiteMage is located behind a CDN or proxy server or if it is located on LiteSpeed Load Balancer in front of the crawler node.

Global setting. Warm up will only run when current server load is less than this limit.

The crawler will check the average server load from the last minute using the first value of sys_getloadavg.
If server load is currently high, the crawler will skip this run. During a crawler run, it will also dynamically check this load limit and adjust the thread count, if only 1 thread is used and server load is still above this limit, the crawler will stop.
Load expresses how many processes are waiting in the queue to access the computer processor. This is calculated for a certain period of time, and the smaller the number the better. This should be set based on your server’s hardware. For example, if your server has a total of 4 cores, you can set the load limit to 4.

Maximum Crawler Threads

Global setting. Multi-threading the crawler will help crawl big sites faster. LiteMage will use multiple threads up to this limit and dynamically adjust it based on current server load. Valid range is 1 - 10.

If this number is higher than 1, the crawler will use multiple threads to crawl. The thread number is dynamically adjusted during crawling based on the current server load and the configured load limit, and won’t exceed this maximum number.
Since the thread number changes based on server load, you can safely set this number higher. The recommended value is the total number of your cpu cores.

Global setting. Limit how long (in seconds) Cache Warm Up can run. Warm up cron job is set to run every 10 minutes. Set this value lower than the cron job time interval.

The crawler will be invoked every 10 minutes. You can adjust this value in the LiteMage config.xml file (crontab:jobs:litemage_warmup_cache:schedule).
When the crawler is invoked, it will continue crawling from where it’s left off during the last run. If it finds that any warm-up configuration has updated, it will restart crawling process.
For each cycle, the crawler will run for the amount of time set here, this is to avoid occupying your server by crawling all the time. If your server is busy, you can set this to a smaller value. If your URL list is huge, you can set this to a larger value. Make sure this is set to a value less than 10 minutes as the crawler is relaunched every 10 minutes.
If your server load is higher than the load limit, the crawler will skip one cycle or will exit earlier.

Set on a per store basis. Specifies how often (in seconds) you want to crawl the full list. Crawler request will always be served by backend directly. If you make the interval less than the public TTL, you can keep your cache always warm.

Set on a per store basis. Sets a custom warm up sequence at the store level. This value should be larger than 0, the lower the number, the higher the priority.

You can also define the sequence of stores that the crawler will crawl. Put lower numbers for your more important stores.
This priority setting is for the whole URL list of a store. You can set a different priority for your customized URL list.

Set on a per store basis. Warm up pages for non-default currencies. Set this to “ALL” to include all available currencies, or list only the most used currencies. Space, comma separated currency codes, like “USD,GBP,EUR”.

If your store supports multiple currencies, you can also warm up non-default currency views. You can turn this off by leaving this setting blank.
You can also list only the popular currencies in the order of priority. The default currency will be crawled first, then the crawler will follow the order of your supplied list.

Available since LiteMage version 1.2.0. Replaced Custom Defined URL List, Custom List Crawl Interval (seconds), and Custom List Priority settings.

Set on a per store basis. You can supply a list of files containing customized URL lists with varying intervals and priorities. This can be set by going to the “Current Configuration Scope” box and selecting a store.

Each line should contain the absolute path to a single file, followed by the interval for that file (in seconds) and the for that file priority. The interval must be a number greater than 600. The priority is a number greater than 0, where a lower number means a higher priority. Each file listed should contain a list of custom-defined URLs (one per line). This setting can be used to create a short list of important URLs to be warmed up first or refreshed more frequently.

Absolute paths of the files containing the list of custom-defined URLs. These should contain 1 URL per line. These URLs are relative to the store baseURL (without http://domain/storepath). You can create a short list of the most important URLs, so they can be warmed up first and refreshed more frequently.
The file names here are an absolute path. Make sure you set the proper permissions on these files so they are readable by Magento.
These files can be updated whenever you want. When the crawler is invoked on each cycle, it will check if these files have changed. If they have, it will update its internal cache and restart crawling.
Custom Lists are set at the store level, and will follow the same rules you defined for the store, for example, what currencies to crawl.
These lists only contain the relative URLs. When the crawler runs, it will add the proper base URL for that store. For example, the base URL could be something like https://www.mystore.com/en/ , but your lists should only contain the rest of the URL without its base like “accessories.html”. Use only one URL per line.