Advanced crawler for recache. some ideas and code

[AndreyPopov]

current crawler contain some issues that not recache more parts of site.

- not recache manufacturer product pages (only list of manufacturers)
- recache only main category and two level sub-categories, but for example my site use three and four level sub-categories
- recached only first page. &page=2, &page=3 .... not cached
- whole Product Catalog (product/catalog) not cached

- recache huge numbers of products can exceed php memory limit

- recache all of above required many times that can exceed php max execution limit

that's why some modifications of recache() function in catalog/controller/extension/module/lscache.php


P.S. some features developers already implemented: cache &page= (but only for category, not for manufacturers or all products) and more category levels

 

[AndreyPopov]

first step add function to calculate number of pages that need to recache:

    protected function CountNumberOfPages($filter_data) {

            if (isset($this->request->get['limit'])) {
                $limit = (int)$this->request->get['limit'];
            } else {
                $limit = $this->journal3->themeConfig('product_limit');
            }

            if (defined('JOURNAL3_ACTIVE')) {
                $this->load->model('journal3/filter');

                $filter_data = array_merge($this->model_journal3_filter->parseFilterData(), $filter_data);

                $this->model_journal3_filter->setFilterData($filter_data);

                \Journal3\Utils\Profiler::start('journal3/filter/total_products');

                $product_total = $this->model_journal3_filter->getTotalProducts();

                \Journal3\Utils\Profiler::end('journal3/filter/total_products');
            } else {
                $product_total = $this->model_catalog_product->getTotalProducts($filter_data);
            }

                $num_pages = ceil($product_total / $limit);

        return $num_pages;

    }

 

[AndreyPopov]

second step
add recache of manufacturers products

        echo 'recache manufacturers urls...' . ($cli ? '' : '<br>') . PHP_EOL;
        $this->load->model('catalog/manufacturer');
        foreach ($this->model_catalog_manufacturer->getManufacturers() as $result) {
                $filter_data = array('filter_manufacturer_id'  => $result['manufacturer_id']);
        $num_pages = $this->CountNumberOfPages($filter_data);

            if ( $num_pages > 1 ) {
                for ($num_page = 1 ; $num_page <= $num_pages ;  $num_page++ ) {
                    if ( $num_page == 1 ) {
                        $urls[] = $this->url->link('product/manufacturer/info', 'manufacturer_id=' . $result['manufacturer_id']);
                    } else {
                            $urls[] = $this->url->link('product/manufacturer/info', 'manufacturer_id=' . $result['manufacturer_id'] . '&page=' . $num_page);
                    }
                }
            } else {
                     $urls[] = $this->url->link('product/manufacturer/info', 'manufacturer_id=' . $result['manufacturer_id']);
            }

        }
        $this->crawlUrls($urls, $cli);
        $urls = array();

 

[AndreyPopov]

third step
add recache to whole Product Catalog

        echo 'recache whole product catalog urls...' . ($cli ? '' : '<br>') . PHP_EOL;

            $filter_data = array('filter_category_id'  => 0);
            $num_pages = $this->CountNumberOfPages($filter_data);
            for ($num_page = 1 ; $num_page <= $num_pages ;  $num_page++ ) {
                if ( $num_page == 1 ) {
                      $urls[] = $this->url->link('product/catalog', '');
                } else {
                      $urls[] = $this->url->link('product/catalog', 'page=' . $num_page);
                }
            }


        $this->crawlUrls($urls, $cli);
        $urls = array();

 

[AndreyPopov]

fourth step
add third and fourth levels of sub-categories to recache

        $this->load->model('catalog/category');
        $this->load->model('catalog/product');
        
        $categories_1 = $this->model_catalog_category->getCategories(0);
        $categoryPath = array();

        echo 'recache category urls...' . ($cli ? '' : '<br>') . PHP_EOL;
        
        foreach ($categories_1 as $category_1) {
            $categoryPath[$category_1['category_id']] = $category_1['category_id'];

            $categories_2 = $this->model_catalog_category->getCategories($category_1['category_id']);

            foreach ($categories_2 as $category_2) {
                $categoryPath[$category_2['category_id']] = $category_1['category_id'] . '_' . $category_2['category_id'];

                $categories_3 = $this->model_catalog_category->getCategories($category_2['category_id']);

                foreach ($categories_3 as $category_3) {
                    $categoryPath[$category_3['category_id']] = $category_1['category_id'] . '_' . $category_2['category_id'] . '_' .  $category_3['category_id'];

                        $categories_4 = $this->model_catalog_category->getCategories($category_3['category_id']);

                        foreach ($categories_4 as $category_4) {
                            $categoryPath[$category_4['category_id']] = $category_1['category_id'] . '_' . $category_2['category_id'] . '_' .  $category_3['category_id'] . '_' .  $category_4['category_id'];

                            $categories_5 = $this->model_catalog_category->getCategories($category_4['category_id']);

                            foreach ($categories_5 as $category_5) {
                                $categoryPath[$category_5['category_id']] = $category_1['category_id'] . '_' . $category_2['category_id'] . '_' .  $category_3['category_id'] . '_' .  $category_4['category_id'] . '_' .  $category_5['category_id'];

                                $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . '_' . $category_5['category_id']);
                            }


                        $filter_data = array('filter_category_id'  => $category_4['category_id']);
                        $num_pages = $this->CountNumberOfPages($filter_data);

                            if ( $num_pages > 1 ) {
                                for ($num_page = 1 ; $num_page <= $num_pages ;  $num_page++ ) {
                                    if ( $num_page == 1 ) {
                                    $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id']);
                                    } else {
                                    $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . '&page=' . $num_page);
                                    }
                                }
                            } else {
                                $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id']);
                                }
                        }

                        $filter_data = array('filter_category_id'  => $category_3['category_id']);
                        $num_pages = $this->CountNumberOfPages($filter_data);

                            if ( $num_pages > 1 ) {
                                for ($num_page = 1 ; $num_page <= $num_pages ;  $num_page++ ) {
                                    if ( $num_page == 1 ) {
                                    $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id']);
                                    } else {
                                    $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '&page=' . $num_page);
                                    }
                                }
                            } else {
                    $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id']);
                                }

                }

                $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id'] . '_' . $category_2['category_id']) ;
            }

            $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id']);
        }
        $this->crawlUrls($urls, $cli);
        $urls = array();

 

[AndreyPopov]

fifth step
add code to choose what recache:

        // choose what recache
        $catalog = true;
        $manufacturer = true;
        $categories = true;
        $products = true;

            if ( isset($this->request->get['what']) ) {
                $what_recache = $this->request->get['what'];
                switch ( $what_recache ) {
                    case "1111":
                    case "all": $catalog = true;$manufacturer = true;$categories = true;$products = true;break;
                    case "1000":
                    case "catalog": $catalog = true;$manufacturer = false;$categories = false;$products = false;break;
                    case "100":
                    case "manufacturer": $catalog = false;$manufacturer = true;$categories = false;$products = false;break;
                    case "10":
                    case "categories": $catalog = false;$manufacturer = false;$categories = true;$products = false;break;
                    case "1":
                    case "products": $catalog = false;$manufacturer = false;$categories = false;$products = true;break;
                    case "1001": $catalog = true;$manufacturer = false;$categories = false;$products = true;break;
                    case "1011": $catalog = true;$manufacturer = false;$categories = true;$products = true;break;
                    case "1101": $catalog = true;$manufacturer = true;$categories = false;$products = true;break;
                    case "11": $catalog = false;$manufacturer = false;$categories = true;$products = true;break;
                    case "101": $catalog = false;$manufacturer = true;$categories = false;$products = true;break;
                    default: $catalog = true;$manufacturer = true;$categories = true;$products = true;
                }
            }

 

[AndreyPopov]

final step
advanced recache() function

 

[AndreyPopov]

now, you can recache all or choose what to recache

1. for recache all
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli"
or
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=all"
or
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=1111"

2. recache only catalog
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=catalog"
or
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=1000"

3. recache only manufacturers
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=manufacturer"
or
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=100"

4. recache only categories
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=categories"
or
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=10"

5. recache only products
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=products"
or
curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=1"

6. or choose combination catalog,manufactrer,categories,products

curl -N "http://yoursite/index.php?route=extension/module/lscache/recache&from=cli&what=XXXX"


where

             X X X X
             | | | - products (0 or 1)
             | | - categories (0 or 1)
             | - manufacturer (0 or 1)
             - catalog (0 or 1)

 

[AndreyPopov]

next steps to advanced recache crawler.

some ideas released and some problems partially solved in this thread

Full recache of huge number of products and PHP max_execution_time 3600s



and next steps are:
Opencart have
- some kind of sorts of product views in catalog,categories,manufacturers
- 15, 25, 50 or 100 (or other) limit of products per page

these views add to url additional parameters
- sort=pd.name&order=ASC
- limit=25

 

[Lee]

This looks very promising but honestly it's over my head as to where to add this to the extension.

Any way to just post the entire file(s) modified and ready to go so I don't mess up? I have no idea where in the file to put your code!

 

[AndreyPopov]

I have no idea where in the file to put your code!

attached files contain code to whole replace
private function recache()

and add additional function to count page.

at this time I add features:
- to add any filter or sorts by GUI
- recache filter/sorts with pages
- add code for prevent problem with php max_execution_time 3600secs not to recache already cached urls

 

[AndreyPopov]

Advanced Crawler READY!!!!!

main features:
- solved exceed php memory limit and php max_execution_time on some hosts
- add recache for catalog
- add recache for manufacturers
- add pages (&page=2.&page=3,etc.) on all possible variants catalog,manufacturers,categories and sorts/filters
- add possibility recache manually added by GUI sorts and/or filters
- add to recache Journal3 QuickView popup and manually added by GUI options required popups
- add possibility to choose what to recache by cli or by GUI

code posted below.....

 

[AndreyPopov]

How modified code to use Advanced Crawler

in file catalog/controller/extension/module/lscache.php

add at the end
- functions to build product list urls

    protected function BuildListOfProductUrls($categoryPath1) {
       
        $UrlsCount1 = 0;
        $this->db->query("CREATE TABLE IF NOT EXISTS `" . DB_PREFIX . "lscache_product_urls_list` ( `url_list_id` int(11) NOT NULL AUTO_INCREMENT, `lscache_product_url` varchar(255) NOT NULL, `recache_status` tinyint(1) NOT NULL, PRIMARY KEY (`url_list_id`)) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;");
        foreach ($this->model_catalog_product->getProducts() as $result) {
            foreach ($this->model_catalog_product->getCategories($result['product_id']) as $category) {
                if(isset( $categoryPath1[$category['category_id']] )){
                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_product_urls_list SET lscache_product_url = '" . 'path=' . $categoryPath1[$category['category_id']] . '&product_id=' . $result['product_id'] . "' ");
                    $UrlsCount1++;
                }
            }

            $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_product_urls_list SET lscache_product_url = '" . 'manufacturer_id=' . $result['manufacturer_id'] . '&product_id=' . $result['product_id'] . "' ");
            $UrlsCount1++;

            $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_product_urls_list SET lscache_product_url = '" . 'product_id=' . $result['product_id'] . "' ");
            $UrlsCount1++;
           
            //Journal3 QuickView url
            if (defined('JOURNAL3_ACTIVE')) {
                $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_product_urls_list SET lscache_product_url = '" . 'product_id=' . $result['product_id'] . '&popup=quickview' . "' ");
                $UrlsCount1++;
            //Journal3 Specific product popup  
                 if($this->lscache->includeFilters) {
                                    foreach($this->lscache->includeFilters as $uri) {
                                        $uri = str_replace('&', '&', $uri);
                                        $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_product_urls_list SET lscache_product_url = '" . 'product_id=' . $result['product_id'] . '&' . $uri . "' ");
                                        $UrlsCount1++;
                                    }
                                }
            }
           
        }
       
        $this->db->query("DELETE FROM " . DB_PREFIX . "setting WHERE store_id = '0' AND `code` = 'module_lscache' AND `key` = 'module_lscache_product_list_recache_status' ");
        $this->db->query("INSERT INTO " . DB_PREFIX . "setting SET store_id = '0', `code` = 'module_lscache', `key` = 'module_lscache_product_list_recache_status', `value` = 'full'");
       
        $this->db->query("DELETE FROM " . DB_PREFIX . "setting WHERE store_id = '0' AND `code` = 'module_lscache' AND `key` = 'module_lscache_product_list_recache_total' ");
        $this->db->query("INSERT INTO " . DB_PREFIX . "setting SET store_id = '0', `code` = 'module_lscache', `key` = 'module_lscache_product_list_recache_total', `value` = '" . $UrlsCount1 . "' ");

        return $UrlsCount1;

    }
   
   
    protected function BuildUrlsListForRecache($FirstItem,$LastItem) {
       
        $UrlsList = array();
        for ($num_item = $FirstItem ; $num_item <= $LastItem ;  $num_item++ ) {
            $PathToRecache = (array)$this->db->query("SELECT `lscache_product_url` FROM `" . DB_PREFIX . "lscache_product_urls_list` WHERE url_list_id = '" . $num_item ."' " );
            if (strpos($PathToRecache['row']['lscache_product_url'], 'popup') !== FALSE) {
                $UrlsList[] = $this->url->link('journal3/product', $PathToRecache['row']['lscache_product_url'] );
            } else {
            $UrlsList[] = $this->url->link('product/product', $PathToRecache['row']['lscache_product_url'] );
            }
        }
        //error_log(print_r($UrlsList,true));
        return $UrlsList;
    }

 

[AndreyPopov]

- functions to build catalog urls

    protected function BuildCatalogUrls() {
        
        $UrlsCount1 = 0;
        $this->db->query("CREATE TABLE IF NOT EXISTS `" . DB_PREFIX . "lscache_catalog_urls_list` ( `url_list_id` int(11) NOT NULL AUTO_INCREMENT, `lscache_catalog_url` varchar(255) NOT NULL, `recache_status` tinyint(1) NOT NULL, PRIMARY KEY (`url_list_id`)) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;");
        
            $filter_data = array('filter_category_id'  => 0);
            $num_pages = $this->CountNumberOfPages($filter_data);
                  $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_catalog_urls_list SET lscache_catalog_url = '' ");
                  $UrlsCount1++;
                              if($this->lscache->includeSorts) {
                                    foreach($this->lscache->includeSorts as $uri) {
                                        $uri = str_replace('&', '&', $uri);
                                        $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_catalog_urls_list SET lscache_catalog_url = '" . $uri . "' ");
                                        $UrlsCount1++;
                                    }
                                }
            for ($num_page = 2 ; $num_page <= $num_pages ;  $num_page++ ) {
                      $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_catalog_urls_list SET lscache_catalog_url = '" . 'page=' . $num_page . "' ");
                      $UrlsCount1++;
                              if($this->lscache->includeSorts) {
                                    foreach($this->lscache->includeSorts as $uri) {
                                        $uri = str_replace('&amp;', '&', $uri);
                                        $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_catalog_urls_list SET lscache_catalog_url = '" . $uri . '&page=' . $num_page . "' ");
                                        $UrlsCount1++;
                                    }
                                }
            }
        
        $this->db->query("DELETE FROM " . DB_PREFIX . "setting WHERE store_id = '0' AND `code` = 'module_lscache' AND `key` = 'module_lscache_catalog_recache_status' ");
        $this->db->query("INSERT INTO " . DB_PREFIX . "setting SET store_id = '0', `code` = 'module_lscache', `key` = 'module_lscache_catalog_recache_status', `value` = 'full'");
        
        $this->db->query("DELETE FROM " . DB_PREFIX . "setting WHERE store_id = '0' AND `code` = 'module_lscache' AND `key` = 'module_lscache_catalog_recache_total' ");
        $this->db->query("INSERT INTO " . DB_PREFIX . "setting SET store_id = '0', `code` = 'module_lscache', `key` = 'module_lscache_catalog_recache_total', `value` = '" . $UrlsCount1 . "' ");

        return $UrlsCount1;

    }


    protected function BuildCatalogUrlsListForRecache($FirstItem,$LastItem) {
        
        $UrlsList = array();
        for ($num_item = $FirstItem ; $num_item <= $LastItem ;  $num_item++ ) {
            $PathToRecache = (array)$this->db->query("SELECT `lscache_catalog_url` FROM `" . DB_PREFIX . "lscache_catalog_urls_list` WHERE url_list_id = '" . $num_item ."' " );
            $UrlsList[] = $this->url->link('product/catalog', $PathToRecache['row']['lscache_catalog_url'] );
        }
        return $UrlsList;
    }

 

[AndreyPopov]

- functions to build manufacturer urls

    protected function BuildManufacturerUrls() {
        
        $UrlsCount1 = 0;
        $this->db->query("CREATE TABLE IF NOT EXISTS `" . DB_PREFIX . "lscache_manufacturer_urls_list` ( `url_list_id` int(11) NOT NULL AUTO_INCREMENT, `lscache_manufacturer_url` varchar(255) NOT NULL, `recache_status` tinyint(1) NOT NULL, PRIMARY KEY (`url_list_id`)) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;");
        
        
        foreach ($this->model_catalog_manufacturer->getManufacturers() as $result) {

                $filter_data = array('filter_manufacturer_id'  => $result['manufacturer_id']);
                     $num_pages = $this->CountNumberOfPages($filter_data);

                     $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_manufacturer_urls_list SET lscache_manufacturer_url = '" . 'manufacturer_id=' . $result['manufacturer_id'] . "' ");
                     $UrlsCount1++;
                                if($this->lscache->includeSorts) {
                                    foreach($this->lscache->includeSorts as $uri) {
                                        $uri = str_replace('&', '&', $uri);
                                        $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_manufacturer_urls_list SET lscache_manufacturer_url = '" . 'manufacturer_id=' . $result['manufacturer_id'] . '&' . $uri . "' ");
                                        $UrlsCount1++;
                                    }
                                }
                for ($num_page = 2 ; $num_page <= $num_pages ;  $num_page++ ) {
                            $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_manufacturer_urls_list SET lscache_manufacturer_url = '" . 'manufacturer_id=' . $result['manufacturer_id'] . '&page=' . $num_page . "' ");
                            $UrlsCount1++;
                                if($this->lscache->includeSorts) {
                                    foreach($this->lscache->includeSorts as $uri) {
                                        $uri = str_replace('&amp;', '&', $uri);
                                        $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_manufacturer_urls_list SET lscache_manufacturer_url = '" . 'manufacturer_id=' . $result['manufacturer_id'] . '&' . $uri . '&page=' . $num_page . "' ");
                                        $UrlsCount1++;
                                    }
                                }
                }
        }
        
        $this->db->query("DELETE FROM " . DB_PREFIX . "setting WHERE store_id = '0' AND `code` = 'module_lscache' AND `key` = 'module_lscache_manufacturer_recache_status' ");
        $this->db->query("INSERT INTO " . DB_PREFIX . "setting SET store_id = '0', `code` = 'module_lscache', `key` = 'module_lscache_manufacturer_recache_status', `value` = 'full'");
        
        $this->db->query("DELETE FROM " . DB_PREFIX . "setting WHERE store_id = '0' AND `code` = 'module_lscache' AND `key` = 'module_lscache_manufacturer_recache_total' ");
        $this->db->query("INSERT INTO " . DB_PREFIX . "setting SET store_id = '0', `code` = 'module_lscache', `key` = 'module_lscache_manufacturer_recache_total', `value` = '" . $UrlsCount1 . "' ");

        return $UrlsCount1;

    }


    protected function BuildManufacturerUrlsListForRecache($FirstItem,$LastItem) {
        
        $UrlsList = array();
        for ($num_item = $FirstItem ; $num_item <= $LastItem ;  $num_item++ ) {
            $PathToRecache = (array)$this->db->query("SELECT `lscache_manufacturer_url` FROM `" . DB_PREFIX . "lscache_manufacturer_urls_list` WHERE url_list_id = '" . $num_item ."' " );
            $UrlsList[] = $this->url->link('product/manufacturer/info', $PathToRecache['row']['lscache_manufacturer_url'] );
        }
        return $UrlsList;
    }

 

[AndreyPopov]

- functions to build categories urls (part1)

    protected function BuildCategoryUrls() {
        
        $UrlsCount1 = 0;
        $this->db->query("CREATE TABLE IF NOT EXISTS `" . DB_PREFIX . "lscache_category_urls_list` ( `url_list_id` int(11) NOT NULL AUTO_INCREMENT, `lscache_category_url` varchar(255) NOT NULL, `recache_status` tinyint(1) NOT NULL, PRIMARY KEY (`url_list_id`)) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;");
        
        $categories_1 = $this->model_catalog_category->getCategories(0);
        
        foreach ($categories_1 as $category_1) {
            $categoryPath[$category_1['category_id']] = $category_1['category_id'];

            $categories_2 = $this->model_catalog_category->getCategories($category_1['category_id']);

            foreach ($categories_2 as $category_2) {
                $categoryPath[$category_2['category_id']] = $category_1['category_id'] . '_' . $category_2['category_id'];

                $categories_3 = $this->model_catalog_category->getCategories($category_2['category_id']);

                foreach ($categories_3 as $category_3) {
                    $categoryPath[$category_3['category_id']] = $category_1['category_id'] . '_' . $category_2['category_id'] . '_' .  $category_3['category_id'];

                        $categories_4 = $this->model_catalog_category->getCategories($category_3['category_id']);

                        foreach ($categories_4 as $category_4) {
                            $categoryPath[$category_4['category_id']] = $category_1['category_id'] . '_' . $category_2['category_id'] . '_' .  $category_3['category_id'] . '_' .  $category_4['category_id'];

                            $categories_5 = $this->model_catalog_category->getCategories($category_4['category_id']);

                            foreach ($categories_5 as $category_5) {
                                $categoryPath[$category_5['category_id']] = $category_1['category_id'] . '_' . $category_2['category_id'] . '_' .  $category_3['category_id'] . '_' .  $category_4['category_id'] . '_' .  $category_5['category_id'];

 

[AndreyPopov]

- functions to build categories urls (part2)

                                $filter_data = array('filter_category_id'  => $category_5['category_id']);
                                $num_pages = $this->CountNumberOfPages($filter_data);
                                $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . '_' . $category_5['category_id'] . "' ");
                                $UrlsCount1++;
                                            if($this->lscache->includeSorts) {
                                                foreach($this->lscache->includeSorts as $uri) {
                                                    $uri = str_replace('&', '&', $uri);
                                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . '_' . $category_5['category_id'] . '&' . $uri . "' ");
                                                    $UrlsCount1++;
                                                }
                                            }
                                    for ($num_page = 2 ; $num_page <= $num_pages ;  $num_page++ ) {
                                        $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . '_' . $category_5['category_id'] . '&page=' . $num_page . "' ");
                                        $UrlsCount1++;
                                            if($this->lscache->includeSorts) {
                                                foreach($this->lscache->includeSorts as $uri) {
                                                    $uri = str_replace('&amp;', '&', $uri);
                                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . '_' . $category_5['category_id'] . '&' . $uri . '&page=' . $num_page . "' ");
                                                    $UrlsCount1++;
                                                }
                                            }
                                    }
                            }

                        $filter_data = array('filter_category_id'  => $category_4['category_id']);
                        $num_pages = $this->CountNumberOfPages($filter_data);

                        $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . "' ");
                        $UrlsCount1++;
                                            if($this->lscache->includeSorts) {
                                                foreach($this->lscache->includeSorts as $uri) {
                                                    $uri = str_replace('&amp;', '&', $uri);
                                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . '&' . $uri . "' ");
                                                    $UrlsCount1++;
                                                }
                                            }
                                for ($num_page = 2 ; $num_page <= $num_pages ;  $num_page++ ) {
                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . '&page=' . $num_page . "' ");
                                    $UrlsCount1++;
                                            if($this->lscache->includeSorts) {
                                                foreach($this->lscache->includeSorts as $uri) {
                                                    $uri = str_replace('&amp;', '&', $uri);
                                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '_' . $category_4['category_id'] . '&' . $uri . '&page=' . $num_page . "' ");
                                                    $UrlsCount1++;
                                                }
                                            }
                                }
                        }

 

[AndreyPopov]

- functions to build categories urls (part3)

                        $filter_data = array('filter_category_id'  => $category_3['category_id']);

                        $num_pages = $this->CountNumberOfPages($filter_data);

                        $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . "' ");

                        $UrlsCount1++;

                                            if($this->lscache->includeSorts) {

                                                foreach($this->lscache->includeSorts as $uri) {

                                                    $uri = str_replace('&', '&', $uri);

                                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '&' . $uri . "' ");

                                                    $UrlsCount1++;

                                                }

                                            }

                                for ($num_page = 2 ; $num_page <= $num_pages ;  $num_page++ ) {

                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '&page=' . $num_page . "' ");

                                    $UrlsCount1++;

                                            if($this->lscache->includeSorts) {

                                                foreach($this->lscache->includeSorts as $uri) {

                                                    $uri = str_replace('&amp;', '&', $uri);

                                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '_' . $category_3['category_id'] . '&' . $uri . '&page=' . $num_page . "' ");

                                                    $UrlsCount1++;

                                                }

                                            }

                                }

                }



                $filter_data = array('filter_category_id'  => $category_2['category_id']);

                $num_pages = $this->CountNumberOfPages($filter_data);

                $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . "' ");

                $UrlsCount1++;

                                            if($this->lscache->includeSorts) {

                                                foreach($this->lscache->includeSorts as $uri) {

                                                    $uri = str_replace('&amp;', '&', $uri);

                                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '&' . $uri . "' ");

                                                    $UrlsCount1++;

                                                }

                                            }

                        for ($num_page = 2 ; $num_page <= $num_pages ;  $num_page++ ) {

                            $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '&page=' . $num_page . "' ");

                            $UrlsCount1++;

                                            if($this->lscache->includeSorts) {

                                                foreach($this->lscache->includeSorts as $uri) {

                                                    $uri = str_replace('&amp;', '&', $uri);

                                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '_' . $category_2['category_id'] . '&' . $uri . '&page=' . $num_page . "' ");

                                                    $UrlsCount1++;

                                                }

                                            }

                        }

            }



            $urls[] =  $this->url->link('product/category', 'path=' . $category_1['category_id']);

            $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . "' ");

            $UrlsCount1++;

                                            if($this->lscache->includeSorts) {

                                                foreach($this->lscache->includeSorts as $uri) {

                                                    $urls[] = $this->url->link('product/category', 'path=' . $category_1['category_id'] . '&' . $uri );

                                                    $uri = str_replace('&amp;', '&', $uri);

                                                    $this->db->query("INSERT INTO " . DB_PREFIX . "lscache_category_urls_list SET lscache_category_url = '" . 'path=' . $category_1['category_id'] . '&' . $uri . "' ");

                                                    $UrlsCount1++;

                                                }

                                            }

        }

      



        $this->db->query("DELETE FROM " . DB_PREFIX . "setting WHERE store_id = '0' AND `code` = 'module_lscache' AND `key` = 'module_lscache_category_recache_status' ");

        $this->db->query("INSERT INTO " . DB_PREFIX . "setting SET store_id = '0', `code` = 'module_lscache', `key` = 'module_lscache_category_recache_status', `value` = 'full'");

      

        $this->db->query("DELETE FROM " . DB_PREFIX . "setting WHERE store_id = '0' AND `code` = 'module_lscache' AND `key` = 'module_lscache_category_recache_total' ");

        $this->db->query("INSERT INTO " . DB_PREFIX . "setting SET store_id = '0', `code` = 'module_lscache', `key` = 'module_lscache_category_recache_total', `value` = '" . $UrlsCount1 . "' ");



        return $UrlsCount1;



    }

please COMBINE "functions to build categories urls part1+part2+part3 !!!!!!

 

[AndreyPopov]

- functions to build categories urls (part4)

    protected function BuildCategoryUrlsListForRecache($FirstItem,$LastItem) {
        
        $UrlsList = array();
        for ($num_item = $FirstItem ; $num_item <= $LastItem ;  $num_item++ ) {
            $PathToRecache = (array)$this->db->query("SELECT `lscache_category_url` FROM `" . DB_PREFIX . "lscache_category_urls_list` WHERE url_list_id = '" . $num_item ."' " );
            $UrlsList[] = $this->url->link('product/category', $PathToRecache['row']['lscache_category_url'] );
        }
        return $UrlsList;
    }

 

[Lee]

Please forgive my ignorance but I've only been at this for a few days but what I understand is these are functions that need to be 'called' in order to be executed.

Where do you place the calls?

And THANK YOU for spending the time and effort to improve this extension for us, very very much appreciated.

Lee