Advanced crawler for recache. some ideas and code

AndreyPopov

Well-Known Member
#24
NEXT

- change line
PHP:
private function crawlUrls($urls, $cli = false)
to
PHP:
private function crawlUrls($urls, $cli=false, $showpathcount=false, $pathcount=1, $totalurls = 0 )


- after line
PHP:
$url = str_replace('&', '&', $url);
add
PHP:
$url = str_replace('&amp%3B', '&', $url);
$url = str_replace('?amp%3B', '?', $url);
$url = str_replace('page%3D', '&page=', $url);



- replace lines
PHP:
            if ($cli) {
                echo $current . '/' . $count . ' ' . $url . ' : ' . $httpcode . PHP_EOL;
            } else {
                echo $current . '/' . $count . ' ' . $url . ' : ' . $httpcode . '<br/>' . PHP_EOL;
            }
with
PHP:
                if ( $showpathcount ) {
                    echo $current . '/' . $count . '/' . $pathcount . '/' . $totalurls . ' ' . $url . ' : ' . $httpcode . ($cli ? '' : '<br>') . PHP_EOL;
                } else {
                    echo $current . '/' . $count . ' ' . $url . ' : ' . $httpcode . ($cli ? '' : '<br>') . PHP_EOL;
                }
 
Last edited:

AndreyPopov

Well-Known Member
#25
NEXT

- after lines
PHP:
        $includeUrls = isset($this->lscache->setting['module_lscache_include_urls']) ? explode(PHP_EOL, $this->lscache->setting['module_lscache_include_urls']) : null;
        $this->lscache->includeUrls = $includeUrls;
add
PHP:
        // additional sorts and filters to recache_options
        $includeSorts = isset($this->lscache->setting['module_lscache_include_sorts']) ? preg_split( '/\n|\r\n?/', $this->lscache->setting['module_lscache_include_sorts']) : null;
        $this->lscache->includeSorts=$includeSorts;
        $includeFilters = isset($this->lscache->setting['module_lscache_include_filters']) ? preg_split( '/\n|\r\n?/', $this->lscache->setting['module_lscache_include_filters']) : null;
        $this->lscache->includeFilters=$includeFilters;
 

AndreyPopov

Well-Known Member
#28
NEXT

appear in GUI

to file admin/language/en-gb/extension/module/lscache.php
add
PHP:
// Include Sorts/Filters
$_['entry_include_sorts']     = 'Include Sorts/Filters';
$_['entry_include_filters']     = 'Include Journal Specific';
$_['entry_module_lscache_product_list_recache_total']     = 'Total Product URLs';
$_['entry_module_lscache_product_list_last_recached']     = 'Already Recached Product URLs';
$_['entry_module_lscache_product_list_recache_status']     = 'Product URLs List Status';
$_['hint_include_sorts'] = "Please input one kind of sorts or filter per line. \neg: \nsort=p.date_added&order=DESC \neg: \nfm=11";
$_['hint_include_filters'] = "Please input one Journal specific product url/popup per line. \neg: \npopup=options&product_quantity=1&";
$_['help_include_sorts']     = 'Additional sorts/filters to recache';
$_['help_include_filters']     = 'Additional Journal specific product url/popup to recache';

// What Recache
$_['entry_recache_option_what']     = 'Choose what to recache';
$_['entry_recache_option_what_button']     = 'Recache';
$_['text_recache_what_default']     = 'recache whole site';
$_['text_recache_what_products']     = 'recache only products';
$_['text_recache_what_category']     = 'recache only categories';
$_['text_recache_what_manufacturer']     = 'recache only manufacturers';
$_['text_recache_what_catalog']     = 'recache only catalog';
$_['text_recache_what_category_products']     = 'recache categories and products';
$_['text_recache_what_manufacturer_category_products']     = 'recache manufacturers, categories and products';
$_['text_recache_what_manufacturer_products']     = 'recache manufacturers and products';
$_['text_recache_what_catalog_products']     = 'recache catalog and products';
$_['text_recache_what_catalog_category_products']     = 'recache catalog, categories and products';
$_['text_recache_what_catalog_manufacturer']     = 'recache catalog and manufacturers';
$_['text_recache_what_catalog_manufacturer_products']     = 'recache catalog, manufacturers and products';
$_['button_recacheWhat'] = 'Rebuild Part of LiteSpeed Cache by your choose';
 

AndreyPopov

Well-Known Member
#29
NEXT

to file admin/controller/extension/module/lscache.php

after line
PHP:
$data["button_recacheAll"] .= $data["text_curl_not_support"];
add
PHP:
$data["button_recacheWhat"] .= $data["text_curl_not_support"];

after line
PHP:
$recacheLink = $siteUrl->link('extension/module/lscache/recache', 'user_token=' . $this->session->data['user_token'],true);
add
PHP:
        $this->load->model('extension/module/lscache');
        $settingWhat = $this->model_extension_module_lscache->getItems();
        $recacheLinkWhat = $siteUrl->link('extension/module/lscache/recache', 'what=' . $settingWhat['module_lscache_recache_option_what'] . '&user_token=' . $this->session->data['user_token'],true);


after lines
PHP:
            if(!isset($oldSetting["module_lscache_vary_safari"])){
                $oldSetting["module_lscache_vary_safari"] = '0';
            }
add
PHP:
            // add variables for what recache
            if(!isset($oldSetting["module_lscache_recache_option_what"])){
                $oldSetting["module_lscache_recache_option_what"] = '1111';
            }


after line
PHP:
$data['recacheAll'] = $this->isCurl()? $recacheLink : '#';
add
PHP:
$data['recacheWhat'] = $this->isCurl()? $recacheLinkWhat : '#';





to file admin/model/extension/module/lscache.php

after line
PHP:
$this->db->query(" insert into " . DB_PREFIX . "setting (store_id, code, `key`, value, serialized) values ('0', 'module_lscache', 'module_lscache_vary_safari', '0', '0')") ;
add
PHP:
        // lscache crawler what recache variabels
        $this->db->query(" insert into " . DB_PREFIX . "setting (store_id, code, `key`, value, serialized) values ('0', 'module_lscache', 'module_lscache_recache_option_what', '1111', '0')") ;
 
Last edited:

AndreyPopov

Well-Known Member
#30
NEXT

in file admin/view/template/extension/module/lscache.twig

after lines
PHP:
                <div class="form-group">
                    <label class="col-sm-2 control-label" for="module_lscache_include_urls"><span data-toggle="tooltip" title="{{ help_include_urls }}">{{ entry_include_urls }}</span></label>
                  <div class="col-sm-10">
                    <textarea  name="module_lscache_include_urls" placeholder="{{ hint_include_urls }}" class="form-control" rows="3">{{ module_lscache_include_urls }}</textarea>
                  </div>
                </div>
add
PHP:
                <!-- Include sorts to recache -->
                <div class="form-group">
                    <label class="col-sm-2 control-label" for="module_lscache_include_sorts"><span data-toggle="tooltip" title="{{ help_include_sorts }}">{{ entry_include_sorts }}</span></label>
                  <div class="col-sm-10">
                    <textarea  name="module_lscache_include_sorts" placeholder="{{ hint_include_sorts }}" class="form-control" rows="3">{{ module_lscache_include_sorts }}</textarea>
                  </div>
                </div>


                <!-- Include filter to recache -->
                <div class="form-group">
                    <label class="col-sm-2 control-label" for="module_lscache_include_filetrs"><span data-toggle="tooltip" title="{{ help_include_filters }}">{{ entry_include_filters }}</span></label>
                  <div class="col-sm-10">
                    <textarea  name="module_lscache_include_filters" placeholder="{{ hint_include_filters }}" class="form-control" rows="3">{{ module_lscache_include_filters }}</textarea>
                  </div>
                </div>



after lines
PHP:
                <div class="form-group">
                  <label class="col-sm-2 control-label" for="module_lscache_purge_url">
                      <button type="submit" name='purgeURL' form="form-lscache-module" data-toggle="tooltip" class="btn btn-warning"><i class="fa fa-trash"> &nbsp; {{ entry_purge_urls }}</i></button>
                  </label>
                  <div class="col-sm-10">
                    <textarea  name="lscache_purge_url" placeholder="{{ hint_purge_urls }}" class="form-control" rows="3">{{ lscache_purge_urls }}</textarea>
                  </div>
                </div>
add
PHP:
                <!-- Choose what to recache -->
                <div class="form-group">
                  <label class="col-sm-2 control-label" for="module_lscache_recache_option_what">{{ entry_recache_option_what }}
                    <a href="{{ recacheWhat }}" data-toggle="tooltip" title="{{ button_recacheWhat }}" class="btn btn-success"><i class="fa fa-flash"> &nbsp; </i></a>
                  </label>
                  
                  <div class="col-sm-10">
                    <select name="module_lscache_recache_option_what" class="form-control">
                      <option value="1111" {{ selectDisable.check(module_lscache_recache_option_what, '1111') }}>{{ text_recache_what_default }}</option>
                      <option value="1" {{ selectDisable.check(module_lscache_recache_option_what, '1') }}>{{ text_recache_what_products }}</option>
                      <option value="10" {{ selectDisable.check(module_lscache_recache_option_what, '10') }}>{{ text_recache_what_category }}</option>
                      <option value="100" {{ selectDisable.check(module_lscache_recache_option_what, '100') }}>{{ text_recache_what_manufacturer }}</option>
                      <option value="1000" {{ selectDisable.check(module_lscache_recache_option_what, '1000') }}>{{ text_recache_what_catalog }}</option>
                      <option value="11" {{ selectDisable.check(module_lscache_recache_option_what, '11') }}>{{ text_recache_what_category_products }}</option>
                      <option value="111" {{ selectDisable.check(module_lscache_recache_option_what, '111') }}>{{ text_recache_what_manufacturer_category_products }}</option>
                      <option value="1001" {{ selectDisable.check(module_lscache_recache_option_what, '1001') }}>{{ text_recache_what_catalog_products }}</option>                     
                      <option value="1011" {{ selectDisable.check(module_lscache_recache_option_what, '1011') }}>{{ text_recache_what_catalog_category_products }}</option>
                      <option value="1100" {{ selectDisable.check(module_lscache_recache_option_what, '1100') }}>{{ text_recache_what_catalog_manufacturer }}</option>
                      <option value="1101" {{ selectDisable.check(module_lscache_recache_option_what, '1101') }}>{{ text_recache_what_catalog_manufacturer_products }}</option>
                      <option value="101" {{ selectDisable.check(module_lscache_recache_option_what, '101') }}>{{ text_recache_what_manufacturer_products }}</option>
                    </select>
                  </div>
                </div>
 

AndreyPopov

Well-Known Member
#31
some issues and how-to-use - later


P.S. if anybody know how to replace block of code by ocmod search regex then I can make modification module.
 

Lee

Well-Known Member
#32
Good heavens you put a ton of work into this! Bravo.

I will try and find time today to put it all together and see what happens...
 

AndreyPopov

Well-Known Member
#33
after edit twig file not forget clear SASS and Theme cache (main admin screen, right up corner )

fill fields, make your choose and click Save.

ATTENTION!!!!! after each Save you must press twice "Rebuild Part of LiteSpeed Cache" before run recache process.

first click (click and wait) start initial process that check and prepare variables.
second click (click and wait) start process to build of urls list.
and only third (and next clicks) start recache process.

(I do not find what I must to do that after just writing to database I can read these settings ????????)
 

Lee

Well-Known Member
#35
OK, I tried to recache "category only" and everytime I try it logs me out of opencart. There may be more issues, there may even be issues with the ocmod file so go through it and debug it.
 

AndreyPopov

Well-Known Member
#36
not sure that this construction:
Code:
            <search><![CDATA[public function recache()]]></search>
            <add position="replace" offset="139">
replace all 139 lines.

tag offset means that:
- ocmod engine find string in "search" tag
- count 139 lines from found string
- replace counted 139th line

replace block of code by ocmod possible in two way:
1. in search tag put whole block to be replaced
2. using tag regex=true with regex expression
 

Lee

Well-Known Member
#37
I'm not sure what you are telling me but in your code you wanted to replace the entire recache() function with your version of the function and it does just that.

Edit: I think you are trying to tell me that it will only replace just one line but that's not the case, it replaces the entire recache() function with your version.
 
Last edited:

AndreyPopov

Well-Known Member
#38
I'm not sure what you are telling me but in your code you wanted to replace the entire recache() function with your version of the function and it does just that.

Edit: I think you are trying to tell me that it will only replace just one line but that's not the case, it replaces the entire recache() function with your version.
tag offset really work after replace!
but to correct replace recache() offset must be 140, not 139 :)
 

Lee

Well-Known Member
#39
When I read the instructions yesterday it said the number starts with 0 not 1 so 139 is correct since it's 140 lines long! In any case it works and may add or subtract an empty line but I'm not going to spend time on that!
 
Top