Opencart rebuild all cache too slow

vacancy

Active Member
#1
Hi

We have a site in Opencart version 3.0.3.2, we have about 14 thousand products. We use Litespeed opencart cache, when we want to do all rebuild after purge cache, the process continues very slowly. We have increased the php.ini and my.cnf limits on the server, there are no restrictions on the litespeed side. We are using the latest version litespeed and lscache. We have Cpanel control panel.

Why can this be caused?
 

serpent_driver

Well-Known Member
#2
You need a different method to rebuild or to warmup the cache. The current method with internal cache crawler works serial. That means each URL must must be requestet first before the next URL will be requested. This way takes time, but maybe too much time for a higher number of URLs.

Solution 1: Find one who is able to develop a crawler that works parallel to request a defined number of URLs at the same time.
Solution 2: Reduce the number or URLs to the most wanded product pages, because why to warmup the cache for product pages that nobody is interested in?
 

AndreyPopov

Well-Known Member
#3
Hi

We have a site in Opencart version 3.0.3.2, we have about 14 thousand products. We use Litespeed opencart cache, when we want to do all rebuild after purge cache, the process continues very slowly. We have increased the php.ini and my.cnf limits on the server, there are no restrictions on the litespeed side. We are using the latest version litespeed and lscache. We have Cpanel control panel.

Why can this be caused?
same problem. 8k products on store.

at that's why I made "advanced crawler"

you can try ocmod module

you can choose what recache:
- products
- categories
- manufacturers
- catalog
- or combination of above

store 14k urls in memory (for standart crawler) also may exceed php memory limit,
also I have php script time limit 3600s on my hoster - advanced crawler also solve these.



some tricks:
1. use cli command by curl for recache (cron tasks - best solution)
2. to increase crawl speed you also can comment in catalog/controller/extension/module/lscache.php line
usleep(round($diff));
(crawler calculate time for one request and sleep)
 
Last edited:

Germont

Well-Known Member
#4
Installing object cache can spare some resources.
You can also add a second simultaneous cronjob, and speed crawling is increased. No need to edit the files.
 

AndreyPopov

Well-Known Member
#5
You can also add a second simultaneous cronjob, and speed crawling is increased. No need to edit the files.
my hoster:
- block same cron jobs execution
- php script execution limit to 3600 secs

by default crawler recache
- "standard" pages (contacts.sitemap,information etc)
- custom added by GUI pages
- categories (with &pages)
- manufacturer (with &pages)
- and only after products


on my hoster by default crawler recache maximum 800 urls (not recached before) per hour
already recached urls crawler check ~1000urls per 10 minutes

when in my store was 3500 products. default crawler make recache 5-6 hours
but after products became 4100 - recache process never ends:(
products from 3900 to 4100 recached by 10 urls per hour :( :( :(

and these only for one User Agent!

P.S. parallel execution two, three or more cron jobs by standard crawler NOT increase speed, because standard crawler always start from beginning: "standard" pages -> custom pages -> categories -> manufacturers -> products.
if be more accuracy - standard crawler always build new url's lists for crawling.
parallel execution two, three or more cron jobs only heavy load CPU and increase site response time
some reduce time of recache (not increase speed) is start cron jobs with time delays (~30-40 mins) from each other.
only if you have muticore CPU and more RAM for php, you can:
- prevent php memory exceed limit. in my example more than 4000 urls exceed 1MB php memory :(
- execute more than one crawler job
 
Last edited:

Germont

Well-Known Member
#6
@AndreyPopov
I agree, you're right on target.
I would rather try another hosting.
Usually a responsive design is preferred, but with so many pages is A MUST.
Have you ever tried manual preloading with two instances simultaneously?
If second it is started when first already crawled 500 urls, then the second doesn't create more database calls until position ~500 is reached. And it happens in very little time, 25 seconds on my VPS. Then both processes continue at the same speed. Cronjob doesn't wait a browser response, so it recovers the delayed start even faster.
I also mapped cache folder to a TMPFS partition, as the VPS is on a mechanical HDD. Made a dramatic difference.
 
Last edited:

AndreyPopov

Well-Known Member
#7
I also mapped cache folder to a TMPFS partition, as the VPS is on a mechanical HDD. Made a dramatic difference.
I use Shared Hosting Server. Hoster provide NVMe SSD for my tariff plan.
problem not in where cache folder or site stored.

Have you ever tried manual preloading with two instances simultaneously?
If second it is started when first already crawled 500 urls, then the second doesn't create more database calls until position ~500 is reached. And it happens in very little time, 25 seconds on my VPS.
I try many-many variants until products became more than 4000.

after 5000 products only separate recache for categories, manufacturer, products solve problem.

also I use Journal Theme and it's feature webp image format:
Desktop View webp
Mobile View webp
Desktop View no-webp(Safari- iMac)
Mobile View no-webp(Safari - iPhone,iPad)

and also Journal Theme use QuickView Product PopUp and Product Options Price PopUp.

no other way to recache all or these - only advanced crawler

any external crawler NOT cache Journal Theme PopUps!
 
Last edited:

serpent_driver

Well-Known Member
#8
@Germont

To prevent any issues caused by hosting limits you always should use a dedicated server or at least a virtual private server (VPS) that allows to set your own custom (script) limits and enough RAM. A shared hosting is never be able to handle a shop with than 2000 items or more. A HTTP cache like Litespeed makes your shop flying, but it flys first if URLs are cached, but flying itself costs more ressources a shared hosting can have, if you have more than 2000 items. This is a wellknown handicap and not a secret. If you or @AndreyPopov have trouble with cache warmup open your mind and create the minimum of conditions to run a shop system with a higher number of items. You will get all benefits of a HTTP cache first if your shop runs proper without a HTTP cache like Litespeed. Litespeed or any other HTTP cache can't fix issues you already have without a HTTP cache, because (and this is a loop) you have to warmup the cache first.
 

AndreyPopov

Well-Known Member
#9
A shared hosting is never be able to handle a shop with than 2000 items or more. This is a wellknown handicap and not a secret.
may be you right, but I never find recommendations about quantity of product items and what server model use.

may be when number of orders become 200 (two hundreds) per day than VPS or dedicate server will best solution.
now 20-30 orders per day on autumn-winter season and max 10 per day on spring-summer.

I'm admin, not owner. I must work in budget limits :(
owner not ready pay me for VPS/dedicate server support :(

To prevent any issues caused by hosting limits you always should use a dedicated server or at least a virtual private server (VPS) that allows to set your own custom (script) limits and enough RAM.
now shared hosting model provide good variants. now I have 3 core / 6GB RAM (4GB for php)
but year ago (2 core/ 4GB RAM) I can use only 1GB for php

and I think crawler algorithm must be work on all possible variants and not stop on exceed php memory limit or php script time execution limit.
 

serpent_driver

Well-Known Member
#10
may be you right, but I never find recommendations about quantity of product items and what server model use.
You can trust me if I say it. I have more than 20 years experience. If you buy a car, you also won't find any documentation that a car can only be used for driving on streets and not on arables. ;)

may be when number of orders become 200 (two hundreds) per day than VPS or dedicate server will best solution.
now 20-30 orders per day on autumn-winter season and max 10 per day on spring-summer.
The number of orders doesn't matter for calculating which hosting could be the best choice. If you have only 20 to 30 orders a day, but 8000 items in your shop there is a disproportion and you (your owner) don't seem to know what your customers really want. If you sell clothes, but not each clothing can be selled in every season, only offer such clothes that is meant for a certain season. This would dramatically reduce the number of items and also would reduce the need for system resources without any business disadvantage. Less sometimes can mean more.

now shared hosting model provide good variants. now I have 3 core / 6GB RAM (4GB for php)
In VPS these are only virtual resources and can't be compared with resources in a dedicated server.

and I think crawler algorithm must be work on all possible variants and not stop on exceed php memory limit or php script time execution limit.
Okay, you want a car that should can drive, fly, swim, diving, transport a whole school class of kids and much more and all together at the same time. And everything for free of charge! Hm, good idea, but reality looks a bit different. For example if you pay $20 a month for hosting, you only get resources and performance for what $20 costs. Not more not less.

If the owner of your shop isn't willing to pay more, it is your job to convince him that he can't expect to get the most. Otherwise you must learn how to perform magic.... ;)
 

AndreyPopov

Well-Known Member
#11
The number of orders doesn't matter for calculating which hosting could be the best choice. If you have only 20 to 30 orders a day, but 8000 items in your shop there is a disproportion and you (your owner) don't seem to know what your customers really want.
you are wrong!

if summer yarns not requested now(winter), I must remove they from site? or not cache they?
some yarns for pre-order or sale only in pack. pack buy one-two orders per week - not cache they? remove from site?
 

serpent_driver

Well-Known Member
#12
you are wrong!
Why?

if summer yarns not requested now(winter), I must remove they from site? or not cache they?
some yarns for pre-order or sale only in pack. pack buy one-two orders per week - not cache they? remove from site?
Anyway, temporary remove (if possible set items inactive) all items that are not meant for the current season.
 

serpent_driver

Well-Known Member
#14
Okay, I am not an expert fpr selling products in your shop. It was only a suggestion to find new ideas to reduce the load because reducing the load is the only way to solve most of your issues.
 

Germont

Well-Known Member
#15
@AndreyPopov
I hope you don't mind an offtopic suggestion. I've seen your shop and maybe offer should be more diverse.
Women who do home tailoring use more than balls of thread and wool.
There are also textiles for skirts, dresses, and then the client can buy everything he needs from one shop.
That's when the owner can afford a VPS and management for it.
 

AndreyPopov

Well-Known Member
#16
Okay, I am not an expert fpr selling products in your shop. It was only a suggestion to find new ideas to reduce the load because reducing the load is the only way to solve most of your issues.
I already say to you: reduce number of product links NOT way to solve problem because:
a) number of products only grow - new manufacturers(brand), yarn's manufacturer add new collections and new (trend of season) colors
b) disable "not-season" yarn with HIGH search rating and wait until new or season yarn receive same search rating - way to dead end :(
 

AndreyPopov

Well-Known Member
#19

serpent_driver

Well-Known Member
#20
I finding solution and ask because may be anyone already solved same problem.
You know that this forum is for LSWS and not OLS?

no, but I want to know everything :)
No, you are resistant against knowledge.

you always try say like: if you want drive faster then buy new more powerful car or leave spare wheel, tools and passenger seats in garage ;)
This is why you always fail. You either can't understand or you don't want to understand. What I try to make you understand is, that you can't transport 100 people if you have only 5 seats in your car. You understand what the different is?
 
Last edited:
Top