Clarifications on purged cache clean-up

A

Andy Badwool

Member
#1
#1
Hello,
I would like a clarification about when purged cache files get deleted.

By reading this: https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:cache:cache-storage

I see that "When LSWS receives a purge request, it will simply mark the related cached objects as dirty. In order to avoid heavy disk operation, it does not delete the actual cache files on disk."

And then, "LSWS will check periodically (triggered by a timer) for expired cache entries based on the entries' defined TTLs. LSWS will select a list of entries based on LRU, and delete the related cache files on disk."

But, it says: "The clean-up process is based on TTL only, it does not check for purge events."

So, to clarify. cache entries that reach their end of life, get deleted automatically periodically.

But what about cache entries that have been purged via a Purge instruction? (such as "Purge All" or purge certain specific ones)

Does that mean that cache entries that have been purged via a purge instruction, will never get deleted from the storage?
Or they do get deleted, but they still have to wait for their original TTL before getting deleted?

At the end of the wiki page it does say:

"When you press the Purge All button, you will not see the cache storage folder get emptied. You don't have to worry about the actual files on disk. LSWS takes care that for you constantly and quietly behind the scenes."

But this phrase kind of collides with "The clean-up process is based on TTL only, it does not check for purge events." which seems to imply that purge events are not taken care by the clean-up process...

So I'm a bit confused as how this works. Could you please clarify?

TLDR: Do cache files purged via purge events get ever deleted from storage? If so, when?

Thanks!
 
N

NiteWave

Administrator
#3
#3
Andy Badwool said:
Does that mean that cache entries that have been purged via a purge instruction, will never get deleted from the storage?
Or they do get deleted, but they still have to wait for their original TTL before getting deleted?
Click to expand...
yes - "they do get deleted, but they still have to wait for their original TTL before getting deleted"

when: when lsws process is idle, it'll actually delete the expired cache files on the disk.
it's like a cron job;
there may be many expired cache files, it may take long time to complete the deletion(especially slow disk io), so "the batch size is small enough that it will not affect the whole server performance"
 
A

Andy Badwool

Member
#4
#4
@NiteWave I see, thanks for the reply.

However, I'm wondering, why do purged entries have to wait for their original TTL before being eligible for deletion? Why are they not already eligible for deletion after being purged? It seems like an unnecessary waste of space to keep purged files around for several days (based on their TTL) if they have already been purged thus marked as invalid.

So why doesn't the cron job also take into account the purged entries via purge action for the deletion process?
 
N

NiteWave

Administrator
#5
#5
while cache is created one by one, purge may purge all or purge by tag, which may need delete for example 100 cached files.

deleting a file need some system calls, deleting 100 files is too expensive for a web server
all visitors may feel the web server hang for a few seconds, although it's just waiting for the kernel to complete the deletion

"Cache Manager SHM" is like database index, can regard it only one file.
"Cached Object" or cached file is like a database record. there may be millions of records.

with current algorithm, when "purge all" or "purge by tag", only operate the index file, mark some records deleted. and return at once to serve new requests, no blocking. so the web server can continue serve high traffic effectively.

so
1. for high traffic websites, which may create many cache files in short time, or
2. some user may set very long TTL
either case may cause big cache space, sysadmin should take the control, using cleancache.sh or cleanlitemage.sh or own script to clean up the cache storage, to prevent the space increase quickly. don't expect the web server can do the heavy-duty disk cleanup job well in all cases. although it may do that well for low traffic web sites
 
A

Andy Badwool

Member
#6
#6
Hello @NiteWave
Thanks for the detailed explanation!

However, I don't mean that files purged via "purge all" or "purge tag" should be deleted immediately. I understand that it's much better to simply mark them as invalid in the index database.

BUT my question is: when the cron job later kicks in to delete expired files, why doesn't it include in the clean up process also the files that have been purged via "purge all" or "purge tag"? Like you said, ""the batch size is small enough that it will not affect the whole server performance", so why can't the batches also include the purged files in the process?

Also, I tried to look into cleancache.sh or cleanlitemage.sh, but even them seem to only target deletion of files with expired TTL, not deletion of files that are no longer valid due to a purge action...
 
N

NiteWave

Administrator
#7
#7
I think the disk clean up algorithm can improve further and further, endlessly.
maybe it gets changed suddenly when our dev has seen a big reason to do it.
can you explain the reason
why it's so necessary to clean up those cached files by "purge all" or "purge tag" events ?

I think current policy is, the disk clean up is a low priority task, so it just use simple algorithm. and can't affect the primary task of a web server.
if disk space has really become problem, sysadmin should kick in.

and it depends on current algorithm on "index database" implementation. you may refer open litespeed source code for deepest understanding.
for example, when "purge tag" event arrive, may delete entries from the index directly. so in order to delete every cached file on disk, you have to search current index to see if it exists. if it exists, not delete it, otherwise delete it. since index is shared by all litespeed workers and update constantly, to access it, you may need get a lock or mutex first. delete 100 cache files, need get or wait for the lock for 100 times. again this low priority task will affect the primary task's performance badly, especially during high traffic.
 
A

Andy Badwool

Member
#8
#8
@NiteWave
NiteWave said:
can you explain the reason
why it's so necessary to clean up those cached files by "purge all" or "purge tag" events ?
Click to expand...
Well, the reason would be because, for example, say that you Set Cache TTL to 10 days. After 2 days for some reason you purge all cache. For the next 8 days, the previous files (useless) will still occupy space on disk for 8 more days. There is little point to them staying around for that long after they're already purged :)

But yeah, of course what you say makes sense too.

Thanks again for the clarifications!
 
You must log in or register to reply here.
Top