Purge Issues

[Karl]

We have Litespeed proxying request to a backend server, which happens to be Varnish - we do this because Varnish does not have SSL support and the Varnish backend has a lot more memory in it than the webserver.

Unfortunately we are not able to purge items from the Litespeed cache, no matter what we try, they never get purged. It's very random though, sometimes the request is met with a 200 Purged, sometimes with a 405 Method Not Allowed:

curl -X PURGE https://<domain>:8443/karl-test.png -I

HTTP/1.1 200 Purged
Content-Length: 0
Date: Wed, 06 Jul 2016 10:55:05 GMT
Accept-Ranges: bytes
Server: LiteSpeed
Connection: close

Other times:

curl -X PURGE https://<domain>:8443/karl-test.png -I

HTTP/1.1 405 Method Not Allowed
Cache-Control: private, no-cache, no-store, must-revalidate, max-age=0
Pragma: no-cache
Content-Type: text/html
Content-Length: 612
Date: Wed, 06 Jul 2016 10:59:41 GMT
Accept-Ranges: bytes
Server: LiteSpeed
Connection: Keep-Alive

Either way, the request for the image returns:

curl -X GET https://<domain>:8443/karl-test.png -I

HTTP/1.1 200 OK
Content-Type:
X-Varnish: 138641631 129828259
Age: 4035
Via: 1.1 varnish-v4
Accept-Ranges: bytes
X-Litespeed-Cache: hit
Content-Length: 9306
Date: Wed, 06 Jul 2016 10:59:33 GMT
Server: LiteSpeed
Connection: Keep-Alive

However it should return:

curl -X GET http://<varnish>/karl-test.png -I

HTTP/1.1 200 OK
Content-Length: 86152
Content-Type:
Date: Wed, 06 Jul 2016 10:58:00 GMT
X-Varnish: 138641775 136086639
Age: 530
Via: 1.1 varnish-v4
Connection: keep-alive
Accept-Ranges: bytes

As you can see, Varnish has a much larger image (as I swapped a tiny image for a much bigger one) in its cache - which is the same image as is on disk right now. Litespeed resolutely refuses to purge the item from the cache, even when it says it has done it, it's always the original image.

 

[Pong]

Not sure what's your setup and why, maybe you can explain a little bit more, but LiteSpeed cache is a replament of varnish and LiteSpeed is a replacment of your web server. To me, LiteSpeed Cache+ LiteSpeed Web Server will be more than enough to achieve your goal with ssl support and much better performance.

If there is any issue between LiteSpeed cache and LSWS, we will investigate a little further.

For litespeed cache purge itself, may need to know how you purge it.

https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:cache

 

[Karl]

Hi,

We cannot use Litespeed alone to achieve what we need for a couple of reasons:

1) Varnish acts as a load balancer in front of the ultimate back end systems (an application written in go), distributing the load of some CPU intensive processes over many application instances and servers.

2) The web servers do not have enough memory to act as a big enough ramdisk to cache in to for Litespeed - Varnish currently has over 100GB cached in it.

We have Litespeed cache the smaller objects that get hit the most often. It caches them for an hour, or at least it should do, up to 2GB of RAM.

Varnish functions exactly as we expect it to, it caches the output of the processes for 2 days, when we make a PURGE request, it purges the object until it is fetched anew.

The problem is absolutely with the cache in Litespeed - it will not purge the content it has cached, even when the time has expired or when we issue a PURGE request (as shown in my original detailed post).

 

[Pong]

So your setup is:
user <-> LiteSpeed proxy server with LScache enabled <-> Varnish <->Mutiple apache backend web servers, is that right?

What's the LSWS version?
How do you setup the LScache?
How do you purge cache?

Please provde more details steps on how to reproduce the issue.

 

[Karl]

User <-> Litespeed (with lscache enabled) acting as a proxy <-> Varnish <-> Multiple Go backend servers

LSWS - Enterprise 5.0.18 (same issue with 5.0.17 as well)

Enable Public Cache   
No

Check Public Cache   
Yes

Max Object Size   
256K

Cache Expire Time (seconds)   
3600

Cache Stale Age (seconds)   
10

Cache Request with Query String   
Yes

Cache Request with Cookie   
Yes

Cache Response with Cookie   
Yes

Ignore Request Cache-Control   
Yes

Ignore Response Cache-Control   
Yes

Enable Private Cache   
No

Check Private Cache   
Not Set

Private Cache Expire Time (seconds)   
4

In the .htaccess for the proxy domain we then have:

CacheEnable public /
CacheDefaultExpire 3600

RewriteEngine On
RewriteBase /

RewriteCond %{REQUEST_METHOD} ^HEAD|GET|PURGE$
RewriteRule (.*) http://<varnish>:6081/$1 [P,L,E=Cache-Control:max-age=3600]

Purge - As per my original post, by issuing an HTTP PURGE request with the path to the object:

curl -X PURGE https://<domain>:8443/karl-test.png -I

HTTP/1.1 200 Purged
Content-Length: 0
Date: Wed, 06 Jul 2016 10:55:05 GMT
Accept-Ranges: bytes
Server: LiteSpeed
Connection: close

Which may come back with a 200 as above, or it may come back with a 405, it's random. Either way the object is not purged from the cache.

 

[NiteWave]

RewriteCond %{REQUEST_METHOD} ^HEAD|GET|PURGE$ RewriteRule (.*) http://<varnish>:6081/$1 [P,L,E=Cache-Control:max-age=3600]

how about remove PURGE method ? i.e., change to
RewriteCond %{REQUEST_METHOD} ^HEAD|GET$
RewriteRule (.*) http://<varnish>:6081/$1 [P,L,E=Cache-Control:max-age=3600]

 

[Karl]

Uhmm.... because then the request just 404s.

Initially we didn't have a RewriteCond at all, but it made no difference. We added it in as a "just in case" because it is what it says in the docs to see if we could get to the bottom of why it wasn't purging.

I really can't believe a product we pay $46-92 per license for doesn't have proper support. Yet we can pay < $10 for a cPanel license and get world class support within minutes on a product that is many many times more complicated. It's crazy.

 

[NiteWave]

PURGE has some limitation or restriction, to prevent anyone from anywhere can PURGE a cache page.

please try /usr/local/lsws/admin/misc/purge_cache_byurl.php
to purge the cache. while I'll mimic your env and try to reproduce the issue.

 

[NiteWave]

update based on my tests:

1) "curl -X PURGE " works, so no need rely on purge_cache_byurl.php
cache purge wiki:
https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:cache:lscache:purge
please note "Whitelist source IP to issue PURGE request", so need run PURGE on the server or a trusted IP

2) I can reproduce the issue.
if original content is from http proxy, when run PURGE, lsws return "404 Not Found"
on the same server, I did some set up to cache /phpinfo.php, when run PURGE, lsws return "200 Purged"(as expected)

I'll report to our staff for further investigation.

 

[NiteWave]

update 2:

looks my first reply introduced more issues.

I did tests on proxy cache, after remove
RewriteCond %{REQUEST_METHOD} ^HEAD|GET$
cache purge just works as expected.

now I tend to agree with you:
either without "RewriteCond %{REQUEST_METHOD}" or "RewriteCond %{REQUEST_METHOD} ^HEAD|GET|PURGE$
results should be same.

I think the problem you encountered:
curl -X PURGE https://<domain>:8443/karl-test.png -I

1) should remove the last "-I"
2) run PURGE on the server or a trusted IP

and the return should always be "200 Purged"

 

[Karl]

Hi,

They were originally made without in, my test script makes them without it (our script purges Litespeed and Varnish, or whichever the user chooses).

Also these commands are already been run from the server. I'd even added in the servers public IP as trusted and 127.0.0.1T as well before I came to posting on here.

Right now it returns 200 Purged, but it isn't, it's still the same old image coming up. I could try again in a minute and it'll come up with a 405

Thanks,

Karl

 

[NiteWave]

curl -X GET https://<domain>:8443/karl-test.png -I HTTP/1.1 200 OK Content-Type: X-Varnish: 138641631 129828259 Age: 4035 Via: 1.1 varnish-v4 Accept-Ranges: bytes X-Litespeed-Cache: hit Content-Length: 9306 Date: Wed, 06 Jul 2016 10:59:33 GMT Server: LiteSpeed Connection: Keep-Alive

for https://<domain>:8443/karl-test.png, can you try following command to purge (run the command on the web server):

#curl -v -X PURGE -H "host:www.domain.com" https://12.34.56.78:8443/karl-test.png

assume:
1) 12.34.56.78 is www.domain.com's IP.
2) the image is accessed from outside by https://www.domain.com:8443/karl-test.png