Swap Folder Cleanup

[Tony]

I'd really like to see some sort of cleanup of the swap folder. It stores a few things in there so I'll address these in separate points.

1) The gzip files data stored should have some sort of cleanup of old files mechanism. It seems like it just grows over time so it hits 500MB of gzip files then 2 weeks it's 1GB. I'm sure we could put it in a partition with 100GB and eventually we'd fill it. So some sort of cleaning up mechanism that is used to clean up files not used or written x amount of time ago. I'd obviously say based on access time but not sure if that data would be available to LSWS or not or how it handled it.


2) Cleaning up of partially uploaded files. This does not seem to be happening. So over time it can grow quite large due to this. So I'd like to have it say remove attempted uploaded files after a period of time and there is no connection using it. There is no point in having a bunch of 20MB uploaded files that are not used by anything.



Right now we wipe the swap folder every few weeks but I don't think that's a great solution. We'd rather see LSWS pro actively cleaning up unused files in this folder. There is no point in keeping gzipped files that were last accessed a week and a half ago. There is also no point in keeping partially uploaded files either.

[ffeingol]

I'll second this one.

[mistwang]

OK, we will do something about it.
Actually, those files can be safely removed with "cd swap_dir; rm -rf *".

[ffeingol]

OK, we will do something about it.
Actually, those files can be safely removed with "cd swap_dir; rm -rf *".

So basically we can just rm -rf all the folders under the swap dir and LSWS will just recreate the structure? Does LSWS need to be down when we do that?

[mistwang]

that's correct, not need to bring down LSWS.

[Tony]

OK, we will do something about it.
Actually, those files can be safely removed with "cd swap_dir; rm -rf *".


Looks like it's part of 4.0.16. Any more details on what exactly the cleanup does?

[mistwang]

It calls lsws/admin/misc/cleancache.sh once a day. The behavior can be customized via the shell script. Default is to remove file more than one day old based on modified time. Can be changed to based on last access time, but some system may use "noatime" option when mount the partition.

[ffeingol]

How does it get called (cron etc.) because it's doing nothing on at least one server.

[mistwang]

It was called by LSWS server daemon when server check for new update. So, you should let LSWS check for update daily instead of monthly in order for daily cleanup.

[ffeingol]

check for updates is already set to daily. Is the execution logged anywhere so we can see why it's not working (or maybe not even running)?

[mistwang]

It should have a log message at NOTICE level
"Start to clean up cache directory."

[mistwang]

You can try running "lsws/admin/misc/cleancache.sh <cache_dir>" see if it does anything for you.

[ffeingol]

OK, I can see the messages that it's starting but it's doing nothing. In looking at the script it looks like it's looking for swap/0/file and our directory is swap/0/0/file. Is this just old junk from upgrades?

[NiteWave]

"it's doing nothing" -- may due to a typo in the script: root_dr should be root_dir
find "$root_dr/$subdir" -type f -mtime +$max_age_days -delete

[ffeingol]

OK, I fixed the script (there are actually 2 occurrences of the typo). Will there be a new release that fixes this typo?

[ffeingol]

Still does not appear to work. If I'm reading the script correctly, it's looking for files in:

/cache/0
/cache/1

etc

but the cache directory is actually organized

/cache/0
/cache/0/0
/cache/0/1
/cache/1
/cache/1/0
/cache/1/1

Can anyone else confirm if this is working or not working?

[anewday]

I see this in the error_log right after I upgraded to 4.0.16

Start to clean up cache directory.
2010-08-08 21:38:49.293 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.300 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.301 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.303 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.304 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.306 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.307 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.309 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.310 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.312 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.313 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.315 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.316 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.318 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.319 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.321 [STDERR] find: invalid predicate `-delete'

LSWS is not configured to use the cache feature.

[mistwang]

Still does not appear to work. If I'm reading the script correctly, it's looking for files in:

/cache/0
/cache/1

etc

but the cache directory is actually organized

/cache/0
/cache/0/0
/cache/0/1
/cache/1
/cache/1/0
/cache/1/1

Can anyone else confirm if this is working or not working?

It should find all files under all sub-directories.

[mistwang]

I see this in the error_log right after I upgraded to 4.0.16

Start to clean up cache directory.
2010-08-08 21:38:49.293 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.300 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.301 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.303 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.304 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.306 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.307 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.309 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.310 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.312 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.313 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.315 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.316 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.318 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.319 [STDERR] find: invalid predicate `-delete'
2010-08-08 21:38:49.321 [STDERR] find: invalid predicate `-delete'
LSWS is not configured to use the cache feature.

looks like your find command does not support "-delete" action?

[ffeingol]

Sorry, but this is still doing absolutely nothing. I've changed the:


# /bin/sh


to


# /bin/sh -x


So the execution gets logged to the stderr log. From the logs it's in the right directory and looping through all the various sub-directories but nothing actually gets deleted. If I manually run the find command as a -print instead of -delete it does print the files that should get deleted.

Very stumped.

[mistwang]

How about run the command with "-delete" from command line as nobody user? LSWS change UID to nobody or the user that lshttpd run as before execute the shell script. I wonder if it is a permission problem.

[ffeingol]

The files are all owned by nobody (which is what I would expect). I'm going to leave it run tonight with the -x in place and the stdout log turned on. If it was a permissions issue I'd expect those messages to show up in the stdout log?

[ffeingol]

How about run the command with "-delete" from command line as nobody user? LSWS change UID to nobody or the user that lshttpd run as before execute the shell script. I wonder if it is a permission problem.

Any suggestions on how to do this from command line?

[mistwang]

sudo -u nobody <path_to_cleancache.sh> <path_to_cache_dir>

[ffeingol]

You get an error from ionice (ioprio_set: Operation not permitted).

[mistwang]

I see, looks like need to set I/O priority in server code before giving up super user privilege, then call the script without ionice.

[ffeingol]

It looks like the cleancache.sh script has been totally removed from the latest builds of LSWS? We're still having issues with lots of files building up in the cache dir. This is especially a problem on VPS's where the tmp folder is pretty small (relative to a physical server).