Expected memory usage?

[subBlue]

We're half way through the 30 day trial of LS Enterprise and have 20 sites up and running using our Rails based CMS. One thing we have noticed is an increasing memory requirement over a few days. From a clean reboot the memory usage starts about 550Mb then over the course of the first day will increase to about 1.3Gb. By the end of the second day it is at 2.5Gb. I had wondered if there is a memory leak in our CMS, but a ps uax trace shows something like this:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
apache 15050 0.0 0.8 39580 35596 ? SN Oct19 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 24868 0.0 0.8 38472 34448 ? SN Oct19 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 31944 0.0 0.9 41124 37428 ? SN Oct19 0:16 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 32649 0.0 0.8 40116 36216 ? SN Oct19 0:06 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
admin 2584 0.0 0.6 29704 26356 ? SNs 07:18 0:03 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 2623 0.0 0.6 29984 26640 ? SNs 07:19 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
admin 10346 0.0 0.6 29704 26356 ? SNs 08:59 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 13587 0.1 0.6 29984 26640 ? SNs 09:31 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 15510 0.3 0.5 24740 21856 ? SNs 09:51 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 15825 0.4 0.5 24612 21708 ? SNs 09:53 0:01 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 15838 0.5 0.5 24740 21856 ? SNs 09:53 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 16148 1.0 0.5 24608 21604 ? SNs 09:57 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
admin 16247 1.6 0.7 35380 31284 ? SN 09:58 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 16263 4.1 0.8 39796 35700 ? SN 09:58 0:05 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
admin 16284 10.5 0.8 40928 36832 ? SN 09:58 0:13 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 16376 2.8 0.5 24472 21588 ? SNs 09:59 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
admin 16383 7.8 0.8 40508 36412 ? SN 09:59 0:06 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 16444 3.5 0.5 24472 21588 ? SNs 09:59 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 16453 6.2 0.8 40604 36856 ? SN 09:59 0:03 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 16455 4.6 0.6 29968 26676 ? SNs 09:59 0:02 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 16462 0.0 0.6 29968 26680 ? SN 09:59 0:00 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 16493 3.7 1.1 56516 47544 ? SN 09:59 0:01 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
apache 16500 1.4 0.6 28716 24964 ? SN 09:59 0:00 ruby /usr/local/lsws/fcgi-bin/RailsRunner.rb
root 2544 0.0 0.0 4668 2516 ? S 07:18 0:01 lshttpd
apache 2546 0.0 0.0 5816 3840 ? S 07:18 0:01 lshttpd
apache 2547 0.1 0.1 8728 6688 ? S 07:18 0:10 lshttpd


There is nothing else listed in the ps trace to suggest another app hogging the memory. Our rails apps are setup using the default Rails VH template and we are running on a Dual Xeon 2.8GHz with 4Gb RAM and Centos 4.2 From the Real-time server log we only have 4-5 of the sites active most of the time as the others are still in development and the busiest site gets about 2500 visitors a day.

The server is usually complete quiet between 2 and 7am so I'm wondering if the memory usage should be dropping back down each morning? Should I be worried about the amount of memory usage on this server, or is this something that you would expect to see?

many thanks for any suggestions/thoughts.

[xing]

From what you posted, LiteSpeed processes are using 14MB of real ram total, which is normal.

Linux/CentOS will agressively cache file read requests in the file buffer. So If all your processes leave 3G of ram "free", the kernel will use all that ram to cache file i/o. The kernel will use whatever "free/unused" process memory to cache disk requests and relinguish those ram to real processes when the need comes.

What this means that even if LiteSpeed and all your Ruby processes are only using 256MB total, the linux kernel is still free to use and display up to max memory usage.

This is normal and expected behavior.

The most important item is to check "swap" value. It should not be increasing and should hold at a very steady value, only up or down very infrequently.

Use vmstat to see some memory related stats.

[xing]

Forgot to one more thing.

There would be a cause of concern for potential memory leak only if:

Swap value keeps going up with constant activity despite the fact you don't see any process should be using that much memory.

We also can't say about your CMS app and the Ruby app in general. Just make sure you are running the latest Ruby binary and not the not the one that came with CentOS 4.2.

[subBlue]

Thanks for your feedback, you have put my mind at ease :) I didn't realise that the kernal will using the remaining ram to cache i/o. Our swap is consistantly empty so all seems fine.

[dreamscape]

If you want to see what amount of RAM is actually available to applications, you can run `free -m`. The first line will show used & free without the cache included. But the second line will read "-/+ buffers/cache:" and show how much is actually being used and how much is actually free to applications if you take the cache & buffers into account. The cache and buffers are also reported in top.

If you just read the reported "free" physical RAM, it doesn't take long for a Linux box to appear to have eaten a ton of RAM for apparently no reason. I used to get thrown off by this all the time as well until I learned that cached RAM was actually available for applications to use if they needed it.