[not solved] Failed to obtain or reinitialize VMemBuf

[anything]

titile: Failed to obtain or reinitialize VMemBuf. possibly running out of swap space.

have been getting these errors on file uploads to litespeed webserver 4.1.4.

Failed to obtain or reinitialize VMemBuf. possibly running out of swap space.

its happened three times so far, several days between incidents. haven't been able to work out what causes it.
once it starts happening it seems to happen to everyone if the upload is above a certain size (i haven't worked out what size exactly).

restarting litespeed solves the problem.

/tmp has plenty of space
and /tmp/lshttpd's permissions appear to be fine

[NiteWave]

please refer:
http://www.litespeedtech.com/support/forum/showthread.php?t=5284
see if /tmp/lshttpd/swap related?

[anything]

i found that last week when looking into the first occurrence but it didnt seem to help
/tmp/lshttpd/swap's permissions are ok. owned by lsws, the same user as litespeed is running as.

there are no other errors in any log files at the same time of each error.
the upload is not even attempted by the browser, a connection reset error is seen in the browser instantly.

as restarting lsws fixes the bug it also seems to suggest its not a file permission error.

[anything]

the problem has happened again.
6 hours of errors over night preventing all file uploads to our sites.

i've rolled back to 4.1.3

[NiteWave]

just now I experienced same issue
2011-09-21 06:41:47.081 [ERROR] [x.x.x.x:4696-0#pkg] Failed to obtain or reinitialize VMemBuf. possibly running out of swap space.

it's 4.0.20. lsphp 5.1.2 with Server API : LiteSpeed 4.10

so it's not likely an issue between 4.1.3 and 4.1.4

I did a change in php.ini:
upload_tmp_dir = /tmp
(it's not set before.) and restarted lsws.

my test shows upload 1M file was ok, but failed for a 1.8M file.

maybe restart lsws resolved the issue as you observed.
just put more info here.

[anything]

upload_tmp_dir is not set in my php.ini either. sys_get_temp_dir() returns /tmp
should i set it anyway?

/tmp has 4.2gig free. according to graphs it got as low as 3.9gig yesterday.

restarting lsws means that the problem goes away for some time. seems to come back on its own for no reason.

for example, I just uploaded a 10mb file with no problem. while it was uploading I could see /tmp/phpnMUhZ6 getting filled.

[anything]

This problem has returned again today.
Strange that only yesterday I upgraded to lsws 4.1.10.

any suggestion as to what might be the cause?

[NiteWave]

does it happen once just after upgrade, or does it happen times(can check the error log) after upgrade ?

[anything]

I don't know what triggered it last time (perhaps an upgrade that I had forgotten), but this time it seems it probably was the upgrade that caused it.
It happened on every website accepting file uploads until i ran "lswsctrl restart".
so far for the last 2 days no error has been reported.

[NiteWave]

I have similar experience on another website. if it happens again, please report it here to trace it.

[anything]

I have similar experience on another website. if it happens again, please report it here to trace it.

Hi NiteWave, it has happened again.

Discovered by users attempting to upload 7mb video files.
Smaller file uploads worked, but larger ones didn't (I didnt troubleshoot far enough to find the largest size possible)

logs report lsws was restarted automatically at 5am, only 9 hours before the first errors were reported.

restarting lsws seemed to be the only solution again.

i checked lsws's log file for any other errors however there are no other error level reports before it (just a bunch of php errors or 404s)

[mistwang]

could be running out of space in swap directory. move

http://www.litespeedtech.com/docs/webserver/config/general/#swappingDir

to a partition with enough free disk space.

[anything]

/tmp has 4.2gig free. according to graphs it got as low as 3.9gig yesterday.


today the /tmp dir has 6.1gig free.

[anything]

after the last problem i stopped lsws completely, made sure no processes were running, and started it fresh.
no problem yet (its been over a week)

could it be that the graceful restart after an upgrade isnt doing things properly.

[NiteWave]

it's kind of hard-tracing issues ... it may due to other bug fixes in 4.1.11(e.g. memory-leak etc) which fix the issue as well. let's assume 4.1.11 has no such issue any more -- if still happen, please re-open this thread.

[DoM]

Hello,
same issue here:

Failed to obtain or reinitialize VMemBuf. possibly running out of swap space.
and
Failed to create temp file with mkstemp(): No such file or directory

disk of course has giga and giga of space.


LSWS 4.1.11

Waiting for your reply

Regards

[NiteWave]

can you do what suggested in previous post:

1.force-reinstall 4.1.11 to latest version

2.stop lsws completely, then start it, not restart gracefully
use
#/usr/local/lsws/bin/lswsctrl stop
and / or
#killall litespeed
to stop it completely. and use
#pstree -p | grep litespeed
to check if no litespeed process is running

this will only cause down time within 1 minute.
then start litespeed.

if still happen, please continue reporting here.

[DoM]

It's not fixed.

Many customers complain on this new version about:

1) upload that can reach maximum 1mbit,
2) all wordpress permalink do not work anymore
3) this error


I think that 's ALL related to same issue.


Waiting for your reply

Regards

[webizen]

...

1) upload that can reach maximum 1mbit,
...



you mean upload speed at 1Mbps max or upload filesize limit 1MB?

[DoM]

Upload size.

[mistwang]

Hello,
same issue here:

Failed to obtain or reinitialize VMemBuf. possibly running out of swap space.
and
Failed to create temp file with mkstemp(): No such file or directory

disk of course has giga and giga of space.


LSWS 4.1.11

Waiting for your reply

Regards
It is likely that the process' virtual memory has been exhausted
Please check output of

ulimit -v

set it to unlimited in /etc/security/limits.conf, then restart LSWS. Also check the number of available inode for the /tmp partition.

[mistwang]

If you set "Swap Directory"
http://www.litespeedtech.com/docs/webserver/config/general/#swappingDir

to a new location, make sure the directory exists, and has write permission for the user that litespeed run as.

[DoM]

Did it. it's already like this.

New dir is /var/lshttpd/swap with 777 permissions and it worked until version 4.1.10


Waiting for your reply

Regards

[mistwang]

you need to verify parent directory permission with

sudo -u nobody touch /var/lshttpd/swap/tempfile

download the latest 4.1.12 build see if it work properly or not. It should help with the CPU usage issue with your server.

[DoM]

Everything works with permissions.

THere is a fix in 4.1.12 ?


Waiting for your reply

Regards

[mistwang]

Please check the ticket reply about 4.1.12.

[DoM]

Link of 4.1.12 does not work:

http://www.litespeedtech.com/packages/4.0/lsws-4.1.12-ent-x86_64-linux.tar.gz

2012-03-10 11:21:04 ERROR 404: Not Found.

[mistwang]

sorry, I probably mistake you with another client. I will PM you the link.

[anything]

why is this thread marked as solved?
what is the solution?

it has happened again to me today, started happening 8 hours ago. Any file uploads above 1MB or so would fail with Error 500.
Failed to obtain or reinitialize VMemBuf. possibly running out of swap space.

[NiteWave]

ok. marked as [not solved]

[mistwang]

why is this thread marked as solved?
what is the solution?

it has happened again to me today, started happening 8 hours ago. Any file uploads above 1MB or so would fail with Error 500.
Failed to obtain or reinitialize VMemBuf. possibly running out of swap space.
Will a restart fix it?
Please send us the section of error log starting to show this issue to bug@litespeed...

[gnu.smtp]

I am also facing this issue now.

/dev/sda7 2.0G 437M 1.5G 24% /tmp

free -m
total used free shared buffers cached
Mem: 8104 7613 491 0 228 6268
-/+ buffers/cache: 1116 6988
Swap: 4094 1 4093


Error getting is upload files shows Crunching and Litespeed error shows "Failed to obtain or reinitialize VMemBuf. possibly running out of swap space"

If we restart the litepseed it gets fixed for a short while but reappears the same issue.

Is there any permanent fix for this.

Please consider this as a serious issue as we cannot keep restarting the lsws for each upload.

[anything]

could be marked as possibly-solved.

i moved the litespeed temp&swap directory to somewhere outside of /tmp and the problem has gone away.

i believed our crons watching /tmp to remove old untouched files (also aka tmpwatch) were inadvertently removing old litespeed temp files that it hadn't touched in several days. then suddenly litespeed expects the file to be there and it isnt. this also might explain why i was only seeing the error every 2-3 days.

[gnu.smtp]

Litespeed Guys,

Is there any other fixes ? I have also seen issues like litespeed stops responding while runinng the cleancache script (eventhough we have not enabled cache)

[NiteWave]

i moved the litespeed temp&swap directory to somewhere outside of /tmp and the problem has gone away.
thanks.

gnu.smtp,

can you follow above clue and see if the issue gone since then ?

if it's confirmed, it'll help us much to identify the issue then can work out a correct fix.

the difficulty of this issue is, it's not easily to reproduce.

[gnu.smtp]

Hello,

Can you guide me how to make such a change ? Do we need to add cron for removing that also ?

Please let me know the exact steps as I do not want to cause any downtime in our shared server.

[mistwang]

/dev/sda7 2.0G 437M 1.5G 24% /tmp
It is too small for uploading large files, you need to move swapping directory to another partition.

http://www.litespeedtech.com/docs/webserver/config/general/#swappingDir

[gnu.smtp]

Hello,

Can you let me know the optimum value of Max I/oBuffer ?

Also Can I change the swap directory to /home/lshttpd/swap folder and what about the security side of setting like this.

On another note, our old issue when running

nobody 623 0.0 0.0 4636 1180 ? SN 07:54 0:00 /bin/sh /usr/local/lsws/admin/misc/cleancache.sh /tmp/lshttpd/swap
nobody 2694 0.0 0.0 4636 1180 ? SN 07:55 0:00 /bin/sh /usr/local/lsws/admin/misc/cleancache.sh /tmp/lshttpd/swap
root 4432 0.0 0.0 4032 724 pts/7 S+ 08:13 0:00 grep http
stackabl 4622 0.0 0.2 30404 18516 ? SN 08:03 0:00 cpanellogd - http logs for stackabl
nobody 6300 0.0 0.0 4636 1180 ? SN 08:03 0:00 /bin/sh /usr/local/lsws/admin/misc/cleancache.sh /tmp/lshttpd/swap
nobody 17075 0.0 0.0 4332 788 ? DN 08:10 0:00 find /tmp/lshttpd/swap/4 -type f -mtime +1 -delete
nobody 17110 0.0 0.0 4332 788 ? DN 08:10 0:00 find /tmp/lshttpd/swap/4 -type f -mtime +1 -delete
nobody 17122 0.0 0.0 4332 792 ? DN 08:10 0:00 find /tmp/lshttpd/swap/4 -type f -mtime +1 -delete
nobody 17127 0.0 0.0 4332 788 ? DN 08:10 0:00 find /tmp/lshttpd/swap/5 -type f -mtime +1 -delete
nobody 17153 0.0 0.0 4332 744 ? DN 08:10 0:00 find /tmp/lshttpd/swap/5 -type f -mtime +1 -delete
nobody 17159 0.0 0.0 4332 780 ? DN 08:10 0:00 find /tmp/lshttpd/swap/9 -type f -mtime +1 -delete
nobody 17210 0.0 0.0 4332 744 ? DN 08:10 0:00 find /tmp/lshttpd/swap/a -type f -mtime +1 -delete
nobody 19261 0.0 0.0 4636 1180 ? SN 07:59 0:00 /bin/sh /usr/local/lsws/admin/misc/cleancache.sh /tmp/lshttpd/swap
nobody 23399 0.0 0.0 4636 1184 ? SN 07:42 0:00 /bin/sh /usr/local/lsws/admin/misc/cleancache.sh /tmp/lshttpd/swap
nobody 31203 0.0 0.0 4636 1180 ? SN 07:54 0:00 /bin/sh /usr/local/lsws/admin/misc/cleancache.sh /tmp/lshttpd/swap
nobody 31250 0.0 0.0 4636 1176 ? SN 07:45 0:00 /bin/sh /usr/local/lsws/admin/misc/cleancache.sh /tmp/lshttpd/swap


when this runs sites gets hung and restart of litespeed do not have any effect.

Any possible solutions ? also If I change the /tmp to /home/lshttp/swap, is there any where else I need to make change s

[NiteWave]

>Can you let me know the optimum value of Max I/oBuffer ?
usually default value is fine, no need change except you have explicit reason.

Also Can I change the swap directory to /home/lshttpd/swap folder and what about the security side of setting like this.
change /tmp/lshttps/swap to /home/lshttpd/swap, should not bring any security issue.

also If I change the /tmp to /home/lshttp/swap, is there any where else I need to make changes
correction: it's not change /tmp to /home/lshttp/swap, but change /tmp/lshttps/swap to /home/lshttp/swap. only change this setting is ok.

On another note, our old issue when running
thanks for the report. looks the script need improvement to prevent multiple
"find /tmp/lshttpd/swap/4 -type f -mtime +1 -delete" processes running at the same time.

just search our forum:
http://www.litespeedtech.com/support/forum/showthread.php?p=34155
We have tried the same and seems to be much better.

But the question is why the original script is having issue for this server only. We have more than 10 servers running on litespeed and none of those server is having such issues.

we have discussed the same issue before. the problem server now is the same one mentioned in your post 6 months ago ?