Failed to generate the realtime reports!

[DraCoola]

Firstly, I suddenly got this :

http://img521.imageshack.us/img521/4768/errorw.th.jpg (http://img521.imageshack.us/my.php?image=errorw.jpg)



Secondly, I switch 4.0.2 to 4.0.1 in hoping to fix that error.

Switched to 4.0.1.
But still got that error and plus I cannot switch back to 4.0.2 again.
I even cannot switch LSWS to Apache in order to reinstall this LSWS to 4.0.2.
Stucked.
In this stucked 4.0.1, LSWS cannot use to access https and cannot read files with 400 chmoded.

Any help? George?

[mistwang]

/tmp is full I think. clean it up.

[DraCoola]

whoa... you are right!
but... how to do that safely?
rm -rf * on /tmp ?

Thank you

[mistwang]

I think it is safe to remove PHP session files, and /tmp/lshttpd/swap

[DraCoola]

Your words is my command!
I'll do that, George!

[DraCoola]

Update

Still stucked...
Cannot switch back to 4.0.2 through WHM LSWS plugin :(

[DraCoola]

still cannot switch to 4.0.2 and apache....
still cannot access https....

would you mind to help me to look at these issue...

thank you!

[auser]

reinstall 4.02 may resolve the issue

[DraCoola]

but reinstall needs to switch to apache first...
while i still got :
----------------------------------------------------------------------
"LiteSpeed Web Server Control Operation ... Switch to Apache

LSWS is still running. Fail to stop within 30 secs

LiteSpeed is still running! Failed to switch to Apache. Please try again!"
-----------------------------------------------------------------------

[DraCoola]

my clients are starting terrorize me by phone now...
they cannot login to their forum and/or their member area on their websites....

[DraCoola]

anyone know how to stop LSWS and then start apache manually from console?
so then I can start to reinstal 4.0.2 immediately...

thanks in advance.

[auser]

FYI:

cd /usr/local/lsws/bin
./lswsctrl stop

cd /usr/local/apache/bin
./apachectl start

[DraCoola]

Thank you auser,

But still LSWS forcing to start as the only webserver.

root@medusa [/tmp]# cd /usr/local/lsws/bin
root@medusa [/usr/local/lsws/bin]# ./lswsctrl stop
[OK] lshttpd: stopped.
root@medusa [/usr/local/lsws/bin]# cd /usr/local/apache/bin
root@medusa [/usr/local/apache/bin]# ./apachectl start
[OK] Send SIGUSR1 to 24651
[OK] Send SIGUSR1 to 24651
[OK] Send SIGUSR1 to 24651
[OK] Send SIGUSR1 to 24651
[OK] Send SIGUSR1 to 24651
[OK] Send SIGUSR1 to 24651
[OK] Send SIGUSR1 to 24651
[OK] Send SIGUSR1 to 24651
[OK] Send SIGUSR1 to 24651
/usr/local/apache/bin/httpd: line 38: 0=0: command not found
root@medusa [/usr/local/apache/bin]#

[DraCoola]

is there any other way to stop this LSWS?
does just kill LSWS pid will safe to do?

can't hold any longer... thank you...

[mistwang]

Just do

killall -9 lshttpd

then

try restart LSWS see if it fix itself.

[anewday]

Set "Auto Reload On Changes" to NO in the admin console and do:

service lsws stop
service httpd start
service httpd restart

[DraCoola]

The fixing steps are :

1. killall -9 lshttpd
2. restart LSWS <--- not fix it self
3. again, killall -9 lshttpd
4. start Apache
5. reinstall 4.0.2 from scratch

pheww...

thanks for all helps, really appreciated :)

[mistwang]

If /tmp/lshttpd/swap use too much disk space, consider moving "Static GZIP Cache Directory"
http://www.litespeedtech.com/docs/webserver/config/tuning/#gzipCacheDir

[DraCoola]

Actually, the main problem that pushed me to run easyapache and then re-setting LSWS is : the cpu load.
Switch to LSWS makes CPU spike randomly to 40-50 from 2.x - 3.x
I don't know what caused this.

Just spikes quickly to 10-20-30-40-50... and then go down to 2.x - 3.x again.
In next couple minutes the CPU load go to super high again and again and again....

Anyone has the same issue?

[DraCoola]

The CPUs are getting wild again....

http://img261.imageshack.us/img261/6121/wild.jpg

:(

[anewday]

What's your lsphp5 environmental settings?

[DraCoola]

What's your lsphp5 environmental settings?

Here they are, bro :
-----------------------------
PHP_LSAPI_MAX_REQUESTS=3000
PHP_LSAPI_CHILDREN=175
LSAPI_MAX_IDLE=5
LSAPI_PGRP_MAX_IDLE=5
LSAPI_MAX_PROCESS_TIME=300
-----------------------------


And again.... :(

http://img412.imageshack.us/img412/9326/wild2.jpg

[mistwang]

Your server start to use swap, that's why you get high load.
your PHP opcode cache size is a bit high to me, since eaccelerator uses /dev/shm, there is no need to set opcode cache too high.
Also, make sure the /dev/shm got cleaned regularly.

[DraCoola]

@George :

PHP opcode cache means.... eaccelerator.shm_size?
Mine is just use the default 16 mb.
So I must to reduce that size again?

Yes I remember you have told me to empty it around every 5 minutes...
Btw I regularly clean my /dev/shm/eaccelerator with hourly cron job.
Is that too long?

[mistwang]

16mb should be fine. you need to find out which process responsible for the swap usage.
monitor the usage of /dev/shm with command "df", I think it should be cleaned more frequently.

[mistwang]

your PHP processes is constantly allocate more than 200MB memory, you need to find out what causes it doing that. Maybe other opcode cache or loader, should reduce the memory usage.

[anewday]

I think PHP_LSAPI_CHILDREN is too high unless your server gets millions of hits per day. What's lsphp5 Max Connections set to?

[mistwang]

PHP_LSAPI_CHILDREN is ignored when PHP suEXEC is used.
http://www.litespeedtech.com/docs/webserver/config/general/#phpSuExec

[DraCoola]

16mb should be fine. you need to find out which process responsible for the swap usage.
monitor the usage of /dev/shm with command "df", I think it should be cleaned more frequently.

Yes it has loaded very fast (it was cleaned up at 5 minutes ago)....

Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 4090756 3926160 164596 96% /dev/shm

I will change my cron job to clean it more frequently.
Is that save to move the eaccelerator.shm_dir to another disk? (that disk is just filled for /home1 only)
Btw.... how to find out which proccess that mostly filled up that cache dir?




your PHP processes is constantly allocate more than 200MB memory, you need to find out what causes it doing that. Maybe other opcode cache or loader, should reduce the memory usage.
I am using eAccelerator only as opcode.
The loaders are zend and ioncube.
There were 2-3 sites which are use ioncube.

[DraCoola]

I think PHP_LSAPI_CHILDREN is too high unless your server gets millions of hits per day. What's lsphp5 Max Connections set to?

lsphp5 Max Connections is 175 (same as PHP_LSAPI_CHILDREN value).



@george :
Yes I use PHP suEXEC, as it shown on my "top" screen shots.
So PHP_LSAPI_CHILDREN value will completely ignored? I'll erase that.

[mistwang]

You can find out how much memory reserved by those loaders from phpinfo() page and reduce it with php.ini .

[DraCoola]

For additional information, some users are randomly just suddenly getting wild with their proccess.
So the CPU load will follows until 40-50 (and that makes LSWS automatically restart with fix 503 error reason)

I found those brutal process on CSF reports.
Here is one for an example :

Time: Tue Apr 28 15:00:58 2009 +0700
Account: waroacom
Process Count: 21 (Killed)

Process Information:

User:waroacom PID:25158 Run Time:11(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25195 Run Time:9(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25203 Run Time:9(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25204 Run Time:9(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25232 Run Time:6(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25244 Run Time:5(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25245 Run Time:5(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25246 Run Time:5(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25253 Run Time:4(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25262 Run Time:4(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25269 Run Time:3(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25276 Run Time:3(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25282 Run Time:3(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25284 Run Time:3(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25287 Run Time:3(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25288 Run Time:2(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25291 Run Time:2(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25293 Run Time:2(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25312 Run Time:0(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25315 Run Time:0(secs) Memory:233788(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5
User:waroacom PID:25316 Run Time:0(secs) Memory:174220(kb) exe:/usr/local/lsws/fcgi-bin/lsphp-5.2.9 cmd:lsphp5


They are many killing process reports at a time from CSF while my CPU scream.

But that wild process has not happen while using mod_suphp...
It is so confusing...

Btw, my php info page --> www.dracoola.com/hph.php

[anewday]

My server shows:

df-h
Filesystem Size Used Avail Use% Mounted on
none 1.5G 71M 1.5G 5% /dev/shm

Doesn't show tmpfs under Filesystem. Is it mounted correctly?

/etc/fstab

# This file is edited by fstab-sync - see 'man fstab-sync' for details

none /dev/shm tmpfs noexec,nosuid 0 0

[DraCoola]

@anewday:

Here is mine :
tmpfs /dev/shm tmpfs defaults,noexec,nosuid 0 0

So... is that mounted correctly?
And how about to move eaccelerator cache dir to dedicate disk which that used for /home1 only?

[mistwang]

You can strace the PHP processes that uses a lot of CPU, if it is from one user, maybe you should check the script from that user or just ban that user see if it helps.
It might be a bug in eaccelerator, so, clean /dev/shm before it is full.

[DraCoola]

So that might be bug from eAcceleartor... noticed that..
Btw I'm not compile eAccelerator by my self.
I have just use /scripts/phpextensionmgr install eAccelerator.
Is that affected?
Because many (all?) forum softwares cannot recognize if there is an eAccelearator on my server (in their forum admin options).

I have try suspend some of users with excessive usage (based on CSF reports) for a half hour.
But still the CPU load is just randomly come and go like twister disaster.
(as I wroted above, there's no 40-50 CPU load with suphp)

I will try to move the eaccelerator cache dir to my /home1 partition (dedicate disk) and wait for any changes on CPU load after that.

[mistwang]

Maybe there are too many running PHP processes cause high memory usage. Set "Max Idle Time" to 10.

http://www.litespeedtech.com/docs/webserver/config/extapps/#extMaxIdleTime

better not using /home1 , it will cause high I/O load.

[DraCoola]

I have set "Max Idle Time" to 4 before.
I'll changed that to 10.
And all right then I'll cancel to move ea cache dir to /home1 partition.

Thanks for advices :)

[mistwang]

It is OK to have it at "4". Check the number of PHP processes running.
You can try disabling ea cache temporarily see if the high CPU usage PHP process goes away.

[DraCoola]

I'll try to disabling ea cache at busy time of my server (it is 5:35 AM here).

top - 05:42:17 up 1 day, 23:31, 1 user, load average: 0.72, 1.33, 1.48

And I will give my update here after that.

So many thanks, George :)

[mistwang]

Have you tried eAccelerator again later. Is the eAccelerator code optimizer enabled? I wonder if the optimizer causes the trouble.
Also, have you tried to strace the PHP process using too much CPU?

Seems eAccelerator is the only solution work well with PHP suEXEC, need to find out why it causes trouble, maybe report it to the developer to have it fixed.