Litesped 3.3.24 trial and high load

[Bono]

First I would like to say that I heard extremely good things about litespeed web server, so I got few questions and i hope somebody can figure it out.

I run one relatively big site, 1 bil. hits per month and lately i'm experiencing attacks on my site, it is no longer syn flood but by looks of it bot net is forcing reload of my site. So i thought it is best time to try litespeed trial version to see if it can cope with my traffic.

My setup: Xeon 3060 (db and web on same box) 4GB of ram
Software: Cpanel 11 Apache 2.2.11 (prefork) with PHP 5.2.8 and Xcache 1.2.2

Litespeed is like i said latest trial version 3.3.24 with compiled PHP 5.2.8 and it looks like Xcache is loaded also.
First thing that happened after I have loaded Litespeed was huge load 12 istead of 3 which is also high for this part of the day.


This is picture of load when litespeed was loaded, i dont need to see that site was unresponsive, i could load forum but if i wanted to go to subsection i would get server is busy. In php info it says xcache is loaded but when i loaded forum it says xcache variables are missing so that is broken too.

Please advince what can I do, I have 13 more days of trial so I would like to switch if it will work better than my current apache setup.

I'm sorry because of so big post, but i'm still in shock because of load, I have expected 0.50 and i got 12.

More info:

mysql 15666 25.1 8.3 496100 348552 ? S<l 2008 12032:55
root 9161 0.1 0.0 5976 2848 ? S 01:23 0:00 lshttpd
nobody 9163 0.4 0.0 6976 3392 ? S 01:23 0:00 \_ lshttpd
user1 9320 12.7 0.2 20596 10484 ? RN 01:23 0:00 | \_ lsphp5
user1 9322 12.6 0.2 20596 10500 ? RN 01:23 0:00 | \_ lsphp5
user1 9328 28.5 0.2 20324 10396 ? RN 01:23 0:00 | \_ lsphp5
user1 9329 23.5 0.2 20320 10348 ? RN 01:23 0:00 | \_ lsphp5
user1 9330 18.5 0.2 20596 10432 ? RN 01:23 0:00 | \_ lsphp5
user1 9331 22.0 0.2 20596 10468 ? RN 01:23 0:00 | \_ lsphp5
user2 9332 0.0 0.0 12144 2084 ? RN 01:23 0:00 | \_ lsphp5
user1 9334 0.0 0.0 12132 1868 ? RN 01:23 0:00 | \_ lsphp5
user1 9335 8.0 0.2 20340 10256 ? RN 01:23 0:00 | \_ lsphp5
user1 9337 6.0 0.1 17612 7596 ? RN 01:23 0:00 | \_ lsphp5
user1 9344 0.0 0.2 20596 10340 ? RN 01:23 0:00 | \_ lsphp5
nobody 9164 0.0 0.0 6024 2652 ? S 01:23 0:00 \_ lshttpd

Server log viewer: Log Size: 325,507.43 KB

In logs i see also bunch of this: 503 Service Unavailable
2009-01-21 01:50:52.417 [NOTICE] The maximum number of file descriptor limit is set to 6000.
2009-01-21 01:50:52.417 [NOTICE] [config:server:epsr:lsphp5]'Process Limit' probably is too low, adjust the limit to: 310.
2009-01-21 01:50:52.417 [NOTICE] [config:server:epsr:lsphp4]'Process Limit' probably is too low, adjust the limit to: 310.

[anewday]

Did you "Build Matching PHP Binary" in Cpanel? Maybe you can try disabling xcache temporarily. Can you post the output of top when it's high load?

Your trial can be extended if you ask mistwang (George).

[mistwang]

Can you show me your xcache configuration in php.ini?
I think the memory limit for lsphp5 may be too low, so it cause 503 errors.

For single large web site, it is better disabling PHP suEXEC.

Test LiteSpeed on another port with an Apache port offset, make sure xcache is working then switch to LiteSpeed from Apache.

[Bono]

Originally Posted by mistwang

Can you show me your xcache configuration in php.ini?
I think the memory limit for lsphp5 may be too low, so it cause 503 errors.

For single large web site, it is better disabling PHP suEXEC.

Test LiteSpeed on another port with an Apache port offset, make sure xcache is working then switch to LiteSpeed from Apache.

Xcache now works, but as you see load is really high, so it is probably ddos or alot of reloading my page, where can i find ddos settings?

I have lowered PHP, and in lsphp5 i have put higher settings:
Memory Soft Limit (bytes)
450M
Memory Hard Limit (bytes)
500M
Process Soft Limit
400
Process Hard Limit

500

27513 nobody 22 1 276m 9248 6432 R 50.8 0.2 0:12.25 lsphp5
27818 nobody 22 1 276m 8232 5504 R 36.8 0.2 0:00.37 lsphp5
27820 nobody 21 1 276m 8044 5320 R 27.9 0.2 0:00.28 lsphp5
15666 mysql 10 -5 486m 341m 3768 S 25.9 8.4 12134:28 mysqld
27127 nobody 21 1 276m 11m 9400 R 19.9 0.3 0:22.94 lsphp5
27695 nobody 22 1 275m 11m 9m S 13.9 0.3 0:05.64 lsphp5
27693 nobody 21 1 275m 11m 9252 S 8.0 0.3 0:05.48 lsphp5
27125 nobody 17 1 275m 13m 11m S 7.0 0.3 0:18.74 lsphp5
27696 nobody 16 1 275m 11m 9.8m S 4.0 0.3 0:06.77 lsphp5
27684 nobody 16 1 275m 10m 9240 S 2.0 0.3 0:05.05 lsphp5
18032 named 25 0 72652 33m 1956 S 1.0 0.8 15:00.25 named
27122 nobody 15 0 8604 5304 1416 S 1.0 0.1 0:01.21 lshttpd
27610 nobody 17 1 275m 11m 9616 S 1.0 0.3 0:04.99 lsphp5

My nice value is also high: Cpu(s): 18.5% us, 10.0% sy, 71.5% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si

[Bono]

Little update, my site is now under heavy load and it looks like apache works better than lite speed.
Lite speed did 40-50 transactions per sec and max load was 45, and apache 2.2.11 does 120 transactions per second and load goes up to 110. But with apache site is more responsive than with lite speed.

114 requests currently being processed, 45 idle workers
193 requests currently being processed, 0 idle workers

DDos protection build in lite speed didn't helped much, i have set 1 transaction per second for dynamic and 10 for static, and load was still high and site was more or less unresponsive.

[anewday]

I think something must be wrong with your setup. I never saw a case where Apache would give lower load or more responsive than litespeed.

Have you set these: http://www.litespeedtech.com/how-tos.html#qa_dos

Maybe you can give root access to mistwang to look into it further.

[Bono]

Originally Posted by anewday

I think something must be wrong with your setup. I never saw a case where Apache would give lower load than litespeed.

Have you set these: http://www.litespeedtech.com/how-tos.html#qa_dos

Maybe you can give root access to mistwang to look into it further.

I was wrong, apache worked better first 5 minutes and then load went to 200. So Lite speed performs better but i'm just stuck with heavy attack.

Atm box is doing 70 req per sec and 40 are waiting in queue.

top - 16:51:55 up 88 days,  8:42,  1 user,  load average: 43.87, 45.77, 39.54
Tasks: 149 total,  27 running, 122 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.7% us,  6.9% sy, 87.4% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   4151296k total,  3382500k used,   768796k free,   286668k buffers
Swap:  2040212k total,      144k used,  2040068k free,  2230652k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                
15666 mysql     10  -5  523m 368m 3768 S 22.9  9.1  12308:41 mysqld                                                                 
30917 nobody    20   1  276m 8456 5756 R 18.9  0.2   0:01.70 lsphp5                                                                 
30767 nobody    26   1  276m 9916 7176 R 10.9  0.2   0:03.04 lsphp5                                                                 
29855 nobody    26   1  276m  12m 9860 R  9.9  0.3   0:14.34 lsphp5                                                                 
30516 nobody    26   1  276m  11m 8992 R  9.9  0.3   0:06.55 lsphp5                                                                 
30713 nobody    26   1  276m 8948 6280 R  9.9  0.2   0:03.80 lsphp5                                                                 
30739 nobody    26   1  276m  11m 8960 R  9.9  0.3   0:02.94 lsphp5                                                                 
30916 nobody    25   1  276m 9028 6328 R  9.9  0.2   0:01.13 lsphp5                                                                 
30932 nobody    25   1  276m 8464 5760 R  9.9  0.2   0:01.16 lsphp5                                                                 
29629 nobody    26   1  276m  13m  10m R  8.9  0.3   0:17.28 lsphp5                                                                 
29678 nobody    25   1  276m  12m  10m S  8.9  0.3   0:12.72 lsphp5                                                                 
30126 nobody    23   1  276m  12m  10m R  8.9  0.3   0:15.42 lsphp5                                                                 
30472 nobody    26   1  276m  11m 8940 R  8.9  0.3   0:07.97 lsphp5                                                                 
30493 nobody    26   1  276m  11m 9368 R  8.9  0.3   0:08.16 lsphp5                                                                 
30495 nobody    26   1  276m  13m  10m R  8.9  0.3   0:06.83 lsphp5                                                                 
30618 nobody    26   1  276m  11m 9408 R  8.9  0.3   0:06.47 lsphp5                                                                 
30912 nobody    25   1  276m 8848 6148 R  8.9  0.2   0:01.32 lsphp5                                                                 
30688 nobody    26   1  275m 9864 7892 S  5.0  0.2   0:05.45 lsphp5                                                                 
30765 nobody    26   1  275m  11m 9956 S  5.0  0.3   0:03.76 lsphp5                                                                 
30885 nobody    18   1  276m 8788 6360 S  2.0  0.2   0:01.06 lsphp5                                                                 
13174 nobody    15   0 10588 7252 2448 S  1.0  0.2   0:36.48 lshttpd                                                                
28169 root      16   0  3036 1052  768 R  1.0  0.0   0:05.45 top                                                                    
29706 nobody    26   1  275m  11m 9964 S  1.0  0.3   0:18.24 lsphp5                                                                 
30585 nobody    26   1  276m  11m 9004 R  1.0  0.3   0:05.71 lsphp5                                                                 
30766 nobody    26   1  275m 8156 6212 S  1.0  0.2   0:04.09 lsphp5                                                                 
30894 nobody    23   1  276m 8952 6280 R  1.0  0.2   0:01.52 lsphp5

[anewday]

27513 nobody 22 1 276m 9248 6432 R 50.8 0.2 0:12.25 lsphp5
27818 nobody 22 1 276m 8232 5504 R 36.8 0.2 0:00.37 lsphp5
27820 nobody 21 1 276m 8044 5320 R 27.9 0.2 0:00.28 lsphp5
15666 mysql 10 -5 486m 341m 3768 S 25.9 8.4 12134:28 mysqld
27127 nobody 21 1 276m 11m 9400 R 19.9 0.3 0:22.94 lsphp5
27695 nobody 22 1 275m 11m 9m S 13.9 0.3 0:05.64 lsphp5
27693 nobody 21 1 275m 11m 9252 S 8.0 0.3 0:05.48 lsphp5
27125 nobody 17 1 275m 13m 11m S 7.0 0.3 0:18.74 lsphp5
27696 nobody 16 1 275m 11m 9.8m S 4.0 0.3 0:06.77 lsphp5
27684 nobody 16 1 275m 10m 9240 S 2.0 0.3 0:05.05 lsphp5
18032 named 25 0 72652 33m 1956 S 1.0 0.8 15:00.25 named
27122 nobody 15 0 8604 5304 1416 S 1.0 0.1 0:01.21 lshttpd
27610 nobody 17 1 275m 11m 9616 S 1.0 0.3 0:04.99 lsphp5

My nice value is also high: Cpu(s): 18.5% us, 10.0% sy, 71.5% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si

According to mistwang, "CPU spikes are normal due to the fact that LiteSpeed only use a handful persistent PHP process to handle all PHP requests."

http://www.litespeedtech.com/support...stent#post9660

[anewday]

Can you paste all the settings under External App > lsphp5?

You may have to tweak the environmental variables: http://www.litespeedtech.com/php-litespeed-sapi.html

[anewday]

Also, in real-time stats, "External Application" Look under the WaitQ column, if it's higher than 0 then you'll need to raise your max connections.