Litesped 3.3.24 trial and high load

Discussion in 'Install/Configuration' started by Bono, Jan 21, 2009.

  1. Bono

    Bono New Member

    First I would like to say that I heard extremely good things about litespeed web server, so I got few questions and i hope somebody can figure it out.

    I run one relatively big site, 1 bil. hits per month and lately i'm experiencing attacks on my site, it is no longer syn flood but by looks of it bot net is forcing reload of my site. So i thought it is best time to try litespeed trial version to see if it can cope with my traffic.

    My setup: Xeon 3060 (db and web on same box) 4GB of ram
    Software: Cpanel 11 Apache 2.2.11 (prefork) with PHP 5.2.8 and Xcache 1.2.2

    Litespeed is like i said latest trial version 3.3.24 with compiled PHP 5.2.8 and it looks like Xcache is loaded also.
    First thing that happened after I have loaded Litespeed was huge load 12 istead of 3 which is also high for this part of the day.
    [​IMG]

    This is picture of load when litespeed was loaded, i dont need to see that site was unresponsive, i could load forum but if i wanted to go to subsection i would get server is busy. In php info it says xcache is loaded but when i loaded forum it says xcache variables are missing so that is broken too.

    Please advince what can I do, I have 13 more days of trial so I would like to switch if it will work better than my current apache setup.

    I'm sorry because of so big post, but i'm still in shock because of load, I have expected 0.50 and i got 12.

    More info:

    Server log viewer: Log Size: 325,507.43 KB

    In logs i see also bunch of this: 503 Service Unavailable
    2009-01-21 01:50:52.417 [NOTICE] The maximum number of file descriptor limit is set to 6000.
    2009-01-21 01:50:52.417 [NOTICE] [config:server:epsr:lsphp5]'Process Limit' probably is too low, adjust the limit to: 310.
    2009-01-21 01:50:52.417 [NOTICE] [config:server:epsr:lsphp4]'Process Limit' probably is too low, adjust the limit to: 310.
    Last edited: Jan 21, 2009
  2. anewday

    anewday Moderator

    Did you "Build Matching PHP Binary" in Cpanel? Maybe you can try disabling xcache temporarily. Can you post the output of top when it's high load?

    Your trial can be extended if you ask mistwang (George). :)
  3. mistwang

    mistwang LiteSpeed Staff

    Can you show me your xcache configuration in php.ini?
    I think the memory limit for lsphp5 may be too low, so it cause 503 errors.

    For single large web site, it is better disabling PHP suEXEC.

    Test LiteSpeed on another port with an Apache port offset, make sure xcache is working then switch to LiteSpeed from Apache.
  4. Bono

    Bono New Member

    Xcache now works, but as you see load is really high, so it is probably ddos or alot of reloading my page, where can i find ddos settings?

    I have lowered PHP, and in lsphp5 i have put higher settings:
    Memory Soft Limit (bytes)
    450M
    Memory Hard Limit (bytes)
    500M
    Process Soft Limit
    400
    Process Hard Limit

    500


    My nice value is also high: Cpu(s): 18.5% us, 10.0% sy, 71.5% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
    Last edited: Jan 21, 2009
  5. Bono

    Bono New Member

    Little update, my site is now under heavy load and it looks like apache works better than lite speed.
    Lite speed did 40-50 transactions per sec and max load was 45, and apache 2.2.11 does 120 transactions per second and load goes up to 110. But with apache site is more responsive than with lite speed.

    114 requests currently being processed, 45 idle workers
    193 requests currently being processed, 0 idle workers

    DDos protection build in lite speed didn't helped much, i have set 1 transaction per second for dynamic and 10 for static, and load was still high and site was more or less unresponsive.
  6. anewday

    anewday Moderator

    I think something must be wrong with your setup. I never saw a case where Apache would give lower load or more responsive than litespeed.

    Have you set these: http://www.litespeedtech.com/how-tos.html#qa_dos

    Maybe you can give root access to mistwang to look into it further.
    Last edited: Jan 21, 2009
  7. Bono

    Bono New Member

    I was wrong, apache worked better first 5 minutes and then load went to 200. So Lite speed performs better but i'm just stuck with heavy attack.

    Atm box is doing 70 req per sec and 40 are waiting in queue.



    Code:
    top - 16:51:55 up 88 days,  8:42,  1 user,  load average: 43.87, 45.77, 39.54
    Tasks: 149 total,  27 running, 122 sleeping,   0 stopped,   0 zombie
    Cpu(s):  5.7% us,  6.9% sy, 87.4% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
    Mem:   4151296k total,  3382500k used,   768796k free,   286668k buffers
    Swap:  2040212k total,      144k used,  2040068k free,  2230652k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                
    15666 mysql     10  -5  523m 368m 3768 S 22.9  9.1  12308:41 mysqld                                                                 
    30917 nobody    20   1  276m 8456 5756 R 18.9  0.2   0:01.70 lsphp5                                                                 
    30767 nobody    26   1  276m 9916 7176 R 10.9  0.2   0:03.04 lsphp5                                                                 
    29855 nobody    26   1  276m  12m 9860 R  9.9  0.3   0:14.34 lsphp5                                                                 
    30516 nobody    26   1  276m  11m 8992 R  9.9  0.3   0:06.55 lsphp5                                                                 
    30713 nobody    26   1  276m 8948 6280 R  9.9  0.2   0:03.80 lsphp5                                                                 
    30739 nobody    26   1  276m  11m 8960 R  9.9  0.3   0:02.94 lsphp5                                                                 
    30916 nobody    25   1  276m 9028 6328 R  9.9  0.2   0:01.13 lsphp5                                                                 
    30932 nobody    25   1  276m 8464 5760 R  9.9  0.2   0:01.16 lsphp5                                                                 
    29629 nobody    26   1  276m  13m  10m R  8.9  0.3   0:17.28 lsphp5                                                                 
    29678 nobody    25   1  276m  12m  10m S  8.9  0.3   0:12.72 lsphp5                                                                 
    30126 nobody    23   1  276m  12m  10m R  8.9  0.3   0:15.42 lsphp5                                                                 
    30472 nobody    26   1  276m  11m 8940 R  8.9  0.3   0:07.97 lsphp5                                                                 
    30493 nobody    26   1  276m  11m 9368 R  8.9  0.3   0:08.16 lsphp5                                                                 
    30495 nobody    26   1  276m  13m  10m R  8.9  0.3   0:06.83 lsphp5                                                                 
    30618 nobody    26   1  276m  11m 9408 R  8.9  0.3   0:06.47 lsphp5                                                                 
    30912 nobody    25   1  276m 8848 6148 R  8.9  0.2   0:01.32 lsphp5                                                                 
    30688 nobody    26   1  275m 9864 7892 S  5.0  0.2   0:05.45 lsphp5                                                                 
    30765 nobody    26   1  275m  11m 9956 S  5.0  0.3   0:03.76 lsphp5                                                                 
    30885 nobody    18   1  276m 8788 6360 S  2.0  0.2   0:01.06 lsphp5                                                                 
    13174 nobody    15   0 10588 7252 2448 S  1.0  0.2   0:36.48 lshttpd                                                                
    28169 root      16   0  3036 1052  768 R  1.0  0.0   0:05.45 top                                                                    
    29706 nobody    26   1  275m  11m 9964 S  1.0  0.3   0:18.24 lsphp5                                                                 
    30585 nobody    26   1  276m  11m 9004 R  1.0  0.3   0:05.71 lsphp5                                                                 
    30766 nobody    26   1  275m 8156 6212 S  1.0  0.2   0:04.09 lsphp5                                                                 
    30894 nobody    23   1  276m 8952 6280 R  1.0  0.2   0:01.52 lsphp5
  8. anewday

    anewday Moderator

    According to mistwang, "CPU spikes are normal due to the fact that LiteSpeed only use a handful persistent PHP process to handle all PHP requests."

    http://www.litespeedtech.com/support/forum/showthread.php?p=9660&highlight=persistent#post9660
  9. anewday

    anewday Moderator

    Last edited: Jan 21, 2009
  10. anewday

    anewday Moderator

    Also, in real-time stats, "External Application" Look under the WaitQ column, if it's higher than 0 then you'll need to raise your max connections.
  11. Bono

    Bono New Member

    Here it is, just in case if you need it i will paste you my memory usage:
    Code:
    Name	 	lsphp5
    Address	 	uds://tmp/lshttpd/lsphp5.sock
    Max Connections 35
    Environment PHP_LSAPI_MAX_REQUESTS=500
    PHP_LSAPI_CHILDREN=35
    Initial Request Timeout (secs) 60
    Retry Timeout (secs) 	0
    Persistent Connection	No
    Connection Keepalive Timeout	Not Set
    Response Buffering	No
    Auto Start	Yes
    Command		$SERVER_ROOT/fcgi-bin/lsphp5
    Back Log	100
    Instances	1
    Run On Start Up		Not Set
    Max Idle Time		Not Set
    Priority		Not Set
    Memory Soft Limit (bytes)	450M
    Memory Hard Limit (bytes)	500M
    Process Soft Limit		400
    Process Hard Limit		500
  12. anewday

    anewday Moderator

    Try setting persistent connection to Yes and raise max connections and PHP_LSAPI_CHILDREN to 60 or higher. Also, check the last post on Page 1.
  13. Bono

    Bono New Member

    Answer to the last post on the first page is zero, nothing is waiting. And i have applied those two values so now i need look and see if it will help.

    It looks like that didn't helped much:

    Last edited: Jan 21, 2009
  14. anewday

    anewday Moderator

    I don't know what else you can do, wait for mistwang's reply. How many req/s do you see in real-time stats?
    Last edited: Jan 21, 2009
  15. Bono

    Bono New Member

    ATM 188 req/s outgoing 632KB/sec. :eek:

    I'm under big attack so it's not best time to switch, but i don't have any choice apache dies after few minutes.
  16. anewday

    anewday Moderator

    It's not that much. Did you set these under Security tab?

    Outbound Bandwidth (bytes/sec)
    Inbound Bandwidth (bytes/sec)

    May be worth a try.
  17. Bono

    Bono New Member

    Yea I have tried that too, but my main problem is extremely high load and too many mysql threads. I will see over night if load will be lower.
  18. anewday

    anewday Moderator

    Maybe the bottleneck lies elsewhere, not the webserver. Is your mysql on a separate hdd/server?
  19. Bono

    Bono New Member

    No same, but there is no iowait. It is just that somebody is attacking and lite speed can't help me much. Only good thing is that it doesn't lock like apache.

    Code:
     r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
    47  0    144 262292 306640 2649820    0    0    29    16    3     0 16  7 76  1
    56  0    144 260628 306640 2650080    0    0     3   184 1370  1214 87 13  0  0
    60  0    144 256212 306640 2650080    0    0     1   165 1286  1205 84 16  0  0
    61  0    144 253716 306640 2650080    0    0     0   204 1217  1042 90 10  0  0
    70  0    144 244676 306648 2650592    0    0     0   411 1492  1369 84 16  0  0
    73  0    144 241396 306652 2650328    0    0     0   279 1399  1354 89 11  0  0
    78  0    144 231540 306656 2650584    0    0     1   188 1477  1351 88 12  0  0
    87  0    144 213172 306656 2650584    0    0     0   117 1197  1064 84 16  0  0
  20. anewday

    anewday Moderator

    What did you set for connection soft, hard and grace period? How many IPs are attacking and how big? I hope you have a firewall installed to block the offenders.

    My site got hit with a ddos from 20 different IPs 2 weeks ago and the forum was still running fine with load of 15, though it was slower than normal. Server is a Xeon 3060 with 3GB of memory. Mysql on a separate drive.

Share This Page