Performance

#1
Hi all,

recently I've upgraded my dedicated server from a single core CPU to a dual core system. After doing some performance tests I ask myself if LSWS ENT #1 CPU is the bottleneck? And what do you think about failed requests?

If you need more detail about the configs please advise.

Thank you.

System:
o AMD 5600 X2, 2GB RAM
o Debian Etch x64

Software:
o nginx 0.5.33 ; 2 worker instances ; php-fcgi 5.2.5 via spawn-fcgi (5 fcgi children), xcache 1.2.1
o LSWS 3.3.1 ENT #1 CPU ; php 5.2.5 via lsapi, default lsapi/php config, xcache 1.2.1

Code:
PHP/Info 
ab -n 10000 -c [concurrent level] http://[ip]:[port]/phpinfo.php 

<HTML>
<BODY>
   <?php phpinfo(); ?>
</BODY>
</HTML>

Server Software:        nginx/0.5.33                                       
Server Hostname:        <del>
Server Port:            80

Document Path:          /phpinfo.php
Document Length:        34572 bytes

Concurrency Level:      100
Time taken for tests:   7.341 seconds
Complete requests:      10000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      346940000 bytes
HTML transferred:       345720000 bytes
Requests per second:    1362.21 [#/sec] (mean)
Time per request:       73.41 [ms] (mean)
Time per request:       0.73 [ms] (mean, across all concurrent requests)
Transfer rate:          47260.59 [Kbytes/sec] received
Code:
Server Software:        LiteSpeed                                          
Server Hostname:        <del>
Server Port:            81

Document Path:          /phpinfo.php
Document Length:        32245 bytes

Concurrency Level:      100
Time taken for tests:   12.660 seconds
Complete requests:      10000
Failed requests:        3
   (Connect: 0, Length: 3, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      323685462 bytes
HTML transferred:       322353265 bytes
Requests per second:    789.89 [#/sec] (mean)
Time per request:       126.60 [ms] (mean)
Time per request:       1.27 [ms] (mean, across all concurrent requests)
Transfer rate:          25567.57 [Kbytes/sec] received
I have tested Apache 2.2.6 / mod_fcgid also. But the load factor was unacceptable.
 
Last edited:

mistwang

LiteSpeed Staff
#2
The failed requests in LiteSpeed test is due the difference in response body size, the request should be successful. I have been seeing that for long time during our internal tests.

The performance difference between LiteSpeed result and nginx result is mainly because LiteSpeed start lsphp dynamically, while nginx start FastCGI PHP processes statically. Starting new PHP processes is expensive, if it happens during the test, it will affect the test results.

Please post your nginx configuration and litespeed PHP configuration, I need to make sure you have compared apple to apple. Are you using the latest 3.3.1 release?
 
#3
Hi,

nginx.conf

Code:
user nginx;
worker_processes 2;
error_log logs/nginx.log;
pid logs/nginx.pid;

events {
  worker_connections  1024;
}

http {
        include       conf/mime.types;
        default_type  application/octet-stream;

        sendfile        on;

        tcp_nopush      on;
        tcp_nodelay     off;

        keepalive_timeout  65;

        gzip  on;
        gzip_http_version 1.0;
        gzip_comp_level 1;
        gzip_proxied any;
        gzip_types      text/plain text/html text/css application/x-javascript text/xml application/xml
                        application/xml+rss text/javascript;

        server {
                listen       <del>:80;
                server_name  <del>;
                                                   
        access_log  off;

        location / {
                root   /home/soellner/www/public_html/;
                index  index.html index.php;
        }

        #error_page  404              /404.html;


        location ~ \.php$ {
                include conf/fastcgi.conf;
                fastcgi_pass  127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param  SCRIPT_FILENAME  /home/soellner/www/public_html$fastcgi_script_name;
        }

    }
}
fastcgi.conf

Code:
fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;
Extract of FCGI Startup

Code:
#!/bin/sh

SPAWNFCGI="/home/soellner/www/bin/spawn-fcgi"
FCGIPROGRAM="/home/soellner/www/bin/php-fcgi"
## FCGISOCKET="/tmp/soellner-fcgi.sock"
FCGIIP=127.0.0.1
FCGIPORT=9000
PHPRC="/home/soellner/www/conf/"

PHP_FCGI_CHILDREN=4
PHP_FCGI_MAX_REQUESTS=1000
FCGI_WEB_SERVER_ADDRS="127.0.0.1"
ALLOWED_ENV="PATH USER"

USERID=stesoell
GROUPID=soellner
LSWS PHP LSAPI

Code:
Name	 	        soellner_php
Address	        uds://home/soellner/www/tmp/lsphp.sock
Notes	 	        Not Set
Max Connections	20

Environment
PHP_LSAPI_MAX_REQUESTS =500
PHP_LSAPI_CHILDREN         =20

Initial Request Timeout (secs) 180
Retry Timeout (secs) 0
Persistent Connection Not Set
Connection Keepalive Timeout Not Set
Response Buffering No
Auto Start	Yes
Command $VH_ROOT/bin/php-lsws -c $VH_ROOT/conf/php.ini
Back Log 100
Instances 1
Run On Start Up Not Set
Max Idle Time Not Set
Priority 0
Memory Soft Limit (bytes) 250M
Memory Hard Limit (bytes) 300M
Process Soft Limit 200
Process Hard Limit 200
 

mistwang

LiteSpeed Staff
#4
OK, I recommend the following changes to soellner_php configurations:

Code:
Max Connections    4

Environment
PHP_LSAPI_MAX_REQUESTS =1000
PHP_LSAPI_CHILDREN     =4
LSAPI_AVOID_FORK = 1

Run On Start Up  Yes
Make sure the php.ini used for litespeed and nginux has the same value.
You can retry your test. LiteSpeed may still need to start children process on demand for the first run, so, run it a couple times.
 
#5
Ok. New results. LSWS now performs better.

changed LSAPI PHP as suggested ...
Max Connections 4
Environment
PHP_LSAPI_MAX_REQUESTS1000
PHP_LSAPI_CHILDREN=4
LSAPI_AVOID_FORK=1
Run On Start Up Yes

Code:
Server Software:        LiteSpeed                                          
Server Hostname:        <del>
Server Port:            81

Document Path:          /test/phpinfo.php
Document Length:        34309 bytes

Concurrency Level:      100
Time taken for tests:   9.665 seconds
Complete requests:      10000
Failed requests:        18
   (Connect: 0, Length: 18, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      344223049 bytes
HTML transferred:       342815478 bytes
Requests per second:    1034.66 [#/sec] (mean)
Time per request:       96.65 [ms] (mean)
Time per request:       0.97 [ms] (mean, across all concurrent requests)
Transfer rate:          35615.42 [Kbytes/sec] received
(Load >1)

nginx: no change in config ...

Code:
Server Software:        nginx/0.5.33                                       
Server Hostname:        <del>
Server Port:            80

Document Path:          /test/phpinfo.php
Document Length:        36575 bytes

Concurrency Level:      100
Time taken for tests:   7.976 seconds
Complete requests:      10000
Failed requests:        9
   (Connect: 0, Length: 9, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      366969955 bytes
HTML transferred:       365749955 bytes
Requests per second:    1253.76 [#/sec] (mean)
Time per request:       79.76 [ms] (mean)
Time per request:       0.80 [ms] (mean, across all concurrent requests)
Transfer rate:          46009.27 [Kbytes/sec] received
(Load >2)
 

mistwang

LiteSpeed Staff
#6
We will setup this in our lab to compare those two.
Which version of PHP are you using for the test?
What is the configuration for building fastcgi PHP binary for nginx?
You run "ab" from the same server?

If there is anything else you would like us know about you setup, please let us know.
 
#7
php 5.2.5 for nginx with xcache 1.2.1

Code:
#! /bin/sh
#
# Created by configure

CFLAGS='-O3 -fno-exceptions -fomit-frame-pointer -march=k8 -msse3 -pipe' \
CXXFLAGS='-O3 -felide-constructors -fno-exceptions -fno-rtti -fomit-frame-pointer -march=k8 -msse3 -pipe' \
'./configure' \
'--prefix=/usr/local/php' \
'--with-php-config=../conf' \
'--enable-fastcgi' \
'--enable-force-cgi-redirect' \
'--enable-discard-path' \
'--enable-sysvmsg' \
'--enable-sysvsem' \
'--enable-sysvshm' \
'--enable-mbstring' \
'--disable-pdo' \
'--with-mysqli=/usr/local/mysql/bin/mysql_config' \
'--without-sqlite' \
'--with-pear' \
"$@"
php 5.2.5 for lsws lsapi (LSWS 3.3.1 Ent #1CPU) with xcache 1.2.1

Code:
#! /bin/sh
#
# Created by configure

CFLAGS='-O3 -fno-exceptions -fomit-frame-pointer -march=k8 -msse3 -pipe' \
CXXFLAGS='-O3 -felide-constructors -fno-exceptions -fno-rtti -fomit-frame-pointer -march=k8 -msse3 -pipe' \
'./configure' \
'--prefix=/usr/local/php' \
'--with-litespeed' \
'--with-php-config=../conf' \
'--enable-fastcgi' \
'--enable-force-cgi-redirect' \
'--enable-discard-path' \
'--enable-sysvmsg' \
'--enable-sysvsem' \
'--enable-sysvshm' \
'--enable-mbstring' \
'--disable-pdo' \
'--with-mysqli=/usr/local/mysql/bin/mysql_config' \
'--without-sqlite' \
'--with-pear' \
"$@"
php.ini

http://soellner.info/test/php.ini.txt

ApacheBench

ab Version 1.3d <$Revision: 1.73 $> run on the same server.
 
Last edited:

mistwang

LiteSpeed Staff
#8
Thanks for the information.
Can you try nginx with "worker_processes 1;"

It will be a much closer match to the 1-CPU license.
 
#9
Damn. I had to re-bench nginx/1worker cause CPU was in powersafe mode.

nginx, 1 worker

Server Software: nginx/0.5.33
Server Hostname: <del>
Server Port: 80

Document Path: /test/phpinfo.php
Document Length: 36609 bytes

Concurrency Level: 100
Time taken for tests: 6.696 seconds
Complete requests: 10000
Failed requests: 10
(Connect: 0, Length: 10, Exceptions: 0)
Broken pipe errors: 0
Total transferred: 367346681 bytes
HTML transferred: 366126559 bytes
Requests per second: 1493.43 [#/sec] (mean)
Time per request: 66.96 [ms] (mean)
Time per request: 0.67 [ms] (mean, across all concurrent requests)
Transfer rate: 54860.62 [Kbytes/sec] received
Server Software: LiteSpeed
Server Hostname: <del>
Server Port: 81

Document Path: /test/phpinfo.php
Document Length: 34309 bytes

Concurrency Level: 100
Time taken for tests: 9.022 seconds
Complete requests: 10000
Failed requests: 11
(Connect: 0, Length: 11, Exceptions: 0)
Broken pipe errors: 0
Total transferred: 344434728 bytes
HTML transferred: 343021337 bytes
Requests per second: 1108.40 [#/sec] (mean)
Time per request: 90.22 [ms] (mean)
Time per request: 0.90 [ms] (mean, across all concurrent requests)
Transfer rate: 38177.20 [Kbytes/sec] received
 
Last edited:

mistwang

LiteSpeed Staff
#14
OK, here is our test results,

Hardware: dual opteron 250
OS: OpenSuse 10.3 x86_64
kernel: 2.6.22.12-0.1-default

LSWS 3.3.1 Enterprise with 1-CPU license

typical LiteSpeed result

ab -n 10000 -c 100 http://localhost:8088/test/phpinfo.php
This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests
Server Software: LiteSpeed
Server Hostname: localhost
Server Port: 8088

Document Path: /test/phpinfo.php
Document Length: 34112 bytes

Concurrency Level: 100
Time taken for tests: 7.048 seconds
Complete requests: 10000
Failed requests: 10
(Connect: 0, Length: 10, Exceptions: 0)
Broken pipe errors: 0
Total transferred: 342518224 bytes
HTML transferred: 341144588 bytes
Requests per second: 1418.84 [#/sec] (mean)
Time per request: 70.48 [ms] (mean)
Time per request: 0.70 [ms] (mean, across all concurrent requests)
Transfer rate: 48597.93 [Kbytes/sec] received

Connnection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 5.4 0 61
Processing: 54 69 15.4 65 199
Waiting: 17 69 15.4 64 198
Total: 54 70 16.3 65 199

Percentage of the requests served within a certain time (ms)
50% 65
66% 68
75% 71
80% 73
90% 78
95% 114
98% 137
99% 143
100% 199 (last request)
typical nginx result:
ab -n 10000 -c 100 http://localhost:80/test/phpinfo.php
This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests
Server Software: nginx/0.5.33
Server Hostname: localhost
Server Port: 80

Document Path: /test/phpinfo.php
Document Length: 36929 bytes

Concurrency Level: 100
Time taken for tests: 8.404 seconds
Complete requests: 10000
Failed requests: 29
(Connect: 0, Length: 29, Exceptions: 0)
Broken pipe errors: 0
Total transferred: 370621087 bytes
HTML transferred: 369400721 bytes
Requests per second: 1189.91 [#/sec] (mean)
Time per request: 84.04 [ms] (mean)
Time per request: 0.84 [ms] (mean, across all concurrent requests)
Transfer rate: 44100.56 [Kbytes/sec] received

Connnection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 9.7 0 133
Processing: 24 82 15.6 81 413
Waiting: 14 82 15.6 81 413
Total: 24 83 18.5 81 413

Percentage of the requests served within a certain time (ms)
50% 81
66% 83
75% 84
80% 85
90% 89
95% 101
98% 163
99% 190
100% 413 (last request)
I have done multiple run for each server, those are the typical results. Sometime, the result is worse when, somehow, CPU idle are high during the test, it happens to both nginux and LSWS. I think it is a kernel scheduler issue.

I found that even with "LSAPI_AVOID_FORK" set, lsphp still kills idle children process, so, I added another environment variable "LSAPI_MAX_IDLE_CHILDREN=4" to lsphp external app configuration.

Another issue I encountered is the confliction of xcache between lsphp and fastcgi php if I keep both running at the same time, both try to lock /tmp/xcache, got some errors, so I have to make sure only have one PHP backend running. Please check if there is any error message in lsws/logs/stderr.log, lsws/logs/error.log.

Maybe you can try the test without xcache in the mix. it would not affect the result much for such a simple script.

LSAPI PHP has a few advantage over nginx + fcgi php:
  • optimized IPC protocol, LSAPI has the minimum overhead in handling the IPC procotol.
  • Persistent connection between web server and backend app server. nginx + fcgi php does not use persistent connections.
  • Unix domain socket vs TCP socket, unix domain socket is faster.
Due to above advantages, is not likely LSAPI lose to a fcgi php backend. However, I do notice that not all CPU cycles has been fully used during a test which result in bad benchmark results. Maybe that happen more frequently on your test environment.

The test do expose one weakness of LSWS when I switch LSWS to use fcgi PHP backend, the result is not as good, and consistently, only half the CPU cycles has been used during the test. I guess it is about time to fine tune our FCGI support as we have been focused on LSAPI for a while. :)
 
#15
Thanks for your effort, George.

I did a re-bench without xcache and using an unix socket for external fcgi.
I added also max_idle_children to lsapi environment.

Code:
Server Software:        LiteSpeed                                          
Server Hostname:        <del>
Server Port:            81

Document Path:          /test/phpinfo.php
Document Length:        31540 bytes

Concurrency Level:      100
Time taken for tests:   9.724 seconds
Complete requests:      10000
Failed requests:        26
   (Connect: 0, Length: 26, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      316266494 bytes
HTML transferred:       314888418 bytes
Requests per second:    1028.38 [#/sec] (mean)
Time per request:       97.24 [ms] (mean)
Time per request:       0.97 [ms] (mean, across all concurrent requests)
Transfer rate:          32524.32 [Kbytes/sec] received
Code:
Server Software:        nginx/0.5.33                                       
Server Hostname:        <del>
Server Port:            80

Document Path:          /test/phpinfo.php
Document Length:        33877 bytes

Concurrency Level:      100
Time taken for tests:   7.104 seconds
Complete requests:      10000
Failed requests:        11
   (Connect: 0, Length: 11, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      339989945 bytes
HTML transferred:       338769945 bytes
Requests per second:    1407.66 [#/sec] (mean)
Time per request:       71.04 [ms] (mean)
Time per request:       0.71 [ms] (mean, across all concurrent requests)
Transfer rate:          47858.95 [Kbytes/sec] received
There are some entries in the error.log I don't know :
http://soellner.info/test/lsws-error.txt
 
Last edited:

mistwang

LiteSpeed Staff
#16
Using unix domain socket with nginx is not reliable. better not using it.

I think it is related to the kernel, somehow, your Linux kernel scheduler assign most process to one CPU when persistent connection is used. Please check the CPU idle time during the test.
You can try increase "max connection" and "PHP_LSAPI_CHILDREN" to 10, see if it helps.

You can apply a trial license, it is equivalent to a 2-CPU license.


Which version of kernel are you using? Linux distribution?

We will test it a little bit more.
 

ts77

Well-Known Member
#18
The test do expose one weakness of LSWS when I switch LSWS to use fcgi PHP backend, the result is not as good, and consistently, only half the CPU cycles has been used during the test. I guess it is about time to fine tune our FCGI support as we have been focused on LSAPI for a while. :)
Yes please :).
Only using fcgi here for being able to use the distributions packages instead of building my own.
 

mistwang

LiteSpeed Staff
#19
Yes please :).
Only using fcgi here for being able to use the distributions packages instead of building my own.
I think it should be a better idea to convert to LSAPI PHP, even with your load balancing setup, LSAPI PHP has everything you need.
 

ts77

Well-Known Member
#20
I agree, it would be a better idea but with too many servers the manual processes take up a lot of time - and yes, I know that I could do this by now through the webinterface.
 
Top