Benchmark

[Test]

Apache vs Litespeed, negative CGI performance

Debian GNU/Linux, kernel 2.4.27-1

Litespeed (without throttling, just after installation):
-----------------
ab -c 100 -n 1000 "http://localhost/cgi-bin/helloworld"
...
Requests per second: 81.99 [#/sec] (mean)
Time per request: 1219.703 [ms] (mean)
Time per request: 12.197 [ms] (mean, across all concurrent requests)
---------------



Apache 1.3.33 (just after apt-get install apache; cp helloworld /usr/lib/cgi-bin/):
-----------------
ab -c 100 -n 1000 "http://localhost/cgi-bin/helloworld"
...
Requests per second: 90.95 [#/sec] (mean)
Time per request: 1099.461 [ms] (mean)
Time per request: 10.995 [ms] (mean, across all concurrent requests)
---------------

apache is ~10% faster %)




.. I'll test FastCGI application performance tomorrow.

[mistwang]

Thank you for posting your benchmark results. :-)
There are a couple issue need to be addressed.
First, It is not good idea to benchmark the helloworld CGI, it is written in plain shell script, very very slow, you should benchmark a CGI written in C/C++.

The out of box LiteSpeed configuration only allows 20 concurrent CGI processes, whereas, Apache starts 100 concurrent CGI processes in your test case. It will give Apache unfair advantage as the helloworld CGI is so slow.

You can change "Max CGI Instances" under "Server"->"Security" tab. You should see better result if you set it to 100.

And you will get more accurate result if you do across network test.

[Test]

Originally Posted by mistwang

Thank you for posting your benchmark results. :-)
There are a couple issue need to be addressed.
First, It is not good idea to benchmark the helloworld CGI, it is written in plain shell script, very very slow, you should benchmark a CGI written in C/C++.

why not ?

You can change "Max CGI Instances" under "Server"->"Security" tab. You should see better result if you set it to 100.

Very interesting results:

ab -c 100 -n 1000 "http://localhost/cgi-bin/helloworld"

"Max CGI Instances"=1
Requests per second: 85.72 [#/sec] (mean)
Time per request: 1166.561 [ms] (mean)
Time per request: 11.666 [ms] (mean, across all concurrent requests)

"Max CGI Instances"=5
Requests per second: 86.70 [#/sec] (mean)
Time per request: 1153.340 [ms] (mean)
Time per request: 11.533 [ms] (mean, across all concurrent requests)

"Max CGI Instances"=10
Requests per second: 82.95 [#/sec] (mean)
Time per request: 1205.486 [ms] (mean)
Time per request: 12.055 [ms] (mean, across all concurrent requests)

"Max CGI Instances"=20
Requests per second: 82.28 [#/sec] (mean)
Time per request: 1215.301 [ms] (mean)
Time per request: 12.153 [ms] (mean, across all concurrent requests)

"Max CGI Instances"=50
Requests per second: 79.52 [#/sec] (mean)
Time per request: 1257.548 [ms] (mean)
Time per request: 12.575 [ms] (mean, across all concurrent requests)

"Max CGI Instances"=100
Requests per second: 75.18 [#/sec] (mean)
Time per request: 1330.107 [ms] (mean)
Time per request: 13.301 [ms] (mean, across all concurrent requests)

"Max CGI Instances"=150
Requests per second: 75.19 [#/sec] (mean)
Time per request: 1330.010 [ms] (mean)
Time per request: 13.300 [ms] (mean, across all concurrent requests)


any ideas ?

[Test]

Binary test.cgi application:
----------------
#include <unistd.h>
int main() { printf( "Content-Type: text/html\n\nHello World!\n" ); }
----------------


Lightspeed (Max CGI Instances 20):
ab -c 100 -n 5000 "http://localhost/cgi-bin/test.cgi"
Requests per second: 250.22 [#/sec] (mean)
Time per request: 399.654 [ms] (mean)
Time per request: 3.997 [ms] (mean, across all concurrent requests)


Apache 1.3.33
ab -c 100 -n 5000 "http://localhost/cgi-bin/test.cgi"
Requests per second: 329.23 [#/sec] (mean)
Time per request: 303.740 [ms] (mean)
Time per request: 3.037 [ms] (mean, across all concurrent requests)

something wrong with Lightspeed :/ ...

[Test]

Binary test.cgi application.
Test from another server (not from localhost):

Lightspeed (Max CGI Instances 20):
ab -c 100 -n 5000 "http://xxx/cgi-bin/test.cgi"
Requests per second: 301.42 [#/sec] (mean)
Time per request: 331.760 [ms] (mean)
Time per request: 3.318 [ms] (mean, across all concurrent requests)

Lightspeed (Max CGI Instances 100):
ab -c 100 -n 5000 "http://xxx/cgi-bin/test.cgi"
Requests per second: 238.76 [#/sec] (mean)
Time per request: 418.834 [ms] (mean)
Time per request: 4.188 [ms] (mean, across all concurrent requests)

Apache 1.3.33
ab -c 100 -n 5000 "http://xxx/cgi-bin/test.cgi"
Requests per second: 353.50 [#/sec] (mean)
Time per request: 282.888 [ms] (mean)
Time per request: 2.829 [ms] (mean, across all concurrent requests)


I am disappointed

[mistwang]

Thank you for your updates. :-)

LiteSpeed use a standalone CGI deamon, for each CGI execution, litespeed need to make a connection to CGI daemon, forword CGI request to CGI daemon, then CGI daemon fork() then execute the CGI. This kind of design is intended to reduce the cost of fork() by adding an extra IPC.

Apache fork and execute the CGI within the server process. In real production, Apache process will have embedded mod_php and mod_perl, which will substantially increase the cost of fork().

And our benchmark shows that CGI performance of Standard edition drops when concurrent level is greater than 100 as well, so if you want the best CGI performance, please try Pro Edition.

CGI is slow and is not recommended for a high traffic web site, should use FCGI.

Looking forward to your FCGI benchmark results. :-)

[Test]

FastCGI test

Apache 1.3.33:
ab -c 20 -n 10000 "http://66.250.130.43/echo-x.cgi"
Requests per second: 515.68 [#/sec] (mean)
Time per request: 38.784 [ms] (mean)
Time per request: 1.939 [ms] (mean, across all concurrent requests)


Litespeed
ab -c 20 -n 10000 "http://66.250.130.43/echo-x.cgi"
Requests per second: 1121.71 [#/sec] (mean)
Time per request: 17.830 [ms] (mean)
Time per request: 0.891 [ms] (mean, across all concurrent requests)


looks good

[mistwang]

Thanks.