Benchmark

[cyberzen]

Could you include more web servers in your benchmark, like thttpd and boa?

[mistwang]

Sure!
A most complete benchmark result will be available soon. :-)

[cyberzen]

I suggest other people who want other webservers to be included in the benchmark to please post in this thread.

[zellster]

Another web server to benchmark: lighttpd

http://jan.kneschke.de/projects/lighttpd/

[mistwang]

Sure, if we have additional time to play with it. Our focus is on Apache though. :-)
We tried it before, and the result shows that its keep-alive performance is not as good as boa.

[ts77]

any news about more benchmarks? :)

[mistwang]

New benchmarks will be released with 2.0 release.

[andy]

Something's not right in the new benchmark.

According to the product page litespeed standard cannot handle more than 300 concurrent connections, yet all the benchmarks show it handling up to 1000 concurrency. What gives?

[mistwang]

Not only LSWS, by default, Apache can only have 255 concurrent connections, but they still be able to survive on the 1000 concurrent level test.


That is true that at give time, Standand Edition can have 300 concurrent connections at most. However, it can more clients when it does not keep connections alive. LSWS will changes to non-keepalive mode when total concurrent connections reach certain level. That's why it can successfully pass the test like "ab -n xxxxx -c 1000 -k ..." with most requests non-keepalive.

ApacheBench does think the concurrent level is at 1000. :-)

[hdghdfh]

Regarding the lighttpd benchmark results: the performance would have been drastically better if the event handler would have been set up correctly (linux-sysepoll). But you probably know that already, and that's why it is commented out in the config...?

[mistwang]

It is just the opposite, we tried sysepoll, the score is lower than using poll() for the concurrent levels we test, not only for lighttpd, same thing with litespeed and other web server we tested before like the 'userver'.
You probably see benefit of sys_epoll for concurrent level at 10,000.

Just do a simple over the network test, not over the loopback, and see the result yourself.

[Nope]

It is true that epoll is slightly slower than poll when used in the same manner. However, if the design of the work loop is changed to actually use the capability of epoll to store an address directly identifying the dataset/class-member, instead going the long way via the port number, the increase should be significant. Then epoll can easily gain several % speed over poll. At least that's my experience from testing several worker core solutions for hssTVS.

If I ever manage to get hssTVS feature complete you can add it to your benchmark too. :wink: It should be around as fast as your professional version for static content (Linux only though) currently, but it is a long way from even offering the most basic features of your servers and it still lacks any support for dynamic content.

[mistwang]

That's always a good idea to use each API in the most efficient way.
But there are weaknesses in epoll that make it not as efficient as poll in such intensive performance tests. As epoll API can only handle one file handle at a time, there are too many kernel/user land context switches. For example, three context switches are required for each non-keepalive connection just for event handling, while poll() only need one. poll() can combine multiple events in one function call, epoll() cannot. In this kind of performance test, usually multiple events will be combined and reported by one poll(), epoll() has to fetch them one by one.

The design of kqueue() in FreeBSD is better.

We can include your hssTVS in next benchmark update if it is feature complete and ready for production use by that time. :-)

[Nope]

My view of epoll is quite different. In my opinion it is superior. While you have to go through your complete fd set when poll returns to find the active ones, you also have to get the right work data set additinally. With epoll you just register the interesting descriptors once and it returns only the ones that have been active since the last call. This can very well be multiple data sets, only limited by the size of the provided buffer. So, if from 200 active connections were 20 active I get 20 data sets at once, ready with the pointer to the class containing the information and the methods to handle it. The problems coming with the needed add/modify/delete are largely a problem of the design philosophy that's commen amoung the event driven servers. I found ways to work around that and that's part of the reason my core is so fast.

In case I finish hssTVS far enough to be worthy I'll contact you. But don't hold your breath, I don't have the time I'd need to get there any time soon. Feel free to check the current version. Most parts needed for the dynamic content are already in there, so the speed of a more final version shouldn't be slower than built 178.

[zellster]

Recent bechmarks from TVS:
http://tvs.minxam.com/benchmarks.html

LSWS-specific comments:

"Update to 2.0... The 2.0 Version uses poll and is overall 11% faster for
non-keep-alive requests and around 22% faster when keep alive is enabled.
I am somewhat baffled though about the very low number shown at 250 concurrent
keep-alive connections. The status of ZB shows that the number of kept-alive
requests is by far too low (5056 from 10000), especially as the max concurrent
value was set to 300(default setting). Here it was even slower than 1.5. Well,
while it is considerable faster than the old version 1.5 I tested before, it
still fails to overcome Boa or even my dptTVS. Actually, comparing my numbers
with the results they show on their page, my hssTVS is able to beat the
professional LiteSpeed and comes even near Tux 2.2, so I have every right to
go party tonight. The 2.0 Version is also considerable faster when it comes
to bigger files. So, in case you stll use 1.5 you really should upgrade to
this newer version."

Is there any plan to add zero-copy support (sendfile/sendfilev) to the Standard edition? If not, what is the anticipated perf/CPU boost of the feature in the Pro edition?

Thanks!

[mistwang]

No plan yet. :-)
sendfile() will greatly reduce CPU utilization when serving large files, for smaller files, the impact is minimum.

For serving large set of static files, enterprise should be used even for a single CPU server, as disk I/O wait will slow down any single process event driven server, unless aio is used for disk i/o.

[Nope]

For zero-copy sendfile to work you'll also need hardware that supports this feature. I've only found some rather expensive 1GBit cards with such a support till now. But it's still more effective than pumping the data with read/write and the usage is straight forward and easy to implement.

At least the 2.6 Linux core supports non-blocking file access. But portability doesn't allow to optimize code to use that feature I guess. On the other hand it still seems to lack signal support for normal filedescriptors.

[grfgguvf]

Regarding the lighttpd benchmark results: the performance would have been drastically better if the event handler would have been set up correctly (linux-sysepoll). But you probably know that already, and that's why it is commented out in the config...?

I suppose this is indeed the first impression of most people looking at the benchmarks (mine too). So maybe the servers should be measured with epoll and plain poll too, just to show there is no malice.

[grfgguvf]

For serving large set of static files, enterprise should be used even for a single CPU server, as disk I/O wait will slow down any single process event driven server, unless aio is used for disk i/o.

aio_sendfile() will be the killer.

And maybe a little late but could you add Cherokee httpd to the benchmark? Thanks.

[BuhBompus]

Yes, could you please include hssTVS in a new benchmark as well. I would be interested to see how it compares in serving up static content (say 10kb static) vs LSWS Pro/Enterprise.

Thanks for the great work guys!

[xing]

BuhBomps,

What is hssTVS? Do you have a link to it?

[ts77]

BuhBomps,

What is hssTVS? Do you have a link to it?

thats a simple one
http://www.google.de/search?hl=de&q=hsstvs&btnG=Google-Suche&meta=
;)


thomas

[xing]

Ah. Thanks. For some reason, my previous googling skill did not result in anything. My ability to mistype must be too good. =)

I see that hssTVS is:

1) Static content only
2) Fast (based on their benchmarks)
3) Uses epoll

We will definitely use them in our next round of benchmarks.

They did include LiteSpeed 2.0 Standard Edition in their benchmark which did very well, better than comparable (apples to apples) product such as Lighttpd despite the fact they had only "poll" enabled for LiteSpeed and epoll enabled for their own product and Lighttpd. LiteSpeed has supported epoll for ages. Strange they didn't turn it on. No conspiracy theory but it must've been a honest mistake at LiteSpeed's expense nonetheless.

[Nope]

Being the programmer of hssTVS ....

Well, I didn't enable epoll for Litespeed because you definitely state in your benchmarks that poll would give better results ... At least I did disable anything else that could degrade performance, like logfiles or htaccess.

I'd be happy to change any config you desire for the next round of benchmarks I do. I'll be also very happy to assist in configuring my hssTVS for your next benchmarks.

What I really missed in your benchmarks was testing the conditional get and HEAD request speed as those methods are usually heavily used too in real life.

That being said, the TVS server supporting dynamic content will be started after hssTVS reaches beta (with my definition of beta being finished and waiting for any bugs to show up before declaring it final).

[xing]

On second thought. Having epoll enabled for LSWS Std on their benchmark probably would not have made much difference in the scalability tests as the LiteSpeed Standard Edition has a 300 concurrent connections cap. epoll only shows it's true use in very high concurrent rates.

[xing]

Nope, for the record, the the post above was posted before I saw your reply. =)

I do have to say I should've read the complete thread more fully as mistwang did have mention about the pros and cons of poll vs epoll and how it applied in concurrency rates. So you are right in that he was hinting that poll could be faster than epoll depending on concurrency rate.

Though I would still liked it having epoll enabled simply because it would put everyone on the same event dispatcher, even if it's at LiteSpeed's disadvantage, as it would be to no-one's advantage at using the same event api, when the product supports it.

Now I know who to contact for hssTVS setup, you will definitely hear from us when it comes to the next round of benchmarking.

As for conditional GETs and HEAD, I would say conditional GETs are much more important than HEAD and it would be a good benchmark metric.

[Nope]

Nope, for the record, the the post above was posted before I saw your reply. =)

I do have to say I should've read the complete thread more fully as mistwang did have mention about the pros and cons of poll vs epoll and how it applied in concurrency rates. So you are right in that he was hinting that poll could be faster than epoll depending on concurrency rate.

Though I would still liked it having epoll enabled simply because it would put everyone on the same event dispatcher, even if it's at LiteSpeed's disadvantage, as it would be to no-one's advantage at using the same event api, when the product supports it.
Well, I admit not reading the complete manuals/guides for all the servers I tested. I did usually look into comments on the pages/forums though. I did also disable all that came to mind which could hinder performance as especially hssTVS didn't support them and I wouldn't give it an unfair advantage. Honestely, I did the benchmarks mostly for my own benefit, wanting to know where my code was compared to others.

Now I know who to contact for hssTVS setup, you will definitely hear from us when it comes to the next round of benchmarking.
Next time I'll ask you too. ;)

As for conditional GETs and HEAD, I would say conditional GETs are much more important than HEAD and it would be a good benchmark metric.
Well, a lot search engine crawlers still use HEAD as do a couple of http/1.0 Proxies out there. But you are right, use of HEAD is definetely decreasing. What I like about HEAD additionally is the fact that it shows the raw speed of the internal parsing without the network/disk/ram speed being much of an issue. Of course a 304 is almost as lean, just adding the date parser to the results.

[xing]

Nope, you should add your webserver to the following wiki:

http://en.wikipedia.org/wiki/Comparison_of_web_servers

[Nope]

I thought about that a while ago, but opted to wait until I get the features finished and at least into the final beta stage. As it's now I would have problems to provide a support to speak of to more than a handfull users and hunting for bugs.
My work as proxy admin/operator/developer is currently stressfull as we are currently in a state of transition between systems. That's on a bigger farm collection providing access to over 50000 users. So, such an exposure has to wait.

Are you sure YOUR forum is the right place to speak about MY server? -lol-
Besides it's getting way off topic now.

[xing]

Competition makes everyone better and I also don't mind mentioning Lighttpd, Zeus, Apache, BillGates 6.0, kryptonite 8.5, or any other server in the LiteSpeed Benchmark thread. =)

We stand by our product and will take on all challengers.

[Nope]

Well, ok then.

Have you had the chance to take a closer look at my baby then? A couple of features might be overkill for a mere static content server I guess, but it's to be the core of the "real" thing later on.

[BuhBompus]

OK.. Just for you stat freaks out there, I "tried" to run a benchmark on LiteSpeed/2.1.15 standard vs hssTVS for static content. My results are below..

Notice to LiteSpeed Tech: Your server would not let me run the benchmark as it would only get to "Completed 10000 requests" in ApacheBench and stop. From what I can tell, the log files didn't give me any info, but I think even though I had all the security/throttle/banning options turned off, it was still banning the IP address from any further connections and wouldn't allow anything else from the IP until I restarted the server. Accessing the page using an alternate IP was successful.

Here are the results:

----
ApacheBench 2.0.40-dev
ab -k -n 100000 -c 100 http://127.0.0.1:65000/<numbytes>.html
Notice: Due to the lack of a private network,
these results are LOW due to the fact AB was running on the same machine as the web servers, 127.0.0.1 <-> 127.0.0.1 , therefore CPU usage was VERY effected. These results should be higher if ran across a network using 2 seperate PC's.
----

^^^^^
LiteSpeed/2.1.15 Standard
- 127.0.0.1 - No Logging - Follow Symbolic - Disable Script - Not Restrained - epoll()
- Max KeepAlive: 500 - Smart KeepAlive - No Security - No .htaccess - No Expires

Results:
100 Concurrent: Decided to stop responding to requests even though security was disabled?
Unable to test...
^^^^^^

^^^^^^
hssTVS 0.218d
- 127.0.0.1 - No Logging - 100 concurrent keepalive

100 byte static file: Requests per second: 9889.81
1000 byte static file: Requests per second: 9834.88
4000 byte static file: Requests per second: 8854.65
- Transfer rate: 36434.78 [Kbytes/sec]
16000 byte static file: Requests per second: 5404.62
- Transfer rate: 85601.56 [Kbytes/sec]
^^^^^^

I would be more than happy to run more benchmarks once I get LiteSpeed to work without banning the IP from what it seems like a LOT of connections at a time (even thought it is set to off). I would also like to run the test between 2 seperate PC's on a LAN so I can get real results.

Aaron

[Nope]

Well, you should always include more information about the used system. So Processor/Ram (amount+speed)/hdd (type/size/speed) are the minimum. The OS including distro and exact kernel version, which runlevel and perhaps other demons/programs which may influence the results. If you use a real network it's often of interest which cards/chipsets/switches you've used.

Beside of that, ab in version 2.0.x is not very good. Better would be the older 1.3.x or, for some tests, httperf (also free and together with autobench really convenient). The local loopback is also not so good, but as long as all contestants run on the same machine it can give at least give a clue about the effectiveness. Sadly it ignores network specific things which can have a huge influence on the real world speed of a server program. In case you want to test huge files over a network you might have to start including the CPU usage as then the connection will be most likely the bottleneck.

If ab stops dead after a few thousand requests, check the number of not completely closed sockets. Might be that LiteSpeed uses something like a linger setting which might be bad in such a local loop test scenario. Then you'd have to run multiple tests with only 10000 (or whatever still works) and compare the averages for the contestants.

[zellster]

I would be interested in any benchmarks that compare LiteSpeed and Accoria's Rock web server: http://www.accoria.com/cgi-bin/load?name=webserver.html&desc=Rock%20Web%20Server

Does anyone have any experience or feedback on this product?

[mistwang]

Interesting product. :-)
Benchmarked it a little bit, LiteSpeed Enterprise is about 15%-30% faster than Rock on a simple small static file test, for both non-keepalive and keepalive.