Server load

[alex]

We seem to be hitting a wall on www.disboards.com. We are getting busier and at about 18-1900 users online load starts to skyrocket dramatically untill about 70 where I have it set to give a server too busy message.
We have a new database server and the load there is very low. I tested this further by turning off search during a period of high load. Search is very intensive and this would normally show up on the database server but it made no difference.

[mistwang]

It is hard to say what is the root cause of the problem, my guess is the table level lock in MySQL. One request locka a table for too long and blocks other requests, so requests are queued up at the front end server, which causes the high load. You can try some vbulletin hacks that minimize table lock problem if you haven't, I might be wrong on this though.
Other things you can try is to upgrade PHP to 4.4.1 along with the ionCube loader/accelerator.

[alex]

I have seen our boards when there is table locking and this doesn't seem to be the same. Usually the DB server load climbs some and it's staying around 1.
I will give the php upgrade a shot if you think that will help.
Is there anyway to tell when a server is just maxed out? Any signs in litespeed that we are just hitting the load limits?

[mistwang]

Is there anyway to tell when a server is just maxed out? Any signs in litespeed that we are just hitting the load limits?

You can check "I/O wait" and "CPU idle", if CPU idle is very low or I/O wait is pretty high, then the server was maxed out.

The load is mainly contributed by PHP scripting, not inside LiteSpeed, it will help alot if you can make PHP run faster. :-)

If possible you can try other PHP opcode cache like APC along with our LiteSpeed PHP sapi, http://www.litespeedtech.com/lsapi/.

[xing]

Run "vmstat" and check for swap in/out columns to make sure that your OS is not hitting the swapping partition like mad which would kill your performance. This can happen if you overallocate too much memory than available to mysql.

If vmstat command is not available, install the "yum install sysstat" package.

Make sure that mysql's my.conf threadcache value is not too high and not to low and just enough to that the "new threads created" vaule within mysql "show variables" collection is not increasing much if at all. Proper mysql threadcache level can make a huge different by minimizing constant thread creation, memory allocation, which can lead to swapping on overburdened servers.

[alex]

Well do I have a story for you...
We upgraded the PHP to 4.1.1 and installed the lsapi
Completely blew up the boards
Could NOT get rid of the of 503 errors to save my life
So about 4 am I finally say the heck with it and reinstall litespeed. I point it to the original config files for each board.
It goes back in with LSAPI as the default so I get zend optimizer working and call it quits figuring I could worry about a caching program later.
Around noon time today we had 2100 simoultaneous visitors! That is 300 more than I have ever seen and the boards were being responsive. Load was a bit high averaging 40 or so but since the boards were working who cares about numbers.
Here's a snapshot with over 2000 users on
Refresh: 2 5 10 15 30 60 90 120 150 secs Real-Time Server Status Report

Snapshot View | |


System Information
uptime 1:56pm up 11:02, 1 user, load average: 41.59, 40.53, 40.27
mpstat 01:56:43 PM CPU %user %nice %system %idle intr/s
01:56:43 PM all 53.71 0.00 5.33 40.96 2369.60


vmstat procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
26 0 1 0 2339076 143236 166460 0 0 2 51 53 264 54 5 41


LiteSpeed/2.1.9 Enterprise
PID Nice Priority %CPU %Mem RSS(KB) VM(KB) Accu. CPU Uptime
5759 0 26 6.3 0.3 12040 14076 00:24:12 06:21:31
5758 0 27 6.2 0.3 12260 14480 00:23:53 06:21:31

Network Statistics
Throughput (KB/Sec) Connections
Plain SSL Max Idle Plain SSL
In Out In Out Used Avail Used Avail
73 327 0 0 600 108 219 381 0 400

Request Rate
Scope Requests in Processing Request/Second
Server 111 78
Example 0 0
disboards 108 78
dreamsboards 0 0
_AdminVHost 0 0

External Apps
Scope Type Name Max CONN Eff Max Pool In Use Idle WaitQ Req/Sec
Server LSAPI phpLsapi 40 40 40 27 13 71 13


Copyright © 2003-2005. Lite Speed Technologies Inc.
All rights reserved.

[mistwang]

Great Story!

Thank you for sharing.