LSWS optimization request

[LiteSpeeder]

Hello,

I'm running a big vBulletin forum. Web and DB servers are separate.

Web server:
2 x E5405 @ 2.00GHz CPUs
8 GB RAM
2 X 120 GB HDDs RAID0

DB server:
2 x E5405 @ 2.00GHz CPUs
4 GB RAM
2 X 120 GB HDDs RAID0

The problems happen in the peak times.

Members see many 503 error pages and i see these warnings in lsws logs:
Running short of concurrent connections.

External processor is not available.

detect loop redirection: /408.shtml

13.000+ online vbulletin users (cookie timeout 15 min)
500-600 instant php & mysql connections

Before requesting tuning recommendations, i want to ask if it's possible to handle this traffic in one box? (web server)

My LSWS settings:
http://img192.imageshack.us/img192/5838/snap1bu.png

http://img705.imageshack.us/img705/6646/snap2.png

http://img697.imageshack.us/img697/9277/snap3c.png

http://img189.imageshack.us/img189/3168/snap4i.png



lsphp5 settings:
http://img697.imageshack.us/img697/2656/snap5q.png


DB server, my.cnf:
[mysqld]
local-infile=0
skip-name-resolve
safe-show-database
#old_passwords
back_log = 50
skip-innodb
max_connections = 1256
key_buffer_size = 800M
myisam_sort_buffer_size = 10000M
myisam_max_sort_file_size = 10000M
myisam_max_extra_sort_file_size = 10000M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 3M
table_cache = 5000
thread_cache_size = 1024
wait_timeout = 300
connect_timeout = 10
tmp_table_size = 128M
max_heap_table_size = 128M
max_allowed_packet = 128M
net_buffer_length = 16384
max_connect_errors = 10
thread_concurrency = 4
concurrent_insert = 2
table_lock_wait_timeout = 30
read_rnd_buffer_size = 1M
bulk_insert_buffer_size = 8M
query_cache_limit = 8M
query_cache_size = 256M
query_cache_type = 1
query_prealloc_size = 262144
query_alloc_block_size = 65536
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
default-storage-engine = MyISAM
max_write_lock_count = 16

[mysqld_safe]
nice = -10
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 128M
sort_buffer = 128M
read_buffer = 64M
write_buffer = 64M

Real-Time Stats:
http://img260.imageshack.us/img260/6655/snap6o.th.png (http://img260.imageshack.us/i/snap6o.png/)

[LiteSpeeder]

Btw, i'm usign APC. I actually have APC, eaccelerator and xcache installed. I'm trying them one by one...when i swith to xcache, EAProc WaitQ is always filled up so i can't use xcache...

[mistwang]

Please download and do a manual upgrade to 4.0.12 again. the latest build is more aggressive in closing files, it may solve your problem.
Since the web server load is pretty high, I think it is better to have a dedicate DB server.
If DB server is not that busy, you can use it as secondary web server by routing some PHP requests to it.

[LiteSpeeder]

What do you mean by dedicate db server? I already have two servers. 1 web, 1 db. I also have a less powered (dual xeon 2.8, GB RAM) web server i'm using it for some static files. I've just installed litespeed web server standard to it.

Ok, thanks for the re-install suggestion. What do you think about my settings above?

[mistwang]

I mean you should keep you current setup instead of trying to use just one server.
You configuration looks fine. You can try increase "Max connections" if you still have the "Running short of concurrent connections" problem after upgrade to latest 4.0.12 .

[anewday]

I can't help much with the mysql/php config.

Your server should be able to handle the load easily. What is the iowait? Try increasing process soft and hard limit for lsphp5 Also, post the realtime stats for lsapi from the web console.

[mistwang]

Seems your DB server has very high I/O wait, that could the cause of the slow down.
You should check DB server with

mysqladmin processlist

when I/O wait is high, maybe it is caused by a bad SQL, or missing index of a large table.