What litespeed can do for that server?

A

ASTRAPI

Well-Known Member
#1
#1
Hello

I own a forum 500 users active per 10 minutes and 550.000 posts 48.000 members and is very slow in my server:

It needs 5-8 seconds to open a topic :(

My server is:

# 2x Intel Xeon Quad 5405
# 12GB DDR2 RAM
# 2x SAS disks in Raid 1 15.000 RPM
# 1.000 Mbit unshared port

I use many php files and one database that is less than 1gb and 2 ajax applications in the forum.

I am using Apache and fcgi.

My results:

top

top - 23:39:24 up 5:42, 1 user, load average: 5.40, 6.24, 8.17
Tasks: 346 total, 4 running, 341 sleeping, 0 stopped, 1 zombie
Cpu0 : 27.9%us, 12.6%sy, 0.1%ni, 58.9%id, 0.5%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu1 : 29.2%us, 12.9%sy, 0.1%ni, 54.1%id, 3.6%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu2 : 31.0%us, 13.3%sy, 0.0%ni, 55.2%id, 0.3%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu3 : 30.9%us, 13.0%sy, 0.0%ni, 55.4%id, 0.5%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu4 : 27.9%us, 12.1%sy, 0.1%ni, 59.4%id, 0.4%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu5 : 25.7%us, 10.8%sy, 0.0%ni, 63.1%id, 0.3%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu6 : 30.3%us, 12.4%sy, 0.1%ni, 56.7%id, 0.3%wa, 0.0%hi, 0.2%si, 0.0%st
Cpu7 : 32.2%us, 8.1%sy, 0.0%ni, 56.9%id, 0.8%wa, 0.2%hi, 1.8%si, 0.0%st
Mem: 12306276k total, 4237576k used, 8068700k free, 175844k buffers
Swap: 15623172k total, 0k used, 15623172k free, 1334500k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28487 mysql 15 0 1093m 419m 3212 S 254.4 3.5 132:22.44 mysqld
30632 amorem 16 0 110m 27m 5884 S 15.1 0.2 0:26.36 php5
29115 amorem 15 0 116m 33m 5892 S 13.9 0.3 0:26.73 php5
30686 amorem 15 0 113m 30m 5876 S 13.9 0.3 0:27.38 php5
28171 amorem 16 0 113m 31m 5916 S 12.6 0.3 0:36.20 php5
28880 amorem 16 0 112m 29m 5476 S 12.6 0.2 0:24.62 php5
30664 amorem 16 0 115m 33m 5884 S 8.8 0.3 0:30.49 php5
29061 amorem 15 0 115m 32m 5468 R 7.6 0.3 0:36.57 php5
28262 amorem 15 0 107m 25m 5912 R 6.3 0.2 0:31.99 php5
30634 amorem 15 0 107m 24m 5904 R 3.8 0.2 0:28.08 php5
8627 root 15 0 12872 1304 820 R 2.5 0.0 0:00.03 top
28247 amorem 15 0 110m 27m 5492 S 2.5 0.2 0:30.79 php5
28872 amorem 16 0 107m 24m 5472 S 2.5 0.2 0:22.88 php5
30680 amorem 16 0 107m 24m 5484 S 2.5 0.2 0:13.30 php5
31798 amorem 16 0 112m 29m 5468 S 2.5 0.2 0:27.40 php5
8101 nobody 15 0 60880 3176 1660 S 1.3 0.0 0:00.11 httpd


netstat -apn | grep :80 |wc -l

7784

mysql
\s

Threads: 44 Questions: 340100 Slow queries: 0 Opens: 568 Flush tables: 1 Open tables: 516 Queries per second avg: 126.337

vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
7 0 0 7863236 176584 1337296 0 0 9 42 244 54 29 12 57 1 0


I was read in the forum that is good to have the database in another hard disk but i think it will not work in my scenario as i use raid.

Also i use mysqltuner and sqlprime scripts and all are ok only one message i got:

Table locks acquired immediately: 68%

Reduce lock wait

It seems more configuration or sql problem.

Can litespeed help for that?

Do i have to select or unselecte something at setup so i do not get my problems in litespeed?

I want to have very fast loading with such a server am i asking a lot? :)

You are my last hope.
 
N

NiteWave

Administrator
#2
#2
it looks the bottle-neck is mysqld
28487 mysql 15 0 1093m 419m 3212 S 254.4 3.5 132:22.44 mysqld
Click to expand...
not IO.

you can try on 1-CPU license LSWS, this is the best way to get the answer.

LSWS will reduce the load, having more efficient PHP.

other suggestions:

1.how about using memcached? this may reduce the load of mysqld a lot.

2.profile a topic page using XHProf, to identify execution time of each php function in that page.
 
A

ASTRAPI

Well-Known Member
#3
#3
If the problem is mysql i can;t think how Litespeed can help ....

I have a lot of resources not used at all like cpu and ram.

Maybe i must tune sql or convert some tables to Innodb ?
 
P

PSS

Well-Known Member
#4
#4
Sounds like you're not using indexes (use that RAM!), or you have FULLTEXT searches going on which lock post table.

What forum software you run and how have you optimized it?
For example, with vBulletin stay with 3.6 branch (4.x is just bad programming), replace search with Sphinx, adjust indexes etc.

Raid is not really needed with your site.

For comparison, http://photography-on-the.net/forum/ runs Litespeed.

2 x Intel Xeon-Nehalem 5520-Quadcore
12GB DDR3 RAM
2 x 64GB Intel X25-E SSD disks (one for database only, one for hotcopies)
2 x 500GB SATA disks
gigabit ports etc.

Current load average: 0.62, 0.89, 0.86 - usually around 1 even with 3000 people in.

Even vB 3.6 is a resource hog and has a lousy database structure for large forums, Litespeed can't help bad unoptimized code. I'm moving to custom forum software soon for that reason and others. There is no commercial out-of-the box forum solution for extra large forums that really works.

Make sure that you have a dedicated server, not just dedicated by hardware. That means: no control panels like Ensim, cPanel or Plesk, no resource hog mail servers like qmail, nothing extra - they do their own stuff to destroy response stability.

Post your my.cnf.
 
Last edited:
A

ASTRAPI

Well-Known Member
#5
#5
Thanks for your reply !

I am using invision forum.

"Sounds like you're not using indexes (use that RAM!)"

Yes i am not using this and i don't know how to enable it :(

"you have FULLTEXT searches going on which lock post table."

No i am not using full text search.

my.cnf:

[mysqld]
skip-innodb
skip-locking
max_allowed_packet = 128M
max_connections = 500
key_buffer = 950M
key_buffer_size = 950M
query_cache_type = 1
query_cache_limit = 256M
query_cache_size = 256M
table_cache = 1024
sort_buffer_size = 512M
read_buffer_size = 8M
read_rnd_buffer_size = 512MB
interactive_timeout = 90
wait_timeout = 15
connect_timeout = 10
thread_cache = 32
thread_cache_size = 8
myisam_sort_buffer_size = 64M
thread_concurrency = 8
join_buffer_size = 128

Thank you
 
P

PSS

Well-Known Member
#6
#6
ASTRAPI said:
"Sounds like you're not using indexes (use that RAM!)"

Yes i am not using this and i don't know how to enable it :(
Click to expand...
How indexes are set up is up to developers of the forum, but you can check if they are active by checking table index setup in phpmyadmin, if there are rows in a table and there are indexes set up to a table, you must have > 0 in "cardinality" column of "indexes".

If you see that cardinality is zero, you can kickstart indexes with REPAIR TABLE tablename (that command is in phpmyadmin GUI).

See how this simplified my.cnf setup for [mysqld] performs:

skip-networking
skip-innodb
myisam_use_mmap = 1
query_cache_type = 1
query_cache_size = 128M
query_cache_limit = 2M
max_connections = 1000
table_cache = 5000
key_buffer_size = 1256M
thread_cache_size = 256
sort_buffer_size = 128K
max_allowed_packet = 64M

sort_buffer_size is very small because that way it uses CPU cache better (a lot faster), query cache has bugs if over 128M. Most defaults work very well in mysql.

Set "myisam_use_mmap = 1" if you use mysql 5.1.
 
Last edited:
A

ASTRAPI

Well-Known Member
#7
#7
With this line inside it doesn't load at all :

myisam_use_mmap = 1

If i remove this line i am getting lod from 5 to 14 and the cpu works as crazy but is not fast at all 3 times slower :(
 
P

PSS

Well-Known Member
#8
#8
ASTRAPI said:
With this line inside it doesn't load at all :

myisam_use_mmap = 1

If i remove this line i am getting lod from 5 to 14 and the cpu works as crazy but is not fast at all 3 times slower :(
Click to expand...
When you reboot your mysql on live server it usually takes a while (5-30 minutes) for the disk caches to fill up, during that time CPU and disk will show relatively high usage.

But overall it sounds very odd. Better contact Inivision support.
 
A

ASTRAPI

Well-Known Member
#11
#11
Amanagment company told me that i have a lot lot php proceses...

But i need to have a huge improvment so i am not sure 100% if i can have it from litespeed...
 
P

PSS

Well-Known Member
#12
#12
ASTRAPI said:
Amanagment company told me that i have a lot lot php proceses...

But i need to have a huge improvment so i am not sure 100% if i can have it from litespeed...
Click to expand...
Each php process using db is tied until mysql returns a result. So, when you get locking issues (like search locks post table), processes cumulate and this creates a lock queue, which creates more stalled php processes...

What you need to do is to check mysql processlist and see what is going on there and what is locked and slow.

Test disk i/o, could be SAS-adapter BIOS acting up. Try without RAID, just one dedicated disk for databases.

Believe me: Litespeed is much better than Apache in every way. Much better php process control. You can always try it out.
 
A

ASTRAPI

Well-Known Member
#13
#13
just one dedicated disk for databases
Click to expand...
How can i do that?

I mean if i disable raid i will have cpanel and apache and mysql in the first disk.

How can i move that to a second disk and do i have to configure anything after that in mysql or my configuration file?

A raid 10 it will help?

I do not have at the moment I/O issues.

I use 2 sas 25k discs.
 
P

PSS

Well-Known Member
#14
#14
I'd suggest that you do these in this order:

1. install litespeed tryout and see if that makes a difference.

then

2. check Phpmyadmin's processes view many times and see which queries give you state "Locked" and stay there over 1s. In a good server you should not see more than 5-10 processes at a time in process view. When you get locking problems you see dozens or hundreds of processes on that list with waiting state, and only few as locked.
If you see that you have locking problem, you best bet is to
a) disable all extra scripts (non-forum) and plugins and see if that helps.
b) contact Invision support and ask them to check it out, especially indexes and that forum scripts are up-to-date.
c) try other hardware. I have seen sql locks cumulating because hardware is faulty (SCSI adapter or disk). I've seen servers go bad.
d) try InnoDb, but you MUST ask Invision support about that first. Note also that InnoDB can not be backed up with mysqlhotcopy and repairing InnoDB can be a pain.

then

3. Stop using RAID, and use empty disk for mysql databases only.

Basically this is done in shell

a. stop mysqld service
b. move active database folder to empty drive (check permisisons and flags!)
c. edit my.cnf, add under [mysqld]
datadir=/newdisk/yourdatabasefolder
d. restart mysqld service

That database drive can be partitioned to EXT3, and you can disable access time updates (noatime) to speed it up.

I use dedicated disk for system, dedicated disk for domains, dedicated disk for databases, dedicated disk for backups.

Don't do anything unless you're sure what you do and what it does.
 
A

ASTRAPI

Well-Known Member
#15
#15
Thanks that helps a lot.

check Phpmyadmin's processes
Click to expand...
How can i check this?

Note also that InnoDB can not be backed up with mysqlhotcopy and repairing InnoDB can be a pain.
Click to expand...
mysqlhotcopy?

I just run from ssh:

/scripts/pkgacct account

and i get a backup in home directory with files and the database inside.

It will not be ok?

move active database folder to empty drive (check permisisons and flags!)
Click to expand...
How can i found where is that folder located?

Thank you
 
P

PSS

Well-Known Member
#16
#16
ASTRAPI said:
Thanks that helps a lot.



How can i check this?



mysqlhotcopy?

I just run from ssh:

/scripts/pkgacct account

and i get a backup in home directory with files and the database inside.

It will not be ok?



How can i found where is that folder located?

Thank you
Click to expand...
Mysqlhotcopy is a Perl:DBI script, http://dev.mysql.com/doc/refman/5.0/en/mysqlhotcopy.html. It basically flushes tables, locks them, copies, releases locks.

I run servers without control panels, only with Webmin if anything, so I have no fresh info how Cpanel affects all this.

With CPanel, you should contact Cpanel or your Hosting company support in order to move database to a dedicated disk. Cpanel has a nasty way to taking hold of all that is in your server, and manually moving things around may not agree with it and you may end up with problems.

About backups: CPanel script "pkgacct" has a reputation to eat lots of resources for a long time.

Way how backups are done is crucial if you want to keep your server running smoothly. In my experience for a 24/7 busy database driven server, backing up had the least effect on service quality when you do

- MyISAM backups: mysqlhotcopy (without indexes) done on shut down forum (and shut down domain even better). I have 6GB database and that way it takes 30 seconds to backup it up from a SSD disk to another. Bzip and tar the backup folder, download tar home with https: or move to another server or remote disk with NFS. Better not do cron database backups, always do them with site shut down and personally monitored. Never keep backup repository in same server (and disk) it was taken from, because when disk goes bad -> backups are gone.

- domain file backups: run a rsync to another disk few times a day with cron. With rsync you can limit disk i/o and exclude logs etc., like

rsync -avrhW --bwlimit=2000 --exclude=*.log --exclude=*.gz --exclude=*.lsz --stats /live/www/ /backup/www

Sorry I can't give you more detailed advice - each server is different. It's all in Google though :)
 
Last edited:
S

Sandro

New Member
#17
#17
hey OP, I'm curious to know if you solved this and how.

It's strange to me that with all the RAM left and CPU you have those problems.

But...this may sound stupid, in your top output if you have one apache process, why? where are the child processes?

Anyway it does seem a MySQL problem even thought I don't get why the mysqlserver is not that "loaded".
 
Last edited:
You must log in or register to reply here.
Top