Disk IO Issues - please help!
I'm experiencing some serious disk IO issues on my web server and I can't seem to pinpoint the problem. During peak and off peak (to a lesser extent) my disk IO is maxing out constantly 98-100% and rarely drops below 80% even during off peak.
I'm running 2-CPU Litespeed on 2 x Harpertown 5410, 4G RAM, 2x750G SATA2, hardware RAID 1, CentOS 4.5/cPanel.
DB server is separate and there are no problems there.
I've been running LS for about 4 years, never have I had disk IO issues and this time last year we had 75% more traffic so the current issues are making no sense at all. Requests per second according to Litespeed real time stats is max. 80 per sec.
I and my host have been looking to see if there is any obscure logging but we have found nothing.
Steps I've taken so far to resolve or at least improve the situation:
- Upgrade from 4.0.12 to 4.2.2
- Disable access logs (domlogs already disabled)
- Enable AIO
- Lower Max connections/Max keep-live requests from 2000 to 1000
- Increase Max MMAP File Size from 256K to 1M
- Increase Total MMAP Cache Size from 40M to 128M
I'm seeing no improvement at all.
The only changes on the server since this time last year are we have upgrade PHP to 5.3.21 and upgraded to vBulletin 4.2. I noticed no IO problems after the upgrades and the problem seems to have worsened over the last week with no traffic increase.
We are a download site, but only serve about 20-30 downloads per minute at the moment which is much lower than in previous years.
iostat output (70% of peak):
You have more writes than reads. that's likely the reason of high i/o. so download (assuming only for disk read) is not the cause. Run tool like 'atop' or 'iotop' to see which process causing the i/o.
I investigated this and kjournald is writing the most alongside pdflush:
Is there anything I can do to troubleshoot further?
Thanks for your help.
Are you suggesting LS causing the I/O? I am a little confused. From your first post, LS (running for quite a while with no issue) is upgraded to help alleviate the problem and it does a bit.
Anyway, you can google "kjournald high IO" for more discussions/answers.
|All times are GMT -7. The time now is 02:15 PM.|