performances issue with x86_64 and cpu cores

Discussion in 'General' started by IrPr, Aug 18, 2008.

  1. IrPr

    IrPr New Member

    Hi there,
    I was on a 32bit system, hosting large major php/mysql cms and many static file size up to 10M, more than 2K concurrent connections with no any lag even at traffic peak, that was awesome

    Now i've migrated to new server with more 4G RAM and better CPU, changes are RAM, CPU, RAID controller PCI and system arch and now im using cPanel
    but its getting IO bottleneck and high iowait even at medium traffic
    I guess odd raid controller, RocketRaid 174x which seems to be fakeraid caused this issue but i doubt in 2 things with LS

    one is system arch that changed from previous case, now i've switched to 64bit and this is my 1st xperience with x86_64
    i know 64bit system increases system abilities well such as big files spport but dont know about performances
    is it possible that 64bit caused this lag problem?

    another one is cpu cores, in previus configuration it was added listeners and VH in LSWS manually, assigned cores to listeners, but now i'm on cPanel and i dont find how to set cores for listeners, its loaded from httpd.conf automatically and i tought its using 1 core

    here is mpstat in at a very low traffic
    Code:
    Linux 2.6.18-8.el5 (hdbox)  08/18/2008      _x86_64_
    
    07:21:47 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:48 AM  all    0.51    0.51    2.53    7.07    0.51    1.01    0.00    0.00   87.88
    07:21:48 AM    0    0.00    1.02    0.00    0.00    0.00    0.00    0.00    0.00   98.98
    07:21:48 AM    1    0.00    0.00    5.00   15.00    1.00    3.00    0.00    0.00   76.00
    
    07:21:48 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:49 AM  all    0.51    3.55    2.03   13.20    0.51    1.02    0.00    0.00   79.19
    07:21:49 AM    0    1.01    1.01    1.01    2.02    0.00    0.00    0.00    0.00   94.95
    07:21:49 AM    1    0.00    5.10    4.08   24.49    0.00    2.04    0.00    0.00   64.29
    
    07:21:49 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:50 AM  all    0.51    4.04    2.53   14.14    0.00    1.01    0.00    0.00   77.78
    07:21:50 AM    0    1.00    0.00    1.00    3.00    0.00    0.00    0.00    0.00   95.00
    07:21:50 AM    1    0.00    8.25    3.09   24.74    1.03    2.06    0.00    0.00   60.82
    
    07:21:50 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:51 AM  all    0.00    0.00    2.04    4.08    0.51    1.02    0.00    0.00   92.35
    07:21:51 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
    07:21:51 AM    1    1.01    0.00    4.04    9.09    0.00    2.02    0.00    0.00   83.84
    
    07:21:51 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:52 AM  all    0.00    0.00    2.02   11.62    0.00    1.01    0.00    0.00   85.35
    07:21:52 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
    07:21:52 AM    1    0.00    0.00    4.04   22.22    1.01    2.02    0.00    0.00   70.71
    
    07:21:52 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:53 AM  all    0.00    0.00    1.53    9.18    0.51    1.02    0.00    0.00   87.76
    07:21:53 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
    07:21:53 AM    1    0.00    0.00    4.04   19.19    1.01    2.02    0.00    0.00   73.74
    
    07:21:53 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:54 AM  all    0.51    1.52    2.02    9.09    0.51    1.01    0.00    0.00   85.35
    07:21:54 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
    07:21:54 AM    1    0.00    4.08    3.06   17.35    0.00    2.04    0.00    0.00   73.47
    
    07:21:54 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:55 AM  all    0.00    0.00    1.54    6.67    0.51    0.51    0.00    0.00   90.77
    07:21:55 AM    0    0.00    0.00    0.00    5.10    0.00    0.00    0.00    0.00   94.90
    07:21:55 AM    1    1.01    0.00    4.04    8.08    1.01    1.01    0.00    0.00   84.85
    
    07:21:55 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:56 AM  all    0.00    0.00    2.03    6.09    0.00    1.02    0.00    0.00   90.86
    07:21:56 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
    07:21:56 AM    1    0.00    0.00    4.00   13.00    1.00    2.00    0.00    0.00   80.00
    
    07:21:56 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:57 AM  all    3.00    6.00    4.50    8.00    0.50    1.00    0.00    0.00   77.00
    07:21:57 AM    0    4.08    2.04    2.04    2.04    0.00    0.00    0.00    0.00   89.80
    07:21:57 AM    1    1.04    9.38    6.25   13.54    0.00    2.08    0.00    0.00   67.71
    
    07:21:57 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:58 AM  all    1.53    6.63    3.06   10.20    0.00    1.02    0.00    0.00   77.55
    07:21:58 AM    0    3.03    6.06    1.01    0.00    0.00    0.00    0.00    0.00   89.90
    07:21:58 AM    1    0.00    8.00    6.00   20.00    1.00    2.00    0.00    0.00   63.00
    
    07:21:58 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:21:59 AM  all    2.53    8.59    4.04    3.54    0.51    1.01    0.00    0.00   79.80
    07:21:59 AM    0    5.00    7.00    2.00    2.00    0.00    0.00    0.00    0.00   84.00
    07:21:59 AM    1    1.00   10.00    6.00    6.00    0.00    2.00    0.00    0.00   75.00
    
    07:21:59 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:22:00 AM  all    7.04    8.54    5.53    8.54    0.00    1.01    0.00    0.00   69.35
    07:22:00 AM    0   12.12    8.08    3.03    1.01    0.00    0.00    0.00    0.00   75.76
    07:22:00 AM    1    1.03    8.25    8.25   16.49    0.00    2.06    0.00    0.00   63.92
    
    07:22:00 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:22:01 AM  all    0.00    0.00    2.55    9.69    0.00    0.51    0.00    0.00   87.24
    07:22:01 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
    07:22:01 AM    1    1.01    0.00    4.04   19.19    1.01    1.01    0.00    0.00   73.74
    
    07:22:01 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
    07:22:02 AM  all    2.51    5.03    2.51   16.58    0.50    1.01    0.00    0.00   71.86
    07:22:02 AM    0    4.00    9.00    2.00    0.00    0.00    0.00    0.00    0.00   85.00
    07:22:02 AM    1    0.00    2.02    3.03   33.33    1.01    2.02    0.00    0.00   58.59
    
    approximately core 0 has no iowait while core 1 has whole system's iowait

    Any commen would be appreciated,
    Thanks
    Last edited: Aug 18, 2008
  2. IrPr

    IrPr New Member

    PS: another major difference is io event dispatcher
    due to raid controller drive issue with 2.6 kernels i'm currently on 2.4 and therefore no epoll IO event dispatcher available
    in previus case it was set to epoll while its forced to be poll in new box
    Last edited: Aug 18, 2008
  3. mistwang

    mistwang LiteSpeed Staff

    You can create the listener manually with IP:port and assign the binding. LSWS will use it together with configuration from httpd.conf. Using multiple CPU to handle I/O intensive task is important.

    epoll may have some impact as well, but should not be as significant.
  4. IrPr

    IrPr New Member

    Thanks for reply,
    any idea about arch ?
  5. mistwang

    mistwang LiteSpeed Staff

    I don't know how good is the 64bit support in 2.4 kernel, hope it is good enough. I think the high I/O wait may related to the RAID card driver. Hope you can get good support for the RAID card in latest 2.6 kernel.

    Not sure what kind RAID is used. RAID0 should be the best, then RAID10, RAID5, etc.
  6. IrPr

    IrPr New Member

    Thanks a lot Mr Wang,
    I know its not LSWS related question and maybe an offtopic thread, but it would be more appreciated if you suggest me more,

    i've planned to upgrade Raid controller to 3WARE which i'v experienced with
    it was workin like a charm pushed upto 350Mbit/s with RAID1 Array over Litespeed Ent on 32bit system with 2.6 kernel using epoll as IO event dispatcher, while now on 64bit 2.4 kernel same files and same RAID1 Array but on RocketRaid fakeraid controller even couldnt push 100Mbit/s
    I host large files upper than 100MB with a lot of concurrent connections up to 2-3k

    What would u recommend for my download server? Which RAID Array, 0 or 1 or 5 ? and 32bit or 64 over 2.6 kernel?

    again, Special thanks
    Last edited: Aug 20, 2008
  7. mistwang

    mistwang LiteSpeed Staff

    3WARE raid card with 2.6 kernel should work better.
    Go with either RAID0 or RAID10 or RAID1, depends on what level of data protection do you need. 32bit or 64bit should not matter much, if you need to serve file larger than 2GB, you have to use 64bit.
  8. IrPr

    IrPr New Member

    Thanks for your advice,
    I did setup 3 SATA II in RAID0 ARRAY using 3WARE 6xxx-7xxx controller on 64bit CentOS 5.1 release with 2.6 kernel

    no better performances yet
    im hosting large files ~100MB and about ~k concurrent connections using epoll as event dispatcher
    my iowait value is still too high and seems just one core is used by lshttpd process:
    thats odd
    im binding to all cpu cores but core 0 is actually idle while core 1 uses more than 70% iowait
    i just host large static files right now on this box with no php/cgi/rubby/mysql process
    maybe something buggy with LSWS when hosting large static contents on SMP systems?

    any suggestion would be appreciated

    Thanks
  9. mistwang

    mistwang LiteSpeed Staff

    How about the CPU usage of lshttpd processes, if two lshttpd are configured properly to serve that site, there should be two lshttpd processes on top of "top" output.
  10. IrPr

    IrPr New Member

    Right, there are 2 lshttpd process in top result


    Last edited: Sep 2, 2008
  11. mistwang

    mistwang LiteSpeed Staff

    From the CPU cycles consumed by the two lshttpd processes, two processes has been well balanced, so the configuration should be all right.

    And it does look like the kernel allocate I/O operations to only one CPU core only, it is up to the kernel I/O scheduler, not much we can do about in user land applications.

    It might be at the peak I/O throughput for the disk array, since those are random read accesses. there are a few things we can improve if have not done yet.

    Make sure NCQ has been enabled for the RAID card and hard disks. try a different I/O scheduler in the kernel.

    We can try a 4-CPU or 8-CPU license on this server, see if it get any better when more I/O worker processes are used. It may help a little with NCQ enabled I think.
  12. IrPr

    IrPr New Member

    WOW! AWESOME!

    switched IO scheduler to deadline and some other tips with kernel, now server performances increased dramatically

    this mrtg graph indicates that:
    [​IMG]
    Look output is about 3x now :D

    Thanks George, you are the man!
    Last edited: Sep 2, 2008
  13. mistwang

    mistwang LiteSpeed Staff

    Glad it work out beautifully. :)

    What is the most important factor on this, I/O scheduler? or other kernel tuning tips?
  14. IrPr

    IrPr New Member


    • Switching to deadline I/O Scheduler has the most efficient
    • set READAHEAD to much higher value, changed 256 to 16M
    • increased VM dirty ratio 20 values
    Last edited: Sep 3, 2008
  15. mistwang

    mistwang LiteSpeed Staff

    Thanks!
    We will add a wiki entry for this.

Share This Page