Help with high load server

Discussion in 'Install/Configuration' started by ARPLink, Jun 7, 2013.

  1. ARPLink

    ARPLink New Member

    Hi,

    I've been using litespeed for some time now, but since about 2-3 days ago, one of the servers started having loading problems. At first I thought it was being atacked, but then I noticed that there were a lot of http processes used(around 300-350) and the load was at a constant 10-12. Normally I wouldn't get alarmed because it's not such a high load, but clients started reporting that the websites backends(wp-admin for example) took forever to load.

    Could you please advise what I can tweak so admin backends start loading normally?(I'm guessing it's because the load exceeds the normal value of smaller than 8). MySQL is not a bottleneck, it has 2000 available connections and the average load is 1-1,5.

    In peak times, the server has around 350-400 processes:

    Code:
    ps -ef|wc
        358    3092   25232
    
    This is a load average from today:

    Code:
    07:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    09:00:03 AM         6      1201      6.33      7.41      7.55
    09:10:01 AM        16      1210      6.97      7.33      7.49
    09:20:02 AM        17      1188      9.76      9.22      8.36
    09:30:01 AM        39      1243      9.51      8.37      8.21
    09:40:01 AM         9      1182      7.27      8.08      8.18
    09:50:01 AM         4      1207     11.87      9.68      8.90
    10:00:01 AM        24      1207      8.66      9.88      9.44
    10:10:01 AM         4      1216     10.31     10.75     10.14
    10:20:03 AM        29      1249     11.44     11.28     10.64
    10:30:02 AM        11      1245     11.18     10.96     10.78
    10:40:02 AM        19      1201      8.88      9.93     10.31
    10:50:01 AM        18      1202     12.02     11.41     10.71
    11:00:01 AM         8      1205     11.44      9.89     10.02
    11:10:01 AM        14      1188      8.34      9.49      9.83
    11:20:02 AM        25      1217     10.42      9.51      9.65
    11:30:01 AM         6      1214      6.99      8.35      9.07
    11:40:01 AM         6      1210      8.40      8.59      8.76
    11:50:02 AM        19      1257      9.30     10.58      9.86
    12:00:01 PM        13      1233      9.34      9.37      9.43
    12:10:01 PM        13      1233      7.44      9.34      9.62
    12:20:01 PM         6      1247      5.84      7.28      8.56
    12:30:02 PM        13      1258      7.87      7.46      7.94
    Average:           13      1177      6.34      6.57      6.59
    
    The hardware setup is as follows:

    1.Http server(Cpanel)
    1 x Intel Xeon(quad-core)
    8 Gb RAM
    4 x 1 TB HDD in RAID 10

    2. MySQL server
    1 x Intel Xeon(quad-core)
    16 Gb RAM
    2 x 150 Gb HDD(SAS) in RAID 1

    My litespeed config is as follows:

    Code:
    <?xml version="1.0" encoding="UTF-8"?>
    <httpServerConfig>
      <serverName>$HOSTNAME</serverName>
      <user>nobody</user>
      <group>nobody</group>
      <priority>10</priority>
      <chrootPath>/</chrootPath>
      <enableChroot>0</enableChroot>
      <enableLVE>2</enableLVE>
      <inMemBufSize>2548M</inMemBufSize>
      <swappingDir>/tmp/lshttpd/swap</swappingDir>
      <autoRestart>1</autoRestart>
      <autoFix503>1</autoFix503>
      <gracefulRestartTimeout></gracefulRestartTimeout>
      <loadApacheConf>1</loadApacheConf>
      <autoReloadApacheConf>2</autoReloadApacheConf>
      <apacheBinPath>/usr/local/apache/bin/httpd</apacheBinPath>
      <apacheConfFile>/usr/local/apache/conf/httpd.conf</apacheConfFile>
      <apachePortOffset>0</apachePortOffset>
      <apacheIpOffset>0</apacheIpOffset>
      <apacheHandledContent></apacheHandledContent>
      <apacheIgnoredModules></apacheIgnoredModules>
      <phpSuExec>1</phpSuExec>
      <phpSuExecMaxConn>15</phpSuExecMaxConn>
      <apacheEnableFP></apacheEnableFP>
      <apacheEnvVars></apacheEnvVars>
      <mime>$SERVER_ROOT/conf/mime.properties</mime>
      <disableInitLogRotation>1</disableInitLogRotation>
      <showVersionNumber>0</showVersionNumber>
      <enableIpGeo></enableIpGeo>
      <useIpInProxyHeader>0</useIpInProxyHeader>
      <autoUpdateInterval>86400</autoUpdateInterval>
      <autoUpdateDownloadPkg>1</autoUpdateDownloadPkg>
      <adminEmails>admin@keyhost.ro</adminEmails>
      <adminRoot>$SERVER_ROOT/admin/</adminRoot>
      <logging>
        <log>
          <fileName>/usr/local/apache/logs/error_log</fileName>
          <logLevel>INFO</logLevel>
    <debugLevel>0</debugLevel>
          <rollingSize>2000M</rollingSize>
          <enableStderrLog>1</enableStderrLog>
        </log>
        <accessLog>
          <fileName>$SERVER_ROOT/logs/access.log</fileName>
          <rollingSize>2000M</rollingSize>
          <keepDays>30</keepDays>
          <compressArchive>0</compressArchive>
        </accessLog>
      </logging>
      <indexFiles>index.html, index.php, index.php5, index.htm</indexFiles>
      <autoIndex>1</autoIndex>
      <autoIndexURI>/_autoindex/default.php</autoIndexURI>
      <htAccess>
        <allowOverride>31</allowOverride>
        <accessFileName>.htaccess</accessFileName>
      </htAccess>
      <expires>
        <enableExpires>1</enableExpires>
        <expiresByType>image/*=A604800, text/css=A604800, application/x-javascript=A604800, application/javascript=A604800</expiresByType>
      </expires>
      <tuning>
        <maxConnections>1000</maxConnections>
        <maxSSLConnections>100</maxSSLConnections>
        <connTimeout>120</connTimeout>
        <maxKeepAliveReq>1</maxKeepAliveReq>
        <smartKeepAlive>0</smartKeepAlive>
        <keepAliveTimeout>3</keepAliveTimeout>
        <sndBufSize>0</sndBufSize>
        <rcvBufSize>0</rcvBufSize>
        <eventDispatcher>best</eventDispatcher>
        <maxCachedFileSize>16k</maxCachedFileSize>
        <totalInMemCacheSize>128M</totalInMemCacheSize>
        <maxMMapFileSize>2M</maxMMapFileSize>
        <totalMMapCacheSize>1024M</totalMMapCacheSize>
        <useSendfile>1</useSendfile>
    <useAIO>2</useAIO>
        <AIOBlockSize>7</AIOBlockSize>
        <fileETag>0</fileETag>
        <SSLCryptoDevice>null</SSLCryptoDevice>
        <maxReqURLLen>8K</maxReqURLLen>
        <maxReqHeaderSize>4096</maxReqHeaderSize>
        <maxReqBodySize>500M</maxReqBodySize>
        <maxDynRespHeaderSize>8K</maxDynRespHeaderSize>
        <maxDynRespSize>500M</maxDynRespSize>
        <enableGzipCompress>1</enableGzipCompress>
        <enableDynGzipCompress>1</enableDynGzipCompress>
        <gzipCompressLevel>2</gzipCompressLevel>
        <compressibleTypes>text/*, application/x-javascript, application/xml, application/javascript</compressibleTypes>
        <gzipAutoUpdateStatic>1</gzipAutoUpdateStatic>
        <gzipCacheDir>/home/loguri/gzip</gzipCacheDir>
        <gzipStaticCompressLevel>6</gzipStaticCompressLevel>
        <gzipMaxFileSize>1M</gzipMaxFileSize>
        <gzipMinFileSize>32K</gzipMinFileSize>
      </tuning>
      <security>
        <fileAccessControl>
          <followSymbolLink>1</followSymbolLink>
          <checkSymbolLink>0</checkSymbolLink>
          <requiredPermissionMask>000</requiredPermissionMask>
          <restrictedPermissionMask>000</restrictedPermissionMask>
        </fileAccessControl>
        <perClientConnLimit>
          <staticReqPerSec></staticReqPerSec>
          <dynReqPerSec></dynReqPerSec>
          <outBandwidth>0</outBandwidth>
          <inBandwidth>0</inBandwidth>
          <softLimit>20</softLimit>
          <hardLimit>50</hardLimit>
          <blockBadReq>1</blockBadReq>
          <gracePeriod>30</gracePeriod>
          <banPeriod>5</banPeriod>
        </perClientConnLimit>
    </perClientConnLimit>
        <CGIRLimit>
          <cgidSock></cgidSock>
          <maxCGIInstances>20</maxCGIInstances>
          <minUID>11</minUID>
          <minGID>10</minGID>
          <forceGID></forceGID>
          <priority>-15</priority>
          <CPUSoftLimit>0</CPUSoftLimit>
          <CPUHardLimit>0</CPUHardLimit>
          <memSoftLimit></memSoftLimit>
          <memHardLimit></memHardLimit>
          <procSoftLimit>20</procSoftLimit>
          <procHardLimit>25</procHardLimit>
        </CGIRLimit>
        <censorshipControl>
          <enableCensorship>1</enableCensorship>
          <logLevel>0</logLevel>
          <defaultAction>deny,log,status:403</defaultAction>
          <scanPOST>0</scanPOST>
          <disableSecHtaccess></disableSecHtaccess>
          <secAuditLogEngine></secAuditLogEngine>
          <secAuditLog>/home/loguri/sec_audit.log</secAuditLog>
        </censorshipControl>
        <accessDenyDir>
          <dir>/</dir>
          <dir>/etc/*</dir>
          <dir>/dev/*</dir>
          <dir>$SERVER_ROOT/conf/*</dir>
          <dir>$SERVER_ROOT/admin/conf/*</dir>
        </accessDenyDir>
        <accessControl>
          <allow>ALL</allow>
          <deny></deny>
        </accessControl>
      </security>
      <extProcessorList>
        <extProcessor>
     <type>lsapi</type>
          <name>lsphp5</name>
          <address>uds://tmp/lshttpd/lsphp5.sock</address>
          <note></note>
          <maxConns>300</maxConns>
          <env>PHP_LSAPI_MAX_REQUESTS=100</env>
          <env>PHP_LSAPI_CHILDREN=300</env>
          <initTimeout>60</initTimeout>
          <retryTimeout>0</retryTimeout>
          <persistConn>0</persistConn>
          <pcKeepAliveTimeout>5</pcKeepAliveTimeout>
          <respBuffer>0</respBuffer>
          <autoStart>1</autoStart>
          <path>$SERVER_ROOT/fcgi-bin/lsphp5</path>
          <backlog>100</backlog>
          <instances>1</instances>
          <extUser></extUser>
          <extGroup></extGroup>
          <runOnStartUp>0</runOnStartUp>
          <extMaxIdleTime>2</extMaxIdleTime>
          <priority>10</priority>
          <memSoftLimit>256M</memSoftLimit>
          <memHardLimit>312M</memHardLimit>
          <procSoftLimit>200</procSoftLimit>
          <procHardLimit>200</procHardLimit>
        </extProcessor>
    
    I cut it short because that's all that I need to tune. I don't use Ruby.
  2. webizen

    webizen New Member

    1. paste 'top -c' from command line so we can see which process takes more cpu cycles.

    2. check Real-Time Stats in LSWS Admin Console => Actions and see if anything in the WaitQ column.

    3. also, try request static URL instead of dynamic (requires db access) and if still takes long time.
  3. ARPLink

    ARPLink New Member

    Hi,

    I have 2 top -c outputs. Most of the heavy sites have cache scripts(most of the are wordpress and I've installed supercache).

    Code:
    top - 11:27:00 up 85 days,  4:25,  3 users,  load average: 7.78, 6.90, 6.10
    Tasks: 252 total,   7 running, 245 sleeping,   0 stopped,   0 zombie
    Cpu(s): 57.0%us, 27.4%sy,  0.0%ni,  8.6%id,  1.3%wa,  0.0%hi,  3.7%si,  2.0%st
    Mem:   7340032k total,  6611876k used,   728156k free,   519160k buffers
    Swap:  4194296k total,    35164k used,  4159132k free,  3153480k cached
    
        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     706820 co***asa  16   0  166m  62m  22m S 57.0  0.9   0:01.12 lsphp5
     706829 pi***ex   17   0  157m  48m  18m R 54.4  0.7   0:01.07 lsphp5:/home/pi***ex/public_html/index.php
     706781 m***tord  16   0  195m  34m  19m S 50.9  0.5   0:01.40 lsphp5
     706828 s***elib  15   0  174m  70m  22m S 50.4  1.0   0:00.99 lsphp5
     706836 u****kid  15   0  167m  64m  22m S 38.2  0.9   0:00.75 lsphp5
     706825 su**t     18   0  159m  55m  22m S 37.1  0.8   0:00.73 lsphp5
     706830 g***pele  15   0  168m  64m  22m S 35.1  0.9   0:00.69 lsphp5:/home/gh****ele/public_html/index.php
     706818 f***ec    16   0  169m  64m  22m S 33.1  0.9   0:00.65 lsphp5
     598709 nobody    18   0  325m 154m  17m D 30.0  2.1   6:00.30 litespeed (lshttpd)
     706823 o****tiv  15   0  158m  52m  22m S 29.5  0.7   0:00.58 lsphp5:/home/o****tiv/public_html/index.php
     706819 gh****le  15   0  152m  49m  22m S 28.5  0.7   0:00.56 lsphp5
     706840 c*****se  17   0  146m  42m  22m S 25.4  0.6   0:00.50 lsphp5
     706826 d****20   18   0  158m  54m  22m S 24.4  0.8   0:00.48 lsphp5
     706842 s****et   18   0  163m  58m  22m R 21.4  0.8   0:00.42 lsphp5:/home/s***et/public_html/index.php
     706827 mo***ord  16   0  202m  39m  18m R 15.8  0.6   0:00.31 lsphp5:/home/m****rd/public_html/index.php
     706816 mo***ord  15   0  201m  40m  19m S 15.3  0.6   0:00.71 lsphp5
     706822 r****re   15   0  139m  31m  17m S 15.3  0.4   0:00.30 lsphp5
     706844 i***za    16   0  152m  44m  19m R 13.7  0.6   0:00.27 lsphp5:/home/i****za/public_html/index.php
     706841 im***me   15   0  156m  35m 9252 S 12.7  0.5   0:00.25 lsphp5:/home/i****me/public_html/index.php
     598708 nobody    17   0  299m 139m  14m D 12.2  1.9   4:43.17 litespeed (lshttpd)
     706766 o****tiv  15   0  159m  54m  22m S 10.7  0.8   0:00.74 lsphp5:/home/ob***tiv/public_html/index.php
     706787 p***ob    18   0  153m  50m  22m S 10.2  0.7   0:01.01 lsphp5
     706848 s***ic    15   0  146m  35m  15m R 10.2  0.5   0:00.20 lsphp5:/home/s***ic/public_html/index.php
     706843 v****nin  17   0  137m  20m 9.9m D  6.1  0.3   0:00.12 lsphp5:/home/v****unin/public_html/index.php
     706837 f***ec    18   0  132m  13m 7652 S  5.1  0.2   0:00.10 lsphp5
     706765 ob**ctiv  15   0  159m  54m  22m S  4.6  0.8   0:00.49 lsphp5
     706838 fi**ec    18   0  132m  13m 7652 S  4.6  0.2   0:00.09 lsphp5
     706839 z***o     18   0  134m  13m 7732 S  4.6  0.2   0:00.09 lsphp5
     706833 o*****iv  18   0  132m  12m 6544 S  4.1  0.2   0:00.08 lsphp5
     706835 i****e    20   0  131m  10m 5628 S  4.1  0.1   0:00.08 lsphp5
     706824 p****67   18   0  131m  10m 5820 S  3.6  0.1   0:00.07 lsphp5
     706834 i****e    18   0  131m  10m 5628 S  3.6  0.1   0:00.07 lsphp5
     706821 an***om   18   0  132m  12m 7232 S  3.1  0.2   0:00.06 lsphp5
     706832 rb***re   18   0  132m  10m 6052 D  3.1  0.2   0:00.06 lsphp5:hemes/branfordmagazine-4/scripts/timthumb.php
      41344 named     18   0  373m  61m 2052 S  1.0  0.9 135:11.51 /usr/sbin/named -u named
     459133 root      15   0 12864 1276  828 S  1.0  0.0   0:32.46 top
     706809 gi***81   18   0  171m  66m  23m S  1.0  0.9   0:00.54 lsphp5:home/g***1/public_html/wp-comments-post.php
    
    Code:
    top - 11:08:57 up 85 days,  4:07,  3 users,  load average: 6.43, 6.09, 5.33
    Tasks: 236 total,   8 running, 227 sleeping,   0 stopped,   1 zombie
    Cpu(s): 31.6%us, 10.7%sy,  0.0%ni, 53.9%id,  1.0%wa,  0.0%hi,  2.4%si,  0.3%st
    Mem:   7340032k total,  6429340k used,   910692k free,   542788k buffers
    Swap:  4194296k total,    35164k used,  4159132k free,  3204008k cached
    
        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     686202 q***n  15   0  171m  68m  22m R 36.2  1.0   0:02.28 lsphp5:/home/q***zin/public_html/index.php
     686252 s***elib  16   0  189m  85m  22m R 28.6  1.2   0:00.86 lsphp5:/home/s***lib/public_html/index.php
     686250 r***ire   16   0  162m  58m  22m R 23.6  0.8   0:00.71 lsphp5:/home/rb***re/public_html/index.php
     686180 r***ire   15   0  161m  55m  22m S 21.6  0.8   0:01.84 lsphp5
     686251 g***pele  15   0  168m  64m  22m S 20.9  0.9   0:00.63 lsphp5:/home/gh***ele/public_html/index.php
     686256 m***ncri  17   0  179m  74m  22m R 19.9  1.0   0:00.60 lsphp5:/home/ma***cri/public_html/index.php
     686248 f***ec    15   0  169m  65m  22m S 16.6  0.9   0:00.50 lsphp5
     686249 g***881   15   0  166m  62m  22m S 15.9  0.9   0:00.48 lsphp5
     686242 ***iaza   15   0  193m  28m  14m S 14.3  0.4   0:00.43 lsphp5
     686228 urb***id  15   0  176m  71m  22m R 13.3  1.0   0:01.36 lsphp5:/home/urb***id/public_html/index.php
     686258 ja***and  15   0  155m  50m  22m R 12.9  0.7   0:00.39 lsphp5:/home/jan***and/public_html/index.php
     686246 pe***ana  20   0  155m  49m  22m S 11.0  0.7   0:00.33 lsphp5
     686253 g***pele  16   0  154m  49m  22m R 10.0  0.7   0:00.30 lsphp5:/home/ghi***le/public_html/index.php
     686255 m***tord  15   0  200m  36m  16m S 10.0  0.5   0:00.30 lsphp5:/home/mo***rd/public_html/index.php
     686233 n***s     17   0  197m  30m  15m S  8.6  0.4   0:00.34 lsphp5
     598709 nobody    15   0  293m 116m  20m S  7.0  1.6   4:24.77 litespeed (lshttpd)
     598708 nobody    18   0  280m 113m  14m S  4.3  1.6   3:25.37 litespeed (lshttpd)
     686257 p***oana  18   0  131m  11m 6272 S  2.0  0.2   0:00.06 lsphp5
     686254 pr***eri  17   0  187m  11m 6408 S  1.7  0.2   0:00.05 lsphp5
      41344 named     18   0  373m  61m 2052 S  1.3  0.9 135:08.96 /usr/sbin/named -u named
     598664 root      15   0  107m  72m 2272 S  1.0  1.0   0:23.13 litespeed (lshttpd)
     395667 root      18   0 13412 1768  780 S  0.3  0.0  84:04.01 top
     657463 root      15   0 12888 1288  832 R  0.3  0.0   0:04.66 top
          1 root      16   1 10372  648  552 S  0.0  0.0   0:39.80 init [3]
          3 root      39  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/0]
          5 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/1]
          7 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/2]
          9 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/3]
         11 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/4]
         13 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/5]
         15 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/6]
         17 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/7]
         18 root      10  -5     0    0    0 S  0.0  0.0   1:11.82 [events/0]
         19 root      10  -5     0    0    0 S  0.0  0.0   0:40.95 [events/1]
         20 root      10  -5     0    0    0 S  0.0  0.0   0:38.57 [events/2]
         21 root      10  -5     0    0    0 S  0.0  0.0   0:38.38 [events/3]
         22 root      10  -5     0    0    0 S  0.0  0.0   0:38.35 [events/4]
    
  4. webizen

    webizen New Member

    for quad-core svr, 10-12 cpu load avg isn't very high.

    have you ever tested any static pages and see if they are slow?

    most cpu cycles on PHP. improve PHP code or put in opcode cache (APC, xcache, eAccelerator) to speed up. Also, recommend followings for per client throttling settings to prevent any http level ddos attack (excessive requests from individual remote IP).

  5. ARPLink

    ARPLink New Member

    Hi webizen,

    Thanks for the suggestions. I have eAccelerator installed but it's not wanting to work.

    Here's an output of the disk space the cache folder occupies.

    root@s1 [/home/l***ri]# du -sh eaccelerator
    1.2M eaccelerator

    Also, permissions:
    root@s1 [/home/l***ri]# ll -h|grep eaccelerator
    drwxrwxrwx 18 root root 4.0K Jun 8 05:16 eaccelerator/

    And php.ini config:

    Code:
    extension="eaccelerator.so"
    eaccelerator.cache_dir="/home/lo***i/eaccelerator"
    eaccelerator.check_mtime="1"
    eaccelerator.compress="1"
    eaccelerator.compress_level="9"
    eaccelerator.debug="0"
    eaccelerator.enable="1"
    eaccelerator.filter=""
    eaccelerator.optimizer="1"
    eaccelerator.shm_max="0"
    eaccelerator.shm_only="0"
    eaccelerator.shm_prune_period="0"
    eaccelerator.shm_size="16"
    eaccelerator.shm_ttl="0"
    
    
  6. webizen

    webizen New Member

    since you are shared hosting, to leverage opcode cache(EA), you need to run lsphp5 in suEXEC Daemon mode:
    EA cache_dir should point to somewhere that anyone can access like
  7. ARPLink

    ARPLink New Member

    Hi,

    I implemented all the changes you indicated in your last reply, but still EA does not want to cache.

    root@s1 [/tmp]# du -s eaccelerator
    1216 eaccelerator

    It just creates the cache folders and that's it.
  8. webizen

    webizen New Member

    switch to apache and see if EA starts to cache.
  9. anewday

    anewday Moderator

    What's the owner of that eaccelerator cache folder? It has to be the same as for lsws.

    Here's mine:

    drwxr-xr-x 18 nobody nobody

Share This Page