Differences

This shows you the differences between two versions of the page.

--- litespeed_wiki:php:server-load-issue [2017/02/01 21:40]
Jackson Zhang
+++ litespeed_wiki:php:server-load-issue [2019/08/27 12:58] (current)
Jackson Zhang
@@ Line 1: / Line 1: @@
-====== How to resolve high server load ======
+====== How to Resolve High Server Load ======
-If your server load is high (server loads that are < = the number of CPU Cores in your server are quite normal), you need to locate the bottleneck and make optimizations targeting it.
+If your server load is high (server loads that are <nowiki><=</nowiki> the number of CPU Cores in your server are quite normal), you need to locate the bottleneck and make optimizations targeting it.
-  * It is always good practice to use the ''top'' command to check which processes are using a lot of CPU resources and figure out whether they are behaving normally.
-  * If disk I/O is high, it is likely a disk I/O problem, which you should address. If you have more free memory available, you can move php sessions to ''/dev/shm''.
+Most of the time, high server load is caused by PHP processes and MySQL, //not// by the LiteSpeed Web Server process itself. No matter how fast LiteSpeed runs, it has no control over how fast a PHP script can run, or how much memory a PHP script will consume. Hence, in general, troubleshooting server load issue is not a matter of LiteSpeed Web Server's configuration, and is beyond our support scope. (Analysis of your DDoS attack is beyond our support scope as well, unless you would like to [[https://store.litespeedtech.com/store/cart.php?gid=5|engage us through paid hourly support]].)
+In the spirit of helpfulness, however, we are happy to provide the following troubleshooting tips for you to use when investigating high server load.
+  * Try to use our [[https://www.litespeedtech.com/products/cache-plugins|LSCache solutions]] to lower the load.
+  * It is always good practice to use the ''top'' or ''atop'' command to check which processes are using a lot of CPU resources, and then figure out whether they are behaving normally.
+  * If disk I/O is high, it is likely a disk I/O problem, which you should address. If you have more free memory available, you can move php sessions to ''/dev/shm''. Disk I/O issue example: <code>top -c</code> iowait ''41.6 wa'' is too high. then check further <code>iostat -x
+Linux 3.10.0-962.3.2.lve1.5.24.10.el7.x86_64 (lbr27.lalunahost.com.br)  04/18/2019      _x86_64_        (8 CPU)
+avg-cpu:  %user   %nice %system %iowait  %steal   %idle
+.65    0.15    2.16   20.24    0.00   70.80
+Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
+sdb               0.16     7.43   16.61    1.80    74.74    78.56    16.65     2.46  133.39    8.38 1284.98   1.45   2.68
+sda              28.25   134.39   93.98   54.24  1445.73  1227.69    36.07     2.11   14.18   46.67   79.17   4.86  72.08</code> ''/dev/sda'' has problem with ''72.08%util'', ''/dev/sdb'' is ok.
   * Check the CPU idle percentage. If it is close to 0%, then your server is CPU-bound and you should try to reduce the CPU usage.
-  * Try to use our LSCache solutions to lower the load.
+  * When CloudLinux is used, you should try to use the LVE limit to prevent any one user from consuming all of the resources.
+  * Try PHP7 instead of PHP5 if your code is compatible with it.
   * Enable PHP opcode cache.
-  * Check the real time stats for the number of PHP processes during peak time (''LSWS Web Admin → Actions → Real-Time Stats→External Application''), check ''In Use'', ''Idle'', and ''WaitQ''. For shared hosting, if one user uses too many PHP processes while others only use a few, you may adjust PHP max concurrency setting to be a smaller number. See our [[https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:php:concurrency|PHP Concurrency wiki]] for more information on determining if PHP SuEXEC is used and how to reduce the max concurrency of PHP.
+  * Check the real time stats for the number of PHP processes during peak time under **Actions > Real-Time Stats > External Application**, check ''In Use'', ''Idle'', and ''WaitQ''. For shared hosting, if one user uses too many PHP processes while others only use a few, you may adjust PHP max concurrency setting to be a smaller number. See our [[https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:php:concurrency|PHP Concurrency wiki]] for more information on determining if PHP SuEXEC is used and how to reduce the max concurrency of PHP.
-  * Check the number of concurrent users to see if it is normal: ''netstat -na | grep 80 | grep ESTA''
+  * DDoS attack may push the server to high load. Check the number of concurrent users to see if it is normal: ''netstat -na | grep 80 | grep ESTA''. \\ \\ To check concurrent connections sorted by IP, run the following: ''netstat -ntu | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr''. \\ \\ Sometimes you may count ''time_waits'' as well by running the command without ''grep ESTABLISHED'', which is ''netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr''. It is useful in some special situations (i.e., an attacker could make a connection and then send requests to an expensive URL, wait a little while, and then close the connection. If the server does not abort the process, the backend will be used up soon and keep serving a request that has been abandoned. Hence, you wait to analyze IP even with ''time_waits''). \\ \\ You can try some of the [[litespeed_wiki:waf|LSWS WAF features]] to mitigate DDoS attacks. If you can identify one or two IPs causing the problem, you should find a way to limit the resources that one or two bad IPs can consume. The web server has no control over how fast a PHP script can run, or how much memory a PHP script will consume. Reasonably, LVE limit by CloudLinux can help when someone abuses one account. If a server is under attack or being abused, you need to find a way to stop the attack or abuse, such as blocking bad IPs with a blacklist or using a mod_security rule set. No matter how fast a web server can run, it can only handle the amount of traffic the backend (PHP and MySQL) can handle, unless content is served from cache.
   * For control panel users, check your PHP configuration and make sure Apache and LiteSpeed have matching PHPs.
   * For cpanel/WHM user, disable PHP open_basedir feature.
-  * Sometimes high load can be caused by poorly written PHP/SQL code and has to be fixed by your PHP Developers. Sometimes it is a MySQL-based problem, for example, you may have a missing index. To debug this, you may try to use ''mysqladmin'' and ''processlist''. No web server configuration change can help such situations and they are beyond our support scope. You can even use ''strace'' to track your PHP processes: ''strace -tt -T -p <pid_of_a_php_process>''
+  * Sometimes high load can be caused by poorly written PHP/SQL code and has to be fixed by your PHP Developers. Sometimes it is a MySQL-based problem, for example, you may have a missing index. To debug this, you may try to use ''mysqladmin'' and ''processlist''. If you have spare RAM, trying to raise database cache pool/buffer may help. No web server configuration change can help such situations and they are beyond our support scope. You can even use ''strace'' to track your PHP processes: ''strace -tt -T -p <pid_of_a_php_process>''
 <code>
   top - 15:28:56 up 1 day, 17:58, 3 users, load average: 187.42, 143.52, 109.09