Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
litespeed_wiki:php_503_error [2014/02/05 17:13]
Michael [Real World Examples]
litespeed_wiki:php_503_error [2015/07/28 13:11]
Michael Alegre removed
Line 1: Line 1:
-===== Troubleshooting 503 Errors with LiteSpeed Web Server ​(Under Construction)=====+====== Troubleshooting 503 Errors with LiteSpeed Web Server ​======
  
 503 errors are often caused by a malfunction in PHP. This wiki will go over basic steps to troubleshoot 503 errors, some common causes of 503 errors, and some examples that show the steps in practice. 503 errors are often caused by a malfunction in PHP. This wiki will go over basic steps to troubleshoot 503 errors, some common causes of 503 errors, and some examples that show the steps in practice.
  
-===== Basic Troubleshooting steps =====+====== Basic Troubleshooting steps ======
  
-==== 1. Check the Error Log ====+===== 1. Check the Error Log =====
  
 For native LSWS setups, the error log is usually named ''​error.log''​. It may be named ''​error_log''​ if you are using WHM/​cPanel. ​ For native LSWS setups, the error log is usually named ''​error.log''​. It may be named ''​error_log''​ if you are using WHM/​cPanel. ​
Line 26: Line 26:
  
  
-=== Interpreting these entries: ===+==== Interpreting these entries: ​====
  
   * The time stamp - We can use this time stamp later to find related entries in other logs.   * The time stamp - We can use this time stamp later to find related entries in other logs.
Line 37: Line 37:
 From these entries, we can tell that the above error did not cause a crash. We know what time the error occurred and to which process. We also know that the problem is probably in the php.ini, since the error occurred before the process took the request. From these entries, we can tell that the above error did not cause a crash. We know what time the error occurred and to which process. We also know that the problem is probably in the php.ini, since the error occurred before the process took the request.
  
-==== 2. Check for Corresponding Entries in stderr.log ====+===== 2. Check for Corresponding Entries in stderr.log ​=====
  
 The stderr.log logs errors from the standard error stream. This log can give you additional information about errors that occurred. Using the time stamps and PIDs you've gotten from the error log, you may be able to find relevant errors in the stderr.log. The stderr.log can usually be found in the same directory as the error log. The stderr.log logs errors from the standard error stream. This log can give you additional information about errors that occurred. Using the time stamps and PIDs you've gotten from the error log, you may be able to find relevant errors in the stderr.log. The stderr.log can usually be found in the same directory as the error log.
Line 51: Line 51:
 If you cannot find errors in stderr.log, you may have to resort to testing common causes of PHP errors (addressed below). If you cannot find errors in stderr.log, you may have to resort to testing common causes of PHP errors (addressed below).
  
-==== 3. Enable Core Dump (or just turn off opcode caching) ====+===== 3. Enable Core Dump (or just turn off opcode caching) ​=====
  
 If you find that PHP has crashed (as demonstrated by the process being "​killed by signal"​),​ a core dump will allow you to look further into the cause of the crash. If you find that PHP has crashed (as demonstrated by the process being "​killed by signal"​),​ a core dump will allow you to look further into the cause of the crash.
  
-To enable a core dump, add the environment value ''​LSAPI_ALLOW_CORE_DUMP=1'' ​to your external application settings (WebAdmin console > Configuration > External App). Next time the application crashes, a core dump will be generated. The core file created can usually be found in the directory holding the PHP script affected.+To enable a core dump, add the environment value  
 +  ​LSAPI_ALLOW_CORE_DUMP=1 ​ 
 +to your external application settings (WebAdmin console > Configuration > External App). Next time the application crashes, a core dump will be generated. The core file created can usually be found in the directory holding the PHP script affected.
  
-**Note:** As noted belowopcode caches ​are frequently a cause of PHP crashesIf you find your PHP crashedyou may want to try turning off any opcode caching you have. This addressed further [[litespeed_wiki:​php_503_error#​opcode_caches_apc_xcache_eaccelerateor|below]].+Your Redhat/​Centos system may already use ABRT(Automatic Bug Reporting Tool) tool to generate core dump files. In this casecore files may be located at /​var/​spool/​abrt/​. ​ ABRT configuration files are normally located at /etc/abrt/ ​Please ensure the setting ​  "​ProcessUnpackaged = yes" in /​etc/​abrt/​abrt-action-save-package-data.conf to enable the core dumpotherwise core file may not be generated even the web server error log says so.  If you want to disable core file dump after the debugging, simply change "​ProcessUnpackaged"​ setting back to "​no"​
  
-==== 4Analyze Core File with GNU Debugger ==== +core_pattern is used to specify a core dumpfile pattern name.  
 +  cat /​proc/​sys/​kernel/​core_pattern 
 +which shows you the pattern template for the output filename.
  
-GNU Debugger (GDB) uses the syntax ''​gdb <​path/​to/​lsphp/​binary>​ <​path/​to/​core/​file>''​. (Your LSPHP binary can usually be found in ''/​usr/​local/​lsws/​fcgi-bin''​.)+If the first character of the pattern is a '​|',​ the kernel will treat the rest of the pattern as a command to run.  The core dump will be written to the standard input of that program instead of to a file.  
 + 
 +By default, /​proc/​sys/​kernel/​core_pattern contains core string and kernel produces core.* files in crashed process'​s current directory. 
 + 
 +Abrt’s C/C++ hook overrides this with: 
 +  |/​usr/​libexec/​abrt-hook-ccpp %s %c %p %u %g %t e 
 +which results in kernel calling abrt-hook-ccpp 
 + 
 +**Note:** As noted below, opcode caches are frequently a cause of PHP crashes. If you find your PHP crashed, you may want to try turning off any opcode caching you have. This is addressed further [[litespeed_wiki:​php_503_error#​opcode_caches_apc_xcache_eaccelerateor|below]]. 
 +===== 4. Analyze Core File with GNU Debugger =====  
 + 
 +GNU Debugger (GDB) uses the syntax ''​gdb <​path/​to/​lsphp/​binary>​ <​path/​to/​core/​file>''​. (Your LSPHP binary can usually be found in ''/​usr/​local/​lsws/​fcgi-bin''​. ​If you installed LSPHP through LiteSpeed Repository, LSPHP binary file usually locates in ''/​usr/​local/​lsws/​lsphp5x/​lsphp''​('​x'​ means PHP version of '​3'​ '​4'​ '​5'​ or '​6'​) ​)
  
 Once you have opened the core file with GDB, use ''​bt''​ command to print a backtrace of the stack (the steps that the process application took coming to the crash). This will often reveal what caused the crash. Once you have opened the core file with GDB, use ''​bt''​ command to print a backtrace of the stack (the steps that the process application took coming to the crash). This will often reveal what caused the crash.
Line 92: Line 107:
 This backtrace shows us that the program crashed soon after accessing the opcode cache, suggesting that the issue is with the opcode cache. The user can then try upgrading the opcode cache or changing their PHP version, or disabling opcode cache if neither of those work. This backtrace shows us that the program crashed soon after accessing the opcode cache, suggesting that the issue is with the opcode cache. The user can then try upgrading the opcode cache or changing their PHP version, or disabling opcode cache if neither of those work.
  
-===== Common Causes of 503 Errors =====+====== Common Causes of 503 Errors ​======
  
-==== 1. Bad php.ini ====+===== 1. Bad php.ini ​=====
  
 Faulty configurations or directives in your php.ini can cause a fatal error and make the process exit right at the beginning. Faulty configurations or directives in your php.ini can cause a fatal error and make the process exit right at the beginning.
Line 100: Line 115:
 php.ini problems generally show an "​error:​ Connection reset by peer!" error and have a confirmed value of ''​0'',​ meaning that the process never took the request. These errors will often be explained in your stderr.log. They may require that you comment out bad directives or fix faulty configurations. php.ini problems generally show an "​error:​ Connection reset by peer!" error and have a confirmed value of ''​0'',​ meaning that the process never took the request. These errors will often be explained in your stderr.log. They may require that you comment out bad directives or fix faulty configurations.
  
-==== 2. PHP Binary-Module Mismatch ====+===== 2. PHP Binary-Module Mismatch ​=====
  
 PHP will return an error if one of your modules uses an API that does not match your PHP version. The following is an example of an error shown when there is a binary-module mismatch: PHP will return an error if one of your modules uses an API that does not match your PHP version. The following is an example of an error shown when there is a binary-module mismatch:
Line 109: Line 124:
   These options need to match in Unknown on line 0   These options need to match in Unknown on line 0
  
-To fix this, you will need to either rebuild the module and PHP, making sure that you are using a version of PHP that works with the module. Also, make sure that the correct extension path is used.+To fix this, you will need to either rebuild the module and/or PHP, making sure that you are using a version of PHP that works with the module. Also, make sure that the correct extension path is used.
  
-==== 3. Opcode Caches (APC, xCache, EAccelerateor) ====+===== 3. Opcode Caches (APC, xCache, EAccelerateor) ​=====
  
 The different opcode caches often have compatibility issues with PHP. These issues may be uncovered when looking through the backtrace of a PHP crash. Often, if you are getting a 503 error, and especially if you see PHP crashing, it may be a good idea to try turning off your opcode cache to see if it solves the problem. The different opcode caches often have compatibility issues with PHP. These issues may be uncovered when looking through the backtrace of a PHP crash. Often, if you are getting a 503 error, and especially if you see PHP crashing, it may be a good idea to try turning off your opcode cache to see if it solves the problem.
Line 121: Line 136:
 If you find that the opcode cache is causing the error, you can try upgrading your version of the opcode cache or using a different version of PHP. If that does not work, disable the opcode cache and possibly try a different opcode cache. You may also want to submit a bug report to the opcode cache developer. If you find that the opcode cache is causing the error, you can try upgrading your version of the opcode cache or using a different version of PHP. If that does not work, disable the opcode cache and possibly try a different opcode cache. You may also want to submit a bug report to the opcode cache developer.
  
-==== 4. Third Party Modules (ZendGuardLoader,​ Suhosin, ionCube, etc.) ====+===== 4. Third Party Modules (ZendGuardLoader,​ Suhosin, ionCube, etc.) =====
  
 Third part modules also often have compatibility issues as well. To turn off the modules, comment out the line with the extension in your php.ini files. Restart LSWS and try the page in question again. Third part modules also often have compatibility issues as well. To turn off the modules, comment out the line with the extension in your php.ini files. Restart LSWS and try the page in question again.
Line 127: Line 142:
 Just as with opcode caches, if you find that a third party module is causing the error, you can try upgrading your version of the module or using a different version of PHP. If that does not work, disable the module and consider submitting a bug report to the module'​s developer. Just as with opcode caches, if you find that a third party module is causing the error, you can try upgrading your version of the module or using a different version of PHP. If that does not work, disable the module and consider submitting a bug report to the module'​s developer.
  
-==== 5. Module Loading Order ====+===== 5. Module Loading Order =====
  
 Sometimes, the module loading order makes a difference. Shuffling the order that modules are listed in your php.ini has been known to fix issues. Sometimes, the module loading order makes a difference. Shuffling the order that modules are listed in your php.ini has been known to fix issues.
  
-==== 6. "​zend_extension"​ Instead of "​extension"​ and Vice Versa ====+===== 6. "​zend_extension"​ Instead of "​extension"​ and Vice Versa =====
  
 ''​zend_extension''​ is used for Zend's own extensions, such as frameworks or optimizers (like ionCube, ZendGuardLoader,​ or ZendOptimizer). ''​extension''​ is for everything else, such as PEAR, PECL, etc. ''​zend_extension''​ is used for Zend's own extensions, such as frameworks or optimizers (like ionCube, ZendGuardLoader,​ or ZendOptimizer). ''​extension''​ is for everything else, such as PEAR, PECL, etc.
Line 142: Line 157:
   - Comment out the ''​extension_dir''​ line to let PHP pick a default.   - Comment out the ''​extension_dir''​ line to let PHP pick a default.
  
-===== Real World Examples =====+===== 7. lsphp process is killed unexpectedly ===== 
 + 
 +when php script is executing, if the process is killed by admin or a process monitoring daemon, it'll simply result 503 error. 
 + 
 +An Example: 
 + 
 +A WHM/cPanel server, installed a plugin "​ConfigServer Security & Firewall"​(i.e.,​CSF / LFD), it killed lsphp5 process from time to time and result 503 error. 
 + 
 +/​etc/​csf/​csf.conf : 
 +  # This User Process Tracking option sends an alert if any cPanel user process 
 +  # exceeds the time usage set (seconds). To ignore specific processes or users 
 +  # use csf.pignore 
 +  # 
 +  # Set to 0 to disable this feature 
 +  PT_USERTIME = "​1800"​ 
 +   
 +so if lsphp5 process has run 1800 seconds(30 minutes), it might be caught by csf and killed. in php suExec Daemon mode or ProcessGroup mode, it's normal that the parent lsphp5 process keep running over 30 minutes. when csf/lfd kill lsphp5 process, it'll leave logs in /​var/​log/​lfd.log,​ like 
 + 
 +  Jun 19 16:29:16 evo lfd[18304]: *User Processing* PID:18264 Kill:1 User:xxxxx VM:538(MB) EXE:/​usr/​local/​lsws/​fcgi-bin/​lsphp-5.4.42 CMD:​lsphp5 
 + 
 +the time stamp match 503 error in /​usr/​local/​apache/​logs/​error_log:​ 
 + 
 +  2015-06-19 16:​29:​16.370 [NOTICE] [173.245.50.197:​61317-0#​APVH_pingje.org] oops! 503 Service Unavailable 
 + 
 +the fix: 
 + 
 +add 
 +  pexe:/​usr/​local/​lsws/​fcgi-bin/​lsphp.* 
 +to end of /​etc/​csf/​csf.pignore , then restart csf / lfd 
 +  # csf -r 
 + 
 +or in WHM, 
 +  Home » Plugins » ConfigServer Security & Firewall 
 +  lfd - Login Failure Daemon 
 +  "​csf.pignore,​ Process Tracking"​ Edit "lfd ignore file"​ 
 +  append following line 
 +  pexe:/​usr/​local/​lsws/​fcgi-bin/​lsphp.* 
 +  "​Restart Lfd" 
 + 
 +===== 8. lsphp process hit memory limit ===== 
 + 
 +when define a lsphp external application,​ there are 2 litespeed specific settings: 
 + 
 +  * Memory Soft Limit (bytes) (https://​www.litespeedtech.com/​docs/​webserver/​config/​extapps#​memSoftLimit) 
 +  * Memory Hard Limit (bytes) (https://​www.litespeedtech.com/​docs/​webserver/​config/​extapps#​memHardLimit) 
 + 
 +As stated in the document: "The main purpose of this limit is to prevent excessive memory usage because of software bugs or intentional attacks, not to impose a limit on normal usage. Make sure to leave enough head room, otherwise your application may fail and 503 error may be returned."​ 
 + 
 +so for example in a shared hosting server, with memory soft/hard limit set, if an php script in an account consumes too many memory, it'll fail(return 503 error) and other accounts are not affected. the ideal solution is to optimize the php script to consume less memory. A quick and temporary workaround is to raise the soft/hard limit to a big value(for example 8G), to see if the 503 error will be gone. here's a use case : https://​www.litespeedtech.com/​support/​forum/​threads/​solved-xenforo-rebuild-attachment-thumbnails-503.12403/​ 
 +====== Real World Examples ​======
  
 [[litespeed_wiki:​php_503_error_example1|Bad Directive in php.ini]] [[litespeed_wiki:​php_503_error_example1|Bad Directive in php.ini]]