Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
litespeed_wiki:config:bug-reporting [2017/11/01 15:24]
Jackson Zhang [How to switch to the debug build of LSWS Enterprise]
litespeed_wiki:config:bug-reporting [2020/01/03 17:52] (current)
George Wang [How to Toggle Debug Logging and Capture Output]
Line 1: Line 1:
-====== How to submit ​bug report ​====== +====== How to Submit ​Bug Report ​======
-To submit a useful bug report you need to provide as much information as you can. Including+
  
-  ​Your Server OS (Linux/​FreeBSD/​Solaris/​MacOSX) +**NOTE**: Always upgrade ​to the latest ​build of the latest version before submitting a report!
-  ​CPU/​Processor family (32bit/​64bit) +
-  ​If you think LSWS does not work properly with a certain feature, you should provide as detailed information as possible on the feature and how you use it, so we can reproduce the problem in our lab. +
-  ​If lshttpd crashed, LSWS Enterprise users should switch ​to the debug build of lshttpd, then submit ​the debug report ​generated and a core file if possible. +
-  * gdb backtrace information on core file.+
  
-If you receive some email alert on the crash, ​you can forward that email to us. If not, you can use ''​gdb''​ to backtrace the crash and send to us for further investigation. If the error log shows server crashed but no core file generated, you may need to enable core dump for your system and reboot your server. +There are two cases where you can log a bug report:
-===== Debug build of LSWS Enterprise ​ =====+
  
-Starting from version 5.0.x, you do not need separate debug build since the production ​LSWS binary installed has debug info enabled by defaultIt has no impact ​on your server performance.+  - Server Crash Report: Submit if the server crashes and core files were generated at ''/​tmp/​lshttpd/​bak_core/''​ 
 +  - Bug Report: Submit if the server doesn'​t crash, but LSWS doesn'​t work as expected, especially when the same thing works with Apache in a control panel environmentTurn on debug logging when reproducing the issue, then send the detailed debug log for us to investigate
  
-For versions before 5.0there are two versions of the lshttpd binary shipped in the LSWS Enterprise package. One is the release ​build. The other is the debug build. The debug build is not normally used when you run LSWS, but, when debugging, the GDB stack backtrace and core file generated by the debug build will be used for bug analysis and bug fix. If there is no debug build of lshttpd binary in your current installation,​ you can download the related ​version and arch of debug build through ​the following command:  +Before you submit any such report to usit is always a good practice to check which release version and which BUILD caused ​the issue
-  /​usr/​local/​lsws/​admin/misc/lsup.sh -d -f -v <​your_lsws_version>​+  ​cat /​usr/​local/​lsws/​VERSION 
 +  cat /usr/​local/​lsws/BUILD
  
-===== How to enable a core dump =====+If it is an old version or build, please run the force upgrade command and check again. This action alone may fix your problem. 
 +  /​usr/​local/​lsws/​admin/​misc/​lsup.sh -f -v <​latest_version_number>​
  
-==== For PHP ==== +Monitor your server to see if the issue reoccursIf it doesplease install the debug build: 
-By default, core dumping is disabled in LSAPI PHP external applicationsTo enable ​it, you need to add an environment variable to your LSPHP external application configuration (WebAdmin console ​Configuration > External App > your external application > Environment):​+  /​usr/​local/​lsws/​admin/​misc/​lsup.sh -f -d -v <​latest_version_number>
  
-  LSAPI_ALLOW_CORE_DUMP=1+Monitor your server to see if the issue reoccurs. If it does, please take the following steps, as appropriate,​ to log a report.
  
-==== Server-wide ==== +===== Submit a Server ​Crash Report ===== 
-If you only need a particular program to be troubleshooted,​ you can use soft limits. +
-  ulimit -S -c +
-Edit ''/​etc/​security/​limits.conf''​ and add the following line entry at the bottom of the file (before # End of file)+
  
-''​* ​             ​soft ​   ​core            ​unlimited''​+A new version or new build of LiteSpeed Web Server may have an undiscovered bug, crash, and generate core files at ''​/​tmp/​lshttpd/​bak_core/​core.*''​. You should normally receive an email alert when this happens. You are welcome to submit a server crash report, either by forwarding the alert email or forwarding ​core files to ''​bug@litespeedtech.com''​.
  
-Check ulimit core file size value. It should be unlimited. +==== Email Alert With Debug Info ==== 
-  ulimit -c +You may receive an email alert similar to the following, if another crash happens again after a forced upgrade:
-  -> unlimited +
- +
-Core dumping is generally disabled by default on Linux. The following guide will lead you through the steps necessary to enable it: [[https://​linux-audit.com/​understand-and-configure-core-dumps-work-on-linux/​|How to enable core dump for Linux]] +
- +
-===== Install GDB and backtrace the core file ===== +
-Install ''​gdb''​ on your server if not installed yet, and use the syntax ''​gdb /​usr/​local/​lsws/​bin/​litespeed <​path/​to/​core/​file>''​ to check the debug infomation in core file. +
- +
-Once you have opened the core file with GDB, use the ''​bt''​ command to print a backtrace of the stack (the steps that the application took leading up to the crash). This will often reveal what caused the crash. ​  +
- +
-You can forward the above backtrace information + core files to bug@litespeedtech.com . +
- +
-===== Email Alert ===== +
-If you enable ​email alert on your server and gdb enabled, you should receive some alert email like the following. You can forward the email to bug@litespeedtech.com.+
  
 +<​code>​
 +  Web server example.com on example.com automatically restarted
 +  ​
   ##​---------- Forwarded message ----------##​   ##​---------- Forwarded message ----------##​
   From: root <​root@example.com>​   From: root <​root@example.com>​
Line 57: Line 41:
   unexpected signal=11, a core file has been created. A new instance of web   unexpected signal=11, a core file has been created. A new instance of web
   server will be started automatically!   server will be started automatically!
-  Please forward the following debug information to bug@litespeedtech.com.+  Please forward the following debug information to [[bug@litespeedtech.com]].
   Environment:​   Environment:​
   Server: LiteSpeed/​5.2.2 Enterprise Build 2   Server: LiteSpeed/​5.2.2 Enterprise Build 2
Line 238: Line 222:
   fs 0x0 0   fs 0x0 0
   gs 0x0 0   gs 0x0 0
-===== Where to send the bug report ===== +</​code>​ 
-Bug reports ​can be sent to our [[bug@litespeedtech.com|bug report email]].+ 
 +This means that LiteSpeed Web Server crashed, and core files have been created at ''/​tmp/​lshttpd/​bak_core''​.  
 +  ll /​tmp/​lshttpd/​bak_core 
 +  drwxr-xr-x 2 root   ​nobody ​    4096 May  3  2018 ./ 
 +  drwxr-x--x 4 nobody nobody ​   20480 Dec 11 11:10 ../ 
 +  -rw------- 1 nobody nobody 53587968 May  3  2018 core.2578440 
 + 
 +==== Email Alert Without Debug Info ==== 
 +You may receive an email alert similar ​to the following, without backtrace information:​ 
 + 
 +<​code>​ 
 +  At [10/​Dec/​2018:​07:​15:​19 +0000], web server with pid=4059 received unexpected signal=6, a core file has been created. A new instance of web server will be started automatically! 
 + 
 +  Please forward the following debug information to bug@litespeedtech.com. 
 +Environment:​ 
 + 
 +Server: LiteSpeed/​5.3.4 Enterprise Build 6 
 +OS: Linux 
 +Release: 3.10.0-514.21.1.el7.x86_64 
 +Version: #1 SMP Thu May 25 17:04:51 UTC 2017 
 +Machine: x86_64 
 + 
 +If the call stack information does not show up here, please compress and forward the core file located in /​tmp/​lshttpd/​.  
 + 
 +</​code>​ 
 + 
 +Please install ''​gdb''​ first, followed by a LSWS debug build: 
 +    yum install gdb 
 +    /​usr/​local/​lsws/​admin/​misc/​lsup.sh -d -f -v <​latest_version_number>​ 
 + 
 +Then, wait for the next crash. After that crash you will receive an email alert with the debug info and you can forward it to ''​bug@litespeedtech.com''​.  
 + 
 +===== Submit a Bug Report ===== 
 + 
 +If the server doesn'​t crash, but it's not working as expected, particularly if the same task //does// work in Apache, you can submit a bug report
 +Enable a high-level debug log on LSWS to reproduce the issue, and send the detailed debug log to ''​bug@litespeedtech.com''​. Be sure to include the detailed steps to reproduce the problem so that our developers may investigate. 
 + 
 +==== How to Toggle Debug Logging and Capture Output ​==== 
 + 
 +Let's give an example in a cPanel EA4 environment. One PHP URL/request works with Apache, but it doesn'​t work with LSWS. Go to the logs folder: 
 +  cd /​etc/​apache2/​logs 
 + 
 +Back up or delete old ''​error_log'':​ 
 +  mv error_log error_log.old 
 + 
 +Toggle LiteSpeed server debug log (run once to turn on, run again to turn off): 
 +  pkill -USR2 litespeed 
 + 
 +Reproduce the issue in your browser. 
 + 
 +Toggle the debug log off: 
 +  pkill -USR2 litespeed 
 + 
 +Move the debug log to ''​error_log.debug'':​ 
 +  mv error_log error_log.debug 
 + 
 +The ''​error_log.debug''​ file could be very big if it is a busy shared production server, since there could be many domains servicing requests at the same time. When forward the log file please compress it.  
 + 
 +If compressed log file is still too big, it might be good to just focus on the IP of your location. (You can easily find your own IP by searching ''​IP''​ at google.com.) 
 +  grep your_IP error_log.debug > error_log.debug.ip 
 + 
 +(Substitute your actual IP address where it says ''​your_IP''​.) ​ Some related debug log entries are not marked with your IP address, so we may miss some important information to diagnose the problem. The original log file should ​be kept just in case we need that. We recommend sending the full log.  
 + 
 +You can then forward ''​error_log.debug.ip''​ along with detailed steps to reproduce issue to ''​bug@litespeedtech.com''​ for investigation    ​
  
-We recommend that Enterprise customers use our ticket system (accessible from [[https://​store.litespeedtech.com/​store/​clientarea.php|the client area]]) for their bug reports. 
  • Admin
  • Last modified: 2017/11/01 15:24
  • by Jackson Zhang