Differences

This shows you the differences between two versions of the page.

--- litespeed_wiki:config:bug-reporting [2018/12/11 16:29]
Jackson Zhang [How to submit a bug report]
+++ litespeed_wiki:config:bug-reporting [2020/01/03 17:52] (current)
George Wang [How to Toggle Debug Logging and Capture Output]
@@ Line 1: / Line 1: @@
-====== How to submit a bug report ======
+====== How to Submit a Bug Report ======
-To submit a useful bug report you need to provide as much information as you can, including:
-  * Your Server OS (Linux/FreeBSD/Solaris/MacOSX)
+**NOTE**: Always upgrade to the latest build of the latest version before submitting a report!
-  * CPU/Processor family (32bit/64bit)
-  * If you think LSWS does not work properly with a certain feature, you should provide as detailed information as possible on the feature and how you use it, so we can reproduce the problem in our lab.
-  * If lshttpd crashed, LSWS Enterprise users should switch to the debug build of lshttpd, then submit the debug report generated and a core file if possible.
-  * GDB backtrace information on core file.
-If you receive some email alert on the crash, you can forward that email to us. If not, you can use ''gdb'' to backtrace the crash and send to us for further investigation. If the error log shows the server crashed but no core file was generated, you may need to enable a core dump for your system and reboot your server.
+There are two cases where you can log a bug report:
-===== LSWS crash report =====
+  - Server Crash Report: Submit if the server crashes and core files were generated at ''/tmp/lshttpd/bak_core/''
+  - Bug Report: Submit if the server doesn't crash, but LSWS doesn't work as expected, especially when the same thing works with Apache in a control panel environment. Turn on debug logging when reproducing the issue, then send the detailed debug log for us to investigate.
-When you receive some email alert similar to the following:
+Before you submit any such report to us, it is always a good practice to check which release version and which BUILD caused the issue:
+  cat /usr/local/lsws/VERSION
+  cat /usr/local/lsws/BUILD
-<code>
+If it is an old version or build, please run the force upgrade command and check again. This action alone may fix your problem.
-  Web server example.com on example.com automatically restarted
+  /usr/local/lsws/admin/misc/lsup.sh -f -v <latest_version_number>
-  --------- Forwarded message ---------
-  From: root <root@abc.example.com>
-  Date: seg, 10 de dez de 2018 às 12:38
-  Subject: Web server example.com on example.com automatically restarted
-  To: <root@localhost>
-  At [10/Dec/2018:12:38:26 -0200], web server with pid=12264 received unexpected signal=11, a core file has been created. A new instance of web server will be started automatically!
-  Please forward the following debug information to bug@litespeedtech.com.
+Monitor your server to see if the issue reoccurs. If it does, please install the debug build:
-  Environment:
+  /usr/local/lsws/admin/misc/lsup.sh -f -d -v <latest_version_number>
-  Server: LiteSpeed/5.3.4 Enterprise Build 4
+Monitor your server to see if the issue reoccurs. If it does, please take the following steps, as appropriate, to log a report.
-  OS: Linux
-  Release: 3.10.0-862.14.4.el7.x86_64
-  Version: #1 SMP Wed Sep 26 15:12:11 UTC 2018
-  Machine: x86_64
-  If the call stack information does not show up here, please compress and
+===== Submit a Server Crash Report =====
-  forward the core file located in /tmp/lshttpd/.
-  [New LWP 12264]
+A new version or new build of LiteSpeed Web Server may have an undiscovered bug, crash, and generate core files at ''/tmp/lshttpd/bak_core/core.*''. You should normally receive an email alert when this happens. You are welcome to submit a server crash report, either by forwarding the alert email or forwarding core files to ''bug@litespeedtech.com''.
-  [New LWP 12265]
-  [New LWP 12266]
-  Core was generated by `litespeed'.
-  Program terminated with signal 11, Segmentation fault.
-  #0 0x0000000000557bc2 in ?? ()
-  #0 0x0000000000557bc2 in ?? ()
-  No symbol table info available.
-  #1 0x00000000042763e0 in ?? ()
-  No symbol table info available.
-  #2 0x00000000042763e0 in ?? ()
-  No symbol table info available.
-  #3 0x00000000042e0110 in ?? ()
-  No symbol table info available.
-  #4 0x0000000000557ccb in ?? ()
-  No symbol table info available.
-  #5 0x000000000420c988 in ?? ()
-  No symbol table info available.
-  #6 0x000000000404b000 in ?? ()
-  No symbol table info available.
-  #7 0x00000000042e0110 in ?? ()
-  No symbol table info available.
-  #8 0x0000000000000008 in ?? ()
-  No symbol table info available.
-  #9 0x00000000042e0d98 in ?? ()
-  No symbol table info available.
-  #10 0x000000000055a0e3 in ?? ()
-  No symbol table info available.
-  #11 0x00000000042e0110 in ?? ()
-  No symbol table info available.
-  #12 0x00000000042e0158 in ?? ()
-  No symbol table info available.
-  #13 0x0000000000000000 in ?? ()
-  No symbol table info available.
-  rax 0x4023200 67252736
-  rbx 0x42763e0 69690336
-  rcx 0x42763d0 69690320
-  rdx 0x0 0
-  rsi 0x404b000 67416064
-  rdi 0x42763e0 69690336
-  rbp 0x42e0110 0x42e0110
-  rsp 0x7ffcc447b170 0x7ffcc447b170
-  r8 0x1 1
-  r9 0x404aff0 67416048
-  r10 0x0 0
-  r11 0x0 0
-  r12 0x404b000 67416064
-  r13 0x3 3
-  r14 0x64 100
-  r15 0x0 0
-  rip 0x557bc2 0x557bc2
-  eflags 0x10246 [ PF ZF IF RF ]
-  cs 0x33 51
-  ss 0x2b 43
-  ds 0x0 0
-  es 0x0 0
-  fs 0x0 0
-  gs 0x0 0
-</code>
-it means LSWS server crashed and a core file has been created at /tmp/lshttpd/bak_core
+==== Email Alert With Debug Info ====
-  ll /tmp/lshttpd/bak_core
+You may receive an email alert similar to the following, if another crash happens again after a forced upgrade:
-  drwxr-xr-x 2 root   nobody     4096 May  3  2018 ./
-  drwxr-x--x 4 nobody nobody    20480 Dec 11 11:10 ../
-  -rw------- 1 nobody nobody 53587968 May  3  2018 core.2578440
-Before you log a bug report to us, please check your version and build number and run force upgrade command to the latest build of lastest version and it may have fixed your problem already.
-  /usr/local/lsws/admin/misc/lsup.sh -f -v <latest_version_number>
-If you receive an email similar to the following email without backtrace information, please install gdb firstly.
 <code>
-  At [10/Dec/2018:07:15:19 +0000], web server with pid=4059 received unexpected signal=6, a core file has been created. A new instance of web server will be started automatically!
+  Web server example.com on example.com automatically restarted
-  Please forward the following debug information to bug@litespeedtech.com.
-Environment:
-Server: LiteSpeed/5.3.4 Enterprise Build 6
-OS: Linux
-Release: 3.10.0-514.21.1.el7.x86_64
-Version: #1 SMP Thu May 25 17:04:51 UTC 2017
-Machine: x86_64
-If the call stack information does not show up here, please compress and forward the core file located in /tmp/lshttpd/.
-</code>
-If you have installed gdb and also LSWS crashed on the latest build of the latest version, please switch to debug build:
-  /usr/local/lsws/admin/misc/lsup.sh -d -f -v <latest_version_number>
-then wait for the next crash to happen and log a bug report ticket by providiong tmp root ssh login for our developer to take a further analysis.
-===== Debug build of LSWS Enterprise  =====
-Starting from version 5.0.x, you do not need a separate debug build since the production LSWS binary installed has debug info enabled by default. It has no impact on your server performance.
-For versions earlier than 5.0, there are two versions of the lshttpd binary shipped in the LSWS Enterprise package. One is the release build. The other is the debug build. The debug build is not normally used when you run LSWS, but, when debugging, the GDB stack backtrace and core file generated by the debug build will be used for bug analysis and bug fixes. If there is no debug build of lshttpd binary in your current installation, you can download the related version and arch of the debug build through the following command:
-  /usr/local/lsws/admin/misc/lsup.sh -d -f -v <your_lsws_version>
-===== How to enable a core dump =====
-==== For PHP ====
-By default, core dumping is disabled in LSAPI PHP external applications. To enable it, you need to add an environment variable to your LSPHP external application configuration (**WebAdmin console > Configuration > External App > your external application > Environment**):
-  LSAPI_ALLOW_CORE_DUMP=1
-==== Server-wide ====
-If you only need a particular program to be troubleshooted, you can use soft limits:
-  ulimit -S -c
-Edit ''/etc/security/limits.conf'' and add the following line at the bottom of the file (before ''# End of file'')
-<code>*              soft    core            unlimited</code>
-Check ulimit core file size value. It should be unlimited.
-  ulimit -c
-  -> unlimited
-Core dumping is generally disabled by default on Linux. The following guide will lead you through the steps necessary to enable it: [[https://linux-audit.com/understand-and-configure-core-dumps-work-on-linux/|How to enable core dump for Linux]]
-===== Install GDB and backtrace the core file =====
-Install GDB on your server if not installed yet, and use the syntax ''gdb /usr/local/lsws/bin/litespeed <path/to/core/file>'' to check the debug infomation in core file.
-Once you have opened the core file with GDB, use the ''bt'' command to print a backtrace of the stack (the steps that the application took leading up to the crash). This will often reveal what caused the crash.
-You can forward the above backtrace information + core files to [[bug@litespeedtech.com]] .
-===== Email Alert =====
-If you enable email alert on your server and GDB is enabled, you should receive some alert email like the following. You can forward the email to [[bug@litespeedtech.com]].
   ##---------- Forwarded message ----------##
   From: root <root@example.com>
@@ Line 358: / Line 222: @@
   fs 0x0 0
   gs 0x0 0
-===== Where to send the bug report =====
+</code>
-Bug reports can be sent to our [[bug@litespeedtech.com|bug report email]].
+This means that LiteSpeed Web Server crashed, and core files have been created at ''/tmp/lshttpd/bak_core''.
+  ll /tmp/lshttpd/bak_core
+  drwxr-xr-x 2 root   nobody     4096 May  3  2018 ./
+  drwxr-x--x 4 nobody nobody    20480 Dec 11 11:10 ../
+  -rw------- 1 nobody nobody 53587968 May  3  2018 core.2578440
+==== Email Alert Without Debug Info ====
+You may receive an email alert similar to the following, without backtrace information:
+<code>
+  At [10/Dec/2018:07:15:19 +0000], web server with pid=4059 received unexpected signal=6, a core file has been created. A new instance of web server will be started automatically!
+  Please forward the following debug information to bug@litespeedtech.com.
+Environment:
+Server: LiteSpeed/5.3.4 Enterprise Build 6
+OS: Linux
+Release: 3.10.0-514.21.1.el7.x86_64
+Version: #1 SMP Thu May 25 17:04:51 UTC 2017
+Machine: x86_64
+If the call stack information does not show up here, please compress and forward the core file located in /tmp/lshttpd/.
+</code>
+Please install ''gdb'' first, followed by a LSWS debug build:
+    yum install gdb
+    /usr/local/lsws/admin/misc/lsup.sh -d -f -v <latest_version_number>
+Then, wait for the next crash. After that crash you will receive an email alert with the debug info and you can forward it to ''bug@litespeedtech.com''.
+===== Submit a Bug Report =====
+If the server doesn't crash, but it's not working as expected, particularly if the same task //does// work in Apache, you can submit a bug report.
+Enable a high-level debug log on LSWS to reproduce the issue, and send the detailed debug log to ''bug@litespeedtech.com''. Be sure to include the detailed steps to reproduce the problem so that our developers may investigate.
+==== How to Toggle Debug Logging and Capture Output ====
+Let's give an example in a cPanel EA4 environment. One PHP URL/request works with Apache, but it doesn't work with LSWS. Go to the logs folder:
+  cd /etc/apache2/logs
+Back up or delete old ''error_log'':
+  mv error_log error_log.old
+Toggle LiteSpeed server debug log (run once to turn on, run again to turn off):
+  pkill -USR2 litespeed
+Reproduce the issue in your browser.
+Toggle the debug log off:
+  pkill -USR2 litespeed
+Move the debug log to ''error_log.debug'':
+  mv error_log error_log.debug
+The ''error_log.debug'' file could be very big if it is a busy shared production server, since there could be many domains servicing requests at the same time. When forward the log file please compress it.
+If compressed log file is still too big, it might be good to just focus on the IP of your location. (You can easily find your own IP by searching ''IP'' at google.com.)
+  grep your_IP error_log.debug > error_log.debug.ip
+(Substitute your actual IP address where it says ''your_IP''.)  Some related debug log entries are not marked with your IP address, so we may miss some important information to diagnose the problem. The original log file should be kept just in case we need that. We recommend sending the full log.
+You can then forward ''error_log.debug.ip'' along with detailed steps to reproduce issue to ''bug@litespeedtech.com'' for investigation.
-We recommend that Enterprise customers use our ticket system (accessible from [[https://store.litespeedtech.com/store/clientarea.php|the client area]]) for their bug reports.