Differences

This shows you the differences between two versions of the page.

--- litespeed_wiki:config:bug-reporting [2017/06/08 12:33]
Eric Leu [Server-wide]
+++ litespeed_wiki:config:bug-reporting [2020/01/03 17:52] (current)
George Wang [How to Toggle Debug Logging and Capture Output]
@@ Line 1: / Line 1: @@
-====== How to submit a bug report ======
+====== How to Submit a Bug Report ======
-To submit a useful bug report you need to provide as much information as you can. Including
-  * Your Server OS (Linux/FreeBSD/Solaris/MacOSX)
+**NOTE**: Always upgrade to the latest build of the latest version before submitting a report!
-  * CPU/Processor family (32bit/64bit)
-  * If you think LSWS does not work properly with a certain feature, you should provide as detailed information as possible on the feature and how you use it, so we can reproduce the problem in our lab.
-  * If lshttpd crashed, LSWS Enterprise users should switch to the debug build of lshttpd, then submit the debug report generated and a core file if possible.
-===== How to switch to the debug build of LSWS Enterprise  =====
+There are two cases where you can log a bug report:
-For 4.x version, there are two versions of the lshttpd binary shipped in the LSWS Enterprise package. One is the release build. The other is the debug build. The debug build is not normally used when you run LSWS, but, when debugging, the GDB stack backtrace and core file generated by the debug build are the most useful information you can provide for fixing the bug.
+  - Server Crash Report: Submit if the server crashes and core files were generated at ''/tmp/lshttpd/bak_core/''
+  - Bug Report: Submit if the server doesn't crash, but LSWS doesn't work as expected, especially when the same thing works with Apache in a control panel environment. Turn on debug logging when reproducing the issue, then send the detailed debug log for us to investigate.
-For latest version such as 5.0.x or 5.1.x, the binary installed is debug info enabled by default hence no separate debug build version required. It has no impact for your server performance.
+Before you submit any such report to us, it is always a good practice to check which release version and which BUILD caused the issue:
+  cat /usr/local/lsws/VERSION
+  cat /usr/local/lsws/BUILD
-===== How to download debug build of LSWS Enterprise  =====
+If it is an old version or build, please run the force upgrade command and check again. This action alone may fix your problem.
-For 4.x version, if there is no debug build of lshttpd binary in your current installation, you can download the related version and arch of debug build from our website. For example:
+  /usr/local/lsws/admin/misc/lsup.sh -f -v <latest_version_number>
-  /usr/local/lsws/admin/misc/lsup.sh -d -f -v <your_lsws_version>
+Monitor your server to see if the issue reoccurs. If it does, please install the debug build:
+  /usr/local/lsws/admin/misc/lsup.sh -f -d -v <latest_version_number>
-Once you have debug version installed, reproduce your problem and submit a bug report.
+Monitor your server to see if the issue reoccurs. If it does, please take the following steps, as appropriate, to log a report.
-For 5.0.x, 5.1.x, you do not need to switch to debug build anymore since the binary installed is debug info enabled by default. It has no impact for your server performance.
+===== Submit a Server Crash Report =====
-===== How to enable a core dump =====
+A new version or new build of LiteSpeed Web Server may have an undiscovered bug, crash, and generate core files at ''/tmp/lshttpd/bak_core/core.*''. You should normally receive an email alert when this happens. You are welcome to submit a server crash report, either by forwarding the alert email or forwarding core files to ''bug@litespeedtech.com''.
-==== For PHP ====
+==== Email Alert With Debug Info ====
-By default, core dumping is disabled in LSAPI PHP external applications. To enable it, you need to add an environment variable to your LSPHP external application configuration (WebAdmin console > Configuration > External App > your external application > Environment):
+You may receive an email alert similar to the following, if another crash happens again after a forced upgrade:
-  LSAPI_ALLOW_CORE_DUMP=1
+<code>
+  Web server example.com on example.com automatically restarted
+  ##---------- Forwarded message ----------##
+  From: root <root@example.com>
+  To: root@localhost
+  Cc:
+  Bcc:
+  Date: Wed, 01 Nov 2017 01:00:27 -0500
+  Subject: Web server example.com on example.com
+  automatically restarted
+  At [01/Nov/2017:01:00:25 -0500], web server with pid=12964 received
+  unexpected signal=11, a core file has been created. A new instance of web
+  server will be started automatically!
+  Please forward the following debug information to [[bug@litespeedtech.com]].
+  Environment:
+  Server: LiteSpeed/5.2.2 Enterprise Build 2
+  OS: Linux
+  Release: 3.10.0-693.5.2.el7.x86_64
+  Version: #1 SMP Fri Oct 20 20:32:50 UTC 2017
+  Machine: x86_64
+  If the call stack information does not show up here, please compress and
+  forward the core file located in /tmp/lshttpd/.
+  [New LWP 12964]
+  [New LWP 12971]
+  [New LWP 12972]
+  [Thread debugging using libthread_db enabled]
+  Using host libthread_db library "/lib64/libthread_db.so.1".
+  Core was generated by `litespeed'.
+  Program terminated with signal 11, Segmentation fault.
+  #0 0x0000000000607440 in lsShm_hElem_s::getLruLinkPtr (this=0x0) at
+  /home/gwang/release/litespeed/src/shm/lsshmhash.h:102
+  #0 0x0000000000607440 in lsShm_hElem_s::getLruLinkPtr (this=0x0) at
+  /home/gwang/release/litespeed/src/shm/lsshmhash.h:102
+  No locals.
+  #1 0x00000000006a6553 in lsShm_hElem_s::setLruLinkNext (this=0x0, off=...)
+  at /home/gwang/release/litespeed/src/shm/lsshmhash.h:110
+  No locals.
+  #2 0x00000000006a681c in LsShmHash::set_linkNext (this=0x144a880,
+  offThis=..., offNext=...) at
+  /home/gwang/release/litespeed/src/shm/lsshmhash.h:686
+  No locals.
+  #3 0x00000000006a5953 in LsShmHash::linkSetTop (this=0x144a880,
+  pElem=0x7f7d53dc4b80, offElem=...) at
+  /home/gwang/release/litespeed/src/shm/lsshmhash.cpp:1581
+  pLru = 0x7f7d53d00a1c
+  prev = {m_iOffset = 610112}
+  __PRETTY_FUNCTION__ = "void LsShmHash::linkSetTop(LsShmHElem*,
+  LsShmHash::iteroffset)"
+  pLink = 0x7f7d53dc4b9c
+  next = {m_iOffset = 12479344}
+  #4 0x00000000006a34c1 in LsShmHash::doGet (this=0x144a880, iterOff=...,
+  key=4039776365, pParms=0x7ffd84259560, pFlag=0x7ffd84259614) at
+  /home/gwang/release/litespeed/src/shm/lsshmhash.cpp:809
+  iter = 0x7f7d53dc4b80
+  #5 0x00000000006a4a9a in LsShmHash::getPtr (pThis=0x144a880,
+  pParms=0x7ffd84259560, pFlag=0x7ffd84259614) at
+  /home/gwang/release/litespeed/src/shm/lsshmhash.cpp:1206
+  key = 4039776365
+  iterOff = {m_iOffset = 805760}
+  #6 0x000000000069f9fb in LsShmHash::getIterator (this=0x144a880,
+  pParms=0x7ffd84259560, pFlag=0x7ffd84259614) at
+  /home/gwang/release/litespeed/src/shm/lsshmhash.h:401
+  iterOff = {m_iOffset = 44971056}
+  #7 0x000000000069f8fd in LsShmHash::get (this=0x144a880, pKey=0x2ae3430,
+  keyLen=8, valLen=0x7ffd84259618, pFlag=0x7ffd84259614) at
+  /home/gwang/release/litespeed/src/shm/lsshmhash.h:323
+  iterOff = {m_iOffset = 2217055712}
+  parms = {key = {ptr = 0x2ae3430 "\357\t\304\330\326\373e8", len =
+}, val = {ptr = 0x0, len = 12}}
+  iter = 0x2ae3430
+  #8 0x000000000069f2bb in ShmCacheManager::addTracking2 (this=0x15024a0,
+  pEntry=0x2ae3410, pTracker=0x144a880) at
+  /home/gwang/release/litespeed/src/cache/shmcachemanager.cpp:1141
+  pData = 0x943b51
+  valLen = 12
+  flag = 0
+  offVal = 0
+  #9 0x000000000069f248 in ShmCacheManager::addTracking (this=0x15024a0,
+  pEntry=0x2ae3410) at
+  /home/gwang/release/litespeed/src/cache/shmcachemanager.cpp:1128
+  pTracker = 0x144a880
+  #10 0x000000000055baef in HttpSession::preparePageCache (this=0x30f9ea0) at
+  /home/gwang/release/litespeed/src/http/httpsession.cpp:6463
+  pEntry = 0xfffffffffffffffe
+  pIP = 0x0
+  ipLen = 0
+  age = 604800
+  priv_cache = 0
+  pResp = 0x30fa358
+  pCtx = 0x149afd0
+  #11 0x0000000000555d79 in HttpSession::setupDynRespBody (this=0x30f9ea0,
+  prepareCache=1) at
+  /home/gwang/release/litespeed/src/http/httpsession.cpp:4439
+  No locals.
+  #12 0x0000000000556641 in HttpSession::appendDynBody (this=0x30f9ea0,
+  pBuf=0xd7b420 <HttpGlobals::g_achBuf> "<!DOCTYPE html>\n<html
+  lang=\"en-US\" prefix=\"og: http://ogp.me/ns# fb:
+  http://ogp.me/ns/fb#\">\n<head
+  >\n<meta charset=\"UTF-8\" />\n<title>My Bookmark -
+  MyReadingManga</title><meta name=\"viewport\" content=\"w"..., len=16384)
+  at /home/gwang/release/litespeed/src/http/httpsession.cpp:4631
+  buffered = 32765
+  ret = 0
+  #13 0x000000000056eec8 in HttpExtConnector::processRespBodyData
+  (this=0x14a4800, pBuf=0xd7b420 <HttpGlobals::g_achBuf> "<!DOCTYPE
+  html>\n<html lang=\"en-US\" prefix=\"og: http://ogp.me/ns# fb:
+  http://ogp.me/ns/fb#\">\n<head >\n<meta charset=\"UTF-8\" />\n<title>My
+  Bookmark - MyReadingManga</title><meta name=\"viewport\" content=\"w"...,
+  len=16384) at
+  /home/gwang/release/litespeed/src/http/httpextconnector.cpp:371
+  ret = 16384
+  #14 0x000000000066d7f7 in LsapiConn::readRespBody (this=0x314a9e0) at
+  /home/gwang/release/litespeed/src/extensions/lsapi/lsapiconn.cpp:943
+  len = 0
+  packetLen = 16384
+  pBuf = 0xd7b420 <HttpGlobals::g_achBuf> "<!DOCTYPE html>\n<html
+  lang=\"en-US\" prefix=\"og: http://ogp.me/ns# fb:
+  http://ogp.me/ns/fb#\">\n<head
+  >\n<meta charset=\"UTF-8\" />\n<title>My Bookmark -
+  MyReadingManga</title><meta name=\"viewport\" content=\"w"...
+  toRead = 16384
+  pHEC = 0x14a4800
+  ret = 16384
+  bufLen = 16384
+  respState = @0x14a4874: 526339
+  total = 16384
+  #15 0x000000000066cb13 in LsapiConn::processResp (this=0x314a9e0) at
+  /home/gwang/release/litespeed/src/extensions/lsapi/lsapiconn.cpp:683
+  ret = 8
+  count = 2
+  #16 0x000000000066c2c3 in LsapiConn::doRead (this=0x314a9e0) at
+  /home/gwang/release/litespeed/src/extensions/lsapi/lsapiconn.cpp:361
+  ret = 0
+  #17 0x000000000067e727 in ExtConn::onRead (this=0x314a9e0) at
+  /home/gwang/release/litespeed/src/extensions/extconn.cpp:344
+  ret = 0
+  #18 0x000000000068720f in EdStream::handleEvents (this=0x314a9e0, event=1)
+  at /home/gwang/release/litespeed/src/edio/ediostream.cpp:70
+  ret = 0
+  #19 0x000000000068b6e2 in epoll::processEvents (this=0x135fa80, events=1)
+  at /home/gwang/release/litespeed/src/edio/epoll.cpp:371
+  fd = 76
+  pReactor = 0x314a9e0
+  p = 0x1370130
+  __PRETTY_FUNCTION__ = "virtual int epoll::processEvents(int)"
+  #20 0x000000000068b5c5 in epoll::waitAndProcessEvents (this=0x135fa80,
+  iTimeoutMilliSec=100) at
+  /home/gwang/release/litespeed/src/edio/epoll.cpp:344
+  ret = 1
+  #21 0x000000000054192d in EventDispatcher::run (this=0x1345c78) at
+  /home/gwang/release/litespeed/src/http/eventdispatcher.cpp:244
+  ret = 1
+  evt_waiting = 0
+  sigEvent = 0
+  errors = 0
+  pQuicEngine = 0x13a3a50
+  #22 0x00000000004ef55b in HttpServerImpl::start (this=0x1345c50) at
+  /home/gwang/release/litespeed/src/main/httpserver.cpp:529
+  No locals.
+  #23 0x00000000004f2bfa in HttpServer::start (this=0xd752d0
+  <TSingleton<HttpServer>::getInstance()::s_instance>) at
+  /home/gwang/release/litespeed/src/main/httpserver.cpp:2031
+  No locals.
+  #24 0x00000000004e83d1 in LshttpdMain::main (this=0x1345870, argc=1,
+  argv=0x7ffd84259ca8) at
+  /home/gwang/release/litespeed/src/main/lshttpdmain.cpp:1950
+  ret = 0
+  #25 0x00000000004e36e6 in main (argc=1, argv=0x7ffd84259ca8) at
+  /home/gwang/release/litespeed/src/main.cpp:140
+  ret = 0
+  rax 0x0 0
+  rbx 0x1345870 20207728
+  rcx 0x7f7d53d00a1c 140176253782556
+  rdx 0x10 16
+  rsi 0xc4b80 805760
+  rdi 0x0 0
+  rbp 0x7ffd84259bc0 0x7ffd84259bc0
+  rsp 0x7ffd84259b90 0x7ffd84259b90
+  r8 0x7ffd84259614 140726820509204
+  r9 0x7f7d59fca340 140176357368640
+  r10 0x4 4
+  r11 0x7f7d5a0032d0 140176357602000
+  r12 0x4e3578 5125496
+  r13 0x7ffd84259ca0 140726820510880
+  r14 0x0 0
+  r15 0x0 0
+  rip 0x4e36e6 0x4e36e6 <main(int, char**)+102>
+  eflags 0x10206 [ PF IF RF ]
+  cs 0x33 51
+  ss 0x2b 43
+  ds 0x0 0
+  es 0x0 0
+  fs 0x0 0
+  gs 0x0 0
+</code>
-==== Server-wide ====
+This means that LiteSpeed Web Server crashed, and core files have been created at ''/tmp/lshttpd/bak_core''.
-If you only need a particular program to be troubleshooted, you can use soft limits.
+  ll /tmp/lshttpd/bak_core
-  ulimit -S -c
+  drwxr-xr-x 2 root   nobody     4096 May  3  2018 ./
-Edit /etc/security/limits.conf and add the following line entry at the bottom of the file (before # End of file)
+  drwxr-x--x 4 nobody nobody    20480 Dec 11 11:10 ../
-''*              soft    core            unlimited''
+  -rw------- 1 nobody nobody 53587968 May  3  2018 core.2578440
-Check ulimit core file size value. It should be unlimited.
+==== Email Alert Without Debug Info ====
-  ulimit -c
+You may receive an email alert similar to the following, without backtrace information:
-  -> unlimited
-Core dumping is generally disabled by default on Linux. The following guide will lead you through the steps necessary to enable it: [[https://linux-audit.com/understand-and-configure-core-dumps-work-on-linux/|How to enable core dump for Linux]]
+<code>
+  At [10/Dec/2018:07:15:19 +0000], web server with pid=4059 received unexpected signal=6, a core file has been created. A new instance of web server will be started automatically!
+  Please forward the following debug information to bug@litespeedtech.com.
+Environment:
-===== Where to send the bug report =====
+Server: LiteSpeed/5.3.4 Enterprise Build 6
-Bug reports can be sent to our [[bug@litespeedtech.com|bug report email]].
+OS: Linux
+Release: 3.10.0-514.21.1.el7.x86_64
+Version: #1 SMP Thu May 25 17:04:51 UTC 2017
+Machine: x86_64
+If the call stack information does not show up here, please compress and forward the core file located in /tmp/lshttpd/.
+</code>
+Please install ''gdb'' first, followed by a LSWS debug build:
+    yum install gdb
+    /usr/local/lsws/admin/misc/lsup.sh -d -f -v <latest_version_number>
+Then, wait for the next crash. After that crash you will receive an email alert with the debug info and you can forward it to ''bug@litespeedtech.com''.
+===== Submit a Bug Report =====
+If the server doesn't crash, but it's not working as expected, particularly if the same task //does// work in Apache, you can submit a bug report.
+Enable a high-level debug log on LSWS to reproduce the issue, and send the detailed debug log to ''bug@litespeedtech.com''. Be sure to include the detailed steps to reproduce the problem so that our developers may investigate.
+==== How to Toggle Debug Logging and Capture Output ====
+Let's give an example in a cPanel EA4 environment. One PHP URL/request works with Apache, but it doesn't work with LSWS. Go to the logs folder:
+  cd /etc/apache2/logs
+Back up or delete old ''error_log'':
+  mv error_log error_log.old
+Toggle LiteSpeed server debug log (run once to turn on, run again to turn off):
+  pkill -USR2 litespeed
+Reproduce the issue in your browser.
+Toggle the debug log off:
+  pkill -USR2 litespeed
+Move the debug log to ''error_log.debug'':
+  mv error_log error_log.debug
+The ''error_log.debug'' file could be very big if it is a busy shared production server, since there could be many domains servicing requests at the same time. When forward the log file please compress it.
+If compressed log file is still too big, it might be good to just focus on the IP of your location. (You can easily find your own IP by searching ''IP'' at google.com.)
+  grep your_IP error_log.debug > error_log.debug.ip
+(Substitute your actual IP address where it says ''your_IP''.)  Some related debug log entries are not marked with your IP address, so we may miss some important information to diagnose the problem. The original log file should be kept just in case we need that. We recommend sending the full log.
+You can then forward ''error_log.debug.ip'' along with detailed steps to reproduce issue to ''bug@litespeedtech.com'' for investigation.
-We recommend that Enterprise customers use our ticket system (accessible from [[https://store.litespeedtech.com/store/clientarea.php|the client area]]) for their bug reports.