LS Dying - Various Errors

XN-Matt · Apr 14, 2017

In the recent versions, we're seeing LS die completely with varying errors.
Last log entry is immediately before it stopped responding.

Code:

[Fri Apr 14 04:59:28.747400 2017] [:notice] [pid 173875:tid 139935212316640] [host xxxxxx] mod_lsapi:  Selfstarter 173875 started
[Fri Apr 14 04:59:57.623652 2017] [core:error] [pid 173967:tid 139934832768768] (104)Connection reset by peer: [client 194.176.105.144:32623] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 05:12:06.205630 2017] [access_compat:error] [pid 173880:tid 139934974887680] [client 78.92.180.101:62434] AH01797: client denied by server configuration: /home/xx/public_html/xx
[Fri Apr 14 05:13:21.801415 2017] [cgid:error] [pid 173880:tid 139934966494976] (70014)End of file found: [client 66.240.205.34:38584] AH01270: Error reading request entity data
[Fri Apr 14 05:13:23.852656 2017] [cgid:error] [pid 173880:tid 139934815983360] (70014)End of file found: [client 66.240.205.34:53353] AH01270: Error reading request entity data
[Fri Apr 14 05:13:25.859728 2017] [cgid:error] [pid 173877:tid 139934966494976] (70014)End of file found: [client 66.240.205.34:60888] AH01270: Error reading request entity data
[Fri Apr 14 05:13:26.499399 2017] [cgid:error] [pid 173880:tid 139934841161472] (70014)End of file found: [client 66.240.205.34:47231] AH01270: Error reading request entity data
[Fri Apr 14 05:13:27.762036 2017] [cgid:error] [pid 173877:tid 139934991673088] (70014)End of file found: [client 66.240.205.34:55030] AH01270: Error reading request entity data
[Fri Apr 14 05:13:35.809388 2017] [cgid:error] [pid 173966:tid 139934807590656] (70014)End of file found: [client 66.240.205.34:53731] AH01270: Error reading request entity data
[Fri Apr 14 05:13:52.545084 2017] [cgid:error] [pid 173967:tid 139934757234432] (70014)End of file found: [client 89.248.167.131:33444] AH01270: Error reading request entity data
[Fri Apr 14 05:13:54.765470 2017] [cgid:error] [pid 173877:tid 139934740449024] (70014)End of file found: [client 89.248.167.131:34001] AH01270: Error reading request entity data
[Fri Apr 14 05:13:54.881406 2017] [cgid:error] [pid 173876:tid 139934849554176] (70014)End of file found: [client 89.248.167.131:34053] AH01270: Error reading request entity data
[Fri Apr 14 05:13:55.105929 2017] [cgid:error] [pid 173966:tid 139934958102272] (70014)End of file found: [client 89.248.167.131:34100] AH01270: Error reading request entity data
[Fri Apr 14 05:13:55.505998 2017] [cgid:error] [pid 173880:tid 139934748841728] (70014)End of file found: [client 89.248.167.131:34276] AH01270: Error reading request entity data
[Fri Apr 14 05:14:04.029373 2017] [cgid:error] [pid 173880:tid 139935000065792] (70014)End of file found: [client 89.248.167.131:34753] AH01270: Error reading request entity data
[Fri Apr 14 05:15:20.739866 2017] [access_compat:error] [pid 173966:tid 139934849554176] [client 89.184.65.130:43462] AH01797: client denied by server configuration: /home/xx/public_html/xx
[Fri Apr 14 05:42:59.193527 2017] [cgid:error] [pid 173877:tid 139934849554176] (70014)End of file found: [client 71.6.165.200:51313] AH01270: Error reading request entity data
[Fri Apr 14 05:43:00.284755 2017] [cgid:error] [pid 173880:tid 139934765627136] (70014)End of file found: [client 71.6.165.200:51592] AH01270: Error reading request entity data
[Fri Apr 14 05:43:00.777708 2017] [cgid:error] [pid 173967:tid 139934983280384] (70014)End of file found: [client 71.6.165.200:51648] AH01270: Error reading request entity data
[Fri Apr 14 05:43:01.256321 2017] [cgid:error] [pid 173877:tid 139934983280384] (70014)End of file found: [client 71.6.165.200:51804] AH01270: Error reading request entity data
[Fri Apr 14 05:43:01.716286 2017] [cgid:error] [pid 173966:tid 139934774019840] (70014)End of file found: [client 71.6.165.200:51857] AH01270: Error reading request entity data
[Fri Apr 14 05:43:08.265280 2017] [cgid:error] [pid 173876:tid 139934949709568] (70014)End of file found: [client 71.6.165.200:52140] AH01270: Error reading request entity data
[Fri Apr 14 05:49:42.079086 2017] [core:error] [pid 173877:tid 139934974887680] (104)Connection reset by peer: [client 87.81.249.226:53496] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 05:49:49.169136 2017] [core:error] [pid 173966:tid 139934765627136] (104)Connection reset by peer: [client 87.81.249.226:53515] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 05:49:56.066341 2017] [core:error] [pid 173876:tid 139934857946880] (104)Connection reset by peer: [client 87.81.249.226:53530] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 05:50:03.026677 2017] [core:error] [pid 173877:tid 139934824376064] (104)Connection reset by peer: [client 87.81.249.226:53544] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 05:52:09.800776 2017] [core:error] [pid 173966:tid 139934774019840] (104)Connection reset by peer: [client 151.226.124.247:49463] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 05:53:52.629831 2017] [access_compat:error] [pid 173967:tid 139934832768768] [client 202.46.54.17:28918] AH01797: client denied by server configuration: /home/xx/public_html/xx/wp-includes/class-wp-post-type.php
[Fri Apr 14 05:53:53.709054 2017] [access_compat:error] [pid 173876:tid 139934832768768] [client 202.46.57.209:29522] AH01797: client denied by server configuration: /home/xx/public_html/xx/wp-includes/class-wp-post-type.php
[Fri Apr 14 05:55:07.283568 2017] [cgid:error] [pid 173873:tid 139935212316640] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: autodiscover.cgi
[Fri Apr 14 05:55:07.283696 2017] [cgid:error] [pid 173967:tid 139934748841728] (32)Broken pipe: [client 151.226.124.247:49528] AH02651: Error writing request body to script /usr/local/cpanel/cgi-sys/autodiscover.cgi
[Fri Apr 14 05:55:07.283735 2017] [cgid:error] [pid 173967:tid 139934748841728] [client 151.226.124.247:49528] End of script output before headers: autodiscover.cgi
[Fri Apr 14 05:55:07.284461 2017] [cgid:error] [pid 173873:tid 139935212316640] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: autodiscover.cgi
[Fri Apr 14 05:55:07.284517 2017] [cgid:error] [pid 173876:tid 139934807590656] (32)Broken pipe: [client 151.226.124.247:49529] AH02651: Error writing request body to script /usr/local/cpanel/cgi-sys/autodiscover.cgi
[Fri Apr 14 05:55:07.284551 2017] [cgid:error] [pid 173876:tid 139934807590656] [client 151.226.124.247:49529] End of script output before headers: autodiscover.cgi
[Fri Apr 14 05:55:07.285290 2017] [cgid:error] [pid 173873:tid 139935212316640] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: autodiscover.cgi
[Fri Apr 14 05:55:07.285348 2017] [cgid:error] [pid 173880:tid 139934732056320] (32)Broken pipe: [client 151.226.124.247:49530] AH02651: Error writing request body to script /usr/local/cpanel/cgi-sys/autodiscover.cgi
[Fri Apr 14 05:55:07.285387 2017] [cgid:error] [pid 173880:tid 139934732056320] [client 151.226.124.247:49530] End of script output before headers: autodiscover.cgi
[Fri Apr 14 05:55:07.286169 2017] [cgid:error] [pid 173873:tid 139935212316640] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: ea-php56
[Fri Apr 14 05:55:07.286499 2017] [cgid:error] [pid 173967:tid 139934748841728] [client 151.226.124.247:49528] End of script output before headers: ea-php56
[Fri Apr 14 05:55:07.287051 2017] [cgid:error] [pid 173873:tid 139935212316640] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: ea-php56
[Fri Apr 14 05:55:07.287099 2017] [cgid:error] [pid 173876:tid 139934807590656] [client 151.226.124.247:49529] End of script output before headers: ea-php56
[Fri Apr 14 05:55:07.287958 2017] [cgid:error] [pid 173873:tid 139935212316640] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: ea-php56
[Fri Apr 14 05:55:07.289266 2017] [cgid:error] [pid 173880:tid 139934732056320] [client 151.226.124.247:49530] End of script output before headers: ea-php56
[Fri Apr 14 05:55:07.333016 2017] [core:error] [pid 173966:tid 139934757234432] (104)Connection reset by peer: [client 151.226.124.247:49527] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 05:55:07.334319 2017] [core:error] [pid 173877:tid 139934757234432] (104)Connection reset by peer: [client 151.226.124.247:49526] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 05:55:08.290198 2017] [:notice] [pid 173875:tid 139935212316640] [host xxxxxxx] mod_lsapi:  Selfstarter 173875 stopped
[Fri Apr 14 05:55:09.291278 2017] [core:notice] [pid 171059:tid 139935212316640] AH00060: seg fault or similar nasty error detected in the parent process
[Fri Apr 14 05:55:10.479850 2017]

This is occurring on old servers (Centos 6) with unchanged settings and new servers (Centos 7) alike.
Memory does not appear to be the issue as there is plenty on each server and the older ones have only started doing that with no change in configuration or load.

Any pointers or obvious things to check as we have to manually start the process again.

XN-Matt · Apr 14, 2017

Server 2

Code:

[Fri Apr 14 08:12:59.684280 2017] [core:error] [pid 569847:tid 140266308560640] (104)Connection reset by peer: [client 87.127.112.191:62876] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 08:12:59.772203 2017] [core:error] [pid 569875:tid 140266400880384] (104)Connection reset by peer: [client 87.127.112.191:62877] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 08:13:15.282876 2017] [core:error] [pid 569819:tid 140266274989824] (104)Connection reset by peer: [client 87.127.112.191:62886] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 08:13:15.372901 2017] [core:error] [pid 569847:tid 140266375702272] (104)Connection reset by peer: [client 87.127.112.191:62887] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 08:13:26.613292 2017] [core:error] [pid 569763:tid 140266400880384] (104)Connection reset by peer: [client 217.33.149.186:53508] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 08:13:26.705488 2017] [core:error] [pid 569791:tid 140266283382528] (104)Connection reset by peer: [client 217.33.149.186:53509] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 08:13:28.050111 2017] [cgid:error] [pid 569760:tid 140266757785568] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: ea-php56
[Fri Apr 14 08:13:28.056537 2017] [cgid:error] [pid 569819:tid 140266350524160] [client 78.148.181.125:21647] End of script output before headers: ea-php56, referer: xx
[Fri Apr 14 08:13:28.059517 2017] [cgid:error] [pid 569760:tid 140266757785568] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: ea-php56
[Fri Apr 14 08:13:28.061429 2017] [cgid:error] [pid 569819:tid 140266350524160] [client 78.148.181.125:21647] End of script output before headers: ea-php56, referer:  xx
[Fri Apr 14 08:13:29.459972 2017] [core:error] [pid 569819:tid 140266417665792] (104)Connection reset by peer: [client 87.127.112.191:62897] AH00574: ap_content_length_filter: apr_bucket_read() failed
[Fri Apr 14 08:13:30.037498 2017] [cgid:error] [pid 569756:tid 140266757785568] AH01239: cgid daemon process died, restarting
[Fri Apr 14 08:13:30.217393 2017] [cgid:error] [pid 569875:tid 140266528753408] (111)Connection refused: [client 87.127.112.191:62898] AH01258: cgid daemon is gone; is Apache terminating?: /usr/local/cpanel/cgi-sys/autodiscover.cgi
[Fri Apr 14 08:13:30.447398 2017] [:notice] [pid 569762:tid 140266757785568] [host xxxx] mod_lsapi:  Selfstarter 569762 stopped
[Fri Apr 14 08:13:31.045500 2017] [core:notice] [pid 569756:tid 140266757785568] AH00060: seg fault or similar nasty error detected in the parent process
[Fri Apr 14 08:13:32.647409 2017] [cgid:error] [pid 569819:tid 140266350524160] [client 78.148.181.125:21647] AH01259: CGI process 0 didn't exit, sending SIGKILL, referer: xxx

NiteWave · Apr 15, 2017

please refer
https://www.litespeedtech.com/support/forum/threads/litespeed-stooped.14815/
see if still occur under 5.1.15

XN-Matt · Apr 15, 2017

Even if already running 5.1.15 do we need to force a re-install?

NiteWave · Apr 15, 2017

when did you install 5.1.5 ? you can check the time stamp of /usr/local/lsws/bin/lshttpd.5.1.15
compare with
~>curl -I http://www.litespeedtech.com/packages/5.0/lsws-5.1.15-ent-x86_64-linux.tar.gz
HTTP/1.1 200 OK
ETag: "6c59d8-58f0eadd-27fd8278a587472f"
Last-Modified: Fri, 14 Apr 2017 15:29:33 GMT
Content-Type: application/gzip
Content-Length: 7100888
Date: Sat, 15 Apr 2017 02:50:06 GMT
Accept-Ranges: bytes
Server: LiteSpeed
Connection: Keep-Alive

if it's linux x86 64 bit system.

if your lshttpd.5.1.5 is newer than "Last-Modified: Fri, 14 Apr 2017 15:29:33 GMT", I think no need to force re-install. otherwise, best to do it to ensure it's latest build.

XN-Matt · Apr 15, 2017

OK, what is the significance in that date?
I can't see any changes in the release log between when we might have first downloaded it and now.

XN-Matt · Apr 15, 2017

Well that made no difference. Crashed again.

Code:

[Sat Apr 15 05:07:08.034194 2017] [:notice] [pid 585649:tid 140077775644640] [host xxx] mod_lsapi:  Selfstarter 585649 stopped
[Sat Apr 15 05:07:08.536042 2017] [cgid:error] [pid 585650:tid 140077329999616] (111)Connection refused: [client 151.226.124.247:53260] AH01258: cgid daemon is gone; is Apache terminating?: /usr/local/cpanel/cgi-sys/ea-php56
[Sat Apr 15 05:07:08.790292 2017] [core:notice] [pid 585646:tid 140077775644640] AH00060: seg fault or similar nasty error detected in the parent process
[Sat Apr 15 05:07:09.171622 2017] [:error] [pid 585650:tid 140077455890176] [client 166.137.14.22] ModSecurity: Access denied with code 403 (phase 2). Matched phrase "/via/" at TX:header_name. [file "/var/cpanel/cwaf/rules/10_HTTP_HTTP.conf"] [line "32"] [id "210740"] [rev "2"] [msg "COMODO WAF: HTTP header is restricted by policy"] [data "/Via/"] [severity "WARNING"] [hostname "xx"] [uri "/images/cps/xxx/SS851308.JPG"] [unique_id "WPGcbRlfaNGVaSJ9Nvg@CAAAAAQ"]
[Sat Apr 15 05:07:10.216830 2017] [cgid:error] [pid 585734:tid 140077413926656] [client 151.226.124.247:53261] AH01259: CGI process 0 didn't exit, sending SIGKILL

XN-Matt · Apr 15, 2017

We might have correlated this to mod_security. We are using the CWAF rules but have not updated automatically.
User memory increase is seen a bit before the crash and our mod_sec logs have timestamps like.

Code:

[14/Apr/2017:04:23:28 +0100] - 217.182.132.65 49084 xx:80 80
[14/Apr/2017:04:23:31 +0100] - 69.162.124.231 8383 xx:80 80
[14/Apr/2017:04:23:46 +0100] - 180.76.15.144 56652 xx:80 80
[14/Apr/2017:04:23:52 +0100] - 69.162.124.228 35495 xx:443 443
[14/Apr/2017:04:23:53 +0100] - 63.143.42.245 4824 xx:80 80
[14/Apr/2017:04:24:01 +0100] - 69.162.124.229 54729 xx:80 80
[14/Apr/2017:04:24:11 +0100] - 180.76.15.157 59377 xx:80 80

However, during the issue (immediately before a restart/start)

Code:

[14/Apr/2017:04:56:16 +0100] WPBIXtFI7MAs9zMwRJZzgQAAAMg 185.159.36.6 60387 xx 80
[14/Apr/2017:04:56:37 +0100] WPBIdfU1X-9qk3-SZBtSvgAAAUY 217.182.132.86 42730 xx 80
[14/Apr/2017:04:56:44 +0100] WPBIfCg@qCI2LK3fcDeeRgAAAA0 180.76.15.138 57508 xx 80
[14/Apr/2017:04:57:13 +0100] WPBImENXch2uI@4kTzoasgAAAUo 185.159.36.6 48209 xx 80
[14/Apr/2017:04:57:36 +0100] WPBIrkNXch2uI@4kTzoauQAAAVg 69.162.124.229 22801 xx 80
[14/Apr/2017:04:57:36 +0100] WPBIsARt7grABznsczoH6QAAAFI 95.108.179.28 54303 xx 80
[14/Apr/2017:04:57:38 +0100] WPBIsCXc6aspDi@0-a1rNAAAABA 69.162.124.229 24691 xx 443

After we reset, the logs don't have the random "WPBIsCXc6aspDi@0-a1rNAAAABA" or similar reference in them.
mod_sec leak or something in recent releases?

Graph attached. Furthest left was the increase in user memory usage (red) and then a restart and then again on the right (red) and a restart right after.

Is the same on all servers affected.

XN-Matt · Apr 18, 2017

No response in over 2 days.

C'mon guys. We expect SOME support here for your software which just keeps dying on us.

mistwang · Apr 19, 2017

XN-Matt said:

Well that made no difference. Crashed again.

Code:

[Sat Apr 15 05:07:08.034194 2017] [:notice] [pid 585649:tid 140077775644640] [host xxx] mod_lsapi:  Selfstarter 585649 stopped
[Sat Apr 15 05:07:08.536042 2017] [cgid:error] [pid 585650:tid 140077329999616] (111)Connection refused: [client 151.226.124.247:53260] AH01258: cgid daemon is gone; is Apache terminating?: /usr/local/cpanel/cgi-sys/ea-php56
[Sat Apr 15 05:07:08.790292 2017] [core:notice] [pid 585646:tid 140077775644640] AH00060: seg fault or similar nasty error detected in the parent process
[Sat Apr 15 05:07:09.171622 2017] [:error] [pid 585650:tid 140077455890176] [client 166.137.14.22] ModSecurity: Access denied with code 403 (phase 2). Matched phrase "/via/" at TX:header_name. [file "/var/cpanel/cwaf/rules/10_HTTP_HTTP.conf"] [line "32"] [id "210740"] [rev "2"] [msg "COMODO WAF: HTTP header is restricted by policy"] [data "/Via/"] [severity "WARNING"] [hostname "xx"] [uri "/images/cps/xxx/SS851308.JPG"] [unique_id "WPGcbRlfaNGVaSJ9Nvg@CAAAAAQ"]
[Sat Apr 15 05:07:10.216830 2017] [cgid:error] [pid 585734:tid 140077413926656] [client 151.226.124.247:53261] AH01259: CGI process 0 didn't exit, sending SIGKILL

Those looks like Apache error log, not generated by LiteSpeed. Maybe your server is still running Apache by accident?

XN-Matt · Apr 19, 2017

Nope. This is LiteSpeed.
It is happening on ALL our LS servers - new installs with CentOS 7 and old installations with CentOS 6

Code:

root@[~]# ps aux | grep litespeed
root      819211  5.9  0.5 112364 90104 ?        SN   14:00   0:02 litespeed (lshttpd)
nobody    819258  1.7  0.5 296240 90940 ?        SNl  14:00   0:00 litespeed (lshttpd)
root      819593  0.0  0.0 103332   896 pts/3    S+   14:01   0:00 grep litespeed

root@[~]# ps aux | grep apache
root      819634  0.0  0.0 103328   888 pts/3    S+   14:01   0:00 grep apache

root@[~]# ps aux | grep http
root      819211  5.0  0.5 112364 90104 ?        SN   14:00   0:02 litespeed (lshttpd)
root      819213  0.0  0.0  12648   688 ?        SN   14:00   0:00 httpd (lscgid)
nobody    819258  1.8  0.5 297180 91144 ?        SNl  14:00   0:00 litespeed (lshttpd)
root      819654  0.0  0.0 103332   892 pts/3    S+   14:01   0:00 grep http

mistwang · Apr 19, 2017

There must be Apache being started temporarily and crashed. if you look at the logging style, the timestamp, litespeed generated log entries are in completely different format.
Are you using cPanel?

XN-Matt · Apr 19, 2017

Yes it is.
Litespeed is crashing (which we can confirm it is), cPanel may be starting Apache but still the issue remains, why is LiteSpeed crashing?

mistwang · Apr 19, 2017

It is probably because cPanel killing LiteSpeed, then start up Apache.
We need to find out what operation trigger Apache being started, and why?
Can you give us temp root access to the server having this problem? Just open a support ticket.

XN-Matt · Apr 19, 2017

There is no support ticket option in the client area.
I really wish you would not expect paying customers to use a forum. It is simply not good enough for a support mechanism and has frankly taken far too long.

mistwang · Apr 19, 2017

you can submit a bug report ticket. should be able to do that from client area.

LS Dying - Various Errors

XN-Matt

Well-Known Member

XN-Matt

Well-Known Member

NiteWave

Administrator

XN-Matt

Well-Known Member

NiteWave

Administrator

XN-Matt

Well-Known Member

XN-Matt

Well-Known Member

XN-Matt

Well-Known Member

Attachments

XN-Matt

Well-Known Member

mistwang

LiteSpeed Staff

XN-Matt

Well-Known Member

mistwang

LiteSpeed Staff

XN-Matt

Well-Known Member

mistwang

LiteSpeed Staff

XN-Matt

Well-Known Member

mistwang

LiteSpeed Staff