[solved] web server with pid=12027 received unexpected signal=11

T

tophostingcenter

Active Member
#1
#1
web server with pid=12027 received unexpected signal=11, no core file is created
Hi,

We are getting this error message at least 10 times per day , can you help on this :

some time this one:

--
At [09/Nov/2010:07:40:09 -0500], web server with pid=12027 received unexpected signal=11, no core file is created. A new instance of web server will be started automatically!
--

And other time :

--
At [09/Nov/2010:07:02:07 -0500], web server with pid=19911 received unexpected signal=6, no core file is created. A new instance of web server will be started automatically!
--

We are using :

Hsphere Linux Web Server
Litespeed Web Server Enterprise v4.1rc3

Let us know if you need the access to our web server to fix this ..

regards

eric
tophostingcenter.com
Your Cloud Server provider
Use this Promocode: THC20
 
Last edited by a moderator:
T

tophostingcenter

Active Member
#2
#2
Update

hi,

I just wanted to update you with this error from the same server

==
LiteSpeed Web server c2web1.dcsl1.tophostingcenter.com on c2web1.dcsl1.tophostingcenter.com restarts automatically to fix 503 Errors
==

Thanks
eric
 
W

webizen

Well-Known Member
#5
#5
Hi Eric,

The following message is normal, "Auto Fix 503 Error" is enabled by default (web admin console -> server -> general). Detailed description is at http://litespeedtech.com/docs/webserver/config/general/#autoFix503

==
LiteSpeed Web server c2web1.dcsl1.tophostingcenter.com on c2web1.dcsl1.tophostingcenter.com restarts automatically to fix 503 Errors
==

The following messages have to do with server passing listeners during graceful restart. Please do a force re-install the latest build (4.1rc3) from LSWS web admin console.
--
At [09/Nov/2010:07:40:09 -0500], web server with pid=12027 received unexpected signal=11, no core file is created. A new instance of web server will be started automatically!
--

And other time :

--
At [09/Nov/2010:07:02:07 -0500], web server with pid=19911 received unexpected signal=6, no core file is created. A new instance of web server will be started automatically!
--

Please do a force reinstall from LSWS web console to use the latest build.

Webizen
 
T

tophostingcenter

Active Member
#6
#6
Hello,

Thank you very much, appreciated!

I did reinstall with force and will see if the issue has been resolved.

I will let you know if I need further support.

Thanks again !

Eric
 
T

tophostingcenter

Active Member
#7
#7
Hi,

Unfortunately, we got the same error again.

--
At [10/Nov/2010:03:47:48 -0500], web server with pid=9347 received unexpected signal=11, a core file is created. A new instance of web server will be started automatically!

Please forward the following debug information to litespeedtech.com.
Environment:

Server: LiteSpeed/4.1RC3 Enterprise
OS: Linux
Release: 2.6.18-028stab070.7
Version: #1 SMP Fri Oct 1 13:53:00 MSD 2010
Machine: x86_64

If the call stack information does not show up here, please compress and forward the core file located in /tmp/lshttpd/.
--
At [10/Nov/2010:03:46:52 -0500], web server with pid=6133 received unexpected signal=11, a core file is created. A new instance of web server will be started automatically!

Please forward the following debug information to litespeedtech.com.
Environment:

Server: LiteSpeed/4.1RC3 Enterprise
OS: Linux
Release: 2.6.18-028stab070.7
Version: #1 SMP Fri Oct 1 13:53:00 MSD 2010
Machine: x86_64

If the call stack information does not show up here, please compress and forward the core file located in /tmp/lshttpd/.
==

The server login details that you have is still good if you want to get in our server to check and fix.

Thanks in advance

Eric
 
W

webizen

Well-Known Member
#10
#10
Hi Eric,

Here are what we found:
1. PHP uses alot of memory (over 100MB/each) out of 850MB total. IOW, a few PHP processes would exhaust system memory.
2. A few Litespeed processes dump core by out of memory issue.
3. Sometimes, we could not log in with following error messages:

Received disconnect from xxx.xxx.xxx.xxx: 2: fork failed: Cannot allocate memory.​

The issue
-----------
PHP is too fat and uses alot of memory (more than it should).

Suggestion
------------
Rebuild lsphp with only necessary modules to avoid the memory shortage issue.

Regards,

Webizen
 
T

tophostingcenter

Active Member
#11
#11
Hello,

I'm sorry to bring this again, but the problem is worst and worst now. Please find below the latest error message that we got, let us know what to do .

--
At [03/Dec/2010:09:47:12 -0500], web server with pid=20073 received unexpected signal=11, a core file is created. A new instance of web server will be started automatically!

Please forward the following debug information to bug@litespeedtech.com.
Environment:

Server: LiteSpeed/4.1RC3 Enterprise
OS: Linux
Release: 2.6.18-028stab070.7
Version: #1 SMP Fri Oct 1 13:53:00 MSD 2010
Machine: x86_64

If the call stack information does not show up here, please compress and forward the core file located in /tmp/lshttpd/.

[New Thread 18051]
[New Thread 18050]
Core was generated by `litespeed'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000004e8d9a in EventReactor::andMask2 (this=0x1ae7d850, mask=-2)
at /home/gwang/release/httpd/httpd/edio/eventreactor.h:52
in /home/gwang/release/httpd/httpd/edio/eventreactor.h
#0 0x00000000004e8d9a in EventReactor::andMask2 (this=0x1ae7d850, mask=-2)
at /home/gwang/release/httpd/httpd/edio/eventreactor.h:52
#1 0x00000000004e8cce in Multiplexer::suspendRead (this=0x1aec6950,
pHandler=0x1ae7d850)
at /home/gwang/release/httpd/httpd/edio/multiplexer.cpp:26
#2 0x000000000045db4d in HttpIOLink::suspendRead (this=0x1ae7d850)
at /home/gwang/release/httpd/httpd/http/httpiolink.cpp:192
#3 0x0000000000459df3 in HttpExtConnector::eek:nRead (this=0x1aece470,
pConn=0x1ae7d850)
at /home/gwang/release/httpd/httpd/http/httpextconnector.cpp:666
#4 0x00000000004a460e in HttpConnection::readReqBody (this=0x1ae7d850)
at /home/gwang/release/httpd/httpd/http/httpconnection.cpp:437
#5 0x00000000004a7b09 in HttpConnection::eek:nReadEx (this=0x1ae7d850)
at /home/gwang/release/httpd/httpd/http/httpconnection.cpp:1679
#6 0x00000000004609c8 in HttpIOLink::doReadT (this=0x1ae7d850)
at /home/gwang/release/httpd/httpd/http/httpiolink.h:145
#7 0x000000000045f217 in HttpIOLink::eek:nReadT (pThis=0x1ae7d850)
at /home/gwang/release/httpd/httpd/http/httpiolink.cpp:893
#8 0x000000000045d9e8 in HttpIOLink::handleEvents (this=0x1ae7d850, evt=1)
at /home/gwang/release/httpd/httpd/http/httpiolink.cpp:162
#9 0x00000000004e8a94 in PollfdReactor::processAllEvents (this=0x1aec6960)
at /home/gwang/release/httpd/httpd/edio/pollfdreactor.h:143
#10 0x00000000004e87b5 in Poller::waitAndProcessEvents (this=0x1aec6950,
iTimeoutMilliSec=100)
at /home/gwang/release/httpd/httpd/edio/poller.cpp:69
#11 0x000000000044d629 in EventDispatcher::run (this=0x1aea1898)
at /home/gwang/release/httpd/httpd/http/eventdispatcher.cpp:225
#12 0x000000000040f5e3 in HttpServerImpl::start (this=0x1aea1870)
at /home/gwang/release/httpd/httpd/main/httpserver.cpp:462
#13 0x000000000041259f in HttpServer::start (this=0x85a4b0)
at /home/gwang/release/httpd/httpd/main/httpserver.cpp:1827
#14 0x0000000000409a9c in LshttpdMain::main (this=0x1aea14b0, argc=1,
argv=0x7fff8dca6d28)
at /home/gwang/release/httpd/httpd/main/lshttpdmain.cpp:1728
#15 0x000000000040583f in main (argc=1, argv=0x7fff8dca6d28)
at /home/gwang/release/httpd/httpd/main.cpp:121
--

Thanks
 
W

webizen

Well-Known Member
#12
#12
Eric,

This is still a resource(memory) shortage issue.

Your current privvmpages failure count keeps increasing (the last column below)
# cat /proc/user_beancounters | grep privvmpages
privvmpages 103663 103705 148000 168000 35854072
# cat /proc/user_beancounters | grep privvmpages
privvmpages 94619 141945 148000 168000 35854093
Click to expand...
Please have your host increase privvmpages barrier and limit to a reasonable level (say 256000 280000).

Webizen
 
Last edited:
W

webizen

Well-Known Member
#14
#14
Great!

So far privvmpages failcnt has not changed.

BTW, you may want to bump up tcpsndbuf barrier and limit to 2700000 3800000 to stop its failcnt.

# cat /proc/user_beancounters | grep tcpsndbuf
tcpsndbuf 215080 343384 1720320 2703360 827560246

# cat /proc/user_beancounters | grep tcpsndbuf
tcpsndbuf 387992 468160 1720320 2703360 827665338
Click to expand...
 
Last edited:
T

tophostingcenter

Active Member
#16
#16
Hi,

Very sorry to bother you again, but we still have the same issue again. I don't know why we have this container having this issue, we have a couple other ones setup in the same way and we don't have that issue .

Here the error:

--
At [06/Dec/2010:09:48:03 -0500], web server with pid=30689 received unexpected signal=11, a core file is created. A new instance of web server will be started automatically!

Please forward the following debug information to bug@litespeedtech.com.
Environment:

Server: LiteSpeed/4.1RC3 Enterprise
OS: Linux
Release: 2.6.18-028stab070.7
Version: #1 SMP Fri Oct 1 13:53:00 MSD 2010
Machine: x86_64

If the call stack information does not show up here, please compress and forward the core file located in /tmp/lshttpd/.

Core was generated by `/bin/sh /usr/local/lsws/admin/misc/cleancache.sh /tmp/lshttpd/swap'.
Program terminated with signal 11, Segmentation fault.
#0 0x00002b025ca8852c in dl_main () from /lib64/ld-linux-x86-64.so.2
#0 0x00002b025ca8852c in dl_main () from /lib64/ld-linux-x86-64.so.2
#1 0x00002b025ca9936b in _dl_sysdep_start () from /lib64/ld-linux-x86-64.so.2
#2 0x00002b025ca87388 in _dl_start () from /lib64/ld-linux-x86-64.so.2
#3 0x00002b025ca86a78 in _start () from /lib64/ld-linux-x86-64.so.2
#4 0x0000000000000003 in ?? ()
#5 0x00007fff3e9d7f1a in ?? ()
#6 0x00007fff3e9d7f22 in ?? ()
#7 0x00007fff3e9d7f4b in ?? ()
#8 0x0000000000000000 in ?? ()
--

Thanks for your help again.

Eric
 
W

webizen

Well-Known Member
#18
#18
Hi Eric,

The resource limits raised previously are not high enough.

in /usr/local/lsws/logs/stderr.log,
2010-12-05 15:39:49.249 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
2010-12-05 15:39:57.496 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
2010-12-05 15:39:57.567 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
2010-12-05 15:39:57.662 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
2010-12-05 15:39:59.606 [STDERR] accept() failed: Cannot allocate memory
2010-12-05 15:40:00.322 [STDERR] lsphp5: error while loading shared libraries: libsepol.so.1: cannot open shared object file: Error 23
2010-12-05 15:51:20.995 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
2010-12-05 15:51:21.055 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
2010-12-05 15:51:21.062 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
2010-12-05 15:51:21.063 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
2010-12-05 15:51:21.078 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
2010-12-05 15:51:21.183 [STDERR] fork() failed, please increase process limit: Cannot allocate memory
Click to expand...
Please double the privvmpages and tcpsndbuf barrier and limit to make it hard to hit and hence failcnt stops.
privvmpages -> 512000 640000
tcpsndbuf -> 6200000 8200000

It is also recommended vm be restarted to clear all the failure counts.

Whether or not these settings are good depends on the traffic hit on the box and the hosted applications (php and others). It could be applications demand more resources than others, etc.

Webizen
 
Last edited:
You must log in or register to reply here.
Top