LiteSpeed Support Forums

LiteSpeed Support Forums (http://www.litespeedtech.com/support/forum/index.php)
-   General (http://www.litespeedtech.com/support/forum/forumdisplay.php?f=25)
-   -   autorestarting and error: Connection reset by peer (http://www.litespeedtech.com/support/forum/showthread.php?t=5810)

hd-sam 04-01-2012 05:48 AM

autorestarting and error: Connection reset by peer
 
Hi,

I've tried running 4.1.11 and 4.1.10 and get these errors with both.
The server is running a high traffic WordPress blog and SMF forum.

I had XCache installed and thought that was the problem. I disabled it, issue exists, installed APC, but it still exists no matter what I do.

The error logs show the server randomly auto restarting all the time.

Here is a debug report:
PHP Code:

2012-04-01 08:03:29.113 [INFORemove pid5440exitcode0
2012
-04-01 08:03:57.814 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::handleEvents() events=4!
2012-04-01 08:03:57.815 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Written to client: 56940
2012-04-01 08:03:57.815 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] m_pHandler->onWrite() return 1
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::handleEvents() events=25!
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::suspendRead()...
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Close socket ...
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Available Connections: 2000, concurrent conn: 0
2012-04-01 08:04:13.530 [NOTICE] [Child4951Shut down successfully!
2012-04-01 08:04:13.531 [INFOPermission deniedlet parent kill it5404
2012
-04-01 08:04:13.531 [INFO] [AdminPHPkill pid5404
2012
-04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4934000
2012
-04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4864000
2012
-04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe48e1000
2012
-04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4a75000
2012
-04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4ab8000
2012
-04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4a6e000
2012
-04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe492d000

….. the release mapped data error goes on for 2 pages…..

2012-04-01 08:04:13.537 [DEBUG] [MMAPRelease mapped data at 0x7f9fe48ac000
2012
-04-01 08:04:13.537 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4a67000
2012
-04-01 08:04:13.541 [NOTICE] [AutoRestartercleanup children processes and unix sockets of process with pid=4951 !
2012-04-01 08:04:13.541 [INFO] [CLEANUPClean up child process with pid5404
2012
-04-01 08:04:13.641 [NOTICE] [AutoRestarterchild process with pid=4951 exited with status=0!
2012-04-01 08:04:50.532 [NOTICE] [Child5413Shut down successfully!
2012-04-01 08:04:50.538 [NOTICE] [AutoRestartercleanup children processes and unix sockets of process with pid=5413 !
2012-04-01 08:04:50.539 [INFO] [CLEANUPClean up child process with pid5515
2012
-04-01 08:04:50.639 [NOTICE] [AutoRestarterchild process with pid=5413 exited with status=0!
2012-04-01 08:04:50.639 [NOTICE] [PID:32577Server Stopped!
2012-04-01 08:05:01.564 [INFO] [127.0.0.1:59670-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:10:01.307 [INFO] [127.0.0.1:36268-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacteddomain.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, e
rrorConnection reset by peer!
2012-04-01 08:15:02.013 [INFO] [127.0.0.1:40754-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:16:09.295 [INFORemove pid6659exitcode0
2012
-04-01 08:20:01.709 [INFO] [127.0.0.1:45336-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:21:54.621 [INFO] [HTAccessUpdating configuration from [/home/redactedusername/public_html/wp-admin/.htaccess]
2012-04-01 08:21:57.058 [INFO] [HTAccessUpdating configuration from [/home/redactedusername/public_html/images/.htaccess]
2012-04-01 08:25:01.473 [INFO] [127.0.0.1:50243-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:30:01.178 [INFO] [127.0.0.1:55391-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status] 

Here is a copy of the .rtreport:

PHP Code:



root
@cr1 [/tmp/lshttpd]# more .rtreport
VERSIONLiteSpeed Web Server/Enterprise/4.1.11
UPTIME
00:42:56
BPS_IN
62BPS_OUT789SSL_BPS_IN0SSL_BPS_OUT0
MAXCONN
2000MAXSSL_CONN500PLAINCONN113AVAILCONN1887IDLECONN107SSLCONN0AVAILSSL500
REQ_RATE 
[]: REQ_PROCESSING6REQ_PER_SEC63.0TOT_REQS108634CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
REQ_RATE 
[APVH_*_redacted.hostname.com]: REQ_PROCESSING0REQ_PER_SEC0.0TOT_REQS0CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
REQ_RATE 
[APVH_redacted.hostname.com]: REQ_PROCESSING0REQ_PER_SEC0.0TOT_REQS0CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
REQ_RATE 
[_AdminVHost]: REQ_PROCESSING0REQ_PER_SEC0.0TOT_REQS8CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
REQ_RATE 
[APVH_98.158.153.71_redacted.hostname.com]: REQ_PROCESSING0REQ_PER_SEC0.0TOT_REQS0CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
REQ_RATE 
[APVH_redacteddomain.com]: REQ_PROCESSING5REQ_PER_SEC63.0TOT_REQS108620CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
EXTAPP 
[LSAPI] [] [lsphp5]: CMAXCONN35EMAXCONN35POOL_SIZE10INUSE_CONN2IDLE_CONN8WAITQUE_DEPTH0REQ_PER_SEC18.4TOT_REQS35842
BLOCKED_IP

EOF
root
@cr1 [/tmp/lshttpd]

I'm also getting these connection reset by peer errors every few mintes:
PHP Code:

2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacted.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, error: Connection reset by peer! 

Any help would be appreciated!

NiteWave 04-01-2012 07:28 AM

run dmesg, see if see many "nf_conntrack: table full, dropping packet"

hd-sam 04-01-2012 11:56 AM

Thanks for the fast response NiteWave.

There are none of those in dmesg.

It does show about 2 pages of entries from the CSF firewall of UDP_IN and TCP_IN blocked, here's a small ex:

PHP Code:

Firewall: *UDP_IN BlockedIN=eth0 OUTMAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.135.203.237 DST=R.E.D.A.C.T.E.DLEN=364 TOS=0x00 PREC=0x00 TTL=114 ID=28681 PROTO=UDP SPT=500
 DPT
=500 LEN=344 
Firewall
: *UDP_IN BlockedIN=eth0 OUTMAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.135.203.237 DST=R.E.D.A.C.T.E.DLEN=316 TOS=0x00 PREC=0x00 TTL=114 ID=28682 PROTO=UDP SPT=500
 DPT
=500 LEN=296 
Firewall
: *UDP_IN BlockedIN=eth0 OUTMAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.22.71.4 DST=R.E.D.A.C.T.E.DLEN=76 TOS=0x00 PREC=0x00 TTL=55 ID=45473 DF PROTO=UDP SPT=53 DPT
=19203 LEN=56 
Firewall
: *UDP_IN BlockedIN=eth0 OUTMAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=208.72.12.115 DST=R.E.D.A.C.T.E.DLEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=267 PROTO=UDP SPT=13396 DPT
=33436 LEN=12 


webizen 04-01-2012 12:52 PM

Quote:

Originally Posted by hd-sam (Post 32453)
...
I'm also getting these connection reset by peer errors every few mintes:
PHP Code:

2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacted.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, error: Connection reset by peer! 

...

looks like php script is crashed. make sure matching php is built.

hd-sam 04-01-2012 12:57 PM

Quote:

Originally Posted by webizen (Post 32456)
looks like php script is crashed. make sure matching php is built.

Thanks, I've actually done that about 5x now... with 4.1.10 and 4.1.11 both. That error has appeared in both versions.

I'm running PHP 5.2.17

hd-sam 04-01-2012 01:46 PM

One other thing I noticed....

Every time I empty the page cache using the W3 Total Cache plugin in WordPress, the web server appears to restart. An important thing to note is my config is with PHP SuExec disabled.

I get this email every time I empty the cache:

PHP Code:

At [01/Apr/2012:16:42:27 -0400], web server with pid=32272 received unexpected signal=11a core file is creatednew instance of web server will be started automatically!

Please forward the following debug information to bug@litespeedtech.com.
Environment:

ServerLiteSpeed/4.1.11 Enterprise
OS
Linux
Release
2.6.32-220.7.1.el6.x86_64
Version
#1 SMP Wed Mar 7 00:52:02 GMT 2012
Machinex86_64

If the call stack information does not show up hereplease compress and forward the core file located in /tmp/lshttpd/.

[New 
Thread 32272]
[New 
Thread 32274]
[New 
Thread 32273]
Core was generated by `litespeed'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000457bb3 in ?? ()
#0  0x0000000000457bb3 in ?? ()
#1  0x000000000081bd63 in ?? ()
#2  0x0000000000000000 in ?? () 


NiteWave 04-01-2012 06:14 PM

have you tried wp-super cache instead of W3-total cache?
if wp-super cache works well, then the issue is narrowed down.

hd-sam 04-02-2012 09:54 AM

Thanks NiteWave,

I was able to fix the issue regarding the crash every time I emptied the cache. I rebuilt Apache/PHP with easyapache, then built matching PHP w/ LS.
That seemed to have fixed that portion of the issue.

However, the remaining issue now appears to be this:
2012-04-02 12:47:25.618 [INFO] [97.75.77.133:58306-0#APVH_redacted.com]
connection to [/tmp/lshttpd/lsphp5.sock.615] on request #500, confirmed, 0, associated process: 29101, running: 1, error: Connection reset by peer!

This appears every 5 minutes in the error log. Exactly the same error (on request #500)
I've disabled APC and XCache as well.

hd-sam 04-02-2012 10:06 AM

I just looked at my access-logs to see what these IPs are accessing that is causing that. It seems to be timthumb.php is causing that error.

The timthumb.php file is the latest version 2.8.10.

webizen 04-02-2012 10:59 AM

Quote:

Originally Posted by hd-sam (Post 32465)
...
However, the remaining issue now appears to be this:
2012-04-02 12:47:25.618 [INFO] [97.75.77.133:58306-0#APVH_redacted.com]
connection to [/tmp/lshttpd/lsphp5.sock.615] on request #500, confirmed, 0, associated process: 29101, running: 1, error: Connection reset by peer!

This appears every 5 minutes in the error log. Exactly the same error (on request #500)
I've disabled APC and XCache as well.

It is due to your lsphp5 external app environment variable:
Code:

PHP_LSAPI_MAX_REQUESTS=500
That is NOT a problem (even with "error: Connection reset by peer!" signature). LSPHP5 parent will spawn another child to take it over.


All times are GMT -7. The time now is 01:32 AM.