autorestarting and error: Connection reset by peer

[hd-sam]

Hi,

I've tried running 4.1.11 and 4.1.10 and get these errors with both.
The server is running a high traffic WordPress blog and SMF forum.

I had XCache installed and thought that was the problem. I disabled it, issue exists, installed APC, but it still exists no matter what I do.

The error logs show the server randomly auto restarting all the time.

Here is a debug report:

PHP Code:

2012-04-01 08:03:29.113 [INFO] Remove pid: 5440, exitcode: 0
2012-04-01 08:03:57.814 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::handleEvents() events=4!
2012-04-01 08:03:57.815 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Written to client: 56940
2012-04-01 08:03:57.815 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] m_pHandler->onWrite() return 1
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::handleEvents() events=25!
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::suspendRead()...
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Close socket ...
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Available Connections: 2000, concurrent conn: 0
2012-04-01 08:04:13.530 [NOTICE] [Child: 4951] Shut down successfully!
2012-04-01 08:04:13.531 [INFO] Permission denied, let parent kill it: 5404
2012-04-01 08:04:13.531 [INFO] [AdminPHP] kill pid: 5404
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4934000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4864000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe48e1000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4a75000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4ab8000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4a6e000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe492d000

….. the release mapped data error goes on for 2 pages…..

2012-04-01 08:04:13.537 [DEBUG] [MMAP] Release mapped data at 0x7f9fe48ac000
2012-04-01 08:04:13.537 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4a67000
2012-04-01 08:04:13.541 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets of process with pid=4951 !
2012-04-01 08:04:13.541 [INFO] [CLEANUP] Clean up child process with pid: 5404
2012-04-01 08:04:13.641 [NOTICE] [AutoRestarter] child process with pid=4951 exited with status=0!
2012-04-01 08:04:50.532 [NOTICE] [Child: 5413] Shut down successfully!
2012-04-01 08:04:50.538 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets of process with pid=5413 !
2012-04-01 08:04:50.539 [INFO] [CLEANUP] Clean up child process with pid: 5515
2012-04-01 08:04:50.639 [NOTICE] [AutoRestarter] child process with pid=5413 exited with status=0!
2012-04-01 08:04:50.639 [NOTICE] [PID:32577] Server Stopped!
2012-04-01 08:05:01.564 [INFO] [127.0.0.1:59670-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:10:01.307 [INFO] [127.0.0.1:36268-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacteddomain.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, e
rror: Connection reset by peer!
2012-04-01 08:15:02.013 [INFO] [127.0.0.1:40754-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:16:09.295 [INFO] Remove pid: 6659, exitcode: 0
2012-04-01 08:20:01.709 [INFO] [127.0.0.1:45336-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:21:54.621 [INFO] [HTAccess] Updating configuration from [/home/redactedusername/public_html/wp-admin/.htaccess]
2012-04-01 08:21:57.058 [INFO] [HTAccess] Updating configuration from [/home/redactedusername/public_html/images/.htaccess]
2012-04-01 08:25:01.473 [INFO] [127.0.0.1:50243-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:30:01.178 [INFO] [127.0.0.1:55391-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status] 


Here is a copy of the .rtreport:

PHP Code:



root@cr1 [/tmp/lshttpd]# more .rtreport
VERSION: LiteSpeed Web Server/Enterprise/4.1.11
UPTIME: 00:42:56
BPS_IN: 62, BPS_OUT: 789, SSL_BPS_IN: 0, SSL_BPS_OUT: 0
MAXCONN: 2000, MAXSSL_CONN: 500, PLAINCONN: 113, AVAILCONN: 1887, IDLECONN: 107, SSLCONN: 0, AVAILSSL: 500
REQ_RATE []: REQ_PROCESSING: 6, REQ_PER_SEC: 63.0, TOT_REQS: 108634, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [APVH_*_redacted.hostname.com]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 0, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [APVH_redacted.hostname.com]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 0, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [_AdminVHost]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 8, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [APVH_98.158.153.71_redacted.hostname.com]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 0, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [APVH_redacteddomain.com]: REQ_PROCESSING: 5, REQ_PER_SEC: 63.0, TOT_REQS: 108620, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
EXTAPP [LSAPI] [] [lsphp5]: CMAXCONN: 35, EMAXCONN: 35, POOL_SIZE: 10, INUSE_CONN: 2, IDLE_CONN: 8, WAITQUE_DEPTH: 0, REQ_PER_SEC: 18.4, TOT_REQS: 35842
BLOCKED_IP: 
EOF
root@cr1 [/tmp/lshttpd]# 


I'm also getting these connection reset by peer errors every few mintes:

PHP Code:

2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacted.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, error: Connection reset by peer! 


Any help would be appreciated!

[NiteWave]

run dmesg, see if see many "nf_conntrack: table full, dropping packet"

[hd-sam]

Thanks for the fast response NiteWave.

There are none of those in dmesg.

It does show about 2 pages of entries from the CSF firewall of UDP_IN and TCP_IN blocked, here's a small ex:

PHP Code:

Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.135.203.237 DST=R.E.D.A.C.T.E.D. LEN=364 TOS=0x00 PREC=0x00 TTL=114 ID=28681 PROTO=UDP SPT=500
 DPT=500 LEN=344 
Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.135.203.237 DST=R.E.D.A.C.T.E.D. LEN=316 TOS=0x00 PREC=0x00 TTL=114 ID=28682 PROTO=UDP SPT=500
 DPT=500 LEN=296 
Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.22.71.4 DST=R.E.D.A.C.T.E.D. LEN=76 TOS=0x00 PREC=0x00 TTL=55 ID=45473 DF PROTO=UDP SPT=53 DPT
=19203 LEN=56 
Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=208.72.12.115 DST=R.E.D.A.C.T.E.D. LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=267 PROTO=UDP SPT=13396 DPT
=33436 LEN=12 


[webizen]

Originally Posted by hd-sam

...
I'm also getting these connection reset by peer errors every few mintes:

PHP Code:

2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacted.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, error: Connection reset by peer! 


...

looks like php script is crashed. make sure matching php is built.

[hd-sam]

Originally Posted by webizen

looks like php script is crashed. make sure matching php is built.

Thanks, I've actually done that about 5x now... with 4.1.10 and 4.1.11 both. That error has appeared in both versions.

I'm running PHP 5.2.17

[hd-sam]

One other thing I noticed....

Every time I empty the page cache using the W3 Total Cache plugin in WordPress, the web server appears to restart. An important thing to note is my config is with PHP SuExec disabled.

I get this email every time I empty the cache:

PHP Code:

At [01/Apr/2012:16:42:27 -0400], web server with pid=32272 received unexpected signal=11, a core file is created. A new instance of web server will be started automatically!

Please forward the following debug information to bug@litespeedtech.com.
Environment:

Server: LiteSpeed/4.1.11 Enterprise
OS: Linux
Release: 2.6.32-220.7.1.el6.x86_64
Version: #1 SMP Wed Mar 7 00:52:02 GMT 2012
Machine: x86_64

If the call stack information does not show up here, please compress and forward the core file located in /tmp/lshttpd/.

[New Thread 32272]
[New Thread 32274]
[New Thread 32273]
Core was generated by `litespeed'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000457bb3 in ?? ()
#0  0x0000000000457bb3 in ?? ()
#1  0x000000000081bd63 in ?? ()
#2  0x0000000000000000 in ?? () 


[NiteWave]

have you tried wp-super cache instead of W3-total cache?
if wp-super cache works well, then the issue is narrowed down.

[hd-sam]

Thanks NiteWave,

I was able to fix the issue regarding the crash every time I emptied the cache. I rebuilt Apache/PHP with easyapache, then built matching PHP w/ LS.
That seemed to have fixed that portion of the issue.

However, the remaining issue now appears to be this:
2012-04-02 12:47:25.618 [INFO] [97.75.77.133:58306-0#APVH_redacted.com]
connection to [/tmp/lshttpd/lsphp5.sock.615] on request #500, confirmed, 0, associated process: 29101, running: 1, error: Connection reset by peer!

This appears every 5 minutes in the error log. Exactly the same error (on request #500)
I've disabled APC and XCache as well.

[hd-sam]

I just looked at my access-logs to see what these IPs are accessing that is causing that. It seems to be timthumb.php is causing that error.

The timthumb.php file is the latest version 2.8.10.

[webizen]

Originally Posted by hd-sam

...
However, the remaining issue now appears to be this:
2012-04-02 12:47:25.618 [INFO] [97.75.77.133:58306-0#APVH_redacted.com]
connection to [/tmp/lshttpd/lsphp5.sock.615] on request #500, confirmed, 0, associated process: 29101, running: 1, error: Connection reset by peer!

This appears every 5 minutes in the error log. Exactly the same error (on request #500)
I've disabled APC and XCache as well.

It is due to your lsphp5 external app environment variable:

PHP_LSAPI_MAX_REQUESTS=500

That is NOT a problem (even with "error: Connection reset by peer!" signature). LSPHP5 parent will spawn another child to take it over.