autorestarting and error: Connection reset by peer

H

hd-sam

Active Member
#1
#1
Hi,

I've tried running 4.1.11 and 4.1.10 and get these errors with both.
The server is running a high traffic WordPress blog and SMF forum.

I had XCache installed and thought that was the problem. I disabled it, issue exists, installed APC, but it still exists no matter what I do.

The error logs show the server randomly auto restarting all the time.

Here is a debug report:
PHP: 
2012-04-01 08:03:29.113 [INFO] Remove pid: 5440, exitcode: 0
2012-04-01 08:03:57.814 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::handleEvents() events=4!
2012-04-01 08:03:57.815 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Written to client: 56940
2012-04-01 08:03:57.815 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] m_pHandler->onWrite() return 1
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::handleEvents() events=25!
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::suspendRead()...
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Close socket ...
2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Available Connections: 2000, concurrent conn: 0
2012-04-01 08:04:13.530 [NOTICE] [Child: 4951] Shut down successfully!
2012-04-01 08:04:13.531 [INFO] Permission denied, let parent kill it: 5404
2012-04-01 08:04:13.531 [INFO] [AdminPHP] kill pid: 5404
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4934000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4864000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe48e1000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4a75000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4ab8000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4a6e000
2012-04-01 08:04:13.532 [DEBUG] [MMAP] Release mapped data at 0x7f9fe492d000

….. the release mapped data error goes on for 2 pages…..

2012-04-01 08:04:13.537 [DEBUG] [MMAP] Release mapped data at 0x7f9fe48ac000
2012-04-01 08:04:13.537 [DEBUG] [MMAP] Release mapped data at 0x7f9fe4a67000
2012-04-01 08:04:13.541 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets of process with pid=4951 !
2012-04-01 08:04:13.541 [INFO] [CLEANUP] Clean up child process with pid: 5404
2012-04-01 08:04:13.641 [NOTICE] [AutoRestarter] child process with pid=4951 exited with status=0!
2012-04-01 08:04:50.532 [NOTICE] [Child: 5413] Shut down successfully!
2012-04-01 08:04:50.538 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets of process with pid=5413 !
2012-04-01 08:04:50.539 [INFO] [CLEANUP] Clean up child process with pid: 5515
2012-04-01 08:04:50.639 [NOTICE] [AutoRestarter] child process with pid=5413 exited with status=0!
2012-04-01 08:04:50.639 [NOTICE] [PID:32577] Server Stopped!
2012-04-01 08:05:01.564 [INFO] [127.0.0.1:59670-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:10:01.307 [INFO] [127.0.0.1:36268-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacteddomain.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, e
rror: Connection reset by peer!
2012-04-01 08:15:02.013 [INFO] [127.0.0.1:40754-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:16:09.295 [INFO] Remove pid: 6659, exitcode: 0
2012-04-01 08:20:01.709 [INFO] [127.0.0.1:45336-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:21:54.621 [INFO] [HTAccess] Updating configuration from [/home/redactedusername/public_html/wp-admin/.htaccess]
2012-04-01 08:21:57.058 [INFO] [HTAccess] Updating configuration from [/home/redactedusername/public_html/images/.htaccess]
2012-04-01 08:25:01.473 [INFO] [127.0.0.1:50243-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2012-04-01 08:30:01.178 [INFO] [127.0.0.1:55391-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
Here is a copy of the .rtreport:

PHP: 
root@cr1 [/tmp/lshttpd]# more .rtreport
VERSION: LiteSpeed Web Server/Enterprise/4.1.11
UPTIME: 00:42:56
BPS_IN: 62, BPS_OUT: 789, SSL_BPS_IN: 0, SSL_BPS_OUT: 0
MAXCONN: 2000, MAXSSL_CONN: 500, PLAINCONN: 113, AVAILCONN: 1887, IDLECONN: 107, SSLCONN: 0, AVAILSSL: 500
REQ_RATE []: REQ_PROCESSING: 6, REQ_PER_SEC: 63.0, TOT_REQS: 108634, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [APVH_*_redacted.hostname.com]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 0, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [APVH_redacted.hostname.com]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 0, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [_AdminVHost]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 8, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [APVH_98.158.153.71_redacted.hostname.com]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 0, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
REQ_RATE [APVH_redacteddomain.com]: REQ_PROCESSING: 5, REQ_PER_SEC: 63.0, TOT_REQS: 108620, CACHE_HITS_PER_SEC: 0.0, TOTAL_CACHE_HITS: 0
EXTAPP [LSAPI] [] [lsphp5]: CMAXCONN: 35, EMAXCONN: 35, POOL_SIZE: 10, INUSE_CONN: 2, IDLE_CONN: 8, WAITQUE_DEPTH: 0, REQ_PER_SEC: 18.4, TOT_REQS: 35842
BLOCKED_IP: 
EOF
root@cr1 [/tmp/lshttpd]#
I'm also getting these connection reset by peer errors every few mintes:
PHP: 
2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacted.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, error: Connection reset by peer!
Any help would be appreciated!
 
Last edited:
H

hd-sam

Active Member
#3
#3
Thanks for the fast response NiteWave.

There are none of those in dmesg.

It does show about 2 pages of entries from the CSF firewall of UDP_IN and TCP_IN blocked, here's a small ex:

PHP: 
Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.135.203.237 DST=R.E.D.A.C.T.E.D. LEN=364 TOS=0x00 PREC=0x00 TTL=114 ID=28681 PROTO=UDP SPT=500
 DPT=500 LEN=344 
Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.135.203.237 DST=R.E.D.A.C.T.E.D. LEN=316 TOS=0x00 PREC=0x00 TTL=114 ID=28682 PROTO=UDP SPT=500
 DPT=500 LEN=296 
Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.22.71.4 DST=R.E.D.A.C.T.E.D. LEN=76 TOS=0x00 PREC=0x00 TTL=55 ID=45473 DF PROTO=UDP SPT=53 DPT
=19203 LEN=56 
Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=208.72.12.115 DST=R.E.D.A.C.T.E.D. LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=267 PROTO=UDP SPT=13396 DPT
=33436 LEN=12
 
W

webizen

Well-Known Member
#4
#4
hd-sam said:
...
I'm also getting these connection reset by peer errors every few mintes:
PHP: 
2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacted.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, error: Connection reset by peer!
...
Click to expand...
looks like php script is crashed. make sure matching php is built.
 
H

hd-sam

Active Member
#6
#6
One other thing I noticed....

Every time I empty the page cache using the W3 Total Cache plugin in WordPress, the web server appears to restart. An important thing to note is my config is with PHP SuExec disabled.

I get this email every time I empty the cache:

PHP: 
At [01/Apr/2012:16:42:27 -0400], web server with pid=32272 received unexpected signal=11, a core file is created. A new instance of web server will be started automatically!

Please forward the following debug information to bug@litespeedtech.com.
Environment:

Server: LiteSpeed/4.1.11 Enterprise
OS: Linux
Release: 2.6.32-220.7.1.el6.x86_64
Version: #1 SMP Wed Mar 7 00:52:02 GMT 2012
Machine: x86_64

If the call stack information does not show up here, please compress and forward the core file located in /tmp/lshttpd/.

[New Thread 32272]
[New Thread 32274]
[New Thread 32273]
Core was generated by `litespeed'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000457bb3 in ?? ()
#0  0x0000000000457bb3 in ?? ()
#1  0x000000000081bd63 in ?? ()
#2  0x0000000000000000 in ?? ()
 
H

hd-sam

Active Member
#8
#8
Thanks NiteWave,

I was able to fix the issue regarding the crash every time I emptied the cache. I rebuilt Apache/PHP with easyapache, then built matching PHP w/ LS.
That seemed to have fixed that portion of the issue.

However, the remaining issue now appears to be this:
2012-04-02 12:47:25.618 [INFO] [97.75.77.133:58306-0#APVH_redacted.com]
connection to [/tmp/lshttpd/lsphp5.sock.615] on request #500, confirmed, 0, associated process: 29101, running: 1, error: Connection reset by peer!

This appears every 5 minutes in the error log. Exactly the same error (on request #500)
I've disabled APC and XCache as well.
 
H

hd-sam

Active Member
#9
#9
I just looked at my access-logs to see what these IPs are accessing that is causing that. It seems to be timthumb.php is causing that error.

The timthumb.php file is the latest version 2.8.10.
 
W

webizen

Well-Known Member
#10
#10
hd-sam said:
...
However, the remaining issue now appears to be this:
2012-04-02 12:47:25.618 [INFO] [97.75.77.133:58306-0#APVH_redacted.com]
connection to [/tmp/lshttpd/lsphp5.sock.615] on request #500, confirmed, 0, associated process: 29101, running: 1, error: Connection reset by peer!

This appears every 5 minutes in the error log. Exactly the same error (on request #500)
I've disabled APC and XCache as well.
Click to expand...
It is due to your lsphp5 external app environment variable:
Code: 
PHP_LSAPI_MAX_REQUESTS=500
That is NOT a problem (even with "error: Connection reset by peer!" signature). LSPHP5 parent will spawn another child to take it over.
 
H

hd-sam

Active Member
#11
#11
webizen said:
It is due to your lsphp5 external app environment variable:
Code: 
PHP_LSAPI_MAX_REQUESTS=500
That is NOT a problem (even with "error: Connection reset by peer!" signature). LSPHP5 parent will spawn another child to take it over.
Click to expand...
Excellent, thank you. That is good to know. Would you recommend I change the that number or leave it?

Another important thing to note we have litespeed installed on many servers (all are cpanel servers), some of them used for shared hosting, some of them for dedicated sites. While I don't have issues with them, I've noticed every single one is autorestarting.

On every single one, the Current Uptime is under 1 hour (checked via Real-Time Stats in LS Web Admin). I've never seen it go more than 2 hours. They are all running 4.1.11 or 4.1.10.

Is that normal too?
Is anyone else experiencing the same thing?
 
W

webizen

Well-Known Member
#12
#12
1. You can leave php_lsapi_max_requests=500 as is. As pointed out, that's not a problem.
2. For self restart every hour, pls check whm => service configuration, make sure apache is disabled. also check error log (i.e. /usr/local/apache/logs/error_log) for details of restart.
 
You must log in or register to reply here.
Top