autorestarting and error: Connection reset by peer

Discussion in 'General' started by hd-sam, Apr 1, 2012.

  1. hd-sam

    hd-sam Member

    Hi,

    I've tried running 4.1.11 and 4.1.10 and get these errors with both.
    The server is running a high traffic WordPress blog and SMF forum.

    I had XCache installed and thought that was the problem. I disabled it, issue exists, installed APC, but it still exists no matter what I do.

    The error logs show the server randomly auto restarting all the time.

    Here is a debug report:
    PHP:
    2012-04-01 08:03:29.113 [INFORemove pid5440exitcode0
    2012
    -04-01 08:03:57.814 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::handleEvents() events=4!
    2012-04-01 08:03:57.815 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Written to client: 56940
    2012-04-01 08:03:57.815 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] m_pHandler->onWrite() return 1
    2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::handleEvents() events=25!
    2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] HttpIOLink::suspendRead()...
    2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Close socket ...
    2012-04-01 08:04:13.530 [DEBUG] [37.54.212.131:63207-0#APVH_redacteddomain.com] Available Connections: 2000, concurrent conn: 0
    2012-04-01 08:04:13.530 [NOTICE] [Child4951Shut down successfully!
    2012-04-01 08:04:13.531 [INFOPermission deniedlet parent kill it5404
    2012
    -04-01 08:04:13.531 [INFO] [AdminPHPkill pid5404
    2012
    -04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4934000
    2012
    -04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4864000
    2012
    -04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe48e1000
    2012
    -04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4a75000
    2012
    -04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4ab8000
    2012
    -04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4a6e000
    2012
    -04-01 08:04:13.532 [DEBUG] [MMAPRelease mapped data at 0x7f9fe492d000

    .. the release mapped data error goes on for 2 pages…..

    2012-04-01 08:04:13.537 [DEBUG] [MMAPRelease mapped data at 0x7f9fe48ac000
    2012
    -04-01 08:04:13.537 [DEBUG] [MMAPRelease mapped data at 0x7f9fe4a67000
    2012
    -04-01 08:04:13.541 [NOTICE] [AutoRestartercleanup children processes and unix sockets of process with pid=4951 !
    2012-04-01 08:04:13.541 [INFO] [CLEANUPClean up child process with pid5404
    2012
    -04-01 08:04:13.641 [NOTICE] [AutoRestarterchild process with pid=4951 exited with status=0!
    2012-04-01 08:04:50.532 [NOTICE] [Child5413Shut down successfully!
    2012-04-01 08:04:50.538 [NOTICE] [AutoRestartercleanup children processes and unix sockets of process with pid=5413 !
    2012-04-01 08:04:50.539 [INFO] [CLEANUPClean up child process with pid5515
    2012
    -04-01 08:04:50.639 [NOTICE] [AutoRestarterchild process with pid=5413 exited with status=0!
    2012-04-01 08:04:50.639 [NOTICE] [PID:32577Server Stopped!
    2012-04-01 08:05:01.564 [INFO] [127.0.0.1:59670-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
    2012-04-01 08:10:01.307 [INFO] [127.0.0.1:36268-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
    2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacteddomain.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, e
    rrorConnection reset by peer!
    2012-04-01 08:15:02.013 [INFO] [127.0.0.1:40754-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
    2012-04-01 08:16:09.295 [INFORemove pid6659exitcode0
    2012
    -04-01 08:20:01.709 [INFO] [127.0.0.1:45336-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
    2012-04-01 08:21:54.621 [INFO] [HTAccessUpdating configuration from [/home/redactedusername/public_html/wp-admin/.htaccess]
    2012-04-01 08:21:57.058 [INFO] [HTAccessUpdating configuration from [/home/redactedusername/public_html/images/.htaccess]
    2012-04-01 08:25:01.473 [INFO] [127.0.0.1:50243-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]
    2012-04-01 08:30:01.178 [INFO] [127.0.0.1:55391-0#APVH_98.158.153.71_redacted.hostname.com] File not found [/usr/local/apache/htdocs/whm-server-status]

    Here is a copy of the .rtreport:

    PHP:


    root
    @cr1 [/tmp/lshttpd]# more .rtreport
    VERSIONLiteSpeed Web Server/Enterprise/4.1.11
    UPTIME
    00:42:56
    BPS_IN
    62BPS_OUT789SSL_BPS_IN0SSL_BPS_OUT0
    MAXCONN
    2000MAXSSL_CONN500PLAINCONN113AVAILCONN1887IDLECONN107SSLCONN0AVAILSSL500
    REQ_RATE 
    []: REQ_PROCESSING6REQ_PER_SEC63.0TOT_REQS108634CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
    REQ_RATE 
    [APVH_*_redacted.hostname.com]: REQ_PROCESSING0REQ_PER_SEC0.0TOT_REQS0CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
    REQ_RATE 
    [APVH_redacted.hostname.com]: REQ_PROCESSING0REQ_PER_SEC0.0TOT_REQS0CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
    REQ_RATE 
    [_AdminVHost]: REQ_PROCESSING0REQ_PER_SEC0.0TOT_REQS8CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
    REQ_RATE 
    [APVH_98.158.153.71_redacted.hostname.com]: REQ_PROCESSING0REQ_PER_SEC0.0TOT_REQS0CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
    REQ_RATE 
    [APVH_redacteddomain.com]: REQ_PROCESSING5REQ_PER_SEC63.0TOT_REQS108620CACHE_HITS_PER_SEC0.0TOTAL_CACHE_HITS0
    EXTAPP 
    [LSAPI] [] [lsphp5]: CMAXCONN35EMAXCONN35POOL_SIZE10INUSE_CONN2IDLE_CONN8WAITQUE_DEPTH0REQ_PER_SEC18.4TOT_REQS35842
    BLOCKED_IP

    EOF
    root
    @cr1 [/tmp/lshttpd]

    I'm also getting these connection reset by peer errors every few mintes:
    PHP:
    2012-04-01 08:13:16.130 [INFO] [50.58.204.5:63997-0#APVH_redacted.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #61, confirmed, 0, associated process: 5618, running: 1, error: Connection reset by peer!
    Any help would be appreciated!
    Last edited: Apr 1, 2012
  2. NiteWave

    NiteWave Administrator

    run dmesg, see if see many "nf_conntrack: table full, dropping packet"
  3. hd-sam

    hd-sam Member

    Thanks for the fast response NiteWave.

    There are none of those in dmesg.

    It does show about 2 pages of entries from the CSF firewall of UDP_IN and TCP_IN blocked, here's a small ex:

    PHP:
    Firewall: *UDP_IN BlockedIN=eth0 OUTMAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.135.203.237 DST=R.E.D.A.C.T.E.DLEN=364 TOS=0x00 PREC=0x00 TTL=114 ID=28681 PROTO=UDP SPT=500
     DPT
    =500 LEN=344 
    Firewall
    : *UDP_IN BlockedIN=eth0 OUTMAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.135.203.237 DST=R.E.D.A.C.T.E.DLEN=316 TOS=0x00 PREC=0x00 TTL=114 ID=28682 PROTO=UDP SPT=500
     DPT
    =500 LEN=296 
    Firewall
    : *UDP_IN BlockedIN=eth0 OUTMAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=50.22.71.4 DST=R.E.D.A.C.T.E.DLEN=76 TOS=0x00 PREC=0x00 TTL=55 ID=45473 DF PROTO=UDP SPT=53 DPT
    =19203 LEN=56 
    Firewall
    : *UDP_IN BlockedIN=eth0 OUTMAC=5e:92:9b:0b:03:d7:00:1b:0d:ee:be:00:08:00 SRC=208.72.12.115 DST=R.E.D.A.C.T.E.DLEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=267 PROTO=UDP SPT=13396 DPT
    =33436 LEN=12 
  4. webizen

    webizen New Member

    looks like php script is crashed. make sure matching php is built.
  5. hd-sam

    hd-sam Member

    Thanks, I've actually done that about 5x now... with 4.1.10 and 4.1.11 both. That error has appeared in both versions.

    I'm running PHP 5.2.17
  6. hd-sam

    hd-sam Member

    One other thing I noticed....

    Every time I empty the page cache using the W3 Total Cache plugin in WordPress, the web server appears to restart. An important thing to note is my config is with PHP SuExec disabled.

    I get this email every time I empty the cache:

    PHP:
    At [01/Apr/2012:16:42:27 -0400], web server with pid=32272 received unexpected signal=11a core file is creatednew instance of web server will be started automatically!

    Please forward the following debug information to bug@litespeedtech.com.
    Environment:

    ServerLiteSpeed/4.1.11 Enterprise
    OS
    Linux
    Release
    2.6.32-220.7.1.el6.x86_64
    Version
    #1 SMP Wed Mar 7 00:52:02 GMT 2012
    Machinex86_64

    If the call stack information does not show up hereplease compress and forward the core file located in /tmp/lshttpd/.

    [New 
    Thread 32272]
    [New 
    Thread 32274]
    [New 
    Thread 32273]
    Core was generated by `litespeed'.
    Program terminated with signal 11, Segmentation fault.
    #0  0x0000000000457bb3 in ?? ()
    #0  0x0000000000457bb3 in ?? ()
    #1  0x000000000081bd63 in ?? ()
    #2  0x0000000000000000 in ?? ()
  7. NiteWave

    NiteWave Administrator

    have you tried wp-super cache instead of W3-total cache?
    if wp-super cache works well, then the issue is narrowed down.
  8. hd-sam

    hd-sam Member

    Thanks NiteWave,

    I was able to fix the issue regarding the crash every time I emptied the cache. I rebuilt Apache/PHP with easyapache, then built matching PHP w/ LS.
    That seemed to have fixed that portion of the issue.

    However, the remaining issue now appears to be this:
    2012-04-02 12:47:25.618 [INFO] [97.75.77.133:58306-0#APVH_redacted.com]
    connection to [/tmp/lshttpd/lsphp5.sock.615] on request #500, confirmed, 0, associated process: 29101, running: 1, error: Connection reset by peer!

    This appears every 5 minutes in the error log. Exactly the same error (on request #500)
    I've disabled APC and XCache as well.
  9. hd-sam

    hd-sam Member

    I just looked at my access-logs to see what these IPs are accessing that is causing that. It seems to be timthumb.php is causing that error.

    The timthumb.php file is the latest version 2.8.10.
  10. webizen

    webizen New Member

    It is due to your lsphp5 external app environment variable:
    Code:
    PHP_LSAPI_MAX_REQUESTS=500
    
    That is NOT a problem (even with "error: Connection reset by peer!" signature). LSPHP5 parent will spawn another child to take it over.
  11. hd-sam

    hd-sam Member

    Excellent, thank you. That is good to know. Would you recommend I change the that number or leave it?

    Another important thing to note we have litespeed installed on many servers (all are cpanel servers), some of them used for shared hosting, some of them for dedicated sites. While I don't have issues with them, I've noticed every single one is autorestarting.

    On every single one, the Current Uptime is under 1 hour (checked via Real-Time Stats in LS Web Admin). I've never seen it go more than 2 hours. They are all running 4.1.11 or 4.1.10.

    Is that normal too?
    Is anyone else experiencing the same thing?
  12. webizen

    webizen New Member

    1. You can leave php_lsapi_max_requests=500 as is. As pointed out, that's not a problem.
    2. For self restart every hour, pls check whm => service configuration, make sure apache is disabled. also check error log (i.e. /usr/local/apache/logs/error_log) for details of restart.

Share This Page