no end of trouble - 504 Gateway Timeout errors daily

eva2000

eva2000

Well-Known Member
#1
#1
Lately every day been getting 504 gateway timeout errors on both load balanced web1 and web2 litespeed servers got this on 4.0.12 and 4.0.13 enterprise

Also has WHM/Cpanel installed = cPanel 11.25.0-R43473 - WHM 11.25.0 - X 3.9
CENTOS 5.4 x86_64 standard on web1

Load balancer - 2x shared haproxy load balancers

I noticed LSAPI complaining of runaway processes in error log as well since then removed eaccelerator from PHP on both servers and upgraded from LSAPI 4.1 to 5.3 (web host did this)

2010-02-27 06:02:17.016 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.822].
2010-02-27 06:02:17.016 [INFO] [APVH_username_Suphp:] pid list size: 10
2010-02-27 06:02:17.016 [INFO] [89.xxx.xx.xxx:49382-0#APVH_forums.mydomain.com] add child process pid: 20817
2010-02-27 06:02:17.917 [INFO] [CLEANUP] Clean up child process with pid: 20609
2010-02-27 06:02:17.921 [INFO] Remove pid: 20609
2010-02-27 06:02:17.921 [INFO] Pid: 20609 associated with [APVH_username_Suphp:]
2010-02-27 06:02:17.921 [INFO] ProcInfo Pid: 20817, socket: /tmp/lshttpd/APVH_mydomain.com_Suphp.sock.822
2010-02-27 06:02:17.921 [INFO] [APVH_username_Suphp:] pid list size: 10, pid stop list size: 0
2010-02-27 06:02:59.055 [NOTICE] [62.xxx.xxx.xxx:49741-0#APVH_forums.mydomain.com] No request delivery notification has been received from LSAPI process:20684, possible run away process.
2010-02-27 06:02:59.055 [NOTICE] [62.xxx.xxx.xxx:49741-0#APVH_forums.mydomain.com] Retry with new instance.
2010-02-27 06:02:59.055 [INFO] Permission denied, let parent kill it: 20684
2010-02-27 06:02:59.055 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.791].
Click to expand...
 
Last edited:
eva2000

eva2000

Well-Known Member
#2
#2
when received 504 gateway timeout, after trying restarting litespeed still won't come back up error logs

web2

root@web2 [/usr/local/apache/logs]# tail -50 error_log
2010-02-28 03:07:37.257 [INFO] Stop listener web2internalipaddress:443.
2010-02-28 03:07:37.257 [INFO] Stop listener web2internalipaddress:80.
2010-02-28 03:07:37.257 [INFO] Stop listener 9x.xx.xx159:443.
2010-02-28 03:07:37.257 [INFO] Stop listener 9x.xx.xx159:80.
2010-02-28 03:07:37.257 [NOTICE] [Child: 6769] Shut down successfully!
2010-02-28 03:07:37.258 [NOTICE] New lshttpd process is ready, lshttpd (6768) stops listeners
2010-02-28 03:07:37.258 [INFO] Stop listener 127.0.0.1:443.
2010-02-28 03:07:37.258 [NOTICE] New lshttpd process is ready, lshttpd (6770) stops listeners
2010-02-28 03:07:37.258 [INFO] Stop listener 127.0.0.1:80.
2010-02-28 03:07:37.258 [INFO] Stop listener 9x.xx.xx.92:443.
2010-02-28 03:07:37.258 [INFO] Stop listener 9x.xx.xx158:443.
2010-02-28 03:07:37.258 [INFO] Stop listener 9x.xx.xx.92:80.
2010-02-28 03:07:37.258 [INFO] Stop listener *:7080.
2010-02-28 03:07:37.258 [INFO] Stop listener 9x.xx.xx156:80.
2010-02-28 03:07:37.258 [NOTICE] [Child: 6768] Shut down successfully!
2010-02-28 03:07:37.259 [NOTICE] New lshttpd process is ready, lshttpd (6771) stops listeners
2010-02-28 03:07:37.259 [INFO] Stop listener 9x.xx.xx157:443.
2010-02-28 03:07:37.259 [INFO] Stop listener 9x.xx.xx157:80.
2010-02-28 03:07:37.259 [NOTICE] [Child: 6771] Shut down successfully!
2010-02-28 03:07:37.260 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets belong to process 6769 !
2010-02-28 03:07:37.260 [NOTICE] [AutoRestarter] child process with pid=6769 exited with status=0!
2010-02-28 03:07:37.260 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets belong to process 6768 !
2010-02-28 03:07:37.261 [NOTICE] [AutoRestarter] child process with pid=6768 exited with status=0!
2010-02-28 03:07:37.261 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets belong to process 6771 !
2010-02-28 03:07:37.273 [INFO] [HTAccess] Updating configuration from [/home/username/public_html/forums/.htaccess]
2010-02-28 03:07:37.274 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock].
2010-02-28 03:07:37.274 [INFO] [APVH_username_Suphp:] pid list size: 1
2010-02-28 03:07:37.274 [INFO] [APVH_username_Suphp:] add child process pid: 6961
2010-02-28 03:07:37.361 [NOTICE] [AutoRestarter] child process with pid=6771 exited with status=0!
2010-02-28 03:07:37.538 [INFO] [HTAccess] Updating configuration from [/home/username/public_html/.htaccess]
2010-02-28 03:07:37.733 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.758].
2010-02-28 03:07:37.733 [INFO] [APVH_username_Suphp:] pid list size: 2
2010-02-28 03:07:37.733 [INFO] [APVH_username_Suphp:] add child process pid: 6962
2010-02-28 03:07:37.975 [INFO] [HTAccess] Updating configuration from [/home/username/public_html/files/.htaccess]
2010-02-28 03:07:38.029 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.602].
2010-02-28 03:07:38.030 [INFO] [APVH_username_Suphp:] pid list size: 3
2010-02-28 03:07:38.030 [INFO] [APVH_username_Suphp:] add child process pid: 6969
2010-02-28 03:07:38.404 [NOTICE] [AutoUpdate] Checking for new releases...
2010-02-28 03:07:38.703 [INFO] [AutoUpdate] No new update.
2010-02-28 03:07:39.186 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.798].
2010-02-28 03:07:39.187 [INFO] [APVH_username_Suphp:] pid list size: 4
2010-02-28 03:07:39.187 [INFO] [APVH_username_Suphp:] add child process pid: 6971
2010-02-28 03:07:39.840 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.042].
2010-02-28 03:07:39.840 [INFO] [APVH_username_Suphp:] pid list size: 5
2010-02-28 03:07:39.840 [INFO] [APVH_username_Suphp:] add child process pid: 6972
2010-02-28 03:07:39.901 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.129].
2010-02-28 03:07:39.901 [INFO] [APVH_username_Suphp:] pid list size: 6
2010-02-28 03:07:39.901 [INFO] [APVH_username_Suphp:] add child process pid: 6973
root@web2 [/usr/local/apache/logs]# date
Sun Feb 28 03:07:43 CST 2010
Click to expand...

web1

root@web1 [/usr/local/apache/logs]# tail -50 error_log
2010-02-28 03:07:50.533 [INFO] [AutoUpdate] No new update.
2010-02-28 03:07:51.086 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.228].
2010-02-28 03:07:51.086 [INFO] [APVH_username_Suphp:] pid list size: 4
2010-02-28 03:07:51.086 [INFO] [APVH_username_Suphp:] add child process pid: 12374
2010-02-28 03:07:51.648 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.273].
2010-02-28 03:07:51.648 [INFO] [APVH_username_Suphp:] pid list size: 5
2010-02-28 03:07:51.648 [INFO] [APVH_username_Suphp:] add child process pid: 12375
2010-02-28 03:07:51.730 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.687].
2010-02-28 03:07:51.730 [INFO] [APVH_username_Suphp:] pid list size: 6
2010-02-28 03:07:51.730 [INFO] [APVH_username_Suphp:] add child process pid: 12376
2010-02-28 03:07:52.650 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.263].
2010-02-28 03:07:52.650 [INFO] [APVH_username_Suphp:] pid list size: 7
2010-02-28 03:07:52.650 [INFO] [APVH_username_Suphp:] add child process pid: 12377
2010-02-28 03:07:52.668 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.358].
2010-02-28 03:07:52.669 [INFO] [APVH_username_Suphp:] pid list size: 8
2010-02-28 03:07:52.669 [INFO] [APVH_username_Suphp:] add child process pid: 12378
2010-02-28 03:07:52.710 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.401].
2010-02-28 03:07:52.710 [INFO] [APVH_username_Suphp:] pid list size: 9
2010-02-28 03:07:52.710 [INFO] [APVH_username_Suphp:] add child process pid: 12379
2010-02-28 03:07:53.239 [INFO] Start listen socket [/tmp/lshttpd/APVH_mydomain.com_Suphp.sock.387].
2010-02-28 03:07:53.239 [INFO] [APVH_username_Suphp:] pid list size: 10
2010-02-28 03:07:53.239 [INFO] [APVH_username_Suphp:] add child process pid: 12400
2010-02-28 03:07:53.769 [INFO] Start listen socket [/tmp/lshttpd/lsphp5.sock].
2010-02-28 03:07:53.770 [INFO] [lsphp5] pid list size: 1
2010-02-28 03:07:53.770 [INFO] [lsphp5] add child process pid: 12401
2010-02-28 03:07:55.030 [INFO] [HTAccess] Updating configuration from [/home/username/public_html/gallery/.htaccess]
2010-02-28 03:07:58.203 [INFO] [HTAccess] Updating configuration from [/home/username/public_html/files/.htaccess]
2010-02-28 03:08:02.779 [INFO] [HTAccess] Updating configuration from [/home/username/public_html/forums/tags/.htaccess]
2010-02-28 03:08:04.003 [INFO] [lsphp5] kill pid: 12401
2010-02-28 03:08:04.042 [INFO] Remove pid: 12401
2010-02-28 03:08:04.042 [INFO] Pid: 12401 associated with [lsphp5]
2010-02-28 03:08:04.042 [INFO] ProcInfo Pid: 12401, socket: /tmp/lshttpd/lsphp5.sock
2010-02-28 03:08:04.042 [INFO] [lsphp5] pid list size: 0, pid stop list size: 0
2010-02-28 03:08:20.145 [NOTICE] lshttpd (12214) stop lingering due to idle
2010-02-28 03:08:20.145 [NOTICE] [Child: 12214] Shut down successfully!
2010-02-28 03:08:20.145 [NOTICE] [APVH_username_Suphp:] stop worker processes
2010-02-28 03:08:20.145 [INFO] [APVH_username_Suphp:] 10 request being processed, kill external app later.
2010-02-28 03:08:20.147 [NOTICE] [AutoRestarter] cleanup children processes and unix sockets belong to process 12214 !
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12217
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12218
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12220
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12221
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12222
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12223
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12231
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12232
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12233
2010-02-28 03:08:20.148 [INFO] [CLEANUP] Clean up child process with pid: 12234
2010-02-28 03:08:20.249 [NOTICE] [AutoRestarter] child process with pid=12214 exited with status=0!
2010-02-28 03:08:20.249 [NOTICE] [PID:12211] Server Stopped!
Click to expand...
 
eva2000

eva2000

Well-Known Member
#3
#3
Also on web 1 get

010-02-28 03:19:41.431 [INFO] [HTAccess] Updating configuration from [/home/username/public_html/forums/tags/.htaccess]
2010-02-28 03:20:01.861 [INFO] [127.0.0.1:41724-0#APVH_9x.xx.xx.152_web1.mydomain.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2010-02-28 03:20:01.861 [INFO] [127.0.0.1:41724-0#APVH_9x.xx.xx.152_web1.mydomain.com] Cannot found appropriate handler for [/whm-server-status]
2010-02-28 03:20:01.865 [INFO] [127.0.0.1:41725-0#APVH_9x.xx.xx.152_web1.mydomain.com] File not found [/usr/local/apache/htdocs/whm-server-status]
2010-02-28 03:20:01.865 [INFO] [127.0.0.1:41725-0#APVH_9x.xx.xx.152_web1.mydomain.com] Cannot found appropriate handler for [/whm-server-status]
2010-02-28 03:20:17.544 [NOTICE] SIGTERM received, shutdown disabled, turn off fake Apache httpd.
Click to expand...
web1 also reports

2010-02-28 05:42:48.477 [NOTICE] Your License key will expire in 7 days
2010-02-28 05:42:48.477 [NOTICE] Loading LiteSpeed/4.0.13 Enterprise ...
Click to expand...
 
Last edited:
M

mistwang

LiteSpeed Staff
#4
#4
eva2000 said:
2010-02-28 03:20:01.865 [INFO] [127.0.0.1:41725-0#APVH_9x.xx.xx.152_web1.mydomain.com] Cannot found appropriate handler for [/whm-server-status]
2010-02-28 03:20:17.544 [NOTICE] SIGTERM received, shutdown disabled, turn off fake Apache httpd.
Click to expand...
Those does not matter, as designed.

When you get 504 timeout, can you access the web server directly, still get error?

what is your lsphp5 configuration?

You can try upgrade to the latest 4.0.13 build, it may work better.
 
You must log in or register to reply here.
Top