503 Errors - APC with cPanel matched PHP binary

[ziggy95]

After trying again we are not getting the 503 errors anymore. There was a problem where the database connection was routing through the network LAN vs the local LAN between the two boxes. I'm not sure how this would have played into things, but now that the connection is direct the errors aren't happenning.... yet.

Will keep you posted.

[mistwang]

You should use LAN for DB connection. always use the fastest available connection.

[ziggy95]

Originally Posted by mistwang

You should use LAN for DB connection. always use the fastest available connection.

Yes, of course we understand this. Someone had made a mistake and it was corrected.

However, we are back to the 503 errors. So far we have only moved a single low traffic site to the server. We are testing and near moving a very high traffic site (vbulletin forum) over.

I have tried setting memory in Security and External App to 3,000M. No good.

2009-02-19 13:17:31.799 [NOTICE] [68.83.66.8:49373-0#APVH_sitename.com] Redirect: #1, URL: /vbseo.php
2009-02-19 13:17:31.801 [INFO] [68.83.66.8:49373-0#APVH_sitename.com:lsapi] connection to [uds://tmp/lshttpd/lsphp5.sock] on request #0, Connection reset by peer!
2009-02-19 13:17:31.804 [INFO] [68.83.66.8:49373-0#APVH_sitename.com:lsapi] connection to [uds://tmp/lshttpd/lsphp5.sock] on request #0, Connection reset by peer!
2009-02-19 13:17:31.807 [INFO] [68.83.66.8:49373-0#APVH_sitename.com:lsapi] connection to [uds://tmp/lshttpd/lsphp5.sock] on request #0, Connection reset by peer!
2009-02-19 13:17:31.807 [NOTICE] [68.83.66.8:49373-0#APVH_sitename.com:lsapi] oops! 503 Service Unavailable

is what shows in error_log.

I guess we don't have to have APC, though it would likely help.

Any advice?

[mistwang]

Maybe you have set process limits of lsphp5 too low?
check how many lsphp5 processes do you get?

I think for some reason, APC crashes PHP, maybe look into other cache solutions like eAccelerator or xcache.

[ziggy95]

The processes are set at default 200/200. I will try xcache instead. I've used them both before with comparable results.

[mistwang]

Since you do not use PHP suEXEC, you can try strace the parent lsphp5 process with command

strace -f -p <pid_of_lsphp5>

when you get 503 errors. It can give you some hints on why PHP crashed for some reason.

[ziggy95]

I installed and configured xcache. Within seconds of restarting the server, the 503 errors come back. I setting max processes to 0/0, no luck. Here is the result of strace:

# strace -f -p 16225
Process 16225 attached - interrupt to quit
select(6, [5], NULL, NULL, {0, 853000}) = 0 (Timeout)
getppid() = 16224
select(6, [5], NULL, NULL, {1, 0}) = 1 (in [5], left {0, 226000})
read(5, "LS\1\0\355\6\0\0\315\3\0\0\0\0\0\0\201\1\0\0\372\ 1\0\0\265\1\0\0n\2\0\0"..., 8192) = 1773
write(5, "LS\7\0\10\0\0\0", 8) = 8
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0
rt_sigaction(SIGPROF, {0x685d80, [PROF], SA_RESTORER|SA_RESTART, 0x3ed4e301b0}, {0x685d80, [PROF], SA_RESTORER|SA_RESTART, 0x3ed4e301b0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
getcwd("/usr/local/lsws/fcgi-bin", 4095) = 25
chdir("/home/sitename/public_html") = 0
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={30, 0}}, NULL) = 0
rt_sigaction(SIGPROF, {0x685d80, [PROF], SA_RESTORER|SA_RESTART, 0x3ed4e301b0}, {0x685d80, [PROF], SA_RESTORER|SA_RESTART, 0x3ed4e301b0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
open("/home/sitename/public_html/vbseo.php", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0644, st_size=48445, ...}) = 0
mmap(NULL, 48445, PROT_READ, MAP_SHARED, 6, 0) = 0x2aaaafef6000
munmap(0x2aaaafef6000, 48445) = 0
close(6) = 0
open("/home/sitename/public_html/vbseo.php", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0644, st_size=48445, ...}) = 0
fcntl(6, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(6, {st_mode=S_IFREG|0644, st_size=48445, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaafef6000
lseek(6, 0, SEEK_CUR) = 0
lseek(6, 0, SEEK_SET) = 0
read(6, "<?php\n\n/************************"..., 4096) = 4096
lseek(6, 4096, SEEK_SET) = 4096
stat("/home/sitename/public_html/vbseo.php", {st_mode=S_IFREG|0644, st_size=48445, ...}) = 0
fcntl(1, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(1, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
lseek(6, -4096, SEEK_CUR) = 0
close(6) = 0
munmap(0x2aaaafef6000, 4096) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 16225 detached

[mistwang]

Which version of PHP is used?
Maybe you can try downgrade PHP version.

How about Apache mod_php? will APC or Xcache crash it?
The problem is between PHP and opcode cache, you can try to let lsphp5 dump core file and exam the core file with GDB.

To let lsphp5 dump core file, you need to add environment variable to lsphp5 external application

LSAPI_ALLOW_CORE_DUMP=1

[ziggy95]

PHP is 5.2.8 as stated. I'm not familiar with examining the core dump. I would need more help on that.

[mistwang]

Once you got the core file, just run

gdb <path_to_lsphp5> <core_file>

then type command

bt

to list the call stack.