Pages Hanging After Server Migration

[bh-tech]

We have recently migrated from CentOS5 to CentOS7.

Our current LiteSpeed Version is 5.1.12, and we also use CloudLinux to use an older version of PHP.

For the most part everything is working perfectly, and the server is responding very fast.

However, occasionally the server will just stop responding. When you try to load a page, it will just hang and continuously load. The only solution so far has been to restart LiteSpeed, in some cases we need to restart LiteSpeed twice before the server will load pages again.

There is a line in STDERR log that seems to line up with the issue, and it reads:

[STDERR] Reached max children process limit: 10, extra: 3, current: 13, please increase LSAPI_CHILDREN.

Looking under LS Admin > Server > External App > lsphp5 > Environment, we have these settings:

PHP_LSAPI_MAX_REQUESTS=5000
PHP_LSAPI_CHILDREN=35

Is there somewhere else I need to increase the LSAPI_CHILDREN limit?

This is what is in our error_log around the same, there are many of these that all look very similar.

2017-02-15 07:30:16.570 [INFO] [123.45.647.89:2949] Connection idle time: 121 while in state: 6 stream flag: 0,close!
2017-02-15 07:30:16.570 [NOTICE] [123.45.647.89:2949] Content len: 0, Request line: 'GET /forums/showthread.php?123456-Embarrassing-moments-when-you-ve-done-a-Gary/page17 HTTP/1.1'
2017-02-15 07:30:16.570 [INFO] [123.45.647.89:2949] Cookie len: 313, bb_lastvisit=1486664710; bb_lastactivity=0; bb_forum_view=88162576a8efd707c7996c274f222a5efe608353a-1-%7Bi-199_i-1486677994_%7D; bb_sessionhash=5865fb613d
9f88827c25c65e2236fa90; bb_thread_lastview=3b98a16bfbeaaf1a2b1bd28d6193163729638339a-3-%7Bi-559835_i-1486691650_i-559747_i-1486683716_i-447786_i-1486793649_%7D
2017-02-15 07:30:16.570 [INFO] [123.45.647.89:2949] HttpExtConnector state: 8, request body sent: 0, response body size: -2, response body sent:0, left in buffer: 0, attempts: 0.
2017-02-15 07:30:16.570 [INFO] [123.45.647.89:2949] Lsapi connection state: 3, watching event: 25, respState: 1, m_iTotalPending:0, m_iPacketLeft:0, m_iCurRespHeader:2, req sent for 56 seconds,Total processing time: 56.
2017-02-15 07:30:16.570 [INFO] [123.45.647.89:2949] abort request..., code: 4
2017-02-15 07:30:16.570 [INFO] [123.45.647.89:2949] Abort request processing by PID:83579, kill: 1, begin time: 56, sent time: 56, req processed: 5

Would love some help as this issue is effecting our site tremendously.

Thanks in advance

 

[NiteWave]

for php suExec, that error message mean
lsws web admin -> Server -> General -> PHP suEXEC Max Conn,
for php non-suExec, it does mean PHP_LSAPI_CHILDREN
so please increase "PHP suEXEC Max Conn" and watch real-time stats -> WaitQ, it should keep 0 or non-0 for very short period.

 

[bh-tech]

We do use PHP suExec - so I guess that's why the LSAPI_CHILDREN does nothing.

I have updated PHP suEXEC Max Conn to 20. So far the issue hasn't happened again - but will be sure about a few days.

Would reaching the max limit cause the server to hang? Like it would cause pages not to load?

 

[NiteWave]

server won't hang, but new requests will put in wait queue, so you'll notice WaitQ > 0.
the user will wait for a bit longer since their requests may be waiting at server side.

 

[bh-tech]

UPDATE

I've just had the issue happen again - right as I was making that last post.

This time I didn't get the "Reached max children process limit" message though.

I got a screenshot of the Real-Time Stats page when it happened though. Attached.

I can see the WaitQ column down the bottom is at 110 and the Max CONN of the EA has been reached. Is that the issue?

 

[NiteWave]

yes, as stated, WaitQ should keep as 0 or non-0 should drop to 0 quickly.
other wise, need optimize your php/mysql to make the response time shorter and make it less happen.

you can see that busy site has higher possibility to have WaitQ > 0.

keep increasing it to see if any improvement, but remember it should be lower than cloudlinux's EP limit(Limit on entry processes).

 

[bh-tech]

For our server, 95% of the time the 'Req in Processing' is under 10.

So the figures in that screenshot are not normal at all.

It looks as though there was a very big spike in the number of requests at one time, so the requests got backed up. However looking through domlogs, there isn't really any evidence of a massive increase of requests to the server. So the reason for the higher Req in Processing is a bit of a mystery.

I have increased the PHP suEXEC Max Conn again and will continue to monitor the site. The updated figure is still lower than that of the CloudLinux EP Limit.