[Resolved] Out of memory errors in Admin interface

[toki]

Hi,

I'm facing out-of-memory errors when accessing the Admin interface with a large complex config.

I've tried increasing the value of memory_limit in /usr/local/lsws/admin/conf/php.ini from 200M to 2048M. This allowed me to log in, and now the Configuration and WebAdmin Console links work. But the "General" link (http://localhost:7080/service/serviceMgr.php) doesn't. I see errors such as the following in my error log:

2015-12-08 13:25:49.959 [NOTICE] [127.0.0.1:38578-1#_AdminVHost:lsapi] [STDERR] PHP Fatal error:  Out of memory (allocated 462684160) (tried to allocate 33292288 bytes) in /usr/local/lsws/admin/html.5.0.9/service/homeCont.php on line 118
2015-12-08 13:33:02.347 [NOTICE] [127.0.0.1:38591] [STDERR] PHP Fatal error:  Out of memory (allocated 462684160) (tried to allocate 33292288 bytes) in /usr/local/lsws/admin/html.5.0.9/service/homeCont.php on line 118
2015-12-08 13:34:31.632 [NOTICE] [127.0.0.1:38593-1#_AdminVHost:lsapi] [STDERR] PHP Fatal error:  Out of memory (allocated 462684160) (tried to allocate 33292288 bytes) in /usr/local/lsws/admin/html.5.0.9/service/homeCont.php on line 118

Any advice to fix this would be appreciated.

Cheers,
Toki

 

[NiteWave]

how about 2000M or 1500M ?

 

[toki]

Same I've tried with 1024M, 1500M, 2000M, 2048M even 4096M.

2015-12-08 14:04:02.432 [NOTICE] [127.0.0.1:38640] [STDERR] PHP Fatal error: Out of memory (allocated 461373440) (tried to allocate 35487744 bytes) in Unknown on line 0

The box has 16GB RAM.

 

[toki]

It, for some reason, seems to be capping itself at ~440MB:

2015-12-08 14:10:45.140 [NOTICE] [127.0.0.1:38663-3#_AdminVHost:lsapi] [STDERR] PHP Fatal error: Out of memory (allocated 461373440) (tried to allocate 35487744 bytes) in Unknown on line 0
root@litespeed-1:/usr/local/lsws/admin/conf# echo "461373440 / 1024 / 1024" | bc
440

 

[toki]

Some strace output on the admin_php5 process

mmap(NULL, 33558528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0x20834000)  = 0x1e834000
mmap(NULL, 33689600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 33558528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0x20834000)  = 0x1e834000
mmap(NULL, 33689600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
writev(5, [{"LS\6\0\246\0\0\0", 8}, {"PHP Fatal error:  Out of memory "..., 158}], 2) = 166

 

[NiteWave]

edit /usr/local/lsws/conf/httpd_config.xml, search
<memSoftLimit>250M</memSoftLimit>
<memHardLimit>300M</memHardLimit>
<memSoftLimit>450M</memSoftLimit>
<memHardLimit>500M</memHardLimit>

you can modify memHardLimit first one by one, restart lsws, then see if works.
if fixed, identify which line has taken effect.

from the error log message, it looks your admin_php5 process is limited by 500M.

 

[toki]

Thanks, but we only impose mem(Soft|Hard)Limit for railsDefaults and CGIRLimit.

Changing either of these values doesn't fix it - I still get the same message in the error log and a blank page is returned.

 

[NiteWave]

but we only impose mem(Soft|Hard)Limit for railsDefaults and CGIRLimit.

in your httpd_config.xml, is there configuration of lsphp5 like following ?
<extProcessor>
<type>lsapi</type>
<name>lsphp5</name>
<address>uds://tmp/lshttpd/lsphp5.sock</address>
<maxConns>35</maxConns>
<env>PHP_LSAPI_MAX_REQUESTS=5000</env>
<env>PHP_LSAPI_CHILDREN=35</env>
<initTimeout>60</initTimeout>
<retryTimeout>0</retryTimeout>
<persistConn>1</persistConn>
<respBuffer>0</respBuffer>
<autoStart>1</autoStart>
<path>$SERVER_ROOT/fcgi-bin/lsphp5</path>
<backlog>100</backlog>
<instances>1</instances>
<priority>0</priority>
<memSoftLimit>450M</memSoftLimit>
<memHardLimit>500M</memHardLimit>
<procSoftLimit>200</procSoftLimit>
<procHardLimit>200</procHardLimit>
</extProcessor>

 

[toki]

There isn't:

root@litespeed-1:/usr/local/lsws/conf# grep lsphp5 httpd_config.xml
root@litespeed-1:/usr/local/lsws/conf#
root@litespeed-1:/usr/local/lsws/conf# grep lsphp5.sock httpd_config.xml
root@litespeed-1:/usr/local/lsws/conf#

root@litespeed-1:/usr/local/lsws/conf# grep mem.*Limit[^/] httpd_config.xml
<memSoftLimit>250M</memSoftLimit>
<memHardLimit>300M</memHardLimit>
<memSoftLimit>550M</memSoftLimit>
<memHardLimit>600M</memHardLimit>

The top two are for CGI, the bottom two are for Rails.

It's worth noting that most of the admin interface works fine, it's just the "General" page that hits this issue. Doesn't the admin interface use the admin_php5 binary anyway?

Our PHP is handled by an external lsapi PHP binary and we don't have limits set:

<extProcessor>
<type>lsapi</type>
<name>php52</name>
<address>192.168.100.5:5201</address>
<note/>
<maxConns>300</maxConns>
<initTimeout>90</initTimeout>
<retryTimeout>1</retryTimeout>
<persistConn>1</persistConn>
<pcKeepAliveTimeout>10</pcKeepAliveTimeout>
<respBuffer>1</respBuffer>
<autoStart>0</autoStart>
<path/>
<backlog/>
<instances>1</instances>
<extUser/>
<extGroup/>
<runOnStartUp>0</runOnStartUp>
<extMaxIdleTime/>
<priority/>
<memSoftLimit/>
<memHardLimit/>
<procSoftLimit/>
<procHardLimit/>
</extProcessor>
...
<scriptHandlerList>
<scriptHandler>
<suffix>php, php5</suffix>
<type>lsapi</type>
<handler>php52</handler>
<note/>
</scriptHandler>
</scriptHandlerList>

 

[NiteWave]

it's just the "General" page that hits this issue.

you can add back the settings I posted, and increase the default
<memSoftLimit>450M</memSoftLimit>
<memHardLimit>500M</memHardLimit>
to see if it can resolve the "Out of memory" issue.

or add
<memSoftLimit>850M</memSoftLimit>
<memHardLimit>900M</memHardLimit>
to php52

anyway, please try first, it won't hurt anything, you can revert back anytime.

 

[toki]

Okay, I tried adding the settings to php52, same result.

I added the extProcessor for lsphp5 as you recommended, same result (increased soft limit to 850M and hard limit to 900M)

2015-12-08 16:04:18.809 [NOTICE] [127.0.0.1:38738-2#_AdminVHost:lsapi] [STDERR] PHP Fatal error: Out of memory (allocated 461635584) (tried to allocate 35483648 bytes) in Unknown on line 0

 

[NiteWave]

can you also try modify the memory limit under CGI ?

 

[toki]

Increased soft to 512M and hard to 1024M for CGIRLimit and restarted - same issue.

I've logged a support ticket via the Client Area for this now, as it is impacting a critical migration.

Appreciate your assistance in resolving this ASAP.

 

[mistwang]

The memory limit for admin_php is hard coded. please try the updated 5.0.9 package with

/usr/local/lsws/admin/misc/lsup.sh -d -f -v 5.0.9

 

[toki]

Perfect - that has fixed the issue.

Thanks!