oom killer on litespeed processes

H

Havri

Member
#1
#1
Hello,

I keep finding these kinds of entries in /var/log/messages:

Code: 
May 18 11:05:17 entry01 kernel: oom_kill_process: 1 callbacks suppressed
May 18 11:05:17 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 18 11:05:17 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
May 18 11:05:17 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
May 18 11:05:17 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 90827
May 18 11:05:17 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:17 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:17 entry01 kernel: Memory cgroup stats for /lve1421: cache:160KB rss:524128KB rss_huge:442368KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:263884KB active_anon:260248KB inactive_file:16KB active_file:8KB unevictable:0KB
May 18 11:05:17 entry01 kernel: Memory cgroup out of memory: Kill process 32246 (lsphp) score 141 or sacrifice child
May 18 11:05:17 entry01 kernel: Killed process 32246 (lsphp) in VE "0" total-vm:415656kB, anon-rss:69556kB, file-rss:3944kB, shmem-rss:4kB
May 18 11:05:17 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 18 11:05:17 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
May 18 11:05:17 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
May 18 11:05:17 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 92291
May 18 11:05:17 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:17 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:17 entry01 kernel: Memory cgroup stats for /lve1421: cache:28KB rss:524260KB rss_huge:430080KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:263632KB active_anon:260632KB inactive_file:8KB active_file:16KB unevictable:0KB
May 18 11:05:17 entry01 kernel: Memory cgroup out of memory: Kill process 32250 (lsphp) score 145 or sacrifice child
May 18 11:05:17 entry01 kernel: Killed process 32250 (lsphp) in VE "0" total-vm:417720kB, anon-rss:71608kB, file-rss:3940kB, shmem-rss:4kB
May 18 11:05:18 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 18 11:05:18 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
May 18 11:05:18 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
May 18 11:05:18 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 93999
May 18 11:05:18 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:18 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:18 entry01 kernel: Memory cgroup stats for /lve1421: cache:32KB rss:524256KB rss_huge:407552KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:264108KB active_anon:260152KB inactive_file:8KB active_file:8KB unevictable:0KB
May 18 11:05:18 entry01 kernel: Memory cgroup out of memory: Kill process 32257 (lsphp) score 165 or sacrifice child
May 18 11:05:18 entry01 kernel: Killed process 32257 (lsphp) in VE "0" total-vm:430440kB, anon-rss:82260kB, file-rss:3960kB, shmem-rss:4kB
May 18 11:05:18 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 18 11:05:18 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
May 18 11:05:18 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
May 18 11:05:18 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 95211
May 18 11:05:18 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:18 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:18 entry01 kernel: Memory cgroup stats for /lve1421: cache:36KB rss:524252KB rss_huge:403456KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:268592KB active_anon:255664KB inactive_file:24KB active_file:8KB unevictable:0KB
May 18 11:05:18 entry01 kernel: Memory cgroup out of memory: Kill process 32258 (lsphp) score 121 or sacrifice child
May 18 11:05:18 entry01 kernel: Killed process 32258 (lsphp) in VE "0" total-vm:405352kB, anon-rss:59204kB, file-rss:3828kB, shmem-rss:4kB
May 18 11:05:19 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 18 11:05:19 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
May 18 11:05:19 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
May 18 11:05:19 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 97351
May 18 11:05:19 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:19 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:19 entry01 kernel: Memory cgroup stats for /lve1421: cache:28KB rss:524260KB rss_huge:397312KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:273228KB active_anon:251036KB inactive_file:8KB active_file:4KB unevictable:0KB
May 18 11:05:19 entry01 kernel: Memory cgroup out of memory: Kill process 32292 (lsphp) score 157 or sacrifice child
May 18 11:05:19 entry01 kernel: Killed process 32292 (lsphp) in VE "0" total-vm:426240kB, anon-rss:77972kB, file-rss:3924kB, shmem-rss:4kB
May 18 11:05:20 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 18 11:05:20 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
May 18 11:05:20 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
May 18 11:05:20 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 98512
May 18 11:05:20 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:20 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:20 entry01 kernel: Memory cgroup stats for /lve1421: cache:28KB rss:524260KB rss_huge:397312KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:292304KB active_anon:231960KB inactive_file:12KB active_file:12KB unevictable:0KB
May 18 11:05:20 entry01 kernel: Memory cgroup out of memory: Kill process 32251 (lsphp) score 136 or sacrifice child
May 18 11:05:20 entry01 kernel: Killed process 32251 (lsphp) in VE "0" total-vm:413620kB, anon-rss:66716kB, file-rss:3940kB, shmem-rss:4kB
May 18 11:05:20 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 18 11:05:20 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
May 18 11:05:20 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
May 18 11:05:20 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 100039
May 18 11:05:20 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:20 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 18 11:05:20 entry01 kernel: Memory cgroup stats for /lve1421: cache:32KB rss:524256KB rss_huge:399360KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:278656KB active_anon:245604KB inactive_file:12KB active_file:12KB unevictable:0KB
May 18 11:05:20 entry01 kernel: Memory cgroup out of memory: Kill process 32252 (lsphp) score 159 or sacrifice child
May 18 11:05:20 entry01 kernel: Killed process 32252 (lsphp) in VE "0" total-vm:426208kB, anon-rss:78764kB, file-rss:3880kB, shmem-rss:4kB

Right now, I have for all my External Apps Memory Soft Limit (bytes): 4096M and Memory Hard Limit (bytes): 4096M. I've also uploaded 2 screenshots. All the external apps have the same settings.

Currently running version 5.2.7, Build 5 of Litespeed. Also running cPanel 11.68.0.38.

Let me know if you need any other info.

Thank you. DeepinScreenshot_select-area_20180518111307.png DeepinScreenshot_select-area_20180518111259.png
 
H

Havri

Member
#5
#5
Hello,

Thank you. I will increase the PMEM for all the users that have 512M and get back to you. I've also found other errors that look pretty much the same, but there isn't any specific LVE name/number. It's just a number:


Code: 
May 20 23:18:51 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 20 23:18:51 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
May 20 23:18:51 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
May 20 23:18:51 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 224006
May 20 23:18:51 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
May 20 23:18:51 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 20 23:18:51 my_server01 kernel: Memory cgroup stats for /1168: cache:16KB rss:1048560KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:279160KB active_anon:769404KB inactive_file:8KB active_file:4KB unevictable:0KB
May 20 23:18:51 my_server01 kernel: Memory cgroup out of memory: Kill process 9597 (lsphp) score 152 or sacrifice child
May 20 23:18:51 my_server01 kernel: Killed process 9597 (lsphp) in VE "0" total-vm:572956kB, anon-rss:153572kB, file-rss:5820kB, shmem-rss:4kB
May 20 23:15:02 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 20 23:15:02 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
May 20 23:15:02 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
May 20 23:15:02 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 222937
May 20 23:15:02 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
May 20 23:15:02 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 20 23:15:02 my_server01 kernel: Memory cgroup stats for /1168: cache:8KB rss:1048568KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:525060KB active_anon:523512KB inactive_file:0KB active_file:4KB unevictable:0KB
May 20 23:15:02 my_server01 kernel: Memory cgroup out of memory: Kill process 5554 (lsphp) score 122 or sacrifice child
May 20 23:15:02 my_server01 kernel: Killed process 5554 (lsphp) in VE "0" total-vm:467536kB, anon-rss:123824kB, file-rss:3776kB, shmem-rss:4kB
May 20 23:15:02 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 20 23:15:02 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
May 20 23:15:02 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
May 20 23:15:02 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 223039
May 20 23:15:02 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
May 20 23:15:02 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 20 23:15:02 my_server01 kernel: Memory cgroup stats for /1168: cache:8KB rss:1048568KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:497324KB active_anon:551124KB inactive_file:4KB active_file:0KB unevictable:0KB
May 20 23:15:02 my_server01 kernel: Memory cgroup out of memory: Kill process 5577 (lsphp) score 120 or sacrifice child
May 20 23:15:02 my_server01 kernel: Killed process 5577 (lsphp) in VE "0" total-vm:465244kB, anon-rss:121892kB, file-rss:3656kB, shmem-rss:4kB
May 20 23:15:03 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 20 23:15:03 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
May 20 23:15:03 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
May 20 23:15:03 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 223135
May 20 23:15:03 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
May 20 23:15:03 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 20 23:15:03 my_server01 kernel: Memory cgroup stats for /1168: cache:8KB rss:1048568KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:524512KB active_anon:524060KB inactive_file:0KB active_file:4KB unevictable:0KB
May 20 23:15:03 my_server01 kernel: Memory cgroup out of memory: Kill process 5580 (lsphp) score 121 or sacrifice child
May 20 23:15:03 my_server01 kernel: Killed process 5580 (lsphp) in VE "0" total-vm:465980kB, anon-rss:122476kB, file-rss:3840kB, shmem-rss:4kB
May 20 23:15:04 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 20 23:15:04 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
May 20 23:15:04 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
May 20 23:15:04 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 223310
May 20 23:15:04 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
May 20 23:15:04 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
May 20 23:15:04 my_server01 kernel: Memory cgroup stats for /1168: cache:8KB rss:1048568KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:524464KB active_anon:523980KB inactive_file:0KB active_file:4KB unevictable:0KB
May 20 23:15:04 my_server01 kernel: Memory cgroup out of memory: Kill process 5611 (lsphp) score 123 or sacrifice child
May 20 23:15:04 my_server01 kernel: Killed process 5611 (lsphp) in VE "0" total-vm:469340kB, anon-rss:125292kB, file-rss:3764kB, shmem-rss:4kB
What I want to know is if the "Memory cgroup stats for /1168:" represents the UID 1168 from /etc/passwd and should be interpreted the same as "Memory cgroup stats for /lve1421:" (note that there is no "lve" before the number).

So in these two examples, "Memory cgroup stats for /1168" represents LVE for UID 1168 and "Memory cgroup stats for /lve1421:" represents LVE for UID 1421. I already found the corresponding UID for these LVEs, but I just want to be sure.


Thank you.
 
You must log in or register to reply here.
Top