oom killer on litespeed processes

Discussion in 'General' started by Havri, May 18, 2018 at 9:15 AM.

  1. Havri

    Havri Member

    Hello,

    I keep finding these kinds of entries in /var/log/messages:

    Code:
    May 18 11:05:17 entry01 kernel: oom_kill_process: 1 callbacks suppressed
    May 18 11:05:17 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 18 11:05:17 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
    May 18 11:05:17 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
    May 18 11:05:17 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 90827
    May 18 11:05:17 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:17 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:17 entry01 kernel: Memory cgroup stats for /lve1421: cache:160KB rss:524128KB rss_huge:442368KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:263884KB active_anon:260248KB inactive_file:16KB active_file:8KB unevictable:0KB
    May 18 11:05:17 entry01 kernel: Memory cgroup out of memory: Kill process 32246 (lsphp) score 141 or sacrifice child
    May 18 11:05:17 entry01 kernel: Killed process 32246 (lsphp) in VE "0" total-vm:415656kB, anon-rss:69556kB, file-rss:3944kB, shmem-rss:4kB
    May 18 11:05:17 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 18 11:05:17 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
    May 18 11:05:17 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
    May 18 11:05:17 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 92291
    May 18 11:05:17 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:17 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:17 entry01 kernel: Memory cgroup stats for /lve1421: cache:28KB rss:524260KB rss_huge:430080KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:263632KB active_anon:260632KB inactive_file:8KB active_file:16KB unevictable:0KB
    May 18 11:05:17 entry01 kernel: Memory cgroup out of memory: Kill process 32250 (lsphp) score 145 or sacrifice child
    May 18 11:05:17 entry01 kernel: Killed process 32250 (lsphp) in VE "0" total-vm:417720kB, anon-rss:71608kB, file-rss:3940kB, shmem-rss:4kB
    May 18 11:05:18 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 18 11:05:18 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
    May 18 11:05:18 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
    May 18 11:05:18 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 93999
    May 18 11:05:18 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:18 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:18 entry01 kernel: Memory cgroup stats for /lve1421: cache:32KB rss:524256KB rss_huge:407552KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:264108KB active_anon:260152KB inactive_file:8KB active_file:8KB unevictable:0KB
    May 18 11:05:18 entry01 kernel: Memory cgroup out of memory: Kill process 32257 (lsphp) score 165 or sacrifice child
    May 18 11:05:18 entry01 kernel: Killed process 32257 (lsphp) in VE "0" total-vm:430440kB, anon-rss:82260kB, file-rss:3960kB, shmem-rss:4kB
    May 18 11:05:18 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 18 11:05:18 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
    May 18 11:05:18 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
    May 18 11:05:18 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 95211
    May 18 11:05:18 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:18 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:18 entry01 kernel: Memory cgroup stats for /lve1421: cache:36KB rss:524252KB rss_huge:403456KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:268592KB active_anon:255664KB inactive_file:24KB active_file:8KB unevictable:0KB
    May 18 11:05:18 entry01 kernel: Memory cgroup out of memory: Kill process 32258 (lsphp) score 121 or sacrifice child
    May 18 11:05:18 entry01 kernel: Killed process 32258 (lsphp) in VE "0" total-vm:405352kB, anon-rss:59204kB, file-rss:3828kB, shmem-rss:4kB
    May 18 11:05:19 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 18 11:05:19 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
    May 18 11:05:19 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
    May 18 11:05:19 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 97351
    May 18 11:05:19 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:19 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:19 entry01 kernel: Memory cgroup stats for /lve1421: cache:28KB rss:524260KB rss_huge:397312KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:273228KB active_anon:251036KB inactive_file:8KB active_file:4KB unevictable:0KB
    May 18 11:05:19 entry01 kernel: Memory cgroup out of memory: Kill process 32292 (lsphp) score 157 or sacrifice child
    May 18 11:05:19 entry01 kernel: Killed process 32292 (lsphp) in VE "0" total-vm:426240kB, anon-rss:77972kB, file-rss:3924kB, shmem-rss:4kB
    May 18 11:05:20 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 18 11:05:20 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
    May 18 11:05:20 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
    May 18 11:05:20 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 98512
    May 18 11:05:20 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:20 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:20 entry01 kernel: Memory cgroup stats for /lve1421: cache:28KB rss:524260KB rss_huge:397312KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:292304KB active_anon:231960KB inactive_file:12KB active_file:12KB unevictable:0KB
    May 18 11:05:20 entry01 kernel: Memory cgroup out of memory: Kill process 32251 (lsphp) score 136 or sacrifice child
    May 18 11:05:20 entry01 kernel: Killed process 32251 (lsphp) in VE "0" total-vm:413620kB, anon-rss:66716kB, file-rss:3940kB, shmem-rss:4kB
    May 18 11:05:20 entry01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 18 11:05:20 entry01 kernel: lsphp cpuset=lve1421 mems_allowed=0-1
    May 18 11:05:20 entry01 kernel: Task in /lve1421 killed as a result of limit of /lve1421
    May 18 11:05:20 entry01 kernel: memory: usage 524288kB, limit 524288kB, failcnt 100039
    May 18 11:05:20 entry01 kernel: memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:20 entry01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 18 11:05:20 entry01 kernel: Memory cgroup stats for /lve1421: cache:32KB rss:524256KB rss_huge:399360KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:278656KB active_anon:245604KB inactive_file:12KB active_file:12KB unevictable:0KB
    May 18 11:05:20 entry01 kernel: Memory cgroup out of memory: Kill process 32252 (lsphp) score 159 or sacrifice child
    May 18 11:05:20 entry01 kernel: Killed process 32252 (lsphp) in VE "0" total-vm:426208kB, anon-rss:78764kB, file-rss:3880kB, shmem-rss:4kB
    

    Right now, I have for all my External Apps Memory Soft Limit (bytes): 4096M and Memory Hard Limit (bytes): 4096M. I've also uploaded 2 screenshots. All the external apps have the same settings.

    Currently running version 5.2.7, Build 5 of Litespeed. Also running cPanel 11.68.0.38.

    Let me know if you need any other info.

    Thank you. DeepinScreenshot_select-area_20180518111307.png DeepinScreenshot_select-area_20180518111259.png
     
  2. NiteWave

    NiteWave Administrator

    what's the cloudlinux's lve settings for user id 1421 ?
     
  3. Havri

    Havri Member

    Hello,

    I've attached a screenshot with the LVE limits for the user with userid 1421.

    Let me know if you need any other info.
     

    Attached Files:

  4. NiteWave

    NiteWave Administrator

    cludlinux lve PMEM = 512M, too small.
    increase it, or set it to 0 (unlimited), see if those errors will be gone.
     
  5. Havri

    Havri Member

    Hello,

    Thank you. I will increase the PMEM for all the users that have 512M and get back to you. I've also found other errors that look pretty much the same, but there isn't any specific LVE name/number. It's just a number:


    Code:
    May 20 23:18:51 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 20 23:18:51 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
    May 20 23:18:51 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
    May 20 23:18:51 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 224006
    May 20 23:18:51 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
    May 20 23:18:51 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 20 23:18:51 my_server01 kernel: Memory cgroup stats for /1168: cache:16KB rss:1048560KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:279160KB active_anon:769404KB inactive_file:8KB active_file:4KB unevictable:0KB
    May 20 23:18:51 my_server01 kernel: Memory cgroup out of memory: Kill process 9597 (lsphp) score 152 or sacrifice child
    May 20 23:18:51 my_server01 kernel: Killed process 9597 (lsphp) in VE "0" total-vm:572956kB, anon-rss:153572kB, file-rss:5820kB, shmem-rss:4kB
    May 20 23:15:02 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 20 23:15:02 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
    May 20 23:15:02 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
    May 20 23:15:02 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 222937
    May 20 23:15:02 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
    May 20 23:15:02 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 20 23:15:02 my_server01 kernel: Memory cgroup stats for /1168: cache:8KB rss:1048568KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:525060KB active_anon:523512KB inactive_file:0KB active_file:4KB unevictable:0KB
    May 20 23:15:02 my_server01 kernel: Memory cgroup out of memory: Kill process 5554 (lsphp) score 122 or sacrifice child
    May 20 23:15:02 my_server01 kernel: Killed process 5554 (lsphp) in VE "0" total-vm:467536kB, anon-rss:123824kB, file-rss:3776kB, shmem-rss:4kB
    May 20 23:15:02 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 20 23:15:02 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
    May 20 23:15:02 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
    May 20 23:15:02 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 223039
    May 20 23:15:02 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
    May 20 23:15:02 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 20 23:15:02 my_server01 kernel: Memory cgroup stats for /1168: cache:8KB rss:1048568KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:497324KB active_anon:551124KB inactive_file:4KB active_file:0KB unevictable:0KB
    May 20 23:15:02 my_server01 kernel: Memory cgroup out of memory: Kill process 5577 (lsphp) score 120 or sacrifice child
    May 20 23:15:02 my_server01 kernel: Killed process 5577 (lsphp) in VE "0" total-vm:465244kB, anon-rss:121892kB, file-rss:3656kB, shmem-rss:4kB
    May 20 23:15:03 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 20 23:15:03 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
    May 20 23:15:03 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
    May 20 23:15:03 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 223135
    May 20 23:15:03 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
    May 20 23:15:03 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 20 23:15:03 my_server01 kernel: Memory cgroup stats for /1168: cache:8KB rss:1048568KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:524512KB active_anon:524060KB inactive_file:0KB active_file:4KB unevictable:0KB
    May 20 23:15:03 my_server01 kernel: Memory cgroup out of memory: Kill process 5580 (lsphp) score 121 or sacrifice child
    May 20 23:15:03 my_server01 kernel: Killed process 5580 (lsphp) in VE "0" total-vm:465980kB, anon-rss:122476kB, file-rss:3840kB, shmem-rss:4kB
    May 20 23:15:04 my_server01 kernel: lsphp invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
    May 20 23:15:04 my_server01 kernel: lsphp cpuset=1168 mems_allowed=0-1
    May 20 23:15:04 my_server01 kernel: Task in /1168 killed as a result of limit of /1168
    May 20 23:15:04 my_server01 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 223310
    May 20 23:15:04 my_server01 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
    May 20 23:15:04 my_server01 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    May 20 23:15:04 my_server01 kernel: Memory cgroup stats for /1168: cache:8KB rss:1048568KB rss_huge:0KB mapped_file:4KB shmem:4KB slab_reclaimable:0KB slab_unreclaimable:0KB swap:0KB inactive_anon:524464KB active_anon:523980KB inactive_file:0KB active_file:4KB unevictable:0KB
    May 20 23:15:04 my_server01 kernel: Memory cgroup out of memory: Kill process 5611 (lsphp) score 123 or sacrifice child
    May 20 23:15:04 my_server01 kernel: Killed process 5611 (lsphp) in VE "0" total-vm:469340kB, anon-rss:125292kB, file-rss:3764kB, shmem-rss:4kB
    What I want to know is if the "Memory cgroup stats for /1168:" represents the UID 1168 from /etc/passwd and should be interpreted the same as "Memory cgroup stats for /lve1421:" (note that there is no "lve" before the number).

    So in these two examples, "Memory cgroup stats for /1168" represents LVE for UID 1168 and "Memory cgroup stats for /lve1421:" represents LVE for UID 1421. I already found the corresponding UID for these LVEs, but I just want to be sure.


    Thank you.
     
  6. mistwang

    mistwang LiteSpeed Staff

    I think CloudLinux can give you better answer to this question, as they are the owner of this feature, producing this log message.
     
  7. Havri

    Havri Member

    Hello,

    Thanks. I've created another thread on their forum.

    Regards.
     

Share This Page