Out of Memory (OOM)

wikipedia:Out of memory

• journalctl -r | grep "Killed process"
• journalctl -r | grep oom_

OOM in Docker

• Docker: --oom-kill-disable=false To enable or disable OOM Killer for the container.

Use docker inspect CONTANINER_NAME | grep OOM to show options. For example:

docker inspect grafana | grep -i OOM
           "OOMKilled": false,
           "OomScoreAdj": 0,
           "OomKillDisable": false,

Kubernetes

• Pod The node had condition: [MemoryPressure].
• kubectl describe nodes | grep MemoryPressure
• CrashLoopBackOff
• kubectl get events --sort-by='.lastTimestamp' | grep -iE "OOM|evict|kill|memory"

OOM Linux kernel logs

Jan 26 04:40:29 SERVER_NAME kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0

Jan 26 04:40:29 SERVER_NAME kernel: oom_reaper: reaped process 22572 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

May 22 14:02:59 ip-10-10-10-10 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/system-postgresql.slice/[email protected],task=postgres,pid=9120,uid=115

May 22 14:02:59 ip-10-10-10-10 kernel: Out of memory: Killed process 9120 (postgres) total-vm:1493744kB, anon-rss:280412kB, file-rss:2052kB, shmem-rss:963576kB, UID:115 pgtables:2856kB oom_score_adj:0

Redis

• OOM command not allowed when used memory ...

Related terms

• Java.lang.OutOfMemoryError: Java heap space
• /proc/sys/vm/panic_on_oom
• /proc/sys/vm/oom_kill_allocating_task
• yarn: Error Command failed with exit code 137
• systemd-oomd
• pm.max_requests
• EarlyOOM
• FailedScheduling .../... Insufficient memory
• oom_kill_process.cold
• sar -r
• Cannot allocate memory

See also

• Memory: memory pages, RAM, virsh Memory Commands, OOM, meminfo, vmstat, NAND, DDR, lsmem, /dev/shm, /proc/meminfo, sar -r, IOMMU, pmem, Memory management, Garbage collector, THP, Linux Huge Page TLB, usage, Remote direct memory access (RMDA), SK Hynix, HBM
• OOM, EarlyOOM, OOM Killer, MemoryPressure (K8s), systemd-oomd, OOMKilled, oom-killer, memcpy, sar -r, Exit code: 137, Killed process, oom_reaper, oom_score_adj
• ENOMEM