• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

The Geek Diary

CONCEPTS | BASICS | HOWTO

  • OS
    • Linux
    • CentOS/RHEL
    • Solaris
    • Oracle Linux
    • Linux Services
    • VCS
  • Database
    • oracle
    • oracle 12c
    • ASM
    • mysql
    • MariaDB
    • Data Guard
  • DevOps
    • Docker
    • Shell Scripting
  • Interview Questions
  • Big Data
    • Hadoop
    • Cloudera
    • Hortonworks HDP

CentOS/RHEL 6 Kernel Panic – “Out of memory” kernel parameter panic_on_oom is enabled

By admin

The Problem

CentOS/RHEL 6 system reboot due too out of memory error:

DUMPFILE: vmcore [PARTIAL DUMP]
  CPUS: 2
  DATE: Mon Feb 29 05:28:02 2016
  UPTIME: 33 days, 09:45:55
LOAD AVERAGE: 1.88, 1.52, 1.41
  TASKS: 218
  NODENAME: localhost
  RELEASE: 2.6.32-431.el6.x86_64
  VERSION: #1 SMP Sun Nov 10 22:19:54 EST 2013
  MACHINE: x86_64 (3000 Mhz)
  MEMORY: 8 GB
  PANIC: "Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled"
  PID: 31010
  COMMAND: "java"
  TASK: ffff88010055a040 [THREAD_INFO: ffff88001583e000]
  CPU: 0
  STATE: TASK_RUNNING (PANIC)

The Solution

Reboot is due to:
1. the OOM killer and the physical/swap memory is completely utilized.
2. vm.panic_on_oom was enabled.

>kmem -i

PAGES TOTAL PERCENTAGE
TOTAL MEM 1979578 7.6 GB ----
FREE 27818 108.7 MB 1% of TOTAL MEM
USED 1951760 7.4 GB 98% of TOTAL MEM   98% memory used
SHARED 270 1.1 MB 0% of TOTAL MEM
BUFFERS 53 212 KB 0% of TOTAL MEM
CACHED 1461 5.7 MB 0% of TOTAL MEM
SLAB 14026 54.8 MB 0% of TOTAL MEM

TOTAL SWAP 4196348 16 GB ----
SWAP USED 4196347 16 GB 99% of TOTAL SWAP
SWAP FREE 1 4 KB 0% of TOTAL SWAP      No free swap space

COMMIT LIMIT 5186137 19.8 GB ----
COMMITTED 1564979 6 GB 30% of TOTAL LIMIT
# sysctl_-a |grep vm.panic_on_oom
 vm.panic_on_oom = 1                 OOM enabled for panic.

In this example, “java” process consumed 80% of the memory.

>ps
..
30934 1 1 ffff88010043aae0 IN 78.3 29303436 7384984 java
30935 1 1 ffff88010043b540 IN 78.3 29303436 7384984 java
30936 1 1 ffff880238f62080 IN 78.3 29303436 7384984 java
31008 1 1 ffff8801060b0aa0 IN 78.3 29303436 7384984 java
31009 1 1 ffff8801060b0040 IN 78.3 29303436 7384984 java
> 31010 1 0 ffff88010055a040 RU 78.3 29303436 7384984 java
31690 1 0 ffff880239fc6aa0 IN 78.3 29303436 7384984 java
...

Physical memory or swap memory has to be increased to avoid OOM, else the memory usage of application has to be reduced. The application memory usage can be limit via ulimit or cgroup.

Filed Under: CentOS/RHEL 6, Linux

Some more articles you might also be interested in …

  1. How to Schedule Jobs with ‘at’ command under Linux
  2. How to monitor the Mounting/Umounting of Mount Points Using Auditd on CentOS/RHEL 6,7
  3. How to find docker storage device and its size (device mapper storage driver)
  4. NFSv4 Client Shows “nobody” As Owner And Group For Mount Point (CentOS/RHEL)
  5. How to Re-generate initramfs and vmlinuz for Rescue Kernel with Current Kernel in CentOS/RHEL 7
  6. How to Disable user list on GNOME login screen in CentOS/RHEL 8
  7. How to check failed or bad login attempts in Linux
  8. How to configure apache virtual host on ubuntu
  9. How to list all modules and check if they are enabled or disabled in CentOS/RHEL 8
  10. LVM and multipathing – sample LVM filter strings

You May Also Like

Primary Sidebar

Recent Posts

  • What are different Oracle Database Vault Roles
  • Unable to export realm protected table using data pump
  • Beginners Guide to Oracle Database Vault
  • How to Disable IPv6 on Ubuntu 18.04 Bionic Beaver Linux
  • Archives
  • Contact Us
  • Copyright

© 2021 · The Geek Diary