The Problem
A Linux OS failed bootup, from console log we can see all goes well in the very beginning, kernel loaded and init scripts were running OK. But suddenly the Linux OS was shutting down (received TERM signal) after auditd daemon started.
... dracut: Switching root mount: mount point /proc/bus/usb does not exist Welcome to Oracle Linux Server Starting udev: udev: starting version 147 (snip) Mounting local filesystems: EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null) EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: (null) EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null) EXT4-fs (dm-9): mounted filesystem with ordered data mode. Opts: (null) EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: (null) EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts: (null) EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null) EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Enabling local filesystem quotas: [ OK ] Enabling /etc/fstab swaps: Adding 16777212k swap on /dev/mapper/vg_ol68-LogVol08. Priority:-1 extents:1 across:16777212k SS [ OK ] Entering non-interactive startup Starting OVM guest daemon: [ OK ] Calling the system activity data collector (sadc)... Starting monitoring for VG vg_ol68: 10 logical volume(s) in volume group "vg_ol68" monitored [ OK ] NET: Registered protocol family 10 Bringing up loopback interface: [ OK ] Bringing up interface eth0: Determining if ip address x.x.x is already in use for device eth0... [ OK ] Starting auditd: type=1305 audit(1500420382.015:3): audit_pid=1626 old=0 auid=4294967295 ses=4294967295 res=1 init: rc main process (1341) killed by TERM signal [ OK ]
Confirmed there was no kernel panic issue happened. The Linux OS had been running well for days. No man-made operation/change before the issue.
The Solution
Normally the Linux OS will not shutdown itself. But some application/utility does. Since every time the TERM signal was received when auditd daemon starts, we found the auditd daemon has the feature to halt Linux OS under some specific situations.
The following items in “man auditd.conf” will shutdown the Linux OS when set value to “halt”.
- space_left_action
- admin_space_left_action
- disk_full_action
- disk_error_action
The Linux OS in this case does have those “halt” settings.
# cat /etc/audit/auditd.conf | grep halt admin_space_left_action = halt disk_full_action = halt disk_error_action = halt
And the volume of /var/log/audit has only 6MB space available.
/dev/mapper/vg_LogVol05 16040 428304 61524 88% /var/log/audit
This the expected behavior when auditd found space issue or disk error, please check Linux OS accordingly. If you don’t want auditd to shutdown the Linux OS, you can modify the “halt” to “syslog”, please refer “man auditd.conf” for more details.
# man auditd.conf