The problem
Audit log and messages file not rotating .
Rotation of /var/log/messages, /var/log/audit.log and /var/log/asmaudit.log files are not happening at OS level.
$ ls -ltr /var/log/audit* -rw------- 1 root root 4436099 Mar 22 03:07 audit.log.10.gz -rw------- 1 root root 3873269 Mar 23 03:07 audit.log.9.gz -rw------- 1 root root 4036499 Mar 25 03:07 audit.log.8.gz -rw------- 1 root root 3454722 Mar 26 03:08 audit.log.7.gz -rw------- 1 root root 6145628 Mar 27 03:07 audit.log.6.gz -rw------- 1 root root 2979527 Mar 28 03:07 audit.log.5.gz -rw------- 1 root root 3797834 Mar 29 03:07 audit.log.4.gz -rw------- 1 root root 4283296 Mar 30 03:08 audit.log.3.gz ----->>> No logs rotated for 20 days -rw------- 1 root root 96190 Apr 20 11:20 audit.log.2.gz ----->>> No logs rotated for 20 days -rw------- 1 root root 387 Apr 22 12:16 audit.log.1.gz -rw------- 1 root root 507157 May 3 10:26 audit.log
$ ls -ltr /var/log/mess* -rw-r----- 1 root orarom 2996230 Mar 28 03:07 messages.6.gz -rw-r----- 1 root orarom 3821592 Mar 29 03:07 messages.5.gz -rw-r----- 1 root orarom 4324393 Mar 30 03:08 messages.4.gz -rw-r----- 1 root orarom 5240535 Mar 31 03:08 messages.3.gz ----->>> No logs rotated for 20 days -rw-r----- 1 root orarom 96226018 Apr 20 11:20 messages.2.gz ----->>> No logs rotated for 20 days -rw-r----- 1 root orarom 10463366 Apr 22 12:16 messages.1.gz -rw-r----- 1 root orarom 1555913366 May 3 10:26 messages
$ ls -ltr /var/log/asmau* -rw------- 1 root root 4005971 Mar 26 03:07 asmaudit.log.4.gz -rw------- 1 root root 4128224 Apr 20 11:19 asmaudit.log.3.gz -rw------- 1 root root 16248312 Apr 22 12:16 asmaudit.log.2.gz -rw------- 1 root root 74179752 Apr 22 12:16 asmaudit.log.1 -rw------- 1 root root 396961019 May 3 10:27 asmaudit.log
$ cat /var/lib/logrotate.status | grep mess "/var/log/messages" 2017-4-22 ----->>> Logs rotated manually
Root Cause
cron.daily job locked by another anacron job. We can see on the ps output that anacron is still running and blocking other processes.
$ ps -ef|grep cron
root 17592 1 0 Mar11 ? 00:00:16 crond
root 132079 232781 0 Mar31 ? 00:00:00 /bin/bash /usr/bin/run-parts /etc/cron.daily
root 232781 1 0 Mar31 ? 00:00:00 /usr/sbin/anacron -s
oracle 264332 233472 0 06:37 pts/6 00:00:00 grep cron
root 392965 132079 0 Mar31 ? 00:00:00 /bin/sh /etc/cron.daily/mlocate.cron
root 392966 132079 0 Mar31 ? 00:00:00 awk -v progname=/etc/cron.daily/mlocate.cron progname {????? print progname ":\n"????? progname="";???? }???? { print; }
The Solution
Please kill stucked cron processes and restart anacron process. For example:
$ ps -ef|grep cron
root 17592 1 0 Mar11 ? 00:00:16 crond
root 132079 232781 0 Mar31 ? 00:00:00 /bin/bash /usr/bin/run-parts /etc/cron.daily
root 1320791 0 Mar31 ? 00:00:00 /usr/sbin/anacron -s
oracle 264332 233472 0 06:37 pts/6 00:00:00 grep cron
root 392965 132079 0 Mar31 ? 00:00:00 /bin/sh /etc/cron.daily/mlocate.cron
root 392966 132079 0 Mar31 ? 00:00:00 awk -v progname=/etc/cron.daily/mlocate.cron progname {????? print progname ":\n"????? progname="";???? }???? { print; }
and it should be like below
$ ps -ef|grep cron oracle 75374 61598 0 00:37 pts/2 00:00:00 grep cron root 387494 1 0 Jul07 ? 00:00:00 crond
Then please monitor the server for a day if it will rotate the logs.
$ egrep "messages|audit" /var/lib/logrotate.status "/var/log/asmaudit.log" 2017-7-9 "/var/log/messages" 2017-7-9 "/var/log/audit.log" 2017-7-8
$ ls -ltr /var/log/audit* -rw------- 1 root root 4036499 Mar 25 03:07 /var/log/audit.log.10.gz -rw------- 1 root root 3454722 Mar 26 03:08 /var/log/audit.log.9.gz -rw------- 1 root root 6145628 Mar 27 03:07 /var/log/audit.log.8.gz -rw------- 1 root root 2979527 Mar 28 03:07 /var/log/audit.log.7.gz -rw------- 1 root root 3797834 Mar 29 03:07 /var/log/audit.log.6.gz -rw------- 1 root root 4283296 Mar 30 03:08 /var/log/audit.log.5.gz -rw------- 1 root root 96190 Apr 20 11:20 /var/log/audit.log.4.gz -rw------- 1 root root 387 Apr 22 12:16 /var/log/audit.log.3.gz -rw----r-- 1 root root 109466 Jun 27 10:39 /var/log/audit.log.2.gz -rw----r-- 1 root root 3530666 Jul 8 03:08 /var/log/audit.log.1.gz -rw----r-- 1 root root 180505494 Jul 10 00:38 /var/log/audit.log
$ ls -ltr /var/log/mess* -rw-r----- 1 root orarom 96226018 Apr 20 11:20 /var/log/messages.6.gz -rw-r----- 1 root orarom 10463366 Apr 22 12:16 /var/log/messages.5.gz -rw-r----- 1 root orarom 206584051 Jun 27 10:39 /var/log/messages.4.gz -rw-r----- 1 root orarom 58913698 Jul 7 11:09 /var/log/messages.3.gz -rw-r----- 1 root orarom 3541858 Jul 8 03:08 /var/log/messages.2.gz -rw-r----- 1 root orarom 6033888 Jul 9 03:08 /var/log/messages.1.gz -rw-r----- 1 root orarom 180613996 Jul 10 00:38 /var/log/messages
$ ls -ltr /var/log/asmau* -rw------- 1 root root 1542952 Jun 27 10:37 /var/log/asmaudit.log.4.gz -rw----r-- 1 root root 22711243 Jul 7 11:08 /var/log/asmaudit.log.3.gz -rw----r-- 1 root root 3626457 Jul 9 03:08 /var/log/asmaudit.log.2.gz -rw----r-- 1 root root 42116158 Jul 9 03:08 /var/log/asmaudit.log.1 -rw----r-- 1 root root 22786400 Jul 10 00:39 /var/log/asmaudit.log