The Geek Diary

A File Is Claimed to Be Disappearing – How to monitor a file for deletion in Linux

by

Question: A file seems to be disappearing and no direct reason for its disappearance can be found. How can we monitor the file access, especially the deletion of a particular file in CentOS/RHEL system?

A file can be monitored through its lifecycle by using the Linux auditd service.

The Steps

1. To install it on CentOS or RHEL, run:

# yum install audit

2. To enable the service at boot time, run:

# chkconfig auditd on        # For CentOS/RHEL 6
# systemctl enable auditd        # For CentOS/RHEL 7

or start it temporarily, until the next reboot:

# service auditd start        # For CentOS/RHEL 6
# systemctl start auditd        # For CentOS/RHEL 7
Note: systemctl restart auditd won’t work in RHEL7 because the kernel handles auditd in special way. You will need to stop and start the service.

3. To watch a specific file to see if something touches the file, run:

# auditctl -w [filename] -p rwxa  -k [filter key]

where:
-w [filename] : watch the specific file.
-p war : permissions filter for write, append, read.
-k [filter key] : a filter key to uniquely identify the audit logs.

For Example:

# auditctl -w /var/opt/sun/comms/messaging64/log/mail.log_yesterday -p rwxa -k mail.log_yesterday

4. To list the current set of rules, run: 

# auditctl -l

For Example:

# auditctl -l
LIST_RULES: exit,always watch=/var/opt/sun/comms/messaging64/log/mail.log_yesterday perm=rwxa key=mail.log_yesterday

Verify

To find out who changed or accessed the watched file:

# ausearch -i -k [filter key]

For Example:

# ausearch -i -k mail.log_yesterday
...
----
type=PATH msg=audit(12/12/2013 00:30:00.407:6209) : item=0 name=/opt/sun/comms/messaging64/data/log/mail.log_yesterday inode=425831 dev=fd:00 mode=file,600 ouid=mailsrv ogid=mail rdev=00:00 
type=CWD msg=audit(12/12/2013 00:30:00.407:6209) :  cwd=/var/opt/sun/comms/messaging64/log 
type=SYSCALL msg=audit(12/12/2013 00:30:00.407:6209) : arch=x86_64 syscall=open success=yes exit=5 a0=7fff62973b1d a1=0 a2=7fff62973870 a3=2 items=1 ppid=16442 pid=16450 auid=root uid=mailsrv gid=mail euid=mailsrv suid=mailsrv fsuid=mailsrv egid=mail sgid=mail fsgid=mail tty=(none) ses=570 comm=cat exe=/bin/cat key=mail.log_yesterday 
----
type=CONFIG_CHANGE msg=audit(12/12/2013 00:30:00.407:6210) : auid=root ses=570 op="updated rules" path=/var/opt/sun/comms/messaging64/log/mail.log_yesterday key=mail.log_yesterday list=exit res=1 
----
type=PATH msg=audit(12/12/2013 00:30:00.407:6211) : item=1 name=/opt/sun/comms/messaging64/data/log/mail.log_yesterday inode=425831 dev=fd:00 mode=file,600 ouid=mailsrv ogid=mail rdev=00:00 
type=PATH msg=audit(12/12/2013 00:30:00.407:6211) : item=0 name=/opt/sun/comms/messaging64/data/log/ inode=423325 dev=fd:00 mode=dir,700 ouid=mailsrv ogid=mail rdev=00:00 
type=CWD msg=audit(12/12/2013 00:30:00.407:6211) :  cwd=/var/opt/sun/comms/messaging64/log 
type=SYSCALL msg=audit(12/12/2013 00:30:00.407:6211) : arch=x86_64 syscall=unlink success=yes exit=0 a0=7ffff94beb1f a1=7ffff94beb1f a2=2 a3=2 items=2 ppid=16442 pid=16452 auid=root uid=mailsrv gid=mail euid=mailsrv suid=mailsrv fsuid=mailsrv egid=mail sgid=mail fsgid=mail tty=(none) ses=570 comm=rm exe=/bin/rm key=mail.log_yesterday 
----
type=CONFIG_CHANGE msg=audit(12/12/2013 00:30:00.407:6212) : auid=root ses=570 op="updated rules" path=/var/opt/sun/comms/messaging64/log/mail.log_yesterday key=mail.log_yesterday list=exit res=1 
----
type=PATH msg=audit(12/12/2013 00:30:00.407:6213) : item=3 name=/opt/sun/comms/messaging64/data/log/mail.log_yesterday inode=425765 dev=fd:00 mode=file,600 ouid=mailsrv ogid=mail rdev=00:00 
type=PATH msg=audit(12/12/2013 00:30:00.407:6213) : item=2 name=/opt/sun/comms/messaging64/data/log/mail.log_current inode=425765 dev=fd:00 mode=file,600 ouid=mailsrv ogid=mail rdev=00:00 
type=PATH msg=audit(12/12/2013 00:30:00.407:6213) : item=1 name=/opt/sun/comms/messaging64/data/log/ inode=423325 dev=fd:00 mode=dir,700 ouid=mailsrv ogid=mail rdev=00:00 
type=PATH msg=audit(12/12/2013 00:30:00.407:6213) : item=0 name=/opt/sun/comms/messaging64/data/log/ inode=423325 dev=fd:00 mode=dir,700 ouid=mailsrv ogid=mail rdev=00:00 
type=CWD msg=audit(12/12/2013 00:30:00.407:6213) :  cwd=/var/opt/sun/comms/messaging64/log 
type=SYSCALL msg=audit(12/12/2013 00:30:00.407:6213) : arch=x86_64 syscall=rename success=yes exit=0 a0=7fff81d5caea a1=7fff81d5cb1f a2=0 a3=0 items=4 ppid=16442 pid=16453 auid=root uid=mailsrv gid=mail euid=mailsrv suid=mailsrv fsuid=mailsrv egid=mail sgid=mail fsgid=mail tty=(none) ses=570 comm=mv exe=/bin/mv key=mail.log_yesterday 
...

Here we see the rotation of mail.log files occurring, which is being done by the ‘return_job‘ which is happening by default at 00:30, and which is ran by user “mailsrv“. You can see the mail.log_yesterday file being cat (into the mail.log), being removed, and the mail.log_current getting renamed to the mail.log_yesterday.

Understanding System auditing with auditd

You May Also Like