• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer navigation

The Geek Diary

  • OS
    • Linux
    • CentOS/RHEL
    • Solaris
    • Oracle Linux
    • VCS
  • Interview Questions
  • Database
    • oracle
    • oracle 12c
    • ASM
    • mysql
    • MariaDB
  • DevOps
    • Docker
    • Shell Scripting
  • Big Data
    • Hadoop
    • Cloudera
    • Hortonworks HDP

Linux filesystem is filling, despite no large files or directories

by admin

The Problem

On a Linux-based system, the root file system is filling up by some unknown process. It continues regardless of what files are moved or cleaned from the file system.

# df -hP /
/dev/mapper/VGExaDb-LVDbSys1    ext3     30G   29G     0 100%    /

There are no large files to account for the full file system:

# find / -xdev -type f -size +100M -exec ls -lh {} \;

There are no large directories to account for the full file system:

# du -h --max-depth=1 /
42M     /sbin
13M     /etc
2.4G    /usr
45M    /tmp
451M   /var
192M   /lib
(and so on)
...

The Solution

At one point in the past, two or more processes were using a file; for example,/tmp/top.log. One process stopped it’s access to /tmp/top.log by deleting it (actually it’s directory entry) and the other process continued to write to the inode reference, allowing the file to continue to grow.

This can be seen in the output of: “lsof +L 1”

# lsof +L 1
COMMAND    PID      USER   FD     TYPE   DEVICE     SIZE/OFF      NLINK          NODE       NAME
top        34261    root   1W     REG   252,0       21460567592   0              1785896    /tmp/top.log (deleted)

Other files were listed too but were much smaller.

This shows that user=root was running a top command that was spooling to /tmp/top.log, and that there are currently no links to that file. That spool file was 21Gb in size but did not report in “du –h –max-depth=1 /” output, where /tmp listed as only 45M.

Follow the steps below to identify and kill such a process.

1. Identify the files on the system that have fewer than 1 link with the command:

# lsof +L 1

2. Kill any processes that are writing to any unusually large file listed. In the example above, you would run:

# kill 34261

3. Space will be released when the final process stops using the file.

Filed Under: Linux

Some more articles you might also be interested in …

  1. extrace Command Examples in Linux
  2. “az provider” Command Examples (Manage resource providers)
  3. cdk: A CLI for AWS Cloud Development Kit (CDK)
  4. How to Disable Daylight Savings Time (DST), and Modify the Timezone on Linux
  5. fstrim Command Examples in Linux
  6. du: command not found
  7. atom: A cross-platform pluggable text editor (Command Examples)
  8. Linux OS Service ‘squid’
  9. bup: Backup system based on the Git packfile format, providing incremental saves and global deduplication
  10. jpegoptim for image optimization

You May Also Like

Primary Sidebar

Recent Posts

  • ctags: Generates an index (or tag) file of language objects found in source files for many popular programming languages
  • csvtool: Utility to filter and extract data from CSV formatted sources
  • csvstat: Print descriptive statistics for all columns in a CSV file
  • csvsql: Generate SQL statements for a CSV file or execute those statements directly on a database

© 2023 · The Geek Diary

  • Archives
  • Contact Us
  • Copyright