One of the most confusing situations in Linux system administration is when the df command shows that a filesystem is almost full, while the du command reports much lower disk usage.

This situation is especially common in production environments. A typical scenario looks like this:

A monitoring system sends a “disk full” alert during the night. You connect to the server and run:

df -h

The output shows that the root filesystem, or another mounted filesystem, is almost completely full. Then you try to find which directory is consuming the space by running du, but the total usage reported by du is much lower than what df shows.

At first glance, this may look like a contradiction. However, in most cases, both commands are correct. The real reason is that df and du calculate disk usage from different layers of the operating system.

In this article, we will explain what df and du actually measure, why their outputs may differ, how deleted files can continue consuming disk space, how to detect them, and how to safely reclaim the lost space.

---

Why Do df and du Show Different Results?

Two of the most commonly used commands for checking disk usage in Linux are:

df -h

and:

du -sh /

A safer and more controlled command for production systems is:

du -xhd1 /

Although these commands seem to serve a similar purpose, they do not measure disk usage in the same way.

df works at the filesystem level. It shows how many blocks are used and how many blocks are available.

du works at the file and directory level. It walks through the directory tree and adds up the sizes of the files it can access.

In other words, df asks the filesystem how much space is allocated, while du calculates the total size of visible files and directories.

This difference becomes very important when a file has been deleted but is still being held open by a running process.

---

What Does the df Command Measure?

df stands for “disk free”. It displays disk usage information for mounted filesystems.

The most common usage is:

df -h

Example output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   47G  3.0G  94% /
/dev/sdb1       200G  150G   50G  75% /data

The columns mean:

Column	Description
Filesystem	The disk partition or filesystem
Size	Total filesystem size
Used	Used disk space
Avail	Available disk space
Use%	Usage percentage
Mounted on	Mount point of the filesystem

The important point is that df does not scan directories one by one. It does not walk through /var, /home, /opt, or other directories to calculate file sizes.

Instead, it asks the kernel and the filesystem metadata how many blocks are allocated and how many blocks are free.

So df answers this question:

“How many blocks are currently used on this filesystem?”

For this reason, df gives a reliable filesystem-level view of real block allocation.

---

What Does the du Command Measure?

du stands for “disk usage”. It calculates the disk space used by files and directories.

For example:

du -sh /var/log

Example output:

2.4G    /var/log

This shows the total disk usage of the /var/log directory.

However, running the following command directly on a production server may not always be ideal:

du -sh /

This command may scan the entire directory tree, including large directories and mounted filesystems. On busy systems, it can take a long time and create unnecessary disk I/O.

A better approach is:

du -xhd1 /

The parameters are important:

Parameter	Meaning
-x	Stay on the same filesystem and do not cross into other mount points
-h	Show human-readable sizes such as KB, MB, and GB
-d1	Show only first-level directories

Example output:

4.5G    /usr
2.8G    /var
1.1G    /home
650M    /opt
20K     /root
9.2G    /

This output helps you quickly identify which top-level directory is consuming space.

However, there is a critical limitation:

du can only calculate the size of files it can see in the directory tree.

If a file has been deleted and no longer exists in the directory structure, du cannot include it in its calculation. But if that deleted file is still open by a running process, the disk space may still be allocated.

This is one of the most common reasons why df and du show different results.

---

How File Deletion Works in Linux

To understand this problem properly, we need to understand how Linux handles file deletion.

In Linux, a file is mainly represented by two important components:

Directory entry
Inode

The directory entry is the visible name and path of the file.

For example:

/var/log/nginx/access.log

This is the file path visible in the directory tree.

The inode stores the actual metadata of the file. This includes ownership, permissions, timestamps, and references to the disk blocks where the file data is stored.

When you delete a file using rm, Linux removes the directory entry first.

For example:

rm /var/log/nginx/access.log

After this command, the file no longer appears in the directory:

ls -lh /var/log/nginx/

You will not see access.log anymore.

However, if a running process still has this file open, the inode is not released immediately. The process may continue writing to the file through its open file descriptor.

In this situation, the filename has been removed, but the actual disk blocks are still in use.

As a result:

ls cannot show the file.

du cannot calculate the file.

The file appears to be deleted.

But df still shows the disk space as used.

From the kernel’s perspective, the file still exists as long as a process keeps an open reference to it.

---

What Is a Deleted but Open File?

A deleted but open file is a file whose directory entry has been removed but which is still being used by a running process through an open file descriptor.

This situation is commonly called:

Deleted but open file
Orphaned open file
Invisible disk usage
Space leak
Unlinked file still held by process

This problem is especially common with long-running services such as:

Nginx
Apache
Tomcat
Java applications
Node.js services
Docker containers
Database services
Log collector agents
Backup applications
Monitoring agents

These services may open a log file and continue writing to it for a long time. If the log file is manually deleted, the application may not notice it immediately. Instead of reopening the file by name, it continues writing to the existing file descriptor.

That is why a deleted file may continue consuming disk space.

---

The Most Common Scenario: Deleting a Large Log File

The most common cause of this problem in production systems is manually deleting a large log file.

Consider the following scenario.

An Nginx access log file grows very large:

/var/log/nginx/access.log

Assume it reaches 20 GB. A disk usage alert is triggered, and an administrator tries to free space quickly by running:

rm /var/log/nginx/access.log

At first, the file appears to be gone.

You check the directory:

ls -lh /var/log/nginx/

The file is no longer visible.

Then you run:

du -sh /var/log

The /var/log directory appears smaller.

But when you run:

df -h

The disk usage has not decreased.

This happens because the Nginx process is still holding the deleted log file open.

The file name has been removed, but the inode and disk blocks are still referenced by the Nginx process. Therefore, the kernel cannot release the disk space.

The space will only be released when one of the following happens:

The process closes the file.
The service is restarted.
The service reopens its log files.
The process exits.
The file descriptor is released.

---

If df Shows Full but du Shows Less, What Should You Check First?

If df -h shows high disk usage but du -xhd1 / does not show where the space is used, the first thing to check is deleted files that are still open.

The best tool for this is lsof.

---

Finding Deleted but Open Files with lsof +L1

The lsof command lists open files on a Linux system.

To find deleted files that are still open, run:

sudo lsof -nP +L1

The parameters mean:

Parameter	Meaning
-n	Do not perform DNS name resolution. This makes the command faster.
-P	Do not convert port numbers to service names.
+L1	List files with a link count lower than 1, which usually means deleted but still open files.

Example output:

COMMAND   PID   USER     FD   TYPE DEVICE   SIZE/OFF NLINK NODE NAME
nginx    1423   root     10w  REG  253,1   524288000     0 1048 /var/log/nginx/access.log (deleted)
java     2201   tomcat   22w  REG  253,1   209715200     0 2341 /tmp/app.log (deleted)

Important columns:

Column	Description
COMMAND	The process holding the file open
PID	Process ID
USER	User running the process
FD	File descriptor
TYPE	File type
DEVICE	Device information
SIZE/OFF	File size or current offset
NLINK	Link count
NAME	File name and status

If the NAME column contains (deleted), the file has been deleted from the filesystem but is still open by a process.

In the example above, Nginx is still holding a deleted access.log file open. The Java application is also holding a deleted /tmp/app.log file open.

These files are not visible with ls and are not counted by du, but their disk blocks are still counted by df.

---

Sorting Large Deleted Files

On some systems, there may be many deleted but open files. To identify the largest ones, you can sort the output by size:

sudo lsof +L1 | awk '{print $7, $0}' | sort -rn

This command tries to sort the output based on the SIZE/OFF field.

A simpler first step is:

sudo lsof -nP +L1

Then manually check entries with large values in the SIZE/OFF column.

If you see files with sizes in GB, they are likely responsible for the missing disk space.

---

How to Reclaim the Disk Space

After identifying deleted but open files, there are several ways to reclaim the disk space.

The most important rule is:

Do not blindly truncate or delete files without understanding what they are.

You must first identify:

Which process is holding the file open
What type of file it is
Whether the file contains critical data
Whether restarting the service is safe
Whether the file is only a normal log file

---

Method 1: Restart the Related Service

The cleanest and usually safest method is to restart the service that is holding the file open.

For Nginx:

sudo systemctl restart nginx

For Apache:

sudo systemctl restart apache2

For Tomcat:

sudo systemctl restart tomcat

For a Docker container:

docker restart container_name

When the service restarts, the open file descriptor is closed. Once the last reference to the inode is gone, the kernel releases the disk blocks.

This method is clean and easy to understand.

The disadvantage is that restarting a service may cause a short interruption. In production environments, you should evaluate the impact before restarting critical services.

---

Method 2: Ask the Service to Reopen Its Log Files

Some services can reopen their log files without a full restart.

For Nginx:

sudo systemctl reload nginx

or:

sudo nginx -s reopen

For Apache:

sudo systemctl reload apache2

Some applications reopen log files when they receive a SIGHUP signal:

sudo kill -HUP <PID>

However, this behavior is application-dependent. Some applications support SIGHUP, while others do not. Some may reload configuration but not reopen log files.

Before using this method, verify how the application handles reload or signal operations.

---

Method 3: Truncate the File Descriptor Through /proc

If restarting the service is not possible, you can truncate the open file descriptor through /proc.

First, identify the problematic file:

sudo lsof -nP +L1

Example output:

COMMAND   PID   USER   FD   TYPE DEVICE   SIZE/OFF NLINK NODE NAME
nginx    1423   root   10w  REG  253,1   524288000     0 1048 /var/log/nginx/access.log (deleted)

Important values:

PID:

1423

FD:

10w

The w means the file is open for writing. Under /proc, the file descriptor can be accessed as:

/proc/1423/fd/10

To truncate it:

sudo truncate -s 0 /proc/1423/fd/10

This command sets the file size to zero and releases the disk space.

The advantage of this method is that the service does not need to be stopped.

However, this method must be used carefully. If you truncate the wrong file, you may cause data loss or application inconsistency.

---

Important Warnings About truncate

The truncate method is not safe for every file.

It is generally acceptable for ordinary log files such as:

Web server access logs
Web server error logs
Debug logs
Temporary application logs
Plain text log files

However, you must be very careful with files such as:

Database WAL files
MySQL binary logs
PostgreSQL transaction logs
Oracle redo logs
Queue files
Crash recovery files
Active data files
State files
Binary application files

Truncating these files may cause data loss, database corruption, or application failure.

Also, some applications may behave unexpectedly after a file is truncated. The process may keep its file offset and continue writing from an unexpected position.

This can be especially problematic with:

Some Java logging implementations
Custom Go logging systems
Stateful services
Applications using binary or structured log formats

Before using truncate, ask the following questions:

Is this file really just a log file?
Does it contain critical data?
Can the service be restarted instead?
Can the service reload and reopen the log file?
Could this operation affect application behavior?

---

Other Reasons for Differences Between df and du

A difference between df and du is not always caused by deleted open files. Other filesystem behaviors may also create differences.

---

1. Reserved Blocks

Filesystems such as ext4 reserve a certain amount of space for the root user.

This reserved space helps prevent the system from becoming completely unusable when the disk is nearly full. The default value is often around 5%.

This reserved area is not available to normal users, so the available space in df may look lower than expected.

To check reserved block information:

sudo tune2fs -l /dev/sdX | grep -i reserved

Example:

sudo tune2fs -l /dev/sda1 | grep -i reserved

Example output:

Reserved block count:     655360
Reserved blocks uid:      0
Reserved blocks gid:      0

On large filesystems, 5% can be a significant amount of space. For example, on a 2 TB filesystem, 5% is about 100 GB.

The reserved block percentage can be changed, but it should be done carefully. Reducing reserved blocks too much on the root filesystem may cause problems when the disk becomes full.

---

2. Hidden Data Under Mount Points

When a filesystem is mounted on top of an existing directory, the old contents of that directory become hidden.

For example, suppose /data already contains files:

/data/old_files

Then another filesystem is mounted on /data:

mount /dev/sdb1 /data

Now the old contents of /data are hidden because /data shows the newly mounted filesystem.

However, the old files may still consume space on the original filesystem.

This can happen after manual mount operations, incorrect fstab entries, or disk migrations.

To inspect mount points:

findmnt

or:

mount

To see block devices and mount relationships:

lsblk

---

3. Sparse Files

Sparse files can appear large logically but consume less physical disk space.

For example, a virtual disk image may appear to be 100 GB but physically consume only 15 GB.

You can compare the physical and apparent size with:

du -sh file.img

and:

du -sh --apparent-size file.img

du -sh file.img shows the actual disk blocks used.

du -sh --apparent-size file.img shows the logical size of the file.

Sparse files are common with:

Virtual machine disks
Database files
Backup images
QCOW2 files
Raw image files
Test files
Large preallocated files

Sparse files may not be the direct reason for a df and du mismatch, but they can make disk usage analysis confusing.

---

4. Container and OverlayFS Layers

On systems running Docker, Podman, or Kubernetes, disk usage can be more complex.

Container platforms use overlay filesystems, image layers, writable container layers, volumes, and logs. Because of this, traditional du checks may not always provide the full picture.

To check Docker disk usage:

docker system df

For more detail:

docker system df -v

Container logs are usually stored under:

/var/lib/docker/containers/

If Docker uses the default json-file log driver, container logs can grow very large over time.

Instead of manually deleting these log files, configure Docker log rotation.

Example /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "5"
  }
}

After changing this configuration, restart Docker:

sudo systemctl restart docker

This limits each container log file to 100 MB and keeps up to 5 rotated files.

---

5. Inode Exhaustion

Sometimes the problem is not disk space but inode exhaustion.

In Linux filesystems, every file and directory uses an inode. If a system creates too many small files, it can run out of inodes even when there is still free disk space.

To check inode usage:

df -i

Example output:

Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1      3276800 3200000  76800   98% /

If IUse% is very high, the system may fail to create new files even though disk space appears available.

This often happens in:

Mail queue directories
Cache directories
Session directories
Temporary upload folders
Directories with many small log files
Container overlay directories

For this reason, always check both:

df -h

and:

df -i

when investigating disk problems.

---

Recommended Troubleshooting Flow for Production Systems

When you receive a disk full alert on a Linux server, avoid randomly deleting files. Follow a controlled troubleshooting flow instead.

First, check general disk usage:

df -h

Then check inode usage:

df -i

Identify which filesystem is full. If the root filesystem is full, run:

du -xhd1 /

If /var looks large:

du -xhd1 /var

If /var/log looks large:

du -xhd1 /var/log

If df shows high usage but du does not explain it, check deleted but open files:

sudo lsof -nP +L1

If Docker is used:

docker system df

To find large files on the same filesystem:

find / -xdev -type f -size +1G -exec ls -lh {} \;

To inspect mount points:

findmnt

and

lsblk

This workflow helps you understand whether the issue is caused by visible files, deleted open files, inode exhaustion, container layers, or mount point behavior.

---

What Should Be Done Instead of Deleting Large Log Files?

In production environments, deleting large log files with rm is usually not the best approach.

Wrong approach:

rm /var/log/nginx/access.log

This removes the directory entry, but if the service still holds the file open, the disk space may not be released.

A better approach is to truncate the file without deleting it:

sudo truncate -s 0 /var/log/nginx/access.log

This keeps the inode and file path intact while clearing the file content. The application can continue writing to the same file.

However, the best long-term solution is proper log rotation.

---

Why Logrotate Is Important

logrotate is one of the most important mechanisms for preventing uncontrolled log growth on Linux systems.

With a correct logrotate configuration:

Logs are rotated periodically.
Old logs are compressed.
Only a defined number of old logs are kept.
Services can be notified to reopen log files.
Disk full incidents are reduced.

Example Nginx logrotate configuration:

/var/log/nginx/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        systemctl reload nginx > /dev/null 2>&1 || true
    endscript
}

Explanation:

Directive	Meaning
daily	Rotate logs every day
rotate 14	Keep 14 old log files
compress	Compress old logs
delaycompress	Delay compression until the next rotation cycle
missingok	Do not return an error if the log file is missing
notifempty	Do not rotate empty log files
create	Create a new log file with specified permissions and ownership
postrotate	Run commands after log rotation

This ensures that log files are managed automatically and do not grow indefinitely.

---

Which Command Should Be Used in Which Situation?

Situation	Command	Description
Check general disk usage	df -h	Shows filesystem-level used and free space
Check inode usage	df -i	Shows inode consumption
Find large top-level directories	du -xhd1 /	Shows first-level directories on the same filesystem
Analyze /var usage	du -xhd1 /var	Shows large directories under /var
Find deleted but open files	sudo lsof -nP +L1	Shows deleted files still held by processes
Find large files	find / -xdev -type f -size +1G -exec ls -lh {} \;	Finds files larger than 1 GB on the same filesystem
Check Docker usage	docker system df	Shows Docker image, container, volume, and cache usage
Check detailed Docker usage	docker system df -v	Shows detailed Docker disk usage
Show mount points	findmnt	Displays mounted filesystems
Show disks and partitions	lsblk	Displays block devices and partitions
Check sparse file size	du -sh file and du -sh --apparent-size file	Compares physical and apparent size
Check ext4 reserved blocks	tune2fs -l /dev/sdX	Shows reserved block information

---

Example Case Analysis

Assume a monitoring system reports that the root filesystem is 95% full.

First check:

df -h

Output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   48G  2.0G  96% /

Then check top-level directory usage:

du -xhd1 /

Output:

5.0G    /usr
3.0G    /var
1.5G    /home
500M    /opt
10G     /

Here, there is a clear mismatch:

df shows 48 GB used.

du shows only about 10 GB used.

This strongly suggests that deleted but open files may be consuming disk space.

Check with:

sudo lsof -nP +L1

Example output:

COMMAND   PID   USER   FD   TYPE DEVICE   SIZE/OFF NLINK NODE NAME
java     2450   app    12w  REG  253,1   32212254720  0 123456 /var/log/app/application.log (deleted)

This shows that a Java application is holding a deleted log file open. The file is about 30 GB.

Possible solutions:

Restart the application:

sudo systemctl restart app-service

Or, if downtime is not acceptable and you are sure it is only a normal log file:

sudo truncate -s 0 /proc/2450/fd/12

Then verify:

df -h

The disk space should now be released.

---

How to Prevent This Problem

Although it may not be possible to prevent every disk usage issue, the risk can be greatly reduced with proper configuration and monitoring.

Recommended practices:

Configure logrotate correctly.
Set maximum log file sizes for applications.
Configure Docker log rotation with max-size and max-file.
Monitor both disk usage and inode usage.
Monitor critical directories such as /var/log, /tmp, and /var/lib/docker.
Ensure applications reopen log files after rotation.
Avoid deleting active log files with rm.
Create a standard operating procedure for disk cleanup.
Periodically check for deleted but open files with lsof +L1.

In Linux, df and du measure disk usage from different perspectives.

df shows filesystem-level block allocation.

du calculates the total size of files visible in the directory tree.

Because of this, their outputs do not always match.

One of the most common causes of a major difference between df and du is deleted files that are still open by running processes. These files are no longer visible with ls, are not counted by du, but still consume disk blocks because the kernel cannot release them until the process closes the file descriptor.

The most effective command for detecting this issue is:

sudo lsof -nP +L1

The safest way to reclaim disk space is usually to restart the related service or make it reopen its log files. If restarting is not possible, and you are sure the file is only a normal log file, you can truncate the open file descriptor through /proc/PID/fd/FD.

When troubleshooting disk full incidents, avoid randomly deleting files. Instead, identify exactly where the space is being held: visible files, deleted open files, inodes, container layers, reserved blocks, or hidden mount point data.

A reliable Linux disk investigation should use df, du, lsof, findmnt, df -i, and container-specific tools such as docker system df together.

Disk usage analysis in Linux is not only about asking “which directory is using how many GB?” The real goal is to understand which process, inode, filesystem behavior, or storage layer is actually holding the disk space.