Linux Disk Space Troubleshooting

System

Kernel Images

List all installed Linux kernel images

apt list --installed 'linux-image*'

Remove specific kernel image and its dependencies

apt remove --autoremove linux-image-6.1.0-9-amd64

APT Cache

Show total size of downloaded package archives

du -sh /var/cache/apt/archives

Clean all downloaded package files (frees space immediately)

apt clean

Note: apt autoremove and apt autoclean don't always help. autoclean only removes packages that can no longer be downloaded (superseded versions), while clean wipes the entire archive cache, including current versions kept after install/upgrade. If /var/cache/apt/archives keeps growing and autoremove/autoclean don't free space, apt clean is the fix.

General .cache Directories

Find all .cache directories and show their sizes, sorted by size

find / -type d -name ".cache" -exec du -sh {} + | sort -hrk 1

This will follow into NFS/other mounted filesystems too — if it hangs or picks up remote directories you don't care about, add -xdev right after find / (see Mounted Filesystems & NFS below).

Disk Usage (du)

Show disk usage of root directory contents, suppress errors

du -sh /* 2>/dev/null

Find largest files/directories system-wide, show top 20

du -a / 2>/dev/null | sort -n -r | head -n 20

Mounted Filesystems & NFS

du, find, and lsof don't know or care about mount boundaries by default — they'll happily walk into every NFS/CIFS/bind mount they find, which either inflates your numbers or hangs the command if the remote share is slow or unresponsive.

List all mounts with filesystem type (much more readable than plain mount)

findmnt -t nfs,nfs4,cifs

Show disk usage excluding specific filesystem types (NFS, CIFS, tmpfs, etc.)

df -hT -x nfs -x nfs4 -x tmpfs -x devtmpfs

Stay on one filesystem — don't cross into any mounted filesystem (this is what actually solves the "exclude" problem, since it skips every other filesystem, not just NFS)

du -x -sh /*

Same idea for find — don't descend into mounted filesystems

find / -xdev -name '*.log' -size +100M

If a command hangs, it's very likely a stale/unresponsive NFS mount — wrap it in timeout instead of waiting it out or Ctrl+C-ing (Ctrl+C often doesn't interrupt a process stuck in NFS I/O wait, and even kill -9 may not)

timeout 10 du -sh /mnt/some-nfs-share

Check for processes stuck in D-state (uninterruptible sleep), usually caused by a dead/unresponsive NFS server

ps aux | awk '$8 ~ /D/'

If the NFS server itself is ZFS (e.g. a zpool0 backing your NFS export): deleting files on the client won't necessarily free space on the server if a ZFS snapshot still references those blocks — this is the number one "I deleted files but it's still full" surprise on ZFS-backed NFS exports.

# run on the ZFS host, not the NFS client
zfs list -o name,used,avail,refer -r zpool0
zfs list -t snapshot -o name,used -s used

Inodes

Disk can show free space but writes still fail with "No space left on device" — that means you're out of inodes, not blocks

df -i

Find which directories are eating the most inodes (most files) — common with mail queues, session caches, or dirs full of tiny files

find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -n 20

Reserved Blocks (ext4)

ext4 reserves 5% of the filesystem for root by default — on a large disk that can be many GB counted as "used" by df but not reclaimable by a normal user

Check the current reserved percentage/block count

tune2fs -l /dev/sdX | grep -i reserved

Lower it (e.g. to 1%) to reclaim space — only do this on non-root/non-boot filesystems unless you know exactly what you're doing

tune2fs -m 1 /dev/sdX

ncdu (Interactive Alternative)

Plain du output is tedious to scan; ncdu gives a navigable, sorted tree and, like du -x, stays on one filesystem by default

apt install ncdu
ncdu -x /

Deleted Files Still Held Open

List open files that have been deleted (zero link count) but still hold disk space

lsof +L1

Summarize space by process name

lsof +L1 | awk 'NR>1 {sum[$1]+=$7} END {for (i in sum) printf "%-20s %.2f MB\n", i, sum[i]/1024/1024}' | sort -rnk 2

Identify culprit files by process name

lsof +L1 | grep "vivaldi-b" | sort -rnk 7 | head -n 10

Summarize space by PID

lsof +L1 | awk 'NR>1 {sum[$1" (PID:"$2")"]+=$7} END {for (i in sum) printf "%-30s %.2f MB\n", i, sum[i]/1024/1024}' | sort -rnk 3

Logs

systemd Journal

Show total disk space used by systemd journal

journalctl --disk-usage

To apply a rule one time, use commands like below.

Clean journal logs older than 14 days

journalctl --vacuum-time=14d

Limit journal logs to 512MB total size

journalctl --vacuum-size=512M

Keep only 3 most recent journal files

journalctl --vacuum-files=3

But it's much better to use /etc/systemd/journald.conf for permanent rules

[Journal]
Compress=yes
SystemMaxUse=512M
SystemMaxFileSize=256M
SystemMaxFiles=7

Restart journald to apply the new configuration

systemctl restart systemd-journald

Find Large Log Files

Find log files larger than 100MB and display their details

find / -name '*.log' -size +100M -exec ls -lh {} \;

Truncate a Log Without Restarting the Service

If a service holds a log file open and keeps writing to it, rm-ing the file won't free space until the process restarts (see Deleted Files Still Held Open above). Truncating in place frees the space immediately, no restart needed

truncate -s 0 /var/log/some-huge.log
# or
: > /var/log/some-huge.log

Docker

Container Logs

Find Docker container log files and sort by size (largest first)

find /var/lib/docker -type f -name "*json.log" -exec ls -lhS {} + | sort -hrk 5

Let's break it down:

• -type f: look for files only.
• -name "*json.log": match Docker container log filenames.
• -exec ls -lhS {} +: for each file found, run ls -lhS (-l long format, -h human-readable sizes, -S sort by size, {} placeholder, + batches as many paths as possible per invocation).
• | sort -hrk 5: re-sort the output — -h human-readable numeric sort, -r descending, -k 5 by the 5th column (file size in ls -lh output).

System-wide Docker Usage & Cleanup

Breakdown of space used by images, containers, local volumes, and build cache

docker system df -v

Remove unused data (stopped containers, dangling images, unused networks, build cache) — doesn't touch volumes or images still referenced by a container

docker system prune

More aggressive: also removes all unused images (not just dangling ones) and all unused volumes — this will delete data in volumes not attached to a running container, double-check first

docker system prune -a --volumes

List dangling (unnamed, unreferenced) volumes before deciding whether to remove them

docker volume ls -f dangling=true

Remove All Docker Data

systemctl stop docker docker.socket
rm -rf /var/lib/docker
systemctl start docker