Disk full recovery on a Linux VPS

A full disk on a VPS cascades fast: new SSH sessions can't write to ~/.ssh, services can't write logs, MariaDB refuses inserts, mail spool stops accepting deliveries. This article covers fast recovery, root-cause investigation, and what to set up so it doesn't happen again.

Step 1 — Get a shell (even if SSH is broken)

If SSH won't connect — "Connection closed" right after login banner, or hangs entirely — the disk is too full for sshd to set up a session. Open the browser console from the LYLIX portal (VPS details → Console). The browser console talks straight to the hypervisor and doesn't need any disk space to start.

Log in as root (or your sudo user) at the console prompt and proceed from there.

Step 2 — Confirm what's full

df -h          # disk usage by mount — which partition is at 100%?
df -ih         # inode usage — rare but possible (millions of tiny files)

If df -h shows the root filesystem at 100%, you're in the standard case. If a non-root mount is full (e.g. /var on its own partition), focus cleanup there.

If df -ih shows 100% Iuse but df -h has plenty of bytes free, you've exhausted inodes — usually millions of small files in one directory (mail queue, session files, log files that never rotate). Find with find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -n | tail.

Step 3 — Fast wins (try in order)

Each of these reliably frees space without risk to running services:

# 1. Trim systemd journal — typically the biggest single recovery
journalctl --vacuum-size=200M

# 2. Old compressed logs (rotated logs older than 30 days)
find /var/log -name '*.gz' -mtime +30 -delete
find /var/log -name '*.[0-9]' -mtime +30 -delete

# 3. Package manager caches
apt clean                                      # Debian/Ubuntu
dnf clean all                                  # AlmaLinux

# 4. /tmp — only obvious junk; sockets and lockfiles must stay
find /tmp -type f -mtime +7 -delete 2>/dev/null

# 5. Crash dumps if any
rm -rf /var/crash/* 2>/dev/null

Re-check with df -h. If you're back below 90%, move to Step 5 to prevent recurrence. If you're still tight, find the actual culprit.

Step 4 — Find what's actually eating the disk

# Biggest top-level dirs
du -sh /* 2>/dev/null | sort -h | tail -10

# Drill into the biggest one (usually /var or /home)
du -sh /var/* 2>/dev/null | sort -h | tail -10

# Single largest files anywhere
find / -xdev -type f -size +100M 2>/dev/null | xargs -r ls -lh | sort -k5 -h | tail -20

If you have ncdu installed (apt/dnf install ncdu), ncdu -x / gives an interactive tree view that's much faster than repeated du.

PBX VPS — common culprits

• /var/spool/asterisk/monitor/ — call recordings. Months of recordings accumulate fast. Off-host transfer + delete from local (see the call recording article).
• /var/spool/asterisk/voicemail/ — voicemail files (.wav + transcripts).
• /var/log/asterisk/ — Asterisk® log files, especially full and messages.
• /var/lib/mysql/mysql-bin.* — MariaDB binary logs. Do not rm these; from the MariaDB CLI run PURGE BINARY LOGS BEFORE NOW() - INTERVAL 7 DAY;.
• /var/spool/asterisk/backup/ — backups created by the PBX backup module. Old ones can be deleted from the GUI or directly.

Mail server — common culprits

• /var/spool/postfix/deferred/ — held mail (often huge after an outage). postsuper -d ALL deferred drops it all; postsuper -d ALL drops everything in the queue. Use carefully.
• /var/mail/ or /home/*/Maildir/ — IMAP storage.
• /var/log/mail.log — extremely chatty under spam attack; rotate aggressively.

Docker hosts — common culprits

• /var/lib/docker/overlay2/ — image and container layers.
• docker image prune -a — removes images not referenced by any container.
• docker system prune -af --volumes — nuclear option; removes stopped containers, unused networks, dangling images, and unused volumes. Don't run this on a production stack without knowing what you'll lose.

Step 5 — Prevent recurrence

• Cap journald size: edit /etc/systemd/journald.conf, set SystemMaxUse=500M, then systemctl restart systemd-journald. journald will never exceed that.
• Verify logrotate is actually running: systemctl status logrotate.timer on systemd distros. If disabled, enable it. Check /var/lib/logrotate/status for last-run timestamps.
• Monitor disk usage from the LYLIX portal's resource graphs. Set an external alert (Uptime Robot, Healthchecks.io) that pings the VPS for a df output and warns at 80%.
• Move large append-only data off-host — call recordings, voicemail archives, mail archives. Restic or rclone to S3-compatible storage. Local disk should hold working set, not archives.
• Plan upgrade adds disk. If your steady-state usage is genuinely outgrowing the plan, upgrade — disk grows on next reboot (CPU/RAM upgrades take effect on next reboot too, per the upgrading-vps-plan article).

What NOT to delete

• Anything in /var/lib/mysql/ or /var/lib/postgresql/ — corrupts the database.
• /var/lib/dpkg/ or /var/lib/rpm/ — breaks the package manager.
• /boot/ contents — kernel + initrd, needed to boot.
• /var/log/wtmp, /var/log/btmp, /var/log/lastlog — login history; truncate with : > /var/log/wtmp instead of rm if you must.

When in doubt: stop services that write to disk, take a snapshot, then experiment. A snapshot rollback is cheap; an accidentally deleted MariaDB datafile is not.