Performance tuning basics — sysctl, ulimit, and the scheduler knobs that actually matter

Linux out of the box runs fine for most workloads. The "performance tuning" articles you find online often paste 50 sysctl lines that were optimal in 2009 — half of which are now defaults, the other half of which might hurt your workload. This article covers the handful of knobs that actually move the needle on a typical VPS, and the rest you should leave alone unless you're profiling a specific bottleneck.

The knobs worth knowing

File descriptor limits

Default per-process file descriptor limit is 1024 on most distros. For anything with lots of connections (a busy nginx, a Postfix mail server, a PBX with hundreds of registrations), this becomes a hard cap. Symptoms: "too many open files" errors in logs, connections refused for no apparent reason.

# Check current limit for a running process
cat /proc/$(pgrep -f myapp)/limits | grep "Max open files"

# Raise it for a systemd service
mkdir -p /etc/systemd/system/myapp.service.d
cat > /etc/systemd/system/myapp.service.d/override.conf <<EOF
[Service]
LimitNOFILE=65536
EOF
systemctl daemon-reload && systemctl restart myapp

For login shells (rare on a VPS, but if you have users running long-lived processes), set in /etc/security/limits.d/90-myapp.conf:

myapp soft nofile 65536
myapp hard nofile 65536

TCP connection limits

For servers receiving many connections (web, mail, PBX SIP):

# /etc/sysctl.d/99-network.conf
net.core.somaxconn = 4096                          # backlog for accept() queue
net.ipv4.tcp_max_syn_backlog = 4096                # pending half-open connections
net.core.netdev_max_backlog = 5000                 # incoming packet queue per CPU
net.ipv4.ip_local_port_range = 10000 65535         # source-port range for outbound conns

# Apply
sysctl --system

The default somaxconn = 4096 in modern kernels is reasonable; the value above is the same. Older defaults (128) are limiting for busy servers.

TIME_WAIT cleanup

Sockets in TIME_WAIT state linger for 60 seconds after the connection closes. For a server making lots of outbound short-lived connections (e.g. a proxy that re-fetches from upstream constantly, or a PBX whose calls open and close trunks quickly), the accumulation can exhaust the source-port range.

# /etc/sysctl.d/99-network.conf
net.ipv4.tcp_tw_reuse = 1                          # allow reuse of TIME_WAIT sockets
net.ipv4.tcp_fin_timeout = 30                      # shorten FIN_WAIT2 timeout

Don't enable tcp_tw_recycle — it's removed from newer kernels for breaking NAT'd clients.

vm.swappiness for database/PBX boxes

Already covered in the swap article — drop from default 60 to 10 for workloads where you want hot data staying in RAM.

Inotify watches

Applications that watch a lot of files (Mastodon, Nextcloud with file watchers, large dev environments) hit the default limit of 8192 watches per user fairly quickly.

# /etc/sysctl.d/99-inotify.conf
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 256

Knobs you don't need to touch on a modern kernel

If a tuning guide tells you to set any of these, ask yourself if the guide is from a 2014 LAMP context and whether it still applies to your kernel:

• net.ipv4.tcp_congestion_control = cubic — already the default.
• net.core.rmem_max / wmem_max at small values — autoscaling has been on by default for years.
• vm.dirty_ratio / vm.dirty_background_ratio — defaults are fine unless you have a very specific I/O bursting problem.
• kernel.shmmax — relevant for huge-shared-memory databases on old kernels; auto-scaled now.

Touching these and not noticing benefit is normal; touching them and breaking things (because the guide was wrong for your kernel version) is also normal.

BBR for high-latency / high-loss networks

If your VPS serves traffic across continents (high RTT), switching the TCP congestion control to BBR can noticeably improve throughput compared to the default cubic.

# Check current
sysctl net.ipv4.tcp_congestion_control

# Enable BBR
cat >> /etc/sysctl.d/99-bbr.conf <<EOF
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr
EOF
sysctl --system

Doesn't help if your users are mostly nearby; helps a lot if you serve a global audience over varying network quality.

How to actually measure

Tuning without measurement is folklore. Before you change anything, capture baseline:

uptime                                              # load
vmstat 1 10                                         # CPU, memory, I/O snapshot
iostat -x 1 5                                       # per-disk I/O (sysstat package)
ss -s                                               # socket summary (TIME_WAIT count)
sar -n DEV 1 5                                      # per-interface throughput

Apply one change. Re-measure. If it didn't help (or hurt something), back it out. Apply the next. The temptation to apply all the knobs from a guide at once is what produces the "I tuned X and it got slower but I don't know which knob" situation.

Persisting changes

Anything you set with sysctl -w is gone at reboot. Put persistent settings in /etc/sysctl.d/*.conf (one file per concern, numbered) so you can see what's been customized at a glance and reverse it later.

sysctl --system                          # reload all sysctl.d files
sysctl -a | grep tcp_congestion          # confirm a specific value took effect