KnowledgebaseLinux VPS › Shipping logs off the VPS — when, why, and to where

Shipping logs off the VPS — when, why, and to where

Keeping logs locally is fine until you actually need them — at which point either the disk is full and they've rotated away, or the VPS that crashed is exactly the one whose logs would have told you why. Shipping logs off-box gives you durable storage outside the box that produced them, search/aggregation across multiple servers, and a record that survives even a complete VPS loss.

When you actually need this

  • Multi-VPS setup where you want unified search across nodes.
  • Post-incident debugging — you want logs from before the crash even if the VPS itself is gone.
  • Compliance frameworks that require log retention beyond what local disk holds.
  • Security event collection (auditd, failed logins, intrusion-detection alerts) that you want isolated from the boxes generating them.

For a single-VPS hobby setup, local logs + a healthchecks integration on critical jobs is usually enough.

Option 1: hosted log aggregator (lowest friction)

Papertrail, Better Stack, Datadog Logs, Logtail, Axiom — all offer hosted syslog endpoints with free or low-cost tiers for low-volume use. Pattern:

# /etc/rsyslog.d/50-papertrail.conf (replace endpoint with the one from your provider)
*.* @logs1.papertrailapp.com:12345    # UDP
*.* @@logs1.papertrailapp.com:12345   # TCP with TLS (preferred)

# Reload
systemctl restart rsyslog

Within a minute, the provider's UI shows your logs flowing in. They handle storage, search, alerting.

Pros: zero infrastructure to maintain on your side. Cons: per-GB pricing once you exceed the free tier; lock-in to the provider's UI/search; sometimes spotty TLS support.

Option 2: self-hosted Loki + Grafana on another LYLIX VPS

Loki is the log-aggregation pair of Grafana — Prometheus-like ingestion model, scales reasonably, runs comfortably on a 2 GB VPS for small fleets. The pattern:

  1. Run Loki + Grafana on a small dedicated LYLIX VPS.
  2. On each VPS you want to ship logs from, run promtail (Loki's log shipper) as a systemd service.
  3. Configure promtail to scrape journalctl output and the files under /var/log/ you care about.

Promtail config skeleton at /etc/promtail/config.yaml:

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https://loki.example.com/loki/api/v1/push
    basic_auth:
      username: vps-shipper
      password: ${LOKI_PASS}

scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
        host: my-vps

  - job_name: apache
    static_configs:
      - targets: [localhost]
        labels:
          job: apache
          __path__: /var/log/apache2/*.log

Pros: no per-GB fees beyond your own storage; full ownership; search via Grafana's LogQL is genuinely good. Cons: another box to maintain, scaling beyond ~10 GB/day requires planning, you eat the storage cost.

Option 3: rsyslog to a second VPS (low-tech, durable)

For the "I just want a backup of logs in case this box dies" use case, rsyslog forwarding to a second VPS is the simplest possible thing. No search UI, no aggregator — just a copy of the logs on a different disk.

On the log-receiver VPS (Debian/Ubuntu):

# /etc/rsyslog.d/00-receiver.conf
module(load="imtcp")
input(type="imtcp" port="6514")

template(name="PerHost" type="string"
         string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
*.* ?PerHost
& stop

systemctl restart rsyslog

On the log-sender VPSes:

# /etc/rsyslog.d/50-forward.conf
*.* @@receiver.example.com:6514

systemctl restart rsyslog

Each sender's logs end up in /var/log/remote/<hostname>/<program>.log on the receiver. Add TLS by switching to imtcp with StreamDriverMode=1 if the logs traverse the open Internet (carrying syslog over public IP unencrypted is a leak risk).

What about journald → journald?

systemd-journald supports remote sending via systemd-journal-remote. Works fine, less commonly deployed than rsyslog forwarding, and limits you to the journald-compatible recipient (a Loki/promtail can't read the journald binary format directly). Worth knowing exists; usually rsyslog is the broader-compatibility choice.

Things to ship vs not

Don't blindly ship everything. Local logs are fine for high-volume application logs that you only need during real-time debugging. Ship the durable-record stuff:

  • auth.log / secure (logins, sudo, ssh)
  • kern.log (kernel messages, OOM kills)
  • application error logs (not access logs unless you need them centrally)
  • fail2ban, audit subsystem output

Application access logs at 10k+ requests/sec quickly become 10 GB/day; ship them only if you actually need cross-host search on them, otherwise rotate locally and archive to off-host storage cold.

Also Read

« « Back

Powered by WHMCompleteSolution