Rescue mode: recovering from a broken kernel or SSH lockout

Rescue mode boots your VPS from a SystemRescue ISO instead of its own disk. The disk is still attached, untouched, and ready to be mounted from the rescue environment — so you can fix whatever broke without losing data. This article covers when to reach for rescue, the exact button sequence, and three common recovery patterns (broken kernel, corrupt fstab, lost root password).

What rescue mode does

Your VM's boot order is temporarily changed to boot from a SystemRescue ISO attached as a CD-ROM. The ISO is a current Linux live distribution with all the standard recovery tools: parted, fdisk, mount, chroot, gpg, nano, fsck.*, tune2fs, cryptsetup, SSH, etc. Your disk shows up as a normal block device (/dev/vda or similar) for you to inspect and repair.

Cloud-init data stays attached, so the rescue environment comes up with your IP already configured — no need to manually ip addr add. SSH works the moment the rescue image is up.

When you're done, you exit rescue mode from the portal; the boot order is restored and the VM reboots back into its own OS.

When to use rescue mode

• The VM won't boot at all — kernel panic, bootloader broken, "no bootable device", or the disk fails an early fsck.
• You broke /etc/fstab and the VM hangs at boot trying to mount something that doesn't exist.
• You broke /etc/ssh/sshd_config AND lost the browser-console root password too — rescue gives you root on the rescue env, then chroot into your actual install.
• Forgot the root password and don't want to do the GRUB-edit dance from the browser console.
• Filesystem corruption requiring fsck on an unmounted root.
• Distro upgrade went sideways and the system won't come back up.

How to enter rescue mode

1. Sign in to customer.lylix.net, open your service.
2. Click the Rescue tab.
3. Click Boot into rescue ISO.
4. Confirm. The VM will reboot into the rescue environment (about a minute).
5. SSH in:

   ssh root@<your-IP>

   SystemRescue's default root password is root by default; check the portal for a session-specific override if present.

You can also use the browser console to log in directly without SSH, useful if you're not sure the rescue env's network is up yet.

Pattern 1: Fix a broken fstab

Symptom: VM hangs at boot with messages about waiting for a device or "dependency failed for /something".

1. Boot into rescue, SSH in.
2. Identify your root partition:

   lsblk
   # Look for the disk's biggest partition, typically /dev/vda1 or
   # /dev/vda2. If LVM, look for /dev/mapper/* entries.

3. Mount it:

   mount /dev/vda1 /mnt
   # If LVM:
   vgchange -ay
   mount /dev/mapper/vg-root /mnt

4. Edit fstab:

   nano /mnt/etc/fstab
   # Comment out or remove the bad line.

5. Unmount and reboot back to normal:

   umount /mnt
   # In the portal: Rescue tab → Exit rescue mode

Pattern 2: Fix broken sshd_config + reset root password

Symptom: SSH refuses connections after a config change AND you can't log in via the browser console either.

1. Boot into rescue, SSH into the rescue env.
2. Mount root and chroot in:

   mount /dev/vda1 /mnt
   mount --bind /dev /mnt/dev
   mount --bind /proc /mnt/proc
   mount --bind /sys /mnt/sys
   chroot /mnt /bin/bash

3. Inside the chroot, fix sshd and reset the password:

   # Restore a sensible sshd config:
   nano /etc/ssh/sshd_config
   # Make sure these are right:
   #   PermitRootLogin yes      (temporarily, to recover)
   #   PasswordAuthentication yes
   # Test:
   sshd -t
   
   # Reset root password:
   passwd
   
   # Exit the chroot:
   exit

4. Unmount and reboot:

   umount /mnt/dev /mnt/proc /mnt/sys
   umount /mnt
   # Portal: Rescue tab → Exit rescue mode

Pattern 3: Run fsck on an unmounted root filesystem

Symptom: VM keeps boot-looping with fsck errors, or you suspect disk corruption.

1. Boot into rescue, SSH in.
2. Identify the filesystem (don't mount it):

   lsblk -f
   # Note the FSTYPE — ext4, xfs, btrfs, etc.

3. Run the appropriate fsck:

   # ext2/3/4:
   fsck.ext4 -y -f /dev/vda1
   
   # xfs (needs xfs_repair, not fsck):
   xfs_repair /dev/vda1
   
   # btrfs:
   btrfs check --repair /dev/vda1   # use cautiously

4. Exit rescue, reboot to normal. If fsck found and fixed errors, watch the next boot carefully — re-corruption within a day or two suggests the disk itself has a problem (open a ticket).

Exiting rescue mode

1. From the portal, click Rescue tab again.
2. Click Exit rescue mode.
3. The VM stops, boot order is reset, and it boots back into its own disk.

The portal shows a banner while rescue mode is active, so you can't forget you're in it.

Things to know

• Rescue is non-destructive by default. Just booting into rescue doesn't modify your disk. You have to explicitly mount and write to make changes.
• Network up automatically. Cloud-init gives the rescue env your assigned IPs, so SSH works immediately.
• Local SSH known_hosts — the rescue env has different SSH host keys than your normal install. Your local machine will complain about a key mismatch. Use ssh -o StrictHostKeyChecking=no or temporarily remove the host-key entry, and remember to remove it from known_hosts again when you exit rescue.
• Don't make permanent OS changes here. The rescue env itself is ephemeral — anything you install or configure in it disappears when you exit. Make changes inside the chroot (your real install), not in the rescue environment.

Stuck? Open a ticket from the portal and include what you saw at boot (browser-console screenshot helps). We've seen most of the failure modes; the typical exchange resolves in one or two replies.