Rescuing Linux

Sometimes you need perform irregular, unpleasant tasks involving a Linux box. One generally picks up the techniques via experience. This kind of thing tends to become less necessary as one learns Linux better, and thus it's easy to lose currency. The introduction of systemd, for instance, changed almost all of this shit up. Almost everything here requires access to the console, relevant only long before sshd is running (though take a look at e.g. RescueInitramfs).

firmware

System firmware configuration can usually be entered by holding one of Delete, F2, or F1. A boot menu can commonly be accessed with F10.

grub

If there is no boot delay, but you need interactive control, try holding down space (or with some versions shift) when grub comes up.

Don't go editing grub's configuration files directly if at all possible. Edit /etc/default/grub and rebuild them with update-grub. Note that update-grub2 is these days just a symlink to update-grub. If what you want to change isn't available via /etc/default/grub, try to do it via /etc/grub.d. If you're directly editing grub.cfg, you're gonna have a bad time (and your changes will be blown away the next time someone runs update-grub).

Do yourself a favor and install memtest86+, which will be automatically added to your grub menu on UEFI machines.

If you're in the interactive GRUB command prompt, and wish not to be (i.e. to return to the menu), use the command normal.

kernel command line

The bootloader can provide command line parameters to the kernel (they can also be specified at build time, see CONFIG_CMDLINE). systemd can take many parameters off of the kernel command line.

For more output, ensure "quiet" is not present. For still more output, add "debug=vc" (by default, debug writes to /run/initramfs/initramfs.debug; this sends it to the console).

The root process can be specified using init= (this can also be specified at build time with CONFIG_DEFAULT_INIT).

Note that the kernel does not by default reboot following a panic. This can be undesirable on remote machines. panic=N will reboot N seconds after a panic, if N is positive. /proc/sys/kernel/panic exposes this.

my video is borked

If you can't get output to the display, netconsole is a pretty decent method for quickly getting console output sent to another machine. You won't have ARP or routing, so the target will need be on the local broadcast domain. Specify the target MAC if you can to avoid broadcasting.

initramfs

Almost all distributions ship kernels making use of initramfs these days. An initramfs can be embedded directly into the kernel image (see kernel config entry CONFIG_INITRAMFS_SOURCE), but it is usually shipped as its own file instead, and specified by the bootloader. An initramfs is a (possibly compressed) cpio archive. On boot, it is unpacked into a tmpfs. The compelling advantage of initramfs is the ability to mount the true root filesystem (which might be on NFS, or encrypted, etc.) from userspace, with a minimal filesystem such as userspace expects. Over time, presence of an initramfs has become more or less assumed, and is is thus now required for all manner of things (i.e. persistent block device names when specifying the root filesystem). In the absence of an initramfs, all code necessary for mounting root must be built into the kernel (i.e. not as modules).

The initramfs-tools-core package ships lsinitramfs and unmkinitramfs to easily list or extract the contents of an initramfs file.

You're unlikely to run into initrd these days, but it can be unpacked the same way (the difference is in how it's mounted during boot). Initramfs on Debian are named initrd-*.

If built into the kernel, there are no extra considerations for the initramfs. If it's a distinct file, it needs to live somewhere visible to the bootloader. If booting directly from UEFI, it needs live in the ESP. So long as it's kept in the same directory as the kernel image, you ought be fine. Be sure to copy the initramfs along with the kernel image if you're ever backing up the kernel, or moving it to another machine, etc.

The initramfs often has copies of various kernel modules, so most changes to modules require an initramfs rebuild.

writing an initramfs

mkinitramfs is a lower-level tool usually called via update-initramfs, controlled by the many configuration files in /etc/initramfs-tools/.

initramfs can't mount root

An unpleasant situation is one where initramfs fails to mount the root partition, in which case you will be dumped to the dreaded BusyBox or klibc shells (ash, as in "a shitty shell"). A day when one sees BusyBox is never a good day. If the machine is remote, you are fucked without server-style out of band access (e.g. Dell iDRAC, BMC, KVM-over-IP). Otherwise, if you have a valid root partition somewhere, you can manually continue the boot by mounting that partition to /mnt/root and running exec switch_root /mnt/root /sbin/init or its non-union equivalent. Usually this means you've specified the wrong root partition in your bootloader; check the root command line option to the kernel.

access sans password

Ideally, you can just provide init=/bin/sh on the kernel command line using grub's interactive mode. This will go through the end of the initramfs and invoke /bin/sh within the root filesystem.

what if we can't use interactive mode?

systemd emergency mode with a locked root account

If you've locked the root account with passwd -l, systemd's sulogin won't let you enter rescue mode. Use the instructions from "access sans password" above to get a shell if possible, and enable root with passwd. Reboot, and sulogin will admit you using the new password.

all my binaries are gone

The situation can arise that one has an active process context (we'll assume a shell) on a machine where the standard POSIX binaries are unavailable. This can happen due to e.g. a network error, a disk error, memory error, wayward rm -rf /, misadventures in package management. What can be done in such a case? Shell builtins, sysfs, /dev entries, and proc usually continue to be available.

cat without cat

The bash shell's echo or printf builtin can be used:

[freebird](0) $ dmesg
-bash: /usr/bin/dmesg: Input/output error
[freebird](126) $ printf "%s" "$(</proc/kmsg)"
.....
[freebird](0) $ echo "$(</proc/kmsg)"
.....
[freebird](0) $

reboot without reboot

Enable and apply sysrq reboot:

echo 128 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

Alt+SysRq+B will accomplish this at the terminal if sysrq reboot is enabled. Unlike reboot, there will be no syncing and unmounting of disks.

moving a root filesystem + bootloader

Ideally, mount the source filesystem ro, so it's not being modified while you copy. Start assuming a clean destination filesystem (otherwise, wipe it). Using the -a flag, cp basic toplevel directories *but do not cross filesystem boundaries on the source* (i.e. if /home is on its own filesystem, do not include it below; this includes /boot/efi). Any that you skip in this fashion ought have their mountpoints created; add them to the mkdir below:

cp -a SOURCE/{bin,boot,etc,home,lib*,root,sbin,usr} DEST/
mkdir -p DEST/{dev,proc,var/tmp,tmp}
ln -s /run DEST/var/run
ln -s /lock DEST/var/lock
chmod 1777 DEST/{var/tmp,tmp}

ESP, bootloader, initramfs...

External links

• Debian wiki page for initramfs
• Kernel documentation for Ramfs, Rootfs, and Initramfs