Rescuing Linux

Sometimes you need perform irregular, unpleasant tasks involving a Linux box. One generally picks up the techniques via experience. This kind of thing tends to become less necessary as one learns Linux better, and thus it's easy to lose currency. The introduction of systemd, for instance, changed almost all of this shit up. Almost everything here requires access to the console, relevant only long before sshd is running (though take a look at e.g. RescueInitramfs).

grub

If there is no boot delay, but you need interactive control, try holding down space when grub comes up.

Don't go editing grub's configuration files directly if at all possible. Edit /etc/default/grub and rebuild them with update-grub. Note that update-grub2 is these days just a symlink to update-grub. If what you want to change isn't available via /etc/default/grub, try to do it via /etc/grub.d. If you're directly editing grub.cfg, you're gonna have a bad time (and your changes will be blown away the next time someone runs update-grub).

Do yourself a favor and install memtest86+, which will be automatically added to your grub menu on UEFI machines.

kernel command line

The bootloader can provide command line parameters to the kernel (they can also be specified at build time, see CONFIG_CMDLINE). systemd can take many parameters off of the kernel command line.

For more output, ensure "quiet" is not present. For still more output, add "debug=vc" (by default, debug writes to /run/initramfs/initramfs.debug; this sends it to the console).

The root process can be specified using init= (this can also be specified at build time with CONFIG_DEFAULT_INIT).

Note that the kernel does not by default reboot following a panic. This can be undesirable on remote machines. panic=N will reboot N seconds after a panic, if N is positive. /proc/sys/kernel/panic exposes this.

initramfs

Almost all distributions ship kernels making use of initramfs these days. An initramfs can be embedded directly into the kernel image (see kernel config entry CONFIG_INITRAMFS_SOURCE), but it is usually shipped as its own file instead, and specified by the bootloader. An initramfs is a (possibly compressed) cpio archive. On boot, it is unpacked into a tmpfs. The compelling advantage of initramfs is the ability to mount the true root filesystem (which might be on NFS, or encrypted, etc.) from userspace, with a minimal filesystem such as userspace expects. Over time, presence of an initramfs has become more or less assumed, and is is thus now required for all manner of things (i.e. persistent block device names when specifying the root filesystem). In the absence of an initramfs, all code necessary for mounting root must be built into the kernel (i.e. not as modules).

The initramfs-tools-core package ships lsinitramfs and unmkinitramfs to easily list or extract the contents of an initramfs file.

You're unlikely to run into initrd these days, but it can be unpacked the same way (the difference is in how it's mounted during boot). Initramfs on Debian are named initrd-*.

If built into the kernel, there are no extra considerations for the initramfs. If it's a distinct file, it needs to live somewhere visible to the bootloader. If booting directly from UEFI, it needs live in the ESP. So long as it's kept in the same directory as the kernel image, you ought be fine. Be sure to copy the initramfs along with the kernel image if you're ever backing up the kernel, or moving it to another machine, etc.

The initramfs often has copies of various kernel modules, so most changes to modules require an initramfs rebuild.

writing an initramfs

mkinitramfs is a lower-level tool usually called via update-initramfs, controlled by the many configuration files in /etc/initramfs-tools/.

initramfs can't mount root

An unpleasant situation is one where initramfs fails to mount the root partition, in which case you will be dumped to the dreaded BusyBox or klibc shells (ash, as in "a shitty shell"). A day when one sees BusyBox is never a good day. If the machine is remote, you are fucked without server-style out of band access (e.g. Dell iDRAC, BMC, KVM-over-IP). Otherwise, if you have a valid root partition somewhere, you can manually continue the boot by mounting that partition to /mnt/root and running exec switch_root /mnt/root /sbin/init or its non-union equivalent. Usually this means you've specified the wrong root partition in your bootloader; check the root command line option to the kernel.

all my binaries are gone

The situation can arise that one has an active process context (we'll assume a shell) on a machine where the standard POSIX binaries are unavailable. This can happen due to e.g. a network error, a disk error, memory error, wayward rm -rf /, misadventures in package management. What can be done in such a case? Shell builtins, sysfs, /dev entries, and proc usually continue to be available.

cat without cat

The bash shell's echo or printf builtin can be used:

[freebird](0) $ dmesg
-bash: /usr/bin/dmesg: Input/output error
[freebird](126) $ printf "%s" "$(</proc/kmsg)"
.....
[freebird](0) $ echo "$(</proc/kmsg)"
.....
[freebird](0) $

reboot without reboot

Enable and apply sysrq reboot:

echo 128 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

Alt+SysRq+B will accomplish this at the terminal if sysrq reboot is enabled. Unlike reboot, there will be no syncing and unmounting of disks.

External links

• Debian wiki page for initramfs
• Kernel documentation for Ramfs, Rootfs, and Initramfs