Rescuing Linux

(21 intermediate revisions by the same user not shown)

Line 1:

Sometimes you need perform irregular, unpleasant tasks involving a Linux box. One generally picks up the techniques via experience. This kind of thing tends to become less necessary as one learns Linux better, and thus it's easy to lose currency. The introduction of [[systemd]], for instance, changed almost all of this shit up.

Sometimes you need perform irregular, unpleasant tasks involving a Linux box. One generally picks up the techniques via experience. This kind of thing tends to become less necessary as one learns Linux better, and thus it's easy to lose currency. The introduction of [[systemd]], for instance, changed almost all of this shit up. Almost everything here requires access to the console, relevant only long before sshd is running (though take a look at e.g. [https://wiki.debian.org/RescueInitramfs RescueInitramfs]).

==firmware==

System firmware configuration can usually be entered by holding one of Delete, F2, or F1. A boot menu can commonly be accessed with F10. All of these are of course motherboard-specific.

If you need clear a firmware password, try removing the battery and unplugging the machine for 30s.

<tt>systemctl reboot --firmware-setup</tt> will attempt to reboot into firmware configuration.

==grub==

If there is no boot delay, but you need interactive control, try holding down space when grub comes up.

If there is no boot delay, but you need interactive control, try holding down space (or with some versions shift) when grub comes up.

Don't go editing grub's configuration files directly if at all possible. Edit <tt>/etc/default/grub</tt> and rebuild them with <tt>update-grub</tt>. Note that <tt>update-grub2</tt> is these days just a symlink to <tt>update-grub</tt>. If what you want to change isn't available via <tt>/etc/default/grub</tt>, try to do it via <tt>/etc/grub.d</tt>. If you're directly editing <tt>grub.cfg</tt>, you're gonna have a bad time (and your changes will be blown away the next time someone runs <tt>update-grub</tt>).

Do yourself a favor and install [https://packages.debian.org/sid/memtest86+ memtest86+], which will be automatically added to your grub menu on [[UEFI]] machines.

~~==access sans password==~~

If you're in the interactive GRUB command prompt, and wish not to be (i.e. to return to the menu), use the command <tt>normal</tt>.

==kernel command line==

The bootloader can provide command line parameters to the kernel (they can also be specified at build time, see <tt>CONFIG_CMDLINE</tt>). ~~[[systemd]]~~ can ~~take many parameters off of~~ the ~~kernel command line~~.

The bootloader can provide command line parameters to the kernel (they can also be specified at build time, see <tt>CONFIG_CMDLINE</tt>). GRUB supports providing them in interactive mode. With UEFI firmware, they can be specified in the EFI shell.

For more output, ensure "quiet" is not present. For still more output, add "debug".

[[systemd]] can take many parameters off of the kernel command line, including <tt>systemd.unit=</tt> to specify a boot target.

For more output, ensure "quiet" is not present. For still more output, add "debug=vc" (by default, <tt>debug</tt> writes to <tt>/run/initramfs/initramfs.debug</tt>; this sends it to the console).

The root process can be specified using <tt>init=</tt> (this can also be specified at build time with <tt>CONFIG_DEFAULT_INIT</tt>).

Note that the kernel does not by default reboot following a panic. This can be undesirable on remote machines. <tt>panic=N</tt> will reboot N seconds after a panic, if N is positive. <tt>/proc/sys/kernel/panic</tt> exposes this.

Modules can be blacklisted with <tt>modprobe.blacklist=</tt>.

===my video is borked===

If you can't get output to the display, [https://www.kernel.org/doc/html/latest/networking/netconsole.html netconsole] is a pretty decent method for quickly getting console output sent to another machine. You won't have ARP or routing, so the target will need be on the local broadcast domain. Specify the target MAC if you can to avoid broadcasting.

==initramfs==

Line 28:

Line 48:

===initramfs can't mount root===

An unpleasant situation is one where initramfs fails to mount the root partition, in which case you will be dumped to the dreaded BusyBox or klibc shells (ash, as in "a shitty shell"). A day when one sees BusyBox is never a good day. If the machine is remote, you are fucked without server-style out of band access (e.g. Dell iDRAC, BMC, KVM-over-IP). Otherwise, if you have a valid root partition somewhere, you can manually continue the boot by mounting that partition to <tt>/mnt/root</tt> and running <tt>exec switch_root /mnt/root /sbin/init</tt>. Usually this means you've specified the wrong root partition in your bootloader; check the <tt>root</tt> command line option to the kernel.

An unpleasant situation is one where initramfs fails to mount the root partition, in which case you will be dumped to the dreaded BusyBox or klibc shells (ash, as in "a shitty shell"). A day when one sees BusyBox is never a good day. If the machine is remote, you are fucked without server-style out of band access (e.g. Dell iDRAC, BMC, KVM-over-IP). Otherwise, if you have a valid root partition somewhere, you can manually continue the boot by mounting that partition to <tt>/mnt/root</tt> and running <tt>exec switch_root /mnt/root /sbin/init</tt> or its non-union equivalent. Usually this means you've specified the wrong root partition in your bootloader; check the <tt>root</tt> command line option to the kernel.

If you have [[systemd]] running in the emergency shell (its <tt>emergency.target</tt>), try manually mounting the root filesystem to <tt>/sysroot</tt> and then running <tt>systemctl restart mount-sysroot</tt>. If this succeeds, exiting the shell ought continue the boot.

==fsck on boot==

this doesn't happen much anymore since most everyone's running at least ext3, but ext2 required <tt>fsck</tt> on boot when not cleanly unmounted, and every so often just to make sure things hadn't shifted around.

if you don't want a filesystem to block booting under [[systemd]], use <tt>nofail</tt> in <tt>/etc/fstab</tt>.

==access sans password==

Ideally, you can just provide <tt>init=/bin/sh</tt> on the kernel command line using grub's interactive mode. This will go through the end of the initramfs and invoke <tt>/bin/sh</tt> within the root filesystem.

<b>what if we can't use interactive mode?</b>

===systemd emergency mode with a locked root account===

If you've locked the root account with <tt>passwd -l</tt>, systemd's <tt>sulogin</tt> won't let you enter rescue mode. Use the instructions from "access sans password" above to get a shell if possible, and enable root with <tt>passwd</tt>. Reboot, and <tt>sulogin</tt> will admit you using the new password.

==all my binaries are gone==

The situation can arise that one has an active process context (we'll assume a shell) on a machine where the standard POSIX binaries are unavailable. This can happen due to e.g. a network error, a disk error, memory error, wayward <tt>rm -rf /</tt>, misadventures in package management. What can be done in such a case? Shell builtins, [[sysfs]], <tt>/dev</tt> entries, and [[proc]] usually continue to be available.

===cat without cat===

The bash shell's <tt>echo</tt> or <tt>printf</tt> builtin can be used:

<pre>

[freebird](0) $ dmesg

-bash: /usr/bin/dmesg: Input/output error

[freebird](126) $ printf "%s" "$(</proc/kmsg)"

.....

[freebird](0) $ echo "$(</proc/kmsg)"

.....

[freebird](0) $

</pre>

===reboot without reboot===

Enable and apply sysrq reboot:

<pre>echo 128 > /proc/sys/kernel/sysrq

echo b > /proc/sysrq-trigger</pre>

Alt+SysRq+B will accomplish this at the terminal if sysrq reboot is enabled. Unlike <tt>reboot</tt>, there will be no syncing and unmounting of disks.

==moving a root filesystem + bootloader==

Ideally, mount the source filesystem <tt>ro</tt>, so it's not being modified while you copy. Start assuming a clean destination filesystem (otherwise, wipe it). Using the <tt>-a</tt> flag, <tt>cp</tt> basic toplevel directories *but do not cross filesystem boundaries on the source* (i.e. if <tt>/home</tt> is on its own filesystem, do not include it below; this includes <tt>/boot/efi</tt>). Any that you skip in this fashion ought have their mountpoints created; add them to the <tt>mkdir</tt> below:

<pre>

cp -a SOURCE/{bin,boot,etc,home,lib*,root,sbin,usr} DEST/

mkdir -p DEST/{dev,proc,var/tmp,tmp}

ln -s /run DEST/var/run

ln -s /lock DEST/var/lock

chmod 1777 DEST/{var/tmp,tmp}

</pre>

<b>ESP, bootloader, initramfs...</b>

==reassembling a degraded MDRAID==

...

==External links==

* Debian wiki page for [https://wiki.debian.org/initramfs initramfs]

* Kernel documentation for [https://www.kernel.org/doc/html/latest/filesystems/ramfs-rootfs-initramfs.html Ramfs, Rootfs, and Initramfs]

Rescuing Linux: Difference between revisions

Dank (talk | contribs)

@@ Line 1: / Line 1: @@
-Sometimes you need perform irregular, unpleasant tasks involving a Linux box. One generally picks up the techniques via experience. This kind of thing tends to become less necessary as one learns Linux better, and thus it's easy to lose currency. The introduction of [[systemd]], for instance, changed almost all of this shit up.
+Sometimes you need perform irregular, unpleasant tasks involving a Linux box. One generally picks up the techniques via experience. This kind of thing tends to become less necessary as one learns Linux better, and thus it's easy to lose currency. The introduction of [[systemd]], for instance, changed almost all of this shit up. Almost everything here requires access to the console, relevant only long before sshd is running (though take a look at e.g. [https://wiki.debian.org/RescueInitramfs RescueInitramfs]).
+==firmware==
+System firmware configuration can usually be entered by holding one of Delete, F2, or F1. A boot menu can commonly be accessed with F10. All of these are of course motherboard-specific.
+If you need clear a firmware password, try removing the battery and unplugging the machine for 30s.
+<tt>systemctl reboot --firmware-setup</tt> will attempt to reboot into firmware configuration.
 ==grub==
-If there is no boot delay, but you need interactive control, try holding down space when grub comes up.
+If there is no boot delay, but you need interactive control, try holding down space (or with some versions shift) when grub comes up.
+Don't go editing grub's configuration files directly if at all possible. Edit <tt>/etc/default/grub</tt> and rebuild them with <tt>update-grub</tt>. Note that <tt>update-grub2</tt> is these days just a symlink to <tt>update-grub</tt>. If what you want to change isn't available via <tt>/etc/default/grub</tt>, try to do it via <tt>/etc/grub.d</tt>. If you're directly editing <tt>grub.cfg</tt>, you're gonna have a bad time (and your changes will be blown away the next time someone runs <tt>update-grub</tt>).
+Do yourself a favor and install [https://packages.debian.org/sid/memtest86+ memtest86+], which will be automatically added to your grub menu on [[UEFI]] machines.
-==access sans password==
+If you're in the interactive GRUB command prompt, and wish not to be (i.e. to return to the menu), use the command <tt>normal</tt>.
 ==kernel command line==
-The bootloader can provide command line parameters to the kernel (they can also be specified at build time, see <tt>CONFIG_CMDLINE</tt>). [[systemd]] can take many parameters off of the kernel command line.
+The bootloader can provide command line parameters to the kernel (they can also be specified at build time, see <tt>CONFIG_CMDLINE</tt>). GRUB supports providing them in interactive mode. With UEFI firmware, they can be specified in the EFI shell.
-For more output, ensure "quiet" is not present. For still more output, add "debug".
+[[systemd]] can take many parameters off of the kernel command line, including <tt>systemd.unit=</tt> to specify a boot target.
+For more output, ensure "quiet" is not present. For still more output, add "debug=vc" (by default, <tt>debug</tt> writes to <tt>/run/initramfs/initramfs.debug</tt>; this sends it to the console).
 The root process can be specified using <tt>init=</tt> (this can also be specified at build time with <tt>CONFIG_DEFAULT_INIT</tt>).
+Note that the kernel does not by default reboot following a panic. This can be undesirable on remote machines. <tt>panic=N</tt> will reboot N seconds after a panic, if N is positive. <tt>/proc/sys/kernel/panic</tt> exposes this.
+Modules can be blacklisted with <tt>modprobe.blacklist=</tt>.
+===my video is borked===
+If you can't get output to the display, [https://www.kernel.org/doc/html/latest/networking/netconsole.html netconsole] is a pretty decent method for quickly getting console output sent to another machine. You won't have ARP or routing, so the target will need be on the local broadcast domain. Specify the target MAC if you can to avoid broadcasting.
 ==initramfs==
@@ Line 28: / Line 48: @@
 ===initramfs can't mount root===
-An unpleasant situation is one where initramfs fails to mount the root partition, in which case you will be dumped to the dreaded BusyBox or klibc shells (ash, as in "a shitty shell"). A day when one sees BusyBox is never a good day. If the machine is remote, you are fucked without server-style out of band access (e.g. Dell iDRAC, BMC, KVM-over-IP). Otherwise, if you have a valid root partition somewhere, you can manually continue the boot by mounting that partition to <tt>/mnt/root</tt> and running <tt>exec switch_root /mnt/root /sbin/init</tt>. Usually this means you've specified the wrong root partition in your bootloader; check the <tt>root</tt> command line option to the kernel.
+An unpleasant situation is one where initramfs fails to mount the root partition, in which case you will be dumped to the dreaded BusyBox or klibc shells (ash, as in "a shitty shell"). A day when one sees BusyBox is never a good day. If the machine is remote, you are fucked without server-style out of band access (e.g. Dell iDRAC, BMC, KVM-over-IP). Otherwise, if you have a valid root partition somewhere, you can manually continue the boot by mounting that partition to <tt>/mnt/root</tt> and running <tt>exec switch_root /mnt/root /sbin/init</tt> or its non-union equivalent. Usually this means you've specified the wrong root partition in your bootloader; check the <tt>root</tt> command line option to the kernel.
+If you have [[systemd]] running in the emergency shell (its <tt>emergency.target</tt>), try manually mounting the root filesystem to <tt>/sysroot</tt> and then running <tt>systemctl restart mount-sysroot</tt>. If this succeeds, exiting the shell ought continue the boot.
 ==fsck on boot==
+this doesn't happen much anymore since most everyone's running at least ext3, but ext2 required <tt>fsck</tt> on boot when not cleanly unmounted, and every so often just to make sure things hadn't shifted around.
+if you don't want a filesystem to block booting under [[systemd]], use <tt>nofail</tt> in <tt>/etc/fstab</tt>.
+==access sans password==
+Ideally, you can just provide <tt>init=/bin/sh</tt> on the kernel command line using grub's interactive mode. This will go through the end of the initramfs and invoke <tt>/bin/sh</tt> within the root filesystem.
+<b>what if we can't use interactive mode?</b>
+===systemd emergency mode with a locked root account===
+If you've locked the root account with <tt>passwd -l</tt>, systemd's <tt>sulogin</tt> won't let you enter rescue mode. Use the instructions from "access sans password" above to get a shell if possible, and enable root with <tt>passwd</tt>. Reboot, and <tt>sulogin</tt> will admit you using the new password.
+==all my binaries are gone==
+The situation can arise that one has an active process context (we'll assume a shell) on a machine where the standard POSIX binaries are unavailable. This can happen due to e.g. a network error, a disk error, memory error, wayward <tt>rm -rf /</tt>, misadventures in package management. What can be done in such a case? Shell builtins, [[sysfs]], <tt>/dev</tt> entries, and [[proc]] usually continue to be available.
+===cat without cat===
+The bash shell's <tt>echo</tt> or <tt>printf</tt> builtin can be used:
+<pre>
+[freebird](0) $ dmesg
+-bash: /usr/bin/dmesg: Input/output error
+[freebird](126) $ printf "%s" "$(</proc/kmsg)"
+.....
+[freebird](0) $ echo "$(</proc/kmsg)"
+.....
+[freebird](0) $
+</pre>
+===reboot without reboot===
+Enable and apply sysrq reboot:
+<pre>echo 128 > /proc/sys/kernel/sysrq
+echo b > /proc/sysrq-trigger</pre>
+Alt+SysRq+B will accomplish this at the terminal if sysrq reboot is enabled. Unlike <tt>reboot</tt>, there will be no syncing and unmounting of disks.
+==moving a root filesystem + bootloader==
+Ideally, mount the source filesystem <tt>ro</tt>, so it's not being modified while you copy. Start assuming a clean destination filesystem (otherwise, wipe it). Using the <tt>-a</tt> flag, <tt>cp</tt> basic toplevel directories *but do not cross filesystem boundaries on the source* (i.e. if <tt>/home</tt> is on its own filesystem, do not include it below; this includes <tt>/boot/efi</tt>). Any that you skip in this fashion ought have their mountpoints created; add them to the <tt>mkdir</tt> below:
+<pre>
+cp -a SOURCE/{bin,boot,etc,home,lib*,root,sbin,usr} DEST/
+mkdir -p DEST/{dev,proc,var/tmp,tmp}
+ln -s /run DEST/var/run
+ln -s /lock DEST/var/lock
+chmod 1777 DEST/{var/tmp,tmp}
+</pre>
+<b>ESP, bootloader, initramfs...</b>
+==reassembling a degraded MDRAID==
+...
 ==External links==
 * Debian wiki page for [https://wiki.debian.org/initramfs initramfs]
 * Kernel documentation for [https://www.kernel.org/doc/html/latest/filesystems/ramfs-rootfs-initramfs.html Ramfs, Rootfs, and Initramfs]