Often when replacing a failed disk in a RAID1 setup (or in many other cases) grub fails to load. Usually it's because the bios is expecting to find a boot loader and the various files on which the boot loader depends on the first disk in the machine. If that's the disk that was replaced, then loading grub will fail.

One way of recovering is to boot the server from a USB stick that has grub installed and then manually tell grub how to boot from the disks that are available.

If you are reading this before you replace a disk - be sure to copy the /boot/grub/menu.lst file from the computer before turning it off. That will save you a lot of pain.

There are a lot of complicated tutorials on how to do this - however, if you don't care about the contents of your USB stick, it's really quite simple.

• Put your USB disk into working linux computer (don't mount it). On my computer it is typically recognized as /dev/sda. If your USB stick is recognized differently (and/or your hard disk is /dev/sda) then you must replace all the instances below of /dev/sda with whatever your actual USB stick is being recognized. This is, well, kinda important.

• Create a single partition. I typically do this with:

  sudo cfdisk /dev/sda
  

  Feel free to create a FAT32 partition if you want to use this stick on different machines with different operating systems (including Windows).

• Mount the partition you just created:

  sudo mount /dev/sda1 /mnt
  

• Install GRUB:

  sudo grub-install --no-floppy --root-directory=/mnt /dev/sda
  

  That should create a /mnt/boot/grub directory.

  It will also create /mnt/boot/grub/device.map, which is a list of disks on the computer it is running. This list will include not only the USB stick but your computer's hard disk as well. For example:

  (hd0)   /dev/hda
  (hd1)   /dev/sda
  

  Change to:

  (hd0)   /dev/sda
  

  So that the only device showing is your USB stick. Then re-run:

  sudo grub-install --root-directory=/mnt /dev/sda
  

• Now comes the hard part. You need a grub configuration that will make sense for the server you are booting.

  If you are using grub1...

  Hopefully you have a copy of the menu.lst file from the server. In that case, simply copy it to /mnt/boot/grub/ and you are ready to go. Otherwise, you'll need to craft one. Below is a sample.

  # uncomment these lines if you want to send grub to a serial console
  #serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
  #terminal serial
  default     0
  timeout     5
  color cyan/blue white/blue
  
  
  # simple setup
  title           Debian GNU/Linux, kernel 2.6.26-2-686
  root            (hd0,5)
  kernel          /boot/vmlinuz-2.6.26-2-686 root=/dev/hda6 ro
  initrd          /boot/initrd.img-2.6.26-2-686
  
  
  # here's a more complicated one
  title       Debian GNU/Linux, kernel 2.6.26-2-vserver-amd64
  root        (hd0,0)
  kernel      /vmlinuz-2.6.26-2-vserver-amd64 root=/dev/mapper/vg_pianeta0-root \
  ro console=ttyS0,115200n8 cryptopts=target=md1_crypt,source=/dev/md1 \
  cryptopts=target=md2_crypt,source=/dev/md2,lvm=vg_pianeta0-root 
  initrd      /initrd.img-2.6.26-2-vserver-amd64
  

  If you are using grub2...

  grub-mkconfig -o /mnt/boot/grub/grub.cfg
  

  Then edit. You might want something like:

  set default=0
  set timeout=5
  
  
  menuentry "Debian GNU/Linux, with Linux 2.6.32-trunk-vserver-amd64" --class debian --class gnu-linux --class gnu --class os {
   set root='(hd0,1)'
   search --no-floppy --fs-uuid --set 7682a24c-b06f-456b-b3d4-bcb7294d81e2
   echo   Loading Linux 2.6.32-trunk-vserver-amd64 ...
   linux  /vmlinuz-2.6.32-trunk-vserver-amd64 root=/dev/mapper/vg_chicken0-root ro  quiet
   echo   Loading initial ramdisk ...
  initrd  /initrd.img-2.6.32-trunk-vserver-amd64
  }
  

• Put the USB stick into the target computer, configure it to boot from the USB stick via bios, and then you should see the GRUB menu come up.

• It's possible that your computer will just boot with your menu.lst file. In that case - congrats! See the last step below to figure out how to ensure it can boot without your USB stick. On the other hand, if it fails, you'll need to experimentally figure out which disk has which partitions and which kernels. Fortunately grub supports tab completion which makes this job easier:

  • When the grub menu comes up, pick from the menu list the most likely candidate and press 'e' for edit.

  • You should see the various lines from the stanzas for the list item you picked (i.e. a root, kernel, and initrd stanza). You may use the up/down arrows to select a line. If that doesn't work, look for hints on the screen for how to get around. Going left and right on a given line may require Ctl-b and Ctl-f for back and forward. You also may need to use the delete key (not backspace) to delete characters

  • Select a line, delete the characters from the end of the line, and then try tab completion with various options. For example, on the root line try typing simply:

    root (

  And then tab. You should be presented with the available disks (numbered 0 and up). Try typing one of the disks and hitting tab and you should be presented with the available partitions. Continue this process until you find the one that seems right.

• When you are done and you have successfully booted, you can ensure that boots will work without the usb key by installing grub on all available disks:

  grub-install /dev/sda
  grub-install /dev/sdb