Re: grub rescue read or write sector outside of partition

[Andrei Borzenkov]

В Sat, 27 Jun 2015 15:17:38 -0700
Dale Carstensen  <address@hidden> пишет:

> 
> The bash history is long gone.  My feeble memory is that it
> was simply
> 
>  grub2-install /dev/sdf
> 
> and it responded there were no errors.
> 
> Eventually I booted from a DVD and used chroot to do
> 
>  grub2-install /dev/sdb
> 
> The disk configuration, as shown by /proc/mdstat, is:
> 
> md126 : active raid6 sdf8[5] sdd1[4] sdc1[3] sdb1[2] sda1[1]
>       87836160 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
>       
> md127 : active raid6 sdf10[5] sdd3[4] sdc3[3] sdb3[2] sda3[1]
>       840640512 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] 
> [UUUUU]
>       bitmap: 1/3 pages [4KB], 65536KB chunk
> 
> / is mounted from md126p1, /home from md127p1.
> 

Do you really have partitioned MD RAID? Why? The only reason to have it
is firmware RAID, but here you already build MD from partitions. Having
additional partitions on top of them just complicates things.

> The sdf8 and sdf10 partitions are on the replacement drive.
> The former partitions those replaced are still on sde8 and
> sde10.
> 

Was drive sde still present in system after you had replaced it with
sdf in MD RAID? You said it failed - how exactly? Was it media failure
on some sectors or was it complete failure, which made drive
inaccessible?

> Grub calls sde (hd0), sdf (hd1), md126 (md/1) and md127 (md/3).
> The DVD boot calls sde sda, and sdf sdb.  All neatly made
> consistent by those long UUID strings.  And grub calls
> md126p1 (md/1,gpt1), but for command input seems to like
> (md/1,1) without the label-type distinction.
> 

Yes, you indeed have partitioned RAID. And grub even does work with it.
Wow! :)

...
> 
> Well, it seems to work again.
> 
> The first baby step was to make a partition on (hd1)/sdb/sdf
> starting at block 34 and ending at block 2047.  Partition 8
> begins at block 2048, and originally I set it to type ef02.

If at this state you run grub2-install /dev/sdf *and* it completed
without errors, it means it overwrote beginning of /dev/sdf8. I wonder
what can we do to check that partition is not in use. May be opening it
exclusively would help in this case.

> Then I changed it to fd00 and made the block 34 partition (11)
> type ef02.  I tried to make that partition 11 ext3 and
> put some of /boot in it, but obviously it's way too short
> for anything but grub's persistent environment.  So I used
> dd with if=/dev/zero to clear it.  And I did grub2-install
> with the --recheck option.  All while booted from DVD and
> using chroot, keeping in mind the device was /dev/sdb.
> 
> That avoided "grub rescue", but the only kernel it found was
> the old one, 3.8.11.
> 

Not sure I understand so far how it could fix "grub rescue".

> I stabbed in the dark through another 4 or 5 reboots, until
> eventually pager=1 and cat to look at /boot/grub/grub.cfg
> showed that the only menuentry in it for Linux was for
> 3.8.11, while I knew the latest grub.cfg I had also had
> the new 4.0.5, as well as older 3.8.9 and 3.8.7 ones.
> I'm still not sure where that grub.cfg came from, but I
> made the assumption that it had to do with grub being too
> liberal about failed members of RAID6 partitions.
> 
> So I ran
> 
>  mdadm --zero-superblock /dev/sde8
> 
> and also for 10.
> 
> I think that fixed things.

Yes, *that* is quite possible. Unfortunately GRUB does not currently
checks whether disk superblock generations match each other, and it
stops scanning as soon as enough disks are found, so it could pick up
stale pieces from sde instead of new one from sdf.

So to be on safe side it is necessary to either remove replaced drive
or zero it out, so it is not detected as part of RAID.

Anyway, I'm happy you fixed it and thank you very much for sharing your
experience, it is quite helpful!