Mirroring Linux system disk

DISCLAIMER: The author is not responsible for the loss of data and other problems caused by execution of the described commands.

Currently, the need for mirroring of the system disk is less in demand, since most servers are virtual and are backed up by a fault-tolerant storage. Even physical servers come with a RAID controller, mirroring system disk by default.

But what if this is not true and you have to mirror system disk? Let's first practice a little with VM. I've taken RedHat 7.2 system installed by Next->Next rule on single disk.

Analysis of the existing system

First, we need to understand what we have at the starting point and what we want to see at the end. Then we plan a migration steps.

[root@localhost ~]# df
Filesystem            1K-blocks   Used Available Use% Mounted on
/dev/mapper/rhel-root  18307072 884460  17422612   5% /
devtmpfs                 497924      0    497924   0% /dev
tmpfs                    508384      0    508384   0% /dev/shm
tmpfs                    508384   6760    501624   2% /run
tmpfs                    508384      0    508384   0% /sys/fs/cgroup
/dev/vda1                508588 109896    398692  22% /boot
tmpfs                    101680      0    101680   0% /run/user/0

It looks like our disk has the first partition mounted as /boot and rootfs is in LVM. Let's check this:

[root@localhost ~]# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: QEMU     Model: QEMU DVD-ROM     Rev: 2.5+
  Type:   CD-ROM                           ANSI  SCSI revision: 05

On SCSI, there are no other devices than the CD-ROM. My virtual disk is defined as VIRTIO, so it's not here.

[root@localhost ~]# cat /proc/partitions
major minor  #blocks  name

  11        0    1048575 sr0
 252        0   20971520 vda
 252        1     512000 vda1
 252        2   20458496 vda2
 253        0   18317312 dm-0
 253        1    2097152 dm-1

It is another good place to check what happens. sr0 is a SCSI CDROM, vda is my virtual disk, vda1 and vda2 are its partitions, dm-* are devices from the "device mapper" and can be translated to a readable form, as it is written here .

[root@localhost ~]# fdisk -l /dev/vda

Disk /dev/vda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0007d9a3

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    41943039    20458496   8e  Linux LVM
[root@localhost ~]# 

It seems that we were lucky. There are no partitions serving the data, everything is serviced by LVM, except /boot.

What will we implement? I'll add one more disk with the same layout, make the first partition mirrored using MD (mdadm), the second partition will be added to LVM, then all LVs will be mirrored by LVM. A boot record should be written on the second disk also, the BIOS will be configured to boot from the second disk as the second choice. Peanuts.

Backup the system

TAKE THE BACKUP !!!

All operations here are very dangerous. Even if they look safe, a small typo can turn your server into a bunch of hardware.

DISCLAIMER: The author is not responsible for the loss of data and other problems caused by execution of the described commands.

Mirroring system

Add a disk. In a real situation, you can not know whether the disk is empty or contains data. If you have the opportunity to connect the drive somewhere else, then clean it there. It is usually enough to clear the first 512b to erase the partition table. If you need to erase the GPT, then probably your first disk is also large and has a GPT on it. In this case, you must follow another article, not this one.

In a virtual world, a reboot is not required. Some hardware also allows you to use hot-swap drives. In the worst case, you must turn off the system, insert a new disk and turn on the system. If the new disk still have content, it can be loaded instead of the primary one, so scratch the disk before inserting.

[root@localhost ~]# tail /var/log/messages
Mar 12 06:25:06 localhost kernel: pci 0000:00:09.0: BAR 4: assigned [mem 0x40000000-0x407fffff 64bit pref]
Mar 12 06:25:06 localhost kernel: pci 0000:00:09.0: BAR 1: assigned [mem 0x40800000-0x40800fff]
Mar 12 06:25:06 localhost kernel: pci 0000:00:09.0: BAR 0: assigned [io  0x1000-0x103f]
Mar 12 06:25:06 localhost kernel: virtio-pci 0000:00:09.0: enabling device (0000 -> 0003)
Mar 12 06:25:06 localhost kernel: vdb: unknown partition table

We decided that the partitioning scheme is suitable for us, so we will simply copy the suitable layout from the old disk to the new one (suppose we had add a disk of the same size for mirroring):

[root@localhost ~]# sfdisk -d /dev/vda | sfdisk /dev/vdb
Checking that no-one is using this disk right now ...
OK

Disk /dev/vdb: 41610 cylinders, 16 heads, 63 sectors/track
sfdisk:  /dev/vdb: unrecognized partition table type

Old situation:
sfdisk: No partitions found

New situation:
Units: sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/vdb1   *      2048   1026047    1024000  83  Linux
/dev/vdb2       1026048  41943039   40916992  8e  Linux LVM
/dev/vdb3             0         -          0   0  Empty
/dev/vdb4             0         -          0   0  Empty
Warning: partition 1 does not start at a cylinder boundary
Warning: partition 1 does not end at a cylinder boundary
Warning: partition 2 does not start at a cylinder boundary
Warning: partition 2 does not end at a cylinder boundary
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
[root@localhost ~]#

The above command is very dangerous. THIS command dump partition table from vda and write it on vdb. YOUR command will be differ. Check seven times before pressing Enter. Overwriting working partition table may turn your server into brick. Did you backup ?

Now, change partition type to 'Linux raid autodetect':

[root@localhost ~]# fdisk /dev/vdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p

Disk /dev/vdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1   *        2048     1026047      512000   83  Linux
/dev/vdb2         1026048    41943039    20458496   8e  Linux LVM

Command (m for help): t
Partition number (1,2, default 2): 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@localhost ~]#

Changing partition type is not necessary, but this will help the kernel detect mirror chains.

Check if the kernel knows about the new partitions:

[root@localhost ~]# cat /proc/partitions 
major minor  #blocks  name

  11        0    1048575 sr0
 252        0   20971520 vda
 252        1     512000 vda1
 252        2   20458496 vda2
 252       16   20971520 vdb
 252       17     512000 vdb1
 252       18   20458496 vdb2
 253        0   18317312 dm-0
 253        1    2097152 dm-1
[root@localhost ~]# 

If you do not see new vdb2 and vdb2 here, run kpartx -a /dev/vdb to force the kernel to re-read the partition table of the new disk. If the partitions were earlier (in the case of a non-empty disk), you may need to run kpartx -d /dev/vdb before re-reading.

Mirroring /boot

The mdadm package is not installed with minimal installation, install it if necessary.

[root@localhost ~]# mdadm -C /dev/md0 -n2 -l1 /dev/vdb1 missing
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

The broken mirror starts with one missing device. The warning is for incompatible tools. Our grub2 knows about MD/v1 metadata, so agree with "yes."

[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 vdb1[0]
      511680 blocks super 1.2 [2/1] [U_]
      
unused devices: 

An underscore in [U_] is our "missing" device. Create /etc/mdadm.conf config file. This is not strictly necessary, but it will help fix the MD names and support RAID. Examine "Array UUID" and make correct config file:

[root@localhost ~]# mdadm --examine /dev/vdb1 | grep "Array UUID"
     Array UUID : 1303fbf7:ede1d764:ef401616:2b234c54
[root@localhost ~]# cat /etc/mdadm.conf
MAILADDR root
ARRAY /dev/md0 uuid=1303fbf7:ede1d764:ef401616:2b234c54
[root@localhost ~]#

Check that everything works as should:

[root@localhost ~]# mdadm -S /dev/md0
mdadm: stopped /dev/md0
[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid1] 
unused devices: 
[root@localhost ~]# mdadm -A /dev/md0
mdadm: /dev/md0 has been started with 1 drive (out of 2).
[root@localhost ~]# mdadm -R /dev/md0
[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 vdb1[0]
      511680 blocks super 1.2 [2/1] [U_]
      
unused devices: 

Copy /boot content to new location:

[root@localhost ~]# mount | grep boot
/dev/vda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
[root@localhost ~]# grep boot /etc/fstab
UUID=98355cc5-4eca-4576-9b97-153755cbd2a3 /boot                   xfs     defaults        0 0
[root@localhost ~]# 

Surprise, /boot formatted as xfs, let's do the same:

[root@localhost ~]# mkfs.xfs /dev/md0
meta-data=/dev/md0               isize=256    agcount=4, agsize=31980 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=127920, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=853, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@localhost ~]# mkdir -p /mnt/d
[root@localhost ~]# mount /dev/md0 /mnt/d
[root@localhost ~]# (cd /boot && tar cf - .)|(cd /mnt/d && tar xpfB -)
[root@localhost ~]# umount /boot /mnt/d
[root@localhost ~]# mount /dev/md0 /boot

Fix /etc/fstab /boot entry to mount /dev/md0:

[root@localhost ~]# grep boot /etc/fstab
/dev/md0                /boot                   xfs     defaults        0 0
#UUID=98355cc5-4eca-4576-9b97-153755cbd2a3 /boot                   xfs     defaults        0 0
[root@localhost ~]# 

Note that the original /boot remains unchanged, because it was unmounted. All changes will be performed in a new location, so we have a chance to boot our system using the old working procedure.

Rebuild initrd to include MD modules and configuration files. They will be included automatically, you just need to rebuild initrd:

# mkinitrd -f /boot/initramfs-$(uname -r).img $(uname -r)

Rebuild grub.cfg and write bootsector:

[root@localhost ~]# grub2-mkconfig > /boot/grub2/grub.cfg    
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-327.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.el7.x86_64.img
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Found linux image: /boot/vmlinuz-0-rescue-b5480fa09a54b645ab8ad5a0eccb24d7
Found initrd image: /boot/initramfs-0-rescue-b5480fa09a54b645ab8ad5a0eccb24d7.img
done
[root@localhost ~]# grub2-install /dev/vdb
Installing for i386-pc platform.
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
[root@localhost ~]# 

Time to test. Shut down the server. Enter the BIOS settings and remove the vda from the boot list, leave only the vdb drive. Boot the server. Check the output of the dmesg command. Try to make sure that /dev/vda1 was not used at boot time, since we will destroy it in the next step.

So far, so good. Let's finish /boot mirroring. First, we will fix partition type of /dev/vda1 to be "fd":

[root@localhost ~]# fdisk /dev/vda
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p

Disk /dev/vda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    41943039    20458496   8e  Linux LVM

Command (m for help): t
Partition number (1,2, default 2): 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@localhost ~]#

A dangerous command again, the old boot partition will be lost for the sake of building a healthy MD device:

[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 vdb1[0]
      511680 blocks super 1.2 [2/1] [U_]
      
unused devices: 
[root@localhost ~]# mdadm -a /dev/md0 /dev/vda1
mdadm: added /dev/vda1
[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 vda1[2] vdb1[0]
      511680 blocks super 1.2 [2/1] [U_]
      [===========>.........]  recovery = 57.8% (295936/511680) finish=0.0min speed=49322K/sec
      
unused devices: 
[root@localhost ~]# 

Wait until the synchronization is complete and recreate the boot sector on the device /dev/vda.

[root@localhost ~]# grub2-install /dev/vda
Installing for i386-pc platform.
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
[root@localhost ~]# 

Reboot the server, add the first disk to the boot sequence as the first device, check the boot procedure.

LVM mirroring of the rest:

The difficult part is over. Add a second partition to an existing VG:

[root@localhost ~]# vgs
  VG   #PV #LV #SN Attr   VSize  VFree 
  rhel   1   2   0 wz--n- 19.51g 40.00m
[root@localhost ~]# pvs
  PV         VG   Fmt  Attr PSize  PFree 
  /dev/vda2  rhel lvm2 a--  19.51g 40.00m
[root@localhost ~]# pvcreate /dev/vdb2
  Physical volume "/dev/vdb2" successfully created
[root@localhost ~]# vgextend rhel /dev/vdb2
  Volume group "rhel" successfully extended
[root@localhost ~]# pvs
  PV         VG   Fmt  Attr PSize  PFree 
  /dev/vda2  rhel lvm2 a--  19.51g 40.00m
  /dev/vdb2  rhel lvm2 a--  19.51g 19.51g
[root@localhost ~]# 

We must have free space on both PVs to store the metadata of the mirror and the log. If you do not have free space, try to reduce something. In my example, reducing swap LV is the simplest way: swapoff, reduce, re-create swap, swapon. But 40.00m of free space is quite enough to store mirrors metadata. So, let's start:

[root@localhost ~]# lvs
  LV   VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root rhel -wi-ao---- 17.47g                                                    
  swap rhel -wi-ao----  2.00g                                                    
[root@localhost ~]# lvconvert -b -m1 /dev/rhel/root 
[root@localhost ~]# lvconvert -b -m1 /dev/rhel/swap 
[root@localhost ~]# lvs
  LV   VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root rhel rwi-aor--- 17.47g                                    2.24            
  swap rhel rwi-aor---  2.00g                                    0.00            
[root@localhost ~]#

Wait until Cpy%Sync is 100% before the destructive tests. You can reboot the server, the LVM will continue to synchronize on its own.

Rebuilt the new initrd to reflect changes in LVM structure:

[root@localhost ~]# mkinitrd -f /boot/initramfs-$(uname -r).img $(uname -r)
/usr/sbin/dracut: line 639: warning: setlocale: LC_MESSAGES: cannot change locale (C.utf8): No such file or directory
/usr/sbin/dracut: line 640: warning: setlocale: LC_CTYPE: cannot change locale (C.utf8): No such file or directory
[root@localhost ~]# 

The rebuild is required, do not skip it, otherwise the server will not boot even with two available disks.

Restart the server to verify the boot procedure is working.

Testing resilience and recovery from drive faults.

Let's remove second drive and check /var/log/messages what happen:

Mar 12 08:48:48 localhost kernel: md: md0 still in use.
Mar 12 08:48:48 localhost kernel: md/raid1:md0: Disk failure on vdb1, disabling device.#012md/raid1:md0: Operation continuing on 1 devices.
Mar 12 08:48:57 localhost kernel: md: super_written gets error=-5, uptodate=0
Mar 12 08:48:57 localhost kernel: md/raid1:mdX: Disk failure on dm-5, disabling device.#012md/raid1:mdX: Operation continuing on 1 devices.
Mar 12 08:48:57 localhost lvm[1476]: Device #1 of raid1 array, rhel-root, has failed.
Mar 12 08:48:57 localhost lvm[1476]: WARNING: Device for PV XOFMua-NHIr-1FrR-fpeu-i47H-kQN1-KJ3UOx not found or rejected by a filter.
Mar 12 08:48:57 localhost lvm[1476]: WARNING: Device for PV XOFMua-NHIr-1FrR-fpeu-i47H-kQN1-KJ3UOx already missing, skipping.
Mar 12 08:48:57 localhost lvm[1476]: WARNING: Device for PV XOFMua-NHIr-1FrR-fpeu-i47H-kQN1-KJ3UOx not found or rejected by a filter.
Mar 12 08:48:57 localhost lvm[1476]: Use 'lvconvert --repair rhel/root' to replace failed device.

As you see, everything still functional. But what about booting ? Reboot the server.

Amazing !

Return the second device back as is. LVM immediately detected a paired drive and resynchronized LVs seamless:

[root@localhost ~]# lvs
  LV   VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root rhel rwi-aor--- 8.47g                                    100.00          
  swap rhel rwi-aor--- 1.00g                                    100.00          
[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [raid1] 
md0 : active raid1 vda1[2]
      511680 blocks super 1.2 [2/1] [_U]
      
unused devices: 
[root@localhost ~]# 

However, the MD (/boot) device was not automatically assembled. Let's help him:

[root@localhost ~]# mdadm -a /dev/md0 /dev/vdb1
mdadm: added /dev/vdb1
[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [raid1] 
md0 : active raid1 vdb1[3] vda1[2]
      511680 blocks super 1.2 [2/1] [_U]
      [=====>...............]  recovery = 25.0% (128000/511680) finish=0.1min speed=32496K/sec
      
unused devices: 
[root@localhost ~]# 

Let's repeat the same test, now with the first disk. Oops. My server is crushed. Probably because of very frequent equipment failure. However, the boot was successful, and the server returned with one second disk. The re-reassignment of the disk went through the same scenario, and the mirror was rebuilt well.

In the next test, we will replace the failed disk with an empty one. The procedure for restructuring will be slightly different. I shut off the server and replaced one disk with blank.

First of all, copy the partition table to a new disk. Double check which drive is the source and which is the destination.

[root@localhost ~]# fdisk -l /dev/vda

Disk /dev/vda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000f2f15

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   fd  Linux raid autodetect
/dev/vda2         1026048    20971519     9972736   8e  Linux LVM
[root@localhost ~]# fdisk -l /dev/vdb

Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@localhost ~]# 

In my case vda will be a source, then command will be:

# sfdisk -d /dev/vda | sfdisk /dev/vdb

Now rebuild the MD device:

[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [raid1] 
md0 : active raid1 vda1[2]
      511680 blocks super 1.2 [2/1] [_U]
      
unused devices: 
[root@localhost ~]# mdadm -a /dev/md0 /dev/vdb1
mdadm: added /dev/vdb1
[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [raid1] 
md0 : active raid1 vdb1[3] vda1[2]
      511680 blocks super 1.2 [2/1] [_U]
      [====>................]  recovery = 23.7% (121792/511680) finish=0.1min speed=60896K/sec
      
unused devices: 
[root@localhost ~]#

Wait until the synchronization is complete, then install the boot sector. Again, in my case I should fix vdb:

[root@localhost ~]# grub2-install /dev/vdb
Installing for i386-pc platform.
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
[root@localhost ~]#

Now let's fix LVM faulty PV. First, delete it:

[root@localhost ~]# pvs
  WARNING: Device for PV uTPvR8-zeuP-tdJT-AW3r-DCuj-hazO-QxqYdB not found or rejected by a filter.
  PV             VG   Fmt  Attr PSize PFree 
  /dev/vda2      rhel lvm2 a--  9.51g 32.00m
  unknown device rhel lvm2 a-m  9.51g 32.00m
[root@localhost ~]# vgreduce rhel --removemissing --force
  WARNING: Device for PV uTPvR8-zeuP-tdJT-AW3r-DCuj-hazO-QxqYdB not found or rejected by a filter.
  WARNING: Device for PV uTPvR8-zeuP-tdJT-AW3r-DCuj-hazO-QxqYdB not found or rejected by a filter.
  Wrote out consistent volume group rhel
[root@localhost ~]# pvcreate /dev/vdb2
  Physical volume "/dev/vdb2" successfully created
[root@localhost ~]# pvs
  PV         VG   Fmt  Attr PSize PFree 
  /dev/vda2  rhel lvm2 a--  9.51g 32.00m
  /dev/vdb2       lvm2 ---  9.51g  9.51g
[root@localhost ~]# vgextend rhel /dev/vdb2
  Volume group "rhel" successfully extended
[root@localhost ~]# pvs
  PV         VG   Fmt  Attr PSize PFree 
  /dev/vda2  rhel lvm2 a--  9.51g 32.00m
  /dev/vdb2  rhel lvm2 a--  9.51g  9.51g
[root@localhost ~]# lvs
  LV   VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root rhel rwi-aor--- 8.47g                                    100.00          
  swap rhel rwi-aor--- 1.00g                                    100.00          

After correcting PV, let's fix LV. First, we convert it back to a normal linear LV, and then convert it back to be a mirror:

[root@localhost ~]# lvs       
  LV   VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root rhel rwi-aor--- 8.47g                                    100.00          
  swap rhel rwi-aor--- 1.00g                                    100.00          
[root@localhost ~]# lvconvert -m0 /dev/rhel/swap
[root@localhost ~]# lvconvert -m0 /dev/rhel/root
[root@localhost ~]# lvs
  LV   VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root rhel -wi-ao---- 8.47g                                                    
  swap rhel -wi-ao---- 1.00g                                                    
[root@localhost ~]# lvconvert -b -m1 /dev/rhel/root 
[root@localhost ~]# lvconvert -b -m1 /dev/rhel/swap
[root@localhost ~]# lvs
  LV   VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root rhel rwi-aor--- 8.47g                                    1.01            
  swap rhel rwi-aor--- 1.00g                                    6.25            
[root@localhost ~]#

Wait for 100% synced.

Rescue boot

If something went wrong, for example, if you skipped the mkinitrd step or the bootrecord creation step, the server will not boot. In my case, I've re-created the VM again from the template, which is much faster. But if your server is valuable, you can try boot from the rescue CD. In the worst case, you have performed a server's backup and will do bare metal restore.

After boot from the rescue CD, it will try to find your installation. In most cases, it will find it and mount it under /mnt/sysimage. Then run "chroot" and do the skipped steps. This section of the article is quite theoretical without any commands for copy-paste, because the case has not happen.


Updated on Sun Mar 12 21:44:47 IST 2017 by Oleg Volkov More documentations here