This manual and command syntax is current for SUSE 15 / Red Hat 8, previous command syntax can be found here.
All volumes used here are small, so all actions are performed relatively quickly. If you have large volumes, use the -b (background) option to return to the shell immediately. It's safe to press Ctrl-C if you forgot to use the -b option. The command will continue to run in the background. The progress of a command can be checked with the lvs command.
At our starting point, we have a logical volume /dev/testvg/data that we want to migrate over the new four disks by converting it to a striped type. It is desirable that all operations are performed online, without interrupting normal work.
# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rootvg lvm2 a-- 31.99g 20.99g /dev/sdb testvg lvm2 a-- 30.00g 0 # df /mnt Filesystem Size Used Avail Use% Mounted on /dev/mapper/testvg-data 30G 1.9G 28G 7% /mnt
Connect the new four devices and, depending on your hardware, rescan the FC, SCSI, and multipath devices. Here I have a small KVM environment that adds devices automatically when they are plugged in, so there is no need to scan. Just in case, here are the commands for scanning FC and SCSI:
# # Rescan Fiber Channel # for FC in /sys/class/fc_host/host?/issue_lip ; do echo "1" > $FC ; sleep 5 ; done # # Rescan SCSI devices # for SH in /sys/class/scsi_host/host?/scan ; do echo "- - -" > $SH ; done
Create a PV on each added disk (or multipath device).
# pvcreate /dev/sd{c,d,e,f}
Physical volume "/dev/sdc" successfully created.
Physical volume "/dev/sdd" successfully created.
Physical volume "/dev/sde" successfully created.
Physical volume "/dev/sdf" successfully created.
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 rootvg lvm2 a-- 31.99g 20.99g
/dev/sdb testvg lvm2 a-- 30.00g 0
/dev/sdc lvm2 --- 10.00g 10.00g
/dev/sdd lvm2 --- 10.00g 10.00g
/dev/sde lvm2 --- 10.00g 10.00g
/dev/sdf lvm2 --- 10.00g 10.00g
Extend our testvg with these four PVs:
# vgextend testvg /dev/sd{c,d,e,f}
Volume group "testvg" successfully extended
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 rootvg lvm2 a-- 31.99g 20.99g
/dev/sdb testvg lvm2 a-- 30.00g 0
/dev/sdc testvg lvm2 a-- 10.00g 10.00g
/dev/sdd testvg lvm2 a-- 10.00g 10.00g
/dev/sde testvg lvm2 a-- 10.00g 10.00g
/dev/sdf testvg lvm2 a-- 10.00g 10.00g
Add a striped mirror chain to an existing LV:
# lvconvert --type mirror /dev/testvg/data --mirrors 1 --stripes 4 --stripesize 4m /dev/sd{c,d,e,f}
Number of extents requested (7679) needs to be divisible by 4.
Unable to allocate extents for mirror(s).
We want to have a striped volume of four stripes, but the original LV contains a number of PEs that is not a multiple of four. In my case, I need to increase it by one extent and then repeat the previous command:
# lvresize -l+1 /dev/testvg/data
Size of logical volume testvg/data changed from 30.00 GiB (7679 extents) to 30.00 GiB (7680 extents).
Logical volume testvg/data successfully resized.
# lvconvert --type mirror /dev/testvg/data --mirrors 1 --stripes 4 --stripesize 4m /dev/sd{c,d,e,f}
Logical volume testvg/data being converted.
testvg/data: Converted: 9.96%
testvg/data: Converted: 100.00%
# lvs -a -o +devices testvg
LV VG Attr LSize ^ Log Cpy%Sync ^ Devices
data testvg mwi-aom--- 30.00g ^ [data_mlog] 100.00 ^ data_mimage_0(0),data_mimage_1(0)
[data_mimage_0] testvg iwi-aom--- 30.00g ^ ^ /dev/sdb(0)
[data_mimage_0] testvg iwi-aom--- 30.00g ^ ^ /dev/sdc(0)
[data_mimage_1] testvg iwi-aom--- 30.00g ^ ^ /dev/sdc(1),/dev/sdd(0),/dev/sde(0),/dev/sdf(0)
[data_mlog] testvg lwi-aom--- 4.00m ^ ^ /dev/sdc(1921)
This output shows that the logical volume named data consists of two mirrored chains. One chain (marked red) is the original volume, which is on sdb, and one additional extent is on sdc. The second chain is striped over all four target devices.
The next step is to remove the original chain from the mirror. You can only do this after a full sync, as shown in the Cpy%Sync field. The location of our original volume has become very complicated due to the increase in its size, so you have to be very precise when you mention the PV to delete. Let's analyze the PV structure before removing the mirror chain. I marked the original chain in red in the output above. Let's find the PE allocation on /dev/sdb and /dev/sdc for data_mimage_0.
# pvdisplay /dev/sdb -m
--- Physical volume ---
PV Name /dev/sdb
VG Name testvg
PV Size 30.00 GiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 7679
Free PE 0
Allocated PE 7679
PV UUID 9R5LEr-Alq1-vY4c-2w3a-HfOO-RX6D-yufAUS
--- Physical Segments ---
Physical extent 0 to 7678:
Logical volume /dev/testvg/data_mimage_0
Logical extents 0 to 7678
# pvdisplay /dev/sdc -m
--- Physical volume ---
PV Name /dev/sdc
VG Name testvg
PV Size 10.00 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 2559
Free PE 637
Allocated PE 1922
PV UUID 0FGf4H-WuXU-rhIq-YIiG-vLn7-ptgc-iriGuk
--- Physical Segments ---
Physical extent 0 to 0:
Logical volume /dev/testvg/data_mimage_0
Logical extents 7679 to 7679
Physical extent 1 to 1920:
Logical volume /dev/testvg/data_mimage_1
Logical extents 0 to 7679
Physical extent 1921 to 1921:
Logical volume /dev/testvg/data_mlog
Logical extents 0 to 0
Physical extent 1922 to 2558:
FREE
That's it, the correct exact location for the original LV is /dev/sdb:0-7678 /dev/sdc:0-0. Now we can drop the correct chain of mirrors:
# lvconvert -m0 /dev/testvg/data /dev/sdb:0-7678 /dev/sdc:0-0 Logical volume testvg/data converted. # lvs -a -o +devices testvg LV VG Attr LSize Po ^ rt Devices data testvg -wi-a----- 30.00g ^ /dev/sdc(1),/dev/sdd(0),/dev/sde(0),/dev/sdf(0)
The last step in the migration procedure is to clean up the freed resources. I make this chapter separate to make easy reference. The cleanup procedure begins with the removal of the PV from the VG.
# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rootvg lvm2 a-- 31.99g 20.99g /dev/sdb testvg lvm2 a-- 30.00g 30.00g /dev/sdc testvg lvm2 a-- 10.00g 2.50g /dev/sdd testvg lvm2 a-- 10.00g 2.50g /dev/sde testvg lvm2 a-- 10.00g 2.50g /dev/sdf testvg lvm2 a-- 10.00g 2.50g # vgreduce testvg /dev/sdb Removed "/dev/sdb" from volume group "testvg" # pvremove /dev/sdb Labels on physical volume "/dev/sdb" successfully wiped.
The next step is usually to remove the multipath devices and then the corresponding SCSI devices. Print the contents of the multipath device, then delete it. In my example, there are no multipath devices, but usually the removal looks like this:
# multipath -ll mpathb .. # multipath -f mpathb
The multipath name must not include the /dev/mapper part in these commands. The next step is to delete the SCSI devices. Disconnecting even unused SCSI devices can cause the kernel to panic, so you need to delete them before physically disconnecting them.
# echo 1 > /sys/block/sdb/device/delete # grep sdb /proc/partitions #
Now you can physically disconnect old FC LUN(s).
This task is not typical, but can be a great exercise. The following output shows that my LV completely occupies the first 10G PV, and I have already added three more 10G PVs to convert this LV to a striped type in place.
# lvdisplay /dev/testvg/data -m
--- Logical volume ---
LV Path /dev/testvg/data
LV Name data
VG Name testvg
LV UUID rdnDtM-4TlU-YWz1-dHs9-HC9Q-pGxO-Ue8QSd
LV Write Access read/write
LV Creation host, time localhost, 2022-02-08 10:54:32 +0200
LV Status available
# open 0
LV Size 10.00 GiB
Current LE 2559
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 254:4
--- Segments ---
Logical extents 0 to 2558:
Type linear
Physical volume /dev/sdc
Physical extents 0 to 2558
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 rootvg lvm2 a-- 31.99g 20.99g
/dev/sdc testvg lvm2 a-- 10.00g 0
/dev/sdd testvg lvm2 a-- 10.00g 10.00g
/dev/sde testvg lvm2 a-- 10.00g 10.00g
/dev/sdf testvg lvm2 a-- 10.00g 10.00g
Unfortunately for us, an LV consists of 2559 PE, which is not a multiple of 4. Also, /dev/sdc doesn't have room for a fourth stripe. So we have to increase LV by one PE and move some blocks to other disks to make room for striping. So I add one extent to make the LV size fit four stripes requirements, putting this extra extent at the end of the next disk.
# pvdisplay /dev/sdd -m
--- Physical volume ---
PV Name /dev/sdd
VG Name testvg
PV Size 10.00 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 2559
Free PE 2559
Allocated PE 0
PV UUID cY5ygo-z8Zt-Q4qg-Lzo9-76g0-H9A7-mjaW2B
--- Physical Segments ---
Physical extent 0 to 2558:
FREE
The last extent is 2558, we will use it:
# lvextend -l+1 /dev/testvg/data /dev/sdd:2558-2558 Size of logical volume testvg/data changed from 10.00 GiB (2559 extents) to 10.00 GiB (2560 extents). Logical volume testvg/data successfully resized.
Now let's calculate how many PEs we need to free from /dev/sdc before creating the mirror. Simple math shows that this should be 640 extents. Needless to say, that PE counts from 0 and commands cares of extents inclusively. Let's move 640 extents to the end of another free disk:
# pvmove /dev/sdc:0-639 /dev/sde:1919-2558 /dev/sdc: Moved: 1.88% /dev/sdc: Moved: 100.00%
Now we have room for the fourth stripe and we can start assembling the mirror. Usually mirroring takes care of placing the mirrored extents on different devices, so in our case the creating will refuse to run unless the --alloc anywhere option is used. The mirror engine has its own log of one extent size, which we did not count at all. Then the --mirrorlog core option comes to the help. This log is stored in memory, not on disk, and is considered not reliable. This should be enough for our temporary transaction.
# lvconvert --type mirror /dev/testvg/data --mirrors 1 --alloc anywhere --mirrorlog core --stripes 4 --stripesize 4m /dev/sd{c,d,e,f}
Logical volume testvg/data being converted.
testvg/data: Converted: 1.02%
testvg/data: Converted: 100.00%
# lvs -a -o +devices testvg
LV VG Attr LSize ^ Cpy%Sync ^ Devices
data testvg mwi-a-m--- 10.00g ^ 100.00 ^ data_mimage_0(0),data_mimage_1(0)
[data_mimage_0] testvg iwi-aom--- 10.00g ^ ^ /dev/sde(1919)
[data_mimage_0] testvg iwi-aom--- 10.00g ^ ^ /dev/sdc(640)
[data_mimage_0] testvg iwi-aom--- 10.00g ^ ^ /dev/sdd(2558)
[data_mimage_1] testvg iwi-aom--- 10.00g ^ ^ /dev/sdc(0),/dev/sdd(0),/dev/sde(0),/dev/sdf(0)
You must calculate the exact location of the PEs, as shown in the previous example, and delete the original volume:
# lvconvert -m0 /dev/testvg/data /dev/sde:1919-2558 /dev/sdc:640-2558 /dev/sdd:2558-2558 Logical volume testvg/data converted. # lvs -a -o +devices testvg LV VG Attr LSize P.. ^ ..t Devices data testvg -wi-a----- 10.00g ^ /dev/sdc(0),/dev/sdd(0),/dev/sde(0),/dev/sdf(0)
Once you've played enough with striped volumes and want to go back to simplicity, you can reverse the conversion in much the same way.
Now we have a striped LV with four stripes:
# lvdisplay /dev/testvg/data -m
--- Logical volume ---
LV Path /dev/testvg/data
LV Name data
VG Name testvg
LV UUID rdnDtM-4TlU-YWz1-dHs9-HC9Q-pGxO-Ue8QSd
LV Write Access read/write
LV Creation host, time localhost, 2022-02-08 10:54:32 +0200
LV Status available
# open 0
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 65536
Block device 254:4
--- Segments ---
Logical extents 0 to 2559:
Type striped
Stripes 4
Stripe size 4.00 MiB
Stripe 0:
Physical volume /dev/sdc
Physical extents 0 to 639
Stripe 1:
Physical volume /dev/sdd
Physical extents 0 to 639
Stripe 2:
Physical volume /dev/sde
Physical extents 0 to 639
Stripe 3:
Physical volume /dev/sdf
Physical extents 0 to 639
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 rootvg lvm2 a-- 31.99g 20.99g
/dev/sdb lvm2 --- 30.00g 30.00g
/dev/sdc testvg lvm2 a-- 10.00g 7.50g
/dev/sdd testvg lvm2 a-- 10.00g 7.50g
/dev/sde testvg lvm2 a-- 10.00g 7.50g
/dev/sdf testvg lvm2 a-- 10.00g 7.50g
The challenge is to convert it online to a regular logical volume located on another separate drive. Add target disk (PV) to VG:
# vgextend testvg /dev/sdb
Add a new PV as a mirror chain to existing striped LV:
# lvconvert --type mirror /dev/testvg/data --mirrors 1 /dev/sdb Logical volume testvg/data being converted. testvg/data: Converted: 0.62% testvg/data: Converted: 100.00% # lvs -a -o +devices testvg LV VG Attr LSize ^ Log Cpy%Sync ^ Devices data testvg mwi-a-m--- 10.00g ^ [data_mlog] 100.00 ^ data_mimage_0(0),data_mimage_1(0) [data_mimage_0] testvg iwi-aom--- 10.00g ^ ^ /dev/sdc(0),/dev/sdd(0),/dev/sde(0),/dev/sdf(0) [data_mimage_1] testvg iwi-aom--- 10.00g ^ ^ /dev/sdb(0) [data_mlog] testvg lwi-aom--- 4.00m ^ ^ /dev/sdb(2560)
Once Cpy%Sync is set to 100 (the mirror is in sync), break the mirror by removing the striped chain:
# lvconvert -m0 /dev/testvg/data /dev/sdc /dev/sdd /dev/sde /dev/sdf Logical volume testvg/data converted. # lvs -a -o +devices testvg LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices data testvg -wi-a----- 10.00g /dev/sdb(0)
Finish migration by cleanup procedure described at Cleanup procedure