HOWTO align VMware Linux VMDK files

Alignment problem is well known and causes multiply IO requests for every single OS request. This saturates IO channels ; performance degrades. Alignment of VMFS done by selecting correct volume type on storage level. Lets suppose that all done well and VMDK file beguns exactly on storage block start. Modern disks (especially SSD) and storages works with 4k block size (pages) emulation 512 block access. This emulating costs in performance degradation. We'll try to bring things back to 4k.

Make existing Linux aligned

Starting point: you have Linux installed on single vmdk file, like:

# fdisk -l /dev/sda

Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        1044     8281507+  8e  Linux LVM
where /boot (and GRUB) sits in /dev/sda1 and MBR, and rest of OS hidden by LVM in /dev/sda2. I am going to separate these partitions to different disks, achieving easy resizing as side effect.

Add (on-line) two new disks (vmdk) to Linux VM. One small disk to fit /boot only (150m enough) and second to hold rest OS (mine fits in 8g). I'd recommend to keep application data on separate disk (vmdk file). This makes management, clones, templates much easy. First disk would be not align, but it works at boot time only, so impact minimal.

Rescan devices in Linux:

# for SH in /sys/class/scsi_host/host?/scan ; do echo "- - -" > $SH ; done
# fdisk -l
...

Disk /dev/sdb: 157 MB, 157286400 bytes
64 heads, 32 sectors/track, 150 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdc doesn't contain a valid partition table

Lets transfer LVM to aligned volume. Resulting disk is unpartitioned with data aligned to 4k boundary.

# pvcreate --dataalignment 4k /dev/sdc 
  Physical volume "/dev/sdc" successfully created
# pvs
  PV         VG     Fmt  Attr PSize PFree
  /dev/sda2  rootvg lvm2 a-   7.88G 2.78G
  /dev/sdc          lvm2 a-   8.00G 8.00G
# vgextend rootvg /dev/sdc
  Volume group "rootvg" successfully extended
# pvmove /dev/sda2
  /dev/sda2: Moved: 0.0%
...
# pvs
  PV         VG     Fmt  Attr PSize PFree
  /dev/sda2  rootvg lvm2 a-   7.88G 7.88G
  /dev/sdc   rootvg lvm2 a-   7.97G 2.88G
# vgreduce rootvg /dev/sda2
  Removed "/dev/sda2" from volume group "rootvg"
# pvremove /dev/sda2
  Labels on physical volume "/dev/sda2" successfully wiped

Transfer /boot and GRUB:

# # DANGEROUS !! THIS WILL REBUILD PARTITION TABLE on /dev/sdb !!!
# echo -e ",,,,\n;\n;\n;" | sfdisk /dev/sdb
# mkfs.ext3 -j -m0 -L /boot /dev/sdb1
# mkdir /boot.new ; mount /dev/sdb1 /boot.new
# (cd /boot && tar cf - .)|(cd /boot.new && tar xpvfB -)
# grub
grub> device (hd1) /dev/sdb

grub> root (hd1,0)
 Filesystem type is ext2fs, partition type 0x83

grub> setup (hd1)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd1)"... failed (this is not fatal)
 Running "embed /grub/e2fs_stage1_5 (hd1,0)"... failed (this is not fatal)
 Running "install /grub/stage1 (hd1) /grub/stage2 p /grub/grub.conf "... succeeded
Done.

grub> quit

It is time to turn off VM, remove old disk and change SCSI ids for new disks, set 0 for small disk and 1 for bigger one.

Adding FS for application

Add (on-line) disk for data (mine 30g). Set it's SCSI id to be (1:0). This will create also "New SCSI Controller" Change it's type to VMware Paravirtual. This type is not so sutable for OS disks, because it limits upgrading possibility, however it good for data/application.

# for SH in /sys/class/scsi_host/host?/scan ; do echo "- - -" > $SH ; done
# fdisk -l
...
Disk /dev/sdc: 32.2 GB, 32212254720 bytes
64 heads, 32 sectors/track, 30720 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdc doesn't contain a valid partition table

Create aligned PV, VG, and format FS with 4k blosk size:

# pvcreate --dataalignment 4k /dev/sdc 
  Physical volume "/dev/sdc" successfully created
# vgcreate orahome /dev/sdc
  Volume group "orahome" successfully created
# lvcreate -n export -L25g /dev/orahome
  Logical volume "export" created
# mkfs.ext3 -j -m0 -b4096 /dev/orahome/export 
# mkdir /export && mount /dev/orahome/export /export

Paravirtual SCSI controller starts after FSCK checks, then its FS should be marked as _netdev at /etc/fstab file:

# grep orahome /etc/fstab
/dev/orahome/export	/export	ext3	_netdev	2 2

Extending OS disk on-line

Due to the fact, that LVM does NOT resides on partition now, there is no need in partition resizing. Then reboot could be avoided and system disk extend can be performed on-line. Change OS disk size to 10g now. Rescan:

# for device in /sys/block/sd* ; do echo 1 > $device/device/rescan ; sleep 2 ; done
# pvs
  PV         VG      Fmt  Attr PSize  PFree
  /dev/sdb   rootvg  lvm2 a-    7.97G 2.88G
  /dev/sdc   orahome lvm2 a-   30.00G 5.00G
# pvresize /dev/sdb
  Physical volume "/dev/sdb" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
# pvs
  PV         VG      Fmt  Attr PSize  PFree
  /dev/sdb   rootvg  lvm2 a-    9.97G 4.88G
  /dev/sdc   orahome lvm2 a-   30.00G 5.00G

The rest is trivial.


If this was helpfull, please donate
Updated on Thu Jun 28 11:06:07 IDT 2012 by Oleg Volkov More documentations here