Linux multipath Boot from SAN (RedHat 5)

Installing on multipath SAN LUN from scratch.

Create a LUN, map it to the server, create the appropriate zones. Modern Linux can be installed on multipath devices natively, so prepare all paths be available and on-line.

Configure BIOS to boot from SAN. Following info related to HP hardware with qla adapters. Use this only as reference, your hardware may be differ. Press <Ctrl-Q> to enter QLogic configuration during server boot procedure. HP G7 servers with iLO3 comes with "nice graphic" booting, that mask all starting messages including QLogic's. Disable this in BIOS or press <Esc> key when this promted. The configuration menu appear. Choose first adapter and enable booting, select paths to boot. Save settings and repeat the same with second adapter.

When finished with BIOS, boot from your installation media. Add an option mpath to other options you usually use. Install as usual.

The multipath daemon starts rather late at boot time (after local mounting of FS). Therefore /boot is not present at an early stage of loading. Init scripts detect FSCK errors and bail out to a single-user mode. Correct /boot entry in /etc/fstab to enable the option _netdev. Because you are in single-user mode, the root FS still mounted in read-only mode, you should remount it read-write before editing /etc/fstab:

# mount -o rw,remount /
# vi /etc/fstab
# grep /boot /etc/fstab
/dev/mapper/mpath0p1    /boot                   ext3    _netdev        1 2
# mount -o ro,remount /
# reboot

Check that everything is OK after OS comes back. The most interesting file is /etc/multipath.conf:

# cat /etc/multipath.conf
defaults {
        user_friendly_names yes
}

blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^(hd|xvd|vd)[a-z]*"
        wwid "*"
}

# Make sure our multipath devices are enabled.

blacklist_exceptions {
        wwid "360000970000292600282533030374133"
}

This file excludes everything except the boot device. This may be appropriate, for example, if other devices need to be controlled using Veritas multipath. If you need other devices to be managed using native multipath too, add them to the list of blacklist_exceptions.

Migrating existing installation to multipath SAN LUN

Add a target LUN to existing Linux server. Rescan FC devices, like:

# for FC in /sys/class/fc_host/host?/issue_lip ; do echo "1" > $FC ; sleep 5 ; done ; sleep 20

Look in /var/log/messages for lines like:

May  2 09:39:58 hostname kernel: qla2xxx 0000:06:00.1: LOOP DOWN detected (4 0 0 0).
May  2 09:39:58 hostname kernel: qla2xxx 0000:06:00.1: LIP reset occured (f7f7).
May  2 09:39:58 hostname kernel: qla2xxx 0000:06:00.1: LOOP UP detected (4 Gbps).
May  2 09:39:58 hostname kernel:   Vendor: EMC       Model: SYMMETRIX         Rev: 5874
May  2 09:39:58 hostname multipathd: sddt: add path (uevent) 
May  2 09:39:58 hostname kernel:   Type:   Direct-Access                      ANSI SCSI revision: 04
May  2 09:39:58 hostname kernel: SCSI device sddt: 253440000 512-byte hdwr sectors (129761 MB)
May  2 09:39:58 hostname kernel: sddt: Write Protect is off
May  2 09:39:58 hostname kernel: SCSI device sddt: drive cache: write through
May  2 09:39:58 hostname kernel: SCSI device sddt: 253440000 512-byte hdwr sectors (129761 MB)
May  2 09:39:58 hostname kernel: sddt: Write Protect is off
May  2 09:39:58 hostname kernel: SCSI device sddt: drive cache: write through
May  2 09:39:58 hostname kernel:  sddt: unknown partition table
May  2 09:39:59 hostname kernel: sd 1:0:0:61: Attached scsi disk sddt
May  2 09:39:59 hostname kernel: sd 1:0:0:61: Attached scsi generic sg123 type 0

Check for new WWID:

# ll /dev/disk/by-id/ | grep sddt
lrwxrwxrwx 1 root root 10 May  2 09:39 scsi-360000970000292600282533030374133 -> ../../sddt

Add it to blacklist_exceptions list and name it unique:

blacklist_exceptions {
# SRC:
        wwid "360000970000292600282533030374136"
# TARGET:
        wwid "360000970000292600282533030374133"
}

multipaths {
        multipath {
                wwid    360000970000292600282533030374133
                alias   mroot0
        }
}

Rebuild multipath tables:

# multipath
mroot0 (360000970000292600282533030374133) dm-11 EMC,SYMMETRIX
[size=121G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 0:0:0:61 sdds 71:160 [active][ready]
 \_ 1:0:0:61 sddt 71:176 [active][ready]
# multipath -ll

You should see both source and target LUNs here. Partition new device /dev/mpath/mroot0 with fdisk. Refresh multipath (otherwise new partitions will not known). You can use kpartx as an alternative for partition rescan.

# fdisk /dev/mpath/mroot0
....
# multipath -f mroot0
# multipath

Format new /boot. Copy content and update /etc/fstab for new /boot mount.

# mkfs.ext3 -j -m0 /dev/mpath/mroot0p1
# mkdir /boot.new
# mount /dev/mpath/mroot0p1 /boot.new
# (cd /boot && tar cf - .)|(cd /boot.new && tar xpvfB -)
# umount /boot /boot.new
# mount /dev/mpath/mroot0p1 /boot
# vi /etc/fstab
# grep /boot /etc/fstab
/dev/mapper/mroot0p1    /boot                   ext3    _netdev        1 2

Now migrate the LVM data using pvremove:

# pvcreate /dev/mpath/mroot0p2
# vgextend rootvg /dev/mpath/mroot0p2
# pvmove /dev/mpath/mpath0p2
# vgreduce rootvg /dev/mpath/mpath0p2
# pvremove /dev/mpath/mpath0p2

Update multipath configuration file, remove old LUN leaving only new WWID. Please check before this, that first partition unmounted and second partition removed from LVM. The LUN have to be not in use.

# multipath -f mpath0
# multipath -ll
# vi /etc/multipath.conf
# multipath
# multipath -ll

Place this filter line into /etc/lvm/lvm.conf to make LVM ignore other disks. It is important to mask real SCSI disks combines multipath device, because they will have LVM headers too, that can confuse LVM. LVM can desides to use single path device instead of multipath, then you can loose redundancy.

filter = [ "a|/dev/mpath/|","a|/dev/mapper/m|","a|/dev/cciss/|","r/.*/" ]

Rebuild initrd to include new /etc/multipath.conf and /etc/lvm/lvm.conf. Old initrd does not know about new LUN and will look for old one. This is most important step if you want your system able to boot.

# cd /boot
# mkinitrd -f initrd-$(uname -r).img $(uname -r)

Update GRUB MBR now similar to following:

# grub
grub> device (hd1) /dev/mapper/mroot0
device (hd1) /dev/mapper/mroot0
grub> root (hd1,0)
root (hd1,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup (hd1)
setup (hd1)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd1)"...  15 sectors are embedded.
succeeded
 Running "install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.
grub> quit

Troubleshooting

If you missed one of steps above and something goes wrong, then you can boot from installation media in rescue mode adding mpath option:

boot: linux rescue mpath

Boot DR servers from SRDF disks

SRDF disks, presented to DR servers, will have their own WWID. They will be ignored by /etc/multipath.conf that will cause single path boot and LVM errors. You have to add this WWID to blacklist_exceptions prior switching to DR site. Detect WWID :

# symdev -sid XXX show 3377 | grep WWN
    Device WWN               : 60000970000292601446533033333737

Linux will see same number with 3 prepends (not sure, what this mean). Add it to blacklist_exceptions and aliases list:

blacklist_exceptions {
        wwid "360000970000292601446533033333737"
        wwid "360000970000292600232533031364336"
}

multipaths {
        multipath {
                wwid    360000970000292600232533031364336
                alias   mroot0
        }
        multipath {
                wwid    360000970000292601446533033333737
                alias   mroot0
        }
}

Duplicate alias definition will not conflicts, because only one of LUN will exist on each site.

Update initrd with changes:

# cd /boot
# mkinitrd -f initrd-$(uname -r).img $(uname -r)

See also HOWTO LUNs on Linux using native tools
If this was helpfull, please donate
Updated on Sun Sep 8 13:41:27 IDT 2013 by Oleg Volkov More documentations here