Create a LUN, map it to the server, create the appropriate zones. Modern Linux can be installed on multipath devices natively, so prepare all paths be available and on-line.
Configure BIOS to boot from SAN. Following info related to HP hardware with qla adapters. Use this only as reference, your hardware may be differ. Press <Ctrl-Q> to enter QLogic configuration during server boot procedure. HP G7 servers with iLO3 comes with "nice graphic" booting, that mask all starting messages including QLogic's. Disable this in BIOS or press <Esc> key when this promted. The configuration menu appear. Choose first adapter and enable booting, select paths to boot. Save settings and repeat the same with second adapter.
When finished with BIOS, boot from your installation media. Add an option mpath to other options you usually use. Install as usual.
The multipath daemon starts rather late at boot time (after local mounting of FS). Therefore /boot is not present at an early stage of loading. Init scripts detect FSCK errors and bail out to a single-user mode. Correct /boot entry in /etc/fstab to enable the option _netdev. Because you are in single-user mode, the root FS still mounted in read-only mode, you should remount it read-write before editing /etc/fstab:
# mount -o rw,remount / # vi /etc/fstab # grep /boot /etc/fstab /dev/mapper/mpath0p1 /boot ext3 _netdev 1 2 # mount -o ro,remount / # reboot
Check that everything is OK after OS comes back. The most interesting file is /etc/multipath.conf:
# cat /etc/multipath.conf
defaults {
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(hd|xvd|vd)[a-z]*"
wwid "*"
}
# Make sure our multipath devices are enabled.
blacklist_exceptions {
wwid "360000970000292600282533030374133"
}
This file excludes everything except the boot device. This may be appropriate, for example, if other devices need to be controlled using Veritas multipath. If you need other devices to be managed using native multipath too, add them to the list of blacklist_exceptions.
Add a target LUN to existing Linux server. Rescan FC devices, like:
# for FC in /sys/class/fc_host/host?/issue_lip ; do echo "1" > $FC ; sleep 5 ; done ; sleep 20
Look in /var/log/messages for lines like:
May 2 09:39:58 hostname kernel: qla2xxx 0000:06:00.1: LOOP DOWN detected (4 0 0 0). May 2 09:39:58 hostname kernel: qla2xxx 0000:06:00.1: LIP reset occured (f7f7). May 2 09:39:58 hostname kernel: qla2xxx 0000:06:00.1: LOOP UP detected (4 Gbps). May 2 09:39:58 hostname kernel: Vendor: EMC Model: SYMMETRIX Rev: 5874 May 2 09:39:58 hostname multipathd: sddt: add path (uevent) May 2 09:39:58 hostname kernel: Type: Direct-Access ANSI SCSI revision: 04 May 2 09:39:58 hostname kernel: SCSI device sddt: 253440000 512-byte hdwr sectors (129761 MB) May 2 09:39:58 hostname kernel: sddt: Write Protect is off May 2 09:39:58 hostname kernel: SCSI device sddt: drive cache: write through May 2 09:39:58 hostname kernel: SCSI device sddt: 253440000 512-byte hdwr sectors (129761 MB) May 2 09:39:58 hostname kernel: sddt: Write Protect is off May 2 09:39:58 hostname kernel: SCSI device sddt: drive cache: write through May 2 09:39:58 hostname kernel: sddt: unknown partition table May 2 09:39:59 hostname kernel: sd 1:0:0:61: Attached scsi disk sddt May 2 09:39:59 hostname kernel: sd 1:0:0:61: Attached scsi generic sg123 type 0
Check for new WWID:
# ll /dev/disk/by-id/ | grep sddt lrwxrwxrwx 1 root root 10 May 2 09:39 scsi-360000970000292600282533030374133 -> ../../sddt
Add it to blacklist_exceptions list and name it unique:
blacklist_exceptions {
# SRC:
wwid "360000970000292600282533030374136"
# TARGET:
wwid "360000970000292600282533030374133"
}
multipaths {
multipath {
wwid 360000970000292600282533030374133
alias mroot0
}
}
Rebuild multipath tables:
# multipath mroot0 (360000970000292600282533030374133) dm-11 EMC,SYMMETRIX [size=121G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=2][active] \_ 0:0:0:61 sdds 71:160 [active][ready] \_ 1:0:0:61 sddt 71:176 [active][ready] # multipath -ll
You should see both source and target LUNs here. Partition new device /dev/mpath/mroot0 with fdisk. Refresh multipath (otherwise new partitions will not known). You can use kpartx as an alternative for partition rescan.
# fdisk /dev/mpath/mroot0 .... # multipath -f mroot0 # multipath
Format new /boot. Copy content and update /etc/fstab for new /boot mount.
# mkfs.ext3 -j -m0 /dev/mpath/mroot0p1 # mkdir /boot.new # mount /dev/mpath/mroot0p1 /boot.new # (cd /boot && tar cf - .)|(cd /boot.new && tar xpvfB -) # umount /boot /boot.new # mount /dev/mpath/mroot0p1 /boot # vi /etc/fstab # grep /boot /etc/fstab /dev/mapper/mroot0p1 /boot ext3 _netdev 1 2
Now migrate the LVM data using pvremove:
# pvcreate /dev/mpath/mroot0p2 # vgextend rootvg /dev/mpath/mroot0p2 # pvmove /dev/mpath/mpath0p2 # vgreduce rootvg /dev/mpath/mpath0p2 # pvremove /dev/mpath/mpath0p2
Update multipath configuration file, remove old LUN leaving only new WWID. Please check before this, that first partition unmounted and second partition removed from LVM. The LUN have to be not in use.
# multipath -f mpath0 # multipath -ll # vi /etc/multipath.conf # multipath # multipath -ll
Place this filter line into /etc/lvm/lvm.conf to make LVM ignore other disks. It is important to mask real SCSI disks combines multipath device, because they will have LVM headers too, that can confuse LVM. LVM can desides to use single path device instead of multipath, then you can loose redundancy.
filter = [ "a|/dev/mpath/|","a|/dev/mapper/m|","a|/dev/cciss/|","r/.*/" ]
Rebuild initrd to include new /etc/multipath.conf and /etc/lvm/lvm.conf. Old initrd does not know about new LUN and will look for old one. This is most important step if you want your system able to boot.
# cd /boot # mkinitrd -f initrd-$(uname -r).img $(uname -r)
Update GRUB MBR now similar to following:
# grub grub> device (hd1) /dev/mapper/mroot0 device (hd1) /dev/mapper/mroot0 grub> root (hd1,0) root (hd1,0) Filesystem type is ext2fs, partition type 0x83 grub> setup (hd1) setup (hd1) Checking if "/boot/grub/stage1" exists... no Checking if "/grub/stage1" exists... yes Checking if "/grub/stage2" exists... yes Checking if "/grub/e2fs_stage1_5" exists... yes Running "embed /grub/e2fs_stage1_5 (hd1)"... 15 sectors are embedded. succeeded Running "install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.conf"... succeeded Done. grub> quit
If you missed one of steps above and something goes wrong, then you can boot from installation media in rescue mode adding mpath option:
boot: linux rescue mpath
SRDF disks, presented to DR servers, will have their own WWID. They will be ignored by /etc/multipath.conf that will cause single path boot and LVM errors. You have to add this WWID to blacklist_exceptions prior switching to DR site. Detect WWID :
# symdev -sid XXX show 3377 | grep WWN
Device WWN : 60000970000292601446533033333737
Linux will see same number with 3 prepends (not sure, what this mean). Add it to blacklist_exceptions and aliases list:
blacklist_exceptions {
wwid "360000970000292601446533033333737"
wwid "360000970000292600232533031364336"
}
multipaths {
multipath {
wwid 360000970000292600232533031364336
alias mroot0
}
multipath {
wwid 360000970000292601446533033333737
alias mroot0
}
}
Duplicate alias definition will not conflicts, because only one of LUN will exist on each site.
Update initrd with changes:
# cd /boot # mkinitrd -f initrd-$(uname -r).img $(uname -r)