Fusion-IO mezzonine UCS cards on RedHat 6 Linux

Create server profile and install minimal RedHat 6 on UCS blade. Check with lspci command if you see the hardware:

# lspci | grep -i fusion
81:00.0 Mass storage controller: Fusion-io ioDrive2 (rev 04)

Software Installation

You have to register at FusionIO support site to download required RPMs. Following RPMs are required: fio-common, fio-firmware, fio-sysvinit, fio-util, libvsl, iomemory-vsl.src.rpm. There are some precompiled modules avaliable for download, but we will use SRC rpm and recompile module. Install RPMs required to this compilation:

# yum install rpm-build kernel-devel

Compile and load driver itself:

root@localhost:/tmp/FIO # rpmbuild --rebuild iomemory-vsl-3.2.10.1509-1.0.el6.src.rpm
.....
Wrote: /root/rpmbuild/RPMS/x86_64/iomemory-vsl-2.6.32-504.12.2.el6.x86_64-3.2.10.1509-1.0.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/iomemory-vsl-config-2.6.32-504.12.2.el6.x86_64-3.2.10.1509-1.0.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/iomemory-vsl-source-3.2.10.1509-1.0.el6.x86_64.rpm
.....
# rpm -ihv /root/rpmbuild/RPMS/x86_64/iomemory-vsl-2.6.32-504.12.2.el6.x86_64-3.2.10.1509-1.0.el6.x86_64.rpm                                         
Preparing...                ########################################### [100%]                                                                                                        
   1:iomemory-vsl-2.6.32-504########################################### [100%]

Install other required RPMs:

# rpm -ihv fio-common-3.2.10.1509-1.0.el6.x86_64.rpm\
 fio-firmware-fusion-3.2.10.20150212-1.noarch.rpm\
 fio-sysvinit-3.2.10.1509-1.0.el6.x86_64.rpm\
 fio-util-3.2.10.1509-1.0.el6.x86_64.rpm\
 libvsl-3.2.10.1509-1.0.el6.x86_64.rpm

Hardware checks and care

Load driver for first time and verify hardware state

# modprobe iomemory-vsl
# lspci | grep Fusion
81:00.0 Mass storage controller: Fusion-io ioDrive2 (rev 04)
# lspci -s 81:00.0 -v
81:00.0 Mass storage controller: Fusion-io ioDrive2 (rev 04)
        Subsystem: Cisco Systems Inc Device 00bc
        Physical Slot: 14
        Flags: bus master, fast devsel, latency 0, IRQ 128
        Memory at fbe20000 (32-bit, non-prefetchable) [size=8K]
        Memory at fbe00000 (32-bit, non-prefetchable) [size=128K]
        Expansion ROM at fbd00000 [disabled] [size=1M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [60] Express Endpoint, MSI 01
        Capabilities: [9c] MSI-X: Enable- Count=1 Masked-
        Kernel driver in use: iodrive
        Kernel modules: iomemory-vsl
# fio-status

Found 1 ioMemory device in this system
Driver version: 3.2.10 build 1509

Adapter: Cisco Mezz
        Cisco UCS 785GB MLC Fusion-io ioDrive2, Product Number:MFIO785M, SN:XXXXXXXXXXX
        External Power: NOT connected
        PCIe Power limit threshold: 24.75W
        Connected ioMemory modules:
          fct0: Product Number:MFIO785M, SN:XXXXXXXXX

fct0    Attached
        Fontana, Product Number:MFIO785M, SN:XXXXXXXXX
        PCI:81:00.0, Slot Number:1
        Firmware v7.1.15, rev 110356 Public
        785.00 GBytes device size
        Internal temperature: 42.82 degC, max 43.31 degC
        Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00%
        Contained VSUs:
          fioa: ID:0, UUID:43d45254-7fa7-42b3-893d-8aa0dbc2224e

fioa    State: Online, Type: block device
        ID:0, UUID:43d45254-7fa7-42b3-893d-8aa0dbc2224e
        785.00 GBytes device size

Using devices

Low level formatting device with 4k page size will reduce memory usage of driver and improve performance. Scince Oracle use 8k as datablock size, 4k page size is quite suitable.

# fio-format -b 4k /dev/fct0

/dev/fct0: Error detected. The device is not detached.

WARNING: Formatting will destroy any existing data on the device!
Do you wish to continue [y/n]? n
Aborted.
# fio-detach /dev/fct0
Detaching: [====================] (100%) -
 fioa - detached.
# fio-format -b 4k /dev/fct0

/dev/fct0: Creating block device.
  Block device of size 785.00GBytes (731.09GiBytes).
  Using block (sector) size of 4096 bytes.

WARNING: Formatting will destroy any existing data on the device!
Do you wish to continue [y/n]? y

WARNING: Do not interrupt the formatting! If interrupted, the fio-sure-erase utility may help recover from format errors. Please see documentation or contact support.

Formatting: [====================] (100%) /
 /dev/fct0 - format successful.

Now attach block device:

# fio-attach /dev/fct?
Attaching: [====================] (100%) /
 fioa - attached.

If you plan to use it with LVM, then create PV aligned to 4k, create VG and create EXT3 using 4k as block size.

Scince I do not see option extending FIO device, I'll not use LVM over it

# mkfs.ext3 -j -m0 -b4096 /dev/fioa

Init scripts and auto mounts

It is looks like active cache resides in huge driver's memory allocation. Then it should be "flushed" to device during shutdown. This procedure includes umount, deactivating VG, deattaching fio devices. Deattaching do real final writing. This mean that these cards does not like power cut-off and data may become corrupted. Consider good backup policy (as usual).

An RPM fio-sysvinit contains initscripts that works well; you just have to configure it. First of all fix /etc/fstab with noauto attribute:

# grep fioa /etc/fstab
/dev/fioa               /oradbs/oratemp         ext3    defaults,noauto 0 0

Then disable udev taking care about FusionIO cards by editing /etc/modprobe.d/iomemory-vsl.conf as follow:

# To keep ioDrive from auto loading at boot when using udev, uncomment below
blacklist iomemory-vsl

# disable auto attach
# options iomemory-vsl auto_attach=0

# disable parallel attach for multiple cards
# options  iomemory-vsl parallel_attach=0

# To allow the ioDrive driver to load on SLES11, set "allow_unsupported_modules 1" in /etc/modprobe.d/unsupported-modules

Verify that iomemory-vsl started at boot time:

# chkconfig --list iomemory-vsl
iomemory-vsl    0:off   1:off   2:on    3:on    4:on    5:on    6:off

Configure init script to do what you need by editting /etc/sysconfig/iomemory-vsl, like:

# egrep -v "^#|^$" /etc/sysconfig/iomemory-vsl
ENABLED=1
TIMEOUT=15
VERBOSE=1
KILL_PROCS_ON_UMOUNT=1
SWAP=0
IOMEMORY_MOD_OPTS=""
MD_ARRAYS=""
LVM_VGS=""
LVM_DETECT=0
MOUNTS="/dev/fioa /oradbs/oratemp"

Edited strings are bolded here. Reboot server and verify boot messages, You SHOULD NOT SEE lines like that:

kernel: fioinf Cisco UCS 785GB MLC Fusion-io ioDrive2 0000:81:00.0: ***************************************************
kernel: fioinf Cisco UCS 785GB MLC Fusion-io ioDrive2 0000:81:00.0: *** unclean shutdown detected, re-scanning log. ***
kernel: fioinf Cisco UCS 785GB MLC Fusion-io ioDrive2 0000:81:00.0: *** this may take several minutes.              ***
kernel: fioinf Cisco UCS 785GB MLC Fusion-io ioDrive2 0000:81:00.0: ***************************************************

Some Performance data

# hdparm -tT /dev/fioa

/dev/fioa:
 Timing cached reads:   19754 MB in  2.00 seconds = 9892.81 MB/sec
 Timing buffered disk reads: 4366 MB in  3.00 seconds = 1455.32 MB/sec

bonnie++ -d /oradbs/oratemp -s 200g -m voleg