Build NAS for Small Buisiness Office

I plan to take the following steps:

  1. The redundancy will be provided by the LVM engine. All LVs will be converted to mirrors.
  2. The /boot partition is not really needed and will be moved to LVM.
  3. The EFI boot partition must exist on both drives.
  4. ZFS will be created on an already redundant LVM volume.
  5. The backup strategy will be implemented by taking snapshots and configuring replication to a remote site.
  6. Setup DHCP, firewall, privoxy and SAMBA services

Hardware and initial installation

For the needs of SBO, a high-performance computer is not required; almost any hardware will do here. I'm planning to use ZFS, known as memory hungry software, so at least 4G RAM is required. Redundancy requires at least two hard drives. This office is really small, so a computer with 8G RAM and two 220G SSDs is perfectly acceptable.

A minimal Fedora 37 server was installed with a slight modification to the disk layout. Please remember to set the new VG to maximum space and not "auto" size, otherwise you won't be able to create a mirror due to lack of space. I made root 5G in size and ext4 file system. Added 1G swap LV. /var and /var/log and /var/lib ext4 were also added, 1G each. After installation, I got the following result:

root@server:~ # lsblk -e 2,7,11,252
sda                 8:0    0 223.6G  0 disk 
├─sda1              8:1    0   600M  0 part /boot/efi
├─sda2              8:2    0     1G  0 part /boot
└─sda3              8:3    0   222G  0 part 
  ├─rootvg-slash  253:0    0     5G  0 lvm  /
  ├─rootvg-swap   253:1    0     1G  0 lvm  [SWAP]
  ├─rootvg-varlog 253:2    0     1G  0 lvm  /var/log
  ├─rootvg-var    253:3    0     1G  0 lvm  /var
  └─rootvg-varlib 253:4    0     1G  0 lvm  /var/lib
sdb                 8:16   0 223.6G  0 disk 
root@server:~ # df -l -x tmpfs -x devtmpfs
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-slash   4.9G  946M  3.7G  21% /
/dev/sda2                  960M  166M  795M  18% /boot
/dev/mapper/rootvg-var     974M  212K  907M   1% /var
/dev/mapper/rootvg-varlib  974M   25M  883M   3% /var/lib
/dev/mapper/rootvg-varlog  974M   18M  889M   2% /var/log
/dev/sda1                  599M  7.1M  592M   2% /boot/efi
Bold font indicates the command that you are typing, the rest is the output of the command.

Add some usefull packages:

root@server:~ # dnf install bash-completion rsync rsyslog gdisk

Migrate /boot partition to LVM

This particular server cannot be switched from EFI to Legacy BIOS (Compatibility Support Mode). GRUB2 itself is implemented as an EFI application and resides on the EFI filesystem. I checked this particular grubx64.efi installed with Fedora 37 for its builtins modules. Found that it can work with mirrored LVM directly. So, let's move the contents of the /boot partition to the LVM volume to mirror it later.

root@server:~ # lvcreate -n boot -L256m rootvg
  Logical volume "boot" created.
root@server:~ # mkfs.ext4 -j -m 0 /dev/rootvg/boot
root@server:~ # mount /dev/rootvg/boot /mnt
root@server:~ # umount /boot/efi
root@server:~ # rsync -av /boot/ /mnt/

Update the /etc/fstab to replace /boot be LVM instead of partition:

root@server:~ # umount /mnt /boot
root@server:~ # vi /etc/fstab
root@server:~ # grep boot /etc/fstab
/dev/rootvg/boot        /boot                   ext4     defaults        1 1
UUID=CC14-D6D2          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
root@server:~ # systemctl daemon-reload 
root@server:~ # mount -a
root@server:~ # df | grep boot
/dev/mapper/rootvg-boot    230M  127M  100M  57% /boot
/dev/sda1                  599M  7.1M  592M   2% /boot/efi

Note that although we copied the data to the LVM volume, the boot process still uses the original partition according to the /boot/efi/EFI/fedora/grub.cfg file located on the EFI partition. Let's change it to use new LV:

root@server:~ # echo 'configfile (lvm/rootvg-boot)/grub2/grub.cfg' > /boot/efi/EFI/fedora/grub.cfg

Then restart the server to check its loading. If the reboot works, but you're still not sure if the boot process is using LVM, you can mount the original partition somewhere and rename its grub.cfg to a different name.

Make Logical Volumes mirrored

It's time to format the second disk according to the planned scheme

root@server:~ # parted -a optimal /dev/sdb
GNU Parted 3.5
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
(parted) unit mib
(parted) mkpart primary fat32 1 20
(parted) name 1 "EFI System Partition"
(parted) set 1 esp on
(parted) mkpart primary 20 100%
(parted) set 2 lvm on
(parted) print
Model: ATA TS240GSSD220S (scsi)
Disk /dev/sdb: 228937MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start    End        Size       File system  Name                  Flags
 1      1.00MiB  20.0MiB    19.0MiB    fat32        EFI System Partition  boot, esp
 2      20.0MiB  228936MiB  228916MiB               primary               lvm

(parted) quit
Information: You may need to update /etc/fstab.

The first section is for EFI and we will deal with it later. The second partition will be formatted as PV and added to the VG.

root@server:~ # pvcreate /dev/sdb2
  Physical volume "/dev/sdb1" successfully created.
root@server:~ # vgextend rootvg /dev/sdb2
  Volume group "rootvg" successfully extended
root@server:~ # pvs
  PV         VG     Fmt  Attr PSize    PFree   
  /dev/sda3  rootvg lvm2 a--   221.98g  212.73g
  /dev/sdb2  rootvg lvm2 a--  <223.55g <223.55g

It is important that both volumes have free space to create not only the data, but also the journal. If you missed the recommendation to set a maximum VG size during installation, then you must enlarge the partition, otherwise it will not be possible to create mirrors.

Convert all existing LVs to mirror ones.

root@server:~ # cd /dev/rootvg
root@server:/dev/rootvg # ls
boot  slash  swap  var  varlib  varlog
root@server:/dev/rootvg # for lv in * ; do lvconvert -m1 -y /dev/rootvg/$lv ; done
  Logical volume rootvg/boot successfully converted.
  Logical volume rootvg/slash successfully converted.
  Logical volume rootvg/swap successfully converted.
  Logical volume rootvg/var successfully converted.
  Logical volume rootvg/varlib successfully converted.
  Logical volume rootvg/varlog successfully converted.

Once the mirrored LVs are created, the existing initrd needs to be rebuilt or the reboot will fail.

root@server:~ # dracut -f /boot/initramfs-$(uname -r).img $(uname -r)

When booting from a broken mirror, the dracut initrd activates only those LVs that are specified on the kernel command line. To fix this behavior, replace CMDLINE in /etc/default/grub to only include "". This option will activate the entire VG rootvg during the boot phase. Then re-create the grub.cfg.

root@server:~ # grep CMDLINE /etc/default/grub
root@server:~ # grub2-mkconfig > /boot/grub2/grub.cfg
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...

Reboot the server to make sure it boots normally.

EFI partition

The second EFI partition must be created and copied over from the first. This data is mostly static, and rarely gets updated. Create a FAT32 file system and copy the original EFI content to its backup:

root@server:~ # mkdosfs /dev/sdb1
mkfs.fat 4.2 (2021-01-31)
root@server:~ # mkdir /boot/efib
root@server:~ # mount -t vfat /dev/sdb1 /boot/efib
root@server:~ # rsync -av /boot/efi/ /boot/efib/

Add the copied EFI partition to fstab for mounting. Since we want the system to boot without any of the disks, add the "nofail" option to fstab. Add the same nofail mount option for the sda1 drive.

/dev/sda1          /boot/efi               vfat    nofail,umask=0077,shortname=winnt 0 2
/dev/sdb1          /boot/efib              vfat    nofail,umask=0077,shortname=winnt 0 2

Reboot the server to make sure it loads well. Then disable the first drive in the BIOS or unplug it to make sure the OS able boots from the only surviving drive. Then reconnect or turn on the drive back, as we will need work on it in the next chapter.

Rebuild the mirror

In this step, we need to destroy everything that exists on the original sda drive and rebuild the mirror. The same procedure will be relevant for rebuilding the mirror in the event of a real disk failure.

Use wipefs command to remove all filesystems and GPT on original /dev/sda.

root@server:~ # wipefs -a -f /dev/sda3
/dev/sda3: 8 bytes were erased at offset 0x00000218 (LVM2_member): 4c 56 4d 32 20 30 30 31
root@server:~ # wipefs -a -f /dev/sda2
/dev/sda2: 4 bytes were erased at offset 0x00000000 (xfs): 58 46 53 42
root@server:~ # wipefs -a -f /dev/sda1
/dev/sda1: 8 bytes were erased at offset 0x00000052 (vfat): 46 41 54 33 32 20 20 20
/dev/sda1: 1 byte was erased at offset 0x00000000 (vfat): eb
/dev/sda1: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa
root@server:~ # wipefs -a -f /dev/sda
/dev/sda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sda: 8 bytes were erased at offset 0x6fc86d5e00 (gpt): 45 46 49 20 50 41 52 54
/dev/sda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
root@server:~ # reboot

After the reboot, we should have the same situation as if we had just replaced the failed sda drive. First we need to create a new partition table on disk. We will use the "gdisk" tool to replicate the existing partition table to the target disk. Run gdisk with the source disk, activate the expert command menu, and write to the target disk.

!! This procedure will copy GPT from /dev/sdb -> to /dev/sda !!
root@server:~ # gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): x

Expert command (? for help): u
Type device filename, or press <Enter> to exit: /dev/sda

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.

Expert command (? for help): q

Fix VG by removing missing PV and adding a new one for a replacement.

root@server:~ # pvs
  WARNING: Couldn't find device with uuid N4aGoj-IyCD-jScK-Cj91-wyEs-U6Ed-23VviJ.
  WARNING: VG rootvg is missing PV N4aGoj-IyCD-jScK-Cj91-wyEs-U6Ed-23VviJ (last written to /dev/sda3).
  PV         VG     Fmt  Attr PSize    PFree   
  /dev/sdb2  rootvg lvm2 a--  <223.55g  214.27g
  [unknown]  rootvg lvm2 a-m   221.98g <212.71g
root@server:~ # vgreduce rootvg --removemissing --force
  Wrote out consistent volume group rootvg.
root@server:~ # pvs
  PV         VG     Fmt  Attr PSize    PFree  
  /dev/sdb2  rootvg lvm2 a--  <223.55g 214.27g
root@server:~ # pvcreate /dev/sda2
  Physical volume "/dev/sda2" successfully created.
root@server:~ # vgextend rootvg /dev/sda2
  Volume group "rootvg" successfully extended
root@server:~ # pvs
  PV         VG     Fmt  Attr PSize    PFree   
  /dev/sda2  rootvg lvm2 a--  <223.55g <223.55g
  /dev/sdb2  rootvg lvm2 a--  <223.55g  214.27g

Then run repair for LVs:

root@server:~ # cd /dev/rootvg
root@server:/dev/rootvg # for lv in * ; do lvconvert --repair /dev/rootvg/$lv -y ; done
  Faulty devices in rootvg/boot successfully replaced.
  Faulty devices in rootvg/slash successfully replaced.
  Faulty devices in rootvg/swap successfully replaced.
  Faulty devices in rootvg/var successfully replaced.
  Faulty devices in rootvg/varlib successfully replaced.
  Faulty devices in rootvg/varlog successfully replaced.

Copy EFI partition content manually:

root@server:~ # mkdosfs /dev/sda1
mkfs.fat 4.2 (2021-01-31)
root@server:~ # mount /dev/sda1
root@server:~ # df | grep efi
/dev/sdb1                   19M  7.1M   12M  38% /boot/efib
/dev/sda1                   19M     0   19M   0% /boot/efi
root@server:~ # rsync -av /boot/efib/ /boot/efi/

Installing and configure ZFS

You can find installation instructions on the OpenZFS website. Briefly, it looks like this.

root@server:~ # dnf install -y$(rpm --eval "%{dist}").noarch.rpm
root@server:~ # dnf install -y kernel-devel
root@server:~ # dnf install -y zfs
root@server:~ # modprobe zfs
root@server:~ # echo zfs > /etc/modules-load.d/zfs.conf

When upgrading the kernel, you must rebuild the ZFS DKMS module. This is usually done with the command (you not need run this now):

root@server:~ # dkms install -m zfs -v $(rpm -q --queryformat %{VERSION} zfs)

The next step is to create a zpool that should use the redundant LV. So first we need to create such LV.

root@server:~ # lvcreate -m1 -n zpool -L100g rootvg
  Logical volume "zpool" created.

Then create zpool on it

root@server:~ # zpool create -o ashift=12 -m none z  /dev/rootvg/zpool
root@server:~ # zpool status 
  pool: z
 state: ONLINE

        NAME        STATE     READ WRITE CKSUM
        z           ONLINE       0     0     0
          zpool     ONLINE       0     0     0

errors: No known data errors

Finally, create filesystem with some flags usefull for SAMBA

root@server:~ # zfs create -o mountpoint=/home/samba -o casesensitivity=mixed z/samba

Updated on Wed Feb 22 08:32:01 IST 2023 More documentations here