Setup SUSE RMT + PXE server on Power

The scope

Installing SUSE LPARs in a Power architecture over a network is much easier than any other installation options. It is wise to set up a PXE environment when installing more than two SUSE LPARs. Refer to a number of articles on my blog detailing network deployment for various flavors of linuxes.

Once the installation is complete, maintenance of software updates becomes important, and an RMT server is definitely the easiest solution for this. It creates a network isolation for patch delivery still allowing access to the latest available subscription software. RMT can also be used directly during the installation phase, installing the latest packages right away.

The RMT server lacks a full PXE stack, in this article we will add it and configure all the missing components to get it. We will install it (RMT + PXE) directly on the Power equipment in order to eliminate doubts about the violation of the purchased licenses.

Brief overview of Power architecture

Like VmWare, for which it is not enough just to turn on the ESX server to get a complete infrastructure, the IBM Power infrastructure has some prerequisites and settings.

A virtual machine in the IBM world is called LPAR (Logical PARtition) because this machine is not virtual, but real hardware is allocated to create a virtual server. This process is managed by the HMC (Hardware Management Console), which plays a role similar to the VC in VmWare world. An HMC can be a physical or virtual server running on any other existing virtualization platform at the customer's site. The HMC is connected to a public network and managed through a web interface. The HMC is also connected to a dedicated private network. This dedicated network is not routable and is directly connected to all IBM Power servers, to the HMC interface. To achieve redundancy, it is common to install two vHMCs and two separate private networks.

The very first LPAR installed is usually the VIOS (Virtual Input Output Server). The VIOS helps share I/O cards (such as FC storage access cards and network cards) between payload LPARs. Again, two VIOS are usually installed to make the solution redundant.

Setup LPAR

NOTE:This part was written theoretically and should be rewritten next time have hands on with hardware itself.

Create on one VIOS "Virtual Library" and create there a virtual disk for new LPAR (usually 120g is enough).

Upload or transfer a desired ISO file (for example SLE-15-SP3-Full-ppc64le-QU2-Media1.iso) to "Virtual library". Configure vitual optical device and load an ISO to it. Pay attention that downlaoded iso is for ppc64le architecture, other versions will not suits.

Create an LPAR with two small (0.1) virtual CPU and 4G RAM. Create virtual SCSI adapter and mark it "required". Connect to it both virtual disk and virtual DVD. An HMC CLI commands could be helpfull:

lsrep
mkrep --sp rootvg --size 20G
mkvopt -name SLE15SP3.iso --file /home/padmin/SLE-15-SP3-Full-ppc64le-QU2-Media1.iso
lsmap --all
mkvdev -fbo -vadapter vhost2
loadopt -disk SLE15SP3.iso -vtd vtopt0
unloadopt -vtd vtopt0

Installation of SUSE

The boot will occur from ISO, and a default GRUB menu appear.

  1. Select Installation and press e to edit. Add vnc=1 vncpassword=P@ssw0rd to the line starting with linux word. When finished editting, press Ctrl-x to begin boot (the hint about Ctrl-x shown at screen bottom).
  2. See boot messages. When displayed a message that VNC server is active and its connecting info, connect using any vnc-viewer (even running on Windows) using password above.
  3. Select SUSE Linux Enterprise Server as a product, Next, accept the license, Next.
  4. The registration could be skipped, if you do not have internet access or registration info, Next.
  5. There is no additional Add On Product required for our purpose, Next.
  6. The Suggested Partitioning page comes with BTRFS suggestions. I do not recommend to use BTRFS due to my own experienced data lost. Select Expert Partitioner -> Start with Current Proposal
  7. Select disk (usually sda), Overview tab, Partition Table -> Create New Partition Table. Confirm partitions deletion. Use MS-DOS partition type, Next.
  8. Select Partitions tab and Add Partition -> Primary Partition, Next. Set Custom Size 8m, Next. Raw Volume, Next. Do not format device, Partition ID = PReP Boot, Next. This partition used to be ppc/ppc64 bootloader. Probably not needed anymore, to be checked.
  9. Add Partition -> Primary Partition, Next. Use Maximum Size, Next. Mark it Raw Volume, Next. Check Do not format device and set Partition ID = Linux LVM, Next.
  10. On the left side, click on Volume Management then Add Volume GRoup at right-bottom corner. Put rootvg at Volume Group Name field, click on Add All button in the middle, Next.
  11. Select rootvg on the left selector and Logical Volumes tab.
  12. Add Logical Volume and name it swap, Next. Set Custom Size to 2g, Next. Role is Swap, Next. Format device with Swap Filesystem, Mount device as swap Next.
  13. Add Logical Volume adn name it slash, Next. Set Custom Size be 8g, Next. Role is Operating System, Next. Format device with ext4 Filesystem, Mount device as /, Next. Please do not use BTRFS !
  14. Add Logical Volume and name it var, Next. Set Custom Size be 80g, Next. Role is Operating System, Next. Format device with xfs Filesystem, Mount device as /var, Next. The filesystem size depends on desired amount of replicated data. 25G is a usual usage of basic single distribution. Everything for single distribution will cost you about 80g. With the time passing you will need add another SP channels and usage can be increased significantly. Please do not use BTRS !!
  15. Accept.
  16. /boot does not required., Yes. Next.
  17. Set the TimeZone. You can try to click on Israel, or just enter Asia / Israel, Next.
  18. Skip User Creation, Next. Of course, you can create a user here, if your corporate policy forbid root user using.
  19. Set root password, Next.
  20. The summary of pending installation is shown. Select only Minimal Base System for Software. Probably disable Firewall, otherwice you have to add all next services to firewall rules. Definetely disable Kdump. Make text mode as default systemd target. Click Install, Install again.

Install and configure RMT

You have to enable module Server Applications (usually enabled) to be able install:

# zypper in rmt-server

Next is about configure an RMT according to the SUSE documentation.

After setting up your company credentials, you can add some channels for mirroring. Use the same official guide for this and "enable" the products you want. Synchronization will take time, mainly depending on the network bandwidth.

Configure an Installation DVD

The DVD media used during installation can be used as the installation source for the rest of the LPARS. First of all, let's mount it permanently into the appropriate location.

The installation source should be available via HTTP to be used during installation. The RMT software uses an NGINX HTTP server as its engine. As a result of installing RMT, there are two virtual hosts configured at NGINX. The /etc/nginx/nginx.conf file configures localhost then it cannot be used to serve external requests. The file /etc/nginx/vhosts.d/rmt-server-http.conf describes the virtual host for $HOSTNAME (we sat it as rmt earlier). This site serves files related to the path /usr/share/rmt/public. We will utilize this configuration and mount our installation disk below this location:

# ll /dev/disk/by-label/
total 0
lrwxrwxrwx 1 root root 9 May  8 12:14 SLE-15-SP3-Full-ppc64le201.3.001 -> ../../sr0
# mkdir /usr/share/rmt/public/install
# echo "/dev/disk/by-label/SLE-15-SP3-Full-ppc64le201.3.001 /usr/share/rmt/public/install auto defaults 0 0" >> /etc/fstab
# mount /usr/share/rmt/public/install
mount: /usr/share/rmt/public/install: WARNING: device write-protected, mounted read-only.
# mkdir /usr/share/rmt/public/autoyast

The /usr/share/rmt/public/autoyast directory will contain your custom autoyast.xml files later.

Configure TFTP

Install TFTP server and client.

# zypper in tftp

In addition to the TFTP server itself, you need to populate the TFTP root directory with the necessary files. This could be done in two ways. You can copy /usr/share/rmt/public/install/boot from an already mounted DVD or install the package like:

# zypper in tftpboot-installation-SLE-15-SP3-ppc64le

This package unpacks some files into the /usr/share/tftpboot-installation/SLE-15-SP3-ppc64le/ directory. Let's set this directory as the TFTPROOT directory by editing the file:

# egrep -v "^$|^#" /etc/sysconfig/tftp
TFTP_USER="tftp"
TFTP_OPTIONS="-v"
TFTP_DIRECTORY="/usr/share/tftpboot-installation/SLE-15-SP3-ppc64le"

Enable TFTP server and add it to firewall rules if firewall still in use:

# systemctl enable --now tftp.socket
Created symlink /etc/systemd/system/sockets.target.wants/tftp.socket → /usr/lib/systemd/system/tftp.socket.
# firewall-cmd --add-service=tftp --add-service=dhcp --permanent
# firewall-cmd --reload
# firewall-cmd --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: ssh dhcpv6-client http https tftp dhcp
  ports: 10050/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

Test TFTP server:

# cd /tmp
/tmp # tftp localhost
tftp> get README
tftp> quit
/tmp # ll README
-rw-r--r-- 1 root root 442 May  8 12:35 README

Configure DHCP

PXE uses a specially configured DHCP server, but usually another DHCP server already exists in the organization. Therefore, we will configure our own DHCP server to serves only certain clients and does not interfere with other DHCP requests.

Install the package:

# zypper in dhcp-server

Put the following configuration file:

# /etc/dhcpd.conf
allow booting;
allow bootp;
ddns-update-style none;
default-lease-time 14400;
deny unknown-clients;

# Send hostname over DHCP
get-lease-hostnames true;
use-host-decl-names on;

subnet 192.168.0.0 netmask 255.255.255.0 {
        option domain-name "localdomain";
        #option domain-name-servers 192.168.0.1;
        option routers          192.168.0.1;
        #option ntp-servers      192.168.0.1;
        option subnet-mask      255.255.255.0;
# This is an IP of RMT server, where TFTP server run and serve "filename" below.
        next-server     192.168.0.20;
# This is GRUB2 for ppc64le architecture. This option could be used on host level too.
	filename        "/boot/ppc64le/grub2-ieee1275/core.elf"; 

	host ppc {
		hardware ethernet 52:54:00:53:36:51;
		fixed-address   192.168.0.11;
	}
}

Define DHCPD_INTERFACE at /etc/sysconfig/dhcpd file, then start DHCP server:

# systemctl enable --now dhcpd

Configure GRUB

There are two grub.cfg files that you can find in the /usr/share/tftpboot-installation/SLE-15-SP3-ppc64le directory we deployed. The effective file is /usr/share/tftpboot-installation/SLE-15-SP3-ppc64le/boot/ppc64le/grub2-ieee1275/grub.cfg, while the second one exists for an unknown reason. The best way is to create a hard link between them:

# rm /usr/share/tftpboot-installation/SLE-15-SP3-ppc64le/boot/grub2/grub.cfg
# ln /usr/share/tftpboot-installation/SLE-15-SP3-ppc64le/boot/ppc64le/grub2-ieee1275/grub.cfg /usr/share/tftpboot-installation/SLE-15-SP3-ppc64le/boot/grub2/grub.cfg

Then you can edit anyone of them to make similar:

# cat /usr/share/tftpboot-installation/SLE-15-SP3-ppc64le/boot/grub2/grub.cfg
with_gfx=0

gfxmode=auto
locale_dir=$prefix/locale
lang=en_US

set default='local'

insmod gettext

if sleep --interruptible 0 ; then
  timeout=60
fi

insmod http
# in our case http server is the same as tftp server, otherwice it should be root=(http,IP-of-HTTP-server)
set root=(http)
# Once root set, the kernel and initrd files are related to it. Rest are kernel option, then should include full url

menuentry 'Installation' --class opensuse --class gnu-linux --class gnu --class os {
  echo 'Loading kernel ...'
  linux /install/boot/ppc64le/linux install=http://192.168.0.20/install autoyast=http://192.168.0.20/autoyast/sap.xml
  echo 'Loading initial ramdisk ...'
  initrd /install/boot/ppc64le/initrd
}

menuentry 'VNC Installation' --class opensuse --class gnu-linux --class gnu --class os {
  echo 'Loading kernel ...'
  linux /install/boot/ppc64le/linux install=http://192.168.0.20/install vnc=1 vncpassword=P@ssw0rd
  echo 'Loading initial ramdisk ...'
  initrd /install/boot/ppc64le/initrd
}

menuentry 'Rescue System' $arch --class opensuse --class gnu-linux --class gnu {
  echo 'Loading kernel ...'
  linux /install/boot/ppc64le/linux rescue=1
  echo 'Loading initial ramdisk ...'
  initrd /install/boot/ppc64le/initrd
}

menuentry 'Upgrade' $arch --class opensuse --class gnu-linux --class gnu {
  echo 'Loading kernel ...'
  linux /install/boot/ppc64le/linux upgrade=1
  echo 'Loading initial ramdisk ...'
  initrd /install/boot/ppc64le/initrd
}

menuentry 'local' {
  exit
}

submenu 'Other options...' {
 menuentry 'Reboot' {
 reboot
 }

 menuentry 'Exit to Open Firmware' {
 exit
 }
}

Get and fix an autoyast file

The autoyast/sap.xml above should exist. Take one of mine.

You have to fix the RMT server name/IP and its certificate SHA1 SSL fingerprint. To get correct fingerprint, do this on any linux:

# wget http://rmt.2cloud4.com/rmt.crt
# openssl x509 -fingerprint -noout -sha1 -in rmt.crt

Deploy any additional LPAR using PXE

Update the /etc/dhcpd.conf with real MAC address for your new LPAR and set relevant IP address. The restart of DHCP server required to make changes active. When powering on the LPAR using HMC, select network boot and explicitely define your RMT as boot server. Take a look on console to see installation progress.


Updated on Fri May 13 14:11:10 IDT 2022 More documentations here