Booting LibreELEC on Raspberry Pi4 using PXE network boot

The new Raspberry Pi4 has arrived. I ordered it to be my STB, HTPC, what you name it, to run Kodi (xbmc).

This is almost my first experience with Paspberry, so I've started with a common starting point. I've downloaded the NOOBS zip file. My work laptop is based on Linux, so the suggested formatting tools for other popular operating systems did not suit me. My SD card is 32G in size. So I've created the first 4G partition of type e (W95 FAT16 (LBA)), formatted it using mkfs.vfat, and unzipped the NOOBS.zip file into this partition. There is no need for other actions, such as installing a boot sector, activating a partition, etc. It is enough to have bootloader files on the first partition formatted as FAT.

After booting, the NOOBS installer suggested that I'll install (ta damm!) exactly what I need - LibreELEC, the minimum system to boot into Kodi. This is the first time I've met this distribution, and it's just wonderful.

The next step was getting rid of the SD card. It gives good read speed, but is poor in write speed, limited in size and sensitive to the number of rewrite cycles. The previous Raspberry Pi3 project, which built the OpenVPN access point, taught me to use SD in mostly read mode, otherwise you might end up with a corrupted file system. Maybe this particular SD was bad, but the less SD you write, the better.

I have my own NAS server at home, which does most of the background work at home, including the PXE suit: DHCP, TFTP, NFS, HTTP. Why not use it to boot RasPi4 with LE?

Enable RasPi4 boot from network

The official EEPROM bootloader repository claims that network boot still in BETA state. You can follow the last link to update BOOTROM and enable booting over the network. The manual assumes that you are using Debian.

This is a good article describing the new bootloader strategy in the RasPi4 architecture. This is my version and configuration, proven to work:

LibreELEC:~ # rpi-eeprom-update
BOOTLOADER: up-to-date
CURRENT: Mon Sep 23 10:24:46 UTC 2019 (1569234286)
 LATEST: Tue Sep 10 10:41:50 UTC 2019 (1568112110)
VL805: up-to-date
CURRENT: 000137ab
 LATEST: 000137ab
LibreELEC:~ # vcgencmd bootloader_config
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=15000
TFTP_IP=
BOOT_ORDER=0x21
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=5
[none]
FREEZE_VERSION=0

You cannot literally follow the procedure described in the links above to update the bootloader within the LibreELEC OS, because it is a very stripped OS. However, you can use rpi-eeprom-update and rpi-eeprom-config in the same way as described there. Download the desired BOOTROM from the repository, make changes to the config, then install BOOTROM:

LibreELEC:~ # wget https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/beta/pieeprom-2020-01-17.bin
Connecting to github.com (140.82.118.4:443)
Connecting to raw.githubusercontent.com (151.101.64.133:443)
saving to 'pieeprom-2020-01-17.bin'
pieeprom-2020-01-17. 100% |**************************************************************|  512k  0:00:00 ETA
'pieeprom-2020-01-17.bin' saved
LibreELEC:~ # rpi-eeprom-config pieeprom-2020-01-17.bin > bootconf.txt
LibreELEC:~ # cat bootconf.txt 
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
TFTP_IP=
TFTP_PREFIX=0
BOOT_ORDER=0x1
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=5
[none]
FREEZE_VERSION=0

Change the BOOT_ORDER be 0x21. You can also add or change other parameters. Then create a customized BOOTROM and install it:

LibreELEC:~ # rpi-eeprom-config pieeprom-2020-01-17.bin --out my-pieeprom.bin --config bootconf.txt
LibreELEC:~ # ls -l *bin
-rw-r--r--    1 root     root        524288 Jan 27 15:19 my-pieeprom.bin
-rw-r--r--    1 root     root        524288 Jan 27 15:09 pieeprom-2020-01-17.bin
LibreELEC:~ # rpi-eeprom-update -d -f my-pieeprom.bin 
*** INSTALLING my-pieeprom.bin  ***
EEPROM update pending. Please reboot to apply the update.
LibreELEC:~ # reboot

This repository has updated files goes to /boot on any RasPi.

DHCP settings

No article on RasPi network booting describes the configuration of an ISC DHCP server. According to them, it just works if you use dnsmasq.

There are a few things you need to know about RasPi4 network boot. BOOTROM is very small, so DHCP is not fully implemented. For this reason, you can see DHCPREQUEST in the log files on your server, then DHCPOFFER. BOOTROM will continue to use this offer without completing the DHCP handshake. The absence of DHCPREQUEST and DHCPACK do not indicate an error in the boot procedure.

There are two DHCP options required to boot RasPi4, and you can see them analyzing DHCPREQUEST by tcpdump: TFTP Server(option 66) and Vendor Specific(option 43), the latter is used as the boot menu. You can implement them in your dhcpd.conf as:

 ..
# Option 43 used by RaspberyPi net boot
option rpi-pxe code 43 = text ;
option rpi-vendor code 60 = string ;
 ..
host raspi4 {
    hardware ethernet dc:a6:32:6e:03:9f;
    fixed-address 192.168.0.68;
    option rpi-pxe "   \nPXE\t\n   Raspberry Pi Boot   " ;
    option rpi-vendor "PXEClient" ;
    option tftp-server-name "192.168.0.50";
}
 ..

You can boot RasPi4 without setting all these options, just write them hardcoded to BOOTROM by fixing bootconf.txt as described in the chapter above.

TFTP setting

Network boot from BOOTROM will search for its files in the my-serial subdirectory first, located in the TFTP root. In case of failure, it will try find them directly in TFTP root. This allows you to share common files with many clients, while client-specific files can be placed in your own subdirectory. This my-serial can be obtained on a working RasPi4 by command:

LibreELEC:~ # vcgencmd otp_dump | awk -F":" '/28:/{print $2}'
15dcf4f5

Here is a reference for this command.

Thus, create a directory with the name 15dcf4f5 (in my case) at TFTP root (/tftpboot in my case) and fill it with everything that you found in the /flash directory on the working LibreELEC.

It is time for a change ! Locate the file cmdline.txt, that you've copied just now, and change its content like:

# cat cmdline.txt
toram boot=NFS=192.168.0.50:/tftpboot/15dcf4f5 disk=NFS=192.168.0.50:/export/LibreELEC ssh ip=dhcp quiet

You correctly guessed that this is the kernel boot command line, at least part of it. It includes additional parameters that the LibreELEC init script can understand and evaluate. You can check the script and get an idea of these and other options.

NFS settings

When the root file system in LibreELEC mounted in read-only mode, /storage is used to store user data and other custom data. According to cmdline.txt above, its contents should be replicated from working LibreELEC to /export/LibreELEC on my NFS server. After that, allow the client (it has a fixed IP address via DHCP) to mount both these directories for read-write. The client usually mounts /flash read-only, but during the upgrade process, it re-mounts /flash as read-write, so the server must allow that.

..
/export/LibreELEC 192.168.0.68(sec=sys,rw,insecure,fsid=0014,no_root_squash)
/tftpboot/15dcf4f5 192.168.0.68(sec=sys,rw,insecure,fsid=0015,no_root_squash)
..

Updated on Mon Jan 27 23:57:38 IST 2020 More documentations here