Raspberry Pi - Diskless Cluster using Ubuntu 20.04.1

I'm attempting to get Ubuntu 20.04 to boot diskless on a Raspberry PI so I can run Ubuntu on a cluster.

So far, I moved the images from the boot partition on an installed RPi4b with Ubuntu installed, into a dnsmasq/TFTP server under the /srv/tftpboot/<serial>/ which is the default for a Raspberry Pi, Ive updated the bootloader and got the PI to boot.

The dnsmasq log shows the files being loaded at boot:

/var/log/dnsmasq.log

==> /var/log/dnsmasq.log <==
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 available DHCP subnet: 192.168.254.254/255.255.255.0
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 vendor class: PXEClient:Arch:00000:UNDI:002001
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 PXE(eth0) dc:a6:32:b4:48:4f proxy
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 tags: eth0
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 broadcast response
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 sent size: 1 option: 53 message-type 2
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 sent size: 4 option: 54 server-identifier 192.168.254.254
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 sent size: 17 option: 97 client-machine-id 00:52:50:69:34:14:31:d0:00:32:b4:48:4f:f6...
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 sent size: 32 option: 43 vendor-encap 06:01:03:0a:04:00:50:58:45:09:14:00:00:11...
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 available DHCP subnet: 192.168.254.254/255.255.255.0
Sep 6 14:25:03 dnsmasq-dhcp[4799]: 1527100745 vendor class: PXEClient:Arch:00000:UNDI:002001
Sep 6 14:25:03 dnsmasq-tftp[4799]: error 0 Early terminate received from 192.168.254.253
Sep 6 14:25:03 dnsmasq-tftp[4799]: failed sending /srv/tftpboot/6b0bb1f6/start4.elf to 192.168.254.253
Sep 6 14:25:03 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/config.txt to 192.168.254.253
Sep 6 14:25:03 dnsmasq-tftp[4799]: file /srv/tftpboot/6b0bb1f6/pieeprom.sig not found
Sep 6 14:25:03 dnsmasq-tftp[4799]: file /srv/tftpboot/6b0bb1f6/recover4.elf not found
Sep 6 14:25:03 dnsmasq-tftp[4799]: file /srv/tftpboot/6b0bb1f6/recovery.elf not found
Sep 6 14:25:04 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/start4.elf to 192.168.254.253
Sep 6 14:25:05 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/fixup4.dat to 192.168.254.253
Sep 6 14:25:05 dnsmasq-tftp[4799]: file /srv/tftpboot/6b0bb1f6/recovery.elf not found
Sep 6 14:25:05 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/config.txt to 192.168.254.253
Sep 6 14:25:05 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/syscfg.txt to 192.168.254.253
Sep 6 14:25:05 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/usercfg.txt to 192.168.254.253
Sep 6 14:25:05 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/dt-blob.bin to 192.168.254.253
Sep 6 14:25:05 dnsmasq-tftp[4799]: file /srv/tftpboot/6b0bb1f6/recovery.elf not found
Sep 6 14:25:05 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/config.txt to 192.168.254.253
Sep 6 14:25:05 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/syscfg.txt to 192.168.254.253
Sep 6 14:25:05 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/usercfg.txt to 192.168.254.253
Sep 6 14:25:05 dnsmasq-tftp[4799]: file /srv/tftpboot/6b0bb1f6/bootcfg.txt not found
Sep 6 14:25:06 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/bcm2711-rpi-4-b.dtb to 192.168.254.253
Sep 6 14:25:06 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/overlays/overlay_map.dtb to 192.168.254.253
Sep 6 14:25:06 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/config.txt to 192.168.254.253
Sep 6 14:25:06 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/syscfg.txt to 192.168.254.253
Sep 6 14:25:06 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/usercfg.txt to 192.168.254.253
Sep 6 14:25:06 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/cmdline.txt to 192.168.254.253
Sep 6 14:25:08 dnsmasq-tftp[4799]: file /srv/tftpboot/6b0bb1f6/armstub8-gic.bin not found
Sep 6 14:25:08 dnsmasq-tftp[4799]: error 0 Early terminate received from 192.168.254.253
Sep 6 14:25:08 dnsmasq-tftp[4799]: failed sending /srv/tftpboot/6b0bb1f6/uboot_rpi_4.bin to 192.168.254.253
Sep 6 14:25:08 dnsmasq-tftp[4799]: sent /srv/tftpboot/6b0bb1f6/uboot_rpi_4.bin to 192.168.254.253

Typically you would use an NFS server and update your cmdline.txt to contain the locations of the rootfs so once the initrd loads, you can use a root FS stored on the network.

I can't find much info on the options required for Ubuntu so I took them directly from the kernel docs at:

So I updated my cmdline.txt on the TFTP server to:

cmdline.txt

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 nfsrootdebug elevator=deadline rootwait fixrtc init=initrd.img ip=dhcp rootfstype=nfs4 root=/dev/nfs nfsroot=192.168.254.254/srv/nfs/6b0bb1f6 rw

I only got as far as the U-Boot console. The vmlinuz image wasn't being loaded so I decompressed my vmlinuz-5.4.0-1016-raspi to vmlinux on my installed pi running off the sdcard:

zcat /boot/vmlinuz-5.4.0-1016-raspi > vmlinux

and placed this into the TFTP server then updated my config.txt in the [all] section to make use of the decompressed image.

config.txt

[all]
arm_64bit=1
device_tree_address=0x03000000
enable_uart=1
cmdline=cmdline.txt
include syscfg.txt
include usercfg.txt
kernel=vmlinux
initramfs initrd.img followkernel

I have managed to get as far as the initrd loading and the NFS mount scripts are not mounting the NFS share....

nfsmount: need a path
Begin: Retrying nfs mount ...

Extracting the initrd:

mkdir /root/initrd-expanded
cd /root/initrd-expanded
lz4 -cd /boot/initrd.img-5.4.0-1016-raspi | cpio -id

and looking inside the nfs script:

# NFS root mounting
nfs_mount_root()
{ nfs_top # For DHCP modprobe af_packet wait_for_udev 10 # Default delay is around 180s delay=${ROOTDELAY:-180} # loop until nfsmount succeeds nfs_mount_root_impl ret=$? nfs_retry_count=0 while [ ${nfs_retry_count} -lt "${delay}" ] \ && [ $ret -ne 0 ] ; do [ "$quiet" != "y" ] && log_begin_msg "Retrying nfs mount" sleep 1 nfs_mount_root_impl ret=$? nfs_retry_count=$(( nfs_retry_count + 1 )) [ "$quiet" != "y" ] && log_end_msg done
}

Nothing was hitting the NFS server, tcpdump port nfs and also adding debug logging:

/etc/default/nfs-kernel-server

RPCMOUNTDOPTS="--manage-gids --debug all"

It appears that the path being passed to the nfsroot is incorrect...now this is where it get's a little confusing.

The handling for the nfsroot= on a base image install

nfsroot=192.168.254.254/srv/nfs/6b0bb1f6 -> works
nfsroot=192.168.254.254:/srv/nfs/6b0bb1f6 -> fails

If you take the installation from an installed SDCard after doing an apt update:

nfsroot=192.168.254.254/srv/nfs/6b0bb1f6 -> fails
nfsroot=192.168.254.254:/srv/nfs/6b0bb1f6 -> works

That was 5 hours spent pulling my hair out.

You cannot (or at least I couldn't) create a bootable NFS installed system from the images provided by Ubuntu for Raspberry PI. Go figure! If you attempt it you will get all sorts of errors about services not starting up.

So, Ill answer my own question.

2 Answers

Create a fully diskless system on Ubuntu 20.04.1 on a Raspberry PI 4b

1) Install a RaspberryPi Lite onto an SDCard

2) Boot the Rpi4 with the Raspberry OS SDCard, login and run the following to enable ssh:

cd /boot
touch ssh
reboot

3) Update the boot loader, from another machine. If you know the IP address of the PI and you enabled ssh (above), this script will set the boot code to 0xf12 which means it will try the network, sdcard, reboot in that order over and over. You might want to ensure the firmware version if this post gets old. It will also give you an env file which contains the serial and mac address which is handy, stored in a <uuid>.rpi.env file

./update-bootloader.sh <ip-address-of-the-pi> <ip-address-of-your-nfs-server>

e.g.

./update-bootloader.sh 192.168.0.254 192.169.0.254

#!/usr/bin/env bash
# update-bootloader.sh - update the boot loader for Rpi4
RPI_IP=$1
KICKSTART_IP=$2
RPI_DEFAULT_PASS="raspberry"
PI_EEPROM_DATE="2020-07-31"
PI_EEPROM_VERSION="pieeprom-${PI_EEPROM_DATE}"
PI_EEPROM_FILE="${PI_EEPROM_VERSION}.bin"
PI_EEPROM_LINK=""
UBUNTU_IMAGE_NAME="ubuntu-20.04.1-preinstalled-server-arm64+raspi.img"
UBUNTU_IMAGE_FILE="${UBUNTU_IMAGE_NAME}.xz"
UUID=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
ssh-keygen -R ${RPI_IP}
ssh-keyscan -H ${RPI_IP} >> ~/.ssh/known_hosts
sshpass -p "${RPI_DEFAULT_PASS}" ssh pi@${RPI_IP} << EOF
if [[ -f ${PI_EEPROM_FILE} ]];then rm ${PI_EEPROM_FILE} echo 'removed eeprom file'
fi
rm *.rpi.env
echo 'removed old env'
rm bootconf.txt
echo 'removed bootconf.txt'
if [[ ! -f ${PI_EEPROM_FILE} ]];then wget ${PI_EEPROM_LINK}
fi
echo "extracting boot config from eeprom"
sudo rpi-eeprom-config ${PI_EEPROM_FILE} > bootconf.txt
echo "updating bootconfig"
sed -i 's/BOOT_ORDER=.*/BOOT_ORDER=0xf12/g' bootconf.txt
echo "MAX_RESTARTS=5" | sudo tee -a bootconf.txt
echo "writing eeprom"
sudo rpi-eeprom-config --out ${PI_EEPROM_VERSION}-netboot.bin --config bootconf.txt ${PI_EEPROM_FILE}
echo "updating eeprom on rpi"
sudo rpi-eeprom-update -d -f ./${PI_EEPROM_VERSION}-netboot.bin
echo "getting serial and mac"
cat /proc/cpuinfo | grep Serial | awk -F ': ' '{print \$2}' | tail -c 9 | awk '{print "RPI_SERIAL="\$1}' > ${UUID}.rpi.env
ip addr show eth0 | grep ether | awk '{print \$2}' | awk '{print "RPI_MAC="\$1}' >> ${UUID}.rpi.env
EOF
# copy the pi env back to get the serial and mac
sshpass -p "${RPI_DEFAULT_PASS}" scp -r pi@${RPI_IP}:~/${UUID}.rpi.env ~/${UUID}.rpi.env
sshpass -p "${RPI_DEFAULT_PASS}" ssh pi@${RPI_IP} << EOF
sudo reboot
EOF
cat ~/${UUID}.rpi.env

Your pi will now have been rebooted and will be pinging your network for the DHCP response.

4) Install Ubuntu 20.04.1 onto the SD Card using the PI Imager

5) Plug the SDCard into your Pi and boot it up

6) Do an

apt update -y; apt upgrade -y

setup your pass, install some packages, put vim on there (obviously, who uses emacs or nano anyway??)

7) shutdown the pi

halt -p

On your server

8) Take the Ubuntu 20.04.1 installed SDCard, plug it into your server, mount the drive, copy the OS files from partition 2, to your nfs location for sharing the root to the PI, probably /srv/nfs/<serial>/

# find the partition where your CD card is (mine was sda on a rpi)
fdisk -l
# mount the sdcard - copy the second partition off (should contain the boot files, you can ignore the boot partition, we can use the files off the root partition)
mkdir /root/p2
mount /dev/sda2 /root/p2
# cp using using -ax will give you a correct copy, man cp if needed
cp -vax /root/p2/. /srv/nfs/<serial>/.
# clean up
umount /root/p2
rm -rf /root/p2

9) Mount the boot folder of the Rpi files (stored in the NFS share) to your TFTP location so your TFTP can serve up the boot files, I'm using a PI here to serve the other PIs so edit as needed)

#/etc/fstab
LABEL=writable / ext4 defaults 0 0
LABEL=system-boot /boot/firmware vfat defaults 0 1
/srv/nfs/<serial>/boot /srv/tftpboot/<serial> none defaults,bind 0 0

Then mount the new location

mount -a

10) Extract your vmlinuz on the boot folder of your nfs share into vmlinux as the pi wont decompress the vmlinuz kernel

zcat /srv/nfs/<serial>/boot/vmlinuz-5.4.0-1016-raspi > /src/nfs/<serial>/boot/vmlinux-5.4.0-1016-raspi

11) Create symlinks inside the /srv/nfs/<serial>/boot partition to point to the bcm2711-rpi-4-b.dtb, start4.elf, fixup4.dat files which are missing in the boot folder for the TFTP to find them in the dtb and firmware folders

Optional - clean out all the junk that is no longer needed - look at my ls -al output to see whats should be there and what I removed.

lrwxrwxrwx 1 root root 41 Sep 7 08:19 bcm2711-rpi-4-b.dtb -> dtbs/5.4.0-1016-raspi/bcm2711-rpi-4-b.dtb
-rw-r--r-- 1 root root 216 Sep 7 08:23 cmdline.txt
-rw-r--r-- 1 root root 220286 Aug 13 15:09 config-5.4.0-1016-raspi
-rw-r--r-- 1 root root 231 Sep 7 08:45 config.txt
lrwxrwxrwx 1 root root 43 Sep 6 20:34 dtb -> dtbs/5.4.0-1016-raspi/./bcm2711-rpi-4-b.dtb
lrwxrwxrwx 1 root root 43 Sep 6 20:34 dtb-5.4.0-1016-raspi -> dtbs/5.4.0-1016-raspi/./bcm2711-rpi-4-b.dtb
drwxr-xr-x 3 root root 4096 Sep 7 07:27 dtbs
drwxr-xr-x 2 root root 4096 Sep 7 07:56 firmware
lrwxrwxrwx 1 root root 19 Sep 7 08:15 fixup4.dat -> firmware/fixup4.dat
lrwxrwxrwx 1 root root 27 Sep 6 20:32 initrd.img -> initrd.img-5.4.0-1016-raspi
-rw-r--r-- 1 root root 29579888 Sep 6 20:34 initrd.img-5.4.0-1016-raspi
lrwxrwxrwx 1 root root 19 Sep 7 07:26 start4.elf -> firmware/start4.elf
-rw-r--r-- 1 root root 327 Sep 7 08:04 syscfg.txt
-rw-r--r-- 1 root root 4162247 Aug 13 15:09 System.map-5.4.0-1016-raspi
-rw-r--r-- 1 root root 200 Sep 7 08:04 usercfg.txt
-rw-r--r-- 1 root root 25907712 Sep 7 08:13 vmlinux-5.4.0-1016-raspi
lrwxrwxrwx 1 root root 24 Sep 6 20:32 vmlinuz -> vmlinuz-5.4.0-1016-raspi
-rw-r--r-- 1 root root 8420251 Aug 13 15:09 vmlinuz-5.4.0-1016-raspi

12) update some configs in the /srv/nfs/<serial>/boot partition

#/srv/nfs/<serial>/boot/config.txt
[pi4]
max_framebuffers=2
[all]
arm_64bit=1
device_tree_address=0x03000000
enable_uart=1
cmdline=cmdline.txt
include syscfg.txt
include usercfg.txt
kernel=vmlinux-5.4.0-1016-raspi
initramfs initrd.img-5.4.0-1016-raspi followkernel

#/srv/nfs/<serial>/boot/cmdline.txt
net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 nfsrootdebug elevator=deadline rootwait fixrtc init=initrd.img ip=dhcp rootfstype=nfs4 root=/dev/nfs nfsroot=<nfs ip>:/srv/nfs/<serial> rw

13) Update the fstab - this is the fstab which is sent to the pi

#/srv/nfs/<serial>/etc/fstab
proc /proc proc defaults 0 0
<nfs ip>:/srv/nfs/<serial> / nfs4 defaults,rw,nolock 0 0 # data to be shared to server
<nfs ip>:/srv/nfs/<serial>/boot/firmware /boot/firmware nfs4 defaults,rw,nolock 0 1 # data to be shared to server
none /tmp tmpfs defaults 0 0
none /var/run tmpfs defaults 0 0
none /var/lock tmpfs defaults 0 0
none /var/tmp tmpfs defaults 0 0

14) Install an NFS server (google this) to serve the Pi

#/etc/exports
/srv/nfs/<serial> *(insecure,rw,async,no_root_squash)

exportfs -ra

15) Install a dnsmasq server (google is your friend) to serve up the dhcp options and tftp the boot images

#/etc/dnsmasq.conf
dhcp-range=<your network subnet>,proxy # e.g. 192.168.254.254,proxy
log-dhcp
enable-tftp
tftp-root=/srv/tftpboot
pxe-service=0,"Raspberry Pi Boot"
log-facility=/var/log/dnsmasq.log

16) Clusters? You have more than one Pi? You can use an overlayfs mount on your server to provide multiple Pis their operating system using a single base root file system and then using overlays to give each Pi its own space for storage and FS modifications.

If you got this far then this should be easy:

On your proper server - not a pi

17) Create mounts for overlay fs based mounts so we can use the root fs as a lower dir (google overlayfs)

#/etc/fstab
overlay /srv/nfs/6b0bb1f6 overlay defaults,lowerdir=/srv/nfs/ubuntu-rpi4-lower,upperdir=/srv/nfs/6b0bb1f6-upper,workdir=/srv/nfs/6b0bb1f6-work,nfs_export=on,index=on 0 0
overlay /srv/nfs/68e71308 overlay defaults,lowerdir=/srv/nfs/ubuntu-rpi4-lower,upperdir=/srv/nfs/68e71308-upper,workdir=/srv/nfs/68e71308-work,nfs_export=on,index=on 0 0

18) Create the FS system to support the overlays, mine looks like this for 3 pis.

# this is inside /srv/nfs
drwxr-xr-x 1 root root 4096 Sep 7 12:47 68e71308
drwxr-xr-x 3 root root 4096 Sep 7 12:47 68e71308-upper
drwxr-xr-x 3 root root 4096 Sep 7 13:25 68e71308-work
drwxr-xr-x 1 root root 4096 Sep 7 12:13 6b0bb1f6
drwxr-xr-x 2 root root 4096 Sep 7 12:13 6b0bb1f6-upper
drwxr-xr-x 4 root root 4096 Sep 7 13:25 6b0bb1f6-work
drwxr-xr-x 1 root root 4096 Sep 7 12:47 917c9833
drwxr-xr-x 2 root root 4096 Sep 7 11:49 917c9833-upper
drwxr-xr-x 2 root root 4096 Sep 7 11:34 917c9833-work
drwxr-xr-x 21 root root 4096 Sep 6 19:58 ubuntu-rpi4-lower

19) You need to put an /etc/fstab inside the merged folder for the mount (not the upper or work dirs, just the plain serial named one), which will override the ubuntu-rpi4-lower provided ones. Google fusefs or overlayfs for more info, (it's how docker containers work don't ya know :)

20) Create a cmdline.txt inside each merged folder inside /srv/nfs/<serial>/boot/cmdline.txt

21) Export the merged folders over nfs so ours pis can use them as before:

#/etc/exports
/srv/nfs/6b0bb1f6 *(rw,sync,no_subtree_check,no_root_squash,fsid=1)
/srv/nfs/917c9833 *(rw,sync,no_subtree_check,no_root_squash,fsid=2)
/srv/nfs/68e71308 *(rw,sync,no_subtree_check,no_root_squash,fsid=3)

exportfs -ra

22) Adding a new Pi is then just a case:

22.0) Update the boot loader

22.1 Creating three empty folders on the server

mkdir /srv/nfs/<serial>
mkdir /srv/nfs/<serial>-work
mkdir /srv/nfs/<serial>-upper

22.2) Adding an fstab with the mount options for the new serial and upper/work dirs

22.3) Adding an cmdline.txt with the right NFS location

23) Uber automation If you want you can create a hook script inside the initrd of your RaspberryPi SD card which updates the bootloader for you and pings a web server with its serial which then will add the mounts by the time the Pi has rebooted itself, its already booting off the network. Ill provide that at some point.

Thank you for the manual. I would add few things. Newer firmwares seem to be fine with uncompressing the kernel. (tested on 5.8 and higher) For me I had issues with permissions on the vmlinuz file though.

Also, this is messing up snap. I guess there are many people against it anyway, but if you would needed. You can add

network inet,
network inet6,

into /etc/apparmor.d/usr.lib.snapd.snap-confine.real