Using PXE boot to install Ubuntu over the network

The goal here is to automate the installation of Ubuntu over the network, rather than manually mounting an ISO and entering settings interactively. (And, for me personally, to replicate what we have set up at work, to learn more about it.)

How it works¶

1. The target machine (whether physical or VM) is set up, PXE ("pixie") boot is enabled, and it is started.
2. UEFI looks for a DHCP server on the local network. The DHCP server assigns the machine an IP address, and tells it where to download the EFI executable for GRUB.
3. UEFI downloads GRUB using the TFTP protocol. and then executes it.
4. GRUB downloads a menu file (and a font) using TFTP, and presents the menu to the user.
5. The user selects an entry such as "Install Ubuntu 22.04".
6. GRUB downloads the Linux kernel (vmlinuz) and live boot image (initrd) from the TFTP server, and then runs it. It passes it the URLs of an ISO image of Ubuntu Server installer and an autoinstall (YAML) file.

I'm not 100% sure about the live boot image part... I think it is called Casper. Or maybe that is just one part of it. ↩

7. The live boot image downloads the full ISO file over HTTP and runs it, which starts the Subiquity installer.¹¹
8. Subiquity downloads the autoinstall file, then installs Ubuntu using the configuration provided.
9. At the end of the install, the system reboots.
10. At the next boot, either:
    1. The hard disk has higher priority, so Ubuntu is booted directly; or
    2. PXE boot happens as above, but "Boot from Local Disk" is selected from the menu, and Ubuntu is booted by GRUB.

Graphical version¶

sequenceDiagram
    actor User
    box Target Machine
	    participant UEFI as UEFI
    	participant GRUB as GRUB
	    participant Live as Live Boot
	    participant Subiquity as Subiquity<br>Installer
	    participant Disk as Ubuntu<br>(Hard Disk)
    end
    box Boot Server
        participant DHCP as DHCP
        participant TFTP as TFTP
        participant HTTP as HTTP
    end
    User       ->> UEFI      : Boot
    activate UEFI
    DHCP      -->> UEFI      : Get IP, server + filename
    TFTP      -->> UEFI      : Get GRUB EFI executable
    UEFI       ->> GRUB      : Run GRUB
    deactivate UEFI
    activate GRUB
    TFTP      -->> GRUB      : Get menu file + font
    User      -->> GRUB      : Select menu item
    TFTP      -->> GRUB      : Get Linux kernel + live boot image
    GRUB       ->> Live      : Run live boot
    deactivate GRUB
    activate Live
    HTTP      -->> Live      : Get Ubuntu installer ISO
    Live       ->> Subiquity : Run installer
    deactivate Live
    activate Subiquity
    HTTP      -->> Subiquity : Get config file
    Subiquity -->> Disk      : Install Ubuntu
    Subiquity  ->> UEFI      : Reboot
    deactivate Subiquity
    activate UEFI
    UEFI       ->> Disk      : Boot Ubuntu
    deactivate UEFI
    activate Disk
    deactivate Disk

Limitations¶

We will only support UEFI systems, rather than older BIOS systems.

We will only support Ubuntu versions that use Subiquity (20.04 and above). Older versions were based on Debian-Installer and used preseed files instead of YAML files.

The steps are very similar though, and all combinations could be supported simultaneously if required.

Test network¶

If you have an existing DHCP server (which is likely), and don't want to replace it or disrupt existing devices, I recommend setting up a separate network for testing. You will also need internet access, so you will probably need two network interfaces per VM - one for local communication between test VMs (used for DHCP, TFTP, HTTP), and one for access to the internet and the rest of the local network.

Creating a second network interface in Hyper-V¶

If you are using Hyper-V, as I am:

• Open Hyper-V Manager
• Go to Action > Virtual Switch Manager
• In "New virtual network switch", select "Internal" and click "Create Virtual Switch"
• Set the name to Hyper-V Internal Network (or another name of your choice)
• Click OK

Now assign a static IP for the host machine so it is able to communicate with the VMs using that interface:

• Go to Start > View network connections
• Right-click "vEthernet (Hyper-V Internal Network)" > Properties
• Select "Internet Protocol Version 4" and click Properties
• Select "Use the following IP address"
• Enter the IP 192.168.5.1 (or another IP of your choice)
• Click OK, then Close

Then for each VM you create:

• During setup:
  • Select "Hyper-V Internal Network"
• After setup:
  • Right-click the machine, then click Settings
  • In "Add Hardware", select "Network Adapter" and click "Add"
  • Set Virtual Switch to "Default Switch"
  • Click OK

That will give each VM access to both a private local network - which we will use for DHCP / PXE boot - and NAT to access the internet. The reason for selecting the private network first is we want the DHCP server to be on the first interface - otherwise UEFI waits for the first interface to time out before moving on to the second one.

The steps for other hypervisors (e.g. VirtualBox) should be fairly similar.

Boot server setup¶

We will need at least one VM to act as our DHCP/TFTP/HTTP server - although we could split the services across multiple VMs (or physical servers) if needed (e.g. in a large network).

(Virtual) hardware requirements¶

The boot server has very modest requirements. I used:

• 2 vCPUs
• 2 GB RAM
• 20 GB disk

But you could probably get away with less if needed.

You will also need a test machine. It will need at least 3 GB of RAM to hold the installer image - otherwise it will probably fail to run.²² The same is true for all machines you intend to install Ubuntu on in the future. Other operating systems may have different requirements.

Secure boot¶

If you are using Hyper-V, you will need to configure Secure Boot. After creating the VM:

• Right-click the machine, then click Settings
• Go to the "Security" section
• Change "Template" from "Microsoft Windows" to "Microsoft UEFI Certificate Authority"
• Click OK

The same applies for all target machines you create later. (Alternatively, you can disable Secure Boot completely.)

Operating system¶

I'm using Ubuntu 22.04 for the server, as well as for the target machines - though they don't have to match, and a single server could host installers for a variety of operating systems.

I won't describe how to install Ubuntu - if you're at the point where you want to automate it, I assume you know how to do it manually already.

You will need to assign a static IP for the private network interface, since there is no DHCP server yet. I used 192.168.5.100, with a subnet of 192.168.5.0/24. Leave "Gateway" and "Name servers" blank, because we aren't using this interface for internet traffic or DNS.

Firewall rules¶

This won't apply if you set up a separate test network, as described above - but if you have a firewall set up, I think you will need to allow traffic from the target servers to the boot server on these ports:³³

• UDP 67 for DHCP
• UDP 69 for TFTP
• TCP 80 for HTTP (and maybe TCP 443 for HTTPS, if you decide to use it)

And from the boot server to the target servers:

• UDP 68 for DHCP replies

Directory structure¶

Just for reference, this is the directory structure we will be creating:

/etc
└── dhcpd
    └── dhcpd.conf

/srv/tftp
├── efi
│   ├── grubx64.efi
│   └── shimx64.efi
├── grub
│   ├── fonts
│   │   └── unicode.pf2
│   └── grub.cfg
└── ubuntu-22.04
    ├── initrd
    └── vmlinuz

/var/www
└── ubuntu-22.04
    ├── standard-configuration.yaml
    └── ubuntu-22.04.2-live-server-amd64.iso

You can move most of these files around to suit your own preferences, as long as you update the relevant config files (e.g. /etc/default/tftpd-hpa, /etc/apache2/sites-available/000-default.conf, and the ones listed below).

However, there are some (e.g. dhcpd.conf, grubx64.efi, grub.cfg) that can't be moved/renamed.

DHCP server¶

I will be using ISC DHCP Server, to match what we use at work. However, it is no longer maintained (EOL), as of Oct 2022, so is probably not the best choice⁴⁴ for a new setup!

Install it:

sudo apt install isc-dhcp-server

Then edit the configuration file:

sudoedit /etc/dhcp/dhcpd.conf

Replace it with something like this (the key parts being option architecture-type code ... and the last few lines):

# The name of the DHCP server (not sure if it's used for anything)
server-name "boot.djm.me";

# The DNS servers to use (required for internet access) - I chose to use Cloudflare
option domain-name-servers 1.1.1.1, 1.0.0.1;

# The domain name to use when resolving hostnames via DNS (optional)
option domain-name "djm.me";

# The lease times in seconds (these are the defaults set by the Ubuntu package)
default-lease-time 600; # 10 minutes
max-lease-time 7200; # 2 hours

# This DNS server is authoritative - i.e. it will send DHCPNAK responses to
# clients trying to renew IPs not assigned to them, rather than ignoring them
authoritative;

# Register the "architecture-type" option, which the DHCP server doesn't know out of the box
option architecture-type code 93 = unsigned integer 16;

# Configure the DHCP pool for this subnet
# Note: It will only listen on interfaces that match - so the NAT interface will be ignored
subnet 192.168.5.0 netmask 255.255.255.0 {

    # Define a smallish range, since (1) we won't be setting up many servers, and
    # (2) we will be assigning fixed IP addresses to each VM once they're set up
    range 192.168.5.200 192.168.5.249;

    # If the server uses UEFI, send the EFI image location to boot from
    # (To support BIOS, we would add further options here below)
    if option architecture-type = 00:07 {
        # Note: You can use a hostname here if you prefer - it will be looked up on the
        # DHCP server and sent as an IP address, so make sure it resolves to the correct
        # IP, and not to 127.0.0.1 or 127.0.1.1! Best to start with an IP address...
        next-server 192.168.5.100;
        filename "efi/shimx64.efi";
    }
}

Then restart the DHCP server to apply the changes, and check that it is working:

sudo systemctl restart isc-dhcp-server
sudo systemctl status isc-dhcp-server

It should output something like:

Mar 26 12:14:00 boot dhcpd[1442]: Listening on LPF/eth0/00:15:5d:ea:01:10/192.168.5.0/24
Mar 26 12:14:00 boot dhcpd[1442]: Sending on   LPF/eth0/00:15:5d:ea:01:10/192.168.5.0/24
Mar 26 12:14:00 boot dhcpd[1442]: Server starting service.

Test DHCP¶

At this stage, you could try booting a test machine to check that it is correctly allocated an IP address. It won't actually be able to boot yet, because there is no server to connect to, and will give an error such as:

NBP filename is efi/shimx64.efi
NBP filename is 0 Byres
PXE-E99: Unexpected network error.

You will also be able to see it in the DHCP server logs by running:

journalctl --lines 100 --follow --unit isc-dhcp-server

It should output something like:⁵⁵

Mar 26 12:15:38 boot dhcpd[1442]: DHCPDISCOVER from 00:15:5d:ea:01:0e via eth0
Mar 26 12:15:39 boot dhcpd[1442]: DHCPOFFER on 192.168.5.200 to 00:15:5d:ea:01:0e via eth0
Mar 26 12:15:42 boot dhcpd[1442]: DHCPREQUEST for 192.168.5.200 (192.168.5.100) from 00:15:5d:ea:01:0e via eth0
Mar 26 12:15:42 boot dhcpd[1442]: DHCPACK on 192.168.5.200 to 00:15:5d:ea:01:0e via eth0

TFTP and HTTP servers¶

We will need both TFTP and HTTP (web) servers to serve files. These are quite simple to set up:⁶⁶

sudo apt install tftpd-hpa apache2

Test TFTP¶

Now you can reboot the test machine and check that it is connecting to the TFTP server. Of course, there is nothing for it to download yet, but the error message should be different - e.g.

NBP filename is efi/shimx64.efi
NBP filename is 0 Byres
PXE-E23: Client received TFTP error from server.

If you would like to, you can also enable verbose logging on the TFTP server. First, edit the config file:

sudoedit /etc/default/tftpd-hpa

And change this line:

TFTP_OPTIONS="--secure"

To:

TFTP_OPTIONS="--secure -vvv"

Then restart the server:

sudo systemctl restart tftpd-hpa

And watch the logs next time you reboot the test machine:

journalctl --lines 100 --follow --unit tftpd-hpa

It should output something like:⁷⁷

Mar 26 12:30:31 boot in.tftpd[2827]: RRQ from 192.168.5.200 filename efi/shimx64.efi
Mar 26 12:30:31 boot in.tftpd[2827]: sending NAK (1, File not found) to 192.168.5.200

Remember to turn verbose logging off afterwards.

GRUB binaries¶

DHCP passes control to GRUB - the GRand Unified Bootloader. We can copy⁸⁸ that from our local Ubuntu install:⁹⁹

sudo mkdir /srv/tftp/efi
sudo cp /usr/lib/shim/shimx64.efi.signed /srv/tftp/efi/shimx64.efi
sudo cp /usr/lib/grub/x86_64-efi-signed/grubnetx64.efi.signed /srv/tftp/efi/grubx64.efi

Note that there are two files required:

• shimx64.efi is signed by Microsoft, and contains Canonical's public key
• grubx64.efi is signed by Canonical, and contains GRUB itself

That allows new versions of GRUB to be released without Microsoft needing to sign every single release.

If you disable Secure Boot, you could use grubx64.efi directly - even the unsigned version. If you run into problems, it may be worth doing that temporarily, to rule out signing/verification issues.

Also note that we have to copy grubnetx64.efi, not grubx64.efi. I assume it contains additional code required for netbooting. We have to rename it to grubx64.efi, because that is what shimx64.efi looks for.

Finally, many tutorials recommend getting grubnetx64.efi.signed directly from the Ubuntu archive - but that doesn't contain a copy of shimx64.efi, and it doesn't seem to work with the local version. It's probably best to get both files from the same source. If you disable Secure Boot, however, the version in the archive works fine.

Test GRUB¶

At this point, you should be able to reboot the test machine and get to a GRUB prompt:¹⁰¹⁰

Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists possible
device or file completions.

grub>

Not very user-friendly yet, but we're getting there!

GRUB menu¶

Now let's create a menu file.

sudo mkdir /srv/tftp/grub
sudoedit /srv/tftp/grub/grub.cfg

Enter the following:

set timeout=10

loadfont unicode

set menu_color_normal=white/black
set menu_color_highlight=black/light-gray

menuentry "Boot from Local Disk" {
    insmod chain
    search --set=root --file /EFI/ubuntu/grubx64.efi
    chainloader /EFI/ubuntu/grubx64.efi
}

menuentry "Install Ubuntu 22.04 (Jammy) - Standard Configuration" {
    linux /ubuntu-22.04/vmlinuz root=/dev/ram0 ip=dhcp url=http://${pxe_default_server}/ubuntu-22.04/ubuntu-22.04.2-live-server-amd64.iso autoinstall cloud-config-url=http://${pxe_default_server}/ubuntu-22.04/standard-configuration.yaml
    initrd /ubuntu-22.04/initrd
}

menuentry "Install Ubuntu 22.04 (Jammy) - Manual Installation" {
    linux /ubuntu-22.04/vmlinuz root=/dev/ram0 ip=dhcp url=http://${pxe_default_server}/ubuntu-22.04/ubuntu-22.04.2-live-server-amd64.iso autoinstall
    initrd /ubuntu-22.04/initrd
}

This configures three menu entries:

• "Boot from Local Disk" assumes that Ubuntu is already installed on the local hard disk, and boots it. This won't work yet, but it will be required after it is installed, in case PXE boot is higher priority than local booting. This is the default, auto-selected after 10 seconds, so that the machine can be booted without manual intervention.
• "Install Ubuntu ... Standard Configuration" will install Ubuntu without any intervention, using the standard configuration that we're going to define later.
• "Install Ubuntu ... Manual Installation" will start the Ubuntu installer, then prompt the user for the rest of the configuration (the same as installing from a regular ISO or disc).

We also need to copy the font referenced above:

sudo mkdir /srv/tftp/grub/fonts
sudo cp /usr/share/grub/unicode.pf2 /srv/tftp/grub/fonts/unicode.pf2

Test GRUB menu¶

At this point, you should be able to reboot the test machine, get to the GRUB menu, and select an entry.

But none of the entries will actually work yet, because we don't have anything to run...

Download Ubuntu live server ISO¶

We're going to skip a step now - the reason will become clear in a moment - and download the Ubuntu Server ISO into the web server root:

sudo mkdir /var/www/html/ubuntu-22.04
sudo wget https://releases.ubuntu.com/22.04.2/ubuntu-22.04.2-live-server-amd64.iso -O /var/www/html/ubuntu-22.04/ubuntu-22.04.2-live-server-amd64.iso

The file is 1.8 GB, so it will take some time to run.

Test HTTP¶

There's no point rebooting the test machine at this point, because it still doesn't know how to download that file.

But you can check the HTTP server is working by running this on the boot server:

curl -I http://192.168.5.100/ubuntu-22.04/ubuntu-22.04.2-live-server-amd64.iso

It should output something like:

HTTP/1.1 200 OK
Date: Sun, 26 Mar 2023 12:51:50 GMT
Server: Apache/2.4.52 (Ubuntu)
Last-Modified: Fri, 17 Feb 2023 21:57:18 GMT
ETag: "75c6f000-5f4ec65a00b80"
Accept-Ranges: bytes
Content-Length: 1975971840
Content-Type: application/x-iso9660-image

Live boot image¶

Next, we need to extract a couple of files from the ISO. They are:

• vmlinuz - The Linux kernel (z indicates it is compressed)
• initrd - The Initial RAM Disk that bootstraps the system

Together, they will allow the system to actually boot.

# Mount the ISO
sudo mkdir /mnt/iso
sudo mount /var/www/html/ubuntu-22.04/ubuntu-22.04.2-live-server-amd64.iso /mnt/iso

# Copy the files from it
sudo mkdir /srv/tftp/ubuntu-22.04
sudo cp /mnt/iso/casper/vmlinuz /srv/tftp/ubuntu-22.04/
sudo cp /mnt/iso/casper/initrd /srv/tftp/ubuntu-22.04/

# Unmount the ISO
sudo umount /mnt/iso
sudo rmdir /mnt/iso

Test manual installation¶

At this point, the "Manual Installation" option should be fully working. You can use it to install Ubuntu on the test machine, if you want, and then you can use "Boot from Local Disk" to boot it.

If that's all you wanted it to do, you can even stop here. (You would probably want to edit the GRUB menu and remove the "Standard Configuration" option though.)

If you try to run the "Standard Configuration" option though, it will attempt to download standard-configuration.yaml, fail, and then take you to the regular manual installer.

Automated installation¶

Finally, we will fully automate the Ubuntu installation by providing an autoinstall file containing the necessary configuration.

sudoedit /var/www/html/ubuntu-22.04/standard-configuration.yaml

This is what my file looks like:

#cloud-config
autoinstall:
  version: 1

  identity:
    hostname: test.djm.me
    realname: Dave James Miller
    username: dave
    password: $6$BDX1Rs8nTlXLeR6n$Yf6JWszbfc5lfXEX9SUEolaH.MnIAQOJgBLPVaIc5MsWGN7/HqAcCFQC/oz7SKWNxvhmgj1Xnh9mDcYAQ8usY0 # test

  keyboard:
    layout: gb

  locale: en_GB.UTF-8

  network:
    ethernets:
      eth0:
        addresses:
          - 192.168.5.101/24
        nameservers:
          addresses: []
          search: []
      eth1:
        dhcp4: true
    version: 2

  refresh-installer:
    update: true

  source:
    id: ubuntu-server
    search_drivers: false

  ssh:
    authorized-keys:
      - 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGRzRUvx7ESGxsXrmNKgVY+pg8uc1ZHxzqVoMVznlzWF d@djm.me'
    install-server: true

  storage:
    config:

      # disk-0
      - id: disk-0
        type: disk
        ptable: gpt
        wipe: superblock

      # disk-0 > partition-0 = /boot/efi
      - id: partition-0
        type: partition
        device: disk-0
        size: 1G
        wipe: superblock
        flag: boot
        grub_device: true

      - id: format-0
        type: format
        volume: partition-0
        fstype: fat32

      - id: mount-0
        type: mount
        device: format-0
        path: /boot/efi

      # disk-0 > partition-1 = /boot
      - id: partition-1
        type: partition
        device: disk-0
        size: 2G
        wipe: superblock

      - id: format-1
        type: format
        volume: partition-1
        fstype: ext4

      - id: mount-1
        type: mount
        device: format-1
        path: /boot

      # disk-0 > partition-2
      - id: partition-2
        type: partition
        device: disk-0
        size: -1
        wipe: superblock

      # disk-0 > partition-2 > ubuntu-vg
      - id: lvm_volgroup-0
        type: lvm_volgroup
        name: ubuntu-vg
        devices:
          - partition-2

      # disk-0 > partition-2 > ubuntu-vg > ubuntu-lv = /
      - id: lvm_partition-0
        type: lvm_partition
        volgroup: lvm_volgroup-0
        name: ubuntu-lv
        wipe: superblock

      - id: format-3
        type: format
        volume: lvm_partition-0
        fstype: ext4

      - id: mount-3
        type: mount
        device: format-3
        path: /

  timezone: Europe/London

  updates: all

It looks a little daunting, but you don't need to write it all from scratch. Instead, set up a server manually, then look in /var/log/installer/autoinstall-user-data to see the configuration that was generated:

cat /var/log/installer/autoinstall-user-data

You can use that as a starting point, and tweak it manually as necessary.

Passwords¶

To generate the password hash under identity:

sudo apt install whois
mkpasswd -m sha512crypt
# enter the password and press enter

Network¶

You may notice that I have specified a static IP address for the internal network, rather than using DHCP. That is so I can use that IP in my DNS records, and it will keep working even if the DHCP server is not available. This works fine, as long as the IP address for each server is unique (see below), and they do not overlap with the range of IPs that the DHCP server allocates.

If you are setting up a permanent DHCP server, you will probably want to register the fixed IP addresses in /etc/dhcp/dhcpd.conf instead:

host test {
    hardware ethernet 00:15:5d:ea:01:0e;
    fixed-address 192.168.5.101;
}

You can find the MAC address in Hyper-V > Networking tab > Hyper-V Internal Network. Alternatively, run ip addr on the server itself (under eth0, since it's the first network adapter). Again, the IP should be within the subnet range, but outside the dynamically allocated range.

Storage¶

The most complex part of the YAML file is storage.

Be aware that the autoinstall-user-data file will contain exact disk sizes and serial numbers - so if you copy it directly, it won't work on a different machine. If you look at my example above, you will see that I have adapted it to use all remaining space (-1) for the last partition, rather than a fixed size. I also converted the other sizes from bytes to gigabytes (and rounded them up), removed the hard-coded serial numbers and paths, and reorganised it to be a little easier to understand. The Curtin documentation (referenced in the Autoinstall documentation) was useful for that.

If you're not sure what to do here, the code above should be sufficient to get started - it allocates 1 GB for the EFI partition, 2 GB for the Ubuntu boot partition (because GRUB doesn't support LVM), and the rest for the main system partition using LVM.

Possible future improvements¶

These are things I might consider doing in the future...

Making the configuration dynamic¶

At the moment, all values - including the hostname and IP - are hard-coded. You will, therefore, need to edit this file before setting up each server.

One way to avoid that is by adding this section, so that the installer prompts for them interactively:

  interactive-sections:
    - identity
    - network

However, I found:

• The interactive installer doesn't allow fully qualified hostnames, which I prefer to use, and
• It doesn't seem to pre-fill the static IP address correctly - though I'm not sure why.

Alternatively, you could make a script (PHP, etc.) that looks up the MAC address in a database to get the hostname and IP, and feed them into the YAML file... Or perhaps get the DHCP server to pass them through, again based on the MAC address...

Other ideas¶

• Find a better GRUB theme, or just change the background image
• Add more operating systems (e.g. other versions of Ubuntu, Debian, Kali Linux, a web kiosk...)
• Add a caching proxy server to speed up repeated downloads
• Add Puppet to automatically configure the new servers