Currently Viewing Posts Tagged kubernetes

This post was drafted by Claude and reviewed by me. Claude checked the commands, the image links, and the libguestfs, systemd and Proxmox source code behind the machine ID fix; I checked it against my own setup.

Back in 2023 I wrote about building cloud images for Proxmox, and updated it in 2025 for Ubuntu 24.04. It’s time for another round: Ubuntu 26.04 is out, Proxmox 9 changed a few things, and I’ve finally worked out the machine ID problem from the bottom of the old post. This time I’ve added Debian 13 and AlmaLinux 10 too.

What’s changed since the old post

• New images: Ubuntu 26.04 LTS (Resolute Raccoon), Debian 13 (Trixie) and AlmaLinux 10.
• AlmaLinux 10 needs a newer CPU type. Like RHEL 10, it’s built for the x86-64-v3 level (AVX2). On Proxmox’s default CPU types it kernel-panics at boot, so the template needs --cpu x86-64-v3 or --cpu host. If your CPU has no AVX2 (some Pentiums and Celerons), Alma publishes an x86_64_v2 build as well.
• Correction: install dhcpcd-base, not dhcpcd. The old note for Proxmox 9 said dhcpcd. The full package also runs a DHCP client service on the host, which grabs leases on bridges and guest firewall interfaces. One of my nodes ended up with three default routes because of it.
• The machine ID problem is fixed in the image, with no need to boot each VM and patch it. The details are below, including why it happened.
• Updated template commands: an explicit CPU type, import-from instead of qm importdisk, --boot order=scsi0 instead of the deprecated boot options, and DHCP set on the template.

On the Proxmox host

You need libguestfs-tools. On Proxmox 9 (Debian 13 underneath) you also need a DHCP client binary: virt-customize boots a small appliance to install packages, and without one it fails with dhcpcd: command not found and has no network.

apt update
apt install libguestfs-tools dhcpcd-base

If you followed my old note and installed dhcpcd, check ip route. If there’s more than one default route, fix it (with a console handy):

systemctl disable --now dhcpcd
apt remove dhcpcd        # keeps dhcpcd-base
ip route

Stopping it doesn’t remove the addresses and routes it added. Delete them with ip addr del, or reboot.

Pick a distro

Set these once; the rest of the commands are the same for all three.

Your storage name will be different. local-zfs is what my nodes use (the default on a ZFS install). Yours might be local-lvm or a name you picked: pvesm status on the host lists them, as does Datacenter → Storage in the web UI.

Ubuntu 26.04 LTS

VMID=9000
NAME=ubuntu-2604-cloudinit-template
IMG=resolute-server-cloudimg-amd64.img
wget https://cloud-images.ubuntu.com/resolute/current/$IMG

Debian 13

VMID=9001
NAME=debian-13-cloudinit-template
IMG=debian-13-genericcloud-amd64.qcow2
wget https://cloud.debian.org/images/cloud/trixie/latest/$IMG

AlmaLinux 10

VMID=9002
NAME=almalinux-10-cloudinit-template
IMG=AlmaLinux-10-GenericCloud-latest.x86_64.qcow2
wget https://repo.almalinux.org/almalinux/10/cloud/x86_64/images/$IMG

For the v2 build, replace x86_64 with x86_64_v2 in both the folder and the file name.

All three:

STORAGE=local-zfs                  # change to your storage
PROXY=http://10.244.71.183:3142    # change to your proxy's IP and port

Customise the image

I point apt at my apt-cacher-ng proxy and install the QEMU guest agent. If you run a proxy, set PROXY to your own proxy’s IP and port: 10.244.71.183 is mine and won’t exist on your network. If you don’t run one, drop the --append-line.

Ubuntu and Debian:

virt-customize -a $IMG \
  --install qemu-guest-agent \
  --append-line "/etc/apt/apt.conf.d/00proxy:Acquire::http { Proxy \"$PROXY\"; };"

AlmaLinux:

virt-customize -a $IMG \
  --install qemu-guest-agent \
  --selinux-relabel

--selinux-relabel matters on Alma: without it, files changed by virt-customize have no SELinux labels and services can fail to start. I skip the proxy here: Alma’s mirrors are HTTPS, and apt-cacher-ng only caches plain HTTP.

Sysprep and reset the machine ID

virt-sysprep -a $IMG
virt-customize -a $IMG \
  --truncate /etc/machine-id \
  --run-command 'rm -f /var/lib/dbus/machine-id && ln -s /etc/machine-id /var/lib/dbus/machine-id'
virt-cat -a $IMG /etc/machine-id | wc -c    # should print 0

Why clones were sharing an IP. systemd-networkd builds its DHCP client ID from /etc/machine-id. Two VMs with the same machine ID look like the same client to the DHCP server, so they get the same address, even with different MAC addresses.

When systemd boots and finds /etc/machine-id empty, it picks a new ID in this order:

1. /var/lib/dbus/machine-id, if it’s a regular file with an ID in it.
2. On KVM, the VM’s SMBIOS UUID.
3. A random ID.

Proxmox gives every clone a new SMBIOS UUID, so a template with an empty machine ID produces a unique ID in every clone, on first boot.

virt-sysprep already empties both files. So if clones still share an ID, one of two things happened:

• The template was booted before qm template. One boot is enough to write an ID, and every clone inherits it.
• Something ran after virt-sysprep. Package installs can create a machine ID, so do sysprep and the reset last.

The virt-customize line above is my old manual fix, applied to the image instead of each VM. If you did boot the template, run cloud-init clean --logs --machine-id before shutting it down.

Create the template

qm create $VMID --name $NAME --ostype l26 --memory 4096 --cores 2 \
  --cpu x86-64-v2-AES --net0 virtio,bridge=vmbr0,tag=72
qm set $VMID --scsihw virtio-scsi-single \
  --scsi0 $STORAGE:0,import-from=$PWD/$IMG,discard=on,ssd=1,iothread=1
qm set $VMID --boot order=scsi0
qm disk resize $VMID scsi0 +50G
qm set $VMID --ide2 $STORAGE:cloudinit
qm set $VMID --serial0 socket --vga serial0
qm set $VMID --agent enabled=1
qm set $VMID --ipconfig0 ip=dhcp
qm template $VMID

For AlmaLinux 10, use --cpu x86-64-v3 (or host).

• --cpu: without it, qm create uses kvm64, a Pentium 4-level CPU. x86-64-v2-AES works on any reasonably modern host and keeps live migration working. host is fastest, but migrating to a node with a different CPU can fail.
• import-from: replaces qm importdisk plus qm set --scsi0. It needs an absolute path, hence $PWD.
• virtio-scsi-single with iothread=1: gives each disk its own I/O thread. discard=on,ssd=1 passes TRIM through to thin storage.
• VLAN 72 is my Kubernetes VLAN; remove ,tag=72 if you don’t need it.
• Disk size: the cloud images are only 2-3 GB, so add space before templating.

Clone it

qm clone 9000 2001 --name k8s-01 --full
qm set 2001 --sshkey ~/godboxv3.pub --ciuser tiernano --memory 4096
qm start 2001

Change the IDs (9000 is the template, 2001 the new VM), the VM name (k8s-01), the username (tiernano), the SSH public key path (~/godboxv3.pub) and the memory to your own.

For a fixed IP, add --ipconfig0 to the qm set line, not to qm clone: qm clone only makes the copy, and the cloud-init settings (--ipconfig0, --sshkey, --ciuser) all belong to qm set. Set it before the first qm start. It replaces the template’s ip=dhcp for that VM only:

qm set 2001 --sshkey ~/godboxv3.pub --ciuser tiernano --memory 4096 \
  --ipconfig0 ip=10.244.72.21/24,gw=10.244.72.1

Use an address, prefix and gateway from your own network.

--full makes an independent copy. Without it you get a linked clone, which is quicker and smaller, but the template can’t be deleted while linked clones exist.

Check the machine ID on the clones

cat /etc/machine-id
journalctl -b | grep -i 'machine id'

Every clone should show a different ID, and the log should say Initializing machine ID from SMBIOS/DMI UUID. If two match, the template has an ID baked in: rebuild it from a fresh image.