How to fix flaky bootloader configuration that prevents rebuilds on Hetzner servers

Troubleshooting
1

This post is useful for users encountering rebuild errors like this:

updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/sda...
Installing for i386-pc platform.
/nix/store/vx3l94jmflz9nfmqngv6gp15hj7z4l0n-grub-2.12/sbin/grub-install: warning: File system `ext2' doesn't support embedding.
/nix/store/vx3l94jmflz9nfmqngv6gp15hj7z4l0n-grub-2.12/sbin/grub-install: error: embedding is not possible, but this is required for cross-disk install.
/nix/store/n6af8sp59mw7izcq8nb2svnc8s21h12g-install-grub.pl: installation of GRUB on /dev/sda failed: Inappropriate ioctl for device
Failed to install bootloader

The root cause is that Linux device names are not stable: /dev/sda and /dev/sdb can refer to different disks after a reboot. In this case, the data volume may become /dev/sda. Because that disk is fully occupied by an ext2/ext4 filesystem, GRUB cannot be installed there - which is correct, because the bootloader belongs on the system disk, now /dev/sdb.

To fix this, you need to:

  1. Read output of ls -la /dev/disk/by-id to determine id of a real system disk. On hetzner, it’s usually scsi-0QEMU_QEMU_HARDDISK_<some numbers>. You will also see ID with -part1, -part14 and -part15 suffixes in the output. You should ignore those entries. Example output:
[root@selfprivacyserver:/etc/nixos]# ls -la /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 160 Jun 26 16:30 .
drwxr-xr-x 8 root root 160 Jun 26 16:30 ..
lrwxrwxrwx 1 root root   9 Jun 26 16:30 ata-QEMU_DVD-ROM_QM00001 -> ../../sr0
lrwxrwxrwx 1 root root   9 Jun 26 16:30 scsi-0HC_Volume_102459416 -> ../../sdb
lrwxrwxrwx 1 root root   9 Jun 26 16:30 scsi-0QEMU_QEMU_HARDDISK_61238073 -> ../../sda
lrwxrwxrwx 1 root root  10 Jun 26 16:30 scsi-0QEMU_QEMU_HARDDISK_61238073-part1 -> ../../sda1
lrwxrwxrwx 1 root root  11 Jun 26 16:30 scsi-0QEMU_QEMU_HARDDISK_61238073-part14 -> ../../sda14
lrwxrwxrwx 1 root root  11 Jun 26 16:30 scsi-0QEMU_QEMU_HARDDISK_61238073-part15 -> ../../sda15
  1. Write disk id (not partition! ignore -part* entries) to bootloader configuration in /etc/nixos/userdata.json. You can either use nano text editor to modify userdata.json by hand to contain:
{
    "dns": ...,
    "server": {
        "provider": "HETZNER",
        ...
        "bootloader": {
            "type": "grub-mbr",
            "device": "/dev/disk/by-id/id-of-system-disk-you-got-in-first-step"
        }
    },
}

or you can use following one-liner:

d=/dev/disk/by-id/id-of-system-disk-you-got-in-first-step; t=$(mktemp); jq --arg d "$d" '.server.bootloader = {type: "grub-mbr", device: $d}' /etc/nixos/userdata.json > "$t" && mv "$t" /etc/nixos/userdata.json

After modifying userdata.json, click Upgrade server in the SelfPrivacy app. The rebuild should then succeed.