Installing NixOS on Hetzner Online (dedicated hosting)

I started running a bunch of stuff on my VPS and it is starting to be a little too full. Since I want the stuff to be accessible even when my home network is down (hello wireless internet), I bought a Hetzner dedicated host. That meant trying to figure out how to get nixos to run on the thing.

Sane people would request KVM and just install the thing interactively. But I am not sane and wanted the ability to reinstall, reimage, etc. everything at any point without too much delay. Which means automation and no request tickets.

Okay, that was enough preable. This time I'll spare you also of my unsuccessful attempts (mostly because I didn't document anything) and go right to what worked.

Networking

I don't know if there is DHCP, but every guide seems to suggest to hardcode the IP addresses and network interfaces. So I did.

The following script, adapted from nixos-install-scripts can be run on the hetzner rescue system and the result can be saved to networking.nix.

I modified it slightly, but the original script worked as well. It's probably a good idea to check if the output matches info on hetzner portal thingy. You can also try installing some supported system and see if the stuff that detects matches the output here.

set -xeu
set -o pipefail

# Find the name of the network interface that connects us to the Internet.
# Inspired by https://unix.stackexchange.com/questions/14961/how-to-find-out-which-interface-am-i-using-for-connecting-to-the-internet/302613#302613
RESCUE_INTERFACE=$(ip -j route get 8.8.8.8 | jq -r '.[].dev')

# Find what its name will be under NixOS, which uses stable interface names.
# See https://major.io/2015/08/21/understanding-systemds-predictable-network-device-names/#comment-545626
# NICs for most Hetzner servers are not onboard, which is why we use
# `ID_NET_NAME_PATH`otherwise it would be `ID_NET_NAME_ONBOARD`.
INTERFACE_DEVICE_PATH=$(udevadm info -e | grep -Po "(?<=^P: )(.*${RESCUE_INTERFACE})")
UDEVADM_PROPERTIES_FOR_INTERFACE=$(udevadm info --query=property "--path=$INTERFACE_DEVICE_PATH")
NIXOS_INTERFACE=$(echo "$UDEVADM_PROPERTIES_FOR_INTERFACE" | grep -o -E 'ID_NET_NAME_PATH=\w+' | cut -d= -f2)
echo "Determined NIXOS_INTERFACE as '$NIXOS_INTERFACE'"

IP_V4=$(ip -j route get 8.8.8.8 | jq -r '.[].prefsrc')
echo "Determined IP_V4 as $IP_V4"

# Determine Internet IPv6 by checking route, and using ::1
# (because Hetzner rescue mode uses ::2 by default).
IP_V6="$(ip -j route get 2001:4860:4860:0:0:0:0:8888 | jq -r '.[].prefsrc' | cut -d: -f1-4)::1"
echo "Determined IP_V6 as $IP_V6"


# From https://stackoverflow.com/questions/1204629/how-do-i-get-the-default-gateway-in-linux-given-the-destination/15973156#15973156
read _ _ DEFAULT_GATEWAY _ < <(ip route list match 0/0); echo "$DEFAULT_GATEWAY"
echo "Determined DEFAULT_GATEWAY as $DEFAULT_GATEWAY"


# Generate networking.nix
cat <<EOF
{
  pkgs,
  config,
  inputs,
  ...
}: {
  networking.useDHCP = false;
  networking.interfaces."$NIXOS_INTERFACE".ipv4.addresses = [
    {
      address = "$IP_V4";
      prefixLength = 24;
    }
  ];
  networking.interfaces."$NIXOS_INTERFACE".ipv6.addresses = [
    {
      address = "$IP_V6";
      prefixLength = 64;
    }
  ];
  networking.defaultGateway = "$DEFAULT_GATEWAY";
  networking.defaultGateway6 = { address = "fe80::1"; interface = "$NIXOS_INTERFACE"; };
  networking.nameservers = ["8.8.8.8"];
}
EOF

Run it by:

ssh root@your-ip bash -s < figure-out-networking.sh

Booting

Newest Hetzner servers have UEFI, but that is somewhat recent addition and I bought an older system. So it was my first time figuring out BIOS¹

The resulting boot.nix look like this:

{
  config,
  lib,
  pkgs,
  modulesPath,
  ...
}: {
  boot.initrd.availableKernelModules = ["ahci" "sd_mod"];
  boot.initrd.kernelModules = ["dm-snapshot"];
  boot.kernelModules = ["kvm-intel"];
  boot.extraModulePackages = [];

  swapDevices = [];

  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
  hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;

  # Use GRUB2 as the boot loader.
  # We don't use systemd-boot because Hetzner uses BIOS legacy boot.
  boot.loader.systemd-boot.enable = false;
  boot.loader.grub = {
    enable = true;
    efiSupport = false;
  };
}

Note the lack of specification of disks here as that will be in separate config.

How did I get to this exact config you ask? I ran parts of the nixos-install-script referenced above. Just enough to get nixos-generate-config working and then removed a bunch of stuff. It took A LOT of trial and error.

Formatting

As the plan is to install using nixos-anywhere I went with disko to specify the partition layout. The following layout works. I ended up with a different one but it was a major milestone to get something booting as I only knew if it worked or not, not why it failed. Is it networking? Is the machine not booting at all? Who knows. I could ask for KVM, but that would be too easy. I wouldn't want it to easy, right? Also it was 2AM and I wasn't thinking straight².

# Example to create a bios compatible gpt partition
{lib, ...}: let
  one = "/dev/disk/by-path/pci-0000:00:17.0-ata-2.0";
  two = "/dev/disk/by-path/pci-0000:00:17.0-ata-3.0";
  content = {
    type = "gpt";
    partitions = {
      boot = {
        name = "boot";
        size = "1M";
        type = "EF02";
      };
      raid1 = {
        size = "1G";
        content = {
          type = "mdraid";
          name = "braid";
        };
      };
      raid2 = {
        size = "100%";
        content = {
          type = "mdraid";
          name = "rraid";
        };
      };
    };
  };
in {
  disko.devices.disk = {
    one = {
      inherit content;
      type = "disk";
      device = one;
    };
    two = {
      inherit content;
      type = "disk";
      device = two;
    };
  };
  disko.devices.mdadm = {
    braid = { # stands for boot raid
      type = "mdadm";
      level = 1;
      content = {
        type = "filesystem";
        format = "ext4";
        mountpoint = "/boot";
      };
    };
    rraid = { # stands for root raid
      type = "mdadm";
      level = 1;
      content = {
        type = "filesystem";
        format = "ext4";
        mountpoint = "/";
        mountOptions = ["defaults"];
      };
    };
  };
}

I came up with this layout by looking at what the hetzner installer created when installing debian.

Some baseline config

Just so that I can connect to the thing, etc. I have this on every server.

{
  pkgs,
  config,
  inputs,
  lib,
  ...
}: let
  authorizedKeys = ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMr5ynyyHtVRtoXOCDmyJv4l6JwBWGgt2b4lo1dWLHoW isabella@isbl.cz"];
in {
  users.users.isabella = {
    isNormalUser = true;
    extraGroups = ["wheel"];
    openssh.authorizedKeys.keys = authorizedKeys;
  };
  environment.variables.EDITOR = "vim";
  users.users.root.openssh.authorizedKeys.keys = authorizedKeys;

  security.sudo.wheelNeedsPassword = false;
  nix.settings.experimental-features = ["nix-command" "flakes"];
  nix.settings.trusted-users = ["isabella"];

  services.openssh = {
    enable = true;
    settings.PasswordAuthentication = false;
    settings.KbdInteractiveAuthentication = false;
    settings.PermitRootLogin = lib.mkForce "no"; # force needed for making live images
  };
}

Putting it together

Now, this part is untested. I plan to wipe the server at some point (to tests backup restore, among other things), but for now, it's just a fiction. I was developing this in bigger repo which has more things going on...

That also means that I can't give you a flake.lock.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    disko = {
      url = "github:nix-community/disko";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs = inputs @ {
    self,
    nixpkgs,
    disko,
    ...
  }: {
    nixosConfigurations.hetzner = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      specialArgs = {inherit inputs;};
      modules = [
        ./base.nix
        ./boot.nix
        ./disks.nix
        ./networking.nix
        disko.nixosModules.disko
        {networking.hostName = "hetzner";}
      ];
    };
  };
}

Run install

One major hurdle was how to run the installer as kexec didn't seem to work for me in the rescue system. What I ended up doing was installing debian first, then running nixos everywhere with the following command:

nix run github:nix-community/nixos-anywhere -- --flake '.#hetzner' root@your-ip

Conclusion

It ended up not being too bad. But I'd definitelly recommend just using the free KVM to debug the install process. It would've helped to know why it's failing.

Next up: atomic backups with btrfs and impermanence (I already have this working, just to write it). And maybe encryption at rest? (this one is more of a pipe dream)

References that didn't fit anywhere else:

• nixos wiki article