Adventures in building disk images of NixOS virtual machines for M1

I work in a thoroughly Nix-based environment, and sometimes I need to test NixOS things, but I have a Mac at work. So, time to get a NixOS VM going on the M1, I guess! This was a partially-planned adventure.

People I know have been saying good things about the (mostly) qemu wrapper UTM (also on GitHub here), so that was the tool of choice.

I kind of didn't realize there were normal installer images that would absolutely just work (since UTM implements UEFI by default), so I tried a more fun option: just build the root filesystem offline and import it.

To do this, you will need a Linux machine with Nix and the capacity to execute aarch64 binaries, either by emulation with binfmt-misc and qemu-user on Linux, or natively because you have an aarch64-linux computer with Nix on it. I have the latter because Oracle Cloud apparently just gives away 24GB memory aarch64 instances for free.

NixOS has infrastructure for building disk images of systems, so it's more or less a case of doing it, with the correct configuration.

The config for the bootloader is also documented on the NixOS wiki here. The listed config worked for me, which is:

{ ... }: {
  boot.loader.grub = {
    efiSupport = true;
    efiInstallAsRemovable = true;
    device = "nodev";
  };
}

Notable things about the hardware UTM configures by default:

• Ethernet: enp0s5, with the host at 192.168.64.1/24 and the guest receiving an IP via DHCP (probably 192.168.64.2)
• Disk: root on vda2, boot on vda1

It's a QEMU VM so everything is virtio; NixOS provides a configuration to get all the necessary modules, which you can import at (modulesPath + "/profiles/qemu-guest.nix").

Here's the configuration I used:

{ config, modulesPath, pkgs, lib, ... }:
{
  imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];

  networking.hostName = "thinnix";
  networking.useDHCP = false;
  networking.interfaces.enp0s5.useDHCP = true;

  users.users.root.initialPassword = "changeme";

  boot.kernelParams = ["console=ttyAMA0,115200n8" "console=tty0"];
  boot.consoleLogLevel = lib.mkDefault 7;

  boot.growPartition = true;
  # not sure if needed
  boot.initrd.kernelModules = [ "nvme" ];
  boot.loader.grub = {
    efiSupport = true;
    efiInstallAsRemovable = true;
    device = "nodev";
  };

  fileSystems."/" = { device = "/dev/vda2"; fsType = "ext4"; };
  fileSystems."/boot" = { device = "/dev/vda1"; fsType = "vfat"; };

  nixpkgs.localSystem.system = "aarch64-linux";

  system.build.image = import <nixpkgs/nixos/lib/make-disk-image.nix> {
    diskSize = 10000;
    format = "qcow2-compressed";
    installBootLoader = true;
    partitionTableType = "efi";
    inherit config lib pkgs;
  };

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "21.11"; # Did you read the comment?
}

Build with:

$ nix-build -I "nixpkgs=channel:nixos-22.05,nixos-config=$(pwd)/configuration.nix" \
    '<nixpkgs>/nixos' -A config.system.build.image

Creating a VM

To do this, create a VM, selecting the "Other" type:

Then disable ISO boot, since we don't need an installer where we're going.

Finish setting up the VM, don't worry about the disk size since we will immediately delete it. Select "Open VM Settings" on the summary page so you can do that.

Then select the disk and delete it:

Finally, create a new disk, selecting import to get the newly built root filesystem image from Nix: