guest: Ubuntu initramfs only partly unpacks at the default 256 MiB, leaving 1,533 files missing #349

Summary

At its default 256 MiB, the Ubuntu microVM guest cannot unpack its whole initramfs. The kernel prints Initramfs unpacking failed: write error and boots anyway, with 1,533 of the archive's 6,703 entries missing, including part of the dpkg database, and with its root file system full. Every CI step that boots the Ubuntu guest uses that default, and none of them notices.

Reproduction

On dev at 06c0b6297a8e5f9ad779948766165db7bf750399, with the guest artifacts and Linux GNU OpenVMM binary of run 37220619269, on Linux/KVM:

printf '%s\n' 'dmesg | grep -i "initramfs unpacking"' 'df -k / | tail -n 1' \
    'ls -d /usr/share/doc/libbz2-1.0' 'nvx-exit 0' |
    python3 scripts/nvx.py run --guest ubuntu --hypervisor kvm

[    0.930645] Initramfs unpacking failed: write error
rootfs             99228 99228         0 100% /
ls: cannot access '/usr/share/doc/libbz2-1.0': No such file or directory

With --memory-mib 384, the kernel reports no error and the directory exists.

Evidence

The run's initramfs-ubuntu.cpio.gz is 38,354,943 bytes (SHA-256 95cf09e32afae157a6e4053ca9cc4bec30089b5bcd522d9ffd044706c4828ae5) and holds 6,703 entries. A probe booted the managed Ubuntu guest the way managed-lifecycle does, with loglevel=7, and checked each archive entry over the control session:

• The unpacked tree needs about 106 MiB, but at 256 MiB the root file system gets about 97 MiB. It is a tmpfs sized at half of the memory that the kernel manages when it mounts it, before it frees the 37,456 KiB compressed image and about 4.4 MiB of init memory: (240,400 - 37,456 - 4,484) / 2 = 99,230 KiB.
• Unpacking stops at the first failed write. /usr/share/doc/libbsd0/copyright is the last entry written, complete; every later entry, from /usr/share/doc/libbz2-1.0 on, is missing.
• The Ubuntu image of Support a working directory for each execution #333's CI run 37219940994 fails the same way, so the image's size, not a change under review, triggers the failure.
• Azure Linux (about 105 MiB of file data, 512 MiB default) and Alpine (about 18 MiB, 128 MiB default) unpack completely.

Impact

The missing entries hold 6.0 MiB of file data:

• The dpkg database no longer describes the installed files, and Perl modules, locales, and terminfo entries are partly missing, so tools that need them can fail inside the guest.
• The root file system has no space left after boot.
• nvx.py run --guest ubuntu and nvx.py test-microvm --guest ubuntu both default to 256 MiB (default_memory_mib in scripts/nvx_tools/guests.py).

Why CI passes

The Ubuntu steps of .github/workflows/run-nvx-microvm-tests.yml pass no --memory-mib. Their scenarios do not touch the missing files, and nothing checks the unpack result: every test-microvm scenario boots with quiet loglevel=0, which keeps the kernel's message off the console, and the guest still reaches its boot marker.

Possible directions

• Raise the Ubuntu guest's default memory. 384 MiB is the smallest size tested that unpacks completely; a margin over the unpacked size would keep a later package update from crossing the limit again.
• Make a truncated root file system fail the boot, for example with a sentinel that the build writes as the archive's last entry and init checks, so that no scenario can pass on a partial guest.
• Trim the image (documentation, man pages, locales, shell completions) to reduce its size, without relying on that for correctness.

Found while investigating the CI failure of #333, which does not change this behavior.