Skip to content

Implement production trusted storage provider for Cloud Hypervisor enclaves #9440

Description

@lpcox

Summary

Cloud Hypervisor enclaves (enclaves[].runtime: cloud-hypervisor) are wired end to end (#9397, #9398) and have deterministic conformance coverage (#9399), but they still fail closed in production. prepareEnclaves() calls assertCloudHypervisorEnclavePrerequisites(config, deps.cloudHypervisorStorageProvider) (src/enclave/manager.ts), and production never supplies a TrustedCloudHypervisorEnclaveStorageProvider (src/enclave/cloud-hypervisor-lifecycle.ts). Every script, agent, or combined configuration is rejected before seeds, runtime probes, the listener, or a VM are created.

This issue covers the next step: implement that provider so that all writable state of a single invocation is charged to one invocation-owned, bounded allocation domain. Recovery must remain identity-checked. This unblocks live broker-to-VM acceptance in #9395.

Contract (from TrustedCloudHypervisorEnclaveStorageProvider)

  • Aggregate per-invocation capacity: script 1 GiB, agent 512 MiB (CLOUD_HYPERVISOR_ENCLAVE_RESOURCE_PROFILES). The limit applies across all writable exports and storage, including sparse files and concurrent writers, artifact/rootfs snapshots, and runtime state.
  • Enforcement is in place before any host or artifact resource is created and remains until close() succeeds.
  • A free-space or statfs capacity check, sparse-file sizing, or an invocation/guest tmpfs alone is insufficient.
  • assertAvailable(config) fails closed when the host cannot provide this. There is no fallback to Docker, no relaxed limits, and no grant of runner-user KVM access.
  • Protocol v2, static script/agent roles, and existing security gates stay unchanged.

Current gaps (see docs/cloud-hypervisor-foundation.md, "Enclave conformance evidence and remaining live gate")

Gap Today
Artifact snapshots createArtifactSnapshot() allocates under /var/lib/awf-cloud-hypervisor/trusted-artifacts/run-* and needs an executable mount. The invocation tmpfs is noexec, so copying the binaries there cannot meet launch confinement as-is.
VM run state createCloudHypervisorRunPaths() uses /run/awf-cloud-hypervisor, independently of the invocation mount.
Rootfs preparation startCloudHypervisor() prepares <workDir>/cloud-hypervisor-rootfs/<vmRunId> and stages a writable rootfs copy in the manager run directory. None of this is charged to invocation capacity.
Recovery ownership HostExecutorResourceJournal.captureSnapshot() and isTrustedArtifactSnapshotDirectory() accept fixed trusted roots only. Redirecting paths without durable mount/inode ownership would break identity-checked recovery.

Scope

  1. Define one invocation-owned allocation domain, for example a per-invocation bounded mount with separate executable read-only artifact staging and writable noexec state. Its aggregate usage must be enforced by a mechanism the kernel accounts for, not by advisory checks.
  2. Route artifact snapshots, rootfs preparation and staging, manager run paths, and writable exports through it via backendDependencies and managerDependencies.
  3. Preserve executable-artifact attestation and verification and launch confinement (network namespace, privilege drop, Landlock/seccomp).
  4. Extend the resource journal so recovery identifies invocation-owned mounts and inodes durably. Do not relax path validation or add a global bind mount.
  5. Wire the provider into production prepareEnclaves() only on eligible hosts (GitHub-hosted Ubuntu x86_64 with KVM and writable cgroup v2). Everywhere else, keep the current fail-closed error.

Acceptance criteria

  • Unit tests: the provider rejects unavailable hosts, and every writable path for both roles is inside the invocation domain. close() releases everything, and a failed close() keeps enforcement in place.
  • Gated privileged probes in .github/workflows/test-cloud-hypervisor-enclaves.yml show that aggregate ENOSPC at the role limit covers snapshots, rootfs, run state, and exports, including sparse and concurrent writes.
  • Crash/recovery test: recovery cleans only identity-matched invocation resources and never replays work.
  • Without the provider or eligibility, behavior is unchanged: the existing admission error, raised before any resource is created.
  • Docs are updated: docs/cloud-hypervisor-foundation.md gap table, docs/enclaves-architecture.md, and ADR 0002 if the design changes.
  • Follow-up: once this lands, live VM boot plus broker acceptance in [awf] Add live KVM conformance coverage for Cloud Hypervisor enclaves #9395 can proceed.

Related

#9394 (bounded storage contract), #9395 (live KVM acceptance, blocked on this), #9396, #9397, #9398, #9399

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions