feat: add systemd mode for full VM experience #50

rgarcia · 2025-12-24T03:22:11Z

Summary

This PR adds support for running systemd-based OCI images, enabling a VM experience where systemd is PID 1 and manages the full system.

Motivation

Previously, hypeman only supported "exec mode" where the Go init binary runs as PID 1 and executes the container entrypoint directly. This works great for Docker-style single-process containers, but doesn't support images designed to run a full init system.

With this change, you can now run images like jrei/systemd-ubuntu:22.04 and get a full Linux system experience:

systemctl works
journalctl works
SSH and other system services can run
The VM feels like an EC2 instance

How it works

Auto-detection

The mode is auto-detected from the image's CMD:

If CMD is /sbin/init, /lib/systemd/systemd, or similar → systemd mode
Otherwise → exec mode (existing behavior)

Boot Flow

Kernel → init.sh (mount /proc /sys /dev) → Go init binary
                                              ↓
                                    ┌─────────┴─────────┐
                                    ↓                   ↓
                              Exec Mode           Systemd Mode
                                    ↓                   ↓
                            Run entrypoint      chroot + exec /sbin/init
                            as child process    (systemd becomes PID 1)

Guest Agent

In systemd mode, the guest-agent is installed as a systemd service (hypeman-agent.service) that starts automatically. This enables hypeman exec, hypeman cp, and other remote operations.

Key Changes

File	Description
`lib/system/init/*.go`	New Go-based init binary with modular boot phases
`lib/system/init/init.sh`	Shell wrapper (Go runtime needs /proc /sys /dev before main())
`lib/images/systemd.go`	`IsSystemdImage()` auto-detection from CMD
`lib/instances/configdisk.go`	Passes `INIT_MODE` to guest via config disk
`integration/systemd_test.go`	Full E2E test

Testing

# Run the systemd integration test
make test TEST=TestSystemdMode

The test:

Pulls jrei/systemd-ubuntu:22.04
Verifies IsSystemdImage() detects it correctly
Boots a VM
Waits for guest-agent to be ready
Verifies:
- PID 1 is systemd
- /opt/hypeman/guest-agent exists
- hypeman-agent.service is active
- journalctl -u hypeman-agent works

Demo

# Run a systemd VM
hypeman run --name demo jrei/systemd-ubuntu:22.04

# Wait a few seconds for boot, then:
hypeman exec demo cat /proc/1/comm
# → systemd

hypeman exec demo systemctl status hypeman-agent
# → active (running)

hypeman exec demo journalctl -u hypeman-agent --no-pager -n 5
# → shows agent logs

Note

Enables full systemd-based VMs and modernizes guest boot/init flow.

New Go init (lib/system/init/*) with structured logging and dual modes: exec (default) and systemd (auto-detected via images.IsSystemdImage) with service injection for hypeman-agent
Replaces shell init script with embedded binaries (InitBinary, wrapper init.sh); initrd build updated to include these and NVIDIA modules; staleness hash logic adjusted
Host→guest configuration switched to JSON (lib/vmconfig/); config disk now writes config.json and includes network, volumes, GPU, and init_mode
Exec API adds wait_for_agent and guest client adds retryable vsock dialing (AgentVSockDialError) and Unavailable handling
Makefile builds embedded guest-agent and new init; build/dev/test/release-prep depend on these; clean targets updated; .gitignore includes built init
Tests: new integration/systemd_test.go; unit tests for IsSystemdImage; exec tests use WaitForAgent and updated log assertions
Docs: lib/system/README.md updated to describe Go init, storage/versioning, and workflows

^{Written by Cursor Bugbot for commit 0dd3098. This will update automatically on new commits. Configure here.}

Replace shell-based init script with Go binary that supports two modes: ## Exec Mode (existing behavior) - Go init runs as PID 1 - Starts guest-agent in background - Runs container entrypoint as child process - Used for standard Docker images (nginx, python, etc.) ## Systemd Mode (new) - Auto-detected when image CMD is /sbin/init or /lib/systemd/systemd - Go init sets up rootfs, then chroots and execs systemd - Systemd becomes PID 1 and manages the full system - guest-agent runs as a systemd service (hypeman-agent.service) - Enables EC2-like experience: ssh, systemctl, journalctl all work ## Key changes: - lib/system/init/: New Go-based init binary with modular boot phases - lib/images/systemd.go: IsSystemdImage() auto-detection from CMD - lib/instances/configdisk.go: Passes INIT_MODE to guest - lib/system/init/init.sh: Shell wrapper to mount /proc /sys /dev before Go runtime (Go requires these during initialization) - integration/systemd_test.go: Full E2E test verifying: - systemd is PID 1 - hypeman-agent.service is active - journalctl works for viewing logs ## Boot flow: 1. Kernel loads initrd with busybox + Go init + guest-agent 2. init.sh mounts /proc, /sys, /dev (Go runtime needs these) 3. init.sh execs Go init binary 4. Go init mounts overlay rootfs, configures network, copies agent 5. Based on INIT_MODE: exec mode (run entrypoint) or systemd mode (chroot + exec /sbin/init)

lib/images/systemd.go

By default, waits up to 30 seconds for the guest agent to become ready. This prevents immediate failures when the VM is still booting. Use --wait-for-agent=0 to fail immediately (old behavior). Depends on: onkernel/hypeman#50

lib/system/init/mode_exec.go

rgarcia · 2025-12-24T12:34:31Z

.cursor/plans/systemd_vm_with_go_init_c0a9c010.plan.md

@@ -0,0 +1,418 @@
+---


will remove before merge

rgarcia · 2025-12-24T13:50:06Z

lib/system/init/main.go

+}
+
+// dropToShell drops to an interactive shell for debugging when boot fails
+func dropToShell() {


not sure if this is actually useful... opus tooks some liberties

Maybe should log something, just so it's clear what's going on from hypeman logs ...

.cursor/plans/systemd_vm_with_go_init_c0a9c010.plan.md

lib/guest/client.go

sjmiller609

Looks great. One primary simplification tip I think would be nice is the serialize / deserialize config as json simplification, shared single type definition in host and guest code (imported in guest code)

sjmiller609 · 2025-12-25T21:52:45Z

lib/images/systemd.go

+//   - /sbin/init
+//   - /lib/systemd/systemd
+//   - /usr/lib/systemd/systemd
+//   - Any path ending in /init


Maybe any path ending in /init is too broad to decide it's a systemd image. I think it's not that uncommon to name an entrypoint script like that

sjmiller609 · 2025-12-25T21:55:44Z

lib/system/init/config.go

+)
+
+// Config holds the parsed configuration from the config disk.
+type Config struct {


should this be imported from somewhere? seems like this type is already defined wherever we serialize it into the file.

sjmiller609 · 2025-12-25T21:57:39Z

lib/system/init/config.go

+
+// parseConfigFile parses a shell-style configuration file.
+// It handles simple KEY=VALUE and KEY="VALUE" assignments.
+func parseConfigFile(path string) (*Config, error) {


would it be simpler to just change to writing json to the file instead of env list? Since maybe it's only an env list as a left over artifact from being used as "source" but now if it's go writing and go reading, might be cleaner to just define the config struct in one place and use normal json serialization / deserialization.

sjmiller609 · 2025-12-25T22:00:37Z

lib/system/init/config.go

+// parseVolumeMounts parses the VOLUME_MOUNTS string.
+// Format: "device:path:mode[:overlay_device] device:path:mode ..."
+func parseVolumeMounts(s string) []VolumeMount {


Yeah, this could be way simplified I would think if we go with json serialization, define types more cleanly assuming they are being json serialized instead of kinda jamming it into env list then pulling back out from that.

sjmiller609 · 2025-12-25T22:02:02Z

lib/system/init/init.sh

+# Minimal init wrapper that sets up environment before running Go init
+# The Go runtime needs /proc and /dev to exist during initialization


sjmiller609 · 2025-12-25T22:04:51Z

lib/system/init/main.go

+}
+
+// dropToShell drops to an interactive shell for debugging when boot fails
+func dropToShell() {


Maybe should log something, just so it's clear what's going on from hypeman logs ...

sjmiller609 · 2025-12-25T22:07:41Z

lib/system/init/mode_systemd.go

+	// Exec systemd - this replaces the current process
+	log.Info("systemd", "exec /sbin/init")
+
+	// syscall.Exec replaces the current process with the new one
+	// /sbin/init is typically a symlink to /lib/systemd/systemd
+	err := syscall.Exec("/sbin/init", []string{"/sbin/init"}, os.Environ())


is it always /sbin/init? shouldn't we run the user's CMD anyways instead of override it to /sbin/init?

Defines Config and VolumeMount types in a shared package that both the host (configdisk.go) and guest init binary can import, eliminating duplication.

- configdisk.go now uses vmconfig.Config instead of local GuestConfig - init binary now imports vmconfig instead of duplicating types - Also adds logging to dropToShell() for better debugging - mode_systemd.go now runs user's CMD instead of hardcoding /sbin/init

Per review feedback, matching any path ending in /init is too aggressive since many entrypoint scripts are named 'init'. Now only matches explicit systemd paths: /sbin/init, /lib/systemd/systemd, /usr/lib/systemd/systemd

lib/system/init/network.go

lib/system/init/mode_exec.go

lib/system/init/mode_systemd.go

lib/system/init/volumes.go

- buildEnv: user's env vars now take precedence over defaults (PATH, HOME) - systemd mode: pass user's env vars via buildEnv instead of os.Environ() - volumes: use device name for overlay mount points to avoid basename collisions

rgarcia added 2 commits December 23, 2025 20:54

plan

d09b2d8

rgarcia changed the title ~~feat: add systemd mode for EC2-like VMs~~ feat: add systemd mode for full VM experience Dec 24, 2025

cursor bot reviewed Dec 24, 2025

View reviewed changes

lib/images/systemd.go Outdated Show resolved Hide resolved

tweaks

1d67894

rgarcia mentioned this pull request Dec 24, 2025

feat: add --wait-for-agent flag to exec command onkernel/hypeman-cli#20

Open

cursor bot reviewed Dec 24, 2025

View reviewed changes

lib/system/init/mode_exec.go Outdated Show resolved Hide resolved

rgarcia commented Dec 24, 2025

View reviewed changes

tweaks

f0d8608

rgarcia commented Dec 24, 2025

View reviewed changes

cursor bot reviewed Dec 24, 2025

View reviewed changes

.cursor/plans/systemd_vm_with_go_init_c0a9c010.plan.md Outdated Show resolved Hide resolved

lib/guest/client.go Show resolved Hide resolved

Fix test

c7349e0

rgarcia requested a review from sjmiller609 December 24, 2025 14:08

rgarcia added 2 commits December 24, 2025 14:24

fix test

a021f6c

rename AgentConnectionError -> AgentVsockDialError

d79a403

sjmiller609 approved these changes Dec 25, 2025

View reviewed changes

rgarcia added 4 commits December 26, 2025 15:22

feat: add shared vmconfig package for host-to-guest config schema

126c111

Defines Config and VolumeMount types in a shared package that both the host (configdisk.go) and guest init binary can import, eliminating duplication.

fix: remove overly-broad /init suffix detection in IsSystemdImage

824d6e0

Per review feedback, matching any path ending in /init is too aggressive since many entrypoint scripts are named 'init'. Now only matches explicit systemd paths: /sbin/init, /lib/systemd/systemd, /usr/lib/systemd/systemd

chore: remove plan file before merge

4ee5678

cursor bot reviewed Dec 26, 2025

View reviewed changes

lib/system/init/network.go Outdated Show resolved Hide resolved

lib/system/init/mode_exec.go Show resolved Hide resolved

lib/system/init/mode_systemd.go Outdated Show resolved Hide resolved

fix: use %d format for GuestCIDR int field

8b82dfd

cursor bot reviewed Dec 26, 2025

View reviewed changes

lib/system/init/volumes.go Outdated Show resolved Hide resolved

rgarcia merged commit a08c2c8 into main Dec 26, 2025
4 checks passed

rgarcia deleted the full-vm branch December 26, 2025 21:03

		# Minimal init wrapper that sets up environment before running Go init
		# The Go runtime needs /proc and /dev to exist during initialization

feat: add systemd mode for full VM experience #50

feat: add systemd mode for full VM experience #50

Uh oh!

Conversation

rgarcia commented Dec 24, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

How it works

Auto-detection

Boot Flow

Guest Agent

Key Changes

Testing

Demo

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sjmiller609 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rgarcia commented Dec 24, 2025 •

edited by cursor bot

Loading