res-os-builder: container-native bootable image builder (design proposal) #27

New issue

Open

opened 2026-10-09 14:13:44 +00:00 by muse-lantern · 0 comments

muse-lantern commented

2026-10-09 14:13:44 +00:00

Member

Migrated from residual/os#1 (2026-10-09) — the backlog lives in residual/.agent per the workspace manual (AGENTS.md §10 rule 8). Original issue follows unchanged.

res-os-builder: container-native bootable image builder (design proposal)

Goal

Build residual boot images from Docker/OCI containers instead of (or alongside) the current mkimage.sh ISO pipeline. Desired flow:

docker build a FROM-based OS image — config scripts baked in via COPY/RUN
Converter tool turns the OCI image into a bootable image
dd to USB

No privileged build containers, no QEMU in the build path. The whole pipeline must run on standard GitHub runners (fully unprivileged).

Why not bootc

Evaluated and rejected:

bootc defines a bootable image as kernel + initramfs + systemd; systemd-as-PID1 is baked into the model, not optional. Adopting it means replacing res-init — a core design decision of this project.
bootc-image-builder is Fedora/RHEL-family oriented ("especially oriented towards Fedora/CentOS bootc or derivatives"). Alpine support is effectively nonexistent and undocumented; prior attempts at Alpine+bootc hit a wall of undocumented Fedora-isms.

Hand-rolled is the right call here.

Proposed pipeline

Two Dockerfile layers, bottom to top:

Dockerfile.os (one per flavor: console, desktop, virt, rpi, …): FROM alpine → apk add → multi-stage COPY of Go binaries (res-init, res-sh, res-console, residual-login, res-edit) → COPY config scripts → RUN scripts. This image is the OS filesystem and runs as a plain Docker image (docker run it) — flavors are just variants of this layer. The knowledge in mkimg.residual*.sh (package lists, config) migrates here.
Dockerfile.residual: FROM the OS image above → adds everything needed to build it into a bootable OS: kernel + modules under /boot, bootloader config, initramfs hooks/config, partition-layout hints. A thin, auditable delta on top of a tested OS image.
Push the residual image to an OCI registry (or docker save a tarball) — the tool accepts both.
res-os-builder convert (new Go tool, lives at cmd/res-os-builder in this repo):
- unpack OCI layers → rootfs/ (umoci/crane, no daemon needed)
- kernel + initramfs: discovered from the image's /boot per the image contract (built in-image by the distro's native initramfs tooling during the residual-layer build — mkinitfs on Alpine, initramfs-tools on Ubuntu, mkinitcpio on Arch)
- mksquashfs rootfs/ → system.squashfs
- assemble the bootable artifact:
  - ISO via xorriso, and/or
  - raw GPT .img: FAT32 ESP via mtools, partitions laid out with sfdisk + dd at offsets — no loop mounts, no privileges
Every step above works unprivileged (mksquashfs, xorriso, mkfs.ext4 -d, mtools all do). Builds on any stock runner.

Boot model: immutable OS + overlay for user data

system.squashfs mounted read-only as the overlay lower → immutable system
overlay upper: tmpfs (stateless) or a labeled persistent partition (RESIDUAL_DATA, ext4) when present — same image boots stateless on any machine, persistent where the data partition exists, no flags needed
initramfs mounts the media, locates the squashfs, sets up the overlay, switch_root → stock init (OpenRC /sbin/init). res-init runs as a supervised sidecar service inside it — not as PID 1 (see refinements below)
This mirrors the current ISO's live-boot model; the only thing that changes is where the rootfs comes from (container instead of apk tree). The knowledge in mkimg.residual*.sh (package lists, config) migrates into Dockerfile.os.

Two-layer Dockerfiles: the OS image (Dockerfile.os, per flavor) is the runnable system — itself a valid Docker image; Dockerfile.residual layers only the boot machinery on top. Keeps the OS image container-testable and the boot delta thin.
res-init as sidecar, not PID 1 (for now): res-init is currently built as a sidecar init that sits within another init system. The bootable image boots stock init; res-init is supervised within it. PID 1 is a future step, not a requirement of this builder.
The builder is a real Go tool, not a script (cmd/res-os-builder). It owns the whole pipeline end-to-end: builds the OCI image locally from the Dockerfiles, passes it through the residual boot layer, and converts it to the boot artifact for the requested boot setup (ISO vs raw .img, BIOS vs UEFI, …).
Custom initramfs still required: a stock initramfs won't carry the persistence drivers. Ours needs exFAT + BTRFS kernel modules and qcow2 support for the persistence backends. Ties into .agent#10's initramfs-init-residual fork decision — this builder consumes that work, it doesn't replace it.
Overlay/persistence designed in from day one: no building without it and retrofitting later. No lock-in by deferral.
Distro-generic by contract, not by distro knowledge (2026-10-09, per user): the tool must handle Ubuntu- or Arch-flavored images, not just Alpine. Mechanism: the Go tool stays distro-dumb and defines a residual image contract — a residual-ready image provides /boot/vmlinuz-* + /boot/initramfs-* (paths discoverable via image labels or /etc/residual/image.toml), and its initramfs contains our portable persistence hook. All distro-specific work (package manager, initramfs builder, hook directory) lives in the Dockerfiles, where the distro's native tooling exists — never in the Go tool.
Consequence of the above: initramfs generation moves into the Dockerfile.residual RUN step (distro-native builder + our hook installed via that builder's hook mechanism); the converter only consumes the artifacts. Our persistence hook ships as portable busybox-ash.
The sidecar-init decision is what makes multi-distro feasible: res-init as PID 1 would need per-distro init integration; as a sidecar it just runs as a service under whatever init the distro ships (OpenRC, systemd, …).
Flash drive partition layout (2026-10-09, per user): partition 1 = boot partition holding the immutable residual base layer (kernel, initramfs, squashfs, bootloader). Rest of the drive = either exFAT carrying a qcow2 image file, or a raw BTRFS partition for residual. The exFAT+qcow2 layout is the project-vision default: the data store stays a plain file — visible, copyable, back-up-able from any OS you plug the drive into. Initramfs driver bill of materials: exfat + btrfs kernel modules, nbd kernel module + nbd client (qemu-nbd) for the qcow2 file. Boot-time discovery: exFAT+qcow2 → nbd-attach → btrfs upper; raw btrfs → direct upper; neither found → tmpfs upper. The initramfs implements the overlay mount with boot-time discovery from the first POC — persistent store found → it becomes the upper; not found → tmpfs upper. Same image, no flags, no later rewrite. Partition layout reserves the data partition from the start.

Open decisions

Output: ISO, raw .img, or both from a single build?
Kernel shipped inside the container image (apk add linux-lts in the Dockerfile)?
Input: tool builds the image locally from Dockerfiles, registry reference, local tarball — which modes?
Image build backend: shell out to docker build/buildkit on the runner, or daemonless layer assembly?
Data partition label/format defaults (now also: exFAT in the mix alongside btrfs)?
qcow2 backend shape: qcow2 file on a data partition (needs nbd tooling in initramfs) vs qcow2 as the VM disk itself (needs nothing in-guest)?

Suggested first milestone

Probe runner capabilities via a workflow (docker daemon? /dev/kvm? loop devices?)
Minimal POC: alpine + kernel + trivial init → bootable image → QEMU boot check on the runner, grepping the serial log for a marker string; upload the .img as a workflow artifact

> **Migrated from `residual/os#1`** (2026-10-09) — the backlog lives in `residual/.agent` per the workspace manual (AGENTS.md §10 rule 8). Original issue follows unchanged. --- # res-os-builder: container-native bootable image builder (design proposal) ## Goal Build residual boot images from Docker/OCI containers instead of (or alongside) the current `mkimage.sh` ISO pipeline. Desired flow: 1. `docker build` a `FROM`-based OS image — config scripts baked in via `COPY`/`RUN` 2. Converter tool turns the OCI image into a bootable image 3. `dd` to USB No privileged build containers, no QEMU in the build path. The whole pipeline must run on **standard GitHub runners** (fully unprivileged). ## Why not bootc Evaluated and rejected: - bootc defines a bootable image as kernel + initramfs + **systemd**; systemd-as-PID1 is baked into the model, not optional. Adopting it means replacing `res-init` — a core design decision of this project. - `bootc-image-builder` is Fedora/RHEL-family oriented ("especially oriented towards Fedora/CentOS bootc or derivatives"). Alpine support is effectively nonexistent and undocumented; prior attempts at Alpine+bootc hit a wall of undocumented Fedora-isms. Hand-rolled is the right call here. ## Proposed pipeline Two Dockerfile layers, bottom to top: 1. **`Dockerfile.os`** (one per flavor: console, desktop, virt, rpi, …): `FROM alpine` → `apk add` → multi-stage `COPY` of Go binaries (`res-init`, `res-sh`, `res-console`, `residual-login`, `res-edit`) → `COPY` config scripts → `RUN` scripts. This image **is the OS filesystem and runs as a plain Docker image** (`docker run` it) — flavors are just variants of this layer. The knowledge in `mkimg.residual*.sh` (package lists, config) migrates here. 2. **`Dockerfile.residual`**: `FROM` the OS image above → adds everything needed to *build* it into a bootable OS: kernel + modules under `/boot`, bootloader config, initramfs hooks/config, partition-layout hints. A thin, auditable delta on top of a tested OS image. 3. Push the residual image to an OCI registry (or `docker save` a tarball) — the tool accepts both. 4. **`res-os-builder convert`** (new Go tool, lives at `cmd/res-os-builder` in this repo): - unpack OCI layers → `rootfs/` (`umoci`/`crane`, no daemon needed) - kernel + initramfs: discovered from the image's `/boot` per the image contract (built in-image by the distro's native initramfs tooling during the residual-layer build — mkinitfs on Alpine, initramfs-tools on Ubuntu, mkinitcpio on Arch) - `mksquashfs rootfs/` → `system.squashfs` - assemble the bootable artifact: - ISO via `xorriso`, and/or - raw GPT `.img`: FAT32 ESP via `mtools`, partitions laid out with `sfdisk` + `dd` at offsets — **no loop mounts, no privileges** 5. Every step above works unprivileged (`mksquashfs`, `xorriso`, `mkfs.ext4 -d`, `mtools` all do). Builds on any stock runner. ## Boot model: immutable OS + overlay for user data - `system.squashfs` mounted read-only as the overlay **lower** → immutable system - overlay **upper**: `tmpfs` (stateless) **or** a labeled persistent partition (`RESIDUAL_DATA`, ext4) when present — same image boots stateless on any machine, persistent where the data partition exists, no flags needed - initramfs mounts the media, locates the squashfs, sets up the overlay, `switch_root` → stock init (OpenRC `/sbin/init`). `res-init` runs as a supervised **sidecar** service inside it — not as PID 1 (see refinements below) - This mirrors the current ISO's live-boot model; the only thing that changes is where the rootfs comes *from* (container instead of apk tree). The knowledge in `mkimg.residual*.sh` (package lists, config) migrates into `Dockerfile.os`. ## Design refinements (2026-10-09, per user) - **Two-layer Dockerfiles**: the OS image (`Dockerfile.os`, per flavor) is the runnable system — itself a valid Docker image; `Dockerfile.residual` layers only the boot machinery on top. Keeps the OS image container-testable and the boot delta thin. - **res-init as sidecar, not PID 1 (for now)**: res-init is currently built as a sidecar init that sits *within* another init system. The bootable image boots stock init; res-init is supervised within it. PID 1 is a future step, not a requirement of this builder. - **The builder is a real Go tool, not a script** (`cmd/res-os-builder`). It owns the whole pipeline end-to-end: builds the OCI image locally from the Dockerfiles, passes it through the residual boot layer, and converts it to the boot artifact for the requested boot setup (ISO vs raw `.img`, BIOS vs UEFI, …). - **Custom initramfs still required**: a stock initramfs won't carry the persistence drivers. Ours needs exFAT + BTRFS kernel modules and qcow2 support for the persistence backends. Ties into `.agent#10`'s `initramfs-init-residual` fork decision — this builder consumes that work, it doesn't replace it. - **Overlay/persistence designed in from day one**: no building without it and retrofitting later. No lock-in by deferral. - **Distro-generic by contract, not by distro knowledge** (2026-10-09, per user): the tool must handle Ubuntu- or Arch-flavored images, not just Alpine. Mechanism: the Go tool stays distro-dumb and defines a *residual image contract* — a residual-ready image provides `/boot/vmlinuz-*` + `/boot/initramfs-*` (paths discoverable via image labels or `/etc/residual/image.toml`), and its initramfs contains our portable persistence hook. All distro-specific work (package manager, initramfs builder, hook directory) lives in the Dockerfiles, where the distro's native tooling exists — never in the Go tool. - Consequence of the above: initramfs *generation* moves into the `Dockerfile.residual` RUN step (distro-native builder + our hook installed via that builder's hook mechanism); the converter only consumes the artifacts. Our persistence hook ships as portable busybox-ash. - The sidecar-init decision is what makes multi-distro feasible: res-init as PID 1 would need per-distro init integration; as a sidecar it just runs as a service under whatever init the distro ships (OpenRC, systemd, …). - **Flash drive partition layout** (2026-10-09, per user): partition 1 = boot partition holding the immutable residual base layer (kernel, initramfs, squashfs, bootloader). Rest of the drive = either exFAT carrying a qcow2 image file, or a raw BTRFS partition for residual. The **exFAT+qcow2 layout is the project-vision default**: the data store stays a plain file — visible, copyable, back-up-able from any OS you plug the drive into. Initramfs driver bill of materials: exfat + btrfs kernel modules, nbd kernel module + nbd client (`qemu-nbd`) for the qcow2 file. Boot-time discovery: exFAT+qcow2 → nbd-attach → btrfs upper; raw btrfs → direct upper; neither found → tmpfs upper. The initramfs implements the overlay mount with boot-time discovery from the first POC — persistent store found → it becomes the upper; not found → tmpfs upper. Same image, no flags, no later rewrite. Partition layout reserves the data partition from the start. ## Open decisions - [ ] Output: ISO, raw `.img`, or both from a single build? - [ ] Kernel shipped inside the container image (`apk add linux-lts` in the Dockerfile)? - [ ] Input: tool builds the image locally from Dockerfiles, registry reference, local tarball — which modes? - [ ] Image build backend: shell out to `docker build`/buildkit on the runner, or daemonless layer assembly? - [ ] Data partition label/format defaults (now also: exFAT in the mix alongside btrfs)? - [ ] qcow2 backend shape: qcow2 *file* on a data partition (needs nbd tooling in initramfs) vs qcow2 as the VM disk itself (needs nothing in-guest)? ## Suggested first milestone 1. Probe runner capabilities via a workflow (docker daemon? `/dev/kvm`? loop devices?) 2. Minimal POC: alpine + kernel + trivial init → bootable image → QEMU boot check on the runner, grepping the serial log for a marker string; upload the `.img` as a workflow artifact

muse-lantern added the

proj-os

tier-3-hard

labels

2026-10-09 14:13:44 +00:00

muse-lantern referenced this issue from residual/os

2026-10-09 14:13:45 +00:00

res-os-builder: container-native bootable image builder (design proposal) #1

muse-lantern referenced this issue from a commit

2026-10-09 19:43:34 +00:00

res-os-builder: Alpine POC pipeline (rootfs, initramfs, GPT image)

muse-lantern referenced this issue from residual/os