I3: Choose how Docker builds run: dedicated build VM or rootless #45

Open
opened 2026-11-09 17:06:00 +00:00 by claude · 5 comments
Member

Compare a dedicated build VM on one of the hypervisors, rootless DinD, and a daemonless builder for image builds. Cover isolation, whether existing workflows keep working, CPU and disk available, and maintenance cost.

Work (Claude): write the comparison and a recommendation in a comment, ideally with a small proof of concept. Keep sensitive details in the private assessment.
Decision (Eric): choose, since a VM needs host resources.

Acceptance: decision recorded, follow-up implementation issue opened.

Depends on: I1.

Compare a dedicated build VM on one of the hypervisors, rootless DinD, and a daemonless builder for image builds. Cover isolation, whether existing workflows keep working, CPU and disk available, and maintenance cost. **Work (Claude)**: write the comparison and a recommendation in a comment, ideally with a small proof of concept. Keep sensitive details in the private assessment. **Decision (Eric)**: choose, since a VM needs host resources. **Acceptance**: decision recorded, follow-up implementation issue opened. Depends on: I1.
claude added this to the CI/CD 1: Runner policy and isolation milestone 2026-11-09 17:06:00 +00:00
claude added the needs-eric
priority
high
cicd
labels 2026-11-09 17:06:00 +00:00
claude added this to the CI/CD Roadmap project 2026-11-09 17:07:33 +00:00
Owner

What is your recommendation? I have a vm for personal use that isn't really being used, but rootless seems more in keeping with my Kubernetes infrastructure. What do you recommend?

What is your recommendation? I have a vm for personal use that isn't really being used, but rootless seems more in keeping with my Kubernetes infrastructure. What do you recommend?
Author
Member

Recommendation: use the spare VM as the CI host, outside the cluster. Rootless in the cluster gains less than its name suggests, and the VM also settles the open node question in S3 (#36).

Comparison

Runner VM outside the cluster Rootless DinD in the cluster Daemonless builder (rootless BuildKit)
Separation from production Its own kernel and its own firewall rules Same nodes and pod network as production Same as rootless, for image builds only
Existing workflows Work unchanged, including jobs that call docker Most work. Jobs that need privileged containers or low ports don't, and networking is slower Break. The runner still needs a Docker daemon to start each job's container, so this replaces only docker build
CPU and disk Whatever the VM has, not shared with services Still needs a worker with a big disk, so the S3 node decision stays open Same as rootless
Upkeep One more host to patch. Declared in this repo like the other VMs Stays in the Helm chart, with one flag A second build path to maintain next to the runner

What I tested

  • Chart gitea/actions 0.1.2 has a statefulset.dind.rootless flag. I read the chart's template: the flag changes the user the Docker daemon runs as and where its socket is, and leaves the container's other security settings as they are.
  • I ran docker:29.5.2-dind-rootless on the workstation three ways. It only started with the same container settings the chart uses today. Started that way, the daemon ran as an unprivileged user, used the overlayfs driver and ran a job container. This was a local test, not on the cluster.
  • Rootless with tighter pod settings would need newer container runtime and kernel versions than most workers have.

Details of the isolation side are in the private assessment.

Why the VM, even though rootless is more Kubernetes-native

  • It's the only one of the three that changes where CI runs, not only how. Network separation (I2, #44) becomes a few firewall rules on one host.
  • It answers S3 without growing a worker's disk, and gives S4 (#37) its own CPU.
  • It's the dedicated build host from L1 (#78), brought forward. It would still be managed from this repo with Ansible, so it stays declarative.
  • Rootless Docker can be turned on later on the VM as a second layer, once the workflows are known to cope.

What the VM needs: 4 vCPUs if possible (2 at least), 8 GB RAM or more, about 100 GB of disk, Debian 13, outbound HTTPS to gitea.eom.dev and the internet, and SSH from the Ansible control host.

If you'd rather stay inside Kubernetes: join that VM to the cluster as a dedicated worker and label it eom.dev/ci-runner=true. Everything already committed for S3 to S5 then deploys as it is. Disk and CPU are fixed, and separation stays as it is today.

To decide

  1. Standalone runner VM (recommended). Tell me the VM's inventory name and size. I'll write the role tasks for a runner on that host, carrying over the S4 and S5 settings (one job at a time, timeouts, pinned job images, scheduled cleanup), and label it ready-to-deploy. The in-cluster runner is removed once the new one is serving jobs.
  2. VM as a dedicated cluster node. Join and label it, then run the command in #36.
  3. Rootless in the cluster. I'd add the flag to the role. S3 still needs a node.

The runner was evicted again today for the same disk reason as on 2026-11-09, so whichever you choose, #36 is the urgent part.

— eom-cicd-roadmap

**Recommendation: use the spare VM as the CI host, outside the cluster.** Rootless in the cluster gains less than its name suggests, and the VM also settles the open node question in S3 (#36). **Comparison** | | Runner VM outside the cluster | Rootless DinD in the cluster | Daemonless builder (rootless BuildKit) | |---|---|---|---| | Separation from production | Its own kernel and its own firewall rules | Same nodes and pod network as production | Same as rootless, for image builds only | | Existing workflows | Work unchanged, including jobs that call `docker` | Most work. Jobs that need privileged containers or low ports don't, and networking is slower | Break. The runner still needs a Docker daemon to start each job's container, so this replaces only `docker build` | | CPU and disk | Whatever the VM has, not shared with services | Still needs a worker with a big disk, so the S3 node decision stays open | Same as rootless | | Upkeep | One more host to patch. Declared in this repo like the other VMs | Stays in the Helm chart, with one flag | A second build path to maintain next to the runner | **What I tested** - Chart `gitea/actions` 0.1.2 has a `statefulset.dind.rootless` flag. I read the chart's template: the flag changes the user the Docker daemon runs as and where its socket is, and leaves the container's other security settings as they are. - I ran `docker:29.5.2-dind-rootless` on the workstation three ways. It only started with the same container settings the chart uses today. Started that way, the daemon ran as an unprivileged user, used the `overlayfs` driver and ran a job container. This was a local test, not on the cluster. - Rootless with tighter pod settings would need newer container runtime and kernel versions than most workers have. Details of the isolation side are in the private assessment. **Why the VM, even though rootless is more Kubernetes-native** - It's the only one of the three that changes where CI runs, not only how. Network separation (I2, #44) becomes a few firewall rules on one host. - It answers S3 without growing a worker's disk, and gives S4 (#37) its own CPU. - It's the dedicated build host from L1 (#78), brought forward. It would still be managed from this repo with Ansible, so it stays declarative. - Rootless Docker can be turned on later on the VM as a second layer, once the workflows are known to cope. **What the VM needs:** 4 vCPUs if possible (2 at least), 8 GB RAM or more, about 100 GB of disk, Debian 13, outbound HTTPS to `gitea.eom.dev` and the internet, and SSH from the Ansible control host. **If you'd rather stay inside Kubernetes:** join that VM to the cluster as a dedicated worker and label it `eom.dev/ci-runner=true`. Everything already committed for S3 to S5 then deploys as it is. Disk and CPU are fixed, and separation stays as it is today. **To decide** 1. **Standalone runner VM (recommended).** Tell me the VM's inventory name and size. I'll write the role tasks for a runner on that host, carrying over the S4 and S5 settings (one job at a time, timeouts, pinned job images, scheduled cleanup), and label it `ready-to-deploy`. The in-cluster runner is removed once the new one is serving jobs. 2. **VM as a dedicated cluster node.** Join and label it, then run the command in #36. 3. **Rootless in the cluster.** I'd add the flag to the role. S3 still needs a node. The runner was evicted again today for the same disk reason as on 2026-11-09, so whichever you choose, #36 is the urgent part. — eom-cicd-roadmap
Owner

The VM doesn't have the resources you described. What about the inspiron-3670? Could we use that instead?

The VM doesn't have the resources you described. What about the inspiron-3670? Could we use that instead?
Author
Member

Yes. inspiron-3670 is a better fit than the VM I described. I checked it read-only today:

inspiron-3670 Needed
CPU 6 cores (i5-8400) 4, at least 2
Memory 62 GB, about 56 GB free 8 GB or more
Disk 100 GB free on / about 100 GB
OS / Docker Debian 13, Docker 26.1 with Compose 2.26 Debian 13

Three things to know first

  1. CPU is already busy. A long-running background workload uses about 4.5 of the 6 cores. CI would share what's left, so builds would be slow. Recommendation: run that workload at the lowest priority (nice 19 or CPUWeight=1 in its service) so CI and OctoPrint get CPU first, and it takes whatever is left over.
  2. Disk. / is on the spinning disk. It has room, but image pulls and builds are slower there. The 466 GB SSD appears to hold another OS's partitions. If that disk is free to use, Docker data on it would make builds much faster. I haven't touched it and won't plan on it without your say. By default the draft uses /var/lib/gitea-runner/docker on /.
  3. The printers. OctoPrint streams G-code over USB, and long CPU stalls can cause print blemishes. The draft caps CI at 4 of 6 CPUs and 16 GB, and gives it a lower CPU weight than the host's other containers. Watch the first print that runs during a build.

What's drafted (committed, not applied)

  • Role: tasks/actions_host.yaml, templates/runner-compose.yaml.j2, templates/runner-config.yaml.j2 and gitea_actions_host_* defaults in DevOps/ansible-role-gitea@9f768ed and @c7874a4.
  • This repo: ci-runner.yaml, the submodule bump, and a section in docs/ci-cd/runner.md (4d6e43e, 5c72efc, c7874a4).
  • Design: a Compose project with its own Docker daemon (DinD) for jobs, separate from the host's Docker, so jobs can't see or touch OctoPrint or CNCjs. It also includes the runner and the S5 cleanup script. Capacity 1, the 2h job timeout and the pinned job images all come from the S4 variables. The token is fetched once from the cluster, used to register, and then deleted.

How it was tested

  • ansible-playbook --syntax-check, ansible-lint (no findings apart from the role's existing lowercase task names) and yamllint passed, run in a container.
  • I rendered the templates and checked them with docker compose config.
  • I ran the rendered dind and prune services on the workstation, with smaller CPU and memory caps. The daemon came up healthy with the overlayfs driver, the caps showed up in its cgroup, and a job-style container ran through it and reached gitea.eom.dev. The cleanup completed a pass. Everything was removed afterwards.
  • Not tested: the runner registering and running a real job, because that needs a registration token, and anything on inspiron-3670 itself.

To go ahead

  1. Reply here with: go/no-go on inspiron-3670, what to do about the background workload (lower its priority, recommended; or leave it), and whether the SSD is free to use.
  2. Then I'll adjust the variables, label this ready-to-deploy, and post the run notice. The command will be:
    ansible-playbook -i inventories/eom.dev.yml ci-runner.yaml -K
    • Effect: a second runner comes up on inspiron-3670 next to the in-cluster one. Both take jobs until the cluster runner is removed in a later step, so nothing running is interrupted.
    • Risk: builds compete with whatever else runs on that machine.
    • Rollback: docker compose -f /opt/gitea-runner/compose.yaml down on the host.

Network separation for this host belongs to I2 (#44). The details are in the private assessment.

— eom-cicd-roadmap

**Yes. inspiron-3670 is a better fit than the VM I described.** I checked it read-only today: | | inspiron-3670 | Needed | |---|---|---| | CPU | 6 cores (i5-8400) | 4, at least 2 | | Memory | 62 GB, about 56 GB free | 8 GB or more | | Disk | 100 GB free on `/` | about 100 GB | | OS / Docker | Debian 13, Docker 26.1 with Compose 2.26 | Debian 13 | **Three things to know first** 1. **CPU is already busy.** A long-running background workload uses about 4.5 of the 6 cores. CI would share what's left, so builds would be slow. **Recommendation:** run that workload at the lowest priority (`nice 19` or `CPUWeight=1` in its service) so CI and OctoPrint get CPU first, and it takes whatever is left over. 2. **Disk.** `/` is on the spinning disk. It has room, but image pulls and builds are slower there. The 466 GB SSD appears to hold another OS's partitions. If that disk is free to use, Docker data on it would make builds much faster. I haven't touched it and won't plan on it without your say. By default the draft uses `/var/lib/gitea-runner/docker` on `/`. 3. **The printers.** OctoPrint streams G-code over USB, and long CPU stalls can cause print blemishes. The draft caps CI at 4 of 6 CPUs and 16 GB, and gives it a lower CPU weight than the host's other containers. Watch the first print that runs during a build. **What's drafted (committed, not applied)** - Role: `tasks/actions_host.yaml`, `templates/runner-compose.yaml.j2`, `templates/runner-config.yaml.j2` and `gitea_actions_host_*` defaults in DevOps/ansible-role-gitea@9f768ed and @c7874a4. - This repo: `ci-runner.yaml`, the submodule bump, and a section in `docs/ci-cd/runner.md` (4d6e43e, 5c72efc, c7874a4). - Design: a Compose project with **its own Docker daemon (DinD)** for jobs, separate from the host's Docker, so jobs can't see or touch OctoPrint or CNCjs. It also includes the runner and the S5 cleanup script. Capacity 1, the 2h job timeout and the pinned job images all come from the S4 variables. The token is fetched once from the cluster, used to register, and then deleted. **How it was tested** - `ansible-playbook --syntax-check`, `ansible-lint` (no findings apart from the role's existing lowercase task names) and `yamllint` passed, run in a container. - I rendered the templates and checked them with `docker compose config`. - I ran the rendered `dind` and `prune` services on the workstation, with smaller CPU and memory caps. The daemon came up healthy with the `overlayfs` driver, the caps showed up in its cgroup, and a job-style container ran through it and reached gitea.eom.dev. The cleanup completed a pass. Everything was removed afterwards. - **Not tested:** the runner registering and running a real job, because that needs a registration token, and anything on inspiron-3670 itself. **To go ahead** 1. Reply here with: go/no-go on inspiron-3670, what to do about the background workload (lower its priority, recommended; or leave it), and whether the SSD is free to use. 2. Then I'll adjust the variables, label this `ready-to-deploy`, and post the run notice. The command will be: `ansible-playbook -i inventories/eom.dev.yml ci-runner.yaml -K` - **Effect:** a second runner comes up on inspiron-3670 next to the in-cluster one. Both take jobs until the cluster runner is removed in a later step, so nothing running is interrupted. - **Risk:** builds compete with whatever else runs on that machine. - **Rollback:** `docker compose -f /opt/gitea-runner/compose.yaml down` on the host. Network separation for this host belongs to I2 (#44). The details are in the private assessment. — eom-cicd-roadmap
Author
Member

One more input for the decision: before choosing, please read the new "I3 follow-up: inspiron-3670" section in the private assessment. It covers this machine's other roles, including the USB-attached tools, and what that means for isolation. It may change which option you prefer.

— eom-cicd-roadmap

One more input for the decision: before choosing, please read the new "I3 follow-up: inspiron-3670" section in the private assessment. It covers this machine's other roles, including the USB-attached tools, and what that means for isolation. It may change which option you prefer. — eom-cicd-roadmap
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DevOps/software-infrastructure#45