Compare a dedicated build VM on one of the hypervisors, rootless DinD, and a daemonless builder for image builds. Cover isolation, whether existing workflows keep working, CPU and disk available, and maintenance cost.
Work (Claude): write the comparison and a recommendation in a comment, ideally with a small proof of concept. Keep sensitive details in the private assessment. Decision (Eric): choose, since a VM needs host resources.
Compare a dedicated build VM on one of the hypervisors, rootless DinD, and a daemonless builder for image builds. Cover isolation, whether existing workflows keep working, CPU and disk available, and maintenance cost.
**Work (Claude)**: write the comparison and a recommendation in a comment, ideally with a small proof of concept. Keep sensitive details in the private assessment.
**Decision (Eric)**: choose, since a VM needs host resources.
**Acceptance**: decision recorded, follow-up implementation issue opened.
Depends on: I1.
claude
added this to the CI/CD 1: Runner policy and isolation milestone 2026-11-09 17:06:00 +00:00
What is your recommendation? I have a vm for personal use that isn't really being used, but rootless seems more in keeping with my Kubernetes infrastructure. What do you recommend?
What is your recommendation? I have a vm for personal use that isn't really being used, but rootless seems more in keeping with my Kubernetes infrastructure. What do you recommend?
Recommendation: use the spare VM as the CI host, outside the cluster. Rootless in the cluster gains less than its name suggests, and the VM also settles the open node question in S3 (#36).
Comparison
Runner VM outside the cluster
Rootless DinD in the cluster
Daemonless builder (rootless BuildKit)
Separation from production
Its own kernel and its own firewall rules
Same nodes and pod network as production
Same as rootless, for image builds only
Existing workflows
Work unchanged, including jobs that call docker
Most work. Jobs that need privileged containers or low ports don't, and networking is slower
Break. The runner still needs a Docker daemon to start each job's container, so this replaces only docker build
CPU and disk
Whatever the VM has, not shared with services
Still needs a worker with a big disk, so the S3 node decision stays open
Same as rootless
Upkeep
One more host to patch. Declared in this repo like the other VMs
Stays in the Helm chart, with one flag
A second build path to maintain next to the runner
What I tested
Chart gitea/actions 0.1.2 has a statefulset.dind.rootless flag. I read the chart's template: the flag changes the user the Docker daemon runs as and where its socket is, and leaves the container's other security settings as they are.
I ran docker:29.5.2-dind-rootless on the workstation three ways. It only started with the same container settings the chart uses today. Started that way, the daemon ran as an unprivileged user, used the overlayfs driver and ran a job container. This was a local test, not on the cluster.
Rootless with tighter pod settings would need newer container runtime and kernel versions than most workers have.
Details of the isolation side are in the private assessment.
Why the VM, even though rootless is more Kubernetes-native
It's the only one of the three that changes where CI runs, not only how. Network separation (I2, #44) becomes a few firewall rules on one host.
It answers S3 without growing a worker's disk, and gives S4 (#37) its own CPU.
It's the dedicated build host from L1 (#78), brought forward. It would still be managed from this repo with Ansible, so it stays declarative.
Rootless Docker can be turned on later on the VM as a second layer, once the workflows are known to cope.
What the VM needs: 4 vCPUs if possible (2 at least), 8 GB RAM or more, about 100 GB of disk, Debian 13, outbound HTTPS to gitea.eom.dev and the internet, and SSH from the Ansible control host.
If you'd rather stay inside Kubernetes: join that VM to the cluster as a dedicated worker and label it eom.dev/ci-runner=true. Everything already committed for S3 to S5 then deploys as it is. Disk and CPU are fixed, and separation stays as it is today.
To decide
Standalone runner VM (recommended). Tell me the VM's inventory name and size. I'll write the role tasks for a runner on that host, carrying over the S4 and S5 settings (one job at a time, timeouts, pinned job images, scheduled cleanup), and label it ready-to-deploy. The in-cluster runner is removed once the new one is serving jobs.
VM as a dedicated cluster node. Join and label it, then run the command in #36.
Rootless in the cluster. I'd add the flag to the role. S3 still needs a node.
The runner was evicted again today for the same disk reason as on 2026-11-09, so whichever you choose, #36 is the urgent part.
— eom-cicd-roadmap
**Recommendation: use the spare VM as the CI host, outside the cluster.** Rootless in the cluster gains less than its name suggests, and the VM also settles the open node question in S3 (#36).
**Comparison**
| | Runner VM outside the cluster | Rootless DinD in the cluster | Daemonless builder (rootless BuildKit) |
|---|---|---|---|
| Separation from production | Its own kernel and its own firewall rules | Same nodes and pod network as production | Same as rootless, for image builds only |
| Existing workflows | Work unchanged, including jobs that call `docker` | Most work. Jobs that need privileged containers or low ports don't, and networking is slower | Break. The runner still needs a Docker daemon to start each job's container, so this replaces only `docker build` |
| CPU and disk | Whatever the VM has, not shared with services | Still needs a worker with a big disk, so the S3 node decision stays open | Same as rootless |
| Upkeep | One more host to patch. Declared in this repo like the other VMs | Stays in the Helm chart, with one flag | A second build path to maintain next to the runner |
**What I tested**
- Chart `gitea/actions` 0.1.2 has a `statefulset.dind.rootless` flag. I read the chart's template: the flag changes the user the Docker daemon runs as and where its socket is, and leaves the container's other security settings as they are.
- I ran `docker:29.5.2-dind-rootless` on the workstation three ways. It only started with the same container settings the chart uses today. Started that way, the daemon ran as an unprivileged user, used the `overlayfs` driver and ran a job container. This was a local test, not on the cluster.
- Rootless with tighter pod settings would need newer container runtime and kernel versions than most workers have.
Details of the isolation side are in the private assessment.
**Why the VM, even though rootless is more Kubernetes-native**
- It's the only one of the three that changes where CI runs, not only how. Network separation (I2, #44) becomes a few firewall rules on one host.
- It answers S3 without growing a worker's disk, and gives S4 (#37) its own CPU.
- It's the dedicated build host from L1 (#78), brought forward. It would still be managed from this repo with Ansible, so it stays declarative.
- Rootless Docker can be turned on later on the VM as a second layer, once the workflows are known to cope.
**What the VM needs:** 4 vCPUs if possible (2 at least), 8 GB RAM or more, about 100 GB of disk, Debian 13, outbound HTTPS to `gitea.eom.dev` and the internet, and SSH from the Ansible control host.
**If you'd rather stay inside Kubernetes:** join that VM to the cluster as a dedicated worker and label it `eom.dev/ci-runner=true`. Everything already committed for S3 to S5 then deploys as it is. Disk and CPU are fixed, and separation stays as it is today.
**To decide**
1. **Standalone runner VM (recommended).** Tell me the VM's inventory name and size. I'll write the role tasks for a runner on that host, carrying over the S4 and S5 settings (one job at a time, timeouts, pinned job images, scheduled cleanup), and label it `ready-to-deploy`. The in-cluster runner is removed once the new one is serving jobs.
2. **VM as a dedicated cluster node.** Join and label it, then run the command in #36.
3. **Rootless in the cluster.** I'd add the flag to the role. S3 still needs a node.
The runner was evicted again today for the same disk reason as on 2026-11-09, so whichever you choose, #36 is the urgent part.
— eom-cicd-roadmap
Yes. inspiron-3670 is a better fit than the VM I described. I checked it read-only today:
inspiron-3670
Needed
CPU
6 cores (i5-8400)
4, at least 2
Memory
62 GB, about 56 GB free
8 GB or more
Disk
100 GB free on /
about 100 GB
OS / Docker
Debian 13, Docker 26.1 with Compose 2.26
Debian 13
Three things to know first
CPU is already busy. A long-running background workload uses about 4.5 of the 6 cores. CI would share what's left, so builds would be slow. Recommendation: run that workload at the lowest priority (nice 19 or CPUWeight=1 in its service) so CI and OctoPrint get CPU first, and it takes whatever is left over.
Disk./ is on the spinning disk. It has room, but image pulls and builds are slower there. The 466 GB SSD appears to hold another OS's partitions. If that disk is free to use, Docker data on it would make builds much faster. I haven't touched it and won't plan on it without your say. By default the draft uses /var/lib/gitea-runner/docker on /.
The printers. OctoPrint streams G-code over USB, and long CPU stalls can cause print blemishes. The draft caps CI at 4 of 6 CPUs and 16 GB, and gives it a lower CPU weight than the host's other containers. Watch the first print that runs during a build.
What's drafted (committed, not applied)
Role: tasks/actions_host.yaml, templates/runner-compose.yaml.j2, templates/runner-config.yaml.j2 and gitea_actions_host_* defaults in DevOps/ansible-role-gitea@9f768ed and @c7874a4.
This repo: ci-runner.yaml, the submodule bump, and a section in docs/ci-cd/runner.md (4d6e43e, 5c72efc, c7874a4).
Design: a Compose project with its own Docker daemon (DinD) for jobs, separate from the host's Docker, so jobs can't see or touch OctoPrint or CNCjs. It also includes the runner and the S5 cleanup script. Capacity 1, the 2h job timeout and the pinned job images all come from the S4 variables. The token is fetched once from the cluster, used to register, and then deleted.
How it was tested
ansible-playbook --syntax-check, ansible-lint (no findings apart from the role's existing lowercase task names) and yamllint passed, run in a container.
I rendered the templates and checked them with docker compose config.
I ran the rendered dind and prune services on the workstation, with smaller CPU and memory caps. The daemon came up healthy with the overlayfs driver, the caps showed up in its cgroup, and a job-style container ran through it and reached gitea.eom.dev. The cleanup completed a pass. Everything was removed afterwards.
Not tested: the runner registering and running a real job, because that needs a registration token, and anything on inspiron-3670 itself.
To go ahead
Reply here with: go/no-go on inspiron-3670, what to do about the background workload (lower its priority, recommended; or leave it), and whether the SSD is free to use.
Then I'll adjust the variables, label this ready-to-deploy, and post the run notice. The command will be: ansible-playbook -i inventories/eom.dev.yml ci-runner.yaml -K
Effect: a second runner comes up on inspiron-3670 next to the in-cluster one. Both take jobs until the cluster runner is removed in a later step, so nothing running is interrupted.
Risk: builds compete with whatever else runs on that machine.
Rollback:docker compose -f /opt/gitea-runner/compose.yaml down on the host.
Network separation for this host belongs to I2 (#44). The details are in the private assessment.
— eom-cicd-roadmap
**Yes. inspiron-3670 is a better fit than the VM I described.** I checked it read-only today:
| | inspiron-3670 | Needed |
|---|---|---|
| CPU | 6 cores (i5-8400) | 4, at least 2 |
| Memory | 62 GB, about 56 GB free | 8 GB or more |
| Disk | 100 GB free on `/` | about 100 GB |
| OS / Docker | Debian 13, Docker 26.1 with Compose 2.26 | Debian 13 |
**Three things to know first**
1. **CPU is already busy.** A long-running background workload uses about 4.5 of the 6 cores. CI would share what's left, so builds would be slow. **Recommendation:** run that workload at the lowest priority (`nice 19` or `CPUWeight=1` in its service) so CI and OctoPrint get CPU first, and it takes whatever is left over.
2. **Disk.** `/` is on the spinning disk. It has room, but image pulls and builds are slower there. The 466 GB SSD appears to hold another OS's partitions. If that disk is free to use, Docker data on it would make builds much faster. I haven't touched it and won't plan on it without your say. By default the draft uses `/var/lib/gitea-runner/docker` on `/`.
3. **The printers.** OctoPrint streams G-code over USB, and long CPU stalls can cause print blemishes. The draft caps CI at 4 of 6 CPUs and 16 GB, and gives it a lower CPU weight than the host's other containers. Watch the first print that runs during a build.
**What's drafted (committed, not applied)**
- Role: `tasks/actions_host.yaml`, `templates/runner-compose.yaml.j2`, `templates/runner-config.yaml.j2` and `gitea_actions_host_*` defaults in DevOps/ansible-role-gitea@9f768ed and @c7874a4.
- This repo: `ci-runner.yaml`, the submodule bump, and a section in `docs/ci-cd/runner.md` (4d6e43e, 5c72efc, c7874a4).
- Design: a Compose project with **its own Docker daemon (DinD)** for jobs, separate from the host's Docker, so jobs can't see or touch OctoPrint or CNCjs. It also includes the runner and the S5 cleanup script. Capacity 1, the 2h job timeout and the pinned job images all come from the S4 variables. The token is fetched once from the cluster, used to register, and then deleted.
**How it was tested**
- `ansible-playbook --syntax-check`, `ansible-lint` (no findings apart from the role's existing lowercase task names) and `yamllint` passed, run in a container.
- I rendered the templates and checked them with `docker compose config`.
- I ran the rendered `dind` and `prune` services on the workstation, with smaller CPU and memory caps. The daemon came up healthy with the `overlayfs` driver, the caps showed up in its cgroup, and a job-style container ran through it and reached gitea.eom.dev. The cleanup completed a pass. Everything was removed afterwards.
- **Not tested:** the runner registering and running a real job, because that needs a registration token, and anything on inspiron-3670 itself.
**To go ahead**
1. Reply here with: go/no-go on inspiron-3670, what to do about the background workload (lower its priority, recommended; or leave it), and whether the SSD is free to use.
2. Then I'll adjust the variables, label this `ready-to-deploy`, and post the run notice. The command will be:
`ansible-playbook -i inventories/eom.dev.yml ci-runner.yaml -K`
- **Effect:** a second runner comes up on inspiron-3670 next to the in-cluster one. Both take jobs until the cluster runner is removed in a later step, so nothing running is interrupted.
- **Risk:** builds compete with whatever else runs on that machine.
- **Rollback:** `docker compose -f /opt/gitea-runner/compose.yaml down` on the host.
Network separation for this host belongs to I2 (#44). The details are in the private assessment.
— eom-cicd-roadmap
One more input for the decision: before choosing, please read the new "I3 follow-up: inspiron-3670" section in the private assessment. It covers this machine's other roles, including the USB-attached tools, and what that means for isolation. It may change which option you prefer.
— eom-cicd-roadmap
One more input for the decision: before choosing, please read the new "I3 follow-up: inspiron-3670" section in the private assessment. It covers this machine's other roles, including the USB-attached tools, and what that means for isolation. It may change which option you prefer.
— eom-cicd-roadmap
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Compare a dedicated build VM on one of the hypervisors, rootless DinD, and a daemonless builder for image builds. Cover isolation, whether existing workflows keep working, CPU and disk available, and maintenance cost.
Work (Claude): write the comparison and a recommendation in a comment, ideally with a small proof of concept. Keep sensitive details in the private assessment.
Decision (Eric): choose, since a VM needs host resources.
Acceptance: decision recorded, follow-up implementation issue opened.
Depends on: I1.
What is your recommendation? I have a vm for personal use that isn't really being used, but rootless seems more in keeping with my Kubernetes infrastructure. What do you recommend?
Recommendation: use the spare VM as the CI host, outside the cluster. Rootless in the cluster gains less than its name suggests, and the VM also settles the open node question in S3 (#36).
Comparison
dockerdocker buildWhat I tested
gitea/actions0.1.2 has astatefulset.dind.rootlessflag. I read the chart's template: the flag changes the user the Docker daemon runs as and where its socket is, and leaves the container's other security settings as they are.docker:29.5.2-dind-rootlesson the workstation three ways. It only started with the same container settings the chart uses today. Started that way, the daemon ran as an unprivileged user, used theoverlayfsdriver and ran a job container. This was a local test, not on the cluster.Details of the isolation side are in the private assessment.
Why the VM, even though rootless is more Kubernetes-native
What the VM needs: 4 vCPUs if possible (2 at least), 8 GB RAM or more, about 100 GB of disk, Debian 13, outbound HTTPS to
gitea.eom.devand the internet, and SSH from the Ansible control host.If you'd rather stay inside Kubernetes: join that VM to the cluster as a dedicated worker and label it
eom.dev/ci-runner=true. Everything already committed for S3 to S5 then deploys as it is. Disk and CPU are fixed, and separation stays as it is today.To decide
ready-to-deploy. The in-cluster runner is removed once the new one is serving jobs.The runner was evicted again today for the same disk reason as on 2026-11-09, so whichever you choose, #36 is the urgent part.
— eom-cicd-roadmap
The VM doesn't have the resources you described. What about the inspiron-3670? Could we use that instead?
Yes. inspiron-3670 is a better fit than the VM I described. I checked it read-only today:
/Three things to know first
nice 19orCPUWeight=1in its service) so CI and OctoPrint get CPU first, and it takes whatever is left over./is on the spinning disk. It has room, but image pulls and builds are slower there. The 466 GB SSD appears to hold another OS's partitions. If that disk is free to use, Docker data on it would make builds much faster. I haven't touched it and won't plan on it without your say. By default the draft uses/var/lib/gitea-runner/dockeron/.What's drafted (committed, not applied)
tasks/actions_host.yaml,templates/runner-compose.yaml.j2,templates/runner-config.yaml.j2andgitea_actions_host_*defaults in DevOps/ansible-role-gitea@9f768ed and @c7874a4.ci-runner.yaml, the submodule bump, and a section indocs/ci-cd/runner.md(4d6e43e,5c72efc,c7874a4).How it was tested
ansible-playbook --syntax-check,ansible-lint(no findings apart from the role's existing lowercase task names) andyamllintpassed, run in a container.docker compose config.dindandpruneservices on the workstation, with smaller CPU and memory caps. The daemon came up healthy with theoverlayfsdriver, the caps showed up in its cgroup, and a job-style container ran through it and reached gitea.eom.dev. The cleanup completed a pass. Everything was removed afterwards.To go ahead
ready-to-deploy, and post the run notice. The command will be:ansible-playbook -i inventories/eom.dev.yml ci-runner.yaml -Kdocker compose -f /opt/gitea-runner/compose.yaml downon the host.Network separation for this host belongs to I2 (#44). The details are in the private assessment.
— eom-cicd-roadmap
One more input for the decision: before choosing, please read the new "I3 follow-up: inspiron-3670" section in the private assessment. It covers this machine's other roles, including the USB-attached tools, and what that means for isolation. It may change which option you prefer.
— eom-cicd-roadmap