The runner's Docker-in-Docker data lands on the node's small disk. One runner image is about 5 GB, and the pod was evicted for low ephemeral storage on 2026-11-09, killing every running job.
Work
Schedule the runner on a node with real disk space, or give DinD dedicated storage that suits overlay2 (NFS does not).
Set an explicit ephemeral-storage request and limit, and size-limit the Docker data volume.
Check what the gitea/actions chart 0.1.2 exposes; if it can't do this, propose the smallest alternative.
Acceptance
Runner placement and storage are declared in the role and survive a redeploy.
Two concurrent large-image jobs run without node disk pressure.
Depends on: S2. Label ready-to-deploy when committed.
The runner's Docker-in-Docker data lands on the node's small disk. One runner image is about 5 GB, and the pod was evicted for low ephemeral storage on 2026-11-09, killing every running job.
**Work**
- Schedule the runner on a node with real disk space, or give DinD dedicated storage that suits overlay2 (NFS does not).
- Set an explicit `ephemeral-storage` request and limit, and size-limit the Docker data volume.
- Check what the `gitea/actions` chart 0.1.2 exposes; if it can't do this, propose the smallest alternative.
**Acceptance**
- Runner placement and storage are declared in the role and survive a redeploy.
- Two concurrent large-image jobs run without node disk pressure.
Depends on: S2. Label `ready-to-deploy` when committed.
claude
added this to the CI/CD 0: Stabilize the runner milestone 2026-11-09 17:05:56 +00:00
What chart gitea/actions 0.1.2 exposes:statefulset.nodeSelector, affinity, extraVolumes, dind.extraVolumeMounts and dind.resources. There's no built-in option for Docker storage, so the change mounts an extra volume at /var/lib/docker. Both cluster storage classes are NFS, which overlay2 doesn't support, so the volume is an emptyDir on the node's local disk.
What changed (all values are role defaults)
gitea_actions_runner_node_selector: {eom.dev/ci-runner: "true"}. The runner schedules only on a node with that label.
New tasks look the label up, and the deploy fails with a clear message if no node has it, instead of leaving a pod Pending.
docker-dataemptyDir with sizeLimit: 50Gi, mounted at /var/lib/docker in DinD.
DinD ephemeral storage: request 20Gi, limit 55Gi. The request also keeps the scheduler off nodes with small disks.
Note: chart 0.1.2 renders nodeSelector correctly for one key only, as commented in the role.
Tested: I rendered the role's Helm values through Ansible and ran helm template on chart 0.1.2 in an alpine/helm container. The StatefulSet gets the nodeSelector, the /var/lib/docker mount, the emptyDirsizeLimit and the ephemeral-storage resources. The playbook passes --syntax-check, and --list-tasks --tags actions shows the two new check tasks. Nothing was applied.
Decision needed: which node gets the label? Most workers have about 15 GB of local disk. The one node with a large disk has 1 CPU, and other services already request about 85% of it.
Grow the local disk of one 2-CPU worker to about 80 GB or more and label it (recommended). CI gets its own CPU headroom, which S4 (#37) needs for real CPU and memory requests.
Label the existing large-disk node now. It's the fastest fix for disk, but CI jobs would compete with the services already there, and S4's CPU request may not fit.
Wait for a dedicated build host (#45 / #78). That leaves the eviction risk in place until then.
Effect: the runner StatefulSet's pod template changes, so the pod is recreated on the labeled node. Gitea itself isn't touched.
Risk: the restart kills any running CI jobs, so check https://gitea.eom.dev/-/admin/actions/runners or the Actions pages first. Docker's cache starts empty, so the first job re-pulls the runner image (about 5 GB). If sizeLimit is exceeded, the kubelet evicts the pod. S5 (#38) adds the cleanup that keeps usage below the limit.
Rollback: in the superproject, git revert dc406a1, then rerun the same --tags actions command. Optionally remove the label with kubectl label node <chosen-node> eom.dev/ci-runner-.
The second acceptance criterion (two large-image jobs at once without disk pressure) can only be checked after deploy. I'll verify it then, together with S8.
— eom-cicd-roadmap
**Prepared, needs a node decision before deploy.** DevOps/ansible-role-gitea@e83262d862, bumped in dc406a1.
**What chart `gitea/actions` 0.1.2 exposes:** `statefulset.nodeSelector`, `affinity`, `extraVolumes`, `dind.extraVolumeMounts` and `dind.resources`. There's no built-in option for Docker storage, so the change mounts an extra volume at `/var/lib/docker`. Both cluster storage classes are NFS, which overlay2 doesn't support, so the volume is an `emptyDir` on the node's local disk.
**What changed (all values are role defaults)**
- `gitea_actions_runner_node_selector: {eom.dev/ci-runner: "true"}`. The runner schedules only on a node with that label.
- New tasks look the label up, and the deploy **fails with a clear message** if no node has it, instead of leaving a pod Pending.
- `docker-data` `emptyDir` with `sizeLimit: 50Gi`, mounted at `/var/lib/docker` in DinD.
- DinD ephemeral storage: request `20Gi`, limit `55Gi`. The request also keeps the scheduler off nodes with small disks.
- Note: chart 0.1.2 renders `nodeSelector` correctly for one key only, as commented in the role.
**Tested:** I rendered the role's Helm values through Ansible and ran `helm template` on chart 0.1.2 in an `alpine/helm` container. The StatefulSet gets the `nodeSelector`, the `/var/lib/docker` mount, the `emptyDir` `sizeLimit` and the ephemeral-storage resources. The playbook passes `--syntax-check`, and `--list-tasks --tags actions` shows the two new check tasks. Nothing was applied.
**Decision needed: which node gets the label?** Most workers have about 15 GB of local disk. The one node with a large disk has 1 CPU, and other services already request about 85% of it.
1. **Grow the local disk of one 2-CPU worker to about 80 GB or more and label it (recommended).** CI gets its own CPU headroom, which S4 (#37) needs for real CPU and memory requests.
2. Label the existing large-disk node now. It's the fastest fix for disk, but CI jobs would compete with the services already there, and S4's CPU request may not fit.
3. Wait for a dedicated build host (#45 / #78). That leaves the eviction risk in place until then.
**Deploy, once a node is chosen:**
```
kubectl label node <chosen-node> eom.dev/ci-runner=true
ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions
```
Add your usual vault option.
- **Effect:** the runner StatefulSet's pod template changes, so the pod is recreated on the labeled node. Gitea itself isn't touched.
- **Risk:** the restart **kills any running CI jobs**, so check https://gitea.eom.dev/-/admin/actions/runners or the Actions pages first. Docker's cache starts empty, so the first job re-pulls the runner image (about 5 GB). If `sizeLimit` is exceeded, the kubelet evicts the pod. S5 (#38) adds the cleanup that keeps usage below the limit.
- **Rollback:** in the superproject, `git revert dc406a1`, then rerun the same `--tags actions` command. Optionally remove the label with `kubectl label node <chosen-node> eom.dev/ci-runner-`.
The second acceptance criterion (two large-image jobs at once without disk pressure) can only be checked after deploy. I'll verify it then, together with S8.
— eom-cicd-roadmap
It happened again: the runner pod was evicted for low ephemeral storage at 15:28 UTC on 2026-11-12 and restarted on another small-disk worker. Any job running at that moment was lost. I checked the live pod read-only: it still has no node selector, no resource settings and no Docker data volume, so the changes for this issue, #37 and #38 aren't applied yet.
The node decision above is still what's needed. There's now a fourth way to settle it: my recommendation in #45 is to move the runner to your spare VM, which would give it its own disk and CPU. If you choose that, this issue is met by the VM and no worker needs a bigger disk. If the VM will take more than a few days, option 2 above (label the existing large-disk node) stops the evictions in the meantime with one label and one playbook run.
— eom-cicd-roadmap
**It happened again: the runner pod was evicted for low ephemeral storage at 15:28 UTC on 2026-11-12** and restarted on another small-disk worker. Any job running at that moment was lost. I checked the live pod read-only: it still has no node selector, no resource settings and no Docker data volume, so the changes for this issue, #37 and #38 aren't applied yet.
The node decision above is still what's needed. There's now a fourth way to settle it: my recommendation in #45 is to move the runner to your spare VM, which would give it its own disk and CPU. If you choose that, this issue is met by the VM and no worker needs a bigger disk. If the VM will take more than a few days, option 2 above (label the existing large-disk node) stops the evictions in the meantime with one label and one playbook run.
— eom-cicd-roadmap
Decision recorded: option 2. From the 2026-11-16 commits (DevOps/ansible-role-gitea@3974f60, bumped in 48d962b), the runner node is the existing large-disk worker, and the eom.dev/ci-runner=true label is already on it. I'm removing needs-eric. ready-to-deploy stays.
Checked today (read-only):
Exactly one node carries the label, so the role's "no labelled node" check will pass.
That node's current CPU requests are 855m of 1000m. The runner pod's three containers now request 110m in total, which comes to 965m, so the pod should schedule. Memory (2Gi DinD request against about 1.6Gi already requested of about 15.5Gi) and ephemeral storage (20Gi against about 127Gi allocatable) fit easily.
The live runner is still the old one: no node selector and no docker-prune sidecar, on a small-disk worker. So nothing from S3 to S5 is live yet.
Effect: the runner pod is recreated on the labelled node with S3 (Docker storage), S4 (resources, capacity 1, timeouts) and S5 (cleanup sidecar). Gitea itself isn't touched.
Risk: the restart kills any running CI jobs, so check the Actions pages first. The first job re-pulls the job image (about 5 GB) onto the new disk. Builds will be slower than before, because DinD is capped at 750m CPU on that node.
Rollback:git revert 48d962b (CPU sizing only), or revert the S3 to S5 bumps listed on #36 to #38, then rerun the same command.
If you'd like G8's cache (#56) in the same restart, it's being prepared now. I'll note on #56 whether it's ready, so the runner restarts only once.
S6 (#39) is separate: it's --tags gitea and restarts Gitea itself, so pick a quiet time for that one.
— eom-cicd-roadmap
**Decision recorded: option 2.** From the 2026-11-16 commits (DevOps/ansible-role-gitea@3974f60, bumped in 48d962b), the runner node is the existing large-disk worker, and the `eom.dev/ci-runner=true` label is already on it. I'm removing `needs-eric`. `ready-to-deploy` stays.
**Checked today (read-only):**
- Exactly one node carries the label, so the role's "no labelled node" check will pass.
- That node's current CPU requests are 855m of 1000m. The runner pod's three containers now request 110m in total, which comes to 965m, so the pod should schedule. Memory (2Gi DinD request against about 1.6Gi already requested of about 15.5Gi) and ephemeral storage (20Gi against about 127Gi allocatable) fit easily.
- The live runner is still the old one: no node selector and no `docker-prune` sidecar, on a small-disk worker. So nothing from S3 to S5 is live yet.
**The one remaining step:**
```
ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions
```
Add your usual vault option.
- **Effect:** the runner pod is recreated on the labelled node with S3 (Docker storage), S4 (resources, capacity 1, timeouts) and S5 (cleanup sidecar). Gitea itself isn't touched.
- **Risk:** the restart **kills any running CI jobs**, so check the Actions pages first. The first job re-pulls the job image (about 5 GB) onto the new disk. Builds will be slower than before, because DinD is capped at 750m CPU on that node.
- **Rollback:** `git revert 48d962b` (CPU sizing only), or revert the S3 to S5 bumps listed on #36 to #38, then rerun the same command.
If you'd like G8's cache (#56) in the same restart, it's being prepared now. I'll note on #56 whether it's ready, so the runner restarts only once.
S6 (#39) is separate: it's `--tags gitea` and restarts Gitea itself, so pick a quiet time for that one.
— eom-cicd-roadmap
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The runner's Docker-in-Docker data lands on the node's small disk. One runner image is about 5 GB, and the pod was evicted for low ephemeral storage on 2026-11-09, killing every running job.
Work
ephemeral-storagerequest and limit, and size-limit the Docker data volume.gitea/actionschart 0.1.2 exposes; if it can't do this, propose the smallest alternative.Acceptance
Depends on: S2. Label
ready-to-deploywhen committed.Prepared, needs a node decision before deploy. DevOps/ansible-role-gitea@e83262d862, bumped in
dc406a1.What chart
gitea/actions0.1.2 exposes:statefulset.nodeSelector,affinity,extraVolumes,dind.extraVolumeMountsanddind.resources. There's no built-in option for Docker storage, so the change mounts an extra volume at/var/lib/docker. Both cluster storage classes are NFS, which overlay2 doesn't support, so the volume is anemptyDiron the node's local disk.What changed (all values are role defaults)
gitea_actions_runner_node_selector: {eom.dev/ci-runner: "true"}. The runner schedules only on a node with that label.docker-dataemptyDirwithsizeLimit: 50Gi, mounted at/var/lib/dockerin DinD.20Gi, limit55Gi. The request also keeps the scheduler off nodes with small disks.nodeSelectorcorrectly for one key only, as commented in the role.Tested: I rendered the role's Helm values through Ansible and ran
helm templateon chart 0.1.2 in analpine/helmcontainer. The StatefulSet gets thenodeSelector, the/var/lib/dockermount, theemptyDirsizeLimitand the ephemeral-storage resources. The playbook passes--syntax-check, and--list-tasks --tags actionsshows the two new check tasks. Nothing was applied.Decision needed: which node gets the label? Most workers have about 15 GB of local disk. The one node with a large disk has 1 CPU, and other services already request about 85% of it.
Deploy, once a node is chosen:
Add your usual vault option.
sizeLimitis exceeded, the kubelet evicts the pod. S5 (#38) adds the cleanup that keeps usage below the limit.git revert dc406a1, then rerun the same--tags actionscommand. Optionally remove the label withkubectl label node <chosen-node> eom.dev/ci-runner-.The second acceptance criterion (two large-image jobs at once without disk pressure) can only be checked after deploy. I'll verify it then, together with S8.
— eom-cicd-roadmap
It happened again: the runner pod was evicted for low ephemeral storage at 15:28 UTC on 2026-11-12 and restarted on another small-disk worker. Any job running at that moment was lost. I checked the live pod read-only: it still has no node selector, no resource settings and no Docker data volume, so the changes for this issue, #37 and #38 aren't applied yet.
The node decision above is still what's needed. There's now a fourth way to settle it: my recommendation in #45 is to move the runner to your spare VM, which would give it its own disk and CPU. If you choose that, this issue is met by the VM and no worker needs a bigger disk. If the VM will take more than a few days, option 2 above (label the existing large-disk node) stops the evictions in the meantime with one label and one playbook run.
— eom-cicd-roadmap
Decision recorded: option 2. From the 2026-11-16 commits (DevOps/ansible-role-gitea@3974f60, bumped in
48d962b), the runner node is the existing large-disk worker, and theeom.dev/ci-runner=truelabel is already on it. I'm removingneeds-eric.ready-to-deploystays.Checked today (read-only):
docker-prunesidecar, on a small-disk worker. So nothing from S3 to S5 is live yet.The one remaining step:
Add your usual vault option.
git revert 48d962b(CPU sizing only), or revert the S3 to S5 bumps listed on #36 to #38, then rerun the same command.If you'd like G8's cache (#56) in the same restart, it's being prepared now. I'll note on #56 whether it's ready, so the runner restarts only once.
S6 (#39) is separate: it's
--tags giteaand restarts Gitea itself, so pick a quiet time for that one.— eom-cicd-roadmap