S3: Give the runner's Docker storage enough disk #36

Open
opened 2026-11-09 17:05:56 +00:00 by claude · 3 comments
Member

The runner's Docker-in-Docker data lands on the node's small disk. One runner image is about 5 GB, and the pod was evicted for low ephemeral storage on 2026-11-09, killing every running job.

Work

  • Schedule the runner on a node with real disk space, or give DinD dedicated storage that suits overlay2 (NFS does not).
  • Set an explicit ephemeral-storage request and limit, and size-limit the Docker data volume.
  • Check what the gitea/actions chart 0.1.2 exposes; if it can't do this, propose the smallest alternative.

Acceptance

  • Runner placement and storage are declared in the role and survive a redeploy.
  • Two concurrent large-image jobs run without node disk pressure.

Depends on: S2. Label ready-to-deploy when committed.

The runner's Docker-in-Docker data lands on the node's small disk. One runner image is about 5 GB, and the pod was evicted for low ephemeral storage on 2026-11-09, killing every running job. **Work** - Schedule the runner on a node with real disk space, or give DinD dedicated storage that suits overlay2 (NFS does not). - Set an explicit `ephemeral-storage` request and limit, and size-limit the Docker data volume. - Check what the `gitea/actions` chart 0.1.2 exposes; if it can't do this, propose the smallest alternative. **Acceptance** - Runner placement and storage are declared in the role and survive a redeploy. - Two concurrent large-image jobs run without node disk pressure. Depends on: S2. Label `ready-to-deploy` when committed.
claude added this to the CI/CD 0: Stabilize the runner milestone 2026-11-09 17:05:56 +00:00
claude added the
priority
critical
cicd
labels 2026-11-09 17:05:56 +00:00
claude added this to the CI/CD Roadmap project 2026-11-09 17:07:33 +00:00
claude moved this to To Do in CI/CD Roadmap on 2026-11-09 17:07:59 +00:00
Author
Member

Prepared, needs a node decision before deploy. DevOps/ansible-role-gitea@e83262d862, bumped in dc406a1.

What chart gitea/actions 0.1.2 exposes: statefulset.nodeSelector, affinity, extraVolumes, dind.extraVolumeMounts and dind.resources. There's no built-in option for Docker storage, so the change mounts an extra volume at /var/lib/docker. Both cluster storage classes are NFS, which overlay2 doesn't support, so the volume is an emptyDir on the node's local disk.

What changed (all values are role defaults)

  • gitea_actions_runner_node_selector: {eom.dev/ci-runner: "true"}. The runner schedules only on a node with that label.
  • New tasks look the label up, and the deploy fails with a clear message if no node has it, instead of leaving a pod Pending.
  • docker-data emptyDir with sizeLimit: 50Gi, mounted at /var/lib/docker in DinD.
  • DinD ephemeral storage: request 20Gi, limit 55Gi. The request also keeps the scheduler off nodes with small disks.
  • Note: chart 0.1.2 renders nodeSelector correctly for one key only, as commented in the role.

Tested: I rendered the role's Helm values through Ansible and ran helm template on chart 0.1.2 in an alpine/helm container. The StatefulSet gets the nodeSelector, the /var/lib/docker mount, the emptyDir sizeLimit and the ephemeral-storage resources. The playbook passes --syntax-check, and --list-tasks --tags actions shows the two new check tasks. Nothing was applied.

Decision needed: which node gets the label? Most workers have about 15 GB of local disk. The one node with a large disk has 1 CPU, and other services already request about 85% of it.

  1. Grow the local disk of one 2-CPU worker to about 80 GB or more and label it (recommended). CI gets its own CPU headroom, which S4 (#37) needs for real CPU and memory requests.
  2. Label the existing large-disk node now. It's the fastest fix for disk, but CI jobs would compete with the services already there, and S4's CPU request may not fit.
  3. Wait for a dedicated build host (#45 / #78). That leaves the eviction risk in place until then.

Deploy, once a node is chosen:

kubectl label node <chosen-node> eom.dev/ci-runner=true
ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions

Add your usual vault option.

  • Effect: the runner StatefulSet's pod template changes, so the pod is recreated on the labeled node. Gitea itself isn't touched.
  • Risk: the restart kills any running CI jobs, so check https://gitea.eom.dev/-/admin/actions/runners or the Actions pages first. Docker's cache starts empty, so the first job re-pulls the runner image (about 5 GB). If sizeLimit is exceeded, the kubelet evicts the pod. S5 (#38) adds the cleanup that keeps usage below the limit.
  • Rollback: in the superproject, git revert dc406a1, then rerun the same --tags actions command. Optionally remove the label with kubectl label node <chosen-node> eom.dev/ci-runner-.

The second acceptance criterion (two large-image jobs at once without disk pressure) can only be checked after deploy. I'll verify it then, together with S8.

— eom-cicd-roadmap

**Prepared, needs a node decision before deploy.** DevOps/ansible-role-gitea@e83262d862, bumped in dc406a1. **What chart `gitea/actions` 0.1.2 exposes:** `statefulset.nodeSelector`, `affinity`, `extraVolumes`, `dind.extraVolumeMounts` and `dind.resources`. There's no built-in option for Docker storage, so the change mounts an extra volume at `/var/lib/docker`. Both cluster storage classes are NFS, which overlay2 doesn't support, so the volume is an `emptyDir` on the node's local disk. **What changed (all values are role defaults)** - `gitea_actions_runner_node_selector: {eom.dev/ci-runner: "true"}`. The runner schedules only on a node with that label. - New tasks look the label up, and the deploy **fails with a clear message** if no node has it, instead of leaving a pod Pending. - `docker-data` `emptyDir` with `sizeLimit: 50Gi`, mounted at `/var/lib/docker` in DinD. - DinD ephemeral storage: request `20Gi`, limit `55Gi`. The request also keeps the scheduler off nodes with small disks. - Note: chart 0.1.2 renders `nodeSelector` correctly for one key only, as commented in the role. **Tested:** I rendered the role's Helm values through Ansible and ran `helm template` on chart 0.1.2 in an `alpine/helm` container. The StatefulSet gets the `nodeSelector`, the `/var/lib/docker` mount, the `emptyDir` `sizeLimit` and the ephemeral-storage resources. The playbook passes `--syntax-check`, and `--list-tasks --tags actions` shows the two new check tasks. Nothing was applied. **Decision needed: which node gets the label?** Most workers have about 15 GB of local disk. The one node with a large disk has 1 CPU, and other services already request about 85% of it. 1. **Grow the local disk of one 2-CPU worker to about 80 GB or more and label it (recommended).** CI gets its own CPU headroom, which S4 (#37) needs for real CPU and memory requests. 2. Label the existing large-disk node now. It's the fastest fix for disk, but CI jobs would compete with the services already there, and S4's CPU request may not fit. 3. Wait for a dedicated build host (#45 / #78). That leaves the eviction risk in place until then. **Deploy, once a node is chosen:** ``` kubectl label node <chosen-node> eom.dev/ci-runner=true ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions ``` Add your usual vault option. - **Effect:** the runner StatefulSet's pod template changes, so the pod is recreated on the labeled node. Gitea itself isn't touched. - **Risk:** the restart **kills any running CI jobs**, so check https://gitea.eom.dev/-/admin/actions/runners or the Actions pages first. Docker's cache starts empty, so the first job re-pulls the runner image (about 5 GB). If `sizeLimit` is exceeded, the kubelet evicts the pod. S5 (#38) adds the cleanup that keeps usage below the limit. - **Rollback:** in the superproject, `git revert dc406a1`, then rerun the same `--tags actions` command. Optionally remove the label with `kubectl label node <chosen-node> eom.dev/ci-runner-`. The second acceptance criterion (two large-image jobs at once without disk pressure) can only be checked after deploy. I'll verify it then, together with S8. — eom-cicd-roadmap
claude added the needs-ericready-to-deploy labels 2026-11-10 01:04:19 +00:00
Author
Member

It happened again: the runner pod was evicted for low ephemeral storage at 15:28 UTC on 2026-11-12 and restarted on another small-disk worker. Any job running at that moment was lost. I checked the live pod read-only: it still has no node selector, no resource settings and no Docker data volume, so the changes for this issue, #37 and #38 aren't applied yet.

The node decision above is still what's needed. There's now a fourth way to settle it: my recommendation in #45 is to move the runner to your spare VM, which would give it its own disk and CPU. If you choose that, this issue is met by the VM and no worker needs a bigger disk. If the VM will take more than a few days, option 2 above (label the existing large-disk node) stops the evictions in the meantime with one label and one playbook run.

— eom-cicd-roadmap

**It happened again: the runner pod was evicted for low ephemeral storage at 15:28 UTC on 2026-11-12** and restarted on another small-disk worker. Any job running at that moment was lost. I checked the live pod read-only: it still has no node selector, no resource settings and no Docker data volume, so the changes for this issue, #37 and #38 aren't applied yet. The node decision above is still what's needed. There's now a fourth way to settle it: my recommendation in #45 is to move the runner to your spare VM, which would give it its own disk and CPU. If you choose that, this issue is met by the VM and no worker needs a bigger disk. If the VM will take more than a few days, option 2 above (label the existing large-disk node) stops the evictions in the meantime with one label and one playbook run. — eom-cicd-roadmap
Author
Member

Decision recorded: option 2. From the 2026-11-16 commits (DevOps/ansible-role-gitea@3974f60, bumped in 48d962b), the runner node is the existing large-disk worker, and the eom.dev/ci-runner=true label is already on it. I'm removing needs-eric. ready-to-deploy stays.

Checked today (read-only):

  • Exactly one node carries the label, so the role's "no labelled node" check will pass.
  • That node's current CPU requests are 855m of 1000m. The runner pod's three containers now request 110m in total, which comes to 965m, so the pod should schedule. Memory (2Gi DinD request against about 1.6Gi already requested of about 15.5Gi) and ephemeral storage (20Gi against about 127Gi allocatable) fit easily.
  • The live runner is still the old one: no node selector and no docker-prune sidecar, on a small-disk worker. So nothing from S3 to S5 is live yet.

The one remaining step:

ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions

Add your usual vault option.

  • Effect: the runner pod is recreated on the labelled node with S3 (Docker storage), S4 (resources, capacity 1, timeouts) and S5 (cleanup sidecar). Gitea itself isn't touched.
  • Risk: the restart kills any running CI jobs, so check the Actions pages first. The first job re-pulls the job image (about 5 GB) onto the new disk. Builds will be slower than before, because DinD is capped at 750m CPU on that node.
  • Rollback: git revert 48d962b (CPU sizing only), or revert the S3 to S5 bumps listed on #36 to #38, then rerun the same command.

If you'd like G8's cache (#56) in the same restart, it's being prepared now. I'll note on #56 whether it's ready, so the runner restarts only once.

S6 (#39) is separate: it's --tags gitea and restarts Gitea itself, so pick a quiet time for that one.

— eom-cicd-roadmap

**Decision recorded: option 2.** From the 2026-11-16 commits (DevOps/ansible-role-gitea@3974f60, bumped in 48d962b), the runner node is the existing large-disk worker, and the `eom.dev/ci-runner=true` label is already on it. I'm removing `needs-eric`. `ready-to-deploy` stays. **Checked today (read-only):** - Exactly one node carries the label, so the role's "no labelled node" check will pass. - That node's current CPU requests are 855m of 1000m. The runner pod's three containers now request 110m in total, which comes to 965m, so the pod should schedule. Memory (2Gi DinD request against about 1.6Gi already requested of about 15.5Gi) and ephemeral storage (20Gi against about 127Gi allocatable) fit easily. - The live runner is still the old one: no node selector and no `docker-prune` sidecar, on a small-disk worker. So nothing from S3 to S5 is live yet. **The one remaining step:** ``` ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions ``` Add your usual vault option. - **Effect:** the runner pod is recreated on the labelled node with S3 (Docker storage), S4 (resources, capacity 1, timeouts) and S5 (cleanup sidecar). Gitea itself isn't touched. - **Risk:** the restart **kills any running CI jobs**, so check the Actions pages first. The first job re-pulls the job image (about 5 GB) onto the new disk. Builds will be slower than before, because DinD is capped at 750m CPU on that node. - **Rollback:** `git revert 48d962b` (CPU sizing only), or revert the S3 to S5 bumps listed on #36 to #38, then rerun the same command. If you'd like G8's cache (#56) in the same restart, it's being prepared now. I'll note on #56 whether it's ready, so the runner restarts only once. S6 (#39) is separate: it's `--tags gitea` and restarts Gitea itself, so pick a quiet time for that one. — eom-cicd-roadmap
claude removed the needs-eric label 2026-11-17 14:14:42 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DevOps/software-infrastructure#36