Nothing prunes stopped containers, unused volumes, or old images inside DinD, so disk use only grows until the pod is evicted.
Work
Add a scheduled prune, by sidecar or a loop in the deployment, that removes containers and volumes older than a few hours and images beyond a size threshold, without removing the current runner images.
Acceptance
Disk use returns to a known baseline after a batch of jobs.
A running job's containers are never pruned (tested).
Depends on: S3. Label ready-to-deploy when committed.
Nothing prunes stopped containers, unused volumes, or old images inside DinD, so disk use only grows until the pod is evicted.
**Work**
- Add a scheduled prune, by sidecar or a loop in the deployment, that removes containers and volumes older than a few hours and images beyond a size threshold, without removing the current runner images.
**Acceptance**
- Disk use returns to a known baseline after a batch of jobs.
- A running job's containers are never pruned (tested).
Depends on: S3. Label `ready-to-deploy` when committed.
claude
added this to the CI/CD 0: Stabilize the runner milestone 2026-11-09 17:05:57 +00:00
I prepared this while S3 waits for the node decision. The prune doesn't depend on which node is chosen, and S3's deploy notes count on it to keep Docker below the 50Gi docker-data cap. Deploying all three at once means one runner restart instead of two.
What changed
New files/docker-prune.sh, run by a docker-prune native sidecar (chart postExtraInitContainers, restartPolicy: Always) placed after DinD. It starts once DinD's socket is up, talks to it through the shared docker-socket volume, and stops before DinD on shutdown. Image docker:29.5.2-cli, which matches the deployed docker:29.5.2-dind. Resources: 10m / 32Mi requested, 200m / 128Mi limit.
Every 15 minutes it:
removes stopped containers older than 6 h (container prune --filter until=). Docker never prunes running containers.
removes volumes that no container references and that are older than 6 h. Docker also refuses to remove an in-use volume, so a race fails safe.
removes dangling images and build cache older than 6 h.
if Docker uses more than 30 GB (by docker system df), removes every image no container uses, except the pinned job images. Those come from gitea_actions_runner_labels, so they stay cached and jobs don't re-pull about 5 GB each. It uses a plain docker image rm, which fails for any image a running or stopped container uses.
logs what it removed and the size before and after: kubectl -n gitea logs gitea-actions-runner-0 -c docker-prune.
Settings are role defaults: gitea_actions_prune_interval_seconds (900), _max_age_minutes (360), _images_above_gb (30), _keep_images (derived from the labels), _image. Each has a comment.
Tested (nothing applied)
shellcheck -s sh: clean.
Functional test against a throwaway docker:29.5.2-dind, one pass with the age set to 1 minute, the image threshold set to 0 GB and a stand-in "job image" kept:
Object
Expected
Result
running container, older than the age limit, with a named volume (a running job)
kept, with its volume
kept
stopped container older than the age limit
removed
removed
stopped container newer than the age limit
kept
kept
unused volume older / newer than the age limit
removed / kept
removed / kept
unused image in the keep list
kept
kept
image used by the running container
kept
kept
image used by the new stopped container
kept
kept
unused image not in the keep list
removed
removed
The size parser agreed byte for byte with docker system df.
The role's Helm values went through Ansible (ansible-core 2.18, stub secrets), then helm template on chart 0.1.2. The StatefulSet's init containers are reach-gitea, dind, docker-prune. The env, mount and resources are as above, and the script in the rendered pod spec is byte-identical to files/docker-prune.sh, so Jinja didn't touch the Go-template {{.Size}} strings.
gitea.yaml --syntax-check passes. --list-tasks --tags actions is unchanged.
kubeconform -strict flags only serviceName missing on the StatefulSet. That comes from chart 0.1.2's own template, is the same before this change, and the live StatefulSet runs with it.
Deploy. Same command as S3/S4, once the node is labeled:
Effect: the runner pod is recreated with the S3/S4 changes and the new sidecar. Gitea itself isn't touched.
Risk: the restart kills any running CI jobs, so check the Actions pages first. The sidecar's first pass runs right away. On a fresh emptyDir there's nothing to remove. If the sidecar misbehaves, jobs aren't affected, because it only deletes things no container uses.
Rollback:git revert 160c82b in the superproject and rerun the same command. To keep S3/S4 and only switch off cleanup, set gitea_actions_prune_interval_seconds very high instead.
Still to verify after deploy: the first acceptance criterion, that disk use returns to a known baseline after a batch of jobs. I'll record the baseline from the sidecar's log and check it during the S8 (#41) soak.
— eom-cicd-roadmap
**Committed, ready to deploy together with S3 (#36) and S4 (#37).** DevOps/ansible-role-gitea@4d7b0b9e01, bumped in 160c82b.
I prepared this while S3 waits for the node decision. The prune doesn't depend on which node is chosen, and S3's deploy notes count on it to keep Docker below the 50Gi `docker-data` cap. Deploying all three at once means one runner restart instead of two.
**What changed**
- New `files/docker-prune.sh`, run by a **`docker-prune` native sidecar** (chart `postExtraInitContainers`, `restartPolicy: Always`) placed after DinD. It starts once DinD's socket is up, talks to it through the shared `docker-socket` volume, and stops before DinD on shutdown. Image `docker:29.5.2-cli`, which matches the deployed `docker:29.5.2-dind`. Resources: 10m / 32Mi requested, 200m / 128Mi limit.
- Every 15 minutes it:
1. removes **stopped** containers older than 6 h (`container prune --filter until=`). Docker never prunes running containers.
2. removes volumes that **no container references** and that are older than 6 h. Docker also refuses to remove an in-use volume, so a race fails safe.
3. removes dangling images and build cache older than 6 h.
4. if Docker uses **more than 30 GB** (by `docker system df`), removes every image no container uses, **except the pinned job images**. Those come from `gitea_actions_runner_labels`, so they stay cached and jobs don't re-pull about 5 GB each. It uses a plain `docker image rm`, which fails for any image a running or stopped container uses.
5. logs what it removed and the size before and after: `kubectl -n gitea logs gitea-actions-runner-0 -c docker-prune`.
- Settings are role defaults: `gitea_actions_prune_interval_seconds` (900), `_max_age_minutes` (360), `_images_above_gb` (30), `_keep_images` (derived from the labels), `_image`. Each has a comment.
**Tested (nothing applied)**
- `shellcheck -s sh`: clean.
- **Functional test against a throwaway `docker:29.5.2-dind`**, one pass with the age set to 1 minute, the image threshold set to 0 GB and a stand-in "job image" kept:
| Object | Expected | Result |
|---|---|---|
| running container, older than the age limit, with a named volume (a running job) | kept, with its volume | kept |
| stopped container older than the age limit | removed | removed |
| stopped container newer than the age limit | kept | kept |
| unused volume older / newer than the age limit | removed / kept | removed / kept |
| unused image in the keep list | kept | kept |
| image used by the running container | kept | kept |
| image used by the new stopped container | kept | kept |
| unused image not in the keep list | removed | removed |
The size parser agreed byte for byte with `docker system df`.
- The role's Helm values went through Ansible (ansible-core 2.18, stub secrets), then `helm template` on chart 0.1.2. The StatefulSet's init containers are `reach-gitea`, `dind`, `docker-prune`. The env, mount and resources are as above, and the script in the rendered pod spec is byte-identical to `files/docker-prune.sh`, so Jinja didn't touch the Go-template `{{.Size}}` strings.
- `gitea.yaml --syntax-check` passes. `--list-tasks --tags actions` is unchanged.
- `kubeconform -strict` flags only `serviceName` missing on the StatefulSet. That comes from chart 0.1.2's own template, is the same before this change, and the live StatefulSet runs with it.
**Deploy.** Same command as S3/S4, once the node is labeled:
```
kubectl label node <chosen-node> eom.dev/ci-runner=true
ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions
```
Add your usual vault option.
- **Effect:** the runner pod is recreated with the S3/S4 changes and the new sidecar. Gitea itself isn't touched.
- **Risk:** the restart **kills any running CI jobs**, so check the Actions pages first. The sidecar's first pass runs right away. On a fresh `emptyDir` there's nothing to remove. If the sidecar misbehaves, jobs aren't affected, because it only deletes things no container uses.
- **Rollback:** `git revert 160c82b` in the superproject and rerun the same command. To keep S3/S4 and only switch off cleanup, set `gitea_actions_prune_interval_seconds` very high instead.
**Still to verify after deploy:** the first acceptance criterion, that disk use returns to a known baseline after a batch of jobs. I'll record the baseline from the sidecar's log and check it during the S8 (#41) soak.
— eom-cicd-roadmap
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Nothing prunes stopped containers, unused volumes, or old images inside DinD, so disk use only grows until the pod is evicted.
Work
Acceptance
Depends on: S3. Label
ready-to-deploywhen committed.Committed, ready to deploy together with S3 (#36) and S4 (#37). DevOps/ansible-role-gitea@4d7b0b9e01, bumped in
160c82b.I prepared this while S3 waits for the node decision. The prune doesn't depend on which node is chosen, and S3's deploy notes count on it to keep Docker below the 50Gi
docker-datacap. Deploying all three at once means one runner restart instead of two.What changed
files/docker-prune.sh, run by adocker-prunenative sidecar (chartpostExtraInitContainers,restartPolicy: Always) placed after DinD. It starts once DinD's socket is up, talks to it through the shareddocker-socketvolume, and stops before DinD on shutdown. Imagedocker:29.5.2-cli, which matches the deployeddocker:29.5.2-dind. Resources: 10m / 32Mi requested, 200m / 128Mi limit.container prune --filter until=). Docker never prunes running containers.docker system df), removes every image no container uses, except the pinned job images. Those come fromgitea_actions_runner_labels, so they stay cached and jobs don't re-pull about 5 GB each. It uses a plaindocker image rm, which fails for any image a running or stopped container uses.kubectl -n gitea logs gitea-actions-runner-0 -c docker-prune.gitea_actions_prune_interval_seconds(900),_max_age_minutes(360),_images_above_gb(30),_keep_images(derived from the labels),_image. Each has a comment.Tested (nothing applied)
shellcheck -s sh: clean.docker:29.5.2-dind, one pass with the age set to 1 minute, the image threshold set to 0 GB and a stand-in "job image" kept:The size parser agreed byte for byte with
docker system df.helm templateon chart 0.1.2. The StatefulSet's init containers arereach-gitea,dind,docker-prune. The env, mount and resources are as above, and the script in the rendered pod spec is byte-identical tofiles/docker-prune.sh, so Jinja didn't touch the Go-template{{.Size}}strings.gitea.yaml --syntax-checkpasses.--list-tasks --tags actionsis unchanged.kubeconform -strictflags onlyserviceNamemissing on the StatefulSet. That comes from chart 0.1.2's own template, is the same before this change, and the live StatefulSet runs with it.Deploy. Same command as S3/S4, once the node is labeled:
Add your usual vault option.
emptyDirthere's nothing to remove. If the sidecar misbehaves, jobs aren't affected, because it only deletes things no container uses.git revert 160c82bin the superproject and rerun the same command. To keep S3/S4 and only switch off cleanup, setgitea_actions_prune_interval_secondsvery high instead.Still to verify after deploy: the first acceptance criterion, that disk use returns to a known baseline after a batch of jobs. I'll record the baseline from the sidecar's log and check it during the S8 (#41) soak.
— eom-cicd-roadmap