Adds docs/ci-cd/soak/soak.yaml (20 soak jobs plus a control failure, for a cicd-test repo) and docs/ci-cd/soak-test.md with preconditions, pass criteria and the first evidence lining up silent job failures with runner evictions. The runbook now says to cancel orphaned in-progress jobs before rerunning. Refs DevOps/software-infrastructure#41 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
12 KiB
Gitea Actions runner: operations runbook
How the shared CI runner for gitea.eom.dev is built, how to change it, and how to recover it. Tracked in DevOps/software-infrastructure#42 (S9).
Status: this describes the runner as configured in
roles/ericomeehan.giteaafter S3–S5 (#36, #37, #38) and Gitea's Actions settings from S6 (#39). Until those are deployed, the live runner still has the older setup: no node label, capacity 2, no Docker cleanup.
This file is public. It names no hosts, addresses, or credentials. Look those up in the inventory and vault.
Architecture
gitea (Deployment, ns gitea) <── polls for jobs ── gitea-actions-runner-0 (StatefulSet, ns gitea)
├─ init: reach-gitea waits until Gitea answers
├─ dind Docker daemon; job containers run here
├─ docker-prune (sidecar) cleans Docker data every 15 min (S5)
└─ runner act_runner; talks to dind over a socket
- One runner for everyone. It is registered at instance level, so every
repo and org on gitea.eom.dev shares it. Capacity is 1 job at a time
(
gitea_actions_runner_capacity). - Helm chart
gitea/actions, pinned togitea_actions_chart_version(0.1.2). Release namegitea-actions, namespacegitea. - Placement. The pod only schedules on a node labeled
eom.dev/ci-runner=true(gitea_actions_runner_node_selector). The role fails with a clear message if no node has the label. - Docker storage. DinD's
/var/lib/dockeris anemptyDiron that node's local disk, capped bygitea_actions_docker_data_size(50Gi). It is emptied whenever the pod is recreated. NFS can't be used because overlay2 doesn't support it. - Runner state.
/datain therunnercontainer is a 1Gi PVC (data-runner-gitea-actions-runner-0). It holds.runner, the runner's registration. Deleting it makes the runner register again on the next start. - Registration token. Kubernetes Secret
gitea-actions-runner-tokenin namespacegitea. The role creates it once withgitea actions generate-runner-tokenand never prints it. - Job images.
ubuntu-latest,ubuntu-24.04andubuntu-22.04map to dateddocker.gitea.com/runner-imagestags (gitea_actions_runner_labels).force_pullis off, so they stay cached in DinD and the cleanup never removes them. - Limits. DinD gets up to 2 CPU and 8Gi memory, and holds every job
container. A job over the memory limit is OOM-killed and fails with a step
error. The runner fails a job after 2h; Gitea fails one after 2h30m
(
ENDLESS_TASK_TIMEOUT), marks one whose runner stops reporting as failed after 5m (ZOMBIE_TASK_TIMEOUT), and cancels one nobody picks up within 12h (ABANDONED_JOB_TIMEOUT). - Monitoring (S7, #40, in
roles/ericomeehan.prometheus, groupeom-ci):GiteaRunnerNotReady,GiteaRunnerRestarted,GiteaRunnerPodReplaced,GiteaRunnerEvicted,NodeDiskPressure,GiteaDown.
All tunables are in roles/ericomeehan.gitea/defaults/main.yml, with comments.
Applying a change
-
Change the role (
DevOps/ansible-role-gitea), push it, then bump the submodule here and push that. -
Check that no job is running: Site Administration → Actions → Runners, or each busy repo's Actions tab. Applying restarts the runner and any running job dies. Post a heads-up in the forum's Agents category topic "Gitea Actions runner: status and changes" if other agents' CI may be busy.
-
Apply only the runner tasks:
ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actionsAdd the vault option you normally use. Gitea itself isn't redeployed.
-
Confirm:
kubectl -n gitea get pod gitea-actions-runner-0shows all containers ready, the runner shows Idle in the admin Runners page, and a small workflow run succeeds.
Rollback: git revert the submodule bump in this repo, then rerun the same
command.
Reading logs
| What | Command |
|---|---|
| Pod state and recent events | kubectl -n gitea describe pod gitea-actions-runner-0 |
| Runner (job pickup, job results) | kubectl -n gitea logs gitea-actions-runner-0 -c runner |
| Docker daemon | kubectl -n gitea logs gitea-actions-runner-0 -c dind |
| Cleanup passes and disk use | kubectl -n gitea logs gitea-actions-runner-0 -c docker-prune |
| Log from before a restart | add --previous to any of the above |
| Gitea's side (task state, zombie/endless timeouts) | kubectl -n gitea logs deploy/gitea -c gitea | grep -i action |
| Docker disk use inside DinD | kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker system df |
| Free space on the Docker volume | kubectl -n gitea exec gitea-actions-runner-0 -c dind -- df -h /var/lib/docker |
Job logs themselves are in the web UI under each run. They are kept for 90 days, and artifacts for 30.
A job that stops logging and later fails with no step error means the
runner lost it: the pod was restarted, evicted or replaced, or Gitea's zombie
timeout fired. Check the alerts above and describe pod events before
blaming the job's code.
Recovery
Jobs queue but never start
- Is the runner online? Admin → Actions → Runners. If it is Offline, go to "Runner pod not running" below.
- Does the job ask for a label the runner doesn't have (
runs-on: macos-*,windows-*, or a custom label)? It will wait until Gitea cancels it after 12h. Fix the workflow, or cancel the runs from the repo's Actions tab. - Is the queue just long? With capacity 1, one long job blocks everyone.
Find the run that is in progress. If it is stuck, cancel it in the UI.
Workflows should use
concurrencywithcancel-in-progress: trueso repeated pushes don't stack up. - A repo flooding the queue (for example workflows inherited from an upstream
.github/workflows): cancel its queued runs, then give it a.gitea/workflows/directory, which makes Gitea ignore.github/workflows.
Runner pod not running
kubectl -n gitea get pod gitea-actions-runner-0 -o wide and describe:
- Pending, "didn't match node selector": no node carries
eom.dev/ci-runner=true, or that node is down. Label a suitable node (enough local disk, see #36) withkubectl label node <node> eom.dev/ci-runner=true. - Pending, insufficient cpu/memory/ephemeral-storage: the labeled node is full. Free it up, or label a bigger node.
- Init:0/1 (
reach-gitea): Gitea isn't answering. Fix Gitea first (GiteaDown). - CrashLoopBackOff in
runner: readlogs -c runner --previous. Registration errors mean the token or.runneris bad; see "Replacing the registration token". - CrashLoopBackOff in
dind: usually a full or broken/var/lib/docker. Deleting the pod gives it a fresh, emptyemptyDir:kubectl -n gitea delete pod gitea-actions-runner-0(the StatefulSet recreates it; the first job re-pulls its image, about 5 GB).
Runner evicted, or the disk is full
Signs: GiteaRunnerEvicted or NodeDiskPressure, events saying
ephemeral local storage usage exceeds, or jobs failing on no space left on device.
-
Check usage:
docker system dfanddf -h /var/lib/docker(table above), and thedocker-prunelog for its last pass. -
If the cleanup sidecar isn't running or can't keep up, free space by hand while no job is running:
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker container prune -f kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker builder prune -af kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker image prune -afThe last command also removes the cached job images; the next job re-pulls.
-
If it keeps happening, a workflow is producing more data than the cap allows. Find it from the runs that were active at the time, then either fix the workflow or raise
gitea_actions_docker_data_sizeand the ephemeral-storage limits together (the node needs that much free local disk). -
If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that were lost, once. A job that was running at the time may stay "in progress" until Gitea's task timeout fails it; cancel it first. See soak-test.md for what this looks like in a job's log.
kubectl itself fails
If kubectl errors with a TLS or certificate message, the problem is the
cluster's control plane, not the runner. CI usually keeps working, because the
runner talks to Gitea directly. On kubeadm clusters, check certificate expiry on
the control plane with kubeadm certs check-expiration and renew with
kubeadm certs renew, following the Kubernetes documentation for restarting
the control-plane components afterwards.
Replacing the registration token
Needed when the runner is deleted in the Gitea UI, the token is suspected leaked, or the runner can't register. No token value ever needs to be copied by hand.
- Make sure no job is running.
- In Site Administration → Actions → Runners, delete the old runner entry.
If the token may have leaked, also use Create new runner → Reset
registration token there.
gitea actions generate-runner-tokenreturns the current active token rather than a new one, so without the reset the old token keeps working for new registrations. - Delete the stored token so the role generates a fresh one:
kubectl -n gitea delete secret gitea-actions-runner-token - Remove the old registration so the runner registers again. Either delete
/data/.runnerfrom therunnercontainer, or delete the pod and its PVC (data-runner-gitea-actions-runner-0); it holds nothing else. - Run the role:
ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions. It fetches the active token withgitea actions generate-runner-token, stores it in the Secret, and the runner registers on start. - Confirm a single runner shows Idle in the admin page and a test workflow succeeds.
Proposed: runner on a dedicated host
Not deployed. Proposed in DevOps/software-infrastructure#45 as the answer to
I3 and S3: run the runner on inspiron-3670 instead of in the cluster.
- Layout. Docker Compose project
gitea-runnerin/opt/gitea-runner:dind(a private Docker daemon that holds every job container, with data in/var/lib/gitea-runner/docker),runner, andprune(the same cleanup script as in the cluster). Jobs never see the host's own Docker daemon or its other containers. - Settings. Capacity, timeouts, job images and cleanup are the same
variables as the in-cluster runner. Host-only ones are
gitea_actions_host_*in the role'sdefaults/main.yml: 4 CPUs and 16 GB for jobs, and a CPU weight below the host's other containers. - Apply.
ansible-playbook -i inventories/eom.dev.yml ci-runner.yaml -K(-Kbecause the host asks for a sudo password). The first run fetches a registration token from the cluster, registers, and deletes the token file. Rerunning recreates only containers whose settings changed; that cancels a running job. - Logs. On the host:
docker compose -f /opt/gitea-runner/compose.yaml logs runner(ordind,prune). - Roll back.
docker compose -f /opt/gitea-runner/compose.yaml downon the host. The in-cluster runner keeps serving jobs until it is removed separately. - Note for jobs.
nprocinside a job reports all of the host's CPUs, not the 4-CPU cap, so builds that size-jfrom it oversubscribe.
Related
- Roadmap: label
cicdon DevOps/software-infrastructure, milestones "CI/CD 0" to "CI/CD 4". - S8 soak test and silent-failure evidence: soak-test.md.
- Agents coordinate CI on the forum's Agents category, topic "Gitea Actions runner: status and changes".