Files
software-infrastructure/docs/ci-cd/runner.md
T
claudeandClaude Opus 5.5 156cbeea60 docs: runner operations runbook (draft)
Add docs/ci-cd/runner.md: architecture of the shared Gitea Actions runner,
how to apply role changes, where to read logs, recovery steps for a stuck
queue, an evicted runner or a full Docker disk, and how to replace the
registration token without handling its value.

Describes the configuration committed for S3-S5; it becomes accurate for the
live runner once those are deployed. The S7 alerts already link here.

Refs DevOps/software-infrastructure#42

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:59:26 -05:00

200 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Gitea Actions runner: operations runbook
How the shared CI runner for gitea.eom.dev is built, how to change it, and how
to recover it. Tracked in DevOps/software-infrastructure#42 (S9).
> **Status:** this describes the runner as configured in
> `roles/ericomeehan.gitea` after S3–S5 (#36, #37, #38) and Gitea's Actions
> settings from S6 (#39). Until those are deployed, the live runner still has
> the older setup: no node label, capacity 2, no Docker cleanup.
This file is public. It names no hosts, addresses, or credentials. Look those
up in the inventory and vault.
## Architecture
```text
gitea (Deployment, ns gitea) <── polls for jobs ── gitea-actions-runner-0 (StatefulSet, ns gitea)
├─ init: reach-gitea waits until Gitea answers
├─ dind Docker daemon; job containers run here
├─ docker-prune (sidecar) cleans Docker data every 15 min (S5)
└─ runner act_runner; talks to dind over a socket
```
- **One runner for everyone.** It is registered at instance level, so every
repo and org on gitea.eom.dev shares it. Capacity is **1 job at a time**
(`gitea_actions_runner_capacity`).
- **Helm chart** `gitea/actions`, pinned to `gitea_actions_chart_version`
(0.1.2). Release name `gitea-actions`, namespace `gitea`.
- **Placement.** The pod only schedules on a node labeled
`eom.dev/ci-runner=true` (`gitea_actions_runner_node_selector`). The role
fails with a clear message if no node has the label.
- **Docker storage.** DinD's `/var/lib/docker` is an `emptyDir` on that node's
local disk, capped by `gitea_actions_docker_data_size` (50Gi). It is emptied
whenever the pod is recreated. NFS can't be used because overlay2 doesn't
support it.
- **Runner state.** `/data` in the `runner` container is a 1Gi PVC
(`data-runner-gitea-actions-runner-0`). It holds `.runner`, the runner's
registration. Deleting it makes the runner register again on the next start.
- **Registration token.** Kubernetes Secret `gitea-actions-runner-token` in
namespace `gitea`. The role creates it once with
`gitea actions generate-runner-token` and never prints it.
- **Job images.** `ubuntu-latest`, `ubuntu-24.04` and `ubuntu-22.04` map to
dated `docker.gitea.com/runner-images` tags (`gitea_actions_runner_labels`).
`force_pull` is off, so they stay cached in DinD and the cleanup never
removes them.
- **Limits.** DinD gets up to 2 CPU and 8Gi memory, and holds every job
container. A job over the memory limit is OOM-killed and fails with a step
error. The runner fails a job after 2h; Gitea fails one after 2h30m
(`ENDLESS_TASK_TIMEOUT`), marks one whose runner stops reporting as failed
after 5m (`ZOMBIE_TASK_TIMEOUT`), and cancels one nobody picks up within 12h
(`ABANDONED_JOB_TIMEOUT`).
- **Monitoring** (S7, #40, in `roles/ericomeehan.prometheus`, group `eom-ci`):
`GiteaRunnerNotReady`, `GiteaRunnerRestarted`, `GiteaRunnerPodReplaced`,
`GiteaRunnerEvicted`, `NodeDiskPressure`, `GiteaDown`.
All tunables are in `roles/ericomeehan.gitea/defaults/main.yml`, with comments.
## Applying a change
1. Change the role (`DevOps/ansible-role-gitea`), push it, then bump the
submodule here and push that.
2. **Check that no job is running**: Site Administration → Actions → Runners,
or each busy repo's Actions tab. Applying restarts the runner and **any
running job dies**. Post a heads-up in the forum's Agents category topic
"Gitea Actions runner: status and changes" if other agents' CI may be busy.
3. Apply only the runner tasks:
```sh
ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions
```
Add the vault option you normally use. Gitea itself isn't redeployed.
4. Confirm: `kubectl -n gitea get pod gitea-actions-runner-0` shows all
containers ready, the runner shows **Idle** in the admin Runners page, and a
small workflow run succeeds.
**Rollback:** `git revert` the submodule bump in this repo, then rerun the same
command.
## Reading logs
| What | Command |
|---|---|
| Pod state and recent events | `kubectl -n gitea describe pod gitea-actions-runner-0` |
| Runner (job pickup, job results) | `kubectl -n gitea logs gitea-actions-runner-0 -c runner` |
| Docker daemon | `kubectl -n gitea logs gitea-actions-runner-0 -c dind` |
| Cleanup passes and disk use | `kubectl -n gitea logs gitea-actions-runner-0 -c docker-prune` |
| Log from before a restart | add `--previous` to any of the above |
| Gitea's side (task state, zombie/endless timeouts) | `kubectl -n gitea logs deploy/gitea -c gitea \| grep -i action` |
| Docker disk use inside DinD | `kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker system df` |
| Free space on the Docker volume | `kubectl -n gitea exec gitea-actions-runner-0 -c dind -- df -h /var/lib/docker` |
Job logs themselves are in the web UI under each run. They are kept for 90
days, and artifacts for 30.
A job that **stops logging and later fails with no step error** means the
runner lost it: the pod was restarted, evicted or replaced, or Gitea's zombie
timeout fired. Check the alerts above and `describe pod` events before
blaming the job's code.
## Recovery
### Jobs queue but never start
1. Is the runner online? Admin → Actions → Runners. If it is **Offline**, go to
"Runner pod not running" below.
2. Does the job ask for a label the runner doesn't have (`runs-on: macos-*`,
`windows-*`, or a custom label)? It will wait until Gitea cancels it after
12h. Fix the workflow, or cancel the runs from the repo's Actions tab.
3. Is the queue just long? With capacity 1, one long job blocks everyone.
Find the run that is in progress. If it is stuck, cancel it in the UI.
Workflows should use `concurrency` with `cancel-in-progress: true` so
repeated pushes don't stack up.
4. A repo flooding the queue (for example workflows inherited from an upstream
`.github/workflows`): cancel its queued runs, then give it a
`.gitea/workflows/` directory, which makes Gitea ignore `.github/workflows`.
### Runner pod not running
`kubectl -n gitea get pod gitea-actions-runner-0 -o wide` and `describe`:
- **Pending, "didn't match node selector"**: no node carries
`eom.dev/ci-runner=true`, or that node is down. Label a suitable node (enough
local disk, see #36) with `kubectl label node <node> eom.dev/ci-runner=true`.
- **Pending, insufficient cpu/memory/ephemeral-storage**: the labeled node is
full. Free it up, or label a bigger node.
- **Init:0/1 (`reach-gitea`)**: Gitea isn't answering. Fix Gitea first
(`GiteaDown`).
- **CrashLoopBackOff in `runner`**: read `logs -c runner --previous`.
Registration errors mean the token or `.runner` is bad; see "Replacing the
registration token".
- **CrashLoopBackOff in `dind`**: usually a full or broken `/var/lib/docker`.
Deleting the pod gives it a fresh, empty `emptyDir`:
`kubectl -n gitea delete pod gitea-actions-runner-0` (the StatefulSet
recreates it; the first job re-pulls its image, about 5 GB).
### Runner evicted, or the disk is full
Signs: `GiteaRunnerEvicted` or `NodeDiskPressure`, events saying
`ephemeral local storage usage exceeds`, or jobs failing on `no space left on
device`.
1. Check usage: `docker system df` and `df -h /var/lib/docker` (table above),
and the `docker-prune` log for its last pass.
2. If the cleanup sidecar isn't running or can't keep up, free space by hand
while no job is running:
```sh
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker container prune -f
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker builder prune -af
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker image prune -af
```
The last command also removes the cached job images; the next job re-pulls.
3. If it keeps happening, a workflow is producing more data than the cap allows.
Find it from the runs that were active at the time, then either fix the
workflow or raise `gitea_actions_docker_data_size` and the ephemeral-storage
limits together (the node needs that much free local disk).
4. If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that
were lost, once.
### kubectl itself fails
If `kubectl` errors with a TLS or certificate message, the problem is the
cluster's control plane, not the runner. CI usually keeps working, because the
runner talks to Gitea directly. On kubeadm clusters, check certificate expiry on
the control plane with `kubeadm certs check-expiration` and renew with
`kubeadm certs renew`, following the Kubernetes documentation for restarting
the control-plane components afterwards.
## Replacing the registration token
Needed when the runner is deleted in the Gitea UI, the token is suspected
leaked, or the runner can't register. No token value ever needs to be copied by
hand.
1. Make sure no job is running.
2. In Site Administration → Actions → Runners, delete the old runner entry.
If the token may have leaked, also use **Create new runner → Reset
registration token** there. `gitea actions generate-runner-token` returns
the current active token rather than a new one, so without the reset the
old token keeps working for new registrations.
3. Delete the stored token so the role generates a fresh one:
`kubectl -n gitea delete secret gitea-actions-runner-token`
4. Remove the old registration so the runner registers again. Either delete
`/data/.runner` from the `runner` container, or delete the pod and its PVC
(`data-runner-gitea-actions-runner-0`); it holds nothing else.
5. Run the role: `ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions`.
It fetches the active token with `gitea actions generate-runner-token`,
stores it in the Secret, and the runner registers on start.
6. Confirm a single runner shows **Idle** in the admin page and a test workflow
succeeds.
## Related
- Roadmap: label `cicd` on DevOps/software-infrastructure, milestones
"CI/CD 0" to "CI/CD 4".
- Agents coordinate CI on the forum's Agents category, topic "Gitea Actions
runner: status and changes".