From 156cbeea604abdd2edd16db345fc7bf098347a50 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 19:59:26 -0500 Subject: [PATCH] docs: runner operations runbook (draft) Add docs/ci-cd/runner.md: architecture of the shared Gitea Actions runner, how to apply role changes, where to read logs, recovery steps for a stuck queue, an evicted runner or a full Docker disk, and how to replace the registration token without handling its value. Describes the configuration committed for S3-S5; it becomes accurate for the live runner once those are deployed. The S7 alerts already link here. Refs DevOps/software-infrastructure#42 Co-Authored-By: Claude Opus 5.5 --- docs/ci-cd/runner.md | 199 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 docs/ci-cd/runner.md diff --git a/docs/ci-cd/runner.md b/docs/ci-cd/runner.md new file mode 100644 index 0000000..42ded4a --- /dev/null +++ b/docs/ci-cd/runner.md @@ -0,0 +1,199 @@ +# Gitea Actions runner: operations runbook + +How the shared CI runner for gitea.eom.dev is built, how to change it, and how +to recover it. Tracked in DevOps/software-infrastructure#42 (S9). + +> **Status:** this describes the runner as configured in +> `roles/ericomeehan.gitea` after S3–S5 (#36, #37, #38) and Gitea's Actions +> settings from S6 (#39). Until those are deployed, the live runner still has +> the older setup: no node label, capacity 2, no Docker cleanup. + +This file is public. It names no hosts, addresses, or credentials. Look those +up in the inventory and vault. + +## Architecture + +```text +gitea (Deployment, ns gitea) <── polls for jobs ── gitea-actions-runner-0 (StatefulSet, ns gitea) + ├─ init: reach-gitea waits until Gitea answers + ├─ dind Docker daemon; job containers run here + ├─ docker-prune (sidecar) cleans Docker data every 15 min (S5) + └─ runner act_runner; talks to dind over a socket +``` + +- **One runner for everyone.** It is registered at instance level, so every + repo and org on gitea.eom.dev shares it. Capacity is **1 job at a time** + (`gitea_actions_runner_capacity`). +- **Helm chart** `gitea/actions`, pinned to `gitea_actions_chart_version` + (0.1.2). Release name `gitea-actions`, namespace `gitea`. +- **Placement.** The pod only schedules on a node labeled + `eom.dev/ci-runner=true` (`gitea_actions_runner_node_selector`). The role + fails with a clear message if no node has the label. +- **Docker storage.** DinD's `/var/lib/docker` is an `emptyDir` on that node's + local disk, capped by `gitea_actions_docker_data_size` (50Gi). It is emptied + whenever the pod is recreated. NFS can't be used because overlay2 doesn't + support it. +- **Runner state.** `/data` in the `runner` container is a 1Gi PVC + (`data-runner-gitea-actions-runner-0`). It holds `.runner`, the runner's + registration. Deleting it makes the runner register again on the next start. +- **Registration token.** Kubernetes Secret `gitea-actions-runner-token` in + namespace `gitea`. The role creates it once with + `gitea actions generate-runner-token` and never prints it. +- **Job images.** `ubuntu-latest`, `ubuntu-24.04` and `ubuntu-22.04` map to + dated `docker.gitea.com/runner-images` tags (`gitea_actions_runner_labels`). + `force_pull` is off, so they stay cached in DinD and the cleanup never + removes them. +- **Limits.** DinD gets up to 2 CPU and 8Gi memory, and holds every job + container. A job over the memory limit is OOM-killed and fails with a step + error. The runner fails a job after 2h; Gitea fails one after 2h30m + (`ENDLESS_TASK_TIMEOUT`), marks one whose runner stops reporting as failed + after 5m (`ZOMBIE_TASK_TIMEOUT`), and cancels one nobody picks up within 12h + (`ABANDONED_JOB_TIMEOUT`). +- **Monitoring** (S7, #40, in `roles/ericomeehan.prometheus`, group `eom-ci`): + `GiteaRunnerNotReady`, `GiteaRunnerRestarted`, `GiteaRunnerPodReplaced`, + `GiteaRunnerEvicted`, `NodeDiskPressure`, `GiteaDown`. + +All tunables are in `roles/ericomeehan.gitea/defaults/main.yml`, with comments. + +## Applying a change + +1. Change the role (`DevOps/ansible-role-gitea`), push it, then bump the + submodule here and push that. +2. **Check that no job is running**: Site Administration → Actions → Runners, + or each busy repo's Actions tab. Applying restarts the runner and **any + running job dies**. Post a heads-up in the forum's Agents category topic + "Gitea Actions runner: status and changes" if other agents' CI may be busy. +3. Apply only the runner tasks: + + ```sh + ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions + ``` + + Add the vault option you normally use. Gitea itself isn't redeployed. +4. Confirm: `kubectl -n gitea get pod gitea-actions-runner-0` shows all + containers ready, the runner shows **Idle** in the admin Runners page, and a + small workflow run succeeds. + +**Rollback:** `git revert` the submodule bump in this repo, then rerun the same +command. + +## Reading logs + +| What | Command | +|---|---| +| Pod state and recent events | `kubectl -n gitea describe pod gitea-actions-runner-0` | +| Runner (job pickup, job results) | `kubectl -n gitea logs gitea-actions-runner-0 -c runner` | +| Docker daemon | `kubectl -n gitea logs gitea-actions-runner-0 -c dind` | +| Cleanup passes and disk use | `kubectl -n gitea logs gitea-actions-runner-0 -c docker-prune` | +| Log from before a restart | add `--previous` to any of the above | +| Gitea's side (task state, zombie/endless timeouts) | `kubectl -n gitea logs deploy/gitea -c gitea \| grep -i action` | +| Docker disk use inside DinD | `kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker system df` | +| Free space on the Docker volume | `kubectl -n gitea exec gitea-actions-runner-0 -c dind -- df -h /var/lib/docker` | + +Job logs themselves are in the web UI under each run. They are kept for 90 +days, and artifacts for 30. + +A job that **stops logging and later fails with no step error** means the +runner lost it: the pod was restarted, evicted or replaced, or Gitea's zombie +timeout fired. Check the alerts above and `describe pod` events before +blaming the job's code. + +## Recovery + +### Jobs queue but never start + +1. Is the runner online? Admin → Actions → Runners. If it is **Offline**, go to + "Runner pod not running" below. +2. Does the job ask for a label the runner doesn't have (`runs-on: macos-*`, + `windows-*`, or a custom label)? It will wait until Gitea cancels it after + 12h. Fix the workflow, or cancel the runs from the repo's Actions tab. +3. Is the queue just long? With capacity 1, one long job blocks everyone. + Find the run that is in progress. If it is stuck, cancel it in the UI. + Workflows should use `concurrency` with `cancel-in-progress: true` so + repeated pushes don't stack up. +4. A repo flooding the queue (for example workflows inherited from an upstream + `.github/workflows`): cancel its queued runs, then give it a + `.gitea/workflows/` directory, which makes Gitea ignore `.github/workflows`. + +### Runner pod not running + +`kubectl -n gitea get pod gitea-actions-runner-0 -o wide` and `describe`: + +- **Pending, "didn't match node selector"**: no node carries + `eom.dev/ci-runner=true`, or that node is down. Label a suitable node (enough + local disk, see #36) with `kubectl label node eom.dev/ci-runner=true`. +- **Pending, insufficient cpu/memory/ephemeral-storage**: the labeled node is + full. Free it up, or label a bigger node. +- **Init:0/1 (`reach-gitea`)**: Gitea isn't answering. Fix Gitea first + (`GiteaDown`). +- **CrashLoopBackOff in `runner`**: read `logs -c runner --previous`. + Registration errors mean the token or `.runner` is bad; see "Replacing the + registration token". +- **CrashLoopBackOff in `dind`**: usually a full or broken `/var/lib/docker`. + Deleting the pod gives it a fresh, empty `emptyDir`: + `kubectl -n gitea delete pod gitea-actions-runner-0` (the StatefulSet + recreates it; the first job re-pulls its image, about 5 GB). + +### Runner evicted, or the disk is full + +Signs: `GiteaRunnerEvicted` or `NodeDiskPressure`, events saying +`ephemeral local storage usage exceeds`, or jobs failing on `no space left on +device`. + +1. Check usage: `docker system df` and `df -h /var/lib/docker` (table above), + and the `docker-prune` log for its last pass. +2. If the cleanup sidecar isn't running or can't keep up, free space by hand + while no job is running: + + ```sh + kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker container prune -f + kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker builder prune -af + kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker image prune -af + ``` + + The last command also removes the cached job images; the next job re-pulls. +3. If it keeps happening, a workflow is producing more data than the cap allows. + Find it from the runs that were active at the time, then either fix the + workflow or raise `gitea_actions_docker_data_size` and the ephemeral-storage + limits together (the node needs that much free local disk). +4. If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that + were lost, once. + +### kubectl itself fails + +If `kubectl` errors with a TLS or certificate message, the problem is the +cluster's control plane, not the runner. CI usually keeps working, because the +runner talks to Gitea directly. On kubeadm clusters, check certificate expiry on +the control plane with `kubeadm certs check-expiration` and renew with +`kubeadm certs renew`, following the Kubernetes documentation for restarting +the control-plane components afterwards. + +## Replacing the registration token + +Needed when the runner is deleted in the Gitea UI, the token is suspected +leaked, or the runner can't register. No token value ever needs to be copied by +hand. + +1. Make sure no job is running. +2. In Site Administration → Actions → Runners, delete the old runner entry. + If the token may have leaked, also use **Create new runner → Reset + registration token** there. `gitea actions generate-runner-token` returns + the current active token rather than a new one, so without the reset the + old token keeps working for new registrations. +3. Delete the stored token so the role generates a fresh one: + `kubectl -n gitea delete secret gitea-actions-runner-token` +4. Remove the old registration so the runner registers again. Either delete + `/data/.runner` from the `runner` container, or delete the pod and its PVC + (`data-runner-gitea-actions-runner-0`); it holds nothing else. +5. Run the role: `ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions`. + It fetches the active token with `gitea actions generate-runner-token`, + stores it in the Secret, and the runner registers on start. +6. Confirm a single runner shows **Idle** in the admin page and a test workflow + succeeds. + +## Related + +- Roadmap: label `cicd` on DevOps/software-infrastructure, milestones + "CI/CD 0" to "CI/CD 4". +- Agents coordinate CI on the forum's Agents category, topic "Gitea Actions + runner: status and changes".