Refs #45 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
227 lines
12 KiB
Markdown
227 lines
12 KiB
Markdown
# Gitea Actions runner: operations runbook
|
||
|
||
How the shared CI runner for gitea.eom.dev is built, how to change it, and how
|
||
to recover it. Tracked in DevOps/software-infrastructure#42 (S9).
|
||
|
||
> **Status:** this describes the runner as configured in
|
||
> `roles/ericomeehan.gitea` after S3–S5 (#36, #37, #38) and Gitea's Actions
|
||
> settings from S6 (#39). Until those are deployed, the live runner still has
|
||
> the older setup: no node label, capacity 2, no Docker cleanup.
|
||
|
||
This file is public. It names no hosts, addresses, or credentials. Look those
|
||
up in the inventory and vault.
|
||
|
||
## Architecture
|
||
|
||
```text
|
||
gitea (Deployment, ns gitea) <── polls for jobs ── gitea-actions-runner-0 (StatefulSet, ns gitea)
|
||
├─ init: reach-gitea waits until Gitea answers
|
||
├─ dind Docker daemon; job containers run here
|
||
├─ docker-prune (sidecar) cleans Docker data every 15 min (S5)
|
||
└─ runner act_runner; talks to dind over a socket
|
||
```
|
||
|
||
- **One runner for everyone.** It is registered at instance level, so every
|
||
repo and org on gitea.eom.dev shares it. Capacity is **1 job at a time**
|
||
(`gitea_actions_runner_capacity`).
|
||
- **Helm chart** `gitea/actions`, pinned to `gitea_actions_chart_version`
|
||
(0.1.2). Release name `gitea-actions`, namespace `gitea`.
|
||
- **Placement.** The pod only schedules on a node labeled
|
||
`eom.dev/ci-runner=true` (`gitea_actions_runner_node_selector`). The role
|
||
fails with a clear message if no node has the label.
|
||
- **Docker storage.** DinD's `/var/lib/docker` is an `emptyDir` on that node's
|
||
local disk, capped by `gitea_actions_docker_data_size` (50Gi). It is emptied
|
||
whenever the pod is recreated. NFS can't be used because overlay2 doesn't
|
||
support it.
|
||
- **Runner state.** `/data` in the `runner` container is a 1Gi PVC
|
||
(`data-runner-gitea-actions-runner-0`). It holds `.runner`, the runner's
|
||
registration. Deleting it makes the runner register again on the next start.
|
||
- **Registration token.** Kubernetes Secret `gitea-actions-runner-token` in
|
||
namespace `gitea`. The role creates it once with
|
||
`gitea actions generate-runner-token` and never prints it.
|
||
- **Job images.** `ubuntu-latest`, `ubuntu-24.04` and `ubuntu-22.04` map to
|
||
dated `docker.gitea.com/runner-images` tags (`gitea_actions_runner_labels`).
|
||
`force_pull` is off, so they stay cached in DinD and the cleanup never
|
||
removes them.
|
||
- **Limits.** DinD gets up to 2 CPU and 8Gi memory, and holds every job
|
||
container. A job over the memory limit is OOM-killed and fails with a step
|
||
error. The runner fails a job after 2h; Gitea fails one after 2h30m
|
||
(`ENDLESS_TASK_TIMEOUT`), marks one whose runner stops reporting as failed
|
||
after 5m (`ZOMBIE_TASK_TIMEOUT`), and cancels one nobody picks up within 12h
|
||
(`ABANDONED_JOB_TIMEOUT`).
|
||
- **Monitoring** (S7, #40, in `roles/ericomeehan.prometheus`, group `eom-ci`):
|
||
`GiteaRunnerNotReady`, `GiteaRunnerRestarted`, `GiteaRunnerPodReplaced`,
|
||
`GiteaRunnerEvicted`, `NodeDiskPressure`, `GiteaDown`.
|
||
|
||
All tunables are in `roles/ericomeehan.gitea/defaults/main.yml`, with comments.
|
||
|
||
## Applying a change
|
||
|
||
1. Change the role (`DevOps/ansible-role-gitea`), push it, then bump the
|
||
submodule here and push that.
|
||
2. **Check that no job is running**: Site Administration → Actions → Runners,
|
||
or each busy repo's Actions tab. Applying restarts the runner and **any
|
||
running job dies**. Post a heads-up in the forum's Agents category topic
|
||
"Gitea Actions runner: status and changes" if other agents' CI may be busy.
|
||
3. Apply only the runner tasks:
|
||
|
||
```sh
|
||
ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions
|
||
```
|
||
|
||
Add the vault option you normally use. Gitea itself isn't redeployed.
|
||
4. Confirm: `kubectl -n gitea get pod gitea-actions-runner-0` shows all
|
||
containers ready, the runner shows **Idle** in the admin Runners page, and a
|
||
small workflow run succeeds.
|
||
|
||
**Rollback:** `git revert` the submodule bump in this repo, then rerun the same
|
||
command.
|
||
|
||
## Reading logs
|
||
|
||
| What | Command |
|
||
|---|---|
|
||
| Pod state and recent events | `kubectl -n gitea describe pod gitea-actions-runner-0` |
|
||
| Runner (job pickup, job results) | `kubectl -n gitea logs gitea-actions-runner-0 -c runner` |
|
||
| Docker daemon | `kubectl -n gitea logs gitea-actions-runner-0 -c dind` |
|
||
| Cleanup passes and disk use | `kubectl -n gitea logs gitea-actions-runner-0 -c docker-prune` |
|
||
| Log from before a restart | add `--previous` to any of the above |
|
||
| Gitea's side (task state, zombie/endless timeouts) | `kubectl -n gitea logs deploy/gitea -c gitea \| grep -i action` |
|
||
| Docker disk use inside DinD | `kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker system df` |
|
||
| Free space on the Docker volume | `kubectl -n gitea exec gitea-actions-runner-0 -c dind -- df -h /var/lib/docker` |
|
||
|
||
Job logs themselves are in the web UI under each run. They are kept for 90
|
||
days, and artifacts for 30.
|
||
|
||
A job that **stops logging and later fails with no step error** means the
|
||
runner lost it: the pod was restarted, evicted or replaced, or Gitea's zombie
|
||
timeout fired. Check the alerts above and `describe pod` events before
|
||
blaming the job's code.
|
||
|
||
## Recovery
|
||
|
||
### Jobs queue but never start
|
||
|
||
1. Is the runner online? Admin → Actions → Runners. If it is **Offline**, go to
|
||
"Runner pod not running" below.
|
||
2. Does the job ask for a label the runner doesn't have (`runs-on: macos-*`,
|
||
`windows-*`, or a custom label)? It will wait until Gitea cancels it after
|
||
12h. Fix the workflow, or cancel the runs from the repo's Actions tab.
|
||
3. Is the queue just long? With capacity 1, one long job blocks everyone.
|
||
Find the run that is in progress. If it is stuck, cancel it in the UI.
|
||
Workflows should use `concurrency` with `cancel-in-progress: true` so
|
||
repeated pushes don't stack up.
|
||
4. A repo flooding the queue (for example workflows inherited from an upstream
|
||
`.github/workflows`): cancel its queued runs, then give it a
|
||
`.gitea/workflows/` directory, which makes Gitea ignore `.github/workflows`.
|
||
|
||
### Runner pod not running
|
||
|
||
`kubectl -n gitea get pod gitea-actions-runner-0 -o wide` and `describe`:
|
||
|
||
- **Pending, "didn't match node selector"**: no node carries
|
||
`eom.dev/ci-runner=true`, or that node is down. Label a suitable node (enough
|
||
local disk, see #36) with `kubectl label node <node> eom.dev/ci-runner=true`.
|
||
- **Pending, insufficient cpu/memory/ephemeral-storage**: the labeled node is
|
||
full. Free it up, or label a bigger node.
|
||
- **Init:0/1 (`reach-gitea`)**: Gitea isn't answering. Fix Gitea first
|
||
(`GiteaDown`).
|
||
- **CrashLoopBackOff in `runner`**: read `logs -c runner --previous`.
|
||
Registration errors mean the token or `.runner` is bad; see "Replacing the
|
||
registration token".
|
||
- **CrashLoopBackOff in `dind`**: usually a full or broken `/var/lib/docker`.
|
||
Deleting the pod gives it a fresh, empty `emptyDir`:
|
||
`kubectl -n gitea delete pod gitea-actions-runner-0` (the StatefulSet
|
||
recreates it; the first job re-pulls its image, about 5 GB).
|
||
|
||
### Runner evicted, or the disk is full
|
||
|
||
Signs: `GiteaRunnerEvicted` or `NodeDiskPressure`, events saying
|
||
`ephemeral local storage usage exceeds`, or jobs failing on `no space left on
|
||
device`.
|
||
|
||
1. Check usage: `docker system df` and `df -h /var/lib/docker` (table above),
|
||
and the `docker-prune` log for its last pass.
|
||
2. If the cleanup sidecar isn't running or can't keep up, free space by hand
|
||
while no job is running:
|
||
|
||
```sh
|
||
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker container prune -f
|
||
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker builder prune -af
|
||
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker image prune -af
|
||
```
|
||
|
||
The last command also removes the cached job images; the next job re-pulls.
|
||
3. If it keeps happening, a workflow is producing more data than the cap allows.
|
||
Find it from the runs that were active at the time, then either fix the
|
||
workflow or raise `gitea_actions_docker_data_size` and the ephemeral-storage
|
||
limits together (the node needs that much free local disk).
|
||
4. If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that
|
||
were lost, once.
|
||
|
||
### kubectl itself fails
|
||
|
||
If `kubectl` errors with a TLS or certificate message, the problem is the
|
||
cluster's control plane, not the runner. CI usually keeps working, because the
|
||
runner talks to Gitea directly. On kubeadm clusters, check certificate expiry on
|
||
the control plane with `kubeadm certs check-expiration` and renew with
|
||
`kubeadm certs renew`, following the Kubernetes documentation for restarting
|
||
the control-plane components afterwards.
|
||
|
||
## Replacing the registration token
|
||
|
||
Needed when the runner is deleted in the Gitea UI, the token is suspected
|
||
leaked, or the runner can't register. No token value ever needs to be copied by
|
||
hand.
|
||
|
||
1. Make sure no job is running.
|
||
2. In Site Administration → Actions → Runners, delete the old runner entry.
|
||
If the token may have leaked, also use **Create new runner → Reset
|
||
registration token** there. `gitea actions generate-runner-token` returns
|
||
the current active token rather than a new one, so without the reset the
|
||
old token keeps working for new registrations.
|
||
3. Delete the stored token so the role generates a fresh one:
|
||
`kubectl -n gitea delete secret gitea-actions-runner-token`
|
||
4. Remove the old registration so the runner registers again. Either delete
|
||
`/data/.runner` from the `runner` container, or delete the pod and its PVC
|
||
(`data-runner-gitea-actions-runner-0`); it holds nothing else.
|
||
5. Run the role: `ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions`.
|
||
It fetches the active token with `gitea actions generate-runner-token`,
|
||
stores it in the Secret, and the runner registers on start.
|
||
6. Confirm a single runner shows **Idle** in the admin page and a test workflow
|
||
succeeds.
|
||
|
||
## Proposed: runner on a dedicated host
|
||
|
||
Not deployed. Proposed in DevOps/software-infrastructure#45 as the answer to
|
||
I3 and S3: run the runner on `inspiron-3670` instead of in the cluster.
|
||
|
||
- **Layout.** Docker Compose project `gitea-runner` in `/opt/gitea-runner`:
|
||
`dind` (a private Docker daemon that holds every job container, with data in
|
||
`/var/lib/gitea-runner/docker`), `runner`, and `prune` (the same cleanup
|
||
script as in the cluster). Jobs never see the host's own Docker daemon or
|
||
its other containers.
|
||
- **Settings.** Capacity, timeouts, job images and cleanup are the same
|
||
variables as the in-cluster runner. Host-only ones are `gitea_actions_host_*`
|
||
in the role's `defaults/main.yml`: 4 CPUs and 16 GB for jobs, and a CPU
|
||
weight below the host's other containers.
|
||
- **Apply.** `ansible-playbook -i inventories/eom.dev.yml ci-runner.yaml -K`
|
||
(`-K` because the host asks for a sudo password). The first run fetches a
|
||
registration token from the cluster, registers, and deletes the token file.
|
||
Rerunning recreates only containers whose settings changed; that cancels a
|
||
running job.
|
||
- **Logs.** On the host: `docker compose -f /opt/gitea-runner/compose.yaml logs
|
||
runner` (or `dind`, `prune`).
|
||
- **Roll back.** `docker compose -f /opt/gitea-runner/compose.yaml down` on the
|
||
host. The in-cluster runner keeps serving jobs until it is removed
|
||
separately.
|
||
- **Note for jobs.** `nproc` inside a job reports all of the host's CPUs, not
|
||
the 4-CPU cap, so builds that size `-j` from it oversubscribe.
|
||
|
||
## Related
|
||
|
||
- Roadmap: label `cicd` on DevOps/software-infrastructure, milestones
|
||
"CI/CD 0" to "CI/CD 4".
|
||
- Agents coordinate CI on the forum's Agents category, topic "Gitea Actions
|
||
runner: status and changes".
|