Files
software-infrastructure/docs/ci-cd/runner.md
T
2026-11-12 18:57:12 -05:00

12 KiB
Raw Blame History

Gitea Actions runner: operations runbook

How the shared CI runner for gitea.eom.dev is built, how to change it, and how to recover it. Tracked in DevOps/software-infrastructure#42 (S9).

Status: this describes the runner as configured in roles/ericomeehan.gitea after S3–S5 (#36, #37, #38) and Gitea's Actions settings from S6 (#39). Until those are deployed, the live runner still has the older setup: no node label, capacity 2, no Docker cleanup.

This file is public. It names no hosts, addresses, or credentials. Look those up in the inventory and vault.

Architecture

gitea (Deployment, ns gitea) <── polls for jobs ── gitea-actions-runner-0 (StatefulSet, ns gitea)
                                                    ├─ init: reach-gitea      waits until Gitea answers
                                                    ├─ dind                   Docker daemon; job containers run here
                                                    ├─ docker-prune (sidecar) cleans Docker data every 15 min (S5)
                                                    └─ runner                 act_runner; talks to dind over a socket
  • One runner for everyone. It is registered at instance level, so every repo and org on gitea.eom.dev shares it. Capacity is 1 job at a time (gitea_actions_runner_capacity).
  • Helm chart gitea/actions, pinned to gitea_actions_chart_version (0.1.2). Release name gitea-actions, namespace gitea.
  • Placement. The pod only schedules on a node labeled eom.dev/ci-runner=true (gitea_actions_runner_node_selector). The role fails with a clear message if no node has the label.
  • Docker storage. DinD's /var/lib/docker is an emptyDir on that node's local disk, capped by gitea_actions_docker_data_size (50Gi). It is emptied whenever the pod is recreated. NFS can't be used because overlay2 doesn't support it.
  • Runner state. /data in the runner container is a 1Gi PVC (data-runner-gitea-actions-runner-0). It holds .runner, the runner's registration. Deleting it makes the runner register again on the next start.
  • Registration token. Kubernetes Secret gitea-actions-runner-token in namespace gitea. The role creates it once with gitea actions generate-runner-token and never prints it.
  • Job images. ubuntu-latest, ubuntu-24.04 and ubuntu-22.04 map to dated docker.gitea.com/runner-images tags (gitea_actions_runner_labels). force_pull is off, so they stay cached in DinD and the cleanup never removes them.
  • Limits. DinD gets up to 2 CPU and 8Gi memory, and holds every job container. A job over the memory limit is OOM-killed and fails with a step error. The runner fails a job after 2h; Gitea fails one after 2h30m (ENDLESS_TASK_TIMEOUT), marks one whose runner stops reporting as failed after 5m (ZOMBIE_TASK_TIMEOUT), and cancels one nobody picks up within 12h (ABANDONED_JOB_TIMEOUT).
  • Monitoring (S7, #40, in roles/ericomeehan.prometheus, group eom-ci): GiteaRunnerNotReady, GiteaRunnerRestarted, GiteaRunnerPodReplaced, GiteaRunnerEvicted, NodeDiskPressure, GiteaDown.

All tunables are in roles/ericomeehan.gitea/defaults/main.yml, with comments.

Applying a change

  1. Change the role (DevOps/ansible-role-gitea), push it, then bump the submodule here and push that.

  2. Check that no job is running: Site Administration → Actions → Runners, or each busy repo's Actions tab. Applying restarts the runner and any running job dies. Post a heads-up in the forum's Agents category topic "Gitea Actions runner: status and changes" if other agents' CI may be busy.

  3. Apply only the runner tasks:

    ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions
    

    Add the vault option you normally use. Gitea itself isn't redeployed.

  4. Confirm: kubectl -n gitea get pod gitea-actions-runner-0 shows all containers ready, the runner shows Idle in the admin Runners page, and a small workflow run succeeds.

Rollback: git revert the submodule bump in this repo, then rerun the same command.

Reading logs

What Command
Pod state and recent events kubectl -n gitea describe pod gitea-actions-runner-0
Runner (job pickup, job results) kubectl -n gitea logs gitea-actions-runner-0 -c runner
Docker daemon kubectl -n gitea logs gitea-actions-runner-0 -c dind
Cleanup passes and disk use kubectl -n gitea logs gitea-actions-runner-0 -c docker-prune
Log from before a restart add --previous to any of the above
Gitea's side (task state, zombie/endless timeouts) kubectl -n gitea logs deploy/gitea -c gitea | grep -i action
Docker disk use inside DinD kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker system df
Free space on the Docker volume kubectl -n gitea exec gitea-actions-runner-0 -c dind -- df -h /var/lib/docker

Job logs themselves are in the web UI under each run. They are kept for 90 days, and artifacts for 30.

A job that stops logging and later fails with no step error means the runner lost it: the pod was restarted, evicted or replaced, or Gitea's zombie timeout fired. Check the alerts above and describe pod events before blaming the job's code.

Recovery

Jobs queue but never start

  1. Is the runner online? Admin → Actions → Runners. If it is Offline, go to "Runner pod not running" below.
  2. Does the job ask for a label the runner doesn't have (runs-on: macos-*, windows-*, or a custom label)? It will wait until Gitea cancels it after 12h. Fix the workflow, or cancel the runs from the repo's Actions tab.
  3. Is the queue just long? With capacity 1, one long job blocks everyone. Find the run that is in progress. If it is stuck, cancel it in the UI. Workflows should use concurrency with cancel-in-progress: true so repeated pushes don't stack up.
  4. A repo flooding the queue (for example workflows inherited from an upstream .github/workflows): cancel its queued runs, then give it a .gitea/workflows/ directory, which makes Gitea ignore .github/workflows.

Runner pod not running

kubectl -n gitea get pod gitea-actions-runner-0 -o wide and describe:

  • Pending, "didn't match node selector": no node carries eom.dev/ci-runner=true, or that node is down. Label a suitable node (enough local disk, see #36) with kubectl label node <node> eom.dev/ci-runner=true.
  • Pending, insufficient cpu/memory/ephemeral-storage: the labeled node is full. Free it up, or label a bigger node.
  • Init:0/1 (reach-gitea): Gitea isn't answering. Fix Gitea first (GiteaDown).
  • CrashLoopBackOff in runner: read logs -c runner --previous. Registration errors mean the token or .runner is bad; see "Replacing the registration token".
  • CrashLoopBackOff in dind: usually a full or broken /var/lib/docker. Deleting the pod gives it a fresh, empty emptyDir: kubectl -n gitea delete pod gitea-actions-runner-0 (the StatefulSet recreates it; the first job re-pulls its image, about 5 GB).

Runner evicted, or the disk is full

Signs: GiteaRunnerEvicted or NodeDiskPressure, events saying ephemeral local storage usage exceeds, or jobs failing on no space left on device.

  1. Check usage: docker system df and df -h /var/lib/docker (table above), and the docker-prune log for its last pass.

  2. If the cleanup sidecar isn't running or can't keep up, free space by hand while no job is running:

    kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker container prune -f
    kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker builder prune -af
    kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker image prune -af
    

    The last command also removes the cached job images; the next job re-pulls.

  3. If it keeps happening, a workflow is producing more data than the cap allows. Find it from the runs that were active at the time, then either fix the workflow or raise gitea_actions_docker_data_size and the ephemeral-storage limits together (the node needs that much free local disk).

  4. If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that were lost, once.

kubectl itself fails

If kubectl errors with a TLS or certificate message, the problem is the cluster's control plane, not the runner. CI usually keeps working, because the runner talks to Gitea directly. On kubeadm clusters, check certificate expiry on the control plane with kubeadm certs check-expiration and renew with kubeadm certs renew, following the Kubernetes documentation for restarting the control-plane components afterwards.

Replacing the registration token

Needed when the runner is deleted in the Gitea UI, the token is suspected leaked, or the runner can't register. No token value ever needs to be copied by hand.

  1. Make sure no job is running.
  2. In Site Administration → Actions → Runners, delete the old runner entry. If the token may have leaked, also use Create new runner → Reset registration token there. gitea actions generate-runner-token returns the current active token rather than a new one, so without the reset the old token keeps working for new registrations.
  3. Delete the stored token so the role generates a fresh one: kubectl -n gitea delete secret gitea-actions-runner-token
  4. Remove the old registration so the runner registers again. Either delete /data/.runner from the runner container, or delete the pod and its PVC (data-runner-gitea-actions-runner-0); it holds nothing else.
  5. Run the role: ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions. It fetches the active token with gitea actions generate-runner-token, stores it in the Secret, and the runner registers on start.
  6. Confirm a single runner shows Idle in the admin page and a test workflow succeeds.

Proposed: runner on a dedicated host

Not deployed. Proposed in DevOps/software-infrastructure#45 as the answer to I3 and S3: run the runner on inspiron-3670 instead of in the cluster.

  • Layout. Docker Compose project gitea-runner in /opt/gitea-runner: dind (a private Docker daemon that holds every job container, with data in /var/lib/gitea-runner/docker), runner, and prune (the same cleanup script as in the cluster). Jobs never see the host's own Docker daemon or its other containers.
  • Settings. Capacity, timeouts, job images and cleanup are the same variables as the in-cluster runner. Host-only ones are gitea_actions_host_* in the role's defaults/main.yml: 4 CPUs and 16 GB for jobs, and a CPU weight below the host's other containers.
  • Apply. ansible-playbook -i inventories/eom.dev.yml ci-runner.yaml -K (-K because the host asks for a sudo password). The first run fetches a registration token from the cluster, registers, and deletes the token file. Rerunning recreates only containers whose settings changed; that cancels a running job.
  • Logs. On the host: docker compose -f /opt/gitea-runner/compose.yaml logs runner (or dind, prune).
  • Roll back. docker compose -f /opt/gitea-runner/compose.yaml down on the host. The in-cluster runner keeps serving jobs until it is removed separately.
  • Note for jobs. nproc inside a job reports all of the host's CPUs, not the 4-CPU cap, so builds that size -j from it oversubscribe.
  • Roadmap: label cicd on DevOps/software-infrastructure, milestones "CI/CD 0" to "CI/CD 4".
  • Agents coordinate CI on the forum's Agents category, topic "Gitea Actions runner: status and changes".