S9: Write the runner operations runbook #42

Open
opened 2026-11-09 17:05:58 +00:00 by claude · 1 comment
Member

Add docs/ci-cd/runner.md: architecture, how to apply changes, how to read job and runner logs, what to do for a stuck queue, an evicted runner, or a full disk, and how to replace the registration token (described, with no secret values).

Acceptance: someone new could recover the runner by following it. No credentials or internal addresses.

Depends on: S3, S4, S5.

Add `docs/ci-cd/runner.md`: architecture, how to apply changes, how to read job and runner logs, what to do for a stuck queue, an evicted runner, or a full disk, and how to replace the registration token (described, with no secret values). **Acceptance**: someone new could recover the runner by following it. No credentials or internal addresses. Depends on: S3, S4, S5.
claude added this to the CI/CD 0: Stabilize the runner milestone 2026-11-09 17:05:58 +00:00
claude added the
priority
low
cicd
labels 2026-11-09 17:05:58 +00:00
claude added this to the CI/CD Roadmap project 2026-11-09 17:07:33 +00:00
claude moved this to To Do in CI/CD Roadmap on 2026-11-09 17:07:59 +00:00
Author
Member

Draft runbook committed: docs/ci-cd/runner.md in 156cbee.

What it covers: the architecture (pod containers, node label, Docker emptyDir, runner PVC, token Secret, pinned job images, limits and Gitea timeouts, S7 alerts). It also covers how to apply a role change with --tags actions and roll it back, a table of log commands, and recovery steps for a stuck queue, a pod that isn't running (by symptom), eviction or a full Docker disk, and kubectl itself failing. Finally, it explains how to replace the registration token without ever handling its value. The S7 alert annotations already link to this file.

How it was checked: the container names come from rendering chart gitea/actions 0.1.2 with helm template in a container. Settings and their values come from the role's defaults/main.yml and tasks/actions.yaml as committed. markdownlint reports only line-length warnings. A grep found no addresses or credentials. None of the recovery steps were run against the live runner.

Why it stays open: the doc describes the S3–S5 configuration, which isn't deployed yet, and it says so at the top. To meet the acceptance criterion, someone new must be able to recover the runner by following it. I'll check it against the live runner after #36, #37 and #38 are applied, fix whatever differs, and close it then.

One point to confirm after deploy: gitea actions generate-runner-token returns the current active token rather than a new one, so the doc says to rotate with Reset registration token in the admin UI. I based this on Gitea's private API behaviour as I understand it, and haven't tested it on 1.27.

— eom-cicd-roadmap

**Draft runbook committed:** `docs/ci-cd/runner.md` in 156cbee. **What it covers:** the architecture (pod containers, node label, Docker `emptyDir`, runner PVC, token Secret, pinned job images, limits and Gitea timeouts, S7 alerts). It also covers how to apply a role change with `--tags actions` and roll it back, a table of log commands, and recovery steps for a stuck queue, a pod that isn't running (by symptom), eviction or a full Docker disk, and `kubectl` itself failing. Finally, it explains how to replace the registration token without ever handling its value. The S7 alert annotations already link to this file. **How it was checked:** the container names come from rendering chart `gitea/actions` 0.1.2 with `helm template` in a container. Settings and their values come from the role's `defaults/main.yml` and `tasks/actions.yaml` as committed. `markdownlint` reports only line-length warnings. A grep found no addresses or credentials. **None of the recovery steps were run against the live runner.** **Why it stays open:** the doc describes the S3–S5 configuration, which isn't deployed yet, and it says so at the top. To meet the acceptance criterion, someone new must be able to recover the runner by following it. I'll check it against the live runner after #36, #37 and #38 are applied, fix whatever differs, and close it then. **One point to confirm after deploy:** `gitea actions generate-runner-token` returns the current active token rather than a new one, so the doc says to rotate with **Reset registration token** in the admin UI. I based this on Gitea's private API behaviour as I understand it, and haven't tested it on 1.27. — eom-cicd-roadmap
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DevOps/software-infrastructure#42