S7: Monitor Gitea, the runner, and the job queue #40

Open
opened 2026-11-09 17:05:58 +00:00 by claude · 1 comment
Member

Gitea has metrics enabled, but Prometheus has no gitea_* series and no pod state metrics, so runner restarts and queue backups go unseen.

Work

  • In ansible-role-prometheus, scrape Gitea's /metrics securely and add kube-state-metrics or an equivalent source of pod status.
  • Add alerts: runner pod not ready, runner pod restarted or evicted, node disk pressure, Gitea down, Actions queue age too high.
  • Check what queue metrics 1.27 exposes. If there are none, add a small check using the Actions API.
  • Add a Grafana panel for runner health and queue depth.

Acceptance

  • Each alert fires in a controlled test or is shown to evaluate against real data.
  • An eviction now shows up in Prometheus.

Label ready-to-deploy when committed.

Gitea has metrics enabled, but Prometheus has no `gitea_*` series and no pod state metrics, so runner restarts and queue backups go unseen. **Work** - In `ansible-role-prometheus`, scrape Gitea's `/metrics` securely and add kube-state-metrics or an equivalent source of pod status. - Add alerts: runner pod not ready, runner pod restarted or evicted, node disk pressure, Gitea down, Actions queue age too high. - Check what queue metrics 1.27 exposes. If there are none, add a small check using the Actions API. - Add a Grafana panel for runner health and queue depth. **Acceptance** - Each alert fires in a controlled test or is shown to evaluate against real data. - An eviction now shows up in Prometheus. Label `ready-to-deploy` when committed.
claude added this to the CI/CD 0: Stabilize the runner milestone 2026-11-09 17:05:58 +00:00
claude added the cicd
priority
high
labels 2026-11-09 17:05:58 +00:00
claude added this to the CI/CD Roadmap project 2026-11-09 17:07:33 +00:00
claude moved this to To Do in CI/CD Roadmap on 2026-11-09 17:07:59 +00:00
Author
Member

Part 1 committed, ready to deploy: pod state and CI alerts. DevOps/ansible-role-prometheus@bba0d30, bumped in 162fbd1.

What changed

  • kube-state-metrics v2.16.0 (built against client-go 1.32, within one minor version of the cluster's 1.31) in the prometheus namespace. It has its own ServiceAccount and a ClusterRole that only grants list/watch on the resources it's told to watch (pods, nodes, PVCs, deployments, statefulsets, daemonsets, jobs, cronjobs). Non-root, read-only root filesystem, no capabilities, 10m CPU / 64Mi request, 256Mi limit.
  • New scrape job kube-state-metrics with honor_labels: true.
  • New alert group eom-ci, routed to the existing Matrix receiver:
Alert Fires when Severity
GiteaRunnerNotReady the runner StatefulSet has fewer ready replicas than desired for 10 min critical
GiteaRunnerRestarted a runner container restarted in the last 15 min warning
GiteaRunnerPodReplaced the runner pod was deleted, evicted or rescheduled in the last 15 min (new pod uid) warning
GiteaRunnerEvicted the runner pod was evicted in the last 15 min critical
NodeDiskPressure any node reports DiskPressure for 5 min warning
GiteaDown the Gitea Deployment has no available pod for 5 min critical

Tested (nothing applied)

  • promtool check rules and promtool test rules (Prometheus 3.5.0, the deployed version) with synthetic series for each alert: it fires at the right time with the right labels and text, it doesn't fire before its for:, and it clears afterwards. It's checked against the rules as written and again after rendering through the Bitnami prometheus chart's tpl, so the {{...}} escaping is right.
  • kube-state-metrics manifests rendered with the role defaults and validated with kubeconform -strict against Kubernetes 1.31.2: 5 of 5 valid.
  • prometheus.yaml --syntax-check passes with a stub secrets file, and --list-tasks shows the new task between the blackbox exporter and Prometheus.

Deploy

ansible-playbook -i inventories/eom.dev.yml prometheus.yaml

Add your usual vault option. (The playbook has no tags, so this reruns the whole role.)

  • Effect: creates kube-state-metrics and its RBAC, and Helm upgrades Prometheus with the new scrape job and rules, which restarts the Prometheus server pod for a short gap in scraping. The runner and Gitea aren't touched, and no CI job is affected.
  • Expect right away: NodeDiskPressure may fire for a node that already reports DiskPressure. That's a real condition, not noise. GiteaRunnerPodReplaced will fire once when the S3/S4 runner deploy recreates the pod. That's expected.
  • Risk: low. kube-state-metrics only reads, and if it fails to start you lose the new alerts, not existing ones (ScrapeTargetDown would report it).
  • Rollback: git revert 162fbd1 in the superproject and rerun the same command. That removes the scrape job and rules. kube-state-metrics itself stays until deleted: kubectl -n prometheus delete deploy,svc,sa kube-state-metrics and kubectl delete clusterrole,clusterrolebinding kube-state-metrics.

Acceptance, after deploy: I'll check that kube_pod_* series exist, that each alert evaluates against real data, and that the S3/S4 runner restart shows up as GiteaRunnerPodReplaced. That's the "eviction now shows up in Prometheus" proof, together with S8 (#41).

Still open in this issue

  • The Gitea /metrics scrape and the queue-age check are on hold for a decision from Eric. The details are in the private assessment.
  • Grafana panel for runner health and queue depth, once the series exist.

— eom-cicd-roadmap

**Part 1 committed, ready to deploy:** pod state and CI alerts. DevOps/ansible-role-prometheus@bba0d30, bumped in 162fbd1. **What changed** - **kube-state-metrics v2.16.0** (built against client-go 1.32, within one minor version of the cluster's 1.31) in the `prometheus` namespace. It has its own ServiceAccount and a ClusterRole that only grants `list`/`watch` on the resources it's told to watch (pods, nodes, PVCs, deployments, statefulsets, daemonsets, jobs, cronjobs). Non-root, read-only root filesystem, no capabilities, 10m CPU / 64Mi request, 256Mi limit. - New scrape job `kube-state-metrics` with `honor_labels: true`. - New alert group `eom-ci`, routed to the existing Matrix receiver: | Alert | Fires when | Severity | |---|---|---| | `GiteaRunnerNotReady` | the runner StatefulSet has fewer ready replicas than desired for 10 min | critical | | `GiteaRunnerRestarted` | a runner container restarted in the last 15 min | warning | | `GiteaRunnerPodReplaced` | the runner pod was deleted, evicted or rescheduled in the last 15 min (new pod uid) | warning | | `GiteaRunnerEvicted` | the runner pod was evicted in the last 15 min | critical | | `NodeDiskPressure` | any node reports DiskPressure for 5 min | warning | | `GiteaDown` | the Gitea Deployment has no available pod for 5 min | critical | **Tested (nothing applied)** - `promtool check rules` and **`promtool test rules`** (Prometheus 3.5.0, the deployed version) with synthetic series for each alert: it fires at the right time with the right labels and text, it doesn't fire before its `for:`, and it clears afterwards. It's checked against the rules as written and again after rendering through the Bitnami `prometheus` chart's `tpl`, so the `{{`...`}}` escaping is right. - kube-state-metrics manifests rendered with the role defaults and validated with `kubeconform -strict` against Kubernetes 1.31.2: 5 of 5 valid. - `prometheus.yaml --syntax-check` passes with a stub secrets file, and `--list-tasks` shows the new task between the blackbox exporter and Prometheus. **Deploy** ``` ansible-playbook -i inventories/eom.dev.yml prometheus.yaml ``` Add your usual vault option. (The playbook has no tags, so this reruns the whole role.) - **Effect:** creates kube-state-metrics and its RBAC, and Helm upgrades Prometheus with the new scrape job and rules, which restarts the Prometheus server pod for a short gap in scraping. The runner and Gitea aren't touched, and no CI job is affected. - **Expect right away:** `NodeDiskPressure` may fire for a node that already reports DiskPressure. That's a real condition, not noise. `GiteaRunnerPodReplaced` will fire once when the S3/S4 runner deploy recreates the pod. That's expected. - **Risk:** low. kube-state-metrics only reads, and if it fails to start you lose the new alerts, not existing ones (`ScrapeTargetDown` would report it). - **Rollback:** `git revert 162fbd1` in the superproject and rerun the same command. That removes the scrape job and rules. kube-state-metrics itself stays until deleted: `kubectl -n prometheus delete deploy,svc,sa kube-state-metrics` and `kubectl delete clusterrole,clusterrolebinding kube-state-metrics`. **Acceptance, after deploy:** I'll check that `kube_pod_*` series exist, that each alert evaluates against real data, and that the S3/S4 runner restart shows up as `GiteaRunnerPodReplaced`. That's the "eviction now shows up in Prometheus" proof, together with S8 (#41). **Still open in this issue** - **The Gitea `/metrics` scrape and the queue-age check are on hold for a decision from Eric.** The details are in the private assessment. - **Grafana panel** for runner health and queue depth, once the series exist. — eom-cicd-roadmap
claude added the needs-ericready-to-deploy labels 2026-11-11 01:02:55 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DevOps/software-infrastructure#40