Gitea has metrics enabled, but Prometheus has no gitea_* series and no pod state metrics, so runner restarts and queue backups go unseen.
Work
In ansible-role-prometheus, scrape Gitea's /metrics securely and add kube-state-metrics or an equivalent source of pod status.
Add alerts: runner pod not ready, runner pod restarted or evicted, node disk pressure, Gitea down, Actions queue age too high.
Check what queue metrics 1.27 exposes. If there are none, add a small check using the Actions API.
Add a Grafana panel for runner health and queue depth.
Acceptance
Each alert fires in a controlled test or is shown to evaluate against real data.
An eviction now shows up in Prometheus.
Label ready-to-deploy when committed.
Gitea has metrics enabled, but Prometheus has no `gitea_*` series and no pod state metrics, so runner restarts and queue backups go unseen.
**Work**
- In `ansible-role-prometheus`, scrape Gitea's `/metrics` securely and add kube-state-metrics or an equivalent source of pod status.
- Add alerts: runner pod not ready, runner pod restarted or evicted, node disk pressure, Gitea down, Actions queue age too high.
- Check what queue metrics 1.27 exposes. If there are none, add a small check using the Actions API.
- Add a Grafana panel for runner health and queue depth.
**Acceptance**
- Each alert fires in a controlled test or is shown to evaluate against real data.
- An eviction now shows up in Prometheus.
Label `ready-to-deploy` when committed.
claude
added this to the CI/CD 0: Stabilize the runner milestone 2026-11-09 17:05:58 +00:00
kube-state-metrics v2.16.0 (built against client-go 1.32, within one minor version of the cluster's 1.31) in the prometheus namespace. It has its own ServiceAccount and a ClusterRole that only grants list/watch on the resources it's told to watch (pods, nodes, PVCs, deployments, statefulsets, daemonsets, jobs, cronjobs). Non-root, read-only root filesystem, no capabilities, 10m CPU / 64Mi request, 256Mi limit.
New scrape job kube-state-metrics with honor_labels: true.
New alert group eom-ci, routed to the existing Matrix receiver:
Alert
Fires when
Severity
GiteaRunnerNotReady
the runner StatefulSet has fewer ready replicas than desired for 10 min
critical
GiteaRunnerRestarted
a runner container restarted in the last 15 min
warning
GiteaRunnerPodReplaced
the runner pod was deleted, evicted or rescheduled in the last 15 min (new pod uid)
warning
GiteaRunnerEvicted
the runner pod was evicted in the last 15 min
critical
NodeDiskPressure
any node reports DiskPressure for 5 min
warning
GiteaDown
the Gitea Deployment has no available pod for 5 min
critical
Tested (nothing applied)
promtool check rules and promtool test rules (Prometheus 3.5.0, the deployed version) with synthetic series for each alert: it fires at the right time with the right labels and text, it doesn't fire before its for:, and it clears afterwards. It's checked against the rules as written and again after rendering through the Bitnami prometheus chart's tpl, so the {{...}} escaping is right.
kube-state-metrics manifests rendered with the role defaults and validated with kubeconform -strict against Kubernetes 1.31.2: 5 of 5 valid.
prometheus.yaml --syntax-check passes with a stub secrets file, and --list-tasks shows the new task between the blackbox exporter and Prometheus.
Add your usual vault option. (The playbook has no tags, so this reruns the whole role.)
Effect: creates kube-state-metrics and its RBAC, and Helm upgrades Prometheus with the new scrape job and rules, which restarts the Prometheus server pod for a short gap in scraping. The runner and Gitea aren't touched, and no CI job is affected.
Expect right away:NodeDiskPressure may fire for a node that already reports DiskPressure. That's a real condition, not noise. GiteaRunnerPodReplaced will fire once when the S3/S4 runner deploy recreates the pod. That's expected.
Risk: low. kube-state-metrics only reads, and if it fails to start you lose the new alerts, not existing ones (ScrapeTargetDown would report it).
Rollback:git revert 162fbd1 in the superproject and rerun the same command. That removes the scrape job and rules. kube-state-metrics itself stays until deleted: kubectl -n prometheus delete deploy,svc,sa kube-state-metrics and kubectl delete clusterrole,clusterrolebinding kube-state-metrics.
Acceptance, after deploy: I'll check that kube_pod_* series exist, that each alert evaluates against real data, and that the S3/S4 runner restart shows up as GiteaRunnerPodReplaced. That's the "eviction now shows up in Prometheus" proof, together with S8 (#41).
Still open in this issue
The Gitea /metrics scrape and the queue-age check are on hold for a decision from Eric. The details are in the private assessment.
Grafana panel for runner health and queue depth, once the series exist.
— eom-cicd-roadmap
**Part 1 committed, ready to deploy:** pod state and CI alerts. DevOps/ansible-role-prometheus@bba0d30, bumped in 162fbd1.
**What changed**
- **kube-state-metrics v2.16.0** (built against client-go 1.32, within one minor version of the cluster's 1.31) in the `prometheus` namespace. It has its own ServiceAccount and a ClusterRole that only grants `list`/`watch` on the resources it's told to watch (pods, nodes, PVCs, deployments, statefulsets, daemonsets, jobs, cronjobs). Non-root, read-only root filesystem, no capabilities, 10m CPU / 64Mi request, 256Mi limit.
- New scrape job `kube-state-metrics` with `honor_labels: true`.
- New alert group `eom-ci`, routed to the existing Matrix receiver:
| Alert | Fires when | Severity |
|---|---|---|
| `GiteaRunnerNotReady` | the runner StatefulSet has fewer ready replicas than desired for 10 min | critical |
| `GiteaRunnerRestarted` | a runner container restarted in the last 15 min | warning |
| `GiteaRunnerPodReplaced` | the runner pod was deleted, evicted or rescheduled in the last 15 min (new pod uid) | warning |
| `GiteaRunnerEvicted` | the runner pod was evicted in the last 15 min | critical |
| `NodeDiskPressure` | any node reports DiskPressure for 5 min | warning |
| `GiteaDown` | the Gitea Deployment has no available pod for 5 min | critical |
**Tested (nothing applied)**
- `promtool check rules` and **`promtool test rules`** (Prometheus 3.5.0, the deployed version) with synthetic series for each alert: it fires at the right time with the right labels and text, it doesn't fire before its `for:`, and it clears afterwards. It's checked against the rules as written and again after rendering through the Bitnami `prometheus` chart's `tpl`, so the `{{`...`}}` escaping is right.
- kube-state-metrics manifests rendered with the role defaults and validated with `kubeconform -strict` against Kubernetes 1.31.2: 5 of 5 valid.
- `prometheus.yaml --syntax-check` passes with a stub secrets file, and `--list-tasks` shows the new task between the blackbox exporter and Prometheus.
**Deploy**
```
ansible-playbook -i inventories/eom.dev.yml prometheus.yaml
```
Add your usual vault option. (The playbook has no tags, so this reruns the whole role.)
- **Effect:** creates kube-state-metrics and its RBAC, and Helm upgrades Prometheus with the new scrape job and rules, which restarts the Prometheus server pod for a short gap in scraping. The runner and Gitea aren't touched, and no CI job is affected.
- **Expect right away:** `NodeDiskPressure` may fire for a node that already reports DiskPressure. That's a real condition, not noise. `GiteaRunnerPodReplaced` will fire once when the S3/S4 runner deploy recreates the pod. That's expected.
- **Risk:** low. kube-state-metrics only reads, and if it fails to start you lose the new alerts, not existing ones (`ScrapeTargetDown` would report it).
- **Rollback:** `git revert 162fbd1` in the superproject and rerun the same command. That removes the scrape job and rules. kube-state-metrics itself stays until deleted: `kubectl -n prometheus delete deploy,svc,sa kube-state-metrics` and `kubectl delete clusterrole,clusterrolebinding kube-state-metrics`.
**Acceptance, after deploy:** I'll check that `kube_pod_*` series exist, that each alert evaluates against real data, and that the S3/S4 runner restart shows up as `GiteaRunnerPodReplaced`. That's the "eviction now shows up in Prometheus" proof, together with S8 (#41).
**Still open in this issue**
- **The Gitea `/metrics` scrape and the queue-age check are on hold for a decision from Eric.** The details are in the private assessment.
- **Grafana panel** for runner health and queue depth, once the series exist.
— eom-cicd-roadmap
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Gitea has metrics enabled, but Prometheus has no
gitea_*series and no pod state metrics, so runner restarts and queue backups go unseen.Work
ansible-role-prometheus, scrape Gitea's/metricssecurely and add kube-state-metrics or an equivalent source of pod status.Acceptance
Label
ready-to-deploywhen committed.Part 1 committed, ready to deploy: pod state and CI alerts. DevOps/ansible-role-prometheus@bba0d30, bumped in
162fbd1.What changed
prometheusnamespace. It has its own ServiceAccount and a ClusterRole that only grantslist/watchon the resources it's told to watch (pods, nodes, PVCs, deployments, statefulsets, daemonsets, jobs, cronjobs). Non-root, read-only root filesystem, no capabilities, 10m CPU / 64Mi request, 256Mi limit.kube-state-metricswithhonor_labels: true.eom-ci, routed to the existing Matrix receiver:GiteaRunnerNotReadyGiteaRunnerRestartedGiteaRunnerPodReplacedGiteaRunnerEvictedNodeDiskPressureGiteaDownTested (nothing applied)
promtool check rulesandpromtool test rules(Prometheus 3.5.0, the deployed version) with synthetic series for each alert: it fires at the right time with the right labels and text, it doesn't fire before itsfor:, and it clears afterwards. It's checked against the rules as written and again after rendering through the Bitnamiprometheuschart'stpl, so the{{...}}escaping is right.kubeconform -strictagainst Kubernetes 1.31.2: 5 of 5 valid.prometheus.yaml --syntax-checkpasses with a stub secrets file, and--list-tasksshows the new task between the blackbox exporter and Prometheus.Deploy
Add your usual vault option. (The playbook has no tags, so this reruns the whole role.)
NodeDiskPressuremay fire for a node that already reports DiskPressure. That's a real condition, not noise.GiteaRunnerPodReplacedwill fire once when the S3/S4 runner deploy recreates the pod. That's expected.ScrapeTargetDownwould report it).git revert 162fbd1in the superproject and rerun the same command. That removes the scrape job and rules. kube-state-metrics itself stays until deleted:kubectl -n prometheus delete deploy,svc,sa kube-state-metricsandkubectl delete clusterrole,clusterrolebinding kube-state-metrics.Acceptance, after deploy: I'll check that
kube_pod_*series exist, that each alert evaluates against real data, and that the S3/S4 runner restart shows up asGiteaRunnerPodReplaced. That's the "eviction now shows up in Prometheus" proof, together with S8 (#41).Still open in this issue
/metricsscrape and the queue-age check are on hold for a decision from Eric. The details are in the private assessment.— eom-cicd-roadmap