- Prometheus alert rules (files/alerting-rules.yml): SiteDown,
CertificateExpiringSoon, ScrapeTargetDown, GpuOverheating (>= 85 C),
HostOverheating and HostRebooted. Disk and RAM are already covered by
Grafana rules. Templates use Helm's {{` `}} escaping because the chart
runs alertingRules through tpl.
- Alertmanager routes everything to the grafana-matrix-forwarder (same
Matrix room as Grafana alerts), resolved notifications on, repeat 24h.
- Blackbox exporter (prom/blackbox-exporter) probing every public site,
plus a scrape job for the NVIDIA DCGM exporter (T4 temperature, clocks).
- forum-digest CronJob (9:00 ET): posts unanswered member topics on
discourse.eom.dev to Matrix through the same forwarder, and nothing when
there are none. Standard-library Python, public Discourse JSON only.
These replace the LocalAI SRE and community agents, which were too slow
and unreliable on the throttled T4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chart defaulted to the "nano" preset (192Mi memory limit), which
Prometheus outgrew, leaving it OOM-killed in a crash loop for ~25 days.
Request 250m/1Gi and limit 1 CPU/2Gi instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claudeandClaude Opus 5.5
