What this prompt does
This prompt has the AI stand up a complete Kubernetes monitoring stack for a [cluster_type] cluster running [workload_count] workloads. It installs kube-prometheus-stack via Helm with [retention_period] retention on [storage_class], creates ServiceMonitors for [custom_apps] scraping [metrics_port], writes PrometheusRules for [alert_scenarios], builds Grafana dashboards including [app_name]-specific [app_metrics], configures Alertmanager routing critical alerts to [critical_channel] and warnings to [warning_channel] with [silence_hours] quiet hours, adds recording rules, and wires blackbox-exporter for [external_endpoints].
The structure works because it separates collection (ServiceMonitors), alerting (PrometheusRules), visualization (Grafana), and routing (Alertmanager) into the distinct objects Prometheus actually uses. Routing critical pages separately from warnings, with quiet hours for non-critical, is what keeps the on-call signal useful instead of noisy. Recording rules pre-compute expensive percentile queries so dashboards stay fast under load. Sizing the install to [workload_count] workloads with [retention_period] retention on [storage_class] keeps the stack itself from becoming a resource hog, which is a common failure mode when monitoring is bolted on without capacity planning.
When to use it
- You are standing up monitoring on a new cluster and want dashboards and alerts before something breaks.
- You need ServiceMonitors to scrape custom application metrics from
[custom_apps]. - Your on-call is noisy and you want critical pages routed separately from warnings.
- You want alert rules for the standard failure classes — crash loops, high error rate, high latency, disk pressure.
- You need endpoint and certificate-expiry monitoring for
[external_endpoints]via blackbox-exporter. - You want recording rules so p50/p95/p99 latency queries do not slow your dashboards down.
- You want
[app_name]-specific dashboards alongside the cluster and namespace overviews, not just generic node graphs.
Example output
Expect a Helm values file for kube-prometheus-stack, ServiceMonitor manifests for [custom_apps], PrometheusRule objects covering [alert_scenarios] with severities and for-durations, Grafana dashboard definitions (cluster overview, namespace consumption, and [app_name] metrics), an Alertmanager routing config splitting [critical_channel] from [warning_channel] with [silence_hours], recording rules for request/error/latency, and a blackbox-exporter probe config for [external_endpoints].
Pro tips
- Route
[critical_channel]and[warning_channel]separately from day one; a flat routing tree is the fastest way to make on-call ignore alerts. - Set sensible
for/duration on each alert in[alert_scenarios]so a brief blip does not page someone — sustained, not momentary, conditions should fire. - Write recording rules for the expensive percentile queries before building dashboards on them, or your Grafana panels will lag under load.
- Size
[retention_period]and[storage_class]together; longer retention on slow storage gets expensive, so match retention to what you actually query. - Add certificate-expiry checks for
[external_endpoints]via blackbox-exporter — expired certs are a predictable, preventable outage. - Keep
[silence_hours]for non-critical only; critical pages must always break through regardless of the quiet window. - Tie the
[app_name]dashboard to the[app_metrics]that actually predict trouble — request rate, error rate, and latency percentiles — rather than every metric the app emits. - Confirm
[custom_apps]expose a Prometheus-format endpoint on[metrics_port]before writing ServiceMonitors, since a monitor scraping a missing endpoint fails silently.
