What this prompt does
This prompt builds a monitoring and evaluation system for a [app_type] handling [volume] requests per day. It tracks latency (TTFT, total response time, tokens per second at p50, p95, p99), cost per request, per user, and per feature with budget alerts, and quality scoring via [eval_method] across [quality_dimensions].
The structure works because once an AI feature is live, dashboards are what tell you whether it is actually working. Hallucination detection with source verification for RAG, user feedback collection, A/B test monitoring with statistical significance, and error-rate tracking by category turn vague "it feels off" worries into measurable signals. Anomaly alerting on quality drops, cost spikes, and latency increases is what catches problems before users do, surfaced in a [dashboard_tool] with drill-down.
Token-usage analytics close a gap most teams miss. Tracking input versus output tokens and wasted context shows where prompts are bloated and where cost is leaking, which directly informs both spend and latency. Combined with per-feature cost attribution and percentile latency, you get a full picture of where an AI feature is slow, expensive, or drifting in quality, so optimization effort goes where the data points rather than where intuition guesses.
When to use it
- You have an AI feature in production and need visibility into latency, cost, and quality.
- You want LLM-as-judge quality scoring across
[quality_dimensions]rather than gut feel. - You need cost tracking per request, user, and feature with budget alerts.
- You are running RAG and want hallucination detection with source verification.
- You are A/B testing prompt variants and need statistical significance, not anecdotes.
- You want anomaly alerts that catch quality drift or cost spikes before users complain.
Example output
Expect a monitoring system in [language] with [storage] for metrics: latency percentile tracking, cost-attribution logic, quality scoring via [eval_method], hallucination detection, a feedback collection mechanism, A/B test analysis, error categorization, and alerting rules. It typically ships with a [dashboard_tool] dashboard definition (such as Grafana JSON) wiring all the metrics together with drill-down.
Pro tips
- Track latency as percentiles, not averages; p95 and p99 reveal the slow tail that a mean hides entirely.
- Attribute cost per feature, not just globally, so you can see which features are expensive and worth optimizing.
- Use a cheaper model for
[eval_method]LLM-as-judge scoring to keep evaluation affordable at[volume]scale. - Define
[quality_dimensions]that map to real user value (accuracy, helpfulness, safety) rather than vanity metrics. - Set alert thresholds from a real baseline, not guesses, so you catch genuine anomalies without constant false alarms.
- Treat quality-drift and cost-spike alerts as the priority; those are the two that catch problems before users do.
- Watch wasted-context tokens; bloated prompts quietly inflate both cost and latency without improving answers.
- Collect user feedback (thumbs, ratings, free text) and correlate it with your
[eval_method]scores to validate that automated judging matches real perception.
