What this prompt does
This prompt builds an LLM chatbot evaluation harness concrete enough to run today — a scored rubric plus 50 diverse test cases, not abstract quality talk. It casts the AI as an ML engineer building evaluation. You provide the [use_case], the [forbidden] must-never behaviors, the required [format], and the [tone] target. It returns a five-dimension rubric (correctness, helpfulness, safety, tone, format adherence) with a 1–5 scale and per-point descriptors, 50 test prompts across happy path, edge cases, ambiguous intent, adversarial, and out-of-scope, expected behavior per prompt including correct refusals, a scoring guide for marginal answers, and a pass threshold.
The value lives in the adversarial and out-of-scope cases, which is where bots leak data or overreach. By deriving refusals directly from your [forbidden] list, the harness tests the exact failures that cause incidents. The per-point rubric descriptors and the marginal-answer scoring guide are what let two reviewers agree, so scores stay consistent across people and across prompt revisions.
When to use it
- Standing up evaluation for a chatbot before shipping or after a prompt change
- Writing refusal tests from a concrete
[forbidden]behavior list - Building a reproducible rubric so two reviewers score consistently
- Catching regressions in safety when you tweak a system prompt
- Covering adversarial and out-of-scope inputs, not just the happy path
- Setting a clear pass threshold that decides what blocks release
Example output
You get the rubric table first — five dimensions, each with a 1–5 scale and one-line descriptors per point — followed by 50 cases listed as (prompt, category, expected behavior). The cases span happy path, edge, ambiguous, adversarial, and out-of-scope, with correct refusals spelled out for the forbidden behaviors. A scoring guide for marginal answers and a pass threshold tell you which failures block release.
Pro tips
- Make
[forbidden]exhaustive and specific — "give refunds outside policy, share another customer's order, invent shipping dates" generates sharper refusal tests than a vague "be safe" - Pin the
[format]exactly (e.g. short answer plus a single next-step link) so the format-adherence dimension has a concrete target - Write the refusals before the happy path — the adversarial and out-of-scope cases are where incidents start
- Use the marginal-answer scoring guide when two reviewers disagree; it is built to resolve exactly those ties
- Set the pass threshold strictly for safety failures, since a single leaked-order case should block release regardless of overall score
- Re-run the 50 cases after every prompt change to catch silent safety regressions before users do
- Grow the adversarial and out-of-scope buckets over time as you discover new ways users try to push the bot off its
[use_case]
