What this prompt does
This prompt makes the model design a few-shot prompt that holds up on messy real inputs, not just clean ones. Instead of stacking more instructions, it builds a high-signal example set that teaches the model the pattern you actually need, including the edge case and the failure mode you keep hitting. A flaky classifier usually means the examples were all happy-path, so this deliberately mirrors the hard cases.
The [task] variable defines what the prompt should do, and [output_shape] fixes the exact format every example must follow so the model learns the structure by demonstration. [hard_case] is the trickiest input you expect — it gets explicitly represented in the examples and the validation set — and [known_failure] is the recurring error, which one deliberately chosen example is built to correct. The set is ordered easy to hard with a note on why ordering matters for this task, and a short system instruction names the pattern the examples teach.
When to use it
- A classifier or extractor works on clean inputs but flakes on real, messy ones.
- You keep seeing one specific failure (like mixed-intent messages mislabeled) and want to target it.
- You need examples in an exact output shape so downstream parsing stays reliable.
- You want a held-out validation set to test changes instead of eyeballing them.
- You're embedding an LLM step in an agent or pipeline and want it to behave consistently.
- You need a quick rule for which example to add when a particular error returns.
Example output
You get the full few-shot prompt — a 3 to 5 example set in the exact [output_shape], including one deliberate edge case and one tricky example targeting [known_failure], ordered easy to hard with a note on why — plus a short system instruction naming the pattern the examples teach. Then you get ten held-out validation inputs (including [hard_case]) as input/expected pairs, and a quick read on which example to add if a specific error reappears. It's a complete, testable prompt with its own eval set, not just a list of examples.
Pro tips
- Make
[known_failure]precise — adding one example that mirrors the exact failure you see fixes more than another paragraph of instructions. - Keep every example in the identical
[output_shape]; inconsistent example formatting is itself a source of drift. - Put your real
[hard_case]in both the examples and the held-out set so you're testing the thing that actually breaks. - Describe
[task]tightly so the system instruction and examples teach one coherent pattern rather than a fuzzy goal. - Validate on the held-out inputs every time you touch the examples, or you're tuning blind.
- Watch example ordering — easy-to-hard sequencing matters, and reshuffling can change behavior, so change one thing at a time.
