What this prompt does
This prompt walks you through the full data pipeline for fine-tuning a specific LLM — from deciding what examples to collect, through formatting them correctly for the target model's API, to building a clean holdout evaluation set that will not bleed into training. It treats data preparation as an engineering problem with defined quality criteria, not a "gather some examples and hope" exercise.
The template is structured around ten sequential concerns that mirror how a serious fine-tuning project actually runs. Because you supply [base_model] explicitly, the formatting guidance stays model-specific — the JSONL schema for OpenAI's fine-tuning API differs meaningfully from the ShareGPT JSON format Axolotl expects for Llama, and this prompt forces that specificity up front. Providing [current_data] and [budget] anchors the collection strategy to what's actually achievable rather than theoretical.
What distinguishes it from generic fine-tuning prompts is the decontamination step (section 10) and the explicit quality rubric (section 4). These are the two things most practitioners skip, and they're the two things that most often produce a model that looks good on training loss but fails on real inputs.
When to use it
- You have a base model (GPT-4o mini, Llama 3.1 8B, Mistral 7B) and a clear task but no structured dataset yet.
- You're converting unstructured internal data (support tickets, code reviews, internal wikis) into fine-tuning examples.
- You want to generate synthetic training data using a stronger model and need generation prompts plus quality filters to do it responsibly.
- You're splitting a collected dataset and need stratification logic so your eval set reflects real production distribution.
- You're debugging a fine-tuned model that underperforms and suspect data quality is the root cause.
- You're working within a fixed budget and need to prioritize which examples to create manually vs. synthetically.
Example output
For [base_model]: gpt-4o-mini, [fine_tuning_purpose]: classify customer support tickets by urgency, [sample_count]: 3:
{"messages": [{"role": "system", "content": "Classify the support ticket urgency as LOW, MEDIUM, or HIGH. Reply with only the label."}, {"role": "user", "content": "My account was charged twice this month and I can't log in."}, {"role": "assistant", "content": "HIGH"}]}
{"messages": [{"role": "system", "content": "Classify the support ticket urgency as LOW, MEDIUM, or HIGH. Reply with only the label."}, {"role": "user", "content": "Can I change my newsletter preferences?"}, {"role": "assistant", "content": "LOW"}]}
{"messages": [{"role": "system", "content": "Classify the support ticket urgency as LOW, MEDIUM, or HIGH. Reply with only the label."}, {"role": "user", "content": "I submitted a refund request 10 days ago and have not heard back."}, {"role": "assistant", "content": "MEDIUM"}]}
The response also includes the quality rubric: score 1 if the label is ambiguous or the ticket is truncated, score 5 if the ticket contains enough signal for a human annotator to label it confidently within 10 seconds. Only examples scoring 4 or 5 enter the training split.
Pro tips
- Lock the system prompt first. The system prompt in your training examples must be byte-for-byte identical to what you'll use in production inference. If it drifts — even whitespace — your fine-tuned model will underperform on the exact task it was trained for. Set it before generating a single example and treat it as frozen.
[target_size]anchors everything. Start with "1000 examples" and let the prompt reason backward — it will tell you how many synthetic vs. human-annotated examples that budget realistically allows, and where hand-labeling time is best spent.- Use
[eval_size]as 5–10% of total, minimum 100 examples. For classification tasks, stratify by label so rare classes are represented. For generative tasks, sample from each difficulty tier. Ask the prompt to generate the eval set first, then hold it out before touching the training split. - Synthetic data needs a judge pass. When you use section 5 to generate synthetic examples, pipe those outputs back through a separate GPT-4o prompt scoring each against your section 4 rubric. Only keep scores of 4 or 5. This is cheap per-example and prevents low-quality noise from compounding across thousands of rows.
- Decontamination is not optional. Before training, run fuzzy-match deduplication (MinHash or SimHash) between your eval and train splits. Even paraphrased overlaps inflate eval metrics and hide real generalization gaps — you'll ship a model that looks strong internally and collapses on production inputs.
