LLMOps: MLOps for the LLM Era

The LLMOps stack, in layers

1. Prompts and tools as code

• Prompts live in version control, not in Slack pastes or Notion docs.
• Tool definitions are typed and tested.
• Diffs of prompts get reviewed like any other code change.
• Templating is explicit (Jinja, MDX, custom) — no string concatenation in business logic.

2. Evals as CI gates

• A standing eval suite (50–500 examples) runs on every prompt change.
• Metrics: correctness, faithfulness, schema validity, refusal rate, regression on golden inputs.
• A change that drops a metric blocks merge.
• Eval set is locked and versioned; you don't iterate it to make scores go up.

3. Observability

• Every production call is traced: prompt, tools called, tokens in/out, latency, cost, model version.
• Traces are searchable by user, request ID, error class.
• Slow / failed / expensive calls bubble up as alerts.

4. Cost and budget controls

• Per-feature, per-tenant token budgets.
• Spike detection and circuit breakers.
• Routing logic to cheaper models when quality allows.
• Monthly review: top-N callers by spend, top-N by tokens-per-call.

5. Safety and compliance

• Input filters (PII, banned categories) and output filters (toxicity, leaks).
• Audit logs for every action an agent took.
• Data-handling policy: what gets sent to third-party APIs, what stays internal.
• Red-team eval suite separate from quality eval.

6. Incident response

• Playbooks for common LLM incidents: hallucinated facts, prompt-injected agents, cost runaways, model regression after a provider update.
• On-call rotation includes someone who can read prompt traces, not just top and kubectl.

What "MLOps for LLMs" gets wrong

• Model retraining is not the centre of LLMOps. Most teams use hosted models. The artifact under management is the prompt + tool + evaluation system, not weights.
• Pipelines are not the primary deliverable. Real-time agent loops are. The cadence is request-by-request, not batch.
• Drift detection looks different. "The world changed" is the new "data drifted." Catch it via fresh eval inputs and user feedback signals, not feature distributions.

Smallest viable LLMOps

You don't need a full stack day one. A bare minimum that catches 80% of pain:

1. Prompts in git, tied to commits.
2. A 50-prompt eval suite that runs on PRs.
3. Tracing that captures every call's prompt, response, tokens, latency, model.
4. A weekly cost / latency report.
5. A simple "rerun on staging with new prompt" tool to feel changes before they hit prod.

That's a couple of days of work and pays back forever.

What scales well later

• Prompt management UI — let non-engineers experiment safely against the eval suite.
• A/B testing harness — ship two prompt versions to a small slice, measure.
• Continuous evals on prod traffic — sample 1% of real calls, judge with a reference model, alert on regressions.
• Prompt registry with lineage — which prompt version, which model, which tool registry shipped together.

In one line

LLMOps is what stops your AI feature from being a heroic Friday-night ship and turns it into something a team can change confidently on a Tuesday.