What this prompt does
This prompt makes the AI run a real batch-versus-stream decision for a specific workload instead of defaulting to the streaming stack everyone reaches for. You provide the [use_case], [data_volume], [event_rate], [source_type], the [freshness_requirement], and a [monthly_budget], and the model maps your requirements onto the batch–stream spectrum (pure batch, micro-batch, near-real-time, true real-time) with justification grounded in latency, volume, and cost.
The structure works because it forces both options to be costed and scored side by side rather than assumed. It recommends [batch_tool_options] with processing time and total cost of ownership, recommends [stream_tool_options] with infrastructure needs, an operational complexity score, and monthly cost, and weighs Lambda versus Kappa architectures for maintainability. It analyzes correctness for [correctness_challenges], gives a migration path with triggers to move from [starting_approach] to the other, and produces a decision matrix scoring latency, cost, complexity, correctness, [team_skills], and scalability. The matrix keeps the choice defensible when the budget gets questioned.
When to use it
- You're about to commit infrastructure spend and want the cheapest correct architecture, not the trendiest.
- You have a latency target and need to know whether micro-batch already satisfies it.
- You need batch and streaming options costed against a real
[monthly_budget]. - You're weighing Lambda versus Kappa and want a maintainability comparison.
- Your team's
[team_skills]should factor into the choice (limited Scala, strong SQL). - You want a documented, defensible decision to justify the architecture when challenged.
Example output
Expect an analysis: where your requirements land on the batch–stream spectrum with reasoning; a batch recommendation using [batch_tool_options] with processing time and cost; a streaming recommendation using [stream_tool_options] with a complexity score and monthly cost; a Lambda-versus-Kappa comparison; a correctness section addressing [correctness_challenges]; a migration path from [starting_approach] with trigger conditions; and a scored decision matrix across latency, cost, complexity, correctness, team skills, and scalability.
Pro tips
- Be honest about
[freshness_requirement]— a true sub-second need justifies streaming, but "real-time" that actually means "within a few minutes" is usually micro-batch territory. - Give a real
[monthly_budget]; the cost comparison is what exposes that the streaming stack often costs far more than the latency gain is worth. - State
[team_skills]accurately — a Flink recommendation is a liability if no one on the team can operate it under pressure. - Use the migration triggers rather than over-building now; starting with
[starting_approach]and moving when a concrete signal fires avoids paying for streaming before you need it. - Press on
[correctness_challenges](late, out-of-order, duplicate events) — these are where streaming and batch genuinely differ and where a wrong choice causes silent errors. - Keep the decision matrix; when budgets get questioned later, a scored comparison defends the choice far better than a recollection.
