What this prompt does
This prompt makes the AI design a Kafka streaming pipeline with the topology decisions that hurt most if skipped. You set [streaming_use_case], the [event_volume] per second, and [event_sources], and the model designs topics for [topic_list] with [partition_count], a replication factor of [replication_factor], and [retention_period], plus schema management through [schema_registry] with backward-compatibility and evolution rules.
The structure works because it walks the full path a message takes and pins the durability and correctness knobs at each step. Producers get batching, [compression_type], idempotence, and acks=[acks_setting]; consumer groups get [consumer_count] consumers, [offset_strategy] offset management, and exactly-once via [exactly_once_method]; stream processing runs on [stream_processor] with [window_size] windows and [late_data_strategy] for late data. It adds a dead letter topic and consumer-lag alerts when lag exceeds [lag_threshold]. The schema-registry and DLQ choices are exactly the ones teams regret skipping under deadline.
When to use it
- You need sub-second analytics or alerting and a batch pipeline can't meet the latency.
- You're designing Kafka topics and need partition, replication, and retention decided from throughput, not guessed.
- You want schema evolution governed by a registry so producers and consumers don't break each other.
- You need exactly-once processing rather than at-least-once with duplicates.
- You have late-arriving or out-of-order events and need a defined windowing and lateness strategy.
- You want consumer-lag alerting and a DLQ with replay before going live.
Example output
Expect an architecture: a topic table with partition counts, replication, and retention per topic; Avro/Protobuf schema definitions and compatibility rules for [schema_registry]; producer config (batching, [compression_type], idempotence, acks=[acks_setting]); consumer-group layout with [offset_strategy] and [exactly_once_method]; a [stream_processor] job with [window_size] aggregations and join logic; a dead letter topic with replay; and monitoring with consumer-lag alerts at [lag_threshold].
Pro tips
- Size
[partition_count]from real throughput and consumer parallelism — too few caps consumer scaling, too many adds overhead and rebalance pain. - Use
acks=allwith idempotence when you need the strong durability the default assumes; weaker acks trade safety for latency and can lose data. - Decide
[exactly_once_method]early — retrofitting Kafka transactions onto an at-least-once pipeline later is far more disruptive than designing for it. - Pin schema compatibility rules in
[schema_registry]from the start; an unmanaged schema change is the classic way a producer silently breaks every consumer. - Set
[late_data_strategy]to match your[window_size]; allowed lateness shorter than real event delay drops valid late events into the void. - Alert on
[lag_threshold]as a leading indicator — rising consumer lag warns you of a falling-behind consumer before it becomes data loss at retention expiry. - Set
[retention_period]per topic role; raw topics often need only days, while enriched topics may need weeks, and over-retaining everything inflates storage cost. - Wire the dead letter topic before launch, not after the first poison message — replaying from a DLQ is far easier than reconstructing lost records.
