What this prompt does
This prompt sets the model up as a senior streaming-data engineer and asks for a concrete Kafka topic and schema strategy with real producer and consumer code -- not a high-level overview. You provide [use_case], [throughput], and [ordering_needs], and it returns topic naming, a partitioning strategy, retention and compaction settings, a Schema Registry plan, consumer group design, and working code.
The partition key is the decision the structure is built around. [ordering_needs] forces the model to justify which key it picks: strict per-entity ordering means keying on that entity, while loose ordering allows round-robin spreading. [throughput] informs partition count and whether one key risks hot-spotting under load. [use_case] defines the event domain and therefore how many topics make sense. By demanding the model defend the key against the ordering guarantee, the prompt avoids the default mistake of round-robin everything, and it ties retention and compaction choices to the way each topic is actually consumed.
When to use it
- You're designing an event backbone and need a defensible partition-key choice.
- You must preserve per-entity ordering (e.g. per-order) and want the keying to prove it.
- You're worried about hot partitions under peak load and need a throughput-aware layout.
- You need Schema Registry evolution rules (backward vs forward compatible) written down.
- You want retention and compaction settings justified per topic, not copy-pasted defaults.
- You need starter producer and consumer code with serialization already wired in.
- You're adding consumers later and want rebalance behaviour predictable from the start.
Example output
You get a topic config table (names, partition counts, retention, compaction), the schema in Avro or Protobuf with its evolution rules spelled out, then producer and consumer code with serialization and consumer-group config wired in. The code is meant to run and illustrate the keying and rebalance behaviour, not to be a snippet you still have to assemble. Together the table and code give you both the design rationale and the runnable proof of it.
Pro tips
- State
[ordering_needs]precisely --strict per-order orderingversus best-effort changes the key and the partition count. - Give a realistic
[throughput](e.g.50k events/sec peak) so partition counts and hot-spot warnings are sized to your load. - Describe
[use_case]as the full domain (orders and payments events) so the model decides topic count sensibly. - Replay a day of events through a new consumer before you call the schema final -- evolution bugs only show on real history.
- If the model defaults to round-robin, push back and make it tie the key choice to your stated ordering guarantee.
- Ask it to spell out the backward-vs-forward compatibility rule for your registry so a producer change can't break consumers silently.
- Pin down consumer-group design up front so rebalance behaviour is predictable when you add consumers later.
