What this prompt does
This prompt asks the model to act as a senior Rust engineer and build a memory-efficient data pipeline that streams over files of arbitrary size. You set [input_format], the [transformation] to apply, and the [output_format], and the hard constraint is bounded memory regardless of how large the input grows. It returns a complete src/main.rs with Cargo dependencies listed at the top.
The structure works because it pins down the failure mode most pipelines hit in production: "it worked on the sample file" becoming an out-of-memory crash. By requiring a streaming reader for [input_format], an incremental writer for [output_format], and a transformation stage that processes [transformation] over the stream, the prompt prevents the model from quietly loading everything into RAM. Rayon parallelism is requested only where ordering and aggregation make it safe, which keeps the speedup from corrupting results. The custom error enum and Result propagation mean failures surface as values you handle rather than panics that kill a long run halfway through.
When to use it
- You're processing files too big to fit in memory and need a streaming approach.
- You want a Rust pipeline with a hard memory ceiling you can reason about.
- You need safe parallelism that won't scramble ordering or break aggregations.
- You want proper error handling with a custom error enum instead of scattered
unwrap. - You're building backend or data tooling where reliability beats raw cleverness.
- You want a progress bar driven by bytes processed for long-running jobs.
- You're moving a script that worked on samples toward something safe for production data.
Example output
Expect a single, complete src/main.rs with the Cargo dependencies declared up top. Inside: a streaming reader for [input_format], the [transformation] stage, an incremental [output_format] writer, rayon used only where it's safe, a custom error enum with Result propagation, and a byte-driven progress bar plus a short note on the memory ceiling. The note on the ceiling lets you sanity-check that memory stays flat as input size grows. It's meant to be a runnable starting point, not a snippet.
Pro tips
- Be specific in
[transformation]— "parse, clean, aggregate daily totals" tells the model where ordering matters and where the merge step needs care. - Decide upfront which stages can parallelize without breaking ordering; for aggregations, the merge step is where bugs hide.
- Match
[input_format]and[output_format]to your real data (CSV in, Parquet out) so the reader and writer are correct rather than generic. - Insist on no
unwrapin the hot path — the custom error enum is what keeps the pipeline from panicking mid-run. - Ask the model to state the actual memory ceiling so you can confirm the streaming claim instead of trusting it.
- If a transformation needs cross-record state, call that out explicitly so the model doesn't naively parallelize it.
