What this prompt does
This prompt asks the AI to build a multi-stage Pandas data pipeline rather than a one-off cleaning script. It defines six ordered stages: ingest from [input_format], clean (nulls, duplicates, outliers via [outlier_method]), validate against [validation_rules], transform with [transformations], enrich from [enrichment_source], and export to [output_destination] — all processed in [chunk_size]-row chunks with quality checks between stages and a profiling report.
The structure works because it separates the pipeline into discrete, inspectable stages instead of one tangled transform. The [data_source] variable frames what you're processing, [outlier_method] and [validation_rules] make cleaning and validation explicit and auditable, and [chunk_size] keeps memory bounded on large, messy datasets. Quality checks between stages are what stop a silent corruption from flowing all the way to [output_destination].
When to use it
- You're processing real, messy data and want a staged pipeline you can inspect at each step.
- You need chunked processing to handle datasets too large to load into memory at once.
- You want validation against
[validation_rules]so bad rows are caught, not exported. - You're enriching records from an external
[enrichment_source]and need that as a discrete stage. - You need error recovery for partial failures rather than an all-or-nothing run.
Example output
Expect a structured Python pipeline: an ingest function for [input_format], a cleaning stage applying [outlier_method], a validation stage checking [validation_rules], transform and enrichment stages, and an export to [output_destination]. Processing iterates over [chunk_size]-row chunks, data-quality checks sit between stages, a profiling report summarizes the run, and logging plus partial-failure recovery keep a long job debuggable — code organized stage by stage, not a single monolithic function.
Pro tips
- Set
[chunk_size]to your memory budget — the default of 10000 is a safe start; raise it for speed, lower it if you hit limits. - Be explicit in
[validation_rules](types, required fields, ranges); vague rules let bad data slip through the quality gates. - Match
[outlier_method]to your data's distribution — IQR suits skewed data, z-score suits roughly normal data. - Name a real
[enrichment_source]; if it's a rate-limited API, plan for caching and the enrichment stage to handle failures. - Keep
[output_destination]honest — writing to both PostgreSQL and Parquet (the default) means designing two writers, so drop one if you don't need it. - Read the between-stage quality checks and tune their thresholds before trusting the profiling report on production data.
