What this prompt does
This prompt makes the model a data engineering expert that designs a complete data quality and error-handling framework for an ETL pipeline. You describe the pipeline with [data_source], [data_destination], [data_volume], [pipeline_frequency], and [etl_tool], and it returns validation rules, an error-handling strategy, implementation code, a quality-score design, monitoring, and recovery procedures.
The structure is what makes it production-grade. Validation is split into schema-level, row-level, and dataset-level checks, so it catches everything from a dropped column to a sudden 10x row count. The error-handling section maps error types to actions and severities, including a dead-letter queue design for rows that fail. Because you name your [etl_tool], it generates reusable, config-driven validation functions in that stack rather than pseudocode. The [custom_validation] and [monitoring_addition] slots let you bake in business rules and cost alerts specific to your pipeline.
When to use it
- When bad source data keeps breaking dashboards or reports downstream.
- Setting up a new pipeline and you want quality gates in from day one, not bolted on later.
- When you need a dead-letter queue and quarantine workflow for rows that fail validation.
- To define a single data-quality score that trends over time and flags gradual degradation.
- When you need idempotent, replayable recovery procedures after a pipeline outage.
Example output
You get layered validation rules (schema, row, dataset) for each entity, an error-handling table mapping error types to detection, remediation, severity, and alerting, plus dead-letter-queue design. Then implementation code in your chosen tool with reusable validators and config-driven rules, a data-quality score (0-100) across completeness, accuracy, timeliness, consistency, and uniqueness with SQL per metric, monitoring and alerting rules, and recovery procedures covering reprocessing, reconciliation, backfill, and idempotency.
Pro tips
- Name your real
[etl_tool]stack (for example Python + Airflow + dbt) so the generated validators use idioms you can actually drop in. - Encode genuine business rules in
[custom_validation]— things like "order total equals the sum of line items" catch errors generic checks miss. - Set
[data_volume]and[pipeline_frequency]accurately; the dataset-level row-count and freshness checks key off your expected ranges. - Use
[monitoring_addition]for cost or SLA concerns specific to your warehouse, such as alerting when compute credits spike unexpectedly. - Start by wiring up the dead-letter queue and dataset-level checks first — in my experience those two catch most production surprises before they spread.
- Treat the generated code as a scaffold: review every threshold and remediation rule against your data before enabling auto-remediation.
