What this prompt does
This prompt makes the AI design a data lineage tracking system that can answer "what breaks if we change this column" without anyone maintaining diagrams by hand. You describe the [data_platform], the [pipeline_count], [table_count], [dashboard_count], and [consumer_count], and the model builds a lineage metadata model capturing source → pipeline → table → column → dashboard relationships with transformation notes at each hop, stored in [lineage_storage].
The structure works because it derives lineage from artifacts you already produce rather than from hand-drawn graphs. It extracts from [extraction_sources] — parsing SQL, reading dbt refs, capturing Airflow dependencies — to build column-level lineage that traces a column like [example_output_column] back through [hop_count] hops. Impact analysis takes a proposed change to [impact_source] and lists every affected downstream table, pipeline, and dashboard with severity. A visualization API returns graph data in [graph_format], a freshness overlay flags stale-upstream nodes, and [notification_system] alerts owners. Column-level impact is what turns a scary migration into a routine one.
When to use it
- Stakeholders keep asking "what depends on this column" and you want a system that answers it.
- You're planning a schema migration and need to know every downstream table, pipeline, and dashboard affected.
- You want lineage extracted automatically from dbt, query history, and orchestrator metadata, not drawn by hand.
- You need column-level (not just table-level) tracing through several hops.
- You want an interactive lineage diagram fed by a graph API.
- You need to notify the right data owners when upstream changes could break their assets.
Example output
Expect a design: a lineage metadata model and [lineage_storage] schema, an extraction approach parsing [extraction_sources] to populate it, a column-level trace example for [example_output_column] across [hop_count] hops, an impact-analysis routine for [impact_source] with severity classification, a visualization API returning [graph_format] nodes and edges, a freshness/staleness overlay, and [notification_system] integration routing alerts to the right owners.
Pro tips
- Lean on
[extraction_sources]you already have (dbt manifest, query history) — hand-maintained lineage rots the moment a model changes, automated lineage stays current. - Insist on column-level, not just table-level, tracing; table-level lineage can't tell you whether your specific
[example_output_column]is actually affected by a change. - Tune
[hop_count]to your real depth — stopping too shallow misses the dashboard at the end of the chain that actually breaks. - Make impact-analysis severity meaningful so a rename touching one stale report isn't ranked the same as one feeding finance dashboards.
- Use the freshness overlay to catch the subtle bug where a downstream node is fresher than its upstream — a sign of an out-of-order or broken refresh.
- Route
[notification_system]alerts by data domain so owners get only the upstream changes that touch their assets, not every change everywhere.
