What this prompt does
This prompt makes the AI design a data lake organization strategy built on the medallion architecture so the lake doesn't quietly turn into a swamp. You set [organization_type], [storage_platform], the [data_types], [source_count], and [storage_volume], and the model lays out Bronze (raw, append-only), Silver (cleaned, typed), and Gold (business aggregates) layers with a folder hierarchy of [bucket_name]/<layer>/<source>/<entity>/<partition>.
The structure works because a clear bronze/silver/gold split keeps cost and access control sane as volume grows. It defines a partitioning scheme ([partition_scheme]) using Hive-style partitions with [target_partition_size] targets to avoid small-file problems, file formats ([raw_format] for bronze, [analytics_format] for silver/gold) with [compression_codec], a metadata catalog via [catalog_tool], access control through [access_model] with roles [data_roles], lifecycle policies moving data to [cold_storage] after [hot_retention], and a compaction job on a [compaction_schedule]. The lifecycle and compaction rules are what keep the lake healthy.
When to use it
- You're standing up a data lake and want bronze/silver/gold structure from day one.
- You're hitting the small-file problem and need partition sizing and compaction to fix it.
- You need a consistent folder hierarchy and naming so datasets are discoverable.
- You want a metadata catalog registering schema, owner, freshness, PII, and lineage.
- You need role-based access with column- and row-level security on PII.
- You want lifecycle rules that tier cold data and archive or delete on a schedule, often for compliance.
Example output
Expect a strategy document: the medallion layer definitions, a folder hierarchy template under [bucket_name], a partitioning scheme with target file sizes, format and compression choices per layer with justification, a catalog design via [catalog_tool], an access model defining [data_roles] with column- and row-level rules, lifecycle policies (hot to [cold_storage], archive, delete), and a compaction job spec on the [compaction_schedule].
Pro tips
- Keep Bronze strictly append-only and in
[raw_format]as received — the moment you start cleaning data in Bronze, you lose your replayable source of truth. - Tune
[target_partition_size]toward the 128 MB–1 GB range; partitions far smaller than that create the small-file problem that compaction then has to undo. - Match
[partition_scheme]to query patterns (date for time-series, source for multi-tenant) — partitioning on a column nobody filters on wastes layout effort. - Put real PII classification in the
[catalog_tool]entries so the[access_model]column- and row-level rules have something to enforce against. - Set lifecycle thresholds (
[hot_retention], archive, delete) from access patterns and compliance, not arbitrary round numbers — a too-short hot window pushes active data into slow cold storage. - Schedule
[compaction_schedule]more often for Bronze than Silver, as the default suggests, since raw ingestion is where small files accumulate fastest. - Pick
[analytics_format]like Parquet for Silver and Gold so analytical queries read columnar data efficiently, and keep[compression_codec]consistent so downstream engines don't trip on mixed encodings. - Register every dataset in
[catalog_tool]with owner and freshness from the start; an uncataloged lake is the first step toward one nobody can navigate.
