delta-lake-and-medallion-architecture

Name: delta-lake-and-medallion-architecture
Author: vaquarkhan

19stars4forks

How to install delta-lake-and-medallion-architecture?

vaquarkhan/data-engineering-agent-skills/delta-lake-and-medallion-architecture

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill delta-lake-and-medallion-architecture

Installs into the current project.

Summary

Guides agents through Delta Lake and medallion-style lakehouse design. Use when building or modifying bronze, silver, and gold layers, Delta Lake mutation patterns, streaming-to-batch lakehouse flows, or Databricks-centered serving architectures.

Files · 1

View on GitHub

SKILL.md

1	# Delta Lake And Medallion Architecture
2
3	## Overview
4
5	Use this skill when a `Delta Lake`-based lakehouse needs more than generic table-format guidance. It helps agents design medallion-style layers, transactional update behavior, CDC merges, and publish-safe transformation paths for Delta-centric platforms.
6
7	## When to Use
8
9	- designing `bronze`, `silver`, and `gold` layering
10	- implementing `Delta Lake` merges, upserts, or deletes
11	- building `Databricks`-centered batch and streaming lakehouse flows
12	- deciding how raw landing data evolves into trusted publish outputs
13
14	Do not use medallion terminology as decoration if the layers do not carry distinct responsibilities.
15
16	## Workflow
17
18	1. Define the layer responsibilities.
19	Typical pattern:
20	- bronze: raw or lightly standardized landing
21	- silver: cleaned, conformed, contract-aware transformations
22	- gold: business-facing publish-ready outputs
23
24	2. Define movement rules between layers.
25	Capture:
26	- validation requirements
27	- mutation behavior
28	- CDC merge strategy
29	- schema enforcement and evolution policy
30
31	3. Keep bronze permissive and gold disciplined.
32	Raw survival and publish trust need different operating rules.
33
34	4. Coordinate batch and streaming writers carefully.
35	Checkpoints, merges, and small-file behavior must support recovery and maintenance.
36
37	5. Plan maintenance as part of the architecture.
38	Include:
39	- compaction
40	- retention
41	- optimization
42	- cleanup
43
44	## Common Rationalizations
45
46	\| Rationalization \| Reality \|
47	\| --- \| --- \|
48	\| "Bronze, silver, and gold are enough architecture by themselves." \| Layer names without contracts and rules create confusion, not design clarity. \|
49	\| "We can merge everything directly into gold." \| Business-facing layers need stronger validation and stability than raw ingestion paths. \|
50	\| "Delta transactions solve every operational problem." \| You still need thoughtful layering, maintenance, and replay design. \|
51
52	## Red Flags
53
54	- bronze, silver, and gold have no explicit responsibilities
55	- gold tables are fed directly from unstable raw landing logic
56	- merge semantics are unclear for CDC or late data
57	- maintenance tasks such as compaction are ignored
58
59	## Verification
60
61	- [ ] Each medallion layer has a distinct purpose and quality bar
62	- [ ] Movement, validation, and mutation rules between layers are explicit
63	- [ ] Streaming and batch interaction with Delta tables is operationally safe
64	- [ ] Maintenance and lifecycle behavior are part of the design

People who liked this also viewed

6 picks

Type

delta-lake-and-medallion-architecture

19stars4forks

View on GitHub

How to install delta-lake-and-medallion-architecture?

vaquarkhan/data-engineering-agent-skills/delta-lake-and-medallion-architecture

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill delta-lake-and-medallion-architecture

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Delta Lake And Medallion Architecture
2
3	## Overview
4
5	Use this skill when a `Delta Lake`-based lakehouse needs more than generic table-format guidance. It helps agents design medallion-style layers, transactional update behavior, CDC merges, and publish-safe transformation paths for Delta-centric platforms.
6
7	## When to Use
8
9	- designing `bronze`, `silver`, and `gold` layering
10	- implementing `Delta Lake` merges, upserts, or deletes
11	- building `Databricks`-centered batch and streaming lakehouse flows
12	- deciding how raw landing data evolves into trusted publish outputs
13
14	Do not use medallion terminology as decoration if the layers do not carry distinct responsibilities.
15
16	## Workflow
17
18	1. Define the layer responsibilities.
19	Typical pattern:
20	- bronze: raw or lightly standardized landing
21	- silver: cleaned, conformed, contract-aware transformations
22	- gold: business-facing publish-ready outputs
23
24	2. Define movement rules between layers.
25	Capture:
26	- validation requirements
27	- mutation behavior
28	- CDC merge strategy
29	- schema enforcement and evolution policy
30
31	3. Keep bronze permissive and gold disciplined.
32	Raw survival and publish trust need different operating rules.
33
34	4. Coordinate batch and streaming writers carefully.
35	Checkpoints, merges, and small-file behavior must support recovery and maintenance.
36
37	5. Plan maintenance as part of the architecture.
38	Include:
39	- compaction
40	- retention
41	- optimization
42	- cleanup
43
44	## Common Rationalizations
45
46	\| Rationalization \| Reality \|
47	\| --- \| --- \|
48	\| "Bronze, silver, and gold are enough architecture by themselves." \| Layer names without contracts and rules create confusion, not design clarity. \|
49	\| "We can merge everything directly into gold." \| Business-facing layers need stronger validation and stability than raw ingestion paths. \|
50	\| "Delta transactions solve every operational problem." \| You still need thoughtful layering, maintenance, and replay design. \|
51
52	## Red Flags
53
54	- bronze, silver, and gold have no explicit responsibilities
55	- gold tables are fed directly from unstable raw landing logic
56	- merge semantics are unclear for CDC or late data
57	- maintenance tasks such as compaction are ignored
58
59	## Verification
60
61	- [ ] Each medallion layer has a distinct purpose and quality bar
62	- [ ] Movement, validation, and mutation rules between layers are explicit
63	- [ ] Streaming and batch interaction with Delta tables is operationally safe
64	- [ ] Maintenance and lifecycle behavior are part of the design

People who liked this also viewed

6 picks

Type