architecting-data

Name: architecting-data
Author: ancoleman

61installs59forks

How to install architecting-data?

ancoleman/ai-design-components/architecting-data

$npx -y skills add ancoleman/ai-design-components --skill architecting-data

Installs into the current project.

Summary

Strategic guidance for designing modern data platforms, covering storage paradigms (data lake, warehouse, lakehouse), modeling approaches (dimensional, normalized, data vault, wide tables), data mesh principles, and medallion architecture patterns. Use when architecting data plat

Files · 1

View on GitHub

SKILL.md

1	# Data Architecture
2
3	## Purpose
4
5	Guide architects and platform engineers through strategic data architecture decisions for modern cloud-native data platforms.
6
7	## When to Use This Skill
8
9	Invoke this skill when:
10	- Designing a new data platform or modernizing legacy systems
11	- Choosing between data lake, data warehouse, or data lakehouse
12	- Deciding on data modeling approaches (dimensional, normalized, data vault, wide tables)
13	- Evaluating centralized vs data mesh architecture
14	- Selecting open table formats (Apache Iceberg, Delta Lake, Apache Hudi)
15	- Designing medallion architecture (bronze, silver, gold layers)
16	- Implementing data governance and cataloging
17
18	## Core Concepts
19
20	### 1. Storage Paradigms
21
22	Three primary patterns for analytical data storage:
23
24	Data Lake: Centralized repository for raw data at scale
25	- Schema-on-read, cost-optimized ($0.02-0.03/GB/month)
26	- Use when: Diverse data sources, exploratory analytics, ML/AI training data
27
28	Data Warehouse: Structured repository optimized for BI
29	- Schema-on-write, ACID transactions, fast queries
30	- Use when: Known BI requirements, strong governance needed
31
32	Data Lakehouse: Hybrid combining lake flexibility with warehouse reliability
33	- Open table formats (Iceberg, Delta Lake), ACID on object storage
34	- Use when: Mixed BI + ML workloads, cost optimization (60-80% cheaper than warehouse)
35
36	Decision Framework:
37	- BI/Reporting only + Known queries → Data Warehouse
38	- ML/AI primary + Raw data needed → Data Lake or Lakehouse
39	- Mixed BI + ML + Cost optimization → Data Lakehouse (recommended)
40	- Exploratory/Unknown use cases → Data Lake
41
42	For detailed comparison, see [references/storage-paradigms.md](references/storage-paradigms.md).
43
44	### 2. Data Modeling Approaches
45
46	Four primary modeling patterns:
47
48	Dimensional (Kimball): Star/snowflake schemas for BI
49	- Use when: Known query patterns, BI dashboards, trend analysis
50
51	Normalized (3NF): Eliminate redundancy for transactional systems
52	- Use when: OLTP systems, frequent updates, strong consistency
53
54	Data Vault 2.0: Flexible model with complete audit trail
55	- Use when: Compliance requirements, multiple sources, agile warehousing
56
57	Wide Tables: Denormalized, optimized for columnar storage
58	- Use when: ML feature stores, data science notebooks, high-performance dashboards
59
60	Decision Framework:
61	- Analytical (BI) + Known queries → Dimensional (Star Schema)
62	- Transactional (OLTP) → Normalized (3NF)
63	- Compliance/Audit → Data Vault 2.0
64	- Data Science/ML → Wide Tables
65
66	For detailed patterns, see [references/modeling-approaches.md](references/modeling-approaches.md).
67
68	### 3. Data Mesh Principles
69
70	Decentralized architecture for large organizations (>500 people).
71
72	Four Core Principles:
73	1. Domain-oriented decentralization
74	2. Data as a product (SLAs, quality, documentation)
75	3. Self-serve data infrastructure
76	4. Federated computational governance
77
78	Readiness Assessment (Score 1-5 each):
79	1. Domain clarity
80	2. Team maturity
81	3. Platform capability
82	4. Governance maturity
83	5. Scale need
84	6. Organizational buy-in
85
86	Scoring: 24-30: Strong candidate \| 18-23: Hybrid \| 12-17: Build foundation first \| 6-11: Centralized
87
88	Red Flags: Small org (<100 people), unclear domains, no platform team, weak governance
89
90	For full guide, see [references/data-mesh-guide.md](references/data-mesh-guide.md).
91
92	### 4. Medallion Architecture
93
94	Standard lakehouse pattern: Bronze (raw) → Silver (cleaned) → Gold (business-level)
95
96	Bronze Layer: Exact copy of source data, immutable, append-only
97
98	Silver Layer: Validated, deduplicated, typed data
99
100	Gold Layer: Business logic, aggregates, dimensional models, ML features
101
102	Data Quality by Layer:
103	- Bronze → Silver: Schema validation, type checks, deduplication
104	- Silver → Gold: Business rule validation, referential integrity
105	- Gold: Anomaly detection, statistical checks
106
107	For patterns, see [references/medallion-pattern.md](references/medallion-pattern.md).
108
109	### 5. Open Table Formats
110
111	Enable ACID transactions on data lakes:
112
113	Apache Iceberg: Multi-engine, vendor-neutral (Context7: 79.7 score)
114	- Use when: Avoid vendor lock-in, multi-engine flexibility
115
116	Delta Lake: Databricks ecosystem, Spark-optimized
117	- Use when: Committed to Databricks
118
119	Apache Hudi: Optimized for CDC and frequent upserts
120	- Use when: CDC-heavy workloads
121
122	Recommendation: Apache Iceberg for new projects (vendor-neutral, broadest support)
123
124	For comparison, see [references/table-formats.md](references/table-formats.md).
125
126	### 6. Modern Data Stack
127
128	**Sta

People who liked this also viewed

6 picks

Type

architecting-data

61installs59forks

View on GitHub

How to install architecting-data?

ancoleman/ai-design-components/architecting-data

$npx -y skills add ancoleman/ai-design-components --skill architecting-data

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Data Architecture
2
3	## Purpose
4
5	Guide architects and platform engineers through strategic data architecture decisions for modern cloud-native data platforms.
6
7	## When to Use This Skill
8
9	Invoke this skill when:
10	- Designing a new data platform or modernizing legacy systems
11	- Choosing between data lake, data warehouse, or data lakehouse
12	- Deciding on data modeling approaches (dimensional, normalized, data vault, wide tables)
13	- Evaluating centralized vs data mesh architecture
14	- Selecting open table formats (Apache Iceberg, Delta Lake, Apache Hudi)
15	- Designing medallion architecture (bronze, silver, gold layers)
16	- Implementing data governance and cataloging
17
18	## Core Concepts
19
20	### 1. Storage Paradigms
21
22	Three primary patterns for analytical data storage:
23
24	Data Lake: Centralized repository for raw data at scale
25	- Schema-on-read, cost-optimized ($0.02-0.03/GB/month)
26	- Use when: Diverse data sources, exploratory analytics, ML/AI training data
27
28	Data Warehouse: Structured repository optimized for BI
29	- Schema-on-write, ACID transactions, fast queries
30	- Use when: Known BI requirements, strong governance needed
31
32	Data Lakehouse: Hybrid combining lake flexibility with warehouse reliability
33	- Open table formats (Iceberg, Delta Lake), ACID on object storage
34	- Use when: Mixed BI + ML workloads, cost optimization (60-80% cheaper than warehouse)
35
36	Decision Framework:
37	- BI/Reporting only + Known queries → Data Warehouse
38	- ML/AI primary + Raw data needed → Data Lake or Lakehouse
39	- Mixed BI + ML + Cost optimization → Data Lakehouse (recommended)
40	- Exploratory/Unknown use cases → Data Lake
41
42	For detailed comparison, see [references/storage-paradigms.md](references/storage-paradigms.md).
43
44	### 2. Data Modeling Approaches
45
46	Four primary modeling patterns:
47
48	Dimensional (Kimball): Star/snowflake schemas for BI
49	- Use when: Known query patterns, BI dashboards, trend analysis
50
51	Normalized (3NF): Eliminate redundancy for transactional systems
52	- Use when: OLTP systems, frequent updates, strong consistency
53
54	Data Vault 2.0: Flexible model with complete audit trail
55	- Use when: Compliance requirements, multiple sources, agile warehousing
56
57	Wide Tables: Denormalized, optimized for columnar storage
58	- Use when: ML feature stores, data science notebooks, high-performance dashboards
59
60	Decision Framework:
61	- Analytical (BI) + Known queries → Dimensional (Star Schema)
62	- Transactional (OLTP) → Normalized (3NF)
63	- Compliance/Audit → Data Vault 2.0
64	- Data Science/ML → Wide Tables
65
66	For detailed patterns, see [references/modeling-approaches.md](references/modeling-approaches.md).
67
68	### 3. Data Mesh Principles
69
70	Decentralized architecture for large organizations (>500 people).
71
72	Four Core Principles:
73	1. Domain-oriented decentralization
74	2. Data as a product (SLAs, quality, documentation)
75	3. Self-serve data infrastructure
76	4. Federated computational governance
77
78	Readiness Assessment (Score 1-5 each):
79	1. Domain clarity
80	2. Team maturity
81	3. Platform capability
82	4. Governance maturity
83	5. Scale need
84	6. Organizational buy-in
85
86	Scoring: 24-30: Strong candidate \| 18-23: Hybrid \| 12-17: Build foundation first \| 6-11: Centralized
87
88	Red Flags: Small org (<100 people), unclear domains, no platform team, weak governance
89
90	For full guide, see [references/data-mesh-guide.md](references/data-mesh-guide.md).
91
92	### 4. Medallion Architecture
93
94	Standard lakehouse pattern: Bronze (raw) → Silver (cleaned) → Gold (business-level)
95
96	Bronze Layer: Exact copy of source data, immutable, append-only
97
98	Silver Layer: Validated, deduplicated, typed data
99
100	Gold Layer: Business logic, aggregates, dimensional models, ML features
101
102	Data Quality by Layer:
103	- Bronze → Silver: Schema validation, type checks, deduplication
104	- Silver → Gold: Business rule validation, referential integrity
105	- Gold: Anomaly detection, statistical checks
106
107	For patterns, see [references/medallion-pattern.md](references/medallion-pattern.md).
108
109	### 5. Open Table Formats
110
111	Enable ACID transactions on data lakes:
112
113	Apache Iceberg: Multi-engine, vendor-neutral (Context7: 79.7 score)
114	- Use when: Avoid vendor lock-in, multi-engine flexibility
115
116	Delta Lake: Databricks ecosystem, Spark-optimized
117	- Use when: Committed to Databricks
118
119	Apache Hudi: Optimized for CDC and frequent upserts
120	- Use when: CDC-heavy workloads
121
122	Recommendation: Apache Iceberg for new projects (vendor-neutral, broadest support)
123
124	For comparison, see [references/table-formats.md](references/table-formats.md).
125
126	### 6. Modern Data Stack
127
128	**Sta

People who liked this also viewed

6 picks

Type