dbt-transformation-patterns

Name: dbt-transformation-patterns
Author: wshobson

8.5kinstalls4.1kforks

How to install dbt-transformation-patterns?

wshobson/agents/dbt-transformation-patterns

$npx -y skills add wshobson/agents --skill dbt-transformation-patterns

Installs into the current project.

Summary

Master dbt (data build tool) for analytics engineering with model organization, testing, documentation, and incremental strategies. Use when building data transformations, creating data models, or implementing analytics engineering best practices.

Files · 1

View on GitHub

SKILL.md

1	# dbt Transformation Patterns
2
3	Production-ready patterns for dbt (data build tool) including model organization, testing strategies, documentation, and incremental processing.
4
5	## When to Use This Skill
6
7	- Building data transformation pipelines with dbt
8	- Organizing models into staging, intermediate, and marts layers
9	- Implementing data quality tests
10	- Creating incremental models for large datasets
11	- Documenting data models and lineage
12	- Setting up dbt project structure
13
14	## Core Concepts
15
16	### 1. Model Layers (Medallion Architecture)
17
18	```
19	sources/ Raw data definitions
20	↓
21	staging/ 1:1 with source, light cleaning
22	↓
23	intermediate/ Business logic, joins, aggregations
24	↓
25	marts/ Final analytics tables
26	```
27
28	### 2. Naming Conventions
29
30	\| Layer \| Prefix \| Example \|
31	\| ------------ \| -------------- \| ----------------------------- \|
32	\| Staging \| `stg_` \| `stg_stripe__payments` \|
33	\| Intermediate \| `int_` \| `int_payments_pivoted` \|
34	\| Marts \| `dim_`, `fct_` \| `dim_customers`, `fct_orders` \|
35
36	## Quick Start
37
38	```yaml
39	# dbt_project.yml
40	name: "analytics"
41	version: "1.0.0"
42	profile: "analytics"
43
44	model-paths: ["models"]
45	analysis-paths: ["analyses"]
46	test-paths: ["tests"]
47	seed-paths: ["seeds"]
48	macro-paths: ["macros"]
49
50	vars:
51	start_date: "2020-01-01"
52
53	models:
54	analytics:
55	staging:
56	+materialized: view
57	+schema: staging
58	intermediate:
59	+materialized: ephemeral
60	marts:
61	+materialized: table
62	+schema: analytics
63	```
64
65	```
66	# Project structure
67	models/
68	├── staging/
69	│ ├── stripe/
70	│ │ ├── _stripe__sources.yml
71	│ │ ├── _stripe__models.yml
72	│ │ ├── stg_stripe__customers.sql
73	│ │ └── stg_stripe__payments.sql
74	│ └── shopify/
75	│ ├── _shopify__sources.yml
76	│ └── stg_shopify__orders.sql
77	├── intermediate/
78	│ └── finance/
79	│ └── int_payments_pivoted.sql
80	└── marts/
81	├── core/
82	│ ├── _core__models.yml
83	│ ├── dim_customers.sql
84	│ └── fct_orders.sql
85	└── finance/
86	└── fct_revenue.sql
87	```
88
89	## Detailed patterns and worked examples
90
91	Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
92
93	## Best Practices
94
95	### Do's
96
97	- Use staging layer - Clean data once, use everywhere
98	- Test aggressively - Not null, unique, relationships
99	- Document everything - Column descriptions, model descriptions
100	- Use incremental - For tables > 1M rows
101	- Version control - dbt project in Git
102
103	### Don'ts
104
105	- Don't skip staging - Raw → mart is tech debt
106	- Don't hardcode dates - Use `{{ var('start_date') }}`
107	- Don't repeat logic - Extract to macros
108	- Don't test in prod - Use dev target
109	- Don't ignore freshness - Monitor source data

People who liked this also viewed

6 picks

Type

dbt-transformation-patterns

8.5kinstalls4.1kforks

View on GitHub

How to install dbt-transformation-patterns?

wshobson/agents/dbt-transformation-patterns

$npx -y skills add wshobson/agents --skill dbt-transformation-patterns

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# dbt Transformation Patterns
2
3	Production-ready patterns for dbt (data build tool) including model organization, testing strategies, documentation, and incremental processing.
4
5	## When to Use This Skill
6
7	- Building data transformation pipelines with dbt
8	- Organizing models into staging, intermediate, and marts layers
9	- Implementing data quality tests
10	- Creating incremental models for large datasets
11	- Documenting data models and lineage
12	- Setting up dbt project structure
13
14	## Core Concepts
15
16	### 1. Model Layers (Medallion Architecture)
17
18	```
19	sources/ Raw data definitions
20	↓
21	staging/ 1:1 with source, light cleaning
22	↓
23	intermediate/ Business logic, joins, aggregations
24	↓
25	marts/ Final analytics tables
26	```
27
28	### 2. Naming Conventions
29
30	\| Layer \| Prefix \| Example \|
31	\| ------------ \| -------------- \| ----------------------------- \|
32	\| Staging \| `stg_` \| `stg_stripe__payments` \|
33	\| Intermediate \| `int_` \| `int_payments_pivoted` \|
34	\| Marts \| `dim_`, `fct_` \| `dim_customers`, `fct_orders` \|
35
36	## Quick Start
37
38	```yaml
39	# dbt_project.yml
40	name: "analytics"
41	version: "1.0.0"
42	profile: "analytics"
43
44	model-paths: ["models"]
45	analysis-paths: ["analyses"]
46	test-paths: ["tests"]
47	seed-paths: ["seeds"]
48	macro-paths: ["macros"]
49
50	vars:
51	start_date: "2020-01-01"
52
53	models:
54	analytics:
55	staging:
56	+materialized: view
57	+schema: staging
58	intermediate:
59	+materialized: ephemeral
60	marts:
61	+materialized: table
62	+schema: analytics
63	```
64
65	```
66	# Project structure
67	models/
68	├── staging/
69	│ ├── stripe/
70	│ │ ├── _stripe__sources.yml
71	│ │ ├── _stripe__models.yml
72	│ │ ├── stg_stripe__customers.sql
73	│ │ └── stg_stripe__payments.sql
74	│ └── shopify/
75	│ ├── _shopify__sources.yml
76	│ └── stg_shopify__orders.sql
77	├── intermediate/
78	│ └── finance/
79	│ └── int_payments_pivoted.sql
80	└── marts/
81	├── core/
82	│ ├── _core__models.yml
83	│ ├── dim_customers.sql
84	│ └── fct_orders.sql
85	└── finance/
86	└── fct_revenue.sql
87	```
88
89	## Detailed patterns and worked examples
90
91	Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
92
93	## Best Practices
94
95	### Do's
96
97	- Use staging layer - Clean data once, use everywhere
98	- Test aggressively - Not null, unique, relationships
99	- Document everything - Column descriptions, model descriptions
100	- Use incremental - For tables > 1M rows
101	- Version control - dbt project in Git
102
103	### Don'ts
104
105	- Don't skip staging - Raw → mart is tech debt
106	- Don't hardcode dates - Use `{{ var('start_date') }}`
107	- Don't repeat logic - Extract to macros
108	- Don't test in prod - Use dev target
109	- Don't ignore freshness - Monitor source data

People who liked this also viewed

6 picks

Type