data-quality-frameworks

Name: data-quality-frameworks
Author: wshobson

8.6kinstalls4.1kforks

How to install data-quality-frameworks?

wshobson/agents/data-quality-frameworks

$npx -y skills add wshobson/agents --skill data-quality-frameworks

Installs into the current project.

Summary

Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.

Files · 1

View on GitHub

SKILL.md

1	# Data Quality Frameworks
2
3	Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.
4
5	## When to Use This Skill
6
7	- Implementing data quality checks in pipelines
8	- Setting up Great Expectations validation
9	- Building comprehensive dbt test suites
10	- Establishing data contracts between teams
11	- Monitoring data quality metrics
12	- Automating data validation in CI/CD
13
14	## Core Concepts
15
16	### 1. Data Quality Dimensions
17
18	\| Dimension \| Description \| Example Check \|
19	\| ---------------- \| ------------------------ \| -------------------------------------------------- \|
20	\| Completeness \| No missing values \| `expect_column_values_to_not_be_null` \|
21	\| Uniqueness \| No duplicates \| `expect_column_values_to_be_unique` \|
22	\| Validity \| Values in expected range \| `expect_column_values_to_be_in_set` \|
23	\| Accuracy \| Data matches reality \| Cross-reference validation \|
24	\| Consistency \| No contradictions \| `expect_column_pair_values_A_to_be_greater_than_B` \|
25	\| Timeliness \| Data is recent \| `expect_column_max_to_be_between` \|
26
27	### 2. Testing Pyramid for Data
28
29	```
30	/\
31	/ \ Integration Tests (cross-table)
32	/────\
33	/ \ Unit Tests (single column)
34	/────────\
35	/ \ Schema Tests (structure)
36	/────────────\
37	```
38
39	## Quick Start
40
41	### Great Expectations Setup
42
43	```bash
44	# Install
45	pip install great_expectations
46
47	# Initialize project
48	great_expectations init
49
50	# Create datasource
51	great_expectations datasource new
52	```
53
54	```python
55	# great_expectations/checkpoints/daily_validation.yml
56	import great_expectations as gx
57
58	# Create context
59	context = gx.get_context()
60
61	# Create expectation suite
62	suite = context.add_expectation_suite("orders_suite")
63
64	# Add expectations
65	suite.add_expectation(
66	gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id")
67	)
68	suite.add_expectation(
69	gx.expectations.ExpectColumnValuesToBeUnique(column="order_id")
70	)
71
72	# Validate
73	results = context.run_checkpoint(checkpoint_name="daily_orders")
74	```
75
76	## Detailed patterns and worked examples
77
78	Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
79
80	## Summary: {total_passed}/{total_tables} tables passed")
81	report.append("")
82
83	for table, result in results.items():
84	status = "✅" if result.passed else "❌"
85	report.append(f"### {status} {table}")
86	report.append(f"- Expectations: {result.total_expectations}")
87	report.append(f"- Failed: {result.failed_expectations}")
88
89	if not result.passed:
90	report.append("- Failed checks:")
91	for detail in result.details:
92	if not detail["success"]:
93	report.append(f" - {detail['expectation']}: {detail['observed_value']}")
94	report.append("")
95
96	return "\n".join(report)
97
98	# Usage
99	context = gx.get_context()
100	pipeline = DataQualityPipeline(context)
101
102	tables_to_validate = {
103	"orders": "orders_suite",
104	"customers": "customers_suite",
105	"products": "products_suite",
106	}
107
108	results = pipeline.run_all(tables_to_validate)
109	report = pipeline.generate_report(results)
110
111	# Fail pipeline if any table failed
112	if not all(r.passed for r in results.values()):
113	print(report)
114	raise ValueError("Data quality checks failed!")
115	```
116
117	## Best Practices
118
119	### Do's
120
121	- Test early - Validate source data before transformations
122	- Test incrementally - Add tests as you find issues
123	- Document expectations - Clear descriptions for each test
124	- Alert on failures - Integrate with monitoring
125	- Version contracts - Track schema changes
126
127	### Don'ts
128
129	- Don't test everything - Focus on critical columns
130	- Don't ignore warnings - They often precede failures
131	- Don't skip freshness - Stale data is bad data
132	- Don't hardcode thresholds - Use dynamic baselines
133	- Don't test in isolation - Test relationships too

People who liked this also viewed

6 picks

Type

data-quality-frameworks

8.6kinstalls4.1kforks

View on GitHub

How to install data-quality-frameworks?

wshobson/agents/data-quality-frameworks

$npx -y skills add wshobson/agents --skill data-quality-frameworks

Installs into the current project.

Summary

Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.

Files · 1

View on GitHub

SKILL.md

1	# Data Quality Frameworks
2
3	Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.
4
5	## When to Use This Skill
6
7	- Implementing data quality checks in pipelines
8	- Setting up Great Expectations validation
9	- Building comprehensive dbt test suites
10	- Establishing data contracts between teams
11	- Monitoring data quality metrics
12	- Automating data validation in CI/CD
13
14	## Core Concepts
15
16	### 1. Data Quality Dimensions
17
18	\| Dimension \| Description \| Example Check \|
19	\| ---------------- \| ------------------------ \| -------------------------------------------------- \|
20	\| Completeness \| No missing values \| `expect_column_values_to_not_be_null` \|
21	\| Uniqueness \| No duplicates \| `expect_column_values_to_be_unique` \|
22	\| Validity \| Values in expected range \| `expect_column_values_to_be_in_set` \|
23	\| Accuracy \| Data matches reality \| Cross-reference validation \|
24	\| Consistency \| No contradictions \| `expect_column_pair_values_A_to_be_greater_than_B` \|
25	\| Timeliness \| Data is recent \| `expect_column_max_to_be_between` \|
26
27	### 2. Testing Pyramid for Data
28
29	```
30	/\
31	/ \ Integration Tests (cross-table)
32	/────\
33	/ \ Unit Tests (single column)
34	/────────\
35	/ \ Schema Tests (structure)
36	/────────────\
37	```
38
39	## Quick Start
40
41	### Great Expectations Setup
42
43	```bash
44	# Install
45	pip install great_expectations
46
47	# Initialize project
48	great_expectations init
49
50	# Create datasource
51	great_expectations datasource new
52	```
53
54	```python
55	# great_expectations/checkpoints/daily_validation.yml
56	import great_expectations as gx
57
58	# Create context
59	context = gx.get_context()
60
61	# Create expectation suite
62	suite = context.add_expectation_suite("orders_suite")
63
64	# Add expectations
65	suite.add_expectation(
66	gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id")
67	)
68	suite.add_expectation(
69	gx.expectations.ExpectColumnValuesToBeUnique(column="order_id")
70	)
71
72	# Validate
73	results = context.run_checkpoint(checkpoint_name="daily_orders")
74	```
75
76	## Detailed patterns and worked examples
77
78	Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
79
80	## Summary: {total_passed}/{total_tables} tables passed")
81	report.append("")
82
83	for table, result in results.items():
84	status = "✅" if result.passed else "❌"
85	report.append(f"### {status} {table}")
86	report.append(f"- Expectations: {result.total_expectations}")
87	report.append(f"- Failed: {result.failed_expectations}")
88
89	if not result.passed:
90	report.append("- Failed checks:")
91	for detail in result.details:
92	if not detail["success"]:
93	report.append(f" - {detail['expectation']}: {detail['observed_value']}")
94	report.append("")
95
96	return "\n".join(report)
97
98	# Usage
99	context = gx.get_context()
100	pipeline = DataQualityPipeline(context)
101
102	tables_to_validate = {
103	"orders": "orders_suite",
104	"customers": "customers_suite",
105	"products": "products_suite",
106	}
107
108	results = pipeline.run_all(tables_to_validate)
109	report = pipeline.generate_report(results)
110
111	# Fail pipeline if any table failed
112	if not all(r.passed for r in results.values()):
113	print(report)
114	raise ValueError("Data quality checks failed!")
115	```
116
117	## Best Practices
118
119	### Do's
120
121	- Test early - Validate source data before transformations
122	- Test incrementally - Add tests as you find issues
123	- Document expectations - Clear descriptions for each test
124	- Alert on failures - Integrate with monitoring
125	- Version contracts - Track schema changes
126
127	### Don'ts
128
129	- Don't test everything - Focus on critical columns
130	- Don't ignore warnings - They often precede failures
131	- Don't skip freshness - Stale data is bad data
132	- Don't hardcode thresholds - Use dynamic baselines
133	- Don't test in isolation - Test relationships too

People who liked this also viewed

6 picks

Type