data-quality-gate

Name: data-quality-gate
Author: zakelfassi

17stars0forks

How to install data-quality-gate?

zakelfassi/skills-driven-development/data-quality-gate

$npx -y skills add zakelfassi/skills-driven-development --skill data-quality-gate

Installs into the current project.

Summary

Add or extend data validation checks in the pipeline when data quality issues are detected — null-percentage thresholds, value ranges, referential integrity, and row-count sanity checks. Use when a pipeline breaks on bad data, when onboarding a new dataset, or when asked to "add

Files · 1

View on GitHub

SKILL.md

1	# Data Quality Gate
2
3	Add or extend validation checks that block bad data from propagating through the pipeline.
4
5	## Inputs
6	- Stage name where the gate should run (e.g., `customer_events`)
7	- Check type: `null_rate` \| `range` \| `referential` \| `row_count` \| `custom`
8	- Column(s) affected
9	- Threshold or reference table (depending on check type)
10	- Severity: `warn` (log and continue) or `fail` (halt the pipeline)
11
12	## Steps
13
14	1. Identify where to add the gate
15	Gates run after ingestion (raw → staging) or after a transform (staging → marts).
16	Most gaps are caught earliest; prefer adding checks at the earliest stage where the data is available.
17
18	2. Write the check
19
20	Null-rate check:
21	```python
22	def check_null_rate(df, column, threshold=0.05):
23	rate = df[column].isnull().mean()
24	if rate > threshold:
25	raise DataQualityError(
26	f"{column} null rate {rate:.1%} exceeds threshold {threshold:.1%}"
27	)
28	```
29
30	Range check:
31	```python
32	def check_range(df, column, min_val, max_val):
33	out_of_range = df[(df[column] < min_val) \| (df[column] > max_val)]
34	if len(out_of_range) > 0:
35	raise DataQualityError(
36	f"{column}: {len(out_of_range)} rows outside [{min_val}, {max_val}]"
37	)
38	```
39
40	Referential integrity check:
41	```python
42	def check_referential(df, fk_column, reference_df, pk_column):
43	orphans = df[~df[fk_column].isin(reference_df[pk_column])]
44	if len(orphans) > 0:
45	raise DataQualityError(
46	f"{fk_column}: {len(orphans)} rows with no matching {pk_column}"
47	)
48	```
49
50	Row-count sanity check:
51	```python
52	def check_row_count(df, min_rows, max_rows=None):
53	n = len(df)
54	if n < min_rows:
55	raise DataQualityError(f"Only {n} rows; expected at least {min_rows}")
56	if max_rows and n > max_rows:
57	raise DataQualityError(f"{n} rows exceeds max {max_rows}")
58	```
59
60	3. Register the check in the stage's test suite
61	Add the check to `pipelines/ingestion/{stage_name}/tests/test_quality.py` or the dbt schema YAML:
62	```yaml
63	# dbt schema
64	- name: {column}
65	tests:
66	- not_null
67	- dbt_utils.accepted_range:
68	min_value: {min}
69	max_value: {max}
70	```
71
72	4. Set the severity
73	- `warn`: emit a metric + log; allow the pipeline to continue.
74	- `fail`: raise `DataQualityError`; the scheduler marks the run as failed.
75	Document the choice and rationale in the check's docstring.
76
77	5. Test the check
78	Write a test that intentionally violates the threshold and confirms the exception is raised:
79	```python
80	def test_null_rate_fails():
81	bad_df = pd.DataFrame({"col": [None] * 10})
82	with pytest.raises(DataQualityError):
83	check_null_rate(bad_df, "col", threshold=0.05)
84	```
85
86	6. Document in the data dictionary
87	Add a "Quality checks" section to `docs/data-dictionary/{stage_name}.md`:
88	\| Check \| Column \| Threshold \| Severity \|
89	\|-------\|--------\|-----------\|----------\|
90	\| null_rate \| {col} \| < 5% \| fail \|
91
92	## Conventions
93	- All checks are functions in `pipelines/quality/checks.py`; imported by stage test files
94	- `DataQualityError` is defined in `pipelines/quality/errors.py`
95	- Metrics emitted: `data_quality.{stage}.{check}.{column}` (gauge, 0=pass, 1=fail)
96	- Null-rate thresholds are per-column, not global
97	- Referential checks are `warn` severity by default unless the FK is critical for downstream joins
98
99	## Edge Cases
100	- Threshold is too tight and causes false positives: Increase the threshold and document the change in git with a comment; don't just bump it silently.
101	- Data arrives in batches (some partitions are empty): Add a `min_rows_per_partition` check rather than a total row-count check.
102	- Reference table is unavailable (API down): Wrap the referential check in a try/except; emit a `warn`-level metric and continue rather than failing the pipeline.
103	- Check runs too slowly on large datasets: Sample 10% of rows for range/null checks; document the sampling in the check's docstring.

People who liked this also viewed

6 picks

Type

data-quality-gate

17stars0forks

View on GitHub

How to install data-quality-gate?

zakelfassi/skills-driven-development/data-quality-gate

$npx -y skills add zakelfassi/skills-driven-development --skill data-quality-gate

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Data Quality Gate
2
3	Add or extend validation checks that block bad data from propagating through the pipeline.
4
5	## Inputs
6	- Stage name where the gate should run (e.g., `customer_events`)
7	- Check type: `null_rate` \| `range` \| `referential` \| `row_count` \| `custom`
8	- Column(s) affected
9	- Threshold or reference table (depending on check type)
10	- Severity: `warn` (log and continue) or `fail` (halt the pipeline)
11
12	## Steps
13
14	1. Identify where to add the gate
15	Gates run after ingestion (raw → staging) or after a transform (staging → marts).
16	Most gaps are caught earliest; prefer adding checks at the earliest stage where the data is available.
17
18	2. Write the check
19
20	Null-rate check:
21	```python
22	def check_null_rate(df, column, threshold=0.05):
23	rate = df[column].isnull().mean()
24	if rate > threshold:
25	raise DataQualityError(
26	f"{column} null rate {rate:.1%} exceeds threshold {threshold:.1%}"
27	)
28	```
29
30	Range check:
31	```python
32	def check_range(df, column, min_val, max_val):
33	out_of_range = df[(df[column] < min_val) \| (df[column] > max_val)]
34	if len(out_of_range) > 0:
35	raise DataQualityError(
36	f"{column}: {len(out_of_range)} rows outside [{min_val}, {max_val}]"
37	)
38	```
39
40	Referential integrity check:
41	```python
42	def check_referential(df, fk_column, reference_df, pk_column):
43	orphans = df[~df[fk_column].isin(reference_df[pk_column])]
44	if len(orphans) > 0:
45	raise DataQualityError(
46	f"{fk_column}: {len(orphans)} rows with no matching {pk_column}"
47	)
48	```
49
50	Row-count sanity check:
51	```python
52	def check_row_count(df, min_rows, max_rows=None):
53	n = len(df)
54	if n < min_rows:
55	raise DataQualityError(f"Only {n} rows; expected at least {min_rows}")
56	if max_rows and n > max_rows:
57	raise DataQualityError(f"{n} rows exceeds max {max_rows}")
58	```
59
60	3. Register the check in the stage's test suite
61	Add the check to `pipelines/ingestion/{stage_name}/tests/test_quality.py` or the dbt schema YAML:
62	```yaml
63	# dbt schema
64	- name: {column}
65	tests:
66	- not_null
67	- dbt_utils.accepted_range:
68	min_value: {min}
69	max_value: {max}
70	```
71
72	4. Set the severity
73	- `warn`: emit a metric + log; allow the pipeline to continue.
74	- `fail`: raise `DataQualityError`; the scheduler marks the run as failed.
75	Document the choice and rationale in the check's docstring.
76
77	5. Test the check
78	Write a test that intentionally violates the threshold and confirms the exception is raised:
79	```python
80	def test_null_rate_fails():
81	bad_df = pd.DataFrame({"col": [None] * 10})
82	with pytest.raises(DataQualityError):
83	check_null_rate(bad_df, "col", threshold=0.05)
84	```
85
86	6. Document in the data dictionary
87	Add a "Quality checks" section to `docs/data-dictionary/{stage_name}.md`:
88	\| Check \| Column \| Threshold \| Severity \|
89	\|-------\|--------\|-----------\|----------\|
90	\| null_rate \| {col} \| < 5% \| fail \|
91
92	## Conventions
93	- All checks are functions in `pipelines/quality/checks.py`; imported by stage test files
94	- `DataQualityError` is defined in `pipelines/quality/errors.py`
95	- Metrics emitted: `data_quality.{stage}.{check}.{column}` (gauge, 0=pass, 1=fail)
96	- Null-rate thresholds are per-column, not global
97	- Referential checks are `warn` severity by default unless the FK is critical for downstream joins
98
99	## Edge Cases
100	- Threshold is too tight and causes false positives: Increase the threshold and document the change in git with a comment; don't just bump it silently.
101	- Data arrives in batches (some partitions are empty): Add a `min_rows_per_partition` check rather than a total row-count check.
102	- Reference table is unavailable (API down): Wrap the referential check in a try/except; emit a `warn`-level metric and continue rather than failing the pipeline.
103	- Check runs too slowly on large datasets: Sample 10% of rows for range/null checks; document the sampling in the check's docstring.

People who liked this also viewed

6 picks

Type