great-expectations-deequ-and-cuallee

Name: great-expectations-deequ-and-cuallee
Author: vaquarkhan

19stars4forks

How to install great-expectations-deequ-and-cuallee?

vaquarkhan/data-engineering-agent-skills/great-expectations-deequ-and-cuallee

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill great-expectations-deequ-and-cuallee

Installs into the current project.

Summary

Guides agents through data-quality frameworks such as Great Expectations, Deequ, and Cuallee. Use when implementing framework-based validation suites, reusable checks, or evidence-driven data-quality enforcement.

Files · 1

View on GitHub

SKILL.md

1	# Great Expectations Deequ And Cuallee
2
3	## Overview
4
5	Use this skill when the team wants structured quality enforcement through a data-quality framework instead of ad hoc checks. It helps agents align contracts with expectation suites, route failures by severity, and produce reviewable validation evidence.
6
7	## When to Use
8
9	- implementing `Great Expectations` expectation suites or checkpoints
10	- adding `Deequ` analyzers and constraints to Spark pipelines
11	- using `Cuallee` for lightweight PySpark or Pandas validation
12	- standardizing reusable quality checks across datasets
13	- integrating framework-based quality gates into CI/CD or publish workflows
14	- producing validation evidence for audits or release gates
15
16	Do not use this when a few inline assertions are sufficient or when the team has no plan to reuse checks across datasets.
17
18	## Workflow
19
20	1. Ground the validation suite in a real contract.
21	- start from the dataset contract: grain, freshness, allowed nulls, value ranges
22	- do not invent checks that are not tied to a documented expectation
23	- map each contract field to one or more framework checks
24	- classify checks by severity: critical (blocks publish), warning (alert only), informational
25
26	2. Choose the right framework for the runtime context.
27	- `Great Expectations`: best for warehouse/lake validation with rich documentation output
28	- `Deequ`: best for Spark pipelines with compile-time constraint definitions
29	- `Cuallee`: best for lightweight validation in PySpark or Pandas without heavy setup
30	- consider execution environment: batch, streaming micro-batch, or CI tests
31	- avoid framework lock-in by keeping contract definitions separate from framework syntax
32
33	3. Design expectation suites with maintenance in mind.
34	- organize expectations by dataset and domain, not by framework capability
35	- keep suites small and focused — one suite per dataset or model output
36	- parameterize thresholds so they can be adjusted without code changes
37	- version suites alongside the pipeline code that produces the data
38	- document when and why each expectation was added
39
40	4. Integrate validation into the pipeline lifecycle.
41	- run validation after transformation, before publish or downstream consumption
42	- critical failures must block publish and trigger alerts
43	- warning failures must be logged and visible but do not block
44	- produce structured evidence output (JSON, HTML, or YAML) for each run
45	- store validation results for historical trending and audit
46
47	5. Make evidence reviewable and actionable.
48	- validation output should be human-readable, not just pass/fail
49	- include row counts, failure percentages, and sample failing records
50	- route evidence to dashboards, Slack, or incident workflows based on severity
51	- link validation results to the specific pipeline run and data partition
52
53	6. Plan for expectation evolution and false positives.
54	- as data changes, expectations need updates — budget for this work
55	- define a process for reviewing and tuning noisy expectations
56	- distinguish true data issues from expectation drift
57	- retire expectations that no longer match the active contract
58
59	## Common Rationalizations
60
61	\| Rationalization \| Reality \|
62	\| --- \| --- \|
63	\| "We'll add expectations later when we have time." \| Quality frameworks deliver value only when integrated during pipeline development. Retrofitting is harder and often abandoned. \|
64	\| "Every column needs an expectation." \| Over-instrumented suites create noise. Expectations should be grounded in contract requirements, not framework capability. \|
65	\| "The framework handles severity for us." \| Severity classification requires human judgment about business impact. Frameworks provide mechanics, not decisions. \|
66	\| "Validation passed so the data is correct." \| Validation proves conformance to documented expectations. It cannot catch requirements that were never specified. \|
67
68	## Red Flags
69
70	- expectation suites with no connection to a dataset contract
71	- all checks are critical severity with no triage or routing plan
72	- validation output is discarded after each run — no historical evidence
73	- suites are never updated when contracts change
74	- framework is chosen based on hype instead of runtime compatibility
75	- validation runs after publish, not before
76	- no plan for handling noisy or flapping expectations
77	- checks exist but nobody reviews failures or trends
78
79	## Verification
80
81	- [ ] Every expectation maps to a documented contract field or business rule
82	- [ ] Severity levels are defined: critical blocks publish, warning alerts, informational logs
83	- [ ] Framework choice matches the runtime context (Spark, warehouse, Pandas)
84	- [ ] Validation runs before publish and produ

People who liked this also viewed

6 picks

Type

great-expectations-deequ-and-cuallee

19stars4forks

View on GitHub

How to install great-expectations-deequ-and-cuallee?

vaquarkhan/data-engineering-agent-skills/great-expectations-deequ-and-cuallee

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill great-expectations-deequ-and-cuallee

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Great Expectations Deequ And Cuallee
2
3	## Overview
4
5	Use this skill when the team wants structured quality enforcement through a data-quality framework instead of ad hoc checks. It helps agents align contracts with expectation suites, route failures by severity, and produce reviewable validation evidence.
6
7	## When to Use
8
9	- implementing `Great Expectations` expectation suites or checkpoints
10	- adding `Deequ` analyzers and constraints to Spark pipelines
11	- using `Cuallee` for lightweight PySpark or Pandas validation
12	- standardizing reusable quality checks across datasets
13	- integrating framework-based quality gates into CI/CD or publish workflows
14	- producing validation evidence for audits or release gates
15
16	Do not use this when a few inline assertions are sufficient or when the team has no plan to reuse checks across datasets.
17
18	## Workflow
19
20	1. Ground the validation suite in a real contract.
21	- start from the dataset contract: grain, freshness, allowed nulls, value ranges
22	- do not invent checks that are not tied to a documented expectation
23	- map each contract field to one or more framework checks
24	- classify checks by severity: critical (blocks publish), warning (alert only), informational
25
26	2. Choose the right framework for the runtime context.
27	- `Great Expectations`: best for warehouse/lake validation with rich documentation output
28	- `Deequ`: best for Spark pipelines with compile-time constraint definitions
29	- `Cuallee`: best for lightweight validation in PySpark or Pandas without heavy setup
30	- consider execution environment: batch, streaming micro-batch, or CI tests
31	- avoid framework lock-in by keeping contract definitions separate from framework syntax
32
33	3. Design expectation suites with maintenance in mind.
34	- organize expectations by dataset and domain, not by framework capability
35	- keep suites small and focused — one suite per dataset or model output
36	- parameterize thresholds so they can be adjusted without code changes
37	- version suites alongside the pipeline code that produces the data
38	- document when and why each expectation was added
39
40	4. Integrate validation into the pipeline lifecycle.
41	- run validation after transformation, before publish or downstream consumption
42	- critical failures must block publish and trigger alerts
43	- warning failures must be logged and visible but do not block
44	- produce structured evidence output (JSON, HTML, or YAML) for each run
45	- store validation results for historical trending and audit
46
47	5. Make evidence reviewable and actionable.
48	- validation output should be human-readable, not just pass/fail
49	- include row counts, failure percentages, and sample failing records
50	- route evidence to dashboards, Slack, or incident workflows based on severity
51	- link validation results to the specific pipeline run and data partition
52
53	6. Plan for expectation evolution and false positives.
54	- as data changes, expectations need updates — budget for this work
55	- define a process for reviewing and tuning noisy expectations
56	- distinguish true data issues from expectation drift
57	- retire expectations that no longer match the active contract
58
59	## Common Rationalizations
60
61	\| Rationalization \| Reality \|
62	\| --- \| --- \|
63	\| "We'll add expectations later when we have time." \| Quality frameworks deliver value only when integrated during pipeline development. Retrofitting is harder and often abandoned. \|
64	\| "Every column needs an expectation." \| Over-instrumented suites create noise. Expectations should be grounded in contract requirements, not framework capability. \|
65	\| "The framework handles severity for us." \| Severity classification requires human judgment about business impact. Frameworks provide mechanics, not decisions. \|
66	\| "Validation passed so the data is correct." \| Validation proves conformance to documented expectations. It cannot catch requirements that were never specified. \|
67
68	## Red Flags
69
70	- expectation suites with no connection to a dataset contract
71	- all checks are critical severity with no triage or routing plan
72	- validation output is discarded after each run — no historical evidence
73	- suites are never updated when contracts change
74	- framework is chosen based on hype instead of runtime compatibility
75	- validation runs after publish, not before
76	- no plan for handling noisy or flapping expectations
77	- checks exist but nobody reviews failures or trends
78
79	## Verification
80
81	- [ ] Every expectation maps to a documented contract field or business rule
82	- [ ] Severity levels are defined: critical blocks publish, warning alerts, informational logs
83	- [ ] Framework choice matches the runtime context (Spark, warehouse, Pandas)
84	- [ ] Validation runs before publish and produ

People who liked this also viewed

6 picks

Type