data-quality-detection

Name: data-quality-detection
Author: kaori-seasons

15stars3forks

How to install data-quality-detection?

kaori-seasons/data-skill-hub/data-quality-detection

$npx -y skills add kaori-seasons/data-skill-hub --skill data-quality-detection

Installs into the current project.

Summary

Use when you need to diagnose data quality problems in tables, SQL pipelines, bridge tables, path fields, enum fields, null-heavy dimensions, or frontend-vs-warehouse discrepancies. This skill traces issues to source pollution, SQL mapping mistakes, missing bridge keys, wrong gra

Files · 1

View on GitHub

SKILL.md

1	# Data Quality Detection
2
3	Use this skill when the task is not just "run null checks", but "explain why the data cannot be trusted and where the break actually happens."
4
5	## What This Skill Solves
6
7	This skill is built for questions like:
8
9	- Which table is safe to use as the analysis base table?
10	- Is the bad value already present in ODS, or introduced by downstream SQL?
11	- Is the issue caused by source pollution, bridge-table failure, wrong field sourcing, or frontend formatting?
12	- Why do path fields, platform fields, or enum fields look wrong?
13	- Why do planned bridge keys exist in design docs but not in runtime tables?
14
15	## First Principles
16
17	Treat data quality as a decision-support problem, not a formatting problem.
18
19	The skill assumes:
20
21	1. A field is "high quality" only if it can still support stable business decisions.
22	2. A finding is incomplete unless it is attached to a concrete break point.
23	3. Single-table evidence is weak; cross-layer evidence is preferred.
24	4. High fill rate does not imply correct semantics.
25	5. Every major conclusion should be backed by representative samples.
26
27	## Workflow
28
29	### 1. Define the expected contract
30
31	Before querying, define:
32
33	- target grain
34	- key fields
35	- required dimensions
36	- required metrics
37	- derived metrics that should not be mistaken for physical columns
38
39	Read these when relevant:
40
41	- `references/four-table.pdf`
42	- `references/data-quality-report-20260408.md`
43
44	### 2. Confirm the runtime object
45
46	Do not trust only the planned table name.
47
48	Check:
49
50	- whether the table exists
51	- whether the runtime columns match the design
52	- whether a result table is missing and requires fallback to source tables
53	- whether same-name fields are actually sourced from different places
54
55	Useful scripts:
56
57	- `scripts/run_table124_quality_report.py`
58	- `scripts/run_total_hours_sql_field_probe.py`
59	- `scripts/build_sql_data_map.py`
60
61	### 3. Run six layers of checks
62
63	#### Structure
64
65	- schema completeness
66	- type mismatches
67	- comment/documentation gaps
68	- all-nullable design
69
70	#### Value quality
71
72	- null / blank rates
73	- dirty characters
74	- overlong values
75	- path suffix anomalies
76	- file-name vs full-path conflicts
77
78	#### Key and grain
79
80	- duplicate primary keys
81	- one-file-many-rows / one-order-many-rows / one-instance-many-rows
82	- distinct-count collapse
83
84	#### Enum and distribution
85
86	- mixed semantic/process/test values
87	- target/reference enum mismatches
88	- values that appear on only one side
89
90	#### Freshness and temporal sanity
91
92	- min/max timestamps
93	- future dates
94	- extreme stale dates
95	- fields that exist in schema but are empty in live runtime
96
97	#### Cross-layer and lineage
98
99	- source vs downstream consistency
100	- bridge-table breaks
101	- semantic miswrite such as `full_path` written into a folder-like field
102	- frontend display-layer formatting differences
103
104	## Root-Cause Buckets
105
106	Force each major issue into one of these buckets:
107
108	1. Source pollution
109	2. SQL mapping mistake
110	3. Missing bridge key or incomplete bridge table
111	4. Wrong grain or wrong key choice
112	5. Rule coverage gap
113	6. Frontend / interface formatting deviation
114
115	If a finding does not fit a bucket, the diagnosis is still incomplete.
116
117	## Script Selection Guide
118
119	Use the smallest relevant artifact set first.
120
121	### General row-level checks
122
123	- `scripts/run_quality_checks.py`
124	- `scripts/run_7table_quality.py`
125
126	### Planned-vs-runtime bridge validation
127
128	- `scripts/run_table124_quality_report.py`
129
130	### Image/path/source diagnostics
131
132	- `scripts/run_current_version_image_source_dq.py`
133	- `scripts/run_frontend_image_path_lineage_trace.py`
134	- `scripts/run_dim_picture_enum_gap_report.py`
135	- `scripts/run_platform_null_root_cause_probe.py`
136	- `scripts/run_platform_path_layer_scan.py`
137	- `scripts/run_sample_table01_platform_null_diagnosis.py`
138	- `scripts/run_pic_backup_video_content_type_probe.py`
139
140	### SQL / lineage / field-source diagnostics
141
142	- `scripts/build_sql_data_map.py`
143	- `scripts/run_total_hours_sql_field_probe.py`
144
145	## Bundled References
146
147	Read only the ones needed for the current question:
148
149	- `references/data-quality-report-20260408.md`
150	- `references/table124-quality-report-20260416.json`
151	- `references/current-version-image-source-dq-report-20260416.json`
152	- `references/four-table.pdf`
153	- `references/total-hours-field-mapping-report.pdf`
154
155	## Output Standard
156
157	A good result should include:
158
159	1. execution summary
160	2. target grain and key definition
161	3. structural findings
162	4. value-level findings
163	5. lineage/bridge findings
164	6. root-cause classification
165	7. representative samples
166	8. remediation priority
167	9. explicit uncertainty or counterarguments
168
169	## Self-Critique Checklist
170
171	Before finalizing, challenge your own di

People who liked this also viewed

6 picks

Type

data-quality-detection

15stars3forks

View on GitHub

How to install data-quality-detection?

kaori-seasons/data-skill-hub/data-quality-detection

$npx -y skills add kaori-seasons/data-skill-hub --skill data-quality-detection

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Data Quality Detection
2
3	Use this skill when the task is not just "run null checks", but "explain why the data cannot be trusted and where the break actually happens."
4
5	## What This Skill Solves
6
7	This skill is built for questions like:
8
9	- Which table is safe to use as the analysis base table?
10	- Is the bad value already present in ODS, or introduced by downstream SQL?
11	- Is the issue caused by source pollution, bridge-table failure, wrong field sourcing, or frontend formatting?
12	- Why do path fields, platform fields, or enum fields look wrong?
13	- Why do planned bridge keys exist in design docs but not in runtime tables?
14
15	## First Principles
16
17	Treat data quality as a decision-support problem, not a formatting problem.
18
19	The skill assumes:
20
21	1. A field is "high quality" only if it can still support stable business decisions.
22	2. A finding is incomplete unless it is attached to a concrete break point.
23	3. Single-table evidence is weak; cross-layer evidence is preferred.
24	4. High fill rate does not imply correct semantics.
25	5. Every major conclusion should be backed by representative samples.
26
27	## Workflow
28
29	### 1. Define the expected contract
30
31	Before querying, define:
32
33	- target grain
34	- key fields
35	- required dimensions
36	- required metrics
37	- derived metrics that should not be mistaken for physical columns
38
39	Read these when relevant:
40
41	- `references/four-table.pdf`
42	- `references/data-quality-report-20260408.md`
43
44	### 2. Confirm the runtime object
45
46	Do not trust only the planned table name.
47
48	Check:
49
50	- whether the table exists
51	- whether the runtime columns match the design
52	- whether a result table is missing and requires fallback to source tables
53	- whether same-name fields are actually sourced from different places
54
55	Useful scripts:
56
57	- `scripts/run_table124_quality_report.py`
58	- `scripts/run_total_hours_sql_field_probe.py`
59	- `scripts/build_sql_data_map.py`
60
61	### 3. Run six layers of checks
62
63	#### Structure
64
65	- schema completeness
66	- type mismatches
67	- comment/documentation gaps
68	- all-nullable design
69
70	#### Value quality
71
72	- null / blank rates
73	- dirty characters
74	- overlong values
75	- path suffix anomalies
76	- file-name vs full-path conflicts
77
78	#### Key and grain
79
80	- duplicate primary keys
81	- one-file-many-rows / one-order-many-rows / one-instance-many-rows
82	- distinct-count collapse
83
84	#### Enum and distribution
85
86	- mixed semantic/process/test values
87	- target/reference enum mismatches
88	- values that appear on only one side
89
90	#### Freshness and temporal sanity
91
92	- min/max timestamps
93	- future dates
94	- extreme stale dates
95	- fields that exist in schema but are empty in live runtime
96
97	#### Cross-layer and lineage
98
99	- source vs downstream consistency
100	- bridge-table breaks
101	- semantic miswrite such as `full_path` written into a folder-like field
102	- frontend display-layer formatting differences
103
104	## Root-Cause Buckets
105
106	Force each major issue into one of these buckets:
107
108	1. Source pollution
109	2. SQL mapping mistake
110	3. Missing bridge key or incomplete bridge table
111	4. Wrong grain or wrong key choice
112	5. Rule coverage gap
113	6. Frontend / interface formatting deviation
114
115	If a finding does not fit a bucket, the diagnosis is still incomplete.
116
117	## Script Selection Guide
118
119	Use the smallest relevant artifact set first.
120
121	### General row-level checks
122
123	- `scripts/run_quality_checks.py`
124	- `scripts/run_7table_quality.py`
125
126	### Planned-vs-runtime bridge validation
127
128	- `scripts/run_table124_quality_report.py`
129
130	### Image/path/source diagnostics
131
132	- `scripts/run_current_version_image_source_dq.py`
133	- `scripts/run_frontend_image_path_lineage_trace.py`
134	- `scripts/run_dim_picture_enum_gap_report.py`
135	- `scripts/run_platform_null_root_cause_probe.py`
136	- `scripts/run_platform_path_layer_scan.py`
137	- `scripts/run_sample_table01_platform_null_diagnosis.py`
138	- `scripts/run_pic_backup_video_content_type_probe.py`
139
140	### SQL / lineage / field-source diagnostics
141
142	- `scripts/build_sql_data_map.py`
143	- `scripts/run_total_hours_sql_field_probe.py`
144
145	## Bundled References
146
147	Read only the ones needed for the current question:
148
149	- `references/data-quality-report-20260408.md`
150	- `references/table124-quality-report-20260416.json`
151	- `references/current-version-image-source-dq-report-20260416.json`
152	- `references/four-table.pdf`
153	- `references/total-hours-field-mapping-report.pdf`
154
155	## Output Standard
156
157	A good result should include:
158
159	1. execution summary
160	2. target grain and key definition
161	3. structural findings
162	4. value-level findings
163	5. lineage/bridge findings
164	6. root-cause classification
165	7. representative samples
166	8. remediation priority
167	9. explicit uncertainty or counterarguments
168
169	## Self-Critique Checklist
170
171	Before finalizing, challenge your own di

People who liked this also viewed

6 picks

Type