clean-data

Name: clean-data
Author: Aperivue

208stars53forks

How to install clean-data?

aperivue/medsci-skills/clean-data

$npx -y skills add Aperivue/medsci-skills --skill clean-data

Installs into the current project.

Summary

Interactive data profiling and cleaning assistant for medical research. Three-stage workflow (profile, flag, code-generate) with user approval gates at each step. Handles missing values, outliers, duplicates, and type mismatches in CSV/Excel clinical data. Does NOT auto-clean — a

Files · 1

View on GitHub

SKILL.md

1	# Data Profiling and Cleaning Skill
2
3	You are assisting a medical researcher with data profiling and cleaning for clinical datasets.
4	This is a three-stage interactive workflow. You generate code and reports -- you do NOT
5	auto-clean data. Every cleaning decision requires explicit researcher confirmation.
6
7	## Philosophy
8
9	This skill is a PROFILING AND FLAGGING ASSISTANT, not an automated data cleaner.
10	Clinical data cleaning requires domain expertise that an LLM cannot replace.
11	Every cleaning decision must be confirmed by the researcher.
12
13	DATA PRIVACY WARNING
14
15	If your dataset contains Protected Health Information (PHI) or Personally Identifiable
16	Information (PII), run `/deidentify` first to remove PHI before proceeding. The deidentify
17	skill provides a standalone Python script (no LLM) that scans for Korean SSN, phone numbers,
18	names, dates, and addresses, then anonymizes them with your confirmation.
19
20	If `_deidentified.` files exist in the working directory, use those instead of raw data.
21
22	Alternatively:
23	1. Provide only the data dictionary / codebook for profiling guidance
24	2. Or use a local-only environment with no network access
25
26	This tool generates CODE that runs on your data -- it does not need to see the raw data
27	to generate useful profiling scripts.
28
29	## Reference Files
30
31	- Profiling template: `${CLAUDE_SKILL_DIR}/references/profiling_template.py` -- reusable profiling script
32	- Cleaning patterns: `${CLAUDE_SKILL_DIR}/references/cleaning_patterns.md` -- common clinical data patterns
33	- Implausible-value & cross-field validity rules: `${CLAUDE_SKILL_DIR}/references/implausible_value_rules.md` -- domain-default hard physiologic bounds (per organ system) + cross-field logical-consistency rules for Stage 2 flagging when the codebook is silent (error-screening, not reference ranges; flag, never auto-fix)
34
35	Read relevant references before generating profiling or cleaning code.
36
37	## Three-Stage Workflow
38
39	### Stage 1: Profiling
40
41	Input: CSV/Excel file path OR data dictionary/codebook
42
43	Actions:
44
45	1. Generate a Python profiling script (pandas-based) that produces:
46	- Variable count, row count, data types
47	- Missing value count and percentage per variable
48	- Unique value counts for categorical variables
49	- Min/max/mean/median/SD for numeric variables
50	- Distribution plots (histograms for numeric, bar charts for categorical)
51	2. If user provides a codebook: cross-reference variable names, expected types, expected ranges
52	3. Present summary table to user
53
54	Use `${CLAUDE_SKILL_DIR}/references/profiling_template.py` as the base script. Adapt it to
55	the specific dataset structure.
56
57	Gate: User reviews profiling output before proceeding. Ask:
58	> "Here is the profiling summary. Would you like to proceed to Stage 2 (Flagging)?
59	> Are there any variables you want to exclude or focus on?"
60
61	### Stage 2: Flagging
62
63	Based on profiling results, flag potential issues in these categories:
64
65	1. Missing values: Variables with >5% missing, pattern analysis (MCAR/MAR/MNAR heuristic)
66	2. Statistical outliers: IQR method (Q1 - 1.5IQR, Q3 + 1.5IQR) and Z-score (\|z\| > 3)
67	3. Duplicates: Exact row duplicates AND near-duplicates (same patient ID, different dates)
68	4. Type mismatches: Numeric stored as string, dates in inconsistent formats
69	5. Implausible values: Use the codebook's valid range when provided; when the codebook is silent, apply the domain-default hard physiologic bounds in `references/implausible_value_rules.md` §1 (compatible-with-life screening bounds, per organ system) as a flag-for-review — distinct from statistical outliers (#2): an implausible value is a likely data-entry/unit/sentinel error (correct-or-set-missing), an outlier is biologically possible (keep + sensitivity). Check units before calling a bound violation an error. Never auto-fix.
70	5b. Cross-field inconsistencies: Logical contradictions between fields per `references/implausible_value_rules.md` §2 — temporal ordering (birth ≤ event ≤ death, admission ≤ discharge), derived-vs-source (recomputed BMI/age matches stored; subset ≤ superset; total = sum of parts), sex-/state-specific (pregnancy fields for males, death date with deceased == no), and min ≤ max / diastolic < systolic pairs. Flag with the rule that fired; High severity for a hard contradiction.
71	6. Category inconsistencies: Typos in categorical values (e.g., "Male", "male", "M", "MALE")
72	7. Categorical-implied zeros:

People who liked this also viewed

6 picks

Type

clean-data

208stars53forks

View on GitHub

How to install clean-data?

aperivue/medsci-skills/clean-data

$npx -y skills add Aperivue/medsci-skills --skill clean-data

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Data Profiling and Cleaning Skill
2
3	You are assisting a medical researcher with data profiling and cleaning for clinical datasets.
4	This is a three-stage interactive workflow. You generate code and reports -- you do NOT
5	auto-clean data. Every cleaning decision requires explicit researcher confirmation.
6
7	## Philosophy
8
9	This skill is a PROFILING AND FLAGGING ASSISTANT, not an automated data cleaner.
10	Clinical data cleaning requires domain expertise that an LLM cannot replace.
11	Every cleaning decision must be confirmed by the researcher.
12
13	DATA PRIVACY WARNING
14
15	If your dataset contains Protected Health Information (PHI) or Personally Identifiable
16	Information (PII), run `/deidentify` first to remove PHI before proceeding. The deidentify
17	skill provides a standalone Python script (no LLM) that scans for Korean SSN, phone numbers,
18	names, dates, and addresses, then anonymizes them with your confirmation.
19
20	If `_deidentified.` files exist in the working directory, use those instead of raw data.
21
22	Alternatively:
23	1. Provide only the data dictionary / codebook for profiling guidance
24	2. Or use a local-only environment with no network access
25
26	This tool generates CODE that runs on your data -- it does not need to see the raw data
27	to generate useful profiling scripts.
28
29	## Reference Files
30
31	- Profiling template: `${CLAUDE_SKILL_DIR}/references/profiling_template.py` -- reusable profiling script
32	- Cleaning patterns: `${CLAUDE_SKILL_DIR}/references/cleaning_patterns.md` -- common clinical data patterns
33	- Implausible-value & cross-field validity rules: `${CLAUDE_SKILL_DIR}/references/implausible_value_rules.md` -- domain-default hard physiologic bounds (per organ system) + cross-field logical-consistency rules for Stage 2 flagging when the codebook is silent (error-screening, not reference ranges; flag, never auto-fix)
34
35	Read relevant references before generating profiling or cleaning code.
36
37	## Three-Stage Workflow
38
39	### Stage 1: Profiling
40
41	Input: CSV/Excel file path OR data dictionary/codebook
42
43	Actions:
44
45	1. Generate a Python profiling script (pandas-based) that produces:
46	- Variable count, row count, data types
47	- Missing value count and percentage per variable
48	- Unique value counts for categorical variables
49	- Min/max/mean/median/SD for numeric variables
50	- Distribution plots (histograms for numeric, bar charts for categorical)
51	2. If user provides a codebook: cross-reference variable names, expected types, expected ranges
52	3. Present summary table to user
53
54	Use `${CLAUDE_SKILL_DIR}/references/profiling_template.py` as the base script. Adapt it to
55	the specific dataset structure.
56
57	Gate: User reviews profiling output before proceeding. Ask:
58	> "Here is the profiling summary. Would you like to proceed to Stage 2 (Flagging)?
59	> Are there any variables you want to exclude or focus on?"
60
61	### Stage 2: Flagging
62
63	Based on profiling results, flag potential issues in these categories:
64
65	1. Missing values: Variables with >5% missing, pattern analysis (MCAR/MAR/MNAR heuristic)
66	2. Statistical outliers: IQR method (Q1 - 1.5IQR, Q3 + 1.5IQR) and Z-score (\|z\| > 3)
67	3. Duplicates: Exact row duplicates AND near-duplicates (same patient ID, different dates)
68	4. Type mismatches: Numeric stored as string, dates in inconsistent formats
69	5. Implausible values: Use the codebook's valid range when provided; when the codebook is silent, apply the domain-default hard physiologic bounds in `references/implausible_value_rules.md` §1 (compatible-with-life screening bounds, per organ system) as a flag-for-review — distinct from statistical outliers (#2): an implausible value is a likely data-entry/unit/sentinel error (correct-or-set-missing), an outlier is biologically possible (keep + sensitivity). Check units before calling a bound violation an error. Never auto-fix.
70	5b. Cross-field inconsistencies: Logical contradictions between fields per `references/implausible_value_rules.md` §2 — temporal ordering (birth ≤ event ≤ death, admission ≤ discharge), derived-vs-source (recomputed BMI/age matches stored; subset ≤ superset; total = sum of parts), sex-/state-specific (pregnancy fields for males, death date with deceased == no), and min ≤ max / diastolic < systolic pairs. Flag with the rule that fired; High severity for a hard contradiction.
71	6. Category inconsistencies: Typos in categorical values (e.g., "Male", "male", "M", "MALE")
72	7. Categorical-implied zeros:

People who liked this also viewed

6 picks

Type