model-validation

Name: model-validation
Author: Aperivue

208stars53forks

How to install model-validation?

aperivue/medsci-skills/model-validation

$npx -y skills add Aperivue/medsci-skills --skill model-validation

Installs into the current project.

Summary

Design or audit the clinical-validation study for an engineer-built medical-imaging model (segmentation, classification, or detection) before the validation report or manuscript is written. Covers patient-level split disjointness and the data-leakage taxonomy, tuning-on-test, int

Files · 1

View on GitHub

SKILL.md

1	# Model-Validation Skill
2
3	## Purpose
4
5	This skill pressure-tests the validation study for an engineer-built medical-imaging model — the
6	common case where a clinical team receives a trained segmentation / classification / detection model
7	from an engineering collaborator and must validate it and write it up. It is the imaging-model
8	specialization of `/design-study`: where `design-study` covers general validity and
9	`design-ai-benchmarking` covers AI-versus-human-expert reader studies, this skill owns the
10	partition, leakage, reproducibility, and metric-selection mechanics that decide whether a reported
11	Dice / AUROC / sensitivity is trustworthy.
12
13	It is advisory and deterministic-audit only. It writes decision notes and runs a stdlib gate on the
14	split table; it never builds, trains, or alters the model, and it never replaces MONAI / nnU-Net /
15	TorchIO — those produce the model, this validates and publishes it.
16
17	## When to use
18	- A trained imaging model (in-house, vendor, or open-weights) needs a clinical-validation study designed
19	or audited before submission.
20	- You have, or can produce, the split-assignment table (which patient went to train / val / test).
21
22	## When NOT to use
23	- Building or training the model → out of scope (integrate MONAI / nnU-Net).
24	- AI-versus-human-expert reader study → `/design-ai-benchmarking`.
25	- LLM / MLLM evaluation → `/mllm-eval` (when available).
26	- General study/validity review → `/design-study`.
27	- Statistical execution (DeLong, ICC, bootstrap CIs, calibration tables) → `/analyze-stats`.
28	- Item-by-item reporting-guideline audit of a finished manuscript → `/check-reporting`.
29	- Reviewing a finished manuscript → `/self-review` or `/peer-review` (which load the MD0–MD8
30	reviewer-side probe).
31
32	## Workflow
33
34	The design/audit rationale behind Phases 2–7 — the full data-leakage taxonomy, the
35	internal-vs-genuine-external validation ladder, comparator design, single-run vs multi-seed
36	variance, test-set sizing, and the CLAIM 2024 / TRIPOD+AI / STARD-AI reporting map — is in
37	`${CLAUDE_SKILL_DIR}/references/validation_design.md` (load on demand). The patient-disjointness
38	verdict itself is proven by `scripts/check_split_leakage.py` (Phase 2), not from that prose.
39
40	### Phase 1 — Reconstruct the task, the intended-use horizon, and the analysis unit
41	State the model's task (segmentation / classification / detection), its intended-use horizon
42	(screening, triage, pre-procedure, post-hoc), the single headline metric the conclusion leans on,
43	and the analysis unit the metric must respect (per-patient vs per-lesion vs per-image). Everything
44	downstream is read against this.
45
46	### Phase 2 — Leakage audit (the deterministic gate, run first)
47	The most metric-inflating defect is a split that is not disjoint at the patient level. Produce the
48	emitted split-assignment table (`patient_id,split`) and run the gate:
49
50	```bash
51	python3 ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage.py \
52	--splits <split_assignment.csv> --out qc/split_leakage.json --strict
53	```
54
55	`PATIENT_OVERLAP` (a patient in ≥ 2 partitions) and `MISSING_SEED` (an unreproducible split) are
56	proven by set arithmetic — not heuristics. Then walk the rest of the leakage taxonomy (Kapoor &
57	Narayanan, Patterns 2023) that the table cannot show: preprocessing-before-split (normalisation,
58	resampling, foundation-model embeddings, or ComBat harmonisation fit on the whole cohort before
59	partitioning), site / scanner / burned-in-label shortcuts, and temporal leakage (a random split
60	where future and past coexist). The decisive question: *could any value used in training have been
61	computed only with knowledge of a test case?*
62
63	### Phase 3 — Validation tier (internal split vs genuine external)
64	Classify the evidence honest

People who liked this also viewed

6 picks

Type

model-validation

208stars53forks

View on GitHub

How to install model-validation?

aperivue/medsci-skills/model-validation

$npx -y skills add Aperivue/medsci-skills --skill model-validation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Model-Validation Skill
2
3	## Purpose
4
5	This skill pressure-tests the validation study for an engineer-built medical-imaging model — the
6	common case where a clinical team receives a trained segmentation / classification / detection model
7	from an engineering collaborator and must validate it and write it up. It is the imaging-model
8	specialization of `/design-study`: where `design-study` covers general validity and
9	`design-ai-benchmarking` covers AI-versus-human-expert reader studies, this skill owns the
10	partition, leakage, reproducibility, and metric-selection mechanics that decide whether a reported
11	Dice / AUROC / sensitivity is trustworthy.
12
13	It is advisory and deterministic-audit only. It writes decision notes and runs a stdlib gate on the
14	split table; it never builds, trains, or alters the model, and it never replaces MONAI / nnU-Net /
15	TorchIO — those produce the model, this validates and publishes it.
16
17	## When to use
18	- A trained imaging model (in-house, vendor, or open-weights) needs a clinical-validation study designed
19	or audited before submission.
20	- You have, or can produce, the split-assignment table (which patient went to train / val / test).
21
22	## When NOT to use
23	- Building or training the model → out of scope (integrate MONAI / nnU-Net).
24	- AI-versus-human-expert reader study → `/design-ai-benchmarking`.
25	- LLM / MLLM evaluation → `/mllm-eval` (when available).
26	- General study/validity review → `/design-study`.
27	- Statistical execution (DeLong, ICC, bootstrap CIs, calibration tables) → `/analyze-stats`.
28	- Item-by-item reporting-guideline audit of a finished manuscript → `/check-reporting`.
29	- Reviewing a finished manuscript → `/self-review` or `/peer-review` (which load the MD0–MD8
30	reviewer-side probe).
31
32	## Workflow
33
34	The design/audit rationale behind Phases 2–7 — the full data-leakage taxonomy, the
35	internal-vs-genuine-external validation ladder, comparator design, single-run vs multi-seed
36	variance, test-set sizing, and the CLAIM 2024 / TRIPOD+AI / STARD-AI reporting map — is in
37	`${CLAUDE_SKILL_DIR}/references/validation_design.md` (load on demand). The patient-disjointness
38	verdict itself is proven by `scripts/check_split_leakage.py` (Phase 2), not from that prose.
39
40	### Phase 1 — Reconstruct the task, the intended-use horizon, and the analysis unit
41	State the model's task (segmentation / classification / detection), its intended-use horizon
42	(screening, triage, pre-procedure, post-hoc), the single headline metric the conclusion leans on,
43	and the analysis unit the metric must respect (per-patient vs per-lesion vs per-image). Everything
44	downstream is read against this.
45
46	### Phase 2 — Leakage audit (the deterministic gate, run first)
47	The most metric-inflating defect is a split that is not disjoint at the patient level. Produce the
48	emitted split-assignment table (`patient_id,split`) and run the gate:
49
50	```bash
51	python3 ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage.py \
52	--splits <split_assignment.csv> --out qc/split_leakage.json --strict
53	```
54
55	`PATIENT_OVERLAP` (a patient in ≥ 2 partitions) and `MISSING_SEED` (an unreproducible split) are
56	proven by set arithmetic — not heuristics. Then walk the rest of the leakage taxonomy (Kapoor &
57	Narayanan, Patterns 2023) that the table cannot show: preprocessing-before-split (normalisation,
58	resampling, foundation-model embeddings, or ComBat harmonisation fit on the whole cohort before
59	partitioning), site / scanner / burned-in-label shortcuts, and temporal leakage (a random split
60	where future and past coexist). The decisive question: *could any value used in training have been
61	computed only with knowledge of a test case?*
62
63	### Phase 3 — Validation tier (internal split vs genuine external)
64	Classify the evidence honest

People who liked this also viewed

6 picks

Type