model-evaluation

Name: model-evaluation
Author: Aperivue

208stars53forks

How to install model-evaluation?

aperivue/medsci-skills/model-evaluation

$npx -y skills add Aperivue/medsci-skills --skill model-evaluation

Installs into the current project.

Summary

Compute and report task-correct held-out metrics for a trained medical-imaging model — segmentation (Dice plus a boundary metric such as HD95 or NSD, per structure), classification (AUROC plus AUPRC and sensitivity/specificity with bootstrap CIs at the deployment prevalence), det

Files · 1

View on GitHub

SKILL.md

1	# Model-Evaluation Skill
2
3	## Purpose
4
5	This skill makes a medical-imaging model's held-out evaluation task-correct and honest: the right
6	metric for the task and the prevalence, with uncertainty, calibration, and subgroup performance. It
7	emits a per-case metric table that the publication statistics build on, and gates the metric choice
8	against Metrics Reloaded (Maier-Hein & Reinke et al., Nat Methods 2024) and CLAIM 2024.
9
10	It sits between `/model-validation` (which audits the split / design) and `/analyze-stats` (which owns
11	the comparative inference). It computes the imaging-specific per-case metrics (surface distances, FROC,
12	ECE of a softmax head); `/analyze-stats` owns DeLong / NRI / IDI / decision curves / MRMC. Like
13	`/analyze-stats`, it generates and executes code on your predictions — numbers are never hand-typed.
14
15	## When to use
16	- You have held-out predictions + ground truth and need task-correct metrics with CIs, calibration, and
17	subgroup slices, plus a per-case table for the manuscript statistics.
18
19	## When NOT to use
20	- Auditing the validation design / leakage → `/model-validation`.
21	- DeLong / NRI / IDI / decision curves / MRMC reader study → `/analyze-stats`.
22	- Building / training the model → `/model-scaffold`; LLM / MLLM → `/mllm-eval`.
23	- Figure rendering → `/make-figures`.
24
25	## Workflow
26
27	### Phase 1 — Fix the analysis unit and the task
28	State the task (segmentation / classification / detection / interactive / generative) and the analysis unit the metric must
29	respect (per-patient vs per-lesion vs per-image). A per-lesion metric must not be reported as
30	per-patient.
31
32	### Phase 2 — Compute task-correct metrics
33	Generate evaluation code that computes, on the held-out predictions:
34	- segmentation: Dice/IoU and a boundary metric (HD95 / NSD), per structure not only a global
35	mean, with bootstrap 95% CIs.
36	- classification: AUROC and AUPRC with bootstrap CIs, sensitivity/specificity, and PPV/NPV **at
37	the deployment prevalence** (not a balanced set).
38	- detection: FROC / mAP with the IoU match criterion stated.
39	- interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive): the segmentation
40	metrics above plus the interaction axis — Dice-vs-interactions / number-of-clicks (NoC) to a
41	target threshold, initial-vs-converged (or peak) Dice, and per-case interaction/inference time
42	(see the metric guide; the study design is in `/design-study` + `/model-validation`).
43	- generative / synthesis (image generation or modification): full-reference similarity
44	(MSE/RMSE/PSNR/SSIM) or no-reference quality (SNR/CNR, standardized visual scores), **plus a
45	downstream-task evaluation** — image quality is not clinical utility (Park et al., Radiol Med 2024).
46	For multiclass classification, state the aggregation scheme (one-vs-rest / macro / micro /
47	pairwise / Obuchowski); time-to-event discrimination (Harrell's C, time-dependent ROC) is handed
48	to `/analyze-stats`.
49	Add calibration (reliability diagram / ECE) and subgroup slices (the Model Card Factors).
50	See `${CLAUDE_SKILL_DIR}/references/metric_guide.md`. Emit a per-case CSV for `/analyze-stats`.
51
52	### Phase 3 — Gate the metric choice (deterministic)
53	```bash
54	python3 ${CLAUDE_SKILL_DIR}/scripts/chec

People who liked this also viewed

6 picks

Type

model-evaluation

208stars53forks

View on GitHub

How to install model-evaluation?

aperivue/medsci-skills/model-evaluation

$npx -y skills add Aperivue/medsci-skills --skill model-evaluation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Model-Evaluation Skill
2
3	## Purpose
4
5	This skill makes a medical-imaging model's held-out evaluation task-correct and honest: the right
6	metric for the task and the prevalence, with uncertainty, calibration, and subgroup performance. It
7	emits a per-case metric table that the publication statistics build on, and gates the metric choice
8	against Metrics Reloaded (Maier-Hein & Reinke et al., Nat Methods 2024) and CLAIM 2024.
9
10	It sits between `/model-validation` (which audits the split / design) and `/analyze-stats` (which owns
11	the comparative inference). It computes the imaging-specific per-case metrics (surface distances, FROC,
12	ECE of a softmax head); `/analyze-stats` owns DeLong / NRI / IDI / decision curves / MRMC. Like
13	`/analyze-stats`, it generates and executes code on your predictions — numbers are never hand-typed.
14
15	## When to use
16	- You have held-out predictions + ground truth and need task-correct metrics with CIs, calibration, and
17	subgroup slices, plus a per-case table for the manuscript statistics.
18
19	## When NOT to use
20	- Auditing the validation design / leakage → `/model-validation`.
21	- DeLong / NRI / IDI / decision curves / MRMC reader study → `/analyze-stats`.
22	- Building / training the model → `/model-scaffold`; LLM / MLLM → `/mllm-eval`.
23	- Figure rendering → `/make-figures`.
24
25	## Workflow
26
27	### Phase 1 — Fix the analysis unit and the task
28	State the task (segmentation / classification / detection / interactive / generative) and the analysis unit the metric must
29	respect (per-patient vs per-lesion vs per-image). A per-lesion metric must not be reported as
30	per-patient.
31
32	### Phase 2 — Compute task-correct metrics
33	Generate evaluation code that computes, on the held-out predictions:
34	- segmentation: Dice/IoU and a boundary metric (HD95 / NSD), per structure not only a global
35	mean, with bootstrap 95% CIs.
36	- classification: AUROC and AUPRC with bootstrap CIs, sensitivity/specificity, and PPV/NPV **at
37	the deployment prevalence** (not a balanced set).
38	- detection: FROC / mAP with the IoU match criterion stated.
39	- interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive): the segmentation
40	metrics above plus the interaction axis — Dice-vs-interactions / number-of-clicks (NoC) to a
41	target threshold, initial-vs-converged (or peak) Dice, and per-case interaction/inference time
42	(see the metric guide; the study design is in `/design-study` + `/model-validation`).
43	- generative / synthesis (image generation or modification): full-reference similarity
44	(MSE/RMSE/PSNR/SSIM) or no-reference quality (SNR/CNR, standardized visual scores), **plus a
45	downstream-task evaluation** — image quality is not clinical utility (Park et al., Radiol Med 2024).
46	For multiclass classification, state the aggregation scheme (one-vs-rest / macro / micro /
47	pairwise / Obuchowski); time-to-event discrimination (Harrell's C, time-dependent ROC) is handed
48	to `/analyze-stats`.
49	Add calibration (reliability diagram / ECE) and subgroup slices (the Model Card Factors).
50	See `${CLAUDE_SKILL_DIR}/references/metric_guide.md`. Emit a per-case CSV for `/analyze-stats`.
51
52	### Phase 3 — Gate the metric choice (deterministic)
53	```bash
54	python3 ${CLAUDE_SKILL_DIR}/scripts/chec

People who liked this also viewed

6 picks

Type