uncertainty-imaging

Name: uncertainty-imaging
Author: Aperivue

208stars53forks

How to install uncertainty-imaging?

aperivue/medsci-skills/uncertainty-imaging

$npx -y skills add Aperivue/medsci-skills --skill uncertainty-imaging

Installs into the current project.

Summary

Design or audit the uncertainty-quantification, out-of-distribution (OOD) detection, and selective-prediction layer of a medical-imaging model framed for deployment — so a clinical-use claim carries calibrated per-case uncertainty (MC-dropout / deep ensemble / conformal / Bayesia

Files · 1

View on GitHub

SKILL.md

1	# Uncertainty-Imaging Skill
2
3	## Purpose
4
5	A medical-imaging model framed for deployment must say more than "class 1, 0.87". It needs a
6	calibrated uncertainty on each case, an out-of-distribution (OOD) guard validated on data known
7	to be out-of-distribution, and — if it abstains — a pre-specified operating point. The failures are
8	predictable and reviewer-visible: a clinical-use claim built on point predictions, conformal intervals
9	quoted without ever measuring their coverage, an "OOD detector" evaluated only on in-distribution data,
10	a deep ensemble whose members share a seed, and uncertainty validated only in-distribution when
11	deployment sees scanner/site/case-mix shift. This skill designs that layer and audits an existing one
12	(Gal 2016; Lakshminarayanan 2017; Angelopoulos & Bates; Ovadia 2019; DECIDE-AI).
13
14	It is the deployment-safety companion in the model-engineering lane: `/model-evaluation` computes the
15	held-out metrics and calibration, and uncertainty-imaging covers the uncertainty / OOD / abstention
16	machinery a deployment claim rests on. It integrates MAPIE (conformal), captum, and pretrained OOD
17	scorers; it does not reimplement them and never runs a model on real patient data.
18
19	## When to use
20	- Your model is framed for clinical use / deployment and a reviewer will ask "what does it do when it is
21	unsure, or off-distribution?"
22	- You report conformal / MC-dropout / ensemble uncertainty and want the coverage, independence, and
23	shift checks right before submission.
24	- You want to audit an existing uncertainty/OOD section for the failure modes below.
25
26	## When NOT to use
27	- Held-out discrimination / calibration metrics of the point predictor → `/model-evaluation` then
28	`/analyze-stats`.
29	- Training-repo scaffolding / the split → `/model-scaffold` (+ `/model-validation`).
30	- Interpretability / saliency of a trained network → `/explainability`.
31	- Classical-ML calibration of a tabular model → `/radiomics-ml` + `/analyze-stats`.
32	- Reimplementing MAPIE / an OOD library → out of scope (this skill wires and audits them).
33
34	## The failure modes (what the gate enforces)
35	1. Point predictions under a deployment claim. A clinical-use claim with no uncertainty method at
36	all — add MC-dropout, a deep ensemble, conformal prediction, or a Bayesian estimate.
37	2. Conformal without coverage validation. Conformal's guarantee holds under exchangeability, which
38	can fail on clinical data — measure achieved coverage on a held-out calibration/test set.
39	3. OOD claim with no held-out OOD set. An OOD detector's operating point and AUROC are unmeasured
40	until you evaluate on data known to be out-of-distribution (different scanner / site / pathology).
41	4. Non-independent ensemble. A deep ensemble whose members share a seed/init (or has < 2 members)
42	underestimates epistemic uncertainty.
43	5. MC-dropout with dropout off at inference. Dropout must stay active during sampling; off, every
44	pass is identical and the estimate collapses to a point prediction.
45	6. Selective prediction without a target. Abstention chosen post hoc inflates accuracy-at-coverage;
46	pre-specify the coverage / risk operating point.
47	7. No calibration under shift. Uncertainty evaluated in-distribution only; deployment uncertainty
48	degrades under shift, so report it on shifted / external data.
49
50	## Workflow
51
52	### Phase 1 — Choose the uncertainty method (integrate, don't reimplement)
53	- Conformal prediction (MAPIE) — distribution-free prediction sets/intervals at a nominal coverage;
54	the strongest default when a calibration set is available. Validate empirical coverage.
55	- Deep ensembles (Lakshminarayanan 2017) — train K independent m

People who liked this also viewed

6 picks

Type

uncertainty-imaging

208stars53forks

View on GitHub

How to install uncertainty-imaging?

aperivue/medsci-skills/uncertainty-imaging

$npx -y skills add Aperivue/medsci-skills --skill uncertainty-imaging

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Uncertainty-Imaging Skill
2
3	## Purpose
4
5	A medical-imaging model framed for deployment must say more than "class 1, 0.87". It needs a
6	calibrated uncertainty on each case, an out-of-distribution (OOD) guard validated on data known
7	to be out-of-distribution, and — if it abstains — a pre-specified operating point. The failures are
8	predictable and reviewer-visible: a clinical-use claim built on point predictions, conformal intervals
9	quoted without ever measuring their coverage, an "OOD detector" evaluated only on in-distribution data,
10	a deep ensemble whose members share a seed, and uncertainty validated only in-distribution when
11	deployment sees scanner/site/case-mix shift. This skill designs that layer and audits an existing one
12	(Gal 2016; Lakshminarayanan 2017; Angelopoulos & Bates; Ovadia 2019; DECIDE-AI).
13
14	It is the deployment-safety companion in the model-engineering lane: `/model-evaluation` computes the
15	held-out metrics and calibration, and uncertainty-imaging covers the uncertainty / OOD / abstention
16	machinery a deployment claim rests on. It integrates MAPIE (conformal), captum, and pretrained OOD
17	scorers; it does not reimplement them and never runs a model on real patient data.
18
19	## When to use
20	- Your model is framed for clinical use / deployment and a reviewer will ask "what does it do when it is
21	unsure, or off-distribution?"
22	- You report conformal / MC-dropout / ensemble uncertainty and want the coverage, independence, and
23	shift checks right before submission.
24	- You want to audit an existing uncertainty/OOD section for the failure modes below.
25
26	## When NOT to use
27	- Held-out discrimination / calibration metrics of the point predictor → `/model-evaluation` then
28	`/analyze-stats`.
29	- Training-repo scaffolding / the split → `/model-scaffold` (+ `/model-validation`).
30	- Interpretability / saliency of a trained network → `/explainability`.
31	- Classical-ML calibration of a tabular model → `/radiomics-ml` + `/analyze-stats`.
32	- Reimplementing MAPIE / an OOD library → out of scope (this skill wires and audits them).
33
34	## The failure modes (what the gate enforces)
35	1. Point predictions under a deployment claim. A clinical-use claim with no uncertainty method at
36	all — add MC-dropout, a deep ensemble, conformal prediction, or a Bayesian estimate.
37	2. Conformal without coverage validation. Conformal's guarantee holds under exchangeability, which
38	can fail on clinical data — measure achieved coverage on a held-out calibration/test set.
39	3. OOD claim with no held-out OOD set. An OOD detector's operating point and AUROC are unmeasured
40	until you evaluate on data known to be out-of-distribution (different scanner / site / pathology).
41	4. Non-independent ensemble. A deep ensemble whose members share a seed/init (or has < 2 members)
42	underestimates epistemic uncertainty.
43	5. MC-dropout with dropout off at inference. Dropout must stay active during sampling; off, every
44	pass is identical and the estimate collapses to a point prediction.
45	6. Selective prediction without a target. Abstention chosen post hoc inflates accuracy-at-coverage;
46	pre-specify the coverage / risk operating point.
47	7. No calibration under shift. Uncertainty evaluated in-distribution only; deployment uncertainty
48	degrades under shift, so report it on shifted / external data.
49
50	## Workflow
51
52	### Phase 1 — Choose the uncertainty method (integrate, don't reimplement)
53	- Conformal prediction (MAPIE) — distribution-free prediction sets/intervals at a nominal coverage;
54	the strongest default when a calibration set is available. Validate empirical coverage.
55	- Deep ensembles (Lakshminarayanan 2017) — train K independent m

People who liked this also viewed

6 picks

Type