explainability

Name: explainability
Author: Aperivue

208stars53forks

How to install explainability?

aperivue/medsci-skills/explainability

$npx -y skills add Aperivue/medsci-skills --skill explainability

Installs into the current project.

Summary

Produce or audit the interpretability/explainability analysis of a medical-imaging model — Grad-CAM / Grad-CAM++ / attention-rollout / saliency / integrated-gradients — so it clears the rigor bar a reviewer expects: mandatory Adebayo sanity checks (model- and data-randomisation),

Files · 1

View on GitHub

SKILL.md

1	# Explainability Skill
2
3	## Purpose
4
5	A saliency / Grad-CAM heat-map is the most over-interpreted artifact in medical-imaging AI: a
6	colourful map over the lesion is routinely presented as proof the model "looks at the right thing."
7	Adebayo et al. (NeurIPS 2018) showed many saliency methods produce visually convincing maps that are
8	independent of the model's learned weights and of the labels — so they explain nothing. This skill
9	produces an explainability analysis that clears the rigor bar, and audits an existing one, so the map
10	is trustworthy before it reaches a manuscript (CLAIM 2024 / TRIPOD+AI interpretability items).
11
12	It sits alongside evaluation in the lane: `/architecture-zoo` → `/preprocess-imaging` →
13	`/model-scaffold` → `/model-validation` → `/model-evaluation` + explainability →
14	`/write-paper` + `/check-reporting`. It integrates captum / pytorch-grad-cam (referenced in the
15	plan); it does not reimplement them and never runs a model on real patient data.
16
17	## When to use
18	- You produced (or are about to produce) saliency / Grad-CAM / attention maps and want them reported
19	to the standard a reviewer expects.
20	- You want to audit an explainability analysis for the four failure modes below.
21
22	## When NOT to use
23	- Discrimination / calibration metrics → `/model-evaluation` then `/analyze-stats`.
24	- Split or preprocessing leakage → `/model-validation` / `/preprocess-imaging`.
25	- LLM/MLLM faithfulness & hallucination → `/mllm-eval`.
26	- Reimplementing captum / pytorch-grad-cam → out of scope (this skill wires and audits them).
27
28	## The four failure modes (what the gate enforces)
29	1. Saliency as validation. A map is attribution, not proof the model is correct or that the
30	relationship is causal. Frame it as "where signal is attributed", never as "the model is right".
31	2. No sanity check. Run the Adebayo model-randomisation and data-randomisation tests. A map
32	that survives neither is uninterpretable; both axes are the minimum bar.
33	3. No quantitative localisation. If you claim the map localises the finding, measure it — IoU /
34	pointing game / Dice against ground-truth masks — do not eyeball a few examples.
35	4. Cherry-picked examples. Report a cohort-level result, not a handful of hand-picked cases.
36
37	## Workflow
38
39	### Phase 1 — Produce the maps (integrate, don't reimplement)
40	Choose the method for the architecture (`references/explainability_guide.md`): Grad-CAM / Grad-CAM++
41	for CNNs, attention-rollout for ViTs, integrated-gradients / SHAP for attribution. Wire captum or
42	pytorch-grad-cam; do not write a new CAM implementation.
43
44	### Phase 2 — Sanity-check and quantify
45	- Run the model-parameter randomisation and data (label) randomisation tests (Adebayo 2018);
46	a faithful map degrades when the model/labels are randomised.
47	- Compute a quantitative localisation metric against ground-truth masks (IoU / pointing game /
48	Dice) over the cohort — not a visual impression.
49
50	### Phase 3 — Emit the explainability-report manifest
51	```json
52	{
53	"method": "grad-cam++",
54	"n_examples": 200,
55	"cohort_level": true,
56	"localization_metric": "iou",
57	"localization_value": 0.63,
58	"sanity_checks": ["model_randomization", "data_randomization"],
59	"interpretation": "localization"
60	}
61	```
62	`interpretation`: `attribution` / `localization` / `faithfulness` (descriptive) — never
63	`validation` / `causal` (overclaim).
64
65	### Phase 4 — Gate the report (deterministic)
66	```bash
67	python3 scripts/check_explainability_report.py --manifest explainability_report.json --strict
68	```
69	Verdicts: `SALIENCY_AS_VALIDATION`, `NO_SANITY_CHECK`, `NO_LOCALIZATION_METRIC` (Major);
70	`INSUFFICIENT_SANITY`, `CHERRY_PICKED_EXAMPLES`, `MISSING_METHOD` (Minor). The verdict is reproduced
71	by rule on the manifest, never asserted from prose.

People who liked this also viewed

6 picks

Type

explainability

208stars53forks

View on GitHub

How to install explainability?

aperivue/medsci-skills/explainability

$npx -y skills add Aperivue/medsci-skills --skill explainability

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Explainability Skill
2
3	## Purpose
4
5	A saliency / Grad-CAM heat-map is the most over-interpreted artifact in medical-imaging AI: a
6	colourful map over the lesion is routinely presented as proof the model "looks at the right thing."
7	Adebayo et al. (NeurIPS 2018) showed many saliency methods produce visually convincing maps that are
8	independent of the model's learned weights and of the labels — so they explain nothing. This skill
9	produces an explainability analysis that clears the rigor bar, and audits an existing one, so the map
10	is trustworthy before it reaches a manuscript (CLAIM 2024 / TRIPOD+AI interpretability items).
11
12	It sits alongside evaluation in the lane: `/architecture-zoo` → `/preprocess-imaging` →
13	`/model-scaffold` → `/model-validation` → `/model-evaluation` + explainability →
14	`/write-paper` + `/check-reporting`. It integrates captum / pytorch-grad-cam (referenced in the
15	plan); it does not reimplement them and never runs a model on real patient data.
16
17	## When to use
18	- You produced (or are about to produce) saliency / Grad-CAM / attention maps and want them reported
19	to the standard a reviewer expects.
20	- You want to audit an explainability analysis for the four failure modes below.
21
22	## When NOT to use
23	- Discrimination / calibration metrics → `/model-evaluation` then `/analyze-stats`.
24	- Split or preprocessing leakage → `/model-validation` / `/preprocess-imaging`.
25	- LLM/MLLM faithfulness & hallucination → `/mllm-eval`.
26	- Reimplementing captum / pytorch-grad-cam → out of scope (this skill wires and audits them).
27
28	## The four failure modes (what the gate enforces)
29	1. Saliency as validation. A map is attribution, not proof the model is correct or that the
30	relationship is causal. Frame it as "where signal is attributed", never as "the model is right".
31	2. No sanity check. Run the Adebayo model-randomisation and data-randomisation tests. A map
32	that survives neither is uninterpretable; both axes are the minimum bar.
33	3. No quantitative localisation. If you claim the map localises the finding, measure it — IoU /
34	pointing game / Dice against ground-truth masks — do not eyeball a few examples.
35	4. Cherry-picked examples. Report a cohort-level result, not a handful of hand-picked cases.
36
37	## Workflow
38
39	### Phase 1 — Produce the maps (integrate, don't reimplement)
40	Choose the method for the architecture (`references/explainability_guide.md`): Grad-CAM / Grad-CAM++
41	for CNNs, attention-rollout for ViTs, integrated-gradients / SHAP for attribution. Wire captum or
42	pytorch-grad-cam; do not write a new CAM implementation.
43
44	### Phase 2 — Sanity-check and quantify
45	- Run the model-parameter randomisation and data (label) randomisation tests (Adebayo 2018);
46	a faithful map degrades when the model/labels are randomised.
47	- Compute a quantitative localisation metric against ground-truth masks (IoU / pointing game /
48	Dice) over the cohort — not a visual impression.
49
50	### Phase 3 — Emit the explainability-report manifest
51	```json
52	{
53	"method": "grad-cam++",
54	"n_examples": 200,
55	"cohort_level": true,
56	"localization_metric": "iou",
57	"localization_value": 0.63,
58	"sanity_checks": ["model_randomization", "data_randomization"],
59	"interpretation": "localization"
60	}
61	```
62	`interpretation`: `attribution` / `localization` / `faithfulness` (descriptive) — never
63	`validation` / `causal` (overclaim).
64
65	### Phase 4 — Gate the report (deterministic)
66	```bash
67	python3 scripts/check_explainability_report.py --manifest explainability_report.json --strict
68	```
69	Verdicts: `SALIENCY_AS_VALIDATION`, `NO_SANITY_CHECK`, `NO_LOCALIZATION_METRIC` (Major);
70	`INSUFFICIENT_SANITY`, `CHERRY_PICKED_EXAMPLES`, `MISSING_METHOD` (Minor). The verdict is reproduced
71	by rule on the manifest, never asserted from prose.

People who liked this also viewed

6 picks

Type