mllm-eval

Name: mllm-eval
Author: Aperivue

208stars53forks

How to install mllm-eval?

aperivue/medsci-skills/mllm-eval

$npx -y skills add Aperivue/medsci-skills --skill mllm-eval

Installs into the current project.

Summary

Design or audit a model-agnostic evaluation harness for an LLM or multimodal LLM on a clinical task (radiology report generation, visual question answering, clinical text extraction/classification) — the adjudicated reference standard, clinical-efficacy metrics (RadGraph-F1 / Che

Files · 1

View on GitHub

SKILL.md

1	# MLLM-Eval Skill
2
3	## Purpose
4
5	This skill makes an LLM / MLLM clinical evaluation defensible: a real adjudicated reference standard,
6	faithfulness measured not assumed, clinical-efficacy metrics beyond n-gram overlap, a pretraining-
7	contamination check, prompt-sensitivity disclosed, and a reader study where text is generated. It is
8	model-agnostic — every check applies to a closed API and to open weights — and read-only (an
9	advisory design/audit skill): it audits the evaluation design and specifies and routes the
10	clinical-efficacy metrics (RadGraph-F1 / CheXbert-F1 via their published extractors) rather than running
11	the model or computing the metrics itself.
12
13	It is the LLM/MLLM evaluation-design counterpart in the lane — an auditor that hands the specified
14	metrics to their extractors and `/analyze-stats`, parallel to how `/model-validation` audits an imaging
15	model's design (the imaging metrics themselves are computed by `/model-evaluation`). The reviewer-side
16	audit of a finished manuscript uses the `mllm_evaluation.md` (ME0–ME8) probe via `/self-review` and
17	`/peer-review`; this skill is the author-side harness design. It routes the reader study to
18	`/design-ai-benchmarking`, the sizing to `/calc-sample-size`, and TRIPOD-LLM / MI-CLEAR-LLM compliance to
19	`/check-reporting`.
20
21	## When to use
22	- You are designing or auditing an evaluation of an LLM/MLLM on a clinical task and want it to cover the
23	axes a reviewer will check (reference standard, faithfulness, contamination, prompt sensitivity, reader
24	study).
25
26	## When NOT to use
27	- AI-vs-human-expert benchmark with a rated rubric → `/design-ai-benchmarking`.
28	- Imaging prediction/segmentation model → `/model-evaluation` + `/model-validation`.
29	- Image-to-image generative model → the `image_synthesis` probe.
30	- Training / serving the LLM → out of scope.
31	- Item-level TRIPOD-LLM / MI-CLEAR-LLM audit of a finished manuscript → `/check-reporting`.
32
33	## Workflow
34
35	### Phase 1 — Pin the task, model, comparator, decoding (ME0)
36	State the task (report generation / VQA / extraction-classification), the exact model + version/date
37	(closed API or open-weights id), the decoding settings (temperature, seed, max tokens), and what the
38	outputs are scored against.
39
40	### Phase 2 — Reference standard + metrics (ME1, ME2)
41	Require an adjudicated expert reference (not a single unverified report or a model-derived label).
42	For report generation, report a clinical-efficacy metric — RadGraph-F1 (Jain et al., NeurIPS
43	2021) or CheXbert-F1 (Smit et al., 2020), or the composite RadCliQ (Yu et al., Patterns 2023)
44	— alongside any BLEU/ROUGE, with CIs. For VQA/classification, report accuracy at the
45	real prevalence with a stated answer-matching rule.
46
47	### Phase 3 — Faithfulness + contamination (ME3, ME4)
48	Add an atomic-fact faithfulness measure + a false-premise / abstention probe (MedVH, Med-HALT) —
49	report a hallucination rate, not just accuracy. For any public benchmark (VQA-RAD, SLAKE, MIMIC-CXR-
50	derived, MedQA), add a contamination statement: training cutoff vs benchmark release, a held-out /
51	post-cutoff set, or a contamination probe.
52
53	### Phase 4 — Prompt sensitivity + reader study (ME5, ME7)
54	Disclose the exact prompt(s), temperature/seed, ≥ 3 runs with variance, and a prompt-robustness
55	check. For a deployment/utility claim, design a blinded reader study with an error taxonomy (route
56	the rubric/IRR to `/design-ai-benchmarking`, ICC/κ to `/analyze-stats`, sizing to `/calc-sample-size`).
57
58	### Phase 5 — Gate the plan (deterministic)
59	```bash
60	python3 ${CLAUDE_SKILL_DIR}/scripts/check_mllm_eval_completeness.py \
61	--plan plan.md --task report_generation\|vqa\|classification --strict
62	```
63	`NGRAM_ONLY` / `FAITHFULNESS_MISSING` / `REFERENCE_STANDARD_MISSING` / `CONTAMINATION_UNADDRESSED` /
64	`READER_STUDY_MISSING` must be resolved.
65
66	### Phase 6 — Hand off
67	Methods/Results → `/write-paper`; compliance (TRIPOD-LLM / MI-CLEAR-LLM) → `/check

People who liked this also viewed

6 picks

Type

mllm-eval

208stars53forks

View on GitHub

How to install mllm-eval?

aperivue/medsci-skills/mllm-eval

$npx -y skills add Aperivue/medsci-skills --skill mllm-eval

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# MLLM-Eval Skill
2
3	## Purpose
4
5	This skill makes an LLM / MLLM clinical evaluation defensible: a real adjudicated reference standard,
6	faithfulness measured not assumed, clinical-efficacy metrics beyond n-gram overlap, a pretraining-
7	contamination check, prompt-sensitivity disclosed, and a reader study where text is generated. It is
8	model-agnostic — every check applies to a closed API and to open weights — and read-only (an
9	advisory design/audit skill): it audits the evaluation design and specifies and routes the
10	clinical-efficacy metrics (RadGraph-F1 / CheXbert-F1 via their published extractors) rather than running
11	the model or computing the metrics itself.
12
13	It is the LLM/MLLM evaluation-design counterpart in the lane — an auditor that hands the specified
14	metrics to their extractors and `/analyze-stats`, parallel to how `/model-validation` audits an imaging
15	model's design (the imaging metrics themselves are computed by `/model-evaluation`). The reviewer-side
16	audit of a finished manuscript uses the `mllm_evaluation.md` (ME0–ME8) probe via `/self-review` and
17	`/peer-review`; this skill is the author-side harness design. It routes the reader study to
18	`/design-ai-benchmarking`, the sizing to `/calc-sample-size`, and TRIPOD-LLM / MI-CLEAR-LLM compliance to
19	`/check-reporting`.
20
21	## When to use
22	- You are designing or auditing an evaluation of an LLM/MLLM on a clinical task and want it to cover the
23	axes a reviewer will check (reference standard, faithfulness, contamination, prompt sensitivity, reader
24	study).
25
26	## When NOT to use
27	- AI-vs-human-expert benchmark with a rated rubric → `/design-ai-benchmarking`.
28	- Imaging prediction/segmentation model → `/model-evaluation` + `/model-validation`.
29	- Image-to-image generative model → the `image_synthesis` probe.
30	- Training / serving the LLM → out of scope.
31	- Item-level TRIPOD-LLM / MI-CLEAR-LLM audit of a finished manuscript → `/check-reporting`.
32
33	## Workflow
34
35	### Phase 1 — Pin the task, model, comparator, decoding (ME0)
36	State the task (report generation / VQA / extraction-classification), the exact model + version/date
37	(closed API or open-weights id), the decoding settings (temperature, seed, max tokens), and what the
38	outputs are scored against.
39
40	### Phase 2 — Reference standard + metrics (ME1, ME2)
41	Require an adjudicated expert reference (not a single unverified report or a model-derived label).
42	For report generation, report a clinical-efficacy metric — RadGraph-F1 (Jain et al., NeurIPS
43	2021) or CheXbert-F1 (Smit et al., 2020), or the composite RadCliQ (Yu et al., Patterns 2023)
44	— alongside any BLEU/ROUGE, with CIs. For VQA/classification, report accuracy at the
45	real prevalence with a stated answer-matching rule.
46
47	### Phase 3 — Faithfulness + contamination (ME3, ME4)
48	Add an atomic-fact faithfulness measure + a false-premise / abstention probe (MedVH, Med-HALT) —
49	report a hallucination rate, not just accuracy. For any public benchmark (VQA-RAD, SLAKE, MIMIC-CXR-
50	derived, MedQA), add a contamination statement: training cutoff vs benchmark release, a held-out /
51	post-cutoff set, or a contamination probe.
52
53	### Phase 4 — Prompt sensitivity + reader study (ME5, ME7)
54	Disclose the exact prompt(s), temperature/seed, ≥ 3 runs with variance, and a prompt-robustness
55	check. For a deployment/utility claim, design a blinded reader study with an error taxonomy (route
56	the rubric/IRR to `/design-ai-benchmarking`, ICC/κ to `/analyze-stats`, sizing to `/calc-sample-size`).
57
58	### Phase 5 — Gate the plan (deterministic)
59	```bash
60	python3 ${CLAUDE_SKILL_DIR}/scripts/check_mllm_eval_completeness.py \
61	--plan plan.md --task report_generation\|vqa\|classification --strict
62	```
63	`NGRAM_ONLY` / `FAITHFULNESS_MISSING` / `REFERENCE_STANDARD_MISSING` / `CONTAMINATION_UNADDRESSED` /
64	`READER_STUDY_MISSING` must be resolved.
65
66	### Phase 6 — Hand off
67	Methods/Results → `/write-paper`; compliance (TRIPOD-LLM / MI-CLEAR-LLM) → `/check

People who liked this also viewed

6 picks

Type