design-ai-benchmarking

Name: design-ai-benchmarking
Author: Aperivue

208stars53forks

How to install design-ai-benchmarking?

aperivue/medsci-skills/design-ai-benchmarking

$npx -y skills add Aperivue/medsci-skills --skill design-ai-benchmarking

Installs into the current project.

Summary

Design and validity review for studies that benchmark one or more AI systems against a human-expert panel as the reference. Covers the evaluation question and arm definition, decoupled multi-dimensional rubrics with anchors, planted calibration probes, reviewer-panel construction

Files · 1

View on GitHub

SKILL.md

1	# Design-AI-Benchmarking Skill
2
3	## Purpose
4
5	This skill pressure-tests an AI-vs-human-expert benchmark before any ratings are collected, so that
6	the comparison is fair, the rubric measures distinct constructs, the scale is calibrated, and the
7	reported reliability is interpretable. It is the AI-evaluation specialization of `/design-study`: where
8	`/design-study` reviews a study in general, this skill owns the specific machinery of comparing AI
9	system(s) to a panel of human experts (or to each other) on rated outputs.
10
11	Use it when:
12	- one or more AI systems will be scored against a human-expert reference (reader study, annotation
13	panel, AI-output evaluation, model-vs-model bench)
14	- a rubric and rating protocol must be locked before reviewers begin
15	- a benchmark feels vulnerable to "the highest score is just the most tautological item" or
16	"low agreement, but we cannot tell why" criticism
17	- a reviewer or editor asks how the evaluation controlled for rater drift, leakage, or judge bias
18
19	Do not use it for: general study/validity review (use `/design-study`); statistical execution such
20	as ICC or DeLong (use `/analyze-stats`); reporting-guideline item audits (use `/check-reporting`);
21	or reviewing an already-written manuscript (use `/peer-review` or `/self-review`).
22
23	---
24
25	## Communication Rules
26
27	- Communicate with the user in their preferred language.
28	- Use English for statistical, machine-learning, and reporting-guideline terminology.
29	- Be direct about evaluation-validity risks, but always propose the smallest feasible fix first.
30	- Never invent reviewer ratings, reference labels, or agreement statistics; those come from collected
31	data only.
32
33	---
34
35	## Standard Output
36
37	```text
38	## AI-Benchmark Design Review
39	Evaluation question: ...
40	Arms / systems compared: ...
41	Reference (human-expert panel): ...
42	Unit of rating: (item / case / output)
43
44	### Rubric (decoupled dimensions)
45	- dimension -> construct -> anchors (1..k)
46
47	### Calibration probes (blinded, randomized)
48	- positive-control / known-bad / instability / mechanism-contradiction
49
50	### Reviewer panel
51	- n reviewers, metadata captured, per-reviewer randomized order
52
53	### Reliability plan
54	- overall IRR target + control-item IRR (reported separately)
55
56	### Judge strategy
57	- human-as-judge / LLM-as-judge / both + adjudication rule
58
59	### Validity risks
60	1. ...
61
62	### Minimal fixes
63	- ...
64
65	### Decision
66	- Ready to collect / Needs rubric revision / Needs arm or judge redesign
67	```
68
69	---
70
71	## Workflow
72
73	### Phase 1: Define the evaluation question and arms
74
75	Pin down, in writing:
76	- the exact claim the benchmark must support (e.g., "system A's outputs are perceptually
77	indistinguishable from expert outputs", not "system A is deployment-ready")
78	- every arm/system being compared, and what each arm receives as input (same items, same information
79	access, same output format) so no arm has a hidden advantage
80	- the human-expert reference: who they are, and whether they set ground truth, provide a comparison
81	arm, or both
82	- the unit of rating (item, case, output) and how many units each reviewer sees
83
84	Gate: Present the reconstructed evaluation question, arms, and reference to the user and confirm
85	before designing the rubric. A wrong reconstruction misdirects the entire benchmark.
86
87	### Phase 2: Design a decoupled multi-dimensional rubric
88
89	- Decouple the axes. Each rated dimension measures one construct. Keep "is the output valid/correct"
90	separate from "is it novel", "is it feasible/measurable", "does it add value over current tools", and
91	"would it change action". A candidate can be high-validity yet low-added-value ("real but redundant");
92	a single blended score hides this divergence.
93	- Anchor every scale point with a short verbal descriptor; pilot the anchors with at least one
94	reviewer before locking.
95	- Pre-specify discriminant validity: hypothesize which dimensions should correlate vs be orthogonal,
96	then report the full inter-dimension correlation matrix to confirm the rubric measures distinct
97	constructs.
98	- A worked rubric template lives in `${CLAUDE_SKILL_DIR}/references/elicitation_rubric_template.md`.
99
100	### Phase 3: Insert and randomize calibr

People who liked this also viewed

6 picks

Type

design-ai-benchmarking

208stars53forks

View on GitHub

How to install design-ai-benchmarking?

aperivue/medsci-skills/design-ai-benchmarking

$npx -y skills add Aperivue/medsci-skills --skill design-ai-benchmarking

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Design-AI-Benchmarking Skill
2
3	## Purpose
4
5	This skill pressure-tests an AI-vs-human-expert benchmark before any ratings are collected, so that
6	the comparison is fair, the rubric measures distinct constructs, the scale is calibrated, and the
7	reported reliability is interpretable. It is the AI-evaluation specialization of `/design-study`: where
8	`/design-study` reviews a study in general, this skill owns the specific machinery of comparing AI
9	system(s) to a panel of human experts (or to each other) on rated outputs.
10
11	Use it when:
12	- one or more AI systems will be scored against a human-expert reference (reader study, annotation
13	panel, AI-output evaluation, model-vs-model bench)
14	- a rubric and rating protocol must be locked before reviewers begin
15	- a benchmark feels vulnerable to "the highest score is just the most tautological item" or
16	"low agreement, but we cannot tell why" criticism
17	- a reviewer or editor asks how the evaluation controlled for rater drift, leakage, or judge bias
18
19	Do not use it for: general study/validity review (use `/design-study`); statistical execution such
20	as ICC or DeLong (use `/analyze-stats`); reporting-guideline item audits (use `/check-reporting`);
21	or reviewing an already-written manuscript (use `/peer-review` or `/self-review`).
22
23	---
24
25	## Communication Rules
26
27	- Communicate with the user in their preferred language.
28	- Use English for statistical, machine-learning, and reporting-guideline terminology.
29	- Be direct about evaluation-validity risks, but always propose the smallest feasible fix first.
30	- Never invent reviewer ratings, reference labels, or agreement statistics; those come from collected
31	data only.
32
33	---
34
35	## Standard Output
36
37	```text
38	## AI-Benchmark Design Review
39	Evaluation question: ...
40	Arms / systems compared: ...
41	Reference (human-expert panel): ...
42	Unit of rating: (item / case / output)
43
44	### Rubric (decoupled dimensions)
45	- dimension -> construct -> anchors (1..k)
46
47	### Calibration probes (blinded, randomized)
48	- positive-control / known-bad / instability / mechanism-contradiction
49
50	### Reviewer panel
51	- n reviewers, metadata captured, per-reviewer randomized order
52
53	### Reliability plan
54	- overall IRR target + control-item IRR (reported separately)
55
56	### Judge strategy
57	- human-as-judge / LLM-as-judge / both + adjudication rule
58
59	### Validity risks
60	1. ...
61
62	### Minimal fixes
63	- ...
64
65	### Decision
66	- Ready to collect / Needs rubric revision / Needs arm or judge redesign
67	```
68
69	---
70
71	## Workflow
72
73	### Phase 1: Define the evaluation question and arms
74
75	Pin down, in writing:
76	- the exact claim the benchmark must support (e.g., "system A's outputs are perceptually
77	indistinguishable from expert outputs", not "system A is deployment-ready")
78	- every arm/system being compared, and what each arm receives as input (same items, same information
79	access, same output format) so no arm has a hidden advantage
80	- the human-expert reference: who they are, and whether they set ground truth, provide a comparison
81	arm, or both
82	- the unit of rating (item, case, output) and how many units each reviewer sees
83
84	Gate: Present the reconstructed evaluation question, arms, and reference to the user and confirm
85	before designing the rubric. A wrong reconstruction misdirects the entire benchmark.
86
87	### Phase 2: Design a decoupled multi-dimensional rubric
88
89	- Decouple the axes. Each rated dimension measures one construct. Keep "is the output valid/correct"
90	separate from "is it novel", "is it feasible/measurable", "does it add value over current tools", and
91	"would it change action". A candidate can be high-validity yet low-added-value ("real but redundant");
92	a single blended score hides this divergence.
93	- Anchor every scale point with a short verbal descriptor; pilot the anchors with at least one
94	reviewer before locking.
95	- Pre-specify discriminant validity: hypothesize which dimensions should correlate vs be orthogonal,
96	then report the full inter-dimension correlation matrix to confirm the rubric measures distinct
97	constructs.
98	- A worked rubric template lives in `${CLAUDE_SKILL_DIR}/references/elicitation_rubric_template.md`.
99
100	### Phase 3: Insert and randomize calibr

People who liked this also viewed

6 picks

Type