Design and validity review for studies that benchmark one or more AI systems against a human-expert panel as the reference. Covers the evaluation question and arm definition, decoupled multi-dimensional rubrics with anchors, planted calibration probes, reviewer-panel construction, inter-rater reliability targets, LLM-as-judge versus human-as-judge adjudication, construct-independence guards, and a structured rating-export schema. Use before data collection on an AI-vs-expert evaluation.
How to install design-ai-benchmarking?
aperivue/medsci-skills/design-ai-benchmarking
$npx -y skills add aperivue/medsci-skills --skill design-ai-benchmarkingInstalls into the current project.
›Prefer a prompt? Paste this to your agent
Run `npx skills use "https://github.com/aperivue/medsci-skills" --skill "aperivue/medsci-skills/design-ai-benchmarking"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.
Use the skills in "https://github.com/aperivue/medsci-skills" that are relevant to the current task. Run `npx skills add "https://github.com/aperivue/medsci-skills"` and select the relevant skills, then follow their instructions.
Related
6 picksType
- researchInvestigate a question against high-trust primary sources and capture the findings as a Markdown file in the repo.Jul 2026
- analyze-projectRigor Analyze / Rigor Audit read-only skill for deep learning research repositories.Jul 2026
- paper-context-resolverRigor Paper Context helper for README-first deep learning repo reproduction.Jul 2026
- repo-intake-and-planRigor Intake helper for README-first deep learning repo reproduction.Jul 2026
- firecrawl-searchWeb search with full page content extraction. Use this skill whenever the user asks to search the web, find articles, research a topic, look something up, find…Jul 2026
- customer-researchWhen the user wants to conduct, analyze, or synthesize customer research.Jul 2026
