human-eval-design

Name: human-eval-design
Author: Productfculty-aipm

70stars33forks

How to install human-eval-design?

productfculty-aipm/pm-copilot-by-product-faculty/human-eval-design

$npx -y skills add Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill human-eval-design

Installs into the current project.

Summary

Use this skill when the user asks to "design a human evaluation", "human eval process", "annotation guidelines", "how to set up human review of AI outputs", "how to get humans to evaluate AI quality", "build a labeling process", "create annotation criteria", or wants to set up a

Files · 1

View on GitHub

SKILL.md

1	# Human Eval Design
2
3	You are designing a human evaluation process for AI outputs — the gold standard for evaluating quality that LLM-as-judge systems are calibrated against. Human evals are slower and more expensive, but they're the source of ground truth for every other evaluation method.
4
5	Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), annotation methodology best practices.
6
7	## Step 1 — Load Context
8
9	Read `memory/user-profile.md` for the AI feature being evaluated. Understand: what are the failure categories from error analysis? What is the principal domain expert's quality bar?
10
11	## Step 2 — When to Run Human Evals
12
13	Run human evals:
14	- When bootstrapping: Before LLM-as-judge, to create the calibration dataset
15	- When investigating a regression: A metric dropped — sample and manually review to understand why
16	- Periodically (ongoing): Random sample of 50–100 live outputs per week/month to catch drift
17	- When launching a new model or prompt change: Before and after to validate quality didn't regress
18
19	Do NOT run human evals:
20	- On every inference in production (too slow and expensive)
21	- Without clear annotation guidelines (produces unreliable data)
22
23	## Step 3 — Annotation Design
24
25	Binary annotation (Recommended for most cases):
26	Each evaluator sees one (input, output) pair and marks: Thumbs Up (Good) / Thumbs Down (Bad).
27	Add: one mandatory reason for "Bad" selections (required for the error analysis feedback loop).
28
29	Pros: Fast (< 30 seconds per annotation), high agreement, easy to aggregate.
30	Cons: No nuance — doesn't tell you HOW bad something is.
31
32	Rubric annotation (Use when more granularity is needed):
33	Each evaluator rates 3–5 criteria on a 1–5 scale.
34
35	Example rubric for a PRD-writing feature:
36	\| Criterion \| 1 (Poor) \| 3 (Acceptable) \| 5 (Excellent) \|
37	\|---\|---\|---\|---\|
38	\| Problem clarity \| Vague or missing \| Present but imprecise \| Sharp, specific, evidence-based \|
39	\| User story quality \| Generic or feature-framed \| Adequate \| Demand-side, outcome-oriented \|
40	\| Acceptance criteria \| Missing or not testable \| Partially testable \| All binary and independently testable \|
41	\| Success metrics \| Output-oriented \| Mixed \| All outcome-oriented and measurable \|
42
43	## Step 4 — Annotation Guidelines
44
45	Write clear, specific annotation guidelines. The most common source of low agreement is ambiguous guidelines.
46
47	For each criterion, define:
48	- What PASS looks like (with 1–2 examples)
49	- What FAIL looks like (with 1–2 examples)
50	- The edge case: what if the output is partially correct?
51	- What to do if you're unsure (escalate to the principal domain expert)
52
53	Golden examples: Before the actual annotation, provide 5–10 examples with the "correct" answer labeled. This calibrates the annotator to the quality bar before they start.
54
55	## Step 5 — Agreement and Reliability
56
57	Inter-annotator agreement: If using multiple annotators, measure: % of cases where both annotators agree.
58	- > 85% agreement: The guidelines are clear and the task is well-defined
59	- 70–85% agreement: Guidelines need clarification on the gray areas
60	- < 70% agreement: The task definition is unclear; redesign guidelines
61
62	When to use one annotator vs. two:
63	- Binary annotation: One annotator is fine for most cases. Two annotators + tiebreaker for high-stakes evals.
64	- Rubric annotation: Two annotators + reconciliation for all cases.
65
66	## Step 6 — Annotation Tooling
67
68	For different scales:
69
70	Under 200 annotations/week: Google Sheets or Notion database. Simple, free, easy to set up.
71
72	200–2,000 annotations/week: Label Studio (free, open source) or Argilla. More structure, better workflows.
73
74	2,000+ annotations/week: Scale AI, Labelbox, or similar professional annotation platforms.
75
76	## Step 7 — Feedback Loop
77
78	The annotation process is only valuable if it feeds back into improvement:
79
80	1. Annotators flag Bad outputs
81	2. Bad outputs are clustered by failure category (error analysis)
82	3. Most common failures become new eval test cases
83	4. Eval suite is updated to catch those failures automatically
84	5. Model/prompt is updated to fix the failures
85	6. Annotation sample confirms the fix didn't create new failures
86
87	This is the continuous improvement flywheel.
88
89	## Step 8 — Output
90
91	Produce:
92	- Annotation interface design (fields, criteria, format)
93	- Annotation guidelines document (per-criterion definitions with examples)
94	- Golden examples set (5–10 labeled examples for calibration)
95	- Agreement measurement plan
96	- Sampling strategy: how many outputs to review per week and how to select them (random vs. targeted)

People who liked this also viewed

6 picks

Type

human-eval-design

70stars33forks

View on GitHub

How to install human-eval-design?

productfculty-aipm/pm-copilot-by-product-faculty/human-eval-design

$npx -y skills add Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill human-eval-design

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Human Eval Design
2
3	You are designing a human evaluation process for AI outputs — the gold standard for evaluating quality that LLM-as-judge systems are calibrated against. Human evals are slower and more expensive, but they're the source of ground truth for every other evaluation method.
4
5	Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), annotation methodology best practices.
6
7	## Step 1 — Load Context
8
9	Read `memory/user-profile.md` for the AI feature being evaluated. Understand: what are the failure categories from error analysis? What is the principal domain expert's quality bar?
10
11	## Step 2 — When to Run Human Evals
12
13	Run human evals:
14	- When bootstrapping: Before LLM-as-judge, to create the calibration dataset
15	- When investigating a regression: A metric dropped — sample and manually review to understand why
16	- Periodically (ongoing): Random sample of 50–100 live outputs per week/month to catch drift
17	- When launching a new model or prompt change: Before and after to validate quality didn't regress
18
19	Do NOT run human evals:
20	- On every inference in production (too slow and expensive)
21	- Without clear annotation guidelines (produces unreliable data)
22
23	## Step 3 — Annotation Design
24
25	Binary annotation (Recommended for most cases):
26	Each evaluator sees one (input, output) pair and marks: Thumbs Up (Good) / Thumbs Down (Bad).
27	Add: one mandatory reason for "Bad" selections (required for the error analysis feedback loop).
28
29	Pros: Fast (< 30 seconds per annotation), high agreement, easy to aggregate.
30	Cons: No nuance — doesn't tell you HOW bad something is.
31
32	Rubric annotation (Use when more granularity is needed):
33	Each evaluator rates 3–5 criteria on a 1–5 scale.
34
35	Example rubric for a PRD-writing feature:
36	\| Criterion \| 1 (Poor) \| 3 (Acceptable) \| 5 (Excellent) \|
37	\|---\|---\|---\|---\|
38	\| Problem clarity \| Vague or missing \| Present but imprecise \| Sharp, specific, evidence-based \|
39	\| User story quality \| Generic or feature-framed \| Adequate \| Demand-side, outcome-oriented \|
40	\| Acceptance criteria \| Missing or not testable \| Partially testable \| All binary and independently testable \|
41	\| Success metrics \| Output-oriented \| Mixed \| All outcome-oriented and measurable \|
42
43	## Step 4 — Annotation Guidelines
44
45	Write clear, specific annotation guidelines. The most common source of low agreement is ambiguous guidelines.
46
47	For each criterion, define:
48	- What PASS looks like (with 1–2 examples)
49	- What FAIL looks like (with 1–2 examples)
50	- The edge case: what if the output is partially correct?
51	- What to do if you're unsure (escalate to the principal domain expert)
52
53	Golden examples: Before the actual annotation, provide 5–10 examples with the "correct" answer labeled. This calibrates the annotator to the quality bar before they start.
54
55	## Step 5 — Agreement and Reliability
56
57	Inter-annotator agreement: If using multiple annotators, measure: % of cases where both annotators agree.
58	- > 85% agreement: The guidelines are clear and the task is well-defined
59	- 70–85% agreement: Guidelines need clarification on the gray areas
60	- < 70% agreement: The task definition is unclear; redesign guidelines
61
62	When to use one annotator vs. two:
63	- Binary annotation: One annotator is fine for most cases. Two annotators + tiebreaker for high-stakes evals.
64	- Rubric annotation: Two annotators + reconciliation for all cases.
65
66	## Step 6 — Annotation Tooling
67
68	For different scales:
69
70	Under 200 annotations/week: Google Sheets or Notion database. Simple, free, easy to set up.
71
72	200–2,000 annotations/week: Label Studio (free, open source) or Argilla. More structure, better workflows.
73
74	2,000+ annotations/week: Scale AI, Labelbox, or similar professional annotation platforms.
75
76	## Step 7 — Feedback Loop
77
78	The annotation process is only valuable if it feeds back into improvement:
79
80	1. Annotators flag Bad outputs
81	2. Bad outputs are clustered by failure category (error analysis)
82	3. Most common failures become new eval test cases
83	4. Eval suite is updated to catch those failures automatically
84	5. Model/prompt is updated to fix the failures
85	6. Annotation sample confirms the fix didn't create new failures
86
87	This is the continuous improvement flywheel.
88
89	## Step 8 — Output
90
91	Produce:
92	- Annotation interface design (fields, criteria, format)
93	- Annotation guidelines document (per-criterion definitions with examples)
94	- Golden examples set (5–10 labeled examples for calibration)
95	- Agreement measurement plan
96	- Sampling strategy: how many outputs to review per week and how to select them (random vs. targeted)

People who liked this also viewed

6 picks

Type