eval-suite-design

Name: eval-suite-design
Author: Productfculty-aipm

70stars33forks

How to install eval-suite-design?

productfculty-aipm/pm-copilot-by-product-faculty/eval-suite-design

$npx -y skills add Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill eval-suite-design

Installs into the current project.

Summary

Use this skill when the user asks to "design an eval suite", "build evals for my AI feature", "create an evaluation framework", "how do I evaluate my AI", "what evals should I run", "build an eval system", or wants to create a systematic evaluation framework for an AI-powered pro

Files · 1

View on GitHub

SKILL.md

1	# Eval Suite Design
2
3	You are designing an evaluation suite for an AI product feature — a systematic set of tests that catches real failure modes before they reach users. The goal is a suite that the team actually runs and acts on, not one that gets ignored.
4
5	Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), Aman Khan (Beyond vibe checks, 2025).
6
7	Key principle: "Evals quietly decide whether your AI product thrives or dies. The ability to write great evals is rapidly becoming the defining skill for AI PMs in 2025 and beyond." — Aman Khan, Lenny's Newsletter (2025)
8
9	## Step 1 — Load Context
10
11	Read the error analysis output (from the error-analysis skill or user input) to understand which failure categories to target. Read `memory/user-profile.md` for the AI feature context.
12
13	## Step 2 — Three Types of Evals
14
15	For each failure category, select the appropriate eval type:
16
17	Type 1 — Code-based evals (deterministic):
18	Best for: Failures with objectively correct / incorrect answers. Format compliance. Structural checks.
19	Examples:
20	- Output contains required sections (assert "## Problem" in output)
21	- Output is within length bounds (assert len(output) < 2000)
22	- Output is valid JSON (try parse; fail if exception)
23	- Required fields are non-empty (assert output.get('metric') is not None)
24	Pros: Fast, cheap, perfectly reliable
25	Cons: Only works for objective correctness — can't evaluate quality
26
27	Type 2 — Human evals:
28	Best for: Subjective quality, domain-specific correctness, complex reasoning, new failure categories being discovered.
29	Format: Annotators see (input, output) pairs and rate: Thumbs up / Thumbs down, or score on a rubric (1–5).
30	Pros: Highest accuracy; catches nuanced failures
31	Cons: Slow, expensive, can't scale; requires clear annotation guidelines
32	Use for: Calibration, sampling for quality assurance, training LLM-as-judge
33
34	Type 3 — LLM-as-judge:
35	Best for: Subjective quality at scale; failures that require reasoning to detect; when human evals are too slow.
36	Structure: A separate LLM (usually a stronger model) reviews (input, output) pairs and provides a judgment.
37	Pros: Scalable; can evaluate complex quality; can explain its reasoning
38	Cons: Not perfectly reliable; needs calibration against human evals; can be biased
39
40	## Step 3 — Eval Design Per Failure Category
41
42	For each top failure category (from error analysis):
43
44	Name: [Failure category name]
45	Eval type: [Code-based / Human / LLM-as-judge]
46	What to test: [Specific aspect of the output being evaluated]
47	Test cases needed: [How many? Where do they come from?]
48	Pass/fail criteria: [What counts as pass? What counts as fail?]
49	Automation plan: [When does this eval run — on every PR? Daily? Weekly?]
50
51	## Step 4 — LLM-as-Judge Prompt Design
52
53	If using LLM-as-judge, write the judge prompt following best practices:
54
55	```
56	You are evaluating an AI assistant's response for [failure type].
57
58	Input to the AI assistant:
59	{input}
60
61	AI assistant's response:
62	{response}
63
64	Evaluation criteria:
65	[Criterion 1]: [Clear definition of what good looks like]
66	[Criterion 2]: [Clear definition of what good looks like]
67
68	Scoring:
69	- PASS: The response [specific pass condition]
70	- FAIL: The response [specific fail condition]
71
72	Your output:
73	First, briefly explain your reasoning (1–2 sentences).
74	Then output: PASS or FAIL
75	```
76
77	Key principles for judge prompts:
78	- Binary outputs (PASS/FAIL) are more reliable than numeric scores
79	- Include examples of PASS and FAIL in the prompt when possible
80	- The judge should explain its reasoning before giving the verdict (chain-of-thought)
81	- Calibrate the judge against 50+ human annotations before trusting it
82
83	## Step 5 — The Principal Domain Expert Model
84
85	From Hamel Husain: designate one "benevolent dictator" for quality — one person whose judgment defines what PASS/FAIL means for subjective evals. This prevents annotation conflicts and anchors the LLM-as-judge calibration.
86
87	This person:
88	- Reviews 50–100 human annotation cases to establish the quality bar
89	- Resolves disagreements between annotators
90	- Periodically reviews LLM-as-judge outputs to catch drift
91
92	## Step 6 — Eval Suite Structure
93
94	Design the full suite as three layers:
95
96	Layer 1 — Pre-commit (fast): Code-based evals only. Run on every code change. Must complete in < 60 seconds. Catches format and structural failures.
97
98	Layer 2 — Pre-deploy (medium): Code-based + LLM-as-judge on a representative sample. Run before any deployment. Should complete in < 10 minutes.
99
100	**Layer 3 — Production monitoring (ongoing):

Productfculty-aipmPublisher · 69 listings

Preview

Productfculty-aipm/PM-Copilot-by-Product-Faculty

# Eval Suite Design

You are designing an evaluation suite for an AI product feature — a systematic set of tests that catches real failure modes before they reach users. The goal is

Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), Aman Khan (Beyond vibe checks, 2025).

Key principle: "Evals quietly decide whether your AI product thrives or dies. The ability to write great evals is rapidly becoming the defining skill for AI PMs

RepoProductfculty-aipm/PM-Copilot-by-Product-Faculty

License—

TypeSkills

CategoryTesting

UpdatedApr 2026

Repo stars70

First seenJul 22, 2026

People who liked this also viewed

6 picks

Type

eval-suite-design

70stars33forks

View on GitHub

How to install eval-suite-design?

productfculty-aipm/pm-copilot-by-product-faculty/eval-suite-design

$npx -y skills add Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill eval-suite-design

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Eval Suite Design
2
3	You are designing an evaluation suite for an AI product feature — a systematic set of tests that catches real failure modes before they reach users. The goal is a suite that the team actually runs and acts on, not one that gets ignored.
4
5	Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), Aman Khan (Beyond vibe checks, 2025).
6
7	Key principle: "Evals quietly decide whether your AI product thrives or dies. The ability to write great evals is rapidly becoming the defining skill for AI PMs in 2025 and beyond." — Aman Khan, Lenny's Newsletter (2025)
8
9	## Step 1 — Load Context
10
11	Read the error analysis output (from the error-analysis skill or user input) to understand which failure categories to target. Read `memory/user-profile.md` for the AI feature context.
12
13	## Step 2 — Three Types of Evals
14
15	For each failure category, select the appropriate eval type:
16
17	Type 1 — Code-based evals (deterministic):
18	Best for: Failures with objectively correct / incorrect answers. Format compliance. Structural checks.
19	Examples:
20	- Output contains required sections (assert "## Problem" in output)
21	- Output is within length bounds (assert len(output) < 2000)
22	- Output is valid JSON (try parse; fail if exception)
23	- Required fields are non-empty (assert output.get('metric') is not None)
24	Pros: Fast, cheap, perfectly reliable
25	Cons: Only works for objective correctness — can't evaluate quality
26
27	Type 2 — Human evals:
28	Best for: Subjective quality, domain-specific correctness, complex reasoning, new failure categories being discovered.
29	Format: Annotators see (input, output) pairs and rate: Thumbs up / Thumbs down, or score on a rubric (1–5).
30	Pros: Highest accuracy; catches nuanced failures
31	Cons: Slow, expensive, can't scale; requires clear annotation guidelines
32	Use for: Calibration, sampling for quality assurance, training LLM-as-judge
33
34	Type 3 — LLM-as-judge:
35	Best for: Subjective quality at scale; failures that require reasoning to detect; when human evals are too slow.
36	Structure: A separate LLM (usually a stronger model) reviews (input, output) pairs and provides a judgment.
37	Pros: Scalable; can evaluate complex quality; can explain its reasoning
38	Cons: Not perfectly reliable; needs calibration against human evals; can be biased
39
40	## Step 3 — Eval Design Per Failure Category
41
42	For each top failure category (from error analysis):
43
44	Name: [Failure category name]
45	Eval type: [Code-based / Human / LLM-as-judge]
46	What to test: [Specific aspect of the output being evaluated]
47	Test cases needed: [How many? Where do they come from?]
48	Pass/fail criteria: [What counts as pass? What counts as fail?]
49	Automation plan: [When does this eval run — on every PR? Daily? Weekly?]
50
51	## Step 4 — LLM-as-Judge Prompt Design
52
53	If using LLM-as-judge, write the judge prompt following best practices:
54
55	```
56	You are evaluating an AI assistant's response for [failure type].
57
58	Input to the AI assistant:
59	{input}
60
61	AI assistant's response:
62	{response}
63
64	Evaluation criteria:
65	[Criterion 1]: [Clear definition of what good looks like]
66	[Criterion 2]: [Clear definition of what good looks like]
67
68	Scoring:
69	- PASS: The response [specific pass condition]
70	- FAIL: The response [specific fail condition]
71
72	Your output:
73	First, briefly explain your reasoning (1–2 sentences).
74	Then output: PASS or FAIL
75	```
76
77	Key principles for judge prompts:
78	- Binary outputs (PASS/FAIL) are more reliable than numeric scores
79	- Include examples of PASS and FAIL in the prompt when possible
80	- The judge should explain its reasoning before giving the verdict (chain-of-thought)
81	- Calibrate the judge against 50+ human annotations before trusting it
82
83	## Step 5 — The Principal Domain Expert Model
84
85	From Hamel Husain: designate one "benevolent dictator" for quality — one person whose judgment defines what PASS/FAIL means for subjective evals. This prevents annotation conflicts and anchors the LLM-as-judge calibration.
86
87	This person:
88	- Reviews 50–100 human annotation cases to establish the quality bar
89	- Resolves disagreements between annotators
90	- Periodically reviews LLM-as-judge outputs to catch drift
91
92	## Step 6 — Eval Suite Structure
93
94	Design the full suite as three layers:
95
96	Layer 1 — Pre-commit (fast): Code-based evals only. Run on every code change. Must complete in < 60 seconds. Catches format and structural failures.
97
98	Layer 2 — Pre-deploy (medium): Code-based + LLM-as-judge on a representative sample. Run before any deployment. Should complete in < 10 minutes.
99
100	**Layer 3 — Production monitoring (ongoing):

Productfculty-aipmPublisher · 69 listings

Preview

Productfculty-aipm/PM-Copilot-by-Product-Faculty

# Eval Suite Design

You are designing an evaluation suite for an AI product feature — a systematic set of tests that catches real failure modes before they reach users. The goal is

Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), Aman Khan (Beyond vibe checks, 2025).

Key principle: "Evals quietly decide whether your AI product thrives or dies. The ability to write great evals is rapidly becoming the defining skill for AI PMs

RepoProductfculty-aipm/PM-Copilot-by-Product-Faculty

License—

TypeSkills

CategoryTesting

UpdatedApr 2026

Repo stars70

First seenJul 22, 2026

People who liked this also viewed

6 picks

Type