llm-as-judge

Name: llm-as-judge
Author: Productfculty-aipm

70stars33forks

How to install llm-as-judge?

productfculty-aipm/pm-copilot-by-product-faculty/llm-as-judge

$npx -y skills add Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill llm-as-judge

Installs into the current project.

Summary

Use this skill when the user asks to "set up LLM as a judge", "write an LLM judge prompt", "automate quality evaluation", "use Claude to evaluate outputs", "build an automated eval", "LLM-based evaluation", or wants to create a scalable automated evaluation system where one LLM g

Files · 1

View on GitHub

SKILL.md

1	# LLM-as-Judge Setup
2
3	You are setting up an LLM-as-judge evaluation system — a scalable way to automatically evaluate AI output quality by using a stronger or specialized LLM to grade the outputs of the product's AI.
4
5	Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), Anthropic's evaluation methodology.
6
7	## Step 1 — Load Context
8
9	Read `memory/user-profile.md` for the AI feature being evaluated. Read the error analysis output if available to understand the failure categories to target.
10
11	## Step 2 — When to Use LLM-as-Judge
12
13	Use LLM-as-judge when:
14	- Human evaluation is the gold standard but too slow or expensive to run at scale
15	- The failure mode requires reasoning to detect (not just structural checking)
16	- You need to evaluate thousands of outputs on a regular cadence
17	- You've calibrated the judge against human annotations and trust it
18
19	Do NOT use LLM-as-judge when:
20	- You haven't calibrated it against human evals first (calibration is mandatory)
21	- The task has an objectively correct answer (use code-based evals instead)
22	- The failure mode is one the judge model is known to be bad at (e.g., detecting subtle factual errors in specialized domains)
23
24	## Step 3 — Judge Architecture Options
25
26	Option A — Binary (Recommended for most cases):
27	Judge outputs PASS or FAIL with a brief explanation. Simple, reliable, easy to aggregate.
28
29	Option B — Rubric (Use when more granularity is needed):
30	Judge scores each of 3–5 criteria on a 1–5 scale. Good for quality tracking over time.
31
32	Option C — Comparative (Use when evaluating prompt variants):
33	Judge sees two outputs side-by-side and picks the better one. Best for A/B testing prompt changes.
34
35	Option D — Error detection (Use when targeting specific failure categories):
36	Judge specifically looks for a known failure type (e.g., "does this output contain any unsupported factual claims?").
37
38	## Step 4 — Judge Prompt Template
39
40	Structure every judge prompt with these sections:
41
42	```markdown
43	# Evaluation Task
44
45	You are evaluating the quality of an AI assistant's response. Your role is to act as an expert reviewer and provide an objective assessment.
46
47	## Context
48	[Brief description of what the AI assistant is supposed to do]
49
50	## What you're evaluating for
51	[Specific quality criteria — be precise about what PASS and FAIL mean]
52
53	## The evaluation
54
55	User input:
56	{{input}}
57
58	AI assistant's response:
59	{{output}}
60
61	## Instructions
62	1. Analyze the response against the criteria above
63	2. Write your reasoning in 2–3 sentences
64	3. State your verdict: PASS or FAIL
65
66	## Output format
67	Reasoning: [Your 2–3 sentence analysis]
68	Verdict: [PASS or FAIL]
69	```
70
71	## Step 5 — Calibration Protocol
72
73	Before deploying LLM-as-judge in production, calibrate it:
74
75	1. Collect 100 (input, output) pairs
76	2. Have the principal domain expert (or 2 human annotators) label each: PASS or FAIL
77	3. Run the judge on the same 100 pairs
78	4. Calculate agreement rate: (matching judgments) / 100
79	5. Target: > 85% agreement with human labels
80
81	If agreement is < 85%:
82	- Add more examples to the judge prompt
83	- Tighten or clarify the evaluation criteria
84	- Try a different model (claude-opus-4-6 is more accurate as a judge for complex quality)
85	- Split one rubric criterion into two more specific criteria
86
87	## Step 6 — Avoiding Common Judge Failures
88
89	Position bias: Judge tends to prefer responses in a certain format or length. Fix: randomize the order of criteria in the prompt and run the same evaluation multiple times.
90
91	Verbosity bias: Judge rates longer responses as better even when they're not. Fix: explicitly state in the prompt "Length is not a quality signal. Evaluate based on [specific criteria]."
92
93	Self-evaluation bias: An LLM tends to give high ratings to outputs that look like its own generation style. Fix: use a different model as judge than the model being evaluated.
94
95	Instruction following: If the judge doesn't follow the output format, extract the verdict with a regex or second parsing step.
96
97	## Step 7 — Integration Pattern
98
99	Provide a Python pseudocode template for integrating LLM-as-judge into the evaluation pipeline:
100
101	```python
102	import anthropic
103
104	def evaluate_output(user_input: str, ai_output: str, judge_prompt: str) -> dict:
105	client = anthropic.Anthropic()
106
107	judge_input = judge_prompt.replace("{{input}}", user_input).replace("{{output}}", ai_output)
108
109	response = client.messages.create(
110	model="claude-opus-4-6", # Use strongest model as judge
111	max_tokens=500,
112	messages=[{"role": "user", "content": judge_input}]
113	)
114
115	verdict_text = response.content[0].text
116	verdict = "PASS" if "Verdict: PASS

People who liked this also viewed

6 picks

Type

llm-as-judge

70stars33forks

View on GitHub

How to install llm-as-judge?

productfculty-aipm/pm-copilot-by-product-faculty/llm-as-judge

$npx -y skills add Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill llm-as-judge

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# LLM-as-Judge Setup
2
3	You are setting up an LLM-as-judge evaluation system — a scalable way to automatically evaluate AI output quality by using a stronger or specialized LLM to grade the outputs of the product's AI.
4
5	Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), Anthropic's evaluation methodology.
6
7	## Step 1 — Load Context
8
9	Read `memory/user-profile.md` for the AI feature being evaluated. Read the error analysis output if available to understand the failure categories to target.
10
11	## Step 2 — When to Use LLM-as-Judge
12
13	Use LLM-as-judge when:
14	- Human evaluation is the gold standard but too slow or expensive to run at scale
15	- The failure mode requires reasoning to detect (not just structural checking)
16	- You need to evaluate thousands of outputs on a regular cadence
17	- You've calibrated the judge against human annotations and trust it
18
19	Do NOT use LLM-as-judge when:
20	- You haven't calibrated it against human evals first (calibration is mandatory)
21	- The task has an objectively correct answer (use code-based evals instead)
22	- The failure mode is one the judge model is known to be bad at (e.g., detecting subtle factual errors in specialized domains)
23
24	## Step 3 — Judge Architecture Options
25
26	Option A — Binary (Recommended for most cases):
27	Judge outputs PASS or FAIL with a brief explanation. Simple, reliable, easy to aggregate.
28
29	Option B — Rubric (Use when more granularity is needed):
30	Judge scores each of 3–5 criteria on a 1–5 scale. Good for quality tracking over time.
31
32	Option C — Comparative (Use when evaluating prompt variants):
33	Judge sees two outputs side-by-side and picks the better one. Best for A/B testing prompt changes.
34
35	Option D — Error detection (Use when targeting specific failure categories):
36	Judge specifically looks for a known failure type (e.g., "does this output contain any unsupported factual claims?").
37
38	## Step 4 — Judge Prompt Template
39
40	Structure every judge prompt with these sections:
41
42	```markdown
43	# Evaluation Task
44
45	You are evaluating the quality of an AI assistant's response. Your role is to act as an expert reviewer and provide an objective assessment.
46
47	## Context
48	[Brief description of what the AI assistant is supposed to do]
49
50	## What you're evaluating for
51	[Specific quality criteria — be precise about what PASS and FAIL mean]
52
53	## The evaluation
54
55	User input:
56	{{input}}
57
58	AI assistant's response:
59	{{output}}
60
61	## Instructions
62	1. Analyze the response against the criteria above
63	2. Write your reasoning in 2–3 sentences
64	3. State your verdict: PASS or FAIL
65
66	## Output format
67	Reasoning: [Your 2–3 sentence analysis]
68	Verdict: [PASS or FAIL]
69	```
70
71	## Step 5 — Calibration Protocol
72
73	Before deploying LLM-as-judge in production, calibrate it:
74
75	1. Collect 100 (input, output) pairs
76	2. Have the principal domain expert (or 2 human annotators) label each: PASS or FAIL
77	3. Run the judge on the same 100 pairs
78	4. Calculate agreement rate: (matching judgments) / 100
79	5. Target: > 85% agreement with human labels
80
81	If agreement is < 85%:
82	- Add more examples to the judge prompt
83	- Tighten or clarify the evaluation criteria
84	- Try a different model (claude-opus-4-6 is more accurate as a judge for complex quality)
85	- Split one rubric criterion into two more specific criteria
86
87	## Step 6 — Avoiding Common Judge Failures
88
89	Position bias: Judge tends to prefer responses in a certain format or length. Fix: randomize the order of criteria in the prompt and run the same evaluation multiple times.
90
91	Verbosity bias: Judge rates longer responses as better even when they're not. Fix: explicitly state in the prompt "Length is not a quality signal. Evaluate based on [specific criteria]."
92
93	Self-evaluation bias: An LLM tends to give high ratings to outputs that look like its own generation style. Fix: use a different model as judge than the model being evaluated.
94
95	Instruction following: If the judge doesn't follow the output format, extract the verdict with a regex or second parsing step.
96
97	## Step 7 — Integration Pattern
98
99	Provide a Python pseudocode template for integrating LLM-as-judge into the evaluation pipeline:
100
101	```python
102	import anthropic
103
104	def evaluate_output(user_input: str, ai_output: str, judge_prompt: str) -> dict:
105	client = anthropic.Anthropic()
106
107	judge_input = judge_prompt.replace("{{input}}", user_input).replace("{{output}}", ai_output)
108
109	response = client.messages.create(
110	model="claude-opus-4-6", # Use strongest model as judge
111	max_tokens=500,
112	messages=[{"role": "user", "content": judge_input}]
113	)
114
115	verdict_text = response.content[0].text
116	verdict = "PASS" if "Verdict: PASS

People who liked this also viewed

6 picks

Type