advanced-evaluation

Name: advanced-evaluation
Author: sickn33

44kstars6.5kforks

How to install advanced-evaluation?

sickn33/agentic-awesome-skills/advanced-evaluation

$npx -y skills add sickn33/agentic-awesome-skills --skill advanced-evaluation

Installs into the current project.

Summary

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

Files · 1

View on GitHub

SKILL.md

1	# Advanced Evaluation
2
3	This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.
4
5	Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.
6
7	## When to Use
8	Activate this skill when:
9
10	- Building automated evaluation pipelines for LLM outputs
11	- Comparing multiple model responses to select the best one
12	- Establishing consistent quality standards across evaluation teams
13	- Debugging evaluation systems that show inconsistent results
14	- Designing A/B tests for prompt or model changes
15	- Creating rubrics for human or automated evaluation
16	- Analyzing correlation between automated and human judgments
17
18	## Core Concepts
19
20	### The Evaluation Taxonomy
21
22	Evaluation approaches fall into two primary categories with distinct reliability profiles:
23
24	Direct Scoring: A single LLM rates one response on a defined scale.
25	- Best for: Objective criteria (factual accuracy, instruction following, toxicity)
26	- Reliability: Moderate to high for well-defined criteria
27	- Failure mode: Score calibration drift, inconsistent scale interpretation
28
29	Pairwise Comparison: An LLM compares two responses and selects the better one.
30	- Best for: Subjective preferences (tone, style, persuasiveness)
31	- Reliability: Higher than direct scoring for preferences
32	- Failure mode: Position bias, length bias
33
34	Research from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth.
35
36	### The Bias Landscape
37
38	LLM judges exhibit systematic biases that must be actively mitigated:
39
40	Position Bias: First-position responses receive preferential treatment in pairwise comparison. Mitigation: Evaluate twice with swapped positions, use majority vote or consistency check.
41
42	Length Bias: Longer responses are rated higher regardless of quality. Mitigation: Explicit prompting to ignore length, length-normalized scoring.
43
44	Self-Enhancement Bias: Models rate their own outputs higher. Mitigation: Use different models for generation and evaluation, or acknowledge limitation.
45
46	Verbosity Bias: Detailed explanations receive higher scores even when unnecessary. Mitigation: Criteria-specific rubrics that penalize irrelevant detail.
47
48	Authority Bias: Confident, authoritative tone rated higher regardless of accuracy. Mitigation: Require evidence citation, fact-checking layer.
49
50	### Metric Selection Framework
51
52	Choose metrics based on the evaluation task structure:
53
54	\| Task Type \| Primary Metrics \| Secondary Metrics \|
55	\|-----------\|-----------------\|-------------------\|
56	\| Binary classification (pass/fail) \| Recall, Precision, F1 \| Cohen's κ \|
57	\| Ordinal scale (1-5 rating) \| Spearman's ρ, Kendall's τ \| Cohen's κ (weighted) \|
58	\| Pairwise preference \| Agreement rate, Position consistency \| Confidence calibration \|
59	\| Multi-label \| Macro-F1, Micro-F1 \| Per-label precision/recall \|
60
61	The critical insight: High absolute agreement matters less than systematic disagreement patterns. A judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.
62
63	## Evaluation Approaches
64
65	### Direct Scoring Implementation
66
67	Direct scoring requires three components: clear criteria, a calibrated scale, and structured output format.
68
69	Criteria Definition Pattern:
70	```
71	Criterion: [Name]
72	Description: [What this criterion measures]
73	Weight: [Relative importance, 0-1]
74	```
75
76	Scale Calibration:
77	- 1-3 scales: Binary with neutral option, lowest cognitive load
78	- 1-5 scales: Standard Likert, good balance of granularity and reliability
79	- 1-10 scales: High granularity but harder to calibrate, use only with detailed rubrics
80
81	Prompt Structure for Direct Scoring:
82	```
83	You are an expert evaluator assessing response quality.
84
85	## Task
86	Evaluate the following response against each criterion.
87
88	## Original Prompt
89	{prompt}
90
91	## Response to Evaluate
92	{response}
93
94	## Criteria
95	{for each criterion: name, description, weight}
96
97	## Instructions
98	For each criterion:
99	1. Find specific evidence in the response
100	2. Score according to the rubric (1-{max} scale)
101	3. Justify your score with evidence
102	4. Suggest one specific improvement
103
104	## Output Forma

People who liked this also viewed

6 picks

Type

advanced-evaluation

44kstars6.5kforks

View on GitHub

How to install advanced-evaluation?

sickn33/agentic-awesome-skills/advanced-evaluation

$npx -y skills add sickn33/agentic-awesome-skills --skill advanced-evaluation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Advanced Evaluation
2
3	This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.
4
5	Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.
6
7	## When to Use
8	Activate this skill when:
9
10	- Building automated evaluation pipelines for LLM outputs
11	- Comparing multiple model responses to select the best one
12	- Establishing consistent quality standards across evaluation teams
13	- Debugging evaluation systems that show inconsistent results
14	- Designing A/B tests for prompt or model changes
15	- Creating rubrics for human or automated evaluation
16	- Analyzing correlation between automated and human judgments
17
18	## Core Concepts
19
20	### The Evaluation Taxonomy
21
22	Evaluation approaches fall into two primary categories with distinct reliability profiles:
23
24	Direct Scoring: A single LLM rates one response on a defined scale.
25	- Best for: Objective criteria (factual accuracy, instruction following, toxicity)
26	- Reliability: Moderate to high for well-defined criteria
27	- Failure mode: Score calibration drift, inconsistent scale interpretation
28
29	Pairwise Comparison: An LLM compares two responses and selects the better one.
30	- Best for: Subjective preferences (tone, style, persuasiveness)
31	- Reliability: Higher than direct scoring for preferences
32	- Failure mode: Position bias, length bias
33
34	Research from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth.
35
36	### The Bias Landscape
37
38	LLM judges exhibit systematic biases that must be actively mitigated:
39
40	Position Bias: First-position responses receive preferential treatment in pairwise comparison. Mitigation: Evaluate twice with swapped positions, use majority vote or consistency check.
41
42	Length Bias: Longer responses are rated higher regardless of quality. Mitigation: Explicit prompting to ignore length, length-normalized scoring.
43
44	Self-Enhancement Bias: Models rate their own outputs higher. Mitigation: Use different models for generation and evaluation, or acknowledge limitation.
45
46	Verbosity Bias: Detailed explanations receive higher scores even when unnecessary. Mitigation: Criteria-specific rubrics that penalize irrelevant detail.
47
48	Authority Bias: Confident, authoritative tone rated higher regardless of accuracy. Mitigation: Require evidence citation, fact-checking layer.
49
50	### Metric Selection Framework
51
52	Choose metrics based on the evaluation task structure:
53
54	\| Task Type \| Primary Metrics \| Secondary Metrics \|
55	\|-----------\|-----------------\|-------------------\|
56	\| Binary classification (pass/fail) \| Recall, Precision, F1 \| Cohen's κ \|
57	\| Ordinal scale (1-5 rating) \| Spearman's ρ, Kendall's τ \| Cohen's κ (weighted) \|
58	\| Pairwise preference \| Agreement rate, Position consistency \| Confidence calibration \|
59	\| Multi-label \| Macro-F1, Micro-F1 \| Per-label precision/recall \|
60
61	The critical insight: High absolute agreement matters less than systematic disagreement patterns. A judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.
62
63	## Evaluation Approaches
64
65	### Direct Scoring Implementation
66
67	Direct scoring requires three components: clear criteria, a calibrated scale, and structured output format.
68
69	Criteria Definition Pattern:
70	```
71	Criterion: [Name]
72	Description: [What this criterion measures]
73	Weight: [Relative importance, 0-1]
74	```
75
76	Scale Calibration:
77	- 1-3 scales: Binary with neutral option, lowest cognitive load
78	- 1-5 scales: Standard Likert, good balance of granularity and reliability
79	- 1-10 scales: High granularity but harder to calibrate, use only with detailed rubrics
80
81	Prompt Structure for Direct Scoring:
82	```
83	You are an expert evaluator assessing response quality.
84
85	## Task
86	Evaluate the following response against each criterion.
87
88	## Original Prompt
89	{prompt}
90
91	## Response to Evaluate
92	{response}
93
94	## Criteria
95	{for each criterion: name, description, weight}
96
97	## Instructions
98	For each criterion:
99	1. Find specific evidence in the response
100	2. Score according to the rubric (1-{max} scale)
101	3. Justify your score with evidence
102	4. Suggest one specific improvement
103
104	## Output Forma

People who liked this also viewed

6 picks

Type