advanced-evaluation

Name: advanced-evaluation
Author: muratcankoylan

45installs1.4kforks

How to install advanced-evaluation?

muratcankoylan/agent-skills-for-context-engineering/advanced-evaluation

$npx -y skills add muratcankoylan/Agent-Skills-for-Context-Engineering --skill advanced-evaluation

Installs into the current project.

Summary

This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated quality assessment.

Files · 1

View on GitHub

SKILL.md

1	# Advanced Evaluation
2
3	This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.
4
5	Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.
6
7	## When to Activate
8
9	Activate this skill when:
10
11	- Building LLM-as-judge systems for LLM outputs
12	- Comparing multiple model responses to select the best one
13	- Establishing consistent quality standards across evaluation teams
14	- Debugging evaluation systems that show inconsistent results
15	- Designing A/B tests for prompt or model changes
16	- Creating rubrics specifically for LLM or human/LLM hybrid judges
17	- Analyzing correlation between automated and human judgments
18
19	Do not activate this skill for adjacent work owned by other skills:
20	- General deterministic checks, regression suites, production quality gates, or outcome metrics: `evaluation`.
21	- Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries: `harness-engineering`.
22	- Tool API contracts for evaluation tools: `tool-design`.
23
24	## Core Concepts
25
26	### The Evaluation Taxonomy
27
28	Select between two primary approaches based on whether ground truth exists:
29
30	Direct Scoring — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.
31
32	Pairwise Comparison — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Pairwise methods often correlate better with human preference than open-ended direct scoring for subjective tasks (claim-advanced-evaluation-position-swap). Watch for position bias and length bias.
33
34	### The Bias Landscape
35
36	Mitigate these systematic biases in every evaluation system:
37
38	Position Bias: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.
39
40	Length Bias: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.
41
42	Self-Enhancement Bias: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.
43
44	Verbosity Bias: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.
45
46	Authority Bias: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.
47
48	### Metric Selection Framework
49
50	Match metrics to the evaluation task structure:
51
52	\| Task Type \| Primary Metrics \| Secondary Metrics \|
53	\|-----------\|-----------------\|-------------------\|
54	\| Binary classification (pass/fail) \| Recall, Precision, F1 \| Cohen's kappa \|
55	\| Ordinal scale (1-5 rating) \| Spearman's rho, Kendall's tau \| Cohen's kappa (weighted) \|
56	\| Pairwise preference \| Agreement rate, Position consistency \| Confidence calibration \|
57	\| Multi-label \| Macro-F1, Micro-F1 \| Per-label precision/recall \|
58
59	Prioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.
60
61	## Evaluation Approaches
62
63	### Direct Scoring Implementation
64
65	Build direct scoring with three components: clear criteria, a calibrated scale, and structured output format.
66
67	Criteria Definition Pattern:
68	```
69	Criterion: [Name]
70	Description: [What this criterion measures]
71	Weight: [Relative importance, 0-1]
72	```
73
74	Scale Calibration — Choose scale granularity based on rubric detail:
75	- 1-3: Binary with neutral option, lowest cognitive load
76	- 1-5: Standard Likert, best balance of granularity and reliability
77	- 1-10: Use only with detailed per-level rubrics because calibration is harder
78
79	Prompt Structure for Direct Scoring:
80	```
81	You are an expert evaluator assessing response quality.
82
83	## Task
84	Evaluate the following response against each criterion.
85
86	## Original Prompt
87	{prompt}
88
89	## Response to Evaluate
90	{response}
91
92	## Criteria
93	{for each criterion: name, description, weight}
94
95	## Instructions
96	For each criterion:
97	1. Find specific evidence in the response
98	2. Score according to the rubric (1-{max} scale)
99	3. Justify your score with evidence
100	4. Suggest one specific improve

People who liked this also viewed

6 picks

Type

advanced-evaluation

45installs1.4kforks

View on GitHub

How to install advanced-evaluation?

muratcankoylan/agent-skills-for-context-engineering/advanced-evaluation

$npx -y skills add muratcankoylan/Agent-Skills-for-Context-Engineering --skill advanced-evaluation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Advanced Evaluation
2
3	This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.
4
5	Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.
6
7	## When to Activate
8
9	Activate this skill when:
10
11	- Building LLM-as-judge systems for LLM outputs
12	- Comparing multiple model responses to select the best one
13	- Establishing consistent quality standards across evaluation teams
14	- Debugging evaluation systems that show inconsistent results
15	- Designing A/B tests for prompt or model changes
16	- Creating rubrics specifically for LLM or human/LLM hybrid judges
17	- Analyzing correlation between automated and human judgments
18
19	Do not activate this skill for adjacent work owned by other skills:
20	- General deterministic checks, regression suites, production quality gates, or outcome metrics: `evaluation`.
21	- Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries: `harness-engineering`.
22	- Tool API contracts for evaluation tools: `tool-design`.
23
24	## Core Concepts
25
26	### The Evaluation Taxonomy
27
28	Select between two primary approaches based on whether ground truth exists:
29
30	Direct Scoring — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.
31
32	Pairwise Comparison — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Pairwise methods often correlate better with human preference than open-ended direct scoring for subjective tasks (claim-advanced-evaluation-position-swap). Watch for position bias and length bias.
33
34	### The Bias Landscape
35
36	Mitigate these systematic biases in every evaluation system:
37
38	Position Bias: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.
39
40	Length Bias: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.
41
42	Self-Enhancement Bias: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.
43
44	Verbosity Bias: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.
45
46	Authority Bias: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.
47
48	### Metric Selection Framework
49
50	Match metrics to the evaluation task structure:
51
52	\| Task Type \| Primary Metrics \| Secondary Metrics \|
53	\|-----------\|-----------------\|-------------------\|
54	\| Binary classification (pass/fail) \| Recall, Precision, F1 \| Cohen's kappa \|
55	\| Ordinal scale (1-5 rating) \| Spearman's rho, Kendall's tau \| Cohen's kappa (weighted) \|
56	\| Pairwise preference \| Agreement rate, Position consistency \| Confidence calibration \|
57	\| Multi-label \| Macro-F1, Micro-F1 \| Per-label precision/recall \|
58
59	Prioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.
60
61	## Evaluation Approaches
62
63	### Direct Scoring Implementation
64
65	Build direct scoring with three components: clear criteria, a calibrated scale, and structured output format.
66
67	Criteria Definition Pattern:
68	```
69	Criterion: [Name]
70	Description: [What this criterion measures]
71	Weight: [Relative importance, 0-1]
72	```
73
74	Scale Calibration — Choose scale granularity based on rubric detail:
75	- 1-3: Binary with neutral option, lowest cognitive load
76	- 1-5: Standard Likert, best balance of granularity and reliability
77	- 1-10: Use only with detailed per-level rubrics because calibration is harder
78
79	Prompt Structure for Direct Scoring:
80	```
81	You are an expert evaluator assessing response quality.
82
83	## Task
84	Evaluate the following response against each criterion.
85
86	## Original Prompt
87	{prompt}
88
89	## Response to Evaluate
90	{response}
91
92	## Criteria
93	{for each criterion: name, description, weight}
94
95	## Instructions
96	For each criterion:
97	1. Find specific evidence in the response
98	2. Score according to the rubric (1-{max} scale)
99	3. Justify your score with evidence
100	4. Suggest one specific improve

People who liked this also viewed

6 picks

Type