agentic-eval

Name: agentic-eval
Author: github

9.9kinstalls4.6kforks

How to install agentic-eval?

github/awesome-copilot/agentic-eval

$npx -y skills add github/awesome-copilot --skill agentic-eval

Installs into the current project.

Summary

Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when: - Implementing self-critique and reflection loops - Building evaluator-optimizer pipelines for quality-critical generation - Creating test-driven code refinement workflows - Designing rubr

Files · 1

View on GitHub

SKILL.md

1	# Agentic Evaluation Patterns
2
3	Patterns for self-improvement through iterative evaluation and refinement.
4
5	## Overview
6
7	Evaluation patterns enable agents to assess and improve their own outputs, moving beyond single-shot generation to iterative refinement loops.
8
9	```
10	Generate → Evaluate → Critique → Refine → Output
11	↑ │
12	└──────────────────────────────┘
13	```
14
15	## When to Use
16
17	- Quality-critical generation: Code, reports, analysis requiring high accuracy
18	- Tasks with clear evaluation criteria: Defined success metrics exist
19	- Content requiring specific standards: Style guides, compliance, formatting
20
21	---
22
23	## Pattern 1: Basic Reflection
24
25	Agent evaluates and improves its own output through self-critique.
26
27	```python
28	def reflect_and_refine(task: str, criteria: list[str], max_iterations: int = 3) -> str:
29	"""Generate with reflection loop."""
30	output = llm(f"Complete this task:\n{task}")
31
32	for i in range(max_iterations):
33	# Self-critique
34	critique = llm(f"""
35	Evaluate this output against criteria: {criteria}
36	Output: {output}
37	Rate each: PASS/FAIL with feedback as JSON.
38	""")
39
40	critique_data = json.loads(critique)
41	all_pass = all(c["status"] == "PASS" for c in critique_data.values())
42	if all_pass:
43	return output
44
45	# Refine based on critique
46	failed = {k: v["feedback"] for k, v in critique_data.items() if v["status"] == "FAIL"}
47	output = llm(f"Improve to address: {failed}\nOriginal: {output}")
48
49	return output
50	```
51
52	Key insight: Use structured JSON output for reliable parsing of critique results.
53
54	---
55
56	## Pattern 2: Evaluator-Optimizer
57
58	Separate generation and evaluation into distinct components for clearer responsibilities.
59
60	```python
61	class EvaluatorOptimizer:
62	def __init__(self, score_threshold: float = 0.8):
63	self.score_threshold = score_threshold
64
65	def generate(self, task: str) -> str:
66	return llm(f"Complete: {task}")
67
68	def evaluate(self, output: str, task: str) -> dict:
69	return json.loads(llm(f"""
70	Evaluate output for task: {task}
71	Output: {output}
72	Return JSON: {{"overall_score": 0-1, "dimensions": {{"accuracy": ..., "clarity": ...}}}}
73	"""))
74
75	def optimize(self, output: str, feedback: dict) -> str:
76	return llm(f"Improve based on feedback: {feedback}\nOutput: {output}")
77
78	def run(self, task: str, max_iterations: int = 3) -> str:
79	output = self.generate(task)
80	for _ in range(max_iterations):
81	evaluation = self.evaluate(output, task)
82	if evaluation["overall_score"] >= self.score_threshold:
83	break
84	output = self.optimize(output, evaluation)
85	return output
86	```
87
88	---
89
90	## Pattern 3: Code-Specific Reflection
91
92	Test-driven refinement loop for code generation.
93
94	```python
95	class CodeReflector:
96	def reflect_and_fix(self, spec: str, max_iterations: int = 3) -> str:
97	code = llm(f"Write Python code for: {spec}")
98	tests = llm(f"Generate pytest tests for: {spec}\nCode: {code}")
99
100	for _ in range(max_iterations):
101	result = run_tests(code, tests)
102	if result["success"]:
103	return code
104	code = llm(f"Fix error: {result['error']}\nCode: {code}")
105	return code
106	```
107
108	---
109
110	## Evaluation Strategies
111
112	### Outcome-Based
113	Evaluate whether output achieves the expected result.
114
115	```python
116	def evaluate_outcome(task: str, output: str, expected: str) -> str:
117	return llm(f"Does output achieve expected outcome? Task: {task}, Expected: {expected}, Output: {output}")
118	```
119
120	### LLM-as-Judge
121	Use LLM to compare and rank outputs.
122
123	```python
124	def llm_judge(output_a: str, output_b: str, criteria: str) -> str:
125	return llm(f"Compare outputs A and B for {criteria}. Which is better and why?")
126	```
127
128	### Rubric-Based
129	Score outputs against weighted dimensions.
130
131	```python
132	RUBRIC = {
133	"accuracy": {"weight": 0.4},
134	"clarity": {"weight": 0.3},
135	"completeness": {"weight": 0.3}
136	}
137
138	def evaluate_with_rubric(output: str, rubric: dict) -> float:
139	scores = json.loads(llm(f"Rate 1-5 for each dimension: {list(rubric.keys())}\nOutput: {output}"))
140	return sum(scores[d] * rubric[d]["weight"] for d in rubric) / 5
141	```
142
143	---
144
145	## Best Practices
146
147	\| Practice \| Rationale \|
148	\|----------\|-----------\|
149	\| Clear criteria \| Define specific, measurable evaluation criteria up

People who liked this also viewed

6 picks

Type

agentic-eval

9.9kinstalls4.6kforks

View on GitHub

How to install agentic-eval?

github/awesome-copilot/agentic-eval

$npx -y skills add github/awesome-copilot --skill agentic-eval

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Agentic Evaluation Patterns
2
3	Patterns for self-improvement through iterative evaluation and refinement.
4
5	## Overview
6
7	Evaluation patterns enable agents to assess and improve their own outputs, moving beyond single-shot generation to iterative refinement loops.
8
9	```
10	Generate → Evaluate → Critique → Refine → Output
11	↑ │
12	└──────────────────────────────┘
13	```
14
15	## When to Use
16
17	- Quality-critical generation: Code, reports, analysis requiring high accuracy
18	- Tasks with clear evaluation criteria: Defined success metrics exist
19	- Content requiring specific standards: Style guides, compliance, formatting
20
21	---
22
23	## Pattern 1: Basic Reflection
24
25	Agent evaluates and improves its own output through self-critique.
26
27	```python
28	def reflect_and_refine(task: str, criteria: list[str], max_iterations: int = 3) -> str:
29	"""Generate with reflection loop."""
30	output = llm(f"Complete this task:\n{task}")
31
32	for i in range(max_iterations):
33	# Self-critique
34	critique = llm(f"""
35	Evaluate this output against criteria: {criteria}
36	Output: {output}
37	Rate each: PASS/FAIL with feedback as JSON.
38	""")
39
40	critique_data = json.loads(critique)
41	all_pass = all(c["status"] == "PASS" for c in critique_data.values())
42	if all_pass:
43	return output
44
45	# Refine based on critique
46	failed = {k: v["feedback"] for k, v in critique_data.items() if v["status"] == "FAIL"}
47	output = llm(f"Improve to address: {failed}\nOriginal: {output}")
48
49	return output
50	```
51
52	Key insight: Use structured JSON output for reliable parsing of critique results.
53
54	---
55
56	## Pattern 2: Evaluator-Optimizer
57
58	Separate generation and evaluation into distinct components for clearer responsibilities.
59
60	```python
61	class EvaluatorOptimizer:
62	def __init__(self, score_threshold: float = 0.8):
63	self.score_threshold = score_threshold
64
65	def generate(self, task: str) -> str:
66	return llm(f"Complete: {task}")
67
68	def evaluate(self, output: str, task: str) -> dict:
69	return json.loads(llm(f"""
70	Evaluate output for task: {task}
71	Output: {output}
72	Return JSON: {{"overall_score": 0-1, "dimensions": {{"accuracy": ..., "clarity": ...}}}}
73	"""))
74
75	def optimize(self, output: str, feedback: dict) -> str:
76	return llm(f"Improve based on feedback: {feedback}\nOutput: {output}")
77
78	def run(self, task: str, max_iterations: int = 3) -> str:
79	output = self.generate(task)
80	for _ in range(max_iterations):
81	evaluation = self.evaluate(output, task)
82	if evaluation["overall_score"] >= self.score_threshold:
83	break
84	output = self.optimize(output, evaluation)
85	return output
86	```
87
88	---
89
90	## Pattern 3: Code-Specific Reflection
91
92	Test-driven refinement loop for code generation.
93
94	```python
95	class CodeReflector:
96	def reflect_and_fix(self, spec: str, max_iterations: int = 3) -> str:
97	code = llm(f"Write Python code for: {spec}")
98	tests = llm(f"Generate pytest tests for: {spec}\nCode: {code}")
99
100	for _ in range(max_iterations):
101	result = run_tests(code, tests)
102	if result["success"]:
103	return code
104	code = llm(f"Fix error: {result['error']}\nCode: {code}")
105	return code
106	```
107
108	---
109
110	## Evaluation Strategies
111
112	### Outcome-Based
113	Evaluate whether output achieves the expected result.
114
115	```python
116	def evaluate_outcome(task: str, output: str, expected: str) -> str:
117	return llm(f"Does output achieve expected outcome? Task: {task}, Expected: {expected}, Output: {output}")
118	```
119
120	### LLM-as-Judge
121	Use LLM to compare and rank outputs.
122
123	```python
124	def llm_judge(output_a: str, output_b: str, criteria: str) -> str:
125	return llm(f"Compare outputs A and B for {criteria}. Which is better and why?")
126	```
127
128	### Rubric-Based
129	Score outputs against weighted dimensions.
130
131	```python
132	RUBRIC = {
133	"accuracy": {"weight": 0.4},
134	"clarity": {"weight": 0.3},
135	"completeness": {"weight": 0.3}
136	}
137
138	def evaluate_with_rubric(output: str, rubric: dict) -> float:
139	scores = json.loads(llm(f"Rate 1-5 for each dimension: {list(rubric.keys())}\nOutput: {output}"))
140	return sum(scores[d] * rubric[d]["weight"] for d in rubric) / 5
141	```
142
143	---
144
145	## Best Practices
146
147	\| Practice \| Rationale \|
148	\|----------\|-----------\|
149	\| Clear criteria \| Define specific, measurable evaluation criteria up

People who liked this also viewed

6 picks

Type