ai-agent-eval

Name: ai-agent-eval
Author: PramodDutta

182stars17forks

How to install ai-agent-eval?

pramoddutta/qaskills/ai-agent-eval

$npx -y skills add PramodDutta/qaskills --skill ai-agent-eval

Installs into the current project.

Summary

Comprehensive evaluation patterns for AI agents including multi-turn conversation testing, LLM-as-judge frameworks, benchmark suites, regression detection, and systematic eval pipelines for measuring agent quality and safety.

Files · 1

View on GitHub

SKILL.md

1	# AI Agent Evaluation Skill
2
3	You are an expert in evaluating AI agents and LLM-powered systems. When the user asks you to build evaluation frameworks, create benchmarks, implement LLM-as-judge patterns, test multi-turn conversations, or measure agent quality, follow these detailed instructions to produce robust, reproducible evaluation systems.
4
5	## Core Principles
6
7	1. Deterministic evaluation pipelines -- Every eval must be reproducible. Pin model versions, temperatures, seed values, and system prompts so results can be compared across runs.
8	2. Multi-dimensional scoring -- Never rely on a single metric. Evaluate correctness, helpfulness, safety, latency, cost, and task completion as separate dimensions.
9	3. LLM-as-judge with calibration -- When using LLMs to judge outputs, calibrate judges against human annotations and measure inter-judge agreement before trusting automated scores.
10	4. Golden dataset management -- Maintain versioned datasets of input/expected-output pairs. Tag each example with difficulty, category, and edge-case classification.
11	5. Regression detection over absolute scores -- Track score changes between agent versions rather than chasing absolute numbers. A 2% drop from a reliable baseline matters more than a 90% absolute score.
12	6. Safety and alignment testing -- Every eval suite must include adversarial inputs, prompt injection attempts, and boundary-testing cases that verify the agent refuses harmful requests.
13	7. Statistical rigor -- Report confidence intervals, run multiple trials, and use proper statistical tests when comparing agent versions. Never declare a winner based on a single run.
14
15	## Project Structure
16
17	```
18	evals/
19	datasets/
20	golden/
21	coding-tasks.jsonl
22	qa-pairs.jsonl
23	multi-turn-conversations.jsonl
24	adversarial-inputs.jsonl
25	edge-cases.jsonl
26	generated/
27	synthetic-tasks.jsonl
28	judges/
29	correctness-judge.ts
30	helpfulness-judge.ts
31	safety-judge.ts
32	code-quality-judge.ts
33	composite-judge.ts
34	runners/
35	eval-runner.ts
36	batch-runner.ts
37	parallel-runner.ts
38	metrics/
39	scoring.ts
40	statistical.ts
41	aggregation.ts
42	reports/
43	html-reporter.ts
44	json-reporter.ts
45	regression-detector.ts
46	config/
47	eval-config.ts
48	model-config.ts
49	tests/
50	judge-calibration.test.ts
51	metric-accuracy.test.ts
52	pipeline-integration.test.ts
53	results/
54	.gitkeep
55	```
56
57	## Eval Dataset Format
58
59	```typescript
60	// evals/datasets/types.ts
61	export interface EvalExample {
62	id: string;
63	input: string \| ConversationTurn[];
64	expectedOutput?: string;
65	expectedBehavior?: string;
66	tags: string[];
67	difficulty: 'easy' \| 'medium' \| 'hard' \| 'adversarial';
68	category: string;
69	metadata?: Record<string, unknown>;
70	}
71
72	export interface ConversationTurn {
73	role: 'user' \| 'assistant' \| 'system';
74	content: string;
75	}
76
77	export interface EvalResult {
78	exampleId: string;
79	agentOutput: string;
80	scores: Record<string, number>;
81	judgeReasonings: Record<string, string>;
82	latencyMs: number;
83	tokenUsage: { input: number; output: number };
84	timestamp: string;
85	agentVersion: string;
86	error?: string;
87	}
88
89	export interface EvalSuiteResult {
90	suiteId: string;
91	agentVersion: string;
92	timestamp: string;
93	results: EvalResult[];
94	aggregateScores: Record<string, AggregateScore>;
95	totalExamples: number;
96	passedExamples: number;
97	failedExamples: number;
98	errorExamples: number;
99	}
100
101	export interface AggregateScore {
102	mean: number;
103	median: number;
104	stdDev: number;
105	min: number;
106	max: number;
107	p5: number;
108	p95: number;
109	confidenceInterval: { lower: number; upper: number };
110	sampleSize: number;
111	}
112	```
113
114	## LLM-as-Judge Implementation
115
116	```typescript
117	// evals/judges/correctness-judge.ts
118	import Anthropic from '@anthropic-ai/sdk';
119
120	export interface JudgeResult {
121	score: number; // 0-10 scale
122	reasoning: string;
123	confidence: number; // 0-1
124	flags: string[];
125	}
126
127	export interface JudgeConfig {
128	model: string;
129	temperature: number;
130	maxTokens: number;
131	systemPrompt: string;
132	scoringRubric: string;
133	}
134
135	const DEFAULT_CORRECTNESS_CONFIG: JudgeConfig = {
136	model: 'claude-sonnet-4-20250514',
137	temperature: 0,
138	maxTokens: 1024,
139	systemPrompt: `You are an expert evaluator assessing the correctness of AI

People who liked this also viewed

6 picks

Type

ai-agent-eval

182stars17forks

View on GitHub

How to install ai-agent-eval?

pramoddutta/qaskills/ai-agent-eval

$npx -y skills add PramodDutta/qaskills --skill ai-agent-eval

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# AI Agent Evaluation Skill
2
3	You are an expert in evaluating AI agents and LLM-powered systems. When the user asks you to build evaluation frameworks, create benchmarks, implement LLM-as-judge patterns, test multi-turn conversations, or measure agent quality, follow these detailed instructions to produce robust, reproducible evaluation systems.
4
5	## Core Principles
6
7	1. Deterministic evaluation pipelines -- Every eval must be reproducible. Pin model versions, temperatures, seed values, and system prompts so results can be compared across runs.
8	2. Multi-dimensional scoring -- Never rely on a single metric. Evaluate correctness, helpfulness, safety, latency, cost, and task completion as separate dimensions.
9	3. LLM-as-judge with calibration -- When using LLMs to judge outputs, calibrate judges against human annotations and measure inter-judge agreement before trusting automated scores.
10	4. Golden dataset management -- Maintain versioned datasets of input/expected-output pairs. Tag each example with difficulty, category, and edge-case classification.
11	5. Regression detection over absolute scores -- Track score changes between agent versions rather than chasing absolute numbers. A 2% drop from a reliable baseline matters more than a 90% absolute score.
12	6. Safety and alignment testing -- Every eval suite must include adversarial inputs, prompt injection attempts, and boundary-testing cases that verify the agent refuses harmful requests.
13	7. Statistical rigor -- Report confidence intervals, run multiple trials, and use proper statistical tests when comparing agent versions. Never declare a winner based on a single run.
14
15	## Project Structure
16
17	```
18	evals/
19	datasets/
20	golden/
21	coding-tasks.jsonl
22	qa-pairs.jsonl
23	multi-turn-conversations.jsonl
24	adversarial-inputs.jsonl
25	edge-cases.jsonl
26	generated/
27	synthetic-tasks.jsonl
28	judges/
29	correctness-judge.ts
30	helpfulness-judge.ts
31	safety-judge.ts
32	code-quality-judge.ts
33	composite-judge.ts
34	runners/
35	eval-runner.ts
36	batch-runner.ts
37	parallel-runner.ts
38	metrics/
39	scoring.ts
40	statistical.ts
41	aggregation.ts
42	reports/
43	html-reporter.ts
44	json-reporter.ts
45	regression-detector.ts
46	config/
47	eval-config.ts
48	model-config.ts
49	tests/
50	judge-calibration.test.ts
51	metric-accuracy.test.ts
52	pipeline-integration.test.ts
53	results/
54	.gitkeep
55	```
56
57	## Eval Dataset Format
58
59	```typescript
60	// evals/datasets/types.ts
61	export interface EvalExample {
62	id: string;
63	input: string \| ConversationTurn[];
64	expectedOutput?: string;
65	expectedBehavior?: string;
66	tags: string[];
67	difficulty: 'easy' \| 'medium' \| 'hard' \| 'adversarial';
68	category: string;
69	metadata?: Record<string, unknown>;
70	}
71
72	export interface ConversationTurn {
73	role: 'user' \| 'assistant' \| 'system';
74	content: string;
75	}
76
77	export interface EvalResult {
78	exampleId: string;
79	agentOutput: string;
80	scores: Record<string, number>;
81	judgeReasonings: Record<string, string>;
82	latencyMs: number;
83	tokenUsage: { input: number; output: number };
84	timestamp: string;
85	agentVersion: string;
86	error?: string;
87	}
88
89	export interface EvalSuiteResult {
90	suiteId: string;
91	agentVersion: string;
92	timestamp: string;
93	results: EvalResult[];
94	aggregateScores: Record<string, AggregateScore>;
95	totalExamples: number;
96	passedExamples: number;
97	failedExamples: number;
98	errorExamples: number;
99	}
100
101	export interface AggregateScore {
102	mean: number;
103	median: number;
104	stdDev: number;
105	min: number;
106	max: number;
107	p5: number;
108	p95: number;
109	confidenceInterval: { lower: number; upper: number };
110	sampleSize: number;
111	}
112	```
113
114	## LLM-as-Judge Implementation
115
116	```typescript
117	// evals/judges/correctness-judge.ts
118	import Anthropic from '@anthropic-ai/sdk';
119
120	export interface JudgeResult {
121	score: number; // 0-10 scale
122	reasoning: string;
123	confidence: number; // 0-1
124	flags: string[];
125	}
126
127	export interface JudgeConfig {
128	model: string;
129	temperature: number;
130	maxTokens: number;
131	systemPrompt: string;
132	scoringRubric: string;
133	}
134
135	const DEFAULT_CORRECTNESS_CONFIG: JudgeConfig = {
136	model: 'claude-sonnet-4-20250514',
137	temperature: 0,
138	maxTokens: 1024,
139	systemPrompt: `You are an expert evaluator assessing the correctness of AI

People who liked this also viewed

6 picks

Type