agent-evaluation

Name: agent-evaluation
Author: sickn33

44kstars6.5kforks

How to install agent-evaluation?

sickn33/agentic-awesome-skills/agent-evaluation

$npx -y skills add sickn33/agentic-awesome-skills --skill agent-evaluation

Installs into the current project.

Summary

Testing and benchmarking LLM agents including behavioral testing,

Files · 1

View on GitHub

SKILL.md

1	# Agent Evaluation
2
3	Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks
4
5	## Capabilities
6
7	- agent-testing
8	- benchmark-design
9	- capability-assessment
10	- reliability-metrics
11	- regression-testing
12
13	## Prerequisites
14
15	- Knowledge: Testing methodologies, Statistical analysis basics, LLM behavior patterns
16	- Skills_recommended: autonomous-agents, multi-agent-orchestration
17	- Required skills: testing-fundamentals, llm-fundamentals
18
19	## Scope
20
21	- Does_not_cover: Model training evaluation (loss, perplexity), Fairness and bias testing, User experience testing
22	- Boundaries: Focus is agent capability and reliability, Covers functional and behavioral testing
23
24	## Ecosystem
25
26	### Primary_tools
27
28	- AgentBench - Multi-environment benchmark for LLM agents (ICLR 2024)
29	- τ-bench (Tau-bench) - Sierra's real-world agent benchmark
30	- ToolEmu - Risky behavior detection for agent tool use
31	- Langsmith - LLM tracing and evaluation platform
32
33	### Alternatives
34
35	- Braintrust - When: Need production monitoring integration LLM evaluation and monitoring
36	- PromptFoo - When: Focus on prompt-level evaluation Prompt testing framework
37
38	### Deprecated
39
40	- Manual testing only
41
42	## Patterns
43
44	### Statistical Test Evaluation
45
46	Run tests multiple times and analyze result distributions
47
48	When to use: Evaluating stochastic agent behavior
49
50	interface TestResult {
51	testId: string;
52	runId: string;
53	passed: boolean;
54	score: number; // 0-1 for partial credit
55	latencyMs: number;
56	tokensUsed: number;
57	output: string;
58	expectedBehaviors: string[];
59	actualBehaviors: string[];
60	}
61
62	interface StatisticalAnalysis {
63	passRate: number;
64	confidence95: [number, number];
65	meanScore: number;
66	stdDevScore: number;
67	meanLatency: number;
68	p95Latency: number;
69	behaviorConsistency: number;
70	}
71
72	class StatisticalEvaluator {
73	private readonly minRuns = 10;
74	private readonly confidenceLevel = 0.95;
75
76	async evaluateAgent(
77	agent: Agent,
78	testSuite: TestCase[]
79	): Promise<EvaluationReport> {
80	const results: TestResult[] = [];
81
82	// Run each test multiple times
83	for (const test of testSuite) {
84	for (let run = 0; run < this.minRuns; run++) {
85	const result = await this.runTest(agent, test, run);
86	results.push(result);
87	}
88	}
89
90	// Analyze by test
91	const byTest = this.groupByTest(results);
92	const testAnalyses = new Map<string, StatisticalAnalysis>();
93
94	for (const [testId, testResults] of byTest) {
95	testAnalyses.set(testId, this.analyzeResults(testResults));
96	}
97
98	// Overall analysis
99	const overall = this.analyzeResults(results);
100
101	return {
102	overall,
103	byTest: testAnalyses,
104	concerns: this.identifyConcerns(testAnalyses),
105	recommendations: this.generateRecommendations(testAnalyses)
106	};
107	}
108
109	private analyzeResults(results: TestResult[]): StatisticalAnalysis {
110	const passes = results.filter(r => r.passed);
111	const passRate = passes.length / results.length;
112
113	// Calculate confidence interval for pass rate
114	const z = 1.96; // 95% confidence
115	const se = Math.sqrt((passRate * (1 - passRate)) / results.length);
116	const confidence95: [number, number] = [
117	Math.max(0, passRate - z * se),
118	Math.min(1, passRate + z * se)
119	];
120
121	const scores = results.map(r => r.score);
122	const latencies = results.map(r => r.latencyMs);
123
124	return {
125	passRate,
126	confidence95,
127	meanScore: this.mean(scores),
128	stdDevScore: this.stdDev(scores),
129	meanLatency: this.mean(latencies),
130	p95Latency: this.percentile(latencies, 95),
131	behaviorConsistency: this.calculateConsistency(results)
132	};
133	}
134
135	private calculateConsistency(results: TestResult[]): number {
136	// How consistent are the behaviors across runs?
137	if (results.length < 2) return 1;
138
139	const behaviorSets = results.map(r => new Set(r.actualBehaviors));
140	let consistencySum = 0;
141	let comparisons = 0;
142
143	for (let i = 0; i < behaviorSets.length; i++) {
144	for (let j = i + 1; j < behaviorSets.length; j++) {
145	const intersection = new Set(
146	[...behaviorSets[i]].filter(x => behaviorSets[j].has(x))
147	);
148	const union = new Set([...behaviorSets[i], ...behaviorSets[j]]

People who liked this also viewed

6 picks

Type

microsoft-foundryDeploy, evaluate, fine-tune, and manage Foundry agents end-to-end with azd: hosted agent scaffold/run/deploy, prompt agent create, batch eval, continuous eval, prompt optimizer, Agent Optimizer scaffoSkillsJul 2026476k1.3k
azure-deployExecute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application SkillsJul 2026472k1.3k
azure-preparePrepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wSkillsJul 2026471k1.3k
azure-validate--- name: azure-validate description: "Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity pSkillsJul 2026471k1.3k
azure-aigatewayConfigure Azure API Management as an AI Gateway for AI models, MCP tools, and agents. WHEN: semantic caching, token limit, content safety, load balancing, AI model governance, MCP rate limiting, jailbSkillsJul 2026470k1.3k
azure-hosted-copilot-sdkOfficial agent plugin providing skills and MCP server configurations for Azure scenarios.SkillsJul 2026424k1.3k

agent-evaluation

44kstars6.5kforks

View on GitHub

How to install agent-evaluation?

sickn33/agentic-awesome-skills/agent-evaluation

$npx -y skills add sickn33/agentic-awesome-skills --skill agent-evaluation

Installs into the current project.

Summary

Testing and benchmarking LLM agents including behavioral testing,

Files · 1

View on GitHub

SKILL.md

1	# Agent Evaluation
2
3	Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks
4
5	## Capabilities
6
7	- agent-testing
8	- benchmark-design
9	- capability-assessment
10	- reliability-metrics
11	- regression-testing
12
13	## Prerequisites
14
15	- Knowledge: Testing methodologies, Statistical analysis basics, LLM behavior patterns
16	- Skills_recommended: autonomous-agents, multi-agent-orchestration
17	- Required skills: testing-fundamentals, llm-fundamentals
18
19	## Scope
20
21	- Does_not_cover: Model training evaluation (loss, perplexity), Fairness and bias testing, User experience testing
22	- Boundaries: Focus is agent capability and reliability, Covers functional and behavioral testing
23
24	## Ecosystem
25
26	### Primary_tools
27
28	- AgentBench - Multi-environment benchmark for LLM agents (ICLR 2024)
29	- τ-bench (Tau-bench) - Sierra's real-world agent benchmark
30	- ToolEmu - Risky behavior detection for agent tool use
31	- Langsmith - LLM tracing and evaluation platform
32
33	### Alternatives
34
35	- Braintrust - When: Need production monitoring integration LLM evaluation and monitoring
36	- PromptFoo - When: Focus on prompt-level evaluation Prompt testing framework
37
38	### Deprecated
39
40	- Manual testing only
41
42	## Patterns
43
44	### Statistical Test Evaluation
45
46	Run tests multiple times and analyze result distributions
47
48	When to use: Evaluating stochastic agent behavior
49
50	interface TestResult {
51	testId: string;
52	runId: string;
53	passed: boolean;
54	score: number; // 0-1 for partial credit
55	latencyMs: number;
56	tokensUsed: number;
57	output: string;
58	expectedBehaviors: string[];
59	actualBehaviors: string[];
60	}
61
62	interface StatisticalAnalysis {
63	passRate: number;
64	confidence95: [number, number];
65	meanScore: number;
66	stdDevScore: number;
67	meanLatency: number;
68	p95Latency: number;
69	behaviorConsistency: number;
70	}
71
72	class StatisticalEvaluator {
73	private readonly minRuns = 10;
74	private readonly confidenceLevel = 0.95;
75
76	async evaluateAgent(
77	agent: Agent,
78	testSuite: TestCase[]
79	): Promise<EvaluationReport> {
80	const results: TestResult[] = [];
81
82	// Run each test multiple times
83	for (const test of testSuite) {
84	for (let run = 0; run < this.minRuns; run++) {
85	const result = await this.runTest(agent, test, run);
86	results.push(result);
87	}
88	}
89
90	// Analyze by test
91	const byTest = this.groupByTest(results);
92	const testAnalyses = new Map<string, StatisticalAnalysis>();
93
94	for (const [testId, testResults] of byTest) {
95	testAnalyses.set(testId, this.analyzeResults(testResults));
96	}
97
98	// Overall analysis
99	const overall = this.analyzeResults(results);
100
101	return {
102	overall,
103	byTest: testAnalyses,
104	concerns: this.identifyConcerns(testAnalyses),
105	recommendations: this.generateRecommendations(testAnalyses)
106	};
107	}
108
109	private analyzeResults(results: TestResult[]): StatisticalAnalysis {
110	const passes = results.filter(r => r.passed);
111	const passRate = passes.length / results.length;
112
113	// Calculate confidence interval for pass rate
114	const z = 1.96; // 95% confidence
115	const se = Math.sqrt((passRate * (1 - passRate)) / results.length);
116	const confidence95: [number, number] = [
117	Math.max(0, passRate - z * se),
118	Math.min(1, passRate + z * se)
119	];
120
121	const scores = results.map(r => r.score);
122	const latencies = results.map(r => r.latencyMs);
123
124	return {
125	passRate,
126	confidence95,
127	meanScore: this.mean(scores),
128	stdDevScore: this.stdDev(scores),
129	meanLatency: this.mean(latencies),
130	p95Latency: this.percentile(latencies, 95),
131	behaviorConsistency: this.calculateConsistency(results)
132	};
133	}
134
135	private calculateConsistency(results: TestResult[]): number {
136	// How consistent are the behaviors across runs?
137	if (results.length < 2) return 1;
138
139	const behaviorSets = results.map(r => new Set(r.actualBehaviors));
140	let consistencySum = 0;
141	let comparisons = 0;
142
143	for (let i = 0; i < behaviorSets.length; i++) {
144	for (let j = i + 1; j < behaviorSets.length; j++) {
145	const intersection = new Set(
146	[...behaviorSets[i]].filter(x => behaviorSets[j].has(x))
147	);
148	const union = new Set([...behaviorSets[i], ...behaviorSets[j]]

People who liked this also viewed

6 picks

Type