agent-benchmark

Name: agent-benchmark
Author: vibeeval

519stars44forks

How to install agent-benchmark?

vibeeval/vibecosystem/agent-benchmark

$npx -y skills add vibeeval/vibecosystem --skill agent-benchmark

Installs into the current project.

Summary

Framework for measuring and tracking agent response quality over time. Detects regressions before they reach production. Use when evaluating agent changes, auditing quality, or establishing performance baselines.

Files · 1

View on GitHub

SKILL.md

1	# Agent Benchmark Framework
2
3	Without benchmarks, we cannot know whether agent changes improve or degrade quality. This skill defines how to measure, track, and protect agent performance.
4
5	## When to Activate
6
7	- Before and after modifying any agent definition file
8	- When adding a new skill that an agent depends on
9	- Periodic quality audits (weekly/monthly)
10	- When a user reports degraded agent output
11	- Before promoting an agent from experimental to production
12
13	## Core Concepts
14
15	### Why Benchmarks Matter
16
17	Agent quality degrades silently. A prompt tweak that improves one response can break ten others. Without a baseline to compare against, every change is a guess. Benchmarks make quality visible and regressions detectable.
18
19	### Benchmark Types
20
21	\| Type \| Scope \| Cost \| Frequency \|
22	\|------\|-------\|------\|-----------\|
23	\| Prompt Benchmark \| Single agent, single task \| Low \| Every agent change \|
24	\| Task Benchmark \| End-to-end scenario \| Medium \| Feature changes \|
25	\| Regression Suite \| All critical agents \| High \| Weekly / before release \|
26
27	## Directory Structure
28
29	```
30	~/.claude/benchmarks/
31	fixtures/
32	code-reviewer/
33	missing-error-handling.ts # Input: code with no try/catch
34	sql-injection.py # Input: unparameterized query
35	clean-code.ts # Input: code with no issues
36	security-reviewer/
37	hardcoded-secret.ts # Input: API key in source
38	parameterized-query.py # Input: safe query (no findings expected)
39	verifier/
40	passing-build/ # Input: project that builds
41	failing-types/ # Input: project with type errors
42	ground-truth/
43	code-reviewer/
44	missing-error-handling.json # Expected findings
45	sql-injection.json # Expected findings
46	clean-code.json # Expected: empty findings
47	security-reviewer/
48	hardcoded-secret.json
49	parameterized-query.json
50	rubrics/
51	code-reviewer.md # Scoring rubric
52	security-reviewer.md
53	verifier.md
54	baselines/
55	code-reviewer-2026-03-01.json # Timestamped baseline scores
56	code-reviewer-2026-03-26.json
57	security-reviewer-2026-03-26.json
58	results/
59	run-2026-03-26T14-00.json # Latest run output
60	```
61
62	## Scoring Rubric Template
63
64	Each agent has its own rubric file. The template:
65
66	```markdown
67	## [Agent Name] Scoring Rubric
68
69	### Completeness (0-30 points)
70	Did the agent find everything it should have found?
71
72	- Found all expected issues: 30
73	- Missed 1 non-critical issue: 22
74	- Missed 1 critical issue: 10
75	- Missed 2+ issues: 5
76	- Found nothing when issues exist: 0
77
78	### Accuracy (0-30 points)
79	Were the findings correct? No false positives?
80
81	- All findings verified correct: 30
82	- 1 false positive: 22
83	- 2 false positives: 12
84	- 3+ false positives: 5
85	- Majority of findings are wrong: 0
86
87	### Actionability (0-20 points)
88	Did the agent give concrete, implementable fixes?
89
90	- Clear fix with file/line reference: 20
91	- Clear fix without location: 14
92	- Vague suggestion (refactor this): 7
93	- No fix suggested: 0
94
95	### Format Compliance (0-20 points)
96	Did the output follow the agent's output contract?
97
98	- Matches contract exactly (VERDICT + sections): 20
99	- Minor deviation (missing one section): 12
100	- Major deviation (no VERDICT): 5
101	- Unstructured free text: 0
102	```
103
104	## Ground Truth Format
105
106	Ground truth files define what a correct agent response must contain:
107
108	```json
109	{
110	"fixture": "missing-error-handling.ts",
111	"agent": "code-reviewer",
112	"required_findings": [
113	{
114	"id": "missing-try-catch",
115	"severity": "HIGH",
116	"description_contains": ["error handling", "try", "catch"],
117	"location_hint": "fetchUserData"
118	}
119	],
120	"forbidden_findings": [],
121	"required_verdict": "FAIL",
122	"min_score": 70
123	}
124	```
125
126	## Scoring Logic
127
128	### How a Run Is Scored
129
130	```
131	1. Load fixture (input code / task)
132	2. Run agent with fixture as input
133	3. Parse agent output
134	4. Check required_findings: each found = +completeness points
135	5. Check forbidden_findings: each false positive = -accuracy points
136	6. Check verdict matches required_verdict
137	7. Check format follows output contract
138	8. Sum scores → final 0-100
139	9. Compare against min_score threshold
140	```
141
142	### Score Interpretation
143
144	\| Score \| Status \| Action \|
145	\|-------\|--------\|--------\|
146	\| 90-100 \| EXCELLENT \| No action needed \|
147	\| 75-89 \| GOOD \| Minor tuning optional \|
148	\| 60-74 \| WARN \| Investigate degradation \|
149	\| 40-59 \| POOR \| Agent needs rework \|
150	\| 0-39 \| CRITICAL \| Block deployment \|
151
152	## Running Benchmarks
153
154	### Run All Benchmarks
155
156	```bash
157	# Full suite
158	node ~/.claude/benchmarks/run.mjs
159
160	# Output: results/run-{timestamp}.json
161	```
162
163	### Run Single Agent
164
165	```bash
166	# Benchmark one agent
167	node ~/.claude/benchmarks/run.mjs --agent code-

People who liked this also viewed

6 picks

Type

agent-benchmark

519stars44forks

View on GitHub

How to install agent-benchmark?

vibeeval/vibecosystem/agent-benchmark

$npx -y skills add vibeeval/vibecosystem --skill agent-benchmark

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Agent Benchmark Framework
2
3	Without benchmarks, we cannot know whether agent changes improve or degrade quality. This skill defines how to measure, track, and protect agent performance.
4
5	## When to Activate
6
7	- Before and after modifying any agent definition file
8	- When adding a new skill that an agent depends on
9	- Periodic quality audits (weekly/monthly)
10	- When a user reports degraded agent output
11	- Before promoting an agent from experimental to production
12
13	## Core Concepts
14
15	### Why Benchmarks Matter
16
17	Agent quality degrades silently. A prompt tweak that improves one response can break ten others. Without a baseline to compare against, every change is a guess. Benchmarks make quality visible and regressions detectable.
18
19	### Benchmark Types
20
21	\| Type \| Scope \| Cost \| Frequency \|
22	\|------\|-------\|------\|-----------\|
23	\| Prompt Benchmark \| Single agent, single task \| Low \| Every agent change \|
24	\| Task Benchmark \| End-to-end scenario \| Medium \| Feature changes \|
25	\| Regression Suite \| All critical agents \| High \| Weekly / before release \|
26
27	## Directory Structure
28
29	```
30	~/.claude/benchmarks/
31	fixtures/
32	code-reviewer/
33	missing-error-handling.ts # Input: code with no try/catch
34	sql-injection.py # Input: unparameterized query
35	clean-code.ts # Input: code with no issues
36	security-reviewer/
37	hardcoded-secret.ts # Input: API key in source
38	parameterized-query.py # Input: safe query (no findings expected)
39	verifier/
40	passing-build/ # Input: project that builds
41	failing-types/ # Input: project with type errors
42	ground-truth/
43	code-reviewer/
44	missing-error-handling.json # Expected findings
45	sql-injection.json # Expected findings
46	clean-code.json # Expected: empty findings
47	security-reviewer/
48	hardcoded-secret.json
49	parameterized-query.json
50	rubrics/
51	code-reviewer.md # Scoring rubric
52	security-reviewer.md
53	verifier.md
54	baselines/
55	code-reviewer-2026-03-01.json # Timestamped baseline scores
56	code-reviewer-2026-03-26.json
57	security-reviewer-2026-03-26.json
58	results/
59	run-2026-03-26T14-00.json # Latest run output
60	```
61
62	## Scoring Rubric Template
63
64	Each agent has its own rubric file. The template:
65
66	```markdown
67	## [Agent Name] Scoring Rubric
68
69	### Completeness (0-30 points)
70	Did the agent find everything it should have found?
71
72	- Found all expected issues: 30
73	- Missed 1 non-critical issue: 22
74	- Missed 1 critical issue: 10
75	- Missed 2+ issues: 5
76	- Found nothing when issues exist: 0
77
78	### Accuracy (0-30 points)
79	Were the findings correct? No false positives?
80
81	- All findings verified correct: 30
82	- 1 false positive: 22
83	- 2 false positives: 12
84	- 3+ false positives: 5
85	- Majority of findings are wrong: 0
86
87	### Actionability (0-20 points)
88	Did the agent give concrete, implementable fixes?
89
90	- Clear fix with file/line reference: 20
91	- Clear fix without location: 14
92	- Vague suggestion (refactor this): 7
93	- No fix suggested: 0
94
95	### Format Compliance (0-20 points)
96	Did the output follow the agent's output contract?
97
98	- Matches contract exactly (VERDICT + sections): 20
99	- Minor deviation (missing one section): 12
100	- Major deviation (no VERDICT): 5
101	- Unstructured free text: 0
102	```
103
104	## Ground Truth Format
105
106	Ground truth files define what a correct agent response must contain:
107
108	```json
109	{
110	"fixture": "missing-error-handling.ts",
111	"agent": "code-reviewer",
112	"required_findings": [
113	{
114	"id": "missing-try-catch",
115	"severity": "HIGH",
116	"description_contains": ["error handling", "try", "catch"],
117	"location_hint": "fetchUserData"
118	}
119	],
120	"forbidden_findings": [],
121	"required_verdict": "FAIL",
122	"min_score": 70
123	}
124	```
125
126	## Scoring Logic
127
128	### How a Run Is Scored
129
130	```
131	1. Load fixture (input code / task)
132	2. Run agent with fixture as input
133	3. Parse agent output
134	4. Check required_findings: each found = +completeness points
135	5. Check forbidden_findings: each false positive = -accuracy points
136	6. Check verdict matches required_verdict
137	7. Check format follows output contract
138	8. Sum scores → final 0-100
139	9. Compare against min_score threshold
140	```
141
142	### Score Interpretation
143
144	\| Score \| Status \| Action \|
145	\|-------\|--------\|--------\|
146	\| 90-100 \| EXCELLENT \| No action needed \|
147	\| 75-89 \| GOOD \| Minor tuning optional \|
148	\| 60-74 \| WARN \| Investigate degradation \|
149	\| 40-59 \| POOR \| Agent needs rework \|
150	\| 0-39 \| CRITICAL \| Block deployment \|
151
152	## Running Benchmarks
153
154	### Run All Benchmarks
155
156	```bash
157	# Full suite
158	node ~/.claude/benchmarks/run.mjs
159
160	# Output: results/run-{timestamp}.json
161	```
162
163	### Run Single Agent
164
165	```bash
166	# Benchmark one agent
167	node ~/.claude/benchmarks/run.mjs --agent code-

People who liked this also viewed

6 picks

Type