eval-harness

Name: eval-harness
Author: affaan-m

232kstars35kforks

How to install eval-harness?

affaan-m/ecc/eval-harness

$npx -y skills add affaan-m/ECC --skill eval-harness

Installs into the current project.

Summary

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles

Files · 1

View on GitHub

SKILL.md

1	# Eval Harness Skill
2
3	A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.
4
5	## When to Activate
6
7	- Setting up eval-driven development (EDD) for AI-assisted workflows
8	- Defining pass/fail criteria for Claude Code task completion
9	- Measuring agent reliability with pass@k metrics
10	- Creating regression test suites for prompt or agent changes
11	- Benchmarking agent performance across model versions
12
13	## Philosophy
14
15	Eval-Driven Development treats evals as the "unit tests of AI development":
16	- Define expected behavior BEFORE implementation
17	- Run evals continuously during development
18	- Track regressions with each change
19	- Use pass@k metrics for reliability measurement
20
21	## Eval Types
22
23	### Capability Evals
24	Test if Claude can do something it couldn't before:
25	```markdown
26	[CAPABILITY EVAL: feature-name]
27	Task: Description of what Claude should accomplish
28	Success Criteria:
29	- [ ] Criterion 1
30	- [ ] Criterion 2
31	- [ ] Criterion 3
32	Expected Output: Description of expected result
33	```
34
35	### Regression Evals
36	Ensure changes don't break existing functionality:
37	```markdown
38	[REGRESSION EVAL: feature-name]
39	Baseline: SHA or checkpoint name
40	Tests:
41	- existing-test-1: PASS/FAIL
42	- existing-test-2: PASS/FAIL
43	- existing-test-3: PASS/FAIL
44	Result: X/Y passed (previously Y/Y)
45	```
46
47	## Grader Types
48
49	### 1. Code-Based Grader
50	Deterministic checks using code:
51	```bash
52	# Check if file contains expected pattern
53	grep -q "export function handleAuth" src/auth.ts && echo "PASS" \|\| echo "FAIL"
54
55	# Check if tests pass
56	npm test -- --testPathPattern="auth" && echo "PASS" \|\| echo "FAIL"
57
58	# Check if build succeeds
59	npm run build && echo "PASS" \|\| echo "FAIL"
60	```
61
62	### 2. Model-Based Grader
63	Use Claude to evaluate open-ended outputs:
64	```markdown
65	[MODEL GRADER PROMPT]
66	Evaluate the following code change:
67	1. Does it solve the stated problem?
68	2. Is it well-structured?
69	3. Are edge cases handled?
70	4. Is error handling appropriate?
71
72	Score: 1-5 (1=poor, 5=excellent)
73	Reasoning: [explanation]
74	```
75
76	### 3. Human Grader
77	Flag for manual review:
78	```markdown
79	[HUMAN REVIEW REQUIRED]
80	Change: Description of what changed
81	Reason: Why human review is needed
82	Risk Level: LOW/MEDIUM/HIGH
83	```
84
85	## Metrics
86
87	### pass@k
88	"At least one success in k attempts"
89	- pass@1: First attempt success rate
90	- pass@3: Success within 3 attempts
91	- Typical target: pass@3 > 90%
92
93	### pass^k
94	"All k trials succeed"
95	- Higher bar for reliability
96	- pass^3: 3 consecutive successes
97	- Use for critical paths
98
99	## Eval Workflow
100
101	### 1. Define (Before Coding)
102	```markdown
103	## EVAL DEFINITION: feature-xyz
104
105	### Capability Evals
106	1. Can create new user account
107	2. Can validate email format
108	3. Can hash password securely
109
110	### Regression Evals
111	1. Existing login still works
112	2. Session management unchanged
113	3. Logout flow intact
114
115	### Success Metrics
116	- pass@3 > 90% for capability evals
117	- pass^3 = 100% for regression evals
118	```
119
120	### 2. Implement
121	Write code to pass the defined evals.
122
123	### 3. Evaluate
124	```bash
125	# Run capability evals
126	[Run each capability eval, record PASS/FAIL]
127
128	# Run regression evals
129	npm test -- --testPathPattern="existing"
130
131	# Generate report
132	```
133
134	### 4. Report
135	```markdown
136	EVAL REPORT: feature-xyz
137	========================
138
139	Capability Evals:
140	create-user: PASS (pass@1)
141	validate-email: PASS (pass@2)
142	hash-password: PASS (pass@1)
143	Overall: 3/3 passed
144
145	Regression Evals:
146	login-flow: PASS
147	session-mgmt: PASS
148	logout-flow: PASS
149	Overall: 3/3 passed
150
151	Metrics:
152	pass@1: 67% (2/3)
153	pass@3: 100% (3/3)
154
155	Status: READY FOR REVIEW
156	```
157
158	## Integration Patterns
159
160	### Pre-Implementation
161	```
162	/eval define feature-name
163	```
164	Creates eval definition file at `.claude/evals/feature-name.md`
165
166	### During Implementation
167	```
168	/eval check feature-name
169	```
170	Runs current evals and reports status
171
172	### Post-Implementation
173	```
174	/eval report feature-name
175	```
176	Generates full eval report
177
178	## Eval Storage
179
180	Store evals in project:
181	```
182	.claude/
183	evals/
184	feature-xyz.md # Eval definition
185	feature-xyz.log # Eval run history
186	baseline.json # Regression baselines
187	```
188
189	## Best Practices
190
191	1. Define evals BEFORE coding - Forces clear thinking about success criteria
192	2. Run evals frequently - Catch regressions early
193	3. Track pass@k over time - Monitor reliability trends
194	4. Use code graders when possible - Deterministic > probabilistic
195	5. Human review for security - Never fully automate security checks
196	6. Keep evals fast - Slow evals don't get run
197	7. Version evals with code - Evals are first-class artifacts
198
199	## Example: Adding Authentication
200
201	```markdown
202	## EVAL: add-authentication
203
204	### Phase 1: Define (10 min)
205	Capability Evals:
206	- [ ] User can register with email/password
207	- [ ] User can login with valid credentials
208	- [ ] Invalid cre

People who liked this also viewed

6 picks

Type

tddSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026501k181k
test-driven-developmentUse when implementing any feature or bugfix, before writing implementation codeSkillsJul 2026175k259k
qaSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026160k181k
verification-before-completionUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidenceSkillsJul 2026153k259k
webapp-testingToolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser lSkillsJul 2026120k163k
playwright-best-practicesUse when writing Playwright tests, fixing flaky tests, debugging failures, implementing Page Object Model, configuring CI/CD, optimizing performance, mocking APIs, handling authentication or OAuth, teSkillsJul 202664k338

eval-harness

232kstars35kforks

View on GitHub

How to install eval-harness?

affaan-m/ecc/eval-harness

$npx -y skills add affaan-m/ECC --skill eval-harness

Installs into the current project.

Summary

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles

Files · 1

View on GitHub

SKILL.md

1	# Eval Harness Skill
2
3	A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.
4
5	## When to Activate
6
7	- Setting up eval-driven development (EDD) for AI-assisted workflows
8	- Defining pass/fail criteria for Claude Code task completion
9	- Measuring agent reliability with pass@k metrics
10	- Creating regression test suites for prompt or agent changes
11	- Benchmarking agent performance across model versions
12
13	## Philosophy
14
15	Eval-Driven Development treats evals as the "unit tests of AI development":
16	- Define expected behavior BEFORE implementation
17	- Run evals continuously during development
18	- Track regressions with each change
19	- Use pass@k metrics for reliability measurement
20
21	## Eval Types
22
23	### Capability Evals
24	Test if Claude can do something it couldn't before:
25	```markdown
26	[CAPABILITY EVAL: feature-name]
27	Task: Description of what Claude should accomplish
28	Success Criteria:
29	- [ ] Criterion 1
30	- [ ] Criterion 2
31	- [ ] Criterion 3
32	Expected Output: Description of expected result
33	```
34
35	### Regression Evals
36	Ensure changes don't break existing functionality:
37	```markdown
38	[REGRESSION EVAL: feature-name]
39	Baseline: SHA or checkpoint name
40	Tests:
41	- existing-test-1: PASS/FAIL
42	- existing-test-2: PASS/FAIL
43	- existing-test-3: PASS/FAIL
44	Result: X/Y passed (previously Y/Y)
45	```
46
47	## Grader Types
48
49	### 1. Code-Based Grader
50	Deterministic checks using code:
51	```bash
52	# Check if file contains expected pattern
53	grep -q "export function handleAuth" src/auth.ts && echo "PASS" \|\| echo "FAIL"
54
55	# Check if tests pass
56	npm test -- --testPathPattern="auth" && echo "PASS" \|\| echo "FAIL"
57
58	# Check if build succeeds
59	npm run build && echo "PASS" \|\| echo "FAIL"
60	```
61
62	### 2. Model-Based Grader
63	Use Claude to evaluate open-ended outputs:
64	```markdown
65	[MODEL GRADER PROMPT]
66	Evaluate the following code change:
67	1. Does it solve the stated problem?
68	2. Is it well-structured?
69	3. Are edge cases handled?
70	4. Is error handling appropriate?
71
72	Score: 1-5 (1=poor, 5=excellent)
73	Reasoning: [explanation]
74	```
75
76	### 3. Human Grader
77	Flag for manual review:
78	```markdown
79	[HUMAN REVIEW REQUIRED]
80	Change: Description of what changed
81	Reason: Why human review is needed
82	Risk Level: LOW/MEDIUM/HIGH
83	```
84
85	## Metrics
86
87	### pass@k
88	"At least one success in k attempts"
89	- pass@1: First attempt success rate
90	- pass@3: Success within 3 attempts
91	- Typical target: pass@3 > 90%
92
93	### pass^k
94	"All k trials succeed"
95	- Higher bar for reliability
96	- pass^3: 3 consecutive successes
97	- Use for critical paths
98
99	## Eval Workflow
100
101	### 1. Define (Before Coding)
102	```markdown
103	## EVAL DEFINITION: feature-xyz
104
105	### Capability Evals
106	1. Can create new user account
107	2. Can validate email format
108	3. Can hash password securely
109
110	### Regression Evals
111	1. Existing login still works
112	2. Session management unchanged
113	3. Logout flow intact
114
115	### Success Metrics
116	- pass@3 > 90% for capability evals
117	- pass^3 = 100% for regression evals
118	```
119
120	### 2. Implement
121	Write code to pass the defined evals.
122
123	### 3. Evaluate
124	```bash
125	# Run capability evals
126	[Run each capability eval, record PASS/FAIL]
127
128	# Run regression evals
129	npm test -- --testPathPattern="existing"
130
131	# Generate report
132	```
133
134	### 4. Report
135	```markdown
136	EVAL REPORT: feature-xyz
137	========================
138
139	Capability Evals:
140	create-user: PASS (pass@1)
141	validate-email: PASS (pass@2)
142	hash-password: PASS (pass@1)
143	Overall: 3/3 passed
144
145	Regression Evals:
146	login-flow: PASS
147	session-mgmt: PASS
148	logout-flow: PASS
149	Overall: 3/3 passed
150
151	Metrics:
152	pass@1: 67% (2/3)
153	pass@3: 100% (3/3)
154
155	Status: READY FOR REVIEW
156	```
157
158	## Integration Patterns
159
160	### Pre-Implementation
161	```
162	/eval define feature-name
163	```
164	Creates eval definition file at `.claude/evals/feature-name.md`
165
166	### During Implementation
167	```
168	/eval check feature-name
169	```
170	Runs current evals and reports status
171
172	### Post-Implementation
173	```
174	/eval report feature-name
175	```
176	Generates full eval report
177
178	## Eval Storage
179
180	Store evals in project:
181	```
182	.claude/
183	evals/
184	feature-xyz.md # Eval definition
185	feature-xyz.log # Eval run history
186	baseline.json # Regression baselines
187	```
188
189	## Best Practices
190
191	1. Define evals BEFORE coding - Forces clear thinking about success criteria
192	2. Run evals frequently - Catch regressions early
193	3. Track pass@k over time - Monitor reliability trends
194	4. Use code graders when possible - Deterministic > probabilistic
195	5. Human review for security - Never fully automate security checks
196	6. Keep evals fast - Slow evals don't get run
197	7. Version evals with code - Evals are first-class artifacts
198
199	## Example: Adding Authentication
200
201	```markdown
202	## EVAL: add-authentication
203
204	### Phase 1: Define (10 min)
205	Capability Evals:
206	- [ ] User can register with email/password
207	- [ ] User can login with valid credentials
208	- [ ] Invalid cre

People who liked this also viewed

6 picks

Type