eval-harness

Name: eval-harness
Author: Archive228

712stars129forks

How to install eval-harness?

archive228/loopkit/eval-harness

$npx -y skills add Archive228/loopkit --skill eval-harness

Installs into the current project.

Summary

Build a repeatable eval loop that grades agent output with an LLM judge, so prompt/skill changes get scored against a baseline instead of eyeballed. Reuses loopkit's verifier subagent as the grader — do not build a new one.

Files · 1

View on GitHub

SKILL.md

1	# Eval Harness
2
3	Every prompt tweak in a long-running agent looks like an improvement in the moment. The only way to know is a graded run against fixed inputs. Loopkit already ships `.claude/agents/verifier.md` — that is your grader. Do not rebuild it.
4
5	## The three-stage loop
6
7	```
8	inputs.jsonl → runner → outputs.jsonl → verifier (per row) → verdicts.jsonl → diff vs baseline
9	```
10
11	Each stage writes to disk. No stage holds the whole run in context.
12
13	## Stage 1 — inputs.jsonl
14
15	One JSON object per row: `{"id": "case-01", "input": "...", "expected": "..."}`.
16
17	- 20-100 cases is enough for a signal. More is nice, not required.
18	- Include known-hard cases, edge cases, and a couple of trivial ones as sanity anchors.
19	- Freeze the file. Rev the eval with a suffix (`inputs-v2.jsonl`) when you change it. Never edit in place — you lose the baseline.
20
21	## Stage 2 — runner
22
23	A dumb loop: for each row, call the model with the current prompt/skill, capture output, write `{"id": ..., "output": ...}` to `outputs.jsonl`. No grading here — just capture.
24
25	- Same temperature every run (usually 0 for evals).
26	- Same seed / model version.
27	- Log the git SHA of the prompt/skill under test in the file header.
28
29	If the runner is smart it will bias the eval. Keep it dumb.
30
31	## Stage 3 — verifier
32
33	Fan out one subagent per row (see `subagent-fanout`). Each gets:
34	- The input.
35	- The expected output (or spec).
36	- The actual output.
37	- The verifier system prompt from `.claude/agents/verifier.md`.
38
39	Verifier returns strict JSON: `{"pass": bool, "why": "..."}`. Collect into `verdicts.jsonl`.
40
41	## Diff vs baseline
42
43	Two runs of the same eval on two prompt versions → compare pass rates per case. What matters:
44
45	- Overall pass rate — the headline.
46	- Regressions — cases that were green and went red. These block ship.
47	- New passes — cases that were red and went green. These justify ship.
48	- Flappy cases — inconsistent across reruns. Investigate; may be genuine model nondeterminism or a bad case.
49
50	A change that raises the mean but adds regressions is usually a loss — the new failures are cases you already knew worked.
51
52	## Red flags
53
54	- Grader is the same model that produced the output, with the same prompt. Self-grading is lenient. Use a different persona at minimum; ideally a different model tier.
55	- Eval passes 100% on day one. The cases are too easy, or the grader is a rubber stamp. Add adversarial cases.
56	- Eval takes >30 minutes. Fan out. A serial 100-case eval is a serial 100-case bottleneck.
57	- Grader sees your prompt under test. It will grade what you wanted, not what happened. Feed it only spec + input + output.
58	- Baseline lost. Without baseline, "improvement" is vibes. Commit `verdicts.jsonl` to git.
59
60	## When NOT to do this
61
62	- One-off script — build a checklist, not a harness.
63	- Prompt that changes daily and won't stabilize — evals need a fixed target.
64	- Task where "correct" isn't checkable (open-ended creative writing) — use human eval or a rubric-based grader, not pass/fail.
65
66	The verifier is already yours. The harness is 100 lines of glue around it.

People who liked this also viewed

6 picks

Type

eval-harness

712stars129forks

View on GitHub

How to install eval-harness?

archive228/loopkit/eval-harness

$npx -y skills add Archive228/loopkit --skill eval-harness

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Eval Harness
2
3	Every prompt tweak in a long-running agent looks like an improvement in the moment. The only way to know is a graded run against fixed inputs. Loopkit already ships `.claude/agents/verifier.md` — that is your grader. Do not rebuild it.
4
5	## The three-stage loop
6
7	```
8	inputs.jsonl → runner → outputs.jsonl → verifier (per row) → verdicts.jsonl → diff vs baseline
9	```
10
11	Each stage writes to disk. No stage holds the whole run in context.
12
13	## Stage 1 — inputs.jsonl
14
15	One JSON object per row: `{"id": "case-01", "input": "...", "expected": "..."}`.
16
17	- 20-100 cases is enough for a signal. More is nice, not required.
18	- Include known-hard cases, edge cases, and a couple of trivial ones as sanity anchors.
19	- Freeze the file. Rev the eval with a suffix (`inputs-v2.jsonl`) when you change it. Never edit in place — you lose the baseline.
20
21	## Stage 2 — runner
22
23	A dumb loop: for each row, call the model with the current prompt/skill, capture output, write `{"id": ..., "output": ...}` to `outputs.jsonl`. No grading here — just capture.
24
25	- Same temperature every run (usually 0 for evals).
26	- Same seed / model version.
27	- Log the git SHA of the prompt/skill under test in the file header.
28
29	If the runner is smart it will bias the eval. Keep it dumb.
30
31	## Stage 3 — verifier
32
33	Fan out one subagent per row (see `subagent-fanout`). Each gets:
34	- The input.
35	- The expected output (or spec).
36	- The actual output.
37	- The verifier system prompt from `.claude/agents/verifier.md`.
38
39	Verifier returns strict JSON: `{"pass": bool, "why": "..."}`. Collect into `verdicts.jsonl`.
40
41	## Diff vs baseline
42
43	Two runs of the same eval on two prompt versions → compare pass rates per case. What matters:
44
45	- Overall pass rate — the headline.
46	- Regressions — cases that were green and went red. These block ship.
47	- New passes — cases that were red and went green. These justify ship.
48	- Flappy cases — inconsistent across reruns. Investigate; may be genuine model nondeterminism or a bad case.
49
50	A change that raises the mean but adds regressions is usually a loss — the new failures are cases you already knew worked.
51
52	## Red flags
53
54	- Grader is the same model that produced the output, with the same prompt. Self-grading is lenient. Use a different persona at minimum; ideally a different model tier.
55	- Eval passes 100% on day one. The cases are too easy, or the grader is a rubber stamp. Add adversarial cases.
56	- Eval takes >30 minutes. Fan out. A serial 100-case eval is a serial 100-case bottleneck.
57	- Grader sees your prompt under test. It will grade what you wanted, not what happened. Feed it only spec + input + output.
58	- Baseline lost. Without baseline, "improvement" is vibes. Commit `verdicts.jsonl` to git.
59
60	## When NOT to do this
61
62	- One-off script — build a checklist, not a harness.
63	- Prompt that changes daily and won't stabilize — evals need a fixed target.
64	- Task where "correct" isn't checkable (open-ended creative writing) — use human eval or a rubric-based grader, not pass/fail.
65
66	The verifier is already yours. The harness is 100 lines of glue around it.

People who liked this also viewed

6 picks

Type