meta-harness

Name: meta-harness
Author: 001TMF

72stars6forks

How to install meta-harness?

001tmf/harness-forge/meta-harness

$npx -y skills add 001TMF/harness-forge --skill meta-harness

Installs into the current project.

Summary

Run a Meta-Harness-style optimization loop NATIVELY — automatically search over the scaffolding around a FIXED base model (memory, retrieval, context construction, prompt templates, summarization, tool-selection logic) by proposing candidate variants, scoring each on a cheap dete

Files · 1

View on GitHub

SKILL.md

1	# Meta-Harness (native)
2
3	## What this is
4
5	*Meta-Harness optimizes the harness, not the model.* The harness is the code around a
6	fixed base model that decides what to store, retrieve, compress, and show while the model
7	works. You hold the model frozen and search over that scaffolding: propose candidate variants,
8	score each on a cheap deterministic eval, keep a Pareto frontier (quality up, cost down),
9	and iterate. The proposer is an LLM agent writing code; the inner loop is a cheap scorer.
10
11	The Stanford repo (`stanford-iris-lab/meta-harness`) ships a Python driver —
12	`claude_wrapper.py` (~720 lines) + `meta_harness.py` (~540 lines) — that **reimplements an
13	agent runtime to drive a headless Claude**: spawn a session, parse stream-json, track tool
14	calls, log everything, loop. You already are that runtime. So you run the same loop with
15	native tools (`Agent`, `Workflow`, `/loop`) and keep only the irreducible domain logic — a $0
16	scorer. The orchestration was never the hard part; your harness provides it.
17
18	This skill is the method, reusable for any harness-optimization task. A fully worked
19	example (optimizing proteus's campaign-memory summarizer) lives at `~/mh-proteus/` and is
20	walked through in `references/proteus-example.md`.
21
22	## When to use this
23
24	Strong fit when several of these hold (full criteria in `references/method.md`):
25
26	- The base model is fixed and the opportunity is better retrieval / memory / context /
27	prompting / tool scaffolding. (This is the whole premise — if the gain must come from the
28	model weights, this is the wrong tool: do RL/fine-tuning instead.)
29	- There are repeated episodes / tasks, not a one-off.
30	- There is a cheap, deterministic eval with a real success signal — or you can build one.
31	- The search set is large enough to expose failure modes, small enough to iterate.
32	- There are recurring error patterns a harness could fix systematically.
33
34	Poor fit: no stable eval loop, or purely subjective quality with no measurable criterion.
35
36	## The loop (mental model)
37
38	```
39	seed frontier with the incumbent harness (the thing to beat)
40	repeat until budget/convergence:
41	PROPOSE k candidate harness variants (proposer agents write code)
42	VALIDATE each imports / type-checks (cheap reject of broken candidates)
43	SCORE each on the held-out-protected eval set ($0 deterministic scorer)
44	FRONTIER Pareto-merge (quality up, cost down), floor-respecting
45	FINAL: score the frontier once on the untouched TEST split
46	```
47
48	The proposer is the mutation+crossover operator. The frontier is the persistent search memory.
49	The held-out test split is touched exactly once, at the end — never during the search.
50
51	## The five things YOU supply (everything else is native)
52
53	The orchestration is native; the domain is yours. Build these five — templates in `assets/`,
54	how-to in `references/building-blocks.md`:
55
56	1. Candidate interface — one clean, swappable boundary (an ABC / Protocol). A candidate is a
57	drop-in implementation. If your harness logic is tangled into one big function, extract the
58	boundary first. → `assets/candidate_base-template.py`
59	2. A $0 deterministic scorer + rubric — the inner loop. It must vary with the candidate
60	(see the frozen-replay trap below) and run with no LLM / no network so you can call it
61	hundreds of times for free. → `assets/scorer-template.py`
62	3. An eval corpus with a held-out split — the tasks/records candidates are graded on, split
63	so the test set shares no leaky structure with the search set.
64	4. A proposer prior — a short mini-SKILL the proposer agents load that steers them toward
65	mechanism-level changes (not constant-tuning) and enforces anti-leakage.
66	→ `assets/proposer-prior-template.md`
67	5. A frontier + run log — the state carried across iterations (a JSON

People who liked this also viewed

6 picks

Type

meta-harness

72stars6forks

View on GitHub

How to install meta-harness?

001tmf/harness-forge/meta-harness

$npx -y skills add 001TMF/harness-forge --skill meta-harness

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Meta-Harness (native)
2
3	## What this is
4
5	*Meta-Harness optimizes the harness, not the model.* The harness is the code around a
6	fixed base model that decides what to store, retrieve, compress, and show while the model
7	works. You hold the model frozen and search over that scaffolding: propose candidate variants,
8	score each on a cheap deterministic eval, keep a Pareto frontier (quality up, cost down),
9	and iterate. The proposer is an LLM agent writing code; the inner loop is a cheap scorer.
10
11	The Stanford repo (`stanford-iris-lab/meta-harness`) ships a Python driver —
12	`claude_wrapper.py` (~720 lines) + `meta_harness.py` (~540 lines) — that **reimplements an
13	agent runtime to drive a headless Claude**: spawn a session, parse stream-json, track tool
14	calls, log everything, loop. You already are that runtime. So you run the same loop with
15	native tools (`Agent`, `Workflow`, `/loop`) and keep only the irreducible domain logic — a $0
16	scorer. The orchestration was never the hard part; your harness provides it.
17
18	This skill is the method, reusable for any harness-optimization task. A fully worked
19	example (optimizing proteus's campaign-memory summarizer) lives at `~/mh-proteus/` and is
20	walked through in `references/proteus-example.md`.
21
22	## When to use this
23
24	Strong fit when several of these hold (full criteria in `references/method.md`):
25
26	- The base model is fixed and the opportunity is better retrieval / memory / context /
27	prompting / tool scaffolding. (This is the whole premise — if the gain must come from the
28	model weights, this is the wrong tool: do RL/fine-tuning instead.)
29	- There are repeated episodes / tasks, not a one-off.
30	- There is a cheap, deterministic eval with a real success signal — or you can build one.
31	- The search set is large enough to expose failure modes, small enough to iterate.
32	- There are recurring error patterns a harness could fix systematically.
33
34	Poor fit: no stable eval loop, or purely subjective quality with no measurable criterion.
35
36	## The loop (mental model)
37
38	```
39	seed frontier with the incumbent harness (the thing to beat)
40	repeat until budget/convergence:
41	PROPOSE k candidate harness variants (proposer agents write code)
42	VALIDATE each imports / type-checks (cheap reject of broken candidates)
43	SCORE each on the held-out-protected eval set ($0 deterministic scorer)
44	FRONTIER Pareto-merge (quality up, cost down), floor-respecting
45	FINAL: score the frontier once on the untouched TEST split
46	```
47
48	The proposer is the mutation+crossover operator. The frontier is the persistent search memory.
49	The held-out test split is touched exactly once, at the end — never during the search.
50
51	## The five things YOU supply (everything else is native)
52
53	The orchestration is native; the domain is yours. Build these five — templates in `assets/`,
54	how-to in `references/building-blocks.md`:
55
56	1. Candidate interface — one clean, swappable boundary (an ABC / Protocol). A candidate is a
57	drop-in implementation. If your harness logic is tangled into one big function, extract the
58	boundary first. → `assets/candidate_base-template.py`
59	2. A $0 deterministic scorer + rubric — the inner loop. It must vary with the candidate
60	(see the frozen-replay trap below) and run with no LLM / no network so you can call it
61	hundreds of times for free. → `assets/scorer-template.py`
62	3. An eval corpus with a held-out split — the tasks/records candidates are graded on, split
63	so the test set shares no leaky structure with the search set.
64	4. A proposer prior — a short mini-SKILL the proposer agents load that steers them toward
65	mechanism-level changes (not constant-tuning) and enforces anti-leakage.
66	→ `assets/proposer-prior-template.md`
67	5. A frontier + run log — the state carried across iterations (a JSON

People who liked this also viewed

6 picks

Type