harness-stripping

Name: harness-stripping
Author: Archive228

712stars129forks

How to install harness-stripping?

archive228/loopkit/harness-stripping

$npx -y skills add Archive228/loopkit --skill harness-stripping

Installs into the current project.

Summary

Systematically remove one harness component at a time and measure impact, killing scaffolding that no longer earns its complexity.

Files · 1

View on GitHub

SKILL.md

1	# Harness Stripping
2
3	Every harness component was added to compensate for a specific model failure. Models improve. Components don't retire themselves. The scaffolding that saved you on Sonnet 4.5 may be dead weight — or actively harmful — on Opus 4.6. Strip it deliberately, one piece at a time, and let evals tell you what still earns its keep.
4
5	Inspired by Prithvi's March 2026 harness post on evaluator-generator separation and the general "re-test your assumptions each model bump" discipline.
6
7	## When to apply
8
9	- A model upgrade just landed and your harness was tuned for the previous generation.
10	- A component's justification is "we added this because the model used to do X" — and you haven't checked whether it still does X.
11	- The harness has accreted over months and nobody remembers what half the machinery is for.
12	- Cost or latency is climbing and you suspect redundant belt-and-suspenders layers.
13
14	## Procedure
15
16	1. Inventory the components. List every distinct piece of scaffolding: prompt sections, tool wrappers, post-hoc validators, retry loops, evaluator personas, structured-output enforcers, sandbox rules. One row per component. Note the failure mode each was added to prevent.
17
18	2. Rank by suspicion. Put the components most likely to be obsolete at the top: anything added before the last two model bumps, anything targeting a failure mode you haven't seen recently, anything whose original justification is now folklore.
19
20	3. Pick a baseline eval. You need a repeatable metric before you touch anything. Reuse an existing eval set if you have one; otherwise pick 20–60 tasks representative of production work. Record baseline score, cost, and wall-clock.
21
22	4. Strip one component. Only one. Comment it out or gate it behind a flag — don't delete yet. Re-run the eval.
23
24	5. Compare against baseline.
25	- Score within noise, cost/latency down → the component is dead weight. Delete.
26	- Score drops measurably → the component still earns its complexity. Restore and note what failure mode returned.
27	- Score improves → the component was actively harmful. Delete and investigate why (often: over-constraining a now-capable model).
28
29	6. Commit the delta. Land the strip (or the restore-with-notes) as its own commit. Do not batch multiple strips into one change — you lose the ability to attribute the score movement.
30
31	7. Repeat for the next component. Re-establish baseline from the new state each round, not the original. Compounding strips have compounding effects.
32
33	## Anti-patterns
34
35	- Stripping two components at once — you can't tell which one mattered. Halve the signal, double the confusion.
36	- Skipping the eval "because it's obviously safe to remove" — the harness accreted for reasons. Some are still real. Measure.
37	- Deleting instead of gating on the first pass — you will want to A/B mid-review. Flag first, delete after the eval confirms.
38	- Trusting anecdotes over the eval — "it feels better without it" is how load-bearing components get removed. If the eval doesn't show it, it isn't there.
39	- Stripping components that guard safety, sandboxing, or cost caps — those aren't compensating for model weakness. Leave them.
40	- Doing this on prod traffic — run against an eval set, not real users. The failure modes you're re-probing are exactly the ones that hurt users.
41
42	## What to strip first
43
44	Highest yield in practice:
45
46	- Structured-output enforcers layered on top of models that now emit valid JSON natively.
47	- Multi-step "plan then execute" wrappers on tasks the model now one-shots.
48	- Retry loops around tool calls that no longer flake.
49	- Evaluator personas whose critiques the generator now anticipates on its own.
50	- Verbose "remember to do X" prompt sections where X is now default behavior.
51
52	## When NOT to apply
53
54	Don't strip mid-project on a live long-running run — you'll perturb sessions in flight. Do it between projects, or on a forked branch. Also skip if you don't have an eval you trust; stripping without measurement is guessing.
55
56	## Related
57
58	- [[shift-notes]] — record which components were stripped and when, so the next audit doesn't re-strip and re-restore the same piece.
59	- [[adversarial-verify]] — the evaluator-generator pattern that may itself be a strip candidate on newer models.
60	- [[broken-window-check]] — if you strip a component and the eval regresses in a specific way, that's your new broken window to hunt.

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

harness-stripping

712stars129forks

View on GitHub

How to install harness-stripping?

archive228/loopkit/harness-stripping

$npx -y skills add Archive228/loopkit --skill harness-stripping

Installs into the current project.

Summary

Systematically remove one harness component at a time and measure impact, killing scaffolding that no longer earns its complexity.

Files · 1

View on GitHub

SKILL.md

1	# Harness Stripping
2
3	Every harness component was added to compensate for a specific model failure. Models improve. Components don't retire themselves. The scaffolding that saved you on Sonnet 4.5 may be dead weight — or actively harmful — on Opus 4.6. Strip it deliberately, one piece at a time, and let evals tell you what still earns its keep.
4
5	Inspired by Prithvi's March 2026 harness post on evaluator-generator separation and the general "re-test your assumptions each model bump" discipline.
6
7	## When to apply
8
9	- A model upgrade just landed and your harness was tuned for the previous generation.
10	- A component's justification is "we added this because the model used to do X" — and you haven't checked whether it still does X.
11	- The harness has accreted over months and nobody remembers what half the machinery is for.
12	- Cost or latency is climbing and you suspect redundant belt-and-suspenders layers.
13
14	## Procedure
15
16	1. Inventory the components. List every distinct piece of scaffolding: prompt sections, tool wrappers, post-hoc validators, retry loops, evaluator personas, structured-output enforcers, sandbox rules. One row per component. Note the failure mode each was added to prevent.
17
18	2. Rank by suspicion. Put the components most likely to be obsolete at the top: anything added before the last two model bumps, anything targeting a failure mode you haven't seen recently, anything whose original justification is now folklore.
19
20	3. Pick a baseline eval. You need a repeatable metric before you touch anything. Reuse an existing eval set if you have one; otherwise pick 20–60 tasks representative of production work. Record baseline score, cost, and wall-clock.
21
22	4. Strip one component. Only one. Comment it out or gate it behind a flag — don't delete yet. Re-run the eval.
23
24	5. Compare against baseline.
25	- Score within noise, cost/latency down → the component is dead weight. Delete.
26	- Score drops measurably → the component still earns its complexity. Restore and note what failure mode returned.
27	- Score improves → the component was actively harmful. Delete and investigate why (often: over-constraining a now-capable model).
28
29	6. Commit the delta. Land the strip (or the restore-with-notes) as its own commit. Do not batch multiple strips into one change — you lose the ability to attribute the score movement.
30
31	7. Repeat for the next component. Re-establish baseline from the new state each round, not the original. Compounding strips have compounding effects.
32
33	## Anti-patterns
34
35	- Stripping two components at once — you can't tell which one mattered. Halve the signal, double the confusion.
36	- Skipping the eval "because it's obviously safe to remove" — the harness accreted for reasons. Some are still real. Measure.
37	- Deleting instead of gating on the first pass — you will want to A/B mid-review. Flag first, delete after the eval confirms.
38	- Trusting anecdotes over the eval — "it feels better without it" is how load-bearing components get removed. If the eval doesn't show it, it isn't there.
39	- Stripping components that guard safety, sandboxing, or cost caps — those aren't compensating for model weakness. Leave them.
40	- Doing this on prod traffic — run against an eval set, not real users. The failure modes you're re-probing are exactly the ones that hurt users.
41
42	## What to strip first
43
44	Highest yield in practice:
45
46	- Structured-output enforcers layered on top of models that now emit valid JSON natively.
47	- Multi-step "plan then execute" wrappers on tasks the model now one-shots.
48	- Retry loops around tool calls that no longer flake.
49	- Evaluator personas whose critiques the generator now anticipates on its own.
50	- Verbose "remember to do X" prompt sections where X is now default behavior.
51
52	## When NOT to apply
53
54	Don't strip mid-project on a live long-running run — you'll perturb sessions in flight. Do it between projects, or on a forked branch. Also skip if you don't have an eval you trust; stripping without measurement is guessing.
55
56	## Related
57
58	- [[shift-notes]] — record which components were stripped and when, so the next audit doesn't re-strip and re-restore the same piece.
59	- [[adversarial-verify]] — the evaluator-generator pattern that may itself be a strip candidate on newer models.
60	- [[broken-window-check]] — if you strip a component and the eval regresses in a specific way, that's your new broken window to hunt.

People who liked this also viewed

6 picks

Type