optimize-skill

Name: optimize-skill
Author: Bennyoooo

21stars2forks

How to install optimize-skill?

bennyoooo/skillmaxxing/optimize-skill

$npx -y skills add Bennyoooo/skillmaxxing --skill optimize-skill

Installs into the current project.

Summary

Improve an existing skill through an evaluation-gated loop — run it on its eval tasks, analyze failures, propose bounded edits, validate a candidate, and (with your approval) promote a strictly-better version. Use when a skill underperforms or the user asks to optimize/tune/impro

Files · 1

View on GitHub

SKILL.md

1	# optimize-skill
2
3	Make a skill measurably better without uncontrolled drift. This is an agent-in-the-loop loop, not a hands-off run: the CLI owns the deterministic machinery (scoring, edit budget, rejected-edit buffer, the gate, atomic promote/revert); you own the reasoning (running the skill, judging prose outputs, proposing edits). Expect several turns per optimization.
4
5	## Preconditions
6
7	- The skill has an eval manifest (`eval.yaml`) with real tasks. If it has none, stop and offer to create one (`create-skill`) — optimization cannot run without an eval set.
8	- Optimization edits a managed copy, never the installed symlink target.
9
10	## The loop (repeat until the gate stops improving)
11
12	1. Rollout. For each eval task `input`, run the current skill yourself and collect its output. Write `rollouts.json`: `[{ "taskId": "...", "output": "..." }]`.
13
14	2. Score.
15
16	```bash
17	scripts/optimize.sh score --eval eval.yaml --rollouts rollouts.json --skill <name> --json
18	```
19
20	Deterministic tasks are scored for you. `agent-judge` tasks come back as pending — score those yourself against each task's rubric and fold them into the aggregate. Record the current score.
21
22	3. Reflect. Read the failing trajectories. Diagnose why they failed (this is your job — the CLI never judges why). Propose a small set of structured edits to `SKILL.md`. Write `edits.json`: an array of `{ op: append\|insert_after\|replace\|delete, target?, content?, sourceType: "failure"\|"success", supportCount? }`. Prefer failure-driven edits.
23
24	4. Apply (bounded).
25
26	```bash
27	scripts/optimize.sh apply --skill <name> --skill-dir <live-dir> --edits edits.json --step <n> --total <N>
28	```
29
30	The CLI caps edits at the budget (annealed over steps), skips edits to the protected `SLOW_UPDATE` region, and writes a candidate copy. Note the candidate dir it prints.
31
32	5. Validate. Re-run rollout + score against the candidate (steps 1–2 pointing at the candidate dir), including the held-out tasks. Then gate:
33
34	```bash
35	scripts/optimize.sh gate --current <currentScore> --candidate <candidateScore> --best <bestScore>
36	```
37
38	A non-zero exit means reject — add those edits to your rejected set so you don't re-propose them, and try a different reflection. Also reject if any held-out task regressed, even if the aggregate improved.
39
40	6. Promote (human gate). Only on a strict improvement with no held-out regression, present the candidate and its score delta to the user. On their approval:
41
42	```bash
43	scripts/optimize.sh promote --skill <name> --live <live-dir> --candidate <candidate-dir> --score <candidateScore>
44	```
45
46	The prior version is retained and the change is reversible.
47
48	## Revert
49
50	```bash
51	scripts/optimize.sh revert --skill <name> --version <prior-version> --live <live-dir>
52	```
53
54	## Honesty
55
56	- "Optimize automatically" means the loop, budget, buffer, and gates are automated — the intelligence (rollout, reflection, edits, agent-judge) is yours. A weak reasoning pass simply makes less progress; the gate guarantees no regression is ever promoted.
57	- Never promote without explicit user approval, even when the gate passes.

People who liked this also viewed

6 picks

Type

optimize-skill

21stars2forks

View on GitHub

How to install optimize-skill?

bennyoooo/skillmaxxing/optimize-skill

$npx -y skills add Bennyoooo/skillmaxxing --skill optimize-skill

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# optimize-skill
2
3	Make a skill measurably better without uncontrolled drift. This is an agent-in-the-loop loop, not a hands-off run: the CLI owns the deterministic machinery (scoring, edit budget, rejected-edit buffer, the gate, atomic promote/revert); you own the reasoning (running the skill, judging prose outputs, proposing edits). Expect several turns per optimization.
4
5	## Preconditions
6
7	- The skill has an eval manifest (`eval.yaml`) with real tasks. If it has none, stop and offer to create one (`create-skill`) — optimization cannot run without an eval set.
8	- Optimization edits a managed copy, never the installed symlink target.
9
10	## The loop (repeat until the gate stops improving)
11
12	1. Rollout. For each eval task `input`, run the current skill yourself and collect its output. Write `rollouts.json`: `[{ "taskId": "...", "output": "..." }]`.
13
14	2. Score.
15
16	```bash
17	scripts/optimize.sh score --eval eval.yaml --rollouts rollouts.json --skill <name> --json
18	```
19
20	Deterministic tasks are scored for you. `agent-judge` tasks come back as pending — score those yourself against each task's rubric and fold them into the aggregate. Record the current score.
21
22	3. Reflect. Read the failing trajectories. Diagnose why they failed (this is your job — the CLI never judges why). Propose a small set of structured edits to `SKILL.md`. Write `edits.json`: an array of `{ op: append\|insert_after\|replace\|delete, target?, content?, sourceType: "failure"\|"success", supportCount? }`. Prefer failure-driven edits.
23
24	4. Apply (bounded).
25
26	```bash
27	scripts/optimize.sh apply --skill <name> --skill-dir <live-dir> --edits edits.json --step <n> --total <N>
28	```
29
30	The CLI caps edits at the budget (annealed over steps), skips edits to the protected `SLOW_UPDATE` region, and writes a candidate copy. Note the candidate dir it prints.
31
32	5. Validate. Re-run rollout + score against the candidate (steps 1–2 pointing at the candidate dir), including the held-out tasks. Then gate:
33
34	```bash
35	scripts/optimize.sh gate --current <currentScore> --candidate <candidateScore> --best <bestScore>
36	```
37
38	A non-zero exit means reject — add those edits to your rejected set so you don't re-propose them, and try a different reflection. Also reject if any held-out task regressed, even if the aggregate improved.
39
40	6. Promote (human gate). Only on a strict improvement with no held-out regression, present the candidate and its score delta to the user. On their approval:
41
42	```bash
43	scripts/optimize.sh promote --skill <name> --live <live-dir> --candidate <candidate-dir> --score <candidateScore>
44	```
45
46	The prior version is retained and the change is reversible.
47
48	## Revert
49
50	```bash
51	scripts/optimize.sh revert --skill <name> --version <prior-version> --live <live-dir>
52	```
53
54	## Honesty
55
56	- "Optimize automatically" means the loop, budget, buffer, and gates are automated — the intelligence (rollout, reflection, edits, agent-judge) is yours. A weak reasoning pass simply makes less progress; the gate guarantees no regression is ever promoted.
57	- Never promote without explicit user approval, even when the gate passes.

People who liked this also viewed

6 picks

Type