autoresearch

Name: autoresearch
Author: byungjunjang

114stars65forks

How to install autoresearch?

byungjunjang/jangpm-meta-skills/autoresearch

$npx -y skills add byungjunjang/jangpm-meta-skills --skill autoresearch

Installs into the current project.

Summary

Autonomously optimize a Claude Code skill or agent system by running it repeatedly, scoring outputs against evals, mutating owned artifacts (prompt, references, scripts, agent definitions), and keeping improvements. Karpathy's autoresearch methodology. Trigger when the user asks

Files · 1

View on GitHub

SKILL.md

1	# Autoresearch for Skills
2
3	## When NOT to use this skill
4
5	One-off skill edits — fixing a typo, tweaking a sentence, adding a description or a small rule — do not need an experiment loop. Just edit the skill file directly. Use autoresearch only when iterative run-score-mutate evaluation is actually wanted.
6
7	Most skills work about 70% of the time. The other 30% you get garbage. The fix isn't to rewrite the skill from scratch. It's to let an agent run it dozens of times, score every output, and tighten the right artifact — prompt, reference asset, or executable code — until that 30% disappears.
8
9	This skill adapts Andrej Karpathy's autoresearch methodology (autonomous experimentation loops) to Claude Code skills and agent systems. Karpathy mutated ML training code; here we mutate every artifact a skill owns: SKILL.md prose, reference assets in `references/`, and executable artifacts the skill invokes (scripts, agent/subagent definitions, MCP servers, hooks, harness code).
10
11	---
12
13	## the core job
14
15	Run a loop: generate outputs → score against evals → mutate the skill → keep improvements → repeat.
16
17	Output: An improved target skill file + `results.json` + `results.tsv` + `changelog.md` + `research-log.json` + live HTML dashboard.
18
19	---
20
21	## project setup (required)
22
23	autoresearch modifies the target skill file on every experiment. To prevent Claude Code from prompting for approval each time, add the following permissions to `.claude/settings.json` at the project root.
24
25	```json
26	{
27	"permissions": {
28	"allow": [
29	"Edit(.claude/skills/**)",
30	"Write(.claude/skills/**)",
31	"Edit(~/.claude/skills/**)",
32	"Write(~/.claude/skills/**)"
33	]
34	}
35	}
36	```
37
38	The project-local paths (`.claude/skills/**`) cover skills in this repo and are safe to commit. If you're optimizing a globally installed skill (`~/.claude/skills/`), also add the global paths locally — but do not commit them, as they grant broad write access to all installed skills on any collaborator's machine.
39
40	If `.claude/settings.json` already exists, add only the entries you need to the `permissions.allow` array. Without these permissions, autoresearch will require manual approval every time it modifies the target skill file, which breaks the autonomous loop.
41
42	---
43
44	## step 0: gather context
45
46	STOP. Do not run any experiments until all fields below are confirmed with the user.
47
48	1. Target skill(s) — Which skill to optimize? (exact path to the target `SKILL.md`). For pipelines, list all skills in execution order.
49	2. Pipeline mode — Single skill or multi-skill pipeline? Default: single. See `references/pipeline-guide.md` for pipeline details.
50	3. Owned executable artifacts — Every script, tool implementation, agent/subagent definition, MCP server, hook, and harness file the skill invokes. List them with paths. Default: all in-scope as L2b mutation candidates unless the user explicitly excludes them. If the skill is pure prompt + static references, this list is empty and L2b is unused. See `references/mutation-guide.md` for the L2a/L2b distinction.
51	4. Test inputs — 3-5 different prompts/scenarios covering different use cases. See `references/eval-guide.md` (Test prompt design section) for what makes a good test input.
52	5. Eval criteria — Binary checks for rules (3-6) + comparative checks for quality dimensions (0-5). See `references/eval-guide.md`.
53	6. Runs per experiment — How many times to run the skill per mutation? Default: 5.
54	7. Budget cap — Optional. Max experiment cycles before stopping. Default: no cap.
55	8. Termination conditions — When to stop auto mode. Default: 95%+ binary pass rate for 3 consecutive experiments. See `references/mutation-guide.md` for custom conditions.
56	9. Human review mode — Review the first few experiments before full auto? Default: yes (first 3). Set to `skip` for fully autonomous.
57
58	If the user provides an `evals.json` file, use that instead of asking for items 4-5.
59
60	---
61
62	## step 1: read the skill
63
64	Before changing anything, read and understand the target skill completely.
65
66	1. Read the full target skill file
67	2. Read any files in `references/` that the target skill links to (L2a candidates)
68	3. Read every executable artifact the skill invokes — scripts, tool implementations, agent/subagent definitions (e.g., `agents/openai.yaml`), MCP server code, hook scripts, harness code (L2b candidates). Use the list captured in `Step 0` item 3 as the starting set; verify by tracing every command, tool call, and subagent dispatch the skill perform

People who liked this also viewed

6 picks

Type

autoresearch

114stars65forks

View on GitHub

How to install autoresearch?

byungjunjang/jangpm-meta-skills/autoresearch

$npx -y skills add byungjunjang/jangpm-meta-skills --skill autoresearch

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Autoresearch for Skills
2
3	## When NOT to use this skill
4
5	One-off skill edits — fixing a typo, tweaking a sentence, adding a description or a small rule — do not need an experiment loop. Just edit the skill file directly. Use autoresearch only when iterative run-score-mutate evaluation is actually wanted.
6
7	Most skills work about 70% of the time. The other 30% you get garbage. The fix isn't to rewrite the skill from scratch. It's to let an agent run it dozens of times, score every output, and tighten the right artifact — prompt, reference asset, or executable code — until that 30% disappears.
8
9	This skill adapts Andrej Karpathy's autoresearch methodology (autonomous experimentation loops) to Claude Code skills and agent systems. Karpathy mutated ML training code; here we mutate every artifact a skill owns: SKILL.md prose, reference assets in `references/`, and executable artifacts the skill invokes (scripts, agent/subagent definitions, MCP servers, hooks, harness code).
10
11	---
12
13	## the core job
14
15	Run a loop: generate outputs → score against evals → mutate the skill → keep improvements → repeat.
16
17	Output: An improved target skill file + `results.json` + `results.tsv` + `changelog.md` + `research-log.json` + live HTML dashboard.
18
19	---
20
21	## project setup (required)
22
23	autoresearch modifies the target skill file on every experiment. To prevent Claude Code from prompting for approval each time, add the following permissions to `.claude/settings.json` at the project root.
24
25	```json
26	{
27	"permissions": {
28	"allow": [
29	"Edit(.claude/skills/**)",
30	"Write(.claude/skills/**)",
31	"Edit(~/.claude/skills/**)",
32	"Write(~/.claude/skills/**)"
33	]
34	}
35	}
36	```
37
38	The project-local paths (`.claude/skills/**`) cover skills in this repo and are safe to commit. If you're optimizing a globally installed skill (`~/.claude/skills/`), also add the global paths locally — but do not commit them, as they grant broad write access to all installed skills on any collaborator's machine.
39
40	If `.claude/settings.json` already exists, add only the entries you need to the `permissions.allow` array. Without these permissions, autoresearch will require manual approval every time it modifies the target skill file, which breaks the autonomous loop.
41
42	---
43
44	## step 0: gather context
45
46	STOP. Do not run any experiments until all fields below are confirmed with the user.
47
48	1. Target skill(s) — Which skill to optimize? (exact path to the target `SKILL.md`). For pipelines, list all skills in execution order.
49	2. Pipeline mode — Single skill or multi-skill pipeline? Default: single. See `references/pipeline-guide.md` for pipeline details.
50	3. Owned executable artifacts — Every script, tool implementation, agent/subagent definition, MCP server, hook, and harness file the skill invokes. List them with paths. Default: all in-scope as L2b mutation candidates unless the user explicitly excludes them. If the skill is pure prompt + static references, this list is empty and L2b is unused. See `references/mutation-guide.md` for the L2a/L2b distinction.
51	4. Test inputs — 3-5 different prompts/scenarios covering different use cases. See `references/eval-guide.md` (Test prompt design section) for what makes a good test input.
52	5. Eval criteria — Binary checks for rules (3-6) + comparative checks for quality dimensions (0-5). See `references/eval-guide.md`.
53	6. Runs per experiment — How many times to run the skill per mutation? Default: 5.
54	7. Budget cap — Optional. Max experiment cycles before stopping. Default: no cap.
55	8. Termination conditions — When to stop auto mode. Default: 95%+ binary pass rate for 3 consecutive experiments. See `references/mutation-guide.md` for custom conditions.
56	9. Human review mode — Review the first few experiments before full auto? Default: yes (first 3). Set to `skip` for fully autonomous.
57
58	If the user provides an `evals.json` file, use that instead of asking for items 4-5.
59
60	---
61
62	## step 1: read the skill
63
64	Before changing anything, read and understand the target skill completely.
65
66	1. Read the full target skill file
67	2. Read any files in `references/` that the target skill links to (L2a candidates)
68	3. Read every executable artifact the skill invokes — scripts, tool implementations, agent/subagent definitions (e.g., `agents/openai.yaml`), MCP server code, hook scripts, harness code (L2b candidates). Use the list captured in `Step 0` item 3 as the starting set; verify by tracing every command, tool call, and subagent dispatch the skill perform

People who liked this also viewed

6 picks

Type