autoresearch

Name: autoresearch
Author: github

37kstars4.6kforks

How to install autoresearch?

github/awesome-copilot/autoresearch

$npx -y skills add github/awesome-copilot --skill autoresearch

Installs into the current project.

Summary

Autonomous iterative experimentation loop for any programming task. Guides the user through defining goals, measurable metrics, and scope constraints, then runs an autonomous loop of code changes, testing, measuring, and keeping/discarding results. Inspired by Karpathy''s autores

Files · 1

View on GitHub

SKILL.md

1	# Autoresearch: Autonomous Iterative Experimentation
2
3	An autonomous experimentation loop for any programming task. You define the goal and how to measure it; the agent iterates autonomously -- modifying code, running experiments, measuring results, and keeping or discarding changes -- until interrupted.
4
5	This skill is inspired by [Karpathy's autoresearch](https://github.com/karpathy/autoresearch), generalized from ML training to any programming task with a measurable outcome.
6
7	---
8
9	## Agent Behavior Rules
10
11	1. DO guide the user through the Setup phase interactively before starting the loop.
12	2. DO establish a baseline measurement before making any changes.
13	3. DO commit every experiment attempt before running it (so it can be reverted cleanly).
14	4. DO keep a results log (TSV) tracking every experiment.
15	5. DO revert changes that do not improve the metric (git reset to last known good).
16	6. DO run autonomously once the loop starts -- never pause to ask "should I continue?".
17	7. DO NOT modify files the user marked as out-of-scope.
18	8. DO NOT skip the measurement step -- every experiment must be measured.
19	9. DO NOT keep changes that regress the metric unless the user explicitly allowed trade-offs.
20	10. DO NOT install new dependencies or make environment changes unless the user approved it.
21
22	---
23
24	## Phase 1: Setup (Interactive)
25
26	Before any experimentation begins, work with the user to establish these parameters.
27	Ask the user directly for each item. Do not assume or skip any.
28
29	### 1.1 Define the Goal
30
31	Ask the user:
32
33	> What are you trying to improve or optimize?
34	>
35	> Examples: execution time, memory usage, binary size, test pass rate, code coverage,
36	> API response latency, throughput, error rate, benchmark score, build time, bundle size,
37	> lines of code, cyclomatic complexity, etc.
38
39	Record the user's answer as the goal.
40
41	### 1.2 Define the Metric
42
43	Ask the user:
44
45	> How do we measure success? What exact command produces the metric?
46	>
47	> I need:
48	> 1. The command to run (e.g., `dotnet test`, `npm run benchmark`, `time ./build.sh`, `pytest --tb=short`)
49	> 2. How to extract the metric from the output (e.g., a regex pattern, a specific line, a JSON field)
50	> 3. Direction: Is lower better or higher better?
51	>
52	> Example: "Run `dotnet test --logger trx`, count passing tests. Higher is better."
53	> Example: "Run `hyperfine './my-program'`, extract mean time. Lower is better."
54
55	Record:
56	- `METRIC_COMMAND`: the command to run
57	- `METRIC_EXTRACTION`: how to extract the numeric metric from output
58	- `METRIC_DIRECTION`: `lower_is_better` or `higher_is_better`
59
60	### 1.3 Define the Scope
61
62	Ask the user:
63
64	> Which files or directories am I allowed to modify?
65	>
66	> And which files are OFF LIMITS (read-only)?
67
68	Record:
69	- `IN_SCOPE_FILES`: files/dirs the agent may edit
70	- `OUT_OF_SCOPE_FILES`: files/dirs that must not be modified
71
72	### 1.4 Define Constraints
73
74	Ask the user:
75
76	> Are there any constraints I should respect?
77	>
78	> Examples:
79	> - Time budget per experiment (e.g., "each run should take < 2 minutes")
80	> - No new dependencies
81	> - Must keep all existing tests passing
82	> - Must not change the public API
83	> - Must maintain backward compatibility
84	> - VRAM/memory limit
85	> - Code complexity limits (prefer simpler solutions)
86
87	Record as `CONSTRAINTS`.
88
89	### 1.5 Define the Experiment Budget (Optional)
90
91	Ask the user:
92
93	> How many experiments should I run, or should I just keep going until you stop me?
94	>
95	> You can say a number (e.g., "try 20 experiments") or "unlimited" (I'll run until you interrupt).
96
97	Record as `MAX_EXPERIMENTS` (number or `unlimited`).
98
99	### 1.6 Simplicity Criterion
100
101	Inform the user of the default simplicity policy:
102
103	> Simplicity policy (default): All else being equal, simpler is better. A small improvement
104	> that adds ugly complexity is not worth it. Removing code while maintaining or improving
105	> the metric is a great outcome. I'll weigh the complexity cost against the improvement
106	> magnitude. Does this policy work for you, or do you want to adjust it?
107
108	Record any adjustments as `SIMPLICITY_POLICY`.
109
110	### 1.7 Confirm

People who liked this also viewed

6 picks

Type

autoresearch

37kstars4.6kforks

View on GitHub

How to install autoresearch?

github/awesome-copilot/autoresearch

$npx -y skills add github/awesome-copilot --skill autoresearch

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Autoresearch: Autonomous Iterative Experimentation
2
3	An autonomous experimentation loop for any programming task. You define the goal and how to measure it; the agent iterates autonomously -- modifying code, running experiments, measuring results, and keeping or discarding changes -- until interrupted.
4
5	This skill is inspired by [Karpathy's autoresearch](https://github.com/karpathy/autoresearch), generalized from ML training to any programming task with a measurable outcome.
6
7	---
8
9	## Agent Behavior Rules
10
11	1. DO guide the user through the Setup phase interactively before starting the loop.
12	2. DO establish a baseline measurement before making any changes.
13	3. DO commit every experiment attempt before running it (so it can be reverted cleanly).
14	4. DO keep a results log (TSV) tracking every experiment.
15	5. DO revert changes that do not improve the metric (git reset to last known good).
16	6. DO run autonomously once the loop starts -- never pause to ask "should I continue?".
17	7. DO NOT modify files the user marked as out-of-scope.
18	8. DO NOT skip the measurement step -- every experiment must be measured.
19	9. DO NOT keep changes that regress the metric unless the user explicitly allowed trade-offs.
20	10. DO NOT install new dependencies or make environment changes unless the user approved it.
21
22	---
23
24	## Phase 1: Setup (Interactive)
25
26	Before any experimentation begins, work with the user to establish these parameters.
27	Ask the user directly for each item. Do not assume or skip any.
28
29	### 1.1 Define the Goal
30
31	Ask the user:
32
33	> What are you trying to improve or optimize?
34	>
35	> Examples: execution time, memory usage, binary size, test pass rate, code coverage,
36	> API response latency, throughput, error rate, benchmark score, build time, bundle size,
37	> lines of code, cyclomatic complexity, etc.
38
39	Record the user's answer as the goal.
40
41	### 1.2 Define the Metric
42
43	Ask the user:
44
45	> How do we measure success? What exact command produces the metric?
46	>
47	> I need:
48	> 1. The command to run (e.g., `dotnet test`, `npm run benchmark`, `time ./build.sh`, `pytest --tb=short`)
49	> 2. How to extract the metric from the output (e.g., a regex pattern, a specific line, a JSON field)
50	> 3. Direction: Is lower better or higher better?
51	>
52	> Example: "Run `dotnet test --logger trx`, count passing tests. Higher is better."
53	> Example: "Run `hyperfine './my-program'`, extract mean time. Lower is better."
54
55	Record:
56	- `METRIC_COMMAND`: the command to run
57	- `METRIC_EXTRACTION`: how to extract the numeric metric from output
58	- `METRIC_DIRECTION`: `lower_is_better` or `higher_is_better`
59
60	### 1.3 Define the Scope
61
62	Ask the user:
63
64	> Which files or directories am I allowed to modify?
65	>
66	> And which files are OFF LIMITS (read-only)?
67
68	Record:
69	- `IN_SCOPE_FILES`: files/dirs the agent may edit
70	- `OUT_OF_SCOPE_FILES`: files/dirs that must not be modified
71
72	### 1.4 Define Constraints
73
74	Ask the user:
75
76	> Are there any constraints I should respect?
77	>
78	> Examples:
79	> - Time budget per experiment (e.g., "each run should take < 2 minutes")
80	> - No new dependencies
81	> - Must keep all existing tests passing
82	> - Must not change the public API
83	> - Must maintain backward compatibility
84	> - VRAM/memory limit
85	> - Code complexity limits (prefer simpler solutions)
86
87	Record as `CONSTRAINTS`.
88
89	### 1.5 Define the Experiment Budget (Optional)
90
91	Ask the user:
92
93	> How many experiments should I run, or should I just keep going until you stop me?
94	>
95	> You can say a number (e.g., "try 20 experiments") or "unlimited" (I'll run until you interrupt).
96
97	Record as `MAX_EXPERIMENTS` (number or `unlimited`).
98
99	### 1.6 Simplicity Criterion
100
101	Inform the user of the default simplicity policy:
102
103	> Simplicity policy (default): All else being equal, simpler is better. A small improvement
104	> that adds ugly complexity is not worth it. Removing code while maintaining or improving
105	> the metric is a great outcome. I'll weigh the complexity cost against the improvement
106	> magnitude. Does this policy work for you, or do you want to adjust it?
107
108	Record any adjustments as `SIMPLICITY_POLICY`.
109
110	### 1.7 Confirm

People who liked this also viewed

6 picks

Type