arbor

Name: arbor
Author: K-Dense-AI

259installs3.1kforks

How to install arbor?

k-dense-ai/scientific-agent-skills/arbor

$npx -y skills add K-Dense-AI/scientific-agent-skills --skill arbor

Installs into the current project.

Summary

Autonomously improve a real artifact (code, training recipe, agent harness, data pipeline, prompt) against an objective and an evaluator, using Hypothesis Tree Refinement (HTR) from the Arbor paper. Use this whenever someone wants to iteratively optimize something over many exper

Files · 1

View on GitHub

SKILL.md

1	# Arbor — Autonomous Optimization via Hypothesis Tree Refinement
2
3	## Overview
4
5	This skill runs an Autonomous Optimization (AO) loop: starting from an existing artifact and a measurable objective, improve it through many rounds of experiment and evaluation — without step-by-step human supervision and without overfitting to the feedback signal. It's the right tool when the bottleneck isn't writing one good change, but organizing dozens of trials so that lessons accumulate instead of evaporating.
6
7	It implements Hypothesis Tree Refinement (HTR) from Arbor (Jin et al., 2026). The key idea: keep the research state in a persistent hypothesis tree rather than in conversation history. Each node binds a hypothesis, the distilled insight it produced, and a pointer to the artifact version that realizes it. You play the long-lived coordinator that owns this tree and decides where to search; short-lived executor subagents test one hypothesis each in isolated git worktrees and report back. A held-out merge gate admits a change only when it improves on a test evaluator the search never optimized against. This is what turns trial-and-error into cumulative, auditable research.
8
9	Use the `scripts/tree.py` state manager for all the bookkeeping (creating nodes, writing evidence, propagating insights, pruning, the merge gate, the Observe projection). It keeps the state consistent and frees you to spend judgment on what the evidence means.
10
11	## When to use this skill
12
13	Reach for Arbor when the task is iterative improvement of a concrete artifact under an evaluator:
14	- Model training: optimizer/architecture/recipe changes to lower loss or hit a target in fewer steps.
15	- Harness/agent engineering: raising pass rate or accuracy of an agent loop, search harness, or tool-use scaffold.
16	- Data synthesis: improving a generation/filtering pipeline judged by downstream model behavior.
17	- Benchmark optimization: MLE-bench / Kaggle-style "improve the submission" tasks.
18	- Prompt/system optimization where you can score outputs automatically.
19
20	The distinguishing signals: there's an artifact you can modify, an objective, a way to score candidates, and you expect to run many experiments. If the user only wants a single fix or a one-shot answer, this is overkill — just do the work directly. If they want open-ended ideation with no evaluator, use `hypothesis-generation` or `scientific-brainstorming` instead.
21
22	## The AO setup — pin this down first
23
24	Before any experiments, establish the task tuple `(M_0, O, E_dev, E_test)`. Getting this right matters more than any later decision, so confirm it explicitly:
25
26	- M_0 — initial material: the artifact to improve (a repo, a script, a config, a prompt). Make sure it's under git and currently runs.
27	- O — objective: the natural-language goal and the metric direction (maximize accuracy? minimize loss/steps?).
28	- E_dev — development evaluator: a command you can run freely during search to score a candidate. Fast, repeatable.
29	- E_test — held-out test evaluator: a separate evaluator (different seeds, different split, or a larger run) used only at the merge gate. It must not be used as a search oracle — that's the whole point.
30
31	If the user hasn't given you a clean dev/test split, construct one and say so. The dev/test separation is the mechanism that catches overfitting: a candidate that wins on dev but not on test isn't a success, it's a warning that you're exploiting the feedback signal. Without it, autonomous search reliably overfits.
32
33	Initialize the run:
34
35	```bash
36	python scripts/tree.py init \
37	--objective "Improve BrowseComp answer accuracy on the search harness" \
38	--dev-eval "python eval.py --split dev --n 50" \
39	--test-eval "python eval.py --split test --n 300" \
40	--material "." --metric-direction max --branching 3 --m

People who liked this also viewed

6 picks

Type

arbor

259installs3.1kforks

View on GitHub

How to install arbor?

k-dense-ai/scientific-agent-skills/arbor

$npx -y skills add K-Dense-AI/scientific-agent-skills --skill arbor

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Arbor — Autonomous Optimization via Hypothesis Tree Refinement
2
3	## Overview
4
5	This skill runs an Autonomous Optimization (AO) loop: starting from an existing artifact and a measurable objective, improve it through many rounds of experiment and evaluation — without step-by-step human supervision and without overfitting to the feedback signal. It's the right tool when the bottleneck isn't writing one good change, but organizing dozens of trials so that lessons accumulate instead of evaporating.
6
7	It implements Hypothesis Tree Refinement (HTR) from Arbor (Jin et al., 2026). The key idea: keep the research state in a persistent hypothesis tree rather than in conversation history. Each node binds a hypothesis, the distilled insight it produced, and a pointer to the artifact version that realizes it. You play the long-lived coordinator that owns this tree and decides where to search; short-lived executor subagents test one hypothesis each in isolated git worktrees and report back. A held-out merge gate admits a change only when it improves on a test evaluator the search never optimized against. This is what turns trial-and-error into cumulative, auditable research.
8
9	Use the `scripts/tree.py` state manager for all the bookkeeping (creating nodes, writing evidence, propagating insights, pruning, the merge gate, the Observe projection). It keeps the state consistent and frees you to spend judgment on what the evidence means.
10
11	## When to use this skill
12
13	Reach for Arbor when the task is iterative improvement of a concrete artifact under an evaluator:
14	- Model training: optimizer/architecture/recipe changes to lower loss or hit a target in fewer steps.
15	- Harness/agent engineering: raising pass rate or accuracy of an agent loop, search harness, or tool-use scaffold.
16	- Data synthesis: improving a generation/filtering pipeline judged by downstream model behavior.
17	- Benchmark optimization: MLE-bench / Kaggle-style "improve the submission" tasks.
18	- Prompt/system optimization where you can score outputs automatically.
19
20	The distinguishing signals: there's an artifact you can modify, an objective, a way to score candidates, and you expect to run many experiments. If the user only wants a single fix or a one-shot answer, this is overkill — just do the work directly. If they want open-ended ideation with no evaluator, use `hypothesis-generation` or `scientific-brainstorming` instead.
21
22	## The AO setup — pin this down first
23
24	Before any experiments, establish the task tuple `(M_0, O, E_dev, E_test)`. Getting this right matters more than any later decision, so confirm it explicitly:
25
26	- M_0 — initial material: the artifact to improve (a repo, a script, a config, a prompt). Make sure it's under git and currently runs.
27	- O — objective: the natural-language goal and the metric direction (maximize accuracy? minimize loss/steps?).
28	- E_dev — development evaluator: a command you can run freely during search to score a candidate. Fast, repeatable.
29	- E_test — held-out test evaluator: a separate evaluator (different seeds, different split, or a larger run) used only at the merge gate. It must not be used as a search oracle — that's the whole point.
30
31	If the user hasn't given you a clean dev/test split, construct one and say so. The dev/test separation is the mechanism that catches overfitting: a candidate that wins on dev but not on test isn't a success, it's a warning that you're exploiting the feedback signal. Without it, autonomous search reliably overfits.
32
33	Initialize the run:
34
35	```bash
36	python scripts/tree.py init \
37	--objective "Improve BrowseComp answer accuracy on the search harness" \
38	--dev-eval "python eval.py --split dev --n 50" \
39	--test-eval "python eval.py --split test --n 300" \
40	--material "." --metric-direction max --branching 3 --m

People who liked this also viewed

6 picks

Type