skillhone-evaluation

Name: skillhone-evaluation
Author: Tencent

58stars5forks

How to install skillhone-evaluation?

tencent/skillhone/skillhone-evaluation

$npx -y skills add Tencent/SkillHone --skill skillhone-evaluation

Installs into the current project.

Summary

Run and interpret skill evaluations. Use when you need to evaluate a skill, run probe/test/PR-val, check if a PR regresses quality, compare two versions, or diagnose why the score dropped. Handles the full eval lifecycle including solver trajectory diagnosis for tool-level error

Files · 1

View on GitHub

SKILL.md

1	# SkillHone Evaluation
2
3	Run evaluations and diagnose results to inform improvement decisions.
4
5	## Evaluator Harness Model
6
7	The evaluator is not just a scorer. It is a harness that runs the skill against
8	private tasks and leaves a structured evidence trail:
9
10	- Input: a skill checkout plus an eval repo containing datasets, verifier
11	logic, task contract, and optional compiler/audit helpers.
12	- Execution: each eval item runs in an isolated solver workdir. The solver
13	may create files, call scripts, and write the required artifact.
14	- Verification: the eval repo verifier scores the produced answer and may
15	call task-local validators, compilers, parsers, renderers, or audit helpers.
16	- Output JSON: aggregate score plus redacted per-item trace summaries.
17	- Workdir evidence: `trajectory.jsonl`, produced artifacts, and stderr-like
18	signals that explain failures the score cannot explain.
19
20	This separation matters. A low score may come from weak skill instructions, but
21	it may also come from missing files, tool crashes, invalid compiled artifacts,
22	over-strict verifier rules, or infrastructure errors. Evaluation work is about
23	mapping the failure to the correct harness layer.
24
25	On Forgejo-backed repos, `status.py` is a read-only context check before PR
26	validation or merge decisions:
27
28	```bash
29	python3 ~/.skillhone/skills/skillhone/scripts/status.py
30	```
31
32	## Core capability: eval.py
33
34	```bash
35	# Probe — fast iteration signal
36	python3 ~/.skillhone/skills/skillhone/scripts/eval.py \
37	--skill-dir /path/to/skill --eval-dir /path/to/eval-repo \
38	--split probe --output _data/probe_result.json
39
40	# Test — final benchmark (NEVER during iteration)
41	python3 ~/.skillhone/skills/skillhone/scripts/eval.py \
42	--skill-dir /path/to/skill --eval-dir /path/to/eval-repo \
43	--split test --output test_result.json
44	```
45
46	The output JSON includes a `"workdir"` field pointing to solver working
47	directories (e.g. `/data/tmp/eval_agent_xyz/`) containing `trajectory.jsonl`
48	files and produced artifacts for deeper diagnosis.
49
50	## Subagents
51
52	\| Subagent \| What it does \|
53	\|----------\|-------------\|
54	\| `trajectory-analyzer` \| Reads `workdir/work_<uid>/trajectory.jsonl` files to diagnose tool errors (rate limits, wrong tool calls, script crashes). Outputs redacted `_data/trajectory_diagnosis.json` safe to share with improver. \|
55	\| `pr-quality-reviewer` \| Merge gate for skill PRs — runs static check + rubric scoring, posts PR comment, returns APPROVE/REQUEST_CHANGES. \|
56
57	## Trajectory Diagnosis
58
59	`probe_result.json` captures scores and some error categories, but it is only
60	the top of the evidence trail. It cannot fully explain runtime behavior such as:
61
62	- Wikipedia API rate limiting (HTTP 429/403)
63	- Agent calling `web_search` directly instead of `Bash("python3 scripts/web_search.py ...")`
64	- Script crashes (exit code 1 with traceback)
65	- The solver never writing the required artifact
66	- The solver writing a file in the wrong location
67
68	The `trajectory-analyzer` subagent fills this gap by reading raw solver logs. Its output distinguishes infrastructure failures (fix scripts/config) from skill failures (fix SKILL.md). This distinction is critical for avoiding wasted iterations.
69
70	## Compiler Feedback Diagnosis
71
72	Some artifact tasks are compiler-like: Mermaid, LaTeX, TypeScript, Python tests,
73	SQL parsers, JSON/YAML schema validators, browser renderers, and similar tools
74	produce actionable stderr or diagnostics. Do not reduce these failures to
75	`wrong_answer`.
76
77	When a failed trace produced an artifact, inspect the solver workdir from the
78	`workdir` field and run the task-local compiler, validator, renderer, or audit
79	helper on that artifact. Prefer commands and helpers shipped by the eval repo or
80	described in its README/contract; if none exist, use the standard local compiler
81	for that artifact type. Capture only concise, non-gold diagnostic summaries:
82
83	- compiler/parser command used
84	- first error line and location, if available
85	- failed rule name or failed score key, if available
86	- artifact-level pattern, e.g. "invalid Mermaid subgraph syntax" or "missing
87	answer file"
88
89	Write this as `_data/compiler_diagnosis.json` or include it in the existing
90	diagnosis file. It is safe to share with the improver when it contains only
91	error messages, failed score names, and artifact snippets needed to identify the
92	syntax class; do not include gold answers or full eval questions.
93
94	If the task-local verifier already exposes detailed failed score keys, preserve
95	those names. They are usually better improvement signals than a rewritten
96	natural-language summary.
97
98	## Interpreting scores
99
100	\| Field \| Meaning \|
101	\|-------\|---------\|
102	\| `sco

People who liked this also viewed

6 picks

Type

skillhone-evaluation

58stars5forks

View on GitHub

How to install skillhone-evaluation?

tencent/skillhone/skillhone-evaluation

$npx -y skills add Tencent/SkillHone --skill skillhone-evaluation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# SkillHone Evaluation
2
3	Run evaluations and diagnose results to inform improvement decisions.
4
5	## Evaluator Harness Model
6
7	The evaluator is not just a scorer. It is a harness that runs the skill against
8	private tasks and leaves a structured evidence trail:
9
10	- Input: a skill checkout plus an eval repo containing datasets, verifier
11	logic, task contract, and optional compiler/audit helpers.
12	- Execution: each eval item runs in an isolated solver workdir. The solver
13	may create files, call scripts, and write the required artifact.
14	- Verification: the eval repo verifier scores the produced answer and may
15	call task-local validators, compilers, parsers, renderers, or audit helpers.
16	- Output JSON: aggregate score plus redacted per-item trace summaries.
17	- Workdir evidence: `trajectory.jsonl`, produced artifacts, and stderr-like
18	signals that explain failures the score cannot explain.
19
20	This separation matters. A low score may come from weak skill instructions, but
21	it may also come from missing files, tool crashes, invalid compiled artifacts,
22	over-strict verifier rules, or infrastructure errors. Evaluation work is about
23	mapping the failure to the correct harness layer.
24
25	On Forgejo-backed repos, `status.py` is a read-only context check before PR
26	validation or merge decisions:
27
28	```bash
29	python3 ~/.skillhone/skills/skillhone/scripts/status.py
30	```
31
32	## Core capability: eval.py
33
34	```bash
35	# Probe — fast iteration signal
36	python3 ~/.skillhone/skills/skillhone/scripts/eval.py \
37	--skill-dir /path/to/skill --eval-dir /path/to/eval-repo \
38	--split probe --output _data/probe_result.json
39
40	# Test — final benchmark (NEVER during iteration)
41	python3 ~/.skillhone/skills/skillhone/scripts/eval.py \
42	--skill-dir /path/to/skill --eval-dir /path/to/eval-repo \
43	--split test --output test_result.json
44	```
45
46	The output JSON includes a `"workdir"` field pointing to solver working
47	directories (e.g. `/data/tmp/eval_agent_xyz/`) containing `trajectory.jsonl`
48	files and produced artifacts for deeper diagnosis.
49
50	## Subagents
51
52	\| Subagent \| What it does \|
53	\|----------\|-------------\|
54	\| `trajectory-analyzer` \| Reads `workdir/work_<uid>/trajectory.jsonl` files to diagnose tool errors (rate limits, wrong tool calls, script crashes). Outputs redacted `_data/trajectory_diagnosis.json` safe to share with improver. \|
55	\| `pr-quality-reviewer` \| Merge gate for skill PRs — runs static check + rubric scoring, posts PR comment, returns APPROVE/REQUEST_CHANGES. \|
56
57	## Trajectory Diagnosis
58
59	`probe_result.json` captures scores and some error categories, but it is only
60	the top of the evidence trail. It cannot fully explain runtime behavior such as:
61
62	- Wikipedia API rate limiting (HTTP 429/403)
63	- Agent calling `web_search` directly instead of `Bash("python3 scripts/web_search.py ...")`
64	- Script crashes (exit code 1 with traceback)
65	- The solver never writing the required artifact
66	- The solver writing a file in the wrong location
67
68	The `trajectory-analyzer` subagent fills this gap by reading raw solver logs. Its output distinguishes infrastructure failures (fix scripts/config) from skill failures (fix SKILL.md). This distinction is critical for avoiding wasted iterations.
69
70	## Compiler Feedback Diagnosis
71
72	Some artifact tasks are compiler-like: Mermaid, LaTeX, TypeScript, Python tests,
73	SQL parsers, JSON/YAML schema validators, browser renderers, and similar tools
74	produce actionable stderr or diagnostics. Do not reduce these failures to
75	`wrong_answer`.
76
77	When a failed trace produced an artifact, inspect the solver workdir from the
78	`workdir` field and run the task-local compiler, validator, renderer, or audit
79	helper on that artifact. Prefer commands and helpers shipped by the eval repo or
80	described in its README/contract; if none exist, use the standard local compiler
81	for that artifact type. Capture only concise, non-gold diagnostic summaries:
82
83	- compiler/parser command used
84	- first error line and location, if available
85	- failed rule name or failed score key, if available
86	- artifact-level pattern, e.g. "invalid Mermaid subgraph syntax" or "missing
87	answer file"
88
89	Write this as `_data/compiler_diagnosis.json` or include it in the existing
90	diagnosis file. It is safe to share with the improver when it contains only
91	error messages, failed score names, and artifact snippets needed to identify the
92	syntax class; do not include gold answers or full eval questions.
93
94	If the task-local verifier already exposes detailed failed score keys, preserve
95	those names. They are usually better improvement signals than a rewritten
96	natural-language summary.
97
98	## Interpreting scores
99
100	\| Field \| Meaning \|
101	\|-------\|---------\|
102	\| `sco

People who liked this also viewed

6 picks

Type