skillhone-synthesis

Name: skillhone-synthesis
Author: Tencent

58stars5forks

How to install skillhone-synthesis?

tencent/skillhone/skillhone-synthesis

$npx -y skills add Tencent/SkillHone --skill skillhone-synthesis

Installs into the current project.

Summary

Use this skill to synthesize closed-form, automatically verifiable benchmark Q/A by exploring a tool environment, building a reusable exploration graph, and mining multiple hard questions from that graph. Use for: building a benchmark, writing eval items, generating evaluation da

Files · 1

View on GitHub

SKILL.md

1	# SkillHone Synthesis
2
3	This skill builds a generic closed-form Q/A dataset from real tool use.
4
5	Closed-form means each question has one intended answer, and the answer can be checked by code: exact match, regex, numeric tolerance, enum, or JSON field checks. The final benchmark should not require a human or an LLM judge for basic grading.
6
7	The central idea is simple:
8
9	1. Explore the target environment with tools.
10	2. Save what was learned as a graph of entities, relations, and reasoning walks.
11	3. Mine several Q/A samples from that graph.
12	4. Validate that each sample is answerable, unique, stable, hard enough, and mechanically gradable.
13
14	The skill is domain-agnostic. Concrete task files, tool wrappers, and domain examples live outside this skill.
15
16	## Task Spec Coverage
17
18	The task description is the source of truth for both the solver prompt and the
19	verifier. When a task spec uses words such as "must", "must not", "required",
20	"forbidden", "output format", "validation", "quality", or "acceptance", treat
21	those clauses as benchmark requirements.
22
23	For every requirement in the task spec that can be checked mechanically, the
24	generated verifier should include a corresponding `scores` key. This includes
25	artifact constraints such as file names, raw-vs-wrapped output, syntax validity,
26	compile/render success, required fields, banned strings, shape/count limits,
27	style tokens, fixed palettes, local-only dependencies, and any other observable
28	property of the submitted answer.
29
30	Do not reduce the verifier to only the gold answer. If the task says the output
31	must have a particular structure or quality, that structure or quality should be
32	scored directly whenever it is observable from the answer or from local helper
33	tools. Subjective requirements may be approximated by deterministic proxies, but
34	clearly subjective-only preferences should not be silently converted into a pass.
35
36	## Inputs
37
38	Ask for these if they are missing:
39
40	- Task description: what environment is being explored and who will answer the questions.
41	- Tool interface: CLI commands, MCP tools, shell commands, or local files the explorer may use.
42	- Answer contract: allowed answer types and formatting rules.
43	- Target yield: desired number of final samples and optional coverage goals.
44
45	Do not infer hidden tools or hidden datasets. If the task needs live data, the Cartographer must collect it through the provided tools.
46
47	## Simple, Hard, And Broken Questions
48
49	A simple question usually has one of these shapes:
50
51	- It asks for a single obvious field.
52	- It names the target entity directly.
53	- It ends on a fact that a strong model may already know.
54	- It can be answered without using the tools.
55
56	A hard question usually forces at least one real operation:
57
58	- Resolve an entity from attributes rather than from its name or ID.
59	- Traverse multiple linked records.
60	- Combine data from two sources.
61	- Compute an aggregation, ratio, rank, or derived threshold.
62	- Apply a meaningful exclusion or tie-breaker.
63	- Return a precise value that cannot be guessed approximately.
64
65	A broken question is not a hard question. Drop or repair it if it is ambiguous, unstable, subjective, unverifiable, or impossible to answer from the collected graph.
66
67	Example:
68
69	- Too easy: "What is the display name of record R001?"
70	- Better: "Among records in segment A during the fixed window, find the record ranked third by metric X after excluding records without a linked event. What is its stable source code?"
71
72	The second question is not hard because the entities are obscure. It is hard because the solver must resolve a set, apply constraints, rank, traverse a relation, and return a precise terminal value.
73
74	## Pipeline
75
76	\| Stage \| Agent \| Purpose \| Tool access \|
77	\|---\|---\|---\|---\|
78	\| 1 \| Cartographer \| Explore with tools and build a graph plus reasoning walks. \| Yes \|
79	\| 2 \| Miner \| Convert walks into Q/A candidates and verification snippets. \| No \|
80	\| 3 \| Validator \| Reject candidates that are not answerable, unique, stable, executable, or hard enough. \| Sometimes \|
81	\| 4 \| Deduper \| Remove duplicate shapes and produce final files. \| No \|
82
83	The Cartographer is the only stage that should gather new facts. Miner and Deduper are pure over saved artifacts. Validator may run the system-under-test solver when an empirical difficulty gate is requested.
84
85	## Exploration Graph
86
87	The graph stores reusable evidence:
88
89	- Entities: records, actors, groups, sources, categories, events, or task-specific eq

People who liked this also viewed

6 picks

Type

skillhone-synthesis

58stars5forks

View on GitHub

How to install skillhone-synthesis?

tencent/skillhone/skillhone-synthesis

$npx -y skills add Tencent/SkillHone --skill skillhone-synthesis

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# SkillHone Synthesis
2
3	This skill builds a generic closed-form Q/A dataset from real tool use.
4
5	Closed-form means each question has one intended answer, and the answer can be checked by code: exact match, regex, numeric tolerance, enum, or JSON field checks. The final benchmark should not require a human or an LLM judge for basic grading.
6
7	The central idea is simple:
8
9	1. Explore the target environment with tools.
10	2. Save what was learned as a graph of entities, relations, and reasoning walks.
11	3. Mine several Q/A samples from that graph.
12	4. Validate that each sample is answerable, unique, stable, hard enough, and mechanically gradable.
13
14	The skill is domain-agnostic. Concrete task files, tool wrappers, and domain examples live outside this skill.
15
16	## Task Spec Coverage
17
18	The task description is the source of truth for both the solver prompt and the
19	verifier. When a task spec uses words such as "must", "must not", "required",
20	"forbidden", "output format", "validation", "quality", or "acceptance", treat
21	those clauses as benchmark requirements.
22
23	For every requirement in the task spec that can be checked mechanically, the
24	generated verifier should include a corresponding `scores` key. This includes
25	artifact constraints such as file names, raw-vs-wrapped output, syntax validity,
26	compile/render success, required fields, banned strings, shape/count limits,
27	style tokens, fixed palettes, local-only dependencies, and any other observable
28	property of the submitted answer.
29
30	Do not reduce the verifier to only the gold answer. If the task says the output
31	must have a particular structure or quality, that structure or quality should be
32	scored directly whenever it is observable from the answer or from local helper
33	tools. Subjective requirements may be approximated by deterministic proxies, but
34	clearly subjective-only preferences should not be silently converted into a pass.
35
36	## Inputs
37
38	Ask for these if they are missing:
39
40	- Task description: what environment is being explored and who will answer the questions.
41	- Tool interface: CLI commands, MCP tools, shell commands, or local files the explorer may use.
42	- Answer contract: allowed answer types and formatting rules.
43	- Target yield: desired number of final samples and optional coverage goals.
44
45	Do not infer hidden tools or hidden datasets. If the task needs live data, the Cartographer must collect it through the provided tools.
46
47	## Simple, Hard, And Broken Questions
48
49	A simple question usually has one of these shapes:
50
51	- It asks for a single obvious field.
52	- It names the target entity directly.
53	- It ends on a fact that a strong model may already know.
54	- It can be answered without using the tools.
55
56	A hard question usually forces at least one real operation:
57
58	- Resolve an entity from attributes rather than from its name or ID.
59	- Traverse multiple linked records.
60	- Combine data from two sources.
61	- Compute an aggregation, ratio, rank, or derived threshold.
62	- Apply a meaningful exclusion or tie-breaker.
63	- Return a precise value that cannot be guessed approximately.
64
65	A broken question is not a hard question. Drop or repair it if it is ambiguous, unstable, subjective, unverifiable, or impossible to answer from the collected graph.
66
67	Example:
68
69	- Too easy: "What is the display name of record R001?"
70	- Better: "Among records in segment A during the fixed window, find the record ranked third by metric X after excluding records without a linked event. What is its stable source code?"
71
72	The second question is not hard because the entities are obscure. It is hard because the solver must resolve a set, apply constraints, rank, traverse a relation, and return a precise terminal value.
73
74	## Pipeline
75
76	\| Stage \| Agent \| Purpose \| Tool access \|
77	\|---\|---\|---\|---\|
78	\| 1 \| Cartographer \| Explore with tools and build a graph plus reasoning walks. \| Yes \|
79	\| 2 \| Miner \| Convert walks into Q/A candidates and verification snippets. \| No \|
80	\| 3 \| Validator \| Reject candidates that are not answerable, unique, stable, executable, or hard enough. \| Sometimes \|
81	\| 4 \| Deduper \| Remove duplicate shapes and produce final files. \| No \|
82
83	The Cartographer is the only stage that should gather new facts. Miner and Deduper are pure over saved artifacts. Validator may run the system-under-test solver when an empirical difficulty gate is requested.
84
85	## Exploration Graph
86
87	The graph stores reusable evidence:
88
89	- Entities: records, actors, groups, sources, categories, events, or task-specific eq

People who liked this also viewed

6 picks

Type