skill-tester

Name: skill-tester
Author: pavel-molyanov

241stars75forks

How to install skill-tester?

pavel-molyanov/molyanov-ai-dev/skill-tester

$npx -y skills add pavel-molyanov/molyanov-ai-dev --skill skill-tester

Installs into the current project.

Summary

Test skills end-to-end: design test cases, run with/without skill, grade results, test description triggering accuracy, produce improvement report. Use when: "протестируй скилл", "запусти тесты для скилла", "проверь скилл", "run skill tests", "test this skill", "skill eval", "оце

Files · 1

View on GitHub

SKILL.md

1	<!-- Generated by sync-to-codex v1. Do not edit directly. -->
2
3	# Skill Tester
4
5	Design tests, run them, grade results, evaluate description triggering, produce
6	actionable report. All in one workflow — no separate test-design step needed.
7
8	You are team lead, test designer, user actor, and analyst — all in one.
9
10	## Phase 1: Understand & Design
11
12	### 1a. Read the target skill
13
14	1. User provides skill name or path
15	2. Read the target skill's SKILL.md + ALL referenced files completely
16	3. Map out:
17	- Skill type: procedural / informational
18	- Input: what does the skill expect? (user message, task file, structured data)
19	- Output: what should the skill produce? (files, messages, actions, decisions)
20	- Phases: list all phases/steps with their checkpoints
21	- References: list all files the skill tells agents to read
22	- Decision points: where does the skill branch based on input?
23	- Dialogue points: where does the skill ask the user questions?
24
25	### 1b. Design test prompts
26
27	Design test prompts applying criteria from [test-design-guide.md](references/test-design-guide.md)
28	(realistic prompts, assertion design, persona setup):
29
30	1. Propose 2-3 test prompts:
31	- 1 happy-path: the most common, standard use case
32	- 1-2 edge cases: where the skill might break or behave unexpectedly
33	2. For each prompt, propose assertions — binary, observable checks.
34	Categories:
35	- [Process]: Did the agent follow the skill's workflow?
36	- [Outcome]: Is the result correct and complete?
37	- [Compliance]: Did the agent obey all skill instructions?
38
39	### 1c. Design trigger eval queries
40
41	Design trigger eval queries applying patterns from [trigger-eval-guide.md](references/trigger-eval-guide.md):
42
43	1. Generate 15-20 trigger eval queries:
44	- 8-10 should-trigger: varied phrasings of the skill's intended use
45	- 8-10 should-not-trigger: near-misses that share keywords but need
46	something different
47	2. These will be used in Phase 4 to test description accuracy
48
49	### 1d. Confirm with user
50
51	Present the full test plan:
52	- Test prompts with assertions
53	- Trigger eval queries
54	- Proposed model for runners
55	- Persona (use default, modify only if user requests)
56
57	"Here are the test cases and trigger queries I've prepared. Do these look
58	right, or do you want to adjust anything?"
59
60	Wait for confirmation before proceeding.
61
62	Checkpoint: User confirmed test plan. All prompts have assertions.
63	Trigger eval queries prepared.
64
65	## Phase 2: Execute Tests
66
67	### 2a. Setup
68
69	1. Create workspace: `~/.claude/skill-tests/{skill-name}/iteration-{N}/`
70	(N = 1 for first run, increment for re-runs)
71	2. Save test plan to workspace:
72	- `evals.json` with all prompts and assertions
73	- `trigger-evals.json` with all trigger queries
74	3. spawn_agent worker/explorer orchestration(team_name="skill-test-{skill-name}")
75	4. Plan runners: per scenario = 2 with-skill + 1 baseline without skill
76
77	Show plan to user: "I'll run {N} scenarios, {M} runners total.
78	Model: {model}. Proceed?"
79
80	### 2b. Spawn runners
81
82	For each scenario, spawn all runners in parallel:
83
84	With-skill runners (2 per scenario):
85	- Prompt = scenario's task prompt (natural, as user would write)
86	- Each runner loads the tested skill: `Skill(skill="{tested-skill-name}")`
87	- Model: as confirmed with user
88	- Use `run_in_background: true`
89
90	Baseline runner (1 per scenario):
91	- Same task prompt, same model
92	- Receives no skill to load
93	- Use `run_in_background: true`
94
95	Save each runner's task_id — needed for grader agents to retrieve transcripts.
96
97	Scenarios run sequentially. Runners within a scenario run in parallel.
98
99	### 2c. Interact as user persona
100
101	If runners send questions, answer in character per the scenario's persona.
102	Rules:
103	- Stay in character: answer as the user would
104	- Be consistent: same question from different runners → same answer
105	- Answer naturally — without guidance toward any specific behavior
106	- Keep conversation purely about the task itself
107	- Baseline runner may ask different questions (no skill to guide it) —
108	this is expected, answer them too
109
110	### 2d. Capture timing data
111
112	When each runner completes, immediately save timing data:
113
114	```json
115	{
116	"total_tokens": 84852,
117	"duration_ms": 23332,
118	"total_duration_seconds": 23.3
119	}
120	```
121
122	This is the only opportunity to capture this data — it comes through the
123	task notification and isn't persisted elsewhere. Process each notification
124	as it arrives rather than trying to batch them.
125
126	Save to `timing.json` in each runner's result directory.
127
128	Checkpoint: All runners completed. Timing data captured.
129
130	## Phase 3: Grade & Analyze
131
132	### 3a. Grade via grader age

People who liked this also viewed

6 picks

Type

skill-tester

241stars75forks

View on GitHub

How to install skill-tester?

pavel-molyanov/molyanov-ai-dev/skill-tester

$npx -y skills add pavel-molyanov/molyanov-ai-dev --skill skill-tester

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	<!-- Generated by sync-to-codex v1. Do not edit directly. -->
2
3	# Skill Tester
4
5	Design tests, run them, grade results, evaluate description triggering, produce
6	actionable report. All in one workflow — no separate test-design step needed.
7
8	You are team lead, test designer, user actor, and analyst — all in one.
9
10	## Phase 1: Understand & Design
11
12	### 1a. Read the target skill
13
14	1. User provides skill name or path
15	2. Read the target skill's SKILL.md + ALL referenced files completely
16	3. Map out:
17	- Skill type: procedural / informational
18	- Input: what does the skill expect? (user message, task file, structured data)
19	- Output: what should the skill produce? (files, messages, actions, decisions)
20	- Phases: list all phases/steps with their checkpoints
21	- References: list all files the skill tells agents to read
22	- Decision points: where does the skill branch based on input?
23	- Dialogue points: where does the skill ask the user questions?
24
25	### 1b. Design test prompts
26
27	Design test prompts applying criteria from [test-design-guide.md](references/test-design-guide.md)
28	(realistic prompts, assertion design, persona setup):
29
30	1. Propose 2-3 test prompts:
31	- 1 happy-path: the most common, standard use case
32	- 1-2 edge cases: where the skill might break or behave unexpectedly
33	2. For each prompt, propose assertions — binary, observable checks.
34	Categories:
35	- [Process]: Did the agent follow the skill's workflow?
36	- [Outcome]: Is the result correct and complete?
37	- [Compliance]: Did the agent obey all skill instructions?
38
39	### 1c. Design trigger eval queries
40
41	Design trigger eval queries applying patterns from [trigger-eval-guide.md](references/trigger-eval-guide.md):
42
43	1. Generate 15-20 trigger eval queries:
44	- 8-10 should-trigger: varied phrasings of the skill's intended use
45	- 8-10 should-not-trigger: near-misses that share keywords but need
46	something different
47	2. These will be used in Phase 4 to test description accuracy
48
49	### 1d. Confirm with user
50
51	Present the full test plan:
52	- Test prompts with assertions
53	- Trigger eval queries
54	- Proposed model for runners
55	- Persona (use default, modify only if user requests)
56
57	"Here are the test cases and trigger queries I've prepared. Do these look
58	right, or do you want to adjust anything?"
59
60	Wait for confirmation before proceeding.
61
62	Checkpoint: User confirmed test plan. All prompts have assertions.
63	Trigger eval queries prepared.
64
65	## Phase 2: Execute Tests
66
67	### 2a. Setup
68
69	1. Create workspace: `~/.claude/skill-tests/{skill-name}/iteration-{N}/`
70	(N = 1 for first run, increment for re-runs)
71	2. Save test plan to workspace:
72	- `evals.json` with all prompts and assertions
73	- `trigger-evals.json` with all trigger queries
74	3. spawn_agent worker/explorer orchestration(team_name="skill-test-{skill-name}")
75	4. Plan runners: per scenario = 2 with-skill + 1 baseline without skill
76
77	Show plan to user: "I'll run {N} scenarios, {M} runners total.
78	Model: {model}. Proceed?"
79
80	### 2b. Spawn runners
81
82	For each scenario, spawn all runners in parallel:
83
84	With-skill runners (2 per scenario):
85	- Prompt = scenario's task prompt (natural, as user would write)
86	- Each runner loads the tested skill: `Skill(skill="{tested-skill-name}")`
87	- Model: as confirmed with user
88	- Use `run_in_background: true`
89
90	Baseline runner (1 per scenario):
91	- Same task prompt, same model
92	- Receives no skill to load
93	- Use `run_in_background: true`
94
95	Save each runner's task_id — needed for grader agents to retrieve transcripts.
96
97	Scenarios run sequentially. Runners within a scenario run in parallel.
98
99	### 2c. Interact as user persona
100
101	If runners send questions, answer in character per the scenario's persona.
102	Rules:
103	- Stay in character: answer as the user would
104	- Be consistent: same question from different runners → same answer
105	- Answer naturally — without guidance toward any specific behavior
106	- Keep conversation purely about the task itself
107	- Baseline runner may ask different questions (no skill to guide it) —
108	this is expected, answer them too
109
110	### 2d. Capture timing data
111
112	When each runner completes, immediately save timing data:
113
114	```json
115	{
116	"total_tokens": 84852,
117	"duration_ms": 23332,
118	"total_duration_seconds": 23.3
119	}
120	```
121
122	This is the only opportunity to capture this data — it comes through the
123	task notification and isn't persisted elsewhere. Process each notification
124	as it arrives rather than trying to batch them.
125
126	Save to `timing.json` in each runner's result directory.
127
128	Checkpoint: All runners completed. Timing data captured.
129
130	## Phase 3: Grade & Analyze
131
132	### 3a. Grade via grader age

People who liked this also viewed

6 picks

Type