skill-forge-eval

Name: skill-forge-eval
Author: AgriciDaniel

78stars33forks

How to install skill-forge-eval?

agricidaniel/skill-forge/skill-forge-eval

$npx -y skills add AgriciDaniel/skill-forge --skill skill-forge-eval

Installs into the current project.

Summary

Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality. Spawns executor, grader, comparator, and analyzer sub-agents for parallel evaluation. Generates eval_metadata.json, grading.json, and feedback reports. Use when u

Files · 1

View on GitHub

SKILL.md

1	# Skill Evaluation Pipeline
2
3	Run structured evaluations against Claude Code skills to verify triggering,
4	correctness, and quality using a multi-agent pipeline.
5
6	## Process
7
8	### Step 1: Define Eval Set
9
10	Accept eval definitions from:
11	- Path to eval set JSON: `evals/evals.json` or user-specified file
12	- Inline prompts: User provides eval queries directly
13	- Auto-generated: Generate from skill description (see Step 1b)
14
15	Eval set JSON schema:
16	```json
17	{
18	"skill_name": "my-skill",
19	"skill_path": "./my-skill",
20	"evals": [
21	{
22	"eval_id": 0,
23	"eval_name": "descriptive-name",
24	"prompt": "The user's task prompt",
25	"input_files": [],
26	"assertions": [
27	{
28	"name": "output-has-score",
29	"check": "Output contains a numeric score between 0-100",
30	"weight": 1.0
31	}
32	],
33	"should_trigger": true
34	}
35	]
36	}
37	```
38
39	#### Step 1b: Auto-Generate Eval Set
40
41	If no eval set exists, generate one:
42	1. Read the skill's SKILL.md description and instructions
43	2. Run `python scripts/generate_eval_set.py <skill-path>` to produce a starter set
44	3. Present the generated set to the user for review and editing
45	4. User approves or modifies before proceeding
46
47	### Step 2: Set Up Workspace
48
49	Create the eval workspace outside the skill directory to avoid confusing eval
50	artifacts with skill files. Use a sibling directory or a dedicated location:
51
52	```
53	eval-workspace/
54	iteration-1/
55	eval-0/
56	eval_metadata.json # Assertions and config for this eval
57	with_skill/
58	outputs/ # Skill execution outputs
59	timing.json # Token count + duration
60	grading.json # Assertion results + evidence
61	baseline/
62	outputs/
63	timing.json
64	grading.json
65	eval-1/
66	eval_metadata.json
67	with_skill/
68	outputs/
69	timing.json
70	grading.json
71	baseline/
72	outputs/
73	timing.json
74	grading.json
75	benchmark.json # Aggregated metrics
76	benchmark.md # Human-readable report
77	```
78
79	For each eval directory, create `eval_metadata.json` from the eval set entry:
80	```json
81	{
82	"eval_id": 0,
83	"eval_name": "descriptive-name",
84	"prompt": "The user's task prompt",
85	"assertions": [...],
86	"should_trigger": true
87	}
88	```
89
90	### Step 3: Execute Eval Runs
91
92	For each eval in the set, spawn two parallel runs:
93
94	With-skill run (delegate to `agents/skill-forge-executor.md`):
95	```
96	Execute this task:
97	- Skill path: <path-to-skill>
98	- Task: <eval prompt>
99	- Input files: <eval files if any, or "none">
100	- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
101	- Outputs to save: <what the assertions check>
102	```
103
104	Baseline run (delegate to `agents/skill-forge-executor.md`):
105	- For new skills: run without the skill loaded
106	- For improved skills: run with the previous version (snapshot it first)
107
108	Save timing data to `timing.json` in each run directory:
109	```json
110	{
111	"total_tokens": 84852,
112	"duration_ms": 23332,
113	"total_duration_seconds": 23.3
114	}
115	```
116
117	### Step 4: Grade Results
118
119	Delegate to `agents/skill-forge-grader.md` for each completed run:
120
121	1. Grade against assertions defined in eval_metadata.json
122	2. Save results to `grading.json` per run:
123	```json
124	{
125	"eval_id": 0,
126	"run_type": "with_skill",
127	"assertions": [
128	{
129	"name": "output-has-score",
130	"passed": true,
131	"evidence": "Found score: 87/100 on line 14"
132	}
133	],
134	"pass_rate": 1.0
135	}
136	```
137
138	### Step 5: Aggregate and Analyze
139
140	1. Run `python scripts/aggregate_benchmark.py <workspace>/iteration-<N> --skill-name <name>`
141	2. This produces `benchmark.json` and `benchmark.md` with:
142	- Pass rate per eval (with_skill vs baseline)
143	- Average time and token usage
144	- Improvement ratio (with_skill / baseline)
145
146	3. Delegate to `agents/skill-forge-analyzer.md` to:
147	- Surface patterns that aggregate stats might hide
148	- Identify consistently failing assertion types
149	- Flag regressions from previous iterations
150
151	### Step 6: Present Results
152
153	Generate a summary report:
154
155	```markdown
156	# Eval Report: [skill-name] — Iteration [N]
157
158	## Overall
159	\| Metric \| With Skill \| Baseline \| Delta \|
160	\|--------\|-----------\|----------\|-------\|
161	\| Pass Rate \| X% \| Y% \| +Z% \|
162	\| Avg Time \| Xs \| Ys \| -Zs \|
163	\| Avg Tokens \| X \| Y \| -Z \|
164
165	## Per-Eval Results
166	\| Eval \| With Skill \| Baseline \| Status \|
167	\|------\|-----------\|----------\|--------\|
168	\| eval-0 \| PASS \| FAIL \| Improved \|
169	\| eval-1 \| PASS \| PASS \| Maintained \|
170
171	## Patterns & Insights

People who liked this also viewed

6 picks

Type

skill-forge-eval

78stars33forks

View on GitHub

How to install skill-forge-eval?

agricidaniel/skill-forge/skill-forge-eval

$npx -y skills add AgriciDaniel/skill-forge --skill skill-forge-eval

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Skill Evaluation Pipeline
2
3	Run structured evaluations against Claude Code skills to verify triggering,
4	correctness, and quality using a multi-agent pipeline.
5
6	## Process
7
8	### Step 1: Define Eval Set
9
10	Accept eval definitions from:
11	- Path to eval set JSON: `evals/evals.json` or user-specified file
12	- Inline prompts: User provides eval queries directly
13	- Auto-generated: Generate from skill description (see Step 1b)
14
15	Eval set JSON schema:
16	```json
17	{
18	"skill_name": "my-skill",
19	"skill_path": "./my-skill",
20	"evals": [
21	{
22	"eval_id": 0,
23	"eval_name": "descriptive-name",
24	"prompt": "The user's task prompt",
25	"input_files": [],
26	"assertions": [
27	{
28	"name": "output-has-score",
29	"check": "Output contains a numeric score between 0-100",
30	"weight": 1.0
31	}
32	],
33	"should_trigger": true
34	}
35	]
36	}
37	```
38
39	#### Step 1b: Auto-Generate Eval Set
40
41	If no eval set exists, generate one:
42	1. Read the skill's SKILL.md description and instructions
43	2. Run `python scripts/generate_eval_set.py <skill-path>` to produce a starter set
44	3. Present the generated set to the user for review and editing
45	4. User approves or modifies before proceeding
46
47	### Step 2: Set Up Workspace
48
49	Create the eval workspace outside the skill directory to avoid confusing eval
50	artifacts with skill files. Use a sibling directory or a dedicated location:
51
52	```
53	eval-workspace/
54	iteration-1/
55	eval-0/
56	eval_metadata.json # Assertions and config for this eval
57	with_skill/
58	outputs/ # Skill execution outputs
59	timing.json # Token count + duration
60	grading.json # Assertion results + evidence
61	baseline/
62	outputs/
63	timing.json
64	grading.json
65	eval-1/
66	eval_metadata.json
67	with_skill/
68	outputs/
69	timing.json
70	grading.json
71	baseline/
72	outputs/
73	timing.json
74	grading.json
75	benchmark.json # Aggregated metrics
76	benchmark.md # Human-readable report
77	```
78
79	For each eval directory, create `eval_metadata.json` from the eval set entry:
80	```json
81	{
82	"eval_id": 0,
83	"eval_name": "descriptive-name",
84	"prompt": "The user's task prompt",
85	"assertions": [...],
86	"should_trigger": true
87	}
88	```
89
90	### Step 3: Execute Eval Runs
91
92	For each eval in the set, spawn two parallel runs:
93
94	With-skill run (delegate to `agents/skill-forge-executor.md`):
95	```
96	Execute this task:
97	- Skill path: <path-to-skill>
98	- Task: <eval prompt>
99	- Input files: <eval files if any, or "none">
100	- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
101	- Outputs to save: <what the assertions check>
102	```
103
104	Baseline run (delegate to `agents/skill-forge-executor.md`):
105	- For new skills: run without the skill loaded
106	- For improved skills: run with the previous version (snapshot it first)
107
108	Save timing data to `timing.json` in each run directory:
109	```json
110	{
111	"total_tokens": 84852,
112	"duration_ms": 23332,
113	"total_duration_seconds": 23.3
114	}
115	```
116
117	### Step 4: Grade Results
118
119	Delegate to `agents/skill-forge-grader.md` for each completed run:
120
121	1. Grade against assertions defined in eval_metadata.json
122	2. Save results to `grading.json` per run:
123	```json
124	{
125	"eval_id": 0,
126	"run_type": "with_skill",
127	"assertions": [
128	{
129	"name": "output-has-score",
130	"passed": true,
131	"evidence": "Found score: 87/100 on line 14"
132	}
133	],
134	"pass_rate": 1.0
135	}
136	```
137
138	### Step 5: Aggregate and Analyze
139
140	1. Run `python scripts/aggregate_benchmark.py <workspace>/iteration-<N> --skill-name <name>`
141	2. This produces `benchmark.json` and `benchmark.md` with:
142	- Pass rate per eval (with_skill vs baseline)
143	- Average time and token usage
144	- Improvement ratio (with_skill / baseline)
145
146	3. Delegate to `agents/skill-forge-analyzer.md` to:
147	- Surface patterns that aggregate stats might hide
148	- Identify consistently failing assertion types
149	- Flag regressions from previous iterations
150
151	### Step 6: Present Results
152
153	Generate a summary report:
154
155	```markdown
156	# Eval Report: [skill-name] — Iteration [N]
157
158	## Overall
159	\| Metric \| With Skill \| Baseline \| Delta \|
160	\|--------\|-----------\|----------\|-------\|
161	\| Pass Rate \| X% \| Y% \| +Z% \|
162	\| Avg Time \| Xs \| Ys \| -Zs \|
163	\| Avg Tokens \| X \| Y \| -Z \|
164
165	## Per-Eval Results
166	\| Eval \| With Skill \| Baseline \| Status \|
167	\|------\|-----------\|----------\|--------\|
168	\| eval-0 \| PASS \| FAIL \| Improved \|
169	\| eval-1 \| PASS \| PASS \| Maintained \|
170
171	## Patterns & Insights

People who liked this also viewed

6 picks

Type