skill-forge-benchmark

Name: skill-forge-benchmark
Author: AgriciDaniel

78stars33forks

How to install skill-forge-benchmark?

agricidaniel/skill-forge/skill-forge-benchmark

$npx -y skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark

Installs into the current project.

Summary

Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations. Runs multiple trials per eval for statistical reliability, aggregates results into benchmark.json, and generates comparison reports between skill

Files · 1

View on GitHub

SKILL.md

1	# Skill Benchmarking & Performance Tracking
2
3	Measure and compare skill performance across iterations with statistical
4	rigor using multiple trials, variance analysis, and trend tracking.
5
6	## Process
7
8	### Step 1: Define Benchmark Configuration
9
10	Accept configuration as:
11	- Existing eval set: Path to `evals/evals.json` (from `/skill-forge eval`)
12	- Benchmark config: Custom config with trial count and thresholds
13
14	Benchmark config schema:
15	```json
16	{
17	"skill_name": "my-skill",
18	"skill_path": "./my-skill",
19	"eval_set_path": "./evals/evals.json",
20	"trials_per_eval": 3,
21	"baseline_type": "no_skill",
22	"previous_benchmark": null,
23	"thresholds": {
24	"min_pass_rate": 0.8,
25	"max_avg_tokens": 100000,
26	"max_avg_duration_seconds": 120,
27	"min_improvement_ratio": 1.0
28	}
29	}
30	```
31
32	### Step 2: Execute Benchmark Runs
33
34	For each eval, run `trials_per_eval` times (default: 3) to get reliable metrics:
35
36	1. Execute with-skill runs (3x per eval)
37	2. Execute baseline runs (3x per eval)
38	3. Capture per-run: pass/fail, token count, duration
39	4. Save each run's `timing.json` and `grading.json`
40
41	Use `agents/skill-forge-executor.md` for parallel execution where possible.
42
43	### Step 3: Aggregate Results
44
45	Run `python scripts/aggregate_benchmark.py <workspace>/iteration-<N> --skill-name <name>`:
46
47	Output `benchmark.json` schema:
48	```json
49	{
50	"skill_name": "my-skill",
51	"iteration": 1,
52	"timestamp": "2026-03-06T12:00:00Z",
53	"summary": {
54	"total_evals": 10,
55	"with_skill": {
56	"pass_rate": 0.87,
57	"pass_rate_std": 0.05,
58	"avg_tokens": 45000,
59	"avg_duration_seconds": 34.2
60	},
61	"baseline": {
62	"pass_rate": 0.60,
63	"pass_rate_std": 0.08,
64	"avg_tokens": 62000,
65	"avg_duration_seconds": 52.1
66	},
67	"improvement_ratio": 1.45,
68	"token_savings_ratio": 0.73,
69	"time_savings_ratio": 0.66
70	},
71	"per_eval": [
72	{
73	"eval_id": 0,
74	"eval_name": "basic-trigger",
75	"with_skill": {"pass_rate": 1.0, "avg_tokens": 30000, "avg_duration_seconds": 20.1},
76	"baseline": {"pass_rate": 0.67, "avg_tokens": 50000, "avg_duration_seconds": 45.0},
77	"trials": 3
78	}
79	],
80	"thresholds_met": {
81	"min_pass_rate": true,
82	"max_avg_tokens": true,
83	"max_avg_duration_seconds": true,
84	"min_improvement_ratio": true
85	}
86	}
87	```
88
89	### Step 4: Compare with Previous Iterations
90
91	If `previous_benchmark` is provided or prior `iteration-<N-1>` exists:
92
93	1. Load previous `benchmark.json`
94	2. Calculate delta per metric:
95	- Pass rate change
96	- Token usage change
97	- Duration change
98	- New regressions (evals that passed before but fail now)
99	- New improvements (evals that failed before but pass now)
100
101	### Step 5: Generate Benchmark Report
102
103	```markdown
104	# Benchmark Report: [skill-name]
105
106	## Iteration [N] vs [N-1]
107
108	### Summary
109	\| Metric \| Current \| Previous \| Delta \| Threshold \| Status \|
110	\|--------\|---------\|----------\|-------\|-----------\|--------\|
111	\| Pass Rate \| 87% \| 78% \| +9% \| >= 80% \| PASS \|
112	\| Avg Tokens \| 45K \| 52K \| -13% \| <= 100K \| PASS \|
113	\| Avg Time \| 34s \| 41s \| -17% \| <= 120s \| PASS \|
114	\| Improvement \| 1.45x \| 1.30x \| +0.15x \| >= 1.0x \| PASS \|
115
116	### Regressions (Action Required)
117	\| Eval \| Previous \| Current \| Notes \|
118	\|------\|----------\|---------\|-------\|
119	\| eval-5 \| PASS \| FAIL \| Output missing required section \|
120
121	### Improvements
122	\| Eval \| Previous \| Current \| Notes \|
123	\|------\|----------\|---------\|-------\|
124	\| eval-3 \| FAIL \| PASS \| Error handling now works \|
125
126	### Per-Eval Detail
127	[Full breakdown table]
128
129	### Variance Analysis
130	\| Eval \| Pass Rate \| Std Dev \| Trials \| Reliability \|
131	\|------\|-----------\|---------\|--------\|-------------\|
132	\| eval-0 \| 100% \| 0.00 \| 3 \| High \|
133	\| eval-1 \| 67% \| 0.47 \| 3 \| Low (investigate) \|
134
135	### Recommendations
136	[Based on regressions, low-reliability evals, and threshold failures]
137	```
138
139	### Step 6: Threshold Gating
140
141	If any threshold fails:
142	1. Flag as FAIL with specific threshold details
143	2. List which evals caused the failure
144	3. Recommend running `/skill-forge evolve` to address issues
145	4. Do NOT approve for publish until thresholds pass
146
147	## Error Handling
148
149	- Flaky trials: If a trial times out or crashes, exclude it from variance calculation and note `"trials_completed"` vs `"trials_requested"` in per-eval results
150	- Insufficient trials: If fewer than 2 trials complete for an eval, flag variance as `"unreliable"` in the report
151	- Missing baseline: If baseline runs fail entirely, report with-skill results

People who liked this also viewed

6 picks

Type

skill-forge-benchmark

78stars33forks

View on GitHub

How to install skill-forge-benchmark?

agricidaniel/skill-forge/skill-forge-benchmark

$npx -y skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Skill Benchmarking & Performance Tracking
2
3	Measure and compare skill performance across iterations with statistical
4	rigor using multiple trials, variance analysis, and trend tracking.
5
6	## Process
7
8	### Step 1: Define Benchmark Configuration
9
10	Accept configuration as:
11	- Existing eval set: Path to `evals/evals.json` (from `/skill-forge eval`)
12	- Benchmark config: Custom config with trial count and thresholds
13
14	Benchmark config schema:
15	```json
16	{
17	"skill_name": "my-skill",
18	"skill_path": "./my-skill",
19	"eval_set_path": "./evals/evals.json",
20	"trials_per_eval": 3,
21	"baseline_type": "no_skill",
22	"previous_benchmark": null,
23	"thresholds": {
24	"min_pass_rate": 0.8,
25	"max_avg_tokens": 100000,
26	"max_avg_duration_seconds": 120,
27	"min_improvement_ratio": 1.0
28	}
29	}
30	```
31
32	### Step 2: Execute Benchmark Runs
33
34	For each eval, run `trials_per_eval` times (default: 3) to get reliable metrics:
35
36	1. Execute with-skill runs (3x per eval)
37	2. Execute baseline runs (3x per eval)
38	3. Capture per-run: pass/fail, token count, duration
39	4. Save each run's `timing.json` and `grading.json`
40
41	Use `agents/skill-forge-executor.md` for parallel execution where possible.
42
43	### Step 3: Aggregate Results
44
45	Run `python scripts/aggregate_benchmark.py <workspace>/iteration-<N> --skill-name <name>`:
46
47	Output `benchmark.json` schema:
48	```json
49	{
50	"skill_name": "my-skill",
51	"iteration": 1,
52	"timestamp": "2026-03-06T12:00:00Z",
53	"summary": {
54	"total_evals": 10,
55	"with_skill": {
56	"pass_rate": 0.87,
57	"pass_rate_std": 0.05,
58	"avg_tokens": 45000,
59	"avg_duration_seconds": 34.2
60	},
61	"baseline": {
62	"pass_rate": 0.60,
63	"pass_rate_std": 0.08,
64	"avg_tokens": 62000,
65	"avg_duration_seconds": 52.1
66	},
67	"improvement_ratio": 1.45,
68	"token_savings_ratio": 0.73,
69	"time_savings_ratio": 0.66
70	},
71	"per_eval": [
72	{
73	"eval_id": 0,
74	"eval_name": "basic-trigger",
75	"with_skill": {"pass_rate": 1.0, "avg_tokens": 30000, "avg_duration_seconds": 20.1},
76	"baseline": {"pass_rate": 0.67, "avg_tokens": 50000, "avg_duration_seconds": 45.0},
77	"trials": 3
78	}
79	],
80	"thresholds_met": {
81	"min_pass_rate": true,
82	"max_avg_tokens": true,
83	"max_avg_duration_seconds": true,
84	"min_improvement_ratio": true
85	}
86	}
87	```
88
89	### Step 4: Compare with Previous Iterations
90
91	If `previous_benchmark` is provided or prior `iteration-<N-1>` exists:
92
93	1. Load previous `benchmark.json`
94	2. Calculate delta per metric:
95	- Pass rate change
96	- Token usage change
97	- Duration change
98	- New regressions (evals that passed before but fail now)
99	- New improvements (evals that failed before but pass now)
100
101	### Step 5: Generate Benchmark Report
102
103	```markdown
104	# Benchmark Report: [skill-name]
105
106	## Iteration [N] vs [N-1]
107
108	### Summary
109	\| Metric \| Current \| Previous \| Delta \| Threshold \| Status \|
110	\|--------\|---------\|----------\|-------\|-----------\|--------\|
111	\| Pass Rate \| 87% \| 78% \| +9% \| >= 80% \| PASS \|
112	\| Avg Tokens \| 45K \| 52K \| -13% \| <= 100K \| PASS \|
113	\| Avg Time \| 34s \| 41s \| -17% \| <= 120s \| PASS \|
114	\| Improvement \| 1.45x \| 1.30x \| +0.15x \| >= 1.0x \| PASS \|
115
116	### Regressions (Action Required)
117	\| Eval \| Previous \| Current \| Notes \|
118	\|------\|----------\|---------\|-------\|
119	\| eval-5 \| PASS \| FAIL \| Output missing required section \|
120
121	### Improvements
122	\| Eval \| Previous \| Current \| Notes \|
123	\|------\|----------\|---------\|-------\|
124	\| eval-3 \| FAIL \| PASS \| Error handling now works \|
125
126	### Per-Eval Detail
127	[Full breakdown table]
128
129	### Variance Analysis
130	\| Eval \| Pass Rate \| Std Dev \| Trials \| Reliability \|
131	\|------\|-----------\|---------\|--------\|-------------\|
132	\| eval-0 \| 100% \| 0.00 \| 3 \| High \|
133	\| eval-1 \| 67% \| 0.47 \| 3 \| Low (investigate) \|
134
135	### Recommendations
136	[Based on regressions, low-reliability evals, and threshold failures]
137	```
138
139	### Step 6: Threshold Gating
140
141	If any threshold fails:
142	1. Flag as FAIL with specific threshold details
143	2. List which evals caused the failure
144	3. Recommend running `/skill-forge evolve` to address issues
145	4. Do NOT approve for publish until thresholds pass
146
147	## Error Handling
148
149	- Flaky trials: If a trial times out or crashes, exclude it from variance calculation and note `"trials_completed"` vs `"trials_requested"` in per-eval results
150	- Insufficient trials: If fewer than 2 trials complete for an eval, flag variance as `"unreliable"` in the report
151	- Missing baseline: If baseline runs fail entirely, report with-skill results

People who liked this also viewed

6 picks

Type