benchmark-summary

Name: benchmark-summary
Author: RConsortium

88stars20forks

How to install benchmark-summary?

rconsortium/pharma-skills/benchmark-summary

$npx -y skills add RConsortium/pharma-skills --skill benchmark-summary

Installs into the current project.

Summary

Generate a combined benchmark analysis for the group-sequential-design skill by reading all benchmark GitHub issues, selecting the latest completed run per issue, and producing a structured three-section report (summary table + overall scorecard + failure pattern analysis). Use t

Files · 1

View on GitHub

SKILL.md

1	# Benchmark Summary Skill
2
3	Produce the combined benchmark analysis document for `RConsortium/pharma-skills`. This involves reading benchmark results that live as comments on GitHub issues, selecting the right run for each, and synthesizing them into a structured report.
4
5	Repository: `RConsortium/pharma-skills`
6	Output file: `benchmark_analysis_YYYY-MM-DD.md` in the repo root
7	GitHub issue: post to `RConsortium/pharma-skills` after saving locally
8
9	---
10
11	## Step 1 — Discover all benchmark issues
12
13	List every open issue with a `[benchmark]` title prefix:
14
15	```bash
16	gh issue list --repo RConsortium/pharma-skills --limit 100 --json number,title,state \
17	\| jq '[.[] \| select(.title \| startswith("[benchmark]"))]'
18	```
19
20	Collect the issue numbers. Also fetch non-prefixed benchmark issues that are known eval cases (e.g. issues identified in prior summaries). When in doubt, fetch the issue and check whether it contains an "Automated Benchmark Results" comment.
21
22	---
23
24	## Step 2 — Fetch results for each issue
25
26	For each issue number, fetch all comments:
27
28	```bash
29	gh issue view <N> --repo RConsortium/pharma-skills --comments
30	```
31
32	Each benchmark result comment contains a Run Metadata table with:
33	- `Run date` — use this to order runs chronologically
34	- `Model` — collapse as described in Step 3
35	- A Scorecard table with With Skill / Without Skill scores
36	- A Verdict paragraph
37
38	Collect all result comments per issue. There may be multiple runs (different models or re-runs).
39
40	---
41
42	## Step 3 — Select the latest completed run per issue
43
44	Apply these rules in order to choose one run per issue:
45
46	Exclude a run if any of these apply:
47	- The scorecard note says "Partial run", "timeout", "rate-limit", "hit limit", or similar
48	- Both agents produced no output files
49	- The run was terminated before either agent finished
50	- The comment explicitly notes the run was superseded by a later one
51
52	From the remaining completed runs, pick the most recent by `Run date`.
53
54	Model name normalisation — collapse version suffixes for display:
55	- `claude-sonnet-4-6`, `claude-sonnet-4-7` → Claude Sonnet
56	- `claude-opus-4-5`, `claude-opus-4-7` → Claude Opus
57	- `claude-haiku-` → Claude Haiku*
58	- `gemini--flash` → Gemini Flash
59	- `gemini--pro` → Gemini Pro
60
61	---
62
63	## Step 4 — Build the three-section document
64
65	Follow the exact format below for the output document.
66
67	### Selection rules
68
69	- One row per benchmark issue. Use the latest completed run for each issue — ignore runs marked partial, timeout, rate-limit hit, or where both agents produced no output.
70	- Do not treat `claude-sonnet-4-7` as a different model from `claude-sonnet-4-6`; list both simply as "Claude Sonnet". Similarly collapse minor version suffixes (e.g. `claude-opus-4-7` → "Claude Opus").
71	- For each issue, record: run date, model, with-skill score (fraction + %), without-skill score, and a one-sentence verdict.
72
73	### Section 1 — Benchmark Summary Table
74
75	```markdown
76	## Benchmark Summary: Latest Completed Run per Issue (Skill vs No Skill)
77
78	\| Issue \| Scenario \| Run Date \| Model \| With Skill \| Without Skill \| Verdict \|
79	\|-------\|----------\|----------\|-------\|-----------\|--------------\|---------\|
80	\| #N \| <short description> \| YYYY-MM-DD \| <model> \| X% (n/d) \| Y% (n/d) \| ✅/❌/➕ <one sentence> \|
81	```
82
83	Verdict icons: ✅ = skill wins, ❌ = no-skill wins, ➕ = tie.
84
85	For no-skill wins, always add a parenthetical explaining whether it is a skill scope gap (wrong framework applied) or an orchestration/environment bug (not a content failure).
86
87	At the bottom of the table, add a note listing any excluded runs and why (partial, timeout, no output).
88
89	### Section 2 — Overall Scorecard
90
91	```markdown
92	## Overall Scorecard
93
94	\| \| With Skill \| Without Skill \|
95	\|--\|------------\|--------------\|
96	\| Wins \| N \| N \|
97	\| Ties \| N \| N \|
98	\| Avg score (excl. trivial) \| X% \| Y% \|
99
100	The N no-skill wins are all structural skill gaps, not base model superiority:
101
102	\| Issue \| No-skill win reason \|
103	\|-------\|-------------------\|
104	\| #N \| <one line> \|
105
106	What the benchmarks confirm the skill does well:
107	- <bullet per consistent value driver>
108	```
109
110	Exclude issues with a trivial assertion (e.g. a dry-run with a single sanity-check assertion) from the average score calculation; note which issues were excluded.
111
112	### Section 3 — Failure Pattern Analysis
113
114	For each failure pattern, use this template:

People who liked this also viewed

6 picks

Type

benchmark-summary

88stars20forks

View on GitHub

How to install benchmark-summary?

rconsortium/pharma-skills/benchmark-summary

$npx -y skills add RConsortium/pharma-skills --skill benchmark-summary

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Benchmark Summary Skill
2
3	Produce the combined benchmark analysis document for `RConsortium/pharma-skills`. This involves reading benchmark results that live as comments on GitHub issues, selecting the right run for each, and synthesizing them into a structured report.
4
5	Repository: `RConsortium/pharma-skills`
6	Output file: `benchmark_analysis_YYYY-MM-DD.md` in the repo root
7	GitHub issue: post to `RConsortium/pharma-skills` after saving locally
8
9	---
10
11	## Step 1 — Discover all benchmark issues
12
13	List every open issue with a `[benchmark]` title prefix:
14
15	```bash
16	gh issue list --repo RConsortium/pharma-skills --limit 100 --json number,title,state \
17	\| jq '[.[] \| select(.title \| startswith("[benchmark]"))]'
18	```
19
20	Collect the issue numbers. Also fetch non-prefixed benchmark issues that are known eval cases (e.g. issues identified in prior summaries). When in doubt, fetch the issue and check whether it contains an "Automated Benchmark Results" comment.
21
22	---
23
24	## Step 2 — Fetch results for each issue
25
26	For each issue number, fetch all comments:
27
28	```bash
29	gh issue view <N> --repo RConsortium/pharma-skills --comments
30	```
31
32	Each benchmark result comment contains a Run Metadata table with:
33	- `Run date` — use this to order runs chronologically
34	- `Model` — collapse as described in Step 3
35	- A Scorecard table with With Skill / Without Skill scores
36	- A Verdict paragraph
37
38	Collect all result comments per issue. There may be multiple runs (different models or re-runs).
39
40	---
41
42	## Step 3 — Select the latest completed run per issue
43
44	Apply these rules in order to choose one run per issue:
45
46	Exclude a run if any of these apply:
47	- The scorecard note says "Partial run", "timeout", "rate-limit", "hit limit", or similar
48	- Both agents produced no output files
49	- The run was terminated before either agent finished
50	- The comment explicitly notes the run was superseded by a later one
51
52	From the remaining completed runs, pick the most recent by `Run date`.
53
54	Model name normalisation — collapse version suffixes for display:
55	- `claude-sonnet-4-6`, `claude-sonnet-4-7` → Claude Sonnet
56	- `claude-opus-4-5`, `claude-opus-4-7` → Claude Opus
57	- `claude-haiku-` → Claude Haiku*
58	- `gemini--flash` → Gemini Flash
59	- `gemini--pro` → Gemini Pro
60
61	---
62
63	## Step 4 — Build the three-section document
64
65	Follow the exact format below for the output document.
66
67	### Selection rules
68
69	- One row per benchmark issue. Use the latest completed run for each issue — ignore runs marked partial, timeout, rate-limit hit, or where both agents produced no output.
70	- Do not treat `claude-sonnet-4-7` as a different model from `claude-sonnet-4-6`; list both simply as "Claude Sonnet". Similarly collapse minor version suffixes (e.g. `claude-opus-4-7` → "Claude Opus").
71	- For each issue, record: run date, model, with-skill score (fraction + %), without-skill score, and a one-sentence verdict.
72
73	### Section 1 — Benchmark Summary Table
74
75	```markdown
76	## Benchmark Summary: Latest Completed Run per Issue (Skill vs No Skill)
77
78	\| Issue \| Scenario \| Run Date \| Model \| With Skill \| Without Skill \| Verdict \|
79	\|-------\|----------\|----------\|-------\|-----------\|--------------\|---------\|
80	\| #N \| <short description> \| YYYY-MM-DD \| <model> \| X% (n/d) \| Y% (n/d) \| ✅/❌/➕ <one sentence> \|
81	```
82
83	Verdict icons: ✅ = skill wins, ❌ = no-skill wins, ➕ = tie.
84
85	For no-skill wins, always add a parenthetical explaining whether it is a skill scope gap (wrong framework applied) or an orchestration/environment bug (not a content failure).
86
87	At the bottom of the table, add a note listing any excluded runs and why (partial, timeout, no output).
88
89	### Section 2 — Overall Scorecard
90
91	```markdown
92	## Overall Scorecard
93
94	\| \| With Skill \| Without Skill \|
95	\|--\|------------\|--------------\|
96	\| Wins \| N \| N \|
97	\| Ties \| N \| N \|
98	\| Avg score (excl. trivial) \| X% \| Y% \|
99
100	The N no-skill wins are all structural skill gaps, not base model superiority:
101
102	\| Issue \| No-skill win reason \|
103	\|-------\|-------------------\|
104	\| #N \| <one line> \|
105
106	What the benchmarks confirm the skill does well:
107	- <bullet per consistent value driver>
108	```
109
110	Exclude issues with a trivial assertion (e.g. a dry-run with a single sanity-check assertion) from the average score calculation; note which issues were excluded.
111
112	### Section 3 — Failure Pattern Analysis
113
114	For each failure pattern, use this template:

People who liked this also viewed

6 picks

Type