benchmark-runner

Name: benchmark-runner
Author: RConsortium

88stars20forks

How to install benchmark-runner?

rconsortium/pharma-skills/benchmark-runner

$npx -y skills add RConsortium/pharma-skills --skill benchmark-runner

Installs into the current project.

Summary

Auto-discover all skills with evals in RConsortium/pharma-skills, benchmark each with vs. without skill using matched isolated sessions, and post scored results to the linked GitHub issue. Use whenever someone says "run benchmarks", "compare skill performance", "eval the skills",

Files · 1

View on GitHub

SKILL.md

1	# Skill Benchmark Runner
2
3	Benchmark every evaluation case in the `_automation/evals/` directory of the `RConsortium/pharma-skills` repository. Each routine invocation is one of two short phases (~20 min each). The routine inspects GitHub issue comments on startup to decide which phase to execute — no configuration needed, no commits required, no repo write access required from the human user.
4
5	Repository: `RConsortium/pharma-skills` (https://github.com/RConsortium/pharma-skills)
6
7	---
8
9	## Model Selection — Series Label and Sub-Agent Model
10
11	`{CURRENT_MODEL_NAME}` throughout this skill is the benchmark series label, fixed to the literal string `Claude Routine`. Use it verbatim everywhere it appears: `get_next_eval.py --model`, `record_run_result.py --model`, the marker JSON (`"model"`), and the report metadata. The scripts normalise this label (lowercase, strip spaces/punctuation) for deduplication, so all runs group into one consistent series.
12
13	Sub-agent model:
14
15	- Agent A and Agent B are launched without an explicit `--model` flag (`claude -p ...`), so each sub-agent inherits this session's default model — the same model orchestrating the routine. This keeps the measurement clean (bare model ± skill, identical model on both sides) without ever writing a concrete model ID into a repository artifact. Never hardcode a concrete model ID anywhere in this skill or its outputs.
16	- The prompt files constructed in Step 2 / Step 6 and the `CLAUDE_CODE_MAX_OUTPUT_TOKENS` setting are the only information passed from the parent session to a sub-agent. Do not forward any other parent context: no conversation history, no additional environment variables, no eval assertions, no scoring prompt, no blinded map. This prevents the orchestrator's context from leaking into either candidate.
17
18	---
19
20	## Routine Setup (one-time)
21
22	Create a single routine at [claude.ai/code/routines](https://claude.ai/code/routines):
23
24	\| Field \| Value \|
25	\|---\|---\|
26	\| Prompt \| `Read _automation/benchmark-runner/SKILL.md and execute.` \|
27	\| Repository \| `RConsortium/pharma-skills` \|
28	\| Schedule \| `0 1,6 * * *` (1 AM and 6 AM UTC — 5 h gap, matches rolling usage window) \|
29
30	That is all. The skill determines its own phase on every invocation.
31
32	---
33
34	## GitHub Access — Use Whichever Method Works
35
36	Throughout this skill you will read issue comments, post issue comments, and create release assets. Use whatever method is available in your environment — pick the one that works without prompting:
37
38	\| Method \| Best when \| Notes \|
39	\|---\|---\|---\|
40	\| `mcp__github__*` MCP tools \| Running inside Claude Code with the GitHub MCP server \| No token required; preferred when available \|
41	\| `gh` CLI (`gh issue view`, `gh release upload`, etc.) \| Running locally with `gh` authenticated \| Concise, supports all operations \|
42	\| REST API via `curl` \| Anywhere with `GH_TOKEN` / `GITHUB_TOKEN` set \| Universal fallback; use for release-asset upload (no MCP equivalent) \|
43	\| Provider-specific GitHub tools (Codex, Gemini, etc.) \| Running under another agent CLI \| Use whatever the host provides \|
44
45	Reason about which method to use; do not enforce a rigid order. If one fails, try another. Always confirm the operation succeeded (e.g., the comment URL came back, the asset was uploaded) before continuing.
46
47	For release-asset upload there is currently no MCP tool — use `gh release upload` or `curl` POST to the upload URL.
48
49	---
50
51	## Phase Detection — Run Before Any Other Step
52
53	Scan all benchmark eval issues to find any that are waiting for Phase 2 (Agent B + scoring) for the model you are running:
54
55	1. List all eval files: `ls _automation/evals/.json` — extract each `id` field (e.g. `github-issue-27` → issue #27*).
56
57	2. For each issue number, fetch comments using whichever GitHub access method is available (see above). Scan each comment body for a `<!-- BENCHMARK_PARTIAL:` marker.
58
59	3. Filter and evaluate each `BENCHMARK_PARTIAL` marker found:
60	- Skip if `state.model` does not match `{CURRENT_MODEL_NAME}`. This is critical: a partial run by another user on a different model belongs to that user. Only pick up partials matching your current model.
61	- Skip if a later comment on the same issue contains a matching `<!-- BENCHMARK_COMPLETE: {"eval_id":"{same}","model":"{same}"` — Phase 2 already finished for that combination.
62	- Otherwise → Phase 2 candidate. Extract the JSON from the marker (see format below) and note the partial comment `id`.
63
64	4. Decision:
65	- One or more Phase 2 candidates found → pick the oldest (earliest `created_at`) → enter Phase 2 with that state.
66	-

People who liked this also viewed

6 picks

Type

benchmark-runner

88stars20forks

View on GitHub

How to install benchmark-runner?

rconsortium/pharma-skills/benchmark-runner

$npx -y skills add RConsortium/pharma-skills --skill benchmark-runner

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Skill Benchmark Runner
2
3	Benchmark every evaluation case in the `_automation/evals/` directory of the `RConsortium/pharma-skills` repository. Each routine invocation is one of two short phases (~20 min each). The routine inspects GitHub issue comments on startup to decide which phase to execute — no configuration needed, no commits required, no repo write access required from the human user.
4
5	Repository: `RConsortium/pharma-skills` (https://github.com/RConsortium/pharma-skills)
6
7	---
8
9	## Model Selection — Series Label and Sub-Agent Model
10
11	`{CURRENT_MODEL_NAME}` throughout this skill is the benchmark series label, fixed to the literal string `Claude Routine`. Use it verbatim everywhere it appears: `get_next_eval.py --model`, `record_run_result.py --model`, the marker JSON (`"model"`), and the report metadata. The scripts normalise this label (lowercase, strip spaces/punctuation) for deduplication, so all runs group into one consistent series.
12
13	Sub-agent model:
14
15	- Agent A and Agent B are launched without an explicit `--model` flag (`claude -p ...`), so each sub-agent inherits this session's default model — the same model orchestrating the routine. This keeps the measurement clean (bare model ± skill, identical model on both sides) without ever writing a concrete model ID into a repository artifact. Never hardcode a concrete model ID anywhere in this skill or its outputs.
16	- The prompt files constructed in Step 2 / Step 6 and the `CLAUDE_CODE_MAX_OUTPUT_TOKENS` setting are the only information passed from the parent session to a sub-agent. Do not forward any other parent context: no conversation history, no additional environment variables, no eval assertions, no scoring prompt, no blinded map. This prevents the orchestrator's context from leaking into either candidate.
17
18	---
19
20	## Routine Setup (one-time)
21
22	Create a single routine at [claude.ai/code/routines](https://claude.ai/code/routines):
23
24	\| Field \| Value \|
25	\|---\|---\|
26	\| Prompt \| `Read _automation/benchmark-runner/SKILL.md and execute.` \|
27	\| Repository \| `RConsortium/pharma-skills` \|
28	\| Schedule \| `0 1,6 * * *` (1 AM and 6 AM UTC — 5 h gap, matches rolling usage window) \|
29
30	That is all. The skill determines its own phase on every invocation.
31
32	---
33
34	## GitHub Access — Use Whichever Method Works
35
36	Throughout this skill you will read issue comments, post issue comments, and create release assets. Use whatever method is available in your environment — pick the one that works without prompting:
37
38	\| Method \| Best when \| Notes \|
39	\|---\|---\|---\|
40	\| `mcp__github__*` MCP tools \| Running inside Claude Code with the GitHub MCP server \| No token required; preferred when available \|
41	\| `gh` CLI (`gh issue view`, `gh release upload`, etc.) \| Running locally with `gh` authenticated \| Concise, supports all operations \|
42	\| REST API via `curl` \| Anywhere with `GH_TOKEN` / `GITHUB_TOKEN` set \| Universal fallback; use for release-asset upload (no MCP equivalent) \|
43	\| Provider-specific GitHub tools (Codex, Gemini, etc.) \| Running under another agent CLI \| Use whatever the host provides \|
44
45	Reason about which method to use; do not enforce a rigid order. If one fails, try another. Always confirm the operation succeeded (e.g., the comment URL came back, the asset was uploaded) before continuing.
46
47	For release-asset upload there is currently no MCP tool — use `gh release upload` or `curl` POST to the upload URL.
48
49	---
50
51	## Phase Detection — Run Before Any Other Step
52
53	Scan all benchmark eval issues to find any that are waiting for Phase 2 (Agent B + scoring) for the model you are running:
54
55	1. List all eval files: `ls _automation/evals/.json` — extract each `id` field (e.g. `github-issue-27` → issue #27*).
56
57	2. For each issue number, fetch comments using whichever GitHub access method is available (see above). Scan each comment body for a `<!-- BENCHMARK_PARTIAL:` marker.
58
59	3. Filter and evaluate each `BENCHMARK_PARTIAL` marker found:
60	- Skip if `state.model` does not match `{CURRENT_MODEL_NAME}`. This is critical: a partial run by another user on a different model belongs to that user. Only pick up partials matching your current model.
61	- Skip if a later comment on the same issue contains a matching `<!-- BENCHMARK_COMPLETE: {"eval_id":"{same}","model":"{same}"` — Phase 2 already finished for that combination.
62	- Otherwise → Phase 2 candidate. Extract the JSON from the marker (see format below) and note the partial comment `id`.
63
64	4. Decision:
65	- One or more Phase 2 candidates found → pick the oldest (earliest `created_at`) → enter Phase 2 with that state.
66	-

People who liked this also viewed

6 picks

Type