issue-to-eval
starsforks
How to install issue-to-eval?
rconsortium/pharma-skills/issue-to-eval
$npx -y skills add RConsortium/pharma-skills --skill issue-to-evalConverts one or more GitHub Issues into standardized benchmark data using automated scripts. Use when a user provides an issue number or URL and wants to add it to the evaluation suite.
SKILL.md
| 1 | # Issue to Eval |
| 2 | |
| 3 | Converts GitHub Issues into standardized benchmark evaluation cases (JSON) and saves them to the `_automation/evals/` directory. It automatically discovers new issues labeled `benchmark` and updates existing evaluations if the issue content has been modified. |
| 4 | |
| 5 | ## Task Flow |
| 6 | |
| 7 | ### Option A: Sync All Benchmarks (Recommended) |
| 8 | Use this to automatically identify and update all evaluations from GitHub: |
| 9 | ```bash |
| 10 | python3 _automation/issue-to-eval/scripts/sync_benchmarks.py |
| 11 | ``` |
| 12 | - Discovers all issues with the `benchmark` label. |
| 13 | - Compares each issue against the local files in `evals/`. |
| 14 | - Adds new cases and updates existing ones if the prompt or assertions have changed. |
| 15 | |
| 16 | ### Option B: Import Specific Issue |
| 17 | Use this for a one-off import or testing: |
| 18 | ```bash |
| 19 | python3 _automation/issue-to-eval/scripts/import_issue_eval.py --issue {ISSUE_NUMBER_OR_URL} |
| 20 | ``` |
| 21 | |
| 22 | ## Review Output |
| 23 | - Report the status for each issue (Success/Updated/Skipped/Error) to the user. |
| 24 | - If an issue is modified on GitHub, running the sync script will propagate those changes to the local evaluation suite. |
| 25 | |
| 26 | ## Requirements |
| 27 | - The issue MUST follow the standard benchmark template with headers: `## Skills`, optional `## Language`, `## Query`, `## Expected Output`, `## Attached Files / Input Context`, and `## Rubric Criteria (Assertions)`. |
| 28 | - `## Skills` may contain one or more skill names, one per line or comma-separated. The generated eval stores them in `target_skills`. |
| 29 | - `## Language` is optional. When present, the generated eval stores it in `language` so the benchmark runner can pass the same language constraint to both agents. |
People who liked this also viewed
6 picksType
- find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 2026
- grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026
- vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026
- improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026
- setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026
- triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026
