rstack-skill-evaluator
starsforks
How to install rstack-skill-evaluator?
rstackjs/agent-skills/rstack-skill-evaluator
$npx -y skills add rstackjs/agent-skills --skill rstack-skill-evaluatorBenchmark agent skills by generating eval cases, comparing skill-guided vs baseline runs, and recording artifacts under skills-test/{skill-name}.
SKILL.md
| 1 | # rstack-skill-evaluator |
| 2 | |
| 3 | A thin repo-specific layer on top of `skill-creator`. For workflow (Test / Improve / Benchmark modes), JSON schemas, grading, and viewer details, defer to `skill-creator`'s own SKILL.md and `references/schemas.md`. |
| 4 | |
| 5 | ## Targeting a skill |
| 6 | |
| 7 | If the user hasn't named a target, ask. Skills live under `skills/` (production) and `.agents/skills/` (internal-only). |
| 8 | |
| 9 | ## Artifact layout |
| 10 | |
| 11 | For skill `<name>`, two paths are tracked in git; everything else under `skills-test/` is gitignored: |
| 12 | |
| 13 | ```plaintext |
| 14 | +--------------------------------------+----------------------------------+ |
| 15 | | Tracked path | Purpose | |
| 16 | +--------------------------------------+----------------------------------+ |
| 17 | | skills-test/<name>/evals/evals.json | eval definitions | |
| 18 | | skills-test/<name>/report.md | human-readable run summary | |
| 19 | +--------------------------------------+----------------------------------+ |
| 20 | ``` |
| 21 | |
| 22 | Workspaces, raw run outputs, and fixtures may live anywhere — under `skills-test/<name>/` or an OS scratch dir — as long as `report.md` references the path so a reader can find them. |
| 23 | |
| 24 | ## `report.md` |
| 25 | |
| 26 | Committed Markdown summary of the latest run. At minimum cover: setup (model, skill version / commit ref, date), aggregate pass rate / tokens / wall time for `with_skill` vs `without_skill`, per-eval breakdown, and pointers to the raw artifacts. |
People who liked this also viewed
6 picksType
- find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 2026
- grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026
- vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026
- improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026
- setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026
- triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026
