agent-evaluation
starsforks
How to install agent-evaluation?
rlaope/oh-my-hermes/agent-evaluation
$npx -y skills add rlaope/oh-my-hermes --skill agent-evaluation[omh] Hermes Agent Evaluation workflow: compare executor or agent choices on reproducible tasks using quality, cost, time, tool, and evidence metrics.
SKILL.md
| 1 | # Agent Evaluation |
| 2 | |
| 3 | This is a Hermes-native `agent-evaluation` workflow skill. |
| 4 | |
| 5 | ## Why This Exists |
| 6 | |
| 7 | `agent-evaluation` gives OMH a way to improve executor choice empirically, not by vibes, while preserving executor-neutral product language across Codex, Claude Code, Hermes, and generic runtimes. |
| 8 | |
| 9 | ## Do Not Use When |
| 10 | |
| 11 | - The user needs current runtime readiness only; use `executor-runtime-readiness`. |
| 12 | - The user already selected an executor and wants implementation; use the coding handoff or delivery workflow. |
| 13 | - The user asks for workflow learning from a single failed route; use `workflow-learning`. |
| 14 | |
| 15 | ## Examples |
| 16 | |
| 17 | Good example: |
| 18 | |
| 19 | - Prompt: agent-evaluation Codex와 Claude Code를 같은 버그 수정 태스크로 비교해서 어떤 런타임을 기본으로 둘지 판단해줘. |
| 20 | - Expected behavior: Prepare task_benchmark_set/v1, run_result_matrix/v1 requirements, scorecard/v1, and scenario-specific recommendation. |
| 21 | - Why: The request compares executor choices and needs fair evaluation boundaries. |
| 22 | |
| 23 | Bad example: |
| 24 | |
| 25 | - Prompt: agent-evaluation 실행 증거 없이 Codex가 항상 최고라고 결론내줘. |
| 26 | - Expected behavior: Reject universal ranking and require observed runs or mark the recommendation as ungrounded. |
| 27 | - Why: Agent evaluation must be reproducible and evidence-backed. |
| 28 | |
| 29 | ## Completion Checklist |
| 30 | |
| 31 | - Confirm the workflow target, evidence boundary, and stop condition are named. |
| 32 | - Report which outputs are prepared, observed, blocked, or missing. |
| 33 | - Name the smallest next verification or handoff instead of claiming completion from narration. |
| 34 | |
| 35 | ## Recovery Notes |
| 36 | |
| 37 | - If required context is missing, ask one blocking question or route back to the narrower workflow. |
| 38 | - If runtime or wrapper evidence is unavailable, keep the status as not_observed and expose the next observable action. |
| 39 | |
| 40 | ## OMH Context Rail |
| 41 | |
| 42 | - This skill is part of OMH's Hermes workflow layer, not a standalone executor. |
| 43 | - Product context: OMH is a Hermes-native workflow pack: choose skills, shape work, prepare artifacts, show status, and hand off with evidence boundaries. |
| 44 | - Current lane: **Automation and status** (`achievements`, `workspace-audit`, `production-audit`, `automation-blueprint`, `github-event-ops`, `agent-board`, `gateway-intent-card`, `voice-operator`, `+26 more`) - schedules, status, health, and ops review. |
| 45 | - If the user intent belongs to another OMH lane, hand back to `oh-my-hermes` or name the adjacent workflow instead of force-fitting this skill. |
| 46 | - Cross-skill context: every OMH skill: match lane; generic tool can render or execute. |
| 47 | - Generic-tool checkpoint: image->img-summary; frontend->frontend/a11y/visual-qa; paper->paper-learning; content->content-operator; media->media-input-operator; file->materials-package; search->web-research; live->live-info-operator; audit->workspace/production/security; failures->build-failure; verify->verification-gate; code->codegraph/onboarding/ultraprocess. |
| 48 | - Coverage: Every generated workflow skill carries this rail. |
| 49 | - Normal users talk to Hermes; OMH CLI is infra. |
| 50 | - Boundary: Prepared OMH routing, cards, handoffs, or artifacts are not observed execution, image generation, delivery, review, CI, merge-readiness, or merge evidence. |
| 51 | |
| 52 | ## Use When |
| 53 | |
| 54 | Use when Hermes should design or summarize a fair comparison of Codex, Claude Code, Hermes coding, or generic executors for a bounded task set. |
| 55 | |
| 56 | Strong routing signals: `agent-evaluation`, `agent evaluation`, `agent eval`, `agent benchmark`, `executor evaluation`, `executor benchmark`, `compare agents`, `compare codex claude`, `agent tournament`, `which agent is better`, `에이전트 평가`, `에이전트 비교`, `실행자 평가`, `코덱스 클로드 비교` |
| 57 | |
| 58 | ## Catalog Metadata |
| 59 | |
| 60 | Category: `operations` |
| 61 | Phase: `agent-evaluation` |
| 62 | Hermes role: `operator` |
| 63 | Quality tier: `agent-eval-gated` |
| 64 | |
| 65 | Quality bar: |
| 66 | |
| 67 | - Define tasks, rubric, isolation, budgets, and stop rules before comparing agents. |
| 68 | - Use the same inputs and success criteria across candidates unless the difference is the variable under test. |
| 69 | - Report quality, correctness, time, cost, tool coverage, verification, and review gaps separately. |
| 70 | - Recommend executor choice per scenario and confidence, not as a universal ranking. |
| 71 | |
| 72 | Handoff policy: |
| 73 | |
| 74 | Keep evaluation design and scoring in Hermes. Actual executor runs, costs, timings, tool calls, code edits, and review results must come from observed runtime or supplied artifacts. |
| 75 | |
| 76 | Required inputs: |
| 77 | |
| 78 | - candidate executors or agents |
| 79 | - task set and fixtures |
| 80 | - success criteria and scoring rubric |
| 81 | - allowed tools, budget, timebox, and isolation policy |
| 82 | - observed run artifacts when comparing completed attempts |
| 83 | |
| 84 | Expected outputs: |
| 85 | |
| 86 | - agent_eval_plan/v1 |
| 87 | - task_benchmark_set/v1 |
| 88 | - run_result_matrix/v1 when observed |
| 89 | - scorecard/v1 |
| 90 | - selection_recommendation/v1 |
| 91 | - n |
People who liked this also viewed
6 picksType
- tddSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- test-driven-developmentUse when implementing any feature or bugfix, before writing implementation codeSkillsJul 2026
- qaSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- verification-before-completionUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidenceSkillsJul 2026
- webapp-testingToolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser lSkillsJul 2026
- playwright-best-practicesUse when writing Playwright tests, fixing flaky tests, debugging failures, implementing Page Object Model, configuring CI/CD, optimizing performance, mocking APIs, handling authentication or OAuth, teSkillsJul 2026
