judge
starsforks
How to install judge?
borda/ai-rig/judge
$npx -y skills add Borda/AI-Rig --skill judgeResearch-supervisor review of program.md — validates experimental methodology (hypothesis clarity, measurement validity, control adequacy, scope, strategy fit), emits APPROVED / NEEDS-REVISION / BLOCKED verdict before expensive run loop.
SKILL.md
| 1 | <objective> |
| 2 | |
| 3 | Research-supervisor review of `program.md` — validates experimental methodology, emits APPROVED / NEEDS-REVISION / BLOCKED verdict before expensive run loop. Read-only; never modifies code or state. |
| 4 | |
| 5 | NOT for: running experiments (use `/research:run`); designing hypotheses (use `research:scientist` agent); config quality (`/foundry:audit` (requires `foundry` plugin)). |
| 6 | </objective> |
| 7 | |
| 8 | <compaction> |
| 9 | |
| 10 | Key boundary: end of J3 — methodology and scientific review agents complete, output files written; before J4 validation and J6 verdict. |
| 11 | Preserve: RUN_DIR (TMPDIR key), PROGRAM_PATH (TMPDIR key), methodology.md path, scientific-review.md path, SKIP_VALIDATION flag, BRANCH. |
| 12 | Clear at J1 start (stale prior run) and at end of J6 after terminal summary printed. |
| 13 | |
| 14 | </compaction> |
| 15 | |
| 16 | <workflow> |
| 17 | |
| 18 | <!-- Agent resolution: see _RESEARCH_SHARED/agent-resolution.md --> |
| 19 | |
| 20 | ## Agent Resolution |
| 21 | |
| 22 | **Agent resolution**: load and follow the protocol below. Contains foundry check + fallback table. If foundry not installed: use table to substitute each `foundry:X` with `general-purpose`. Agents: `foundry:solution-architect`, `research:scientist`. |
| 23 | ```bash |
| 24 | # loads: compaction-contract.md |
| 25 | _RESEARCH_SHARED=$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_research}/bin/resolve_shared.py" 2>/dev/null) # timeout: 5000 |
| 26 | [ -z "$_RESEARCH_SHARED" ] && { echo "! Plugin path resolution failed — ensure research plugin installed and CLAUDE_PLUGIN_ROOT set, or invoke from project root."; exit 1; } |
| 27 | cat "$_RESEARCH_SHARED/agent-resolution.md" |
| 28 | ``` |
| 29 | |
| 30 | | Agent | Fallback if absent | |
| 31 | | --- | --- | |
| 32 | | `foundry:solution-architect` | `general-purpose` (methodology review quality reduced — **⚠ general-purpose agent may not emit `methodology_rating` in required format; verdict defaults to NEEDS-REVISION**) | |
| 33 | | `research:scientist` | `general-purpose` (scientific rigor review quality reduced — **⚠ general-purpose agent may not emit `scientific_rating`; verdict defaults to NEEDS-REVISION**) | |
| 34 | |
| 35 | ## Judge Mode (Steps J1–J6) |
| 36 | |
| 37 | Triggered by `judge` or `judge <file.md>`. |
| 38 | |
| 39 | **Task tracking**: create tasks for J1, J2, J3, J4, J5a, J5b, J6 at start — before any tool calls. (J5a = Codex adversarial review; J5b = resolve rating source.) |
| 40 | |
| 41 | ## Step J1: Locate and parse program.md |
| 42 | |
| 43 | **Flag parsing** (first): |
| 44 | |
| 45 | ```bash |
| 46 | SKIP_VALIDATION=false |
| 47 | [[ "$ARGUMENTS" == *"--skip-validation"* ]] && SKIP_VALIDATION=true |
| 48 | ARGUMENTS="${ARGUMENTS/--skip-validation/}" |
| 49 | ARGUMENTS="${ARGUMENTS#"${ARGUMENTS%%[![:space:]]*}"}" # trim leading whitespace |
| 50 | ``` |
| 51 | |
| 52 | ```bash |
| 53 | # Extract --keep quoted value (compaction-contract.md §keep semantics) |
| 54 | KEEP_ITEMS="" |
| 55 | if [[ "$ARGUMENTS" =~ --keep[[:space:]]\"([^\"]+)\" ]]; then |
| 56 | KEEP_ITEMS="${BASH_REMATCH[1]}" |
| 57 | fi |
| 58 | export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}" |
| 59 | # Clear stale contract from any prior incomplete run (compaction-contract.md §Lifecycle) |
| 60 | rm -f .temp/state/skill-contract.md # timeout: 5000 |
| 61 | echo "${KEEP_ITEMS:-}" > "${TMPDIR:-/tmp}/judge-keep-items-${CSID}" # persist for J3 contract write |
| 62 | ``` |
| 63 | |
| 64 | **Unsupported flag check**: load and follow the protocol below. Supported flags for this skill: `--skip-validation`, `--keep`. |
| 65 | ```bash |
| 66 | # loads: unsupported-flag-protocol.md |
| 67 | _RESEARCH_SHARED=$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_research}/bin/resolve_shared.py" 2>/dev/null) # timeout: 5000 |
| 68 | cat "$_RESEARCH_SHARED/unsupported-flag-protocol.md" |
| 69 | ``` |
| 70 | |
| 71 | **Input resolution** (priority order): |
| 72 | |
| 73 | 1. Explicit argument: `/research:judge path/to/plan.md` |
| 74 | 2. Auto-detect: `program.md` at project root |
| 75 | 3. Latest state: scan `.experiments/state/*/state.json` for most recent with `status: running` and non-null `program_file` field |
| 76 | 4. If nothing found: stop with error: |
| 77 | ```text |
| 78 | No program.md found. Run /research:plan <goal> first, or provide a path: /research:judge <path.md> |
| 79 | ``` |
| 80 | |
| 81 | **Parsing** — find `## <Section>` headings in program.md, extract first fenced code block per section, parse as `key: value` lines, warn on unrecognized keys. `--skip-validation` and `colab_hw` judge-specific, extracted independently. |
| 82 | |
| 83 | **Placeholder substitution** — after parsing, apply same substitution as R1: resolve all `{field_name}` tokens in `metric_cmd` and `guard_cmd` using `## Config` fields, fallback to declared default. No `clarification_prompt` in judge — skip clarification-override step. |
| 84 | |
| 85 | Extract `<program_title>` from `# Program: <title>` line for reports (fallback `# Campaign: <title>` for legacy files). |
| 86 | |
| 87 | ## Step J2: Completeness audit |
| 88 | |
| 89 | Check 12 items. Produce findings list with severity. Each finding has: `id`, `check`, |
People who liked this also viewed
6 picksType
- requesting-code-reviewUse when completing tasks, implementing major features, or before merging to verify work meets requirementsSkillsJul 2026
- code-reviewSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- receiving-code-reviewUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performativeSkillsJul 2026
- baoyu-format-markdownFormats plain text or markdown files with frontmatter, titles, summaries, headings, bold, lists, and code blocks. Use when user asks to "format markdown", "beautify article", "add formatting", or imprSkillsJul 2026
- code-review-excellenceMaster effective code review practices to provide constructive feedback, catch bugs early, and foster knowledge sharing while maintaining team morale. Use when reviewing pull requests, establishing reSkillsJul 2026
- code-review-and-qualityConducts multi-axis code review. Use before merging any change. Use when reviewing code written by yourself, another agent, or a human. Use when you need to assess code quality across multiple dimensiSkillsJul 2026
