calibrate

Name: calibrate
Author: Borda

23stars3forks

How to install calibrate?

borda/ai-rig/calibrate

$npx -y skills add Borda/AI-Rig --skill calibrate

Installs into the current project.

Summary

Calibration testing for agents and skills. Generates synthetic problems with known outcomes (quasi-ground-truth), runs targets against them, measures recall, precision, confidence calibration — reveals whether self-reported confidence scores track actual quality.

Files · 1

View on GitHub

SKILL.md

1	<objective>
2
3	Validate agents and skills by measuring outputs against synthetic problems with defined ground truth. Primary signal: calibration bias — gap between self-reported confidence and actual recall. Well-calibrated agent reports 0.9 when it finds ~90% of issues. Miscalibrated: reports 0.9, finds 60%.
4
5	Calibration data drives improvement loop: systematic gaps → instruction updates; persistent overconfidence → adjusted re-run thresholds in MEMORY.md.
6
7	NOT for: static routing overlap analysis (use /foundry:audit); manually reviewing skill output quality (use /develop:review (requires `develop` plugin)).
8
9	</objective>
10
11	<inputs>
12
13	- $ARGUMENTS: parse `--flags` first, then resolve remaining tokens as scope targets
14
15	Flags (order independent):
16	- `--fast` — 3 problems per target (default when neither pace flag passed)
17	- `--full` — 10 problems per target; mutually exclusive with `--fast`
18	- `--ab-test` — also run `general-purpose` baseline and report delta metrics; requires benchmark (default `--fast` if no pace flag); mutually exclusive with `--apply`
19	- `--apply` — apply proposals: with `--fast`/`--full`: run benchmark then immediately apply; without pace flag: skip benchmark, apply proposals from most recent past run; mutually exclusive with `--ab-test`
20	- `--skip-gate` — suppress follow-up gate; for programmatic callers
21	- `--local` — resolve target agent/skill files from source tree (`plugins/*/`) instead of installed plugin cache; for plugin-dev workflows where local edits aren't yet installed; sets `LOCAL_MODE=true` in all pipeline spawns
22
23	Mutual exclusion validation (check before any work):
24	- `--ab-test` + `--apply` together → hard error: "`--ab-test` and `--apply` are mutually exclusive. Pass one or neither."
25	- `--fast` + `--full` together → hard error: "Pass `--fast` or `--full`, not both."
26	- `--ab-test` without pace flag → default `--fast` silently (no error)
27
28	Unsupported flag check — after all supported flags extracted (`--fast`, `--full`, `--ab-test`, `--apply`, `--skip-gate`, `--local`, `--keep`), scan `$ARGUMENTS` for remaining `--<token>` tokens. If found: print `! Unknown flag(s): \`--<token>\`. Supported: \`--fast\`, \`--full\`, \`--ab-test\`, \`--apply\`, \`--skip-gate\`, \`--local\`, \`--keep\`.` then invoke `AskUserQuestion` — (a) Abort (stop, re-invoke with correct flags) · (b) Continue ignoring (skip unknown flags, proceed). On Abort: stop.
29
30	Legacy positional tokens (`ab`, `apply`, `fast`, `full`) — hard error: print migration hint and stop. Example: "`ab` removed — use `--ab-test` flag: `/calibrate curator --ab-test`."
31
32	Scope tokens (positional, space-separated — defaults to `all`):
33	- `all` — all agents + relevant skills + routing + communication + all rules
34	- `agents` — all agents only (full agent list in `modes/agents.md`)
35	- `skills` — calibratable skills only (`/audit` and others per `modes/skills.md`; `/oss:review` (requires `oss` plugin) excluded — requires live GitHub PR)
36	- `routing` — routing accuracy test: measures how accurately `general-purpose` orchestrator selects correct `subagent_type` for synthetic task prompts (not per-agent quality benchmark; included in `all`)
37	- `communication` — handover + team protocol compliance: runs `foundry:curator` against synthetic agent responses and team transcripts with injected protocol violations (missing JSON envelope, missing `summary`, AgentSpeak v2 breaches); included in `all`
38	- `rules` — rule adherence test: for each global rule file (no `paths:`) and each path-scoped rule when matching file is in context, generates synthetic tasks that should trigger rule's key directives, measures whether `general-purpose` agent with rule loaded correctly applies them; reports rules that are ignored, misapplied, or redundant; included in `all`
39	- `plugins` — all agents + calibratable skills from all `plugins/*/` directories (union of all plugin-namespaced agents and calibratable skills)
40	- `<plugin-name>` — tier 2: bare plugin directory name (e.g. `oss`, `foundry`, `research`, `develop`) auto-resolved when token matches `plugins/<name>/` directory; calibrates all agents + calibratable skills in that plugin
41	- `<agent-name>` — tier 3: single agent (e.g., `foundry:sw-engineer`); also accepts bare name (e.g. `sw-engineer`) and resolves via `plugins/*/agents/<name>.md`
42	- `/foundry:audit` — single s

People who liked this also viewed

6 picks

Type

calibrate

23stars3forks

View on GitHub

How to install calibrate?

borda/ai-rig/calibrate

$npx -y skills add Borda/AI-Rig --skill calibrate

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	<objective>
2
3	Validate agents and skills by measuring outputs against synthetic problems with defined ground truth. Primary signal: calibration bias — gap between self-reported confidence and actual recall. Well-calibrated agent reports 0.9 when it finds ~90% of issues. Miscalibrated: reports 0.9, finds 60%.
4
5	Calibration data drives improvement loop: systematic gaps → instruction updates; persistent overconfidence → adjusted re-run thresholds in MEMORY.md.
6
7	NOT for: static routing overlap analysis (use /foundry:audit); manually reviewing skill output quality (use /develop:review (requires `develop` plugin)).
8
9	</objective>
10
11	<inputs>
12
13	- $ARGUMENTS: parse `--flags` first, then resolve remaining tokens as scope targets
14
15	Flags (order independent):
16	- `--fast` — 3 problems per target (default when neither pace flag passed)
17	- `--full` — 10 problems per target; mutually exclusive with `--fast`
18	- `--ab-test` — also run `general-purpose` baseline and report delta metrics; requires benchmark (default `--fast` if no pace flag); mutually exclusive with `--apply`
19	- `--apply` — apply proposals: with `--fast`/`--full`: run benchmark then immediately apply; without pace flag: skip benchmark, apply proposals from most recent past run; mutually exclusive with `--ab-test`
20	- `--skip-gate` — suppress follow-up gate; for programmatic callers
21	- `--local` — resolve target agent/skill files from source tree (`plugins/*/`) instead of installed plugin cache; for plugin-dev workflows where local edits aren't yet installed; sets `LOCAL_MODE=true` in all pipeline spawns
22
23	Mutual exclusion validation (check before any work):
24	- `--ab-test` + `--apply` together → hard error: "`--ab-test` and `--apply` are mutually exclusive. Pass one or neither."
25	- `--fast` + `--full` together → hard error: "Pass `--fast` or `--full`, not both."
26	- `--ab-test` without pace flag → default `--fast` silently (no error)
27
28	Unsupported flag check — after all supported flags extracted (`--fast`, `--full`, `--ab-test`, `--apply`, `--skip-gate`, `--local`, `--keep`), scan `$ARGUMENTS` for remaining `--<token>` tokens. If found: print `! Unknown flag(s): \`--<token>\`. Supported: \`--fast\`, \`--full\`, \`--ab-test\`, \`--apply\`, \`--skip-gate\`, \`--local\`, \`--keep\`.` then invoke `AskUserQuestion` — (a) Abort (stop, re-invoke with correct flags) · (b) Continue ignoring (skip unknown flags, proceed). On Abort: stop.
29
30	Legacy positional tokens (`ab`, `apply`, `fast`, `full`) — hard error: print migration hint and stop. Example: "`ab` removed — use `--ab-test` flag: `/calibrate curator --ab-test`."
31
32	Scope tokens (positional, space-separated — defaults to `all`):
33	- `all` — all agents + relevant skills + routing + communication + all rules
34	- `agents` — all agents only (full agent list in `modes/agents.md`)
35	- `skills` — calibratable skills only (`/audit` and others per `modes/skills.md`; `/oss:review` (requires `oss` plugin) excluded — requires live GitHub PR)
36	- `routing` — routing accuracy test: measures how accurately `general-purpose` orchestrator selects correct `subagent_type` for synthetic task prompts (not per-agent quality benchmark; included in `all`)
37	- `communication` — handover + team protocol compliance: runs `foundry:curator` against synthetic agent responses and team transcripts with injected protocol violations (missing JSON envelope, missing `summary`, AgentSpeak v2 breaches); included in `all`
38	- `rules` — rule adherence test: for each global rule file (no `paths:`) and each path-scoped rule when matching file is in context, generates synthetic tasks that should trigger rule's key directives, measures whether `general-purpose` agent with rule loaded correctly applies them; reports rules that are ignored, misapplied, or redundant; included in `all`
39	- `plugins` — all agents + calibratable skills from all `plugins/*/` directories (union of all plugin-namespaced agents and calibratable skills)
40	- `<plugin-name>` — tier 2: bare plugin directory name (e.g. `oss`, `foundry`, `research`, `develop`) auto-resolved when token matches `plugins/<name>/` directory; calibrates all agents + calibratable skills in that plugin
41	- `<agent-name>` — tier 3: single agent (e.g., `foundry:sw-engineer`); also accepts bare name (e.g. `sw-engineer`) and resolves via `plugins/*/agents/<name>.md`
42	- `/foundry:audit` — single s

People who liked this also viewed

6 picks

Type