terminal-agent-benchmark
starsforks
How to install terminal-agent-benchmark?
nowledge-co/con-terminal/terminal-agent-benchmark
$npx -y skills add nowledge-co/con-terminal --skill terminal-agent-benchmarkRun and maintain Con's terminal-agent benchmark against a live app session. Use when validating con-cli, SSH workspace reuse, tmux awareness, agent-target preparation, or when collecting benchmark evidence for regressions and release notes.
SKILL.md
| 1 | # Terminal Agent Benchmark |
| 2 | |
| 3 | Use this skill when you need to evaluate Con as a terminal-native agent, not just compile it. |
| 4 | |
| 5 | Primary references: |
| 6 | |
| 7 | - [`benchmarks/terminal-agent/README.md`](../../benchmarks/terminal-agent/README.md) |
| 8 | - [`docs/impl/terminal-agent-benchmark.md`](../../docs/impl/terminal-agent-benchmark.md) |
| 9 | - [`docs/impl/con-cli-e2e.md`](../../docs/impl/con-cli-e2e.md) |
| 10 | |
| 11 | ## Default workflow |
| 12 | |
| 13 | 1. Confirm a live app session and socket exist. |
| 14 | 2. Run the strict benchmark: |
| 15 | - `python3 benchmarks/terminal-agent/run.py --suite strict` |
| 16 | 3. If provider setup is present, run: |
| 17 | - `CON_BENCH_ENABLE_AGENT=1 python3 benchmarks/terminal-agent/run.py --suite all` |
| 18 | 4. Prefer a built-in profile when one matches the workflow: |
| 19 | - `python3 benchmarks/terminal-agent/run.py --list-profiles` |
| 20 | - `python3 benchmarks/terminal-agent/run.py --profile basic-local-shell` |
| 21 | - `CON_BENCH_ENABLE_AGENT=1 python3 benchmarks/terminal-agent/run.py --profile basic-local-codex --suite all` |
| 22 | 5. Use starter profiles for quick regression checks and operator profiles for richer coding, SSH maintenance, or tmux dev-loop evaluation. |
| 23 | - `python3 benchmarks/terminal-agent/run.py --profile operator-local-codex-devloop --suite operator` |
| 24 | - `python3 benchmarks/terminal-agent/run.py --profile operator-local-claude-devloop --suite operator` |
| 25 | - `python3 benchmarks/terminal-agent/run.py --profile operator-local-opencode-devloop --suite operator` |
| 26 | - `python3 benchmarks/terminal-agent/run.py --profile operator-ssh-dual-host-maintenance --suite operator` |
| 27 | - `python3 benchmarks/terminal-agent/run.py --profile operator-ssh-tmux-devloop --suite operator` |
| 28 | 6. For SSH/tmux changes, run the relevant playbook under `benchmarks/terminal-agent/playbooks/`. |
| 29 | 7. Save the JSON record under `.context/benchmarks/` and cite it in your report. |
| 30 | 8. If the run is an operator benchmark, score it with: |
| 31 | - `python3 benchmarks/terminal-agent/score.py --profile ... --record ... --score ...` |
| 32 | 9. Generate a trend report when comparing many runs: |
| 33 | - `python3 benchmarks/terminal-agent/report.py` |
| 34 | |
| 35 | ## Rules |
| 36 | |
| 37 | - Prefer `pane_id` over `pane_index` when following up on benchmark findings. |
| 38 | - Keep benchmark control on the existing `panes.*` API unless the benchmark |
| 39 | explicitly targets pane-local surfaces. Surface support is additive and should |
| 40 | not change the built-in agent's pane/tool contract. |
| 41 | - Do not treat playbook observations as strict pass/fail evidence unless the behavior is actually deterministic. |
| 42 | - If a scenario depends on host setup, say so explicitly. |
| 43 | - Keep operator playbooks safe-by-default. Prefer read-only checks first, and treat destructive or privileged steps as explicit branches. |
| 44 | - If a benchmark reveals a product limit, document the limit instead of hiding it behind a softer assertion. |
| 45 | - Operator suites intentionally serialize `agent ask` turns. If a tab already has a pending agent request, let the runner wait and reuse the same tab instead of opening a parallel benchmark against it. |
People who liked this also viewed
6 picksType
- find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 2026
- grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026
- vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026
- improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026
- setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026
- triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026
