google-agents-cli-eval
installsforks
How to install google-agents-cli-eval?
google/agents-cli/google-agents-cli-eval
$npx -y skills add google/agents-cli --skill google-agents-cli-evalgoogle-agents-cli-eval is an agent skill published from google/agents-cli, installable through the skills CLI.
SKILL.md
| 1 | # Agent Evaluation Guide |
| 2 | |
| 3 | > **Requires:** `agents-cli` (`uv tool install google-agents-cli`) — [install uv](https://docs.astral.sh/uv/getting-started/installation/index.md) first if needed. |
| 4 | |
| 5 | > **Scaffolded project?** If you used `/google-agents-cli-scaffold`, you already have `agents-cli eval run` (chains `generate` + `grade`), `tests/eval/datasets/`, and `tests/eval/eval_config.yaml`. Start with executing `eval run` and iterate from there. |
| 6 | |
| 7 | ## Reference Files |
| 8 | |
| 9 | | File | Contents | |
| 10 | |------|----------| |
| 11 | | `references/dataset_schema.md` | Canonical EvaluationDataset schema — all field types, JSON examples for single-turn / multi-turn / multi-agent, common mistakes | |
| 12 | | `references/metrics-guide.md` | Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config | |
| 13 | | `references/user-simulation.md` | Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics | |
| 14 | | `references/builtin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility | |
| 15 | | `references/multimodal-eval.md` | Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern | |
| 16 | |
| 17 | --- |
| 18 | |
| 19 | ## The Quality Flywheel |
| 20 | |
| 21 | Improving agent quality is iterative. The 5 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Agent Platform Eval Service for better quality and scale. |
| 22 | |
| 23 | ### 1. Prepare Data |
| 24 | |
| 25 | **Default:** Use or edit the scaffolded `tests/eval/datasets/basic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases. |
| 26 | |
| 27 | **Opt-in:** `agents-cli eval dataset synthesize` — user-simulate multi-turn datasets when you lack data; its output includes traces, so skip Stage 2 and grade directly. See *Eval Commands* and `references/user-simulation.md`. |
| 28 | |
| 29 | ### 2. Run Inference |
| 30 | |
| 31 | `agents-cli eval generate` — executes the agent over the dataset and writes traces to `artifacts/traces/`. Run this when you wrote the dataset by hand in Stage 1 (default path). **Skip this stage if you used `eval dataset synthesize`** — that command already produced traces. |
| 32 | |
| 33 | ### 3. Grade Traces (always run) |
| 34 | |
| 35 | `agents-cli eval grade` — scores the traces and writes `results_<ts>.{json,html}` to `artifacts/grade_results/`. No opt-in alternative; this is the core. Always run, regardless of how Stages 1 and 2 produced the traces. |
| 36 | |
| 37 | > **Shortcut:** `agents-cli eval run` chains Stages 2 + 3 in one command using the default `artifacts/traces/` directory between them. Use it for the common path; drop back to the two-step form when you need a custom traces location or want to grade an existing traces file. |
| 38 | |
| 39 | ### 4. Analyze Failures |
| 40 | |
| 41 | **Default:** Open the latest `artifacts/grade_results/results_<ts>.html` (or `.json`) and identify failed metrics — see *What to fix when scores fail* below for the fix table. |
| 42 | |
| 43 | **Opt-in:** `agents-cli eval analyze` — LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See *Eval Commands*. |
| 44 | |
| 45 | ### 5. Optimize & Code Fix |
| 46 | |
| 47 | **Default:** Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See *What to fix when scores fail* below for the failure → fix mapping. |
| 48 | |
| 49 | **Opt-in:** `agents-cli eval optimize` — runs ADK GEPA prompt optimization against a target metric. Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file. |
| 50 | |
| 51 | > **Long-running and expensive.** GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a **single** final `eval optimize` — never loop on this command. |
| 52 | |
| 53 | ### Running the loop |
| 54 | |
| 55 | Iterate stages 2 → 3 → 4 → 5 → 2 (or 1 → 3 → 4 → 5 → 1 if using `synthesize`). After each fix, run `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes |
People who liked this also viewed
6 picksType
- hyperframes-cliUse the HyperFrames CLI development loop: init, add, catalog, capture, lint, check, snapshot, compare, grade-compare, preview, play, present, beats, keyframes, single or batch render, publish, cloud, SkillsJul 2026
- request-refactor-planSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- analyze-projectRigor Analyze / Rigor Audit read-only skill for deep learning research repositories. Use when the user wants to read and understand a repository, inspect model structure and training or inference entrSkillsMay 2026
- explore-codeRigor Improve implementation leaf skill for auditable candidate implementation in deep learning research repositories. Use when the researcher explicitly authorizes exploratory work on an isolated braSkillsMay 2026
- ubiquitous-languageSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- scaffold-exercisesSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
