google-agents-cli-eval

Name: google-agents-cli-eval
Author: google

bygoogle· 116 skills

Installs

65k

Stars

5.4k

Forks

565

How to install google-agents-cli-eval?

google/agents-cli/google-agents-cli-eval

$npx -y skills add google/agents-cli --skill google-agents-cli-eval

Installs into the current project.

›Prefer a prompt? Paste this to your agent

Use this skill

Run `npx skills use "https://github.com/google/agents-cli" --skill "google/agents-cli/google-agents-cli-eval"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.

Use the whole pack

Use the skills in "https://github.com/google/agents-cli" that are relevant to the current task. Run `npx skills add "https://github.com/google/agents-cli"` and select the relevant skills, then follow their instructions.

Files · 1

View on GitHub

SKILL.md

1	# Agent Evaluation Guide
2
3	> Requires: `agents-cli` (`uv tool install google-agents-cli`) — [install uv](https://docs.astral.sh/uv/getting-started/installation/index.md) first if needed.
4
5	> Scaffolded project? If you used `/google-agents-cli-scaffold`, you already have `agents-cli eval run` (chains `generate` + `grade`), `tests/eval/datasets/`, and `tests/eval/eval_config.yaml`. Start with executing `eval run` and iterate from there.
6
7	## Reference Files
8
9	\| File \| Contents \|
10	\|------\|----------\|
11	\| `references/dataset_schema.md` \| Canonical EvaluationDataset schema — all field types, JSON examples for single-turn / multi-turn / multi-agent, common mistakes \|
12	\| `references/metrics-guide.md` \| Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config \|
13	\| `references/user-simulation.md` \| Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics \|
14	\| `references/builtin-tools-eval.md` \| google_search and model-internal tools — trajectory behavior, metric compatibility \|
15	\| `references/multimodal-eval.md` \| Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern \|
16
17	---
18
19	## The Quality Flywheel
20
21	Improving agent quality is iterative. The 5 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Agent Platform Eval Service for better quality and scale.
22
23	### 1. Prepare Data
24
25	Default: Use or edit the scaffolded `tests/eval/datasets/basic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases.
26
27	Opt-in: `agents-cli eval dataset synthesize` — user-simulate multi-turn datasets when you lack data; its output includes traces, so skip Stage 2 and grade directly. See Eval Commands and `references/user-simulation.md`.
28
29	### 2. Run Inference
30
31	`agents-cli eval generate` — executes the agent over the dataset and writes traces to `artifacts/traces/`. Run this when you wrote the dataset by hand in Stage 1 (default path). Skip this stage if you used `eval dataset synthesize` — that command already produced traces.
32
33	### 3. Grade Traces (always run)
34
35	`agents-cli eval grade` — scores the traces and writes `results_<ts>.{json,html}` to `artifacts/grade_results/`. No opt-in alternative; this is the core. Always run, regardless of how Stages 1 and 2 produced the traces.
36
37	> Shortcut: `agents-cli eval run` chains Stages 2 + 3 in one command using the default `artifacts/traces/` directory between them. Use it for the common path; drop back to the two-step form when you need a custom traces location or want to grade an existing traces file.
38
39	### 4. Analyze Failures
40
41	Default: Open the latest `artifacts/grade_results/results_<ts>.html` (or `.json`) and identify failed metrics — see What to fix when scores fail below for the fix table.
42
43	Opt-in: `agents-cli eval analyze` — LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See Eval Commands.
44
45	### 5. Optimize & Code Fix
46
47	Default: Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See What to fix when scores fail below for the failure → fix mapping.
48
49	Opt-in: `agents-cli eval optimize` — runs ADK GEPA prompt optimization against a target metric. Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.
50
51	> Long-running and expensive. GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a single final `eval optimize` — never loop on this command.
52
53	### Running the loop
54
55	Iterate stages 2 → 3 → 4 → 5 → 2 (or 1 → 3 → 4 → 5 → 1 if using `synthesize`). After each fix, run `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes — this is normal. Only after a case passes should you expand coverage with more eval cases.
56
57	When doing 5+ iterations, maintain a task list of which cases are fixed, which are still failing, and what fixes you've tried. Prevents re-attempting the same fix.
58
59	### Shortcuts That Waste Time
60
61	Recognize these rationalizations and push back — they always cost more time than they save:
62
63	\| Shortcut \| Why it fails \|
64	\|----------\|-------------\|
65	\| "I'll tune the eval thresholds down to make it pass" \| Lowering thresholds hides real failures. If the agent can't meet the bar, fix the agent — don't move the bar. \|
66	\| "This eval case is flaky, I'll skip it" \| Flaky evals reveal non-determinism in your agent. Fix with `temperature=0`, rubric-based metrics, or more specific instructions — don't delete the signal. \|
67	\| "I just need to fix the eval dataset, not the agent" \| If you're always adjusting expected outputs, your agent has a behavior problem. Fix the instructions or tool logic first. \|
68
69	## Choosing the Right Metrics
70
71	Pick built-in metrics by what you want to measure. Multi-turn metrics evaluate the full conversation; single-turn metrics evaluate one prompt-response pair (with intermediate tool calls). When no built-in fits, write a custom metric (see Evaluation Configuration Schema below).
72
73	\| Goal \| Recommended built-in metrics \|
74	\|------\|------------------------------\|
75	\| Did the agent achieve the user's goal? (catch-all for multi-turn agents) \| `multi_turn_task_success` \|
76	\| Was the agent's reasoning path logical and efficient? \| `multi_turn_trajectory_quality` \|
77	\| Quality of tool / function calling across turns \| `multi_turn_tool_use_quality` \|
78	\| Final response quality (no ground-truth reference needed) \| `final_response_quality` \|
79	\| Factual grounding (catch hallucinated claims, e.g., RAG agents) \| `hallucination` \|
80	\| Safety policy compliance \| `safety` \|
81	\| Domain-specific check no built-in covers \| Write a custom `LLMMetric` (LLM-judge) or `CodeExecutionMetric` (deterministic Python). See Evaluation Configuration Schema below. \|
82
83	Run `agents-cli eval metric list` to see all available built-ins. For full metric definitions and rubric details, see the [Agent Platform metric docs](https://cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/manage-metrics) and `references/metrics-guide.md`.
84
85	---
86
87	## What to fix when scores fail
88
89	After `agents-cli eval grade` completes, inspect the latest `artifacts/grade_results/results_<timestamp>.json` (or open the `.html` file) for per-case scores and judge rationales — that's the input to every fix decision below.
90
91	\| Failure \| What to change \|
92	\|---------\|---------------\|
93	\| `multi_turn_task_success` low \| The agent isn't completing the user's goal — fix orchestration, missing tool calls, premature termination, or wrong tool selection \|
94	\| `multi_turn_trajectory_quality` low \| The agent reaches the goal inefficiently or takes wrong steps — refine planning prompts, tighten instruction order, or remove redundant tool calls \|
95	\| `multi_turn_tool_use_quality` low \| Fix tool descriptions, parameter docstrings, or agent instructions for tool selection \|
96	\| `final_response_quality` low \| Read the auto-generated rubric verdicts; refine agent instructions to address the worst-scoring criterion (often clarity, completeness, or instruction-following) \|
97	\| `hallucination` low \| Tighten agent instructions to stay grounded in tool output; verify the tool actually returned the data the agent claimed \|
98	\| `safety` low \| Add safety guardrails to instructions; review the violating content category in the rubric verdict \|
99	\| Agent calls wrong tools \| Fix tool descriptions, agent instructions, or `tool_config` \|
100	\| Agent calls extra tools \| Add strict stop instructions, or switch to `multi_turn_tool_use_quality` \|
101
102	After applying a fix, rerun `agents-cli eval generate && agents-cli eval grade` and use `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the fix improved the target metric without regressing others.
103
104	---
105
106	## Eval Commands
107
108	All `agents-cli eval` subcommands support `--help` for the authoritative flag list and defaults — run `agents-cli eval <subcommand> --help` (or `agents-cli eval dataset <subcommand> --help`) when in doubt. The examples below show the most common invocations; flags can change between releases.
109
110	### `eval generate`
111
112	Runs an agent over an evaluation dataset and writes traces to disk.
113
114	```bash
115	# Basic — uses tests/eval/datasets/, writes to artifacts/traces/
116	agents-cli eval generate
117
118	# Advanced — custom dataset and output dir
119	agents-cli eval generate --dataset tests/eval/datasets/custom.json -o ./custom_traces/
120	```
121
122	### `eval grade`
123
124	Scores generated traces against built-in or custom metrics. Writes timestamped `results_<YYYYMMDD_HHMMSS>.json` (consumed by `eval compare`) and `.html` (open in a browser) into the output dir, and prints a summary table to the console.
125
126	```bash
127	# Basic — defaults: traces from artifacts/traces/, results to artifacts/grade_results/,
128	# metrics from tests/eval/eval_config.yaml's metrics_to_run
129	agents-cli eval grade
130
131	# Advanced 1 — grade traces from a non-default location (the canonical
132	# pairing for `eval generate --output custom_traces/`)
133	agents-cli eval grade --traces custom_traces/
134
135	# Advanced 2 — pick built-in metrics, custom output dir
136	agents-cli eval grade --metrics tool_use_quality,safety --output ./out/
137
138	# Advanced 3 — load metrics to run from a config file (YAML or JSON) on a specified trace file.
139	agents-cli eval grade --traces ./artifacts/traces/trace_1.json --config tests/eval/eval_config.yaml
140	```
141
142	See Evaluation Configuration Schema below for the config file format.
143
144	### `eval compare`
145
146	Diffs two `results_*.json` files produced by `eval grade`. Run it after a fix to confirm the target metric improved without regressing others.
147
148	```bash
149	agents-cli eval compare baseline.json candidate.json
150	```
151
152	### `eval metric list`
153
154	Lists the built-in metric names usable with `eval grade --metrics`.
155
156	```bash
157	agents-cli eval metric list
158	```
159
160	### `eval analyze`
161
162	Runs LLM-based failure clustering and root-cause analysis over a `results_*.json` produced by `eval grade`. Use when you have 10+ failing cases and want categorized failure modes instead of reading the HTML case-by-case. Supported `--metric` values: `multi_turn_task_success`, `multi_turn_tool_use_quality`.
163
164	```bash
165	# Basic — analyze a results file with default settings
166	agents-cli eval analyze --eval-result artifacts/grade_results/results_<ts>.json
167
168	# Advanced — restrict to a specific metric and cap loss clusters
169	agents-cli eval analyze \
170	--eval-result artifacts/grade_results/results_<ts>.json \
171	--metric multi_turn_tool_use_quality \
172	--top-k 5 \
173	--output artifacts/analysis_<ts>.json
174	```
175
176	### `eval dataset synthesize`
177
178	Generates user scenarios server-side from your agent's tools and instructions, then plays each scenario against an LLM-backed user simulator. The output is a graded-ready trace file with full `agent_data.turns` populated — feed it directly to `eval grade` (skip `eval generate`).
179
180	```bash
181	# Basic — generate 3 default scenarios (up to 5 turns each) into artifacts/traces/
182	# (where eval grade reads from by default, so synthesize → grade works without flags)
183	agents-cli eval dataset synthesize
184
185	# Advanced — guide scenario generation with optional instruction and environment context
186	agents-cli eval dataset synthesize \
187	-n 5 \
188	--instruction "Customer asking about refunds" \
189	--environment-context "E-commerce support" \
190	--max-turns 8 \
191	-o tests/eval/datasets/refund_scenarios.json
192	```
193
194	For scenario semantics, the full `eval dataset synthesize` flag table, and which simulator internals are not user-configurable, see `references/user-simulation.md`.
195
196	### `eval optimize`
197
198	Runs ADK GEPA prompt optimization against a target metric. Suitable after `eval grade` identifies prompt-only failures (wording, not tool/orchestration logic). `--dataset` and `--target-metric` override values in `--config` when both are passed. Long-running and expensive — see Stage 5 of the Quality Flywheel for usage guidance.
199
200	```bash
201	# Basic — optimize against a single metric on a dataset
202	agents-cli eval optimize --dataset tests/eval/datasets/basic-dataset.json --target-metric final_response_quality
203
204	# Advanced — drive multi-metric / multi-dataset optimization from a config file
205	agents-cli eval optimize --config tests/eval/optimization_config.json
206	```
207
208	### `eval submit` / `eval results` (cloud-side)
209
210	The managed, asynchronous counterpart to the local path, for large or CI-driven runs: `eval submit` hands the dataset and metrics to the Agent Platform Eval Service, and `eval results` polls and downloads the scores. Pass `--resource-name <agent>` to also run inference server-side (managed `generate` + `grade`); omit it to grade an existing trace (managed `grade`).
211
212	```bash
213	# Grade an existing trace server-side; returns a run resource name to poll
214	agents-cli eval submit --dataset tests/eval/datasets/basic-dataset.json --dest gs://my-bucket
215	# Add --resource-name projects/<p>/locations/<l>/reasoningEngines/<id> to run inference too
216
217	agents-cli eval results --run-id <run-resource-name>
218	```
219
220	---
221
222	## Evaluation Dataset Format
223
224	An `EvaluationDataset` is a JSON file with an `eval_cases` array. Cases come in two shapes depending on how they're used:
225
226	- Inference input (what you give to `eval generate`) — a user prompt or a partial conversation ending in a user prompt. The agent runs and produces traces.
227	- Grading input (what you give to `eval grade`) — a complete trace including the agent's responses and tool calls. Normally produced by `eval generate` or `eval dataset synthesize`; you don't write these by hand.
228
229	See `references/dataset_schema.md` for the full canonical schema, all field types, and common mistakes.
230
231	### Inference input format
232
233	Two shapes are supported.
234
235	(a) Simple single-turn prompt — what the scaffolded `tests/eval/datasets/basic-dataset.json` uses. The agent runs from scratch.
236
237	```json
238	{
239	"eval_cases": [
240	{
241	"eval_case_id": "greeting",
242	"prompt": {
243	"role": "user",
244	"parts": [{"text": "Hello, what can you help me with?"}]
245	}
246	},
247	{
248	"eval_case_id": "weather_query",
249	"prompt": {
250	"role": "user",
251	"parts": [{"text": "What's the weather like in San Francisco?"}]
252	}
253	}
254	]
255	}
256	```
257
258	(b) Multi-turn continuation via `agent_data` — a partial conversation whose last turn ends with a user message; the agent's next response is evaluated. See `references/dataset_schema.md` (Multi-Turn / Multi-Agent Dataset) for the JSON shape.
259
260	### Grading input format (traces)
261
262	A complete trace — agent responses plus `function_call` / `function_response` parts — normally produced by `eval generate` / `eval dataset synthesize` (you don't write these by hand). Authors are `"user"`, an agent ID from the `agents` map, or `"tool"`. See `references/dataset_schema.md` for the trace shape, multi-agent examples, and the full type reference.
263
264	---
265
266	## Evaluation Configuration Schema
267
268	`agents-cli eval grade --config <path>` accepts a single configuration file in either YAML (`.yaml` / `.yml`) or JSON (`.json`). The file declares two parts:
269
270	- `metrics_to_run` — the selection list of metric names to execute on this run. Names resolve to built-in metrics first, then to entries in `custom_metrics`.
271	- `custom_metrics` — a definition pool of custom metrics available to this project. Defining a metric here does not run it; it must also appear in `metrics_to_run` (or be passed via `--metrics name1,name2` on the CLI, which is equivalent to overriding `metrics_to_run` for that invocation).
272
273	Minimal example (YAML preferred — human-readable, no JSON escaping for prompts and Python):
274
275	```yaml
276	metrics_to_run:
277	- multi_turn_task_success # built-in
278	- example_llm_metric # selected from custom_metrics pool below
279	- agent_turn_count # selected from custom_metrics pool below
280
281	custom_metrics:
282	- name: example_llm_metric
283	prompt_template: \|
284	Rate the agent's response 1-5 for helpfulness and accuracy.
285	Prompt: {prompt}
286	Final response: {response}
287	Full trace (for tool-call and reasoning context): {agent_data}
288	Return JSON: {"score": <1\|2\|3\|4\|5>, "explanation": "<reason>"}
289
290	- name: agent_turn_count
291	custom_function: \|
292	def evaluate(instance):
293	turns = (instance.get("agent_data") or {}).get("turns", [])
294	return {'score': len(turns)}
295	```
296
297	JSON is also accepted (same field names, with `prompt_template` and `custom_function` as escaped strings) — but always prefer YAML for human-readable configs.
298
299	Each entry in `custom_metrics` is dispatched by field: presence of `custom_function` makes it a `CodeExecutionMetric` (deterministic Python); otherwise it's an `LLMMetric` (LLM-as-judge with `prompt_template`). Run `agents-cli eval metric list` to see available built-ins. For full custom-metric field reference (judge model options, sampling counts), see `references/metrics-guide.md`.
300
301	Agent trace field model. For datasets produced by `agents-cli eval generate` (or `eval dataset synthesize`), each eval case exposes three standard fields to a metric:
302
303	- `{prompt}` — the user message (or first user turn).
304	- `{response}` — the agent's final text response, extracted from the last text-bearing event. In `custom_function` callbacks this is `instance['response']` with shape `{"role": "model", "parts": [{"text": "..."}]}`.
305	- `{agent_data}` — the full structured `turns`/`events` trace, useful when the judge needs to reason about tool calls or intermediate reasoning.
306
307	`{reference}` and `{context}` resolve only when the eval case has `reference` / `context` fields populated (e.g., golden-answer datasets); they are not populated by `eval generate` / `eval dataset synthesize`.
308
309	Code-based metrics default to local in-process execution (no GCP project or region required, but the `evaluate(instance)` function runs with the CLI's privileges). Set `execution: "remote"` on the metric to run it server-side in Vertex AI's `CodeExecutionMetric` sandbox instead — that path requires a configured GCP project + region.
310
311	---
312
313
314	## Common Gotchas
315
316	### Use Rubric-Based Tool Evaluation instead of Hardcoded Sequences
317
318	Evaluating agent tool usage using strict sequence matching is fragile because agents may call helper tools (like searches or geocoding) in different orders or perform extra proactive steps.
319
320	Instead, use `multi_turn_tool_use_quality` / `multi_turn_trajectory_quality`. These metrics automatically generate content-based and intent-based adaptive rubrics, assessing technical correctness and technical sequence logic semantically using an LLM judge rather than forcing a rigid match.
321
322	### App name must match directory name
323
324	The `App` object's `name` parameter MUST match the directory containing your agent:
325
326	```python
327	# CORRE

6 picks

Type

Run `npx skills use "https://github.com/google/agents-cli" --skill "google/agents-cli/google-agents-cli-eval"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.

Use the skills in "https://github.com/google/agents-cli" that are relevant to the current task. Run `npx skills add "https://github.com/google/agents-cli"` and select the relevant skills, then follow their instructions.

google-agents-cli-eval

How to install google-agents-cli-eval?

Related

google-agents-cli-eval

How to install google-agents-cli-eval?

Related