google-agents-cli-observability
bygoogle· 116 skills
This skill should be used when the user wants to "set up tracing", "monitor my ADK agent", "configure logging", "add observability", "debug production traffic", or needs guidance on monitoring deployed ADK (Agent Development Kit) agents. Covers Cloud Trace, prompt-response logging, BigQuery Agent Analytics, third-party integrations (AgentOps, Phoenix, MLflow, etc.), and troubleshooting. Part of the Google ADK (Agent Development Kit) skills suite. Do NOT use for deployment setup (use google-agents-cli-deploy) or API code patterns (use google-agents-cli-adk-code).
How to install google-agents-cli-observability?
google/agents-cli/google-agents-cli-observability
$npx -y skills add google/agents-cli --skill google-agents-cli-observabilityInstalls into the current project.
›Prefer a prompt? Paste this to your agent
Run `npx skills use "https://github.com/google/agents-cli" --skill "google/agents-cli/google-agents-cli-observability"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.
Use the skills in "https://github.com/google/agents-cli" that are relevant to the current task. Run `npx skills add "https://github.com/google/agents-cli"` and select the relevant skills, then follow their instructions.
SKILL.md
| 1 | # ADK Observability Guide |
| 2 | |
| 3 | > **Cloud Trace** works out of the box — no infrastructure needed. **Prompt-response logging** and **BigQuery Agent Analytics** require Terraform-provisioned infrastructure (service account, GCS bucket, BigQuery dataset). Run `agents-cli infra single-project --project PROJECT_ID` to provision these resources. See `references/cloud-trace-and-logging.md` for details, env vars, and verification commands. If your project isn't scaffolded yet, see `/google-agents-cli-scaffold` first. |
| 4 | |
| 5 | ### Order of operations for `agent_runtime` deployments |
| 6 | |
| 7 | For `deployment_target = agent_runtime`, run `agents-cli infra single-project` **before** the first `agents-cli deploy`. The Terraform module owns the entire Reasoning Engine resource (service account, deployment spec, env vars), so applying it after an SDK-based deploy creates a state mismatch Terraform can't reconcile without taking ownership of the whole resource. |
| 8 | |
| 9 | Already ran `agents-cli deploy`? Two options: |
| 10 | |
| 11 | 1. **Switch to Terraform-managed** — delete the SDK-deployed Reasoning Engine, then run `agents-cli infra single-project` and `agents-cli deploy` (sessions and in-flight state are lost). |
| 12 | 2. **Keep the SDK-deployed instance** — skip `infra single-project` and set the observability env vars by re-running `agents-cli deploy --update-env-vars "KEY=VALUE,..."`; deploy matches the existing Reasoning Engine by display name and updates it in place, preserving env vars set outside the deploy. You must also grant its service account the telemetry IAM roles the Terraform module would otherwise provision: `roles/storage.admin` (write completions to the logs bucket), `roles/logging.logWriter`, `roles/cloudtrace.agent`, plus `roles/bigquery.dataOwner` + `roles/bigquery.jobUser` when scaffolded with `--bq-analytics`. The full set lives in `deployment/terraform/single-project/iam.tf` (from `app_sa_roles`) and `telemetry.tf`. Terraform-managed env vars aren't available in this mode. |
| 13 | |
| 14 | ### Reference Files |
| 15 | |
| 16 | | File | Contents | |
| 17 | |------|----------| |
| 18 | | `references/cloud-trace-and-logging.md` | Scaffolded project details — Terraform-provisioned resources, environment variables, verification commands, enabling/disabling locally | |
| 19 | | `references/bigquery-agent-analytics.md` | BQ Agent Analytics plugin — enabling, key features, GCS offloading, tool provenance | |
| 20 | |
| 21 | --- |
| 22 | |
| 23 | ## Observability Tiers |
| 24 | |
| 25 | Choose the right level of observability based on your needs: |
| 26 | |
| 27 | | Tier | What It Does | Scope | Default State | Best For | |
| 28 | |------|-------------|-------|---------------|----------| |
| 29 | | **Cloud Trace** | Distributed tracing — execution flow, latency, errors via OpenTelemetry spans | All templates, all environments | Always enabled | Debugging latency, understanding agent execution flow | |
| 30 | | **Prompt-Response Logging** | GenAI interactions exported to GCS, BigQuery, and Cloud Logging | ADK agents only | Disabled locally, enabled when deployed | Auditing LLM interactions, compliance | |
| 31 | | **BigQuery Agent Analytics** | Structured agent events (LLM calls, tool use, outcomes) to BigQuery | ADK agents with plugin enabled | Opt-in (`--bq-analytics` at scaffold time) | Conversational analytics, custom dashboards, LLM-as-judge evals | |
| 32 | | **Third-Party Integrations** | External observability platforms (AgentOps, Phoenix, MLflow, etc.) | Any ADK agent | Opt-in, per-provider setup | Team collaboration, specialized visualization, prompt management | |
| 33 | |
| 34 | **Ask the user** which tier(s) they need — they can be combined. Cloud Trace is always on; the others are additive. |
| 35 | |
| 36 | --- |
| 37 | |
| 38 | ## Cloud Trace |
| 39 | |
| 40 | ADK uses OpenTelemetry to emit distributed traces. Every agent invocation produces spans that track the full execution flow. |
| 41 | |
| 42 | ### Span Hierarchy |
| 43 | |
| 44 | ``` |
| 45 | invoke_workflow (top-level run) |
| 46 | └── invoke_agent (one per agent in the chain) |
| 47 | ├── call_llm (model request) |
| 48 | │ └── generate_content (underlying GenAI model call) |
| 49 | └── execute_tool (tool execution) |
| 50 | ``` |
| 51 | |
| 52 | ### Setup by Deployment Type |
| 53 | |
| 54 | | Deployment | Setup | |
| 55 | |-----------|-------| |
| 56 | | **Agent Runtime** | Automatic — `get_fast_api_app(otel_to_cloud=True)`, gated on `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY` (set by deploy); exports to Cloud Trace/Logging + Agent Engine console | |
| 57 | | **Cloud Run / GKE (scaffolded)** | Automatic — `get_fast_api_app(otel_to_cloud=True)` exports to Cloud Trace/Logging | |
| 58 | | **Cloud Run / GKE (manual)** | Configure OpenTelemetry exporter in your app | |
| 59 | | **Local dev** | Works with `agents-cli playground`; traces visible in Cloud Console | |
| 60 | |
| 61 | View traces: **Cloud Console → Trace → Trace explorer** |
| 62 | |
| 63 | For detailed setup instructions (Agent Runtime CLI/SDK, Cloud Run, custom deployments), fetch `https://adk.dev/integrations/cloud-trace/index.md`. |
| 64 | |
| 65 | --- |
| 66 | |
| 67 | ## Prompt-Response Logging |
| 68 | |
| 69 | Captures GenAI interactions and exports to GCS (JSONL) and BigQuery (via log sinks + external tables). Content is governed by **two independent tiers**; the net Terraform-deploy default is **full content in GCS/BigQuery, none in traces**: |
| 70 | |
| 71 | | Tier | Captures | Controlled by | Default (Terraform deploy) | |
| 72 | |------|----------|---------------|----------------------------| |
| 73 | | **GCS/BigQuery completions** | Full prompts/responses (the prompt-response logging feature) | `OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK=upload` + `LOGS_BUCKET_NAME` | **On** — full content | |
| 74 | | **Trace spans / Cloud Logging events** | Span/event content | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` + `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false` | **Off** — `NO_CONTENT` | |
| 75 | |
| 76 | The tiers are independent: GCS/BigQuery uploads capture full content whenever their upload vars are set and do **not** honor `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`, which governs the traces/events tier only. Its valid (experimental-semconv) values: |
| 77 | |
| 78 | - `NO_CONTENT` — no content in spans/events (scaffolded default) |
| 79 | - `EVENT_ONLY` — content in Cloud Logging events |
| 80 | - `SPAN_ONLY` / `SPAN_AND_EVENT` — content in trace spans |
| 81 | - `true` / `false` — **invalid**; fall back to `NO_CONTENT` |
| 82 | |
| 83 | For the full mechanics (semconv opt-in, declarative Terraform config, env-var table, enabling/disabling, verification commands), see `references/cloud-trace-and-logging.md`. For ADK logging docs (log levels, configuration, debugging), fetch `https://adk.dev/observability/logging/index.md`. |
| 84 | |
| 85 | --- |
| 86 | |
| 87 | ## BigQuery Agent Analytics Plugin |
| 88 | |
| 89 | Optional plugin that logs structured agent events to BigQuery. Enable with `--bq-analytics` at scaffold time. See `references/bigquery-agent-analytics.md` for details. |
| 90 | |
| 91 | --- |
| 92 | |
| 93 | ## Third-Party Integrations |
| 94 | |
| 95 | ADK supports many third-party observability platforms (via OpenTelemetry or custom instrumentation). The table below covers common ones; the full list is larger (see the pointer below it). |
| 96 | |
| 97 | | Platform | Key Differentiator | Setup Complexity | Self-Hosted Option | |
| 98 | |----------|-------------------|-----------------|-------------------| |
| 99 | | **AgentOps** | Session replays, 2-line setup, replaces native telemetry | Minimal | No (SaaS) | |
| 100 | | **Arize AX** | Commercial platform, production monitoring, evaluation dashboards | Low | No (SaaS) | |
| 101 | | **Phoenix** | Open-source, custom evaluators, experiment testing | Low | Yes | |
| 102 | | **MLflow** | OTel traces to MLflow Tracking Server, span tree visualization | Medium (needs SQL backend) | Yes | |
| 103 | | **Monocle** | 1-call setup, VS Code Gantt chart visualizer | Minimal | Yes (local files) | |
| 104 | | **Weave** | W&B platform, team collaboration, timeline views | Low | No (SaaS) | |
| 105 | | **Freeplay** | Prompt management + evals + observability in one platform | Low | No (SaaS) | |
| 106 | |
| 107 | **Ask the user** which platform they prefer — present the trade-offs and let them choose. Fetch a platform's setup page at `https://adk.dev/integrations/<slug>/index.md` (slugs for the table above: `agentops`, `arize-ax`, `phoenix`, `mlflow-tracing`, `monocle`, `weave`, `freeplay`). ADK has more observability integrations (Datadog, Galileo, LangWatch, Latitude, Future AGI, Respan, Zespan, …) — browse the complete, current list at `https://adk.dev/integrations/` (observability topic). |
| 108 | |
| 109 | --- |
| 110 | |
| 111 | ## Troubleshooting |
| 112 | |
| 113 | | Issue | Solution | |
| 114 | |-------|----------| |
| 115 | | No traces in Cloud Trace | Verify `fast_api_app.py` uses `get_fast_api_app(otel_to_cloud=True)` (Agent Runtime gates it on `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY`) and the SA has the `cloudtrace.agent` role | |
| 116 | | Prompt-response data not appearing | Check `LOGS_BUCKET_NAME` is set; verify SA has `storage.objectCreator` on the bucket; check app logs for telemetry setup warnings | |
| 117 | | Content in traces/events (unwanted) | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT` keeps content out of spans/events. NOTE: GCS/BigQuery completions still capture full content — to stop that, remove `LOGS_BUCKET_NAME`/`OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK` (drop the upload block in `service.tf`) | |
| 118 | | BigQuery Analytics not logging | Verify plugin is configured in `app/agent.py`; check `BQ_ANALYTICS_DATASET_ID` env var is set | |
| 119 | | Third-party integration not capturing spans | Check provider-specific env vars (API keys, endpoints); some providers (AgentOps) replace native telemetry | |
| 120 | | Traces missing tool spans | Tool execution spans appear under `execute_tool` — check trace explorer filters | |
| 121 | | High telemetry costs | Switch to `NO_CONTENT` mode; reduce BigQuery retention; disable unused tiers | |
| 122 | |
| 123 | --- |
| 124 | |
| 125 | ## Deep Dive: ADK Docs (WebFetch URLs) |
| 126 | |
| 127 | For detailed documentation beyond what this skill covers, fetch these pages: |
| 128 | |
| 129 | | Topic | URL | |
| 130 | |-------|-----| |
| 131 | | Observability overview | `https://adk.dev/observability/index.md` | |
| 132 | | Agent activity logging | `https://adk.dev/observability/logging/index.md` | |
| 133 | | Cloud Trace integration | `https://adk.dev/integrations/cloud-trace/index.md` | |
| 134 | | BigQuery Agent Analytics | `https://adk.dev/integrations/bigquery-agent-analytics/index.md` | |
| 135 | |
| 136 | --- |
| 137 | |
| 138 | ## Related Skills |
| 139 | |
| 140 | - `/google-agents-cli-deploy` — Deployment targets, CI/CD pipelines, and production workflows |
| 141 | - `/google-agents-cli-workflow` — Development workflow, coding guidelines, and operational rules |
| 142 | - `/google-agents-cli-adk-code` — ADK Python API quick reference for writing agent code |
Related
6 picksType
- find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express…Jul 2026
- mcp-builderGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools.Jul 2026
- google-agents-cli-adk-codeThis skill should be used when the user wants to "write agent code", "build an agent with ADK", "add a tool", "create a callback", "define an agent", "use…Jul 2026
- google-agents-cli-workflowThis skill should be used when the user wants to "develop an agent", "build an agent using ADK", "run the agent locally", "debug agent code", "test an agent",…Jul 2026
- google-agents-cli-evalThis skill should be used when the user wants to "run an evaluation", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval…Jul 2026
- google-agents-cli-scaffoldThis skill should be used when the user wants to "create an agent project", "start a new ADK project", "build me a new agent", "add CI/CD to my project", "add…Jul 2026
