agent-observability

Name: agent-observability
Author: BagelHole

42stars4forks

How to install agent-observability?

bagelhole/devops-security-agent-skills/agent-observability

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability

Installs into the current project.

Summary

Instrument AI agents with tracing, token metrics, latency, and cost visibility. Use for reliability and debugging.

Files · 1

View on GitHub

SKILL.md

1	# Agent Observability
2
3	Monitor AI agent behavior with logs, traces, metrics, and cost telemetry. This skill covers the full observability stack for LLM-powered applications: from raw Prometheus counters to Grafana dashboards, OpenTelemetry tracing, structured logging, cost tracking, SLO definition, and PII redaction.
4
5	---
6
7	## When to Use
8
9	Apply this skill whenever you operate:
10
11	- Autonomous AI agents that make multi-step tool calls (e.g., coding agents, support agents, data-pipeline agents).
12	- LLM-backed APIs serving chat completions, summarisation, or classification behind a REST or gRPC gateway.
13	- RAG pipelines where a retriever fetches context from a vector store before prompting a model.
14	- Multi-agent orchestrations (crew-style or graph-based) where several agents collaborate on a single task.
15	- Batch inference jobs that process thousands of prompts against a model endpoint.
16
17	Key signals that you need this skill:
18
19	1. You cannot answer "what is p95 latency for agent responses this week?"
20	2. You have no per-request cost attribution.
21	3. Debugging a bad agent response requires grepping raw application logs.
22	4. You have no alerting on token-usage spikes or elevated error rates.
23
24	---
25
26	## Core Metrics
27
28	Define these metrics at the application layer. All examples use the Prometheus client library naming conventions.
29
30	### Latency
31
32	```python
33	from prometheus_client import Histogram
34
35	# Total end-to-end latency for a full agent turn (user prompt -> final response)
36	AGENT_LATENCY = Histogram(
37	"agent_request_duration_seconds",
38	"End-to-end latency of an agent request",
39	labelnames=["agent_name", "model", "status"],
40	buckets=(0.25, 0.5, 1, 2, 5, 10, 30, 60, 120),
41	)
42
43	# Latency of a single LLM API call (one completion request)
44	LLM_CALL_LATENCY = Histogram(
45	"llm_call_duration_seconds",
46	"Latency of an individual LLM API call",
47	labelnames=["model", "provider", "stream"],
48	buckets=(0.1, 0.25, 0.5, 1, 2, 5, 10, 30),
49	)
50
51	# Latency of tool/function calls executed by the agent
52	TOOL_CALL_LATENCY = Histogram(
53	"agent_tool_call_duration_seconds",
54	"Latency of a tool call executed by the agent",
55	labelnames=["tool_name", "agent_name", "status"],
56	buckets=(0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10),
57	)
58	```
59
60	### Token Usage
61
62	```python
63	from prometheus_client import Counter, Histogram
64
65	PROMPT_TOKENS = Counter(
66	"llm_prompt_tokens_total",
67	"Total prompt tokens sent to the model",
68	labelnames=["model", "agent_name"],
69	)
70
71	COMPLETION_TOKENS = Counter(
72	"llm_completion_tokens_total",
73	"Total completion tokens received from the model",
74	labelnames=["model", "agent_name"],
75	)
76
77	CACHED_TOKENS = Counter(
78	"llm_cached_tokens_total",
79	"Prompt tokens served from KV-cache (provider-reported)",
80	labelnames=["model", "agent_name"],
81	)
82
83	TOKENS_PER_REQUEST = Histogram(
84	"llm_tokens_per_request",
85	"Total tokens (prompt + completion) per request",
86	labelnames=["model", "agent_name"],
87	buckets=(100, 500, 1000, 2000, 4000, 8000, 16000, 32000, 64000, 128000),
88	)
89	```
90
91	### Cost
92
93	```python
94	from prometheus_client import Counter
95
96	LLM_COST = Counter(
97	"llm_cost_dollars_total",
98	"Estimated cost in USD for LLM usage",
99	labelnames=["model", "agent_name", "cost_type"], # cost_type: prompt \| completion
100	)
101	```
102
103	### Tool Calls
104
105	```python
106	from prometheus_client import Counter
107
108	TOOL_CALLS_TOTAL = Counter(
109	"agent_tool_calls_total",
110	"Total tool calls made by agents",
111	labelnames=["tool_name", "agent_name", "status"], # status: success \| error \| timeout
112	)
113	```
114
115	### Errors and Retries
116
117	```python
118	from prometheus_client import Counter, Gauge
119
120	LLM_ERRORS = Counter(
121	"llm_errors_total",
122	"Errors returned by the LLM provider",
123	labelnames=["model", "provider", "error_type"], # error_type: rate_limit \| timeout \| 5xx \| auth
124	)
125
126	LLM_RETRIES = Counter(
127	"llm_retries_total",
128	"Retried LLM API calls",
129	labelnames=["model", "provider", "retry_reason"],
130	)
131
132	AGENT_ACTIVE_REQUESTS = Gauge(
133	"agent_active_requests",
134	"Number of agent requests currently in flight",
135	labelnames=["agent_name"],
136	)
137	```
138
139	---
140
141	## OpenTelemetry Integration
142
143	Use the OpenTelemetry Python SDK to create traces that capture every step of an agent turn: the top-level request, each LLM call, each tool execution, and retrieval operations.
144
145	### Setup
146
147	```python
148	# otel_setup.py
149	from opentelemetry import trace
150	from opentelemetry.sdk.trace import TracerProvider
151	from opentelemetry.sdk.trace.export import BatchSpanProcessor
152	from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
153	from opentelemetry.sdk.resources import Resource
154
155	def init_tracing(service_name: str, otlp_endpoint: str = "http://localhost:4317"):
156	resource = Resource.create({
157	"servi

People who liked this also viewed

6 picks

Type

lark-openapi-explorer飞书/Lark 原生 OpenAPI 探索：从官方文档库中挖掘未经 CLI 封装的原生 OpenAPI 接口。当用户的需求无法被现有 lark-* skill 或 lark-cli 已注册命令满足，需要查找并调用原生飞书 OpenAPI 时使用。SkillsJul 2026378k16k
lark-skill-maker创建 lark-cli 的自定义 Skill。当用户需要把飞书 API 操作封装成可复用的 Skill（包装原子 API 或编排多步流程）时使用。SkillsJul 2026377k16k
azure-cost-optimizationOfficial agent plugin providing skills and MCP server configurations for Azure scenarios.SkillsJul 2026209k1.3k
vercel-react-native-skillsVercel's official collection of agent skillsSkillsJul 2026172k29k
mcp-builderGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate externaSkillsJul 202693k163k
gws-sharedgws CLI: Shared patterns for authentication, global flags, and output formatting.SkillsJul 202653k30k

agent-observability

42stars4forks

View on GitHub

How to install agent-observability?

bagelhole/devops-security-agent-skills/agent-observability

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability

Installs into the current project.

Summary

Instrument AI agents with tracing, token metrics, latency, and cost visibility. Use for reliability and debugging.

Files · 1

View on GitHub

SKILL.md

1	# Agent Observability
2
3	Monitor AI agent behavior with logs, traces, metrics, and cost telemetry. This skill covers the full observability stack for LLM-powered applications: from raw Prometheus counters to Grafana dashboards, OpenTelemetry tracing, structured logging, cost tracking, SLO definition, and PII redaction.
4
5	---
6
7	## When to Use
8
9	Apply this skill whenever you operate:
10
11	- Autonomous AI agents that make multi-step tool calls (e.g., coding agents, support agents, data-pipeline agents).
12	- LLM-backed APIs serving chat completions, summarisation, or classification behind a REST or gRPC gateway.
13	- RAG pipelines where a retriever fetches context from a vector store before prompting a model.
14	- Multi-agent orchestrations (crew-style or graph-based) where several agents collaborate on a single task.
15	- Batch inference jobs that process thousands of prompts against a model endpoint.
16
17	Key signals that you need this skill:
18
19	1. You cannot answer "what is p95 latency for agent responses this week?"
20	2. You have no per-request cost attribution.
21	3. Debugging a bad agent response requires grepping raw application logs.
22	4. You have no alerting on token-usage spikes or elevated error rates.
23
24	---
25
26	## Core Metrics
27
28	Define these metrics at the application layer. All examples use the Prometheus client library naming conventions.
29
30	### Latency
31
32	```python
33	from prometheus_client import Histogram
34
35	# Total end-to-end latency for a full agent turn (user prompt -> final response)
36	AGENT_LATENCY = Histogram(
37	"agent_request_duration_seconds",
38	"End-to-end latency of an agent request",
39	labelnames=["agent_name", "model", "status"],
40	buckets=(0.25, 0.5, 1, 2, 5, 10, 30, 60, 120),
41	)
42
43	# Latency of a single LLM API call (one completion request)
44	LLM_CALL_LATENCY = Histogram(
45	"llm_call_duration_seconds",
46	"Latency of an individual LLM API call",
47	labelnames=["model", "provider", "stream"],
48	buckets=(0.1, 0.25, 0.5, 1, 2, 5, 10, 30),
49	)
50
51	# Latency of tool/function calls executed by the agent
52	TOOL_CALL_LATENCY = Histogram(
53	"agent_tool_call_duration_seconds",
54	"Latency of a tool call executed by the agent",
55	labelnames=["tool_name", "agent_name", "status"],
56	buckets=(0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10),
57	)
58	```
59
60	### Token Usage
61
62	```python
63	from prometheus_client import Counter, Histogram
64
65	PROMPT_TOKENS = Counter(
66	"llm_prompt_tokens_total",
67	"Total prompt tokens sent to the model",
68	labelnames=["model", "agent_name"],
69	)
70
71	COMPLETION_TOKENS = Counter(
72	"llm_completion_tokens_total",
73	"Total completion tokens received from the model",
74	labelnames=["model", "agent_name"],
75	)
76
77	CACHED_TOKENS = Counter(
78	"llm_cached_tokens_total",
79	"Prompt tokens served from KV-cache (provider-reported)",
80	labelnames=["model", "agent_name"],
81	)
82
83	TOKENS_PER_REQUEST = Histogram(
84	"llm_tokens_per_request",
85	"Total tokens (prompt + completion) per request",
86	labelnames=["model", "agent_name"],
87	buckets=(100, 500, 1000, 2000, 4000, 8000, 16000, 32000, 64000, 128000),
88	)
89	```
90
91	### Cost
92
93	```python
94	from prometheus_client import Counter
95
96	LLM_COST = Counter(
97	"llm_cost_dollars_total",
98	"Estimated cost in USD for LLM usage",
99	labelnames=["model", "agent_name", "cost_type"], # cost_type: prompt \| completion
100	)
101	```
102
103	### Tool Calls
104
105	```python
106	from prometheus_client import Counter
107
108	TOOL_CALLS_TOTAL = Counter(
109	"agent_tool_calls_total",
110	"Total tool calls made by agents",
111	labelnames=["tool_name", "agent_name", "status"], # status: success \| error \| timeout
112	)
113	```
114
115	### Errors and Retries
116
117	```python
118	from prometheus_client import Counter, Gauge
119
120	LLM_ERRORS = Counter(
121	"llm_errors_total",
122	"Errors returned by the LLM provider",
123	labelnames=["model", "provider", "error_type"], # error_type: rate_limit \| timeout \| 5xx \| auth
124	)
125
126	LLM_RETRIES = Counter(
127	"llm_retries_total",
128	"Retried LLM API calls",
129	labelnames=["model", "provider", "retry_reason"],
130	)
131
132	AGENT_ACTIVE_REQUESTS = Gauge(
133	"agent_active_requests",
134	"Number of agent requests currently in flight",
135	labelnames=["agent_name"],
136	)
137	```
138
139	---
140
141	## OpenTelemetry Integration
142
143	Use the OpenTelemetry Python SDK to create traces that capture every step of an agent turn: the top-level request, each LLM call, each tool execution, and retrieval operations.
144
145	### Setup
146
147	```python
148	# otel_setup.py
149	from opentelemetry import trace
150	from opentelemetry.sdk.trace import TracerProvider
151	from opentelemetry.sdk.trace.export import BatchSpanProcessor
152	from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
153	from opentelemetry.sdk.resources import Resource
154
155	def init_tracing(service_name: str, otlp_endpoint: str = "http://localhost:4317"):
156	resource = Resource.create({
157	"servi

People who liked this also viewed

6 picks

Type