observability-and-instrumentation

Name: observability-and-instrumentation
Author: addyosmani

8.3kinstalls8.6kforks

How to install observability-and-instrumentation?

addyosmani/agent-skills/observability-and-instrumentation

$npx -y skills add addyosmani/agent-skills --skill observability-and-instrumentation

Installs into the current project.

Summary

Instruments code so production behavior is visible and diagnosable. Use when adding logging, metrics, tracing, or alerting. Use when shipping any feature that runs in production and you need evidence it works. Use when production issues are reported but you can't tell what happen

Files · 1

View on GitHub

SKILL.md

1	# Observability and Instrumentation
2
3	## Overview
4
5	Code you can't observe is code you can't operate. Observability is the ability to answer "what is the system doing and why?" from the outside, using the telemetry the code emits. Instrumentation is not a post-launch add-on — it's written alongside the feature, the same way tests are. If a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.
6
7	## When to Use
8
9	- Building any feature that will run in production
10	- Adding a new service, endpoint, background job, or external integration
11	- A production incident took too long to diagnose ("we couldn't tell what happened")
12	- Setting up or reviewing alerting rules
13	- Reviewing a PR that adds I/O, retries, queues, or cross-service calls
14
15	NOT for:
16	- Diagnosing a failure happening right now — use the `debugging-and-error-recovery` skill (observability is what makes that skill fast next time)
17	- Profiling and optimizing measured slowness — use the `performance-optimization` skill
18	- Launch-day monitoring checklists and rollback triggers — see the `shipping-and-launch` skill; this skill covers the instrumentation that feeds them
19
20	## Process
21
22	### 1. Define "working" before instrumenting
23
24	Telemetry without a question is noise. Before adding any instrumentation, write down 2–4 questions an on-call engineer will ask about this feature:
25
26	```
27	FEATURE: checkout payment retry
28	QUESTIONS ON-CALL WILL ASK:
29	1. What fraction of payments succeed on first attempt vs after retry?
30	2. When a payment fails permanently, why? (provider error? timeout? validation?)
31	3. Is the payment provider slower than usual?
32	→ Every signal below must help answer one of these.
33	```
34
35	If you can't name the questions, you're not ready to instrument — you'll log everything and learn nothing.
36
37	### 2. Pick the right signal for each question
38
39	\| Signal \| Answers \| Cost profile \| Example \|
40	\|---\|---\|---\|---\|
41	\| Structured log \| "What happened in this specific case?" \| Per-event; grows with traffic \| `payment_failed` with provider error code \|
42	\| Metric \| "How often / how fast, in aggregate?" \| Fixed per series; cheap to query \| p99 latency of provider calls \|
43	\| Trace \| "Where did time go across services?" \| Per-request; usually sampled \| One slow checkout, broken down by hop \|
44
45	Rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.
46
47	### 3. Structured logging
48
49	Log events, not prose. Every log line is a JSON object with a stable event name and machine-readable fields:
50
51	```typescript
52	// BAD: string interpolation — unqueryable, inconsistent
53	logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);
54
55	// GOOD: stable event name + structured fields
56	logger.warn({
57	event: 'payment_failed',
58	paymentId: id,
59	provider: 'stripe',
60	errorCode: err.code,
61	attempt: n,
62	}, 'payment failed');
63	```
64
65	Log levels — use them consistently:
66
67	\| Level \| Meaning \| On-call action \|
68	\|---\|---\|---\|
69	\| `error` \| Invariant broken; someone may need to act \| Investigate \|
70	\| `warn` \| Degraded but handled (retry succeeded, fallback used) \| Watch for trends \|
71	\| `info` \| Significant business event (order placed, job finished) \| None \|
72	\| `debug` \| Diagnostic detail \| Off in production by default \|
73
74	Correlation IDs are mandatory. Generate (or accept) a request ID at the system boundary and attach it to every log line, span, and outbound call. Without it, you cannot reconstruct a single request from interleaved logs:
75
76	```typescript
77	// Express: child logger per request, ID propagated downstream
78	app.use((req, res, next) => {
79	req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
80	req.log = logger.child({ requestId: req.id });
81	res.setHeader('x-request-id', req.id);
82	next();
83	});
84	```
85
86	Never log secrets, tokens, passwords, or full PII. This is a hard rule from the `security-and-hardening` skill — telemetry pipelines are a classic data-leak path. Allowlist fields; don't log whole request bodies.
87
88	### 4. Metrics
89
90	For request-driven services, instrument RED on every endpoint and every external dependency: Rate (requests/sec), Errors (failure rate), Duration (latency histogram, not average). For resources (queues, pools, hosts), use USE: Utilization, Saturation, Errors.
91
92	As with tracing, the vendor-neutral path is the OpenTelemetry metrics API (same SDK and context as step 5). The example below uses Prometheus' `prom-client` — one common backend choice, not the only one; the RED/USE and cardinality rules are identical either way.
93
94	```typescript
95	import { Histogram } from 'prom-

People who liked this also viewed

6 picks

Type

observability-and-instrumentation

8.3kinstalls8.6kforks

View on GitHub

How to install observability-and-instrumentation?

addyosmani/agent-skills/observability-and-instrumentation

$npx -y skills add addyosmani/agent-skills --skill observability-and-instrumentation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Observability and Instrumentation
2
3	## Overview
4
5	Code you can't observe is code you can't operate. Observability is the ability to answer "what is the system doing and why?" from the outside, using the telemetry the code emits. Instrumentation is not a post-launch add-on — it's written alongside the feature, the same way tests are. If a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.
6
7	## When to Use
8
9	- Building any feature that will run in production
10	- Adding a new service, endpoint, background job, or external integration
11	- A production incident took too long to diagnose ("we couldn't tell what happened")
12	- Setting up or reviewing alerting rules
13	- Reviewing a PR that adds I/O, retries, queues, or cross-service calls
14
15	NOT for:
16	- Diagnosing a failure happening right now — use the `debugging-and-error-recovery` skill (observability is what makes that skill fast next time)
17	- Profiling and optimizing measured slowness — use the `performance-optimization` skill
18	- Launch-day monitoring checklists and rollback triggers — see the `shipping-and-launch` skill; this skill covers the instrumentation that feeds them
19
20	## Process
21
22	### 1. Define "working" before instrumenting
23
24	Telemetry without a question is noise. Before adding any instrumentation, write down 2–4 questions an on-call engineer will ask about this feature:
25
26	```
27	FEATURE: checkout payment retry
28	QUESTIONS ON-CALL WILL ASK:
29	1. What fraction of payments succeed on first attempt vs after retry?
30	2. When a payment fails permanently, why? (provider error? timeout? validation?)
31	3. Is the payment provider slower than usual?
32	→ Every signal below must help answer one of these.
33	```
34
35	If you can't name the questions, you're not ready to instrument — you'll log everything and learn nothing.
36
37	### 2. Pick the right signal for each question
38
39	\| Signal \| Answers \| Cost profile \| Example \|
40	\|---\|---\|---\|---\|
41	\| Structured log \| "What happened in this specific case?" \| Per-event; grows with traffic \| `payment_failed` with provider error code \|
42	\| Metric \| "How often / how fast, in aggregate?" \| Fixed per series; cheap to query \| p99 latency of provider calls \|
43	\| Trace \| "Where did time go across services?" \| Per-request; usually sampled \| One slow checkout, broken down by hop \|
44
45	Rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.
46
47	### 3. Structured logging
48
49	Log events, not prose. Every log line is a JSON object with a stable event name and machine-readable fields:
50
51	```typescript
52	// BAD: string interpolation — unqueryable, inconsistent
53	logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);
54
55	// GOOD: stable event name + structured fields
56	logger.warn({
57	event: 'payment_failed',
58	paymentId: id,
59	provider: 'stripe',
60	errorCode: err.code,
61	attempt: n,
62	}, 'payment failed');
63	```
64
65	Log levels — use them consistently:
66
67	\| Level \| Meaning \| On-call action \|
68	\|---\|---\|---\|
69	\| `error` \| Invariant broken; someone may need to act \| Investigate \|
70	\| `warn` \| Degraded but handled (retry succeeded, fallback used) \| Watch for trends \|
71	\| `info` \| Significant business event (order placed, job finished) \| None \|
72	\| `debug` \| Diagnostic detail \| Off in production by default \|
73
74	Correlation IDs are mandatory. Generate (or accept) a request ID at the system boundary and attach it to every log line, span, and outbound call. Without it, you cannot reconstruct a single request from interleaved logs:
75
76	```typescript
77	// Express: child logger per request, ID propagated downstream
78	app.use((req, res, next) => {
79	req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
80	req.log = logger.child({ requestId: req.id });
81	res.setHeader('x-request-id', req.id);
82	next();
83	});
84	```
85
86	Never log secrets, tokens, passwords, or full PII. This is a hard rule from the `security-and-hardening` skill — telemetry pipelines are a classic data-leak path. Allowlist fields; don't log whole request bodies.
87
88	### 4. Metrics
89
90	For request-driven services, instrument RED on every endpoint and every external dependency: Rate (requests/sec), Errors (failure rate), Duration (latency histogram, not average). For resources (queues, pools, hosts), use USE: Utilization, Saturation, Errors.
91
92	As with tracing, the vendor-neutral path is the OpenTelemetry metrics API (same SDK and context as step 5). The example below uses Prometheus' `prom-client` — one common backend choice, not the only one; the RED/USE and cardinality rules are identical either way.
93
94	```typescript
95	import { Histogram } from 'prom-

People who liked this also viewed

6 picks

Type