observability

Name: observability
Author: proyecto26

47installs1forks

How to install observability?

proyecto26/system-design-skills/observability

$npx -y skills add proyecto26/system-design-skills --skill observability

Installs into the current project.

Summary

This skill should be used when the user asks about "observability" or "monitoring", what "metrics, logs, and traces" to collect, "health checks" (liveness/readiness), "alerting" or "on-call", "SLO/SLI" or "error budgets", the "RED" or "USE" method, "dashboards", or names a tool l

Files · 1

View on GitHub

SKILL.md

1	# Observability
2
3	Decide what to measure so a system can be seen, alerted on, and debugged in
4	production. Getting this wrong is failure mode #6 — ignoring failure: a design
5	that works on the whiteboard but goes dark under load, where the first signal of
6	an outage is a user complaint instead of a page.
7
8	## When to reach for this
9	Any production design needs an answer to "how would we know this broke, and how
10	fast?" Reach for this when defining what the system measures, what pages a human,
11	what an acceptable level of service is (SLO), or how a request is traced across
12	services. It is the design move that makes every other block's stress section
13	real — you cannot mitigate a thundering herd or a hot shard you can't see.
14
15	## When NOT to
16	Do not build a full metrics-logs-traces stack for a prototype or an internal tool
17	with no users to disappoint (YAGNI) — a health check and error logging are
18	enough. Do not invent SLOs nobody will defend, or wire alerts before knowing the
19	symptom that matters; an alert with no owner and no runbook is noise that trains
20	the team to ignore pages. This skill owns what to measure and alert on; the
21	high-volume log pipeline (collect → buffer → ship → index → retain) lives in
22	`distributed-logging` — summarize and link, don't rebuild it here.
23
24	## Clarify first
25	- What is "healthy" from a user's view? The symptom that defines a bad
26	experience (slow checkout, failed upload) — alerts target this, not CPU.
27	- SLO target and window? e.g. 99.9% of requests < 300 ms over 30 days. This
28	sets the error budget and the alert thresholds. (→ `back-of-the-envelope` for the nines.)
29	- Request volume and cardinality? QPS drives metric/trace sample rates;
30	high-cardinality labels (user ID, URL) blow up a metrics store.
31	- Is this request-driven or resource-driven? Picks RED (services) vs USE
32	(CPU/disk/queues) as the measurement frame.
33	- Multi-service request path? If a request crosses services, tracing earns
34	its keep; a single service may not need it yet.
35
36	## The options
37
38	The three pillars (complementary, not either/or):
39	- Metrics — cheap numeric time series (counters, gauges, histograms). Use for
40	dashboards, trend analysis, and alerting — the always-on signal.
41	- Logs — discrete events with context. Use for debugging the specific failure
42	after an alert fires. (The pipeline that moves them is `distributed-logging`.)
43	- Traces — one request's journey across services with timing per hop. Use to
44	find which service or dependency is the latency/error source.
45
46	What to measure (pick a frame per component):
47	- RED (request-driven services): Rate, Errors, Duration. Use for
48	APIs, web tiers, anything serving requests.
49	- USE (resources): Utilization, Saturation, Errors. Use for CPU,
50	memory, disk, connection pools, queues.
51	- Four golden signals (latency, traffic, errors, saturation) — the superset;
52	use as the default service dashboard.
53
54	Health checks (the signal, consumed by `load-balancing`/orchestrator):
55	- Liveness — is the process alive / not deadlocked? Failure ⇒ restart. Keep
56	it cheap; don't check dependencies. Use to recover stuck processes.
57	- Readiness — can this instance serve traffic now (deps reachable, warmup
58	done)? Failure ⇒ pull from rotation, don't restart. Use to gate cold/struggling instances.
59
60	Alerting:
61	- Symptom-based (SLO burn) — page when the user-facing SLO is at risk. Use as
62	the default; it is actionable and low-noise.
63	- Cause-based (resource thresholds) — ticket/warn on CPU, disk, saturation.
64	Use for capacity planning, not paging.
65
66	## Trade-offs
67
68	\| Option \| What it solves \| What it worsens \| Change it when \|
69	\|---\|---\|---\|---\|
70	\| Metrics \| Cheap, always-on alerting + trends \| No per-request detail; high-cardinality labels explode storage/cost \| You need to debug a specific request → add traces/logs \|
71	\| Logs \| Rich context for debugging an incident \| Volume + cost; needs the `distributed-logging` pipeline to scale \| Volume is unmanageable → sample, or shift detail to metrics \|
72	\| Traces \| Pinpoints the slow/failing hop across services \| Instrumentation effort; sampling needed at high QPS \| Single-service or low volume → defer; metrics suffice \|
73	\| RED \| Right frame for request services \| Misses resource exhaustion that hasn't yet hurt requests \| Component is a resource (queue, disk) → use USE \|

People who liked this also viewed

6 picks

Type

observability

47installs1forks

View on GitHub

How to install observability?

proyecto26/system-design-skills/observability

$npx -y skills add proyecto26/system-design-skills --skill observability

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Observability
2
3	Decide what to measure so a system can be seen, alerted on, and debugged in
4	production. Getting this wrong is failure mode #6 — ignoring failure: a design
5	that works on the whiteboard but goes dark under load, where the first signal of
6	an outage is a user complaint instead of a page.
7
8	## When to reach for this
9	Any production design needs an answer to "how would we know this broke, and how
10	fast?" Reach for this when defining what the system measures, what pages a human,
11	what an acceptable level of service is (SLO), or how a request is traced across
12	services. It is the design move that makes every other block's stress section
13	real — you cannot mitigate a thundering herd or a hot shard you can't see.
14
15	## When NOT to
16	Do not build a full metrics-logs-traces stack for a prototype or an internal tool
17	with no users to disappoint (YAGNI) — a health check and error logging are
18	enough. Do not invent SLOs nobody will defend, or wire alerts before knowing the
19	symptom that matters; an alert with no owner and no runbook is noise that trains
20	the team to ignore pages. This skill owns what to measure and alert on; the
21	high-volume log pipeline (collect → buffer → ship → index → retain) lives in
22	`distributed-logging` — summarize and link, don't rebuild it here.
23
24	## Clarify first
25	- What is "healthy" from a user's view? The symptom that defines a bad
26	experience (slow checkout, failed upload) — alerts target this, not CPU.
27	- SLO target and window? e.g. 99.9% of requests < 300 ms over 30 days. This
28	sets the error budget and the alert thresholds. (→ `back-of-the-envelope` for the nines.)
29	- Request volume and cardinality? QPS drives metric/trace sample rates;
30	high-cardinality labels (user ID, URL) blow up a metrics store.
31	- Is this request-driven or resource-driven? Picks RED (services) vs USE
32	(CPU/disk/queues) as the measurement frame.
33	- Multi-service request path? If a request crosses services, tracing earns
34	its keep; a single service may not need it yet.
35
36	## The options
37
38	The three pillars (complementary, not either/or):
39	- Metrics — cheap numeric time series (counters, gauges, histograms). Use for
40	dashboards, trend analysis, and alerting — the always-on signal.
41	- Logs — discrete events with context. Use for debugging the specific failure
42	after an alert fires. (The pipeline that moves them is `distributed-logging`.)
43	- Traces — one request's journey across services with timing per hop. Use to
44	find which service or dependency is the latency/error source.
45
46	What to measure (pick a frame per component):
47	- RED (request-driven services): Rate, Errors, Duration. Use for
48	APIs, web tiers, anything serving requests.
49	- USE (resources): Utilization, Saturation, Errors. Use for CPU,
50	memory, disk, connection pools, queues.
51	- Four golden signals (latency, traffic, errors, saturation) — the superset;
52	use as the default service dashboard.
53
54	Health checks (the signal, consumed by `load-balancing`/orchestrator):
55	- Liveness — is the process alive / not deadlocked? Failure ⇒ restart. Keep
56	it cheap; don't check dependencies. Use to recover stuck processes.
57	- Readiness — can this instance serve traffic now (deps reachable, warmup
58	done)? Failure ⇒ pull from rotation, don't restart. Use to gate cold/struggling instances.
59
60	Alerting:
61	- Symptom-based (SLO burn) — page when the user-facing SLO is at risk. Use as
62	the default; it is actionable and low-noise.
63	- Cause-based (resource thresholds) — ticket/warn on CPU, disk, saturation.
64	Use for capacity planning, not paging.
65
66	## Trade-offs
67
68	\| Option \| What it solves \| What it worsens \| Change it when \|
69	\|---\|---\|---\|---\|
70	\| Metrics \| Cheap, always-on alerting + trends \| No per-request detail; high-cardinality labels explode storage/cost \| You need to debug a specific request → add traces/logs \|
71	\| Logs \| Rich context for debugging an incident \| Volume + cost; needs the `distributed-logging` pipeline to scale \| Volume is unmanageable → sample, or shift detail to metrics \|
72	\| Traces \| Pinpoints the slow/failing hop across services \| Instrumentation effort; sampling needed at high QPS \| Single-service or low volume → defer; metrics suffice \|
73	\| RED \| Right frame for request services \| Misses resource exhaustion that hasn't yet hurt requests \| Component is a resource (queue, disk) → use USE \|

People who liked this also viewed

6 picks

Type