observability-and-instrumentation

Name: observability-and-instrumentation
Author: jmxt3

32stars3forks

How to install observability-and-instrumentation?

jmxt3/gitscape.ai/observability-and-instrumentation

$npx -y skills add jmxt3/gitscape.ai --skill observability-and-instrumentation

Installs into the current project.

Summary

Structured logging, RED metrics, and OpenTelemetry tracing. Use when adding telemetry, or shipping anything that runs in production. Instrument as you build — not after incidents.

Files · 1

View on GitHub

SKILL.md

1	# Observability and Instrumentation
2
3	## Overview
4
5	Instrument production code so that on-call engineers (including future AI agents) can answer: Is it working? What's broken? Why? Logs answer why. Metrics answer that something is wrong. Traces answer where. Build all three in as you go — retrofitting observability after an incident is painful and late.
6
7	## When to Use
8
9	- Adding a new API endpoint or background job
10	- Shipping a feature that touches external services (GitHub API, Gemini)
11	- Modifying error handling or retry logic
12	- Any change to Cloud Run that affects request processing
13
14	## The Three Pillars
15
16	### 1. Structured Logging
17
18	Every log line should be machine-parseable JSON, not a free-form string.
19
20	```python
21	import structlog
22
23	log = structlog.get_logger()
24
25	# GOOD: Structured, queryable
26	log.info(
27	"skill_generation_started",
28	repo=repo,
29	tier=tier,
30	request_id=request_id,
31	)
32
33	log.error(
34	"github_api_error",
35	repo=repo,
36	status_code=e.status,
37	error=str(e),
38	request_id=request_id,
39	)
40
41	# BAD: Free-form string — unsearchable
42	print(f"Error fetching {repo}: {e}")
43	logger.info("Starting skill generation for " + repo)
44	```
45
46	Log level conventions:
47	- `error` — invariant broken, someone may need to act
48	- `warn` — degraded but handled (rate limit approaching, retrying)
49	- `info` — significant business event (skill generated, export downloaded)
50	- `debug` — off in production; verbose tracing for local debugging
51
52	Never log:
53	- Secrets, API keys, or tokens
54	- Full request/response bodies
55	- Unredacted PII or GitHub personal access tokens
56
57	### 2. RED Metrics for Every Endpoint
58
59	For every API endpoint, instrument:
60	- Rate — requests per second
61	- Errors — error rate (%)
62	- Duration — p50/p95/p99 latency (histogram, never average)
63
64	```python
65	from prometheus_client import Counter, Histogram
66
67	skill_generation_requests = Counter(
68	"skill_generation_requests_total",
69	"Total skill generation requests",
70	["tier", "status"] # labels: tier=standard\|hd, status=success\|error
71	)
72
73	skill_generation_duration = Histogram(
74	"skill_generation_duration_seconds",
75	"Skill generation duration",
76	["tier"],
77	buckets=[0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0]
78	)
79
80	# Usage
81	with skill_generation_duration.labels(tier=tier).time():
82	try:
83	result = await generate_skill(repo, tier)
84	skill_generation_requests.labels(tier=tier, status="success").inc()
85	except Exception as e:
86	skill_generation_requests.labels(tier=tier, status="error").inc()
87	raise
88	```
89
90	### 3. Correlation IDs
91
92	Every request should carry a correlation ID that propagates through all log lines and outbound calls:
93
94	```python
95	import uuid
96	from fastapi import Request
97
98	@app.middleware("http")
99	async def add_correlation_id(request: Request, call_next):
100	request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
101	# Bind to structlog context so all logs in this request carry it
102	with structlog.contextvars.bound_contextvars(request_id=request_id):
103	response = await call_next(request)
104	response.headers["X-Request-ID"] = request_id
105	return response
106	```
107
108	## Alerting Principles
109
110	Alert on symptoms, not causes. Users experience symptoms.
111
112	\| Wrong (cause-based alert) \| Right (symptom-based alert) \|
113	\|---\|---\|
114	\| "CPU > 80%" \| "Error rate > 1% for 5 minutes" \|
115	\| "GitHub API calls failing" \| "Skill generation success rate < 95%" \|
116	\| "Memory usage high" \| "p95 latency > 5s" \|
117
118	Actionable alerts only. If an alert fires and you don't know what to do, the alert is not ready.
119
120	## Cloud Logging on Cloud Run
121
122	GitScape deploys to Cloud Run. Logs written to stdout/stderr are automatically captured by Cloud Logging:
123
124	```python
125	# structlog outputs JSON to stdout → Cloud Logging ingests automatically
126	structlog.configure(
127	processors=[
128	structlog.stdlib.add_log_level,
129	structlog.processors.TimeStamper(fmt="iso"),
130	structlog.processors.JSONRenderer(),
131	],
132	wrapper_class=structlog.stdlib.BoundLogger,
133	logger_factory=structlog.PrintLoggerFactory(),
134	)
135	```
136
137	Query logs in Cloud Logging:
138	```
139	resource.type="cloud_run_revision"
140	resource.labels.service_name="gitscape-api"
141	jsonPayload.event="skill_generation_started"
142	```
143
144	## GitScape-Specific Instrumentation Checklist
145
146	For every new endpoint in `api/app/`:
147
148	- [ ] Log `skill_generation_started` (or equivalent) with `repo`, `tier`, `request_id`
149	- [ ] Log `skill_generation_completed` with `repo`, `tier`, `duration_ms`
150	- [ ] Log `skill_generation_failed` with `repo`, `tier`, `error`, `request_id`
151	- [ ] External API calls (GitHub, Gemini) logged with `endpoint`, `status_code`, `latency_ms`
152	- [ ] No secrets or tokens in any log line
153
154	## Common Rationalizations
155
156	\| Rationalization \|

People who liked this also viewed

6 picks

Type

observability-and-instrumentation

32stars3forks

View on GitHub

How to install observability-and-instrumentation?

jmxt3/gitscape.ai/observability-and-instrumentation

$npx -y skills add jmxt3/gitscape.ai --skill observability-and-instrumentation

Installs into the current project.

Summary

Structured logging, RED metrics, and OpenTelemetry tracing. Use when adding telemetry, or shipping anything that runs in production. Instrument as you build — not after incidents.

Files · 1

View on GitHub

SKILL.md

1	# Observability and Instrumentation
2
3	## Overview
4
5	Instrument production code so that on-call engineers (including future AI agents) can answer: Is it working? What's broken? Why? Logs answer why. Metrics answer that something is wrong. Traces answer where. Build all three in as you go — retrofitting observability after an incident is painful and late.
6
7	## When to Use
8
9	- Adding a new API endpoint or background job
10	- Shipping a feature that touches external services (GitHub API, Gemini)
11	- Modifying error handling or retry logic
12	- Any change to Cloud Run that affects request processing
13
14	## The Three Pillars
15
16	### 1. Structured Logging
17
18	Every log line should be machine-parseable JSON, not a free-form string.
19
20	```python
21	import structlog
22
23	log = structlog.get_logger()
24
25	# GOOD: Structured, queryable
26	log.info(
27	"skill_generation_started",
28	repo=repo,
29	tier=tier,
30	request_id=request_id,
31	)
32
33	log.error(
34	"github_api_error",
35	repo=repo,
36	status_code=e.status,
37	error=str(e),
38	request_id=request_id,
39	)
40
41	# BAD: Free-form string — unsearchable
42	print(f"Error fetching {repo}: {e}")
43	logger.info("Starting skill generation for " + repo)
44	```
45
46	Log level conventions:
47	- `error` — invariant broken, someone may need to act
48	- `warn` — degraded but handled (rate limit approaching, retrying)
49	- `info` — significant business event (skill generated, export downloaded)
50	- `debug` — off in production; verbose tracing for local debugging
51
52	Never log:
53	- Secrets, API keys, or tokens
54	- Full request/response bodies
55	- Unredacted PII or GitHub personal access tokens
56
57	### 2. RED Metrics for Every Endpoint
58
59	For every API endpoint, instrument:
60	- Rate — requests per second
61	- Errors — error rate (%)
62	- Duration — p50/p95/p99 latency (histogram, never average)
63
64	```python
65	from prometheus_client import Counter, Histogram
66
67	skill_generation_requests = Counter(
68	"skill_generation_requests_total",
69	"Total skill generation requests",
70	["tier", "status"] # labels: tier=standard\|hd, status=success\|error
71	)
72
73	skill_generation_duration = Histogram(
74	"skill_generation_duration_seconds",
75	"Skill generation duration",
76	["tier"],
77	buckets=[0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0]
78	)
79
80	# Usage
81	with skill_generation_duration.labels(tier=tier).time():
82	try:
83	result = await generate_skill(repo, tier)
84	skill_generation_requests.labels(tier=tier, status="success").inc()
85	except Exception as e:
86	skill_generation_requests.labels(tier=tier, status="error").inc()
87	raise
88	```
89
90	### 3. Correlation IDs
91
92	Every request should carry a correlation ID that propagates through all log lines and outbound calls:
93
94	```python
95	import uuid
96	from fastapi import Request
97
98	@app.middleware("http")
99	async def add_correlation_id(request: Request, call_next):
100	request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
101	# Bind to structlog context so all logs in this request carry it
102	with structlog.contextvars.bound_contextvars(request_id=request_id):
103	response = await call_next(request)
104	response.headers["X-Request-ID"] = request_id
105	return response
106	```
107
108	## Alerting Principles
109
110	Alert on symptoms, not causes. Users experience symptoms.
111
112	\| Wrong (cause-based alert) \| Right (symptom-based alert) \|
113	\|---\|---\|
114	\| "CPU > 80%" \| "Error rate > 1% for 5 minutes" \|
115	\| "GitHub API calls failing" \| "Skill generation success rate < 95%" \|
116	\| "Memory usage high" \| "p95 latency > 5s" \|
117
118	Actionable alerts only. If an alert fires and you don't know what to do, the alert is not ready.
119
120	## Cloud Logging on Cloud Run
121
122	GitScape deploys to Cloud Run. Logs written to stdout/stderr are automatically captured by Cloud Logging:
123
124	```python
125	# structlog outputs JSON to stdout → Cloud Logging ingests automatically
126	structlog.configure(
127	processors=[
128	structlog.stdlib.add_log_level,
129	structlog.processors.TimeStamper(fmt="iso"),
130	structlog.processors.JSONRenderer(),
131	],
132	wrapper_class=structlog.stdlib.BoundLogger,
133	logger_factory=structlog.PrintLoggerFactory(),
134	)
135	```
136
137	Query logs in Cloud Logging:
138	```
139	resource.type="cloud_run_revision"
140	resource.labels.service_name="gitscape-api"
141	jsonPayload.event="skill_generation_started"
142	```
143
144	## GitScape-Specific Instrumentation Checklist
145
146	For every new endpoint in `api/app/`:
147
148	- [ ] Log `skill_generation_started` (or equivalent) with `repo`, `tier`, `request_id`
149	- [ ] Log `skill_generation_completed` with `repo`, `tier`, `duration_ms`
150	- [ ] Log `skill_generation_failed` with `repo`, `tier`, `error`, `request_id`
151	- [ ] External API calls (GitHub, Gemini) logged with `endpoint`, `status_code`, `latency_ms`
152	- [ ] No secrets or tokens in any log line
153
154	## Common Rationalizations
155
156	\| Rationalization \|

People who liked this also viewed

6 picks

Type