$npx -y skills add ancoleman/ai-design-components --skill implementing-observabilityMonitoring, logging, and tracing implementation using OpenTelemetry as the unified standard. Use when building production systems requiring visibility into performance, errors, and behavior. Covers OpenTelemetry (metrics, logs, traces), Prometheus, Grafana, Loki, Jaeger, Tempo, s
| 1 | # Production Observability with OpenTelemetry |
| 2 | |
| 3 | ## Purpose |
| 4 | |
| 5 | Implement production-grade observability using OpenTelemetry as the 2025 industry standard. Covers the three pillars (metrics, logs, traces), LGTM stack deployment, and critical log-trace correlation patterns. |
| 6 | |
| 7 | ## When to Use |
| 8 | |
| 9 | Use when: |
| 10 | - Building production systems requiring visibility into performance and errors |
| 11 | - Debugging distributed systems with multiple services |
| 12 | - Setting up monitoring, logging, or tracing infrastructure |
| 13 | - Implementing structured logging with trace correlation |
| 14 | - Configuring alerting rules for production systems |
| 15 | |
| 16 | Skip if: |
| 17 | - Building proof-of-concept without production deployment |
| 18 | - System has < 100 requests/day (console logging may suffice) |
| 19 | |
| 20 | ## The OpenTelemetry Standard (2025) |
| 21 | |
| 22 | OpenTelemetry is the CNCF graduated project unifying observability: |
| 23 | |
| 24 | ``` |
| 25 | ┌────────────────────────────────────────────────────────┐ |
| 26 | │ OpenTelemetry: The Unified Standard │ |
| 27 | ├────────────────────────────────────────────────────────┤ |
| 28 | │ │ |
| 29 | │ ONE SDK for ALL signals: │ |
| 30 | │ ├── Metrics (Prometheus-compatible) │ |
| 31 | │ ├── Logs (structured, correlated) │ |
| 32 | │ ├── Traces (distributed, standardized) │ |
| 33 | │ └── Context (propagates across services) │ |
| 34 | │ │ |
| 35 | │ Language SDKs: │ |
| 36 | │ ├── Python: opentelemetry-api, opentelemetry-sdk │ |
| 37 | │ ├── Rust: opentelemetry, tracing-opentelemetry │ |
| 38 | │ ├── Go: go.opentelemetry.io/otel │ |
| 39 | │ └── TypeScript: @opentelemetry/api │ |
| 40 | │ │ |
| 41 | │ Export to ANY backend: │ |
| 42 | │ ├── LGTM Stack (Loki, Grafana, Tempo, Mimir) │ |
| 43 | │ ├── Prometheus + Jaeger │ |
| 44 | │ ├── Datadog, New Relic, Honeycomb (SaaS) │ |
| 45 | │ └── Custom backends via OTLP protocol │ |
| 46 | │ │ |
| 47 | └────────────────────────────────────────────────────────┘ |
| 48 | ``` |
| 49 | |
| 50 | **Context7 Reference**: `/websites/opentelemetry_io` (Trust: High, Snippets: 5,888, Score: 85.9) |
| 51 | |
| 52 | ## The Three Pillars of Observability |
| 53 | |
| 54 | ### 1. Metrics (What is happening?) |
| 55 | |
| 56 | Track system health and performance over time. |
| 57 | |
| 58 | **Metric Types**: Counters (always increase), Gauges (up/down), Histograms (distributions), Summaries (percentiles). |
| 59 | |
| 60 | **Brief Example (Python)**: |
| 61 | ```python |
| 62 | from opentelemetry import metrics |
| 63 | |
| 64 | meter = metrics.get_meter(__name__) |
| 65 | http_requests = meter.create_counter("http.server.requests") |
| 66 | http_requests.add(1, {"method": "GET", "status": 200}) |
| 67 | ``` |
| 68 | |
| 69 | ### 2. Logs (What happened?) |
| 70 | |
| 71 | Record discrete events with context. |
| 72 | |
| 73 | **CRITICAL**: Always inject trace_id/span_id for log-trace correlation. |
| 74 | |
| 75 | **Brief Example (Python + structlog)**: |
| 76 | ```python |
| 77 | import structlog |
| 78 | from opentelemetry import trace |
| 79 | |
| 80 | logger = structlog.get_logger() |
| 81 | span = trace.get_current_span() |
| 82 | ctx = span.get_span_context() |
| 83 | |
| 84 | logger.info( |
| 85 | "processing_request", |
| 86 | trace_id=format(ctx.trace_id, '032x'), |
| 87 | span_id=format(ctx.span_id, '016x'), |
| 88 | user_id=user_id |
| 89 | ) |
| 90 | ``` |
| 91 | |
| 92 | **See**: `references/structured-logging.md` for complete configuration. |
| 93 | |
| 94 | ### 3. Traces (Where did time go?) |
| 95 | |
| 96 | Track request flow across distributed services. |
| 97 | |
| 98 | **Key Concepts**: Trace (end-to-end journey), Span (individual operation), Parent-Child (nested operations). |
| 99 | |
| 100 | **Brief Example (Python + FastAPI)**: |
| 101 | ```python |
| 102 | from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor |
| 103 | |
| 104 | app = FastAPI() |
| 105 | FastAPIInstrumentor.instrument_app(app) # Auto-traces all HTTP requests |
| 106 | ``` |
| 107 | |
| 108 | **See**: `references/opentelemetry-setup.md` for SDK installation by language. |
| 109 | |
| 110 | ## The LGTM Stack (Self-Hosted Observability) |
| 111 | |
| 112 | LGTM = **L**oki (Logs) + **G**rafana (Visualization) + **T**empo (Traces) + **M**imir (Metrics) |
| 113 | |
| 114 | ``` |
| 115 | ┌────────────────────────────────────────────────────────┐ |
| 116 | │ LGTM Architecture │ |
| 117 | ├────────────────────────────────────────────────────────┤ |
| 118 | │ │ |
| 119 | │ ┌──────────────────────────────────────────────┐ │ |
| 120 | │ │ Grafana Dashboard (Port 3000) │ │ |
| 121 | │ │ Unified UI for Logs, Metrics, Traces │ │ |
| 122 | │ └──────┬──────────────┬─────────────┬─────────┘ │ |
| 123 | │ │ │ │ │ |
| 124 | │ ▼ ▼ ▼ │ |
| 125 | │ ┌──────────┐ ┌──────────┐ ┌─── |