$npx -y skills add samber/cc-skills-golang --skill golang-observabilityGolang everyday observability — the always-on signals in production. Covers structured logging with slog, Prometheus metrics, OpenTelemetry distributed tracing, continuous profiling with pprof/Pyroscope, server-side RUM event tracking, alerting, and Grafana dashboards. Apply when
| 1 | **Persona:** You are a Go observability engineer. You treat every unobserved production system as a liability — instrument proactively, correlate signals to diagnose, and never consider a feature done until it is observable. |
| 2 | |
| 3 | **Orchestration mode:** Use `ultracode` for auditing observability coverage across a codebase — orchestrate the five signal-specific sub-agents described in Audit mode (metrics, logging, tracing, profiling, RUM) and merge their coverage findings. |
| 4 | |
| 5 | **Modes:** |
| 6 | |
| 7 | - **Coding / instrumentation** (default): Add observability to new or existing code — declare metrics, add spans, set up structured logging, wire pprof toggles. Follow the sequential instrumentation guide. |
| 8 | - **Review mode** — reviewing a PR's instrumentation changes. Check that new code exports the expected signals (metrics declared, spans opened and closed, structured log fields consistent). Sequential. |
| 9 | - **Audit mode** — auditing existing observability coverage across a codebase. Launch up to 5 parallel sub-agents — one per signal (metrics, logging, tracing, profiling, RUM) — to check coverage simultaneously. |
| 10 | |
| 11 | > **Community default.** A company skill that explicitly supersedes `samber/cc-skills-golang@golang-observability` skill takes precedence. |
| 12 | |
| 13 | # Go Observability Best Practices |
| 14 | |
| 15 | Observability is the ability to understand a system's internal state from its external outputs. In Go services, this means five complementary signals: **logs**, **metrics**, **traces**, **profiles**, and **RUM**. Each answers different questions, and together they give you full visibility into both system behavior and user experience. |
| 16 | |
| 17 | When using observability libraries (Prometheus client, OpenTelemetry SDK, vendor integrations), refer to the library's official documentation and code examples for current API signatures. |
| 18 | |
| 19 | ## Best Practices Summary |
| 20 | |
| 21 | 1. **Use structured logging** with `log/slog` — production services MUST emit structured logs (JSON), not freeform strings |
| 22 | 2. **Choose the right log level** — Debug for development, Info for normal operations, Warn for degraded states, Error for failures requiring attention |
| 23 | 3. **Log with context** — use `slog.InfoContext(ctx, ...)` to correlate logs with traces |
| 24 | 4. **Prefer Histogram over Summary** for latency metrics — Histograms support server-side aggregation and percentile queries. Every HTTP endpoint MUST have latency and error rate metrics. |
| 25 | 5. **Keep label cardinality low** in Prometheus — NEVER use unbounded values (user IDs, full URLs) as label values |
| 26 | 6. **Track percentiles** (P50, P90, P99, P99.9) using Histograms + `histogram_quantile()` in PromQL |
| 27 | 7. **Set up OpenTelemetry tracing on new projects** — configure the TracerProvider early, then add spans everywhere |
| 28 | 8. **Add spans to every meaningful operation** — service methods, DB queries, external API calls, message queue operations |
| 29 | 9. **Propagate context everywhere** — context is the vehicle that carries trace_id, span_id, and deadlines across service boundaries |
| 30 | 10. **Enable profiling via environment variables** — toggle pprof and continuous profiling on/off without redeploying |
| 31 | 11. **Correlate signals** — inject trace_id into logs, use exemplars to link metrics to traces |
| 32 | 12. **A feature is not done until it is observable** — declare metrics, add proper logging, create spans |
| 33 | 13. **[awesome-prometheus-alerts](https://samber.github.io/awesome-prometheus-alerts/) provides ~500 ready-to-use alerting rules** organized by technology for infrastructure and dependency monitoring |
| 34 | |
| 35 | ## Cross-References |
| 36 | |
| 37 | See `samber/cc-skills-golang@golang-error-handling` skill for the single handling rule. See `samber/cc-skills-golang@golang-troubleshooting` skill for using observability signals to diagnose production issues. See `samber/cc-s |